2023-10-30T12:10:55.618 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-10-30T12:10:55.619 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} --verbose --name leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi --owner scheduled_leonidus@teuthology --archive /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441018 -- /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441018/orig.config.yaml 2023-10-30T12:10:55.671 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-30T12:10:55.758 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441018 branch: wip-lusov-balance-after-stop description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} email: null first_in_suite: false job_id: '7441018' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-lusov-balance-after-stop ceph: cephfs: max_mds: 3 session_timeout: 300 conf: client: client mount timeout: 600 client_shutdown_timeout: 120 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: true mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: false mds_sleep_rank_change: 5000000.0 mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds osd ms inject socket failures: 2500 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon client directed command retry: 5 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds ms inject socket failures: 2500 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - Replacing daemon mds sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 kclient: syntax: v1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-balance-after-stop sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 owner: scheduled_leonidus@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 8397 sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 sleep_before_teardown: 0 subset: 464/1024 suite: fs suite_branch: wip-lusov-balance-after-stop suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi089.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOag3my8IamJNicmi37LZgXQEl7nS8WHId2QtJcbtUnA2zSfW/3voFcVlZ9NqpZOgjD5TywEhthoXXM1YcaRGLg= smithi192.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJcRRf5xTajm0BZJQJLWhF2zIJij3ZMLY/mNaetV4sVgWPO7jUH10GB2CXSpPhG7M4/lSP5SCEra8BopJzAdnzY= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - mds_thrash: null - workunit: clients: all: - fs/snaps timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-10-30_06:34:16 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.770726 2023-10-30T12:10:55.759 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa; will attempt to use it 2023-10-30T12:10:55.759 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks 2023-10-30T12:10:55.760 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-30T12:10:55.761 INFO:teuthology.task.internal:Checking packages... 2023-10-30T12:10:55.786 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '00c1c13cea21a686b046584596fbeac5315a7d71' 2023-10-30T12:10:55.786 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-30T12:10:55.787 INFO:teuthology.packaging:ref: None 2023-10-30T12:10:55.787 INFO:teuthology.packaging:tag: None 2023-10-30T12:10:55.787 INFO:teuthology.packaging:branch: wip-lusov-balance-after-stop 2023-10-30T12:10:55.787 INFO:teuthology.packaging:sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T12:10:55.788 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-lusov-balance-after-stop 2023-10-30T12:10:56.007 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6835-g00c1c13c-1jammy 2023-10-30T12:10:56.008 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-30T12:10:56.021 INFO:teuthology.task.internal:no buildpackages task found 2023-10-30T12:10:56.022 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-30T12:10:56.035 INFO:teuthology.task.internal:Saving configuration 2023-10-30T12:10:56.056 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-30T12:10:56.065 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T12:10:56.091 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441018', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 12:05:16.663906', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFxkV+yA/lYScqE2+W3ZgbwTedPqIXmLZgnca/qtDloJ'} 2023-10-30T12:10:56.115 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441018', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 12:05:16.662130', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKAtT2njF3CsjtuOlO/fvg37VpbzGBIsSR69FYBNxPPW'} 2023-10-30T12:10:56.116 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-30T12:10:56.127 INFO:teuthology.task.internal:roles: ubuntu@smithi089.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2023-10-30T12:10:56.127 INFO:teuthology.task.internal:roles: ubuntu@smithi192.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-10-30T12:10:56.127 INFO:teuthology.run_tasks:Running task console_log... 2023-10-30T12:10:56.204 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4aacd04c10>, signals=[15]) 2023-10-30T12:10:56.205 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-30T12:10:56.212 INFO:teuthology.task.internal:Opening connections... 2023-10-30T12:10:56.213 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi089.front.sepia.ceph.com 2023-10-30T12:10:56.215 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T12:10:56.290 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi192.front.sepia.ceph.com 2023-10-30T12:10:56.291 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T12:10:56.370 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-30T12:10:56.395 DEBUG:teuthology.orchestra.run.smithi089:> uname -m 2023-10-30T12:10:56.403 INFO:teuthology.orchestra.run.smithi089.stdout:x86_64 2023-10-30T12:10:56.403 DEBUG:teuthology.orchestra.run.smithi089:> cat /etc/os-release 2023-10-30T12:10:56.451 INFO:teuthology.orchestra.run.smithi089.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-30T12:10:56.451 INFO:teuthology.orchestra.run.smithi089.stdout:NAME="Ubuntu" 2023-10-30T12:10:56.452 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_ID="22.04" 2023-10-30T12:10:56.452 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-30T12:10:56.452 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_CODENAME=jammy 2023-10-30T12:10:56.452 INFO:teuthology.orchestra.run.smithi089.stdout:ID=ubuntu 2023-10-30T12:10:56.452 INFO:teuthology.orchestra.run.smithi089.stdout:ID_LIKE=debian 2023-10-30T12:10:56.453 INFO:teuthology.orchestra.run.smithi089.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-30T12:10:56.453 INFO:teuthology.orchestra.run.smithi089.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-30T12:10:56.453 INFO:teuthology.orchestra.run.smithi089.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-30T12:10:56.453 INFO:teuthology.orchestra.run.smithi089.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-30T12:10:56.453 INFO:teuthology.orchestra.run.smithi089.stdout:UBUNTU_CODENAME=jammy 2023-10-30T12:10:56.454 INFO:teuthology.lock.ops:Updating smithi089.front.sepia.ceph.com on lock server 2023-10-30T12:10:56.479 DEBUG:teuthology.orchestra.run.smithi192:> uname -m 2023-10-30T12:10:56.485 INFO:teuthology.orchestra.run.smithi192.stdout:x86_64 2023-10-30T12:10:56.486 DEBUG:teuthology.orchestra.run.smithi192:> cat /etc/os-release 2023-10-30T12:10:56.533 INFO:teuthology.orchestra.run.smithi192.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-30T12:10:56.533 INFO:teuthology.orchestra.run.smithi192.stdout:NAME="Ubuntu" 2023-10-30T12:10:56.534 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_ID="22.04" 2023-10-30T12:10:56.534 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-30T12:10:56.534 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_CODENAME=jammy 2023-10-30T12:10:56.534 INFO:teuthology.orchestra.run.smithi192.stdout:ID=ubuntu 2023-10-30T12:10:56.534 INFO:teuthology.orchestra.run.smithi192.stdout:ID_LIKE=debian 2023-10-30T12:10:56.535 INFO:teuthology.orchestra.run.smithi192.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-30T12:10:56.535 INFO:teuthology.orchestra.run.smithi192.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-30T12:10:56.535 INFO:teuthology.orchestra.run.smithi192.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-30T12:10:56.535 INFO:teuthology.orchestra.run.smithi192.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-30T12:10:56.536 INFO:teuthology.orchestra.run.smithi192.stdout:UBUNTU_CODENAME=jammy 2023-10-30T12:10:56.536 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2023-10-30T12:10:56.562 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-30T12:10:56.575 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-30T12:10:56.585 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-30T12:10:56.586 DEBUG:teuthology.orchestra.run.smithi089:> test '!' -e /home/ubuntu/cephtest 2023-10-30T12:10:56.588 DEBUG:teuthology.orchestra.run.smithi192:> test '!' -e /home/ubuntu/cephtest 2023-10-30T12:10:56.593 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-30T12:10:56.636 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-30T12:10:56.637 DEBUG:teuthology.orchestra.run.smithi089:> test -z $(ls -A /var/lib/ceph) 2023-10-30T12:10:56.639 DEBUG:teuthology.orchestra.run.smithi192:> test -z $(ls -A /var/lib/ceph) 2023-10-30T12:10:56.653 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-30T12:10:56.703 INFO:teuthology.run_tasks:Running task kernel... 2023-10-30T12:10:56.719 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2023-10-30T12:10:56.720 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2023-10-30T12:10:56.720 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2023-10-30T12:10:56.721 DEBUG:teuthology.orchestra.run.smithi089:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-30T12:10:56.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:10:56.727 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=testing 2023-10-30T12:10:56.969 DEBUG:teuthology.task.kernel:sha1 for client.0 is eacf00cc51e204f401b4d5d4f84825a68f373c74 2023-10-30T12:10:56.970 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "eacf00cc51e204f401b4d5d4f84825a68f373c74"... 2023-10-30T12:10:56.970 DEBUG:teuthology.orchestra.run.smithi089:> uname -r 2023-10-30T12:10:56.976 INFO:teuthology.orchestra.run.smithi089.stdout:5.15.0-56-generic 2023-10-30T12:10:56.976 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs eacf00cc51e204f401b4d5d4f84825a68f373c74 2023-10-30T12:10:56.977 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-10-30T12:10:56.977 INFO:teuthology.task.kernel:Installing linux-firmware on client.0... 2023-10-30T12:10:56.977 DEBUG:teuthology.orchestra.run.smithi089:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2023-10-30T12:10:57.034 INFO:teuthology.orchestra.run.smithi089.stderr:bash: line 1: cd: /lib/firmware/updates: No such file or directory 2023-10-30T12:10:57.212 INFO:teuthology.orchestra.run.smithi089.stderr:hint: Using 'master' as the name for the initial branch. This default branch name 2023-10-30T12:10:57.212 INFO:teuthology.orchestra.run.smithi089.stderr:hint: is subject to change. To configure the initial branch name to use in all 2023-10-30T12:10:57.212 INFO:teuthology.orchestra.run.smithi089.stderr:hint: of your new repositories, which will suppress this warning, call: 2023-10-30T12:10:57.213 INFO:teuthology.orchestra.run.smithi089.stderr:hint: 2023-10-30T12:10:57.213 INFO:teuthology.orchestra.run.smithi089.stderr:hint: git config --global init.defaultBranch 2023-10-30T12:10:57.213 INFO:teuthology.orchestra.run.smithi089.stderr:hint: 2023-10-30T12:10:57.213 INFO:teuthology.orchestra.run.smithi089.stderr:hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2023-10-30T12:10:57.213 INFO:teuthology.orchestra.run.smithi089.stderr:hint: 'development'. The just-created branch can be renamed via this command: 2023-10-30T12:10:57.214 INFO:teuthology.orchestra.run.smithi089.stderr:hint: 2023-10-30T12:10:57.214 INFO:teuthology.orchestra.run.smithi089.stderr:hint: git branch -m 2023-10-30T12:10:57.214 INFO:teuthology.orchestra.run.smithi089.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2023-10-30T12:10:57.216 DEBUG:teuthology.orchestra.run.smithi089:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2023-10-30T12:10:57.305 DEBUG:teuthology.orchestra.run.smithi089:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2023-10-30T12:10:57.319 DEBUG:teuthology.orchestra.run.smithi089:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2023-10-30T12:13:49.683 INFO:teuthology.orchestra.run.smithi089.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2023-10-30T12:13:49.684 INFO:teuthology.orchestra.run.smithi089.stderr: * [new branch] main -> origin/main 2023-10-30T12:13:49.714 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20190312 -> 20190312 2023-10-30T12:13:49.714 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20190416 -> 20190416 2023-10-30T12:13:49.714 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20190514 -> 20190514 2023-10-30T12:13:49.714 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20190618 -> 20190618 2023-10-30T12:13:49.715 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20190717 -> 20190717 2023-10-30T12:13:49.715 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20190815 -> 20190815 2023-10-30T12:13:49.715 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20190923 -> 20190923 2023-10-30T12:13:49.715 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20191022 -> 20191022 2023-10-30T12:13:49.715 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20191215 -> 20191215 2023-10-30T12:13:49.716 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20200122 -> 20200122 2023-10-30T12:13:49.716 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20200316 -> 20200316 2023-10-30T12:13:49.719 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20200421 -> 20200421 2023-10-30T12:13:49.720 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20200519 -> 20200519 2023-10-30T12:13:49.720 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20200619 -> 20200619 2023-10-30T12:13:49.720 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20200721 -> 20200721 2023-10-30T12:13:49.720 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20200817 -> 20200817 2023-10-30T12:13:49.720 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20200918 -> 20200918 2023-10-30T12:13:49.721 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20201022 -> 20201022 2023-10-30T12:13:49.721 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20201118 -> 20201118 2023-10-30T12:13:49.721 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20201218 -> 20201218 2023-10-30T12:13:49.721 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20210208 -> 20210208 2023-10-30T12:13:49.721 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20210315 -> 20210315 2023-10-30T12:13:49.722 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20210511 -> 20210511 2023-10-30T12:13:49.722 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20210716 -> 20210716 2023-10-30T12:13:49.722 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20210818 -> 20210818 2023-10-30T12:13:49.722 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20210919 -> 20210919 2023-10-30T12:13:49.722 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20211027 -> 20211027 2023-10-30T12:13:49.723 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20211216 -> 20211216 2023-10-30T12:13:49.723 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20220209 -> 20220209 2023-10-30T12:13:49.723 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20220310 -> 20220310 2023-10-30T12:13:49.723 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20220411 -> 20220411 2023-10-30T12:13:49.723 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20220509 -> 20220509 2023-10-30T12:13:49.724 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20220610 -> 20220610 2023-10-30T12:13:49.724 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20220708 -> 20220708 2023-10-30T12:13:49.724 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20220815 -> 20220815 2023-10-30T12:13:49.724 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20220913 -> 20220913 2023-10-30T12:13:49.724 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20221012 -> 20221012 2023-10-30T12:13:49.725 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20221109 -> 20221109 2023-10-30T12:13:49.725 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20221214 -> 20221214 2023-10-30T12:13:49.725 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20230117 -> 20230117 2023-10-30T12:13:49.725 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20230210 -> 20230210 2023-10-30T12:13:49.725 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20230310 -> 20230310 2023-10-30T12:13:49.726 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20230404 -> 20230404 2023-10-30T12:13:49.726 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20230515 -> 20230515 2023-10-30T12:13:49.726 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20230625 -> 20230625 2023-10-30T12:13:49.726 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20230804 -> 20230804 2023-10-30T12:13:49.726 INFO:teuthology.orchestra.run.smithi089.stderr: * [new tag] 20230919 -> 20230919 2023-10-30T12:13:53.424 INFO:teuthology.orchestra.run.smithi089.stderr:Updating files: 51% (1640/3168) Updating files: 52% (1648/3168) Updating files: 53% (1680/3168) Updating files: 54% (1711/3168) Updating files: 55% (1743/3168) Updating files: 56% (1775/3168) Updating files: 57% (1806/3168) Updating files: 58% (1838/3168) Updating files: 59% (1870/3168) Updating files: 60% (1901/3168) Updating files: 61% (1933/3168) Updating files: 62% (1965/3168) Updating files: 63% (1996/3168) Updating files: 64% (2028/3168) Updating files: 65% (2060/3168) Updating files: 66% (2091/3168) Updating files: 66% (2113/3168) Updating files: 67% (2123/3168) Updating files: 68% (2155/3168) Updating files: 69% (2186/3168) Updating files: 70% (2218/3168) Updating files: 71% (2250/3168) Updating files: 72% (2281/3168) Updating files: 73% (2313/3168) Updating files: 74% (2345/3168) Updating files: 75% (2376/3168) Updating files: 76% (2408/3168) Updating files: 77% (2440/3168) Updating files: 78% (2472/3168) Updating files: 79% (2503/3168) Updating files: 80% (2535/3168) Updating files: 81% (2567/3168) Updating files: 81% (2596/3168) Updating files: 82% (2598/3168) Updating files: 83% (2630/3168) Updating files: 84% (2662/3168) Updating files: 85% (2693/3168) Updating files: 86% (2725/3168) Updating files: 87% (2757/3168) Updating files: 88% (2788/3168) Updating files: 89% (2820/3168) Updating files: 90% (2852/3168) Updating files: 91% (2883/3168) Updating files: 92% (2915/3168) Updating files: 93% (2947/3168) Updating files: 94% (2978/3168) Updating files: 95% (3010/3168) Updating files: 96% (3042/3168) Updating files: 97% (3073/3168) Updating files: 98% (3105/3168) Updating files: 99% (3137/3168) Updating files: 100% (3168/3168) Updating files: 100% (3168/3168), done. 2023-10-30T12:13:53.428 INFO:teuthology.orchestra.run.smithi089.stdout:HEAD is now at 2b304bfe Merge branch 'main' into 'main' 2023-10-30T12:13:53.448 INFO:teuthology.task.kernel:Downloading kernel eacf00cc51e204f401b4d5d4f84825a68f373c74 on client.0... 2023-10-30T12:13:53.449 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=eacf00cc51e204f401b4d5d4f84825a68f373c74 2023-10-30T12:13:53.821 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/eacf00cc51e204f401b4d5d4f84825a68f373c74/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-rc5-geacf00cc51e2/ 2023-10-30T12:13:53.822 DEBUG:teuthology.orchestra.run.smithi089:> rm -f /tmp/linux-image.deb && echo linux-image-6.6.0-rc5-geacf00cc51e2_6.6.0-rc5-geacf00cc51e2-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://4.chacra.ceph.com/r/kernel/testing/eacf00cc51e204f401b4d5d4f84825a68f373c74/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-rc5-geacf00cc51e2/ --input-file=- 2023-10-30T12:13:53.825 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi089.front.sepia.ceph.com to complete... 2023-10-30T12:13:57.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30 12:13:57 URL:https://4.chacra.ceph.com/r/kernel/testing/eacf00cc51e204f401b4d5d4f84825a68f373c74/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-rc5-geacf00cc51e2/linux-image-6.6.0-rc5-geacf00cc51e2_6.6.0-rc5-geacf00cc51e2-1_amd64.deb [38909658/38909658] -> "/tmp/linux-image.deb" [1] 2023-10-30T12:13:57.403 INFO:teuthology.orchestra.run.smithi089.stderr:FINISHED --2023-10-30 12:13:57-- 2023-10-30T12:13:57.403 INFO:teuthology.orchestra.run.smithi089.stderr:Total wall clock time: 3.6s 2023-10-30T12:13:57.404 INFO:teuthology.orchestra.run.smithi089.stderr:Downloaded: 1 files, 37M in 3.2s (11.4 MB/s) 2023-10-30T12:13:57.406 INFO:teuthology.task.kernel:Installing kernel eacf00cc51e204f401b4d5d4f84825a68f373c74 on client.0... 2023-10-30T12:13:57.406 DEBUG:teuthology.orchestra.run.smithi089:> sudo dpkg -i /tmp/linux-image.deb 2023-10-30T12:13:57.618 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package linux-image-6.6.0-rc5-geacf00cc51e2. 2023-10-30T12:13:59.643 INFO:teuthology.orchestra.run.smithi089.stdout:(Reading database ... 129542 files and directories currently installed.) 2023-10-30T12:13:59.647 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack /tmp/linux-image.deb ... 2023-10-30T12:13:59.827 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking linux-image-6.6.0-rc5-geacf00cc51e2 (6.6.0-rc5-geacf00cc51e2-1) ... 2023-10-30T12:14:03.269 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up linux-image-6.6.0-rc5-geacf00cc51e2 (6.6.0-rc5-geacf00cc51e2-1) ... 2023-10-30T12:14:03.375 INFO:teuthology.orchestra.run.smithi089.stderr:update-initramfs: Generating /boot/initrd.img-6.6.0-rc5-geacf00cc51e2 2023-10-30T12:14:24.316 INFO:teuthology.orchestra.run.smithi089.stderr:Sourcing file `/etc/default/grub' 2023-10-30T12:14:24.331 INFO:teuthology.orchestra.run.smithi089.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-30T12:14:24.360 INFO:teuthology.orchestra.run.smithi089.stderr:Generating grub configuration file ... 2023-10-30T12:14:24.754 INFO:teuthology.orchestra.run.smithi089.stderr:Found linux image: /boot/vmlinuz-6.6.0-rc5-geacf00cc51e2 2023-10-30T12:14:24.771 INFO:teuthology.orchestra.run.smithi089.stderr:Found initrd image: /boot/initrd.img-6.6.0-rc5-geacf00cc51e2 2023-10-30T12:14:25.106 INFO:teuthology.orchestra.run.smithi089.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-30T12:14:25.113 INFO:teuthology.orchestra.run.smithi089.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-30T12:14:25.177 INFO:teuthology.orchestra.run.smithi089.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-30T12:14:25.183 INFO:teuthology.orchestra.run.smithi089.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-30T12:14:25.363 INFO:teuthology.orchestra.run.smithi089.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-30T12:14:25.363 INFO:teuthology.orchestra.run.smithi089.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-30T12:14:25.363 INFO:teuthology.orchestra.run.smithi089.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-30T12:14:25.402 INFO:teuthology.orchestra.run.smithi089.stderr:done 2023-10-30T12:14:25.592 DEBUG:teuthology.orchestra.run.smithi089:> dpkg-deb -c /tmp/linux-image.deb 2023-10-30T12:14:25.600 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./ 2023-10-30T12:14:25.601 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./boot/ 2023-10-30T12:14:25.601 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 4415097 2023-10-30 03:46 ./boot/System.map-6.6.0-rc5-geacf00cc51e2 2023-10-30T12:14:25.612 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 169233 2023-10-30 03:46 ./boot/config-6.6.0-rc5-geacf00cc51e2 2023-10-30T12:14:25.613 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11352928 2023-10-30 03:46 ./boot/vmlinuz-6.6.0-rc5-geacf00cc51e2 2023-10-30T12:14:25.632 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/ 2023-10-30T12:14:25.632 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/kernel/ 2023-10-30T12:14:25.633 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/kernel/postinst.d/ 2023-10-30T12:14:25.633 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/kernel/postrm.d/ 2023-10-30T12:14:25.633 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/kernel/preinst.d/ 2023-10-30T12:14:25.633 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/kernel/prerm.d/ 2023-10-30T12:14:25.633 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/ 2023-10-30T12:14:25.634 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/ 2023-10-30T12:14:25.634 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/ 2023-10-30T12:14:25.634 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/ 2023-10-30T12:14:25.634 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/ 2023-10-30T12:14:25.634 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/ 2023-10-30T12:14:25.635 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/ 2023-10-30T12:14:25.635 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 490233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/aesni-intel.ko 2023-10-30T12:14:25.637 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-10-30T12:14:25.638 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-10-30T12:14:25.638 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-10-30T12:14:25.638 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 93113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-10-30T12:14:25.638 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 106729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-10-30T12:14:25.639 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 152241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-10-30T12:14:25.639 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-10-30T12:14:25.639 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-10-30T12:14:25.639 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 83889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-10-30T12:14:25.639 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-10-30T12:14:25.640 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 47225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-10-30T12:14:25.640 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 47209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/serpent-avx2.ko 2023-10-30T12:14:25.640 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 51801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-10-30T12:14:25.640 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 38937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-10-30T12:14:25.640 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 40529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-10-30T12:14:25.641 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 58585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-10-30T12:14:25.641 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 50497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-10-30T12:14:25.641 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-10-30T12:14:25.641 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-10-30T12:14:25.641 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/ 2023-10-30T12:14:25.642 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/cpu/ 2023-10-30T12:14:25.642 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/cpu/mce/ 2023-10-30T12:14:25.642 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 40113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-10-30T12:14:25.642 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/cpuid.ko 2023-10-30T12:14:25.642 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/msr.ko 2023-10-30T12:14:25.643 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kvm/ 2023-10-30T12:14:25.643 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 353761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kvm/kvm-amd.ko 2023-10-30T12:14:25.643 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 881713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kvm/kvm-intel.ko 2023-10-30T12:14:25.643 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 2276817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kvm/kvm.ko 2023-10-30T12:14:25.690 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/ 2023-10-30T12:14:25.691 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 53761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/af_alg.ko 2023-10-30T12:14:25.691 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/algif_hash.ko 2023-10-30T12:14:25.691 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/algif_skcipher.ko 2023-10-30T12:14:25.691 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/ansi_cprng.ko 2023-10-30T12:14:25.691 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/anubis.ko 2023-10-30T12:14:25.692 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/arc4.ko 2023-10-30T12:14:25.692 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/asymmetric_keys/ 2023-10-30T12:14:25.692 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2023-10-30T12:14:25.692 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/ 2023-10-30T12:14:25.692 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/async_memcpy.ko 2023-10-30T12:14:25.693 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/async_pq.ko 2023-10-30T12:14:25.693 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/async_raid6_recov.ko 2023-10-30T12:14:25.693 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/async_tx.ko 2023-10-30T12:14:25.693 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/async_xor.ko 2023-10-30T12:14:25.693 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/raid6test.ko 2023-10-30T12:14:25.694 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/authenc.ko 2023-10-30T12:14:25.694 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/authencesn.ko 2023-10-30T12:14:25.694 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/blake2b_generic.ko 2023-10-30T12:14:25.694 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/blowfish_common.ko 2023-10-30T12:14:25.694 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/blowfish_generic.ko 2023-10-30T12:14:25.695 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 38873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/camellia_generic.ko 2023-10-30T12:14:25.695 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cast5_generic.ko 2023-10-30T12:14:25.695 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cast6_generic.ko 2023-10-30T12:14:25.695 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cast_common.ko 2023-10-30T12:14:25.695 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/ccm.ko 2023-10-30T12:14:25.696 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cmac.ko 2023-10-30T12:14:25.696 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/crc32_generic.ko 2023-10-30T12:14:25.696 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 47505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cryptd.ko 2023-10-30T12:14:25.696 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/crypto_engine.ko 2023-10-30T12:14:25.696 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/crypto_simd.ko 2023-10-30T12:14:25.697 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/crypto_user.ko 2023-10-30T12:14:25.697 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cts.ko 2023-10-30T12:14:25.697 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/des_generic.ko 2023-10-30T12:14:25.697 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/dh_generic.ko 2023-10-30T12:14:25.697 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/echainiv.ko 2023-10-30T12:14:25.698 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/essiv.ko 2023-10-30T12:14:25.698 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/fcrypt.ko 2023-10-30T12:14:25.698 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/khazad.ko 2023-10-30T12:14:25.698 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/lrw.ko 2023-10-30T12:14:25.698 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/lz4.ko 2023-10-30T12:14:25.699 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/lz4hc.ko 2023-10-30T12:14:25.699 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/md4.ko 2023-10-30T12:14:25.699 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/michael_mic.ko 2023-10-30T12:14:25.699 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/pcbc.ko 2023-10-30T12:14:25.699 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/pcrypt.ko 2023-10-30T12:14:25.700 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/rmd160.ko 2023-10-30T12:14:25.700 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/seed.ko 2023-10-30T12:14:25.700 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/serpent_generic.ko 2023-10-30T12:14:25.700 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 149193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/tcrypt.ko 2023-10-30T12:14:25.700 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/tea.ko 2023-10-30T12:14:25.701 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/twofish_common.ko 2023-10-30T12:14:25.701 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/twofish_generic.ko 2023-10-30T12:14:25.701 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/vmac.ko 2023-10-30T12:14:25.701 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/wp512.ko 2023-10-30T12:14:25.701 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/xcbc.ko 2023-10-30T12:14:25.702 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/xor.ko 2023-10-30T12:14:25.702 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/xts.ko 2023-10-30T12:14:25.702 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/xxhash_generic.ko 2023-10-30T12:14:25.702 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ 2023-10-30T12:14:25.702 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/ 2023-10-30T12:14:25.703 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/acpi_extlog.ko 2023-10-30T12:14:25.703 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/acpi_ipmi.ko 2023-10-30T12:14:25.703 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/acpi_pad.ko 2023-10-30T12:14:25.703 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/apei/ 2023-10-30T12:14:25.703 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/apei/einj.ko 2023-10-30T12:14:25.704 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/ec_sys.ko 2023-10-30T12:14:25.704 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/platform_profile.ko 2023-10-30T12:14:25.704 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/sbs.ko 2023-10-30T12:14:25.704 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/sbshc.ko 2023-10-30T12:14:25.705 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 103025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/video.ko 2023-10-30T12:14:25.705 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/ 2023-10-30T12:14:25.705 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/acard-ahci.ko 2023-10-30T12:14:25.705 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 88865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/ahci.ko 2023-10-30T12:14:25.705 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/ahci_platform.ko 2023-10-30T12:14:25.706 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 88289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/libahci.ko 2023-10-30T12:14:25.706 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/libahci_platform.ko 2023-10-30T12:14:25.706 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_acpi.ko 2023-10-30T12:14:25.706 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_ali.ko 2023-10-30T12:14:25.706 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_amd.ko 2023-10-30T12:14:25.707 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_artop.ko 2023-10-30T12:14:25.707 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_atiixp.ko 2023-10-30T12:14:25.707 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_atp867x.ko 2023-10-30T12:14:25.707 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_cmd640.ko 2023-10-30T12:14:25.708 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_cmd64x.ko 2023-10-30T12:14:25.708 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_cypress.ko 2023-10-30T12:14:25.708 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_efar.ko 2023-10-30T12:14:25.708 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_hpt366.ko 2023-10-30T12:14:25.708 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_hpt37x.ko 2023-10-30T12:14:25.709 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_hpt3x2n.ko 2023-10-30T12:14:25.709 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_hpt3x3.ko 2023-10-30T12:14:25.709 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_it8213.ko 2023-10-30T12:14:25.709 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_it821x.ko 2023-10-30T12:14:25.709 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_jmicron.ko 2023-10-30T12:14:25.710 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 51481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_legacy.ko 2023-10-30T12:14:25.710 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_marvell.ko 2023-10-30T12:14:25.710 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_mpiix.ko 2023-10-30T12:14:25.710 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_netcell.ko 2023-10-30T12:14:25.710 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_ninja32.ko 2023-10-30T12:14:25.711 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_ns87410.ko 2023-10-30T12:14:25.711 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_ns87415.ko 2023-10-30T12:14:25.711 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_oldpiix.ko 2023-10-30T12:14:25.711 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_opti.ko 2023-10-30T12:14:25.711 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_optidma.ko 2023-10-30T12:14:25.711 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_pcmcia.ko 2023-10-30T12:14:25.712 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 37793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_pdc2027x.ko 2023-10-30T12:14:25.712 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-10-30T12:14:25.712 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_piccolo.ko 2023-10-30T12:14:25.712 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_radisys.ko 2023-10-30T12:14:25.712 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_rdc.ko 2023-10-30T12:14:25.713 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_rz1000.ko 2023-10-30T12:14:25.713 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_sch.ko 2023-10-30T12:14:25.713 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_serverworks.ko 2023-10-30T12:14:25.713 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_sil680.ko 2023-10-30T12:14:25.713 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_sl82c105.ko 2023-10-30T12:14:25.714 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_triflex.ko 2023-10-30T12:14:25.714 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_via.ko 2023-10-30T12:14:25.714 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pdc_adma.ko 2023-10-30T12:14:25.714 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_inic162x.ko 2023-10-30T12:14:25.714 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 83345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_mv.ko 2023-10-30T12:14:25.715 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 70393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_nv.ko 2023-10-30T12:14:25.715 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 37377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_promise.ko 2023-10-30T12:14:25.715 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_qstor.ko 2023-10-30T12:14:25.715 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_sil.ko 2023-10-30T12:14:25.715 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 38337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_sil24.ko 2023-10-30T12:14:25.716 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_sis.ko 2023-10-30T12:14:25.716 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_svw.ko 2023-10-30T12:14:25.716 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 44793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_sx4.ko 2023-10-30T12:14:25.716 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_uli.ko 2023-10-30T12:14:25.716 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_via.ko 2023-10-30T12:14:25.717 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_vsc.ko 2023-10-30T12:14:25.717 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/auxdisplay/ 2023-10-30T12:14:25.717 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/auxdisplay/ks0108.ko 2023-10-30T12:14:25.717 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/bcma/ 2023-10-30T12:14:25.717 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 124473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/bcma/bcma.ko 2023-10-30T12:14:25.718 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/ 2023-10-30T12:14:25.718 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/aoe/ 2023-10-30T12:14:25.718 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 113665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/aoe/aoe.ko 2023-10-30T12:14:25.718 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/drbd/ 2023-10-30T12:14:25.718 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 965585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/drbd/drbd.ko 2023-10-30T12:14:25.719 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 173225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/floppy.ko 2023-10-30T12:14:25.719 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/mtip32xx/ 2023-10-30T12:14:25.719 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 94297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-10-30T12:14:25.719 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 115793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/nbd.ko 2023-10-30T12:14:25.719 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/null_blk/ 2023-10-30T12:14:25.719 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 110449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/null_blk/null_blk.ko 2023-10-30T12:14:25.720 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 98193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/pktcdvd.ko 2023-10-30T12:14:25.720 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 263769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/rbd.ko 2023-10-30T12:14:25.720 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/xen-blkback/ 2023-10-30T12:14:25.720 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 98145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-10-30T12:14:25.720 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/zram/ 2023-10-30T12:14:25.721 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 60217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/zram/zram.ko 2023-10-30T12:14:25.721 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ 2023-10-30T12:14:25.721 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/applicom.ko 2023-10-30T12:14:25.721 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hangcheck-timer.ko 2023-10-30T12:14:25.721 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/ 2023-10-30T12:14:25.722 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/amd-rng.ko 2023-10-30T12:14:25.722 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/intel-rng.ko 2023-10-30T12:14:25.722 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-10-30T12:14:25.722 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/via-rng.ko 2023-10-30T12:14:25.722 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/virtio-rng.ko 2023-10-30T12:14:25.723 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ 2023-10-30T12:14:25.723 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-10-30T12:14:25.723 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 148601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-10-30T12:14:25.723 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-10-30T12:14:25.724 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 139313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_si.ko 2023-10-30T12:14:25.724 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 70921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-10-30T12:14:25.724 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 53513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-10-30T12:14:25.724 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/lp.ko 2023-10-30T12:14:25.724 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/mwave/ 2023-10-30T12:14:25.725 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 84097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/mwave/mwave.ko 2023-10-30T12:14:25.725 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/nvram.ko 2023-10-30T12:14:25.725 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ppdev.ko 2023-10-30T12:14:25.725 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 51113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tlclk.ko 2023-10-30T12:14:25.725 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/ 2023-10-30T12:14:25.726 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_atmel.ko 2023-10-30T12:14:25.726 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2023-10-30T12:14:25.726 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2023-10-30T12:14:25.726 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2023-10-30T12:14:25.726 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_infineon.ko 2023-10-30T12:14:25.727 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_nsc.ko 2023-10-30T12:14:25.727 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/xen-tpmfront.ko 2023-10-30T12:14:25.727 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/xillybus/ 2023-10-30T12:14:25.727 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/xillybus/xillybus_class.ko 2023-10-30T12:14:25.727 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 50945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/xillybus/xillybus_core.ko 2023-10-30T12:14:25.728 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/xillybus/xillybus_pcie.ko 2023-10-30T12:14:25.728 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/clk/ 2023-10-30T12:14:25.728 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/clk/clk-palmas.ko 2023-10-30T12:14:25.728 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 47849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/clk/clk-si5351.ko 2023-10-30T12:14:25.728 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/clk/clk-twl6040.ko 2023-10-30T12:14:25.729 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/clk/clk-wm831x.ko 2023-10-30T12:14:25.729 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/cpufreq/ 2023-10-30T12:14:25.729 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-10-30T12:14:25.729 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/cpufreq/p4-clockmod.ko 2023-10-30T12:14:25.729 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/cpufreq/speedstep-lib.ko 2023-10-30T12:14:25.730 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/ 2023-10-30T12:14:25.730 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/ccp/ 2023-10-30T12:14:25.730 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 70801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-10-30T12:14:25.730 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 243025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/ccp/ccp.ko 2023-10-30T12:14:25.730 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/ 2023-10-30T12:14:25.731 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/qat/ 2023-10-30T12:14:25.731 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/qat/qat_common/ 2023-10-30T12:14:25.731 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 438761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-10-30T12:14:25.731 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2023-10-30T12:14:25.731 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-10-30T12:14:25.732 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/padlock-aes.ko 2023-10-30T12:14:25.732 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/padlock-sha.ko 2023-10-30T12:14:25.732 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/virtio/ 2023-10-30T12:14:25.732 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 68857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-10-30T12:14:25.732 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dca/ 2023-10-30T12:14:25.733 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dca/dca.ko 2023-10-30T12:14:25.733 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dma/ 2023-10-30T12:14:25.733 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dma/dw/ 2023-10-30T12:14:25.733 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dma/dw/dw_dmac.ko 2023-10-30T12:14:25.733 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dma/ioat/ 2023-10-30T12:14:25.734 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 122569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dma/ioat/ioatdma.ko 2023-10-30T12:14:25.734 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/ 2023-10-30T12:14:25.734 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 79953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/amd64_edac.ko 2023-10-30T12:14:25.734 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/e752x_edac.ko 2023-10-30T12:14:25.734 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 63617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/edac_mce_amd.ko 2023-10-30T12:14:25.735 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i3000_edac.ko 2023-10-30T12:14:25.735 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i3200_edac.ko 2023-10-30T12:14:25.735 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i5100_edac.ko 2023-10-30T12:14:25.735 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i5400_edac.ko 2023-10-30T12:14:25.735 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i7300_edac.ko 2023-10-30T12:14:25.736 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 52041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i7core_edac.ko 2023-10-30T12:14:25.736 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i82975x_edac.ko 2023-10-30T12:14:25.736 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/ie31200_edac.ko 2023-10-30T12:14:25.736 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 56825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/sb_edac.ko 2023-10-30T12:14:25.736 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/x38_edac.ko 2023-10-30T12:14:25.736 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/ 2023-10-30T12:14:25.737 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-gpio.ko 2023-10-30T12:14:25.737 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-max14577.ko 2023-10-30T12:14:25.737 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-max77693.ko 2023-10-30T12:14:25.737 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-max8997.ko 2023-10-30T12:14:25.737 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-palmas.ko 2023-10-30T12:14:25.738 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-rt8973a.ko 2023-10-30T12:14:25.738 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-sm5502.ko 2023-10-30T12:14:25.738 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/firmware/ 2023-10-30T12:14:25.738 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/firmware/dmi-sysfs.ko 2023-10-30T12:14:25.738 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/firmware/efi/ 2023-10-30T12:14:25.739 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/firmware/efi/efi-pstore.ko 2023-10-30T12:14:25.739 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/firmware/iscsi_ibft.ko 2023-10-30T12:14:25.739 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/ 2023-10-30T12:14:25.739 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-adp5520.ko 2023-10-30T12:14:25.739 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-amd8111.ko 2023-10-30T12:14:25.740 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-arizona.ko 2023-10-30T12:14:25.740 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-da9052.ko 2023-10-30T12:14:25.740 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-da9055.ko 2023-10-30T12:14:25.740 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-dln2.ko 2023-10-30T12:14:25.740 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 77849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-f7188x.ko 2023-10-30T12:14:25.741 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-generic.ko 2023-10-30T12:14:25.741 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-ich.ko 2023-10-30T12:14:25.741 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-janz-ttl.ko 2023-10-30T12:14:25.741 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-kempld.ko 2023-10-30T12:14:25.741 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-lp3943.ko 2023-10-30T12:14:25.742 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-max7300.ko 2023-10-30T12:14:25.742 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-max730x.ko 2023-10-30T12:14:25.742 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-max732x.ko 2023-10-30T12:14:25.742 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-ml-ioh.ko 2023-10-30T12:14:25.742 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 47073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-pca953x.ko 2023-10-30T12:14:25.743 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-pcf857x.ko 2023-10-30T12:14:25.743 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-rdc321x.ko 2023-10-30T12:14:25.743 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-sch.ko 2023-10-30T12:14:25.743 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-sch311x.ko 2023-10-30T12:14:25.743 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-tps65912.ko 2023-10-30T12:14:25.744 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-twl4030.ko 2023-10-30T12:14:25.744 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-twl6040.ko 2023-10-30T12:14:25.744 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-viperboard.ko 2023-10-30T12:14:25.744 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-vx855.ko 2023-10-30T12:14:25.744 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-wm831x.ko 2023-10-30T12:14:25.745 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-wm8350.ko 2023-10-30T12:14:25.745 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-wm8994.ko 2023-10-30T12:14:25.745 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/ 2023-10-30T12:14:25.745 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-a4tech.ko 2023-10-30T12:14:25.745 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-apple.ko 2023-10-30T12:14:25.746 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-appleir.ko 2023-10-30T12:14:25.746 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-aureal.ko 2023-10-30T12:14:25.746 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-axff.ko 2023-10-30T12:14:25.746 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-belkin.ko 2023-10-30T12:14:25.746 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-cherry.ko 2023-10-30T12:14:25.747 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-chicony.ko 2023-10-30T12:14:25.747 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 60649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-cp2112.ko 2023-10-30T12:14:25.747 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-cypress.ko 2023-10-30T12:14:25.747 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-dr.ko 2023-10-30T12:14:25.747 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-elecom.ko 2023-10-30T12:14:25.748 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-elo.ko 2023-10-30T12:14:25.748 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-emsff.ko 2023-10-30T12:14:25.748 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-ezkey.ko 2023-10-30T12:14:25.748 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-gaff.ko 2023-10-30T12:14:25.748 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-generic.ko 2023-10-30T12:14:25.749 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-gt683r.ko 2023-10-30T12:14:25.749 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-gyration.ko 2023-10-30T12:14:25.749 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-holtek-kbd.ko 2023-10-30T12:14:25.749 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-holtek-mouse.ko 2023-10-30T12:14:25.749 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-holtekff.ko 2023-10-30T12:14:25.750 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-hyperv.ko 2023-10-30T12:14:25.750 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-icade.ko 2023-10-30T12:14:25.750 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-kensington.ko 2023-10-30T12:14:25.750 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-keytouch.ko 2023-10-30T12:14:25.750 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-kye.ko 2023-10-30T12:14:25.751 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-lcpower.ko 2023-10-30T12:14:25.751 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-led.ko 2023-10-30T12:14:25.751 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 42361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-lenovo.ko 2023-10-30T12:14:25.751 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-lg-g15.ko 2023-10-30T12:14:25.751 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 58185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-logitech-dj.ko 2023-10-30T12:14:25.751 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 101369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-10-30T12:14:25.752 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 80993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-logitech.ko 2023-10-30T12:14:25.752 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-magicmouse.ko 2023-10-30T12:14:25.752 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-microsoft.ko 2023-10-30T12:14:25.752 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-monterey.ko 2023-10-30T12:14:25.752 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 50161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-multitouch.ko 2023-10-30T12:14:25.752 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-ntrig.ko 2023-10-30T12:14:25.753 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-ortek.ko 2023-10-30T12:14:25.753 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-penmount.ko 2023-10-30T12:14:25.753 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-petalynx.ko 2023-10-30T12:14:25.753 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 72937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-picolcd.ko 2023-10-30T12:14:25.753 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-pl.ko 2023-10-30T12:14:25.754 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-plantronics.ko 2023-10-30T12:14:25.754 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-primax.ko 2023-10-30T12:14:25.754 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-rmi.ko 2023-10-30T12:14:25.754 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-arvo.ko 2023-10-30T12:14:25.754 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-common.ko 2023-10-30T12:14:25.755 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-isku.ko 2023-10-30T12:14:25.755 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-kone.ko 2023-10-30T12:14:25.755 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-10-30T12:14:25.755 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-konepure.ko 2023-10-30T12:14:25.755 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-10-30T12:14:25.756 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-lua.ko 2023-10-30T12:14:25.756 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-pyra.ko 2023-10-30T12:14:25.756 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-ryos.ko 2023-10-30T12:14:25.756 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-savu.ko 2023-10-30T12:14:25.756 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat.ko 2023-10-30T12:14:25.757 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-saitek.ko 2023-10-30T12:14:25.757 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-samsung.ko 2023-10-30T12:14:25.757 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-sensor-hub.ko 2023-10-30T12:14:25.757 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-sjoy.ko 2023-10-30T12:14:25.757 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 51249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-sony.ko 2023-10-30T12:14:25.758 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-speedlink.ko 2023-10-30T12:14:25.758 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-steelseries.ko 2023-10-30T12:14:25.758 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-sunplus.ko 2023-10-30T12:14:25.758 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-thrustmaster.ko 2023-10-30T12:14:25.758 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-tivo.ko 2023-10-30T12:14:25.759 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-tmff.ko 2023-10-30T12:14:25.759 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-topseed.ko 2023-10-30T12:14:25.759 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-twinhan.ko 2023-10-30T12:14:25.759 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 71073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-uclogic.ko 2023-10-30T12:14:25.759 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-waltop.ko 2023-10-30T12:14:25.760 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 143473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-wiimote.ko 2023-10-30T12:14:25.760 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-xinmo.ko 2023-10-30T12:14:25.760 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-zpff.ko 2023-10-30T12:14:25.760 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-zydacron.ko 2023-10-30T12:14:25.760 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 267825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid.ko 2023-10-30T12:14:25.761 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/uhid.ko 2023-10-30T12:14:25.761 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/usbhid/ 2023-10-30T12:14:25.761 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 125721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/usbhid/usbhid.ko 2023-10-30T12:14:25.761 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/usbhid/usbkbd.ko 2023-10-30T12:14:25.761 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/usbhid/usbmouse.ko 2023-10-30T12:14:25.762 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 217873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/wacom.ko 2023-10-30T12:14:25.762 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hv/ 2023-10-30T12:14:25.762 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 64201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hv/hv_balloon.ko 2023-10-30T12:14:25.762 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 89625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hv/hv_utils.ko 2023-10-30T12:14:25.762 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 305465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hv/hv_vmbus.ko 2023-10-30T12:14:25.763 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ 2023-10-30T12:14:25.763 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 66521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/abituguru.ko 2023-10-30T12:14:25.763 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 61393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/abituguru3.ko 2023-10-30T12:14:25.763 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/acpi_power_meter.ko 2023-10-30T12:14:25.763 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ad7414.ko 2023-10-30T12:14:25.764 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ad7418.ko 2023-10-30T12:14:25.764 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adc128d818.ko 2023-10-30T12:14:25.764 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adm1025.ko 2023-10-30T12:14:25.764 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 97977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adm1026.ko 2023-10-30T12:14:25.764 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adm1029.ko 2023-10-30T12:14:25.765 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 45553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adm1031.ko 2023-10-30T12:14:25.765 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adm9240.ko 2023-10-30T12:14:25.765 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ads7828.ko 2023-10-30T12:14:25.765 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7410.ko 2023-10-30T12:14:25.765 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7411.ko 2023-10-30T12:14:25.766 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 74433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7462.ko 2023-10-30T12:14:25.766 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7470.ko 2023-10-30T12:14:25.766 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 67929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7475.ko 2023-10-30T12:14:25.766 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7x10.ko 2023-10-30T12:14:25.766 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/amc6821.ko 2023-10-30T12:14:25.767 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 55265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/applesmc.ko 2023-10-30T12:14:25.767 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 49913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/asb100.ko 2023-10-30T12:14:25.767 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 58617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/asc7621.ko 2023-10-30T12:14:25.767 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 40889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/asus_atk0110.ko 2023-10-30T12:14:25.767 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/atxp1.ko 2023-10-30T12:14:25.768 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/coretemp.ko 2023-10-30T12:14:25.768 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/da9052-hwmon.ko 2023-10-30T12:14:25.768 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/da9055-hwmon.ko 2023-10-30T12:14:25.768 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 44465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-10-30T12:14:25.768 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 99601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/dme1737.ko 2023-10-30T12:14:25.769 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ds1621.ko 2023-10-30T12:14:25.769 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ds620.ko 2023-10-30T12:14:25.769 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/emc1403.ko 2023-10-30T12:14:25.769 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/emc2103.ko 2023-10-30T12:14:25.769 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/emc6w201.ko 2023-10-30T12:14:25.770 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 66985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/f71805f.ko 2023-10-30T12:14:25.770 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 94457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/f71882fg.ko 2023-10-30T12:14:25.770 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 41401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/f75375s.ko 2023-10-30T12:14:25.770 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/fam15h_power.ko 2023-10-30T12:14:25.770 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 55449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/fschmd.ko 2023-10-30T12:14:25.771 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/g760a.ko 2023-10-30T12:14:25.771 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/g762.ko 2023-10-30T12:14:25.771 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 46801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/gl518sm.ko 2023-10-30T12:14:25.771 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 45529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/gl520sm.ko 2023-10-30T12:14:25.771 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/hih6130.ko 2023-10-30T12:14:25.771 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/hwmon-vid.ko 2023-10-30T12:14:25.772 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/i5500_temp.ko 2023-10-30T12:14:25.772 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/i5k_amb.ko 2023-10-30T12:14:25.772 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ibmaem.ko 2023-10-30T12:14:25.772 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ibmpex.ko 2023-10-30T12:14:25.772 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ina209.ko 2023-10-30T12:14:25.773 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ina2xx.ko 2023-10-30T12:14:25.773 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 130449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/it87.ko 2023-10-30T12:14:25.773 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/jc42.ko 2023-10-30T12:14:25.773 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/k10temp.ko 2023-10-30T12:14:25.773 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/k8temp.ko 2023-10-30T12:14:25.773 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lineage-pem.ko 2023-10-30T12:14:25.774 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 48601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm63.ko 2023-10-30T12:14:25.774 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm73.ko 2023-10-30T12:14:25.774 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm75.ko 2023-10-30T12:14:25.774 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm77.ko 2023-10-30T12:14:25.774 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm78.ko 2023-10-30T12:14:25.775 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm80.ko 2023-10-30T12:14:25.775 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm83.ko 2023-10-30T12:14:25.775 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 72833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm85.ko 2023-10-30T12:14:25.775 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 46153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm87.ko 2023-10-30T12:14:25.775 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 37913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm90.ko 2023-10-30T12:14:25.776 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm92.ko 2023-10-30T12:14:25.776 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 108841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm93.ko 2023-10-30T12:14:25.776 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm95234.ko 2023-10-30T12:14:25.776 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm95241.ko 2023-10-30T12:14:25.776 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm95245.ko 2023-10-30T12:14:25.776 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc2945.ko 2023-10-30T12:14:25.777 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4151.ko 2023-10-30T12:14:25.777 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4215.ko 2023-10-30T12:14:25.777 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4222.ko 2023-10-30T12:14:25.777 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4245.ko 2023-10-30T12:14:25.777 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4260.ko 2023-10-30T12:14:25.778 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4261.ko 2023-10-30T12:14:25.778 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max16065.ko 2023-10-30T12:14:25.778 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max1619.ko 2023-10-30T12:14:25.778 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max1668.ko 2023-10-30T12:14:25.778 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max197.ko 2023-10-30T12:14:25.779 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max6639.ko 2023-10-30T12:14:25.779 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max6650.ko 2023-10-30T12:14:25.779 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max6697.ko 2023-10-30T12:14:25.779 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/mc13783-adc.ko 2023-10-30T12:14:25.779 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/mcp3021.ko 2023-10-30T12:14:25.780 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2023-10-30T12:14:25.780 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 44129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/nct6683.ko 2023-10-30T12:14:25.780 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 119817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/nct6775-core.ko 2023-10-30T12:14:25.780 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 60705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/nct6775.ko 2023-10-30T12:14:25.780 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 59737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/nct7802.ko 2023-10-30T12:14:25.781 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 82337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pc87360.ko 2023-10-30T12:14:25.781 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 54433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pc87427.ko 2023-10-30T12:14:25.781 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pcf8591.ko 2023-10-30T12:14:25.781 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/ 2023-10-30T12:14:25.781 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-10-30T12:14:25.781 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-10-30T12:14:25.782 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-10-30T12:14:25.782 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/max16064.ko 2023-10-30T12:14:25.782 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/max34440.ko 2023-10-30T12:14:25.782 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/max8688.ko 2023-10-30T12:14:25.782 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-10-30T12:14:25.783 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 82577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-10-30T12:14:25.783 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-10-30T12:14:25.783 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-10-30T12:14:25.783 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-10-30T12:14:25.783 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-10-30T12:14:25.783 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/powr1220.ko 2023-10-30T12:14:25.784 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sch5627.ko 2023-10-30T12:14:25.784 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sch5636.ko 2023-10-30T12:14:25.784 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sch56xx-common.ko 2023-10-30T12:14:25.784 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sht15.ko 2023-10-30T12:14:25.784 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sht21.ko 2023-10-30T12:14:25.785 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/shtc1.ko 2023-10-30T12:14:25.785 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 40105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sis5595.ko 2023-10-30T12:14:25.785 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/smsc47b397.ko 2023-10-30T12:14:25.785 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/smsc47m1.ko 2023-10-30T12:14:25.785 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 37385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/smsc47m192.ko 2023-10-30T12:14:25.786 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/thmc50.ko 2023-10-30T12:14:25.786 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/tmp102.ko 2023-10-30T12:14:25.786 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/tmp103.ko 2023-10-30T12:14:25.786 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/tmp401.ko 2023-10-30T12:14:25.787 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/tmp421.ko 2023-10-30T12:14:25.787 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/via-cputemp.ko 2023-10-30T12:14:25.787 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 41945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/via686a.ko 2023-10-30T12:14:25.787 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 52457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/vt1211.ko 2023-10-30T12:14:25.787 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 51545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/vt8231.ko 2023-10-30T12:14:25.788 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 57513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83627ehf.ko 2023-10-30T12:14:25.788 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 78873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83627hf.ko 2023-10-30T12:14:25.788 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 71345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83781d.ko 2023-10-30T12:14:25.788 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 63289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83791d.ko 2023-10-30T12:14:25.788 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 71321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83792d.ko 2023-10-30T12:14:25.789 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 101473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83793.ko 2023-10-30T12:14:25.789 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 119505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83795.ko 2023-10-30T12:14:25.789 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83l785ts.ko 2023-10-30T12:14:25.789 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83l786ng.ko 2023-10-30T12:14:25.789 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/wm831x-hwmon.ko 2023-10-30T12:14:25.790 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/wm8350-hwmon.ko 2023-10-30T12:14:25.790 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/ 2023-10-30T12:14:25.790 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/algos/ 2023-10-30T12:14:25.790 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-10-30T12:14:25.790 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-10-30T12:14:25.791 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/ 2023-10-30T12:14:25.791 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ali1535.ko 2023-10-30T12:14:25.791 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ali1563.ko 2023-10-30T12:14:25.791 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2023-10-30T12:14:25.791 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-10-30T12:14:25.792 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-10-30T12:14:25.792 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-10-30T12:14:25.792 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2023-10-30T12:14:25.792 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 6929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-10-30T12:14:25.792 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2023-10-30T12:14:25.793 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 50201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-10-30T12:14:25.793 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-10-30T12:14:25.793 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-10-30T12:14:25.793 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-10-30T12:14:25.793 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-dln2.ko 2023-10-30T12:14:25.794 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-gpio.ko 2023-10-30T12:14:25.794 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 62817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-i801.ko 2023-10-30T12:14:25.794 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-isch.ko 2023-10-30T12:14:25.794 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 47889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-10-30T12:14:25.794 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-kempld.ko 2023-10-30T12:14:25.795 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-10-30T12:14:25.795 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-10-30T12:14:25.795 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ocores.ko 2023-10-30T12:14:25.795 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-parport.ko 2023-10-30T12:14:25.795 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-10-30T12:14:25.796 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 47241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-10-30T12:14:25.796 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2023-10-30T12:14:25.796 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-10-30T12:14:25.796 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-10-30T12:14:25.796 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-sis5595.ko 2023-10-30T12:14:25.797 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-sis630.ko 2023-10-30T12:14:25.797 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-10-30T12:14:25.797 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2023-10-30T12:14:25.797 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-10-30T12:14:25.797 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-via.ko 2023-10-30T12:14:25.798 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-10-30T12:14:25.798 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-10-30T12:14:25.798 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 47657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-xiic.ko 2023-10-30T12:14:25.798 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/i2c-mux.ko 2023-10-30T12:14:25.798 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/i2c-smbus.ko 2023-10-30T12:14:25.799 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/i2c-stub.ko 2023-10-30T12:14:25.799 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/muxes/ 2023-10-30T12:14:25.799 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2023-10-30T12:14:25.799 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2023-10-30T12:14:25.799 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2023-10-30T12:14:25.800 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ 2023-10-30T12:14:25.800 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/ 2023-10-30T12:14:25.800 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 273609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/ib_cm.ko 2023-10-30T12:14:25.800 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 879969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/ib_core.ko 2023-10-30T12:14:25.800 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 69337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/ib_umad.ko 2023-10-30T12:14:25.801 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 324217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-10-30T12:14:25.801 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 97985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/iw_cm.ko 2023-10-30T12:14:25.801 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 282273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/rdma_cm.ko 2023-10-30T12:14:25.801 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 66201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-10-30T12:14:25.801 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/ 2023-10-30T12:14:25.801 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/cxgb4/ 2023-10-30T12:14:25.802 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 493073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-10-30T12:14:25.802 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mlx4/ 2023-10-30T12:14:25.802 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 419737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-10-30T12:14:25.802 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mlx5/ 2023-10-30T12:14:25.802 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 739025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-10-30T12:14:25.803 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mthca/ 2023-10-30T12:14:25.803 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 296625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-10-30T12:14:25.803 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/ocrdma/ 2023-10-30T12:14:25.803 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 153697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2023-10-30T12:14:25.804 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/usnic/ 2023-10-30T12:14:25.804 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 168969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2023-10-30T12:14:25.950 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/ 2023-10-30T12:14:25.951 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/ipoib/ 2023-10-30T12:14:25.951 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 263081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-10-30T12:14:25.951 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/iser/ 2023-10-30T12:14:25.951 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 100241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-10-30T12:14:25.952 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/isert/ 2023-10-30T12:14:25.952 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 111425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-10-30T12:14:25.952 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/srp/ 2023-10-30T12:14:25.952 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 128753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-10-30T12:14:25.952 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/srpt/ 2023-10-30T12:14:25.953 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 129529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-10-30T12:14:25.953 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/ 2023-10-30T12:14:25.953 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/evbug.ko 2023-10-30T12:14:25.953 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/ff-memless.ko 2023-10-30T12:14:25.953 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/gameport/ 2023-10-30T12:14:25.953 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-10-30T12:14:25.954 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/gameport/fm801-gp.ko 2023-10-30T12:14:25.954 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 41241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/gameport/gameport.ko 2023-10-30T12:14:25.954 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 41801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/joydev.ko 2023-10-30T12:14:25.954 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/ 2023-10-30T12:14:25.954 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/adp5520-keys.ko 2023-10-30T12:14:25.955 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/adp5588-keys.ko 2023-10-30T12:14:25.955 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/adp5589-keys.ko 2023-10-30T12:14:25.955 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2023-10-30T12:14:25.955 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/gpio_keys.ko 2023-10-30T12:14:25.955 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2023-10-30T12:14:25.956 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/lkkbd.ko 2023-10-30T12:14:25.956 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/lm8323.ko 2023-10-30T12:14:25.956 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/lm8333.ko 2023-10-30T12:14:25.956 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/matrix_keypad.ko 2023-10-30T12:14:25.956 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/max7359_keypad.ko 2023-10-30T12:14:25.957 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/mcs_touchkey.ko 2023-10-30T12:14:25.957 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2023-10-30T12:14:25.957 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/newtonkbd.ko 2023-10-30T12:14:25.957 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/opencores-kbd.ko 2023-10-30T12:14:25.957 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/qt1070.ko 2023-10-30T12:14:25.958 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/qt2160.ko 2023-10-30T12:14:25.958 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/samsung-keypad.ko 2023-10-30T12:14:25.958 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/stowaway.ko 2023-10-30T12:14:25.958 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/sunkbd.ko 2023-10-30T12:14:25.958 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/tca6416-keypad.ko 2023-10-30T12:14:25.959 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/tca8418_keypad.ko 2023-10-30T12:14:25.959 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/twl4030_keypad.ko 2023-10-30T12:14:25.959 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/xtkbd.ko 2023-10-30T12:14:25.959 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/matrix-keymap.ko 2023-10-30T12:14:25.959 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/ 2023-10-30T12:14:25.960 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/appletouch.ko 2023-10-30T12:14:25.960 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/bcm5974.ko 2023-10-30T12:14:25.960 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 123681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/cyapatp.ko 2023-10-30T12:14:25.960 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 89449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/elan_i2c.ko 2023-10-30T12:14:25.960 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/gpio_mouse.ko 2023-10-30T12:14:25.961 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 335977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/psmouse.ko 2023-10-30T12:14:25.961 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/sermouse.ko 2023-10-30T12:14:25.961 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-10-30T12:14:25.961 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/synaptics_usb.ko 2023-10-30T12:14:25.961 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/vsxxxaa.ko 2023-10-30T12:14:25.962 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/rmi4/ 2023-10-30T12:14:25.962 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 109873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/rmi4/rmi_core.ko 2023-10-30T12:14:25.962 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/ 2023-10-30T12:14:25.962 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/altera_ps2.ko 2023-10-30T12:14:25.962 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/arc_ps2.ko 2023-10-30T12:14:25.963 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/ct82c710.ko 2023-10-30T12:14:25.963 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-10-30T12:14:25.963 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/parkbd.ko 2023-10-30T12:14:25.963 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/pcips2.ko 2023-10-30T12:14:25.963 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/ps2mult.ko 2023-10-30T12:14:25.964 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/serio_raw.ko 2023-10-30T12:14:25.964 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/serport.ko 2023-10-30T12:14:25.964 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/sparse-keymap.ko 2023-10-30T12:14:25.964 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/iommu/ 2023-10-30T12:14:25.964 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/iommu/amd/ 2023-10-30T12:14:25.964 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/iommu/amd/iommu_v2.ko 2023-10-30T12:14:25.965 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/ 2023-10-30T12:14:25.965 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-88pm860x.ko 2023-10-30T12:14:25.965 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-adp5520.ko 2023-10-30T12:14:25.965 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 59753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-bd2802.ko 2023-10-30T12:14:25.965 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-blinkm.ko 2023-10-30T12:14:25.965 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-da903x.ko 2023-10-30T12:14:25.966 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-da9052.ko 2023-10-30T12:14:25.966 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-gpio.ko 2023-10-30T12:14:25.966 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lm3530.ko 2023-10-30T12:14:25.966 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lm3533.ko 2023-10-30T12:14:25.967 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lm355x.ko 2023-10-30T12:14:25.967 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lm3642.ko 2023-10-30T12:14:25.967 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lp3944.ko 2023-10-30T12:14:25.967 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lp8788.ko 2023-10-30T12:14:25.967 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-max8997.ko 2023-10-30T12:14:25.967 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-mc13783.ko 2023-10-30T12:14:25.968 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-menf21bmc.ko 2023-10-30T12:14:25.968 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-pca9532.ko 2023-10-30T12:14:25.968 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-pca955x.ko 2023-10-30T12:14:25.968 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-pca963x.ko 2023-10-30T12:14:25.968 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-ss4200.ko 2023-10-30T12:14:25.969 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-tca6507.ko 2023-10-30T12:14:25.969 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-wm831x-status.ko 2023-10-30T12:14:25.969 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-wm8350.ko 2023-10-30T12:14:25.969 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ 2023-10-30T12:14:25.969 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-10-30T12:14:25.970 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-10-30T12:14:25.970 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-10-30T12:14:25.970 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-10-30T12:14:25.970 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-10-30T12:14:25.970 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-10-30T12:14:25.971 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-10-30T12:14:25.971 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-10-30T12:14:25.971 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/ 2023-10-30T12:14:25.971 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/bcache/ 2023-10-30T12:14:25.971 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 654969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/bcache/bcache.ko 2023-10-30T12:14:25.972 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 40577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-bio-prison.ko 2023-10-30T12:14:25.972 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 89601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-bufio.ko 2023-10-30T12:14:25.972 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 47529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-cache-smq.ko 2023-10-30T12:14:25.972 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 156009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-cache.ko 2023-10-30T12:14:25.972 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 106897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-crypt.ko 2023-10-30T12:14:25.972 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-delay.ko 2023-10-30T12:14:25.973 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 53857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-era.ko 2023-10-30T12:14:25.973 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-flakey.ko 2023-10-30T12:14:25.973 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 41593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-log-userspace.ko 2023-10-30T12:14:25.973 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-log.ko 2023-10-30T12:14:25.974 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 49545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-mirror.ko 2023-10-30T12:14:25.974 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 78033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-multipath.ko 2023-10-30T12:14:25.974 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-queue-length.ko 2023-10-30T12:14:25.974 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 70825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-raid.ko 2023-10-30T12:14:25.974 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-region-hash.ko 2023-10-30T12:14:25.974 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-round-robin.ko 2023-10-30T12:14:25.975 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-service-time.ko 2023-10-30T12:14:25.975 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 109593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-snapshot.ko 2023-10-30T12:14:25.975 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-switch.ko 2023-10-30T12:14:25.975 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 181865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-thin-pool.ko 2023-10-30T12:14:25.975 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 51737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-verity.ko 2023-10-30T12:14:25.976 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-zero.ko 2023-10-30T12:14:25.976 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/faulty.ko 2023-10-30T12:14:25.976 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/linear.ko 2023-10-30T12:14:25.976 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/multipath.ko 2023-10-30T12:14:25.976 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/persistent-data/ 2023-10-30T12:14:25.977 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 199937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-10-30T12:14:25.977 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 37337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/raid0.ko 2023-10-30T12:14:25.977 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 113073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/raid1.ko 2023-10-30T12:14:25.977 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 158681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/raid10.ko 2023-10-30T12:14:25.977 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 395721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/raid456.ko 2023-10-30T12:14:25.978 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/ 2023-10-30T12:14:25.978 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/ 2023-10-30T12:14:25.978 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 178137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptbase.ko 2023-10-30T12:14:25.978 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 63385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptctl.ko 2023-10-30T12:14:25.978 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 44665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptfc.ko 2023-10-30T12:14:25.979 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 37225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptlan.ko 2023-10-30T12:14:25.979 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 123321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptsas.ko 2023-10-30T12:14:25.979 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 79041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptscsih.ko 2023-10-30T12:14:25.979 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 53081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptspi.ko 2023-10-30T12:14:25.980 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/ 2023-10-30T12:14:25.980 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/88pm800.ko 2023-10-30T12:14:25.980 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/88pm805.ko 2023-10-30T12:14:25.980 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/88pm80x.ko 2023-10-30T12:14:25.980 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/arizona-i2c.ko 2023-10-30T12:14:25.981 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 123721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/arizona.ko 2023-10-30T12:14:25.981 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/bcm590xx.ko 2023-10-30T12:14:25.981 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/cros_ec_dev.ko 2023-10-30T12:14:25.981 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/dln2.ko 2023-10-30T12:14:25.981 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/janz-cmodio.ko 2023-10-30T12:14:25.982 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 53585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/kempld-core.ko 2023-10-30T12:14:25.982 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/lm3533-core.ko 2023-10-30T12:14:25.982 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/lm3533-ctrlbank.ko 2023-10-30T12:14:25.982 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/lp3943.ko 2023-10-30T12:14:25.982 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 54569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/lpc_ich.ko 2023-10-30T12:14:25.983 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/lpc_sch.ko 2023-10-30T12:14:25.983 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/max8907.ko 2023-10-30T12:14:25.983 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/mc13xxx-core.ko 2023-10-30T12:14:25.983 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/mc13xxx-i2c.ko 2023-10-30T12:14:25.983 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/menf21bmc.ko 2023-10-30T12:14:25.984 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/pcf50633-adc.ko 2023-10-30T12:14:25.984 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/pcf50633-gpio.ko 2023-10-30T12:14:25.984 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/pcf50633.ko 2023-10-30T12:14:25.984 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/rdc321x-southbridge.ko 2023-10-30T12:14:25.984 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/retu-mfd.ko 2023-10-30T12:14:25.985 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 80425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/si476x-core.ko 2023-10-30T12:14:25.985 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 48457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/sm501.ko 2023-10-30T12:14:25.985 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/ti_am335x_tscadc.ko 2023-10-30T12:14:25.985 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 51137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/tps65010.ko 2023-10-30T12:14:25.985 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/tps6507x.ko 2023-10-30T12:14:25.986 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/viperboard.ko 2023-10-30T12:14:25.986 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/vx855.ko 2023-10-30T12:14:25.986 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/wl1273-core.ko 2023-10-30T12:14:25.986 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 44345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/wm8994.ko 2023-10-30T12:14:25.986 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ 2023-10-30T12:14:25.987 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ad525x_dpot-i2c.ko 2023-10-30T12:14:25.987 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 48489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ad525x_dpot.ko 2023-10-30T12:14:25.987 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/altera-stapl/ 2023-10-30T12:14:25.987 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 46585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-10-30T12:14:25.987 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/apds9802als.ko 2023-10-30T12:14:25.988 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/apds990x.ko 2023-10-30T12:14:25.988 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 45593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/bh1770glc.ko 2023-10-30T12:14:25.988 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/c2port/ 2023-10-30T12:14:25.988 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2023-10-30T12:14:25.988 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/c2port/core.ko 2023-10-30T12:14:25.989 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/cb710/ 2023-10-30T12:14:25.989 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/cb710/cb710.ko 2023-10-30T12:14:25.989 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ds1682.ko 2023-10-30T12:14:25.989 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/dummy-irq.ko 2023-10-30T12:14:25.989 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/echo/ 2023-10-30T12:14:25.989 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/echo/echo.ko 2023-10-30T12:14:25.990 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/eeprom/ 2023-10-30T12:14:25.990 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 37849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/eeprom/at24.ko 2023-10-30T12:14:25.990 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/eeprom/eeprom.ko 2023-10-30T12:14:25.990 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-10-30T12:14:25.990 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/eeprom/max6875.ko 2023-10-30T12:14:25.991 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/enclosure.ko 2023-10-30T12:14:25.991 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/genwqe/ 2023-10-30T12:14:25.991 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 152201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/genwqe/genwqe_card.ko 2023-10-30T12:14:25.991 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/hmc6352.ko 2023-10-30T12:14:25.991 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/hpilo.ko 2023-10-30T12:14:25.992 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ibmasm/ 2023-10-30T12:14:25.992 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 72129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ibmasm/ibmasm.ko 2023-10-30T12:14:25.992 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ics932s401.ko 2023-10-30T12:14:25.992 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/isl29003.ko 2023-10-30T12:14:25.992 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/isl29020.ko 2023-10-30T12:14:25.992 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/lis3lv02d/ 2023-10-30T12:14:25.993 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 42513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-10-30T12:14:25.993 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-10-30T12:14:25.993 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/mei/ 2023-10-30T12:14:25.993 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 125209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/mei/mei-me.ko 2023-10-30T12:14:25.993 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 56521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/mei/mei-txe.ko 2023-10-30T12:14:25.993 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 310113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/mei/mei.ko 2023-10-30T12:14:25.994 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/phantom.ko 2023-10-30T12:14:25.994 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ti-st/ 2023-10-30T12:14:25.994 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 74265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ti-st/st_drv.ko 2023-10-30T12:14:25.994 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/tifm_7xx1.ko 2023-10-30T12:14:25.994 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/tifm_core.ko 2023-10-30T12:14:25.995 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/tsl2550.ko 2023-10-30T12:14:25.995 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 48697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/vmw_balloon.ko 2023-10-30T12:14:25.995 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/vmw_vmci/ 2023-10-30T12:14:25.995 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 183609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-10-30T12:14:25.995 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ 2023-10-30T12:14:25.996 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/bonding/ 2023-10-30T12:14:25.996 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 490129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/bonding/bonding.ko 2023-10-30T12:14:25.996 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/ 2023-10-30T12:14:25.996 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/b53/ 2023-10-30T12:14:25.996 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 106033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/b53/b53_common.ko 2023-10-30T12:14:25.997 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 40297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/bcm-sf2.ko 2023-10-30T12:14:25.997 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/mv88e6060.ko 2023-10-30T12:14:25.997 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/mv88e6xxx/ 2023-10-30T12:14:25.997 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 453737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-10-30T12:14:25.997 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dummy.ko 2023-10-30T12:14:25.998 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/eql.ko 2023-10-30T12:14:25.998 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/ 2023-10-30T12:14:25.998 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/3com/ 2023-10-30T12:14:25.998 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 45785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-10-30T12:14:25.998 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 42105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-10-30T12:14:25.999 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 130553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-10-30T12:14:25.999 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 68849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-10-30T12:14:25.999 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/8390/ 2023-10-30T12:14:25.999 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 37649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/8390/8390.ko 2023-10-30T12:14:25.999 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 59689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-10-30T12:14:25.999 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-10-30T12:14:26.000 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 99113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-10-30T12:14:26.000 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/adaptec/ 2023-10-30T12:14:26.000 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 54521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-10-30T12:14:26.000 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/agere/ 2023-10-30T12:14:26.000 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 69201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/agere/et131x.ko 2023-10-30T12:14:26.001 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/alteon/ 2023-10-30T12:14:26.001 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 59585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-10-30T12:14:26.001 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/altera/ 2023-10-30T12:14:26.001 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 41409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-10-30T12:14:26.001 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/amd/ 2023-10-30T12:14:26.001 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 54609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-10-30T12:14:26.002 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 49289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-10-30T12:14:26.002 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 87297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-10-30T12:14:26.002 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/ 2023-10-30T12:14:26.002 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/alx/ 2023-10-30T12:14:26.002 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 97337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-10-30T12:14:26.003 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atl1c/ 2023-10-30T12:14:26.003 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 104673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-10-30T12:14:26.003 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atl1e/ 2023-10-30T12:14:26.003 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 90257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-10-30T12:14:26.003 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atlx/ 2023-10-30T12:14:26.004 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 81569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-10-30T12:14:26.004 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 69729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-10-30T12:14:26.004 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/ 2023-10-30T12:14:26.004 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 78401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-10-30T12:14:26.004 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 176761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-10-30T12:14:26.004 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2023-10-30T12:14:26.005 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 1421497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-10-30T12:14:26.005 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 128569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-10-30T12:14:26.005 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/genet/ 2023-10-30T12:14:26.005 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 132089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-10-30T12:14:26.005 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 312081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-10-30T12:14:26.005 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/brocade/ 2023-10-30T12:14:26.006 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/brocade/bna/ 2023-10-30T12:14:26.006 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 323513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-10-30T12:14:26.006 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/ 2023-10-30T12:14:26.006 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb/ 2023-10-30T12:14:26.006 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 193497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-10-30T12:14:26.007 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2023-10-30T12:14:26.007 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 347417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-10-30T12:14:26.007 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2023-10-30T12:14:26.007 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 1044433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-10-30T12:14:26.007 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2023-10-30T12:14:26.007 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 166193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-10-30T12:14:26.008 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2023-10-30T12:14:26.008 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-10-30T12:14:26.008 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/cisco/ 2023-10-30T12:14:26.008 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/cisco/enic/ 2023-10-30T12:14:26.008 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 204345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-10-30T12:14:26.008 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/ 2023-10-30T12:14:26.009 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/ 2023-10-30T12:14:26.009 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 69449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-10-30T12:14:26.009 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 61489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-10-30T12:14:26.009 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 141225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-10-30T12:14:26.009 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 51497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-10-30T12:14:26.009 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 62817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-10-30T12:14:26.010 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-10-30T12:14:26.010 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dlink/ 2023-10-30T12:14:26.010 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 60873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-10-30T12:14:26.010 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 62833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-10-30T12:14:26.010 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 45721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dnet.ko 2023-10-30T12:14:26.011 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/ec_bhf.ko 2023-10-30T12:14:26.011 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/emulex/ 2023-10-30T12:14:26.011 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/emulex/benet/ 2023-10-30T12:14:26.011 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 281137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-10-30T12:14:26.011 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 49457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/ethoc.ko 2023-10-30T12:14:26.011 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 51737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/fealnx.ko 2023-10-30T12:14:26.012 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/fujitsu/ 2023-10-30T12:14:26.012 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 45345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2023-10-30T12:14:26.012 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/ 2023-10-30T12:14:26.012 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 90465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/e100.ko 2023-10-30T12:14:26.012 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/e1000/ 2023-10-30T12:14:26.012 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 295577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-10-30T12:14:26.013 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/e1000e/ 2023-10-30T12:14:26.013 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 584497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-10-30T12:14:26.013 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/fm10k/ 2023-10-30T12:14:26.013 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 246209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-10-30T12:14:26.013 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/i40e/ 2023-10-30T12:14:26.013 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 925697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-10-30T12:14:26.014 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/iavf/ 2023-10-30T12:14:26.014 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 302849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-10-30T12:14:26.014 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/igb/ 2023-10-30T12:14:26.014 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 514681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-10-30T12:14:26.014 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/igbvf/ 2023-10-30T12:14:26.015 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 93593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-10-30T12:14:26.015 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/ixgbe/ 2023-10-30T12:14:26.015 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 734305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-10-30T12:14:26.015 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/ixgbevf/ 2023-10-30T12:14:26.015 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 151769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-10-30T12:14:26.016 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 86921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/jme.ko 2023-10-30T12:14:26.016 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/marvell/ 2023-10-30T12:14:26.016 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-10-30T12:14:26.016 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 102289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/marvell/skge.ko 2023-10-30T12:14:26.016 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 122649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-10-30T12:14:26.017 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/ 2023-10-30T12:14:26.017 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx4/ 2023-10-30T12:14:26.017 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 727601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-10-30T12:14:26.017 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 278937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-10-30T12:14:26.017 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx5/ 2023-10-30T12:14:26.017 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2023-10-30T12:14:26.018 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 1113993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-10-30T12:14:26.018 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/micrel/ 2023-10-30T12:14:26.018 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 53105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/micrel/ks8842.ko 2023-10-30T12:14:26.018 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 40993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2023-10-30T12:14:26.018 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2023-10-30T12:14:26.019 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 95665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-10-30T12:14:26.019 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/myricom/ 2023-10-30T12:14:26.019 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/myricom/myri10ge/ 2023-10-30T12:14:26.019 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 108649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-10-30T12:14:26.019 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/natsemi/ 2023-10-30T12:14:26.020 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 73441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-10-30T12:14:26.020 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 47777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-10-30T12:14:26.020 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/neterion/ 2023-10-30T12:14:26.020 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 150905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-10-30T12:14:26.020 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/nvidia/ 2023-10-30T12:14:26.020 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 136721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-10-30T12:14:26.021 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/packetengines/ 2023-10-30T12:14:26.021 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 50481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-10-30T12:14:26.021 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 43145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-10-30T12:14:26.021 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/ 2023-10-30T12:14:26.021 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/netxen/ 2023-10-30T12:14:26.022 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 227897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-10-30T12:14:26.022 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 87081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-10-30T12:14:26.022 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2023-10-30T12:14:26.022 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 672049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-10-30T12:14:26.022 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/rdc/ 2023-10-30T12:14:26.023 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 43017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-10-30T12:14:26.023 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/realtek/ 2023-10-30T12:14:26.023 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 67385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-10-30T12:14:26.023 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 83985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-10-30T12:14:26.023 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/realtek/atp.ko 2023-10-30T12:14:26.024 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 189801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-10-30T12:14:26.024 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/samsung/ 2023-10-30T12:14:26.024 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/samsung/sxgbe/ 2023-10-30T12:14:26.024 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 104241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2023-10-30T12:14:26.024 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sfc/ 2023-10-30T12:14:26.024 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 1053921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-10-30T12:14:26.025 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/silan/ 2023-10-30T12:14:26.025 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 47097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-10-30T12:14:26.025 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sis/ 2023-10-30T12:14:26.025 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 54817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sis/sis190.ko 2023-10-30T12:14:26.025 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 63337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sis/sis900.ko 2023-10-30T12:14:26.026 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/smsc/ 2023-10-30T12:14:26.026 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 61601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-10-30T12:14:26.026 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 67193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-10-30T12:14:26.026 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 73409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-10-30T12:14:26.026 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 68177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-10-30T12:14:26.027 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/ 2023-10-30T12:14:26.027 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/stmmac/ 2023-10-30T12:14:26.027 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2023-10-30T12:14:26.027 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-10-30T12:14:26.027 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2023-10-30T12:14:26.027 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 486961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-10-30T12:14:26.028 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sun/ 2023-10-30T12:14:26.028 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 103337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sun/cassini.ko 2023-10-30T12:14:26.028 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 166161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sun/niu.ko 2023-10-30T12:14:26.028 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 66329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sun/sungem.ko 2023-10-30T12:14:26.028 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 78801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-10-30T12:14:26.028 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/tehuti/ 2023-10-30T12:14:26.029 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 50057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-10-30T12:14:26.029 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/ti/ 2023-10-30T12:14:26.029 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 86529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/ti/tlan.ko 2023-10-30T12:14:26.029 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/via/ 2023-10-30T12:14:26.029 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 80273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-10-30T12:14:26.029 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 83633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-10-30T12:14:26.030 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/wiznet/ 2023-10-30T12:14:26.030 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 46241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-10-30T12:14:26.030 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-10-30T12:14:26.030 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/xircom/ 2023-10-30T12:14:26.030 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 60057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-10-30T12:14:26.031 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/fddi/ 2023-10-30T12:14:26.031 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 41065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/fddi/defxx.ko 2023-10-30T12:14:26.031 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/fddi/skfp/ 2023-10-30T12:14:26.031 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 216985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/fddi/skfp/skfp.ko 2023-10-30T12:14:26.031 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 77345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/geneve.ko 2023-10-30T12:14:26.031 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/hyperv/ 2023-10-30T12:14:26.032 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 235521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-10-30T12:14:26.032 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ieee802154/ 2023-10-30T12:14:26.032 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ieee802154/fakelb.ko 2023-10-30T12:14:26.032 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ifb.ko 2023-10-30T12:14:26.032 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ipvlan/ 2023-10-30T12:14:26.032 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 83769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ipvlan/ipvlan.ko 2023-10-30T12:14:26.033 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 62209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/macvlan.ko 2023-10-30T12:14:26.033 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/macvtap.ko 2023-10-30T12:14:26.033 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio/ 2023-10-30T12:14:26.033 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-10-30T12:14:26.033 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-10-30T12:14:26.033 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio/mdio-gpio.ko 2023-10-30T12:14:26.034 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio/mdio-regmap.ko 2023-10-30T12:14:26.034 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio.ko 2023-10-30T12:14:26.034 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mii.ko 2023-10-30T12:14:26.034 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 43953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/netconsole.ko 2023-10-30T12:14:26.034 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/nlmon.ko 2023-10-30T12:14:26.035 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/pcs/ 2023-10-30T12:14:26.035 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/pcs/pcs-lynx.ko 2023-10-30T12:14:26.035 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 38977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-10-30T12:14:26.035 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/ 2023-10-30T12:14:26.035 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/amd.ko 2023-10-30T12:14:26.035 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 50097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-10-30T12:14:26.036 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 44505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/bcm7xxx.ko 2023-10-30T12:14:26.036 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/bcm87xx.ko 2023-10-30T12:14:26.036 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 42241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/broadcom.ko 2023-10-30T12:14:26.036 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/cicada.ko 2023-10-30T12:14:26.036 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/davicom.ko 2023-10-30T12:14:26.037 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/et1011c.ko 2023-10-30T12:14:26.037 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/icplus.ko 2023-10-30T12:14:26.037 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/lxt.ko 2023-10-30T12:14:26.037 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 82049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/marvell.ko 2023-10-30T12:14:26.037 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 76177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/micrel.ko 2023-10-30T12:14:26.038 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/national.ko 2023-10-30T12:14:26.038 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 97857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/phylink.ko 2023-10-30T12:14:26.038 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/qsemi.ko 2023-10-30T12:14:26.038 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 50065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/realtek.ko 2023-10-30T12:14:26.038 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/smsc.ko 2023-10-30T12:14:26.038 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/ste10Xp.ko 2023-10-30T12:14:26.039 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/vitesse.ko 2023-10-30T12:14:26.039 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/plip/ 2023-10-30T12:14:26.039 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 43569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/plip/plip.ko 2023-10-30T12:14:26.039 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/ 2023-10-30T12:14:26.039 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/bsd_comp.ko 2023-10-30T12:14:26.039 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/ppp_async.ko 2023-10-30T12:14:26.040 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/ppp_deflate.ko 2023-10-30T12:14:26.040 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/ppp_mppe.ko 2023-10-30T12:14:26.040 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/ppp_synctty.ko 2023-10-30T12:14:26.040 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 42545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/pppoe.ko 2023-10-30T12:14:26.040 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/pppox.ko 2023-10-30T12:14:26.040 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/pptp.ko 2023-10-30T12:14:26.041 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/sb1000.ko 2023-10-30T12:14:26.041 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/slip/ 2023-10-30T12:14:26.041 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 42465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/slip/slip.ko 2023-10-30T12:14:26.041 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/sungem_phy.ko 2023-10-30T12:14:26.041 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 64385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/tap.ko 2023-10-30T12:14:26.042 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/ 2023-10-30T12:14:26.042 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 106169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team.ko 2023-10-30T12:14:26.042 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team_mode_activebackup.ko 2023-10-30T12:14:26.042 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team_mode_broadcast.ko 2023-10-30T12:14:26.042 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-10-30T12:14:26.042 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team_mode_random.ko 2023-10-30T12:14:26.043 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-10-30T12:14:26.043 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 87017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/veth.ko 2023-10-30T12:14:26.043 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/vmxnet3/ 2023-10-30T12:14:26.043 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 144297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-10-30T12:14:26.043 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/vxlan/ 2023-10-30T12:14:26.044 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 297329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/vxlan/vxlan.ko 2023-10-30T12:14:26.044 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/xen-netback/ 2023-10-30T12:14:26.044 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 144649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/xen-netback/xen-netback.ko 2023-10-30T12:14:26.044 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ntb/ 2023-10-30T12:14:26.044 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ntb/ntb.ko 2023-10-30T12:14:26.044 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/nvme/ 2023-10-30T12:14:26.045 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/nvme/host/ 2023-10-30T12:14:26.045 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 264937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/nvme/host/nvme-core.ko 2023-10-30T12:14:26.045 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 102209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/nvme/host/nvme.ko 2023-10-30T12:14:26.045 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/parport/ 2023-10-30T12:14:26.045 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 133841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/parport/parport.ko 2023-10-30T12:14:26.046 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/parport/parport_cs.ko 2023-10-30T12:14:26.046 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 84465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/parport/parport_pc.ko 2023-10-30T12:14:26.046 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/parport/parport_serial.ko 2023-10-30T12:14:26.046 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/ 2023-10-30T12:14:26.046 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/hotplug/ 2023-10-30T12:14:26.047 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-10-30T12:14:26.047 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2023-10-30T12:14:26.047 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2023-10-30T12:14:26.047 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/pci-stub.ko 2023-10-30T12:14:26.047 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 46529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/xen-pcifront.ko 2023-10-30T12:14:26.048 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/ 2023-10-30T12:14:26.048 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/i82092.ko 2023-10-30T12:14:26.048 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 145809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/pcmcia.ko 2023-10-30T12:14:26.048 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 62137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/pcmcia_core.ko 2023-10-30T12:14:26.048 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2023-10-30T12:14:26.048 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/pd6729.ko 2023-10-30T12:14:26.049 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 112889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/yenta_socket.ko 2023-10-30T12:14:26.049 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/phy/ 2023-10-30T12:14:26.049 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/phy/broadcom/ 2023-10-30T12:14:26.049 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2023-10-30T12:14:26.049 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/phy/samsung/ 2023-10-30T12:14:26.049 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2023-10-30T12:14:26.050 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pinctrl/ 2023-10-30T12:14:26.050 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pinctrl/intel/ 2023-10-30T12:14:26.050 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 70153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-10-30T12:14:26.050 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/ 2023-10-30T12:14:26.050 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/ 2023-10-30T12:14:26.051 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 50889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2023-10-30T12:14:26.051 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_ec.ko 2023-10-30T12:14:26.051 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2023-10-30T12:14:26.051 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2023-10-30T12:14:26.051 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2023-10-30T12:14:26.051 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2023-10-30T12:14:26.052 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2023-10-30T12:14:26.052 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/ 2023-10-30T12:14:26.052 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 67097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/acer-wmi.ko 2023-10-30T12:14:26.052 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/acerhdf.ko 2023-10-30T12:14:26.052 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/apple-gmux.ko 2023-10-30T12:14:26.052 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 59665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/asus-laptop.ko 2023-10-30T12:14:26.053 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-10-30T12:14:26.053 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 127569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/asus-wmi.ko 2023-10-30T12:14:26.053 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/classmate-laptop.ko 2023-10-30T12:14:26.053 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 49121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-10-30T12:14:26.053 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-10-30T12:14:26.054 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 40265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-10-30T12:14:26.054 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-10-30T12:14:26.054 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/hdaps.ko 2023-10-30T12:14:26.054 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/ibm_rtl.ko 2023-10-30T12:14:26.054 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/intel/ 2023-10-30T12:14:26.054 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-10-30T12:14:26.055 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2023-10-30T12:14:26.055 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 43497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/intel_ips.ko 2023-10-30T12:14:26.055 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/msi-wmi.ko 2023-10-30T12:14:26.055 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/mxm-wmi.ko 2023-10-30T12:14:26.055 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-10-30T12:14:26.056 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 42329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/samsung-laptop.ko 2023-10-30T12:14:26.056 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/samsung-q10.ko 2023-10-30T12:14:26.056 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/topstar-laptop.ko 2023-10-30T12:14:26.056 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-10-30T12:14:26.056 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/toshiba_haps.ko 2023-10-30T12:14:26.056 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/wmi-bmof.ko 2023-10-30T12:14:26.057 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 57569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/wmi.ko 2023-10-30T12:14:26.057 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/ 2023-10-30T12:14:26.057 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/ 2023-10-30T12:14:26.057 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/88pm860x_battery.ko 2023-10-30T12:14:26.057 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/88pm860x_charger.ko 2023-10-30T12:14:26.058 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 55057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/bq2415x_charger.ko 2023-10-30T12:14:26.058 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 54081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/bq24190_charger.ko 2023-10-30T12:14:26.058 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/bq24735-charger.ko 2023-10-30T12:14:26.058 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/cros_peripheral_charger.ko 2023-10-30T12:14:26.058 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/da9030_battery.ko 2023-10-30T12:14:26.059 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/da9052-battery.ko 2023-10-30T12:14:26.059 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/ds2760_battery.ko 2023-10-30T12:14:26.059 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/ds2780_battery.ko 2023-10-30T12:14:26.059 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/ds2781_battery.ko 2023-10-30T12:14:26.059 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/ds2782_battery.ko 2023-10-30T12:14:26.059 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/gpio-charger.ko 2023-10-30T12:14:26.060 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/isp1704_charger.ko 2023-10-30T12:14:26.060 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/lp8727_charger.ko 2023-10-30T12:14:26.060 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/max14577_charger.ko 2023-10-30T12:14:26.060 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/max17040_battery.ko 2023-10-30T12:14:26.060 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/max17042_battery.ko 2023-10-30T12:14:26.060 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/max8903_charger.ko 2023-10-30T12:14:26.061 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/max8925_power.ko 2023-10-30T12:14:26.061 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/pcf50633-charger.ko 2023-10-30T12:14:26.061 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 38009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/sbs-battery.ko 2023-10-30T12:14:26.061 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/test_power.ko 2023-10-30T12:14:26.061 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/tps65090-charger.ko 2023-10-30T12:14:26.061 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/wm831x_backup.ko 2023-10-30T12:14:26.062 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/wm831x_power.ko 2023-10-30T12:14:26.062 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/wm8350_power.ko 2023-10-30T12:14:26.062 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/powercap/ 2023-10-30T12:14:26.062 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 57729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/powercap/intel_rapl_common.ko 2023-10-30T12:14:26.063 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/powercap/intel_rapl_msr.ko 2023-10-30T12:14:26.063 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/ 2023-10-30T12:14:26.063 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/clients/ 2023-10-30T12:14:26.063 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/clients/pps-gpio.ko 2023-10-30T12:14:26.063 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/clients/pps-ldisc.ko 2023-10-30T12:14:26.064 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/clients/pps_parport.ko 2023-10-30T12:14:26.064 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 41873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/pps_core.ko 2023-10-30T12:14:26.064 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ptp/ 2023-10-30T12:14:26.064 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 66385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ptp/ptp.ko 2023-10-30T12:14:26.064 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ptp/ptp_kvm.ko 2023-10-30T12:14:26.065 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/ 2023-10-30T12:14:26.065 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-88pm80x.ko 2023-10-30T12:14:26.065 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-88pm860x.ko 2023-10-30T12:14:26.065 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-bq32k.ko 2023-10-30T12:14:26.065 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-da9052.ko 2023-10-30T12:14:26.066 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-da9055.ko 2023-10-30T12:14:26.066 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-da9063.ko 2023-10-30T12:14:26.066 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1286.ko 2023-10-30T12:14:26.066 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 68577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1307.ko 2023-10-30T12:14:26.066 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1374.ko 2023-10-30T12:14:26.067 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1511.ko 2023-10-30T12:14:26.067 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1553.ko 2023-10-30T12:14:26.067 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1672.ko 2023-10-30T12:14:26.067 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1742.ko 2023-10-30T12:14:26.067 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds2404.ko 2023-10-30T12:14:26.067 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds3232.ko 2023-10-30T12:14:26.068 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-em3027.ko 2023-10-30T12:14:26.068 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-fm3130.ko 2023-10-30T12:14:26.068 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-isl12022.ko 2023-10-30T12:14:26.068 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-isl1208.ko 2023-10-30T12:14:26.068 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-lp8788.ko 2023-10-30T12:14:26.069 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 38097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-m41t80.ko 2023-10-30T12:14:26.069 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-m48t35.ko 2023-10-30T12:14:26.069 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-m48t59.ko 2023-10-30T12:14:26.069 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-m48t86.ko 2023-10-30T12:14:26.069 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-max6900.ko 2023-10-30T12:14:26.069 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-max8907.ko 2023-10-30T12:14:26.070 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-max8925.ko 2023-10-30T12:14:26.070 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-max8997.ko 2023-10-30T12:14:26.070 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-max8998.ko 2023-10-30T12:14:26.070 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-mc13xxx.ko 2023-10-30T12:14:26.070 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-msm6242.ko 2023-10-30T12:14:26.071 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-palmas.ko 2023-10-30T12:14:26.071 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 46633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf2127.ko 2023-10-30T12:14:26.071 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf50633.ko 2023-10-30T12:14:26.071 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf85063.ko 2023-10-30T12:14:26.071 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf8523.ko 2023-10-30T12:14:26.072 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf8563.ko 2023-10-30T12:14:26.072 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf8583.ko 2023-10-30T12:14:26.072 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rc5t583.ko 2023-10-30T12:14:26.072 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rp5c01.ko 2023-10-30T12:14:26.072 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rs5c372.ko 2023-10-30T12:14:26.072 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-10-30T12:14:26.073 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rx8025.ko 2023-10-30T12:14:26.073 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rx8581.ko 2023-10-30T12:14:26.073 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-s35390a.ko 2023-10-30T12:14:26.073 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-10-30T12:14:26.073 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-tps6586x.ko 2023-10-30T12:14:26.074 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-tps65910.ko 2023-10-30T12:14:26.074 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-wm831x.ko 2023-10-30T12:14:26.074 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-wm8350.ko 2023-10-30T12:14:26.074 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-x1205.ko 2023-10-30T12:14:26.074 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ 2023-10-30T12:14:26.075 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 77617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/3w-9xxx.ko 2023-10-30T12:14:26.075 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 57665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/3w-sas.ko 2023-10-30T12:14:26.075 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 55169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/3w-xxxx.ko 2023-10-30T12:14:26.075 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 135185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/BusLogic.ko 2023-10-30T12:14:26.075 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/a100u2w.ko 2023-10-30T12:14:26.076 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aacraid/ 2023-10-30T12:14:26.076 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 272305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aacraid/aacraid.ko 2023-10-30T12:14:26.076 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 100617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/advansys.ko 2023-10-30T12:14:26.076 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aic7xxx/ 2023-10-30T12:14:26.076 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 356265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-10-30T12:14:26.076 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 295801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-10-30T12:14:26.077 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aic94xx/ 2023-10-30T12:14:26.077 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 189185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aic94xx/aic94xx.ko 2023-10-30T12:14:26.077 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/am53c974.ko 2023-10-30T12:14:26.077 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/arcmsr/ 2023-10-30T12:14:26.077 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 106881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-10-30T12:14:26.078 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 38185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/atp870u.ko 2023-10-30T12:14:26.078 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/be2iscsi/ 2023-10-30T12:14:26.078 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 246985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-10-30T12:14:26.078 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bfa/ 2023-10-30T12:14:26.078 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 1171945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bfa/bfa.ko 2023-10-30T12:14:26.079 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bnx2fc/ 2023-10-30T12:14:26.079 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 237273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-10-30T12:14:26.079 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bnx2i/ 2023-10-30T12:14:26.079 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 115369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-10-30T12:14:26.079 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 40585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ch.ko 2023-10-30T12:14:26.080 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/csiostor/ 2023-10-30T12:14:26.080 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 308921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/csiostor/csiostor.ko 2023-10-30T12:14:26.080 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/ 2023-10-30T12:14:26.080 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/cxgb3i/ 2023-10-30T12:14:26.080 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 76145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-10-30T12:14:26.080 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/cxgb4i/ 2023-10-30T12:14:26.081 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 101449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-10-30T12:14:26.081 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 146689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-10-30T12:14:26.081 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 75553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/dc395x.ko 2023-10-30T12:14:26.081 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 38137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/dmx3191d.ko 2023-10-30T12:14:26.081 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/esas2r/ 2023-10-30T12:14:26.081 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 212425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/esas2r/esas2r.ko 2023-10-30T12:14:26.082 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 60505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/esp_scsi.ko 2023-10-30T12:14:26.082 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fcoe/ 2023-10-30T12:14:26.082 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 78209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fcoe/fcoe.ko 2023-10-30T12:14:26.082 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 158505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-10-30T12:14:26.082 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fdomain.ko 2023-10-30T12:14:26.083 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fnic/ 2023-10-30T12:14:26.083 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 263489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fnic/fnic.ko 2023-10-30T12:14:26.083 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 201817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/hpsa.ko 2023-10-30T12:14:26.083 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 51209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/hptiop.ko 2023-10-30T12:14:26.083 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 49041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/hv_storvsc.ko 2023-10-30T12:14:26.083 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/imm.ko 2023-10-30T12:14:26.084 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 44057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/initio.ko 2023-10-30T12:14:26.084 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 266537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ipr.ko 2023-10-30T12:14:26.084 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 101881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ips.ko 2023-10-30T12:14:26.084 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/isci/ 2023-10-30T12:14:26.084 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 317153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/isci/isci.ko 2023-10-30T12:14:26.085 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-10-30T12:14:26.085 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 52961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/iscsi_tcp.ko 2023-10-30T12:14:26.085 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libfc/ 2023-10-30T12:14:26.085 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 314865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libfc/libfc.ko 2023-10-30T12:14:26.085 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 145585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libiscsi.ko 2023-10-30T12:14:26.086 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 53521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libiscsi_tcp.ko 2023-10-30T12:14:26.086 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libsas/ 2023-10-30T12:14:26.086 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 216665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libsas/libsas.ko 2023-10-30T12:14:26.086 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/lpfc/ 2023-10-30T12:14:26.086 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 2197249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/lpfc/lpfc.ko 2023-10-30T12:14:26.087 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/megaraid/ 2023-10-30T12:14:26.087 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 73569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-10-30T12:14:26.087 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-10-30T12:14:26.087 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 310617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-10-30T12:14:26.087 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 84721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/megaraid.ko 2023-10-30T12:14:26.088 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/mpt3sas/ 2023-10-30T12:14:26.088 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 659777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-10-30T12:14:26.088 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/mvsas/ 2023-10-30T12:14:26.088 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 119561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/mvsas/mvsas.ko 2023-10-30T12:14:26.088 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 78129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/mvumi.ko 2023-10-30T12:14:26.088 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pcmcia/ 2023-10-30T12:14:26.089 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 81625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2023-10-30T12:14:26.089 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2023-10-30T12:14:26.089 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2023-10-30T12:14:26.089 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2023-10-30T12:14:26.089 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pm8001/ 2023-10-30T12:14:26.089 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 364105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-10-30T12:14:26.090 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 123617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pmcraid.ko 2023-10-30T12:14:26.090 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ppa.ko 2023-10-30T12:14:26.090 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 55361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla1280.ko 2023-10-30T12:14:26.090 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla2xxx/ 2023-10-30T12:14:26.090 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 1876625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-10-30T12:14:26.091 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 85289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-10-30T12:14:26.091 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla4xxx/ 2023-10-30T12:14:26.091 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 559033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-10-30T12:14:26.091 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qlogicfas408.ko 2023-10-30T12:14:26.091 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/raid_class.ko 2023-10-30T12:14:26.091 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 238825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_debug.ko 2023-10-30T12:14:26.092 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 173937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_transport_fc.ko 2023-10-30T12:14:26.092 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 279313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-10-30T12:14:26.092 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 97729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_transport_sas.ko 2023-10-30T12:14:26.092 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 73881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_transport_spi.ko 2023-10-30T12:14:26.092 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 44993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_transport_srp.ko 2023-10-30T12:14:26.092 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ses.ko 2023-10-30T12:14:26.093 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 115889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/st.ko 2023-10-30T12:14:26.093 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 46825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/stex.ko 2023-10-30T12:14:26.093 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/sym53c8xx_2/ 2023-10-30T12:14:26.093 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 162121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-10-30T12:14:26.093 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 42297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/virtio_scsi.ko 2023-10-30T12:14:26.093 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 50321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/vmw_pvscsi.ko 2023-10-30T12:14:26.094 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 40025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/wd719x.ko 2023-10-30T12:14:26.094 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 50025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/xen-scsifront.ko 2023-10-30T12:14:26.094 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ssb/ 2023-10-30T12:14:26.094 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 129441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ssb/ssb.ko 2023-10-30T12:14:26.094 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/ 2023-10-30T12:14:26.095 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/iscsi/ 2023-10-30T12:14:26.095 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 736601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-10-30T12:14:26.095 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/loopback/ 2023-10-30T12:14:26.095 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 49985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/loopback/tcm_loop.ko 2023-10-30T12:14:26.095 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/target_core_file.ko 2023-10-30T12:14:26.096 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 43025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/target_core_iblock.ko 2023-10-30T12:14:26.096 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 930321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/target_core_mod.ko 2023-10-30T12:14:26.096 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 44305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/target_core_pscsi.ko 2023-10-30T12:14:26.096 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 122521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/target_core_user.ko 2023-10-30T12:14:26.096 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/tcm_fc/ 2023-10-30T12:14:26.097 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 75593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-10-30T12:14:26.097 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/ 2023-10-30T12:14:26.097 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/ 2023-10-30T12:14:26.097 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/ 2023-10-30T12:14:26.097 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-10-30T12:14:26.098 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-10-30T12:14:26.098 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-10-30T12:14:26.098 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-10-30T12:14:26.098 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-10-30T12:14:26.098 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-10-30T12:14:26.099 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-10-30T12:14:26.099 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-10-30T12:14:26.099 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-10-30T12:14:26.099 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-10-30T12:14:26.099 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-10-30T12:14:26.100 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 37065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-10-30T12:14:26.100 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/intel_powerclamp.ko 2023-10-30T12:14:26.100 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-10-30T12:14:26.100 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-10-30T12:14:26.101 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-10-30T12:14:26.101 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/ 2023-10-30T12:14:26.101 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/ipwireless/ 2023-10-30T12:14:26.101 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 78425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-10-30T12:14:26.101 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 57865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/moxa.ko 2023-10-30T12:14:26.102 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 50953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/mxser.ko 2023-10-30T12:14:26.102 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 108601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/n_gsm.ko 2023-10-30T12:14:26.102 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/n_hdlc.ko 2023-10-30T12:14:26.102 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 67929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/nozomi.ko 2023-10-30T12:14:26.102 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/ 2023-10-30T12:14:26.103 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/8250/ 2023-10-30T12:14:26.103 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/8250/8250_dw.ko 2023-10-30T12:14:26.103 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 59497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-10-30T12:14:26.103 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/altera_jtaguart.ko 2023-10-30T12:14:26.103 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/altera_uart.ko 2023-10-30T12:14:26.104 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/arc_uart.ko 2023-10-30T12:14:26.104 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 72425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/fsl_lpuart.ko 2023-10-30T12:14:26.104 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/jsm/ 2023-10-30T12:14:26.104 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 103073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/jsm/jsm.ko 2023-10-30T12:14:26.104 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/rp2.ko 2023-10-30T12:14:26.104 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 52817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/sc16is7xx.ko 2023-10-30T12:14:26.105 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/ 2023-10-30T12:14:26.105 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/core/ 2023-10-30T12:14:26.105 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 531265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/core/ufshcd-core.ko 2023-10-30T12:14:26.105 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/host/ 2023-10-30T12:14:26.105 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-10-30T12:14:26.106 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2023-10-30T12:14:26.106 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/ 2023-10-30T12:14:26.106 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 44513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio.ko 2023-10-30T12:14:26.106 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_aec.ko 2023-10-30T12:14:26.106 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_cif.ko 2023-10-30T12:14:26.107 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_dmem_genirq.ko 2023-10-30T12:14:26.107 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_mf624.ko 2023-10-30T12:14:26.107 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_netx.ko 2023-10-30T12:14:26.107 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_pci_generic.ko 2023-10-30T12:14:26.107 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_pdrv_genirq.ko 2023-10-30T12:14:26.108 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_sercos3.ko 2023-10-30T12:14:26.108 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/ 2023-10-30T12:14:26.108 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/c67x00/ 2023-10-30T12:14:26.108 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 73297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/c67x00/c67x00.ko 2023-10-30T12:14:26.108 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/chipidea/ 2023-10-30T12:14:26.109 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 144377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/chipidea/ci_hdrc.ko 2023-10-30T12:14:26.109 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2023-10-30T12:14:26.109 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2023-10-30T12:14:26.109 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2023-10-30T12:14:26.109 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/class/ 2023-10-30T12:14:26.109 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 87089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/class/cdc-acm.ko 2023-10-30T12:14:26.110 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 51409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/class/cdc-wdm.ko 2023-10-30T12:14:26.110 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 47369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/class/usblp.ko 2023-10-30T12:14:26.110 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 89857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/class/usbtmc.ko 2023-10-30T12:14:26.110 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/common/ 2023-10-30T12:14:26.110 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/common/ulpi.ko 2023-10-30T12:14:26.111 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc2/ 2023-10-30T12:14:26.111 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc2/dwc2_pci.ko 2023-10-30T12:14:26.111 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc3/ 2023-10-30T12:14:26.111 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc3/dwc3-haps.ko 2023-10-30T12:14:26.111 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc3/dwc3-pci.ko 2023-10-30T12:14:26.112 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 444553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc3/dwc3.ko 2023-10-30T12:14:26.112 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/ 2023-10-30T12:14:26.112 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/ 2023-10-30T12:14:26.112 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 60177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/u_ether.ko 2023-10-30T12:14:26.112 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 51881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/u_serial.ko 2023-10-30T12:14:26.113 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2023-10-30T12:14:26.113 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 42449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2023-10-30T12:14:26.113 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2023-10-30T12:14:26.113 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2023-10-30T12:14:26.113 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 92489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2023-10-30T12:14:26.113 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 51049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2023-10-30T12:14:26.114 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 122081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2023-10-30T12:14:26.114 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 57617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2023-10-30T12:14:26.114 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2023-10-30T12:14:26.114 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2023-10-30T12:14:26.114 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 57729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2023-10-30T12:14:26.115 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 85609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2023-10-30T12:14:26.115 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2023-10-30T12:14:26.115 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 58113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2023-10-30T12:14:26.115 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 67337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2023-10-30T12:14:26.115 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/ 2023-10-30T12:14:26.116 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2023-10-30T12:14:26.116 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2023-10-30T12:14:26.116 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2023-10-30T12:14:26.116 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_ether.ko 2023-10-30T12:14:26.116 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2023-10-30T12:14:26.117 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_hid.ko 2023-10-30T12:14:26.117 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2023-10-30T12:14:26.117 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2023-10-30T12:14:26.117 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2023-10-30T12:14:26.117 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_printer.ko 2023-10-30T12:14:26.118 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_serial.ko 2023-10-30T12:14:26.118 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_zero.ko 2023-10-30T12:14:26.118 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 62185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2023-10-30T12:14:26.118 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2023-10-30T12:14:26.118 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 161225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/libcomposite.ko 2023-10-30T12:14:26.119 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/ 2023-10-30T12:14:26.119 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2023-10-30T12:14:26.119 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/bdc/ 2023-10-30T12:14:26.119 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 157681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2023-10-30T12:14:26.119 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/goku_udc.ko 2023-10-30T12:14:26.119 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 52721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/gr_udc.ko 2023-10-30T12:14:26.120 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 69705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2023-10-30T12:14:26.120 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 56033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/mv_udc.ko 2023-10-30T12:14:26.120 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 76057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/net2272.ko 2023-10-30T12:14:26.120 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 75073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/net2280.ko 2023-10-30T12:14:26.120 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 73169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/pch_udc.ko 2023-10-30T12:14:26.121 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 65809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2023-10-30T12:14:26.121 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 66097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2023-10-30T12:14:26.121 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 81505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2023-10-30T12:14:26.121 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 156073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/udc-core.ko 2023-10-30T12:14:26.121 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/ 2023-10-30T12:14:26.121 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/bcma-hcd.ko 2023-10-30T12:14:26.122 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 80113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/isp116x-hcd.ko 2023-10-30T12:14:26.122 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 87353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/oxu210hp-hcd.ko 2023-10-30T12:14:26.122 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 78873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/r8a66597-hcd.ko 2023-10-30T12:14:26.122 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 51329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/sl811-hcd.ko 2023-10-30T12:14:26.122 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/sl811_cs.ko 2023-10-30T12:14:26.123 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/ssb-hcd.ko 2023-10-30T12:14:26.123 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/xhci-plat-hcd.ko 2023-10-30T12:14:26.123 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/image/ 2023-10-30T12:14:26.123 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/image/mdc800.ko 2023-10-30T12:14:26.123 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/image/microtek.ko 2023-10-30T12:14:26.124 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/ 2023-10-30T12:14:26.124 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 41721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/adutux.ko 2023-10-30T12:14:26.124 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/appledisplay.ko 2023-10-30T12:14:26.124 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/cypress_cy7c63.ko 2023-10-30T12:14:26.124 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/cytherm.ko 2023-10-30T12:14:26.125 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/ehset.ko 2023-10-30T12:14:26.125 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/emi26.ko 2023-10-30T12:14:26.125 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/emi62.ko 2023-10-30T12:14:26.125 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/ezusb.ko 2023-10-30T12:14:26.125 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/idmouse.ko 2023-10-30T12:14:26.126 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/iowarrior.ko 2023-10-30T12:14:26.126 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/isight_firmware.ko 2023-10-30T12:14:26.126 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/ldusb.ko 2023-10-30T12:14:26.126 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/legousbtower.ko 2023-10-30T12:14:26.126 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/lvstest.ko 2023-10-30T12:14:26.127 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/sisusbvga/ 2023-10-30T12:14:26.127 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 64841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-10-30T12:14:26.127 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/trancevibrator.ko 2023-10-30T12:14:26.127 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/usb3503.ko 2023-10-30T12:14:26.127 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/usblcd.ko 2023-10-30T12:14:26.128 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/usbsevseg.ko 2023-10-30T12:14:26.129 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 70873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/usbtest.ko 2023-10-30T12:14:26.130 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 41969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/uss720.ko 2023-10-30T12:14:26.130 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/yurex.ko 2023-10-30T12:14:26.130 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/mon/ 2023-10-30T12:14:26.130 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 72953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/mon/usbmon.ko 2023-10-30T12:14:26.131 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/musb/ 2023-10-30T12:14:26.131 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 301545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/musb/musb_hdrc.ko 2023-10-30T12:14:26.131 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/phy/ 2023-10-30T12:14:26.131 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/phy/phy-generic.ko 2023-10-30T12:14:26.131 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2023-10-30T12:14:26.131 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/phy/phy-isp1301.ko 2023-10-30T12:14:26.132 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/phy/phy-tahvo.ko 2023-10-30T12:14:26.132 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ 2023-10-30T12:14:26.132 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/aircable.ko 2023-10-30T12:14:26.132 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ark3116.ko 2023-10-30T12:14:26.132 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/belkin_sa.ko 2023-10-30T12:14:26.132 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 38105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ch341.ko 2023-10-30T12:14:26.133 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 70257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/cp210x.ko 2023-10-30T12:14:26.133 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/cyberjack.ko 2023-10-30T12:14:26.133 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 58217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/cypress_m8.ko 2023-10-30T12:14:26.133 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 49817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/digi_acceleport.ko 2023-10-30T12:14:26.133 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/empeg.ko 2023-10-30T12:14:26.134 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 44337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/f81232.ko 2023-10-30T12:14:26.134 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 175137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ftdi_sio.ko 2023-10-30T12:14:26.134 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 53113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/garmin_gps.ko 2023-10-30T12:14:26.134 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 137081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/io_edgeport.ko 2023-10-30T12:14:26.134 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 124833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/io_ti.ko 2023-10-30T12:14:26.135 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 67769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ipaq.ko 2023-10-30T12:14:26.135 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ipw.ko 2023-10-30T12:14:26.135 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ir-usb.ko 2023-10-30T12:14:26.135 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 58233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-10-30T12:14:26.135 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 107521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/keyspan.ko 2023-10-30T12:14:26.135 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/keyspan_pda.ko 2023-10-30T12:14:26.136 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/kl5kusb105.ko 2023-10-30T12:14:26.136 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/kobil_sct.ko 2023-10-30T12:14:26.136 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/mct_u232.ko 2023-10-30T12:14:26.136 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/metro-usb.ko 2023-10-30T12:14:26.136 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 66449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/mos7720.ko 2023-10-30T12:14:26.137 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 78217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/mos7840.ko 2023-10-30T12:14:26.137 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 56337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/mxuport.ko 2023-10-30T12:14:26.137 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/navman.ko 2023-10-30T12:14:26.137 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/omninet.ko 2023-10-30T12:14:26.137 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/opticon.ko 2023-10-30T12:14:26.138 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 190633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/option.ko 2023-10-30T12:14:26.138 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/oti6858.ko 2023-10-30T12:14:26.138 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 54353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/pl2303.ko 2023-10-30T12:14:26.138 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/qcaux.ko 2023-10-30T12:14:26.138 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 37825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/qcserial.ko 2023-10-30T12:14:26.139 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/quatech2.ko 2023-10-30T12:14:26.139 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/safe_serial.ko 2023-10-30T12:14:26.139 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 51625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/sierra.ko 2023-10-30T12:14:26.139 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/spcp8x5.ko 2023-10-30T12:14:26.139 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ssu100.ko 2023-10-30T12:14:26.139 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/symbolserial.ko 2023-10-30T12:14:26.140 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 61489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-10-30T12:14:26.140 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-10-30T12:14:26.140 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/usb_debug.ko 2023-10-30T12:14:26.140 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/usb_wwan.ko 2023-10-30T12:14:26.140 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 125609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/usbserial.ko 2023-10-30T12:14:26.140 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/visor.ko 2023-10-30T12:14:26.141 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 37409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/whiteheat.ko 2023-10-30T12:14:26.141 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/wishbone-serial.ko 2023-10-30T12:14:26.141 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/xsens_mt.ko 2023-10-30T12:14:26.141 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ 2023-10-30T12:14:26.141 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 52913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/uas.ko 2023-10-30T12:14:26.142 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-alauda.ko 2023-10-30T12:14:26.142 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-cypress.ko 2023-10-30T12:14:26.142 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-datafab.ko 2023-10-30T12:14:26.142 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-10-30T12:14:26.142 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-freecom.ko 2023-10-30T12:14:26.143 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-isd200.ko 2023-10-30T12:14:26.143 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-10-30T12:14:26.143 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-karma.ko 2023-10-30T12:14:26.143 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-onetouch.ko 2023-10-30T12:14:26.143 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-realtek.ko 2023-10-30T12:14:26.143 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-sddr09.ko 2023-10-30T12:14:26.144 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-sddr55.ko 2023-10-30T12:14:26.144 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-usbat.ko 2023-10-30T12:14:26.144 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 179969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/usb-storage.ko 2023-10-30T12:14:26.144 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/usbip/ 2023-10-30T12:14:26.144 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 67481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/usbip/usbip-core.ko 2023-10-30T12:14:26.145 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 72177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/usbip/usbip-host.ko 2023-10-30T12:14:26.145 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 107001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-10-30T12:14:26.145 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/ 2023-10-30T12:14:26.145 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/pci/ 2023-10-30T12:14:26.145 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 154297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-10-30T12:14:26.146 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/pci/vfio-pci.ko 2023-10-30T12:14:26.146 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 108737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/vfio.ko 2023-10-30T12:14:26.146 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 77609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-10-30T12:14:26.146 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vhost/ 2023-10-30T12:14:26.146 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 113873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vhost/vhost.ko 2023-10-30T12:14:26.147 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vhost/vhost_iotlb.ko 2023-10-30T12:14:26.147 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 58089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vhost/vhost_net.ko 2023-10-30T12:14:26.147 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 89249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vhost/vhost_scsi.ko 2023-10-30T12:14:26.147 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/ 2023-10-30T12:14:26.147 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/ 2023-10-30T12:14:26.147 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/88pm860x_bl.ko 2023-10-30T12:14:26.148 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/aat2870_bl.ko 2023-10-30T12:14:26.148 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/adp5520_bl.ko 2023-10-30T12:14:26.148 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/adp8860_bl.ko 2023-10-30T12:14:26.148 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/adp8870_bl.ko 2023-10-30T12:14:26.148 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/apple_bl.ko 2023-10-30T12:14:26.149 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/as3711_bl.ko 2023-10-30T12:14:26.149 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/bd6107.ko 2023-10-30T12:14:26.149 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/da903x_bl.ko 2023-10-30T12:14:26.149 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/da9052_bl.ko 2023-10-30T12:14:26.149 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/gpio_backlight.ko 2023-10-30T12:14:26.150 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/kb3886_bl.ko 2023-10-30T12:14:26.150 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/lcd.ko 2023-10-30T12:14:26.150 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/lm3533_bl.ko 2023-10-30T12:14:26.150 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/lm3639_bl.ko 2023-10-30T12:14:26.150 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/lv5207lp.ko 2023-10-30T12:14:26.151 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/max8925_bl.ko 2023-10-30T12:14:26.151 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/pandora_bl.ko 2023-10-30T12:14:26.151 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/pcf50633-backlight.ko 2023-10-30T12:14:26.151 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/platform_lcd.ko 2023-10-30T12:14:26.151 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/wm831x_bl.ko 2023-10-30T12:14:26.151 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/ 2023-10-30T12:14:26.152 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/masters/ 2023-10-30T12:14:26.152 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/masters/ds2482.ko 2023-10-30T12:14:26.152 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 38985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/masters/ds2490.ko 2023-10-30T12:14:26.152 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/masters/matrox_w1.ko 2023-10-30T12:14:26.152 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/masters/w1-gpio.ko 2023-10-30T12:14:26.152 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/ 2023-10-30T12:14:26.153 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-10-30T12:14:26.153 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-10-30T12:14:26.153 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-10-30T12:14:26.153 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-10-30T12:14:26.153 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-10-30T12:14:26.153 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-10-30T12:14:26.154 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-10-30T12:14:26.154 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-10-30T12:14:26.154 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-10-30T12:14:26.154 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_smem.ko 2023-10-30T12:14:26.154 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 56425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_therm.ko 2023-10-30T12:14:26.155 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 102673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/wire.ko 2023-10-30T12:14:26.155 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/ 2023-10-30T12:14:26.155 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/acquirewdt.ko 2023-10-30T12:14:26.155 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/advantechwdt.ko 2023-10-30T12:14:26.155 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/alim1535_wdt.ko 2023-10-30T12:14:26.155 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/alim7101_wdt.ko 2023-10-30T12:14:26.156 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/cpu5wdt.ko 2023-10-30T12:14:26.156 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/da9052_wdt.ko 2023-10-30T12:14:26.156 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/da9055_wdt.ko 2023-10-30T12:14:26.156 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/da9063_wdt.ko 2023-10-30T12:14:26.156 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/dw_wdt.ko 2023-10-30T12:14:26.157 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/eurotechwdt.ko 2023-10-30T12:14:26.157 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/f71808e_wdt.ko 2023-10-30T12:14:26.157 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/hpwdt.ko 2023-10-30T12:14:26.157 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/i6300esb.ko 2023-10-30T12:14:26.157 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-10-30T12:14:26.157 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/iTCO_wdt.ko 2023-10-30T12:14:26.158 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/ib700wdt.ko 2023-10-30T12:14:26.158 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/ibmasr.ko 2023-10-30T12:14:26.158 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-10-30T12:14:26.158 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/it8712f_wdt.ko 2023-10-30T12:14:26.158 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/it87_wdt.ko 2023-10-30T12:14:26.159 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/kempld_wdt.ko 2023-10-30T12:14:26.159 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/machzwd.ko 2023-10-30T12:14:26.159 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/mena21_wdt.ko 2023-10-30T12:14:26.159 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/menf21bmc_wdt.ko 2023-10-30T12:14:26.159 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/nv_tco.ko 2023-10-30T12:14:26.159 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/of_xilinx_wdt.ko 2023-10-30T12:14:26.160 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/pc87413_wdt.ko 2023-10-30T12:14:26.160 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 38657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/pcwd_pci.ko 2023-10-30T12:14:26.160 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/pcwd_usb.ko 2023-10-30T12:14:26.160 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/retu_wdt.ko 2023-10-30T12:14:26.160 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sbc60xxwdt.ko 2023-10-30T12:14:26.160 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sbc_epx_c3.ko 2023-10-30T12:14:26.161 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-10-30T12:14:26.161 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sc1200wdt.ko 2023-10-30T12:14:26.161 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sch311x_wdt.ko 2023-10-30T12:14:26.161 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/smsc37b787_wdt.ko 2023-10-30T12:14:26.161 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/softdog.ko 2023-10-30T12:14:26.162 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sp5100_tco.ko 2023-10-30T12:14:26.162 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/twl4030_wdt.ko 2023-10-30T12:14:26.162 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/via_wdt.ko 2023-10-30T12:14:26.162 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-10-30T12:14:26.162 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/w83877f_wdt.ko 2023-10-30T12:14:26.163 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/w83977f_wdt.ko 2023-10-30T12:14:26.163 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/wafer5823wdt.ko 2023-10-30T12:14:26.163 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/wdt_pci.ko 2023-10-30T12:14:26.163 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/wm831x_wdt.ko 2023-10-30T12:14:26.163 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/wm8350_wdt.ko 2023-10-30T12:14:26.164 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/xen_wdt.ko 2023-10-30T12:14:26.164 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/ 2023-10-30T12:14:26.164 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-evtchn.ko 2023-10-30T12:14:26.164 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-gntalloc.ko 2023-10-30T12:14:26.164 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 47865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-gntdev.ko 2023-10-30T12:14:26.164 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-pciback/ 2023-10-30T12:14:26.165 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 176441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-10-30T12:14:26.165 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 40121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-privcmd.ko 2023-10-30T12:14:26.165 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 71249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-scsiback.ko 2023-10-30T12:14:26.165 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xenfs/ 2023-10-30T12:14:26.165 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xenfs/xenfs.ko 2023-10-30T12:14:26.165 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ 2023-10-30T12:14:26.166 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/9p/ 2023-10-30T12:14:26.166 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 166425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/9p/9p.ko 2023-10-30T12:14:26.166 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/adfs/ 2023-10-30T12:14:26.166 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 61073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/adfs/adfs.ko 2023-10-30T12:14:26.166 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/affs/ 2023-10-30T12:14:26.167 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 163593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/affs/affs.ko 2023-10-30T12:14:26.167 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/afs/ 2023-10-30T12:14:26.167 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 932817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/afs/kafs.ko 2023-10-30T12:14:26.167 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/autofs/ 2023-10-30T12:14:26.167 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 101057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/autofs/autofs4.ko 2023-10-30T12:14:26.168 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/befs/ 2023-10-30T12:14:26.168 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 64545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/befs/befs.ko 2023-10-30T12:14:26.168 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/bfs/ 2023-10-30T12:14:26.168 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 41257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/bfs/bfs.ko 2023-10-30T12:14:26.168 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/binfmt_misc.ko 2023-10-30T12:14:26.169 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/btrfs/ 2023-10-30T12:14:26.169 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 3402585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/btrfs/btrfs.ko 2023-10-30T12:14:26.169 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/cachefiles/ 2023-10-30T12:14:26.169 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 306073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/cachefiles/cachefiles.ko 2023-10-30T12:14:26.169 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ceph/ 2023-10-30T12:14:26.170 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 1323945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ceph/ceph.ko 2023-10-30T12:14:26.170 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/coda/ 2023-10-30T12:14:26.170 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 100801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/coda/coda.ko 2023-10-30T12:14:26.170 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/configfs/ 2023-10-30T12:14:26.170 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 108393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/configfs/configfs.ko 2023-10-30T12:14:26.170 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/cramfs/ 2023-10-30T12:14:26.171 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 29625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/cramfs/cramfs.ko 2023-10-30T12:14:26.171 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/dlm/ 2023-10-30T12:14:26.171 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 642577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/dlm/dlm.ko 2023-10-30T12:14:26.171 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/efs/ 2023-10-30T12:14:26.171 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/efs/efs.ko 2023-10-30T12:14:26.172 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/f2fs/ 2023-10-30T12:14:26.172 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 1505025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/f2fs/f2fs.ko 2023-10-30T12:14:26.172 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fat/ 2023-10-30T12:14:26.172 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fat/msdos.ko 2023-10-30T12:14:26.172 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/freevxfs/ 2023-10-30T12:14:26.172 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 42545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/freevxfs/freevxfs.ko 2023-10-30T12:14:26.173 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fscache/ 2023-10-30T12:14:26.173 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 226481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fscache/fscache.ko 2023-10-30T12:14:26.173 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fuse/ 2023-10-30T12:14:26.173 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fuse/cuse.ko 2023-10-30T12:14:26.174 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/gfs2/ 2023-10-30T12:14:26.174 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 1231529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/gfs2/gfs2.ko 2023-10-30T12:14:26.174 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hfs/ 2023-10-30T12:14:26.174 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 141209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hfs/hfs.ko 2023-10-30T12:14:26.174 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hfsplus/ 2023-10-30T12:14:26.175 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 222817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hfsplus/hfsplus.ko 2023-10-30T12:14:26.175 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hpfs/ 2023-10-30T12:14:26.175 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 192329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hpfs/hpfs.ko 2023-10-30T12:14:26.175 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/isofs/ 2023-10-30T12:14:26.175 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 74065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/isofs/isofs.ko 2023-10-30T12:14:26.175 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/jfs/ 2023-10-30T12:14:26.176 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 419497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/jfs/jfs.ko 2023-10-30T12:14:26.176 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/lockd/ 2023-10-30T12:14:26.176 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 280313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/lockd/lockd.ko 2023-10-30T12:14:26.176 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/minix/ 2023-10-30T12:14:26.176 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 89193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/minix/minix.ko 2023-10-30T12:14:26.177 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/netfs/ 2023-10-30T12:14:26.177 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 124649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/netfs/netfs.ko 2023-10-30T12:14:26.177 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/ 2023-10-30T12:14:26.177 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/blocklayout/ 2023-10-30T12:14:26.177 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 81449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-10-30T12:14:26.178 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/filelayout/ 2023-10-30T12:14:26.178 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 59449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-10-30T12:14:26.178 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/flexfilelayout/ 2023-10-30T12:14:26.178 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 117513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-10-30T12:14:26.178 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 1055649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/nfs.ko 2023-10-30T12:14:26.178 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 66441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/nfsv2.ko 2023-10-30T12:14:26.179 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 120177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/nfsv3.ko 2023-10-30T12:14:26.179 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 2041689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/nfsv4.ko 2023-10-30T12:14:26.179 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs_common/ 2023-10-30T12:14:26.179 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs_common/grace.ko 2023-10-30T12:14:26.179 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs_common/nfs_acl.ko 2023-10-30T12:14:26.180 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfsd/ 2023-10-30T12:14:26.180 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 1516809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfsd/nfsd.ko 2023-10-30T12:14:26.180 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nilfs2/ 2023-10-30T12:14:26.180 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 592833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nilfs2/nilfs2.ko 2023-10-30T12:14:26.180 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/ 2023-10-30T12:14:26.181 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-celtic.ko 2023-10-30T12:14:26.181 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-centeuro.ko 2023-10-30T12:14:26.181 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-croatian.ko 2023-10-30T12:14:26.181 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-cyrillic.ko 2023-10-30T12:14:26.181 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-gaelic.ko 2023-10-30T12:14:26.182 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-greek.ko 2023-10-30T12:14:26.182 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-iceland.ko 2023-10-30T12:14:26.182 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-inuit.ko 2023-10-30T12:14:26.182 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-roman.ko 2023-10-30T12:14:26.182 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-romanian.ko 2023-10-30T12:14:26.183 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-turkish.ko 2023-10-30T12:14:26.183 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_ascii.ko 2023-10-30T12:14:26.183 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp1250.ko 2023-10-30T12:14:26.183 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp1251.ko 2023-10-30T12:14:26.183 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp1255.ko 2023-10-30T12:14:26.184 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp737.ko 2023-10-30T12:14:26.184 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp775.ko 2023-10-30T12:14:26.184 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp850.ko 2023-10-30T12:14:26.184 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp852.ko 2023-10-30T12:14:26.184 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp855.ko 2023-10-30T12:14:26.185 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp857.ko 2023-10-30T12:14:26.185 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp860.ko 2023-10-30T12:14:26.185 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp861.ko 2023-10-30T12:14:26.185 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp862.ko 2023-10-30T12:14:26.185 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp863.ko 2023-10-30T12:14:26.186 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp864.ko 2023-10-30T12:14:26.186 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp865.ko 2023-10-30T12:14:26.186 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp866.ko 2023-10-30T12:14:26.186 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp869.ko 2023-10-30T12:14:26.186 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp874.ko 2023-10-30T12:14:26.187 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 94137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp932.ko 2023-10-30T12:14:26.187 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 142673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp936.ko 2023-10-30T12:14:26.187 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 165745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp949.ko 2023-10-30T12:14:26.187 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 118121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp950.ko 2023-10-30T12:14:26.187 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_euc-jp.ko 2023-10-30T12:14:26.188 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-1.ko 2023-10-30T12:14:26.188 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-13.ko 2023-10-30T12:14:26.188 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-14.ko 2023-10-30T12:14:26.188 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-15.ko 2023-10-30T12:14:26.188 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-2.ko 2023-10-30T12:14:26.189 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-3.ko 2023-10-30T12:14:26.189 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-4.ko 2023-10-30T12:14:26.189 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-5.ko 2023-10-30T12:14:26.189 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-6.ko 2023-10-30T12:14:26.189 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-7.ko 2023-10-30T12:14:26.190 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-9.ko 2023-10-30T12:14:26.190 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_koi8-r.ko 2023-10-30T12:14:26.190 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_koi8-ru.ko 2023-10-30T12:14:26.190 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_koi8-u.ko 2023-10-30T12:14:26.190 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_ucs2_utils.ko 2023-10-30T12:14:26.190 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_utf8.ko 2023-10-30T12:14:26.191 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ntfs/ 2023-10-30T12:14:26.191 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 219417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ntfs/ntfs.ko 2023-10-30T12:14:26.191 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/ 2023-10-30T12:14:26.191 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/cluster/ 2023-10-30T12:14:26.191 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 244409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-10-30T12:14:26.191 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/dlm/ 2023-10-30T12:14:26.192 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 440737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-10-30T12:14:26.192 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/dlmfs/ 2023-10-30T12:14:26.192 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 54049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-10-30T12:14:26.192 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 2592433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/ocfs2.ko 2023-10-30T12:14:26.192 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-10-30T12:14:26.193 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-10-30T12:14:26.193 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-10-30T12:14:26.193 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/omfs/ 2023-10-30T12:14:26.193 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 53433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/omfs/omfs.ko 2023-10-30T12:14:26.193 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/overlayfs/ 2023-10-30T12:14:26.194 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 352153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/overlayfs/overlay.ko 2023-10-30T12:14:26.194 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/pstore/ 2023-10-30T12:14:26.194 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 54809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/pstore/ramoops.ko 2023-10-30T12:14:26.194 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/qnx4/ 2023-10-30T12:14:26.194 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/qnx4/qnx4.ko 2023-10-30T12:14:26.195 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/qnx6/ 2023-10-30T12:14:26.195 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 46689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/qnx6/qnx6.ko 2023-10-30T12:14:26.195 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/quota/ 2023-10-30T12:14:26.195 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/quota/quota_tree.ko 2023-10-30T12:14:26.195 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/quota/quota_v1.ko 2023-10-30T12:14:26.195 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/quota/quota_v2.ko 2023-10-30T12:14:26.196 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/reiserfs/ 2023-10-30T12:14:26.196 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 532673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/reiserfs/reiserfs.ko 2023-10-30T12:14:26.196 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/romfs/ 2023-10-30T12:14:26.196 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/romfs/romfs.ko 2023-10-30T12:14:26.196 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/ 2023-10-30T12:14:26.197 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/client/ 2023-10-30T12:14:26.197 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 2668121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/client/cifs.ko 2023-10-30T12:14:26.197 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/common/ 2023-10-30T12:14:26.197 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/common/cifs_arc4.ko 2023-10-30T12:14:26.197 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/common/cifs_md4.ko 2023-10-30T12:14:26.197 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/squashfs/ 2023-10-30T12:14:26.198 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 134641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/squashfs/squashfs.ko 2023-10-30T12:14:26.198 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/sysv/ 2023-10-30T12:14:26.198 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 90721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/sysv/sysv.ko 2023-10-30T12:14:26.198 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/udf/ 2023-10-30T12:14:26.198 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 247737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/udf/udf.ko 2023-10-30T12:14:26.199 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ufs/ 2023-10-30T12:14:26.199 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 162497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ufs/ufs.ko 2023-10-30T12:14:26.199 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/xfs/ 2023-10-30T12:14:26.199 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 4010729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/xfs/xfs.ko 2023-10-30T12:14:26.199 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/ 2023-10-30T12:14:26.200 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/locking/ 2023-10-30T12:14:26.200 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 75489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/locking/locktorture.ko 2023-10-30T12:14:26.200 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/time/ 2023-10-30T12:14:26.200 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/time/test_udelay.ko 2023-10-30T12:14:26.200 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 60553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/torture.ko 2023-10-30T12:14:26.201 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/ 2023-10-30T12:14:26.201 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 6833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crc-itu-t.ko 2023-10-30T12:14:26.201 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 6529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crc7.ko 2023-10-30T12:14:26.201 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crc8.ko 2023-10-30T12:14:26.201 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crypto/ 2023-10-30T12:14:26.202 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crypto/libarc4.ko 2023-10-30T12:14:26.202 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 37081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crypto/libdes.ko 2023-10-30T12:14:26.202 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/interval_tree_test.ko 2023-10-30T12:14:26.202 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/libcrc32c.ko 2023-10-30T12:14:26.203 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/lru_cache.ko 2023-10-30T12:14:26.203 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/lz4/ 2023-10-30T12:14:26.203 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/lz4/lz4_compress.ko 2023-10-30T12:14:26.203 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/lz4/lz4hc_compress.ko 2023-10-30T12:14:26.203 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/math/ 2023-10-30T12:14:26.203 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 6617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/math/cordic.ko 2023-10-30T12:14:26.204 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/memory-notifier-error-inject.ko 2023-10-30T12:14:26.204 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/notifier-error-inject.ko 2023-10-30T12:14:26.204 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/percpu_test.ko 2023-10-30T12:14:26.204 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/pm-notifier-error-inject.ko 2023-10-30T12:14:26.204 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/raid6/ 2023-10-30T12:14:26.205 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 143433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/raid6/raid6_pq.ko 2023-10-30T12:14:26.205 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/rbtree_test.ko 2023-10-30T12:14:26.205 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/reed_solomon/ 2023-10-30T12:14:26.205 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/reed_solomon/reed_solomon.ko 2023-10-30T12:14:26.205 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 38841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test-kstrtox.ko 2023-10-30T12:14:26.206 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test-string_helpers.ko 2023-10-30T12:14:26.206 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 4760425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test_bpf.ko 2023-10-30T12:14:26.206 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 49089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test_firmware.ko 2023-10-30T12:14:26.206 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 6369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test_module.ko 2023-10-30T12:14:26.206 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test_user_copy.ko 2023-10-30T12:14:26.207 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/ts_bm.ko 2023-10-30T12:14:26.207 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/ts_fsm.ko 2023-10-30T12:14:26.207 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/ts_kmp.ko 2023-10-30T12:14:26.207 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/xz/ 2023-10-30T12:14:26.207 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/xz/xz_dec_test.ko 2023-10-30T12:14:26.208 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/mm/ 2023-10-30T12:14:26.208 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/mm/hwpoison-inject.ko 2023-10-30T12:14:26.208 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ 2023-10-30T12:14:26.208 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/ 2023-10-30T12:14:26.208 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 55737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/6lowpan.ko 2023-10-30T12:14:26.209 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_dest.ko 2023-10-30T12:14:26.209 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_fragment.ko 2023-10-30T12:14:26.209 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_hop.ko 2023-10-30T12:14:26.209 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 6985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_ipv6.ko 2023-10-30T12:14:26.209 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_mobility.ko 2023-10-30T12:14:26.210 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_routing.ko 2023-10-30T12:14:26.210 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_udp.ko 2023-10-30T12:14:26.210 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/ 2023-10-30T12:14:26.210 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/garp.ko 2023-10-30T12:14:26.210 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/mrp.ko 2023-10-30T12:14:26.211 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/p8022.ko 2023-10-30T12:14:26.211 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/psnap.ko 2023-10-30T12:14:26.211 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/stp.ko 2023-10-30T12:14:26.211 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/8021q/ 2023-10-30T12:14:26.211 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 81737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/8021q/8021q.ko 2023-10-30T12:14:26.211 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/9p/ 2023-10-30T12:14:26.212 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 198337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/9p/9pnet.ko 2023-10-30T12:14:26.212 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/9p/9pnet_fd.ko 2023-10-30T12:14:26.212 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/9p/9pnet_rdma.ko 2023-10-30T12:14:26.212 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/9p/9pnet_virtio.ko 2023-10-30T12:14:26.212 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/batman-adv/ 2023-10-30T12:14:26.213 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 649297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/batman-adv/batman-adv.ko 2023-10-30T12:14:26.213 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/ 2023-10-30T12:14:26.213 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 63809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/br_netfilter.ko 2023-10-30T12:14:26.213 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 777689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/bridge.ko 2023-10-30T12:14:26.213 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ 2023-10-30T12:14:26.213 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-10-30T12:14:26.214 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_among.ko 2023-10-30T12:14:26.214 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_arp.ko 2023-10-30T12:14:26.214 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-10-30T12:14:26.214 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-10-30T12:14:26.214 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_ip.ko 2023-10-30T12:14:26.215 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-10-30T12:14:26.215 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_limit.ko 2023-10-30T12:14:26.215 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_log.ko 2023-10-30T12:14:26.215 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_mark.ko 2023-10-30T12:14:26.215 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-10-30T12:14:26.216 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-10-30T12:14:26.216 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-10-30T12:14:26.216 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-10-30T12:14:26.216 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_snat.ko 2023-10-30T12:14:26.216 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_stp.ko 2023-10-30T12:14:26.216 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-10-30T12:14:26.217 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-10-30T12:14:26.217 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-10-30T12:14:26.217 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-10-30T12:14:26.217 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 55929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebtables.ko 2023-10-30T12:14:26.217 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-10-30T12:14:26.218 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-10-30T12:14:26.218 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ceph/ 2023-10-30T12:14:26.218 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 1047497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ceph/libceph.ko 2023-10-30T12:14:26.218 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/core/ 2023-10-30T12:14:26.218 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 125697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/core/pktgen.ko 2023-10-30T12:14:26.219 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dccp/ 2023-10-30T12:14:26.219 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 194209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dccp/dccp.ko 2023-10-30T12:14:26.219 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dccp/dccp_diag.ko 2023-10-30T12:14:26.219 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 57881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dccp/dccp_ipv4.ko 2023-10-30T12:14:26.219 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 60337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dccp/dccp_ipv6.ko 2023-10-30T12:14:26.219 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/ 2023-10-30T12:14:26.220 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 390473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/dsa_core.ko 2023-10-30T12:14:26.220 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/tag_brcm.ko 2023-10-30T12:14:26.220 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/tag_dsa.ko 2023-10-30T12:14:26.220 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/tag_none.ko 2023-10-30T12:14:26.220 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/tag_trailer.ko 2023-10-30T12:14:26.221 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/hsr/ 2023-10-30T12:14:26.221 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 99729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/hsr/hsr.ko 2023-10-30T12:14:26.221 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ieee802154/ 2023-10-30T12:14:26.221 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ieee802154/6lowpan/ 2023-10-30T12:14:26.221 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 45233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-10-30T12:14:26.222 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 254625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ieee802154/ieee802154.ko 2023-10-30T12:14:26.222 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 47289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ieee802154/ieee802154_socket.ko 2023-10-30T12:14:26.222 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ 2023-10-30T12:14:26.222 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ah4.ko 2023-10-30T12:14:26.222 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 38905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/esp4.ko 2023-10-30T12:14:26.222 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 48401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/fou.ko 2023-10-30T12:14:26.223 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/gre.ko 2023-10-30T12:14:26.223 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 48657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/inet_diag.ko 2023-10-30T12:14:26.223 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 60633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ip_gre.ko 2023-10-30T12:14:26.223 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 61225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ip_tunnel.ko 2023-10-30T12:14:26.223 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ip_vti.ko 2023-10-30T12:14:26.224 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ipcomp.ko 2023-10-30T12:14:26.224 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ipip.ko 2023-10-30T12:14:26.224 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ 2023-10-30T12:14:26.224 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 37641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/arp_tables.ko 2023-10-30T12:14:26.224 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-10-30T12:14:26.225 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-10-30T12:14:26.225 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 41673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ip_tables.ko 2023-10-30T12:14:26.225 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-10-30T12:14:26.225 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2023-10-30T12:14:26.225 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-10-30T12:14:26.226 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-10-30T12:14:26.226 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-10-30T12:14:26.226 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/iptable_filter.ko 2023-10-30T12:14:26.226 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-10-30T12:14:26.226 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-10-30T12:14:26.227 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-10-30T12:14:26.227 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/iptable_security.ko 2023-10-30T12:14:26.227 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-10-30T12:14:26.227 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-10-30T12:14:26.227 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-10-30T12:14:26.228 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-10-30T12:14:26.228 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-10-30T12:14:26.228 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2023-10-30T12:14:26.228 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-10-30T12:14:26.228 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-10-30T12:14:26.229 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-10-30T12:14:26.229 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-10-30T12:14:26.229 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-10-30T12:14:26.229 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_bic.ko 2023-10-30T12:14:26.229 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_dctcp.ko 2023-10-30T12:14:26.229 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_diag.ko 2023-10-30T12:14:26.230 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_highspeed.ko 2023-10-30T12:14:26.230 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_htcp.ko 2023-10-30T12:14:26.230 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_hybla.ko 2023-10-30T12:14:26.230 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_illinois.ko 2023-10-30T12:14:26.230 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_lp.ko 2023-10-30T12:14:26.231 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_scalable.ko 2023-10-30T12:14:26.231 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_vegas.ko 2023-10-30T12:14:26.231 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_veno.ko 2023-10-30T12:14:26.231 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_westwood.ko 2023-10-30T12:14:26.231 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_yeah.ko 2023-10-30T12:14:26.232 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tunnel4.ko 2023-10-30T12:14:26.232 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/udp_diag.ko 2023-10-30T12:14:26.232 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 41665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/udp_tunnel.ko 2023-10-30T12:14:26.232 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/xfrm4_tunnel.ko 2023-10-30T12:14:26.232 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ 2023-10-30T12:14:26.232 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ah6.ko 2023-10-30T12:14:26.233 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/esp6.ko 2023-10-30T12:14:26.233 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/fou6.ko 2023-10-30T12:14:26.233 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 80897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ip6_gre.ko 2023-10-30T12:14:26.233 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 93577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ip6_tunnel.ko 2023-10-30T12:14:26.233 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-10-30T12:14:26.234 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 53273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ip6_vti.ko 2023-10-30T12:14:26.234 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ipcomp6.ko 2023-10-30T12:14:26.234 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/mip6.ko 2023-10-30T12:14:26.234 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ 2023-10-30T12:14:26.234 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 42169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-10-30T12:14:26.235 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2023-10-30T12:14:26.235 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-10-30T12:14:26.235 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-10-30T12:14:26.235 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-10-30T12:14:26.235 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-10-30T12:14:26.235 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-10-30T12:14:26.236 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-10-30T12:14:26.236 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-10-30T12:14:26.236 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-10-30T12:14:26.236 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-10-30T12:14:26.236 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-10-30T12:14:26.237 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-10-30T12:14:26.237 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-10-30T12:14:26.237 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-10-30T12:14:26.237 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-10-30T12:14:26.237 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-10-30T12:14:26.238 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-10-30T12:14:26.238 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-10-30T12:14:26.238 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-10-30T12:14:26.238 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-10-30T12:14:26.239 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-10-30T12:14:26.239 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-10-30T12:14:26.239 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-10-30T12:14:26.239 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-10-30T12:14:26.239 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 74073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/sit.ko 2023-10-30T12:14:26.240 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/tunnel6.ko 2023-10-30T12:14:26.240 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/xfrm6_tunnel.ko 2023-10-30T12:14:26.240 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/key/ 2023-10-30T12:14:26.240 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 82833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/key/af_key.ko 2023-10-30T12:14:26.240 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/ 2023-10-30T12:14:26.240 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 125281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_core.ko 2023-10-30T12:14:26.241 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_debugfs.ko 2023-10-30T12:14:26.241 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 25953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_eth.ko 2023-10-30T12:14:26.241 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_ip.ko 2023-10-30T12:14:26.241 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_ip6.ko 2023-10-30T12:14:26.241 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_netlink.ko 2023-10-30T12:14:26.242 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 53241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_ppp.ko 2023-10-30T12:14:26.242 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/lapb/ 2023-10-30T12:14:26.242 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 47977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/lapb/lapb.ko 2023-10-30T12:14:26.242 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/llc/ 2023-10-30T12:14:26.242 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/llc/llc.ko 2023-10-30T12:14:26.242 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 267777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/llc/llc2.ko 2023-10-30T12:14:26.243 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/mac802154/ 2023-10-30T12:14:26.243 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 290417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/mac802154/mac802154.ko 2023-10-30T12:14:26.243 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/mpls/ 2023-10-30T12:14:26.243 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/mpls/mpls_gso.ko 2023-10-30T12:14:26.243 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ 2023-10-30T12:14:26.244 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ 2023-10-30T12:14:26.244 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 119585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set.ko 2023-10-30T12:14:26.244 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-10-30T12:14:26.244 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-10-30T12:14:26.244 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-10-30T12:14:26.245 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 100297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-10-30T12:14:26.245 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 101441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-10-30T12:14:26.245 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 102817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-10-30T12:14:26.245 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 103481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-10-30T12:14:26.245 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 112849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-10-30T12:14:26.246 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 60329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-10-30T12:14:26.246 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 109593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-10-30T12:14:26.246 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 113129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-10-30T12:14:26.246 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 112585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-10-30T12:14:26.246 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 111585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-10-30T12:14:26.247 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 114465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-10-30T12:14:26.247 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-10-30T12:14:26.247 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ 2023-10-30T12:14:26.247 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 411729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs.ko 2023-10-30T12:14:26.247 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-10-30T12:14:26.248 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-10-30T12:14:26.248 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-10-30T12:14:26.248 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-10-30T12:14:26.248 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-10-30T12:14:26.248 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-10-30T12:14:26.248 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-10-30T12:14:26.249 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-10-30T12:14:26.249 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-10-30T12:14:26.249 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-10-30T12:14:26.249 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-10-30T12:14:26.249 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-10-30T12:14:26.250 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-10-30T12:14:26.250 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 49257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conncount.ko 2023-10-30T12:14:26.250 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 408257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack.ko 2023-10-30T12:14:26.250 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-10-30T12:14:26.250 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-10-30T12:14:26.251 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-10-30T12:14:26.251 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 118145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_h323.ko 2023-10-30T12:14:26.251 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_irc.ko 2023-10-30T12:14:26.251 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-10-30T12:14:26.251 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 107297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-10-30T12:14:26.252 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-10-30T12:14:26.252 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_sane.ko 2023-10-30T12:14:26.252 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 66641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_sip.ko 2023-10-30T12:14:26.252 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-10-30T12:14:26.252 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-10-30T12:14:26.253 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_dup_netdev.ko 2023-10-30T12:14:26.253 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 89473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_flow_table.ko 2023-10-30T12:14:26.253 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_flow_table_inet.ko 2023-10-30T12:14:26.253 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 37129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_log_syslog.ko 2023-10-30T12:14:26.253 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 95417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat.ko 2023-10-30T12:14:26.253 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat_amanda.ko 2023-10-30T12:14:26.254 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat_ftp.ko 2023-10-30T12:14:26.254 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat_irc.ko 2023-10-30T12:14:26.254 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat_sip.ko 2023-10-30T12:14:26.254 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat_tftp.ko 2023-10-30T12:14:26.254 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 44065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_synproxy_core.ko 2023-10-30T12:14:26.255 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 615313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_tables.ko 2023-10-30T12:14:26.255 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 42561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink.ko 2023-10-30T12:14:26.255 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink_acct.ko 2023-10-30T12:14:26.255 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink_cttimeout.ko 2023-10-30T12:14:26.255 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 39785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink_log.ko 2023-10-30T12:14:26.255 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink_osf.ko 2023-10-30T12:14:26.256 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 49377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink_queue.ko 2023-10-30T12:14:26.256 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_chain_nat.ko 2023-10-30T12:14:26.256 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_compat.ko 2023-10-30T12:14:26.256 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 40721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_ct.ko 2023-10-30T12:14:26.256 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_dup_netdev.ko 2023-10-30T12:14:26.256 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_fib.ko 2023-10-30T12:14:26.257 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_fwd_netdev.ko 2023-10-30T12:14:26.257 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_hash.ko 2023-10-30T12:14:26.257 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_limit.ko 2023-10-30T12:14:26.257 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_log.ko 2023-10-30T12:14:26.257 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_masq.ko 2023-10-30T12:14:26.258 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_nat.ko 2023-10-30T12:14:26.258 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_numgen.ko 2023-10-30T12:14:26.258 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_osf.ko 2023-10-30T12:14:26.258 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_queue.ko 2023-10-30T12:14:26.258 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_redir.ko 2023-10-30T12:14:26.259 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_reject.ko 2023-10-30T12:14:26.259 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_reject_inet.ko 2023-10-30T12:14:26.259 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_socket.ko 2023-10-30T12:14:26.259 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_synproxy.ko 2023-10-30T12:14:26.259 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_tproxy.ko 2023-10-30T12:14:26.260 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 89313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/x_tables.ko 2023-10-30T12:14:26.260 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_AUDIT.ko 2023-10-30T12:14:26.260 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_CHECKSUM.ko 2023-10-30T12:14:26.260 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_CLASSIFY.ko 2023-10-30T12:14:26.260 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-10-30T12:14:26.260 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 18169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_CT.ko 2023-10-30T12:14:26.261 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_DSCP.ko 2023-10-30T12:14:26.261 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_HL.ko 2023-10-30T12:14:26.261 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_HMARK.ko 2023-10-30T12:14:26.261 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 38873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_IDLETIMER.ko 2023-10-30T12:14:26.261 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_LED.ko 2023-10-30T12:14:26.261 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_LOG.ko 2023-10-30T12:14:26.262 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_MASQUERADE.ko 2023-10-30T12:14:26.262 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_NETMAP.ko 2023-10-30T12:14:26.262 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_NFLOG.ko 2023-10-30T12:14:26.262 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_NFQUEUE.ko 2023-10-30T12:14:26.262 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_RATEEST.ko 2023-10-30T12:14:26.263 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_REDIRECT.ko 2023-10-30T12:14:26.263 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 14337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_SECMARK.ko 2023-10-30T12:14:26.263 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_TCPMSS.ko 2023-10-30T12:14:26.263 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-10-30T12:14:26.263 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_TEE.ko 2023-10-30T12:14:26.263 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_TPROXY.ko 2023-10-30T12:14:26.264 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_TRACE.ko 2023-10-30T12:14:26.264 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_addrtype.ko 2023-10-30T12:14:26.264 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_bpf.ko 2023-10-30T12:14:26.264 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_cgroup.ko 2023-10-30T12:14:26.264 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_cluster.ko 2023-10-30T12:14:26.265 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_comment.ko 2023-10-30T12:14:26.265 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_connbytes.ko 2023-10-30T12:14:26.265 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_connlabel.ko 2023-10-30T12:14:26.265 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_connlimit.ko 2023-10-30T12:14:26.265 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_connmark.ko 2023-10-30T12:14:26.265 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_conntrack.ko 2023-10-30T12:14:26.266 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_cpu.ko 2023-10-30T12:14:26.266 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_dccp.ko 2023-10-30T12:14:26.266 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_devgroup.ko 2023-10-30T12:14:26.266 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_dscp.ko 2023-10-30T12:14:26.266 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_ecn.ko 2023-10-30T12:14:26.267 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_esp.ko 2023-10-30T12:14:26.267 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 43289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_hashlimit.ko 2023-10-30T12:14:26.267 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_helper.ko 2023-10-30T12:14:26.267 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_hl.ko 2023-10-30T12:14:26.267 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_ipcomp.ko 2023-10-30T12:14:26.268 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_iprange.ko 2023-10-30T12:14:26.268 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_ipvs.ko 2023-10-30T12:14:26.268 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_l2tp.ko 2023-10-30T12:14:26.268 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_length.ko 2023-10-30T12:14:26.268 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_limit.ko 2023-10-30T12:14:26.269 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_mac.ko 2023-10-30T12:14:26.269 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_mark.ko 2023-10-30T12:14:26.269 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_multiport.ko 2023-10-30T12:14:26.269 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_nat.ko 2023-10-30T12:14:26.269 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_nfacct.ko 2023-10-30T12:14:26.269 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_osf.ko 2023-10-30T12:14:26.270 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_owner.ko 2023-10-30T12:14:26.270 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_physdev.ko 2023-10-30T12:14:26.270 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_pkttype.ko 2023-10-30T12:14:26.270 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_policy.ko 2023-10-30T12:14:26.271 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_quota.ko 2023-10-30T12:14:26.271 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_rateest.ko 2023-10-30T12:14:26.271 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_realm.ko 2023-10-30T12:14:26.271 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_recent.ko 2023-10-30T12:14:26.271 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_sctp.ko 2023-10-30T12:14:26.271 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_set.ko 2023-10-30T12:14:26.272 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 16089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_socket.ko 2023-10-30T12:14:26.272 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_state.ko 2023-10-30T12:14:26.272 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_statistic.ko 2023-10-30T12:14:26.272 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_string.ko 2023-10-30T12:14:26.272 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 8553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_tcpmss.ko 2023-10-30T12:14:26.273 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_tcpudp.ko 2023-10-30T12:14:26.273 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_time.ko 2023-10-30T12:14:26.273 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_u32.ko 2023-10-30T12:14:26.273 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netlink/ 2023-10-30T12:14:26.273 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netlink/netlink_diag.ko 2023-10-30T12:14:26.274 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/nsh/ 2023-10-30T12:14:26.274 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 12425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/nsh/nsh.ko 2023-10-30T12:14:26.274 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/openvswitch/ 2023-10-30T12:14:26.274 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 380033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/openvswitch/openvswitch.ko 2023-10-30T12:14:26.274 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/openvswitch/vport-geneve.ko 2023-10-30T12:14:26.275 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/openvswitch/vport-gre.ko 2023-10-30T12:14:26.275 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 11793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/openvswitch/vport-vxlan.ko 2023-10-30T12:14:26.275 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/packet/ 2023-10-30T12:14:26.275 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/packet/af_packet_diag.ko 2023-10-30T12:14:26.275 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/phonet/ 2023-10-30T12:14:26.275 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 110201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/phonet/phonet.ko 2023-10-30T12:14:26.276 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 54697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/phonet/pn_pep.ko 2023-10-30T12:14:26.276 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rds/ 2023-10-30T12:14:26.276 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 251481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rds/rds.ko 2023-10-30T12:14:26.276 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 164937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rds/rds_rdma.ko 2023-10-30T12:14:26.276 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 67377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rds/rds_tcp.ko 2023-10-30T12:14:26.277 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rxrpc/ 2023-10-30T12:14:26.277 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 819657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rxrpc/rxrpc.ko 2023-10-30T12:14:26.277 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/ 2023-10-30T12:14:26.277 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_csum.ko 2023-10-30T12:14:26.277 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_gact.ko 2023-10-30T12:14:26.277 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_ipt.ko 2023-10-30T12:14:26.278 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_mirred.ko 2023-10-30T12:14:26.278 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_nat.ko 2023-10-30T12:14:26.278 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 43449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_pedit.ko 2023-10-30T12:14:26.278 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_police.ko 2023-10-30T12:14:26.278 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_simple.ko 2023-10-30T12:14:26.279 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_skbedit.ko 2023-10-30T12:14:26.279 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_vlan.ko 2023-10-30T12:14:26.279 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_basic.ko 2023-10-30T12:14:26.279 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_bpf.ko 2023-10-30T12:14:26.279 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_cgroup.ko 2023-10-30T12:14:26.279 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_flow.ko 2023-10-30T12:14:26.280 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_fw.ko 2023-10-30T12:14:26.280 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 38489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_route.ko 2023-10-30T12:14:26.280 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 61409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_u32.ko 2023-10-30T12:14:26.280 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_cmp.ko 2023-10-30T12:14:26.280 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 13409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_ipset.ko 2023-10-30T12:14:26.281 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_meta.ko 2023-10-30T12:14:26.281 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_nbyte.ko 2023-10-30T12:14:26.281 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_text.ko 2023-10-30T12:14:26.281 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_u32.ko 2023-10-30T12:14:26.281 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 21649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_choke.ko 2023-10-30T12:14:26.281 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_codel.ko 2023-10-30T12:14:26.282 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 34633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_drr.ko 2023-10-30T12:14:26.282 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 33609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_fq.ko 2023-10-30T12:14:26.282 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_fq_codel.ko 2023-10-30T12:14:26.282 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 35433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_gred.ko 2023-10-30T12:14:26.282 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 49601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_hfsc.ko 2023-10-30T12:14:26.283 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_hhf.ko 2023-10-30T12:14:26.283 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 66393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_htb.ko 2023-10-30T12:14:26.283 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_ingress.ko 2023-10-30T12:14:26.283 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 30273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_mqprio.ko 2023-10-30T12:14:26.283 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_mqprio_lib.ko 2023-10-30T12:14:26.284 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_multiq.ko 2023-10-30T12:14:26.284 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 36137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_netem.ko 2023-10-30T12:14:26.284 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 27497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_pie.ko 2023-10-30T12:14:26.284 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_plug.ko 2023-10-30T12:14:26.284 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 24729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_prio.ko 2023-10-30T12:14:26.285 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 48793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_qfq.ko 2023-10-30T12:14:26.285 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 32905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_red.ko 2023-10-30T12:14:26.285 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 26881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_sfb.ko 2023-10-30T12:14:26.285 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 31353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_sfq.ko 2023-10-30T12:14:26.285 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_tbf.ko 2023-10-30T12:14:26.286 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 28105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_teql.ko 2023-10-30T12:14:26.286 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sctp/ 2023-10-30T12:14:26.286 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 929745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sctp/sctp.ko 2023-10-30T12:14:26.286 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 19033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sctp/sctp_diag.ko 2023-10-30T12:14:26.287 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/ 2023-10-30T12:14:26.287 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/auth_gss/ 2023-10-30T12:14:26.287 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 336529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-10-30T12:14:26.287 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 54737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-10-30T12:14:26.287 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 1520161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/sunrpc.ko 2023-10-30T12:14:26.288 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/xprtrdma/ 2023-10-30T12:14:26.288 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 707721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-10-30T12:14:26.288 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/tipc/ 2023-10-30T12:14:26.288 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/tipc/diag.ko 2023-10-30T12:14:26.288 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 862641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/tipc/tipc.ko 2023-10-30T12:14:26.289 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/unix/ 2023-10-30T12:14:26.289 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/unix/unix_diag.ko 2023-10-30T12:14:26.289 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/ 2023-10-30T12:14:26.289 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 85337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-10-30T12:14:26.289 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 68049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-10-30T12:14:26.290 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 96193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/vsock.ko 2023-10-30T12:14:26.290 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 10705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/vsock_diag.ko 2023-10-30T12:14:26.290 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 17521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/vsock_loopback.ko 2023-10-30T12:14:26.290 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/x25/ 2023-10-30T12:14:26.290 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 125177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/x25/x25.ko 2023-10-30T12:14:26.291 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/xfrm/ 2023-10-30T12:14:26.291 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 22657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/xfrm/xfrm_algo.ko 2023-10-30T12:14:26.291 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 20649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/xfrm/xfrm_ipcomp.ko 2023-10-30T12:14:26.291 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 102257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/xfrm/xfrm_user.ko 2023-10-30T12:14:26.291 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/virt/ 2023-10-30T12:14:26.292 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/virt/lib/ 2023-10-30T12:14:26.292 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 15537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/virt/lib/irqbypass.ko 2023-10-30T12:14:26.292 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 639863 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.alias 2023-10-30T12:14:26.292 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 623919 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.alias.bin 2023-10-30T12:14:26.292 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 7663 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.builtin 2023-10-30T12:14:26.293 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 23428 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.builtin.alias.bin 2023-10-30T12:14:26.293 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 9821 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.builtin.bin 2023-10-30T12:14:26.293 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 70004 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.builtin.modinfo 2023-10-30T12:14:26.293 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 146313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.dep 2023-10-30T12:14:26.293 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 232548 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.dep.bin 2023-10-30T12:14:26.294 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 210 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.devname 2023-10-30T12:14:26.294 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 72606 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.order 2023-10-30T12:14:26.294 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 879 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.softdep 2023-10-30T12:14:26.294 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 254438 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.symbols 2023-10-30T12:14:26.294 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 309363 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.symbols.bin 2023-10-30T12:14:26.295 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./usr/ 2023-10-30T12:14:26.295 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./usr/share/ 2023-10-30T12:14:26.295 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./usr/share/doc/ 2023-10-30T12:14:26.295 INFO:teuthology.orchestra.run.smithi089.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./usr/share/doc/linux-image-6.6.0-rc5-geacf00cc51e2/ 2023-10-30T12:14:26.295 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 173 2023-10-30 03:46 ./usr/share/doc/linux-image-6.6.0-rc5-geacf00cc51e2/changelog.Debian.gz 2023-10-30T12:14:26.295 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r-- root/root 691 2023-10-30 03:46 ./usr/share/doc/linux-image-6.6.0-rc5-geacf00cc51e2/copyright 2023-10-30T12:14:26.296 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-rc5-geacf00cc51e2 2023-10-30T12:14:26.297 INFO:teuthology.task.kernel:searching for kernel 6.6.0-rc5-geacf00cc51e2 2023-10-30T12:14:26.297 DEBUG:teuthology.orchestra.run.smithi089:> egrep '(submenu|menuentry.*6.6.0-rc5-geacf00cc51e2).*{' /boot/grub/grub.cfg 2023-10-30T12:14:26.305 INFO:teuthology.orchestra.run.smithi089.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-10-30T12:14:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: menuentry 'Ubuntu, with Linux 6.6.0-rc5-geacf00cc51e2' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.6.0-rc5-geacf00cc51e2-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-10-30T12:14:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: menuentry 'Ubuntu, with Linux 6.6.0-rc5-geacf00cc51e2 (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.6.0-rc5-geacf00cc51e2-recovery-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-10-30T12:14:26.306 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2023-10-30T12:14:26.306 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.6.0-rc5-geacf00cc51e2 2023-10-30T12:14:26.306 DEBUG:teuthology.orchestra.run.smithi089:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2023-10-30T12:14:26.349 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi089.front.sepia.ceph.com to complete... 2023-10-30T12:14:26.761 INFO:teuthology.orchestra.run.smithi089.stderr:Sourcing file `/etc/default/grub' 2023-10-30T12:14:26.762 INFO:teuthology.orchestra.run.smithi089.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-30T12:14:26.790 INFO:teuthology.orchestra.run.smithi089.stderr:Generating grub configuration file ... 2023-10-30T12:14:27.045 INFO:teuthology.orchestra.run.smithi089.stderr:Found linux image: /boot/vmlinuz-6.6.0-rc5-geacf00cc51e2 2023-10-30T12:14:27.052 INFO:teuthology.orchestra.run.smithi089.stderr:Found initrd image: /boot/initrd.img-6.6.0-rc5-geacf00cc51e2 2023-10-30T12:14:27.316 INFO:teuthology.orchestra.run.smithi089.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-30T12:14:27.321 INFO:teuthology.orchestra.run.smithi089.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-30T12:14:27.361 INFO:teuthology.orchestra.run.smithi089.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-30T12:14:27.366 INFO:teuthology.orchestra.run.smithi089.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-30T12:14:27.499 INFO:teuthology.orchestra.run.smithi089.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-30T12:14:27.499 INFO:teuthology.orchestra.run.smithi089.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-30T12:14:27.499 INFO:teuthology.orchestra.run.smithi089.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-30T12:14:27.523 INFO:teuthology.orchestra.run.smithi089.stderr:done 2023-10-30T12:14:58.654 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-10-30T12:14:58.655 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-10-30T12:14:58.656 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T12:15:17.118 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-10-30T12:15:26.120 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-10-30T12:15:26.121 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T12:15:29.178 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-10-30T12:15:41.179 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-10-30T12:15:41.180 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T12:15:59.610 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-10-30T12:16:14.612 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-10-30T12:16:14.613 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T12:16:14.826 DEBUG:teuthology.orchestra.run.smithi089:> true 2023-10-30T12:16:15.520 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-10-30T12:16:15.520 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.6.0-rc5-geacf00cc51e2"... 2023-10-30T12:16:15.521 DEBUG:teuthology.orchestra.run.smithi089:> uname -r 2023-10-30T12:16:15.572 INFO:teuthology.orchestra.run.smithi089.stdout:6.6.0-rc5-geacf00cc51e2 2023-10-30T12:16:15.573 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-rc5-geacf00cc51e2 vs 6.6.0-rc5-geacf00cc51e2 2023-10-30T12:16:15.573 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-30T12:16:15.573 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-10-30T12:16:16.574 DEBUG:teuthology.parallel:result is None 2023-10-30T12:16:16.575 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-30T12:16:16.585 INFO:teuthology.task.internal:Creating test directory... 2023-10-30T12:16:16.585 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-30T12:16:16.589 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-30T12:16:16.595 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-30T12:16:16.602 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-30T12:16:16.610 INFO:teuthology.task.internal:Creating archive directory... 2023-10-30T12:16:16.610 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-30T12:16:16.644 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-30T12:16:16.673 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-30T12:16:16.680 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-30T12:16:16.680 DEBUG:teuthology.orchestra.run.smithi192:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-30T12:16:16.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:16:16.706 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-30T12:16:16.716 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-30T12:16:16.783 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T12:16:16.795 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T12:16:16.808 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T12:16:16.816 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T12:16:16.818 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-30T12:16:16.825 INFO:teuthology.task.internal:Configuring sudo... 2023-10-30T12:16:16.826 DEBUG:teuthology.orchestra.run.smithi089:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-30T12:16:16.829 DEBUG:teuthology.orchestra.run.smithi192:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-30T12:16:16.867 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-30T12:16:16.876 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-30T12:16:16.876 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-30T12:16:16.892 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-30T12:16:16.914 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T12:16:16.953 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T12:16:17.009 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-10-30T12:16:17.010 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-30T12:16:17.082 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T12:16:17.088 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T12:16:17.134 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-30T12:16:17.135 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-30T12:16:17.205 DEBUG:teuthology.orchestra.run.smithi089:> sudo service rsyslog restart 2023-10-30T12:16:17.209 DEBUG:teuthology.orchestra.run.smithi192:> sudo service rsyslog restart 2023-10-30T12:16:17.329 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-30T12:16:17.337 INFO:teuthology.task.internal:Starting timer... 2023-10-30T12:16:17.338 INFO:teuthology.run_tasks:Running task pcp... 2023-10-30T12:16:17.348 INFO:teuthology.run_tasks:Running task selinux... 2023-10-30T12:16:17.356 DEBUG:teuthology.task.selinux:Excluding smithi089: OS 'ubuntu' does not support SELinux 2023-10-30T12:16:17.356 DEBUG:teuthology.task.selinux:Excluding smithi192: OS 'ubuntu' does not support SELinux 2023-10-30T12:16:17.357 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-30T12:16:17.357 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-10-30T12:16:17.357 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-30T12:16:17.358 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-30T12:16:17.367 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-30T12:16:17.373 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-30T12:16:17.503 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-30T12:16:17.517 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-30T12:16:17.518 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi089.front.sepia.ceph.com,smithi192.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-30T12:24:46.864 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi089.front.sepia.ceph.com'), Remote(name='ubuntu@smithi192.front.sepia.ceph.com')] 2023-10-30T12:24:46.866 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-10-30T12:24:46.867 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T12:24:46.947 DEBUG:teuthology.orchestra.run.smithi089:> true 2023-10-30T12:24:47.069 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-10-30T12:24:47.070 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-10-30T12:24:47.071 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T12:24:47.143 DEBUG:teuthology.orchestra.run.smithi192:> true 2023-10-30T12:24:47.243 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-10-30T12:24:47.243 INFO:teuthology.run_tasks:Running task clock... 2023-10-30T12:24:47.261 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-30T12:24:47.262 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-30T12:24:47.263 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T12:24:47.268 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-30T12:24:47.268 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T12:24:47.304 INFO:teuthology.orchestra.run.smithi089.stdout:30 Oct 12:24:47 ntpd[27982]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-30T12:24:47.304 INFO:teuthology.orchestra.run.smithi089.stdout:30 Oct 12:24:47 ntpd[27982]: Command line: ntpd -gq 2023-10-30T12:24:47.305 INFO:teuthology.orchestra.run.smithi089.stdout:30 Oct 12:24:47 ntpd[27982]: ---------------------------------------------------- 2023-10-30T12:24:47.305 INFO:teuthology.orchestra.run.smithi089.stdout:30 Oct 12:24:47 ntpd[27982]: ntp-4 is maintained by Network Time Foundation, 2023-10-30T12:24:47.305 INFO:teuthology.orchestra.run.smithi089.stdout:30 Oct 12:24:47 ntpd[27982]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-30T12:24:47.305 INFO:teuthology.orchestra.run.smithi089.stdout:30 Oct 12:24:47 ntpd[27982]: corporation. Support and training for ntp-4 are 2023-10-30T12:24:47.305 INFO:teuthology.orchestra.run.smithi089.stdout:30 Oct 12:24:47 ntpd[27982]: available at https://www.nwtime.org/support 2023-10-30T12:24:47.306 INFO:teuthology.orchestra.run.smithi089.stdout:30 Oct 12:24:47 ntpd[27982]: ---------------------------------------------------- 2023-10-30T12:24:47.307 INFO:teuthology.orchestra.run.smithi089.stdout:30 Oct 12:24:47 ntpd[27982]: proto: precision = 0.041 usec (-24) 2023-10-30T12:24:47.307 INFO:teuthology.orchestra.run.smithi089.stdout:30 Oct 12:24:47 ntpd[27982]: basedate set to 2022-02-04 2023-10-30T12:24:47.307 INFO:teuthology.orchestra.run.smithi089.stdout:30 Oct 12:24:47 ntpd[27982]: gps base set to 2022-02-06 (week 2196) 2023-10-30T12:24:47.308 INFO:teuthology.orchestra.run.smithi089.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-30T12:24:47.308 INFO:teuthology.orchestra.run.smithi089.stdout:30 Oct 12:24:47 ntpd[27982]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-30T12:24:47.309 INFO:teuthology.orchestra.run.smithi089.stdout:30 Oct 12:24:47 ntpd[27982]: restrict ::: KOD does nothing without LIMITED. 2023-10-30T12:24:47.309 INFO:teuthology.orchestra.run.smithi089.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-30T12:24:47.310 INFO:teuthology.orchestra.run.smithi089.stdout:30 Oct 12:24:47 ntpd[27982]: Listen and drop on 0 v6wildcard [::]:123 2023-10-30T12:24:47.310 INFO:teuthology.orchestra.run.smithi089.stdout:30 Oct 12:24:47 ntpd[27982]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-30T12:24:47.311 INFO:teuthology.orchestra.run.smithi089.stdout:30 Oct 12:24:47 ntpd[27982]: Listen normally on 2 lo 127.0.0.1:123 2023-10-30T12:24:47.311 INFO:teuthology.orchestra.run.smithi089.stdout:30 Oct 12:24:47 ntpd[27982]: Listen normally on 3 enp3s0f1 172.21.15.89:123 2023-10-30T12:24:47.311 INFO:teuthology.orchestra.run.smithi089.stdout:30 Oct 12:24:47 ntpd[27982]: Listen normally on 4 lo [::1]:123 2023-10-30T12:24:47.312 INFO:teuthology.orchestra.run.smithi089.stdout:30 Oct 12:24:47 ntpd[27982]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d501%5]:123 2023-10-30T12:24:47.312 INFO:teuthology.orchestra.run.smithi089.stdout:30 Oct 12:24:47 ntpd[27982]: Listening on routing socket on fd #22 for interface updates 2023-10-30T12:24:47.313 INFO:teuthology.orchestra.run.smithi192.stdout:30 Oct 12:24:47 ntpd[28427]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-30T12:24:47.313 INFO:teuthology.orchestra.run.smithi192.stdout:30 Oct 12:24:47 ntpd[28427]: Command line: ntpd -gq 2023-10-30T12:24:47.313 INFO:teuthology.orchestra.run.smithi192.stdout:30 Oct 12:24:47 ntpd[28427]: ---------------------------------------------------- 2023-10-30T12:24:47.313 INFO:teuthology.orchestra.run.smithi192.stdout:30 Oct 12:24:47 ntpd[28427]: ntp-4 is maintained by Network Time Foundation, 2023-10-30T12:24:47.314 INFO:teuthology.orchestra.run.smithi192.stdout:30 Oct 12:24:47 ntpd[28427]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-30T12:24:47.314 INFO:teuthology.orchestra.run.smithi192.stdout:30 Oct 12:24:47 ntpd[28427]: corporation. Support and training for ntp-4 are 2023-10-30T12:24:47.314 INFO:teuthology.orchestra.run.smithi192.stdout:30 Oct 12:24:47 ntpd[28427]: available at https://www.nwtime.org/support 2023-10-30T12:24:47.314 INFO:teuthology.orchestra.run.smithi192.stdout:30 Oct 12:24:47 ntpd[28427]: ---------------------------------------------------- 2023-10-30T12:24:47.314 INFO:teuthology.orchestra.run.smithi192.stdout:30 Oct 12:24:47 ntpd[28427]: proto: precision = 0.045 usec (-24) 2023-10-30T12:24:47.315 INFO:teuthology.orchestra.run.smithi192.stdout:30 Oct 12:24:47 ntpd[28427]: basedate set to 2022-02-04 2023-10-30T12:24:47.315 INFO:teuthology.orchestra.run.smithi192.stdout:30 Oct 12:24:47 ntpd[28427]: gps base set to 2022-02-06 (week 2196) 2023-10-30T12:24:47.315 INFO:teuthology.orchestra.run.smithi192.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-30T12:24:47.316 INFO:teuthology.orchestra.run.smithi192.stdout:30 Oct 12:24:47 ntpd[28427]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-30T12:24:47.316 INFO:teuthology.orchestra.run.smithi192.stdout:30 Oct 12:24:47 ntpd[28427]: restrict ::: KOD does nothing without LIMITED. 2023-10-30T12:24:47.316 INFO:teuthology.orchestra.run.smithi192.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-30T12:24:47.318 INFO:teuthology.orchestra.run.smithi192.stdout:30 Oct 12:24:47 ntpd[28427]: Listen and drop on 0 v6wildcard [::]:123 2023-10-30T12:24:47.318 INFO:teuthology.orchestra.run.smithi192.stdout:30 Oct 12:24:47 ntpd[28427]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-30T12:24:47.318 INFO:teuthology.orchestra.run.smithi192.stdout:30 Oct 12:24:47 ntpd[28427]: Listen normally on 2 lo 127.0.0.1:123 2023-10-30T12:24:47.318 INFO:teuthology.orchestra.run.smithi192.stdout:30 Oct 12:24:47 ntpd[28427]: Listen normally on 3 enp3s0f1 172.21.15.192:123 2023-10-30T12:24:47.319 INFO:teuthology.orchestra.run.smithi192.stdout:30 Oct 12:24:47 ntpd[28427]: Listen normally on 4 lo [::1]:123 2023-10-30T12:24:47.319 INFO:teuthology.orchestra.run.smithi192.stdout:30 Oct 12:24:47 ntpd[28427]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:8049%5]:123 2023-10-30T12:24:47.319 INFO:teuthology.orchestra.run.smithi192.stdout:30 Oct 12:24:47 ntpd[28427]: Listening on routing socket on fd #22 for interface updates 2023-10-30T12:24:48.312 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:48 ntpd[28427]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:48.312 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:48 ntpd[28427]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:48.313 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:48 ntpd[28427]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:48.313 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:48 ntpd[28427]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:48.313 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:48 ntpd[28427]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:48.313 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:48 ntpd[28427]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:48.314 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:48 ntpd[28427]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:48.314 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:48 ntpd[28427]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:48.314 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:48 ntpd[28427]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:24:48.314 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:48 ntpd[28427]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:24:48.314 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:48 ntpd[28427]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:48.315 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:48 ntpd[28427]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:48.315 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:48 ntpd[28427]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:24:48.315 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:48 ntpd[28427]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:24:50.305 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:50 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:50.305 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:50 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:50.306 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:50 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:50.306 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:50 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:50.306 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:50 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:50.306 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:50 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:50.306 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:50 ntpd[27982]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:24:50.307 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:50 ntpd[27982]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:24:50.307 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:50 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:50.307 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:50 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:50.307 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:50 ntpd[27982]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:24:50.307 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:50 ntpd[27982]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:24:50.312 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:50 ntpd[28427]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:50.312 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:50 ntpd[28427]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:50.313 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:50 ntpd[28427]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:50.313 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:50 ntpd[28427]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:50.313 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:50 ntpd[28427]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:50.313 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:50 ntpd[28427]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:50.313 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:50 ntpd[28427]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:50.314 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:50 ntpd[28427]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:50.314 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:50 ntpd[28427]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:24:50.314 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:50 ntpd[28427]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:24:50.314 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:50 ntpd[28427]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:50.314 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:50 ntpd[28427]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:50.315 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:50 ntpd[28427]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:24:50.315 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:50 ntpd[28427]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:24:50.315 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:50 ntpd[28427]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:50.315 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:50 ntpd[28427]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:50.315 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:50 ntpd[28427]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:24:50.316 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:50 ntpd[28427]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:24:51.304 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:51 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:51.304 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:51 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:52.305 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:52 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:52.306 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:52 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:52.306 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:52 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:52.306 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:52 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:52.307 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:52 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:52.307 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:52 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:52.307 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:52 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:52.307 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:52 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:52.307 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:52 ntpd[27982]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:24:52.308 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:52 ntpd[27982]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:24:52.308 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:52 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:52.308 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:52 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:52.308 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:52 ntpd[27982]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:24:52.308 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:52 ntpd[27982]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:24:52.309 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:52 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:52.309 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:52 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:52.309 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:52 ntpd[27982]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:24:52.309 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:52 ntpd[27982]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:24:52.312 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:52 ntpd[28427]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:52.312 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:52 ntpd[28427]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:52.313 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:52 ntpd[28427]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:52.313 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:52 ntpd[28427]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:52.313 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:52 ntpd[28427]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:52.313 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:52 ntpd[28427]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:52.313 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:52 ntpd[28427]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:52.314 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:52 ntpd[28427]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:52.314 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:52 ntpd[28427]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:24:52.314 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:52 ntpd[28427]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:24:52.314 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:52 ntpd[28427]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:52.315 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:52 ntpd[28427]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:52.315 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:52 ntpd[28427]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:24:52.315 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:52 ntpd[28427]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:24:52.315 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:52 ntpd[28427]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:52.315 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:52 ntpd[28427]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:52.315 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:52 ntpd[28427]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:24:52.316 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:52 ntpd[28427]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:24:54.305 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:54 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:54.306 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:54 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:54.309 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:54 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:54.309 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:54 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:54.309 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:54 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:54.310 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:54 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:54.310 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:54 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:54.310 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:54 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:54.310 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:54 ntpd[27982]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:24:54.311 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:54 ntpd[27982]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:24:54.311 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:54 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:54.311 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:54 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:54.311 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:54 ntpd[27982]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:24:54.311 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:54 ntpd[27982]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:24:54.312 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:54 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:54.312 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:54 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:54.312 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:54 ntpd[27982]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:24:54.312 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:54 ntpd[27982]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:24:54.314 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:54 ntpd[28427]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:54.314 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:54 ntpd[28427]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:54.314 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:54 ntpd[28427]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:54.314 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:54 ntpd[28427]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:54.315 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:54 ntpd[28427]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:54.315 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:54 ntpd[28427]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:54.315 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:54 ntpd[28427]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:54.315 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:54 ntpd[28427]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:54.316 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:54 ntpd[28427]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:24:54.316 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:54 ntpd[28427]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:24:54.317 INFO:teuthology.orchestra.run.smithi192.stdout:30 Oct 12:24:54 ntpd[28427]: ntpd: time slew -0.005913 s 2023-10-30T12:24:54.317 INFO:teuthology.orchestra.run.smithi192.stdout:ntpd: time slew -0.005913s 2023-10-30T12:24:54.318 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:54 ntpd[28427]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-30T12:24:54.318 INFO:teuthology.orchestra.run.smithi192.stderr:30 Oct 12:24:54 ntpd[28427]: can't open /var/log/ntpstats/loopstats.20231030: Permission denied 2023-10-30T12:24:54.423 INFO:teuthology.orchestra.run.smithi192.stdout: remote refid st t when poll reach delay offset jitter 2023-10-30T12:24:54.424 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================== 2023-10-30T12:24:54.424 INFO:teuthology.orchestra.run.smithi192.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T12:24:54.424 INFO:teuthology.orchestra.run.smithi192.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T12:24:54.425 INFO:teuthology.orchestra.run.smithi192.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T12:24:54.426 INFO:teuthology.orchestra.run.smithi192.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T12:24:56.305 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:56 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:56.305 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:56 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:56.305 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:56 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:56.306 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:56 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:56.306 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:56 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:56.306 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:56 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:56.306 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:56 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:56.307 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:56 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:56.307 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:56 ntpd[27982]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:24:56.307 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:56 ntpd[27982]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:24:56.307 INFO:teuthology.orchestra.run.smithi089.stdout:30 Oct 12:24:56 ntpd[27982]: ntpd: time slew -0.000583 s 2023-10-30T12:24:56.308 INFO:teuthology.orchestra.run.smithi089.stdout:ntpd: time slew -0.000583s 2023-10-30T12:24:56.308 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:56 ntpd[27982]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-30T12:24:56.308 INFO:teuthology.orchestra.run.smithi089.stderr:30 Oct 12:24:56 ntpd[27982]: can't open /var/log/ntpstats/loopstats.20231030: Permission denied 2023-10-30T12:24:56.407 INFO:teuthology.orchestra.run.smithi089.stdout: remote refid st t when poll reach delay offset jitter 2023-10-30T12:24:56.408 INFO:teuthology.orchestra.run.smithi089.stdout:============================================================================== 2023-10-30T12:24:56.408 INFO:teuthology.orchestra.run.smithi089.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T12:24:56.408 INFO:teuthology.orchestra.run.smithi089.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T12:24:56.409 INFO:teuthology.orchestra.run.smithi089.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T12:24:56.409 INFO:teuthology.orchestra.run.smithi089.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T12:24:56.409 INFO:teuthology.run_tasks:Running task install... 2023-10-30T12:24:56.423 DEBUG:teuthology.task.install:project ceph 2023-10-30T12:24:56.424 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'}} 2023-10-30T12:24:56.424 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'} 2023-10-30T12:24:56.425 INFO:teuthology.task.install:Using flavor: default 2023-10-30T12:24:56.431 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-30T12:24:56.431 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-10-30T12:24:56.431 DEBUG:teuthology.orchestra.run.smithi089:> sudo apt-key list | grep Ceph 2023-10-30T12:24:56.432 DEBUG:teuthology.orchestra.run.smithi192:> sudo apt-key list | grep Ceph 2023-10-30T12:24:56.495 INFO:teuthology.orchestra.run.smithi192.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-30T12:24:56.525 INFO:teuthology.orchestra.run.smithi192.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-30T12:24:56.526 INFO:teuthology.orchestra.run.smithi192.stdout:uid [ unknown] Ceph.com (release key) 2023-10-30T12:24:56.526 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-10-30T12:24:56.526 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2023-10-30T12:24:56.527 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T12:24:56.559 INFO:teuthology.orchestra.run.smithi089.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-30T12:24:56.612 INFO:teuthology.orchestra.run.smithi089.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-30T12:24:56.612 INFO:teuthology.orchestra.run.smithi089.stdout:uid [ unknown] Ceph.com (release key) 2023-10-30T12:24:56.614 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-10-30T12:24:56.614 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2023-10-30T12:24:56.615 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T12:24:56.748 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default/ 2023-10-30T12:24:56.749 INFO:teuthology.task.install.deb:Package version is 18.0.0-6835-g00c1c13c-1jammy 2023-10-30T12:24:56.837 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default/ 2023-10-30T12:24:56.837 INFO:teuthology.task.install.deb:Package version is 18.0.0-6835-g00c1c13c-1jammy 2023-10-30T12:24:56.930 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-30T12:24:56.931 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-30T12:24:56.945 DEBUG:teuthology.orchestra.run.smithi192:> sudo apt-get update 2023-10-30T12:24:56.952 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-10-30T12:24:56.952 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-30T12:24:56.974 DEBUG:teuthology.orchestra.run.smithi089:> sudo apt-get update 2023-10-30T12:24:57.161 INFO:teuthology.orchestra.run.smithi089.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-30T12:24:57.161 INFO:teuthology.orchestra.run.smithi089.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-30T12:24:57.170 INFO:teuthology.orchestra.run.smithi089.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-30T12:24:57.191 INFO:teuthology.orchestra.run.smithi089.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-30T12:24:57.208 INFO:teuthology.orchestra.run.smithi089.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy InRelease 2023-10-30T12:24:57.232 INFO:teuthology.orchestra.run.smithi089.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-10-30T12:24:57.240 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-30T12:24:57.240 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-30T12:24:57.240 INFO:teuthology.orchestra.run.smithi192.stdout:Ign:3 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy InRelease 2023-10-30T12:24:57.254 INFO:teuthology.orchestra.run.smithi089.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-30T12:24:57.277 INFO:teuthology.orchestra.run.smithi089.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,407 B] 2023-10-30T12:24:57.277 INFO:teuthology.orchestra.run.smithi192.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-10-30T12:24:57.312 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-30T12:24:57.314 INFO:teuthology.orchestra.run.smithi192.stdout:Ign:6 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-30T12:24:57.324 INFO:teuthology.orchestra.run.smithi089.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-10-30T12:24:57.351 INFO:teuthology.orchestra.run.smithi192.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,407 B] 2023-10-30T12:24:57.393 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-30T12:24:57.394 INFO:teuthology.orchestra.run.smithi192.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-10-30T12:24:57.615 INFO:teuthology.orchestra.run.smithi192.stdout:Fetched 28.5 kB in 0s (58.0 kB/s) 2023-10-30T12:24:57.764 INFO:teuthology.orchestra.run.smithi089.stdout:Fetched 28.5 kB in 1s (50.0 kB/s) 2023-10-30T12:24:58.476 INFO:teuthology.orchestra.run.smithi192.stdout:Reading package lists... 2023-10-30T12:24:58.490 DEBUG:teuthology.orchestra.run.smithi192:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6835-g00c1c13c-1jammy cephadm=18.0.0-6835-g00c1c13c-1jammy ceph-mds=18.0.0-6835-g00c1c13c-1jammy ceph-mgr=18.0.0-6835-g00c1c13c-1jammy ceph-common=18.0.0-6835-g00c1c13c-1jammy ceph-fuse=18.0.0-6835-g00c1c13c-1jammy ceph-test=18.0.0-6835-g00c1c13c-1jammy ceph-volume=18.0.0-6835-g00c1c13c-1jammy radosgw=18.0.0-6835-g00c1c13c-1jammy python3-rados=18.0.0-6835-g00c1c13c-1jammy python3-rgw=18.0.0-6835-g00c1c13c-1jammy python3-cephfs=18.0.0-6835-g00c1c13c-1jammy python3-rbd=18.0.0-6835-g00c1c13c-1jammy libcephfs2=18.0.0-6835-g00c1c13c-1jammy libcephfs-dev=18.0.0-6835-g00c1c13c-1jammy librados2=18.0.0-6835-g00c1c13c-1jammy librbd1=18.0.0-6835-g00c1c13c-1jammy rbd-fuse=18.0.0-6835-g00c1c13c-1jammy python3-cephfs=18.0.0-6835-g00c1c13c-1jammy cephfs-shell=18.0.0-6835-g00c1c13c-1jammy cephfs-top=18.0.0-6835-g00c1c13c-1jammy cephfs-mirror=18.0.0-6835-g00c1c13c-1jammy 2023-10-30T12:24:58.541 INFO:teuthology.orchestra.run.smithi192.stdout:Reading package lists... 2023-10-30T12:24:58.672 INFO:teuthology.orchestra.run.smithi089.stdout:Reading package lists... 2023-10-30T12:24:58.686 DEBUG:teuthology.orchestra.run.smithi089:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6835-g00c1c13c-1jammy cephadm=18.0.0-6835-g00c1c13c-1jammy ceph-mds=18.0.0-6835-g00c1c13c-1jammy ceph-mgr=18.0.0-6835-g00c1c13c-1jammy ceph-common=18.0.0-6835-g00c1c13c-1jammy ceph-fuse=18.0.0-6835-g00c1c13c-1jammy ceph-test=18.0.0-6835-g00c1c13c-1jammy ceph-volume=18.0.0-6835-g00c1c13c-1jammy radosgw=18.0.0-6835-g00c1c13c-1jammy python3-rados=18.0.0-6835-g00c1c13c-1jammy python3-rgw=18.0.0-6835-g00c1c13c-1jammy python3-cephfs=18.0.0-6835-g00c1c13c-1jammy python3-rbd=18.0.0-6835-g00c1c13c-1jammy libcephfs2=18.0.0-6835-g00c1c13c-1jammy libcephfs-dev=18.0.0-6835-g00c1c13c-1jammy librados2=18.0.0-6835-g00c1c13c-1jammy librbd1=18.0.0-6835-g00c1c13c-1jammy rbd-fuse=18.0.0-6835-g00c1c13c-1jammy python3-cephfs=18.0.0-6835-g00c1c13c-1jammy cephfs-shell=18.0.0-6835-g00c1c13c-1jammy cephfs-top=18.0.0-6835-g00c1c13c-1jammy cephfs-mirror=18.0.0-6835-g00c1c13c-1jammy 2023-10-30T12:24:58.706 INFO:teuthology.orchestra.run.smithi192.stdout:Building dependency tree... 2023-10-30T12:24:58.706 INFO:teuthology.orchestra.run.smithi192.stdout:Reading state information... 2023-10-30T12:24:58.761 INFO:teuthology.orchestra.run.smithi089.stdout:Reading package lists... 2023-10-30T12:24:58.855 INFO:teuthology.orchestra.run.smithi192.stdout:The following packages were automatically installed and are no longer required: 2023-10-30T12:24:58.855 INFO:teuthology.orchestra.run.smithi192.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-30T12:24:58.856 INFO:teuthology.orchestra.run.smithi192.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-30T12:24:58.857 INFO:teuthology.orchestra.run.smithi192.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-30T12:24:58.857 INFO:teuthology.orchestra.run.smithi192.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-10-30T12:24:58.857 INFO:teuthology.orchestra.run.smithi192.stdout: python2.7-minimal 2023-10-30T12:24:58.857 INFO:teuthology.orchestra.run.smithi192.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-30T12:24:58.858 INFO:teuthology.orchestra.run.smithi192.stdout:The following additional packages will be installed: 2023-10-30T12:24:58.858 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-30T12:24:58.858 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-30T12:24:58.858 INFO:teuthology.orchestra.run.smithi192.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-30T12:24:58.858 INFO:teuthology.orchestra.run.smithi192.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-30T12:24:58.859 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-30T12:24:58.860 INFO:teuthology.orchestra.run.smithi192.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-30T12:24:58.860 INFO:teuthology.orchestra.run.smithi192.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-30T12:24:58.860 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-30T12:24:58.860 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2023-10-30T12:24:58.860 INFO:teuthology.orchestra.run.smithi192.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-30T12:24:58.861 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-30T12:24:58.861 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-30T12:24:58.861 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-30T12:24:58.861 INFO:teuthology.orchestra.run.smithi192.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-30T12:24:58.861 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-30T12:24:58.862 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-30T12:24:58.862 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2023-10-30T12:24:58.862 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2023-10-30T12:24:58.862 INFO:teuthology.orchestra.run.smithi192.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-10-30T12:24:58.863 INFO:teuthology.orchestra.run.smithi192.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2023-10-30T12:24:58.863 INFO:teuthology.orchestra.run.smithi192.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2023-10-30T12:24:58.863 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2023-10-30T12:24:58.863 INFO:teuthology.orchestra.run.smithi192.stdout: qttranslations5-l10n socat xmlstarlet 2023-10-30T12:24:58.863 INFO:teuthology.orchestra.run.smithi192.stdout:Suggested packages: 2023-10-30T12:24:58.864 INFO:teuthology.orchestra.run.smithi192.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-30T12:24:58.864 INFO:teuthology.orchestra.run.smithi192.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-30T12:24:58.864 INFO:teuthology.orchestra.run.smithi192.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-30T12:24:58.864 INFO:teuthology.orchestra.run.smithi192.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-30T12:24:58.864 INFO:teuthology.orchestra.run.smithi192.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-30T12:24:58.864 INFO:teuthology.orchestra.run.smithi192.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-30T12:24:58.865 INFO:teuthology.orchestra.run.smithi192.stdout: python3-watchdog 2023-10-30T12:24:58.865 INFO:teuthology.orchestra.run.smithi192.stdout:Recommended packages: 2023-10-30T12:24:58.865 INFO:teuthology.orchestra.run.smithi192.stdout: btrfs-tools 2023-10-30T12:24:58.926 INFO:teuthology.orchestra.run.smithi089.stdout:Building dependency tree... 2023-10-30T12:24:58.926 INFO:teuthology.orchestra.run.smithi089.stdout:Reading state information... 2023-10-30T12:24:58.958 INFO:teuthology.orchestra.run.smithi192.stdout:The following NEW packages will be installed: 2023-10-30T12:24:58.958 INFO:teuthology.orchestra.run.smithi192.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-30T12:24:58.958 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-30T12:24:58.959 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-10-30T12:24:58.959 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2023-10-30T12:24:58.959 INFO:teuthology.orchestra.run.smithi192.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-30T12:24:58.959 INFO:teuthology.orchestra.run.smithi192.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2023-10-30T12:24:58.959 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-30T12:24:58.961 INFO:teuthology.orchestra.run.smithi192.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-30T12:24:58.961 INFO:teuthology.orchestra.run.smithi192.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-30T12:24:58.961 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-30T12:24:58.961 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2023-10-30T12:24:58.961 INFO:teuthology.orchestra.run.smithi192.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2023-10-30T12:24:58.962 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-30T12:24:58.962 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-30T12:24:58.962 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-30T12:24:58.962 INFO:teuthology.orchestra.run.smithi192.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-30T12:24:58.962 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-30T12:24:58.963 INFO:teuthology.orchestra.run.smithi192.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-30T12:24:58.963 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2023-10-30T12:24:58.963 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-30T12:24:58.963 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-30T12:24:58.963 INFO:teuthology.orchestra.run.smithi192.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-30T12:24:58.964 INFO:teuthology.orchestra.run.smithi192.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-30T12:24:58.964 INFO:teuthology.orchestra.run.smithi192.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-30T12:24:58.964 INFO:teuthology.orchestra.run.smithi192.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-30T12:24:58.964 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-30T12:24:58.964 INFO:teuthology.orchestra.run.smithi192.stdout: socat xmlstarlet 2023-10-30T12:24:59.026 INFO:teuthology.orchestra.run.smithi192.stdout:0 upgraded, 115 newly installed, 0 to remove and 260 not upgraded. 2023-10-30T12:24:59.027 INFO:teuthology.orchestra.run.smithi192.stdout:Need to get 184 MB of archives. 2023-10-30T12:24:59.027 INFO:teuthology.orchestra.run.smithi192.stdout:After this operation, 826 MB of additional disk space will be used. 2023-10-30T12:24:59.027 INFO:teuthology.orchestra.run.smithi192.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2023-10-30T12:24:59.048 INFO:teuthology.orchestra.run.smithi192.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-30T12:24:59.060 INFO:teuthology.orchestra.run.smithi192.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2023-10-30T12:24:59.077 INFO:teuthology.orchestra.run.smithi089.stdout:The following packages were automatically installed and are no longer required: 2023-10-30T12:24:59.077 INFO:teuthology.orchestra.run.smithi089.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-30T12:24:59.078 INFO:teuthology.orchestra.run.smithi089.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-30T12:24:59.078 INFO:teuthology.orchestra.run.smithi089.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-30T12:24:59.078 INFO:teuthology.orchestra.run.smithi089.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-30T12:24:59.078 INFO:teuthology.orchestra.run.smithi089.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-30T12:24:59.078 INFO:teuthology.orchestra.run.smithi089.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-30T12:24:59.080 INFO:teuthology.orchestra.run.smithi089.stdout:The following additional packages will be installed: 2023-10-30T12:24:59.080 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-30T12:24:59.080 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-30T12:24:59.080 INFO:teuthology.orchestra.run.smithi089.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-30T12:24:59.081 INFO:teuthology.orchestra.run.smithi089.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-30T12:24:59.081 INFO:teuthology.orchestra.run.smithi089.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-30T12:24:59.081 INFO:teuthology.orchestra.run.smithi089.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-30T12:24:59.081 INFO:teuthology.orchestra.run.smithi089.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-30T12:24:59.081 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-30T12:24:59.082 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2023-10-30T12:24:59.082 INFO:teuthology.orchestra.run.smithi089.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-30T12:24:59.082 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-30T12:24:59.082 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-30T12:24:59.082 INFO:teuthology.orchestra.run.smithi089.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-30T12:24:59.083 INFO:teuthology.orchestra.run.smithi089.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-30T12:24:59.083 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-30T12:24:59.083 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-30T12:24:59.083 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2023-10-30T12:24:59.084 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2023-10-30T12:24:59.084 INFO:teuthology.orchestra.run.smithi089.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-10-30T12:24:59.084 INFO:teuthology.orchestra.run.smithi089.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2023-10-30T12:24:59.084 INFO:teuthology.orchestra.run.smithi089.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2023-10-30T12:24:59.085 INFO:teuthology.orchestra.run.smithi089.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2023-10-30T12:24:59.085 INFO:teuthology.orchestra.run.smithi089.stdout: qttranslations5-l10n socat xmlstarlet 2023-10-30T12:24:59.085 INFO:teuthology.orchestra.run.smithi089.stdout:Suggested packages: 2023-10-30T12:24:59.085 INFO:teuthology.orchestra.run.smithi089.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-30T12:24:59.085 INFO:teuthology.orchestra.run.smithi089.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-30T12:24:59.086 INFO:teuthology.orchestra.run.smithi089.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-30T12:24:59.086 INFO:teuthology.orchestra.run.smithi089.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-30T12:24:59.086 INFO:teuthology.orchestra.run.smithi089.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-30T12:24:59.086 INFO:teuthology.orchestra.run.smithi089.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-30T12:24:59.086 INFO:teuthology.orchestra.run.smithi089.stdout: python3-watchdog 2023-10-30T12:24:59.087 INFO:teuthology.orchestra.run.smithi089.stdout:Recommended packages: 2023-10-30T12:24:59.087 INFO:teuthology.orchestra.run.smithi089.stdout: btrfs-tools 2023-10-30T12:24:59.114 INFO:teuthology.orchestra.run.smithi192.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-30T12:24:59.116 INFO:teuthology.orchestra.run.smithi192.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6835-g00c1c13c-1jammy [3,450 kB] 2023-10-30T12:24:59.122 INFO:teuthology.orchestra.run.smithi192.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-30T12:24:59.142 INFO:teuthology.orchestra.run.smithi192.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-30T12:24:59.147 INFO:teuthology.orchestra.run.smithi192.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-30T12:24:59.159 INFO:teuthology.orchestra.run.smithi192.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-30T12:24:59.216 INFO:teuthology.orchestra.run.smithi192.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-30T12:24:59.219 INFO:teuthology.orchestra.run.smithi192.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-30T12:24:59.222 INFO:teuthology.orchestra.run.smithi089.stdout:The following NEW packages will be installed: 2023-10-30T12:24:59.222 INFO:teuthology.orchestra.run.smithi089.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-30T12:24:59.222 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-30T12:24:59.222 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-10-30T12:24:59.223 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2023-10-30T12:24:59.224 INFO:teuthology.orchestra.run.smithi089.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-30T12:24:59.224 INFO:teuthology.orchestra.run.smithi089.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2023-10-30T12:24:59.224 INFO:teuthology.orchestra.run.smithi089.stdout: libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-30T12:24:59.224 INFO:teuthology.orchestra.run.smithi089.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-30T12:24:59.225 INFO:teuthology.orchestra.run.smithi089.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-30T12:24:59.225 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-30T12:24:59.225 INFO:teuthology.orchestra.run.smithi089.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2023-10-30T12:24:59.225 INFO:teuthology.orchestra.run.smithi089.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2023-10-30T12:24:59.225 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-30T12:24:59.226 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-30T12:24:59.226 INFO:teuthology.orchestra.run.smithi089.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-30T12:24:59.226 INFO:teuthology.orchestra.run.smithi089.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-30T12:24:59.226 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-30T12:24:59.226 INFO:teuthology.orchestra.run.smithi089.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-30T12:24:59.227 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2023-10-30T12:24:59.227 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-30T12:24:59.227 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-30T12:24:59.229 INFO:teuthology.orchestra.run.smithi089.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-30T12:24:59.230 INFO:teuthology.orchestra.run.smithi089.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-30T12:24:59.230 INFO:teuthology.orchestra.run.smithi089.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-30T12:24:59.230 INFO:teuthology.orchestra.run.smithi089.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-30T12:24:59.230 INFO:teuthology.orchestra.run.smithi089.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-30T12:24:59.231 INFO:teuthology.orchestra.run.smithi089.stdout: socat xmlstarlet 2023-10-30T12:24:59.231 INFO:teuthology.orchestra.run.smithi192.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-30T12:24:59.232 INFO:teuthology.orchestra.run.smithi192.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-30T12:24:59.233 INFO:teuthology.orchestra.run.smithi192.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-30T12:24:59.242 INFO:teuthology.orchestra.run.smithi192.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-30T12:24:59.243 INFO:teuthology.orchestra.run.smithi192.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-30T12:24:59.244 INFO:teuthology.orchestra.run.smithi192.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-30T12:24:59.245 INFO:teuthology.orchestra.run.smithi192.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-30T12:24:59.251 INFO:teuthology.orchestra.run.smithi192.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-30T12:24:59.268 INFO:teuthology.orchestra.run.smithi192.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-30T12:24:59.269 INFO:teuthology.orchestra.run.smithi192.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-30T12:24:59.269 INFO:teuthology.orchestra.run.smithi192.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-30T12:24:59.270 INFO:teuthology.orchestra.run.smithi192.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-30T12:24:59.270 INFO:teuthology.orchestra.run.smithi192.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-30T12:24:59.273 INFO:teuthology.orchestra.run.smithi192.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-30T12:24:59.273 INFO:teuthology.orchestra.run.smithi192.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-30T12:24:59.274 INFO:teuthology.orchestra.run.smithi192.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-30T12:24:59.274 INFO:teuthology.orchestra.run.smithi192.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-30T12:24:59.295 INFO:teuthology.orchestra.run.smithi192.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-30T12:24:59.295 INFO:teuthology.orchestra.run.smithi192.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-30T12:24:59.296 INFO:teuthology.orchestra.run.smithi192.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-30T12:24:59.296 INFO:teuthology.orchestra.run.smithi089.stdout:0 upgraded, 115 newly installed, 0 to remove and 260 not upgraded. 2023-10-30T12:24:59.296 INFO:teuthology.orchestra.run.smithi089.stdout:Need to get 184 MB of archives. 2023-10-30T12:24:59.296 INFO:teuthology.orchestra.run.smithi089.stdout:After this operation, 826 MB of additional disk space will be used. 2023-10-30T12:24:59.297 INFO:teuthology.orchestra.run.smithi089.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2023-10-30T12:24:59.297 INFO:teuthology.orchestra.run.smithi192.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-30T12:24:59.298 INFO:teuthology.orchestra.run.smithi192.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-30T12:24:59.298 INFO:teuthology.orchestra.run.smithi192.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-30T12:24:59.314 INFO:teuthology.orchestra.run.smithi089.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-30T12:24:59.316 INFO:teuthology.orchestra.run.smithi192.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-30T12:24:59.317 INFO:teuthology.orchestra.run.smithi192.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-30T12:24:59.318 INFO:teuthology.orchestra.run.smithi192.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-30T12:24:59.318 INFO:teuthology.orchestra.run.smithi192.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-30T12:24:59.324 INFO:teuthology.orchestra.run.smithi089.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2023-10-30T12:24:59.336 INFO:teuthology.orchestra.run.smithi192.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-30T12:24:59.338 INFO:teuthology.orchestra.run.smithi192.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-30T12:24:59.339 INFO:teuthology.orchestra.run.smithi192.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-30T12:24:59.340 INFO:teuthology.orchestra.run.smithi192.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-30T12:24:59.341 INFO:teuthology.orchestra.run.smithi192.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-30T12:24:59.344 INFO:teuthology.orchestra.run.smithi192.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-30T12:24:59.357 INFO:teuthology.orchestra.run.smithi192.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-30T12:24:59.359 INFO:teuthology.orchestra.run.smithi192.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-30T12:24:59.361 INFO:teuthology.orchestra.run.smithi192.stdout:Get:47 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6835-g00c1c13c-1jammy [3,131 kB] 2023-10-30T12:24:59.366 INFO:teuthology.orchestra.run.smithi089.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-30T12:24:59.374 INFO:teuthology.orchestra.run.smithi089.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-30T12:24:59.391 INFO:teuthology.orchestra.run.smithi089.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-30T12:24:59.393 INFO:teuthology.orchestra.run.smithi089.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-30T12:24:59.403 INFO:teuthology.orchestra.run.smithi089.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-30T12:24:59.427 INFO:teuthology.orchestra.run.smithi192.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-30T12:24:59.428 INFO:teuthology.orchestra.run.smithi192.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-30T12:24:59.430 INFO:teuthology.orchestra.run.smithi192.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-30T12:24:59.431 INFO:teuthology.orchestra.run.smithi192.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-30T12:24:59.434 INFO:teuthology.orchestra.run.smithi192.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-30T12:24:59.434 INFO:teuthology.orchestra.run.smithi192.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-30T12:24:59.439 INFO:teuthology.orchestra.run.smithi089.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-30T12:24:59.441 INFO:teuthology.orchestra.run.smithi089.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-30T12:24:59.449 INFO:teuthology.orchestra.run.smithi089.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-30T12:24:59.451 INFO:teuthology.orchestra.run.smithi089.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-30T12:24:59.452 INFO:teuthology.orchestra.run.smithi089.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-30T12:24:59.456 INFO:teuthology.orchestra.run.smithi192.stdout:Get:54 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6835-g00c1c13c-1jammy [689 kB] 2023-10-30T12:24:59.458 INFO:teuthology.orchestra.run.smithi089.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-30T12:24:59.458 INFO:teuthology.orchestra.run.smithi089.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-30T12:24:59.459 INFO:teuthology.orchestra.run.smithi089.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-30T12:24:59.460 INFO:teuthology.orchestra.run.smithi089.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-30T12:24:59.460 INFO:teuthology.orchestra.run.smithi089.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-30T12:24:59.462 INFO:teuthology.orchestra.run.smithi192.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-30T12:24:59.466 INFO:teuthology.orchestra.run.smithi192.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-30T12:24:59.466 INFO:teuthology.orchestra.run.smithi192.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-30T12:24:59.475 INFO:teuthology.orchestra.run.smithi089.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-30T12:24:59.476 INFO:teuthology.orchestra.run.smithi089.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-30T12:24:59.476 INFO:teuthology.orchestra.run.smithi089.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-30T12:24:59.476 INFO:teuthology.orchestra.run.smithi089.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-30T12:24:59.478 INFO:teuthology.orchestra.run.smithi192.stdout:Get:58 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6835-g00c1c13c-1jammy [356 kB] 2023-10-30T12:24:59.489 INFO:teuthology.orchestra.run.smithi192.stdout:Get:59 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6835-g00c1c13c-1jammy [32.8 kB] 2023-10-30T12:24:59.490 INFO:teuthology.orchestra.run.smithi192.stdout:Get:60 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6835-g00c1c13c-1jammy [180 kB] 2023-10-30T12:24:59.492 INFO:teuthology.orchestra.run.smithi089.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-30T12:24:59.494 INFO:teuthology.orchestra.run.smithi089.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-30T12:24:59.494 INFO:teuthology.orchestra.run.smithi089.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-30T12:24:59.495 INFO:teuthology.orchestra.run.smithi089.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-30T12:24:59.495 INFO:teuthology.orchestra.run.smithi089.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-30T12:24:59.496 INFO:teuthology.orchestra.run.smithi192.stdout:Get:61 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6835-g00c1c13c-1jammy [62.4 kB] 2023-10-30T12:24:59.496 INFO:teuthology.orchestra.run.smithi089.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-30T12:24:59.498 INFO:teuthology.orchestra.run.smithi192.stdout:Get:62 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6835-g00c1c13c-1jammy [330 kB] 2023-10-30T12:24:59.500 INFO:teuthology.orchestra.run.smithi192.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-30T12:24:59.500 INFO:teuthology.orchestra.run.smithi192.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-30T12:24:59.501 INFO:teuthology.orchestra.run.smithi192.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-30T12:24:59.501 INFO:teuthology.orchestra.run.smithi192.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-30T12:24:59.502 INFO:teuthology.orchestra.run.smithi192.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-30T12:24:59.503 INFO:teuthology.orchestra.run.smithi192.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-30T12:24:59.509 INFO:teuthology.orchestra.run.smithi089.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-30T12:24:59.509 INFO:teuthology.orchestra.run.smithi089.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-30T12:24:59.510 INFO:teuthology.orchestra.run.smithi089.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-30T12:24:59.511 INFO:teuthology.orchestra.run.smithi089.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-30T12:24:59.521 INFO:teuthology.orchestra.run.smithi192.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-30T12:24:59.521 INFO:teuthology.orchestra.run.smithi192.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-30T12:24:59.526 INFO:teuthology.orchestra.run.smithi192.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-30T12:24:59.526 INFO:teuthology.orchestra.run.smithi192.stdout:Get:72 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6835-g00c1c13c-1jammy [6,226 kB] 2023-10-30T12:24:59.526 INFO:teuthology.orchestra.run.smithi089.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-30T12:24:59.530 INFO:teuthology.orchestra.run.smithi089.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-30T12:24:59.530 INFO:teuthology.orchestra.run.smithi089.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-30T12:24:59.531 INFO:teuthology.orchestra.run.smithi089.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-30T12:24:59.531 INFO:teuthology.orchestra.run.smithi089.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-30T12:24:59.533 INFO:teuthology.orchestra.run.smithi089.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-30T12:24:59.542 INFO:teuthology.orchestra.run.smithi089.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-30T12:24:59.543 INFO:teuthology.orchestra.run.smithi089.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-30T12:24:59.544 INFO:teuthology.orchestra.run.smithi089.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-30T12:24:59.545 INFO:teuthology.orchestra.run.smithi089.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-30T12:24:59.553 INFO:teuthology.orchestra.run.smithi192.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-30T12:24:59.555 INFO:teuthology.orchestra.run.smithi192.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-30T12:24:59.556 INFO:teuthology.orchestra.run.smithi192.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-30T12:24:59.559 INFO:teuthology.orchestra.run.smithi089.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-30T12:24:59.560 INFO:teuthology.orchestra.run.smithi192.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-30T12:24:59.560 INFO:teuthology.orchestra.run.smithi089.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-30T12:24:59.562 INFO:teuthology.orchestra.run.smithi089.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-30T12:24:59.563 INFO:teuthology.orchestra.run.smithi192.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2023-10-30T12:24:59.564 INFO:teuthology.orchestra.run.smithi192.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-30T12:24:59.569 INFO:teuthology.orchestra.run.smithi192.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-30T12:24:59.599 INFO:teuthology.orchestra.run.smithi192.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-30T12:24:59.599 INFO:teuthology.orchestra.run.smithi192.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-30T12:24:59.600 INFO:teuthology.orchestra.run.smithi192.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-30T12:24:59.606 INFO:teuthology.orchestra.run.smithi089.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-30T12:24:59.607 INFO:teuthology.orchestra.run.smithi089.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-30T12:24:59.608 INFO:teuthology.orchestra.run.smithi089.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-30T12:24:59.609 INFO:teuthology.orchestra.run.smithi089.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-30T12:24:59.609 INFO:teuthology.orchestra.run.smithi089.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-30T12:24:59.610 INFO:teuthology.orchestra.run.smithi089.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-30T12:24:59.629 INFO:teuthology.orchestra.run.smithi089.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-30T12:24:59.632 INFO:teuthology.orchestra.run.smithi089.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-30T12:24:59.632 INFO:teuthology.orchestra.run.smithi089.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-30T12:24:59.639 INFO:teuthology.orchestra.run.smithi192.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-30T12:24:59.642 INFO:teuthology.orchestra.run.smithi192.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-30T12:24:59.643 INFO:teuthology.orchestra.run.smithi192.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-30T12:24:59.650 INFO:teuthology.orchestra.run.smithi089.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-30T12:24:59.650 INFO:teuthology.orchestra.run.smithi089.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-30T12:24:59.651 INFO:teuthology.orchestra.run.smithi089.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-30T12:24:59.651 INFO:teuthology.orchestra.run.smithi089.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-30T12:24:59.651 INFO:teuthology.orchestra.run.smithi089.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-30T12:24:59.652 INFO:teuthology.orchestra.run.smithi089.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-30T12:24:59.655 INFO:teuthology.orchestra.run.smithi192.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-30T12:24:59.656 INFO:teuthology.orchestra.run.smithi192.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-30T12:24:59.656 INFO:teuthology.orchestra.run.smithi192.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-30T12:24:59.659 INFO:teuthology.orchestra.run.smithi089.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-30T12:24:59.659 INFO:teuthology.orchestra.run.smithi089.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-30T12:24:59.660 INFO:teuthology.orchestra.run.smithi192.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-30T12:24:59.663 INFO:teuthology.orchestra.run.smithi089.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-30T12:24:59.676 INFO:teuthology.orchestra.run.smithi089.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-30T12:24:59.677 INFO:teuthology.orchestra.run.smithi089.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-30T12:24:59.678 INFO:teuthology.orchestra.run.smithi089.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-30T12:24:59.681 INFO:teuthology.orchestra.run.smithi089.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-30T12:24:59.684 INFO:teuthology.orchestra.run.smithi089.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2023-10-30T12:24:59.684 INFO:teuthology.orchestra.run.smithi089.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-30T12:24:59.689 INFO:teuthology.orchestra.run.smithi089.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-30T12:24:59.693 INFO:teuthology.orchestra.run.smithi089.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-30T12:24:59.693 INFO:teuthology.orchestra.run.smithi089.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-30T12:24:59.694 INFO:teuthology.orchestra.run.smithi192.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-30T12:24:59.694 INFO:teuthology.orchestra.run.smithi089.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-30T12:24:59.710 INFO:teuthology.orchestra.run.smithi089.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-30T12:24:59.711 INFO:teuthology.orchestra.run.smithi089.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-30T12:24:59.712 INFO:teuthology.orchestra.run.smithi089.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-30T12:24:59.719 INFO:teuthology.orchestra.run.smithi089.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-30T12:24:59.720 INFO:teuthology.orchestra.run.smithi089.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-30T12:24:59.720 INFO:teuthology.orchestra.run.smithi089.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-30T12:24:59.722 INFO:teuthology.orchestra.run.smithi089.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-30T12:24:59.727 INFO:teuthology.orchestra.run.smithi089.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-30T12:24:59.736 INFO:teuthology.orchestra.run.smithi192.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6835-g00c1c13c-1jammy [112 kB] 2023-10-30T12:24:59.746 INFO:teuthology.orchestra.run.smithi192.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6835-g00c1c13c-1jammy [439 kB] 2023-10-30T12:24:59.787 INFO:teuthology.orchestra.run.smithi192.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6835-g00c1c13c-1jammy [22.6 MB] 2023-10-30T12:25:00.143 INFO:teuthology.orchestra.run.smithi089.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6835-g00c1c13c-1jammy [3,450 kB] 2023-10-30T12:25:00.639 INFO:teuthology.orchestra.run.smithi192.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6835-g00c1c13c-1jammy [6,858 kB] 2023-10-30T12:25:00.778 INFO:teuthology.orchestra.run.smithi192.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6835-g00c1c13c-1jammy [238 kB] 2023-10-30T12:25:00.784 INFO:teuthology.orchestra.run.smithi192.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6835-g00c1c13c-1jammy [128 kB] 2023-10-30T12:25:00.788 INFO:teuthology.orchestra.run.smithi192.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6835-g00c1c13c-1jammy [1,496 kB] 2023-10-30T12:25:00.841 INFO:teuthology.orchestra.run.smithi192.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6835-g00c1c13c-1jammy [9,892 kB] 2023-10-30T12:25:01.212 INFO:teuthology.orchestra.run.smithi192.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6835-g00c1c13c-1jammy [31.7 MB] 2023-10-30T12:25:01.836 INFO:teuthology.orchestra.run.smithi089.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6835-g00c1c13c-1jammy [3,131 kB] 2023-10-30T12:25:02.494 INFO:teuthology.orchestra.run.smithi192.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6835-g00c1c13c-1jammy [14.1 kB] 2023-10-30T12:25:02.495 INFO:teuthology.orchestra.run.smithi192.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6835-g00c1c13c-1jammy [839 kB] 2023-10-30T12:25:02.511 INFO:teuthology.orchestra.run.smithi192.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6835-g00c1c13c-1jammy [2,148 kB] 2023-10-30T12:25:02.549 INFO:teuthology.orchestra.run.smithi192.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6835-g00c1c13c-1jammy [284 kB] 2023-10-30T12:25:02.553 INFO:teuthology.orchestra.run.smithi192.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6835-g00c1c13c-1jammy [138 kB] 2023-10-30T12:25:02.555 INFO:teuthology.orchestra.run.smithi192.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6835-g00c1c13c-1jammy [1,907 kB] 2023-10-30T12:25:02.579 INFO:teuthology.orchestra.run.smithi192.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6835-g00c1c13c-1jammy [8,625 kB] 2023-10-30T12:25:02.926 INFO:teuthology.orchestra.run.smithi192.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6835-g00c1c13c-1jammy [14.2 kB] 2023-10-30T12:25:02.932 INFO:teuthology.orchestra.run.smithi192.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6835-g00c1c13c-1jammy [47.8 MB] 2023-10-30T12:25:04.296 INFO:teuthology.orchestra.run.smithi089.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6835-g00c1c13c-1jammy [689 kB] 2023-10-30T12:25:04.474 INFO:teuthology.orchestra.run.smithi089.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6835-g00c1c13c-1jammy [356 kB] 2023-10-30T12:25:04.597 INFO:teuthology.orchestra.run.smithi089.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6835-g00c1c13c-1jammy [32.8 kB] 2023-10-30T12:25:04.597 INFO:teuthology.orchestra.run.smithi089.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6835-g00c1c13c-1jammy [180 kB] 2023-10-30T12:25:04.599 INFO:teuthology.orchestra.run.smithi089.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6835-g00c1c13c-1jammy [62.4 kB] 2023-10-30T12:25:04.647 INFO:teuthology.orchestra.run.smithi089.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6835-g00c1c13c-1jammy [330 kB] 2023-10-30T12:25:04.707 INFO:teuthology.orchestra.run.smithi089.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6835-g00c1c13c-1jammy [6,226 kB] 2023-10-30T12:25:04.994 INFO:teuthology.orchestra.run.smithi192.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6835-g00c1c13c-1jammy [120 kB] 2023-10-30T12:25:04.996 INFO:teuthology.orchestra.run.smithi192.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 18.0.0-6835-g00c1c13c-1jammy [227 kB] 2023-10-30T12:25:05.001 INFO:teuthology.orchestra.run.smithi192.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 18.0.0-6835-g00c1c13c-1jammy [34.0 kB] 2023-10-30T12:25:05.002 INFO:teuthology.orchestra.run.smithi192.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 18.0.0-6835-g00c1c13c-1jammy [23.4 kB] 2023-10-30T12:25:05.003 INFO:teuthology.orchestra.run.smithi192.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6835-g00c1c13c-1jammy [38.3 kB] 2023-10-30T12:25:05.004 INFO:teuthology.orchestra.run.smithi192.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6835-g00c1c13c-1jammy [12.3 MB] 2023-10-30T12:25:05.355 INFO:teuthology.orchestra.run.smithi192.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6835-g00c1c13c-1jammy [91.6 kB] 2023-10-30T12:25:05.531 INFO:teuthology.orchestra.run.smithi089.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6835-g00c1c13c-1jammy [112 kB] 2023-10-30T12:25:05.532 INFO:teuthology.orchestra.run.smithi089.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6835-g00c1c13c-1jammy [439 kB] 2023-10-30T12:25:05.570 INFO:teuthology.orchestra.run.smithi089.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6835-g00c1c13c-1jammy [22.6 MB] 2023-10-30T12:25:05.898 INFO:teuthology.orchestra.run.smithi192.stdout:Fetched 184 MB in 6s (28.9 MB/s) 2023-10-30T12:25:06.064 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pyperclip. 2023-10-30T12:25:06.149 INFO:teuthology.orchestra.run.smithi192.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 130506 files and directories currently installed.) 2023-10-30T12:25:06.153 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2023-10-30T12:25:06.195 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2023-10-30T12:25:06.423 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-30T12:25:06.435 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-30T12:25:06.465 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-30T12:25:06.809 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-cmd2. 2023-10-30T12:25:06.830 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2023-10-30T12:25:06.868 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2023-10-30T12:25:07.136 INFO:teuthology.orchestra.run.smithi089.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6835-g00c1c13c-1jammy [6,858 kB] 2023-10-30T12:25:07.212 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-30T12:25:07.232 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../003-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-30T12:25:07.312 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-30T12:25:07.682 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-30T12:25:07.702 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../004-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-30T12:25:07.740 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-30T12:25:07.794 INFO:teuthology.orchestra.run.smithi089.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6835-g00c1c13c-1jammy [238 kB] 2023-10-30T12:25:07.829 INFO:teuthology.orchestra.run.smithi089.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6835-g00c1c13c-1jammy [128 kB] 2023-10-30T12:25:07.846 INFO:teuthology.orchestra.run.smithi089.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6835-g00c1c13c-1jammy [1,496 kB] 2023-10-30T12:25:08.042 INFO:teuthology.orchestra.run.smithi089.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6835-g00c1c13c-1jammy [9,892 kB] 2023-10-30T12:25:08.069 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-30T12:25:08.089 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../005-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-30T12:25:08.118 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-30T12:25:08.488 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-30T12:25:08.509 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../006-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-30T12:25:08.546 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-30T12:25:09.017 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-30T12:25:09.037 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../007-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T12:25:09.082 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T12:25:09.249 INFO:teuthology.orchestra.run.smithi089.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6835-g00c1c13c-1jammy [31.7 MB] 2023-10-30T12:25:09.470 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-30T12:25:09.490 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../008-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T12:25:09.519 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T12:25:10.065 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-30T12:25:10.085 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../009-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T12:25:10.123 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T12:25:10.501 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-30T12:25:10.521 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../010-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-30T12:25:10.559 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-30T12:25:10.870 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package librados2. 2023-10-30T12:25:10.891 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../011-librados2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:10.920 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking librados2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:11.390 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package librbd1. 2023-10-30T12:25:11.410 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../012-librbd1_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:11.448 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking librbd1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:11.901 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libcephfs2. 2023-10-30T12:25:11.922 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../013-libcephfs2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:11.959 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libcephfs2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:12.245 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-rados. 2023-10-30T12:25:12.257 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../014-python3-rados_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:12.319 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-rados (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:12.606 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-30T12:25:12.626 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../015-python3-ceph-argparse_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:25:12.655 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-ceph-argparse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:12.941 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-cephfs. 2023-10-30T12:25:12.953 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../016-python3-cephfs_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:12.983 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-cephfs (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:13.327 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-30T12:25:13.348 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../017-python3-ceph-common_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:25:13.385 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:13.646 INFO:teuthology.orchestra.run.smithi089.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6835-g00c1c13c-1jammy [14.1 kB] 2023-10-30T12:25:13.646 INFO:teuthology.orchestra.run.smithi089.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6835-g00c1c13c-1jammy [839 kB] 2023-10-30T12:25:13.747 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-prettytable. 2023-10-30T12:25:13.759 INFO:teuthology.orchestra.run.smithi089.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6835-g00c1c13c-1jammy [2,148 kB] 2023-10-30T12:25:13.767 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../018-python3-prettytable_2.5.0-2_all.deb ... 2023-10-30T12:25:13.805 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-30T12:25:14.056 INFO:teuthology.orchestra.run.smithi089.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6835-g00c1c13c-1jammy [284 kB] 2023-10-30T12:25:14.083 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-rbd. 2023-10-30T12:25:14.096 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../019-python3-rbd_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:14.097 INFO:teuthology.orchestra.run.smithi089.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6835-g00c1c13c-1jammy [138 kB] 2023-10-30T12:25:14.115 INFO:teuthology.orchestra.run.smithi089.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6835-g00c1c13c-1jammy [1,907 kB] 2023-10-30T12:25:14.133 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-rbd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:14.381 INFO:teuthology.orchestra.run.smithi089.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6835-g00c1c13c-1jammy [8,625 kB] 2023-10-30T12:25:14.519 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-30T12:25:14.540 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../020-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-30T12:25:14.577 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-30T12:25:15.315 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package librgw2. 2023-10-30T12:25:15.335 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../021-librgw2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:15.364 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking librgw2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:15.556 INFO:teuthology.orchestra.run.smithi089.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6835-g00c1c13c-1jammy [14.2 kB] 2023-10-30T12:25:15.556 INFO:teuthology.orchestra.run.smithi089.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6835-g00c1c13c-1jammy [47.8 MB] 2023-10-30T12:25:15.900 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-rgw. 2023-10-30T12:25:15.921 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../022-python3-rgw_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:15.958 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-rgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:16.336 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-30T12:25:16.357 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../023-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-30T12:25:16.394 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-30T12:25:16.714 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libradosstriper1. 2023-10-30T12:25:16.734 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../024-libradosstriper1_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:16.764 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libradosstriper1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:17.167 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-common. 2023-10-30T12:25:17.188 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../025-ceph-common_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:17.225 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:18.548 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-base. 2023-10-30T12:25:18.568 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../026-ceph-base_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:18.640 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-base (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:19.377 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-30T12:25:19.398 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../027-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-30T12:25:19.435 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-30T12:25:19.805 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-cheroot. 2023-10-30T12:25:19.825 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../028-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-30T12:25:19.871 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-30T12:25:20.283 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-30T12:25:20.303 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../029-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-30T12:25:20.349 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-30T12:25:20.426 INFO:teuthology.orchestra.run.smithi089.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6835-g00c1c13c-1jammy [120 kB] 2023-10-30T12:25:20.427 INFO:teuthology.orchestra.run.smithi089.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 18.0.0-6835-g00c1c13c-1jammy [227 kB] 2023-10-30T12:25:20.443 INFO:teuthology.orchestra.run.smithi089.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 18.0.0-6835-g00c1c13c-1jammy [34.0 kB] 2023-10-30T12:25:20.445 INFO:teuthology.orchestra.run.smithi089.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 18.0.0-6835-g00c1c13c-1jammy [23.4 kB] 2023-10-30T12:25:20.446 INFO:teuthology.orchestra.run.smithi089.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6835-g00c1c13c-1jammy [38.3 kB] 2023-10-30T12:25:20.448 INFO:teuthology.orchestra.run.smithi089.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6835-g00c1c13c-1jammy [12.3 MB] 2023-10-30T12:25:20.744 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-30T12:25:20.764 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../030-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-30T12:25:20.802 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-30T12:25:21.163 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-30T12:25:21.184 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../031-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-30T12:25:21.238 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-30T12:25:21.541 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-tempora. 2023-10-30T12:25:21.551 INFO:teuthology.orchestra.run.smithi089.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6835-g00c1c13c-1jammy [91.6 kB] 2023-10-30T12:25:21.561 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../032-python3-tempora_4.1.2-1_all.deb ... 2023-10-30T12:25:21.599 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-30T12:25:21.902 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-portend. 2023-10-30T12:25:21.922 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../033-python3-portend_3.0.0-1_all.deb ... 2023-10-30T12:25:21.980 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-30T12:25:22.136 INFO:teuthology.orchestra.run.smithi089.stdout:Fetched 184 MB in 22s (8,256 kB/s) 2023-10-30T12:25:22.288 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-30T12:25:22.308 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../034-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-30T12:25:22.337 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-pyperclip. 2023-10-30T12:25:22.346 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-30T12:25:22.405 INFO:teuthology.orchestra.run.smithi089.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 132932 files and directories currently installed.) 2023-10-30T12:25:22.409 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2023-10-30T12:25:22.463 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2023-10-30T12:25:22.657 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-30T12:25:22.669 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../035-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-30T12:25:22.707 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-30T12:25:22.799 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-30T12:25:22.813 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-30T12:25:22.858 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-30T12:25:23.078 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-natsort. 2023-10-30T12:25:23.098 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../036-python3-natsort_8.0.2-1_all.deb ... 2023-10-30T12:25:23.136 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-30T12:25:23.210 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-cmd2. 2023-10-30T12:25:23.224 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2023-10-30T12:25:23.277 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2023-10-30T12:25:23.481 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-logutils. 2023-10-30T12:25:23.501 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../037-python3-logutils_0.3.3-8_all.deb ... 2023-10-30T12:25:23.539 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-30T12:25:23.672 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-30T12:25:23.686 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../003-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-30T12:25:23.789 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-30T12:25:23.800 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-30T12:25:23.820 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../038-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-30T12:25:23.850 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-30T12:25:24.191 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-30T12:25:24.194 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-mako. 2023-10-30T12:25:24.206 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../004-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-30T12:25:24.215 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../039-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-30T12:25:24.253 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-30T12:25:24.258 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-30T12:25:24.606 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-30T12:25:24.626 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../040-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-30T12:25:24.664 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-30T12:25:24.687 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-30T12:25:24.702 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../005-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-30T12:25:24.745 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-30T12:25:24.992 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-30T12:25:25.012 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../041-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-30T12:25:25.050 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-30T12:25:25.131 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-30T12:25:25.146 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../006-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-30T12:25:25.206 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-30T12:25:25.569 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-webob. 2023-10-30T12:25:25.590 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../042-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-30T12:25:25.619 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-30T12:25:25.660 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-30T12:25:25.675 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../007-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T12:25:25.727 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T12:25:26.006 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-waitress. 2023-10-30T12:25:26.026 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../043-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-30T12:25:26.067 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-30T12:25:26.130 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-30T12:25:26.145 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../008-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T12:25:26.188 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T12:25:26.384 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-tempita. 2023-10-30T12:25:26.404 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../044-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-30T12:25:26.467 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-30T12:25:26.582 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-30T12:25:26.598 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../009-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T12:25:26.641 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T12:25:26.770 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-paste. 2023-10-30T12:25:26.790 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../045-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-30T12:25:26.828 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-30T12:25:26.936 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-30T12:25:26.950 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../010-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-30T12:25:26.994 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-30T12:25:27.198 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-30T12:25:27.238 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../046-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-30T12:25:27.256 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-30T12:25:27.331 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package librados2. 2023-10-30T12:25:27.346 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../011-librados2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:27.389 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking librados2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:27.576 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-30T12:25:27.596 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../047-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-30T12:25:27.634 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-30T12:25:27.833 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package librbd1. 2023-10-30T12:25:27.848 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../012-librbd1_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:27.892 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking librbd1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:27.962 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-webtest. 2023-10-30T12:25:27.982 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../048-python3-webtest_2.0.35-1_all.deb ... 2023-10-30T12:25:28.020 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-30T12:25:28.253 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libcephfs2. 2023-10-30T12:25:28.267 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../013-libcephfs2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:28.302 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libcephfs2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:28.306 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pecan. 2023-10-30T12:25:28.327 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../049-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-30T12:25:28.356 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-30T12:25:28.555 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-rados. 2023-10-30T12:25:28.570 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../014-python3-rados_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:28.614 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-rados (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:28.719 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-certifi. 2023-10-30T12:25:28.740 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../050-python3-certifi_2020.6.20-1_all.deb ... 2023-10-30T12:25:28.777 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-30T12:25:28.966 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-30T12:25:28.981 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../015-python3-ceph-argparse_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:25:29.025 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-ceph-argparse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:29.139 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-idna. 2023-10-30T12:25:29.159 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../051-python3-idna_3.3-1_all.deb ... 2023-10-30T12:25:29.197 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-30T12:25:29.277 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-cephfs. 2023-10-30T12:25:29.292 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../016-python3-cephfs_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:29.335 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-cephfs (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:29.542 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-urllib3. 2023-10-30T12:25:29.562 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../052-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-30T12:25:29.600 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-30T12:25:29.613 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-30T12:25:29.628 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../017-python3-ceph-common_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:25:29.663 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:29.966 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-prettytable. 2023-10-30T12:25:29.969 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-requests. 2023-10-30T12:25:29.981 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../018-python3-prettytable_2.5.0-2_all.deb ... 2023-10-30T12:25:29.990 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../053-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-30T12:25:30.024 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-30T12:25:30.028 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-30T12:25:30.277 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-rbd. 2023-10-30T12:25:30.291 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../019-python3-rbd_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:30.335 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-rbd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:30.689 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-30T12:25:30.703 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../020-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-30T12:25:30.747 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-30T12:25:31.074 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package librgw2. 2023-10-30T12:25:31.089 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../021-librgw2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:31.124 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking librgw2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:31.215 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-30T12:25:31.236 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../054-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-30T12:25:31.265 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-30T12:25:31.577 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-rgw. 2023-10-30T12:25:31.591 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../022-python3-rgw_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:31.601 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-30T12:25:31.622 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../055-ceph-mgr-modules-core_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:25:31.635 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-rgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:31.659 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:31.971 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-30T12:25:31.986 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../023-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-30T12:25:32.030 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-30T12:25:32.081 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-30T12:25:32.102 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../056-libsqlite3-mod-ceph_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:32.139 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:32.492 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-openssl. 2023-10-30T12:25:32.513 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../057-python3-openssl_21.0.0-1_all.deb ... 2023-10-30T12:25:32.550 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-30T12:25:32.895 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mgr. 2023-10-30T12:25:32.915 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../058-ceph-mgr_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:32.945 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mgr (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:32.983 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libradosstriper1. 2023-10-30T12:25:32.998 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../024-libradosstriper1_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:33.058 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libradosstriper1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:33.340 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mon. 2023-10-30T12:25:33.361 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../059-ceph-mon_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:33.389 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mon (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:33.469 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ceph-common. 2023-10-30T12:25:33.484 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../025-ceph-common_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:33.527 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:34.377 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-osd. 2023-10-30T12:25:34.397 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../060-ceph-osd_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:34.434 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-osd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:34.609 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ceph-base. 2023-10-30T12:25:34.623 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../026-ceph-base_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:34.668 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ceph-base (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:35.438 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-30T12:25:35.452 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../027-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-30T12:25:35.497 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-30T12:25:36.057 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-cheroot. 2023-10-30T12:25:36.073 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../028-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-30T12:25:36.107 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-30T12:25:36.419 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-30T12:25:36.434 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../029-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-30T12:25:36.469 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-30T12:25:36.480 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph. 2023-10-30T12:25:36.501 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../061-ceph_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:36.538 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:36.772 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-30T12:25:36.786 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../030-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-30T12:25:36.830 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-30T12:25:36.841 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-fuse. 2023-10-30T12:25:36.862 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../062-ceph-fuse_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:36.891 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:37.133 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-30T12:25:37.148 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../031-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-30T12:25:37.191 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-30T12:25:37.294 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mds. 2023-10-30T12:25:37.314 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../063-ceph-mds_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:37.352 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mds (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:37.519 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-tempora. 2023-10-30T12:25:37.534 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../032-python3-tempora_4.1.2-1_all.deb ... 2023-10-30T12:25:37.577 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-30T12:25:37.780 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package cephadm. 2023-10-30T12:25:37.801 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../064-cephadm_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:37.838 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:37.855 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-portend. 2023-10-30T12:25:37.870 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../033-python3-portend_3.0.0-1_all.deb ... 2023-10-30T12:25:37.913 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-30T12:25:38.133 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-30T12:25:38.145 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../065-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-30T12:25:38.202 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-30T12:25:38.216 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-30T12:25:38.231 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../034-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-30T12:25:38.274 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-30T12:25:38.536 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-30T12:25:38.548 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../066-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-30T12:25:38.585 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-30T12:25:38.602 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-30T12:25:38.617 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../035-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-30T12:25:38.661 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-30T12:25:39.006 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-natsort. 2023-10-30T12:25:39.020 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../036-python3-natsort_8.0.2-1_all.deb ... 2023-10-30T12:25:39.064 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-30T12:25:39.376 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-logutils. 2023-10-30T12:25:39.390 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../037-python3-logutils_0.3.3-8_all.deb ... 2023-10-30T12:25:39.459 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-30T12:25:39.716 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-babel. 2023-10-30T12:25:39.737 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../067-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-30T12:25:39.745 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-30T12:25:39.759 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../038-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-30T12:25:39.774 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-30T12:25:39.803 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-30T12:25:40.081 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-mako. 2023-10-30T12:25:40.096 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../039-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-30T12:25:40.139 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-30T12:25:40.144 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-jinja2. 2023-10-30T12:25:40.165 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../068-python3-jinja2_3.0.3-1_all.deb ... 2023-10-30T12:25:40.210 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-30T12:25:40.459 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-30T12:25:40.474 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../040-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-30T12:25:40.534 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-30T12:25:40.563 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-30T12:25:40.584 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../069-ceph-mgr-cephadm_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:25:40.655 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:40.952 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-30T12:25:40.970 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-30T12:25:40.973 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../070-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-30T12:25:40.986 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../041-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-30T12:25:41.002 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-30T12:25:41.037 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-30T12:25:41.330 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-30T12:25:41.340 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-webob. 2023-10-30T12:25:41.351 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../071-python3-repoze.lru_0.7-2_all.deb ... 2023-10-30T12:25:41.355 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../042-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-30T12:25:41.388 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-30T12:25:41.398 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-30T12:25:41.725 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-routes. 2023-10-30T12:25:41.746 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../072-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-30T12:25:41.759 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-waitress. 2023-10-30T12:25:41.774 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../043-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-30T12:25:41.791 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-30T12:25:41.812 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-30T12:25:42.121 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-tempita. 2023-10-30T12:25:42.135 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../044-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-30T12:25:42.145 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-30T12:25:42.166 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../073-ceph-mgr-dashboard_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:25:42.187 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-30T12:25:42.203 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:42.474 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-paste. 2023-10-30T12:25:42.488 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../045-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-30T12:25:42.532 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-30T12:25:42.674 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-30T12:25:42.695 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-30T12:25:42.731 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-30T12:25:42.868 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-30T12:25:42.882 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../046-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-30T12:25:42.926 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-30T12:25:43.254 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-30T12:25:43.268 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../047-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-30T12:25:43.269 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-joblib. 2023-10-30T12:25:43.290 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../075-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-30T12:25:43.312 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-30T12:25:43.327 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-30T12:25:43.615 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-webtest. 2023-10-30T12:25:43.629 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../048-python3-webtest_2.0.35-1_all.deb ... 2023-10-30T12:25:43.698 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-30T12:25:43.781 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-30T12:25:43.802 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../076-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-30T12:25:43.847 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-30T12:25:43.976 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-pecan. 2023-10-30T12:25:43.990 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../049-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-30T12:25:44.026 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-30T12:25:44.234 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-sklearn. 2023-10-30T12:25:44.255 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../077-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-30T12:25:44.292 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-30T12:25:44.348 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-certifi. 2023-10-30T12:25:44.361 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../050-python3-certifi_2020.6.20-1_all.deb ... 2023-10-30T12:25:44.406 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-30T12:25:44.751 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-idna. 2023-10-30T12:25:44.764 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../051-python3-idna_3.3-1_all.deb ... 2023-10-30T12:25:44.809 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-30T12:25:45.093 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-30T12:25:45.117 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../078-ceph-mgr-diskprediction-local_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:25:45.120 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-urllib3. 2023-10-30T12:25:45.133 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../052-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-30T12:25:45.159 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:45.178 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-30T12:25:45.499 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-requests. 2023-10-30T12:25:45.512 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../053-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-30T12:25:45.557 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-30T12:25:45.893 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-30T12:25:45.906 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../054-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-30T12:25:45.943 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-30T12:25:46.163 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-cachetools. 2023-10-30T12:25:46.184 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../079-python3-cachetools_5.0.0-1_all.deb ... 2023-10-30T12:25:46.212 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-30T12:25:46.226 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../055-ceph-mgr-modules-core_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:25:46.229 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-30T12:25:46.279 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:46.533 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-30T12:25:46.558 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../080-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-30T12:25:46.651 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-30T12:25:46.664 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../056-libsqlite3-mod-ceph_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:46.709 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:46.807 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-30T12:25:47.037 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-openssl. 2023-10-30T12:25:47.050 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../057-python3-openssl_21.0.0-1_all.deb ... 2023-10-30T12:25:47.087 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-30T12:25:47.120 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-30T12:25:47.141 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../081-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-30T12:25:47.178 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-30T12:25:47.415 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ceph-mgr. 2023-10-30T12:25:47.428 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../058-ceph-mgr_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:47.473 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ceph-mgr (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:47.497 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-rsa. 2023-10-30T12:25:47.518 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../082-python3-rsa_4.8-1_all.deb ... 2023-10-30T12:25:47.555 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-30T12:25:47.851 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ceph-mon. 2023-10-30T12:25:47.865 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../059-ceph-mon_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:47.926 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ceph-mon (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:47.986 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-google-auth. 2023-10-30T12:25:48.008 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../083-python3-google-auth_1.5.1-3_all.deb ... 2023-10-30T12:25:48.052 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-30T12:25:48.406 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-30T12:25:48.427 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../084-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-30T12:25:48.464 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-30T12:25:48.682 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ceph-osd. 2023-10-30T12:25:48.697 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../060-ceph-osd_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:48.740 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ceph-osd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:48.801 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-websocket. 2023-10-30T12:25:48.822 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../085-python3-websocket_1.2.3-1_all.deb ... 2023-10-30T12:25:48.859 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-30T12:25:49.188 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-30T12:25:49.208 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../086-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-30T12:25:49.279 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-30T12:25:49.977 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-30T12:25:49.989 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../087-ceph-mgr-k8sevents_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:25:50.026 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:50.085 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ceph. 2023-10-30T12:25:50.100 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../061-ceph_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:50.143 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:50.355 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-30T12:25:50.376 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../088-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-30T12:25:50.421 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-30T12:25:50.462 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ceph-fuse. 2023-10-30T12:25:50.477 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../062-ceph-fuse_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:50.521 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ceph-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:50.782 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-30T12:25:50.804 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../089-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-30T12:25:50.840 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-30T12:25:50.849 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ceph-mds. 2023-10-30T12:25:50.864 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../063-ceph-mds_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:50.907 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ceph-mds (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:51.093 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package jq. 2023-10-30T12:25:51.115 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../090-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-30T12:25:51.177 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-30T12:25:51.294 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package cephadm. 2023-10-30T12:25:51.308 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../064-cephadm_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:51.352 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:51.404 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package socat. 2023-10-30T12:25:51.417 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../091-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-30T12:25:51.446 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-30T12:25:51.805 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-30T12:25:51.820 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../065-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-30T12:25:51.863 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-30T12:25:51.925 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package xmlstarlet. 2023-10-30T12:25:51.937 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../092-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-30T12:25:51.974 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-30T12:25:52.200 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-30T12:25:52.215 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../066-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-30T12:25:52.258 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-30T12:25:52.379 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-test. 2023-10-30T12:25:52.401 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../093-ceph-test_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:52.445 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-test (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:53.271 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-babel. 2023-10-30T12:25:53.285 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../067-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-30T12:25:53.338 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-30T12:25:53.658 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-jinja2. 2023-10-30T12:25:53.671 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../068-python3-jinja2_3.0.3-1_all.deb ... 2023-10-30T12:25:53.716 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-30T12:25:54.061 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-30T12:25:54.074 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../069-ceph-mgr-cephadm_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:25:54.119 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:54.472 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-30T12:25:54.486 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../070-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-30T12:25:54.530 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-30T12:25:54.716 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-volume. 2023-10-30T12:25:54.738 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../094-ceph-volume_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:25:54.783 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-volume (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:54.800 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-30T12:25:54.814 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../071-python3-repoze.lru_0.7-2_all.deb ... 2023-10-30T12:25:54.858 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-30T12:25:55.096 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package cephfs-mirror. 2023-10-30T12:25:55.117 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../095-cephfs-mirror_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:55.161 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-routes. 2023-10-30T12:25:55.175 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../072-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-30T12:25:55.195 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking cephfs-mirror (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:55.219 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-30T12:25:55.524 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-colorama. 2023-10-30T12:25:55.545 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../096-python3-colorama_0.4.4-1_all.deb ... 2023-10-30T12:25:55.573 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-30T12:25:55.586 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../073-ceph-mgr-dashboard_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:25:55.590 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-colorama (0.4.4-1) ... 2023-10-30T12:25:55.640 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:55.885 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package cephfs-shell. 2023-10-30T12:25:55.907 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../097-cephfs-shell_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:25:55.952 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking cephfs-shell (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:56.043 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-30T12:25:56.057 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-30T12:25:56.102 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-30T12:25:56.229 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package cephfs-top. 2023-10-30T12:25:56.242 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../098-cephfs-top_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:25:56.271 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking cephfs-top (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:56.507 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-30T12:25:56.520 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../099-libcephfs-dev_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:56.557 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libcephfs-dev (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:56.597 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-joblib. 2023-10-30T12:25:56.610 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../075-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-30T12:25:56.647 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-30T12:25:56.978 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package nvme-cli. 2023-10-30T12:25:57.000 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../100-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-30T12:25:57.009 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-30T12:25:57.022 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../076-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-30T12:25:57.044 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-30T12:25:57.067 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-30T12:25:57.370 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-sklearn. 2023-10-30T12:25:57.383 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../077-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-30T12:25:57.416 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-30T12:25:57.428 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../101-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-30T12:25:57.453 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-30T12:25:57.465 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-30T12:25:57.919 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-30T12:25:57.941 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../102-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-30T12:25:57.986 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-30T12:25:58.153 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-30T12:25:58.167 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../078-ceph-mgr-diskprediction-local_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:25:58.212 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:58.297 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pastescript. 2023-10-30T12:25:58.318 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-4_all.deb ... 2023-10-30T12:25:58.363 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-30T12:25:58.717 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pluggy. 2023-10-30T12:25:58.739 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../104-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-30T12:25:58.783 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-30T12:25:59.137 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-psutil. 2023-10-30T12:25:59.159 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../105-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-30T12:25:59.203 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-30T12:25:59.256 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-cachetools. 2023-10-30T12:25:59.271 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../079-python3-cachetools_5.0.0-1_all.deb ... 2023-10-30T12:25:59.323 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-30T12:25:59.573 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-py. 2023-10-30T12:25:59.595 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../106-python3-py_1.10.0-1_all.deb ... 2023-10-30T12:25:59.639 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-30T12:25:59.693 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-30T12:25:59.708 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../080-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-30T12:25:59.751 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-30T12:25:59.968 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pygments. 2023-10-30T12:25:59.990 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../107-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-30T12:26:00.043 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-30T12:26:00.088 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-30T12:26:00.102 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../081-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-30T12:26:00.146 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-30T12:26:00.507 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-rsa. 2023-10-30T12:26:00.521 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../082-python3-rsa_4.8-1_all.deb ... 2023-10-30T12:26:00.564 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-30T12:26:00.573 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-30T12:26:00.585 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../108-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-30T12:26:00.630 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-30T12:26:00.925 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-toml. 2023-10-30T12:26:00.947 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../109-python3-toml_0.10.2-1_all.deb ... 2023-10-30T12:26:01.000 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-30T12:26:01.062 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-google-auth. 2023-10-30T12:26:01.077 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../083-python3-google-auth_1.5.1-3_all.deb ... 2023-10-30T12:26:01.120 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-30T12:26:01.311 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pytest. 2023-10-30T12:26:01.333 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../110-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-30T12:26:01.378 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-30T12:26:01.457 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-30T12:26:01.472 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../084-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-30T12:26:01.507 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-30T12:26:01.740 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-simplejson. 2023-10-30T12:26:01.761 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../111-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-30T12:26:01.806 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-30T12:26:01.818 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-websocket. 2023-10-30T12:26:01.834 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../085-python3-websocket_1.2.3-1_all.deb ... 2023-10-30T12:26:01.877 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-30T12:26:02.197 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-30T12:26:02.212 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../086-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-30T12:26:02.254 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-30T12:26:02.418 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-30T12:26:02.439 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../112-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-30T12:26:02.492 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-30T12:26:02.960 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-30T12:26:02.974 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../087-ceph-mgr-k8sevents_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:26:03.018 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:03.288 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package radosgw. 2023-10-30T12:26:03.310 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../113-radosgw_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:26:03.338 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-30T12:26:03.351 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../088-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-30T12:26:03.354 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking radosgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:03.396 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-30T12:26:03.732 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-30T12:26:03.746 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../089-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-30T12:26:03.791 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-30T12:26:04.052 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package jq. 2023-10-30T12:26:04.065 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../090-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-30T12:26:04.110 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-30T12:26:04.124 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package rbd-fuse. 2023-10-30T12:26:04.137 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../114-rbd-fuse_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:26:04.174 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking rbd-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:04.363 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package socat. 2023-10-30T12:26:04.376 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../091-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-30T12:26:04.421 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-30T12:26:04.510 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-30T12:26:04.733 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package xmlstarlet. 2023-10-30T12:26:04.746 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../092-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-30T12:26:04.791 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-30T12:26:04.796 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-30T12:26:04.947 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-30T12:26:05.162 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ceph-test. 2023-10-30T12:26:05.176 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../093-ceph-test_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:26:05.217 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-30T12:26:05.220 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ceph-test (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:05.559 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-30T12:26:06.003 INFO:teuthology.orchestra.run.smithi192.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-30T12:26:06.066 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:06.618 INFO:teuthology.orchestra.run.smithi192.stdout:Adding system user cephadm....done 2023-10-30T12:26:06.839 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-30T12:26:07.116 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-30T12:26:07.374 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ceph-volume. 2023-10-30T12:26:07.389 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../094-ceph-volume_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:26:07.441 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ceph-volume (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:07.517 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-30T12:26:07.643 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-30T12:26:07.845 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package cephfs-mirror. 2023-10-30T12:26:07.860 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../095-cephfs-mirror_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:26:07.903 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking cephfs-mirror (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:07.944 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-30T12:26:08.229 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-30T12:26:08.248 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-colorama. 2023-10-30T12:26:08.263 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../096-python3-colorama_0.4.4-1_all.deb ... 2023-10-30T12:26:08.306 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-colorama (0.4.4-1) ... 2023-10-30T12:26:08.355 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-30T12:26:08.534 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package cephfs-shell. 2023-10-30T12:26:08.549 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../097-cephfs-shell_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:26:08.584 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking cephfs-shell (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:08.690 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-30T12:26:08.845 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package cephfs-top. 2023-10-30T12:26:08.860 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../098-cephfs-top_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:26:08.903 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking cephfs-top (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:09.051 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-30T12:26:09.139 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-30T12:26:09.154 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../099-libcephfs-dev_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:26:09.197 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libcephfs-dev (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:09.335 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-colorama (0.4.4-1) ... 2023-10-30T12:26:09.570 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-30T12:26:09.609 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package nvme-cli. 2023-10-30T12:26:09.624 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../100-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-30T12:26:09.667 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-30T12:26:09.854 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-30T12:26:10.014 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-30T12:26:10.028 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../101-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-30T12:26:10.072 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-30T12:26:10.130 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-30T12:26:10.457 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-30T12:26:10.517 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-30T12:26:10.530 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../102-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-30T12:26:10.575 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-30T12:26:10.716 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-ceph-argparse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:10.878 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-pastescript. 2023-10-30T12:26:10.892 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-4_all.deb ... 2023-10-30T12:26:10.936 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-30T12:26:10.967 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-30T12:26:11.130 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2023-10-30T12:26:11.256 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-pluggy. 2023-10-30T12:26:11.269 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../104-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-30T12:26:11.314 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-30T12:26:11.378 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-30T12:26:11.629 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-30T12:26:11.642 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-psutil. 2023-10-30T12:26:11.656 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../105-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-30T12:26:11.700 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-30T12:26:11.755 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-30T12:26:12.028 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-py. 2023-10-30T12:26:12.042 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../106-python3-py_1.10.0-1_all.deb ... 2023-10-30T12:26:12.078 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-30T12:26:12.115 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-30T12:26:12.216 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-30T12:26:12.381 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-pygments. 2023-10-30T12:26:12.395 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../107-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-30T12:26:12.461 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-30T12:26:12.484 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-30T12:26:12.777 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-30T12:26:12.902 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-30T12:26:12.916 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../108-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-30T12:26:12.952 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-30T12:26:13.112 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-30T12:26:13.254 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-toml. 2023-10-30T12:26:13.268 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../109-python3-toml_0.10.2-1_all.deb ... 2023-10-30T12:26:13.313 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-30T12:26:13.649 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-pytest. 2023-10-30T12:26:13.663 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../110-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-30T12:26:13.707 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-30T12:26:13.789 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-30T12:26:14.027 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package python3-simplejson. 2023-10-30T12:26:14.041 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../111-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-30T12:26:14.064 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-30T12:26:14.085 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-30T12:26:14.191 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-30T12:26:14.326 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-30T12:26:14.355 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-30T12:26:14.369 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../112-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-30T12:26:14.413 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-30T12:26:14.635 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-30T12:26:15.020 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-30T12:26:15.092 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package radosgw. 2023-10-30T12:26:15.106 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../113-radosgw_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:26:15.150 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking radosgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:15.279 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-30T12:26:15.664 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-30T12:26:15.761 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package rbd-fuse. 2023-10-30T12:26:15.776 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../114-rbd-fuse_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:26:15.820 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking rbd-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:15.949 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2023-10-30T12:26:16.110 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-30T12:26:16.283 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-30T12:26:16.383 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-30T12:26:16.501 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-30T12:26:16.660 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-30T12:26:16.736 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-30T12:26:16.911 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-30T12:26:17.037 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:17.118 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-30T12:26:17.355 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-idna (3.3-1) ... 2023-10-30T12:26:17.723 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-30T12:26:17.777 INFO:teuthology.orchestra.run.smithi089.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-30T12:26:17.791 INFO:teuthology.orchestra.run.smithi089.stdout:Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. 2023-10-30T12:26:17.871 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:18.888 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T12:26:18.914 INFO:teuthology.orchestra.run.smithi089.stdout:Adding system user cephadm....done 2023-10-30T12:26:19.022 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-30T12:26:19.134 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-30T12:26:19.290 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-30T12:26:19.416 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-30T12:26:19.504 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-30T12:26:19.551 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-30T12:26:19.780 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-30T12:26:19.844 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-30T12:26:20.048 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-30T12:26:20.095 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-30T12:26:20.316 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-30T12:26:20.380 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T12:26:20.506 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-30T12:26:20.600 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-30T12:26:20.632 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-30T12:26:20.751 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-30T12:26:20.950 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-30T12:26:21.078 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-30T12:26:21.210 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-30T12:26:21.447 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-30T12:26:21.469 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-30T12:26:21.714 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-colorama (0.4.4-1) ... 2023-10-30T12:26:21.729 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-30T12:26:21.980 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-30T12:26:21.982 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-30T12:26:22.248 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-30T12:26:22.334 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-30T12:26:22.357 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-30T12:26:22.610 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-30T12:26:22.642 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-30T12:26:22.776 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-30T12:26:22.894 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-30T12:26:23.094 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-30T12:26:23.187 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-ceph-argparse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:23.404 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-30T12:26:23.471 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-30T12:26:23.597 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2023-10-30T12:26:23.665 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-30T12:26:23.865 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-30T12:26:24.057 INFO:teuthology.orchestra.run.smithi192.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-30T12:26:24.175 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-30T12:26:24.326 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-30T12:26:24.341 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-30T12:26:24.468 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-30T12:26:24.702 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-30T12:26:24.811 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-30T12:26:24.911 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-30T12:26:25.062 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-30T12:26:25.204 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-30T12:26:25.339 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-30T12:26:25.504 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T12:26:25.631 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-30T12:26:25.648 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-30T12:26:25.999 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-30T12:26:26.300 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-30T12:26:26.334 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-30T12:26:26.593 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-30T12:26:26.635 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-30T12:26:26.719 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-30T12:26:26.786 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-30T12:26:26.937 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-30T12:26:26.987 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-30T12:26:27.255 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-30T12:26:27.271 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-30T12:26:27.522 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-30T12:26:27.706 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-30T12:26:27.924 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-30T12:26:27.973 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-30T12:26:28.219 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-30T12:26:28.417 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-30T12:26:28.588 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-30T12:26:28.714 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up librados2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:28.735 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2023-10-30T12:26:28.815 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up librgw2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:29.051 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:29.061 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-30T12:26:29.177 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libcephfs2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:29.303 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libradosstriper1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:29.454 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-30T12:26:29.463 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-30T12:26:29.713 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-30T12:26:29.764 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up librbd1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:29.839 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:29.899 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mgr-modules-core (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:30.025 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:30.149 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-idna (3.3-1) ... 2023-10-30T12:26:30.235 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-30T12:26:30.235 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-30T12:26:30.517 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-30T12:26:30.818 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libcephfs-dev (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:30.919 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-rados (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:31.045 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up cephfs-top (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:31.171 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-30T12:26:31.773 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T12:26:31.932 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-30T12:26:32.234 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-30T12:26:32.384 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-30T12:26:32.408 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-rbd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:32.534 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up rbd-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:32.535 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-30T12:26:32.660 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-rgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:32.828 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-cephfs (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:32.878 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-30T12:26:32.954 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up cephfs-shell (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:33.080 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:33.146 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-30T12:26:33.406 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T12:26:33.549 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-30T12:26:33.671 INFO:teuthology.orchestra.run.smithi192.stdout:Adding group ceph....done 2023-10-30T12:26:33.727 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-30T12:26:34.053 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-30T12:26:34.066 INFO:teuthology.orchestra.run.smithi192.stdout:Adding system user ceph....done 2023-10-30T12:26:34.276 INFO:teuthology.orchestra.run.smithi192.stdout:Setting system user ceph properties....done 2023-10-30T12:26:34.292 INFO:teuthology.orchestra.run.smithi192.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-30T12:26:34.312 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-30T12:26:34.427 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-30T12:26:34.581 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-30T12:26:34.788 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-30T12:26:34.848 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-30T12:26:35.116 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-30T12:26:35.390 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up cephfs-mirror (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:35.434 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-30T12:26:35.560 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-30T12:26:35.601 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-30T12:26:35.601 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-30T12:26:35.845 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-30T12:26:35.995 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-30T12:26:36.158 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-test (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:36.284 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up radosgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:36.313 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-30T12:26:36.623 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-30T12:26:36.771 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-30T12:26:36.772 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-30T12:26:36.890 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-30T12:26:37.306 INFO:teuthology.orchestra.run.smithi089.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-30T12:26:37.311 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-base (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:37.663 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-30T12:26:37.727 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-30T12:26:37.903 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-30T12:26:38.321 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-30T12:26:38.456 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mds (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:38.597 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-30T12:26:38.747 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-30T12:26:38.748 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-30T12:26:38.865 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T12:26:39.016 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-30T12:26:39.274 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mgr (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:39.359 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-30T12:26:39.492 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-30T12:26:39.492 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-30T12:26:39.702 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-30T12:26:40.013 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-30T12:26:40.042 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-osd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:40.145 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-30T12:26:40.343 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-30T12:26:40.343 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-30T12:26:40.422 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-30T12:26:40.706 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-30T12:26:40.943 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:40.999 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-30T12:26:41.078 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:41.237 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mon (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:41.434 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-30T12:26:41.466 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-30T12:26:41.467 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-30T12:26:41.744 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-30T12:26:42.030 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mgr-cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:42.137 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-30T12:26:42.181 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:42.280 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up librados2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:42.307 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mgr-dashboard (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:42.414 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up librgw2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:42.483 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-volume (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:42.660 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:42.811 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libcephfs2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:42.811 INFO:teuthology.orchestra.run.smithi192.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-30T12:26:42.962 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libradosstriper1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:43.105 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-30T12:26:43.423 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up librbd1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:43.574 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ceph-mgr-modules-core (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:43.725 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ceph-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:43.950 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-30T12:26:43.950 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-30T12:26:44.433 INFO:teuthology.orchestra.run.smithi192.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-30T12:26:44.593 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libcephfs-dev (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:44.694 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-rados (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:44.820 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up cephfs-top (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:44.911 INFO:teuthology.orchestra.run.smithi192.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-30T12:26:44.914 DEBUG:teuthology.orchestra.run.smithi192:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2023-10-30T12:26:44.946 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-30T12:26:45.009 INFO:teuthology.orchestra.run.smithi192.stdout:Reading package lists... 2023-10-30T12:26:45.182 INFO:teuthology.orchestra.run.smithi192.stdout:Building dependency tree... 2023-10-30T12:26:45.182 INFO:teuthology.orchestra.run.smithi192.stdout:Reading state information... 2023-10-30T12:26:45.328 INFO:teuthology.orchestra.run.smithi192.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2023-10-30T12:26:45.328 INFO:teuthology.orchestra.run.smithi192.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2023-10-30T12:26:45.329 INFO:teuthology.orchestra.run.smithi192.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2023-10-30T12:26:45.329 INFO:teuthology.orchestra.run.smithi192.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2023-10-30T12:26:45.329 INFO:teuthology.orchestra.run.smithi192.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2023-10-30T12:26:45.329 INFO:teuthology.orchestra.run.smithi192.stdout:The following packages were automatically installed and are no longer required: 2023-10-30T12:26:45.329 INFO:teuthology.orchestra.run.smithi192.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-30T12:26:45.330 INFO:teuthology.orchestra.run.smithi192.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-30T12:26:45.330 INFO:teuthology.orchestra.run.smithi192.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-30T12:26:45.330 INFO:teuthology.orchestra.run.smithi192.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-10-30T12:26:45.330 INFO:teuthology.orchestra.run.smithi192.stdout: python2.7-minimal 2023-10-30T12:26:45.331 INFO:teuthology.orchestra.run.smithi192.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-30T12:26:45.332 INFO:teuthology.orchestra.run.smithi192.stdout:The following additional packages will be installed: 2023-10-30T12:26:45.332 INFO:teuthology.orchestra.run.smithi192.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2023-10-30T12:26:45.332 INFO:teuthology.orchestra.run.smithi192.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2023-10-30T12:26:45.332 INFO:teuthology.orchestra.run.smithi192.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2023-10-30T12:26:45.332 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2023-10-30T12:26:45.333 INFO:teuthology.orchestra.run.smithi192.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-30T12:26:45.333 INFO:teuthology.orchestra.run.smithi192.stdout:Suggested packages: 2023-10-30T12:26:45.333 INFO:teuthology.orchestra.run.smithi192.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2023-10-30T12:26:45.333 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2023-10-30T12:26:45.333 INFO:teuthology.orchestra.run.smithi192.stdout: libengine-pkcs11-openssl 2023-10-30T12:26:45.368 INFO:teuthology.orchestra.run.smithi192.stdout:The following NEW packages will be installed: 2023-10-30T12:26:45.369 INFO:teuthology.orchestra.run.smithi192.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2023-10-30T12:26:45.369 INFO:teuthology.orchestra.run.smithi192.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2023-10-30T12:26:45.369 INFO:teuthology.orchestra.run.smithi192.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2023-10-30T12:26:45.370 INFO:teuthology.orchestra.run.smithi192.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2023-10-30T12:26:45.370 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2023-10-30T12:26:45.370 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2023-10-30T12:26:45.371 INFO:teuthology.orchestra.run.smithi192.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-30T12:26:45.371 INFO:teuthology.orchestra.run.smithi192.stdout:The following packages will be upgraded: 2023-10-30T12:26:45.371 INFO:teuthology.orchestra.run.smithi192.stdout: libssl3 2023-10-30T12:26:45.421 INFO:teuthology.orchestra.run.smithi192.stdout:1 upgraded, 39 newly installed, 0 to remove and 259 not upgraded. 2023-10-30T12:26:45.422 INFO:teuthology.orchestra.run.smithi192.stdout:Need to get 54.6 MB of archives. 2023-10-30T12:26:45.422 INFO:teuthology.orchestra.run.smithi192.stdout:After this operation, 202 MB of additional disk space will be used. 2023-10-30T12:26:45.422 INFO:teuthology.orchestra.run.smithi192.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2023-10-30T12:26:45.520 INFO:teuthology.orchestra.run.smithi192.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.12 [1,901 kB] 2023-10-30T12:26:45.580 INFO:teuthology.orchestra.run.smithi192.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2023-10-30T12:26:45.594 INFO:teuthology.orchestra.run.smithi192.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2023-10-30T12:26:45.595 INFO:teuthology.orchestra.run.smithi192.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2023-10-30T12:26:45.598 INFO:teuthology.orchestra.run.smithi192.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2023-10-30T12:26:45.600 INFO:teuthology.orchestra.run.smithi192.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2023-10-30T12:26:45.600 INFO:teuthology.orchestra.run.smithi192.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 libbluetooth3 amd64 5.64-0ubuntu1 [86.9 kB] 2023-10-30T12:26:45.602 INFO:teuthology.orchestra.run.smithi192.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2023-10-30T12:26:45.603 INFO:teuthology.orchestra.run.smithi192.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2023-10-30T12:26:45.604 INFO:teuthology.orchestra.run.smithi192.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2023-10-30T12:26:45.604 INFO:teuthology.orchestra.run.smithi192.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2023-10-30T12:26:45.616 INFO:teuthology.orchestra.run.smithi192.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2023-10-30T12:26:45.957 INFO:teuthology.orchestra.run.smithi192.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2023-10-30T12:26:45.959 INFO:teuthology.orchestra.run.smithi192.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2023-10-30T12:26:45.960 INFO:teuthology.orchestra.run.smithi192.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.2 [263 kB] 2023-10-30T12:26:45.963 INFO:teuthology.orchestra.run.smithi192.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2023-10-30T12:26:45.963 INFO:teuthology.orchestra.run.smithi192.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2023-10-30T12:26:45.967 INFO:teuthology.orchestra.run.smithi192.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.9-0ubuntu0.22.04.1 [141 kB] 2023-10-30T12:26:45.969 INFO:teuthology.orchestra.run.smithi192.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2023-10-30T12:26:45.977 INFO:teuthology.orchestra.run.smithi192.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2023-10-30T12:26:45.977 INFO:teuthology.orchestra.run.smithi192.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.12 [2,373 kB] 2023-10-30T12:26:46.000 INFO:teuthology.orchestra.run.smithi192.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2023-10-30T12:26:46.001 INFO:teuthology.orchestra.run.smithi192.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.2 [1,094 kB] 2023-10-30T12:26:46.011 INFO:teuthology.orchestra.run.smithi192.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2023-10-30T12:26:46.031 INFO:teuthology.orchestra.run.smithi192.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2023-10-30T12:26:46.034 INFO:teuthology.orchestra.run.smithi192.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2023-10-30T12:26:46.035 INFO:teuthology.orchestra.run.smithi192.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2023-10-30T12:26:46.035 INFO:teuthology.orchestra.run.smithi192.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2023-10-30T12:26:46.036 INFO:teuthology.orchestra.run.smithi192.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.9-0ubuntu0.22.04.1 [1,222 kB] 2023-10-30T12:26:46.046 INFO:teuthology.orchestra.run.smithi192.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2023-10-30T12:26:46.047 INFO:teuthology.orchestra.run.smithi192.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.9-0ubuntu0.22.04.1 [16.1 MB] 2023-10-30T12:26:46.217 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-rbd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:46.242 INFO:teuthology.orchestra.run.smithi192.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2023-10-30T12:26:46.242 INFO:teuthology.orchestra.run.smithi192.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2023-10-30T12:26:46.242 INFO:teuthology.orchestra.run.smithi192.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2023-10-30T12:26:46.243 INFO:teuthology.orchestra.run.smithi192.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2023-10-30T12:26:46.243 INFO:teuthology.orchestra.run.smithi192.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2023-10-30T12:26:46.243 INFO:teuthology.orchestra.run.smithi192.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2023-10-30T12:26:46.275 INFO:teuthology.orchestra.run.smithi192.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2023-10-30T12:26:46.275 INFO:teuthology.orchestra.run.smithi192.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2023-10-30T12:26:46.350 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up rbd-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:46.470 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-rgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:46.596 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up python3-cephfs (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:46.679 INFO:teuthology.orchestra.run.smithi192.stdout:Preconfiguring packages ... 2023-10-30T12:26:46.721 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up cephfs-shell (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:46.848 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:46.878 INFO:teuthology.orchestra.run.smithi192.stdout:Fetched 54.6 MB in 1s (60.8 MB/s) 2023-10-30T12:26:47.014 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package flex. 2023-10-30T12:26:47.071 INFO:teuthology.orchestra.run.smithi192.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137425 files and directories currently installed.) 2023-10-30T12:26:47.075 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2023-10-30T12:26:47.121 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking flex (2.6.4-8build2) ... 2023-10-30T12:26:47.443 INFO:teuthology.orchestra.run.smithi089.stdout:Adding group ceph....done 2023-10-30T12:26:47.715 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.12_amd64.deb ... 2023-10-30T12:26:47.810 INFO:teuthology.orchestra.run.smithi089.stdout:Adding system user ceph....done 2023-10-30T12:26:47.927 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.12) over (3.0.2-0ubuntu1.7) ... 2023-10-30T12:26:48.022 INFO:teuthology.orchestra.run.smithi089.stdout:Setting system user ceph properties....done 2023-10-30T12:26:48.041 INFO:teuthology.orchestra.run.smithi089.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-30T12:26:48.184 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-30T12:26:48.376 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T12:26:48.589 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-30T12:26:48.936 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package bison. 2023-10-30T12:26:49.025 INFO:teuthology.orchestra.run.smithi192.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137498 files and directories currently installed.) 2023-10-30T12:26:49.030 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2023-10-30T12:26:49.089 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2023-10-30T12:26:49.299 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up cephfs-mirror (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:49.507 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-30T12:26:49.508 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-30T12:26:49.597 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-10-30T12:26:49.619 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2023-10-30T12:26:49.697 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2023-10-30T12:26:50.100 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package dump. 2023-10-30T12:26:50.122 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2023-10-30T12:26:50.134 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ceph-test (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:50.200 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking dump (0.4b47-3) ... 2023-10-30T12:26:50.260 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up radosgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:50.470 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package indent. 2023-10-30T12:26:50.483 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T12:26:50.520 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-30T12:26:50.804 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-30T12:26:50.804 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-30T12:26:50.814 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-10-30T12:26:50.827 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2023-10-30T12:26:50.864 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-30T12:26:51.218 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-10-30T12:26:51.240 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1_amd64.deb ... 2023-10-30T12:26:51.284 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-30T12:26:51.453 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ceph-base (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:51.546 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-10-30T12:26:51.567 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2023-10-30T12:26:51.637 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2023-10-30T12:26:51.841 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-30T12:26:51.916 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libfl2:amd64. 2023-10-30T12:26:51.937 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2023-10-30T12:26:51.974 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2023-10-30T12:26:52.243 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libfl-dev:amd64. 2023-10-30T12:26:52.255 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2023-10-30T12:26:52.293 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-30T12:26:52.497 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ceph-mds (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:52.546 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libjson-perl. 2023-10-30T12:26:52.558 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2023-10-30T12:26:52.620 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libjson-perl (4.04000-1) ... 2023-10-30T12:26:52.737 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-30T12:26:52.737 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-30T12:26:52.957 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libllvm14:amd64. 2023-10-30T12:26:52.979 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2023-10-30T12:26:53.023 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-30T12:26:53.307 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ceph-mgr (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:53.519 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-30T12:26:53.519 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-30T12:26:54.134 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ceph-osd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:54.576 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-30T12:26:54.576 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-30T12:26:54.718 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-10-30T12:26:54.740 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-30T12:26:54.784 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T12:26:55.037 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libmbim-proxy. 2023-10-30T12:26:55.050 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-30T12:26:55.104 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T12:26:55.210 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:55.353 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:55.465 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-10-30T12:26:55.487 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-30T12:26:55.496 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ceph-mon (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:55.558 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T12:26:55.705 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-30T12:26:55.705 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-30T12:26:55.910 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libndp0:amd64. 2023-10-30T12:26:55.931 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2023-10-30T12:26:55.976 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-30T12:26:56.339 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ceph-mgr-cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:56.354 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libnm0:amd64. 2023-10-30T12:26:56.376 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2023-10-30T12:26:56.421 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-30T12:26:56.481 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:56.608 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ceph-mgr-dashboard (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:56.750 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ceph-volume (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:56.857 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libpq5:amd64. 2023-10-30T12:26:56.879 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../16-libpq5_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T12:26:56.932 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T12:26:57.103 INFO:teuthology.orchestra.run.smithi089.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-30T12:26:57.335 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-10-30T12:26:57.356 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T12:26:57.401 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T12:26:57.746 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libqmi-proxy. 2023-10-30T12:26:57.758 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T12:26:57.796 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T12:26:57.878 INFO:teuthology.orchestra.run.smithi089.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-30T12:26:58.098 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-10-30T12:26:58.120 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.12_amd64.deb ... 2023-10-30T12:26:58.173 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T12:26:58.439 INFO:teuthology.orchestra.run.smithi089.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-30T12:26:58.444 DEBUG:teuthology.orchestra.run.smithi089:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2023-10-30T12:26:58.520 INFO:teuthology.orchestra.run.smithi089.stdout:Reading package lists... 2023-10-30T12:26:58.685 INFO:teuthology.orchestra.run.smithi089.stdout:Building dependency tree... 2023-10-30T12:26:58.686 INFO:teuthology.orchestra.run.smithi089.stdout:Reading state information... 2023-10-30T12:26:58.785 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-10-30T12:26:58.807 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2023-10-30T12:26:58.833 INFO:teuthology.orchestra.run.smithi089.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2023-10-30T12:26:58.834 INFO:teuthology.orchestra.run.smithi089.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2023-10-30T12:26:58.834 INFO:teuthology.orchestra.run.smithi089.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2023-10-30T12:26:58.834 INFO:teuthology.orchestra.run.smithi089.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2023-10-30T12:26:58.834 INFO:teuthology.orchestra.run.smithi089.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2023-10-30T12:26:58.834 INFO:teuthology.orchestra.run.smithi089.stdout:The following packages were automatically installed and are no longer required: 2023-10-30T12:26:58.835 INFO:teuthology.orchestra.run.smithi089.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-30T12:26:58.835 INFO:teuthology.orchestra.run.smithi089.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-30T12:26:58.835 INFO:teuthology.orchestra.run.smithi089.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-30T12:26:58.835 INFO:teuthology.orchestra.run.smithi089.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-30T12:26:58.836 INFO:teuthology.orchestra.run.smithi089.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-30T12:26:58.836 INFO:teuthology.orchestra.run.smithi089.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-30T12:26:58.837 INFO:teuthology.orchestra.run.smithi089.stdout:The following additional packages will be installed: 2023-10-30T12:26:58.837 INFO:teuthology.orchestra.run.smithi089.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2023-10-30T12:26:58.837 INFO:teuthology.orchestra.run.smithi089.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2023-10-30T12:26:58.837 INFO:teuthology.orchestra.run.smithi089.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2023-10-30T12:26:58.837 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2023-10-30T12:26:58.838 INFO:teuthology.orchestra.run.smithi089.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-30T12:26:58.838 INFO:teuthology.orchestra.run.smithi089.stdout:Suggested packages: 2023-10-30T12:26:58.838 INFO:teuthology.orchestra.run.smithi089.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2023-10-30T12:26:58.838 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2023-10-30T12:26:58.839 INFO:teuthology.orchestra.run.smithi089.stdout: libengine-pkcs11-openssl 2023-10-30T12:26:58.852 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2023-10-30T12:26:58.885 INFO:teuthology.orchestra.run.smithi089.stdout:The following NEW packages will be installed: 2023-10-30T12:26:58.886 INFO:teuthology.orchestra.run.smithi089.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2023-10-30T12:26:58.886 INFO:teuthology.orchestra.run.smithi089.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2023-10-30T12:26:58.887 INFO:teuthology.orchestra.run.smithi089.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2023-10-30T12:26:58.887 INFO:teuthology.orchestra.run.smithi089.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2023-10-30T12:26:58.887 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2023-10-30T12:26:58.888 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2023-10-30T12:26:58.888 INFO:teuthology.orchestra.run.smithi089.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-30T12:26:58.888 INFO:teuthology.orchestra.run.smithi089.stdout:The following packages will be upgraded: 2023-10-30T12:26:58.889 INFO:teuthology.orchestra.run.smithi089.stdout: libssl3 2023-10-30T12:26:58.943 INFO:teuthology.orchestra.run.smithi089.stdout:1 upgraded, 39 newly installed, 0 to remove and 259 not upgraded. 2023-10-30T12:26:58.943 INFO:teuthology.orchestra.run.smithi089.stdout:Need to get 54.6 MB of archives. 2023-10-30T12:26:58.943 INFO:teuthology.orchestra.run.smithi089.stdout:After this operation, 202 MB of additional disk space will be used. 2023-10-30T12:26:58.944 INFO:teuthology.orchestra.run.smithi089.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2023-10-30T12:26:59.047 INFO:teuthology.orchestra.run.smithi089.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.12 [1,901 kB] 2023-10-30T12:26:59.101 INFO:teuthology.orchestra.run.smithi089.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2023-10-30T12:26:59.109 INFO:teuthology.orchestra.run.smithi089.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2023-10-30T12:26:59.110 INFO:teuthology.orchestra.run.smithi089.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2023-10-30T12:26:59.112 INFO:teuthology.orchestra.run.smithi089.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2023-10-30T12:26:59.113 INFO:teuthology.orchestra.run.smithi089.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2023-10-30T12:26:59.113 INFO:teuthology.orchestra.run.smithi089.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 libbluetooth3 amd64 5.64-0ubuntu1 [86.9 kB] 2023-10-30T12:26:59.114 INFO:teuthology.orchestra.run.smithi089.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2023-10-30T12:26:59.115 INFO:teuthology.orchestra.run.smithi089.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2023-10-30T12:26:59.117 INFO:teuthology.orchestra.run.smithi089.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2023-10-30T12:26:59.120 INFO:teuthology.orchestra.run.smithi089.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2023-10-30T12:26:59.133 INFO:teuthology.orchestra.run.smithi089.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2023-10-30T12:26:59.163 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package modemmanager. 2023-10-30T12:26:59.185 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-30T12:26:59.229 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T12:26:59.360 INFO:teuthology.orchestra.run.smithi089.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2023-10-30T12:26:59.362 INFO:teuthology.orchestra.run.smithi089.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2023-10-30T12:26:59.362 INFO:teuthology.orchestra.run.smithi089.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.2 [263 kB] 2023-10-30T12:26:59.365 INFO:teuthology.orchestra.run.smithi089.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2023-10-30T12:26:59.366 INFO:teuthology.orchestra.run.smithi089.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2023-10-30T12:26:59.370 INFO:teuthology.orchestra.run.smithi089.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.9-0ubuntu0.22.04.1 [141 kB] 2023-10-30T12:26:59.372 INFO:teuthology.orchestra.run.smithi089.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2023-10-30T12:26:59.379 INFO:teuthology.orchestra.run.smithi089.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2023-10-30T12:26:59.379 INFO:teuthology.orchestra.run.smithi089.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.12 [2,373 kB] 2023-10-30T12:26:59.403 INFO:teuthology.orchestra.run.smithi089.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2023-10-30T12:26:59.403 INFO:teuthology.orchestra.run.smithi089.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.2 [1,094 kB] 2023-10-30T12:26:59.414 INFO:teuthology.orchestra.run.smithi089.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2023-10-30T12:26:59.436 INFO:teuthology.orchestra.run.smithi089.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2023-10-30T12:26:59.440 INFO:teuthology.orchestra.run.smithi089.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2023-10-30T12:26:59.441 INFO:teuthology.orchestra.run.smithi089.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2023-10-30T12:26:59.441 INFO:teuthology.orchestra.run.smithi089.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2023-10-30T12:26:59.442 INFO:teuthology.orchestra.run.smithi089.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.9-0ubuntu0.22.04.1 [1,222 kB] 2023-10-30T12:26:59.456 INFO:teuthology.orchestra.run.smithi089.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2023-10-30T12:26:59.458 INFO:teuthology.orchestra.run.smithi089.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.9-0ubuntu0.22.04.1 [16.1 MB] 2023-10-30T12:26:59.613 INFO:teuthology.orchestra.run.smithi089.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2023-10-30T12:26:59.613 INFO:teuthology.orchestra.run.smithi089.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2023-10-30T12:26:59.613 INFO:teuthology.orchestra.run.smithi089.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2023-10-30T12:26:59.613 INFO:teuthology.orchestra.run.smithi089.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2023-10-30T12:26:59.614 INFO:teuthology.orchestra.run.smithi089.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2023-10-30T12:26:59.614 INFO:teuthology.orchestra.run.smithi089.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2023-10-30T12:26:59.626 INFO:teuthology.orchestra.run.smithi089.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2023-10-30T12:26:59.627 INFO:teuthology.orchestra.run.smithi089.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2023-10-30T12:26:59.927 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package network-manager. 2023-10-30T12:26:59.949 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2023-10-30T12:27:00.003 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2023-10-30T12:27:00.006 INFO:teuthology.orchestra.run.smithi089.stdout:Preconfiguring packages ... 2023-10-30T12:27:00.203 INFO:teuthology.orchestra.run.smithi089.stdout:Fetched 54.6 MB in 1s (74.6 MB/s) 2023-10-30T12:27:00.345 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package flex. 2023-10-30T12:27:00.417 INFO:teuthology.orchestra.run.smithi089.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 139851 files and directories currently installed.) 2023-10-30T12:27:00.420 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2023-10-30T12:27:00.460 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking flex (2.6.4-8build2) ... 2023-10-30T12:27:00.634 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ppp. 2023-10-30T12:27:00.656 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2023-10-30T12:27:00.943 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2023-10-30T12:27:01.029 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.12_amd64.deb ... 2023-10-30T12:27:01.233 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.12) over (3.0.2-0ubuntu1.7) ... 2023-10-30T12:27:01.247 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package pptp-linux. 2023-10-30T12:27:01.270 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2023-10-30T12:27:01.314 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2023-10-30T12:27:01.592 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package network-manager-pptp. 2023-10-30T12:27:01.613 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2023-10-30T12:27:01.623 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T12:27:01.658 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2023-10-30T12:27:01.986 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package postgresql-client-common. 2023-10-30T12:27:02.008 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2023-10-30T12:27:02.052 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking postgresql-client-common (238) ... 2023-10-30T12:27:02.150 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package bison. 2023-10-30T12:27:02.220 INFO:teuthology.orchestra.run.smithi089.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 139924 files and directories currently installed.) 2023-10-30T12:27:02.224 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2023-10-30T12:27:02.277 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2023-10-30T12:27:02.356 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package postgresql-client-14. 2023-10-30T12:27:02.379 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../27-postgresql-client-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T12:27:02.414 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T12:27:02.703 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-10-30T12:27:02.717 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2023-10-30T12:27:02.761 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2023-10-30T12:27:02.986 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package postgresql-common. 2023-10-30T12:27:03.008 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2023-10-30T12:27:03.064 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package dump. 2023-10-30T12:27:03.070 INFO:teuthology.orchestra.run.smithi192.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2023-10-30T12:27:03.078 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2023-10-30T12:27:03.108 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking postgresql-common (238) ... 2023-10-30T12:27:03.114 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking dump (0.4b47-3) ... 2023-10-30T12:27:03.519 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package indent. 2023-10-30T12:27:03.533 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T12:27:03.569 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-30T12:27:03.585 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package postgresql-14. 2023-10-30T12:27:03.607 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../29-postgresql-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T12:27:03.655 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T12:27:03.813 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-10-30T12:27:03.827 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2023-10-30T12:27:03.872 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-30T12:27:04.200 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-10-30T12:27:04.215 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1_amd64.deb ... 2023-10-30T12:27:04.258 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-30T12:27:04.553 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-10-30T12:27:04.568 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2023-10-30T12:27:04.603 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2023-10-30T12:27:05.057 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libfl2:amd64. 2023-10-30T12:27:05.071 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2023-10-30T12:27:05.114 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2023-10-30T12:27:05.138 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package postgresql. 2023-10-30T12:27:05.151 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2023-10-30T12:27:05.188 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking postgresql (14+238) ... 2023-10-30T12:27:05.400 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libfl-dev:amd64. 2023-10-30T12:27:05.415 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2023-10-30T12:27:05.448 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package postgresql-client. 2023-10-30T12:27:05.458 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-30T12:27:05.461 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2023-10-30T12:27:05.498 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking postgresql-client (14+238) ... 2023-10-30T12:27:05.720 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libjson-perl. 2023-10-30T12:27:05.735 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2023-10-30T12:27:05.767 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package postgresql-contrib. 2023-10-30T12:27:05.778 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libjson-perl (4.04000-1) ... 2023-10-30T12:27:05.780 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2023-10-30T12:27:05.817 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking postgresql-contrib (14+238) ... 2023-10-30T12:27:06.073 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libllvm14:amd64. 2023-10-30T12:27:06.088 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2023-10-30T12:27:06.103 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package tcl8.6. 2023-10-30T12:27:06.122 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-30T12:27:06.125 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2023-10-30T12:27:06.211 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-30T12:27:06.498 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package tcl. 2023-10-30T12:27:06.520 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2023-10-30T12:27:06.569 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking tcl (8.6.11+1build2) ... 2023-10-30T12:27:06.950 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package wpasupplicant. 2023-10-30T12:27:06.972 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2023-10-30T12:27:07.017 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-30T12:27:07.388 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-10-30T12:27:07.401 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2023-10-30T12:27:07.438 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2023-10-30T12:27:07.508 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-10-30T12:27:07.524 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-30T12:27:07.567 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T12:27:07.833 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package usb-modeswitch. 2023-10-30T12:27:07.836 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libmbim-proxy. 2023-10-30T12:27:07.851 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-30T12:27:07.855 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2023-10-30T12:27:07.894 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T12:27:07.908 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-30T12:27:08.222 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-10-30T12:27:08.233 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up postgresql-client-common (238) ... 2023-10-30T12:27:08.237 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-30T12:27:08.280 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T12:27:08.497 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up flex (2.6.4-8build2) ... 2023-10-30T12:27:08.657 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-30T12:27:08.783 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T12:27:08.916 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2023-10-30T12:27:09.075 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libndp0:amd64. 2023-10-30T12:27:09.090 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2023-10-30T12:27:09.092 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-30T12:27:09.133 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-30T12:27:09.243 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T12:27:09.445 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libnm0:amd64. 2023-10-30T12:27:09.460 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2023-10-30T12:27:09.503 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-30T12:27:09.732 INFO:teuthology.orchestra.run.smithi192.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2023-10-30T12:27:09.922 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libpq5:amd64. 2023-10-30T12:27:09.938 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../16-libpq5_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T12:27:09.981 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T12:27:10.259 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-10-30T12:27:10.273 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-30T12:27:10.274 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T12:27:10.308 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T12:27:10.594 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libqmi-proxy. 2023-10-30T12:27:10.610 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T12:27:10.653 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T12:27:10.662 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-30T12:27:10.662 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-30T12:27:10.922 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-10-30T12:27:10.937 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.12_amd64.deb ... 2023-10-30T12:27:10.980 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T12:27:11.176 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2023-10-30T12:27:11.327 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T12:27:11.479 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2023-10-30T12:27:11.483 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-10-30T12:27:11.498 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2023-10-30T12:27:11.541 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2023-10-30T12:27:11.655 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2023-10-30T12:27:11.805 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2023-10-30T12:27:11.927 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package modemmanager. 2023-10-30T12:27:11.932 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-30T12:27:11.942 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-30T12:27:11.986 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T12:27:12.083 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2023-10-30T12:27:12.592 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package network-manager. 2023-10-30T12:27:12.606 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2023-10-30T12:27:12.658 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2023-10-30T12:27:12.857 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-30T12:27:13.016 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up dump (0.4b47-3) ... 2023-10-30T12:27:13.110 INFO:teuthology.orchestra.run.smithi192.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-10-30T12:27:13.184 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T12:27:13.274 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package ppp. 2023-10-30T12:27:13.288 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2023-10-30T12:27:13.335 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-30T12:27:13.511 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up tcl (8.6.11+1build2) ... 2023-10-30T12:27:13.644 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2023-10-30T12:27:13.662 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libjson-perl (4.04000-1) ... 2023-10-30T12:27:13.813 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2023-10-30T12:27:13.930 INFO:teuthology.orchestra.run.smithi192.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2023-10-30T12:27:14.023 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-30T12:27:14.063 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package pptp-linux. 2023-10-30T12:27:14.077 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2023-10-30T12:27:14.121 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2023-10-30T12:27:14.149 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-30T12:27:14.342 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-30T12:27:14.366 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package network-manager-pptp. 2023-10-30T12:27:14.380 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2023-10-30T12:27:14.415 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2023-10-30T12:27:14.527 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T12:27:14.678 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2023-10-30T12:27:14.685 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package postgresql-client-common. 2023-10-30T12:27:14.700 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2023-10-30T12:27:14.744 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking postgresql-client-common (238) ... 2023-10-30T12:27:14.879 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up postgresql-client (14+238) ... 2023-10-30T12:27:15.063 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2023-10-30T12:27:15.064 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package postgresql-client-14. 2023-10-30T12:27:15.079 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../27-postgresql-client-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T12:27:15.122 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T12:27:15.702 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package postgresql-common. 2023-10-30T12:27:15.717 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2023-10-30T12:27:15.777 INFO:teuthology.orchestra.run.smithi089.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2023-10-30T12:27:15.809 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking postgresql-common (238) ... 2023-10-30T12:27:16.142 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:16.142 INFO:teuthology.orchestra.run.smithi192.stdout:The following network interfaces were found in /etc/network/interfaces 2023-10-30T12:27:16.142 INFO:teuthology.orchestra.run.smithi192.stdout:which means they are currently configured by ifupdown: 2023-10-30T12:27:16.142 INFO:teuthology.orchestra.run.smithi192.stdout:- enp3s0f1 2023-10-30T12:27:16.143 INFO:teuthology.orchestra.run.smithi192.stdout:If you want to manage those interfaces with NetworkManager instead 2023-10-30T12:27:16.143 INFO:teuthology.orchestra.run.smithi192.stdout:remove their configuration from /etc/network/interfaces. 2023-10-30T12:27:16.143 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:16.271 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-10-30T12:27:16.301 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package postgresql-14. 2023-10-30T12:27:16.315 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../29-postgresql-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T12:27:16.354 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T12:27:16.604 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-10-30T12:27:16.947 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-10-30T12:27:17.567 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T12:27:17.588 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package postgresql. 2023-10-30T12:27:17.602 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2023-10-30T12:27:17.637 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking postgresql (14+238) ... 2023-10-30T12:27:17.726 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-30T12:27:17.848 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package postgresql-client. 2023-10-30T12:27:17.862 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2023-10-30T12:27:17.903 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2023-10-30T12:27:17.906 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking postgresql-client (14+238) ... 2023-10-30T12:27:18.053 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up postgresql-common (238) ... 2023-10-30T12:27:18.208 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package postgresql-contrib. 2023-10-30T12:27:18.223 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2023-10-30T12:27:18.267 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking postgresql-contrib (14+238) ... 2023-10-30T12:27:18.595 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package tcl8.6. 2023-10-30T12:27:18.609 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2023-10-30T12:27:18.653 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-30T12:27:18.975 INFO:teuthology.orchestra.run.smithi192.stdout:Adding user postgres to group ssl-cert 2023-10-30T12:27:19.083 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package tcl. 2023-10-30T12:27:19.098 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2023-10-30T12:27:19.146 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking tcl (8.6.11+1build2) ... 2023-10-30T12:27:19.168 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:19.168 INFO:teuthology.orchestra.run.smithi192.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2023-10-30T12:27:19.367 INFO:teuthology.orchestra.run.smithi192.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2023-10-30T12:27:19.367 INFO:teuthology.orchestra.run.smithi192.stdout:Removing obsolete dictionary files: 2023-10-30T12:27:19.511 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package wpasupplicant. 2023-10-30T12:27:19.527 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2023-10-30T12:27:19.570 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-30T12:27:19.899 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-10-30T12:27:19.914 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2023-10-30T12:27:19.948 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2023-10-30T12:27:20.006 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2023-10-30T12:27:20.335 INFO:teuthology.orchestra.run.smithi089.stdout:Selecting previously unselected package usb-modeswitch. 2023-10-30T12:27:20.351 INFO:teuthology.orchestra.run.smithi089.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2023-10-30T12:27:20.402 INFO:teuthology.orchestra.run.smithi089.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-30T12:27:20.472 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T12:27:20.677 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up postgresql-client-common (238) ... 2023-10-30T12:27:20.802 INFO:teuthology.orchestra.run.smithi192.stdout:Creating new PostgreSQL cluster 14/main ... 2023-10-30T12:27:20.816 INFO:teuthology.orchestra.run.smithi192.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2023-10-30T12:27:20.853 INFO:teuthology.orchestra.run.smithi192.stdout:The files belonging to this database system will be owned by user "postgres". 2023-10-30T12:27:20.854 INFO:teuthology.orchestra.run.smithi192.stdout:This user must also own the server process. 2023-10-30T12:27:20.854 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:20.854 INFO:teuthology.orchestra.run.smithi192.stdout:The database cluster will be initialized with locale "en_US". 2023-10-30T12:27:20.854 INFO:teuthology.orchestra.run.smithi192.stdout:The default database encoding has accordingly been set to "LATIN1". 2023-10-30T12:27:20.855 INFO:teuthology.orchestra.run.smithi192.stdout:The default text search configuration will be set to "english". 2023-10-30T12:27:20.855 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:20.855 INFO:teuthology.orchestra.run.smithi192.stdout:Data page checksums are disabled. 2023-10-30T12:27:20.855 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:20.856 INFO:teuthology.orchestra.run.smithi192.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2023-10-30T12:27:20.856 INFO:teuthology.orchestra.run.smithi192.stdout:creating subdirectories ... ok 2023-10-30T12:27:20.857 INFO:teuthology.orchestra.run.smithi192.stdout:selecting dynamic shared memory implementation ... posix 2023-10-30T12:27:20.867 INFO:teuthology.orchestra.run.smithi192.stdout:selecting default max_connections ... 100 2023-10-30T12:27:20.874 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up flex (2.6.4-8build2) ... 2023-10-30T12:27:20.888 INFO:teuthology.orchestra.run.smithi192.stdout:selecting default shared_buffers ... 128MB 2023-10-30T12:27:20.898 INFO:teuthology.orchestra.run.smithi192.stdout:selecting default time zone ... Etc/UTC 2023-10-30T12:27:20.899 INFO:teuthology.orchestra.run.smithi192.stdout:creating configuration files ... ok 2023-10-30T12:27:21.000 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-30T12:27:21.013 INFO:teuthology.orchestra.run.smithi192.stdout:running bootstrap script ... ok 2023-10-30T12:27:21.126 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T12:27:21.252 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2023-10-30T12:27:21.403 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-30T12:27:21.529 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T12:27:21.721 INFO:teuthology.orchestra.run.smithi192.stdout:performing post-bootstrap initialization ... ok 2023-10-30T12:27:22.536 INFO:teuthology.orchestra.run.smithi089.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2023-10-30T12:27:22.635 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-30T12:27:22.760 INFO:teuthology.orchestra.run.smithi192.stdout:syncing data to disk ... ok 2023-10-30T12:27:22.999 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-30T12:27:23.000 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-30T12:27:23.185 INFO:teuthology.orchestra.run.smithi192.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2023-10-30T12:27:23.673 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2023-10-30T12:27:23.799 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T12:27:23.933 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2023-10-30T12:27:24.059 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2023-10-30T12:27:24.185 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2023-10-30T12:27:24.311 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-30T12:27:24.446 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2023-10-30T12:27:25.185 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-30T12:27:25.311 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up dump (0.4b47-3) ... 2023-10-30T12:27:25.406 INFO:teuthology.orchestra.run.smithi089.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-10-30T12:27:25.471 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T12:27:25.630 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-30T12:27:25.756 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up tcl (8.6.11+1build2) ... 2023-10-30T12:27:25.874 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libjson-perl (4.04000-1) ... 2023-10-30T12:27:25.975 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2023-10-30T12:27:26.056 INFO:teuthology.orchestra.run.smithi089.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2023-10-30T12:27:26.103 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T12:27:26.118 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-30T12:27:26.244 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-30T12:27:26.263 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up postgresql-contrib (14+238) ... 2023-10-30T12:27:26.389 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up postgresql (14+238) ... 2023-10-30T12:27:26.420 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-30T12:27:26.515 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T12:27:26.546 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T12:27:26.674 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T12:27:26.706 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2023-10-30T12:27:26.874 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up postgresql-client (14+238) ... 2023-10-30T12:27:26.929 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-10-30T12:27:26.930 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-10-30T12:27:26.983 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2023-10-30T12:27:27.540 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:27.540 INFO:teuthology.orchestra.run.smithi089.stdout:The following network interfaces were found in /etc/network/interfaces 2023-10-30T12:27:27.540 INFO:teuthology.orchestra.run.smithi089.stdout:which means they are currently configured by ifupdown: 2023-10-30T12:27:27.540 INFO:teuthology.orchestra.run.smithi089.stdout:- enp3s0f1 2023-10-30T12:27:27.541 INFO:teuthology.orchestra.run.smithi089.stdout:If you want to manage those interfaces with NetworkManager instead 2023-10-30T12:27:27.541 INFO:teuthology.orchestra.run.smithi089.stdout:remove their configuration from /etc/network/interfaces. 2023-10-30T12:27:27.541 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:27.644 INFO:teuthology.orchestra.run.smithi192.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-10-30T12:27:27.674 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-10-30T12:27:27.962 INFO:teuthology.orchestra.run.smithi192.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-30T12:27:28.057 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-10-30T12:27:28.139 INFO:teuthology.orchestra.run.smithi192.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-30T12:27:28.434 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-10-30T12:27:29.403 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T12:27:29.538 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-30T12:27:29.650 INFO:teuthology.orchestra.run.smithi192.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2023-10-30T12:27:29.655 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2023-10-30T12:27:29.741 INFO:teuthology.orchestra.run.smithi192.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-10-30T12:27:29.806 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up postgresql-common (238) ... 2023-10-30T12:27:30.419 INFO:teuthology.orchestra.run.smithi192.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-30T12:27:30.422 DEBUG:teuthology.parallel:result is None 2023-10-30T12:27:30.708 INFO:teuthology.orchestra.run.smithi089.stdout:Adding user postgres to group ssl-cert 2023-10-30T12:27:30.956 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:30.956 INFO:teuthology.orchestra.run.smithi089.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2023-10-30T12:27:31.253 INFO:teuthology.orchestra.run.smithi089.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2023-10-30T12:27:31.253 INFO:teuthology.orchestra.run.smithi089.stdout:Removing obsolete dictionary files: 2023-10-30T12:27:31.965 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2023-10-30T12:27:32.591 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T12:27:32.932 INFO:teuthology.orchestra.run.smithi089.stdout:Creating new PostgreSQL cluster 14/main ... 2023-10-30T12:27:32.951 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2023-10-30T12:27:33.000 INFO:teuthology.orchestra.run.smithi089.stdout:The files belonging to this database system will be owned by user "postgres". 2023-10-30T12:27:33.001 INFO:teuthology.orchestra.run.smithi089.stdout:This user must also own the server process. 2023-10-30T12:27:33.001 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:33.001 INFO:teuthology.orchestra.run.smithi089.stdout:The database cluster will be initialized with locale "en_US". 2023-10-30T12:27:33.001 INFO:teuthology.orchestra.run.smithi089.stdout:The default database encoding has accordingly been set to "LATIN1". 2023-10-30T12:27:33.001 INFO:teuthology.orchestra.run.smithi089.stdout:The default text search configuration will be set to "english". 2023-10-30T12:27:33.002 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:33.002 INFO:teuthology.orchestra.run.smithi089.stdout:Data page checksums are disabled. 2023-10-30T12:27:33.002 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:33.002 INFO:teuthology.orchestra.run.smithi089.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2023-10-30T12:27:33.003 INFO:teuthology.orchestra.run.smithi089.stdout:creating subdirectories ... ok 2023-10-30T12:27:33.003 INFO:teuthology.orchestra.run.smithi089.stdout:selecting dynamic shared memory implementation ... posix 2023-10-30T12:27:33.026 INFO:teuthology.orchestra.run.smithi089.stdout:selecting default max_connections ... 100 2023-10-30T12:27:33.062 INFO:teuthology.orchestra.run.smithi089.stdout:selecting default shared_buffers ... 128MB 2023-10-30T12:27:33.075 INFO:teuthology.orchestra.run.smithi089.stdout:selecting default time zone ... Etc/UTC 2023-10-30T12:27:33.076 INFO:teuthology.orchestra.run.smithi089.stdout:creating configuration files ... ok 2023-10-30T12:27:33.244 INFO:teuthology.orchestra.run.smithi089.stdout:running bootstrap script ... ok 2023-10-30T12:27:33.857 INFO:teuthology.orchestra.run.smithi089.stdout:performing post-bootstrap initialization ... ok 2023-10-30T12:27:34.423 INFO:teuthology.orchestra.run.smithi089.stdout:syncing data to disk ... ok 2023-10-30T12:27:34.940 INFO:teuthology.orchestra.run.smithi089.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2023-10-30T12:27:37.916 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T12:27:38.025 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up postgresql-contrib (14+238) ... 2023-10-30T12:27:38.151 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up postgresql (14+238) ... 2023-10-30T12:27:38.277 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T12:27:38.403 INFO:teuthology.orchestra.run.smithi089.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T12:27:38.689 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-10-30T12:27:38.689 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-10-30T12:27:39.431 INFO:teuthology.orchestra.run.smithi089.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-10-30T12:27:39.808 INFO:teuthology.orchestra.run.smithi089.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-30T12:27:39.951 INFO:teuthology.orchestra.run.smithi089.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-30T12:27:41.795 INFO:teuthology.orchestra.run.smithi089.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2023-10-30T12:27:41.929 INFO:teuthology.orchestra.run.smithi089.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-10-30T12:27:42.622 INFO:teuthology.orchestra.run.smithi089.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-30T12:27:42.627 DEBUG:teuthology.parallel:result is None 2023-10-30T12:27:42.628 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T12:27:42.854 DEBUG:teuthology.orchestra.run.smithi089:> dpkg-query -W -f '${Version}' ceph 2023-10-30T12:27:42.876 INFO:teuthology.orchestra.run.smithi089.stdout:18.0.0-6835-g00c1c13c-1jammy 2023-10-30T12:27:42.876 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6835-g00c1c13c-1jammy 2023-10-30T12:27:42.877 INFO:teuthology.task.install:The correct ceph version 18.0.0-6835-g00c1c13c-1jammy is installed. 2023-10-30T12:27:42.877 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T12:27:43.028 DEBUG:teuthology.orchestra.run.smithi192:> dpkg-query -W -f '${Version}' ceph 2023-10-30T12:27:43.048 INFO:teuthology.orchestra.run.smithi192.stdout:18.0.0-6835-g00c1c13c-1jammy 2023-10-30T12:27:43.048 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6835-g00c1c13c-1jammy 2023-10-30T12:27:43.048 INFO:teuthology.task.install:The correct ceph version 18.0.0-6835-g00c1c13c-1jammy is installed. 2023-10-30T12:27:43.049 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-30T12:27:43.049 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-10-30T12:27:43.049 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-30T12:27:43.070 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-30T12:27:43.071 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-30T12:27:43.104 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-30T12:27:43.104 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-10-30T12:27:43.105 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/daemon-helper 2023-10-30T12:27:43.133 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-30T12:27:43.192 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-30T12:27:43.193 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/daemon-helper 2023-10-30T12:27:43.206 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-30T12:27:43.258 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-30T12:27:43.259 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-10-30T12:27:43.259 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-30T12:27:43.279 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-30T12:27:43.340 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-30T12:27:43.340 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-30T12:27:43.354 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-30T12:27:43.407 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-30T12:27:43.408 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-10-30T12:27:43.408 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/stdin-killer 2023-10-30T12:27:43.428 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-30T12:27:43.489 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-30T12:27:43.490 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/stdin-killer 2023-10-30T12:27:43.503 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-30T12:27:43.559 INFO:teuthology.run_tasks:Running task ceph... 2023-10-30T12:27:43.622 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-30T12:27:43.623 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 777 /var/log/ceph 2023-10-30T12:27:43.626 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 777 /var/log/ceph 2023-10-30T12:27:43.642 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-30T12:27:43.642 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-30T12:27:43.689 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-30T12:27:43.704 INFO:tasks.ceph:Creating extra log directories... 2023-10-30T12:27:43.705 DEBUG:teuthology.orchestra.run.smithi089:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-30T12:27:43.749 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-30T12:27:43.771 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-10-30T12:27:43.772 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-10-30T12:27:43.772 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-10-30T12:27:43.772 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-10-30T12:27:43.773 DEBUG:teuthology.orchestra.run.smithi089:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-30T12:27:43.814 DEBUG:teuthology.orchestra.run.smithi089:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-30T12:27:43.903 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-30T12:27:43.904 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-30T12:27:43.910 DEBUG:teuthology.orchestra.run.smithi192:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-30T12:27:43.987 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-30T12:27:43.987 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-30T12:27:43.987 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441018', 'branch': 'wip-lusov-balance-after-stop', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}', 'email': None, 'first_in_suite': False, 'job_id': '7441018', 'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-lusov-balance-after-stop'}, 'ceph': {'cephfs': {'max_mds': 3, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds'], 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'}}, 'kclient': {'syntax': 'v1'}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lusov-balance-after-stop', 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'}}, 'owner': 'scheduled_leonidus@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 8397, 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71', 'sleep_before_teardown': 0, 'subset': '464/1024', 'suite': 'fs', 'suite_branch': 'wip-lusov-balance-after-stop', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '00c1c13cea21a686b046584596fbeac5315a7d71', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi089.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOag3my8IamJNicmi37LZgXQEl7nS8WHId2QtJcbtUnA2zSfW/3voFcVlZ9NqpZOgjD5TywEhthoXXM1YcaRGLg=', 'smithi192.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJcRRf5xTajm0BZJQJLWhF2zIJij3ZMLY/mNaetV4sVgWPO7jUH10GB2CXSpPhG7M4/lSP5SCEra8BopJzAdnzY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds'], 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71', 'cluster': 'ceph'}}, {'kclient': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['fs/snaps']}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '6899cd26fceddb2fec83dc1a1349394b28c8998e', 'timestamp': '2023-10-30_06:34:16', 'tube': 'smithi', 'user': 'leonidus', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.770726'} 2023-10-30T12:27:43.988 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-30T12:27:43.992 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-30T12:27:44.028 DEBUG:teuthology.orchestra.run.smithi089:> sudo install -d -m0777 -- /var/run/ceph 2023-10-30T12:27:44.045 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m0777 -- /var/run/ceph 2023-10-30T12:27:44.088 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-10-30T12:27:44.089 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/scratch_devs of=/dev/stdout 2023-10-30T12:27:44.110 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-30T12:27:44.110 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_1 2023-10-30T12:27:44.161 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-30T12:27:44.162 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:27:44.162 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 900 Links: 1 2023-10-30T12:27:44.162 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:27:44.162 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2023-10-30 12:21:16.905039990 +0000 2023-10-30T12:27:44.162 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2023-10-30 12:21:16.501043599 +0000 2023-10-30T12:27:44.163 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2023-10-30 12:21:16.501043599 +0000 2023-10-30T12:27:44.163 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2023-10-30 12:21:16.501043599 +0000 2023-10-30T12:27:44.163 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-30T12:27:44.227 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2023-10-30T12:27:44.227 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2023-10-30T12:27:44.227 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000350194 s, 1.5 MB/s 2023-10-30T12:27:44.228 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-30T12:27:44.279 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_2 2023-10-30T12:27:44.334 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-30T12:27:44.334 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:27:44.334 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 915 Links: 1 2023-10-30T12:27:44.334 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:27:44.335 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2023-10-30 12:21:17.581033952 +0000 2023-10-30T12:27:44.335 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2023-10-30 12:21:17.169037632 +0000 2023-10-30T12:27:44.335 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2023-10-30 12:21:17.169037632 +0000 2023-10-30T12:27:44.335 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2023-10-30 12:21:17.169037632 +0000 2023-10-30T12:27:44.335 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-30T12:27:44.395 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2023-10-30T12:27:44.396 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2023-10-30T12:27:44.396 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000648182 s, 790 kB/s 2023-10-30T12:27:44.397 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-30T12:27:44.448 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_3 2023-10-30T12:27:44.498 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-30T12:27:44.498 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:27:44.498 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 931 Links: 1 2023-10-30T12:27:44.499 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:27:44.499 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2023-10-30 12:21:18.209028343 +0000 2023-10-30T12:27:44.499 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2023-10-30 12:21:17.837031666 +0000 2023-10-30T12:27:44.499 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2023-10-30 12:21:17.837031666 +0000 2023-10-30T12:27:44.499 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2023-10-30 12:21:17.837031666 +0000 2023-10-30T12:27:44.500 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-30T12:27:44.559 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2023-10-30T12:27:44.560 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2023-10-30T12:27:44.560 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000356176 s, 1.4 MB/s 2023-10-30T12:27:44.561 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-30T12:27:44.611 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_4 2023-10-30T12:27:44.666 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-30T12:27:44.666 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:27:44.667 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 951 Links: 1 2023-10-30T12:27:44.667 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:27:44.667 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2023-10-30 12:21:18.889022269 +0000 2023-10-30T12:27:44.667 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2023-10-30 12:21:18.521025555 +0000 2023-10-30T12:27:44.668 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2023-10-30 12:21:18.521025555 +0000 2023-10-30T12:27:44.668 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2023-10-30 12:21:18.521025555 +0000 2023-10-30T12:27:44.668 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-30T12:27:44.728 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2023-10-30T12:27:44.728 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2023-10-30T12:27:44.728 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000369072 s, 1.4 MB/s 2023-10-30T12:27:44.730 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-30T12:27:44.784 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T12:27:44.784 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-30T12:27:44.784 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/scratch_devs of=/dev/stdout 2023-10-30T12:27:44.790 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-30T12:27:44.791 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_1 2023-10-30T12:27:44.836 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-30T12:27:44.837 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:27:44.837 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 885 Links: 1 2023-10-30T12:27:44.837 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:27:44.837 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2023-10-30 12:20:49.967222418 +0000 2023-10-30T12:27:44.838 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2023-10-30 12:20:49.587224807 +0000 2023-10-30T12:27:44.838 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2023-10-30 12:20:49.587224807 +0000 2023-10-30T12:27:44.838 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2023-10-30T12:27:44.838 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-30T12:27:44.892 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2023-10-30T12:27:44.893 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2023-10-30T12:27:44.893 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000555055 s, 922 kB/s 2023-10-30T12:27:44.894 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-30T12:27:44.941 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_2 2023-10-30T12:27:44.988 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-30T12:27:44.988 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:27:44.988 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 901 Links: 1 2023-10-30T12:27:44.989 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:27:44.989 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2023-10-30 12:20:50.579218565 +0000 2023-10-30T12:27:44.989 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2023-10-30 12:20:50.215220857 +0000 2023-10-30T12:27:44.989 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2023-10-30 12:20:50.215220857 +0000 2023-10-30T12:27:44.990 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2023-10-30T12:27:44.990 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-30T12:27:45.042 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2023-10-30T12:27:45.043 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2023-10-30T12:27:45.043 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000556452 s, 920 kB/s 2023-10-30T12:27:45.044 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-30T12:27:45.096 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_3 2023-10-30T12:27:45.144 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-30T12:27:45.145 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:27:45.145 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 924 Links: 1 2023-10-30T12:27:45.145 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:27:45.145 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2023-10-30 12:20:51.183214760 +0000 2023-10-30T12:27:45.146 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2023-10-30 12:20:50.827217003 +0000 2023-10-30T12:27:45.146 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2023-10-30 12:20:50.827217003 +0000 2023-10-30T12:27:45.146 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2023-10-30T12:27:45.146 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-30T12:27:45.198 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2023-10-30T12:27:45.198 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2023-10-30T12:27:45.198 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000452752 s, 1.1 MB/s 2023-10-30T12:27:45.199 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-30T12:27:45.249 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_4 2023-10-30T12:27:45.304 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-30T12:27:45.305 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:27:45.305 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 937 Links: 1 2023-10-30T12:27:45.305 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:27:45.306 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2023-10-30 12:20:51.815210775 +0000 2023-10-30T12:27:45.306 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2023-10-30 12:20:51.455213045 +0000 2023-10-30T12:27:45.306 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2023-10-30 12:20:51.455213045 +0000 2023-10-30T12:27:45.306 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2023-10-30T12:27:45.307 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-30T12:27:45.359 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2023-10-30T12:27:45.359 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2023-10-30T12:27:45.359 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000546442 s, 937 kB/s 2023-10-30T12:27:45.360 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-30T12:27:45.409 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T12:27:45.410 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi089.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi192.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-30T12:27:45.410 INFO:tasks.ceph:Generating config... 2023-10-30T12:27:45.411 INFO:tasks.ceph:[client] client mount timeout = 600 2023-10-30T12:27:45.412 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2023-10-30T12:27:45.412 INFO:tasks.ceph:[client] debug client = 20 2023-10-30T12:27:45.412 INFO:tasks.ceph:[client] debug ms = 1 2023-10-30T12:27:45.412 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-10-30T12:27:45.413 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-10-30T12:27:45.413 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-10-30T12:27:45.413 INFO:tasks.ceph:[mds] debug mds = 20 2023-10-30T12:27:45.413 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-10-30T12:27:45.414 INFO:tasks.ceph:[mds] debug ms = 1 2023-10-30T12:27:45.414 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2023-10-30T12:27:45.414 INFO:tasks.ceph:[mds] mds bal merge size = 5 2023-10-30T12:27:45.415 INFO:tasks.ceph:[mds] mds bal split bits = 3 2023-10-30T12:27:45.415 INFO:tasks.ceph:[mds] mds bal split size = 100 2023-10-30T12:27:45.415 INFO:tasks.ceph:[mds] mds debug frag = True 2023-10-30T12:27:45.415 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-10-30T12:27:45.416 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = True 2023-10-30T12:27:45.416 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-10-30T12:27:45.416 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-10-30T12:27:45.416 INFO:tasks.ceph:[mds] mds_dir_prefetch = False 2023-10-30T12:27:45.417 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2023-10-30T12:27:45.417 INFO:tasks.ceph:[mds] mon client directed command retry = 5 2023-10-30T12:27:45.417 INFO:tasks.ceph:[mds] ms inject delay max = 1 2023-10-30T12:27:45.417 INFO:tasks.ceph:[mds] ms inject delay probability = 0.005 2023-10-30T12:27:45.418 INFO:tasks.ceph:[mds] ms inject delay type = client mds osd 2023-10-30T12:27:45.418 INFO:tasks.ceph:[mds] ms inject socket failures = 2500 2023-10-30T12:27:45.418 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-10-30T12:27:45.418 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-10-30T12:27:45.419 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-10-30T12:27:45.419 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-30T12:27:45.419 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-30T12:27:45.419 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-30T12:27:45.420 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-30T12:27:45.420 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-30T12:27:45.420 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-10-30T12:27:45.420 INFO:tasks.ceph:[osd] bdev async discard = True 2023-10-30T12:27:45.421 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-10-30T12:27:45.421 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-10-30T12:27:45.421 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-30T12:27:45.421 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-30T12:27:45.422 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-10-30T12:27:45.422 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-10-30T12:27:45.422 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-30T12:27:45.422 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-30T12:27:45.422 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-10-30T12:27:45.423 INFO:tasks.ceph:[osd] mon client directed command retry = 5 2023-10-30T12:27:45.423 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-30T12:27:45.423 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-30T12:27:45.423 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-30T12:27:45.424 INFO:tasks.ceph:[osd] ms inject delay max = 1 2023-10-30T12:27:45.424 INFO:tasks.ceph:[osd] ms inject delay probability = 0.005 2023-10-30T12:27:45.424 INFO:tasks.ceph:[osd] ms inject delay type = client mds 2023-10-30T12:27:45.424 INFO:tasks.ceph:[osd] ms inject socket failures = 2500 2023-10-30T12:27:45.424 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2023-10-30T12:27:45.425 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-30T12:27:45.425 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-10-30T12:27:45.425 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-30T12:27:45.426 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-10-30T12:27:45.426 INFO:tasks.ceph:Setting up mon.a... 2023-10-30T12:27:45.427 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-30T12:27:45.472 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/ceph.keyring 2023-10-30T12:27:45.488 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-30T12:27:45.583 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T12:27:45.604 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.89'), ('mon.b', '172.21.15.192'), ('mon.c', '[v2:172.21.15.192:3301,v1:172.21.15.192:6790]')] 2023-10-30T12:27:45.605 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.89,172.21.15.192,[v2:172.21.15.192:3301,v1:172.21.15.192:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-10-30T12:27:45.606 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-10-30T12:27:45.606 DEBUG:teuthology.orchestra.run.smithi089:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-30T12:27:45.654 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.89 --add b 172.21.15.192 --addv c '[v2:172.21.15.192:3301,v1:172.21.15.192:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:27:45.745 INFO:teuthology.orchestra.run.smithi089.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:27:45.745 INFO:teuthology.orchestra.run.smithi089.stdout:monmaptool: generated fsid 401641be-db6e-4c16-b3ac-fd84a86c4de4 2023-10-30T12:27:45.746 INFO:teuthology.orchestra.run.smithi089.stdout:setting min_mon_release = pacific 2023-10-30T12:27:45.746 INFO:teuthology.orchestra.run.smithi089.stdout:epoch 0 2023-10-30T12:27:45.746 INFO:teuthology.orchestra.run.smithi089.stdout:fsid 401641be-db6e-4c16-b3ac-fd84a86c4de4 2023-10-30T12:27:45.746 INFO:teuthology.orchestra.run.smithi089.stdout:last_changed 2023-10-30T12:27:45.746729+0000 2023-10-30T12:27:45.747 INFO:teuthology.orchestra.run.smithi089.stdout:created 2023-10-30T12:27:45.746729+0000 2023-10-30T12:27:45.747 INFO:teuthology.orchestra.run.smithi089.stdout:min_mon_release 16 (pacific) 2023-10-30T12:27:45.747 INFO:teuthology.orchestra.run.smithi089.stdout:election_strategy: 1 2023-10-30T12:27:45.747 INFO:teuthology.orchestra.run.smithi089.stdout:0: [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] mon.a 2023-10-30T12:27:45.747 INFO:teuthology.orchestra.run.smithi089.stdout:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-10-30T12:27:45.747 INFO:teuthology.orchestra.run.smithi089.stdout:2: [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] mon.c 2023-10-30T12:27:45.748 INFO:teuthology.orchestra.run.smithi089.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-10-30T12:27:45.749 DEBUG:teuthology.orchestra.run.smithi089:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-30T12:27:45.757 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 401641be-db6e-4c16-b3ac-fd84a86c4de4... 2023-10-30T12:27:45.758 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-30T12:27:45.800 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-30T12:27:45.831 INFO:teuthology.orchestra.run.smithi192.stdout:[global] 2023-10-30T12:27:45.831 INFO:teuthology.orchestra.run.smithi192.stdout: chdir = "" 2023-10-30T12:27:45.831 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-30T12:27:45.831 INFO:teuthology.orchestra.run.smithi192.stdout: auth supported = cephx 2023-10-30T12:27:45.831 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:45.832 INFO:teuthology.orchestra.run.smithi192.stdout: filestore xattr use omap = true 2023-10-30T12:27:45.832 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:45.832 INFO:teuthology.orchestra.run.smithi192.stdout: mon clock drift allowed = 1.000 2023-10-30T12:27:45.832 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:45.832 INFO:teuthology.orchestra.run.smithi192.stdout: osd crush chooseleaf type = 0 2023-10-30T12:27:45.833 INFO:teuthology.orchestra.run.smithi192.stdout: auth debug = true 2023-10-30T12:27:45.833 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:45.833 INFO:teuthology.orchestra.run.smithi192.stdout: ms die on old message = true 2023-10-30T12:27:45.833 INFO:teuthology.orchestra.run.smithi192.stdout: ms die on bug = true 2023-10-30T12:27:45.833 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:45.833 INFO:teuthology.orchestra.run.smithi192.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-30T12:27:45.834 INFO:teuthology.orchestra.run.smithi192.stdout: mon pg warn max object skew = 0 2023-10-30T12:27:45.834 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:45.834 INFO:teuthology.orchestra.run.smithi192.stdout: # disable pg_autoscaler by default for new pools 2023-10-30T12:27:45.834 INFO:teuthology.orchestra.run.smithi192.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-30T12:27:45.834 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:45.835 INFO:teuthology.orchestra.run.smithi192.stdout: osd pool default size = 2 2023-10-30T12:27:45.835 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:45.835 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd allow primary affinity = true 2023-10-30T12:27:45.835 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd allow pg remap = true 2023-10-30T12:27:45.835 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on legacy crush tunables = false 2023-10-30T12:27:45.836 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on crush straw calc version zero = false 2023-10-30T12:27:45.836 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on no sortbitwise = false 2023-10-30T12:27:45.836 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on osd down out interval zero = false 2023-10-30T12:27:45.836 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on too few osds = false 2023-10-30T12:27:45.836 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-30T12:27:45.837 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-30T12:27:45.837 INFO:teuthology.orchestra.run.smithi192.stdout: mon_allow_pool_size_one = true 2023-10-30T12:27:45.837 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:45.837 INFO:teuthology.orchestra.run.smithi192.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-30T12:27:45.837 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:45.837 INFO:teuthology.orchestra.run.smithi192.stdout: osd default data pool replay window = 5 2023-10-30T12:27:45.838 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:45.838 INFO:teuthology.orchestra.run.smithi192.stdout: mon allow pool delete = true 2023-10-30T12:27:45.838 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:45.838 INFO:teuthology.orchestra.run.smithi192.stdout: mon cluster log file level = debug 2023-10-30T12:27:45.838 INFO:teuthology.orchestra.run.smithi192.stdout: debug asserts on shutdown = true 2023-10-30T12:27:45.839 INFO:teuthology.orchestra.run.smithi192.stdout: mon health detail to clog = false 2023-10-30T12:27:45.839 INFO:teuthology.orchestra.run.smithi192.stdout: mon host = "172.21.15.89,172.21.15.192,[v2:172.21.15.192:3301,v1:172.21.15.192:6790]" 2023-10-30T12:27:45.839 INFO:teuthology.orchestra.run.smithi192.stdout: ms die on skipped message = False 2023-10-30T12:27:45.839 INFO:teuthology.orchestra.run.smithi192.stdout: fsid = 401641be-db6e-4c16-b3ac-fd84a86c4de4 2023-10-30T12:27:45.839 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:45.839 INFO:teuthology.orchestra.run.smithi192.stdout:[osd] 2023-10-30T12:27:45.840 INFO:teuthology.orchestra.run.smithi192.stdout: osd journal size = 100 2023-10-30T12:27:45.840 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:45.840 INFO:teuthology.orchestra.run.smithi192.stdout: osd scrub load threshold = 5.0 2023-10-30T12:27:45.840 INFO:teuthology.orchestra.run.smithi192.stdout: osd scrub max interval = 600 2023-10-30T12:27:45.840 INFO:teuthology.orchestra.run.smithi192.stdout: osd mclock profile = high_recovery_ops 2023-10-30T12:27:45.841 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:45.841 INFO:teuthology.orchestra.run.smithi192.stdout: osd recover clone overlap = true 2023-10-30T12:27:45.841 INFO:teuthology.orchestra.run.smithi192.stdout: osd recovery max chunk = 1048576 2023-10-30T12:27:45.841 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:45.841 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug shutdown = true 2023-10-30T12:27:45.842 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug op order = true 2023-10-30T12:27:45.842 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug verify stray on activate = true 2023-10-30T12:27:45.842 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:45.842 INFO:teuthology.orchestra.run.smithi192.stdout: osd open classes on start = true 2023-10-30T12:27:45.842 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug pg log writeout = true 2023-10-30T12:27:45.842 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:45.843 INFO:teuthology.orchestra.run.smithi192.stdout: osd deep scrub update digest min age = 30 2023-10-30T12:27:45.843 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:45.843 INFO:teuthology.orchestra.run.smithi192.stdout: osd map max advance = 10 2023-10-30T12:27:45.843 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:45.843 INFO:teuthology.orchestra.run.smithi192.stdout: journal zero on create = true 2023-10-30T12:27:45.844 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:45.844 INFO:teuthology.orchestra.run.smithi192.stdout: filestore ondisk finisher threads = 3 2023-10-30T12:27:45.844 INFO:teuthology.orchestra.run.smithi192.stdout: filestore apply finisher threads = 3 2023-10-30T12:27:45.844 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:45.844 INFO:teuthology.orchestra.run.smithi192.stdout: bdev debug aio = true 2023-10-30T12:27:45.844 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug misdirected ops = true 2023-10-30T12:27:45.845 INFO:teuthology.orchestra.run.smithi192.stdout: bdev async discard = True 2023-10-30T12:27:45.845 INFO:teuthology.orchestra.run.smithi192.stdout: bdev enable discard = True 2023-10-30T12:27:45.845 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore allocator = bitmap 2023-10-30T12:27:45.845 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore block size = 96636764160 2023-10-30T12:27:45.845 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore fsck on mount = True 2023-10-30T12:27:45.846 INFO:teuthology.orchestra.run.smithi192.stdout: debug bluefs = 1/20 2023-10-30T12:27:45.846 INFO:teuthology.orchestra.run.smithi192.stdout: debug bluestore = 1/20 2023-10-30T12:27:45.846 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2023-10-30T12:27:45.846 INFO:teuthology.orchestra.run.smithi192.stdout: debug osd = 20 2023-10-30T12:27:45.846 INFO:teuthology.orchestra.run.smithi192.stdout: debug rocksdb = 4/10 2023-10-30T12:27:45.846 INFO:teuthology.orchestra.run.smithi192.stdout: mon client directed command retry = 5 2023-10-30T12:27:45.847 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd backfillfull_ratio = 0.85 2023-10-30T12:27:45.847 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd full ratio = 0.9 2023-10-30T12:27:45.847 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd nearfull ratio = 0.8 2023-10-30T12:27:45.847 INFO:teuthology.orchestra.run.smithi192.stdout: ms inject delay max = 1 2023-10-30T12:27:45.847 INFO:teuthology.orchestra.run.smithi192.stdout: ms inject delay probability = 0.005 2023-10-30T12:27:45.848 INFO:teuthology.orchestra.run.smithi192.stdout: ms inject delay type = client mds 2023-10-30T12:27:45.848 INFO:teuthology.orchestra.run.smithi192.stdout: ms inject socket failures = 2500 2023-10-30T12:27:45.848 INFO:teuthology.orchestra.run.smithi192.stdout: osd blocked scrub grace period = 3600 2023-10-30T12:27:45.848 INFO:teuthology.orchestra.run.smithi192.stdout: osd failsafe full ratio = 0.95 2023-10-30T12:27:45.848 INFO:teuthology.orchestra.run.smithi192.stdout: osd max markdown count = 1000 2023-10-30T12:27:45.848 INFO:teuthology.orchestra.run.smithi192.stdout: osd objectstore = bluestore 2023-10-30T12:27:45.849 INFO:teuthology.orchestra.run.smithi192.stdout: osd op complaint time = 180 2023-10-30T12:27:45.849 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:45.849 INFO:teuthology.orchestra.run.smithi192.stdout:[mgr] 2023-10-30T12:27:45.849 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2023-10-30T12:27:45.849 INFO:teuthology.orchestra.run.smithi192.stdout: debug mgr = 20 2023-10-30T12:27:45.850 INFO:teuthology.orchestra.run.smithi192.stdout: debug mon = 20 2023-10-30T12:27:45.850 INFO:teuthology.orchestra.run.smithi192.stdout: debug auth = 20 2023-10-30T12:27:45.850 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min pgs per osd = 4 2023-10-30T12:27:45.850 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min bytes per osd = 10 2023-10-30T12:27:45.850 INFO:teuthology.orchestra.run.smithi192.stdout: mgr/telemetry/nag = false 2023-10-30T12:27:45.851 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:45.851 INFO:teuthology.orchestra.run.smithi192.stdout:[mon] 2023-10-30T12:27:45.851 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2023-10-30T12:27:45.851 INFO:teuthology.orchestra.run.smithi192.stdout: debug mon = 20 2023-10-30T12:27:45.851 INFO:teuthology.orchestra.run.smithi192.stdout: debug paxos = 20 2023-10-30T12:27:45.851 INFO:teuthology.orchestra.run.smithi192.stdout: debug auth = 20 2023-10-30T12:27:45.852 INFO:teuthology.orchestra.run.smithi192.stdout: mon data avail warn = 5 2023-10-30T12:27:45.852 INFO:teuthology.orchestra.run.smithi192.stdout: mon mgr mkfs grace = 240 2023-10-30T12:27:45.852 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min pgs per osd = 4 2023-10-30T12:27:45.852 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd reporter subtree level = osd 2023-10-30T12:27:45.852 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd prime pg temp = true 2023-10-30T12:27:45.853 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min bytes per osd = 10 2023-10-30T12:27:45.853 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:45.853 INFO:teuthology.orchestra.run.smithi192.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-30T12:27:45.853 INFO:teuthology.orchestra.run.smithi192.stdout: auth mon ticket ttl = 660 # 11m 2023-10-30T12:27:45.853 INFO:teuthology.orchestra.run.smithi192.stdout: auth service ticket ttl = 240 # 4m 2023-10-30T12:27:45.853 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:45.854 INFO:teuthology.orchestra.run.smithi192.stdout: # don't complain about insecure global_id in the test suite 2023-10-30T12:27:45.854 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-30T12:27:45.854 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-30T12:27:45.854 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:45.854 INFO:teuthology.orchestra.run.smithi192.stdout: # 1m isn't quite enough 2023-10-30T12:27:45.855 INFO:teuthology.orchestra.run.smithi192.stdout: mon_down_mkfs_grace = 2m 2023-10-30T12:27:45.855 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:45.855 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_filestore_osds = false 2023-10-30T12:27:45.855 INFO:teuthology.orchestra.run.smithi192.stdout: mon op complaint time = 120 2023-10-30T12:27:45.855 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-30T12:27:45.856 INFO:teuthology.orchestra.run.smithi192.stdout:[client] 2023-10-30T12:27:45.856 INFO:teuthology.orchestra.run.smithi192.stdout: rgw cache enabled = true 2023-10-30T12:27:45.856 INFO:teuthology.orchestra.run.smithi192.stdout: rgw enable ops log = true 2023-10-30T12:27:45.856 INFO:teuthology.orchestra.run.smithi192.stdout: rgw enable usage log = true 2023-10-30T12:27:45.856 INFO:teuthology.orchestra.run.smithi192.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-30T12:27:45.856 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-30T12:27:45.857 INFO:teuthology.orchestra.run.smithi192.stdout: client mount timeout = 600 2023-10-30T12:27:45.857 INFO:teuthology.orchestra.run.smithi192.stdout: client_shutdown_timeout = 120 2023-10-30T12:27:45.857 INFO:teuthology.orchestra.run.smithi192.stdout: debug client = 20 2023-10-30T12:27:45.857 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2023-10-30T12:27:45.857 INFO:teuthology.orchestra.run.smithi192.stdout: rados mon op timeout = 900 2023-10-30T12:27:45.857 INFO:teuthology.orchestra.run.smithi192.stdout: rados osd op timeout = 900 2023-10-30T12:27:45.858 INFO:teuthology.orchestra.run.smithi192.stdout:[mon.a] 2023-10-30T12:27:45.858 INFO:teuthology.orchestra.run.smithi192.stdout:[mon.b] 2023-10-30T12:27:45.858 INFO:teuthology.orchestra.run.smithi192.stdout:[mon.c] 2023-10-30T12:27:45.858 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.a] 2023-10-30T12:27:45.858 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.c] 2023-10-30T12:27:45.859 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.e] 2023-10-30T12:27:45.859 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.b] 2023-10-30T12:27:45.859 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.d] 2023-10-30T12:27:45.859 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.f] 2023-10-30T12:27:45.859 INFO:teuthology.orchestra.run.smithi192.stdout:[mds] 2023-10-30T12:27:45.859 INFO:teuthology.orchestra.run.smithi192.stdout: debug mds = 20 2023-10-30T12:27:45.860 INFO:teuthology.orchestra.run.smithi192.stdout: debug mds balancer = 20 2023-10-30T12:27:45.860 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2023-10-30T12:27:45.860 INFO:teuthology.orchestra.run.smithi192.stdout: mds bal fragment size max = 10000 2023-10-30T12:27:45.860 INFO:teuthology.orchestra.run.smithi192.stdout: mds bal merge size = 5 2023-10-30T12:27:45.860 INFO:teuthology.orchestra.run.smithi192.stdout: mds bal split bits = 3 2023-10-30T12:27:45.860 INFO:teuthology.orchestra.run.smithi192.stdout: mds bal split size = 100 2023-10-30T12:27:45.861 INFO:teuthology.orchestra.run.smithi192.stdout: mds debug frag = True 2023-10-30T12:27:45.861 INFO:teuthology.orchestra.run.smithi192.stdout: mds debug scatterstat = True 2023-10-30T12:27:45.861 INFO:teuthology.orchestra.run.smithi192.stdout: mds oft prefetch dirfrags = True 2023-10-30T12:27:45.861 INFO:teuthology.orchestra.run.smithi192.stdout: mds op complaint time = 180 2023-10-30T12:27:45.861 INFO:teuthology.orchestra.run.smithi192.stdout: mds verify scatter = True 2023-10-30T12:27:45.861 INFO:teuthology.orchestra.run.smithi192.stdout: mds_dir_prefetch = False 2023-10-30T12:27:45.862 INFO:teuthology.orchestra.run.smithi192.stdout: mds_sleep_rank_change = 5000000.0 2023-10-30T12:27:45.862 INFO:teuthology.orchestra.run.smithi192.stdout: mon client directed command retry = 5 2023-10-30T12:27:45.862 INFO:teuthology.orchestra.run.smithi192.stdout: ms inject delay max = 1 2023-10-30T12:27:45.862 INFO:teuthology.orchestra.run.smithi192.stdout: ms inject delay probability = 0.005 2023-10-30T12:27:45.862 INFO:teuthology.orchestra.run.smithi192.stdout: ms inject delay type = client mds osd 2023-10-30T12:27:45.863 INFO:teuthology.orchestra.run.smithi192.stdout: ms inject socket failures = 2500 2023-10-30T12:27:45.863 INFO:teuthology.orchestra.run.smithi192.stdout: osd op complaint time = 180 2023-10-30T12:27:45.863 INFO:teuthology.orchestra.run.smithi192.stdout: rados mon op timeout = 900 2023-10-30T12:27:45.863 INFO:teuthology.orchestra.run.smithi192.stdout: rados osd op timeout = 900 2023-10-30T12:27:45.864 INFO:teuthology.orchestra.run.smithi089.stdout:[global] 2023-10-30T12:27:45.865 INFO:teuthology.orchestra.run.smithi089.stdout: chdir = "" 2023-10-30T12:27:45.865 INFO:teuthology.orchestra.run.smithi089.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-30T12:27:45.865 INFO:teuthology.orchestra.run.smithi089.stdout: auth supported = cephx 2023-10-30T12:27:45.865 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:45.865 INFO:teuthology.orchestra.run.smithi089.stdout: filestore xattr use omap = true 2023-10-30T12:27:45.866 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:45.866 INFO:teuthology.orchestra.run.smithi089.stdout: mon clock drift allowed = 1.000 2023-10-30T12:27:45.866 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:45.866 INFO:teuthology.orchestra.run.smithi089.stdout: osd crush chooseleaf type = 0 2023-10-30T12:27:45.866 INFO:teuthology.orchestra.run.smithi089.stdout: auth debug = true 2023-10-30T12:27:45.867 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:45.867 INFO:teuthology.orchestra.run.smithi089.stdout: ms die on old message = true 2023-10-30T12:27:45.867 INFO:teuthology.orchestra.run.smithi089.stdout: ms die on bug = true 2023-10-30T12:27:45.867 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:45.867 INFO:teuthology.orchestra.run.smithi089.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-30T12:27:45.868 INFO:teuthology.orchestra.run.smithi089.stdout: mon pg warn max object skew = 0 2023-10-30T12:27:45.868 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:45.868 INFO:teuthology.orchestra.run.smithi089.stdout: # disable pg_autoscaler by default for new pools 2023-10-30T12:27:45.868 INFO:teuthology.orchestra.run.smithi089.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-30T12:27:45.868 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:45.868 INFO:teuthology.orchestra.run.smithi089.stdout: osd pool default size = 2 2023-10-30T12:27:45.869 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:45.869 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd allow primary affinity = true 2023-10-30T12:27:45.869 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd allow pg remap = true 2023-10-30T12:27:45.869 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on legacy crush tunables = false 2023-10-30T12:27:45.869 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on crush straw calc version zero = false 2023-10-30T12:27:45.870 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on no sortbitwise = false 2023-10-30T12:27:45.870 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on osd down out interval zero = false 2023-10-30T12:27:45.870 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on too few osds = false 2023-10-30T12:27:45.870 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-30T12:27:45.870 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-30T12:27:45.870 INFO:teuthology.orchestra.run.smithi089.stdout: mon_allow_pool_size_one = true 2023-10-30T12:27:45.871 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:45.871 INFO:teuthology.orchestra.run.smithi089.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-30T12:27:45.871 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:45.871 INFO:teuthology.orchestra.run.smithi089.stdout: osd default data pool replay window = 5 2023-10-30T12:27:45.871 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:45.872 INFO:teuthology.orchestra.run.smithi089.stdout: mon allow pool delete = true 2023-10-30T12:27:45.872 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:45.872 INFO:teuthology.orchestra.run.smithi089.stdout: mon cluster log file level = debug 2023-10-30T12:27:45.872 INFO:teuthology.orchestra.run.smithi089.stdout: debug asserts on shutdown = true 2023-10-30T12:27:45.872 INFO:teuthology.orchestra.run.smithi089.stdout: mon health detail to clog = false 2023-10-30T12:27:45.873 INFO:teuthology.orchestra.run.smithi089.stdout: mon host = "172.21.15.89,172.21.15.192,[v2:172.21.15.192:3301,v1:172.21.15.192:6790]" 2023-10-30T12:27:45.873 INFO:teuthology.orchestra.run.smithi089.stdout: ms die on skipped message = False 2023-10-30T12:27:45.873 INFO:teuthology.orchestra.run.smithi089.stdout: fsid = 401641be-db6e-4c16-b3ac-fd84a86c4de4 2023-10-30T12:27:45.873 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:45.873 INFO:teuthology.orchestra.run.smithi089.stdout:[osd] 2023-10-30T12:27:45.873 INFO:teuthology.orchestra.run.smithi089.stdout: osd journal size = 100 2023-10-30T12:27:45.874 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:45.874 INFO:teuthology.orchestra.run.smithi089.stdout: osd scrub load threshold = 5.0 2023-10-30T12:27:45.874 INFO:teuthology.orchestra.run.smithi089.stdout: osd scrub max interval = 600 2023-10-30T12:27:45.874 INFO:teuthology.orchestra.run.smithi089.stdout: osd mclock profile = high_recovery_ops 2023-10-30T12:27:45.874 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:45.875 INFO:teuthology.orchestra.run.smithi089.stdout: osd recover clone overlap = true 2023-10-30T12:27:45.875 INFO:teuthology.orchestra.run.smithi089.stdout: osd recovery max chunk = 1048576 2023-10-30T12:27:45.875 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:45.875 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug shutdown = true 2023-10-30T12:27:45.875 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug op order = true 2023-10-30T12:27:45.876 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug verify stray on activate = true 2023-10-30T12:27:45.876 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:45.876 INFO:teuthology.orchestra.run.smithi089.stdout: osd open classes on start = true 2023-10-30T12:27:45.876 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug pg log writeout = true 2023-10-30T12:27:45.876 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:45.876 INFO:teuthology.orchestra.run.smithi089.stdout: osd deep scrub update digest min age = 30 2023-10-30T12:27:45.877 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:45.877 INFO:teuthology.orchestra.run.smithi089.stdout: osd map max advance = 10 2023-10-30T12:27:45.877 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:45.877 INFO:teuthology.orchestra.run.smithi089.stdout: journal zero on create = true 2023-10-30T12:27:45.877 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:45.877 INFO:teuthology.orchestra.run.smithi089.stdout: filestore ondisk finisher threads = 3 2023-10-30T12:27:45.878 INFO:teuthology.orchestra.run.smithi089.stdout: filestore apply finisher threads = 3 2023-10-30T12:27:45.878 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:45.878 INFO:teuthology.orchestra.run.smithi089.stdout: bdev debug aio = true 2023-10-30T12:27:45.878 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug misdirected ops = true 2023-10-30T12:27:45.878 INFO:teuthology.orchestra.run.smithi089.stdout: bdev async discard = True 2023-10-30T12:27:45.879 INFO:teuthology.orchestra.run.smithi089.stdout: bdev enable discard = True 2023-10-30T12:27:45.879 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore allocator = bitmap 2023-10-30T12:27:45.879 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore block size = 96636764160 2023-10-30T12:27:45.879 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore fsck on mount = True 2023-10-30T12:27:45.879 INFO:teuthology.orchestra.run.smithi089.stdout: debug bluefs = 1/20 2023-10-30T12:27:45.880 INFO:teuthology.orchestra.run.smithi089.stdout: debug bluestore = 1/20 2023-10-30T12:27:45.880 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2023-10-30T12:27:45.880 INFO:teuthology.orchestra.run.smithi089.stdout: debug osd = 20 2023-10-30T12:27:45.880 INFO:teuthology.orchestra.run.smithi089.stdout: debug rocksdb = 4/10 2023-10-30T12:27:45.880 INFO:teuthology.orchestra.run.smithi089.stdout: mon client directed command retry = 5 2023-10-30T12:27:45.880 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd backfillfull_ratio = 0.85 2023-10-30T12:27:45.881 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd full ratio = 0.9 2023-10-30T12:27:45.881 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd nearfull ratio = 0.8 2023-10-30T12:27:45.881 INFO:teuthology.orchestra.run.smithi089.stdout: ms inject delay max = 1 2023-10-30T12:27:45.881 INFO:teuthology.orchestra.run.smithi089.stdout: ms inject delay probability = 0.005 2023-10-30T12:27:45.881 INFO:teuthology.orchestra.run.smithi089.stdout: ms inject delay type = client mds 2023-10-30T12:27:45.882 INFO:teuthology.orchestra.run.smithi089.stdout: ms inject socket failures = 2500 2023-10-30T12:27:45.882 INFO:teuthology.orchestra.run.smithi089.stdout: osd blocked scrub grace period = 3600 2023-10-30T12:27:45.882 INFO:teuthology.orchestra.run.smithi089.stdout: osd failsafe full ratio = 0.95 2023-10-30T12:27:45.882 INFO:teuthology.orchestra.run.smithi089.stdout: osd max markdown count = 1000 2023-10-30T12:27:45.882 INFO:teuthology.orchestra.run.smithi089.stdout: osd objectstore = bluestore 2023-10-30T12:27:45.883 INFO:teuthology.orchestra.run.smithi089.stdout: osd op complaint time = 180 2023-10-30T12:27:45.883 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:45.883 INFO:teuthology.orchestra.run.smithi089.stdout:[mgr] 2023-10-30T12:27:45.883 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2023-10-30T12:27:45.883 INFO:teuthology.orchestra.run.smithi089.stdout: debug mgr = 20 2023-10-30T12:27:45.883 INFO:teuthology.orchestra.run.smithi089.stdout: debug mon = 20 2023-10-30T12:27:45.884 INFO:teuthology.orchestra.run.smithi089.stdout: debug auth = 20 2023-10-30T12:27:45.884 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min pgs per osd = 4 2023-10-30T12:27:45.884 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min bytes per osd = 10 2023-10-30T12:27:45.884 INFO:teuthology.orchestra.run.smithi089.stdout: mgr/telemetry/nag = false 2023-10-30T12:27:45.884 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:45.885 INFO:teuthology.orchestra.run.smithi089.stdout:[mon] 2023-10-30T12:27:45.885 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2023-10-30T12:27:45.885 INFO:teuthology.orchestra.run.smithi089.stdout: debug mon = 20 2023-10-30T12:27:45.885 INFO:teuthology.orchestra.run.smithi089.stdout: debug paxos = 20 2023-10-30T12:27:45.885 INFO:teuthology.orchestra.run.smithi089.stdout: debug auth = 20 2023-10-30T12:27:45.886 INFO:teuthology.orchestra.run.smithi089.stdout: mon data avail warn = 5 2023-10-30T12:27:45.886 INFO:teuthology.orchestra.run.smithi089.stdout: mon mgr mkfs grace = 240 2023-10-30T12:27:45.886 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min pgs per osd = 4 2023-10-30T12:27:45.886 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd reporter subtree level = osd 2023-10-30T12:27:45.886 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd prime pg temp = true 2023-10-30T12:27:45.890 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min bytes per osd = 10 2023-10-30T12:27:45.891 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:45.891 INFO:teuthology.orchestra.run.smithi089.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-30T12:27:45.891 INFO:teuthology.orchestra.run.smithi089.stdout: auth mon ticket ttl = 660 # 11m 2023-10-30T12:27:45.891 INFO:teuthology.orchestra.run.smithi089.stdout: auth service ticket ttl = 240 # 4m 2023-10-30T12:27:45.891 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:45.892 INFO:teuthology.orchestra.run.smithi089.stdout: # don't complain about insecure global_id in the test suite 2023-10-30T12:27:45.892 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-30T12:27:45.892 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-30T12:27:45.892 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:45.892 INFO:teuthology.orchestra.run.smithi089.stdout: # 1m isn't quite enough 2023-10-30T12:27:45.893 INFO:teuthology.orchestra.run.smithi089.stdout: mon_down_mkfs_grace = 2m 2023-10-30T12:27:45.893 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:45.893 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_filestore_osds = false 2023-10-30T12:27:45.893 INFO:teuthology.orchestra.run.smithi089.stdout: mon op complaint time = 120 2023-10-30T12:27:45.893 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:27:45.893 INFO:teuthology.orchestra.run.smithi089.stdout:[client] 2023-10-30T12:27:45.894 INFO:teuthology.orchestra.run.smithi089.stdout: rgw cache enabled = true 2023-10-30T12:27:45.894 INFO:teuthology.orchestra.run.smithi089.stdout: rgw enable ops log = true 2023-10-30T12:27:45.894 INFO:teuthology.orchestra.run.smithi089.stdout: rgw enable usage log = true 2023-10-30T12:27:45.894 INFO:teuthology.orchestra.run.smithi089.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-30T12:27:45.894 INFO:teuthology.orchestra.run.smithi089.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-30T12:27:45.895 INFO:teuthology.orchestra.run.smithi089.stdout: client mount timeout = 600 2023-10-30T12:27:45.895 INFO:teuthology.orchestra.run.smithi089.stdout: client_shutdown_timeout = 120 2023-10-30T12:27:45.895 INFO:teuthology.orchestra.run.smithi089.stdout: debug client = 20 2023-10-30T12:27:45.895 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2023-10-30T12:27:45.895 INFO:teuthology.orchestra.run.smithi089.stdout: rados mon op timeout = 900 2023-10-30T12:27:45.896 INFO:teuthology.orchestra.run.smithi089.stdout: rados osd op timeout = 900 2023-10-30T12:27:45.896 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.a] 2023-10-30T12:27:45.896 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.b] 2023-10-30T12:27:45.896 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.c] 2023-10-30T12:27:45.896 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.a] 2023-10-30T12:27:45.896 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.c] 2023-10-30T12:27:45.897 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.e] 2023-10-30T12:27:45.897 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.b] 2023-10-30T12:27:45.897 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.d] 2023-10-30T12:27:45.897 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.f] 2023-10-30T12:27:45.898 INFO:teuthology.orchestra.run.smithi089.stdout:[mds] 2023-10-30T12:27:45.898 INFO:teuthology.orchestra.run.smithi089.stdout: debug mds = 20 2023-10-30T12:27:45.898 INFO:teuthology.orchestra.run.smithi089.stdout: debug mds balancer = 20 2023-10-30T12:27:45.898 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2023-10-30T12:27:45.898 INFO:teuthology.orchestra.run.smithi089.stdout: mds bal fragment size max = 10000 2023-10-30T12:27:45.899 INFO:teuthology.orchestra.run.smithi089.stdout: mds bal merge size = 5 2023-10-30T12:27:45.899 INFO:teuthology.orchestra.run.smithi089.stdout: mds bal split bits = 3 2023-10-30T12:27:45.899 INFO:teuthology.orchestra.run.smithi089.stdout: mds bal split size = 100 2023-10-30T12:27:45.899 INFO:teuthology.orchestra.run.smithi089.stdout: mds debug frag = True 2023-10-30T12:27:45.899 INFO:teuthology.orchestra.run.smithi089.stdout: mds debug scatterstat = True 2023-10-30T12:27:45.899 INFO:teuthology.orchestra.run.smithi089.stdout: mds oft prefetch dirfrags = True 2023-10-30T12:27:45.900 INFO:teuthology.orchestra.run.smithi089.stdout: mds op complaint time = 180 2023-10-30T12:27:45.900 INFO:teuthology.orchestra.run.smithi089.stdout: mds verify scatter = True 2023-10-30T12:27:45.900 INFO:teuthology.orchestra.run.smithi089.stdout: mds_dir_prefetch = False 2023-10-30T12:27:45.900 INFO:teuthology.orchestra.run.smithi089.stdout: mds_sleep_rank_change = 5000000.0 2023-10-30T12:27:45.900 INFO:teuthology.orchestra.run.smithi089.stdout: mon client directed command retry = 5 2023-10-30T12:27:45.900 INFO:teuthology.orchestra.run.smithi089.stdout: ms inject delay max = 1 2023-10-30T12:27:45.901 INFO:teuthology.orchestra.run.smithi089.stdout: ms inject delay probability = 0.005 2023-10-30T12:27:45.901 INFO:teuthology.orchestra.run.smithi089.stdout: ms inject delay type = client mds osd 2023-10-30T12:27:45.901 INFO:teuthology.orchestra.run.smithi089.stdout: ms inject socket failures = 2500 2023-10-30T12:27:45.901 INFO:teuthology.orchestra.run.smithi089.stdout: osd op complaint time = 180 2023-10-30T12:27:45.901 INFO:teuthology.orchestra.run.smithi089.stdout: rados mon op timeout = 900 2023-10-30T12:27:45.902 INFO:teuthology.orchestra.run.smithi089.stdout: rados osd op timeout = 900 2023-10-30T12:27:45.902 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-30T12:27:45.903 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-30T12:27:45.962 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-30T12:27:45.963 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-10-30T12:27:45.964 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-30T12:27:45.972 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-10-30T12:27:45.973 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-30T12:27:46.021 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi089.front.sepia.ceph.com 2023-10-30T12:27:46.022 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-10-30T12:27:46.022 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-30T12:27:46.022 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T12:27:46.093 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-10-30T12:27:46.093 DEBUG:teuthology.orchestra.run.smithi089:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:27:46.143 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi192.front.sepia.ceph.com 2023-10-30T12:27:46.144 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-30T12:27:46.144 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-30T12:27:46.144 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T12:27:46.167 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-30T12:27:46.168 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:27:46.216 INFO:tasks.ceph:Setting up mon nodes... 2023-10-30T12:27:46.216 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-30T12:27:46.217 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-10-30T12:27:46.272 INFO:teuthology.orchestra.run.smithi089.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-10-30T12:27:46.277 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-30T12:27:46.323 INFO:teuthology.orchestra.run.smithi192.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-30T12:27:46.339 INFO:tasks.ceph:Setting up mds nodes... 2023-10-30T12:27:46.340 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-10-30T12:27:46.396 INFO:teuthology.orchestra.run.smithi089.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-10-30T12:27:46.400 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-10-30T12:27:46.420 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-10-30T12:27:46.520 INFO:teuthology.orchestra.run.smithi089.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-10-30T12:27:46.524 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-10-30T12:27:46.545 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2023-10-30T12:27:46.644 INFO:teuthology.orchestra.run.smithi089.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2023-10-30T12:27:46.648 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2023-10-30T12:27:46.668 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-10-30T12:27:46.713 INFO:teuthology.orchestra.run.smithi192.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-10-30T12:27:46.716 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-10-30T12:27:46.771 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-10-30T12:27:46.854 INFO:teuthology.orchestra.run.smithi192.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-10-30T12:27:46.857 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-10-30T12:27:46.911 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2023-10-30T12:27:46.996 INFO:teuthology.orchestra.run.smithi192.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2023-10-30T12:27:46.998 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2023-10-30T12:27:47.050 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-30T12:27:47.051 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-30T12:27:47.094 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-30T12:27:47.111 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-30T12:27:47.112 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi089.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi192.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-30T12:27:47.112 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-30T12:27:47.177 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T12:27:47.178 INFO:tasks.ceph:role: osd.0 2023-10-30T12:27:47.178 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi089.front.sepia.ceph.com 2023-10-30T12:27:47.178 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-30T12:27:47.245 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:27:47.245 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:27:47.245 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:27:47.246 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T12:27:47.246 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:27:47.246 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:27:47.246 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:27:47.246 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T12:27:47.246 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:27:47.247 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:27:47.699 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2023-10-30T12:27:47.701 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2023-10-30T12:27:47.701 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-30T12:27:47.814 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-30T12:27:47.830 INFO:teuthology.orchestra.run.smithi089.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T12:27:47.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:27:47.832 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-30T12:27:47.893 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T12:27:47.893 INFO:tasks.ceph:role: osd.1 2023-10-30T12:27:47.894 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi089.front.sepia.ceph.com 2023-10-30T12:27:47.894 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-30T12:27:47.962 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:27:47.963 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:27:47.963 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:27:47.963 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T12:27:47.963 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:27:47.964 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:27:47.964 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:27:47.964 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T12:27:47.964 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:27:47.964 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:27:48.452 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2023-10-30T12:27:48.456 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2023-10-30T12:27:48.456 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-30T12:27:48.580 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-30T12:27:48.599 INFO:teuthology.orchestra.run.smithi089.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T12:27:48.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:27:48.600 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-30T12:27:48.663 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T12:27:48.663 INFO:tasks.ceph:role: osd.2 2023-10-30T12:27:48.664 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi089.front.sepia.ceph.com 2023-10-30T12:27:48.664 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-30T12:27:48.738 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:27:48.738 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:27:48.739 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:27:48.739 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T12:27:48.739 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:27:48.739 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:27:48.739 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:27:48.740 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T12:27:48.740 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:27:48.740 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:27:49.218 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2023-10-30T12:27:49.221 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2023-10-30T12:27:49.221 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-30T12:27:49.375 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-30T12:27:49.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:27:49.395 INFO:teuthology.orchestra.run.smithi089.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T12:27:49.396 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-10-30T12:27:49.459 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T12:27:49.459 INFO:tasks.ceph:role: osd.3 2023-10-30T12:27:49.459 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi089.front.sepia.ceph.com 2023-10-30T12:27:49.460 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-30T12:27:49.530 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:27:49.530 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:27:49.530 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:27:49.531 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T12:27:49.531 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:27:49.531 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:27:49.531 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:27:49.531 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T12:27:49.532 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:27:49.532 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:27:49.981 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2023-10-30T12:27:49.983 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2023-10-30T12:27:49.984 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-10-30T12:27:50.104 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-10-30T12:27:50.122 INFO:teuthology.orchestra.run.smithi089.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T12:27:50.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:27:50.124 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:27:50.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:27:50.259+0000 7fde1ab0b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-30T12:27:50.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:27:50.259+0000 7fde1ab0b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-30T12:27:50.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:27:50.259+0000 7fde1ab0b640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T12:27:50.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:27:50.259+0000 7fde1ab0b640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T12:27:50.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:27:50.259+0000 7fde1ab0b640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T12:27:50.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:27:50.263+0000 7fde1ab0b640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-30T12:27:53.254 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-30T12:27:53.276 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:27:53.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:27:53.371+0000 7f9362119640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-30T12:27:53.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:27:53.371+0000 7f9362119640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-30T12:27:53.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:27:53.371+0000 7f9362119640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T12:27:53.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:27:53.371+0000 7f9362119640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T12:27:53.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:27:53.371+0000 7f9362119640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T12:27:53.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:27:53.375+0000 7f9362119640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-30T12:27:56.365 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-30T12:27:56.387 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:27:56.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:27:56.483+0000 7f100b9e8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-30T12:27:56.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:27:56.483+0000 7f100b9e8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-30T12:27:56.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:27:56.483+0000 7f100b9e8640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T12:27:56.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:27:56.483+0000 7f100b9e8640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T12:27:56.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:27:56.483+0000 7f100b9e8640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T12:27:56.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:27:56.483+0000 7f100b9e8640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-30T12:27:59.461 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-30T12:27:59.483 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:27:59.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:27:59.583+0000 7fc2c4596640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-10-30T12:27:59.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:27:59.583+0000 7fc2c4596640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-10-30T12:27:59.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:27:59.583+0000 7fc2c4596640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T12:27:59.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:27:59.583+0000 7fc2c4596640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T12:27:59.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:27:59.583+0000 7fc2c4596640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T12:27:59.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:27:59.587+0000 7fc2c4596640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-10-30T12:28:02.565 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-10-30T12:28:02.586 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-10-30T12:28:02.601 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T12:28:02.601 INFO:tasks.ceph:role: osd.4 2023-10-30T12:28:02.602 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi192.front.sepia.ceph.com 2023-10-30T12:28:02.602 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-30T12:28:02.663 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:28:02.664 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:28:02.664 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:28:02.664 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T12:28:02.664 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:28:02.664 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:28:02.665 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:28:02.665 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T12:28:02.665 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:28:02.665 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:28:03.124 INFO:teuthology.orchestra.run.smithi192.stdout:Discarding blocks...Done. 2023-10-30T12:28:03.125 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2023-10-30T12:28:03.125 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-10-30T12:28:03.218 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-10-30T12:28:03.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:28:03.229 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T12:28:03.229 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-10-30T12:28:03.280 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T12:28:03.281 INFO:tasks.ceph:role: osd.5 2023-10-30T12:28:03.281 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi192.front.sepia.ceph.com 2023-10-30T12:28:03.281 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-30T12:28:03.345 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:28:03.345 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:28:03.345 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:28:03.346 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T12:28:03.346 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:28:03.346 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:28:03.346 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:28:03.346 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T12:28:03.347 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:28:03.347 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:28:03.810 INFO:teuthology.orchestra.run.smithi192.stdout:Discarding blocks...Done. 2023-10-30T12:28:03.812 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2023-10-30T12:28:03.812 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-10-30T12:28:03.933 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-10-30T12:28:03.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:28:03.945 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T12:28:03.946 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-10-30T12:28:03.999 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T12:28:03.999 INFO:tasks.ceph:role: osd.6 2023-10-30T12:28:04.000 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi192.front.sepia.ceph.com 2023-10-30T12:28:04.000 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-30T12:28:04.063 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:28:04.064 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:28:04.064 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:28:04.064 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T12:28:04.064 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:28:04.064 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:28:04.065 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:28:04.065 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T12:28:04.065 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:28:04.065 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:28:04.548 INFO:teuthology.orchestra.run.smithi192.stdout:Discarding blocks...Done. 2023-10-30T12:28:04.550 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2023-10-30T12:28:04.550 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-10-30T12:28:04.694 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-10-30T12:28:04.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:28:04.706 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T12:28:04.707 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-10-30T12:28:04.758 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T12:28:04.759 INFO:tasks.ceph:role: osd.7 2023-10-30T12:28:04.759 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi192.front.sepia.ceph.com 2023-10-30T12:28:04.759 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-30T12:28:04.824 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:28:04.824 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:28:04.824 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:28:04.824 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T12:28:04.824 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:28:04.825 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:28:04.825 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:28:04.825 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T12:28:04.825 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:28:04.825 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:28:05.244 INFO:teuthology.orchestra.run.smithi192.stdout:Discarding blocks...Done. 2023-10-30T12:28:05.246 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2023-10-30T12:28:05.246 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-10-30T12:28:05.329 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-10-30T12:28:05.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:28:05.381 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T12:28:05.381 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:28:05.515 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:05.509+0000 7f3ecad8d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-10-30T12:28:05.520 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:05.517+0000 7f3ecad8d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-10-30T12:28:05.520 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:05.517+0000 7f3ecad8d640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T12:28:05.521 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:05.517+0000 7f3ecad8d640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T12:28:05.521 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:05.517+0000 7f3ecad8d640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T12:28:05.523 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:05.517+0000 7f3ecad8d640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-10-30T12:28:08.532 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-10-30T12:28:08.547 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:28:08.629 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:08.625+0000 7f0d4fbe8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-10-30T12:28:08.631 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:08.625+0000 7f0d4fbe8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-10-30T12:28:08.631 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:08.625+0000 7f0d4fbe8640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T12:28:08.631 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:08.625+0000 7f0d4fbe8640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T12:28:08.632 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:08.625+0000 7f0d4fbe8640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T12:28:08.632 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:08.625+0000 7f0d4fbe8640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-10-30T12:28:11.686 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-10-30T12:28:11.743 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:28:11.829 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:11.825+0000 7f2044c3b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-10-30T12:28:11.830 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:11.825+0000 7f2044c3b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-10-30T12:28:11.830 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:11.825+0000 7f2044c3b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-30T12:28:11.830 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:11.825+0000 7f2044c3b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-30T12:28:11.830 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:11.825+0000 7f2044c3b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-30T12:28:11.830 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:11.825+0000 7f2044c3b640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-10-30T12:28:13.990 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:28:13.995 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:28:14.837 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-10-30T12:28:14.850 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:28:14.930 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:14.925+0000 7f6d623a9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-10-30T12:28:14.931 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:14.925+0000 7f6d623a9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-10-30T12:28:14.931 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:14.925+0000 7f6d623a9640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-30T12:28:14.932 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:14.925+0000 7f6d623a9640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-30T12:28:14.932 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:14.925+0000 7f6d623a9640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-30T12:28:14.932 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:14.925+0000 7f6d623a9640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-10-30T12:28:17.936 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-10-30T12:28:17.992 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-30T12:28:17.993 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-10-30T12:28:17.993 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-10-30T12:28:18.016 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-10-30T12:28:18.016 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-10-30T12:28:18.079 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-10-30T12:28:18.079 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-10-30T12:28:18.143 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-10-30T12:28:18.144 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2023-10-30T12:28:18.207 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-10-30T12:28:18.207 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-30T12:28:18.271 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-10-30T12:28:18.271 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-30T12:28:18.339 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-10-30T12:28:18.339 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-30T12:28:18.407 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-10-30T12:28:18.407 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-10-30T12:28:18.475 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-30T12:28:18.475 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-30T12:28:18.489 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-30T12:28:18.490 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-10-30T12:28:18.543 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-30T12:28:18.544 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-10-30T12:28:18.600 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-30T12:28:18.600 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2023-10-30T12:28:18.655 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-30T12:28:18.655 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-10-30T12:28:18.707 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-30T12:28:18.707 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-10-30T12:28:18.759 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-30T12:28:18.760 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-10-30T12:28:18.816 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-30T12:28:18.816 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-10-30T12:28:18.872 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-10-30T12:28:18.872 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-30T12:28:18.882 INFO:tasks.ceph:Adding keys to all mons... 2023-10-30T12:28:18.882 DEBUG:teuthology.orchestra.run.smithi089:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-30T12:28:18.933 DEBUG:teuthology.orchestra.run.smithi192:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-30T12:28:18.945 INFO:teuthology.orchestra.run.smithi192.stdout:[mgr.y] 2023-10-30T12:28:18.945 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQBCoT9ljKVREBAAptlCxxsBA6nQR4ZJi7xbFA== 2023-10-30T12:28:18.946 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.a] 2023-10-30T12:28:18.946 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQBCoT9lWr+0FxAA9T1y+z3r8HbYOhjP67hVrA== 2023-10-30T12:28:18.946 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.c] 2023-10-30T12:28:18.946 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQBCoT9lOF0gHxAA+YlSN6wrfNz3hNXa20y++A== 2023-10-30T12:28:18.947 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.e] 2023-10-30T12:28:18.947 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQBCoT9lnMB+JhAAJ0clcUGoukNnWG5ZyWJ7jQ== 2023-10-30T12:28:18.947 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.0] 2023-10-30T12:28:18.947 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQBGoT9lJnqNDxAAo37co2jgIxSy2rSeN8n+8Q== 2023-10-30T12:28:18.947 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.1] 2023-10-30T12:28:18.948 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQBJoT9l4yJYFhAATBeKdpEW58A3Bz10hqcj2g== 2023-10-30T12:28:18.948 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.2] 2023-10-30T12:28:18.948 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQBMoT9ljKLcHBAAtNQBrqkKb79JHjPLEj+bdw== 2023-10-30T12:28:18.948 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.3] 2023-10-30T12:28:18.948 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQBPoT9lM5PkIhAAF01gbhPX46l62NBys2qg7A== 2023-10-30T12:28:18.949 INFO:teuthology.orchestra.run.smithi192.stdout:[mgr.x] 2023-10-30T12:28:18.949 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQBCoT9lF7cbExAAc60QSSj/MiOQ45/51quoxw== 2023-10-30T12:28:18.949 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.b] 2023-10-30T12:28:18.949 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQBCoT9lfeWUKhAAcosNUlwtIndFw8jF/smatg== 2023-10-30T12:28:18.950 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.d] 2023-10-30T12:28:18.950 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQBCoT9liREBMxAAELu0uANMbBxs6ndIj2n5Mw== 2023-10-30T12:28:18.950 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.f] 2023-10-30T12:28:18.950 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQBCoT9leqxwOxAA6DeGLLqObT/B7ZanhKAgTg== 2023-10-30T12:28:18.950 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.4] 2023-10-30T12:28:18.951 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQBVoT9liP/LHhAAAirnVVabUUmIN5uHXCZDwQ== 2023-10-30T12:28:18.951 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.5] 2023-10-30T12:28:18.951 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQBYoT9lKJufJRAAPmVQwjXwM3CCLr1to69IAQ== 2023-10-30T12:28:18.951 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.6] 2023-10-30T12:28:18.951 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQBboT9lh55wMRAANryvv/ofmTbphZLbr5LKKQ== 2023-10-30T12:28:18.952 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.7] 2023-10-30T12:28:18.952 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQBeoT9l/yyENxAAyix2rm4OGZQB5hJp9FTwBQ== 2023-10-30T12:28:18.952 INFO:teuthology.orchestra.run.smithi192.stdout:[client.0] 2023-10-30T12:28:18.952 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQBDoT9lJY+6BRAAKnjh4CNZuC6Y9RgyY4BnQA== 2023-10-30T12:28:18.954 INFO:teuthology.orchestra.run.smithi089.stdout:[mgr.y] 2023-10-30T12:28:18.954 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQBCoT9ljKVREBAAptlCxxsBA6nQR4ZJi7xbFA== 2023-10-30T12:28:18.954 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.a] 2023-10-30T12:28:18.954 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQBCoT9lWr+0FxAA9T1y+z3r8HbYOhjP67hVrA== 2023-10-30T12:28:18.955 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.c] 2023-10-30T12:28:18.955 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQBCoT9lOF0gHxAA+YlSN6wrfNz3hNXa20y++A== 2023-10-30T12:28:18.955 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.e] 2023-10-30T12:28:18.955 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQBCoT9lnMB+JhAAJ0clcUGoukNnWG5ZyWJ7jQ== 2023-10-30T12:28:18.955 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.0] 2023-10-30T12:28:18.956 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQBGoT9lJnqNDxAAo37co2jgIxSy2rSeN8n+8Q== 2023-10-30T12:28:18.956 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.1] 2023-10-30T12:28:18.956 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQBJoT9l4yJYFhAATBeKdpEW58A3Bz10hqcj2g== 2023-10-30T12:28:18.956 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.2] 2023-10-30T12:28:18.956 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQBMoT9ljKLcHBAAtNQBrqkKb79JHjPLEj+bdw== 2023-10-30T12:28:18.957 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.3] 2023-10-30T12:28:18.957 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQBPoT9lM5PkIhAAF01gbhPX46l62NBys2qg7A== 2023-10-30T12:28:18.957 INFO:teuthology.orchestra.run.smithi089.stdout:[mgr.x] 2023-10-30T12:28:18.957 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQBCoT9lF7cbExAAc60QSSj/MiOQ45/51quoxw== 2023-10-30T12:28:18.957 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.b] 2023-10-30T12:28:18.958 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQBCoT9lfeWUKhAAcosNUlwtIndFw8jF/smatg== 2023-10-30T12:28:18.958 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.d] 2023-10-30T12:28:18.958 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQBCoT9liREBMxAAELu0uANMbBxs6ndIj2n5Mw== 2023-10-30T12:28:18.958 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.f] 2023-10-30T12:28:18.959 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQBCoT9leqxwOxAA6DeGLLqObT/B7ZanhKAgTg== 2023-10-30T12:28:18.959 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.4] 2023-10-30T12:28:18.959 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQBVoT9liP/LHhAAAirnVVabUUmIN5uHXCZDwQ== 2023-10-30T12:28:18.959 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.5] 2023-10-30T12:28:18.959 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQBYoT9lKJufJRAAPmVQwjXwM3CCLr1to69IAQ== 2023-10-30T12:28:18.960 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.6] 2023-10-30T12:28:18.960 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQBboT9lh55wMRAANryvv/ofmTbphZLbr5LKKQ== 2023-10-30T12:28:18.960 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.7] 2023-10-30T12:28:18.960 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQBeoT9l/yyENxAAyix2rm4OGZQB5hJp9FTwBQ== 2023-10-30T12:28:18.960 INFO:teuthology.orchestra.run.smithi089.stdout:[client.0] 2023-10-30T12:28:18.960 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQBDoT9lJY+6BRAAKnjh4CNZuC6Y9RgyY4BnQA== 2023-10-30T12:28:18.961 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T12:28:19.001 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T12:28:19.057 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T12:28:19.061 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T12:28:19.140 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T12:28:19.144 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T12:28:19.198 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T12:28:19.202 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T12:28:19.268 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:19.272 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:19.350 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:19.354 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:19.436 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:19.440 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:19.519 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:19.523 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:19.607 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T12:28:19.611 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T12:28:19.696 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T12:28:19.700 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T12:28:19.753 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T12:28:19.757 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T12:28:19.835 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T12:28:19.839 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T12:28:19.922 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:19.925 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:20.007 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:20.011 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:20.092 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:20.096 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:20.147 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:20.151 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:20.216 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-30T12:28:20.220 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-30T12:28:20.307 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-30T12:28:20.308 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-30T12:28:20.328 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T12:28:20.441 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-30T12:28:20.461 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-10-30T12:28:20.476 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T12:28:20.575 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-10-30T12:28:20.586 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-10-30T12:28:20.637 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T12:28:20.738 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-10-30T12:28:20.750 DEBUG:teuthology.orchestra.run.smithi089:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:28:20.754 DEBUG:teuthology.orchestra.run.smithi192:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:28:20.796 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-30T12:28:20.796 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-30T12:28:20.797 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-30T12:28:20.805 INFO:tasks.ceph.mon.a:Started 2023-10-30T12:28:20.805 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-30T12:28:20.805 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-10-30T12:28:20.841 INFO:tasks.ceph.mon.b:Started 2023-10-30T12:28:20.841 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-30T12:28:20.842 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-10-30T12:28:20.844 INFO:tasks.ceph.mon.c:Started 2023-10-30T12:28:20.844 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-30T12:28:20.845 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-10-30T12:28:20.845 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-10-30T12:28:20.848 INFO:tasks.ceph.mgr.y:Started 2023-10-30T12:28:20.849 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-30T12:28:20.849 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-30T12:28:20.851 INFO:tasks.ceph.mgr.x:Started 2023-10-30T12:28:20.852 INFO:tasks.ceph:Setting crush tunables to default 2023-10-30T12:28:20.852 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster ceph osd crush tunables default 2023-10-30T12:28:20.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:20.990+0000 7f378610e640 1 Processor -- start 2023-10-30T12:28:20.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:20.990+0000 7f378610e640 1 -- start start 2023-10-30T12:28:20.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:20.990+0000 7f378610e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f37801508c0 0x7f37800c6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:20.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:20.990+0000 7f378610e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f37800722a0 0x7f37800c6e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:20.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:20.990+0000 7f378610e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37800bfec0 0x7f37800c0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:20.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:20.990+0000 7f378610e640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37800c85b0 con 0x7f37801508c0 2023-10-30T12:28:20.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:20.990+0000 7f378610e640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37800c0a30 con 0x7f37800722a0 2023-10-30T12:28:20.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:20.990+0000 7f378610e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f378006f650 con 0x7f37800bfec0 2023-10-30T12:28:20.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:20.990+0000 7f377f7fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f37801508c0 0x7f37800c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:20.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:20.990+0000 7f377f7fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f37801508c0 0x7f37800c6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49388/0 (socket says 172.21.15.89:49388) 2023-10-30T12:28:20.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:20.990+0000 7f377f7fe640 1 -- 172.21.15.89:0/4158000172 learned_addr learned my addr 172.21.15.89:0/4158000172 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:20.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:20.990+0000 7f377ffff640 1 --2- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37800bfec0 0x7f37800c0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:20.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:20.990+0000 7f377effd640 1 --2- 172.21.15.89:0/4158000172 >> v2:172.21.15.192:3300/0 conn(0x7f37800722a0 0x7f37800c6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:20.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:20.990+0000 7f377f7fe640 1 -- 172.21.15.89:0/4158000172 >> v2:172.21.15.89:3300/0 conn(0x7f37801508c0 msgr2=0x7f37800c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:28:20.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:20.990+0000 7f377f7fe640 1 -- 172.21.15.89:0/4158000172 >> v2:172.21.15.89:3300/0 conn(0x7f37801508c0 msgr2=0x7f37800c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:28:20.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:20.990+0000 7f377f7fe640 1 --2- 172.21.15.89:0/4158000172 >> v2:172.21.15.89:3300/0 conn(0x7f37801508c0 0x7f37800c6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:28:20.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:20.990+0000 7f377f7fe640 1 --2- 172.21.15.89:0/4158000172 >> v2:172.21.15.89:3300/0 conn(0x7f37801508c0 0x7f37800c6920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:28:20.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:20.990+0000 7f377effd640 1 -- 172.21.15.89:0/4158000172 >> v2:172.21.15.192:3300/0 conn(0x7f37800722a0 msgr2=0x7f37800c6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:28:20.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:20.990+0000 7f377effd640 1 -- 172.21.15.89:0/4158000172 >> v2:172.21.15.192:3300/0 conn(0x7f37800722a0 msgr2=0x7f37800c6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:28:20.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:20.990+0000 7f377effd640 1 --2- 172.21.15.89:0/4158000172 >> v2:172.21.15.192:3300/0 conn(0x7f37800722a0 0x7f37800c6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:28:20.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:20.990+0000 7f377ffff640 1 -- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37800bfec0 msgr2=0x7f37800c0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:28:20.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:20.990+0000 7f377effd640 1 --2- 172.21.15.89:0/4158000172 >> v2:172.21.15.192:3300/0 conn(0x7f37800722a0 0x7f37800c6e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:28:20.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:20.990+0000 7f377ffff640 1 -- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37800bfec0 msgr2=0x7f37800c0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:28:20.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:20.990+0000 7f377ffff640 1 --2- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37800bfec0 0x7f37800c0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:28:20.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:20.990+0000 7f377ffff640 1 --2- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37800bfec0 0x7f37800c0350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:28:21.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:21.194+0000 7f377f7fe640 1 --2- 172.21.15.89:0/4158000172 >> v2:172.21.15.89:3300/0 conn(0x7f37801508c0 0x7f37800c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:21.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:21.194+0000 7f377f7fe640 1 -- 172.21.15.89:0/4158000172 >> v2:172.21.15.89:3300/0 conn(0x7f37801508c0 msgr2=0x7f37800c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:28:21.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:21.194+0000 7f377f7fe640 1 -- 172.21.15.89:0/4158000172 >> v2:172.21.15.89:3300/0 conn(0x7f37801508c0 msgr2=0x7f37800c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:28:21.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:21.194+0000 7f377f7fe640 1 --2- 172.21.15.89:0/4158000172 >> v2:172.21.15.89:3300/0 conn(0x7f37801508c0 0x7f37800c6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:28:21.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:21.194+0000 7f377f7fe640 1 --2- 172.21.15.89:0/4158000172 >> v2:172.21.15.89:3300/0 conn(0x7f37801508c0 0x7f37800c6920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:28:21.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:21.194+0000 7f377ffff640 1 --2- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37800bfec0 0x7f37800c0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:21.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:21.194+0000 7f377effd640 1 --2- 172.21.15.89:0/4158000172 >> v2:172.21.15.192:3300/0 conn(0x7f37800722a0 0x7f37800c6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:21.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:21.194+0000 7f377effd640 1 -- 172.21.15.89:0/4158000172 >> v2:172.21.15.192:3300/0 conn(0x7f37800722a0 msgr2=0x7f37800c6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:28:21.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:21.194+0000 7f377effd640 1 -- 172.21.15.89:0/4158000172 >> v2:172.21.15.192:3300/0 conn(0x7f37800722a0 msgr2=0x7f37800c6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:28:21.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:21.194+0000 7f377effd640 1 --2- 172.21.15.89:0/4158000172 >> v2:172.21.15.192:3300/0 conn(0x7f37800722a0 0x7f37800c6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:28:21.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:21.194+0000 7f377effd640 1 --2- 172.21.15.89:0/4158000172 >> v2:172.21.15.192:3300/0 conn(0x7f37800722a0 0x7f37800c6e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:28:21.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:21.194+0000 7f377ffff640 1 -- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37800bfec0 msgr2=0x7f37800c0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:28:21.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:21.194+0000 7f377ffff640 1 -- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37800bfec0 msgr2=0x7f37800c0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:28:21.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:21.194+0000 7f377ffff640 1 --2- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37800bfec0 0x7f37800c0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:28:21.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:21.194+0000 7f377ffff640 1 --2- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37800bfec0 0x7f37800c0350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:28:21.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:21.598+0000 7f377f7fe640 1 --2- 172.21.15.89:0/4158000172 >> v2:172.21.15.89:3300/0 conn(0x7f37801508c0 0x7f37800c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:21.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:21.598+0000 7f377f7fe640 1 -- 172.21.15.89:0/4158000172 >> v2:172.21.15.89:3300/0 conn(0x7f37801508c0 msgr2=0x7f37800c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:28:21.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:21.598+0000 7f377f7fe640 1 -- 172.21.15.89:0/4158000172 >> v2:172.21.15.89:3300/0 conn(0x7f37801508c0 msgr2=0x7f37800c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:28:21.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:21.598+0000 7f377f7fe640 1 --2- 172.21.15.89:0/4158000172 >> v2:172.21.15.89:3300/0 conn(0x7f37801508c0 0x7f37800c6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:28:21.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:21.598+0000 7f377f7fe640 1 --2- 172.21.15.89:0/4158000172 >> v2:172.21.15.89:3300/0 conn(0x7f37801508c0 0x7f37800c6920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:28:21.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:21.598+0000 7f377ffff640 1 --2- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37800bfec0 0x7f37800c0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:21.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:21.598+0000 7f377effd640 1 --2- 172.21.15.89:0/4158000172 >> v2:172.21.15.192:3300/0 conn(0x7f37800722a0 0x7f37800c6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:21.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:21.598+0000 7f377ffff640 1 -- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37800bfec0 msgr2=0x7f37800c0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:28:21.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:21.598+0000 7f377ffff640 1 -- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37800bfec0 msgr2=0x7f37800c0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:28:21.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:21.598+0000 7f377ffff640 1 --2- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37800bfec0 0x7f37800c0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:28:21.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:21.598+0000 7f377effd640 1 -- 172.21.15.89:0/4158000172 >> v2:172.21.15.192:3300/0 conn(0x7f37800722a0 msgr2=0x7f37800c6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:28:21.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:21.598+0000 7f377effd640 1 -- 172.21.15.89:0/4158000172 >> v2:172.21.15.192:3300/0 conn(0x7f37800722a0 msgr2=0x7f37800c6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:28:21.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:21.598+0000 7f377effd640 1 --2- 172.21.15.89:0/4158000172 >> v2:172.21.15.192:3300/0 conn(0x7f37800722a0 0x7f37800c6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:28:21.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:21.598+0000 7f377ffff640 1 --2- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37800bfec0 0x7f37800c0350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:28:21.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:21.598+0000 7f377effd640 1 --2- 172.21.15.89:0/4158000172 >> v2:172.21.15.192:3300/0 conn(0x7f37800722a0 0x7f37800c6e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-30T12:28:22.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:22.402+0000 7f377f7fe640 1 --2- 172.21.15.89:0/4158000172 >> v2:172.21.15.89:3300/0 conn(0x7f37801508c0 0x7f37800c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:22.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:22.402+0000 7f377effd640 1 --2- 172.21.15.89:0/4158000172 >> v2:172.21.15.192:3300/0 conn(0x7f37800722a0 0x7f37800c6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:22.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:22.402+0000 7f377ffff640 1 --2- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37800bfec0 0x7f37800c0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:22.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:22.402+0000 7f377f7fe640 1 -- 172.21.15.89:0/4158000172 >> v2:172.21.15.89:3300/0 conn(0x7f37801508c0 msgr2=0x7f37800c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:28:22.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:22.402+0000 7f377f7fe640 1 -- 172.21.15.89:0/4158000172 >> v2:172.21.15.89:3300/0 conn(0x7f37801508c0 msgr2=0x7f37800c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:28:22.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:22.402+0000 7f377f7fe640 1 --2- 172.21.15.89:0/4158000172 >> v2:172.21.15.89:3300/0 conn(0x7f37801508c0 0x7f37800c6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:28:22.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:22.402+0000 7f377f7fe640 1 --2- 172.21.15.89:0/4158000172 >> v2:172.21.15.89:3300/0 conn(0x7f37801508c0 0x7f37800c6920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:28:22.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:22.402+0000 7f377effd640 1 -- 172.21.15.89:0/4158000172 >> v2:172.21.15.192:3300/0 conn(0x7f37800722a0 msgr2=0x7f37800c6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-30T12:28:22.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:22.402+0000 7f377effd640 1 -- 172.21.15.89:0/4158000172 >> v2:172.21.15.192:3300/0 conn(0x7f37800722a0 msgr2=0x7f37800c6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:28:22.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:22.402+0000 7f377effd640 1 --2- 172.21.15.89:0/4158000172 >> v2:172.21.15.192:3300/0 conn(0x7f37800722a0 0x7f37800c6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:28:22.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:22.402+0000 7f377effd640 1 --2- 172.21.15.89:0/4158000172 >> v2:172.21.15.192:3300/0 conn(0x7f37800722a0 0x7f37800c6e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:28:22.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:22.402+0000 7f377ffff640 1 -- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37800bfec0 msgr2=0x7f37800c0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-30T12:28:22.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:22.402+0000 7f377ffff640 1 -- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37800bfec0 msgr2=0x7f37800c0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:28:22.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:22.402+0000 7f377ffff640 1 --2- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37800bfec0 0x7f37800c0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:28:22.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:22.402+0000 7f377ffff640 1 --2- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37800bfec0 0x7f37800c0350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-30T12:28:23.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:23.993+0000 7f377d7fa640 1 -- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37800bfec0 msgr2=0x7f37800c0350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:28:23.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:23.993+0000 7f377d7fa640 1 --2- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f37800bfec0 0x7f37800c0350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:23.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:23.993+0000 7f377d7fa640 1 -- 172.21.15.89:0/4158000172 >> v2:172.21.15.192:3300/0 conn(0x7f37800722a0 msgr2=0x7f37800c6e60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:28:23.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:23.993+0000 7f377d7fa640 1 --2- 172.21.15.89:0/4158000172 >> v2:172.21.15.192:3300/0 conn(0x7f37800722a0 0x7f37800c6e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:23.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:23.993+0000 7f377d7fa640 1 -- 172.21.15.89:0/4158000172 >> v2:172.21.15.89:3300/0 conn(0x7f37801508c0 msgr2=0x7f37800c6920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:28:23.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:23.993+0000 7f377d7fa640 1 --2- 172.21.15.89:0/4158000172 >> v2:172.21.15.89:3300/0 conn(0x7f37801508c0 0x7f37800c6920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:23.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:23.993+0000 7f377d7fa640 1 -- 172.21.15.89:0/4158000172 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f378006f930 con 0x7f37680036a0 2023-10-30T12:28:23.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:23.993+0000 7f377d7fa640 1 -- 172.21.15.89:0/4158000172 --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f376800c090 con 0x7f3768007310 2023-10-30T12:28:26.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.237+0000 7f377e7fc640 1 -- 172.21.15.89:0/4158000172 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3328985826 0 0) 0x7f378006f930 con 0x7f37680036a0 2023-10-30T12:28:26.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.237+0000 7f377e7fc640 1 -- 172.21.15.89:0/4158000172 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f376c003530 con 0x7f37680036a0 2023-10-30T12:28:26.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.241+0000 7f377e7fc640 1 -- 172.21.15.89:0/4158000172 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3111313508 0 0) 0x7f376c003530 con 0x7f37680036a0 2023-10-30T12:28:26.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.241+0000 7f377e7fc640 1 -- 172.21.15.89:0/4158000172 >> v1:172.21.15.192:6789/0 conn(0x7f3768007310 legacy=0x7f37680097d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:26.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.241+0000 7f377e7fc640 1 -- 172.21.15.89:0/4158000172 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f376800d290 con 0x7f37680036a0 2023-10-30T12:28:26.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.241+0000 7f377e7fc640 1 -- 172.21.15.89:0/4158000172 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f3770010670 con 0x7f37680036a0 2023-10-30T12:28:26.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.241+0000 7f377e7fc640 1 -- 172.21.15.89:0/4158000172 >> v1:172.21.15.89:6789/0 conn(0x7f37680036a0 legacy=0x7f3768005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:26.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.241+0000 7f377ffff640 1 -- 172.21.15.89:0/4158000172 reap_dead start 2023-10-30T12:28:26.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.241+0000 7f377e7fc640 1 --2- 172.21.15.89:0/4158000172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f376c003a00 0x7f376c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:26.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.241+0000 7f377e7fc640 1 --2- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f376c0047d0 0x7f376c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:26.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.241+0000 7f377e7fc640 1 --2- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f376c009980 0x7f376c00bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:26.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.241+0000 7f377e7fc640 1 -- 172.21.15.89:0/4158000172 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376c00c360 con 0x7f376c003a00 2023-10-30T12:28:26.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.241+0000 7f377e7fc640 1 -- 172.21.15.89:0/4158000172 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376c00c4a0 con 0x7f376c0047d0 2023-10-30T12:28:26.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.241+0000 7f377e7fc640 1 -- 172.21.15.89:0/4158000172 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f376c00c660 con 0x7f376c009980 2023-10-30T12:28:26.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.241+0000 7f377f7fe640 1 --2- 172.21.15.89:0/4158000172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f376c003a00 0x7f376c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:26.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.241+0000 7f377effd640 1 --2- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f376c0047d0 0x7f376c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:26.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.241+0000 7f377ffff640 1 --2- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f376c009980 0x7f376c00bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:26.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.241+0000 7f377f7fe640 1 -- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f376c009980 msgr2=0x7f376c00bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:26.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.241+0000 7f377f7fe640 1 --2- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f376c009980 0x7f376c00bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:26.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.241+0000 7f377f7fe640 1 -- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f376c0047d0 msgr2=0x7f376c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:26.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.241+0000 7f377f7fe640 1 --2- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f376c0047d0 0x7f376c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:26.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.241+0000 7f377f7fe640 1 -- 172.21.15.89:0/4158000172 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f376c003530 con 0x7f376c003a00 2023-10-30T12:28:26.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.241+0000 7f377effd640 1 --2- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f376c0047d0 0x7f376c004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:26.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.245+0000 7f377f7fe640 1 --2- 172.21.15.89:0/4158000172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f376c003a00 0x7f376c003e00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3770006fa0 tx=0x7f377001d7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81eeb3eb351393c0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:26.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.245+0000 7f377e7fc640 1 -- 172.21.15.89:0/4158000172 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37700aa680 con 0x7f376c003a00 2023-10-30T12:28:26.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.245+0000 7f377e7fc640 1 -- 172.21.15.89:0/4158000172 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f377000c040 con 0x7f376c003a00 2023-10-30T12:28:26.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.245+0000 7f377e7fc640 1 -- 172.21.15.89:0/4158000172 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37700aab00 con 0x7f376c003a00 2023-10-30T12:28:26.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.245+0000 7f378610e640 1 -- 172.21.15.89:0/4158000172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f376c003a00 msgr2=0x7f376c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:26.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.245+0000 7f378610e640 1 --2- 172.21.15.89:0/4158000172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f376c003a00 0x7f376c003e00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3770006fa0 tx=0x7f377001d7d0 comp rx=0 tx=0).stop 2023-10-30T12:28:26.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.245+0000 7f378610e640 1 -- 172.21.15.89:0/4158000172 shutdown_connections 2023-10-30T12:28:26.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.245+0000 7f378610e640 1 --2- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f376c0047d0 0x7f376c004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:26.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.245+0000 7f378610e640 1 --2- 172.21.15.89:0/4158000172 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f376c009980 0x7f376c00bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:26.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.245+0000 7f378610e640 1 --2- 172.21.15.89:0/4158000172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f376c003a00 0x7f376c003e00 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:26.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.245+0000 7f378610e640 1 -- 172.21.15.89:0/4158000172 >> 172.21.15.89:0/4158000172 conn(0x7f37800c7910 msgr2=0x7f37800c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:26.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.245+0000 7f378610e640 1 -- 172.21.15.89:0/4158000172 shutdown_connections 2023-10-30T12:28:26.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.245+0000 7f378610e640 1 -- 172.21.15.89:0/4158000172 wait complete. 2023-10-30T12:28:26.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.249+0000 7f378610e640 1 Processor -- start 2023-10-30T12:28:26.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.249+0000 7f378610e640 1 -- start start 2023-10-30T12:28:26.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.249+0000 7f378610e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f376c003a00 0x7f37801511a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:26.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.249+0000 7f378610e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f376c0047d0 0x7f3780142b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:26.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.249+0000 7f378610e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f376c009980 0x7f37801538f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:26.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.249+0000 7f378610e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3780156080 con 0x7f376c009980 2023-10-30T12:28:26.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.249+0000 7f378610e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37801565b0 con 0x7f376c003a00 2023-10-30T12:28:26.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.249+0000 7f378610e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3780153460 con 0x7f376c0047d0 2023-10-30T12:28:26.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.249+0000 7f377ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f376c009980 0x7f37801538f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:26.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.249+0000 7f377ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f376c009980 0x7f37801538f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49470/0 (socket says 172.21.15.89:49470) 2023-10-30T12:28:26.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.249+0000 7f377ffff640 1 -- 172.21.15.89:0/1253700332 learned_addr learned my addr 172.21.15.89:0/1253700332 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:26.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.249+0000 7f377f7fe640 1 --2- 172.21.15.89:0/1253700332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f376c003a00 0x7f37801511a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:26.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.249+0000 7f377effd640 1 --2- 172.21.15.89:0/1253700332 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f376c0047d0 0x7f3780142b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:26.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.249+0000 7f377ffff640 1 -- 172.21.15.89:0/1253700332 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f376c0047d0 msgr2=0x7f3780142b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:26.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.249+0000 7f377ffff640 1 --2- 172.21.15.89:0/1253700332 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f376c0047d0 0x7f3780142b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:26.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.249+0000 7f377ffff640 1 -- 172.21.15.89:0/1253700332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f376c003a00 msgr2=0x7f37801511a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:26.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.249+0000 7f377ffff640 1 --2- 172.21.15.89:0/1253700332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f376c003a00 0x7f37801511a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:26.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.249+0000 7f377ffff640 1 -- 172.21.15.89:0/1253700332 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3780143090 con 0x7f376c009980 2023-10-30T12:28:26.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.249+0000 7f377ffff640 1 --2- 172.21.15.89:0/1253700332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f376c009980 0x7f37801538f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3778009610 tx=0x7f3778014690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:26.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.249+0000 7f3767fff640 1 -- 172.21.15.89:0/1253700332 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3778003ea0 con 0x7f376c009980 2023-10-30T12:28:26.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.249+0000 7f3767fff640 1 -- 172.21.15.89:0/1253700332 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f377800f040 con 0x7f376c009980 2023-10-30T12:28:26.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.249+0000 7f378610e640 1 -- 172.21.15.89:0/1253700332 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37801f5f60 con 0x7f376c009980 2023-10-30T12:28:26.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.249+0000 7f3767fff640 1 -- 172.21.15.89:0/1253700332 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3778017440 con 0x7f376c009980 2023-10-30T12:28:26.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.249+0000 7f378610e640 1 -- 172.21.15.89:0/1253700332 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37801530f0 con 0x7f376c009980 2023-10-30T12:28:26.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.249+0000 7f3767fff640 1 -- 172.21.15.89:0/1253700332 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f37780178a0 con 0x7f376c009980 2023-10-30T12:28:26.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.249+0000 7f3767fff640 1 -- 172.21.15.89:0/1253700332 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3778020ba0 con 0x7f376c009980 2023-10-30T12:28:26.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.253+0000 7f378610e640 1 -- 172.21.15.89:0/1253700332 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f378004e170 con 0x7f376c009980 2023-10-30T12:28:26.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.265+0000 7f3767fff640 1 -- 172.21.15.89:0/1253700332 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f377803eb40 con 0x7f376c009980 2023-10-30T12:28:26.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.325+0000 7f378610e640 1 -- 172.21.15.89:0/1253700332 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f37801535a0 con 0x7f376c009980 2023-10-30T12:28:26.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.329+0000 7f3767fff640 1 -- 172.21.15.89:0/1253700332 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f377802d020 con 0x7f376c009980 2023-10-30T12:28:26.334 INFO:teuthology.orchestra.run.smithi089.stderr:adjusted tunables profile to default 2023-10-30T12:28:26.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.333+0000 7f378610e640 1 -- 172.21.15.89:0/1253700332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f376c009980 msgr2=0x7f37801538f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:26.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.333+0000 7f378610e640 1 --2- 172.21.15.89:0/1253700332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f376c009980 0x7f37801538f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3778009610 tx=0x7f3778014690 comp rx=0 tx=0).stop 2023-10-30T12:28:26.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.333+0000 7f378610e640 1 -- 172.21.15.89:0/1253700332 shutdown_connections 2023-10-30T12:28:26.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.333+0000 7f378610e640 1 --2- 172.21.15.89:0/1253700332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f376c009980 0x7f37801538f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:26.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.333+0000 7f378610e640 1 --2- 172.21.15.89:0/1253700332 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f376c0047d0 0x7f3780142b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:26.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.333+0000 7f378610e640 1 --2- 172.21.15.89:0/1253700332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f376c003a00 0x7f37801511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:26.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.333+0000 7f378610e640 1 -- 172.21.15.89:0/1253700332 >> 172.21.15.89:0/1253700332 conn(0x7f37800c7910 msgr2=0x7f37800c0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:26.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.333+0000 7f378610e640 1 -- 172.21.15.89:0/1253700332 shutdown_connections 2023-10-30T12:28:26.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:26.333+0000 7f378610e640 1 -- 172.21.15.89:0/1253700332 wait complete. 2023-10-30T12:28:26.354 INFO:tasks.ceph:check_enable_crimson: False 2023-10-30T12:28:26.355 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-30T12:28:26.355 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-10-30T12:28:26.355 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-30T12:28:26.377 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-10-30T12:28:26.427 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-30T12:28:26.449 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-10-30T12:28:26.449 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-30T12:28:26.459 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-30T12:28:26.456+0000 7fb4e738e180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-30T12:28:26.475 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-10-30T12:28:26.473+0000 7f7edc42e180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-30T12:28:26.495 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-10-30T12:28:26.495 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-10-30T12:28:26.558 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-30T12:28:26.559 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-10-30T12:28:26.571 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-30T12:28:26.571 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-10-30T12:28:26.623 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-30T12:28:26.624 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-10-30T12:28:26.675 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-30T12:28:26.675 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-10-30T12:28:26.727 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 72595c54-fd6e-4a5a-bb14-ff27e9377b07 0 2023-10-30T12:28:26.890 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.888+0000 7f25ac0b7640 1 Processor -- start 2023-10-30T12:28:26.891 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.888+0000 7f25ac0b7640 1 -- start start 2023-10-30T12:28:26.892 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.888+0000 7f25ac0b7640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25a415a780 0x7f25a40c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:26.892 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.888+0000 7f25ac0b7640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a4159be0 con 0x7f25a40c2e20 2023-10-30T12:28:26.892 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.888+0000 7f25ac0b7640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a4156850 con 0x7f25a4150800 2023-10-30T12:28:26.892 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.888+0000 7f25ac0b7640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a40c7020 con 0x7f25a415a780 2023-10-30T12:28:26.893 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.888+0000 7f25a9e53640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f25a4150800 0x7f25a40cb170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:48420/0 (socket says 172.21.15.192:48420) 2023-10-30T12:28:26.893 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.888+0000 7f25a9e53640 1 -- 172.21.15.192:0/702242941 learned_addr learned my addr 172.21.15.192:0/702242941 (peer_addr_for_me v1:172.21.15.192:0/0) 2023-10-30T12:28:26.893 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.888+0000 7f25aa654640 1 --2- 172.21.15.192:0/702242941 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25a415a780 0x7f25a40c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:26.893 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.888+0000 7f25a8e51640 1 -- 172.21.15.192:0/702242941 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4254490118 0 0) 0x7f25a4156850 con 0x7f25a4150800 2023-10-30T12:28:26.893 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.892+0000 7f25a8e51640 1 -- 172.21.15.192:0/702242941 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f258c003530 con 0x7f25a4150800 2023-10-30T12:28:26.894 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.892+0000 7f25aa654640 1 -- 172.21.15.192:0/702242941 >> v1:172.21.15.192:6789/0 conn(0x7f25a4150800 legacy=0x7f25a40cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:26.894 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.892+0000 7f25aa654640 1 -- 172.21.15.192:0/702242941 >> v1:172.21.15.89:6789/0 conn(0x7f25a40c2e20 legacy=0x7f25a415a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:26.894 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.892+0000 7f25aa654640 1 -- 172.21.15.192:0/702242941 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a4156b30 con 0x7f25a415a780 2023-10-30T12:28:26.895 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.892+0000 7f25aa654640 1 --2- 172.21.15.192:0/702242941 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25a415a780 0x7f25a40c6ae0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2594009dc0 tx=0x7f259401f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2dfea0ec9821a02 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:26.895 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.892+0000 7f25a8e51640 1 -- 172.21.15.192:0/702242941 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25940af600 con 0x7f25a415a780 2023-10-30T12:28:26.896 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.892+0000 7f25a8e51640 1 -- 172.21.15.192:0/702242941 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25940ab050 con 0x7f25a415a780 2023-10-30T12:28:26.896 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.892+0000 7f25a8e51640 1 -- 172.21.15.192:0/702242941 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25940af980 con 0x7f25a415a780 2023-10-30T12:28:26.896 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.892+0000 7f25ac0b7640 1 -- 172.21.15.192:0/702242941 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25a415a780 msgr2=0x7f25a40c6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:26.896 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.892+0000 7f25ac0b7640 1 --2- 172.21.15.192:0/702242941 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25a415a780 0x7f25a40c6ae0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2594009dc0 tx=0x7f259401f150 comp rx=0 tx=0).stop 2023-10-30T12:28:26.896 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.892+0000 7f25ac0b7640 1 -- 172.21.15.192:0/702242941 shutdown_connections 2023-10-30T12:28:26.897 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.892+0000 7f25ac0b7640 1 --2- 172.21.15.192:0/702242941 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25a415a780 0x7f25a40c6ae0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:26.897 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.892+0000 7f25ac0b7640 1 -- 172.21.15.192:0/702242941 >> 172.21.15.192:0/702242941 conn(0x7f25a406c850 msgr2=0x7f25a406cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:26.897 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.892+0000 7f25ac0b7640 1 -- 172.21.15.192:0/702242941 shutdown_connections 2023-10-30T12:28:26.897 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.892+0000 7f25ac0b7640 1 -- 172.21.15.192:0/702242941 wait complete. 2023-10-30T12:28:26.899 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.896+0000 7f25ac0b7640 1 Processor -- start 2023-10-30T12:28:26.899 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.896+0000 7f25ac0b7640 1 -- start start 2023-10-30T12:28:26.900 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.896+0000 7f25ac0b7640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25a40c2e20 0x7f25a41590d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:26.900 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.896+0000 7f25ac0b7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25a4150800 0x7f25a414df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:26.900 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.896+0000 7f25ac0b7640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25a415a780 0x7f25a4146530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:26.900 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.896+0000 7f25ac0b7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a4159990 con 0x7f25a4150800 2023-10-30T12:28:26.900 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.896+0000 7f25ac0b7640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a413b4a0 con 0x7f25a40c2e20 2023-10-30T12:28:26.901 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.896+0000 7f25ac0b7640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a40c8220 con 0x7f25a415a780 2023-10-30T12:28:26.901 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.896+0000 7f25aa654640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25a415a780 0x7f25a4146530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:26.901 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.896+0000 7f25a9e53640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25a40c2e20 0x7f25a41590d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:26.901 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.896+0000 7f25aa654640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25a415a780 0x7f25a4146530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.192:60588/0 (socket says 172.21.15.192:60588) 2023-10-30T12:28:26.901 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.896+0000 7f25a9e53640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25a40c2e20 0x7f25a41590d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:42104/0 (socket says 172.21.15.192:42104) 2023-10-30T12:28:26.902 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.896+0000 7f25aa654640 1 -- 172.21.15.192:0/246999709 learned_addr learned my addr 172.21.15.192:0/246999709 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-10-30T12:28:26.902 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.896+0000 7f25a9652640 1 --2- 172.21.15.192:0/246999709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25a4150800 0x7f25a414df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:26.902 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.896+0000 7f25a9e53640 1 -- 172.21.15.192:0/246999709 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25a415a780 msgr2=0x7f25a4146530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:26.902 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.896+0000 7f25a9e53640 1 --2- 172.21.15.192:0/246999709 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25a415a780 0x7f25a4146530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:26.902 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.896+0000 7f25a9e53640 1 -- 172.21.15.192:0/246999709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25a4150800 msgr2=0x7f25a414df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:26.902 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.896+0000 7f25a9e53640 1 --2- 172.21.15.192:0/246999709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25a4150800 0x7f25a414df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:26.903 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.896+0000 7f25a9e53640 1 -- 172.21.15.192:0/246999709 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2594009a70 con 0x7f25a40c2e20 2023-10-30T12:28:26.903 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.896+0000 7f25aa654640 1 --2- 172.21.15.192:0/246999709 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25a415a780 0x7f25a4146530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:26.903 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.896+0000 7f25a9e53640 1 --2- 172.21.15.192:0/246999709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25a40c2e20 0x7f25a41590d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f259c00c400 tx=0x7f259c00c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:26.903 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.896+0000 7f25a27fc640 1 -- 172.21.15.192:0/246999709 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c005cf0 con 0x7f25a40c2e20 2023-10-30T12:28:26.903 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.896+0000 7f25a27fc640 1 -- 172.21.15.192:0/246999709 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f259c005e50 con 0x7f25a40c2e20 2023-10-30T12:28:26.904 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.896+0000 7f25ac0b7640 1 -- 172.21.15.192:0/246999709 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a41596b0 con 0x7f25a40c2e20 2023-10-30T12:28:26.904 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.896+0000 7f25a27fc640 1 -- 172.21.15.192:0/246999709 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c0072d0 con 0x7f25a40c2e20 2023-10-30T12:28:26.904 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.896+0000 7f25ac0b7640 1 -- 172.21.15.192:0/246999709 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a414e490 con 0x7f25a40c2e20 2023-10-30T12:28:26.904 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.896+0000 7f25a27fc640 1 -- 172.21.15.192:0/246999709 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f259c01c440 con 0x7f25a40c2e20 2023-10-30T12:28:26.904 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.896+0000 7f25a27fc640 1 -- 172.21.15.192:0/246999709 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f259c012780 con 0x7f25a40c2e20 2023-10-30T12:28:26.904 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.896+0000 7f25ac0b7640 1 -- 172.21.15.192:0/246999709 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25a4158c50 con 0x7f25a40c2e20 2023-10-30T12:28:26.915 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.912+0000 7f25a27fc640 1 -- 172.21.15.192:0/246999709 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f25a4158c50 con 0x7f25a40c2e20 2023-10-30T12:28:26.983 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.980+0000 7f25ac0b7640 1 -- 172.21.15.192:0/246999709 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd new", "uuid": "72595c54-fd6e-4a5a-bb14-ff27e9377b07", "id": 0} v 0) v1 -- 0x7f25a404e170 con 0x7f25a40c2e20 2023-10-30T12:28:26.988 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.988+0000 7f25a27fc640 1 -- 172.21.15.192:0/246999709 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "72595c54-fd6e-4a5a-bb14-ff27e9377b07", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f259c021060 con 0x7f25a40c2e20 2023-10-30T12:28:26.988 INFO:teuthology.orchestra.run.smithi192.stdout:0 2023-10-30T12:28:26.990 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.988+0000 7f25ac0b7640 1 -- 172.21.15.192:0/246999709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25a40c2e20 msgr2=0x7f25a41590d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:26.990 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.988+0000 7f25ac0b7640 1 --2- 172.21.15.192:0/246999709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25a40c2e20 0x7f25a41590d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f259c00c400 tx=0x7f259c00c8c0 comp rx=0 tx=0).stop 2023-10-30T12:28:26.990 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.988+0000 7f25ac0b7640 1 -- 172.21.15.192:0/246999709 shutdown_connections 2023-10-30T12:28:26.990 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.988+0000 7f25ac0b7640 1 --2- 172.21.15.192:0/246999709 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25a415a780 0x7f25a4146530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:26.990 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.988+0000 7f25ac0b7640 1 --2- 172.21.15.192:0/246999709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25a4150800 0x7f25a414df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:26.991 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.988+0000 7f25ac0b7640 1 --2- 172.21.15.192:0/246999709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25a40c2e20 0x7f25a41590d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:26.991 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.988+0000 7f25ac0b7640 1 -- 172.21.15.192:0/246999709 >> 172.21.15.192:0/246999709 conn(0x7f25a406c850 msgr2=0x7f25a406cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:26.991 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.988+0000 7f25ac0b7640 1 -- 172.21.15.192:0/246999709 shutdown_connections 2023-10-30T12:28:26.991 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:26.988+0000 7f25ac0b7640 1 -- 172.21.15.192:0/246999709 wait complete. 2023-10-30T12:28:27.004 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 0305670b-d7c4-4193-bec2-8b7351ff590a 1 2023-10-30T12:28:27.017 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-30T12:28:27.016+0000 7fb4e738e180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-30T12:28:27.046 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-10-30T12:28:27.041+0000 7f7edc42e180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-30T12:28:27.119 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.116+0000 7f98454fb640 1 Processor -- start 2023-10-30T12:28:27.119 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.116+0000 7f98454fb640 1 -- start start 2023-10-30T12:28:27.120 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.116+0000 7f98454fb640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9840150800 0x7f98400c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:27.121 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.116+0000 7f98454fb640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9840159be0 con 0x7f984015a070 2023-10-30T12:28:27.121 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.120+0000 7f98454fb640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9840156850 con 0x7f98400c2e20 2023-10-30T12:28:27.121 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.120+0000 7f98454fb640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98400c7020 con 0x7f9840150800 2023-10-30T12:28:27.121 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.120+0000 7f983effd640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9840150800 0x7f98400c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:27.121 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.120+0000 7f983e7fc640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f98400c2e20 0x7f98400cb170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:48422/0 (socket says 172.21.15.192:48422) 2023-10-30T12:28:27.121 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.120+0000 7f983e7fc640 1 -- 172.21.15.192:0/3363210744 learned_addr learned my addr 172.21.15.192:0/3363210744 (peer_addr_for_me v1:172.21.15.192:0/0) 2023-10-30T12:28:27.122 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.120+0000 7f983effd640 1 -- 172.21.15.192:0/3363210744 >> v1:172.21.15.192:6789/0 conn(0x7f98400c2e20 legacy=0x7f98400cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:27.122 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.120+0000 7f983effd640 1 -- 172.21.15.192:0/3363210744 >> v1:172.21.15.89:6789/0 conn(0x7f984015a070 legacy=0x7f9840160e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:27.122 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.120+0000 7f983effd640 1 -- 172.21.15.192:0/3363210744 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9840156b30 con 0x7f9840150800 2023-10-30T12:28:27.123 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.120+0000 7f983effd640 1 --2- 172.21.15.192:0/3363210744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9840150800 0x7f98400c6ae0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f98300097c0 tx=0x7f983001ec30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5dbec51fb82fd06 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:27.123 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.120+0000 7f983dffb640 1 -- 172.21.15.192:0/3363210744 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98300a8930 con 0x7f9840150800 2023-10-30T12:28:27.123 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.120+0000 7f983dffb640 1 -- 172.21.15.192:0/3363210744 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98300a8a90 con 0x7f9840150800 2023-10-30T12:28:27.123 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.120+0000 7f983dffb640 1 -- 172.21.15.192:0/3363210744 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98300a8e20 con 0x7f9840150800 2023-10-30T12:28:27.124 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.120+0000 7f98454fb640 1 -- 172.21.15.192:0/3363210744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9840150800 msgr2=0x7f98400c6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:27.125 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.120+0000 7f98454fb640 1 --2- 172.21.15.192:0/3363210744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9840150800 0x7f98400c6ae0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f98300097c0 tx=0x7f983001ec30 comp rx=0 tx=0).stop 2023-10-30T12:28:27.125 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.120+0000 7f98454fb640 1 -- 172.21.15.192:0/3363210744 shutdown_connections 2023-10-30T12:28:27.125 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.120+0000 7f98454fb640 1 --2- 172.21.15.192:0/3363210744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9840150800 0x7f98400c6ae0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:27.125 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.120+0000 7f98454fb640 1 -- 172.21.15.192:0/3363210744 >> 172.21.15.192:0/3363210744 conn(0x7f984006c850 msgr2=0x7f984006cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:27.125 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.120+0000 7f98454fb640 1 -- 172.21.15.192:0/3363210744 shutdown_connections 2023-10-30T12:28:27.126 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.120+0000 7f98454fb640 1 -- 172.21.15.192:0/3363210744 wait complete. 2023-10-30T12:28:27.126 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.124+0000 7f98454fb640 1 Processor -- start 2023-10-30T12:28:27.126 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.124+0000 7f98454fb640 1 -- start start 2023-10-30T12:28:27.128 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.124+0000 7f98454fb640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98400c2e20 0x7f98400b1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:27.128 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.124+0000 7f98454fb640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9840150800 0x7f984014b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:27.128 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.124+0000 7f98454fb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f984015a070 0x7f98401ef4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:27.129 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.124+0000 7f98454fb640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9840155930 con 0x7f984015a070 2023-10-30T12:28:27.129 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.124+0000 7f98454fb640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f984014be80 con 0x7f9840150800 2023-10-30T12:28:27.129 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.124+0000 7f98454fb640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f984014c830 con 0x7f98400c2e20 2023-10-30T12:28:27.129 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.124+0000 7f983e7fc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9840150800 0x7f984014b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:27.129 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.124+0000 7f983effd640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98400c2e20 0x7f98400b1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:27.130 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.124+0000 7f983effd640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98400c2e20 0x7f98400b1c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.192:60590/0 (socket says 172.21.15.192:60590) 2023-10-30T12:28:27.130 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.124+0000 7f983e7fc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9840150800 0x7f984014b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:42120/0 (socket says 172.21.15.192:42120) 2023-10-30T12:28:27.130 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.124+0000 7f983effd640 1 -- 172.21.15.192:0/2735602533 learned_addr learned my addr 172.21.15.192:0/2735602533 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-10-30T12:28:27.130 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.124+0000 7f983effd640 1 -- 172.21.15.192:0/2735602533 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9840150800 msgr2=0x7f984014b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:27.130 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.124+0000 7f983effd640 1 --2- 172.21.15.192:0/2735602533 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9840150800 0x7f984014b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:27.131 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.124+0000 7f983effd640 1 -- 172.21.15.192:0/2735602533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f984015a070 msgr2=0x7f98401ef4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:27.131 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.124+0000 7f983effd640 1 --2- 172.21.15.192:0/2735602533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f984015a070 0x7f98401ef4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:27.131 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.124+0000 7f983effd640 1 -- 172.21.15.192:0/2735602533 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9830009420 con 0x7f98400c2e20 2023-10-30T12:28:27.131 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.124+0000 7f983e7fc640 1 --2- 172.21.15.192:0/2735602533 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9840150800 0x7f984014b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:27.131 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.124+0000 7f983effd640 1 --2- 172.21.15.192:0/2735602533 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98400c2e20 0x7f98400b1c00 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f983001b840 tx=0x7f983001b9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:27.131 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.124+0000 7f98277fe640 1 -- 172.21.15.192:0/2735602533 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98300a8930 con 0x7f98400c2e20 2023-10-30T12:28:27.132 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.124+0000 7f98277fe640 1 -- 172.21.15.192:0/2735602533 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98300b9410 con 0x7f98400c2e20 2023-10-30T12:28:27.132 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.124+0000 7f98454fb640 1 -- 172.21.15.192:0/2735602533 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f984014cb10 con 0x7f98400c2e20 2023-10-30T12:28:27.132 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.124+0000 7f98277fe640 1 -- 172.21.15.192:0/2735602533 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98300b97a0 con 0x7f98400c2e20 2023-10-30T12:28:27.132 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.124+0000 7f98454fb640 1 -- 172.21.15.192:0/2735602533 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98401552e0 con 0x7f98400c2e20 2023-10-30T12:28:27.132 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.124+0000 7f98277fe640 1 -- 172.21.15.192:0/2735602533 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f98300b7040 con 0x7f98400c2e20 2023-10-30T12:28:27.133 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.124+0000 7f98277fe640 1 -- 172.21.15.192:0/2735602533 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f98300c6d60 con 0x7f98400c2e20 2023-10-30T12:28:27.133 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.124+0000 7f98454fb640 1 -- 172.21.15.192:0/2735602533 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f984004e170 con 0x7f98400c2e20 2023-10-30T12:28:27.133 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.128+0000 7f98277fe640 1 -- 172.21.15.192:0/2735602533 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f98300b5070 con 0x7f98400c2e20 2023-10-30T12:28:27.198 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.196+0000 7f98454fb640 1 -- 172.21.15.192:0/2735602533 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd new", "uuid": "0305670b-d7c4-4193-bec2-8b7351ff590a", "id": 1} v 0) v1 -- 0x7f984004e170 con 0x7f98400c2e20 2023-10-30T12:28:27.202 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-30T12:28:27.200+0000 7fb4e738e180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-30T12:28:27.205 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.204+0000 7f98277fe640 1 -- 172.21.15.192:0/2735602533 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0305670b-d7c4-4193-bec2-8b7351ff590a", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f98300b9d40 con 0x7f98400c2e20 2023-10-30T12:28:27.206 INFO:teuthology.orchestra.run.smithi192.stdout:1 2023-10-30T12:28:27.207 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.204+0000 7f98454fb640 1 -- 172.21.15.192:0/2735602533 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98400c2e20 msgr2=0x7f98400b1c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:27.207 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.204+0000 7f98454fb640 1 --2- 172.21.15.192:0/2735602533 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98400c2e20 0x7f98400b1c00 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f983001b840 tx=0x7f983001b9d0 comp rx=0 tx=0).stop 2023-10-30T12:28:27.207 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.204+0000 7f98454fb640 1 -- 172.21.15.192:0/2735602533 shutdown_connections 2023-10-30T12:28:27.207 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.204+0000 7f98454fb640 1 --2- 172.21.15.192:0/2735602533 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9840150800 0x7f984014b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:27.208 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.204+0000 7f98454fb640 1 --2- 172.21.15.192:0/2735602533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f984015a070 0x7f98401ef4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:27.208 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.204+0000 7f98454fb640 1 --2- 172.21.15.192:0/2735602533 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f98400c2e20 0x7f98400b1c00 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:27.208 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.204+0000 7f98454fb640 1 -- 172.21.15.192:0/2735602533 >> 172.21.15.192:0/2735602533 conn(0x7f984006c850 msgr2=0x7f984006df40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:27.208 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.204+0000 7f98454fb640 1 -- 172.21.15.192:0/2735602533 shutdown_connections 2023-10-30T12:28:27.208 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.204+0000 7f98454fb640 1 -- 172.21.15.192:0/2735602533 wait complete. 2023-10-30T12:28:27.222 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 4bab5426-a3c3-46b0-91fc-546c254006c3 2 2023-10-30T12:28:27.242 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-10-30T12:28:27.237+0000 7f7edc42e180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-30T12:28:27.375 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.372+0000 7fedbdc36640 1 Processor -- start 2023-10-30T12:28:27.375 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.372+0000 7fedbdc36640 1 -- start start 2023-10-30T12:28:27.377 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.376+0000 7fedbdc36640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fedb80c2e20 0x7fedb80c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:27.377 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.376+0000 7fedbdc36640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fedb815a070 0x7fedb815a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:27.377 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.376+0000 7fedbdc36640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb8159be0 con 0x7fedb8150800 2023-10-30T12:28:27.377 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.376+0000 7fedbdc36640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedb80c7020 con 0x7fedb815a070 2023-10-30T12:28:27.378 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.376+0000 7fedbdc36640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedb815aa40 con 0x7fedb80c2e20 2023-10-30T12:28:27.378 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.376+0000 7fedb6ffd640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fedb80c2e20 0x7fedb80c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:27.378 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.376+0000 7fedb77fe640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fedb8150800 0x7fedb80cb170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:48436/0 (socket says 172.21.15.192:48436) 2023-10-30T12:28:27.378 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.376+0000 7fedb6ffd640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fedb80c2e20 0x7fedb80c6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:42134/0 (socket says 172.21.15.192:42134) 2023-10-30T12:28:27.379 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.376+0000 7fedb77fe640 1 -- 172.21.15.192:0/752494097 learned_addr learned my addr 172.21.15.192:0/752494097 (peer_addr_for_me v1:172.21.15.192:0/0) 2023-10-30T12:28:27.379 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.376+0000 7fedb7fff640 1 --2- 172.21.15.192:0/752494097 >> v2:172.21.15.89:3300/0 conn(0x7fedb815a070 0x7fedb815a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:27.379 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.376+0000 7fedb6ffd640 1 -- 172.21.15.192:0/752494097 >> v2:172.21.15.89:3300/0 conn(0x7fedb815a070 msgr2=0x7fedb815a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:27.379 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.376+0000 7fedb6ffd640 1 --2- 172.21.15.192:0/752494097 >> v2:172.21.15.89:3300/0 conn(0x7fedb815a070 0x7fedb815a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:27.379 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.376+0000 7fedb6ffd640 1 -- 172.21.15.192:0/752494097 >> v1:172.21.15.192:6789/0 conn(0x7fedb8150800 legacy=0x7fedb80cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:27.379 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.376+0000 7fedb67fc640 1 -- 172.21.15.192:0/752494097 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1831051453 0 0) 0x7fedb8159be0 con 0x7fedb8150800 2023-10-30T12:28:27.380 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.376+0000 7fedb6ffd640 1 -- 172.21.15.192:0/752494097 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb8156850 con 0x7fedb80c2e20 2023-10-30T12:28:27.380 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.376+0000 7fedb7fff640 1 --2- 172.21.15.192:0/752494097 >> v2:172.21.15.89:3300/0 conn(0x7fedb815a070 0x7fedb815a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:28:27.381 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.376+0000 7fedb6ffd640 1 --2- 172.21.15.192:0/752494097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedb80c2e20 0x7fedb80c6ae0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fedb80c9a70 tx=0x7feda001ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa5184309238da6b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:27.381 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.376+0000 7fedb67fc640 1 -- 172.21.15.192:0/752494097 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda00a8950 con 0x7fedb80c2e20 2023-10-30T12:28:27.382 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.376+0000 7fedb67fc640 1 -- 172.21.15.192:0/752494097 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feda00a8ab0 con 0x7fedb80c2e20 2023-10-30T12:28:27.382 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.376+0000 7fedb67fc640 1 -- 172.21.15.192:0/752494097 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda00a8e40 con 0x7fedb80c2e20 2023-10-30T12:28:27.382 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.376+0000 7fedbdc36640 1 -- 172.21.15.192:0/752494097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedb80c2e20 msgr2=0x7fedb80c6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:27.382 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.376+0000 7fedbdc36640 1 --2- 172.21.15.192:0/752494097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedb80c2e20 0x7fedb80c6ae0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fedb80c9a70 tx=0x7feda001ec50 comp rx=0 tx=0).stop 2023-10-30T12:28:27.382 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.376+0000 7fedbdc36640 1 -- 172.21.15.192:0/752494097 shutdown_connections 2023-10-30T12:28:27.383 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.376+0000 7fedbdc36640 1 --2- 172.21.15.192:0/752494097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedb80c2e20 0x7fedb80c6ae0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:27.383 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.376+0000 7fedbdc36640 1 --2- 172.21.15.192:0/752494097 >> v2:172.21.15.89:3300/0 conn(0x7fedb815a070 0x7fedb815a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:27.383 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.376+0000 7fedbdc36640 1 -- 172.21.15.192:0/752494097 >> 172.21.15.192:0/752494097 conn(0x7fedb806c850 msgr2=0x7fedb806cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:27.383 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.376+0000 7fedbdc36640 1 -- 172.21.15.192:0/752494097 shutdown_connections 2023-10-30T12:28:27.383 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.376+0000 7fedbdc36640 1 -- 172.21.15.192:0/752494097 wait complete. 2023-10-30T12:28:27.384 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.380+0000 7fedbdc36640 1 Processor -- start 2023-10-30T12:28:27.384 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.380+0000 7fedbdc36640 1 -- start start 2023-10-30T12:28:27.386 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.380+0000 7fedbdc36640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedb80c2e20 0x7fedb8141030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:27.386 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.380+0000 7fedbdc36640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedb8150800 0x7fedb814f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:27.386 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.380+0000 7fedbdc36640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fedb815a070 0x7fedb814fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:27.387 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.380+0000 7fedbdc36640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb8157be0 con 0x7fedb8150800 2023-10-30T12:28:27.387 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.380+0000 7fedbdc36640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb81552f0 con 0x7fedb80c2e20 2023-10-30T12:28:27.387 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.380+0000 7fedbdc36640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedb814e030 con 0x7fedb815a070 2023-10-30T12:28:27.387 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.380+0000 7fedb77fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedb80c2e20 0x7fedb8141030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:27.387 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.380+0000 7fedb7fff640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fedb815a070 0x7fedb814fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:27.388 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.380+0000 7fedb77fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedb80c2e20 0x7fedb8141030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:42148/0 (socket says 172.21.15.192:42148) 2023-10-30T12:28:27.388 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.380+0000 7fedb7fff640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fedb815a070 0x7fedb814fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.192:60592/0 (socket says 172.21.15.192:60592) 2023-10-30T12:28:27.388 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.380+0000 7fedb77fe640 1 -- 172.21.15.192:0/1842412701 learned_addr learned my addr 172.21.15.192:0/1842412701 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-10-30T12:28:27.388 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.380+0000 7fedb6ffd640 1 --2- 172.21.15.192:0/1842412701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedb8150800 0x7fedb814f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:27.388 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.380+0000 7fedb77fe640 1 -- 172.21.15.192:0/1842412701 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fedb815a070 msgr2=0x7fedb814fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:27.389 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.380+0000 7fedb77fe640 1 --2- 172.21.15.192:0/1842412701 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fedb815a070 0x7fedb814fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:27.389 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.380+0000 7fedb77fe640 1 -- 172.21.15.192:0/1842412701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedb8150800 msgr2=0x7fedb814f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:27.389 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.380+0000 7fedb77fe640 1 --2- 172.21.15.192:0/1842412701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedb8150800 0x7fedb814f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:27.389 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.380+0000 7fedb77fe640 1 -- 172.21.15.192:0/1842412701 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda0009420 con 0x7fedb80c2e20 2023-10-30T12:28:27.389 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.380+0000 7fedb7fff640 1 --2- 172.21.15.192:0/1842412701 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fedb815a070 0x7fedb814fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:27.389 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.380+0000 7fedb77fe640 1 --2- 172.21.15.192:0/1842412701 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedb80c2e20 0x7fedb8141030 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7feda800c890 tx=0x7feda800cd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:27.390 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.380+0000 7fedaffff640 1 -- 172.21.15.192:0/1842412701 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda800a640 con 0x7fedb80c2e20 2023-10-30T12:28:27.390 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.380+0000 7fedaffff640 1 -- 172.21.15.192:0/1842412701 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feda800a7c0 con 0x7fedb80c2e20 2023-10-30T12:28:27.390 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.380+0000 7fedbdc36640 1 -- 172.21.15.192:0/1842412701 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedb814e310 con 0x7fedb80c2e20 2023-10-30T12:28:27.390 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.380+0000 7fedaffff640 1 -- 172.21.15.192:0/1842412701 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda800ab70 con 0x7fedb80c2e20 2023-10-30T12:28:27.391 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.380+0000 7fedbdc36640 1 -- 172.21.15.192:0/1842412701 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedb8143910 con 0x7fedb80c2e20 2023-10-30T12:28:27.391 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.380+0000 7fedaffff640 1 -- 172.21.15.192:0/1842412701 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7feda80070d0 con 0x7fedb80c2e20 2023-10-30T12:28:27.391 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.380+0000 7fedaffff640 1 -- 172.21.15.192:0/1842412701 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7feda8012820 con 0x7fedb80c2e20 2023-10-30T12:28:27.391 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.380+0000 7fedbdc36640 1 -- 172.21.15.192:0/1842412701 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedb804e170 con 0x7fedb80c2e20 2023-10-30T12:28:27.391 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.384+0000 7fedaffff640 1 -- 172.21.15.192:0/1842412701 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7feda8018070 con 0x7fedb80c2e20 2023-10-30T12:28:27.453 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.452+0000 7fedbdc36640 1 -- 172.21.15.192:0/1842412701 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4bab5426-a3c3-46b0-91fc-546c254006c3", "id": 2} v 0) v1 -- 0x7fedb814f380 con 0x7fedb80c2e20 2023-10-30T12:28:27.461 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.460+0000 7fedaffff640 1 -- 172.21.15.192:0/1842412701 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4bab5426-a3c3-46b0-91fc-546c254006c3", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7feda8012530 con 0x7fedb80c2e20 2023-10-30T12:28:27.461 INFO:teuthology.orchestra.run.smithi192.stdout:2 2023-10-30T12:28:27.462 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.460+0000 7fedbdc36640 1 -- 172.21.15.192:0/1842412701 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedb80c2e20 msgr2=0x7fedb8141030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:27.462 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.460+0000 7fedbdc36640 1 --2- 172.21.15.192:0/1842412701 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedb80c2e20 0x7fedb8141030 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7feda800c890 tx=0x7feda800cd50 comp rx=0 tx=0).stop 2023-10-30T12:28:27.463 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.460+0000 7fedbdc36640 1 -- 172.21.15.192:0/1842412701 shutdown_connections 2023-10-30T12:28:27.463 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.460+0000 7fedbdc36640 1 --2- 172.21.15.192:0/1842412701 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fedb815a070 0x7fedb814fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:27.463 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.460+0000 7fedbdc36640 1 --2- 172.21.15.192:0/1842412701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedb8150800 0x7fedb814f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:27.463 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.460+0000 7fedbdc36640 1 --2- 172.21.15.192:0/1842412701 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedb80c2e20 0x7fedb8141030 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:27.463 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.460+0000 7fedbdc36640 1 -- 172.21.15.192:0/1842412701 >> 172.21.15.192:0/1842412701 conn(0x7fedb806c850 msgr2=0x7fedb80cb650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:27.464 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.460+0000 7fedbdc36640 1 -- 172.21.15.192:0/1842412701 shutdown_connections 2023-10-30T12:28:27.464 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.460+0000 7fedbdc36640 1 -- 172.21.15.192:0/1842412701 wait complete. 2023-10-30T12:28:27.477 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 7d47aea8-27cd-42be-8591-d63ad68c90e2 3 2023-10-30T12:28:27.589 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.588+0000 7fd4b4c3e640 1 Processor -- start 2023-10-30T12:28:27.589 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.588+0000 7fd4b4c3e640 1 -- start start 2023-10-30T12:28:27.591 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.588+0000 7fd4b4c3e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd4b0151900 0x7fd4b00c22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:27.591 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.588+0000 7fd4b4c3e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4b006e570 0x7fd4b006d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:27.591 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.588+0000 7fd4b4c3e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4b006e360 con 0x7fd4b00c79a0 2023-10-30T12:28:27.591 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.588+0000 7fd4b4c3e640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4b006eab0 con 0x7fd4b0151900 2023-10-30T12:28:27.592 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.588+0000 7fd4b4c3e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b00c7480 con 0x7fd4b006e570 2023-10-30T12:28:27.592 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.588+0000 7fd4a7fff640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4b006e570 0x7fd4b006d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:27.592 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.588+0000 7fd4ae59c640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd4b0151900 0x7fd4b00c22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:27.592 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.588+0000 7fd4ae59c640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd4b0151900 0x7fd4b00c22f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:42150/0 (socket says 172.21.15.192:42150) 2023-10-30T12:28:27.592 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.588+0000 7fd4aed9d640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fd4b00c79a0 0x7fd4b006f1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:48440/0 (socket says 172.21.15.192:48440) 2023-10-30T12:28:27.593 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.588+0000 7fd4a7fff640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4b006e570 0x7fd4b006d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.192:60600/0 (socket says 172.21.15.192:60600) 2023-10-30T12:28:27.593 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.588+0000 7fd4ae59c640 1 -- 172.21.15.192:0/2167649661 learned_addr learned my addr 172.21.15.192:0/2167649661 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-10-30T12:28:27.593 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.588+0000 7fd4ae59c640 1 -- 172.21.15.192:0/2167649661 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4b006e570 msgr2=0x7fd4b006d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:27.593 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.588+0000 7fd4ae59c640 1 --2- 172.21.15.192:0/2167649661 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4b006e570 0x7fd4b006d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:27.594 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.588+0000 7fd4add9b640 1 -- 172.21.15.192:0/2167649661 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2793481006 0 0) 0x7fd4b006e360 con 0x7fd4b00c79a0 2023-10-30T12:28:27.594 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.588+0000 7fd4ae59c640 1 -- 172.21.15.192:0/2167649661 >> v1:172.21.15.192:6789/0 conn(0x7fd4b00c79a0 legacy=0x7fd4b006f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:27.594 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.588+0000 7fd4ae59c640 1 -- 172.21.15.192:0/2167649661 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b006d6f0 con 0x7fd4b0151900 2023-10-30T12:28:27.594 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.588+0000 7fd4a7fff640 1 --2- 172.21.15.192:0/2167649661 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4b006e570 0x7fd4b006d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:27.598 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4ae59c640 1 --2- 172.21.15.192:0/2167649661 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4b0151900 0x7fd4b00c22f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd498009870 tx=0x7fd49801c510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52f00da0d05eaf36 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:27.598 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4add9b640 1 -- 172.21.15.192:0/2167649661 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4980aa430 con 0x7fd4b0151900 2023-10-30T12:28:27.598 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4add9b640 1 -- 172.21.15.192:0/2167649661 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4980aa590 con 0x7fd4b0151900 2023-10-30T12:28:27.598 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4add9b640 1 -- 172.21.15.192:0/2167649661 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4980aa920 con 0x7fd4b0151900 2023-10-30T12:28:27.599 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4b4c3e640 1 -- 172.21.15.192:0/2167649661 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4b0151900 msgr2=0x7fd4b00c22f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:27.599 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4b4c3e640 1 --2- 172.21.15.192:0/2167649661 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4b0151900 0x7fd4b00c22f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd498009870 tx=0x7fd49801c510 comp rx=0 tx=0).stop 2023-10-30T12:28:27.599 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4b4c3e640 1 -- 172.21.15.192:0/2167649661 shutdown_connections 2023-10-30T12:28:27.599 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4b4c3e640 1 --2- 172.21.15.192:0/2167649661 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4b006e570 0x7fd4b006d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:27.599 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4b4c3e640 1 --2- 172.21.15.192:0/2167649661 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4b0151900 0x7fd4b00c22f0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:27.600 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4b4c3e640 1 -- 172.21.15.192:0/2167649661 >> 172.21.15.192:0/2167649661 conn(0x7fd4b00ccfe0 msgr2=0x7fd4b00cd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:27.600 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4b4c3e640 1 -- 172.21.15.192:0/2167649661 shutdown_connections 2023-10-30T12:28:27.600 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4b4c3e640 1 -- 172.21.15.192:0/2167649661 wait complete. 2023-10-30T12:28:27.600 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4b4c3e640 1 Processor -- start 2023-10-30T12:28:27.600 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4b4c3e640 1 -- start start 2023-10-30T12:28:27.601 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4b4c3e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4b006e570 0x7fd4b0141b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:27.601 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4b4c3e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4b00c79a0 0x7fd4b014ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:27.601 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4b4c3e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4b0151900 0x7fd4b014bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:27.601 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4b4c3e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b0142090 con 0x7fd4b0151900 2023-10-30T12:28:27.601 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4b4c3e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b00c5fd0 con 0x7fd4b006e570 2023-10-30T12:28:27.601 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4b4c3e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b0150740 con 0x7fd4b00c79a0 2023-10-30T12:28:27.602 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4ae59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4b006e570 0x7fd4b0141b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:27.602 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4ae59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4b006e570 0x7fd4b0141b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:42152/0 (socket says 172.21.15.192:42152) 2023-10-30T12:28:27.602 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4ae59c640 1 -- 172.21.15.192:0/2087842925 learned_addr learned my addr 172.21.15.192:0/2087842925 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-10-30T12:28:27.602 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4a7fff640 1 --2- 172.21.15.192:0/2087842925 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4b00c79a0 0x7fd4b014ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:27.602 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4aed9d640 1 --2- 172.21.15.192:0/2087842925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4b0151900 0x7fd4b014bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:27.603 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4ae59c640 1 -- 172.21.15.192:0/2087842925 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4b00c79a0 msgr2=0x7fd4b014ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:27.603 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4ae59c640 1 --2- 172.21.15.192:0/2087842925 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4b00c79a0 0x7fd4b014ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:27.603 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4ae59c640 1 -- 172.21.15.192:0/2087842925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4b0151900 msgr2=0x7fd4b014bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:27.603 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4ae59c640 1 --2- 172.21.15.192:0/2087842925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4b0151900 0x7fd4b014bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:27.604 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4ae59c640 1 -- 172.21.15.192:0/2087842925 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a0009420 con 0x7fd4b006e570 2023-10-30T12:28:27.604 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.592+0000 7fd4aed9d640 1 --2- 172.21.15.192:0/2087842925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4b0151900 0x7fd4b014bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:28:27.604 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.596+0000 7fd4ae59c640 1 --2- 172.21.15.192:0/2087842925 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4b006e570 0x7fd4b0141b50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd49801cdb0 tx=0x7fd49801ad50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:27.604 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.596+0000 7fd4a6ffd640 1 -- 172.21.15.192:0/2087842925 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4980aa430 con 0x7fd4b006e570 2023-10-30T12:28:27.604 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.596+0000 7fd4a6ffd640 1 -- 172.21.15.192:0/2087842925 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4980b3820 con 0x7fd4b006e570 2023-10-30T12:28:27.604 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.596+0000 7fd4b4c3e640 1 -- 172.21.15.192:0/2087842925 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4980094d0 con 0x7fd4b006e570 2023-10-30T12:28:27.605 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.596+0000 7fd4a6ffd640 1 -- 172.21.15.192:0/2087842925 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4980b3bd0 con 0x7fd4b006e570 2023-10-30T12:28:27.605 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.596+0000 7fd4b4c3e640 1 -- 172.21.15.192:0/2087842925 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4b01f0fc0 con 0x7fd4b006e570 2023-10-30T12:28:27.605 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.596+0000 7fd4a6ffd640 1 -- 172.21.15.192:0/2087842925 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd4980bd6f0 con 0x7fd4b006e570 2023-10-30T12:28:27.605 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.596+0000 7fd4a6ffd640 1 -- 172.21.15.192:0/2087842925 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7fd4980b2810 con 0x7fd4b006e570 2023-10-30T12:28:27.605 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.596+0000 7fd4b4c3e640 1 -- 172.21.15.192:0/2087842925 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4b0142a90 con 0x7fd4b006e570 2023-10-30T12:28:27.606 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.596+0000 7fd4a6ffd640 1 -- 172.21.15.192:0/2087842925 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fd4980b2380 con 0x7fd4b006e570 2023-10-30T12:28:27.665 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.664+0000 7fd4b4c3e640 1 -- 172.21.15.192:0/2087842925 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7d47aea8-27cd-42be-8591-d63ad68c90e2", "id": 3} v 0) v1 -- 0x7fd4b004e170 con 0x7fd4b006e570 2023-10-30T12:28:27.669 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.668+0000 7fd4a6ffd640 1 -- 172.21.15.192:0/2087842925 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7d47aea8-27cd-42be-8591-d63ad68c90e2", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd4980b25e0 con 0x7fd4b006e570 2023-10-30T12:28:27.670 INFO:teuthology.orchestra.run.smithi192.stdout:3 2023-10-30T12:28:27.671 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.668+0000 7fd4b4c3e640 1 -- 172.21.15.192:0/2087842925 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4b006e570 msgr2=0x7fd4b0141b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:27.671 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.668+0000 7fd4b4c3e640 1 --2- 172.21.15.192:0/2087842925 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4b006e570 0x7fd4b0141b50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd49801cdb0 tx=0x7fd49801ad50 comp rx=0 tx=0).stop 2023-10-30T12:28:27.672 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.668+0000 7fd4b4c3e640 1 -- 172.21.15.192:0/2087842925 shutdown_connections 2023-10-30T12:28:27.672 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.668+0000 7fd4b4c3e640 1 --2- 172.21.15.192:0/2087842925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4b0151900 0x7fd4b014bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:27.672 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.668+0000 7fd4b4c3e640 1 --2- 172.21.15.192:0/2087842925 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4b00c79a0 0x7fd4b014ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:27.672 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.668+0000 7fd4b4c3e640 1 --2- 172.21.15.192:0/2087842925 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4b006e570 0x7fd4b0141b50 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:27.672 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.668+0000 7fd4b4c3e640 1 -- 172.21.15.192:0/2087842925 >> 172.21.15.192:0/2087842925 conn(0x7fd4b00ccfe0 msgr2=0x7fd4b00c9030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:27.673 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.668+0000 7fd4b4c3e640 1 -- 172.21.15.192:0/2087842925 shutdown_connections 2023-10-30T12:28:27.673 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.668+0000 7fd4b4c3e640 1 -- 172.21.15.192:0/2087842925 wait complete. 2023-10-30T12:28:27.687 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 2e2ae8f8-d097-4bba-9448-95047fbfacad 4 2023-10-30T12:28:27.784 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-30T12:28:27.784+0000 7fb4e738e180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-30T12:28:27.803 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.800+0000 7f311f261640 1 Processor -- start 2023-10-30T12:28:27.803 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.800+0000 7f311f261640 1 -- start start 2023-10-30T12:28:27.805 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.804+0000 7f311f261640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311806e570 0x7f31180c22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:27.805 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.804+0000 7f311f261640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311806e360 con 0x7f3118151900 2023-10-30T12:28:27.805 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.804+0000 7f311f261640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31180c7480 con 0x7f31180c79a0 2023-10-30T12:28:27.806 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.804+0000 7f311f261640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311806eab0 con 0x7f311806e570 2023-10-30T12:28:27.806 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.804+0000 7f3117fff640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311806e570 0x7f31180c22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:27.806 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.804+0000 7f311d7fe640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f31180c79a0 0x7f311806f1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:48446/0 (socket says 172.21.15.192:48446) 2023-10-30T12:28:27.806 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.804+0000 7f311d7fe640 1 -- 172.21.15.192:0/264397127 learned_addr learned my addr 172.21.15.192:0/264397127 (peer_addr_for_me v1:172.21.15.192:0/0) 2023-10-30T12:28:27.806 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.804+0000 7f3117fff640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311806e570 0x7f31180c22f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.192:60628/0 (socket says 172.21.15.192:60628) 2023-10-30T12:28:27.807 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.804+0000 7f3117fff640 1 -- 172.21.15.192:0/264397127 >> v1:172.21.15.192:6789/0 conn(0x7f31180c79a0 legacy=0x7f311806f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:27.807 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.804+0000 7f3117fff640 1 -- 172.21.15.192:0/264397127 >> v1:172.21.15.89:6789/0 conn(0x7f3118151900 legacy=0x7f311806d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:27.807 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.804+0000 7f3117fff640 1 -- 172.21.15.192:0/264397127 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3118158320 con 0x7f311806e570 2023-10-30T12:28:27.808 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.804+0000 7f3117fff640 1 --2- 172.21.15.192:0/264397127 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311806e570 0x7f31180c22f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f31040097f0 tx=0x7f310401c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ff5c59c612b8d5b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:27.808 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.804+0000 7f31177fe640 1 -- 172.21.15.192:0/264397127 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31040ac600 con 0x7f311806e570 2023-10-30T12:28:27.809 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.804+0000 7f31177fe640 1 -- 172.21.15.192:0/264397127 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31040a8050 con 0x7f311806e570 2023-10-30T12:28:27.809 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.804+0000 7f31177fe640 1 -- 172.21.15.192:0/264397127 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31040ac980 con 0x7f311806e570 2023-10-30T12:28:27.809 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.804+0000 7f311f261640 1 -- 172.21.15.192:0/264397127 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311806e570 msgr2=0x7f31180c22f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:27.809 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.804+0000 7f311f261640 1 --2- 172.21.15.192:0/264397127 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311806e570 0x7f31180c22f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f31040097f0 tx=0x7f310401c490 comp rx=0 tx=0).stop 2023-10-30T12:28:27.809 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.804+0000 7f311f261640 1 -- 172.21.15.192:0/264397127 shutdown_connections 2023-10-30T12:28:27.810 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.804+0000 7f311f261640 1 --2- 172.21.15.192:0/264397127 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311806e570 0x7f31180c22f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:27.810 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.804+0000 7f311f261640 1 -- 172.21.15.192:0/264397127 >> 172.21.15.192:0/264397127 conn(0x7f31180ccfe0 msgr2=0x7f31180cd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:27.810 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.808+0000 7f311f261640 1 -- 172.21.15.192:0/264397127 shutdown_connections 2023-10-30T12:28:27.810 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.808+0000 7f311f261640 1 -- 172.21.15.192:0/264397127 wait complete. 2023-10-30T12:28:27.813 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.808+0000 7f311f261640 1 Processor -- start 2023-10-30T12:28:27.813 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.808+0000 7f311f261640 1 -- start start 2023-10-30T12:28:27.813 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.808+0000 7f311f261640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f311806e570 0x7f31180c49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:27.813 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.808+0000 7f311f261640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31180c79a0 0x7f3118155560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:27.813 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.808+0000 7f311f261640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3118151900 0x7f31181f5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:27.814 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.808+0000 7f311f261640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311814e280 con 0x7f3118151900 2023-10-30T12:28:27.814 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.808+0000 7f311f261640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31180c4ef0 con 0x7f311806e570 2023-10-30T12:28:27.814 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.808+0000 7f311f261640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311806b6a0 con 0x7f31180c79a0 2023-10-30T12:28:27.814 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.808+0000 7f3117fff640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31180c79a0 0x7f3118155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:27.814 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.808+0000 7f311cffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f311806e570 0x7f31180c49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:27.815 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.808+0000 7f3117fff640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31180c79a0 0x7f3118155560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.192:60640/0 (socket says 172.21.15.192:60640) 2023-10-30T12:28:27.815 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.808+0000 7f3117fff640 1 -- 172.21.15.192:0/2292353228 learned_addr learned my addr 172.21.15.192:0/2292353228 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-10-30T12:28:27.815 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.808+0000 7f311d7fe640 1 --2- 172.21.15.192:0/2292353228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3118151900 0x7f31181f5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:27.815 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.808+0000 7f3117fff640 1 -- 172.21.15.192:0/2292353228 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f311806e570 msgr2=0x7f31180c49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:27.815 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.808+0000 7f3117fff640 1 --2- 172.21.15.192:0/2292353228 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f311806e570 0x7f31180c49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:27.816 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.808+0000 7f3117fff640 1 -- 172.21.15.192:0/2292353228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3118151900 msgr2=0x7f31181f5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:27.816 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.808+0000 7f3117fff640 1 --2- 172.21.15.192:0/2292353228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3118151900 0x7f31181f5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:27.816 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.808+0000 7f3117fff640 1 -- 172.21.15.192:0/2292353228 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3104009420 con 0x7f31180c79a0 2023-10-30T12:28:27.816 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.808+0000 7f311d7fe640 1 --2- 172.21.15.192:0/2292353228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3118151900 0x7f31181f5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:27.816 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.808+0000 7f311cffd640 1 --2- 172.21.15.192:0/2292353228 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f311806e570 0x7f31180c49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:27.817 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.808+0000 7f3117fff640 1 --2- 172.21.15.192:0/2292353228 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31180c79a0 0x7f3118155560 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f310401a0c0 tx=0x7f310401acd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:27.817 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.808+0000 7f31157fa640 1 -- 172.21.15.192:0/2292353228 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31040b5830 con 0x7f31180c79a0 2023-10-30T12:28:27.817 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.808+0000 7f31157fa640 1 -- 172.21.15.192:0/2292353228 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31040b59b0 con 0x7f31180c79a0 2023-10-30T12:28:27.817 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.808+0000 7f311f261640 1 -- 172.21.15.192:0/2292353228 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3118151e60 con 0x7f31180c79a0 2023-10-30T12:28:27.817 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.808+0000 7f31157fa640 1 -- 172.21.15.192:0/2292353228 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31040b5d60 con 0x7f31180c79a0 2023-10-30T12:28:27.818 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.808+0000 7f311f261640 1 -- 172.21.15.192:0/2292353228 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f311806b3c0 con 0x7f31180c79a0 2023-10-30T12:28:27.818 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.808+0000 7f31157fa640 1 -- 172.21.15.192:0/2292353228 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f31040b3070 con 0x7f31180c79a0 2023-10-30T12:28:27.818 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.808+0000 7f31157fa640 1 -- 172.21.15.192:0/2292353228 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f31040b68e0 con 0x7f31180c79a0 2023-10-30T12:28:27.818 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.812+0000 7f311f261640 1 -- 172.21.15.192:0/2292353228 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f311804e170 con 0x7f31180c79a0 2023-10-30T12:28:27.819 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.812+0000 7f31157fa640 1 -- 172.21.15.192:0/2292353228 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f31040b1070 con 0x7f31180c79a0 2023-10-30T12:28:27.853 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-30T12:28:27.852+0000 7fb4e738e180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-30T12:28:27.875 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-10-30T12:28:27.873+0000 7f7edc42e180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-30T12:28:27.891 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.888+0000 7f311f261640 1 -- 172.21.15.192:0/2292353228 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd new", "uuid": "2e2ae8f8-d097-4bba-9448-95047fbfacad", "id": 4} v 0) v1 -- 0x7f31180c5f20 con 0x7f31180c79a0 2023-10-30T12:28:27.897 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.896+0000 7f31157fa640 1 -- 172.21.15.192:0/2292353228 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2e2ae8f8-d097-4bba-9448-95047fbfacad", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f31040b6c00 con 0x7f31180c79a0 2023-10-30T12:28:27.898 INFO:teuthology.orchestra.run.smithi192.stdout:4 2023-10-30T12:28:27.899 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.896+0000 7f311f261640 1 -- 172.21.15.192:0/2292353228 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31180c79a0 msgr2=0x7f3118155560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:27.899 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.896+0000 7f311f261640 1 --2- 172.21.15.192:0/2292353228 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31180c79a0 0x7f3118155560 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f310401a0c0 tx=0x7f310401acd0 comp rx=0 tx=0).stop 2023-10-30T12:28:27.899 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.896+0000 7f311f261640 1 -- 172.21.15.192:0/2292353228 shutdown_connections 2023-10-30T12:28:27.899 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.896+0000 7f311f261640 1 --2- 172.21.15.192:0/2292353228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3118151900 0x7f31181f5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:27.900 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.896+0000 7f311f261640 1 --2- 172.21.15.192:0/2292353228 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31180c79a0 0x7f3118155560 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:27.900 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.896+0000 7f311f261640 1 --2- 172.21.15.192:0/2292353228 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f311806e570 0x7f31180c49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:27.900 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.896+0000 7f311f261640 1 -- 172.21.15.192:0/2292353228 >> 172.21.15.192:0/2292353228 conn(0x7f31180ccfe0 msgr2=0x7f311814fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:27.900 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.896+0000 7f311f261640 1 -- 172.21.15.192:0/2292353228 shutdown_connections 2023-10-30T12:28:27.900 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:27.896+0000 7f311f261640 1 -- 172.21.15.192:0/2292353228 wait complete. 2023-10-30T12:28:27.910 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-30T12:28:27.908+0000 7fb4e738e180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-30T12:28:27.914 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 49be5ce5-12f4-4c26-8d39-864b6ff67313 5 2023-10-30T12:28:27.937 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-10-30T12:28:27.933+0000 7f7edc42e180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-30T12:28:27.994 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-30T12:28:27.992+0000 7fb4e738e180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-30T12:28:27.997 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-10-30T12:28:27.993+0000 7f7edc42e180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-30T12:28:28.019 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.016+0000 7f0276cf7640 1 Processor -- start 2023-10-30T12:28:28.020 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.016+0000 7f0276cf7640 1 -- start start 2023-10-30T12:28:28.021 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.020+0000 7f0276cf7640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f02700cbb50 0x7f027015ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:28.021 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.020+0000 7f0276cf7640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f027006fe90 0x7f027015d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:28.021 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.020+0000 7f0276cf7640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02700c6650 con 0x7f02700cb770 2023-10-30T12:28:28.022 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.020+0000 7f0276cf7640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0270159750 con 0x7f02700cbb50 2023-10-30T12:28:28.022 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.020+0000 7f0276cf7640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0270159940 con 0x7f027006fe90 2023-10-30T12:28:28.022 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.020+0000 7f026ffff640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f02700cbb50 0x7f027015ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:28.022 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.020+0000 7f0275294640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f027006fe90 0x7f027015d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:28.022 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.020+0000 7f0275294640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f027006fe90 0x7f027015d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.192:60646/0 (socket says 172.21.15.192:60646) 2023-10-30T12:28:28.023 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.020+0000 7f026ffff640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f02700cbb50 0x7f027015ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:42162/0 (socket says 172.21.15.192:42162) 2023-10-30T12:28:28.023 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.020+0000 7f0274a93640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f02700cb770 0x7f02700c5f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:48456/0 (socket says 172.21.15.192:48456) 2023-10-30T12:28:28.023 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.020+0000 7f0275294640 1 -- 172.21.15.192:0/755224362 learned_addr learned my addr 172.21.15.192:0/755224362 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-10-30T12:28:28.023 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.020+0000 7f0275294640 1 -- 172.21.15.192:0/755224362 >> v2:172.21.15.192:3300/0 conn(0x7f02700cbb50 msgr2=0x7f027015ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:28.023 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.020+0000 7f0275294640 1 --2- 172.21.15.192:0/755224362 >> v2:172.21.15.192:3300/0 conn(0x7f02700cbb50 0x7f027015ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.024 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.020+0000 7f0275294640 1 -- 172.21.15.192:0/755224362 >> v1:172.21.15.192:6789/0 conn(0x7f02700cb770 legacy=0x7f02700c5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:28.024 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.020+0000 7f0275294640 1 -- 172.21.15.192:0/755224362 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0270159c20 con 0x7f027006fe90 2023-10-30T12:28:28.024 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.020+0000 7f026f7fe640 1 -- 172.21.15.192:0/755224362 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1959499027 0 0) 0x7f02700c6650 con 0x7f02700cb770 2023-10-30T12:28:28.024 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.020+0000 7f026ffff640 1 --2- 172.21.15.192:0/755224362 >> v2:172.21.15.192:3300/0 conn(0x7f02700cbb50 0x7f027015ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:28.027 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.020+0000 7f0275294640 1 --2- 172.21.15.192:0/755224362 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f027006fe90 0x7f027015d380 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0268009e50 tx=0x7f026801f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa6d3233244357bf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:28.028 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.020+0000 7f026f7fe640 1 -- 172.21.15.192:0/755224362 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02680af400 con 0x7f027006fe90 2023-10-30T12:28:28.028 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.020+0000 7f026f7fe640 1 -- 172.21.15.192:0/755224362 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02680ab050 con 0x7f027006fe90 2023-10-30T12:28:28.028 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.020+0000 7f026f7fe640 1 -- 172.21.15.192:0/755224362 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02680af780 con 0x7f027006fe90 2023-10-30T12:28:28.028 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.020+0000 7f0276cf7640 1 -- 172.21.15.192:0/755224362 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f027006fe90 msgr2=0x7f027015d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:28.028 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.020+0000 7f0276cf7640 1 --2- 172.21.15.192:0/755224362 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f027006fe90 0x7f027015d380 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0268009e50 tx=0x7f026801f1f0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.029 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.020+0000 7f0276cf7640 1 -- 172.21.15.192:0/755224362 shutdown_connections 2023-10-30T12:28:28.029 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.020+0000 7f0276cf7640 1 --2- 172.21.15.192:0/755224362 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f027006fe90 0x7f027015d380 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.029 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.020+0000 7f0276cf7640 1 --2- 172.21.15.192:0/755224362 >> v2:172.21.15.192:3300/0 conn(0x7f02700cbb50 0x7f027015ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.029 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.020+0000 7f0276cf7640 1 -- 172.21.15.192:0/755224362 >> 172.21.15.192:0/755224362 conn(0x7f027006c620 msgr2=0x7f0270069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:28.029 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.020+0000 7f0276cf7640 1 -- 172.21.15.192:0/755224362 shutdown_connections 2023-10-30T12:28:28.030 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.020+0000 7f0276cf7640 1 -- 172.21.15.192:0/755224362 wait complete. 2023-10-30T12:28:28.030 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.024+0000 7f0276cf7640 1 Processor -- start 2023-10-30T12:28:28.030 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.024+0000 7f0276cf7640 1 -- start start 2023-10-30T12:28:28.030 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.024+0000 7f0276cf7640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f027006fe90 0x7f02701f2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:28.030 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.024+0000 7f0276cf7640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02700cb770 0x7f027014d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:28.031 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.024+0000 7f0276cf7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02700cbb50 0x7f027014f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:28.031 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.024+0000 7f0276cf7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02701547e0 con 0x7f02700cbb50 2023-10-30T12:28:28.031 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.024+0000 7f0276cf7640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02700c4890 con 0x7f02700cb770 2023-10-30T12:28:28.031 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.024+0000 7f0276cf7640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02701f2980 con 0x7f027006fe90 2023-10-30T12:28:28.031 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.024+0000 7f026ffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02700cb770 0x7f027014d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:28.032 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.024+0000 7f0274a93640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f027006fe90 0x7f02701f2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:28.032 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.024+0000 7f026ffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02700cb770 0x7f027014d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:42176/0 (socket says 172.21.15.192:42176) 2023-10-30T12:28:28.032 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.024+0000 7f026ffff640 1 -- 172.21.15.192:0/1241461398 learned_addr learned my addr 172.21.15.192:0/1241461398 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-10-30T12:28:28.032 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.024+0000 7f0275294640 1 --2- 172.21.15.192:0/1241461398 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02700cbb50 0x7f027014f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:28.032 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.024+0000 7f026ffff640 1 -- 172.21.15.192:0/1241461398 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f027006fe90 msgr2=0x7f02701f2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:28.033 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.024+0000 7f026ffff640 1 --2- 172.21.15.192:0/1241461398 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f027006fe90 0x7f02701f2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.033 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.024+0000 7f026ffff640 1 -- 172.21.15.192:0/1241461398 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02700cbb50 msgr2=0x7f027014f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:28.033 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.024+0000 7f026ffff640 1 --2- 172.21.15.192:0/1241461398 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02700cbb50 0x7f027014f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.033 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.024+0000 7f026ffff640 1 -- 172.21.15.192:0/1241461398 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f025c009420 con 0x7f02700cb770 2023-10-30T12:28:28.033 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.024+0000 7f026ffff640 1 --2- 172.21.15.192:0/1241461398 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02700cb770 0x7f027014d290 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f025c00e930 tx=0x7f025c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:28.033 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.024+0000 7f026d7fa640 1 -- 172.21.15.192:0/1241461398 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025c00ca00 con 0x7f02700cb770 2023-10-30T12:28:28.034 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.024+0000 7f026d7fa640 1 -- 172.21.15.192:0/1241461398 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025c00cb60 con 0x7f02700cb770 2023-10-30T12:28:28.034 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.024+0000 7f0276cf7640 1 -- 172.21.15.192:0/1241461398 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0268009b30 con 0x7f02700cb770 2023-10-30T12:28:28.034 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.024+0000 7f026d7fa640 1 -- 172.21.15.192:0/1241461398 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025c013470 con 0x7f02700cb770 2023-10-30T12:28:28.034 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.024+0000 7f0276cf7640 1 -- 172.21.15.192:0/1241461398 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02700c3c80 con 0x7f02700cb770 2023-10-30T12:28:28.034 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.024+0000 7f026d7fa640 1 -- 172.21.15.192:0/1241461398 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f025c005ca0 con 0x7f02700cb770 2023-10-30T12:28:28.035 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.024+0000 7f026d7fa640 1 -- 172.21.15.192:0/1241461398 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f025c0067f0 con 0x7f02700cb770 2023-10-30T12:28:28.035 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.024+0000 7f0276cf7640 1 -- 172.21.15.192:0/1241461398 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f027015a470 con 0x7f02700cb770 2023-10-30T12:28:28.035 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.028+0000 7f026d7fa640 1 -- 172.21.15.192:0/1241461398 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f025c021060 con 0x7f02700cb770 2023-10-30T12:28:28.050 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-30T12:28:28.048+0000 7fb4e738e180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-30T12:28:28.084 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-10-30T12:28:28.081+0000 7f7edc42e180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-30T12:28:28.092 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.092+0000 7f0276cf7640 1 -- 172.21.15.192:0/1241461398 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd new", "uuid": "49be5ce5-12f4-4c26-8d39-864b6ff67313", "id": 5} v 0) v1 -- 0x7f027004e170 con 0x7f02700cb770 2023-10-30T12:28:28.097 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.096+0000 7f026d7fa640 1 -- 172.21.15.192:0/1241461398 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "49be5ce5-12f4-4c26-8d39-864b6ff67313", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f025c017070 con 0x7f02700cb770 2023-10-30T12:28:28.097 INFO:teuthology.orchestra.run.smithi192.stdout:5 2023-10-30T12:28:28.098 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.096+0000 7f0276cf7640 1 -- 172.21.15.192:0/1241461398 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02700cb770 msgr2=0x7f027014d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:28.098 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.096+0000 7f0276cf7640 1 --2- 172.21.15.192:0/1241461398 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02700cb770 0x7f027014d290 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f025c00e930 tx=0x7f025c00edf0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.098 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.096+0000 7f0276cf7640 1 -- 172.21.15.192:0/1241461398 shutdown_connections 2023-10-30T12:28:28.099 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.096+0000 7f0276cf7640 1 --2- 172.21.15.192:0/1241461398 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02700cb770 0x7f027014d290 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.099 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.096+0000 7f0276cf7640 1 --2- 172.21.15.192:0/1241461398 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02700cbb50 0x7f027014f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.099 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.096+0000 7f0276cf7640 1 --2- 172.21.15.192:0/1241461398 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f027006fe90 0x7f02701f2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.099 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.096+0000 7f0276cf7640 1 -- 172.21.15.192:0/1241461398 >> 172.21.15.192:0/1241461398 conn(0x7f027006c620 msgr2=0x7f027006e340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:28.099 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.096+0000 7f0276cf7640 1 -- 172.21.15.192:0/1241461398 shutdown_connections 2023-10-30T12:28:28.100 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.096+0000 7f0276cf7640 1 -- 172.21.15.192:0/1241461398 wait complete. 2023-10-30T12:28:28.113 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 48623452-d456-4ea2-a0de-5fa4b37e6d10 6 2023-10-30T12:28:28.130 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-30T12:28:28.128+0000 7fb4e738e180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-30T12:28:28.144 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-10-30T12:28:28.141+0000 7f7edc42e180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-30T12:28:28.184 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-30T12:28:28.180+0000 7fb4e738e180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-30T12:28:28.223 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.220+0000 7ff7e40a9640 1 Processor -- start 2023-10-30T12:28:28.223 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.220+0000 7ff7e40a9640 1 -- start start 2023-10-30T12:28:28.225 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.220+0000 7ff7e40a9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7dc151900 0x7ff7dc0c22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:28.225 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.220+0000 7ff7e40a9640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff7dc06e570 0x7ff7dc06d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:28.225 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.220+0000 7ff7e40a9640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7dc06e360 con 0x7ff7dc0c79a0 2023-10-30T12:28:28.225 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.220+0000 7ff7e40a9640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7dc06eab0 con 0x7ff7dc06e570 2023-10-30T12:28:28.226 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.220+0000 7ff7e40a9640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7dc0c7480 con 0x7ff7dc151900 2023-10-30T12:28:28.226 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.220+0000 7ff7e1644640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff7dc06e570 0x7ff7dc06d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:28.226 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.220+0000 7ff7e1644640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff7dc06e570 0x7ff7dc06d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:42184/0 (socket says 172.21.15.192:42184) 2023-10-30T12:28:28.226 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.220+0000 7ff7e1644640 1 -- 172.21.15.192:0/2889938408 learned_addr learned my addr 172.21.15.192:0/2889938408 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-10-30T12:28:28.226 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.224+0000 7ff7e1e45640 1 --2- 172.21.15.192:0/2889938408 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7dc151900 0x7ff7dc0c22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:28.227 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.224+0000 7ff7e1644640 1 -- 172.21.15.192:0/2889938408 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7dc151900 msgr2=0x7ff7dc0c22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:28.227 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.224+0000 7ff7e1644640 1 --2- 172.21.15.192:0/2889938408 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7dc151900 0x7ff7dc0c22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.227 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.224+0000 7ff7e1644640 1 -- 172.21.15.192:0/2889938408 >> v1:172.21.15.89:6789/0 conn(0x7ff7dc0c79a0 legacy=0x7ff7dc06f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:28.227 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.224+0000 7ff7e1644640 1 -- 172.21.15.192:0/2889938408 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7dc06d6f0 con 0x7ff7dc06e570 2023-10-30T12:28:28.227 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.224+0000 7ff7e1e45640 1 --2- 172.21.15.192:0/2889938408 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7dc151900 0x7ff7dc0c22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:28.228 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.224+0000 7ff7e1644640 1 --2- 172.21.15.192:0/2889938408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff7dc06e570 0x7ff7dc06d1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc009870 tx=0x7ff7cc01c510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a9b3d4d014078fa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:28.229 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.224+0000 7ff7e0e43640 1 -- 172.21.15.192:0/2889938408 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc0ac640 con 0x7ff7dc06e570 2023-10-30T12:28:28.229 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.224+0000 7ff7e0e43640 1 -- 172.21.15.192:0/2889938408 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7cc0ac7a0 con 0x7ff7dc06e570 2023-10-30T12:28:28.229 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.224+0000 7ff7e0e43640 1 -- 172.21.15.192:0/2889938408 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc0acb30 con 0x7ff7dc06e570 2023-10-30T12:28:28.229 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.224+0000 7ff7e40a9640 1 -- 172.21.15.192:0/2889938408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff7dc06e570 msgr2=0x7ff7dc06d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:28.230 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.224+0000 7ff7e40a9640 1 --2- 172.21.15.192:0/2889938408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff7dc06e570 0x7ff7dc06d1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc009870 tx=0x7ff7cc01c510 comp rx=0 tx=0).stop 2023-10-30T12:28:28.230 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.224+0000 7ff7e40a9640 1 -- 172.21.15.192:0/2889938408 shutdown_connections 2023-10-30T12:28:28.230 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.224+0000 7ff7e40a9640 1 --2- 172.21.15.192:0/2889938408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff7dc06e570 0x7ff7dc06d1b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.230 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.224+0000 7ff7e40a9640 1 --2- 172.21.15.192:0/2889938408 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7dc151900 0x7ff7dc0c22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.230 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.224+0000 7ff7e40a9640 1 -- 172.21.15.192:0/2889938408 >> 172.21.15.192:0/2889938408 conn(0x7ff7dc0ccfe0 msgr2=0x7ff7dc0cd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:28.231 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.224+0000 7ff7e40a9640 1 -- 172.21.15.192:0/2889938408 shutdown_connections 2023-10-30T12:28:28.231 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.224+0000 7ff7e40a9640 1 -- 172.21.15.192:0/2889938408 wait complete. 2023-10-30T12:28:28.233 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.228+0000 7ff7e40a9640 1 Processor -- start 2023-10-30T12:28:28.233 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.228+0000 7ff7e40a9640 1 -- start start 2023-10-30T12:28:28.233 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.228+0000 7ff7e40a9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7dc06e570 0x7ff7dc1f3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:28.233 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.228+0000 7ff7e40a9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7dc0c79a0 0x7ff7dc159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:28.234 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.228+0000 7ff7e40a9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff7dc151900 0x7ff7dc1efb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:28.234 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.228+0000 7ff7e40a9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7dc1553b0 con 0x7ff7dc06e570 2023-10-30T12:28:28.234 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.228+0000 7ff7e40a9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7dc1f0040 con 0x7ff7dc151900 2023-10-30T12:28:28.234 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.228+0000 7ff7e40a9640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7dc1f2ce0 con 0x7ff7dc0c79a0 2023-10-30T12:28:28.234 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.228+0000 7ff7e1644640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7dc0c79a0 0x7ff7dc159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:28.235 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.228+0000 7ff7e2646640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff7dc151900 0x7ff7dc1efb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:28.235 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.228+0000 7ff7e1644640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7dc0c79a0 0x7ff7dc159b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.192:60662/0 (socket says 172.21.15.192:60662) 2023-10-30T12:28:28.235 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.228+0000 7ff7e1644640 1 -- 172.21.15.192:0/3460464036 learned_addr learned my addr 172.21.15.192:0/3460464036 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-10-30T12:28:28.235 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.228+0000 7ff7e1e45640 1 --2- 172.21.15.192:0/3460464036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7dc06e570 0x7ff7dc1f3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:28.235 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.228+0000 7ff7e1644640 1 -- 172.21.15.192:0/3460464036 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff7dc151900 msgr2=0x7ff7dc1efb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:28.236 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.228+0000 7ff7e1644640 1 --2- 172.21.15.192:0/3460464036 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff7dc151900 0x7ff7dc1efb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.236 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.228+0000 7ff7e1644640 1 -- 172.21.15.192:0/3460464036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7dc06e570 msgr2=0x7ff7dc1f3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:28.236 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.228+0000 7ff7e1644640 1 --2- 172.21.15.192:0/3460464036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7dc06e570 0x7ff7dc1f3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.236 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.228+0000 7ff7e1644640 1 -- 172.21.15.192:0/3460464036 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7cc0094d0 con 0x7ff7dc0c79a0 2023-10-30T12:28:28.236 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.228+0000 7ff7e1644640 1 --2- 172.21.15.192:0/3460464036 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7dc0c79a0 0x7ff7dc159b20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc0af040 tx=0x7ff7cc0192b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:28.237 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.228+0000 7ff7d27fc640 1 -- 172.21.15.192:0/3460464036 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc0b5640 con 0x7ff7dc0c79a0 2023-10-30T12:28:28.237 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.228+0000 7ff7d27fc640 1 -- 172.21.15.192:0/3460464036 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7cc0b57c0 con 0x7ff7dc0c79a0 2023-10-30T12:28:28.237 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.228+0000 7ff7d27fc640 1 -- 172.21.15.192:0/3460464036 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc0b5b70 con 0x7ff7dc0c79a0 2023-10-30T12:28:28.237 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.228+0000 7ff7e40a9640 1 -- 172.21.15.192:0/3460464036 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7dc0c6170 con 0x7ff7dc0c79a0 2023-10-30T12:28:28.237 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.228+0000 7ff7e40a9640 1 -- 172.21.15.192:0/3460464036 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7dc0c5e90 con 0x7ff7dc0c79a0 2023-10-30T12:28:28.238 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.228+0000 7ff7d27fc640 1 -- 172.21.15.192:0/3460464036 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff7cc0b3070 con 0x7ff7dc0c79a0 2023-10-30T12:28:28.238 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.228+0000 7ff7d27fc640 1 -- 172.21.15.192:0/3460464036 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7ff7cc0ab3e0 con 0x7ff7dc0c79a0 2023-10-30T12:28:28.238 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.232+0000 7ff7e40a9640 1 -- 172.21.15.192:0/3460464036 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7dc04e170 con 0x7ff7dc0c79a0 2023-10-30T12:28:28.239 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-10-30T12:28:28.229+0000 7f7edc42e180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-30T12:28:28.239 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.232+0000 7ff7d27fc640 1 -- 172.21.15.192:0/3460464036 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7ff7cc0b1070 con 0x7ff7dc0c79a0 2023-10-30T12:28:28.292 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-10-30T12:28:28.289+0000 7f7edc42e180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-30T12:28:28.298 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.296+0000 7ff7e40a9640 1 -- 172.21.15.192:0/3460464036 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd new", "uuid": "48623452-d456-4ea2-a0de-5fa4b37e6d10", "id": 6} v 0) v1 -- 0x7ff7dc071020 con 0x7ff7dc0c79a0 2023-10-30T12:28:28.304 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.300+0000 7ff7d27fc640 1 -- 172.21.15.192:0/3460464036 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "48623452-d456-4ea2-a0de-5fa4b37e6d10", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff7cc0c0460 con 0x7ff7dc0c79a0 2023-10-30T12:28:28.304 INFO:teuthology.orchestra.run.smithi192.stdout:6 2023-10-30T12:28:28.305 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.304+0000 7ff7e40a9640 1 -- 172.21.15.192:0/3460464036 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7dc0c79a0 msgr2=0x7ff7dc159b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:28.306 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.304+0000 7ff7e40a9640 1 --2- 172.21.15.192:0/3460464036 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7dc0c79a0 0x7ff7dc159b20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc0af040 tx=0x7ff7cc0192b0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.306 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.304+0000 7ff7e40a9640 1 -- 172.21.15.192:0/3460464036 shutdown_connections 2023-10-30T12:28:28.306 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.304+0000 7ff7e40a9640 1 --2- 172.21.15.192:0/3460464036 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff7dc151900 0x7ff7dc1efb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.306 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.304+0000 7ff7e40a9640 1 --2- 172.21.15.192:0/3460464036 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7dc0c79a0 0x7ff7dc159b20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.307 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.304+0000 7ff7e40a9640 1 --2- 172.21.15.192:0/3460464036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7dc06e570 0x7ff7dc1f3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.307 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.304+0000 7ff7e40a9640 1 -- 172.21.15.192:0/3460464036 >> 172.21.15.192:0/3460464036 conn(0x7ff7dc0ccfe0 msgr2=0x7ff7dc0c8390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:28.307 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.304+0000 7ff7e40a9640 1 -- 172.21.15.192:0/3460464036 shutdown_connections 2023-10-30T12:28:28.307 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.304+0000 7ff7e40a9640 1 -- 172.21.15.192:0/3460464036 wait complete. 2023-10-30T12:28:28.319 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new ff0a4384-54f8-43ec-a3ce-9c8ae3bf8a94 7 2023-10-30T12:28:28.333 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-30T12:28:28.332+0000 7fb4e738e180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-30T12:28:28.395 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-30T12:28:28.392+0000 7fb4e738e180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-30T12:28:28.442 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.440+0000 7f53884c0640 1 Processor -- start 2023-10-30T12:28:28.442 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.440+0000 7f53884c0640 1 -- start start 2023-10-30T12:28:28.444 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.440+0000 7f53884c0640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f538006e570 0x7f53800c22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:28.444 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.440+0000 7f53884c0640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f53800c79a0 0x7f538006f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:28.444 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.440+0000 7f53884c0640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f538006e360 con 0x7f5380151900 2023-10-30T12:28:28.444 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.440+0000 7f53884c0640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f538006eab0 con 0x7f53800c79a0 2023-10-30T12:28:28.445 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.440+0000 7f53884c0640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53800c7480 con 0x7f538006e570 2023-10-30T12:28:28.445 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.440+0000 7f5385a5b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f538006e570 0x7f53800c22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:28.445 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.440+0000 7f5385a5b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f538006e570 0x7f53800c22f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.192:60670/0 (socket says 172.21.15.192:60670) 2023-10-30T12:28:28.445 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.440+0000 7f5385a5b640 1 -- 172.21.15.192:0/1510484034 learned_addr learned my addr 172.21.15.192:0/1510484034 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-10-30T12:28:28.445 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.440+0000 7f5386a5d640 1 --2- 172.21.15.192:0/1510484034 >> v2:172.21.15.89:3300/0 conn(0x7f53800c79a0 0x7f538006f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:28.446 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.440+0000 7f5385a5b640 1 -- 172.21.15.192:0/1510484034 >> v2:172.21.15.89:3300/0 conn(0x7f53800c79a0 msgr2=0x7f538006f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:28.446 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.440+0000 7f5385a5b640 1 --2- 172.21.15.192:0/1510484034 >> v2:172.21.15.89:3300/0 conn(0x7f53800c79a0 0x7f538006f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.446 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.440+0000 7f5385a5b640 1 -- 172.21.15.192:0/1510484034 >> v1:172.21.15.89:6789/0 conn(0x7f5380151900 legacy=0x7f538006d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:28.446 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.440+0000 7f5385a5b640 1 -- 172.21.15.192:0/1510484034 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5380158320 con 0x7f538006e570 2023-10-30T12:28:28.448 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.444+0000 7f5385a5b640 1 --2- 172.21.15.192:0/1510484034 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f538006e570 0x7f53800c22f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f537c009870 tx=0x7f537c01c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2caf57ad248d9024 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:28.448 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.444+0000 7f538525a640 1 -- 172.21.15.192:0/1510484034 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f537c0aa5f0 con 0x7f538006e570 2023-10-30T12:28:28.448 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.444+0000 7f538525a640 1 -- 172.21.15.192:0/1510484034 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f537c0aa750 con 0x7f538006e570 2023-10-30T12:28:28.448 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.444+0000 7f538525a640 1 -- 172.21.15.192:0/1510484034 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f537c0aaae0 con 0x7f538006e570 2023-10-30T12:28:28.449 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.444+0000 7f53884c0640 1 -- 172.21.15.192:0/1510484034 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f538006e570 msgr2=0x7f53800c22f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:28.449 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.444+0000 7f53884c0640 1 --2- 172.21.15.192:0/1510484034 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f538006e570 0x7f53800c22f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f537c009870 tx=0x7f537c01c4d0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.449 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.444+0000 7f53884c0640 1 -- 172.21.15.192:0/1510484034 shutdown_connections 2023-10-30T12:28:28.449 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.444+0000 7f53884c0640 1 --2- 172.21.15.192:0/1510484034 >> v2:172.21.15.89:3300/0 conn(0x7f53800c79a0 0x7f538006f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.449 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.444+0000 7f53884c0640 1 --2- 172.21.15.192:0/1510484034 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f538006e570 0x7f53800c22f0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.450 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.444+0000 7f53884c0640 1 -- 172.21.15.192:0/1510484034 >> 172.21.15.192:0/1510484034 conn(0x7f53800ccfe0 msgr2=0x7f53800cd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:28.450 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.444+0000 7f53884c0640 1 -- 172.21.15.192:0/1510484034 shutdown_connections 2023-10-30T12:28:28.450 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.444+0000 7f53884c0640 1 -- 172.21.15.192:0/1510484034 wait complete. 2023-10-30T12:28:28.450 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.444+0000 7f53884c0640 1 Processor -- start 2023-10-30T12:28:28.450 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.444+0000 7f53884c0640 1 -- start start 2023-10-30T12:28:28.451 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-10-30T12:28:28.445+0000 7f7edc42e180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-30T12:28:28.453 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.448+0000 7f53884c0640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f538006e570 0x7f53800c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:28.453 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.448+0000 7f53884c0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53800c79a0 0x7f538006c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:28.453 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.448+0000 7f53884c0640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5380151900 0x7f53801f5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:28.454 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.448+0000 7f53884c0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53800c5770 con 0x7f53800c79a0 2023-10-30T12:28:28.454 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.448+0000 7f53884c0640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f538006afa0 con 0x7f538006e570 2023-10-30T12:28:28.454 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.448+0000 7f53884c0640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53801f4ea0 con 0x7f5380151900 2023-10-30T12:28:28.454 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.448+0000 7f538625c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f538006e570 0x7f53800c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:28.454 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.448+0000 7f5386a5d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5380151900 0x7f53801f5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:28.455 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.448+0000 7f538625c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f538006e570 0x7f53800c8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:42202/0 (socket says 172.21.15.192:42202) 2023-10-30T12:28:28.455 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.448+0000 7f538625c640 1 -- 172.21.15.192:0/3482567181 learned_addr learned my addr 172.21.15.192:0/3482567181 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-10-30T12:28:28.455 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.448+0000 7f5385a5b640 1 --2- 172.21.15.192:0/3482567181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53800c79a0 0x7f538006c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:28.455 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.448+0000 7f538625c640 1 -- 172.21.15.192:0/3482567181 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5380151900 msgr2=0x7f53801f5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:28.455 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.448+0000 7f538625c640 1 --2- 172.21.15.192:0/3482567181 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5380151900 0x7f53801f5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.456 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.448+0000 7f538625c640 1 -- 172.21.15.192:0/3482567181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53800c79a0 msgr2=0x7f538006c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:28.456 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.448+0000 7f538625c640 1 --2- 172.21.15.192:0/3482567181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53800c79a0 0x7f538006c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.456 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.448+0000 7f538625c640 1 -- 172.21.15.192:0/3482567181 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f537c0094d0 con 0x7f538006e570 2023-10-30T12:28:28.456 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.448+0000 7f5386a5d640 1 --2- 172.21.15.192:0/3482567181 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5380151900 0x7f53801f5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:28.456 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.448+0000 7f538625c640 1 --2- 172.21.15.192:0/3482567181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f538006e570 0x7f53800c8ca0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f537400b470 tx=0x7f537400b930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:28.457 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.448+0000 7f5372ffd640 1 -- 172.21.15.192:0/3482567181 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5374004020 con 0x7f538006e570 2023-10-30T12:28:28.457 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.448+0000 7f5372ffd640 1 -- 172.21.15.192:0/3482567181 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5374004180 con 0x7f538006e570 2023-10-30T12:28:28.457 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.448+0000 7f5372ffd640 1 -- 172.21.15.192:0/3482567181 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5374011690 con 0x7f538006e570 2023-10-30T12:28:28.457 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.448+0000 7f53884c0640 1 -- 172.21.15.192:0/3482567181 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5380151e60 con 0x7f538006e570 2023-10-30T12:28:28.457 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.448+0000 7f53884c0640 1 -- 172.21.15.192:0/3482567181 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53801f5180 con 0x7f538006e570 2023-10-30T12:28:28.458 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.448+0000 7f5372ffd640 1 -- 172.21.15.192:0/3482567181 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f5374011af0 con 0x7f538006e570 2023-10-30T12:28:28.458 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.448+0000 7f5372ffd640 1 -- 172.21.15.192:0/3482567181 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f5374003d80 con 0x7f538006e570 2023-10-30T12:28:28.458 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.448+0000 7f53884c0640 1 -- 172.21.15.192:0/3482567181 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f538004e170 con 0x7f538006e570 2023-10-30T12:28:28.458 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.452+0000 7f5372ffd640 1 -- 172.21.15.192:0/3482567181 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f5374015030 con 0x7f538006e570 2023-10-30T12:28:28.515 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-10-30T12:28:28.513+0000 7f7edc42e180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-30T12:28:28.515 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.512+0000 7f53884c0640 1 -- 172.21.15.192:0/3482567181 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ff0a4384-54f8-43ec-a3ce-9c8ae3bf8a94", "id": 7} v 0) v1 -- 0x7f538006b4d0 con 0x7f538006e570 2023-10-30T12:28:28.522 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.520+0000 7f5372ffd640 1 -- 172.21.15.192:0/3482567181 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ff0a4384-54f8-43ec-a3ce-9c8ae3bf8a94", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5374003600 con 0x7f538006e570 2023-10-30T12:28:28.522 INFO:teuthology.orchestra.run.smithi192.stdout:7 2023-10-30T12:28:28.523 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.520+0000 7f53884c0640 1 -- 172.21.15.192:0/3482567181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f538006e570 msgr2=0x7f53800c8ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:28.523 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.520+0000 7f53884c0640 1 --2- 172.21.15.192:0/3482567181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f538006e570 0x7f53800c8ca0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f537400b470 tx=0x7f537400b930 comp rx=0 tx=0).stop 2023-10-30T12:28:28.524 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.520+0000 7f53884c0640 1 -- 172.21.15.192:0/3482567181 shutdown_connections 2023-10-30T12:28:28.524 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.520+0000 7f53884c0640 1 --2- 172.21.15.192:0/3482567181 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5380151900 0x7f53801f5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.524 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.520+0000 7f53884c0640 1 --2- 172.21.15.192:0/3482567181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53800c79a0 0x7f538006c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.524 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.520+0000 7f53884c0640 1 --2- 172.21.15.192:0/3482567181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f538006e570 0x7f53800c8ca0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.525 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.520+0000 7f53884c0640 1 -- 172.21.15.192:0/3482567181 >> 172.21.15.192:0/3482567181 conn(0x7f53800ccfe0 msgr2=0x7f53800cbb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:28.525 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.520+0000 7f53884c0640 1 -- 172.21.15.192:0/3482567181 shutdown_connections 2023-10-30T12:28:28.525 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-30T12:28:28.520+0000 7f53884c0640 1 -- 172.21.15.192:0/3482567181 wait complete. 2023-10-30T12:28:28.537 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-30T12:28:28.538 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-30T12:28:28.541 INFO:tasks.ceph.osd.0:Started 2023-10-30T12:28:28.542 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-30T12:28:28.542 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-30T12:28:28.545 INFO:tasks.ceph.osd.1:Started 2023-10-30T12:28:28.546 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-30T12:28:28.546 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-30T12:28:28.550 INFO:tasks.ceph.osd.2:Started 2023-10-30T12:28:28.550 INFO:tasks.ceph.osd.3:Restarting daemon 2023-10-30T12:28:28.550 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-10-30T12:28:28.555 INFO:tasks.ceph.osd.3:Started 2023-10-30T12:28:28.556 INFO:tasks.ceph.osd.4:Restarting daemon 2023-10-30T12:28:28.556 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-10-30T12:28:28.558 INFO:tasks.ceph.osd.4:Started 2023-10-30T12:28:28.558 INFO:tasks.ceph.osd.5:Restarting daemon 2023-10-30T12:28:28.559 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-10-30T12:28:28.561 INFO:tasks.ceph.osd.5:Started 2023-10-30T12:28:28.563 INFO:tasks.ceph.osd.6:Restarting daemon 2023-10-30T12:28:28.563 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-10-30T12:28:28.565 INFO:tasks.ceph.osd.6:Started 2023-10-30T12:28:28.565 INFO:tasks.ceph.osd.7:Restarting daemon 2023-10-30T12:28:28.566 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-10-30T12:28:28.567 INFO:tasks.ceph.osd.7:Started 2023-10-30T12:28:28.568 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:28:28.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.745+0000 7fa48ac17640 1 Processor -- start 2023-10-30T12:28:28.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.745+0000 7fa48ac17640 1 -- start start 2023-10-30T12:28:28.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.745+0000 7fa48ac17640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4840c6f60 0x7fa4841566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:28.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.745+0000 7fa48ac17640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4840c0a50 con 0x7fa4840c6b80 2023-10-30T12:28:28.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.745+0000 7fa48ac17640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa48406f6e0 con 0x7fa4840c05d0 2023-10-30T12:28:28.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.745+0000 7fa48ac17640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa48406f9c0 con 0x7fa4840c6f60 2023-10-30T12:28:28.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.745+0000 7fa4889b3640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa4840c6b80 0x7fa4840bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58354/0 (socket says 172.21.15.89:58354) 2023-10-30T12:28:28.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.745+0000 7fa4889b3640 1 -- 172.21.15.89:0/2509785414 learned_addr learned my addr 172.21.15.89:0/2509785414 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:28:28.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.745+0000 7fa4891b4640 1 --2- 172.21.15.89:0/2509785414 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4840c6f60 0x7fa4841566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:28.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.745+0000 7fa4837fe640 1 -- 172.21.15.89:0/2509785414 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2632226916 0 0) 0x7fa4840c0a50 con 0x7fa4840c6b80 2023-10-30T12:28:28.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.745+0000 7fa4837fe640 1 -- 172.21.15.89:0/2509785414 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa46c003530 con 0x7fa4840c6b80 2023-10-30T12:28:28.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.745+0000 7fa4837fe640 1 -- 172.21.15.89:0/2509785414 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 809360692 0 0) 0x7fa48406f6e0 con 0x7fa4840c05d0 2023-10-30T12:28:28.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.745+0000 7fa4837fe640 1 -- 172.21.15.89:0/2509785414 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4840c0a50 con 0x7fa4840c05d0 2023-10-30T12:28:28.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.745+0000 7fa4891b4640 1 -- 172.21.15.89:0/2509785414 >> v1:172.21.15.192:6789/0 conn(0x7fa4840c05d0 legacy=0x7fa48413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:28.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.745+0000 7fa4891b4640 1 -- 172.21.15.89:0/2509785414 >> v1:172.21.15.89:6789/0 conn(0x7fa4840c6b80 legacy=0x7fa4840bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:28.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.745+0000 7fa4891b4640 1 -- 172.21.15.89:0/2509785414 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa48406fca0 con 0x7fa4840c6f60 2023-10-30T12:28:28.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.749+0000 7fa4891b4640 1 --2- 172.21.15.89:0/2509785414 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4840c6f60 0x7fa4841566a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa47c009dc0 tx=0x7fa47c01c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5b83c2e2e0fa679 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:28.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.749+0000 7fa4837fe640 1 -- 172.21.15.89:0/2509785414 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47c0a5840 con 0x7fa4840c6f60 2023-10-30T12:28:28.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.749+0000 7fa4837fe640 1 -- 172.21.15.89:0/2509785414 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa47c0a59a0 con 0x7fa4840c6f60 2023-10-30T12:28:28.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.749+0000 7fa4837fe640 1 -- 172.21.15.89:0/2509785414 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47c0a5d30 con 0x7fa4840c6f60 2023-10-30T12:28:28.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.753+0000 7fa48ac17640 1 -- 172.21.15.89:0/2509785414 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4840c6f60 msgr2=0x7fa4841566a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:28.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.753+0000 7fa48ac17640 1 --2- 172.21.15.89:0/2509785414 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4840c6f60 0x7fa4841566a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa47c009dc0 tx=0x7fa47c01c970 comp rx=0 tx=0).stop 2023-10-30T12:28:28.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.753+0000 7fa48ac17640 1 -- 172.21.15.89:0/2509785414 shutdown_connections 2023-10-30T12:28:28.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.753+0000 7fa48ac17640 1 --2- 172.21.15.89:0/2509785414 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4840c6f60 0x7fa4841566a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.753+0000 7fa48ac17640 1 -- 172.21.15.89:0/2509785414 >> 172.21.15.89:0/2509785414 conn(0x7fa4840c7910 msgr2=0x7fa4840c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:28.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.753+0000 7fa48ac17640 1 -- 172.21.15.89:0/2509785414 shutdown_connections 2023-10-30T12:28:28.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.753+0000 7fa48ac17640 1 -- 172.21.15.89:0/2509785414 wait complete. 2023-10-30T12:28:28.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.753+0000 7fa48ac17640 1 Processor -- start 2023-10-30T12:28:28.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.753+0000 7fa48ac17640 1 -- start start 2023-10-30T12:28:28.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.753+0000 7fa48ac17640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4840c05d0 0x7fa48413db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:28.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.753+0000 7fa48ac17640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4840c6b80 0x7fa4840c9240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:28.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.753+0000 7fa48ac17640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4840c6f60 0x7fa4840c9780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:28.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.753+0000 7fa48ac17640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa48413e840 con 0x7fa4840c6f60 2023-10-30T12:28:28.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.753+0000 7fa48ac17640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4841f2fe0 con 0x7fa4840c6b80 2023-10-30T12:28:28.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.753+0000 7fa48ac17640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4841f3280 con 0x7fa4840c05d0 2023-10-30T12:28:28.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.753+0000 7fa483fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4840c6b80 0x7fa4840c9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:28.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.753+0000 7fa4889b3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4840c05d0 0x7fa48413db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:28.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.753+0000 7fa4891b4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4840c6f60 0x7fa4840c9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:28.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.753+0000 7fa4889b3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4840c05d0 0x7fa48413db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:51632/0 (socket says 172.21.15.89:51632) 2023-10-30T12:28:28.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.753+0000 7fa483fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4840c6b80 0x7fa4840c9240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:43558/0 (socket says 172.21.15.89:43558) 2023-10-30T12:28:28.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.753+0000 7fa4889b3640 1 -- 172.21.15.89:0/111207493 learned_addr learned my addr 172.21.15.89:0/111207493 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:28.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.753+0000 7fa4891b4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4840c6f60 0x7fa4840c9780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49522/0 (socket says 172.21.15.89:49522) 2023-10-30T12:28:28.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.753+0000 7fa4891b4640 1 -- 172.21.15.89:0/111207493 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4840c05d0 msgr2=0x7fa48413db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:28.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.753+0000 7fa4891b4640 1 --2- 172.21.15.89:0/111207493 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4840c05d0 0x7fa48413db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.753+0000 7fa4891b4640 1 -- 172.21.15.89:0/111207493 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4840c6b80 msgr2=0x7fa4840c9240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:28.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.753+0000 7fa4891b4640 1 --2- 172.21.15.89:0/111207493 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4840c6b80 0x7fa4840c9240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.753+0000 7fa4891b4640 1 -- 172.21.15.89:0/111207493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa47c009a70 con 0x7fa4840c6f60 2023-10-30T12:28:28.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.757+0000 7fa4891b4640 1 --2- 172.21.15.89:0/111207493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4840c6f60 0x7fa4840c9780 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa47c0ac000 tx=0x7fa47c019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:28.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.757+0000 7fa4817fa640 1 -- 172.21.15.89:0/111207493 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47c0be070 con 0x7fa4840c6f60 2023-10-30T12:28:28.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.757+0000 7fa4817fa640 1 -- 172.21.15.89:0/111207493 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa47c0b9450 con 0x7fa4840c6f60 2023-10-30T12:28:28.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.757+0000 7fa4817fa640 1 -- 172.21.15.89:0/111207493 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47c0b97e0 con 0x7fa4840c6f60 2023-10-30T12:28:28.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.757+0000 7fa48ac17640 1 -- 172.21.15.89:0/111207493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa484155910 con 0x7fa4840c6f60 2023-10-30T12:28:28.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.757+0000 7fa48ac17640 1 -- 172.21.15.89:0/111207493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa484155ed0 con 0x7fa4840c6f60 2023-10-30T12:28:28.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.757+0000 7fa4817fa640 1 -- 172.21.15.89:0/111207493 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa47c0b9c40 con 0x7fa4840c6f60 2023-10-30T12:28:28.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.757+0000 7fa4817fa640 1 -- 172.21.15.89:0/111207493 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fa47c0adbc0 con 0x7fa4840c6f60 2023-10-30T12:28:28.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.757+0000 7fa48ac17640 1 -- 172.21.15.89:0/111207493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4841560c0 con 0x7fa4840c6f60 2023-10-30T12:28:28.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.761+0000 7fa4817fa640 1 -- 172.21.15.89:0/111207493 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fa47c0b2070 con 0x7fa4840c6f60 2023-10-30T12:28:28.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.813+0000 7fa48ac17640 1 -- 172.21.15.89:0/111207493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa48404e170 con 0x7fa4840c6f60 2023-10-30T12:28:28.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.813+0000 7fa4817fa640 1 -- 172.21.15.89:0/111207493 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fa47c0add80 con 0x7fa4840c6f60 2023-10-30T12:28:28.819 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:28:28.820 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":10,"fsid":"401641be-db6e-4c16-b3ac-fd84a86c4de4","created":"2023-10-30T12:28:26.198532+0000","modified":"2023-10-30T12:28:28.517734+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T12:28:28.517734+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"72595c54-fd6e-4a5a-bb14-ff27e9377b07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0305670b-d7c4-4193-bec2-8b7351ff590a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4bab5426-a3c3-46b0-91fc-546c254006c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"7d47aea8-27cd-42be-8591-d63ad68c90e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2e2ae8f8-d097-4bba-9448-95047fbfacad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"49be5ce5-12f4-4c26-8d39-864b6ff67313","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"48623452-d456-4ea2-a0de-5fa4b37e6d10","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ff0a4384-54f8-43ec-a3ce-9c8ae3bf8a94","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:28:28.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.817+0000 7fa48ac17640 1 -- 172.21.15.89:0/111207493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4840c6f60 msgr2=0x7fa4840c9780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:28.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.817+0000 7fa48ac17640 1 --2- 172.21.15.89:0/111207493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4840c6f60 0x7fa4840c9780 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa47c0ac000 tx=0x7fa47c019710 comp rx=0 tx=0).stop 2023-10-30T12:28:28.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.817+0000 7fa48ac17640 1 -- 172.21.15.89:0/111207493 shutdown_connections 2023-10-30T12:28:28.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.817+0000 7fa48ac17640 1 --2- 172.21.15.89:0/111207493 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4840c6b80 0x7fa4840c9240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.817+0000 7fa48ac17640 1 --2- 172.21.15.89:0/111207493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4840c6f60 0x7fa4840c9780 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.817+0000 7fa48ac17640 1 --2- 172.21.15.89:0/111207493 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4840c05d0 0x7fa48413db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:28.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.817+0000 7fa48ac17640 1 -- 172.21.15.89:0/111207493 >> 172.21.15.89:0/111207493 conn(0x7fa4840c7910 msgr2=0x7fa484138950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:28.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.817+0000 7fa48ac17640 1 -- 172.21.15.89:0/111207493 shutdown_connections 2023-10-30T12:28:28.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:28.817+0000 7fa48ac17640 1 -- 172.21.15.89:0/111207493 wait complete. 2023-10-30T12:28:28.838 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-30T12:28:28.839 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-30T12:28:29.057 INFO:tasks.ceph.mgr.x.smithi192.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-10-30T12:28:29.058 INFO:tasks.ceph.mgr.x.smithi192.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-10-30T12:28:29.058 INFO:tasks.ceph.mgr.x.smithi192.stderr: from numpy import show_config as show_numpy_config 2023-10-30T12:28:29.093 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-30T12:28:29.092+0000 7fb4e738e180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-30T12:28:29.190 INFO:tasks.ceph.osd.4.smithi192.stderr:2023-10-30T12:28:29.188+0000 7f08af7f9640 -1 Falling back to public interface 2023-10-30T12:28:29.190 INFO:tasks.ceph.osd.5.smithi192.stderr:2023-10-30T12:28:29.188+0000 7f923e239640 -1 Falling back to public interface 2023-10-30T12:28:29.193 INFO:tasks.ceph.osd.7.smithi192.stderr:2023-10-30T12:28:29.192+0000 7f6f602bd640 -1 Falling back to public interface 2023-10-30T12:28:29.205 INFO:tasks.ceph.osd.6.smithi192.stderr:2023-10-30T12:28:29.204+0000 7f99049ce640 -1 Falling back to public interface 2023-10-30T12:28:29.216 INFO:tasks.ceph.osd.0.smithi089.stderr:2023-10-30T12:28:29.213+0000 7fe354bfd640 -1 Falling back to public interface 2023-10-30T12:28:29.227 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-30T12:28:29.224+0000 7fb4e738e180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-30T12:28:29.228 INFO:tasks.ceph.osd.2.smithi089.stderr:2023-10-30T12:28:29.225+0000 7f9694a2b640 -1 Falling back to public interface 2023-10-30T12:28:29.230 INFO:tasks.ceph.mgr.y.smithi089.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-10-30T12:28:29.230 INFO:tasks.ceph.mgr.y.smithi089.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-10-30T12:28:29.230 INFO:tasks.ceph.mgr.y.smithi089.stderr: from numpy import show_config as show_numpy_config 2023-10-30T12:28:29.248 INFO:tasks.ceph.osd.1.smithi089.stderr:2023-10-30T12:28:29.245+0000 7fbe7ce2d640 -1 Falling back to public interface 2023-10-30T12:28:29.283 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-10-30T12:28:29.281+0000 7f7edc42e180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-30T12:28:29.288 INFO:tasks.ceph.osd.3.smithi089.stderr:2023-10-30T12:28:29.285+0000 7f0618396640 -1 Falling back to public interface 2023-10-30T12:28:29.418 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-10-30T12:28:29.413+0000 7f7edc42e180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-30T12:28:29.429 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-30T12:28:29.428+0000 7fb4e738e180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-30T12:28:29.499 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-30T12:28:29.496+0000 7fb4e738e180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-30T12:28:29.579 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-10-30T12:28:29.573+0000 7f7edc42e180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-30T12:28:29.622 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-30T12:28:29.620+0000 7fb4e738e180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-30T12:28:29.645 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T12:28:29.664 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-10-30T12:28:29.661+0000 7f7edc42e180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-30T12:28:29.772 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.769+0000 7fa0ab780640 1 Processor -- start 2023-10-30T12:28:29.772 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.769+0000 7fa0ab780640 1 -- start start 2023-10-30T12:28:29.773 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.769+0000 7fa0ab780640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa0a40c7d40 0x7fa0a40c4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:29.774 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.769+0000 7fa0ab780640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0a415ceb0 0x7fa0a415d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:29.774 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.769+0000 7fa0ab780640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0a40c69c0 con 0x7fa0a4153b50 2023-10-30T12:28:29.774 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.769+0000 7fa0ab780640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0a40c81b0 con 0x7fa0a40c7d40 2023-10-30T12:28:29.774 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.769+0000 7fa0ab780640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a40c6ca0 con 0x7fa0a415ceb0 2023-10-30T12:28:29.775 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.769+0000 7fa0a8d1b640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa0a40c7d40 0x7fa0a40c4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:29.775 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.769+0000 7fa0a9d1d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0a415ceb0 0x7fa0a415d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:29.775 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.769+0000 7fa0a951c640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fa0a4153b50 0x7fa0a4071e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:41528/0 (socket says 172.21.15.89:41528) 2023-10-30T12:28:29.776 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.769+0000 7fa0a951c640 1 -- 172.21.15.89:0/2989632696 learned_addr learned my addr 172.21.15.89:0/2989632696 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:28:29.776 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.769+0000 7fa0a8d1b640 1 -- 172.21.15.89:0/2989632696 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0a415ceb0 msgr2=0x7fa0a415d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:29.776 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.769+0000 7fa0a8d1b640 1 --2- 172.21.15.89:0/2989632696 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0a415ceb0 0x7fa0a415d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:29.776 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.769+0000 7fa0a8d1b640 1 -- 172.21.15.89:0/2989632696 >> v1:172.21.15.192:6789/0 conn(0x7fa0a4153b50 legacy=0x7fa0a4071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:29.777 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.769+0000 7fa0a8d1b640 1 -- 172.21.15.89:0/2989632696 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a40c6f80 con 0x7fa0a40c7d40 2023-10-30T12:28:29.777 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.773+0000 7fa0a8d1b640 1 --2- 172.21.15.89:0/2989632696 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0a40c7d40 0x7fa0a40c4820 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa0940097c0 tx=0x7fa09401ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4e7a42233f58e2b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:29.777 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.773+0000 7fa09bfff640 1 -- 172.21.15.89:0/2989632696 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0940a8970 con 0x7fa0a40c7d40 2023-10-30T12:28:29.778 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.773+0000 7fa09bfff640 1 -- 172.21.15.89:0/2989632696 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0940a8ad0 con 0x7fa0a40c7d40 2023-10-30T12:28:29.778 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.773+0000 7fa09bfff640 1 -- 172.21.15.89:0/2989632696 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0940a8e60 con 0x7fa0a40c7d40 2023-10-30T12:28:29.779 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.773+0000 7fa0ab780640 1 -- 172.21.15.89:0/2989632696 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0a40c7d40 msgr2=0x7fa0a40c4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:29.779 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.773+0000 7fa0ab780640 1 --2- 172.21.15.89:0/2989632696 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0a40c7d40 0x7fa0a40c4820 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa0940097c0 tx=0x7fa09401ec70 comp rx=0 tx=0).stop 2023-10-30T12:28:29.779 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.773+0000 7fa0ab780640 1 -- 172.21.15.89:0/2989632696 shutdown_connections 2023-10-30T12:28:29.779 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.773+0000 7fa0ab780640 1 --2- 172.21.15.89:0/2989632696 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0a415ceb0 0x7fa0a415d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:29.780 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.773+0000 7fa0ab780640 1 --2- 172.21.15.89:0/2989632696 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0a40c7d40 0x7fa0a40c4820 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:29.780 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.773+0000 7fa0ab780640 1 -- 172.21.15.89:0/2989632696 >> 172.21.15.89:0/2989632696 conn(0x7fa0a40c8920 msgr2=0x7fa0a40c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:29.780 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.773+0000 7fa0ab780640 1 -- 172.21.15.89:0/2989632696 shutdown_connections 2023-10-30T12:28:29.780 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.773+0000 7fa0ab780640 1 -- 172.21.15.89:0/2989632696 wait complete. 2023-10-30T12:28:29.782 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.777+0000 7fa0ab780640 1 Processor -- start 2023-10-30T12:28:29.783 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.777+0000 7fa0ab780640 1 -- start start 2023-10-30T12:28:29.783 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.777+0000 7fa0ab780640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0a40c7d40 0x7fa0a41497f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:29.783 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.777+0000 7fa0ab780640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0a4153b50 0x7fa0a414ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:29.783 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.777+0000 7fa0ab780640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0a415ceb0 0x7fa0a41f2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:29.783 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.777+0000 7fa0ab780640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a4148ad0 con 0x7fa0a415ceb0 2023-10-30T12:28:29.784 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.777+0000 7fa0ab780640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a414add0 con 0x7fa0a40c7d40 2023-10-30T12:28:29.784 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.777+0000 7fa0ab780640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a414b0b0 con 0x7fa0a4153b50 2023-10-30T12:28:29.784 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.777+0000 7fa0a9d1d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0a415ceb0 0x7fa0a41f2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:29.784 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.777+0000 7fa0a9d1d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0a415ceb0 0x7fa0a41f2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58182/0 (socket says 172.21.15.89:58182) 2023-10-30T12:28:29.784 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.777+0000 7fa0a9d1d640 1 -- 172.21.15.89:0/315626084 learned_addr learned my addr 172.21.15.89:0/315626084 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:29.785 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.777+0000 7fa0a951c640 1 --2- 172.21.15.89:0/315626084 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0a40c7d40 0x7fa0a41497f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:29.785 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.777+0000 7fa0a8d1b640 1 --2- 172.21.15.89:0/315626084 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0a4153b50 0x7fa0a414ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:29.785 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.777+0000 7fa0a9d1d640 1 -- 172.21.15.89:0/315626084 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0a4153b50 msgr2=0x7fa0a414ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:29.785 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.777+0000 7fa0a9d1d640 1 --2- 172.21.15.89:0/315626084 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0a4153b50 0x7fa0a414ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:29.785 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.777+0000 7fa0a9d1d640 1 -- 172.21.15.89:0/315626084 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0a40c7d40 msgr2=0x7fa0a41497f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:29.785 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.777+0000 7fa0a9d1d640 1 --2- 172.21.15.89:0/315626084 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0a40c7d40 0x7fa0a41497f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:29.786 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.777+0000 7fa0a9d1d640 1 -- 172.21.15.89:0/315626084 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa094009420 con 0x7fa0a415ceb0 2023-10-30T12:28:29.786 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.777+0000 7fa0a8d1b640 1 --2- 172.21.15.89:0/315626084 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0a4153b50 0x7fa0a414ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:29.786 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.777+0000 7fa0a9d1d640 1 --2- 172.21.15.89:0/315626084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0a415ceb0 0x7fa0a41f2180 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa09c00efc0 tx=0x7fa09c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:29.786 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.777+0000 7fa099ffb640 1 -- 172.21.15.89:0/315626084 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa09c014480 con 0x7fa0a415ceb0 2023-10-30T12:28:29.787 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.777+0000 7fa099ffb640 1 -- 172.21.15.89:0/315626084 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa09c00f040 con 0x7fa0a415ceb0 2023-10-30T12:28:29.787 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.777+0000 7fa099ffb640 1 -- 172.21.15.89:0/315626084 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa09c014820 con 0x7fa0a415ceb0 2023-10-30T12:28:29.787 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.777+0000 7fa0ab780640 1 -- 172.21.15.89:0/315626084 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0a41481a0 con 0x7fa0a415ceb0 2023-10-30T12:28:29.787 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.777+0000 7fa0ab780640 1 -- 172.21.15.89:0/315626084 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0a4148760 con 0x7fa0a415ceb0 2023-10-30T12:28:29.788 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.781+0000 7fa099ffb640 1 -- 172.21.15.89:0/315626084 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa09c014c80 con 0x7fa0a415ceb0 2023-10-30T12:28:29.788 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.781+0000 7fa099ffb640 1 -- 172.21.15.89:0/315626084 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fa09c013ab0 con 0x7fa0a415ceb0 2023-10-30T12:28:29.788 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.781+0000 7fa0ab780640 1 -- 172.21.15.89:0/315626084 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0a4148c40 con 0x7fa0a415ceb0 2023-10-30T12:28:29.788 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.785+0000 7fa099ffb640 1 -- 172.21.15.89:0/315626084 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fa09c018020 con 0x7fa0a415ceb0 2023-10-30T12:28:29.808 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-10-30T12:28:29.805+0000 7f7edc42e180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-30T12:28:29.843 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.841+0000 7fa0ab780640 1 -- 172.21.15.89:0/315626084 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0a404e170 con 0x7fa0a415ceb0 2023-10-30T12:28:29.844 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.841+0000 7fa099ffb640 1 -- 172.21.15.89:0/315626084 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fa09c013d90 con 0x7fa0a415ceb0 2023-10-30T12:28:29.845 INFO:teuthology.misc.health.smithi089.stdout: 2023-10-30T12:28:29.845 INFO:teuthology.misc.health.smithi089.stdout:{"epoch":10,"fsid":"401641be-db6e-4c16-b3ac-fd84a86c4de4","created":"2023-10-30T12:28:26.198532+0000","modified":"2023-10-30T12:28:28.517734+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T12:28:28.517734+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"72595c54-fd6e-4a5a-bb14-ff27e9377b07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0305670b-d7c4-4193-bec2-8b7351ff590a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4bab5426-a3c3-46b0-91fc-546c254006c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"7d47aea8-27cd-42be-8591-d63ad68c90e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2e2ae8f8-d097-4bba-9448-95047fbfacad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"49be5ce5-12f4-4c26-8d39-864b6ff67313","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"48623452-d456-4ea2-a0de-5fa4b37e6d10","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ff0a4384-54f8-43ec-a3ce-9c8ae3bf8a94","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:28:29.846 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.841+0000 7fa0ab780640 1 -- 172.21.15.89:0/315626084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0a415ceb0 msgr2=0x7fa0a41f2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:29.846 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.841+0000 7fa0ab780640 1 --2- 172.21.15.89:0/315626084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0a415ceb0 0x7fa0a41f2180 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa09c00efc0 tx=0x7fa09c00c490 comp rx=0 tx=0).stop 2023-10-30T12:28:29.846 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.841+0000 7fa0ab780640 1 -- 172.21.15.89:0/315626084 shutdown_connections 2023-10-30T12:28:29.847 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.841+0000 7fa0ab780640 1 --2- 172.21.15.89:0/315626084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0a415ceb0 0x7fa0a41f2180 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:29.847 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.841+0000 7fa0ab780640 1 --2- 172.21.15.89:0/315626084 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0a4153b50 0x7fa0a414ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:29.847 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.841+0000 7fa0ab780640 1 --2- 172.21.15.89:0/315626084 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0a40c7d40 0x7fa0a41497f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:29.847 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.841+0000 7fa0ab780640 1 -- 172.21.15.89:0/315626084 >> 172.21.15.89:0/315626084 conn(0x7fa0a40c8920 msgr2=0x7fa0a40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:29.847 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.841+0000 7fa0ab780640 1 -- 172.21.15.89:0/315626084 shutdown_connections 2023-10-30T12:28:29.848 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:29.841+0000 7fa0ab780640 1 -- 172.21.15.89:0/315626084 wait complete. 2023-10-30T12:28:29.863 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-30T12:28:30.220 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-30T12:28:30.216+0000 7fb4e738e180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-30T12:28:30.296 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-30T12:28:30.292+0000 7fb4e738e180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-30T12:28:30.466 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-30T12:28:30.464+0000 7fb4e738e180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-30T12:28:30.504 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-10-30T12:28:30.501+0000 7f7edc42e180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-30T12:28:30.539 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-30T12:28:30.536+0000 7fb4e738e180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-30T12:28:30.590 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-10-30T12:28:30.585+0000 7f7edc42e180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-30T12:28:30.778 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-10-30T12:28:30.773+0000 7f7edc42e180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-30T12:28:30.825 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-30T12:28:30.824+0000 7fb4e738e180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-30T12:28:30.862 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-10-30T12:28:30.857+0000 7f7edc42e180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-30T12:28:30.883 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-30T12:28:30.880+0000 7fb4e738e180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-30T12:28:31.196 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-10-30T12:28:31.193+0000 7f7edc42e180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-30T12:28:31.265 INFO:tasks.ceph.mgr.y.smithi089.stderr:2023-10-30T12:28:31.261+0000 7f7edc42e180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-30T12:28:31.936 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-30T12:28:31.932+0000 7fb48eb02640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-10-30T12:28:31.937 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-30T12:28:31.936+0000 7fb48eb02640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-10-30T12:28:33.082 INFO:tasks.ceph.osd.4.smithi192.stderr:2023-10-30T12:28:33.080+0000 7f08af7f9640 -1 osd.4 0 log_to_monitors true 2023-10-30T12:28:33.118 INFO:tasks.ceph.osd.7.smithi192.stderr:2023-10-30T12:28:33.116+0000 7f6f602bd640 -1 osd.7 0 log_to_monitors true 2023-10-30T12:28:33.135 INFO:tasks.ceph.osd.5.smithi192.stderr:2023-10-30T12:28:33.132+0000 7f923e239640 -1 osd.5 0 log_to_monitors true 2023-10-30T12:28:33.155 INFO:tasks.ceph.osd.0.smithi089.stderr:2023-10-30T12:28:33.153+0000 7fe354bfd640 -1 osd.0 0 log_to_monitors true 2023-10-30T12:28:33.171 INFO:tasks.ceph.osd.6.smithi192.stderr:2023-10-30T12:28:33.168+0000 7f99049ce640 -1 osd.6 0 log_to_monitors true 2023-10-30T12:28:33.223 INFO:tasks.ceph.osd.2.smithi089.stderr:2023-10-30T12:28:33.217+0000 7f9694a2b640 -1 osd.2 0 log_to_monitors true 2023-10-30T12:28:33.223 INFO:tasks.ceph.osd.1.smithi089.stderr:2023-10-30T12:28:33.221+0000 7fbe7ce2d640 -1 osd.1 0 log_to_monitors true 2023-10-30T12:28:33.349 INFO:tasks.ceph.osd.3.smithi089.stderr:2023-10-30T12:28:33.345+0000 7f0618396640 -1 osd.3 0 log_to_monitors true 2023-10-30T12:28:36.209 INFO:tasks.ceph.osd.0.smithi089.stderr:2023-10-30T12:28:36.205+0000 7fe350384640 -1 osd.0 0 waiting for initial osdmap 2023-10-30T12:28:36.239 INFO:tasks.ceph.osd.0.smithi089.stderr:2023-10-30T12:28:36.233+0000 7fe3489cb640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:28:36.281 INFO:tasks.ceph.osd.2.smithi089.stderr:2023-10-30T12:28:36.277+0000 7f968f6ab640 -1 osd.2 0 waiting for initial osdmap 2023-10-30T12:28:36.318 INFO:tasks.ceph.osd.2.smithi089.stderr:2023-10-30T12:28:36.313+0000 7f9687cf2640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:28:36.339 INFO:tasks.ceph.osd.1.smithi089.stderr:2023-10-30T12:28:36.337+0000 7fbe785b2640 -1 osd.1 0 waiting for initial osdmap 2023-10-30T12:28:36.369 INFO:tasks.ceph.osd.1.smithi089.stderr:2023-10-30T12:28:36.365+0000 7fbe70bf9640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:28:36.528 INFO:tasks.ceph.osd.4.smithi192.stderr:2023-10-30T12:28:36.524+0000 7f08ac7b6640 -1 osd.4 0 waiting for initial osdmap 2023-10-30T12:28:36.547 INFO:tasks.ceph.osd.5.smithi192.stderr:2023-10-30T12:28:36.544+0000 7f92399be640 -1 osd.5 0 waiting for initial osdmap 2023-10-30T12:28:36.565 INFO:tasks.ceph.osd.4.smithi192.stderr:2023-10-30T12:28:36.564+0000 7f08a35c5640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:28:36.573 INFO:tasks.ceph.osd.5.smithi192.stderr:2023-10-30T12:28:36.572+0000 7f9232005640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:28:36.669 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T12:28:36.688 INFO:tasks.ceph.osd.6.smithi192.stderr:2023-10-30T12:28:36.684+0000 7f9900153640 -1 osd.6 0 waiting for initial osdmap 2023-10-30T12:28:36.698 INFO:tasks.ceph.osd.7.smithi192.stderr:2023-10-30T12:28:36.696+0000 7f6f5ba42640 -1 osd.7 0 waiting for initial osdmap 2023-10-30T12:28:36.709 INFO:tasks.ceph.osd.6.smithi192.stderr:2023-10-30T12:28:36.708+0000 7f98f879a640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:28:36.718 INFO:tasks.ceph.osd.7.smithi192.stderr:2023-10-30T12:28:36.716+0000 7f6f54089640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:28:36.776 INFO:tasks.ceph.osd.3.smithi089.stderr:2023-10-30T12:28:36.773+0000 7f0613b1d640 -1 osd.3 0 waiting for initial osdmap 2023-10-30T12:28:36.803 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.801+0000 7f19d71a0640 1 Processor -- start 2023-10-30T12:28:36.804 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.801+0000 7f19d71a0640 1 -- start start 2023-10-30T12:28:36.804 INFO:tasks.ceph.osd.3.smithi089.stderr:2023-10-30T12:28:36.801+0000 7f060c164640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:28:36.806 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.801+0000 7f19d71a0640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f19d0153900 0x7f19d0159f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:36.806 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.801+0000 7f19d71a0640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19d006aa00 0x7f19d00cbca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:36.806 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.801+0000 7f19d71a0640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d015a460 con 0x7f19d00c9db0 2023-10-30T12:28:36.806 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.801+0000 7f19d71a0640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19d00ca380 con 0x7f19d0153900 2023-10-30T12:28:36.806 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.801+0000 7f19d71a0640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19d0153ce0 con 0x7f19d006aa00 2023-10-30T12:28:36.807 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.801+0000 7f19d619e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f19d0153900 0x7f19d0159f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:36.807 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.801+0000 7f19d619e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f19d0153900 0x7f19d0159f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58238/0 (socket says 172.21.15.89:58238) 2023-10-30T12:28:36.807 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.801+0000 7f19d619e640 1 -- 172.21.15.89:0/2740668992 learned_addr learned my addr 172.21.15.89:0/2740668992 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:36.807 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.801+0000 7f19d599d640 1 --2- 172.21.15.89:0/2740668992 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19d006aa00 0x7f19d00cbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:36.808 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.801+0000 7f19d619e640 1 -- 172.21.15.89:0/2740668992 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19d006aa00 msgr2=0x7f19d00cbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:36.808 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.801+0000 7f19d619e640 1 --2- 172.21.15.89:0/2740668992 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19d006aa00 0x7f19d00cbca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:36.808 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.801+0000 7f19d619e640 1 -- 172.21.15.89:0/2740668992 >> v1:172.21.15.192:6789/0 conn(0x7f19d00c9db0 legacy=0x7f19d0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:36.808 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.801+0000 7f19d619e640 1 -- 172.21.15.89:0/2740668992 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d00cac00 con 0x7f19d0153900 2023-10-30T12:28:36.809 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.805+0000 7f19d619e640 1 --2- 172.21.15.89:0/2740668992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19d0153900 0x7f19d0159f20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f19cc0098a0 tx=0x7f19cc01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d115487c54466c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:36.809 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.805+0000 7f19d519c640 1 -- 172.21.15.89:0/2740668992 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19cc0aa670 con 0x7f19d0153900 2023-10-30T12:28:36.809 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.805+0000 7f19d519c640 1 -- 172.21.15.89:0/2740668992 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19cc0aa7d0 con 0x7f19d0153900 2023-10-30T12:28:36.810 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.805+0000 7f19d519c640 1 -- 172.21.15.89:0/2740668992 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19cc0aab60 con 0x7f19d0153900 2023-10-30T12:28:36.811 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.805+0000 7f19d71a0640 1 -- 172.21.15.89:0/2740668992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19d0153900 msgr2=0x7f19d0159f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:36.811 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.805+0000 7f19d71a0640 1 --2- 172.21.15.89:0/2740668992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19d0153900 0x7f19d0159f20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f19cc0098a0 tx=0x7f19cc01c4f0 comp rx=0 tx=0).stop 2023-10-30T12:28:36.811 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.805+0000 7f19d71a0640 1 -- 172.21.15.89:0/2740668992 shutdown_connections 2023-10-30T12:28:36.811 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.805+0000 7f19d71a0640 1 --2- 172.21.15.89:0/2740668992 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19d006aa00 0x7f19d00cbca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:36.812 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.805+0000 7f19d71a0640 1 --2- 172.21.15.89:0/2740668992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19d0153900 0x7f19d0159f20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:36.812 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.805+0000 7f19d71a0640 1 -- 172.21.15.89:0/2740668992 >> 172.21.15.89:0/2740668992 conn(0x7f19d006bf40 msgr2=0x7f19d006c350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:36.812 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.805+0000 7f19d71a0640 1 -- 172.21.15.89:0/2740668992 shutdown_connections 2023-10-30T12:28:36.812 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.805+0000 7f19d71a0640 1 -- 172.21.15.89:0/2740668992 wait complete. 2023-10-30T12:28:36.815 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.809+0000 7f19d71a0640 1 Processor -- start 2023-10-30T12:28:36.815 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.809+0000 7f19d71a0640 1 -- start start 2023-10-30T12:28:36.815 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.809+0000 7f19d71a0640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19d006aa00 0x7f19d0145cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:36.815 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.809+0000 7f19d71a0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19d00c9db0 0x7f19d0154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:36.816 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.809+0000 7f19d71a0640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19d0153900 0x7f19d0154770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:36.816 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.809+0000 7f19d71a0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d015acb0 con 0x7f19d00c9db0 2023-10-30T12:28:36.816 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.809+0000 7f19d71a0640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d0146200 con 0x7f19d0153900 2023-10-30T12:28:36.816 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.809+0000 7f19d71a0640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19d01f2df0 con 0x7f19d006aa00 2023-10-30T12:28:36.816 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.809+0000 7f19d599d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19d00c9db0 0x7f19d0154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:36.817 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.809+0000 7f19d599d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19d00c9db0 0x7f19d0154230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58248/0 (socket says 172.21.15.89:58248) 2023-10-30T12:28:36.817 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.809+0000 7f19d599d640 1 -- 172.21.15.89:0/1519721907 learned_addr learned my addr 172.21.15.89:0/1519721907 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:36.817 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.809+0000 7f19d619e640 1 --2- 172.21.15.89:0/1519721907 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19d006aa00 0x7f19d0145cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:36.817 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.809+0000 7f19d699f640 1 --2- 172.21.15.89:0/1519721907 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19d0153900 0x7f19d0154770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:36.817 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.809+0000 7f19d599d640 1 -- 172.21.15.89:0/1519721907 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19d006aa00 msgr2=0x7f19d0145cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:36.818 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.809+0000 7f19d599d640 1 --2- 172.21.15.89:0/1519721907 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19d006aa00 0x7f19d0145cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:36.818 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.809+0000 7f19d599d640 1 -- 172.21.15.89:0/1519721907 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19d0153900 msgr2=0x7f19d0154770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:36.818 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.809+0000 7f19d599d640 1 --2- 172.21.15.89:0/1519721907 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19d0153900 0x7f19d0154770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:36.818 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.809+0000 7f19d599d640 1 -- 172.21.15.89:0/1519721907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19cc0094d0 con 0x7f19d00c9db0 2023-10-30T12:28:36.819 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.809+0000 7f19d599d640 1 --2- 172.21.15.89:0/1519721907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19d00c9db0 0x7f19d0154230 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f19c000e930 tx=0x7f19c000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:36.819 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.809+0000 7f19c6ffd640 1 -- 172.21.15.89:0/1519721907 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c0009690 con 0x7f19d00c9db0 2023-10-30T12:28:36.819 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.809+0000 7f19c6ffd640 1 -- 172.21.15.89:0/1519721907 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19c000ca00 con 0x7f19d00c9db0 2023-10-30T12:28:36.819 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.809+0000 7f19c6ffd640 1 -- 172.21.15.89:0/1519721907 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c000cd70 con 0x7f19d00c9db0 2023-10-30T12:28:36.819 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.809+0000 7f19d71a0640 1 -- 172.21.15.89:0/1519721907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19d01f30d0 con 0x7f19d00c9db0 2023-10-30T12:28:36.820 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.809+0000 7f19d71a0640 1 -- 172.21.15.89:0/1519721907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19d01f3690 con 0x7f19d00c9db0 2023-10-30T12:28:36.820 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.809+0000 7f19d71a0640 1 -- 172.21.15.89:0/1519721907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d0154010 con 0x7f19d00c9db0 2023-10-30T12:28:36.821 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.813+0000 7f19c6ffd640 1 -- 172.21.15.89:0/1519721907 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 4) v1 ==== 85794+0+0 (secure 0 0 0) 0x7f19c0013470 con 0x7f19d00c9db0 2023-10-30T12:28:36.821 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.813+0000 7f19c6ffd640 1 --2- 172.21.15.89:0/1519721907 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f19b4066410 0x7f19b40688d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:36.821 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.813+0000 7f19c6ffd640 1 -- 172.21.15.89:0/1519721907 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f19c00830f0 con 0x7f19d00c9db0 2023-10-30T12:28:36.821 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.813+0000 7f19d619e640 1 --2- 172.21.15.89:0/1519721907 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f19b4066410 0x7f19b40688d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:36.822 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.813+0000 7f19d619e640 1 --2- 172.21.15.89:0/1519721907 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f19b4066410 0x7f19b40688d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f19cc0ae000 tx=0x7f19cc0b5040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:36.822 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.817+0000 7f19c6ffd640 1 -- 172.21.15.89:0/1519721907 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f19c00515e0 con 0x7f19d00c9db0 2023-10-30T12:28:36.970 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.965+0000 7f19d71a0640 1 -- 172.21.15.89:0/1519721907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19d00ca8b0 con 0x7f19d00c9db0 2023-10-30T12:28:36.971 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.965+0000 7f19c6ffd640 1 -- 172.21.15.89:0/1519721907 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f19c0017090 con 0x7f19d00c9db0 2023-10-30T12:28:36.971 INFO:teuthology.misc.health.smithi089.stdout: 2023-10-30T12:28:36.971 INFO:teuthology.misc.health.smithi089.stdout:{"epoch":13,"fsid":"401641be-db6e-4c16-b3ac-fd84a86c4de4","created":"2023-10-30T12:28:26.198532+0000","modified":"2023-10-30T12:28:35.264718+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T12:28:28.517734+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"72595c54-fd6e-4a5a-bb14-ff27e9377b07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0305670b-d7c4-4193-bec2-8b7351ff590a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4bab5426-a3c3-46b0-91fc-546c254006c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"7d47aea8-27cd-42be-8591-d63ad68c90e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2e2ae8f8-d097-4bba-9448-95047fbfacad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"49be5ce5-12f4-4c26-8d39-864b6ff67313","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"48623452-d456-4ea2-a0de-5fa4b37e6d10","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ff0a4384-54f8-43ec-a3ce-9c8ae3bf8a94","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:28:36.973 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.969+0000 7f19d71a0640 1 -- 172.21.15.89:0/1519721907 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f19b4066410 msgr2=0x7f19b40688d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:36.973 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.969+0000 7f19d71a0640 1 --2- 172.21.15.89:0/1519721907 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f19b4066410 0x7f19b40688d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f19cc0ae000 tx=0x7f19cc0b5040 comp rx=0 tx=0).stop 2023-10-30T12:28:36.973 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.969+0000 7f19d71a0640 1 -- 172.21.15.89:0/1519721907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19d00c9db0 msgr2=0x7f19d0154230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:36.974 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.969+0000 7f19d71a0640 1 --2- 172.21.15.89:0/1519721907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19d00c9db0 0x7f19d0154230 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f19c000e930 tx=0x7f19c000edf0 comp rx=0 tx=0).stop 2023-10-30T12:28:36.974 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.969+0000 7f19d71a0640 1 -- 172.21.15.89:0/1519721907 shutdown_connections 2023-10-30T12:28:36.974 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.969+0000 7f19d71a0640 1 --2- 172.21.15.89:0/1519721907 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f19b4066410 0x7f19b40688d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:36.974 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.969+0000 7f19d71a0640 1 --2- 172.21.15.89:0/1519721907 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19d0153900 0x7f19d0154770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:36.975 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.969+0000 7f19d71a0640 1 --2- 172.21.15.89:0/1519721907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19d00c9db0 0x7f19d0154230 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:36.975 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.969+0000 7f19d71a0640 1 --2- 172.21.15.89:0/1519721907 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19d006aa00 0x7f19d0145cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:36.975 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.969+0000 7f19d71a0640 1 -- 172.21.15.89:0/1519721907 >> 172.21.15.89:0/1519721907 conn(0x7f19d006bf40 msgr2=0x7f19d0146460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:36.975 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.969+0000 7f19d71a0640 1 -- 172.21.15.89:0/1519721907 shutdown_connections 2023-10-30T12:28:36.975 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:36.969+0000 7f19d71a0640 1 -- 172.21.15.89:0/1519721907 wait complete. 2023-10-30T12:28:36.990 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-30T12:28:43.797 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T12:28:43.930 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.924+0000 7fa22e5b8640 1 Processor -- start 2023-10-30T12:28:43.930 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.924+0000 7fa22e5b8640 1 -- start start 2023-10-30T12:28:43.931 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.924+0000 7fa22e5b8640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa22806fb80 0x7fa2280c2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:43.932 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.924+0000 7fa22e5b8640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa2280708e0 0x7fa228157c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:43.932 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.924+0000 7fa22e5b8640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa228070070 con 0x7fa2281515d0 2023-10-30T12:28:43.932 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.924+0000 7fa22e5b8640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa228070350 con 0x7fa2280708e0 2023-10-30T12:28:43.932 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.924+0000 7fa22e5b8640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2280b0a10 con 0x7fa22806fb80 2023-10-30T12:28:43.933 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.924+0000 7fa22cb55640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa2280708e0 0x7fa228157c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:43.933 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.928+0000 7fa2277fe640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa22806fb80 0x7fa2280c2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:43.933 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.928+0000 7fa22cb55640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa2280708e0 0x7fa228157c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35742/0 (socket says 172.21.15.89:35742) 2023-10-30T12:28:43.933 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.928+0000 7fa22cb55640 1 -- 172.21.15.89:0/521011477 learned_addr learned my addr 172.21.15.89:0/521011477 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:43.933 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.928+0000 7fa22cb55640 1 -- 172.21.15.89:0/521011477 >> v2:172.21.15.192:3300/0 conn(0x7fa22806fb80 msgr2=0x7fa2280c2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:43.934 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.928+0000 7fa22cb55640 1 --2- 172.21.15.89:0/521011477 >> v2:172.21.15.192:3300/0 conn(0x7fa22806fb80 0x7fa2280c2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:43.934 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.928+0000 7fa22cb55640 1 -- 172.21.15.89:0/521011477 >> v1:172.21.15.192:6789/0 conn(0x7fa2281515d0 legacy=0x7fa2280c4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:43.934 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.928+0000 7fa22cb55640 1 -- 172.21.15.89:0/521011477 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2280b0cf0 con 0x7fa2280708e0 2023-10-30T12:28:43.935 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.928+0000 7fa22cb55640 1 --2- 172.21.15.89:0/521011477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2280708e0 0x7fa228157c90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa22806eb60 tx=0x7fa22001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a396f06986c2095 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:43.935 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.928+0000 7fa226ffd640 1 -- 172.21.15.89:0/521011477 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2200af400 con 0x7fa2280708e0 2023-10-30T12:28:43.935 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.928+0000 7fa226ffd640 1 -- 172.21.15.89:0/521011477 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2200ab050 con 0x7fa2280708e0 2023-10-30T12:28:43.935 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.928+0000 7fa226ffd640 1 -- 172.21.15.89:0/521011477 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2200af780 con 0x7fa2280708e0 2023-10-30T12:28:43.936 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.932+0000 7fa22e5b8640 1 -- 172.21.15.89:0/521011477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2280708e0 msgr2=0x7fa228157c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:43.936 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.932+0000 7fa22e5b8640 1 --2- 172.21.15.89:0/521011477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2280708e0 0x7fa228157c90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa22806eb60 tx=0x7fa22001f1c0 comp rx=0 tx=0).stop 2023-10-30T12:28:43.937 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.932+0000 7fa22e5b8640 1 -- 172.21.15.89:0/521011477 shutdown_connections 2023-10-30T12:28:43.937 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.932+0000 7fa22e5b8640 1 --2- 172.21.15.89:0/521011477 >> v2:172.21.15.192:3300/0 conn(0x7fa22806fb80 0x7fa2280c2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:43.937 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.932+0000 7fa22e5b8640 1 --2- 172.21.15.89:0/521011477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2280708e0 0x7fa228157c90 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:43.937 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.932+0000 7fa22e5b8640 1 -- 172.21.15.89:0/521011477 >> 172.21.15.89:0/521011477 conn(0x7fa2280cb0b0 msgr2=0x7fa2280cb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:43.937 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.932+0000 7fa22e5b8640 1 -- 172.21.15.89:0/521011477 shutdown_connections 2023-10-30T12:28:43.938 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.932+0000 7fa22e5b8640 1 -- 172.21.15.89:0/521011477 wait complete. 2023-10-30T12:28:43.940 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.932+0000 7fa22e5b8640 1 Processor -- start 2023-10-30T12:28:43.940 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.932+0000 7fa22e5b8640 1 -- start start 2023-10-30T12:28:43.940 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.932+0000 7fa22e5b8640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa22806fb80 0x7fa228156f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:43.940 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.932+0000 7fa22e5b8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2280708e0 0x7fa2280b0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:43.940 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.932+0000 7fa22e5b8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2281515d0 0x7fa228156460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:43.941 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.932+0000 7fa22e5b8640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2281f2140 con 0x7fa2281515d0 2023-10-30T12:28:43.941 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.932+0000 7fa22e5b8640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2280b0840 con 0x7fa2280708e0 2023-10-30T12:28:43.941 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.932+0000 7fa22e5b8640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa228152030 con 0x7fa22806fb80 2023-10-30T12:28:43.941 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.932+0000 7fa22cb55640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2281515d0 0x7fa228156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:43.941 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.932+0000 7fa22cb55640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2281515d0 0x7fa228156460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35746/0 (socket says 172.21.15.89:35746) 2023-10-30T12:28:43.942 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.932+0000 7fa22cb55640 1 -- 172.21.15.89:0/1458311442 learned_addr learned my addr 172.21.15.89:0/1458311442 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:43.942 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.932+0000 7fa227fff640 1 --2- 172.21.15.89:0/1458311442 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa22806fb80 0x7fa228156f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:43.942 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.932+0000 7fa2277fe640 1 --2- 172.21.15.89:0/1458311442 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2280708e0 0x7fa2280b0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:43.942 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.932+0000 7fa22cb55640 1 -- 172.21.15.89:0/1458311442 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa22806fb80 msgr2=0x7fa228156f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:43.942 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.932+0000 7fa22cb55640 1 --2- 172.21.15.89:0/1458311442 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa22806fb80 0x7fa228156f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:43.943 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.932+0000 7fa22cb55640 1 -- 172.21.15.89:0/1458311442 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2280708e0 msgr2=0x7fa2280b0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:43.943 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.932+0000 7fa22cb55640 1 --2- 172.21.15.89:0/1458311442 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2280708e0 0x7fa2280b0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:43.943 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.932+0000 7fa22cb55640 1 -- 172.21.15.89:0/1458311442 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa220009b30 con 0x7fa2281515d0 2023-10-30T12:28:43.943 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.932+0000 7fa227fff640 1 --2- 172.21.15.89:0/1458311442 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa22806fb80 0x7fa228156f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:43.943 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.936+0000 7fa22cb55640 1 --2- 172.21.15.89:0/1458311442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2281515d0 0x7fa228156460 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa2200b2040 tx=0x7fa22001bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:43.943 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.936+0000 7fa224ff9640 1 -- 172.21.15.89:0/1458311442 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2200afd30 con 0x7fa2281515d0 2023-10-30T12:28:43.944 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.936+0000 7fa224ff9640 1 -- 172.21.15.89:0/1458311442 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa220009510 con 0x7fa2281515d0 2023-10-30T12:28:43.944 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.936+0000 7fa224ff9640 1 -- 172.21.15.89:0/1458311442 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2200ba680 con 0x7fa2281515d0 2023-10-30T12:28:43.944 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.936+0000 7fa22e5b8640 1 -- 172.21.15.89:0/1458311442 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2281f2f30 con 0x7fa2281515d0 2023-10-30T12:28:43.944 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.936+0000 7fa22e5b8640 1 -- 172.21.15.89:0/1458311442 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2281f2600 con 0x7fa2281515d0 2023-10-30T12:28:43.945 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.936+0000 7fa22e5b8640 1 -- 172.21.15.89:0/1458311442 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa22804e170 con 0x7fa2281515d0 2023-10-30T12:28:43.945 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.936+0000 7fa224ff9640 1 -- 172.21.15.89:0/1458311442 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa2200b6040 con 0x7fa2281515d0 2023-10-30T12:28:43.946 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.936+0000 7fa224ff9640 1 --2- 172.21.15.89:0/1458311442 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa2140664d0 0x7fa214068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:43.946 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.936+0000 7fa224ff9640 1 -- 172.21.15.89:0/1458311442 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa220121d80 con 0x7fa2281515d0 2023-10-30T12:28:43.946 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.936+0000 7fa227fff640 1 --2- 172.21.15.89:0/1458311442 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa2140664d0 0x7fa214068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:43.946 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.940+0000 7fa227fff640 1 --2- 172.21.15.89:0/1458311442 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa2140664d0 0x7fa214068990 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa218000f70 tx=0x7fa218009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:43.947 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:43.940+0000 7fa224ff9640 1 -- 172.21.15.89:0/1458311442 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa2200ef600 con 0x7fa2281515d0 2023-10-30T12:28:44.066 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:28:44.071 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:28:44.098 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:44.092+0000 7fa22e5b8640 1 -- 172.21.15.89:0/1458311442 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2280706c0 con 0x7fa2281515d0 2023-10-30T12:28:44.099 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:44.092+0000 7fa224ff9640 1 -- 172.21.15.89:0/1458311442 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12850 (secure 0 0 0) 0x7fa2200f2c20 con 0x7fa2281515d0 2023-10-30T12:28:44.100 INFO:teuthology.misc.health.smithi089.stdout: 2023-10-30T12:28:44.100 INFO:teuthology.misc.health.smithi089.stdout:{"epoch":18,"fsid":"401641be-db6e-4c16-b3ac-fd84a86c4de4","created":"2023-10-30T12:28:26.198532+0000","modified":"2023-10-30T12:28:41.916008+0000","last_up_change":"2023-10-30T12:28:37.212205+0000","last_in_change":"2023-10-30T12:28:28.517734+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:39.004943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72595c54-fd6e-4a5a-bb14-ff27e9377b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6801","nonce":1673955622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6803","nonce":1673955622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6807","nonce":1673955622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6805","nonce":1673955622}]},"public_addr":"172.21.15.89:6801/1673955622","cluster_addr":"172.21.15.89:6803/1673955622","heartbeat_back_addr":"172.21.15.89:6807/1673955622","heartbeat_front_addr":"172.21.15.89:6805/1673955622","state":["exists","up"]},{"osd":1,"uuid":"0305670b-d7c4-4193-bec2-8b7351ff590a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6817","nonce":3340572039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6819","nonce":3340572039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6823","nonce":3340572039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6821","nonce":3340572039}]},"public_addr":"172.21.15.89:6817/3340572039","cluster_addr":"172.21.15.89:6819/3340572039","heartbeat_back_addr":"172.21.15.89:6823/3340572039","heartbeat_front_addr":"172.21.15.89:6821/3340572039","state":["exists","up"]},{"osd":2,"uuid":"4bab5426-a3c3-46b0-91fc-546c254006c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6809","nonce":1929940799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6811","nonce":1929940799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6815","nonce":1929940799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6813","nonce":1929940799}]},"public_addr":"172.21.15.89:6809/1929940799","cluster_addr":"172.21.15.89:6811/1929940799","heartbeat_back_addr":"172.21.15.89:6815/1929940799","heartbeat_front_addr":"172.21.15.89:6813/1929940799","state":["exists","up"]},{"osd":3,"uuid":"7d47aea8-27cd-42be-8591-d63ad68c90e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6825","nonce":345087939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6827","nonce":345087939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6831","nonce":345087939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6829","nonce":345087939}]},"public_addr":"172.21.15.89:6825/345087939","cluster_addr":"172.21.15.89:6827/345087939","heartbeat_back_addr":"172.21.15.89:6831/345087939","heartbeat_front_addr":"172.21.15.89:6829/345087939","state":["exists","up"]},{"osd":4,"uuid":"2e2ae8f8-d097-4bba-9448-95047fbfacad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6801","nonce":2138125545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6803","nonce":2138125545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6809","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6811","nonce":2138125545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6805","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6807","nonce":2138125545}]},"public_addr":"172.21.15.192:6801/2138125545","cluster_addr":"172.21.15.192:6803/2138125545","heartbeat_back_addr":"172.21.15.192:6811/2138125545","heartbeat_front_addr":"172.21.15.192:6807/2138125545","state":["exists","up"]},{"osd":5,"uuid":"49be5ce5-12f4-4c26-8d39-864b6ff67313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6806","nonce":2360218671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6810","nonce":2360218671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6815","nonce":2360218671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6813","nonce":2360218671}]},"public_addr":"172.21.15.192:6806/2360218671","cluster_addr":"172.21.15.192:6810/2360218671","heartbeat_back_addr":"172.21.15.192:6815/2360218671","heartbeat_front_addr":"172.21.15.192:6813/2360218671","state":["exists","up"]},{"osd":6,"uuid":"48623452-d456-4ea2-a0de-5fa4b37e6d10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6825","nonce":3707671287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6827","nonce":3707671287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6831","nonce":3707671287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6829","nonce":3707671287}]},"public_addr":"172.21.15.192:6825/3707671287","cluster_addr":"172.21.15.192:6827/3707671287","heartbeat_back_addr":"172.21.15.192:6831/3707671287","heartbeat_front_addr":"172.21.15.192:6829/3707671287","state":["exists","up"]},{"osd":7,"uuid":"ff0a4384-54f8-43ec-a3ce-9c8ae3bf8a94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6817","nonce":3111945871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6819","nonce":3111945871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6823","nonce":3111945871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6821","nonce":3111945871}]},"public_addr":"172.21.15.192:6817/3111945871","cluster_addr":"172.21.15.192:6819/3111945871","heartbeat_back_addr":"172.21.15.192:6823/3111945871","heartbeat_front_addr":"172.21.15.192:6821/3111945871","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.183065+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.248650+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.194576+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.397483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.043648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.186624+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.159674+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.100851+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:28:44.101 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:44.096+0000 7fa22e5b8640 1 -- 172.21.15.89:0/1458311442 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa2140664d0 msgr2=0x7fa214068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:44.102 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:44.096+0000 7fa22e5b8640 1 --2- 172.21.15.89:0/1458311442 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa2140664d0 0x7fa214068990 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa218000f70 tx=0x7fa218009040 comp rx=0 tx=0).stop 2023-10-30T12:28:44.102 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:44.096+0000 7fa22e5b8640 1 -- 172.21.15.89:0/1458311442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2281515d0 msgr2=0x7fa228156460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:44.102 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:44.096+0000 7fa22e5b8640 1 --2- 172.21.15.89:0/1458311442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2281515d0 0x7fa228156460 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa2200b2040 tx=0x7fa22001bf60 comp rx=0 tx=0).stop 2023-10-30T12:28:44.102 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:44.096+0000 7fa22e5b8640 1 -- 172.21.15.89:0/1458311442 shutdown_connections 2023-10-30T12:28:44.102 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:44.096+0000 7fa22e5b8640 1 --2- 172.21.15.89:0/1458311442 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa2140664d0 0x7fa214068990 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.103 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:44.096+0000 7fa22e5b8640 1 --2- 172.21.15.89:0/1458311442 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2280708e0 0x7fa2280b0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.103 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:44.096+0000 7fa22e5b8640 1 --2- 172.21.15.89:0/1458311442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2281515d0 0x7fa228156460 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.103 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:44.096+0000 7fa22e5b8640 1 --2- 172.21.15.89:0/1458311442 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa22806fb80 0x7fa228156f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.103 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:44.096+0000 7fa22e5b8640 1 -- 172.21.15.89:0/1458311442 >> 172.21.15.89:0/1458311442 conn(0x7fa2280cb0b0 msgr2=0x7fa2280c4380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:44.103 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:44.096+0000 7fa22e5b8640 1 -- 172.21.15.89:0/1458311442 shutdown_connections 2023-10-30T12:28:44.104 INFO:teuthology.misc.health.smithi089.stderr:2023-10-30T12:28:44.096+0000 7fa22e5b8640 1 -- 172.21.15.89:0/1458311442 wait complete. 2023-10-30T12:28:44.118 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-10-30T12:28:44.119 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-30T12:28:44.119 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T12:28:44.119 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T12:28:44.165 INFO:tasks.ceph.mds.a:Started 2023-10-30T12:28:44.166 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T12:28:44.166 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T12:28:44.170 INFO:tasks.ceph.mds.c:Started 2023-10-30T12:28:44.170 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T12:28:44.170 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T12:28:44.174 INFO:tasks.ceph.mds.e:Started 2023-10-30T12:28:44.174 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-30T12:28:44.174 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-30T12:28:44.177 INFO:tasks.ceph.mds.b:Started 2023-10-30T12:28:44.177 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-30T12:28:44.178 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-30T12:28:44.181 INFO:tasks.ceph.mds.d:Started 2023-10-30T12:28:44.181 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-30T12:28:44.181 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-30T12:28:44.184 INFO:tasks.ceph.mds.f:Started 2023-10-30T12:28:44.184 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-10-30T12:28:44.185 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:28:44.278 INFO:tasks.ceph.mds.e.smithi089.stdout:starting mds.e at 2023-10-30T12:28:44.279 INFO:tasks.ceph.mds.a.smithi089.stdout:starting mds.a at 2023-10-30T12:28:44.291 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2023-10-30T12:28:44.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.340+0000 7f654d0dc640 1 Processor -- start 2023-10-30T12:28:44.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.340+0000 7f654d0dc640 1 -- start start 2023-10-30T12:28:44.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.340+0000 7f654d0dc640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f654815cd50 0x7f65480c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:44.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.340+0000 7f654d0dc640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f65480cefa0 0x7f65480cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:44.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.340+0000 7f654d0dc640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f654815a6c0 con 0x7f65481538f0 2023-10-30T12:28:44.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.340+0000 7f654d0dc640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65480cf8f0 con 0x7f65480cefa0 2023-10-30T12:28:44.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.340+0000 7f654d0dc640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f654815d300 con 0x7f654815cd50 2023-10-30T12:28:44.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.340+0000 7f654c8db640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f65480cefa0 0x7f65480cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:44.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.340+0000 7f654c8db640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f65480cefa0 0x7f65480cf3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35810/0 (socket says 172.21.15.89:35810) 2023-10-30T12:28:44.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.340+0000 7f654c8db640 1 -- 172.21.15.89:0/3550631531 learned_addr learned my addr 172.21.15.89:0/3550631531 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:44.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.340+0000 7f65477fe640 1 --2- 172.21.15.89:0/3550631531 >> v2:172.21.15.192:3300/0 conn(0x7f654815cd50 0x7f65480c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:44.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.340+0000 7f654c8db640 1 -- 172.21.15.89:0/3550631531 >> v2:172.21.15.192:3300/0 conn(0x7f654815cd50 msgr2=0x7f65480c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:44.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.340+0000 7f654c8db640 1 --2- 172.21.15.89:0/3550631531 >> v2:172.21.15.192:3300/0 conn(0x7f654815cd50 0x7f65480c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.340+0000 7f654c8db640 1 -- 172.21.15.89:0/3550631531 >> v1:172.21.15.192:6789/0 conn(0x7f65481538f0 legacy=0x7f6548159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:44.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.340+0000 7f654c8db640 1 -- 172.21.15.89:0/3550631531 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f654815d5e0 con 0x7f65480cefa0 2023-10-30T12:28:44.357 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.344+0000 7f654c8db640 1 --2- 172.21.15.89:0/3550631531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65480cefa0 0x7f65480cf3b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6548072e90 tx=0x7f654001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcf86665bc37a11b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:44.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.344+0000 7f6546ffd640 1 -- 172.21.15.89:0/3550631531 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65400af400 con 0x7f65480cefa0 2023-10-30T12:28:44.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.344+0000 7f6546ffd640 1 -- 172.21.15.89:0/3550631531 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65400ab050 con 0x7f65480cefa0 2023-10-30T12:28:44.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.344+0000 7f6546ffd640 1 -- 172.21.15.89:0/3550631531 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65400af780 con 0x7f65480cefa0 2023-10-30T12:28:44.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.344+0000 7f654d0dc640 1 -- 172.21.15.89:0/3550631531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65480cefa0 msgr2=0x7f65480cf3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:44.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.344+0000 7f654d0dc640 1 --2- 172.21.15.89:0/3550631531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65480cefa0 0x7f65480cf3b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6548072e90 tx=0x7f654001f1c0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.344+0000 7f654d0dc640 1 -- 172.21.15.89:0/3550631531 shutdown_connections 2023-10-30T12:28:44.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.344+0000 7f654d0dc640 1 --2- 172.21.15.89:0/3550631531 >> v2:172.21.15.192:3300/0 conn(0x7f654815cd50 0x7f65480c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.344+0000 7f654d0dc640 1 --2- 172.21.15.89:0/3550631531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65480cefa0 0x7f65480cf3b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.344+0000 7f654d0dc640 1 -- 172.21.15.89:0/3550631531 >> 172.21.15.89:0/3550631531 conn(0x7f654806a640 msgr2=0x7f6548065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:44.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.344+0000 7f654d0dc640 1 -- 172.21.15.89:0/3550631531 shutdown_connections 2023-10-30T12:28:44.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.344+0000 7f654d0dc640 1 -- 172.21.15.89:0/3550631531 wait complete. 2023-10-30T12:28:44.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.348+0000 7f654d0dc640 1 Processor -- start 2023-10-30T12:28:44.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.348+0000 7f654d0dc640 1 -- start start 2023-10-30T12:28:44.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.348+0000 7f654d0dc640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65480cefa0 0x7f65481573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:44.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.348+0000 7f654d0dc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65481538f0 0x7f6548144de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:44.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.348+0000 7f654d0dc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f654815cd50 0x7f6548145320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:44.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.348+0000 7f654d0dc640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65480c7a20 con 0x7f654815cd50 2023-10-30T12:28:44.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.348+0000 7f654d0dc640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6548152b30 con 0x7f65481538f0 2023-10-30T12:28:44.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.348+0000 7f654d0dc640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6548152df0 con 0x7f65480cefa0 2023-10-30T12:28:44.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.348+0000 7f654c8db640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f654815cd50 0x7f6548145320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:44.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.348+0000 7f654c8db640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f654815cd50 0x7f6548145320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35816/0 (socket says 172.21.15.89:35816) 2023-10-30T12:28:44.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.348+0000 7f654c8db640 1 -- 172.21.15.89:0/3003523454 learned_addr learned my addr 172.21.15.89:0/3003523454 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:44.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.348+0000 7f6547fff640 1 --2- 172.21.15.89:0/3003523454 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65480cefa0 0x7f65481573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:44.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.348+0000 7f65477fe640 1 --2- 172.21.15.89:0/3003523454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65481538f0 0x7f6548144de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:44.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.348+0000 7f654c8db640 1 -- 172.21.15.89:0/3003523454 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65480cefa0 msgr2=0x7f65481573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:44.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.348+0000 7f654c8db640 1 --2- 172.21.15.89:0/3003523454 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65480cefa0 0x7f65481573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.348+0000 7f654c8db640 1 -- 172.21.15.89:0/3003523454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65481538f0 msgr2=0x7f6548144de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:44.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.348+0000 7f654c8db640 1 --2- 172.21.15.89:0/3003523454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65481538f0 0x7f6548144de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.348+0000 7f654c8db640 1 -- 172.21.15.89:0/3003523454 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6540009b30 con 0x7f654815cd50 2023-10-30T12:28:44.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.348+0000 7f654c8db640 1 --2- 172.21.15.89:0/3003523454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f654815cd50 0x7f6548145320 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f65400b2040 tx=0x7f654001bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:44.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.348+0000 7f6544ff9640 1 -- 172.21.15.89:0/3003523454 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65400afd30 con 0x7f654815cd50 2023-10-30T12:28:44.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.348+0000 7f6544ff9640 1 -- 172.21.15.89:0/3003523454 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6540009510 con 0x7f654815cd50 2023-10-30T12:28:44.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.348+0000 7f654d0dc640 1 -- 172.21.15.89:0/3003523454 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65481530d0 con 0x7f654815cd50 2023-10-30T12:28:44.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.348+0000 7f6544ff9640 1 -- 172.21.15.89:0/3003523454 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65400ba660 con 0x7f654815cd50 2023-10-30T12:28:44.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.348+0000 7f654d0dc640 1 -- 172.21.15.89:0/3003523454 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6548152250 con 0x7f654815cd50 2023-10-30T12:28:44.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.352+0000 7f654d0dc640 1 -- 172.21.15.89:0/3003523454 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f654804e170 con 0x7f654815cd50 2023-10-30T12:28:44.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.352+0000 7f6544ff9640 1 -- 172.21.15.89:0/3003523454 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f65400b6040 con 0x7f654815cd50 2023-10-30T12:28:44.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.352+0000 7f6544ff9640 1 --2- 172.21.15.89:0/3003523454 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f65340664d0 0x7f6534068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:44.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.352+0000 7f6544ff9640 1 -- 172.21.15.89:0/3003523454 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6540121c70 con 0x7f654815cd50 2023-10-30T12:28:44.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.352+0000 7f6547fff640 1 --2- 172.21.15.89:0/3003523454 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f65340664d0 0x7f6534068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:44.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.352+0000 7f6547fff640 1 --2- 172.21.15.89:0/3003523454 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f65340664d0 0x7f6534068990 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6538007fd0 tx=0x7f6538009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:44.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.356+0000 7f6544ff9640 1 -- 172.21.15.89:0/3003523454 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f65400ef470 con 0x7f654815cd50 2023-10-30T12:28:44.405 INFO:tasks.ceph.mds.b.smithi192.stdout:starting mds.b at 2023-10-30T12:28:44.407 INFO:tasks.ceph.mds.f.smithi192.stdout:starting mds.f at 2023-10-30T12:28:44.414 INFO:tasks.ceph.mds.d.smithi192.stdout:starting mds.d at 2023-10-30T12:28:44.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.508+0000 7f654d0dc640 1 -- 172.21.15.89:0/3003523454 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65480b2290 con 0x7f654815cd50 2023-10-30T12:28:44.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.508+0000 7f6544ff9640 1 -- 172.21.15.89:0/3003523454 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12850 (secure 0 0 0) 0x7f65400f2a90 con 0x7f654815cd50 2023-10-30T12:28:44.515 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:28:44.516 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":18,"fsid":"401641be-db6e-4c16-b3ac-fd84a86c4de4","created":"2023-10-30T12:28:26.198532+0000","modified":"2023-10-30T12:28:41.916008+0000","last_up_change":"2023-10-30T12:28:37.212205+0000","last_in_change":"2023-10-30T12:28:28.517734+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:39.004943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72595c54-fd6e-4a5a-bb14-ff27e9377b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6801","nonce":1673955622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6803","nonce":1673955622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6807","nonce":1673955622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6805","nonce":1673955622}]},"public_addr":"172.21.15.89:6801/1673955622","cluster_addr":"172.21.15.89:6803/1673955622","heartbeat_back_addr":"172.21.15.89:6807/1673955622","heartbeat_front_addr":"172.21.15.89:6805/1673955622","state":["exists","up"]},{"osd":1,"uuid":"0305670b-d7c4-4193-bec2-8b7351ff590a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6817","nonce":3340572039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6819","nonce":3340572039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6823","nonce":3340572039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6821","nonce":3340572039}]},"public_addr":"172.21.15.89:6817/3340572039","cluster_addr":"172.21.15.89:6819/3340572039","heartbeat_back_addr":"172.21.15.89:6823/3340572039","heartbeat_front_addr":"172.21.15.89:6821/3340572039","state":["exists","up"]},{"osd":2,"uuid":"4bab5426-a3c3-46b0-91fc-546c254006c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6809","nonce":1929940799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6811","nonce":1929940799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6815","nonce":1929940799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6813","nonce":1929940799}]},"public_addr":"172.21.15.89:6809/1929940799","cluster_addr":"172.21.15.89:6811/1929940799","heartbeat_back_addr":"172.21.15.89:6815/1929940799","heartbeat_front_addr":"172.21.15.89:6813/1929940799","state":["exists","up"]},{"osd":3,"uuid":"7d47aea8-27cd-42be-8591-d63ad68c90e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6825","nonce":345087939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6827","nonce":345087939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6831","nonce":345087939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6829","nonce":345087939}]},"public_addr":"172.21.15.89:6825/345087939","cluster_addr":"172.21.15.89:6827/345087939","heartbeat_back_addr":"172.21.15.89:6831/345087939","heartbeat_front_addr":"172.21.15.89:6829/345087939","state":["exists","up"]},{"osd":4,"uuid":"2e2ae8f8-d097-4bba-9448-95047fbfacad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6801","nonce":2138125545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6803","nonce":2138125545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6809","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6811","nonce":2138125545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6805","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6807","nonce":2138125545}]},"public_addr":"172.21.15.192:6801/2138125545","cluster_addr":"172.21.15.192:6803/2138125545","heartbeat_back_addr":"172.21.15.192:6811/2138125545","heartbeat_front_addr":"172.21.15.192:6807/2138125545","state":["exists","up"]},{"osd":5,"uuid":"49be5ce5-12f4-4c26-8d39-864b6ff67313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6806","nonce":2360218671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6810","nonce":2360218671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6815","nonce":2360218671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6813","nonce":2360218671}]},"public_addr":"172.21.15.192:6806/2360218671","cluster_addr":"172.21.15.192:6810/2360218671","heartbeat_back_addr":"172.21.15.192:6815/2360218671","heartbeat_front_addr":"172.21.15.192:6813/2360218671","state":["exists","up"]},{"osd":6,"uuid":"48623452-d456-4ea2-a0de-5fa4b37e6d10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6825","nonce":3707671287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6827","nonce":3707671287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6831","nonce":3707671287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6829","nonce":3707671287}]},"public_addr":"172.21.15.192:6825/3707671287","cluster_addr":"172.21.15.192:6827/3707671287","heartbeat_back_addr":"172.21.15.192:6831/3707671287","heartbeat_front_addr":"172.21.15.192:6829/3707671287","state":["exists","up"]},{"osd":7,"uuid":"ff0a4384-54f8-43ec-a3ce-9c8ae3bf8a94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6817","nonce":3111945871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6819","nonce":3111945871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6823","nonce":3111945871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6821","nonce":3111945871}]},"public_addr":"172.21.15.192:6817/3111945871","cluster_addr":"172.21.15.192:6819/3111945871","heartbeat_back_addr":"172.21.15.192:6823/3111945871","heartbeat_front_addr":"172.21.15.192:6821/3111945871","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.183065+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.248650+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.194576+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.397483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.043648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.186624+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.159674+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.100851+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:28:44.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.512+0000 7f654d0dc640 1 -- 172.21.15.89:0/3003523454 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f65340664d0 msgr2=0x7f6534068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:44.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.512+0000 7f654d0dc640 1 --2- 172.21.15.89:0/3003523454 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f65340664d0 0x7f6534068990 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6538007fd0 tx=0x7f6538009040 comp rx=0 tx=0).stop 2023-10-30T12:28:44.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.512+0000 7f654d0dc640 1 -- 172.21.15.89:0/3003523454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f654815cd50 msgr2=0x7f6548145320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:44.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.512+0000 7f654d0dc640 1 --2- 172.21.15.89:0/3003523454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f654815cd50 0x7f6548145320 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f65400b2040 tx=0x7f654001bf60 comp rx=0 tx=0).stop 2023-10-30T12:28:44.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.512+0000 7f654d0dc640 1 -- 172.21.15.89:0/3003523454 shutdown_connections 2023-10-30T12:28:44.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.512+0000 7f654d0dc640 1 --2- 172.21.15.89:0/3003523454 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f65340664d0 0x7f6534068990 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.512+0000 7f654d0dc640 1 --2- 172.21.15.89:0/3003523454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65481538f0 0x7f6548144de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.512+0000 7f654d0dc640 1 --2- 172.21.15.89:0/3003523454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f654815cd50 0x7f6548145320 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.512+0000 7f654d0dc640 1 --2- 172.21.15.89:0/3003523454 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65480cefa0 0x7f65481573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.512+0000 7f654d0dc640 1 -- 172.21.15.89:0/3003523454 >> 172.21.15.89:0/3003523454 conn(0x7f654806a640 msgr2=0x7f65480c48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:44.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.512+0000 7f654d0dc640 1 -- 172.21.15.89:0/3003523454 shutdown_connections 2023-10-30T12:28:44.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.512+0000 7f654d0dc640 1 -- 172.21.15.89:0/3003523454 wait complete. 2023-10-30T12:28:44.536 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T12:28:39.004943+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T12:28:44.537 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T12:28:44.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.672+0000 7f4e9b468640 1 Processor -- start 2023-10-30T12:28:44.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.672+0000 7f4e9b468640 1 -- start start 2023-10-30T12:28:44.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.672+0000 7f4e9b468640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e940722a0 0x7f4e940bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:44.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.672+0000 7f4e9b468640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e940c0a30 con 0x7f4e941508c0 2023-10-30T12:28:44.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.672+0000 7f4e9b468640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e9406f650 con 0x7f4e940c0400 2023-10-30T12:28:44.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.672+0000 7f4e9b468640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e940c85b0 con 0x7f4e940722a0 2023-10-30T12:28:44.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.672+0000 7f4e99204640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f4e941508c0 0x7f4e940c6920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42560/0 (socket says 172.21.15.89:42560) 2023-10-30T12:28:44.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.672+0000 7f4e99204640 1 -- 172.21.15.89:0/475739970 learned_addr learned my addr 172.21.15.89:0/475739970 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:28:44.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.672+0000 7f4e98a03640 1 --2- 172.21.15.89:0/475739970 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e940722a0 0x7f4e940bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:44.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.672+0000 7f4e8bfff640 1 -- 172.21.15.89:0/475739970 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2684269160 0 0) 0x7f4e940c0a30 con 0x7f4e941508c0 2023-10-30T12:28:44.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.676+0000 7f4e8bfff640 1 -- 172.21.15.89:0/475739970 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e80003530 con 0x7f4e941508c0 2023-10-30T12:28:44.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.676+0000 7f4e8bfff640 1 -- 172.21.15.89:0/475739970 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3164919258 0 0) 0x7f4e80003530 con 0x7f4e941508c0 2023-10-30T12:28:44.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.676+0000 7f4e8bfff640 1 -- 172.21.15.89:0/475739970 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e940722a0 msgr2=0x7f4e940bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:44.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.676+0000 7f4e8bfff640 1 --2- 172.21.15.89:0/475739970 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e940722a0 0x7f4e940bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.676+0000 7f4e8bfff640 1 -- 172.21.15.89:0/475739970 >> v1:172.21.15.192:6789/0 conn(0x7f4e940c0400 legacy=0x7f4e94156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:44.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.676+0000 7f4e8bfff640 1 -- 172.21.15.89:0/475739970 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9406f930 con 0x7f4e941508c0 2023-10-30T12:28:44.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.676+0000 7f4e8bfff640 1 -- 172.21.15.89:0/475739970 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f4e900050a0 con 0x7f4e941508c0 2023-10-30T12:28:44.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.676+0000 7f4e8bfff640 1 -- 172.21.15.89:0/475739970 >> v1:172.21.15.89:6789/0 conn(0x7f4e941508c0 legacy=0x7f4e940c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:44.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.676+0000 7f4e8bfff640 1 --2- 172.21.15.89:0/475739970 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e940722a0 0x7f4e800039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:44.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.676+0000 7f4e8bfff640 1 --2- 172.21.15.89:0/475739970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e80003ef0 0x7f4e800047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:44.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.676+0000 7f4e8bfff640 1 --2- 172.21.15.89:0/475739970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e80004d10 0x7f4e80005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:44.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.676+0000 7f4e8bfff640 1 -- 172.21.15.89:0/475739970 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e940c85b0 con 0x7f4e80004d10 2023-10-30T12:28:44.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.676+0000 7f4e8bfff640 1 -- 172.21.15.89:0/475739970 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8000baa0 con 0x7f4e80003ef0 2023-10-30T12:28:44.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.676+0000 7f4e8bfff640 1 -- 172.21.15.89:0/475739970 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8000de70 con 0x7f4e940722a0 2023-10-30T12:28:44.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.676+0000 7f4e99a05640 1 --2- 172.21.15.89:0/475739970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e80004d10 0x7f4e80005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:44.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.676+0000 7f4e98a03640 1 --2- 172.21.15.89:0/475739970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e80003ef0 0x7f4e800047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:44.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.676+0000 7f4e99204640 1 --2- 172.21.15.89:0/475739970 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e940722a0 0x7f4e800039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:44.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.676+0000 7f4e99a05640 1 -- 172.21.15.89:0/475739970 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e940722a0 msgr2=0x7f4e800039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:44.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.676+0000 7f4e99a05640 1 --2- 172.21.15.89:0/475739970 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e940722a0 0x7f4e800039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.676+0000 7f4e99a05640 1 -- 172.21.15.89:0/475739970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e80003ef0 msgr2=0x7f4e800047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:44.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.676+0000 7f4e99a05640 1 --2- 172.21.15.89:0/475739970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e80003ef0 0x7f4e800047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.676+0000 7f4e99a05640 1 -- 172.21.15.89:0/475739970 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e80003530 con 0x7f4e80004d10 2023-10-30T12:28:44.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.676+0000 7f4e99a05640 1 --2- 172.21.15.89:0/475739970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e80004d10 0x7f4e80005190 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c001f60 tx=0x7f4e8c01efd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8d44e00914b986c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:44.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.680+0000 7f4e8bfff640 1 -- 172.21.15.89:0/475739970 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c0a8bf0 con 0x7f4e80004d10 2023-10-30T12:28:44.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.680+0000 7f4e8bfff640 1 -- 172.21.15.89:0/475739970 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e8c0a8d50 con 0x7f4e80004d10 2023-10-30T12:28:44.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.680+0000 7f4e8bfff640 1 -- 172.21.15.89:0/475739970 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c0b0680 con 0x7f4e80004d10 2023-10-30T12:28:44.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.680+0000 7f4e9b468640 1 -- 172.21.15.89:0/475739970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e80004d10 msgr2=0x7f4e80005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:44.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.680+0000 7f4e9b468640 1 --2- 172.21.15.89:0/475739970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e80004d10 0x7f4e80005190 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c001f60 tx=0x7f4e8c01efd0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.680+0000 7f4e99a05640 1 -- 172.21.15.89:0/475739970 reap_dead start 2023-10-30T12:28:44.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.680+0000 7f4e9b468640 1 -- 172.21.15.89:0/475739970 shutdown_connections 2023-10-30T12:28:44.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.680+0000 7f4e9b468640 1 -- 172.21.15.89:0/475739970 >> 172.21.15.89:0/475739970 conn(0x7f4e940c7910 msgr2=0x7f4e940c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:44.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.680+0000 7f4e9b468640 1 -- 172.21.15.89:0/475739970 shutdown_connections 2023-10-30T12:28:44.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.680+0000 7f4e9b468640 1 -- 172.21.15.89:0/475739970 wait complete. 2023-10-30T12:28:44.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.680+0000 7f4e9b468640 1 Processor -- start 2023-10-30T12:28:44.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.680+0000 7f4e9b468640 1 -- start start 2023-10-30T12:28:44.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.680+0000 7f4e9b468640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e80004d10 0x7f4e940c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:44.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.680+0000 7f4e9b468640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e941435b0 0x7f4e94071660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:44.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.680+0000 7f4e9b468640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e941f2430 0x7f4e941f2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:44.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.680+0000 7f4e9b468640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e940cac50 con 0x7f4e941f2430 2023-10-30T12:28:44.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.680+0000 7f4e9b468640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e940ca280 con 0x7f4e941435b0 2023-10-30T12:28:44.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.680+0000 7f4e9b468640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e94143120 con 0x7f4e80004d10 2023-10-30T12:28:44.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.680+0000 7f4e99a05640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e941f2430 0x7f4e941f2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:44.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.684+0000 7f4e99a05640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e941f2430 0x7f4e941f2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35840/0 (socket says 172.21.15.89:35840) 2023-10-30T12:28:44.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.684+0000 7f4e99a05640 1 -- 172.21.15.89:0/3869110117 learned_addr learned my addr 172.21.15.89:0/3869110117 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:44.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.684+0000 7f4e99204640 1 --2- 172.21.15.89:0/3869110117 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e80004d10 0x7f4e940c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:44.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.684+0000 7f4e98a03640 1 --2- 172.21.15.89:0/3869110117 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e941435b0 0x7f4e94071660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:44.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.684+0000 7f4e99a05640 1 -- 172.21.15.89:0/3869110117 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e80004d10 msgr2=0x7f4e940c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:44.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.684+0000 7f4e99a05640 1 --2- 172.21.15.89:0/3869110117 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e80004d10 0x7f4e940c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.684+0000 7f4e99a05640 1 -- 172.21.15.89:0/3869110117 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e941435b0 msgr2=0x7f4e94071660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:44.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.684+0000 7f4e99a05640 1 --2- 172.21.15.89:0/3869110117 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e941435b0 0x7f4e94071660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.684+0000 7f4e99a05640 1 -- 172.21.15.89:0/3869110117 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e941dd5e0 con 0x7f4e941f2430 2023-10-30T12:28:44.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.684+0000 7f4e98a03640 1 --2- 172.21.15.89:0/3869110117 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e941435b0 0x7f4e94071660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:44.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.684+0000 7f4e99a05640 1 --2- 172.21.15.89:0/3869110117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e941f2430 0x7f4e941f2890 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c002050 tx=0x7f4e80005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:44.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.684+0000 7f4e89ffb640 1 -- 172.21.15.89:0/3869110117 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c0a8bf0 con 0x7f4e941f2430 2023-10-30T12:28:44.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.684+0000 7f4e89ffb640 1 -- 172.21.15.89:0/3869110117 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e8c003980 con 0x7f4e941f2430 2023-10-30T12:28:44.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.684+0000 7f4e9b468640 1 -- 172.21.15.89:0/3869110117 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e940ca690 con 0x7f4e941f2430 2023-10-30T12:28:44.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.684+0000 7f4e89ffb640 1 -- 172.21.15.89:0/3869110117 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c003cf0 con 0x7f4e941f2430 2023-10-30T12:28:44.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.684+0000 7f4e9b468640 1 -- 172.21.15.89:0/3869110117 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e940c9dd0 con 0x7f4e941f2430 2023-10-30T12:28:44.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.684+0000 7f4e9b468640 1 -- 172.21.15.89:0/3869110117 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e940cabb0 con 0x7f4e941f2430 2023-10-30T12:28:44.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.684+0000 7f4e89ffb640 1 -- 172.21.15.89:0/3869110117 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4e8c0b7040 con 0x7f4e941f2430 2023-10-30T12:28:44.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.684+0000 7f4e89ffb640 1 --2- 172.21.15.89:0/3869110117 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4e780664a0 0x7f4e78068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:44.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.688+0000 7f4e89ffb640 1 -- 172.21.15.89:0/3869110117 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4e8c1202a0 con 0x7f4e941f2430 2023-10-30T12:28:44.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.688+0000 7f4e99204640 1 --2- 172.21.15.89:0/3869110117 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4e780664a0 0x7f4e78068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:44.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.688+0000 7f4e99204640 1 --2- 172.21.15.89:0/3869110117 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4e780664a0 0x7f4e78068960 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4e90004b10 tx=0x7f4e90009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:44.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.692+0000 7f4e89ffb640 1 -- 172.21.15.89:0/3869110117 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4e8c0edc20 con 0x7f4e941f2430 2023-10-30T12:28:44.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.840+0000 7f4e9b468640 1 -- 172.21.15.89:0/3869110117 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4e9406fbe0 con 0x7f4e941f2430 2023-10-30T12:28:44.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.840+0000 7f4e89ffb640 1 -- 172.21.15.89:0/3869110117 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4e8c0f1240 con 0x7f4e941f2430 2023-10-30T12:28:44.847 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-10-30T12:28:44.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.844+0000 7f4e9b468640 1 -- 172.21.15.89:0/3869110117 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4e780664a0 msgr2=0x7f4e78068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:44.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.844+0000 7f4e9b468640 1 --2- 172.21.15.89:0/3869110117 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4e780664a0 0x7f4e78068960 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4e90004b10 tx=0x7f4e90009040 comp rx=0 tx=0).stop 2023-10-30T12:28:44.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.844+0000 7f4e9b468640 1 -- 172.21.15.89:0/3869110117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e941f2430 msgr2=0x7f4e941f2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:44.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.844+0000 7f4e9b468640 1 --2- 172.21.15.89:0/3869110117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e941f2430 0x7f4e941f2890 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c002050 tx=0x7f4e80005800 comp rx=0 tx=0).stop 2023-10-30T12:28:44.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.844+0000 7f4e9b468640 1 -- 172.21.15.89:0/3869110117 shutdown_connections 2023-10-30T12:28:44.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.844+0000 7f4e9b468640 1 --2- 172.21.15.89:0/3869110117 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4e780664a0 0x7f4e78068960 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.844+0000 7f4e9b468640 1 --2- 172.21.15.89:0/3869110117 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e941435b0 0x7f4e94071660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.844+0000 7f4e9b468640 1 --2- 172.21.15.89:0/3869110117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e941f2430 0x7f4e941f2890 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.844+0000 7f4e9b468640 1 --2- 172.21.15.89:0/3869110117 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e80004d10 0x7f4e940c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.844+0000 7f4e9b468640 1 -- 172.21.15.89:0/3869110117 >> 172.21.15.89:0/3869110117 conn(0x7f4e940c7910 msgr2=0x7f4e941e4760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:44.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.844+0000 7f4e9b468640 1 -- 172.21.15.89:0/3869110117 shutdown_connections 2023-10-30T12:28:44.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:44.844+0000 7f4e9b468640 1 -- 172.21.15.89:0/3869110117 wait complete. 2023-10-30T12:28:44.867 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:28:45.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.000+0000 7fb4facc9640 1 Processor -- start 2023-10-30T12:28:45.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.000+0000 7fb4facc9640 1 -- start start 2023-10-30T12:28:45.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.004+0000 7fb4facc9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4f40c6540 0x7fb4f415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:45.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.004+0000 7fb4facc9640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f40c69d0 con 0x7fb4f41580e0 2023-10-30T12:28:45.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.004+0000 7fb4facc9640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f40c5810 con 0x7fb4f4070740 2023-10-30T12:28:45.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.004+0000 7fb4facc9640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f406acb0 con 0x7fb4f40c6540 2023-10-30T12:28:45.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.004+0000 7fb4f8a65640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb4f41580e0 0x7fb4f4069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42564/0 (socket says 172.21.15.89:42564) 2023-10-30T12:28:45.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.004+0000 7fb4f8a65640 1 -- 172.21.15.89:0/687163738 learned_addr learned my addr 172.21.15.89:0/687163738 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:28:45.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.004+0000 7fb4f9266640 1 --2- 172.21.15.89:0/687163738 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4f40c6540 0x7fb4f415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:45.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.004+0000 7fb4f37fe640 1 -- 172.21.15.89:0/687163738 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2413684839 0 0) 0x7fb4f40c69d0 con 0x7fb4f41580e0 2023-10-30T12:28:45.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.004+0000 7fb4f37fe640 1 -- 172.21.15.89:0/687163738 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4e4003530 con 0x7fb4f41580e0 2023-10-30T12:28:45.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.004+0000 7fb4f37fe640 1 -- 172.21.15.89:0/687163738 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2697983400 0 0) 0x7fb4e4003530 con 0x7fb4f41580e0 2023-10-30T12:28:45.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.004+0000 7fb4f37fe640 1 -- 172.21.15.89:0/687163738 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4f40c6540 msgr2=0x7fb4f415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:45.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.004+0000 7fb4f37fe640 1 --2- 172.21.15.89:0/687163738 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4f40c6540 0x7fb4f415e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.004+0000 7fb4f37fe640 1 -- 172.21.15.89:0/687163738 >> v1:172.21.15.192:6789/0 conn(0x7fb4f4070740 legacy=0x7fb4f406a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:45.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.004+0000 7fb4f37fe640 1 -- 172.21.15.89:0/687163738 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f40c6bb0 con 0x7fb4f41580e0 2023-10-30T12:28:45.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.004+0000 7fb4f37fe640 1 -- 172.21.15.89:0/687163738 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7fb4e8001680 con 0x7fb4f41580e0 2023-10-30T12:28:45.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.004+0000 7fb4f37fe640 1 -- 172.21.15.89:0/687163738 >> v1:172.21.15.89:6789/0 conn(0x7fb4f41580e0 legacy=0x7fb4f4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:45.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.004+0000 7fb4f37fe640 1 --2- 172.21.15.89:0/687163738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4e40039b0 0x7fb4e4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:45.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.004+0000 7fb4f37fe640 1 --2- 172.21.15.89:0/687163738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4e4004360 0x7fb4e4004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:45.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.004+0000 7fb4f37fe640 1 --2- 172.21.15.89:0/687163738 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4f40c6540 0x7fb4e4005090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:45.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.004+0000 7fb4f37fe640 1 -- 172.21.15.89:0/687163738 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e400b990 con 0x7fb4e4004360 2023-10-30T12:28:45.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.004+0000 7fb4f37fe640 1 -- 172.21.15.89:0/687163738 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e400bad0 con 0x7fb4e40039b0 2023-10-30T12:28:45.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.004+0000 7fb4f37fe640 1 -- 172.21.15.89:0/687163738 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e400bc90 con 0x7fb4f40c6540 2023-10-30T12:28:45.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.004+0000 7fb4f3fff640 1 --2- 172.21.15.89:0/687163738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4e4004360 0x7fb4e4004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:45.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.004+0000 7fb4f8a65640 1 --2- 172.21.15.89:0/687163738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4e40039b0 0x7fb4e4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:45.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.004+0000 7fb4f9266640 1 --2- 172.21.15.89:0/687163738 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4f40c6540 0x7fb4e4005090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:45.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.004+0000 7fb4f3fff640 1 -- 172.21.15.89:0/687163738 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4f40c6540 msgr2=0x7fb4e4005090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:45.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.004+0000 7fb4f3fff640 1 --2- 172.21.15.89:0/687163738 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4f40c6540 0x7fb4e4005090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.004+0000 7fb4f3fff640 1 -- 172.21.15.89:0/687163738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4e40039b0 msgr2=0x7fb4e4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:45.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.004+0000 7fb4f3fff640 1 --2- 172.21.15.89:0/687163738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4e40039b0 0x7fb4e4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.004+0000 7fb4f3fff640 1 -- 172.21.15.89:0/687163738 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e4003530 con 0x7fb4e4004360 2023-10-30T12:28:45.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.008+0000 7fb4f3fff640 1 --2- 172.21.15.89:0/687163738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4e4004360 0x7fb4e4004830 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0001860 tx=0x7fb4e001c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da3fcec63d44c93d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:45.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.008+0000 7fb4f37fe640 1 -- 172.21.15.89:0/687163738 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e00ab600 con 0x7fb4e4004360 2023-10-30T12:28:45.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.008+0000 7fb4f37fe640 1 -- 172.21.15.89:0/687163738 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e00a7050 con 0x7fb4e4004360 2023-10-30T12:28:45.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.008+0000 7fb4f37fe640 1 -- 172.21.15.89:0/687163738 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e00ab9a0 con 0x7fb4e4004360 2023-10-30T12:28:45.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.008+0000 7fb4facc9640 1 -- 172.21.15.89:0/687163738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4e4004360 msgr2=0x7fb4e4004830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:45.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.008+0000 7fb4facc9640 1 --2- 172.21.15.89:0/687163738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4e4004360 0x7fb4e4004830 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0001860 tx=0x7fb4e001c4e0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.008+0000 7fb4f9266640 1 -- 172.21.15.89:0/687163738 reap_dead start 2023-10-30T12:28:45.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.008+0000 7fb4facc9640 1 -- 172.21.15.89:0/687163738 shutdown_connections 2023-10-30T12:28:45.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.008+0000 7fb4facc9640 1 -- 172.21.15.89:0/687163738 >> 172.21.15.89:0/687163738 conn(0x7fb4f40c7910 msgr2=0x7fb4f40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:45.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.008+0000 7fb4facc9640 1 -- 172.21.15.89:0/687163738 shutdown_connections 2023-10-30T12:28:45.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.008+0000 7fb4facc9640 1 -- 172.21.15.89:0/687163738 wait complete. 2023-10-30T12:28:45.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.008+0000 7fb4facc9640 1 Processor -- start 2023-10-30T12:28:45.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.008+0000 7fb4facc9640 1 -- start start 2023-10-30T12:28:45.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.012+0000 7fb4facc9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4e4004360 0x7fb4f41f6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:45.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.012+0000 7fb4facc9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4f40c0ad0 0x7fb4f414c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:45.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.012+0000 7fb4facc9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4f414c770 0x7fb4f40c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:45.023 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.012+0000 7fb4facc9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f40c9260 con 0x7fb4e4004360 2023-10-30T12:28:45.023 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.012+0000 7fb4facc9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f40c16c0 con 0x7fb4f40c0ad0 2023-10-30T12:28:45.023 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.012+0000 7fb4facc9640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f40c19a0 con 0x7fb4f414c770 2023-10-30T12:28:45.023 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.012+0000 7fb4f8a65640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4e4004360 0x7fb4f41f6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:45.023 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.012+0000 7fb4f8a65640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4e4004360 0x7fb4f41f6e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35858/0 (socket says 172.21.15.89:35858) 2023-10-30T12:28:45.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.012+0000 7fb4f8a65640 1 -- 172.21.15.89:0/2830713231 learned_addr learned my addr 172.21.15.89:0/2830713231 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:45.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.012+0000 7fb4f3fff640 1 --2- 172.21.15.89:0/2830713231 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4f40c0ad0 0x7fb4f414c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:45.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.012+0000 7fb4f9266640 1 --2- 172.21.15.89:0/2830713231 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4f414c770 0x7fb4f40c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:45.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.012+0000 7fb4f8a65640 1 -- 172.21.15.89:0/2830713231 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4f414c770 msgr2=0x7fb4f40c96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:45.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.012+0000 7fb4f8a65640 1 --2- 172.21.15.89:0/2830713231 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4f414c770 0x7fb4f40c96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.012+0000 7fb4f8a65640 1 -- 172.21.15.89:0/2830713231 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4f40c0ad0 msgr2=0x7fb4f414c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:45.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.012+0000 7fb4f8a65640 1 --2- 172.21.15.89:0/2830713231 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4f40c0ad0 0x7fb4f414c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.012+0000 7fb4f8a65640 1 -- 172.21.15.89:0/2830713231 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f40c1c80 con 0x7fb4e4004360 2023-10-30T12:28:45.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.012+0000 7fb4f9266640 1 --2- 172.21.15.89:0/2830713231 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4f414c770 0x7fb4f40c96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:28:45.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.012+0000 7fb4f3fff640 1 --2- 172.21.15.89:0/2830713231 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4f40c0ad0 0x7fb4f414c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:28:45.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.012+0000 7fb4f8a65640 1 --2- 172.21.15.89:0/2830713231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4e4004360 0x7fb4f41f6e00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb4e800e710 tx=0x7fb4e800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:45.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.012+0000 7fb4f17fa640 1 -- 172.21.15.89:0/2830713231 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e800ee20 con 0x7fb4e4004360 2023-10-30T12:28:45.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.012+0000 7fb4f17fa640 1 -- 172.21.15.89:0/2830713231 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e800c8d0 con 0x7fb4e4004360 2023-10-30T12:28:45.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.012+0000 7fb4f17fa640 1 -- 172.21.15.89:0/2830713231 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e800cc80 con 0x7fb4e4004360 2023-10-30T12:28:45.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.012+0000 7fb4facc9640 1 -- 172.21.15.89:0/2830713231 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4f40c8c10 con 0x7fb4e4004360 2023-10-30T12:28:45.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.012+0000 7fb4facc9640 1 -- 172.21.15.89:0/2830713231 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4f414cd90 con 0x7fb4e4004360 2023-10-30T12:28:45.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.012+0000 7fb4facc9640 1 -- 172.21.15.89:0/2830713231 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4f414cb50 con 0x7fb4e4004360 2023-10-30T12:28:45.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.012+0000 7fb4f17fa640 1 -- 172.21.15.89:0/2830713231 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb4e801d460 con 0x7fb4e4004360 2023-10-30T12:28:45.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.016+0000 7fb4f17fa640 1 --2- 172.21.15.89:0/2830713231 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb4d80664a0 0x7fb4d8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:45.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.016+0000 7fb4f17fa640 1 -- 172.21.15.89:0/2830713231 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb4e8083c40 con 0x7fb4e4004360 2023-10-30T12:28:45.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.016+0000 7fb4f3fff640 1 --2- 172.21.15.89:0/2830713231 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb4d80664a0 0x7fb4d8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:45.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.016+0000 7fb4f3fff640 1 --2- 172.21.15.89:0/2830713231 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb4d80664a0 0x7fb4d8068960 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0001860 tx=0x7fb4e00a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:45.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.020+0000 7fb4f17fa640 1 -- 172.21.15.89:0/2830713231 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb4e80512e0 con 0x7fb4e4004360 2023-10-30T12:28:45.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.200+0000 7fb4facc9640 1 -- 172.21.15.89:0/2830713231 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4f404e170 con 0x7fb4e4004360 2023-10-30T12:28:45.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.200+0000 7fb4f17fa640 1 -- 172.21.15.89:0/2830713231 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1324 (secure 0 0 0) 0x7fb4e8054900 con 0x7fb4e4004360 2023-10-30T12:28:45.207 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:28:45.207 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2023-10-30T12:28:45.208 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 2 2023-10-30T12:28:45.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.204+0000 7fb4facc9640 1 -- 172.21.15.89:0/2830713231 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb4d80664a0 msgr2=0x7fb4d8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:45.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.204+0000 7fb4facc9640 1 --2- 172.21.15.89:0/2830713231 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb4d80664a0 0x7fb4d8068960 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0001860 tx=0x7fb4e00a6000 comp rx=0 tx=0).stop 2023-10-30T12:28:45.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.204+0000 7fb4facc9640 1 -- 172.21.15.89:0/2830713231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4e4004360 msgr2=0x7fb4f41f6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:45.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.204+0000 7fb4facc9640 1 --2- 172.21.15.89:0/2830713231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4e4004360 0x7fb4f41f6e00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb4e800e710 tx=0x7fb4e800ebd0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.204+0000 7fb4facc9640 1 -- 172.21.15.89:0/2830713231 shutdown_connections 2023-10-30T12:28:45.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.204+0000 7fb4facc9640 1 --2- 172.21.15.89:0/2830713231 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb4d80664a0 0x7fb4d8068960 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.204+0000 7fb4facc9640 1 --2- 172.21.15.89:0/2830713231 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4f40c0ad0 0x7fb4f414c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.204+0000 7fb4facc9640 1 --2- 172.21.15.89:0/2830713231 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb4f414c770 0x7fb4f40c96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.204+0000 7fb4facc9640 1 --2- 172.21.15.89:0/2830713231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4e4004360 0x7fb4f41f6e00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.204+0000 7fb4facc9640 1 -- 172.21.15.89:0/2830713231 >> 172.21.15.89:0/2830713231 conn(0x7fb4f40c7910 msgr2=0x7fb4f414e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:45.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.204+0000 7fb4facc9640 1 -- 172.21.15.89:0/2830713231 shutdown_connections 2023-10-30T12:28:45.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:45.204+0000 7fb4facc9640 1 -- 172.21.15.89:0/2830713231 wait complete. 2023-10-30T12:28:47.228 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:28:47.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.376+0000 7f473f59e640 1 Processor -- start 2023-10-30T12:28:47.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.376+0000 7f473f59e640 1 -- start start 2023-10-30T12:28:47.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.376+0000 7f473f59e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47401538f0 0x7f4740159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:47.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.376+0000 7f473f59e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f474015cd50 0x7f47400c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:47.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.376+0000 7f473f59e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f474015d300 con 0x7f47400cefa0 2023-10-30T12:28:47.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.376+0000 7f473f59e640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f474015d5e0 con 0x7f474015cd50 2023-10-30T12:28:47.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.376+0000 7f473f59e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f474015d8c0 con 0x7f47401538f0 2023-10-30T12:28:47.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.376+0000 7f4735d9b640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f474015cd50 0x7f47400c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:47.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.376+0000 7f4735d9b640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f474015cd50 0x7f47400c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35862/0 (socket says 172.21.15.89:35862) 2023-10-30T12:28:47.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.376+0000 7f4735d9b640 1 -- 172.21.15.89:0/1907721273 learned_addr learned my addr 172.21.15.89:0/1907721273 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:47.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.376+0000 7f473e59c640 1 --2- 172.21.15.89:0/1907721273 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47401538f0 0x7f4740159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:47.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.376+0000 7f4735d9b640 1 -- 172.21.15.89:0/1907721273 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47401538f0 msgr2=0x7f4740159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:47.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.376+0000 7f4735d9b640 1 --2- 172.21.15.89:0/1907721273 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47401538f0 0x7f4740159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.376+0000 7f4735d9b640 1 -- 172.21.15.89:0/1907721273 >> v1:172.21.15.192:6789/0 conn(0x7f47400cefa0 legacy=0x7f47400cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:47.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.376+0000 7f4735d9b640 1 -- 172.21.15.89:0/1907721273 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f474015dba0 con 0x7f474015cd50 2023-10-30T12:28:47.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.380+0000 7f4735d9b640 1 --2- 172.21.15.89:0/1907721273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f474015cd50 0x7f47400c9f90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4728009870 tx=0x7f472801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8607f79752022bac server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:47.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.380+0000 7f473dd9b640 1 -- 172.21.15.89:0/1907721273 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47280a8a30 con 0x7f474015cd50 2023-10-30T12:28:47.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.380+0000 7f473dd9b640 1 -- 172.21.15.89:0/1907721273 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47280a8b90 con 0x7f474015cd50 2023-10-30T12:28:47.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.380+0000 7f473dd9b640 1 -- 172.21.15.89:0/1907721273 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47280b0440 con 0x7f474015cd50 2023-10-30T12:28:47.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.380+0000 7f473f59e640 1 -- 172.21.15.89:0/1907721273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f474015cd50 msgr2=0x7f47400c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:47.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.380+0000 7f473f59e640 1 --2- 172.21.15.89:0/1907721273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f474015cd50 0x7f47400c9f90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4728009870 tx=0x7f472801ed30 comp rx=0 tx=0).stop 2023-10-30T12:28:47.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.380+0000 7f473f59e640 1 -- 172.21.15.89:0/1907721273 shutdown_connections 2023-10-30T12:28:47.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.380+0000 7f473f59e640 1 --2- 172.21.15.89:0/1907721273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f474015cd50 0x7f47400c9f90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.380+0000 7f473f59e640 1 --2- 172.21.15.89:0/1907721273 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47401538f0 0x7f4740159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.380+0000 7f473f59e640 1 -- 172.21.15.89:0/1907721273 >> 172.21.15.89:0/1907721273 conn(0x7f474006a640 msgr2=0x7f4740065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:47.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.380+0000 7f473f59e640 1 -- 172.21.15.89:0/1907721273 shutdown_connections 2023-10-30T12:28:47.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.380+0000 7f473f59e640 1 -- 172.21.15.89:0/1907721273 wait complete. 2023-10-30T12:28:47.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.384+0000 7f473f59e640 1 Processor -- start 2023-10-30T12:28:47.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.384+0000 7f473f59e640 1 -- start start 2023-10-30T12:28:47.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.384+0000 7f473f59e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47400cefa0 0x7f47401484d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:47.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.384+0000 7f473f59e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47401538f0 0x7f47400ce1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:47.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.384+0000 7f473f59e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f474015cd50 0x7f47401f2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:47.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.384+0000 7f473f59e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47401f26e0 con 0x7f47401538f0 2023-10-30T12:28:47.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.384+0000 7f473f59e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474014d750 con 0x7f47400cefa0 2023-10-30T12:28:47.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.384+0000 7f473f59e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4740147f50 con 0x7f474015cd50 2023-10-30T12:28:47.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.384+0000 7f4735d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47401538f0 0x7f47400ce1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:47.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.384+0000 7f4735d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47401538f0 0x7f47400ce1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35874/0 (socket says 172.21.15.89:35874) 2023-10-30T12:28:47.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.384+0000 7f4735d9b640 1 -- 172.21.15.89:0/1786331447 learned_addr learned my addr 172.21.15.89:0/1786331447 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:47.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.384+0000 7f473ed9d640 1 --2- 172.21.15.89:0/1786331447 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f474015cd50 0x7f47401f2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:47.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.384+0000 7f473e59c640 1 --2- 172.21.15.89:0/1786331447 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47400cefa0 0x7f47401484d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:47.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.384+0000 7f4735d9b640 1 -- 172.21.15.89:0/1786331447 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f474015cd50 msgr2=0x7f47401f2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:47.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.384+0000 7f4735d9b640 1 --2- 172.21.15.89:0/1786331447 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f474015cd50 0x7f47401f2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.384+0000 7f4735d9b640 1 -- 172.21.15.89:0/1786331447 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47400cefa0 msgr2=0x7f47401484d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:47.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.384+0000 7f4735d9b640 1 --2- 172.21.15.89:0/1786331447 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47400cefa0 0x7f47401484d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.384+0000 7f4735d9b640 1 -- 172.21.15.89:0/1786331447 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47280094d0 con 0x7f47401538f0 2023-10-30T12:28:47.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.384+0000 7f4735d9b640 1 --2- 172.21.15.89:0/1786331447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47401538f0 0x7f47400ce1c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f47280b3040 tx=0x7f472801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:47.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.384+0000 7f47377fe640 1 -- 172.21.15.89:0/1786331447 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47280b9620 con 0x7f47401538f0 2023-10-30T12:28:47.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.384+0000 7f47377fe640 1 -- 172.21.15.89:0/1786331447 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47280b97a0 con 0x7f47401538f0 2023-10-30T12:28:47.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.384+0000 7f47377fe640 1 -- 172.21.15.89:0/1786331447 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47280b9b50 con 0x7f47401538f0 2023-10-30T12:28:47.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.384+0000 7f473f59e640 1 -- 172.21.15.89:0/1786331447 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47401f23b0 con 0x7f47401538f0 2023-10-30T12:28:47.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.384+0000 7f473f59e640 1 -- 172.21.15.89:0/1786331447 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47401481f0 con 0x7f47401538f0 2023-10-30T12:28:47.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.384+0000 7f473f59e640 1 -- 172.21.15.89:0/1786331447 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f474004e170 con 0x7f47401538f0 2023-10-30T12:28:47.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.388+0000 7f47377fe640 1 -- 172.21.15.89:0/1786331447 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f47280b7070 con 0x7f47401538f0 2023-10-30T12:28:47.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.388+0000 7f47377fe640 1 --2- 172.21.15.89:0/1786331447 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f47200664a0 0x7f4720068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:47.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.388+0000 7f47377fe640 1 -- 172.21.15.89:0/1786331447 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4728121a60 con 0x7f47401538f0 2023-10-30T12:28:47.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.388+0000 7f473e59c640 1 --2- 172.21.15.89:0/1786331447 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f47200664a0 0x7f4720068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:47.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.388+0000 7f473e59c640 1 --2- 172.21.15.89:0/1786331447 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f47200664a0 0x7f4720068960 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4730003fd0 tx=0x7f47300039a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:47.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.392+0000 7f47377fe640 1 -- 172.21.15.89:0/1786331447 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f47280ef100 con 0x7f47401538f0 2023-10-30T12:28:47.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.572+0000 7f473f59e640 1 -- 172.21.15.89:0/1786331447 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47400b2290 con 0x7f47401538f0 2023-10-30T12:28:47.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.572+0000 7f47377fe640 1 -- 172.21.15.89:0/1786331447 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+5283 (secure 0 0 0) 0x7f47280b5070 con 0x7f47401538f0 2023-10-30T12:28:47.580 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:28:47.580 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1289220491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1289220491},{"type":"v1","addr":"172.21.15.192:6835","nonce":1289220491}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4318,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2023-10-30T12:28:47.580 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 3 2023-10-30T12:28:47.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.576+0000 7f473f59e640 1 -- 172.21.15.89:0/1786331447 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f47200664a0 msgr2=0x7f4720068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:47.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.576+0000 7f473f59e640 1 --2- 172.21.15.89:0/1786331447 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f47200664a0 0x7f4720068960 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4730003fd0 tx=0x7f47300039a0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.576+0000 7f473f59e640 1 -- 172.21.15.89:0/1786331447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47401538f0 msgr2=0x7f47400ce1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:47.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.576+0000 7f473f59e640 1 --2- 172.21.15.89:0/1786331447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47401538f0 0x7f47400ce1c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f47280b3040 tx=0x7f472801bad0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.576+0000 7f473f59e640 1 -- 172.21.15.89:0/1786331447 shutdown_connections 2023-10-30T12:28:47.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.576+0000 7f473f59e640 1 --2- 172.21.15.89:0/1786331447 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f47200664a0 0x7f4720068960 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.576+0000 7f473f59e640 1 --2- 172.21.15.89:0/1786331447 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f474015cd50 0x7f47401f2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.576+0000 7f473f59e640 1 --2- 172.21.15.89:0/1786331447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47401538f0 0x7f47400ce1c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.576+0000 7f473f59e640 1 --2- 172.21.15.89:0/1786331447 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47400cefa0 0x7f47401484d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.576+0000 7f473f59e640 1 -- 172.21.15.89:0/1786331447 >> 172.21.15.89:0/1786331447 conn(0x7f474006a640 msgr2=0x7f47400c4de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:47.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.576+0000 7f473f59e640 1 -- 172.21.15.89:0/1786331447 shutdown_connections 2023-10-30T12:28:47.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.576+0000 7f473f59e640 1 -- 172.21.15.89:0/1786331447 wait complete. 2023-10-30T12:28:47.600 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:28:47.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.732+0000 7fcd36a32640 1 Processor -- start 2023-10-30T12:28:47.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.732+0000 7fcd36a32640 1 -- start start 2023-10-30T12:28:47.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.736+0000 7fcd36a32640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcd300c6bd0 0x7fcd3006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:47.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.736+0000 7fcd36a32640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcd30071110 0x7fcd30158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:47.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.736+0000 7fcd36a32640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd30066cf0 con 0x7fcd3006f7a0 2023-10-30T12:28:47.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.736+0000 7fcd36a32640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd300700c0 con 0x7fcd300c6bd0 2023-10-30T12:28:47.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.736+0000 7fcd36a32640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd300c70d0 con 0x7fcd30071110 2023-10-30T12:28:47.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.736+0000 7fcd2f7fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcd300c6bd0 0x7fcd3006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:47.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.736+0000 7fcd2f7fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcd300c6bd0 0x7fcd3006fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35876/0 (socket says 172.21.15.89:35876) 2023-10-30T12:28:47.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.736+0000 7fcd2f7fe640 1 -- 172.21.15.89:0/1876904749 learned_addr learned my addr 172.21.15.89:0/1876904749 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:47.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.736+0000 7fcd34fcf640 1 --2- 172.21.15.89:0/1876904749 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcd30071110 0x7fcd30158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:47.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.736+0000 7fcd2f7fe640 1 -- 172.21.15.89:0/1876904749 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcd30071110 msgr2=0x7fcd30158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:47.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.736+0000 7fcd2f7fe640 1 --2- 172.21.15.89:0/1876904749 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcd30071110 0x7fcd30158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.736+0000 7fcd2f7fe640 1 -- 172.21.15.89:0/1876904749 >> v1:172.21.15.192:6789/0 conn(0x7fcd3006f7a0 legacy=0x7fcd30069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:47.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.736+0000 7fcd2f7fe640 1 -- 172.21.15.89:0/1876904749 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd300c7390 con 0x7fcd300c6bd0 2023-10-30T12:28:47.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.740+0000 7fcd2f7fe640 1 --2- 172.21.15.89:0/1876904749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd300c6bd0 0x7fcd3006fb80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcd24009870 tx=0x7fcd2401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efc5a24bf131960d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:47.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.740+0000 7fcd2effd640 1 -- 172.21.15.89:0/1876904749 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd240a8a30 con 0x7fcd300c6bd0 2023-10-30T12:28:47.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.740+0000 7fcd2effd640 1 -- 172.21.15.89:0/1876904749 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd240a8b90 con 0x7fcd300c6bd0 2023-10-30T12:28:47.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.740+0000 7fcd2effd640 1 -- 172.21.15.89:0/1876904749 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd240b0440 con 0x7fcd300c6bd0 2023-10-30T12:28:47.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.740+0000 7fcd36a32640 1 -- 172.21.15.89:0/1876904749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd300c6bd0 msgr2=0x7fcd3006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:47.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.740+0000 7fcd36a32640 1 --2- 172.21.15.89:0/1876904749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd300c6bd0 0x7fcd3006fb80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcd24009870 tx=0x7fcd2401ed30 comp rx=0 tx=0).stop 2023-10-30T12:28:47.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.740+0000 7fcd36a32640 1 -- 172.21.15.89:0/1876904749 shutdown_connections 2023-10-30T12:28:47.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.740+0000 7fcd36a32640 1 --2- 172.21.15.89:0/1876904749 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcd30071110 0x7fcd30158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.740+0000 7fcd36a32640 1 --2- 172.21.15.89:0/1876904749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd300c6bd0 0x7fcd3006fb80 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.740+0000 7fcd36a32640 1 -- 172.21.15.89:0/1876904749 >> 172.21.15.89:0/1876904749 conn(0x7fcd300c7910 msgr2=0x7fcd300c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:47.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.740+0000 7fcd36a32640 1 -- 172.21.15.89:0/1876904749 shutdown_connections 2023-10-30T12:28:47.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.740+0000 7fcd36a32640 1 -- 172.21.15.89:0/1876904749 wait complete. 2023-10-30T12:28:47.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.740+0000 7fcd36a32640 1 Processor -- start 2023-10-30T12:28:47.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.740+0000 7fcd36a32640 1 -- start start 2023-10-30T12:28:47.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.740+0000 7fcd36a32640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcd3006f7a0 0x7fcd301ee7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:47.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.740+0000 7fcd36a32640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcd30071110 0x7fcd301f2790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:47.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.740+0000 7fcd36a32640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd300c6bd0 0x7fcd3014feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:47.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.740+0000 7fcd36a32640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd301f2370 con 0x7fcd300c6bd0 2023-10-30T12:28:47.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.740+0000 7fcd36a32640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd301f1d40 con 0x7fcd30071110 2023-10-30T12:28:47.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.740+0000 7fcd36a32640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd30155f30 con 0x7fcd3006f7a0 2023-10-30T12:28:47.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.744+0000 7fcd34fcf640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd300c6bd0 0x7fcd3014feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:47.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.744+0000 7fcd34fcf640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd300c6bd0 0x7fcd3014feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35886/0 (socket says 172.21.15.89:35886) 2023-10-30T12:28:47.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.744+0000 7fcd34fcf640 1 -- 172.21.15.89:0/2332859428 learned_addr learned my addr 172.21.15.89:0/2332859428 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:47.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.744+0000 7fcd2ffff640 1 --2- 172.21.15.89:0/2332859428 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcd3006f7a0 0x7fcd301ee7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:47.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.744+0000 7fcd2f7fe640 1 --2- 172.21.15.89:0/2332859428 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcd30071110 0x7fcd301f2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:47.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.744+0000 7fcd34fcf640 1 -- 172.21.15.89:0/2332859428 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcd3006f7a0 msgr2=0x7fcd301ee7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:47.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.744+0000 7fcd34fcf640 1 --2- 172.21.15.89:0/2332859428 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcd3006f7a0 0x7fcd301ee7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.744+0000 7fcd34fcf640 1 -- 172.21.15.89:0/2332859428 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcd30071110 msgr2=0x7fcd301f2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:47.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.744+0000 7fcd34fcf640 1 --2- 172.21.15.89:0/2332859428 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcd30071110 0x7fcd301f2790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.744+0000 7fcd34fcf640 1 -- 172.21.15.89:0/2332859428 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd240094d0 con 0x7fcd300c6bd0 2023-10-30T12:28:47.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.744+0000 7fcd34fcf640 1 --2- 172.21.15.89:0/2332859428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd300c6bd0 0x7fcd3014feb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcd28007e40 tx=0x7fcd2800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:47.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.744+0000 7fcd2cff9640 1 -- 172.21.15.89:0/2332859428 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd28014620 con 0x7fcd300c6bd0 2023-10-30T12:28:47.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.744+0000 7fcd2cff9640 1 -- 172.21.15.89:0/2332859428 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2800ee90 con 0x7fcd300c6bd0 2023-10-30T12:28:47.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.744+0000 7fcd2cff9640 1 -- 172.21.15.89:0/2332859428 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd280149e0 con 0x7fcd300c6bd0 2023-10-30T12:28:47.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.744+0000 7fcd36a32640 1 -- 172.21.15.89:0/2332859428 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd30156210 con 0x7fcd300c6bd0 2023-10-30T12:28:47.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.744+0000 7fcd36a32640 1 -- 172.21.15.89:0/2332859428 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd301f1f90 con 0x7fcd300c6bd0 2023-10-30T12:28:47.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.744+0000 7fcd36a32640 1 -- 172.21.15.89:0/2332859428 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd301f24b0 con 0x7fcd300c6bd0 2023-10-30T12:28:47.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.744+0000 7fcd2cff9640 1 -- 172.21.15.89:0/2332859428 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcd28014b40 con 0x7fcd300c6bd0 2023-10-30T12:28:47.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.744+0000 7fcd2cff9640 1 --2- 172.21.15.89:0/2332859428 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fcd1c0664a0 0x7fcd1c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:47.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.744+0000 7fcd2cff9640 1 -- 172.21.15.89:0/2332859428 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcd280844b0 con 0x7fcd300c6bd0 2023-10-30T12:28:47.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.748+0000 7fcd2ffff640 1 --2- 172.21.15.89:0/2332859428 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fcd1c0664a0 0x7fcd1c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:47.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.748+0000 7fcd2ffff640 1 --2- 172.21.15.89:0/2332859428 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fcd1c0664a0 0x7fcd1c068960 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcd20000f70 tx=0x7fcd20009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:47.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.748+0000 7fcd2cff9640 1 -- 172.21.15.89:0/2332859428 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcd301f24b0 con 0x7fcd300c6bd0 2023-10-30T12:28:47.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.900+0000 7fcd36a32640 1 -- 172.21.15.89:0/2332859428 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd301f24b0 con 0x7fcd300c6bd0 2023-10-30T12:28:47.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.900+0000 7fcd2cff9640 1 -- 172.21.15.89:0/2332859428 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12850 (secure 0 0 0) 0x7fcd301f24b0 con 0x7fcd300c6bd0 2023-10-30T12:28:47.906 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:28:47.906 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":18,"fsid":"401641be-db6e-4c16-b3ac-fd84a86c4de4","created":"2023-10-30T12:28:26.198532+0000","modified":"2023-10-30T12:28:41.916008+0000","last_up_change":"2023-10-30T12:28:37.212205+0000","last_in_change":"2023-10-30T12:28:28.517734+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:39.004943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72595c54-fd6e-4a5a-bb14-ff27e9377b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6801","nonce":1673955622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6803","nonce":1673955622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6807","nonce":1673955622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6805","nonce":1673955622}]},"public_addr":"172.21.15.89:6801/1673955622","cluster_addr":"172.21.15.89:6803/1673955622","heartbeat_back_addr":"172.21.15.89:6807/1673955622","heartbeat_front_addr":"172.21.15.89:6805/1673955622","state":["exists","up"]},{"osd":1,"uuid":"0305670b-d7c4-4193-bec2-8b7351ff590a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6817","nonce":3340572039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6819","nonce":3340572039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6823","nonce":3340572039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6821","nonce":3340572039}]},"public_addr":"172.21.15.89:6817/3340572039","cluster_addr":"172.21.15.89:6819/3340572039","heartbeat_back_addr":"172.21.15.89:6823/3340572039","heartbeat_front_addr":"172.21.15.89:6821/3340572039","state":["exists","up"]},{"osd":2,"uuid":"4bab5426-a3c3-46b0-91fc-546c254006c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6809","nonce":1929940799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6811","nonce":1929940799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6815","nonce":1929940799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6813","nonce":1929940799}]},"public_addr":"172.21.15.89:6809/1929940799","cluster_addr":"172.21.15.89:6811/1929940799","heartbeat_back_addr":"172.21.15.89:6815/1929940799","heartbeat_front_addr":"172.21.15.89:6813/1929940799","state":["exists","up"]},{"osd":3,"uuid":"7d47aea8-27cd-42be-8591-d63ad68c90e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6825","nonce":345087939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6827","nonce":345087939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6831","nonce":345087939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6829","nonce":345087939}]},"public_addr":"172.21.15.89:6825/345087939","cluster_addr":"172.21.15.89:6827/345087939","heartbeat_back_addr":"172.21.15.89:6831/345087939","heartbeat_front_addr":"172.21.15.89:6829/345087939","state":["exists","up"]},{"osd":4,"uuid":"2e2ae8f8-d097-4bba-9448-95047fbfacad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6801","nonce":2138125545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6803","nonce":2138125545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6809","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6811","nonce":2138125545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6805","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6807","nonce":2138125545}]},"public_addr":"172.21.15.192:6801/2138125545","cluster_addr":"172.21.15.192:6803/2138125545","heartbeat_back_addr":"172.21.15.192:6811/2138125545","heartbeat_front_addr":"172.21.15.192:6807/2138125545","state":["exists","up"]},{"osd":5,"uuid":"49be5ce5-12f4-4c26-8d39-864b6ff67313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6806","nonce":2360218671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6810","nonce":2360218671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6815","nonce":2360218671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6813","nonce":2360218671}]},"public_addr":"172.21.15.192:6806/2360218671","cluster_addr":"172.21.15.192:6810/2360218671","heartbeat_back_addr":"172.21.15.192:6815/2360218671","heartbeat_front_addr":"172.21.15.192:6813/2360218671","state":["exists","up"]},{"osd":6,"uuid":"48623452-d456-4ea2-a0de-5fa4b37e6d10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6825","nonce":3707671287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6827","nonce":3707671287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6831","nonce":3707671287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6829","nonce":3707671287}]},"public_addr":"172.21.15.192:6825/3707671287","cluster_addr":"172.21.15.192:6827/3707671287","heartbeat_back_addr":"172.21.15.192:6831/3707671287","heartbeat_front_addr":"172.21.15.192:6829/3707671287","state":["exists","up"]},{"osd":7,"uuid":"ff0a4384-54f8-43ec-a3ce-9c8ae3bf8a94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6817","nonce":3111945871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6819","nonce":3111945871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6823","nonce":3111945871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6821","nonce":3111945871}]},"public_addr":"172.21.15.192:6817/3111945871","cluster_addr":"172.21.15.192:6819/3111945871","heartbeat_back_addr":"172.21.15.192:6823/3111945871","heartbeat_front_addr":"172.21.15.192:6821/3111945871","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.183065+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.248650+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.194576+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.397483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.043648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.186624+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.159674+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.100851+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:28:47.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.900+0000 7fcd36a32640 1 -- 172.21.15.89:0/2332859428 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fcd1c0664a0 msgr2=0x7fcd1c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:47.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.900+0000 7fcd36a32640 1 --2- 172.21.15.89:0/2332859428 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fcd1c0664a0 0x7fcd1c068960 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcd20000f70 tx=0x7fcd20009040 comp rx=0 tx=0).stop 2023-10-30T12:28:47.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.900+0000 7fcd36a32640 1 -- 172.21.15.89:0/2332859428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd300c6bd0 msgr2=0x7fcd3014feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:47.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.900+0000 7fcd36a32640 1 --2- 172.21.15.89:0/2332859428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd300c6bd0 0x7fcd3014feb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcd28007e40 tx=0x7fcd2800e4b0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.900+0000 7fcd36a32640 1 -- 172.21.15.89:0/2332859428 shutdown_connections 2023-10-30T12:28:47.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.900+0000 7fcd36a32640 1 --2- 172.21.15.89:0/2332859428 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fcd1c0664a0 0x7fcd1c068960 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.900+0000 7fcd36a32640 1 --2- 172.21.15.89:0/2332859428 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcd30071110 0x7fcd301f2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.900+0000 7fcd36a32640 1 --2- 172.21.15.89:0/2332859428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd300c6bd0 0x7fcd3014feb0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.900+0000 7fcd36a32640 1 --2- 172.21.15.89:0/2332859428 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcd3006f7a0 0x7fcd301ee7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.900+0000 7fcd36a32640 1 -- 172.21.15.89:0/2332859428 >> 172.21.15.89:0/2332859428 conn(0x7fcd300c7910 msgr2=0x7fcd300c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:47.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.904+0000 7fcd36a32640 1 -- 172.21.15.89:0/2332859428 shutdown_connections 2023-10-30T12:28:47.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:47.904+0000 7fcd36a32640 1 -- 172.21.15.89:0/2332859428 wait complete. 2023-10-30T12:28:47.926 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T12:28:39.004943+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T12:28:47.927 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T12:28:48.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.060+0000 7fed3bc54640 1 Processor -- start 2023-10-30T12:28:48.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed3bc54640 1 -- start start 2023-10-30T12:28:48.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed3bc54640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed341508c0 0x7fed340c6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:48.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed3bc54640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fed340722a0 0x7fed340c6e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:48.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed3bc54640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed340c0a30 con 0x7fed340bfec0 2023-10-30T12:28:48.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed3bc54640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed340c85b0 con 0x7fed340722a0 2023-10-30T12:28:48.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed3bc54640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed3406f650 con 0x7fed341508c0 2023-10-30T12:28:48.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed3a1f1640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fed340bfec0 0x7fed34156930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42580/0 (socket says 172.21.15.89:42580) 2023-10-30T12:28:48.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed3a1f1640 1 -- 172.21.15.89:0/3644435243 learned_addr learned my addr 172.21.15.89:0/3644435243 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:28:48.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed399f0640 1 --2- 172.21.15.89:0/3644435243 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed341508c0 0x7fed340c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:48.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed391ef640 1 --2- 172.21.15.89:0/3644435243 >> v2:172.21.15.192:3300/0 conn(0x7fed340722a0 0x7fed340c6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:48.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed389ee640 1 -- 172.21.15.89:0/3644435243 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3347991657 0 0) 0x7fed340c0a30 con 0x7fed340bfec0 2023-10-30T12:28:48.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed389ee640 1 -- 172.21.15.89:0/3644435243 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed20003530 con 0x7fed340bfec0 2023-10-30T12:28:48.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed389ee640 1 -- 172.21.15.89:0/3644435243 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4048722227 0 0) 0x7fed20003530 con 0x7fed340bfec0 2023-10-30T12:28:48.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed389ee640 1 -- 172.21.15.89:0/3644435243 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed341508c0 msgr2=0x7fed340c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:48.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed389ee640 1 --2- 172.21.15.89:0/3644435243 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed341508c0 0x7fed340c6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed389ee640 1 -- 172.21.15.89:0/3644435243 >> v2:172.21.15.192:3300/0 conn(0x7fed340722a0 msgr2=0x7fed340c6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:48.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed389ee640 1 --2- 172.21.15.89:0/3644435243 >> v2:172.21.15.192:3300/0 conn(0x7fed340722a0 0x7fed340c6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed389ee640 1 -- 172.21.15.89:0/3644435243 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3406f930 con 0x7fed340bfec0 2023-10-30T12:28:48.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed389ee640 1 -- 172.21.15.89:0/3644435243 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7fed2c001a00 con 0x7fed340bfec0 2023-10-30T12:28:48.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed389ee640 1 -- 172.21.15.89:0/3644435243 >> v1:172.21.15.89:6789/0 conn(0x7fed340bfec0 legacy=0x7fed34156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:48.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed389ee640 1 --2- 172.21.15.89:0/3644435243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed20003980 0x7fed20003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:48.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed389ee640 1 --2- 172.21.15.89:0/3644435243 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed341508c0 0x7fed200047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:48.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed389ee640 1 --2- 172.21.15.89:0/3644435243 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed20004d10 0x7fed20005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:48.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed389ee640 1 -- 172.21.15.89:0/3644435243 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed340c85b0 con 0x7fed20003980 2023-10-30T12:28:48.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed389ee640 1 -- 172.21.15.89:0/3644435243 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3406f650 con 0x7fed20004d10 2023-10-30T12:28:48.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed389ee640 1 -- 172.21.15.89:0/3644435243 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed2000b990 con 0x7fed341508c0 2023-10-30T12:28:48.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed399f0640 1 --2- 172.21.15.89:0/3644435243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed20003980 0x7fed20003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:48.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed391ef640 1 --2- 172.21.15.89:0/3644435243 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed341508c0 0x7fed200047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:48.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed3a1f1640 1 --2- 172.21.15.89:0/3644435243 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed20004d10 0x7fed20005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:48.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed399f0640 1 -- 172.21.15.89:0/3644435243 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed341508c0 msgr2=0x7fed200047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:48.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed399f0640 1 --2- 172.21.15.89:0/3644435243 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed341508c0 0x7fed200047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed399f0640 1 -- 172.21.15.89:0/3644435243 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed20004d10 msgr2=0x7fed20005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:48.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed399f0640 1 --2- 172.21.15.89:0/3644435243 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed20004d10 0x7fed20005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed399f0640 1 -- 172.21.15.89:0/3644435243 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed20003530 con 0x7fed20003980 2023-10-30T12:28:48.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed391ef640 1 --2- 172.21.15.89:0/3644435243 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed341508c0 0x7fed200047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:28:48.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.064+0000 7fed3a1f1640 1 --2- 172.21.15.89:0/3644435243 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed20004d10 0x7fed20005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:28:48.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.068+0000 7fed399f0640 1 --2- 172.21.15.89:0/3644435243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed20003980 0x7fed20003da0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fed30009ac0 tx=0x7fed3001ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89bae40072b48277 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:48.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.068+0000 7fed389ee640 1 -- 172.21.15.89:0/3644435243 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed300059a0 con 0x7fed20003980 2023-10-30T12:28:48.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.068+0000 7fed389ee640 1 -- 172.21.15.89:0/3644435243 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed30005b00 con 0x7fed20003980 2023-10-30T12:28:48.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.068+0000 7fed389ee640 1 -- 172.21.15.89:0/3644435243 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed30006030 con 0x7fed20003980 2023-10-30T12:28:48.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.068+0000 7fed3bc54640 1 -- 172.21.15.89:0/3644435243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed20003980 msgr2=0x7fed20003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:48.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.068+0000 7fed3bc54640 1 --2- 172.21.15.89:0/3644435243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed20003980 0x7fed20003da0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fed30009ac0 tx=0x7fed3001ca30 comp rx=0 tx=0).stop 2023-10-30T12:28:48.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.068+0000 7fed3a1f1640 1 -- 172.21.15.89:0/3644435243 reap_dead start 2023-10-30T12:28:48.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.068+0000 7fed3bc54640 1 -- 172.21.15.89:0/3644435243 shutdown_connections 2023-10-30T12:28:48.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.068+0000 7fed3bc54640 1 -- 172.21.15.89:0/3644435243 >> 172.21.15.89:0/3644435243 conn(0x7fed340c7910 msgr2=0x7fed340c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:48.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.068+0000 7fed3bc54640 1 -- 172.21.15.89:0/3644435243 shutdown_connections 2023-10-30T12:28:48.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.068+0000 7fed3bc54640 1 -- 172.21.15.89:0/3644435243 wait complete. 2023-10-30T12:28:48.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.072+0000 7fed3bc54640 1 Processor -- start 2023-10-30T12:28:48.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.072+0000 7fed3bc54640 1 -- start start 2023-10-30T12:28:48.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.072+0000 7fed3bc54640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed20003980 0x7fed34152750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:48.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.072+0000 7fed3bc54640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed34151a20 0x7fed34155510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:48.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.072+0000 7fed3bc54640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed34146690 0x7fed34146b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:48.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.072+0000 7fed3bc54640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed34147050 con 0x7fed34151a20 2023-10-30T12:28:48.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.072+0000 7fed3bc54640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3415dba0 con 0x7fed20003980 2023-10-30T12:28:48.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.072+0000 7fed3bc54640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed3415de60 con 0x7fed34146690 2023-10-30T12:28:48.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.072+0000 7fed391ef640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed34151a20 0x7fed34155510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:48.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.072+0000 7fed391ef640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed34151a20 0x7fed34155510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35902/0 (socket says 172.21.15.89:35902) 2023-10-30T12:28:48.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.072+0000 7fed391ef640 1 -- 172.21.15.89:0/3686610870 learned_addr learned my addr 172.21.15.89:0/3686610870 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:48.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.072+0000 7fed399f0640 1 --2- 172.21.15.89:0/3686610870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed20003980 0x7fed34152750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:48.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.072+0000 7fed3a1f1640 1 --2- 172.21.15.89:0/3686610870 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed34146690 0x7fed34146b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:48.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.072+0000 7fed391ef640 1 -- 172.21.15.89:0/3686610870 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed34146690 msgr2=0x7fed34146b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:48.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.072+0000 7fed391ef640 1 --2- 172.21.15.89:0/3686610870 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed34146690 0x7fed34146b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.072+0000 7fed391ef640 1 -- 172.21.15.89:0/3686610870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed20003980 msgr2=0x7fed34152750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:48.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.072+0000 7fed391ef640 1 --2- 172.21.15.89:0/3686610870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed20003980 0x7fed34152750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.072+0000 7fed391ef640 1 -- 172.21.15.89:0/3686610870 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed34151fd0 con 0x7fed34151a20 2023-10-30T12:28:48.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.072+0000 7fed3a1f1640 1 --2- 172.21.15.89:0/3686610870 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed34146690 0x7fed34146b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:28:48.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.072+0000 7fed391ef640 1 --2- 172.21.15.89:0/3686610870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed34151a20 0x7fed34155510 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fed24010980 tx=0x7fed24010e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:48.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.072+0000 7fed2a7fc640 1 -- 172.21.15.89:0/3686610870 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed2400ebc0 con 0x7fed34151a20 2023-10-30T12:28:48.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.072+0000 7fed2a7fc640 1 -- 172.21.15.89:0/3686610870 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed2400ed20 con 0x7fed34151a20 2023-10-30T12:28:48.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.072+0000 7fed2a7fc640 1 -- 172.21.15.89:0/3686610870 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed2401c600 con 0x7fed34151a20 2023-10-30T12:28:48.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.072+0000 7fed3bc54640 1 -- 172.21.15.89:0/3686610870 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed341522b0 con 0x7fed34151a20 2023-10-30T12:28:48.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.072+0000 7fed3bc54640 1 -- 172.21.15.89:0/3686610870 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed3415e3b0 con 0x7fed34151a20 2023-10-30T12:28:48.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.072+0000 7fed3bc54640 1 -- 172.21.15.89:0/3686610870 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed340c6f00 con 0x7fed34151a20 2023-10-30T12:28:48.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.076+0000 7fed2a7fc640 1 -- 172.21.15.89:0/3686610870 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fed2401c760 con 0x7fed34151a20 2023-10-30T12:28:48.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.076+0000 7fed2a7fc640 1 --2- 172.21.15.89:0/3686610870 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fed180664a0 0x7fed18068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:48.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.076+0000 7fed2a7fc640 1 -- 172.21.15.89:0/3686610870 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fed24083fe0 con 0x7fed34151a20 2023-10-30T12:28:48.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.076+0000 7fed399f0640 1 --2- 172.21.15.89:0/3686610870 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fed180664a0 0x7fed18068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:48.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.076+0000 7fed399f0640 1 --2- 172.21.15.89:0/3686610870 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fed180664a0 0x7fed18068960 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fed300ae000 tx=0x7fed300b0040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:48.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.080+0000 7fed2a7fc640 1 -- 172.21.15.89:0/3686610870 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fed240517c0 con 0x7fed34151a20 2023-10-30T12:28:48.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.232+0000 7fed3bc54640 1 -- 172.21.15.89:0/3686610870 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fed3404e170 con 0x7fed34151a20 2023-10-30T12:28:48.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.232+0000 7fed2a7fc640 1 -- 172.21.15.89:0/3686610870 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7fed24054de0 con 0x7fed34151a20 2023-10-30T12:28:48.239 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-10-30T12:28:48.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.236+0000 7fed3bc54640 1 -- 172.21.15.89:0/3686610870 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fed180664a0 msgr2=0x7fed18068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:48.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.236+0000 7fed3bc54640 1 --2- 172.21.15.89:0/3686610870 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fed180664a0 0x7fed18068960 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fed300ae000 tx=0x7fed300b0040 comp rx=0 tx=0).stop 2023-10-30T12:28:48.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.236+0000 7fed3bc54640 1 -- 172.21.15.89:0/3686610870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed34151a20 msgr2=0x7fed34155510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:48.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.236+0000 7fed3bc54640 1 --2- 172.21.15.89:0/3686610870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed34151a20 0x7fed34155510 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fed24010980 tx=0x7fed24010e40 comp rx=0 tx=0).stop 2023-10-30T12:28:48.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.236+0000 7fed3bc54640 1 -- 172.21.15.89:0/3686610870 shutdown_connections 2023-10-30T12:28:48.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.236+0000 7fed3bc54640 1 --2- 172.21.15.89:0/3686610870 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fed180664a0 0x7fed18068960 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.236+0000 7fed3bc54640 1 --2- 172.21.15.89:0/3686610870 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed34146690 0x7fed34146b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.236+0000 7fed3bc54640 1 --2- 172.21.15.89:0/3686610870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed34151a20 0x7fed34155510 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.236+0000 7fed3bc54640 1 --2- 172.21.15.89:0/3686610870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed20003980 0x7fed34152750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.236+0000 7fed3bc54640 1 -- 172.21.15.89:0/3686610870 >> 172.21.15.89:0/3686610870 conn(0x7fed340c7910 msgr2=0x7fed34153220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:48.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.236+0000 7fed3bc54640 1 -- 172.21.15.89:0/3686610870 shutdown_connections 2023-10-30T12:28:48.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.236+0000 7fed3bc54640 1 -- 172.21.15.89:0/3686610870 wait complete. 2023-10-30T12:28:48.259 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-30T12:28:48.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.392+0000 7fd1036d6640 1 Processor -- start 2023-10-30T12:28:48.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.392+0000 7fd1036d6640 1 -- start start 2023-10-30T12:28:48.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.392+0000 7fd1036d6640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd0fc153d70 0x7fd0fc06ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:48.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.392+0000 7fd1036d6640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd0fc0c8490 0x7fd0fc0c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:48.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.392+0000 7fd1036d6640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0fc06afa0 con 0x7fd0fc06e470 2023-10-30T12:28:48.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.392+0000 7fd1036d6640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0fc06d780 con 0x7fd0fc153d70 2023-10-30T12:28:48.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.392+0000 7fd1036d6640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0fc154150 con 0x7fd0fc0c8490 2023-10-30T12:28:48.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.392+0000 7fd101472640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd0fc153d70 0x7fd0fc06ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:48.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.392+0000 7fd101472640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd0fc153d70 0x7fd0fc06ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35918/0 (socket says 172.21.15.89:35918) 2023-10-30T12:28:48.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.392+0000 7fd101472640 1 -- 172.21.15.89:0/3364960285 learned_addr learned my addr 172.21.15.89:0/3364960285 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:48.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.392+0000 7fd101c73640 1 --2- 172.21.15.89:0/3364960285 >> v2:172.21.15.192:3300/0 conn(0x7fd0fc0c8490 0x7fd0fc0c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:48.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.392+0000 7fd101472640 1 -- 172.21.15.89:0/3364960285 >> v2:172.21.15.192:3300/0 conn(0x7fd0fc0c8490 msgr2=0x7fd0fc0c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:48.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.392+0000 7fd101472640 1 --2- 172.21.15.89:0/3364960285 >> v2:172.21.15.192:3300/0 conn(0x7fd0fc0c8490 0x7fd0fc0c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.392+0000 7fd101472640 1 -- 172.21.15.89:0/3364960285 >> v1:172.21.15.192:6789/0 conn(0x7fd0fc06e470 legacy=0x7fd0fc0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:48.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.392+0000 7fd101472640 1 -- 172.21.15.89:0/3364960285 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0fc06e030 con 0x7fd0fc153d70 2023-10-30T12:28:48.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.396+0000 7fd101472640 1 --2- 172.21.15.89:0/3364960285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0fc153d70 0x7fd0fc06ca80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc0cddf0 tx=0x7fd0f801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6549e08a15433608 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:48.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.396+0000 7fd0f3fff640 1 -- 172.21.15.89:0/3364960285 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f80aa430 con 0x7fd0fc153d70 2023-10-30T12:28:48.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.396+0000 7fd0f3fff640 1 -- 172.21.15.89:0/3364960285 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0f80aa590 con 0x7fd0fc153d70 2023-10-30T12:28:48.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.396+0000 7fd0f3fff640 1 -- 172.21.15.89:0/3364960285 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f80aa920 con 0x7fd0fc153d70 2023-10-30T12:28:48.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.396+0000 7fd1036d6640 1 -- 172.21.15.89:0/3364960285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0fc153d70 msgr2=0x7fd0fc06ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:48.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.396+0000 7fd1036d6640 1 --2- 172.21.15.89:0/3364960285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0fc153d70 0x7fd0fc06ca80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc0cddf0 tx=0x7fd0f801c4d0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.396+0000 7fd1036d6640 1 -- 172.21.15.89:0/3364960285 shutdown_connections 2023-10-30T12:28:48.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.396+0000 7fd1036d6640 1 --2- 172.21.15.89:0/3364960285 >> v2:172.21.15.192:3300/0 conn(0x7fd0fc0c8490 0x7fd0fc0c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.396+0000 7fd1036d6640 1 --2- 172.21.15.89:0/3364960285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0fc153d70 0x7fd0fc06ca80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.396+0000 7fd1036d6640 1 -- 172.21.15.89:0/3364960285 >> 172.21.15.89:0/3364960285 conn(0x7fd0fc071ea0 msgr2=0x7fd0fc065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:48.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.396+0000 7fd1036d6640 1 -- 172.21.15.89:0/3364960285 shutdown_connections 2023-10-30T12:28:48.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.396+0000 7fd1036d6640 1 -- 172.21.15.89:0/3364960285 wait complete. 2023-10-30T12:28:48.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.400+0000 7fd1036d6640 1 Processor -- start 2023-10-30T12:28:48.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.400+0000 7fd1036d6640 1 -- start start 2023-10-30T12:28:48.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.400+0000 7fd1036d6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0fc06e470 0x7fd0fc0cb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:48.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.400+0000 7fd1036d6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0fc0c8490 0x7fd0fc1495b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:48.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.400+0000 7fd1036d6640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd0fc153d70 0x7fd0fc149af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:48.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.400+0000 7fd1036d6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc0ccc00 con 0x7fd0fc0c8490 2023-10-30T12:28:48.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.400+0000 7fd1036d6640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc0cbae0 con 0x7fd0fc06e470 2023-10-30T12:28:48.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.400+0000 7fd1036d6640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc14c2c0 con 0x7fd0fc153d70 2023-10-30T12:28:48.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.400+0000 7fd100c71640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0fc0c8490 0x7fd0fc1495b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:48.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.400+0000 7fd100c71640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0fc0c8490 0x7fd0fc1495b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35920/0 (socket says 172.21.15.89:35920) 2023-10-30T12:28:48.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.400+0000 7fd100c71640 1 -- 172.21.15.89:0/1738828659 learned_addr learned my addr 172.21.15.89:0/1738828659 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:48.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.400+0000 7fd101472640 1 --2- 172.21.15.89:0/1738828659 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0fc06e470 0x7fd0fc0cb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:48.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.400+0000 7fd101c73640 1 --2- 172.21.15.89:0/1738828659 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd0fc153d70 0x7fd0fc149af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:48.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.400+0000 7fd100c71640 1 -- 172.21.15.89:0/1738828659 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd0fc153d70 msgr2=0x7fd0fc149af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:48.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.400+0000 7fd100c71640 1 --2- 172.21.15.89:0/1738828659 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd0fc153d70 0x7fd0fc149af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.400+0000 7fd100c71640 1 -- 172.21.15.89:0/1738828659 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0fc06e470 msgr2=0x7fd0fc0cb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:48.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.400+0000 7fd100c71640 1 --2- 172.21.15.89:0/1738828659 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0fc06e470 0x7fd0fc0cb5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.400+0000 7fd100c71640 1 -- 172.21.15.89:0/1738828659 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f80094d0 con 0x7fd0fc0c8490 2023-10-30T12:28:48.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.400+0000 7fd100c71640 1 --2- 172.21.15.89:0/1738828659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0fc0c8490 0x7fd0fc1495b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec00c3c0 tx=0x7fd0ec00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:48.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.400+0000 7fd0f1ffb640 1 -- 172.21.15.89:0/1738828659 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec0052d0 con 0x7fd0fc0c8490 2023-10-30T12:28:48.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.400+0000 7fd0f1ffb640 1 -- 172.21.15.89:0/1738828659 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0ec005430 con 0x7fd0fc0c8490 2023-10-30T12:28:48.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.400+0000 7fd1036d6640 1 -- 172.21.15.89:0/1738828659 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0fc143db0 con 0x7fd0fc0c8490 2023-10-30T12:28:48.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.400+0000 7fd0f1ffb640 1 -- 172.21.15.89:0/1738828659 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec003840 con 0x7fd0fc0c8490 2023-10-30T12:28:48.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.400+0000 7fd1036d6640 1 -- 172.21.15.89:0/1738828659 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0fc0cc640 con 0x7fd0fc0c8490 2023-10-30T12:28:48.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.404+0000 7fd1036d6640 1 -- 172.21.15.89:0/1738828659 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0fc04e170 con 0x7fd0fc0c8490 2023-10-30T12:28:48.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.404+0000 7fd0f1ffb640 1 -- 172.21.15.89:0/1738828659 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd0ec0039a0 con 0x7fd0fc0c8490 2023-10-30T12:28:48.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.404+0000 7fd0f1ffb640 1 --2- 172.21.15.89:0/1738828659 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd0e00664a0 0x7fd0e0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:48.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.404+0000 7fd0f1ffb640 1 -- 172.21.15.89:0/1738828659 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd0ec082f40 con 0x7fd0fc0c8490 2023-10-30T12:28:48.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.404+0000 7fd101472640 1 --2- 172.21.15.89:0/1738828659 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd0e00664a0 0x7fd0e0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:48.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.404+0000 7fd101472640 1 --2- 172.21.15.89:0/1738828659 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd0e00664a0 0x7fd0e0068960 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd0f80ae000 tx=0x7fd0f80b5040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:48.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.408+0000 7fd0f1ffb640 1 -- 172.21.15.89:0/1738828659 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd0ec050850 con 0x7fd0fc0c8490 2023-10-30T12:28:48.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.560+0000 7fd1036d6640 1 -- 172.21.15.89:0/1738828659 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd0fc06c2a0 con 0x7fd0fc0c8490 2023-10-30T12:28:48.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.560+0000 7fd0f1ffb640 1 -- 172.21.15.89:0/1738828659 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) v1 ==== 84+0+65 (secure 0 0 0) 0x7fd0ec017090 con 0x7fd0fc0c8490 2023-10-30T12:28:48.565 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:28:48.565 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2023-10-30T12:28:48.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2023-10-30T12:28:48.566 INFO:teuthology.orchestra.run.smithi089.stdout: "poolnum": 1, 2023-10-30T12:28:48.566 INFO:teuthology.orchestra.run.smithi089.stdout: "poolname": ".mgr" 2023-10-30T12:28:48.566 INFO:teuthology.orchestra.run.smithi089.stdout: } 2023-10-30T12:28:48.566 INFO:teuthology.orchestra.run.smithi089.stdout:] 2023-10-30T12:28:48.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.560+0000 7fd1036d6640 1 -- 172.21.15.89:0/1738828659 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd0e00664a0 msgr2=0x7fd0e0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:48.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.560+0000 7fd1036d6640 1 --2- 172.21.15.89:0/1738828659 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd0e00664a0 0x7fd0e0068960 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd0f80ae000 tx=0x7fd0f80b5040 comp rx=0 tx=0).stop 2023-10-30T12:28:48.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.560+0000 7fd1036d6640 1 -- 172.21.15.89:0/1738828659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0fc0c8490 msgr2=0x7fd0fc1495b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:48.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.560+0000 7fd1036d6640 1 --2- 172.21.15.89:0/1738828659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0fc0c8490 0x7fd0fc1495b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec00c3c0 tx=0x7fd0ec00c880 comp rx=0 tx=0).stop 2023-10-30T12:28:48.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.560+0000 7fd1036d6640 1 -- 172.21.15.89:0/1738828659 shutdown_connections 2023-10-30T12:28:48.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.560+0000 7fd1036d6640 1 --2- 172.21.15.89:0/1738828659 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd0e00664a0 0x7fd0e0068960 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.560+0000 7fd1036d6640 1 --2- 172.21.15.89:0/1738828659 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd0fc153d70 0x7fd0fc149af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.560+0000 7fd1036d6640 1 --2- 172.21.15.89:0/1738828659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0fc0c8490 0x7fd0fc1495b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.560+0000 7fd1036d6640 1 --2- 172.21.15.89:0/1738828659 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0fc06e470 0x7fd0fc0cb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.560+0000 7fd1036d6640 1 -- 172.21.15.89:0/1738828659 >> 172.21.15.89:0/1738828659 conn(0x7fd0fc071ea0 msgr2=0x7fd0fc14e640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:48.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.560+0000 7fd1036d6640 1 -- 172.21.15.89:0/1738828659 shutdown_connections 2023-10-30T12:28:48.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.560+0000 7fd1036d6640 1 -- 172.21.15.89:0/1738828659 wait complete. 2023-10-30T12:28:48.585 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-30T12:28:48.586 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-30T12:28:48.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.720+0000 7fbb4c5d3640 1 Processor -- start 2023-10-30T12:28:48.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.720+0000 7fbb4c5d3640 1 -- start start 2023-10-30T12:28:48.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.724+0000 7fbb4c5d3640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbb4415d610 0x7fbb440c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:48.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.724+0000 7fbb4c5d3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb4406e620 0x7fbb440be770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:48.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.724+0000 7fbb4c5d3640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb441596f0 con 0x7fbb4415d9f0 2023-10-30T12:28:48.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.724+0000 7fbb4c5d3640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb440c64c0 con 0x7fbb4415d610 2023-10-30T12:28:48.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.724+0000 7fbb4c5d3640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb440c67a0 con 0x7fbb4406e620 2023-10-30T12:28:48.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.724+0000 7fbb4a36f640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbb4415d610 0x7fbb440c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:48.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.724+0000 7fbb4a36f640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbb4415d610 0x7fbb440c78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35926/0 (socket says 172.21.15.89:35926) 2023-10-30T12:28:48.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.724+0000 7fbb4a36f640 1 -- 172.21.15.89:0/4112783205 learned_addr learned my addr 172.21.15.89:0/4112783205 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:48.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.724+0000 7fbb49b6e640 1 --2- 172.21.15.89:0/4112783205 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb4406e620 0x7fbb440be770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:48.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.724+0000 7fbb4a36f640 1 -- 172.21.15.89:0/4112783205 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb4406e620 msgr2=0x7fbb440be770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:48.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.724+0000 7fbb4a36f640 1 --2- 172.21.15.89:0/4112783205 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb4406e620 0x7fbb440be770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.724+0000 7fbb4a36f640 1 -- 172.21.15.89:0/4112783205 >> v1:172.21.15.192:6789/0 conn(0x7fbb4415d9f0 legacy=0x7fbb44069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:48.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.724+0000 7fbb4a36f640 1 -- 172.21.15.89:0/4112783205 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb44159970 con 0x7fbb4415d610 2023-10-30T12:28:48.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.728+0000 7fbb4a36f640 1 --2- 172.21.15.89:0/4112783205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb4415d610 0x7fbb440c78e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbb40009870 tx=0x7fbb4001c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76f9dcbcc03047eb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:48.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.728+0000 7fbb4936d640 1 -- 172.21.15.89:0/4112783205 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb400a95f0 con 0x7fbb4415d610 2023-10-30T12:28:48.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.728+0000 7fbb4936d640 1 -- 172.21.15.89:0/4112783205 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb4406a7b0 con 0x7fbb4415d610 2023-10-30T12:28:48.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.728+0000 7fbb4936d640 1 -- 172.21.15.89:0/4112783205 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb400a9970 con 0x7fbb4415d610 2023-10-30T12:28:48.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.728+0000 7fbb4c5d3640 1 -- 172.21.15.89:0/4112783205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb4415d610 msgr2=0x7fbb440c78e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:48.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.728+0000 7fbb4c5d3640 1 --2- 172.21.15.89:0/4112783205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb4415d610 0x7fbb440c78e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbb40009870 tx=0x7fbb4001c510 comp rx=0 tx=0).stop 2023-10-30T12:28:48.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.728+0000 7fbb4c5d3640 1 -- 172.21.15.89:0/4112783205 shutdown_connections 2023-10-30T12:28:48.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.728+0000 7fbb4c5d3640 1 --2- 172.21.15.89:0/4112783205 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb4406e620 0x7fbb440be770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.728+0000 7fbb4c5d3640 1 --2- 172.21.15.89:0/4112783205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb4415d610 0x7fbb440c78e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.728+0000 7fbb4c5d3640 1 -- 172.21.15.89:0/4112783205 >> 172.21.15.89:0/4112783205 conn(0x7fbb440cb490 msgr2=0x7fbb440cb8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:48.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.728+0000 7fbb4c5d3640 1 -- 172.21.15.89:0/4112783205 shutdown_connections 2023-10-30T12:28:48.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.728+0000 7fbb4c5d3640 1 -- 172.21.15.89:0/4112783205 wait complete. 2023-10-30T12:28:48.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.728+0000 7fbb4c5d3640 1 Processor -- start 2023-10-30T12:28:48.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.728+0000 7fbb4c5d3640 1 -- start start 2023-10-30T12:28:48.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.732+0000 7fbb4c5d3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb4406e620 0x7fbb441524e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:48.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.732+0000 7fbb4c5d3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb4415d610 0x7fbb441f2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:48.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.732+0000 7fbb4c5d3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb4415d9f0 0x7fbb44158160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:48.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.732+0000 7fbb4c5d3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb440bf540 con 0x7fbb4415d9f0 2023-10-30T12:28:48.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.732+0000 7fbb4c5d3640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb440c1860 con 0x7fbb4415d610 2023-10-30T12:28:48.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.732+0000 7fbb4c5d3640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb441f30a0 con 0x7fbb4406e620 2023-10-30T12:28:48.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.732+0000 7fbb4ab70640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb4415d9f0 0x7fbb44158160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:48.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.732+0000 7fbb4ab70640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb4415d9f0 0x7fbb44158160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35932/0 (socket says 172.21.15.89:35932) 2023-10-30T12:28:48.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.732+0000 7fbb4ab70640 1 -- 172.21.15.89:0/1524552563 learned_addr learned my addr 172.21.15.89:0/1524552563 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:48.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.732+0000 7fbb4a36f640 1 --2- 172.21.15.89:0/1524552563 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb4406e620 0x7fbb441524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:48.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.732+0000 7fbb49b6e640 1 --2- 172.21.15.89:0/1524552563 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb4415d610 0x7fbb441f2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:48.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.732+0000 7fbb4ab70640 1 -- 172.21.15.89:0/1524552563 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb4406e620 msgr2=0x7fbb441524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:48.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.732+0000 7fbb4ab70640 1 --2- 172.21.15.89:0/1524552563 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb4406e620 0x7fbb441524e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.732+0000 7fbb4ab70640 1 -- 172.21.15.89:0/1524552563 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb4415d610 msgr2=0x7fbb441f2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:48.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.732+0000 7fbb4ab70640 1 --2- 172.21.15.89:0/1524552563 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb4415d610 0x7fbb441f2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.732+0000 7fbb4ab70640 1 -- 172.21.15.89:0/1524552563 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb400094d0 con 0x7fbb4415d9f0 2023-10-30T12:28:48.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.732+0000 7fbb4a36f640 1 --2- 172.21.15.89:0/1524552563 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb4406e620 0x7fbb441524e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:48.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.732+0000 7fbb4ab70640 1 --2- 172.21.15.89:0/1524552563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb4415d9f0 0x7fbb44158160 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c00c730 tx=0x7fbb3c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:48.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.732+0000 7fbb3affd640 1 -- 172.21.15.89:0/1524552563 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c003b80 con 0x7fbb4415d9f0 2023-10-30T12:28:48.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.732+0000 7fbb3affd640 1 -- 172.21.15.89:0/1524552563 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb3c003ce0 con 0x7fbb4415d9f0 2023-10-30T12:28:48.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.732+0000 7fbb3affd640 1 -- 172.21.15.89:0/1524552563 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c005310 con 0x7fbb4415d9f0 2023-10-30T12:28:48.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.732+0000 7fbb4c5d3640 1 -- 172.21.15.89:0/1524552563 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb44152bb0 con 0x7fbb4415d9f0 2023-10-30T12:28:48.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.732+0000 7fbb4c5d3640 1 -- 172.21.15.89:0/1524552563 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb44150df0 con 0x7fbb4415d9f0 2023-10-30T12:28:48.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.732+0000 7fbb4c5d3640 1 -- 172.21.15.89:0/1524552563 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb4404e170 con 0x7fbb4415d9f0 2023-10-30T12:28:48.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.732+0000 7fbb3affd640 1 -- 172.21.15.89:0/1524552563 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbb3c005680 con 0x7fbb4415d9f0 2023-10-30T12:28:48.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.736+0000 7fbb3affd640 1 --2- 172.21.15.89:0/1524552563 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbb280664a0 0x7fbb28068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:48.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.736+0000 7fbb3affd640 1 -- 172.21.15.89:0/1524552563 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbb3c083a20 con 0x7fbb4415d9f0 2023-10-30T12:28:48.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.736+0000 7fbb4a36f640 1 --2- 172.21.15.89:0/1524552563 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbb280664a0 0x7fbb28068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:48.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.736+0000 7fbb4a36f640 1 --2- 172.21.15.89:0/1524552563 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbb280664a0 0x7fbb28068960 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbb400ae000 tx=0x7fbb400b3040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:48.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.740+0000 7fbb3affd640 1 -- 172.21.15.89:0/1524552563 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbb3c0511f0 con 0x7fbb4415d9f0 2023-10-30T12:28:48.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:48.888+0000 7fbb4c5d3640 1 -- 172.21.15.89:0/1524552563 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fbb44159220 con 0x7fbb4415d9f0 2023-10-30T12:28:49.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.328+0000 7fbb3affd640 1 -- 172.21.15.89:0/1524552563 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v19) v1 ==== 138+0+0 (secure 0 0 0) 0x7fbb3c054810 con 0x7fbb4415d9f0 2023-10-30T12:28:49.333 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' created 2023-10-30T12:28:49.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.328+0000 7fbb4c5d3640 1 -- 172.21.15.89:0/1524552563 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbb280664a0 msgr2=0x7fbb28068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:49.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.328+0000 7fbb4c5d3640 1 --2- 172.21.15.89:0/1524552563 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbb280664a0 0x7fbb28068960 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbb400ae000 tx=0x7fbb400b3040 comp rx=0 tx=0).stop 2023-10-30T12:28:49.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.328+0000 7fbb4c5d3640 1 -- 172.21.15.89:0/1524552563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb4415d9f0 msgr2=0x7fbb44158160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:49.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.328+0000 7fbb4c5d3640 1 --2- 172.21.15.89:0/1524552563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb4415d9f0 0x7fbb44158160 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c00c730 tx=0x7fbb3c00cbf0 comp rx=0 tx=0).stop 2023-10-30T12:28:49.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.328+0000 7fbb4c5d3640 1 -- 172.21.15.89:0/1524552563 shutdown_connections 2023-10-30T12:28:49.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.328+0000 7fbb4c5d3640 1 --2- 172.21.15.89:0/1524552563 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbb280664a0 0x7fbb28068960 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:49.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.328+0000 7fbb4c5d3640 1 --2- 172.21.15.89:0/1524552563 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb4415d610 0x7fbb441f2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:49.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.328+0000 7fbb4c5d3640 1 --2- 172.21.15.89:0/1524552563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb4415d9f0 0x7fbb44158160 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:49.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.328+0000 7fbb4c5d3640 1 --2- 172.21.15.89:0/1524552563 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb4406e620 0x7fbb441524e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:49.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.328+0000 7fbb4c5d3640 1 -- 172.21.15.89:0/1524552563 >> 172.21.15.89:0/1524552563 conn(0x7fbb440cb490 msgr2=0x7fbb440bcf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:49.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.328+0000 7fbb4c5d3640 1 -- 172.21.15.89:0/1524552563 shutdown_connections 2023-10-30T12:28:49.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.328+0000 7fbb4c5d3640 1 -- 172.21.15.89:0/1524552563 wait complete. 2023-10-30T12:28:49.361 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-30T12:28:49.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.500+0000 7f52109c5640 1 Processor -- start 2023-10-30T12:28:49.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.500+0000 7f52109c5640 1 -- start start 2023-10-30T12:28:49.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.504+0000 7f52109c5640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f520c06d420 0x7f520c1650b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:49.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.504+0000 7f52109c5640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f520c15a670 con 0x7f520c15a1a0 2023-10-30T12:28:49.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.504+0000 7f52109c5640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f520c0c0f70 con 0x7f520c1539e0 2023-10-30T12:28:49.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.504+0000 7f52109c5640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f520c0ca3c0 con 0x7f520c06d420 2023-10-30T12:28:49.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.504+0000 7f520ad9d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f520c06d420 0x7f520c1650b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:49.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.504+0000 7f520ad9d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f520c06d420 0x7f520c1650b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35940/0 (socket says 172.21.15.89:35940) 2023-10-30T12:28:49.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.504+0000 7f520ad9d640 1 -- 172.21.15.89:0/1877987968 learned_addr learned my addr 172.21.15.89:0/1877987968 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:49.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.504+0000 7f5209d9b640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f520c15a1a0 0x7f520c0c78f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42596/0 (socket says 172.21.15.89:42596) 2023-10-30T12:28:49.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.504+0000 7f520ad9d640 1 -- 172.21.15.89:0/1877987968 >> v1:172.21.15.192:6789/0 conn(0x7f520c1539e0 legacy=0x7f520c0c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:49.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.504+0000 7f520ad9d640 1 -- 172.21.15.89:0/1877987968 >> v1:172.21.15.89:6789/0 conn(0x7f520c15a1a0 legacy=0x7f520c0c78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:49.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.504+0000 7f520959a640 1 -- 172.21.15.89:0/1877987968 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3798512834 0 0) 0x7f520c15a670 con 0x7f520c15a1a0 2023-10-30T12:28:49.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.504+0000 7f520ad9d640 1 -- 172.21.15.89:0/1877987968 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f520c0c1250 con 0x7f520c06d420 2023-10-30T12:28:49.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.504+0000 7f520ad9d640 1 --2- 172.21.15.89:0/1877987968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f520c06d420 0x7f520c1650b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5204009870 tx=0x7f520401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e21dfd89b56c6052 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:49.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.504+0000 7f520959a640 1 -- 172.21.15.89:0/1877987968 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52040a8a30 con 0x7f520c06d420 2023-10-30T12:28:49.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.504+0000 7f520959a640 1 -- 172.21.15.89:0/1877987968 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52040a8b90 con 0x7f520c06d420 2023-10-30T12:28:49.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.504+0000 7f520959a640 1 -- 172.21.15.89:0/1877987968 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52040b0440 con 0x7f520c06d420 2023-10-30T12:28:49.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.508+0000 7f52109c5640 1 -- 172.21.15.89:0/1877987968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f520c06d420 msgr2=0x7f520c1650b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:49.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.508+0000 7f52109c5640 1 --2- 172.21.15.89:0/1877987968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f520c06d420 0x7f520c1650b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5204009870 tx=0x7f520401ed30 comp rx=0 tx=0).stop 2023-10-30T12:28:49.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.508+0000 7f52109c5640 1 -- 172.21.15.89:0/1877987968 shutdown_connections 2023-10-30T12:28:49.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.508+0000 7f52109c5640 1 --2- 172.21.15.89:0/1877987968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f520c06d420 0x7f520c1650b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:49.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.508+0000 7f52109c5640 1 -- 172.21.15.89:0/1877987968 >> 172.21.15.89:0/1877987968 conn(0x7f520c06c850 msgr2=0x7f520c06cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:49.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.508+0000 7f52109c5640 1 -- 172.21.15.89:0/1877987968 shutdown_connections 2023-10-30T12:28:49.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.508+0000 7f52109c5640 1 -- 172.21.15.89:0/1877987968 wait complete. 2023-10-30T12:28:49.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.508+0000 7f52109c5640 1 Processor -- start 2023-10-30T12:28:49.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.508+0000 7f52109c5640 1 -- start start 2023-10-30T12:28:49.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.508+0000 7f52109c5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f520c06d420 0x7f520c158db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:49.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.508+0000 7f52109c5640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f520c1539e0 0x7f520c0c56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:49.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.508+0000 7f52109c5640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f520c15a1a0 0x7f520c1ef250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:49.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.508+0000 7f52109c5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520c1eee30 con 0x7f520c06d420 2023-10-30T12:28:49.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.508+0000 7f52109c5640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520c0b0fd0 con 0x7f520c1539e0 2023-10-30T12:28:49.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.508+0000 7f52109c5640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f520c1592f0 con 0x7f520c15a1a0 2023-10-30T12:28:49.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.508+0000 7f520a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f520c06d420 0x7f520c158db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:49.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.508+0000 7f520a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f520c06d420 0x7f520c158db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35952/0 (socket says 172.21.15.89:35952) 2023-10-30T12:28:49.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.508+0000 7f520a59c640 1 -- 172.21.15.89:0/2277537921 learned_addr learned my addr 172.21.15.89:0/2277537921 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:49.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.508+0000 7f5209d9b640 1 --2- 172.21.15.89:0/2277537921 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f520c1539e0 0x7f520c0c56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:49.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.512+0000 7f520ad9d640 1 --2- 172.21.15.89:0/2277537921 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f520c15a1a0 0x7f520c1ef250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:49.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.512+0000 7f520a59c640 1 -- 172.21.15.89:0/2277537921 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f520c15a1a0 msgr2=0x7f520c1ef250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:49.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.512+0000 7f520a59c640 1 --2- 172.21.15.89:0/2277537921 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f520c15a1a0 0x7f520c1ef250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:49.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.512+0000 7f520a59c640 1 -- 172.21.15.89:0/2277537921 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f520c1539e0 msgr2=0x7f520c0c56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:49.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.512+0000 7f520a59c640 1 --2- 172.21.15.89:0/2277537921 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f520c1539e0 0x7f520c0c56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:49.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.512+0000 7f520a59c640 1 -- 172.21.15.89:0/2277537921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52040094d0 con 0x7f520c06d420 2023-10-30T12:28:49.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.512+0000 7f520a59c640 1 --2- 172.21.15.89:0/2277537921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f520c06d420 0x7f520c158db0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f51fc00c820 tx=0x7f51fc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:49.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.512+0000 7f51faffd640 1 -- 172.21.15.89:0/2277537921 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51fc003b80 con 0x7f520c06d420 2023-10-30T12:28:49.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.512+0000 7f51faffd640 1 -- 172.21.15.89:0/2277537921 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51fc003ce0 con 0x7f520c06d420 2023-10-30T12:28:49.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.512+0000 7f52109c5640 1 -- 172.21.15.89:0/2277537921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f520c0c6020 con 0x7f520c06d420 2023-10-30T12:28:49.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.512+0000 7f51faffd640 1 -- 172.21.15.89:0/2277537921 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51fc00aa30 con 0x7f520c06d420 2023-10-30T12:28:49.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.512+0000 7f52109c5640 1 -- 172.21.15.89:0/2277537921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f520c1eea70 con 0x7f520c06d420 2023-10-30T12:28:49.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.512+0000 7f52109c5640 1 -- 172.21.15.89:0/2277537921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f520c1ef0a0 con 0x7f520c06d420 2023-10-30T12:28:49.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.512+0000 7f51faffd640 1 -- 172.21.15.89:0/2277537921 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f51fc00ab90 con 0x7f520c06d420 2023-10-30T12:28:49.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.512+0000 7f51faffd640 1 --2- 172.21.15.89:0/2277537921 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f51ec0664a0 0x7f51ec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:49.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.516+0000 7f51faffd640 1 -- 172.21.15.89:0/2277537921 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f51fc083ca0 con 0x7f520c06d420 2023-10-30T12:28:49.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.516+0000 7f5209d9b640 1 --2- 172.21.15.89:0/2277537921 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f51ec0664a0 0x7f51ec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:49.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.516+0000 7f5209d9b640 1 --2- 172.21.15.89:0/2277537921 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f51ec0664a0 0x7f51ec068960 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f520c1e93b0 tx=0x7f5200007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:49.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.520+0000 7f51faffd640 1 -- 172.21.15.89:0/2277537921 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f51fc051260 con 0x7f520c06d420 2023-10-30T12:28:49.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:49.668+0000 7f52109c5640 1 -- 172.21.15.89:0/2277537921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f520c04e170 con 0x7f520c06d420 2023-10-30T12:28:50.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.336+0000 7f51faffd640 1 -- 172.21.15.89:0/2277537921 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v20) v1 ==== 170+0+0 (secure 0 0 0) 0x7f51fc054880 con 0x7f520c06d420 2023-10-30T12:28:50.343 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' created 2023-10-30T12:28:50.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.340+0000 7f52109c5640 1 -- 172.21.15.89:0/2277537921 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f51ec0664a0 msgr2=0x7f51ec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:50.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.340+0000 7f52109c5640 1 --2- 172.21.15.89:0/2277537921 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f51ec0664a0 0x7f51ec068960 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f520c1e93b0 tx=0x7f5200007d30 comp rx=0 tx=0).stop 2023-10-30T12:28:50.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.340+0000 7f52109c5640 1 -- 172.21.15.89:0/2277537921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f520c06d420 msgr2=0x7f520c158db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:50.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.340+0000 7f52109c5640 1 --2- 172.21.15.89:0/2277537921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f520c06d420 0x7f520c158db0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f51fc00c820 tx=0x7f51fc00cce0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.340+0000 7f52109c5640 1 -- 172.21.15.89:0/2277537921 shutdown_connections 2023-10-30T12:28:50.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.340+0000 7f52109c5640 1 --2- 172.21.15.89:0/2277537921 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f51ec0664a0 0x7f51ec068960 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.340+0000 7f52109c5640 1 --2- 172.21.15.89:0/2277537921 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f520c1539e0 0x7f520c0c56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.340+0000 7f52109c5640 1 --2- 172.21.15.89:0/2277537921 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f520c15a1a0 0x7f520c1ef250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.340+0000 7f52109c5640 1 --2- 172.21.15.89:0/2277537921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f520c06d420 0x7f520c158db0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.340+0000 7f52109c5640 1 -- 172.21.15.89:0/2277537921 >> 172.21.15.89:0/2277537921 conn(0x7f520c06c850 msgr2=0x7f520c0c80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:50.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.340+0000 7f52109c5640 1 -- 172.21.15.89:0/2277537921 shutdown_connections 2023-10-30T12:28:50.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.340+0000 7f52109c5640 1 -- 172.21.15.89:0/2277537921 wait complete. 2023-10-30T12:28:50.386 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-30T12:28:50.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.524+0000 7efd3b009640 1 Processor -- start 2023-10-30T12:28:50.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.524+0000 7efd3b009640 1 -- start start 2023-10-30T12:28:50.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.528+0000 7efd3b009640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7efd340c5810 0x7efd340c5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:50.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.528+0000 7efd3b009640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efd340c6130 0x7efd3406a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:50.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.528+0000 7efd3b009640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd340c6cb0 con 0x7efd3406f880 2023-10-30T12:28:50.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.528+0000 7efd3b009640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd34071d90 con 0x7efd340c6130 2023-10-30T12:28:50.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.528+0000 7efd3b009640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd341585d0 con 0x7efd340c5810 2023-10-30T12:28:50.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.528+0000 7efd2bfff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efd340c6130 0x7efd3406a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:50.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.528+0000 7efd395a6640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7efd3406f880 0x7efd3415ab80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55646/0 (socket says 172.21.15.89:55646) 2023-10-30T12:28:50.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.528+0000 7efd395a6640 1 -- 172.21.15.89:0/1094014894 learned_addr learned my addr 172.21.15.89:0/1094014894 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:28:50.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.528+0000 7efd38da5640 1 --2- 172.21.15.89:0/1094014894 >> v2:172.21.15.192:3300/0 conn(0x7efd340c5810 0x7efd340c5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:50.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.528+0000 7efd2bfff640 1 -- 172.21.15.89:0/1094014894 >> v2:172.21.15.192:3300/0 conn(0x7efd340c5810 msgr2=0x7efd340c5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:50.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.528+0000 7efd2bfff640 1 --2- 172.21.15.89:0/1094014894 >> v2:172.21.15.192:3300/0 conn(0x7efd340c5810 0x7efd340c5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.528+0000 7efd2bfff640 1 -- 172.21.15.89:0/1094014894 >> v1:172.21.15.89:6789/0 conn(0x7efd3406f880 legacy=0x7efd3415ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:50.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.528+0000 7efd2bfff640 1 -- 172.21.15.89:0/1094014894 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd340c6620 con 0x7efd340c6130 2023-10-30T12:28:50.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.528+0000 7efd2b7fe640 1 -- 172.21.15.89:0/1094014894 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2781515774 0 0) 0x7efd340c6cb0 con 0x7efd3406f880 2023-10-30T12:28:50.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.528+0000 7efd2bfff640 1 --2- 172.21.15.89:0/1094014894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd340c6130 0x7efd3406a270 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efd3406b380 tx=0x7efd2401c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34627397059c67ae server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:50.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.532+0000 7efd2b7fe640 1 -- 172.21.15.89:0/1094014894 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd240ab600 con 0x7efd340c6130 2023-10-30T12:28:50.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.532+0000 7efd2b7fe640 1 -- 172.21.15.89:0/1094014894 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd240a7050 con 0x7efd340c6130 2023-10-30T12:28:50.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.532+0000 7efd2b7fe640 1 -- 172.21.15.89:0/1094014894 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd240ab980 con 0x7efd340c6130 2023-10-30T12:28:50.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.532+0000 7efd3b009640 1 -- 172.21.15.89:0/1094014894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd340c6130 msgr2=0x7efd3406a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:50.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.532+0000 7efd3b009640 1 --2- 172.21.15.89:0/1094014894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd340c6130 0x7efd3406a270 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efd3406b380 tx=0x7efd2401c320 comp rx=0 tx=0).stop 2023-10-30T12:28:50.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.532+0000 7efd3b009640 1 -- 172.21.15.89:0/1094014894 shutdown_connections 2023-10-30T12:28:50.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.532+0000 7efd3b009640 1 --2- 172.21.15.89:0/1094014894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd340c6130 0x7efd3406a270 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.532+0000 7efd3b009640 1 --2- 172.21.15.89:0/1094014894 >> v2:172.21.15.192:3300/0 conn(0x7efd340c5810 0x7efd340c5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.532+0000 7efd3b009640 1 -- 172.21.15.89:0/1094014894 >> 172.21.15.89:0/1094014894 conn(0x7efd340c7910 msgr2=0x7efd340c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:50.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.532+0000 7efd3b009640 1 -- 172.21.15.89:0/1094014894 shutdown_connections 2023-10-30T12:28:50.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.532+0000 7efd3b009640 1 -- 172.21.15.89:0/1094014894 wait complete. 2023-10-30T12:28:50.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.532+0000 7efd3b009640 1 Processor -- start 2023-10-30T12:28:50.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.532+0000 7efd3b009640 1 -- start start 2023-10-30T12:28:50.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.532+0000 7efd3b009640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd3406f880 0x7efd341f6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:50.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.532+0000 7efd3b009640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd340c5810 0x7efd340c1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:50.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.532+0000 7efd3b009640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd340c6130 0x7efd340c4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:50.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.532+0000 7efd3b009640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd341f5620 con 0x7efd3406f880 2023-10-30T12:28:50.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.532+0000 7efd3b009640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd341f3080 con 0x7efd340c6130 2023-10-30T12:28:50.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.532+0000 7efd3b009640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd341f6570 con 0x7efd340c5810 2023-10-30T12:28:50.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.532+0000 7efd38da5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd3406f880 0x7efd341f6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:50.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.532+0000 7efd38da5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd3406f880 0x7efd341f6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55194/0 (socket says 172.21.15.89:55194) 2023-10-30T12:28:50.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.532+0000 7efd38da5640 1 -- 172.21.15.89:0/1527783879 learned_addr learned my addr 172.21.15.89:0/1527783879 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:50.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.536+0000 7efd2bfff640 1 --2- 172.21.15.89:0/1527783879 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd340c5810 0x7efd340c1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:50.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.536+0000 7efd395a6640 1 --2- 172.21.15.89:0/1527783879 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd340c6130 0x7efd340c4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:50.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.536+0000 7efd38da5640 1 -- 172.21.15.89:0/1527783879 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd340c5810 msgr2=0x7efd340c1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:50.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.536+0000 7efd38da5640 1 --2- 172.21.15.89:0/1527783879 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd340c5810 0x7efd340c1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.536+0000 7efd38da5640 1 -- 172.21.15.89:0/1527783879 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd340c6130 msgr2=0x7efd340c4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:50.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.536+0000 7efd38da5640 1 --2- 172.21.15.89:0/1527783879 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd340c6130 0x7efd340c4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.536+0000 7efd38da5640 1 -- 172.21.15.89:0/1527783879 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd24009420 con 0x7efd3406f880 2023-10-30T12:28:50.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.536+0000 7efd395a6640 1 --2- 172.21.15.89:0/1527783879 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd340c6130 0x7efd340c4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:50.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.536+0000 7efd38da5640 1 --2- 172.21.15.89:0/1527783879 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd3406f880 0x7efd341f6030 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efd3000d7d0 tx=0x7efd3000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:50.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.536+0000 7efd297fa640 1 -- 172.21.15.89:0/1527783879 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3000b660 con 0x7efd3406f880 2023-10-30T12:28:50.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.536+0000 7efd297fa640 1 -- 172.21.15.89:0/1527783879 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd3000b7c0 con 0x7efd3406f880 2023-10-30T12:28:50.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.536+0000 7efd297fa640 1 -- 172.21.15.89:0/1527783879 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3000bb70 con 0x7efd3406f880 2023-10-30T12:28:50.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.536+0000 7efd3b009640 1 -- 172.21.15.89:0/1527783879 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd341f5ba0 con 0x7efd3406f880 2023-10-30T12:28:50.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.536+0000 7efd3b009640 1 -- 172.21.15.89:0/1527783879 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd340ca4c0 con 0x7efd3406f880 2023-10-30T12:28:50.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.536+0000 7efd3b009640 1 -- 172.21.15.89:0/1527783879 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd340c75d0 con 0x7efd3406f880 2023-10-30T12:28:50.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.540+0000 7efd297fa640 1 -- 172.21.15.89:0/1527783879 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efd3000bcd0 con 0x7efd3406f880 2023-10-30T12:28:50.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.540+0000 7efd297fa640 1 --2- 172.21.15.89:0/1527783879 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7efd200664d0 0x7efd20068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:50.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.540+0000 7efd297fa640 1 -- 172.21.15.89:0/1527783879 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5609+0+0 (secure 0 0 0) 0x7efd30084cb0 con 0x7efd3406f880 2023-10-30T12:28:50.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.540+0000 7efd2bfff640 1 --2- 172.21.15.89:0/1527783879 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7efd200664d0 0x7efd20068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:50.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.540+0000 7efd2bfff640 1 --2- 172.21.15.89:0/1527783879 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7efd200664d0 0x7efd20068990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efd240ad000 tx=0x7efd24008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:50.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.544+0000 7efd297fa640 1 -- 172.21.15.89:0/1527783879 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efd30055c70 con 0x7efd3406f880 2023-10-30T12:28:50.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:50.724+0000 7efd3b009640 1 -- 172.21.15.89:0/1527783879 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7efd3404e170 con 0x7efd3406f880 2023-10-30T12:28:51.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.340+0000 7efd297fa640 1 -- 172.21.15.89:0/1527783879 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7efd30004650 con 0x7efd3406f880 2023-10-30T12:28:51.346 INFO:teuthology.orchestra.run.smithi089.stderr:new fs with metadata pool 2 and data pool 3 2023-10-30T12:28:51.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.344+0000 7efd3b009640 1 -- 172.21.15.89:0/1527783879 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7efd200664d0 msgr2=0x7efd20068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:51.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.344+0000 7efd3b009640 1 --2- 172.21.15.89:0/1527783879 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7efd200664d0 0x7efd20068990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efd240ad000 tx=0x7efd24008040 comp rx=0 tx=0).stop 2023-10-30T12:28:51.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.344+0000 7efd3b009640 1 -- 172.21.15.89:0/1527783879 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd3406f880 msgr2=0x7efd341f6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:51.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.344+0000 7efd3b009640 1 --2- 172.21.15.89:0/1527783879 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd3406f880 0x7efd341f6030 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efd3000d7d0 tx=0x7efd3000dc90 comp rx=0 tx=0).stop 2023-10-30T12:28:51.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.344+0000 7efd3b009640 1 -- 172.21.15.89:0/1527783879 shutdown_connections 2023-10-30T12:28:51.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.344+0000 7efd3b009640 1 --2- 172.21.15.89:0/1527783879 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7efd200664d0 0x7efd20068990 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.344+0000 7efd3b009640 1 --2- 172.21.15.89:0/1527783879 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd340c6130 0x7efd340c4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.344+0000 7efd3b009640 1 --2- 172.21.15.89:0/1527783879 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd340c5810 0x7efd340c1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.344+0000 7efd3b009640 1 --2- 172.21.15.89:0/1527783879 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd3406f880 0x7efd341f6030 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.344+0000 7efd3b009640 1 -- 172.21.15.89:0/1527783879 >> 172.21.15.89:0/1527783879 conn(0x7efd340c7910 msgr2=0x7efd3406d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:51.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.344+0000 7efd3b009640 1 -- 172.21.15.89:0/1527783879 shutdown_connections 2023-10-30T12:28:51.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.344+0000 7efd3b009640 1 -- 172.21.15.89:0/1527783879 wait complete. 2023-10-30T12:28:51.369 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:28:51.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.500+0000 7f5dc8054640 1 Processor -- start 2023-10-30T12:28:51.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.500+0000 7f5dc8054640 1 -- start start 2023-10-30T12:28:51.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.504+0000 7f5dc8054640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5dc006e470 0x7f5dc006ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:51.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.504+0000 7f5dc8054640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5dc00c8490 0x7f5dc00c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:51.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.504+0000 7f5dc8054640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc006afa0 con 0x7f5dc0153d70 2023-10-30T12:28:51.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.504+0000 7f5dc8054640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dc006d780 con 0x7f5dc006e470 2023-10-30T12:28:51.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.504+0000 7f5dc8054640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dc0154150 con 0x7f5dc00c8490 2023-10-30T12:28:51.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.504+0000 7f5dc55ef640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5dc006e470 0x7f5dc006ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:51.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.504+0000 7f5dc55ef640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5dc006e470 0x7f5dc006ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55196/0 (socket says 172.21.15.89:55196) 2023-10-30T12:28:51.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.504+0000 7f5dc55ef640 1 -- 172.21.15.89:0/4269860552 learned_addr learned my addr 172.21.15.89:0/4269860552 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:51.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.504+0000 7f5dc65f1640 1 --2- 172.21.15.89:0/4269860552 >> v2:172.21.15.192:3300/0 conn(0x7f5dc00c8490 0x7f5dc00c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:51.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.504+0000 7f5dc55ef640 1 -- 172.21.15.89:0/4269860552 >> v2:172.21.15.192:3300/0 conn(0x7f5dc00c8490 msgr2=0x7f5dc00c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:51.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.504+0000 7f5dc55ef640 1 --2- 172.21.15.89:0/4269860552 >> v2:172.21.15.192:3300/0 conn(0x7f5dc00c8490 0x7f5dc00c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.504+0000 7f5dc55ef640 1 -- 172.21.15.89:0/4269860552 >> v1:172.21.15.192:6789/0 conn(0x7f5dc0153d70 legacy=0x7f5dc00cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:51.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.504+0000 7f5dc55ef640 1 -- 172.21.15.89:0/4269860552 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc006e030 con 0x7f5dc006e470 2023-10-30T12:28:51.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.504+0000 7f5dc65f1640 1 --2- 172.21.15.89:0/4269860552 >> v2:172.21.15.192:3300/0 conn(0x7f5dc00c8490 0x7f5dc00c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:51.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.504+0000 7f5dc55ef640 1 --2- 172.21.15.89:0/4269860552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dc006e470 0x7f5dc006ca80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5dc00cddf0 tx=0x7f5db001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d222afa7fded2699 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:51.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.508+0000 7f5dc4dee640 1 -- 172.21.15.89:0/4269860552 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db00aa430 con 0x7f5dc006e470 2023-10-30T12:28:51.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.508+0000 7f5dc4dee640 1 -- 172.21.15.89:0/4269860552 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db00aa590 con 0x7f5dc006e470 2023-10-30T12:28:51.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.508+0000 7f5dc4dee640 1 -- 172.21.15.89:0/4269860552 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db00aa920 con 0x7f5dc006e470 2023-10-30T12:28:51.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.508+0000 7f5dc8054640 1 -- 172.21.15.89:0/4269860552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dc006e470 msgr2=0x7f5dc006ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:51.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.508+0000 7f5dc8054640 1 --2- 172.21.15.89:0/4269860552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dc006e470 0x7f5dc006ca80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5dc00cddf0 tx=0x7f5db001c4d0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.508+0000 7f5dc8054640 1 -- 172.21.15.89:0/4269860552 shutdown_connections 2023-10-30T12:28:51.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.508+0000 7f5dc8054640 1 --2- 172.21.15.89:0/4269860552 >> v2:172.21.15.192:3300/0 conn(0x7f5dc00c8490 0x7f5dc00c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.508+0000 7f5dc8054640 1 --2- 172.21.15.89:0/4269860552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dc006e470 0x7f5dc006ca80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.508+0000 7f5dc8054640 1 -- 172.21.15.89:0/4269860552 >> 172.21.15.89:0/4269860552 conn(0x7f5dc0071ea0 msgr2=0x7f5dc0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:51.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.508+0000 7f5dc8054640 1 -- 172.21.15.89:0/4269860552 shutdown_connections 2023-10-30T12:28:51.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.508+0000 7f5dc8054640 1 -- 172.21.15.89:0/4269860552 wait complete. 2023-10-30T12:28:51.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.508+0000 7f5dc8054640 1 Processor -- start 2023-10-30T12:28:51.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.508+0000 7f5dc8054640 1 -- start start 2023-10-30T12:28:51.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.508+0000 7f5dc8054640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dc006e470 0x7f5dc0147f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:51.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.508+0000 7f5dc8054640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5dc00c8490 0x7f5dc014f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:51.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.508+0000 7f5dc8054640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5dc0153d70 0x7f5dc01414a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:51.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.508+0000 7f5dc8054640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc0149660 con 0x7f5dc006e470 2023-10-30T12:28:51.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.508+0000 7f5dc8054640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc013db40 con 0x7f5dc00c8490 2023-10-30T12:28:51.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.508+0000 7f5dc8054640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc01638c0 con 0x7f5dc0153d70 2023-10-30T12:28:51.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.508+0000 7f5dc5df0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dc006e470 0x7f5dc0147f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:51.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.508+0000 7f5dc5df0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dc006e470 0x7f5dc0147f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55208/0 (socket says 172.21.15.89:55208) 2023-10-30T12:28:51.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.508+0000 7f5dc5df0640 1 -- 172.21.15.89:0/1691664345 learned_addr learned my addr 172.21.15.89:0/1691664345 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:51.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.508+0000 7f5dc55ef640 1 --2- 172.21.15.89:0/1691664345 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5dc00c8490 0x7f5dc014f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:51.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.512+0000 7f5dc65f1640 1 --2- 172.21.15.89:0/1691664345 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5dc0153d70 0x7f5dc01414a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:51.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.512+0000 7f5dc5df0640 1 -- 172.21.15.89:0/1691664345 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5dc0153d70 msgr2=0x7f5dc01414a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:51.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.512+0000 7f5dc5df0640 1 --2- 172.21.15.89:0/1691664345 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5dc0153d70 0x7f5dc01414a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.512+0000 7f5dc5df0640 1 -- 172.21.15.89:0/1691664345 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5dc00c8490 msgr2=0x7f5dc014f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:51.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.512+0000 7f5dc5df0640 1 --2- 172.21.15.89:0/1691664345 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5dc00c8490 0x7f5dc014f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.512+0000 7f5dc5df0640 1 -- 172.21.15.89:0/1691664345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db00094d0 con 0x7f5dc006e470 2023-10-30T12:28:51.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.512+0000 7f5dc5df0640 1 --2- 172.21.15.89:0/1691664345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dc006e470 0x7f5dc0147f80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc00c500 tx=0x7f5dbc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:51.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.512+0000 7f5db67fc640 1 -- 172.21.15.89:0/1691664345 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc005220 con 0x7f5dc006e470 2023-10-30T12:28:51.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.512+0000 7f5db67fc640 1 -- 172.21.15.89:0/1691664345 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dbc005380 con 0x7f5dc006e470 2023-10-30T12:28:51.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.512+0000 7f5dc8054640 1 -- 172.21.15.89:0/1691664345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc0145b20 con 0x7f5dc006e470 2023-10-30T12:28:51.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.512+0000 7f5db67fc640 1 -- 172.21.15.89:0/1691664345 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc007270 con 0x7f5dc006e470 2023-10-30T12:28:51.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.512+0000 7f5dc8054640 1 -- 172.21.15.89:0/1691664345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc0145f60 con 0x7f5dc006e470 2023-10-30T12:28:51.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.512+0000 7f5dc8054640 1 -- 172.21.15.89:0/1691664345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc0145db0 con 0x7f5dc006e470 2023-10-30T12:28:51.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.512+0000 7f5db67fc640 1 -- 172.21.15.89:0/1691664345 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5dbc0073d0 con 0x7f5dc006e470 2023-10-30T12:28:51.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.512+0000 7f5db67fc640 1 --2- 172.21.15.89:0/1691664345 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5da40664a0 0x7f5da4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:51.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.512+0000 7f5db67fc640 1 -- 172.21.15.89:0/1691664345 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5dbc083830 con 0x7f5dc006e470 2023-10-30T12:28:51.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.512+0000 7f5dc55ef640 1 --2- 172.21.15.89:0/1691664345 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5da40664a0 0x7f5da4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:51.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.516+0000 7f5dc55ef640 1 --2- 172.21.15.89:0/1691664345 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5da40664a0 0x7f5da4068960 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5db00ae000 tx=0x7f5db00a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:51.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.516+0000 7f5db67fc640 1 -- 172.21.15.89:0/1691664345 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5dbc050c20 con 0x7f5dc006e470 2023-10-30T12:28:51.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.668+0000 7f5dc8054640 1 -- 172.21.15.89:0/1691664345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5dc004e170 con 0x7f5dc006e470 2023-10-30T12:28:51.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.668+0000 7f5db67fc640 1 -- 172.21.15.89:0/1691664345 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16518 (secure 0 0 0) 0x7f5dbc054240 con 0x7f5dc006e470 2023-10-30T12:28:51.675 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:28:51.675 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":21,"fsid":"401641be-db6e-4c16-b3ac-fd84a86c4de4","created":"2023-10-30T12:28:26.198532+0000","modified":"2023-10-30T12:28:50.728893+0000","last_up_change":"2023-10-30T12:28:37.212205+0000","last_in_change":"2023-10-30T12:28:28.517734+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:39.004943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T12:28:48.935909+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T12:28:49.716098+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72595c54-fd6e-4a5a-bb14-ff27e9377b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6801","nonce":1673955622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6803","nonce":1673955622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6807","nonce":1673955622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6805","nonce":1673955622}]},"public_addr":"172.21.15.89:6801/1673955622","cluster_addr":"172.21.15.89:6803/1673955622","heartbeat_back_addr":"172.21.15.89:6807/1673955622","heartbeat_front_addr":"172.21.15.89:6805/1673955622","state":["exists","up"]},{"osd":1,"uuid":"0305670b-d7c4-4193-bec2-8b7351ff590a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6817","nonce":3340572039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6819","nonce":3340572039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6823","nonce":3340572039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6821","nonce":3340572039}]},"public_addr":"172.21.15.89:6817/3340572039","cluster_addr":"172.21.15.89:6819/3340572039","heartbeat_back_addr":"172.21.15.89:6823/3340572039","heartbeat_front_addr":"172.21.15.89:6821/3340572039","state":["exists","up"]},{"osd":2,"uuid":"4bab5426-a3c3-46b0-91fc-546c254006c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6809","nonce":1929940799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6811","nonce":1929940799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6815","nonce":1929940799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6813","nonce":1929940799}]},"public_addr":"172.21.15.89:6809/1929940799","cluster_addr":"172.21.15.89:6811/1929940799","heartbeat_back_addr":"172.21.15.89:6815/1929940799","heartbeat_front_addr":"172.21.15.89:6813/1929940799","state":["exists","up"]},{"osd":3,"uuid":"7d47aea8-27cd-42be-8591-d63ad68c90e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6825","nonce":345087939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6827","nonce":345087939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6831","nonce":345087939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6829","nonce":345087939}]},"public_addr":"172.21.15.89:6825/345087939","cluster_addr":"172.21.15.89:6827/345087939","heartbeat_back_addr":"172.21.15.89:6831/345087939","heartbeat_front_addr":"172.21.15.89:6829/345087939","state":["exists","up"]},{"osd":4,"uuid":"2e2ae8f8-d097-4bba-9448-95047fbfacad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6801","nonce":2138125545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6803","nonce":2138125545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6809","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6811","nonce":2138125545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6805","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6807","nonce":2138125545}]},"public_addr":"172.21.15.192:6801/2138125545","cluster_addr":"172.21.15.192:6803/2138125545","heartbeat_back_addr":"172.21.15.192:6811/2138125545","heartbeat_front_addr":"172.21.15.192:6807/2138125545","state":["exists","up"]},{"osd":5,"uuid":"49be5ce5-12f4-4c26-8d39-864b6ff67313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6806","nonce":2360218671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6810","nonce":2360218671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6815","nonce":2360218671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6813","nonce":2360218671}]},"public_addr":"172.21.15.192:6806/2360218671","cluster_addr":"172.21.15.192:6810/2360218671","heartbeat_back_addr":"172.21.15.192:6815/2360218671","heartbeat_front_addr":"172.21.15.192:6813/2360218671","state":["exists","up"]},{"osd":6,"uuid":"48623452-d456-4ea2-a0de-5fa4b37e6d10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6825","nonce":3707671287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6827","nonce":3707671287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6831","nonce":3707671287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6829","nonce":3707671287}]},"public_addr":"172.21.15.192:6825/3707671287","cluster_addr":"172.21.15.192:6827/3707671287","heartbeat_back_addr":"172.21.15.192:6831/3707671287","heartbeat_front_addr":"172.21.15.192:6829/3707671287","state":["exists","up"]},{"osd":7,"uuid":"ff0a4384-54f8-43ec-a3ce-9c8ae3bf8a94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6817","nonce":3111945871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6819","nonce":3111945871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6823","nonce":3111945871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6821","nonce":3111945871}]},"public_addr":"172.21.15.192:6817/3111945871","cluster_addr":"172.21.15.192:6819/3111945871","heartbeat_back_addr":"172.21.15.192:6823/3111945871","heartbeat_front_addr":"172.21.15.192:6821/3111945871","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.183065+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.248650+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.194576+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.397483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.043648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.186624+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.159674+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.100851+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:28:51.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.672+0000 7f5dc8054640 1 -- 172.21.15.89:0/1691664345 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5da40664a0 msgr2=0x7f5da4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:51.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.672+0000 7f5dc8054640 1 --2- 172.21.15.89:0/1691664345 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5da40664a0 0x7f5da4068960 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5db00ae000 tx=0x7f5db00a7000 comp rx=0 tx=0).stop 2023-10-30T12:28:51.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.672+0000 7f5dc8054640 1 -- 172.21.15.89:0/1691664345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dc006e470 msgr2=0x7f5dc0147f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:51.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.672+0000 7f5dc8054640 1 --2- 172.21.15.89:0/1691664345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dc006e470 0x7f5dc0147f80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc00c500 tx=0x7f5dbc00c9c0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.672+0000 7f5dc8054640 1 -- 172.21.15.89:0/1691664345 shutdown_connections 2023-10-30T12:28:51.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.672+0000 7f5dc8054640 1 --2- 172.21.15.89:0/1691664345 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5da40664a0 0x7f5da4068960 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.672+0000 7f5dc8054640 1 --2- 172.21.15.89:0/1691664345 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5dc00c8490 0x7f5dc014f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.672+0000 7f5dc8054640 1 --2- 172.21.15.89:0/1691664345 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5dc0153d70 0x7f5dc01414a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.672+0000 7f5dc8054640 1 --2- 172.21.15.89:0/1691664345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5dc006e470 0x7f5dc0147f80 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.672+0000 7f5dc8054640 1 -- 172.21.15.89:0/1691664345 >> 172.21.15.89:0/1691664345 conn(0x7f5dc0071ea0 msgr2=0x7f5dc0150af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:51.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.672+0000 7f5dc8054640 1 -- 172.21.15.89:0/1691664345 shutdown_connections 2023-10-30T12:28:51.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.672+0000 7f5dc8054640 1 -- 172.21.15.89:0/1691664345 wait complete. 2023-10-30T12:28:51.696 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:28:51.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.828+0000 7fa1357c4640 1 Processor -- start 2023-10-30T12:28:51.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.828+0000 7fa1357c4640 1 -- start start 2023-10-30T12:28:51.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.828+0000 7fa1357c4640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa1301538a0 0x7fa13015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:51.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.828+0000 7fa1357c4640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa1300ca0e0 0x7fa13015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:51.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.828+0000 7fa1357c4640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa1300cf160 0x7fa1300cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:51.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.828+0000 7fa1357c4640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1300cb800 con 0x7fa1300cf160 2023-10-30T12:28:51.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.828+0000 7fa1357c4640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa13015dc30 con 0x7fa1300ca0e0 2023-10-30T12:28:51.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.828+0000 7fa1357c4640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1300cb330 con 0x7fa1301538a0 2023-10-30T12:28:51.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.828+0000 7fa12f7fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa1300cf160 0x7fa1300cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:51.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.828+0000 7fa12f7fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa1300cf160 0x7fa1300cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55214/0 (socket says 172.21.15.89:55214) 2023-10-30T12:28:51.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.828+0000 7fa12f7fe640 1 -- 172.21.15.89:0/2239983464 learned_addr learned my addr 172.21.15.89:0/2239983464 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:51.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.828+0000 7fa12effd640 1 --2- 172.21.15.89:0/2239983464 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa1301538a0 0x7fa13015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:51.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.828+0000 7fa1267fc640 1 --2- 172.21.15.89:0/2239983464 >> v2:172.21.15.192:3300/0 conn(0x7fa1300ca0e0 0x7fa13015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:51.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.828+0000 7fa12f7fe640 1 -- 172.21.15.89:0/2239983464 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa1301538a0 msgr2=0x7fa13015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:51.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.828+0000 7fa12f7fe640 1 --2- 172.21.15.89:0/2239983464 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa1301538a0 0x7fa13015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.828+0000 7fa12f7fe640 1 -- 172.21.15.89:0/2239983464 >> v2:172.21.15.192:3300/0 conn(0x7fa1300ca0e0 msgr2=0x7fa13015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:51.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.828+0000 7fa12f7fe640 1 --2- 172.21.15.89:0/2239983464 >> v2:172.21.15.192:3300/0 conn(0x7fa1300ca0e0 0x7fa13015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.828+0000 7fa12f7fe640 1 -- 172.21.15.89:0/2239983464 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1300cad70 con 0x7fa1300cf160 2023-10-30T12:28:51.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.828+0000 7fa12effd640 1 --2- 172.21.15.89:0/2239983464 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa1301538a0 0x7fa13015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:51.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.832+0000 7fa12f7fe640 1 --2- 172.21.15.89:0/2239983464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1300cf160 0x7fa1300cf590 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa1300ccd10 tx=0x7fa12801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e73f8f03de91e0d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:51.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.832+0000 7fa12e7fc640 1 -- 172.21.15.89:0/2239983464 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1280af400 con 0x7fa1300cf160 2023-10-30T12:28:51.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.832+0000 7fa12e7fc640 1 -- 172.21.15.89:0/2239983464 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1280ab050 con 0x7fa1300cf160 2023-10-30T12:28:51.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.832+0000 7fa12e7fc640 1 -- 172.21.15.89:0/2239983464 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1280af780 con 0x7fa1300cf160 2023-10-30T12:28:51.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.832+0000 7fa1357c4640 1 -- 172.21.15.89:0/2239983464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1300cf160 msgr2=0x7fa1300cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:51.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.832+0000 7fa1357c4640 1 --2- 172.21.15.89:0/2239983464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1300cf160 0x7fa1300cf590 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa1300ccd10 tx=0x7fa12801f1c0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.832+0000 7fa1357c4640 1 -- 172.21.15.89:0/2239983464 shutdown_connections 2023-10-30T12:28:51.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.832+0000 7fa1357c4640 1 --2- 172.21.15.89:0/2239983464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1300cf160 0x7fa1300cf590 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.832+0000 7fa1357c4640 1 --2- 172.21.15.89:0/2239983464 >> v2:172.21.15.192:3300/0 conn(0x7fa1300ca0e0 0x7fa13015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.832+0000 7fa1357c4640 1 --2- 172.21.15.89:0/2239983464 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa1301538a0 0x7fa13015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.832+0000 7fa1357c4640 1 -- 172.21.15.89:0/2239983464 >> 172.21.15.89:0/2239983464 conn(0x7fa13006bfe0 msgr2=0x7fa13006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:51.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.832+0000 7fa1357c4640 1 -- 172.21.15.89:0/2239983464 shutdown_connections 2023-10-30T12:28:51.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.832+0000 7fa1357c4640 1 -- 172.21.15.89:0/2239983464 wait complete. 2023-10-30T12:28:51.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.836+0000 7fa1357c4640 1 Processor -- start 2023-10-30T12:28:51.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.836+0000 7fa1357c4640 1 -- start start 2023-10-30T12:28:51.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.836+0000 7fa1357c4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1300ca0e0 0x7fa1301efb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:51.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.836+0000 7fa1357c4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1300cf160 0x7fa1301e5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:51.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.836+0000 7fa1357c4640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa1301538a0 0x7fa1301e2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:51.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.836+0000 7fa1357c4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa13015afa0 con 0x7fa1300ca0e0 2023-10-30T12:28:51.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.836+0000 7fa1357c4640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa13015a9e0 con 0x7fa1300cf160 2023-10-30T12:28:51.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.836+0000 7fa1357c4640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa13015acc0 con 0x7fa1301538a0 2023-10-30T12:28:51.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.836+0000 7fa12effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1300ca0e0 0x7fa1301efb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:51.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.836+0000 7fa12effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1300ca0e0 0x7fa1301efb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55228/0 (socket says 172.21.15.89:55228) 2023-10-30T12:28:51.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.836+0000 7fa12effd640 1 -- 172.21.15.89:0/1789180857 learned_addr learned my addr 172.21.15.89:0/1789180857 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:51.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.836+0000 7fa1267fc640 1 --2- 172.21.15.89:0/1789180857 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1300cf160 0x7fa1301e5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:51.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.836+0000 7fa12f7fe640 1 --2- 172.21.15.89:0/1789180857 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa1301538a0 0x7fa1301e2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:51.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.836+0000 7fa12effd640 1 -- 172.21.15.89:0/1789180857 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa1301538a0 msgr2=0x7fa1301e2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:51.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.836+0000 7fa12effd640 1 --2- 172.21.15.89:0/1789180857 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa1301538a0 0x7fa1301e2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.836+0000 7fa12effd640 1 -- 172.21.15.89:0/1789180857 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1300cf160 msgr2=0x7fa1301e5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:51.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.836+0000 7fa12effd640 1 --2- 172.21.15.89:0/1789180857 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1300cf160 0x7fa1301e5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.836+0000 7fa12effd640 1 -- 172.21.15.89:0/1789180857 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa128009b30 con 0x7fa1300ca0e0 2023-10-30T12:28:51.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.836+0000 7fa1267fc640 1 --2- 172.21.15.89:0/1789180857 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1300cf160 0x7fa1301e5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:51.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.836+0000 7fa12effd640 1 --2- 172.21.15.89:0/1789180857 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1300ca0e0 0x7fa1301efb60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa12000d800 tx=0x7fa12000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:51.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.836+0000 7fa127fff640 1 -- 172.21.15.89:0/1789180857 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1200096c0 con 0x7fa1300ca0e0 2023-10-30T12:28:51.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.836+0000 7fa127fff640 1 -- 172.21.15.89:0/1789180857 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa120009820 con 0x7fa1300ca0e0 2023-10-30T12:28:51.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.836+0000 7fa1357c4640 1 -- 172.21.15.89:0/1789180857 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1301ef080 con 0x7fa1300ca0e0 2023-10-30T12:28:51.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.836+0000 7fa127fff640 1 -- 172.21.15.89:0/1789180857 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa120009bd0 con 0x7fa1300ca0e0 2023-10-30T12:28:51.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.836+0000 7fa1357c4640 1 -- 172.21.15.89:0/1789180857 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1301ef640 con 0x7fa1300ca0e0 2023-10-30T12:28:51.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.840+0000 7fa1357c4640 1 -- 172.21.15.89:0/1789180857 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1300b2a50 con 0x7fa1300ca0e0 2023-10-30T12:28:51.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.840+0000 7fa127fff640 1 -- 172.21.15.89:0/1789180857 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa120009d30 con 0x7fa1300ca0e0 2023-10-30T12:28:51.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.840+0000 7fa127fff640 1 --2- 172.21.15.89:0/1789180857 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa1100664a0 0x7fa110068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:51.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.840+0000 7fa127fff640 1 -- 172.21.15.89:0/1789180857 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa120084590 con 0x7fa1300ca0e0 2023-10-30T12:28:51.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.840+0000 7fa1267fc640 1 --2- 172.21.15.89:0/1789180857 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa1100664a0 0x7fa110068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:51.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.840+0000 7fa1267fc640 1 --2- 172.21.15.89:0/1789180857 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa1100664a0 0x7fa110068960 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa118004660 tx=0x7fa118008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:51.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.844+0000 7fa127fff640 1 -- 172.21.15.89:0/1789180857 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa120051a00 con 0x7fa1300ca0e0 2023-10-30T12:28:52.002 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.996+0000 7fa1357c4640 1 -- 172.21.15.89:0/1789180857 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa130159830 con 0x7fa1300ca0e0 2023-10-30T12:28:52.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:51.996+0000 7fa127fff640 1 -- 172.21.15.89:0/1789180857 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16505 (secure 0 0 0) 0x7fa120055020 con 0x7fa1300ca0e0 2023-10-30T12:28:52.004 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:28:52.004 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":22,"fsid":"401641be-db6e-4c16-b3ac-fd84a86c4de4","created":"2023-10-30T12:28:26.198532+0000","modified":"2023-10-30T12:28:51.918285+0000","last_up_change":"2023-10-30T12:28:37.212205+0000","last_in_change":"2023-10-30T12:28:28.517734+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:39.004943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T12:28:48.935909+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T12:28:49.716098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72595c54-fd6e-4a5a-bb14-ff27e9377b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6801","nonce":1673955622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6803","nonce":1673955622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6807","nonce":1673955622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6805","nonce":1673955622}]},"public_addr":"172.21.15.89:6801/1673955622","cluster_addr":"172.21.15.89:6803/1673955622","heartbeat_back_addr":"172.21.15.89:6807/1673955622","heartbeat_front_addr":"172.21.15.89:6805/1673955622","state":["exists","up"]},{"osd":1,"uuid":"0305670b-d7c4-4193-bec2-8b7351ff590a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6817","nonce":3340572039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6819","nonce":3340572039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6823","nonce":3340572039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6821","nonce":3340572039}]},"public_addr":"172.21.15.89:6817/3340572039","cluster_addr":"172.21.15.89:6819/3340572039","heartbeat_back_addr":"172.21.15.89:6823/3340572039","heartbeat_front_addr":"172.21.15.89:6821/3340572039","state":["exists","up"]},{"osd":2,"uuid":"4bab5426-a3c3-46b0-91fc-546c254006c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6809","nonce":1929940799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6811","nonce":1929940799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6815","nonce":1929940799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6813","nonce":1929940799}]},"public_addr":"172.21.15.89:6809/1929940799","cluster_addr":"172.21.15.89:6811/1929940799","heartbeat_back_addr":"172.21.15.89:6815/1929940799","heartbeat_front_addr":"172.21.15.89:6813/1929940799","state":["exists","up"]},{"osd":3,"uuid":"7d47aea8-27cd-42be-8591-d63ad68c90e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6825","nonce":345087939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6827","nonce":345087939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6831","nonce":345087939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6829","nonce":345087939}]},"public_addr":"172.21.15.89:6825/345087939","cluster_addr":"172.21.15.89:6827/345087939","heartbeat_back_addr":"172.21.15.89:6831/345087939","heartbeat_front_addr":"172.21.15.89:6829/345087939","state":["exists","up"]},{"osd":4,"uuid":"2e2ae8f8-d097-4bba-9448-95047fbfacad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6801","nonce":2138125545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6803","nonce":2138125545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6809","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6811","nonce":2138125545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6805","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6807","nonce":2138125545}]},"public_addr":"172.21.15.192:6801/2138125545","cluster_addr":"172.21.15.192:6803/2138125545","heartbeat_back_addr":"172.21.15.192:6811/2138125545","heartbeat_front_addr":"172.21.15.192:6807/2138125545","state":["exists","up"]},{"osd":5,"uuid":"49be5ce5-12f4-4c26-8d39-864b6ff67313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6806","nonce":2360218671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6810","nonce":2360218671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6815","nonce":2360218671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6813","nonce":2360218671}]},"public_addr":"172.21.15.192:6806/2360218671","cluster_addr":"172.21.15.192:6810/2360218671","heartbeat_back_addr":"172.21.15.192:6815/2360218671","heartbeat_front_addr":"172.21.15.192:6813/2360218671","state":["exists","up"]},{"osd":6,"uuid":"48623452-d456-4ea2-a0de-5fa4b37e6d10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6825","nonce":3707671287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6827","nonce":3707671287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6831","nonce":3707671287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6829","nonce":3707671287}]},"public_addr":"172.21.15.192:6825/3707671287","cluster_addr":"172.21.15.192:6827/3707671287","heartbeat_back_addr":"172.21.15.192:6831/3707671287","heartbeat_front_addr":"172.21.15.192:6829/3707671287","state":["exists","up"]},{"osd":7,"uuid":"ff0a4384-54f8-43ec-a3ce-9c8ae3bf8a94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6817","nonce":3111945871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6819","nonce":3111945871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6823","nonce":3111945871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6821","nonce":3111945871}]},"public_addr":"172.21.15.192:6817/3111945871","cluster_addr":"172.21.15.192:6819/3111945871","heartbeat_back_addr":"172.21.15.192:6823/3111945871","heartbeat_front_addr":"172.21.15.192:6821/3111945871","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.183065+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.248650+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.194576+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.397483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.043648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.186624+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.159674+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.100851+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:28:52.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.000+0000 7fa1357c4640 1 -- 172.21.15.89:0/1789180857 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa1100664a0 msgr2=0x7fa110068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.000+0000 7fa1357c4640 1 --2- 172.21.15.89:0/1789180857 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa1100664a0 0x7fa110068960 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa118004660 tx=0x7fa118008040 comp rx=0 tx=0).stop 2023-10-30T12:28:52.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.000+0000 7fa1357c4640 1 -- 172.21.15.89:0/1789180857 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1300ca0e0 msgr2=0x7fa1301efb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.000+0000 7fa1357c4640 1 --2- 172.21.15.89:0/1789180857 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1300ca0e0 0x7fa1301efb60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa12000d800 tx=0x7fa12000dcc0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.000+0000 7fa1357c4640 1 -- 172.21.15.89:0/1789180857 shutdown_connections 2023-10-30T12:28:52.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.000+0000 7fa1357c4640 1 --2- 172.21.15.89:0/1789180857 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa1100664a0 0x7fa110068960 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.000+0000 7fa1357c4640 1 --2- 172.21.15.89:0/1789180857 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1300cf160 0x7fa1301e5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.000+0000 7fa1357c4640 1 --2- 172.21.15.89:0/1789180857 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa1301538a0 0x7fa1301e2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.000+0000 7fa1357c4640 1 --2- 172.21.15.89:0/1789180857 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1300ca0e0 0x7fa1301efb60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.000+0000 7fa1357c4640 1 -- 172.21.15.89:0/1789180857 >> 172.21.15.89:0/1789180857 conn(0x7fa13006bfe0 msgr2=0x7fa1301526f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:52.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.000+0000 7fa1357c4640 1 -- 172.21.15.89:0/1789180857 shutdown_connections 2023-10-30T12:28:52.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.000+0000 7fa1357c4640 1 -- 172.21.15.89:0/1789180857 wait complete. 2023-10-30T12:28:52.024 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-30T12:28:52.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.160+0000 7f9ec0479640 1 Processor -- start 2023-10-30T12:28:52.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.160+0000 7f9ec0479640 1 -- start start 2023-10-30T12:28:52.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.160+0000 7f9ec0479640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9eb8159930 0x7f9eb80cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.160+0000 7f9ec0479640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9eb80cf6e0 0x7f9eb815d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.160+0000 7f9ec0479640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eb815a920 con 0x7f9eb8153980 2023-10-30T12:28:52.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.160+0000 7f9ec0479640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eb80cb930 con 0x7f9eb80cf6e0 2023-10-30T12:28:52.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.160+0000 7f9ec0479640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb8159dc0 con 0x7f9eb8159930 2023-10-30T12:28:52.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.160+0000 7f9ebe215640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9eb8153980 0x7f9eb80caca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55654/0 (socket says 172.21.15.89:55654) 2023-10-30T12:28:52.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.160+0000 7f9ebe215640 1 -- 172.21.15.89:0/2386131510 learned_addr learned my addr 172.21.15.89:0/2386131510 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:28:52.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.160+0000 7f9ebea16640 1 --2- 172.21.15.89:0/2386131510 >> v2:172.21.15.89:3300/0 conn(0x7f9eb80cf6e0 0x7f9eb815d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.160+0000 7f9ebda14640 1 --2- 172.21.15.89:0/2386131510 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9eb8159930 0x7f9eb80cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.160+0000 7f9ebd213640 1 -- 172.21.15.89:0/2386131510 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1384629249 0 0) 0x7f9eb815a920 con 0x7f9eb8153980 2023-10-30T12:28:52.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.160+0000 7f9ebd213640 1 -- 172.21.15.89:0/2386131510 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ea4003530 con 0x7f9eb8153980 2023-10-30T12:28:52.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.164+0000 7f9ebea16640 1 -- 172.21.15.89:0/2386131510 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9eb8159930 msgr2=0x7f9eb80cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:52.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.164+0000 7f9ebea16640 1 --2- 172.21.15.89:0/2386131510 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9eb8159930 0x7f9eb80cf1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.164+0000 7f9ebea16640 1 -- 172.21.15.89:0/2386131510 >> v1:172.21.15.89:6789/0 conn(0x7f9eb8153980 legacy=0x7f9eb80caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.164+0000 7f9ebea16640 1 -- 172.21.15.89:0/2386131510 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb815a080 con 0x7f9eb80cf6e0 2023-10-30T12:28:52.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.164+0000 7f9ebea16640 1 --2- 172.21.15.89:0/2386131510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eb80cf6e0 0x7f9eb815d570 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0009d90 tx=0x7f9eb001c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79e88483b9079f73 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:52.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.164+0000 7f9ebd213640 1 -- 172.21.15.89:0/2386131510 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb00a58f0 con 0x7f9eb80cf6e0 2023-10-30T12:28:52.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.164+0000 7f9ebd213640 1 -- 172.21.15.89:0/2386131510 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb00a5a50 con 0x7f9eb80cf6e0 2023-10-30T12:28:52.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.164+0000 7f9ebd213640 1 -- 172.21.15.89:0/2386131510 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb00a5de0 con 0x7f9eb80cf6e0 2023-10-30T12:28:52.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.164+0000 7f9ec0479640 1 -- 172.21.15.89:0/2386131510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eb80cf6e0 msgr2=0x7f9eb815d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.164+0000 7f9ec0479640 1 --2- 172.21.15.89:0/2386131510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eb80cf6e0 0x7f9eb815d570 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0009d90 tx=0x7f9eb001c920 comp rx=0 tx=0).stop 2023-10-30T12:28:52.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.164+0000 7f9ec0479640 1 -- 172.21.15.89:0/2386131510 shutdown_connections 2023-10-30T12:28:52.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.164+0000 7f9ec0479640 1 --2- 172.21.15.89:0/2386131510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eb80cf6e0 0x7f9eb815d570 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.164+0000 7f9ec0479640 1 --2- 172.21.15.89:0/2386131510 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9eb8159930 0x7f9eb80cf1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.164+0000 7f9ec0479640 1 -- 172.21.15.89:0/2386131510 >> 172.21.15.89:0/2386131510 conn(0x7f9eb806c0f0 msgr2=0x7f9eb806c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:52.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.164+0000 7f9ec0479640 1 -- 172.21.15.89:0/2386131510 shutdown_connections 2023-10-30T12:28:52.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.164+0000 7f9ec0479640 1 -- 172.21.15.89:0/2386131510 wait complete. 2023-10-30T12:28:52.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.168+0000 7f9ec0479640 1 Processor -- start 2023-10-30T12:28:52.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.168+0000 7f9ec0479640 1 -- start start 2023-10-30T12:28:52.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.168+0000 7f9ec0479640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9eb80cf6e0 0x7f9eb815aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.168+0000 7f9ec0479640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9eb8153980 0x7f9eb8148b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.168+0000 7f9ec0479640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eb8156f30 0x7f9eb81447e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.168+0000 7f9ec0479640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb81f1ac0 con 0x7f9eb8156f30 2023-10-30T12:28:52.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.168+0000 7f9ec0479640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb80b2c80 con 0x7f9eb80cf6e0 2023-10-30T12:28:52.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.168+0000 7f9ec0479640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb80b2f60 con 0x7f9eb8153980 2023-10-30T12:28:52.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.168+0000 7f9ebea16640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eb8156f30 0x7f9eb81447e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.168+0000 7f9ebea16640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eb8156f30 0x7f9eb81447e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55244/0 (socket says 172.21.15.89:55244) 2023-10-30T12:28:52.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.168+0000 7f9ebea16640 1 -- 172.21.15.89:0/4036494493 learned_addr learned my addr 172.21.15.89:0/4036494493 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:52.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.168+0000 7f9ebda14640 1 --2- 172.21.15.89:0/4036494493 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9eb8153980 0x7f9eb8148b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.168+0000 7f9ebe215640 1 --2- 172.21.15.89:0/4036494493 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9eb80cf6e0 0x7f9eb815aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.168+0000 7f9ebea16640 1 -- 172.21.15.89:0/4036494493 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9eb8153980 msgr2=0x7f9eb8148b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.168+0000 7f9ebea16640 1 --2- 172.21.15.89:0/4036494493 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9eb8153980 0x7f9eb8148b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.168+0000 7f9ebea16640 1 -- 172.21.15.89:0/4036494493 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9eb80cf6e0 msgr2=0x7f9eb815aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.168+0000 7f9ebea16640 1 --2- 172.21.15.89:0/4036494493 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9eb80cf6e0 0x7f9eb815aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.168+0000 7f9ebea16640 1 -- 172.21.15.89:0/4036494493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb0009a70 con 0x7f9eb8156f30 2023-10-30T12:28:52.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.168+0000 7f9ebda14640 1 --2- 172.21.15.89:0/4036494493 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9eb8153980 0x7f9eb8148b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:52.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.168+0000 7f9ebea16640 1 --2- 172.21.15.89:0/4036494493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eb8156f30 0x7f9eb81447e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9eb00b0040 tx=0x7f9eb00196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:52.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.168+0000 7f9eaeffd640 1 -- 172.21.15.89:0/4036494493 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb00ad640 con 0x7f9eb8156f30 2023-10-30T12:28:52.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.168+0000 7f9eaeffd640 1 -- 172.21.15.89:0/4036494493 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb00ad7a0 con 0x7f9eb8156f30 2023-10-30T12:28:52.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.168+0000 7f9eaeffd640 1 -- 172.21.15.89:0/4036494493 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb00adb70 con 0x7f9eb8156f30 2023-10-30T12:28:52.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.168+0000 7f9ec0479640 1 -- 172.21.15.89:0/4036494493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eb80b3240 con 0x7f9eb8156f30 2023-10-30T12:28:52.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.168+0000 7f9ec0479640 1 -- 172.21.15.89:0/4036494493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eb815b030 con 0x7f9eb8156f30 2023-10-30T12:28:52.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.172+0000 7f9ec0479640 1 -- 172.21.15.89:0/4036494493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eb804e170 con 0x7f9eb8156f30 2023-10-30T12:28:52.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.172+0000 7f9eaeffd640 1 -- 172.21.15.89:0/4036494493 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9eb00b4050 con 0x7f9eb8156f30 2023-10-30T12:28:52.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.172+0000 7f9eaeffd640 1 --2- 172.21.15.89:0/4036494493 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9e9c0664d0 0x7f9e9c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.172+0000 7f9eaeffd640 1 -- 172.21.15.89:0/4036494493 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9eb011ec60 con 0x7f9eb8156f30 2023-10-30T12:28:52.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.172+0000 7f9ebe215640 1 --2- 172.21.15.89:0/4036494493 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9e9c0664d0 0x7f9e9c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.172+0000 7f9ebe215640 1 --2- 172.21.15.89:0/4036494493 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9e9c0664d0 0x7f9e9c068990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9eb40011f0 tx=0x7f9eb4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:52.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.176+0000 7f9eaeffd640 1 -- 172.21.15.89:0/4036494493 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9eb00ec1f0 con 0x7f9eb8156f30 2023-10-30T12:28:52.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:52.360+0000 7f9ec0479640 1 -- 172.21.15.89:0/4036494493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f9eb80caa80 con 0x7f9eb8156f30 2023-10-30T12:28:53.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.360+0000 7f9eaeffd640 1 -- 172.21.15.89:0/4036494493 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9eb00b20b0 con 0x7f9eb8156f30 2023-10-30T12:28:53.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.364+0000 7f9ec0479640 1 -- 172.21.15.89:0/4036494493 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9e9c0664d0 msgr2=0x7f9e9c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.364+0000 7f9ec0479640 1 --2- 172.21.15.89:0/4036494493 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9e9c0664d0 0x7f9e9c068990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9eb40011f0 tx=0x7f9eb4009040 comp rx=0 tx=0).stop 2023-10-30T12:28:53.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.364+0000 7f9ec0479640 1 -- 172.21.15.89:0/4036494493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eb8156f30 msgr2=0x7f9eb81447e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.364+0000 7f9ec0479640 1 --2- 172.21.15.89:0/4036494493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eb8156f30 0x7f9eb81447e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9eb00b0040 tx=0x7f9eb00196c0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.364+0000 7f9ec0479640 1 -- 172.21.15.89:0/4036494493 shutdown_connections 2023-10-30T12:28:53.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.364+0000 7f9ec0479640 1 --2- 172.21.15.89:0/4036494493 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9e9c0664d0 0x7f9e9c068990 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.364+0000 7f9ec0479640 1 --2- 172.21.15.89:0/4036494493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eb8156f30 0x7f9eb81447e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.364+0000 7f9ec0479640 1 --2- 172.21.15.89:0/4036494493 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9eb8153980 0x7f9eb8148b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.364+0000 7f9ec0479640 1 --2- 172.21.15.89:0/4036494493 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9eb80cf6e0 0x7f9eb815aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.364+0000 7f9ec0479640 1 -- 172.21.15.89:0/4036494493 >> 172.21.15.89:0/4036494493 conn(0x7f9eb806c0f0 msgr2=0x7f9eb80b1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:53.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.364+0000 7f9ec0479640 1 -- 172.21.15.89:0/4036494493 shutdown_connections 2023-10-30T12:28:53.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.364+0000 7f9ec0479640 1 -- 172.21.15.89:0/4036494493 wait complete. 2023-10-30T12:28:53.392 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300} 2023-10-30T12:28:53.393 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2023-10-30T12:28:53.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.532+0000 7fa2d7046640 1 Processor -- start 2023-10-30T12:28:53.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.532+0000 7fa2d7046640 1 -- start start 2023-10-30T12:28:53.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.532+0000 7fa2d7046640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa2d006c9e0 0x7fa2d006afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.532+0000 7fa2d7046640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2d006b4e0 con 0x7fa2d01570d0 2023-10-30T12:28:53.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.532+0000 7fa2d7046640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2d01599b0 con 0x7fa2d006fdd0 2023-10-30T12:28:53.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.532+0000 7fa2d7046640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2d015a630 con 0x7fa2d006c9e0 2023-10-30T12:28:53.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.532+0000 7fa2c7fff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa2d006c9e0 0x7fa2d006afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.532+0000 7fa2d4de2640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa2d01570d0 0x7fa2d0071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55656/0 (socket says 172.21.15.89:55656) 2023-10-30T12:28:53.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.532+0000 7fa2d4de2640 1 -- 172.21.15.89:0/1993427560 learned_addr learned my addr 172.21.15.89:0/1993427560 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:28:53.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.536+0000 7fa2c7fff640 1 -- 172.21.15.89:0/1993427560 >> v1:172.21.15.192:6789/0 conn(0x7fa2d006fdd0 legacy=0x7fa2d00ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.536+0000 7fa2c7fff640 1 -- 172.21.15.89:0/1993427560 >> v1:172.21.15.89:6789/0 conn(0x7fa2d01570d0 legacy=0x7fa2d0071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.536+0000 7fa2c7fff640 1 -- 172.21.15.89:0/1993427560 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d0159c90 con 0x7fa2d006c9e0 2023-10-30T12:28:53.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.536+0000 7fa2c77fe640 1 -- 172.21.15.89:0/1993427560 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3847542361 0 0) 0x7fa2cc001120 con 0x7fa2d01570d0 2023-10-30T12:28:53.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.536+0000 7fa2c7fff640 1 --2- 172.21.15.89:0/1993427560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2d006c9e0 0x7fa2d006afa0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa2c00097c0 tx=0x7fa2c001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65530822c72bb1a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.536+0000 7fa2c77fe640 1 -- 172.21.15.89:0/1993427560 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c00a8970 con 0x7fa2d006c9e0 2023-10-30T12:28:53.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.536+0000 7fa2c77fe640 1 -- 172.21.15.89:0/1993427560 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2c00a8ad0 con 0x7fa2d006c9e0 2023-10-30T12:28:53.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.536+0000 7fa2c77fe640 1 -- 172.21.15.89:0/1993427560 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c00a8e60 con 0x7fa2d006c9e0 2023-10-30T12:28:53.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.536+0000 7fa2d7046640 1 -- 172.21.15.89:0/1993427560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2d006c9e0 msgr2=0x7fa2d006afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.536+0000 7fa2d7046640 1 --2- 172.21.15.89:0/1993427560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2d006c9e0 0x7fa2d006afa0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa2c00097c0 tx=0x7fa2c001ec70 comp rx=0 tx=0).stop 2023-10-30T12:28:53.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.536+0000 7fa2d7046640 1 -- 172.21.15.89:0/1993427560 shutdown_connections 2023-10-30T12:28:53.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.536+0000 7fa2d7046640 1 --2- 172.21.15.89:0/1993427560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2d006c9e0 0x7fa2d006afa0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.536+0000 7fa2d7046640 1 -- 172.21.15.89:0/1993427560 >> 172.21.15.89:0/1993427560 conn(0x7fa2d0069e90 msgr2=0x7fa2d006a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:53.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.540+0000 7fa2d7046640 1 -- 172.21.15.89:0/1993427560 shutdown_connections 2023-10-30T12:28:53.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.540+0000 7fa2d7046640 1 -- 172.21.15.89:0/1993427560 wait complete. 2023-10-30T12:28:53.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.540+0000 7fa2d7046640 1 Processor -- start 2023-10-30T12:28:53.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.540+0000 7fa2d7046640 1 -- start start 2023-10-30T12:28:53.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.540+0000 7fa2d7046640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2d006c9e0 0x7fa2d0151c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.540+0000 7fa2d7046640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2d006fdd0 0x7fa2d0065ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.540+0000 7fa2d7046640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2d01570d0 0x7fa2d0158250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.540+0000 7fa2d7046640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d0151970 con 0x7fa2d006c9e0 2023-10-30T12:28:53.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.540+0000 7fa2d7046640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d0152300 con 0x7fa2d006fdd0 2023-10-30T12:28:53.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.540+0000 7fa2d7046640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d01525a0 con 0x7fa2d01570d0 2023-10-30T12:28:53.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.540+0000 7fa2d4de2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2d006c9e0 0x7fa2d0151c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.540+0000 7fa2d4de2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2d006c9e0 0x7fa2d0151c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55258/0 (socket says 172.21.15.89:55258) 2023-10-30T12:28:53.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.540+0000 7fa2d4de2640 1 -- 172.21.15.89:0/2652246003 learned_addr learned my addr 172.21.15.89:0/2652246003 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:53.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.540+0000 7fa2c7fff640 1 --2- 172.21.15.89:0/2652246003 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2d006fdd0 0x7fa2d0065ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.540+0000 7fa2d55e3640 1 --2- 172.21.15.89:0/2652246003 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2d01570d0 0x7fa2d0158250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.540+0000 7fa2d4de2640 1 -- 172.21.15.89:0/2652246003 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2d01570d0 msgr2=0x7fa2d0158250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.540+0000 7fa2d4de2640 1 --2- 172.21.15.89:0/2652246003 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2d01570d0 0x7fa2d0158250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.540+0000 7fa2d4de2640 1 -- 172.21.15.89:0/2652246003 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2d006fdd0 msgr2=0x7fa2d0065ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.540+0000 7fa2d4de2640 1 --2- 172.21.15.89:0/2652246003 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2d006fdd0 0x7fa2d0065ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.540+0000 7fa2d4de2640 1 -- 172.21.15.89:0/2652246003 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c0009420 con 0x7fa2d006c9e0 2023-10-30T12:28:53.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.540+0000 7fa2d4de2640 1 --2- 172.21.15.89:0/2652246003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2d006c9e0 0x7fa2d0151c50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc00c410 tx=0x7fa2cc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.540+0000 7fa2c57fa640 1 -- 172.21.15.89:0/2652246003 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc0074e0 con 0x7fa2d006c9e0 2023-10-30T12:28:53.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.540+0000 7fa2c57fa640 1 -- 172.21.15.89:0/2652246003 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2cc007640 con 0x7fa2d006c9e0 2023-10-30T12:28:53.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.540+0000 7fa2d7046640 1 -- 172.21.15.89:0/2652246003 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2d0152840 con 0x7fa2d006c9e0 2023-10-30T12:28:53.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.540+0000 7fa2c57fa640 1 -- 172.21.15.89:0/2652246003 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc004a30 con 0x7fa2d006c9e0 2023-10-30T12:28:53.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.540+0000 7fa2d7046640 1 -- 172.21.15.89:0/2652246003 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2d0157a50 con 0x7fa2d006c9e0 2023-10-30T12:28:53.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.544+0000 7fa2d7046640 1 -- 172.21.15.89:0/2652246003 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2d004e170 con 0x7fa2d006c9e0 2023-10-30T12:28:53.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.544+0000 7fa2c57fa640 1 -- 172.21.15.89:0/2652246003 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa2cc004b90 con 0x7fa2d006c9e0 2023-10-30T12:28:53.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.544+0000 7fa2c57fa640 1 --2- 172.21.15.89:0/2652246003 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa2bc0664a0 0x7fa2bc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.544+0000 7fa2c57fa640 1 -- 172.21.15.89:0/2652246003 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa2cc083b60 con 0x7fa2d006c9e0 2023-10-30T12:28:53.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.544+0000 7fa2c7fff640 1 --2- 172.21.15.89:0/2652246003 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa2bc0664a0 0x7fa2bc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.548+0000 7fa2c7fff640 1 --2- 172.21.15.89:0/2652246003 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa2bc0664a0 0x7fa2bc068960 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa2c00b2000 tx=0x7fa2c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.548+0000 7fa2c57fa640 1 -- 172.21.15.89:0/2652246003 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa2cc050f50 con 0x7fa2d006c9e0 2023-10-30T12:28:53.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:53.732+0000 7fa2d7046640 1 -- 172.21.15.89:0/2652246003 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7fa2d006b4e0 con 0x7fa2d006c9e0 2023-10-30T12:28:54.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.364+0000 7fa2c57fa640 1 -- 172.21.15.89:0/2652246003 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7fa2d006b4e0 con 0x7fa2d006c9e0 2023-10-30T12:28:54.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.368+0000 7fa2d7046640 1 -- 172.21.15.89:0/2652246003 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa2bc0664a0 msgr2=0x7fa2bc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:54.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.368+0000 7fa2d7046640 1 --2- 172.21.15.89:0/2652246003 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa2bc0664a0 0x7fa2bc068960 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa2c00b2000 tx=0x7fa2c0008040 comp rx=0 tx=0).stop 2023-10-30T12:28:54.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.368+0000 7fa2d7046640 1 -- 172.21.15.89:0/2652246003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2d006c9e0 msgr2=0x7fa2d0151c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:54.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.368+0000 7fa2d7046640 1 --2- 172.21.15.89:0/2652246003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2d006c9e0 0x7fa2d0151c50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc00c410 tx=0x7fa2cc00c8d0 comp rx=0 tx=0).stop 2023-10-30T12:28:54.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.368+0000 7fa2d7046640 1 -- 172.21.15.89:0/2652246003 shutdown_connections 2023-10-30T12:28:54.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.368+0000 7fa2d7046640 1 --2- 172.21.15.89:0/2652246003 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa2bc0664a0 0x7fa2bc068960 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:54.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.368+0000 7fa2d7046640 1 --2- 172.21.15.89:0/2652246003 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2d006fdd0 0x7fa2d0065ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:54.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.368+0000 7fa2d7046640 1 --2- 172.21.15.89:0/2652246003 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2d01570d0 0x7fa2d0158250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:54.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.368+0000 7fa2d7046640 1 --2- 172.21.15.89:0/2652246003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2d006c9e0 0x7fa2d0151c50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:54.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.368+0000 7fa2d7046640 1 -- 172.21.15.89:0/2652246003 >> 172.21.15.89:0/2652246003 conn(0x7fa2d0069e90 msgr2=0x7fa2d00caec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:54.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.368+0000 7fa2d7046640 1 -- 172.21.15.89:0/2652246003 shutdown_connections 2023-10-30T12:28:54.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.368+0000 7fa2d7046640 1 -- 172.21.15.89:0/2652246003 wait complete. 2023-10-30T12:28:54.396 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-30T12:28:54.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.532+0000 7fefead3f640 1 Processor -- start 2023-10-30T12:28:54.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.532+0000 7fefead3f640 1 -- start start 2023-10-30T12:28:54.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.536+0000 7fefead3f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fefe4153840 0x7fefe415a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:54.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.536+0000 7fefead3f640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fefe40cb4c0 0x7fefe40cef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:54.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.536+0000 7fefead3f640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe415a6b0 con 0x7fefe4153c20 2023-10-30T12:28:54.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.536+0000 7fefead3f640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefe415dbc0 con 0x7fefe40cb4c0 2023-10-30T12:28:54.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.536+0000 7fefead3f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefe415d730 con 0x7fefe4153840 2023-10-30T12:28:54.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.536+0000 7fefe9d3d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fefe4153840 0x7fefe415a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:54.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.536+0000 7fefea53e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fefe40cb4c0 0x7fefe40cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:54.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.536+0000 7fefe9d3d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fefe4153840 0x7fefe415a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:37486/0 (socket says 172.21.15.89:37486) 2023-10-30T12:28:54.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.536+0000 7fefe9d3d640 1 -- 172.21.15.89:0/1688837697 learned_addr learned my addr 172.21.15.89:0/1688837697 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:54.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.536+0000 7fefe953c640 1 --1- 172.21.15.89:0/1688837697 >> v1:172.21.15.192:6789/0 conn(0x7fefe4153c20 0x7fefe40c3eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:46348/0 (socket says 172.21.15.89:46348) 2023-10-30T12:28:54.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.536+0000 7fefe9d3d640 1 -- 172.21.15.89:0/1688837697 >> v2:172.21.15.192:3300/0 conn(0x7fefe40cb4c0 msgr2=0x7fefe40cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:54.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.536+0000 7fefe9d3d640 1 --2- 172.21.15.89:0/1688837697 >> v2:172.21.15.192:3300/0 conn(0x7fefe40cb4c0 0x7fefe40cef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:54.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.536+0000 7fefe9d3d640 1 -- 172.21.15.89:0/1688837697 >> v1:172.21.15.192:6789/0 conn(0x7fefe4153c20 legacy=0x7fefe40c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:54.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.536+0000 7fefe9d3d640 1 -- 172.21.15.89:0/1688837697 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe40ca8f0 con 0x7fefe4153840 2023-10-30T12:28:54.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.536+0000 7fefea53e640 1 --2- 172.21.15.89:0/1688837697 >> v2:172.21.15.192:3300/0 conn(0x7fefe40cb4c0 0x7fefe40cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:54.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.540+0000 7fefe9d3d640 1 --2- 172.21.15.89:0/1688837697 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fefe4153840 0x7fefe415a170 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fefe0009870 tx=0x7fefe001ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97e870255a9602b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:54.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.540+0000 7fefe8d3b640 1 -- 172.21.15.89:0/1688837697 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe00a8a30 con 0x7fefe4153840 2023-10-30T12:28:54.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.540+0000 7fefe8d3b640 1 -- 172.21.15.89:0/1688837697 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefe00a8b90 con 0x7fefe4153840 2023-10-30T12:28:54.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.540+0000 7fefe8d3b640 1 -- 172.21.15.89:0/1688837697 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe00b0440 con 0x7fefe4153840 2023-10-30T12:28:54.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.540+0000 7fefead3f640 1 -- 172.21.15.89:0/1688837697 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fefe4153840 msgr2=0x7fefe415a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:54.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.540+0000 7fefead3f640 1 --2- 172.21.15.89:0/1688837697 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fefe4153840 0x7fefe415a170 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fefe0009870 tx=0x7fefe001ed30 comp rx=0 tx=0).stop 2023-10-30T12:28:54.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.540+0000 7fefead3f640 1 -- 172.21.15.89:0/1688837697 shutdown_connections 2023-10-30T12:28:54.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.540+0000 7fefead3f640 1 --2- 172.21.15.89:0/1688837697 >> v2:172.21.15.192:3300/0 conn(0x7fefe40cb4c0 0x7fefe40cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:54.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.540+0000 7fefead3f640 1 --2- 172.21.15.89:0/1688837697 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fefe4153840 0x7fefe415a170 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:54.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.540+0000 7fefead3f640 1 -- 172.21.15.89:0/1688837697 >> 172.21.15.89:0/1688837697 conn(0x7fefe406be40 msgr2=0x7fefe406c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:54.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.540+0000 7fefead3f640 1 -- 172.21.15.89:0/1688837697 shutdown_connections 2023-10-30T12:28:54.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.540+0000 7fefead3f640 1 -- 172.21.15.89:0/1688837697 wait complete. 2023-10-30T12:28:54.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.540+0000 7fefead3f640 1 Processor -- start 2023-10-30T12:28:54.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.540+0000 7fefead3f640 1 -- start start 2023-10-30T12:28:54.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.544+0000 7fefead3f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fefe40cb4c0 0x7fefe40b4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:54.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.544+0000 7fefead3f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fefe4153840 0x7fefe41583e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:54.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.544+0000 7fefead3f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefe4153c20 0x7fefe4158920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:54.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.544+0000 7fefead3f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe41593d0 con 0x7fefe4153c20 2023-10-30T12:28:54.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.544+0000 7fefead3f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe4157000 con 0x7fefe40cb4c0 2023-10-30T12:28:54.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.544+0000 7fefead3f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefe41efb00 con 0x7fefe4153840 2023-10-30T12:28:54.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.544+0000 7fefea53e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefe4153c20 0x7fefe4158920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:54.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.544+0000 7fefea53e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefe4153c20 0x7fefe4158920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55266/0 (socket says 172.21.15.89:55266) 2023-10-30T12:28:54.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.544+0000 7fefea53e640 1 -- 172.21.15.89:0/153795659 learned_addr learned my addr 172.21.15.89:0/153795659 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:54.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.544+0000 7fefe953c640 1 --2- 172.21.15.89:0/153795659 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fefe4153840 0x7fefe41583e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:54.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.544+0000 7fefe9d3d640 1 --2- 172.21.15.89:0/153795659 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fefe40cb4c0 0x7fefe40b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:54.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.544+0000 7fefea53e640 1 -- 172.21.15.89:0/153795659 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fefe4153840 msgr2=0x7fefe41583e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:54.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.544+0000 7fefea53e640 1 --2- 172.21.15.89:0/153795659 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fefe4153840 0x7fefe41583e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:54.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.544+0000 7fefea53e640 1 -- 172.21.15.89:0/153795659 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fefe40cb4c0 msgr2=0x7fefe40b4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:54.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.544+0000 7fefea53e640 1 --2- 172.21.15.89:0/153795659 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fefe40cb4c0 0x7fefe40b4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:54.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.544+0000 7fefea53e640 1 -- 172.21.15.89:0/153795659 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefdc009a70 con 0x7fefe4153c20 2023-10-30T12:28:54.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.544+0000 7fefea53e640 1 --2- 172.21.15.89:0/153795659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefe4153c20 0x7fefe4158920 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fefdc00efc0 tx=0x7fefdc009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:54.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.544+0000 7fefda7fc640 1 -- 172.21.15.89:0/153795659 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefdc009d70 con 0x7fefe4153c20 2023-10-30T12:28:54.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.544+0000 7fefda7fc640 1 -- 172.21.15.89:0/153795659 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefdc00f040 con 0x7fefe4153c20 2023-10-30T12:28:54.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.544+0000 7fefda7fc640 1 -- 172.21.15.89:0/153795659 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefdc0039a0 con 0x7fefe4153c20 2023-10-30T12:28:54.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.544+0000 7fefead3f640 1 -- 172.21.15.89:0/153795659 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefe00094d0 con 0x7fefe4153c20 2023-10-30T12:28:54.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.544+0000 7fefead3f640 1 -- 172.21.15.89:0/153795659 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefe4157560 con 0x7fefe4153c20 2023-10-30T12:28:54.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.544+0000 7fefead3f640 1 -- 172.21.15.89:0/153795659 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefe404e9c0 con 0x7fefe4153c20 2023-10-30T12:28:54.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.544+0000 7fefda7fc640 1 -- 172.21.15.89:0/153795659 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fefdc0144a0 con 0x7fefe4153c20 2023-10-30T12:28:54.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.548+0000 7fefda7fc640 1 --2- 172.21.15.89:0/153795659 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fefc80664a0 0x7fefc8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:54.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.548+0000 7fefda7fc640 1 -- 172.21.15.89:0/153795659 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fefdc084a70 con 0x7fefe4153c20 2023-10-30T12:28:54.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.548+0000 7fefe9d3d640 1 --2- 172.21.15.89:0/153795659 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fefc80664a0 0x7fefc8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:54.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.548+0000 7fefe9d3d640 1 --2- 172.21.15.89:0/153795659 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fefc80664a0 0x7fefc8068960 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fefe00b2000 tx=0x7fefe00aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:54.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.552+0000 7fefda7fc640 1 -- 172.21.15.89:0/153795659 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fefdc051f00 con 0x7fefe4153c20 2023-10-30T12:28:54.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:54.732+0000 7fefead3f640 1 -- 172.21.15.89:0/153795659 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fefe414fd90 con 0x7fefe4153c20 2023-10-30T12:28:55.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.380+0000 7fefda7fc640 1 -- 172.21.15.89:0/153795659 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) v1 ==== 122+0+0 (secure 0 0 0) 0x7fefdc0180c0 con 0x7fefe4153c20 2023-10-30T12:28:55.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.384+0000 7fefead3f640 1 -- 172.21.15.89:0/153795659 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fefc80664a0 msgr2=0x7fefc8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.384+0000 7fefead3f640 1 --2- 172.21.15.89:0/153795659 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fefc80664a0 0x7fefc8068960 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fefe00b2000 tx=0x7fefe00aa070 comp rx=0 tx=0).stop 2023-10-30T12:28:55.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.384+0000 7fefead3f640 1 -- 172.21.15.89:0/153795659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefe4153c20 msgr2=0x7fefe4158920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.384+0000 7fefead3f640 1 --2- 172.21.15.89:0/153795659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefe4153c20 0x7fefe4158920 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fefdc00efc0 tx=0x7fefdc009f90 comp rx=0 tx=0).stop 2023-10-30T12:28:55.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.384+0000 7fefead3f640 1 -- 172.21.15.89:0/153795659 shutdown_connections 2023-10-30T12:28:55.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.384+0000 7fefead3f640 1 --2- 172.21.15.89:0/153795659 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fefc80664a0 0x7fefc8068960 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.384+0000 7fefead3f640 1 --2- 172.21.15.89:0/153795659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefe4153c20 0x7fefe4158920 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.384+0000 7fefead3f640 1 --2- 172.21.15.89:0/153795659 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fefe4153840 0x7fefe41583e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.384+0000 7fefead3f640 1 --2- 172.21.15.89:0/153795659 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fefe40cb4c0 0x7fefe40b4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.384+0000 7fefead3f640 1 -- 172.21.15.89:0/153795659 >> 172.21.15.89:0/153795659 conn(0x7fefe406be40 msgr2=0x7fefe4150590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:55.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.384+0000 7fefead3f640 1 -- 172.21.15.89:0/153795659 shutdown_connections 2023-10-30T12:28:55.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.384+0000 7fefead3f640 1 -- 172.21.15.89:0/153795659 wait complete. 2023-10-30T12:28:55.413 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-10-30T12:28:55.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.552+0000 7f69f6d3e640 1 Processor -- start 2023-10-30T12:28:55.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.552+0000 7f69f6d3e640 1 -- start start 2023-10-30T12:28:55.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.552+0000 7f69f6d3e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69f01508c0 0x7f69f00c6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:55.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.552+0000 7f69f6d3e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f69f00c6e60 0x7f69f0156930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:55.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.552+0000 7f69f6d3e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f00c0a30 con 0x7f69f00722a0 2023-10-30T12:28:55.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.552+0000 7f69f6d3e640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69f00c85b0 con 0x7f69f00c6e60 2023-10-30T12:28:55.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.552+0000 7f69f6d3e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f006f650 con 0x7f69f01508c0 2023-10-30T12:28:55.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.552+0000 7f69f52db640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f69f00c6e60 0x7f69f0156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:55.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.552+0000 7f69f52db640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f69f00c6e60 0x7f69f0156930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55276/0 (socket says 172.21.15.89:55276) 2023-10-30T12:28:55.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.552+0000 7f69f52db640 1 -- 172.21.15.89:0/2733629439 learned_addr learned my addr 172.21.15.89:0/2733629439 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:55.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.552+0000 7f69f4ada640 1 --2- 172.21.15.89:0/2733629439 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69f01508c0 0x7f69f00c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:55.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.552+0000 7f69f52db640 1 -- 172.21.15.89:0/2733629439 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69f01508c0 msgr2=0x7f69f00c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:55.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.552+0000 7f69f52db640 1 --2- 172.21.15.89:0/2733629439 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69f01508c0 0x7f69f00c6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.552+0000 7f69f52db640 1 -- 172.21.15.89:0/2733629439 >> v1:172.21.15.192:6789/0 conn(0x7f69f00722a0 legacy=0x7f69f00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.552+0000 7f69f52db640 1 -- 172.21.15.89:0/2733629439 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f006f930 con 0x7f69f00c6e60 2023-10-30T12:28:55.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.556+0000 7f69f52db640 1 --2- 172.21.15.89:0/2733629439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f00c6e60 0x7f69f0156930 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f69e8009e50 tx=0x7f69e801ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ff3847c7930b5f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:55.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.556+0000 7f69e77fe640 1 -- 172.21.15.89:0/2733629439 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e80a78e0 con 0x7f69f00c6e60 2023-10-30T12:28:55.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.556+0000 7f69e77fe640 1 -- 172.21.15.89:0/2733629439 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e80a7a40 con 0x7f69f00c6e60 2023-10-30T12:28:55.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.556+0000 7f69e77fe640 1 -- 172.21.15.89:0/2733629439 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e80a7dd0 con 0x7f69f00c6e60 2023-10-30T12:28:55.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.556+0000 7f69f6d3e640 1 -- 172.21.15.89:0/2733629439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f00c6e60 msgr2=0x7f69f0156930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.556+0000 7f69f6d3e640 1 --2- 172.21.15.89:0/2733629439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f00c6e60 0x7f69f0156930 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f69e8009e50 tx=0x7f69e801ec20 comp rx=0 tx=0).stop 2023-10-30T12:28:55.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.556+0000 7f69f6d3e640 1 -- 172.21.15.89:0/2733629439 shutdown_connections 2023-10-30T12:28:55.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.556+0000 7f69f6d3e640 1 --2- 172.21.15.89:0/2733629439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f00c6e60 0x7f69f0156930 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.556+0000 7f69f6d3e640 1 --2- 172.21.15.89:0/2733629439 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69f01508c0 0x7f69f00c6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.556+0000 7f69f6d3e640 1 -- 172.21.15.89:0/2733629439 >> 172.21.15.89:0/2733629439 conn(0x7f69f00c7910 msgr2=0x7f69f00c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:55.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.556+0000 7f69f6d3e640 1 -- 172.21.15.89:0/2733629439 shutdown_connections 2023-10-30T12:28:55.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.556+0000 7f69f6d3e640 1 -- 172.21.15.89:0/2733629439 wait complete. 2023-10-30T12:28:55.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.560+0000 7f69f6d3e640 1 Processor -- start 2023-10-30T12:28:55.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.560+0000 7f69f6d3e640 1 -- start start 2023-10-30T12:28:55.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.560+0000 7f69f6d3e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f00722a0 0x7f69f013c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:55.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.560+0000 7f69f6d3e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69f00c6e60 0x7f69f0154480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:55.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.560+0000 7f69f6d3e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69f01508c0 0x7f69f01419b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:55.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.560+0000 7f69f6d3e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f0155610 con 0x7f69f00722a0 2023-10-30T12:28:55.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.560+0000 7f69f6d3e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f0155ef0 con 0x7f69f01508c0 2023-10-30T12:28:55.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.560+0000 7f69f6d3e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f00c9480 con 0x7f69f00c6e60 2023-10-30T12:28:55.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.560+0000 7f69f4ada640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f00722a0 0x7f69f013c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:55.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.560+0000 7f69f4ada640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f00722a0 0x7f69f013c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55284/0 (socket says 172.21.15.89:55284) 2023-10-30T12:28:55.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.560+0000 7f69f4ada640 1 -- 172.21.15.89:0/4236084060 learned_addr learned my addr 172.21.15.89:0/4236084060 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:55.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.560+0000 7f69f52db640 1 --2- 172.21.15.89:0/4236084060 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69f01508c0 0x7f69f01419b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:55.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.560+0000 7f69e7fff640 1 --2- 172.21.15.89:0/4236084060 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69f00c6e60 0x7f69f0154480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:55.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.560+0000 7f69f4ada640 1 -- 172.21.15.89:0/4236084060 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69f00c6e60 msgr2=0x7f69f0154480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.560+0000 7f69f4ada640 1 --2- 172.21.15.89:0/4236084060 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69f00c6e60 0x7f69f0154480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.560+0000 7f69f4ada640 1 -- 172.21.15.89:0/4236084060 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69f01508c0 msgr2=0x7f69f01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.560+0000 7f69f4ada640 1 --2- 172.21.15.89:0/4236084060 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69f01508c0 0x7f69f01419b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.560+0000 7f69f4ada640 1 -- 172.21.15.89:0/4236084060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e8009b30 con 0x7f69f00722a0 2023-10-30T12:28:55.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.560+0000 7f69f4ada640 1 --2- 172.21.15.89:0/4236084060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f00722a0 0x7f69f013c980 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f69ec00d800 tx=0x7f69ec00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:55.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.560+0000 7f69e57fa640 1 -- 172.21.15.89:0/4236084060 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ec00b660 con 0x7f69f00722a0 2023-10-30T12:28:55.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.560+0000 7f69e57fa640 1 -- 172.21.15.89:0/4236084060 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ec00b7c0 con 0x7f69f00722a0 2023-10-30T12:28:55.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.560+0000 7f69e57fa640 1 -- 172.21.15.89:0/4236084060 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ec00bb70 con 0x7f69f00722a0 2023-10-30T12:28:55.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.560+0000 7f69f6d3e640 1 -- 172.21.15.89:0/4236084060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f00ca980 con 0x7f69f00722a0 2023-10-30T12:28:55.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.560+0000 7f69f6d3e640 1 -- 172.21.15.89:0/4236084060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f00c9ad0 con 0x7f69f00722a0 2023-10-30T12:28:55.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.560+0000 7f69f6d3e640 1 -- 172.21.15.89:0/4236084060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69f004e170 con 0x7f69f00722a0 2023-10-30T12:28:55.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.564+0000 7f69e57fa640 1 -- 172.21.15.89:0/4236084060 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f69ec00bcd0 con 0x7f69f00722a0 2023-10-30T12:28:55.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.564+0000 7f69e57fa640 1 --2- 172.21.15.89:0/4236084060 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f69dc0664a0 0x7f69dc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:55.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.564+0000 7f69e57fa640 1 -- 172.21.15.89:0/4236084060 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69ec084390 con 0x7f69f00722a0 2023-10-30T12:28:55.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.564+0000 7f69e7fff640 1 --2- 172.21.15.89:0/4236084060 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f69dc0664a0 0x7f69dc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:55.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.564+0000 7f69e7fff640 1 --2- 172.21.15.89:0/4236084060 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f69dc0664a0 0x7f69dc068960 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f69f00c0920 tx=0x7f69e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:55.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.568+0000 7f69e57fa640 1 -- 172.21.15.89:0/4236084060 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f69ec051770 con 0x7f69f00722a0 2023-10-30T12:28:55.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:55.752+0000 7f69f6d3e640 1 -- 172.21.15.89:0/4236084060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f69f014a6b0 con 0x7f69f00722a0 2023-10-30T12:28:56.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.392+0000 7f69e57fa640 1 -- 172.21.15.89:0/4236084060 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v8) v1 ==== 115+0+0 (secure 0 0 0) 0x7f69ec054d90 con 0x7f69f00722a0 2023-10-30T12:28:56.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.396+0000 7f69f6d3e640 1 -- 172.21.15.89:0/4236084060 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f69dc0664a0 msgr2=0x7f69dc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.396+0000 7f69f6d3e640 1 --2- 172.21.15.89:0/4236084060 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f69dc0664a0 0x7f69dc068960 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f69f00c0920 tx=0x7f69e0009040 comp rx=0 tx=0).stop 2023-10-30T12:28:56.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.396+0000 7f69f6d3e640 1 -- 172.21.15.89:0/4236084060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f00722a0 msgr2=0x7f69f013c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.396+0000 7f69f6d3e640 1 --2- 172.21.15.89:0/4236084060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f00722a0 0x7f69f013c980 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f69ec00d800 tx=0x7f69ec00dcc0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.396+0000 7f69f6d3e640 1 -- 172.21.15.89:0/4236084060 shutdown_connections 2023-10-30T12:28:56.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.396+0000 7f69f6d3e640 1 --2- 172.21.15.89:0/4236084060 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f69dc0664a0 0x7f69dc068960 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.396+0000 7f69f6d3e640 1 --2- 172.21.15.89:0/4236084060 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f69f01508c0 0x7f69f01419b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.396+0000 7f69f6d3e640 1 --2- 172.21.15.89:0/4236084060 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69f00c6e60 0x7f69f0154480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.396+0000 7f69f6d3e640 1 --2- 172.21.15.89:0/4236084060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69f00722a0 0x7f69f013c980 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.396+0000 7f69f6d3e640 1 -- 172.21.15.89:0/4236084060 >> 172.21.15.89:0/4236084060 conn(0x7f69f00c7910 msgr2=0x7f69f00c7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:56.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.396+0000 7f69f6d3e640 1 -- 172.21.15.89:0/4236084060 shutdown_connections 2023-10-30T12:28:56.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.396+0000 7f69f6d3e640 1 -- 172.21.15.89:0/4236084060 wait complete. 2023-10-30T12:28:56.422 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-30T12:28:56.422 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:28:56.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.556+0000 7f76937d9640 1 Processor -- start 2023-10-30T12:28:56.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.556+0000 7f76937d9640 1 -- start start 2023-10-30T12:28:56.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.556+0000 7f76937d9640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f768c15d6f0 0x7f768c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.556+0000 7f76937d9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f768c0cf160 0x7f768c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.556+0000 7f76937d9640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768c0cb800 con 0x7f768c1538a0 2023-10-30T12:28:56.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.556+0000 7f76937d9640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f768c0cb330 con 0x7f768c15d6f0 2023-10-30T12:28:56.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.556+0000 7f76937d9640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f768c0cad70 con 0x7f768c0cf160 2023-10-30T12:28:56.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.556+0000 7f7690d74640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f768c15d6f0 0x7f768c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.556+0000 7f7690d74640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f768c15d6f0 0x7f768c15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55288/0 (socket says 172.21.15.89:55288) 2023-10-30T12:28:56.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.556+0000 7f7690d74640 1 -- 172.21.15.89:0/1879664061 learned_addr learned my addr 172.21.15.89:0/1879664061 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:56.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.556+0000 7f7691d76640 1 --2- 172.21.15.89:0/1879664061 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f768c0cf160 0x7f768c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.556+0000 7f7690d74640 1 -- 172.21.15.89:0/1879664061 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f768c0cf160 msgr2=0x7f768c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:56.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.556+0000 7f7690d74640 1 --2- 172.21.15.89:0/1879664061 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f768c0cf160 0x7f768c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.556+0000 7f7690d74640 1 -- 172.21.15.89:0/1879664061 >> v1:172.21.15.89:6789/0 conn(0x7f768c1538a0 legacy=0x7f768c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.556+0000 7f7690d74640 1 -- 172.21.15.89:0/1879664061 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c0cb050 con 0x7f768c15d6f0 2023-10-30T12:28:56.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.560+0000 7f7690d74640 1 --2- 172.21.15.89:0/1879664061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f768c15d6f0 0x7f768c15a090 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f767c009870 tx=0x7f767c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ac55964b9ccf07f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:56.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.560+0000 7f7683fff640 1 -- 172.21.15.89:0/1879664061 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c0ac620 con 0x7f768c15d6f0 2023-10-30T12:28:56.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.560+0000 7f7683fff640 1 -- 172.21.15.89:0/1879664061 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767c0ac780 con 0x7f768c15d6f0 2023-10-30T12:28:56.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.560+0000 7f7683fff640 1 -- 172.21.15.89:0/1879664061 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c0acb10 con 0x7f768c15d6f0 2023-10-30T12:28:56.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.560+0000 7f76937d9640 1 -- 172.21.15.89:0/1879664061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f768c15d6f0 msgr2=0x7f768c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.560+0000 7f76937d9640 1 --2- 172.21.15.89:0/1879664061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f768c15d6f0 0x7f768c15a090 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f767c009870 tx=0x7f767c01c460 comp rx=0 tx=0).stop 2023-10-30T12:28:56.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.560+0000 7f76937d9640 1 -- 172.21.15.89:0/1879664061 shutdown_connections 2023-10-30T12:28:56.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.560+0000 7f76937d9640 1 --2- 172.21.15.89:0/1879664061 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f768c0cf160 0x7f768c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.560+0000 7f76937d9640 1 --2- 172.21.15.89:0/1879664061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f768c15d6f0 0x7f768c15a090 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.560+0000 7f76937d9640 1 -- 172.21.15.89:0/1879664061 >> 172.21.15.89:0/1879664061 conn(0x7f768c06bfe0 msgr2=0x7f768c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:56.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.560+0000 7f76937d9640 1 -- 172.21.15.89:0/1879664061 shutdown_connections 2023-10-30T12:28:56.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.560+0000 7f76937d9640 1 -- 172.21.15.89:0/1879664061 wait complete. 2023-10-30T12:28:56.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.564+0000 7f76937d9640 1 Processor -- start 2023-10-30T12:28:56.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.564+0000 7f76937d9640 1 -- start start 2023-10-30T12:28:56.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.564+0000 7f76937d9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f768c0cf160 0x7f768c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.564+0000 7f76937d9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f768c1538a0 0x7f768c145170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.564+0000 7f76937d9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f768c15d6f0 0x7f768c148ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.564+0000 7f76937d9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c145d10 con 0x7f768c0cf160 2023-10-30T12:28:56.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.564+0000 7f76937d9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c148390 con 0x7f768c15d6f0 2023-10-30T12:28:56.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.564+0000 7f76937d9640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f768c146830 con 0x7f768c1538a0 2023-10-30T12:28:56.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.564+0000 7f7691575640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f768c0cf160 0x7f768c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.564+0000 7f7691575640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f768c0cf160 0x7f768c0b4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55292/0 (socket says 172.21.15.89:55292) 2023-10-30T12:28:56.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.564+0000 7f7691575640 1 -- 172.21.15.89:0/2402327783 learned_addr learned my addr 172.21.15.89:0/2402327783 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:56.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.564+0000 7f7690d74640 1 --2- 172.21.15.89:0/2402327783 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f768c1538a0 0x7f768c145170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.564+0000 7f7691d76640 1 --2- 172.21.15.89:0/2402327783 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f768c15d6f0 0x7f768c148ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.564+0000 7f7691575640 1 -- 172.21.15.89:0/2402327783 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f768c1538a0 msgr2=0x7f768c145170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.564+0000 7f7691575640 1 --2- 172.21.15.89:0/2402327783 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f768c1538a0 0x7f768c145170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.564+0000 7f7691575640 1 -- 172.21.15.89:0/2402327783 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f768c15d6f0 msgr2=0x7f768c148ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.564+0000 7f7691575640 1 --2- 172.21.15.89:0/2402327783 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f768c15d6f0 0x7f768c148ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.564+0000 7f7691575640 1 -- 172.21.15.89:0/2402327783 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f767c0094d0 con 0x7f768c0cf160 2023-10-30T12:28:56.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.564+0000 7f7691575640 1 --2- 172.21.15.89:0/2402327783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f768c0cf160 0x7f768c0b4d90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7688009f00 tx=0x7f768800c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:56.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.564+0000 7f7681ffb640 1 -- 172.21.15.89:0/2402327783 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768800ce70 con 0x7f768c0cf160 2023-10-30T12:28:56.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.564+0000 7f7681ffb640 1 -- 172.21.15.89:0/2402327783 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7688004020 con 0x7f768c0cf160 2023-10-30T12:28:56.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.564+0000 7f7681ffb640 1 -- 172.21.15.89:0/2402327783 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76880043f0 con 0x7f768c0cf160 2023-10-30T12:28:56.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.564+0000 7f76937d9640 1 -- 172.21.15.89:0/2402327783 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f768c148630 con 0x7f768c0cf160 2023-10-30T12:28:56.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.564+0000 7f76937d9640 1 -- 172.21.15.89:0/2402327783 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f768c1efde0 con 0x7f768c0cf160 2023-10-30T12:28:56.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.568+0000 7f76937d9640 1 -- 172.21.15.89:0/2402327783 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f768c0b2a50 con 0x7f768c0cf160 2023-10-30T12:28:56.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.568+0000 7f7681ffb640 1 -- 172.21.15.89:0/2402327783 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7688003600 con 0x7f768c0cf160 2023-10-30T12:28:56.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.568+0000 7f7681ffb640 1 --2- 172.21.15.89:0/2402327783 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f76700664a0 0x7f7670068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.568+0000 7f7681ffb640 1 -- 172.21.15.89:0/2402327783 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7688086840 con 0x7f768c0cf160 2023-10-30T12:28:56.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.568+0000 7f7690d74640 1 --2- 172.21.15.89:0/2402327783 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f76700664a0 0x7f7670068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.572+0000 7f7690d74640 1 --2- 172.21.15.89:0/2402327783 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f76700664a0 0x7f7670068960 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f767c0ae000 tx=0x7f767c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:56.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.572+0000 7f7681ffb640 1 -- 172.21.15.89:0/2402327783 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7688053b90 con 0x7f768c0cf160 2023-10-30T12:28:56.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.752+0000 7f76937d9640 1 -- 172.21.15.89:0/2402327783 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f768c15a570 con 0x7f768c0cf160 2023-10-30T12:28:56.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.752+0000 7f7681ffb640 1 -- 172.21.15.89:0/2402327783 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6513 (secure 0 0 0) 0x7f7688018090 con 0x7f768c0cf160 2023-10-30T12:28:56.758 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:28:56.758 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1289220491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1289220491},{"type":"v1","addr":"172.21.15.192:6835","nonce":1289220491}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:28:56.400026+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1],"up":{"mds_0":4318,"mds_1":4262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:creating","state_seq":1,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:28:56.758 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 9 2023-10-30T12:28:56.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.752+0000 7f76937d9640 1 -- 172.21.15.89:0/2402327783 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f76700664a0 msgr2=0x7f7670068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.752+0000 7f76937d9640 1 --2- 172.21.15.89:0/2402327783 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f76700664a0 0x7f7670068960 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f767c0ae000 tx=0x7f767c008040 comp rx=0 tx=0).stop 2023-10-30T12:28:56.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.752+0000 7f76937d9640 1 -- 172.21.15.89:0/2402327783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f768c0cf160 msgr2=0x7f768c0b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.752+0000 7f76937d9640 1 --2- 172.21.15.89:0/2402327783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f768c0cf160 0x7f768c0b4d90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7688009f00 tx=0x7f768800c430 comp rx=0 tx=0).stop 2023-10-30T12:28:56.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.752+0000 7f76937d9640 1 -- 172.21.15.89:0/2402327783 shutdown_connections 2023-10-30T12:28:56.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.752+0000 7f76937d9640 1 --2- 172.21.15.89:0/2402327783 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f76700664a0 0x7f7670068960 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.752+0000 7f76937d9640 1 --2- 172.21.15.89:0/2402327783 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f768c15d6f0 0x7f768c148ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.752+0000 7f76937d9640 1 --2- 172.21.15.89:0/2402327783 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f768c1538a0 0x7f768c145170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.752+0000 7f76937d9640 1 --2- 172.21.15.89:0/2402327783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f768c0cf160 0x7f768c0b4d90 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.752+0000 7f76937d9640 1 -- 172.21.15.89:0/2402327783 >> 172.21.15.89:0/2402327783 conn(0x7f768c06bfe0 msgr2=0x7f768c152790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:56.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.752+0000 7f76937d9640 1 -- 172.21.15.89:0/2402327783 shutdown_connections 2023-10-30T12:28:56.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.752+0000 7f76937d9640 1 -- 172.21.15.89:0/2402327783 wait complete. 2023-10-30T12:28:56.778 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:28:56.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.908+0000 7fc051b0f640 1 Processor -- start 2023-10-30T12:28:56.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.912+0000 7fc051b0f640 1 -- start start 2023-10-30T12:28:56.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.912+0000 7fc051b0f640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc04c153bf0 0x7fc04c071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.912+0000 7fc051b0f640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc04c0c6fe0 0x7fc04c0c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.912+0000 7fc051b0f640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc04c0700b0 con 0x7fc04c0c79e0 2023-10-30T12:28:56.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.912+0000 7fc051b0f640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc04c0c6810 con 0x7fc04c153bf0 2023-10-30T12:28:56.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.912+0000 7fc051b0f640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc04c159900 con 0x7fc04c0c6fe0 2023-10-30T12:28:56.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.912+0000 7fc04b7fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc04c153bf0 0x7fc04c071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.912+0000 7fc04bfff640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc04c0c79e0 0x7fc04c0cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55678/0 (socket says 172.21.15.89:55678) 2023-10-30T12:28:56.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.912+0000 7fc04bfff640 1 -- 172.21.15.89:0/2468978863 learned_addr learned my addr 172.21.15.89:0/2468978863 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:28:56.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.912+0000 7fc043fff640 1 --2- 172.21.15.89:0/2468978863 >> v2:172.21.15.192:3300/0 conn(0x7fc04c0c6fe0 0x7fc04c0c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.912+0000 7fc04b7fe640 1 -- 172.21.15.89:0/2468978863 >> v2:172.21.15.192:3300/0 conn(0x7fc04c0c6fe0 msgr2=0x7fc04c0c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:56.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.912+0000 7fc04b7fe640 1 --2- 172.21.15.89:0/2468978863 >> v2:172.21.15.192:3300/0 conn(0x7fc04c0c6fe0 0x7fc04c0c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.912+0000 7fc04b7fe640 1 -- 172.21.15.89:0/2468978863 >> v1:172.21.15.89:6789/0 conn(0x7fc04c0c79e0 legacy=0x7fc04c0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.912+0000 7fc04affd640 1 -- 172.21.15.89:0/2468978863 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2708450992 0 0) 0x7fc04c0700b0 con 0x7fc04c0c79e0 2023-10-30T12:28:56.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.912+0000 7fc04b7fe640 1 -- 172.21.15.89:0/2468978863 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc04c0c6530 con 0x7fc04c153bf0 2023-10-30T12:28:56.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.916+0000 7fc04b7fe640 1 --2- 172.21.15.89:0/2468978863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc04c153bf0 0x7fc04c071260 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc04c0723d0 tx=0x7fc03c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62d3ee39842bf6bd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:56.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.916+0000 7fc04affd640 1 -- 172.21.15.89:0/2468978863 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c0a8950 con 0x7fc04c153bf0 2023-10-30T12:28:56.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.916+0000 7fc04affd640 1 -- 172.21.15.89:0/2468978863 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc03c0a8ab0 con 0x7fc04c153bf0 2023-10-30T12:28:56.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.916+0000 7fc04affd640 1 -- 172.21.15.89:0/2468978863 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c0a8e40 con 0x7fc04c153bf0 2023-10-30T12:28:56.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.916+0000 7fc051b0f640 1 -- 172.21.15.89:0/2468978863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc04c153bf0 msgr2=0x7fc04c071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.916+0000 7fc051b0f640 1 --2- 172.21.15.89:0/2468978863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc04c153bf0 0x7fc04c071260 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc04c0723d0 tx=0x7fc03c01ec50 comp rx=0 tx=0).stop 2023-10-30T12:28:56.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.916+0000 7fc051b0f640 1 -- 172.21.15.89:0/2468978863 shutdown_connections 2023-10-30T12:28:56.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.916+0000 7fc051b0f640 1 --2- 172.21.15.89:0/2468978863 >> v2:172.21.15.192:3300/0 conn(0x7fc04c0c6fe0 0x7fc04c0c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.916+0000 7fc051b0f640 1 --2- 172.21.15.89:0/2468978863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc04c153bf0 0x7fc04c071260 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.916+0000 7fc051b0f640 1 -- 172.21.15.89:0/2468978863 >> 172.21.15.89:0/2468978863 conn(0x7fc04c06c6e0 msgr2=0x7fc04c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:56.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.916+0000 7fc051b0f640 1 -- 172.21.15.89:0/2468978863 shutdown_connections 2023-10-30T12:28:56.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.916+0000 7fc051b0f640 1 -- 172.21.15.89:0/2468978863 wait complete. 2023-10-30T12:28:56.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.916+0000 7fc051b0f640 1 Processor -- start 2023-10-30T12:28:56.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.916+0000 7fc051b0f640 1 -- start start 2023-10-30T12:28:56.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.920+0000 7fc051b0f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc04c0c6fe0 0x7fc04c0c5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.920+0000 7fc051b0f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc04c0c79e0 0x7fc04c0c56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.920+0000 7fc051b0f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc04c153bf0 0x7fc04c155540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.920+0000 7fc051b0f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c0c3150 con 0x7fc04c153bf0 2023-10-30T12:28:56.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.920+0000 7fc051b0f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c154ce0 con 0x7fc04c0c79e0 2023-10-30T12:28:56.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.920+0000 7fc051b0f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c155090 con 0x7fc04c0c6fe0 2023-10-30T12:28:56.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.920+0000 7fc04bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc04c153bf0 0x7fc04c155540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.920+0000 7fc04bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc04c153bf0 0x7fc04c155540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55316/0 (socket says 172.21.15.89:55316) 2023-10-30T12:28:56.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.920+0000 7fc04bfff640 1 -- 172.21.15.89:0/2057682890 learned_addr learned my addr 172.21.15.89:0/2057682890 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:56.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.920+0000 7fc04b7fe640 1 --2- 172.21.15.89:0/2057682890 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc04c0c6fe0 0x7fc04c0c5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.920+0000 7fc043fff640 1 --2- 172.21.15.89:0/2057682890 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc04c0c79e0 0x7fc04c0c56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.920+0000 7fc04bfff640 1 -- 172.21.15.89:0/2057682890 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc04c0c6fe0 msgr2=0x7fc04c0c5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.920+0000 7fc04bfff640 1 --2- 172.21.15.89:0/2057682890 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc04c0c6fe0 0x7fc04c0c5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.920+0000 7fc04bfff640 1 -- 172.21.15.89:0/2057682890 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc04c0c79e0 msgr2=0x7fc04c0c56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.920+0000 7fc04bfff640 1 --2- 172.21.15.89:0/2057682890 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc04c0c79e0 0x7fc04c0c56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.920+0000 7fc04bfff640 1 -- 172.21.15.89:0/2057682890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03c009420 con 0x7fc04c153bf0 2023-10-30T12:28:56.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.920+0000 7fc04bfff640 1 --2- 172.21.15.89:0/2057682890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc04c153bf0 0x7fc04c155540 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc04400c810 tx=0x7fc04400ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:56.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.920+0000 7fc048ff9640 1 -- 172.21.15.89:0/2057682890 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc04400a640 con 0x7fc04c153bf0 2023-10-30T12:28:56.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.920+0000 7fc048ff9640 1 -- 172.21.15.89:0/2057682890 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc04400a7c0 con 0x7fc04c153bf0 2023-10-30T12:28:56.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.920+0000 7fc048ff9640 1 -- 172.21.15.89:0/2057682890 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc04400ab70 con 0x7fc04c153bf0 2023-10-30T12:28:56.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.920+0000 7fc051b0f640 1 -- 172.21.15.89:0/2057682890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc04c0c3380 con 0x7fc04c153bf0 2023-10-30T12:28:56.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.920+0000 7fc051b0f640 1 -- 172.21.15.89:0/2057682890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc04c0c4330 con 0x7fc04c153bf0 2023-10-30T12:28:56.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.920+0000 7fc051b0f640 1 -- 172.21.15.89:0/2057682890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc04c155260 con 0x7fc04c153bf0 2023-10-30T12:28:56.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.920+0000 7fc048ff9640 1 -- 172.21.15.89:0/2057682890 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc04400acd0 con 0x7fc04c153bf0 2023-10-30T12:28:56.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.924+0000 7fc048ff9640 1 --2- 172.21.15.89:0/2057682890 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc02c0664a0 0x7fc02c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.924+0000 7fc048ff9640 1 -- 172.21.15.89:0/2057682890 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc0440834c0 con 0x7fc04c153bf0 2023-10-30T12:28:56.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.924+0000 7fc04b7fe640 1 --2- 172.21.15.89:0/2057682890 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc02c0664a0 0x7fc02c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.924+0000 7fc04b7fe640 1 --2- 172.21.15.89:0/2057682890 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc02c0664a0 0x7fc02c068960 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc03c0b2000 tx=0x7fc03c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:56.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:56.928+0000 7fc048ff9640 1 -- 172.21.15.89:0/2057682890 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc044050830 con 0x7fc04c153bf0 2023-10-30T12:28:57.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.080+0000 7fc051b0f640 1 -- 172.21.15.89:0/2057682890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc04c159dc0 con 0x7fc04c153bf0 2023-10-30T12:28:57.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.080+0000 7fc048ff9640 1 -- 172.21.15.89:0/2057682890 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16505 (secure 0 0 0) 0x7fc044053e50 con 0x7fc04c153bf0 2023-10-30T12:28:57.194 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:28:57.195 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":22,"fsid":"401641be-db6e-4c16-b3ac-fd84a86c4de4","created":"2023-10-30T12:28:26.198532+0000","modified":"2023-10-30T12:28:51.918285+0000","last_up_change":"2023-10-30T12:28:37.212205+0000","last_in_change":"2023-10-30T12:28:28.517734+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:39.004943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T12:28:48.935909+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T12:28:49.716098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72595c54-fd6e-4a5a-bb14-ff27e9377b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6801","nonce":1673955622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6803","nonce":1673955622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6807","nonce":1673955622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6805","nonce":1673955622}]},"public_addr":"172.21.15.89:6801/1673955622","cluster_addr":"172.21.15.89:6803/1673955622","heartbeat_back_addr":"172.21.15.89:6807/1673955622","heartbeat_front_addr":"172.21.15.89:6805/1673955622","state":["exists","up"]},{"osd":1,"uuid":"0305670b-d7c4-4193-bec2-8b7351ff590a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6817","nonce":3340572039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6819","nonce":3340572039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6823","nonce":3340572039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6821","nonce":3340572039}]},"public_addr":"172.21.15.89:6817/3340572039","cluster_addr":"172.21.15.89:6819/3340572039","heartbeat_back_addr":"172.21.15.89:6823/3340572039","heartbeat_front_addr":"172.21.15.89:6821/3340572039","state":["exists","up"]},{"osd":2,"uuid":"4bab5426-a3c3-46b0-91fc-546c254006c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6809","nonce":1929940799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6811","nonce":1929940799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6815","nonce":1929940799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6813","nonce":1929940799}]},"public_addr":"172.21.15.89:6809/1929940799","cluster_addr":"172.21.15.89:6811/1929940799","heartbeat_back_addr":"172.21.15.89:6815/1929940799","heartbeat_front_addr":"172.21.15.89:6813/1929940799","state":["exists","up"]},{"osd":3,"uuid":"7d47aea8-27cd-42be-8591-d63ad68c90e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6825","nonce":345087939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6827","nonce":345087939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6831","nonce":345087939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6829","nonce":345087939}]},"public_addr":"172.21.15.89:6825/345087939","cluster_addr":"172.21.15.89:6827/345087939","heartbeat_back_addr":"172.21.15.89:6831/345087939","heartbeat_front_addr":"172.21.15.89:6829/345087939","state":["exists","up"]},{"osd":4,"uuid":"2e2ae8f8-d097-4bba-9448-95047fbfacad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6801","nonce":2138125545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6803","nonce":2138125545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6809","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6811","nonce":2138125545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6805","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6807","nonce":2138125545}]},"public_addr":"172.21.15.192:6801/2138125545","cluster_addr":"172.21.15.192:6803/2138125545","heartbeat_back_addr":"172.21.15.192:6811/2138125545","heartbeat_front_addr":"172.21.15.192:6807/2138125545","state":["exists","up"]},{"osd":5,"uuid":"49be5ce5-12f4-4c26-8d39-864b6ff67313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6806","nonce":2360218671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6810","nonce":2360218671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6815","nonce":2360218671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6813","nonce":2360218671}]},"public_addr":"172.21.15.192:6806/2360218671","cluster_addr":"172.21.15.192:6810/2360218671","heartbeat_back_addr":"172.21.15.192:6815/2360218671","heartbeat_front_addr":"172.21.15.192:6813/2360218671","state":["exists","up"]},{"osd":6,"uuid":"48623452-d456-4ea2-a0de-5fa4b37e6d10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6825","nonce":3707671287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6827","nonce":3707671287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6831","nonce":3707671287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6829","nonce":3707671287}]},"public_addr":"172.21.15.192:6825/3707671287","cluster_addr":"172.21.15.192:6827/3707671287","heartbeat_back_addr":"172.21.15.192:6831/3707671287","heartbeat_front_addr":"172.21.15.192:6829/3707671287","state":["exists","up"]},{"osd":7,"uuid":"ff0a4384-54f8-43ec-a3ce-9c8ae3bf8a94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6817","nonce":3111945871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6819","nonce":3111945871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6823","nonce":3111945871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6821","nonce":3111945871}]},"public_addr":"172.21.15.192:6817/3111945871","cluster_addr":"172.21.15.192:6819/3111945871","heartbeat_back_addr":"172.21.15.192:6823/3111945871","heartbeat_front_addr":"172.21.15.192:6821/3111945871","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.183065+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.248650+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.194576+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.397483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.043648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.186624+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.159674+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.100851+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:28:57.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.084+0000 7fc051b0f640 1 -- 172.21.15.89:0/2057682890 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc02c0664a0 msgr2=0x7fc02c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:57.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.084+0000 7fc051b0f640 1 --2- 172.21.15.89:0/2057682890 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc02c0664a0 0x7fc02c068960 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc03c0b2000 tx=0x7fc03c008040 comp rx=0 tx=0).stop 2023-10-30T12:28:57.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.084+0000 7fc051b0f640 1 -- 172.21.15.89:0/2057682890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc04c153bf0 msgr2=0x7fc04c155540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:57.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.084+0000 7fc051b0f640 1 --2- 172.21.15.89:0/2057682890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc04c153bf0 0x7fc04c155540 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc04400c810 tx=0x7fc04400ccd0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.084+0000 7fc051b0f640 1 -- 172.21.15.89:0/2057682890 shutdown_connections 2023-10-30T12:28:57.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.084+0000 7fc051b0f640 1 --2- 172.21.15.89:0/2057682890 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc02c0664a0 0x7fc02c068960 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.084+0000 7fc051b0f640 1 --2- 172.21.15.89:0/2057682890 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc04c0c79e0 0x7fc04c0c56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.084+0000 7fc051b0f640 1 --2- 172.21.15.89:0/2057682890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc04c153bf0 0x7fc04c155540 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.084+0000 7fc051b0f640 1 --2- 172.21.15.89:0/2057682890 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc04c0c6fe0 0x7fc04c0c5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.084+0000 7fc051b0f640 1 -- 172.21.15.89:0/2057682890 >> 172.21.15.89:0/2057682890 conn(0x7fc04c06c6e0 msgr2=0x7fc04c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:57.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.084+0000 7fc051b0f640 1 -- 172.21.15.89:0/2057682890 shutdown_connections 2023-10-30T12:28:57.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.084+0000 7fc051b0f640 1 -- 172.21.15.89:0/2057682890 wait complete. 2023-10-30T12:28:57.199 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-30T12:28:57.200 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:28:57.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.332+0000 7fd0c606b640 1 Processor -- start 2023-10-30T12:28:57.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.336+0000 7fd0c606b640 1 -- start start 2023-10-30T12:28:57.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.336+0000 7fd0c606b640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd0c015d6f0 0x7fd0c015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:57.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.336+0000 7fd0c606b640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd0c00cf160 0x7fd0c00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:57.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.336+0000 7fd0c606b640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0c00cb800 con 0x7fd0c01538a0 2023-10-30T12:28:57.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.336+0000 7fd0c606b640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0c00cb330 con 0x7fd0c00cf160 2023-10-30T12:28:57.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.336+0000 7fd0c606b640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0c00cad70 con 0x7fd0c015d6f0 2023-10-30T12:28:57.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.336+0000 7fd0bffff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd0c00cf160 0x7fd0c00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:57.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.336+0000 7fd0bffff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd0c00cf160 0x7fd0c00cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55324/0 (socket says 172.21.15.89:55324) 2023-10-30T12:28:57.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.336+0000 7fd0bffff640 1 -- 172.21.15.89:0/577799271 learned_addr learned my addr 172.21.15.89:0/577799271 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:57.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.336+0000 7fd0bf7fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd0c01538a0 0x7fd0c00ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55694/0 (socket says 172.21.15.89:55694) 2023-10-30T12:28:57.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.336+0000 7fd0beffd640 1 --2- 172.21.15.89:0/577799271 >> v2:172.21.15.192:3300/0 conn(0x7fd0c015d6f0 0x7fd0c015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:57.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.336+0000 7fd0bffff640 1 -- 172.21.15.89:0/577799271 >> v2:172.21.15.192:3300/0 conn(0x7fd0c015d6f0 msgr2=0x7fd0c015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:57.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.336+0000 7fd0bffff640 1 --2- 172.21.15.89:0/577799271 >> v2:172.21.15.192:3300/0 conn(0x7fd0c015d6f0 0x7fd0c015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.336+0000 7fd0bffff640 1 -- 172.21.15.89:0/577799271 >> v1:172.21.15.89:6789/0 conn(0x7fd0c01538a0 legacy=0x7fd0c00ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:57.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.336+0000 7fd0bffff640 1 -- 172.21.15.89:0/577799271 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c00cb050 con 0x7fd0c00cf160 2023-10-30T12:28:57.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.336+0000 7fd0be7fc640 1 -- 172.21.15.89:0/577799271 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2785049212 0 0) 0x7fd0c00cb800 con 0x7fd0c01538a0 2023-10-30T12:28:57.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.340+0000 7fd0bffff640 1 --2- 172.21.15.89:0/577799271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0c00cf160 0x7fd0c00cf590 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd0c00b30a0 tx=0x7fd0b801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77761a83f7b7bdf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:57.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.340+0000 7fd0be7fc640 1 -- 172.21.15.89:0/577799271 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b80af400 con 0x7fd0c00cf160 2023-10-30T12:28:57.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.340+0000 7fd0be7fc640 1 -- 172.21.15.89:0/577799271 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0b80ab050 con 0x7fd0c00cf160 2023-10-30T12:28:57.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.340+0000 7fd0be7fc640 1 -- 172.21.15.89:0/577799271 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b80af780 con 0x7fd0c00cf160 2023-10-30T12:28:57.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.340+0000 7fd0c606b640 1 -- 172.21.15.89:0/577799271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0c00cf160 msgr2=0x7fd0c00cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:57.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.340+0000 7fd0c606b640 1 --2- 172.21.15.89:0/577799271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0c00cf160 0x7fd0c00cf590 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd0c00b30a0 tx=0x7fd0b801f1c0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.340+0000 7fd0c606b640 1 -- 172.21.15.89:0/577799271 shutdown_connections 2023-10-30T12:28:57.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.340+0000 7fd0c606b640 1 --2- 172.21.15.89:0/577799271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0c00cf160 0x7fd0c00cf590 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.340+0000 7fd0c606b640 1 --2- 172.21.15.89:0/577799271 >> v2:172.21.15.192:3300/0 conn(0x7fd0c015d6f0 0x7fd0c015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.340+0000 7fd0c606b640 1 -- 172.21.15.89:0/577799271 >> 172.21.15.89:0/577799271 conn(0x7fd0c006bfe0 msgr2=0x7fd0c006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:57.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.340+0000 7fd0c606b640 1 -- 172.21.15.89:0/577799271 shutdown_connections 2023-10-30T12:28:57.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.340+0000 7fd0c606b640 1 -- 172.21.15.89:0/577799271 wait complete. 2023-10-30T12:28:57.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.340+0000 7fd0c606b640 1 Processor -- start 2023-10-30T12:28:57.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.340+0000 7fd0c606b640 1 -- start start 2023-10-30T12:28:57.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.344+0000 7fd0c606b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd0c00cf160 0x7fd0c01504e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:57.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.344+0000 7fd0c606b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0c01538a0 0x7fd0c0155df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:57.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.344+0000 7fd0c606b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0c015d6f0 0x7fd0c0156330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:57.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.344+0000 7fd0c606b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c014d700 con 0x7fd0c01538a0 2023-10-30T12:28:57.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.344+0000 7fd0c606b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c01544f0 con 0x7fd0c015d6f0 2023-10-30T12:28:57.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.344+0000 7fd0c606b640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c0153f90 con 0x7fd0c00cf160 2023-10-30T12:28:57.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.344+0000 7fd0beffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0c01538a0 0x7fd0c0155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:57.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.344+0000 7fd0beffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0c01538a0 0x7fd0c0155df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55338/0 (socket says 172.21.15.89:55338) 2023-10-30T12:28:57.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.344+0000 7fd0beffd640 1 -- 172.21.15.89:0/3376714031 learned_addr learned my addr 172.21.15.89:0/3376714031 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:57.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.344+0000 7fd0bf7fe640 1 --2- 172.21.15.89:0/3376714031 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd0c00cf160 0x7fd0c01504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:57.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.344+0000 7fd0bffff640 1 --2- 172.21.15.89:0/3376714031 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0c015d6f0 0x7fd0c0156330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:57.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.344+0000 7fd0beffd640 1 -- 172.21.15.89:0/3376714031 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd0c00cf160 msgr2=0x7fd0c01504e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:57.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.344+0000 7fd0beffd640 1 --2- 172.21.15.89:0/3376714031 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd0c00cf160 0x7fd0c01504e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.344+0000 7fd0beffd640 1 -- 172.21.15.89:0/3376714031 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0c015d6f0 msgr2=0x7fd0c0156330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:57.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.344+0000 7fd0beffd640 1 --2- 172.21.15.89:0/3376714031 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0c015d6f0 0x7fd0c0156330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.344+0000 7fd0beffd640 1 -- 172.21.15.89:0/3376714031 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b8009b30 con 0x7fd0c01538a0 2023-10-30T12:28:57.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.344+0000 7fd0beffd640 1 --2- 172.21.15.89:0/3376714031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0c01538a0 0x7fd0c0155df0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd0b400d5f0 tx=0x7fd0b400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:57.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.344+0000 7fd0a7fff640 1 -- 172.21.15.89:0/3376714031 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b40125f0 con 0x7fd0c01538a0 2023-10-30T12:28:57.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.344+0000 7fd0a7fff640 1 -- 172.21.15.89:0/3376714031 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0b4012750 con 0x7fd0c01538a0 2023-10-30T12:28:57.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.344+0000 7fd0a7fff640 1 -- 172.21.15.89:0/3376714031 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b4012ae0 con 0x7fd0c01538a0 2023-10-30T12:28:57.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.344+0000 7fd0c606b640 1 -- 172.21.15.89:0/3376714031 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0c0154230 con 0x7fd0c01538a0 2023-10-30T12:28:57.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.344+0000 7fd0c606b640 1 -- 172.21.15.89:0/3376714031 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0c01590b0 con 0x7fd0c01538a0 2023-10-30T12:28:57.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.344+0000 7fd0c606b640 1 -- 172.21.15.89:0/3376714031 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0c00b2a50 con 0x7fd0c01538a0 2023-10-30T12:28:57.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.344+0000 7fd0a7fff640 1 -- 172.21.15.89:0/3376714031 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd0b4012c40 con 0x7fd0c01538a0 2023-10-30T12:28:57.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.348+0000 7fd0a7fff640 1 --2- 172.21.15.89:0/3376714031 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd0ac0664a0 0x7fd0ac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:57.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.348+0000 7fd0a7fff640 1 -- 172.21.15.89:0/3376714031 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd0b4084410 con 0x7fd0c01538a0 2023-10-30T12:28:57.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.348+0000 7fd0bf7fe640 1 --2- 172.21.15.89:0/3376714031 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd0ac0664a0 0x7fd0ac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:57.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.348+0000 7fd0bf7fe640 1 --2- 172.21.15.89:0/3376714031 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd0ac0664a0 0x7fd0ac068960 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0007fd0 tx=0x7fd0b0007d60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:57.357 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.352+0000 7fd0a7fff640 1 -- 172.21.15.89:0/3376714031 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd0b4051760 con 0x7fd0c01538a0 2023-10-30T12:28:57.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.500+0000 7fd0c606b640 1 -- 172.21.15.89:0/3376714031 --> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd0c0159b70 con 0x7fd0ac0664a0 2023-10-30T12:28:57.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.524+0000 7fd0a7fff640 1 -- 172.21.15.89:0/3376714031 <== mgr.4105 v2:172.21.15.192:6832/4076276638 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264444 (secure 0 0 0) 0x7fd0b0004c80 con 0x7fd0ac0664a0 2023-10-30T12:28:57.528 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:28:57.530 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-10-30T12:28:57.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.524+0000 7fd0c606b640 1 -- 172.21.15.89:0/3376714031 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd0ac0664a0 msgr2=0x7fd0ac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:57.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.524+0000 7fd0c606b640 1 --2- 172.21.15.89:0/3376714031 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd0ac0664a0 0x7fd0ac068960 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0007fd0 tx=0x7fd0b0007d60 comp rx=0 tx=0).stop 2023-10-30T12:28:57.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.524+0000 7fd0c606b640 1 -- 172.21.15.89:0/3376714031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0c01538a0 msgr2=0x7fd0c0155df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:57.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.524+0000 7fd0c606b640 1 --2- 172.21.15.89:0/3376714031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0c01538a0 0x7fd0c0155df0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd0b400d5f0 tx=0x7fd0b400dab0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.524+0000 7fd0c606b640 1 -- 172.21.15.89:0/3376714031 shutdown_connections 2023-10-30T12:28:57.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.524+0000 7fd0c606b640 1 --2- 172.21.15.89:0/3376714031 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd0ac0664a0 0x7fd0ac068960 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.524+0000 7fd0c606b640 1 --2- 172.21.15.89:0/3376714031 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0c015d6f0 0x7fd0c0156330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.524+0000 7fd0c606b640 1 --2- 172.21.15.89:0/3376714031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0c01538a0 0x7fd0c0155df0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.524+0000 7fd0c606b640 1 --2- 172.21.15.89:0/3376714031 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd0c00cf160 0x7fd0c01504e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.524+0000 7fd0c606b640 1 -- 172.21.15.89:0/3376714031 >> 172.21.15.89:0/3376714031 conn(0x7fd0c006bfe0 msgr2=0x7fd0c00c49f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:57.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.524+0000 7fd0c606b640 1 -- 172.21.15.89:0/3376714031 shutdown_connections 2023-10-30T12:28:57.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.524+0000 7fd0c606b640 1 -- 172.21.15.89:0/3376714031 wait complete. 2023-10-30T12:28:57.548 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-10-30T12:28:56.920690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":111,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219048,"kb_used_data":2836,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755672,"statfs":{"total":773094113280,"available":772869808128,"internally_reserved":0,"allocated":2904064,"data_stored":1771991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12741,"internal_metadata":221236795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":49,"apply_latency_ms":49,"commit_latency_ns":49000000,"apply_latency_ns":49000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.577185"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.984200+0000","last_change":"2023-10-30T12:28:50.979797+0000","last_active":"2023-10-30T12:28:50.984200+0000","last_peered":"2023-10-30T12:28:50.984200+0000","last_clean":"2023-10-30T12:28:50.984200+0000","last_became_active":"2023-10-30T12:28:50.979351+0000","last_became_peered":"2023-10-30T12:28:50.979351+0000","last_unstale":"2023-10-30T12:28:50.984200+0000","last_undegraded":"2023-10-30T12:28:50.984200+0000","last_fullsized":"2023-10-30T12:28:50.984200+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:33:02.139822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.932043+0000","last_change":"2023-10-30T12:28:50.929403+0000","last_active":"2023-10-30T12:28:51.932043+0000","last_peered":"2023-10-30T12:28:51.932043+0000","last_clean":"2023-10-30T12:28:51.932043+0000","last_became_active":"2023-10-30T12:28:50.929133+0000","last_became_peered":"2023-10-30T12:28:50.929133+0000","last_unstale":"2023-10-30T12:28:51.932043+0000","last_undegraded":"2023-10-30T12:28:51.932043+0000","last_fullsized":"2023-10-30T12:28:51.932043+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:12:57.327440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.936209+0000","last_change":"2023-10-30T12:28:50.934201+0000","last_active":"2023-10-30T12:28:50.936209+0000","last_peered":"2023-10-30T12:28:50.936209+0000","last_clean":"2023-10-30T12:28:50.936209+0000","last_became_active":"2023-10-30T12:28:50.933896+0000","last_became_peered":"2023-10-30T12:28:50.933896+0000","last_unstale":"2023-10-30T12:28:50.936209+0000","last_undegraded":"2023-10-30T12:28:50.936209+0000","last_fullsized":"2023-10-30T12:28:50.936209+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:50:22.654643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.978480+0000","last_change":"2023-10-30T12:28:50.969193+0000","last_active":"2023-10-30T12:28:50.978480+0000","last_peered":"2023-10-30T12:28:50.978480+0000","last_clean":"2023-10-30T12:28:50.978480+0000","last_became_active":"2023-10-30T12:28:50.967774+0000","last_became_peered":"2023-10-30T12:28:50.967774+0000","last_unstale":"2023-10-30T12:28:50.978480+0000","last_undegraded":"2023-10-30T12:28:50.978480+0000","last_fullsized":"2023-10-30T12:28:50.978480+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:32:01.665321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.983029+0000","last_change":"2023-10-30T12:28:50.973751+0000","last_active":"2023-10-30T12:28:50.983029+0000","last_peered":"2023-10-30T12:28:50.983029+0000","last_clean":"2023-10-30T12:28:50.983029+0000","last_became_active":"2023-10-30T12:28:50.973132+0000","last_became_peered":"2023-10-30T12:28:50.973132+0000","last_unstale":"2023-10-30T12:28:50.983029+0000","last_undegraded":"2023-10-30T12:28:50.983029+0000","last_fullsized":"2023-10-30T12:28:50.983029+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:50:49.899151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.978576+0000","last_change":"2023-10-30T12:28:50.968823+0000","last_active":"2023-10-30T12:28:50.978576+0000","last_peered":"2023-10-30T12:28:50.978576+0000","last_clean":"2023-10-30T12:28:50.978576+0000","last_became_active":"2023-10-30T12:28:50.967910+0000","last_became_peered":"2023-10-30T12:28:50.967910+0000","last_unstale":"2023-10-30T12:28:50.978576+0000","last_undegraded":"2023-10-30T12:28:50.978576+0000","last_fullsized":"2023-10-30T12:28:50.978576+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:59:35.262638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.983085+0000","last_change":"2023-10-30T12:28:50.974164+0000","last_active":"2023-10-30T12:28:50.983085+0000","last_peered":"2023-10-30T12:28:50.983085+0000","last_clean":"2023-10-30T12:28:50.983085+0000","last_became_active":"2023-10-30T12:28:50.973288+0000","last_became_peered":"2023-10-30T12:28:50.973288+0000","last_unstale":"2023-10-30T12:28:50.983085+0000","last_undegraded":"2023-10-30T12:28:50.983085+0000","last_fullsized":"2023-10-30T12:28:50.983085+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:38:25.761235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.958001+0000","last_change":"2023-10-30T12:28:50.955287+0000","last_active":"2023-10-30T12:28:50.958001+0000","last_peered":"2023-10-30T12:28:50.958001+0000","last_clean":"2023-10-30T12:28:50.958001+0000","last_became_active":"2023-10-30T12:28:50.954988+0000","last_became_peered":"2023-10-30T12:28:50.954988+0000","last_unstale":"2023-10-30T12:28:50.958001+0000","last_undegraded":"2023-10-30T12:28:50.958001+0000","last_fullsized":"2023-10-30T12:28:50.958001+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:46:16.004666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.983305+0000","last_change":"2023-10-30T12:28:50.977144+0000","last_active":"2023-10-30T12:28:50.983305+0000","last_peered":"2023-10-30T12:28:50.983305+0000","last_clean":"2023-10-30T12:28:50.983305+0000","last_became_active":"2023-10-30T12:28:50.976564+0000","last_became_peered":"2023-10-30T12:28:50.976564+0000","last_unstale":"2023-10-30T12:28:50.983305+0000","last_undegraded":"2023-10-30T12:28:50.983305+0000","last_fullsized":"2023-10-30T12:28:50.983305+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:50.229780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.938542+0000","last_change":"2023-10-30T12:28:50.952564+0000","last_active":"2023-10-30T12:28:51.938542+0000","last_peered":"2023-10-30T12:28:51.938542+0000","last_clean":"2023-10-30T12:28:51.938542+0000","last_became_active":"2023-10-30T12:28:50.951809+0000","last_became_peered":"2023-10-30T12:28:50.951809+0000","last_unstale":"2023-10-30T12:28:51.938542+0000","last_undegraded":"2023-10-30T12:28:51.938542+0000","last_fullsized":"2023-10-30T12:28:51.938542+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:34:29.771577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.937215+0000","last_change":"2023-10-30T12:28:50.970779+0000","last_active":"2023-10-30T12:28:51.937215+0000","last_peered":"2023-10-30T12:28:51.937215+0000","last_clean":"2023-10-30T12:28:51.937215+0000","last_became_active":"2023-10-30T12:28:50.969877+0000","last_became_peered":"2023-10-30T12:28:50.969877+0000","last_unstale":"2023-10-30T12:28:51.937215+0000","last_undegraded":"2023-10-30T12:28:51.937215+0000","last_fullsized":"2023-10-30T12:28:51.937215+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:10.539915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.940164+0000","last_change":"2023-10-30T12:28:50.944452+0000","last_active":"2023-10-30T12:28:51.940164+0000","last_peered":"2023-10-30T12:28:51.940164+0000","last_clean":"2023-10-30T12:28:51.940164+0000","last_became_active":"2023-10-30T12:28:50.943787+0000","last_became_peered":"2023-10-30T12:28:50.943787+0000","last_unstale":"2023-10-30T12:28:51.940164+0000","last_undegraded":"2023-10-30T12:28:51.940164+0000","last_fullsized":"2023-10-30T12:28:51.940164+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:30:34.585298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.943369+0000","last_change":"2023-10-30T12:28:50.938181+0000","last_active":"2023-10-30T12:28:50.943369+0000","last_peered":"2023-10-30T12:28:50.943369+0000","last_clean":"2023-10-30T12:28:50.943369+0000","last_became_active":"2023-10-30T12:28:50.937906+0000","last_became_peered":"2023-10-30T12:28:50.937906+0000","last_unstale":"2023-10-30T12:28:50.943369+0000","last_undegraded":"2023-10-30T12:28:50.943369+0000","last_fullsized":"2023-10-30T12:28:50.943369+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:35:50.470679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.936206+0000","last_change":"2023-10-30T12:28:50.934085+0000","last_active":"2023-10-30T12:28:50.936206+0000","last_peered":"2023-10-30T12:28:50.936206+0000","last_clean":"2023-10-30T12:28:50.936206+0000","last_became_active":"2023-10-30T12:28:50.932753+0000","last_became_peered":"2023-10-30T12:28:50.932753+0000","last_unstale":"2023-10-30T12:28:50.936206+0000","last_undegraded":"2023-10-30T12:28:50.936206+0000","last_fullsized":"2023-10-30T12:28:50.936206+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:24.146920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931710+0000","last_change":"2023-10-30T12:28:50.943343+0000","last_active":"2023-10-30T12:28:51.931710+0000","last_peered":"2023-10-30T12:28:51.931710+0000","last_clean":"2023-10-30T12:28:51.931710+0000","last_became_active":"2023-10-30T12:28:50.943011+0000","last_became_peered":"2023-10-30T12:28:50.943011+0000","last_unstale":"2023-10-30T12:28:51.931710+0000","last_undegraded":"2023-10-30T12:28:51.931710+0000","last_fullsized":"2023-10-30T12:28:51.931710+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:59:35.881115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.974555+0000","last_change":"2023-10-30T12:28:50.953573+0000","last_active":"2023-10-30T12:28:50.974555+0000","last_peered":"2023-10-30T12:28:50.974555+0000","last_clean":"2023-10-30T12:28:50.974555+0000","last_became_active":"2023-10-30T12:28:50.952744+0000","last_became_peered":"2023-10-30T12:28:50.952744+0000","last_unstale":"2023-10-30T12:28:50.974555+0000","last_undegraded":"2023-10-30T12:28:50.974555+0000","last_fullsized":"2023-10-30T12:28:50.974555+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:13:11.797130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.978742+0000","last_change":"2023-10-30T12:28:50.975156+0000","last_active":"2023-10-30T12:28:50.978742+0000","last_peered":"2023-10-30T12:28:50.978742+0000","last_clean":"2023-10-30T12:28:50.978742+0000","last_became_active":"2023-10-30T12:28:50.974576+0000","last_became_peered":"2023-10-30T12:28:50.974576+0000","last_unstale":"2023-10-30T12:28:50.978742+0000","last_undegraded":"2023-10-30T12:28:50.978742+0000","last_fullsized":"2023-10-30T12:28:50.978742+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:26:53.504301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.938728+0000","last_change":"2023-10-30T12:28:50.937685+0000","last_active":"2023-10-30T12:28:50.938728+0000","last_peered":"2023-10-30T12:28:50.938728+0000","last_clean":"2023-10-30T12:28:50.938728+0000","last_became_active":"2023-10-30T12:28:50.937384+0000","last_became_peered":"2023-10-30T12:28:50.937384+0000","last_unstale":"2023-10-30T12:28:50.938728+0000","last_undegraded":"2023-10-30T12:28:50.938728+0000","last_fullsized":"2023-10-30T12:28:50.938728+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:33:59.082334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.983151+0000","last_change":"2023-10-30T12:28:50.974918+0000","last_active":"2023-10-30T12:28:50.983151+0000","last_peered":"2023-10-30T12:28:50.983151+0000","last_clean":"2023-10-30T12:28:50.983151+0000","last_became_active":"2023-10-30T12:28:50.973369+0000","last_became_peered":"2023-10-30T12:28:50.973369+0000","last_unstale":"2023-10-30T12:28:50.983151+0000","last_undegraded":"2023-10-30T12:28:50.983151+0000","last_fullsized":"2023-10-30T12:28:50.983151+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:54:34.723806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.937226+0000","last_change":"2023-10-30T12:28:50.955835+0000","last_active":"2023-10-30T12:28:51.937226+0000","last_peered":"2023-10-30T12:28:51.937226+0000","last_clean":"2023-10-30T12:28:51.937226+0000","last_became_active":"2023-10-30T12:28:50.954831+0000","last_became_peered":"2023-10-30T12:28:50.954831+0000","last_unstale":"2023-10-30T12:28:51.937226+0000","last_undegraded":"2023-10-30T12:28:51.937226+0000","last_fullsized":"2023-10-30T12:28:51.937226+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:56:25.554582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.963406+0000","last_change":"2023-10-30T12:28:50.962417+0000","last_active":"2023-10-30T12:28:50.963406+0000","last_peered":"2023-10-30T12:28:50.963406+0000","last_clean":"2023-10-30T12:28:50.963406+0000","last_became_active":"2023-10-30T12:28:50.962065+0000","last_became_peered":"2023-10-30T12:28:50.962065+0000","last_unstale":"2023-10-30T12:28:50.963406+0000","last_undegraded":"2023-10-30T12:28:50.963406+0000","last_fullsized":"2023-10-30T12:28:50.963406+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:46:22.958672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.978608+0000","last_change":"2023-10-30T12:28:50.968992+0000","last_active":"2023-10-30T12:28:50.978608+0000","last_peered":"2023-10-30T12:28:50.978608+0000","last_clean":"2023-10-30T12:28:50.978608+0000","last_became_active":"2023-10-30T12:28:50.967651+0000","last_became_peered":"2023-10-30T12:28:50.967651+0000","last_unstale":"2023-10-30T12:28:50.978608+0000","last_undegraded":"2023-10-30T12:28:50.978608+0000","last_fullsized":"2023-10-30T12:28:50.978608+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:42.061451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.963293+0000","last_change":"2023-10-30T12:28:50.962609+0000","last_active":"2023-10-30T12:28:50.963293+0000","last_peered":"2023-10-30T12:28:50.963293+0000","last_clean":"2023-10-30T12:28:50.963293+0000","last_became_active":"2023-10-30T12:28:50.962290+0000","last_became_peered":"2023-10-30T12:28:50.962290+0000","last_unstale":"2023-10-30T12:28:50.963293+0000","last_undegraded":"2023-10-30T12:28:50.963293+0000","last_fullsized":"2023-10-30T12:28:50.963293+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:57:13.022671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.978406+0000","last_change":"2023-10-30T12:28:50.966653+0000","last_active":"2023-10-30T12:28:50.978406+0000","last_peered":"2023-10-30T12:28:50.978406+0000","last_clean":"2023-10-30T12:28:50.978406+0000","last_became_active":"2023-10-30T12:28:50.965713+0000","last_became_peered":"2023-10-30T12:28:50.965713+0000","last_unstale":"2023-10-30T12:28:50.978406+0000","last_undegraded":"2023-10-30T12:28:50.978406+0000","last_fullsized":"2023-10-30T12:28:50.978406+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:42:07.786343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.952821+0000","last_change":"2023-10-30T12:28:50.951300+0000","last_active":"2023-10-30T12:28:50.952821+0000","last_peered":"2023-10-30T12:28:50.952821+0000","last_clean":"2023-10-30T12:28:50.952821+0000","last_became_active":"2023-10-30T12:28:50.951026+0000","last_became_peered":"2023-10-30T12:28:50.951026+0000","last_unstale":"2023-10-30T12:28:50.952821+0000","last_undegraded":"2023-10-30T12:28:50.952821+0000","last_fullsized":"2023-10-30T12:28:50.952821+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:29:11.435330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.938879+0000","last_change":"2023-10-30T12:28:50.963507+0000","last_active":"2023-10-30T12:28:51.938879+0000","last_peered":"2023-10-30T12:28:51.938879+0000","last_clean":"2023-10-30T12:28:51.938879+0000","last_became_active":"2023-10-30T12:28:50.962484+0000","last_became_peered":"2023-10-30T12:28:50.962484+0000","last_unstale":"2023-10-30T12:28:51.938879+0000","last_undegraded":"2023-10-30T12:28:51.938879+0000","last_fullsized":"2023-10-30T12:28:51.938879+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:11:24.160220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931933+0000","last_change":"2023-10-30T12:28:50.933362+0000","last_active":"2023-10-30T12:28:51.931933+0000","last_peered":"2023-10-30T12:28:51.931933+0000","last_clean":"2023-10-30T12:28:51.931933+0000","last_became_active":"2023-10-30T12:28:50.933093+0000","last_became_peered":"2023-10-30T12:28:50.933093+0000","last_unstale":"2023-10-30T12:28:51.931933+0000","last_undegraded":"2023-10-30T12:28:51.931933+0000","last_fullsized":"2023-10-30T12:28:51.931933+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:54:00.852482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.955331+0000","last_change":"2023-10-30T12:28:50.952857+0000","last_active":"2023-10-30T12:28:50.955331+0000","last_peered":"2023-10-30T12:28:50.955331+0000","last_clean":"2023-10-30T12:28:50.955331+0000","last_became_active":"2023-10-30T12:28:50.952531+0000","last_became_peered":"2023-10-30T12:28:50.952531+0000","last_unstale":"2023-10-30T12:28:50.955331+0000","last_undegraded":"2023-10-30T12:28:50.955331+0000","last_fullsized":"2023-10-30T12:28:50.955331+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:44:51.152114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.935136+0000","last_change":"2023-10-30T12:28:50.933408+0000","last_active":"2023-10-30T12:28:50.935136+0000","last_peered":"2023-10-30T12:28:50.935136+0000","last_clean":"2023-10-30T12:28:50.935136+0000","last_became_active":"2023-10-30T12:28:50.933124+0000","last_became_peered":"2023-10-30T12:28:50.933124+0000","last_unstale":"2023-10-30T12:28:50.935136+0000","last_undegraded":"2023-10-30T12:28:50.935136+0000","last_fullsized":"2023-10-30T12:28:50.935136+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:39:57.679819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.978541+0000","last_change":"2023-10-30T12:28:50.966388+0000","last_active":"2023-10-30T12:28:50.978541+0000","last_peered":"2023-10-30T12:28:50.978541+0000","last_clean":"2023-10-30T12:28:50.978541+0000","last_became_active":"2023-10-30T12:28:50.965639+0000","last_became_peered":"2023-10-30T12:28:50.965639+0000","last_unstale":"2023-10-30T12:28:50.978541+0000","last_undegraded":"2023-10-30T12:28:50.978541+0000","last_fullsized":"2023-10-30T12:28:50.978541+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:46:28.766578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.957736+0000","last_change":"2023-10-30T12:28:50.955391+0000","last_active":"2023-10-30T12:28:50.957736+0000","last_peered":"2023-10-30T12:28:50.957736+0000","last_clean":"2023-10-30T12:28:50.957736+0000","last_became_active":"2023-10-30T12:28:50.955084+0000","last_became_peered":"2023-10-30T12:28:50.955084+0000","last_unstale":"2023-10-30T12:28:50.957736+0000","last_undegraded":"2023-10-30T12:28:50.957736+0000","last_fullsized":"2023-10-30T12:28:50.957736+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:08:17.139917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.936598+0000","last_change":"2023-10-30T12:28:50.945609+0000","last_active":"2023-10-30T12:28:51.936598+0000","last_peered":"2023-10-30T12:28:51.936598+0000","last_clean":"2023-10-30T12:28:51.936598+0000","last_became_active":"2023-10-30T12:28:50.944775+0000","last_became_peered":"2023-10-30T12:28:50.944775+0000","last_unstale":"2023-10-30T12:28:51.936598+0000","last_undegraded":"2023-10-30T12:28:51.936598+0000","last_fullsized":"2023-10-30T12:28:51.936598+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:08:55.718193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.939146+0000","last_change":"2023-10-30T12:28:50.941378+0000","last_active":"2023-10-30T12:28:51.939146+0000","last_peered":"2023-10-30T12:28:51.939146+0000","last_clean":"2023-10-30T12:28:51.939146+0000","last_became_active":"2023-10-30T12:28:50.940567+0000","last_became_peered":"2023-10-30T12:28:50.940567+0000","last_unstale":"2023-10-30T12:28:51.939146+0000","last_undegraded":"2023-10-30T12:28:51.939146+0000","last_fullsized":"2023-10-30T12:28:51.939146+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:31:23.772343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.930221+0000","last_change":"2023-10-30T12:28:50.395670+0000","last_active":"2023-10-30T12:28:50.930221+0000","last_peered":"2023-10-30T12:28:50.930221+0000","last_clean":"2023-10-30T12:28:50.930221+0000","last_became_active":"2023-10-30T12:28:50.395382+0000","last_became_peered":"2023-10-30T12:28:50.395382+0000","last_unstale":"2023-10-30T12:28:50.930221+0000","last_undegraded":"2023-10-30T12:28:50.930221+0000","last_fullsized":"2023-10-30T12:28:50.930221+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:14.858121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.957837+0000","last_change":"2023-10-30T12:28:50.953202+0000","last_active":"2023-10-30T12:28:50.957837+0000","last_peered":"2023-10-30T12:28:50.957837+0000","last_clean":"2023-10-30T12:28:50.957837+0000","last_became_active":"2023-10-30T12:28:50.952613+0000","last_became_peered":"2023-10-30T12:28:50.952613+0000","last_unstale":"2023-10-30T12:28:50.957837+0000","last_undegraded":"2023-10-30T12:28:50.957837+0000","last_fullsized":"2023-10-30T12:28:50.957837+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:07.934471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.938991+0000","last_change":"2023-10-30T12:28:50.384598+0000","last_active":"2023-10-30T12:28:51.938991+0000","last_peered":"2023-10-30T12:28:51.938991+0000","last_clean":"2023-10-30T12:28:51.938991+0000","last_became_active":"2023-10-30T12:28:50.383898+0000","last_became_peered":"2023-10-30T12:28:50.383898+0000","last_unstale":"2023-10-30T12:28:51.938991+0000","last_undegraded":"2023-10-30T12:28:51.938991+0000","last_fullsized":"2023-10-30T12:28:51.938991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:09:50.290004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.929895+0000","last_change":"2023-10-30T12:28:50.386193+0000","last_active":"2023-10-30T12:28:50.929895+0000","last_peered":"2023-10-30T12:28:50.929895+0000","last_clean":"2023-10-30T12:28:50.929895+0000","last_became_active":"2023-10-30T12:28:50.385963+0000","last_became_peered":"2023-10-30T12:28:50.385963+0000","last_unstale":"2023-10-30T12:28:50.929895+0000","last_undegraded":"2023-10-30T12:28:50.929895+0000","last_fullsized":"2023-10-30T12:28:50.929895+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:00:55.603212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.938352+0000","last_change":"2023-10-30T12:28:50.936038+0000","last_active":"2023-10-30T12:28:50.938352+0000","last_peered":"2023-10-30T12:28:50.938352+0000","last_clean":"2023-10-30T12:28:50.938352+0000","last_became_active":"2023-10-30T12:28:50.935764+0000","last_became_peered":"2023-10-30T12:28:50.935764+0000","last_unstale":"2023-10-30T12:28:50.938352+0000","last_undegraded":"2023-10-30T12:28:50.938352+0000","last_fullsized":"2023-10-30T12:28:50.938352+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:24:28.355705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.952323+0000","last_change":"2023-10-30T12:28:50.392213+0000","last_active":"2023-10-30T12:28:50.952323+0000","last_peered":"2023-10-30T12:28:50.952323+0000","last_clean":"2023-10-30T12:28:50.952323+0000","last_became_active":"2023-10-30T12:28:50.391494+0000","last_became_peered":"2023-10-30T12:28:50.391494+0000","last_unstale":"2023-10-30T12:28:50.952323+0000","last_undegraded":"2023-10-30T12:28:50.952323+0000","last_fullsized":"2023-10-30T12:28:50.952323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:38:37.301840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931619+0000","last_change":"2023-10-30T12:28:50.933319+0000","last_active":"2023-10-30T12:28:51.931619+0000","last_peered":"2023-10-30T12:28:51.931619+0000","last_clean":"2023-10-30T12:28:51.931619+0000","last_became_active":"2023-10-30T12:28:50.933062+0000","last_became_peered":"2023-10-30T12:28:50.933062+0000","last_unstale":"2023-10-30T12:28:51.931619+0000","last_undegraded":"2023-10-30T12:28:51.931619+0000","last_fullsized":"2023-10-30T12:28:51.931619+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:26:48.475038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.937437+0000","last_change":"2023-10-30T12:28:50.394368+0000","last_active":"2023-10-30T12:28:51.937437+0000","last_peered":"2023-10-30T12:28:51.937437+0000","last_clean":"2023-10-30T12:28:51.937437+0000","last_became_active":"2023-10-30T12:28:50.393686+0000","last_became_peered":"2023-10-30T12:28:50.393686+0000","last_unstale":"2023-10-30T12:28:51.937437+0000","last_undegraded":"2023-10-30T12:28:51.937437+0000","last_fullsized":"2023-10-30T12:28:51.937437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:43:17.726462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.975448+0000","last_change":"2023-10-30T12:28:50.963703+0000","last_active":"2023-10-30T12:28:50.975448+0000","last_peered":"2023-10-30T12:28:50.975448+0000","last_clean":"2023-10-30T12:28:50.975448+0000","last_became_active":"2023-10-30T12:28:50.963105+0000","last_became_peered":"2023-10-30T12:28:50.963105+0000","last_unstale":"2023-10-30T12:28:50.975448+0000","last_undegraded":"2023-10-30T12:28:50.975448+0000","last_fullsized":"2023-10-30T12:28:50.975448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:05:31.103581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.957188+0000","last_change":"2023-10-30T12:28:50.397679+0000","last_active":"2023-10-30T12:28:50.957188+0000","last_peered":"2023-10-30T12:28:50.957188+0000","last_clean":"2023-10-30T12:28:50.957188+0000","last_became_active":"2023-10-30T12:28:50.397041+0000","last_became_peered":"2023-10-30T12:28:50.397041+0000","last_unstale":"2023-10-30T12:28:50.957188+0000","last_undegraded":"2023-10-30T12:28:50.957188+0000","last_fullsized":"2023-10-30T12:28:50.957188+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:52:22.564616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.969024+0000","last_change":"2023-10-30T12:28:50.966838+0000","last_active":"2023-10-30T12:28:50.969024+0000","last_peered":"2023-10-30T12:28:50.969024+0000","last_clean":"2023-10-30T12:28:50.969024+0000","last_became_active":"2023-10-30T12:28:50.966569+0000","last_became_peered":"2023-10-30T12:28:50.966569+0000","last_unstale":"2023-10-30T12:28:50.969024+0000","last_undegraded":"2023-10-30T12:28:50.969024+0000","last_fullsized":"2023-10-30T12:28:50.969024+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:37:20.093654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931949+0000","last_change":"2023-10-30T12:28:50.391527+0000","last_active":"2023-10-30T12:28:51.931949+0000","last_peered":"2023-10-30T12:28:51.931949+0000","last_clean":"2023-10-30T12:28:51.931949+0000","last_became_active":"2023-10-30T12:28:50.391260+0000","last_became_peered":"2023-10-30T12:28:50.391260+0000","last_unstale":"2023-10-30T12:28:51.931949+0000","last_undegraded":"2023-10-30T12:28:51.931949+0000","last_fullsized":"2023-10-30T12:28:51.931949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:29:34.772490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.940211+0000","last_change":"2023-10-30T12:28:50.975441+0000","last_active":"2023-10-30T12:28:51.940211+0000","last_peered":"2023-10-30T12:28:51.940211+0000","last_clean":"2023-10-30T12:28:51.940211+0000","last_became_active":"2023-10-30T12:28:50.974843+0000","last_became_peered":"2023-10-30T12:28:50.974843+0000","last_unstale":"2023-10-30T12:28:51.940211+0000","last_undegraded":"2023-10-30T12:28:51.940211+0000","last_fullsized":"2023-10-30T12:28:51.940211+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:06:12.060752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.937470+0000","last_change":"2023-10-30T12:28:50.374838+0000","last_active":"2023-10-30T12:28:51.937470+0000","last_peered":"2023-10-30T12:28:51.937470+0000","last_clean":"2023-10-30T12:28:51.937470+0000","last_became_active":"2023-10-30T12:28:50.373581+0000","last_became_peered":"2023-10-30T12:28:50.373581+0000","last_unstale":"2023-10-30T12:28:51.937470+0000","last_undegraded":"2023-10-30T12:28:51.937470+0000","last_fullsized":"2023-10-30T12:28:51.937470+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:14:51.506745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.983033+0000","last_change":"2023-10-30T12:28:50.974012+0000","last_active":"2023-10-30T12:28:50.983033+0000","last_peered":"2023-10-30T12:28:50.983033+0000","last_clean":"2023-10-30T12:28:50.983033+0000","last_became_active":"2023-10-30T12:28:50.973269+0000","last_became_peered":"2023-10-30T12:28:50.973269+0000","last_unstale":"2023-10-30T12:28:50.983033+0000","last_undegraded":"2023-10-30T12:28:50.983033+0000","last_fullsized":"2023-10-30T12:28:50.983033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:36:49.701055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.925887+0000","last_change":"2023-10-30T12:28:50.366590+0000","last_active":"2023-10-30T12:28:50.925887+0000","last_peered":"2023-10-30T12:28:50.925887+0000","last_clean":"2023-10-30T12:28:50.925887+0000","last_became_active":"2023-10-30T12:28:50.366269+0000","last_became_peered":"2023-10-30T12:28:50.366269+0000","last_unstale":"2023-10-30T12:28:50.925887+0000","last_undegraded":"2023-10-30T12:28:50.925887+0000","last_fullsized":"2023-10-30T12:28:50.925887+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:15:11.963438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.963379+0000","last_change":"2023-10-30T12:28:50.962058+0000","last_active":"2023-10-30T12:28:50.963379+0000","last_peered":"2023-10-30T12:28:50.963379+0000","last_clean":"2023-10-30T12:28:50.963379+0000","last_became_active":"2023-10-30T12:28:50.961805+0000","last_became_peered":"2023-10-30T12:28:50.961805+0000","last_unstale":"2023-10-30T12:28:50.963379+0000","last_undegraded":"2023-10-30T12:28:50.963379+0000","last_fullsized":"2023-10-30T12:28:50.963379+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:42:35.249989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.956067+0000","last_change":"2023-10-30T12:28:50.397244+0000","last_active":"2023-10-30T12:28:50.956067+0000","last_peered":"2023-10-30T12:28:50.956067+0000","last_clean":"2023-10-30T12:28:50.956067+0000","last_became_active":"2023-10-30T12:28:50.396727+0000","last_became_peered":"2023-10-30T12:28:50.396727+0000","last_unstale":"2023-10-30T12:28:50.956067+0000","last_undegraded":"2023-10-30T12:28:50.956067+0000","last_fullsized":"2023-10-30T12:28:50.956067+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:33.493343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.984268+0000","last_change":"2023-10-30T12:28:50.981498+0000","last_active":"2023-10-30T12:28:50.984268+0000","last_peered":"2023-10-30T12:28:50.984268+0000","last_clean":"2023-10-30T12:28:50.984268+0000","last_became_active":"2023-10-30T12:28:50.981063+0000","last_became_peered":"2023-10-30T12:28:50.981063+0000","last_unstale":"2023-10-30T12:28:50.984268+0000","last_undegraded":"2023-10-30T12:28:50.984268+0000","last_fullsized":"2023-10-30T12:28:50.984268+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:13:43.443210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.937078+0000","last_change":"2023-10-30T12:28:50.383880+0000","last_active":"2023-10-30T12:28:51.937078+0000","last_peered":"2023-10-30T12:28:51.937078+0000","last_clean":"2023-10-30T12:28:51.937078+0000","last_became_active":"2023-10-30T12:28:50.383570+0000","last_became_peered":"2023-10-30T12:28:50.383570+0000","last_unstale":"2023-10-30T12:28:51.937078+0000","last_undegraded":"2023-10-30T12:28:51.937078+0000","last_fullsized":"2023-10-30T12:28:51.937078+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:19:42.047520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.936983+0000","last_change":"2023-10-30T12:28:50.950312+0000","last_active":"2023-10-30T12:28:51.936983+0000","last_peered":"2023-10-30T12:28:51.936983+0000","last_clean":"2023-10-30T12:28:51.936983+0000","last_became_active":"2023-10-30T12:28:50.949485+0000","last_became_peered":"2023-10-30T12:28:50.949485+0000","last_unstale":"2023-10-30T12:28:51.936983+0000","last_undegraded":"2023-10-30T12:28:51.936983+0000","last_fullsized":"2023-10-30T12:28:51.936983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:13:04.669915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.924804+0000","last_change":"2023-10-30T12:28:50.362979+0000","last_active":"2023-10-30T12:28:50.924804+0000","last_peered":"2023-10-30T12:28:50.924804+0000","last_clean":"2023-10-30T12:28:50.924804+0000","last_became_active":"2023-10-30T12:28:50.362624+0000","last_became_peered":"2023-10-30T12:28:50.362624+0000","last_unstale":"2023-10-30T12:28:50.924804+0000","last_undegraded":"2023-10-30T12:28:50.924804+0000","last_fullsized":"2023-10-30T12:28:50.924804+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:01:13.111816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.984219+0000","last_change":"2023-10-30T12:28:50.979815+0000","last_active":"2023-10-30T12:28:50.984219+0000","last_peered":"2023-10-30T12:28:50.984219+0000","last_clean":"2023-10-30T12:28:50.984219+0000","last_became_active":"2023-10-30T12:28:50.979344+0000","last_became_peered":"2023-10-30T12:28:50.979344+0000","last_unstale":"2023-10-30T12:28:50.984219+0000","last_undegraded":"2023-10-30T12:28:50.984219+0000","last_fullsized":"2023-10-30T12:28:50.984219+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:57:05.538953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.940171+0000","last_change":"2023-10-30T12:28:50.382175+0000","last_active":"2023-10-30T12:28:51.940171+0000","last_peered":"2023-10-30T12:28:51.940171+0000","last_clean":"2023-10-30T12:28:51.940171+0000","last_became_active":"2023-10-30T12:28:50.381125+0000","last_became_peered":"2023-10-30T12:28:50.381125+0000","last_unstale":"2023-10-30T12:28:51.940171+0000","last_undegraded":"2023-10-30T12:28:51.940171+0000","last_fullsized":"2023-10-30T12:28:51.940171+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:36:02.652665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.983341+0000","last_change":"2023-10-30T12:28:50.978034+0000","last_active":"2023-10-30T12:28:50.983341+0000","last_peered":"2023-10-30T12:28:50.983341+0000","last_clean":"2023-10-30T12:28:50.983341+0000","last_became_active":"2023-10-30T12:28:50.977585+0000","last_became_peered":"2023-10-30T12:28:50.977585+0000","last_unstale":"2023-10-30T12:28:50.983341+0000","last_undegraded":"2023-10-30T12:28:50.983341+0000","last_fullsized":"2023-10-30T12:28:50.983341+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:21:32.868831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.924936+0000","last_change":"2023-10-30T12:28:50.367258+0000","last_active":"2023-10-30T12:28:50.924936+0000","last_peered":"2023-10-30T12:28:50.924936+0000","last_clean":"2023-10-30T12:28:50.924936+0000","last_became_active":"2023-10-30T12:28:50.366974+0000","last_became_peered":"2023-10-30T12:28:50.366974+0000","last_unstale":"2023-10-30T12:28:50.924936+0000","last_undegraded":"2023-10-30T12:28:50.924936+0000","last_fullsized":"2023-10-30T12:28:50.924936+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:14:48.823902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.984257+0000","last_change":"2023-10-30T12:28:50.981784+0000","last_active":"2023-10-30T12:28:50.984257+0000","last_peered":"2023-10-30T12:28:50.984257+0000","last_clean":"2023-10-30T12:28:50.984257+0000","last_became_active":"2023-10-30T12:28:50.981193+0000","last_became_peered":"2023-10-30T12:28:50.981193+0000","last_unstale":"2023-10-30T12:28:50.984257+0000","last_undegraded":"2023-10-30T12:28:50.984257+0000","last_fullsized":"2023-10-30T12:28:50.984257+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:22:53.466806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.932011+0000","last_change":"2023-10-30T12:28:50.368011+0000","last_active":"2023-10-30T12:28:51.932011+0000","last_peered":"2023-10-30T12:28:51.932011+0000","last_clean":"2023-10-30T12:28:51.932011+0000","last_became_active":"2023-10-30T12:28:50.367730+0000","last_became_peered":"2023-10-30T12:28:50.367730+0000","last_unstale":"2023-10-30T12:28:51.932011+0000","last_undegraded":"2023-10-30T12:28:51.932011+0000","last_fullsized":"2023-10-30T12:28:51.932011+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:59:41.946550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.978412+0000","last_change":"2023-10-30T12:28:50.968262+0000","last_active":"2023-10-30T12:28:50.978412+0000","last_peered":"2023-10-30T12:28:50.978412+0000","last_clean":"2023-10-30T12:28:50.978412+0000","last_became_active":"2023-10-30T12:28:50.966246+0000","last_became_peered":"2023-10-30T12:28:50.966246+0000","last_unstale":"2023-10-30T12:28:50.978412+0000","last_undegraded":"2023-10-30T12:28:50.978412+0000","last_fullsized":"2023-10-30T12:28:50.978412+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:16:29.181523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.925526+0000","last_change":"2023-10-30T12:28:50.394060+0000","last_active":"2023-10-30T12:28:50.925526+0000","last_peered":"2023-10-30T12:28:50.925526+0000","last_clean":"2023-10-30T12:28:50.925526+0000","last_became_active":"2023-10-30T12:28:50.393642+0000","last_became_peered":"2023-10-30T12:28:50.393642+0000","last_unstale":"2023-10-30T12:28:50.925526+0000","last_undegraded":"2023-10-30T12:28:50.925526+0000","last_fullsized":"2023-10-30T12:28:50.925526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:21.643882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.983297+0000","last_change":"2023-10-30T12:28:50.976814+0000","last_active":"2023-10-30T12:28:50.983297+0000","last_peered":"2023-10-30T12:28:50.983297+0000","last_clean":"2023-10-30T12:28:50.983297+0000","last_became_active":"2023-10-30T12:28:50.976343+0000","last_became_peered":"2023-10-30T12:28:50.976343+0000","last_unstale":"2023-10-30T12:28:50.983297+0000","last_undegraded":"2023-10-30T12:28:50.983297+0000","last_fullsized":"2023-10-30T12:28:50.983297+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:19:44.549840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":164,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.371491+0000","last_change":"2023-10-30T12:28:41.203990+0000","last_active":"2023-10-30T12:28:50.371491+0000","last_peered":"2023-10-30T12:28:50.371491+0000","last_clean":"2023-10-30T12:28:50.371491+0000","last_became_active":"2023-10-30T12:28:41.201796+0000","last_became_peered":"2023-10-30T12:28:41.201796+0000","last_unstale":"2023-10-30T12:28:50.371491+0000","last_undegraded":"2023-10-30T12:28:50.371491+0000","last_fullsized":"2023-10-30T12:28:50.371491+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:39.218257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:39.218257+0000","last_clean_scrub_stamp":"2023-10-30T12:28:39.218257+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:21:00.605462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.956372+0000","last_change":"2023-10-30T12:28:50.372651+0000","last_active":"2023-10-30T12:28:50.956372+0000","last_peered":"2023-10-30T12:28:50.956372+0000","last_clean":"2023-10-30T12:28:50.956372+0000","last_became_active":"2023-10-30T12:28:50.367593+0000","last_became_peered":"2023-10-30T12:28:50.367593+0000","last_unstale":"2023-10-30T12:28:50.956372+0000","last_undegraded":"2023-10-30T12:28:50.956372+0000","last_fullsized":"2023-10-30T12:28:50.956372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:32:10.492061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.974612+0000","last_change":"2023-10-30T12:28:50.970536+0000","last_active":"2023-10-30T12:28:50.974612+0000","last_peered":"2023-10-30T12:28:50.974612+0000","last_clean":"2023-10-30T12:28:50.974612+0000","last_became_active":"2023-10-30T12:28:50.970326+0000","last_became_peered":"2023-10-30T12:28:50.970326+0000","last_unstale":"2023-10-30T12:28:50.974612+0000","last_undegraded":"2023-10-30T12:28:50.974612+0000","last_fullsized":"2023-10-30T12:28:50.974612+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:50.431623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.945861+0000","last_change":"2023-10-30T12:28:50.392461+0000","last_active":"2023-10-30T12:28:50.945861+0000","last_peered":"2023-10-30T12:28:50.945861+0000","last_clean":"2023-10-30T12:28:50.945861+0000","last_became_active":"2023-10-30T12:28:50.391824+0000","last_became_peered":"2023-10-30T12:28:50.391824+0000","last_unstale":"2023-10-30T12:28:50.945861+0000","last_undegraded":"2023-10-30T12:28:50.945861+0000","last_fullsized":"2023-10-30T12:28:50.945861+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:56:10.373046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.957824+0000","last_change":"2023-10-30T12:28:50.953046+0000","last_active":"2023-10-30T12:28:50.957824+0000","last_peered":"2023-10-30T12:28:50.957824+0000","last_clean":"2023-10-30T12:28:50.957824+0000","last_became_active":"2023-10-30T12:28:50.952625+0000","last_became_peered":"2023-10-30T12:28:50.952625+0000","last_unstale":"2023-10-30T12:28:50.957824+0000","last_undegraded":"2023-10-30T12:28:50.957824+0000","last_fullsized":"2023-10-30T12:28:50.957824+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:07:42.377331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.924652+0000","last_change":"2023-10-30T12:28:50.367547+0000","last_active":"2023-10-30T12:28:50.924652+0000","last_peered":"2023-10-30T12:28:50.924652+0000","last_clean":"2023-10-30T12:28:50.924652+0000","last_became_active":"2023-10-30T12:28:50.367230+0000","last_became_peered":"2023-10-30T12:28:50.367230+0000","last_unstale":"2023-10-30T12:28:50.924652+0000","last_undegraded":"2023-10-30T12:28:50.924652+0000","last_fullsized":"2023-10-30T12:28:50.924652+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:42.570786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931639+0000","last_change":"2023-10-30T12:28:50.928888+0000","last_active":"2023-10-30T12:28:51.931639+0000","last_peered":"2023-10-30T12:28:51.931639+0000","last_clean":"2023-10-30T12:28:51.931639+0000","last_became_active":"2023-10-30T12:28:50.928608+0000","last_became_peered":"2023-10-30T12:28:50.928608+0000","last_unstale":"2023-10-30T12:28:51.931639+0000","last_undegraded":"2023-10-30T12:28:51.931639+0000","last_fullsized":"2023-10-30T12:28:51.931639+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:40:51.042396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.956457+0000","last_change":"2023-10-30T12:28:50.391240+0000","last_active":"2023-10-30T12:28:50.956457+0000","last_peered":"2023-10-30T12:28:50.956457+0000","last_clean":"2023-10-30T12:28:50.956457+0000","last_became_active":"2023-10-30T12:28:50.390864+0000","last_became_peered":"2023-10-30T12:28:50.390864+0000","last_unstale":"2023-10-30T12:28:50.956457+0000","last_undegraded":"2023-10-30T12:28:50.956457+0000","last_fullsized":"2023-10-30T12:28:50.956457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:59:40.285830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.939756+0000","last_change":"2023-10-30T12:28:50.949990+0000","last_active":"2023-10-30T12:28:51.939756+0000","last_peered":"2023-10-30T12:28:51.939756+0000","last_clean":"2023-10-30T12:28:51.939756+0000","last_became_active":"2023-10-30T12:28:50.949122+0000","last_became_peered":"2023-10-30T12:28:50.949122+0000","last_unstale":"2023-10-30T12:28:51.939756+0000","last_undegraded":"2023-10-30T12:28:51.939756+0000","last_fullsized":"2023-10-30T12:28:51.939756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:30:56.099810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931803+0000","last_change":"2023-10-30T12:28:50.358700+0000","last_active":"2023-10-30T12:28:51.931803+0000","last_peered":"2023-10-30T12:28:51.931803+0000","last_clean":"2023-10-30T12:28:51.931803+0000","last_became_active":"2023-10-30T12:28:50.358310+0000","last_became_peered":"2023-10-30T12:28:50.358310+0000","last_unstale":"2023-10-30T12:28:51.931803+0000","last_undegraded":"2023-10-30T12:28:51.931803+0000","last_fullsized":"2023-10-30T12:28:51.931803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:23:30.910622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.978335+0000","last_change":"2023-10-30T12:28:50.967650+0000","last_active":"2023-10-30T12:28:50.978335+0000","last_peered":"2023-10-30T12:28:50.978335+0000","last_clean":"2023-10-30T12:28:50.978335+0000","last_became_active":"2023-10-30T12:28:50.965626+0000","last_became_peered":"2023-10-30T12:28:50.965626+0000","last_unstale":"2023-10-30T12:28:50.978335+0000","last_undegraded":"2023-10-30T12:28:50.978335+0000","last_fullsized":"2023-10-30T12:28:50.978335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:57:33.494828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.960484+0000","last_change":"2023-10-30T12:28:50.395645+0000","last_active":"2023-10-30T12:28:50.960484+0000","last_peered":"2023-10-30T12:28:50.960484+0000","last_clean":"2023-10-30T12:28:50.960484+0000","last_became_active":"2023-10-30T12:28:50.395090+0000","last_became_peered":"2023-10-30T12:28:50.395090+0000","last_unstale":"2023-10-30T12:28:50.960484+0000","last_undegraded":"2023-10-30T12:28:50.960484+0000","last_fullsized":"2023-10-30T12:28:50.960484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:45:44.678383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931844+0000","last_change":"2023-10-30T12:28:50.929586+0000","last_active":"2023-10-30T12:28:51.931844+0000","last_peered":"2023-10-30T12:28:51.931844+0000","last_clean":"2023-10-30T12:28:51.931844+0000","last_became_active":"2023-10-30T12:28:50.929337+0000","last_became_peered":"2023-10-30T12:28:50.929337+0000","last_unstale":"2023-10-30T12:28:51.931844+0000","last_undegraded":"2023-10-30T12:28:51.931844+0000","last_fullsized":"2023-10-30T12:28:51.931844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:11:13.067698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.932075+0000","last_change":"2023-10-30T12:28:50.380474+0000","last_active":"2023-10-30T12:28:51.932075+0000","last_peered":"2023-10-30T12:28:51.932075+0000","last_clean":"2023-10-30T12:28:51.932075+0000","last_became_active":"2023-10-30T12:28:50.380247+0000","last_became_peered":"2023-10-30T12:28:50.380247+0000","last_unstale":"2023-10-30T12:28:51.932075+0000","last_undegraded":"2023-10-30T12:28:51.932075+0000","last_fullsized":"2023-10-30T12:28:51.932075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:52:03.952617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.977080+0000","last_change":"2023-10-30T12:28:50.973361+0000","last_active":"2023-10-30T12:28:50.977080+0000","last_peered":"2023-10-30T12:28:50.977080+0000","last_clean":"2023-10-30T12:28:50.977080+0000","last_became_active":"2023-10-30T12:28:50.973086+0000","last_became_peered":"2023-10-30T12:28:50.973086+0000","last_unstale":"2023-10-30T12:28:50.977080+0000","last_undegraded":"2023-10-30T12:28:50.977080+0000","last_fullsized":"2023-10-30T12:28:50.977080+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:20:16.414696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.937662+0000","last_change":"2023-10-30T12:28:50.386277+0000","last_active":"2023-10-30T12:28:51.937662+0000","last_peered":"2023-10-30T12:28:51.937662+0000","last_clean":"2023-10-30T12:28:51.937662+0000","last_became_active":"2023-10-30T12:28:50.385924+0000","last_became_peered":"2023-10-30T12:28:50.385924+0000","last_unstale":"2023-10-30T12:28:51.937662+0000","last_undegraded":"2023-10-30T12:28:51.937662+0000","last_fullsized":"2023-10-30T12:28:51.937662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:35:03.837523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.933959+0000","last_change":"2023-10-30T12:28:50.929670+0000","last_active":"2023-10-30T12:28:50.933959+0000","last_peered":"2023-10-30T12:28:50.933959+0000","last_clean":"2023-10-30T12:28:50.933959+0000","last_became_active":"2023-10-30T12:28:50.929259+0000","last_became_peered":"2023-10-30T12:28:50.929259+0000","last_unstale":"2023-10-30T12:28:50.933959+0000","last_undegraded":"2023-10-30T12:28:50.933959+0000","last_fullsized":"2023-10-30T12:28:50.933959+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:50:04.591856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.945507+0000","last_change":"2023-10-30T12:28:50.386426+0000","last_active":"2023-10-30T12:28:50.945507+0000","last_peered":"2023-10-30T12:28:50.945507+0000","last_clean":"2023-10-30T12:28:50.945507+0000","last_became_active":"2023-10-30T12:28:50.385305+0000","last_became_peered":"2023-10-30T12:28:50.385305+0000","last_unstale":"2023-10-30T12:28:50.945507+0000","last_undegraded":"2023-10-30T12:28:50.945507+0000","last_fullsized":"2023-10-30T12:28:50.945507+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:13:05.376581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.938368+0000","last_change":"2023-10-30T12:28:50.941607+0000","last_active":"2023-10-30T12:28:51.938368+0000","last_peered":"2023-10-30T12:28:51.938368+0000","last_clean":"2023-10-30T12:28:51.938368+0000","last_became_active":"2023-10-30T12:28:50.940846+0000","last_became_peered":"2023-10-30T12:28:50.940846+0000","last_unstale":"2023-10-30T12:28:51.938368+0000","last_undegraded":"2023-10-30T12:28:51.938368+0000","last_fullsized":"2023-10-30T12:28:51.938368+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:29.837077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931710+0000","last_change":"2023-10-30T12:28:50.392333+0000","last_active":"2023-10-30T12:28:51.931710+0000","last_peered":"2023-10-30T12:28:51.931710+0000","last_clean":"2023-10-30T12:28:51.931710+0000","last_became_active":"2023-10-30T12:28:50.392068+0000","last_became_peered":"2023-10-30T12:28:50.392068+0000","last_unstale":"2023-10-30T12:28:51.931710+0000","last_undegraded":"2023-10-30T12:28:51.931710+0000","last_fullsized":"2023-10-30T12:28:51.931710+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:47:24.674925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.937725+0000","last_change":"2023-10-30T12:28:50.932167+0000","last_active":"2023-10-30T12:28:50.937725+0000","last_peered":"2023-10-30T12:28:50.937725+0000","last_clean":"2023-10-30T12:28:50.937725+0000","last_became_active":"2023-10-30T12:28:50.931898+0000","last_became_peered":"2023-10-30T12:28:50.931898+0000","last_unstale":"2023-10-30T12:28:50.937725+0000","last_undegraded":"2023-10-30T12:28:50.937725+0000","last_fullsized":"2023-10-30T12:28:50.937725+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:02:42.275987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931812+0000","last_change":"2023-10-30T12:28:50.365055+0000","last_active":"2023-10-30T12:28:51.931812+0000","last_peered":"2023-10-30T12:28:51.931812+0000","last_clean":"2023-10-30T12:28:51.931812+0000","last_became_active":"2023-10-30T12:28:50.364797+0000","last_became_peered":"2023-10-30T12:28:50.364797+0000","last_unstale":"2023-10-30T12:28:51.931812+0000","last_undegraded":"2023-10-30T12:28:51.931812+0000","last_fullsized":"2023-10-30T12:28:51.931812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:06:23.067895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.938503+0000","last_change":"2023-10-30T12:28:50.935908+0000","last_active":"2023-10-30T12:28:50.938503+0000","last_peered":"2023-10-30T12:28:50.938503+0000","last_clean":"2023-10-30T12:28:50.938503+0000","last_became_active":"2023-10-30T12:28:50.935657+0000","last_became_peered":"2023-10-30T12:28:50.935657+0000","last_unstale":"2023-10-30T12:28:50.938503+0000","last_undegraded":"2023-10-30T12:28:50.938503+0000","last_fullsized":"2023-10-30T12:28:50.938503+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:27:34.869337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.948828+0000","last_change":"2023-10-30T12:28:50.393465+0000","last_active":"2023-10-30T12:28:50.948828+0000","last_peered":"2023-10-30T12:28:50.948828+0000","last_clean":"2023-10-30T12:28:50.948828+0000","last_became_active":"2023-10-30T12:28:50.392807+0000","last_became_peered":"2023-10-30T12:28:50.392807+0000","last_unstale":"2023-10-30T12:28:50.948828+0000","last_undegraded":"2023-10-30T12:28:50.948828+0000","last_fullsized":"2023-10-30T12:28:50.948828+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:35:37.976956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.938767+0000","last_change":"2023-10-30T12:28:50.954834+0000","last_active":"2023-10-30T12:28:51.938767+0000","last_peered":"2023-10-30T12:28:51.938767+0000","last_clean":"2023-10-30T12:28:51.938767+0000","last_became_active":"2023-10-30T12:28:50.954291+0000","last_became_peered":"2023-10-30T12:28:50.954291+0000","last_unstale":"2023-10-30T12:28:51.938767+0000","last_undegraded":"2023-10-30T12:28:51.938767+0000","last_fullsized":"2023-10-30T12:28:51.938767+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:19:44.345918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.975322+0000","last_change":"2023-10-30T12:28:50.963620+0000","last_active":"2023-10-30T12:28:50.975322+0000","last_peered":"2023-10-30T12:28:50.975322+0000","last_clean":"2023-10-30T12:28:50.975322+0000","last_became_active":"2023-10-30T12:28:50.962817+0000","last_became_peered":"2023-10-30T12:28:50.962817+0000","last_unstale":"2023-10-30T12:28:50.975322+0000","last_undegraded":"2023-10-30T12:28:50.975322+0000","last_fullsized":"2023-10-30T12:28:50.975322+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:34:54.782063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.946929+0000","last_change":"2023-10-30T12:28:50.402179+0000","last_active":"2023-10-30T12:28:50.946929+0000","last_peered":"2023-10-30T12:28:50.946929+0000","last_clean":"2023-10-30T12:28:50.946929+0000","last_became_active":"2023-10-30T12:28:50.401677+0000","last_became_peered":"2023-10-30T12:28:50.401677+0000","last_unstale":"2023-10-30T12:28:50.946929+0000","last_undegraded":"2023-10-30T12:28:50.946929+0000","last_fullsized":"2023-10-30T12:28:50.946929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:39:44.432214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.978448+0000","last_change":"2023-10-30T12:28:50.968805+0000","last_active":"2023-10-30T12:28:50.978448+0000","last_peered":"2023-10-30T12:28:50.978448+0000","last_clean":"2023-10-30T12:28:50.978448+0000","last_became_active":"2023-10-30T12:28:50.967591+0000","last_became_peered":"2023-10-30T12:28:50.967591+0000","last_unstale":"2023-10-30T12:28:50.978448+0000","last_undegraded":"2023-10-30T12:28:50.978448+0000","last_fullsized":"2023-10-30T12:28:50.978448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:25:43.057459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.949875+0000","last_change":"2023-10-30T12:28:50.392617+0000","last_active":"2023-10-30T12:28:50.949875+0000","last_peered":"2023-10-30T12:28:50.949875+0000","last_clean":"2023-10-30T12:28:50.949875+0000","last_became_active":"2023-10-30T12:28:50.392021+0000","last_became_peered":"2023-10-30T12:28:50.392021+0000","last_unstale":"2023-10-30T12:28:50.949875+0000","last_undegraded":"2023-10-30T12:28:50.949875+0000","last_fullsized":"2023-10-30T12:28:50.949875+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:52:30.621125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.969439+0000","last_change":"2023-10-30T12:28:50.967491+0000","last_active":"2023-10-30T12:28:50.969439+0000","last_peered":"2023-10-30T12:28:50.969439+0000","last_clean":"2023-10-30T12:28:50.969439+0000","last_became_active":"2023-10-30T12:28:50.967186+0000","last_became_peered":"2023-10-30T12:28:50.967186+0000","last_unstale":"2023-10-30T12:28:50.969439+0000","last_undegraded":"2023-10-30T12:28:50.969439+0000","last_fullsized":"2023-10-30T12:28:50.969439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:21:15.521473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.930309+0000","last_change":"2023-10-30T12:28:50.363230+0000","last_active":"2023-10-30T12:28:50.930309+0000","last_peered":"2023-10-30T12:28:50.930309+0000","last_clean":"2023-10-30T12:28:50.930309+0000","last_became_active":"2023-10-30T12:28:50.362929+0000","last_became_peered":"2023-10-30T12:28:50.362929+0000","last_unstale":"2023-10-30T12:28:50.930309+0000","last_undegraded":"2023-10-30T12:28:50.930309+0000","last_fullsized":"2023-10-30T12:28:50.930309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:41:27.760578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.940360+0000","last_change":"2023-10-30T12:28:50.937780+0000","last_active":"2023-10-30T12:28:50.940360+0000","last_peered":"2023-10-30T12:28:50.940360+0000","last_clean":"2023-10-30T12:28:50.940360+0000","last_became_active":"2023-10-30T12:28:50.937458+0000","last_became_peered":"2023-10-30T12:28:50.937458+0000","last_unstale":"2023-10-30T12:28:50.940360+0000","last_undegraded":"2023-10-30T12:28:50.940360+0000","last_fullsized":"2023-10-30T12:28:50.940360+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:49.787839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.926291+0000","last_change":"2023-10-30T12:28:50.386782+0000","last_active":"2023-10-30T12:28:50.926291+0000","last_peered":"2023-10-30T12:28:50.926291+0000","last_clean":"2023-10-30T12:28:50.926291+0000","last_became_active":"2023-10-30T12:28:50.386522+0000","last_became_peered":"2023-10-30T12:28:50.386522+0000","last_unstale":"2023-10-30T12:28:50.926291+0000","last_undegraded":"2023-10-30T12:28:50.926291+0000","last_fullsized":"2023-10-30T12:28:50.926291+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:44:03.517641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":806912,"data_stored":662932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0149999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.228}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.181}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.359}]}]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":78864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.131}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.264}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3380000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542150,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":71945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0640000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1379999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.089}]}]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":78864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0740000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.21}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1779999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]}]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":212992,"data_stored":72564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]}]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":71945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.87}]}]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":72564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]}]},{"osd":0,"up_from":14,"seq":60129542150,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":802816,"data_stored":662313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80700000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:28:57.554 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:28:57.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b97a6b640 1 Processor -- start 2023-10-30T12:28:57.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b97a6b640 1 -- start start 2023-10-30T12:28:57.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b97a6b640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4b900c6f60 0x7f4b901566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:57.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b97a6b640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b900c0a50 con 0x7f4b900c6b80 2023-10-30T12:28:57.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b97a6b640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b9006f6e0 con 0x7f4b900c05d0 2023-10-30T12:28:57.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b97a6b640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b9006f9c0 con 0x7f4b900c6f60 2023-10-30T12:28:57.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b95807640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f4b900c6b80 0x7f4b900bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55708/0 (socket says 172.21.15.89:55708) 2023-10-30T12:28:57.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b95807640 1 -- 172.21.15.89:0/1994281384 learned_addr learned my addr 172.21.15.89:0/1994281384 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:28:57.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b96008640 1 --2- 172.21.15.89:0/1994281384 >> v2:172.21.15.192:3300/0 conn(0x7f4b900c6f60 0x7f4b901566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:57.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b94805640 1 -- 172.21.15.89:0/1994281384 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3604684733 0 0) 0x7f4b900c0a50 con 0x7f4b900c6b80 2023-10-30T12:28:57.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b94805640 1 -- 172.21.15.89:0/1994281384 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b7c003530 con 0x7f4b900c6b80 2023-10-30T12:28:57.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b94805640 1 -- 172.21.15.89:0/1994281384 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 137709098 0 0) 0x7f4b7c003530 con 0x7f4b900c6b80 2023-10-30T12:28:57.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b94805640 1 -- 172.21.15.89:0/1994281384 >> v2:172.21.15.192:3300/0 conn(0x7f4b900c6f60 msgr2=0x7f4b901566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:57.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b94805640 1 --2- 172.21.15.89:0/1994281384 >> v2:172.21.15.192:3300/0 conn(0x7f4b900c6f60 0x7f4b901566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b94805640 1 -- 172.21.15.89:0/1994281384 >> v1:172.21.15.192:6789/0 conn(0x7f4b900c05d0 legacy=0x7f4b9013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:57.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b94805640 1 -- 172.21.15.89:0/1994281384 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b9006fca0 con 0x7f4b900c6b80 2023-10-30T12:28:57.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b94805640 1 -- 172.21.15.89:0/1994281384 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f4b8c001680 con 0x7f4b900c6b80 2023-10-30T12:28:57.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b94805640 1 -- 172.21.15.89:0/1994281384 >> v1:172.21.15.89:6789/0 conn(0x7f4b900c6b80 legacy=0x7f4b900bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:57.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b94805640 1 --2- 172.21.15.89:0/1994281384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b7c003980 0x7f4b7c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:57.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b94805640 1 --2- 172.21.15.89:0/1994281384 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b7c0047d0 0x7f4b7c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:57.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b94805640 1 --2- 172.21.15.89:0/1994281384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b7c005190 0x7f4b7c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:57.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b94805640 1 -- 172.21.15.89:0/1994281384 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9006f9c0 con 0x7f4b7c005190 2023-10-30T12:28:57.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b94805640 1 -- 172.21.15.89:0/1994281384 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7c00c030 con 0x7f4b7c003980 2023-10-30T12:28:57.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b94805640 1 -- 172.21.15.89:0/1994281384 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7c00c1f0 con 0x7f4b7c0047d0 2023-10-30T12:28:57.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b96008640 1 --2- 172.21.15.89:0/1994281384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b7c005190 0x7f4b7c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:57.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b95807640 1 --2- 172.21.15.89:0/1994281384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b7c003980 0x7f4b7c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:57.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b95006640 1 --2- 172.21.15.89:0/1994281384 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b7c0047d0 0x7f4b7c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:57.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b96008640 1 -- 172.21.15.89:0/1994281384 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b7c0047d0 msgr2=0x7f4b7c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:57.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b96008640 1 --2- 172.21.15.89:0/1994281384 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b7c0047d0 0x7f4b7c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b96008640 1 -- 172.21.15.89:0/1994281384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b7c003980 msgr2=0x7f4b7c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:57.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b96008640 1 --2- 172.21.15.89:0/1994281384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b7c003980 0x7f4b7c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.728+0000 7f4b96008640 1 -- 172.21.15.89:0/1994281384 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b7c003530 con 0x7f4b7c005190 2023-10-30T12:28:57.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.732+0000 7f4b96008640 1 -- 172.21.15.89:0/1994281384 reap_dead start 2023-10-30T12:28:57.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.732+0000 7f4b96008640 1 --2- 172.21.15.89:0/1994281384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b7c005190 0x7f4b7c00b990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4b88009fc0 tx=0x7f4b8801cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f20c169ba713e5dd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:57.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.732+0000 7f4b94805640 1 -- 172.21.15.89:0/1994281384 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b880a5cb0 con 0x7f4b7c005190 2023-10-30T12:28:57.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.732+0000 7f4b94805640 1 -- 172.21.15.89:0/1994281384 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b8800f040 con 0x7f4b7c005190 2023-10-30T12:28:57.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.732+0000 7f4b94805640 1 -- 172.21.15.89:0/1994281384 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b88003be0 con 0x7f4b7c005190 2023-10-30T12:28:57.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.732+0000 7f4b97a6b640 1 -- 172.21.15.89:0/1994281384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b7c005190 msgr2=0x7f4b7c00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:57.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.732+0000 7f4b97a6b640 1 --2- 172.21.15.89:0/1994281384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b7c005190 0x7f4b7c00b990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4b88009fc0 tx=0x7f4b8801cca0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.732+0000 7f4b97a6b640 1 -- 172.21.15.89:0/1994281384 shutdown_connections 2023-10-30T12:28:57.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.732+0000 7f4b97a6b640 1 --2- 172.21.15.89:0/1994281384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b7c005190 0x7f4b7c00b990 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.732+0000 7f4b97a6b640 1 -- 172.21.15.89:0/1994281384 >> 172.21.15.89:0/1994281384 conn(0x7f4b900c7910 msgr2=0x7f4b900c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:57.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.732+0000 7f4b97a6b640 1 -- 172.21.15.89:0/1994281384 shutdown_connections 2023-10-30T12:28:57.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.732+0000 7f4b97a6b640 1 -- 172.21.15.89:0/1994281384 wait complete. 2023-10-30T12:28:57.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.736+0000 7f4b97a6b640 1 Processor -- start 2023-10-30T12:28:57.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.736+0000 7f4b97a6b640 1 -- start start 2023-10-30T12:28:57.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.736+0000 7f4b97a6b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b7c005190 0x7f4b901df040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:57.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.736+0000 7f4b97a6b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b900ca090 0x7f4b901ee9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:57.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.736+0000 7f4b97a6b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b90153d40 0x7f4b901eef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:57.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.736+0000 7f4b97a6b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b900ca910 con 0x7f4b7c005190 2023-10-30T12:28:57.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.736+0000 7f4b97a6b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b900c9560 con 0x7f4b90153d40 2023-10-30T12:28:57.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.736+0000 7f4b97a6b640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b900c8c10 con 0x7f4b900ca090 2023-10-30T12:28:57.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.736+0000 7f4b95807640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b7c005190 0x7f4b901df040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:57.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.736+0000 7f4b95807640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b7c005190 0x7f4b901df040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55364/0 (socket says 172.21.15.89:55364) 2023-10-30T12:28:57.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.736+0000 7f4b95807640 1 -- 172.21.15.89:0/319596163 learned_addr learned my addr 172.21.15.89:0/319596163 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:57.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.736+0000 7f4b95006640 1 --2- 172.21.15.89:0/319596163 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b900ca090 0x7f4b901ee9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:57.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.736+0000 7f4b96008640 1 --2- 172.21.15.89:0/319596163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b90153d40 0x7f4b901eef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:57.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.736+0000 7f4b95807640 1 -- 172.21.15.89:0/319596163 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b900ca090 msgr2=0x7f4b901ee9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:57.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.736+0000 7f4b95807640 1 --2- 172.21.15.89:0/319596163 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b900ca090 0x7f4b901ee9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.736+0000 7f4b95807640 1 -- 172.21.15.89:0/319596163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b90153d40 msgr2=0x7f4b901eef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:57.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.736+0000 7f4b95807640 1 --2- 172.21.15.89:0/319596163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b90153d40 0x7f4b901eef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.736+0000 7f4b95807640 1 -- 172.21.15.89:0/319596163 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b901ef450 con 0x7f4b7c005190 2023-10-30T12:28:57.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.736+0000 7f4b95807640 1 --2- 172.21.15.89:0/319596163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b7c005190 0x7f4b901df040 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c00e780 tx=0x7f4b8c00ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:57.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.736+0000 7f4b867fc640 1 -- 172.21.15.89:0/319596163 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b8c00ee30 con 0x7f4b7c005190 2023-10-30T12:28:57.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.736+0000 7f4b867fc640 1 -- 172.21.15.89:0/319596163 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b8c00c8d0 con 0x7f4b7c005190 2023-10-30T12:28:57.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.736+0000 7f4b867fc640 1 -- 172.21.15.89:0/319596163 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b8c00cc60 con 0x7f4b7c005190 2023-10-30T12:28:57.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.736+0000 7f4b97a6b640 1 -- 172.21.15.89:0/319596163 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b901e0860 con 0x7f4b7c005190 2023-10-30T12:28:57.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.736+0000 7f4b97a6b640 1 -- 172.21.15.89:0/319596163 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b900c9100 con 0x7f4b7c005190 2023-10-30T12:28:57.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.740+0000 7f4b97a6b640 1 -- 172.21.15.89:0/319596163 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b9004e170 con 0x7f4b7c005190 2023-10-30T12:28:57.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.740+0000 7f4b867fc640 1 -- 172.21.15.89:0/319596163 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4b8c01d460 con 0x7f4b7c005190 2023-10-30T12:28:57.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.740+0000 7f4b867fc640 1 --2- 172.21.15.89:0/319596163 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4b740664a0 0x7f4b74068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:57.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.740+0000 7f4b867fc640 1 -- 172.21.15.89:0/319596163 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4b8c083dc0 con 0x7f4b7c005190 2023-10-30T12:28:57.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.740+0000 7f4b95006640 1 --2- 172.21.15.89:0/319596163 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4b740664a0 0x7f4b74068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:57.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.740+0000 7f4b95006640 1 --2- 172.21.15.89:0/319596163 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4b740664a0 0x7f4b74068960 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4b900ca470 tx=0x7f4b80009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:57.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.744+0000 7f4b867fc640 1 -- 172.21.15.89:0/319596163 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4b8c051240 con 0x7f4b7c005190 2023-10-30T12:28:57.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.892+0000 7f4b97a6b640 1 -- 172.21.15.89:0/319596163 --> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4b900c0be0 con 0x7f4b740664a0 2023-10-30T12:28:57.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.912+0000 7f4b867fc640 1 -- 172.21.15.89:0/319596163 <== mgr.4105 v2:172.21.15.192:6832/4076276638 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264444 (secure 0 0 0) 0x7f4b80014400 con 0x7f4b740664a0 2023-10-30T12:28:57.918 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:28:57.920 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-10-30T12:28:57.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.912+0000 7f4b97a6b640 1 -- 172.21.15.89:0/319596163 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4b740664a0 msgr2=0x7f4b74068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:57.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.912+0000 7f4b97a6b640 1 --2- 172.21.15.89:0/319596163 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4b740664a0 0x7f4b74068960 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4b900ca470 tx=0x7f4b80009040 comp rx=0 tx=0).stop 2023-10-30T12:28:57.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.912+0000 7f4b97a6b640 1 -- 172.21.15.89:0/319596163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b7c005190 msgr2=0x7f4b901df040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:57.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.912+0000 7f4b97a6b640 1 --2- 172.21.15.89:0/319596163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b7c005190 0x7f4b901df040 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c00e780 tx=0x7f4b8c00ec40 comp rx=0 tx=0).stop 2023-10-30T12:28:57.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.912+0000 7f4b97a6b640 1 -- 172.21.15.89:0/319596163 shutdown_connections 2023-10-30T12:28:57.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.912+0000 7f4b97a6b640 1 --2- 172.21.15.89:0/319596163 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4b740664a0 0x7f4b74068960 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.912+0000 7f4b97a6b640 1 --2- 172.21.15.89:0/319596163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b90153d40 0x7f4b901eef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.912+0000 7f4b97a6b640 1 --2- 172.21.15.89:0/319596163 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b900ca090 0x7f4b901ee9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.912+0000 7f4b97a6b640 1 --2- 172.21.15.89:0/319596163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b7c005190 0x7f4b901df040 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.912+0000 7f4b97a6b640 1 -- 172.21.15.89:0/319596163 >> 172.21.15.89:0/319596163 conn(0x7f4b900c7910 msgr2=0x7f4b90069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:57.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.912+0000 7f4b97a6b640 1 -- 172.21.15.89:0/319596163 shutdown_connections 2023-10-30T12:28:57.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:57.912+0000 7f4b97a6b640 1 -- 172.21.15.89:0/319596163 wait complete. 2023-10-30T12:28:57.938 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-10-30T12:28:56.920690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":111,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219048,"kb_used_data":2836,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755672,"statfs":{"total":773094113280,"available":772869808128,"internally_reserved":0,"allocated":2904064,"data_stored":1771991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12741,"internal_metadata":221236795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":49,"apply_latency_ms":49,"commit_latency_ns":49000000,"apply_latency_ns":49000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.577185"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.984200+0000","last_change":"2023-10-30T12:28:50.979797+0000","last_active":"2023-10-30T12:28:50.984200+0000","last_peered":"2023-10-30T12:28:50.984200+0000","last_clean":"2023-10-30T12:28:50.984200+0000","last_became_active":"2023-10-30T12:28:50.979351+0000","last_became_peered":"2023-10-30T12:28:50.979351+0000","last_unstale":"2023-10-30T12:28:50.984200+0000","last_undegraded":"2023-10-30T12:28:50.984200+0000","last_fullsized":"2023-10-30T12:28:50.984200+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:33:02.139822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.932043+0000","last_change":"2023-10-30T12:28:50.929403+0000","last_active":"2023-10-30T12:28:51.932043+0000","last_peered":"2023-10-30T12:28:51.932043+0000","last_clean":"2023-10-30T12:28:51.932043+0000","last_became_active":"2023-10-30T12:28:50.929133+0000","last_became_peered":"2023-10-30T12:28:50.929133+0000","last_unstale":"2023-10-30T12:28:51.932043+0000","last_undegraded":"2023-10-30T12:28:51.932043+0000","last_fullsized":"2023-10-30T12:28:51.932043+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:12:57.327440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.936209+0000","last_change":"2023-10-30T12:28:50.934201+0000","last_active":"2023-10-30T12:28:50.936209+0000","last_peered":"2023-10-30T12:28:50.936209+0000","last_clean":"2023-10-30T12:28:50.936209+0000","last_became_active":"2023-10-30T12:28:50.933896+0000","last_became_peered":"2023-10-30T12:28:50.933896+0000","last_unstale":"2023-10-30T12:28:50.936209+0000","last_undegraded":"2023-10-30T12:28:50.936209+0000","last_fullsized":"2023-10-30T12:28:50.936209+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:50:22.654643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.978480+0000","last_change":"2023-10-30T12:28:50.969193+0000","last_active":"2023-10-30T12:28:50.978480+0000","last_peered":"2023-10-30T12:28:50.978480+0000","last_clean":"2023-10-30T12:28:50.978480+0000","last_became_active":"2023-10-30T12:28:50.967774+0000","last_became_peered":"2023-10-30T12:28:50.967774+0000","last_unstale":"2023-10-30T12:28:50.978480+0000","last_undegraded":"2023-10-30T12:28:50.978480+0000","last_fullsized":"2023-10-30T12:28:50.978480+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:32:01.665321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.983029+0000","last_change":"2023-10-30T12:28:50.973751+0000","last_active":"2023-10-30T12:28:50.983029+0000","last_peered":"2023-10-30T12:28:50.983029+0000","last_clean":"2023-10-30T12:28:50.983029+0000","last_became_active":"2023-10-30T12:28:50.973132+0000","last_became_peered":"2023-10-30T12:28:50.973132+0000","last_unstale":"2023-10-30T12:28:50.983029+0000","last_undegraded":"2023-10-30T12:28:50.983029+0000","last_fullsized":"2023-10-30T12:28:50.983029+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:50:49.899151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.978576+0000","last_change":"2023-10-30T12:28:50.968823+0000","last_active":"2023-10-30T12:28:50.978576+0000","last_peered":"2023-10-30T12:28:50.978576+0000","last_clean":"2023-10-30T12:28:50.978576+0000","last_became_active":"2023-10-30T12:28:50.967910+0000","last_became_peered":"2023-10-30T12:28:50.967910+0000","last_unstale":"2023-10-30T12:28:50.978576+0000","last_undegraded":"2023-10-30T12:28:50.978576+0000","last_fullsized":"2023-10-30T12:28:50.978576+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:59:35.262638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.983085+0000","last_change":"2023-10-30T12:28:50.974164+0000","last_active":"2023-10-30T12:28:50.983085+0000","last_peered":"2023-10-30T12:28:50.983085+0000","last_clean":"2023-10-30T12:28:50.983085+0000","last_became_active":"2023-10-30T12:28:50.973288+0000","last_became_peered":"2023-10-30T12:28:50.973288+0000","last_unstale":"2023-10-30T12:28:50.983085+0000","last_undegraded":"2023-10-30T12:28:50.983085+0000","last_fullsized":"2023-10-30T12:28:50.983085+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:38:25.761235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.958001+0000","last_change":"2023-10-30T12:28:50.955287+0000","last_active":"2023-10-30T12:28:50.958001+0000","last_peered":"2023-10-30T12:28:50.958001+0000","last_clean":"2023-10-30T12:28:50.958001+0000","last_became_active":"2023-10-30T12:28:50.954988+0000","last_became_peered":"2023-10-30T12:28:50.954988+0000","last_unstale":"2023-10-30T12:28:50.958001+0000","last_undegraded":"2023-10-30T12:28:50.958001+0000","last_fullsized":"2023-10-30T12:28:50.958001+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:46:16.004666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.983305+0000","last_change":"2023-10-30T12:28:50.977144+0000","last_active":"2023-10-30T12:28:50.983305+0000","last_peered":"2023-10-30T12:28:50.983305+0000","last_clean":"2023-10-30T12:28:50.983305+0000","last_became_active":"2023-10-30T12:28:50.976564+0000","last_became_peered":"2023-10-30T12:28:50.976564+0000","last_unstale":"2023-10-30T12:28:50.983305+0000","last_undegraded":"2023-10-30T12:28:50.983305+0000","last_fullsized":"2023-10-30T12:28:50.983305+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:50.229780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.938542+0000","last_change":"2023-10-30T12:28:50.952564+0000","last_active":"2023-10-30T12:28:51.938542+0000","last_peered":"2023-10-30T12:28:51.938542+0000","last_clean":"2023-10-30T12:28:51.938542+0000","last_became_active":"2023-10-30T12:28:50.951809+0000","last_became_peered":"2023-10-30T12:28:50.951809+0000","last_unstale":"2023-10-30T12:28:51.938542+0000","last_undegraded":"2023-10-30T12:28:51.938542+0000","last_fullsized":"2023-10-30T12:28:51.938542+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:34:29.771577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.937215+0000","last_change":"2023-10-30T12:28:50.970779+0000","last_active":"2023-10-30T12:28:51.937215+0000","last_peered":"2023-10-30T12:28:51.937215+0000","last_clean":"2023-10-30T12:28:51.937215+0000","last_became_active":"2023-10-30T12:28:50.969877+0000","last_became_peered":"2023-10-30T12:28:50.969877+0000","last_unstale":"2023-10-30T12:28:51.937215+0000","last_undegraded":"2023-10-30T12:28:51.937215+0000","last_fullsized":"2023-10-30T12:28:51.937215+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:10.539915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.940164+0000","last_change":"2023-10-30T12:28:50.944452+0000","last_active":"2023-10-30T12:28:51.940164+0000","last_peered":"2023-10-30T12:28:51.940164+0000","last_clean":"2023-10-30T12:28:51.940164+0000","last_became_active":"2023-10-30T12:28:50.943787+0000","last_became_peered":"2023-10-30T12:28:50.943787+0000","last_unstale":"2023-10-30T12:28:51.940164+0000","last_undegraded":"2023-10-30T12:28:51.940164+0000","last_fullsized":"2023-10-30T12:28:51.940164+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:30:34.585298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.943369+0000","last_change":"2023-10-30T12:28:50.938181+0000","last_active":"2023-10-30T12:28:50.943369+0000","last_peered":"2023-10-30T12:28:50.943369+0000","last_clean":"2023-10-30T12:28:50.943369+0000","last_became_active":"2023-10-30T12:28:50.937906+0000","last_became_peered":"2023-10-30T12:28:50.937906+0000","last_unstale":"2023-10-30T12:28:50.943369+0000","last_undegraded":"2023-10-30T12:28:50.943369+0000","last_fullsized":"2023-10-30T12:28:50.943369+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:35:50.470679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.936206+0000","last_change":"2023-10-30T12:28:50.934085+0000","last_active":"2023-10-30T12:28:50.936206+0000","last_peered":"2023-10-30T12:28:50.936206+0000","last_clean":"2023-10-30T12:28:50.936206+0000","last_became_active":"2023-10-30T12:28:50.932753+0000","last_became_peered":"2023-10-30T12:28:50.932753+0000","last_unstale":"2023-10-30T12:28:50.936206+0000","last_undegraded":"2023-10-30T12:28:50.936206+0000","last_fullsized":"2023-10-30T12:28:50.936206+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:24.146920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931710+0000","last_change":"2023-10-30T12:28:50.943343+0000","last_active":"2023-10-30T12:28:51.931710+0000","last_peered":"2023-10-30T12:28:51.931710+0000","last_clean":"2023-10-30T12:28:51.931710+0000","last_became_active":"2023-10-30T12:28:50.943011+0000","last_became_peered":"2023-10-30T12:28:50.943011+0000","last_unstale":"2023-10-30T12:28:51.931710+0000","last_undegraded":"2023-10-30T12:28:51.931710+0000","last_fullsized":"2023-10-30T12:28:51.931710+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:59:35.881115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.974555+0000","last_change":"2023-10-30T12:28:50.953573+0000","last_active":"2023-10-30T12:28:50.974555+0000","last_peered":"2023-10-30T12:28:50.974555+0000","last_clean":"2023-10-30T12:28:50.974555+0000","last_became_active":"2023-10-30T12:28:50.952744+0000","last_became_peered":"2023-10-30T12:28:50.952744+0000","last_unstale":"2023-10-30T12:28:50.974555+0000","last_undegraded":"2023-10-30T12:28:50.974555+0000","last_fullsized":"2023-10-30T12:28:50.974555+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:13:11.797130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.978742+0000","last_change":"2023-10-30T12:28:50.975156+0000","last_active":"2023-10-30T12:28:50.978742+0000","last_peered":"2023-10-30T12:28:50.978742+0000","last_clean":"2023-10-30T12:28:50.978742+0000","last_became_active":"2023-10-30T12:28:50.974576+0000","last_became_peered":"2023-10-30T12:28:50.974576+0000","last_unstale":"2023-10-30T12:28:50.978742+0000","last_undegraded":"2023-10-30T12:28:50.978742+0000","last_fullsized":"2023-10-30T12:28:50.978742+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:26:53.504301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.938728+0000","last_change":"2023-10-30T12:28:50.937685+0000","last_active":"2023-10-30T12:28:50.938728+0000","last_peered":"2023-10-30T12:28:50.938728+0000","last_clean":"2023-10-30T12:28:50.938728+0000","last_became_active":"2023-10-30T12:28:50.937384+0000","last_became_peered":"2023-10-30T12:28:50.937384+0000","last_unstale":"2023-10-30T12:28:50.938728+0000","last_undegraded":"2023-10-30T12:28:50.938728+0000","last_fullsized":"2023-10-30T12:28:50.938728+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:33:59.082334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.983151+0000","last_change":"2023-10-30T12:28:50.974918+0000","last_active":"2023-10-30T12:28:50.983151+0000","last_peered":"2023-10-30T12:28:50.983151+0000","last_clean":"2023-10-30T12:28:50.983151+0000","last_became_active":"2023-10-30T12:28:50.973369+0000","last_became_peered":"2023-10-30T12:28:50.973369+0000","last_unstale":"2023-10-30T12:28:50.983151+0000","last_undegraded":"2023-10-30T12:28:50.983151+0000","last_fullsized":"2023-10-30T12:28:50.983151+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:54:34.723806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.937226+0000","last_change":"2023-10-30T12:28:50.955835+0000","last_active":"2023-10-30T12:28:51.937226+0000","last_peered":"2023-10-30T12:28:51.937226+0000","last_clean":"2023-10-30T12:28:51.937226+0000","last_became_active":"2023-10-30T12:28:50.954831+0000","last_became_peered":"2023-10-30T12:28:50.954831+0000","last_unstale":"2023-10-30T12:28:51.937226+0000","last_undegraded":"2023-10-30T12:28:51.937226+0000","last_fullsized":"2023-10-30T12:28:51.937226+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:56:25.554582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.963406+0000","last_change":"2023-10-30T12:28:50.962417+0000","last_active":"2023-10-30T12:28:50.963406+0000","last_peered":"2023-10-30T12:28:50.963406+0000","last_clean":"2023-10-30T12:28:50.963406+0000","last_became_active":"2023-10-30T12:28:50.962065+0000","last_became_peered":"2023-10-30T12:28:50.962065+0000","last_unstale":"2023-10-30T12:28:50.963406+0000","last_undegraded":"2023-10-30T12:28:50.963406+0000","last_fullsized":"2023-10-30T12:28:50.963406+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:46:22.958672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.978608+0000","last_change":"2023-10-30T12:28:50.968992+0000","last_active":"2023-10-30T12:28:50.978608+0000","last_peered":"2023-10-30T12:28:50.978608+0000","last_clean":"2023-10-30T12:28:50.978608+0000","last_became_active":"2023-10-30T12:28:50.967651+0000","last_became_peered":"2023-10-30T12:28:50.967651+0000","last_unstale":"2023-10-30T12:28:50.978608+0000","last_undegraded":"2023-10-30T12:28:50.978608+0000","last_fullsized":"2023-10-30T12:28:50.978608+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:42.061451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.963293+0000","last_change":"2023-10-30T12:28:50.962609+0000","last_active":"2023-10-30T12:28:50.963293+0000","last_peered":"2023-10-30T12:28:50.963293+0000","last_clean":"2023-10-30T12:28:50.963293+0000","last_became_active":"2023-10-30T12:28:50.962290+0000","last_became_peered":"2023-10-30T12:28:50.962290+0000","last_unstale":"2023-10-30T12:28:50.963293+0000","last_undegraded":"2023-10-30T12:28:50.963293+0000","last_fullsized":"2023-10-30T12:28:50.963293+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:57:13.022671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.978406+0000","last_change":"2023-10-30T12:28:50.966653+0000","last_active":"2023-10-30T12:28:50.978406+0000","last_peered":"2023-10-30T12:28:50.978406+0000","last_clean":"2023-10-30T12:28:50.978406+0000","last_became_active":"2023-10-30T12:28:50.965713+0000","last_became_peered":"2023-10-30T12:28:50.965713+0000","last_unstale":"2023-10-30T12:28:50.978406+0000","last_undegraded":"2023-10-30T12:28:50.978406+0000","last_fullsized":"2023-10-30T12:28:50.978406+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:42:07.786343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.952821+0000","last_change":"2023-10-30T12:28:50.951300+0000","last_active":"2023-10-30T12:28:50.952821+0000","last_peered":"2023-10-30T12:28:50.952821+0000","last_clean":"2023-10-30T12:28:50.952821+0000","last_became_active":"2023-10-30T12:28:50.951026+0000","last_became_peered":"2023-10-30T12:28:50.951026+0000","last_unstale":"2023-10-30T12:28:50.952821+0000","last_undegraded":"2023-10-30T12:28:50.952821+0000","last_fullsized":"2023-10-30T12:28:50.952821+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:29:11.435330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.938879+0000","last_change":"2023-10-30T12:28:50.963507+0000","last_active":"2023-10-30T12:28:51.938879+0000","last_peered":"2023-10-30T12:28:51.938879+0000","last_clean":"2023-10-30T12:28:51.938879+0000","last_became_active":"2023-10-30T12:28:50.962484+0000","last_became_peered":"2023-10-30T12:28:50.962484+0000","last_unstale":"2023-10-30T12:28:51.938879+0000","last_undegraded":"2023-10-30T12:28:51.938879+0000","last_fullsized":"2023-10-30T12:28:51.938879+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:11:24.160220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931933+0000","last_change":"2023-10-30T12:28:50.933362+0000","last_active":"2023-10-30T12:28:51.931933+0000","last_peered":"2023-10-30T12:28:51.931933+0000","last_clean":"2023-10-30T12:28:51.931933+0000","last_became_active":"2023-10-30T12:28:50.933093+0000","last_became_peered":"2023-10-30T12:28:50.933093+0000","last_unstale":"2023-10-30T12:28:51.931933+0000","last_undegraded":"2023-10-30T12:28:51.931933+0000","last_fullsized":"2023-10-30T12:28:51.931933+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:54:00.852482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.955331+0000","last_change":"2023-10-30T12:28:50.952857+0000","last_active":"2023-10-30T12:28:50.955331+0000","last_peered":"2023-10-30T12:28:50.955331+0000","last_clean":"2023-10-30T12:28:50.955331+0000","last_became_active":"2023-10-30T12:28:50.952531+0000","last_became_peered":"2023-10-30T12:28:50.952531+0000","last_unstale":"2023-10-30T12:28:50.955331+0000","last_undegraded":"2023-10-30T12:28:50.955331+0000","last_fullsized":"2023-10-30T12:28:50.955331+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:44:51.152114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.935136+0000","last_change":"2023-10-30T12:28:50.933408+0000","last_active":"2023-10-30T12:28:50.935136+0000","last_peered":"2023-10-30T12:28:50.935136+0000","last_clean":"2023-10-30T12:28:50.935136+0000","last_became_active":"2023-10-30T12:28:50.933124+0000","last_became_peered":"2023-10-30T12:28:50.933124+0000","last_unstale":"2023-10-30T12:28:50.935136+0000","last_undegraded":"2023-10-30T12:28:50.935136+0000","last_fullsized":"2023-10-30T12:28:50.935136+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:39:57.679819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.978541+0000","last_change":"2023-10-30T12:28:50.966388+0000","last_active":"2023-10-30T12:28:50.978541+0000","last_peered":"2023-10-30T12:28:50.978541+0000","last_clean":"2023-10-30T12:28:50.978541+0000","last_became_active":"2023-10-30T12:28:50.965639+0000","last_became_peered":"2023-10-30T12:28:50.965639+0000","last_unstale":"2023-10-30T12:28:50.978541+0000","last_undegraded":"2023-10-30T12:28:50.978541+0000","last_fullsized":"2023-10-30T12:28:50.978541+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:46:28.766578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.957736+0000","last_change":"2023-10-30T12:28:50.955391+0000","last_active":"2023-10-30T12:28:50.957736+0000","last_peered":"2023-10-30T12:28:50.957736+0000","last_clean":"2023-10-30T12:28:50.957736+0000","last_became_active":"2023-10-30T12:28:50.955084+0000","last_became_peered":"2023-10-30T12:28:50.955084+0000","last_unstale":"2023-10-30T12:28:50.957736+0000","last_undegraded":"2023-10-30T12:28:50.957736+0000","last_fullsized":"2023-10-30T12:28:50.957736+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:08:17.139917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.936598+0000","last_change":"2023-10-30T12:28:50.945609+0000","last_active":"2023-10-30T12:28:51.936598+0000","last_peered":"2023-10-30T12:28:51.936598+0000","last_clean":"2023-10-30T12:28:51.936598+0000","last_became_active":"2023-10-30T12:28:50.944775+0000","last_became_peered":"2023-10-30T12:28:50.944775+0000","last_unstale":"2023-10-30T12:28:51.936598+0000","last_undegraded":"2023-10-30T12:28:51.936598+0000","last_fullsized":"2023-10-30T12:28:51.936598+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:08:55.718193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.939146+0000","last_change":"2023-10-30T12:28:50.941378+0000","last_active":"2023-10-30T12:28:51.939146+0000","last_peered":"2023-10-30T12:28:51.939146+0000","last_clean":"2023-10-30T12:28:51.939146+0000","last_became_active":"2023-10-30T12:28:50.940567+0000","last_became_peered":"2023-10-30T12:28:50.940567+0000","last_unstale":"2023-10-30T12:28:51.939146+0000","last_undegraded":"2023-10-30T12:28:51.939146+0000","last_fullsized":"2023-10-30T12:28:51.939146+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:31:23.772343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.930221+0000","last_change":"2023-10-30T12:28:50.395670+0000","last_active":"2023-10-30T12:28:50.930221+0000","last_peered":"2023-10-30T12:28:50.930221+0000","last_clean":"2023-10-30T12:28:50.930221+0000","last_became_active":"2023-10-30T12:28:50.395382+0000","last_became_peered":"2023-10-30T12:28:50.395382+0000","last_unstale":"2023-10-30T12:28:50.930221+0000","last_undegraded":"2023-10-30T12:28:50.930221+0000","last_fullsized":"2023-10-30T12:28:50.930221+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:14.858121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.957837+0000","last_change":"2023-10-30T12:28:50.953202+0000","last_active":"2023-10-30T12:28:50.957837+0000","last_peered":"2023-10-30T12:28:50.957837+0000","last_clean":"2023-10-30T12:28:50.957837+0000","last_became_active":"2023-10-30T12:28:50.952613+0000","last_became_peered":"2023-10-30T12:28:50.952613+0000","last_unstale":"2023-10-30T12:28:50.957837+0000","last_undegraded":"2023-10-30T12:28:50.957837+0000","last_fullsized":"2023-10-30T12:28:50.957837+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:07.934471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.938991+0000","last_change":"2023-10-30T12:28:50.384598+0000","last_active":"2023-10-30T12:28:51.938991+0000","last_peered":"2023-10-30T12:28:51.938991+0000","last_clean":"2023-10-30T12:28:51.938991+0000","last_became_active":"2023-10-30T12:28:50.383898+0000","last_became_peered":"2023-10-30T12:28:50.383898+0000","last_unstale":"2023-10-30T12:28:51.938991+0000","last_undegraded":"2023-10-30T12:28:51.938991+0000","last_fullsized":"2023-10-30T12:28:51.938991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:09:50.290004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.929895+0000","last_change":"2023-10-30T12:28:50.386193+0000","last_active":"2023-10-30T12:28:50.929895+0000","last_peered":"2023-10-30T12:28:50.929895+0000","last_clean":"2023-10-30T12:28:50.929895+0000","last_became_active":"2023-10-30T12:28:50.385963+0000","last_became_peered":"2023-10-30T12:28:50.385963+0000","last_unstale":"2023-10-30T12:28:50.929895+0000","last_undegraded":"2023-10-30T12:28:50.929895+0000","last_fullsized":"2023-10-30T12:28:50.929895+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:00:55.603212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.938352+0000","last_change":"2023-10-30T12:28:50.936038+0000","last_active":"2023-10-30T12:28:50.938352+0000","last_peered":"2023-10-30T12:28:50.938352+0000","last_clean":"2023-10-30T12:28:50.938352+0000","last_became_active":"2023-10-30T12:28:50.935764+0000","last_became_peered":"2023-10-30T12:28:50.935764+0000","last_unstale":"2023-10-30T12:28:50.938352+0000","last_undegraded":"2023-10-30T12:28:50.938352+0000","last_fullsized":"2023-10-30T12:28:50.938352+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:24:28.355705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.952323+0000","last_change":"2023-10-30T12:28:50.392213+0000","last_active":"2023-10-30T12:28:50.952323+0000","last_peered":"2023-10-30T12:28:50.952323+0000","last_clean":"2023-10-30T12:28:50.952323+0000","last_became_active":"2023-10-30T12:28:50.391494+0000","last_became_peered":"2023-10-30T12:28:50.391494+0000","last_unstale":"2023-10-30T12:28:50.952323+0000","last_undegraded":"2023-10-30T12:28:50.952323+0000","last_fullsized":"2023-10-30T12:28:50.952323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:38:37.301840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931619+0000","last_change":"2023-10-30T12:28:50.933319+0000","last_active":"2023-10-30T12:28:51.931619+0000","last_peered":"2023-10-30T12:28:51.931619+0000","last_clean":"2023-10-30T12:28:51.931619+0000","last_became_active":"2023-10-30T12:28:50.933062+0000","last_became_peered":"2023-10-30T12:28:50.933062+0000","last_unstale":"2023-10-30T12:28:51.931619+0000","last_undegraded":"2023-10-30T12:28:51.931619+0000","last_fullsized":"2023-10-30T12:28:51.931619+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:26:48.475038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.937437+0000","last_change":"2023-10-30T12:28:50.394368+0000","last_active":"2023-10-30T12:28:51.937437+0000","last_peered":"2023-10-30T12:28:51.937437+0000","last_clean":"2023-10-30T12:28:51.937437+0000","last_became_active":"2023-10-30T12:28:50.393686+0000","last_became_peered":"2023-10-30T12:28:50.393686+0000","last_unstale":"2023-10-30T12:28:51.937437+0000","last_undegraded":"2023-10-30T12:28:51.937437+0000","last_fullsized":"2023-10-30T12:28:51.937437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:43:17.726462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.975448+0000","last_change":"2023-10-30T12:28:50.963703+0000","last_active":"2023-10-30T12:28:50.975448+0000","last_peered":"2023-10-30T12:28:50.975448+0000","last_clean":"2023-10-30T12:28:50.975448+0000","last_became_active":"2023-10-30T12:28:50.963105+0000","last_became_peered":"2023-10-30T12:28:50.963105+0000","last_unstale":"2023-10-30T12:28:50.975448+0000","last_undegraded":"2023-10-30T12:28:50.975448+0000","last_fullsized":"2023-10-30T12:28:50.975448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:05:31.103581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.957188+0000","last_change":"2023-10-30T12:28:50.397679+0000","last_active":"2023-10-30T12:28:50.957188+0000","last_peered":"2023-10-30T12:28:50.957188+0000","last_clean":"2023-10-30T12:28:50.957188+0000","last_became_active":"2023-10-30T12:28:50.397041+0000","last_became_peered":"2023-10-30T12:28:50.397041+0000","last_unstale":"2023-10-30T12:28:50.957188+0000","last_undegraded":"2023-10-30T12:28:50.957188+0000","last_fullsized":"2023-10-30T12:28:50.957188+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:52:22.564616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.969024+0000","last_change":"2023-10-30T12:28:50.966838+0000","last_active":"2023-10-30T12:28:50.969024+0000","last_peered":"2023-10-30T12:28:50.969024+0000","last_clean":"2023-10-30T12:28:50.969024+0000","last_became_active":"2023-10-30T12:28:50.966569+0000","last_became_peered":"2023-10-30T12:28:50.966569+0000","last_unstale":"2023-10-30T12:28:50.969024+0000","last_undegraded":"2023-10-30T12:28:50.969024+0000","last_fullsized":"2023-10-30T12:28:50.969024+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:37:20.093654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931949+0000","last_change":"2023-10-30T12:28:50.391527+0000","last_active":"2023-10-30T12:28:51.931949+0000","last_peered":"2023-10-30T12:28:51.931949+0000","last_clean":"2023-10-30T12:28:51.931949+0000","last_became_active":"2023-10-30T12:28:50.391260+0000","last_became_peered":"2023-10-30T12:28:50.391260+0000","last_unstale":"2023-10-30T12:28:51.931949+0000","last_undegraded":"2023-10-30T12:28:51.931949+0000","last_fullsized":"2023-10-30T12:28:51.931949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:29:34.772490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.940211+0000","last_change":"2023-10-30T12:28:50.975441+0000","last_active":"2023-10-30T12:28:51.940211+0000","last_peered":"2023-10-30T12:28:51.940211+0000","last_clean":"2023-10-30T12:28:51.940211+0000","last_became_active":"2023-10-30T12:28:50.974843+0000","last_became_peered":"2023-10-30T12:28:50.974843+0000","last_unstale":"2023-10-30T12:28:51.940211+0000","last_undegraded":"2023-10-30T12:28:51.940211+0000","last_fullsized":"2023-10-30T12:28:51.940211+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:06:12.060752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.937470+0000","last_change":"2023-10-30T12:28:50.374838+0000","last_active":"2023-10-30T12:28:51.937470+0000","last_peered":"2023-10-30T12:28:51.937470+0000","last_clean":"2023-10-30T12:28:51.937470+0000","last_became_active":"2023-10-30T12:28:50.373581+0000","last_became_peered":"2023-10-30T12:28:50.373581+0000","last_unstale":"2023-10-30T12:28:51.937470+0000","last_undegraded":"2023-10-30T12:28:51.937470+0000","last_fullsized":"2023-10-30T12:28:51.937470+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:14:51.506745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.983033+0000","last_change":"2023-10-30T12:28:50.974012+0000","last_active":"2023-10-30T12:28:50.983033+0000","last_peered":"2023-10-30T12:28:50.983033+0000","last_clean":"2023-10-30T12:28:50.983033+0000","last_became_active":"2023-10-30T12:28:50.973269+0000","last_became_peered":"2023-10-30T12:28:50.973269+0000","last_unstale":"2023-10-30T12:28:50.983033+0000","last_undegraded":"2023-10-30T12:28:50.983033+0000","last_fullsized":"2023-10-30T12:28:50.983033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:36:49.701055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.925887+0000","last_change":"2023-10-30T12:28:50.366590+0000","last_active":"2023-10-30T12:28:50.925887+0000","last_peered":"2023-10-30T12:28:50.925887+0000","last_clean":"2023-10-30T12:28:50.925887+0000","last_became_active":"2023-10-30T12:28:50.366269+0000","last_became_peered":"2023-10-30T12:28:50.366269+0000","last_unstale":"2023-10-30T12:28:50.925887+0000","last_undegraded":"2023-10-30T12:28:50.925887+0000","last_fullsized":"2023-10-30T12:28:50.925887+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:15:11.963438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.963379+0000","last_change":"2023-10-30T12:28:50.962058+0000","last_active":"2023-10-30T12:28:50.963379+0000","last_peered":"2023-10-30T12:28:50.963379+0000","last_clean":"2023-10-30T12:28:50.963379+0000","last_became_active":"2023-10-30T12:28:50.961805+0000","last_became_peered":"2023-10-30T12:28:50.961805+0000","last_unstale":"2023-10-30T12:28:50.963379+0000","last_undegraded":"2023-10-30T12:28:50.963379+0000","last_fullsized":"2023-10-30T12:28:50.963379+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:42:35.249989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.956067+0000","last_change":"2023-10-30T12:28:50.397244+0000","last_active":"2023-10-30T12:28:50.956067+0000","last_peered":"2023-10-30T12:28:50.956067+0000","last_clean":"2023-10-30T12:28:50.956067+0000","last_became_active":"2023-10-30T12:28:50.396727+0000","last_became_peered":"2023-10-30T12:28:50.396727+0000","last_unstale":"2023-10-30T12:28:50.956067+0000","last_undegraded":"2023-10-30T12:28:50.956067+0000","last_fullsized":"2023-10-30T12:28:50.956067+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:33.493343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.984268+0000","last_change":"2023-10-30T12:28:50.981498+0000","last_active":"2023-10-30T12:28:50.984268+0000","last_peered":"2023-10-30T12:28:50.984268+0000","last_clean":"2023-10-30T12:28:50.984268+0000","last_became_active":"2023-10-30T12:28:50.981063+0000","last_became_peered":"2023-10-30T12:28:50.981063+0000","last_unstale":"2023-10-30T12:28:50.984268+0000","last_undegraded":"2023-10-30T12:28:50.984268+0000","last_fullsized":"2023-10-30T12:28:50.984268+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:13:43.443210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.937078+0000","last_change":"2023-10-30T12:28:50.383880+0000","last_active":"2023-10-30T12:28:51.937078+0000","last_peered":"2023-10-30T12:28:51.937078+0000","last_clean":"2023-10-30T12:28:51.937078+0000","last_became_active":"2023-10-30T12:28:50.383570+0000","last_became_peered":"2023-10-30T12:28:50.383570+0000","last_unstale":"2023-10-30T12:28:51.937078+0000","last_undegraded":"2023-10-30T12:28:51.937078+0000","last_fullsized":"2023-10-30T12:28:51.937078+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:19:42.047520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.936983+0000","last_change":"2023-10-30T12:28:50.950312+0000","last_active":"2023-10-30T12:28:51.936983+0000","last_peered":"2023-10-30T12:28:51.936983+0000","last_clean":"2023-10-30T12:28:51.936983+0000","last_became_active":"2023-10-30T12:28:50.949485+0000","last_became_peered":"2023-10-30T12:28:50.949485+0000","last_unstale":"2023-10-30T12:28:51.936983+0000","last_undegraded":"2023-10-30T12:28:51.936983+0000","last_fullsized":"2023-10-30T12:28:51.936983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:13:04.669915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.924804+0000","last_change":"2023-10-30T12:28:50.362979+0000","last_active":"2023-10-30T12:28:50.924804+0000","last_peered":"2023-10-30T12:28:50.924804+0000","last_clean":"2023-10-30T12:28:50.924804+0000","last_became_active":"2023-10-30T12:28:50.362624+0000","last_became_peered":"2023-10-30T12:28:50.362624+0000","last_unstale":"2023-10-30T12:28:50.924804+0000","last_undegraded":"2023-10-30T12:28:50.924804+0000","last_fullsized":"2023-10-30T12:28:50.924804+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:01:13.111816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.984219+0000","last_change":"2023-10-30T12:28:50.979815+0000","last_active":"2023-10-30T12:28:50.984219+0000","last_peered":"2023-10-30T12:28:50.984219+0000","last_clean":"2023-10-30T12:28:50.984219+0000","last_became_active":"2023-10-30T12:28:50.979344+0000","last_became_peered":"2023-10-30T12:28:50.979344+0000","last_unstale":"2023-10-30T12:28:50.984219+0000","last_undegraded":"2023-10-30T12:28:50.984219+0000","last_fullsized":"2023-10-30T12:28:50.984219+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:57:05.538953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.940171+0000","last_change":"2023-10-30T12:28:50.382175+0000","last_active":"2023-10-30T12:28:51.940171+0000","last_peered":"2023-10-30T12:28:51.940171+0000","last_clean":"2023-10-30T12:28:51.940171+0000","last_became_active":"2023-10-30T12:28:50.381125+0000","last_became_peered":"2023-10-30T12:28:50.381125+0000","last_unstale":"2023-10-30T12:28:51.940171+0000","last_undegraded":"2023-10-30T12:28:51.940171+0000","last_fullsized":"2023-10-30T12:28:51.940171+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:36:02.652665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.983341+0000","last_change":"2023-10-30T12:28:50.978034+0000","last_active":"2023-10-30T12:28:50.983341+0000","last_peered":"2023-10-30T12:28:50.983341+0000","last_clean":"2023-10-30T12:28:50.983341+0000","last_became_active":"2023-10-30T12:28:50.977585+0000","last_became_peered":"2023-10-30T12:28:50.977585+0000","last_unstale":"2023-10-30T12:28:50.983341+0000","last_undegraded":"2023-10-30T12:28:50.983341+0000","last_fullsized":"2023-10-30T12:28:50.983341+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:21:32.868831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.924936+0000","last_change":"2023-10-30T12:28:50.367258+0000","last_active":"2023-10-30T12:28:50.924936+0000","last_peered":"2023-10-30T12:28:50.924936+0000","last_clean":"2023-10-30T12:28:50.924936+0000","last_became_active":"2023-10-30T12:28:50.366974+0000","last_became_peered":"2023-10-30T12:28:50.366974+0000","last_unstale":"2023-10-30T12:28:50.924936+0000","last_undegraded":"2023-10-30T12:28:50.924936+0000","last_fullsized":"2023-10-30T12:28:50.924936+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:14:48.823902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.984257+0000","last_change":"2023-10-30T12:28:50.981784+0000","last_active":"2023-10-30T12:28:50.984257+0000","last_peered":"2023-10-30T12:28:50.984257+0000","last_clean":"2023-10-30T12:28:50.984257+0000","last_became_active":"2023-10-30T12:28:50.981193+0000","last_became_peered":"2023-10-30T12:28:50.981193+0000","last_unstale":"2023-10-30T12:28:50.984257+0000","last_undegraded":"2023-10-30T12:28:50.984257+0000","last_fullsized":"2023-10-30T12:28:50.984257+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:22:53.466806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.932011+0000","last_change":"2023-10-30T12:28:50.368011+0000","last_active":"2023-10-30T12:28:51.932011+0000","last_peered":"2023-10-30T12:28:51.932011+0000","last_clean":"2023-10-30T12:28:51.932011+0000","last_became_active":"2023-10-30T12:28:50.367730+0000","last_became_peered":"2023-10-30T12:28:50.367730+0000","last_unstale":"2023-10-30T12:28:51.932011+0000","last_undegraded":"2023-10-30T12:28:51.932011+0000","last_fullsized":"2023-10-30T12:28:51.932011+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:59:41.946550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.978412+0000","last_change":"2023-10-30T12:28:50.968262+0000","last_active":"2023-10-30T12:28:50.978412+0000","last_peered":"2023-10-30T12:28:50.978412+0000","last_clean":"2023-10-30T12:28:50.978412+0000","last_became_active":"2023-10-30T12:28:50.966246+0000","last_became_peered":"2023-10-30T12:28:50.966246+0000","last_unstale":"2023-10-30T12:28:50.978412+0000","last_undegraded":"2023-10-30T12:28:50.978412+0000","last_fullsized":"2023-10-30T12:28:50.978412+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:16:29.181523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.925526+0000","last_change":"2023-10-30T12:28:50.394060+0000","last_active":"2023-10-30T12:28:50.925526+0000","last_peered":"2023-10-30T12:28:50.925526+0000","last_clean":"2023-10-30T12:28:50.925526+0000","last_became_active":"2023-10-30T12:28:50.393642+0000","last_became_peered":"2023-10-30T12:28:50.393642+0000","last_unstale":"2023-10-30T12:28:50.925526+0000","last_undegraded":"2023-10-30T12:28:50.925526+0000","last_fullsized":"2023-10-30T12:28:50.925526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:21.643882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.983297+0000","last_change":"2023-10-30T12:28:50.976814+0000","last_active":"2023-10-30T12:28:50.983297+0000","last_peered":"2023-10-30T12:28:50.983297+0000","last_clean":"2023-10-30T12:28:50.983297+0000","last_became_active":"2023-10-30T12:28:50.976343+0000","last_became_peered":"2023-10-30T12:28:50.976343+0000","last_unstale":"2023-10-30T12:28:50.983297+0000","last_undegraded":"2023-10-30T12:28:50.983297+0000","last_fullsized":"2023-10-30T12:28:50.983297+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:19:44.549840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":164,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.371491+0000","last_change":"2023-10-30T12:28:41.203990+0000","last_active":"2023-10-30T12:28:50.371491+0000","last_peered":"2023-10-30T12:28:50.371491+0000","last_clean":"2023-10-30T12:28:50.371491+0000","last_became_active":"2023-10-30T12:28:41.201796+0000","last_became_peered":"2023-10-30T12:28:41.201796+0000","last_unstale":"2023-10-30T12:28:50.371491+0000","last_undegraded":"2023-10-30T12:28:50.371491+0000","last_fullsized":"2023-10-30T12:28:50.371491+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:39.218257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:39.218257+0000","last_clean_scrub_stamp":"2023-10-30T12:28:39.218257+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:21:00.605462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.956372+0000","last_change":"2023-10-30T12:28:50.372651+0000","last_active":"2023-10-30T12:28:50.956372+0000","last_peered":"2023-10-30T12:28:50.956372+0000","last_clean":"2023-10-30T12:28:50.956372+0000","last_became_active":"2023-10-30T12:28:50.367593+0000","last_became_peered":"2023-10-30T12:28:50.367593+0000","last_unstale":"2023-10-30T12:28:50.956372+0000","last_undegraded":"2023-10-30T12:28:50.956372+0000","last_fullsized":"2023-10-30T12:28:50.956372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:32:10.492061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.974612+0000","last_change":"2023-10-30T12:28:50.970536+0000","last_active":"2023-10-30T12:28:50.974612+0000","last_peered":"2023-10-30T12:28:50.974612+0000","last_clean":"2023-10-30T12:28:50.974612+0000","last_became_active":"2023-10-30T12:28:50.970326+0000","last_became_peered":"2023-10-30T12:28:50.970326+0000","last_unstale":"2023-10-30T12:28:50.974612+0000","last_undegraded":"2023-10-30T12:28:50.974612+0000","last_fullsized":"2023-10-30T12:28:50.974612+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:50.431623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.945861+0000","last_change":"2023-10-30T12:28:50.392461+0000","last_active":"2023-10-30T12:28:50.945861+0000","last_peered":"2023-10-30T12:28:50.945861+0000","last_clean":"2023-10-30T12:28:50.945861+0000","last_became_active":"2023-10-30T12:28:50.391824+0000","last_became_peered":"2023-10-30T12:28:50.391824+0000","last_unstale":"2023-10-30T12:28:50.945861+0000","last_undegraded":"2023-10-30T12:28:50.945861+0000","last_fullsized":"2023-10-30T12:28:50.945861+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:56:10.373046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.957824+0000","last_change":"2023-10-30T12:28:50.953046+0000","last_active":"2023-10-30T12:28:50.957824+0000","last_peered":"2023-10-30T12:28:50.957824+0000","last_clean":"2023-10-30T12:28:50.957824+0000","last_became_active":"2023-10-30T12:28:50.952625+0000","last_became_peered":"2023-10-30T12:28:50.952625+0000","last_unstale":"2023-10-30T12:28:50.957824+0000","last_undegraded":"2023-10-30T12:28:50.957824+0000","last_fullsized":"2023-10-30T12:28:50.957824+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:07:42.377331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.924652+0000","last_change":"2023-10-30T12:28:50.367547+0000","last_active":"2023-10-30T12:28:50.924652+0000","last_peered":"2023-10-30T12:28:50.924652+0000","last_clean":"2023-10-30T12:28:50.924652+0000","last_became_active":"2023-10-30T12:28:50.367230+0000","last_became_peered":"2023-10-30T12:28:50.367230+0000","last_unstale":"2023-10-30T12:28:50.924652+0000","last_undegraded":"2023-10-30T12:28:50.924652+0000","last_fullsized":"2023-10-30T12:28:50.924652+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:42.570786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931639+0000","last_change":"2023-10-30T12:28:50.928888+0000","last_active":"2023-10-30T12:28:51.931639+0000","last_peered":"2023-10-30T12:28:51.931639+0000","last_clean":"2023-10-30T12:28:51.931639+0000","last_became_active":"2023-10-30T12:28:50.928608+0000","last_became_peered":"2023-10-30T12:28:50.928608+0000","last_unstale":"2023-10-30T12:28:51.931639+0000","last_undegraded":"2023-10-30T12:28:51.931639+0000","last_fullsized":"2023-10-30T12:28:51.931639+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:40:51.042396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.956457+0000","last_change":"2023-10-30T12:28:50.391240+0000","last_active":"2023-10-30T12:28:50.956457+0000","last_peered":"2023-10-30T12:28:50.956457+0000","last_clean":"2023-10-30T12:28:50.956457+0000","last_became_active":"2023-10-30T12:28:50.390864+0000","last_became_peered":"2023-10-30T12:28:50.390864+0000","last_unstale":"2023-10-30T12:28:50.956457+0000","last_undegraded":"2023-10-30T12:28:50.956457+0000","last_fullsized":"2023-10-30T12:28:50.956457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:59:40.285830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.939756+0000","last_change":"2023-10-30T12:28:50.949990+0000","last_active":"2023-10-30T12:28:51.939756+0000","last_peered":"2023-10-30T12:28:51.939756+0000","last_clean":"2023-10-30T12:28:51.939756+0000","last_became_active":"2023-10-30T12:28:50.949122+0000","last_became_peered":"2023-10-30T12:28:50.949122+0000","last_unstale":"2023-10-30T12:28:51.939756+0000","last_undegraded":"2023-10-30T12:28:51.939756+0000","last_fullsized":"2023-10-30T12:28:51.939756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:30:56.099810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931803+0000","last_change":"2023-10-30T12:28:50.358700+0000","last_active":"2023-10-30T12:28:51.931803+0000","last_peered":"2023-10-30T12:28:51.931803+0000","last_clean":"2023-10-30T12:28:51.931803+0000","last_became_active":"2023-10-30T12:28:50.358310+0000","last_became_peered":"2023-10-30T12:28:50.358310+0000","last_unstale":"2023-10-30T12:28:51.931803+0000","last_undegraded":"2023-10-30T12:28:51.931803+0000","last_fullsized":"2023-10-30T12:28:51.931803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:23:30.910622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.978335+0000","last_change":"2023-10-30T12:28:50.967650+0000","last_active":"2023-10-30T12:28:50.978335+0000","last_peered":"2023-10-30T12:28:50.978335+0000","last_clean":"2023-10-30T12:28:50.978335+0000","last_became_active":"2023-10-30T12:28:50.965626+0000","last_became_peered":"2023-10-30T12:28:50.965626+0000","last_unstale":"2023-10-30T12:28:50.978335+0000","last_undegraded":"2023-10-30T12:28:50.978335+0000","last_fullsized":"2023-10-30T12:28:50.978335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:57:33.494828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.960484+0000","last_change":"2023-10-30T12:28:50.395645+0000","last_active":"2023-10-30T12:28:50.960484+0000","last_peered":"2023-10-30T12:28:50.960484+0000","last_clean":"2023-10-30T12:28:50.960484+0000","last_became_active":"2023-10-30T12:28:50.395090+0000","last_became_peered":"2023-10-30T12:28:50.395090+0000","last_unstale":"2023-10-30T12:28:50.960484+0000","last_undegraded":"2023-10-30T12:28:50.960484+0000","last_fullsized":"2023-10-30T12:28:50.960484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:45:44.678383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931844+0000","last_change":"2023-10-30T12:28:50.929586+0000","last_active":"2023-10-30T12:28:51.931844+0000","last_peered":"2023-10-30T12:28:51.931844+0000","last_clean":"2023-10-30T12:28:51.931844+0000","last_became_active":"2023-10-30T12:28:50.929337+0000","last_became_peered":"2023-10-30T12:28:50.929337+0000","last_unstale":"2023-10-30T12:28:51.931844+0000","last_undegraded":"2023-10-30T12:28:51.931844+0000","last_fullsized":"2023-10-30T12:28:51.931844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:11:13.067698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.932075+0000","last_change":"2023-10-30T12:28:50.380474+0000","last_active":"2023-10-30T12:28:51.932075+0000","last_peered":"2023-10-30T12:28:51.932075+0000","last_clean":"2023-10-30T12:28:51.932075+0000","last_became_active":"2023-10-30T12:28:50.380247+0000","last_became_peered":"2023-10-30T12:28:50.380247+0000","last_unstale":"2023-10-30T12:28:51.932075+0000","last_undegraded":"2023-10-30T12:28:51.932075+0000","last_fullsized":"2023-10-30T12:28:51.932075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:52:03.952617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.977080+0000","last_change":"2023-10-30T12:28:50.973361+0000","last_active":"2023-10-30T12:28:50.977080+0000","last_peered":"2023-10-30T12:28:50.977080+0000","last_clean":"2023-10-30T12:28:50.977080+0000","last_became_active":"2023-10-30T12:28:50.973086+0000","last_became_peered":"2023-10-30T12:28:50.973086+0000","last_unstale":"2023-10-30T12:28:50.977080+0000","last_undegraded":"2023-10-30T12:28:50.977080+0000","last_fullsized":"2023-10-30T12:28:50.977080+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:20:16.414696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.937662+0000","last_change":"2023-10-30T12:28:50.386277+0000","last_active":"2023-10-30T12:28:51.937662+0000","last_peered":"2023-10-30T12:28:51.937662+0000","last_clean":"2023-10-30T12:28:51.937662+0000","last_became_active":"2023-10-30T12:28:50.385924+0000","last_became_peered":"2023-10-30T12:28:50.385924+0000","last_unstale":"2023-10-30T12:28:51.937662+0000","last_undegraded":"2023-10-30T12:28:51.937662+0000","last_fullsized":"2023-10-30T12:28:51.937662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:35:03.837523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.933959+0000","last_change":"2023-10-30T12:28:50.929670+0000","last_active":"2023-10-30T12:28:50.933959+0000","last_peered":"2023-10-30T12:28:50.933959+0000","last_clean":"2023-10-30T12:28:50.933959+0000","last_became_active":"2023-10-30T12:28:50.929259+0000","last_became_peered":"2023-10-30T12:28:50.929259+0000","last_unstale":"2023-10-30T12:28:50.933959+0000","last_undegraded":"2023-10-30T12:28:50.933959+0000","last_fullsized":"2023-10-30T12:28:50.933959+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:50:04.591856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.945507+0000","last_change":"2023-10-30T12:28:50.386426+0000","last_active":"2023-10-30T12:28:50.945507+0000","last_peered":"2023-10-30T12:28:50.945507+0000","last_clean":"2023-10-30T12:28:50.945507+0000","last_became_active":"2023-10-30T12:28:50.385305+0000","last_became_peered":"2023-10-30T12:28:50.385305+0000","last_unstale":"2023-10-30T12:28:50.945507+0000","last_undegraded":"2023-10-30T12:28:50.945507+0000","last_fullsized":"2023-10-30T12:28:50.945507+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:13:05.376581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.938368+0000","last_change":"2023-10-30T12:28:50.941607+0000","last_active":"2023-10-30T12:28:51.938368+0000","last_peered":"2023-10-30T12:28:51.938368+0000","last_clean":"2023-10-30T12:28:51.938368+0000","last_became_active":"2023-10-30T12:28:50.940846+0000","last_became_peered":"2023-10-30T12:28:50.940846+0000","last_unstale":"2023-10-30T12:28:51.938368+0000","last_undegraded":"2023-10-30T12:28:51.938368+0000","last_fullsized":"2023-10-30T12:28:51.938368+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:29.837077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931710+0000","last_change":"2023-10-30T12:28:50.392333+0000","last_active":"2023-10-30T12:28:51.931710+0000","last_peered":"2023-10-30T12:28:51.931710+0000","last_clean":"2023-10-30T12:28:51.931710+0000","last_became_active":"2023-10-30T12:28:50.392068+0000","last_became_peered":"2023-10-30T12:28:50.392068+0000","last_unstale":"2023-10-30T12:28:51.931710+0000","last_undegraded":"2023-10-30T12:28:51.931710+0000","last_fullsized":"2023-10-30T12:28:51.931710+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:47:24.674925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.937725+0000","last_change":"2023-10-30T12:28:50.932167+0000","last_active":"2023-10-30T12:28:50.937725+0000","last_peered":"2023-10-30T12:28:50.937725+0000","last_clean":"2023-10-30T12:28:50.937725+0000","last_became_active":"2023-10-30T12:28:50.931898+0000","last_became_peered":"2023-10-30T12:28:50.931898+0000","last_unstale":"2023-10-30T12:28:50.937725+0000","last_undegraded":"2023-10-30T12:28:50.937725+0000","last_fullsized":"2023-10-30T12:28:50.937725+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:02:42.275987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931812+0000","last_change":"2023-10-30T12:28:50.365055+0000","last_active":"2023-10-30T12:28:51.931812+0000","last_peered":"2023-10-30T12:28:51.931812+0000","last_clean":"2023-10-30T12:28:51.931812+0000","last_became_active":"2023-10-30T12:28:50.364797+0000","last_became_peered":"2023-10-30T12:28:50.364797+0000","last_unstale":"2023-10-30T12:28:51.931812+0000","last_undegraded":"2023-10-30T12:28:51.931812+0000","last_fullsized":"2023-10-30T12:28:51.931812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:06:23.067895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.938503+0000","last_change":"2023-10-30T12:28:50.935908+0000","last_active":"2023-10-30T12:28:50.938503+0000","last_peered":"2023-10-30T12:28:50.938503+0000","last_clean":"2023-10-30T12:28:50.938503+0000","last_became_active":"2023-10-30T12:28:50.935657+0000","last_became_peered":"2023-10-30T12:28:50.935657+0000","last_unstale":"2023-10-30T12:28:50.938503+0000","last_undegraded":"2023-10-30T12:28:50.938503+0000","last_fullsized":"2023-10-30T12:28:50.938503+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:27:34.869337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.948828+0000","last_change":"2023-10-30T12:28:50.393465+0000","last_active":"2023-10-30T12:28:50.948828+0000","last_peered":"2023-10-30T12:28:50.948828+0000","last_clean":"2023-10-30T12:28:50.948828+0000","last_became_active":"2023-10-30T12:28:50.392807+0000","last_became_peered":"2023-10-30T12:28:50.392807+0000","last_unstale":"2023-10-30T12:28:50.948828+0000","last_undegraded":"2023-10-30T12:28:50.948828+0000","last_fullsized":"2023-10-30T12:28:50.948828+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:35:37.976956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.938767+0000","last_change":"2023-10-30T12:28:50.954834+0000","last_active":"2023-10-30T12:28:51.938767+0000","last_peered":"2023-10-30T12:28:51.938767+0000","last_clean":"2023-10-30T12:28:51.938767+0000","last_became_active":"2023-10-30T12:28:50.954291+0000","last_became_peered":"2023-10-30T12:28:50.954291+0000","last_unstale":"2023-10-30T12:28:51.938767+0000","last_undegraded":"2023-10-30T12:28:51.938767+0000","last_fullsized":"2023-10-30T12:28:51.938767+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:19:44.345918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.975322+0000","last_change":"2023-10-30T12:28:50.963620+0000","last_active":"2023-10-30T12:28:50.975322+0000","last_peered":"2023-10-30T12:28:50.975322+0000","last_clean":"2023-10-30T12:28:50.975322+0000","last_became_active":"2023-10-30T12:28:50.962817+0000","last_became_peered":"2023-10-30T12:28:50.962817+0000","last_unstale":"2023-10-30T12:28:50.975322+0000","last_undegraded":"2023-10-30T12:28:50.975322+0000","last_fullsized":"2023-10-30T12:28:50.975322+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:34:54.782063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.946929+0000","last_change":"2023-10-30T12:28:50.402179+0000","last_active":"2023-10-30T12:28:50.946929+0000","last_peered":"2023-10-30T12:28:50.946929+0000","last_clean":"2023-10-30T12:28:50.946929+0000","last_became_active":"2023-10-30T12:28:50.401677+0000","last_became_peered":"2023-10-30T12:28:50.401677+0000","last_unstale":"2023-10-30T12:28:50.946929+0000","last_undegraded":"2023-10-30T12:28:50.946929+0000","last_fullsized":"2023-10-30T12:28:50.946929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:39:44.432214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.978448+0000","last_change":"2023-10-30T12:28:50.968805+0000","last_active":"2023-10-30T12:28:50.978448+0000","last_peered":"2023-10-30T12:28:50.978448+0000","last_clean":"2023-10-30T12:28:50.978448+0000","last_became_active":"2023-10-30T12:28:50.967591+0000","last_became_peered":"2023-10-30T12:28:50.967591+0000","last_unstale":"2023-10-30T12:28:50.978448+0000","last_undegraded":"2023-10-30T12:28:50.978448+0000","last_fullsized":"2023-10-30T12:28:50.978448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:25:43.057459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.949875+0000","last_change":"2023-10-30T12:28:50.392617+0000","last_active":"2023-10-30T12:28:50.949875+0000","last_peered":"2023-10-30T12:28:50.949875+0000","last_clean":"2023-10-30T12:28:50.949875+0000","last_became_active":"2023-10-30T12:28:50.392021+0000","last_became_peered":"2023-10-30T12:28:50.392021+0000","last_unstale":"2023-10-30T12:28:50.949875+0000","last_undegraded":"2023-10-30T12:28:50.949875+0000","last_fullsized":"2023-10-30T12:28:50.949875+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:52:30.621125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.969439+0000","last_change":"2023-10-30T12:28:50.967491+0000","last_active":"2023-10-30T12:28:50.969439+0000","last_peered":"2023-10-30T12:28:50.969439+0000","last_clean":"2023-10-30T12:28:50.969439+0000","last_became_active":"2023-10-30T12:28:50.967186+0000","last_became_peered":"2023-10-30T12:28:50.967186+0000","last_unstale":"2023-10-30T12:28:50.969439+0000","last_undegraded":"2023-10-30T12:28:50.969439+0000","last_fullsized":"2023-10-30T12:28:50.969439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:21:15.521473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.930309+0000","last_change":"2023-10-30T12:28:50.363230+0000","last_active":"2023-10-30T12:28:50.930309+0000","last_peered":"2023-10-30T12:28:50.930309+0000","last_clean":"2023-10-30T12:28:50.930309+0000","last_became_active":"2023-10-30T12:28:50.362929+0000","last_became_peered":"2023-10-30T12:28:50.362929+0000","last_unstale":"2023-10-30T12:28:50.930309+0000","last_undegraded":"2023-10-30T12:28:50.930309+0000","last_fullsized":"2023-10-30T12:28:50.930309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:41:27.760578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.940360+0000","last_change":"2023-10-30T12:28:50.937780+0000","last_active":"2023-10-30T12:28:50.940360+0000","last_peered":"2023-10-30T12:28:50.940360+0000","last_clean":"2023-10-30T12:28:50.940360+0000","last_became_active":"2023-10-30T12:28:50.937458+0000","last_became_peered":"2023-10-30T12:28:50.937458+0000","last_unstale":"2023-10-30T12:28:50.940360+0000","last_undegraded":"2023-10-30T12:28:50.940360+0000","last_fullsized":"2023-10-30T12:28:50.940360+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:49.787839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.926291+0000","last_change":"2023-10-30T12:28:50.386782+0000","last_active":"2023-10-30T12:28:50.926291+0000","last_peered":"2023-10-30T12:28:50.926291+0000","last_clean":"2023-10-30T12:28:50.926291+0000","last_became_active":"2023-10-30T12:28:50.386522+0000","last_became_peered":"2023-10-30T12:28:50.386522+0000","last_unstale":"2023-10-30T12:28:50.926291+0000","last_undegraded":"2023-10-30T12:28:50.926291+0000","last_fullsized":"2023-10-30T12:28:50.926291+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:44:03.517641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":806912,"data_stored":662932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0149999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.228}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.181}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.359}]}]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":78864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.131}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.264}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3380000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542150,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":71945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0640000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1379999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.089}]}]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":78864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0740000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.21}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1779999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]}]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":212992,"data_stored":72564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]}]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":71945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.87}]}]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":72564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]}]},{"osd":0,"up_from":14,"seq":60129542150,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":802816,"data_stored":662313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80700000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:28:57.943 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-30T12:28:57.944 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-30T12:28:57.944 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-30T12:28:57.944 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-30T12:28:57.945 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-30T12:28:58.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d39303640 1 Processor -- start 2023-10-30T12:28:58.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d39303640 1 -- start start 2023-10-30T12:28:58.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d39303640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4d3415cd50 0x7f4d340c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d39303640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4d340cefa0 0x7f4d340cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d39303640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d3415a6c0 con 0x7f4d341538f0 2023-10-30T12:28:58.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d39303640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d3415d300 con 0x7f4d3415cd50 2023-10-30T12:28:58.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d39303640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3415d5e0 con 0x7f4d340cefa0 2023-10-30T12:28:58.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d33fff640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f4d341538f0 0x7f4d34159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55710/0 (socket says 172.21.15.89:55710) 2023-10-30T12:28:58.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d33fff640 1 -- 172.21.15.89:0/2907987506 learned_addr learned my addr 172.21.15.89:0/2907987506 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:28:58.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d337fe640 1 --2- 172.21.15.89:0/2907987506 >> v2:172.21.15.192:3300/0 conn(0x7f4d3415cd50 0x7f4d340c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d38b02640 1 --2- 172.21.15.89:0/2907987506 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4d340cefa0 0x7f4d340cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d32ffd640 1 -- 172.21.15.89:0/2907987506 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3629295087 0 0) 0x7f4d3415a6c0 con 0x7f4d341538f0 2023-10-30T12:28:58.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d32ffd640 1 -- 172.21.15.89:0/2907987506 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d1c003530 con 0x7f4d341538f0 2023-10-30T12:28:58.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d32ffd640 1 -- 172.21.15.89:0/2907987506 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2465183723 0 0) 0x7f4d1c003530 con 0x7f4d341538f0 2023-10-30T12:28:58.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d32ffd640 1 -- 172.21.15.89:0/2907987506 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4d340cefa0 msgr2=0x7f4d340cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:58.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d32ffd640 1 --2- 172.21.15.89:0/2907987506 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4d340cefa0 0x7f4d340cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.126 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d32ffd640 1 -- 172.21.15.89:0/2907987506 >> v2:172.21.15.192:3300/0 conn(0x7f4d3415cd50 msgr2=0x7f4d340c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:58.126 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d32ffd640 1 --2- 172.21.15.89:0/2907987506 >> v2:172.21.15.192:3300/0 conn(0x7f4d3415cd50 0x7f4d340c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.126 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d32ffd640 1 -- 172.21.15.89:0/2907987506 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d3415d8c0 con 0x7f4d341538f0 2023-10-30T12:28:58.126 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d32ffd640 1 -- 172.21.15.89:0/2907987506 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f4d24001680 con 0x7f4d341538f0 2023-10-30T12:28:58.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d38b02640 1 --2- 172.21.15.89:0/2907987506 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4d340cefa0 0x7f4d340cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:58.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d32ffd640 1 -- 172.21.15.89:0/2907987506 >> v1:172.21.15.89:6789/0 conn(0x7f4d341538f0 legacy=0x7f4d34159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d32ffd640 1 --2- 172.21.15.89:0/2907987506 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d1c003980 0x7f4d1c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d32ffd640 1 --2- 172.21.15.89:0/2907987506 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4d340cefa0 0x7f4d1c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d32ffd640 1 --2- 172.21.15.89:0/2907987506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c004d10 0x7f4d1c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d32ffd640 1 -- 172.21.15.89:0/2907987506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3415d300 con 0x7f4d1c004d10 2023-10-30T12:28:58.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d32ffd640 1 -- 172.21.15.89:0/2907987506 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3415d5e0 con 0x7f4d1c003980 2023-10-30T12:28:58.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d32ffd640 1 -- 172.21.15.89:0/2907987506 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c00b990 con 0x7f4d340cefa0 2023-10-30T12:28:58.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d38b02640 1 --2- 172.21.15.89:0/2907987506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c004d10 0x7f4d1c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d33fff640 1 --2- 172.21.15.89:0/2907987506 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d1c003980 0x7f4d1c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d337fe640 1 --2- 172.21.15.89:0/2907987506 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4d340cefa0 0x7f4d1c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d38b02640 1 -- 172.21.15.89:0/2907987506 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4d340cefa0 msgr2=0x7f4d1c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:58.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d38b02640 1 --2- 172.21.15.89:0/2907987506 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4d340cefa0 0x7f4d1c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d38b02640 1 -- 172.21.15.89:0/2907987506 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d1c003980 msgr2=0x7f4d1c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:58.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d38b02640 1 --2- 172.21.15.89:0/2907987506 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d1c003980 0x7f4d1c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d38b02640 1 -- 172.21.15.89:0/2907987506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d2c009a70 con 0x7f4d1c004d10 2023-10-30T12:28:58.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.116+0000 7f4d337fe640 1 --2- 172.21.15.89:0/2907987506 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4d340cefa0 0x7f4d1c0047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:58.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.120+0000 7f4d38b02640 1 --2- 172.21.15.89:0/2907987506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c004d10 0x7f4d1c005190 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c00e350 tx=0x7f4d2c01cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f876440a57208d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:58.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.120+0000 7f4d32ffd640 1 -- 172.21.15.89:0/2907987506 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c0a5df0 con 0x7f4d1c004d10 2023-10-30T12:28:58.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.120+0000 7f4d32ffd640 1 -- 172.21.15.89:0/2907987506 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d2c00f040 con 0x7f4d1c004d10 2023-10-30T12:28:58.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.120+0000 7f4d32ffd640 1 -- 172.21.15.89:0/2907987506 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c0ad640 con 0x7f4d1c004d10 2023-10-30T12:28:58.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.120+0000 7f4d39303640 1 -- 172.21.15.89:0/2907987506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c004d10 msgr2=0x7f4d1c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.120+0000 7f4d39303640 1 --2- 172.21.15.89:0/2907987506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c004d10 0x7f4d1c005190 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c00e350 tx=0x7f4d2c01cde0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.120+0000 7f4d38b02640 1 -- 172.21.15.89:0/2907987506 reap_dead start 2023-10-30T12:28:58.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.120+0000 7f4d39303640 1 -- 172.21.15.89:0/2907987506 shutdown_connections 2023-10-30T12:28:58.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.120+0000 7f4d39303640 1 -- 172.21.15.89:0/2907987506 >> 172.21.15.89:0/2907987506 conn(0x7f4d3406a640 msgr2=0x7f4d34065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:58.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.120+0000 7f4d39303640 1 -- 172.21.15.89:0/2907987506 shutdown_connections 2023-10-30T12:28:58.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.120+0000 7f4d39303640 1 -- 172.21.15.89:0/2907987506 wait complete. 2023-10-30T12:28:58.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.124+0000 7f4d39303640 1 Processor -- start 2023-10-30T12:28:58.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.124+0000 7f4d39303640 1 -- start start 2023-10-30T12:28:58.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.124+0000 7f4d39303640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c004d10 0x7f4d34147dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.124+0000 7f4d39303640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4d34152f10 0x7f4d340b2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.124+0000 7f4d39303640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d340b3350 0x7f4d3414fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.124+0000 7f4d39303640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d341f3310 con 0x7f4d1c004d10 2023-10-30T12:28:58.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.124+0000 7f4d39303640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d341f2d90 con 0x7f4d340b3350 2023-10-30T12:28:58.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.124+0000 7f4d39303640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d341efdc0 con 0x7f4d34152f10 2023-10-30T12:28:58.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.124+0000 7f4d33fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c004d10 0x7f4d34147dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.124+0000 7f4d33fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c004d10 0x7f4d34147dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55392/0 (socket says 172.21.15.89:55392) 2023-10-30T12:28:58.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.124+0000 7f4d33fff640 1 -- 172.21.15.89:0/2311081725 learned_addr learned my addr 172.21.15.89:0/2311081725 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:58.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.124+0000 7f4d38b02640 1 --2- 172.21.15.89:0/2311081725 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d340b3350 0x7f4d3414fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.124+0000 7f4d337fe640 1 --2- 172.21.15.89:0/2311081725 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4d34152f10 0x7f4d340b2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.124+0000 7f4d33fff640 1 -- 172.21.15.89:0/2311081725 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4d34152f10 msgr2=0x7f4d340b2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.124+0000 7f4d33fff640 1 --2- 172.21.15.89:0/2311081725 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4d34152f10 0x7f4d340b2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.124+0000 7f4d33fff640 1 -- 172.21.15.89:0/2311081725 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d340b3350 msgr2=0x7f4d3414fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.124+0000 7f4d33fff640 1 --2- 172.21.15.89:0/2311081725 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d340b3350 0x7f4d3414fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.124+0000 7f4d33fff640 1 -- 172.21.15.89:0/2311081725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d341f3030 con 0x7f4d1c004d10 2023-10-30T12:28:58.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.124+0000 7f4d33fff640 1 --2- 172.21.15.89:0/2311081725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c004d10 0x7f4d34147dc0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4d24007310 tx=0x7f4d240077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:58.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.124+0000 7f4d30ff9640 1 -- 172.21.15.89:0/2311081725 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d2400f9c0 con 0x7f4d1c004d10 2023-10-30T12:28:58.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.124+0000 7f4d30ff9640 1 -- 172.21.15.89:0/2311081725 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d2400fb60 con 0x7f4d1c004d10 2023-10-30T12:28:58.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.124+0000 7f4d39303640 1 -- 172.21.15.89:0/2311081725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d341ef000 con 0x7f4d1c004d10 2023-10-30T12:28:58.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.124+0000 7f4d30ff9640 1 -- 172.21.15.89:0/2311081725 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d240133f0 con 0x7f4d1c004d10 2023-10-30T12:28:58.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.124+0000 7f4d39303640 1 -- 172.21.15.89:0/2311081725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d341ef5c0 con 0x7f4d1c004d10 2023-10-30T12:28:58.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.128+0000 7f4d39303640 1 -- 172.21.15.89:0/2311081725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d3404e170 con 0x7f4d1c004d10 2023-10-30T12:28:58.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.128+0000 7f4d30ff9640 1 -- 172.21.15.89:0/2311081725 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4d24013550 con 0x7f4d1c004d10 2023-10-30T12:28:58.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.128+0000 7f4d30ff9640 1 --2- 172.21.15.89:0/2311081725 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4d200664a0 0x7f4d20068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.128+0000 7f4d30ff9640 1 -- 172.21.15.89:0/2311081725 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4d240841e0 con 0x7f4d1c004d10 2023-10-30T12:28:58.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.128+0000 7f4d337fe640 1 --2- 172.21.15.89:0/2311081725 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4d200664a0 0x7f4d20068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.128+0000 7f4d337fe640 1 --2- 172.21.15.89:0/2311081725 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4d200664a0 0x7f4d20068960 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4d340b1b00 tx=0x7f4d28008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:58.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.132+0000 7f4d30ff9640 1 -- 172.21.15.89:0/2311081725 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4d240515c0 con 0x7f4d1c004d10 2023-10-30T12:28:58.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.320+0000 7f4d39303640 1 -- 172.21.15.89:0/2311081725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f4d340b2290 con 0x7f4d1c004d10 2023-10-30T12:28:58.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.324+0000 7f4d30ff9640 1 -- 172.21.15.89:0/2311081725 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+164437 (secure 0 0 0) 0x7f4d24054be0 con 0x7f4d1c004d10 2023-10-30T12:28:58.330 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:28:58.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.324+0000 7f4d39303640 1 -- 172.21.15.89:0/2311081725 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4d200664a0 msgr2=0x7f4d20068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.324+0000 7f4d39303640 1 --2- 172.21.15.89:0/2311081725 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4d200664a0 0x7f4d20068960 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4d340b1b00 tx=0x7f4d28008040 comp rx=0 tx=0).stop 2023-10-30T12:28:58.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.324+0000 7f4d39303640 1 -- 172.21.15.89:0/2311081725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c004d10 msgr2=0x7f4d34147dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.324+0000 7f4d39303640 1 --2- 172.21.15.89:0/2311081725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c004d10 0x7f4d34147dc0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4d24007310 tx=0x7f4d240077d0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.324+0000 7f4d39303640 1 -- 172.21.15.89:0/2311081725 shutdown_connections 2023-10-30T12:28:58.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.324+0000 7f4d39303640 1 --2- 172.21.15.89:0/2311081725 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4d200664a0 0x7f4d20068960 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.324+0000 7f4d39303640 1 --2- 172.21.15.89:0/2311081725 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4d340b3350 0x7f4d3414fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.324+0000 7f4d39303640 1 --2- 172.21.15.89:0/2311081725 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4d34152f10 0x7f4d340b2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.324+0000 7f4d39303640 1 --2- 172.21.15.89:0/2311081725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4d1c004d10 0x7f4d34147dc0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.324+0000 7f4d39303640 1 -- 172.21.15.89:0/2311081725 >> 172.21.15.89:0/2311081725 conn(0x7f4d3406a640 msgr2=0x7f4d3406c680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:58.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.324+0000 7f4d39303640 1 -- 172.21.15.89:0/2311081725 shutdown_connections 2023-10-30T12:28:58.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.324+0000 7f4d39303640 1 -- 172.21.15.89:0/2311081725 wait complete. 2023-10-30T12:28:58.350 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":5,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4076276638},{"type":"v1","addr":"172.21.15.192:6833","nonce":4076276638}]},"active_addr":"172.21.15.192:6833/4076276638","active_change":"2023-10-30T12:28:30.904394+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4106,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.192:0","nonce":3893213247}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.192:0","nonce":3642602388}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.192:0","nonce":1389649332}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.192:0","nonce":2041538997}]}]} 2023-10-30T12:28:58.354 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-30T12:28:58.354 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-30T12:28:58.354 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:28:58.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.492+0000 7f23d1f65640 1 Processor -- start 2023-10-30T12:28:58.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.492+0000 7f23d1f65640 1 -- start start 2023-10-30T12:28:58.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.492+0000 7f23d1f65640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f23cc15d6f0 0x7f23cc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.492+0000 7f23d1f65640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f23cc0cf160 0x7f23cc0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.492+0000 7f23d1f65640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23cc0cb800 con 0x7f23cc1538a0 2023-10-30T12:28:58.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.492+0000 7f23d1f65640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23cc0cb330 con 0x7f23cc0cf160 2023-10-30T12:28:58.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.492+0000 7f23d1f65640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23cc0cad70 con 0x7f23cc15d6f0 2023-10-30T12:28:58.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.492+0000 7f23cbfff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f23cc0cf160 0x7f23cc0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.492+0000 7f23cbfff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f23cc0cf160 0x7f23cc0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55402/0 (socket says 172.21.15.89:55402) 2023-10-30T12:28:58.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.492+0000 7f23cbfff640 1 -- 172.21.15.89:0/2284419611 learned_addr learned my addr 172.21.15.89:0/2284419611 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:58.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.492+0000 7f23cb7fe640 1 --1- 172.21.15.89:0/2284419611 >> v1:172.21.15.89:6789/0 conn(0x7f23cc1538a0 0x7f23cc0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55718/0 (socket says 172.21.15.89:55718) 2023-10-30T12:28:58.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.492+0000 7f23cbfff640 1 -- 172.21.15.89:0/2284419611 >> v2:172.21.15.192:3300/0 conn(0x7f23cc15d6f0 msgr2=0x7f23cc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:58.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.492+0000 7f23cbfff640 1 --2- 172.21.15.89:0/2284419611 >> v2:172.21.15.192:3300/0 conn(0x7f23cc15d6f0 0x7f23cc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.492+0000 7f23cbfff640 1 -- 172.21.15.89:0/2284419611 >> v1:172.21.15.89:6789/0 conn(0x7f23cc1538a0 legacy=0x7f23cc0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.492+0000 7f23cbfff640 1 -- 172.21.15.89:0/2284419611 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23cc0cb050 con 0x7f23cc0cf160 2023-10-30T12:28:58.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.496+0000 7f23cbfff640 1 --2- 172.21.15.89:0/2284419611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23cc0cf160 0x7f23cc0cf590 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f23cc0b30a0 tx=0x7f23c401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecd5b27ed0205ac8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:58.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.496+0000 7f23caffd640 1 -- 172.21.15.89:0/2284419611 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c40af400 con 0x7f23cc0cf160 2023-10-30T12:28:58.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.496+0000 7f23caffd640 1 -- 172.21.15.89:0/2284419611 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c40ab050 con 0x7f23cc0cf160 2023-10-30T12:28:58.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.496+0000 7f23caffd640 1 -- 172.21.15.89:0/2284419611 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c40af780 con 0x7f23cc0cf160 2023-10-30T12:28:58.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.496+0000 7f23d1f65640 1 -- 172.21.15.89:0/2284419611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23cc0cf160 msgr2=0x7f23cc0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.496+0000 7f23d1f65640 1 --2- 172.21.15.89:0/2284419611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23cc0cf160 0x7f23cc0cf590 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f23cc0b30a0 tx=0x7f23c401f1c0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.496+0000 7f23d1f65640 1 -- 172.21.15.89:0/2284419611 shutdown_connections 2023-10-30T12:28:58.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.496+0000 7f23d1f65640 1 --2- 172.21.15.89:0/2284419611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23cc0cf160 0x7f23cc0cf590 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.496+0000 7f23d1f65640 1 --2- 172.21.15.89:0/2284419611 >> v2:172.21.15.192:3300/0 conn(0x7f23cc15d6f0 0x7f23cc15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.496+0000 7f23d1f65640 1 -- 172.21.15.89:0/2284419611 >> 172.21.15.89:0/2284419611 conn(0x7f23cc06bfe0 msgr2=0x7f23cc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:58.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.496+0000 7f23d1f65640 1 -- 172.21.15.89:0/2284419611 shutdown_connections 2023-10-30T12:28:58.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.496+0000 7f23d1f65640 1 -- 172.21.15.89:0/2284419611 wait complete. 2023-10-30T12:28:58.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.500+0000 7f23d1f65640 1 Processor -- start 2023-10-30T12:28:58.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.500+0000 7f23d1f65640 1 -- start start 2023-10-30T12:28:58.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.500+0000 7f23d1f65640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23cc0cf160 0x7f23cc157f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.500+0000 7f23d1f65640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23cc1538a0 0x7f23cc1e3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.500+0000 7f23d1f65640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23cc15d6f0 0x7f23cc15b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.500+0000 7f23d1f65640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc158470 con 0x7f23cc0cf160 2023-10-30T12:28:58.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.500+0000 7f23d1f65640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc14dd50 con 0x7f23cc15d6f0 2023-10-30T12:28:58.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.500+0000 7f23d1f65640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc1e30a0 con 0x7f23cc1538a0 2023-10-30T12:28:58.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.500+0000 7f23cb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23cc0cf160 0x7f23cc157f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.500+0000 7f23cb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23cc0cf160 0x7f23cc157f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55406/0 (socket says 172.21.15.89:55406) 2023-10-30T12:28:58.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.500+0000 7f23cb7fe640 1 -- 172.21.15.89:0/233264660 learned_addr learned my addr 172.21.15.89:0/233264660 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:58.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.500+0000 7f23c3fff640 1 --2- 172.21.15.89:0/233264660 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23cc1538a0 0x7f23cc1e3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.500+0000 7f23cbfff640 1 --2- 172.21.15.89:0/233264660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23cc15d6f0 0x7f23cc15b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.500+0000 7f23cb7fe640 1 -- 172.21.15.89:0/233264660 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23cc1538a0 msgr2=0x7f23cc1e3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.500+0000 7f23cb7fe640 1 --2- 172.21.15.89:0/233264660 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23cc1538a0 0x7f23cc1e3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.500+0000 7f23cb7fe640 1 -- 172.21.15.89:0/233264660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23cc15d6f0 msgr2=0x7f23cc15b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.500+0000 7f23cb7fe640 1 --2- 172.21.15.89:0/233264660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23cc15d6f0 0x7f23cc15b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.500+0000 7f23cb7fe640 1 -- 172.21.15.89:0/233264660 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c4009b30 con 0x7f23cc0cf160 2023-10-30T12:28:58.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.500+0000 7f23cbfff640 1 --2- 172.21.15.89:0/233264660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23cc15d6f0 0x7f23cc15b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:28:58.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.500+0000 7f23cb7fe640 1 --2- 172.21.15.89:0/233264660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23cc0cf160 0x7f23cc157f30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f23bc00c490 tx=0x7f23bc00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:58.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.500+0000 7f23c8ff9640 1 -- 172.21.15.89:0/233264660 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23bc007480 con 0x7f23cc0cf160 2023-10-30T12:28:58.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.500+0000 7f23c8ff9640 1 -- 172.21.15.89:0/233264660 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23bc0075e0 con 0x7f23cc0cf160 2023-10-30T12:28:58.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.500+0000 7f23c8ff9640 1 -- 172.21.15.89:0/233264660 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23bc004a10 con 0x7f23cc0cf160 2023-10-30T12:28:58.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.500+0000 7f23d1f65640 1 -- 172.21.15.89:0/233264660 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23cc1f2ee0 con 0x7f23cc0cf160 2023-10-30T12:28:58.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.500+0000 7f23d1f65640 1 -- 172.21.15.89:0/233264660 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23cc1e2ae0 con 0x7f23cc0cf160 2023-10-30T12:28:58.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.500+0000 7f23d1f65640 1 -- 172.21.15.89:0/233264660 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23cc1e5890 con 0x7f23cc0cf160 2023-10-30T12:28:58.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.504+0000 7f23c8ff9640 1 -- 172.21.15.89:0/233264660 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f23bc004b70 con 0x7f23cc0cf160 2023-10-30T12:28:58.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.504+0000 7f23c8ff9640 1 --2- 172.21.15.89:0/233264660 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23ac0664d0 0x7f23ac068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.504+0000 7f23c8ff9640 1 -- 172.21.15.89:0/233264660 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f23bc082ff0 con 0x7f23cc0cf160 2023-10-30T12:28:58.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.504+0000 7f23c3fff640 1 --2- 172.21.15.89:0/233264660 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23ac0664d0 0x7f23ac068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.504+0000 7f23c3fff640 1 --2- 172.21.15.89:0/233264660 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23ac0664d0 0x7f23ac068990 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f23b4006fd0 tx=0x7f23b4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:58.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.508+0000 7f23c8ff9640 1 -- 172.21.15.89:0/233264660 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f23cc1e5890 con 0x7f23cc0cf160 2023-10-30T12:28:58.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.660+0000 7f23d1f65640 1 -- 172.21.15.89:0/233264660 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23cc1e5890 con 0x7f23cc0cf160 2023-10-30T12:28:58.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.660+0000 7f23c8ff9640 1 -- 172.21.15.89:0/233264660 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f23cc1e5890 con 0x7f23cc0cf160 2023-10-30T12:28:58.666 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:28:58.667 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":22,"fsid":"401641be-db6e-4c16-b3ac-fd84a86c4de4","created":"2023-10-30T12:28:26.198532+0000","modified":"2023-10-30T12:28:51.918285+0000","last_up_change":"2023-10-30T12:28:37.212205+0000","last_in_change":"2023-10-30T12:28:28.517734+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:39.004943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T12:28:48.935909+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T12:28:49.716098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72595c54-fd6e-4a5a-bb14-ff27e9377b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6801","nonce":1673955622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6803","nonce":1673955622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6807","nonce":1673955622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6805","nonce":1673955622}]},"public_addr":"172.21.15.89:6801/1673955622","cluster_addr":"172.21.15.89:6803/1673955622","heartbeat_back_addr":"172.21.15.89:6807/1673955622","heartbeat_front_addr":"172.21.15.89:6805/1673955622","state":["exists","up"]},{"osd":1,"uuid":"0305670b-d7c4-4193-bec2-8b7351ff590a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6817","nonce":3340572039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6819","nonce":3340572039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6823","nonce":3340572039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6821","nonce":3340572039}]},"public_addr":"172.21.15.89:6817/3340572039","cluster_addr":"172.21.15.89:6819/3340572039","heartbeat_back_addr":"172.21.15.89:6823/3340572039","heartbeat_front_addr":"172.21.15.89:6821/3340572039","state":["exists","up"]},{"osd":2,"uuid":"4bab5426-a3c3-46b0-91fc-546c254006c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6809","nonce":1929940799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6811","nonce":1929940799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6815","nonce":1929940799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6813","nonce":1929940799}]},"public_addr":"172.21.15.89:6809/1929940799","cluster_addr":"172.21.15.89:6811/1929940799","heartbeat_back_addr":"172.21.15.89:6815/1929940799","heartbeat_front_addr":"172.21.15.89:6813/1929940799","state":["exists","up"]},{"osd":3,"uuid":"7d47aea8-27cd-42be-8591-d63ad68c90e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6825","nonce":345087939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6827","nonce":345087939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6831","nonce":345087939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6829","nonce":345087939}]},"public_addr":"172.21.15.89:6825/345087939","cluster_addr":"172.21.15.89:6827/345087939","heartbeat_back_addr":"172.21.15.89:6831/345087939","heartbeat_front_addr":"172.21.15.89:6829/345087939","state":["exists","up"]},{"osd":4,"uuid":"2e2ae8f8-d097-4bba-9448-95047fbfacad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6801","nonce":2138125545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6803","nonce":2138125545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6809","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6811","nonce":2138125545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6805","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6807","nonce":2138125545}]},"public_addr":"172.21.15.192:6801/2138125545","cluster_addr":"172.21.15.192:6803/2138125545","heartbeat_back_addr":"172.21.15.192:6811/2138125545","heartbeat_front_addr":"172.21.15.192:6807/2138125545","state":["exists","up"]},{"osd":5,"uuid":"49be5ce5-12f4-4c26-8d39-864b6ff67313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6806","nonce":2360218671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6810","nonce":2360218671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6815","nonce":2360218671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6813","nonce":2360218671}]},"public_addr":"172.21.15.192:6806/2360218671","cluster_addr":"172.21.15.192:6810/2360218671","heartbeat_back_addr":"172.21.15.192:6815/2360218671","heartbeat_front_addr":"172.21.15.192:6813/2360218671","state":["exists","up"]},{"osd":6,"uuid":"48623452-d456-4ea2-a0de-5fa4b37e6d10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6825","nonce":3707671287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6827","nonce":3707671287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6831","nonce":3707671287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6829","nonce":3707671287}]},"public_addr":"172.21.15.192:6825/3707671287","cluster_addr":"172.21.15.192:6827/3707671287","heartbeat_back_addr":"172.21.15.192:6831/3707671287","heartbeat_front_addr":"172.21.15.192:6829/3707671287","state":["exists","up"]},{"osd":7,"uuid":"ff0a4384-54f8-43ec-a3ce-9c8ae3bf8a94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6817","nonce":3111945871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6819","nonce":3111945871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6823","nonce":3111945871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6821","nonce":3111945871}]},"public_addr":"172.21.15.192:6817/3111945871","cluster_addr":"172.21.15.192:6819/3111945871","heartbeat_back_addr":"172.21.15.192:6823/3111945871","heartbeat_front_addr":"172.21.15.192:6821/3111945871","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.183065+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.248650+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.194576+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.397483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.043648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.186624+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.159674+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.100851+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:28:58.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.664+0000 7f23d1f65640 1 -- 172.21.15.89:0/233264660 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23ac0664d0 msgr2=0x7f23ac068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.664+0000 7f23d1f65640 1 --2- 172.21.15.89:0/233264660 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23ac0664d0 0x7f23ac068990 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f23b4006fd0 tx=0x7f23b4008040 comp rx=0 tx=0).stop 2023-10-30T12:28:58.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.664+0000 7f23d1f65640 1 -- 172.21.15.89:0/233264660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23cc0cf160 msgr2=0x7f23cc157f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.664+0000 7f23d1f65640 1 --2- 172.21.15.89:0/233264660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23cc0cf160 0x7f23cc157f30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f23bc00c490 tx=0x7f23bc00c950 comp rx=0 tx=0).stop 2023-10-30T12:28:58.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.664+0000 7f23d1f65640 1 -- 172.21.15.89:0/233264660 shutdown_connections 2023-10-30T12:28:58.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.664+0000 7f23d1f65640 1 --2- 172.21.15.89:0/233264660 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23ac0664d0 0x7f23ac068990 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.664+0000 7f23d1f65640 1 --2- 172.21.15.89:0/233264660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23cc15d6f0 0x7f23cc15b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.664+0000 7f23d1f65640 1 --2- 172.21.15.89:0/233264660 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23cc1538a0 0x7f23cc1e3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.664+0000 7f23d1f65640 1 --2- 172.21.15.89:0/233264660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23cc0cf160 0x7f23cc157f30 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.664+0000 7f23d1f65640 1 -- 172.21.15.89:0/233264660 >> 172.21.15.89:0/233264660 conn(0x7f23cc06bfe0 msgr2=0x7f23cc15aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:58.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.664+0000 7f23d1f65640 1 -- 172.21.15.89:0/233264660 shutdown_connections 2023-10-30T12:28:58.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.664+0000 7f23d1f65640 1 -- 172.21.15.89:0/233264660 wait complete. 2023-10-30T12:28:58.687 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-30T12:28:58.687 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:28:58.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.816+0000 7face75ab640 1 Processor -- start 2023-10-30T12:28:58.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.816+0000 7face75ab640 1 -- start start 2023-10-30T12:28:58.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.820+0000 7face75ab640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7face00c6b80 0x7face00bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.820+0000 7face75ab640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7face00c6f60 0x7face00c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.820+0000 7face75ab640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7face00c8430 0x7face013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.820+0000 7face75ab640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7face00c0a50 con 0x7face00c6f60 2023-10-30T12:28:58.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.820+0000 7face75ab640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7face006f6e0 con 0x7face00c6b80 2023-10-30T12:28:58.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.820+0000 7face75ab640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face006f9c0 con 0x7face00c8430 2023-10-30T12:28:58.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.820+0000 7face4b46640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7face00c6f60 0x7face00c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.820+0000 7face4b46640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7face00c6f60 0x7face00c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55412/0 (socket says 172.21.15.89:55412) 2023-10-30T12:28:58.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.820+0000 7face4b46640 1 -- 172.21.15.89:0/1320354753 learned_addr learned my addr 172.21.15.89:0/1320354753 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:58.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.820+0000 7face5347640 1 --2- 172.21.15.89:0/1320354753 >> v2:172.21.15.192:3300/0 conn(0x7face00c6b80 0x7face00bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.820+0000 7face5b48640 1 --2- 172.21.15.89:0/1320354753 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7face00c8430 0x7face013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.820+0000 7face4b46640 1 -- 172.21.15.89:0/1320354753 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7face00c8430 msgr2=0x7face013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:58.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.820+0000 7face4b46640 1 --2- 172.21.15.89:0/1320354753 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7face00c8430 0x7face013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.820+0000 7face4b46640 1 -- 172.21.15.89:0/1320354753 >> v2:172.21.15.192:3300/0 conn(0x7face00c6b80 msgr2=0x7face00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:58.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.820+0000 7face4b46640 1 --2- 172.21.15.89:0/1320354753 >> v2:172.21.15.192:3300/0 conn(0x7face00c6b80 0x7face00bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.820+0000 7face4b46640 1 -- 172.21.15.89:0/1320354753 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face006fca0 con 0x7face00c6f60 2023-10-30T12:28:58.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.824+0000 7face4b46640 1 --2- 172.21.15.89:0/1320354753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7face00c6f60 0x7face00c0400 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7face006f1b0 tx=0x7facd001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57d223d1eb143162 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:58.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.824+0000 7facd7fff640 1 -- 172.21.15.89:0/1320354753 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd00ac630 con 0x7face00c6f60 2023-10-30T12:28:58.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.824+0000 7facd7fff640 1 -- 172.21.15.89:0/1320354753 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd00ac790 con 0x7face00c6f60 2023-10-30T12:28:58.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.824+0000 7facd7fff640 1 -- 172.21.15.89:0/1320354753 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd00acb20 con 0x7face00c6f60 2023-10-30T12:28:58.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.824+0000 7face75ab640 1 -- 172.21.15.89:0/1320354753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7face00c6f60 msgr2=0x7face00c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.824+0000 7face75ab640 1 --2- 172.21.15.89:0/1320354753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7face00c6f60 0x7face00c0400 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7face006f1b0 tx=0x7facd001c500 comp rx=0 tx=0).stop 2023-10-30T12:28:58.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.824+0000 7face75ab640 1 -- 172.21.15.89:0/1320354753 shutdown_connections 2023-10-30T12:28:58.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.824+0000 7face75ab640 1 --2- 172.21.15.89:0/1320354753 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7face00c8430 0x7face013ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.824+0000 7face75ab640 1 --2- 172.21.15.89:0/1320354753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7face00c6f60 0x7face00c0400 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.824+0000 7face75ab640 1 --2- 172.21.15.89:0/1320354753 >> v2:172.21.15.192:3300/0 conn(0x7face00c6b80 0x7face00bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.824+0000 7face75ab640 1 -- 172.21.15.89:0/1320354753 >> 172.21.15.89:0/1320354753 conn(0x7face00c7910 msgr2=0x7face00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:58.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.824+0000 7face75ab640 1 -- 172.21.15.89:0/1320354753 shutdown_connections 2023-10-30T12:28:58.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.824+0000 7face75ab640 1 -- 172.21.15.89:0/1320354753 wait complete. 2023-10-30T12:28:58.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.824+0000 7face75ab640 1 Processor -- start 2023-10-30T12:28:58.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.824+0000 7face75ab640 1 -- start start 2023-10-30T12:28:58.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.828+0000 7face75ab640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7face00c6b80 0x7face00c3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.828+0000 7face75ab640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7face00c6f60 0x7face013a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.828+0000 7face75ab640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7face00c8430 0x7face00c9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.828+0000 7face75ab640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face0155d90 con 0x7face00c8430 2023-10-30T12:28:58.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.828+0000 7face75ab640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face01395b0 con 0x7face00c6b80 2023-10-30T12:28:58.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.828+0000 7face75ab640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face0154ea0 con 0x7face00c6f60 2023-10-30T12:28:58.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.828+0000 7face5b48640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7face00c8430 0x7face00c9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.828+0000 7face5b48640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7face00c8430 0x7face00c9c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55414/0 (socket says 172.21.15.89:55414) 2023-10-30T12:28:58.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.828+0000 7face5b48640 1 -- 172.21.15.89:0/2208730261 learned_addr learned my addr 172.21.15.89:0/2208730261 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:58.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.828+0000 7face4b46640 1 --2- 172.21.15.89:0/2208730261 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7face00c6f60 0x7face013a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.828+0000 7face5347640 1 --2- 172.21.15.89:0/2208730261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7face00c6b80 0x7face00c3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.828+0000 7face5b48640 1 -- 172.21.15.89:0/2208730261 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7face00c6f60 msgr2=0x7face013a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.828+0000 7face5b48640 1 --2- 172.21.15.89:0/2208730261 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7face00c6f60 0x7face013a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.828+0000 7face5b48640 1 -- 172.21.15.89:0/2208730261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7face00c6b80 msgr2=0x7face00c3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.828+0000 7face5b48640 1 --2- 172.21.15.89:0/2208730261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7face00c6b80 0x7face00c3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.828+0000 7face5b48640 1 -- 172.21.15.89:0/2208730261 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd00094d0 con 0x7face00c8430 2023-10-30T12:28:58.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.828+0000 7face5347640 1 --2- 172.21.15.89:0/2208730261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7face00c6b80 0x7face00c3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:58.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.828+0000 7face5b48640 1 --2- 172.21.15.89:0/2208730261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7face00c8430 0x7face00c9c60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7facd8007e60 tx=0x7facd800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:58.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.828+0000 7facd5ffb640 1 -- 172.21.15.89:0/2208730261 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd8014420 con 0x7face00c8430 2023-10-30T12:28:58.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.828+0000 7facd5ffb640 1 -- 172.21.15.89:0/2208730261 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd80145a0 con 0x7face00c8430 2023-10-30T12:28:58.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.828+0000 7face75ab640 1 -- 172.21.15.89:0/2208730261 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face0155180 con 0x7face00c8430 2023-10-30T12:28:58.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.828+0000 7facd5ffb640 1 -- 172.21.15.89:0/2208730261 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd8014950 con 0x7face00c8430 2023-10-30T12:28:58.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.828+0000 7face75ab640 1 -- 172.21.15.89:0/2208730261 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face0155740 con 0x7face00c8430 2023-10-30T12:28:58.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.828+0000 7face75ab640 1 -- 172.21.15.89:0/2208730261 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7face00b0f40 con 0x7face00c8430 2023-10-30T12:28:58.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.828+0000 7facd5ffb640 1 -- 172.21.15.89:0/2208730261 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7facd8014ab0 con 0x7face00c8430 2023-10-30T12:28:58.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.832+0000 7facd5ffb640 1 --2- 172.21.15.89:0/2208730261 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7facc40664a0 0x7facc4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.832+0000 7facd5ffb640 1 -- 172.21.15.89:0/2208730261 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7facd8084720 con 0x7face00c8430 2023-10-30T12:28:58.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.832+0000 7face5347640 1 --2- 172.21.15.89:0/2208730261 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7facc40664a0 0x7facc4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.832+0000 7face5347640 1 --2- 172.21.15.89:0/2208730261 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7facc40664a0 0x7facc4068960 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7facdc004660 tx=0x7facdc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:58.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.836+0000 7facd5ffb640 1 -- 172.21.15.89:0/2208730261 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7facd8051ba0 con 0x7face00c8430 2023-10-30T12:28:58.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.988+0000 7face75ab640 1 -- 172.21.15.89:0/2208730261 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7face0155f40 con 0x7face00c8430 2023-10-30T12:28:58.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.988+0000 7facd5ffb640 1 -- 172.21.15.89:0/2208730261 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16505 (secure 0 0 0) 0x7face0155f40 con 0x7face00c8430 2023-10-30T12:28:58.995 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:28:58.995 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":22,"fsid":"401641be-db6e-4c16-b3ac-fd84a86c4de4","created":"2023-10-30T12:28:26.198532+0000","modified":"2023-10-30T12:28:51.918285+0000","last_up_change":"2023-10-30T12:28:37.212205+0000","last_in_change":"2023-10-30T12:28:28.517734+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:39.004943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T12:28:48.935909+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T12:28:49.716098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72595c54-fd6e-4a5a-bb14-ff27e9377b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6801","nonce":1673955622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6803","nonce":1673955622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6807","nonce":1673955622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6805","nonce":1673955622}]},"public_addr":"172.21.15.89:6801/1673955622","cluster_addr":"172.21.15.89:6803/1673955622","heartbeat_back_addr":"172.21.15.89:6807/1673955622","heartbeat_front_addr":"172.21.15.89:6805/1673955622","state":["exists","up"]},{"osd":1,"uuid":"0305670b-d7c4-4193-bec2-8b7351ff590a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6817","nonce":3340572039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6819","nonce":3340572039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6823","nonce":3340572039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6821","nonce":3340572039}]},"public_addr":"172.21.15.89:6817/3340572039","cluster_addr":"172.21.15.89:6819/3340572039","heartbeat_back_addr":"172.21.15.89:6823/3340572039","heartbeat_front_addr":"172.21.15.89:6821/3340572039","state":["exists","up"]},{"osd":2,"uuid":"4bab5426-a3c3-46b0-91fc-546c254006c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6809","nonce":1929940799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6811","nonce":1929940799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6815","nonce":1929940799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6813","nonce":1929940799}]},"public_addr":"172.21.15.89:6809/1929940799","cluster_addr":"172.21.15.89:6811/1929940799","heartbeat_back_addr":"172.21.15.89:6815/1929940799","heartbeat_front_addr":"172.21.15.89:6813/1929940799","state":["exists","up"]},{"osd":3,"uuid":"7d47aea8-27cd-42be-8591-d63ad68c90e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6825","nonce":345087939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6827","nonce":345087939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6831","nonce":345087939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6829","nonce":345087939}]},"public_addr":"172.21.15.89:6825/345087939","cluster_addr":"172.21.15.89:6827/345087939","heartbeat_back_addr":"172.21.15.89:6831/345087939","heartbeat_front_addr":"172.21.15.89:6829/345087939","state":["exists","up"]},{"osd":4,"uuid":"2e2ae8f8-d097-4bba-9448-95047fbfacad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6801","nonce":2138125545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6803","nonce":2138125545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6809","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6811","nonce":2138125545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6805","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6807","nonce":2138125545}]},"public_addr":"172.21.15.192:6801/2138125545","cluster_addr":"172.21.15.192:6803/2138125545","heartbeat_back_addr":"172.21.15.192:6811/2138125545","heartbeat_front_addr":"172.21.15.192:6807/2138125545","state":["exists","up"]},{"osd":5,"uuid":"49be5ce5-12f4-4c26-8d39-864b6ff67313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6806","nonce":2360218671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6810","nonce":2360218671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6815","nonce":2360218671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6813","nonce":2360218671}]},"public_addr":"172.21.15.192:6806/2360218671","cluster_addr":"172.21.15.192:6810/2360218671","heartbeat_back_addr":"172.21.15.192:6815/2360218671","heartbeat_front_addr":"172.21.15.192:6813/2360218671","state":["exists","up"]},{"osd":6,"uuid":"48623452-d456-4ea2-a0de-5fa4b37e6d10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6825","nonce":3707671287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6827","nonce":3707671287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6831","nonce":3707671287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6829","nonce":3707671287}]},"public_addr":"172.21.15.192:6825/3707671287","cluster_addr":"172.21.15.192:6827/3707671287","heartbeat_back_addr":"172.21.15.192:6831/3707671287","heartbeat_front_addr":"172.21.15.192:6829/3707671287","state":["exists","up"]},{"osd":7,"uuid":"ff0a4384-54f8-43ec-a3ce-9c8ae3bf8a94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6817","nonce":3111945871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6819","nonce":3111945871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6823","nonce":3111945871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6821","nonce":3111945871}]},"public_addr":"172.21.15.192:6817/3111945871","cluster_addr":"172.21.15.192:6819/3111945871","heartbeat_back_addr":"172.21.15.192:6823/3111945871","heartbeat_front_addr":"172.21.15.192:6821/3111945871","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.183065+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.248650+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.194576+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.397483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.043648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.186624+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.159674+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.100851+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:28:58.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.992+0000 7face75ab640 1 -- 172.21.15.89:0/2208730261 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7facc40664a0 msgr2=0x7facc4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.992+0000 7face75ab640 1 --2- 172.21.15.89:0/2208730261 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7facc40664a0 0x7facc4068960 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7facdc004660 tx=0x7facdc008040 comp rx=0 tx=0).stop 2023-10-30T12:28:58.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.992+0000 7face75ab640 1 -- 172.21.15.89:0/2208730261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7face00c8430 msgr2=0x7face00c9c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.992+0000 7face75ab640 1 --2- 172.21.15.89:0/2208730261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7face00c8430 0x7face00c9c60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7facd8007e60 tx=0x7facd800e4b0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.992+0000 7face75ab640 1 -- 172.21.15.89:0/2208730261 shutdown_connections 2023-10-30T12:28:58.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.992+0000 7face75ab640 1 --2- 172.21.15.89:0/2208730261 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7facc40664a0 0x7facc4068960 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.992+0000 7face75ab640 1 --2- 172.21.15.89:0/2208730261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7face00c8430 0x7face00c9c60 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.992+0000 7face75ab640 1 --2- 172.21.15.89:0/2208730261 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7face00c6f60 0x7face013a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.992+0000 7face75ab640 1 --2- 172.21.15.89:0/2208730261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7face00c6b80 0x7face00c3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.992+0000 7face75ab640 1 -- 172.21.15.89:0/2208730261 >> 172.21.15.89:0/2208730261 conn(0x7face00c7910 msgr2=0x7face00c0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:58.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.992+0000 7face75ab640 1 -- 172.21.15.89:0/2208730261 shutdown_connections 2023-10-30T12:28:58.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:58.992+0000 7face75ab640 1 -- 172.21.15.89:0/2208730261 wait complete. 2023-10-30T12:28:59.016 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-30T12:28:59.017 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-30T12:28:59.017 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-30T12:28:59.018 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-10-30T12:28:59.018 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-10-30T12:28:59.019 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-10-30T12:28:59.020 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-10-30T12:28:59.020 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-10-30T12:28:59.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.256+0000 7f8ad0c3a640 1 Processor -- start 2023-10-30T12:28:59.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.256+0000 7fd5a60dc640 1 Processor -- start 2023-10-30T12:28:59.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.256+0000 7f8ad0c3a640 1 -- start start 2023-10-30T12:28:59.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.256+0000 7f8ad0c3a640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8ac40dd410 0x7f8ac404adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.256+0000 7f8ad0c3a640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8ac404c690 0x7f8ac404b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.256+0000 7fd5a60dc640 1 -- start start 2023-10-30T12:28:59.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.260+0000 7f8acb7fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8ac40dd410 0x7f8ac404adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.260+0000 7f8acb7fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8ac40dd410 0x7f8ac404adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55426/0 (socket says 172.21.15.89:55426) 2023-10-30T12:28:59.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.260+0000 7f8acb7fe640 1 -- 172.21.15.89:0/447726739 learned_addr learned my addr 172.21.15.89:0/447726739 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:59.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.260+0000 7f8ad0c3a640 1 -- 172.21.15.89:0/447726739 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac4035d20 con 0x7f8ac40e3180 2023-10-30T12:28:59.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.260+0000 7f8ad0c3a640 1 -- 172.21.15.89:0/447726739 --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ac40e3d60 con 0x7f8ac40dd410 2023-10-30T12:28:59.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.260+0000 7f8ad0c3a640 1 -- 172.21.15.89:0/447726739 --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ac4045520 con 0x7f8ac404c690 2023-10-30T12:28:59.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.260+0000 7f8acaffd640 1 --2- 172.21.15.89:0/447726739 >> v2:172.21.15.192:3300/0 conn(0x7f8ac404c690 0x7f8ac404b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.260+0000 7fd5a60dc640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd598039d30 0x7fd598050760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.260+0000 7fd5a60dc640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd59804fff0 con 0x7fd5980e31d0 2023-10-30T12:28:59.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.260+0000 7fd5a60dc640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5980e3680 con 0x7fd598049560 2023-10-30T12:28:59.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.260+0000 7fd5a60dc640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd598050340 con 0x7fd598039d30 2023-10-30T12:28:59.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.260+0000 7f8acb7fe640 1 -- 172.21.15.89:0/447726739 >> v2:172.21.15.192:3300/0 conn(0x7f8ac404c690 msgr2=0x7f8ac404b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:59.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.260+0000 7fd5a50da640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd598039d30 0x7fd598050760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.260+0000 7f8acb7fe640 1 --2- 172.21.15.89:0/447726739 >> v2:172.21.15.192:3300/0 conn(0x7f8ac404c690 0x7f8ac404b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.260+0000 7f8acb7fe640 1 -- 172.21.15.89:0/447726739 >> v1:172.21.15.89:6789/0 conn(0x7f8ac40e3180 legacy=0x7f8ac40e35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.260+0000 7f8acb7fe640 1 -- 172.21.15.89:0/447726739 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac40e4020 con 0x7f8ac40dd410 2023-10-30T12:28:59.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.260+0000 7f8acaffd640 1 --2- 172.21.15.89:0/447726739 >> v2:172.21.15.192:3300/0 conn(0x7f8ac404c690 0x7f8ac404b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:59.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.260+0000 7fd5a48d9640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd5980e31d0 0x7fd5980d96f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55732/0 (socket says 172.21.15.89:55732) 2023-10-30T12:28:59.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.260+0000 7fd5a48d9640 1 -- 172.21.15.89:0/3107837158 learned_addr learned my addr 172.21.15.89:0/3107837158 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:28:59.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.264+0000 7fd597fff640 1 -- 172.21.15.89:0/3107837158 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569557409 0 0) 0x7fd59804fff0 con 0x7fd5980e31d0 2023-10-30T12:28:59.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.264+0000 7fd597fff640 1 -- 172.21.15.89:0/3107837158 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd58c003530 con 0x7fd5980e31d0 2023-10-30T12:28:59.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.264+0000 7fd597fff640 1 -- 172.21.15.89:0/3107837158 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 998402213 0 0) 0x7fd58c003530 con 0x7fd5980e31d0 2023-10-30T12:28:59.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.264+0000 7fd597fff640 1 -- 172.21.15.89:0/3107837158 >> v2:172.21.15.192:3300/0 conn(0x7fd598039d30 msgr2=0x7fd598050760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:59.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.264+0000 7fd597fff640 1 --2- 172.21.15.89:0/3107837158 >> v2:172.21.15.192:3300/0 conn(0x7fd598039d30 0x7fd598050760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.264+0000 7fd597fff640 1 -- 172.21.15.89:0/3107837158 >> v1:172.21.15.192:6789/0 conn(0x7fd598049560 legacy=0x7fd5980edb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.264+0000 7fd597fff640 1 -- 172.21.15.89:0/3107837158 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5980e3960 con 0x7fd5980e31d0 2023-10-30T12:28:59.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.264+0000 7fd597fff640 1 -- 172.21.15.89:0/3107837158 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7fd590001660 con 0x7fd5980e31d0 2023-10-30T12:28:59.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.264+0000 7fd597fff640 1 -- 172.21.15.89:0/3107837158 >> v1:172.21.15.89:6789/0 conn(0x7fd5980e31d0 legacy=0x7fd5980d96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.264+0000 7fd597fff640 1 --2- 172.21.15.89:0/3107837158 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd58c0039e0 0x7fd58c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.264+0000 7fd597fff640 1 --2- 172.21.15.89:0/3107837158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd58c0047d0 0x7fd58c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.264+0000 7fd597fff640 1 --2- 172.21.15.89:0/3107837158 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd58c005190 0x7fd58c00b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.264+0000 7fd597fff640 1 -- 172.21.15.89:0/3107837158 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd598050340 con 0x7fd58c0047d0 2023-10-30T12:28:59.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.264+0000 7fd597fff640 1 -- 172.21.15.89:0/3107837158 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58c00c030 con 0x7fd58c0039e0 2023-10-30T12:28:59.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.264+0000 7fd597fff640 1 -- 172.21.15.89:0/3107837158 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd58c00c1f0 con 0x7fd58c005190 2023-10-30T12:28:59.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.264+0000 7f0f94068640 1 Processor -- start 2023-10-30T12:28:59.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.264+0000 7fd5a48d9640 1 --2- 172.21.15.89:0/3107837158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd58c0047d0 0x7fd58c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.264+0000 7f0f94068640 1 -- start start 2023-10-30T12:28:59.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.264+0000 7fd5a48d9640 1 -- 172.21.15.89:0/3107837158 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd58c005190 msgr2=0x7fd58c00b900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:28:59.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.264+0000 7fd5a48d9640 1 --2- 172.21.15.89:0/3107837158 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd58c005190 0x7fd58c00b900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.264+0000 7fd5a48d9640 1 -- 172.21.15.89:0/3107837158 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd58c0039e0 msgr2=0x7fd58c003e20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-30T12:28:59.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.264+0000 7fd5a48d9640 1 --2- 172.21.15.89:0/3107837158 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd58c0039e0 0x7fd58c003e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.264+0000 7fd5a48d9640 1 -- 172.21.15.89:0/3107837158 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5980e3960 con 0x7fd58c0047d0 2023-10-30T12:28:59.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.268+0000 7f0f94068640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0f8c06ad90 0x7f0f8c152de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.268+0000 7f8acb7fe640 1 --2- 172.21.15.89:0/447726739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ac40dd410 0x7f8ac404adf0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8ac4047390 tx=0x7f8ac001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4219b45442b666ae server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.268+0000 7f0f94068640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0f8c0c8430 0x7f0f8c156280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.268+0000 7f0f91603640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0f8c0c8430 0x7f0f8c156280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.268+0000 7f0f91e04640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0f8c06ad90 0x7f0f8c152de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.268+0000 7f0f91603640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0f8c0c8430 0x7f0f8c156280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55446/0 (socket says 172.21.15.89:55446) 2023-10-30T12:28:59.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.268+0000 7f0f94068640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f8c1567c0 con 0x7f0f8c0c6670 2023-10-30T12:28:59.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.268+0000 7f0f94068640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f8c070820 con 0x7f0f8c0c8430 2023-10-30T12:28:59.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.268+0000 7f0f94068640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f8c0c7400 con 0x7f0f8c06ad90 2023-10-30T12:28:59.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.268+0000 7f0f91603640 1 -- 172.21.15.89:0/1113260968 learned_addr learned my addr 172.21.15.89:0/1113260968 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:59.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.268+0000 7f8aca7fc640 1 -- 172.21.15.89:0/447726739 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac00a89f0 con 0x7f8ac40dd410 2023-10-30T12:28:59.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.268+0000 7f8aca7fc640 1 -- 172.21.15.89:0/447726739 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ac00a8b50 con 0x7f8ac40dd410 2023-10-30T12:28:59.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.268+0000 7f0f91603640 1 -- 172.21.15.89:0/1113260968 >> v2:172.21.15.192:3300/0 conn(0x7f0f8c06ad90 msgr2=0x7f0f8c152de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:59.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.268+0000 7f0f91603640 1 --2- 172.21.15.89:0/1113260968 >> v2:172.21.15.192:3300/0 conn(0x7f0f8c06ad90 0x7f0f8c152de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.268+0000 7f0f90e02640 1 -- 172.21.15.89:0/1113260968 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1921784909 0 0) 0x7f0f8c1567c0 con 0x7f0f8c0c6670 2023-10-30T12:28:59.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.268+0000 7f8aca7fc640 1 -- 172.21.15.89:0/447726739 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac00b0440 con 0x7f8ac40dd410 2023-10-30T12:28:59.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.268+0000 7f0f91603640 1 -- 172.21.15.89:0/1113260968 >> v1:172.21.15.192:6789/0 conn(0x7f0f8c0c6670 legacy=0x7f0f8c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.268+0000 7f0f91603640 1 -- 172.21.15.89:0/1113260968 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f8c0c76c0 con 0x7f0f8c0c8430 2023-10-30T12:28:59.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.268+0000 7f0f91e04640 1 --2- 172.21.15.89:0/1113260968 >> v2:172.21.15.192:3300/0 conn(0x7f0f8c06ad90 0x7f0f8c152de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:59.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.268+0000 7fd5a58db640 1 -- 172.21.15.89:0/3107837158 reap_dead start 2023-10-30T12:28:59.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.268+0000 7f1d617c4640 1 Processor -- start 2023-10-30T12:28:59.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.268+0000 7f1d617c4640 1 -- start start 2023-10-30T12:28:59.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.268+0000 7f75a449e640 1 Processor -- start 2023-10-30T12:28:59.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.268+0000 7f75a449e640 1 -- start start 2023-10-30T12:28:59.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.268+0000 7fd5a48d9640 1 --2- 172.21.15.89:0/3107837158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd58c0047d0 0x7fd58c004c50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd59000c340 tx=0x7fd59001ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bbdf72bba665b6c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.268+0000 7fd597fff640 1 -- 172.21.15.89:0/3107837158 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5900a59b0 con 0x7fd58c0047d0 2023-10-30T12:28:59.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.268+0000 7fd597fff640 1 -- 172.21.15.89:0/3107837158 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5900a5b10 con 0x7fd58c0047d0 2023-10-30T12:28:59.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.268+0000 7fd597fff640 1 -- 172.21.15.89:0/3107837158 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5900a59b0 con 0x7fd58c0047d0 2023-10-30T12:28:59.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7f8ad0c3a640 1 -- 172.21.15.89:0/447726739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ac40dd410 msgr2=0x7f8ac404adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7f8ad0c3a640 1 --2- 172.21.15.89:0/447726739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ac40dd410 0x7f8ac404adf0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8ac4047390 tx=0x7f8ac001ecf0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7f8ad0c3a640 1 -- 172.21.15.89:0/447726739 shutdown_connections 2023-10-30T12:28:59.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7f8ad0c3a640 1 --2- 172.21.15.89:0/447726739 >> v2:172.21.15.192:3300/0 conn(0x7f8ac404c690 0x7f8ac404b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7f8ad0c3a640 1 --2- 172.21.15.89:0/447726739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ac40dd410 0x7f8ac404adf0 secure :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8ac4047390 tx=0x7f8ac001ecf0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7f8ad0c3a640 1 -- 172.21.15.89:0/447726739 >> 172.21.15.89:0/447726739 conn(0x7f8ac40502e0 msgr2=0x7f8ac40506f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:59.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7f8ad0c3a640 1 -- 172.21.15.89:0/447726739 shutdown_connections 2023-10-30T12:28:59.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7f8ad0c3a640 1 -- 172.21.15.89:0/447726739 wait complete. 2023-10-30T12:28:59.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7fd5a60dc640 1 -- 172.21.15.89:0/3107837158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd58c0047d0 msgr2=0x7fd58c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7fd5a60dc640 1 --2- 172.21.15.89:0/3107837158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd58c0047d0 0x7fd58c004c50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd59000c340 tx=0x7fd59001ccf0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7f1d617c4640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1d5c153810 0x7f1d5c15a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7f1d617c4640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d5c070840 0x7f1d5c070c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7f1d617c4640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c0c6880 con 0x7f1d5c069e90 2023-10-30T12:28:59.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7f1d617c4640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d5c15a820 con 0x7f1d5c153810 2023-10-30T12:28:59.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7f75a449e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f759c0bfec0 0x7f759c0c0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7f75a449e640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f759c0c04e0 con 0x7f759c1599d0 2023-10-30T12:28:59.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7f75a449e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f759c0c07c0 con 0x7f759c06f540 2023-10-30T12:28:59.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7f75a449e640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f759c1595b0 con 0x7f759c0bfec0 2023-10-30T12:28:59.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7fd5a60dc640 1 -- 172.21.15.89:0/3107837158 shutdown_connections 2023-10-30T12:28:59.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7fd5a60dc640 1 --2- 172.21.15.89:0/3107837158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd58c0047d0 0x7fd58c004c50 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7fd5a60dc640 1 -- 172.21.15.89:0/3107837158 >> 172.21.15.89:0/3107837158 conn(0x7fd5980062c0 msgr2=0x7fd5980066d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:59.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7f75a223a640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f759c06f540 0x7f759c152c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:46398/0 (socket says 172.21.15.89:46398) 2023-10-30T12:28:59.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7f75a223a640 1 -- 172.21.15.89:0/4000226710 learned_addr learned my addr 172.21.15.89:0/4000226710 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:28:59.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7f1d617c4640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5c0711c0 con 0x7f1d5c070840 2023-10-30T12:28:59.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7f1d5b7fe640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f1d5c069e90 0x7f1d5c06a300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:46384/0 (socket says 172.21.15.89:46384) 2023-10-30T12:28:59.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7f1d5b7fe640 1 -- 172.21.15.89:0/1954267183 learned_addr learned my addr 172.21.15.89:0/1954267183 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:28:59.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7f1d5a7fc640 1 --2- 172.21.15.89:0/1954267183 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d5c070840 0x7f1d5c070c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7f1d5affd640 1 --2- 172.21.15.89:0/1954267183 >> v2:172.21.15.89:3300/0 conn(0x7f1d5c153810 0x7f1d5c15a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7f1d5a7fc640 1 -- 172.21.15.89:0/1954267183 >> v2:172.21.15.89:3300/0 conn(0x7f1d5c153810 msgr2=0x7f1d5c15a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:59.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7f1d5a7fc640 1 --2- 172.21.15.89:0/1954267183 >> v2:172.21.15.89:3300/0 conn(0x7f1d5c153810 0x7f1d5c15a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7f1d5a7fc640 1 -- 172.21.15.89:0/1954267183 >> v1:172.21.15.192:6789/0 conn(0x7f1d5c069e90 legacy=0x7f1d5c06a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7f1d5a7fc640 1 -- 172.21.15.89:0/1954267183 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d5c0c5990 con 0x7f1d5c070840 2023-10-30T12:28:59.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7f75a1238640 1 -- 172.21.15.89:0/4000226710 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1467347539 0 0) 0x7f759c0c07c0 con 0x7f759c06f540 2023-10-30T12:28:59.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7f75a1238640 1 -- 172.21.15.89:0/4000226710 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7588003530 con 0x7f759c06f540 2023-10-30T12:28:59.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.272+0000 7f75a2a3b640 1 --2- 172.21.15.89:0/4000226710 >> v2:172.21.15.89:3300/0 conn(0x7f759c0bfec0 0x7f759c0c0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f75a1238640 1 -- 172.21.15.89:0/4000226710 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2856424091 0 0) 0x7f7588003530 con 0x7f759c06f540 2023-10-30T12:28:59.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f75a1238640 1 -- 172.21.15.89:0/4000226710 >> v2:172.21.15.89:3300/0 conn(0x7f759c0bfec0 msgr2=0x7f759c0c0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:59.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f75a1238640 1 --2- 172.21.15.89:0/4000226710 >> v2:172.21.15.89:3300/0 conn(0x7f759c0bfec0 0x7f759c0c0ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7fd5a60dc640 1 -- 172.21.15.89:0/3107837158 shutdown_connections 2023-10-30T12:28:59.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7fd5a60dc640 1 -- 172.21.15.89:0/3107837158 wait complete. 2023-10-30T12:28:59.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f75a1238640 1 -- 172.21.15.89:0/4000226710 >> v1:172.21.15.89:6789/0 conn(0x7f759c1599d0 legacy=0x7f759c0c6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:59.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f75a1238640 1 -- 172.21.15.89:0/4000226710 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759c158ef0 con 0x7f759c06f540 2023-10-30T12:28:59.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f75a1238640 1 -- 172.21.15.89:0/4000226710 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f758c001660 con 0x7f759c06f540 2023-10-30T12:28:59.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f75a1238640 1 -- 172.21.15.89:0/4000226710 >> v1:172.21.15.192:6789/0 conn(0x7f759c06f540 legacy=0x7f759c152c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f75a1238640 1 --2- 172.21.15.89:0/4000226710 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7588003980 0x7f7588003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f75a1238640 1 --2- 172.21.15.89:0/4000226710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75880047d0 0x7f7588004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f75a1238640 1 --2- 172.21.15.89:0/4000226710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75880052b0 0x7f758800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f8ad0c3a640 1 Processor -- start 2023-10-30T12:28:59.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f75a1238640 1 -- 172.21.15.89:0/4000226710 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c1595b0 con 0x7f75880047d0 2023-10-30T12:28:59.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f75a1238640 1 -- 172.21.15.89:0/4000226710 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758800c1f0 con 0x7f75880052b0 2023-10-30T12:28:59.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f75a1238640 1 -- 172.21.15.89:0/4000226710 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f758800c3b0 con 0x7f7588003980 2023-10-30T12:28:59.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f75a2a3b640 1 --2- 172.21.15.89:0/4000226710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75880052b0 0x7f758800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f8ad0c3a640 1 -- start start 2023-10-30T12:28:59.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f0f91603640 1 --2- 172.21.15.89:0/1113260968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f8c0c8430 0x7f0f8c156280 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c06c750 tx=0x7f0f7c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49f1aea68c3d0213 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f75a223a640 1 --2- 172.21.15.89:0/4000226710 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7588003980 0x7f7588003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f75a1a39640 1 --2- 172.21.15.89:0/4000226710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75880047d0 0x7f7588004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f75a2a3b640 1 -- 172.21.15.89:0/4000226710 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7588003980 msgr2=0x7f7588003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:59.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f75a2a3b640 1 --2- 172.21.15.89:0/4000226710 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7588003980 0x7f7588003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f75a2a3b640 1 -- 172.21.15.89:0/4000226710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75880047d0 msgr2=0x7f7588004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:59.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f75a2a3b640 1 --2- 172.21.15.89:0/4000226710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75880047d0 0x7f7588004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f75a2a3b640 1 -- 172.21.15.89:0/4000226710 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759c158ef0 con 0x7f75880052b0 2023-10-30T12:28:59.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f75a1a39640 1 --2- 172.21.15.89:0/4000226710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75880047d0 0x7f7588004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:28:59.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7fd5a60dc640 1 Processor -- start 2023-10-30T12:28:59.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f8ad0c3a640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ac404c690 0x7f8ac41796c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f8ad0c3a640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ac40e3180 0x7f8ac40d2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f8ad0c3a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ac40d9170 0x7f8ac4047660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f8ad0c3a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac4048790 con 0x7f8ac40d9170 2023-10-30T12:28:59.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f8ad0c3a640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac40d75e0 con 0x7f8ac40e3180 2023-10-30T12:28:59.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f8ad0c3a640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac40d78c0 con 0x7f8ac404c690 2023-10-30T12:28:59.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f8acb7fe640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ac404c690 0x7f8ac41796c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f8acaffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ac40e3180 0x7f8ac40d2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f8acb7fe640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ac404c690 0x7f8ac41796c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:37636/0 (socket says 172.21.15.89:37636) 2023-10-30T12:28:59.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f8acb7fe640 1 -- 172.21.15.89:0/2409092753 learned_addr learned my addr 172.21.15.89:0/2409092753 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:59.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f0b9886a640 1 Processor -- start 2023-10-30T12:28:59.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f0f90e02640 1 -- 172.21.15.89:0/1113260968 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c0ab610 con 0x7f0f8c0c8430 2023-10-30T12:28:59.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7fd5a60dc640 1 -- start start 2023-10-30T12:28:59.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f0f90e02640 1 -- 172.21.15.89:0/1113260968 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f7c0ab770 con 0x7f0f8c0c8430 2023-10-30T12:28:59.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f0f90e02640 1 -- 172.21.15.89:0/1113260968 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c0abb00 con 0x7f0f8c0c8430 2023-10-30T12:28:59.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f0f94068640 1 -- 172.21.15.89:0/1113260968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f8c0c8430 msgr2=0x7f0f8c156280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f0f94068640 1 --2- 172.21.15.89:0/1113260968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f8c0c8430 0x7f0f8c156280 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c06c750 tx=0x7f0f7c01c500 comp rx=0 tx=0).stop 2023-10-30T12:28:59.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f8acbfff640 1 --2- 172.21.15.89:0/2409092753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ac40d9170 0x7f8ac4047660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f0f94068640 1 -- 172.21.15.89:0/1113260968 shutdown_connections 2023-10-30T12:28:59.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f0f94068640 1 --2- 172.21.15.89:0/1113260968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f8c0c8430 0x7f0f8c156280 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f0f94068640 1 --2- 172.21.15.89:0/1113260968 >> v2:172.21.15.192:3300/0 conn(0x7f0f8c06ad90 0x7f0f8c152de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f0f94068640 1 -- 172.21.15.89:0/1113260968 >> 172.21.15.89:0/1113260968 conn(0x7f0f8c0c7910 msgr2=0x7f0f8c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:59.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f8acb7fe640 1 -- 172.21.15.89:0/2409092753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ac40e3180 msgr2=0x7f8ac40d2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f8acb7fe640 1 --2- 172.21.15.89:0/2409092753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ac40e3180 0x7f8ac40d2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f8acb7fe640 1 -- 172.21.15.89:0/2409092753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ac40d9170 msgr2=0x7f8ac4047660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f8acb7fe640 1 --2- 172.21.15.89:0/2409092753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ac40d9170 0x7f8ac4047660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f8acb7fe640 1 -- 172.21.15.89:0/2409092753 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac00094d0 con 0x7f8ac404c690 2023-10-30T12:28:59.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f1d5a7fc640 1 --2- 172.21.15.89:0/1954267183 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d5c070840 0x7f1d5c070c80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1d440097c0 tx=0x7f1d4401ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46caa2fd27fbe358 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f8acbfff640 1 --2- 172.21.15.89:0/2409092753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ac40d9170 0x7f8ac4047660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:28:59.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f0f94068640 1 -- 172.21.15.89:0/1113260968 shutdown_connections 2023-10-30T12:28:59.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f0f94068640 1 -- 172.21.15.89:0/1113260968 wait complete. 2023-10-30T12:28:59.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.276+0000 7f0b9886a640 1 -- start start 2023-10-30T12:28:59.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f8acb7fe640 1 --2- 172.21.15.89:0/2409092753 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ac404c690 0x7f8ac41796c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8ac001c920 tx=0x7f8ac001ba90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f1d59ffb640 1 -- 172.21.15.89:0/1954267183 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d440a8970 con 0x7f1d5c070840 2023-10-30T12:28:59.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f8ab7fff640 1 -- 172.21.15.89:0/2409092753 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac00b0e80 con 0x7f8ac404c690 2023-10-30T12:28:59.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f8ab7fff640 1 -- 172.21.15.89:0/2409092753 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ac00bb480 con 0x7f8ac404c690 2023-10-30T12:28:59.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f8ab7fff640 1 -- 172.21.15.89:0/2409092753 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac00bb810 con 0x7f8ac404c690 2023-10-30T12:28:59.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7fd5a60dc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5980cb1c0 0x7fd5980cb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7fd5a60dc640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd598053850 0x7fd5980d80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7fd5a60dc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd598053f20 0x7fd5980d0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7fd5a60dc640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59817c2d0 con 0x7fd598053f20 2023-10-30T12:28:59.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7fd5a60dc640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59817b970 con 0x7fd5980cb1c0 2023-10-30T12:28:59.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7fd5a60dc640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd59817f960 con 0x7fd598053850 2023-10-30T12:28:59.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7fd5a58db640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd598053f20 0x7fd5980d0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7fd5a58db640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd598053f20 0x7fd5980d0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55492/0 (socket says 172.21.15.89:55492) 2023-10-30T12:28:59.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7fd5a58db640 1 -- 172.21.15.89:0/1168731836 learned_addr learned my addr 172.21.15.89:0/1168731836 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:59.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7fd5a48d9640 1 --2- 172.21.15.89:0/1168731836 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd598053850 0x7fd5980d80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f1d59ffb640 1 -- 172.21.15.89:0/1954267183 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d440a8ad0 con 0x7f1d5c070840 2023-10-30T12:28:59.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f1d59ffb640 1 -- 172.21.15.89:0/1954267183 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d440a8e60 con 0x7f1d5c070840 2023-10-30T12:28:59.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f0f94068640 1 Processor -- start 2023-10-30T12:28:59.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f8ad0c3a640 1 -- 172.21.15.89:0/2409092753 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ac40d7f50 con 0x7f8ac404c690 2023-10-30T12:28:59.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f8ad0c3a640 1 -- 172.21.15.89:0/2409092753 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ac40d7ba0 con 0x7f8ac404c690 2023-10-30T12:28:59.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f8ad0c3a640 1 -- 172.21.15.89:0/2409092753 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8ac4045ac0 con 0x7f8ac404c690 2023-10-30T12:28:59.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f0b9886a640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b94157c80 0x7f0b94070fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f0b9886a640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0b94069e90 0x7f0b940c6890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f0b9886a640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b9406a2e0 con 0x7f0b9406f340 2023-10-30T12:28:59.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f0b9886a640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b940702a0 con 0x7f0b94069e90 2023-10-30T12:28:59.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f0b9886a640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b940c8390 con 0x7f0b94157c80 2023-10-30T12:28:59.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7fd5a58db640 1 -- 172.21.15.89:0/1168731836 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd598053850 msgr2=0x7fd5980d80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7fd5a58db640 1 --2- 172.21.15.89:0/1168731836 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd598053850 0x7fd5980d80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7fd5a58db640 1 -- 172.21.15.89:0/1168731836 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5980cb1c0 msgr2=0x7fd5980cb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7fd5a58db640 1 --2- 172.21.15.89:0/1168731836 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5980cb1c0 0x7fd5980cb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7fd5a58db640 1 -- 172.21.15.89:0/1168731836 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd59817bc10 con 0x7fd598053f20 2023-10-30T12:28:59.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f75a2a3b640 1 -- 172.21.15.89:0/4000226710 reap_dead start 2023-10-30T12:28:59.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f0b9259c640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f0b9406f340 0x7f0b94161000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:46406/0 (socket says 172.21.15.89:46406) 2023-10-30T12:28:59.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f0b9259c640 1 -- 172.21.15.89:0/3289629890 learned_addr learned my addr 172.21.15.89:0/3289629890 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:28:59.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7fd5a58db640 1 --2- 172.21.15.89:0/1168731836 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd598053f20 0x7fd5980d0c90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0063ef0 tx=0x7fd5a0066e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.326 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f0f94068640 1 -- start start 2023-10-30T12:28:59.326 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f0b9359e640 1 --2- 172.21.15.89:0/3289629890 >> v2:172.21.15.192:3300/0 conn(0x7f0b94069e90 0x7f0b940c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.326 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.280+0000 7f0b92d9d640 1 --2- 172.21.15.89:0/3289629890 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b94157c80 0x7f0b94070fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f1d617c4640 1 -- 172.21.15.89:0/1954267183 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d5c070840 msgr2=0x7f1d5c070c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f1d617c4640 1 --2- 172.21.15.89:0/1954267183 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d5c070840 0x7f1d5c070c80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1d440097c0 tx=0x7f1d4401ec70 comp rx=0 tx=0).stop 2023-10-30T12:28:59.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7fd595ffb640 1 -- 172.21.15.89:0/1168731836 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a0072770 con 0x7fd598053f20 2023-10-30T12:28:59.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7fd595ffb640 1 -- 172.21.15.89:0/1168731836 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a0069e90 con 0x7fd598053f20 2023-10-30T12:28:59.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7fd595ffb640 1 -- 172.21.15.89:0/1168731836 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a0072b50 con 0x7fd598053f20 2023-10-30T12:28:59.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f0b91d9b640 1 -- 172.21.15.89:0/3289629890 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1718247320 0 0) 0x7f0b7c001120 con 0x7f0b9406f340 2023-10-30T12:28:59.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f0b91d9b640 1 -- 172.21.15.89:0/3289629890 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b80003530 con 0x7f0b9406f340 2023-10-30T12:28:59.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f75a2a3b640 1 --2- 172.21.15.89:0/4000226710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75880052b0 0x7f758800b990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7594007f10 tx=0x7f759401ce10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20b601d4178ca822 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f0f94068640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f8c06ad90 0x7f0f8c140550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f0f94068640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f8c0c6670 0x7f0f8c0c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f0f94068640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f8c0c8430 0x7f0f8c0c9c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f0f94068640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c0c9260 con 0x7f0f8c0c6670 2023-10-30T12:28:59.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f0f94068640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c13c000 con 0x7f0f8c0c8430 2023-10-30T12:28:59.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f0f94068640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c137620 con 0x7f0f8c06ad90 2023-10-30T12:28:59.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f0b9359e640 1 -- 172.21.15.89:0/3289629890 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b94157c80 msgr2=0x7f0b94070fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:59.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f0b9359e640 1 --2- 172.21.15.89:0/3289629890 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b94157c80 0x7f0b94070fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f0b9359e640 1 -- 172.21.15.89:0/3289629890 >> v1:172.21.15.192:6789/0 conn(0x7f0b9406f340 legacy=0x7f0b94161000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f0b9359e640 1 -- 172.21.15.89:0/3289629890 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b94161ab0 con 0x7f0b94069e90 2023-10-30T12:28:59.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f1d617c4640 1 -- 172.21.15.89:0/1954267183 shutdown_connections 2023-10-30T12:28:59.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f1d617c4640 1 --2- 172.21.15.89:0/1954267183 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d5c070840 0x7f1d5c070c80 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f1d617c4640 1 --2- 172.21.15.89:0/1954267183 >> v2:172.21.15.89:3300/0 conn(0x7f1d5c153810 0x7f1d5c15a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f1d617c4640 1 -- 172.21.15.89:0/1954267183 >> 172.21.15.89:0/1954267183 conn(0x7f1d5c0c7910 msgr2=0x7f1d5c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:59.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f0f92605640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f8c0c8430 0x7f0f8c0c9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f0f92605640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f8c0c8430 0x7f0f8c0c9c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:39030/0 (socket says 172.21.15.89:39030) 2023-10-30T12:28:59.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f0f92605640 1 -- 172.21.15.89:0/169038631 learned_addr learned my addr 172.21.15.89:0/169038631 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:59.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f1d617c4640 1 -- 172.21.15.89:0/1954267183 shutdown_connections 2023-10-30T12:28:59.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f0f91603640 1 --2- 172.21.15.89:0/169038631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f8c0c6670 0x7f0f8c0c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7fd5a60dc640 1 -- 172.21.15.89:0/1168731836 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd59817bef0 con 0x7fd598053f20 2023-10-30T12:28:59.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7fd5a60dc640 1 -- 172.21.15.89:0/1168731836 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd59817fc40 con 0x7fd598053f20 2023-10-30T12:28:59.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7fd5a60dc640 1 -- 172.21.15.89:0/1168731836 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd598009910 con 0x7fd598053f20 2023-10-30T12:28:59.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f0f91e04640 1 --2- 172.21.15.89:0/169038631 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f8c06ad90 0x7f0f8c140550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f0f92605640 1 -- 172.21.15.89:0/169038631 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f8c06ad90 msgr2=0x7f0f8c140550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f1d617c4640 1 -- 172.21.15.89:0/1954267183 wait complete. 2023-10-30T12:28:59.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f75a1238640 1 -- 172.21.15.89:0/4000226710 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75940a5e20 con 0x7f75880052b0 2023-10-30T12:28:59.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f75a1238640 1 -- 172.21.15.89:0/4000226710 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75940ad440 con 0x7f75880052b0 2023-10-30T12:28:59.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f75a1238640 1 -- 172.21.15.89:0/4000226710 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75940a5e20 con 0x7f75880052b0 2023-10-30T12:28:59.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f0f92605640 1 --2- 172.21.15.89:0/169038631 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f8c06ad90 0x7f0f8c140550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f0f92605640 1 -- 172.21.15.89:0/169038631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f8c0c6670 msgr2=0x7f0f8c0c96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f0f92605640 1 --2- 172.21.15.89:0/169038631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f8c0c6670 0x7f0f8c0c96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f0f92605640 1 -- 172.21.15.89:0/169038631 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f88009420 con 0x7f0f8c0c8430 2023-10-30T12:28:59.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f623f474640 1 Processor -- start 2023-10-30T12:28:59.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f0f91e04640 1 --2- 172.21.15.89:0/169038631 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f8c06ad90 0x7f0f8c140550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:59.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f0f91603640 1 --2- 172.21.15.89:0/169038631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f8c0c6670 0x7f0f8c0c96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:59.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f623f474640 1 -- start start 2023-10-30T12:28:59.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f75a449e640 1 -- 172.21.15.89:0/4000226710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75880052b0 msgr2=0x7f758800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f75a449e640 1 --2- 172.21.15.89:0/4000226710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75880052b0 0x7f758800b990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7594007f10 tx=0x7f759401ce10 comp rx=0 tx=0).stop 2023-10-30T12:28:59.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f8ab7fff640 1 -- 172.21.15.89:0/2409092753 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8ac00bb970 con 0x7f8ac404c690 2023-10-30T12:28:59.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f0f92605640 1 --2- 172.21.15.89:0/169038631 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f8c0c8430 0x7f0f8c0c9c30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0f8400c780 tx=0x7f0f8400cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f75a449e640 1 -- 172.21.15.89:0/4000226710 shutdown_connections 2023-10-30T12:28:59.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f75a449e640 1 --2- 172.21.15.89:0/4000226710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f75880052b0 0x7f758800b990 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.284+0000 7f75a449e640 1 -- 172.21.15.89:0/4000226710 >> 172.21.15.89:0/4000226710 conn(0x7f759c0c7910 msgr2=0x7f759c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:59.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f0f827fc640 1 -- 172.21.15.89:0/169038631 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f84003db0 con 0x7f0f8c0c8430 2023-10-30T12:28:59.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f75a449e640 1 -- 172.21.15.89:0/4000226710 shutdown_connections 2023-10-30T12:28:59.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f1d617c4640 1 Processor -- start 2023-10-30T12:28:59.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f75a449e640 1 -- 172.21.15.89:0/4000226710 wait complete. 2023-10-30T12:28:59.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f8ab7fff640 1 --2- 172.21.15.89:0/2409092753 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8ab00665a0 0x7f8ab0068a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f0f94068640 1 -- 172.21.15.89:0/169038631 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f7c0094d0 con 0x7f0f8c0c8430 2023-10-30T12:28:59.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f1d617c4640 1 -- start start 2023-10-30T12:28:59.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f0f827fc640 1 -- 172.21.15.89:0/169038631 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f8400ce90 con 0x7f0f8c0c8430 2023-10-30T12:28:59.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f0f827fc640 1 -- 172.21.15.89:0/169038631 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f84003db0 con 0x7f0f8c0c8430 2023-10-30T12:28:59.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f0f94068640 1 -- 172.21.15.89:0/169038631 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f8c13c580 con 0x7f0f8c0c8430 2023-10-30T12:28:59.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f0b9359e640 1 --2- 172.21.15.89:0/3289629890 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b94069e90 0x7f0b940c6890 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c009d90 tx=0x7f0b8c01c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9ae2d5acd927f8d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f623f474640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f62380c80f0 0x7f62380c2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f623f474640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6238152eb0 0x7f62380713a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f8ab7fff640 1 -- 172.21.15.89:0/2409092753 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8ac0122cc0 con 0x7f8ac404c690 2023-10-30T12:28:59.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f8ab7fff640 1 --2- 172.21.15.89:0/2409092753 >> [v2:172.21.15.89:6800/1673955622,v1:172.21.15.89:6801/1673955622] conn(0x7f8ab006e7c0 0x7f8ab0070be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f8ab7fff640 1 -- 172.21.15.89:0/2409092753 --> [v2:172.21.15.89:6800/1673955622,v1:172.21.15.89:6801/1673955622] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8ab0071290 con 0x7f8ab006e7c0 2023-10-30T12:28:59.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f8ab7fff640 1 -- 172.21.15.89:0/2409092753 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ac0123070 con 0x7f8ac404c690 2023-10-30T12:28:59.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f8acaffd640 1 --2- 172.21.15.89:0/2409092753 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8ab00665a0 0x7f8ab0068a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7fd595ffb640 1 -- 172.21.15.89:0/1168731836 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd5a0072cb0 con 0x7fd598053f20 2023-10-30T12:28:59.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f623f474640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6238155e30 con 0x7f62380700c0 2023-10-30T12:28:59.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f623f474640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6238156670 con 0x7f62380c80f0 2023-10-30T12:28:59.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f623f474640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62381560b0 con 0x7f6238152eb0 2023-10-30T12:28:59.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f0f94068640 1 -- 172.21.15.89:0/169038631 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0f8c0c6150 con 0x7f0f8c0c8430 2023-10-30T12:28:59.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f8acbfff640 1 --2- 172.21.15.89:0/2409092753 >> [v2:172.21.15.89:6800/1673955622,v1:172.21.15.89:6801/1673955622] conn(0x7f8ab006e7c0 0x7f8ab0070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f1d617c4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d5c069e90 0x7f1d5c0c2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f1d617c4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d5c070840 0x7f1d5c158c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f1d617c4640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d5c153810 0x7f1d5c1599d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f1d617c4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5c1587d0 con 0x7f1d5c069e90 2023-10-30T12:28:59.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f1d617c4640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5c0b07d0 con 0x7f1d5c070840 2023-10-30T12:28:59.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f1d617c4640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5c0b0ab0 con 0x7f1d5c153810 2023-10-30T12:28:59.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f0b91d9b640 1 -- 172.21.15.89:0/3289629890 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c0a5920 con 0x7f0b94069e90 2023-10-30T12:28:59.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f0b91d9b640 1 -- 172.21.15.89:0/3289629890 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b8c0a5a80 con 0x7f0b94069e90 2023-10-30T12:28:59.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f0b91d9b640 1 -- 172.21.15.89:0/3289629890 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c0a5e10 con 0x7f0b94069e90 2023-10-30T12:28:59.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f623dc71640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6238152eb0 0x7f62380713a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f623dc71640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6238152eb0 0x7f62380713a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:37678/0 (socket says 172.21.15.89:37678) 2023-10-30T12:28:59.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f623dc71640 1 -- 172.21.15.89:0/527888041 learned_addr learned my addr 172.21.15.89:0/527888041 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:59.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f623e472640 1 --2- 172.21.15.89:0/527888041 >> v2:172.21.15.192:3300/0 conn(0x7f62380c80f0 0x7f62380c2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f1d5b7fe640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d5c153810 0x7f1d5c1599d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f1d5b7fe640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d5c153810 0x7f1d5c1599d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:37690/0 (socket says 172.21.15.89:37690) 2023-10-30T12:28:59.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f1d5b7fe640 1 -- 172.21.15.89:0/1953870259 learned_addr learned my addr 172.21.15.89:0/1953870259 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:59.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f1d5affd640 1 --2- 172.21.15.89:0/1953870259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d5c069e90 0x7f1d5c0c2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f623dc71640 1 -- 172.21.15.89:0/527888041 >> v2:172.21.15.192:3300/0 conn(0x7f62380c80f0 msgr2=0x7f62380c2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:59.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f623dc71640 1 --2- 172.21.15.89:0/527888041 >> v2:172.21.15.192:3300/0 conn(0x7f62380c80f0 0x7f62380c2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f623dc71640 1 -- 172.21.15.89:0/527888041 >> v1:172.21.15.192:6789/0 conn(0x7f62380700c0 legacy=0x7f6238155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f623dc71640 1 -- 172.21.15.89:0/527888041 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6238156390 con 0x7f6238152eb0 2023-10-30T12:28:59.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f623e472640 1 --2- 172.21.15.89:0/527888041 >> v2:172.21.15.192:3300/0 conn(0x7f62380c80f0 0x7f62380c2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:59.357 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7fd595ffb640 1 --2- 172.21.15.89:0/1168731836 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd5840664f0 0x7fd5840689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.357 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f0b9886a640 1 -- 172.21.15.89:0/3289629890 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b94069e90 msgr2=0x7f0b940c6890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.357 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f0b9886a640 1 --2- 172.21.15.89:0/3289629890 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b94069e90 0x7f0b940c6890 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c009d90 tx=0x7f0b8c01c950 comp rx=0 tx=0).stop 2023-10-30T12:28:59.357 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f6fff054640 1 Processor -- start 2023-10-30T12:28:59.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7fd5a50da640 1 --2- 172.21.15.89:0/1168731836 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd5840664f0 0x7fd5840689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f8acbfff640 1 --2- 172.21.15.89:0/2409092753 >> [v2:172.21.15.89:6800/1673955622,v1:172.21.15.89:6801/1673955622] conn(0x7f8ab006e7c0 0x7f8ab0070be0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f8acaffd640 1 --2- 172.21.15.89:0/2409092753 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8ab00665a0 0x7f8ab0068a60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8ac416d560 tx=0x7f8ab8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7fd595ffb640 1 -- 172.21.15.89:0/1168731836 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd5a0090020 con 0x7fd598053f20 2023-10-30T12:28:59.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7fd595ffb640 1 --2- 172.21.15.89:0/1168731836 >> [v2:172.21.15.89:6816/3340572039,v1:172.21.15.89:6817/3340572039] conn(0x7fd58406e710 0x7fd584070b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7fd595ffb640 1 -- 172.21.15.89:0/1168731836 --> [v2:172.21.15.89:6816/3340572039,v1:172.21.15.89:6817/3340572039] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd5840711e0 con 0x7fd58406e710 2023-10-30T12:28:59.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7fd595ffb640 1 -- 172.21.15.89:0/1168731836 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd5a00fe840 con 0x7fd598053f20 2023-10-30T12:28:59.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7fd5a48d9640 1 --2- 172.21.15.89:0/1168731836 >> [v2:172.21.15.89:6816/3340572039,v1:172.21.15.89:6817/3340572039] conn(0x7fd58406e710 0x7fd584070b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7fd5a48d9640 1 --2- 172.21.15.89:0/1168731836 >> [v2:172.21.15.89:6816/3340572039,v1:172.21.15.89:6817/3340572039] conn(0x7fd58406e710 0x7fd584070b30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7fd5a50da640 1 --2- 172.21.15.89:0/1168731836 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd5840664f0 0x7fd5840689b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd59c006aa0 tx=0x7fd59c006a30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f0f827fc640 1 -- 172.21.15.89:0/169038631 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0f84005070 con 0x7f0f8c0c8430 2023-10-30T12:28:59.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f0f827fc640 1 --2- 172.21.15.89:0/169038631 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0f70066530 0x7f0f700689f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f0f827fc640 1 -- 172.21.15.89:0/169038631 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0f840844c0 con 0x7f0f8c0c8430 2023-10-30T12:28:59.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f1d5b7fe640 1 -- 172.21.15.89:0/1953870259 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d5c070840 msgr2=0x7f1d5c158c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.288+0000 7f1d5a7fc640 1 --2- 172.21.15.89:0/1953870259 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d5c070840 0x7f1d5c158c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f1d5b7fe640 1 --2- 172.21.15.89:0/1953870259 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d5c070840 0x7f1d5c158c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f1d5b7fe640 1 -- 172.21.15.89:0/1953870259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d5c069e90 msgr2=0x7f1d5c0c2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f1d5b7fe640 1 --2- 172.21.15.89:0/1953870259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d5c069e90 0x7f1d5c0c2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f1d5b7fe640 1 -- 172.21.15.89:0/1953870259 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d44009420 con 0x7f1d5c153810 2023-10-30T12:28:59.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f1d5a7fc640 1 --2- 172.21.15.89:0/1953870259 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d5c070840 0x7f1d5c158c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:28:59.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f1d5b7fe640 1 --2- 172.21.15.89:0/1953870259 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d5c153810 0x7f1d5c1599d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1d5400c390 tx=0x7f1d5400c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f1d537fe640 1 -- 172.21.15.89:0/1953870259 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d54005cf0 con 0x7f1d5c153810 2023-10-30T12:28:59.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f1d617c4640 1 -- 172.21.15.89:0/1953870259 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d5c158180 con 0x7f1d5c153810 2023-10-30T12:28:59.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f1d537fe640 1 -- 172.21.15.89:0/1953870259 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d54005e50 con 0x7f1d5c153810 2023-10-30T12:28:59.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f1d537fe640 1 -- 172.21.15.89:0/1953870259 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d54007310 con 0x7f1d5c153810 2023-10-30T12:28:59.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f1d617c4640 1 -- 172.21.15.89:0/1953870259 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d5c14a620 con 0x7f1d5c153810 2023-10-30T12:28:59.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f1d617c4640 1 -- 172.21.15.89:0/1953870259 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1d5c0c5f50 con 0x7f1d5c153810 2023-10-30T12:28:59.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f75a449e640 1 Processor -- start 2023-10-30T12:28:59.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f75a449e640 1 -- start start 2023-10-30T12:28:59.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f75a449e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75880052b0 0x7f759c0c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f75a449e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f759c150630 0x7f759c0c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f75a449e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f759c0c2f50 0x7f759c154470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f75a449e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c0c3f90 con 0x7f75880052b0 2023-10-30T12:28:59.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f75a449e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c0c5710 con 0x7f759c150630 2023-10-30T12:28:59.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f75a449e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f759c0c33d0 con 0x7f759c0c2f50 2023-10-30T12:28:59.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f75a1a39640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f759c150630 0x7f759c0c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f75a2a3b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f759c0c2f50 0x7f759c154470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f75a2a3b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f759c0c2f50 0x7f759c154470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:37696/0 (socket says 172.21.15.89:37696) 2023-10-30T12:28:59.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f75a2a3b640 1 -- 172.21.15.89:0/3392379781 learned_addr learned my addr 172.21.15.89:0/3392379781 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:59.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f75a223a640 1 --2- 172.21.15.89:0/3392379781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75880052b0 0x7f759c0c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f0b9886a640 1 -- 172.21.15.89:0/3289629890 shutdown_connections 2023-10-30T12:28:59.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f0b9886a640 1 --2- 172.21.15.89:0/3289629890 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b94069e90 0x7f0b940c6890 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f0b9886a640 1 --2- 172.21.15.89:0/3289629890 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b94157c80 0x7f0b94070fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f0b9886a640 1 -- 172.21.15.89:0/3289629890 >> 172.21.15.89:0/3289629890 conn(0x7f0b940c75d0 msgr2=0x7f0b940c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:59.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f6fff054640 1 -- start start 2023-10-30T12:28:59.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f6fff054640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6ff8153810 0x7f6ff815a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f6fff054640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6ff8069e90 0x7f6ff806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f6fff054640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff80c6880 con 0x7f6ff8070840 2023-10-30T12:28:59.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f6fff054640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ff815a820 con 0x7f6ff8069e90 2023-10-30T12:28:59.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f6fff054640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ff80c5990 con 0x7f6ff8153810 2023-10-30T12:28:59.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f6ffd5f1640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6ff8069e90 0x7f6ff806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f6ffd5f1640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6ff8069e90 0x7f6ff806a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55546/0 (socket says 172.21.15.89:55546) 2023-10-30T12:28:59.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f6ffd5f1640 1 -- 172.21.15.89:0/3928146266 learned_addr learned my addr 172.21.15.89:0/3928146266 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:59.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f75a2a3b640 1 -- 172.21.15.89:0/3392379781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f759c150630 msgr2=0x7f759c0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f75a2a3b640 1 --2- 172.21.15.89:0/3392379781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f759c150630 0x7f759c0c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f75a2a3b640 1 -- 172.21.15.89:0/3392379781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75880052b0 msgr2=0x7f759c0c43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f0f827fc640 1 --2- 172.21.15.89:0/169038631 >> [v2:172.21.15.89:6808/1929940799,v1:172.21.15.89:6809/1929940799] conn(0x7f0f7006e6d0 0x7f0f70070af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f0f827fc640 1 -- 172.21.15.89:0/169038631 --> [v2:172.21.15.89:6808/1929940799,v1:172.21.15.89:6809/1929940799] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0f700711a0 con 0x7f0f7006e6d0 2023-10-30T12:28:59.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f0f827fc640 1 -- 172.21.15.89:0/169038631 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f84084910 con 0x7f0f8c0c8430 2023-10-30T12:28:59.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f75a2a3b640 1 --2- 172.21.15.89:0/3392379781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75880052b0 0x7f759c0c43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f0f91603640 1 --2- 172.21.15.89:0/169038631 >> [v2:172.21.15.89:6808/1929940799,v1:172.21.15.89:6809/1929940799] conn(0x7f0f7006e6d0 0x7f0f70070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f75a2a3b640 1 -- 172.21.15.89:0/3392379781 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759c0c5430 con 0x7f759c0c2f50 2023-10-30T12:28:59.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f6ffd5f1640 1 -- 172.21.15.89:0/3928146266 >> v2:172.21.15.192:3300/0 conn(0x7f6ff8153810 msgr2=0x7f6ff815a2e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:28:59.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f75a223a640 1 --2- 172.21.15.89:0/3392379781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75880052b0 0x7f759c0c43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:28:59.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f0b9886a640 1 -- 172.21.15.89:0/3289629890 shutdown_connections 2023-10-30T12:28:59.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f0b9886a640 1 -- 172.21.15.89:0/3289629890 wait complete. 2023-10-30T12:28:59.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f6ffd5f1640 1 --2- 172.21.15.89:0/3928146266 >> v2:172.21.15.192:3300/0 conn(0x7f6ff8153810 0x7f6ff815a2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f6ffd5f1640 1 -- 172.21.15.89:0/3928146266 >> v1:172.21.15.192:6789/0 conn(0x7f6ff8070840 legacy=0x7f6ff8070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:59.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f6ffd5f1640 1 -- 172.21.15.89:0/3928146266 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff80c5c70 con 0x7f6ff8069e90 2023-10-30T12:28:59.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.292+0000 7f0f91e04640 1 --2- 172.21.15.89:0/169038631 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0f70066530 0x7f0f700689f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.296+0000 7f75a2a3b640 1 --2- 172.21.15.89:0/3392379781 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f759c0c2f50 0x7f759c154470 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7594007f10 tx=0x7f7588005800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.296+0000 7f8ab7fff640 1 -- 172.21.15.89:0/2409092753 <== osd.0 v2:172.21.15.89:6800/1673955622 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f8ab0071290 con 0x7f8ab006e7c0 2023-10-30T12:28:59.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.296+0000 7f0f91603640 1 --2- 172.21.15.89:0/169038631 >> [v2:172.21.15.89:6808/1929940799,v1:172.21.15.89:6809/1929940799] conn(0x7f0f7006e6d0 0x7f0f70070af0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.296+0000 7f7592ffd640 1 -- 172.21.15.89:0/3392379781 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75940be070 con 0x7f759c0c2f50 2023-10-30T12:28:59.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.296+0000 7f7592ffd640 1 -- 172.21.15.89:0/3392379781 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75940b9660 con 0x7f759c0c2f50 2023-10-30T12:28:59.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.296+0000 7fd595ffb640 1 -- 172.21.15.89:0/1168731836 <== osd.1 v2:172.21.15.89:6816/3340572039 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fd5840711e0 con 0x7fd58406e710 2023-10-30T12:28:59.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.296+0000 7f7592ffd640 1 -- 172.21.15.89:0/3392379781 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75940a5e20 con 0x7f759c0c2f50 2023-10-30T12:28:59.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.296+0000 7f75a449e640 1 -- 172.21.15.89:0/3392379781 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f759c1eaed0 con 0x7f759c0c2f50 2023-10-30T12:28:59.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.296+0000 7f75a449e640 1 -- 172.21.15.89:0/3392379781 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f759c0c2210 con 0x7f759c0c2f50 2023-10-30T12:28:59.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.296+0000 7f0f91e04640 1 --2- 172.21.15.89:0/169038631 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0f70066530 0x7f0f700689f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009fd0 tx=0x7f0f88009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.296+0000 7f1d537fe640 1 -- 172.21.15.89:0/1953870259 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1d54007470 con 0x7f1d5c153810 2023-10-30T12:28:59.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.296+0000 7f75a449e640 1 -- 172.21.15.89:0/3392379781 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f759c1591d0 con 0x7f759c0c2f50 2023-10-30T12:28:59.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.296+0000 7f0f827fc640 1 -- 172.21.15.89:0/169038631 <== osd.2 v2:172.21.15.89:6808/1929940799 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f0f700711a0 con 0x7f0f7006e6d0 2023-10-30T12:28:59.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.296+0000 7f0b9886a640 1 Processor -- start 2023-10-30T12:28:59.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.296+0000 7f0b9886a640 1 -- start start 2023-10-30T12:28:59.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.296+0000 7f623dc71640 1 --2- 172.21.15.89:0/527888041 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6238152eb0 0x7f62380713a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f622800b920 tx=0x7f622801c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ae0c6c46ab1b82d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.296+0000 7f1d537fe640 1 --2- 172.21.15.89:0/1953870259 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1d3c0664d0 0x7f1d3c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.296+0000 7f1d537fe640 1 -- 172.21.15.89:0/1953870259 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1d54083390 con 0x7f1d5c153810 2023-10-30T12:28:59.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f1d5affd640 1 --2- 172.21.15.89:0/1953870259 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1d3c0664d0 0x7f1d3c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f1d537fe640 1 --2- 172.21.15.89:0/1953870259 >> [v2:172.21.15.89:6824/345087939,v1:172.21.15.89:6825/345087939] conn(0x7f1d3c06e670 0x7f1d3c070a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f1d537fe640 1 -- 172.21.15.89:0/1953870259 --> [v2:172.21.15.89:6824/345087939,v1:172.21.15.89:6825/345087939] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1d3c071140 con 0x7f1d3c06e670 2023-10-30T12:28:59.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f1d537fe640 1 -- 172.21.15.89:0/1953870259 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d540837e0 con 0x7f1d5c153810 2023-10-30T12:28:59.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f0b9886a640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b94069e90 0x7f0b941f6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f623d470640 1 -- 172.21.15.89:0/527888041 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62280a59e0 con 0x7f6238152eb0 2023-10-30T12:28:59.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f623d470640 1 -- 172.21.15.89:0/527888041 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62280a5b40 con 0x7f6238152eb0 2023-10-30T12:28:59.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f623d470640 1 -- 172.21.15.89:0/527888041 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62280ad440 con 0x7f6238152eb0 2023-10-30T12:28:59.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f0b9886a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b9406f340 0x7f0b9414e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f0b9886a640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b94157c80 0x7f0b941f90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f0b9886a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b94162d50 con 0x7f0b9406f340 2023-10-30T12:28:59.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f0b9886a640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b94168050 con 0x7f0b94157c80 2023-10-30T12:28:59.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f0b9886a640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b941f4520 con 0x7f0b94069e90 2023-10-30T12:28:59.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f1d5a7fc640 1 --2- 172.21.15.89:0/1953870259 >> [v2:172.21.15.89:6824/345087939,v1:172.21.15.89:6825/345087939] conn(0x7f1d3c06e670 0x7f1d3c070a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f0b92d9d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b94069e90 0x7f0b941f6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f0b92d9d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b94069e90 0x7f0b941f6fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:37698/0 (socket says 172.21.15.89:37698) 2023-10-30T12:28:59.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f0b9259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b9406f340 0x7f0b9414e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f1d5affd640 1 --2- 172.21.15.89:0/1953870259 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1d3c0664d0 0x7f1d3c068990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c00f510 tx=0x7f1d4c0077b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f0b92d9d640 1 -- 172.21.15.89:0/4012783335 learned_addr learned my addr 172.21.15.89:0/4012783335 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:59.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f7592ffd640 1 -- 172.21.15.89:0/3392379781 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f75940b9810 con 0x7f759c0c2f50 2023-10-30T12:28:59.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f6ffd5f1640 1 --2- 172.21.15.89:0/3928146266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ff8069e90 0x7f6ff806a2e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8072640 tx=0x7f6fe001ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1a44cf4ece367d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f1d5a7fc640 1 --2- 172.21.15.89:0/1953870259 >> [v2:172.21.15.89:6824/345087939,v1:172.21.15.89:6825/345087939] conn(0x7f1d3c06e670 0x7f1d3c070a90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f0b92d9d640 1 -- 172.21.15.89:0/4012783335 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b94157c80 msgr2=0x7f0b941f90a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-30T12:28:59.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f0b92d9d640 1 --2- 172.21.15.89:0/4012783335 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b94157c80 0x7f0b941f90a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f0b92d9d640 1 -- 172.21.15.89:0/4012783335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b9406f340 msgr2=0x7f0b9414e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f0b92d9d640 1 --2- 172.21.15.89:0/4012783335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b9406f340 0x7f0b9414e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f0b92d9d640 1 -- 172.21.15.89:0/4012783335 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b8c009a70 con 0x7f0b94069e90 2023-10-30T12:28:59.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f623f474640 1 -- 172.21.15.89:0/527888041 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6238152eb0 msgr2=0x7f62380713a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f623f474640 1 --2- 172.21.15.89:0/527888041 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6238152eb0 0x7f62380713a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f622800b920 tx=0x7f622801c920 comp rx=0 tx=0).stop 2023-10-30T12:28:59.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f0b92d9d640 1 --2- 172.21.15.89:0/4012783335 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b94069e90 0x7f0b941f6fa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0b8400e870 tx=0x7f0b8400ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f623f474640 1 -- 172.21.15.89:0/527888041 shutdown_connections 2023-10-30T12:28:59.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f623f474640 1 --2- 172.21.15.89:0/527888041 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6238152eb0 0x7f62380713a0 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f622800b920 tx=0x7f622801c920 comp rx=0 tx=0).stop 2023-10-30T12:28:59.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f623f474640 1 --2- 172.21.15.89:0/527888041 >> v2:172.21.15.192:3300/0 conn(0x7f62380c80f0 0x7f62380c2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f623f474640 1 -- 172.21.15.89:0/527888041 >> 172.21.15.89:0/527888041 conn(0x7f62380c75d0 msgr2=0x7f62380c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:59.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f7592ffd640 1 --2- 172.21.15.89:0/3392379781 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f7580066570 0x7f7580068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f7592ffd640 1 -- 172.21.15.89:0/3392379781 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f759411eb70 con 0x7f759c0c2f50 2023-10-30T12:28:59.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f7592ffd640 1 --2- 172.21.15.89:0/3392379781 >> [v2:172.21.15.192:6800/2138125545,v1:172.21.15.192:6801/2138125545] conn(0x7f758006e790 0x7f7580070bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f7592ffd640 1 -- 172.21.15.89:0/3392379781 --> [v2:172.21.15.192:6800/2138125545,v1:172.21.15.192:6801/2138125545] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7580071260 con 0x7f758006e790 2023-10-30T12:28:59.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f7592ffd640 1 -- 172.21.15.89:0/3392379781 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f759411ef20 con 0x7f759c0c2f50 2023-10-30T12:28:59.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f0b8b7fe640 1 -- 172.21.15.89:0/4012783335 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8400eec0 con 0x7f0b94069e90 2023-10-30T12:28:59.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f0b8b7fe640 1 -- 172.21.15.89:0/4012783335 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b8400c940 con 0x7f0b94069e90 2023-10-30T12:28:59.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f0b8b7fe640 1 -- 172.21.15.89:0/4012783335 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8400ccb0 con 0x7f0b94069e90 2023-10-30T12:28:59.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f0b9886a640 1 -- 172.21.15.89:0/4012783335 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b9414ef00 con 0x7f0b94069e90 2023-10-30T12:28:59.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f0b9886a640 1 -- 172.21.15.89:0/4012783335 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b941f4770 con 0x7f0b94069e90 2023-10-30T12:28:59.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f75a223a640 1 --2- 172.21.15.89:0/3392379781 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f7580066570 0x7f7580068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f6ff77fe640 1 -- 172.21.15.89:0/3928146266 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe00a89d0 con 0x7f6ff8069e90 2023-10-30T12:28:59.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f6ff77fe640 1 -- 172.21.15.89:0/3928146266 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe00a8b30 con 0x7f6ff8069e90 2023-10-30T12:28:59.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.300+0000 7f6ff77fe640 1 -- 172.21.15.89:0/3928146266 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe00b0440 con 0x7f6ff8069e90 2023-10-30T12:28:59.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.304+0000 7f75a1a39640 1 --2- 172.21.15.89:0/3392379781 >> [v2:172.21.15.192:6800/2138125545,v1:172.21.15.192:6801/2138125545] conn(0x7f758006e790 0x7f7580070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.304+0000 7f1d537fe640 1 -- 172.21.15.89:0/1953870259 <== osd.3 v2:172.21.15.89:6824/345087939 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f1d3c071140 con 0x7f1d3c06e670 2023-10-30T12:28:59.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.304+0000 7f623f474640 1 -- 172.21.15.89:0/527888041 shutdown_connections 2023-10-30T12:28:59.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.304+0000 7f0b9886a640 1 -- 172.21.15.89:0/4012783335 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0b94070ab0 con 0x7f0b94069e90 2023-10-30T12:28:59.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.304+0000 7f75a223a640 1 --2- 172.21.15.89:0/3392379781 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f7580066570 0x7f7580068a30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f758c001ec0 tx=0x7f758c000ee0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.304+0000 7f6fff054640 1 -- 172.21.15.89:0/3928146266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ff8069e90 msgr2=0x7f6ff806a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.304+0000 7f6fff054640 1 --2- 172.21.15.89:0/3928146266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ff8069e90 0x7f6ff806a2e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8072640 tx=0x7f6fe001ecd0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.304+0000 7f6fff054640 1 -- 172.21.15.89:0/3928146266 shutdown_connections 2023-10-30T12:28:59.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.304+0000 7f6fff054640 1 --2- 172.21.15.89:0/3928146266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ff8069e90 0x7f6ff806a2e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.304+0000 7f6fff054640 1 --2- 172.21.15.89:0/3928146266 >> v2:172.21.15.192:3300/0 conn(0x7f6ff8153810 0x7f6ff815a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.304+0000 7f623f474640 1 -- 172.21.15.89:0/527888041 wait complete. 2023-10-30T12:28:59.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.304+0000 7f0b8b7fe640 1 -- 172.21.15.89:0/4012783335 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0b84026480 con 0x7f0b94069e90 2023-10-30T12:28:59.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.304+0000 7f0b8b7fe640 1 --2- 172.21.15.89:0/4012783335 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0b740664f0 0x7f0b740689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.304+0000 7f623f474640 1 Processor -- start 2023-10-30T12:28:59.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.304+0000 7f6fff054640 1 -- 172.21.15.89:0/3928146266 >> 172.21.15.89:0/3928146266 conn(0x7f6ff80c7910 msgr2=0x7f6ff80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:59.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.304+0000 7f0b8b7fe640 1 -- 172.21.15.89:0/4012783335 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0b84055ed0 con 0x7f0b94069e90 2023-10-30T12:28:59.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.304+0000 7f0b8b7fe640 1 --2- 172.21.15.89:0/4012783335 >> [v2:172.21.15.192:6804/2360218671,v1:172.21.15.192:6806/2360218671] conn(0x7f0b7406f3f0 0x7f0b74071890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.304+0000 7f0b8b7fe640 1 -- 172.21.15.89:0/4012783335 --> [v2:172.21.15.192:6804/2360218671,v1:172.21.15.192:6806/2360218671] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0b74071f60 con 0x7f0b7406f3f0 2023-10-30T12:28:59.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.304+0000 7f0b8b7fe640 1 -- 172.21.15.89:0/4012783335 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b840555a0 con 0x7f0b94069e90 2023-10-30T12:28:59.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.304+0000 7f0b9259c640 1 --2- 172.21.15.89:0/4012783335 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0b740664f0 0x7f0b740689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.308+0000 7f0b9359e640 1 --2- 172.21.15.89:0/4012783335 >> [v2:172.21.15.192:6804/2360218671,v1:172.21.15.192:6806/2360218671] conn(0x7f0b7406f3f0 0x7f0b74071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.304+0000 7f6fff054640 1 -- 172.21.15.89:0/3928146266 shutdown_connections 2023-10-30T12:28:59.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.308+0000 7f6fff054640 1 -- 172.21.15.89:0/3928146266 wait complete. 2023-10-30T12:28:59.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.308+0000 7f75a1a39640 1 --2- 172.21.15.89:0/3392379781 >> [v2:172.21.15.192:6800/2138125545,v1:172.21.15.192:6801/2138125545] conn(0x7f758006e790 0x7f7580070bb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.308+0000 7f623f474640 1 -- start start 2023-10-30T12:28:59.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.308+0000 7f0b9359e640 1 --2- 172.21.15.89:0/4012783335 >> [v2:172.21.15.192:6804/2360218671,v1:172.21.15.192:6806/2360218671] conn(0x7f0b7406f3f0 0x7f0b74071890 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.308+0000 7f0b9259c640 1 --2- 172.21.15.89:0/4012783335 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0b740664f0 0x7f0b740689b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0b9414f300 tx=0x7f0b7c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.308+0000 7f7592ffd640 1 -- 172.21.15.89:0/3392379781 <== osd.4 v2:172.21.15.192:6800/2138125545 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f7580071260 con 0x7f758006e790 2023-10-30T12:28:59.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.308+0000 7f0b8b7fe640 1 -- 172.21.15.89:0/4012783335 <== osd.5 v2:172.21.15.192:6804/2360218671 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f0b74071f60 con 0x7f0b7406f3f0 2023-10-30T12:28:59.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.308+0000 7f623f474640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62380700c0 0x7f6238151490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.308+0000 7f6fff054640 1 Processor -- start 2023-10-30T12:28:59.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.308+0000 7f6fff054640 1 -- start start 2023-10-30T12:28:59.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.308+0000 7f623f474640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62380c80f0 0x7f62380c9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.308+0000 7f623f474640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62380ca240 0x7f6238140b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.312+0000 7f623f474640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6238152070 con 0x7f62380c80f0 2023-10-30T12:28:59.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.312+0000 7f623f474640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62381519d0 con 0x7f62380ca240 2023-10-30T12:28:59.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.312+0000 7f623e472640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62380700c0 0x7f6238151490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.312+0000 7f623f474640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6238140120 con 0x7f62380700c0 2023-10-30T12:28:59.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.312+0000 7f623e472640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62380700c0 0x7f6238151490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:37712/0 (socket says 172.21.15.89:37712) 2023-10-30T12:28:59.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.312+0000 7f623e472640 1 -- 172.21.15.89:0/1030450109 learned_addr learned my addr 172.21.15.89:0/1030450109 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:59.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.312+0000 7f623ec73640 1 --2- 172.21.15.89:0/1030450109 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62380ca240 0x7f6238140b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.312+0000 7f623dc71640 1 --2- 172.21.15.89:0/1030450109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62380c80f0 0x7f62380c9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.312+0000 7f623e472640 1 -- 172.21.15.89:0/1030450109 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62380ca240 msgr2=0x7f6238140b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.312+0000 7f623e472640 1 --2- 172.21.15.89:0/1030450109 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62380ca240 0x7f6238140b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.312+0000 7f623e472640 1 -- 172.21.15.89:0/1030450109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62380c80f0 msgr2=0x7f62380c9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.312+0000 7f623e472640 1 --2- 172.21.15.89:0/1030450109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62380c80f0 0x7f62380c9d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.312+0000 7f623e472640 1 -- 172.21.15.89:0/1030450109 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f623400d850 con 0x7f62380700c0 2023-10-30T12:28:59.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.312+0000 7f623e472640 1 --2- 172.21.15.89:0/1030450109 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62380700c0 0x7f6238151490 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6234012fc0 tx=0x7f6234010490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.312+0000 7f6fff054640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ff8069e90 0x7f6ff8145420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.312+0000 7f622effd640 1 -- 172.21.15.89:0/1030450109 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623400db60 con 0x7f62380700c0 2023-10-30T12:28:59.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.312+0000 7f623f474640 1 -- 172.21.15.89:0/1030450109 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f622800b580 con 0x7f62380700c0 2023-10-30T12:28:59.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.312+0000 7f623f474640 1 -- 172.21.15.89:0/1030450109 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6238152580 con 0x7f62380700c0 2023-10-30T12:28:59.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.312+0000 7f622effd640 1 -- 172.21.15.89:0/1030450109 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6234013040 con 0x7f62380700c0 2023-10-30T12:28:59.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.312+0000 7f622effd640 1 -- 172.21.15.89:0/1030450109 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623400d2e0 con 0x7f62380700c0 2023-10-30T12:28:59.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.312+0000 7f6fff054640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ff8070840 0x7f6ff80c2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.312+0000 7f6ffcdf0640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ff8069e90 0x7f6ff8145420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.312+0000 7f6ffcdf0640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ff8069e90 0x7f6ff8145420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:37722/0 (socket says 172.21.15.89:37722) 2023-10-30T12:28:59.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.312+0000 7f6ffcdf0640 1 -- 172.21.15.89:0/3518626517 learned_addr learned my addr 172.21.15.89:0/3518626517 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:59.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.312+0000 7f6fff054640 1 --2- 172.21.15.89:0/3518626517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ff8153810 0x7f6ff80ca290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.312+0000 7f6ff7fff640 1 --2- 172.21.15.89:0/3518626517 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ff8070840 0x7f6ff80c2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.312+0000 7f6ffd5f1640 1 --2- 172.21.15.89:0/3518626517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ff8153810 0x7f6ff80ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.312+0000 7f6fff054640 1 -- 172.21.15.89:0/3518626517 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff8142660 con 0x7f6ff8153810 2023-10-30T12:28:59.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.312+0000 7f6fff054640 1 -- 172.21.15.89:0/3518626517 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff8145960 con 0x7f6ff8070840 2023-10-30T12:28:59.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.312+0000 7f6fff054640 1 -- 172.21.15.89:0/3518626517 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff815a600 con 0x7f6ff8069e90 2023-10-30T12:28:59.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.316+0000 7f6ffcdf0640 1 -- 172.21.15.89:0/3518626517 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ff8070840 msgr2=0x7f6ff80c2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.316+0000 7f6ffcdf0640 1 --2- 172.21.15.89:0/3518626517 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ff8070840 0x7f6ff80c2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.316+0000 7f6ffcdf0640 1 -- 172.21.15.89:0/3518626517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ff8153810 msgr2=0x7f6ff80ca290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.316+0000 7f6ffcdf0640 1 --2- 172.21.15.89:0/3518626517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ff8153810 0x7f6ff80ca290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.316+0000 7f6ffcdf0640 1 -- 172.21.15.89:0/3518626517 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe00094d0 con 0x7f6ff8069e90 2023-10-30T12:28:59.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.316+0000 7f6ffcdf0640 1 --2- 172.21.15.89:0/3518626517 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ff8069e90 0x7f6ff8145420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6ff000ba50 tx=0x7f6ff000bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.316+0000 7f6ff57fa640 1 -- 172.21.15.89:0/3518626517 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff000cc10 con 0x7f6ff8069e90 2023-10-30T12:28:59.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.316+0000 7f6fff054640 1 -- 172.21.15.89:0/3518626517 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff8146740 con 0x7f6ff8069e90 2023-10-30T12:28:59.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.316+0000 7f6ff57fa640 1 -- 172.21.15.89:0/3518626517 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff000cd70 con 0x7f6ff8069e90 2023-10-30T12:28:59.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.316+0000 7f6ff57fa640 1 -- 172.21.15.89:0/3518626517 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff0014680 con 0x7f6ff8069e90 2023-10-30T12:28:59.419 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.316+0000 7f6fff054640 1 -- 172.21.15.89:0/3518626517 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff80c4460 con 0x7f6ff8069e90 2023-10-30T12:28:59.419 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.320+0000 7f0f94068640 1 -- 172.21.15.89:0/169038631 --> [v2:172.21.15.89:6808/1929940799,v1:172.21.15.89:6809/1929940799] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0f8c0c5e70 con 0x7f0f7006e6d0 2023-10-30T12:28:59.419 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.320+0000 7f6fff054640 1 -- 172.21.15.89:0/3518626517 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6ff80c6230 con 0x7f6ff8069e90 2023-10-30T12:28:59.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.316+0000 7f622effd640 1 -- 172.21.15.89:0/1030450109 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f62340059a0 con 0x7f62380700c0 2023-10-30T12:28:59.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.316+0000 7f622cff9640 1 -- 172.21.15.89:0/1030450109 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6238070e80 con 0x7f62380700c0 2023-10-30T12:28:59.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.320+0000 7f622effd640 1 --2- 172.21.15.89:0/1030450109 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f621c06edb0 0x7f621c071270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.320+0000 7f622effd640 1 -- 172.21.15.89:0/1030450109 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f623400b020 con 0x7f62380700c0 2023-10-30T12:28:59.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.320+0000 7f622effd640 1 --2- 172.21.15.89:0/1030450109 >> [v2:172.21.15.192:6824/3707671287,v1:172.21.15.192:6825/3707671287] conn(0x7f621c076f50 0x7f621c079370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.320+0000 7f623dc71640 1 --2- 172.21.15.89:0/1030450109 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f621c06edb0 0x7f621c071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.320+0000 7f622effd640 1 -- 172.21.15.89:0/1030450109 --> [v2:172.21.15.192:6824/3707671287,v1:172.21.15.192:6825/3707671287] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f621c079a20 con 0x7f621c076f50 2023-10-30T12:28:59.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.320+0000 7f622effd640 1 -- 172.21.15.89:0/1030450109 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6234050f10 con 0x7f62380700c0 2023-10-30T12:28:59.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.320+0000 7f623ec73640 1 --2- 172.21.15.89:0/1030450109 >> [v2:172.21.15.192:6824/3707671287,v1:172.21.15.192:6825/3707671287] conn(0x7f621c076f50 0x7f621c079370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.320+0000 7f623dc71640 1 --2- 172.21.15.89:0/1030450109 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f621c06edb0 0x7f621c071270 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f62280af000 tx=0x7f62280a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.320+0000 7f623ec73640 1 --2- 172.21.15.89:0/1030450109 >> [v2:172.21.15.192:6824/3707671287,v1:172.21.15.192:6825/3707671287] conn(0x7f621c076f50 0x7f621c079370 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.320+0000 7f6ff57fa640 1 -- 172.21.15.89:0/3518626517 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6ff00147e0 con 0x7f6ff8069e90 2023-10-30T12:28:59.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.320+0000 7f0f827fc640 1 -- 172.21.15.89:0/169038631 <== osd.2 v2:172.21.15.89:6808/1929940799 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0f7c0ab490 con 0x7f0f7006e6d0 2023-10-30T12:28:59.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.320+0000 7f0f94068640 1 -- 172.21.15.89:0/169038631 >> [v2:172.21.15.89:6808/1929940799,v1:172.21.15.89:6809/1929940799] conn(0x7f0f7006e6d0 msgr2=0x7f0f70070af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.320+0000 7f0f94068640 1 --2- 172.21.15.89:0/169038631 >> [v2:172.21.15.89:6808/1929940799,v1:172.21.15.89:6809/1929940799] conn(0x7f0f7006e6d0 0x7f0f70070af0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.324+0000 7f0f94068640 1 -- 172.21.15.89:0/169038631 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0f70066530 msgr2=0x7f0f700689f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.324+0000 7f0f94068640 1 --2- 172.21.15.89:0/169038631 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0f70066530 0x7f0f700689f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009fd0 tx=0x7f0f88009190 comp rx=0 tx=0).stop 2023-10-30T12:28:59.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.324+0000 7f0f94068640 1 -- 172.21.15.89:0/169038631 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f8c0c8430 msgr2=0x7f0f8c0c9c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.324+0000 7f0f94068640 1 --2- 172.21.15.89:0/169038631 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f8c0c8430 0x7f0f8c0c9c30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0f8400c780 tx=0x7f0f8400cc40 comp rx=0 tx=0).stop 2023-10-30T12:28:59.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.324+0000 7f0f92605640 1 -- 172.21.15.89:0/169038631 reap_dead start 2023-10-30T12:28:59.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.324+0000 7f6ff57fa640 1 --2- 172.21.15.89:0/3518626517 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6fd80664a0 0x7f6fd8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.324+0000 7f0f94068640 1 -- 172.21.15.89:0/169038631 shutdown_connections 2023-10-30T12:28:59.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.324+0000 7f0f94068640 1 -- 172.21.15.89:0/169038631 >> 172.21.15.89:0/169038631 conn(0x7f0f8c0c7910 msgr2=0x7f0f8c150920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:59.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.324+0000 7f0f94068640 1 -- 172.21.15.89:0/169038631 shutdown_connections 2023-10-30T12:28:59.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.324+0000 7f0f94068640 1 -- 172.21.15.89:0/169038631 wait complete. 2023-10-30T12:28:59.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.324+0000 7f6ff57fa640 1 -- 172.21.15.89:0/3518626517 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ff0084460 con 0x7f6ff8069e90 2023-10-30T12:28:59.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.324+0000 7f622effd640 1 -- 172.21.15.89:0/1030450109 <== osd.6 v2:172.21.15.192:6824/3707671287 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f621c079a20 con 0x7f621c076f50 2023-10-30T12:28:59.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.324+0000 7f6ff57fa640 1 --2- 172.21.15.89:0/3518626517 >> [v2:172.21.15.192:6816/3111945871,v1:172.21.15.192:6817/3111945871] conn(0x7f6fd806e6c0 0x7f6fd8070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.324+0000 7f6ff57fa640 1 -- 172.21.15.89:0/3518626517 --> [v2:172.21.15.192:6816/3111945871,v1:172.21.15.192:6817/3111945871] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6fd8071190 con 0x7f6fd806e6c0 2023-10-30T12:28:59.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.324+0000 7f6ff57fa640 1 -- 172.21.15.89:0/3518626517 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ff0084810 con 0x7f6ff8069e90 2023-10-30T12:28:59.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.324+0000 7f6ff7fff640 1 --2- 172.21.15.89:0/3518626517 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6fd80664a0 0x7f6fd8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.324+0000 7fd5a60dc640 1 -- 172.21.15.89:0/1168731836 --> [v2:172.21.15.89:6816/3340572039,v1:172.21.15.89:6817/3340572039] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd598037410 con 0x7fd58406e710 2023-10-30T12:28:59.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.324+0000 7fd595ffb640 1 -- 172.21.15.89:0/1168731836 <== osd.1 v2:172.21.15.89:6816/3340572039 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd590001600 con 0x7fd58406e710 2023-10-30T12:28:59.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.324+0000 7f6ffd5f1640 1 --2- 172.21.15.89:0/3518626517 >> [v2:172.21.15.192:6816/3111945871,v1:172.21.15.192:6817/3111945871] conn(0x7f6fd806e6c0 0x7f6fd8070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.324+0000 7f6ffd5f1640 1 --2- 172.21.15.89:0/3518626517 >> [v2:172.21.15.192:6816/3111945871,v1:172.21.15.192:6817/3111945871] conn(0x7f6fd806e6c0 0x7f6fd8070ae0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.328+0000 7fd5a60dc640 1 -- 172.21.15.89:0/1168731836 >> [v2:172.21.15.89:6816/3340572039,v1:172.21.15.89:6817/3340572039] conn(0x7fd58406e710 msgr2=0x7fd584070b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.328+0000 7fd5a60dc640 1 --2- 172.21.15.89:0/1168731836 >> [v2:172.21.15.89:6816/3340572039,v1:172.21.15.89:6817/3340572039] conn(0x7fd58406e710 0x7fd584070b30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.328+0000 7f6ff57fa640 1 -- 172.21.15.89:0/3518626517 <== osd.7 v2:172.21.15.192:6816/3111945871 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f6fd8071190 con 0x7f6fd806e6c0 2023-10-30T12:28:59.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.328+0000 7f8ad0c3a640 1 -- 172.21.15.89:0/2409092753 --> [v2:172.21.15.89:6800/1673955622,v1:172.21.15.89:6801/1673955622] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8ac40457e0 con 0x7f8ab006e7c0 2023-10-30T12:28:59.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.328+0000 7f1d617c4640 1 -- 172.21.15.89:0/1953870259 --> [v2:172.21.15.89:6824/345087939,v1:172.21.15.89:6825/345087939] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1d5c0c5c70 con 0x7f1d3c06e670 2023-10-30T12:28:59.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.328+0000 7fd5a60dc640 1 -- 172.21.15.89:0/1168731836 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd5840664f0 msgr2=0x7fd5840689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.328+0000 7fd5a60dc640 1 --2- 172.21.15.89:0/1168731836 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd5840664f0 0x7fd5840689b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd59c006aa0 tx=0x7fd59c006a30 comp rx=0 tx=0).stop 2023-10-30T12:28:59.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.328+0000 7fd5a60dc640 1 -- 172.21.15.89:0/1168731836 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd598053f20 msgr2=0x7fd5980d0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.328+0000 7fd5a60dc640 1 --2- 172.21.15.89:0/1168731836 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd598053f20 0x7fd5980d0c90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0063ef0 tx=0x7fd5a0066e60 comp rx=0 tx=0).stop 2023-10-30T12:28:59.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.332+0000 7fd5a58db640 1 -- 172.21.15.89:0/1168731836 reap_dead start 2023-10-30T12:28:59.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.332+0000 7f1d537fe640 1 -- 172.21.15.89:0/1953870259 <== osd.3 v2:172.21.15.89:6824/345087939 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1d44004040 con 0x7f1d3c06e670 2023-10-30T12:28:59.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.332+0000 7f6ff7fff640 1 --2- 172.21.15.89:0/3518626517 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6fd80664a0 0x7f6fd8068960 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6ff815ac00 tx=0x7f6fe8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.332+0000 7fd5a60dc640 1 -- 172.21.15.89:0/1168731836 shutdown_connections 2023-10-30T12:28:59.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.332+0000 7fd5a60dc640 1 -- 172.21.15.89:0/1168731836 >> 172.21.15.89:0/1168731836 conn(0x7fd5980062c0 msgr2=0x7fd598052b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:59.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.332+0000 7fd5a60dc640 1 -- 172.21.15.89:0/1168731836 shutdown_connections 2023-10-30T12:28:59.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.332+0000 7fd5a60dc640 1 -- 172.21.15.89:0/1168731836 wait complete. 2023-10-30T12:28:59.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.332+0000 7f8ab7fff640 1 -- 172.21.15.89:0/2409092753 <== osd.0 v2:172.21.15.89:6800/1673955622 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8acc04e760 con 0x7f8ab006e7c0 2023-10-30T12:28:59.435 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.332+0000 7f1d617c4640 1 -- 172.21.15.89:0/1953870259 >> [v2:172.21.15.89:6824/345087939,v1:172.21.15.89:6825/345087939] conn(0x7f1d3c06e670 msgr2=0x7f1d3c070a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.435 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.332+0000 7f1d617c4640 1 --2- 172.21.15.89:0/1953870259 >> [v2:172.21.15.89:6824/345087939,v1:172.21.15.89:6825/345087939] conn(0x7f1d3c06e670 0x7f1d3c070a90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.435 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.332+0000 7f1d617c4640 1 -- 172.21.15.89:0/1953870259 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1d3c0664d0 msgr2=0x7f1d3c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.436 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.332+0000 7f8ad0c3a640 1 -- 172.21.15.89:0/2409092753 >> [v2:172.21.15.89:6800/1673955622,v1:172.21.15.89:6801/1673955622] conn(0x7f8ab006e7c0 msgr2=0x7f8ab0070be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.436 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.332+0000 7f1d617c4640 1 --2- 172.21.15.89:0/1953870259 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1d3c0664d0 0x7f1d3c068990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c00f510 tx=0x7f1d4c0077b0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.332+0000 7f1d617c4640 1 -- 172.21.15.89:0/1953870259 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d5c153810 msgr2=0x7f1d5c1599d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.332+0000 7f1d617c4640 1 --2- 172.21.15.89:0/1953870259 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d5c153810 0x7f1d5c1599d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1d5400c390 tx=0x7f1d5400c850 comp rx=0 tx=0).stop 2023-10-30T12:28:59.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.332+0000 7f8ad0c3a640 1 --2- 172.21.15.89:0/2409092753 >> [v2:172.21.15.89:6800/1673955622,v1:172.21.15.89:6801/1673955622] conn(0x7f8ab006e7c0 0x7f8ab0070be0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.332+0000 7f8ad0c3a640 1 -- 172.21.15.89:0/2409092753 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8ab00665a0 msgr2=0x7f8ab0068a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.332+0000 7f8ad0c3a640 1 --2- 172.21.15.89:0/2409092753 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8ab00665a0 0x7f8ab0068a60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8ac416d560 tx=0x7f8ab8008040 comp rx=0 tx=0).stop 2023-10-30T12:28:59.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.332+0000 7f8ad0c3a640 1 -- 172.21.15.89:0/2409092753 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ac404c690 msgr2=0x7f8ac41796c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.332+0000 7f8ad0c3a640 1 --2- 172.21.15.89:0/2409092753 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ac404c690 0x7f8ac41796c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8ac001c920 tx=0x7f8ac001ba90 comp rx=0 tx=0).stop 2023-10-30T12:28:59.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.332+0000 7f8acbfff640 1 -- 172.21.15.89:0/2409092753 reap_dead start 2023-10-30T12:28:59.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.332+0000 7f8ad0c3a640 1 -- 172.21.15.89:0/2409092753 shutdown_connections 2023-10-30T12:28:59.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.332+0000 7f8ad0c3a640 1 -- 172.21.15.89:0/2409092753 >> 172.21.15.89:0/2409092753 conn(0x7f8ac40502e0 msgr2=0x7f8ac40462c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:59.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.332+0000 7f8ad0c3a640 1 -- 172.21.15.89:0/2409092753 shutdown_connections 2023-10-30T12:28:59.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.332+0000 7f8ad0c3a640 1 -- 172.21.15.89:0/2409092753 wait complete. 2023-10-30T12:28:59.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.332+0000 7f1d5b7fe640 1 -- 172.21.15.89:0/1953870259 reap_dead start 2023-10-30T12:28:59.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.336+0000 7f75a449e640 1 -- 172.21.15.89:0/3392379781 --> [v2:172.21.15.192:6800/2138125545,v1:172.21.15.192:6801/2138125545] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f759c0c0310 con 0x7f758006e790 2023-10-30T12:28:59.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.336+0000 7f1d617c4640 1 -- 172.21.15.89:0/1953870259 shutdown_connections 2023-10-30T12:28:59.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.336+0000 7f1d617c4640 1 -- 172.21.15.89:0/1953870259 >> 172.21.15.89:0/1953870259 conn(0x7f1d5c0c7910 msgr2=0x7f1d5c150b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:59.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.336+0000 7f1d617c4640 1 -- 172.21.15.89:0/1953870259 shutdown_connections 2023-10-30T12:28:59.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.336+0000 7f1d617c4640 1 -- 172.21.15.89:0/1953870259 wait complete. 2023-10-30T12:28:59.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.336+0000 7f7592ffd640 1 -- 172.21.15.89:0/3392379781 <== osd.4 v2:172.21.15.192:6800/2138125545 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f759800c870 con 0x7f758006e790 2023-10-30T12:28:59.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.336+0000 7f75a449e640 1 -- 172.21.15.89:0/3392379781 >> [v2:172.21.15.192:6800/2138125545,v1:172.21.15.192:6801/2138125545] conn(0x7f758006e790 msgr2=0x7f7580070bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.336+0000 7f75a449e640 1 --2- 172.21.15.89:0/3392379781 >> [v2:172.21.15.192:6800/2138125545,v1:172.21.15.192:6801/2138125545] conn(0x7f758006e790 0x7f7580070bb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.336+0000 7f75a449e640 1 -- 172.21.15.89:0/3392379781 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f7580066570 msgr2=0x7f7580068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.336+0000 7f75a449e640 1 --2- 172.21.15.89:0/3392379781 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f7580066570 0x7f7580068a30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f758c001ec0 tx=0x7f758c000ee0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.336+0000 7f75a449e640 1 -- 172.21.15.89:0/3392379781 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f759c0c2f50 msgr2=0x7f759c154470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.336+0000 7f75a449e640 1 --2- 172.21.15.89:0/3392379781 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f759c0c2f50 0x7f759c154470 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7594007f10 tx=0x7f7588005800 comp rx=0 tx=0).stop 2023-10-30T12:28:59.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.336+0000 7f75a2a3b640 1 -- 172.21.15.89:0/3392379781 reap_dead start 2023-10-30T12:28:59.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.340+0000 7f75a449e640 1 -- 172.21.15.89:0/3392379781 shutdown_connections 2023-10-30T12:28:59.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.340+0000 7f75a449e640 1 -- 172.21.15.89:0/3392379781 >> 172.21.15.89:0/3392379781 conn(0x7f759c0c7910 msgr2=0x7f759c0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:59.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.340+0000 7f75a449e640 1 -- 172.21.15.89:0/3392379781 shutdown_connections 2023-10-30T12:28:59.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.340+0000 7f75a449e640 1 -- 172.21.15.89:0/3392379781 wait complete. 2023-10-30T12:28:59.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.340+0000 7f0b9886a640 1 -- 172.21.15.89:0/4012783335 --> [v2:172.21.15.192:6804/2360218671,v1:172.21.15.192:6806/2360218671] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0b940707d0 con 0x7f0b7406f3f0 2023-10-30T12:28:59.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.344+0000 7f0b8b7fe640 1 -- 172.21.15.89:0/4012783335 <== osd.5 v2:172.21.15.192:6804/2360218671 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0b8c0a58c0 con 0x7f0b7406f3f0 2023-10-30T12:28:59.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.348+0000 7f0b897fa640 1 -- 172.21.15.89:0/4012783335 >> [v2:172.21.15.192:6804/2360218671,v1:172.21.15.192:6806/2360218671] conn(0x7f0b7406f3f0 msgr2=0x7f0b74071890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.348+0000 7f0b897fa640 1 --2- 172.21.15.89:0/4012783335 >> [v2:172.21.15.192:6804/2360218671,v1:172.21.15.192:6806/2360218671] conn(0x7f0b7406f3f0 0x7f0b74071890 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.348+0000 7f0b897fa640 1 -- 172.21.15.89:0/4012783335 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0b740664f0 msgr2=0x7f0b740689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.348+0000 7f0b897fa640 1 --2- 172.21.15.89:0/4012783335 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0b740664f0 0x7f0b740689b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0b9414f300 tx=0x7f0b7c009040 comp rx=0 tx=0).stop 2023-10-30T12:28:59.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.348+0000 7f0b897fa640 1 -- 172.21.15.89:0/4012783335 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b94069e90 msgr2=0x7f0b941f6fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.348+0000 7f0b897fa640 1 --2- 172.21.15.89:0/4012783335 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b94069e90 0x7f0b941f6fa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0b8400e870 tx=0x7f0b8400ec70 comp rx=0 tx=0).stop 2023-10-30T12:28:59.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.348+0000 7f0b9359e640 1 -- 172.21.15.89:0/4012783335 reap_dead start 2023-10-30T12:28:59.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.348+0000 7f0b897fa640 1 -- 172.21.15.89:0/4012783335 shutdown_connections 2023-10-30T12:28:59.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.348+0000 7f0b897fa640 1 -- 172.21.15.89:0/4012783335 >> 172.21.15.89:0/4012783335 conn(0x7f0b940c75d0 msgr2=0x7f0b940c9810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:59.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.348+0000 7f0b897fa640 1 -- 172.21.15.89:0/4012783335 shutdown_connections 2023-10-30T12:28:59.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.348+0000 7f0b897fa640 1 -- 172.21.15.89:0/4012783335 wait complete. 2023-10-30T12:28:59.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.352+0000 7f622cff9640 1 -- 172.21.15.89:0/1030450109 --> [v2:172.21.15.192:6824/3707671287,v1:172.21.15.192:6825/3707671287] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f62380ca620 con 0x7f621c076f50 2023-10-30T12:28:59.449 INFO:teuthology.orchestra.run.smithi089.stdout:60129542152 2023-10-30T12:28:59.450 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-30T12:28:59.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.352+0000 7f622effd640 1 -- 172.21.15.89:0/1030450109 <== osd.6 v2:172.21.15.192:6824/3707671287 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f621c079a20 con 0x7f621c076f50 2023-10-30T12:28:59.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.360+0000 7f622cff9640 1 -- 172.21.15.89:0/1030450109 >> [v2:172.21.15.192:6824/3707671287,v1:172.21.15.192:6825/3707671287] conn(0x7f621c076f50 msgr2=0x7f621c079370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.360+0000 7f622cff9640 1 --2- 172.21.15.89:0/1030450109 >> [v2:172.21.15.192:6824/3707671287,v1:172.21.15.192:6825/3707671287] conn(0x7f621c076f50 0x7f621c079370 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.360+0000 7f6fff054640 1 -- 172.21.15.89:0/3518626517 --> [v2:172.21.15.192:6816/3111945871,v1:172.21.15.192:6817/3111945871] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6ff80c5f50 con 0x7f6fd806e6c0 2023-10-30T12:28:59.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.360+0000 7f622cff9640 1 -- 172.21.15.89:0/1030450109 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f621c06edb0 msgr2=0x7f621c071270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.360+0000 7f622cff9640 1 --2- 172.21.15.89:0/1030450109 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f621c06edb0 0x7f621c071270 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f62280af000 tx=0x7f62280a7070 comp rx=0 tx=0).stop 2023-10-30T12:28:59.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.360+0000 7f622cff9640 1 -- 172.21.15.89:0/1030450109 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62380700c0 msgr2=0x7f6238151490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.360+0000 7f622cff9640 1 --2- 172.21.15.89:0/1030450109 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62380700c0 0x7f6238151490 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6234012fc0 tx=0x7f6234010490 comp rx=0 tx=0).stop 2023-10-30T12:28:59.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.360+0000 7f623ec73640 1 -- 172.21.15.89:0/1030450109 reap_dead start 2023-10-30T12:28:59.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.360+0000 7f622cff9640 1 -- 172.21.15.89:0/1030450109 shutdown_connections 2023-10-30T12:28:59.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.360+0000 7f622cff9640 1 -- 172.21.15.89:0/1030450109 >> 172.21.15.89:0/1030450109 conn(0x7f62380c75d0 msgr2=0x7f623806ca90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:59.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.360+0000 7f622cff9640 1 -- 172.21.15.89:0/1030450109 shutdown_connections 2023-10-30T12:28:59.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.360+0000 7f622cff9640 1 -- 172.21.15.89:0/1030450109 wait complete. 2023-10-30T12:28:59.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.360+0000 7f6ff57fa640 1 -- 172.21.15.89:0/3518626517 <== osd.7 v2:172.21.15.192:6816/3111945871 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6fe00a89b0 con 0x7f6fd806e6c0 2023-10-30T12:28:59.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.364+0000 7f6fff054640 1 -- 172.21.15.89:0/3518626517 >> [v2:172.21.15.192:6816/3111945871,v1:172.21.15.192:6817/3111945871] conn(0x7f6fd806e6c0 msgr2=0x7f6fd8070ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.364+0000 7f6fff054640 1 --2- 172.21.15.89:0/3518626517 >> [v2:172.21.15.192:6816/3111945871,v1:172.21.15.192:6817/3111945871] conn(0x7f6fd806e6c0 0x7f6fd8070ae0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.368+0000 7f6fff054640 1 -- 172.21.15.89:0/3518626517 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6fd80664a0 msgr2=0x7f6fd8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.368+0000 7f6fff054640 1 --2- 172.21.15.89:0/3518626517 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6fd80664a0 0x7f6fd8068960 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6ff815ac00 tx=0x7f6fe8008040 comp rx=0 tx=0).stop 2023-10-30T12:28:59.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.368+0000 7f6fff054640 1 -- 172.21.15.89:0/3518626517 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ff8069e90 msgr2=0x7f6ff8145420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.368+0000 7f6fff054640 1 --2- 172.21.15.89:0/3518626517 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ff8069e90 0x7f6ff8145420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6ff000ba50 tx=0x7f6ff000bf10 comp rx=0 tx=0).stop 2023-10-30T12:28:59.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.368+0000 7f6ffd5f1640 1 -- 172.21.15.89:0/3518626517 reap_dead start 2023-10-30T12:28:59.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.368+0000 7f6fff054640 1 -- 172.21.15.89:0/3518626517 shutdown_connections 2023-10-30T12:28:59.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.368+0000 7f6fff054640 1 -- 172.21.15.89:0/3518626517 >> 172.21.15.89:0/3518626517 conn(0x7f6ff80c7910 msgr2=0x7f6ff80c0830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:59.457 INFO:teuthology.orchestra.run.smithi089.stdout:60129542152 2023-10-30T12:28:59.458 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-30T12:28:59.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.372+0000 7f6fff054640 1 -- 172.21.15.89:0/3518626517 shutdown_connections 2023-10-30T12:28:59.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.372+0000 7f6fff054640 1 -- 172.21.15.89:0/3518626517 wait complete. 2023-10-30T12:28:59.459 INFO:teuthology.orchestra.run.smithi089.stdout:60129542152 2023-10-30T12:28:59.460 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-30T12:28:59.460 INFO:teuthology.orchestra.run.smithi089.stdout:60129542151 2023-10-30T12:28:59.461 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-30T12:28:59.462 INFO:teuthology.orchestra.run.smithi089.stdout:60129542152 2023-10-30T12:28:59.462 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-30T12:28:59.463 INFO:teuthology.orchestra.run.smithi089.stdout:60129542152 2023-10-30T12:28:59.463 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-30T12:28:59.464 INFO:teuthology.orchestra.run.smithi089.stdout:60129542152 2023-10-30T12:28:59.464 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-30T12:28:59.465 INFO:teuthology.orchestra.run.smithi089.stdout:60129542152 2023-10-30T12:28:59.466 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-30T12:28:59.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.732+0000 7f35a1217640 1 Processor -- start 2023-10-30T12:28:59.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.732+0000 7f35a1217640 1 -- start start 2023-10-30T12:28:59.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.732+0000 7f35a1217640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f359c0b24b0 0x7f359c0b3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.732+0000 7f35a1217640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f359c0b2a00 con 0x7f359c0c8fa0 2023-10-30T12:28:59.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.732+0000 7f35a1217640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f359c0c9380 con 0x7f359c159890 2023-10-30T12:28:59.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.732+0000 7f35a1217640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f359c159db0 con 0x7f359c0b24b0 2023-10-30T12:28:59.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.732+0000 7f359ad9d640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f359c0c8fa0 0x7f359c15d7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:52066/0 (socket says 172.21.15.89:52066) 2023-10-30T12:28:59.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.732+0000 7f359ad9d640 1 -- 172.21.15.89:0/3000333962 learned_addr learned my addr 172.21.15.89:0/3000333962 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:28:59.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.736+0000 7f359b59e640 1 --2- 172.21.15.89:0/3000333962 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f359c0b24b0 0x7f359c0b3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.736+0000 7f3599d9b640 1 -- 172.21.15.89:0/3000333962 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1542538625 0 0) 0x7f359c0b2a00 con 0x7f359c0c8fa0 2023-10-30T12:28:59.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.736+0000 7f3599d9b640 1 -- 172.21.15.89:0/3000333962 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3584003530 con 0x7f359c0c8fa0 2023-10-30T12:28:59.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.736+0000 7f3599d9b640 1 -- 172.21.15.89:0/3000333962 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2427895996 0 0) 0x7f3584003530 con 0x7f359c0c8fa0 2023-10-30T12:28:59.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.736+0000 7f3599d9b640 1 -- 172.21.15.89:0/3000333962 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f359c0b24b0 msgr2=0x7f359c0b3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:59.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.736+0000 7f3599d9b640 1 --2- 172.21.15.89:0/3000333962 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f359c0b24b0 0x7f359c0b3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.736+0000 7f3599d9b640 1 -- 172.21.15.89:0/3000333962 >> v1:172.21.15.192:6789/0 conn(0x7f359c159890 legacy=0x7f359c0c9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:59.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.736+0000 7f3599d9b640 1 -- 172.21.15.89:0/3000333962 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359c15a090 con 0x7f359c0c8fa0 2023-10-30T12:28:59.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.736+0000 7f3599d9b640 1 -- 172.21.15.89:0/3000333962 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f358c001680 con 0x7f359c0c8fa0 2023-10-30T12:28:59.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.736+0000 7f3599d9b640 1 -- 172.21.15.89:0/3000333962 >> v1:172.21.15.89:6789/0 conn(0x7f359c0c8fa0 legacy=0x7f359c15d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.736+0000 7f3599d9b640 1 --2- 172.21.15.89:0/3000333962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35840039b0 0x7f3584003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.736+0000 7f3599d9b640 1 --2- 172.21.15.89:0/3000333962 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35840047d0 0x7f3584004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.736+0000 7f3599d9b640 1 --2- 172.21.15.89:0/3000333962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3584005190 0x7f358400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.736+0000 7f3599d9b640 1 -- 172.21.15.89:0/3000333962 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359c159db0 con 0x7f3584005190 2023-10-30T12:28:59.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.736+0000 7f3599d9b640 1 -- 172.21.15.89:0/3000333962 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358400bfe0 con 0x7f35840039b0 2023-10-30T12:28:59.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.736+0000 7f3599d9b640 1 -- 172.21.15.89:0/3000333962 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f358400c1a0 con 0x7f35840047d0 2023-10-30T12:28:59.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.736+0000 7f359ad9d640 1 --2- 172.21.15.89:0/3000333962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35840039b0 0x7f3584003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.736+0000 7f359b59e640 1 --2- 172.21.15.89:0/3000333962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3584005190 0x7f358400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.736+0000 7f359ad9d640 1 -- 172.21.15.89:0/3000333962 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35840047d0 msgr2=0x7f3584004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:59.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.736+0000 7f359a59c640 1 --2- 172.21.15.89:0/3000333962 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35840047d0 0x7f3584004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.736+0000 7f359ad9d640 1 --2- 172.21.15.89:0/3000333962 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35840047d0 0x7f3584004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.736+0000 7f359ad9d640 1 -- 172.21.15.89:0/3000333962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3584005190 msgr2=0x7f358400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:59.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.736+0000 7f359ad9d640 1 --2- 172.21.15.89:0/3000333962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3584005190 0x7f358400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.736+0000 7f359ad9d640 1 -- 172.21.15.89:0/3000333962 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359c15a090 con 0x7f35840039b0 2023-10-30T12:28:59.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.736+0000 7f359a59c640 1 --2- 172.21.15.89:0/3000333962 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35840047d0 0x7f3584004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:28:59.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.740+0000 7f2179874640 1 Processor -- start 2023-10-30T12:28:59.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.740+0000 7f2179874640 1 -- start start 2023-10-30T12:28:59.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.740+0000 7f2179874640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f216c0e24a0 0x7f216c04f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.740+0000 7f2179874640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f216c0495b0 0x7f216c0499d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.740+0000 7f2179874640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f216c050d20 con 0x7f216c0e2880 2023-10-30T12:28:59.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.740+0000 7f2179874640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f216c051000 con 0x7f216c0e24a0 2023-10-30T12:28:59.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.740+0000 7f2179874640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f216c006e90 con 0x7f216c0495b0 2023-10-30T12:28:59.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.740+0000 7f2178872640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f216c0e24a0 0x7f216c04f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.740+0000 7f2173fff640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f216c0495b0 0x7f216c0499d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.740+0000 7f2178872640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f216c0e24a0 0x7f216c04f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38964/0 (socket says 172.21.15.89:38964) 2023-10-30T12:28:59.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.740+0000 7f2179073640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f216c0e2880 0x7f216c0e2cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:36038/0 (socket says 172.21.15.89:36038) 2023-10-30T12:28:59.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.740+0000 7f2178872640 1 -- 172.21.15.89:0/3388089146 learned_addr learned my addr 172.21.15.89:0/3388089146 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:59.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.740+0000 7fd64ddf2640 1 Processor -- start 2023-10-30T12:28:59.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.740+0000 7f2173fff640 1 -- 172.21.15.89:0/3388089146 >> v2:172.21.15.89:3300/0 conn(0x7f216c0e24a0 msgr2=0x7f216c04f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:59.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.740+0000 7f2173fff640 1 --2- 172.21.15.89:0/3388089146 >> v2:172.21.15.89:3300/0 conn(0x7f216c0e24a0 0x7f216c04f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.740+0000 7f2173fff640 1 -- 172.21.15.89:0/3388089146 >> v1:172.21.15.192:6789/0 conn(0x7f216c0e2880 legacy=0x7f216c0e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.744+0000 7f21737fe640 1 -- 172.21.15.89:0/3388089146 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839137105 0 0) 0x7f216c050d20 con 0x7f216c0e2880 2023-10-30T12:28:59.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.744+0000 7f2173fff640 1 -- 172.21.15.89:0/3388089146 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f216c007170 con 0x7f216c0495b0 2023-10-30T12:28:59.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.744+0000 7fd64ddf2640 1 -- start start 2023-10-30T12:28:59.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.744+0000 7fd64ddf2640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd6481598f0 0x7fd64814fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.744+0000 7f359ad9d640 1 --2- 172.21.15.89:0/3000333962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35840039b0 0x7f3584003db0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f358c00c370 tx=0x7f358c01cda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=367e736bf59e5c6a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.744+0000 7fd64ddf2640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd6480c6f00 0x7fd6480c2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.744+0000 7fd64cdf0640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd6481598f0 0x7fd64814fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.744+0000 7fd64cdf0640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd6481598f0 0x7fd64814fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:59246/0 (socket says 172.21.15.89:59246) 2023-10-30T12:28:59.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.744+0000 7fd64cdf0640 1 -- 172.21.15.89:0/3153978992 learned_addr learned my addr 172.21.15.89:0/3153978992 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:59.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.744+0000 7fd64ddf2640 1 -- 172.21.15.89:0/3153978992 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6480c2e90 con 0x7fd648155700 2023-10-30T12:28:59.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.744+0000 7fd647fff640 1 --2- 172.21.15.89:0/3153978992 >> v2:172.21.15.89:3300/0 conn(0x7fd6480c6f00 0x7fd6480c2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.744+0000 7fd64ddf2640 1 -- 172.21.15.89:0/3153978992 --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd648156700 con 0x7fd6480c6f00 2023-10-30T12:28:59.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.744+0000 7fd64ddf2640 1 -- 172.21.15.89:0/3153978992 --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd64806f600 con 0x7fd6481598f0 2023-10-30T12:28:59.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7fd6477fe640 1 -- 172.21.15.89:0/3153978992 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3279950225 0 0) 0x7fd6480c2e90 con 0x7fd648155700 2023-10-30T12:28:59.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7fd6477fe640 1 -- 172.21.15.89:0/3153978992 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd630003530 con 0x7fd648155700 2023-10-30T12:28:59.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7f3599d9b640 1 -- 172.21.15.89:0/3000333962 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c0a5ca0 con 0x7f35840039b0 2023-10-30T12:28:59.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7f3599d9b640 1 -- 172.21.15.89:0/3000333962 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f358c0a5e00 con 0x7f35840039b0 2023-10-30T12:28:59.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7f2173fff640 1 --2- 172.21.15.89:0/3388089146 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f216c0495b0 0x7f216c0499d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f216c012200 tx=0x7f216001ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae80f6e76fa2959b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7fd647fff640 1 -- 172.21.15.89:0/3153978992 >> v2:172.21.15.192:3300/0 conn(0x7fd6481598f0 msgr2=0x7fd64814fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:59.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7fd647fff640 1 --2- 172.21.15.89:0/3153978992 >> v2:172.21.15.192:3300/0 conn(0x7fd6481598f0 0x7fd64814fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7fd647fff640 1 -- 172.21.15.89:0/3153978992 >> v1:172.21.15.89:6789/0 conn(0x7fd648155700 legacy=0x7fd64806cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7fd6477fe640 1 -- 172.21.15.89:0/3153978992 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1813530771 0 0) 0x7fd630003530 con 0x7fd648155700 2023-10-30T12:28:59.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7fd647fff640 1 -- 172.21.15.89:0/3153978992 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd648156140 con 0x7fd6480c6f00 2023-10-30T12:28:59.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7f3599d9b640 1 -- 172.21.15.89:0/3000333962 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c0a5ca0 con 0x7f35840039b0 2023-10-30T12:28:59.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7f21737fe640 1 -- 172.21.15.89:0/3388089146 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21600a8950 con 0x7f216c0495b0 2023-10-30T12:28:59.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7f21737fe640 1 -- 172.21.15.89:0/3388089146 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21600a8ab0 con 0x7f216c0495b0 2023-10-30T12:28:59.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7f21737fe640 1 -- 172.21.15.89:0/3388089146 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21600a8e40 con 0x7f216c0495b0 2023-10-30T12:28:59.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7f35a1217640 1 -- 172.21.15.89:0/3000333962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35840039b0 msgr2=0x7f3584003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7f35a1217640 1 --2- 172.21.15.89:0/3000333962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35840039b0 0x7f3584003db0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f358c00c370 tx=0x7f358c01cda0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7f359b59e640 1 -- 172.21.15.89:0/3000333962 reap_dead start 2023-10-30T12:28:59.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7f35a1217640 1 -- 172.21.15.89:0/3000333962 shutdown_connections 2023-10-30T12:28:59.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7f35a1217640 1 -- 172.21.15.89:0/3000333962 >> 172.21.15.89:0/3000333962 conn(0x7f359c06bda0 msgr2=0x7f359c06c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:59.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7f06d0fc4640 1 Processor -- start 2023-10-30T12:28:59.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7f35a1217640 1 -- 172.21.15.89:0/3000333962 shutdown_connections 2023-10-30T12:28:59.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7f06d0fc4640 1 -- start start 2023-10-30T12:28:59.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7f2179874640 1 -- 172.21.15.89:0/3388089146 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f216c0495b0 msgr2=0x7f216c0499d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7f2179874640 1 --2- 172.21.15.89:0/3388089146 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f216c0495b0 0x7f216c0499d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f216c012200 tx=0x7f216001ec50 comp rx=0 tx=0).stop 2023-10-30T12:28:59.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7f2179874640 1 -- 172.21.15.89:0/3388089146 shutdown_connections 2023-10-30T12:28:59.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7f2179874640 1 --2- 172.21.15.89:0/3388089146 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f216c0495b0 0x7f216c0499d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7f2179874640 1 --2- 172.21.15.89:0/3388089146 >> v2:172.21.15.89:3300/0 conn(0x7f216c0e24a0 0x7f216c04f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7f2179874640 1 -- 172.21.15.89:0/3388089146 >> 172.21.15.89:0/3388089146 conn(0x7f216c054710 msgr2=0x7f216c054b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:59.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7f2179874640 1 -- 172.21.15.89:0/3388089146 shutdown_connections 2023-10-30T12:28:59.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7f2179874640 1 -- 172.21.15.89:0/3388089146 wait complete. 2023-10-30T12:28:59.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.748+0000 7f35a1217640 1 -- 172.21.15.89:0/3000333962 wait complete. 2023-10-30T12:28:59.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.752+0000 7f06d0fc4640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f06cc070840 0x7f06cc070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.752+0000 7f06d0fc4640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06cc069e90 0x7f06cc06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.752+0000 7f06d0fc4640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06cc0c6880 con 0x7f06cc153810 2023-10-30T12:28:59.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.752+0000 7f06d0fc4640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06cc0711f0 con 0x7f06cc070840 2023-10-30T12:28:59.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.752+0000 7f06d0fc4640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06cc0c5990 con 0x7f06cc069e90 2023-10-30T12:28:59.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.752+0000 7f06cad9d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06cc069e90 0x7f06cc06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.752+0000 7f35a1217640 1 Processor -- start 2023-10-30T12:28:59.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.752+0000 7f06cad9d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06cc069e90 0x7f06cc06a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:57212/0 (socket says 172.21.15.89:57212) 2023-10-30T12:28:59.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.752+0000 7f06cad9d640 1 -- 172.21.15.89:0/1497772438 learned_addr learned my addr 172.21.15.89:0/1497772438 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:59.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.752+0000 7f35a1217640 1 -- start start 2023-10-30T12:28:59.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.752+0000 7f06c9d9b640 1 --2- 172.21.15.89:0/1497772438 >> v2:172.21.15.89:3300/0 conn(0x7f06cc070840 0x7f06cc070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.752+0000 7f06cad9d640 1 -- 172.21.15.89:0/1497772438 >> v2:172.21.15.89:3300/0 conn(0x7f06cc070840 msgr2=0x7f06cc070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:59.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.752+0000 7f06cad9d640 1 --2- 172.21.15.89:0/1497772438 >> v2:172.21.15.89:3300/0 conn(0x7f06cc070840 0x7f06cc070cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.752+0000 7f06cad9d640 1 -- 172.21.15.89:0/1497772438 >> v1:172.21.15.192:6789/0 conn(0x7f06cc153810 legacy=0x7f06cc15a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.752+0000 7f06cad9d640 1 -- 172.21.15.89:0/1497772438 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06cc0c5c70 con 0x7f06cc069e90 2023-10-30T12:28:59.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.752+0000 7fd647fff640 1 --2- 172.21.15.89:0/3153978992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6480c6f00 0x7fd6480c2950 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd6480c87e0 tx=0x7fd64001f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f30dd09953c0a452 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.752+0000 7f2179874640 1 Processor -- start 2023-10-30T12:28:59.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.752+0000 7fd6477fe640 1 -- 172.21.15.89:0/3153978992 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6400b0440 con 0x7fd6480c6f00 2023-10-30T12:28:59.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.752+0000 7f2179874640 1 -- start start 2023-10-30T12:28:59.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.752+0000 7f983359e640 1 Processor -- start 2023-10-30T12:28:59.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.752+0000 7f35a1217640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35840039b0 0x7f359c156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.752+0000 7f35a1217640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f359c0cf6b0 0x7f359c15c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.752+0000 7f983359e640 1 -- start start 2023-10-30T12:28:59.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.752+0000 7f73807a1640 1 Processor -- start 2023-10-30T12:28:59.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.752+0000 7f73807a1640 1 -- start start 2023-10-30T12:28:59.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.752+0000 7f35a1217640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f359c14cf60 0x7f359c14d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.752+0000 7f359a59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f359c0cf6b0 0x7f359c15c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f359b59e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f359c14cf60 0x7f359c14d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f359a59c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f359c0cf6b0 0x7f359c15c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:59258/0 (socket says 172.21.15.89:59258) 2023-10-30T12:28:59.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f359ad9d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35840039b0 0x7f359c156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f359ad9d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35840039b0 0x7f359c156bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:57220/0 (socket says 172.21.15.89:57220) 2023-10-30T12:28:59.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.752+0000 7fd6477fe640 1 -- 172.21.15.89:0/3153978992 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6400b05a0 con 0x7fd6480c6f00 2023-10-30T12:28:59.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.752+0000 7fd6477fe640 1 -- 172.21.15.89:0/3153978992 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6400b0930 con 0x7fd6480c6f00 2023-10-30T12:28:59.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f359b59e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f359c14cf60 0x7f359c14d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38998/0 (socket says 172.21.15.89:38998) 2023-10-30T12:28:59.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f359a59c640 1 -- 172.21.15.89:0/1681253381 learned_addr learned my addr 172.21.15.89:0/1681253381 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:59.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f35a1217640 1 -- 172.21.15.89:0/1681253381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359c1512b0 con 0x7f359c14cf60 2023-10-30T12:28:59.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f35a1217640 1 -- 172.21.15.89:0/1681253381 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359c14d920 con 0x7f359c0cf6b0 2023-10-30T12:28:59.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f35a1217640 1 -- 172.21.15.89:0/1681253381 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f359c1ef9b0 con 0x7f35840039b0 2023-10-30T12:28:59.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f2179874640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f216c0495b0 0x7f216c04d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f2179874640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f216c0e24a0 0x7f216c17c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f2179874640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f216c0e2880 0x7f216c04c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f2179874640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216c04ec00 con 0x7f216c0495b0 2023-10-30T12:28:59.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f2179874640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216c17be80 con 0x7f216c0e24a0 2023-10-30T12:28:59.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f2179874640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f216c04cf10 con 0x7f216c0e2880 2023-10-30T12:28:59.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7fd64ddf2640 1 -- 172.21.15.89:0/3153978992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6480c6f00 msgr2=0x7fd6480c2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7fd64ddf2640 1 --2- 172.21.15.89:0/3153978992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6480c6f00 0x7fd6480c2950 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd6480c87e0 tx=0x7fd64001f0f0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7fd64ddf2640 1 -- 172.21.15.89:0/3153978992 shutdown_connections 2023-10-30T12:28:59.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7fd64ddf2640 1 --2- 172.21.15.89:0/3153978992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6480c6f00 0x7fd6480c2950 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7fd64ddf2640 1 --2- 172.21.15.89:0/3153978992 >> v2:172.21.15.192:3300/0 conn(0x7fd6481598f0 0x7fd64814fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7fd64ddf2640 1 -- 172.21.15.89:0/3153978992 >> 172.21.15.89:0/3153978992 conn(0x7fd6480cae10 msgr2=0x7fd6480cb220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:59.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f983359e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f98341598f0 0x7f98340c2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f983359e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f98340c6f00 0x7f983414f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f983359e640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98340c2e90 con 0x7f9834155700 2023-10-30T12:28:59.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f983359e640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9834156700 con 0x7f98341598f0 2023-10-30T12:28:59.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f983359e640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9834156140 con 0x7f98340c6f00 2023-10-30T12:28:59.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f2179073640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f216c0e2880 0x7f216c04c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f2179073640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f216c0e2880 0x7f216c04c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:57232/0 (socket says 172.21.15.89:57232) 2023-10-30T12:28:59.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f359b59e640 1 -- 172.21.15.89:0/1681253381 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35840039b0 msgr2=0x7f359c156bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f359b59e640 1 --2- 172.21.15.89:0/1681253381 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35840039b0 0x7f359c156bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f359b59e640 1 -- 172.21.15.89:0/1681253381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f359c0cf6b0 msgr2=0x7f359c15c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f359b59e640 1 --2- 172.21.15.89:0/1681253381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f359c0cf6b0 0x7f359c15c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f359b59e640 1 -- 172.21.15.89:0/1681253381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359c0ce180 con 0x7f359c14cf60 2023-10-30T12:28:59.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7fd64ddf2640 1 -- 172.21.15.89:0/3153978992 shutdown_connections 2023-10-30T12:28:59.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7fd64ddf2640 1 -- 172.21.15.89:0/3153978992 wait complete. 2023-10-30T12:28:59.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f2178872640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f216c0495b0 0x7f216c04d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f2178872640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f216c0495b0 0x7f216c04d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39010/0 (socket says 172.21.15.89:39010) 2023-10-30T12:28:59.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f2178872640 1 -- 172.21.15.89:0/2346991954 learned_addr learned my addr 172.21.15.89:0/2346991954 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:59.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f06c9d9b640 1 --2- 172.21.15.89:0/1497772438 >> v2:172.21.15.89:3300/0 conn(0x7f06cc070840 0x7f06cc070cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:28:59.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f9832d9d640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9834155700 0x7f983406cec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:52086/0 (socket says 172.21.15.89:52086) 2023-10-30T12:28:59.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f2173fff640 1 --2- 172.21.15.89:0/2346991954 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f216c0e24a0 0x7f216c17c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f359ad9d640 1 --2- 172.21.15.89:0/1681253381 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35840039b0 0x7f359c156bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:59.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f9829d9b640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f98340c6f00 0x7f983414f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f9832d9d640 1 -- 172.21.15.89:0/4131563235 learned_addr learned my addr 172.21.15.89:0/4131563235 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:28:59.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f73807a1640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f73781599d0 0x7f73780bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f73807a1640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f73780c0ce0 0x7f7378156290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f73807a1640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73781567d0 con 0x7f737806f540 2023-10-30T12:28:59.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f73807a1640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73781595b0 con 0x7f73780c0ce0 2023-10-30T12:28:59.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f73807a1640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73780c1150 con 0x7f73781599d0 2023-10-30T12:28:59.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f983259c640 1 --2- 172.21.15.89:0/4131563235 >> v2:172.21.15.89:3300/0 conn(0x7f98341598f0 0x7f98340c2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f737ed3e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f73780c0ce0 0x7f7378156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f737ed3e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f73780c0ce0 0x7f7378156290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39024/0 (socket says 172.21.15.89:39024) 2023-10-30T12:28:59.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f737ed3e640 1 -- 172.21.15.89:0/3872378266 learned_addr learned my addr 172.21.15.89:0/3872378266 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:59.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.752+0000 7f41b6503640 1 Processor -- start 2023-10-30T12:28:59.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f41b6503640 1 -- start start 2023-10-30T12:28:59.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f41b6503640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41a80dd300 0x7f41a804ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f41b5501640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41a80dd300 0x7f41a804ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f41b5501640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41a80dd300 0x7f41a804ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:57238/0 (socket says 172.21.15.89:57238) 2023-10-30T12:28:59.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f41b5501640 1 -- 172.21.15.89:0/903622477 learned_addr learned my addr 172.21.15.89:0/903622477 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:59.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f41b6503640 1 --2- 172.21.15.89:0/903622477 >> v2:172.21.15.89:3300/0 conn(0x7f41a80e3780 0x7f41a804b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f2178872640 1 -- 172.21.15.89:0/2346991954 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f216c0e2880 msgr2=0x7f216c04c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f2178872640 1 --2- 172.21.15.89:0/2346991954 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f216c0e2880 0x7f216c04c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f2178872640 1 -- 172.21.15.89:0/2346991954 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f216c0e24a0 msgr2=0x7f216c17c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f2178872640 1 --2- 172.21.15.89:0/2346991954 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f216c0e24a0 0x7f216c17c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f2178872640 1 -- 172.21.15.89:0/2346991954 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21600093b0 con 0x7f216c0495b0 2023-10-30T12:28:59.796 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f2173fff640 1 --2- 172.21.15.89:0/2346991954 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f216c0e24a0 0x7f216c17c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:59.796 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f41b6503640 1 -- 172.21.15.89:0/903622477 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a8035c10 con 0x7f41a804c580 2023-10-30T12:28:59.796 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f737dd3c640 1 --2- 172.21.15.89:0/3872378266 >> v2:172.21.15.192:3300/0 conn(0x7f73781599d0 0x7f73780bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.797 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f41b6503640 1 -- 172.21.15.89:0/903622477 --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41a80e3c50 con 0x7f41a80e3780 2023-10-30T12:28:59.797 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.756+0000 7f06cad9d640 1 --2- 172.21.15.89:0/1497772438 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06cc069e90 0x7f06cc06a2e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f06c4009e50 tx=0x7f06c401f230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aefca8155f0e5766 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.798 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f9829d9b640 1 -- 172.21.15.89:0/4131563235 >> v2:172.21.15.89:3300/0 conn(0x7f98341598f0 msgr2=0x7f98340c2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:59.798 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f9829d9b640 1 --2- 172.21.15.89:0/4131563235 >> v2:172.21.15.89:3300/0 conn(0x7f98341598f0 0x7f98340c2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.798 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f9829d9b640 1 -- 172.21.15.89:0/4131563235 >> v1:172.21.15.89:6789/0 conn(0x7f9834155700 legacy=0x7f983406cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.798 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f9829d9b640 1 -- 172.21.15.89:0/4131563235 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9834156420 con 0x7f98340c6f00 2023-10-30T12:28:59.799 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f41b6503640 1 -- 172.21.15.89:0/903622477 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41a8045410 con 0x7f41a80dd300 2023-10-30T12:28:59.799 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f2178872640 1 --2- 172.21.15.89:0/2346991954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f216c0495b0 0x7f216c04d340 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f216800e890 tx=0x7f216800ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.800 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f06c959a640 1 -- 172.21.15.89:0/1497772438 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c40af400 con 0x7f06cc069e90 2023-10-30T12:28:59.800 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f06c959a640 1 -- 172.21.15.89:0/1497772438 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c40ab050 con 0x7f06cc069e90 2023-10-30T12:28:59.800 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f06c959a640 1 -- 172.21.15.89:0/1497772438 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c40af7c0 con 0x7f06cc069e90 2023-10-30T12:28:59.801 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f737ed3e640 1 -- 172.21.15.89:0/3872378266 >> v2:172.21.15.192:3300/0 conn(0x7f73781599d0 msgr2=0x7f73780bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:59.801 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f737ed3e640 1 --2- 172.21.15.89:0/3872378266 >> v2:172.21.15.192:3300/0 conn(0x7f73781599d0 0x7f73780bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.802 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f737ed3e640 1 -- 172.21.15.89:0/3872378266 >> v1:172.21.15.89:6789/0 conn(0x7f737806f540 legacy=0x7f7378152c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:59.802 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f737ed3e640 1 -- 172.21.15.89:0/3872378266 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73780c04e0 con 0x7f73780c0ce0 2023-10-30T12:28:59.803 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f41b5501640 1 -- 172.21.15.89:0/903622477 >> v2:172.21.15.89:3300/0 conn(0x7f41a80e3780 msgr2=0x7f41a804b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:59.803 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f41b5501640 1 --2- 172.21.15.89:0/903622477 >> v2:172.21.15.89:3300/0 conn(0x7f41a80e3780 0x7f41a804b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.803 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f41b5501640 1 -- 172.21.15.89:0/903622477 >> v1:172.21.15.89:6789/0 conn(0x7f41a804c580 legacy=0x7f41a80e3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:59.804 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f41b5501640 1 -- 172.21.15.89:0/903622477 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a80e3f10 con 0x7f41a80dd300 2023-10-30T12:28:59.805 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7fd64ddf2640 1 Processor -- start 2023-10-30T12:28:59.805 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7fd64ddf2640 1 -- start start 2023-10-30T12:28:59.806 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f06d0fc4640 1 -- 172.21.15.89:0/1497772438 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06cc069e90 msgr2=0x7f06cc06a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.806 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f06d0fc4640 1 --2- 172.21.15.89:0/1497772438 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06cc069e90 0x7f06cc06a2e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f06c4009e50 tx=0x7f06c401f230 comp rx=0 tx=0).stop 2023-10-30T12:28:59.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f21717fa640 1 -- 172.21.15.89:0/2346991954 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2168009620 con 0x7f216c0495b0 2023-10-30T12:28:59.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f21717fa640 1 -- 172.21.15.89:0/2346991954 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f216800c940 con 0x7f216c0495b0 2023-10-30T12:28:59.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f21717fa640 1 -- 172.21.15.89:0/2346991954 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216800ccf0 con 0x7f216c0495b0 2023-10-30T12:28:59.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f2179874640 1 -- 172.21.15.89:0/2346991954 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f216c04e940 con 0x7f216c0495b0 2023-10-30T12:28:59.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f2179874640 1 -- 172.21.15.89:0/2346991954 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f216c17c0d0 con 0x7f216c0495b0 2023-10-30T12:28:59.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f359b59e640 1 --2- 172.21.15.89:0/1681253381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f359c14cf60 0x7f359c14d3e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3584003980 tx=0x7f35940076c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f06d0fc4640 1 -- 172.21.15.89:0/1497772438 shutdown_connections 2023-10-30T12:28:59.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f06d0fc4640 1 --2- 172.21.15.89:0/1497772438 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06cc069e90 0x7f06cc06a2e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f06d0fc4640 1 --2- 172.21.15.89:0/1497772438 >> v2:172.21.15.89:3300/0 conn(0x7f06cc070840 0x7f06cc070cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f06d0fc4640 1 -- 172.21.15.89:0/1497772438 >> 172.21.15.89:0/1497772438 conn(0x7f06cc0c7910 msgr2=0x7f06cc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:59.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f06d0fc4640 1 -- 172.21.15.89:0/1497772438 shutdown_connections 2023-10-30T12:28:59.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f35837fe640 1 -- 172.21.15.89:0/1681253381 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3594011050 con 0x7f359c14cf60 2023-10-30T12:28:59.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f35a1217640 1 -- 172.21.15.89:0/1681253381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f359c1eeb50 con 0x7f359c14cf60 2023-10-30T12:28:59.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f35a1217640 1 -- 172.21.15.89:0/1681253381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f359c1ef110 con 0x7f359c14cf60 2023-10-30T12:28:59.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f35837fe640 1 -- 172.21.15.89:0/1681253381 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3594012040 con 0x7f359c14cf60 2023-10-30T12:28:59.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7f2179874640 1 -- 172.21.15.89:0/2346991954 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f216c0054a0 con 0x7f216c0495b0 2023-10-30T12:28:59.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.760+0000 7f06d0fc4640 1 -- 172.21.15.89:0/1497772438 wait complete. 2023-10-30T12:28:59.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7f01fdb3e640 1 Processor -- start 2023-10-30T12:28:59.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7fd64ddf2640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6480c6f00 0x7fd648151e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7fd64ddf2640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd648155700 0x7fd6480700b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7fd64ddf2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6481598f0 0x7fd6480705f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7fd64ddf2640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd648151000 con 0x7fd6481598f0 2023-10-30T12:28:59.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7fd64ddf2640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6480718f0 con 0x7fd648155700 2023-10-30T12:28:59.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7fd64ddf2640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd648150680 con 0x7fd6480c6f00 2023-10-30T12:28:59.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7f35837fe640 1 -- 172.21.15.89:0/1681253381 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3594011050 con 0x7f359c14cf60 2023-10-30T12:28:59.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7f01fdb3e640 1 -- start start 2023-10-30T12:28:59.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7fd64cdf0640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6480c6f00 0x7fd648151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7fd64cdf0640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6480c6f00 0x7fd648151e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:57240/0 (socket says 172.21.15.89:57240) 2023-10-30T12:28:59.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7fd64cdf0640 1 -- 172.21.15.89:0/4107909011 learned_addr learned my addr 172.21.15.89:0/4107909011 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:59.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7fd647fff640 1 --2- 172.21.15.89:0/4107909011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd648155700 0x7fd6480700b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7f9829d9b640 1 --2- 172.21.15.89:0/4131563235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98340c6f00 0x7f983414f8e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f983406f7d0 tx=0x7f981c01ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e28ceaa018cf6cc5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7f06d0fc4640 1 Processor -- start 2023-10-30T12:28:59.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7f21717fa640 1 -- 172.21.15.89:0/2346991954 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2168013470 con 0x7f216c0495b0 2023-10-30T12:28:59.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7fd64cdf0640 1 -- 172.21.15.89:0/4107909011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd648155700 msgr2=0x7fd6480700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7fd64cdf0640 1 --2- 172.21.15.89:0/4107909011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd648155700 0x7fd6480700b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7fd64cdf0640 1 -- 172.21.15.89:0/4107909011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6481598f0 msgr2=0x7fd6480705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7fd64cdf0640 1 --2- 172.21.15.89:0/4107909011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6481598f0 0x7fd6480705f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7f35a1217640 1 -- 172.21.15.89:0/1681253381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f357c005260 con 0x7f359c14cf60 2023-10-30T12:28:59.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7f737ed3e640 1 --2- 172.21.15.89:0/3872378266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73780c0ce0 0x7f7378156290 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f73780cbe10 tx=0x7f737001f180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3296bf2ac366fa9f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7fd64cdf0640 1 -- 172.21.15.89:0/4107909011 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd640009a70 con 0x7fd6480c6f00 2023-10-30T12:28:59.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7f06d0fc4640 1 -- start start 2023-10-30T12:28:59.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7f01fdb3e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01f8157f70 0x7f01f8161b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7f9831d9b640 1 -- 172.21.15.89:0/4131563235 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981c0a8950 con 0x7f98340c6f00 2023-10-30T12:28:59.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7f9831d9b640 1 -- 172.21.15.89:0/4131563235 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f981c0a8ab0 con 0x7f98340c6f00 2023-10-30T12:28:59.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7f9831d9b640 1 -- 172.21.15.89:0/4131563235 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981c0a8e40 con 0x7f98340c6f00 2023-10-30T12:28:59.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7f41b5501640 1 --2- 172.21.15.89:0/903622477 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41a80dd300 0x7f41a804ace0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f41ac009870 tx=0x7f41ac01ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f085888fdb7ef27a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7f21717fa640 1 --2- 172.21.15.89:0/2346991954 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f21580664f0 0x7f21580689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7f21717fa640 1 -- 172.21.15.89:0/2346991954 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f21680842f0 con 0x7f216c0495b0 2023-10-30T12:28:59.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7fd64cdf0640 1 --2- 172.21.15.89:0/4107909011 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6480c6f00 0x7fd648151e10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd63800f9e0 tx=0x7fd63800fde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7f41a7fff640 1 -- 172.21.15.89:0/903622477 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ac0a89d0 con 0x7f41a80dd300 2023-10-30T12:28:59.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7fd6457fa640 1 -- 172.21.15.89:0/4107909011 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63800c070 con 0x7fd6480c6f00 2023-10-30T12:28:59.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7fd6457fa640 1 -- 172.21.15.89:0/4107909011 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd63800dc10 con 0x7fd6480c6f00 2023-10-30T12:28:59.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7fd6457fa640 1 -- 172.21.15.89:0/4107909011 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd638009580 con 0x7fd6480c6f00 2023-10-30T12:28:59.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7f2173fff640 1 --2- 172.21.15.89:0/2346991954 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f21580664f0 0x7f21580689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7fd64ddf2640 1 -- 172.21.15.89:0/4107909011 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6481500e0 con 0x7fd6480c6f00 2023-10-30T12:28:59.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7f01f77fe640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01f8157f70 0x7f01f8161b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7f01f77fe640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01f8157f70 0x7f01f8161b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:57250/0 (socket says 172.21.15.89:57250) 2023-10-30T12:28:59.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.764+0000 7f01f77fe640 1 -- 172.21.15.89:0/1578196436 learned_addr learned my addr 172.21.15.89:0/1578196436 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:59.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f01fdb3e640 1 -- 172.21.15.89:0/1578196436 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f81620d0 con 0x7f01f80cad80 2023-10-30T12:28:59.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f01fdb3e640 1 -- 172.21.15.89:0/1578196436 --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f8158350 con 0x7f01f80cf160 2023-10-30T12:28:59.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f01fdb3e640 1 -- 172.21.15.89:0/1578196436 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f80ca8b0 con 0x7f01f8157f70 2023-10-30T12:28:59.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7fd64ddf2640 1 -- 172.21.15.89:0/4107909011 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd648070fc0 con 0x7fd6480c6f00 2023-10-30T12:28:59.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f41a7fff640 1 -- 172.21.15.89:0/903622477 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41ac0a8b30 con 0x7f41a80dd300 2023-10-30T12:28:59.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f41a7fff640 1 -- 172.21.15.89:0/903622477 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ac0b0440 con 0x7f41a80dd300 2023-10-30T12:28:59.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f983359e640 1 -- 172.21.15.89:0/4131563235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98340c6f00 msgr2=0x7f983414f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f983359e640 1 --2- 172.21.15.89:0/4131563235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98340c6f00 0x7f983414f8e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f983406f7d0 tx=0x7f981c01ec50 comp rx=0 tx=0).stop 2023-10-30T12:28:59.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f06d0fc4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06cc069e90 0x7f06cc1500a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f06d0fc4640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06cc070840 0x7f06cc1f6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f41b6503640 1 -- 172.21.15.89:0/903622477 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41a80dd300 msgr2=0x7f41a804ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f06d0fc4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06cc153810 0x7f06cc148cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f06d0fc4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06cc14e980 con 0x7f06cc153810 2023-10-30T12:28:59.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f06d0fc4640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06cc1f74a0 con 0x7f06cc069e90 2023-10-30T12:28:59.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f06d0fc4640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06cc14e6a0 con 0x7f06cc070840 2023-10-30T12:28:59.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f983359e640 1 -- 172.21.15.89:0/4131563235 shutdown_connections 2023-10-30T12:28:59.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f983359e640 1 --2- 172.21.15.89:0/4131563235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98340c6f00 0x7f983414f8e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f983359e640 1 --2- 172.21.15.89:0/4131563235 >> v2:172.21.15.89:3300/0 conn(0x7f98341598f0 0x7f98340c2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f983359e640 1 -- 172.21.15.89:0/4131563235 >> 172.21.15.89:0/4131563235 conn(0x7f98340cae10 msgr2=0x7f98340cb220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:59.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7fd64ddf2640 1 -- 172.21.15.89:0/4107909011 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd64804e170 con 0x7fd6480c6f00 2023-10-30T12:28:59.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f01f77fe640 1 -- 172.21.15.89:0/1578196436 >> v1:172.21.15.192:6789/0 conn(0x7f01f80cf160 legacy=0x7f01f8168510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:59.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f01f77fe640 1 -- 172.21.15.89:0/1578196436 >> v1:172.21.15.89:6789/0 conn(0x7f01f80cad80 legacy=0x7f01f80cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f01f77fe640 1 -- 172.21.15.89:0/1578196436 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f80ca2f0 con 0x7f01f8157f70 2023-10-30T12:28:59.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f35837fe640 1 -- 172.21.15.89:0/1681253381 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3594020460 con 0x7f359c14cf60 2023-10-30T12:28:59.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f35837fe640 1 --2- 172.21.15.89:0/1681253381 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f35880664c0 0x7f3588068980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f35837fe640 1 -- 172.21.15.89:0/1681253381 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35940840e0 con 0x7f359c14cf60 2023-10-30T12:28:59.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f06cad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06cc153810 0x7f06cc148cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f06c9d9b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06cc070840 0x7f06cc1f6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f06cad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06cc153810 0x7f06cc148cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39060/0 (socket says 172.21.15.89:39060) 2023-10-30T12:28:59.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f06cad9d640 1 -- 172.21.15.89:0/2728258506 learned_addr learned my addr 172.21.15.89:0/2728258506 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:59.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f41b6503640 1 --2- 172.21.15.89:0/903622477 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41a80dd300 0x7f41a804ace0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f41ac009870 tx=0x7f41ac01ed10 comp rx=0 tx=0).stop 2023-10-30T12:28:59.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f983359e640 1 -- 172.21.15.89:0/4131563235 shutdown_connections 2023-10-30T12:28:59.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f41b6503640 1 -- 172.21.15.89:0/903622477 shutdown_connections 2023-10-30T12:28:59.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f359ad9d640 1 --2- 172.21.15.89:0/1681253381 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f35880664c0 0x7f3588068980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f41b6503640 1 --2- 172.21.15.89:0/903622477 >> v2:172.21.15.89:3300/0 conn(0x7f41a80e3780 0x7f41a804b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f41b6503640 1 --2- 172.21.15.89:0/903622477 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41a80dd300 0x7f41a804ace0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f41b6503640 1 -- 172.21.15.89:0/903622477 >> 172.21.15.89:0/903622477 conn(0x7f41a80501d0 msgr2=0x7f41a80505e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:59.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f2173fff640 1 --2- 172.21.15.89:0/2346991954 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f21580664f0 0x7f21580689b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f21600b2000 tx=0x7f2160008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f06ca59c640 1 --2- 172.21.15.89:0/2728258506 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06cc069e90 0x7f06cc1500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f06cad9d640 1 -- 172.21.15.89:0/2728258506 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06cc070840 msgr2=0x7f06cc1f6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f06cad9d640 1 --2- 172.21.15.89:0/2728258506 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06cc070840 0x7f06cc1f6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f06cad9d640 1 -- 172.21.15.89:0/2728258506 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06cc069e90 msgr2=0x7f06cc1500a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f06cad9d640 1 --2- 172.21.15.89:0/2728258506 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06cc069e90 0x7f06cc1500a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f06cad9d640 1 -- 172.21.15.89:0/2728258506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c4009b30 con 0x7f06cc153810 2023-10-30T12:28:59.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f06ca59c640 1 --2- 172.21.15.89:0/2728258506 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06cc069e90 0x7f06cc1500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:28:59.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f41b6503640 1 -- 172.21.15.89:0/903622477 shutdown_connections 2023-10-30T12:28:59.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.772+0000 7f21717fa640 1 -- 172.21.15.89:0/2346991954 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2168051800 con 0x7f216c0495b0 2023-10-30T12:28:59.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f359ad9d640 1 --2- 172.21.15.89:0/1681253381 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f35880664c0 0x7f3588068980 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f358c0ab000 tx=0x7f358c0a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7fd6457fa640 1 -- 172.21.15.89:0/4107909011 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd6380059a0 con 0x7fd6480c6f00 2023-10-30T12:28:59.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.772+0000 7fd6457fa640 1 --2- 172.21.15.89:0/4107909011 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd634066570 0x7fd634068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.772+0000 7fd6457fa640 1 -- 172.21.15.89:0/4107909011 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd638015070 con 0x7fd6480c6f00 2023-10-30T12:28:59.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.772+0000 7fd647fff640 1 --2- 172.21.15.89:0/4107909011 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd634066570 0x7fd634068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f06cad9d640 1 --2- 172.21.15.89:0/2728258506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06cc153810 0x7f06cc148cb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f06c401ce60 tx=0x7f06c401bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f983359e640 1 -- 172.21.15.89:0/4131563235 wait complete. 2023-10-30T12:28:59.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.772+0000 7f983359e640 1 Processor -- start 2023-10-30T12:28:59.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.772+0000 7f983359e640 1 -- start start 2023-10-30T12:28:59.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.772+0000 7f737d53b640 1 -- 172.21.15.89:0/3872378266 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73700af400 con 0x7f73780c0ce0 2023-10-30T12:28:59.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.768+0000 7f41b6503640 1 -- 172.21.15.89:0/903622477 wait complete. 2023-10-30T12:28:59.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.772+0000 7f41b6503640 1 Processor -- start 2023-10-30T12:28:59.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.772+0000 7f06baffd640 1 -- 172.21.15.89:0/2728258506 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c40c0070 con 0x7f06cc153810 2023-10-30T12:28:59.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.772+0000 7f06d0fc4640 1 -- 172.21.15.89:0/2728258506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06cc14c230 con 0x7f06cc153810 2023-10-30T12:28:59.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.772+0000 7f06d0fc4640 1 -- 172.21.15.89:0/2728258506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06cc1505e0 con 0x7f06cc153810 2023-10-30T12:28:59.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.772+0000 7f41b6503640 1 -- start start 2023-10-30T12:28:59.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.772+0000 7f06b8ff9640 1 -- 172.21.15.89:0/2728258506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06b0005260 con 0x7f06cc153810 2023-10-30T12:28:59.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.772+0000 7f983359e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98340c6f00 0x7f9834150340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.772+0000 7fd647fff640 1 --2- 172.21.15.89:0/4107909011 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd634066570 0x7fd634068a30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd6400b2000 tx=0x7fd6400aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.772+0000 7f983359e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9834155700 0x7f983413c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.772+0000 7f983359e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98341598f0 0x7f98341514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.772+0000 7f983359e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98341510d0 con 0x7f98341598f0 2023-10-30T12:28:59.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.772+0000 7f983359e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98341f6f90 con 0x7f98340c6f00 2023-10-30T12:28:59.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.772+0000 7f983359e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98341f6410 con 0x7f9834155700 2023-10-30T12:28:59.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.772+0000 7f9832d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98341598f0 0x7f98341514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.776+0000 7f9832d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98341598f0 0x7f98341514f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39072/0 (socket says 172.21.15.89:39072) 2023-10-30T12:28:59.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.776+0000 7f9832d9d640 1 -- 172.21.15.89:0/94908658 learned_addr learned my addr 172.21.15.89:0/94908658 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:59.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.776+0000 7f983259c640 1 --2- 172.21.15.89:0/94908658 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98340c6f00 0x7f9834150340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.776+0000 7f41b6503640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41a804c580 0x7f41a816ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.776+0000 7f9829d9b640 1 --2- 172.21.15.89:0/94908658 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9834155700 0x7f983413c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.855 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.776+0000 7f41b6503640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41a80dd300 0x7f41a804dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.855 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.776+0000 7f41b6503640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41a80e3780 0x7f41a804aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.855 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.776+0000 7f41b6503640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a804e510 con 0x7f41a804c580 2023-10-30T12:28:59.855 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.776+0000 7f41b6503640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a8174100 con 0x7f41a80e3780 2023-10-30T12:28:59.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.776+0000 7f41b6503640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41a8174530 con 0x7f41a80dd300 2023-10-30T12:28:59.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.776+0000 7f41b5501640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41a804c580 0x7f41a816ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.776+0000 7f9832d9d640 1 -- 172.21.15.89:0/94908658 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9834155700 msgr2=0x7f983413c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.776+0000 7f9832d9d640 1 --2- 172.21.15.89:0/94908658 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9834155700 0x7f983413c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.776+0000 7f9832d9d640 1 -- 172.21.15.89:0/94908658 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98340c6f00 msgr2=0x7f9834150340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.776+0000 7f9832d9d640 1 --2- 172.21.15.89:0/94908658 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98340c6f00 0x7f9834150340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.776+0000 7f9832d9d640 1 -- 172.21.15.89:0/94908658 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f981c009420 con 0x7f98341598f0 2023-10-30T12:28:59.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.776+0000 7f41b5d02640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41a80e3780 0x7f41a804aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.776+0000 7f41b5501640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41a804c580 0x7f41a816ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39078/0 (socket says 172.21.15.89:39078) 2023-10-30T12:28:59.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.776+0000 7f41b4d00640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41a80dd300 0x7f41a804dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.776+0000 7f41b5501640 1 -- 172.21.15.89:0/1212470265 learned_addr learned my addr 172.21.15.89:0/1212470265 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:59.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.776+0000 7f01f77fe640 1 --2- 172.21.15.89:0/1578196436 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01f8157f70 0x7f01f8161b90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f01e80098a0 tx=0x7f01e801ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ff585e0e07d45a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.776+0000 7f9832d9d640 1 --2- 172.21.15.89:0/94908658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98341598f0 0x7f98341514f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f982c00c9f0 tx=0x7f982c00ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.776+0000 7f41b5501640 1 -- 172.21.15.89:0/1212470265 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41a80dd300 msgr2=0x7f41a804dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.776+0000 7f41b5501640 1 --2- 172.21.15.89:0/1212470265 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41a80dd300 0x7f41a804dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.776+0000 7f41b5501640 1 -- 172.21.15.89:0/1212470265 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41a80e3780 msgr2=0x7f41a804aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.776+0000 7f41b5501640 1 --2- 172.21.15.89:0/1212470265 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41a80e3780 0x7f41a804aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.776+0000 7f41b5501640 1 -- 172.21.15.89:0/1212470265 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ac0094d0 con 0x7f41a804c580 2023-10-30T12:28:59.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.776+0000 7f41b5501640 1 --2- 172.21.15.89:0/1212470265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41a804c580 0x7f41a816ef80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f41ac01d3c0 tx=0x7f41ac01bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.780+0000 7f737d53b640 1 -- 172.21.15.89:0/3872378266 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73700ab050 con 0x7f73780c0ce0 2023-10-30T12:28:59.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.780+0000 7f982b7fe640 1 -- 172.21.15.89:0/94908658 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982c00ab70 con 0x7f98341598f0 2023-10-30T12:28:59.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.780+0000 7f737d53b640 1 -- 172.21.15.89:0/3872378266 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73700af400 con 0x7f73780c0ce0 2023-10-30T12:28:59.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.780+0000 7f06baffd640 1 -- 172.21.15.89:0/2728258506 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c40099b0 con 0x7f06cc153810 2023-10-30T12:28:59.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.780+0000 7f06baffd640 1 -- 172.21.15.89:0/2728258506 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c40afe00 con 0x7f06cc153810 2023-10-30T12:28:59.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.780+0000 7f983359e640 1 -- 172.21.15.89:0/94908658 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98341f66f0 con 0x7f98341598f0 2023-10-30T12:28:59.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.780+0000 7f983359e640 1 -- 172.21.15.89:0/94908658 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98341f6cb0 con 0x7f98341598f0 2023-10-30T12:28:59.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.780+0000 7f35837fe640 1 -- 172.21.15.89:0/1681253381 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f35940514c0 con 0x7f359c14cf60 2023-10-30T12:28:59.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.780+0000 7f41a5ffb640 1 -- 172.21.15.89:0/1212470265 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ac0bb670 con 0x7f41a804c580 2023-10-30T12:28:59.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.780+0000 7f41b6503640 1 -- 172.21.15.89:0/1212470265 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41a8185880 con 0x7f41a804c580 2023-10-30T12:28:59.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.780+0000 7f41b6503640 1 -- 172.21.15.89:0/1212470265 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41a8172f40 con 0x7f41a804c580 2023-10-30T12:28:59.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.780+0000 7f73807a1640 1 -- 172.21.15.89:0/3872378266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73780c0ce0 msgr2=0x7f7378156290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.780+0000 7f73807a1640 1 --2- 172.21.15.89:0/3872378266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73780c0ce0 0x7f7378156290 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f73780cbe10 tx=0x7f737001f180 comp rx=0 tx=0).stop 2023-10-30T12:28:59.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.780+0000 7f41a5ffb640 1 -- 172.21.15.89:0/1212470265 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41ac0a8970 con 0x7f41a804c580 2023-10-30T12:28:59.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.780+0000 7f41a5ffb640 1 -- 172.21.15.89:0/1212470265 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ac0bb9d0 con 0x7f41a804c580 2023-10-30T12:28:59.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.780+0000 7f9828d99640 1 -- 172.21.15.89:0/94908658 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9834151210 con 0x7f98341598f0 2023-10-30T12:28:59.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.780+0000 7f73807a1640 1 -- 172.21.15.89:0/3872378266 shutdown_connections 2023-10-30T12:28:59.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.780+0000 7f73807a1640 1 --2- 172.21.15.89:0/3872378266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73780c0ce0 0x7f7378156290 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.780+0000 7f73807a1640 1 --2- 172.21.15.89:0/3872378266 >> v2:172.21.15.192:3300/0 conn(0x7f73781599d0 0x7f73780bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.780+0000 7f73807a1640 1 -- 172.21.15.89:0/3872378266 >> 172.21.15.89:0/3872378266 conn(0x7f73780c7910 msgr2=0x7f73780c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:59.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.780+0000 7f73807a1640 1 -- 172.21.15.89:0/3872378266 shutdown_connections 2023-10-30T12:28:59.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.780+0000 7f73807a1640 1 -- 172.21.15.89:0/3872378266 wait complete. 2023-10-30T12:28:59.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.780+0000 7f982b7fe640 1 -- 172.21.15.89:0/94908658 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f982c001ae0 con 0x7f98341598f0 2023-10-30T12:28:59.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.784+0000 7f41b6503640 1 -- 172.21.15.89:0/1212470265 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4190005260 con 0x7f41a804c580 2023-10-30T12:28:59.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.784+0000 7f982b7fe640 1 -- 172.21.15.89:0/94908658 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982c01c440 con 0x7f98341598f0 2023-10-30T12:28:59.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.784+0000 7f01f67fc640 1 -- 172.21.15.89:0/1578196436 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e80a8a10 con 0x7f01f8157f70 2023-10-30T12:28:59.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.784+0000 7f01f67fc640 1 -- 172.21.15.89:0/1578196436 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e80a8b70 con 0x7f01f8157f70 2023-10-30T12:28:59.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.784+0000 7f01f67fc640 1 -- 172.21.15.89:0/1578196436 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e80a8a10 con 0x7f01f8157f70 2023-10-30T12:28:59.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.784+0000 7fd6457fa640 1 -- 172.21.15.89:0/4107909011 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd638051a70 con 0x7fd6480c6f00 2023-10-30T12:28:59.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.784+0000 7f982b7fe640 1 -- 172.21.15.89:0/94908658 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f982c007070 con 0x7f98341598f0 2023-10-30T12:28:59.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.784+0000 7f06baffd640 1 -- 172.21.15.89:0/2728258506 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f06c40b6040 con 0x7f06cc153810 2023-10-30T12:28:59.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.784+0000 7f06baffd640 1 --2- 172.21.15.89:0/2728258506 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f06ac0664a0 0x7f06ac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.784+0000 7f06baffd640 1 -- 172.21.15.89:0/2728258506 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f06c4122010 con 0x7f06cc153810 2023-10-30T12:28:59.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.784+0000 7f73807a1640 1 Processor -- start 2023-10-30T12:28:59.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.784+0000 7f01fdb3e640 1 -- 172.21.15.89:0/1578196436 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01f8157f70 msgr2=0x7f01f8161b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.784+0000 7f01fdb3e640 1 --2- 172.21.15.89:0/1578196436 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01f8157f70 0x7f01f8161b90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f01e80098a0 tx=0x7f01e801ed10 comp rx=0 tx=0).stop 2023-10-30T12:28:59.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.784+0000 7f73807a1640 1 -- start start 2023-10-30T12:28:59.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.784+0000 7f01fdb3e640 1 -- 172.21.15.89:0/1578196436 shutdown_connections 2023-10-30T12:28:59.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.784+0000 7f01fdb3e640 1 --2- 172.21.15.89:0/1578196436 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01f8157f70 0x7f01f8161b90 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.784+0000 7f01fdb3e640 1 -- 172.21.15.89:0/1578196436 >> 172.21.15.89:0/1578196436 conn(0x7f01f806bfe0 msgr2=0x7f01f806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:59.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.784+0000 7f06ca59c640 1 --2- 172.21.15.89:0/2728258506 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f06ac0664a0 0x7f06ac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.784+0000 7f982b7fe640 1 --2- 172.21.15.89:0/94908658 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f98140664a0 0x7f9814068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.784+0000 7f01fdb3e640 1 -- 172.21.15.89:0/1578196436 shutdown_connections 2023-10-30T12:28:59.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.784+0000 7f01fdb3e640 1 -- 172.21.15.89:0/1578196436 wait complete. 2023-10-30T12:28:59.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.784+0000 7f06ca59c640 1 --2- 172.21.15.89:0/2728258506 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f06ac0664a0 0x7f06ac068960 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f06bc000f70 tx=0x7f06bc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.784+0000 7f983259c640 1 --2- 172.21.15.89:0/94908658 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f98140664a0 0x7f9814068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.788+0000 7f73807a1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f737806f540 0x7f737813dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.788+0000 7f73807a1640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73780c0ce0 0x7f7378154d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.788+0000 7f73807a1640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73781599d0 0x7f737813f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.788+0000 7f983259c640 1 --2- 172.21.15.89:0/94908658 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f98140664a0 0x7f9814068960 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9824009b60 tx=0x7f9824009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.788+0000 7f73807a1640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7378153920 con 0x7f737806f540 2023-10-30T12:28:59.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.788+0000 7f73807a1640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7378156620 con 0x7f73780c0ce0 2023-10-30T12:28:59.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.788+0000 7f73807a1640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f737813d580 con 0x7f73781599d0 2023-10-30T12:28:59.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.788+0000 7f737dd3c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73780c0ce0 0x7f7378154d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.788+0000 7f737dd3c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73780c0ce0 0x7f7378154d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:59330/0 (socket says 172.21.15.89:59330) 2023-10-30T12:28:59.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.788+0000 7f737ed3e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73781599d0 0x7f737813f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.788+0000 7f737dd3c640 1 -- 172.21.15.89:0/2308884482 learned_addr learned my addr 172.21.15.89:0/2308884482 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:59.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.788+0000 7f01fdb3e640 1 Processor -- start 2023-10-30T12:28:59.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.788+0000 7f737e53d640 1 --2- 172.21.15.89:0/2308884482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f737806f540 0x7f737813dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.788+0000 7f737ed3e640 1 -- 172.21.15.89:0/2308884482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73780c0ce0 msgr2=0x7f7378154d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.788+0000 7f737ed3e640 1 --2- 172.21.15.89:0/2308884482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73780c0ce0 0x7f7378154d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.788+0000 7f737ed3e640 1 -- 172.21.15.89:0/2308884482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f737806f540 msgr2=0x7f737813dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.788+0000 7f737ed3e640 1 --2- 172.21.15.89:0/2308884482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f737806f540 0x7f737813dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.788+0000 7f737ed3e640 1 -- 172.21.15.89:0/2308884482 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7370009b30 con 0x7f73781599d0 2023-10-30T12:28:59.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.788+0000 7f737ed3e640 1 --2- 172.21.15.89:0/2308884482 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73781599d0 0x7f737813f0e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7378156960 tx=0x7f737001bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.788+0000 7f01fdb3e640 1 -- start start 2023-10-30T12:28:59.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.788+0000 7f736effd640 1 -- 172.21.15.89:0/2308884482 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73700be070 con 0x7f73781599d0 2023-10-30T12:28:59.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.788+0000 7f73807a1640 1 -- 172.21.15.89:0/2308884482 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f737813d860 con 0x7f73781599d0 2023-10-30T12:28:59.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.788+0000 7f73807a1640 1 -- 172.21.15.89:0/2308884482 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73781562b0 con 0x7f73781599d0 2023-10-30T12:28:59.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.788+0000 7f982b7fe640 1 -- 172.21.15.89:0/94908658 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f982c055150 con 0x7f98341598f0 2023-10-30T12:28:59.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.788+0000 7f736effd640 1 -- 172.21.15.89:0/2308884482 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7370004040 con 0x7f73781599d0 2023-10-30T12:28:59.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.788+0000 7f736effd640 1 -- 172.21.15.89:0/2308884482 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73700043f0 con 0x7f73781599d0 2023-10-30T12:28:59.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.792+0000 7f01fdb3e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01f80cad80 0x7f01f8151070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.792+0000 7f01fdb3e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01f80cf160 0x7f01f814fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.792+0000 7f01fdb3e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01f8157f70 0x7f01f81502d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.792+0000 7f01fdb3e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f80ceb50 con 0x7f01f8157f70 2023-10-30T12:28:59.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.792+0000 7f01fdb3e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f80cd530 con 0x7f01f80cf160 2023-10-30T12:28:59.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.792+0000 7f01fdb3e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f81536f0 con 0x7f01f80cad80 2023-10-30T12:28:59.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.792+0000 7f01f7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01f8157f70 0x7f01f81502d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.792+0000 7f01f7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01f8157f70 0x7f01f81502d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39104/0 (socket says 172.21.15.89:39104) 2023-10-30T12:28:59.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.792+0000 7f01f7fff640 1 -- 172.21.15.89:0/3769323033 learned_addr learned my addr 172.21.15.89:0/3769323033 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:28:59.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.792+0000 7f01f6ffd640 1 --2- 172.21.15.89:0/3769323033 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01f80cf160 0x7f01f814fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.792+0000 7f01f77fe640 1 --2- 172.21.15.89:0/3769323033 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01f80cad80 0x7f01f8151070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.792+0000 7f01f7fff640 1 -- 172.21.15.89:0/3769323033 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01f80cad80 msgr2=0x7f01f8151070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.792+0000 7f01f7fff640 1 --2- 172.21.15.89:0/3769323033 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01f80cad80 0x7f01f8151070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.792+0000 7f01f7fff640 1 -- 172.21.15.89:0/3769323033 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01f80cf160 msgr2=0x7f01f814fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.792+0000 7f01f7fff640 1 --2- 172.21.15.89:0/3769323033 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01f80cf160 0x7f01f814fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.792+0000 7f41a5ffb640 1 -- 172.21.15.89:0/1212470265 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f41ac0b7070 con 0x7f41a804c580 2023-10-30T12:28:59.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.792+0000 7f01f7fff640 1 -- 172.21.15.89:0/3769323033 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e80094d0 con 0x7f01f8157f70 2023-10-30T12:28:59.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.792+0000 7f01f6ffd640 1 --2- 172.21.15.89:0/3769323033 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01f80cf160 0x7f01f814fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:59.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.792+0000 7f06baffd640 1 -- 172.21.15.89:0/2728258506 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f06c40ef5c0 con 0x7f06cc153810 2023-10-30T12:28:59.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.792+0000 7f01f7fff640 1 --2- 172.21.15.89:0/3769323033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01f8157f70 0x7f01f81502d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f01f000c780 tx=0x7f01f000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.792+0000 7f736effd640 1 -- 172.21.15.89:0/2308884482 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f73700b6070 con 0x7f73781599d0 2023-10-30T12:28:59.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.792+0000 7f41a5ffb640 1 --2- 172.21.15.89:0/1212470265 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f41940664a0 0x7f4194068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.796+0000 7f41b4d00640 1 --2- 172.21.15.89:0/1212470265 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f41940664a0 0x7f4194068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.796+0000 7f73807a1640 1 -- 172.21.15.89:0/2308884482 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7360005260 con 0x7f73781599d0 2023-10-30T12:28:59.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.796+0000 7f736effd640 1 --2- 172.21.15.89:0/2308884482 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f735c0664d0 0x7f735c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.796+0000 7f736effd640 1 -- 172.21.15.89:0/2308884482 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7370121460 con 0x7f73781599d0 2023-10-30T12:28:59.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.796+0000 7f41b4d00640 1 --2- 172.21.15.89:0/1212470265 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f41940664a0 0x7f4194068960 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f41a8173bc0 tx=0x7f41a0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.796+0000 7f737e53d640 1 --2- 172.21.15.89:0/2308884482 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f735c0664d0 0x7f735c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.796+0000 7f01dffff640 1 -- 172.21.15.89:0/3769323033 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f0004b70 con 0x7f01f8157f70 2023-10-30T12:28:59.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.796+0000 7f737e53d640 1 --2- 172.21.15.89:0/2308884482 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f735c0664d0 0x7f735c068990 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7374009870 tx=0x7f7374008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.796+0000 7f982b7fe640 1 -- 172.21.15.89:0/94908658 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f982c051490 con 0x7f98341598f0 2023-10-30T12:28:59.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.796+0000 7f01fdb3e640 1 -- 172.21.15.89:0/3769323033 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f80c7150 con 0x7f01f8157f70 2023-10-30T12:28:59.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.796+0000 7f01fdb3e640 1 -- 172.21.15.89:0/3769323033 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f80b4cd0 con 0x7f01f8157f70 2023-10-30T12:28:59.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.796+0000 7f41a5ffb640 1 -- 172.21.15.89:0/1212470265 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f41ac0b0e30 con 0x7f41a804c580 2023-10-30T12:28:59.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.800+0000 7f01fdb3e640 1 -- 172.21.15.89:0/3769323033 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01f804e170 con 0x7f01f8157f70 2023-10-30T12:28:59.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.800+0000 7f41a5ffb640 1 -- 172.21.15.89:0/1212470265 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f41ac0effa0 con 0x7f41a804c580 2023-10-30T12:28:59.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.800+0000 7f01dffff640 1 -- 172.21.15.89:0/3769323033 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01f000ce90 con 0x7f01f8157f70 2023-10-30T12:28:59.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.800+0000 7f01dffff640 1 -- 172.21.15.89:0/3769323033 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f0004b70 con 0x7f01f8157f70 2023-10-30T12:28:59.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.804+0000 7f01dffff640 1 -- 172.21.15.89:0/3769323033 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f01f0004cd0 con 0x7f01f8157f70 2023-10-30T12:28:59.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.808+0000 7f01dffff640 1 --2- 172.21.15.89:0/3769323033 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f01e4066500 0x7f01e40689c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.808+0000 7f01dffff640 1 -- 172.21.15.89:0/3769323033 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01f0084180 con 0x7f01f8157f70 2023-10-30T12:28:59.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.808+0000 7f01f77fe640 1 --2- 172.21.15.89:0/3769323033 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f01e4066500 0x7f01e40689c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.808+0000 7f01f77fe640 1 --2- 172.21.15.89:0/3769323033 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f01e4066500 0x7f01e40689c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f01e80b2000 tx=0x7f01e8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.808+0000 7f736effd640 1 -- 172.21.15.89:0/2308884482 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f73700ee8d0 con 0x7f73781599d0 2023-10-30T12:28:59.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:28:59.816+0000 7f01dffff640 1 -- 172.21.15.89:0/3769323033 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f01f0051720 con 0x7f01f8157f70 2023-10-30T12:29:00.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.092+0000 7f2179874640 1 -- 172.21.15.89:0/2346991954 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f216c03a760 con 0x7f216c0495b0 2023-10-30T12:29:00.103 INFO:teuthology.orchestra.run.smithi089.stdout:60129542151 2023-10-30T12:29:00.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.096+0000 7f21717fa640 1 -- 172.21.15.89:0/2346991954 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2168054e20 con 0x7f216c0495b0 2023-10-30T12:29:00.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.100+0000 7f2179874640 1 -- 172.21.15.89:0/2346991954 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f21580664f0 msgr2=0x7f21580689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.100+0000 7f2179874640 1 --2- 172.21.15.89:0/2346991954 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f21580664f0 0x7f21580689b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f21600b2000 tx=0x7f2160008040 comp rx=0 tx=0).stop 2023-10-30T12:29:00.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.100+0000 7f2179874640 1 -- 172.21.15.89:0/2346991954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f216c0495b0 msgr2=0x7f216c04d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.100+0000 7f2179874640 1 --2- 172.21.15.89:0/2346991954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f216c0495b0 0x7f216c04d340 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f216800e890 tx=0x7f216800ed50 comp rx=0 tx=0).stop 2023-10-30T12:29:00.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.100+0000 7f2179874640 1 -- 172.21.15.89:0/2346991954 shutdown_connections 2023-10-30T12:29:00.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.100+0000 7f2179874640 1 --2- 172.21.15.89:0/2346991954 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f21580664f0 0x7f21580689b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.100+0000 7f2179874640 1 --2- 172.21.15.89:0/2346991954 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f216c0e24a0 0x7f216c17c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.100+0000 7f2179874640 1 --2- 172.21.15.89:0/2346991954 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f216c0e2880 0x7f216c04c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.100+0000 7f2179874640 1 --2- 172.21.15.89:0/2346991954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f216c0495b0 0x7f216c04d340 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.100+0000 7f2179874640 1 -- 172.21.15.89:0/2346991954 >> 172.21.15.89:0/2346991954 conn(0x7f216c054710 msgr2=0x7f216c0391f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:00.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.108+0000 7f2179874640 1 -- 172.21.15.89:0/2346991954 shutdown_connections 2023-10-30T12:29:00.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.108+0000 7f2179874640 1 -- 172.21.15.89:0/2346991954 wait complete. 2023-10-30T12:29:00.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.112+0000 7f9828d99640 1 -- 172.21.15.89:0/94908658 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f9834151210 con 0x7f98341598f0 2023-10-30T12:29:00.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.120+0000 7f982b7fe640 1 -- 172.21.15.89:0/94908658 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9834151210 con 0x7f98341598f0 2023-10-30T12:29:00.124 INFO:teuthology.orchestra.run.smithi089.stdout:60129542151 2023-10-30T12:29:00.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.124+0000 7f06b8ff9640 1 -- 172.21.15.89:0/2728258506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f06b00057e0 con 0x7f06cc153810 2023-10-30T12:29:00.131 INFO:teuthology.orchestra.run.smithi089.stdout:60129542150 2023-10-30T12:29:00.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.124+0000 7f06baffd640 1 -- 172.21.15.89:0/2728258506 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f06c40f2be0 con 0x7f06cc153810 2023-10-30T12:29:00.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.124+0000 7f9828d99640 1 -- 172.21.15.89:0/94908658 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f98140664a0 msgr2=0x7f9814068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.124+0000 7f9828d99640 1 --2- 172.21.15.89:0/94908658 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f98140664a0 0x7f9814068960 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9824009b60 tx=0x7f9824009190 comp rx=0 tx=0).stop 2023-10-30T12:29:00.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.124+0000 7f9828d99640 1 -- 172.21.15.89:0/94908658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98341598f0 msgr2=0x7f98341514f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.124+0000 7f9828d99640 1 --2- 172.21.15.89:0/94908658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98341598f0 0x7f98341514f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f982c00c9f0 tx=0x7f982c00ceb0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.124+0000 7f9828d99640 1 -- 172.21.15.89:0/94908658 shutdown_connections 2023-10-30T12:29:00.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.124+0000 7f9828d99640 1 --2- 172.21.15.89:0/94908658 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f98140664a0 0x7f9814068960 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.124+0000 7f9828d99640 1 --2- 172.21.15.89:0/94908658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98341598f0 0x7f98341514f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.124+0000 7f9828d99640 1 --2- 172.21.15.89:0/94908658 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9834155700 0x7f983413c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.124+0000 7f9828d99640 1 --2- 172.21.15.89:0/94908658 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98340c6f00 0x7f9834150340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.124+0000 7f9828d99640 1 -- 172.21.15.89:0/94908658 >> 172.21.15.89:0/94908658 conn(0x7f98340cae10 msgr2=0x7f9834153f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:00.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.128+0000 7f9828d99640 1 -- 172.21.15.89:0/94908658 shutdown_connections 2023-10-30T12:29:00.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.128+0000 7f9828d99640 1 -- 172.21.15.89:0/94908658 wait complete. 2023-10-30T12:29:00.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.128+0000 7f06b8ff9640 1 -- 172.21.15.89:0/2728258506 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f06ac0664a0 msgr2=0x7f06ac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.128+0000 7f06b8ff9640 1 --2- 172.21.15.89:0/2728258506 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f06ac0664a0 0x7f06ac068960 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f06bc000f70 tx=0x7f06bc009040 comp rx=0 tx=0).stop 2023-10-30T12:29:00.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.128+0000 7f06b8ff9640 1 -- 172.21.15.89:0/2728258506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06cc153810 msgr2=0x7f06cc148cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.128+0000 7f06b8ff9640 1 --2- 172.21.15.89:0/2728258506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06cc153810 0x7f06cc148cb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f06c401ce60 tx=0x7f06c401bfd0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.128+0000 7f06b8ff9640 1 -- 172.21.15.89:0/2728258506 shutdown_connections 2023-10-30T12:29:00.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.128+0000 7f06b8ff9640 1 --2- 172.21.15.89:0/2728258506 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f06ac0664a0 0x7f06ac068960 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.128+0000 7f06b8ff9640 1 --2- 172.21.15.89:0/2728258506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06cc153810 0x7f06cc148cb0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.128+0000 7f06b8ff9640 1 --2- 172.21.15.89:0/2728258506 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f06cc070840 0x7f06cc1f6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.128+0000 7f06b8ff9640 1 --2- 172.21.15.89:0/2728258506 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f06cc069e90 0x7f06cc1500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.128+0000 7f06b8ff9640 1 -- 172.21.15.89:0/2728258506 >> 172.21.15.89:0/2728258506 conn(0x7f06cc0c7910 msgr2=0x7f06cc0ca550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:00.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.128+0000 7f06b8ff9640 1 -- 172.21.15.89:0/2728258506 shutdown_connections 2023-10-30T12:29:00.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.128+0000 7f06b8ff9640 1 -- 172.21.15.89:0/2728258506 wait complete. 2023-10-30T12:29:00.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.132+0000 7f41b6503640 1 -- 172.21.15.89:0/1212470265 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f41900057e0 con 0x7f41a804c580 2023-10-30T12:29:00.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.132+0000 7f41a5ffb640 1 -- 172.21.15.89:0/1212470265 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f41ac0b5070 con 0x7f41a804c580 2023-10-30T12:29:00.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.132+0000 7f73807a1640 1 -- 172.21.15.89:0/2308884482 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f7360005510 con 0x7f73781599d0 2023-10-30T12:29:00.141 INFO:teuthology.orchestra.run.smithi089.stdout:60129542151 2023-10-30T12:29:00.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.132+0000 7f736effd640 1 -- 172.21.15.89:0/2308884482 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f73700f1ef0 con 0x7f73781599d0 2023-10-30T12:29:00.141 INFO:teuthology.orchestra.run.smithi089.stdout:60129542151 2023-10-30T12:29:00.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.136+0000 7f35a1217640 1 -- 172.21.15.89:0/1681253381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f357c005510 con 0x7f359c14cf60 2023-10-30T12:29:00.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.136+0000 7f35837fe640 1 -- 172.21.15.89:0/1681253381 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3594054ae0 con 0x7f359c14cf60 2023-10-30T12:29:00.143 INFO:teuthology.orchestra.run.smithi089.stdout:60129542151 2023-10-30T12:29:00.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.136+0000 7f41b6503640 1 -- 172.21.15.89:0/1212470265 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f41940664a0 msgr2=0x7f4194068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.140+0000 7f41b6503640 1 --2- 172.21.15.89:0/1212470265 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f41940664a0 0x7f4194068960 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f41a8173bc0 tx=0x7f41a0009040 comp rx=0 tx=0).stop 2023-10-30T12:29:00.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.140+0000 7f41b6503640 1 -- 172.21.15.89:0/1212470265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41a804c580 msgr2=0x7f41a816ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.140+0000 7f41b6503640 1 --2- 172.21.15.89:0/1212470265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41a804c580 0x7f41a816ef80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f41ac01d3c0 tx=0x7f41ac01bab0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.140+0000 7f01fdb3e640 1 -- 172.21.15.89:0/3769323033 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f01f80c7150 con 0x7f01f8157f70 2023-10-30T12:29:00.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.140+0000 7f01dffff640 1 -- 172.21.15.89:0/3769323033 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f01f80c7150 con 0x7f01f8157f70 2023-10-30T12:29:00.146 INFO:teuthology.orchestra.run.smithi089.stdout:60129542151 2023-10-30T12:29:00.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.140+0000 7f73807a1640 1 -- 172.21.15.89:0/2308884482 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f735c0664d0 msgr2=0x7f735c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.140+0000 7f73807a1640 1 --2- 172.21.15.89:0/2308884482 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f735c0664d0 0x7f735c068990 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7374009870 tx=0x7f7374008040 comp rx=0 tx=0).stop 2023-10-30T12:29:00.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.140+0000 7f73807a1640 1 -- 172.21.15.89:0/2308884482 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73781599d0 msgr2=0x7f737813f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.140+0000 7f73807a1640 1 --2- 172.21.15.89:0/2308884482 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73781599d0 0x7f737813f0e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7378156960 tx=0x7f737001bf20 comp rx=0 tx=0).stop 2023-10-30T12:29:00.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.140+0000 7f41b6503640 1 -- 172.21.15.89:0/1212470265 shutdown_connections 2023-10-30T12:29:00.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.140+0000 7f41b6503640 1 --2- 172.21.15.89:0/1212470265 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f41940664a0 0x7f4194068960 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.140+0000 7f41b6503640 1 --2- 172.21.15.89:0/1212470265 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41a80e3780 0x7f41a804aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.140+0000 7f41b6503640 1 --2- 172.21.15.89:0/1212470265 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41a80dd300 0x7f41a804dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.140+0000 7f41b6503640 1 --2- 172.21.15.89:0/1212470265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41a804c580 0x7f41a816ef80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.140+0000 7f41b6503640 1 -- 172.21.15.89:0/1212470265 >> 172.21.15.89:0/1212470265 conn(0x7f41a80501d0 msgr2=0x7f41a80e1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:00.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.140+0000 7f73807a1640 1 -- 172.21.15.89:0/2308884482 shutdown_connections 2023-10-30T12:29:00.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.140+0000 7f41b6503640 1 -- 172.21.15.89:0/1212470265 shutdown_connections 2023-10-30T12:29:00.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.140+0000 7f41b6503640 1 -- 172.21.15.89:0/1212470265 wait complete. 2023-10-30T12:29:00.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.140+0000 7f35a1217640 1 -- 172.21.15.89:0/1681253381 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f35880664c0 msgr2=0x7f3588068980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.140+0000 7f35a1217640 1 --2- 172.21.15.89:0/1681253381 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f35880664c0 0x7f3588068980 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f358c0ab000 tx=0x7f358c0a7070 comp rx=0 tx=0).stop 2023-10-30T12:29:00.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.140+0000 7f35a1217640 1 -- 172.21.15.89:0/1681253381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f359c14cf60 msgr2=0x7f359c14d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.140+0000 7f35a1217640 1 --2- 172.21.15.89:0/1681253381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f359c14cf60 0x7f359c14d3e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3584003980 tx=0x7f35940076c0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.140+0000 7f73807a1640 1 --2- 172.21.15.89:0/2308884482 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f735c0664d0 0x7f735c068990 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.140+0000 7f35a1217640 1 -- 172.21.15.89:0/1681253381 shutdown_connections 2023-10-30T12:29:00.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.140+0000 7f35a1217640 1 --2- 172.21.15.89:0/1681253381 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f35880664c0 0x7f3588068980 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.140+0000 7f35a1217640 1 --2- 172.21.15.89:0/1681253381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f359c0cf6b0 0x7f359c15c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.140+0000 7f35a1217640 1 --2- 172.21.15.89:0/1681253381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f359c14cf60 0x7f359c14d3e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.140+0000 7f35a1217640 1 --2- 172.21.15.89:0/1681253381 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35840039b0 0x7f359c156bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.140+0000 7f35a1217640 1 -- 172.21.15.89:0/1681253381 >> 172.21.15.89:0/1681253381 conn(0x7f359c06bda0 msgr2=0x7f359c157ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:00.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.144+0000 7f73807a1640 1 --2- 172.21.15.89:0/2308884482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73780c0ce0 0x7f7378154d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.144+0000 7f73807a1640 1 --2- 172.21.15.89:0/2308884482 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73781599d0 0x7f737813f0e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.144+0000 7f73807a1640 1 --2- 172.21.15.89:0/2308884482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f737806f540 0x7f737813dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.144+0000 7f73807a1640 1 -- 172.21.15.89:0/2308884482 >> 172.21.15.89:0/2308884482 conn(0x7f73780c7910 msgr2=0x7f73780c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:00.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.144+0000 7f35a1217640 1 -- 172.21.15.89:0/1681253381 shutdown_connections 2023-10-30T12:29:00.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.144+0000 7f35a1217640 1 -- 172.21.15.89:0/1681253381 wait complete. 2023-10-30T12:29:00.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.144+0000 7f73807a1640 1 -- 172.21.15.89:0/2308884482 shutdown_connections 2023-10-30T12:29:00.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.144+0000 7f73807a1640 1 -- 172.21.15.89:0/2308884482 wait complete. 2023-10-30T12:29:00.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.144+0000 7f01fdb3e640 1 -- 172.21.15.89:0/3769323033 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f01e4066500 msgr2=0x7f01e40689c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.144+0000 7f01fdb3e640 1 --2- 172.21.15.89:0/3769323033 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f01e4066500 0x7f01e40689c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f01e80b2000 tx=0x7f01e8008040 comp rx=0 tx=0).stop 2023-10-30T12:29:00.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.144+0000 7f01fdb3e640 1 -- 172.21.15.89:0/3769323033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01f8157f70 msgr2=0x7f01f81502d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.144+0000 7f01fdb3e640 1 --2- 172.21.15.89:0/3769323033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01f8157f70 0x7f01f81502d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f01f000c780 tx=0x7f01f000cc40 comp rx=0 tx=0).stop 2023-10-30T12:29:00.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.144+0000 7f01fdb3e640 1 -- 172.21.15.89:0/3769323033 shutdown_connections 2023-10-30T12:29:00.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.144+0000 7f01fdb3e640 1 --2- 172.21.15.89:0/3769323033 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f01e4066500 0x7f01e40689c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.144+0000 7f01fdb3e640 1 --2- 172.21.15.89:0/3769323033 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01f80cf160 0x7f01f814fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.144+0000 7f01fdb3e640 1 --2- 172.21.15.89:0/3769323033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01f8157f70 0x7f01f81502d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.144+0000 7f01fdb3e640 1 --2- 172.21.15.89:0/3769323033 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01f80cad80 0x7f01f8151070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.144+0000 7f01fdb3e640 1 -- 172.21.15.89:0/3769323033 >> 172.21.15.89:0/3769323033 conn(0x7f01f806bfe0 msgr2=0x7f01f80c4230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:00.159 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.0 2023-10-30T12:29:00.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.144+0000 7f01fdb3e640 1 -- 172.21.15.89:0/3769323033 shutdown_connections 2023-10-30T12:29:00.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.144+0000 7f01fdb3e640 1 -- 172.21.15.89:0/3769323033 wait complete. 2023-10-30T12:29:00.182 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542150 for osd.4 2023-10-30T12:29:00.184 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.3 2023-10-30T12:29:00.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.176+0000 7fd64ddf2640 1 -- 172.21.15.89:0/4107909011 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd64806ccb0 con 0x7fd6480c6f00 2023-10-30T12:29:00.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.180+0000 7fd6457fa640 1 -- 172.21.15.89:0/4107909011 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd638051a70 con 0x7fd6480c6f00 2023-10-30T12:29:00.185 INFO:teuthology.orchestra.run.smithi089.stdout:60129542151 2023-10-30T12:29:00.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.180+0000 7fd64ddf2640 1 -- 172.21.15.89:0/4107909011 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd634066570 msgr2=0x7fd634068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.180+0000 7fd64ddf2640 1 --2- 172.21.15.89:0/4107909011 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd634066570 0x7fd634068a30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd6400b2000 tx=0x7fd6400aa070 comp rx=0 tx=0).stop 2023-10-30T12:29:00.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.180+0000 7fd64ddf2640 1 -- 172.21.15.89:0/4107909011 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6480c6f00 msgr2=0x7fd648151e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.180+0000 7fd64ddf2640 1 --2- 172.21.15.89:0/4107909011 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6480c6f00 0x7fd648151e10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd63800f9e0 tx=0x7fd63800fde0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.180+0000 7fd64ddf2640 1 -- 172.21.15.89:0/4107909011 shutdown_connections 2023-10-30T12:29:00.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.180+0000 7fd64ddf2640 1 --2- 172.21.15.89:0/4107909011 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd634066570 0x7fd634068a30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.180+0000 7fd64ddf2640 1 --2- 172.21.15.89:0/4107909011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd648155700 0x7fd6480700b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.180+0000 7fd64ddf2640 1 --2- 172.21.15.89:0/4107909011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6481598f0 0x7fd6480705f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.180+0000 7fd64ddf2640 1 --2- 172.21.15.89:0/4107909011 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6480c6f00 0x7fd648151e10 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.180+0000 7fd64ddf2640 1 -- 172.21.15.89:0/4107909011 >> 172.21.15.89:0/4107909011 conn(0x7fd6480cae10 msgr2=0x7fd6480c54c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:00.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.180+0000 7fd64ddf2640 1 -- 172.21.15.89:0/4107909011 shutdown_connections 2023-10-30T12:29:00.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:00.180+0000 7fd64ddf2640 1 -- 172.21.15.89:0/4107909011 wait complete. 2023-10-30T12:29:00.190 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.7 2023-10-30T12:29:00.191 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.5 2023-10-30T12:29:00.192 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.6 2023-10-30T12:29:00.195 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.2 2023-10-30T12:29:00.209 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.1 2023-10-30T12:29:01.161 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-30T12:29:01.183 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-30T12:29:01.184 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-30T12:29:01.192 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-30T12:29:01.192 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-30T12:29:01.193 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-30T12:29:01.196 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-30T12:29:01.210 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-30T12:29:01.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.372+0000 7ff92abdd640 1 Processor -- start 2023-10-30T12:29:01.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.372+0000 7ff92abdd640 1 -- start start 2023-10-30T12:29:01.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.376+0000 7ff92abdd640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff91c0d4e20 0x7ff91c034880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.376+0000 7ff92abdd640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff91c034dc0 0x7ff91c0deaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.376+0000 7ff92a3dc640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff91c034dc0 0x7ff91c0deaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.376+0000 7ff92a3dc640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff91c034dc0 0x7ff91c0deaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:59352/0 (socket says 172.21.15.89:59352) 2023-10-30T12:29:01.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.376+0000 7ff92a3dc640 1 -- 172.21.15.89:0/586419839 learned_addr learned my addr 172.21.15.89:0/586419839 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:01.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.376+0000 7ff92abdd640 1 -- 172.21.15.89:0/586419839 --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff91c0452f0 con 0x7ff91c04b740 2023-10-30T12:29:01.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.376+0000 7ff929bdb640 1 --2- 172.21.15.89:0/586419839 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff91c0d4e20 0x7ff91c034880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.376+0000 7ff92abdd640 1 -- 172.21.15.89:0/586419839 --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff91c007940 con 0x7ff91c034dc0 2023-10-30T12:29:01.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.376+0000 7ff92abdd640 1 -- 172.21.15.89:0/586419839 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c0df0e0 con 0x7ff91c0d4e20 2023-10-30T12:29:01.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.376+0000 7ff928bd9640 1 -- 172.21.15.89:0/586419839 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3791541559 0 0) 0x7ff91c0452f0 con 0x7ff91c04b740 2023-10-30T12:29:01.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.376+0000 7ff928bd9640 1 -- 172.21.15.89:0/586419839 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff910003430 con 0x7ff91c04b740 2023-10-30T12:29:01.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.380+0000 7ff928bd9640 1 -- 172.21.15.89:0/586419839 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4204850290 0 0) 0x7ff910003430 con 0x7ff91c04b740 2023-10-30T12:29:01.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.380+0000 7ff92a3dc640 1 -- 172.21.15.89:0/586419839 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff91c0d4e20 msgr2=0x7ff91c034880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:01.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.380+0000 7ff92a3dc640 1 --2- 172.21.15.89:0/586419839 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff91c0d4e20 0x7ff91c034880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.380+0000 7ff92a3dc640 1 -- 172.21.15.89:0/586419839 >> v1:172.21.15.192:6789/0 conn(0x7ff91c04b740 legacy=0x7ff91c0de390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.380+0000 7ff92a3dc640 1 -- 172.21.15.89:0/586419839 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff91c006b80 con 0x7ff91c034dc0 2023-10-30T12:29:01.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.380+0000 7ff929bdb640 1 --2- 172.21.15.89:0/586419839 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff91c0d4e20 0x7ff91c034880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:29:01.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.384+0000 7ff92a3dc640 1 --2- 172.21.15.89:0/586419839 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff91c034dc0 0x7ff91c0deaa0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff924069c10 tx=0x7ff92001c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a97eab3d504071ae server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.384+0000 7ff928bd9640 1 -- 172.21.15.89:0/586419839 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92404e9c0 con 0x7ff91c034dc0 2023-10-30T12:29:01.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.384+0000 7ff928bd9640 1 -- 172.21.15.89:0/586419839 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9200a7050 con 0x7ff91c034dc0 2023-10-30T12:29:01.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.384+0000 7ff928bd9640 1 -- 172.21.15.89:0/586419839 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff920008be0 con 0x7ff91c034dc0 2023-10-30T12:29:01.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.388+0000 7ff92abdd640 1 -- 172.21.15.89:0/586419839 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff91c034dc0 msgr2=0x7ff91c0deaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.388+0000 7ff92abdd640 1 --2- 172.21.15.89:0/586419839 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff91c034dc0 0x7ff91c0deaa0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff924069c10 tx=0x7ff92001c3e0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.388+0000 7ff92abdd640 1 -- 172.21.15.89:0/586419839 shutdown_connections 2023-10-30T12:29:01.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.388+0000 7ff92abdd640 1 --2- 172.21.15.89:0/586419839 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff91c034dc0 0x7ff91c0deaa0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.388+0000 7ff92abdd640 1 --2- 172.21.15.89:0/586419839 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff91c0d4e20 0x7ff91c034880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.388+0000 7ff92abdd640 1 -- 172.21.15.89:0/586419839 >> 172.21.15.89:0/586419839 conn(0x7ff91c0502e0 msgr2=0x7ff91c0506f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:01.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.392+0000 7ff92abdd640 1 -- 172.21.15.89:0/586419839 shutdown_connections 2023-10-30T12:29:01.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.392+0000 7ff92abdd640 1 -- 172.21.15.89:0/586419839 wait complete. 2023-10-30T12:29:01.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.392+0000 7ff92abdd640 1 Processor -- start 2023-10-30T12:29:01.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.392+0000 7ff92abdd640 1 -- start start 2023-10-30T12:29:01.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.396+0000 7ff92abdd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff91c034dc0 0x7ff91c0c5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.396+0000 7ff92abdd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff91c04b740 0x7ff91c0c4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.396+0000 7ff92abdd640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff91c0d4e20 0x7ff91c0c51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.396+0000 7ff92abdd640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c0d68a0 con 0x7ff91c034dc0 2023-10-30T12:29:01.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.396+0000 7ff92abdd640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c0d7170 con 0x7ff91c04b740 2023-10-30T12:29:01.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.396+0000 7ff92abdd640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c0cfb80 con 0x7ff91c0d4e20 2023-10-30T12:29:01.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.396+0000 7ff9293da640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff91c04b740 0x7ff91c0c4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.396+0000 7ff92a3dc640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff91c0d4e20 0x7ff91c0c51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.396+0000 7ff9293da640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff91c04b740 0x7ff91c0c4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:59364/0 (socket says 172.21.15.89:59364) 2023-10-30T12:29:01.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.396+0000 7ff9293da640 1 -- 172.21.15.89:0/4032853748 learned_addr learned my addr 172.21.15.89:0/4032853748 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:01.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.396+0000 7ff929bdb640 1 --2- 172.21.15.89:0/4032853748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff91c034dc0 0x7ff91c0c5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.396+0000 7ff92a3dc640 1 -- 172.21.15.89:0/4032853748 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff91c04b740 msgr2=0x7ff91c0c4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.400+0000 7ff92a3dc640 1 --2- 172.21.15.89:0/4032853748 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff91c04b740 0x7ff91c0c4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.400+0000 7ff92a3dc640 1 -- 172.21.15.89:0/4032853748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff91c034dc0 msgr2=0x7ff91c0c5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.400+0000 7ff92a3dc640 1 --2- 172.21.15.89:0/4032853748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff91c034dc0 0x7ff91c0c5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.400+0000 7ff92a3dc640 1 -- 172.21.15.89:0/4032853748 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff914009420 con 0x7ff91c0d4e20 2023-10-30T12:29:01.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.400+0000 7ff929bdb640 1 --2- 172.21.15.89:0/4032853748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff91c034dc0 0x7ff91c0c5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:29:01.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.400+0000 7ff9293da640 1 --2- 172.21.15.89:0/4032853748 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff91c04b740 0x7ff91c0c4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:29:01.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.400+0000 7ff92a3dc640 1 --2- 172.21.15.89:0/4032853748 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff91c0d4e20 0x7ff91c0c51a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff9200aa000 tx=0x7ff92000abe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.400+0000 7ff91a7fc640 1 -- 172.21.15.89:0/4032853748 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff920004440 con 0x7ff91c0d4e20 2023-10-30T12:29:01.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.400+0000 7ff91a7fc640 1 -- 172.21.15.89:0/4032853748 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff92000a150 con 0x7ff91c0d4e20 2023-10-30T12:29:01.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.400+0000 7ff92abdd640 1 -- 172.21.15.89:0/4032853748 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff920008050 con 0x7ff91c0d4e20 2023-10-30T12:29:01.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.400+0000 7ff91a7fc640 1 -- 172.21.15.89:0/4032853748 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92000a4c0 con 0x7ff91c0d4e20 2023-10-30T12:29:01.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.400+0000 7ff92abdd640 1 -- 172.21.15.89:0/4032853748 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff91c0de370 con 0x7ff91c0d4e20 2023-10-30T12:29:01.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.400+0000 7ff92abdd640 1 -- 172.21.15.89:0/4032853748 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff91c0c6f70 con 0x7ff91c0d4e20 2023-10-30T12:29:01.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.404+0000 7ff91a7fc640 1 -- 172.21.15.89:0/4032853748 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff92000a620 con 0x7ff91c0d4e20 2023-10-30T12:29:01.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.404+0000 7ff91a7fc640 1 --2- 172.21.15.89:0/4032853748 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff908066570 0x7ff908068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.404+0000 7ff91a7fc640 1 -- 172.21.15.89:0/4032853748 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff92011e210 con 0x7ff91c0d4e20 2023-10-30T12:29:01.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.408+0000 7ff929bdb640 1 --2- 172.21.15.89:0/4032853748 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff908066570 0x7ff908068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.412+0000 7ff929bdb640 1 --2- 172.21.15.89:0/4032853748 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff908066570 0x7ff908068a30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff914009bd0 tx=0x7ff914008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.435 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.428+0000 7ff91a7fc640 1 -- 172.21.15.89:0/4032853748 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff9200eb820 con 0x7ff91c0d4e20 2023-10-30T12:29:01.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.448+0000 7f317acdc640 1 Processor -- start 2023-10-30T12:29:01.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.448+0000 7f467acdc640 1 Processor -- start 2023-10-30T12:29:01.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.448+0000 7f317acdc640 1 -- start start 2023-10-30T12:29:01.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.452+0000 7f317acdc640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f31741598f0 0x7f31740c2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.448+0000 7f467acdc640 1 -- start start 2023-10-30T12:29:01.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.452+0000 7f467acdc640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f466c0495b0 0x7f466c0499d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.452+0000 7f467acdc640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f466c0e2880 0x7f466c0e2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.452+0000 7f467acdc640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f466c0e3250 con 0x7f466c0e24a0 2023-10-30T12:29:01.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.452+0000 7f467acdc640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f466c051000 con 0x7f466c0e2880 2023-10-30T12:29:01.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.452+0000 7f467acdc640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f466c050d20 con 0x7f466c0495b0 2023-10-30T12:29:01.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.452+0000 7f317acdc640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31740c2e90 con 0x7f31740c6f00 2023-10-30T12:29:01.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.452+0000 7f317acdc640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f317414ff20 con 0x7f317414fb40 2023-10-30T12:29:01.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.452+0000 7f317acdc640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3174156700 con 0x7f31741598f0 2023-10-30T12:29:01.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.452+0000 7f46794d9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f466c0495b0 0x7f466c0499d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.452+0000 7f46794d9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f466c0495b0 0x7f466c0499d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:57342/0 (socket says 172.21.15.89:57342) 2023-10-30T12:29:01.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.452+0000 7f46794d9640 1 -- 172.21.15.89:0/2647259366 learned_addr learned my addr 172.21.15.89:0/2647259366 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:01.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.452+0000 7f3179cda640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f317414fb40 0x7f3174155700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:36068/0 (socket says 172.21.15.89:36068) 2023-10-30T12:29:01.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.452+0000 7f3179cda640 1 -- 172.21.15.89:0/3791773833 learned_addr learned my addr 172.21.15.89:0/3791773833 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:29:01.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.452+0000 7f467a4db640 1 --2- 172.21.15.89:0/2647259366 >> v2:172.21.15.192:3300/0 conn(0x7f466c0e2880 0x7f466c0e2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.452+0000 7f31794d9640 1 --2- 172.21.15.89:0/3791773833 >> v2:172.21.15.89:3300/0 conn(0x7f31741598f0 0x7f31740c2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.452+0000 7f3178cd8640 1 -- 172.21.15.89:0/3791773833 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 863595227 0 0) 0x7f317414ff20 con 0x7f317414fb40 2023-10-30T12:29:01.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.452+0000 7f3178cd8640 1 -- 172.21.15.89:0/3791773833 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3160003530 con 0x7f317414fb40 2023-10-30T12:29:01.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.452+0000 7f467a4db640 1 -- 172.21.15.89:0/2647259366 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f466c0495b0 msgr2=0x7f466c0499d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:01.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.452+0000 7f467a4db640 1 --2- 172.21.15.89:0/2647259366 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f466c0495b0 0x7f466c0499d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.452+0000 7f467a4db640 1 -- 172.21.15.89:0/2647259366 >> v1:172.21.15.192:6789/0 conn(0x7f466c0e24a0 legacy=0x7f466c04f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.452+0000 7f467a4db640 1 -- 172.21.15.89:0/2647259366 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f466c006e90 con 0x7f466c0e2880 2023-10-30T12:29:01.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.456+0000 7f31794d9640 1 -- 172.21.15.89:0/3791773833 >> v1:172.21.15.192:6789/0 conn(0x7f317414fb40 legacy=0x7f3174155700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.456+0000 7f31794d9640 1 -- 172.21.15.89:0/3791773833 >> v1:172.21.15.89:6789/0 conn(0x7f31740c6f00 legacy=0x7f317406d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:01.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.456+0000 7f31794d9640 1 -- 172.21.15.89:0/3791773833 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f317406f600 con 0x7f31741598f0 2023-10-30T12:29:01.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.456+0000 7f31af73e640 1 Processor -- start 2023-10-30T12:29:01.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.456+0000 7f31af73e640 1 -- start start 2023-10-30T12:29:01.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.456+0000 7ff97ff1d640 1 Processor -- start 2023-10-30T12:29:01.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.460+0000 7ff97ff1d640 1 -- start start 2023-10-30T12:29:01.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.460+0000 7f31af73e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31a00d1650 0x7f31a00dacd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.460+0000 7f31794d9640 1 --2- 172.21.15.89:0/3791773833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31741598f0 0x7f31740c2950 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f31700097c0 tx=0x7f317001c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aaa9b8382e6a21c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.460+0000 7f31af73e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f31a004b0e0 0x7f31a004b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.460+0000 7f3178cd8640 1 -- 172.21.15.89:0/3791773833 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31700ab610 con 0x7f31741598f0 2023-10-30T12:29:01.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.460+0000 7f31accd9640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f31a0005ee0 0x7f31a0045170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:36098/0 (socket says 172.21.15.89:36098) 2023-10-30T12:29:01.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.460+0000 7f3178cd8640 1 -- 172.21.15.89:0/3791773833 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31700ab770 con 0x7f31741598f0 2023-10-30T12:29:01.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.460+0000 7f3178cd8640 1 -- 172.21.15.89:0/3791773833 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31700abae0 con 0x7f31741598f0 2023-10-30T12:29:01.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.460+0000 7f467a4db640 1 --2- 172.21.15.89:0/2647259366 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f466c0e2880 0x7f466c0e2d10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f46700097c0 tx=0x7f467001ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f7b2564ddbd4934 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.460+0000 7f31ad4da640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31a00d1650 0x7f31a00dacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.460+0000 7ff97ff1d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff9700d4de0 0x7ff97004b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.460+0000 7f31ad4da640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31a00d1650 0x7f31a00dacd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:57352/0 (socket says 172.21.15.89:57352) 2023-10-30T12:29:01.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.460+0000 7f31af73e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31a00072f0 con 0x7f31a0005ee0 2023-10-30T12:29:01.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.460+0000 7f31af73e640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31a0007780 con 0x7f31a004b0e0 2023-10-30T12:29:01.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.460+0000 7f31af73e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a0006420 con 0x7f31a00d1650 2023-10-30T12:29:01.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.460+0000 7f31adcdb640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f31a004b0e0 0x7f31a004b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.460+0000 7f31adcdb640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f31a004b0e0 0x7f31a004b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:59392/0 (socket says 172.21.15.89:59392) 2023-10-30T12:29:01.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.460+0000 7f31adcdb640 1 -- 172.21.15.89:0/3372149157 learned_addr learned my addr 172.21.15.89:0/3372149157 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:01.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.460+0000 7f317acdc640 1 -- 172.21.15.89:0/3791773833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31741598f0 msgr2=0x7f31740c2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.460+0000 7f317acdc640 1 --2- 172.21.15.89:0/3791773833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31741598f0 0x7f31740c2950 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f31700097c0 tx=0x7f317001c430 comp rx=0 tx=0).stop 2023-10-30T12:29:01.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.460+0000 7f4678cd8640 1 -- 172.21.15.89:0/2647259366 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46700a8970 con 0x7f466c0e2880 2023-10-30T12:29:01.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.460+0000 7f4678cd8640 1 -- 172.21.15.89:0/2647259366 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46700a8ad0 con 0x7f466c0e2880 2023-10-30T12:29:01.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.460+0000 7f4678cd8640 1 -- 172.21.15.89:0/2647259366 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46700a8e60 con 0x7f466c0e2880 2023-10-30T12:29:01.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7f317acdc640 1 -- 172.21.15.89:0/3791773833 shutdown_connections 2023-10-30T12:29:01.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7f317acdc640 1 --2- 172.21.15.89:0/3791773833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31741598f0 0x7f31740c2950 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7f317acdc640 1 -- 172.21.15.89:0/3791773833 >> 172.21.15.89:0/3791773833 conn(0x7f31740cae10 msgr2=0x7f31740cb220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:01.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7f317acdc640 1 -- 172.21.15.89:0/3791773833 shutdown_connections 2023-10-30T12:29:01.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7f319ffff640 1 -- 172.21.15.89:0/3372149157 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528498951 0 0) 0x7f31a00072f0 con 0x7f31a0005ee0 2023-10-30T12:29:01.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7f319ffff640 1 -- 172.21.15.89:0/3372149157 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3194003530 con 0x7f31a0005ee0 2023-10-30T12:29:01.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7f31ad4da640 1 -- 172.21.15.89:0/3372149157 >> v2:172.21.15.192:3300/0 conn(0x7f31a004b0e0 msgr2=0x7f31a004b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:01.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7f31ad4da640 1 --2- 172.21.15.89:0/3372149157 >> v2:172.21.15.192:3300/0 conn(0x7f31a004b0e0 0x7f31a004b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7f31ad4da640 1 -- 172.21.15.89:0/3372149157 >> v1:172.21.15.192:6789/0 conn(0x7f31a0005ee0 legacy=0x7f31a0045170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7f31ad4da640 1 -- 172.21.15.89:0/3372149157 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a00066e0 con 0x7f31a00d1650 2023-10-30T12:29:01.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7f317acdc640 1 -- 172.21.15.89:0/3791773833 wait complete. 2023-10-30T12:29:01.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7ff97ff1d640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff970050ec0 0x7ff9700060b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7ff97ff1d640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff97004bc00 con 0x7ff9700471c0 2023-10-30T12:29:01.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7ff97ff1d640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff97004bec0 con 0x7ff9700d4de0 2023-10-30T12:29:01.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7ff97ff1d640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff970046400 con 0x7ff970050ec0 2023-10-30T12:29:01.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7ff97dcb9640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff970050ec0 0x7ff9700060b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7ff97dcb9640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff970050ec0 0x7ff9700060b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:59394/0 (socket says 172.21.15.89:59394) 2023-10-30T12:29:01.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7ff97dcb9640 1 -- 172.21.15.89:0/2668755254 learned_addr learned my addr 172.21.15.89:0/2668755254 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:01.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7ff97d4b8640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff9700d4de0 0x7ff97004b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7ff97dcb9640 1 -- 172.21.15.89:0/2668755254 >> v2:172.21.15.89:3300/0 conn(0x7ff9700d4de0 msgr2=0x7ff97004b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:01.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7ff97dcb9640 1 --2- 172.21.15.89:0/2668755254 >> v2:172.21.15.89:3300/0 conn(0x7ff9700d4de0 0x7ff97004b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7ff97dcb9640 1 -- 172.21.15.89:0/2668755254 >> v1:172.21.15.89:6789/0 conn(0x7ff9700471c0 legacy=0x7ff9700db650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:01.480 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7ff97dcb9640 1 -- 172.21.15.89:0/2668755254 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9700466e0 con 0x7ff970050ec0 2023-10-30T12:29:01.480 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7f467acdc640 1 -- 172.21.15.89:0/2647259366 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f466c0e2880 msgr2=0x7f466c0e2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.480 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7f467acdc640 1 --2- 172.21.15.89:0/2647259366 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f466c0e2880 0x7f466c0e2d10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f46700097c0 tx=0x7f467001ecb0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7ff97d4b8640 1 --2- 172.21.15.89:0/2668755254 >> v2:172.21.15.89:3300/0 conn(0x7ff9700d4de0 0x7ff97004b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:29:01.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7f467acdc640 1 -- 172.21.15.89:0/2647259366 shutdown_connections 2023-10-30T12:29:01.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7f467acdc640 1 --2- 172.21.15.89:0/2647259366 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f466c0e2880 0x7f466c0e2d10 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7f317acdc640 1 Processor -- start 2023-10-30T12:29:01.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7f467acdc640 1 --2- 172.21.15.89:0/2647259366 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f466c0495b0 0x7f466c0499d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7f467acdc640 1 -- 172.21.15.89:0/2647259366 >> 172.21.15.89:0/2647259366 conn(0x7f466c054710 msgr2=0x7f466c054b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:01.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7f317acdc640 1 -- start start 2023-10-30T12:29:01.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.464+0000 7f8a7ddd3640 1 Processor -- start 2023-10-30T12:29:01.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.468+0000 7f467acdc640 1 -- 172.21.15.89:0/2647259366 shutdown_connections 2023-10-30T12:29:01.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.468+0000 7f8a7ddd3640 1 -- start start 2023-10-30T12:29:01.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.468+0000 7f467acdc640 1 -- 172.21.15.89:0/2647259366 wait complete. 2023-10-30T12:29:01.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.468+0000 7f317acdc640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31740c6f00 0x7f317414b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.468+0000 7f8a7ddd3640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8a78152de0 0x7f8a78156280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.468+0000 7f8a7ddd3640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a781567c0 con 0x7f8a78069e90 2023-10-30T12:29:01.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.468+0000 7f8a7ddd3640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a780c7400 con 0x7f8a7806ad90 2023-10-30T12:29:01.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.468+0000 7f8a7ddd3640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a78070820 con 0x7f8a78152de0 2023-10-30T12:29:01.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.468+0000 7f31ad4da640 1 --2- 172.21.15.89:0/3372149157 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31a00d1650 0x7f31a00dacd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f31a40097c0 tx=0x7f31a40230a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d7b5b9ea58c3f73 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.468+0000 7f8a777fe640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8a78152de0 0x7f8a78156280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.468+0000 7f319ffff640 1 -- 172.21.15.89:0/3372149157 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a40acd80 con 0x7f31a00d1650 2023-10-30T12:29:01.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.468+0000 7f8a76ffd640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f8a7806ad90 0x7f8a780c6670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:36108/0 (socket says 172.21.15.89:36108) 2023-10-30T12:29:01.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.468+0000 7f8a76ffd640 1 -- 172.21.15.89:0/2286522941 learned_addr learned my addr 172.21.15.89:0/2286522941 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:29:01.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.468+0000 7f467acdc640 1 Processor -- start 2023-10-30T12:29:01.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.468+0000 7f319ffff640 1 -- 172.21.15.89:0/3372149157 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a40b4440 con 0x7f31a00d1650 2023-10-30T12:29:01.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.468+0000 7f319ffff640 1 -- 172.21.15.89:0/3372149157 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a40b47d0 con 0x7f31a00d1650 2023-10-30T12:29:01.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.468+0000 7ff97dcb9640 1 --2- 172.21.15.89:0/2668755254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff970050ec0 0x7ff9700060b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff96c013150 tx=0x7ff96c024bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=567f8584c23106c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.468+0000 7f3179cda640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31740c6f00 0x7f317414b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.468+0000 7f3179cda640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31740c6f00 0x7f317414b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:57358/0 (socket says 172.21.15.89:57358) 2023-10-30T12:29:01.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.468+0000 7f3179cda640 1 -- 172.21.15.89:0/3093624752 learned_addr learned my addr 172.21.15.89:0/3093624752 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:01.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.468+0000 7f8a777fe640 1 -- 172.21.15.89:0/2286522941 >> v1:172.21.15.192:6789/0 conn(0x7f8a7806ad90 legacy=0x7f8a780c6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.468+0000 7f8a777fe640 1 -- 172.21.15.89:0/2286522941 >> v1:172.21.15.89:6789/0 conn(0x7f8a78069e90 legacy=0x7f8a78160180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.468+0000 7f8a777fe640 1 -- 172.21.15.89:0/2286522941 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a780c76c0 con 0x7f8a78152de0 2023-10-30T12:29:01.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.468+0000 7f317acdc640 1 --2- 172.21.15.89:0/3093624752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f317414fb40 0x7f31741484d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.468+0000 7f317acdc640 1 --2- 172.21.15.89:0/3093624752 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f317406d570 0x7f3174148d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f467acdc640 1 -- start start 2023-10-30T12:29:01.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f31af73e640 1 -- 172.21.15.89:0/3372149157 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31a00d1650 msgr2=0x7f31a00dacd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f317a4db640 1 --2- 172.21.15.89:0/3093624752 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f317406d570 0x7f3174148d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f31af73e640 1 --2- 172.21.15.89:0/3372149157 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31a00d1650 0x7f31a00dacd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f31a40097c0 tx=0x7f31a40230a0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f317acdc640 1 -- 172.21.15.89:0/3093624752 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31741f6dd0 con 0x7f317414fb40 2023-10-30T12:29:01.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f31794d9640 1 --2- 172.21.15.89:0/3093624752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f317414fb40 0x7f31741484d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f317acdc640 1 -- 172.21.15.89:0/3093624752 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31741f61e0 con 0x7f317406d570 2023-10-30T12:29:01.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f317acdc640 1 -- 172.21.15.89:0/3093624752 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f317414bee0 con 0x7f31740c6f00 2023-10-30T12:29:01.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7ff97ecbb640 1 -- 172.21.15.89:0/2668755254 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96c0ada80 con 0x7ff970050ec0 2023-10-30T12:29:01.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7ff97ecbb640 1 -- 172.21.15.89:0/2668755254 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff96c0adbe0 con 0x7ff970050ec0 2023-10-30T12:29:01.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7ff97ecbb640 1 -- 172.21.15.89:0/2668755254 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96c0110d0 con 0x7ff970050ec0 2023-10-30T12:29:01.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f31af73e640 1 -- 172.21.15.89:0/3372149157 shutdown_connections 2023-10-30T12:29:01.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f31af73e640 1 --2- 172.21.15.89:0/3372149157 >> v2:172.21.15.192:3300/0 conn(0x7f31a004b0e0 0x7f31a004b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f31af73e640 1 --2- 172.21.15.89:0/3372149157 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31a00d1650 0x7f31a00dacd0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f31af73e640 1 -- 172.21.15.89:0/3372149157 >> 172.21.15.89:0/3372149157 conn(0x7f31a00501d0 msgr2=0x7f31a00505e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:01.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f467acdc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f466c0495b0 0x7f466c175580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f467acdc640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f466c0e24a0 0x7f466c0cd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f4679cda640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f466c0495b0 0x7f466c175580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f4679cda640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f466c0495b0 0x7f466c175580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:59416/0 (socket says 172.21.15.89:59416) 2023-10-30T12:29:01.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f4679cda640 1 -- 172.21.15.89:0/1817135296 learned_addr learned my addr 172.21.15.89:0/1817135296 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:01.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f467acdc640 1 --2- 172.21.15.89:0/1817135296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f466c0e2880 0x7f466c168d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f46794d9640 1 --2- 172.21.15.89:0/1817135296 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f466c0e24a0 0x7f466c0cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f467acdc640 1 -- 172.21.15.89:0/1817135296 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466c0e2220 con 0x7f466c0e2880 2023-10-30T12:29:01.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f467acdc640 1 -- 172.21.15.89:0/1817135296 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466c168870 con 0x7f466c0495b0 2023-10-30T12:29:01.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f467acdc640 1 -- 172.21.15.89:0/1817135296 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f466c039d90 con 0x7f466c0e24a0 2023-10-30T12:29:01.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f4679cda640 1 -- 172.21.15.89:0/1817135296 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f466c0e24a0 msgr2=0x7f466c0cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f4679cda640 1 --2- 172.21.15.89:0/1817135296 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f466c0e24a0 0x7f466c0cd330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f4679cda640 1 -- 172.21.15.89:0/1817135296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f466c0e2880 msgr2=0x7f466c168d00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:29:01.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f467a4db640 1 --2- 172.21.15.89:0/1817135296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f466c0e2880 0x7f466c168d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f4679cda640 1 --2- 172.21.15.89:0/1817135296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f466c0e2880 0x7f466c168d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.476+0000 7f4679cda640 1 -- 172.21.15.89:0/1817135296 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4670009420 con 0x7f466c0495b0 2023-10-30T12:29:01.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.476+0000 7f467a4db640 1 --2- 172.21.15.89:0/1817135296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f466c0e2880 0x7f466c168d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:29:01.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.476+0000 7f4679cda640 1 --2- 172.21.15.89:0/1817135296 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f466c0495b0 0x7f466c175580 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f467407ad90 tx=0x7f4674066e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.476+0000 7f466a7fc640 1 -- 172.21.15.89:0/1817135296 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4674072770 con 0x7f466c0495b0 2023-10-30T12:29:01.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.476+0000 7f467acdc640 1 -- 172.21.15.89:0/1817135296 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f466c0397d0 con 0x7f466c0495b0 2023-10-30T12:29:01.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.476+0000 7f466a7fc640 1 -- 172.21.15.89:0/1817135296 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46740728d0 con 0x7f466c0495b0 2023-10-30T12:29:01.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f317a4db640 1 -- 172.21.15.89:0/3093624752 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31740c6f00 msgr2=0x7f317414b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f317a4db640 1 --2- 172.21.15.89:0/3093624752 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31740c6f00 0x7f317414b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f317a4db640 1 -- 172.21.15.89:0/3093624752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f317414fb40 msgr2=0x7f31741484d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f317a4db640 1 --2- 172.21.15.89:0/3093624752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f317414fb40 0x7f31741484d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f317a4db640 1 -- 172.21.15.89:0/3093624752 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3170009420 con 0x7f317406d570 2023-10-30T12:29:01.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f317a4db640 1 --2- 172.21.15.89:0/3093624752 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f317406d570 0x7f3174148d70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f316400c410 tx=0x7f316400c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f316a7fc640 1 -- 172.21.15.89:0/3093624752 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3164003600 con 0x7f317406d570 2023-10-30T12:29:01.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f316a7fc640 1 -- 172.21.15.89:0/3093624752 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3164003760 con 0x7f317406d570 2023-10-30T12:29:01.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f316a7fc640 1 -- 172.21.15.89:0/3093624752 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3164003b50 con 0x7f317406d570 2023-10-30T12:29:01.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.476+0000 7f317acdc640 1 -- 172.21.15.89:0/3093624752 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31741f64a0 con 0x7f317406d570 2023-10-30T12:29:01.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.476+0000 7f317acdc640 1 -- 172.21.15.89:0/3093624752 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31741f6a60 con 0x7f317406d570 2023-10-30T12:29:01.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7ff97ff1d640 1 -- 172.21.15.89:0/2668755254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff970050ec0 msgr2=0x7ff9700060b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7ff97ff1d640 1 --2- 172.21.15.89:0/2668755254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff970050ec0 0x7ff9700060b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff96c013150 tx=0x7ff96c024bd0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7ff97ff1d640 1 -- 172.21.15.89:0/2668755254 shutdown_connections 2023-10-30T12:29:01.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7ff97ff1d640 1 --2- 172.21.15.89:0/2668755254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff970050ec0 0x7ff9700060b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7ff97ff1d640 1 --2- 172.21.15.89:0/2668755254 >> v2:172.21.15.89:3300/0 conn(0x7ff9700d4de0 0x7ff97004b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7ff97ff1d640 1 -- 172.21.15.89:0/2668755254 >> 172.21.15.89:0/2668755254 conn(0x7ff9700503a0 msgr2=0x7ff9700507b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:01.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.476+0000 7ff97ff1d640 1 -- 172.21.15.89:0/2668755254 shutdown_connections 2023-10-30T12:29:01.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.476+0000 7ff97ff1d640 1 -- 172.21.15.89:0/2668755254 wait complete. 2023-10-30T12:29:01.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f31af73e640 1 -- 172.21.15.89:0/3372149157 shutdown_connections 2023-10-30T12:29:01.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.472+0000 7f31af73e640 1 -- 172.21.15.89:0/3372149157 wait complete. 2023-10-30T12:29:01.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.476+0000 7f31af73e640 1 Processor -- start 2023-10-30T12:29:01.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.476+0000 7f467acdc640 1 -- 172.21.15.89:0/1817135296 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f466c168500 con 0x7f466c0495b0 2023-10-30T12:29:01.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.476+0000 7f466a7fc640 1 -- 172.21.15.89:0/1817135296 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4674072ca0 con 0x7f466c0495b0 2023-10-30T12:29:01.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.476+0000 7ff97ff1d640 1 Processor -- start 2023-10-30T12:29:01.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.476+0000 7f317acdc640 1 -- 172.21.15.89:0/3093624752 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f317404e170 con 0x7f317406d570 2023-10-30T12:29:01.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.476+0000 7f8a777fe640 1 --2- 172.21.15.89:0/2286522941 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a78152de0 0x7f8a78156280 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c00a8e0 tx=0x7f8a6c01d680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a433254fa2e32d19 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.476+0000 7f31af73e640 1 -- start start 2023-10-30T12:29:01.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.476+0000 7ff97ff1d640 1 -- start start 2023-10-30T12:29:01.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.476+0000 7f7e1bf01640 1 Processor -- start 2023-10-30T12:29:01.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.476+0000 7f467acdc640 1 -- 172.21.15.89:0/1817135296 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f466c168360 con 0x7f466c0495b0 2023-10-30T12:29:01.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.476+0000 7f7e1bf01640 1 -- start start 2023-10-30T12:29:01.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.476+0000 7ff97ff1d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9700471c0 0x7ff9700cb070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.476+0000 7ff97ff1d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff970050ec0 0x7ff97016c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7ff97ff1d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9700d4de0 0x7ff9700d7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7ff97ff1d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9700cb5b0 con 0x7ff9700d4de0 2023-10-30T12:29:01.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7ff97ff1d640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff970172c00 con 0x7ff9700471c0 2023-10-30T12:29:01.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7ff97ff1d640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9700d5890 con 0x7ff970050ec0 2023-10-30T12:29:01.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f466a7fc640 1 -- 172.21.15.89:0/1817135296 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f467409d3f0 con 0x7f466c0495b0 2023-10-30T12:29:01.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7ff97ccb7640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff970050ec0 0x7ff97016c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7ff97dcb9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9700d4de0 0x7ff9700d7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7ff97ccb7640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff970050ec0 0x7ff97016c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:57370/0 (socket says 172.21.15.89:57370) 2023-10-30T12:29:01.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7ff97ccb7640 1 -- 172.21.15.89:0/868636229 learned_addr learned my addr 172.21.15.89:0/868636229 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:01.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7ff97dcb9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9700d4de0 0x7ff9700d7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39152/0 (socket says 172.21.15.89:39152) 2023-10-30T12:29:01.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7ff97d4b8640 1 --2- 172.21.15.89:0/868636229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9700471c0 0x7ff9700cb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f8a767fc640 1 -- 172.21.15.89:0/2286522941 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c0aa660 con 0x7f8a78152de0 2023-10-30T12:29:01.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f8a767fc640 1 -- 172.21.15.89:0/2286522941 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a6c0aa7c0 con 0x7f8a78152de0 2023-10-30T12:29:01.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f8a767fc640 1 -- 172.21.15.89:0/2286522941 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c0aab50 con 0x7f8a78152de0 2023-10-30T12:29:01.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7ff97dcb9640 1 -- 172.21.15.89:0/868636229 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff970050ec0 msgr2=0x7ff97016c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7ff97dcb9640 1 --2- 172.21.15.89:0/868636229 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff970050ec0 0x7ff97016c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7ff97dcb9640 1 -- 172.21.15.89:0/868636229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9700471c0 msgr2=0x7ff9700cb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7ff97dcb9640 1 --2- 172.21.15.89:0/868636229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9700471c0 0x7ff9700cb070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7ff97dcb9640 1 -- 172.21.15.89:0/868636229 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff96c011d70 con 0x7ff9700d4de0 2023-10-30T12:29:01.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f31af73e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31a0005ee0 0x7f31a00dc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f31af73e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31a004b0e0 0x7f31a00c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f31af73e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31a00d1650 0x7f31a00de890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f31accd9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31a004b0e0 0x7f31a00c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f31ad4da640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31a0005ee0 0x7f31a00dc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f31accd9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31a004b0e0 0x7f31a00c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:59430/0 (socket says 172.21.15.89:59430) 2023-10-30T12:29:01.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7ff97dcb9640 1 --2- 172.21.15.89:0/868636229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9700d4de0 0x7ff9700d7330 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff96c0b3000 tx=0x7ff96c021970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f466a7fc640 1 --2- 172.21.15.89:0/1817135296 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f46580664a0 0x7f4658068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7ff9667fc640 1 -- 172.21.15.89:0/868636229 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96c0112d0 con 0x7ff9700d4de0 2023-10-30T12:29:01.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7ff9667fc640 1 -- 172.21.15.89:0/868636229 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff96c011430 con 0x7ff9700d4de0 2023-10-30T12:29:01.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7ff9667fc640 1 -- 172.21.15.89:0/868636229 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96c0c0470 con 0x7ff9700d4de0 2023-10-30T12:29:01.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f7e1bf01640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7e140c8cf0 0x7f7e140c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f7e1bf01640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7e140c90d0 0x7f7e140ca340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f7e1bf01640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e1415ce30 con 0x7f7e140cef90 2023-10-30T12:29:01.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f7e1bf01640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e14159880 con 0x7f7e140c8cf0 2023-10-30T12:29:01.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f7e1bf01640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e14159ae0 con 0x7f7e140c90d0 2023-10-30T12:29:01.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f316a7fc640 1 -- 172.21.15.89:0/3093624752 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3164003cb0 con 0x7f317406d570 2023-10-30T12:29:01.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f46794d9640 1 --2- 172.21.15.89:0/1817135296 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f46580664a0 0x7f4658068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f316a7fc640 1 --2- 172.21.15.89:0/3093624752 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f31580664f0 0x7f31580689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f3179cda640 1 --2- 172.21.15.89:0/3093624752 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f31580664f0 0x7f31580689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f316a7fc640 1 -- 172.21.15.89:0/3093624752 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f31640545e0 con 0x7f317406d570 2023-10-30T12:29:01.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f31ad4da640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31a0005ee0 0x7f31a00dc8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:57380/0 (socket says 172.21.15.89:57380) 2023-10-30T12:29:01.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f31af73e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a00c6680 con 0x7f31a00d1650 2023-10-30T12:29:01.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f31af73e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a00c78e0 con 0x7f31a004b0e0 2023-10-30T12:29:01.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f31af73e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a00cc700 con 0x7f31a0005ee0 2023-10-30T12:29:01.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f7e1a49e640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f7e140cef90 0x7f7e1415d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:52146/0 (socket says 172.21.15.89:52146) 2023-10-30T12:29:01.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f31accd9640 1 -- 172.21.15.89:0/1127765517 learned_addr learned my addr 172.21.15.89:0/1127765517 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:01.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f31adcdb640 1 --2- 172.21.15.89:0/1127765517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31a00d1650 0x7f31a00de890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f7e1a49e640 1 -- 172.21.15.89:0/578885602 learned_addr learned my addr 172.21.15.89:0/578885602 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:29:01.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f31accd9640 1 -- 172.21.15.89:0/1127765517 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31a0005ee0 msgr2=0x7f31a00dc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f31accd9640 1 --2- 172.21.15.89:0/1127765517 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31a0005ee0 0x7f31a00dc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f31accd9640 1 -- 172.21.15.89:0/1127765517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31a00d1650 msgr2=0x7f31a00de890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f31accd9640 1 --2- 172.21.15.89:0/1127765517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31a00d1650 0x7f31a00de890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f31accd9640 1 -- 172.21.15.89:0/1127765517 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a4009420 con 0x7f31a004b0e0 2023-10-30T12:29:01.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f31ad4da640 1 --2- 172.21.15.89:0/1127765517 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31a0005ee0 0x7f31a00dc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:29:01.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f466a7fc640 1 -- 172.21.15.89:0/1817135296 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46740fe030 con 0x7f466c0495b0 2023-10-30T12:29:01.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f3179cda640 1 --2- 172.21.15.89:0/3093624752 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f31580664f0 0x7f31580689b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f316c001570 tx=0x7f316c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7f31accd9640 1 --2- 172.21.15.89:0/1127765517 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31a004b0e0 0x7f31a00c51f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f319800c530 tx=0x7f319800c9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f7e18c9b640 1 -- 172.21.15.89:0/578885602 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1993981866 0 0) 0x7f7e1415ce30 con 0x7f7e140cef90 2023-10-30T12:29:01.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f7e18c9b640 1 -- 172.21.15.89:0/578885602 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e00003530 con 0x7f7e140cef90 2023-10-30T12:29:01.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7ff97ff1d640 1 -- 172.21.15.89:0/868636229 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9700d5b70 con 0x7ff9700d4de0 2023-10-30T12:29:01.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.480+0000 7ff97ff1d640 1 -- 172.21.15.89:0/868636229 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9701731a0 con 0x7ff9700d4de0 2023-10-30T12:29:01.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7ff97ff1d640 1 -- 172.21.15.89:0/868636229 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff970005520 con 0x7ff9700d4de0 2023-10-30T12:29:01.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f8a7ddd3640 1 -- 172.21.15.89:0/2286522941 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a78152de0 msgr2=0x7f8a78156280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f8a7ddd3640 1 --2- 172.21.15.89:0/2286522941 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a78152de0 0x7f8a78156280 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c00a8e0 tx=0x7f8a6c01d680 comp rx=0 tx=0).stop 2023-10-30T12:29:01.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f7e19c9d640 1 --2- 172.21.15.89:0/578885602 >> v2:172.21.15.89:3300/0 conn(0x7f7e140c8cf0 0x7f7e140c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f46794d9640 1 --2- 172.21.15.89:0/1817135296 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f46580664a0 0x7f4658068960 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f466c0dca30 tx=0x7f4664008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f8a7ddd3640 1 -- 172.21.15.89:0/2286522941 shutdown_connections 2023-10-30T12:29:01.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f8a7ddd3640 1 --2- 172.21.15.89:0/2286522941 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a78152de0 0x7f8a78156280 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f8a7ddd3640 1 -- 172.21.15.89:0/2286522941 >> 172.21.15.89:0/2286522941 conn(0x7f8a780c7910 msgr2=0x7f8a780c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:01.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f8a7ddd3640 1 -- 172.21.15.89:0/2286522941 shutdown_connections 2023-10-30T12:29:01.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f8a7ddd3640 1 -- 172.21.15.89:0/2286522941 wait complete. 2023-10-30T12:29:01.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f7e1949c640 1 --2- 172.21.15.89:0/578885602 >> v2:172.21.15.192:3300/0 conn(0x7f7e140c90d0 0x7f7e140ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f7e18c9b640 1 -- 172.21.15.89:0/578885602 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3097566863 0 0) 0x7f7e00003530 con 0x7f7e140cef90 2023-10-30T12:29:01.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f7e18c9b640 1 -- 172.21.15.89:0/578885602 >> v2:172.21.15.192:3300/0 conn(0x7f7e140c90d0 msgr2=0x7f7e140ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:01.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f7e18c9b640 1 --2- 172.21.15.89:0/578885602 >> v2:172.21.15.192:3300/0 conn(0x7f7e140c90d0 0x7f7e140ca340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f7e18c9b640 1 -- 172.21.15.89:0/578885602 >> v2:172.21.15.89:3300/0 conn(0x7f7e140c8cf0 msgr2=0x7f7e140c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:01.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f7e18c9b640 1 --2- 172.21.15.89:0/578885602 >> v2:172.21.15.89:3300/0 conn(0x7f7e140c8cf0 0x7f7e140c9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f7e18c9b640 1 -- 172.21.15.89:0/578885602 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e14159dc0 con 0x7f7e140cef90 2023-10-30T12:29:01.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f7e18c9b640 1 -- 172.21.15.89:0/578885602 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f7e0c001a00 con 0x7f7e140cef90 2023-10-30T12:29:01.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f7e18c9b640 1 -- 172.21.15.89:0/578885602 >> v1:172.21.15.89:6789/0 conn(0x7f7e140cef90 legacy=0x7f7e1415d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f7e18c9b640 1 --2- 172.21.15.89:0/578885602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e000039d0 0x7f7e00003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f7e18c9b640 1 --2- 172.21.15.89:0/578885602 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e000047d0 0x7f7e00004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f7e18c9b640 1 --2- 172.21.15.89:0/578885602 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e00005190 0x7f7e0000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f7e18c9b640 1 -- 172.21.15.89:0/578885602 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e14159880 con 0x7f7e000039d0 2023-10-30T12:29:01.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f7e18c9b640 1 -- 172.21.15.89:0/578885602 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e14159ae0 con 0x7f7e00005190 2023-10-30T12:29:01.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f7e18c9b640 1 -- 172.21.15.89:0/578885602 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0000bf80 con 0x7f7e000047d0 2023-10-30T12:29:01.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f7e19c9d640 1 --2- 172.21.15.89:0/578885602 >> v2:172.21.15.89:3300/0 conn(0x7f7e140c8cf0 0x7f7e140c9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:29:01.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f7e1949c640 1 --2- 172.21.15.89:0/578885602 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e000047d0 0x7f7e00004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f7e1a49e640 1 --2- 172.21.15.89:0/578885602 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e00005190 0x7f7e0000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f7e1949c640 1 -- 172.21.15.89:0/578885602 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e00005190 msgr2=0x7f7e0000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:01.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f7e1949c640 1 --2- 172.21.15.89:0/578885602 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e00005190 0x7f7e0000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f7e1949c640 1 -- 172.21.15.89:0/578885602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e000039d0 msgr2=0x7f7e00003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:01.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f7e1949c640 1 --2- 172.21.15.89:0/578885602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e000039d0 0x7f7e00003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f7e1949c640 1 -- 172.21.15.89:0/578885602 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e00003530 con 0x7f7e000047d0 2023-10-30T12:29:01.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f7e1a49e640 1 --2- 172.21.15.89:0/578885602 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e00005190 0x7f7e0000b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:01.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f7e1a49e640 1 -- 172.21.15.89:0/578885602 reap_dead start 2023-10-30T12:29:01.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f8a7ddd3640 1 Processor -- start 2023-10-30T12:29:01.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.484+0000 7f8a7ddd3640 1 -- start start 2023-10-30T12:29:01.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.488+0000 7ff9667fc640 1 -- 172.21.15.89:0/868636229 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff96c0add50 con 0x7ff9700d4de0 2023-10-30T12:29:01.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.488+0000 7f8a7ddd3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a78069e90 0x7f8a780c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.488+0000 7f8a7ddd3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a7806ad90 0x7f8a78142310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.488+0000 7f8a7ddd3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a78152de0 0x7f8a781fa6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.488+0000 7f8a777fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a78069e90 0x7f8a780c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.488+0000 7f8a7ddd3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7814de10 con 0x7f8a78069e90 2023-10-30T12:29:01.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.488+0000 7f8a7ddd3640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a780ca210 con 0x7f8a78152de0 2023-10-30T12:29:01.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.488+0000 7f8a7ddd3640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a780ca590 con 0x7f8a7806ad90 2023-10-30T12:29:01.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.488+0000 7f8a777fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a78069e90 0x7f8a780c8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39180/0 (socket says 172.21.15.89:39180) 2023-10-30T12:29:01.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.488+0000 7f8a777fe640 1 -- 172.21.15.89:0/2591068123 learned_addr learned my addr 172.21.15.89:0/2591068123 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:01.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.488+0000 7f8a76ffd640 1 --2- 172.21.15.89:0/2591068123 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a7806ad90 0x7f8a78142310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.488+0000 7f8a77fff640 1 --2- 172.21.15.89:0/2591068123 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a78152de0 0x7f8a781fa6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.488+0000 7ff9667fc640 1 --2- 172.21.15.89:0/868636229 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff960066570 0x7ff960068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.488+0000 7f8a777fe640 1 -- 172.21.15.89:0/2591068123 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a7806ad90 msgr2=0x7f8a78142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.488+0000 7ff9667fc640 1 -- 172.21.15.89:0/868636229 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff96c126960 con 0x7ff9700d4de0 2023-10-30T12:29:01.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.488+0000 7ff97d4b8640 1 --2- 172.21.15.89:0/868636229 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff960066570 0x7ff960068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.488+0000 7f8a777fe640 1 --2- 172.21.15.89:0/2591068123 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a7806ad90 0x7f8a78142310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.488+0000 7f8a777fe640 1 -- 172.21.15.89:0/2591068123 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a78152de0 msgr2=0x7f8a781fa6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.488+0000 7f8a777fe640 1 --2- 172.21.15.89:0/2591068123 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a78152de0 0x7f8a781fa6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.488+0000 7f8a777fe640 1 -- 172.21.15.89:0/2591068123 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a6c00a5c0 con 0x7f8a78069e90 2023-10-30T12:29:01.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.488+0000 7f8a76ffd640 1 --2- 172.21.15.89:0/2591068123 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a7806ad90 0x7f8a78142310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:29:01.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.488+0000 7f8a777fe640 1 --2- 172.21.15.89:0/2591068123 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a78069e90 0x7f8a780c8c10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c01a290 tx=0x7f8a6c01a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.488+0000 7ff97d4b8640 1 --2- 172.21.15.89:0/868636229 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff960066570 0x7ff960068a30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff968006fd0 tx=0x7ff968008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.488+0000 7f7e1949c640 1 --2- 172.21.15.89:0/578885602 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e000047d0 0x7f7e00004c50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7e04009ae0 tx=0x7f7e0401ca20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2311043e58179b30 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.488+0000 7f7e18c9b640 1 -- 172.21.15.89:0/578885602 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e040a55f0 con 0x7f7e000047d0 2023-10-30T12:29:01.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.488+0000 7f7e18c9b640 1 -- 172.21.15.89:0/578885602 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e040a5750 con 0x7f7e000047d0 2023-10-30T12:29:01.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.488+0000 7f7e18c9b640 1 -- 172.21.15.89:0/578885602 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e040a5b00 con 0x7f7e000047d0 2023-10-30T12:29:01.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.492+0000 7f7e1bf01640 1 -- 172.21.15.89:0/578885602 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e000047d0 msgr2=0x7f7e00004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.492+0000 7f7e1bf01640 1 --2- 172.21.15.89:0/578885602 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e000047d0 0x7f7e00004c50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7e04009ae0 tx=0x7f7e0401ca20 comp rx=0 tx=0).stop 2023-10-30T12:29:01.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.492+0000 7f7e1bf01640 1 -- 172.21.15.89:0/578885602 shutdown_connections 2023-10-30T12:29:01.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.492+0000 7f7e1bf01640 1 --2- 172.21.15.89:0/578885602 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e000047d0 0x7f7e00004c50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.492+0000 7f7e1bf01640 1 -- 172.21.15.89:0/578885602 >> 172.21.15.89:0/578885602 conn(0x7f7e1406be40 msgr2=0x7f7e1406c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:01.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.492+0000 7f8a5ffff640 1 -- 172.21.15.89:0/2591068123 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c0b3a20 con 0x7f8a78069e90 2023-10-30T12:29:01.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.492+0000 7f8a5ffff640 1 -- 172.21.15.89:0/2591068123 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a6c0b3b80 con 0x7f8a78069e90 2023-10-30T12:29:01.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.492+0000 7f8a5ffff640 1 -- 172.21.15.89:0/2591068123 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c0bc440 con 0x7f8a78069e90 2023-10-30T12:29:01.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.492+0000 7f8a7ddd3640 1 -- 172.21.15.89:0/2591068123 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a78142850 con 0x7f8a78069e90 2023-10-30T12:29:01.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.492+0000 7f7e1bf01640 1 -- 172.21.15.89:0/578885602 shutdown_connections 2023-10-30T12:29:01.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.492+0000 7ff9667fc640 1 -- 172.21.15.89:0/868636229 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff96c0f3dc0 con 0x7ff9700d4de0 2023-10-30T12:29:01.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.492+0000 7f8a7ddd3640 1 -- 172.21.15.89:0/2591068123 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a780caae0 con 0x7f8a78069e90 2023-10-30T12:29:01.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.492+0000 7f7e1bf01640 1 -- 172.21.15.89:0/578885602 wait complete. 2023-10-30T12:29:01.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.492+0000 7f8a7ddd3640 1 -- 172.21.15.89:0/2591068123 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a7804e170 con 0x7f8a78069e90 2023-10-30T12:29:01.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.496+0000 7f7e1bf01640 1 Processor -- start 2023-10-30T12:29:01.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.496+0000 7f7e1bf01640 1 -- start start 2023-10-30T12:29:01.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.496+0000 7f8a5ffff640 1 -- 172.21.15.89:0/2591068123 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8a6c0a93c0 con 0x7f8a78069e90 2023-10-30T12:29:01.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.496+0000 7f466a7fc640 1 -- 172.21.15.89:0/1817135296 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f466c168360 con 0x7f466c0495b0 2023-10-30T12:29:01.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.496+0000 7f7e1bf01640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e000047d0 0x7f7e140c7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.496+0000 7f8a5ffff640 1 --2- 172.21.15.89:0/2591068123 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8a640664a0 0x7f8a64068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.496+0000 7f8a5ffff640 1 -- 172.21.15.89:0/2591068123 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8a6c11eda0 con 0x7f8a78069e90 2023-10-30T12:29:01.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.496+0000 7f7e1bf01640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e140c7080 0x7f7e140c7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.496+0000 7f7e1bf01640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e141554a0 0x7f7e14155920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.496+0000 7f7e1bf01640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e14155080 con 0x7f7e000047d0 2023-10-30T12:29:01.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.496+0000 7f7e1bf01640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e14155e60 con 0x7f7e141554a0 2023-10-30T12:29:01.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.496+0000 7f7e1bf01640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1415c350 con 0x7f7e140c7080 2023-10-30T12:29:01.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.496+0000 7f8a76ffd640 1 --2- 172.21.15.89:0/2591068123 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8a640664a0 0x7f8a64068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.496+0000 7f7e1949c640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e140c7080 0x7f7e140c7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.496+0000 7f7e19c9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e000047d0 0x7f7e140c7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.496+0000 7f7e19c9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e000047d0 0x7f7e140c7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39186/0 (socket says 172.21.15.89:39186) 2023-10-30T12:29:01.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.496+0000 7f7e19c9d640 1 -- 172.21.15.89:0/2699864873 learned_addr learned my addr 172.21.15.89:0/2699864873 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:01.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.496+0000 7f7e1a49e640 1 --2- 172.21.15.89:0/2699864873 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e141554a0 0x7f7e14155920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.496+0000 7f7e19c9d640 1 -- 172.21.15.89:0/2699864873 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e140c7080 msgr2=0x7f7e140c7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.496+0000 7f7e19c9d640 1 --2- 172.21.15.89:0/2699864873 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e140c7080 0x7f7e140c7500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.496+0000 7f7e19c9d640 1 -- 172.21.15.89:0/2699864873 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e141554a0 msgr2=0x7f7e14155920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.496+0000 7f7e19c9d640 1 --2- 172.21.15.89:0/2699864873 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e141554a0 0x7f7e14155920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.496+0000 7f7e19c9d640 1 -- 172.21.15.89:0/2699864873 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1415bab0 con 0x7f7e000047d0 2023-10-30T12:29:01.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.500+0000 7f8a76ffd640 1 --2- 172.21.15.89:0/2591068123 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8a640664a0 0x7f8a64068960 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8a78143d00 tx=0x7f8a680047f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.500+0000 7f7e19c9d640 1 --2- 172.21.15.89:0/2699864873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e000047d0 0x7f7e140c7d70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7e1000eb60 tx=0x7f7e1000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.500+0000 7f7e0a7fc640 1 -- 172.21.15.89:0/2699864873 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1000ce40 con 0x7f7e000047d0 2023-10-30T12:29:01.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.500+0000 7f7e0a7fc640 1 -- 172.21.15.89:0/2699864873 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e1000ec50 con 0x7f7e000047d0 2023-10-30T12:29:01.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.500+0000 7f7e0a7fc640 1 -- 172.21.15.89:0/2699864873 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e10013470 con 0x7f7e000047d0 2023-10-30T12:29:01.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.500+0000 7f7e1bf01640 1 -- 172.21.15.89:0/2699864873 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e1415bd90 con 0x7f7e000047d0 2023-10-30T12:29:01.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.500+0000 7f8a5ffff640 1 -- 172.21.15.89:0/2591068123 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8a6c0ec3e0 con 0x7f8a78069e90 2023-10-30T12:29:01.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.500+0000 7f7e1bf01640 1 -- 172.21.15.89:0/2699864873 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e14154ca0 con 0x7f7e000047d0 2023-10-30T12:29:01.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.500+0000 7f7e1bf01640 1 -- 172.21.15.89:0/2699864873 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e141552f0 con 0x7f7e000047d0 2023-10-30T12:29:01.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.504+0000 7f7e0a7fc640 1 -- 172.21.15.89:0/2699864873 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7e100135d0 con 0x7f7e000047d0 2023-10-30T12:29:01.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.504+0000 7f7e0a7fc640 1 --2- 172.21.15.89:0/2699864873 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f7df80664f0 0x7f7df80689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.504+0000 7f7e0a7fc640 1 -- 172.21.15.89:0/2699864873 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e100842d0 con 0x7f7e000047d0 2023-10-30T12:29:01.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.508+0000 7f7e1949c640 1 --2- 172.21.15.89:0/2699864873 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f7df80664f0 0x7f7df80689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.508+0000 7f316a7fc640 1 -- 172.21.15.89:0/3093624752 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f31640510b0 con 0x7f317406d570 2023-10-30T12:29:01.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.508+0000 7f7e1949c640 1 --2- 172.21.15.89:0/2699864873 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f7df80664f0 0x7f7df80689b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7e040af000 tx=0x7f7e04008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.508+0000 7f319dffb640 1 -- 172.21.15.89:0/1127765517 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31980074e0 con 0x7f31a004b0e0 2023-10-30T12:29:01.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.508+0000 7f319dffb640 1 -- 172.21.15.89:0/1127765517 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3198007640 con 0x7f31a004b0e0 2023-10-30T12:29:01.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.508+0000 7f319dffb640 1 -- 172.21.15.89:0/1127765517 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3198004a30 con 0x7f31a004b0e0 2023-10-30T12:29:01.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.508+0000 7f31af73e640 1 -- 172.21.15.89:0/1127765517 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a00de100 con 0x7f31a004b0e0 2023-10-30T12:29:01.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.508+0000 7f31af73e640 1 -- 172.21.15.89:0/1127765517 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a00ccc50 con 0x7f31a004b0e0 2023-10-30T12:29:01.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.508+0000 7f31937fe640 1 -- 172.21.15.89:0/1127765517 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3188005260 con 0x7f31a004b0e0 2023-10-30T12:29:01.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.512+0000 7f319dffb640 1 -- 172.21.15.89:0/1127765517 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3198004b90 con 0x7f31a004b0e0 2023-10-30T12:29:01.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.512+0000 7f319dffb640 1 --2- 172.21.15.89:0/1127765517 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f318c0664a0 0x7f318c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.512+0000 7f319dffb640 1 -- 172.21.15.89:0/1127765517 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3198083090 con 0x7f31a004b0e0 2023-10-30T12:29:01.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.512+0000 7f31ad4da640 1 --2- 172.21.15.89:0/1127765517 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f318c0664a0 0x7f318c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.512+0000 7f7e0a7fc640 1 -- 172.21.15.89:0/2699864873 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7e100516a0 con 0x7f7e000047d0 2023-10-30T12:29:01.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.516+0000 7f2ee6fc5640 1 Processor -- start 2023-10-30T12:29:01.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.516+0000 7f31ad4da640 1 --2- 172.21.15.89:0/1127765517 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f318c0664a0 0x7f318c068960 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f31a40b6000 tx=0x7f31a40ae070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.516+0000 7f2ee6fc5640 1 -- start start 2023-10-30T12:29:01.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.516+0000 7f2ee6fc5640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2ee00c4430 0x7f2ee00ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.516+0000 7f2ee6fc5640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee00cb780 con 0x7f2ee0159830 2023-10-30T12:29:01.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.516+0000 7f2ee6fc5640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee015a2e0 con 0x7f2ee0153c60 2023-10-30T12:29:01.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.516+0000 7f2ee6fc5640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ee0154040 con 0x7f2ee00c4430 2023-10-30T12:29:01.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.516+0000 7f2edffff640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2ee00c4430 0x7f2ee00ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.516+0000 7f2ee4d61640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f2ee0153c60 0x7f2ee00cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:36118/0 (socket says 172.21.15.89:36118) 2023-10-30T12:29:01.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.516+0000 7f2edffff640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2ee00c4430 0x7f2ee00ca360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:59476/0 (socket says 172.21.15.89:59476) 2023-10-30T12:29:01.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.516+0000 7f2edffff640 1 -- 172.21.15.89:0/2640864535 learned_addr learned my addr 172.21.15.89:0/2640864535 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:01.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.520+0000 7f2edffff640 1 -- 172.21.15.89:0/2640864535 >> v1:172.21.15.192:6789/0 conn(0x7f2ee0153c60 legacy=0x7f2ee00cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:01.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.520+0000 7f2edffff640 1 -- 172.21.15.89:0/2640864535 >> v1:172.21.15.89:6789/0 conn(0x7f2ee0159830 legacy=0x7f2ee00cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.520+0000 7f2edffff640 1 -- 172.21.15.89:0/2640864535 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee015a5c0 con 0x7f2ee00c4430 2023-10-30T12:29:01.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.524+0000 7f2edffff640 1 --2- 172.21.15.89:0/2640864535 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ee00c4430 0x7f2ee00ca360 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc009870 tx=0x7f2ecc01c500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=967168b1855d7dab server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.528+0000 7f2edf7fe640 1 -- 172.21.15.89:0/2640864535 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc0ab600 con 0x7f2ee00c4430 2023-10-30T12:29:01.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.528+0000 7f2edf7fe640 1 -- 172.21.15.89:0/2640864535 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ecc0a7050 con 0x7f2ee00c4430 2023-10-30T12:29:01.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.528+0000 7f2edf7fe640 1 -- 172.21.15.89:0/2640864535 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc0ab980 con 0x7f2ee00c4430 2023-10-30T12:29:01.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.528+0000 7f2ee6fc5640 1 -- 172.21.15.89:0/2640864535 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ee00c4430 msgr2=0x7f2ee00ca360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.528+0000 7f2ee6fc5640 1 --2- 172.21.15.89:0/2640864535 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ee00c4430 0x7f2ee00ca360 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc009870 tx=0x7f2ecc01c500 comp rx=0 tx=0).stop 2023-10-30T12:29:01.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.528+0000 7f319dffb640 1 -- 172.21.15.89:0/1127765517 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3198050530 con 0x7f31a004b0e0 2023-10-30T12:29:01.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.528+0000 7f2ee6fc5640 1 -- 172.21.15.89:0/2640864535 shutdown_connections 2023-10-30T12:29:01.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.528+0000 7f2ee6fc5640 1 --2- 172.21.15.89:0/2640864535 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ee00c4430 0x7f2ee00ca360 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.528+0000 7f2ee6fc5640 1 -- 172.21.15.89:0/2640864535 >> 172.21.15.89:0/2640864535 conn(0x7f2ee006bfe0 msgr2=0x7f2ee006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:01.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.528+0000 7f2ee6fc5640 1 -- 172.21.15.89:0/2640864535 shutdown_connections 2023-10-30T12:29:01.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.528+0000 7f2ee6fc5640 1 -- 172.21.15.89:0/2640864535 wait complete. 2023-10-30T12:29:01.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.532+0000 7f2ee6fc5640 1 Processor -- start 2023-10-30T12:29:01.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.532+0000 7f2ee6fc5640 1 -- start start 2023-10-30T12:29:01.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.532+0000 7f2ee6fc5640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ee00c4430 0x7f2ee014f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.532+0000 7f2ee6fc5640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ee0153c60 0x7f2ee01ff710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.532+0000 7f2ee6fc5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ee0159830 0x7f2ee00ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.536+0000 7f2edffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ee0153c60 0x7f2ee01ff710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.536+0000 7f2edffff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ee0153c60 0x7f2ee01ff710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:59486/0 (socket says 172.21.15.89:59486) 2023-10-30T12:29:01.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.536+0000 7f2ee4d61640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ee00c4430 0x7f2ee014f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.536+0000 7f2ee4d61640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ee00c4430 0x7f2ee014f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:57412/0 (socket says 172.21.15.89:57412) 2023-10-30T12:29:01.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.536+0000 7f2edffff640 1 -- 172.21.15.89:0/2647807726 learned_addr learned my addr 172.21.15.89:0/2647807726 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:01.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.536+0000 7f2ee6fc5640 1 -- 172.21.15.89:0/2647807726 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee01fef30 con 0x7f2ee0159830 2023-10-30T12:29:01.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.536+0000 7f2ee5562640 1 --2- 172.21.15.89:0/2647807726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ee0159830 0x7f2ee00ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.536+0000 7f2ee6fc5640 1 -- 172.21.15.89:0/2647807726 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee00ced30 con 0x7f2ee0153c60 2023-10-30T12:29:01.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.536+0000 7f2ee6fc5640 1 -- 172.21.15.89:0/2647807726 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee00c7b90 con 0x7f2ee00c4430 2023-10-30T12:29:01.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.536+0000 7f2ee5562640 1 -- 172.21.15.89:0/2647807726 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ee00c4430 msgr2=0x7f2ee014f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.536+0000 7f2ee5562640 1 --2- 172.21.15.89:0/2647807726 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ee00c4430 0x7f2ee014f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.536+0000 7f2ee5562640 1 -- 172.21.15.89:0/2647807726 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ee0153c60 msgr2=0x7f2ee01ff710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.536+0000 7f2ee5562640 1 --2- 172.21.15.89:0/2647807726 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ee0153c60 0x7f2ee01ff710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.536+0000 7f2ee5562640 1 -- 172.21.15.89:0/2647807726 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ecc0094d0 con 0x7f2ee0159830 2023-10-30T12:29:01.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.536+0000 7f2ee5562640 1 --2- 172.21.15.89:0/2647807726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ee0159830 0x7f2ee00ccfb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2ed800c730 tx=0x7f2ed800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.540+0000 7f2edd7fa640 1 -- 172.21.15.89:0/2647807726 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8006d60 con 0x7f2ee0159830 2023-10-30T12:29:01.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.540+0000 7f2edd7fa640 1 -- 172.21.15.89:0/2647807726 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed8003980 con 0x7f2ee0159830 2023-10-30T12:29:01.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.540+0000 7f2edd7fa640 1 -- 172.21.15.89:0/2647807726 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8006d60 con 0x7f2ee0159830 2023-10-30T12:29:01.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.540+0000 7f2ee6fc5640 1 -- 172.21.15.89:0/2647807726 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ee00cea50 con 0x7f2ee0159830 2023-10-30T12:29:01.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.540+0000 7f2ee6fc5640 1 -- 172.21.15.89:0/2647807726 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ee0203230 con 0x7f2ee0159830 2023-10-30T12:29:01.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.540+0000 7f2ee6fc5640 1 -- 172.21.15.89:0/2647807726 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ee01ff430 con 0x7f2ee0159830 2023-10-30T12:29:01.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.548+0000 7f2edd7fa640 1 -- 172.21.15.89:0/2647807726 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2ed8003b30 con 0x7f2ee0159830 2023-10-30T12:29:01.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.552+0000 7f2edd7fa640 1 --2- 172.21.15.89:0/2647807726 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2ec4066530 0x7f2ec40689f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.552+0000 7f2ee4d61640 1 --2- 172.21.15.89:0/2647807726 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2ec4066530 0x7f2ec40689f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.556+0000 7f2edd7fa640 1 -- 172.21.15.89:0/2647807726 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ed8055260 con 0x7f2ee0159830 2023-10-30T12:29:01.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.556+0000 7f2ee4d61640 1 --2- 172.21.15.89:0/2647807726 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2ec4066530 0x7f2ec40689f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2ed4007240 tx=0x7f2ed4007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.556+0000 7f2edd7fa640 1 -- 172.21.15.89:0/2647807726 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2ed80515a0 con 0x7f2ee0159830 2023-10-30T12:29:01.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.740+0000 7ff92abdd640 1 -- 172.21.15.89:0/4032853748 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7ff91c04bda0 con 0x7ff91c0d4e20 2023-10-30T12:29:01.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.740+0000 7ff91a7fc640 1 -- 172.21.15.89:0/4032853748 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff9200eb820 con 0x7ff91c0d4e20 2023-10-30T12:29:01.747 INFO:teuthology.orchestra.run.smithi089.stdout:60129542151 2023-10-30T12:29:01.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.748+0000 7ff92abdd640 1 -- 172.21.15.89:0/4032853748 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff908066570 msgr2=0x7ff908068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.748+0000 7ff92abdd640 1 --2- 172.21.15.89:0/4032853748 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff908066570 0x7ff908068a30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff914009bd0 tx=0x7ff914008040 comp rx=0 tx=0).stop 2023-10-30T12:29:01.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.748+0000 7ff92abdd640 1 -- 172.21.15.89:0/4032853748 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff91c0d4e20 msgr2=0x7ff91c0c51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.748+0000 7ff92abdd640 1 --2- 172.21.15.89:0/4032853748 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff91c0d4e20 0x7ff91c0c51a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff9200aa000 tx=0x7ff92000abe0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.748+0000 7ff92abdd640 1 -- 172.21.15.89:0/4032853748 shutdown_connections 2023-10-30T12:29:01.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.748+0000 7ff92abdd640 1 --2- 172.21.15.89:0/4032853748 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff908066570 0x7ff908068a30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.748+0000 7ff92abdd640 1 --2- 172.21.15.89:0/4032853748 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff91c04b740 0x7ff91c0c4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.748+0000 7ff92abdd640 1 --2- 172.21.15.89:0/4032853748 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff91c0d4e20 0x7ff91c0c51a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.748+0000 7ff92abdd640 1 --2- 172.21.15.89:0/4032853748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff91c034dc0 0x7ff91c0c5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.748+0000 7ff92abdd640 1 -- 172.21.15.89:0/4032853748 >> 172.21.15.89:0/4032853748 conn(0x7ff91c0502e0 msgr2=0x7ff91c035e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:01.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.748+0000 7ff92abdd640 1 -- 172.21.15.89:0/4032853748 shutdown_connections 2023-10-30T12:29:01.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.748+0000 7ff92abdd640 1 -- 172.21.15.89:0/4032853748 wait complete. 2023-10-30T12:29:01.785 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.0 2023-10-30T12:29:01.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.808+0000 7f7e1bf01640 1 -- 172.21.15.89:0/2699864873 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f7e14154ae0 con 0x7f7e000047d0 2023-10-30T12:29:01.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.808+0000 7f7e0a7fc640 1 -- 172.21.15.89:0/2699864873 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7e10054cc0 con 0x7f7e000047d0 2023-10-30T12:29:01.815 INFO:teuthology.orchestra.run.smithi089.stdout:60129542151 2023-10-30T12:29:01.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.812+0000 7f7e1bf01640 1 -- 172.21.15.89:0/2699864873 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f7df80664f0 msgr2=0x7f7df80689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.812+0000 7f7e1bf01640 1 --2- 172.21.15.89:0/2699864873 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f7df80664f0 0x7f7df80689b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7e040af000 tx=0x7f7e04008040 comp rx=0 tx=0).stop 2023-10-30T12:29:01.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.812+0000 7f7e1bf01640 1 -- 172.21.15.89:0/2699864873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e000047d0 msgr2=0x7f7e140c7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.812+0000 7f7e1bf01640 1 --2- 172.21.15.89:0/2699864873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e000047d0 0x7f7e140c7d70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7e1000eb60 tx=0x7f7e1000c520 comp rx=0 tx=0).stop 2023-10-30T12:29:01.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.812+0000 7f7e1bf01640 1 -- 172.21.15.89:0/2699864873 shutdown_connections 2023-10-30T12:29:01.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.812+0000 7f7e1bf01640 1 --2- 172.21.15.89:0/2699864873 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f7df80664f0 0x7f7df80689b0 secure :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7e040af000 tx=0x7f7e04008040 comp rx=0 tx=0).stop 2023-10-30T12:29:01.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.812+0000 7f7e1bf01640 1 --2- 172.21.15.89:0/2699864873 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e141554a0 0x7f7e14155920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.812+0000 7f7e1bf01640 1 --2- 172.21.15.89:0/2699864873 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e140c7080 0x7f7e140c7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.812+0000 7f7e1bf01640 1 --2- 172.21.15.89:0/2699864873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e000047d0 0x7f7e140c7d70 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.812+0000 7f7e1bf01640 1 -- 172.21.15.89:0/2699864873 >> 172.21.15.89:0/2699864873 conn(0x7f7e1406be40 msgr2=0x7f7e140b2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:01.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.812+0000 7f7e1bf01640 1 -- 172.21.15.89:0/2699864873 shutdown_connections 2023-10-30T12:29:01.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.812+0000 7f7e1bf01640 1 -- 172.21.15.89:0/2699864873 wait complete. 2023-10-30T12:29:01.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.812+0000 7ff97ff1d640 1 -- 172.21.15.89:0/868636229 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7ff970032dc0 con 0x7ff9700d4de0 2023-10-30T12:29:01.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.812+0000 7ff9667fc640 1 -- 172.21.15.89:0/868636229 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff96c0f73e0 con 0x7ff9700d4de0 2023-10-30T12:29:01.822 INFO:teuthology.orchestra.run.smithi089.stdout:60129542151 2023-10-30T12:29:01.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.816+0000 7ff97ff1d640 1 -- 172.21.15.89:0/868636229 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff960066570 msgr2=0x7ff960068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.816+0000 7ff97ff1d640 1 --2- 172.21.15.89:0/868636229 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff960066570 0x7ff960068a30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff968006fd0 tx=0x7ff968008040 comp rx=0 tx=0).stop 2023-10-30T12:29:01.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.816+0000 7ff97ff1d640 1 -- 172.21.15.89:0/868636229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9700d4de0 msgr2=0x7ff9700d7330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.816+0000 7ff97ff1d640 1 --2- 172.21.15.89:0/868636229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9700d4de0 0x7ff9700d7330 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff96c0b3000 tx=0x7ff96c021970 comp rx=0 tx=0).stop 2023-10-30T12:29:01.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.816+0000 7ff97ff1d640 1 -- 172.21.15.89:0/868636229 shutdown_connections 2023-10-30T12:29:01.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.816+0000 7ff97ff1d640 1 --2- 172.21.15.89:0/868636229 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff960066570 0x7ff960068a30 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff968006fd0 tx=0x7ff968008040 comp rx=0 tx=0).stop 2023-10-30T12:29:01.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.816+0000 7ff97ff1d640 1 --2- 172.21.15.89:0/868636229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9700d4de0 0x7ff9700d7330 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.816+0000 7ff97ff1d640 1 --2- 172.21.15.89:0/868636229 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff970050ec0 0x7ff97016c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.816+0000 7ff97ff1d640 1 --2- 172.21.15.89:0/868636229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9700471c0 0x7ff9700cb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.816+0000 7ff97ff1d640 1 -- 172.21.15.89:0/868636229 >> 172.21.15.89:0/868636229 conn(0x7ff9700503a0 msgr2=0x7ff970050780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:01.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.820+0000 7ff97ff1d640 1 -- 172.21.15.89:0/868636229 shutdown_connections 2023-10-30T12:29:01.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.820+0000 7ff97ff1d640 1 -- 172.21.15.89:0/868636229 wait complete. 2023-10-30T12:29:01.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.824+0000 7f8a7ddd3640 1 -- 172.21.15.89:0/2591068123 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f8a781555e0 con 0x7f8a78069e90 2023-10-30T12:29:01.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.824+0000 7f8a5ffff640 1 -- 172.21.15.89:0/2591068123 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8a6c0efa00 con 0x7f8a78069e90 2023-10-30T12:29:01.830 INFO:teuthology.orchestra.run.smithi089.stdout:60129542151 2023-10-30T12:29:01.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.828+0000 7f467acdc640 1 -- 172.21.15.89:0/1817135296 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f466c168360 con 0x7f466c0495b0 2023-10-30T12:29:01.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.828+0000 7f8a7ddd3640 1 -- 172.21.15.89:0/2591068123 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8a640664a0 msgr2=0x7f8a64068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.828+0000 7f8a7ddd3640 1 --2- 172.21.15.89:0/2591068123 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8a640664a0 0x7f8a64068960 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8a78143d00 tx=0x7f8a680047f0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.828+0000 7f8a7ddd3640 1 -- 172.21.15.89:0/2591068123 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a78069e90 msgr2=0x7f8a780c8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.828+0000 7f8a7ddd3640 1 --2- 172.21.15.89:0/2591068123 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a78069e90 0x7f8a780c8c10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c01a290 tx=0x7f8a6c01a420 comp rx=0 tx=0).stop 2023-10-30T12:29:01.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.828+0000 7f466a7fc640 1 -- 172.21.15.89:0/1817135296 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f466c168360 con 0x7f466c0495b0 2023-10-30T12:29:01.835 INFO:teuthology.orchestra.run.smithi089.stdout:60129542150 2023-10-30T12:29:01.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.828+0000 7f8a7ddd3640 1 -- 172.21.15.89:0/2591068123 shutdown_connections 2023-10-30T12:29:01.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.828+0000 7f8a7ddd3640 1 --2- 172.21.15.89:0/2591068123 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8a640664a0 0x7f8a64068960 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.828+0000 7f8a7ddd3640 1 --2- 172.21.15.89:0/2591068123 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a78152de0 0x7f8a781fa6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.828+0000 7f8a7ddd3640 1 --2- 172.21.15.89:0/2591068123 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a7806ad90 0x7f8a78142310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.828+0000 7f8a7ddd3640 1 --2- 172.21.15.89:0/2591068123 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a78069e90 0x7f8a780c8c10 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.828+0000 7f8a7ddd3640 1 -- 172.21.15.89:0/2591068123 >> 172.21.15.89:0/2591068123 conn(0x7f8a780c7910 msgr2=0x7f8a78155f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:01.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.828+0000 7f8a7ddd3640 1 -- 172.21.15.89:0/2591068123 shutdown_connections 2023-10-30T12:29:01.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.828+0000 7f8a7ddd3640 1 -- 172.21.15.89:0/2591068123 wait complete. 2023-10-30T12:29:01.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.832+0000 7f317acdc640 1 -- 172.21.15.89:0/3093624752 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f31741f6f10 con 0x7f317406d570 2023-10-30T12:29:01.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.832+0000 7f467acdc640 1 -- 172.21.15.89:0/1817135296 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f46580664a0 msgr2=0x7f4658068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.832+0000 7f467acdc640 1 --2- 172.21.15.89:0/1817135296 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f46580664a0 0x7f4658068960 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f466c0dca30 tx=0x7f4664008040 comp rx=0 tx=0).stop 2023-10-30T12:29:01.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.832+0000 7f467acdc640 1 -- 172.21.15.89:0/1817135296 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f466c0495b0 msgr2=0x7f466c175580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.832+0000 7f467acdc640 1 --2- 172.21.15.89:0/1817135296 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f466c0495b0 0x7f466c175580 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f467407ad90 tx=0x7f4674066e60 comp rx=0 tx=0).stop 2023-10-30T12:29:01.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.832+0000 7f467acdc640 1 -- 172.21.15.89:0/1817135296 shutdown_connections 2023-10-30T12:29:01.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.832+0000 7f467acdc640 1 --2- 172.21.15.89:0/1817135296 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f46580664a0 0x7f4658068960 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f466c0dca30 tx=0x7f4664008040 comp rx=0 tx=0).stop 2023-10-30T12:29:01.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.832+0000 7f467acdc640 1 --2- 172.21.15.89:0/1817135296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f466c0e2880 0x7f466c168d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.832+0000 7f467acdc640 1 --2- 172.21.15.89:0/1817135296 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f466c0e24a0 0x7f466c0cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.832+0000 7f467acdc640 1 --2- 172.21.15.89:0/1817135296 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f466c0495b0 0x7f466c175580 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.832+0000 7f467acdc640 1 -- 172.21.15.89:0/1817135296 >> 172.21.15.89:0/1817135296 conn(0x7f466c054710 msgr2=0x7f466c0d7500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:01.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.832+0000 7f316a7fc640 1 -- 172.21.15.89:0/3093624752 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3164017030 con 0x7f317406d570 2023-10-30T12:29:01.840 INFO:teuthology.orchestra.run.smithi089.stdout:60129542151 2023-10-30T12:29:01.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.832+0000 7f467acdc640 1 -- 172.21.15.89:0/1817135296 shutdown_connections 2023-10-30T12:29:01.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.832+0000 7f317acdc640 1 -- 172.21.15.89:0/3093624752 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f31580664f0 msgr2=0x7f31580689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.832+0000 7f317acdc640 1 --2- 172.21.15.89:0/3093624752 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f31580664f0 0x7f31580689b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f316c001570 tx=0x7f316c009040 comp rx=0 tx=0).stop 2023-10-30T12:29:01.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.836+0000 7f317acdc640 1 -- 172.21.15.89:0/3093624752 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f317406d570 msgr2=0x7f3174148d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.836+0000 7f317acdc640 1 --2- 172.21.15.89:0/3093624752 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f317406d570 0x7f3174148d70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f316400c410 tx=0x7f316400c8d0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.836+0000 7f317acdc640 1 -- 172.21.15.89:0/3093624752 shutdown_connections 2023-10-30T12:29:01.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.836+0000 7f317acdc640 1 --2- 172.21.15.89:0/3093624752 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f31580664f0 0x7f31580689b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.836+0000 7f317acdc640 1 --2- 172.21.15.89:0/3093624752 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f317406d570 0x7f3174148d70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.836+0000 7f317acdc640 1 --2- 172.21.15.89:0/3093624752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f317414fb40 0x7f31741484d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.836+0000 7f317acdc640 1 --2- 172.21.15.89:0/3093624752 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31740c6f00 0x7f317414b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.836+0000 7f317acdc640 1 -- 172.21.15.89:0/3093624752 >> 172.21.15.89:0/3093624752 conn(0x7f31740cae10 msgr2=0x7f31740c4270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:01.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.836+0000 7f467acdc640 1 -- 172.21.15.89:0/1817135296 wait complete. 2023-10-30T12:29:01.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.836+0000 7f317acdc640 1 -- 172.21.15.89:0/3093624752 shutdown_connections 2023-10-30T12:29:01.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.836+0000 7f317acdc640 1 -- 172.21.15.89:0/3093624752 wait complete. 2023-10-30T12:29:01.845 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.2 2023-10-30T12:29:01.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.852+0000 7f31937fe640 1 -- 172.21.15.89:0/1127765517 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f3188005510 con 0x7f31a004b0e0 2023-10-30T12:29:01.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.852+0000 7f319dffb640 1 -- 172.21.15.89:0/1127765517 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3198018020 con 0x7f31a004b0e0 2023-10-30T12:29:01.857 INFO:teuthology.orchestra.run.smithi089.stdout:60129542151 2023-10-30T12:29:01.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.856+0000 7f31937fe640 1 -- 172.21.15.89:0/1127765517 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f318c0664a0 msgr2=0x7f318c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.856+0000 7f31937fe640 1 --2- 172.21.15.89:0/1127765517 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f318c0664a0 0x7f318c068960 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f31a40b6000 tx=0x7f31a40ae070 comp rx=0 tx=0).stop 2023-10-30T12:29:01.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.856+0000 7f31937fe640 1 -- 172.21.15.89:0/1127765517 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31a004b0e0 msgr2=0x7f31a00c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.856+0000 7f31937fe640 1 --2- 172.21.15.89:0/1127765517 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31a004b0e0 0x7f31a00c51f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f319800c530 tx=0x7f319800c9f0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.856+0000 7f31937fe640 1 -- 172.21.15.89:0/1127765517 shutdown_connections 2023-10-30T12:29:01.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.856+0000 7f31937fe640 1 --2- 172.21.15.89:0/1127765517 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f318c0664a0 0x7f318c068960 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.856+0000 7f31937fe640 1 --2- 172.21.15.89:0/1127765517 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31a004b0e0 0x7f31a00c51f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.856+0000 7f31937fe640 1 --2- 172.21.15.89:0/1127765517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31a00d1650 0x7f31a00de890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.856+0000 7f31937fe640 1 --2- 172.21.15.89:0/1127765517 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31a0005ee0 0x7f31a00dc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.856+0000 7f31937fe640 1 -- 172.21.15.89:0/1127765517 >> 172.21.15.89:0/1127765517 conn(0x7f31a00501d0 msgr2=0x7f31a00caeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:01.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.856+0000 7f31937fe640 1 -- 172.21.15.89:0/1127765517 shutdown_connections 2023-10-30T12:29:01.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.856+0000 7f31937fe640 1 -- 172.21.15.89:0/1127765517 wait complete. 2023-10-30T12:29:01.864 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.7 2023-10-30T12:29:01.868 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.3 2023-10-30T12:29:01.869 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542150 for osd.4 2023-10-30T12:29:01.876 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.5 2023-10-30T12:29:01.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.872+0000 7f2ee6fc5640 1 -- 172.21.15.89:0/2647807726 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f2ee01ff430 con 0x7f2ee0159830 2023-10-30T12:29:01.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.876+0000 7f2edd7fa640 1 -- 172.21.15.89:0/2647807726 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2ee01ff430 con 0x7f2ee0159830 2023-10-30T12:29:01.880 INFO:teuthology.orchestra.run.smithi089.stdout:60129542151 2023-10-30T12:29:01.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.876+0000 7f2ee6fc5640 1 -- 172.21.15.89:0/2647807726 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2ec4066530 msgr2=0x7f2ec40689f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.876+0000 7f2ee6fc5640 1 --2- 172.21.15.89:0/2647807726 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2ec4066530 0x7f2ec40689f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2ed4007240 tx=0x7f2ed4007e70 comp rx=0 tx=0).stop 2023-10-30T12:29:01.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.876+0000 7f2ee6fc5640 1 -- 172.21.15.89:0/2647807726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ee0159830 msgr2=0x7f2ee00ccfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.876+0000 7f2ee6fc5640 1 --2- 172.21.15.89:0/2647807726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ee0159830 0x7f2ee00ccfb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2ed800c730 tx=0x7f2ed800cbf0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.884 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.6 2023-10-30T12:29:01.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.876+0000 7f2ee6fc5640 1 -- 172.21.15.89:0/2647807726 shutdown_connections 2023-10-30T12:29:01.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.876+0000 7f2ee6fc5640 1 --2- 172.21.15.89:0/2647807726 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2ec4066530 0x7f2ec40689f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.876+0000 7f2ee6fc5640 1 --2- 172.21.15.89:0/2647807726 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ee0153c60 0x7f2ee01ff710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.876+0000 7f2ee6fc5640 1 --2- 172.21.15.89:0/2647807726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ee0159830 0x7f2ee00ccfb0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.876+0000 7f2ee6fc5640 1 --2- 172.21.15.89:0/2647807726 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ee00c4430 0x7f2ee014f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.876+0000 7f2ee6fc5640 1 -- 172.21.15.89:0/2647807726 >> 172.21.15.89:0/2647807726 conn(0x7f2ee006bfe0 msgr2=0x7f2ee0152f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:01.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.876+0000 7f2ee6fc5640 1 -- 172.21.15.89:0/2647807726 shutdown_connections 2023-10-30T12:29:01.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:01.876+0000 7f2ee6fc5640 1 -- 172.21.15.89:0/2647807726 wait complete. 2023-10-30T12:29:01.901 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.1 2023-10-30T12:29:02.787 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-30T12:29:02.846 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-30T12:29:02.864 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-30T12:29:02.869 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-30T12:29:02.871 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-30T12:29:02.877 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-30T12:29:02.885 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-30T12:29:02.902 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-30T12:29:02.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.948+0000 7ff42fe09640 1 Processor -- start 2023-10-30T12:29:02.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.948+0000 7ff42fe09640 1 -- start start 2023-10-30T12:29:02.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.952+0000 7ff42fe09640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff4280cce00 0x7ff4280cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:02.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.952+0000 7ff42fe09640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff4280c7830 0x7ff4280c7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:02.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.952+0000 7ff42fe09640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4280c2220 con 0x7ff42806e430 2023-10-30T12:29:02.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.952+0000 7ff42fe09640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4280c4040 con 0x7ff4280cce00 2023-10-30T12:29:02.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.952+0000 7ff42fe09640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4280cd720 con 0x7ff4280c7830 2023-10-30T12:29:02.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.952+0000 7ff42e3a6640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7ff42806e430 0x7ff42815ac10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:36120/0 (socket says 172.21.15.89:36120) 2023-10-30T12:29:02.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.952+0000 7ff42e3a6640 1 -- 172.21.15.89:0/3218497082 learned_addr learned my addr 172.21.15.89:0/3218497082 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:29:02.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.956+0000 7ff42cba3640 1 -- 172.21.15.89:0/3218497082 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1450628059 0 0) 0x7ff4280c2220 con 0x7ff42806e430 2023-10-30T12:29:02.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.956+0000 7ff42cba3640 1 -- 172.21.15.89:0/3218497082 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff414003530 con 0x7ff42806e430 2023-10-30T12:29:02.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.956+0000 7ff42d3a4640 1 --2- 172.21.15.89:0/3218497082 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff4280c7830 0x7ff4280c7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:02.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.956+0000 7ff42cba3640 1 -- 172.21.15.89:0/3218497082 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 516213522 0 0) 0x7ff414003530 con 0x7ff42806e430 2023-10-30T12:29:02.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.956+0000 7ff42cba3640 1 -- 172.21.15.89:0/3218497082 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff4280c7830 msgr2=0x7ff4280c7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:02.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.956+0000 7ff42cba3640 1 --2- 172.21.15.89:0/3218497082 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff4280c7830 0x7ff4280c7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:02.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.956+0000 7ff42cba3640 1 -- 172.21.15.89:0/3218497082 >> v2:172.21.15.89:3300/0 conn(0x7ff4280cce00 msgr2=0x7ff4280cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:02.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.956+0000 7ff42cba3640 1 --2- 172.21.15.89:0/3218497082 >> v2:172.21.15.89:3300/0 conn(0x7ff4280cce00 0x7ff4280cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:02.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.956+0000 7ff42cba3640 1 -- 172.21.15.89:0/3218497082 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4280c3ac0 con 0x7ff42806e430 2023-10-30T12:29:02.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.956+0000 7ff42cba3640 1 -- 172.21.15.89:0/3218497082 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7ff420001a50 con 0x7ff42806e430 2023-10-30T12:29:02.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.956+0000 7ff42cba3640 1 -- 172.21.15.89:0/3218497082 >> v1:172.21.15.192:6789/0 conn(0x7ff42806e430 legacy=0x7ff42815ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:02.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.956+0000 7ff42cba3640 1 --2- 172.21.15.89:0/3218497082 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4140039a0 0x7ff414003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:02.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.956+0000 7ff42cba3640 1 --2- 172.21.15.89:0/3218497082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4140047d0 0x7ff414004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:02.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.956+0000 7ff42cba3640 1 --2- 172.21.15.89:0/3218497082 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff4280c7830 0x7ff41400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:02.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.956+0000 7ff42cba3640 1 -- 172.21.15.89:0/3218497082 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4280c4040 con 0x7ff4140047d0 2023-10-30T12:29:02.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.956+0000 7ff42cba3640 1 -- 172.21.15.89:0/3218497082 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4280cd720 con 0x7ff4140039a0 2023-10-30T12:29:02.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.956+0000 7ff42cba3640 1 -- 172.21.15.89:0/3218497082 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff414005570 con 0x7ff4280c7830 2023-10-30T12:29:02.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.956+0000 7ff42e3a6640 1 --2- 172.21.15.89:0/3218497082 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff4280c7830 0x7ff41400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:02.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.956+0000 7ff42d3a4640 1 --2- 172.21.15.89:0/3218497082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4140047d0 0x7ff414004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:02.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.956+0000 7ff42dba5640 1 --2- 172.21.15.89:0/3218497082 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4140039a0 0x7ff414003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:02.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.956+0000 7ff42e3a6640 1 -- 172.21.15.89:0/3218497082 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4140039a0 msgr2=0x7ff414003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:02.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.956+0000 7ff42e3a6640 1 --2- 172.21.15.89:0/3218497082 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4140039a0 0x7ff414003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:02.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.956+0000 7ff42e3a6640 1 -- 172.21.15.89:0/3218497082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4140047d0 msgr2=0x7ff414004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:02.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.956+0000 7ff42e3a6640 1 --2- 172.21.15.89:0/3218497082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff4140047d0 0x7ff414004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:02.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.956+0000 7ff42e3a6640 1 -- 172.21.15.89:0/3218497082 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4280c2220 con 0x7ff4280c7830 2023-10-30T12:29:02.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.964+0000 7ff42e3a6640 1 --2- 172.21.15.89:0/3218497082 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff4280c7830 0x7ff41400b990 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff42000a7d0 tx=0x7ff42001d2e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef098b597b14dbb3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:02.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.964+0000 7ff42cba3640 1 -- 172.21.15.89:0/3218497082 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4200ad620 con 0x7ff4280c7830 2023-10-30T12:29:02.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.964+0000 7ff42cba3640 1 -- 172.21.15.89:0/3218497082 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4200a8070 con 0x7ff4280c7830 2023-10-30T12:29:02.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.964+0000 7ff42cba3640 1 -- 172.21.15.89:0/3218497082 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4200ad9e0 con 0x7ff4280c7830 2023-10-30T12:29:02.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.964+0000 7ff42fe09640 1 -- 172.21.15.89:0/3218497082 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff4280c7830 msgr2=0x7ff41400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:02.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.964+0000 7ff42fe09640 1 --2- 172.21.15.89:0/3218497082 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff4280c7830 0x7ff41400b990 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff42000a7d0 tx=0x7ff42001d2e0 comp rx=0 tx=0).stop 2023-10-30T12:29:02.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.964+0000 7ff42e3a6640 1 -- 172.21.15.89:0/3218497082 reap_dead start 2023-10-30T12:29:02.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.964+0000 7ff42fe09640 1 -- 172.21.15.89:0/3218497082 shutdown_connections 2023-10-30T12:29:02.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.964+0000 7ff42fe09640 1 -- 172.21.15.89:0/3218497082 >> 172.21.15.89:0/3218497082 conn(0x7ff428071ea0 msgr2=0x7ff428065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:02.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.964+0000 7ff42fe09640 1 -- 172.21.15.89:0/3218497082 shutdown_connections 2023-10-30T12:29:02.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.964+0000 7ff42fe09640 1 -- 172.21.15.89:0/3218497082 wait complete. 2023-10-30T12:29:02.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.968+0000 7ff42fe09640 1 Processor -- start 2023-10-30T12:29:02.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.968+0000 7ff42fe09640 1 -- start start 2023-10-30T12:29:02.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.972+0000 7ff42fe09640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4280c7830 0x7ff42814be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:02.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.972+0000 7ff42fe09640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff42806cd70 0x7ff42814c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:02.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.972+0000 7ff42fe09640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff42806b310 0x7ff42806b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:02.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.972+0000 7ff42fe09640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42815c410 con 0x7ff42806cd70 2023-10-30T12:29:02.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.972+0000 7ff42fe09640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42806bcd0 con 0x7ff4280c7830 2023-10-30T12:29:02.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.972+0000 7ff42e3a6640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff42806b310 0x7ff42806b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:02.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.972+0000 7ff42e3a6640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff42806b310 0x7ff42806b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:57442/0 (socket says 172.21.15.89:57442) 2023-10-30T12:29:02.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.972+0000 7ff42e3a6640 1 -- 172.21.15.89:0/728312692 learned_addr learned my addr 172.21.15.89:0/728312692 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:02.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.972+0000 7ff42fe09640 1 -- 172.21.15.89:0/728312692 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4280ccad0 con 0x7ff42806b310 2023-10-30T12:29:02.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.972+0000 7ff42dba5640 1 --2- 172.21.15.89:0/728312692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4280c7830 0x7ff42814be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:02.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.972+0000 7ff42d3a4640 1 --2- 172.21.15.89:0/728312692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff42806cd70 0x7ff42814c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:02.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.976+0000 7ff42e3a6640 1 -- 172.21.15.89:0/728312692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4280c7830 msgr2=0x7ff42814be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:02.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.976+0000 7ff42e3a6640 1 --2- 172.21.15.89:0/728312692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4280c7830 0x7ff42814be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:02.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.976+0000 7ff42e3a6640 1 -- 172.21.15.89:0/728312692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff42806cd70 msgr2=0x7ff42814c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:02.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.976+0000 7ff42e3a6640 1 --2- 172.21.15.89:0/728312692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff42806cd70 0x7ff42814c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:02.982 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.976+0000 7ff42e3a6640 1 -- 172.21.15.89:0/728312692 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff42815da90 con 0x7ff42806b310 2023-10-30T12:29:02.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.976+0000 7ff42dba5640 1 --2- 172.21.15.89:0/728312692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4280c7830 0x7ff42814be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:02.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.976+0000 7ff42e3a6640 1 --2- 172.21.15.89:0/728312692 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff42806b310 0x7ff42806b790 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff42001cd50 tx=0x7ff414005800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:02.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.976+0000 7ff42d3a4640 1 --2- 172.21.15.89:0/728312692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff42806cd70 0x7ff42814c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:29:02.984 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.976+0000 7ff41e7fc640 1 -- 172.21.15.89:0/728312692 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff420001a10 con 0x7ff42806b310 2023-10-30T12:29:02.984 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.976+0000 7ff41e7fc640 1 -- 172.21.15.89:0/728312692 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff420001fc0 con 0x7ff42806b310 2023-10-30T12:29:02.984 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.976+0000 7ff41e7fc640 1 -- 172.21.15.89:0/728312692 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4200adba0 con 0x7ff42806b310 2023-10-30T12:29:02.984 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.976+0000 7ff42fe09640 1 -- 172.21.15.89:0/728312692 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff42815dd30 con 0x7ff42806b310 2023-10-30T12:29:02.984 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.976+0000 7ff42fe09640 1 -- 172.21.15.89:0/728312692 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff42815c0a0 con 0x7ff42806b310 2023-10-30T12:29:02.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.980+0000 7ff42fe09640 1 -- 172.21.15.89:0/728312692 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff42804e170 con 0x7ff42806b310 2023-10-30T12:29:02.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.980+0000 7ff41e7fc640 1 -- 172.21.15.89:0/728312692 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff4200aa020 con 0x7ff42806b310 2023-10-30T12:29:02.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.984+0000 7ff41e7fc640 1 --2- 172.21.15.89:0/728312692 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff40c0665a0 0x7ff40c068a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:02.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.984+0000 7ff41e7fc640 1 -- 172.21.15.89:0/728312692 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff42011df90 con 0x7ff42806b310 2023-10-30T12:29:02.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.984+0000 7ff42dba5640 1 --2- 172.21.15.89:0/728312692 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff40c0665a0 0x7ff40c068a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:02.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:02.984+0000 7ff42dba5640 1 --2- 172.21.15.89:0/728312692 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff40c0665a0 0x7ff40c068a60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff424006fd0 tx=0x7ff424008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:03.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.004+0000 7ff41e7fc640 1 -- 172.21.15.89:0/728312692 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff4200be020 con 0x7ff42806b310 2023-10-30T12:29:03.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.056+0000 7f10e7f3e640 1 Processor -- start 2023-10-30T12:29:03.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.056+0000 7f10e7f3e640 1 -- start start 2023-10-30T12:29:03.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.060+0000 7f10e7f3e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10e014c140 0x7f10e0156020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.060+0000 7f10e7f3e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f10e00c2290 0x7f10e00c26d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.060+0000 7f10e7f3e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f10e006d0e0 0x7f10e006d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.060+0000 7f10e7f3e640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10e00c6370 con 0x7f10e00c2290 2023-10-30T12:29:03.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.060+0000 7f10e7f3e640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10e0159af0 con 0x7f10e006d0e0 2023-10-30T12:29:03.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.060+0000 7f10e7f3e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10e0156560 con 0x7f10e014c140 2023-10-30T12:29:03.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.060+0000 7f10e54d9640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f10e00c2290 0x7f10e00c26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.060+0000 7f10e64db640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f10e006d0e0 0x7f10e006d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.060+0000 7f10e64db640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f10e006d0e0 0x7f10e006d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:59518/0 (socket says 172.21.15.89:59518) 2023-10-30T12:29:03.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.060+0000 7f10e64db640 1 -- 172.21.15.89:0/3701970369 learned_addr learned my addr 172.21.15.89:0/3701970369 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:03.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.060+0000 7f10e5cda640 1 --2- 172.21.15.89:0/3701970369 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10e014c140 0x7f10e0156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.064+0000 7f10e64db640 1 -- 172.21.15.89:0/3701970369 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10e014c140 msgr2=0x7f10e0156020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:03.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.064+0000 7f10e64db640 1 --2- 172.21.15.89:0/3701970369 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10e014c140 0x7f10e0156020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.064+0000 7f10e64db640 1 -- 172.21.15.89:0/3701970369 >> v2:172.21.15.89:3300/0 conn(0x7f10e00c2290 msgr2=0x7f10e00c26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:03.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.064+0000 7f10e64db640 1 --2- 172.21.15.89:0/3701970369 >> v2:172.21.15.89:3300/0 conn(0x7f10e00c2290 0x7f10e00c26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.064+0000 7f10e64db640 1 -- 172.21.15.89:0/3701970369 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e00c6090 con 0x7f10e006d0e0 2023-10-30T12:29:03.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.064+0000 7f10e5cda640 1 --2- 172.21.15.89:0/3701970369 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10e014c140 0x7f10e0156020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:29:03.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.068+0000 7f10e64db640 1 --2- 172.21.15.89:0/3701970369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10e006d0e0 0x7f10e006d510 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f10e0066d80 tx=0x7f10d8021780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e184a7ed924acc9d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:03.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.068+0000 7f10e4cd8640 1 -- 172.21.15.89:0/3701970369 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d80af670 con 0x7f10e006d0e0 2023-10-30T12:29:03.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.068+0000 7f10e4cd8640 1 -- 172.21.15.89:0/3701970369 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d80af7d0 con 0x7f10e006d0e0 2023-10-30T12:29:03.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.068+0000 7f10e7f3e640 1 -- 172.21.15.89:0/3701970369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10e006d0e0 msgr2=0x7f10e006d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.068+0000 7f10e7f3e640 1 --2- 172.21.15.89:0/3701970369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10e006d0e0 0x7f10e006d510 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f10e0066d80 tx=0x7f10d8021780 comp rx=0 tx=0).stop 2023-10-30T12:29:03.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.068+0000 7f10e7f3e640 1 -- 172.21.15.89:0/3701970369 shutdown_connections 2023-10-30T12:29:03.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.072+0000 7f10e7f3e640 1 --2- 172.21.15.89:0/3701970369 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10e006d0e0 0x7f10e006d510 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.072+0000 7f10e7f3e640 1 --2- 172.21.15.89:0/3701970369 >> v2:172.21.15.89:3300/0 conn(0x7f10e00c2290 0x7f10e00c26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.072+0000 7f10e7f3e640 1 --2- 172.21.15.89:0/3701970369 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10e014c140 0x7f10e0156020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.072+0000 7f10e7f3e640 1 -- 172.21.15.89:0/3701970369 >> 172.21.15.89:0/3701970369 conn(0x7f10e00cac50 msgr2=0x7f10e00cb060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:03.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.072+0000 7f10e7f3e640 1 -- 172.21.15.89:0/3701970369 shutdown_connections 2023-10-30T12:29:03.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.072+0000 7f10e7f3e640 1 -- 172.21.15.89:0/3701970369 wait complete. 2023-10-30T12:29:03.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.072+0000 7f10e7f3e640 1 Processor -- start 2023-10-30T12:29:03.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.076+0000 7f10e7f3e640 1 -- start start 2023-10-30T12:29:03.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.076+0000 7f10e7f3e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10e006d0e0 0x7f10e00bdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.076+0000 7f10e7f3e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10e00c2290 0x7f10e00be4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.076+0000 7f10e7f3e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10e014c140 0x7f10e01fb850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.076+0000 7f10e54d9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10e00c2290 0x7f10e00be4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.076+0000 7f10e7f3e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e0155610 con 0x7f10e014c140 2023-10-30T12:29:03.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.076+0000 7f10e7f3e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e00c01e0 con 0x7f10e00c2290 2023-10-30T12:29:03.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.076+0000 7f10e7f3e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10e00c04c0 con 0x7f10e006d0e0 2023-10-30T12:29:03.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.076+0000 7f10e54d9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10e00c2290 0x7f10e00be4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:59528/0 (socket says 172.21.15.89:59528) 2023-10-30T12:29:03.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.076+0000 7f10e54d9640 1 -- 172.21.15.89:0/2866946276 learned_addr learned my addr 172.21.15.89:0/2866946276 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:03.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.080+0000 7f10e5cda640 1 --2- 172.21.15.89:0/2866946276 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10e006d0e0 0x7f10e00bdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.080+0000 7f10e54d9640 1 -- 172.21.15.89:0/2866946276 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10e006d0e0 msgr2=0x7f10e00bdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.080+0000 7f10e54d9640 1 --2- 172.21.15.89:0/2866946276 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10e006d0e0 0x7f10e00bdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.080+0000 7f10e54d9640 1 -- 172.21.15.89:0/2866946276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10e014c140 msgr2=0x7f10e01fb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.080+0000 7f10e54d9640 1 --2- 172.21.15.89:0/2866946276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10e014c140 0x7f10e01fb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.080+0000 7f10e54d9640 1 -- 172.21.15.89:0/2866946276 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d0009420 con 0x7f10e00c2290 2023-10-30T12:29:03.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.080+0000 7f10e54d9640 1 --2- 172.21.15.89:0/2866946276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10e00c2290 0x7f10e00be4e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f10dc00d800 tx=0x7f10dc00dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:03.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.080+0000 7f10d67fc640 1 -- 172.21.15.89:0/2866946276 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10dc0096a0 con 0x7f10e00c2290 2023-10-30T12:29:03.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.080+0000 7f10e7f3e640 1 -- 172.21.15.89:0/2866946276 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10d800c050 con 0x7f10e00c2290 2023-10-30T12:29:03.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.080+0000 7f10e7f3e640 1 -- 172.21.15.89:0/2866946276 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10e00c0a80 con 0x7f10e00c2290 2023-10-30T12:29:03.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.080+0000 7f10d67fc640 1 -- 172.21.15.89:0/2866946276 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10dc009800 con 0x7f10e00c2290 2023-10-30T12:29:03.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.080+0000 7f10d67fc640 1 -- 172.21.15.89:0/2866946276 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10dc0096a0 con 0x7f10e00c2290 2023-10-30T12:29:03.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.084+0000 7f10e7f3e640 1 -- 172.21.15.89:0/2866946276 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10c8005260 con 0x7f10e00c2290 2023-10-30T12:29:03.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.084+0000 7f10d67fc640 1 -- 172.21.15.89:0/2866946276 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f10dc009970 con 0x7f10e00c2290 2023-10-30T12:29:03.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.088+0000 7f10d67fc640 1 --2- 172.21.15.89:0/2866946276 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f10c4066500 0x7f10c40689c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.088+0000 7f10d67fc640 1 -- 172.21.15.89:0/2866946276 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f10dc084290 con 0x7f10e00c2290 2023-10-30T12:29:03.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.088+0000 7f10e5cda640 1 --2- 172.21.15.89:0/2866946276 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f10c4066500 0x7f10c40689c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.088+0000 7f10e5cda640 1 --2- 172.21.15.89:0/2866946276 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f10c4066500 0x7f10c40689c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f10d0009b60 tx=0x7f10d0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:03.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.096+0000 7f10d67fc640 1 -- 172.21.15.89:0/2866946276 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f10dc051780 con 0x7f10e00c2290 2023-10-30T12:29:03.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.128+0000 7ff95cdaa640 1 Processor -- start 2023-10-30T12:29:03.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.128+0000 7ff95cdaa640 1 -- start start 2023-10-30T12:29:03.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.128+0000 7ff95cdaa640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff95806ad90 0x7ff9580c6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.128+0000 7ff95cdaa640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff958069e90 0x7ff95806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.128+0000 7ff95cdaa640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9580c7400 con 0x7ff958152de0 2023-10-30T12:29:03.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.128+0000 7ff95cdaa640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff958070820 con 0x7ff958069e90 2023-10-30T12:29:03.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.128+0000 7ff95cdaa640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9580c76c0 con 0x7ff95806ad90 2023-10-30T12:29:03.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.132+0000 7ff956d9d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff958069e90 0x7ff95806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.128+0000 7fa14e9a1640 1 Processor -- start 2023-10-30T12:29:03.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.132+0000 7ff956d9d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff958069e90 0x7ff95806a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39232/0 (socket says 172.21.15.89:39232) 2023-10-30T12:29:03.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.132+0000 7ff955d9b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff95806ad90 0x7ff9580c6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.132+0000 7ff956d9d640 1 -- 172.21.15.89:0/3650620632 learned_addr learned my addr 172.21.15.89:0/3650620632 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:03.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.132+0000 7ff95559a640 1 -- 172.21.15.89:0/3650620632 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 257316381 0 0) 0x7ff9580c7400 con 0x7ff958152de0 2023-10-30T12:29:03.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.132+0000 7ff95559a640 1 -- 172.21.15.89:0/3650620632 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff940003530 con 0x7ff958152de0 2023-10-30T12:29:03.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.132+0000 7ff955d9b640 1 -- 172.21.15.89:0/3650620632 >> v2:172.21.15.89:3300/0 conn(0x7ff958069e90 msgr2=0x7ff95806a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:03.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.132+0000 7ff955d9b640 1 --2- 172.21.15.89:0/3650620632 >> v2:172.21.15.89:3300/0 conn(0x7ff958069e90 0x7ff95806a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.132+0000 7ff955d9b640 1 -- 172.21.15.89:0/3650620632 >> v1:172.21.15.89:6789/0 conn(0x7ff958152de0 legacy=0x7ff958156280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.132+0000 7ff955d9b640 1 -- 172.21.15.89:0/3650620632 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9580c58b0 con 0x7ff95806ad90 2023-10-30T12:29:03.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.132+0000 7fa14e9a1640 1 -- start start 2023-10-30T12:29:03.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.132+0000 7fa14e9a1640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa148158090 0x7fa148161be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.132+0000 7fa14e9a1640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa1480c6b40 0x7fa148069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.132+0000 7fa14e9a1640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa14806a3d0 0x7fa14806a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.132+0000 7fa14e9a1640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa148162120 con 0x7fa14806a3d0 2023-10-30T12:29:03.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.132+0000 7fa14e9a1640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa14806ad60 con 0x7fa1480c6b40 2023-10-30T12:29:03.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.132+0000 7fa14e9a1640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1480c72b0 con 0x7fa148158090 2023-10-30T12:29:03.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.136+0000 7fb5796ad640 1 Processor -- start 2023-10-30T12:29:03.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.132+0000 7fa147fff640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa148158090 0x7fa148161be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.132+0000 7fa147fff640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa148158090 0x7fa148161be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:57476/0 (socket says 172.21.15.89:57476) 2023-10-30T12:29:03.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.132+0000 7fa147fff640 1 -- 172.21.15.89:0/307387209 learned_addr learned my addr 172.21.15.89:0/307387209 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:03.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.132+0000 7fa13f7fe640 1 --2- 172.21.15.89:0/307387209 >> v2:172.21.15.192:3300/0 conn(0x7fa1480c6b40 0x7fa148069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.136+0000 7fa13f7fe640 1 -- 172.21.15.89:0/307387209 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa148158090 msgr2=0x7fa148161be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:03.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.136+0000 7fa13f7fe640 1 --2- 172.21.15.89:0/307387209 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa148158090 0x7fa148161be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.136+0000 7fa13f7fe640 1 -- 172.21.15.89:0/307387209 >> v2:172.21.15.89:3300/0 conn(0x7fa14806a3d0 msgr2=0x7fa14806a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:03.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.136+0000 7fa13f7fe640 1 --2- 172.21.15.89:0/307387209 >> v2:172.21.15.89:3300/0 conn(0x7fa14806a3d0 0x7fa14806a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.136+0000 7fa13f7fe640 1 -- 172.21.15.89:0/307387209 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1480c6fd0 con 0x7fa1480c6b40 2023-10-30T12:29:03.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.136+0000 7fa211646640 1 Processor -- start 2023-10-30T12:29:03.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.136+0000 7fb5796ad640 1 -- start start 2023-10-30T12:29:03.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7ff955d9b640 1 --2- 172.21.15.89:0/3650620632 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff95806ad90 0x7ff9580c6670 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff94c0071d0 tx=0x7ff94c01d490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9159848cd2ecf44 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:03.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7fa211646640 1 -- start start 2023-10-30T12:29:03.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7f6307868640 1 Processor -- start 2023-10-30T12:29:03.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7fb5796ad640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb56c04f420 0x7fb56c0dec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7fb5796ad640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb56c04af40 con 0x7fb56c050a30 2023-10-30T12:29:03.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7fb5796ad640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb56c04f8f0 con 0x7fb56c0d8e50 2023-10-30T12:29:03.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7fa13f7fe640 1 --2- 172.21.15.89:0/307387209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1480c6b40 0x7fa148069e90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa14806f170 tx=0x7fa13801ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3dab9d94a577f95 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:03.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7ff95559a640 1 -- 172.21.15.89:0/3650620632 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94c008be0 con 0x7ff95806ad90 2023-10-30T12:29:03.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7ff95559a640 1 -- 172.21.15.89:0/3650620632 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff94c008d40 con 0x7ff95806ad90 2023-10-30T12:29:03.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7ff95559a640 1 -- 172.21.15.89:0/3650620632 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94c00a2f0 con 0x7ff95806ad90 2023-10-30T12:29:03.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7fb5796ad640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb56c049a70 con 0x7fb56c04f420 2023-10-30T12:29:03.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7fb5737fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb56c04f420 0x7fb56c0dec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7fa1477fe640 1 -- 172.21.15.89:0/307387209 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1380a8910 con 0x7fa1480c6b40 2023-10-30T12:29:03.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7fa1477fe640 1 -- 172.21.15.89:0/307387209 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1380a8a70 con 0x7fa1480c6b40 2023-10-30T12:29:03.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7fa1477fe640 1 -- 172.21.15.89:0/307387209 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1380a8e00 con 0x7fa1480c6b40 2023-10-30T12:29:03.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7fb573fff640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fb56c0d8e50 0x7fb56c04a830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:36126/0 (socket says 172.21.15.89:36126) 2023-10-30T12:29:03.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7fb573fff640 1 -- 172.21.15.89:0/3601054855 learned_addr learned my addr 172.21.15.89:0/3601054855 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:29:03.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7ff95cdaa640 1 -- 172.21.15.89:0/3650620632 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff95806ad90 msgr2=0x7ff9580c6670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7ff95cdaa640 1 --2- 172.21.15.89:0/3650620632 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff95806ad90 0x7ff9580c6670 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff94c0071d0 tx=0x7ff94c01d490 comp rx=0 tx=0).stop 2023-10-30T12:29:03.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7fb572ffd640 1 -- 172.21.15.89:0/3601054855 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2317913125 0 0) 0x7fb56c04f8f0 con 0x7fb56c0d8e50 2023-10-30T12:29:03.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7f6307868640 1 -- start start 2023-10-30T12:29:03.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7f6307868640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6300152de0 0x7f6300156280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7fb572ffd640 1 -- 172.21.15.89:0/3601054855 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb564003530 con 0x7fb56c0d8e50 2023-10-30T12:29:03.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7f6307868640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6300069e90 0x7f630006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7f6307868640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63001567c0 con 0x7f630006ad90 2023-10-30T12:29:03.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7f6307868640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6300070820 con 0x7f6300069e90 2023-10-30T12:29:03.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7f6307868640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63000c6d80 con 0x7f6300152de0 2023-10-30T12:29:03.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7fa211646640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa20c156350 0x7fa20c1615d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7fa211646640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa20c0c0060 con 0x7fa20c14fc00 2023-10-30T12:29:03.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7fa211646640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa20c0c0340 con 0x7fa20c0c74e0 2023-10-30T12:29:03.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7fa211646640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa20c071090 con 0x7fa20c156350 2023-10-30T12:29:03.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7fa14e9a1640 1 -- 172.21.15.89:0/307387209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1480c6b40 msgr2=0x7fa148069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7fa14e9a1640 1 --2- 172.21.15.89:0/307387209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1480c6b40 0x7fa148069e90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa14806f170 tx=0x7fa13801ec70 comp rx=0 tx=0).stop 2023-10-30T12:29:03.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7ff95cdaa640 1 -- 172.21.15.89:0/3650620632 shutdown_connections 2023-10-30T12:29:03.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.140+0000 7ff95cdaa640 1 --2- 172.21.15.89:0/3650620632 >> v2:172.21.15.89:3300/0 conn(0x7ff958069e90 0x7ff95806a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7ff95cdaa640 1 --2- 172.21.15.89:0/3650620632 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff95806ad90 0x7ff9580c6670 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7ff95cdaa640 1 -- 172.21.15.89:0/3650620632 >> 172.21.15.89:0/3650620632 conn(0x7ff9580c7910 msgr2=0x7ff9580c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:03.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7fa20b7fe640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa20c156350 0x7fa20c1615d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7fb572ffd640 1 -- 172.21.15.89:0/3601054855 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 406717435 0 0) 0x7fb564003530 con 0x7fb56c0d8e50 2023-10-30T12:29:03.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7ff95cdaa640 1 -- 172.21.15.89:0/3650620632 shutdown_connections 2023-10-30T12:29:03.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7fa20b7fe640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa20c156350 0x7fa20c1615d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:59552/0 (socket says 172.21.15.89:59552) 2023-10-30T12:29:03.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7fa20b7fe640 1 -- 172.21.15.89:0/2123361982 learned_addr learned my addr 172.21.15.89:0/2123361982 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:03.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7f6304e03640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f630006ad90 0x7f63000c6670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:52182/0 (socket says 172.21.15.89:52182) 2023-10-30T12:29:03.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7ff95cdaa640 1 -- 172.21.15.89:0/3650620632 wait complete. 2023-10-30T12:29:03.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7f6304e03640 1 -- 172.21.15.89:0/2708301514 learned_addr learned my addr 172.21.15.89:0/2708301514 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:29:03.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7fb5737fe640 1 -- 172.21.15.89:0/3601054855 >> v1:172.21.15.192:6789/0 conn(0x7fb56c0d8e50 legacy=0x7fb56c04a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7fb5737fe640 1 -- 172.21.15.89:0/3601054855 >> v1:172.21.15.89:6789/0 conn(0x7fb56c050a30 legacy=0x7fb56c0df1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7f6305604640 1 --2- 172.21.15.89:0/2708301514 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6300152de0 0x7f6300156280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7fb5737fe640 1 -- 172.21.15.89:0/3601054855 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb56c049d50 con 0x7fb56c04f420 2023-10-30T12:29:03.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7fa20b7fe640 1 -- 172.21.15.89:0/2123361982 >> v1:172.21.15.192:6789/0 conn(0x7fa20c0c74e0 legacy=0x7fa20c155c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7f62f7fff640 1 -- 172.21.15.89:0/2708301514 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3948148814 0 0) 0x7f63001567c0 con 0x7f630006ad90 2023-10-30T12:29:03.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7f62f7fff640 1 -- 172.21.15.89:0/2708301514 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62ec003530 con 0x7f630006ad90 2023-10-30T12:29:03.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7f62f7fff640 1 -- 172.21.15.89:0/2708301514 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1822214878 0 0) 0x7f62ec003530 con 0x7f630006ad90 2023-10-30T12:29:03.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7f62f7fff640 1 -- 172.21.15.89:0/2708301514 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6300152de0 msgr2=0x7f6300156280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:03.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7f62f7fff640 1 --2- 172.21.15.89:0/2708301514 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6300152de0 0x7f6300156280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7f62f7fff640 1 -- 172.21.15.89:0/2708301514 >> v2:172.21.15.192:3300/0 conn(0x7f6300069e90 msgr2=0x7f630006a2e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-30T12:29:03.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7f62f7fff640 1 --2- 172.21.15.89:0/2708301514 >> v2:172.21.15.192:3300/0 conn(0x7f6300069e90 0x7f630006a2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7f62f7fff640 1 -- 172.21.15.89:0/2708301514 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63000c7400 con 0x7f630006ad90 2023-10-30T12:29:03.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7f62f7fff640 1 -- 172.21.15.89:0/2708301514 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f62f0001680 con 0x7f630006ad90 2023-10-30T12:29:03.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7f62f7fff640 1 -- 172.21.15.89:0/2708301514 >> v1:172.21.15.89:6789/0 conn(0x7f630006ad90 legacy=0x7f63000c6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7f62f7fff640 1 --2- 172.21.15.89:0/2708301514 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62ec003770 0x7f62ec003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7f62f7fff640 1 --2- 172.21.15.89:0/2708301514 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62ec0040d0 0x7f62ec00b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7f62f7fff640 1 --2- 172.21.15.89:0/2708301514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62ec00b560 0x7f62ec00d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7f62f7fff640 1 -- 172.21.15.89:0/2708301514 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63000c6d80 con 0x7f62ec00b560 2023-10-30T12:29:03.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7f62f7fff640 1 -- 172.21.15.89:0/2708301514 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec00e230 con 0x7f62ec0040d0 2023-10-30T12:29:03.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7f62f7fff640 1 -- 172.21.15.89:0/2708301514 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec00e3f0 con 0x7f62ec003770 2023-10-30T12:29:03.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7ff95cdaa640 1 Processor -- start 2023-10-30T12:29:03.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7fa20b7fe640 1 -- 172.21.15.89:0/2123361982 >> v1:172.21.15.89:6789/0 conn(0x7fa20c14fc00 legacy=0x7fa20c06a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7fa20b7fe640 1 -- 172.21.15.89:0/2123361982 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa20c071370 con 0x7fa20c156350 2023-10-30T12:29:03.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.148+0000 7ff95cdaa640 1 -- start start 2023-10-30T12:29:03.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7fa14e9a1640 1 -- 172.21.15.89:0/307387209 shutdown_connections 2023-10-30T12:29:03.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7fa14e9a1640 1 --2- 172.21.15.89:0/307387209 >> v2:172.21.15.89:3300/0 conn(0x7fa14806a3d0 0x7fa14806a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7fa14e9a1640 1 --2- 172.21.15.89:0/307387209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1480c6b40 0x7fa148069e90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.144+0000 7fa14e9a1640 1 --2- 172.21.15.89:0/307387209 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa148158090 0x7fa148161be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.148+0000 7fa14e9a1640 1 -- 172.21.15.89:0/307387209 >> 172.21.15.89:0/307387209 conn(0x7fa1480c7910 msgr2=0x7fa1480c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:03.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.148+0000 7f6304e03640 1 --2- 172.21.15.89:0/2708301514 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62ec0040d0 0x7f62ec00b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.148+0000 7f6305e05640 1 --2- 172.21.15.89:0/2708301514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62ec00b560 0x7f62ec00d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.148+0000 7f6304e03640 1 -- 172.21.15.89:0/2708301514 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62ec003770 msgr2=0x7f62ec003b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:29:03.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.148+0000 7f6304e03640 1 --2- 172.21.15.89:0/2708301514 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f62ec003770 0x7f62ec003b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.148+0000 7f6304e03640 1 -- 172.21.15.89:0/2708301514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62ec00b560 msgr2=0x7f62ec00d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:03.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.148+0000 7f6304e03640 1 --2- 172.21.15.89:0/2708301514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62ec00b560 0x7f62ec00d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.148+0000 7f6304e03640 1 -- 172.21.15.89:0/2708301514 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ec003530 con 0x7f62ec0040d0 2023-10-30T12:29:03.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.148+0000 7f6305e05640 1 --2- 172.21.15.89:0/2708301514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62ec00b560 0x7f62ec00d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:03.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.148+0000 7fa14e9a1640 1 -- 172.21.15.89:0/307387209 shutdown_connections 2023-10-30T12:29:03.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.148+0000 7fa14e9a1640 1 -- 172.21.15.89:0/307387209 wait complete. 2023-10-30T12:29:03.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.148+0000 7ff95cdaa640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff958069e90 0x7ff9581fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.148+0000 7ff95cdaa640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff95806ad90 0x7ff95814ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.148+0000 7fb5737fe640 1 --2- 172.21.15.89:0/3601054855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb56c04f420 0x7fb56c0dec90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb5600097c0 tx=0x7fb56001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7347b639053027f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:03.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.148+0000 7ff95659c640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff958069e90 0x7ff9581fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.148+0000 7ff95659c640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff958069e90 0x7ff9581fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:57498/0 (socket says 172.21.15.89:57498) 2023-10-30T12:29:03.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.148+0000 7ff95659c640 1 -- 172.21.15.89:0/3789257816 learned_addr learned my addr 172.21.15.89:0/3789257816 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:03.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.148+0000 7fa20b7fe640 1 --2- 172.21.15.89:0/2123361982 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa20c156350 0x7fa20c1615d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc009d40 tx=0x7fa1fc020300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84dc5801a8bc88c2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:03.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.148+0000 7ff95cdaa640 1 --2- 172.21.15.89:0/3789257816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff958152de0 0x7ff95814b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.148+0000 7ff955d9b640 1 --2- 172.21.15.89:0/3789257816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff95806ad90 0x7ff95814ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.148+0000 7fa209ffb640 1 -- 172.21.15.89:0/2123361982 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc0090d0 con 0x7fa20c156350 2023-10-30T12:29:03.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.148+0000 7fa209ffb640 1 -- 172.21.15.89:0/2123361982 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1fc0ac050 con 0x7fa20c156350 2023-10-30T12:29:03.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.148+0000 7fb572ffd640 1 -- 172.21.15.89:0/3601054855 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5680016d0 con 0x7fb56c04f420 2023-10-30T12:29:03.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.148+0000 7fb572ffd640 1 -- 172.21.15.89:0/3601054855 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5600a7050 con 0x7fb56c04f420 2023-10-30T12:29:03.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.148+0000 7fb572ffd640 1 -- 172.21.15.89:0/3601054855 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5600ab620 con 0x7fb56c04f420 2023-10-30T12:29:03.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.148+0000 7fa209ffb640 1 -- 172.21.15.89:0/2123361982 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc0090d0 con 0x7fa20c156350 2023-10-30T12:29:03.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7ff95cdaa640 1 -- 172.21.15.89:0/3789257816 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95814c1d0 con 0x7ff958152de0 2023-10-30T12:29:03.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fa14e9a1640 1 Processor -- start 2023-10-30T12:29:03.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7ff95cdaa640 1 -- 172.21.15.89:0/3789257816 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9581569c0 con 0x7ff95806ad90 2023-10-30T12:29:03.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fa14e9a1640 1 -- start start 2023-10-30T12:29:03.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7ff95cdaa640 1 -- 172.21.15.89:0/3789257816 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff958158b80 con 0x7ff958069e90 2023-10-30T12:29:03.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7ff956d9d640 1 --2- 172.21.15.89:0/3789257816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff958152de0 0x7ff95814b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fa211646640 1 -- 172.21.15.89:0/2123361982 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa20c156350 msgr2=0x7fa20c1615d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fa211646640 1 --2- 172.21.15.89:0/2123361982 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa20c156350 0x7fa20c1615d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc009d40 tx=0x7fa1fc020300 comp rx=0 tx=0).stop 2023-10-30T12:29:03.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fa211646640 1 -- 172.21.15.89:0/2123361982 shutdown_connections 2023-10-30T12:29:03.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fa211646640 1 --2- 172.21.15.89:0/2123361982 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa20c156350 0x7fa20c1615d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fa211646640 1 -- 172.21.15.89:0/2123361982 >> 172.21.15.89:0/2123361982 conn(0x7fa20c0c7910 msgr2=0x7fa20c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:03.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fb5796ad640 1 -- 172.21.15.89:0/3601054855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb56c04f420 msgr2=0x7fb56c0dec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fb5796ad640 1 --2- 172.21.15.89:0/3601054855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb56c04f420 0x7fb56c0dec90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb5600097c0 tx=0x7fb56001c360 comp rx=0 tx=0).stop 2023-10-30T12:29:03.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7ff956d9d640 1 -- 172.21.15.89:0/3789257816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff958069e90 msgr2=0x7ff9581fc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7ff956d9d640 1 --2- 172.21.15.89:0/3789257816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff958069e90 0x7ff9581fc360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7ff956d9d640 1 -- 172.21.15.89:0/3789257816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff95806ad90 msgr2=0x7ff95814ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7ff956d9d640 1 --2- 172.21.15.89:0/3789257816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff95806ad90 0x7ff95814ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7ff956d9d640 1 -- 172.21.15.89:0/3789257816 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94c00ae50 con 0x7ff958152de0 2023-10-30T12:29:03.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fa211646640 1 -- 172.21.15.89:0/2123361982 shutdown_connections 2023-10-30T12:29:03.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fa211646640 1 -- 172.21.15.89:0/2123361982 wait complete. 2023-10-30T12:29:03.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7ff956d9d640 1 --2- 172.21.15.89:0/3789257816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff958152de0 0x7ff95814b1d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff950009d90 tx=0x7ff95000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:03.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7f6304e03640 1 --2- 172.21.15.89:0/2708301514 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62ec0040d0 0x7f62ec00b020 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f62f000c290 tx=0x7f62f001cc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb9bb9e58b278e2e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:03.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fa14e9a1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa14806a3d0 0x7fa148149b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fa14e9a1640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa148158090 0x7fa14815f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fb5796ad640 1 -- 172.21.15.89:0/3601054855 shutdown_connections 2023-10-30T12:29:03.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fb5796ad640 1 --2- 172.21.15.89:0/3601054855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb56c04f420 0x7fb56c0dec90 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fb5796ad640 1 -- 172.21.15.89:0/3601054855 >> 172.21.15.89:0/3601054855 conn(0x7fb56c051040 msgr2=0x7fb56c051450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:03.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fa14e9a1640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa14815eda0 0x7fa14815fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fa14e9a1640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1480c1040 con 0x7fa14806a3d0 2023-10-30T12:29:03.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fa14e9a1640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1480c1600 con 0x7fa14815eda0 2023-10-30T12:29:03.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fa14e9a1640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa14815f270 con 0x7fa148158090 2023-10-30T12:29:03.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7ff946ffd640 1 -- 172.21.15.89:0/3789257816 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95000cd20 con 0x7ff958152de0 2023-10-30T12:29:03.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7f62f7fff640 1 -- 172.21.15.89:0/2708301514 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f00a5980 con 0x7f62ec0040d0 2023-10-30T12:29:03.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7ff946ffd640 1 -- 172.21.15.89:0/3789257816 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff95000f040 con 0x7ff958152de0 2023-10-30T12:29:03.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7ff946ffd640 1 -- 172.21.15.89:0/3789257816 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff950003bc0 con 0x7ff958152de0 2023-10-30T12:29:03.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7ff95cdaa640 1 -- 172.21.15.89:0/3789257816 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff95814a520 con 0x7ff958152de0 2023-10-30T12:29:03.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fa14cf3e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa14815eda0 0x7fa14815fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fa14cf3e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa14815eda0 0x7fa14815fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:59576/0 (socket says 172.21.15.89:59576) 2023-10-30T12:29:03.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fa14cf3e640 1 -- 172.21.15.89:0/729009924 learned_addr learned my addr 172.21.15.89:0/729009924 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:03.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fa147fff640 1 --2- 172.21.15.89:0/729009924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa14806a3d0 0x7fa148149b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7ff95cdaa640 1 -- 172.21.15.89:0/3789257816 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff95814a800 con 0x7ff958152de0 2023-10-30T12:29:03.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fb5796ad640 1 -- 172.21.15.89:0/3601054855 shutdown_connections 2023-10-30T12:29:03.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fb5796ad640 1 -- 172.21.15.89:0/3601054855 wait complete. 2023-10-30T12:29:03.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fa14cf3e640 1 -- 172.21.15.89:0/729009924 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa148158090 msgr2=0x7fa14815f640 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-30T12:29:03.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fa14cf3e640 1 --2- 172.21.15.89:0/729009924 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa148158090 0x7fa14815f640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fa14cf3e640 1 -- 172.21.15.89:0/729009924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa14806a3d0 msgr2=0x7fa148149b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fa14cf3e640 1 --2- 172.21.15.89:0/729009924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa14806a3d0 0x7fa148149b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fa14cf3e640 1 -- 172.21.15.89:0/729009924 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa138009420 con 0x7fa14815eda0 2023-10-30T12:29:03.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7fa14cf3e640 1 --2- 172.21.15.89:0/729009924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa14815eda0 0x7fa14815fb80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa14000b970 tx=0x7fa14000be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:03.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7f62f7fff640 1 -- 172.21.15.89:0/2708301514 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f00a5ae0 con 0x7f62ec0040d0 2023-10-30T12:29:03.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.152+0000 7f62f7fff640 1 -- 172.21.15.89:0/2708301514 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f00a5980 con 0x7f62ec0040d0 2023-10-30T12:29:03.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.156+0000 7fa211646640 1 Processor -- start 2023-10-30T12:29:03.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.156+0000 7fa211646640 1 -- start start 2023-10-30T12:29:03.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.156+0000 7fa1457fa640 1 -- 172.21.15.89:0/729009924 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14000cb40 con 0x7fa14815eda0 2023-10-30T12:29:03.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.156+0000 7fa1457fa640 1 -- 172.21.15.89:0/729009924 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa14000ccc0 con 0x7fa14815eda0 2023-10-30T12:29:03.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.156+0000 7fa1457fa640 1 -- 172.21.15.89:0/729009924 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa140014680 con 0x7fa14815eda0 2023-10-30T12:29:03.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.156+0000 7fb5796ad640 1 Processor -- start 2023-10-30T12:29:03.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.156+0000 7fa14e9a1640 1 -- 172.21.15.89:0/729009924 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa148143530 con 0x7fa14815eda0 2023-10-30T12:29:03.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.156+0000 7fa14e9a1640 1 -- 172.21.15.89:0/729009924 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1480c1290 con 0x7fa14815eda0 2023-10-30T12:29:03.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.156+0000 7fb5796ad640 1 -- start start 2023-10-30T12:29:03.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.156+0000 7f6307868640 1 -- 172.21.15.89:0/2708301514 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62ec0040d0 msgr2=0x7f62ec00b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.156+0000 7f6307868640 1 --2- 172.21.15.89:0/2708301514 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f62ec0040d0 0x7f62ec00b020 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f62f000c290 tx=0x7f62f001cc80 comp rx=0 tx=0).stop 2023-10-30T12:29:03.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.156+0000 7ff944ff9640 1 -- 172.21.15.89:0/3789257816 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff95814a990 con 0x7ff958152de0 2023-10-30T12:29:03.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.156+0000 7ff946ffd640 1 -- 172.21.15.89:0/3789257816 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff950003d20 con 0x7ff958152de0 2023-10-30T12:29:03.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.156+0000 7fa211646640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa20c0c74e0 0x7fa20c0c1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.156+0000 7fa211646640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa20c14fc00 0x7fa20c155890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.156+0000 7fb5796ad640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb56c04f420 0x7fb56c04e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.156+0000 7fb5796ad640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb56c050a30 0x7fb56c04ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.156+0000 7fb5796ad640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb56c0d8e50 0x7fb56c0d41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.156+0000 7fb5796ad640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56c174e30 con 0x7fb56c04f420 2023-10-30T12:29:03.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.156+0000 7fb5796ad640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56c175360 con 0x7fb56c0d8e50 2023-10-30T12:29:03.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.156+0000 7fb5796ad640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb56c175080 con 0x7fb56c050a30 2023-10-30T12:29:03.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.156+0000 7fa14e9a1640 1 -- 172.21.15.89:0/729009924 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa14804e170 con 0x7fa14815eda0 2023-10-30T12:29:03.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7f6305e05640 1 -- 172.21.15.89:0/2708301514 reap_dead start 2023-10-30T12:29:03.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7f6307868640 1 -- 172.21.15.89:0/2708301514 shutdown_connections 2023-10-30T12:29:03.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7f6307868640 1 -- 172.21.15.89:0/2708301514 >> 172.21.15.89:0/2708301514 conn(0x7f63000c7910 msgr2=0x7f63000c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:03.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fa211646640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa20c156350 0x7fa20c1e22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fa20affd640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa20c0c74e0 0x7fa20c0c1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fa20affd640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa20c0c74e0 0x7fa20c0c1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:57524/0 (socket says 172.21.15.89:57524) 2023-10-30T12:29:03.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fa211646640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20c1ebaf0 con 0x7fa20c156350 2023-10-30T12:29:03.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fb578eac640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb56c0d8e50 0x7fb56c0d41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fb578eac640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb56c0d8e50 0x7fb56c0d41a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:59580/0 (socket says 172.21.15.89:59580) 2023-10-30T12:29:03.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fb578eac640 1 -- 172.21.15.89:0/1428133338 learned_addr learned my addr 172.21.15.89:0/1428133338 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:03.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fa20b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa20c156350 0x7fa20c1e22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fa20b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa20c156350 0x7fa20c1e22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39276/0 (socket says 172.21.15.89:39276) 2023-10-30T12:29:03.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fa20b7fe640 1 -- 172.21.15.89:0/2077303524 learned_addr learned my addr 172.21.15.89:0/2077303524 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:03.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fa211646640 1 -- 172.21.15.89:0/2077303524 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20c1e1e30 con 0x7fa20c14fc00 2023-10-30T12:29:03.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fa211646640 1 -- 172.21.15.89:0/2077303524 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa20c1eaf70 con 0x7fa20c0c74e0 2023-10-30T12:29:03.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fb5737fe640 1 --2- 172.21.15.89:0/1428133338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb56c050a30 0x7fb56c04ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7ff946ffd640 1 --2- 172.21.15.89:0/3789257816 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff9380665a0 0x7ff938068a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fb578eac640 1 -- 172.21.15.89:0/1428133338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb56c050a30 msgr2=0x7fb56c04ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fb578eac640 1 --2- 172.21.15.89:0/1428133338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb56c050a30 0x7fb56c04ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fb578eac640 1 -- 172.21.15.89:0/1428133338 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb56c04f420 msgr2=0x7fb56c04e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fb578eac640 1 --2- 172.21.15.89:0/1428133338 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb56c04f420 0x7fb56c04e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fb578eac640 1 -- 172.21.15.89:0/1428133338 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb560009420 con 0x7fb56c0d8e50 2023-10-30T12:29:03.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7ff946ffd640 1 -- 172.21.15.89:0/3789257816 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff9500843f0 con 0x7ff958152de0 2023-10-30T12:29:03.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fa20a7fc640 1 --2- 172.21.15.89:0/2077303524 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa20c14fc00 0x7fa20c155890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fa20b7fe640 1 -- 172.21.15.89:0/2077303524 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa20c0c74e0 msgr2=0x7fa20c0c1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fa20b7fe640 1 --2- 172.21.15.89:0/2077303524 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa20c0c74e0 0x7fa20c0c1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fb578eac640 1 --2- 172.21.15.89:0/1428133338 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb56c0d8e50 0x7fb56c0d41a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb574000b70 tx=0x7fb574071490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:03.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fa1457fa640 1 -- 172.21.15.89:0/729009924 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa1400147e0 con 0x7fa14815eda0 2023-10-30T12:29:03.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7ff95659c640 1 --2- 172.21.15.89:0/3789257816 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff9380665a0 0x7ff938068a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fb570ff9640 1 -- 172.21.15.89:0/1428133338 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb574072560 con 0x7fb56c0d8e50 2023-10-30T12:29:03.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fb570ff9640 1 -- 172.21.15.89:0/1428133338 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb574095480 con 0x7fb56c0d8e50 2023-10-30T12:29:03.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fb570ff9640 1 -- 172.21.15.89:0/1428133338 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5740958b0 con 0x7fb56c0d8e50 2023-10-30T12:29:03.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fb5796ad640 1 -- 172.21.15.89:0/1428133338 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb56c0cd220 con 0x7fb56c0d8e50 2023-10-30T12:29:03.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fb5796ad640 1 -- 172.21.15.89:0/1428133338 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb56c0cd7e0 con 0x7fb56c0d8e50 2023-10-30T12:29:03.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7f6307868640 1 -- 172.21.15.89:0/2708301514 shutdown_connections 2023-10-30T12:29:03.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fa20b7fe640 1 -- 172.21.15.89:0/2077303524 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa20c14fc00 msgr2=0x7fa20c155890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fa20b7fe640 1 --2- 172.21.15.89:0/2077303524 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa20c14fc00 0x7fa20c155890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fa20b7fe640 1 -- 172.21.15.89:0/2077303524 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1fc00d040 con 0x7fa20c156350 2023-10-30T12:29:03.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fa20affd640 1 --2- 172.21.15.89:0/2077303524 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa20c0c74e0 0x7fa20c0c1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:29:03.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fa20a7fc640 1 --2- 172.21.15.89:0/2077303524 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa20c14fc00 0x7fa20c155890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:03.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fa1457fa640 1 --2- 172.21.15.89:0/729009924 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa1280664f0 0x7fa1280689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.160+0000 7fa20b7fe640 1 --2- 172.21.15.89:0/2077303524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa20c156350 0x7fa20c1e22c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc01cf10 tx=0x7fa1fc01d0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:03.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.164+0000 7ff95659c640 1 --2- 172.21.15.89:0/3789257816 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff9380665a0 0x7ff938068a60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff9480011f0 tx=0x7ff948009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:03.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.164+0000 7fa1457fa640 1 -- 172.21.15.89:0/729009924 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa140083cf0 con 0x7fa14815eda0 2023-10-30T12:29:03.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.164+0000 7f6307868640 1 -- 172.21.15.89:0/2708301514 wait complete. 2023-10-30T12:29:03.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.164+0000 7fa147fff640 1 --2- 172.21.15.89:0/729009924 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa1280664f0 0x7fa1280689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.164+0000 7fb5796ad640 1 -- 172.21.15.89:0/1428133338 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb554005260 con 0x7fb56c0d8e50 2023-10-30T12:29:03.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.164+0000 7fa147fff640 1 --2- 172.21.15.89:0/729009924 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa1280664f0 0x7fa1280689b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa130009a40 tx=0x7fa130008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:03.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.164+0000 7f6307868640 1 Processor -- start 2023-10-30T12:29:03.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.164+0000 7f6307868640 1 -- start start 2023-10-30T12:29:03.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.164+0000 7f6307868640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6300153f30 0x7f6300144650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.164+0000 7f6307868640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6300154310 0x7f6300144b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.164+0000 7f6307868640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63001533a0 0x7f6300155000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.164+0000 7f6307868640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6300154b70 con 0x7f6300153f30 2023-10-30T12:29:03.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.164+0000 7f6307868640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630014c990 con 0x7f63001533a0 2023-10-30T12:29:03.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.164+0000 7f6307868640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f630014cc70 con 0x7f6300154310 2023-10-30T12:29:03.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.164+0000 7f6305604640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6300153f30 0x7f6300144650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.164+0000 7f6304e03640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6300154310 0x7f6300144b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.164+0000 7f6305604640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6300153f30 0x7f6300144650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39288/0 (socket says 172.21.15.89:39288) 2023-10-30T12:29:03.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.164+0000 7f6305604640 1 -- 172.21.15.89:0/3909387308 learned_addr learned my addr 172.21.15.89:0/3909387308 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:03.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.164+0000 7f6305e05640 1 --2- 172.21.15.89:0/3909387308 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63001533a0 0x7f6300155000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.168+0000 7f6305604640 1 -- 172.21.15.89:0/3909387308 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6300154310 msgr2=0x7f6300144b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.168+0000 7f6305604640 1 --2- 172.21.15.89:0/3909387308 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6300154310 0x7f6300144b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.168+0000 7f6305604640 1 -- 172.21.15.89:0/3909387308 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63001533a0 msgr2=0x7f6300155000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.168+0000 7f6305604640 1 --2- 172.21.15.89:0/3909387308 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63001533a0 0x7f6300155000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.168+0000 7f6305604640 1 -- 172.21.15.89:0/3909387308 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f630014cf50 con 0x7f6300153f30 2023-10-30T12:29:03.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.168+0000 7fa1f37fe640 1 -- 172.21.15.89:0/2077303524 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc004040 con 0x7fa20c156350 2023-10-30T12:29:03.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.168+0000 7f6305604640 1 --2- 172.21.15.89:0/3909387308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6300153f30 0x7f6300144650 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f62fc00d4d0 tx=0x7f62fc00d990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:03.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.168+0000 7ff946ffd640 1 -- 172.21.15.89:0/3789257816 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff950051870 con 0x7ff958152de0 2023-10-30T12:29:03.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.168+0000 7fa1f37fe640 1 -- 172.21.15.89:0/2077303524 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1fc0041a0 con 0x7fa20c156350 2023-10-30T12:29:03.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.168+0000 7fa1f37fe640 1 -- 172.21.15.89:0/2077303524 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc004530 con 0x7fa20c156350 2023-10-30T12:29:03.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.168+0000 7fa211646640 1 -- 172.21.15.89:0/2077303524 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa20c1eb250 con 0x7fa20c156350 2023-10-30T12:29:03.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.168+0000 7fa211646640 1 -- 172.21.15.89:0/2077303524 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa20c1eb810 con 0x7fa20c156350 2023-10-30T12:29:03.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.168+0000 7f62f5ffb640 1 -- 172.21.15.89:0/3909387308 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc0125f0 con 0x7f6300153f30 2023-10-30T12:29:03.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.168+0000 7fa211646640 1 -- 172.21.15.89:0/2077303524 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa20c1e1fa0 con 0x7fa20c156350 2023-10-30T12:29:03.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.168+0000 7f6307868640 1 -- 172.21.15.89:0/3909387308 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6300154800 con 0x7f6300153f30 2023-10-30T12:29:03.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.168+0000 7f62f5ffb640 1 -- 172.21.15.89:0/3909387308 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62fc012770 con 0x7f6300153f30 2023-10-30T12:29:03.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.168+0000 7f62f5ffb640 1 -- 172.21.15.89:0/3909387308 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc012b20 con 0x7f6300153f30 2023-10-30T12:29:03.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.168+0000 7f6307868640 1 -- 172.21.15.89:0/3909387308 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63000c51f0 con 0x7f6300153f30 2023-10-30T12:29:03.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.172+0000 7f62f5ffb640 1 -- 172.21.15.89:0/3909387308 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f62fc012c80 con 0x7f6300153f30 2023-10-30T12:29:03.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.172+0000 7f6307868640 1 -- 172.21.15.89:0/3909387308 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f630006a470 con 0x7f6300153f30 2023-10-30T12:29:03.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.172+0000 7fa1f37fe640 1 -- 172.21.15.89:0/2077303524 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa1fc0b7070 con 0x7fa20c156350 2023-10-30T12:29:03.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.172+0000 7f62f5ffb640 1 --2- 172.21.15.89:0/3909387308 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f62e406a8b0 0x7f62e406cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.172+0000 7f62f5ffb640 1 -- 172.21.15.89:0/3909387308 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f62fc0842f0 con 0x7f6300153f30 2023-10-30T12:29:03.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.172+0000 7fa1f37fe640 1 --2- 172.21.15.89:0/2077303524 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa1f80664a0 0x7fa1f8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.172+0000 7fa1f37fe640 1 -- 172.21.15.89:0/2077303524 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa1fc120ce0 con 0x7fa20c156350 2023-10-30T12:29:03.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.172+0000 7f6304e03640 1 --2- 172.21.15.89:0/3909387308 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f62e406a8b0 0x7f62e406cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.172+0000 7fa20affd640 1 --2- 172.21.15.89:0/2077303524 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa1f80664a0 0x7fa1f8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.172+0000 7f6304e03640 1 --2- 172.21.15.89:0/3909387308 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f62e406a8b0 0x7f62e406cd70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f62f00aa000 tx=0x7f62f00b1040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:03.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.176+0000 7fa20affd640 1 --2- 172.21.15.89:0/2077303524 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa1f80664a0 0x7fa1f8068960 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa204007fd0 tx=0x7fa204009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:03.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.176+0000 7fa1457fa640 1 -- 172.21.15.89:0/729009924 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa140054c10 con 0x7fa14815eda0 2023-10-30T12:29:03.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.176+0000 7fa1f37fe640 1 -- 172.21.15.89:0/2077303524 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa1fc0ee210 con 0x7fa20c156350 2023-10-30T12:29:03.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.180+0000 7fb570ff9640 1 -- 172.21.15.89:0/1428133338 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb574095a10 con 0x7fb56c0d8e50 2023-10-30T12:29:03.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.180+0000 7fb570ff9640 1 --2- 172.21.15.89:0/1428133338 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb5580664f0 0x7fb5580689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.180+0000 7fb570ff9640 1 -- 172.21.15.89:0/1428133338 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb574101bc0 con 0x7fb56c0d8e50 2023-10-30T12:29:03.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.180+0000 7fb573fff640 1 --2- 172.21.15.89:0/1428133338 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb5580664f0 0x7fb5580689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.184+0000 7fb573fff640 1 --2- 172.21.15.89:0/1428133338 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb5580664f0 0x7fb5580689b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb5680013e0 tx=0x7fb568001040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:03.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.188+0000 7f62f5ffb640 1 -- 172.21.15.89:0/3909387308 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f62fc051640 con 0x7f6300153f30 2023-10-30T12:29:03.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.188+0000 7fb570ff9640 1 -- 172.21.15.89:0/1428133338 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb5740cf1b0 con 0x7fb56c0d8e50 2023-10-30T12:29:03.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.220+0000 7f41b3fff640 1 Processor -- start 2023-10-30T12:29:03.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.220+0000 7f41b3fff640 1 -- start start 2023-10-30T12:29:03.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.220+0000 7f41b3fff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f41b40700c0 0x7f41b40c2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.220+0000 7f41b3fff640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f41b40713a0 0x7f41b4069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.220+0000 7f41b3fff640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b4071830 con 0x7f41b4152eb0 2023-10-30T12:29:03.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.220+0000 7f41b3fff640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41b4155e30 con 0x7f41b40700c0 2023-10-30T12:29:03.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.220+0000 7f41b3fff640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41b4156670 con 0x7f41b40713a0 2023-10-30T12:29:03.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.220+0000 7f41aa7fc640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f41b40700c0 0x7f41b40c2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.220+0000 7f41aa7fc640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f41b40700c0 0x7f41b40c2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39290/0 (socket says 172.21.15.89:39290) 2023-10-30T12:29:03.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.220+0000 7f41aa7fc640 1 -- 172.21.15.89:0/2601824942 learned_addr learned my addr 172.21.15.89:0/2601824942 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:03.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.224+0000 7f41aa7fc640 1 -- 172.21.15.89:0/2601824942 >> v2:172.21.15.192:3300/0 conn(0x7f41b40713a0 msgr2=0x7f41b4069e90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-30T12:29:03.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.224+0000 7f41aa7fc640 1 --2- 172.21.15.89:0/2601824942 >> v2:172.21.15.192:3300/0 conn(0x7f41b40713a0 0x7f41b4069e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.224+0000 7f41aa7fc640 1 -- 172.21.15.89:0/2601824942 >> v1:172.21.15.192:6789/0 conn(0x7f41b4152eb0 legacy=0x7f41b4155720 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-30T12:29:03.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.224+0000 7f41aa7fc640 1 -- 172.21.15.89:0/2601824942 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b41560b0 con 0x7f41b40700c0 2023-10-30T12:29:03.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.228+0000 7f41aa7fc640 1 --2- 172.21.15.89:0/2601824942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41b40700c0 0x7f41b40c2890 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f41b406c820 tx=0x7f41a401c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f088d3640bbd0fc5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:03.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.228+0000 7f41b27fc640 1 -- 172.21.15.89:0/2601824942 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a40a5750 con 0x7f41b40700c0 2023-10-30T12:29:03.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.228+0000 7f41b27fc640 1 -- 172.21.15.89:0/2601824942 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41a40a58b0 con 0x7f41b40700c0 2023-10-30T12:29:03.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.228+0000 7f41b27fc640 1 -- 172.21.15.89:0/2601824942 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a40a5c40 con 0x7f41b40700c0 2023-10-30T12:29:03.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.232+0000 7f41b3fff640 1 -- 172.21.15.89:0/2601824942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41b40700c0 msgr2=0x7f41b40c2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.232+0000 7f41b3fff640 1 --2- 172.21.15.89:0/2601824942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41b40700c0 0x7f41b40c2890 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f41b406c820 tx=0x7f41a401c980 comp rx=0 tx=0).stop 2023-10-30T12:29:03.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.232+0000 7f41b3fff640 1 -- 172.21.15.89:0/2601824942 shutdown_connections 2023-10-30T12:29:03.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.232+0000 7f41b3fff640 1 --2- 172.21.15.89:0/2601824942 >> v2:172.21.15.192:3300/0 conn(0x7f41b40713a0 0x7f41b4069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.232+0000 7f41b3fff640 1 --2- 172.21.15.89:0/2601824942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41b40700c0 0x7f41b40c2890 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.232+0000 7f41b3fff640 1 -- 172.21.15.89:0/2601824942 >> 172.21.15.89:0/2601824942 conn(0x7f41b40c75d0 msgr2=0x7f41b40c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:03.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.232+0000 7f41b3fff640 1 -- 172.21.15.89:0/2601824942 shutdown_connections 2023-10-30T12:29:03.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.232+0000 7f41b3fff640 1 -- 172.21.15.89:0/2601824942 wait complete. 2023-10-30T12:29:03.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.236+0000 7f41b3fff640 1 Processor -- start 2023-10-30T12:29:03.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.236+0000 7f41b3fff640 1 -- start start 2023-10-30T12:29:03.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.236+0000 7f41b3fff640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41b40713a0 0x7f41b413db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.236+0000 7f41b3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41b4152eb0 0x7f41b41482f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.236+0000 7f41b3fff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41b40c7190 0x7f41b40c3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.236+0000 7f41b2ffd640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41b40713a0 0x7f41b413db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.236+0000 7f41b2ffd640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41b40713a0 0x7f41b413db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:57560/0 (socket says 172.21.15.89:57560) 2023-10-30T12:29:03.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.236+0000 7f41b3fff640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b40c6ed0 con 0x7f41b4152eb0 2023-10-30T12:29:03.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.236+0000 7f41b3fff640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b41f1610 con 0x7f41b40c7190 2023-10-30T12:29:03.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.236+0000 7f41b3fff640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41b41f1b40 con 0x7f41b40713a0 2023-10-30T12:29:03.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.240+0000 7f41b2ffd640 1 -- 172.21.15.89:0/2231086092 learned_addr learned my addr 172.21.15.89:0/2231086092 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:03.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.240+0000 7f41aa7fc640 1 --2- 172.21.15.89:0/2231086092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41b4152eb0 0x7f41b41482f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.240+0000 7f41b37fe640 1 --2- 172.21.15.89:0/2231086092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41b40c7190 0x7f41b40c3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.240+0000 7f41b2ffd640 1 -- 172.21.15.89:0/2231086092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41b40c7190 msgr2=0x7f41b40c3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.240+0000 7f41b2ffd640 1 --2- 172.21.15.89:0/2231086092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41b40c7190 0x7f41b40c3e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.240+0000 7f41b2ffd640 1 -- 172.21.15.89:0/2231086092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41b4152eb0 msgr2=0x7f41b41482f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.240+0000 7f41b2ffd640 1 --2- 172.21.15.89:0/2231086092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41b4152eb0 0x7f41b41482f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.240+0000 7f41b2ffd640 1 -- 172.21.15.89:0/2231086092 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a4009b30 con 0x7f41b40713a0 2023-10-30T12:29:03.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.240+0000 7f41b2ffd640 1 --2- 172.21.15.89:0/2231086092 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41b40713a0 0x7f41b413db50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f419c00b450 tx=0x7f419c00b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:03.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.240+0000 7f41abfff640 1 -- 172.21.15.89:0/2231086092 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419c011890 con 0x7f41b40713a0 2023-10-30T12:29:03.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.240+0000 7f41abfff640 1 -- 172.21.15.89:0/2231086092 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f419c0119f0 con 0x7f41b40713a0 2023-10-30T12:29:03.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.240+0000 7f41abfff640 1 -- 172.21.15.89:0/2231086092 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419c011dc0 con 0x7f41b40713a0 2023-10-30T12:29:03.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.240+0000 7f41b3fff640 1 -- 172.21.15.89:0/2231086092 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41b413e090 con 0x7f41b40713a0 2023-10-30T12:29:03.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.240+0000 7f41b3fff640 1 -- 172.21.15.89:0/2231086092 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41b4143230 con 0x7f41b40713a0 2023-10-30T12:29:03.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.248+0000 7f41b3fff640 1 -- 172.21.15.89:0/2231086092 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4198005260 con 0x7f41b40713a0 2023-10-30T12:29:03.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.248+0000 7f41abfff640 1 -- 172.21.15.89:0/2231086092 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f419c01a400 con 0x7f41b40713a0 2023-10-30T12:29:03.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.248+0000 7f41abfff640 1 --2- 172.21.15.89:0/2231086092 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f41940664d0 0x7f4194068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.248+0000 7f41abfff640 1 -- 172.21.15.89:0/2231086092 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f419c0841e0 con 0x7f41b40713a0 2023-10-30T12:29:03.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.252+0000 7f41aa7fc640 1 --2- 172.21.15.89:0/2231086092 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f41940664d0 0x7f4194068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.252+0000 7f41aa7fc640 1 --2- 172.21.15.89:0/2231086092 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f41940664d0 0x7f4194068990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f41a40ae000 tx=0x7f41a40a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:03.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.260+0000 7f41abfff640 1 -- 172.21.15.89:0/2231086092 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f419c0515c0 con 0x7f41b40713a0 2023-10-30T12:29:03.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.332+0000 7ff42fe09640 1 -- 172.21.15.89:0/728312692 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7ff4280c2650 con 0x7ff42806b310 2023-10-30T12:29:03.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.332+0000 7ff41e7fc640 1 -- 172.21.15.89:0/728312692 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff4200eb540 con 0x7ff42806b310 2023-10-30T12:29:03.341 INFO:teuthology.orchestra.run.smithi089.stdout:60129542153 2023-10-30T12:29:03.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.336+0000 7ff42fe09640 1 -- 172.21.15.89:0/728312692 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff40c0665a0 msgr2=0x7ff40c068a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.336+0000 7ff42fe09640 1 --2- 172.21.15.89:0/728312692 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff40c0665a0 0x7ff40c068a60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff424006fd0 tx=0x7ff424008040 comp rx=0 tx=0).stop 2023-10-30T12:29:03.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.336+0000 7ff42fe09640 1 -- 172.21.15.89:0/728312692 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff42806b310 msgr2=0x7ff42806b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.336+0000 7ff42fe09640 1 --2- 172.21.15.89:0/728312692 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff42806b310 0x7ff42806b790 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff42001cd50 tx=0x7ff414005800 comp rx=0 tx=0).stop 2023-10-30T12:29:03.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.336+0000 7ff42fe09640 1 -- 172.21.15.89:0/728312692 shutdown_connections 2023-10-30T12:29:03.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.336+0000 7ff42fe09640 1 --2- 172.21.15.89:0/728312692 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff40c0665a0 0x7ff40c068a60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.336+0000 7ff42fe09640 1 --2- 172.21.15.89:0/728312692 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff42806b310 0x7ff42806b790 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.336+0000 7ff42fe09640 1 --2- 172.21.15.89:0/728312692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff42806cd70 0x7ff42814c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.336+0000 7ff42fe09640 1 --2- 172.21.15.89:0/728312692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4280c7830 0x7ff42814be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.336+0000 7ff42fe09640 1 -- 172.21.15.89:0/728312692 >> 172.21.15.89:0/728312692 conn(0x7ff428071ea0 msgr2=0x7ff428073480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:03.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.340+0000 7ff42fe09640 1 -- 172.21.15.89:0/728312692 shutdown_connections 2023-10-30T12:29:03.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.340+0000 7ff42fe09640 1 -- 172.21.15.89:0/728312692 wait complete. 2023-10-30T12:29:03.378 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542153 for osd.0 2023-10-30T12:29:03.378 DEBUG:teuthology.parallel:result is None 2023-10-30T12:29:03.419 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.412+0000 7f10e7f3e640 1 -- 172.21.15.89:0/2866946276 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f10c80057e0 con 0x7f10e00c2290 2023-10-30T12:29:03.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.416+0000 7f10d67fc640 1 -- 172.21.15.89:0/2866946276 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f10dc00b440 con 0x7f10e00c2290 2023-10-30T12:29:03.423 INFO:teuthology.orchestra.run.smithi089.stdout:60129542153 2023-10-30T12:29:03.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.420+0000 7f10e7f3e640 1 -- 172.21.15.89:0/2866946276 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f10c4066500 msgr2=0x7f10c40689c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.420+0000 7f10e7f3e640 1 --2- 172.21.15.89:0/2866946276 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f10c4066500 0x7f10c40689c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f10d0009b60 tx=0x7f10d0008040 comp rx=0 tx=0).stop 2023-10-30T12:29:03.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.420+0000 7f10e7f3e640 1 -- 172.21.15.89:0/2866946276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10e00c2290 msgr2=0x7f10e00be4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.420+0000 7f10e7f3e640 1 --2- 172.21.15.89:0/2866946276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10e00c2290 0x7f10e00be4e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f10dc00d800 tx=0x7f10dc00dcc0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.420+0000 7f10e7f3e640 1 -- 172.21.15.89:0/2866946276 shutdown_connections 2023-10-30T12:29:03.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.420+0000 7f10e7f3e640 1 --2- 172.21.15.89:0/2866946276 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f10c4066500 0x7f10c40689c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.420+0000 7f10e7f3e640 1 --2- 172.21.15.89:0/2866946276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10e00c2290 0x7f10e00be4e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.420+0000 7f10e7f3e640 1 --2- 172.21.15.89:0/2866946276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10e014c140 0x7f10e01fb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.420+0000 7f10e7f3e640 1 --2- 172.21.15.89:0/2866946276 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10e006d0e0 0x7f10e00bdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.420+0000 7f10e7f3e640 1 -- 172.21.15.89:0/2866946276 >> 172.21.15.89:0/2866946276 conn(0x7f10e00cac50 msgr2=0x7f10e00c12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:03.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.420+0000 7f10e7f3e640 1 -- 172.21.15.89:0/2866946276 shutdown_connections 2023-10-30T12:29:03.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.420+0000 7f10e7f3e640 1 -- 172.21.15.89:0/2866946276 wait complete. 2023-10-30T12:29:03.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.448+0000 7ff944ff9640 1 -- 172.21.15.89:0/3789257816 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7ff958158790 con 0x7ff958152de0 2023-10-30T12:29:03.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.448+0000 7ff946ffd640 1 -- 172.21.15.89:0/3789257816 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff950054e90 con 0x7ff958152de0 2023-10-30T12:29:03.455 INFO:teuthology.orchestra.run.smithi089.stdout:60129542153 2023-10-30T12:29:03.456 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542153 for osd.2 2023-10-30T12:29:03.456 DEBUG:teuthology.parallel:result is None 2023-10-30T12:29:03.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.452+0000 7ff944ff9640 1 -- 172.21.15.89:0/3789257816 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff9380665a0 msgr2=0x7ff938068a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.452+0000 7ff944ff9640 1 --2- 172.21.15.89:0/3789257816 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff9380665a0 0x7ff938068a60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff9480011f0 tx=0x7ff948009040 comp rx=0 tx=0).stop 2023-10-30T12:29:03.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.452+0000 7ff944ff9640 1 -- 172.21.15.89:0/3789257816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff958152de0 msgr2=0x7ff95814b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.452+0000 7ff944ff9640 1 --2- 172.21.15.89:0/3789257816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff958152de0 0x7ff95814b1d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff950009d90 tx=0x7ff95000c490 comp rx=0 tx=0).stop 2023-10-30T12:29:03.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.452+0000 7ff944ff9640 1 -- 172.21.15.89:0/3789257816 shutdown_connections 2023-10-30T12:29:03.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.452+0000 7ff944ff9640 1 --2- 172.21.15.89:0/3789257816 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff9380665a0 0x7ff938068a60 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.452+0000 7ff944ff9640 1 --2- 172.21.15.89:0/3789257816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff95806ad90 0x7ff95814ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.452+0000 7ff944ff9640 1 --2- 172.21.15.89:0/3789257816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff958152de0 0x7ff95814b1d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.452+0000 7ff944ff9640 1 --2- 172.21.15.89:0/3789257816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff958069e90 0x7ff9581fc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.452+0000 7ff944ff9640 1 -- 172.21.15.89:0/3789257816 >> 172.21.15.89:0/3789257816 conn(0x7ff9580c7910 msgr2=0x7ff958151040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:03.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.452+0000 7ff944ff9640 1 -- 172.21.15.89:0/3789257816 shutdown_connections 2023-10-30T12:29:03.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.452+0000 7ff944ff9640 1 -- 172.21.15.89:0/3789257816 wait complete. 2023-10-30T12:29:03.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.456+0000 7fb5796ad640 1 -- 172.21.15.89:0/1428133338 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fb5540050d0 con 0x7fb56c0d8e50 2023-10-30T12:29:03.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.456+0000 7fb570ff9640 1 -- 172.21.15.89:0/1428133338 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb5740d27d0 con 0x7fb56c0d8e50 2023-10-30T12:29:03.462 INFO:teuthology.orchestra.run.smithi089.stdout:60129542153 2023-10-30T12:29:03.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.460+0000 7fb5796ad640 1 -- 172.21.15.89:0/1428133338 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb5580664f0 msgr2=0x7fb5580689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.460+0000 7fb5796ad640 1 --2- 172.21.15.89:0/1428133338 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb5580664f0 0x7fb5580689b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb5680013e0 tx=0x7fb568001040 comp rx=0 tx=0).stop 2023-10-30T12:29:03.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.460+0000 7fb5796ad640 1 -- 172.21.15.89:0/1428133338 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb56c0d8e50 msgr2=0x7fb56c0d41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.460+0000 7fb5796ad640 1 --2- 172.21.15.89:0/1428133338 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb56c0d8e50 0x7fb56c0d41a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb574000b70 tx=0x7fb574071490 comp rx=0 tx=0).stop 2023-10-30T12:29:03.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.460+0000 7fb5796ad640 1 -- 172.21.15.89:0/1428133338 shutdown_connections 2023-10-30T12:29:03.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.460+0000 7fb5796ad640 1 --2- 172.21.15.89:0/1428133338 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb5580664f0 0x7fb5580689b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.460+0000 7fb5796ad640 1 --2- 172.21.15.89:0/1428133338 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb56c0d8e50 0x7fb56c0d41a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.460+0000 7fb5796ad640 1 --2- 172.21.15.89:0/1428133338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb56c050a30 0x7fb56c04ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.460+0000 7fb5796ad640 1 --2- 172.21.15.89:0/1428133338 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb56c04f420 0x7fb56c04e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.460+0000 7fb5796ad640 1 -- 172.21.15.89:0/1428133338 >> 172.21.15.89:0/1428133338 conn(0x7fb56c051040 msgr2=0x7fb56c005a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:03.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.460+0000 7fb5796ad640 1 -- 172.21.15.89:0/1428133338 shutdown_connections 2023-10-30T12:29:03.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.460+0000 7fb5796ad640 1 -- 172.21.15.89:0/1428133338 wait complete. 2023-10-30T12:29:03.480 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542153 for osd.3 2023-10-30T12:29:03.480 DEBUG:teuthology.parallel:result is None 2023-10-30T12:29:03.485 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542153 for osd.7 2023-10-30T12:29:03.485 DEBUG:teuthology.parallel:result is None 2023-10-30T12:29:03.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.488+0000 7fa211646640 1 -- 172.21.15.89:0/2077303524 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fa20c1e1c00 con 0x7fa20c156350 2023-10-30T12:29:03.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.488+0000 7fa1f37fe640 1 -- 172.21.15.89:0/2077303524 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa1fc0b5020 con 0x7fa20c156350 2023-10-30T12:29:03.495 INFO:teuthology.orchestra.run.smithi089.stdout:60129542153 2023-10-30T12:29:03.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.492+0000 7fa211646640 1 -- 172.21.15.89:0/2077303524 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa1f80664a0 msgr2=0x7fa1f8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.492+0000 7fa211646640 1 --2- 172.21.15.89:0/2077303524 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa1f80664a0 0x7fa1f8068960 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa204007fd0 tx=0x7fa204009040 comp rx=0 tx=0).stop 2023-10-30T12:29:03.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.492+0000 7fa211646640 1 -- 172.21.15.89:0/2077303524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa20c156350 msgr2=0x7fa20c1e22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.492+0000 7fa211646640 1 --2- 172.21.15.89:0/2077303524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa20c156350 0x7fa20c1e22c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc01cf10 tx=0x7fa1fc01d0a0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.492+0000 7fa211646640 1 -- 172.21.15.89:0/2077303524 shutdown_connections 2023-10-30T12:29:03.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.492+0000 7fa211646640 1 --2- 172.21.15.89:0/2077303524 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa1f80664a0 0x7fa1f8068960 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.492+0000 7fa211646640 1 --2- 172.21.15.89:0/2077303524 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa20c14fc00 0x7fa20c155890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.492+0000 7fa211646640 1 --2- 172.21.15.89:0/2077303524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa20c156350 0x7fa20c1e22c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.492+0000 7fa211646640 1 --2- 172.21.15.89:0/2077303524 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa20c0c74e0 0x7fa20c0c1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.492+0000 7fa211646640 1 -- 172.21.15.89:0/2077303524 >> 172.21.15.89:0/2077303524 conn(0x7fa20c0c7910 msgr2=0x7fa20c14df40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:03.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.492+0000 7fa211646640 1 -- 172.21.15.89:0/2077303524 shutdown_connections 2023-10-30T12:29:03.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.492+0000 7fa211646640 1 -- 172.21.15.89:0/2077303524 wait complete. 2023-10-30T12:29:03.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.500+0000 7fa14e9a1640 1 -- 172.21.15.89:0/729009924 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fa1480b0860 con 0x7fa14815eda0 2023-10-30T12:29:03.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.500+0000 7fa1457fa640 1 -- 172.21.15.89:0/729009924 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa140019090 con 0x7fa14815eda0 2023-10-30T12:29:03.506 INFO:teuthology.orchestra.run.smithi089.stdout:60129542152 2023-10-30T12:29:03.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.504+0000 7fa14e9a1640 1 -- 172.21.15.89:0/729009924 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa1280664f0 msgr2=0x7fa1280689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.504+0000 7fa14e9a1640 1 --2- 172.21.15.89:0/729009924 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa1280664f0 0x7fa1280689b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa130009a40 tx=0x7fa130008040 comp rx=0 tx=0).stop 2023-10-30T12:29:03.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.504+0000 7fa14e9a1640 1 -- 172.21.15.89:0/729009924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa14815eda0 msgr2=0x7fa14815fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.504+0000 7fa14e9a1640 1 --2- 172.21.15.89:0/729009924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa14815eda0 0x7fa14815fb80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa14000b970 tx=0x7fa14000be30 comp rx=0 tx=0).stop 2023-10-30T12:29:03.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.504+0000 7fa14e9a1640 1 -- 172.21.15.89:0/729009924 shutdown_connections 2023-10-30T12:29:03.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.504+0000 7fa14e9a1640 1 --2- 172.21.15.89:0/729009924 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa1280664f0 0x7fa1280689b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.504+0000 7fa14e9a1640 1 --2- 172.21.15.89:0/729009924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa14815eda0 0x7fa14815fb80 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.504+0000 7fa14e9a1640 1 --2- 172.21.15.89:0/729009924 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa148158090 0x7fa14815f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.504+0000 7fa14e9a1640 1 --2- 172.21.15.89:0/729009924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa14806a3d0 0x7fa148149b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.504+0000 7fa14e9a1640 1 -- 172.21.15.89:0/729009924 >> 172.21.15.89:0/729009924 conn(0x7fa1480c7910 msgr2=0x7fa14815c710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:03.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.504+0000 7fa14e9a1640 1 -- 172.21.15.89:0/729009924 shutdown_connections 2023-10-30T12:29:03.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.504+0000 7fa14e9a1640 1 -- 172.21.15.89:0/729009924 wait complete. 2023-10-30T12:29:03.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.512+0000 7f6307868640 1 -- 172.21.15.89:0/3909387308 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f63000b0af0 con 0x7f6300153f30 2023-10-30T12:29:03.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.512+0000 7f62f5ffb640 1 -- 172.21.15.89:0/3909387308 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f62fc054c60 con 0x7f6300153f30 2023-10-30T12:29:03.519 INFO:teuthology.orchestra.run.smithi089.stdout:60129542153 2023-10-30T12:29:03.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.516+0000 7f6307868640 1 -- 172.21.15.89:0/3909387308 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f62e406a8b0 msgr2=0x7f62e406cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.516+0000 7f6307868640 1 --2- 172.21.15.89:0/3909387308 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f62e406a8b0 0x7f62e406cd70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f62f00aa000 tx=0x7f62f00b1040 comp rx=0 tx=0).stop 2023-10-30T12:29:03.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.516+0000 7f6307868640 1 -- 172.21.15.89:0/3909387308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6300153f30 msgr2=0x7f6300144650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.516+0000 7f6307868640 1 --2- 172.21.15.89:0/3909387308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6300153f30 0x7f6300144650 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f62fc00d4d0 tx=0x7f62fc00d990 comp rx=0 tx=0).stop 2023-10-30T12:29:03.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.516+0000 7f6307868640 1 -- 172.21.15.89:0/3909387308 shutdown_connections 2023-10-30T12:29:03.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.516+0000 7f6307868640 1 --2- 172.21.15.89:0/3909387308 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f62e406a8b0 0x7f62e406cd70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.516+0000 7f6307868640 1 --2- 172.21.15.89:0/3909387308 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63001533a0 0x7f6300155000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.516+0000 7f6307868640 1 --2- 172.21.15.89:0/3909387308 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6300154310 0x7f6300144b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.516+0000 7f6307868640 1 --2- 172.21.15.89:0/3909387308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6300153f30 0x7f6300144650 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.516+0000 7f6307868640 1 -- 172.21.15.89:0/3909387308 >> 172.21.15.89:0/3909387308 conn(0x7f63000c7910 msgr2=0x7f63000ca480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:03.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.516+0000 7f6307868640 1 -- 172.21.15.89:0/3909387308 shutdown_connections 2023-10-30T12:29:03.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.516+0000 7f6307868640 1 -- 172.21.15.89:0/3909387308 wait complete. 2023-10-30T12:29:03.527 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542153 for osd.6 2023-10-30T12:29:03.527 DEBUG:teuthology.parallel:result is None 2023-10-30T12:29:03.539 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542152 for osd.4 2023-10-30T12:29:03.539 DEBUG:teuthology.parallel:result is None 2023-10-30T12:29:03.543 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542153 for osd.5 2023-10-30T12:29:03.543 DEBUG:teuthology.parallel:result is None 2023-10-30T12:29:03.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.556+0000 7f41b3fff640 1 -- 172.21.15.89:0/2231086092 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f4198005510 con 0x7f41b40713a0 2023-10-30T12:29:03.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.556+0000 7f41abfff640 1 -- 172.21.15.89:0/2231086092 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f419c015020 con 0x7f41b40713a0 2023-10-30T12:29:03.561 INFO:teuthology.orchestra.run.smithi089.stdout:60129542153 2023-10-30T12:29:03.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.556+0000 7f41b3fff640 1 -- 172.21.15.89:0/2231086092 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f41940664d0 msgr2=0x7f4194068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.556+0000 7f41b3fff640 1 --2- 172.21.15.89:0/2231086092 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f41940664d0 0x7f4194068990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f41a40ae000 tx=0x7f41a40a7070 comp rx=0 tx=0).stop 2023-10-30T12:29:03.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.556+0000 7f41b3fff640 1 -- 172.21.15.89:0/2231086092 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41b40713a0 msgr2=0x7f41b413db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.556+0000 7f41b3fff640 1 --2- 172.21.15.89:0/2231086092 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41b40713a0 0x7f41b413db50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f419c00b450 tx=0x7f419c00b910 comp rx=0 tx=0).stop 2023-10-30T12:29:03.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.556+0000 7f41b3fff640 1 -- 172.21.15.89:0/2231086092 shutdown_connections 2023-10-30T12:29:03.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.556+0000 7f41b3fff640 1 --2- 172.21.15.89:0/2231086092 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f41940664d0 0x7f4194068990 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.556+0000 7f41b3fff640 1 --2- 172.21.15.89:0/2231086092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41b40c7190 0x7f41b40c3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.556+0000 7f41b3fff640 1 --2- 172.21.15.89:0/2231086092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41b4152eb0 0x7f41b41482f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.556+0000 7f41b3fff640 1 --2- 172.21.15.89:0/2231086092 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41b40713a0 0x7f41b413db50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.556+0000 7f41b3fff640 1 -- 172.21.15.89:0/2231086092 >> 172.21.15.89:0/2231086092 conn(0x7f41b40c75d0 msgr2=0x7f41b406ca90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:03.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.556+0000 7f41b3fff640 1 -- 172.21.15.89:0/2231086092 shutdown_connections 2023-10-30T12:29:03.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.560+0000 7f41b3fff640 1 -- 172.21.15.89:0/2231086092 wait complete. 2023-10-30T12:29:03.582 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542153 for osd.1 2023-10-30T12:29:03.582 DEBUG:teuthology.parallel:result is None 2023-10-30T12:29:03.583 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-30T12:29:03.583 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:29:03.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.756+0000 7f0b01046640 1 Processor -- start 2023-10-30T12:29:03.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.756+0000 7f0b01046640 1 -- start start 2023-10-30T12:29:03.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.760+0000 7f0b01046640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0afc06e470 0x7f0afc06ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.760+0000 7f0b01046640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0afc0c8490 0x7f0afc0c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.760+0000 7f0b01046640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0afc06afa0 con 0x7f0afc153d70 2023-10-30T12:29:03.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.760+0000 7f0b01046640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0afc06d780 con 0x7f0afc06e470 2023-10-30T12:29:03.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.760+0000 7f0b01046640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0afc154150 con 0x7f0afc0c8490 2023-10-30T12:29:03.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.760+0000 7f0afa59c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0afc06e470 0x7f0afc06ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.760+0000 7f0afa59c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0afc06e470 0x7f0afc06ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39312/0 (socket says 172.21.15.89:39312) 2023-10-30T12:29:03.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.760+0000 7f0afa59c640 1 -- 172.21.15.89:0/3443159050 learned_addr learned my addr 172.21.15.89:0/3443159050 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:03.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.760+0000 7f0afad9d640 1 --1- 172.21.15.89:0/3443159050 >> v1:172.21.15.89:6789/0 conn(0x7f0afc153d70 0x7f0afc0cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:52194/0 (socket says 172.21.15.89:52194) 2023-10-30T12:29:03.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.760+0000 7f0afb59e640 1 --2- 172.21.15.89:0/3443159050 >> v2:172.21.15.192:3300/0 conn(0x7f0afc0c8490 0x7f0afc0c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.760+0000 7f0afa59c640 1 -- 172.21.15.89:0/3443159050 >> v2:172.21.15.192:3300/0 conn(0x7f0afc0c8490 msgr2=0x7f0afc0c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:03.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.760+0000 7f0afa59c640 1 --2- 172.21.15.89:0/3443159050 >> v2:172.21.15.192:3300/0 conn(0x7f0afc0c8490 0x7f0afc0c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.760+0000 7f0afa59c640 1 -- 172.21.15.89:0/3443159050 >> v1:172.21.15.89:6789/0 conn(0x7f0afc153d70 legacy=0x7f0afc0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.760+0000 7f0afa59c640 1 -- 172.21.15.89:0/3443159050 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0afc06e030 con 0x7f0afc06e470 2023-10-30T12:29:03.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.760+0000 7f0afb59e640 1 --2- 172.21.15.89:0/3443159050 >> v2:172.21.15.192:3300/0 conn(0x7f0afc0c8490 0x7f0afc0c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:29:03.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.760+0000 7f0afa59c640 1 --2- 172.21.15.89:0/3443159050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0afc06e470 0x7f0afc06ca80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0afc0cddf0 tx=0x7f0af001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d7b30a30f8019a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:03.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.764+0000 7f0af9d9b640 1 -- 172.21.15.89:0/3443159050 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af00aa430 con 0x7f0afc06e470 2023-10-30T12:29:03.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.764+0000 7f0af9d9b640 1 -- 172.21.15.89:0/3443159050 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0af00aa590 con 0x7f0afc06e470 2023-10-30T12:29:03.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.764+0000 7f0af9d9b640 1 -- 172.21.15.89:0/3443159050 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af00aa920 con 0x7f0afc06e470 2023-10-30T12:29:03.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.764+0000 7f0b01046640 1 -- 172.21.15.89:0/3443159050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0afc06e470 msgr2=0x7f0afc06ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.764+0000 7f0b01046640 1 --2- 172.21.15.89:0/3443159050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0afc06e470 0x7f0afc06ca80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0afc0cddf0 tx=0x7f0af001c4d0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.764+0000 7f0b01046640 1 -- 172.21.15.89:0/3443159050 shutdown_connections 2023-10-30T12:29:03.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.764+0000 7f0b01046640 1 --2- 172.21.15.89:0/3443159050 >> v2:172.21.15.192:3300/0 conn(0x7f0afc0c8490 0x7f0afc0c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.764+0000 7f0b01046640 1 --2- 172.21.15.89:0/3443159050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0afc06e470 0x7f0afc06ca80 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.764+0000 7f0b01046640 1 -- 172.21.15.89:0/3443159050 >> 172.21.15.89:0/3443159050 conn(0x7f0afc071ea0 msgr2=0x7f0afc065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:03.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.764+0000 7f0b01046640 1 -- 172.21.15.89:0/3443159050 shutdown_connections 2023-10-30T12:29:03.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.764+0000 7f0b01046640 1 -- 172.21.15.89:0/3443159050 wait complete. 2023-10-30T12:29:03.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.764+0000 7f0b01046640 1 Processor -- start 2023-10-30T12:29:03.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.764+0000 7f0b01046640 1 -- start start 2023-10-30T12:29:03.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.764+0000 7f0b01046640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0afc06e470 0x7f0afc144a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.764+0000 7f0b01046640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0afc0c8490 0x7f0afc143a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.764+0000 7f0b01046640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0afc153d70 0x7f0afc143f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.764+0000 7f0b01046640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0afc0cb680 con 0x7f0afc0c8490 2023-10-30T12:29:03.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.764+0000 7f0b01046640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0afc0cb960 con 0x7f0afc06e470 2023-10-30T12:29:03.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.764+0000 7f0b01046640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0afc1462d0 con 0x7f0afc153d70 2023-10-30T12:29:03.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.764+0000 7f0afa59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0afc0c8490 0x7f0afc143a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.768+0000 7f0afa59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0afc0c8490 0x7f0afc143a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39316/0 (socket says 172.21.15.89:39316) 2023-10-30T12:29:03.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.768+0000 7f0afa59c640 1 -- 172.21.15.89:0/3935873407 learned_addr learned my addr 172.21.15.89:0/3935873407 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:03.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.768+0000 7f0afad9d640 1 --2- 172.21.15.89:0/3935873407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0afc06e470 0x7f0afc144a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.768+0000 7f0afb59e640 1 --2- 172.21.15.89:0/3935873407 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0afc153d70 0x7f0afc143f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.768+0000 7f0afa59c640 1 -- 172.21.15.89:0/3935873407 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0afc153d70 msgr2=0x7f0afc143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.768+0000 7f0afa59c640 1 --2- 172.21.15.89:0/3935873407 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0afc153d70 0x7f0afc143f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.768+0000 7f0afa59c640 1 -- 172.21.15.89:0/3935873407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0afc06e470 msgr2=0x7f0afc144a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.768+0000 7f0afa59c640 1 --2- 172.21.15.89:0/3935873407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0afc06e470 0x7f0afc144a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.768+0000 7f0afa59c640 1 -- 172.21.15.89:0/3935873407 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af00094d0 con 0x7f0afc0c8490 2023-10-30T12:29:03.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.768+0000 7f0afa59c640 1 --2- 172.21.15.89:0/3935873407 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0afc0c8490 0x7f0afc143a30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0af001c9b0 tx=0x7f0af001ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:03.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.768+0000 7f0ae37fe640 1 -- 172.21.15.89:0/3935873407 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af00b3800 con 0x7f0afc0c8490 2023-10-30T12:29:03.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.768+0000 7f0ae37fe640 1 -- 172.21.15.89:0/3935873407 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0af00b39a0 con 0x7f0afc0c8490 2023-10-30T12:29:03.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.768+0000 7f0b01046640 1 -- 172.21.15.89:0/3935873407 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0afc0cbdf0 con 0x7f0afc0c8490 2023-10-30T12:29:03.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.768+0000 7f0ae37fe640 1 -- 172.21.15.89:0/3935873407 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af00b3d50 con 0x7f0afc0c8490 2023-10-30T12:29:03.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.768+0000 7f0b01046640 1 -- 172.21.15.89:0/3935873407 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0afc146550 con 0x7f0afc0c8490 2023-10-30T12:29:03.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.768+0000 7f0b01046640 1 -- 172.21.15.89:0/3935873407 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0afc04e170 con 0x7f0afc0c8490 2023-10-30T12:29:03.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.768+0000 7f0ae37fe640 1 -- 172.21.15.89:0/3935873407 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0af00c6390 con 0x7f0afc0c8490 2023-10-30T12:29:03.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.772+0000 7f0ae37fe640 1 --2- 172.21.15.89:0/3935873407 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0ae80664a0 0x7f0ae8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.772+0000 7f0ae37fe640 1 -- 172.21.15.89:0/3935873407 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0af011e7e0 con 0x7f0afc0c8490 2023-10-30T12:29:03.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.772+0000 7f0afad9d640 1 --2- 172.21.15.89:0/3935873407 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0ae80664a0 0x7f0ae8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.772+0000 7f0afad9d640 1 --2- 172.21.15.89:0/3935873407 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0ae80664a0 0x7f0ae8068960 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0aec000f70 tx=0x7f0aec007c90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:03.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.776+0000 7f0ae37fe640 1 -- 172.21.15.89:0/3935873407 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0af00ebbc0 con 0x7f0afc0c8490 2023-10-30T12:29:03.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.920+0000 7f0b01046640 1 -- 172.21.15.89:0/3935873407 --> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0afc0c9610 con 0x7f0ae80664a0 2023-10-30T12:29:03.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.944+0000 7f0ae37fe640 1 -- 172.21.15.89:0/3935873407 <== mgr.4105 v2:172.21.15.192:6832/4076276638 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264637 (secure 0 0 0) 0x7f0aec005a60 con 0x7f0ae80664a0 2023-10-30T12:29:03.948 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:29:03.950 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-10-30T12:29:03.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.944+0000 7f0b01046640 1 -- 172.21.15.89:0/3935873407 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0ae80664a0 msgr2=0x7f0ae8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.944+0000 7f0b01046640 1 --2- 172.21.15.89:0/3935873407 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0ae80664a0 0x7f0ae8068960 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0aec000f70 tx=0x7f0aec007c90 comp rx=0 tx=0).stop 2023-10-30T12:29:03.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.944+0000 7f0b01046640 1 -- 172.21.15.89:0/3935873407 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0afc0c8490 msgr2=0x7f0afc143a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.944+0000 7f0b01046640 1 --2- 172.21.15.89:0/3935873407 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0afc0c8490 0x7f0afc143a30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0af001c9b0 tx=0x7f0af001ad10 comp rx=0 tx=0).stop 2023-10-30T12:29:03.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.944+0000 7f0b01046640 1 -- 172.21.15.89:0/3935873407 shutdown_connections 2023-10-30T12:29:03.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.944+0000 7f0b01046640 1 --2- 172.21.15.89:0/3935873407 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0ae80664a0 0x7f0ae8068960 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.944+0000 7f0b01046640 1 --2- 172.21.15.89:0/3935873407 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0afc153d70 0x7f0afc143f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.944+0000 7f0b01046640 1 --2- 172.21.15.89:0/3935873407 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0afc0c8490 0x7f0afc143a30 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.944+0000 7f0b01046640 1 --2- 172.21.15.89:0/3935873407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0afc06e470 0x7f0afc144a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.944+0000 7f0b01046640 1 -- 172.21.15.89:0/3935873407 >> 172.21.15.89:0/3935873407 conn(0x7f0afc071ea0 msgr2=0x7f0afc150af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:03.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.944+0000 7f0b01046640 1 -- 172.21.15.89:0/3935873407 shutdown_connections 2023-10-30T12:29:03.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:03.944+0000 7f0b01046640 1 -- 172.21.15.89:0/3935873407 wait complete. 2023-10-30T12:29:03.969 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2023-10-30T12:29:02.923152+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594184,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":243,"num_write_kb":2442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":190,"ondisk_log_size":190,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219148,"kb_used_data":3020,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755572,"statfs":{"total":773094113280,"available":772869705728,"internally_reserved":0,"allocated":3092480,"data_stored":1817423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12741,"internal_metadata":221236795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3816,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.993756"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028790+0000","last_change":"2023-10-30T12:28:50.979797+0000","last_active":"2023-10-30T12:28:53.028790+0000","last_peered":"2023-10-30T12:28:53.028790+0000","last_clean":"2023-10-30T12:28:53.028790+0000","last_became_active":"2023-10-30T12:28:50.979351+0000","last_became_peered":"2023-10-30T12:28:50.979351+0000","last_unstale":"2023-10-30T12:28:53.028790+0000","last_undegraded":"2023-10-30T12:28:53.028790+0000","last_fullsized":"2023-10-30T12:28:53.028790+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:53:00.556539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.932043+0000","last_change":"2023-10-30T12:28:50.929403+0000","last_active":"2023-10-30T12:28:51.932043+0000","last_peered":"2023-10-30T12:28:51.932043+0000","last_clean":"2023-10-30T12:28:51.932043+0000","last_became_active":"2023-10-30T12:28:50.929133+0000","last_became_peered":"2023-10-30T12:28:50.929133+0000","last_unstale":"2023-10-30T12:28:51.932043+0000","last_undegraded":"2023-10-30T12:28:51.932043+0000","last_fullsized":"2023-10-30T12:28:51.932043+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:12:57.327440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579508+0000","last_change":"2023-10-30T12:28:50.934201+0000","last_active":"2023-10-30T12:28:53.579508+0000","last_peered":"2023-10-30T12:28:53.579508+0000","last_clean":"2023-10-30T12:28:53.579508+0000","last_became_active":"2023-10-30T12:28:50.933896+0000","last_became_peered":"2023-10-30T12:28:50.933896+0000","last_unstale":"2023-10-30T12:28:53.579508+0000","last_undegraded":"2023-10-30T12:28:53.579508+0000","last_fullsized":"2023-10-30T12:28:53.579508+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:20:46.597550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581831+0000","last_change":"2023-10-30T12:28:50.969193+0000","last_active":"2023-10-30T12:28:53.581831+0000","last_peered":"2023-10-30T12:28:53.581831+0000","last_clean":"2023-10-30T12:28:53.581831+0000","last_became_active":"2023-10-30T12:28:50.967774+0000","last_became_peered":"2023-10-30T12:28:50.967774+0000","last_unstale":"2023-10-30T12:28:53.581831+0000","last_undegraded":"2023-10-30T12:28:53.581831+0000","last_fullsized":"2023-10-30T12:28:53.581831+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:23:07.329372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028482+0000","last_change":"2023-10-30T12:28:50.973751+0000","last_active":"2023-10-30T12:28:53.028482+0000","last_peered":"2023-10-30T12:28:53.028482+0000","last_clean":"2023-10-30T12:28:53.028482+0000","last_became_active":"2023-10-30T12:28:50.973132+0000","last_became_peered":"2023-10-30T12:28:50.973132+0000","last_unstale":"2023-10-30T12:28:53.028482+0000","last_undegraded":"2023-10-30T12:28:53.028482+0000","last_fullsized":"2023-10-30T12:28:53.028482+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:43:58.223555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581257+0000","last_change":"2023-10-30T12:28:50.968823+0000","last_active":"2023-10-30T12:28:53.581257+0000","last_peered":"2023-10-30T12:28:53.581257+0000","last_clean":"2023-10-30T12:28:53.581257+0000","last_became_active":"2023-10-30T12:28:50.967910+0000","last_became_peered":"2023-10-30T12:28:50.967910+0000","last_unstale":"2023-10-30T12:28:53.581257+0000","last_undegraded":"2023-10-30T12:28:53.581257+0000","last_fullsized":"2023-10-30T12:28:53.581257+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:34:36.882757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028044+0000","last_change":"2023-10-30T12:28:50.974164+0000","last_active":"2023-10-30T12:28:53.028044+0000","last_peered":"2023-10-30T12:28:53.028044+0000","last_clean":"2023-10-30T12:28:53.028044+0000","last_became_active":"2023-10-30T12:28:50.973288+0000","last_became_peered":"2023-10-30T12:28:50.973288+0000","last_unstale":"2023-10-30T12:28:53.028044+0000","last_undegraded":"2023-10-30T12:28:53.028044+0000","last_fullsized":"2023-10-30T12:28:53.028044+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:09:10.450492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.578880+0000","last_change":"2023-10-30T12:28:50.955287+0000","last_active":"2023-10-30T12:28:53.578880+0000","last_peered":"2023-10-30T12:28:53.578880+0000","last_clean":"2023-10-30T12:28:53.578880+0000","last_became_active":"2023-10-30T12:28:50.954988+0000","last_became_peered":"2023-10-30T12:28:50.954988+0000","last_unstale":"2023-10-30T12:28:53.578880+0000","last_undegraded":"2023-10-30T12:28:53.578880+0000","last_fullsized":"2023-10-30T12:28:53.578880+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:36:30.846920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.029019+0000","last_change":"2023-10-30T12:28:50.977144+0000","last_active":"2023-10-30T12:28:53.029019+0000","last_peered":"2023-10-30T12:28:53.029019+0000","last_clean":"2023-10-30T12:28:53.029019+0000","last_became_active":"2023-10-30T12:28:50.976564+0000","last_became_peered":"2023-10-30T12:28:50.976564+0000","last_unstale":"2023-10-30T12:28:53.029019+0000","last_undegraded":"2023-10-30T12:28:53.029019+0000","last_fullsized":"2023-10-30T12:28:53.029019+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:37:51.341935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.938542+0000","last_change":"2023-10-30T12:28:50.952564+0000","last_active":"2023-10-30T12:28:51.938542+0000","last_peered":"2023-10-30T12:28:51.938542+0000","last_clean":"2023-10-30T12:28:51.938542+0000","last_became_active":"2023-10-30T12:28:50.951809+0000","last_became_peered":"2023-10-30T12:28:50.951809+0000","last_unstale":"2023-10-30T12:28:51.938542+0000","last_undegraded":"2023-10-30T12:28:51.938542+0000","last_fullsized":"2023-10-30T12:28:51.938542+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:34:29.771577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.937215+0000","last_change":"2023-10-30T12:28:50.970779+0000","last_active":"2023-10-30T12:28:51.937215+0000","last_peered":"2023-10-30T12:28:51.937215+0000","last_clean":"2023-10-30T12:28:51.937215+0000","last_became_active":"2023-10-30T12:28:50.969877+0000","last_became_peered":"2023-10-30T12:28:50.969877+0000","last_unstale":"2023-10-30T12:28:51.937215+0000","last_undegraded":"2023-10-30T12:28:51.937215+0000","last_fullsized":"2023-10-30T12:28:51.937215+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:10.539915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.940164+0000","last_change":"2023-10-30T12:28:50.944452+0000","last_active":"2023-10-30T12:28:51.940164+0000","last_peered":"2023-10-30T12:28:51.940164+0000","last_clean":"2023-10-30T12:28:51.940164+0000","last_became_active":"2023-10-30T12:28:50.943787+0000","last_became_peered":"2023-10-30T12:28:50.943787+0000","last_unstale":"2023-10-30T12:28:51.940164+0000","last_undegraded":"2023-10-30T12:28:51.940164+0000","last_fullsized":"2023-10-30T12:28:51.940164+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:30:34.585298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579240+0000","last_change":"2023-10-30T12:28:50.938181+0000","last_active":"2023-10-30T12:28:53.579240+0000","last_peered":"2023-10-30T12:28:53.579240+0000","last_clean":"2023-10-30T12:28:53.579240+0000","last_became_active":"2023-10-30T12:28:50.937906+0000","last_became_peered":"2023-10-30T12:28:50.937906+0000","last_unstale":"2023-10-30T12:28:53.579240+0000","last_undegraded":"2023-10-30T12:28:53.579240+0000","last_fullsized":"2023-10-30T12:28:53.579240+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:41:19.990452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579303+0000","last_change":"2023-10-30T12:28:50.934085+0000","last_active":"2023-10-30T12:28:53.579303+0000","last_peered":"2023-10-30T12:28:53.579303+0000","last_clean":"2023-10-30T12:28:53.579303+0000","last_became_active":"2023-10-30T12:28:50.932753+0000","last_became_peered":"2023-10-30T12:28:50.932753+0000","last_unstale":"2023-10-30T12:28:53.579303+0000","last_undegraded":"2023-10-30T12:28:53.579303+0000","last_fullsized":"2023-10-30T12:28:53.579303+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:23:26.736283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931710+0000","last_change":"2023-10-30T12:28:50.943343+0000","last_active":"2023-10-30T12:28:51.931710+0000","last_peered":"2023-10-30T12:28:51.931710+0000","last_clean":"2023-10-30T12:28:51.931710+0000","last_became_active":"2023-10-30T12:28:50.943011+0000","last_became_peered":"2023-10-30T12:28:50.943011+0000","last_unstale":"2023-10-30T12:28:51.931710+0000","last_undegraded":"2023-10-30T12:28:51.931710+0000","last_fullsized":"2023-10-30T12:28:51.931710+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:59:35.881115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581157+0000","last_change":"2023-10-30T12:28:50.953573+0000","last_active":"2023-10-30T12:28:53.581157+0000","last_peered":"2023-10-30T12:28:53.581157+0000","last_clean":"2023-10-30T12:28:53.581157+0000","last_became_active":"2023-10-30T12:28:50.952744+0000","last_became_peered":"2023-10-30T12:28:50.952744+0000","last_unstale":"2023-10-30T12:28:53.581157+0000","last_undegraded":"2023-10-30T12:28:53.581157+0000","last_fullsized":"2023-10-30T12:28:53.581157+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:41:16.209759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.582342+0000","last_change":"2023-10-30T12:28:50.975156+0000","last_active":"2023-10-30T12:28:53.582342+0000","last_peered":"2023-10-30T12:28:53.582342+0000","last_clean":"2023-10-30T12:28:53.582342+0000","last_became_active":"2023-10-30T12:28:50.974576+0000","last_became_peered":"2023-10-30T12:28:50.974576+0000","last_unstale":"2023-10-30T12:28:53.582342+0000","last_undegraded":"2023-10-30T12:28:53.582342+0000","last_fullsized":"2023-10-30T12:28:53.582342+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:46.159555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.131817+0000","last_change":"2023-10-30T12:28:50.937685+0000","last_active":"2023-10-30T12:28:53.131817+0000","last_peered":"2023-10-30T12:28:53.131817+0000","last_clean":"2023-10-30T12:28:53.131817+0000","last_became_active":"2023-10-30T12:28:50.937384+0000","last_became_peered":"2023-10-30T12:28:50.937384+0000","last_unstale":"2023-10-30T12:28:53.131817+0000","last_undegraded":"2023-10-30T12:28:53.131817+0000","last_fullsized":"2023-10-30T12:28:53.131817+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:16:29.181523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.029181+0000","last_change":"2023-10-30T12:28:50.974918+0000","last_active":"2023-10-30T12:28:53.029181+0000","last_peered":"2023-10-30T12:28:53.029181+0000","last_clean":"2023-10-30T12:28:53.029181+0000","last_became_active":"2023-10-30T12:28:50.973369+0000","last_became_peered":"2023-10-30T12:28:50.973369+0000","last_unstale":"2023-10-30T12:28:53.029181+0000","last_undegraded":"2023-10-30T12:28:53.029181+0000","last_fullsized":"2023-10-30T12:28:53.029181+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:44:04.468611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.937226+0000","last_change":"2023-10-30T12:28:50.955835+0000","last_active":"2023-10-30T12:28:51.937226+0000","last_peered":"2023-10-30T12:28:51.937226+0000","last_clean":"2023-10-30T12:28:51.937226+0000","last_became_active":"2023-10-30T12:28:50.954831+0000","last_became_peered":"2023-10-30T12:28:50.954831+0000","last_unstale":"2023-10-30T12:28:51.937226+0000","last_undegraded":"2023-10-30T12:28:51.937226+0000","last_fullsized":"2023-10-30T12:28:51.937226+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:56:25.554582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.580290+0000","last_change":"2023-10-30T12:28:50.962417+0000","last_active":"2023-10-30T12:28:53.580290+0000","last_peered":"2023-10-30T12:28:53.580290+0000","last_clean":"2023-10-30T12:28:53.580290+0000","last_became_active":"2023-10-30T12:28:50.962065+0000","last_became_peered":"2023-10-30T12:28:50.962065+0000","last_unstale":"2023-10-30T12:28:53.580290+0000","last_undegraded":"2023-10-30T12:28:53.580290+0000","last_fullsized":"2023-10-30T12:28:53.580290+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:13:52.455824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581454+0000","last_change":"2023-10-30T12:28:50.968992+0000","last_active":"2023-10-30T12:28:53.581454+0000","last_peered":"2023-10-30T12:28:53.581454+0000","last_clean":"2023-10-30T12:28:53.581454+0000","last_became_active":"2023-10-30T12:28:50.967651+0000","last_became_peered":"2023-10-30T12:28:50.967651+0000","last_unstale":"2023-10-30T12:28:53.581454+0000","last_undegraded":"2023-10-30T12:28:53.581454+0000","last_fullsized":"2023-10-30T12:28:53.581454+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:39:31.893024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579014+0000","last_change":"2023-10-30T12:28:50.962609+0000","last_active":"2023-10-30T12:28:53.579014+0000","last_peered":"2023-10-30T12:28:53.579014+0000","last_clean":"2023-10-30T12:28:53.579014+0000","last_became_active":"2023-10-30T12:28:50.962290+0000","last_became_peered":"2023-10-30T12:28:50.962290+0000","last_unstale":"2023-10-30T12:28:53.579014+0000","last_undegraded":"2023-10-30T12:28:53.579014+0000","last_fullsized":"2023-10-30T12:28:53.579014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:44:35.484971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581381+0000","last_change":"2023-10-30T12:28:50.966653+0000","last_active":"2023-10-30T12:28:53.581381+0000","last_peered":"2023-10-30T12:28:53.581381+0000","last_clean":"2023-10-30T12:28:53.581381+0000","last_became_active":"2023-10-30T12:28:50.965713+0000","last_became_peered":"2023-10-30T12:28:50.965713+0000","last_unstale":"2023-10-30T12:28:53.581381+0000","last_undegraded":"2023-10-30T12:28:53.581381+0000","last_fullsized":"2023-10-30T12:28:53.581381+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:10:31.600981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579351+0000","last_change":"2023-10-30T12:28:50.951300+0000","last_active":"2023-10-30T12:28:53.579351+0000","last_peered":"2023-10-30T12:28:53.579351+0000","last_clean":"2023-10-30T12:28:53.579351+0000","last_became_active":"2023-10-30T12:28:50.951026+0000","last_became_peered":"2023-10-30T12:28:50.951026+0000","last_unstale":"2023-10-30T12:28:53.579351+0000","last_undegraded":"2023-10-30T12:28:53.579351+0000","last_fullsized":"2023-10-30T12:28:53.579351+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:44:58.872457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.938879+0000","last_change":"2023-10-30T12:28:50.963507+0000","last_active":"2023-10-30T12:28:51.938879+0000","last_peered":"2023-10-30T12:28:51.938879+0000","last_clean":"2023-10-30T12:28:51.938879+0000","last_became_active":"2023-10-30T12:28:50.962484+0000","last_became_peered":"2023-10-30T12:28:50.962484+0000","last_unstale":"2023-10-30T12:28:51.938879+0000","last_undegraded":"2023-10-30T12:28:51.938879+0000","last_fullsized":"2023-10-30T12:28:51.938879+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:11:24.160220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931933+0000","last_change":"2023-10-30T12:28:50.933362+0000","last_active":"2023-10-30T12:28:51.931933+0000","last_peered":"2023-10-30T12:28:51.931933+0000","last_clean":"2023-10-30T12:28:51.931933+0000","last_became_active":"2023-10-30T12:28:50.933093+0000","last_became_peered":"2023-10-30T12:28:50.933093+0000","last_unstale":"2023-10-30T12:28:51.931933+0000","last_undegraded":"2023-10-30T12:28:51.931933+0000","last_fullsized":"2023-10-30T12:28:51.931933+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:54:00.852482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579508+0000","last_change":"2023-10-30T12:28:50.952857+0000","last_active":"2023-10-30T12:28:53.579508+0000","last_peered":"2023-10-30T12:28:53.579508+0000","last_clean":"2023-10-30T12:28:53.579508+0000","last_became_active":"2023-10-30T12:28:50.952531+0000","last_became_peered":"2023-10-30T12:28:50.952531+0000","last_unstale":"2023-10-30T12:28:53.579508+0000","last_undegraded":"2023-10-30T12:28:53.579508+0000","last_fullsized":"2023-10-30T12:28:53.579508+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:01:03.980850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.580366+0000","last_change":"2023-10-30T12:28:50.933408+0000","last_active":"2023-10-30T12:28:53.580366+0000","last_peered":"2023-10-30T12:28:53.580366+0000","last_clean":"2023-10-30T12:28:53.580366+0000","last_became_active":"2023-10-30T12:28:50.933124+0000","last_became_peered":"2023-10-30T12:28:50.933124+0000","last_unstale":"2023-10-30T12:28:53.580366+0000","last_undegraded":"2023-10-30T12:28:53.580366+0000","last_fullsized":"2023-10-30T12:28:53.580366+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:53:50.772420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581630+0000","last_change":"2023-10-30T12:28:50.966388+0000","last_active":"2023-10-30T12:28:53.581630+0000","last_peered":"2023-10-30T12:28:53.581630+0000","last_clean":"2023-10-30T12:28:53.581630+0000","last_became_active":"2023-10-30T12:28:50.965639+0000","last_became_peered":"2023-10-30T12:28:50.965639+0000","last_unstale":"2023-10-30T12:28:53.581630+0000","last_undegraded":"2023-10-30T12:28:53.581630+0000","last_fullsized":"2023-10-30T12:28:53.581630+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:55:46.506378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.131583+0000","last_change":"2023-10-30T12:28:50.955391+0000","last_active":"2023-10-30T12:28:53.131583+0000","last_peered":"2023-10-30T12:28:53.131583+0000","last_clean":"2023-10-30T12:28:53.131583+0000","last_became_active":"2023-10-30T12:28:50.955084+0000","last_became_peered":"2023-10-30T12:28:50.955084+0000","last_unstale":"2023-10-30T12:28:53.131583+0000","last_undegraded":"2023-10-30T12:28:53.131583+0000","last_fullsized":"2023-10-30T12:28:53.131583+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:08:52.389085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.936598+0000","last_change":"2023-10-30T12:28:50.945609+0000","last_active":"2023-10-30T12:28:51.936598+0000","last_peered":"2023-10-30T12:28:51.936598+0000","last_clean":"2023-10-30T12:28:51.936598+0000","last_became_active":"2023-10-30T12:28:50.944775+0000","last_became_peered":"2023-10-30T12:28:50.944775+0000","last_unstale":"2023-10-30T12:28:51.936598+0000","last_undegraded":"2023-10-30T12:28:51.936598+0000","last_fullsized":"2023-10-30T12:28:51.936598+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:08:55.718193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.939146+0000","last_change":"2023-10-30T12:28:50.941378+0000","last_active":"2023-10-30T12:28:51.939146+0000","last_peered":"2023-10-30T12:28:51.939146+0000","last_clean":"2023-10-30T12:28:51.939146+0000","last_became_active":"2023-10-30T12:28:50.940567+0000","last_became_peered":"2023-10-30T12:28:50.940567+0000","last_unstale":"2023-10-30T12:28:51.939146+0000","last_undegraded":"2023-10-30T12:28:51.939146+0000","last_fullsized":"2023-10-30T12:28:51.939146+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:31:23.772343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'5","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.435783+0000","last_change":"2023-10-30T12:28:50.395670+0000","last_active":"2023-10-30T12:29:01.435783+0000","last_peered":"2023-10-30T12:29:01.435783+0000","last_clean":"2023-10-30T12:29:01.435783+0000","last_became_active":"2023-10-30T12:28:50.395382+0000","last_became_peered":"2023-10-30T12:28:50.395382+0000","last_unstale":"2023-10-30T12:29:01.435783+0000","last_undegraded":"2023-10-30T12:29:01.435783+0000","last_fullsized":"2023-10-30T12:29:01.435783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:14.858121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.580427+0000","last_change":"2023-10-30T12:28:50.953202+0000","last_active":"2023-10-30T12:28:53.580427+0000","last_peered":"2023-10-30T12:28:53.580427+0000","last_clean":"2023-10-30T12:28:53.580427+0000","last_became_active":"2023-10-30T12:28:50.952613+0000","last_became_peered":"2023-10-30T12:28:50.952613+0000","last_unstale":"2023-10-30T12:28:53.580427+0000","last_undegraded":"2023-10-30T12:28:53.580427+0000","last_fullsized":"2023-10-30T12:28:53.580427+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:38:56.947997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:52.304063+0000","last_change":"2023-10-30T12:28:50.384598+0000","last_active":"2023-10-30T12:28:52.304063+0000","last_peered":"2023-10-30T12:28:52.304063+0000","last_clean":"2023-10-30T12:28:52.304063+0000","last_became_active":"2023-10-30T12:28:50.383898+0000","last_became_peered":"2023-10-30T12:28:50.383898+0000","last_unstale":"2023-10-30T12:28:52.304063+0000","last_undegraded":"2023-10-30T12:28:52.304063+0000","last_fullsized":"2023-10-30T12:28:52.304063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:09:50.290004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:52.347534+0000","last_change":"2023-10-30T12:28:50.386193+0000","last_active":"2023-10-30T12:28:52.347534+0000","last_peered":"2023-10-30T12:28:52.347534+0000","last_clean":"2023-10-30T12:28:52.347534+0000","last_became_active":"2023-10-30T12:28:50.385963+0000","last_became_peered":"2023-10-30T12:28:50.385963+0000","last_unstale":"2023-10-30T12:28:52.347534+0000","last_undegraded":"2023-10-30T12:28:52.347534+0000","last_fullsized":"2023-10-30T12:28:52.347534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:00:55.603212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.578966+0000","last_change":"2023-10-30T12:28:50.936038+0000","last_active":"2023-10-30T12:28:53.578966+0000","last_peered":"2023-10-30T12:28:53.578966+0000","last_clean":"2023-10-30T12:28:53.578966+0000","last_became_active":"2023-10-30T12:28:50.935764+0000","last_became_peered":"2023-10-30T12:28:50.935764+0000","last_unstale":"2023-10-30T12:28:53.578966+0000","last_undegraded":"2023-10-30T12:28:53.578966+0000","last_fullsized":"2023-10-30T12:28:53.578966+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:23:01.597158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581945+0000","last_change":"2023-10-30T12:28:50.392213+0000","last_active":"2023-10-30T12:28:53.581945+0000","last_peered":"2023-10-30T12:28:53.581945+0000","last_clean":"2023-10-30T12:28:53.581945+0000","last_became_active":"2023-10-30T12:28:50.391494+0000","last_became_peered":"2023-10-30T12:28:50.391494+0000","last_unstale":"2023-10-30T12:28:53.581945+0000","last_undegraded":"2023-10-30T12:28:53.581945+0000","last_fullsized":"2023-10-30T12:28:53.581945+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:38:37.301840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931619+0000","last_change":"2023-10-30T12:28:50.933319+0000","last_active":"2023-10-30T12:28:51.931619+0000","last_peered":"2023-10-30T12:28:51.931619+0000","last_clean":"2023-10-30T12:28:51.931619+0000","last_became_active":"2023-10-30T12:28:50.933062+0000","last_became_peered":"2023-10-30T12:28:50.933062+0000","last_unstale":"2023-10-30T12:28:51.931619+0000","last_undegraded":"2023-10-30T12:28:51.931619+0000","last_fullsized":"2023-10-30T12:28:51.931619+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:26:48.475038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.424753+0000","last_change":"2023-10-30T12:28:50.394368+0000","last_active":"2023-10-30T12:29:01.424753+0000","last_peered":"2023-10-30T12:29:01.424753+0000","last_clean":"2023-10-30T12:29:01.424753+0000","last_became_active":"2023-10-30T12:28:50.393686+0000","last_became_peered":"2023-10-30T12:28:50.393686+0000","last_unstale":"2023-10-30T12:29:01.424753+0000","last_undegraded":"2023-10-30T12:29:01.424753+0000","last_fullsized":"2023-10-30T12:29:01.424753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:43:17.726462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.582247+0000","last_change":"2023-10-30T12:28:50.963703+0000","last_active":"2023-10-30T12:28:53.582247+0000","last_peered":"2023-10-30T12:28:53.582247+0000","last_clean":"2023-10-30T12:28:53.582247+0000","last_became_active":"2023-10-30T12:28:50.963105+0000","last_became_peered":"2023-10-30T12:28:50.963105+0000","last_unstale":"2023-10-30T12:28:53.582247+0000","last_undegraded":"2023-10-30T12:28:53.582247+0000","last_fullsized":"2023-10-30T12:28:53.582247+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:32.423943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.420755+0000","last_change":"2023-10-30T12:28:50.397679+0000","last_active":"2023-10-30T12:29:01.420755+0000","last_peered":"2023-10-30T12:29:01.420755+0000","last_clean":"2023-10-30T12:29:01.420755+0000","last_became_active":"2023-10-30T12:28:50.397041+0000","last_became_peered":"2023-10-30T12:28:50.397041+0000","last_unstale":"2023-10-30T12:29:01.420755+0000","last_undegraded":"2023-10-30T12:29:01.420755+0000","last_fullsized":"2023-10-30T12:29:01.420755+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:52:22.564616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.580443+0000","last_change":"2023-10-30T12:28:50.966838+0000","last_active":"2023-10-30T12:28:53.580443+0000","last_peered":"2023-10-30T12:28:53.580443+0000","last_clean":"2023-10-30T12:28:53.580443+0000","last_became_active":"2023-10-30T12:28:50.966569+0000","last_became_peered":"2023-10-30T12:28:50.966569+0000","last_unstale":"2023-10-30T12:28:53.580443+0000","last_undegraded":"2023-10-30T12:28:53.580443+0000","last_fullsized":"2023-10-30T12:28:53.580443+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:44:33.875449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:56.364393+0000","last_change":"2023-10-30T12:28:50.391527+0000","last_active":"2023-10-30T12:28:56.364393+0000","last_peered":"2023-10-30T12:28:56.364393+0000","last_clean":"2023-10-30T12:28:56.364393+0000","last_became_active":"2023-10-30T12:28:50.391260+0000","last_became_peered":"2023-10-30T12:28:50.391260+0000","last_unstale":"2023-10-30T12:28:56.364393+0000","last_undegraded":"2023-10-30T12:28:56.364393+0000","last_fullsized":"2023-10-30T12:28:56.364393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:29:34.772490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.940211+0000","last_change":"2023-10-30T12:28:50.975441+0000","last_active":"2023-10-30T12:28:51.940211+0000","last_peered":"2023-10-30T12:28:51.940211+0000","last_clean":"2023-10-30T12:28:51.940211+0000","last_became_active":"2023-10-30T12:28:50.974843+0000","last_became_peered":"2023-10-30T12:28:50.974843+0000","last_unstale":"2023-10-30T12:28:51.940211+0000","last_undegraded":"2023-10-30T12:28:51.940211+0000","last_fullsized":"2023-10-30T12:28:51.940211+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:06:12.060752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"22'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.427140+0000","last_change":"2023-10-30T12:28:50.374838+0000","last_active":"2023-10-30T12:29:01.427140+0000","last_peered":"2023-10-30T12:29:01.427140+0000","last_clean":"2023-10-30T12:29:01.427140+0000","last_became_active":"2023-10-30T12:28:50.373581+0000","last_became_peered":"2023-10-30T12:28:50.373581+0000","last_unstale":"2023-10-30T12:29:01.427140+0000","last_undegraded":"2023-10-30T12:29:01.427140+0000","last_fullsized":"2023-10-30T12:29:01.427140+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:14:51.506745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.029046+0000","last_change":"2023-10-30T12:28:50.974012+0000","last_active":"2023-10-30T12:28:53.029046+0000","last_peered":"2023-10-30T12:28:53.029046+0000","last_clean":"2023-10-30T12:28:53.029046+0000","last_became_active":"2023-10-30T12:28:50.973269+0000","last_became_peered":"2023-10-30T12:28:50.973269+0000","last_unstale":"2023-10-30T12:28:53.029046+0000","last_undegraded":"2023-10-30T12:28:53.029046+0000","last_fullsized":"2023-10-30T12:28:53.029046+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:19:21.993898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.427284+0000","last_change":"2023-10-30T12:28:50.366590+0000","last_active":"2023-10-30T12:29:01.427284+0000","last_peered":"2023-10-30T12:29:01.427284+0000","last_clean":"2023-10-30T12:29:01.427284+0000","last_became_active":"2023-10-30T12:28:50.366269+0000","last_became_peered":"2023-10-30T12:28:50.366269+0000","last_unstale":"2023-10-30T12:29:01.427284+0000","last_undegraded":"2023-10-30T12:29:01.427284+0000","last_fullsized":"2023-10-30T12:29:01.427284+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:15:11.963438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579425+0000","last_change":"2023-10-30T12:28:50.962058+0000","last_active":"2023-10-30T12:28:53.579425+0000","last_peered":"2023-10-30T12:28:53.579425+0000","last_clean":"2023-10-30T12:28:53.579425+0000","last_became_active":"2023-10-30T12:28:50.961805+0000","last_became_peered":"2023-10-30T12:28:50.961805+0000","last_unstale":"2023-10-30T12:28:53.579425+0000","last_undegraded":"2023-10-30T12:28:53.579425+0000","last_fullsized":"2023-10-30T12:28:53.579425+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:45:16.373386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.427282+0000","last_change":"2023-10-30T12:28:50.397244+0000","last_active":"2023-10-30T12:29:01.427282+0000","last_peered":"2023-10-30T12:29:01.427282+0000","last_clean":"2023-10-30T12:29:01.427282+0000","last_became_active":"2023-10-30T12:28:50.396727+0000","last_became_peered":"2023-10-30T12:28:50.396727+0000","last_unstale":"2023-10-30T12:29:01.427282+0000","last_undegraded":"2023-10-30T12:29:01.427282+0000","last_fullsized":"2023-10-30T12:29:01.427282+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:33.493343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028618+0000","last_change":"2023-10-30T12:28:50.981498+0000","last_active":"2023-10-30T12:28:53.028618+0000","last_peered":"2023-10-30T12:28:53.028618+0000","last_clean":"2023-10-30T12:28:53.028618+0000","last_became_active":"2023-10-30T12:28:50.981063+0000","last_became_peered":"2023-10-30T12:28:50.981063+0000","last_unstale":"2023-10-30T12:28:53.028618+0000","last_undegraded":"2023-10-30T12:28:53.028618+0000","last_fullsized":"2023-10-30T12:28:53.028618+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:55:17.971880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.937078+0000","last_change":"2023-10-30T12:28:50.383880+0000","last_active":"2023-10-30T12:28:51.937078+0000","last_peered":"2023-10-30T12:28:51.937078+0000","last_clean":"2023-10-30T12:28:51.937078+0000","last_became_active":"2023-10-30T12:28:50.383570+0000","last_became_peered":"2023-10-30T12:28:50.383570+0000","last_unstale":"2023-10-30T12:28:51.937078+0000","last_undegraded":"2023-10-30T12:28:51.937078+0000","last_fullsized":"2023-10-30T12:28:51.937078+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:19:42.047520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.936983+0000","last_change":"2023-10-30T12:28:50.950312+0000","last_active":"2023-10-30T12:28:51.936983+0000","last_peered":"2023-10-30T12:28:51.936983+0000","last_clean":"2023-10-30T12:28:51.936983+0000","last_became_active":"2023-10-30T12:28:50.949485+0000","last_became_peered":"2023-10-30T12:28:50.949485+0000","last_unstale":"2023-10-30T12:28:51.936983+0000","last_undegraded":"2023-10-30T12:28:51.936983+0000","last_fullsized":"2023-10-30T12:28:51.936983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:13:04.669915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"22'10","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.434540+0000","last_change":"2023-10-30T12:28:50.362979+0000","last_active":"2023-10-30T12:29:01.434540+0000","last_peered":"2023-10-30T12:29:01.434540+0000","last_clean":"2023-10-30T12:29:01.434540+0000","last_became_active":"2023-10-30T12:28:50.362624+0000","last_became_peered":"2023-10-30T12:28:50.362624+0000","last_unstale":"2023-10-30T12:29:01.434540+0000","last_undegraded":"2023-10-30T12:29:01.434540+0000","last_fullsized":"2023-10-30T12:29:01.434540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:01:13.111816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028233+0000","last_change":"2023-10-30T12:28:50.979815+0000","last_active":"2023-10-30T12:28:53.028233+0000","last_peered":"2023-10-30T12:28:53.028233+0000","last_clean":"2023-10-30T12:28:53.028233+0000","last_became_active":"2023-10-30T12:28:50.979344+0000","last_became_peered":"2023-10-30T12:28:50.979344+0000","last_unstale":"2023-10-30T12:28:53.028233+0000","last_undegraded":"2023-10-30T12:28:53.028233+0000","last_fullsized":"2023-10-30T12:28:53.028233+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:35:37.018282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:52.284562+0000","last_change":"2023-10-30T12:28:50.382175+0000","last_active":"2023-10-30T12:28:52.284562+0000","last_peered":"2023-10-30T12:28:52.284562+0000","last_clean":"2023-10-30T12:28:52.284562+0000","last_became_active":"2023-10-30T12:28:50.381125+0000","last_became_peered":"2023-10-30T12:28:50.381125+0000","last_unstale":"2023-10-30T12:28:52.284562+0000","last_undegraded":"2023-10-30T12:28:52.284562+0000","last_fullsized":"2023-10-30T12:28:52.284562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:36:02.652665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028807+0000","last_change":"2023-10-30T12:28:50.978034+0000","last_active":"2023-10-30T12:28:53.028807+0000","last_peered":"2023-10-30T12:28:53.028807+0000","last_clean":"2023-10-30T12:28:53.028807+0000","last_became_active":"2023-10-30T12:28:50.977585+0000","last_became_peered":"2023-10-30T12:28:50.977585+0000","last_unstale":"2023-10-30T12:28:53.028807+0000","last_undegraded":"2023-10-30T12:28:53.028807+0000","last_fullsized":"2023-10-30T12:28:53.028807+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:42:56.806184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:56.364009+0000","last_change":"2023-10-30T12:28:50.367258+0000","last_active":"2023-10-30T12:28:56.364009+0000","last_peered":"2023-10-30T12:28:56.364009+0000","last_clean":"2023-10-30T12:28:56.364009+0000","last_became_active":"2023-10-30T12:28:50.366974+0000","last_became_peered":"2023-10-30T12:28:50.366974+0000","last_unstale":"2023-10-30T12:28:56.364009+0000","last_undegraded":"2023-10-30T12:28:56.364009+0000","last_fullsized":"2023-10-30T12:28:56.364009+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:14:48.823902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.029189+0000","last_change":"2023-10-30T12:28:50.981784+0000","last_active":"2023-10-30T12:28:53.029189+0000","last_peered":"2023-10-30T12:28:53.029189+0000","last_clean":"2023-10-30T12:28:53.029189+0000","last_became_active":"2023-10-30T12:28:50.981193+0000","last_became_peered":"2023-10-30T12:28:50.981193+0000","last_unstale":"2023-10-30T12:28:53.029189+0000","last_undegraded":"2023-10-30T12:28:53.029189+0000","last_fullsized":"2023-10-30T12:28:53.029189+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:36:17.384732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:52.280239+0000","last_change":"2023-10-30T12:28:50.368011+0000","last_active":"2023-10-30T12:28:52.280239+0000","last_peered":"2023-10-30T12:28:52.280239+0000","last_clean":"2023-10-30T12:28:52.280239+0000","last_became_active":"2023-10-30T12:28:50.367730+0000","last_became_peered":"2023-10-30T12:28:50.367730+0000","last_unstale":"2023-10-30T12:28:52.280239+0000","last_undegraded":"2023-10-30T12:28:52.280239+0000","last_fullsized":"2023-10-30T12:28:52.280239+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:59:41.946550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.582218+0000","last_change":"2023-10-30T12:28:50.968262+0000","last_active":"2023-10-30T12:28:53.582218+0000","last_peered":"2023-10-30T12:28:53.582218+0000","last_clean":"2023-10-30T12:28:53.582218+0000","last_became_active":"2023-10-30T12:28:50.966246+0000","last_became_peered":"2023-10-30T12:28:50.966246+0000","last_unstale":"2023-10-30T12:28:53.582218+0000","last_undegraded":"2023-10-30T12:28:53.582218+0000","last_fullsized":"2023-10-30T12:28:53.582218+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:00:01.991935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:52.277996+0000","last_change":"2023-10-30T12:28:50.394060+0000","last_active":"2023-10-30T12:28:52.277996+0000","last_peered":"2023-10-30T12:28:52.277996+0000","last_clean":"2023-10-30T12:28:52.277996+0000","last_became_active":"2023-10-30T12:28:50.393642+0000","last_became_peered":"2023-10-30T12:28:50.393642+0000","last_unstale":"2023-10-30T12:28:52.277996+0000","last_undegraded":"2023-10-30T12:28:52.277996+0000","last_fullsized":"2023-10-30T12:28:52.277996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:21.643882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028738+0000","last_change":"2023-10-30T12:28:50.976814+0000","last_active":"2023-10-30T12:28:53.028738+0000","last_peered":"2023-10-30T12:28:53.028738+0000","last_clean":"2023-10-30T12:28:53.028738+0000","last_became_active":"2023-10-30T12:28:50.976343+0000","last_became_peered":"2023-10-30T12:28:50.976343+0000","last_unstale":"2023-10-30T12:28:53.028738+0000","last_undegraded":"2023-10-30T12:28:53.028738+0000","last_fullsized":"2023-10-30T12:28:53.028738+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:33:23.491998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.946067+0000","last_change":"2023-10-30T12:28:41.203990+0000","last_active":"2023-10-30T12:28:50.946067+0000","last_peered":"2023-10-30T12:28:50.946067+0000","last_clean":"2023-10-30T12:28:50.946067+0000","last_became_active":"2023-10-30T12:28:41.201796+0000","last_became_peered":"2023-10-30T12:28:41.201796+0000","last_unstale":"2023-10-30T12:28:50.946067+0000","last_undegraded":"2023-10-30T12:28:50.946067+0000","last_fullsized":"2023-10-30T12:28:50.946067+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:39.218257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:39.218257+0000","last_clean_scrub_stamp":"2023-10-30T12:28:39.218257+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:21:00.605462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.425894+0000","last_change":"2023-10-30T12:28:50.372651+0000","last_active":"2023-10-30T12:29:01.425894+0000","last_peered":"2023-10-30T12:29:01.425894+0000","last_clean":"2023-10-30T12:29:01.425894+0000","last_became_active":"2023-10-30T12:28:50.367593+0000","last_became_peered":"2023-10-30T12:28:50.367593+0000","last_unstale":"2023-10-30T12:29:01.425894+0000","last_undegraded":"2023-10-30T12:29:01.425894+0000","last_fullsized":"2023-10-30T12:29:01.425894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:32:10.492061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579128+0000","last_change":"2023-10-30T12:28:50.970536+0000","last_active":"2023-10-30T12:28:53.579128+0000","last_peered":"2023-10-30T12:28:53.579128+0000","last_clean":"2023-10-30T12:28:53.579128+0000","last_became_active":"2023-10-30T12:28:50.970326+0000","last_became_peered":"2023-10-30T12:28:50.970326+0000","last_unstale":"2023-10-30T12:28:53.579128+0000","last_undegraded":"2023-10-30T12:28:53.579128+0000","last_fullsized":"2023-10-30T12:28:53.579128+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:53:54.589174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.027894+0000","last_change":"2023-10-30T12:28:50.392461+0000","last_active":"2023-10-30T12:28:53.027894+0000","last_peered":"2023-10-30T12:28:53.027894+0000","last_clean":"2023-10-30T12:28:53.027894+0000","last_became_active":"2023-10-30T12:28:50.391824+0000","last_became_peered":"2023-10-30T12:28:50.391824+0000","last_unstale":"2023-10-30T12:28:53.027894+0000","last_undegraded":"2023-10-30T12:28:53.027894+0000","last_fullsized":"2023-10-30T12:28:53.027894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:56:10.373046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579229+0000","last_change":"2023-10-30T12:28:50.953046+0000","last_active":"2023-10-30T12:28:53.579229+0000","last_peered":"2023-10-30T12:28:53.579229+0000","last_clean":"2023-10-30T12:28:53.579229+0000","last_became_active":"2023-10-30T12:28:50.952625+0000","last_became_peered":"2023-10-30T12:28:50.952625+0000","last_unstale":"2023-10-30T12:28:53.579229+0000","last_undegraded":"2023-10-30T12:28:53.579229+0000","last_fullsized":"2023-10-30T12:28:53.579229+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:00:14.643989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.131708+0000","last_change":"2023-10-30T12:28:50.367547+0000","last_active":"2023-10-30T12:28:53.131708+0000","last_peered":"2023-10-30T12:28:53.131708+0000","last_clean":"2023-10-30T12:28:53.131708+0000","last_became_active":"2023-10-30T12:28:50.367230+0000","last_became_peered":"2023-10-30T12:28:50.367230+0000","last_unstale":"2023-10-30T12:28:53.131708+0000","last_undegraded":"2023-10-30T12:28:53.131708+0000","last_fullsized":"2023-10-30T12:28:53.131708+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:42.570786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931639+0000","last_change":"2023-10-30T12:28:50.928888+0000","last_active":"2023-10-30T12:28:51.931639+0000","last_peered":"2023-10-30T12:28:51.931639+0000","last_clean":"2023-10-30T12:28:51.931639+0000","last_became_active":"2023-10-30T12:28:50.928608+0000","last_became_peered":"2023-10-30T12:28:50.928608+0000","last_unstale":"2023-10-30T12:28:51.931639+0000","last_undegraded":"2023-10-30T12:28:51.931639+0000","last_fullsized":"2023-10-30T12:28:51.931639+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:40:51.042396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:52.285682+0000","last_change":"2023-10-30T12:28:50.391240+0000","last_active":"2023-10-30T12:28:52.285682+0000","last_peered":"2023-10-30T12:28:52.285682+0000","last_clean":"2023-10-30T12:28:52.285682+0000","last_became_active":"2023-10-30T12:28:50.390864+0000","last_became_peered":"2023-10-30T12:28:50.390864+0000","last_unstale":"2023-10-30T12:28:52.285682+0000","last_undegraded":"2023-10-30T12:28:52.285682+0000","last_fullsized":"2023-10-30T12:28:52.285682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:59:40.285830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.939756+0000","last_change":"2023-10-30T12:28:50.949990+0000","last_active":"2023-10-30T12:28:51.939756+0000","last_peered":"2023-10-30T12:28:51.939756+0000","last_clean":"2023-10-30T12:28:51.939756+0000","last_became_active":"2023-10-30T12:28:50.949122+0000","last_became_peered":"2023-10-30T12:28:50.949122+0000","last_unstale":"2023-10-30T12:28:51.939756+0000","last_undegraded":"2023-10-30T12:28:51.939756+0000","last_fullsized":"2023-10-30T12:28:51.939756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:30:56.099810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931803+0000","last_change":"2023-10-30T12:28:50.358700+0000","last_active":"2023-10-30T12:28:51.931803+0000","last_peered":"2023-10-30T12:28:51.931803+0000","last_clean":"2023-10-30T12:28:51.931803+0000","last_became_active":"2023-10-30T12:28:50.358310+0000","last_became_peered":"2023-10-30T12:28:50.358310+0000","last_unstale":"2023-10-30T12:28:51.931803+0000","last_undegraded":"2023-10-30T12:28:51.931803+0000","last_fullsized":"2023-10-30T12:28:51.931803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:23:30.910622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.582250+0000","last_change":"2023-10-30T12:28:50.967650+0000","last_active":"2023-10-30T12:28:53.582250+0000","last_peered":"2023-10-30T12:28:53.582250+0000","last_clean":"2023-10-30T12:28:53.582250+0000","last_became_active":"2023-10-30T12:28:50.965626+0000","last_became_peered":"2023-10-30T12:28:50.965626+0000","last_unstale":"2023-10-30T12:28:53.582250+0000","last_undegraded":"2023-10-30T12:28:53.582250+0000","last_fullsized":"2023-10-30T12:28:53.582250+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:49.050703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.029231+0000","last_change":"2023-10-30T12:28:50.395645+0000","last_active":"2023-10-30T12:28:53.029231+0000","last_peered":"2023-10-30T12:28:53.029231+0000","last_clean":"2023-10-30T12:28:53.029231+0000","last_became_active":"2023-10-30T12:28:50.395090+0000","last_became_peered":"2023-10-30T12:28:50.395090+0000","last_unstale":"2023-10-30T12:28:53.029231+0000","last_undegraded":"2023-10-30T12:28:53.029231+0000","last_fullsized":"2023-10-30T12:28:53.029231+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:45:44.678383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931844+0000","last_change":"2023-10-30T12:28:50.929586+0000","last_active":"2023-10-30T12:28:51.931844+0000","last_peered":"2023-10-30T12:28:51.931844+0000","last_clean":"2023-10-30T12:28:51.931844+0000","last_became_active":"2023-10-30T12:28:50.929337+0000","last_became_peered":"2023-10-30T12:28:50.929337+0000","last_unstale":"2023-10-30T12:28:51.931844+0000","last_undegraded":"2023-10-30T12:28:51.931844+0000","last_fullsized":"2023-10-30T12:28:51.931844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:11:13.067698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:52.280274+0000","last_change":"2023-10-30T12:28:50.380474+0000","last_active":"2023-10-30T12:28:52.280274+0000","last_peered":"2023-10-30T12:28:52.280274+0000","last_clean":"2023-10-30T12:28:52.280274+0000","last_became_active":"2023-10-30T12:28:50.380247+0000","last_became_peered":"2023-10-30T12:28:50.380247+0000","last_unstale":"2023-10-30T12:28:52.280274+0000","last_undegraded":"2023-10-30T12:28:52.280274+0000","last_fullsized":"2023-10-30T12:28:52.280274+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:52:03.952617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579476+0000","last_change":"2023-10-30T12:28:50.973361+0000","last_active":"2023-10-30T12:28:53.579476+0000","last_peered":"2023-10-30T12:28:53.579476+0000","last_clean":"2023-10-30T12:28:53.579476+0000","last_became_active":"2023-10-30T12:28:50.973086+0000","last_became_peered":"2023-10-30T12:28:50.973086+0000","last_unstale":"2023-10-30T12:28:53.579476+0000","last_undegraded":"2023-10-30T12:28:53.579476+0000","last_fullsized":"2023-10-30T12:28:53.579476+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:49:48.589889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.421918+0000","last_change":"2023-10-30T12:28:50.386277+0000","last_active":"2023-10-30T12:29:01.421918+0000","last_peered":"2023-10-30T12:29:01.421918+0000","last_clean":"2023-10-30T12:29:01.421918+0000","last_became_active":"2023-10-30T12:28:50.385924+0000","last_became_peered":"2023-10-30T12:28:50.385924+0000","last_unstale":"2023-10-30T12:29:01.421918+0000","last_undegraded":"2023-10-30T12:29:01.421918+0000","last_fullsized":"2023-10-30T12:29:01.421918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:35:03.837523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.580322+0000","last_change":"2023-10-30T12:28:50.929670+0000","last_active":"2023-10-30T12:28:53.580322+0000","last_peered":"2023-10-30T12:28:53.580322+0000","last_clean":"2023-10-30T12:28:53.580322+0000","last_became_active":"2023-10-30T12:28:50.929259+0000","last_became_peered":"2023-10-30T12:28:50.929259+0000","last_unstale":"2023-10-30T12:28:53.580322+0000","last_undegraded":"2023-10-30T12:28:53.580322+0000","last_fullsized":"2023-10-30T12:28:53.580322+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:50:16.507660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.420373+0000","last_change":"2023-10-30T12:28:50.386426+0000","last_active":"2023-10-30T12:29:01.420373+0000","last_peered":"2023-10-30T12:29:01.420373+0000","last_clean":"2023-10-30T12:29:01.420373+0000","last_became_active":"2023-10-30T12:28:50.385305+0000","last_became_peered":"2023-10-30T12:28:50.385305+0000","last_unstale":"2023-10-30T12:29:01.420373+0000","last_undegraded":"2023-10-30T12:29:01.420373+0000","last_fullsized":"2023-10-30T12:29:01.420373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:13:05.376581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.938368+0000","last_change":"2023-10-30T12:28:50.941607+0000","last_active":"2023-10-30T12:28:51.938368+0000","last_peered":"2023-10-30T12:28:51.938368+0000","last_clean":"2023-10-30T12:28:51.938368+0000","last_became_active":"2023-10-30T12:28:50.940846+0000","last_became_peered":"2023-10-30T12:28:50.940846+0000","last_unstale":"2023-10-30T12:28:51.938368+0000","last_undegraded":"2023-10-30T12:28:51.938368+0000","last_fullsized":"2023-10-30T12:28:51.938368+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:29.837077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.433290+0000","last_change":"2023-10-30T12:28:50.392333+0000","last_active":"2023-10-30T12:29:01.433290+0000","last_peered":"2023-10-30T12:29:01.433290+0000","last_clean":"2023-10-30T12:29:01.433290+0000","last_became_active":"2023-10-30T12:28:50.392068+0000","last_became_peered":"2023-10-30T12:28:50.392068+0000","last_unstale":"2023-10-30T12:29:01.433290+0000","last_undegraded":"2023-10-30T12:29:01.433290+0000","last_fullsized":"2023-10-30T12:29:01.433290+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:47:24.674925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579044+0000","last_change":"2023-10-30T12:28:50.932167+0000","last_active":"2023-10-30T12:28:53.579044+0000","last_peered":"2023-10-30T12:28:53.579044+0000","last_clean":"2023-10-30T12:28:53.579044+0000","last_became_active":"2023-10-30T12:28:50.931898+0000","last_became_peered":"2023-10-30T12:28:50.931898+0000","last_unstale":"2023-10-30T12:28:53.579044+0000","last_undegraded":"2023-10-30T12:28:53.579044+0000","last_fullsized":"2023-10-30T12:28:53.579044+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:44.429045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931812+0000","last_change":"2023-10-30T12:28:50.365055+0000","last_active":"2023-10-30T12:28:51.931812+0000","last_peered":"2023-10-30T12:28:51.931812+0000","last_clean":"2023-10-30T12:28:51.931812+0000","last_became_active":"2023-10-30T12:28:50.364797+0000","last_became_peered":"2023-10-30T12:28:50.364797+0000","last_unstale":"2023-10-30T12:28:51.931812+0000","last_undegraded":"2023-10-30T12:28:51.931812+0000","last_fullsized":"2023-10-30T12:28:51.931812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:06:23.067895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579266+0000","last_change":"2023-10-30T12:28:50.935908+0000","last_active":"2023-10-30T12:28:53.579266+0000","last_peered":"2023-10-30T12:28:53.579266+0000","last_clean":"2023-10-30T12:28:53.579266+0000","last_became_active":"2023-10-30T12:28:50.935657+0000","last_became_peered":"2023-10-30T12:28:50.935657+0000","last_unstale":"2023-10-30T12:28:53.579266+0000","last_undegraded":"2023-10-30T12:28:53.579266+0000","last_fullsized":"2023-10-30T12:28:53.579266+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:36:13.242429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.416976+0000","last_change":"2023-10-30T12:28:50.393465+0000","last_active":"2023-10-30T12:29:01.416976+0000","last_peered":"2023-10-30T12:29:01.416976+0000","last_clean":"2023-10-30T12:29:01.416976+0000","last_became_active":"2023-10-30T12:28:50.392807+0000","last_became_peered":"2023-10-30T12:28:50.392807+0000","last_unstale":"2023-10-30T12:29:01.416976+0000","last_undegraded":"2023-10-30T12:29:01.416976+0000","last_fullsized":"2023-10-30T12:29:01.416976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:35:37.976956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.938767+0000","last_change":"2023-10-30T12:28:50.954834+0000","last_active":"2023-10-30T12:28:51.938767+0000","last_peered":"2023-10-30T12:28:51.938767+0000","last_clean":"2023-10-30T12:28:51.938767+0000","last_became_active":"2023-10-30T12:28:50.954291+0000","last_became_peered":"2023-10-30T12:28:50.954291+0000","last_unstale":"2023-10-30T12:28:51.938767+0000","last_undegraded":"2023-10-30T12:28:51.938767+0000","last_fullsized":"2023-10-30T12:28:51.938767+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:19:44.345918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581732+0000","last_change":"2023-10-30T12:28:50.963620+0000","last_active":"2023-10-30T12:28:53.581732+0000","last_peered":"2023-10-30T12:28:53.581732+0000","last_clean":"2023-10-30T12:28:53.581732+0000","last_became_active":"2023-10-30T12:28:50.962817+0000","last_became_peered":"2023-10-30T12:28:50.962817+0000","last_unstale":"2023-10-30T12:28:53.581732+0000","last_undegraded":"2023-10-30T12:28:53.581732+0000","last_fullsized":"2023-10-30T12:28:53.581732+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:23:23.276086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:56.367962+0000","last_change":"2023-10-30T12:28:50.402179+0000","last_active":"2023-10-30T12:28:56.367962+0000","last_peered":"2023-10-30T12:28:56.367962+0000","last_clean":"2023-10-30T12:28:56.367962+0000","last_became_active":"2023-10-30T12:28:50.401677+0000","last_became_peered":"2023-10-30T12:28:50.401677+0000","last_unstale":"2023-10-30T12:28:56.367962+0000","last_undegraded":"2023-10-30T12:28:56.367962+0000","last_fullsized":"2023-10-30T12:28:56.367962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:39:44.432214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581874+0000","last_change":"2023-10-30T12:28:50.968805+0000","last_active":"2023-10-30T12:28:53.581874+0000","last_peered":"2023-10-30T12:28:53.581874+0000","last_clean":"2023-10-30T12:28:53.581874+0000","last_became_active":"2023-10-30T12:28:50.967591+0000","last_became_peered":"2023-10-30T12:28:50.967591+0000","last_unstale":"2023-10-30T12:28:53.581874+0000","last_undegraded":"2023-10-30T12:28:53.581874+0000","last_fullsized":"2023-10-30T12:28:53.581874+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:11:41.973858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:56.366897+0000","last_change":"2023-10-30T12:28:50.392617+0000","last_active":"2023-10-30T12:28:56.366897+0000","last_peered":"2023-10-30T12:28:56.366897+0000","last_clean":"2023-10-30T12:28:56.366897+0000","last_became_active":"2023-10-30T12:28:50.392021+0000","last_became_peered":"2023-10-30T12:28:50.392021+0000","last_unstale":"2023-10-30T12:28:56.366897+0000","last_undegraded":"2023-10-30T12:28:56.366897+0000","last_fullsized":"2023-10-30T12:28:56.366897+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:52:30.621125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.580378+0000","last_change":"2023-10-30T12:28:50.967491+0000","last_active":"2023-10-30T12:28:53.580378+0000","last_peered":"2023-10-30T12:28:53.580378+0000","last_clean":"2023-10-30T12:28:53.580378+0000","last_became_active":"2023-10-30T12:28:50.967186+0000","last_became_peered":"2023-10-30T12:28:50.967186+0000","last_unstale":"2023-10-30T12:28:53.580378+0000","last_undegraded":"2023-10-30T12:28:53.580378+0000","last_fullsized":"2023-10-30T12:28:53.580378+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:05:23.759940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579648+0000","last_change":"2023-10-30T12:28:50.363230+0000","last_active":"2023-10-30T12:28:53.579648+0000","last_peered":"2023-10-30T12:28:53.579648+0000","last_clean":"2023-10-30T12:28:53.579648+0000","last_became_active":"2023-10-30T12:28:50.362929+0000","last_became_peered":"2023-10-30T12:28:50.362929+0000","last_unstale":"2023-10-30T12:28:53.579648+0000","last_undegraded":"2023-10-30T12:28:53.579648+0000","last_fullsized":"2023-10-30T12:28:53.579648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:41:27.760578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579661+0000","last_change":"2023-10-30T12:28:50.937780+0000","last_active":"2023-10-30T12:28:53.579661+0000","last_peered":"2023-10-30T12:28:53.579661+0000","last_clean":"2023-10-30T12:28:53.579661+0000","last_became_active":"2023-10-30T12:28:50.937458+0000","last_became_peered":"2023-10-30T12:28:50.937458+0000","last_unstale":"2023-10-30T12:28:53.579661+0000","last_undegraded":"2023-10-30T12:28:53.579661+0000","last_fullsized":"2023-10-30T12:28:53.579661+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:02.083169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:56.366749+0000","last_change":"2023-10-30T12:28:50.386782+0000","last_active":"2023-10-30T12:28:56.366749+0000","last_peered":"2023-10-30T12:28:56.366749+0000","last_clean":"2023-10-30T12:28:56.366749+0000","last_became_active":"2023-10-30T12:28:50.386522+0000","last_became_peered":"2023-10-30T12:28:50.386522+0000","last_unstale":"2023-10-30T12:28:56.366749+0000","last_undegraded":"2023-10-30T12:28:56.366749+0000","last_fullsized":"2023-10-30T12:28:56.366749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:44:03.517641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":3816,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":7632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542153,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27836,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344004,"statfs":{"total":96636764160,"available":96608260096,"internally_reserved":0,"allocated":839680,"data_stored":670548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2729999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.143}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2270000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542153,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":241664,"data_stored":80566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0149999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.089}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2470000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89500000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.145}]}]},{"osd":5,"up_from":14,"seq":60129542153,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":79007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0980000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.169}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2430000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1359999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.272}]}]},{"osd":4,"up_from":14,"seq":60129542152,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":78990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.028}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3049999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542153,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":245760,"data_stored":81098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.109}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.085}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.2370000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.1950000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.1319999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0279999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.2480000000000002}]}]},{"osd":2,"up_from":14,"seq":60129542153,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":78245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.0129999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.218}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5529999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5710000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3330000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4750000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6560000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542153,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":241664,"data_stored":79696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0969999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.1050000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.5529999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.1790000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.5960000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]}]},{"osd":0,"up_from":14,"seq":60129542153,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":831488,"data_stored":669273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:29:03.975 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:29:04.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.144+0000 7f2532e46640 1 Processor -- start 2023-10-30T12:29:04.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.144+0000 7f2532e46640 1 -- start start 2023-10-30T12:29:04.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.144+0000 7f2532e46640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f252c15d6f0 0x7f252c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.144+0000 7f2532e46640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f252c0cf160 0x7f252c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.144+0000 7f2532e46640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f252c0cb800 con 0x7f252c1538a0 2023-10-30T12:29:04.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.144+0000 7f2532e46640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f252c0cb330 con 0x7f252c0cf160 2023-10-30T12:29:04.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.144+0000 7f2532e46640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f252c0cad70 con 0x7f252c15d6f0 2023-10-30T12:29:04.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.144+0000 7f25313e3640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f252c0cf160 0x7f252c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.144+0000 7f25313e3640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f252c0cf160 0x7f252c0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39332/0 (socket says 172.21.15.89:39332) 2023-10-30T12:29:04.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.144+0000 7f2530be2640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f252c1538a0 0x7f252c0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:52208/0 (socket says 172.21.15.89:52208) 2023-10-30T12:29:04.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.144+0000 7f25313e3640 1 -- 172.21.15.89:0/2759675620 learned_addr learned my addr 172.21.15.89:0/2759675620 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:04.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.144+0000 7f2523fff640 1 --2- 172.21.15.89:0/2759675620 >> v2:172.21.15.192:3300/0 conn(0x7f252c15d6f0 0x7f252c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.144+0000 7f25313e3640 1 -- 172.21.15.89:0/2759675620 >> v2:172.21.15.192:3300/0 conn(0x7f252c15d6f0 msgr2=0x7f252c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:04.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.144+0000 7f25313e3640 1 --2- 172.21.15.89:0/2759675620 >> v2:172.21.15.192:3300/0 conn(0x7f252c15d6f0 0x7f252c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.144+0000 7f25313e3640 1 -- 172.21.15.89:0/2759675620 >> v1:172.21.15.89:6789/0 conn(0x7f252c1538a0 legacy=0x7f252c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:04.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.144+0000 7f25313e3640 1 -- 172.21.15.89:0/2759675620 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f252c0cb050 con 0x7f252c0cf160 2023-10-30T12:29:04.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.144+0000 7f25237fe640 1 -- 172.21.15.89:0/2759675620 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4122881954 0 0) 0x7f252c0cb800 con 0x7f252c1538a0 2023-10-30T12:29:04.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.148+0000 7f25313e3640 1 --2- 172.21.15.89:0/2759675620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f252c0cf160 0x7f252c0cf590 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f252c0b30a0 tx=0x7f252401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73b31e5e1a525e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:04.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.148+0000 7f25237fe640 1 -- 172.21.15.89:0/2759675620 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25240af400 con 0x7f252c0cf160 2023-10-30T12:29:04.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.148+0000 7f25237fe640 1 -- 172.21.15.89:0/2759675620 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25240ab050 con 0x7f252c0cf160 2023-10-30T12:29:04.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.148+0000 7f25237fe640 1 -- 172.21.15.89:0/2759675620 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25240af780 con 0x7f252c0cf160 2023-10-30T12:29:04.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.148+0000 7f2532e46640 1 -- 172.21.15.89:0/2759675620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f252c0cf160 msgr2=0x7f252c0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:04.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.148+0000 7f2532e46640 1 --2- 172.21.15.89:0/2759675620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f252c0cf160 0x7f252c0cf590 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f252c0b30a0 tx=0x7f252401f1c0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.148+0000 7f2532e46640 1 -- 172.21.15.89:0/2759675620 shutdown_connections 2023-10-30T12:29:04.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.148+0000 7f2532e46640 1 --2- 172.21.15.89:0/2759675620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f252c0cf160 0x7f252c0cf590 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.148+0000 7f2532e46640 1 --2- 172.21.15.89:0/2759675620 >> v2:172.21.15.192:3300/0 conn(0x7f252c15d6f0 0x7f252c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.148+0000 7f2532e46640 1 -- 172.21.15.89:0/2759675620 >> 172.21.15.89:0/2759675620 conn(0x7f252c06bfe0 msgr2=0x7f252c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:04.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.148+0000 7f2532e46640 1 -- 172.21.15.89:0/2759675620 shutdown_connections 2023-10-30T12:29:04.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.148+0000 7f2532e46640 1 -- 172.21.15.89:0/2759675620 wait complete. 2023-10-30T12:29:04.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.152+0000 7f2532e46640 1 Processor -- start 2023-10-30T12:29:04.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.152+0000 7f2532e46640 1 -- start start 2023-10-30T12:29:04.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.152+0000 7f2532e46640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f252c0cf160 0x7f252c0c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.152+0000 7f2532e46640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f252c1538a0 0x7f252c0ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.152+0000 7f2532e46640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f252c15d6f0 0x7f252c1f2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.152+0000 7f2532e46640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252c0b3ed0 con 0x7f252c0cf160 2023-10-30T12:29:04.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.152+0000 7f2532e46640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252c159500 con 0x7f252c1538a0 2023-10-30T12:29:04.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.152+0000 7f2532e46640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f252c0b4100 con 0x7f252c15d6f0 2023-10-30T12:29:04.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.152+0000 7f2530be2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f252c0cf160 0x7f252c0c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.152+0000 7f2530be2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f252c0cf160 0x7f252c0c65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39334/0 (socket says 172.21.15.89:39334) 2023-10-30T12:29:04.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.152+0000 7f2530be2640 1 -- 172.21.15.89:0/1057995267 learned_addr learned my addr 172.21.15.89:0/1057995267 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:04.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.152+0000 7f2523fff640 1 --2- 172.21.15.89:0/1057995267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f252c1538a0 0x7f252c0ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.152+0000 7f25313e3640 1 --2- 172.21.15.89:0/1057995267 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f252c15d6f0 0x7f252c1f2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.152+0000 7f2530be2640 1 -- 172.21.15.89:0/1057995267 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f252c15d6f0 msgr2=0x7f252c1f2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:04.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.152+0000 7f2530be2640 1 --2- 172.21.15.89:0/1057995267 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f252c15d6f0 0x7f252c1f2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.152+0000 7f2530be2640 1 -- 172.21.15.89:0/1057995267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f252c1538a0 msgr2=0x7f252c0ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:04.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.152+0000 7f2530be2640 1 --2- 172.21.15.89:0/1057995267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f252c1538a0 0x7f252c0ce390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.152+0000 7f2530be2640 1 -- 172.21.15.89:0/1057995267 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2524009b30 con 0x7f252c0cf160 2023-10-30T12:29:04.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.152+0000 7f2523fff640 1 --2- 172.21.15.89:0/1057995267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f252c1538a0 0x7f252c0ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:04.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.152+0000 7f2530be2640 1 --2- 172.21.15.89:0/1057995267 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f252c0cf160 0x7f252c0c65d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f25280011f0 tx=0x7f252800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:04.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.152+0000 7f25217fa640 1 -- 172.21.15.89:0/1057995267 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25280074e0 con 0x7f252c0cf160 2023-10-30T12:29:04.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.152+0000 7f25217fa640 1 -- 172.21.15.89:0/1057995267 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2528007640 con 0x7f252c0cf160 2023-10-30T12:29:04.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.152+0000 7f2532e46640 1 -- 172.21.15.89:0/1057995267 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f252c0b4f30 con 0x7f252c0cf160 2023-10-30T12:29:04.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.152+0000 7f25217fa640 1 -- 172.21.15.89:0/1057995267 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2528004a10 con 0x7f252c0cf160 2023-10-30T12:29:04.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.152+0000 7f2532e46640 1 -- 172.21.15.89:0/1057995267 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f252c1575c0 con 0x7f252c0cf160 2023-10-30T12:29:04.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.156+0000 7f2532e46640 1 -- 172.21.15.89:0/1057995267 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f252c0b2a50 con 0x7f252c0cf160 2023-10-30T12:29:04.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.156+0000 7f25217fa640 1 -- 172.21.15.89:0/1057995267 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2528004b70 con 0x7f252c0cf160 2023-10-30T12:29:04.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.156+0000 7f25217fa640 1 --2- 172.21.15.89:0/1057995267 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f25180664a0 0x7f2518068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.156+0000 7f25217fa640 1 -- 172.21.15.89:0/1057995267 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25280833e0 con 0x7f252c0cf160 2023-10-30T12:29:04.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.156+0000 7f2523fff640 1 --2- 172.21.15.89:0/1057995267 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f25180664a0 0x7f2518068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.156+0000 7f2523fff640 1 --2- 172.21.15.89:0/1057995267 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f25180664a0 0x7f2518068960 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f252c0b1da0 tx=0x7f251c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:04.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.160+0000 7f25217fa640 1 -- 172.21.15.89:0/1057995267 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2528050850 con 0x7f252c0cf160 2023-10-30T12:29:04.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.308+0000 7f2532e46640 1 -- 172.21.15.89:0/1057995267 --> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f252c159b70 con 0x7f25180664a0 2023-10-30T12:29:04.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.328+0000 7f25217fa640 1 -- 172.21.15.89:0/1057995267 <== mgr.4105 v2:172.21.15.192:6832/4076276638 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264637 (secure 0 0 0) 0x7f251c00f830 con 0x7f25180664a0 2023-10-30T12:29:04.337 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:29:04.338 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-10-30T12:29:04.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.332+0000 7f2532e46640 1 -- 172.21.15.89:0/1057995267 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f25180664a0 msgr2=0x7f2518068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:04.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.332+0000 7f2532e46640 1 --2- 172.21.15.89:0/1057995267 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f25180664a0 0x7f2518068960 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f252c0b1da0 tx=0x7f251c008040 comp rx=0 tx=0).stop 2023-10-30T12:29:04.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.332+0000 7f2532e46640 1 -- 172.21.15.89:0/1057995267 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f252c0cf160 msgr2=0x7f252c0c65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:04.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.332+0000 7f2532e46640 1 --2- 172.21.15.89:0/1057995267 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f252c0cf160 0x7f252c0c65d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f25280011f0 tx=0x7f252800c9b0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.332+0000 7f2532e46640 1 -- 172.21.15.89:0/1057995267 shutdown_connections 2023-10-30T12:29:04.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.332+0000 7f2532e46640 1 --2- 172.21.15.89:0/1057995267 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f25180664a0 0x7f2518068960 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.332+0000 7f2532e46640 1 --2- 172.21.15.89:0/1057995267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f252c1538a0 0x7f252c0ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.332+0000 7f2532e46640 1 --2- 172.21.15.89:0/1057995267 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f252c15d6f0 0x7f252c1f2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.332+0000 7f2532e46640 1 --2- 172.21.15.89:0/1057995267 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f252c0cf160 0x7f252c0c65d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.332+0000 7f2532e46640 1 -- 172.21.15.89:0/1057995267 >> 172.21.15.89:0/1057995267 conn(0x7f252c06bfe0 msgr2=0x7f252c15b560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:04.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.332+0000 7f2532e46640 1 -- 172.21.15.89:0/1057995267 shutdown_connections 2023-10-30T12:29:04.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.332+0000 7f2532e46640 1 -- 172.21.15.89:0/1057995267 wait complete. 2023-10-30T12:29:04.358 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2023-10-30T12:29:02.923152+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594184,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":243,"num_write_kb":2442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":190,"ondisk_log_size":190,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219148,"kb_used_data":3020,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755572,"statfs":{"total":773094113280,"available":772869705728,"internally_reserved":0,"allocated":3092480,"data_stored":1817423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12741,"internal_metadata":221236795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3816,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.993756"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028790+0000","last_change":"2023-10-30T12:28:50.979797+0000","last_active":"2023-10-30T12:28:53.028790+0000","last_peered":"2023-10-30T12:28:53.028790+0000","last_clean":"2023-10-30T12:28:53.028790+0000","last_became_active":"2023-10-30T12:28:50.979351+0000","last_became_peered":"2023-10-30T12:28:50.979351+0000","last_unstale":"2023-10-30T12:28:53.028790+0000","last_undegraded":"2023-10-30T12:28:53.028790+0000","last_fullsized":"2023-10-30T12:28:53.028790+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:53:00.556539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.932043+0000","last_change":"2023-10-30T12:28:50.929403+0000","last_active":"2023-10-30T12:28:51.932043+0000","last_peered":"2023-10-30T12:28:51.932043+0000","last_clean":"2023-10-30T12:28:51.932043+0000","last_became_active":"2023-10-30T12:28:50.929133+0000","last_became_peered":"2023-10-30T12:28:50.929133+0000","last_unstale":"2023-10-30T12:28:51.932043+0000","last_undegraded":"2023-10-30T12:28:51.932043+0000","last_fullsized":"2023-10-30T12:28:51.932043+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:12:57.327440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579508+0000","last_change":"2023-10-30T12:28:50.934201+0000","last_active":"2023-10-30T12:28:53.579508+0000","last_peered":"2023-10-30T12:28:53.579508+0000","last_clean":"2023-10-30T12:28:53.579508+0000","last_became_active":"2023-10-30T12:28:50.933896+0000","last_became_peered":"2023-10-30T12:28:50.933896+0000","last_unstale":"2023-10-30T12:28:53.579508+0000","last_undegraded":"2023-10-30T12:28:53.579508+0000","last_fullsized":"2023-10-30T12:28:53.579508+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:20:46.597550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581831+0000","last_change":"2023-10-30T12:28:50.969193+0000","last_active":"2023-10-30T12:28:53.581831+0000","last_peered":"2023-10-30T12:28:53.581831+0000","last_clean":"2023-10-30T12:28:53.581831+0000","last_became_active":"2023-10-30T12:28:50.967774+0000","last_became_peered":"2023-10-30T12:28:50.967774+0000","last_unstale":"2023-10-30T12:28:53.581831+0000","last_undegraded":"2023-10-30T12:28:53.581831+0000","last_fullsized":"2023-10-30T12:28:53.581831+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:23:07.329372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028482+0000","last_change":"2023-10-30T12:28:50.973751+0000","last_active":"2023-10-30T12:28:53.028482+0000","last_peered":"2023-10-30T12:28:53.028482+0000","last_clean":"2023-10-30T12:28:53.028482+0000","last_became_active":"2023-10-30T12:28:50.973132+0000","last_became_peered":"2023-10-30T12:28:50.973132+0000","last_unstale":"2023-10-30T12:28:53.028482+0000","last_undegraded":"2023-10-30T12:28:53.028482+0000","last_fullsized":"2023-10-30T12:28:53.028482+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:43:58.223555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581257+0000","last_change":"2023-10-30T12:28:50.968823+0000","last_active":"2023-10-30T12:28:53.581257+0000","last_peered":"2023-10-30T12:28:53.581257+0000","last_clean":"2023-10-30T12:28:53.581257+0000","last_became_active":"2023-10-30T12:28:50.967910+0000","last_became_peered":"2023-10-30T12:28:50.967910+0000","last_unstale":"2023-10-30T12:28:53.581257+0000","last_undegraded":"2023-10-30T12:28:53.581257+0000","last_fullsized":"2023-10-30T12:28:53.581257+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:34:36.882757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028044+0000","last_change":"2023-10-30T12:28:50.974164+0000","last_active":"2023-10-30T12:28:53.028044+0000","last_peered":"2023-10-30T12:28:53.028044+0000","last_clean":"2023-10-30T12:28:53.028044+0000","last_became_active":"2023-10-30T12:28:50.973288+0000","last_became_peered":"2023-10-30T12:28:50.973288+0000","last_unstale":"2023-10-30T12:28:53.028044+0000","last_undegraded":"2023-10-30T12:28:53.028044+0000","last_fullsized":"2023-10-30T12:28:53.028044+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:09:10.450492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.578880+0000","last_change":"2023-10-30T12:28:50.955287+0000","last_active":"2023-10-30T12:28:53.578880+0000","last_peered":"2023-10-30T12:28:53.578880+0000","last_clean":"2023-10-30T12:28:53.578880+0000","last_became_active":"2023-10-30T12:28:50.954988+0000","last_became_peered":"2023-10-30T12:28:50.954988+0000","last_unstale":"2023-10-30T12:28:53.578880+0000","last_undegraded":"2023-10-30T12:28:53.578880+0000","last_fullsized":"2023-10-30T12:28:53.578880+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:36:30.846920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.029019+0000","last_change":"2023-10-30T12:28:50.977144+0000","last_active":"2023-10-30T12:28:53.029019+0000","last_peered":"2023-10-30T12:28:53.029019+0000","last_clean":"2023-10-30T12:28:53.029019+0000","last_became_active":"2023-10-30T12:28:50.976564+0000","last_became_peered":"2023-10-30T12:28:50.976564+0000","last_unstale":"2023-10-30T12:28:53.029019+0000","last_undegraded":"2023-10-30T12:28:53.029019+0000","last_fullsized":"2023-10-30T12:28:53.029019+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:37:51.341935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.938542+0000","last_change":"2023-10-30T12:28:50.952564+0000","last_active":"2023-10-30T12:28:51.938542+0000","last_peered":"2023-10-30T12:28:51.938542+0000","last_clean":"2023-10-30T12:28:51.938542+0000","last_became_active":"2023-10-30T12:28:50.951809+0000","last_became_peered":"2023-10-30T12:28:50.951809+0000","last_unstale":"2023-10-30T12:28:51.938542+0000","last_undegraded":"2023-10-30T12:28:51.938542+0000","last_fullsized":"2023-10-30T12:28:51.938542+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:34:29.771577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.937215+0000","last_change":"2023-10-30T12:28:50.970779+0000","last_active":"2023-10-30T12:28:51.937215+0000","last_peered":"2023-10-30T12:28:51.937215+0000","last_clean":"2023-10-30T12:28:51.937215+0000","last_became_active":"2023-10-30T12:28:50.969877+0000","last_became_peered":"2023-10-30T12:28:50.969877+0000","last_unstale":"2023-10-30T12:28:51.937215+0000","last_undegraded":"2023-10-30T12:28:51.937215+0000","last_fullsized":"2023-10-30T12:28:51.937215+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:10.539915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.940164+0000","last_change":"2023-10-30T12:28:50.944452+0000","last_active":"2023-10-30T12:28:51.940164+0000","last_peered":"2023-10-30T12:28:51.940164+0000","last_clean":"2023-10-30T12:28:51.940164+0000","last_became_active":"2023-10-30T12:28:50.943787+0000","last_became_peered":"2023-10-30T12:28:50.943787+0000","last_unstale":"2023-10-30T12:28:51.940164+0000","last_undegraded":"2023-10-30T12:28:51.940164+0000","last_fullsized":"2023-10-30T12:28:51.940164+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:30:34.585298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579240+0000","last_change":"2023-10-30T12:28:50.938181+0000","last_active":"2023-10-30T12:28:53.579240+0000","last_peered":"2023-10-30T12:28:53.579240+0000","last_clean":"2023-10-30T12:28:53.579240+0000","last_became_active":"2023-10-30T12:28:50.937906+0000","last_became_peered":"2023-10-30T12:28:50.937906+0000","last_unstale":"2023-10-30T12:28:53.579240+0000","last_undegraded":"2023-10-30T12:28:53.579240+0000","last_fullsized":"2023-10-30T12:28:53.579240+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:41:19.990452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579303+0000","last_change":"2023-10-30T12:28:50.934085+0000","last_active":"2023-10-30T12:28:53.579303+0000","last_peered":"2023-10-30T12:28:53.579303+0000","last_clean":"2023-10-30T12:28:53.579303+0000","last_became_active":"2023-10-30T12:28:50.932753+0000","last_became_peered":"2023-10-30T12:28:50.932753+0000","last_unstale":"2023-10-30T12:28:53.579303+0000","last_undegraded":"2023-10-30T12:28:53.579303+0000","last_fullsized":"2023-10-30T12:28:53.579303+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:23:26.736283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931710+0000","last_change":"2023-10-30T12:28:50.943343+0000","last_active":"2023-10-30T12:28:51.931710+0000","last_peered":"2023-10-30T12:28:51.931710+0000","last_clean":"2023-10-30T12:28:51.931710+0000","last_became_active":"2023-10-30T12:28:50.943011+0000","last_became_peered":"2023-10-30T12:28:50.943011+0000","last_unstale":"2023-10-30T12:28:51.931710+0000","last_undegraded":"2023-10-30T12:28:51.931710+0000","last_fullsized":"2023-10-30T12:28:51.931710+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:59:35.881115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581157+0000","last_change":"2023-10-30T12:28:50.953573+0000","last_active":"2023-10-30T12:28:53.581157+0000","last_peered":"2023-10-30T12:28:53.581157+0000","last_clean":"2023-10-30T12:28:53.581157+0000","last_became_active":"2023-10-30T12:28:50.952744+0000","last_became_peered":"2023-10-30T12:28:50.952744+0000","last_unstale":"2023-10-30T12:28:53.581157+0000","last_undegraded":"2023-10-30T12:28:53.581157+0000","last_fullsized":"2023-10-30T12:28:53.581157+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:41:16.209759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.582342+0000","last_change":"2023-10-30T12:28:50.975156+0000","last_active":"2023-10-30T12:28:53.582342+0000","last_peered":"2023-10-30T12:28:53.582342+0000","last_clean":"2023-10-30T12:28:53.582342+0000","last_became_active":"2023-10-30T12:28:50.974576+0000","last_became_peered":"2023-10-30T12:28:50.974576+0000","last_unstale":"2023-10-30T12:28:53.582342+0000","last_undegraded":"2023-10-30T12:28:53.582342+0000","last_fullsized":"2023-10-30T12:28:53.582342+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:46.159555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.131817+0000","last_change":"2023-10-30T12:28:50.937685+0000","last_active":"2023-10-30T12:28:53.131817+0000","last_peered":"2023-10-30T12:28:53.131817+0000","last_clean":"2023-10-30T12:28:53.131817+0000","last_became_active":"2023-10-30T12:28:50.937384+0000","last_became_peered":"2023-10-30T12:28:50.937384+0000","last_unstale":"2023-10-30T12:28:53.131817+0000","last_undegraded":"2023-10-30T12:28:53.131817+0000","last_fullsized":"2023-10-30T12:28:53.131817+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:16:29.181523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.029181+0000","last_change":"2023-10-30T12:28:50.974918+0000","last_active":"2023-10-30T12:28:53.029181+0000","last_peered":"2023-10-30T12:28:53.029181+0000","last_clean":"2023-10-30T12:28:53.029181+0000","last_became_active":"2023-10-30T12:28:50.973369+0000","last_became_peered":"2023-10-30T12:28:50.973369+0000","last_unstale":"2023-10-30T12:28:53.029181+0000","last_undegraded":"2023-10-30T12:28:53.029181+0000","last_fullsized":"2023-10-30T12:28:53.029181+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:44:04.468611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.937226+0000","last_change":"2023-10-30T12:28:50.955835+0000","last_active":"2023-10-30T12:28:51.937226+0000","last_peered":"2023-10-30T12:28:51.937226+0000","last_clean":"2023-10-30T12:28:51.937226+0000","last_became_active":"2023-10-30T12:28:50.954831+0000","last_became_peered":"2023-10-30T12:28:50.954831+0000","last_unstale":"2023-10-30T12:28:51.937226+0000","last_undegraded":"2023-10-30T12:28:51.937226+0000","last_fullsized":"2023-10-30T12:28:51.937226+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:56:25.554582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.580290+0000","last_change":"2023-10-30T12:28:50.962417+0000","last_active":"2023-10-30T12:28:53.580290+0000","last_peered":"2023-10-30T12:28:53.580290+0000","last_clean":"2023-10-30T12:28:53.580290+0000","last_became_active":"2023-10-30T12:28:50.962065+0000","last_became_peered":"2023-10-30T12:28:50.962065+0000","last_unstale":"2023-10-30T12:28:53.580290+0000","last_undegraded":"2023-10-30T12:28:53.580290+0000","last_fullsized":"2023-10-30T12:28:53.580290+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:13:52.455824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581454+0000","last_change":"2023-10-30T12:28:50.968992+0000","last_active":"2023-10-30T12:28:53.581454+0000","last_peered":"2023-10-30T12:28:53.581454+0000","last_clean":"2023-10-30T12:28:53.581454+0000","last_became_active":"2023-10-30T12:28:50.967651+0000","last_became_peered":"2023-10-30T12:28:50.967651+0000","last_unstale":"2023-10-30T12:28:53.581454+0000","last_undegraded":"2023-10-30T12:28:53.581454+0000","last_fullsized":"2023-10-30T12:28:53.581454+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:39:31.893024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579014+0000","last_change":"2023-10-30T12:28:50.962609+0000","last_active":"2023-10-30T12:28:53.579014+0000","last_peered":"2023-10-30T12:28:53.579014+0000","last_clean":"2023-10-30T12:28:53.579014+0000","last_became_active":"2023-10-30T12:28:50.962290+0000","last_became_peered":"2023-10-30T12:28:50.962290+0000","last_unstale":"2023-10-30T12:28:53.579014+0000","last_undegraded":"2023-10-30T12:28:53.579014+0000","last_fullsized":"2023-10-30T12:28:53.579014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:44:35.484971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581381+0000","last_change":"2023-10-30T12:28:50.966653+0000","last_active":"2023-10-30T12:28:53.581381+0000","last_peered":"2023-10-30T12:28:53.581381+0000","last_clean":"2023-10-30T12:28:53.581381+0000","last_became_active":"2023-10-30T12:28:50.965713+0000","last_became_peered":"2023-10-30T12:28:50.965713+0000","last_unstale":"2023-10-30T12:28:53.581381+0000","last_undegraded":"2023-10-30T12:28:53.581381+0000","last_fullsized":"2023-10-30T12:28:53.581381+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:10:31.600981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579351+0000","last_change":"2023-10-30T12:28:50.951300+0000","last_active":"2023-10-30T12:28:53.579351+0000","last_peered":"2023-10-30T12:28:53.579351+0000","last_clean":"2023-10-30T12:28:53.579351+0000","last_became_active":"2023-10-30T12:28:50.951026+0000","last_became_peered":"2023-10-30T12:28:50.951026+0000","last_unstale":"2023-10-30T12:28:53.579351+0000","last_undegraded":"2023-10-30T12:28:53.579351+0000","last_fullsized":"2023-10-30T12:28:53.579351+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:44:58.872457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.938879+0000","last_change":"2023-10-30T12:28:50.963507+0000","last_active":"2023-10-30T12:28:51.938879+0000","last_peered":"2023-10-30T12:28:51.938879+0000","last_clean":"2023-10-30T12:28:51.938879+0000","last_became_active":"2023-10-30T12:28:50.962484+0000","last_became_peered":"2023-10-30T12:28:50.962484+0000","last_unstale":"2023-10-30T12:28:51.938879+0000","last_undegraded":"2023-10-30T12:28:51.938879+0000","last_fullsized":"2023-10-30T12:28:51.938879+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:11:24.160220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931933+0000","last_change":"2023-10-30T12:28:50.933362+0000","last_active":"2023-10-30T12:28:51.931933+0000","last_peered":"2023-10-30T12:28:51.931933+0000","last_clean":"2023-10-30T12:28:51.931933+0000","last_became_active":"2023-10-30T12:28:50.933093+0000","last_became_peered":"2023-10-30T12:28:50.933093+0000","last_unstale":"2023-10-30T12:28:51.931933+0000","last_undegraded":"2023-10-30T12:28:51.931933+0000","last_fullsized":"2023-10-30T12:28:51.931933+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:54:00.852482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579508+0000","last_change":"2023-10-30T12:28:50.952857+0000","last_active":"2023-10-30T12:28:53.579508+0000","last_peered":"2023-10-30T12:28:53.579508+0000","last_clean":"2023-10-30T12:28:53.579508+0000","last_became_active":"2023-10-30T12:28:50.952531+0000","last_became_peered":"2023-10-30T12:28:50.952531+0000","last_unstale":"2023-10-30T12:28:53.579508+0000","last_undegraded":"2023-10-30T12:28:53.579508+0000","last_fullsized":"2023-10-30T12:28:53.579508+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:01:03.980850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.580366+0000","last_change":"2023-10-30T12:28:50.933408+0000","last_active":"2023-10-30T12:28:53.580366+0000","last_peered":"2023-10-30T12:28:53.580366+0000","last_clean":"2023-10-30T12:28:53.580366+0000","last_became_active":"2023-10-30T12:28:50.933124+0000","last_became_peered":"2023-10-30T12:28:50.933124+0000","last_unstale":"2023-10-30T12:28:53.580366+0000","last_undegraded":"2023-10-30T12:28:53.580366+0000","last_fullsized":"2023-10-30T12:28:53.580366+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:53:50.772420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581630+0000","last_change":"2023-10-30T12:28:50.966388+0000","last_active":"2023-10-30T12:28:53.581630+0000","last_peered":"2023-10-30T12:28:53.581630+0000","last_clean":"2023-10-30T12:28:53.581630+0000","last_became_active":"2023-10-30T12:28:50.965639+0000","last_became_peered":"2023-10-30T12:28:50.965639+0000","last_unstale":"2023-10-30T12:28:53.581630+0000","last_undegraded":"2023-10-30T12:28:53.581630+0000","last_fullsized":"2023-10-30T12:28:53.581630+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:55:46.506378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.131583+0000","last_change":"2023-10-30T12:28:50.955391+0000","last_active":"2023-10-30T12:28:53.131583+0000","last_peered":"2023-10-30T12:28:53.131583+0000","last_clean":"2023-10-30T12:28:53.131583+0000","last_became_active":"2023-10-30T12:28:50.955084+0000","last_became_peered":"2023-10-30T12:28:50.955084+0000","last_unstale":"2023-10-30T12:28:53.131583+0000","last_undegraded":"2023-10-30T12:28:53.131583+0000","last_fullsized":"2023-10-30T12:28:53.131583+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:08:52.389085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.936598+0000","last_change":"2023-10-30T12:28:50.945609+0000","last_active":"2023-10-30T12:28:51.936598+0000","last_peered":"2023-10-30T12:28:51.936598+0000","last_clean":"2023-10-30T12:28:51.936598+0000","last_became_active":"2023-10-30T12:28:50.944775+0000","last_became_peered":"2023-10-30T12:28:50.944775+0000","last_unstale":"2023-10-30T12:28:51.936598+0000","last_undegraded":"2023-10-30T12:28:51.936598+0000","last_fullsized":"2023-10-30T12:28:51.936598+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:08:55.718193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.939146+0000","last_change":"2023-10-30T12:28:50.941378+0000","last_active":"2023-10-30T12:28:51.939146+0000","last_peered":"2023-10-30T12:28:51.939146+0000","last_clean":"2023-10-30T12:28:51.939146+0000","last_became_active":"2023-10-30T12:28:50.940567+0000","last_became_peered":"2023-10-30T12:28:50.940567+0000","last_unstale":"2023-10-30T12:28:51.939146+0000","last_undegraded":"2023-10-30T12:28:51.939146+0000","last_fullsized":"2023-10-30T12:28:51.939146+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:31:23.772343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'5","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.435783+0000","last_change":"2023-10-30T12:28:50.395670+0000","last_active":"2023-10-30T12:29:01.435783+0000","last_peered":"2023-10-30T12:29:01.435783+0000","last_clean":"2023-10-30T12:29:01.435783+0000","last_became_active":"2023-10-30T12:28:50.395382+0000","last_became_peered":"2023-10-30T12:28:50.395382+0000","last_unstale":"2023-10-30T12:29:01.435783+0000","last_undegraded":"2023-10-30T12:29:01.435783+0000","last_fullsized":"2023-10-30T12:29:01.435783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:14.858121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.580427+0000","last_change":"2023-10-30T12:28:50.953202+0000","last_active":"2023-10-30T12:28:53.580427+0000","last_peered":"2023-10-30T12:28:53.580427+0000","last_clean":"2023-10-30T12:28:53.580427+0000","last_became_active":"2023-10-30T12:28:50.952613+0000","last_became_peered":"2023-10-30T12:28:50.952613+0000","last_unstale":"2023-10-30T12:28:53.580427+0000","last_undegraded":"2023-10-30T12:28:53.580427+0000","last_fullsized":"2023-10-30T12:28:53.580427+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:38:56.947997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:52.304063+0000","last_change":"2023-10-30T12:28:50.384598+0000","last_active":"2023-10-30T12:28:52.304063+0000","last_peered":"2023-10-30T12:28:52.304063+0000","last_clean":"2023-10-30T12:28:52.304063+0000","last_became_active":"2023-10-30T12:28:50.383898+0000","last_became_peered":"2023-10-30T12:28:50.383898+0000","last_unstale":"2023-10-30T12:28:52.304063+0000","last_undegraded":"2023-10-30T12:28:52.304063+0000","last_fullsized":"2023-10-30T12:28:52.304063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:09:50.290004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:52.347534+0000","last_change":"2023-10-30T12:28:50.386193+0000","last_active":"2023-10-30T12:28:52.347534+0000","last_peered":"2023-10-30T12:28:52.347534+0000","last_clean":"2023-10-30T12:28:52.347534+0000","last_became_active":"2023-10-30T12:28:50.385963+0000","last_became_peered":"2023-10-30T12:28:50.385963+0000","last_unstale":"2023-10-30T12:28:52.347534+0000","last_undegraded":"2023-10-30T12:28:52.347534+0000","last_fullsized":"2023-10-30T12:28:52.347534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:00:55.603212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.578966+0000","last_change":"2023-10-30T12:28:50.936038+0000","last_active":"2023-10-30T12:28:53.578966+0000","last_peered":"2023-10-30T12:28:53.578966+0000","last_clean":"2023-10-30T12:28:53.578966+0000","last_became_active":"2023-10-30T12:28:50.935764+0000","last_became_peered":"2023-10-30T12:28:50.935764+0000","last_unstale":"2023-10-30T12:28:53.578966+0000","last_undegraded":"2023-10-30T12:28:53.578966+0000","last_fullsized":"2023-10-30T12:28:53.578966+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:23:01.597158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581945+0000","last_change":"2023-10-30T12:28:50.392213+0000","last_active":"2023-10-30T12:28:53.581945+0000","last_peered":"2023-10-30T12:28:53.581945+0000","last_clean":"2023-10-30T12:28:53.581945+0000","last_became_active":"2023-10-30T12:28:50.391494+0000","last_became_peered":"2023-10-30T12:28:50.391494+0000","last_unstale":"2023-10-30T12:28:53.581945+0000","last_undegraded":"2023-10-30T12:28:53.581945+0000","last_fullsized":"2023-10-30T12:28:53.581945+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:38:37.301840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931619+0000","last_change":"2023-10-30T12:28:50.933319+0000","last_active":"2023-10-30T12:28:51.931619+0000","last_peered":"2023-10-30T12:28:51.931619+0000","last_clean":"2023-10-30T12:28:51.931619+0000","last_became_active":"2023-10-30T12:28:50.933062+0000","last_became_peered":"2023-10-30T12:28:50.933062+0000","last_unstale":"2023-10-30T12:28:51.931619+0000","last_undegraded":"2023-10-30T12:28:51.931619+0000","last_fullsized":"2023-10-30T12:28:51.931619+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:26:48.475038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.424753+0000","last_change":"2023-10-30T12:28:50.394368+0000","last_active":"2023-10-30T12:29:01.424753+0000","last_peered":"2023-10-30T12:29:01.424753+0000","last_clean":"2023-10-30T12:29:01.424753+0000","last_became_active":"2023-10-30T12:28:50.393686+0000","last_became_peered":"2023-10-30T12:28:50.393686+0000","last_unstale":"2023-10-30T12:29:01.424753+0000","last_undegraded":"2023-10-30T12:29:01.424753+0000","last_fullsized":"2023-10-30T12:29:01.424753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:43:17.726462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.582247+0000","last_change":"2023-10-30T12:28:50.963703+0000","last_active":"2023-10-30T12:28:53.582247+0000","last_peered":"2023-10-30T12:28:53.582247+0000","last_clean":"2023-10-30T12:28:53.582247+0000","last_became_active":"2023-10-30T12:28:50.963105+0000","last_became_peered":"2023-10-30T12:28:50.963105+0000","last_unstale":"2023-10-30T12:28:53.582247+0000","last_undegraded":"2023-10-30T12:28:53.582247+0000","last_fullsized":"2023-10-30T12:28:53.582247+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:32.423943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.420755+0000","last_change":"2023-10-30T12:28:50.397679+0000","last_active":"2023-10-30T12:29:01.420755+0000","last_peered":"2023-10-30T12:29:01.420755+0000","last_clean":"2023-10-30T12:29:01.420755+0000","last_became_active":"2023-10-30T12:28:50.397041+0000","last_became_peered":"2023-10-30T12:28:50.397041+0000","last_unstale":"2023-10-30T12:29:01.420755+0000","last_undegraded":"2023-10-30T12:29:01.420755+0000","last_fullsized":"2023-10-30T12:29:01.420755+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:52:22.564616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.580443+0000","last_change":"2023-10-30T12:28:50.966838+0000","last_active":"2023-10-30T12:28:53.580443+0000","last_peered":"2023-10-30T12:28:53.580443+0000","last_clean":"2023-10-30T12:28:53.580443+0000","last_became_active":"2023-10-30T12:28:50.966569+0000","last_became_peered":"2023-10-30T12:28:50.966569+0000","last_unstale":"2023-10-30T12:28:53.580443+0000","last_undegraded":"2023-10-30T12:28:53.580443+0000","last_fullsized":"2023-10-30T12:28:53.580443+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:44:33.875449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:56.364393+0000","last_change":"2023-10-30T12:28:50.391527+0000","last_active":"2023-10-30T12:28:56.364393+0000","last_peered":"2023-10-30T12:28:56.364393+0000","last_clean":"2023-10-30T12:28:56.364393+0000","last_became_active":"2023-10-30T12:28:50.391260+0000","last_became_peered":"2023-10-30T12:28:50.391260+0000","last_unstale":"2023-10-30T12:28:56.364393+0000","last_undegraded":"2023-10-30T12:28:56.364393+0000","last_fullsized":"2023-10-30T12:28:56.364393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:29:34.772490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.940211+0000","last_change":"2023-10-30T12:28:50.975441+0000","last_active":"2023-10-30T12:28:51.940211+0000","last_peered":"2023-10-30T12:28:51.940211+0000","last_clean":"2023-10-30T12:28:51.940211+0000","last_became_active":"2023-10-30T12:28:50.974843+0000","last_became_peered":"2023-10-30T12:28:50.974843+0000","last_unstale":"2023-10-30T12:28:51.940211+0000","last_undegraded":"2023-10-30T12:28:51.940211+0000","last_fullsized":"2023-10-30T12:28:51.940211+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:06:12.060752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"22'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.427140+0000","last_change":"2023-10-30T12:28:50.374838+0000","last_active":"2023-10-30T12:29:01.427140+0000","last_peered":"2023-10-30T12:29:01.427140+0000","last_clean":"2023-10-30T12:29:01.427140+0000","last_became_active":"2023-10-30T12:28:50.373581+0000","last_became_peered":"2023-10-30T12:28:50.373581+0000","last_unstale":"2023-10-30T12:29:01.427140+0000","last_undegraded":"2023-10-30T12:29:01.427140+0000","last_fullsized":"2023-10-30T12:29:01.427140+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:14:51.506745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.029046+0000","last_change":"2023-10-30T12:28:50.974012+0000","last_active":"2023-10-30T12:28:53.029046+0000","last_peered":"2023-10-30T12:28:53.029046+0000","last_clean":"2023-10-30T12:28:53.029046+0000","last_became_active":"2023-10-30T12:28:50.973269+0000","last_became_peered":"2023-10-30T12:28:50.973269+0000","last_unstale":"2023-10-30T12:28:53.029046+0000","last_undegraded":"2023-10-30T12:28:53.029046+0000","last_fullsized":"2023-10-30T12:28:53.029046+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:19:21.993898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.427284+0000","last_change":"2023-10-30T12:28:50.366590+0000","last_active":"2023-10-30T12:29:01.427284+0000","last_peered":"2023-10-30T12:29:01.427284+0000","last_clean":"2023-10-30T12:29:01.427284+0000","last_became_active":"2023-10-30T12:28:50.366269+0000","last_became_peered":"2023-10-30T12:28:50.366269+0000","last_unstale":"2023-10-30T12:29:01.427284+0000","last_undegraded":"2023-10-30T12:29:01.427284+0000","last_fullsized":"2023-10-30T12:29:01.427284+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:15:11.963438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579425+0000","last_change":"2023-10-30T12:28:50.962058+0000","last_active":"2023-10-30T12:28:53.579425+0000","last_peered":"2023-10-30T12:28:53.579425+0000","last_clean":"2023-10-30T12:28:53.579425+0000","last_became_active":"2023-10-30T12:28:50.961805+0000","last_became_peered":"2023-10-30T12:28:50.961805+0000","last_unstale":"2023-10-30T12:28:53.579425+0000","last_undegraded":"2023-10-30T12:28:53.579425+0000","last_fullsized":"2023-10-30T12:28:53.579425+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:45:16.373386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.427282+0000","last_change":"2023-10-30T12:28:50.397244+0000","last_active":"2023-10-30T12:29:01.427282+0000","last_peered":"2023-10-30T12:29:01.427282+0000","last_clean":"2023-10-30T12:29:01.427282+0000","last_became_active":"2023-10-30T12:28:50.396727+0000","last_became_peered":"2023-10-30T12:28:50.396727+0000","last_unstale":"2023-10-30T12:29:01.427282+0000","last_undegraded":"2023-10-30T12:29:01.427282+0000","last_fullsized":"2023-10-30T12:29:01.427282+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:33.493343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028618+0000","last_change":"2023-10-30T12:28:50.981498+0000","last_active":"2023-10-30T12:28:53.028618+0000","last_peered":"2023-10-30T12:28:53.028618+0000","last_clean":"2023-10-30T12:28:53.028618+0000","last_became_active":"2023-10-30T12:28:50.981063+0000","last_became_peered":"2023-10-30T12:28:50.981063+0000","last_unstale":"2023-10-30T12:28:53.028618+0000","last_undegraded":"2023-10-30T12:28:53.028618+0000","last_fullsized":"2023-10-30T12:28:53.028618+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:55:17.971880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.937078+0000","last_change":"2023-10-30T12:28:50.383880+0000","last_active":"2023-10-30T12:28:51.937078+0000","last_peered":"2023-10-30T12:28:51.937078+0000","last_clean":"2023-10-30T12:28:51.937078+0000","last_became_active":"2023-10-30T12:28:50.383570+0000","last_became_peered":"2023-10-30T12:28:50.383570+0000","last_unstale":"2023-10-30T12:28:51.937078+0000","last_undegraded":"2023-10-30T12:28:51.937078+0000","last_fullsized":"2023-10-30T12:28:51.937078+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:19:42.047520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.936983+0000","last_change":"2023-10-30T12:28:50.950312+0000","last_active":"2023-10-30T12:28:51.936983+0000","last_peered":"2023-10-30T12:28:51.936983+0000","last_clean":"2023-10-30T12:28:51.936983+0000","last_became_active":"2023-10-30T12:28:50.949485+0000","last_became_peered":"2023-10-30T12:28:50.949485+0000","last_unstale":"2023-10-30T12:28:51.936983+0000","last_undegraded":"2023-10-30T12:28:51.936983+0000","last_fullsized":"2023-10-30T12:28:51.936983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:13:04.669915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"22'10","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.434540+0000","last_change":"2023-10-30T12:28:50.362979+0000","last_active":"2023-10-30T12:29:01.434540+0000","last_peered":"2023-10-30T12:29:01.434540+0000","last_clean":"2023-10-30T12:29:01.434540+0000","last_became_active":"2023-10-30T12:28:50.362624+0000","last_became_peered":"2023-10-30T12:28:50.362624+0000","last_unstale":"2023-10-30T12:29:01.434540+0000","last_undegraded":"2023-10-30T12:29:01.434540+0000","last_fullsized":"2023-10-30T12:29:01.434540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:01:13.111816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028233+0000","last_change":"2023-10-30T12:28:50.979815+0000","last_active":"2023-10-30T12:28:53.028233+0000","last_peered":"2023-10-30T12:28:53.028233+0000","last_clean":"2023-10-30T12:28:53.028233+0000","last_became_active":"2023-10-30T12:28:50.979344+0000","last_became_peered":"2023-10-30T12:28:50.979344+0000","last_unstale":"2023-10-30T12:28:53.028233+0000","last_undegraded":"2023-10-30T12:28:53.028233+0000","last_fullsized":"2023-10-30T12:28:53.028233+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:35:37.018282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:52.284562+0000","last_change":"2023-10-30T12:28:50.382175+0000","last_active":"2023-10-30T12:28:52.284562+0000","last_peered":"2023-10-30T12:28:52.284562+0000","last_clean":"2023-10-30T12:28:52.284562+0000","last_became_active":"2023-10-30T12:28:50.381125+0000","last_became_peered":"2023-10-30T12:28:50.381125+0000","last_unstale":"2023-10-30T12:28:52.284562+0000","last_undegraded":"2023-10-30T12:28:52.284562+0000","last_fullsized":"2023-10-30T12:28:52.284562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:36:02.652665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028807+0000","last_change":"2023-10-30T12:28:50.978034+0000","last_active":"2023-10-30T12:28:53.028807+0000","last_peered":"2023-10-30T12:28:53.028807+0000","last_clean":"2023-10-30T12:28:53.028807+0000","last_became_active":"2023-10-30T12:28:50.977585+0000","last_became_peered":"2023-10-30T12:28:50.977585+0000","last_unstale":"2023-10-30T12:28:53.028807+0000","last_undegraded":"2023-10-30T12:28:53.028807+0000","last_fullsized":"2023-10-30T12:28:53.028807+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:42:56.806184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:56.364009+0000","last_change":"2023-10-30T12:28:50.367258+0000","last_active":"2023-10-30T12:28:56.364009+0000","last_peered":"2023-10-30T12:28:56.364009+0000","last_clean":"2023-10-30T12:28:56.364009+0000","last_became_active":"2023-10-30T12:28:50.366974+0000","last_became_peered":"2023-10-30T12:28:50.366974+0000","last_unstale":"2023-10-30T12:28:56.364009+0000","last_undegraded":"2023-10-30T12:28:56.364009+0000","last_fullsized":"2023-10-30T12:28:56.364009+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:14:48.823902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.029189+0000","last_change":"2023-10-30T12:28:50.981784+0000","last_active":"2023-10-30T12:28:53.029189+0000","last_peered":"2023-10-30T12:28:53.029189+0000","last_clean":"2023-10-30T12:28:53.029189+0000","last_became_active":"2023-10-30T12:28:50.981193+0000","last_became_peered":"2023-10-30T12:28:50.981193+0000","last_unstale":"2023-10-30T12:28:53.029189+0000","last_undegraded":"2023-10-30T12:28:53.029189+0000","last_fullsized":"2023-10-30T12:28:53.029189+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:36:17.384732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:52.280239+0000","last_change":"2023-10-30T12:28:50.368011+0000","last_active":"2023-10-30T12:28:52.280239+0000","last_peered":"2023-10-30T12:28:52.280239+0000","last_clean":"2023-10-30T12:28:52.280239+0000","last_became_active":"2023-10-30T12:28:50.367730+0000","last_became_peered":"2023-10-30T12:28:50.367730+0000","last_unstale":"2023-10-30T12:28:52.280239+0000","last_undegraded":"2023-10-30T12:28:52.280239+0000","last_fullsized":"2023-10-30T12:28:52.280239+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:59:41.946550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.582218+0000","last_change":"2023-10-30T12:28:50.968262+0000","last_active":"2023-10-30T12:28:53.582218+0000","last_peered":"2023-10-30T12:28:53.582218+0000","last_clean":"2023-10-30T12:28:53.582218+0000","last_became_active":"2023-10-30T12:28:50.966246+0000","last_became_peered":"2023-10-30T12:28:50.966246+0000","last_unstale":"2023-10-30T12:28:53.582218+0000","last_undegraded":"2023-10-30T12:28:53.582218+0000","last_fullsized":"2023-10-30T12:28:53.582218+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:00:01.991935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:52.277996+0000","last_change":"2023-10-30T12:28:50.394060+0000","last_active":"2023-10-30T12:28:52.277996+0000","last_peered":"2023-10-30T12:28:52.277996+0000","last_clean":"2023-10-30T12:28:52.277996+0000","last_became_active":"2023-10-30T12:28:50.393642+0000","last_became_peered":"2023-10-30T12:28:50.393642+0000","last_unstale":"2023-10-30T12:28:52.277996+0000","last_undegraded":"2023-10-30T12:28:52.277996+0000","last_fullsized":"2023-10-30T12:28:52.277996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:21.643882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028738+0000","last_change":"2023-10-30T12:28:50.976814+0000","last_active":"2023-10-30T12:28:53.028738+0000","last_peered":"2023-10-30T12:28:53.028738+0000","last_clean":"2023-10-30T12:28:53.028738+0000","last_became_active":"2023-10-30T12:28:50.976343+0000","last_became_peered":"2023-10-30T12:28:50.976343+0000","last_unstale":"2023-10-30T12:28:53.028738+0000","last_undegraded":"2023-10-30T12:28:53.028738+0000","last_fullsized":"2023-10-30T12:28:53.028738+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:33:23.491998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.946067+0000","last_change":"2023-10-30T12:28:41.203990+0000","last_active":"2023-10-30T12:28:50.946067+0000","last_peered":"2023-10-30T12:28:50.946067+0000","last_clean":"2023-10-30T12:28:50.946067+0000","last_became_active":"2023-10-30T12:28:41.201796+0000","last_became_peered":"2023-10-30T12:28:41.201796+0000","last_unstale":"2023-10-30T12:28:50.946067+0000","last_undegraded":"2023-10-30T12:28:50.946067+0000","last_fullsized":"2023-10-30T12:28:50.946067+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:39.218257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:39.218257+0000","last_clean_scrub_stamp":"2023-10-30T12:28:39.218257+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:21:00.605462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.425894+0000","last_change":"2023-10-30T12:28:50.372651+0000","last_active":"2023-10-30T12:29:01.425894+0000","last_peered":"2023-10-30T12:29:01.425894+0000","last_clean":"2023-10-30T12:29:01.425894+0000","last_became_active":"2023-10-30T12:28:50.367593+0000","last_became_peered":"2023-10-30T12:28:50.367593+0000","last_unstale":"2023-10-30T12:29:01.425894+0000","last_undegraded":"2023-10-30T12:29:01.425894+0000","last_fullsized":"2023-10-30T12:29:01.425894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:32:10.492061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579128+0000","last_change":"2023-10-30T12:28:50.970536+0000","last_active":"2023-10-30T12:28:53.579128+0000","last_peered":"2023-10-30T12:28:53.579128+0000","last_clean":"2023-10-30T12:28:53.579128+0000","last_became_active":"2023-10-30T12:28:50.970326+0000","last_became_peered":"2023-10-30T12:28:50.970326+0000","last_unstale":"2023-10-30T12:28:53.579128+0000","last_undegraded":"2023-10-30T12:28:53.579128+0000","last_fullsized":"2023-10-30T12:28:53.579128+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:53:54.589174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.027894+0000","last_change":"2023-10-30T12:28:50.392461+0000","last_active":"2023-10-30T12:28:53.027894+0000","last_peered":"2023-10-30T12:28:53.027894+0000","last_clean":"2023-10-30T12:28:53.027894+0000","last_became_active":"2023-10-30T12:28:50.391824+0000","last_became_peered":"2023-10-30T12:28:50.391824+0000","last_unstale":"2023-10-30T12:28:53.027894+0000","last_undegraded":"2023-10-30T12:28:53.027894+0000","last_fullsized":"2023-10-30T12:28:53.027894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:56:10.373046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579229+0000","last_change":"2023-10-30T12:28:50.953046+0000","last_active":"2023-10-30T12:28:53.579229+0000","last_peered":"2023-10-30T12:28:53.579229+0000","last_clean":"2023-10-30T12:28:53.579229+0000","last_became_active":"2023-10-30T12:28:50.952625+0000","last_became_peered":"2023-10-30T12:28:50.952625+0000","last_unstale":"2023-10-30T12:28:53.579229+0000","last_undegraded":"2023-10-30T12:28:53.579229+0000","last_fullsized":"2023-10-30T12:28:53.579229+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:00:14.643989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.131708+0000","last_change":"2023-10-30T12:28:50.367547+0000","last_active":"2023-10-30T12:28:53.131708+0000","last_peered":"2023-10-30T12:28:53.131708+0000","last_clean":"2023-10-30T12:28:53.131708+0000","last_became_active":"2023-10-30T12:28:50.367230+0000","last_became_peered":"2023-10-30T12:28:50.367230+0000","last_unstale":"2023-10-30T12:28:53.131708+0000","last_undegraded":"2023-10-30T12:28:53.131708+0000","last_fullsized":"2023-10-30T12:28:53.131708+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:42.570786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931639+0000","last_change":"2023-10-30T12:28:50.928888+0000","last_active":"2023-10-30T12:28:51.931639+0000","last_peered":"2023-10-30T12:28:51.931639+0000","last_clean":"2023-10-30T12:28:51.931639+0000","last_became_active":"2023-10-30T12:28:50.928608+0000","last_became_peered":"2023-10-30T12:28:50.928608+0000","last_unstale":"2023-10-30T12:28:51.931639+0000","last_undegraded":"2023-10-30T12:28:51.931639+0000","last_fullsized":"2023-10-30T12:28:51.931639+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:40:51.042396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:52.285682+0000","last_change":"2023-10-30T12:28:50.391240+0000","last_active":"2023-10-30T12:28:52.285682+0000","last_peered":"2023-10-30T12:28:52.285682+0000","last_clean":"2023-10-30T12:28:52.285682+0000","last_became_active":"2023-10-30T12:28:50.390864+0000","last_became_peered":"2023-10-30T12:28:50.390864+0000","last_unstale":"2023-10-30T12:28:52.285682+0000","last_undegraded":"2023-10-30T12:28:52.285682+0000","last_fullsized":"2023-10-30T12:28:52.285682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:59:40.285830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.939756+0000","last_change":"2023-10-30T12:28:50.949990+0000","last_active":"2023-10-30T12:28:51.939756+0000","last_peered":"2023-10-30T12:28:51.939756+0000","last_clean":"2023-10-30T12:28:51.939756+0000","last_became_active":"2023-10-30T12:28:50.949122+0000","last_became_peered":"2023-10-30T12:28:50.949122+0000","last_unstale":"2023-10-30T12:28:51.939756+0000","last_undegraded":"2023-10-30T12:28:51.939756+0000","last_fullsized":"2023-10-30T12:28:51.939756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:30:56.099810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931803+0000","last_change":"2023-10-30T12:28:50.358700+0000","last_active":"2023-10-30T12:28:51.931803+0000","last_peered":"2023-10-30T12:28:51.931803+0000","last_clean":"2023-10-30T12:28:51.931803+0000","last_became_active":"2023-10-30T12:28:50.358310+0000","last_became_peered":"2023-10-30T12:28:50.358310+0000","last_unstale":"2023-10-30T12:28:51.931803+0000","last_undegraded":"2023-10-30T12:28:51.931803+0000","last_fullsized":"2023-10-30T12:28:51.931803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:23:30.910622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.582250+0000","last_change":"2023-10-30T12:28:50.967650+0000","last_active":"2023-10-30T12:28:53.582250+0000","last_peered":"2023-10-30T12:28:53.582250+0000","last_clean":"2023-10-30T12:28:53.582250+0000","last_became_active":"2023-10-30T12:28:50.965626+0000","last_became_peered":"2023-10-30T12:28:50.965626+0000","last_unstale":"2023-10-30T12:28:53.582250+0000","last_undegraded":"2023-10-30T12:28:53.582250+0000","last_fullsized":"2023-10-30T12:28:53.582250+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:49.050703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.029231+0000","last_change":"2023-10-30T12:28:50.395645+0000","last_active":"2023-10-30T12:28:53.029231+0000","last_peered":"2023-10-30T12:28:53.029231+0000","last_clean":"2023-10-30T12:28:53.029231+0000","last_became_active":"2023-10-30T12:28:50.395090+0000","last_became_peered":"2023-10-30T12:28:50.395090+0000","last_unstale":"2023-10-30T12:28:53.029231+0000","last_undegraded":"2023-10-30T12:28:53.029231+0000","last_fullsized":"2023-10-30T12:28:53.029231+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:45:44.678383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931844+0000","last_change":"2023-10-30T12:28:50.929586+0000","last_active":"2023-10-30T12:28:51.931844+0000","last_peered":"2023-10-30T12:28:51.931844+0000","last_clean":"2023-10-30T12:28:51.931844+0000","last_became_active":"2023-10-30T12:28:50.929337+0000","last_became_peered":"2023-10-30T12:28:50.929337+0000","last_unstale":"2023-10-30T12:28:51.931844+0000","last_undegraded":"2023-10-30T12:28:51.931844+0000","last_fullsized":"2023-10-30T12:28:51.931844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:11:13.067698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:52.280274+0000","last_change":"2023-10-30T12:28:50.380474+0000","last_active":"2023-10-30T12:28:52.280274+0000","last_peered":"2023-10-30T12:28:52.280274+0000","last_clean":"2023-10-30T12:28:52.280274+0000","last_became_active":"2023-10-30T12:28:50.380247+0000","last_became_peered":"2023-10-30T12:28:50.380247+0000","last_unstale":"2023-10-30T12:28:52.280274+0000","last_undegraded":"2023-10-30T12:28:52.280274+0000","last_fullsized":"2023-10-30T12:28:52.280274+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:52:03.952617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579476+0000","last_change":"2023-10-30T12:28:50.973361+0000","last_active":"2023-10-30T12:28:53.579476+0000","last_peered":"2023-10-30T12:28:53.579476+0000","last_clean":"2023-10-30T12:28:53.579476+0000","last_became_active":"2023-10-30T12:28:50.973086+0000","last_became_peered":"2023-10-30T12:28:50.973086+0000","last_unstale":"2023-10-30T12:28:53.579476+0000","last_undegraded":"2023-10-30T12:28:53.579476+0000","last_fullsized":"2023-10-30T12:28:53.579476+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:49:48.589889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.421918+0000","last_change":"2023-10-30T12:28:50.386277+0000","last_active":"2023-10-30T12:29:01.421918+0000","last_peered":"2023-10-30T12:29:01.421918+0000","last_clean":"2023-10-30T12:29:01.421918+0000","last_became_active":"2023-10-30T12:28:50.385924+0000","last_became_peered":"2023-10-30T12:28:50.385924+0000","last_unstale":"2023-10-30T12:29:01.421918+0000","last_undegraded":"2023-10-30T12:29:01.421918+0000","last_fullsized":"2023-10-30T12:29:01.421918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:35:03.837523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.580322+0000","last_change":"2023-10-30T12:28:50.929670+0000","last_active":"2023-10-30T12:28:53.580322+0000","last_peered":"2023-10-30T12:28:53.580322+0000","last_clean":"2023-10-30T12:28:53.580322+0000","last_became_active":"2023-10-30T12:28:50.929259+0000","last_became_peered":"2023-10-30T12:28:50.929259+0000","last_unstale":"2023-10-30T12:28:53.580322+0000","last_undegraded":"2023-10-30T12:28:53.580322+0000","last_fullsized":"2023-10-30T12:28:53.580322+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:50:16.507660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.420373+0000","last_change":"2023-10-30T12:28:50.386426+0000","last_active":"2023-10-30T12:29:01.420373+0000","last_peered":"2023-10-30T12:29:01.420373+0000","last_clean":"2023-10-30T12:29:01.420373+0000","last_became_active":"2023-10-30T12:28:50.385305+0000","last_became_peered":"2023-10-30T12:28:50.385305+0000","last_unstale":"2023-10-30T12:29:01.420373+0000","last_undegraded":"2023-10-30T12:29:01.420373+0000","last_fullsized":"2023-10-30T12:29:01.420373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:13:05.376581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.938368+0000","last_change":"2023-10-30T12:28:50.941607+0000","last_active":"2023-10-30T12:28:51.938368+0000","last_peered":"2023-10-30T12:28:51.938368+0000","last_clean":"2023-10-30T12:28:51.938368+0000","last_became_active":"2023-10-30T12:28:50.940846+0000","last_became_peered":"2023-10-30T12:28:50.940846+0000","last_unstale":"2023-10-30T12:28:51.938368+0000","last_undegraded":"2023-10-30T12:28:51.938368+0000","last_fullsized":"2023-10-30T12:28:51.938368+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:29.837077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.433290+0000","last_change":"2023-10-30T12:28:50.392333+0000","last_active":"2023-10-30T12:29:01.433290+0000","last_peered":"2023-10-30T12:29:01.433290+0000","last_clean":"2023-10-30T12:29:01.433290+0000","last_became_active":"2023-10-30T12:28:50.392068+0000","last_became_peered":"2023-10-30T12:28:50.392068+0000","last_unstale":"2023-10-30T12:29:01.433290+0000","last_undegraded":"2023-10-30T12:29:01.433290+0000","last_fullsized":"2023-10-30T12:29:01.433290+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:47:24.674925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579044+0000","last_change":"2023-10-30T12:28:50.932167+0000","last_active":"2023-10-30T12:28:53.579044+0000","last_peered":"2023-10-30T12:28:53.579044+0000","last_clean":"2023-10-30T12:28:53.579044+0000","last_became_active":"2023-10-30T12:28:50.931898+0000","last_became_peered":"2023-10-30T12:28:50.931898+0000","last_unstale":"2023-10-30T12:28:53.579044+0000","last_undegraded":"2023-10-30T12:28:53.579044+0000","last_fullsized":"2023-10-30T12:28:53.579044+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:44.429045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931812+0000","last_change":"2023-10-30T12:28:50.365055+0000","last_active":"2023-10-30T12:28:51.931812+0000","last_peered":"2023-10-30T12:28:51.931812+0000","last_clean":"2023-10-30T12:28:51.931812+0000","last_became_active":"2023-10-30T12:28:50.364797+0000","last_became_peered":"2023-10-30T12:28:50.364797+0000","last_unstale":"2023-10-30T12:28:51.931812+0000","last_undegraded":"2023-10-30T12:28:51.931812+0000","last_fullsized":"2023-10-30T12:28:51.931812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:06:23.067895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579266+0000","last_change":"2023-10-30T12:28:50.935908+0000","last_active":"2023-10-30T12:28:53.579266+0000","last_peered":"2023-10-30T12:28:53.579266+0000","last_clean":"2023-10-30T12:28:53.579266+0000","last_became_active":"2023-10-30T12:28:50.935657+0000","last_became_peered":"2023-10-30T12:28:50.935657+0000","last_unstale":"2023-10-30T12:28:53.579266+0000","last_undegraded":"2023-10-30T12:28:53.579266+0000","last_fullsized":"2023-10-30T12:28:53.579266+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:36:13.242429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.416976+0000","last_change":"2023-10-30T12:28:50.393465+0000","last_active":"2023-10-30T12:29:01.416976+0000","last_peered":"2023-10-30T12:29:01.416976+0000","last_clean":"2023-10-30T12:29:01.416976+0000","last_became_active":"2023-10-30T12:28:50.392807+0000","last_became_peered":"2023-10-30T12:28:50.392807+0000","last_unstale":"2023-10-30T12:29:01.416976+0000","last_undegraded":"2023-10-30T12:29:01.416976+0000","last_fullsized":"2023-10-30T12:29:01.416976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:35:37.976956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.938767+0000","last_change":"2023-10-30T12:28:50.954834+0000","last_active":"2023-10-30T12:28:51.938767+0000","last_peered":"2023-10-30T12:28:51.938767+0000","last_clean":"2023-10-30T12:28:51.938767+0000","last_became_active":"2023-10-30T12:28:50.954291+0000","last_became_peered":"2023-10-30T12:28:50.954291+0000","last_unstale":"2023-10-30T12:28:51.938767+0000","last_undegraded":"2023-10-30T12:28:51.938767+0000","last_fullsized":"2023-10-30T12:28:51.938767+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:19:44.345918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581732+0000","last_change":"2023-10-30T12:28:50.963620+0000","last_active":"2023-10-30T12:28:53.581732+0000","last_peered":"2023-10-30T12:28:53.581732+0000","last_clean":"2023-10-30T12:28:53.581732+0000","last_became_active":"2023-10-30T12:28:50.962817+0000","last_became_peered":"2023-10-30T12:28:50.962817+0000","last_unstale":"2023-10-30T12:28:53.581732+0000","last_undegraded":"2023-10-30T12:28:53.581732+0000","last_fullsized":"2023-10-30T12:28:53.581732+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:23:23.276086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:56.367962+0000","last_change":"2023-10-30T12:28:50.402179+0000","last_active":"2023-10-30T12:28:56.367962+0000","last_peered":"2023-10-30T12:28:56.367962+0000","last_clean":"2023-10-30T12:28:56.367962+0000","last_became_active":"2023-10-30T12:28:50.401677+0000","last_became_peered":"2023-10-30T12:28:50.401677+0000","last_unstale":"2023-10-30T12:28:56.367962+0000","last_undegraded":"2023-10-30T12:28:56.367962+0000","last_fullsized":"2023-10-30T12:28:56.367962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:39:44.432214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581874+0000","last_change":"2023-10-30T12:28:50.968805+0000","last_active":"2023-10-30T12:28:53.581874+0000","last_peered":"2023-10-30T12:28:53.581874+0000","last_clean":"2023-10-30T12:28:53.581874+0000","last_became_active":"2023-10-30T12:28:50.967591+0000","last_became_peered":"2023-10-30T12:28:50.967591+0000","last_unstale":"2023-10-30T12:28:53.581874+0000","last_undegraded":"2023-10-30T12:28:53.581874+0000","last_fullsized":"2023-10-30T12:28:53.581874+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:11:41.973858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:56.366897+0000","last_change":"2023-10-30T12:28:50.392617+0000","last_active":"2023-10-30T12:28:56.366897+0000","last_peered":"2023-10-30T12:28:56.366897+0000","last_clean":"2023-10-30T12:28:56.366897+0000","last_became_active":"2023-10-30T12:28:50.392021+0000","last_became_peered":"2023-10-30T12:28:50.392021+0000","last_unstale":"2023-10-30T12:28:56.366897+0000","last_undegraded":"2023-10-30T12:28:56.366897+0000","last_fullsized":"2023-10-30T12:28:56.366897+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:52:30.621125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.580378+0000","last_change":"2023-10-30T12:28:50.967491+0000","last_active":"2023-10-30T12:28:53.580378+0000","last_peered":"2023-10-30T12:28:53.580378+0000","last_clean":"2023-10-30T12:28:53.580378+0000","last_became_active":"2023-10-30T12:28:50.967186+0000","last_became_peered":"2023-10-30T12:28:50.967186+0000","last_unstale":"2023-10-30T12:28:53.580378+0000","last_undegraded":"2023-10-30T12:28:53.580378+0000","last_fullsized":"2023-10-30T12:28:53.580378+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:05:23.759940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579648+0000","last_change":"2023-10-30T12:28:50.363230+0000","last_active":"2023-10-30T12:28:53.579648+0000","last_peered":"2023-10-30T12:28:53.579648+0000","last_clean":"2023-10-30T12:28:53.579648+0000","last_became_active":"2023-10-30T12:28:50.362929+0000","last_became_peered":"2023-10-30T12:28:50.362929+0000","last_unstale":"2023-10-30T12:28:53.579648+0000","last_undegraded":"2023-10-30T12:28:53.579648+0000","last_fullsized":"2023-10-30T12:28:53.579648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:41:27.760578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579661+0000","last_change":"2023-10-30T12:28:50.937780+0000","last_active":"2023-10-30T12:28:53.579661+0000","last_peered":"2023-10-30T12:28:53.579661+0000","last_clean":"2023-10-30T12:28:53.579661+0000","last_became_active":"2023-10-30T12:28:50.937458+0000","last_became_peered":"2023-10-30T12:28:50.937458+0000","last_unstale":"2023-10-30T12:28:53.579661+0000","last_undegraded":"2023-10-30T12:28:53.579661+0000","last_fullsized":"2023-10-30T12:28:53.579661+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:02.083169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:56.366749+0000","last_change":"2023-10-30T12:28:50.386782+0000","last_active":"2023-10-30T12:28:56.366749+0000","last_peered":"2023-10-30T12:28:56.366749+0000","last_clean":"2023-10-30T12:28:56.366749+0000","last_became_active":"2023-10-30T12:28:50.386522+0000","last_became_peered":"2023-10-30T12:28:50.386522+0000","last_unstale":"2023-10-30T12:28:56.366749+0000","last_undegraded":"2023-10-30T12:28:56.366749+0000","last_fullsized":"2023-10-30T12:28:56.366749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:44:03.517641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":3816,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":7632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542153,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27836,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344004,"statfs":{"total":96636764160,"available":96608260096,"internally_reserved":0,"allocated":839680,"data_stored":670548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2729999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.143}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2270000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542153,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":241664,"data_stored":80566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0149999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.089}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2470000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89500000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.145}]}]},{"osd":5,"up_from":14,"seq":60129542153,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":79007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0980000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.169}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2430000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1359999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.272}]}]},{"osd":4,"up_from":14,"seq":60129542152,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":78990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.028}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3049999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542153,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":245760,"data_stored":81098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.109}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.085}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.2370000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.1950000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.1319999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0279999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.2480000000000002}]}]},{"osd":2,"up_from":14,"seq":60129542153,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":78245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.0129999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.218}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5529999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5710000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3330000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4750000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6560000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542153,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":241664,"data_stored":79696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0969999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.1050000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.5529999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.1790000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.5960000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]}]},{"osd":0,"up_from":14,"seq":60129542153,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":831488,"data_stored":669273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:29:04.364 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-30T12:29:04.364 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-30T12:29:04.364 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-30T12:29:04.364 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-30T12:29:04.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.528+0000 7f5b572dc640 1 Processor -- start 2023-10-30T12:29:04.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.528+0000 7f5b572dc640 1 -- start start 2023-10-30T12:29:04.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.532+0000 7f5b572dc640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5b5015cd50 0x7f5b500c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.532+0000 7f5b572dc640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b5015a6c0 con 0x7f5b500cefa0 2023-10-30T12:29:04.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.532+0000 7f5b572dc640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b5015d300 con 0x7f5b501538f0 2023-10-30T12:29:04.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.532+0000 7f5b572dc640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b5015d5e0 con 0x7f5b5015cd50 2023-10-30T12:29:04.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.532+0000 7f5b55ad9640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5b5015cd50 0x7f5b500c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.532+0000 7f5b55ad9640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5b5015cd50 0x7f5b500c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39336/0 (socket says 172.21.15.89:39336) 2023-10-30T12:29:04.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.532+0000 7f5b55ad9640 1 -- 172.21.15.89:0/2801151962 learned_addr learned my addr 172.21.15.89:0/2801151962 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:04.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.532+0000 7f5b56adb640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f5b500cefa0 0x7f5b501640e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:52222/0 (socket says 172.21.15.89:52222) 2023-10-30T12:29:04.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.532+0000 7f5b55ad9640 1 -- 172.21.15.89:0/2801151962 >> v1:172.21.15.192:6789/0 conn(0x7f5b501538f0 legacy=0x7f5b50159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:04.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.532+0000 7f5b55ad9640 1 -- 172.21.15.89:0/2801151962 >> v1:172.21.15.89:6789/0 conn(0x7f5b500cefa0 legacy=0x7f5b501640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:04.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.532+0000 7f5b55ad9640 1 -- 172.21.15.89:0/2801151962 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b5015d8c0 con 0x7f5b5015cd50 2023-10-30T12:29:04.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.532+0000 7f5b55ad9640 1 --2- 172.21.15.89:0/2801151962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b5015cd50 0x7f5b500c9f90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5b40009870 tx=0x7f5b4001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56591b408eb54150 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:04.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.536+0000 7f5b552d8640 1 -- 172.21.15.89:0/2801151962 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b400a8a30 con 0x7f5b5015cd50 2023-10-30T12:29:04.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.536+0000 7f5b552d8640 1 -- 172.21.15.89:0/2801151962 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b400a8b90 con 0x7f5b5015cd50 2023-10-30T12:29:04.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.536+0000 7f5b552d8640 1 -- 172.21.15.89:0/2801151962 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b400b0440 con 0x7f5b5015cd50 2023-10-30T12:29:04.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.536+0000 7f5b572dc640 1 -- 172.21.15.89:0/2801151962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b5015cd50 msgr2=0x7f5b500c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:04.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.536+0000 7f5b572dc640 1 --2- 172.21.15.89:0/2801151962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b5015cd50 0x7f5b500c9f90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5b40009870 tx=0x7f5b4001ed30 comp rx=0 tx=0).stop 2023-10-30T12:29:04.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.536+0000 7f5b572dc640 1 -- 172.21.15.89:0/2801151962 shutdown_connections 2023-10-30T12:29:04.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.536+0000 7f5b572dc640 1 --2- 172.21.15.89:0/2801151962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b5015cd50 0x7f5b500c9f90 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.536+0000 7f5b572dc640 1 -- 172.21.15.89:0/2801151962 >> 172.21.15.89:0/2801151962 conn(0x7f5b5006a640 msgr2=0x7f5b50065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:04.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.536+0000 7f5b572dc640 1 -- 172.21.15.89:0/2801151962 shutdown_connections 2023-10-30T12:29:04.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.536+0000 7f5b572dc640 1 -- 172.21.15.89:0/2801151962 wait complete. 2023-10-30T12:29:04.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.536+0000 7f5b572dc640 1 Processor -- start 2023-10-30T12:29:04.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.536+0000 7f5b572dc640 1 -- start start 2023-10-30T12:29:04.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.536+0000 7f5b572dc640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b500cefa0 0x7f5b5015c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.536+0000 7f5b572dc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b501538f0 0x7f5b500b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.536+0000 7f5b572dc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b5015cd50 0x7f5b5015acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.536+0000 7f5b572dc640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b501ef540 con 0x7f5b501538f0 2023-10-30T12:29:04.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.536+0000 7f5b572dc640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b501ef820 con 0x7f5b5015cd50 2023-10-30T12:29:04.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.536+0000 7f5b572dc640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b501efb00 con 0x7f5b500cefa0 2023-10-30T12:29:04.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.536+0000 7f5b55ad9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b501538f0 0x7f5b500b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.536+0000 7f5b55ad9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b501538f0 0x7f5b500b4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39344/0 (socket says 172.21.15.89:39344) 2023-10-30T12:29:04.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.536+0000 7f5b55ad9640 1 -- 172.21.15.89:0/398970174 learned_addr learned my addr 172.21.15.89:0/398970174 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:04.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.540+0000 7f5b562da640 1 --2- 172.21.15.89:0/398970174 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b500cefa0 0x7f5b5015c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.540+0000 7f5b56adb640 1 --2- 172.21.15.89:0/398970174 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b5015cd50 0x7f5b5015acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.540+0000 7f5b55ad9640 1 -- 172.21.15.89:0/398970174 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b500cefa0 msgr2=0x7f5b5015c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:04.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.540+0000 7f5b55ad9640 1 --2- 172.21.15.89:0/398970174 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b500cefa0 0x7f5b5015c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.540+0000 7f5b55ad9640 1 -- 172.21.15.89:0/398970174 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b5015cd50 msgr2=0x7f5b5015acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:04.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.540+0000 7f5b55ad9640 1 --2- 172.21.15.89:0/398970174 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b5015cd50 0x7f5b5015acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.540+0000 7f5b55ad9640 1 -- 172.21.15.89:0/398970174 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b400094d0 con 0x7f5b501538f0 2023-10-30T12:29:04.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.540+0000 7f5b562da640 1 --2- 172.21.15.89:0/398970174 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b500cefa0 0x7f5b5015c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:29:04.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.540+0000 7f5b56adb640 1 --2- 172.21.15.89:0/398970174 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b5015cd50 0x7f5b5015acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:29:04.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.540+0000 7f5b55ad9640 1 --2- 172.21.15.89:0/398970174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b501538f0 0x7f5b500b4450 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5b400b3040 tx=0x7f5b4001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:04.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.540+0000 7f5b46ffd640 1 -- 172.21.15.89:0/398970174 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b400be070 con 0x7f5b501538f0 2023-10-30T12:29:04.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.540+0000 7f5b46ffd640 1 -- 172.21.15.89:0/398970174 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b400ba400 con 0x7f5b501538f0 2023-10-30T12:29:04.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.540+0000 7f5b46ffd640 1 -- 172.21.15.89:0/398970174 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b400ba7b0 con 0x7f5b501538f0 2023-10-30T12:29:04.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.540+0000 7f5b572dc640 1 -- 172.21.15.89:0/398970174 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b501efde0 con 0x7f5b501538f0 2023-10-30T12:29:04.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.540+0000 7f5b572dc640 1 -- 172.21.15.89:0/398970174 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b5015caa0 con 0x7f5b501538f0 2023-10-30T12:29:04.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.540+0000 7f5b572dc640 1 -- 172.21.15.89:0/398970174 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b5004e170 con 0x7f5b501538f0 2023-10-30T12:29:04.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.540+0000 7f5b46ffd640 1 -- 172.21.15.89:0/398970174 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5b400ba910 con 0x7f5b501538f0 2023-10-30T12:29:04.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.540+0000 7f5b46ffd640 1 --2- 172.21.15.89:0/398970174 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5b340664d0 0x7f5b34068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.540+0000 7f5b46ffd640 1 -- 172.21.15.89:0/398970174 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5b40121c30 con 0x7f5b501538f0 2023-10-30T12:29:04.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.540+0000 7f5b562da640 1 --2- 172.21.15.89:0/398970174 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5b340664d0 0x7f5b34068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.544+0000 7f5b562da640 1 --2- 172.21.15.89:0/398970174 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5b340664d0 0x7f5b34068990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c000f70 tx=0x7f5b4c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:04.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.544+0000 7f5b46ffd640 1 -- 172.21.15.89:0/398970174 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5b400ef0b0 con 0x7f5b501538f0 2023-10-30T12:29:04.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.728+0000 7f5b572dc640 1 -- 172.21.15.89:0/398970174 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5b500b5220 con 0x7f5b501538f0 2023-10-30T12:29:04.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.728+0000 7f5b46ffd640 1 -- 172.21.15.89:0/398970174 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f5b400b5040 con 0x7f5b501538f0 2023-10-30T12:29:04.733 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:29:04.733 INFO:teuthology.orchestra.run.smithi089.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-30T12:29:04.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.728+0000 7f5b572dc640 1 -- 172.21.15.89:0/398970174 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5b340664d0 msgr2=0x7f5b34068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:04.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.728+0000 7f5b572dc640 1 --2- 172.21.15.89:0/398970174 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5b340664d0 0x7f5b34068990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c000f70 tx=0x7f5b4c009040 comp rx=0 tx=0).stop 2023-10-30T12:29:04.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.728+0000 7f5b572dc640 1 -- 172.21.15.89:0/398970174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b501538f0 msgr2=0x7f5b500b4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:04.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.728+0000 7f5b572dc640 1 --2- 172.21.15.89:0/398970174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b501538f0 0x7f5b500b4450 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5b400b3040 tx=0x7f5b4001bad0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.728+0000 7f5b572dc640 1 -- 172.21.15.89:0/398970174 shutdown_connections 2023-10-30T12:29:04.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.728+0000 7f5b572dc640 1 --2- 172.21.15.89:0/398970174 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5b340664d0 0x7f5b34068990 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.728+0000 7f5b572dc640 1 --2- 172.21.15.89:0/398970174 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b5015cd50 0x7f5b5015acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.728+0000 7f5b572dc640 1 --2- 172.21.15.89:0/398970174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b501538f0 0x7f5b500b4450 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.728+0000 7f5b572dc640 1 --2- 172.21.15.89:0/398970174 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b500cefa0 0x7f5b5015c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.728+0000 7f5b572dc640 1 -- 172.21.15.89:0/398970174 >> 172.21.15.89:0/398970174 conn(0x7f5b5006a640 msgr2=0x7f5b50065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:04.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.728+0000 7f5b572dc640 1 -- 172.21.15.89:0/398970174 shutdown_connections 2023-10-30T12:29:04.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.728+0000 7f5b572dc640 1 -- 172.21.15.89:0/398970174 wait complete. 2023-10-30T12:29:04.753 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-30T12:29:04.754 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:29:04.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.884+0000 7fb0c5980640 1 Processor -- start 2023-10-30T12:29:04.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.884+0000 7fb0c5980640 1 -- start start 2023-10-30T12:29:04.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.888+0000 7fb0c5980640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb0c015d6f0 0x7fb0c015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.888+0000 7fb0c5980640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb0c00cf160 0x7fb0c00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.888+0000 7fb0c5980640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c00cb800 con 0x7fb0c01538a0 2023-10-30T12:29:04.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.888+0000 7fb0c5980640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c00cb330 con 0x7fb0c015d6f0 2023-10-30T12:29:04.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.888+0000 7fb0c5980640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c00cad70 con 0x7fb0c00cf160 2023-10-30T12:29:04.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.888+0000 7fb0b67fc640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb0c015d6f0 0x7fb0c015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.888+0000 7fb0bf7fe640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb0c00cf160 0x7fb0c00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.888+0000 7fb0b67fc640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb0c015d6f0 0x7fb0c015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:59662/0 (socket says 172.21.15.89:59662) 2023-10-30T12:29:04.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.888+0000 7fb0beffd640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fb0c01538a0 0x7fb0c00ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:36172/0 (socket says 172.21.15.89:36172) 2023-10-30T12:29:04.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.888+0000 7fb0b67fc640 1 -- 172.21.15.89:0/1269064570 learned_addr learned my addr 172.21.15.89:0/1269064570 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:04.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.888+0000 7fb0b67fc640 1 -- 172.21.15.89:0/1269064570 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb0c00cf160 msgr2=0x7fb0c00cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:04.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.888+0000 7fb0b67fc640 1 --2- 172.21.15.89:0/1269064570 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb0c00cf160 0x7fb0c00cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.888+0000 7fb0b67fc640 1 -- 172.21.15.89:0/1269064570 >> v1:172.21.15.192:6789/0 conn(0x7fb0c01538a0 legacy=0x7fb0c00ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:04.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.888+0000 7fb0b67fc640 1 -- 172.21.15.89:0/1269064570 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c00cb050 con 0x7fb0c015d6f0 2023-10-30T12:29:04.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.892+0000 7fb0b67fc640 1 --2- 172.21.15.89:0/1269064570 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0c015d6f0 0x7fb0c015a090 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8009870 tx=0x7fb0a801ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd63542312c7cf9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:04.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.892+0000 7fb0be7fc640 1 -- 172.21.15.89:0/1269064570 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a80a8a30 con 0x7fb0c015d6f0 2023-10-30T12:29:04.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.892+0000 7fb0be7fc640 1 -- 172.21.15.89:0/1269064570 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a80a8b90 con 0x7fb0c015d6f0 2023-10-30T12:29:04.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.892+0000 7fb0be7fc640 1 -- 172.21.15.89:0/1269064570 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a80b0440 con 0x7fb0c015d6f0 2023-10-30T12:29:04.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.892+0000 7fb0c5980640 1 -- 172.21.15.89:0/1269064570 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0c015d6f0 msgr2=0x7fb0c015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:04.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.892+0000 7fb0c5980640 1 --2- 172.21.15.89:0/1269064570 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0c015d6f0 0x7fb0c015a090 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8009870 tx=0x7fb0a801ed30 comp rx=0 tx=0).stop 2023-10-30T12:29:04.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.892+0000 7fb0c5980640 1 -- 172.21.15.89:0/1269064570 shutdown_connections 2023-10-30T12:29:04.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.892+0000 7fb0c5980640 1 --2- 172.21.15.89:0/1269064570 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb0c00cf160 0x7fb0c00cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.892+0000 7fb0c5980640 1 --2- 172.21.15.89:0/1269064570 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0c015d6f0 0x7fb0c015a090 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.892+0000 7fb0c5980640 1 -- 172.21.15.89:0/1269064570 >> 172.21.15.89:0/1269064570 conn(0x7fb0c006bfe0 msgr2=0x7fb0c006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:04.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.892+0000 7fb0c5980640 1 -- 172.21.15.89:0/1269064570 shutdown_connections 2023-10-30T12:29:04.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.892+0000 7fb0c5980640 1 -- 172.21.15.89:0/1269064570 wait complete. 2023-10-30T12:29:04.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.892+0000 7fb0c5980640 1 Processor -- start 2023-10-30T12:29:04.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.892+0000 7fb0c5980640 1 -- start start 2023-10-30T12:29:04.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.896+0000 7fb0c5980640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0c00cf160 0x7fb0c01f3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.896+0000 7fb0c5980640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0c01538a0 0x7fb0c01efe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.896+0000 7fb0c5980640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb0c015d6f0 0x7fb0c00b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.896+0000 7fb0c5980640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c01589e0 con 0x7fb0c01538a0 2023-10-30T12:29:04.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.896+0000 7fb0c5980640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c01f3840 con 0x7fb0c00cf160 2023-10-30T12:29:04.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.896+0000 7fb0c5980640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c01f0340 con 0x7fb0c015d6f0 2023-10-30T12:29:04.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.896+0000 7fb0b67fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0c01538a0 0x7fb0c01efe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.896+0000 7fb0b67fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0c01538a0 0x7fb0c01efe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39348/0 (socket says 172.21.15.89:39348) 2023-10-30T12:29:04.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.896+0000 7fb0b67fc640 1 -- 172.21.15.89:0/1600085949 learned_addr learned my addr 172.21.15.89:0/1600085949 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:04.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.896+0000 7fb0bf7fe640 1 --2- 172.21.15.89:0/1600085949 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb0c015d6f0 0x7fb0c00b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.896+0000 7fb0beffd640 1 --2- 172.21.15.89:0/1600085949 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0c00cf160 0x7fb0c01f3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.896+0000 7fb0b67fc640 1 -- 172.21.15.89:0/1600085949 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb0c015d6f0 msgr2=0x7fb0c00b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:04.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.896+0000 7fb0b67fc640 1 --2- 172.21.15.89:0/1600085949 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb0c015d6f0 0x7fb0c00b43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.896+0000 7fb0b67fc640 1 -- 172.21.15.89:0/1600085949 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0c00cf160 msgr2=0x7fb0c01f3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:04.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.896+0000 7fb0b67fc640 1 --2- 172.21.15.89:0/1600085949 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0c00cf160 0x7fb0c01f3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.896+0000 7fb0b67fc640 1 -- 172.21.15.89:0/1600085949 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a80094d0 con 0x7fb0c01538a0 2023-10-30T12:29:04.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.896+0000 7fb0b67fc640 1 --2- 172.21.15.89:0/1600085949 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0c01538a0 0x7fb0c01efe00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8000c10 tx=0x7fb0a801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:04.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.896+0000 7fb0bcff9640 1 -- 172.21.15.89:0/1600085949 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a80bf070 con 0x7fb0c01538a0 2023-10-30T12:29:04.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.896+0000 7fb0bcff9640 1 -- 172.21.15.89:0/1600085949 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a80b1be0 con 0x7fb0c01538a0 2023-10-30T12:29:04.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.896+0000 7fb0c5980640 1 -- 172.21.15.89:0/1600085949 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c01ef380 con 0x7fb0c01538a0 2023-10-30T12:29:04.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.896+0000 7fb0bcff9640 1 -- 172.21.15.89:0/1600085949 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a80b9410 con 0x7fb0c01538a0 2023-10-30T12:29:04.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.896+0000 7fb0c5980640 1 -- 172.21.15.89:0/1600085949 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c0158f40 con 0x7fb0c01538a0 2023-10-30T12:29:04.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.896+0000 7fb0c5980640 1 -- 172.21.15.89:0/1600085949 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0c00b2a50 con 0x7fb0c01538a0 2023-10-30T12:29:04.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.900+0000 7fb0bcff9640 1 -- 172.21.15.89:0/1600085949 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb0a80b7040 con 0x7fb0c01538a0 2023-10-30T12:29:04.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.900+0000 7fb0bcff9640 1 --2- 172.21.15.89:0/1600085949 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb0a00664a0 0x7fb0a0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.900+0000 7fb0bcff9640 1 -- 172.21.15.89:0/1600085949 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb0a8121400 con 0x7fb0c01538a0 2023-10-30T12:29:04.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.900+0000 7fb0beffd640 1 --2- 172.21.15.89:0/1600085949 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb0a00664a0 0x7fb0a0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.900+0000 7fb0beffd640 1 --2- 172.21.15.89:0/1600085949 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb0a00664a0 0x7fb0a0068960 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0007fd0 tx=0x7fb0b0005b80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:04.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:04.904+0000 7fb0bcff9640 1 -- 172.21.15.89:0/1600085949 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb0a80ee750 con 0x7fb0c01538a0 2023-10-30T12:29:05.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.056+0000 7fb0c5980640 1 -- 172.21.15.89:0/1600085949 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0c0159840 con 0x7fb0c01538a0 2023-10-30T12:29:05.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.056+0000 7fb0bcff9640 1 -- 172.21.15.89:0/1600085949 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16505 (secure 0 0 0) 0x7fb0a80b5090 con 0x7fb0c01538a0 2023-10-30T12:29:05.062 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:29:05.063 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":22,"fsid":"401641be-db6e-4c16-b3ac-fd84a86c4de4","created":"2023-10-30T12:28:26.198532+0000","modified":"2023-10-30T12:28:51.918285+0000","last_up_change":"2023-10-30T12:28:37.212205+0000","last_in_change":"2023-10-30T12:28:28.517734+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:39.004943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T12:28:48.935909+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T12:28:49.716098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72595c54-fd6e-4a5a-bb14-ff27e9377b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6801","nonce":1673955622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6803","nonce":1673955622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6807","nonce":1673955622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6805","nonce":1673955622}]},"public_addr":"172.21.15.89:6801/1673955622","cluster_addr":"172.21.15.89:6803/1673955622","heartbeat_back_addr":"172.21.15.89:6807/1673955622","heartbeat_front_addr":"172.21.15.89:6805/1673955622","state":["exists","up"]},{"osd":1,"uuid":"0305670b-d7c4-4193-bec2-8b7351ff590a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6817","nonce":3340572039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6819","nonce":3340572039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6823","nonce":3340572039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6821","nonce":3340572039}]},"public_addr":"172.21.15.89:6817/3340572039","cluster_addr":"172.21.15.89:6819/3340572039","heartbeat_back_addr":"172.21.15.89:6823/3340572039","heartbeat_front_addr":"172.21.15.89:6821/3340572039","state":["exists","up"]},{"osd":2,"uuid":"4bab5426-a3c3-46b0-91fc-546c254006c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6809","nonce":1929940799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6811","nonce":1929940799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6815","nonce":1929940799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6813","nonce":1929940799}]},"public_addr":"172.21.15.89:6809/1929940799","cluster_addr":"172.21.15.89:6811/1929940799","heartbeat_back_addr":"172.21.15.89:6815/1929940799","heartbeat_front_addr":"172.21.15.89:6813/1929940799","state":["exists","up"]},{"osd":3,"uuid":"7d47aea8-27cd-42be-8591-d63ad68c90e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6825","nonce":345087939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6827","nonce":345087939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6831","nonce":345087939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6829","nonce":345087939}]},"public_addr":"172.21.15.89:6825/345087939","cluster_addr":"172.21.15.89:6827/345087939","heartbeat_back_addr":"172.21.15.89:6831/345087939","heartbeat_front_addr":"172.21.15.89:6829/345087939","state":["exists","up"]},{"osd":4,"uuid":"2e2ae8f8-d097-4bba-9448-95047fbfacad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6801","nonce":2138125545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6803","nonce":2138125545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6809","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6811","nonce":2138125545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6805","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6807","nonce":2138125545}]},"public_addr":"172.21.15.192:6801/2138125545","cluster_addr":"172.21.15.192:6803/2138125545","heartbeat_back_addr":"172.21.15.192:6811/2138125545","heartbeat_front_addr":"172.21.15.192:6807/2138125545","state":["exists","up"]},{"osd":5,"uuid":"49be5ce5-12f4-4c26-8d39-864b6ff67313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6806","nonce":2360218671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6810","nonce":2360218671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6815","nonce":2360218671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6813","nonce":2360218671}]},"public_addr":"172.21.15.192:6806/2360218671","cluster_addr":"172.21.15.192:6810/2360218671","heartbeat_back_addr":"172.21.15.192:6815/2360218671","heartbeat_front_addr":"172.21.15.192:6813/2360218671","state":["exists","up"]},{"osd":6,"uuid":"48623452-d456-4ea2-a0de-5fa4b37e6d10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6825","nonce":3707671287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6827","nonce":3707671287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6831","nonce":3707671287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6829","nonce":3707671287}]},"public_addr":"172.21.15.192:6825/3707671287","cluster_addr":"172.21.15.192:6827/3707671287","heartbeat_back_addr":"172.21.15.192:6831/3707671287","heartbeat_front_addr":"172.21.15.192:6829/3707671287","state":["exists","up"]},{"osd":7,"uuid":"ff0a4384-54f8-43ec-a3ce-9c8ae3bf8a94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6817","nonce":3111945871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6819","nonce":3111945871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6823","nonce":3111945871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6821","nonce":3111945871}]},"public_addr":"172.21.15.192:6817/3111945871","cluster_addr":"172.21.15.192:6819/3111945871","heartbeat_back_addr":"172.21.15.192:6823/3111945871","heartbeat_front_addr":"172.21.15.192:6821/3111945871","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.183065+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.248650+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.194576+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.397483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.043648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.186624+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.159674+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.100851+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:29:05.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.060+0000 7fb0c5980640 1 -- 172.21.15.89:0/1600085949 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb0a00664a0 msgr2=0x7fb0a0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:05.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.060+0000 7fb0c5980640 1 --2- 172.21.15.89:0/1600085949 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb0a00664a0 0x7fb0a0068960 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0007fd0 tx=0x7fb0b0005b80 comp rx=0 tx=0).stop 2023-10-30T12:29:05.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.060+0000 7fb0c5980640 1 -- 172.21.15.89:0/1600085949 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0c01538a0 msgr2=0x7fb0c01efe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:05.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.060+0000 7fb0c5980640 1 --2- 172.21.15.89:0/1600085949 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0c01538a0 0x7fb0c01efe00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8000c10 tx=0x7fb0a801bad0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.060+0000 7fb0c5980640 1 -- 172.21.15.89:0/1600085949 shutdown_connections 2023-10-30T12:29:05.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.060+0000 7fb0c5980640 1 --2- 172.21.15.89:0/1600085949 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb0a00664a0 0x7fb0a0068960 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.060+0000 7fb0c5980640 1 --2- 172.21.15.89:0/1600085949 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb0c015d6f0 0x7fb0c00b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.060+0000 7fb0c5980640 1 --2- 172.21.15.89:0/1600085949 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0c01538a0 0x7fb0c01efe00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.060+0000 7fb0c5980640 1 --2- 172.21.15.89:0/1600085949 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0c00cf160 0x7fb0c01f3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.060+0000 7fb0c5980640 1 -- 172.21.15.89:0/1600085949 >> 172.21.15.89:0/1600085949 conn(0x7fb0c006bfe0 msgr2=0x7fb0c006c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:05.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.060+0000 7fb0c5980640 1 -- 172.21.15.89:0/1600085949 shutdown_connections 2023-10-30T12:29:05.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.060+0000 7fb0c5980640 1 -- 172.21.15.89:0/1600085949 wait complete. 2023-10-30T12:29:05.083 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T12:28:39.004943+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T12:28:48.935909+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T12:28:49.716098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T12:29:05.084 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T12:29:05.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.220+0000 7f5d37479640 1 Processor -- start 2023-10-30T12:29:05.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.220+0000 7f5d37479640 1 -- start start 2023-10-30T12:29:05.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.220+0000 7f5d37479640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5d30153980 0x7f5d300caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:05.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.220+0000 7f5d37479640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d300cb1e0 0x7f5d3015d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:05.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.220+0000 7f5d37479640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d3015a920 con 0x7f5d30159930 2023-10-30T12:29:05.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.220+0000 7f5d37479640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d300cb930 con 0x7f5d30153980 2023-10-30T12:29:05.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.220+0000 7f5d37479640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d30159dc0 con 0x7f5d300cb1e0 2023-10-30T12:29:05.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.220+0000 7f5d35215640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5d30153980 0x7f5d300caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:05.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.220+0000 7f5d35215640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5d30153980 0x7f5d300caca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39352/0 (socket says 172.21.15.89:39352) 2023-10-30T12:29:05.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.220+0000 7f5d35215640 1 -- 172.21.15.89:0/517270916 learned_addr learned my addr 172.21.15.89:0/517270916 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:05.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.220+0000 7f5d35a16640 1 --2- 172.21.15.89:0/517270916 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d300cb1e0 0x7f5d3015d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:05.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.220+0000 7f5d35215640 1 -- 172.21.15.89:0/517270916 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d300cb1e0 msgr2=0x7f5d3015d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:05.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.220+0000 7f5d35215640 1 --2- 172.21.15.89:0/517270916 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d300cb1e0 0x7f5d3015d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.220+0000 7f5d35215640 1 -- 172.21.15.89:0/517270916 >> v1:172.21.15.192:6789/0 conn(0x7f5d30159930 legacy=0x7f5d300cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:05.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.220+0000 7f5d35215640 1 -- 172.21.15.89:0/517270916 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d3015a080 con 0x7f5d30153980 2023-10-30T12:29:05.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.220+0000 7f5d27fff640 1 -- 172.21.15.89:0/517270916 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536787509 0 0) 0x7f5d3015a920 con 0x7f5d30159930 2023-10-30T12:29:05.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.224+0000 7f5d35215640 1 --2- 172.21.15.89:0/517270916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d30153980 0x7f5d300caca0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c009870 tx=0x7f5d2c01c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a71e2a6b3e6fbc8a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:05.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.224+0000 7f5d27fff640 1 -- 172.21.15.89:0/517270916 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d2c0ab620 con 0x7f5d30153980 2023-10-30T12:29:05.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.224+0000 7f5d27fff640 1 -- 172.21.15.89:0/517270916 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d2c0ab780 con 0x7f5d30153980 2023-10-30T12:29:05.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.224+0000 7f5d27fff640 1 -- 172.21.15.89:0/517270916 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d2c0abb10 con 0x7f5d30153980 2023-10-30T12:29:05.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.224+0000 7f5d37479640 1 -- 172.21.15.89:0/517270916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d30153980 msgr2=0x7f5d300caca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:05.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.224+0000 7f5d37479640 1 --2- 172.21.15.89:0/517270916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d30153980 0x7f5d300caca0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c009870 tx=0x7f5d2c01c510 comp rx=0 tx=0).stop 2023-10-30T12:29:05.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.224+0000 7f5d37479640 1 -- 172.21.15.89:0/517270916 shutdown_connections 2023-10-30T12:29:05.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.224+0000 7f5d37479640 1 --2- 172.21.15.89:0/517270916 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d300cb1e0 0x7f5d3015d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.224+0000 7f5d37479640 1 --2- 172.21.15.89:0/517270916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d30153980 0x7f5d300caca0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.224+0000 7f5d37479640 1 -- 172.21.15.89:0/517270916 >> 172.21.15.89:0/517270916 conn(0x7f5d3006c0f0 msgr2=0x7f5d3006c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:05.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.224+0000 7f5d37479640 1 -- 172.21.15.89:0/517270916 shutdown_connections 2023-10-30T12:29:05.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.224+0000 7f5d37479640 1 -- 172.21.15.89:0/517270916 wait complete. 2023-10-30T12:29:05.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.228+0000 7f5d37479640 1 Processor -- start 2023-10-30T12:29:05.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.228+0000 7f5d37479640 1 -- start start 2023-10-30T12:29:05.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.228+0000 7f5d37479640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5d300cb1e0 0x7f5d300cd250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:05.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.228+0000 7f5d37479640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d30153980 0x7f5d30151e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:05.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.228+0000 7f5d37479640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d300b4cf0 0x7f5d301f5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:05.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.228+0000 7f5d37479640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d300ce0c0 con 0x7f5d300b4cf0 2023-10-30T12:29:05.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.228+0000 7f5d37479640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d300b48f0 con 0x7f5d300cb1e0 2023-10-30T12:29:05.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.228+0000 7f5d37479640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d300cf000 con 0x7f5d30153980 2023-10-30T12:29:05.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.228+0000 7f5d35a16640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d300b4cf0 0x7f5d301f5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:05.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.228+0000 7f5d35a16640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d300b4cf0 0x7f5d301f5130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39356/0 (socket says 172.21.15.89:39356) 2023-10-30T12:29:05.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.228+0000 7f5d35a16640 1 -- 172.21.15.89:0/1363843943 learned_addr learned my addr 172.21.15.89:0/1363843943 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:05.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.228+0000 7f5d35215640 1 --2- 172.21.15.89:0/1363843943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5d300cb1e0 0x7f5d300cd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:05.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.228+0000 7f5d34a14640 1 --2- 172.21.15.89:0/1363843943 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d30153980 0x7f5d30151e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:05.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.228+0000 7f5d35a16640 1 -- 172.21.15.89:0/1363843943 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d30153980 msgr2=0x7f5d30151e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:05.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.228+0000 7f5d35a16640 1 --2- 172.21.15.89:0/1363843943 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d30153980 0x7f5d30151e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.228+0000 7f5d35a16640 1 -- 172.21.15.89:0/1363843943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5d300cb1e0 msgr2=0x7f5d300cd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:05.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.228+0000 7f5d35a16640 1 --2- 172.21.15.89:0/1363843943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5d300cb1e0 0x7f5d300cd250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.228+0000 7f5d35a16640 1 -- 172.21.15.89:0/1363843943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d2c0094d0 con 0x7f5d300b4cf0 2023-10-30T12:29:05.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.228+0000 7f5d34a14640 1 --2- 172.21.15.89:0/1363843943 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d30153980 0x7f5d30151e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:29:05.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.228+0000 7f5d35a16640 1 --2- 172.21.15.89:0/1363843943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d300b4cf0 0x7f5d301f5130 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5d28009dd0 tx=0x7f5d2800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:05.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.228+0000 7f5d25ffb640 1 -- 172.21.15.89:0/1363843943 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d2800ced0 con 0x7f5d300b4cf0 2023-10-30T12:29:05.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.228+0000 7f5d25ffb640 1 -- 172.21.15.89:0/1363843943 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d2800f040 con 0x7f5d300b4cf0 2023-10-30T12:29:05.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.228+0000 7f5d25ffb640 1 -- 172.21.15.89:0/1363843943 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d280146a0 con 0x7f5d300b4cf0 2023-10-30T12:29:05.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.228+0000 7f5d37479640 1 -- 172.21.15.89:0/1363843943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d300ceb20 con 0x7f5d300b4cf0 2023-10-30T12:29:05.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.228+0000 7f5d37479640 1 -- 172.21.15.89:0/1363843943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d30157f40 con 0x7f5d300b4cf0 2023-10-30T12:29:05.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.228+0000 7f5d37479640 1 -- 172.21.15.89:0/1363843943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d3004e170 con 0x7f5d300b4cf0 2023-10-30T12:29:05.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.232+0000 7f5d25ffb640 1 -- 172.21.15.89:0/1363843943 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5d28014800 con 0x7f5d300b4cf0 2023-10-30T12:29:05.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.232+0000 7f5d25ffb640 1 --2- 172.21.15.89:0/1363843943 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5d140664d0 0x7f5d14068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:05.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.232+0000 7f5d25ffb640 1 -- 172.21.15.89:0/1363843943 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5d28084e00 con 0x7f5d300b4cf0 2023-10-30T12:29:05.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.232+0000 7f5d35215640 1 --2- 172.21.15.89:0/1363843943 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5d140664d0 0x7f5d14068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:05.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.232+0000 7f5d35215640 1 --2- 172.21.15.89:0/1363843943 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5d140664d0 0x7f5d14068990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c0af000 tx=0x7f5d2c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:05.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.236+0000 7f5d25ffb640 1 -- 172.21.15.89:0/1363843943 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5d280521e0 con 0x7f5d300b4cf0 2023-10-30T12:29:05.318 INFO:tasks.ceph.mds.f.smithi192.stderr:2023-10-30T12:29:05.315+0000 7f0036e29640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T12:29:05.318 INFO:tasks.ceph.mds.f.smithi192.stderr:2023-10-30T12:29:05.315+0000 7f0036e29640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T12:29:05.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.388+0000 7f5d37479640 1 -- 172.21.15.89:0/1363843943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5d300caa80 con 0x7f5d300b4cf0 2023-10-30T12:29:05.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.388+0000 7f5d25ffb640 1 -- 172.21.15.89:0/1363843943 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5d28055800 con 0x7f5d300b4cf0 2023-10-30T12:29:05.393 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-10-30T12:29:05.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.388+0000 7f5d37479640 1 -- 172.21.15.89:0/1363843943 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5d140664d0 msgr2=0x7f5d14068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:05.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.388+0000 7f5d37479640 1 --2- 172.21.15.89:0/1363843943 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5d140664d0 0x7f5d14068990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c0af000 tx=0x7f5d2c0a7000 comp rx=0 tx=0).stop 2023-10-30T12:29:05.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.388+0000 7f5d37479640 1 -- 172.21.15.89:0/1363843943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d300b4cf0 msgr2=0x7f5d301f5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:05.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.388+0000 7f5d37479640 1 --2- 172.21.15.89:0/1363843943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d300b4cf0 0x7f5d301f5130 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5d28009dd0 tx=0x7f5d2800c490 comp rx=0 tx=0).stop 2023-10-30T12:29:05.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.388+0000 7f5d37479640 1 -- 172.21.15.89:0/1363843943 shutdown_connections 2023-10-30T12:29:05.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.388+0000 7f5d37479640 1 --2- 172.21.15.89:0/1363843943 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5d140664d0 0x7f5d14068990 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.388+0000 7f5d37479640 1 --2- 172.21.15.89:0/1363843943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d300b4cf0 0x7f5d301f5130 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.388+0000 7f5d37479640 1 --2- 172.21.15.89:0/1363843943 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5d30153980 0x7f5d30151e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.388+0000 7f5d37479640 1 --2- 172.21.15.89:0/1363843943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5d300cb1e0 0x7f5d300cd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.388+0000 7f5d37479640 1 -- 172.21.15.89:0/1363843943 >> 172.21.15.89:0/1363843943 conn(0x7f5d3006c0f0 msgr2=0x7f5d3015ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:05.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.388+0000 7f5d37479640 1 -- 172.21.15.89:0/1363843943 shutdown_connections 2023-10-30T12:29:05.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.388+0000 7f5d37479640 1 -- 172.21.15.89:0/1363843943 wait complete. 2023-10-30T12:29:05.414 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T12:29:05.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.548+0000 7f352573e640 1 Processor -- start 2023-10-30T12:29:05.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.548+0000 7f352573e640 1 -- start start 2023-10-30T12:29:05.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.548+0000 7f352573e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3520153980 0x7f35200caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:05.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.548+0000 7f352573e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3520159930 0x7f35200cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:05.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.548+0000 7f352573e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f35200cf1a0 0x7f35200cf630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:05.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.548+0000 7f352573e640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f352015a920 con 0x7f3520159930 2023-10-30T12:29:05.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.548+0000 7f352573e640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35200cb930 con 0x7f35200cf1a0 2023-10-30T12:29:05.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.548+0000 7f352573e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3520159dc0 con 0x7f3520153980 2023-10-30T12:29:05.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.548+0000 7f35167fc640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3520159930 0x7f35200cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:05.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.548+0000 7f35167fc640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3520159930 0x7f35200cb1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39358/0 (socket says 172.21.15.89:39358) 2023-10-30T12:29:05.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.548+0000 7f35167fc640 1 -- 172.21.15.89:0/1505636351 learned_addr learned my addr 172.21.15.89:0/1505636351 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:05.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.548+0000 7f351f7fe640 1 --2- 172.21.15.89:0/1505636351 >> v2:172.21.15.192:3300/0 conn(0x7f35200cf1a0 0x7f35200cf630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:05.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.548+0000 7f351effd640 1 --2- 172.21.15.89:0/1505636351 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3520153980 0x7f35200caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:05.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.548+0000 7f35167fc640 1 -- 172.21.15.89:0/1505636351 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3520153980 msgr2=0x7f35200caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:05.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.548+0000 7f35167fc640 1 --2- 172.21.15.89:0/1505636351 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3520153980 0x7f35200caca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.548+0000 7f35167fc640 1 -- 172.21.15.89:0/1505636351 >> v2:172.21.15.192:3300/0 conn(0x7f35200cf1a0 msgr2=0x7f35200cf630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:05.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.548+0000 7f35167fc640 1 --2- 172.21.15.89:0/1505636351 >> v2:172.21.15.192:3300/0 conn(0x7f35200cf1a0 0x7f35200cf630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.548+0000 7f35167fc640 1 -- 172.21.15.89:0/1505636351 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f352015a080 con 0x7f3520159930 2023-10-30T12:29:05.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.548+0000 7f351effd640 1 --2- 172.21.15.89:0/1505636351 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3520153980 0x7f35200caca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:29:05.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.548+0000 7f351f7fe640 1 --2- 172.21.15.89:0/1505636351 >> v2:172.21.15.192:3300/0 conn(0x7f35200cf1a0 0x7f35200cf630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:05.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.552+0000 7f35167fc640 1 --2- 172.21.15.89:0/1505636351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3520159930 0x7f35200cb1e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3520072820 tx=0x7f351001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=537a14c099e9dd5e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:05.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.552+0000 7f351e7fc640 1 -- 172.21.15.89:0/1505636351 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35100aa5f0 con 0x7f3520159930 2023-10-30T12:29:05.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.552+0000 7f351e7fc640 1 -- 172.21.15.89:0/1505636351 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35100aa750 con 0x7f3520159930 2023-10-30T12:29:05.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.552+0000 7f351e7fc640 1 -- 172.21.15.89:0/1505636351 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35100aaae0 con 0x7f3520159930 2023-10-30T12:29:05.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.552+0000 7f352573e640 1 -- 172.21.15.89:0/1505636351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3520159930 msgr2=0x7f35200cb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:05.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.552+0000 7f352573e640 1 --2- 172.21.15.89:0/1505636351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3520159930 0x7f35200cb1e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3520072820 tx=0x7f351001c4d0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.552+0000 7f352573e640 1 -- 172.21.15.89:0/1505636351 shutdown_connections 2023-10-30T12:29:05.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.552+0000 7f352573e640 1 --2- 172.21.15.89:0/1505636351 >> v2:172.21.15.192:3300/0 conn(0x7f35200cf1a0 0x7f35200cf630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.552+0000 7f352573e640 1 --2- 172.21.15.89:0/1505636351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3520159930 0x7f35200cb1e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.552+0000 7f352573e640 1 --2- 172.21.15.89:0/1505636351 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3520153980 0x7f35200caca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.552+0000 7f352573e640 1 -- 172.21.15.89:0/1505636351 >> 172.21.15.89:0/1505636351 conn(0x7f352006c0f0 msgr2=0x7f352006c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:05.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.552+0000 7f352573e640 1 -- 172.21.15.89:0/1505636351 shutdown_connections 2023-10-30T12:29:05.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.552+0000 7f352573e640 1 -- 172.21.15.89:0/1505636351 wait complete. 2023-10-30T12:29:05.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.556+0000 7f352573e640 1 Processor -- start 2023-10-30T12:29:05.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.556+0000 7f352573e640 1 -- start start 2023-10-30T12:29:05.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.556+0000 7f352573e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35200cf1a0 0x7f352015b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:05.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.556+0000 7f352573e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3520153980 0x7f3520140810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:05.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.556+0000 7f352573e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f352015c9d0 0x7f35201f5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:05.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.556+0000 7f352573e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35201404a0 con 0x7f35200cf1a0 2023-10-30T12:29:05.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.556+0000 7f352573e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35201512c0 con 0x7f3520153980 2023-10-30T12:29:05.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.556+0000 7f352573e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3520151560 con 0x7f352015c9d0 2023-10-30T12:29:05.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.556+0000 7f351effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35200cf1a0 0x7f352015b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:05.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.556+0000 7f351effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35200cf1a0 0x7f352015b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39370/0 (socket says 172.21.15.89:39370) 2023-10-30T12:29:05.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.556+0000 7f351effd640 1 -- 172.21.15.89:0/3280140896 learned_addr learned my addr 172.21.15.89:0/3280140896 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:05.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.556+0000 7f351f7fe640 1 --2- 172.21.15.89:0/3280140896 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f352015c9d0 0x7f35201f5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:05.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.556+0000 7f35167fc640 1 --2- 172.21.15.89:0/3280140896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3520153980 0x7f3520140810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:05.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.556+0000 7f351effd640 1 -- 172.21.15.89:0/3280140896 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f352015c9d0 msgr2=0x7f35201f5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:05.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.556+0000 7f351effd640 1 --2- 172.21.15.89:0/3280140896 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f352015c9d0 0x7f35201f5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.556+0000 7f351effd640 1 -- 172.21.15.89:0/3280140896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3520153980 msgr2=0x7f3520140810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:05.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.556+0000 7f351effd640 1 --2- 172.21.15.89:0/3280140896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3520153980 0x7f3520140810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.556+0000 7f351effd640 1 -- 172.21.15.89:0/3280140896 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35100094d0 con 0x7f35200cf1a0 2023-10-30T12:29:05.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.556+0000 7f351f7fe640 1 --2- 172.21.15.89:0/3280140896 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f352015c9d0 0x7f35201f5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:29:05.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.556+0000 7f351effd640 1 --2- 172.21.15.89:0/3280140896 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35200cf1a0 0x7f352015b830 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f350800c7c0 tx=0x7f350800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:05.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.556+0000 7f3517fff640 1 -- 172.21.15.89:0/3280140896 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3508007a30 con 0x7f35200cf1a0 2023-10-30T12:29:05.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.556+0000 7f3517fff640 1 -- 172.21.15.89:0/3280140896 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3508007b90 con 0x7f35200cf1a0 2023-10-30T12:29:05.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.556+0000 7f3517fff640 1 -- 172.21.15.89:0/3280140896 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350800ce70 con 0x7f35200cf1a0 2023-10-30T12:29:05.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.556+0000 7f352573e640 1 -- 172.21.15.89:0/3280140896 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3520151820 con 0x7f35200cf1a0 2023-10-30T12:29:05.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.556+0000 7f352573e640 1 -- 172.21.15.89:0/3280140896 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35201f56d0 con 0x7f35200cf1a0 2023-10-30T12:29:05.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.560+0000 7f352573e640 1 -- 172.21.15.89:0/3280140896 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f352004e170 con 0x7f35200cf1a0 2023-10-30T12:29:05.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.560+0000 7f3517fff640 1 -- 172.21.15.89:0/3280140896 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f35080113f0 con 0x7f35200cf1a0 2023-10-30T12:29:05.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.560+0000 7f3517fff640 1 --2- 172.21.15.89:0/3280140896 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f35000664d0 0x7f3500068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:05.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.560+0000 7f3517fff640 1 -- 172.21.15.89:0/3280140896 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35080842b0 con 0x7f35200cf1a0 2023-10-30T12:29:05.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.560+0000 7f35167fc640 1 --2- 172.21.15.89:0/3280140896 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f35000664d0 0x7f3500068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:05.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.560+0000 7f35167fc640 1 --2- 172.21.15.89:0/3280140896 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f35000664d0 0x7f3500068990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f35100ae000 tx=0x7f35100a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:05.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.564+0000 7f3517fff640 1 -- 172.21.15.89:0/3280140896 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3508051730 con 0x7f35200cf1a0 2023-10-30T12:29:05.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.712+0000 7f352573e640 1 -- 172.21.15.89:0/3280140896 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3520164090 con 0x7f35200cf1a0 2023-10-30T12:29:05.718 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.712+0000 7f3517fff640 1 -- 172.21.15.89:0/3280140896 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3508054d50 con 0x7f35200cf1a0 2023-10-30T12:29:05.718 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-10-30T12:29:05.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.712+0000 7f352573e640 1 -- 172.21.15.89:0/3280140896 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f35000664d0 msgr2=0x7f3500068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:05.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.712+0000 7f352573e640 1 --2- 172.21.15.89:0/3280140896 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f35000664d0 0x7f3500068990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f35100ae000 tx=0x7f35100a7000 comp rx=0 tx=0).stop 2023-10-30T12:29:05.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.712+0000 7f352573e640 1 -- 172.21.15.89:0/3280140896 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35200cf1a0 msgr2=0x7f352015b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:05.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.712+0000 7f352573e640 1 --2- 172.21.15.89:0/3280140896 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35200cf1a0 0x7f352015b830 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f350800c7c0 tx=0x7f350800cc80 comp rx=0 tx=0).stop 2023-10-30T12:29:05.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.712+0000 7f352573e640 1 -- 172.21.15.89:0/3280140896 shutdown_connections 2023-10-30T12:29:05.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.712+0000 7f352573e640 1 --2- 172.21.15.89:0/3280140896 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f35000664d0 0x7f3500068990 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.712+0000 7f352573e640 1 --2- 172.21.15.89:0/3280140896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3520153980 0x7f3520140810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.712+0000 7f352573e640 1 --2- 172.21.15.89:0/3280140896 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f352015c9d0 0x7f35201f5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.712+0000 7f352573e640 1 --2- 172.21.15.89:0/3280140896 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35200cf1a0 0x7f352015b830 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.712+0000 7f352573e640 1 -- 172.21.15.89:0/3280140896 >> 172.21.15.89:0/3280140896 conn(0x7f352006c0f0 msgr2=0x7f3520157580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:05.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.716+0000 7f352573e640 1 -- 172.21.15.89:0/3280140896 shutdown_connections 2023-10-30T12:29:05.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.716+0000 7f352573e640 1 -- 172.21.15.89:0/3280140896 wait complete. 2023-10-30T12:29:05.739 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T12:29:05.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.872+0000 7fc1bc417640 1 Processor -- start 2023-10-30T12:29:05.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.872+0000 7fc1bc417640 1 -- start start 2023-10-30T12:29:05.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.872+0000 7fc1bc417640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc1b4153980 0x7fc1b40caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:05.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.872+0000 7fc1bc417640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1b40cb1e0 0x7fc1b415d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:05.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1bc417640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b415a920 con 0x7fc1b4159930 2023-10-30T12:29:05.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1bc417640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1b40cb930 con 0x7fc1b4153980 2023-10-30T12:29:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1bc417640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b4159dc0 con 0x7fc1b40cb1e0 2023-10-30T12:29:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1b99b2640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc1b4159930 0x7fc1b40cf1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:52234/0 (socket says 172.21.15.89:52234) 2023-10-30T12:29:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1b99b2640 1 -- 172.21.15.89:0/2419463546 learned_addr learned my addr 172.21.15.89:0/2419463546 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:29:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1ba1b3640 1 --2- 172.21.15.89:0/2419463546 >> v2:172.21.15.192:3300/0 conn(0x7fc1b4153980 0x7fc1b40caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:05.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1ba9b4640 1 --2- 172.21.15.89:0/2419463546 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1b40cb1e0 0x7fc1b415d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:05.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1b91b1640 1 -- 172.21.15.89:0/2419463546 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3142789357 0 0) 0x7fc1b415a920 con 0x7fc1b4159930 2023-10-30T12:29:05.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1b91b1640 1 -- 172.21.15.89:0/2419463546 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1a0003530 con 0x7fc1b4159930 2023-10-30T12:29:05.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1b91b1640 1 -- 172.21.15.89:0/2419463546 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3947598095 0 0) 0x7fc1a0003530 con 0x7fc1b4159930 2023-10-30T12:29:05.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1b91b1640 1 -- 172.21.15.89:0/2419463546 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1b40cb1e0 msgr2=0x7fc1b415d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:05.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1b91b1640 1 --2- 172.21.15.89:0/2419463546 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1b40cb1e0 0x7fc1b415d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1b91b1640 1 -- 172.21.15.89:0/2419463546 >> v2:172.21.15.192:3300/0 conn(0x7fc1b4153980 msgr2=0x7fc1b40caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:05.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1b91b1640 1 --2- 172.21.15.89:0/2419463546 >> v2:172.21.15.192:3300/0 conn(0x7fc1b4153980 0x7fc1b40caca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1b91b1640 1 -- 172.21.15.89:0/2419463546 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b415a080 con 0x7fc1b4159930 2023-10-30T12:29:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1b91b1640 1 -- 172.21.15.89:0/2419463546 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7fc1a4001680 con 0x7fc1b4159930 2023-10-30T12:29:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1b91b1640 1 -- 172.21.15.89:0/2419463546 >> v1:172.21.15.89:6789/0 conn(0x7fc1b4159930 legacy=0x7fc1b40cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1b91b1640 1 --2- 172.21.15.89:0/2419463546 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1b40cb1e0 0x7fc1a0003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1b91b1640 1 --2- 172.21.15.89:0/2419463546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1a0003f00 0x7fc1a00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:05.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1b91b1640 1 --2- 172.21.15.89:0/2419463546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1a0004d10 0x7fc1a0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:05.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1b91b1640 1 -- 172.21.15.89:0/2419463546 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b40cb930 con 0x7fc1a0004d10 2023-10-30T12:29:05.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1b91b1640 1 -- 172.21.15.89:0/2419463546 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b4159dc0 con 0x7fc1a0003f00 2023-10-30T12:29:05.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1b91b1640 1 -- 172.21.15.89:0/2419463546 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a000b990 con 0x7fc1b40cb1e0 2023-10-30T12:29:05.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1ba9b4640 1 --2- 172.21.15.89:0/2419463546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1a0004d10 0x7fc1a0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:05.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1b99b2640 1 --2- 172.21.15.89:0/2419463546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1a0003f00 0x7fc1a00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:05.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1ba1b3640 1 --2- 172.21.15.89:0/2419463546 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1b40cb1e0 0x7fc1a0003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:05.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1ba9b4640 1 -- 172.21.15.89:0/2419463546 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1b40cb1e0 msgr2=0x7fc1a0003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:05.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1ba9b4640 1 --2- 172.21.15.89:0/2419463546 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1b40cb1e0 0x7fc1a0003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1ba9b4640 1 -- 172.21.15.89:0/2419463546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1a0003f00 msgr2=0x7fc1a00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:05.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1ba9b4640 1 --2- 172.21.15.89:0/2419463546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1a0003f00 0x7fc1a00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.876+0000 7fc1ba9b4640 1 -- 172.21.15.89:0/2419463546 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a0003530 con 0x7fc1a0004d10 2023-10-30T12:29:05.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.880+0000 7fc1ba9b4640 1 --2- 172.21.15.89:0/2419463546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1a0004d10 0x7fc1a0005190 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac009fc0 tx=0x7fc1ac01cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fe890d11b5627fa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:05.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.880+0000 7fc1b91b1640 1 -- 172.21.15.89:0/2419463546 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac0a5c70 con 0x7fc1a0004d10 2023-10-30T12:29:05.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.880+0000 7fc1b91b1640 1 -- 172.21.15.89:0/2419463546 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1ac00f040 con 0x7fc1a0004d10 2023-10-30T12:29:05.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.880+0000 7fc1b91b1640 1 -- 172.21.15.89:0/2419463546 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac0ad440 con 0x7fc1a0004d10 2023-10-30T12:29:05.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.880+0000 7fc1bc417640 1 -- 172.21.15.89:0/2419463546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1a0004d10 msgr2=0x7fc1a0005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:05.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.880+0000 7fc1bc417640 1 --2- 172.21.15.89:0/2419463546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1a0004d10 0x7fc1a0005190 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac009fc0 tx=0x7fc1ac01cc00 comp rx=0 tx=0).stop 2023-10-30T12:29:05.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.880+0000 7fc1ba9b4640 1 -- 172.21.15.89:0/2419463546 reap_dead start 2023-10-30T12:29:05.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.880+0000 7fc1bc417640 1 -- 172.21.15.89:0/2419463546 shutdown_connections 2023-10-30T12:29:05.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.880+0000 7fc1bc417640 1 -- 172.21.15.89:0/2419463546 >> 172.21.15.89:0/2419463546 conn(0x7fc1b406c0f0 msgr2=0x7fc1b406c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:05.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.880+0000 7fc1bc417640 1 -- 172.21.15.89:0/2419463546 shutdown_connections 2023-10-30T12:29:05.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.880+0000 7fc1bc417640 1 -- 172.21.15.89:0/2419463546 wait complete. 2023-10-30T12:29:05.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.880+0000 7fc1bc417640 1 Processor -- start 2023-10-30T12:29:05.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.880+0000 7fc1bc417640 1 -- start start 2023-10-30T12:29:05.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.884+0000 7fc1bc417640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1a0004d10 0x7fc1b40c72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:05.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.884+0000 7fc1bc417640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1b40c8ce0 0x7fc1b40c9140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:05.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.884+0000 7fc1bc417640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1b4152850 0x7fc1b4152cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:05.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.884+0000 7fc1bc417640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b40c7820 con 0x7fc1b4152850 2023-10-30T12:29:05.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.884+0000 7fc1bc417640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b40c7ba0 con 0x7fc1a0004d10 2023-10-30T12:29:05.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.884+0000 7fc1bc417640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b40c6530 con 0x7fc1b40c8ce0 2023-10-30T12:29:05.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.884+0000 7fc1ba9b4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1b4152850 0x7fc1b4152cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:05.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.884+0000 7fc1ba9b4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1b4152850 0x7fc1b4152cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39378/0 (socket says 172.21.15.89:39378) 2023-10-30T12:29:05.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.884+0000 7fc1ba9b4640 1 -- 172.21.15.89:0/4086251324 learned_addr learned my addr 172.21.15.89:0/4086251324 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:05.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.884+0000 7fc1ba1b3640 1 --2- 172.21.15.89:0/4086251324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1a0004d10 0x7fc1b40c72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:05.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.884+0000 7fc1b99b2640 1 --2- 172.21.15.89:0/4086251324 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1b40c8ce0 0x7fc1b40c9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:05.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.884+0000 7fc1ba9b4640 1 -- 172.21.15.89:0/4086251324 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1b40c8ce0 msgr2=0x7fc1b40c9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:05.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.884+0000 7fc1ba9b4640 1 --2- 172.21.15.89:0/4086251324 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1b40c8ce0 0x7fc1b40c9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.884+0000 7fc1ba9b4640 1 -- 172.21.15.89:0/4086251324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1a0004d10 msgr2=0x7fc1b40c72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:05.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.884+0000 7fc1ba9b4640 1 --2- 172.21.15.89:0/4086251324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1a0004d10 0x7fc1b40c72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:05.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.884+0000 7fc1ba9b4640 1 -- 172.21.15.89:0/4086251324 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b4153210 con 0x7fc1b4152850 2023-10-30T12:29:05.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.884+0000 7fc1ba9b4640 1 --2- 172.21.15.89:0/4086251324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1b4152850 0x7fc1b4152cd0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac0b0040 tx=0x7fc1ac009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:05.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.884+0000 7fc1aaffd640 1 -- 172.21.15.89:0/4086251324 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac0bc070 con 0x7fc1b4152850 2023-10-30T12:29:05.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.884+0000 7fc1aaffd640 1 -- 172.21.15.89:0/4086251324 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1ac0039a0 con 0x7fc1b4152850 2023-10-30T12:29:05.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.884+0000 7fc1bc417640 1 -- 172.21.15.89:0/4086251324 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1b40c7f20 con 0x7fc1b4152850 2023-10-30T12:29:05.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.884+0000 7fc1aaffd640 1 -- 172.21.15.89:0/4086251324 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac003b30 con 0x7fc1b4152850 2023-10-30T12:29:05.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.884+0000 7fc1bc417640 1 -- 172.21.15.89:0/4086251324 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1b40c84e0 con 0x7fc1b4152850 2023-10-30T12:29:05.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.884+0000 7fc1bc417640 1 -- 172.21.15.89:0/4086251324 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1b404e170 con 0x7fc1b4152850 2023-10-30T12:29:05.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.884+0000 7fc1aaffd640 1 -- 172.21.15.89:0/4086251324 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc1ac0b4040 con 0x7fc1b4152850 2023-10-30T12:29:05.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.888+0000 7fc1aaffd640 1 --2- 172.21.15.89:0/4086251324 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc1980664d0 0x7fc198068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:05.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.888+0000 7fc1aaffd640 1 -- 172.21.15.89:0/4086251324 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc1ac11dcd0 con 0x7fc1b4152850 2023-10-30T12:29:05.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.888+0000 7fc1ba1b3640 1 --2- 172.21.15.89:0/4086251324 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc1980664d0 0x7fc198068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:05.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.888+0000 7fc1ba1b3640 1 --2- 172.21.15.89:0/4086251324 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc1980664d0 0x7fc198068990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc1b0009b60 tx=0x7fc1b0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:05.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:05.892+0000 7fc1aaffd640 1 -- 172.21.15.89:0/4086251324 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc1ac0eb320 con 0x7fc1b4152850 2023-10-30T12:29:06.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.040+0000 7fc1bc417640 1 -- 172.21.15.89:0/4086251324 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc1b40caa80 con 0x7fc1b4152850 2023-10-30T12:29:06.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.040+0000 7fc1aaffd640 1 -- 172.21.15.89:0/4086251324 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc1ac0b20a0 con 0x7fc1b4152850 2023-10-30T12:29:06.047 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-10-30T12:29:06.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.044+0000 7fc1bc417640 1 -- 172.21.15.89:0/4086251324 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc1980664d0 msgr2=0x7fc198068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:06.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.044+0000 7fc1bc417640 1 --2- 172.21.15.89:0/4086251324 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc1980664d0 0x7fc198068990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc1b0009b60 tx=0x7fc1b0008040 comp rx=0 tx=0).stop 2023-10-30T12:29:06.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.044+0000 7fc1bc417640 1 -- 172.21.15.89:0/4086251324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1b4152850 msgr2=0x7fc1b4152cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:06.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.044+0000 7fc1bc417640 1 --2- 172.21.15.89:0/4086251324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1b4152850 0x7fc1b4152cd0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac0b0040 tx=0x7fc1ac009750 comp rx=0 tx=0).stop 2023-10-30T12:29:06.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.044+0000 7fc1bc417640 1 -- 172.21.15.89:0/4086251324 shutdown_connections 2023-10-30T12:29:06.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.044+0000 7fc1bc417640 1 --2- 172.21.15.89:0/4086251324 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc1980664d0 0x7fc198068990 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.044+0000 7fc1bc417640 1 --2- 172.21.15.89:0/4086251324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1b4152850 0x7fc1b4152cd0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.044+0000 7fc1bc417640 1 --2- 172.21.15.89:0/4086251324 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc1b40c8ce0 0x7fc1b40c9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.044+0000 7fc1bc417640 1 --2- 172.21.15.89:0/4086251324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc1a0004d10 0x7fc1b40c72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.044+0000 7fc1bc417640 1 -- 172.21.15.89:0/4086251324 >> 172.21.15.89:0/4086251324 conn(0x7fc1b406c0f0 msgr2=0x7fc1b40ce5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:06.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.044+0000 7fc1bc417640 1 -- 172.21.15.89:0/4086251324 shutdown_connections 2023-10-30T12:29:06.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.044+0000 7fc1bc417640 1 -- 172.21.15.89:0/4086251324 wait complete. 2023-10-30T12:29:06.068 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:29:06.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.200+0000 7f6db1646640 1 Processor -- start 2023-10-30T12:29:06.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.200+0000 7f6db1646640 1 -- start start 2023-10-30T12:29:06.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6db1646640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6dac15d6f0 0x7f6dac15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:06.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6db1646640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dac0cb800 con 0x7f6dac0cf160 2023-10-30T12:29:06.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6db1646640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dac0cb330 con 0x7f6dac1538a0 2023-10-30T12:29:06.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6db1646640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dac0cad70 con 0x7f6dac15d6f0 2023-10-30T12:29:06.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6dab7fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6dac0cf160 0x7f6dac164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:52248/0 (socket says 172.21.15.89:52248) 2023-10-30T12:29:06.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6dab7fe640 1 -- 172.21.15.89:0/2604996472 learned_addr learned my addr 172.21.15.89:0/2604996472 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:29:06.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6daa7fc640 1 --2- 172.21.15.89:0/2604996472 >> v2:172.21.15.192:3300/0 conn(0x7f6dac15d6f0 0x7f6dac15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:06.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6da9ffb640 1 -- 172.21.15.89:0/2604996472 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2119916238 0 0) 0x7f6dac0cb800 con 0x7f6dac0cf160 2023-10-30T12:29:06.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6da9ffb640 1 -- 172.21.15.89:0/2604996472 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d94003530 con 0x7f6dac0cf160 2023-10-30T12:29:06.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6da9ffb640 1 -- 172.21.15.89:0/2604996472 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 129708876 0 0) 0x7f6d94003530 con 0x7f6dac0cf160 2023-10-30T12:29:06.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6da9ffb640 1 -- 172.21.15.89:0/2604996472 >> v2:172.21.15.192:3300/0 conn(0x7f6dac15d6f0 msgr2=0x7f6dac15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:06.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6da9ffb640 1 --2- 172.21.15.89:0/2604996472 >> v2:172.21.15.192:3300/0 conn(0x7f6dac15d6f0 0x7f6dac15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6da9ffb640 1 -- 172.21.15.89:0/2604996472 >> v1:172.21.15.192:6789/0 conn(0x7f6dac1538a0 legacy=0x7f6dac0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:06.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6da9ffb640 1 -- 172.21.15.89:0/2604996472 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dac0cb050 con 0x7f6dac0cf160 2023-10-30T12:29:06.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6da9ffb640 1 -- 172.21.15.89:0/2604996472 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f6da4001a00 con 0x7f6dac0cf160 2023-10-30T12:29:06.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6da9ffb640 1 -- 172.21.15.89:0/2604996472 >> v1:172.21.15.89:6789/0 conn(0x7f6dac0cf160 legacy=0x7f6dac164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:06.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6da9ffb640 1 --2- 172.21.15.89:0/2604996472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d94003980 0x7f6d94003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:06.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6da9ffb640 1 --2- 172.21.15.89:0/2604996472 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6d940047d0 0x7f6d94004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:06.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6da9ffb640 1 --2- 172.21.15.89:0/2604996472 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d94005190 0x7f6d9400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:06.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6da9ffb640 1 -- 172.21.15.89:0/2604996472 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dac0cad70 con 0x7f6d94003980 2023-10-30T12:29:06.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6da9ffb640 1 -- 172.21.15.89:0/2604996472 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9400c040 con 0x7f6d94005190 2023-10-30T12:29:06.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6da9ffb640 1 -- 172.21.15.89:0/2604996472 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9400c200 con 0x7f6d940047d0 2023-10-30T12:29:06.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6daa7fc640 1 --2- 172.21.15.89:0/2604996472 >> v2:172.21.15.192:3300/0 conn(0x7f6dac15d6f0 0x7f6dac15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:29:06.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6daaffd640 1 --2- 172.21.15.89:0/2604996472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d94003980 0x7f6d94003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:06.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6dab7fe640 1 --2- 172.21.15.89:0/2604996472 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d94005190 0x7f6d9400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:06.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6daa7fc640 1 --2- 172.21.15.89:0/2604996472 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6d940047d0 0x7f6d94004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:06.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6daaffd640 1 -- 172.21.15.89:0/2604996472 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6d940047d0 msgr2=0x7f6d94004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:06.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6daaffd640 1 --2- 172.21.15.89:0/2604996472 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6d940047d0 0x7f6d94004c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6daaffd640 1 -- 172.21.15.89:0/2604996472 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d94005190 msgr2=0x7f6d9400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:06.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6daaffd640 1 --2- 172.21.15.89:0/2604996472 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d94005190 0x7f6d9400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6daaffd640 1 -- 172.21.15.89:0/2604996472 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d94003530 con 0x7f6d94003980 2023-10-30T12:29:06.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.204+0000 7f6dab7fe640 1 -- 172.21.15.89:0/2604996472 reap_dead start 2023-10-30T12:29:06.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.208+0000 7f6daaffd640 1 --2- 172.21.15.89:0/2604996472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d94003980 0x7f6d94003da0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c001780 tx=0x7f6d9c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3843f2da88eabfac server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:06.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.208+0000 7f6da9ffb640 1 -- 172.21.15.89:0/2604996472 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c0a4ea0 con 0x7f6d94003980 2023-10-30T12:29:06.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.208+0000 7f6da9ffb640 1 -- 172.21.15.89:0/2604996472 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d9c0a7050 con 0x7f6d94003980 2023-10-30T12:29:06.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.208+0000 7f6da9ffb640 1 -- 172.21.15.89:0/2604996472 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c003600 con 0x7f6d94003980 2023-10-30T12:29:06.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.208+0000 7f6db1646640 1 -- 172.21.15.89:0/2604996472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d94003980 msgr2=0x7f6d94003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:06.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.208+0000 7f6db1646640 1 --2- 172.21.15.89:0/2604996472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d94003980 0x7f6d94003da0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c001780 tx=0x7f6d9c01c3e0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.208+0000 7f6db1646640 1 -- 172.21.15.89:0/2604996472 shutdown_connections 2023-10-30T12:29:06.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.208+0000 7f6db1646640 1 --2- 172.21.15.89:0/2604996472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d94003980 0x7f6d94003da0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.208+0000 7f6db1646640 1 -- 172.21.15.89:0/2604996472 >> 172.21.15.89:0/2604996472 conn(0x7f6dac06bfe0 msgr2=0x7f6dac06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:06.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.208+0000 7f6db1646640 1 -- 172.21.15.89:0/2604996472 shutdown_connections 2023-10-30T12:29:06.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.208+0000 7f6db1646640 1 -- 172.21.15.89:0/2604996472 wait complete. 2023-10-30T12:29:06.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.212+0000 7f6db1646640 1 Processor -- start 2023-10-30T12:29:06.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.212+0000 7f6db1646640 1 -- start start 2023-10-30T12:29:06.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.212+0000 7f6db1646640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d94003980 0x7f6dac0ce800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:06.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.212+0000 7f6db1646640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6dac159490 0x7f6dac157c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:06.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.212+0000 7f6db1646640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6dac158140 0x7f6dac1e2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:06.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.212+0000 7f6db1646640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dac1e5950 con 0x7f6d94003980 2023-10-30T12:29:06.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.212+0000 7f6db1646640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dac158df0 con 0x7f6dac158140 2023-10-30T12:29:06.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.212+0000 7f6db1646640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dac1585d0 con 0x7f6dac159490 2023-10-30T12:29:06.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.212+0000 7f6daaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d94003980 0x7f6dac0ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:06.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.212+0000 7f6daaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d94003980 0x7f6dac0ce800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39388/0 (socket says 172.21.15.89:39388) 2023-10-30T12:29:06.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.212+0000 7f6daaffd640 1 -- 172.21.15.89:0/2643179386 learned_addr learned my addr 172.21.15.89:0/2643179386 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:06.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.212+0000 7f6daa7fc640 1 --2- 172.21.15.89:0/2643179386 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6dac159490 0x7f6dac157c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:06.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.212+0000 7f6dab7fe640 1 --2- 172.21.15.89:0/2643179386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6dac158140 0x7f6dac1e2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:06.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.212+0000 7f6daaffd640 1 -- 172.21.15.89:0/2643179386 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6dac159490 msgr2=0x7f6dac157c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:06.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.212+0000 7f6daaffd640 1 --2- 172.21.15.89:0/2643179386 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6dac159490 0x7f6dac157c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.212+0000 7f6daaffd640 1 -- 172.21.15.89:0/2643179386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6dac158140 msgr2=0x7f6dac1e2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:06.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.212+0000 7f6daaffd640 1 --2- 172.21.15.89:0/2643179386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6dac158140 0x7f6dac1e2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.212+0000 7f6daaffd640 1 -- 172.21.15.89:0/2643179386 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dac158b30 con 0x7f6d94003980 2023-10-30T12:29:06.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.212+0000 7f6daaffd640 1 --2- 172.21.15.89:0/2643179386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d94003980 0x7f6dac0ce800 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c005040 tx=0x7f6d94005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:06.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.212+0000 7f6d937fe640 1 -- 172.21.15.89:0/2643179386 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c003dc0 con 0x7f6d94003980 2023-10-30T12:29:06.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.212+0000 7f6d937fe640 1 -- 172.21.15.89:0/2643179386 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d9c0a4ea0 con 0x7f6d94003980 2023-10-30T12:29:06.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.212+0000 7f6d937fe640 1 -- 172.21.15.89:0/2643179386 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c007250 con 0x7f6d94003980 2023-10-30T12:29:06.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.212+0000 7f6db1646640 1 -- 172.21.15.89:0/2643179386 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dac1e4760 con 0x7f6d94003980 2023-10-30T12:29:06.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.212+0000 7f6db1646640 1 -- 172.21.15.89:0/2643179386 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dac1576e0 con 0x7f6d94003980 2023-10-30T12:29:06.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.212+0000 7f6db1646640 1 -- 172.21.15.89:0/2643179386 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dac159870 con 0x7f6d94003980 2023-10-30T12:29:06.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.216+0000 7f6d937fe640 1 -- 172.21.15.89:0/2643179386 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6d9c0073b0 con 0x7f6d94003980 2023-10-30T12:29:06.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.216+0000 7f6d937fe640 1 --2- 172.21.15.89:0/2643179386 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6d980664a0 0x7f6d98068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:06.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.216+0000 7f6d937fe640 1 -- 172.21.15.89:0/2643179386 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6d9c11e320 con 0x7f6d94003980 2023-10-30T12:29:06.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.216+0000 7f6daa7fc640 1 --2- 172.21.15.89:0/2643179386 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6d980664a0 0x7f6d98068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:06.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.216+0000 7f6daa7fc640 1 --2- 172.21.15.89:0/2643179386 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6d980664a0 0x7f6d98068960 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6dac0c4270 tx=0x7f6da0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:06.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.220+0000 7f6d937fe640 1 -- 172.21.15.89:0/2643179386 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6dac159870 con 0x7f6d94003980 2023-10-30T12:29:06.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.404+0000 7f6db1646640 1 -- 172.21.15.89:0/2643179386 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6dac159870 con 0x7f6d94003980 2023-10-30T12:29:06.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.404+0000 7f6d937fe640 1 -- 172.21.15.89:0/2643179386 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f6dac159870 con 0x7f6d94003980 2023-10-30T12:29:06.409 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:29:06.409 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:29:01.934899+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4253":{"gid":4253,"name":"b","rank":2,"incarnation":11,"state":"up:creating","state_seq":1,"addr":"172.21.15.192:6835/1289220491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1289220491},{"type":"v1","addr":"172.21.15.192:6835","nonce":1289220491}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:29:06.410 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 11 2023-10-30T12:29:06.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.404+0000 7f6db1646640 1 -- 172.21.15.89:0/2643179386 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6d980664a0 msgr2=0x7f6d98068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:06.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.404+0000 7f6db1646640 1 --2- 172.21.15.89:0/2643179386 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6d980664a0 0x7f6d98068960 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6dac0c4270 tx=0x7f6da0008040 comp rx=0 tx=0).stop 2023-10-30T12:29:06.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.404+0000 7f6db1646640 1 -- 172.21.15.89:0/2643179386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d94003980 msgr2=0x7f6dac0ce800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:06.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.404+0000 7f6db1646640 1 --2- 172.21.15.89:0/2643179386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d94003980 0x7f6dac0ce800 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c005040 tx=0x7f6d94005800 comp rx=0 tx=0).stop 2023-10-30T12:29:06.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.404+0000 7f6db1646640 1 -- 172.21.15.89:0/2643179386 shutdown_connections 2023-10-30T12:29:06.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.404+0000 7f6db1646640 1 --2- 172.21.15.89:0/2643179386 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6d980664a0 0x7f6d98068960 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.404+0000 7f6db1646640 1 --2- 172.21.15.89:0/2643179386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6dac158140 0x7f6dac1e2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.404+0000 7f6db1646640 1 --2- 172.21.15.89:0/2643179386 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6dac159490 0x7f6dac157c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.404+0000 7f6db1646640 1 --2- 172.21.15.89:0/2643179386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d94003980 0x7f6dac0ce800 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.404+0000 7f6db1646640 1 -- 172.21.15.89:0/2643179386 >> 172.21.15.89:0/2643179386 conn(0x7f6dac06bfe0 msgr2=0x7f6dac065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:06.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.404+0000 7f6db1646640 1 -- 172.21.15.89:0/2643179386 shutdown_connections 2023-10-30T12:29:06.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.404+0000 7f6db1646640 1 -- 172.21.15.89:0/2643179386 wait complete. 2023-10-30T12:29:06.430 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:29:06.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.563+0000 7f5cd359e640 1 Processor -- start 2023-10-30T12:29:06.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.563+0000 7f5cd359e640 1 -- start start 2023-10-30T12:29:06.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.563+0000 7f5cd359e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5cd41538f0 0x7f5cd4159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:06.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.563+0000 7f5cd359e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5cd40cefa0 0x7f5cd40cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:06.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.563+0000 7f5cd359e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cd40ca6a0 con 0x7f5cd415cd50 2023-10-30T12:29:06.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.563+0000 7f5cd359e640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cd415d300 con 0x7f5cd40cefa0 2023-10-30T12:29:06.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.563+0000 7f5cd359e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd415d5e0 con 0x7f5cd41538f0 2023-10-30T12:29:06.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.563+0000 7f5cd2d9d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5cd40cefa0 0x7f5cd40cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:06.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.563+0000 7f5cd2d9d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5cd40cefa0 0x7f5cd40cf3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39396/0 (socket says 172.21.15.89:39396) 2023-10-30T12:29:06.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.563+0000 7f5cd2d9d640 1 -- 172.21.15.89:0/2040507377 learned_addr learned my addr 172.21.15.89:0/2040507377 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:06.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.563+0000 7f5cd259c640 1 --2- 172.21.15.89:0/2040507377 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5cd41538f0 0x7f5cd4159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:06.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.563+0000 7f5cd2d9d640 1 -- 172.21.15.89:0/2040507377 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5cd41538f0 msgr2=0x7f5cd4159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:06.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.563+0000 7f5cd2d9d640 1 --2- 172.21.15.89:0/2040507377 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5cd41538f0 0x7f5cd4159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.563+0000 7f5cd2d9d640 1 -- 172.21.15.89:0/2040507377 >> v1:172.21.15.192:6789/0 conn(0x7f5cd415cd50 legacy=0x7f5cd40c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:06.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.563+0000 7f5cd2d9d640 1 -- 172.21.15.89:0/2040507377 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd415d8c0 con 0x7f5cd40cefa0 2023-10-30T12:29:06.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.567+0000 7f5cd2d9d640 1 --2- 172.21.15.89:0/2040507377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cd40cefa0 0x7f5cd40cf3f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc009e50 tx=0x7f5ccc01f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=237e15644765029c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:06.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.567+0000 7f5cd1d9b640 1 -- 172.21.15.89:0/2040507377 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc0af400 con 0x7f5cd40cefa0 2023-10-30T12:29:06.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.567+0000 7f5cd1d9b640 1 -- 172.21.15.89:0/2040507377 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ccc0ab050 con 0x7f5cd40cefa0 2023-10-30T12:29:06.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.567+0000 7f5cd1d9b640 1 -- 172.21.15.89:0/2040507377 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc0af780 con 0x7f5cd40cefa0 2023-10-30T12:29:06.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.567+0000 7f5cd359e640 1 -- 172.21.15.89:0/2040507377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cd40cefa0 msgr2=0x7f5cd40cf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:06.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.567+0000 7f5cd359e640 1 --2- 172.21.15.89:0/2040507377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cd40cefa0 0x7f5cd40cf3f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc009e50 tx=0x7f5ccc01f1f0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.567+0000 7f5cd359e640 1 -- 172.21.15.89:0/2040507377 shutdown_connections 2023-10-30T12:29:06.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.567+0000 7f5cd359e640 1 --2- 172.21.15.89:0/2040507377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cd40cefa0 0x7f5cd40cf3f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.567+0000 7f5cd359e640 1 --2- 172.21.15.89:0/2040507377 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5cd41538f0 0x7f5cd4159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.567+0000 7f5cd359e640 1 -- 172.21.15.89:0/2040507377 >> 172.21.15.89:0/2040507377 conn(0x7f5cd406a640 msgr2=0x7f5cd4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:06.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.567+0000 7f5cd359e640 1 -- 172.21.15.89:0/2040507377 shutdown_connections 2023-10-30T12:29:06.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.567+0000 7f5cd359e640 1 -- 172.21.15.89:0/2040507377 wait complete. 2023-10-30T12:29:06.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.567+0000 7f5cd359e640 1 Processor -- start 2023-10-30T12:29:06.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.567+0000 7f5cd359e640 1 -- start start 2023-10-30T12:29:06.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.571+0000 7f5cd359e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5cd40cefa0 0x7f5cd4158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:06.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.571+0000 7f5cd359e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cd41538f0 0x7f5cd4142390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:06.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.571+0000 7f5cd359e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5cd415cd50 0x7f5cd4147f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:06.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.571+0000 7f5cd359e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd415bcf0 con 0x7f5cd41538f0 2023-10-30T12:29:06.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.571+0000 7f5cd359e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd41efb30 con 0x7f5cd40cefa0 2023-10-30T12:29:06.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.571+0000 7f5cd359e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd4152210 con 0x7f5cd415cd50 2023-10-30T12:29:06.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.571+0000 7f5ccbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cd41538f0 0x7f5cd4142390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:06.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.571+0000 7f5ccbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cd41538f0 0x7f5cd4142390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39404/0 (socket says 172.21.15.89:39404) 2023-10-30T12:29:06.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.571+0000 7f5ccbfff640 1 -- 172.21.15.89:0/3233670577 learned_addr learned my addr 172.21.15.89:0/3233670577 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:06.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.571+0000 7f5cd2d9d640 1 --2- 172.21.15.89:0/3233670577 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5cd415cd50 0x7f5cd4147f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:06.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.571+0000 7f5cd259c640 1 --2- 172.21.15.89:0/3233670577 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5cd40cefa0 0x7f5cd4158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:06.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.571+0000 7f5ccbfff640 1 -- 172.21.15.89:0/3233670577 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5cd415cd50 msgr2=0x7f5cd4147f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:06.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.571+0000 7f5ccbfff640 1 --2- 172.21.15.89:0/3233670577 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5cd415cd50 0x7f5cd4147f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.571+0000 7f5ccbfff640 1 -- 172.21.15.89:0/3233670577 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5cd40cefa0 msgr2=0x7f5cd4158300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:06.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.571+0000 7f5ccbfff640 1 --2- 172.21.15.89:0/3233670577 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5cd40cefa0 0x7f5cd4158300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.571+0000 7f5ccbfff640 1 -- 172.21.15.89:0/3233670577 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ccc009b30 con 0x7f5cd41538f0 2023-10-30T12:29:06.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.571+0000 7f5cd259c640 1 --2- 172.21.15.89:0/3233670577 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5cd40cefa0 0x7f5cd4158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:06.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.571+0000 7f5ccbfff640 1 --2- 172.21.15.89:0/3233670577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cd41538f0 0x7f5cd4142390 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc00c3c0 tx=0x7f5cbc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:06.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.571+0000 7f5ccaffd640 1 -- 172.21.15.89:0/3233670577 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc0052d0 con 0x7f5cd41538f0 2023-10-30T12:29:06.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.571+0000 7f5ccaffd640 1 -- 172.21.15.89:0/3233670577 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cbc005430 con 0x7f5cd41538f0 2023-10-30T12:29:06.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.571+0000 7f5cd359e640 1 -- 172.21.15.89:0/3233670577 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cd41524f0 con 0x7f5cd41538f0 2023-10-30T12:29:06.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.571+0000 7f5ccaffd640 1 -- 172.21.15.89:0/3233670577 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc003840 con 0x7f5cd41538f0 2023-10-30T12:29:06.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.571+0000 7f5cd359e640 1 -- 172.21.15.89:0/3233670577 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cd41ef7c0 con 0x7f5cd41538f0 2023-10-30T12:29:06.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.571+0000 7f5cd359e640 1 -- 172.21.15.89:0/3233670577 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cd404e170 con 0x7f5cd41538f0 2023-10-30T12:29:06.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.575+0000 7f5ccaffd640 1 -- 172.21.15.89:0/3233670577 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5cbc0039a0 con 0x7f5cd41538f0 2023-10-30T12:29:06.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.575+0000 7f5ccaffd640 1 --2- 172.21.15.89:0/3233670577 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5cb40664d0 0x7f5cb4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:06.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.575+0000 7f5ccaffd640 1 -- 172.21.15.89:0/3233670577 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5cbc083320 con 0x7f5cd41538f0 2023-10-30T12:29:06.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.575+0000 7f5cd259c640 1 --2- 172.21.15.89:0/3233670577 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5cb40664d0 0x7f5cb4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:06.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.575+0000 7f5cd259c640 1 --2- 172.21.15.89:0/3233670577 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5cb40664d0 0x7f5cb4068990 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4007a70 tx=0x7f5cc4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:06.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.579+0000 7f5ccaffd640 1 -- 172.21.15.89:0/3233670577 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5cbc050840 con 0x7f5cd41538f0 2023-10-30T12:29:06.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.727+0000 7f5cd359e640 1 -- 172.21.15.89:0/3233670577 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5cd41464d0 con 0x7f5cd41538f0 2023-10-30T12:29:06.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.731+0000 7f5ccaffd640 1 -- 172.21.15.89:0/3233670577 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f5cbc053e60 con 0x7f5cd41538f0 2023-10-30T12:29:06.737 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:29:06.737 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":22,"fsid":"401641be-db6e-4c16-b3ac-fd84a86c4de4","created":"2023-10-30T12:28:26.198532+0000","modified":"2023-10-30T12:28:51.918285+0000","last_up_change":"2023-10-30T12:28:37.212205+0000","last_in_change":"2023-10-30T12:28:28.517734+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:39.004943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T12:28:48.935909+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T12:28:49.716098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72595c54-fd6e-4a5a-bb14-ff27e9377b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6801","nonce":1673955622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6803","nonce":1673955622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6807","nonce":1673955622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6805","nonce":1673955622}]},"public_addr":"172.21.15.89:6801/1673955622","cluster_addr":"172.21.15.89:6803/1673955622","heartbeat_back_addr":"172.21.15.89:6807/1673955622","heartbeat_front_addr":"172.21.15.89:6805/1673955622","state":["exists","up"]},{"osd":1,"uuid":"0305670b-d7c4-4193-bec2-8b7351ff590a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6817","nonce":3340572039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6819","nonce":3340572039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6823","nonce":3340572039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6821","nonce":3340572039}]},"public_addr":"172.21.15.89:6817/3340572039","cluster_addr":"172.21.15.89:6819/3340572039","heartbeat_back_addr":"172.21.15.89:6823/3340572039","heartbeat_front_addr":"172.21.15.89:6821/3340572039","state":["exists","up"]},{"osd":2,"uuid":"4bab5426-a3c3-46b0-91fc-546c254006c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6809","nonce":1929940799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6811","nonce":1929940799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6815","nonce":1929940799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6813","nonce":1929940799}]},"public_addr":"172.21.15.89:6809/1929940799","cluster_addr":"172.21.15.89:6811/1929940799","heartbeat_back_addr":"172.21.15.89:6815/1929940799","heartbeat_front_addr":"172.21.15.89:6813/1929940799","state":["exists","up"]},{"osd":3,"uuid":"7d47aea8-27cd-42be-8591-d63ad68c90e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6825","nonce":345087939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6827","nonce":345087939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6831","nonce":345087939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6829","nonce":345087939}]},"public_addr":"172.21.15.89:6825/345087939","cluster_addr":"172.21.15.89:6827/345087939","heartbeat_back_addr":"172.21.15.89:6831/345087939","heartbeat_front_addr":"172.21.15.89:6829/345087939","state":["exists","up"]},{"osd":4,"uuid":"2e2ae8f8-d097-4bba-9448-95047fbfacad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6801","nonce":2138125545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6803","nonce":2138125545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6809","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6811","nonce":2138125545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6805","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6807","nonce":2138125545}]},"public_addr":"172.21.15.192:6801/2138125545","cluster_addr":"172.21.15.192:6803/2138125545","heartbeat_back_addr":"172.21.15.192:6811/2138125545","heartbeat_front_addr":"172.21.15.192:6807/2138125545","state":["exists","up"]},{"osd":5,"uuid":"49be5ce5-12f4-4c26-8d39-864b6ff67313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6806","nonce":2360218671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6810","nonce":2360218671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6815","nonce":2360218671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6813","nonce":2360218671}]},"public_addr":"172.21.15.192:6806/2360218671","cluster_addr":"172.21.15.192:6810/2360218671","heartbeat_back_addr":"172.21.15.192:6815/2360218671","heartbeat_front_addr":"172.21.15.192:6813/2360218671","state":["exists","up"]},{"osd":6,"uuid":"48623452-d456-4ea2-a0de-5fa4b37e6d10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6825","nonce":3707671287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6827","nonce":3707671287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6831","nonce":3707671287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6829","nonce":3707671287}]},"public_addr":"172.21.15.192:6825/3707671287","cluster_addr":"172.21.15.192:6827/3707671287","heartbeat_back_addr":"172.21.15.192:6831/3707671287","heartbeat_front_addr":"172.21.15.192:6829/3707671287","state":["exists","up"]},{"osd":7,"uuid":"ff0a4384-54f8-43ec-a3ce-9c8ae3bf8a94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6817","nonce":3111945871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6819","nonce":3111945871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6823","nonce":3111945871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6821","nonce":3111945871}]},"public_addr":"172.21.15.192:6817/3111945871","cluster_addr":"172.21.15.192:6819/3111945871","heartbeat_back_addr":"172.21.15.192:6823/3111945871","heartbeat_front_addr":"172.21.15.192:6821/3111945871","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.183065+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.248650+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.194576+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.397483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.043648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.186624+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.159674+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.100851+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:29:06.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.731+0000 7f5cd359e640 1 -- 172.21.15.89:0/3233670577 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5cb40664d0 msgr2=0x7f5cb4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:06.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.731+0000 7f5cd359e640 1 --2- 172.21.15.89:0/3233670577 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5cb40664d0 0x7f5cb4068990 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4007a70 tx=0x7f5cc4008040 comp rx=0 tx=0).stop 2023-10-30T12:29:06.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.731+0000 7f5cd359e640 1 -- 172.21.15.89:0/3233670577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cd41538f0 msgr2=0x7f5cd4142390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:06.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.731+0000 7f5cd359e640 1 --2- 172.21.15.89:0/3233670577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cd41538f0 0x7f5cd4142390 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc00c3c0 tx=0x7f5cbc00c880 comp rx=0 tx=0).stop 2023-10-30T12:29:06.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.731+0000 7f5cd359e640 1 -- 172.21.15.89:0/3233670577 shutdown_connections 2023-10-30T12:29:06.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.731+0000 7f5cd359e640 1 --2- 172.21.15.89:0/3233670577 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5cb40664d0 0x7f5cb4068990 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.731+0000 7f5cd359e640 1 --2- 172.21.15.89:0/3233670577 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5cd415cd50 0x7f5cd4147f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.731+0000 7f5cd359e640 1 --2- 172.21.15.89:0/3233670577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5cd41538f0 0x7f5cd4142390 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.731+0000 7f5cd359e640 1 --2- 172.21.15.89:0/3233670577 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5cd40cefa0 0x7f5cd4158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.731+0000 7f5cd359e640 1 -- 172.21.15.89:0/3233670577 >> 172.21.15.89:0/3233670577 conn(0x7f5cd406a640 msgr2=0x7f5cd40cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:06.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.731+0000 7f5cd359e640 1 -- 172.21.15.89:0/3233670577 shutdown_connections 2023-10-30T12:29:06.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.731+0000 7f5cd359e640 1 -- 172.21.15.89:0/3233670577 wait complete. 2023-10-30T12:29:06.758 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T12:28:39.004943+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T12:28:48.935909+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T12:28:49.716098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T12:29:06.759 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T12:29:06.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.891+0000 7f321c874640 1 Processor -- start 2023-10-30T12:29:06.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.891+0000 7f321c874640 1 -- start start 2023-10-30T12:29:06.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.895+0000 7f321c874640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3218153c20 0x7f321815a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:06.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.895+0000 7f321c874640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f32180cb4c0 0x7f32180cef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:06.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.895+0000 7f321c874640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f321815a6b0 con 0x7f3218153840 2023-10-30T12:29:06.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.895+0000 7f321c874640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f321815dbc0 con 0x7f32180cb4c0 2023-10-30T12:29:06.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.895+0000 7f321c874640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f321815d730 con 0x7f3218153c20 2023-10-30T12:29:06.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.895+0000 7f3217fff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f32180cb4c0 0x7f32180cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:06.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.895+0000 7f3217fff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f32180cb4c0 0x7f32180cef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39408/0 (socket says 172.21.15.89:39408) 2023-10-30T12:29:06.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.895+0000 7f3217fff640 1 -- 172.21.15.89:0/3866483201 learned_addr learned my addr 172.21.15.89:0/3866483201 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:06.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.895+0000 7f32177fe640 1 --1- 172.21.15.89:0/3866483201 >> v1:172.21.15.89:6789/0 conn(0x7f3218153840 0x7f32180c3eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:52264/0 (socket says 172.21.15.89:52264) 2023-10-30T12:29:06.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.895+0000 7f3216ffd640 1 --2- 172.21.15.89:0/3866483201 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3218153c20 0x7f321815a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:06.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.895+0000 7f3217fff640 1 -- 172.21.15.89:0/3866483201 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3218153c20 msgr2=0x7f321815a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:06.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.895+0000 7f3217fff640 1 --2- 172.21.15.89:0/3866483201 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3218153c20 0x7f321815a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.895+0000 7f3217fff640 1 -- 172.21.15.89:0/3866483201 >> v1:172.21.15.89:6789/0 conn(0x7f3218153840 legacy=0x7f32180c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:06.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.895+0000 7f3217fff640 1 -- 172.21.15.89:0/3866483201 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32180ca8f0 con 0x7f32180cb4c0 2023-10-30T12:29:06.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.895+0000 7f32167fc640 1 -- 172.21.15.89:0/3866483201 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3004208247 0 0) 0x7f3208001120 con 0x7f3218153840 2023-10-30T12:29:06.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.899+0000 7f3217fff640 1 --2- 172.21.15.89:0/3866483201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32180cb4c0 0x7f32180cef40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3210009e50 tx=0x7f321001f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=807980794fdf5d4d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:06.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.899+0000 7f32167fc640 1 -- 172.21.15.89:0/3866483201 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32100af400 con 0x7f32180cb4c0 2023-10-30T12:29:06.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.899+0000 7f32167fc640 1 -- 172.21.15.89:0/3866483201 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32100ab050 con 0x7f32180cb4c0 2023-10-30T12:29:06.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.899+0000 7f32167fc640 1 -- 172.21.15.89:0/3866483201 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32100af780 con 0x7f32180cb4c0 2023-10-30T12:29:06.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.899+0000 7f321c874640 1 -- 172.21.15.89:0/3866483201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32180cb4c0 msgr2=0x7f32180cef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:06.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.899+0000 7f321c874640 1 --2- 172.21.15.89:0/3866483201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32180cb4c0 0x7f32180cef40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3210009e50 tx=0x7f321001f1f0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.899+0000 7f321c874640 1 -- 172.21.15.89:0/3866483201 shutdown_connections 2023-10-30T12:29:06.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.899+0000 7f321c874640 1 --2- 172.21.15.89:0/3866483201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f32180cb4c0 0x7f32180cef40 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.899+0000 7f321c874640 1 --2- 172.21.15.89:0/3866483201 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3218153c20 0x7f321815a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.899+0000 7f321c874640 1 -- 172.21.15.89:0/3866483201 >> 172.21.15.89:0/3866483201 conn(0x7f321806be40 msgr2=0x7f321806c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:06.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.899+0000 7f321c874640 1 -- 172.21.15.89:0/3866483201 shutdown_connections 2023-10-30T12:29:06.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.899+0000 7f321c874640 1 -- 172.21.15.89:0/3866483201 wait complete. 2023-10-30T12:29:06.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.899+0000 7f321c874640 1 Processor -- start 2023-10-30T12:29:06.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.899+0000 7f321c874640 1 -- start start 2023-10-30T12:29:06.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.899+0000 7f321c874640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32180cb4c0 0x7f3218154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:06.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.899+0000 7f321c874640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3218153840 0x7f32181f25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:06.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.899+0000 7f321c874640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3218153c20 0x7f32180c5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:06.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.899+0000 7f321c874640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32180c8b50 con 0x7f3218153c20 2023-10-30T12:29:06.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.899+0000 7f321c874640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32181f2160 con 0x7f32180cb4c0 2023-10-30T12:29:06.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.899+0000 7f321c874640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3218156640 con 0x7f3218153840 2023-10-30T12:29:06.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.903+0000 7f3217fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3218153c20 0x7f32180c5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:06.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.903+0000 7f3217fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3218153c20 0x7f32180c5120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39420/0 (socket says 172.21.15.89:39420) 2023-10-30T12:29:06.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.903+0000 7f3217fff640 1 -- 172.21.15.89:0/1114991956 learned_addr learned my addr 172.21.15.89:0/1114991956 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:06.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.903+0000 7f32177fe640 1 --2- 172.21.15.89:0/1114991956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32180cb4c0 0x7f3218154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:06.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.903+0000 7f3216ffd640 1 --2- 172.21.15.89:0/1114991956 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3218153840 0x7f32181f25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:06.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.903+0000 7f3217fff640 1 -- 172.21.15.89:0/1114991956 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3218153840 msgr2=0x7f32181f25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:06.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.903+0000 7f3217fff640 1 --2- 172.21.15.89:0/1114991956 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3218153840 0x7f32181f25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.903+0000 7f3217fff640 1 -- 172.21.15.89:0/1114991956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32180cb4c0 msgr2=0x7f3218154060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:06.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.903+0000 7f3217fff640 1 --2- 172.21.15.89:0/1114991956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32180cb4c0 0x7f3218154060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:06.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.903+0000 7f3217fff640 1 -- 172.21.15.89:0/1114991956 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3210009b30 con 0x7f3218153c20 2023-10-30T12:29:06.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.903+0000 7f32177fe640 1 --2- 172.21.15.89:0/1114991956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32180cb4c0 0x7f3218154060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:06.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.903+0000 7f3217fff640 1 --2- 172.21.15.89:0/1114991956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3218153c20 0x7f32180c5120 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f321001ce20 tx=0x7f321001bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:06.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.903+0000 7f31fffff640 1 -- 172.21.15.89:0/1114991956 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32100afde0 con 0x7f3218153c20 2023-10-30T12:29:06.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.903+0000 7f31fffff640 1 -- 172.21.15.89:0/1114991956 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3210009510 con 0x7f3218153c20 2023-10-30T12:29:06.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.903+0000 7f321c874640 1 -- 172.21.15.89:0/1114991956 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3218156920 con 0x7f3218153c20 2023-10-30T12:29:06.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.903+0000 7f31fffff640 1 -- 172.21.15.89:0/1114991956 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32100ba660 con 0x7f3218153c20 2023-10-30T12:29:06.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.903+0000 7f321c874640 1 -- 172.21.15.89:0/1114991956 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32181f1b10 con 0x7f3218153c20 2023-10-30T12:29:06.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.903+0000 7f321c874640 1 -- 172.21.15.89:0/1114991956 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f321804e9c0 con 0x7f3218153c20 2023-10-30T12:29:06.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.903+0000 7f31fffff640 1 -- 172.21.15.89:0/1114991956 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f32100b6040 con 0x7f3218153c20 2023-10-30T12:29:06.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.903+0000 7f31fffff640 1 --2- 172.21.15.89:0/1114991956 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f32040664a0 0x7f3204068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:06.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.907+0000 7f31fffff640 1 -- 172.21.15.89:0/1114991956 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3210122200 con 0x7f3218153c20 2023-10-30T12:29:06.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.907+0000 7f32177fe640 1 --2- 172.21.15.89:0/1114991956 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f32040664a0 0x7f3204068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:06.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.907+0000 7f32177fe640 1 --2- 172.21.15.89:0/1114991956 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f32040664a0 0x7f3204068960 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3208000f70 tx=0x7f3208007a60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:06.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:06.907+0000 7f31fffff640 1 -- 172.21.15.89:0/1114991956 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f32100ef670 con 0x7f3218153c20 2023-10-30T12:29:07.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.067+0000 7f321c874640 1 -- 172.21.15.89:0/1114991956 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f32181f22a0 con 0x7f3218153c20 2023-10-30T12:29:07.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.071+0000 7f31fffff640 1 -- 172.21.15.89:0/1114991956 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f32100f2c90 con 0x7f3218153c20 2023-10-30T12:29:07.076 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-10-30T12:29:07.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.071+0000 7f321c874640 1 -- 172.21.15.89:0/1114991956 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f32040664a0 msgr2=0x7f3204068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.071+0000 7f321c874640 1 --2- 172.21.15.89:0/1114991956 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f32040664a0 0x7f3204068960 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3208000f70 tx=0x7f3208007a60 comp rx=0 tx=0).stop 2023-10-30T12:29:07.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.071+0000 7f321c874640 1 -- 172.21.15.89:0/1114991956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3218153c20 msgr2=0x7f32180c5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.071+0000 7f321c874640 1 --2- 172.21.15.89:0/1114991956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3218153c20 0x7f32180c5120 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f321001ce20 tx=0x7f321001bf90 comp rx=0 tx=0).stop 2023-10-30T12:29:07.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.071+0000 7f321c874640 1 -- 172.21.15.89:0/1114991956 shutdown_connections 2023-10-30T12:29:07.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.071+0000 7f321c874640 1 --2- 172.21.15.89:0/1114991956 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f32040664a0 0x7f3204068960 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.071+0000 7f321c874640 1 --2- 172.21.15.89:0/1114991956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3218153c20 0x7f32180c5120 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.071+0000 7f321c874640 1 --2- 172.21.15.89:0/1114991956 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3218153840 0x7f32181f25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.071+0000 7f321c874640 1 --2- 172.21.15.89:0/1114991956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32180cb4c0 0x7f3218154060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.071+0000 7f321c874640 1 -- 172.21.15.89:0/1114991956 >> 172.21.15.89:0/1114991956 conn(0x7f321806be40 msgr2=0x7f321806c220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:07.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.071+0000 7f321c874640 1 -- 172.21.15.89:0/1114991956 shutdown_connections 2023-10-30T12:29:07.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.071+0000 7f321c874640 1 -- 172.21.15.89:0/1114991956 wait complete. 2023-10-30T12:29:07.098 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T12:29:07.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.231+0000 7f3554fd3640 1 Processor -- start 2023-10-30T12:29:07.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.231+0000 7f3554fd3640 1 -- start start 2023-10-30T12:29:07.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.231+0000 7f3554fd3640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f355006fe40 0x7f35500c7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.231+0000 7f3554fd3640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35500c9200 con 0x7f3550151650 2023-10-30T12:29:07.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.231+0000 7f3554fd3640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3550157d80 con 0x7f35500c1f50 2023-10-30T12:29:07.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.231+0000 7f3554fd3640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35500c8ed0 con 0x7f355006fe40 2023-10-30T12:29:07.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.231+0000 7f3547fff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f355006fe40 0x7f35500c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.231+0000 7f3547fff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f355006fe40 0x7f35500c7990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39426/0 (socket says 172.21.15.89:39426) 2023-10-30T12:29:07.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.231+0000 7f3547fff640 1 -- 172.21.15.89:0/209217087 learned_addr learned my addr 172.21.15.89:0/209217087 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:07.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.231+0000 7f354e59c640 1 --1- 172.21.15.89:0/209217087 >> v1:172.21.15.89:6789/0 conn(0x7f3550151650 0x7f3550157670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:52280/0 (socket says 172.21.15.89:52280) 2023-10-30T12:29:07.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.235+0000 7f3547fff640 1 -- 172.21.15.89:0/209217087 >> v1:172.21.15.192:6789/0 conn(0x7f35500c1f50 legacy=0x7f355006f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.235+0000 7f3547fff640 1 -- 172.21.15.89:0/209217087 >> v1:172.21.15.89:6789/0 conn(0x7f3550151650 legacy=0x7f3550157670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.235+0000 7f3547fff640 1 -- 172.21.15.89:0/209217087 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35500c3b10 con 0x7f355006fe40 2023-10-30T12:29:07.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.235+0000 7f354dd9b640 1 -- 172.21.15.89:0/209217087 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2116174616 0 0) 0x7f35500c9200 con 0x7f3550151650 2023-10-30T12:29:07.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.235+0000 7f3547fff640 1 --2- 172.21.15.89:0/209217087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f355006fe40 0x7f35500c7990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3538009870 tx=0x7f353801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aab76875337a49e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:07.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.235+0000 7f354dd9b640 1 -- 172.21.15.89:0/209217087 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35380a8a30 con 0x7f355006fe40 2023-10-30T12:29:07.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.235+0000 7f354dd9b640 1 -- 172.21.15.89:0/209217087 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35380a8b90 con 0x7f355006fe40 2023-10-30T12:29:07.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.235+0000 7f354dd9b640 1 -- 172.21.15.89:0/209217087 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35380b0440 con 0x7f355006fe40 2023-10-30T12:29:07.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.235+0000 7f3554fd3640 1 -- 172.21.15.89:0/209217087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f355006fe40 msgr2=0x7f35500c7990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.235+0000 7f3554fd3640 1 --2- 172.21.15.89:0/209217087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f355006fe40 0x7f35500c7990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3538009870 tx=0x7f353801ed30 comp rx=0 tx=0).stop 2023-10-30T12:29:07.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.235+0000 7f3554fd3640 1 -- 172.21.15.89:0/209217087 shutdown_connections 2023-10-30T12:29:07.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.235+0000 7f3554fd3640 1 --2- 172.21.15.89:0/209217087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f355006fe40 0x7f35500c7990 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.235+0000 7f3554fd3640 1 -- 172.21.15.89:0/209217087 >> 172.21.15.89:0/209217087 conn(0x7f3550069e90 msgr2=0x7f355006a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:07.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.239+0000 7f3554fd3640 1 -- 172.21.15.89:0/209217087 shutdown_connections 2023-10-30T12:29:07.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.239+0000 7f3554fd3640 1 -- 172.21.15.89:0/209217087 wait complete. 2023-10-30T12:29:07.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.239+0000 7f3554fd3640 1 Processor -- start 2023-10-30T12:29:07.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.239+0000 7f3554fd3640 1 -- start start 2023-10-30T12:29:07.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.239+0000 7f3554fd3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f355006fe40 0x7f355006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.239+0000 7f3554fd3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35500c1f50 0x7f35501f2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.239+0000 7f3554fd3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3550151650 0x7f35501f3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.239+0000 7f3554fd3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35501ef230 con 0x7f35500c1f50 2023-10-30T12:29:07.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.239+0000 7f3554fd3640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3550154af0 con 0x7f3550151650 2023-10-30T12:29:07.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.239+0000 7f3554fd3640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35501f2960 con 0x7f355006fe40 2023-10-30T12:29:07.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.239+0000 7f3547fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35500c1f50 0x7f35501f2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.239+0000 7f3547fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35500c1f50 0x7f35501f2df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39432/0 (socket says 172.21.15.89:39432) 2023-10-30T12:29:07.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.239+0000 7f3547fff640 1 -- 172.21.15.89:0/1975965534 learned_addr learned my addr 172.21.15.89:0/1975965534 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:07.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.239+0000 7f354e59c640 1 --2- 172.21.15.89:0/1975965534 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f355006fe40 0x7f355006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.239+0000 7f354ed9d640 1 --2- 172.21.15.89:0/1975965534 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3550151650 0x7f35501f3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.239+0000 7f3547fff640 1 -- 172.21.15.89:0/1975965534 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f355006fe40 msgr2=0x7f355006c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.239+0000 7f3547fff640 1 --2- 172.21.15.89:0/1975965534 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f355006fe40 0x7f355006c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.239+0000 7f3547fff640 1 -- 172.21.15.89:0/1975965534 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3550151650 msgr2=0x7f35501f3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.239+0000 7f3547fff640 1 --2- 172.21.15.89:0/1975965534 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3550151650 0x7f35501f3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.239+0000 7f3547fff640 1 -- 172.21.15.89:0/1975965534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35380094d0 con 0x7f35500c1f50 2023-10-30T12:29:07.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.239+0000 7f3547fff640 1 --2- 172.21.15.89:0/1975965534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35500c1f50 0x7f35501f2df0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f35380b3040 tx=0x7f353801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:07.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.243+0000 7f3546ffd640 1 -- 172.21.15.89:0/1975965534 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35380be070 con 0x7f35500c1f50 2023-10-30T12:29:07.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.243+0000 7f3546ffd640 1 -- 172.21.15.89:0/1975965534 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35380ba400 con 0x7f35500c1f50 2023-10-30T12:29:07.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.243+0000 7f3546ffd640 1 -- 172.21.15.89:0/1975965534 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35380ba7b0 con 0x7f35500c1f50 2023-10-30T12:29:07.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.243+0000 7f3554fd3640 1 -- 172.21.15.89:0/1975965534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3550153f70 con 0x7f35500c1f50 2023-10-30T12:29:07.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.243+0000 7f3554fd3640 1 -- 172.21.15.89:0/1975965534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3550154530 con 0x7f35500c1f50 2023-10-30T12:29:07.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.243+0000 7f3554fd3640 1 -- 172.21.15.89:0/1975965534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f355004e170 con 0x7f35500c1f50 2023-10-30T12:29:07.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.243+0000 7f3546ffd640 1 -- 172.21.15.89:0/1975965534 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f35380ba910 con 0x7f35500c1f50 2023-10-30T12:29:07.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.243+0000 7f3546ffd640 1 --2- 172.21.15.89:0/1975965534 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f35300664d0 0x7f3530068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.243+0000 7f3546ffd640 1 -- 172.21.15.89:0/1975965534 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3538121c70 con 0x7f35500c1f50 2023-10-30T12:29:07.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.243+0000 7f354e59c640 1 --2- 172.21.15.89:0/1975965534 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f35300664d0 0x7f3530068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.247+0000 7f354e59c640 1 --2- 172.21.15.89:0/1975965534 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f35300664d0 0x7f3530068990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3540000f70 tx=0x7f3540007d10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:07.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.247+0000 7f3546ffd640 1 -- 172.21.15.89:0/1975965534 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f35380ef180 con 0x7f35500c1f50 2023-10-30T12:29:07.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.399+0000 7f3554fd3640 1 -- 172.21.15.89:0/1975965534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f35500724d0 con 0x7f35500c1f50 2023-10-30T12:29:07.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.399+0000 7f3546ffd640 1 -- 172.21.15.89:0/1975965534 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f35380f27a0 con 0x7f35500c1f50 2023-10-30T12:29:07.405 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-10-30T12:29:07.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.399+0000 7f3554fd3640 1 -- 172.21.15.89:0/1975965534 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f35300664d0 msgr2=0x7f3530068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.399+0000 7f3554fd3640 1 --2- 172.21.15.89:0/1975965534 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f35300664d0 0x7f3530068990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3540000f70 tx=0x7f3540007d10 comp rx=0 tx=0).stop 2023-10-30T12:29:07.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.399+0000 7f3554fd3640 1 -- 172.21.15.89:0/1975965534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35500c1f50 msgr2=0x7f35501f2df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.399+0000 7f3554fd3640 1 --2- 172.21.15.89:0/1975965534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35500c1f50 0x7f35501f2df0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f35380b3040 tx=0x7f353801bad0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.399+0000 7f3554fd3640 1 -- 172.21.15.89:0/1975965534 shutdown_connections 2023-10-30T12:29:07.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.399+0000 7f3554fd3640 1 --2- 172.21.15.89:0/1975965534 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f35300664d0 0x7f3530068990 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.399+0000 7f3554fd3640 1 --2- 172.21.15.89:0/1975965534 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3550151650 0x7f35501f3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.399+0000 7f3554fd3640 1 --2- 172.21.15.89:0/1975965534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35500c1f50 0x7f35501f2df0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.399+0000 7f3554fd3640 1 --2- 172.21.15.89:0/1975965534 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f355006fe40 0x7f355006c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.399+0000 7f3554fd3640 1 -- 172.21.15.89:0/1975965534 >> 172.21.15.89:0/1975965534 conn(0x7f3550069e90 msgr2=0x7f355006a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:07.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.399+0000 7f3554fd3640 1 -- 172.21.15.89:0/1975965534 shutdown_connections 2023-10-30T12:29:07.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.399+0000 7f3554fd3640 1 -- 172.21.15.89:0/1975965534 wait complete. 2023-10-30T12:29:07.426 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T12:29:07.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.559+0000 7f823313e640 1 Processor -- start 2023-10-30T12:29:07.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.559+0000 7f823313e640 1 -- start start 2023-10-30T12:29:07.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.559+0000 7f823313e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f822c151a50 0x7f822c06c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.559+0000 7f823313e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f822c0c4580 0x7f822c0c33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.559+0000 7f823313e640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f822c0c2960 con 0x7f822c15ab80 2023-10-30T12:29:07.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.559+0000 7f823313e640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f822c0c2f90 con 0x7f822c0c4580 2023-10-30T12:29:07.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.559+0000 7f823313e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f822c06c6e0 con 0x7f822c151a50 2023-10-30T12:29:07.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.563+0000 7f8223fff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f822c0c4580 0x7f822c0c33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.563+0000 7f82316db640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f822c15ab80 0x7f822c06b2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:52288/0 (socket says 172.21.15.89:52288) 2023-10-30T12:29:07.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.563+0000 7f8223fff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f822c0c4580 0x7f822c0c33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39434/0 (socket says 172.21.15.89:39434) 2023-10-30T12:29:07.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.563+0000 7f82316db640 1 -- 172.21.15.89:0/3191886968 learned_addr learned my addr 172.21.15.89:0/3191886968 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:29:07.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.563+0000 7f8230eda640 1 --2- 172.21.15.89:0/3191886968 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f822c151a50 0x7f822c06c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.563+0000 7f82237fe640 1 -- 172.21.15.89:0/3191886968 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3428258734 0 0) 0x7f822c0c2960 con 0x7f822c15ab80 2023-10-30T12:29:07.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.563+0000 7f82237fe640 1 -- 172.21.15.89:0/3191886968 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8214003530 con 0x7f822c15ab80 2023-10-30T12:29:07.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.563+0000 7f8223fff640 1 -- 172.21.15.89:0/3191886968 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f822c151a50 msgr2=0x7f822c06c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:07.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.563+0000 7f8223fff640 1 --2- 172.21.15.89:0/3191886968 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f822c151a50 0x7f822c06c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.563+0000 7f8223fff640 1 -- 172.21.15.89:0/3191886968 >> v1:172.21.15.89:6789/0 conn(0x7f822c15ab80 legacy=0x7f822c06b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.563+0000 7f8223fff640 1 -- 172.21.15.89:0/3191886968 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f822c06e370 con 0x7f822c0c4580 2023-10-30T12:29:07.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.563+0000 7f8230eda640 1 --2- 172.21.15.89:0/3191886968 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f822c151a50 0x7f822c06c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:07.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.563+0000 7f8223fff640 1 --2- 172.21.15.89:0/3191886968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f822c0c4580 0x7f822c0c33b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f821c0097f0 tx=0x7f821c01c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ff6e4b05468c89 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:07.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.563+0000 7f82237fe640 1 -- 172.21.15.89:0/3191886968 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f821c0ab600 con 0x7f822c0c4580 2023-10-30T12:29:07.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.563+0000 7f82237fe640 1 -- 172.21.15.89:0/3191886968 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f821c0a7050 con 0x7f822c0c4580 2023-10-30T12:29:07.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.563+0000 7f82237fe640 1 -- 172.21.15.89:0/3191886968 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f821c0ab980 con 0x7f822c0c4580 2023-10-30T12:29:07.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.567+0000 7f823313e640 1 -- 172.21.15.89:0/3191886968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f822c0c4580 msgr2=0x7f822c0c33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.567+0000 7f823313e640 1 --2- 172.21.15.89:0/3191886968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f822c0c4580 0x7f822c0c33b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f821c0097f0 tx=0x7f821c01c330 comp rx=0 tx=0).stop 2023-10-30T12:29:07.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.567+0000 7f823313e640 1 -- 172.21.15.89:0/3191886968 shutdown_connections 2023-10-30T12:29:07.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.567+0000 7f823313e640 1 --2- 172.21.15.89:0/3191886968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f822c0c4580 0x7f822c0c33b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.567+0000 7f823313e640 1 --2- 172.21.15.89:0/3191886968 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f822c151a50 0x7f822c06c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.567+0000 7f823313e640 1 -- 172.21.15.89:0/3191886968 >> 172.21.15.89:0/3191886968 conn(0x7f822c0c7fe0 msgr2=0x7f822c0c83f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:07.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.567+0000 7f823313e640 1 -- 172.21.15.89:0/3191886968 shutdown_connections 2023-10-30T12:29:07.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.567+0000 7f823313e640 1 -- 172.21.15.89:0/3191886968 wait complete. 2023-10-30T12:29:07.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.567+0000 7f823313e640 1 Processor -- start 2023-10-30T12:29:07.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.567+0000 7f823313e640 1 -- start start 2023-10-30T12:29:07.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.567+0000 7f823313e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f822c0c4580 0x7f822c15a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.567+0000 7f823313e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f822c151a50 0x7f822c14a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.567+0000 7f823313e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f822c15ab80 0x7f822c159160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.567+0000 7f823313e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f822c1494c0 con 0x7f822c15ab80 2023-10-30T12:29:07.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.567+0000 7f823313e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f822c154010 con 0x7f822c0c4580 2023-10-30T12:29:07.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.567+0000 7f823313e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f822c1498e0 con 0x7f822c151a50 2023-10-30T12:29:07.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.567+0000 7f82316db640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f822c15ab80 0x7f822c159160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.567+0000 7f82316db640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f822c15ab80 0x7f822c159160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39442/0 (socket says 172.21.15.89:39442) 2023-10-30T12:29:07.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.567+0000 7f82316db640 1 -- 172.21.15.89:0/1762923303 learned_addr learned my addr 172.21.15.89:0/1762923303 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:07.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.567+0000 7f8230eda640 1 --2- 172.21.15.89:0/1762923303 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f822c0c4580 0x7f822c15a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.567+0000 7f8223fff640 1 --2- 172.21.15.89:0/1762923303 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f822c151a50 0x7f822c14a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.571+0000 7f82316db640 1 -- 172.21.15.89:0/1762923303 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f822c151a50 msgr2=0x7f822c14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.571+0000 7f82316db640 1 --2- 172.21.15.89:0/1762923303 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f822c151a50 0x7f822c14a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.571+0000 7f82316db640 1 -- 172.21.15.89:0/1762923303 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f822c0c4580 msgr2=0x7f822c15a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.571+0000 7f82316db640 1 --2- 172.21.15.89:0/1762923303 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f822c0c4580 0x7f822c15a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.571+0000 7f82316db640 1 -- 172.21.15.89:0/1762923303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f821c009420 con 0x7f822c15ab80 2023-10-30T12:29:07.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.571+0000 7f8223fff640 1 --2- 172.21.15.89:0/1762923303 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f822c151a50 0x7f822c14a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:07.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.571+0000 7f82316db640 1 --2- 172.21.15.89:0/1762923303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f822c15ab80 0x7f822c159160 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f822400caa0 tx=0x7f822400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:07.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.571+0000 7f82217fa640 1 -- 172.21.15.89:0/1762923303 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822400aba0 con 0x7f822c15ab80 2023-10-30T12:29:07.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.571+0000 7f82217fa640 1 -- 172.21.15.89:0/1762923303 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f822400ad00 con 0x7f822c15ab80 2023-10-30T12:29:07.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.571+0000 7f823313e640 1 -- 172.21.15.89:0/1762923303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f822c149170 con 0x7f822c15ab80 2023-10-30T12:29:07.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.571+0000 7f82217fa640 1 -- 172.21.15.89:0/1762923303 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8224012640 con 0x7f822c15ab80 2023-10-30T12:29:07.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.571+0000 7f823313e640 1 -- 172.21.15.89:0/1762923303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f822c1501b0 con 0x7f822c15ab80 2023-10-30T12:29:07.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.571+0000 7f823313e640 1 -- 172.21.15.89:0/1762923303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f822c04e170 con 0x7f822c15ab80 2023-10-30T12:29:07.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.571+0000 7f82217fa640 1 -- 172.21.15.89:0/1762923303 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8224007070 con 0x7f822c15ab80 2023-10-30T12:29:07.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.571+0000 7f82217fa640 1 --2- 172.21.15.89:0/1762923303 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f82180664a0 0x7f8218068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.571+0000 7f82217fa640 1 -- 172.21.15.89:0/1762923303 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8224083ce0 con 0x7f822c15ab80 2023-10-30T12:29:07.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.571+0000 7f8230eda640 1 --2- 172.21.15.89:0/1762923303 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f82180664a0 0x7f8218068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.575+0000 7f8230eda640 1 --2- 172.21.15.89:0/1762923303 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f82180664a0 0x7f8218068960 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f82280045f0 tx=0x7f8228008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:07.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.575+0000 7f82217fa640 1 -- 172.21.15.89:0/1762923303 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8224051200 con 0x7f822c15ab80 2023-10-30T12:29:07.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.727+0000 7f823313e640 1 -- 172.21.15.89:0/1762923303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f822c154fb0 con 0x7f822c15ab80 2023-10-30T12:29:07.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.727+0000 7f82217fa640 1 -- 172.21.15.89:0/1762923303 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f82240170a0 con 0x7f822c15ab80 2023-10-30T12:29:07.736 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-10-30T12:29:07.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.731+0000 7f823313e640 1 -- 172.21.15.89:0/1762923303 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f82180664a0 msgr2=0x7f8218068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.731+0000 7f823313e640 1 --2- 172.21.15.89:0/1762923303 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f82180664a0 0x7f8218068960 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f82280045f0 tx=0x7f8228008040 comp rx=0 tx=0).stop 2023-10-30T12:29:07.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.731+0000 7f823313e640 1 -- 172.21.15.89:0/1762923303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f822c15ab80 msgr2=0x7f822c159160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.731+0000 7f823313e640 1 --2- 172.21.15.89:0/1762923303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f822c15ab80 0x7f822c159160 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f822400caa0 tx=0x7f822400cf60 comp rx=0 tx=0).stop 2023-10-30T12:29:07.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.731+0000 7f823313e640 1 -- 172.21.15.89:0/1762923303 shutdown_connections 2023-10-30T12:29:07.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.731+0000 7f823313e640 1 --2- 172.21.15.89:0/1762923303 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f82180664a0 0x7f8218068960 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.731+0000 7f823313e640 1 --2- 172.21.15.89:0/1762923303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f822c15ab80 0x7f822c159160 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.731+0000 7f823313e640 1 --2- 172.21.15.89:0/1762923303 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f822c151a50 0x7f822c14a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.731+0000 7f823313e640 1 --2- 172.21.15.89:0/1762923303 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f822c0c4580 0x7f822c15a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.731+0000 7f823313e640 1 -- 172.21.15.89:0/1762923303 >> 172.21.15.89:0/1762923303 conn(0x7f822c0c7fe0 msgr2=0x7f822c06b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:07.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.731+0000 7f823313e640 1 -- 172.21.15.89:0/1762923303 shutdown_connections 2023-10-30T12:29:07.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.731+0000 7f823313e640 1 -- 172.21.15.89:0/1762923303 wait complete. 2023-10-30T12:29:07.756 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:29:07.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.887+0000 7f0614646640 1 Processor -- start 2023-10-30T12:29:07.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.887+0000 7f0614646640 1 -- start start 2023-10-30T12:29:07.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.887+0000 7f0614646640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f060c0c5f60 0x7f060c071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.887+0000 7f0614646640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f060c0700b0 con 0x7f060c0c6fe0 2023-10-30T12:29:07.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.887+0000 7f0614646640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f060c0c6530 con 0x7f060c153bf0 2023-10-30T12:29:07.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.887+0000 7f0614646640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f060c0c6810 con 0x7f060c0c5f60 2023-10-30T12:29:07.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.887+0000 7f0611be1640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f060c0c6fe0 0x7f060c0cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:52292/0 (socket says 172.21.15.89:52292) 2023-10-30T12:29:07.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.887+0000 7f0611be1640 1 -- 172.21.15.89:0/138071264 learned_addr learned my addr 172.21.15.89:0/138071264 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:29:07.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.891+0000 7f0612be3640 1 --2- 172.21.15.89:0/138071264 >> v2:172.21.15.192:3300/0 conn(0x7f060c0c5f60 0x7f060c071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.891+0000 7f06113e0640 1 -- 172.21.15.89:0/138071264 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2662042550 0 0) 0x7f060c0700b0 con 0x7f060c0c6fe0 2023-10-30T12:29:07.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.891+0000 7f06113e0640 1 -- 172.21.15.89:0/138071264 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05f8003530 con 0x7f060c0c6fe0 2023-10-30T12:29:07.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.891+0000 7f0612be3640 1 -- 172.21.15.89:0/138071264 >> v1:172.21.15.192:6789/0 conn(0x7f060c153bf0 legacy=0x7f060c0c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.891+0000 7f0612be3640 1 -- 172.21.15.89:0/138071264 >> v1:172.21.15.89:6789/0 conn(0x7f060c0c6fe0 legacy=0x7f060c0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.891+0000 7f0612be3640 1 -- 172.21.15.89:0/138071264 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f060c159b10 con 0x7f060c0c5f60 2023-10-30T12:29:07.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.891+0000 7f0612be3640 1 --2- 172.21.15.89:0/138071264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f060c0c5f60 0x7f060c071260 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0604009d90 tx=0x7f060401f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4d017017fefd5b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:07.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.891+0000 7f06113e0640 1 -- 172.21.15.89:0/138071264 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06040af600 con 0x7f060c0c5f60 2023-10-30T12:29:07.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.895+0000 7f06113e0640 1 -- 172.21.15.89:0/138071264 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06040ab050 con 0x7f060c0c5f60 2023-10-30T12:29:07.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.895+0000 7f06113e0640 1 -- 172.21.15.89:0/138071264 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06040af960 con 0x7f060c0c5f60 2023-10-30T12:29:07.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.895+0000 7f0614646640 1 -- 172.21.15.89:0/138071264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f060c0c5f60 msgr2=0x7f060c071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.895+0000 7f0614646640 1 --2- 172.21.15.89:0/138071264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f060c0c5f60 0x7f060c071260 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0604009d90 tx=0x7f060401f130 comp rx=0 tx=0).stop 2023-10-30T12:29:07.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.895+0000 7f0614646640 1 -- 172.21.15.89:0/138071264 shutdown_connections 2023-10-30T12:29:07.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.895+0000 7f0614646640 1 --2- 172.21.15.89:0/138071264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f060c0c5f60 0x7f060c071260 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.895+0000 7f0614646640 1 -- 172.21.15.89:0/138071264 >> 172.21.15.89:0/138071264 conn(0x7f060c06c6e0 msgr2=0x7f060c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:07.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.895+0000 7f0614646640 1 -- 172.21.15.89:0/138071264 shutdown_connections 2023-10-30T12:29:07.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.895+0000 7f0614646640 1 -- 172.21.15.89:0/138071264 wait complete. 2023-10-30T12:29:07.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.895+0000 7f0614646640 1 Processor -- start 2023-10-30T12:29:07.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.895+0000 7f0614646640 1 -- start start 2023-10-30T12:29:07.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.895+0000 7f0614646640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f060c0c5f60 0x7f060c1544d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.895+0000 7f0614646640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f060c0c6fe0 0x7f060c1f0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.895+0000 7f0614646640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f060c153bf0 0x7f060c1f05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.895+0000 7f0614646640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060c0c4810 con 0x7f060c0c5f60 2023-10-30T12:29:07.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.895+0000 7f0614646640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060c14e150 con 0x7f060c0c6fe0 2023-10-30T12:29:07.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.895+0000 7f0614646640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f060c14e3d0 con 0x7f060c153bf0 2023-10-30T12:29:07.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.895+0000 7f06123e2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f060c0c5f60 0x7f060c1544d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.895+0000 7f06123e2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f060c0c5f60 0x7f060c1544d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39450/0 (socket says 172.21.15.89:39450) 2023-10-30T12:29:07.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.895+0000 7f06123e2640 1 -- 172.21.15.89:0/2357349288 learned_addr learned my addr 172.21.15.89:0/2357349288 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:07.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.895+0000 7f0611be1640 1 --2- 172.21.15.89:0/2357349288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f060c0c6fe0 0x7f060c1f0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.895+0000 7f0612be3640 1 --2- 172.21.15.89:0/2357349288 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f060c153bf0 0x7f060c1f05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.899+0000 7f06123e2640 1 -- 172.21.15.89:0/2357349288 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f060c153bf0 msgr2=0x7f060c1f05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.899+0000 7f06123e2640 1 --2- 172.21.15.89:0/2357349288 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f060c153bf0 0x7f060c1f05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.899+0000 7f06123e2640 1 -- 172.21.15.89:0/2357349288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f060c0c6fe0 msgr2=0x7f060c1f0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.899+0000 7f06123e2640 1 --2- 172.21.15.89:0/2357349288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f060c0c6fe0 0x7f060c1f0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.899+0000 7f06123e2640 1 -- 172.21.15.89:0/2357349288 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0604009a70 con 0x7f060c0c5f60 2023-10-30T12:29:07.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.899+0000 7f06123e2640 1 --2- 172.21.15.89:0/2357349288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f060c0c5f60 0x7f060c1544d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f060800c420 tx=0x7f060800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:07.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.899+0000 7f0602ffd640 1 -- 172.21.15.89:0/2357349288 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06080052d0 con 0x7f060c0c5f60 2023-10-30T12:29:07.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.899+0000 7f0602ffd640 1 -- 172.21.15.89:0/2357349288 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0608005430 con 0x7f060c0c5f60 2023-10-30T12:29:07.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.899+0000 7f0614646640 1 -- 172.21.15.89:0/2357349288 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f060c0c3c00 con 0x7f060c0c5f60 2023-10-30T12:29:07.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.899+0000 7f0602ffd640 1 -- 172.21.15.89:0/2357349288 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0608007260 con 0x7f060c0c5f60 2023-10-30T12:29:07.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.899+0000 7f0614646640 1 -- 172.21.15.89:0/2357349288 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f060c0c41c0 con 0x7f060c0c5f60 2023-10-30T12:29:07.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.899+0000 7f0614646640 1 -- 172.21.15.89:0/2357349288 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f060c04e170 con 0x7f060c0c5f60 2023-10-30T12:29:07.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.899+0000 7f0602ffd640 1 -- 172.21.15.89:0/2357349288 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f06080073c0 con 0x7f060c0c5f60 2023-10-30T12:29:07.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.899+0000 7f0602ffd640 1 --2- 172.21.15.89:0/2357349288 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f05f00664a0 0x7f05f0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.903+0000 7f0602ffd640 1 -- 172.21.15.89:0/2357349288 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0608083270 con 0x7f060c0c5f60 2023-10-30T12:29:07.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.903+0000 7f0611be1640 1 --2- 172.21.15.89:0/2357349288 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f05f00664a0 0x7f05f0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.903+0000 7f0611be1640 1 --2- 172.21.15.89:0/2357349288 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f05f00664a0 0x7f05f0068960 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f060c157470 tx=0x7f05fc007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:07.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:07.903+0000 7f0602ffd640 1 -- 172.21.15.89:0/2357349288 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0608050710 con 0x7f060c0c5f60 2023-10-30T12:29:08.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.087+0000 7f0614646640 1 -- 172.21.15.89:0/2357349288 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f060c06fe90 con 0x7f060c0c5f60 2023-10-30T12:29:08.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.087+0000 7f0602ffd640 1 -- 172.21.15.89:0/2357349288 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6528 (secure 0 0 0) 0x7f0608053d30 con 0x7f060c0c5f60 2023-10-30T12:29:08.096 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:29:08.096 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:29:06.969397+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4253":{"gid":4253,"name":"b","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6835/1289220491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1289220491},{"type":"v1","addr":"172.21.15.192:6835","nonce":1289220491}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:29:08.097 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 12 2023-10-30T12:29:08.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.091+0000 7f0614646640 1 -- 172.21.15.89:0/2357349288 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f05f00664a0 msgr2=0x7f05f0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:08.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.091+0000 7f0614646640 1 --2- 172.21.15.89:0/2357349288 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f05f00664a0 0x7f05f0068960 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f060c157470 tx=0x7f05fc007d40 comp rx=0 tx=0).stop 2023-10-30T12:29:08.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.091+0000 7f0614646640 1 -- 172.21.15.89:0/2357349288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f060c0c5f60 msgr2=0x7f060c1544d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:08.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.091+0000 7f0614646640 1 --2- 172.21.15.89:0/2357349288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f060c0c5f60 0x7f060c1544d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f060800c420 tx=0x7f060800c8e0 comp rx=0 tx=0).stop 2023-10-30T12:29:08.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.091+0000 7f0614646640 1 -- 172.21.15.89:0/2357349288 shutdown_connections 2023-10-30T12:29:08.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.091+0000 7f0614646640 1 --2- 172.21.15.89:0/2357349288 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f05f00664a0 0x7f05f0068960 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:08.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.091+0000 7f0614646640 1 --2- 172.21.15.89:0/2357349288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f060c0c6fe0 0x7f060c1f0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:08.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.091+0000 7f0614646640 1 --2- 172.21.15.89:0/2357349288 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f060c153bf0 0x7f060c1f05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:08.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.091+0000 7f0614646640 1 --2- 172.21.15.89:0/2357349288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f060c0c5f60 0x7f060c1544d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:08.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.091+0000 7f0614646640 1 -- 172.21.15.89:0/2357349288 >> 172.21.15.89:0/2357349288 conn(0x7f060c06c6e0 msgr2=0x7f060c152350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:08.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.091+0000 7f0614646640 1 -- 172.21.15.89:0/2357349288 shutdown_connections 2023-10-30T12:29:08.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.091+0000 7f0614646640 1 -- 172.21.15.89:0/2357349288 wait complete. 2023-10-30T12:29:08.116 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:29:08.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.247+0000 7f9e9913e640 1 Processor -- start 2023-10-30T12:29:08.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.247+0000 7f9e9913e640 1 -- start start 2023-10-30T12:29:08.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.247+0000 7f9e9913e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9e940c6b80 0x7f9e940bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:08.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.247+0000 7f9e9913e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9e9413f4a0 0x7f9e940c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:08.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.247+0000 7f9e9913e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e940c0a50 con 0x7f9e940c6f60 2023-10-30T12:29:08.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.247+0000 7f9e9913e640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e9406f6e0 con 0x7f9e940c6b80 2023-10-30T12:29:08.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.247+0000 7f9e9913e640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e9406f9c0 con 0x7f9e9413f4a0 2023-10-30T12:29:08.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.247+0000 7f9e92d9d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9e940c6b80 0x7f9e940bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:08.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.247+0000 7f9e92d9d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9e940c6b80 0x7f9e940bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39460/0 (socket says 172.21.15.89:39460) 2023-10-30T12:29:08.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.247+0000 7f9e92d9d640 1 -- 172.21.15.89:0/890549649 learned_addr learned my addr 172.21.15.89:0/890549649 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:08.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.247+0000 7f9e9359e640 1 --2- 172.21.15.89:0/890549649 >> v2:172.21.15.192:3300/0 conn(0x7f9e9413f4a0 0x7f9e940c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:08.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.247+0000 7f9e92d9d640 1 -- 172.21.15.89:0/890549649 >> v2:172.21.15.192:3300/0 conn(0x7f9e9413f4a0 msgr2=0x7f9e940c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:08.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.247+0000 7f9e92d9d640 1 --2- 172.21.15.89:0/890549649 >> v2:172.21.15.192:3300/0 conn(0x7f9e9413f4a0 0x7f9e940c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:08.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.247+0000 7f9e92d9d640 1 -- 172.21.15.89:0/890549649 >> v1:172.21.15.192:6789/0 conn(0x7f9e940c6f60 legacy=0x7f9e9413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:08.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.247+0000 7f9e92d9d640 1 -- 172.21.15.89:0/890549649 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e9406fca0 con 0x7f9e940c6b80 2023-10-30T12:29:08.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.251+0000 7f9e92d9d640 1 --2- 172.21.15.89:0/890549649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e940c6b80 0x7f9e940bfec0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9e940c8980 tx=0x7f9e8401ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b21ff886d96dafed server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:08.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.251+0000 7f9e9259c640 1 -- 172.21.15.89:0/890549649 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e840a7bb0 con 0x7f9e940c6b80 2023-10-30T12:29:08.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.251+0000 7f9e9259c640 1 -- 172.21.15.89:0/890549649 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e840a7d10 con 0x7f9e940c6b80 2023-10-30T12:29:08.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.251+0000 7f9e9259c640 1 -- 172.21.15.89:0/890549649 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e840af660 con 0x7f9e940c6b80 2023-10-30T12:29:08.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.251+0000 7f9e9913e640 1 -- 172.21.15.89:0/890549649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e940c6b80 msgr2=0x7f9e940bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:08.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.251+0000 7f9e9913e640 1 --2- 172.21.15.89:0/890549649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e940c6b80 0x7f9e940bfec0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9e940c8980 tx=0x7f9e8401ebd0 comp rx=0 tx=0).stop 2023-10-30T12:29:08.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.251+0000 7f9e9913e640 1 -- 172.21.15.89:0/890549649 shutdown_connections 2023-10-30T12:29:08.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.251+0000 7f9e9913e640 1 --2- 172.21.15.89:0/890549649 >> v2:172.21.15.192:3300/0 conn(0x7f9e9413f4a0 0x7f9e940c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:08.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.251+0000 7f9e9913e640 1 --2- 172.21.15.89:0/890549649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e940c6b80 0x7f9e940bfec0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:08.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.251+0000 7f9e9913e640 1 -- 172.21.15.89:0/890549649 >> 172.21.15.89:0/890549649 conn(0x7f9e940c7910 msgr2=0x7f9e940c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:08.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.255+0000 7f9e9913e640 1 -- 172.21.15.89:0/890549649 shutdown_connections 2023-10-30T12:29:08.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.255+0000 7f9e9913e640 1 -- 172.21.15.89:0/890549649 wait complete. 2023-10-30T12:29:08.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.255+0000 7f9e9913e640 1 Processor -- start 2023-10-30T12:29:08.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.255+0000 7f9e9913e640 1 -- start start 2023-10-30T12:29:08.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.255+0000 7f9e9913e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e940c6b80 0x7f9e9413d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:08.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.255+0000 7f9e9913e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e940c6f60 0x7f9e9413d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:08.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.255+0000 7f9e9913e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e9413f4a0 0x7f9e941f1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:08.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.255+0000 7f9e9913e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e94155a70 con 0x7f9e940c6b80 2023-10-30T12:29:08.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.255+0000 7f9e9913e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e94156240 con 0x7f9e9413f4a0 2023-10-30T12:29:08.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.255+0000 7f9e9913e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e940c1bf0 con 0x7f9e940c6f60 2023-10-30T12:29:08.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.255+0000 7f9e92d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e940c6b80 0x7f9e9413d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:08.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.255+0000 7f9e92d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e940c6b80 0x7f9e9413d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39464/0 (socket says 172.21.15.89:39464) 2023-10-30T12:29:08.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.255+0000 7f9e92d9d640 1 -- 172.21.15.89:0/1613793348 learned_addr learned my addr 172.21.15.89:0/1613793348 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:08.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.255+0000 7f9e9359e640 1 --2- 172.21.15.89:0/1613793348 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e9413f4a0 0x7f9e941f1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:08.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.255+0000 7f9e8bfff640 1 --2- 172.21.15.89:0/1613793348 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e940c6f60 0x7f9e9413d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:08.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.255+0000 7f9e92d9d640 1 -- 172.21.15.89:0/1613793348 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e940c6f60 msgr2=0x7f9e9413d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:08.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.255+0000 7f9e92d9d640 1 --2- 172.21.15.89:0/1613793348 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e940c6f60 0x7f9e9413d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:08.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.255+0000 7f9e92d9d640 1 -- 172.21.15.89:0/1613793348 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e9413f4a0 msgr2=0x7f9e941f1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:08.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.255+0000 7f9e92d9d640 1 --2- 172.21.15.89:0/1613793348 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e9413f4a0 0x7f9e941f1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:08.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.255+0000 7f9e92d9d640 1 -- 172.21.15.89:0/1613793348 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e8400b770 con 0x7f9e940c6b80 2023-10-30T12:29:08.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.255+0000 7f9e8bfff640 1 --2- 172.21.15.89:0/1613793348 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e940c6f60 0x7f9e9413d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:29:08.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.255+0000 7f9e92d9d640 1 --2- 172.21.15.89:0/1613793348 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e940c6b80 0x7f9e9413d080 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9e8401b7e0 tx=0x7f9e8401b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:08.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.255+0000 7f9e8b7fe640 1 -- 172.21.15.89:0/1613793348 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e84006ee0 con 0x7f9e940c6b80 2023-10-30T12:29:08.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.255+0000 7f9e8b7fe640 1 -- 172.21.15.89:0/1613793348 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e840ba640 con 0x7f9e940c6b80 2023-10-30T12:29:08.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.255+0000 7f9e8b7fe640 1 -- 172.21.15.89:0/1613793348 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e840ba9f0 con 0x7f9e940c6b80 2023-10-30T12:29:08.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.259+0000 7f9e9913e640 1 -- 172.21.15.89:0/1613793348 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e9413cbf0 con 0x7f9e940c6b80 2023-10-30T12:29:08.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.259+0000 7f9e9913e640 1 -- 172.21.15.89:0/1613793348 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e94155c80 con 0x7f9e940c6b80 2023-10-30T12:29:08.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.259+0000 7f9e9913e640 1 -- 172.21.15.89:0/1613793348 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e9413c9e0 con 0x7f9e940c6b80 2023-10-30T12:29:08.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.259+0000 7f9e8b7fe640 1 -- 172.21.15.89:0/1613793348 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9e840b6040 con 0x7f9e940c6b80 2023-10-30T12:29:08.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.259+0000 7f9e8b7fe640 1 --2- 172.21.15.89:0/1613793348 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9e740664a0 0x7f9e74068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:08.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.259+0000 7f9e8b7fe640 1 -- 172.21.15.89:0/1613793348 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9e84121540 con 0x7f9e940c6b80 2023-10-30T12:29:08.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.259+0000 7f9e8bfff640 1 --2- 172.21.15.89:0/1613793348 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9e740664a0 0x7f9e74068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:08.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.263+0000 7f9e8bfff640 1 --2- 172.21.15.89:0/1613793348 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9e740664a0 0x7f9e74068960 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9e940c1af0 tx=0x7f9e7c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:08.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.263+0000 7f9e8b7fe640 1 -- 172.21.15.89:0/1613793348 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9e840ee890 con 0x7f9e940c6b80 2023-10-30T12:29:08.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.415+0000 7f9e9913e640 1 -- 172.21.15.89:0/1613793348 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e9404e170 con 0x7f9e940c6b80 2023-10-30T12:29:08.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.415+0000 7f9e8b7fe640 1 -- 172.21.15.89:0/1613793348 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f9e840b4090 con 0x7f9e940c6b80 2023-10-30T12:29:08.423 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:29:08.423 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":22,"fsid":"401641be-db6e-4c16-b3ac-fd84a86c4de4","created":"2023-10-30T12:28:26.198532+0000","modified":"2023-10-30T12:28:51.918285+0000","last_up_change":"2023-10-30T12:28:37.212205+0000","last_in_change":"2023-10-30T12:28:28.517734+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:39.004943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T12:28:48.935909+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T12:28:49.716098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72595c54-fd6e-4a5a-bb14-ff27e9377b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6801","nonce":1673955622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6803","nonce":1673955622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6807","nonce":1673955622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6805","nonce":1673955622}]},"public_addr":"172.21.15.89:6801/1673955622","cluster_addr":"172.21.15.89:6803/1673955622","heartbeat_back_addr":"172.21.15.89:6807/1673955622","heartbeat_front_addr":"172.21.15.89:6805/1673955622","state":["exists","up"]},{"osd":1,"uuid":"0305670b-d7c4-4193-bec2-8b7351ff590a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6817","nonce":3340572039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6819","nonce":3340572039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6823","nonce":3340572039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6821","nonce":3340572039}]},"public_addr":"172.21.15.89:6817/3340572039","cluster_addr":"172.21.15.89:6819/3340572039","heartbeat_back_addr":"172.21.15.89:6823/3340572039","heartbeat_front_addr":"172.21.15.89:6821/3340572039","state":["exists","up"]},{"osd":2,"uuid":"4bab5426-a3c3-46b0-91fc-546c254006c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6809","nonce":1929940799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6811","nonce":1929940799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6815","nonce":1929940799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6813","nonce":1929940799}]},"public_addr":"172.21.15.89:6809/1929940799","cluster_addr":"172.21.15.89:6811/1929940799","heartbeat_back_addr":"172.21.15.89:6815/1929940799","heartbeat_front_addr":"172.21.15.89:6813/1929940799","state":["exists","up"]},{"osd":3,"uuid":"7d47aea8-27cd-42be-8591-d63ad68c90e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6825","nonce":345087939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6827","nonce":345087939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6831","nonce":345087939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6829","nonce":345087939}]},"public_addr":"172.21.15.89:6825/345087939","cluster_addr":"172.21.15.89:6827/345087939","heartbeat_back_addr":"172.21.15.89:6831/345087939","heartbeat_front_addr":"172.21.15.89:6829/345087939","state":["exists","up"]},{"osd":4,"uuid":"2e2ae8f8-d097-4bba-9448-95047fbfacad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6801","nonce":2138125545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6803","nonce":2138125545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6809","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6811","nonce":2138125545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6805","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6807","nonce":2138125545}]},"public_addr":"172.21.15.192:6801/2138125545","cluster_addr":"172.21.15.192:6803/2138125545","heartbeat_back_addr":"172.21.15.192:6811/2138125545","heartbeat_front_addr":"172.21.15.192:6807/2138125545","state":["exists","up"]},{"osd":5,"uuid":"49be5ce5-12f4-4c26-8d39-864b6ff67313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6806","nonce":2360218671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6810","nonce":2360218671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6815","nonce":2360218671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6813","nonce":2360218671}]},"public_addr":"172.21.15.192:6806/2360218671","cluster_addr":"172.21.15.192:6810/2360218671","heartbeat_back_addr":"172.21.15.192:6815/2360218671","heartbeat_front_addr":"172.21.15.192:6813/2360218671","state":["exists","up"]},{"osd":6,"uuid":"48623452-d456-4ea2-a0de-5fa4b37e6d10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6825","nonce":3707671287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6827","nonce":3707671287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6831","nonce":3707671287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6829","nonce":3707671287}]},"public_addr":"172.21.15.192:6825/3707671287","cluster_addr":"172.21.15.192:6827/3707671287","heartbeat_back_addr":"172.21.15.192:6831/3707671287","heartbeat_front_addr":"172.21.15.192:6829/3707671287","state":["exists","up"]},{"osd":7,"uuid":"ff0a4384-54f8-43ec-a3ce-9c8ae3bf8a94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6817","nonce":3111945871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6819","nonce":3111945871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6823","nonce":3111945871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6821","nonce":3111945871}]},"public_addr":"172.21.15.192:6817/3111945871","cluster_addr":"172.21.15.192:6819/3111945871","heartbeat_back_addr":"172.21.15.192:6823/3111945871","heartbeat_front_addr":"172.21.15.192:6821/3111945871","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.183065+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.248650+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.194576+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.397483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.043648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.186624+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.159674+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.100851+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:29:08.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.419+0000 7f9e9913e640 1 -- 172.21.15.89:0/1613793348 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9e740664a0 msgr2=0x7f9e74068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:08.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.419+0000 7f9e9913e640 1 --2- 172.21.15.89:0/1613793348 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9e740664a0 0x7f9e74068960 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9e940c1af0 tx=0x7f9e7c009040 comp rx=0 tx=0).stop 2023-10-30T12:29:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.419+0000 7f9e9913e640 1 -- 172.21.15.89:0/1613793348 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e940c6b80 msgr2=0x7f9e9413d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.419+0000 7f9e9913e640 1 --2- 172.21.15.89:0/1613793348 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e940c6b80 0x7f9e9413d080 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9e8401b7e0 tx=0x7f9e8401b970 comp rx=0 tx=0).stop 2023-10-30T12:29:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.419+0000 7f9e9913e640 1 -- 172.21.15.89:0/1613793348 shutdown_connections 2023-10-30T12:29:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.419+0000 7f9e9913e640 1 --2- 172.21.15.89:0/1613793348 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9e740664a0 0x7f9e74068960 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.419+0000 7f9e9913e640 1 --2- 172.21.15.89:0/1613793348 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e9413f4a0 0x7f9e941f1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.419+0000 7f9e9913e640 1 --2- 172.21.15.89:0/1613793348 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e940c6f60 0x7f9e9413d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.419+0000 7f9e9913e640 1 --2- 172.21.15.89:0/1613793348 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e940c6b80 0x7f9e9413d080 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.419+0000 7f9e9913e640 1 -- 172.21.15.89:0/1613793348 >> 172.21.15.89:0/1613793348 conn(0x7f9e940c7910 msgr2=0x7f9e94069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.419+0000 7f9e9913e640 1 -- 172.21.15.89:0/1613793348 shutdown_connections 2023-10-30T12:29:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.419+0000 7f9e9913e640 1 -- 172.21.15.89:0/1613793348 wait complete. 2023-10-30T12:29:08.444 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:29:08.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.575+0000 7fbb5bb01640 1 Processor -- start 2023-10-30T12:29:08.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.575+0000 7fbb5bb01640 1 -- start start 2023-10-30T12:29:08.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.575+0000 7fbb5bb01640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb541538a0 0x7fbb5415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:08.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.575+0000 7fbb5bb01640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbb540ca0e0 0x7fbb5415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:08.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.575+0000 7fbb5bb01640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb540cb800 con 0x7fbb540cf160 2023-10-30T12:29:08.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.575+0000 7fbb5bb01640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb5415dc30 con 0x7fbb540ca0e0 2023-10-30T12:29:08.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.575+0000 7fbb5bb01640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb540cb330 con 0x7fbb541538a0 2023-10-30T12:29:08.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.575+0000 7fbb5a09e640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fbb540cf160 0x7fbb540cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:52298/0 (socket says 172.21.15.89:52298) 2023-10-30T12:29:08.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.575+0000 7fbb5a09e640 1 -- 172.21.15.89:0/2396672604 learned_addr learned my addr 172.21.15.89:0/2396672604 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:29:08.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.575+0000 7fbb5909c640 1 --2- 172.21.15.89:0/2396672604 >> v2:172.21.15.192:3300/0 conn(0x7fbb540ca0e0 0x7fbb5415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:08.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.575+0000 7fbb5989d640 1 --2- 172.21.15.89:0/2396672604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb541538a0 0x7fbb5415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:08.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.575+0000 7fbb5889b640 1 -- 172.21.15.89:0/2396672604 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3678015207 0 0) 0x7fbb540cb800 con 0x7fbb540cf160 2023-10-30T12:29:08.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.575+0000 7fbb5889b640 1 -- 172.21.15.89:0/2396672604 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb40003530 con 0x7fbb540cf160 2023-10-30T12:29:08.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.575+0000 7fbb5889b640 1 -- 172.21.15.89:0/2396672604 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2788816579 0 0) 0x7fbb40003530 con 0x7fbb540cf160 2023-10-30T12:29:08.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.575+0000 7fbb5889b640 1 -- 172.21.15.89:0/2396672604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb541538a0 msgr2=0x7fbb5415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:08.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.575+0000 7fbb5889b640 1 --2- 172.21.15.89:0/2396672604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb541538a0 0x7fbb5415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:08.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.575+0000 7fbb5889b640 1 -- 172.21.15.89:0/2396672604 >> v2:172.21.15.192:3300/0 conn(0x7fbb540ca0e0 msgr2=0x7fbb5415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:08.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.575+0000 7fbb5889b640 1 --2- 172.21.15.89:0/2396672604 >> v2:172.21.15.192:3300/0 conn(0x7fbb540ca0e0 0x7fbb5415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:08.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.575+0000 7fbb5889b640 1 -- 172.21.15.89:0/2396672604 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb540cad70 con 0x7fbb540cf160 2023-10-30T12:29:08.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.575+0000 7fbb5889b640 1 -- 172.21.15.89:0/2396672604 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7fbb4c001a00 con 0x7fbb540cf160 2023-10-30T12:29:08.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.579+0000 7fbb5889b640 1 -- 172.21.15.89:0/2396672604 >> v1:172.21.15.89:6789/0 conn(0x7fbb540cf160 legacy=0x7fbb540cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:08.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.579+0000 7fbb5889b640 1 --2- 172.21.15.89:0/2396672604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb541538a0 0x7fbb40003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:08.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.579+0000 7fbb5889b640 1 --2- 172.21.15.89:0/2396672604 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb40003f00 0x7fbb400047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:08.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.579+0000 7fbb5889b640 1 --2- 172.21.15.89:0/2396672604 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb40004d10 0x7fbb40005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:08.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.579+0000 7fbb5889b640 1 -- 172.21.15.89:0/2396672604 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5415dc30 con 0x7fbb40004d10 2023-10-30T12:29:08.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.579+0000 7fbb5889b640 1 -- 172.21.15.89:0/2396672604 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb540cb330 con 0x7fbb40003f00 2023-10-30T12:29:08.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.579+0000 7fbb5889b640 1 -- 172.21.15.89:0/2396672604 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4000b9c0 con 0x7fbb541538a0 2023-10-30T12:29:08.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.579+0000 7fbb5a09e640 1 --2- 172.21.15.89:0/2396672604 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb40004d10 0x7fbb40005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:08.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.579+0000 7fbb5989d640 1 --2- 172.21.15.89:0/2396672604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb541538a0 0x7fbb40003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:08.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.579+0000 7fbb5909c640 1 --2- 172.21.15.89:0/2396672604 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb40003f00 0x7fbb400047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:08.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.579+0000 7fbb5a09e640 1 -- 172.21.15.89:0/2396672604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb541538a0 msgr2=0x7fbb40003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:08.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.579+0000 7fbb5a09e640 1 --2- 172.21.15.89:0/2396672604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb541538a0 0x7fbb40003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:08.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.579+0000 7fbb5a09e640 1 -- 172.21.15.89:0/2396672604 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb40003f00 msgr2=0x7fbb400047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:08.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.579+0000 7fbb5a09e640 1 --2- 172.21.15.89:0/2396672604 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb40003f00 0x7fbb400047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:08.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.579+0000 7fbb5a09e640 1 -- 172.21.15.89:0/2396672604 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb40003530 con 0x7fbb40004d10 2023-10-30T12:29:08.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.579+0000 7fbb5989d640 1 --2- 172.21.15.89:0/2396672604 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb541538a0 0x7fbb40003980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:08.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.579+0000 7fbb5909c640 1 --2- 172.21.15.89:0/2396672604 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb40003f00 0x7fbb400047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:08.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.579+0000 7fbb5a09e640 1 --2- 172.21.15.89:0/2396672604 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb40004d10 0x7fbb40005190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c00a860 tx=0x7fbb4c01d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99d551d3a08b8c9f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:08.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.579+0000 7fbb5889b640 1 -- 172.21.15.89:0/2396672604 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4c0ad630 con 0x7fbb40004d10 2023-10-30T12:29:08.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.579+0000 7fbb5889b640 1 -- 172.21.15.89:0/2396672604 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb4c0a8070 con 0x7fbb40004d10 2023-10-30T12:29:08.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.579+0000 7fbb5889b640 1 -- 172.21.15.89:0/2396672604 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4c0ada30 con 0x7fbb40004d10 2023-10-30T12:29:08.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.583+0000 7fbb5bb01640 1 -- 172.21.15.89:0/2396672604 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb40004d10 msgr2=0x7fbb40005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:08.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.583+0000 7fbb5bb01640 1 --2- 172.21.15.89:0/2396672604 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb40004d10 0x7fbb40005190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c00a860 tx=0x7fbb4c01d360 comp rx=0 tx=0).stop 2023-10-30T12:29:08.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.583+0000 7fbb5a09e640 1 -- 172.21.15.89:0/2396672604 reap_dead start 2023-10-30T12:29:08.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.583+0000 7fbb5bb01640 1 -- 172.21.15.89:0/2396672604 shutdown_connections 2023-10-30T12:29:08.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.583+0000 7fbb5bb01640 1 -- 172.21.15.89:0/2396672604 >> 172.21.15.89:0/2396672604 conn(0x7fbb5406bfe0 msgr2=0x7fbb5406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:08.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.583+0000 7fbb5bb01640 1 -- 172.21.15.89:0/2396672604 shutdown_connections 2023-10-30T12:29:08.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.583+0000 7fbb5bb01640 1 -- 172.21.15.89:0/2396672604 wait complete. 2023-10-30T12:29:08.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.583+0000 7fbb5bb01640 1 Processor -- start 2023-10-30T12:29:08.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.583+0000 7fbb5bb01640 1 -- start start 2023-10-30T12:29:08.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.583+0000 7fbb5bb01640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb40004d10 0x7fbb541f0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:08.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.583+0000 7fbb5bb01640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb54151540 0x7fbb540c8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:08.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.583+0000 7fbb5bb01640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb541e52f0 0x7fbb54145a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:08.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.583+0000 7fbb5bb01640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb541e2ae0 con 0x7fbb40004d10 2023-10-30T12:29:08.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.583+0000 7fbb5bb01640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb541e3660 con 0x7fbb541e52f0 2023-10-30T12:29:08.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.583+0000 7fbb5bb01640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb541e5800 con 0x7fbb54151540 2023-10-30T12:29:08.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.583+0000 7fbb5989d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb40004d10 0x7fbb541f0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:08.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.583+0000 7fbb5989d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb40004d10 0x7fbb541f0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39484/0 (socket says 172.21.15.89:39484) 2023-10-30T12:29:08.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.583+0000 7fbb5989d640 1 -- 172.21.15.89:0/760359999 learned_addr learned my addr 172.21.15.89:0/760359999 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:08.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.583+0000 7fbb5909c640 1 --2- 172.21.15.89:0/760359999 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb54151540 0x7fbb540c8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:08.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.583+0000 7fbb5a09e640 1 --2- 172.21.15.89:0/760359999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb541e52f0 0x7fbb54145a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:08.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.587+0000 7fbb5989d640 1 -- 172.21.15.89:0/760359999 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb54151540 msgr2=0x7fbb540c8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:08.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.587+0000 7fbb5989d640 1 --2- 172.21.15.89:0/760359999 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb54151540 0x7fbb540c8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:08.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.587+0000 7fbb5989d640 1 -- 172.21.15.89:0/760359999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb541e52f0 msgr2=0x7fbb54145a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:08.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.587+0000 7fbb5989d640 1 --2- 172.21.15.89:0/760359999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb541e52f0 0x7fbb54145a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:08.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.587+0000 7fbb5989d640 1 -- 172.21.15.89:0/760359999 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb541e2d30 con 0x7fbb40004d10 2023-10-30T12:29:08.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.587+0000 7fbb5a09e640 1 --2- 172.21.15.89:0/760359999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb541e52f0 0x7fbb54145a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:29:08.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.587+0000 7fbb5989d640 1 --2- 172.21.15.89:0/760359999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb40004d10 0x7fbb541f0120 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbb500116a0 tx=0x7fbb50011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:08.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.587+0000 7fbb4a7fc640 1 -- 172.21.15.89:0/760359999 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb50004220 con 0x7fbb40004d10 2023-10-30T12:29:08.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.587+0000 7fbb4a7fc640 1 -- 172.21.15.89:0/760359999 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb50004380 con 0x7fbb40004d10 2023-10-30T12:29:08.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.587+0000 7fbb4a7fc640 1 -- 172.21.15.89:0/760359999 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5000f9e0 con 0x7fbb40004d10 2023-10-30T12:29:08.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.587+0000 7fbb5bb01640 1 -- 172.21.15.89:0/760359999 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb541e3010 con 0x7fbb40004d10 2023-10-30T12:29:08.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.587+0000 7fbb5bb01640 1 -- 172.21.15.89:0/760359999 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb541f0660 con 0x7fbb40004d10 2023-10-30T12:29:08.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.587+0000 7fbb5bb01640 1 -- 172.21.15.89:0/760359999 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb5404e170 con 0x7fbb40004d10 2023-10-30T12:29:08.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.587+0000 7fbb4a7fc640 1 -- 172.21.15.89:0/760359999 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbb5000fb40 con 0x7fbb40004d10 2023-10-30T12:29:08.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.587+0000 7fbb4a7fc640 1 --2- 172.21.15.89:0/760359999 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbb380664d0 0x7fbb38068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:08.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.587+0000 7fbb4a7fc640 1 -- 172.21.15.89:0/760359999 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbb500844c0 con 0x7fbb40004d10 2023-10-30T12:29:08.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.591+0000 7fbb5909c640 1 --2- 172.21.15.89:0/760359999 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbb380664d0 0x7fbb38068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:08.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.591+0000 7fbb5909c640 1 --2- 172.21.15.89:0/760359999 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbb380664d0 0x7fbb38068990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbb540c51b0 tx=0x7fbb44008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:08.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.591+0000 7fbb4a7fc640 1 -- 172.21.15.89:0/760359999 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbb500518a0 con 0x7fbb40004d10 2023-10-30T12:29:08.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.771+0000 7fbb5bb01640 1 -- 172.21.15.89:0/760359999 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb540b1dd0 con 0x7fbb40004d10 2023-10-30T12:29:08.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.775+0000 7fbb4a7fc640 1 -- 172.21.15.89:0/760359999 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6528 (secure 0 0 0) 0x7fbb50054ec0 con 0x7fbb40004d10 2023-10-30T12:29:08.780 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:29:08.781 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:29:06.969397+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4253":{"gid":4253,"name":"b","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6835/1289220491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1289220491},{"type":"v1","addr":"172.21.15.192:6835","nonce":1289220491}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:29:08.781 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 12 2023-10-30T12:29:08.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.775+0000 7fbb5bb01640 1 -- 172.21.15.89:0/760359999 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbb380664d0 msgr2=0x7fbb38068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:08.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.775+0000 7fbb5bb01640 1 --2- 172.21.15.89:0/760359999 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbb380664d0 0x7fbb38068990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbb540c51b0 tx=0x7fbb44008040 comp rx=0 tx=0).stop 2023-10-30T12:29:08.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.775+0000 7fbb5bb01640 1 -- 172.21.15.89:0/760359999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb40004d10 msgr2=0x7fbb541f0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:08.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.775+0000 7fbb5bb01640 1 --2- 172.21.15.89:0/760359999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb40004d10 0x7fbb541f0120 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbb500116a0 tx=0x7fbb50011b60 comp rx=0 tx=0).stop 2023-10-30T12:29:08.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.775+0000 7fbb5bb01640 1 -- 172.21.15.89:0/760359999 shutdown_connections 2023-10-30T12:29:08.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.775+0000 7fbb5bb01640 1 --2- 172.21.15.89:0/760359999 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbb380664d0 0x7fbb38068990 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:08.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.775+0000 7fbb5bb01640 1 --2- 172.21.15.89:0/760359999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb541e52f0 0x7fbb54145a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:08.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.775+0000 7fbb5bb01640 1 --2- 172.21.15.89:0/760359999 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbb54151540 0x7fbb540c8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:08.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.775+0000 7fbb5bb01640 1 --2- 172.21.15.89:0/760359999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb40004d10 0x7fbb541f0120 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:08.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.775+0000 7fbb5bb01640 1 -- 172.21.15.89:0/760359999 >> 172.21.15.89:0/760359999 conn(0x7fbb5406bfe0 msgr2=0x7fbb540c5700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:08.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.775+0000 7fbb5bb01640 1 -- 172.21.15.89:0/760359999 shutdown_connections 2023-10-30T12:29:08.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:08.775+0000 7fbb5bb01640 1 -- 172.21.15.89:0/760359999 wait complete. 2023-10-30T12:29:09.404 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 12, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-30T12:28:50.729245+0000', 'modified': '2023-10-30T12:29:06.969397+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 4318, 'mds_1': 4262, 'mds_2': 4253}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4253': {'gid': 4253, 'name': 'b', 'rank': 2, 'incarnation': 11, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.192:6835/1289220491', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6834', 'nonce': 1289220491}, {'type': 'v1', 'addr': '172.21.15.192:6835', 'nonce': 1289220491}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4262': {'gid': 4262, 'name': 'd', 'rank': 1, 'incarnation': 9, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.192:6839/3339589587', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6838', 'nonce': 3339589587}, {'type': 'v1', 'addr': '172.21.15.192:6839', 'nonce': 3339589587}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4318': {'gid': 4318, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.192:6837/1002124471', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6836', 'nonce': 1002124471}, {'type': 'v1', 'addr': '172.21.15.192:6837', 'nonce': 1002124471}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-30T12:29:09.404 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-10-30T12:29:09.404 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-30T12:29:09.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.539+0000 7f7411ebb640 1 Processor -- start 2023-10-30T12:29:09.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.539+0000 7f7411ebb640 1 -- start start 2023-10-30T12:29:09.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.539+0000 7f7411ebb640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f740c0cab00 0x7f740c164ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:09.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.539+0000 7f7411ebb640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f740c0cba30 con 0x7f740c0c5550 2023-10-30T12:29:09.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.539+0000 7f7411ebb640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f740c0ca290 con 0x7f740c0cb050 2023-10-30T12:29:09.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.539+0000 7f7411ebb640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f740c0c9cf0 con 0x7f740c0cab00 2023-10-30T12:29:09.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.539+0000 7f74116ba640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f740c0cab00 0x7f740c164ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:09.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.539+0000 7f7410eb9640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f740c0c5550 0x7f740c0cef00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:52308/0 (socket says 172.21.15.89:52308) 2023-10-30T12:29:09.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.539+0000 7f7410eb9640 1 -- 172.21.15.89:0/2003679092 learned_addr learned my addr 172.21.15.89:0/2003679092 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:29:09.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.539+0000 7f74116ba640 1 -- 172.21.15.89:0/2003679092 >> v1:172.21.15.192:6789/0 conn(0x7f740c0cb050 legacy=0x7f740c0cbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:09.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.539+0000 7f74037fe640 1 -- 172.21.15.89:0/2003679092 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 450239765 0 0) 0x7f740c0cba30 con 0x7f740c0c5550 2023-10-30T12:29:09.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.539+0000 7f74116ba640 1 -- 172.21.15.89:0/2003679092 >> v1:172.21.15.89:6789/0 conn(0x7f740c0c5550 legacy=0x7f740c0cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:09.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.539+0000 7f74116ba640 1 -- 172.21.15.89:0/2003679092 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f740c15d990 con 0x7f740c0cab00 2023-10-30T12:29:09.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.543+0000 7f74116ba640 1 --2- 172.21.15.89:0/2003679092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f740c0cab00 0x7f740c164ee0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7404009d90 tx=0x7f740401c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30f9af5feceb8749 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:09.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.543+0000 7f74037fe640 1 -- 172.21.15.89:0/2003679092 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74040a5900 con 0x7f740c0cab00 2023-10-30T12:29:09.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.543+0000 7f74037fe640 1 -- 172.21.15.89:0/2003679092 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74040a5a60 con 0x7f740c0cab00 2023-10-30T12:29:09.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.543+0000 7f74037fe640 1 -- 172.21.15.89:0/2003679092 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74040a5df0 con 0x7f740c0cab00 2023-10-30T12:29:09.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.543+0000 7f7411ebb640 1 -- 172.21.15.89:0/2003679092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f740c0cab00 msgr2=0x7f740c164ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:09.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.543+0000 7f7411ebb640 1 --2- 172.21.15.89:0/2003679092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f740c0cab00 0x7f740c164ee0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7404009d90 tx=0x7f740401c950 comp rx=0 tx=0).stop 2023-10-30T12:29:09.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.543+0000 7f7411ebb640 1 -- 172.21.15.89:0/2003679092 shutdown_connections 2023-10-30T12:29:09.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.543+0000 7f7411ebb640 1 --2- 172.21.15.89:0/2003679092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f740c0cab00 0x7f740c164ee0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:09.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.543+0000 7f7411ebb640 1 -- 172.21.15.89:0/2003679092 >> 172.21.15.89:0/2003679092 conn(0x7f740c06bed0 msgr2=0x7f740c06c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:09.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.543+0000 7f7411ebb640 1 -- 172.21.15.89:0/2003679092 shutdown_connections 2023-10-30T12:29:09.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.543+0000 7f7411ebb640 1 -- 172.21.15.89:0/2003679092 wait complete. 2023-10-30T12:29:09.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.547+0000 7f7411ebb640 1 Processor -- start 2023-10-30T12:29:09.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.547+0000 7f7411ebb640 1 -- start start 2023-10-30T12:29:09.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.547+0000 7f7411ebb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f740c0c5550 0x7f740c14b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:09.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.547+0000 7f7411ebb640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f740c0cab00 0x7f740c150d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:09.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.547+0000 7f7411ebb640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f740c0cb050 0x7f740c14c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:09.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.547+0000 7f7411ebb640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f740c1f5410 con 0x7f740c0c5550 2023-10-30T12:29:09.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.547+0000 7f7411ebb640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f740c15aa50 con 0x7f740c0cb050 2023-10-30T12:29:09.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.547+0000 7f7411ebb640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f740c1f4ae0 con 0x7f740c0cab00 2023-10-30T12:29:09.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.547+0000 7f7410eb9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f740c0c5550 0x7f740c14b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:09.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.547+0000 7f7410eb9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f740c0c5550 0x7f740c14b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39504/0 (socket says 172.21.15.89:39504) 2023-10-30T12:29:09.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.547+0000 7f7410eb9640 1 -- 172.21.15.89:0/3373305388 learned_addr learned my addr 172.21.15.89:0/3373305388 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:09.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.547+0000 7f7403fff640 1 --2- 172.21.15.89:0/3373305388 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f740c0cab00 0x7f740c150d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:09.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.547+0000 7f74116ba640 1 --2- 172.21.15.89:0/3373305388 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f740c0cb050 0x7f740c14c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:09.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.547+0000 7f7410eb9640 1 -- 172.21.15.89:0/3373305388 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f740c0cab00 msgr2=0x7f740c150d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:09.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.547+0000 7f7410eb9640 1 --2- 172.21.15.89:0/3373305388 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f740c0cab00 0x7f740c150d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:09.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.547+0000 7f7410eb9640 1 -- 172.21.15.89:0/3373305388 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f740c0cb050 msgr2=0x7f740c14c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:09.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.547+0000 7f7410eb9640 1 --2- 172.21.15.89:0/3373305388 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f740c0cb050 0x7f740c14c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:09.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.547+0000 7f7410eb9640 1 -- 172.21.15.89:0/3373305388 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7404009a70 con 0x7f740c0c5550 2023-10-30T12:29:09.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.547+0000 7f7410eb9640 1 --2- 172.21.15.89:0/3373305388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f740c0c5550 0x7f740c14b710 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f740800c4a0 tx=0x7f740800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:09.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.547+0000 7f74017fa640 1 -- 172.21.15.89:0/3373305388 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74080074e0 con 0x7f740c0c5550 2023-10-30T12:29:09.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.547+0000 7f74017fa640 1 -- 172.21.15.89:0/3373305388 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7408007640 con 0x7f740c0c5550 2023-10-30T12:29:09.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.547+0000 7f74017fa640 1 -- 172.21.15.89:0/3373305388 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7408004a30 con 0x7f740c0c5550 2023-10-30T12:29:09.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.547+0000 7f7411ebb640 1 -- 172.21.15.89:0/3373305388 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f740c1f4dc0 con 0x7f740c0c5550 2023-10-30T12:29:09.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.547+0000 7f7411ebb640 1 -- 172.21.15.89:0/3373305388 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f740c0c7d70 con 0x7f740c0c5550 2023-10-30T12:29:09.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.551+0000 7f74017fa640 1 -- 172.21.15.89:0/3373305388 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7408004b90 con 0x7f740c0c5550 2023-10-30T12:29:09.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.551+0000 7f74017fa640 1 --2- 172.21.15.89:0/3373305388 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f73f80664a0 0x7f73f8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:09.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.551+0000 7f74017fa640 1 -- 172.21.15.89:0/3373305388 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74080831b0 con 0x7f740c0c5550 2023-10-30T12:29:09.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.551+0000 7f7403fff640 1 --2- 172.21.15.89:0/3373305388 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f73f80664a0 0x7f73f8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:09.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.551+0000 7f741649f000 1 Processor -- start 2023-10-30T12:29:09.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.551+0000 7f741649f000 1 -- start start 2023-10-30T12:29:09.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.551+0000 7f7403fff640 1 --2- 172.21.15.89:0/3373305388 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f73f80664a0 0x7f73f8068960 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f740c164ee0 tx=0x7f73fc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:09.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.551+0000 7f741649f000 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x559101fdc810 0x559101fdcbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:09.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.551+0000 7f741649f000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559101fdd130 0x559101fdd510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:09.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.551+0000 7f741649f000 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559101fdda50 0x559101fe7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:09.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.551+0000 7f741649f000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559101d90cc0 con 0x559101fdd130 2023-10-30T12:29:09.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.551+0000 7f741649f000 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x559101df3920 con 0x559101fdda50 2023-10-30T12:29:09.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.551+0000 7f741649f000 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x559101df0e50 con 0x559101fdc810 2023-10-30T12:29:09.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.551+0000 7f7403fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559101fdd130 0x559101fdd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:09.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.551+0000 7f7403fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559101fdd130 0x559101fdd510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39512/0 (socket says 172.21.15.89:39512) 2023-10-30T12:29:09.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.551+0000 7f7410eb9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x559101fdc810 0x559101fdcbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:09.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.551+0000 7f7403fff640 1 -- 172.21.15.89:0/866239995 learned_addr learned my addr 172.21.15.89:0/866239995 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:09.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.551+0000 7f74116ba640 1 --2- 172.21.15.89:0/866239995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559101fdda50 0x559101fe7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:09.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f7403fff640 1 -- 172.21.15.89:0/866239995 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x559101fdc810 msgr2=0x559101fdcbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:09.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f7403fff640 1 --2- 172.21.15.89:0/866239995 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x559101fdc810 0x559101fdcbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:09.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f7403fff640 1 -- 172.21.15.89:0/866239995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559101fdda50 msgr2=0x559101fe7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:09.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f7403fff640 1 --2- 172.21.15.89:0/866239995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559101fdda50 0x559101fe7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:09.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f7403fff640 1 -- 172.21.15.89:0/866239995 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559101ae5890 con 0x559101fdd130 2023-10-30T12:29:09.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f7403fff640 1 --2- 172.21.15.89:0/866239995 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559101fdd130 0x559101fdd510 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f73fc0052b0 tx=0x7f73fc011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad938ad6acc8e70 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:09.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f73f27fc640 1 -- 172.21.15.89:0/866239995 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73fc00e4a0 con 0x559101fdd130 2023-10-30T12:29:09.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f73f27fc640 1 -- 172.21.15.89:0/866239995 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73fc00e600 con 0x559101fdd130 2023-10-30T12:29:09.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f73f27fc640 1 -- 172.21.15.89:0/866239995 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73fc00e970 con 0x559101fdd130 2023-10-30T12:29:09.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f741649f000 1 -- 172.21.15.89:0/866239995 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559101fdd130 msgr2=0x559101fdd510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:09.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f741649f000 1 --2- 172.21.15.89:0/866239995 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559101fdd130 0x559101fdd510 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f73fc0052b0 tx=0x7f73fc011000 comp rx=0 tx=0).stop 2023-10-30T12:29:09.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f741649f000 1 -- 172.21.15.89:0/866239995 shutdown_connections 2023-10-30T12:29:09.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f741649f000 1 --2- 172.21.15.89:0/866239995 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559101fdda50 0x559101fe7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:09.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f741649f000 1 --2- 172.21.15.89:0/866239995 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559101fdd130 0x559101fdd510 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:09.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f741649f000 1 --2- 172.21.15.89:0/866239995 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x559101fdc810 0x559101fdcbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:09.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f741649f000 1 -- 172.21.15.89:0/866239995 >> 172.21.15.89:0/866239995 conn(0x559101f92fc0 msgr2=0x559101fda5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:09.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f741649f000 1 -- 172.21.15.89:0/866239995 shutdown_connections 2023-10-30T12:29:09.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f741649f000 1 -- 172.21.15.89:0/866239995 wait complete. 2023-10-30T12:29:09.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f741649f000 1 -- start start 2023-10-30T12:29:09.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f741649f000 1 Processor -- start 2023-10-30T12:29:09.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f741649f000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559101fdc810 0x559101fd6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:09.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f741649f000 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x559101fdd130 0x559101fd7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:09.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f741649f000 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559101fdda50 0x559101fe2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:09.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f741649f000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559101dec740 con 0x559101fdc810 2023-10-30T12:29:09.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f741649f000 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x559101dabd10 con 0x559101fdda50 2023-10-30T12:29:09.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f741649f000 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x559101dab9b0 con 0x559101fdd130 2023-10-30T12:29:09.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f741649f000 10 client.0 _ll_register_callbacks cb 0x559101f87390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T12:29:09.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f7410eb9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559101fdc810 0x559101fd6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:09.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f7410eb9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559101fdc810 0x559101fd6f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39528/0 (socket says 172.21.15.89:39528) 2023-10-30T12:29:09.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f7410eb9640 1 -- 172.21.15.89:0/14059155 learned_addr learned my addr 172.21.15.89:0/14059155 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:09.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.555+0000 7f74116ba640 1 --2- 172.21.15.89:0/14059155 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559101fdda50 0x559101fe2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:09.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.559+0000 7f7403fff640 1 --2- 172.21.15.89:0/14059155 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x559101fdd130 0x559101fd7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:09.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.559+0000 7f7410eb9640 1 -- 172.21.15.89:0/14059155 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x559101fdd130 msgr2=0x559101fd7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:09.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.559+0000 7f7410eb9640 1 --2- 172.21.15.89:0/14059155 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x559101fdd130 0x559101fd7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:09.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.559+0000 7f7410eb9640 1 -- 172.21.15.89:0/14059155 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559101fdda50 msgr2=0x559101fe2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:09.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.559+0000 7f7410eb9640 1 --2- 172.21.15.89:0/14059155 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559101fdda50 0x559101fe2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:09.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.559+0000 7f7410eb9640 1 -- 172.21.15.89:0/14059155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73fc00f040 con 0x559101fdc810 2023-10-30T12:29:09.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.559+0000 7f7403fff640 1 --2- 172.21.15.89:0/14059155 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x559101fdd130 0x559101fd7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:09.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.559+0000 7f7410eb9640 1 --2- 172.21.15.89:0/14059155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559101fdc810 0x559101fd6f50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f740800ce60 tx=0x7f7408004e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0579e0dcf38a2b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:09.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.559+0000 7f73eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-10-30T12:29:09.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.559+0000 7f73eb7fe640 1 -- 172.21.15.89:0/14059155 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74080583a0 con 0x559101fdc810 2023-10-30T12:29:09.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.559+0000 7f73eb7fe640 1 -- 172.21.15.89:0/14059155 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f740805c550 con 0x559101fdc810 2023-10-30T12:29:09.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.559+0000 7f741649f000 1 -- 172.21.15.89:0/14059155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559101da0370 con 0x559101fdc810 2023-10-30T12:29:09.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.559+0000 7f73eb7fe640 1 -- 172.21.15.89:0/14059155 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7408081170 con 0x559101fdc810 2023-10-30T12:29:09.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.559+0000 7f741649f000 10 client.4677 fetch_fsmap learned FSMap version 12 2023-10-30T12:29:09.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.559+0000 7f741649f000 1 -- 172.21.15.89:0/14059155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0x559101dae460 con 0x559101fdc810 2023-10-30T12:29:09.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.559+0000 7f73eb7fe640 1 -- 172.21.15.89:0/14059155 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559101da0370 con 0x559101fdc810 2023-10-30T12:29:09.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.559+0000 7f73eb7fe640 1 -- 172.21.15.89:0/14059155 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 12) v1 ==== 4468+0+0 (secure 0 0 0) 0x7f740805c0c0 con 0x559101fdc810 2023-10-30T12:29:09.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.559+0000 7f741649f000 10 client.4677 fetch_fsmap finished waiting for FSMap version 12 2023-10-30T12:29:09.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.559+0000 7f741649f000 10 client.4677 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-30T12:29:09.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.559+0000 7f741649f000 20 client.4677 populate_metadata read hostname 'smithi089' 2023-10-30T12:29:09.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.559+0000 7f741649f000 1 --2- 172.21.15.89:0/14059155 >> [v2:172.21.15.192:6834/1289220491,v1:172.21.15.192:6835/1289220491] conn(0x559101fe3790 0x559101fe3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:09.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.559+0000 7f741649f000 4 client.4677 mds_command: new command op to 4253 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:29:09.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.559+0000 7f741649f000 1 -- 172.21.15.89:0/14059155 --> [v2:172.21.15.192:6834/1289220491,v1:172.21.15.192:6835/1289220491] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559101d862d0 con 0x559101fe3790 2023-10-30T12:29:09.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.559+0000 7f7403fff640 1 --2- 172.21.15.89:0/14059155 >> [v2:172.21.15.192:6834/1289220491,v1:172.21.15.192:6835/1289220491] conn(0x559101fe3790 0x559101fe3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:09.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.559+0000 7f73eb7fe640 10 client.4677 ms_handle_connect on v2:172.21.15.192:6834/1289220491 2023-10-30T12:29:09.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:09.559+0000 7f7403fff640 1 --2- 172.21.15.89:0/14059155 >> [v2:172.21.15.192:6834/1289220491,v1:172.21.15.192:6835/1289220491] conn(0x559101fe3790 0x559101fe3b70 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8e50be7ff4d4ee86 server_cookie=76a2c171940ac13e in_seq=0 out_seq=0 2023-10-30T12:29:10.318 INFO:tasks.ceph.mds.f.smithi192.stderr:2023-10-30T12:29:10.315+0000 7f0036e29640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T12:29:12.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.971+0000 7f73eb7fe640 1 -- 172.21.15.89:0/14059155 <== mds.2 v2:172.21.15.192:6834/1289220491 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559101d862d0 con 0x559101fe3790 2023-10-30T12:29:12.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.971+0000 7f73eb7fe640 10 client.4677 handle_command_reply: tid=0 2023-10-30T12:29:12.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.971+0000 7f741649f000 1 client.4677 shutdown 2023-10-30T12:29:12.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.971+0000 7f741649f000 1 -- 172.21.15.89:0/14059155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559101fdc810 msgr2=0x559101fd6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:12.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.971+0000 7f741649f000 1 --2- 172.21.15.89:0/14059155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559101fdc810 0x559101fd6f50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f740800ce60 tx=0x7f7408004e40 comp rx=0 tx=0).stop 2023-10-30T12:29:12.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.971+0000 7f741649f000 1 -- 172.21.15.89:0/14059155 shutdown_connections 2023-10-30T12:29:12.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.971+0000 7f741649f000 1 --2- 172.21.15.89:0/14059155 >> [v2:172.21.15.192:6834/1289220491,v1:172.21.15.192:6835/1289220491] conn(0x559101fe3790 0x559101fe3b70 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.971+0000 7f741649f000 1 --2- 172.21.15.89:0/14059155 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559101fdda50 0x559101fe2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.971+0000 7f741649f000 1 --2- 172.21.15.89:0/14059155 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x559101fdd130 0x559101fd7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.971+0000 7f741649f000 1 --2- 172.21.15.89:0/14059155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559101fdc810 0x559101fd6f50 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.971+0000 7f73eb7fe640 0 client.4677 ms_handle_reset on v2:172.21.15.192:6834/1289220491 2023-10-30T12:29:12.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.971+0000 7f741649f000 1 -- 172.21.15.89:0/14059155 >> 172.21.15.89:0/14059155 conn(0x559101f92fc0 msgr2=0x559101fdc090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:12.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.971+0000 7f741649f000 1 -- 172.21.15.89:0/14059155 shutdown_connections 2023-10-30T12:29:12.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.971+0000 7f741649f000 1 -- 172.21.15.89:0/14059155 wait complete. 2023-10-30T12:29:12.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.971+0000 7f741649f000 20 client.4677 trim_cache size 0 max 16384 2023-10-30T12:29:12.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.987+0000 7f741649f000 1 Processor -- start 2023-10-30T12:29:12.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.987+0000 7f741649f000 1 -- start start 2023-10-30T12:29:12.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f741649f000 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559101fdc810 0x559101fa1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:12.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f741649f000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559101fdd130 0x559101fa1770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:12.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f741649f000 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x559101fdda50 0x559102038830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:12.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f741649f000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559101aa6940 con 0x559101fdd130 2023-10-30T12:29:12.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f741649f000 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x559101dab4d0 con 0x559101fdc810 2023-10-30T12:29:12.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f741649f000 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x559101fdbc60 con 0x559101fdda50 2023-10-30T12:29:12.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f7403fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559101fdd130 0x559101fa1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:12.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f7403fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559101fdd130 0x559101fa1770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57668/0 (socket says 172.21.15.89:57668) 2023-10-30T12:29:12.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f7403fff640 1 -- 172.21.15.89:0/3410117254 learned_addr learned my addr 172.21.15.89:0/3410117254 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:12.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f7410eb9640 1 --2- 172.21.15.89:0/3410117254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559101fdc810 0x559101fa1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:12.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f74116ba640 1 --2- 172.21.15.89:0/3410117254 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x559101fdda50 0x559102038830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:12.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f7403fff640 1 -- 172.21.15.89:0/3410117254 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x559101fdda50 msgr2=0x559102038830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:12.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f7403fff640 1 --2- 172.21.15.89:0/3410117254 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x559101fdda50 0x559102038830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f7403fff640 1 -- 172.21.15.89:0/3410117254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559101fdc810 msgr2=0x559101fa1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:12.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f7403fff640 1 --2- 172.21.15.89:0/3410117254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559101fdc810 0x559101fa1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f7403fff640 1 -- 172.21.15.89:0/3410117254 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559101d94760 con 0x559101fdd130 2023-10-30T12:29:12.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f7410eb9640 1 --2- 172.21.15.89:0/3410117254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559101fdc810 0x559101fa1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:13.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f7403fff640 1 --2- 172.21.15.89:0/3410117254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559101fdd130 0x559101fa1770 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f73fc007b50 tx=0x7f73fc017070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e98f1a6abde5e3fe server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:13.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f73f2ffd640 1 -- 172.21.15.89:0/3410117254 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73fc027040 con 0x559101fdd130 2023-10-30T12:29:13.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f73f2ffd640 1 -- 172.21.15.89:0/3410117254 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73fc019070 con 0x559101fdd130 2023-10-30T12:29:13.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f73f2ffd640 1 -- 172.21.15.89:0/3410117254 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73fc0225a0 con 0x559101fdd130 2023-10-30T12:29:13.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f741649f000 1 -- 172.21.15.89:0/3410117254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559101fdd130 msgr2=0x559101fa1770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f741649f000 1 --2- 172.21.15.89:0/3410117254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559101fdd130 0x559101fa1770 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f73fc007b50 tx=0x7f73fc017070 comp rx=0 tx=0).stop 2023-10-30T12:29:13.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f741649f000 1 -- 172.21.15.89:0/3410117254 shutdown_connections 2023-10-30T12:29:13.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f741649f000 1 --2- 172.21.15.89:0/3410117254 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x559101fdda50 0x559102038830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.002 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f741649f000 1 --2- 172.21.15.89:0/3410117254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559101fdd130 0x559101fa1770 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.002 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f741649f000 1 --2- 172.21.15.89:0/3410117254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559101fdc810 0x559101fa1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f741649f000 1 -- 172.21.15.89:0/3410117254 >> 172.21.15.89:0/3410117254 conn(0x559101f92fc0 msgr2=0x559101f98b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:13.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f741649f000 1 -- 172.21.15.89:0/3410117254 shutdown_connections 2023-10-30T12:29:13.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f741649f000 1 -- 172.21.15.89:0/3410117254 wait complete. 2023-10-30T12:29:13.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.991+0000 7f741649f000 1 -- start start 2023-10-30T12:29:13.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f741649f000 1 Processor -- start 2023-10-30T12:29:13.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f741649f000 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559101fdc810 0x559101fa19a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f741649f000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559101fdd130 0x55910203d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f741649f000 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x559101fdda50 0x55910203deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f741649f000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559101de6c20 con 0x559101fdd130 2023-10-30T12:29:13.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f741649f000 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x559101df0740 con 0x559101fdc810 2023-10-30T12:29:13.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f741649f000 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x559101de6a40 con 0x559101fdda50 2023-10-30T12:29:13.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f741649f000 10 client.0 _ll_register_callbacks cb 0x7f73fc0013a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T12:29:13.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f7403fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559101fdd130 0x55910203d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f7403fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559101fdd130 0x55910203d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57672/0 (socket says 172.21.15.89:57672) 2023-10-30T12:29:13.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f7403fff640 1 -- 172.21.15.89:0/4123815682 learned_addr learned my addr 172.21.15.89:0/4123815682 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:13.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f7410eb9640 1 --2- 172.21.15.89:0/4123815682 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559101fdc810 0x559101fa19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f74116ba640 1 --2- 172.21.15.89:0/4123815682 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x559101fdda50 0x55910203deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f7403fff640 1 -- 172.21.15.89:0/4123815682 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x559101fdda50 msgr2=0x55910203deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:13.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f7403fff640 1 --2- 172.21.15.89:0/4123815682 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x559101fdda50 0x55910203deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f7403fff640 1 -- 172.21.15.89:0/4123815682 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559101fdc810 msgr2=0x559101fa19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:13.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f7403fff640 1 --2- 172.21.15.89:0/4123815682 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559101fdc810 0x559101fa19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f7403fff640 1 -- 172.21.15.89:0/4123815682 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559101ae5890 con 0x559101fdd130 2023-10-30T12:29:13.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f7403fff640 1 --2- 172.21.15.89:0/4123815682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559101fdd130 0x55910203d970 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f73fc003eb0 tx=0x7f73fc000f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e613d72ba8db3336 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:13.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f73eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-10-30T12:29:13.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f73eb7fe640 1 -- 172.21.15.89:0/4123815682 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73fc027050 con 0x559101fdd130 2023-10-30T12:29:13.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f73eb7fe640 1 -- 172.21.15.89:0/4123815682 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f73fc0224e0 con 0x559101fdd130 2023-10-30T12:29:13.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f741649f000 1 -- 172.21.15.89:0/4123815682 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559101d846f0 con 0x559101fdd130 2023-10-30T12:29:13.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f73eb7fe640 1 -- 172.21.15.89:0/4123815682 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f73fc0226a0 con 0x559101fdd130 2023-10-30T12:29:13.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f741649f000 10 client.4683 fetch_fsmap learned FSMap version 12 2023-10-30T12:29:13.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f741649f000 1 -- 172.21.15.89:0/4123815682 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0x559101ae5cc0 con 0x559101fdd130 2023-10-30T12:29:13.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f73eb7fe640 1 -- 172.21.15.89:0/4123815682 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559101d846f0 con 0x559101fdd130 2023-10-30T12:29:13.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f73eb7fe640 1 -- 172.21.15.89:0/4123815682 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 12) v1 ==== 4468+0+0 (secure 0 0 0) 0x7f73fc022980 con 0x559101fdd130 2023-10-30T12:29:13.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f741649f000 10 client.4683 fetch_fsmap finished waiting for FSMap version 12 2023-10-30T12:29:13.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f741649f000 10 client.4683 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-30T12:29:13.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f741649f000 20 client.4683 populate_metadata read hostname 'smithi089' 2023-10-30T12:29:13.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f741649f000 1 --2- 172.21.15.89:0/4123815682 >> [v2:172.21.15.192:6834/1289220491,v1:172.21.15.192:6835/1289220491] conn(0x55910203c750 0x55910203cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f741649f000 4 client.4683 mds_command: new command op to 4253 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-30T12:29:13.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.995+0000 7f741649f000 1 -- 172.21.15.89:0/4123815682 --> [v2:172.21.15.192:6834/1289220491,v1:172.21.15.192:6835/1289220491] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559101de5720 con 0x55910203c750 2023-10-30T12:29:13.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.999+0000 7f7410eb9640 1 --2- 172.21.15.89:0/4123815682 >> [v2:172.21.15.192:6834/1289220491,v1:172.21.15.192:6835/1289220491] conn(0x55910203c750 0x55910203cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.999+0000 7f7410eb9640 1 --2- 172.21.15.89:0/4123815682 >> [v2:172.21.15.192:6834/1289220491,v1:172.21.15.192:6835/1289220491] conn(0x55910203c750 0x55910203cb30 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ac2523b3c403ea27 server_cookie=4d80394d84648eff in_seq=0 out_seq=0 2023-10-30T12:29:13.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.999+0000 7f73eb7fe640 10 client.4683 ms_handle_connect on v2:172.21.15.192:6834/1289220491 2023-10-30T12:29:13.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.999+0000 7f73eb7fe640 1 -- 172.21.15.89:0/4123815682 <== mds.2 v2:172.21.15.192:6834/1289220491 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x559101de5720 con 0x55910203c750 2023-10-30T12:29:13.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.999+0000 7f73eb7fe640 10 client.4683 handle_command_reply: tid=0 2023-10-30T12:29:13.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.999+0000 7f741649f000 1 client.4683 shutdown 2023-10-30T12:29:13.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.999+0000 7f741649f000 1 -- 172.21.15.89:0/4123815682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559101fdd130 msgr2=0x55910203d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.999+0000 7f741649f000 1 --2- 172.21.15.89:0/4123815682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559101fdd130 0x55910203d970 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f73fc003eb0 tx=0x7f73fc000f30 comp rx=0 tx=0).stop 2023-10-30T12:29:13.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.999+0000 7f741649f000 1 -- 172.21.15.89:0/4123815682 shutdown_connections 2023-10-30T12:29:13.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.999+0000 7f741649f000 1 --2- 172.21.15.89:0/4123815682 >> [v2:172.21.15.192:6834/1289220491,v1:172.21.15.192:6835/1289220491] conn(0x55910203c750 0x55910203cb30 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.999+0000 7f741649f000 1 --2- 172.21.15.89:0/4123815682 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x559101fdda50 0x55910203deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.999+0000 7f741649f000 1 --2- 172.21.15.89:0/4123815682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559101fdd130 0x55910203d970 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.999+0000 7f73eb7fe640 0 client.4683 ms_handle_reset on v2:172.21.15.192:6834/1289220491 2023-10-30T12:29:13.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.999+0000 7f741649f000 1 --2- 172.21.15.89:0/4123815682 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559101fdc810 0x559101fa19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.999+0000 7f741649f000 1 -- 172.21.15.89:0/4123815682 >> 172.21.15.89:0/4123815682 conn(0x559101f92fc0 msgr2=0x559101fe1490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:13.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.999+0000 7f741649f000 1 -- 172.21.15.89:0/4123815682 shutdown_connections 2023-10-30T12:29:13.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.999+0000 7f741649f000 1 -- 172.21.15.89:0/4123815682 wait complete. 2023-10-30T12:29:13.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.999+0000 7f741649f000 20 client.4683 trim_cache size 0 max 16384 2023-10-30T12:29:13.016 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-10-30T12:29:13.016 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "401641be-db6e-4c16-b3ac-fd84a86c4de4", 2023-10-30T12:29:13.017 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 2, 2023-10-30T12:29:13.017 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 4253, 2023-10-30T12:29:13.017 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2023-10-30T12:29:13.017 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2023-10-30T12:29:13.017 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2023-10-30T12:29:13.018 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 11.054384719, 2023-10-30T12:29:13.018 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 12, 2023-10-30T12:29:13.018 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 22, 2023-10-30T12:29:13.018 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 22, 2023-10-30T12:29:13.018 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 28.593791019000001 2023-10-30T12:29:13.019 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-10-30T12:29:13.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.999+0000 7f7411ebb640 1 -- 172.21.15.89:0/3373305388 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f73f80664a0 msgr2=0x7f73f8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.999+0000 7f7411ebb640 1 --2- 172.21.15.89:0/3373305388 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f73f80664a0 0x7f73f8068960 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f740c164ee0 tx=0x7f73fc009040 comp rx=0 tx=0).stop 2023-10-30T12:29:13.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.999+0000 7f7411ebb640 1 -- 172.21.15.89:0/3373305388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f740c0c5550 msgr2=0x7f740c14b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.999+0000 7f7411ebb640 1 --2- 172.21.15.89:0/3373305388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f740c0c5550 0x7f740c14b710 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f740800c4a0 tx=0x7f740800c960 comp rx=0 tx=0).stop 2023-10-30T12:29:13.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.999+0000 7f7411ebb640 1 -- 172.21.15.89:0/3373305388 shutdown_connections 2023-10-30T12:29:13.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.999+0000 7f7411ebb640 1 --2- 172.21.15.89:0/3373305388 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f73f80664a0 0x7f73f8068960 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.999+0000 7f7411ebb640 1 --2- 172.21.15.89:0/3373305388 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f740c0cb050 0x7f740c14c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.999+0000 7f7411ebb640 1 --2- 172.21.15.89:0/3373305388 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f740c0cab00 0x7f740c150d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.999+0000 7f7411ebb640 1 --2- 172.21.15.89:0/3373305388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f740c0c5550 0x7f740c14b710 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.999+0000 7f7411ebb640 1 -- 172.21.15.89:0/3373305388 >> 172.21.15.89:0/3373305388 conn(0x7f740c06bed0 msgr2=0x7f740c0cd930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:13.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.999+0000 7f7411ebb640 1 -- 172.21.15.89:0/3373305388 shutdown_connections 2023-10-30T12:29:13.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:12.999+0000 7f7411ebb640 1 -- 172.21.15.89:0/3373305388 wait complete. 2023-10-30T12:29:13.025 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-30T12:29:13.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.163+0000 7f074849c640 1 Processor -- start 2023-10-30T12:29:13.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.163+0000 7f074849c640 1 -- start start 2023-10-30T12:29:13.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.167+0000 7f074849c640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f07401570f0 0x7f074006ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.167+0000 7f074849c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f074006f370 0x7f07401640d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.167+0000 7f074849c640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f074006b3b0 con 0x7f07400b2300 2023-10-30T12:29:13.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.167+0000 7f074849c640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07400c38a0 con 0x7f074006f370 2023-10-30T12:29:13.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.167+0000 7f074849c640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07400c3b20 con 0x7f07401570f0 2023-10-30T12:29:13.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.167+0000 7f0746a39640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f074006f370 0x7f07401640d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.167+0000 7f0746a39640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f074006f370 0x7f07401640d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57680/0 (socket says 172.21.15.89:57680) 2023-10-30T12:29:13.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.167+0000 7f0746a39640 1 -- 172.21.15.89:0/4244870088 learned_addr learned my addr 172.21.15.89:0/4244870088 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:13.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.167+0000 7f0746238640 1 --2- 172.21.15.89:0/4244870088 >> v2:172.21.15.192:3300/0 conn(0x7f07401570f0 0x7f074006ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.167+0000 7f0746a39640 1 -- 172.21.15.89:0/4244870088 >> v2:172.21.15.192:3300/0 conn(0x7f07401570f0 msgr2=0x7f074006ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:13.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.167+0000 7f0746a39640 1 --2- 172.21.15.89:0/4244870088 >> v2:172.21.15.192:3300/0 conn(0x7f07401570f0 0x7f074006ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.167+0000 7f0746a39640 1 -- 172.21.15.89:0/4244870088 >> v1:172.21.15.192:6789/0 conn(0x7f07400b2300 legacy=0x7f0740071b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.167+0000 7f0746a39640 1 -- 172.21.15.89:0/4244870088 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07400ca9f0 con 0x7f074006f370 2023-10-30T12:29:13.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.171+0000 7f0746a39640 1 --2- 172.21.15.89:0/4244870088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f074006f370 0x7f07401640d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f07400ca680 tx=0x7f073801c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e75396eca48d52b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:13.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.171+0000 7f0745236640 1 -- 172.21.15.89:0/4244870088 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07380a5780 con 0x7f074006f370 2023-10-30T12:29:13.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.171+0000 7f0745236640 1 -- 172.21.15.89:0/4244870088 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07380a58e0 con 0x7f074006f370 2023-10-30T12:29:13.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.171+0000 7f0745236640 1 -- 172.21.15.89:0/4244870088 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07380a5c70 con 0x7f074006f370 2023-10-30T12:29:13.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.171+0000 7f074849c640 1 -- 172.21.15.89:0/4244870088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f074006f370 msgr2=0x7f07401640d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.171+0000 7f074849c640 1 --2- 172.21.15.89:0/4244870088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f074006f370 0x7f07401640d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f07400ca680 tx=0x7f073801c9d0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.171+0000 7f074849c640 1 -- 172.21.15.89:0/4244870088 shutdown_connections 2023-10-30T12:29:13.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.171+0000 7f074849c640 1 --2- 172.21.15.89:0/4244870088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f074006f370 0x7f07401640d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.171+0000 7f074849c640 1 --2- 172.21.15.89:0/4244870088 >> v2:172.21.15.192:3300/0 conn(0x7f07401570f0 0x7f074006ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.171+0000 7f074849c640 1 -- 172.21.15.89:0/4244870088 >> 172.21.15.89:0/4244870088 conn(0x7f0740069e90 msgr2=0x7f074006a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:13.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.171+0000 7f074849c640 1 -- 172.21.15.89:0/4244870088 shutdown_connections 2023-10-30T12:29:13.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.171+0000 7f074849c640 1 -- 172.21.15.89:0/4244870088 wait complete. 2023-10-30T12:29:13.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.171+0000 7f074849c640 1 Processor -- start 2023-10-30T12:29:13.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.171+0000 7f074849c640 1 -- start start 2023-10-30T12:29:13.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.175+0000 7f074849c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f074006f370 0x7f07401552b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.175+0000 7f074849c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07400b2300 0x7f0740148a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.175+0000 7f074849c640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f07401570f0 0x7f0740148fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.175+0000 7f074849c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0740159030 con 0x7f07400b2300 2023-10-30T12:29:13.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.175+0000 7f074849c640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0740157e70 con 0x7f074006f370 2023-10-30T12:29:13.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.175+0000 7f074849c640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f074014ecd0 con 0x7f07401570f0 2023-10-30T12:29:13.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.175+0000 7f0745a37640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07400b2300 0x7f0740148a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.175+0000 7f0745a37640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07400b2300 0x7f0740148a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57688/0 (socket says 172.21.15.89:57688) 2023-10-30T12:29:13.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.175+0000 7f0745a37640 1 -- 172.21.15.89:0/3921826349 learned_addr learned my addr 172.21.15.89:0/3921826349 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:13.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.175+0000 7f0746238640 1 --2- 172.21.15.89:0/3921826349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f074006f370 0x7f07401552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.175+0000 7f0746a39640 1 --2- 172.21.15.89:0/3921826349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f07401570f0 0x7f0740148fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.175+0000 7f0745a37640 1 -- 172.21.15.89:0/3921826349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f07401570f0 msgr2=0x7f0740148fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.175+0000 7f0745a37640 1 --2- 172.21.15.89:0/3921826349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f07401570f0 0x7f0740148fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.175+0000 7f0745a37640 1 -- 172.21.15.89:0/3921826349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f074006f370 msgr2=0x7f07401552b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.175+0000 7f0745a37640 1 --2- 172.21.15.89:0/3921826349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f074006f370 0x7f07401552b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.175+0000 7f0745a37640 1 -- 172.21.15.89:0/3921826349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0738009b30 con 0x7f07400b2300 2023-10-30T12:29:13.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.175+0000 7f0746a39640 1 --2- 172.21.15.89:0/3921826349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f07401570f0 0x7f0740148fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:13.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.175+0000 7f0745a37640 1 --2- 172.21.15.89:0/3921826349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07400b2300 0x7f0740148a90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f073000c3c0 tx=0x7f073000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:13.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.175+0000 7f0736ffd640 1 -- 172.21.15.89:0/3921826349 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07300052d0 con 0x7f07400b2300 2023-10-30T12:29:13.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.175+0000 7f0736ffd640 1 -- 172.21.15.89:0/3921826349 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0730005430 con 0x7f07400b2300 2023-10-30T12:29:13.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.175+0000 7f0736ffd640 1 -- 172.21.15.89:0/3921826349 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0730003840 con 0x7f07400b2300 2023-10-30T12:29:13.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.175+0000 7f074849c640 1 -- 172.21.15.89:0/3921826349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f074014e790 con 0x7f07400b2300 2023-10-30T12:29:13.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.175+0000 7f074849c640 1 -- 172.21.15.89:0/3921826349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07401578b0 con 0x7f07400b2300 2023-10-30T12:29:13.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.179+0000 7f0736ffd640 1 -- 172.21.15.89:0/3921826349 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f07300039a0 con 0x7f07400b2300 2023-10-30T12:29:13.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.179+0000 7f0736ffd640 1 --2- 172.21.15.89:0/3921826349 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f07240664d0 0x7f0724068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.179+0000 7f0736ffd640 1 -- 172.21.15.89:0/3921826349 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0730083330 con 0x7f07400b2300 2023-10-30T12:29:13.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.179+0000 7f074b831000 1 Processor -- start 2023-10-30T12:29:13.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.179+0000 7f0746238640 1 --2- 172.21.15.89:0/3921826349 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f07240664d0 0x7f0724068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.179+0000 7f074b831000 1 -- start start 2023-10-30T12:29:13.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.179+0000 7f0746238640 1 --2- 172.21.15.89:0/3921826349 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f07240664d0 0x7f0724068990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f073c0079c0 tx=0x7f073c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:13.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.179+0000 7f074b831000 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b901778810 0x55b901778bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.179+0000 7f074b831000 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b901779130 0x55b901779510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.179+0000 7f074b831000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b901779a50 0x55b901783aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.179+0000 7f074b831000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b90152ccc0 con 0x55b901779a50 2023-10-30T12:29:13.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.179+0000 7f074b831000 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b90158f920 con 0x55b901778810 2023-10-30T12:29:13.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.179+0000 7f074b831000 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b90158ce50 con 0x55b901779130 2023-10-30T12:29:13.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.179+0000 7f0746a39640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b901779a50 0x55b901783aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.179+0000 7f0746a39640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b901779a50 0x55b901783aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57702/0 (socket says 172.21.15.89:57702) 2023-10-30T12:29:13.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.179+0000 7f0746a39640 1 -- 172.21.15.89:0/1596305998 learned_addr learned my addr 172.21.15.89:0/1596305998 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:13.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.179+0000 7f0746238640 1 --2- 172.21.15.89:0/1596305998 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b901778810 0x55b901778bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.179+0000 7f0745a37640 1 --2- 172.21.15.89:0/1596305998 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b901779130 0x55b901779510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.179+0000 7f0746a39640 1 -- 172.21.15.89:0/1596305998 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b901779130 msgr2=0x55b901779510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:13.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.179+0000 7f0746a39640 1 --2- 172.21.15.89:0/1596305998 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b901779130 0x55b901779510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.179+0000 7f0746a39640 1 -- 172.21.15.89:0/1596305998 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b901778810 msgr2=0x55b901778bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:13.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.179+0000 7f0746a39640 1 --2- 172.21.15.89:0/1596305998 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b901778810 0x55b901778bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.179+0000 7f0746a39640 1 -- 172.21.15.89:0/1596305998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b901281890 con 0x55b901779a50 2023-10-30T12:29:13.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.179+0000 7f0746a39640 1 --2- 172.21.15.89:0/1596305998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b901779a50 0x55b901783aa0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f073801ca10 tx=0x7f0738004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d52d2c95bcf7112 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:13.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f0723fff640 1 -- 172.21.15.89:0/1596305998 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07380a5a50 con 0x55b901779a50 2023-10-30T12:29:13.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f0723fff640 1 -- 172.21.15.89:0/1596305998 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07380a5bb0 con 0x55b901779a50 2023-10-30T12:29:13.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f0723fff640 1 -- 172.21.15.89:0/1596305998 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07380b6420 con 0x55b901779a50 2023-10-30T12:29:13.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f074b831000 1 -- 172.21.15.89:0/1596305998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b901779a50 msgr2=0x55b901783aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f074b831000 1 --2- 172.21.15.89:0/1596305998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b901779a50 0x55b901783aa0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f073801ca10 tx=0x7f0738004ac0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f074b831000 1 -- 172.21.15.89:0/1596305998 shutdown_connections 2023-10-30T12:29:13.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f074b831000 1 --2- 172.21.15.89:0/1596305998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b901779a50 0x55b901783aa0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f074b831000 1 --2- 172.21.15.89:0/1596305998 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b901779130 0x55b901779510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f074b831000 1 --2- 172.21.15.89:0/1596305998 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b901778810 0x55b901778bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f074b831000 1 -- 172.21.15.89:0/1596305998 >> 172.21.15.89:0/1596305998 conn(0x55b90172efc0 msgr2=0x55b9017765d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:13.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f074b831000 1 -- 172.21.15.89:0/1596305998 shutdown_connections 2023-10-30T12:29:13.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f074b831000 1 -- 172.21.15.89:0/1596305998 wait complete. 2023-10-30T12:29:13.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f074b831000 1 -- start start 2023-10-30T12:29:13.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f074b831000 1 Processor -- start 2023-10-30T12:29:13.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f074b831000 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b901778810 0x55b901772f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f074b831000 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b901779130 0x55b901773490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f074b831000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b901779a50 0x55b90177ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f074b831000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b901588740 con 0x55b901779a50 2023-10-30T12:29:13.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f074b831000 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b901547d10 con 0x55b901779130 2023-10-30T12:29:13.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f074b831000 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b9015479b0 con 0x55b901778810 2023-10-30T12:29:13.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f074b831000 10 client.0 _ll_register_callbacks cb 0x55b901723390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T12:29:13.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f0746a39640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b901779a50 0x55b90177ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f0746a39640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b901779a50 0x55b90177ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57708/0 (socket says 172.21.15.89:57708) 2023-10-30T12:29:13.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f0746a39640 1 -- 172.21.15.89:0/825924025 learned_addr learned my addr 172.21.15.89:0/825924025 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:13.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f0746238640 1 --2- 172.21.15.89:0/825924025 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b901778810 0x55b901772f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f0745a37640 1 --2- 172.21.15.89:0/825924025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b901779130 0x55b901773490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f0746a39640 1 -- 172.21.15.89:0/825924025 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b901778810 msgr2=0x55b901772f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:13.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f0746a39640 1 --2- 172.21.15.89:0/825924025 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b901778810 0x55b901772f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f0746a39640 1 -- 172.21.15.89:0/825924025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b901779130 msgr2=0x55b901773490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:13.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f0746a39640 1 --2- 172.21.15.89:0/825924025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b901779130 0x55b901773490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.183+0000 7f0746a39640 1 -- 172.21.15.89:0/825924025 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07400ca9f0 con 0x55b901779a50 2023-10-30T12:29:13.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.187+0000 7f0746a39640 1 --2- 172.21.15.89:0/825924025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b901779a50 0x55b90177ec30 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f073801b080 tx=0x7f073801b210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f76a1775ceaffa08 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:13.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.187+0000 7f07217fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-10-30T12:29:13.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.187+0000 7f07217fa640 1 -- 172.21.15.89:0/825924025 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07380039a0 con 0x55b901779a50 2023-10-30T12:29:13.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.187+0000 7f07217fa640 1 -- 172.21.15.89:0/825924025 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0738003b00 con 0x55b901779a50 2023-10-30T12:29:13.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.187+0000 7f074b831000 1 -- 172.21.15.89:0/825924025 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b90153c370 con 0x55b901779a50 2023-10-30T12:29:13.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.187+0000 7f07217fa640 1 -- 172.21.15.89:0/825924025 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0738003cc0 con 0x55b901779a50 2023-10-30T12:29:13.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.187+0000 7f074b831000 10 client.4695 fetch_fsmap learned FSMap version 12 2023-10-30T12:29:13.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.187+0000 7f074b831000 1 -- 172.21.15.89:0/825924025 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0x55b90154a460 con 0x55b901779a50 2023-10-30T12:29:13.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.187+0000 7f07217fa640 1 -- 172.21.15.89:0/825924025 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b90153c370 con 0x55b901779a50 2023-10-30T12:29:13.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.187+0000 7f07217fa640 1 -- 172.21.15.89:0/825924025 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 12) v1 ==== 4468+0+0 (secure 0 0 0) 0x7f0738005d20 con 0x55b901779a50 2023-10-30T12:29:13.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.187+0000 7f074b831000 10 client.4695 fetch_fsmap finished waiting for FSMap version 12 2023-10-30T12:29:13.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.187+0000 7f074b831000 10 client.4695 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-30T12:29:13.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.187+0000 7f074b831000 20 client.4695 populate_metadata read hostname 'smithi089' 2023-10-30T12:29:13.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.187+0000 7f074b831000 1 --2- 172.21.15.89:0/825924025 >> [v2:172.21.15.192:6838/3339589587,v1:172.21.15.192:6839/3339589587] conn(0x55b90177f9a0 0x55b90177fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.187+0000 7f074b831000 4 client.4695 mds_command: new command op to 4262 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:29:13.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.187+0000 7f074b831000 1 -- 172.21.15.89:0/825924025 --> [v2:172.21.15.192:6838/3339589587,v1:172.21.15.192:6839/3339589587] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b9015222d0 con 0x55b90177f9a0 2023-10-30T12:29:13.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.187+0000 7f0746238640 1 --2- 172.21.15.89:0/825924025 >> [v2:172.21.15.192:6838/3339589587,v1:172.21.15.192:6839/3339589587] conn(0x55b90177f9a0 0x55b90177fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.187+0000 7f0746238640 1 --2- 172.21.15.89:0/825924025 >> [v2:172.21.15.192:6838/3339589587,v1:172.21.15.192:6839/3339589587] conn(0x55b90177f9a0 0x55b90177fd80 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c20e29e1d789d739 server_cookie=7666c3eb4ff7e645 in_seq=0 out_seq=0 2023-10-30T12:29:13.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.187+0000 7f07217fa640 10 client.4695 ms_handle_connect on v2:172.21.15.192:6838/3339589587 2023-10-30T12:29:13.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.191+0000 7f07217fa640 1 -- 172.21.15.89:0/825924025 <== mds.1 v2:172.21.15.192:6838/3339589587 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b9015222d0 con 0x55b90177f9a0 2023-10-30T12:29:13.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.191+0000 7f07217fa640 10 client.4695 handle_command_reply: tid=0 2023-10-30T12:29:13.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.191+0000 7f074b831000 1 client.4695 shutdown 2023-10-30T12:29:13.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.191+0000 7f074b831000 1 -- 172.21.15.89:0/825924025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b901779a50 msgr2=0x55b90177ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.191+0000 7f074b831000 1 --2- 172.21.15.89:0/825924025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b901779a50 0x55b90177ec30 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f073801b080 tx=0x7f073801b210 comp rx=0 tx=0).stop 2023-10-30T12:29:13.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.191+0000 7f074b831000 1 -- 172.21.15.89:0/825924025 shutdown_connections 2023-10-30T12:29:13.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.191+0000 7f074b831000 1 --2- 172.21.15.89:0/825924025 >> [v2:172.21.15.192:6838/3339589587,v1:172.21.15.192:6839/3339589587] conn(0x55b90177f9a0 0x55b90177fd80 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.191+0000 7f074b831000 1 --2- 172.21.15.89:0/825924025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b901779130 0x55b901773490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.191+0000 7f074b831000 1 --2- 172.21.15.89:0/825924025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b901779a50 0x55b90177ec30 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.191+0000 7f07217fa640 0 client.4695 ms_handle_reset on v2:172.21.15.192:6838/3339589587 2023-10-30T12:29:13.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.191+0000 7f074b831000 1 --2- 172.21.15.89:0/825924025 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b901778810 0x55b901772f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.191+0000 7f074b831000 1 -- 172.21.15.89:0/825924025 >> 172.21.15.89:0/825924025 conn(0x55b90172efc0 msgr2=0x55b901778090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:13.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.191+0000 7f074b831000 1 -- 172.21.15.89:0/825924025 shutdown_connections 2023-10-30T12:29:13.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.191+0000 7f074b831000 1 -- 172.21.15.89:0/825924025 wait complete. 2023-10-30T12:29:13.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.191+0000 7f074b831000 20 client.4695 trim_cache size 0 max 16384 2023-10-30T12:29:13.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.203+0000 7f074b831000 1 Processor -- start 2023-10-30T12:29:13.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.203+0000 7f074b831000 1 -- start start 2023-10-30T12:29:13.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f074b831000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b901778810 0x55b901780d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f074b831000 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b901779130 0x55b9017812d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f074b831000 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b901779a50 0x55b9017d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f074b831000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b90178aba0 con 0x55b901778810 2023-10-30T12:29:13.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f074b831000 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b901242940 con 0x55b901779130 2023-10-30T12:29:13.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f074b831000 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b9015c9ba0 con 0x55b901779a50 2023-10-30T12:29:13.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f0746238640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b901778810 0x55b901780d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f0746238640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b901778810 0x55b901780d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57714/0 (socket says 172.21.15.89:57714) 2023-10-30T12:29:13.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f0746238640 1 -- 172.21.15.89:0/565085330 learned_addr learned my addr 172.21.15.89:0/565085330 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:13.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f0746a39640 1 --2- 172.21.15.89:0/565085330 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b901779a50 0x55b9017d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f0746238640 1 -- 172.21.15.89:0/565085330 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b901779a50 msgr2=0x55b9017d2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:13.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f0746238640 1 --2- 172.21.15.89:0/565085330 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b901779a50 0x55b9017d2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f0746238640 1 -- 172.21.15.89:0/565085330 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b901779130 msgr2=0x55b9017812d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:13.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f0745a37640 1 --2- 172.21.15.89:0/565085330 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b901779130 0x55b9017812d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f0746238640 1 --2- 172.21.15.89:0/565085330 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b901779130 0x55b9017812d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f0746238640 1 -- 172.21.15.89:0/565085330 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b901530760 con 0x55b901778810 2023-10-30T12:29:13.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f0745a37640 1 --2- 172.21.15.89:0/565085330 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b901779130 0x55b9017812d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:29:13.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f0746238640 1 --2- 172.21.15.89:0/565085330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b901778810 0x55b901780d90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x55b901589c80 tx=0x7f073c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46a3829597178ec5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:13.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f0734ff9640 1 -- 172.21.15.89:0/565085330 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f073c029070 con 0x55b901778810 2023-10-30T12:29:13.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f0734ff9640 1 -- 172.21.15.89:0/565085330 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f073c010d90 con 0x55b901778810 2023-10-30T12:29:13.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f0734ff9640 1 -- 172.21.15.89:0/565085330 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f073c003a70 con 0x55b901778810 2023-10-30T12:29:13.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f074b831000 1 -- 172.21.15.89:0/565085330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b901778810 msgr2=0x55b901780d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f074b831000 1 --2- 172.21.15.89:0/565085330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b901778810 0x55b901780d90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x55b901589c80 tx=0x7f073c013000 comp rx=0 tx=0).stop 2023-10-30T12:29:13.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f074b831000 1 -- 172.21.15.89:0/565085330 shutdown_connections 2023-10-30T12:29:13.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f074b831000 1 --2- 172.21.15.89:0/565085330 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b901779130 0x55b9017812d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f074b831000 1 --2- 172.21.15.89:0/565085330 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b901779a50 0x55b9017d2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f074b831000 1 --2- 172.21.15.89:0/565085330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b901778810 0x55b901780d90 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f074b831000 1 -- 172.21.15.89:0/565085330 >> 172.21.15.89:0/565085330 conn(0x55b90172efc0 msgr2=0x55b9017d0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:13.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f074b831000 1 -- 172.21.15.89:0/565085330 shutdown_connections 2023-10-30T12:29:13.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f074b831000 1 -- 172.21.15.89:0/565085330 wait complete. 2023-10-30T12:29:13.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f074b831000 1 -- start start 2023-10-30T12:29:13.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.207+0000 7f074b831000 1 Processor -- start 2023-10-30T12:29:13.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f074b831000 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b901778810 0x55b90177c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f074b831000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b901779130 0x55b90177cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f074b831000 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b901779a50 0x55b90177d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f074b831000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b901585b40 con 0x55b901779130 2023-10-30T12:29:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f074b831000 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b901777a00 con 0x55b901779a50 2023-10-30T12:29:13.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f074b831000 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b9015840b0 con 0x55b901778810 2023-10-30T12:29:13.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f074b831000 10 client.0 _ll_register_callbacks cb 0x7f0738009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T12:29:13.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f0745a37640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b901779130 0x55b90177cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f0745a37640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b901779130 0x55b90177cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57720/0 (socket says 172.21.15.89:57720) 2023-10-30T12:29:13.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f0745a37640 1 -- 172.21.15.89:0/566639689 learned_addr learned my addr 172.21.15.89:0/566639689 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:13.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f0746a39640 1 --2- 172.21.15.89:0/566639689 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b901779a50 0x55b90177d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f0746238640 1 --2- 172.21.15.89:0/566639689 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b901778810 0x55b90177c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f0745a37640 1 -- 172.21.15.89:0/566639689 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b901778810 msgr2=0x55b90177c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:13.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f0745a37640 1 --2- 172.21.15.89:0/566639689 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b901778810 0x55b90177c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f0745a37640 1 -- 172.21.15.89:0/566639689 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b901779a50 msgr2=0x55b90177d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:13.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f0745a37640 1 --2- 172.21.15.89:0/566639689 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b901779a50 0x55b90177d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f0745a37640 1 -- 172.21.15.89:0/566639689 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f073c00e040 con 0x55b901779130 2023-10-30T12:29:13.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f0745a37640 1 --2- 172.21.15.89:0/566639689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b901779130 0x55b90177cc60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f073005c230 tx=0x7f07300586f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94ea1b64d7b2fe7d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:13.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f07217fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-10-30T12:29:13.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f07217fa640 1 -- 172.21.15.89:0/566639689 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0730058150 con 0x55b901779130 2023-10-30T12:29:13.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f07217fa640 1 -- 172.21.15.89:0/566639689 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f073007ce20 con 0x55b901779130 2023-10-30T12:29:13.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f074b831000 1 -- 172.21.15.89:0/566639689 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b9015206f0 con 0x55b901779130 2023-10-30T12:29:13.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f07217fa640 1 -- 172.21.15.89:0/566639689 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f07300582b0 con 0x55b901779130 2023-10-30T12:29:13.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f074b831000 10 client.4701 fetch_fsmap learned FSMap version 12 2023-10-30T12:29:13.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f074b831000 1 -- 172.21.15.89:0/566639689 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0x55b901281cc0 con 0x55b901779130 2023-10-30T12:29:13.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f07217fa640 1 -- 172.21.15.89:0/566639689 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b9015206f0 con 0x55b901779130 2023-10-30T12:29:13.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f07217fa640 1 -- 172.21.15.89:0/566639689 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 12) v1 ==== 4468+0+0 (secure 0 0 0) 0x7f073005c2d0 con 0x55b901779130 2023-10-30T12:29:13.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f074b831000 10 client.4701 fetch_fsmap finished waiting for FSMap version 12 2023-10-30T12:29:13.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f074b831000 10 client.4701 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-30T12:29:13.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f074b831000 20 client.4701 populate_metadata read hostname 'smithi089' 2023-10-30T12:29:13.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f074b831000 1 --2- 172.21.15.89:0/566639689 >> [v2:172.21.15.192:6838/3339589587,v1:172.21.15.192:6839/3339589587] conn(0x55b9017e1d90 0x55b9017e4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f074b831000 4 client.4701 mds_command: new command op to 4262 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-30T12:29:13.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f074b831000 1 -- 172.21.15.89:0/566639689 --> [v2:172.21.15.192:6838/3339589587,v1:172.21.15.192:6839/3339589587] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b901581720 con 0x55b9017e1d90 2023-10-30T12:29:13.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f0746238640 1 --2- 172.21.15.89:0/566639689 >> [v2:172.21.15.192:6838/3339589587,v1:172.21.15.192:6839/3339589587] conn(0x55b9017e1d90 0x55b9017e4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f07217fa640 10 client.4701 ms_handle_connect on v2:172.21.15.192:6838/3339589587 2023-10-30T12:29:13.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f0746238640 1 --2- 172.21.15.89:0/566639689 >> [v2:172.21.15.192:6838/3339589587,v1:172.21.15.192:6839/3339589587] conn(0x55b9017e1d90 0x55b9017e4180 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=24046aac2bdd81f6 server_cookie=cde13602c1c047e4 in_seq=0 out_seq=0 2023-10-30T12:29:13.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f07217fa640 1 -- 172.21.15.89:0/566639689 <== mds.1 v2:172.21.15.192:6838/3339589587 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55b901581720 con 0x55b9017e1d90 2023-10-30T12:29:13.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f07217fa640 10 client.4701 handle_command_reply: tid=0 2023-10-30T12:29:13.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f074b831000 1 client.4701 shutdown 2023-10-30T12:29:13.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f074b831000 1 -- 172.21.15.89:0/566639689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b901779130 msgr2=0x55b90177cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f074b831000 1 --2- 172.21.15.89:0/566639689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b901779130 0x55b90177cc60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f073005c230 tx=0x7f07300586f0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f074b831000 1 -- 172.21.15.89:0/566639689 shutdown_connections 2023-10-30T12:29:13.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f074b831000 1 --2- 172.21.15.89:0/566639689 >> [v2:172.21.15.192:6838/3339589587,v1:172.21.15.192:6839/3339589587] conn(0x55b9017e1d90 0x55b9017e4180 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f074b831000 1 --2- 172.21.15.89:0/566639689 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b901779a50 0x55b90177d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f07217fa640 0 client.4701 ms_handle_reset on v2:172.21.15.192:6838/3339589587 2023-10-30T12:29:13.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f074b831000 1 --2- 172.21.15.89:0/566639689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b901779130 0x55b90177cc60 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f074b831000 1 --2- 172.21.15.89:0/566639689 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55b901778810 0x55b90177c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f074b831000 1 -- 172.21.15.89:0/566639689 >> 172.21.15.89:0/566639689 conn(0x55b90172efc0 msgr2=0x55b90173dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:13.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f074b831000 1 -- 172.21.15.89:0/566639689 shutdown_connections 2023-10-30T12:29:13.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f074b831000 1 -- 172.21.15.89:0/566639689 wait complete. 2023-10-30T12:29:13.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.211+0000 7f074b831000 20 client.4701 trim_cache size 0 max 16384 2023-10-30T12:29:13.229 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-10-30T12:29:13.229 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "401641be-db6e-4c16-b3ac-fd84a86c4de4", 2023-10-30T12:29:13.229 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 1, 2023-10-30T12:29:13.230 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 4262, 2023-10-30T12:29:13.230 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2023-10-30T12:29:13.230 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2023-10-30T12:29:13.230 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2023-10-30T12:29:13.230 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 16.808336141000002, 2023-10-30T12:29:13.230 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 12, 2023-10-30T12:29:13.231 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 22, 2023-10-30T12:29:13.231 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 22, 2023-10-30T12:29:13.231 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 28.799103500000001 2023-10-30T12:29:13.231 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-10-30T12:29:13.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.215+0000 7f074849c640 1 -- 172.21.15.89:0/3921826349 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f07240664d0 msgr2=0x7f0724068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.215+0000 7f074849c640 1 --2- 172.21.15.89:0/3921826349 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f07240664d0 0x7f0724068990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f073c0079c0 tx=0x7f073c008040 comp rx=0 tx=0).stop 2023-10-30T12:29:13.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.215+0000 7f074849c640 1 -- 172.21.15.89:0/3921826349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07400b2300 msgr2=0x7f0740148a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.215+0000 7f074849c640 1 --2- 172.21.15.89:0/3921826349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07400b2300 0x7f0740148a90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f073000c3c0 tx=0x7f073000c880 comp rx=0 tx=0).stop 2023-10-30T12:29:13.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.215+0000 7f074849c640 1 -- 172.21.15.89:0/3921826349 shutdown_connections 2023-10-30T12:29:13.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.215+0000 7f074849c640 1 --2- 172.21.15.89:0/3921826349 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f07240664d0 0x7f0724068990 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.215+0000 7f074849c640 1 --2- 172.21.15.89:0/3921826349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f07401570f0 0x7f0740148fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.215+0000 7f074849c640 1 --2- 172.21.15.89:0/3921826349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07400b2300 0x7f0740148a90 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.215+0000 7f074849c640 1 --2- 172.21.15.89:0/3921826349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f074006f370 0x7f07401552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.215+0000 7f074849c640 1 -- 172.21.15.89:0/3921826349 >> 172.21.15.89:0/3921826349 conn(0x7f0740069e90 msgr2=0x7f074014ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:13.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.215+0000 7f074849c640 1 -- 172.21.15.89:0/3921826349 shutdown_connections 2023-10-30T12:29:13.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.215+0000 7f074849c640 1 -- 172.21.15.89:0/3921826349 wait complete. 2023-10-30T12:29:13.235 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2023-10-30T12:29:13.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.371+0000 7f778269c640 1 Processor -- start 2023-10-30T12:29:13.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.371+0000 7f778269c640 1 -- start start 2023-10-30T12:29:13.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f778269c640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f777c13fd10 0x7f777c1565a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f778269c640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f777c072030 con 0x7f777c0c6f60 2023-10-30T12:29:13.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f778269c640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f777c070010 con 0x7f777c0c6b80 2023-10-30T12:29:13.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f778269c640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f777c072260 con 0x7f777c13fd10 2023-10-30T12:29:13.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f777b7fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f777c0c6f60 0x7f777c13f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58344/0 (socket says 172.21.15.89:58344) 2023-10-30T12:29:13.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f777b7fe640 1 -- 172.21.15.89:0/1359921741 learned_addr learned my addr 172.21.15.89:0/1359921741 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:29:13.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f7780c39640 1 --2- 172.21.15.89:0/1359921741 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f777c13fd10 0x7f777c1565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f777affd640 1 -- 172.21.15.89:0/1359921741 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3577409010 0 0) 0x7f777c072030 con 0x7f777c0c6f60 2023-10-30T12:29:13.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f777affd640 1 -- 172.21.15.89:0/1359921741 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7764005740 con 0x7f777c0c6f60 2023-10-30T12:29:13.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f777affd640 1 -- 172.21.15.89:0/1359921741 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1869234135 0 0) 0x7f7764005740 con 0x7f777c0c6f60 2023-10-30T12:29:13.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f777affd640 1 -- 172.21.15.89:0/1359921741 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f777c13fd10 msgr2=0x7f777c1565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:13.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f777affd640 1 --2- 172.21.15.89:0/1359921741 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f777c13fd10 0x7f777c1565a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f777affd640 1 -- 172.21.15.89:0/1359921741 >> v1:172.21.15.192:6789/0 conn(0x7f777c0c6b80 legacy=0x7f777c071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f777affd640 1 -- 172.21.15.89:0/1359921741 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777c06f6e0 con 0x7f777c0c6f60 2023-10-30T12:29:13.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f777affd640 1 -- 172.21.15.89:0/1359921741 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f7770001680 con 0x7f777c0c6f60 2023-10-30T12:29:13.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f7780c39640 1 --2- 172.21.15.89:0/1359921741 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f777c13fd10 0x7f777c1565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:29:13.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f777affd640 1 -- 172.21.15.89:0/1359921741 >> v1:172.21.15.89:6789/0 conn(0x7f777c0c6f60 legacy=0x7f777c13f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f777affd640 1 --2- 172.21.15.89:0/1359921741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7764005bc0 0x7f7764005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f777affd640 1 --2- 172.21.15.89:0/1359921741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7764006570 0x7f7764006a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f777affd640 1 --2- 172.21.15.89:0/1359921741 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7764007240 0x7f776400dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f777affd640 1 -- 172.21.15.89:0/1359921741 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777c072260 con 0x7f7764006570 2023-10-30T12:29:13.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f777affd640 1 -- 172.21.15.89:0/1359921741 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7764007840 con 0x7f7764005bc0 2023-10-30T12:29:13.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f777affd640 1 -- 172.21.15.89:0/1359921741 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f776400e110 con 0x7f7764007240 2023-10-30T12:29:13.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f777b7fe640 1 --2- 172.21.15.89:0/1359921741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7764006570 0x7f7764006a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f777bfff640 1 --2- 172.21.15.89:0/1359921741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7764005bc0 0x7f7764005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f7780c39640 1 --2- 172.21.15.89:0/1359921741 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7764007240 0x7f776400dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f777b7fe640 1 -- 172.21.15.89:0/1359921741 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7764007240 msgr2=0x7f776400dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:13.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f777b7fe640 1 --2- 172.21.15.89:0/1359921741 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7764007240 0x7f776400dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f777b7fe640 1 -- 172.21.15.89:0/1359921741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7764005bc0 msgr2=0x7f7764005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:13.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f777b7fe640 1 --2- 172.21.15.89:0/1359921741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7764005bc0 0x7f7764005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f777b7fe640 1 -- 172.21.15.89:0/1359921741 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7774009a70 con 0x7f7764006570 2023-10-30T12:29:13.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.375+0000 7f7780c39640 1 --2- 172.21.15.89:0/1359921741 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7764007240 0x7f776400dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:13.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.379+0000 7f777b7fe640 1 --2- 172.21.15.89:0/1359921741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7764006570 0x7f7764006a40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f777000c160 tx=0x7f777001cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a8bd951bca32b2e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:13.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.379+0000 7f777affd640 1 -- 172.21.15.89:0/1359921741 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7770005ae0 con 0x7f7764006570 2023-10-30T12:29:13.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.379+0000 7f777affd640 1 -- 172.21.15.89:0/1359921741 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7770005c60 con 0x7f7764006570 2023-10-30T12:29:13.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.379+0000 7f777affd640 1 -- 172.21.15.89:0/1359921741 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77700a5880 con 0x7f7764006570 2023-10-30T12:29:13.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.379+0000 7f778269c640 1 -- 172.21.15.89:0/1359921741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7764006570 msgr2=0x7f7764006a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.379+0000 7f778269c640 1 --2- 172.21.15.89:0/1359921741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7764006570 0x7f7764006a40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f777000c160 tx=0x7f777001cb80 comp rx=0 tx=0).stop 2023-10-30T12:29:13.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.379+0000 7f7780c39640 1 -- 172.21.15.89:0/1359921741 reap_dead start 2023-10-30T12:29:13.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.379+0000 7f778269c640 1 -- 172.21.15.89:0/1359921741 shutdown_connections 2023-10-30T12:29:13.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.379+0000 7f778269c640 1 -- 172.21.15.89:0/1359921741 >> 172.21.15.89:0/1359921741 conn(0x7f777c0c7910 msgr2=0x7f777c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:13.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.379+0000 7f778269c640 1 -- 172.21.15.89:0/1359921741 shutdown_connections 2023-10-30T12:29:13.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.379+0000 7f778269c640 1 -- 172.21.15.89:0/1359921741 wait complete. 2023-10-30T12:29:13.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.383+0000 7f778269c640 1 Processor -- start 2023-10-30T12:29:13.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.383+0000 7f778269c640 1 -- start start 2023-10-30T12:29:13.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.383+0000 7f778269c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7764006570 0x7f777c1396c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.383+0000 7f778269c640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f777c155d20 0x7f777c1e43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.383+0000 7f778269c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f777c1e48f0 0x7f777c137580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.383+0000 7f778269c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777c1574d0 con 0x7f777c1e48f0 2023-10-30T12:29:13.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.383+0000 7f778269c640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777c156270 con 0x7f7764006570 2023-10-30T12:29:13.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.383+0000 7f778269c640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f777c1564a0 con 0x7f777c155d20 2023-10-30T12:29:13.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.383+0000 7f7780c39640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f777c1e48f0 0x7f777c137580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.383+0000 7f7780c39640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f777c1e48f0 0x7f777c137580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57734/0 (socket says 172.21.15.89:57734) 2023-10-30T12:29:13.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.383+0000 7f7780c39640 1 -- 172.21.15.89:0/3677098267 learned_addr learned my addr 172.21.15.89:0/3677098267 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:13.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.383+0000 7f777bfff640 1 --2- 172.21.15.89:0/3677098267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7764006570 0x7f777c1396c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.383+0000 7f777b7fe640 1 --2- 172.21.15.89:0/3677098267 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f777c155d20 0x7f777c1e43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.383+0000 7f7780c39640 1 -- 172.21.15.89:0/3677098267 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f777c155d20 msgr2=0x7f777c1e43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.383+0000 7f7780c39640 1 --2- 172.21.15.89:0/3677098267 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f777c155d20 0x7f777c1e43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.383+0000 7f7780c39640 1 -- 172.21.15.89:0/3677098267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7764006570 msgr2=0x7f777c1396c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.383+0000 7f7780c39640 1 --2- 172.21.15.89:0/3677098267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7764006570 0x7f777c1396c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.383+0000 7f7780c39640 1 -- 172.21.15.89:0/3677098267 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777c156780 con 0x7f777c1e48f0 2023-10-30T12:29:13.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.383+0000 7f7780c39640 1 --2- 172.21.15.89:0/3677098267 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f777c1e48f0 0x7f777c137580 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f777c0c21e0 tx=0x7f77740117d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:13.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.383+0000 7f7778ff9640 1 -- 172.21.15.89:0/3677098267 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777400e070 con 0x7f777c1e48f0 2023-10-30T12:29:13.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.383+0000 7f7778ff9640 1 -- 172.21.15.89:0/3677098267 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77740039a0 con 0x7f777c1e48f0 2023-10-30T12:29:13.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.383+0000 7f7778ff9640 1 -- 172.21.15.89:0/3677098267 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7774003b50 con 0x7f777c1e48f0 2023-10-30T12:29:13.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.383+0000 7f778269c640 1 -- 172.21.15.89:0/3677098267 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f777c156e10 con 0x7f777c1e48f0 2023-10-30T12:29:13.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.383+0000 7f778269c640 1 -- 172.21.15.89:0/3677098267 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f777c139c00 con 0x7f777c1e48f0 2023-10-30T12:29:13.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.387+0000 7f7778ff9640 1 -- 172.21.15.89:0/3677098267 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7774003cb0 con 0x7f777c1e48f0 2023-10-30T12:29:13.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.387+0000 7f7778ff9640 1 --2- 172.21.15.89:0/3677098267 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f77680664a0 0x7f7768068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.387+0000 7f7778ff9640 1 -- 172.21.15.89:0/3677098267 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f77740845f0 con 0x7f777c1e48f0 2023-10-30T12:29:13.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.387+0000 7f7785a2b000 1 Processor -- start 2023-10-30T12:29:13.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.387+0000 7f777bfff640 1 --2- 172.21.15.89:0/3677098267 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f77680664a0 0x7f7768068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.387+0000 7f7785a2b000 1 -- start start 2023-10-30T12:29:13.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.387+0000 7f7785a2b000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1f4e76810 0x55d1f4e76bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.387+0000 7f7785a2b000 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55d1f4e77130 0x55d1f4e77510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.387+0000 7f7785a2b000 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d1f4e77a50 0x55d1f4e81aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.387+0000 7f7785a2b000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f4c2acc0 con 0x55d1f4e76810 2023-10-30T12:29:13.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.387+0000 7f7785a2b000 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f4c8d920 con 0x55d1f4e77a50 2023-10-30T12:29:13.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.387+0000 7f7785a2b000 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d1f4c8ae50 con 0x55d1f4e77130 2023-10-30T12:29:13.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.387+0000 7f777bfff640 1 --2- 172.21.15.89:0/3677098267 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f77680664a0 0x7f7768068960 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f776c007fd0 tx=0x7f776c007cd0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:13.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.387+0000 7f777bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1f4e76810 0x55d1f4e76bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.387+0000 7f777bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1f4e76810 0x55d1f4e76bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57744/0 (socket says 172.21.15.89:57744) 2023-10-30T12:29:13.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.387+0000 7f777bfff640 1 -- 172.21.15.89:0/3039951303 learned_addr learned my addr 172.21.15.89:0/3039951303 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:13.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.387+0000 7f777b7fe640 1 --2- 172.21.15.89:0/3039951303 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55d1f4e77130 0x55d1f4e77510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.387+0000 7f7780c39640 1 --2- 172.21.15.89:0/3039951303 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d1f4e77a50 0x55d1f4e81aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f777bfff640 1 -- 172.21.15.89:0/3039951303 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55d1f4e77130 msgr2=0x55d1f4e77510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:13.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f777bfff640 1 --2- 172.21.15.89:0/3039951303 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55d1f4e77130 0x55d1f4e77510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f777bfff640 1 -- 172.21.15.89:0/3039951303 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d1f4e77a50 msgr2=0x55d1f4e81aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:13.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f777bfff640 1 --2- 172.21.15.89:0/3039951303 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d1f4e77a50 0x55d1f4e81aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f777bfff640 1 -- 172.21.15.89:0/3039951303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d1f497f890 con 0x55d1f4e76810 2023-10-30T12:29:13.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f777b7fe640 1 --2- 172.21.15.89:0/3039951303 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55d1f4e77130 0x55d1f4e77510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:13.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f777bfff640 1 --2- 172.21.15.89:0/3039951303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1f4e76810 0x55d1f4e76bf0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f776c0069a0 tx=0x7f776c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ceb1ad46ab2474e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:13.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f7761ffb640 1 -- 172.21.15.89:0/3039951303 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776c006e60 con 0x55d1f4e76810 2023-10-30T12:29:13.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f7761ffb640 1 -- 172.21.15.89:0/3039951303 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f776c023460 con 0x55d1f4e76810 2023-10-30T12:29:13.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f7761ffb640 1 -- 172.21.15.89:0/3039951303 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776c0235d0 con 0x55d1f4e76810 2023-10-30T12:29:13.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f7785a2b000 1 -- 172.21.15.89:0/3039951303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1f4e76810 msgr2=0x55d1f4e76bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f7785a2b000 1 --2- 172.21.15.89:0/3039951303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1f4e76810 0x55d1f4e76bf0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f776c0069a0 tx=0x7f776c017000 comp rx=0 tx=0).stop 2023-10-30T12:29:13.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f7785a2b000 1 -- 172.21.15.89:0/3039951303 shutdown_connections 2023-10-30T12:29:13.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f7785a2b000 1 --2- 172.21.15.89:0/3039951303 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d1f4e77a50 0x55d1f4e81aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f7785a2b000 1 --2- 172.21.15.89:0/3039951303 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55d1f4e77130 0x55d1f4e77510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f7785a2b000 1 --2- 172.21.15.89:0/3039951303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1f4e76810 0x55d1f4e76bf0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f7785a2b000 1 -- 172.21.15.89:0/3039951303 >> 172.21.15.89:0/3039951303 conn(0x55d1f4e2cfc0 msgr2=0x55d1f4e745d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:13.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f7785a2b000 1 -- 172.21.15.89:0/3039951303 shutdown_connections 2023-10-30T12:29:13.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f7785a2b000 1 -- 172.21.15.89:0/3039951303 wait complete. 2023-10-30T12:29:13.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f7785a2b000 1 -- start start 2023-10-30T12:29:13.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f7785a2b000 1 Processor -- start 2023-10-30T12:29:13.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f7785a2b000 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d1f4e76810 0x55d1f4e70b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f7785a2b000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1f4e77130 0x55d1f4e710a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f7785a2b000 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55d1f4e77a50 0x55d1f4e715e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f7785a2b000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f4c86740 con 0x55d1f4e77130 2023-10-30T12:29:13.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f7785a2b000 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f4c45d10 con 0x55d1f4e76810 2023-10-30T12:29:13.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f7785a2b000 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d1f4c459b0 con 0x55d1f4e77a50 2023-10-30T12:29:13.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f7785a2b000 10 client.0 _ll_register_callbacks cb 0x55d1f4e21390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T12:29:13.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f777b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1f4e77130 0x55d1f4e710a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f777b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1f4e77130 0x55d1f4e710a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57748/0 (socket says 172.21.15.89:57748) 2023-10-30T12:29:13.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f777bfff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d1f4e76810 0x55d1f4e70b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f777b7fe640 1 -- 172.21.15.89:0/3732551617 learned_addr learned my addr 172.21.15.89:0/3732551617 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:13.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.391+0000 7f7780c39640 1 --2- 172.21.15.89:0/3732551617 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55d1f4e77a50 0x55d1f4e715e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.395+0000 7f777b7fe640 1 -- 172.21.15.89:0/3732551617 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55d1f4e77a50 msgr2=0x55d1f4e715e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:13.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.395+0000 7f777b7fe640 1 --2- 172.21.15.89:0/3732551617 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55d1f4e77a50 0x55d1f4e715e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.395+0000 7f777b7fe640 1 -- 172.21.15.89:0/3732551617 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d1f4e76810 msgr2=0x55d1f4e70b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:13.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.395+0000 7f777b7fe640 1 --2- 172.21.15.89:0/3732551617 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d1f4e76810 0x55d1f4e70b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.395+0000 7f777b7fe640 1 -- 172.21.15.89:0/3732551617 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f776c00ae10 con 0x55d1f4e77130 2023-10-30T12:29:13.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.395+0000 7f777b7fe640 1 --2- 172.21.15.89:0/3732551617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1f4e77130 0x55d1f4e710a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f777001c9b0 tx=0x7f7770019920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fbe49e90cd64966 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:13.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.395+0000 7f775affd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-10-30T12:29:13.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.395+0000 7f775affd640 1 -- 172.21.15.89:0/3732551617 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77700b4040 con 0x55d1f4e77130 2023-10-30T12:29:13.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.395+0000 7f775affd640 1 -- 172.21.15.89:0/3732551617 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f77700a5a80 con 0x55d1f4e77130 2023-10-30T12:29:13.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.395+0000 7f7785a2b000 1 -- 172.21.15.89:0/3732551617 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d1f4c3a370 con 0x55d1f4e77130 2023-10-30T12:29:13.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.395+0000 7f775affd640 1 -- 172.21.15.89:0/3732551617 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f77700a5c40 con 0x55d1f4e77130 2023-10-30T12:29:13.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.395+0000 7f7785a2b000 10 client.4713 fetch_fsmap learned FSMap version 12 2023-10-30T12:29:13.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.395+0000 7f7785a2b000 1 -- 172.21.15.89:0/3732551617 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0x55d1f4c48460 con 0x55d1f4e77130 2023-10-30T12:29:13.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.395+0000 7f775affd640 1 -- 172.21.15.89:0/3732551617 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d1f4c3a370 con 0x55d1f4e77130 2023-10-30T12:29:13.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.395+0000 7f775affd640 1 -- 172.21.15.89:0/3732551617 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 12) v1 ==== 4468+0+0 (secure 0 0 0) 0x7f77700047f0 con 0x55d1f4e77130 2023-10-30T12:29:13.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.395+0000 7f7785a2b000 10 client.4713 fetch_fsmap finished waiting for FSMap version 12 2023-10-30T12:29:13.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.395+0000 7f7785a2b000 10 client.4713 resolve_mds: resolved name 'f' to daemon mds.f 2023-10-30T12:29:13.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.395+0000 7f7785a2b000 20 client.4713 populate_metadata read hostname 'smithi089' 2023-10-30T12:29:13.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.395+0000 7f7785a2b000 1 --2- 172.21.15.89:0/3732551617 >> [v2:172.21.15.192:6836/1002124471,v1:172.21.15.192:6837/1002124471] conn(0x55d1f4e8a260 0x55d1f4e8c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.395+0000 7f7785a2b000 4 client.4713 mds_command: new command op to 4318 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:29:13.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.395+0000 7f7785a2b000 1 -- 172.21.15.89:0/3732551617 --> [v2:172.21.15.192:6836/1002124471,v1:172.21.15.192:6837/1002124471] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d1f4c202d0 con 0x55d1f4e8a260 2023-10-30T12:29:13.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.395+0000 7f777bfff640 1 --2- 172.21.15.89:0/3732551617 >> [v2:172.21.15.192:6836/1002124471,v1:172.21.15.192:6837/1002124471] conn(0x55d1f4e8a260 0x55d1f4e8c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.395+0000 7f777bfff640 1 --2- 172.21.15.89:0/3732551617 >> [v2:172.21.15.192:6836/1002124471,v1:172.21.15.192:6837/1002124471] conn(0x55d1f4e8a260 0x55d1f4e8c650 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cec2c5f96fa0103 server_cookie=b4a394506c30d740 in_seq=0 out_seq=0 2023-10-30T12:29:13.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.395+0000 7f775affd640 10 client.4713 ms_handle_connect on v2:172.21.15.192:6836/1002124471 2023-10-30T12:29:13.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.399+0000 7f775affd640 1 -- 172.21.15.89:0/3732551617 <== mds.0 v2:172.21.15.192:6836/1002124471 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d1f4c202d0 con 0x55d1f4e8a260 2023-10-30T12:29:13.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.399+0000 7f775affd640 10 client.4713 handle_command_reply: tid=0 2023-10-30T12:29:13.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.399+0000 7f7785a2b000 1 client.4713 shutdown 2023-10-30T12:29:13.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.399+0000 7f7785a2b000 1 -- 172.21.15.89:0/3732551617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1f4e77130 msgr2=0x55d1f4e710a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.399+0000 7f7785a2b000 1 --2- 172.21.15.89:0/3732551617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1f4e77130 0x55d1f4e710a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f777001c9b0 tx=0x7f7770019920 comp rx=0 tx=0).stop 2023-10-30T12:29:13.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.399+0000 7f7785a2b000 1 -- 172.21.15.89:0/3732551617 shutdown_connections 2023-10-30T12:29:13.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.399+0000 7f7785a2b000 1 --2- 172.21.15.89:0/3732551617 >> [v2:172.21.15.192:6836/1002124471,v1:172.21.15.192:6837/1002124471] conn(0x55d1f4e8a260 0x55d1f4e8c650 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.399+0000 7f7785a2b000 1 --2- 172.21.15.89:0/3732551617 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55d1f4e77a50 0x55d1f4e715e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.399+0000 7f7785a2b000 1 --2- 172.21.15.89:0/3732551617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1f4e77130 0x55d1f4e710a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.399+0000 7f7785a2b000 1 --2- 172.21.15.89:0/3732551617 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d1f4e76810 0x55d1f4e70b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.399+0000 7f775affd640 0 client.4713 ms_handle_reset on v2:172.21.15.192:6836/1002124471 2023-10-30T12:29:13.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.399+0000 7f7785a2b000 1 -- 172.21.15.89:0/3732551617 >> 172.21.15.89:0/3732551617 conn(0x55d1f4e2cfc0 msgr2=0x55d1f4e76090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:13.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.399+0000 7f7785a2b000 1 -- 172.21.15.89:0/3732551617 shutdown_connections 2023-10-30T12:29:13.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.399+0000 7f7785a2b000 1 -- 172.21.15.89:0/3732551617 wait complete. 2023-10-30T12:29:13.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.399+0000 7f7785a2b000 20 client.4713 trim_cache size 0 max 16384 2023-10-30T12:29:13.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.411+0000 7f7785a2b000 1 Processor -- start 2023-10-30T12:29:13.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.411+0000 7f7785a2b000 1 -- start start 2023-10-30T12:29:13.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7785a2b000 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d1f4e76810 0x55d1f4ed85b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7785a2b000 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55d1f4e77130 0x55d1f4ed8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.419 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7785a2b000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1f4e77a50 0x55d1f4ed9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.419 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7785a2b000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f4940940 con 0x55d1f4e77a50 2023-10-30T12:29:13.419 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7785a2b000 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f4c204c0 con 0x55d1f4e76810 2023-10-30T12:29:13.419 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7785a2b000 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d1f4c820b0 con 0x55d1f4e77130 2023-10-30T12:29:13.419 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7780c39640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1f4e77a50 0x55d1f4ed9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7780c39640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1f4e77a50 0x55d1f4ed9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57758/0 (socket says 172.21.15.89:57758) 2023-10-30T12:29:13.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7780c39640 1 -- 172.21.15.89:0/1945751824 learned_addr learned my addr 172.21.15.89:0/1945751824 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:13.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f777b7fe640 1 --2- 172.21.15.89:0/1945751824 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55d1f4e77130 0x55d1f4ed8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f777bfff640 1 --2- 172.21.15.89:0/1945751824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d1f4e76810 0x55d1f4ed85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7780c39640 1 -- 172.21.15.89:0/1945751824 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55d1f4e77130 msgr2=0x55d1f4ed8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:13.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7780c39640 1 --2- 172.21.15.89:0/1945751824 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55d1f4e77130 0x55d1f4ed8af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7780c39640 1 -- 172.21.15.89:0/1945751824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d1f4e76810 msgr2=0x55d1f4ed85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:13.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7780c39640 1 --2- 172.21.15.89:0/1945751824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d1f4e76810 0x55d1f4ed85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7780c39640 1 -- 172.21.15.89:0/1945751824 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d1f4e88f50 con 0x55d1f4e77a50 2023-10-30T12:29:13.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f777bfff640 1 --2- 172.21.15.89:0/1945751824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d1f4e76810 0x55d1f4ed85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:29:13.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7780c39640 1 --2- 172.21.15.89:0/1945751824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1f4e77a50 0x55d1f4ed9030 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f77740556c0 tx=0x7f7774055270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1af001c99a4b3fad server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:13.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f77627fc640 1 -- 172.21.15.89:0/1945751824 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7774055730 con 0x55d1f4e77a50 2023-10-30T12:29:13.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f77627fc640 1 -- 172.21.15.89:0/1945751824 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7774059880 con 0x55d1f4e77a50 2023-10-30T12:29:13.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f77627fc640 1 -- 172.21.15.89:0/1945751824 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777405da30 con 0x55d1f4e77a50 2023-10-30T12:29:13.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7785a2b000 1 -- 172.21.15.89:0/1945751824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1f4e77a50 msgr2=0x55d1f4ed9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7785a2b000 1 --2- 172.21.15.89:0/1945751824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1f4e77a50 0x55d1f4ed9030 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f77740556c0 tx=0x7f7774055270 comp rx=0 tx=0).stop 2023-10-30T12:29:13.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7785a2b000 1 -- 172.21.15.89:0/1945751824 shutdown_connections 2023-10-30T12:29:13.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7785a2b000 1 --2- 172.21.15.89:0/1945751824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1f4e77a50 0x55d1f4ed9030 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7785a2b000 1 --2- 172.21.15.89:0/1945751824 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55d1f4e77130 0x55d1f4ed8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7785a2b000 1 --2- 172.21.15.89:0/1945751824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d1f4e76810 0x55d1f4ed85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7785a2b000 1 -- 172.21.15.89:0/1945751824 >> 172.21.15.89:0/1945751824 conn(0x55d1f4e2cfc0 msgr2=0x55d1f4e7c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:13.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7785a2b000 1 -- 172.21.15.89:0/1945751824 shutdown_connections 2023-10-30T12:29:13.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7785a2b000 1 -- 172.21.15.89:0/1945751824 wait complete. 2023-10-30T12:29:13.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7785a2b000 1 -- start start 2023-10-30T12:29:13.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7785a2b000 1 Processor -- start 2023-10-30T12:29:13.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7785a2b000 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1f4e76810 0x55d1f4e7b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7785a2b000 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55d1f4e77130 0x55d1f4ee0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7785a2b000 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d1f4e77a50 0x55d1f4ee11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7785a2b000 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f4c8a740 con 0x55d1f4e76810 2023-10-30T12:29:13.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7785a2b000 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f4c8b8b0 con 0x55d1f4e77a50 2023-10-30T12:29:13.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7785a2b000 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d1f4c454d0 con 0x55d1f4e77130 2023-10-30T12:29:13.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.415+0000 7f7785a2b000 10 client.0 _ll_register_callbacks cb 0x55d1f4c520f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T12:29:13.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f777bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1f4e76810 0x55d1f4e7b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f777bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1f4e76810 0x55d1f4e7b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57770/0 (socket says 172.21.15.89:57770) 2023-10-30T12:29:13.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f777bfff640 1 -- 172.21.15.89:0/2854092449 learned_addr learned my addr 172.21.15.89:0/2854092449 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:13.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f7780c39640 1 --2- 172.21.15.89:0/2854092449 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d1f4e77a50 0x55d1f4ee11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f777b7fe640 1 --2- 172.21.15.89:0/2854092449 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55d1f4e77130 0x55d1f4ee0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f777bfff640 1 -- 172.21.15.89:0/2854092449 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55d1f4e77130 msgr2=0x55d1f4ee0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:13.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f777bfff640 1 --2- 172.21.15.89:0/2854092449 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55d1f4e77130 0x55d1f4ee0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f777bfff640 1 -- 172.21.15.89:0/2854092449 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d1f4e77a50 msgr2=0x55d1f4ee11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:13.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f777bfff640 1 --2- 172.21.15.89:0/2854092449 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d1f4e77a50 0x55d1f4ee11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f777bfff640 1 -- 172.21.15.89:0/2854092449 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f777c139c00 con 0x55d1f4e76810 2023-10-30T12:29:13.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f777bfff640 1 --2- 172.21.15.89:0/2854092449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1f4e76810 0x55d1f4e7b7e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x55d1f4c84060 tx=0x7f776c001330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c360c5d6ff398e0c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:13.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f775affd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2023-10-30T12:29:13.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f775affd640 1 -- 172.21.15.89:0/2854092449 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776c03e070 con 0x55d1f4e76810 2023-10-30T12:29:13.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f775affd640 1 -- 172.21.15.89:0/2854092449 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f776c005020 con 0x55d1f4e76810 2023-10-30T12:29:13.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f7785a2b000 1 -- 172.21.15.89:0/2854092449 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d1f4e88950 con 0x55d1f4e76810 2023-10-30T12:29:13.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f775affd640 1 -- 172.21.15.89:0/2854092449 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f776c033040 con 0x55d1f4e76810 2023-10-30T12:29:13.435 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f7785a2b000 10 client.4719 fetch_fsmap learned FSMap version 12 2023-10-30T12:29:13.435 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f7785a2b000 1 -- 172.21.15.89:0/2854092449 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0x55d1f4c2e760 con 0x55d1f4e76810 2023-10-30T12:29:13.435 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f775affd640 1 -- 172.21.15.89:0/2854092449 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d1f4e88950 con 0x55d1f4e76810 2023-10-30T12:29:13.435 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f775affd640 1 -- 172.21.15.89:0/2854092449 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 12) v1 ==== 4468+0+0 (secure 0 0 0) 0x7f776c041020 con 0x55d1f4e76810 2023-10-30T12:29:13.435 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f7785a2b000 10 client.4719 fetch_fsmap finished waiting for FSMap version 12 2023-10-30T12:29:13.436 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f7785a2b000 10 client.4719 resolve_mds: resolved name 'f' to daemon mds.f 2023-10-30T12:29:13.436 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f7785a2b000 20 client.4719 populate_metadata read hostname 'smithi089' 2023-10-30T12:29:13.436 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f7785a2b000 1 --2- 172.21.15.89:0/2854092449 >> [v2:172.21.15.192:6836/1002124471,v1:172.21.15.192:6837/1002124471] conn(0x55d1f4edb320 0x55d1f4edb700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.436 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f7785a2b000 4 client.4719 mds_command: new command op to 4318 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-30T12:29:13.436 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f7785a2b000 1 -- 172.21.15.89:0/2854092449 --> [v2:172.21.15.192:6836/1002124471,v1:172.21.15.192:6837/1002124471] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d1f4c1e6f0 con 0x55d1f4edb320 2023-10-30T12:29:13.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f777b7fe640 1 --2- 172.21.15.89:0/2854092449 >> [v2:172.21.15.192:6836/1002124471,v1:172.21.15.192:6837/1002124471] conn(0x55d1f4edb320 0x55d1f4edb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f775affd640 10 client.4719 ms_handle_connect on v2:172.21.15.192:6836/1002124471 2023-10-30T12:29:13.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f777b7fe640 1 --2- 172.21.15.89:0/2854092449 >> [v2:172.21.15.192:6836/1002124471,v1:172.21.15.192:6837/1002124471] conn(0x55d1f4edb320 0x55d1f4edb700 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f316d71474ad74e9 server_cookie=a0834ac9953b6c95 in_seq=0 out_seq=0 2023-10-30T12:29:13.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f775affd640 1 -- 172.21.15.89:0/2854092449 <== mds.0 v2:172.21.15.192:6836/1002124471 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55d1f4c1e6f0 con 0x55d1f4edb320 2023-10-30T12:29:13.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f775affd640 10 client.4719 handle_command_reply: tid=0 2023-10-30T12:29:13.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f7785a2b000 1 client.4719 shutdown 2023-10-30T12:29:13.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f7785a2b000 1 -- 172.21.15.89:0/2854092449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1f4e76810 msgr2=0x55d1f4e7b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f7785a2b000 1 --2- 172.21.15.89:0/2854092449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1f4e76810 0x55d1f4e7b7e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x55d1f4c84060 tx=0x7f776c001330 comp rx=0 tx=0).stop 2023-10-30T12:29:13.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f7785a2b000 1 -- 172.21.15.89:0/2854092449 shutdown_connections 2023-10-30T12:29:13.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f7785a2b000 1 --2- 172.21.15.89:0/2854092449 >> [v2:172.21.15.192:6836/1002124471,v1:172.21.15.192:6837/1002124471] conn(0x55d1f4edb320 0x55d1f4edb700 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f7785a2b000 1 --2- 172.21.15.89:0/2854092449 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d1f4e77a50 0x55d1f4ee11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f7785a2b000 1 --2- 172.21.15.89:0/2854092449 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x55d1f4e77130 0x55d1f4ee0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f7785a2b000 1 --2- 172.21.15.89:0/2854092449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1f4e76810 0x55d1f4e7b7e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f775affd640 0 client.4719 ms_handle_reset on v2:172.21.15.192:6836/1002124471 2023-10-30T12:29:13.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f7785a2b000 1 -- 172.21.15.89:0/2854092449 >> 172.21.15.89:0/2854092449 conn(0x55d1f4e2cfc0 msgr2=0x55d1f4e7e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:13.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f7785a2b000 1 -- 172.21.15.89:0/2854092449 shutdown_connections 2023-10-30T12:29:13.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.419+0000 7f7785a2b000 1 -- 172.21.15.89:0/2854092449 wait complete. 2023-10-30T12:29:13.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.423+0000 7f7785a2b000 20 client.4719 trim_cache size 0 max 16384 2023-10-30T12:29:13.440 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-10-30T12:29:13.440 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "401641be-db6e-4c16-b3ac-fd84a86c4de4", 2023-10-30T12:29:13.441 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 0, 2023-10-30T12:29:13.441 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 4318, 2023-10-30T12:29:13.441 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2023-10-30T12:29:13.441 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2023-10-30T12:29:13.441 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2023-10-30T12:29:13.441 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 22.074511432000001, 2023-10-30T12:29:13.442 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 12, 2023-10-30T12:29:13.442 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 22, 2023-10-30T12:29:13.442 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 22, 2023-10-30T12:29:13.442 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 29.012223972000001 2023-10-30T12:29:13.442 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-10-30T12:29:13.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.423+0000 7f778269c640 1 -- 172.21.15.89:0/3677098267 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f77680664a0 msgr2=0x7f7768068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.423+0000 7f778269c640 1 --2- 172.21.15.89:0/3677098267 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f77680664a0 0x7f7768068960 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f776c007fd0 tx=0x7f776c007cd0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.423+0000 7f778269c640 1 -- 172.21.15.89:0/3677098267 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f777c1e48f0 msgr2=0x7f777c137580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.423+0000 7f778269c640 1 --2- 172.21.15.89:0/3677098267 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f777c1e48f0 0x7f777c137580 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f777c0c21e0 tx=0x7f77740117d0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.423+0000 7f778269c640 1 -- 172.21.15.89:0/3677098267 shutdown_connections 2023-10-30T12:29:13.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.423+0000 7f778269c640 1 --2- 172.21.15.89:0/3677098267 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f77680664a0 0x7f7768068960 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.423+0000 7f778269c640 1 --2- 172.21.15.89:0/3677098267 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f777c1e48f0 0x7f777c137580 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.423+0000 7f778269c640 1 --2- 172.21.15.89:0/3677098267 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f777c155d20 0x7f777c1e43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.423+0000 7f778269c640 1 --2- 172.21.15.89:0/3677098267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7764006570 0x7f777c1396c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.423+0000 7f778269c640 1 -- 172.21.15.89:0/3677098267 >> 172.21.15.89:0/3677098267 conn(0x7f777c0c7910 msgr2=0x7f777c06a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:13.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.423+0000 7f778269c640 1 -- 172.21.15.89:0/3677098267 shutdown_connections 2023-10-30T12:29:13.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.423+0000 7f778269c640 1 -- 172.21.15.89:0/3677098267 wait complete. 2023-10-30T12:29:13.446 INFO:teuthology.run_tasks:Running task kclient... 2023-10-30T12:29:13.467 INFO:tasks.kclient:Mounting kernel clients... 2023-10-30T12:29:13.467 INFO:tasks.kclient:config is {'client.0': {}} 2023-10-30T12:29:13.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:13.468 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2023-10-30T12:29:13.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:13.478 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link delete ceph-brx 2023-10-30T12:29:13.542 INFO:teuthology.orchestra.run.smithi089.stderr:Cannot find device "ceph-brx" 2023-10-30T12:29:13.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:29:13.545 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2023-10-30T12:29:13.545 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-30T12:29:13.545 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-30T12:29:13.546 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-30T12:29:13.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.719+0000 7fe8ea5ab640 1 Processor -- start 2023-10-30T12:29:13.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.719+0000 7fe8ea5ab640 1 -- start start 2023-10-30T12:29:13.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8ea5ab640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe8e415a410 0x7fe8e406a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8ea5ab640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e40c6c10 con 0x7fe8e406cb80 2023-10-30T12:29:13.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8ea5ab640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e40c6ef0 con 0x7fe8e4069e90 2023-10-30T12:29:13.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8ea5ab640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8e40c8570 con 0x7fe8e415a410 2023-10-30T12:29:13.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8e8b48640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe8e406cb80 0x7fe8e406cff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58348/0 (socket says 172.21.15.89:58348) 2023-10-30T12:29:13.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8e8b48640 1 -- 172.21.15.89:0/1248781268 learned_addr learned my addr 172.21.15.89:0/1248781268 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:29:13.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8e37fe640 1 --2- 172.21.15.89:0/1248781268 >> v2:172.21.15.192:3300/0 conn(0x7fe8e415a410 0x7fe8e406a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8e2ffd640 1 -- 172.21.15.89:0/1248781268 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3889189509 0 0) 0x7fe8e40c6c10 con 0x7fe8e406cb80 2023-10-30T12:29:13.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8e2ffd640 1 -- 172.21.15.89:0/1248781268 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8cc003530 con 0x7fe8e406cb80 2023-10-30T12:29:13.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8e2ffd640 1 -- 172.21.15.89:0/1248781268 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3101099855 0 0) 0x7fe8cc003530 con 0x7fe8e406cb80 2023-10-30T12:29:13.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8e2ffd640 1 -- 172.21.15.89:0/1248781268 >> v2:172.21.15.192:3300/0 conn(0x7fe8e415a410 msgr2=0x7fe8e406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:13.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8e2ffd640 1 --2- 172.21.15.89:0/1248781268 >> v2:172.21.15.192:3300/0 conn(0x7fe8e415a410 0x7fe8e406a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8e2ffd640 1 -- 172.21.15.89:0/1248781268 >> v1:172.21.15.192:6789/0 conn(0x7fe8e4069e90 legacy=0x7fe8e40c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8e2ffd640 1 -- 172.21.15.89:0/1248781268 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e40c71d0 con 0x7fe8e406cb80 2023-10-30T12:29:13.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8e2ffd640 1 -- 172.21.15.89:0/1248781268 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7fe8dc001a00 con 0x7fe8e406cb80 2023-10-30T12:29:13.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8e2ffd640 1 -- 172.21.15.89:0/1248781268 >> v1:172.21.15.89:6789/0 conn(0x7fe8e406cb80 legacy=0x7fe8e406cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8e2ffd640 1 --2- 172.21.15.89:0/1248781268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8cc003980 0x7fe8cc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8e2ffd640 1 --2- 172.21.15.89:0/1248781268 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8cc0047d0 0x7fe8cc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8e2ffd640 1 --2- 172.21.15.89:0/1248781268 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8cc005190 0x7fe8cc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8e2ffd640 1 -- 172.21.15.89:0/1248781268 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e40c8570 con 0x7fe8cc003980 2023-10-30T12:29:13.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8e2ffd640 1 -- 172.21.15.89:0/1248781268 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8cc00c030 con 0x7fe8cc005190 2023-10-30T12:29:13.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8e2ffd640 1 -- 172.21.15.89:0/1248781268 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8cc00c1f0 con 0x7fe8cc0047d0 2023-10-30T12:29:13.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8e3fff640 1 --2- 172.21.15.89:0/1248781268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8cc003980 0x7fe8cc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8e37fe640 1 --2- 172.21.15.89:0/1248781268 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8cc0047d0 0x7fe8cc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8e8b48640 1 --2- 172.21.15.89:0/1248781268 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8cc005190 0x7fe8cc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8e3fff640 1 -- 172.21.15.89:0/1248781268 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8cc0047d0 msgr2=0x7fe8cc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:13.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8e3fff640 1 --2- 172.21.15.89:0/1248781268 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8cc0047d0 0x7fe8cc004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8e3fff640 1 -- 172.21.15.89:0/1248781268 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8cc005190 msgr2=0x7fe8cc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:13.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8e3fff640 1 --2- 172.21.15.89:0/1248781268 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8cc005190 0x7fe8cc00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8e3fff640 1 -- 172.21.15.89:0/1248781268 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8cc003530 con 0x7fe8cc003980 2023-10-30T12:29:13.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.723+0000 7fe8e8b48640 1 -- 172.21.15.89:0/1248781268 reap_dead start 2023-10-30T12:29:13.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.727+0000 7fe8e3fff640 1 --2- 172.21.15.89:0/1248781268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8cc003980 0x7fe8cc003da0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe8d4001780 tx=0x7fe8d401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f80650ff6d49f8e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:13.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.727+0000 7fe8e2ffd640 1 -- 172.21.15.89:0/1248781268 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d40052d0 con 0x7fe8cc003980 2023-10-30T12:29:13.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.727+0000 7fe8e2ffd640 1 -- 172.21.15.89:0/1248781268 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8d40a7050 con 0x7fe8cc003980 2023-10-30T12:29:13.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.727+0000 7fe8e2ffd640 1 -- 172.21.15.89:0/1248781268 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d4003600 con 0x7fe8cc003980 2023-10-30T12:29:13.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.727+0000 7fe8ea5ab640 1 -- 172.21.15.89:0/1248781268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8cc003980 msgr2=0x7fe8cc003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.727+0000 7fe8ea5ab640 1 --2- 172.21.15.89:0/1248781268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8cc003980 0x7fe8cc003da0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe8d4001780 tx=0x7fe8d401c3e0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.727+0000 7fe8ea5ab640 1 -- 172.21.15.89:0/1248781268 shutdown_connections 2023-10-30T12:29:13.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.727+0000 7fe8ea5ab640 1 --2- 172.21.15.89:0/1248781268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8cc003980 0x7fe8cc003da0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.727+0000 7fe8ea5ab640 1 -- 172.21.15.89:0/1248781268 >> 172.21.15.89:0/1248781268 conn(0x7fe8e406dfd0 msgr2=0x7fe8e406e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:13.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.727+0000 7fe8ea5ab640 1 -- 172.21.15.89:0/1248781268 shutdown_connections 2023-10-30T12:29:13.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.727+0000 7fe8ea5ab640 1 -- 172.21.15.89:0/1248781268 wait complete. 2023-10-30T12:29:13.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.727+0000 7fe8ea5ab640 1 Processor -- start 2023-10-30T12:29:13.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.727+0000 7fe8ea5ab640 1 -- start start 2023-10-30T12:29:13.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.731+0000 7fe8ea5ab640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8cc003980 0x7fe8e40708b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.731+0000 7fe8ea5ab640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8e4154790 0x7fe8e4154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.731+0000 7fe8ea5ab640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8e4155630 0x7fe8e4155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.731+0000 7fe8ea5ab640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e41596d0 con 0x7fe8e4155630 2023-10-30T12:29:13.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.731+0000 7fe8ea5ab640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e414ce10 con 0x7fe8cc003980 2023-10-30T12:29:13.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.731+0000 7fe8ea5ab640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e41f6100 con 0x7fe8e4154790 2023-10-30T12:29:13.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.731+0000 7fe8e8b48640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8e4155630 0x7fe8e4155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.731+0000 7fe8e8b48640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8e4155630 0x7fe8e4155ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57776/0 (socket says 172.21.15.89:57776) 2023-10-30T12:29:13.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.731+0000 7fe8e8b48640 1 -- 172.21.15.89:0/1191971662 learned_addr learned my addr 172.21.15.89:0/1191971662 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:13.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.731+0000 7fe8e37fe640 1 --2- 172.21.15.89:0/1191971662 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8e4154790 0x7fe8e4154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.731+0000 7fe8e3fff640 1 --2- 172.21.15.89:0/1191971662 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8cc003980 0x7fe8e40708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.731+0000 7fe8e8b48640 1 -- 172.21.15.89:0/1191971662 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8e4154790 msgr2=0x7fe8e4154bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.731+0000 7fe8e8b48640 1 --2- 172.21.15.89:0/1191971662 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8e4154790 0x7fe8e4154bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.731+0000 7fe8e8b48640 1 -- 172.21.15.89:0/1191971662 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8cc003980 msgr2=0x7fe8e40708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.731+0000 7fe8e8b48640 1 --2- 172.21.15.89:0/1191971662 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8cc003980 0x7fe8e40708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.731+0000 7fe8e8b48640 1 -- 172.21.15.89:0/1191971662 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e41f63e0 con 0x7fe8e4155630 2023-10-30T12:29:13.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.731+0000 7fe8e8b48640 1 --2- 172.21.15.89:0/1191971662 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8e4155630 0x7fe8e4155ab0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe8e40c6570 tx=0x7fe8dc00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:13.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.731+0000 7fe8e0ff9640 1 -- 172.21.15.89:0/1191971662 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8dc00ec10 con 0x7fe8e4155630 2023-10-30T12:29:13.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.731+0000 7fe8e0ff9640 1 -- 172.21.15.89:0/1191971662 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8dc00ed70 con 0x7fe8e4155630 2023-10-30T12:29:13.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.731+0000 7fe8ea5ab640 1 -- 172.21.15.89:0/1191971662 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8e414d1b0 con 0x7fe8e4155630 2023-10-30T12:29:13.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.731+0000 7fe8e0ff9640 1 -- 172.21.15.89:0/1191971662 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8dc00eee0 con 0x7fe8e4155630 2023-10-30T12:29:13.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.731+0000 7fe8ea5ab640 1 -- 172.21.15.89:0/1191971662 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8e414c8f0 con 0x7fe8e4155630 2023-10-30T12:29:13.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.731+0000 7fe8ea5ab640 1 -- 172.21.15.89:0/1191971662 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8e414c730 con 0x7fe8e4155630 2023-10-30T12:29:13.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.731+0000 7fe8e0ff9640 1 -- 172.21.15.89:0/1191971662 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe8dc013620 con 0x7fe8e4155630 2023-10-30T12:29:13.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.735+0000 7fe8e0ff9640 1 --2- 172.21.15.89:0/1191971662 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe8d00664d0 0x7fe8d0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.735+0000 7fe8e0ff9640 1 -- 172.21.15.89:0/1191971662 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe8dc084390 con 0x7fe8e4155630 2023-10-30T12:29:13.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.735+0000 7fe8e3fff640 1 --2- 172.21.15.89:0/1191971662 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe8d00664d0 0x7fe8d0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.735+0000 7fe8e3fff640 1 --2- 172.21.15.89:0/1191971662 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe8d00664d0 0x7fe8d0068990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe8d4001750 tx=0x7fe8d40011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:13.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.739+0000 7fe8e0ff9640 1 -- 172.21.15.89:0/1191971662 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe8dc0516e0 con 0x7fe8e4155630 2023-10-30T12:29:13.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.915+0000 7fe8ea5ab640 1 -- 172.21.15.89:0/1191971662 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe8e404e170 con 0x7fe8e4155630 2023-10-30T12:29:13.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.919+0000 7fe8e0ff9640 1 -- 172.21.15.89:0/1191971662 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v12) v1 ==== 53+0+73 (secure 0 0 0) 0x7fe8dc054d00 con 0x7fe8e4155630 2023-10-30T12:29:13.920 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-30T12:29:13.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.919+0000 7fe8ea5ab640 1 -- 172.21.15.89:0/1191971662 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe8d00664d0 msgr2=0x7fe8d0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.919+0000 7fe8ea5ab640 1 --2- 172.21.15.89:0/1191971662 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe8d00664d0 0x7fe8d0068990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe8d4001750 tx=0x7fe8d40011a0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.919+0000 7fe8ea5ab640 1 -- 172.21.15.89:0/1191971662 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8e4155630 msgr2=0x7fe8e4155ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.919+0000 7fe8ea5ab640 1 --2- 172.21.15.89:0/1191971662 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8e4155630 0x7fe8e4155ab0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe8e40c6570 tx=0x7fe8dc00e530 comp rx=0 tx=0).stop 2023-10-30T12:29:13.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.919+0000 7fe8ea5ab640 1 -- 172.21.15.89:0/1191971662 shutdown_connections 2023-10-30T12:29:13.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.919+0000 7fe8ea5ab640 1 --2- 172.21.15.89:0/1191971662 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe8d00664d0 0x7fe8d0068990 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.919+0000 7fe8ea5ab640 1 --2- 172.21.15.89:0/1191971662 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8e4155630 0x7fe8e4155ab0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.919+0000 7fe8ea5ab640 1 --2- 172.21.15.89:0/1191971662 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8e4154790 0x7fe8e4154bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.919+0000 7fe8ea5ab640 1 --2- 172.21.15.89:0/1191971662 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8cc003980 0x7fe8e40708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.919+0000 7fe8ea5ab640 1 -- 172.21.15.89:0/1191971662 >> 172.21.15.89:0/1191971662 conn(0x7fe8e406dfd0 msgr2=0x7fe8e4071440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:13.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.919+0000 7fe8ea5ab640 1 -- 172.21.15.89:0/1191971662 shutdown_connections 2023-10-30T12:29:13.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:13.919+0000 7fe8ea5ab640 1 -- 172.21.15.89:0/1191971662 wait complete. 2023-10-30T12:29:13.942 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-30T12:29:13.942 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-30T12:29:13.943 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi089.front.sepia.ceph.com 2023-10-30T12:29:13.943 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-30T12:29:13.943 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-30T12:29:13.943 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-30T12:29:13.943 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-30T12:29:13.944 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-30T12:29:13.944 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-30T12:29:13.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:13.945 DEBUG:teuthology.orchestra.run.smithi089:> ip addr 2023-10-30T12:29:13.954 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-30T12:29:13.954 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-30T12:29:13.954 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2023-10-30T12:29:13.955 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-10-30T12:29:13.955 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2023-10-30T12:29:13.955 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-10-30T12:29:13.955 INFO:teuthology.orchestra.run.smithi089.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-30T12:29:13.955 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c6 brd ff:ff:ff:ff:ff:ff 2023-10-30T12:29:13.955 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f0 2023-10-30T12:29:13.956 INFO:teuthology.orchestra.run.smithi089.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-30T12:29:13.956 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c7 brd ff:ff:ff:ff:ff:ff 2023-10-30T12:29:13.956 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f1 2023-10-30T12:29:13.956 INFO:teuthology.orchestra.run.smithi089.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-30T12:29:13.956 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:00 brd ff:ff:ff:ff:ff:ff 2023-10-30T12:29:13.957 INFO:teuthology.orchestra.run.smithi089.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-30T12:29:13.957 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:01 brd ff:ff:ff:ff:ff:ff 2023-10-30T12:29:13.957 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.21.15.89/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-10-30T12:29:13.957 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft 42416sec preferred_lft 42416sec 2023-10-30T12:29:13.957 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::ec4:7aff:fe8f:d501/64 scope link 2023-10-30T12:29:13.958 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-10-30T12:29:13.958 INFO:teuthology.orchestra.run.smithi089.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-10-30T12:29:13.958 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 02:42:e3:f4:3c:a8 brd ff:ff:ff:ff:ff:ff 2023-10-30T12:29:13.958 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-10-30T12:29:13.958 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-10-30T12:29:13.959 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-30T12:29:13.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:13.960 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2023-10-30T12:29:13.960 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2023-10-30T12:29:13.960 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link add name ceph-brx type bridge 2023-10-30T12:29:13.961 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip addr flush dev ceph-brx 2023-10-30T12:29:13.961 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set ceph-brx up 2023-10-30T12:29:13.961 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-30T12:29:13.962 DEBUG:teuthology.orchestra.run.smithi089:> ') 2023-10-30T12:29:14.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:14.075 DEBUG:teuthology.orchestra.run.smithi089:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-30T12:29:14.094 INFO:teuthology.orchestra.run.smithi089.stdout:1 2023-10-30T12:29:14.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:14.097 DEBUG:teuthology.orchestra.run.smithi089:> route 2023-10-30T12:29:14.175 INFO:teuthology.orchestra.run.smithi089.stdout:Kernel IP routing table 2023-10-30T12:29:14.175 INFO:teuthology.orchestra.run.smithi089.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-30T12:29:14.175 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-10-30T12:29:14.175 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-30T12:29:14.176 INFO:teuthology.orchestra.run.smithi089.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-10-30T12:29:14.176 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-30T12:29:14.176 INFO:teuthology.orchestra.run.smithi089.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-10-30T12:29:14.176 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-10-30T12:29:14.176 INFO:teuthology.orchestra.run.smithi089.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-10-30T12:29:14.177 INFO:teuthology.orchestra.run.smithi089.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-30T12:29:14.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:14.177 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2023-10-30T12:29:14.178 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2023-10-30T12:29:14.178 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-30T12:29:14.179 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-30T12:29:14.179 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-30T12:29:14.180 DEBUG:teuthology.orchestra.run.smithi089:> ') 2023-10-30T12:29:14.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:14.266 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2023-10-30T12:29:14.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:14.276 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list-id 2023-10-30T12:29:14.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:14.332 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2023-10-30T12:29:14.332 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2023-10-30T12:29:14.333 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-30T12:29:14.333 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-10-30T12:29:14.333 DEBUG:teuthology.orchestra.run.smithi089:> ') 2023-10-30T12:29:14.376 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:29:14.384 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:29:14.421 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-10-30T12:29:14.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:14.421 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2023-10-30T12:29:14.422 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2023-10-30T12:29:14.422 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-10-30T12:29:14.422 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-30T12:29:14.423 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-10-30T12:29:14.423 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-10-30T12:29:14.423 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-10-30T12:29:14.423 DEBUG:teuthology.orchestra.run.smithi089:> ') 2023-10-30T12:29:14.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:14.599 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2023-10-30T12:29:14.599 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2023-10-30T12:29:14.600 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set brx.0 up 2023-10-30T12:29:14.600 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set dev brx.0 master ceph-brx 2023-10-30T12:29:14.600 DEBUG:teuthology.orchestra.run.smithi089:> ') 2023-10-30T12:29:14.641 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-30T12:29:14.642 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-30T12:29:14.693 INFO:teuthology.orchestra.run.smithi089.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-30T12:29:14.693 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-30T12:29:14.693 DEBUG:teuthology.orchestra.run.smithi089:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-30T12:29:14.747 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-30T12:29:14.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:14.747 DEBUG:teuthology.orchestra.run.smithi089:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-30T12:29:14.847 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-30T12:29:14.847 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-30T12:29:14.847 INFO:teuthology.orchestra.run.smithi089.stdout:invalid new device string format 2023-10-30T12:29:14.982 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-30T12:29:14.982 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-30T12:29:14.982 INFO:teuthology.orchestra.run.smithi089.stdout:invalid new device string format 2023-10-30T12:29:14.983 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.192:3300,172.21.15.192:3301" 2023-10-30T12:29:14.983 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-30T12:29:14.995 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-30T12:29:14.996 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-30T12:29:14.996 INFO:teuthology.orchestra.run.smithi089.stdout:invalid new device string format 2023-10-30T12:29:14.996 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.192:3300,172.21.15.192:3301" 2023-10-30T12:29:14.996 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-30T12:29:14.996 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:6789,172.21.15.192:6789,172.21.15.192:6790" 2023-10-30T12:29:14.997 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.89:6789,172.21.15.192:6789,172.21.15.192:6790:/ 2023-10-30T12:29:14.997 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=401641be-db6e-4c16-b3ac-fd84a86c4de4" will pass to kernel 2023-10-30T12:29:15.000 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-30T12:29:15.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:15.001 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-30T12:29:15.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:15.029 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-30T12:29:15.105 INFO:teuthology.orchestra.run.smithi089.stdout:client4728 2023-10-30T12:29:15.106 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-10-30T12:29:15.106 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-30T12:29:15.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:15.156 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-30T12:29:15.227 INFO:teuthology.orchestra.run.smithi089.stdout:client4728 2023-10-30T12:29:15.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:15.228 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-30T12:29:15.295 INFO:teuthology.orchestra.run.smithi089.stdout:client4728 2023-10-30T12:29:15.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:15.296 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/401641be-db6e-4c16-b3ac-fd84a86c4de4.client4728/status') 2023-10-30T12:29:15.372 INFO:teuthology.orchestra.run.smithi089.stdout:instance: client.4728 (3)192.168.0.1:0/2816847172 2023-10-30T12:29:15.372 INFO:teuthology.orchestra.run.smithi089.stdout:blocklisted: false 2023-10-30T12:29:15.373 INFO:teuthology.orchestra.run.smithi089.stderr:0+1 records in 2023-10-30T12:29:15.373 INFO:teuthology.orchestra.run.smithi089.stderr:0+1 records out 2023-10-30T12:29:15.373 INFO:teuthology.orchestra.run.smithi089.stderr:69 bytes copied, 6.1537e-05 s, 1.1 MB/s 2023-10-30T12:29:15.374 INFO:teuthology.run_tasks:Running task mds_thrash... 2023-10-30T12:29:15.385 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:29:15.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.547+0000 7efca1f0f640 1 Processor -- start 2023-10-30T12:29:15.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.551+0000 7efca1f0f640 1 -- start start 2023-10-30T12:29:15.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.551+0000 7efca1f0f640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efc9c15d6f0 0x7efc9c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.551+0000 7efca1f0f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc9c0cf160 0x7efc9c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.551+0000 7efca1f0f640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc9c0cb800 con 0x7efc9c1538a0 2023-10-30T12:29:15.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.551+0000 7efca1f0f640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc9c0cb330 con 0x7efc9c15d6f0 2023-10-30T12:29:15.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.551+0000 7efca1f0f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c0cad70 con 0x7efc9c0cf160 2023-10-30T12:29:15.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.551+0000 7efc9affd640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efc9c15d6f0 0x7efc9c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.551+0000 7efc9affd640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efc9c15d6f0 0x7efc9c15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57786/0 (socket says 172.21.15.89:57786) 2023-10-30T12:29:15.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.551+0000 7efc9affd640 1 -- 172.21.15.89:0/2984275096 learned_addr learned my addr 172.21.15.89:0/2984275096 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:15.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.551+0000 7efc9bfff640 1 --2- 172.21.15.89:0/2984275096 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc9c0cf160 0x7efc9c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.551+0000 7efc9affd640 1 -- 172.21.15.89:0/2984275096 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc9c0cf160 msgr2=0x7efc9c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:15.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.551+0000 7efc9affd640 1 --2- 172.21.15.89:0/2984275096 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc9c0cf160 0x7efc9c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.551+0000 7efc9affd640 1 -- 172.21.15.89:0/2984275096 >> v1:172.21.15.192:6789/0 conn(0x7efc9c1538a0 legacy=0x7efc9c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.551+0000 7efc9affd640 1 -- 172.21.15.89:0/2984275096 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc9c0cb050 con 0x7efc9c15d6f0 2023-10-30T12:29:15.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.555+0000 7efc9affd640 1 --2- 172.21.15.89:0/2984275096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc9c15d6f0 0x7efc9c15a090 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7efc90009870 tx=0x7efc9001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad7a82d7d66326d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:15.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.555+0000 7efc9a7fc640 1 -- 172.21.15.89:0/2984275096 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc900a8a30 con 0x7efc9c15d6f0 2023-10-30T12:29:15.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.555+0000 7efc9a7fc640 1 -- 172.21.15.89:0/2984275096 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc900a8b90 con 0x7efc9c15d6f0 2023-10-30T12:29:15.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.555+0000 7efc9a7fc640 1 -- 172.21.15.89:0/2984275096 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc900b0440 con 0x7efc9c15d6f0 2023-10-30T12:29:15.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.555+0000 7efca1f0f640 1 -- 172.21.15.89:0/2984275096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc9c15d6f0 msgr2=0x7efc9c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.555+0000 7efca1f0f640 1 --2- 172.21.15.89:0/2984275096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc9c15d6f0 0x7efc9c15a090 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7efc90009870 tx=0x7efc9001ed30 comp rx=0 tx=0).stop 2023-10-30T12:29:15.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.555+0000 7efca1f0f640 1 -- 172.21.15.89:0/2984275096 shutdown_connections 2023-10-30T12:29:15.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.555+0000 7efca1f0f640 1 --2- 172.21.15.89:0/2984275096 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc9c0cf160 0x7efc9c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.555+0000 7efca1f0f640 1 --2- 172.21.15.89:0/2984275096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc9c15d6f0 0x7efc9c15a090 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.555+0000 7efca1f0f640 1 -- 172.21.15.89:0/2984275096 >> 172.21.15.89:0/2984275096 conn(0x7efc9c06bfe0 msgr2=0x7efc9c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:15.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.555+0000 7efca1f0f640 1 -- 172.21.15.89:0/2984275096 shutdown_connections 2023-10-30T12:29:15.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.555+0000 7efca1f0f640 1 -- 172.21.15.89:0/2984275096 wait complete. 2023-10-30T12:29:15.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.555+0000 7efca1f0f640 1 Processor -- start 2023-10-30T12:29:15.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.555+0000 7efca1f0f640 1 -- start start 2023-10-30T12:29:15.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.559+0000 7efca1f0f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc9c0cf160 0x7efc9c0b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.559+0000 7efca1f0f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc9c1538a0 0x7efc9c1efbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.559+0000 7efca1f0f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc9c15d6f0 0x7efc9c1587e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.559+0000 7efca1f0f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c1ef740 con 0x7efc9c1538a0 2023-10-30T12:29:15.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.559+0000 7efca1f0f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c157560 con 0x7efc9c0cf160 2023-10-30T12:29:15.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.559+0000 7efca1f0f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c1ef3f0 con 0x7efc9c15d6f0 2023-10-30T12:29:15.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.559+0000 7efc9affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc9c1538a0 0x7efc9c1efbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.559+0000 7efc9affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc9c1538a0 0x7efc9c1efbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57798/0 (socket says 172.21.15.89:57798) 2023-10-30T12:29:15.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.559+0000 7efc9affd640 1 -- 172.21.15.89:0/2764887438 learned_addr learned my addr 172.21.15.89:0/2764887438 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:15.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.559+0000 7efc9b7fe640 1 --2- 172.21.15.89:0/2764887438 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc9c0cf160 0x7efc9c0b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.559+0000 7efc9bfff640 1 --2- 172.21.15.89:0/2764887438 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc9c15d6f0 0x7efc9c1587e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.559+0000 7efc9affd640 1 -- 172.21.15.89:0/2764887438 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc9c15d6f0 msgr2=0x7efc9c1587e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.559+0000 7efc9affd640 1 --2- 172.21.15.89:0/2764887438 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc9c15d6f0 0x7efc9c1587e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.559+0000 7efc9affd640 1 -- 172.21.15.89:0/2764887438 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc9c0cf160 msgr2=0x7efc9c0b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.559+0000 7efc9affd640 1 --2- 172.21.15.89:0/2764887438 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc9c0cf160 0x7efc9c0b43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.559+0000 7efc9affd640 1 -- 172.21.15.89:0/2764887438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc900094d0 con 0x7efc9c1538a0 2023-10-30T12:29:15.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.559+0000 7efc9b7fe640 1 --2- 172.21.15.89:0/2764887438 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc9c0cf160 0x7efc9c0b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:15.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.559+0000 7efc9affd640 1 --2- 172.21.15.89:0/2764887438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc9c1538a0 0x7efc9c1efbd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7efc90000c10 tx=0x7efc9001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:15.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.559+0000 7efc83fff640 1 -- 172.21.15.89:0/2764887438 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc900bf070 con 0x7efc9c1538a0 2023-10-30T12:29:15.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.559+0000 7efc83fff640 1 -- 172.21.15.89:0/2764887438 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc900b1c90 con 0x7efc9c1538a0 2023-10-30T12:29:15.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.559+0000 7efc83fff640 1 -- 172.21.15.89:0/2764887438 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc900b9410 con 0x7efc9c1538a0 2023-10-30T12:29:15.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.559+0000 7efca1f0f640 1 -- 172.21.15.89:0/2764887438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc9c1f2e30 con 0x7efc9c1538a0 2023-10-30T12:29:15.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.559+0000 7efca1f0f640 1 -- 172.21.15.89:0/2764887438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc9c1f33f0 con 0x7efc9c1538a0 2023-10-30T12:29:15.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.559+0000 7efca1f0f640 1 -- 172.21.15.89:0/2764887438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc9c0b2a50 con 0x7efc9c1538a0 2023-10-30T12:29:15.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.559+0000 7efc83fff640 1 -- 172.21.15.89:0/2764887438 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efc900b7040 con 0x7efc9c1538a0 2023-10-30T12:29:15.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.563+0000 7efc83fff640 1 --2- 172.21.15.89:0/2764887438 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7efc880664d0 0x7efc88068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.563+0000 7efc83fff640 1 -- 172.21.15.89:0/2764887438 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc90121420 con 0x7efc9c1538a0 2023-10-30T12:29:15.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.563+0000 7efc9b7fe640 1 --2- 172.21.15.89:0/2764887438 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7efc880664d0 0x7efc88068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.563+0000 7efc9b7fe640 1 --2- 172.21.15.89:0/2764887438 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7efc880664d0 0x7efc88068990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efc8c001710 tx=0x7efc8c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:15.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.567+0000 7efc83fff640 1 -- 172.21.15.89:0/2764887438 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efc900ee800 con 0x7efc9c1538a0 2023-10-30T12:29:15.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.715+0000 7efca1f0f640 1 -- 172.21.15.89:0/2764887438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc9c1ef8f0 con 0x7efc9c1538a0 2023-10-30T12:29:15.718 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.715+0000 7efc83fff640 1 -- 172.21.15.89:0/2764887438 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16505 (secure 0 0 0) 0x7efc9c1ef8f0 con 0x7efc9c1538a0 2023-10-30T12:29:15.719 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:29:15.719 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":22,"fsid":"401641be-db6e-4c16-b3ac-fd84a86c4de4","created":"2023-10-30T12:28:26.198532+0000","modified":"2023-10-30T12:28:51.918285+0000","last_up_change":"2023-10-30T12:28:37.212205+0000","last_in_change":"2023-10-30T12:28:28.517734+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:39.004943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T12:28:48.935909+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T12:28:49.716098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72595c54-fd6e-4a5a-bb14-ff27e9377b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6801","nonce":1673955622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6803","nonce":1673955622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6807","nonce":1673955622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6805","nonce":1673955622}]},"public_addr":"172.21.15.89:6801/1673955622","cluster_addr":"172.21.15.89:6803/1673955622","heartbeat_back_addr":"172.21.15.89:6807/1673955622","heartbeat_front_addr":"172.21.15.89:6805/1673955622","state":["exists","up"]},{"osd":1,"uuid":"0305670b-d7c4-4193-bec2-8b7351ff590a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6817","nonce":3340572039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6819","nonce":3340572039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6823","nonce":3340572039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6821","nonce":3340572039}]},"public_addr":"172.21.15.89:6817/3340572039","cluster_addr":"172.21.15.89:6819/3340572039","heartbeat_back_addr":"172.21.15.89:6823/3340572039","heartbeat_front_addr":"172.21.15.89:6821/3340572039","state":["exists","up"]},{"osd":2,"uuid":"4bab5426-a3c3-46b0-91fc-546c254006c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6809","nonce":1929940799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6811","nonce":1929940799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6815","nonce":1929940799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6813","nonce":1929940799}]},"public_addr":"172.21.15.89:6809/1929940799","cluster_addr":"172.21.15.89:6811/1929940799","heartbeat_back_addr":"172.21.15.89:6815/1929940799","heartbeat_front_addr":"172.21.15.89:6813/1929940799","state":["exists","up"]},{"osd":3,"uuid":"7d47aea8-27cd-42be-8591-d63ad68c90e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6825","nonce":345087939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6827","nonce":345087939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6831","nonce":345087939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6829","nonce":345087939}]},"public_addr":"172.21.15.89:6825/345087939","cluster_addr":"172.21.15.89:6827/345087939","heartbeat_back_addr":"172.21.15.89:6831/345087939","heartbeat_front_addr":"172.21.15.89:6829/345087939","state":["exists","up"]},{"osd":4,"uuid":"2e2ae8f8-d097-4bba-9448-95047fbfacad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6801","nonce":2138125545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6803","nonce":2138125545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6809","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6811","nonce":2138125545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6805","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6807","nonce":2138125545}]},"public_addr":"172.21.15.192:6801/2138125545","cluster_addr":"172.21.15.192:6803/2138125545","heartbeat_back_addr":"172.21.15.192:6811/2138125545","heartbeat_front_addr":"172.21.15.192:6807/2138125545","state":["exists","up"]},{"osd":5,"uuid":"49be5ce5-12f4-4c26-8d39-864b6ff67313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6806","nonce":2360218671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6810","nonce":2360218671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6815","nonce":2360218671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6813","nonce":2360218671}]},"public_addr":"172.21.15.192:6806/2360218671","cluster_addr":"172.21.15.192:6810/2360218671","heartbeat_back_addr":"172.21.15.192:6815/2360218671","heartbeat_front_addr":"172.21.15.192:6813/2360218671","state":["exists","up"]},{"osd":6,"uuid":"48623452-d456-4ea2-a0de-5fa4b37e6d10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6825","nonce":3707671287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6827","nonce":3707671287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6831","nonce":3707671287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6829","nonce":3707671287}]},"public_addr":"172.21.15.192:6825/3707671287","cluster_addr":"172.21.15.192:6827/3707671287","heartbeat_back_addr":"172.21.15.192:6831/3707671287","heartbeat_front_addr":"172.21.15.192:6829/3707671287","state":["exists","up"]},{"osd":7,"uuid":"ff0a4384-54f8-43ec-a3ce-9c8ae3bf8a94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6817","nonce":3111945871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6819","nonce":3111945871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6823","nonce":3111945871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6821","nonce":3111945871}]},"public_addr":"172.21.15.192:6817/3111945871","cluster_addr":"172.21.15.192:6819/3111945871","heartbeat_back_addr":"172.21.15.192:6823/3111945871","heartbeat_front_addr":"172.21.15.192:6821/3111945871","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.183065+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.248650+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.194576+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.397483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.043648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.186624+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.159674+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.100851+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:29:15.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.719+0000 7efca1f0f640 1 -- 172.21.15.89:0/2764887438 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7efc880664d0 msgr2=0x7efc88068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.719+0000 7efca1f0f640 1 --2- 172.21.15.89:0/2764887438 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7efc880664d0 0x7efc88068990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efc8c001710 tx=0x7efc8c009040 comp rx=0 tx=0).stop 2023-10-30T12:29:15.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.719+0000 7efca1f0f640 1 -- 172.21.15.89:0/2764887438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc9c1538a0 msgr2=0x7efc9c1efbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.719+0000 7efca1f0f640 1 --2- 172.21.15.89:0/2764887438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc9c1538a0 0x7efc9c1efbd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7efc90000c10 tx=0x7efc9001bad0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.719+0000 7efca1f0f640 1 -- 172.21.15.89:0/2764887438 shutdown_connections 2023-10-30T12:29:15.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.719+0000 7efca1f0f640 1 --2- 172.21.15.89:0/2764887438 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7efc880664d0 0x7efc88068990 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.719+0000 7efca1f0f640 1 --2- 172.21.15.89:0/2764887438 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efc9c15d6f0 0x7efc9c1587e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.719+0000 7efca1f0f640 1 --2- 172.21.15.89:0/2764887438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc9c1538a0 0x7efc9c1efbd0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.719+0000 7efca1f0f640 1 --2- 172.21.15.89:0/2764887438 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efc9c0cf160 0x7efc9c0b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.719+0000 7efca1f0f640 1 -- 172.21.15.89:0/2764887438 >> 172.21.15.89:0/2764887438 conn(0x7efc9c06bfe0 msgr2=0x7efc9c06c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:15.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.719+0000 7efca1f0f640 1 -- 172.21.15.89:0/2764887438 shutdown_connections 2023-10-30T12:29:15.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.719+0000 7efca1f0f640 1 -- 172.21.15.89:0/2764887438 wait complete. 2023-10-30T12:29:15.739 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T12:28:39.004943+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T12:28:48.935909+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T12:28:49.716098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T12:29:15.740 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T12:29:15.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.879+0000 7fc199c46640 1 Processor -- start 2023-10-30T12:29:15.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.879+0000 7fc199c46640 1 -- start start 2023-10-30T12:29:15.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.879+0000 7fc199c46640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc194150170 0x7fc194155d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.879+0000 7fc199c46640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1941592c0 con 0x7fc1940b0cb0 2023-10-30T12:29:15.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.879+0000 7fc199c46640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc19415a040 con 0x7fc194156260 2023-10-30T12:29:15.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.879+0000 7fc199c46640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc194072170 con 0x7fc194150170 2023-10-30T12:29:15.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.879+0000 7fc192ffd640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc1940b0cb0 0x7fc19406a720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58356/0 (socket says 172.21.15.89:58356) 2023-10-30T12:29:15.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.879+0000 7fc192ffd640 1 -- 172.21.15.89:0/2387291103 learned_addr learned my addr 172.21.15.89:0/2387291103 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:29:15.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.879+0000 7fc1937fe640 1 --2- 172.21.15.89:0/2387291103 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc194150170 0x7fc194155d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.879+0000 7fc1927fc640 1 -- 172.21.15.89:0/2387291103 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1337815830 0 0) 0x7fc1941592c0 con 0x7fc1940b0cb0 2023-10-30T12:29:15.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.879+0000 7fc1927fc640 1 -- 172.21.15.89:0/2387291103 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc17c003450 con 0x7fc1940b0cb0 2023-10-30T12:29:15.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.883+0000 7fc1927fc640 1 -- 172.21.15.89:0/2387291103 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262782703 0 0) 0x7fc19415a040 con 0x7fc194156260 2023-10-30T12:29:15.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.883+0000 7fc1927fc640 1 -- 172.21.15.89:0/2387291103 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1941592c0 con 0x7fc194156260 2023-10-30T12:29:15.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.883+0000 7fc1927fc640 1 -- 172.21.15.89:0/2387291103 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3680485931 0 0) 0x7fc17c003450 con 0x7fc1940b0cb0 2023-10-30T12:29:15.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.883+0000 7fc1927fc640 1 -- 172.21.15.89:0/2387291103 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc194150170 msgr2=0x7fc194155d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:15.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.883+0000 7fc1927fc640 1 --2- 172.21.15.89:0/2387291103 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc194150170 0x7fc194155d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.883+0000 7fc1927fc640 1 -- 172.21.15.89:0/2387291103 >> v1:172.21.15.192:6789/0 conn(0x7fc194156260 legacy=0x7fc194160590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.883+0000 7fc1927fc640 1 -- 172.21.15.89:0/2387291103 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc194159560 con 0x7fc1940b0cb0 2023-10-30T12:29:15.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.883+0000 7fc1927fc640 1 -- 172.21.15.89:0/2387291103 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7fc188001680 con 0x7fc1940b0cb0 2023-10-30T12:29:15.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.883+0000 7fc1927fc640 1 -- 172.21.15.89:0/2387291103 >> v1:172.21.15.89:6789/0 conn(0x7fc1940b0cb0 legacy=0x7fc19406a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.883+0000 7fc1927fc640 1 --2- 172.21.15.89:0/2387291103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc17c003c30 0x7fc17c008860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.883+0000 7fc1927fc640 1 --2- 172.21.15.89:0/2387291103 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc17c004110 0x7fc17c00adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.883+0000 7fc1927fc640 1 --2- 172.21.15.89:0/2387291103 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc194150170 0x7fc17c00d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.883+0000 7fc1927fc640 1 -- 172.21.15.89:0/2387291103 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc194072170 con 0x7fc17c003c30 2023-10-30T12:29:15.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.883+0000 7fc1927fc640 1 -- 172.21.15.89:0/2387291103 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c00dc10 con 0x7fc17c004110 2023-10-30T12:29:15.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.883+0000 7fc1927fc640 1 -- 172.21.15.89:0/2387291103 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c00ddd0 con 0x7fc194150170 2023-10-30T12:29:15.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.883+0000 7fc1937fe640 1 --2- 172.21.15.89:0/2387291103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc17c003c30 0x7fc17c008860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.883+0000 7fc192ffd640 1 --2- 172.21.15.89:0/2387291103 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc17c004110 0x7fc17c00adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.883+0000 7fc193fff640 1 --2- 172.21.15.89:0/2387291103 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc194150170 0x7fc17c00d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.883+0000 7fc1937fe640 1 -- 172.21.15.89:0/2387291103 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc194150170 msgr2=0x7fc17c00d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:15.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.883+0000 7fc1937fe640 1 --2- 172.21.15.89:0/2387291103 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc194150170 0x7fc17c00d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.883+0000 7fc1937fe640 1 -- 172.21.15.89:0/2387291103 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc17c004110 msgr2=0x7fc17c00adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:15.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.883+0000 7fc1937fe640 1 --2- 172.21.15.89:0/2387291103 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc17c004110 0x7fc17c00adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.883+0000 7fc1937fe640 1 -- 172.21.15.89:0/2387291103 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17c003450 con 0x7fc17c003c30 2023-10-30T12:29:15.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.883+0000 7fc192ffd640 1 --2- 172.21.15.89:0/2387291103 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc17c004110 0x7fc17c00adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:15.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.887+0000 7fc1937fe640 1 --2- 172.21.15.89:0/2387291103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc17c003c30 0x7fc17c008860 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc184009ac0 tx=0x7fc18401ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc47385bde702ce9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:15.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.887+0000 7fc1927fc640 1 -- 172.21.15.89:0/2387291103 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1840059a0 con 0x7fc17c003c30 2023-10-30T12:29:15.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.887+0000 7fc1927fc640 1 -- 172.21.15.89:0/2387291103 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc184005b00 con 0x7fc17c003c30 2023-10-30T12:29:15.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.887+0000 7fc1927fc640 1 -- 172.21.15.89:0/2387291103 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc184006030 con 0x7fc17c003c30 2023-10-30T12:29:15.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.887+0000 7fc199c46640 1 -- 172.21.15.89:0/2387291103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc17c003c30 msgr2=0x7fc17c008860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.887+0000 7fc199c46640 1 --2- 172.21.15.89:0/2387291103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc17c003c30 0x7fc17c008860 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc184009ac0 tx=0x7fc18401ca00 comp rx=0 tx=0).stop 2023-10-30T12:29:15.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.887+0000 7fc193fff640 1 -- 172.21.15.89:0/2387291103 reap_dead start 2023-10-30T12:29:15.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.887+0000 7fc199c46640 1 -- 172.21.15.89:0/2387291103 shutdown_connections 2023-10-30T12:29:15.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.887+0000 7fc199c46640 1 -- 172.21.15.89:0/2387291103 >> 172.21.15.89:0/2387291103 conn(0x7fc1940c7910 msgr2=0x7fc1940c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:15.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.887+0000 7fc199c46640 1 -- 172.21.15.89:0/2387291103 shutdown_connections 2023-10-30T12:29:15.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.887+0000 7fc199c46640 1 -- 172.21.15.89:0/2387291103 wait complete. 2023-10-30T12:29:15.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.887+0000 7fc199c46640 1 Processor -- start 2023-10-30T12:29:15.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.887+0000 7fc199c46640 1 -- start start 2023-10-30T12:29:15.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.891+0000 7fc199c46640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc17c003c30 0x7fc19414efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.891+0000 7fc199c46640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc194158e10 0x7fc1940c1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.891+0000 7fc199c46640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc19414e9a0 0x7fc194156e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.891+0000 7fc199c46640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1941f2ae0 con 0x7fc194158e10 2023-10-30T12:29:15.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.891+0000 7fc199c46640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1941f1f30 con 0x7fc17c003c30 2023-10-30T12:29:15.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.891+0000 7fc199c46640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc19414fa50 con 0x7fc19414e9a0 2023-10-30T12:29:15.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.891+0000 7fc192ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc194158e10 0x7fc1940c1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.891+0000 7fc192ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc194158e10 0x7fc1940c1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57814/0 (socket says 172.21.15.89:57814) 2023-10-30T12:29:15.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.891+0000 7fc192ffd640 1 -- 172.21.15.89:0/3316660965 learned_addr learned my addr 172.21.15.89:0/3316660965 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:15.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.891+0000 7fc1937fe640 1 --2- 172.21.15.89:0/3316660965 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc17c003c30 0x7fc19414efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.891+0000 7fc193fff640 1 --2- 172.21.15.89:0/3316660965 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc19414e9a0 0x7fc194156e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.891+0000 7fc192ffd640 1 -- 172.21.15.89:0/3316660965 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc19414e9a0 msgr2=0x7fc194156e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.891+0000 7fc192ffd640 1 --2- 172.21.15.89:0/3316660965 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc19414e9a0 0x7fc194156e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.891+0000 7fc192ffd640 1 -- 172.21.15.89:0/3316660965 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc17c003c30 msgr2=0x7fc19414efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.891+0000 7fc192ffd640 1 --2- 172.21.15.89:0/3316660965 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc17c003c30 0x7fc19414efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.891+0000 7fc192ffd640 1 -- 172.21.15.89:0/3316660965 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1941f2f00 con 0x7fc194158e10 2023-10-30T12:29:15.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.891+0000 7fc192ffd640 1 --2- 172.21.15.89:0/3316660965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc194158e10 0x7fc1940c1130 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc188000fc0 tx=0x7fc18800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:15.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.891+0000 7fc17bfff640 1 -- 172.21.15.89:0/3316660965 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc18800c8d0 con 0x7fc194158e10 2023-10-30T12:29:15.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.891+0000 7fc17bfff640 1 -- 172.21.15.89:0/3316660965 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc18800ca30 con 0x7fc194158e10 2023-10-30T12:29:15.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.891+0000 7fc17bfff640 1 -- 172.21.15.89:0/3316660965 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc18800cde0 con 0x7fc194158e10 2023-10-30T12:29:15.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.891+0000 7fc199c46640 1 -- 172.21.15.89:0/3316660965 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc19414f500 con 0x7fc194158e10 2023-10-30T12:29:15.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.891+0000 7fc199c46640 1 -- 172.21.15.89:0/3316660965 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1941f2140 con 0x7fc194158e10 2023-10-30T12:29:15.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.891+0000 7fc199c46640 1 -- 172.21.15.89:0/3316660965 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1941f2c20 con 0x7fc194158e10 2023-10-30T12:29:15.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.891+0000 7fc17bfff640 1 -- 172.21.15.89:0/3316660965 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc18801d460 con 0x7fc194158e10 2023-10-30T12:29:15.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.895+0000 7fc17bfff640 1 --2- 172.21.15.89:0/3316660965 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc1800664d0 0x7fc180068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.895+0000 7fc17bfff640 1 -- 172.21.15.89:0/3316660965 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc188083de0 con 0x7fc194158e10 2023-10-30T12:29:15.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.895+0000 7fc1937fe640 1 --2- 172.21.15.89:0/3316660965 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc1800664d0 0x7fc180068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.895+0000 7fc1937fe640 1 --2- 172.21.15.89:0/3316660965 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc1800664d0 0x7fc180068990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc1840ae000 tx=0x7fc1840046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:15.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:15.895+0000 7fc17bfff640 1 -- 172.21.15.89:0/3316660965 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc1941f2c20 con 0x7fc194158e10 2023-10-30T12:29:16.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.047+0000 7fc199c46640 1 -- 172.21.15.89:0/3316660965 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc1941f2c20 con 0x7fc194158e10 2023-10-30T12:29:16.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.047+0000 7fc17bfff640 1 -- 172.21.15.89:0/3316660965 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc1941f2c20 con 0x7fc194158e10 2023-10-30T12:29:16.051 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-10-30T12:29:16.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.051+0000 7fc199c46640 1 -- 172.21.15.89:0/3316660965 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc1800664d0 msgr2=0x7fc180068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.051+0000 7fc199c46640 1 --2- 172.21.15.89:0/3316660965 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc1800664d0 0x7fc180068990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc1840ae000 tx=0x7fc1840046d0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.051+0000 7fc199c46640 1 -- 172.21.15.89:0/3316660965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc194158e10 msgr2=0x7fc1940c1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.051+0000 7fc199c46640 1 --2- 172.21.15.89:0/3316660965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc194158e10 0x7fc1940c1130 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc188000fc0 tx=0x7fc18800ece0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.051+0000 7fc199c46640 1 -- 172.21.15.89:0/3316660965 shutdown_connections 2023-10-30T12:29:16.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.051+0000 7fc199c46640 1 --2- 172.21.15.89:0/3316660965 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc1800664d0 0x7fc180068990 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.051+0000 7fc199c46640 1 --2- 172.21.15.89:0/3316660965 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc19414e9a0 0x7fc194156e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.051+0000 7fc199c46640 1 --2- 172.21.15.89:0/3316660965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc194158e10 0x7fc1940c1130 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.051+0000 7fc199c46640 1 --2- 172.21.15.89:0/3316660965 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc17c003c30 0x7fc19414efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.051+0000 7fc199c46640 1 -- 172.21.15.89:0/3316660965 >> 172.21.15.89:0/3316660965 conn(0x7fc1940c7910 msgr2=0x7fc19406d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:16.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.051+0000 7fc199c46640 1 -- 172.21.15.89:0/3316660965 shutdown_connections 2023-10-30T12:29:16.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.051+0000 7fc199c46640 1 -- 172.21.15.89:0/3316660965 wait complete. 2023-10-30T12:29:16.072 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T12:29:16.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.211+0000 7f0cefe09640 1 Processor -- start 2023-10-30T12:29:16.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.211+0000 7f0cefe09640 1 -- start start 2023-10-30T12:29:16.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.211+0000 7f0cefe09640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0ce815d570 0x7f0ce815d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.211+0000 7f0cefe09640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce815a920 con 0x7f0ce8153980 2023-10-30T12:29:16.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.211+0000 7f0cefe09640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce80cb930 con 0x7f0ce8159930 2023-10-30T12:29:16.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.211+0000 7f0cefe09640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ce8159dc0 con 0x7f0ce815d570 2023-10-30T12:29:16.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.211+0000 7f0cedba5640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f0ce8153980 0x7f0ce80caca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58360/0 (socket says 172.21.15.89:58360) 2023-10-30T12:29:16.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.211+0000 7f0cedba5640 1 -- 172.21.15.89:0/3206597067 learned_addr learned my addr 172.21.15.89:0/3206597067 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:29:16.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.211+0000 7f0cee3a6640 1 --2- 172.21.15.89:0/3206597067 >> v2:172.21.15.192:3300/0 conn(0x7f0ce815d570 0x7f0ce815d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.211+0000 7f0cecba3640 1 -- 172.21.15.89:0/3206597067 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 331991233 0 0) 0x7f0ce815a920 con 0x7f0ce8153980 2023-10-30T12:29:16.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.211+0000 7f0cecba3640 1 -- 172.21.15.89:0/3206597067 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cd4003530 con 0x7f0ce8153980 2023-10-30T12:29:16.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.211+0000 7f0cecba3640 1 -- 172.21.15.89:0/3206597067 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2572462840 0 0) 0x7f0cd4003530 con 0x7f0ce8153980 2023-10-30T12:29:16.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.211+0000 7f0cecba3640 1 -- 172.21.15.89:0/3206597067 >> v2:172.21.15.192:3300/0 conn(0x7f0ce815d570 msgr2=0x7f0ce815d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:16.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.211+0000 7f0cecba3640 1 --2- 172.21.15.89:0/3206597067 >> v2:172.21.15.192:3300/0 conn(0x7f0ce815d570 0x7f0ce815d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.211+0000 7f0cecba3640 1 -- 172.21.15.89:0/3206597067 >> v1:172.21.15.192:6789/0 conn(0x7f0ce8159930 legacy=0x7f0ce80cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.211+0000 7f0cecba3640 1 -- 172.21.15.89:0/3206597067 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce815a080 con 0x7f0ce8153980 2023-10-30T12:29:16.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.211+0000 7f0cecba3640 1 -- 172.21.15.89:0/3206597067 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f0ce4001680 con 0x7f0ce8153980 2023-10-30T12:29:16.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.211+0000 7f0cecba3640 1 -- 172.21.15.89:0/3206597067 >> v1:172.21.15.89:6789/0 conn(0x7f0ce8153980 legacy=0x7f0ce80caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.215+0000 7f0cecba3640 1 --2- 172.21.15.89:0/3206597067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cd4003980 0x7f0cd4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.215+0000 7f0cecba3640 1 --2- 172.21.15.89:0/3206597067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0cd40047d0 0x7f0cd4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.215+0000 7f0cecba3640 1 --2- 172.21.15.89:0/3206597067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0cd4005190 0x7f0cd400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.215+0000 7f0cecba3640 1 -- 172.21.15.89:0/3206597067 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce8159dc0 con 0x7f0cd4003980 2023-10-30T12:29:16.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.215+0000 7f0cecba3640 1 -- 172.21.15.89:0/3206597067 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd400c030 con 0x7f0cd40047d0 2023-10-30T12:29:16.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.215+0000 7f0cecba3640 1 -- 172.21.15.89:0/3206597067 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd400c1f0 con 0x7f0cd4005190 2023-10-30T12:29:16.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.215+0000 7f0cedba5640 1 --2- 172.21.15.89:0/3206597067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cd4003980 0x7f0cd4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.215+0000 7f0cee3a6640 1 --2- 172.21.15.89:0/3206597067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0cd4005190 0x7f0cd400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.215+0000 7f0ced3a4640 1 --2- 172.21.15.89:0/3206597067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0cd40047d0 0x7f0cd4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.215+0000 7f0cedba5640 1 -- 172.21.15.89:0/3206597067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0cd4005190 msgr2=0x7f0cd400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:16.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.215+0000 7f0cedba5640 1 --2- 172.21.15.89:0/3206597067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0cd4005190 0x7f0cd400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.215+0000 7f0cedba5640 1 -- 172.21.15.89:0/3206597067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0cd40047d0 msgr2=0x7f0cd4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:16.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.215+0000 7f0cedba5640 1 --2- 172.21.15.89:0/3206597067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0cd40047d0 0x7f0cd4004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.215+0000 7f0cedba5640 1 -- 172.21.15.89:0/3206597067 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd4003530 con 0x7f0cd4003980 2023-10-30T12:29:16.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.215+0000 7f0cee3a6640 1 -- 172.21.15.89:0/3206597067 reap_dead start 2023-10-30T12:29:16.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.215+0000 7f0cedba5640 1 --2- 172.21.15.89:0/3206597067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cd4003980 0x7f0cd4003da0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0ce400c240 tx=0x7f0ce401cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19cf1dc7721c0e4b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:16.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.215+0000 7f0cecba3640 1 -- 172.21.15.89:0/3206597067 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4005ae0 con 0x7f0cd4003980 2023-10-30T12:29:16.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.215+0000 7f0cecba3640 1 -- 172.21.15.89:0/3206597067 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce4005c40 con 0x7f0cd4003980 2023-10-30T12:29:16.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.215+0000 7f0cecba3640 1 -- 172.21.15.89:0/3206597067 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce40a5ab0 con 0x7f0cd4003980 2023-10-30T12:29:16.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.219+0000 7f0cefe09640 1 -- 172.21.15.89:0/3206597067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cd4003980 msgr2=0x7f0cd4003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.219+0000 7f0cefe09640 1 --2- 172.21.15.89:0/3206597067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cd4003980 0x7f0cd4003da0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0ce400c240 tx=0x7f0ce401cb90 comp rx=0 tx=0).stop 2023-10-30T12:29:16.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.219+0000 7f0cefe09640 1 -- 172.21.15.89:0/3206597067 shutdown_connections 2023-10-30T12:29:16.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.219+0000 7f0cefe09640 1 --2- 172.21.15.89:0/3206597067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cd4003980 0x7f0cd4003da0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.219+0000 7f0cefe09640 1 -- 172.21.15.89:0/3206597067 >> 172.21.15.89:0/3206597067 conn(0x7f0ce806c0f0 msgr2=0x7f0ce806c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:16.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.219+0000 7f0cefe09640 1 -- 172.21.15.89:0/3206597067 shutdown_connections 2023-10-30T12:29:16.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.219+0000 7f0cefe09640 1 -- 172.21.15.89:0/3206597067 wait complete. 2023-10-30T12:29:16.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.219+0000 7f0cefe09640 1 Processor -- start 2023-10-30T12:29:16.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.219+0000 7f0cefe09640 1 -- start start 2023-10-30T12:29:16.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.219+0000 7f0cefe09640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0cd4003980 0x7f0ce814e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.219+0000 7f0cefe09640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ce80b2be0 0x7f0ce814eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.219+0000 7f0cefe09640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ce814f000 0x7f0ce8149ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.219+0000 7f0cefe09640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce80b3400 con 0x7f0ce80b2be0 2023-10-30T12:29:16.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.219+0000 7f0cefe09640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce81f6370 con 0x7f0ce814f000 2023-10-30T12:29:16.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.219+0000 7f0cefe09640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce80b3120 con 0x7f0cd4003980 2023-10-30T12:29:16.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.219+0000 7f0ced3a4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ce80b2be0 0x7f0ce814eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.219+0000 7f0ced3a4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ce80b2be0 0x7f0ce814eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57834/0 (socket says 172.21.15.89:57834) 2023-10-30T12:29:16.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.219+0000 7f0ced3a4640 1 -- 172.21.15.89:0/3588329846 learned_addr learned my addr 172.21.15.89:0/3588329846 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:16.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.219+0000 7f0cee3a6640 1 --2- 172.21.15.89:0/3588329846 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ce814f000 0x7f0ce8149ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.223+0000 7f0cedba5640 1 --2- 172.21.15.89:0/3588329846 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0cd4003980 0x7f0ce814e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.223+0000 7f0ced3a4640 1 -- 172.21.15.89:0/3588329846 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0cd4003980 msgr2=0x7f0ce814e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.223+0000 7f0ced3a4640 1 --2- 172.21.15.89:0/3588329846 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0cd4003980 0x7f0ce814e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.223+0000 7f0ced3a4640 1 -- 172.21.15.89:0/3588329846 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ce814f000 msgr2=0x7f0ce8149ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.223+0000 7f0ced3a4640 1 --2- 172.21.15.89:0/3588329846 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ce814f000 0x7f0ce8149ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.223+0000 7f0ced3a4640 1 -- 172.21.15.89:0/3588329846 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce81f55b0 con 0x7f0ce80b2be0 2023-10-30T12:29:16.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.223+0000 7f0cee3a6640 1 --2- 172.21.15.89:0/3588329846 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ce814f000 0x7f0ce8149ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:16.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.223+0000 7f0ced3a4640 1 --2- 172.21.15.89:0/3588329846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ce80b2be0 0x7f0ce814eac0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8001ec0 tx=0x7f0cd800e8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:16.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.223+0000 7f0cde7fc640 1 -- 172.21.15.89:0/3588329846 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd80052e0 con 0x7f0ce80b2be0 2023-10-30T12:29:16.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.223+0000 7f0cde7fc640 1 -- 172.21.15.89:0/3588329846 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd8005440 con 0x7f0ce80b2be0 2023-10-30T12:29:16.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.223+0000 7f0cde7fc640 1 -- 172.21.15.89:0/3588329846 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd8003820 con 0x7f0ce80b2be0 2023-10-30T12:29:16.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.223+0000 7f0cefe09640 1 -- 172.21.15.89:0/3588329846 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ce81f5890 con 0x7f0ce80b2be0 2023-10-30T12:29:16.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.223+0000 7f0cefe09640 1 -- 172.21.15.89:0/3588329846 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce81f5e50 con 0x7f0ce80b2be0 2023-10-30T12:29:16.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.223+0000 7f0cefe09640 1 -- 172.21.15.89:0/3588329846 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ce81f6010 con 0x7f0ce80b2be0 2023-10-30T12:29:16.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.223+0000 7f0cde7fc640 1 -- 172.21.15.89:0/3588329846 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0cd8003980 con 0x7f0ce80b2be0 2023-10-30T12:29:16.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.223+0000 7f0cde7fc640 1 --2- 172.21.15.89:0/3588329846 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0ccc0664d0 0x7f0ccc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.223+0000 7f0cde7fc640 1 -- 172.21.15.89:0/3588329846 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0cd8083650 con 0x7f0ce80b2be0 2023-10-30T12:29:16.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.227+0000 7f0cedba5640 1 --2- 172.21.15.89:0/3588329846 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0ccc0664d0 0x7f0ccc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.227+0000 7f0cedba5640 1 --2- 172.21.15.89:0/3588329846 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0ccc0664d0 0x7f0ccc068990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0ce40ab000 tx=0x7f0ce40a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:16.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.231+0000 7f0cde7fc640 1 -- 172.21.15.89:0/3588329846 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0cd8050b70 con 0x7f0ce80b2be0 2023-10-30T12:29:16.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.379+0000 7f0cefe09640 1 -- 172.21.15.89:0/3588329846 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0ce804e170 con 0x7f0ce80b2be0 2023-10-30T12:29:16.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.379+0000 7f0cde7fc640 1 -- 172.21.15.89:0/3588329846 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0cd8054190 con 0x7f0ce80b2be0 2023-10-30T12:29:16.382 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-10-30T12:29:16.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.379+0000 7f0cefe09640 1 -- 172.21.15.89:0/3588329846 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0ccc0664d0 msgr2=0x7f0ccc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.379+0000 7f0cefe09640 1 --2- 172.21.15.89:0/3588329846 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0ccc0664d0 0x7f0ccc068990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0ce40ab000 tx=0x7f0ce40a7070 comp rx=0 tx=0).stop 2023-10-30T12:29:16.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.379+0000 7f0cefe09640 1 -- 172.21.15.89:0/3588329846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ce80b2be0 msgr2=0x7f0ce814eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.379+0000 7f0cefe09640 1 --2- 172.21.15.89:0/3588329846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ce80b2be0 0x7f0ce814eac0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8001ec0 tx=0x7f0cd800e8a0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.379+0000 7f0cefe09640 1 -- 172.21.15.89:0/3588329846 shutdown_connections 2023-10-30T12:29:16.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.379+0000 7f0cefe09640 1 --2- 172.21.15.89:0/3588329846 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0ccc0664d0 0x7f0ccc068990 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.379+0000 7f0cefe09640 1 --2- 172.21.15.89:0/3588329846 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ce814f000 0x7f0ce8149ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.379+0000 7f0cefe09640 1 --2- 172.21.15.89:0/3588329846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ce80b2be0 0x7f0ce814eac0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.379+0000 7f0cefe09640 1 --2- 172.21.15.89:0/3588329846 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0cd4003980 0x7f0ce814e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.379+0000 7f0cefe09640 1 -- 172.21.15.89:0/3588329846 >> 172.21.15.89:0/3588329846 conn(0x7f0ce806c0f0 msgr2=0x7f0ce8158da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:16.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.383+0000 7f0cefe09640 1 -- 172.21.15.89:0/3588329846 shutdown_connections 2023-10-30T12:29:16.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.383+0000 7f0cefe09640 1 -- 172.21.15.89:0/3588329846 wait complete. 2023-10-30T12:29:16.402 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T12:29:16.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.539+0000 7f6cd09c4640 1 Processor -- start 2023-10-30T12:29:16.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.539+0000 7f6cd09c4640 1 -- start start 2023-10-30T12:29:16.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.543+0000 7f6cd09c4640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6ccc156fd0 0x7f6ccc06ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.543+0000 7f6cd09c4640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6ccc06fda0 0x7f6ccc071b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.543+0000 7f6cd09c4640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ccc072090 0x7f6ccc1598a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.543+0000 7f6cd09c4640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ccc06bbe0 con 0x7f6ccc06fda0 2023-10-30T12:29:16.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.543+0000 7f6cd09c4640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ccc1573b0 con 0x7f6ccc156fd0 2023-10-30T12:29:16.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.543+0000 7f6cd09c4640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ccc06b5c0 con 0x7f6ccc072090 2023-10-30T12:29:16.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.543+0000 7f6cc9d9b640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6ccc06fda0 0x7f6ccc071b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.543+0000 7f6cc9d9b640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6ccc06fda0 0x7f6ccc071b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57848/0 (socket says 172.21.15.89:57848) 2023-10-30T12:29:16.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.543+0000 7f6cc9d9b640 1 -- 172.21.15.89:0/2499059587 learned_addr learned my addr 172.21.15.89:0/2499059587 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:16.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.543+0000 7f6cca59c640 1 --2- 172.21.15.89:0/2499059587 >> v2:172.21.15.192:3300/0 conn(0x7f6ccc156fd0 0x7f6ccc06ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.543+0000 7f6ccad9d640 1 --2- 172.21.15.89:0/2499059587 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ccc072090 0x7f6ccc1598a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.543+0000 7f6cc9d9b640 1 -- 172.21.15.89:0/2499059587 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ccc072090 msgr2=0x7f6ccc1598a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:16.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.543+0000 7f6cc9d9b640 1 --2- 172.21.15.89:0/2499059587 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ccc072090 0x7f6ccc1598a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.543+0000 7f6cc9d9b640 1 -- 172.21.15.89:0/2499059587 >> v2:172.21.15.192:3300/0 conn(0x7f6ccc156fd0 msgr2=0x7f6ccc06ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:16.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.543+0000 7f6cc9d9b640 1 --2- 172.21.15.89:0/2499059587 >> v2:172.21.15.192:3300/0 conn(0x7f6ccc156fd0 0x7f6ccc06ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.543+0000 7f6cc9d9b640 1 -- 172.21.15.89:0/2499059587 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ccc06b2e0 con 0x7f6ccc06fda0 2023-10-30T12:29:16.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.547+0000 7f6cc9d9b640 1 --2- 172.21.15.89:0/2499059587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ccc06fda0 0x7f6ccc071b50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc0ca6a0 tx=0x7f6cc001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee99aa581a9a0c51 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:16.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.547+0000 7f6cc959a640 1 -- 172.21.15.89:0/2499059587 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc00a89b0 con 0x7f6ccc06fda0 2023-10-30T12:29:16.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.547+0000 7f6cc959a640 1 -- 172.21.15.89:0/2499059587 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cc00a8b10 con 0x7f6ccc06fda0 2023-10-30T12:29:16.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.547+0000 7f6cc959a640 1 -- 172.21.15.89:0/2499059587 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc00a8ea0 con 0x7f6ccc06fda0 2023-10-30T12:29:16.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.547+0000 7f6cd09c4640 1 -- 172.21.15.89:0/2499059587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ccc06fda0 msgr2=0x7f6ccc071b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.547+0000 7f6cd09c4640 1 --2- 172.21.15.89:0/2499059587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ccc06fda0 0x7f6ccc071b50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc0ca6a0 tx=0x7f6cc001ecf0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.547+0000 7f6cd09c4640 1 -- 172.21.15.89:0/2499059587 shutdown_connections 2023-10-30T12:29:16.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.547+0000 7f6cd09c4640 1 --2- 172.21.15.89:0/2499059587 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ccc072090 0x7f6ccc1598a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.547+0000 7f6cd09c4640 1 --2- 172.21.15.89:0/2499059587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ccc06fda0 0x7f6ccc071b50 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.547+0000 7f6cd09c4640 1 --2- 172.21.15.89:0/2499059587 >> v2:172.21.15.192:3300/0 conn(0x7f6ccc156fd0 0x7f6ccc06ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.547+0000 7f6cd09c4640 1 -- 172.21.15.89:0/2499059587 >> 172.21.15.89:0/2499059587 conn(0x7f6ccc069e90 msgr2=0x7f6ccc06a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:16.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.547+0000 7f6cd09c4640 1 -- 172.21.15.89:0/2499059587 shutdown_connections 2023-10-30T12:29:16.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.547+0000 7f6cd09c4640 1 -- 172.21.15.89:0/2499059587 wait complete. 2023-10-30T12:29:16.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.547+0000 7f6cd09c4640 1 Processor -- start 2023-10-30T12:29:16.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.547+0000 7f6cd09c4640 1 -- start start 2023-10-30T12:29:16.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.551+0000 7f6cd09c4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ccc06fda0 0x7f6ccc1521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.551+0000 7f6cd09c4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ccc072090 0x7f6ccc1f26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.551+0000 7f6cd09c4640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ccc156fd0 0x7f6ccc1f2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.551+0000 7f6cd09c4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ccc1f2290 con 0x7f6ccc06fda0 2023-10-30T12:29:16.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.551+0000 7f6cd09c4640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ccc1507c0 con 0x7f6ccc072090 2023-10-30T12:29:16.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.551+0000 7f6cd09c4640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ccc149c20 con 0x7f6ccc156fd0 2023-10-30T12:29:16.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.551+0000 7f6cca59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ccc06fda0 0x7f6ccc1521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.551+0000 7f6cca59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ccc06fda0 0x7f6ccc1521b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57862/0 (socket says 172.21.15.89:57862) 2023-10-30T12:29:16.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.551+0000 7f6cca59c640 1 -- 172.21.15.89:0/1494830833 learned_addr learned my addr 172.21.15.89:0/1494830833 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:16.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.551+0000 7f6ccad9d640 1 --2- 172.21.15.89:0/1494830833 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ccc156fd0 0x7f6ccc1f2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.551+0000 7f6cc9d9b640 1 --2- 172.21.15.89:0/1494830833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ccc072090 0x7f6ccc1f26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.551+0000 7f6cca59c640 1 -- 172.21.15.89:0/1494830833 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ccc156fd0 msgr2=0x7f6ccc1f2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.551+0000 7f6cca59c640 1 --2- 172.21.15.89:0/1494830833 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ccc156fd0 0x7f6ccc1f2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.551+0000 7f6cca59c640 1 -- 172.21.15.89:0/1494830833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ccc072090 msgr2=0x7f6ccc1f26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.551+0000 7f6cca59c640 1 --2- 172.21.15.89:0/1494830833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ccc072090 0x7f6ccc1f26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.551+0000 7f6cca59c640 1 -- 172.21.15.89:0/1494830833 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc00094d0 con 0x7f6ccc06fda0 2023-10-30T12:29:16.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.551+0000 7f6ccad9d640 1 --2- 172.21.15.89:0/1494830833 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ccc156fd0 0x7f6ccc1f2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:16.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.551+0000 7f6cca59c640 1 --2- 172.21.15.89:0/1494830833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ccc06fda0 0x7f6ccc1521b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc00d5f0 tx=0x7f6cbc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:16.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.551+0000 7f6cbaffd640 1 -- 172.21.15.89:0/1494830833 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc0125f0 con 0x7f6ccc06fda0 2023-10-30T12:29:16.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.551+0000 7f6cbaffd640 1 -- 172.21.15.89:0/1494830833 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cbc012750 con 0x7f6ccc06fda0 2023-10-30T12:29:16.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.551+0000 7f6cbaffd640 1 -- 172.21.15.89:0/1494830833 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc012ae0 con 0x7f6ccc06fda0 2023-10-30T12:29:16.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.551+0000 7f6cd09c4640 1 -- 172.21.15.89:0/1494830833 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ccc0c7190 con 0x7f6ccc06fda0 2023-10-30T12:29:16.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.551+0000 7f6cd09c4640 1 -- 172.21.15.89:0/1494830833 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ccc149940 con 0x7f6ccc06fda0 2023-10-30T12:29:16.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.551+0000 7f6cd09c4640 1 -- 172.21.15.89:0/1494830833 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ccc04e170 con 0x7f6ccc06fda0 2023-10-30T12:29:16.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.551+0000 7f6cbaffd640 1 -- 172.21.15.89:0/1494830833 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6cbc012c40 con 0x7f6ccc06fda0 2023-10-30T12:29:16.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.555+0000 7f6cbaffd640 1 --2- 172.21.15.89:0/1494830833 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6cac0664a0 0x7f6cac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.555+0000 7f6cbaffd640 1 -- 172.21.15.89:0/1494830833 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6cbc084320 con 0x7f6ccc06fda0 2023-10-30T12:29:16.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.555+0000 7f6cc9d9b640 1 --2- 172.21.15.89:0/1494830833 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6cac0664a0 0x7f6cac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.555+0000 7f6cc9d9b640 1 --2- 172.21.15.89:0/1494830833 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6cac0664a0 0x7f6cac068960 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6cc00b2000 tx=0x7f6cc0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:16.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.555+0000 7f6cbaffd640 1 -- 172.21.15.89:0/1494830833 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6cbc0517a0 con 0x7f6ccc06fda0 2023-10-30T12:29:16.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.707+0000 7f6cd09c4640 1 -- 172.21.15.89:0/1494830833 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6ccc1f2500 con 0x7f6ccc06fda0 2023-10-30T12:29:16.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.707+0000 7f6cbaffd640 1 -- 172.21.15.89:0/1494830833 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6cbc054dc0 con 0x7f6ccc06fda0 2023-10-30T12:29:16.708 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-10-30T12:29:16.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.707+0000 7f6cd09c4640 1 -- 172.21.15.89:0/1494830833 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6cac0664a0 msgr2=0x7f6cac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.707+0000 7f6cd09c4640 1 --2- 172.21.15.89:0/1494830833 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6cac0664a0 0x7f6cac068960 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6cc00b2000 tx=0x7f6cc0008040 comp rx=0 tx=0).stop 2023-10-30T12:29:16.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.707+0000 7f6cd09c4640 1 -- 172.21.15.89:0/1494830833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ccc06fda0 msgr2=0x7f6ccc1521b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.707+0000 7f6cd09c4640 1 --2- 172.21.15.89:0/1494830833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ccc06fda0 0x7f6ccc1521b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc00d5f0 tx=0x7f6cbc00dab0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.707+0000 7f6cd09c4640 1 -- 172.21.15.89:0/1494830833 shutdown_connections 2023-10-30T12:29:16.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.707+0000 7f6cd09c4640 1 --2- 172.21.15.89:0/1494830833 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6cac0664a0 0x7f6cac068960 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.707+0000 7f6cd09c4640 1 --2- 172.21.15.89:0/1494830833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ccc072090 0x7f6ccc1f26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.707+0000 7f6cd09c4640 1 --2- 172.21.15.89:0/1494830833 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ccc156fd0 0x7f6ccc1f2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.707+0000 7f6cd09c4640 1 --2- 172.21.15.89:0/1494830833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ccc06fda0 0x7f6ccc1521b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.707+0000 7f6cd09c4640 1 -- 172.21.15.89:0/1494830833 >> 172.21.15.89:0/1494830833 conn(0x7f6ccc069e90 msgr2=0x7f6ccc1547e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:16.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.707+0000 7f6cd09c4640 1 -- 172.21.15.89:0/1494830833 shutdown_connections 2023-10-30T12:29:16.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.707+0000 7f6cd09c4640 1 -- 172.21.15.89:0/1494830833 wait complete. 2023-10-30T12:29:16.729 INFO:tasks.mds_thrash:mds thrasher using random seed: 1698668956 2023-10-30T12:29:16.730 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:29:16.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.863+0000 7f84058ea640 1 Processor -- start 2023-10-30T12:29:16.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.863+0000 7f84058ea640 1 -- start start 2023-10-30T12:29:16.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f84058ea640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f840015d6f0 0x7f840015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f84058ea640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84000cb800 con 0x7f84000cf160 2023-10-30T12:29:16.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f84058ea640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84000cb330 con 0x7f84001538a0 2023-10-30T12:29:16.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f84058ea640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84000cad70 con 0x7f840015d6f0 2023-10-30T12:29:16.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f83ff7fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f84000cf160 0x7f8400164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58372/0 (socket says 172.21.15.89:58372) 2023-10-30T12:29:16.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f83ff7fe640 1 -- 172.21.15.89:0/4000731656 learned_addr learned my addr 172.21.15.89:0/4000731656 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:29:16.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f83f7fff640 1 --2- 172.21.15.89:0/4000731656 >> v2:172.21.15.192:3300/0 conn(0x7f840015d6f0 0x7f840015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f83fe7fc640 1 -- 172.21.15.89:0/4000731656 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2519706216 0 0) 0x7f84000cb800 con 0x7f84000cf160 2023-10-30T12:29:16.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f83fe7fc640 1 -- 172.21.15.89:0/4000731656 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83ec003530 con 0x7f84000cf160 2023-10-30T12:29:16.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f83fe7fc640 1 -- 172.21.15.89:0/4000731656 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 464330614 0 0) 0x7f83ec003530 con 0x7f84000cf160 2023-10-30T12:29:16.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f83fe7fc640 1 -- 172.21.15.89:0/4000731656 >> v2:172.21.15.192:3300/0 conn(0x7f840015d6f0 msgr2=0x7f840015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:16.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f83fe7fc640 1 --2- 172.21.15.89:0/4000731656 >> v2:172.21.15.192:3300/0 conn(0x7f840015d6f0 0x7f840015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f83fe7fc640 1 -- 172.21.15.89:0/4000731656 >> v1:172.21.15.192:6789/0 conn(0x7f84001538a0 legacy=0x7f84000ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f83fe7fc640 1 -- 172.21.15.89:0/4000731656 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84000cb050 con 0x7f84000cf160 2023-10-30T12:29:16.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f83fe7fc640 1 -- 172.21.15.89:0/4000731656 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f83f8001a00 con 0x7f84000cf160 2023-10-30T12:29:16.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f83fe7fc640 1 -- 172.21.15.89:0/4000731656 >> v1:172.21.15.89:6789/0 conn(0x7f84000cf160 legacy=0x7f8400164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f83fe7fc640 1 --2- 172.21.15.89:0/4000731656 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83ec003980 0x7f83ec003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f83fe7fc640 1 --2- 172.21.15.89:0/4000731656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83ec0047d0 0x7f83ec004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f83fe7fc640 1 --2- 172.21.15.89:0/4000731656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83ec005190 0x7f83ec00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f83fe7fc640 1 -- 172.21.15.89:0/4000731656 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84000cad70 con 0x7f83ec005190 2023-10-30T12:29:16.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f83fe7fc640 1 -- 172.21.15.89:0/4000731656 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ec00c040 con 0x7f83ec0047d0 2023-10-30T12:29:16.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f83fe7fc640 1 -- 172.21.15.89:0/4000731656 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83ec00c200 con 0x7f83ec003980 2023-10-30T12:29:16.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f83ff7fe640 1 --2- 172.21.15.89:0/4000731656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83ec005190 0x7f83ec00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f83f7fff640 1 --2- 172.21.15.89:0/4000731656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83ec0047d0 0x7f83ec004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f83feffd640 1 --2- 172.21.15.89:0/4000731656 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83ec003980 0x7f83ec003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f83ff7fe640 1 -- 172.21.15.89:0/4000731656 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83ec003980 msgr2=0x7f83ec003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:16.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f83ff7fe640 1 --2- 172.21.15.89:0/4000731656 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83ec003980 0x7f83ec003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f83ff7fe640 1 -- 172.21.15.89:0/4000731656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83ec0047d0 msgr2=0x7f83ec004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:16.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f83ff7fe640 1 --2- 172.21.15.89:0/4000731656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83ec0047d0 0x7f83ec004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f83ff7fe640 1 -- 172.21.15.89:0/4000731656 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ec003530 con 0x7f83ec005190 2023-10-30T12:29:16.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.867+0000 7f83f7fff640 1 --2- 172.21.15.89:0/4000731656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83ec0047d0 0x7f83ec004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:16.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.871+0000 7f83ff7fe640 1 -- 172.21.15.89:0/4000731656 reap_dead start 2023-10-30T12:29:16.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.871+0000 7f83ff7fe640 1 --2- 172.21.15.89:0/4000731656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83ec005190 0x7f83ec00b990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f83f800a830 tx=0x7f83f801d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a187573af2c4d949 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:16.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.871+0000 7f83fe7fc640 1 -- 172.21.15.89:0/4000731656 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f80072d0 con 0x7f83ec005190 2023-10-30T12:29:16.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.871+0000 7f83fe7fc640 1 -- 172.21.15.89:0/4000731656 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83f80a8070 con 0x7f83ec005190 2023-10-30T12:29:16.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.871+0000 7f83fe7fc640 1 -- 172.21.15.89:0/4000731656 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f80076b0 con 0x7f83ec005190 2023-10-30T12:29:16.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.871+0000 7f84058ea640 1 -- 172.21.15.89:0/4000731656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83ec005190 msgr2=0x7f83ec00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.871+0000 7f84058ea640 1 --2- 172.21.15.89:0/4000731656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83ec005190 0x7f83ec00b990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f83f800a830 tx=0x7f83f801d340 comp rx=0 tx=0).stop 2023-10-30T12:29:16.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.871+0000 7f84058ea640 1 -- 172.21.15.89:0/4000731656 shutdown_connections 2023-10-30T12:29:16.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.871+0000 7f84058ea640 1 --2- 172.21.15.89:0/4000731656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83ec005190 0x7f83ec00b990 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.871+0000 7f84058ea640 1 -- 172.21.15.89:0/4000731656 >> 172.21.15.89:0/4000731656 conn(0x7f840006bfe0 msgr2=0x7f840006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:16.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.871+0000 7f84058ea640 1 -- 172.21.15.89:0/4000731656 shutdown_connections 2023-10-30T12:29:16.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.871+0000 7f84058ea640 1 -- 172.21.15.89:0/4000731656 wait complete. 2023-10-30T12:29:16.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.871+0000 7f84058ea640 1 Processor -- start 2023-10-30T12:29:16.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.871+0000 7f84058ea640 1 -- start start 2023-10-30T12:29:16.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.875+0000 7f84058ea640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83ec005190 0x7f84000b4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.875+0000 7f84058ea640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84001ef860 0x7f84001f2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.875+0000 7f84058ea640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84001e3030 0x7f840014fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.875+0000 7f84058ea640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84001f3260 con 0x7f84001e3030 2023-10-30T12:29:16.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.875+0000 7f84058ea640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84000c50d0 con 0x7f84001ef860 2023-10-30T12:29:16.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.875+0000 7f84058ea640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84001efca0 con 0x7f83ec005190 2023-10-30T12:29:16.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.875+0000 7f83ff7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84001e3030 0x7f840014fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.875+0000 7f83ff7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84001e3030 0x7f840014fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57882/0 (socket says 172.21.15.89:57882) 2023-10-30T12:29:16.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.875+0000 7f83ff7fe640 1 -- 172.21.15.89:0/1842677741 learned_addr learned my addr 172.21.15.89:0/1842677741 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:16.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.875+0000 7f83feffd640 1 --2- 172.21.15.89:0/1842677741 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83ec005190 0x7f84000b4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.875+0000 7f83f7fff640 1 --2- 172.21.15.89:0/1842677741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84001ef860 0x7f84001f2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.875+0000 7f83ff7fe640 1 -- 172.21.15.89:0/1842677741 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83ec005190 msgr2=0x7f84000b4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.875+0000 7f83ff7fe640 1 --2- 172.21.15.89:0/1842677741 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83ec005190 0x7f84000b4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.875+0000 7f83ff7fe640 1 -- 172.21.15.89:0/1842677741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84001ef860 msgr2=0x7f84001f2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.875+0000 7f83ff7fe640 1 --2- 172.21.15.89:0/1842677741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84001ef860 0x7f84001f2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.875+0000 7f83ff7fe640 1 -- 172.21.15.89:0/1842677741 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84001eeda0 con 0x7f84001e3030 2023-10-30T12:29:16.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.875+0000 7f83ff7fe640 1 --2- 172.21.15.89:0/1842677741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84001e3030 0x7f840014fda0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f83f8007040 tx=0x7f83ec005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:16.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.875+0000 7f83fcff9640 1 -- 172.21.15.89:0/1842677741 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f80019c0 con 0x7f84001e3030 2023-10-30T12:29:16.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.875+0000 7f83fcff9640 1 -- 172.21.15.89:0/1842677741 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83f8007450 con 0x7f84001e3030 2023-10-30T12:29:16.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.875+0000 7f84058ea640 1 -- 172.21.15.89:0/1842677741 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84000c5320 con 0x7f84001e3030 2023-10-30T12:29:16.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.875+0000 7f83fcff9640 1 -- 172.21.15.89:0/1842677741 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f80077e0 con 0x7f84001e3030 2023-10-30T12:29:16.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.875+0000 7f84058ea640 1 -- 172.21.15.89:0/1842677741 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84000c58e0 con 0x7f84001e3030 2023-10-30T12:29:16.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.875+0000 7f84058ea640 1 -- 172.21.15.89:0/1842677741 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8400159870 con 0x7f84001e3030 2023-10-30T12:29:16.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.875+0000 7f83fcff9640 1 -- 172.21.15.89:0/1842677741 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f83f80aa070 con 0x7f84001e3030 2023-10-30T12:29:16.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.879+0000 7f83fcff9640 1 --2- 172.21.15.89:0/1842677741 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f83e00664d0 0x7f83e0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.879+0000 7f83fcff9640 1 -- 172.21.15.89:0/1842677741 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f83f811d760 con 0x7f84001e3030 2023-10-30T12:29:16.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.879+0000 7f83feffd640 1 --2- 172.21.15.89:0/1842677741 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f83e00664d0 0x7f83e0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.879+0000 7f83feffd640 1 --2- 172.21.15.89:0/1842677741 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f83e00664d0 0x7f83e0068990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f83f0005410 tx=0x7f83f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:16.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:16.883+0000 7f83fcff9640 1 -- 172.21.15.89:0/1842677741 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8400159870 con 0x7f84001e3030 2023-10-30T12:29:17.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.035+0000 7f84058ea640 1 -- 172.21.15.89:0/1842677741 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8400159870 con 0x7f84001e3030 2023-10-30T12:29:17.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.035+0000 7f83fcff9640 1 -- 172.21.15.89:0/1842677741 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f8400159870 con 0x7f84001e3030 2023-10-30T12:29:17.038 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:29:17.038 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":22,"fsid":"401641be-db6e-4c16-b3ac-fd84a86c4de4","created":"2023-10-30T12:28:26.198532+0000","modified":"2023-10-30T12:28:51.918285+0000","last_up_change":"2023-10-30T12:28:37.212205+0000","last_in_change":"2023-10-30T12:28:28.517734+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:39.004943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T12:28:48.935909+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T12:28:49.716098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72595c54-fd6e-4a5a-bb14-ff27e9377b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6801","nonce":1673955622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6803","nonce":1673955622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6807","nonce":1673955622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6805","nonce":1673955622}]},"public_addr":"172.21.15.89:6801/1673955622","cluster_addr":"172.21.15.89:6803/1673955622","heartbeat_back_addr":"172.21.15.89:6807/1673955622","heartbeat_front_addr":"172.21.15.89:6805/1673955622","state":["exists","up"]},{"osd":1,"uuid":"0305670b-d7c4-4193-bec2-8b7351ff590a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6817","nonce":3340572039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6819","nonce":3340572039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6823","nonce":3340572039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6821","nonce":3340572039}]},"public_addr":"172.21.15.89:6817/3340572039","cluster_addr":"172.21.15.89:6819/3340572039","heartbeat_back_addr":"172.21.15.89:6823/3340572039","heartbeat_front_addr":"172.21.15.89:6821/3340572039","state":["exists","up"]},{"osd":2,"uuid":"4bab5426-a3c3-46b0-91fc-546c254006c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6809","nonce":1929940799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6811","nonce":1929940799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6815","nonce":1929940799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6813","nonce":1929940799}]},"public_addr":"172.21.15.89:6809/1929940799","cluster_addr":"172.21.15.89:6811/1929940799","heartbeat_back_addr":"172.21.15.89:6815/1929940799","heartbeat_front_addr":"172.21.15.89:6813/1929940799","state":["exists","up"]},{"osd":3,"uuid":"7d47aea8-27cd-42be-8591-d63ad68c90e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6825","nonce":345087939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6827","nonce":345087939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6831","nonce":345087939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6829","nonce":345087939}]},"public_addr":"172.21.15.89:6825/345087939","cluster_addr":"172.21.15.89:6827/345087939","heartbeat_back_addr":"172.21.15.89:6831/345087939","heartbeat_front_addr":"172.21.15.89:6829/345087939","state":["exists","up"]},{"osd":4,"uuid":"2e2ae8f8-d097-4bba-9448-95047fbfacad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6801","nonce":2138125545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6803","nonce":2138125545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6809","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6811","nonce":2138125545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6805","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6807","nonce":2138125545}]},"public_addr":"172.21.15.192:6801/2138125545","cluster_addr":"172.21.15.192:6803/2138125545","heartbeat_back_addr":"172.21.15.192:6811/2138125545","heartbeat_front_addr":"172.21.15.192:6807/2138125545","state":["exists","up"]},{"osd":5,"uuid":"49be5ce5-12f4-4c26-8d39-864b6ff67313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6806","nonce":2360218671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6810","nonce":2360218671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6815","nonce":2360218671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6813","nonce":2360218671}]},"public_addr":"172.21.15.192:6806/2360218671","cluster_addr":"172.21.15.192:6810/2360218671","heartbeat_back_addr":"172.21.15.192:6815/2360218671","heartbeat_front_addr":"172.21.15.192:6813/2360218671","state":["exists","up"]},{"osd":6,"uuid":"48623452-d456-4ea2-a0de-5fa4b37e6d10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6825","nonce":3707671287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6827","nonce":3707671287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6831","nonce":3707671287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6829","nonce":3707671287}]},"public_addr":"172.21.15.192:6825/3707671287","cluster_addr":"172.21.15.192:6827/3707671287","heartbeat_back_addr":"172.21.15.192:6831/3707671287","heartbeat_front_addr":"172.21.15.192:6829/3707671287","state":["exists","up"]},{"osd":7,"uuid":"ff0a4384-54f8-43ec-a3ce-9c8ae3bf8a94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6817","nonce":3111945871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6819","nonce":3111945871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6823","nonce":3111945871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6821","nonce":3111945871}]},"public_addr":"172.21.15.192:6817/3111945871","cluster_addr":"172.21.15.192:6819/3111945871","heartbeat_back_addr":"172.21.15.192:6823/3111945871","heartbeat_front_addr":"172.21.15.192:6821/3111945871","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.183065+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.248650+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.194576+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.397483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.043648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.186624+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.159674+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.100851+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:29:17.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.035+0000 7f84058ea640 1 -- 172.21.15.89:0/1842677741 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f83e00664d0 msgr2=0x7f83e0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:17.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.035+0000 7f84058ea640 1 --2- 172.21.15.89:0/1842677741 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f83e00664d0 0x7f83e0068990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f83f0005410 tx=0x7f83f0009040 comp rx=0 tx=0).stop 2023-10-30T12:29:17.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.039+0000 7f84058ea640 1 -- 172.21.15.89:0/1842677741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84001e3030 msgr2=0x7f840014fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:17.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.039+0000 7f84058ea640 1 --2- 172.21.15.89:0/1842677741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84001e3030 0x7f840014fda0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f83f8007040 tx=0x7f83ec005800 comp rx=0 tx=0).stop 2023-10-30T12:29:17.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.039+0000 7f84058ea640 1 -- 172.21.15.89:0/1842677741 shutdown_connections 2023-10-30T12:29:17.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.039+0000 7f84058ea640 1 --2- 172.21.15.89:0/1842677741 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f83e00664d0 0x7f83e0068990 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.039+0000 7f84058ea640 1 --2- 172.21.15.89:0/1842677741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84001ef860 0x7f84001f2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.039+0000 7f84058ea640 1 --2- 172.21.15.89:0/1842677741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f84001e3030 0x7f840014fda0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.039+0000 7f84058ea640 1 --2- 172.21.15.89:0/1842677741 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83ec005190 0x7f84000b4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.039+0000 7f84058ea640 1 -- 172.21.15.89:0/1842677741 >> 172.21.15.89:0/1842677741 conn(0x7f840006bfe0 msgr2=0x7f8400066090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:17.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.039+0000 7f84058ea640 1 -- 172.21.15.89:0/1842677741 shutdown_connections 2023-10-30T12:29:17.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.039+0000 7f84058ea640 1 -- 172.21.15.89:0/1842677741 wait complete. 2023-10-30T12:29:17.059 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T12:28:39.004943+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T12:28:48.935909+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T12:28:49.716098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T12:29:17.059 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T12:29:17.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.199+0000 7fc83eb01640 1 Processor -- start 2023-10-30T12:29:17.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.199+0000 7fc83eb01640 1 -- start start 2023-10-30T12:29:17.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.199+0000 7fc83eb01640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc838153980 0x7fc8380caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:17.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.199+0000 7fc83eb01640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc838159930 0x7fc8380cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:17.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.199+0000 7fc83eb01640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc83815a920 con 0x7fc8380cf1a0 2023-10-30T12:29:17.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.199+0000 7fc83eb01640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8380cb930 con 0x7fc838153980 2023-10-30T12:29:17.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.199+0000 7fc83eb01640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc838159dc0 con 0x7fc838159930 2023-10-30T12:29:17.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.199+0000 7fc83c89d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc838153980 0x7fc8380caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:17.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.199+0000 7fc83c89d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc838153980 0x7fc8380caca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57888/0 (socket says 172.21.15.89:57888) 2023-10-30T12:29:17.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.199+0000 7fc83c89d640 1 -- 172.21.15.89:0/1957108196 learned_addr learned my addr 172.21.15.89:0/1957108196 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:17.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.199+0000 7fc82ffff640 1 --2- 172.21.15.89:0/1957108196 >> v2:172.21.15.192:3300/0 conn(0x7fc838159930 0x7fc8380cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:17.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.199+0000 7fc83c89d640 1 -- 172.21.15.89:0/1957108196 >> v2:172.21.15.192:3300/0 conn(0x7fc838159930 msgr2=0x7fc8380cb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:17.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.199+0000 7fc83c89d640 1 --2- 172.21.15.89:0/1957108196 >> v2:172.21.15.192:3300/0 conn(0x7fc838159930 0x7fc8380cb1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.199+0000 7fc83c89d640 1 -- 172.21.15.89:0/1957108196 >> v1:172.21.15.89:6789/0 conn(0x7fc8380cf1a0 legacy=0x7fc83815d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:17.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.199+0000 7fc83c89d640 1 -- 172.21.15.89:0/1957108196 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc83815a080 con 0x7fc838153980 2023-10-30T12:29:17.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.203+0000 7fc83c89d640 1 --2- 172.21.15.89:0/1957108196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc838153980 0x7fc8380caca0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc8380742c0 tx=0x7fc83401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a9ad8bfcdaeff92 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:17.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.203+0000 7fc82f7fe640 1 -- 172.21.15.89:0/1957108196 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8340aa430 con 0x7fc838153980 2023-10-30T12:29:17.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.203+0000 7fc82f7fe640 1 -- 172.21.15.89:0/1957108196 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8340aa590 con 0x7fc838153980 2023-10-30T12:29:17.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.203+0000 7fc82f7fe640 1 -- 172.21.15.89:0/1957108196 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8340aa920 con 0x7fc838153980 2023-10-30T12:29:17.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.203+0000 7fc83eb01640 1 -- 172.21.15.89:0/1957108196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc838153980 msgr2=0x7fc8380caca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:17.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.203+0000 7fc83eb01640 1 --2- 172.21.15.89:0/1957108196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc838153980 0x7fc8380caca0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc8380742c0 tx=0x7fc83401c4d0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.203+0000 7fc83eb01640 1 -- 172.21.15.89:0/1957108196 shutdown_connections 2023-10-30T12:29:17.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.203+0000 7fc83eb01640 1 --2- 172.21.15.89:0/1957108196 >> v2:172.21.15.192:3300/0 conn(0x7fc838159930 0x7fc8380cb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.203+0000 7fc83eb01640 1 --2- 172.21.15.89:0/1957108196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc838153980 0x7fc8380caca0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.203+0000 7fc83eb01640 1 -- 172.21.15.89:0/1957108196 >> 172.21.15.89:0/1957108196 conn(0x7fc83806c0f0 msgr2=0x7fc83806c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:17.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.203+0000 7fc83eb01640 1 -- 172.21.15.89:0/1957108196 shutdown_connections 2023-10-30T12:29:17.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.203+0000 7fc83eb01640 1 -- 172.21.15.89:0/1957108196 wait complete. 2023-10-30T12:29:17.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.207+0000 7fc83eb01640 1 Processor -- start 2023-10-30T12:29:17.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.207+0000 7fc83eb01640 1 -- start start 2023-10-30T12:29:17.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.207+0000 7fc83eb01640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8380cf1a0 0x7fc838154450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:17.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.207+0000 7fc83eb01640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc838153980 0x7fc8380661f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:17.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.207+0000 7fc83eb01640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc8380b39a0 0x7fc8380b3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:17.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.207+0000 7fc83eb01640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8380b3580 con 0x7fc8380cf1a0 2023-10-30T12:29:17.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.207+0000 7fc83eb01640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc838154990 con 0x7fc838153980 2023-10-30T12:29:17.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.207+0000 7fc83eb01640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8380b4330 con 0x7fc8380b39a0 2023-10-30T12:29:17.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.207+0000 7fc83c89d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8380cf1a0 0x7fc838154450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:17.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.207+0000 7fc83c89d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8380cf1a0 0x7fc838154450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57894/0 (socket says 172.21.15.89:57894) 2023-10-30T12:29:17.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.207+0000 7fc83c89d640 1 -- 172.21.15.89:0/972954017 learned_addr learned my addr 172.21.15.89:0/972954017 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:17.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.207+0000 7fc82ffff640 1 --2- 172.21.15.89:0/972954017 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc838153980 0x7fc8380661f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:17.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.207+0000 7fc83d09e640 1 --2- 172.21.15.89:0/972954017 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc8380b39a0 0x7fc8380b3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:17.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.207+0000 7fc83c89d640 1 -- 172.21.15.89:0/972954017 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc8380b39a0 msgr2=0x7fc8380b3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:17.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.207+0000 7fc83c89d640 1 --2- 172.21.15.89:0/972954017 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc8380b39a0 0x7fc8380b3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.207+0000 7fc83c89d640 1 -- 172.21.15.89:0/972954017 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc838153980 msgr2=0x7fc8380661f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:17.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.207+0000 7fc83c89d640 1 --2- 172.21.15.89:0/972954017 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc838153980 0x7fc8380661f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.207+0000 7fc83c89d640 1 -- 172.21.15.89:0/972954017 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8340094d0 con 0x7fc8380cf1a0 2023-10-30T12:29:17.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.207+0000 7fc82ffff640 1 --2- 172.21.15.89:0/972954017 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc838153980 0x7fc8380661f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:29:17.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.207+0000 7fc83c89d640 1 --2- 172.21.15.89:0/972954017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8380cf1a0 0x7fc838154450 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc8340190e0 tx=0x7fc83401ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:17.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.207+0000 7fc82d7fa640 1 -- 172.21.15.89:0/972954017 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8340b3800 con 0x7fc8380cf1a0 2023-10-30T12:29:17.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.207+0000 7fc82d7fa640 1 -- 172.21.15.89:0/972954017 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8340b3960 con 0x7fc8380cf1a0 2023-10-30T12:29:17.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.207+0000 7fc82d7fa640 1 -- 172.21.15.89:0/972954017 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8340b3d10 con 0x7fc8380cf1a0 2023-10-30T12:29:17.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.207+0000 7fc83eb01640 1 -- 172.21.15.89:0/972954017 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc83815c9c0 con 0x7fc8380cf1a0 2023-10-30T12:29:17.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.207+0000 7fc83eb01640 1 -- 172.21.15.89:0/972954017 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc83815b8d0 con 0x7fc8380cf1a0 2023-10-30T12:29:17.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.211+0000 7fc83eb01640 1 -- 172.21.15.89:0/972954017 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8380b36c0 con 0x7fc8380cf1a0 2023-10-30T12:29:17.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.211+0000 7fc82d7fa640 1 -- 172.21.15.89:0/972954017 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc8340b2380 con 0x7fc8380cf1a0 2023-10-30T12:29:17.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.211+0000 7fc82d7fa640 1 --2- 172.21.15.89:0/972954017 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc8240664a0 0x7fc824068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:17.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.211+0000 7fc82d7fa640 1 -- 172.21.15.89:0/972954017 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc83411e3c0 con 0x7fc8380cf1a0 2023-10-30T12:29:17.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.211+0000 7fc82ffff640 1 --2- 172.21.15.89:0/972954017 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc8240664a0 0x7fc824068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:17.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.211+0000 7fc82ffff640 1 --2- 172.21.15.89:0/972954017 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc8240664a0 0x7fc824068960 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc838152c20 tx=0x7fc828008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:17.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.215+0000 7fc82d7fa640 1 -- 172.21.15.89:0/972954017 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc8380b36c0 con 0x7fc8380cf1a0 2023-10-30T12:29:17.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.367+0000 7fc83eb01640 1 -- 172.21.15.89:0/972954017 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc8380b36c0 con 0x7fc8380cf1a0 2023-10-30T12:29:17.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.367+0000 7fc82d7fa640 1 -- 172.21.15.89:0/972954017 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc8380b36c0 con 0x7fc8380cf1a0 2023-10-30T12:29:17.371 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-10-30T12:29:17.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.371+0000 7fc83eb01640 1 -- 172.21.15.89:0/972954017 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc8240664a0 msgr2=0x7fc824068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:17.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.371+0000 7fc83eb01640 1 --2- 172.21.15.89:0/972954017 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc8240664a0 0x7fc824068960 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc838152c20 tx=0x7fc828008040 comp rx=0 tx=0).stop 2023-10-30T12:29:17.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.371+0000 7fc83eb01640 1 -- 172.21.15.89:0/972954017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8380cf1a0 msgr2=0x7fc838154450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:17.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.371+0000 7fc83eb01640 1 --2- 172.21.15.89:0/972954017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8380cf1a0 0x7fc838154450 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc8340190e0 tx=0x7fc83401ad10 comp rx=0 tx=0).stop 2023-10-30T12:29:17.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.371+0000 7fc83eb01640 1 -- 172.21.15.89:0/972954017 shutdown_connections 2023-10-30T12:29:17.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.371+0000 7fc83eb01640 1 --2- 172.21.15.89:0/972954017 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc8240664a0 0x7fc824068960 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.371+0000 7fc83eb01640 1 --2- 172.21.15.89:0/972954017 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc838153980 0x7fc8380661f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.371+0000 7fc83eb01640 1 --2- 172.21.15.89:0/972954017 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc8380b39a0 0x7fc8380b3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.371+0000 7fc83eb01640 1 --2- 172.21.15.89:0/972954017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc8380cf1a0 0x7fc838154450 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.371+0000 7fc83eb01640 1 -- 172.21.15.89:0/972954017 >> 172.21.15.89:0/972954017 conn(0x7fc83806c0f0 msgr2=0x7fc838158500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:17.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.371+0000 7fc83eb01640 1 -- 172.21.15.89:0/972954017 shutdown_connections 2023-10-30T12:29:17.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.371+0000 7fc83eb01640 1 -- 172.21.15.89:0/972954017 wait complete. 2023-10-30T12:29:17.391 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T12:29:17.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.527+0000 7f0e9f59e640 1 Processor -- start 2023-10-30T12:29:17.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.527+0000 7f0e9f59e640 1 -- start start 2023-10-30T12:29:17.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.531+0000 7f0e9f59e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0ea0153840 0x7f0ea015a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:17.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.531+0000 7f0e9f59e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0ea00cb4c0 0x7f0ea00cef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:17.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.531+0000 7f0e9f59e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ea015a6b0 con 0x7f0ea0153c20 2023-10-30T12:29:17.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.531+0000 7f0e9f59e640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ea015dbc0 con 0x7f0ea0153840 2023-10-30T12:29:17.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.531+0000 7f0e9f59e640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ea015d730 con 0x7f0ea00cb4c0 2023-10-30T12:29:17.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.531+0000 7f0e9e59c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0ea0153840 0x7f0ea015a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:17.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.531+0000 7f0e9e59c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0ea0153840 0x7f0ea015a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57898/0 (socket says 172.21.15.89:57898) 2023-10-30T12:29:17.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.531+0000 7f0e9e59c640 1 -- 172.21.15.89:0/2696932825 learned_addr learned my addr 172.21.15.89:0/2696932825 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:17.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.531+0000 7f0e9ed9d640 1 --2- 172.21.15.89:0/2696932825 >> v2:172.21.15.192:3300/0 conn(0x7f0ea00cb4c0 0x7f0ea00cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.531+0000 7f0e9e59c640 1 -- 172.21.15.89:0/2696932825 >> v2:172.21.15.192:3300/0 conn(0x7f0ea00cb4c0 msgr2=0x7f0ea00cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.531+0000 7f0e9e59c640 1 --2- 172.21.15.89:0/2696932825 >> v2:172.21.15.192:3300/0 conn(0x7f0ea00cb4c0 0x7f0ea00cef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.531+0000 7f0e9e59c640 1 -- 172.21.15.89:0/2696932825 >> v1:172.21.15.192:6789/0 conn(0x7f0ea0153c20 legacy=0x7f0ea00c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.531+0000 7f0e9e59c640 1 -- 172.21.15.89:0/2696932825 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea00ca8f0 con 0x7f0ea0153840 2023-10-30T12:29:17.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.531+0000 7f0e9e59c640 1 --2- 172.21.15.89:0/2696932825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ea0153840 0x7f0ea015a170 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0ea00cfb80 tx=0x7f0e9001ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c4e27a6ece3b306 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:17.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.535+0000 7f0e9dd9b640 1 -- 172.21.15.89:0/2696932825 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e900a89b0 con 0x7f0ea0153840 2023-10-30T12:29:17.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.535+0000 7f0e9dd9b640 1 -- 172.21.15.89:0/2696932825 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e900a8b10 con 0x7f0ea0153840 2023-10-30T12:29:17.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.535+0000 7f0e9dd9b640 1 -- 172.21.15.89:0/2696932825 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e900a8ea0 con 0x7f0ea0153840 2023-10-30T12:29:17.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.535+0000 7f0e9f59e640 1 -- 172.21.15.89:0/2696932825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ea0153840 msgr2=0x7f0ea015a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:17.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.535+0000 7f0e9f59e640 1 --2- 172.21.15.89:0/2696932825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ea0153840 0x7f0ea015a170 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0ea00cfb80 tx=0x7f0e9001ecb0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.535+0000 7f0e9f59e640 1 -- 172.21.15.89:0/2696932825 shutdown_connections 2023-10-30T12:29:17.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.535+0000 7f0e9f59e640 1 --2- 172.21.15.89:0/2696932825 >> v2:172.21.15.192:3300/0 conn(0x7f0ea00cb4c0 0x7f0ea00cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.535+0000 7f0e9f59e640 1 --2- 172.21.15.89:0/2696932825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ea0153840 0x7f0ea015a170 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.535+0000 7f0e9f59e640 1 -- 172.21.15.89:0/2696932825 >> 172.21.15.89:0/2696932825 conn(0x7f0ea006be40 msgr2=0x7f0ea006c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:17.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.535+0000 7f0e9f59e640 1 -- 172.21.15.89:0/2696932825 shutdown_connections 2023-10-30T12:29:17.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.535+0000 7f0e9f59e640 1 -- 172.21.15.89:0/2696932825 wait complete. 2023-10-30T12:29:17.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.535+0000 7f0e9f59e640 1 Processor -- start 2023-10-30T12:29:17.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.535+0000 7f0e9f59e640 1 -- start start 2023-10-30T12:29:17.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.535+0000 7f0e9f59e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ea00cb4c0 0x7f0ea014ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:17.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.535+0000 7f0e9f59e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ea0153840 0x7f0ea014f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:17.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.535+0000 7f0e9f59e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ea0153c20 0x7f0ea01f06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:17.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.535+0000 7f0e9f59e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea015ab60 con 0x7f0ea0153c20 2023-10-30T12:29:17.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.535+0000 7f0e9f59e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea015ae20 con 0x7f0ea00cb4c0 2023-10-30T12:29:17.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.535+0000 7f0e9f59e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea015b100 con 0x7f0ea0153840 2023-10-30T12:29:17.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.539+0000 7f0e9ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ea0153c20 0x7f0ea01f06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:17.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.539+0000 7f0e9ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ea0153c20 0x7f0ea01f06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57912/0 (socket says 172.21.15.89:57912) 2023-10-30T12:29:17.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.539+0000 7f0e9ed9d640 1 -- 172.21.15.89:0/3947947533 learned_addr learned my addr 172.21.15.89:0/3947947533 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:17.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.539+0000 7f0e9e59c640 1 --2- 172.21.15.89:0/3947947533 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ea00cb4c0 0x7f0ea014ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:17.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.539+0000 7f0e95d9b640 1 --2- 172.21.15.89:0/3947947533 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ea0153840 0x7f0ea014f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:17.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.539+0000 7f0e9ed9d640 1 -- 172.21.15.89:0/3947947533 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ea0153840 msgr2=0x7f0ea014f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:17.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.539+0000 7f0e9ed9d640 1 --2- 172.21.15.89:0/3947947533 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ea0153840 0x7f0ea014f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.539+0000 7f0e9ed9d640 1 -- 172.21.15.89:0/3947947533 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ea00cb4c0 msgr2=0x7f0ea014ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:17.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.539+0000 7f0e9ed9d640 1 --2- 172.21.15.89:0/3947947533 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ea00cb4c0 0x7f0ea014ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.539+0000 7f0e9ed9d640 1 -- 172.21.15.89:0/3947947533 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e900094d0 con 0x7f0ea0153c20 2023-10-30T12:29:17.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.539+0000 7f0e95d9b640 1 --2- 172.21.15.89:0/3947947533 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ea0153840 0x7f0ea014f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:17.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.539+0000 7f0e9ed9d640 1 --2- 172.21.15.89:0/3947947533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ea0153c20 0x7f0ea01f06e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0e98007e40 tx=0x7f0e9800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:17.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.539+0000 7f0e977fe640 1 -- 172.21.15.89:0/3947947533 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e98014620 con 0x7f0ea0153c20 2023-10-30T12:29:17.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.539+0000 7f0e977fe640 1 -- 172.21.15.89:0/3947947533 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e9800ee90 con 0x7f0ea0153c20 2023-10-30T12:29:17.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.539+0000 7f0e9f59e640 1 -- 172.21.15.89:0/3947947533 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ea015b3e0 con 0x7f0ea0153c20 2023-10-30T12:29:17.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.539+0000 7f0e977fe640 1 -- 172.21.15.89:0/3947947533 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e980149e0 con 0x7f0ea0153c20 2023-10-30T12:29:17.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.539+0000 7f0e9f59e640 1 -- 172.21.15.89:0/3947947533 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ea015b9a0 con 0x7f0ea0153c20 2023-10-30T12:29:17.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.539+0000 7f0e9f59e640 1 -- 172.21.15.89:0/3947947533 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ea004e9c0 con 0x7f0ea0153c20 2023-10-30T12:29:17.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.539+0000 7f0e977fe640 1 -- 172.21.15.89:0/3947947533 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0e98014b40 con 0x7f0ea0153c20 2023-10-30T12:29:17.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.543+0000 7f0e977fe640 1 --2- 172.21.15.89:0/3947947533 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0e800664a0 0x7f0e80068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:17.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.543+0000 7f0e977fe640 1 -- 172.21.15.89:0/3947947533 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e98084800 con 0x7f0ea0153c20 2023-10-30T12:29:17.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.543+0000 7f0e9e59c640 1 --2- 172.21.15.89:0/3947947533 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0e800664a0 0x7f0e80068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:17.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.543+0000 7f0e9e59c640 1 --2- 172.21.15.89:0/3947947533 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0e800664a0 0x7f0e80068960 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0e900b2000 tx=0x7f0e90009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:17.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.543+0000 7f0e977fe640 1 -- 172.21.15.89:0/3947947533 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0e98051bf0 con 0x7f0ea0153c20 2023-10-30T12:29:17.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.691+0000 7f0e9f59e640 1 -- 172.21.15.89:0/3947947533 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0ea015d390 con 0x7f0ea0153c20 2023-10-30T12:29:17.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.691+0000 7f0e977fe640 1 -- 172.21.15.89:0/3947947533 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0e98055210 con 0x7f0ea0153c20 2023-10-30T12:29:17.695 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-10-30T12:29:17.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.695+0000 7f0e9f59e640 1 -- 172.21.15.89:0/3947947533 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0e800664a0 msgr2=0x7f0e80068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:17.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.695+0000 7f0e9f59e640 1 --2- 172.21.15.89:0/3947947533 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0e800664a0 0x7f0e80068960 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0e900b2000 tx=0x7f0e90009040 comp rx=0 tx=0).stop 2023-10-30T12:29:17.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.695+0000 7f0e9f59e640 1 -- 172.21.15.89:0/3947947533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ea0153c20 msgr2=0x7f0ea01f06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:17.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.695+0000 7f0e9f59e640 1 --2- 172.21.15.89:0/3947947533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ea0153c20 0x7f0ea01f06e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0e98007e40 tx=0x7f0e9800e4b0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.695+0000 7f0e9f59e640 1 -- 172.21.15.89:0/3947947533 shutdown_connections 2023-10-30T12:29:17.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.695+0000 7f0e9f59e640 1 --2- 172.21.15.89:0/3947947533 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0e800664a0 0x7f0e80068960 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.695+0000 7f0e9f59e640 1 --2- 172.21.15.89:0/3947947533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ea0153c20 0x7f0ea01f06e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.695+0000 7f0e9f59e640 1 --2- 172.21.15.89:0/3947947533 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ea0153840 0x7f0ea014f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.695+0000 7f0e9f59e640 1 --2- 172.21.15.89:0/3947947533 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ea00cb4c0 0x7f0ea014ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.695+0000 7f0e9f59e640 1 -- 172.21.15.89:0/3947947533 >> 172.21.15.89:0/3947947533 conn(0x7f0ea006be40 msgr2=0x7f0ea0152730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:17.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.695+0000 7f0e9f59e640 1 -- 172.21.15.89:0/3947947533 shutdown_connections 2023-10-30T12:29:17.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.695+0000 7f0e9f59e640 1 -- 172.21.15.89:0/3947947533 wait complete. 2023-10-30T12:29:17.716 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T12:29:17.855 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.851+0000 7f235cf3e640 1 Processor -- start 2023-10-30T12:29:17.855 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.851+0000 7f235cf3e640 1 -- start start 2023-10-30T12:29:17.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f235cf3e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f235806fac0 0x7f23580ccd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:17.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f235cf3e640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2358159810 con 0x7f2358142b00 2023-10-30T12:29:17.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f235cf3e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2358159a80 con 0x7f235806fec0 2023-10-30T12:29:17.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f235cf3e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2358159d60 con 0x7f235806fac0 2023-10-30T12:29:17.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f235659c640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2358142b00 0x7f235815d6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58376/0 (socket says 172.21.15.89:58376) 2023-10-30T12:29:17.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f235659c640 1 -- 172.21.15.89:0/2943342816 learned_addr learned my addr 172.21.15.89:0/2943342816 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:29:17.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f234dd9b640 1 --2- 172.21.15.89:0/2943342816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f235806fac0 0x7f23580ccd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:17.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f2355d9b640 1 -- 172.21.15.89:0/2943342816 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4010152088 0 0) 0x7f2358159810 con 0x7f2358142b00 2023-10-30T12:29:17.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f2355d9b640 1 -- 172.21.15.89:0/2943342816 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2344003530 con 0x7f2358142b00 2023-10-30T12:29:17.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f2355d9b640 1 -- 172.21.15.89:0/2943342816 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4152296359 0 0) 0x7f2358159a80 con 0x7f235806fec0 2023-10-30T12:29:17.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f2355d9b640 1 -- 172.21.15.89:0/2943342816 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2358159810 con 0x7f235806fec0 2023-10-30T12:29:17.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f2355d9b640 1 -- 172.21.15.89:0/2943342816 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3604263447 0 0) 0x7f2344003530 con 0x7f2358142b00 2023-10-30T12:29:17.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f2355d9b640 1 -- 172.21.15.89:0/2943342816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f235806fac0 msgr2=0x7f23580ccd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:17.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f2355d9b640 1 --2- 172.21.15.89:0/2943342816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f235806fac0 0x7f23580ccd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f2355d9b640 1 -- 172.21.15.89:0/2943342816 >> v1:172.21.15.192:6789/0 conn(0x7f235806fec0 legacy=0x7f23580c2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:17.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f2355d9b640 1 -- 172.21.15.89:0/2943342816 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f235815a040 con 0x7f2358142b00 2023-10-30T12:29:17.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f2355d9b640 1 -- 172.21.15.89:0/2943342816 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f2348001980 con 0x7f2358142b00 2023-10-30T12:29:17.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f2355d9b640 1 -- 172.21.15.89:0/2943342816 >> v1:172.21.15.89:6789/0 conn(0x7f2358142b00 legacy=0x7f235815d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:17.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f2355d9b640 1 --2- 172.21.15.89:0/2943342816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2344003b40 0x7f2344003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:17.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f2355d9b640 1 --2- 172.21.15.89:0/2943342816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23440047d0 0x7f2344004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:17.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f2355d9b640 1 --2- 172.21.15.89:0/2943342816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f235806fac0 0x7f2344005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:17.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f2355d9b640 1 -- 172.21.15.89:0/2943342816 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2358159d60 con 0x7f2344003b40 2023-10-30T12:29:17.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f2355d9b640 1 -- 172.21.15.89:0/2943342816 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f234400ba40 con 0x7f23440047d0 2023-10-30T12:29:17.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f2355d9b640 1 -- 172.21.15.89:0/2943342816 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f234400bc00 con 0x7f235806fac0 2023-10-30T12:29:17.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f235659c640 1 --2- 172.21.15.89:0/2943342816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2344003b40 0x7f2344003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:17.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f234dd9b640 1 --2- 172.21.15.89:0/2943342816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23440047d0 0x7f2344004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:17.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f2356d9d640 1 --2- 172.21.15.89:0/2943342816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f235806fac0 0x7f2344005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:17.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f235659c640 1 -- 172.21.15.89:0/2943342816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f235806fac0 msgr2=0x7f2344005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:17.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f235659c640 1 --2- 172.21.15.89:0/2943342816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f235806fac0 0x7f2344005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f235659c640 1 -- 172.21.15.89:0/2943342816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23440047d0 msgr2=0x7f2344004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:17.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f235659c640 1 --2- 172.21.15.89:0/2943342816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23440047d0 0x7f2344004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f235659c640 1 -- 172.21.15.89:0/2943342816 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2344003530 con 0x7f2344003b40 2023-10-30T12:29:17.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.855+0000 7f234dd9b640 1 --2- 172.21.15.89:0/2943342816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23440047d0 0x7f2344004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:17.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.859+0000 7f235659c640 1 --2- 172.21.15.89:0/2943342816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2344003b40 0x7f2344003f40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f234800a6b0 tx=0x7f234801d1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=761e0e8d06241d29 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:17.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.859+0000 7f2355d9b640 1 -- 172.21.15.89:0/2943342816 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2348007330 con 0x7f2344003b40 2023-10-30T12:29:17.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.859+0000 7f2355d9b640 1 -- 172.21.15.89:0/2943342816 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23480a8070 con 0x7f2344003b40 2023-10-30T12:29:17.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.859+0000 7f2355d9b640 1 -- 172.21.15.89:0/2943342816 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2348007730 con 0x7f2344003b40 2023-10-30T12:29:17.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.859+0000 7f235cf3e640 1 -- 172.21.15.89:0/2943342816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2344003b40 msgr2=0x7f2344003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:17.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.859+0000 7f235cf3e640 1 --2- 172.21.15.89:0/2943342816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2344003b40 0x7f2344003f40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f234800a6b0 tx=0x7f234801d1a0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.859+0000 7f2356d9d640 1 -- 172.21.15.89:0/2943342816 reap_dead start 2023-10-30T12:29:17.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.859+0000 7f235cf3e640 1 -- 172.21.15.89:0/2943342816 shutdown_connections 2023-10-30T12:29:17.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.859+0000 7f235cf3e640 1 -- 172.21.15.89:0/2943342816 >> 172.21.15.89:0/2943342816 conn(0x7f2358071790 msgr2=0x7f2358071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:17.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.859+0000 7f235cf3e640 1 -- 172.21.15.89:0/2943342816 shutdown_connections 2023-10-30T12:29:17.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.859+0000 7f235cf3e640 1 -- 172.21.15.89:0/2943342816 wait complete. 2023-10-30T12:29:17.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.863+0000 7f235cf3e640 1 Processor -- start 2023-10-30T12:29:17.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.863+0000 7f235cf3e640 1 -- start start 2023-10-30T12:29:17.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.863+0000 7f235cf3e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2344003b40 0x7f235815c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:17.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.863+0000 7f235cf3e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23581e5010 0x7f23581554f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:17.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.863+0000 7f235cf3e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23581e5490 0x7f2358155a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:17.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.863+0000 7f235cf3e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2358154f30 con 0x7f2344003b40 2023-10-30T12:29:17.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.863+0000 7f235cf3e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23580c98a0 con 0x7f23581e5490 2023-10-30T12:29:17.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.863+0000 7f235cf3e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23580c8fb0 con 0x7f23581e5010 2023-10-30T12:29:17.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.863+0000 7f235659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2344003b40 0x7f235815c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:17.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.863+0000 7f235659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2344003b40 0x7f235815c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57922/0 (socket says 172.21.15.89:57922) 2023-10-30T12:29:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.863+0000 7f235659c640 1 -- 172.21.15.89:0/2653520766 learned_addr learned my addr 172.21.15.89:0/2653520766 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.863+0000 7f234dd9b640 1 --2- 172.21.15.89:0/2653520766 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23581e5010 0x7f23581554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.863+0000 7f2356d9d640 1 --2- 172.21.15.89:0/2653520766 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23581e5490 0x7f2358155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.863+0000 7f235659c640 1 -- 172.21.15.89:0/2653520766 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23581e5010 msgr2=0x7f23581554f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:17.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.863+0000 7f235659c640 1 --2- 172.21.15.89:0/2653520766 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23581e5010 0x7f23581554f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.863+0000 7f235659c640 1 -- 172.21.15.89:0/2653520766 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23581e5490 msgr2=0x7f2358155a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:17.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.863+0000 7f235659c640 1 --2- 172.21.15.89:0/2653520766 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23581e5490 0x7f2358155a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:17.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.863+0000 7f235659c640 1 -- 172.21.15.89:0/2653520766 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f235815b710 con 0x7f2344003b40 2023-10-30T12:29:17.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.863+0000 7f2356d9d640 1 --2- 172.21.15.89:0/2653520766 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23581e5490 0x7f2358155a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:29:17.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.863+0000 7f235659c640 1 --2- 172.21.15.89:0/2653520766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2344003b40 0x7f235815c460 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f234800a6b0 tx=0x7f2344005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:17.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.863+0000 7f234f7fe640 1 -- 172.21.15.89:0/2653520766 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2348004d70 con 0x7f2344003b40 2023-10-30T12:29:17.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.863+0000 7f234f7fe640 1 -- 172.21.15.89:0/2653520766 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2348001940 con 0x7f2344003b40 2023-10-30T12:29:17.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.863+0000 7f235cf3e640 1 -- 172.21.15.89:0/2653520766 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f235815b9f0 con 0x7f2344003b40 2023-10-30T12:29:17.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.863+0000 7f234f7fe640 1 -- 172.21.15.89:0/2653520766 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2348007330 con 0x7f2344003b40 2023-10-30T12:29:17.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.863+0000 7f235cf3e640 1 -- 172.21.15.89:0/2653520766 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23580c9530 con 0x7f2344003b40 2023-10-30T12:29:17.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.863+0000 7f235cf3e640 1 -- 172.21.15.89:0/2653520766 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f235804e170 con 0x7f2344003b40 2023-10-30T12:29:17.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.867+0000 7f234f7fe640 1 -- 172.21.15.89:0/2653520766 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f23480aa030 con 0x7f2344003b40 2023-10-30T12:29:17.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.867+0000 7f234f7fe640 1 --2- 172.21.15.89:0/2653520766 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23380664d0 0x7f2338068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:17.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.867+0000 7f234f7fe640 1 -- 172.21.15.89:0/2653520766 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f234811e660 con 0x7f2344003b40 2023-10-30T12:29:17.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.867+0000 7f234dd9b640 1 --2- 172.21.15.89:0/2653520766 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23380664d0 0x7f2338068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:17.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.867+0000 7f234dd9b640 1 --2- 172.21.15.89:0/2653520766 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23380664d0 0x7f2338068990 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f235806b890 tx=0x7f2340008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:17.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:17.871+0000 7f234f7fe640 1 -- 172.21.15.89:0/2653520766 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f23480ebac0 con 0x7f2344003b40 2023-10-30T12:29:18.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.019+0000 7f235cf3e640 1 -- 172.21.15.89:0/2653520766 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2358158d30 con 0x7f2344003b40 2023-10-30T12:29:18.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.019+0000 7f234f7fe640 1 -- 172.21.15.89:0/2653520766 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f23480bd0a0 con 0x7f2344003b40 2023-10-30T12:29:18.023 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-10-30T12:29:18.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.023+0000 7f235cf3e640 1 -- 172.21.15.89:0/2653520766 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23380664d0 msgr2=0x7f2338068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:18.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.023+0000 7f235cf3e640 1 --2- 172.21.15.89:0/2653520766 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23380664d0 0x7f2338068990 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f235806b890 tx=0x7f2340008040 comp rx=0 tx=0).stop 2023-10-30T12:29:18.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.023+0000 7f235cf3e640 1 -- 172.21.15.89:0/2653520766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2344003b40 msgr2=0x7f235815c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:18.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.023+0000 7f235cf3e640 1 --2- 172.21.15.89:0/2653520766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2344003b40 0x7f235815c460 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f234800a6b0 tx=0x7f2344005800 comp rx=0 tx=0).stop 2023-10-30T12:29:18.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.023+0000 7f235cf3e640 1 -- 172.21.15.89:0/2653520766 shutdown_connections 2023-10-30T12:29:18.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.023+0000 7f235cf3e640 1 --2- 172.21.15.89:0/2653520766 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23380664d0 0x7f2338068990 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:18.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.023+0000 7f235cf3e640 1 --2- 172.21.15.89:0/2653520766 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23581e5490 0x7f2358155a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:18.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.023+0000 7f235cf3e640 1 --2- 172.21.15.89:0/2653520766 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23581e5010 0x7f23581554f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:18.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.023+0000 7f235cf3e640 1 --2- 172.21.15.89:0/2653520766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2344003b40 0x7f235815c460 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:18.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.023+0000 7f235cf3e640 1 -- 172.21.15.89:0/2653520766 >> 172.21.15.89:0/2653520766 conn(0x7f2358071790 msgr2=0x7f2358157e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:18.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.023+0000 7f235cf3e640 1 -- 172.21.15.89:0/2653520766 shutdown_connections 2023-10-30T12:29:18.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.023+0000 7f235cf3e640 1 -- 172.21.15.89:0/2653520766 wait complete. 2023-10-30T12:29:18.044 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2023-10-30T12:29:18.044 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:29:18.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.179+0000 7f2fced9d640 1 Processor -- start 2023-10-30T12:29:18.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.179+0000 7f2fced9d640 1 -- start start 2023-10-30T12:29:18.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.179+0000 7f2fced9d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fc80c6bd0 0x7f2fc806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:18.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.179+0000 7f2fced9d640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc8066cf0 con 0x7f2fc806f7a0 2023-10-30T12:29:18.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.179+0000 7f2fced9d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc80c70d0 con 0x7f2fc8071110 2023-10-30T12:29:18.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.179+0000 7f2fced9d640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc80700c0 con 0x7f2fc80c6bd0 2023-10-30T12:29:18.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.179+0000 7f2fccb39640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2fc806f7a0 0x7f2fc8069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58384/0 (socket says 172.21.15.89:58384) 2023-10-30T12:29:18.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.179+0000 7f2fccb39640 1 -- 172.21.15.89:0/2336902686 learned_addr learned my addr 172.21.15.89:0/2336902686 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:29:18.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.179+0000 7f2fbffff640 1 --2- 172.21.15.89:0/2336902686 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fc80c6bd0 0x7f2fc806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:18.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.183+0000 7f2fbf7fe640 1 -- 172.21.15.89:0/2336902686 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1888003160 0 0) 0x7f2fc8066cf0 con 0x7f2fc806f7a0 2023-10-30T12:29:18.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.183+0000 7f2fbf7fe640 1 -- 172.21.15.89:0/2336902686 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb0003530 con 0x7f2fc806f7a0 2023-10-30T12:29:18.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.183+0000 7f2fbf7fe640 1 -- 172.21.15.89:0/2336902686 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1532378862 0 0) 0x7f2fb0003530 con 0x7f2fc806f7a0 2023-10-30T12:29:18.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.183+0000 7f2fbf7fe640 1 -- 172.21.15.89:0/2336902686 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fc80c6bd0 msgr2=0x7f2fc806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:18.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.183+0000 7f2fbf7fe640 1 --2- 172.21.15.89:0/2336902686 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fc80c6bd0 0x7f2fc806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:18.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.183+0000 7f2fbf7fe640 1 -- 172.21.15.89:0/2336902686 >> v1:172.21.15.192:6789/0 conn(0x7f2fc8071110 legacy=0x7f2fc8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:18.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.183+0000 7f2fbf7fe640 1 -- 172.21.15.89:0/2336902686 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc80c7390 con 0x7f2fc806f7a0 2023-10-30T12:29:18.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.183+0000 7f2fbf7fe640 1 -- 172.21.15.89:0/2336902686 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f2fc4001680 con 0x7f2fc806f7a0 2023-10-30T12:29:18.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.183+0000 7f2fbf7fe640 1 -- 172.21.15.89:0/2336902686 >> v1:172.21.15.89:6789/0 conn(0x7f2fc806f7a0 legacy=0x7f2fc8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:18.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.183+0000 7f2fbf7fe640 1 --2- 172.21.15.89:0/2336902686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2fb00039b0 0x7f2fb0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:18.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.183+0000 7f2fbf7fe640 1 --2- 172.21.15.89:0/2336902686 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fb00047d0 0x7f2fb0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:18.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.183+0000 7f2fbf7fe640 1 --2- 172.21.15.89:0/2336902686 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fc80c6bd0 0x7f2fb0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:18.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.183+0000 7f2fbf7fe640 1 -- 172.21.15.89:0/2336902686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc80700c0 con 0x7f2fb00039b0 2023-10-30T12:29:18.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.183+0000 7f2fbf7fe640 1 -- 172.21.15.89:0/2336902686 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb000bad0 con 0x7f2fb00047d0 2023-10-30T12:29:18.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.183+0000 7f2fbf7fe640 1 -- 172.21.15.89:0/2336902686 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb000bc90 con 0x7f2fc80c6bd0 2023-10-30T12:29:18.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.183+0000 7f2fccb39640 1 --2- 172.21.15.89:0/2336902686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2fb00039b0 0x7f2fb0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:18.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.183+0000 7f2fbffff640 1 --2- 172.21.15.89:0/2336902686 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fb00047d0 0x7f2fb0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:18.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.183+0000 7f2fcd33a640 1 --2- 172.21.15.89:0/2336902686 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fc80c6bd0 0x7f2fb0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:18.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.183+0000 7f2fccb39640 1 -- 172.21.15.89:0/2336902686 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fc80c6bd0 msgr2=0x7f2fb0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:18.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.183+0000 7f2fccb39640 1 --2- 172.21.15.89:0/2336902686 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fc80c6bd0 0x7f2fb0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:18.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.183+0000 7f2fccb39640 1 -- 172.21.15.89:0/2336902686 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fb00047d0 msgr2=0x7f2fb0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:18.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.183+0000 7f2fccb39640 1 --2- 172.21.15.89:0/2336902686 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fb00047d0 0x7f2fb0004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:18.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.183+0000 7f2fccb39640 1 -- 172.21.15.89:0/2336902686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb0003530 con 0x7f2fb00039b0 2023-10-30T12:29:18.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.183+0000 7f2fbffff640 1 --2- 172.21.15.89:0/2336902686 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fb00047d0 0x7f2fb0004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:18.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.183+0000 7f2fccb39640 1 --2- 172.21.15.89:0/2336902686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2fb00039b0 0x7f2fb0003dd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2fc400c210 tx=0x7f2fc401f350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e27ec19855b9c0b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:18.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.187+0000 7f2fbf7fe640 1 -- 172.21.15.89:0/2336902686 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc4005ae0 con 0x7f2fb00039b0 2023-10-30T12:29:18.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.187+0000 7f2fbf7fe640 1 -- 172.21.15.89:0/2336902686 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc40ab070 con 0x7f2fb00039b0 2023-10-30T12:29:18.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.187+0000 7f2fbf7fe640 1 -- 172.21.15.89:0/2336902686 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc4007070 con 0x7f2fb00039b0 2023-10-30T12:29:18.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.187+0000 7f2fced9d640 1 -- 172.21.15.89:0/2336902686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2fb00039b0 msgr2=0x7f2fb0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:18.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.187+0000 7f2fced9d640 1 --2- 172.21.15.89:0/2336902686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2fb00039b0 0x7f2fb0003dd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2fc400c210 tx=0x7f2fc401f350 comp rx=0 tx=0).stop 2023-10-30T12:29:18.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.187+0000 7f2fcd33a640 1 -- 172.21.15.89:0/2336902686 reap_dead start 2023-10-30T12:29:18.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.187+0000 7f2fced9d640 1 -- 172.21.15.89:0/2336902686 shutdown_connections 2023-10-30T12:29:18.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.187+0000 7f2fced9d640 1 -- 172.21.15.89:0/2336902686 >> 172.21.15.89:0/2336902686 conn(0x7f2fc80c7910 msgr2=0x7f2fc80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:18.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.187+0000 7f2fced9d640 1 -- 172.21.15.89:0/2336902686 shutdown_connections 2023-10-30T12:29:18.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.187+0000 7f2fced9d640 1 -- 172.21.15.89:0/2336902686 wait complete. 2023-10-30T12:29:18.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.187+0000 7f2fced9d640 1 Processor -- start 2023-10-30T12:29:18.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.187+0000 7f2fced9d640 1 -- start start 2023-10-30T12:29:18.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.187+0000 7f2fced9d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fb00039b0 0x7f2fc8157bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:18.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.187+0000 7f2fced9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2fc80aff50 0x7f2fc80c37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:18.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.187+0000 7f2fced9d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fc80c0f40 0x7f2fc80c13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:18.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.187+0000 7f2fced9d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc80afc90 con 0x7f2fc80aff50 2023-10-30T12:29:18.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.187+0000 7f2fced9d640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc81f1b80 con 0x7f2fc80c0f40 2023-10-30T12:29:18.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.187+0000 7f2fced9d640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc81583f0 con 0x7f2fb00039b0 2023-10-30T12:29:18.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.187+0000 7f2fbffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2fc80aff50 0x7f2fc80c37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:18.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.187+0000 7f2fbffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2fc80aff50 0x7f2fc80c37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57940/0 (socket says 172.21.15.89:57940) 2023-10-30T12:29:18.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.187+0000 7f2fbffff640 1 -- 172.21.15.89:0/3659808875 learned_addr learned my addr 172.21.15.89:0/3659808875 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:18.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.187+0000 7f2fcd33a640 1 --2- 172.21.15.89:0/3659808875 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fc80c0f40 0x7f2fc80c13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:18.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.191+0000 7f2fccb39640 1 --2- 172.21.15.89:0/3659808875 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fb00039b0 0x7f2fc8157bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:18.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.191+0000 7f2fbffff640 1 -- 172.21.15.89:0/3659808875 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fb00039b0 msgr2=0x7f2fc8157bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:18.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.191+0000 7f2fbffff640 1 --2- 172.21.15.89:0/3659808875 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fb00039b0 0x7f2fc8157bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:18.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.191+0000 7f2fbffff640 1 -- 172.21.15.89:0/3659808875 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fc80c0f40 msgr2=0x7f2fc80c13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:18.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.191+0000 7f2fbffff640 1 --2- 172.21.15.89:0/3659808875 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fc80c0f40 0x7f2fc80c13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:18.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.191+0000 7f2fbffff640 1 -- 172.21.15.89:0/3659808875 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc81f1dd0 con 0x7f2fc80aff50 2023-10-30T12:29:18.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.191+0000 7f2fccb39640 1 --2- 172.21.15.89:0/3659808875 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fb00039b0 0x7f2fc8157bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:18.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.191+0000 7f2fbffff640 1 --2- 172.21.15.89:0/3659808875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2fc80aff50 0x7f2fc80c37a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8009fd0 tx=0x7f2fb8011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:18.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.191+0000 7f2fbd7fa640 1 -- 172.21.15.89:0/3659808875 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb8004220 con 0x7f2fc80aff50 2023-10-30T12:29:18.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.191+0000 7f2fbd7fa640 1 -- 172.21.15.89:0/3659808875 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb8004380 con 0x7f2fc80aff50 2023-10-30T12:29:18.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.191+0000 7f2fbd7fa640 1 -- 172.21.15.89:0/3659808875 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb800f9e0 con 0x7f2fc80aff50 2023-10-30T12:29:18.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.191+0000 7f2fced9d640 1 -- 172.21.15.89:0/3659808875 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc81f20b0 con 0x7f2fc80aff50 2023-10-30T12:29:18.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.191+0000 7f2fced9d640 1 -- 172.21.15.89:0/3659808875 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc81f2670 con 0x7f2fc80aff50 2023-10-30T12:29:18.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.191+0000 7f2fced9d640 1 -- 172.21.15.89:0/3659808875 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc81f2830 con 0x7f2fc80aff50 2023-10-30T12:29:18.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.191+0000 7f2fbd7fa640 1 -- 172.21.15.89:0/3659808875 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2fb800fb40 con 0x7f2fc80aff50 2023-10-30T12:29:18.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.191+0000 7f2fbd7fa640 1 --2- 172.21.15.89:0/3659808875 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2fb40664d0 0x7f2fb4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:18.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.191+0000 7f2fbd7fa640 1 -- 172.21.15.89:0/3659808875 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2fb80842b0 con 0x7f2fc80aff50 2023-10-30T12:29:18.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.195+0000 7f2fccb39640 1 --2- 172.21.15.89:0/3659808875 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2fb40664d0 0x7f2fb4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:18.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.195+0000 7f2fccb39640 1 --2- 172.21.15.89:0/3659808875 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2fb40664d0 0x7f2fb4068990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2fc40af000 tx=0x7f2fc40aa000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:18.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.195+0000 7f2fbd7fa640 1 -- 172.21.15.89:0/3659808875 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2fb80517e0 con 0x7f2fc80aff50 2023-10-30T12:29:18.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.383+0000 7f2fced9d640 1 -- 172.21.15.89:0/3659808875 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fc804e170 con 0x7f2fc80aff50 2023-10-30T12:29:18.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.383+0000 7f2fbd7fa640 1 -- 172.21.15.89:0/3659808875 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6528 (secure 0 0 0) 0x7f2fb8054e00 con 0x7f2fc80aff50 2023-10-30T12:29:18.385 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:29:18.386 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:29:06.969397+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4253":{"gid":4253,"name":"b","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6835/1289220491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1289220491},{"type":"v1","addr":"172.21.15.192:6835","nonce":1289220491}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:29:18.386 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 12 2023-10-30T12:29:18.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.383+0000 7f2fced9d640 1 -- 172.21.15.89:0/3659808875 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2fb40664d0 msgr2=0x7f2fb4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:18.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.383+0000 7f2fced9d640 1 --2- 172.21.15.89:0/3659808875 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2fb40664d0 0x7f2fb4068990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2fc40af000 tx=0x7f2fc40aa000 comp rx=0 tx=0).stop 2023-10-30T12:29:18.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.383+0000 7f2fced9d640 1 -- 172.21.15.89:0/3659808875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2fc80aff50 msgr2=0x7f2fc80c37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:18.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.383+0000 7f2fced9d640 1 --2- 172.21.15.89:0/3659808875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2fc80aff50 0x7f2fc80c37a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8009fd0 tx=0x7f2fb8011b30 comp rx=0 tx=0).stop 2023-10-30T12:29:18.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.383+0000 7f2fced9d640 1 -- 172.21.15.89:0/3659808875 shutdown_connections 2023-10-30T12:29:18.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.383+0000 7f2fced9d640 1 --2- 172.21.15.89:0/3659808875 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2fb40664d0 0x7f2fb4068990 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:18.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.383+0000 7f2fced9d640 1 --2- 172.21.15.89:0/3659808875 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fc80c0f40 0x7f2fc80c13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:18.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.383+0000 7f2fced9d640 1 --2- 172.21.15.89:0/3659808875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2fc80aff50 0x7f2fc80c37a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:18.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.383+0000 7f2fced9d640 1 --2- 172.21.15.89:0/3659808875 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2fb00039b0 0x7f2fc8157bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:18.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.383+0000 7f2fced9d640 1 -- 172.21.15.89:0/3659808875 >> 172.21.15.89:0/3659808875 conn(0x7f2fc80c7910 msgr2=0x7f2fc80c2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:18.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.383+0000 7f2fced9d640 1 -- 172.21.15.89:0/3659808875 shutdown_connections 2023-10-30T12:29:18.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.383+0000 7f2fced9d640 1 -- 172.21.15.89:0/3659808875 wait complete. 2023-10-30T12:29:18.406 INFO:tasks.mds_thrash:Ready to start thrashing 2023-10-30T12:29:18.406 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2023-10-30T12:29:18.407 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:29:18.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.543+0000 7fe6847b8640 1 Processor -- start 2023-10-30T12:29:18.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.543+0000 7fe6847b8640 1 -- start start 2023-10-30T12:29:18.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.543+0000 7fe6847b8640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe67c0c5f60 0x7fe67c071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:18.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.543+0000 7fe6847b8640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe67c0700b0 con 0x7fe67c0c6fe0 2023-10-30T12:29:18.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.543+0000 7fe6847b8640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe67c0c6530 con 0x7fe67c153bf0 2023-10-30T12:29:18.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.543+0000 7fe6847b8640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe67c0c6810 con 0x7fe67c0c5f60 2023-10-30T12:29:18.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.543+0000 7fe681d53640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe67c0c6fe0 0x7fe67c0cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58394/0 (socket says 172.21.15.89:58394) 2023-10-30T12:29:18.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.543+0000 7fe681d53640 1 -- 172.21.15.89:0/5525111 learned_addr learned my addr 172.21.15.89:0/5525111 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:29:18.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.543+0000 7fe682d55640 1 --2- 172.21.15.89:0/5525111 >> v2:172.21.15.89:3300/0 conn(0x7fe67c0c5f60 0x7fe67c071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:18.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.543+0000 7fe682d55640 1 -- 172.21.15.89:0/5525111 >> v1:172.21.15.192:6789/0 conn(0x7fe67c153bf0 legacy=0x7fe67c0c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:18.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.543+0000 7fe682d55640 1 -- 172.21.15.89:0/5525111 >> v1:172.21.15.89:6789/0 conn(0x7fe67c0c6fe0 legacy=0x7fe67c0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:18.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.543+0000 7fe682d55640 1 -- 172.21.15.89:0/5525111 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67c159b10 con 0x7fe67c0c5f60 2023-10-30T12:29:18.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.543+0000 7fe681552640 1 -- 172.21.15.89:0/5525111 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3341003823 0 0) 0x7fe67c0700b0 con 0x7fe67c0c6fe0 2023-10-30T12:29:18.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.547+0000 7fe682d55640 1 --2- 172.21.15.89:0/5525111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe67c0c5f60 0x7fe67c071260 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe674009d90 tx=0x7fe67401f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e9c48007205c4c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:18.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.547+0000 7fe681552640 1 -- 172.21.15.89:0/5525111 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6740af600 con 0x7fe67c0c5f60 2023-10-30T12:29:18.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.547+0000 7fe681552640 1 -- 172.21.15.89:0/5525111 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6740ab050 con 0x7fe67c0c5f60 2023-10-30T12:29:18.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.547+0000 7fe681552640 1 -- 172.21.15.89:0/5525111 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6740af980 con 0x7fe67c0c5f60 2023-10-30T12:29:18.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.547+0000 7fe6847b8640 1 -- 172.21.15.89:0/5525111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe67c0c5f60 msgr2=0x7fe67c071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:18.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.547+0000 7fe6847b8640 1 --2- 172.21.15.89:0/5525111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe67c0c5f60 0x7fe67c071260 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe674009d90 tx=0x7fe67401f130 comp rx=0 tx=0).stop 2023-10-30T12:29:18.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.547+0000 7fe6847b8640 1 -- 172.21.15.89:0/5525111 shutdown_connections 2023-10-30T12:29:18.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.547+0000 7fe6847b8640 1 --2- 172.21.15.89:0/5525111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe67c0c5f60 0x7fe67c071260 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:18.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.547+0000 7fe6847b8640 1 -- 172.21.15.89:0/5525111 >> 172.21.15.89:0/5525111 conn(0x7fe67c06c6e0 msgr2=0x7fe67c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:18.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.547+0000 7fe6847b8640 1 -- 172.21.15.89:0/5525111 shutdown_connections 2023-10-30T12:29:18.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.547+0000 7fe6847b8640 1 -- 172.21.15.89:0/5525111 wait complete. 2023-10-30T12:29:18.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.551+0000 7fe6847b8640 1 Processor -- start 2023-10-30T12:29:18.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.551+0000 7fe6847b8640 1 -- start start 2023-10-30T12:29:18.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.551+0000 7fe6847b8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe67c0c5f60 0x7fe67c14c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:18.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.551+0000 7fe6847b8640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe67c0c6fe0 0x7fe67c0bc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:18.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.551+0000 7fe6847b8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe67c153bf0 0x7fe67c1511f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:18.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.551+0000 7fe6847b8640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c1ef970 con 0x7fe67c0c5f60 2023-10-30T12:29:18.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.551+0000 7fe6847b8640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c1543b0 con 0x7fe67c153bf0 2023-10-30T12:29:18.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.551+0000 7fe6847b8640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c154650 con 0x7fe67c0c6fe0 2023-10-30T12:29:18.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.551+0000 7fe682554640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe67c0c5f60 0x7fe67c14c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:18.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.551+0000 7fe682554640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe67c0c5f60 0x7fe67c14c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57944/0 (socket says 172.21.15.89:57944) 2023-10-30T12:29:18.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.551+0000 7fe682554640 1 -- 172.21.15.89:0/1024071444 learned_addr learned my addr 172.21.15.89:0/1024071444 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:18.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.551+0000 7fe682d55640 1 --2- 172.21.15.89:0/1024071444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe67c153bf0 0x7fe67c1511f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:18.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.551+0000 7fe681d53640 1 --2- 172.21.15.89:0/1024071444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe67c0c6fe0 0x7fe67c0bc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:18.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.551+0000 7fe682554640 1 -- 172.21.15.89:0/1024071444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe67c0c6fe0 msgr2=0x7fe67c0bc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:18.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.551+0000 7fe682554640 1 --2- 172.21.15.89:0/1024071444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe67c0c6fe0 0x7fe67c0bc460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:18.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.551+0000 7fe682554640 1 -- 172.21.15.89:0/1024071444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe67c153bf0 msgr2=0x7fe67c1511f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:18.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.551+0000 7fe682554640 1 --2- 172.21.15.89:0/1024071444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe67c153bf0 0x7fe67c1511f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:18.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.551+0000 7fe682554640 1 -- 172.21.15.89:0/1024071444 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe674009a70 con 0x7fe67c0c5f60 2023-10-30T12:29:18.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.551+0000 7fe681d53640 1 --2- 172.21.15.89:0/1024071444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe67c0c6fe0 0x7fe67c0bc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:29:18.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.551+0000 7fe682554640 1 --2- 172.21.15.89:0/1024071444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe67c0c5f60 0x7fe67c14c9f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe67800c500 tx=0x7fe67800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:18.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.551+0000 7fe672ffd640 1 -- 172.21.15.89:0/1024071444 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe678005220 con 0x7fe67c0c5f60 2023-10-30T12:29:18.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.551+0000 7fe672ffd640 1 -- 172.21.15.89:0/1024071444 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe678005380 con 0x7fe67c0c5f60 2023-10-30T12:29:18.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.551+0000 7fe672ffd640 1 -- 172.21.15.89:0/1024071444 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe678007270 con 0x7fe67c0c5f60 2023-10-30T12:29:18.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.551+0000 7fe6847b8640 1 -- 172.21.15.89:0/1024071444 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe67c1ef040 con 0x7fe67c0c5f60 2023-10-30T12:29:18.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.551+0000 7fe6847b8640 1 -- 172.21.15.89:0/1024071444 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe67c1ef600 con 0x7fe67c0c5f60 2023-10-30T12:29:18.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.551+0000 7fe6847b8640 1 -- 172.21.15.89:0/1024071444 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe67c04e170 con 0x7fe67c0c5f60 2023-10-30T12:29:18.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.555+0000 7fe672ffd640 1 -- 172.21.15.89:0/1024071444 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe6780073d0 con 0x7fe67c0c5f60 2023-10-30T12:29:18.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.555+0000 7fe672ffd640 1 --2- 172.21.15.89:0/1024071444 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe6600664a0 0x7fe660068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:18.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.555+0000 7fe672ffd640 1 -- 172.21.15.89:0/1024071444 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe678083800 con 0x7fe67c0c5f60 2023-10-30T12:29:18.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.555+0000 7fe681d53640 1 --2- 172.21.15.89:0/1024071444 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe6600664a0 0x7fe660068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:18.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.555+0000 7fe681d53640 1 --2- 172.21.15.89:0/1024071444 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe6600664a0 0x7fe660068960 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe67c0c4d90 tx=0x7fe66c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:18.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.559+0000 7fe672ffd640 1 -- 172.21.15.89:0/1024071444 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe678050bf0 con 0x7fe67c0c5f60 2023-10-30T12:29:18.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.703+0000 7fe6847b8640 1 -- 172.21.15.89:0/1024071444 --> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe67c15a0d0 con 0x7fe6600664a0 2023-10-30T12:29:18.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.723+0000 7fe672ffd640 1 -- 172.21.15.89:0/1024071444 <== mgr.4105 v2:172.21.15.192:6832/4076276638 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264569 (secure 0 0 0) 0x7fe66c001d10 con 0x7fe6600664a0 2023-10-30T12:29:18.730 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:29:18.732 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-10-30T12:29:18.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.727+0000 7fe6847b8640 1 -- 172.21.15.89:0/1024071444 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe6600664a0 msgr2=0x7fe660068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:18.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.727+0000 7fe6847b8640 1 --2- 172.21.15.89:0/1024071444 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe6600664a0 0x7fe660068960 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe67c0c4d90 tx=0x7fe66c009040 comp rx=0 tx=0).stop 2023-10-30T12:29:18.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.727+0000 7fe6847b8640 1 -- 172.21.15.89:0/1024071444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe67c0c5f60 msgr2=0x7fe67c14c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:18.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.727+0000 7fe6847b8640 1 --2- 172.21.15.89:0/1024071444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe67c0c5f60 0x7fe67c14c9f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe67800c500 tx=0x7fe67800c9c0 comp rx=0 tx=0).stop 2023-10-30T12:29:18.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.727+0000 7fe6847b8640 1 -- 172.21.15.89:0/1024071444 shutdown_connections 2023-10-30T12:29:18.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.727+0000 7fe6847b8640 1 --2- 172.21.15.89:0/1024071444 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe6600664a0 0x7fe660068960 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:18.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.727+0000 7fe6847b8640 1 --2- 172.21.15.89:0/1024071444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe67c153bf0 0x7fe67c1511f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:18.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.727+0000 7fe6847b8640 1 --2- 172.21.15.89:0/1024071444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe67c0c6fe0 0x7fe67c0bc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.727+0000 7fe6847b8640 1 --2- 172.21.15.89:0/1024071444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe67c0c5f60 0x7fe67c14c9f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.727+0000 7fe6847b8640 1 -- 172.21.15.89:0/1024071444 >> 172.21.15.89:0/1024071444 conn(0x7fe67c06c6e0 msgr2=0x7fe67c152e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.727+0000 7fe6847b8640 1 -- 172.21.15.89:0/1024071444 shutdown_connections 2023-10-30T12:29:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.727+0000 7fe6847b8640 1 -- 172.21.15.89:0/1024071444 wait complete. 2023-10-30T12:29:18.750 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2023-10-30T12:29:16.928396+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595590,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":283,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":225,"ondisk_log_size":225,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219196,"kb_used_data":3068,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755524,"statfs":{"total":773094113280,"available":772869656576,"internally_reserved":0,"allocated":3141632,"data_stored":1820235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12741,"internal_metadata":221236795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1406,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004326"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028790+0000","last_change":"2023-10-30T12:28:50.979797+0000","last_active":"2023-10-30T12:28:53.028790+0000","last_peered":"2023-10-30T12:28:53.028790+0000","last_clean":"2023-10-30T12:28:53.028790+0000","last_became_active":"2023-10-30T12:28:50.979351+0000","last_became_peered":"2023-10-30T12:28:50.979351+0000","last_unstale":"2023-10-30T12:28:53.028790+0000","last_undegraded":"2023-10-30T12:28:53.028790+0000","last_fullsized":"2023-10-30T12:28:53.028790+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:53:00.556539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.932043+0000","last_change":"2023-10-30T12:28:50.929403+0000","last_active":"2023-10-30T12:28:51.932043+0000","last_peered":"2023-10-30T12:28:51.932043+0000","last_clean":"2023-10-30T12:28:51.932043+0000","last_became_active":"2023-10-30T12:28:50.929133+0000","last_became_peered":"2023-10-30T12:28:50.929133+0000","last_unstale":"2023-10-30T12:28:51.932043+0000","last_undegraded":"2023-10-30T12:28:51.932043+0000","last_fullsized":"2023-10-30T12:28:51.932043+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:12:57.327440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579508+0000","last_change":"2023-10-30T12:28:50.934201+0000","last_active":"2023-10-30T12:28:53.579508+0000","last_peered":"2023-10-30T12:28:53.579508+0000","last_clean":"2023-10-30T12:28:53.579508+0000","last_became_active":"2023-10-30T12:28:50.933896+0000","last_became_peered":"2023-10-30T12:28:50.933896+0000","last_unstale":"2023-10-30T12:28:53.579508+0000","last_undegraded":"2023-10-30T12:28:53.579508+0000","last_fullsized":"2023-10-30T12:28:53.579508+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:20:46.597550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581831+0000","last_change":"2023-10-30T12:28:50.969193+0000","last_active":"2023-10-30T12:28:53.581831+0000","last_peered":"2023-10-30T12:28:53.581831+0000","last_clean":"2023-10-30T12:28:53.581831+0000","last_became_active":"2023-10-30T12:28:50.967774+0000","last_became_peered":"2023-10-30T12:28:50.967774+0000","last_unstale":"2023-10-30T12:28:53.581831+0000","last_undegraded":"2023-10-30T12:28:53.581831+0000","last_fullsized":"2023-10-30T12:28:53.581831+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:23:07.329372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028482+0000","last_change":"2023-10-30T12:28:50.973751+0000","last_active":"2023-10-30T12:28:53.028482+0000","last_peered":"2023-10-30T12:28:53.028482+0000","last_clean":"2023-10-30T12:28:53.028482+0000","last_became_active":"2023-10-30T12:28:50.973132+0000","last_became_peered":"2023-10-30T12:28:50.973132+0000","last_unstale":"2023-10-30T12:28:53.028482+0000","last_undegraded":"2023-10-30T12:28:53.028482+0000","last_fullsized":"2023-10-30T12:28:53.028482+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:43:58.223555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581257+0000","last_change":"2023-10-30T12:28:50.968823+0000","last_active":"2023-10-30T12:28:53.581257+0000","last_peered":"2023-10-30T12:28:53.581257+0000","last_clean":"2023-10-30T12:28:53.581257+0000","last_became_active":"2023-10-30T12:28:50.967910+0000","last_became_peered":"2023-10-30T12:28:50.967910+0000","last_unstale":"2023-10-30T12:28:53.581257+0000","last_undegraded":"2023-10-30T12:28:53.581257+0000","last_fullsized":"2023-10-30T12:28:53.581257+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:34:36.882757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028044+0000","last_change":"2023-10-30T12:28:50.974164+0000","last_active":"2023-10-30T12:28:53.028044+0000","last_peered":"2023-10-30T12:28:53.028044+0000","last_clean":"2023-10-30T12:28:53.028044+0000","last_became_active":"2023-10-30T12:28:50.973288+0000","last_became_peered":"2023-10-30T12:28:50.973288+0000","last_unstale":"2023-10-30T12:28:53.028044+0000","last_undegraded":"2023-10-30T12:28:53.028044+0000","last_fullsized":"2023-10-30T12:28:53.028044+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:09:10.450492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.578880+0000","last_change":"2023-10-30T12:28:50.955287+0000","last_active":"2023-10-30T12:28:53.578880+0000","last_peered":"2023-10-30T12:28:53.578880+0000","last_clean":"2023-10-30T12:28:53.578880+0000","last_became_active":"2023-10-30T12:28:50.954988+0000","last_became_peered":"2023-10-30T12:28:50.954988+0000","last_unstale":"2023-10-30T12:28:53.578880+0000","last_undegraded":"2023-10-30T12:28:53.578880+0000","last_fullsized":"2023-10-30T12:28:53.578880+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:36:30.846920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.029019+0000","last_change":"2023-10-30T12:28:50.977144+0000","last_active":"2023-10-30T12:28:53.029019+0000","last_peered":"2023-10-30T12:28:53.029019+0000","last_clean":"2023-10-30T12:28:53.029019+0000","last_became_active":"2023-10-30T12:28:50.976564+0000","last_became_peered":"2023-10-30T12:28:50.976564+0000","last_unstale":"2023-10-30T12:28:53.029019+0000","last_undegraded":"2023-10-30T12:28:53.029019+0000","last_fullsized":"2023-10-30T12:28:53.029019+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:37:51.341935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.938542+0000","last_change":"2023-10-30T12:28:50.952564+0000","last_active":"2023-10-30T12:28:51.938542+0000","last_peered":"2023-10-30T12:28:51.938542+0000","last_clean":"2023-10-30T12:28:51.938542+0000","last_became_active":"2023-10-30T12:28:50.951809+0000","last_became_peered":"2023-10-30T12:28:50.951809+0000","last_unstale":"2023-10-30T12:28:51.938542+0000","last_undegraded":"2023-10-30T12:28:51.938542+0000","last_fullsized":"2023-10-30T12:28:51.938542+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:34:29.771577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.937215+0000","last_change":"2023-10-30T12:28:50.970779+0000","last_active":"2023-10-30T12:28:51.937215+0000","last_peered":"2023-10-30T12:28:51.937215+0000","last_clean":"2023-10-30T12:28:51.937215+0000","last_became_active":"2023-10-30T12:28:50.969877+0000","last_became_peered":"2023-10-30T12:28:50.969877+0000","last_unstale":"2023-10-30T12:28:51.937215+0000","last_undegraded":"2023-10-30T12:28:51.937215+0000","last_fullsized":"2023-10-30T12:28:51.937215+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:10.539915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.940164+0000","last_change":"2023-10-30T12:28:50.944452+0000","last_active":"2023-10-30T12:28:51.940164+0000","last_peered":"2023-10-30T12:28:51.940164+0000","last_clean":"2023-10-30T12:28:51.940164+0000","last_became_active":"2023-10-30T12:28:50.943787+0000","last_became_peered":"2023-10-30T12:28:50.943787+0000","last_unstale":"2023-10-30T12:28:51.940164+0000","last_undegraded":"2023-10-30T12:28:51.940164+0000","last_fullsized":"2023-10-30T12:28:51.940164+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:30:34.585298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579240+0000","last_change":"2023-10-30T12:28:50.938181+0000","last_active":"2023-10-30T12:28:53.579240+0000","last_peered":"2023-10-30T12:28:53.579240+0000","last_clean":"2023-10-30T12:28:53.579240+0000","last_became_active":"2023-10-30T12:28:50.937906+0000","last_became_peered":"2023-10-30T12:28:50.937906+0000","last_unstale":"2023-10-30T12:28:53.579240+0000","last_undegraded":"2023-10-30T12:28:53.579240+0000","last_fullsized":"2023-10-30T12:28:53.579240+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:41:19.990452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579303+0000","last_change":"2023-10-30T12:28:50.934085+0000","last_active":"2023-10-30T12:28:53.579303+0000","last_peered":"2023-10-30T12:28:53.579303+0000","last_clean":"2023-10-30T12:28:53.579303+0000","last_became_active":"2023-10-30T12:28:50.932753+0000","last_became_peered":"2023-10-30T12:28:50.932753+0000","last_unstale":"2023-10-30T12:28:53.579303+0000","last_undegraded":"2023-10-30T12:28:53.579303+0000","last_fullsized":"2023-10-30T12:28:53.579303+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:23:26.736283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931710+0000","last_change":"2023-10-30T12:28:50.943343+0000","last_active":"2023-10-30T12:28:51.931710+0000","last_peered":"2023-10-30T12:28:51.931710+0000","last_clean":"2023-10-30T12:28:51.931710+0000","last_became_active":"2023-10-30T12:28:50.943011+0000","last_became_peered":"2023-10-30T12:28:50.943011+0000","last_unstale":"2023-10-30T12:28:51.931710+0000","last_undegraded":"2023-10-30T12:28:51.931710+0000","last_fullsized":"2023-10-30T12:28:51.931710+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:59:35.881115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581157+0000","last_change":"2023-10-30T12:28:50.953573+0000","last_active":"2023-10-30T12:28:53.581157+0000","last_peered":"2023-10-30T12:28:53.581157+0000","last_clean":"2023-10-30T12:28:53.581157+0000","last_became_active":"2023-10-30T12:28:50.952744+0000","last_became_peered":"2023-10-30T12:28:50.952744+0000","last_unstale":"2023-10-30T12:28:53.581157+0000","last_undegraded":"2023-10-30T12:28:53.581157+0000","last_fullsized":"2023-10-30T12:28:53.581157+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:41:16.209759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.582342+0000","last_change":"2023-10-30T12:28:50.975156+0000","last_active":"2023-10-30T12:28:53.582342+0000","last_peered":"2023-10-30T12:28:53.582342+0000","last_clean":"2023-10-30T12:28:53.582342+0000","last_became_active":"2023-10-30T12:28:50.974576+0000","last_became_peered":"2023-10-30T12:28:50.974576+0000","last_unstale":"2023-10-30T12:28:53.582342+0000","last_undegraded":"2023-10-30T12:28:53.582342+0000","last_fullsized":"2023-10-30T12:28:53.582342+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:46.159555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.131817+0000","last_change":"2023-10-30T12:28:50.937685+0000","last_active":"2023-10-30T12:28:53.131817+0000","last_peered":"2023-10-30T12:28:53.131817+0000","last_clean":"2023-10-30T12:28:53.131817+0000","last_became_active":"2023-10-30T12:28:50.937384+0000","last_became_peered":"2023-10-30T12:28:50.937384+0000","last_unstale":"2023-10-30T12:28:53.131817+0000","last_undegraded":"2023-10-30T12:28:53.131817+0000","last_fullsized":"2023-10-30T12:28:53.131817+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:16:29.181523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.029181+0000","last_change":"2023-10-30T12:28:50.974918+0000","last_active":"2023-10-30T12:28:53.029181+0000","last_peered":"2023-10-30T12:28:53.029181+0000","last_clean":"2023-10-30T12:28:53.029181+0000","last_became_active":"2023-10-30T12:28:50.973369+0000","last_became_peered":"2023-10-30T12:28:50.973369+0000","last_unstale":"2023-10-30T12:28:53.029181+0000","last_undegraded":"2023-10-30T12:28:53.029181+0000","last_fullsized":"2023-10-30T12:28:53.029181+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:44:04.468611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.937226+0000","last_change":"2023-10-30T12:28:50.955835+0000","last_active":"2023-10-30T12:28:51.937226+0000","last_peered":"2023-10-30T12:28:51.937226+0000","last_clean":"2023-10-30T12:28:51.937226+0000","last_became_active":"2023-10-30T12:28:50.954831+0000","last_became_peered":"2023-10-30T12:28:50.954831+0000","last_unstale":"2023-10-30T12:28:51.937226+0000","last_undegraded":"2023-10-30T12:28:51.937226+0000","last_fullsized":"2023-10-30T12:28:51.937226+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:56:25.554582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.580290+0000","last_change":"2023-10-30T12:28:50.962417+0000","last_active":"2023-10-30T12:28:53.580290+0000","last_peered":"2023-10-30T12:28:53.580290+0000","last_clean":"2023-10-30T12:28:53.580290+0000","last_became_active":"2023-10-30T12:28:50.962065+0000","last_became_peered":"2023-10-30T12:28:50.962065+0000","last_unstale":"2023-10-30T12:28:53.580290+0000","last_undegraded":"2023-10-30T12:28:53.580290+0000","last_fullsized":"2023-10-30T12:28:53.580290+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:13:52.455824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581454+0000","last_change":"2023-10-30T12:28:50.968992+0000","last_active":"2023-10-30T12:28:53.581454+0000","last_peered":"2023-10-30T12:28:53.581454+0000","last_clean":"2023-10-30T12:28:53.581454+0000","last_became_active":"2023-10-30T12:28:50.967651+0000","last_became_peered":"2023-10-30T12:28:50.967651+0000","last_unstale":"2023-10-30T12:28:53.581454+0000","last_undegraded":"2023-10-30T12:28:53.581454+0000","last_fullsized":"2023-10-30T12:28:53.581454+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:39:31.893024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579014+0000","last_change":"2023-10-30T12:28:50.962609+0000","last_active":"2023-10-30T12:28:53.579014+0000","last_peered":"2023-10-30T12:28:53.579014+0000","last_clean":"2023-10-30T12:28:53.579014+0000","last_became_active":"2023-10-30T12:28:50.962290+0000","last_became_peered":"2023-10-30T12:28:50.962290+0000","last_unstale":"2023-10-30T12:28:53.579014+0000","last_undegraded":"2023-10-30T12:28:53.579014+0000","last_fullsized":"2023-10-30T12:28:53.579014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:44:35.484971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581381+0000","last_change":"2023-10-30T12:28:50.966653+0000","last_active":"2023-10-30T12:28:53.581381+0000","last_peered":"2023-10-30T12:28:53.581381+0000","last_clean":"2023-10-30T12:28:53.581381+0000","last_became_active":"2023-10-30T12:28:50.965713+0000","last_became_peered":"2023-10-30T12:28:50.965713+0000","last_unstale":"2023-10-30T12:28:53.581381+0000","last_undegraded":"2023-10-30T12:28:53.581381+0000","last_fullsized":"2023-10-30T12:28:53.581381+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:10:31.600981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579351+0000","last_change":"2023-10-30T12:28:50.951300+0000","last_active":"2023-10-30T12:28:53.579351+0000","last_peered":"2023-10-30T12:28:53.579351+0000","last_clean":"2023-10-30T12:28:53.579351+0000","last_became_active":"2023-10-30T12:28:50.951026+0000","last_became_peered":"2023-10-30T12:28:50.951026+0000","last_unstale":"2023-10-30T12:28:53.579351+0000","last_undegraded":"2023-10-30T12:28:53.579351+0000","last_fullsized":"2023-10-30T12:28:53.579351+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:44:58.872457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.938879+0000","last_change":"2023-10-30T12:28:50.963507+0000","last_active":"2023-10-30T12:28:51.938879+0000","last_peered":"2023-10-30T12:28:51.938879+0000","last_clean":"2023-10-30T12:28:51.938879+0000","last_became_active":"2023-10-30T12:28:50.962484+0000","last_became_peered":"2023-10-30T12:28:50.962484+0000","last_unstale":"2023-10-30T12:28:51.938879+0000","last_undegraded":"2023-10-30T12:28:51.938879+0000","last_fullsized":"2023-10-30T12:28:51.938879+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:11:24.160220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931933+0000","last_change":"2023-10-30T12:28:50.933362+0000","last_active":"2023-10-30T12:28:51.931933+0000","last_peered":"2023-10-30T12:28:51.931933+0000","last_clean":"2023-10-30T12:28:51.931933+0000","last_became_active":"2023-10-30T12:28:50.933093+0000","last_became_peered":"2023-10-30T12:28:50.933093+0000","last_unstale":"2023-10-30T12:28:51.931933+0000","last_undegraded":"2023-10-30T12:28:51.931933+0000","last_fullsized":"2023-10-30T12:28:51.931933+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:54:00.852482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579508+0000","last_change":"2023-10-30T12:28:50.952857+0000","last_active":"2023-10-30T12:28:53.579508+0000","last_peered":"2023-10-30T12:28:53.579508+0000","last_clean":"2023-10-30T12:28:53.579508+0000","last_became_active":"2023-10-30T12:28:50.952531+0000","last_became_peered":"2023-10-30T12:28:50.952531+0000","last_unstale":"2023-10-30T12:28:53.579508+0000","last_undegraded":"2023-10-30T12:28:53.579508+0000","last_fullsized":"2023-10-30T12:28:53.579508+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:01:03.980850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.580366+0000","last_change":"2023-10-30T12:28:50.933408+0000","last_active":"2023-10-30T12:28:53.580366+0000","last_peered":"2023-10-30T12:28:53.580366+0000","last_clean":"2023-10-30T12:28:53.580366+0000","last_became_active":"2023-10-30T12:28:50.933124+0000","last_became_peered":"2023-10-30T12:28:50.933124+0000","last_unstale":"2023-10-30T12:28:53.580366+0000","last_undegraded":"2023-10-30T12:28:53.580366+0000","last_fullsized":"2023-10-30T12:28:53.580366+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:53:50.772420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581630+0000","last_change":"2023-10-30T12:28:50.966388+0000","last_active":"2023-10-30T12:28:53.581630+0000","last_peered":"2023-10-30T12:28:53.581630+0000","last_clean":"2023-10-30T12:28:53.581630+0000","last_became_active":"2023-10-30T12:28:50.965639+0000","last_became_peered":"2023-10-30T12:28:50.965639+0000","last_unstale":"2023-10-30T12:28:53.581630+0000","last_undegraded":"2023-10-30T12:28:53.581630+0000","last_fullsized":"2023-10-30T12:28:53.581630+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:55:46.506378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.131583+0000","last_change":"2023-10-30T12:28:50.955391+0000","last_active":"2023-10-30T12:28:53.131583+0000","last_peered":"2023-10-30T12:28:53.131583+0000","last_clean":"2023-10-30T12:28:53.131583+0000","last_became_active":"2023-10-30T12:28:50.955084+0000","last_became_peered":"2023-10-30T12:28:50.955084+0000","last_unstale":"2023-10-30T12:28:53.131583+0000","last_undegraded":"2023-10-30T12:28:53.131583+0000","last_fullsized":"2023-10-30T12:28:53.131583+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:08:52.389085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.936598+0000","last_change":"2023-10-30T12:28:50.945609+0000","last_active":"2023-10-30T12:28:51.936598+0000","last_peered":"2023-10-30T12:28:51.936598+0000","last_clean":"2023-10-30T12:28:51.936598+0000","last_became_active":"2023-10-30T12:28:50.944775+0000","last_became_peered":"2023-10-30T12:28:50.944775+0000","last_unstale":"2023-10-30T12:28:51.936598+0000","last_undegraded":"2023-10-30T12:28:51.936598+0000","last_fullsized":"2023-10-30T12:28:51.936598+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:08:55.718193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.939146+0000","last_change":"2023-10-30T12:28:50.941378+0000","last_active":"2023-10-30T12:28:51.939146+0000","last_peered":"2023-10-30T12:28:51.939146+0000","last_clean":"2023-10-30T12:28:51.939146+0000","last_became_active":"2023-10-30T12:28:50.940567+0000","last_became_peered":"2023-10-30T12:28:50.940567+0000","last_unstale":"2023-10-30T12:28:51.939146+0000","last_undegraded":"2023-10-30T12:28:51.939146+0000","last_fullsized":"2023-10-30T12:28:51.939146+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:31:23.772343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'5","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.435783+0000","last_change":"2023-10-30T12:28:50.395670+0000","last_active":"2023-10-30T12:29:01.435783+0000","last_peered":"2023-10-30T12:29:01.435783+0000","last_clean":"2023-10-30T12:29:01.435783+0000","last_became_active":"2023-10-30T12:28:50.395382+0000","last_became_peered":"2023-10-30T12:28:50.395382+0000","last_unstale":"2023-10-30T12:29:01.435783+0000","last_undegraded":"2023-10-30T12:29:01.435783+0000","last_fullsized":"2023-10-30T12:29:01.435783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:14.858121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.580427+0000","last_change":"2023-10-30T12:28:50.953202+0000","last_active":"2023-10-30T12:28:53.580427+0000","last_peered":"2023-10-30T12:28:53.580427+0000","last_clean":"2023-10-30T12:28:53.580427+0000","last_became_active":"2023-10-30T12:28:50.952613+0000","last_became_peered":"2023-10-30T12:28:50.952613+0000","last_unstale":"2023-10-30T12:28:53.580427+0000","last_undegraded":"2023-10-30T12:28:53.580427+0000","last_fullsized":"2023-10-30T12:28:53.580427+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:38:56.947997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.959082+0000","last_change":"2023-10-30T12:28:50.384598+0000","last_active":"2023-10-30T12:29:06.959082+0000","last_peered":"2023-10-30T12:29:06.959082+0000","last_clean":"2023-10-30T12:29:06.959082+0000","last_became_active":"2023-10-30T12:28:50.383898+0000","last_became_peered":"2023-10-30T12:28:50.383898+0000","last_unstale":"2023-10-30T12:29:06.959082+0000","last_undegraded":"2023-10-30T12:29:06.959082+0000","last_fullsized":"2023-10-30T12:29:06.959082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:09:50.290004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.954822+0000","last_change":"2023-10-30T12:28:50.386193+0000","last_active":"2023-10-30T12:29:06.954822+0000","last_peered":"2023-10-30T12:29:06.954822+0000","last_clean":"2023-10-30T12:29:06.954822+0000","last_became_active":"2023-10-30T12:28:50.385963+0000","last_became_peered":"2023-10-30T12:28:50.385963+0000","last_unstale":"2023-10-30T12:29:06.954822+0000","last_undegraded":"2023-10-30T12:29:06.954822+0000","last_fullsized":"2023-10-30T12:29:06.954822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:00:55.603212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.578966+0000","last_change":"2023-10-30T12:28:50.936038+0000","last_active":"2023-10-30T12:28:53.578966+0000","last_peered":"2023-10-30T12:28:53.578966+0000","last_clean":"2023-10-30T12:28:53.578966+0000","last_became_active":"2023-10-30T12:28:50.935764+0000","last_became_peered":"2023-10-30T12:28:50.935764+0000","last_unstale":"2023-10-30T12:28:53.578966+0000","last_undegraded":"2023-10-30T12:28:53.578966+0000","last_fullsized":"2023-10-30T12:28:53.578966+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:23:01.597158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581945+0000","last_change":"2023-10-30T12:28:50.392213+0000","last_active":"2023-10-30T12:28:53.581945+0000","last_peered":"2023-10-30T12:28:53.581945+0000","last_clean":"2023-10-30T12:28:53.581945+0000","last_became_active":"2023-10-30T12:28:50.391494+0000","last_became_peered":"2023-10-30T12:28:50.391494+0000","last_unstale":"2023-10-30T12:28:53.581945+0000","last_undegraded":"2023-10-30T12:28:53.581945+0000","last_fullsized":"2023-10-30T12:28:53.581945+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:38:37.301840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931619+0000","last_change":"2023-10-30T12:28:50.933319+0000","last_active":"2023-10-30T12:28:51.931619+0000","last_peered":"2023-10-30T12:28:51.931619+0000","last_clean":"2023-10-30T12:28:51.931619+0000","last_became_active":"2023-10-30T12:28:50.933062+0000","last_became_peered":"2023-10-30T12:28:50.933062+0000","last_unstale":"2023-10-30T12:28:51.931619+0000","last_undegraded":"2023-10-30T12:28:51.931619+0000","last_fullsized":"2023-10-30T12:28:51.931619+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:26:48.475038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.424753+0000","last_change":"2023-10-30T12:28:50.394368+0000","last_active":"2023-10-30T12:29:01.424753+0000","last_peered":"2023-10-30T12:29:01.424753+0000","last_clean":"2023-10-30T12:29:01.424753+0000","last_became_active":"2023-10-30T12:28:50.393686+0000","last_became_peered":"2023-10-30T12:28:50.393686+0000","last_unstale":"2023-10-30T12:29:01.424753+0000","last_undegraded":"2023-10-30T12:29:01.424753+0000","last_fullsized":"2023-10-30T12:29:01.424753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:43:17.726462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.582247+0000","last_change":"2023-10-30T12:28:50.963703+0000","last_active":"2023-10-30T12:28:53.582247+0000","last_peered":"2023-10-30T12:28:53.582247+0000","last_clean":"2023-10-30T12:28:53.582247+0000","last_became_active":"2023-10-30T12:28:50.963105+0000","last_became_peered":"2023-10-30T12:28:50.963105+0000","last_unstale":"2023-10-30T12:28:53.582247+0000","last_undegraded":"2023-10-30T12:28:53.582247+0000","last_fullsized":"2023-10-30T12:28:53.582247+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:32.423943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.420755+0000","last_change":"2023-10-30T12:28:50.397679+0000","last_active":"2023-10-30T12:29:01.420755+0000","last_peered":"2023-10-30T12:29:01.420755+0000","last_clean":"2023-10-30T12:29:01.420755+0000","last_became_active":"2023-10-30T12:28:50.397041+0000","last_became_peered":"2023-10-30T12:28:50.397041+0000","last_unstale":"2023-10-30T12:29:01.420755+0000","last_undegraded":"2023-10-30T12:29:01.420755+0000","last_fullsized":"2023-10-30T12:29:01.420755+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:52:22.564616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.580443+0000","last_change":"2023-10-30T12:28:50.966838+0000","last_active":"2023-10-30T12:28:53.580443+0000","last_peered":"2023-10-30T12:28:53.580443+0000","last_clean":"2023-10-30T12:28:53.580443+0000","last_became_active":"2023-10-30T12:28:50.966569+0000","last_became_peered":"2023-10-30T12:28:50.966569+0000","last_unstale":"2023-10-30T12:28:53.580443+0000","last_undegraded":"2023-10-30T12:28:53.580443+0000","last_fullsized":"2023-10-30T12:28:53.580443+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:44:33.875449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:56.364393+0000","last_change":"2023-10-30T12:28:50.391527+0000","last_active":"2023-10-30T12:28:56.364393+0000","last_peered":"2023-10-30T12:28:56.364393+0000","last_clean":"2023-10-30T12:28:56.364393+0000","last_became_active":"2023-10-30T12:28:50.391260+0000","last_became_peered":"2023-10-30T12:28:50.391260+0000","last_unstale":"2023-10-30T12:28:56.364393+0000","last_undegraded":"2023-10-30T12:28:56.364393+0000","last_fullsized":"2023-10-30T12:28:56.364393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:29:34.772490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.940211+0000","last_change":"2023-10-30T12:28:50.975441+0000","last_active":"2023-10-30T12:28:51.940211+0000","last_peered":"2023-10-30T12:28:51.940211+0000","last_clean":"2023-10-30T12:28:51.940211+0000","last_became_active":"2023-10-30T12:28:50.974843+0000","last_became_peered":"2023-10-30T12:28:50.974843+0000","last_unstale":"2023-10-30T12:28:51.940211+0000","last_undegraded":"2023-10-30T12:28:51.940211+0000","last_fullsized":"2023-10-30T12:28:51.940211+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:06:12.060752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"22'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.427140+0000","last_change":"2023-10-30T12:28:50.374838+0000","last_active":"2023-10-30T12:29:01.427140+0000","last_peered":"2023-10-30T12:29:01.427140+0000","last_clean":"2023-10-30T12:29:01.427140+0000","last_became_active":"2023-10-30T12:28:50.373581+0000","last_became_peered":"2023-10-30T12:28:50.373581+0000","last_unstale":"2023-10-30T12:29:01.427140+0000","last_undegraded":"2023-10-30T12:29:01.427140+0000","last_fullsized":"2023-10-30T12:29:01.427140+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:14:51.506745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.029046+0000","last_change":"2023-10-30T12:28:50.974012+0000","last_active":"2023-10-30T12:28:53.029046+0000","last_peered":"2023-10-30T12:28:53.029046+0000","last_clean":"2023-10-30T12:28:53.029046+0000","last_became_active":"2023-10-30T12:28:50.973269+0000","last_became_peered":"2023-10-30T12:28:50.973269+0000","last_unstale":"2023-10-30T12:28:53.029046+0000","last_undegraded":"2023-10-30T12:28:53.029046+0000","last_fullsized":"2023-10-30T12:28:53.029046+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:19:21.993898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"22'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.957225+0000","last_change":"2023-10-30T12:28:50.366590+0000","last_active":"2023-10-30T12:29:06.957225+0000","last_peered":"2023-10-30T12:29:06.957225+0000","last_clean":"2023-10-30T12:29:06.957225+0000","last_became_active":"2023-10-30T12:28:50.366269+0000","last_became_peered":"2023-10-30T12:28:50.366269+0000","last_unstale":"2023-10-30T12:29:06.957225+0000","last_undegraded":"2023-10-30T12:29:06.957225+0000","last_fullsized":"2023-10-30T12:29:06.957225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:15:11.963438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579425+0000","last_change":"2023-10-30T12:28:50.962058+0000","last_active":"2023-10-30T12:28:53.579425+0000","last_peered":"2023-10-30T12:28:53.579425+0000","last_clean":"2023-10-30T12:28:53.579425+0000","last_became_active":"2023-10-30T12:28:50.961805+0000","last_became_peered":"2023-10-30T12:28:50.961805+0000","last_unstale":"2023-10-30T12:28:53.579425+0000","last_undegraded":"2023-10-30T12:28:53.579425+0000","last_fullsized":"2023-10-30T12:28:53.579425+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:45:16.373386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.427282+0000","last_change":"2023-10-30T12:28:50.397244+0000","last_active":"2023-10-30T12:29:01.427282+0000","last_peered":"2023-10-30T12:29:01.427282+0000","last_clean":"2023-10-30T12:29:01.427282+0000","last_became_active":"2023-10-30T12:28:50.396727+0000","last_became_peered":"2023-10-30T12:28:50.396727+0000","last_unstale":"2023-10-30T12:29:01.427282+0000","last_undegraded":"2023-10-30T12:29:01.427282+0000","last_fullsized":"2023-10-30T12:29:01.427282+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:33.493343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028618+0000","last_change":"2023-10-30T12:28:50.981498+0000","last_active":"2023-10-30T12:28:53.028618+0000","last_peered":"2023-10-30T12:28:53.028618+0000","last_clean":"2023-10-30T12:28:53.028618+0000","last_became_active":"2023-10-30T12:28:50.981063+0000","last_became_peered":"2023-10-30T12:28:50.981063+0000","last_unstale":"2023-10-30T12:28:53.028618+0000","last_undegraded":"2023-10-30T12:28:53.028618+0000","last_fullsized":"2023-10-30T12:28:53.028618+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:55:17.971880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.959250+0000","last_change":"2023-10-30T12:28:50.383880+0000","last_active":"2023-10-30T12:29:06.959250+0000","last_peered":"2023-10-30T12:29:06.959250+0000","last_clean":"2023-10-30T12:29:06.959250+0000","last_became_active":"2023-10-30T12:28:50.383570+0000","last_became_peered":"2023-10-30T12:28:50.383570+0000","last_unstale":"2023-10-30T12:29:06.959250+0000","last_undegraded":"2023-10-30T12:29:06.959250+0000","last_fullsized":"2023-10-30T12:29:06.959250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:19:42.047520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.936983+0000","last_change":"2023-10-30T12:28:50.950312+0000","last_active":"2023-10-30T12:28:51.936983+0000","last_peered":"2023-10-30T12:28:51.936983+0000","last_clean":"2023-10-30T12:28:51.936983+0000","last_became_active":"2023-10-30T12:28:50.949485+0000","last_became_peered":"2023-10-30T12:28:50.949485+0000","last_unstale":"2023-10-30T12:28:51.936983+0000","last_undegraded":"2023-10-30T12:28:51.936983+0000","last_fullsized":"2023-10-30T12:28:51.936983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:13:04.669915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"22'15","reported_seq":27,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.968260+0000","last_change":"2023-10-30T12:28:50.362979+0000","last_active":"2023-10-30T12:29:06.968260+0000","last_peered":"2023-10-30T12:29:06.968260+0000","last_clean":"2023-10-30T12:29:06.968260+0000","last_became_active":"2023-10-30T12:28:50.362624+0000","last_became_peered":"2023-10-30T12:28:50.362624+0000","last_unstale":"2023-10-30T12:29:06.968260+0000","last_undegraded":"2023-10-30T12:29:06.968260+0000","last_fullsized":"2023-10-30T12:29:06.968260+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:01:13.111816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028233+0000","last_change":"2023-10-30T12:28:50.979815+0000","last_active":"2023-10-30T12:28:53.028233+0000","last_peered":"2023-10-30T12:28:53.028233+0000","last_clean":"2023-10-30T12:28:53.028233+0000","last_became_active":"2023-10-30T12:28:50.979344+0000","last_became_peered":"2023-10-30T12:28:50.979344+0000","last_unstale":"2023-10-30T12:28:53.028233+0000","last_undegraded":"2023-10-30T12:28:53.028233+0000","last_fullsized":"2023-10-30T12:28:53.028233+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:35:37.018282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:52.284562+0000","last_change":"2023-10-30T12:28:50.382175+0000","last_active":"2023-10-30T12:28:52.284562+0000","last_peered":"2023-10-30T12:28:52.284562+0000","last_clean":"2023-10-30T12:28:52.284562+0000","last_became_active":"2023-10-30T12:28:50.381125+0000","last_became_peered":"2023-10-30T12:28:50.381125+0000","last_unstale":"2023-10-30T12:28:52.284562+0000","last_undegraded":"2023-10-30T12:28:52.284562+0000","last_fullsized":"2023-10-30T12:28:52.284562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:36:02.652665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028807+0000","last_change":"2023-10-30T12:28:50.978034+0000","last_active":"2023-10-30T12:28:53.028807+0000","last_peered":"2023-10-30T12:28:53.028807+0000","last_clean":"2023-10-30T12:28:53.028807+0000","last_became_active":"2023-10-30T12:28:50.977585+0000","last_became_peered":"2023-10-30T12:28:50.977585+0000","last_unstale":"2023-10-30T12:28:53.028807+0000","last_undegraded":"2023-10-30T12:28:53.028807+0000","last_fullsized":"2023-10-30T12:28:53.028807+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:42:56.806184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:56.364009+0000","last_change":"2023-10-30T12:28:50.367258+0000","last_active":"2023-10-30T12:28:56.364009+0000","last_peered":"2023-10-30T12:28:56.364009+0000","last_clean":"2023-10-30T12:28:56.364009+0000","last_became_active":"2023-10-30T12:28:50.366974+0000","last_became_peered":"2023-10-30T12:28:50.366974+0000","last_unstale":"2023-10-30T12:28:56.364009+0000","last_undegraded":"2023-10-30T12:28:56.364009+0000","last_fullsized":"2023-10-30T12:28:56.364009+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:14:48.823902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.029189+0000","last_change":"2023-10-30T12:28:50.981784+0000","last_active":"2023-10-30T12:28:53.029189+0000","last_peered":"2023-10-30T12:28:53.029189+0000","last_clean":"2023-10-30T12:28:53.029189+0000","last_became_active":"2023-10-30T12:28:50.981193+0000","last_became_peered":"2023-10-30T12:28:50.981193+0000","last_unstale":"2023-10-30T12:28:53.029189+0000","last_undegraded":"2023-10-30T12:28:53.029189+0000","last_fullsized":"2023-10-30T12:28:53.029189+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:36:17.384732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:52.280239+0000","last_change":"2023-10-30T12:28:50.368011+0000","last_active":"2023-10-30T12:28:52.280239+0000","last_peered":"2023-10-30T12:28:52.280239+0000","last_clean":"2023-10-30T12:28:52.280239+0000","last_became_active":"2023-10-30T12:28:50.367730+0000","last_became_peered":"2023-10-30T12:28:50.367730+0000","last_unstale":"2023-10-30T12:28:52.280239+0000","last_undegraded":"2023-10-30T12:28:52.280239+0000","last_fullsized":"2023-10-30T12:28:52.280239+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:59:41.946550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.582218+0000","last_change":"2023-10-30T12:28:50.968262+0000","last_active":"2023-10-30T12:28:53.582218+0000","last_peered":"2023-10-30T12:28:53.582218+0000","last_clean":"2023-10-30T12:28:53.582218+0000","last_became_active":"2023-10-30T12:28:50.966246+0000","last_became_peered":"2023-10-30T12:28:50.966246+0000","last_unstale":"2023-10-30T12:28:53.582218+0000","last_undegraded":"2023-10-30T12:28:53.582218+0000","last_fullsized":"2023-10-30T12:28:53.582218+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:00:01.991935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.962658+0000","last_change":"2023-10-30T12:28:50.394060+0000","last_active":"2023-10-30T12:29:06.962658+0000","last_peered":"2023-10-30T12:29:06.962658+0000","last_clean":"2023-10-30T12:29:06.962658+0000","last_became_active":"2023-10-30T12:28:50.393642+0000","last_became_peered":"2023-10-30T12:28:50.393642+0000","last_unstale":"2023-10-30T12:29:06.962658+0000","last_undegraded":"2023-10-30T12:29:06.962658+0000","last_fullsized":"2023-10-30T12:29:06.962658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:21.643882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028738+0000","last_change":"2023-10-30T12:28:50.976814+0000","last_active":"2023-10-30T12:28:53.028738+0000","last_peered":"2023-10-30T12:28:53.028738+0000","last_clean":"2023-10-30T12:28:53.028738+0000","last_became_active":"2023-10-30T12:28:50.976343+0000","last_became_peered":"2023-10-30T12:28:50.976343+0000","last_unstale":"2023-10-30T12:28:53.028738+0000","last_undegraded":"2023-10-30T12:28:53.028738+0000","last_fullsized":"2023-10-30T12:28:53.028738+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:33:23.491998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.946067+0000","last_change":"2023-10-30T12:28:41.203990+0000","last_active":"2023-10-30T12:28:50.946067+0000","last_peered":"2023-10-30T12:28:50.946067+0000","last_clean":"2023-10-30T12:28:50.946067+0000","last_became_active":"2023-10-30T12:28:41.201796+0000","last_became_peered":"2023-10-30T12:28:41.201796+0000","last_unstale":"2023-10-30T12:28:50.946067+0000","last_undegraded":"2023-10-30T12:28:50.946067+0000","last_fullsized":"2023-10-30T12:28:50.946067+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:39.218257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:39.218257+0000","last_clean_scrub_stamp":"2023-10-30T12:28:39.218257+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:21:00.605462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.426165+0000","last_change":"2023-10-30T12:28:50.372651+0000","last_active":"2023-10-30T12:29:01.426165+0000","last_peered":"2023-10-30T12:29:01.426165+0000","last_clean":"2023-10-30T12:29:01.426165+0000","last_became_active":"2023-10-30T12:28:50.367593+0000","last_became_peered":"2023-10-30T12:28:50.367593+0000","last_unstale":"2023-10-30T12:29:01.426165+0000","last_undegraded":"2023-10-30T12:29:01.426165+0000","last_fullsized":"2023-10-30T12:29:01.426165+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:32:10.492061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579128+0000","last_change":"2023-10-30T12:28:50.970536+0000","last_active":"2023-10-30T12:28:53.579128+0000","last_peered":"2023-10-30T12:28:53.579128+0000","last_clean":"2023-10-30T12:28:53.579128+0000","last_became_active":"2023-10-30T12:28:50.970326+0000","last_became_peered":"2023-10-30T12:28:50.970326+0000","last_unstale":"2023-10-30T12:28:53.579128+0000","last_undegraded":"2023-10-30T12:28:53.579128+0000","last_fullsized":"2023-10-30T12:28:53.579128+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:53:54.589174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.958451+0000","last_change":"2023-10-30T12:28:50.392461+0000","last_active":"2023-10-30T12:29:06.958451+0000","last_peered":"2023-10-30T12:29:06.958451+0000","last_clean":"2023-10-30T12:29:06.958451+0000","last_became_active":"2023-10-30T12:28:50.391824+0000","last_became_peered":"2023-10-30T12:28:50.391824+0000","last_unstale":"2023-10-30T12:29:06.958451+0000","last_undegraded":"2023-10-30T12:29:06.958451+0000","last_fullsized":"2023-10-30T12:29:06.958451+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:56:10.373046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579229+0000","last_change":"2023-10-30T12:28:50.953046+0000","last_active":"2023-10-30T12:28:53.579229+0000","last_peered":"2023-10-30T12:28:53.579229+0000","last_clean":"2023-10-30T12:28:53.579229+0000","last_became_active":"2023-10-30T12:28:50.952625+0000","last_became_peered":"2023-10-30T12:28:50.952625+0000","last_unstale":"2023-10-30T12:28:53.579229+0000","last_undegraded":"2023-10-30T12:28:53.579229+0000","last_fullsized":"2023-10-30T12:28:53.579229+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:00:14.643989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.962549+0000","last_change":"2023-10-30T12:28:50.367547+0000","last_active":"2023-10-30T12:29:06.962549+0000","last_peered":"2023-10-30T12:29:06.962549+0000","last_clean":"2023-10-30T12:29:06.962549+0000","last_became_active":"2023-10-30T12:28:50.367230+0000","last_became_peered":"2023-10-30T12:28:50.367230+0000","last_unstale":"2023-10-30T12:29:06.962549+0000","last_undegraded":"2023-10-30T12:29:06.962549+0000","last_fullsized":"2023-10-30T12:29:06.962549+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:42.570786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931639+0000","last_change":"2023-10-30T12:28:50.928888+0000","last_active":"2023-10-30T12:28:51.931639+0000","last_peered":"2023-10-30T12:28:51.931639+0000","last_clean":"2023-10-30T12:28:51.931639+0000","last_became_active":"2023-10-30T12:28:50.928608+0000","last_became_peered":"2023-10-30T12:28:50.928608+0000","last_unstale":"2023-10-30T12:28:51.931639+0000","last_undegraded":"2023-10-30T12:28:51.931639+0000","last_fullsized":"2023-10-30T12:28:51.931639+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:40:51.042396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.582018+0000","last_change":"2023-10-30T12:28:50.391240+0000","last_active":"2023-10-30T12:28:53.582018+0000","last_peered":"2023-10-30T12:28:53.582018+0000","last_clean":"2023-10-30T12:28:53.582018+0000","last_became_active":"2023-10-30T12:28:50.390864+0000","last_became_peered":"2023-10-30T12:28:50.390864+0000","last_unstale":"2023-10-30T12:28:53.582018+0000","last_undegraded":"2023-10-30T12:28:53.582018+0000","last_fullsized":"2023-10-30T12:28:53.582018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:59:40.285830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.939756+0000","last_change":"2023-10-30T12:28:50.949990+0000","last_active":"2023-10-30T12:28:51.939756+0000","last_peered":"2023-10-30T12:28:51.939756+0000","last_clean":"2023-10-30T12:28:51.939756+0000","last_became_active":"2023-10-30T12:28:50.949122+0000","last_became_peered":"2023-10-30T12:28:50.949122+0000","last_unstale":"2023-10-30T12:28:51.939756+0000","last_undegraded":"2023-10-30T12:28:51.939756+0000","last_fullsized":"2023-10-30T12:28:51.939756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:30:56.099810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931803+0000","last_change":"2023-10-30T12:28:50.358700+0000","last_active":"2023-10-30T12:28:51.931803+0000","last_peered":"2023-10-30T12:28:51.931803+0000","last_clean":"2023-10-30T12:28:51.931803+0000","last_became_active":"2023-10-30T12:28:50.358310+0000","last_became_peered":"2023-10-30T12:28:50.358310+0000","last_unstale":"2023-10-30T12:28:51.931803+0000","last_undegraded":"2023-10-30T12:28:51.931803+0000","last_fullsized":"2023-10-30T12:28:51.931803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:23:30.910622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.582250+0000","last_change":"2023-10-30T12:28:50.967650+0000","last_active":"2023-10-30T12:28:53.582250+0000","last_peered":"2023-10-30T12:28:53.582250+0000","last_clean":"2023-10-30T12:28:53.582250+0000","last_became_active":"2023-10-30T12:28:50.965626+0000","last_became_peered":"2023-10-30T12:28:50.965626+0000","last_unstale":"2023-10-30T12:28:53.582250+0000","last_undegraded":"2023-10-30T12:28:53.582250+0000","last_fullsized":"2023-10-30T12:28:53.582250+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:49.050703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.029231+0000","last_change":"2023-10-30T12:28:50.395645+0000","last_active":"2023-10-30T12:28:53.029231+0000","last_peered":"2023-10-30T12:28:53.029231+0000","last_clean":"2023-10-30T12:28:53.029231+0000","last_became_active":"2023-10-30T12:28:50.395090+0000","last_became_peered":"2023-10-30T12:28:50.395090+0000","last_unstale":"2023-10-30T12:28:53.029231+0000","last_undegraded":"2023-10-30T12:28:53.029231+0000","last_fullsized":"2023-10-30T12:28:53.029231+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:45:44.678383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931844+0000","last_change":"2023-10-30T12:28:50.929586+0000","last_active":"2023-10-30T12:28:51.931844+0000","last_peered":"2023-10-30T12:28:51.931844+0000","last_clean":"2023-10-30T12:28:51.931844+0000","last_became_active":"2023-10-30T12:28:50.929337+0000","last_became_peered":"2023-10-30T12:28:50.929337+0000","last_unstale":"2023-10-30T12:28:51.931844+0000","last_undegraded":"2023-10-30T12:28:51.931844+0000","last_fullsized":"2023-10-30T12:28:51.931844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:11:13.067698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.958476+0000","last_change":"2023-10-30T12:28:50.380474+0000","last_active":"2023-10-30T12:29:06.958476+0000","last_peered":"2023-10-30T12:29:06.958476+0000","last_clean":"2023-10-30T12:29:06.958476+0000","last_became_active":"2023-10-30T12:28:50.380247+0000","last_became_peered":"2023-10-30T12:28:50.380247+0000","last_unstale":"2023-10-30T12:29:06.958476+0000","last_undegraded":"2023-10-30T12:29:06.958476+0000","last_fullsized":"2023-10-30T12:29:06.958476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:52:03.952617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579476+0000","last_change":"2023-10-30T12:28:50.973361+0000","last_active":"2023-10-30T12:28:53.579476+0000","last_peered":"2023-10-30T12:28:53.579476+0000","last_clean":"2023-10-30T12:28:53.579476+0000","last_became_active":"2023-10-30T12:28:50.973086+0000","last_became_peered":"2023-10-30T12:28:50.973086+0000","last_unstale":"2023-10-30T12:28:53.579476+0000","last_undegraded":"2023-10-30T12:28:53.579476+0000","last_fullsized":"2023-10-30T12:28:53.579476+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:49:48.589889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.421918+0000","last_change":"2023-10-30T12:28:50.386277+0000","last_active":"2023-10-30T12:29:01.421918+0000","last_peered":"2023-10-30T12:29:01.421918+0000","last_clean":"2023-10-30T12:29:01.421918+0000","last_became_active":"2023-10-30T12:28:50.385924+0000","last_became_peered":"2023-10-30T12:28:50.385924+0000","last_unstale":"2023-10-30T12:29:01.421918+0000","last_undegraded":"2023-10-30T12:29:01.421918+0000","last_fullsized":"2023-10-30T12:29:01.421918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:35:03.837523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.580322+0000","last_change":"2023-10-30T12:28:50.929670+0000","last_active":"2023-10-30T12:28:53.580322+0000","last_peered":"2023-10-30T12:28:53.580322+0000","last_clean":"2023-10-30T12:28:53.580322+0000","last_became_active":"2023-10-30T12:28:50.929259+0000","last_became_peered":"2023-10-30T12:28:50.929259+0000","last_unstale":"2023-10-30T12:28:53.580322+0000","last_undegraded":"2023-10-30T12:28:53.580322+0000","last_fullsized":"2023-10-30T12:28:53.580322+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:50:16.507660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.420373+0000","last_change":"2023-10-30T12:28:50.386426+0000","last_active":"2023-10-30T12:29:01.420373+0000","last_peered":"2023-10-30T12:29:01.420373+0000","last_clean":"2023-10-30T12:29:01.420373+0000","last_became_active":"2023-10-30T12:28:50.385305+0000","last_became_peered":"2023-10-30T12:28:50.385305+0000","last_unstale":"2023-10-30T12:29:01.420373+0000","last_undegraded":"2023-10-30T12:29:01.420373+0000","last_fullsized":"2023-10-30T12:29:01.420373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:13:05.376581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.938368+0000","last_change":"2023-10-30T12:28:50.941607+0000","last_active":"2023-10-30T12:28:51.938368+0000","last_peered":"2023-10-30T12:28:51.938368+0000","last_clean":"2023-10-30T12:28:51.938368+0000","last_became_active":"2023-10-30T12:28:50.940846+0000","last_became_peered":"2023-10-30T12:28:50.940846+0000","last_unstale":"2023-10-30T12:28:51.938368+0000","last_undegraded":"2023-10-30T12:28:51.938368+0000","last_fullsized":"2023-10-30T12:28:51.938368+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:29.837077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.433290+0000","last_change":"2023-10-30T12:28:50.392333+0000","last_active":"2023-10-30T12:29:01.433290+0000","last_peered":"2023-10-30T12:29:01.433290+0000","last_clean":"2023-10-30T12:29:01.433290+0000","last_became_active":"2023-10-30T12:28:50.392068+0000","last_became_peered":"2023-10-30T12:28:50.392068+0000","last_unstale":"2023-10-30T12:29:01.433290+0000","last_undegraded":"2023-10-30T12:29:01.433290+0000","last_fullsized":"2023-10-30T12:29:01.433290+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:47:24.674925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579044+0000","last_change":"2023-10-30T12:28:50.932167+0000","last_active":"2023-10-30T12:28:53.579044+0000","last_peered":"2023-10-30T12:28:53.579044+0000","last_clean":"2023-10-30T12:28:53.579044+0000","last_became_active":"2023-10-30T12:28:50.931898+0000","last_became_peered":"2023-10-30T12:28:50.931898+0000","last_unstale":"2023-10-30T12:28:53.579044+0000","last_undegraded":"2023-10-30T12:28:53.579044+0000","last_fullsized":"2023-10-30T12:28:53.579044+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:44.429045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:52.280154+0000","last_change":"2023-10-30T12:28:50.365055+0000","last_active":"2023-10-30T12:28:52.280154+0000","last_peered":"2023-10-30T12:28:52.280154+0000","last_clean":"2023-10-30T12:28:52.280154+0000","last_became_active":"2023-10-30T12:28:50.364797+0000","last_became_peered":"2023-10-30T12:28:50.364797+0000","last_unstale":"2023-10-30T12:28:52.280154+0000","last_undegraded":"2023-10-30T12:28:52.280154+0000","last_fullsized":"2023-10-30T12:28:52.280154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:06:23.067895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579266+0000","last_change":"2023-10-30T12:28:50.935908+0000","last_active":"2023-10-30T12:28:53.579266+0000","last_peered":"2023-10-30T12:28:53.579266+0000","last_clean":"2023-10-30T12:28:53.579266+0000","last_became_active":"2023-10-30T12:28:50.935657+0000","last_became_peered":"2023-10-30T12:28:50.935657+0000","last_unstale":"2023-10-30T12:28:53.579266+0000","last_undegraded":"2023-10-30T12:28:53.579266+0000","last_fullsized":"2023-10-30T12:28:53.579266+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:36:13.242429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.416976+0000","last_change":"2023-10-30T12:28:50.393465+0000","last_active":"2023-10-30T12:29:01.416976+0000","last_peered":"2023-10-30T12:29:01.416976+0000","last_clean":"2023-10-30T12:29:01.416976+0000","last_became_active":"2023-10-30T12:28:50.392807+0000","last_became_peered":"2023-10-30T12:28:50.392807+0000","last_unstale":"2023-10-30T12:29:01.416976+0000","last_undegraded":"2023-10-30T12:29:01.416976+0000","last_fullsized":"2023-10-30T12:29:01.416976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:35:37.976956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.938767+0000","last_change":"2023-10-30T12:28:50.954834+0000","last_active":"2023-10-30T12:28:51.938767+0000","last_peered":"2023-10-30T12:28:51.938767+0000","last_clean":"2023-10-30T12:28:51.938767+0000","last_became_active":"2023-10-30T12:28:50.954291+0000","last_became_peered":"2023-10-30T12:28:50.954291+0000","last_unstale":"2023-10-30T12:28:51.938767+0000","last_undegraded":"2023-10-30T12:28:51.938767+0000","last_fullsized":"2023-10-30T12:28:51.938767+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:19:44.345918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581732+0000","last_change":"2023-10-30T12:28:50.963620+0000","last_active":"2023-10-30T12:28:53.581732+0000","last_peered":"2023-10-30T12:28:53.581732+0000","last_clean":"2023-10-30T12:28:53.581732+0000","last_became_active":"2023-10-30T12:28:50.962817+0000","last_became_peered":"2023-10-30T12:28:50.962817+0000","last_unstale":"2023-10-30T12:28:53.581732+0000","last_undegraded":"2023-10-30T12:28:53.581732+0000","last_fullsized":"2023-10-30T12:28:53.581732+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:23:23.276086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:56.367962+0000","last_change":"2023-10-30T12:28:50.402179+0000","last_active":"2023-10-30T12:28:56.367962+0000","last_peered":"2023-10-30T12:28:56.367962+0000","last_clean":"2023-10-30T12:28:56.367962+0000","last_became_active":"2023-10-30T12:28:50.401677+0000","last_became_peered":"2023-10-30T12:28:50.401677+0000","last_unstale":"2023-10-30T12:28:56.367962+0000","last_undegraded":"2023-10-30T12:28:56.367962+0000","last_fullsized":"2023-10-30T12:28:56.367962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:39:44.432214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581874+0000","last_change":"2023-10-30T12:28:50.968805+0000","last_active":"2023-10-30T12:28:53.581874+0000","last_peered":"2023-10-30T12:28:53.581874+0000","last_clean":"2023-10-30T12:28:53.581874+0000","last_became_active":"2023-10-30T12:28:50.967591+0000","last_became_peered":"2023-10-30T12:28:50.967591+0000","last_unstale":"2023-10-30T12:28:53.581874+0000","last_undegraded":"2023-10-30T12:28:53.581874+0000","last_fullsized":"2023-10-30T12:28:53.581874+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:11:41.973858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.957870+0000","last_change":"2023-10-30T12:28:50.392617+0000","last_active":"2023-10-30T12:29:06.957870+0000","last_peered":"2023-10-30T12:29:06.957870+0000","last_clean":"2023-10-30T12:29:06.957870+0000","last_became_active":"2023-10-30T12:28:50.392021+0000","last_became_peered":"2023-10-30T12:28:50.392021+0000","last_unstale":"2023-10-30T12:29:06.957870+0000","last_undegraded":"2023-10-30T12:29:06.957870+0000","last_fullsized":"2023-10-30T12:29:06.957870+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:52:30.621125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.580378+0000","last_change":"2023-10-30T12:28:50.967491+0000","last_active":"2023-10-30T12:28:53.580378+0000","last_peered":"2023-10-30T12:28:53.580378+0000","last_clean":"2023-10-30T12:28:53.580378+0000","last_became_active":"2023-10-30T12:28:50.967186+0000","last_became_peered":"2023-10-30T12:28:50.967186+0000","last_unstale":"2023-10-30T12:28:53.580378+0000","last_undegraded":"2023-10-30T12:28:53.580378+0000","last_fullsized":"2023-10-30T12:28:53.580378+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:05:23.759940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:56.365702+0000","last_change":"2023-10-30T12:28:50.363230+0000","last_active":"2023-10-30T12:28:56.365702+0000","last_peered":"2023-10-30T12:28:56.365702+0000","last_clean":"2023-10-30T12:28:56.365702+0000","last_became_active":"2023-10-30T12:28:50.362929+0000","last_became_peered":"2023-10-30T12:28:50.362929+0000","last_unstale":"2023-10-30T12:28:56.365702+0000","last_undegraded":"2023-10-30T12:28:56.365702+0000","last_fullsized":"2023-10-30T12:28:56.365702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:41:27.760578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579661+0000","last_change":"2023-10-30T12:28:50.937780+0000","last_active":"2023-10-30T12:28:53.579661+0000","last_peered":"2023-10-30T12:28:53.579661+0000","last_clean":"2023-10-30T12:28:53.579661+0000","last_became_active":"2023-10-30T12:28:50.937458+0000","last_became_peered":"2023-10-30T12:28:50.937458+0000","last_unstale":"2023-10-30T12:28:53.579661+0000","last_undegraded":"2023-10-30T12:28:53.579661+0000","last_fullsized":"2023-10-30T12:28:53.579661+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:02.083169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"22'9","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.962486+0000","last_change":"2023-10-30T12:28:50.386782+0000","last_active":"2023-10-30T12:29:06.962486+0000","last_peered":"2023-10-30T12:29:06.962486+0000","last_clean":"2023-10-30T12:29:06.962486+0000","last_became_active":"2023-10-30T12:28:50.386522+0000","last_became_peered":"2023-10-30T12:28:50.386522+0000","last_unstale":"2023-10-30T12:29:06.962486+0000","last_undegraded":"2023-10-30T12:29:06.962486+0000","last_fullsized":"2023-10-30T12:29:06.962486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:44:03.517641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5222,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":124,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":10444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108,"ondisk_log_size":108,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542155,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27836,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344004,"statfs":{"total":96636764160,"available":96608260096,"internally_reserved":0,"allocated":839680,"data_stored":670548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.194}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0169999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.161}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.26}]}]},{"osd":6,"up_from":14,"seq":60129542155,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":245760,"data_stored":81154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94999999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1819999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542155,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":241664,"data_stored":79623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1559999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0269999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2030000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542154,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":241664,"data_stored":79578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.05}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.012}]}]},{"osd":3,"up_from":14,"seq":60129542155,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":249856,"data_stored":81188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94599999999999995}]}]},{"osd":2,"up_from":14,"seq":60129542155,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":241664,"data_stored":79063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0009999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80400000000000005}]}]},{"osd":1,"up_from":14,"seq":60129542155,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":249856,"data_stored":79808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]}]},{"osd":0,"up_from":14,"seq":60129542155,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":831488,"data_stored":669273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.052}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:29:18.756 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:29:18.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.927+0000 7f03c8646640 1 Processor -- start 2023-10-30T12:29:18.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.927+0000 7f03c8646640 1 -- start start 2023-10-30T12:29:18.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.931+0000 7f03c8646640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f03c01538a0 0x7f03c015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:18.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.931+0000 7f03c8646640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03c00ca0e0 0x7f03c015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:18.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.931+0000 7f03c8646640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c00cb800 con 0x7f03c00cf160 2023-10-30T12:29:18.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.931+0000 7f03c8646640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03c015dc30 con 0x7f03c01538a0 2023-10-30T12:29:18.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.931+0000 7f03c8646640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c00cb330 con 0x7f03c00ca0e0 2023-10-30T12:29:18.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.931+0000 7f03c6be3640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f03c00cf160 0x7f03c00cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58396/0 (socket says 172.21.15.89:58396) 2023-10-30T12:29:18.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.931+0000 7f03c6be3640 1 -- 172.21.15.89:0/191878061 learned_addr learned my addr 172.21.15.89:0/191878061 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:29:18.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.931+0000 7f03c63e2640 1 --2- 172.21.15.89:0/191878061 >> v2:172.21.15.192:3300/0 conn(0x7f03c01538a0 0x7f03c015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:18.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.931+0000 7f03c5be1640 1 --2- 172.21.15.89:0/191878061 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03c00ca0e0 0x7f03c015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:18.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.931+0000 7f03c53e0640 1 -- 172.21.15.89:0/191878061 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 358301542 0 0) 0x7f03c00cb800 con 0x7f03c00cf160 2023-10-30T12:29:18.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.931+0000 7f03c53e0640 1 -- 172.21.15.89:0/191878061 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03ac003530 con 0x7f03c00cf160 2023-10-30T12:29:18.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.931+0000 7f03c63e2640 1 -- 172.21.15.89:0/191878061 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03c00ca0e0 msgr2=0x7f03c015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:18.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.931+0000 7f03c53e0640 1 -- 172.21.15.89:0/191878061 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1007940670 0 0) 0x7f03ac003530 con 0x7f03c00cf160 2023-10-30T12:29:18.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.931+0000 7f03c63e2640 1 --2- 172.21.15.89:0/191878061 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03c00ca0e0 0x7f03c015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:18.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.931+0000 7f03c63e2640 1 -- 172.21.15.89:0/191878061 >> v1:172.21.15.89:6789/0 conn(0x7f03c00cf160 legacy=0x7f03c00cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:18.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.931+0000 7f03c63e2640 1 -- 172.21.15.89:0/191878061 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c00cad70 con 0x7f03c01538a0 2023-10-30T12:29:18.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.935+0000 7f03c63e2640 1 --2- 172.21.15.89:0/191878061 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03c01538a0 0x7f03c015d6f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f03bc0097c0 tx=0x7f03bc01eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17b53519948dbfd7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:18.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.935+0000 7f03c53e0640 1 -- 172.21.15.89:0/191878061 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b8001a00 con 0x7f03c01538a0 2023-10-30T12:29:18.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.935+0000 7f03c53e0640 1 -- 172.21.15.89:0/191878061 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03bc0a85d0 con 0x7f03c01538a0 2023-10-30T12:29:18.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.935+0000 7f03c53e0640 1 -- 172.21.15.89:0/191878061 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03bc0a8940 con 0x7f03c01538a0 2023-10-30T12:29:18.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.935+0000 7f03c8646640 1 -- 172.21.15.89:0/191878061 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03c01538a0 msgr2=0x7f03c015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:18.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.935+0000 7f03c8646640 1 --2- 172.21.15.89:0/191878061 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03c01538a0 0x7f03c015d6f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f03bc0097c0 tx=0x7f03bc01eb90 comp rx=0 tx=0).stop 2023-10-30T12:29:18.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.935+0000 7f03c8646640 1 -- 172.21.15.89:0/191878061 shutdown_connections 2023-10-30T12:29:18.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.935+0000 7f03c8646640 1 --2- 172.21.15.89:0/191878061 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03c00ca0e0 0x7f03c015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:18.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.935+0000 7f03c8646640 1 --2- 172.21.15.89:0/191878061 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03c01538a0 0x7f03c015d6f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:18.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.935+0000 7f03c8646640 1 -- 172.21.15.89:0/191878061 >> 172.21.15.89:0/191878061 conn(0x7f03c006bfe0 msgr2=0x7f03c006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:18.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.935+0000 7f03c8646640 1 -- 172.21.15.89:0/191878061 shutdown_connections 2023-10-30T12:29:18.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.935+0000 7f03c8646640 1 -- 172.21.15.89:0/191878061 wait complete. 2023-10-30T12:29:18.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.939+0000 7f03c8646640 1 Processor -- start 2023-10-30T12:29:18.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.939+0000 7f03c8646640 1 -- start start 2023-10-30T12:29:18.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.939+0000 7f03c8646640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03c00ca0e0 0x7f03c01efe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:18.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.939+0000 7f03c8646640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03c00cf160 0x7f03c015c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:18.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.939+0000 7f03c8646640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03c01538a0 0x7f03c01f3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:18.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.939+0000 7f03c8646640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c01f36d0 con 0x7f03c00ca0e0 2023-10-30T12:29:18.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.939+0000 7f03c8646640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c01ef970 con 0x7f03c01538a0 2023-10-30T12:29:18.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.939+0000 7f03c8646640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c01436f0 con 0x7f03c00cf160 2023-10-30T12:29:18.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.939+0000 7f03c63e2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03c00ca0e0 0x7f03c01efe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:18.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.939+0000 7f03c63e2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03c00ca0e0 0x7f03c01efe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57952/0 (socket says 172.21.15.89:57952) 2023-10-30T12:29:18.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.939+0000 7f03c63e2640 1 -- 172.21.15.89:0/3801010436 learned_addr learned my addr 172.21.15.89:0/3801010436 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:18.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.939+0000 7f03c6be3640 1 --2- 172.21.15.89:0/3801010436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03c01538a0 0x7f03c01f3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:18.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.939+0000 7f03c5be1640 1 --2- 172.21.15.89:0/3801010436 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03c00cf160 0x7f03c015c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:18.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.939+0000 7f03c63e2640 1 -- 172.21.15.89:0/3801010436 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03c00cf160 msgr2=0x7f03c015c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:18.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.939+0000 7f03c63e2640 1 --2- 172.21.15.89:0/3801010436 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03c00cf160 0x7f03c015c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:18.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.939+0000 7f03c63e2640 1 -- 172.21.15.89:0/3801010436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03c01538a0 msgr2=0x7f03c01f3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:18.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.939+0000 7f03c63e2640 1 --2- 172.21.15.89:0/3801010436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03c01538a0 0x7f03c01f3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:18.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.939+0000 7f03c63e2640 1 -- 172.21.15.89:0/3801010436 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03bc009420 con 0x7f03c00ca0e0 2023-10-30T12:29:18.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.939+0000 7f03c6be3640 1 --2- 172.21.15.89:0/3801010436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03c01538a0 0x7f03c01f3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:18.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.939+0000 7f03c63e2640 1 --2- 172.21.15.89:0/3801010436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03c00ca0e0 0x7f03c01efe00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f03bc01d240 tx=0x7f03bc01b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:18.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.939+0000 7f03b6ffd640 1 -- 172.21.15.89:0/3801010436 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03bc0b0850 con 0x7f03c00ca0e0 2023-10-30T12:29:18.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.939+0000 7f03b6ffd640 1 -- 172.21.15.89:0/3801010436 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03bc0b09b0 con 0x7f03c00ca0e0 2023-10-30T12:29:18.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.939+0000 7f03b6ffd640 1 -- 172.21.15.89:0/3801010436 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03bc0b0d20 con 0x7f03c00ca0e0 2023-10-30T12:29:18.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.939+0000 7f03c8646640 1 -- 172.21.15.89:0/3801010436 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c01508e0 con 0x7f03c00ca0e0 2023-10-30T12:29:18.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.939+0000 7f03c8646640 1 -- 172.21.15.89:0/3801010436 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c0150b80 con 0x7f03c00ca0e0 2023-10-30T12:29:18.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.939+0000 7f03c8646640 1 -- 172.21.15.89:0/3801010436 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c00b2a50 con 0x7f03c00ca0e0 2023-10-30T12:29:18.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.943+0000 7f03b6ffd640 1 -- 172.21.15.89:0/3801010436 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f03bc0b7040 con 0x7f03c00ca0e0 2023-10-30T12:29:18.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.943+0000 7f03b6ffd640 1 --2- 172.21.15.89:0/3801010436 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f03a40664a0 0x7f03a4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:18.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.943+0000 7f03b6ffd640 1 -- 172.21.15.89:0/3801010436 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f03bc121530 con 0x7f03c00ca0e0 2023-10-30T12:29:18.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.943+0000 7f03c5be1640 1 --2- 172.21.15.89:0/3801010436 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f03a40664a0 0x7f03a4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:18.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.943+0000 7f03c5be1640 1 --2- 172.21.15.89:0/3801010436 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f03a40664a0 0x7f03a4068960 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f03c00c5a60 tx=0x7f03b0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:18.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:18.947+0000 7f03b6ffd640 1 -- 172.21.15.89:0/3801010436 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f03bc0ee9a0 con 0x7f03c00ca0e0 2023-10-30T12:29:19.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.091+0000 7f03c8646640 1 -- 172.21.15.89:0/3801010436 --> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f03c0159890 con 0x7f03a40664a0 2023-10-30T12:29:19.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.111+0000 7f03b6ffd640 1 -- 172.21.15.89:0/3801010436 <== mgr.4105 v2:172.21.15.192:6832/4076276638 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264666 (secure 0 0 0) 0x7f03b00118b0 con 0x7f03a40664a0 2023-10-30T12:29:19.118 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:29:19.120 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-10-30T12:29:19.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.115+0000 7f03c8646640 1 -- 172.21.15.89:0/3801010436 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f03a40664a0 msgr2=0x7f03a4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:19.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.115+0000 7f03c8646640 1 --2- 172.21.15.89:0/3801010436 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f03a40664a0 0x7f03a4068960 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f03c00c5a60 tx=0x7f03b0008040 comp rx=0 tx=0).stop 2023-10-30T12:29:19.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.115+0000 7f03c8646640 1 -- 172.21.15.89:0/3801010436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03c00ca0e0 msgr2=0x7f03c01efe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:19.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.115+0000 7f03c8646640 1 --2- 172.21.15.89:0/3801010436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03c00ca0e0 0x7f03c01efe00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f03bc01d240 tx=0x7f03bc01b930 comp rx=0 tx=0).stop 2023-10-30T12:29:19.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.115+0000 7f03c8646640 1 -- 172.21.15.89:0/3801010436 shutdown_connections 2023-10-30T12:29:19.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.115+0000 7f03c8646640 1 --2- 172.21.15.89:0/3801010436 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f03a40664a0 0x7f03a4068960 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:19.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.115+0000 7f03c8646640 1 --2- 172.21.15.89:0/3801010436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03c01538a0 0x7f03c01f3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:19.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.115+0000 7f03c8646640 1 --2- 172.21.15.89:0/3801010436 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03c00cf160 0x7f03c015c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:19.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.115+0000 7f03c8646640 1 --2- 172.21.15.89:0/3801010436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03c00ca0e0 0x7f03c01efe00 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:19.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.115+0000 7f03c8646640 1 -- 172.21.15.89:0/3801010436 >> 172.21.15.89:0/3801010436 conn(0x7f03c006bfe0 msgr2=0x7f03c006c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:19.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.115+0000 7f03c8646640 1 -- 172.21.15.89:0/3801010436 shutdown_connections 2023-10-30T12:29:19.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.115+0000 7f03c8646640 1 -- 172.21.15.89:0/3801010436 wait complete. 2023-10-30T12:29:19.138 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2023-10-30T12:29:18.928978+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596921,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":288,"num_write_kb":2458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":230,"ondisk_log_size":230,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219196,"kb_used_data":3068,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755524,"statfs":{"total":773094113280,"available":772869656576,"internally_reserved":0,"allocated":3141632,"data_stored":1822082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12741,"internal_metadata":221236795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2737,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004369"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028790+0000","last_change":"2023-10-30T12:28:50.979797+0000","last_active":"2023-10-30T12:28:53.028790+0000","last_peered":"2023-10-30T12:28:53.028790+0000","last_clean":"2023-10-30T12:28:53.028790+0000","last_became_active":"2023-10-30T12:28:50.979351+0000","last_became_peered":"2023-10-30T12:28:50.979351+0000","last_unstale":"2023-10-30T12:28:53.028790+0000","last_undegraded":"2023-10-30T12:28:53.028790+0000","last_fullsized":"2023-10-30T12:28:53.028790+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:53:00.556539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.932043+0000","last_change":"2023-10-30T12:28:50.929403+0000","last_active":"2023-10-30T12:28:51.932043+0000","last_peered":"2023-10-30T12:28:51.932043+0000","last_clean":"2023-10-30T12:28:51.932043+0000","last_became_active":"2023-10-30T12:28:50.929133+0000","last_became_peered":"2023-10-30T12:28:50.929133+0000","last_unstale":"2023-10-30T12:28:51.932043+0000","last_undegraded":"2023-10-30T12:28:51.932043+0000","last_fullsized":"2023-10-30T12:28:51.932043+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:12:57.327440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579508+0000","last_change":"2023-10-30T12:28:50.934201+0000","last_active":"2023-10-30T12:28:53.579508+0000","last_peered":"2023-10-30T12:28:53.579508+0000","last_clean":"2023-10-30T12:28:53.579508+0000","last_became_active":"2023-10-30T12:28:50.933896+0000","last_became_peered":"2023-10-30T12:28:50.933896+0000","last_unstale":"2023-10-30T12:28:53.579508+0000","last_undegraded":"2023-10-30T12:28:53.579508+0000","last_fullsized":"2023-10-30T12:28:53.579508+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:20:46.597550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581831+0000","last_change":"2023-10-30T12:28:50.969193+0000","last_active":"2023-10-30T12:28:53.581831+0000","last_peered":"2023-10-30T12:28:53.581831+0000","last_clean":"2023-10-30T12:28:53.581831+0000","last_became_active":"2023-10-30T12:28:50.967774+0000","last_became_peered":"2023-10-30T12:28:50.967774+0000","last_unstale":"2023-10-30T12:28:53.581831+0000","last_undegraded":"2023-10-30T12:28:53.581831+0000","last_fullsized":"2023-10-30T12:28:53.581831+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:23:07.329372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028482+0000","last_change":"2023-10-30T12:28:50.973751+0000","last_active":"2023-10-30T12:28:53.028482+0000","last_peered":"2023-10-30T12:28:53.028482+0000","last_clean":"2023-10-30T12:28:53.028482+0000","last_became_active":"2023-10-30T12:28:50.973132+0000","last_became_peered":"2023-10-30T12:28:50.973132+0000","last_unstale":"2023-10-30T12:28:53.028482+0000","last_undegraded":"2023-10-30T12:28:53.028482+0000","last_fullsized":"2023-10-30T12:28:53.028482+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:43:58.223555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581257+0000","last_change":"2023-10-30T12:28:50.968823+0000","last_active":"2023-10-30T12:28:53.581257+0000","last_peered":"2023-10-30T12:28:53.581257+0000","last_clean":"2023-10-30T12:28:53.581257+0000","last_became_active":"2023-10-30T12:28:50.967910+0000","last_became_peered":"2023-10-30T12:28:50.967910+0000","last_unstale":"2023-10-30T12:28:53.581257+0000","last_undegraded":"2023-10-30T12:28:53.581257+0000","last_fullsized":"2023-10-30T12:28:53.581257+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:34:36.882757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028044+0000","last_change":"2023-10-30T12:28:50.974164+0000","last_active":"2023-10-30T12:28:53.028044+0000","last_peered":"2023-10-30T12:28:53.028044+0000","last_clean":"2023-10-30T12:28:53.028044+0000","last_became_active":"2023-10-30T12:28:50.973288+0000","last_became_peered":"2023-10-30T12:28:50.973288+0000","last_unstale":"2023-10-30T12:28:53.028044+0000","last_undegraded":"2023-10-30T12:28:53.028044+0000","last_fullsized":"2023-10-30T12:28:53.028044+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:09:10.450492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.578880+0000","last_change":"2023-10-30T12:28:50.955287+0000","last_active":"2023-10-30T12:28:53.578880+0000","last_peered":"2023-10-30T12:28:53.578880+0000","last_clean":"2023-10-30T12:28:53.578880+0000","last_became_active":"2023-10-30T12:28:50.954988+0000","last_became_peered":"2023-10-30T12:28:50.954988+0000","last_unstale":"2023-10-30T12:28:53.578880+0000","last_undegraded":"2023-10-30T12:28:53.578880+0000","last_fullsized":"2023-10-30T12:28:53.578880+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:36:30.846920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.029019+0000","last_change":"2023-10-30T12:28:50.977144+0000","last_active":"2023-10-30T12:28:53.029019+0000","last_peered":"2023-10-30T12:28:53.029019+0000","last_clean":"2023-10-30T12:28:53.029019+0000","last_became_active":"2023-10-30T12:28:50.976564+0000","last_became_peered":"2023-10-30T12:28:50.976564+0000","last_unstale":"2023-10-30T12:28:53.029019+0000","last_undegraded":"2023-10-30T12:28:53.029019+0000","last_fullsized":"2023-10-30T12:28:53.029019+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:37:51.341935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.938542+0000","last_change":"2023-10-30T12:28:50.952564+0000","last_active":"2023-10-30T12:28:51.938542+0000","last_peered":"2023-10-30T12:28:51.938542+0000","last_clean":"2023-10-30T12:28:51.938542+0000","last_became_active":"2023-10-30T12:28:50.951809+0000","last_became_peered":"2023-10-30T12:28:50.951809+0000","last_unstale":"2023-10-30T12:28:51.938542+0000","last_undegraded":"2023-10-30T12:28:51.938542+0000","last_fullsized":"2023-10-30T12:28:51.938542+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:34:29.771577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.937215+0000","last_change":"2023-10-30T12:28:50.970779+0000","last_active":"2023-10-30T12:28:51.937215+0000","last_peered":"2023-10-30T12:28:51.937215+0000","last_clean":"2023-10-30T12:28:51.937215+0000","last_became_active":"2023-10-30T12:28:50.969877+0000","last_became_peered":"2023-10-30T12:28:50.969877+0000","last_unstale":"2023-10-30T12:28:51.937215+0000","last_undegraded":"2023-10-30T12:28:51.937215+0000","last_fullsized":"2023-10-30T12:28:51.937215+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:10.539915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.940164+0000","last_change":"2023-10-30T12:28:50.944452+0000","last_active":"2023-10-30T12:28:51.940164+0000","last_peered":"2023-10-30T12:28:51.940164+0000","last_clean":"2023-10-30T12:28:51.940164+0000","last_became_active":"2023-10-30T12:28:50.943787+0000","last_became_peered":"2023-10-30T12:28:50.943787+0000","last_unstale":"2023-10-30T12:28:51.940164+0000","last_undegraded":"2023-10-30T12:28:51.940164+0000","last_fullsized":"2023-10-30T12:28:51.940164+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:30:34.585298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579240+0000","last_change":"2023-10-30T12:28:50.938181+0000","last_active":"2023-10-30T12:28:53.579240+0000","last_peered":"2023-10-30T12:28:53.579240+0000","last_clean":"2023-10-30T12:28:53.579240+0000","last_became_active":"2023-10-30T12:28:50.937906+0000","last_became_peered":"2023-10-30T12:28:50.937906+0000","last_unstale":"2023-10-30T12:28:53.579240+0000","last_undegraded":"2023-10-30T12:28:53.579240+0000","last_fullsized":"2023-10-30T12:28:53.579240+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:41:19.990452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579303+0000","last_change":"2023-10-30T12:28:50.934085+0000","last_active":"2023-10-30T12:28:53.579303+0000","last_peered":"2023-10-30T12:28:53.579303+0000","last_clean":"2023-10-30T12:28:53.579303+0000","last_became_active":"2023-10-30T12:28:50.932753+0000","last_became_peered":"2023-10-30T12:28:50.932753+0000","last_unstale":"2023-10-30T12:28:53.579303+0000","last_undegraded":"2023-10-30T12:28:53.579303+0000","last_fullsized":"2023-10-30T12:28:53.579303+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:23:26.736283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931710+0000","last_change":"2023-10-30T12:28:50.943343+0000","last_active":"2023-10-30T12:28:51.931710+0000","last_peered":"2023-10-30T12:28:51.931710+0000","last_clean":"2023-10-30T12:28:51.931710+0000","last_became_active":"2023-10-30T12:28:50.943011+0000","last_became_peered":"2023-10-30T12:28:50.943011+0000","last_unstale":"2023-10-30T12:28:51.931710+0000","last_undegraded":"2023-10-30T12:28:51.931710+0000","last_fullsized":"2023-10-30T12:28:51.931710+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:59:35.881115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581157+0000","last_change":"2023-10-30T12:28:50.953573+0000","last_active":"2023-10-30T12:28:53.581157+0000","last_peered":"2023-10-30T12:28:53.581157+0000","last_clean":"2023-10-30T12:28:53.581157+0000","last_became_active":"2023-10-30T12:28:50.952744+0000","last_became_peered":"2023-10-30T12:28:50.952744+0000","last_unstale":"2023-10-30T12:28:53.581157+0000","last_undegraded":"2023-10-30T12:28:53.581157+0000","last_fullsized":"2023-10-30T12:28:53.581157+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:41:16.209759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.582342+0000","last_change":"2023-10-30T12:28:50.975156+0000","last_active":"2023-10-30T12:28:53.582342+0000","last_peered":"2023-10-30T12:28:53.582342+0000","last_clean":"2023-10-30T12:28:53.582342+0000","last_became_active":"2023-10-30T12:28:50.974576+0000","last_became_peered":"2023-10-30T12:28:50.974576+0000","last_unstale":"2023-10-30T12:28:53.582342+0000","last_undegraded":"2023-10-30T12:28:53.582342+0000","last_fullsized":"2023-10-30T12:28:53.582342+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:46.159555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.131817+0000","last_change":"2023-10-30T12:28:50.937685+0000","last_active":"2023-10-30T12:28:53.131817+0000","last_peered":"2023-10-30T12:28:53.131817+0000","last_clean":"2023-10-30T12:28:53.131817+0000","last_became_active":"2023-10-30T12:28:50.937384+0000","last_became_peered":"2023-10-30T12:28:50.937384+0000","last_unstale":"2023-10-30T12:28:53.131817+0000","last_undegraded":"2023-10-30T12:28:53.131817+0000","last_fullsized":"2023-10-30T12:28:53.131817+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:16:29.181523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.029181+0000","last_change":"2023-10-30T12:28:50.974918+0000","last_active":"2023-10-30T12:28:53.029181+0000","last_peered":"2023-10-30T12:28:53.029181+0000","last_clean":"2023-10-30T12:28:53.029181+0000","last_became_active":"2023-10-30T12:28:50.973369+0000","last_became_peered":"2023-10-30T12:28:50.973369+0000","last_unstale":"2023-10-30T12:28:53.029181+0000","last_undegraded":"2023-10-30T12:28:53.029181+0000","last_fullsized":"2023-10-30T12:28:53.029181+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:44:04.468611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.937226+0000","last_change":"2023-10-30T12:28:50.955835+0000","last_active":"2023-10-30T12:28:51.937226+0000","last_peered":"2023-10-30T12:28:51.937226+0000","last_clean":"2023-10-30T12:28:51.937226+0000","last_became_active":"2023-10-30T12:28:50.954831+0000","last_became_peered":"2023-10-30T12:28:50.954831+0000","last_unstale":"2023-10-30T12:28:51.937226+0000","last_undegraded":"2023-10-30T12:28:51.937226+0000","last_fullsized":"2023-10-30T12:28:51.937226+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:56:25.554582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.580290+0000","last_change":"2023-10-30T12:28:50.962417+0000","last_active":"2023-10-30T12:28:53.580290+0000","last_peered":"2023-10-30T12:28:53.580290+0000","last_clean":"2023-10-30T12:28:53.580290+0000","last_became_active":"2023-10-30T12:28:50.962065+0000","last_became_peered":"2023-10-30T12:28:50.962065+0000","last_unstale":"2023-10-30T12:28:53.580290+0000","last_undegraded":"2023-10-30T12:28:53.580290+0000","last_fullsized":"2023-10-30T12:28:53.580290+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:13:52.455824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581454+0000","last_change":"2023-10-30T12:28:50.968992+0000","last_active":"2023-10-30T12:28:53.581454+0000","last_peered":"2023-10-30T12:28:53.581454+0000","last_clean":"2023-10-30T12:28:53.581454+0000","last_became_active":"2023-10-30T12:28:50.967651+0000","last_became_peered":"2023-10-30T12:28:50.967651+0000","last_unstale":"2023-10-30T12:28:53.581454+0000","last_undegraded":"2023-10-30T12:28:53.581454+0000","last_fullsized":"2023-10-30T12:28:53.581454+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:39:31.893024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579014+0000","last_change":"2023-10-30T12:28:50.962609+0000","last_active":"2023-10-30T12:28:53.579014+0000","last_peered":"2023-10-30T12:28:53.579014+0000","last_clean":"2023-10-30T12:28:53.579014+0000","last_became_active":"2023-10-30T12:28:50.962290+0000","last_became_peered":"2023-10-30T12:28:50.962290+0000","last_unstale":"2023-10-30T12:28:53.579014+0000","last_undegraded":"2023-10-30T12:28:53.579014+0000","last_fullsized":"2023-10-30T12:28:53.579014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:44:35.484971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581381+0000","last_change":"2023-10-30T12:28:50.966653+0000","last_active":"2023-10-30T12:28:53.581381+0000","last_peered":"2023-10-30T12:28:53.581381+0000","last_clean":"2023-10-30T12:28:53.581381+0000","last_became_active":"2023-10-30T12:28:50.965713+0000","last_became_peered":"2023-10-30T12:28:50.965713+0000","last_unstale":"2023-10-30T12:28:53.581381+0000","last_undegraded":"2023-10-30T12:28:53.581381+0000","last_fullsized":"2023-10-30T12:28:53.581381+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:10:31.600981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579351+0000","last_change":"2023-10-30T12:28:50.951300+0000","last_active":"2023-10-30T12:28:53.579351+0000","last_peered":"2023-10-30T12:28:53.579351+0000","last_clean":"2023-10-30T12:28:53.579351+0000","last_became_active":"2023-10-30T12:28:50.951026+0000","last_became_peered":"2023-10-30T12:28:50.951026+0000","last_unstale":"2023-10-30T12:28:53.579351+0000","last_undegraded":"2023-10-30T12:28:53.579351+0000","last_fullsized":"2023-10-30T12:28:53.579351+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:44:58.872457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.938879+0000","last_change":"2023-10-30T12:28:50.963507+0000","last_active":"2023-10-30T12:28:51.938879+0000","last_peered":"2023-10-30T12:28:51.938879+0000","last_clean":"2023-10-30T12:28:51.938879+0000","last_became_active":"2023-10-30T12:28:50.962484+0000","last_became_peered":"2023-10-30T12:28:50.962484+0000","last_unstale":"2023-10-30T12:28:51.938879+0000","last_undegraded":"2023-10-30T12:28:51.938879+0000","last_fullsized":"2023-10-30T12:28:51.938879+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:11:24.160220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931933+0000","last_change":"2023-10-30T12:28:50.933362+0000","last_active":"2023-10-30T12:28:51.931933+0000","last_peered":"2023-10-30T12:28:51.931933+0000","last_clean":"2023-10-30T12:28:51.931933+0000","last_became_active":"2023-10-30T12:28:50.933093+0000","last_became_peered":"2023-10-30T12:28:50.933093+0000","last_unstale":"2023-10-30T12:28:51.931933+0000","last_undegraded":"2023-10-30T12:28:51.931933+0000","last_fullsized":"2023-10-30T12:28:51.931933+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:54:00.852482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579508+0000","last_change":"2023-10-30T12:28:50.952857+0000","last_active":"2023-10-30T12:28:53.579508+0000","last_peered":"2023-10-30T12:28:53.579508+0000","last_clean":"2023-10-30T12:28:53.579508+0000","last_became_active":"2023-10-30T12:28:50.952531+0000","last_became_peered":"2023-10-30T12:28:50.952531+0000","last_unstale":"2023-10-30T12:28:53.579508+0000","last_undegraded":"2023-10-30T12:28:53.579508+0000","last_fullsized":"2023-10-30T12:28:53.579508+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:01:03.980850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.580366+0000","last_change":"2023-10-30T12:28:50.933408+0000","last_active":"2023-10-30T12:28:53.580366+0000","last_peered":"2023-10-30T12:28:53.580366+0000","last_clean":"2023-10-30T12:28:53.580366+0000","last_became_active":"2023-10-30T12:28:50.933124+0000","last_became_peered":"2023-10-30T12:28:50.933124+0000","last_unstale":"2023-10-30T12:28:53.580366+0000","last_undegraded":"2023-10-30T12:28:53.580366+0000","last_fullsized":"2023-10-30T12:28:53.580366+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:53:50.772420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581630+0000","last_change":"2023-10-30T12:28:50.966388+0000","last_active":"2023-10-30T12:28:53.581630+0000","last_peered":"2023-10-30T12:28:53.581630+0000","last_clean":"2023-10-30T12:28:53.581630+0000","last_became_active":"2023-10-30T12:28:50.965639+0000","last_became_peered":"2023-10-30T12:28:50.965639+0000","last_unstale":"2023-10-30T12:28:53.581630+0000","last_undegraded":"2023-10-30T12:28:53.581630+0000","last_fullsized":"2023-10-30T12:28:53.581630+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:55:46.506378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.131583+0000","last_change":"2023-10-30T12:28:50.955391+0000","last_active":"2023-10-30T12:28:53.131583+0000","last_peered":"2023-10-30T12:28:53.131583+0000","last_clean":"2023-10-30T12:28:53.131583+0000","last_became_active":"2023-10-30T12:28:50.955084+0000","last_became_peered":"2023-10-30T12:28:50.955084+0000","last_unstale":"2023-10-30T12:28:53.131583+0000","last_undegraded":"2023-10-30T12:28:53.131583+0000","last_fullsized":"2023-10-30T12:28:53.131583+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:08:52.389085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.936598+0000","last_change":"2023-10-30T12:28:50.945609+0000","last_active":"2023-10-30T12:28:51.936598+0000","last_peered":"2023-10-30T12:28:51.936598+0000","last_clean":"2023-10-30T12:28:51.936598+0000","last_became_active":"2023-10-30T12:28:50.944775+0000","last_became_peered":"2023-10-30T12:28:50.944775+0000","last_unstale":"2023-10-30T12:28:51.936598+0000","last_undegraded":"2023-10-30T12:28:51.936598+0000","last_fullsized":"2023-10-30T12:28:51.936598+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:08:55.718193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.939146+0000","last_change":"2023-10-30T12:28:50.941378+0000","last_active":"2023-10-30T12:28:51.939146+0000","last_peered":"2023-10-30T12:28:51.939146+0000","last_clean":"2023-10-30T12:28:51.939146+0000","last_became_active":"2023-10-30T12:28:50.940567+0000","last_became_peered":"2023-10-30T12:28:50.940567+0000","last_unstale":"2023-10-30T12:28:51.939146+0000","last_undegraded":"2023-10-30T12:28:51.939146+0000","last_fullsized":"2023-10-30T12:28:51.939146+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:31:23.772343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'5","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.435783+0000","last_change":"2023-10-30T12:28:50.395670+0000","last_active":"2023-10-30T12:29:01.435783+0000","last_peered":"2023-10-30T12:29:01.435783+0000","last_clean":"2023-10-30T12:29:01.435783+0000","last_became_active":"2023-10-30T12:28:50.395382+0000","last_became_peered":"2023-10-30T12:28:50.395382+0000","last_unstale":"2023-10-30T12:29:01.435783+0000","last_undegraded":"2023-10-30T12:29:01.435783+0000","last_fullsized":"2023-10-30T12:29:01.435783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:14.858121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.580427+0000","last_change":"2023-10-30T12:28:50.953202+0000","last_active":"2023-10-30T12:28:53.580427+0000","last_peered":"2023-10-30T12:28:53.580427+0000","last_clean":"2023-10-30T12:28:53.580427+0000","last_became_active":"2023-10-30T12:28:50.952613+0000","last_became_peered":"2023-10-30T12:28:50.952613+0000","last_unstale":"2023-10-30T12:28:53.580427+0000","last_undegraded":"2023-10-30T12:28:53.580427+0000","last_fullsized":"2023-10-30T12:28:53.580427+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:38:56.947997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.959082+0000","last_change":"2023-10-30T12:28:50.384598+0000","last_active":"2023-10-30T12:29:06.959082+0000","last_peered":"2023-10-30T12:29:06.959082+0000","last_clean":"2023-10-30T12:29:06.959082+0000","last_became_active":"2023-10-30T12:28:50.383898+0000","last_became_peered":"2023-10-30T12:28:50.383898+0000","last_unstale":"2023-10-30T12:29:06.959082+0000","last_undegraded":"2023-10-30T12:29:06.959082+0000","last_fullsized":"2023-10-30T12:29:06.959082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:09:50.290004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.954822+0000","last_change":"2023-10-30T12:28:50.386193+0000","last_active":"2023-10-30T12:29:06.954822+0000","last_peered":"2023-10-30T12:29:06.954822+0000","last_clean":"2023-10-30T12:29:06.954822+0000","last_became_active":"2023-10-30T12:28:50.385963+0000","last_became_peered":"2023-10-30T12:28:50.385963+0000","last_unstale":"2023-10-30T12:29:06.954822+0000","last_undegraded":"2023-10-30T12:29:06.954822+0000","last_fullsized":"2023-10-30T12:29:06.954822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:00:55.603212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.578966+0000","last_change":"2023-10-30T12:28:50.936038+0000","last_active":"2023-10-30T12:28:53.578966+0000","last_peered":"2023-10-30T12:28:53.578966+0000","last_clean":"2023-10-30T12:28:53.578966+0000","last_became_active":"2023-10-30T12:28:50.935764+0000","last_became_peered":"2023-10-30T12:28:50.935764+0000","last_unstale":"2023-10-30T12:28:53.578966+0000","last_undegraded":"2023-10-30T12:28:53.578966+0000","last_fullsized":"2023-10-30T12:28:53.578966+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:23:01.597158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581945+0000","last_change":"2023-10-30T12:28:50.392213+0000","last_active":"2023-10-30T12:28:53.581945+0000","last_peered":"2023-10-30T12:28:53.581945+0000","last_clean":"2023-10-30T12:28:53.581945+0000","last_became_active":"2023-10-30T12:28:50.391494+0000","last_became_peered":"2023-10-30T12:28:50.391494+0000","last_unstale":"2023-10-30T12:28:53.581945+0000","last_undegraded":"2023-10-30T12:28:53.581945+0000","last_fullsized":"2023-10-30T12:28:53.581945+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:38:37.301840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931619+0000","last_change":"2023-10-30T12:28:50.933319+0000","last_active":"2023-10-30T12:28:51.931619+0000","last_peered":"2023-10-30T12:28:51.931619+0000","last_clean":"2023-10-30T12:28:51.931619+0000","last_became_active":"2023-10-30T12:28:50.933062+0000","last_became_peered":"2023-10-30T12:28:50.933062+0000","last_unstale":"2023-10-30T12:28:51.931619+0000","last_undegraded":"2023-10-30T12:28:51.931619+0000","last_fullsized":"2023-10-30T12:28:51.931619+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:26:48.475038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.424753+0000","last_change":"2023-10-30T12:28:50.394368+0000","last_active":"2023-10-30T12:29:01.424753+0000","last_peered":"2023-10-30T12:29:01.424753+0000","last_clean":"2023-10-30T12:29:01.424753+0000","last_became_active":"2023-10-30T12:28:50.393686+0000","last_became_peered":"2023-10-30T12:28:50.393686+0000","last_unstale":"2023-10-30T12:29:01.424753+0000","last_undegraded":"2023-10-30T12:29:01.424753+0000","last_fullsized":"2023-10-30T12:29:01.424753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:43:17.726462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.582247+0000","last_change":"2023-10-30T12:28:50.963703+0000","last_active":"2023-10-30T12:28:53.582247+0000","last_peered":"2023-10-30T12:28:53.582247+0000","last_clean":"2023-10-30T12:28:53.582247+0000","last_became_active":"2023-10-30T12:28:50.963105+0000","last_became_peered":"2023-10-30T12:28:50.963105+0000","last_unstale":"2023-10-30T12:28:53.582247+0000","last_undegraded":"2023-10-30T12:28:53.582247+0000","last_fullsized":"2023-10-30T12:28:53.582247+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:32.423943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.420755+0000","last_change":"2023-10-30T12:28:50.397679+0000","last_active":"2023-10-30T12:29:01.420755+0000","last_peered":"2023-10-30T12:29:01.420755+0000","last_clean":"2023-10-30T12:29:01.420755+0000","last_became_active":"2023-10-30T12:28:50.397041+0000","last_became_peered":"2023-10-30T12:28:50.397041+0000","last_unstale":"2023-10-30T12:29:01.420755+0000","last_undegraded":"2023-10-30T12:29:01.420755+0000","last_fullsized":"2023-10-30T12:29:01.420755+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:52:22.564616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.580443+0000","last_change":"2023-10-30T12:28:50.966838+0000","last_active":"2023-10-30T12:28:53.580443+0000","last_peered":"2023-10-30T12:28:53.580443+0000","last_clean":"2023-10-30T12:28:53.580443+0000","last_became_active":"2023-10-30T12:28:50.966569+0000","last_became_peered":"2023-10-30T12:28:50.966569+0000","last_unstale":"2023-10-30T12:28:53.580443+0000","last_undegraded":"2023-10-30T12:28:53.580443+0000","last_fullsized":"2023-10-30T12:28:53.580443+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:44:33.875449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:56.364393+0000","last_change":"2023-10-30T12:28:50.391527+0000","last_active":"2023-10-30T12:28:56.364393+0000","last_peered":"2023-10-30T12:28:56.364393+0000","last_clean":"2023-10-30T12:28:56.364393+0000","last_became_active":"2023-10-30T12:28:50.391260+0000","last_became_peered":"2023-10-30T12:28:50.391260+0000","last_unstale":"2023-10-30T12:28:56.364393+0000","last_undegraded":"2023-10-30T12:28:56.364393+0000","last_fullsized":"2023-10-30T12:28:56.364393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:29:34.772490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.940211+0000","last_change":"2023-10-30T12:28:50.975441+0000","last_active":"2023-10-30T12:28:51.940211+0000","last_peered":"2023-10-30T12:28:51.940211+0000","last_clean":"2023-10-30T12:28:51.940211+0000","last_became_active":"2023-10-30T12:28:50.974843+0000","last_became_peered":"2023-10-30T12:28:50.974843+0000","last_unstale":"2023-10-30T12:28:51.940211+0000","last_undegraded":"2023-10-30T12:28:51.940211+0000","last_fullsized":"2023-10-30T12:28:51.940211+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:06:12.060752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"22'11","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:15.025529+0000","last_change":"2023-10-30T12:28:50.374838+0000","last_active":"2023-10-30T12:29:15.025529+0000","last_peered":"2023-10-30T12:29:15.025529+0000","last_clean":"2023-10-30T12:29:15.025529+0000","last_became_active":"2023-10-30T12:28:50.373581+0000","last_became_peered":"2023-10-30T12:28:50.373581+0000","last_unstale":"2023-10-30T12:29:15.025529+0000","last_undegraded":"2023-10-30T12:29:15.025529+0000","last_fullsized":"2023-10-30T12:29:15.025529+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:14:51.506745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2685,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.029046+0000","last_change":"2023-10-30T12:28:50.974012+0000","last_active":"2023-10-30T12:28:53.029046+0000","last_peered":"2023-10-30T12:28:53.029046+0000","last_clean":"2023-10-30T12:28:53.029046+0000","last_became_active":"2023-10-30T12:28:50.973269+0000","last_became_peered":"2023-10-30T12:28:50.973269+0000","last_unstale":"2023-10-30T12:28:53.029046+0000","last_undegraded":"2023-10-30T12:28:53.029046+0000","last_fullsized":"2023-10-30T12:28:53.029046+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:19:21.993898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"22'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.957225+0000","last_change":"2023-10-30T12:28:50.366590+0000","last_active":"2023-10-30T12:29:06.957225+0000","last_peered":"2023-10-30T12:29:06.957225+0000","last_clean":"2023-10-30T12:29:06.957225+0000","last_became_active":"2023-10-30T12:28:50.366269+0000","last_became_peered":"2023-10-30T12:28:50.366269+0000","last_unstale":"2023-10-30T12:29:06.957225+0000","last_undegraded":"2023-10-30T12:29:06.957225+0000","last_fullsized":"2023-10-30T12:29:06.957225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:15:11.963438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579425+0000","last_change":"2023-10-30T12:28:50.962058+0000","last_active":"2023-10-30T12:28:53.579425+0000","last_peered":"2023-10-30T12:28:53.579425+0000","last_clean":"2023-10-30T12:28:53.579425+0000","last_became_active":"2023-10-30T12:28:50.961805+0000","last_became_peered":"2023-10-30T12:28:50.961805+0000","last_unstale":"2023-10-30T12:28:53.579425+0000","last_undegraded":"2023-10-30T12:28:53.579425+0000","last_fullsized":"2023-10-30T12:28:53.579425+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:45:16.373386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.427282+0000","last_change":"2023-10-30T12:28:50.397244+0000","last_active":"2023-10-30T12:29:01.427282+0000","last_peered":"2023-10-30T12:29:01.427282+0000","last_clean":"2023-10-30T12:29:01.427282+0000","last_became_active":"2023-10-30T12:28:50.396727+0000","last_became_peered":"2023-10-30T12:28:50.396727+0000","last_unstale":"2023-10-30T12:29:01.427282+0000","last_undegraded":"2023-10-30T12:29:01.427282+0000","last_fullsized":"2023-10-30T12:29:01.427282+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:33.493343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028618+0000","last_change":"2023-10-30T12:28:50.981498+0000","last_active":"2023-10-30T12:28:53.028618+0000","last_peered":"2023-10-30T12:28:53.028618+0000","last_clean":"2023-10-30T12:28:53.028618+0000","last_became_active":"2023-10-30T12:28:50.981063+0000","last_became_peered":"2023-10-30T12:28:50.981063+0000","last_unstale":"2023-10-30T12:28:53.028618+0000","last_undegraded":"2023-10-30T12:28:53.028618+0000","last_fullsized":"2023-10-30T12:28:53.028618+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:55:17.971880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.963531+0000","last_change":"2023-10-30T12:28:50.383880+0000","last_active":"2023-10-30T12:29:06.963531+0000","last_peered":"2023-10-30T12:29:06.963531+0000","last_clean":"2023-10-30T12:29:06.963531+0000","last_became_active":"2023-10-30T12:28:50.383570+0000","last_became_peered":"2023-10-30T12:28:50.383570+0000","last_unstale":"2023-10-30T12:29:06.963531+0000","last_undegraded":"2023-10-30T12:29:06.963531+0000","last_fullsized":"2023-10-30T12:29:06.963531+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:19:42.047520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":846,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.936983+0000","last_change":"2023-10-30T12:28:50.950312+0000","last_active":"2023-10-30T12:28:51.936983+0000","last_peered":"2023-10-30T12:28:51.936983+0000","last_clean":"2023-10-30T12:28:51.936983+0000","last_became_active":"2023-10-30T12:28:50.949485+0000","last_became_peered":"2023-10-30T12:28:50.949485+0000","last_unstale":"2023-10-30T12:28:51.936983+0000","last_undegraded":"2023-10-30T12:28:51.936983+0000","last_fullsized":"2023-10-30T12:28:51.936983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:13:04.669915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"22'15","reported_seq":27,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.968260+0000","last_change":"2023-10-30T12:28:50.362979+0000","last_active":"2023-10-30T12:29:06.968260+0000","last_peered":"2023-10-30T12:29:06.968260+0000","last_clean":"2023-10-30T12:29:06.968260+0000","last_became_active":"2023-10-30T12:28:50.362624+0000","last_became_peered":"2023-10-30T12:28:50.362624+0000","last_unstale":"2023-10-30T12:29:06.968260+0000","last_undegraded":"2023-10-30T12:29:06.968260+0000","last_fullsized":"2023-10-30T12:29:06.968260+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:01:13.111816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028233+0000","last_change":"2023-10-30T12:28:50.979815+0000","last_active":"2023-10-30T12:28:53.028233+0000","last_peered":"2023-10-30T12:28:53.028233+0000","last_clean":"2023-10-30T12:28:53.028233+0000","last_became_active":"2023-10-30T12:28:50.979344+0000","last_became_peered":"2023-10-30T12:28:50.979344+0000","last_unstale":"2023-10-30T12:28:53.028233+0000","last_undegraded":"2023-10-30T12:28:53.028233+0000","last_fullsized":"2023-10-30T12:28:53.028233+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:35:37.018282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:56.363098+0000","last_change":"2023-10-30T12:28:50.382175+0000","last_active":"2023-10-30T12:28:56.363098+0000","last_peered":"2023-10-30T12:28:56.363098+0000","last_clean":"2023-10-30T12:28:56.363098+0000","last_became_active":"2023-10-30T12:28:50.381125+0000","last_became_peered":"2023-10-30T12:28:50.381125+0000","last_unstale":"2023-10-30T12:28:56.363098+0000","last_undegraded":"2023-10-30T12:28:56.363098+0000","last_fullsized":"2023-10-30T12:28:56.363098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:36:02.652665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028807+0000","last_change":"2023-10-30T12:28:50.978034+0000","last_active":"2023-10-30T12:28:53.028807+0000","last_peered":"2023-10-30T12:28:53.028807+0000","last_clean":"2023-10-30T12:28:53.028807+0000","last_became_active":"2023-10-30T12:28:50.977585+0000","last_became_peered":"2023-10-30T12:28:50.977585+0000","last_unstale":"2023-10-30T12:28:53.028807+0000","last_undegraded":"2023-10-30T12:28:53.028807+0000","last_fullsized":"2023-10-30T12:28:53.028807+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:42:56.806184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:56.364009+0000","last_change":"2023-10-30T12:28:50.367258+0000","last_active":"2023-10-30T12:28:56.364009+0000","last_peered":"2023-10-30T12:28:56.364009+0000","last_clean":"2023-10-30T12:28:56.364009+0000","last_became_active":"2023-10-30T12:28:50.366974+0000","last_became_peered":"2023-10-30T12:28:50.366974+0000","last_unstale":"2023-10-30T12:28:56.364009+0000","last_undegraded":"2023-10-30T12:28:56.364009+0000","last_fullsized":"2023-10-30T12:28:56.364009+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:14:48.823902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.029189+0000","last_change":"2023-10-30T12:28:50.981784+0000","last_active":"2023-10-30T12:28:53.029189+0000","last_peered":"2023-10-30T12:28:53.029189+0000","last_clean":"2023-10-30T12:28:53.029189+0000","last_became_active":"2023-10-30T12:28:50.981193+0000","last_became_peered":"2023-10-30T12:28:50.981193+0000","last_unstale":"2023-10-30T12:28:53.029189+0000","last_undegraded":"2023-10-30T12:28:53.029189+0000","last_fullsized":"2023-10-30T12:28:53.029189+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:36:17.384732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:52.280239+0000","last_change":"2023-10-30T12:28:50.368011+0000","last_active":"2023-10-30T12:28:52.280239+0000","last_peered":"2023-10-30T12:28:52.280239+0000","last_clean":"2023-10-30T12:28:52.280239+0000","last_became_active":"2023-10-30T12:28:50.367730+0000","last_became_peered":"2023-10-30T12:28:50.367730+0000","last_unstale":"2023-10-30T12:28:52.280239+0000","last_undegraded":"2023-10-30T12:28:52.280239+0000","last_fullsized":"2023-10-30T12:28:52.280239+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:59:41.946550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.582218+0000","last_change":"2023-10-30T12:28:50.968262+0000","last_active":"2023-10-30T12:28:53.582218+0000","last_peered":"2023-10-30T12:28:53.582218+0000","last_clean":"2023-10-30T12:28:53.582218+0000","last_became_active":"2023-10-30T12:28:50.966246+0000","last_became_peered":"2023-10-30T12:28:50.966246+0000","last_unstale":"2023-10-30T12:28:53.582218+0000","last_undegraded":"2023-10-30T12:28:53.582218+0000","last_fullsized":"2023-10-30T12:28:53.582218+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:00:01.991935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.962658+0000","last_change":"2023-10-30T12:28:50.394060+0000","last_active":"2023-10-30T12:29:06.962658+0000","last_peered":"2023-10-30T12:29:06.962658+0000","last_clean":"2023-10-30T12:29:06.962658+0000","last_became_active":"2023-10-30T12:28:50.393642+0000","last_became_peered":"2023-10-30T12:28:50.393642+0000","last_unstale":"2023-10-30T12:29:06.962658+0000","last_undegraded":"2023-10-30T12:29:06.962658+0000","last_fullsized":"2023-10-30T12:29:06.962658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:21.643882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028738+0000","last_change":"2023-10-30T12:28:50.976814+0000","last_active":"2023-10-30T12:28:53.028738+0000","last_peered":"2023-10-30T12:28:53.028738+0000","last_clean":"2023-10-30T12:28:53.028738+0000","last_became_active":"2023-10-30T12:28:50.976343+0000","last_became_peered":"2023-10-30T12:28:50.976343+0000","last_unstale":"2023-10-30T12:28:53.028738+0000","last_undegraded":"2023-10-30T12:28:53.028738+0000","last_fullsized":"2023-10-30T12:28:53.028738+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:33:23.491998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.946067+0000","last_change":"2023-10-30T12:28:41.203990+0000","last_active":"2023-10-30T12:28:50.946067+0000","last_peered":"2023-10-30T12:28:50.946067+0000","last_clean":"2023-10-30T12:28:50.946067+0000","last_became_active":"2023-10-30T12:28:41.201796+0000","last_became_peered":"2023-10-30T12:28:41.201796+0000","last_unstale":"2023-10-30T12:28:50.946067+0000","last_undegraded":"2023-10-30T12:28:50.946067+0000","last_fullsized":"2023-10-30T12:28:50.946067+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:39.218257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:39.218257+0000","last_clean_scrub_stamp":"2023-10-30T12:28:39.218257+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:21:00.605462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.426165+0000","last_change":"2023-10-30T12:28:50.372651+0000","last_active":"2023-10-30T12:29:01.426165+0000","last_peered":"2023-10-30T12:29:01.426165+0000","last_clean":"2023-10-30T12:29:01.426165+0000","last_became_active":"2023-10-30T12:28:50.367593+0000","last_became_peered":"2023-10-30T12:28:50.367593+0000","last_unstale":"2023-10-30T12:29:01.426165+0000","last_undegraded":"2023-10-30T12:29:01.426165+0000","last_fullsized":"2023-10-30T12:29:01.426165+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:32:10.492061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579128+0000","last_change":"2023-10-30T12:28:50.970536+0000","last_active":"2023-10-30T12:28:53.579128+0000","last_peered":"2023-10-30T12:28:53.579128+0000","last_clean":"2023-10-30T12:28:53.579128+0000","last_became_active":"2023-10-30T12:28:50.970326+0000","last_became_peered":"2023-10-30T12:28:50.970326+0000","last_unstale":"2023-10-30T12:28:53.579128+0000","last_undegraded":"2023-10-30T12:28:53.579128+0000","last_fullsized":"2023-10-30T12:28:53.579128+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:53:54.589174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.958451+0000","last_change":"2023-10-30T12:28:50.392461+0000","last_active":"2023-10-30T12:29:06.958451+0000","last_peered":"2023-10-30T12:29:06.958451+0000","last_clean":"2023-10-30T12:29:06.958451+0000","last_became_active":"2023-10-30T12:28:50.391824+0000","last_became_peered":"2023-10-30T12:28:50.391824+0000","last_unstale":"2023-10-30T12:29:06.958451+0000","last_undegraded":"2023-10-30T12:29:06.958451+0000","last_fullsized":"2023-10-30T12:29:06.958451+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:56:10.373046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579229+0000","last_change":"2023-10-30T12:28:50.953046+0000","last_active":"2023-10-30T12:28:53.579229+0000","last_peered":"2023-10-30T12:28:53.579229+0000","last_clean":"2023-10-30T12:28:53.579229+0000","last_became_active":"2023-10-30T12:28:50.952625+0000","last_became_peered":"2023-10-30T12:28:50.952625+0000","last_unstale":"2023-10-30T12:28:53.579229+0000","last_undegraded":"2023-10-30T12:28:53.579229+0000","last_fullsized":"2023-10-30T12:28:53.579229+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:00:14.643989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.962549+0000","last_change":"2023-10-30T12:28:50.367547+0000","last_active":"2023-10-30T12:29:06.962549+0000","last_peered":"2023-10-30T12:29:06.962549+0000","last_clean":"2023-10-30T12:29:06.962549+0000","last_became_active":"2023-10-30T12:28:50.367230+0000","last_became_peered":"2023-10-30T12:28:50.367230+0000","last_unstale":"2023-10-30T12:29:06.962549+0000","last_undegraded":"2023-10-30T12:29:06.962549+0000","last_fullsized":"2023-10-30T12:29:06.962549+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:42.570786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931639+0000","last_change":"2023-10-30T12:28:50.928888+0000","last_active":"2023-10-30T12:28:51.931639+0000","last_peered":"2023-10-30T12:28:51.931639+0000","last_clean":"2023-10-30T12:28:51.931639+0000","last_became_active":"2023-10-30T12:28:50.928608+0000","last_became_peered":"2023-10-30T12:28:50.928608+0000","last_unstale":"2023-10-30T12:28:51.931639+0000","last_undegraded":"2023-10-30T12:28:51.931639+0000","last_fullsized":"2023-10-30T12:28:51.931639+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:40:51.042396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.582018+0000","last_change":"2023-10-30T12:28:50.391240+0000","last_active":"2023-10-30T12:28:53.582018+0000","last_peered":"2023-10-30T12:28:53.582018+0000","last_clean":"2023-10-30T12:28:53.582018+0000","last_became_active":"2023-10-30T12:28:50.390864+0000","last_became_peered":"2023-10-30T12:28:50.390864+0000","last_unstale":"2023-10-30T12:28:53.582018+0000","last_undegraded":"2023-10-30T12:28:53.582018+0000","last_fullsized":"2023-10-30T12:28:53.582018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:59:40.285830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.939756+0000","last_change":"2023-10-30T12:28:50.949990+0000","last_active":"2023-10-30T12:28:51.939756+0000","last_peered":"2023-10-30T12:28:51.939756+0000","last_clean":"2023-10-30T12:28:51.939756+0000","last_became_active":"2023-10-30T12:28:50.949122+0000","last_became_peered":"2023-10-30T12:28:50.949122+0000","last_unstale":"2023-10-30T12:28:51.939756+0000","last_undegraded":"2023-10-30T12:28:51.939756+0000","last_fullsized":"2023-10-30T12:28:51.939756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:30:56.099810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931803+0000","last_change":"2023-10-30T12:28:50.358700+0000","last_active":"2023-10-30T12:28:51.931803+0000","last_peered":"2023-10-30T12:28:51.931803+0000","last_clean":"2023-10-30T12:28:51.931803+0000","last_became_active":"2023-10-30T12:28:50.358310+0000","last_became_peered":"2023-10-30T12:28:50.358310+0000","last_unstale":"2023-10-30T12:28:51.931803+0000","last_undegraded":"2023-10-30T12:28:51.931803+0000","last_fullsized":"2023-10-30T12:28:51.931803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:23:30.910622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.582250+0000","last_change":"2023-10-30T12:28:50.967650+0000","last_active":"2023-10-30T12:28:53.582250+0000","last_peered":"2023-10-30T12:28:53.582250+0000","last_clean":"2023-10-30T12:28:53.582250+0000","last_became_active":"2023-10-30T12:28:50.965626+0000","last_became_peered":"2023-10-30T12:28:50.965626+0000","last_unstale":"2023-10-30T12:28:53.582250+0000","last_undegraded":"2023-10-30T12:28:53.582250+0000","last_fullsized":"2023-10-30T12:28:53.582250+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:49.050703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.029231+0000","last_change":"2023-10-30T12:28:50.395645+0000","last_active":"2023-10-30T12:28:53.029231+0000","last_peered":"2023-10-30T12:28:53.029231+0000","last_clean":"2023-10-30T12:28:53.029231+0000","last_became_active":"2023-10-30T12:28:50.395090+0000","last_became_peered":"2023-10-30T12:28:50.395090+0000","last_unstale":"2023-10-30T12:28:53.029231+0000","last_undegraded":"2023-10-30T12:28:53.029231+0000","last_fullsized":"2023-10-30T12:28:53.029231+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:45:44.678383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931844+0000","last_change":"2023-10-30T12:28:50.929586+0000","last_active":"2023-10-30T12:28:51.931844+0000","last_peered":"2023-10-30T12:28:51.931844+0000","last_clean":"2023-10-30T12:28:51.931844+0000","last_became_active":"2023-10-30T12:28:50.929337+0000","last_became_peered":"2023-10-30T12:28:50.929337+0000","last_unstale":"2023-10-30T12:28:51.931844+0000","last_undegraded":"2023-10-30T12:28:51.931844+0000","last_fullsized":"2023-10-30T12:28:51.931844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:11:13.067698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.958476+0000","last_change":"2023-10-30T12:28:50.380474+0000","last_active":"2023-10-30T12:29:06.958476+0000","last_peered":"2023-10-30T12:29:06.958476+0000","last_clean":"2023-10-30T12:29:06.958476+0000","last_became_active":"2023-10-30T12:28:50.380247+0000","last_became_peered":"2023-10-30T12:28:50.380247+0000","last_unstale":"2023-10-30T12:29:06.958476+0000","last_undegraded":"2023-10-30T12:29:06.958476+0000","last_fullsized":"2023-10-30T12:29:06.958476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:52:03.952617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579476+0000","last_change":"2023-10-30T12:28:50.973361+0000","last_active":"2023-10-30T12:28:53.579476+0000","last_peered":"2023-10-30T12:28:53.579476+0000","last_clean":"2023-10-30T12:28:53.579476+0000","last_became_active":"2023-10-30T12:28:50.973086+0000","last_became_peered":"2023-10-30T12:28:50.973086+0000","last_unstale":"2023-10-30T12:28:53.579476+0000","last_undegraded":"2023-10-30T12:28:53.579476+0000","last_fullsized":"2023-10-30T12:28:53.579476+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:49:48.589889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.421918+0000","last_change":"2023-10-30T12:28:50.386277+0000","last_active":"2023-10-30T12:29:01.421918+0000","last_peered":"2023-10-30T12:29:01.421918+0000","last_clean":"2023-10-30T12:29:01.421918+0000","last_became_active":"2023-10-30T12:28:50.385924+0000","last_became_peered":"2023-10-30T12:28:50.385924+0000","last_unstale":"2023-10-30T12:29:01.421918+0000","last_undegraded":"2023-10-30T12:29:01.421918+0000","last_fullsized":"2023-10-30T12:29:01.421918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:35:03.837523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.580322+0000","last_change":"2023-10-30T12:28:50.929670+0000","last_active":"2023-10-30T12:28:53.580322+0000","last_peered":"2023-10-30T12:28:53.580322+0000","last_clean":"2023-10-30T12:28:53.580322+0000","last_became_active":"2023-10-30T12:28:50.929259+0000","last_became_peered":"2023-10-30T12:28:50.929259+0000","last_unstale":"2023-10-30T12:28:53.580322+0000","last_undegraded":"2023-10-30T12:28:53.580322+0000","last_fullsized":"2023-10-30T12:28:53.580322+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:50:16.507660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.420373+0000","last_change":"2023-10-30T12:28:50.386426+0000","last_active":"2023-10-30T12:29:01.420373+0000","last_peered":"2023-10-30T12:29:01.420373+0000","last_clean":"2023-10-30T12:29:01.420373+0000","last_became_active":"2023-10-30T12:28:50.385305+0000","last_became_peered":"2023-10-30T12:28:50.385305+0000","last_unstale":"2023-10-30T12:29:01.420373+0000","last_undegraded":"2023-10-30T12:29:01.420373+0000","last_fullsized":"2023-10-30T12:29:01.420373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:13:05.376581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.938368+0000","last_change":"2023-10-30T12:28:50.941607+0000","last_active":"2023-10-30T12:28:51.938368+0000","last_peered":"2023-10-30T12:28:51.938368+0000","last_clean":"2023-10-30T12:28:51.938368+0000","last_became_active":"2023-10-30T12:28:50.940846+0000","last_became_peered":"2023-10-30T12:28:50.940846+0000","last_unstale":"2023-10-30T12:28:51.938368+0000","last_undegraded":"2023-10-30T12:28:51.938368+0000","last_fullsized":"2023-10-30T12:28:51.938368+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:29.837077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.433290+0000","last_change":"2023-10-30T12:28:50.392333+0000","last_active":"2023-10-30T12:29:01.433290+0000","last_peered":"2023-10-30T12:29:01.433290+0000","last_clean":"2023-10-30T12:29:01.433290+0000","last_became_active":"2023-10-30T12:28:50.392068+0000","last_became_peered":"2023-10-30T12:28:50.392068+0000","last_unstale":"2023-10-30T12:29:01.433290+0000","last_undegraded":"2023-10-30T12:29:01.433290+0000","last_fullsized":"2023-10-30T12:29:01.433290+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:47:24.674925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579044+0000","last_change":"2023-10-30T12:28:50.932167+0000","last_active":"2023-10-30T12:28:53.579044+0000","last_peered":"2023-10-30T12:28:53.579044+0000","last_clean":"2023-10-30T12:28:53.579044+0000","last_became_active":"2023-10-30T12:28:50.931898+0000","last_became_peered":"2023-10-30T12:28:50.931898+0000","last_unstale":"2023-10-30T12:28:53.579044+0000","last_undegraded":"2023-10-30T12:28:53.579044+0000","last_fullsized":"2023-10-30T12:28:53.579044+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:44.429045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:52.280154+0000","last_change":"2023-10-30T12:28:50.365055+0000","last_active":"2023-10-30T12:28:52.280154+0000","last_peered":"2023-10-30T12:28:52.280154+0000","last_clean":"2023-10-30T12:28:52.280154+0000","last_became_active":"2023-10-30T12:28:50.364797+0000","last_became_peered":"2023-10-30T12:28:50.364797+0000","last_unstale":"2023-10-30T12:28:52.280154+0000","last_undegraded":"2023-10-30T12:28:52.280154+0000","last_fullsized":"2023-10-30T12:28:52.280154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:06:23.067895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579266+0000","last_change":"2023-10-30T12:28:50.935908+0000","last_active":"2023-10-30T12:28:53.579266+0000","last_peered":"2023-10-30T12:28:53.579266+0000","last_clean":"2023-10-30T12:28:53.579266+0000","last_became_active":"2023-10-30T12:28:50.935657+0000","last_became_peered":"2023-10-30T12:28:50.935657+0000","last_unstale":"2023-10-30T12:28:53.579266+0000","last_undegraded":"2023-10-30T12:28:53.579266+0000","last_fullsized":"2023-10-30T12:28:53.579266+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:36:13.242429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.416976+0000","last_change":"2023-10-30T12:28:50.393465+0000","last_active":"2023-10-30T12:29:01.416976+0000","last_peered":"2023-10-30T12:29:01.416976+0000","last_clean":"2023-10-30T12:29:01.416976+0000","last_became_active":"2023-10-30T12:28:50.392807+0000","last_became_peered":"2023-10-30T12:28:50.392807+0000","last_unstale":"2023-10-30T12:29:01.416976+0000","last_undegraded":"2023-10-30T12:29:01.416976+0000","last_fullsized":"2023-10-30T12:29:01.416976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:35:37.976956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.938767+0000","last_change":"2023-10-30T12:28:50.954834+0000","last_active":"2023-10-30T12:28:51.938767+0000","last_peered":"2023-10-30T12:28:51.938767+0000","last_clean":"2023-10-30T12:28:51.938767+0000","last_became_active":"2023-10-30T12:28:50.954291+0000","last_became_peered":"2023-10-30T12:28:50.954291+0000","last_unstale":"2023-10-30T12:28:51.938767+0000","last_undegraded":"2023-10-30T12:28:51.938767+0000","last_fullsized":"2023-10-30T12:28:51.938767+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:19:44.345918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581732+0000","last_change":"2023-10-30T12:28:50.963620+0000","last_active":"2023-10-30T12:28:53.581732+0000","last_peered":"2023-10-30T12:28:53.581732+0000","last_clean":"2023-10-30T12:28:53.581732+0000","last_became_active":"2023-10-30T12:28:50.962817+0000","last_became_peered":"2023-10-30T12:28:50.962817+0000","last_unstale":"2023-10-30T12:28:53.581732+0000","last_undegraded":"2023-10-30T12:28:53.581732+0000","last_fullsized":"2023-10-30T12:28:53.581732+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:23:23.276086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:56.367962+0000","last_change":"2023-10-30T12:28:50.402179+0000","last_active":"2023-10-30T12:28:56.367962+0000","last_peered":"2023-10-30T12:28:56.367962+0000","last_clean":"2023-10-30T12:28:56.367962+0000","last_became_active":"2023-10-30T12:28:50.401677+0000","last_became_peered":"2023-10-30T12:28:50.401677+0000","last_unstale":"2023-10-30T12:28:56.367962+0000","last_undegraded":"2023-10-30T12:28:56.367962+0000","last_fullsized":"2023-10-30T12:28:56.367962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:39:44.432214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581874+0000","last_change":"2023-10-30T12:28:50.968805+0000","last_active":"2023-10-30T12:28:53.581874+0000","last_peered":"2023-10-30T12:28:53.581874+0000","last_clean":"2023-10-30T12:28:53.581874+0000","last_became_active":"2023-10-30T12:28:50.967591+0000","last_became_peered":"2023-10-30T12:28:50.967591+0000","last_unstale":"2023-10-30T12:28:53.581874+0000","last_undegraded":"2023-10-30T12:28:53.581874+0000","last_fullsized":"2023-10-30T12:28:53.581874+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:11:41.973858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.957870+0000","last_change":"2023-10-30T12:28:50.392617+0000","last_active":"2023-10-30T12:29:06.957870+0000","last_peered":"2023-10-30T12:29:06.957870+0000","last_clean":"2023-10-30T12:29:06.957870+0000","last_became_active":"2023-10-30T12:28:50.392021+0000","last_became_peered":"2023-10-30T12:28:50.392021+0000","last_unstale":"2023-10-30T12:29:06.957870+0000","last_undegraded":"2023-10-30T12:29:06.957870+0000","last_fullsized":"2023-10-30T12:29:06.957870+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:52:30.621125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.580378+0000","last_change":"2023-10-30T12:28:50.967491+0000","last_active":"2023-10-30T12:28:53.580378+0000","last_peered":"2023-10-30T12:28:53.580378+0000","last_clean":"2023-10-30T12:28:53.580378+0000","last_became_active":"2023-10-30T12:28:50.967186+0000","last_became_peered":"2023-10-30T12:28:50.967186+0000","last_unstale":"2023-10-30T12:28:53.580378+0000","last_undegraded":"2023-10-30T12:28:53.580378+0000","last_fullsized":"2023-10-30T12:28:53.580378+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:05:23.759940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:56.365702+0000","last_change":"2023-10-30T12:28:50.363230+0000","last_active":"2023-10-30T12:28:56.365702+0000","last_peered":"2023-10-30T12:28:56.365702+0000","last_clean":"2023-10-30T12:28:56.365702+0000","last_became_active":"2023-10-30T12:28:50.362929+0000","last_became_peered":"2023-10-30T12:28:50.362929+0000","last_unstale":"2023-10-30T12:28:56.365702+0000","last_undegraded":"2023-10-30T12:28:56.365702+0000","last_fullsized":"2023-10-30T12:28:56.365702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:41:27.760578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579661+0000","last_change":"2023-10-30T12:28:50.937780+0000","last_active":"2023-10-30T12:28:53.579661+0000","last_peered":"2023-10-30T12:28:53.579661+0000","last_clean":"2023-10-30T12:28:53.579661+0000","last_became_active":"2023-10-30T12:28:50.937458+0000","last_became_peered":"2023-10-30T12:28:50.937458+0000","last_unstale":"2023-10-30T12:28:53.579661+0000","last_undegraded":"2023-10-30T12:28:53.579661+0000","last_fullsized":"2023-10-30T12:28:53.579661+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:02.083169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"22'9","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.962486+0000","last_change":"2023-10-30T12:28:50.386782+0000","last_active":"2023-10-30T12:29:06.962486+0000","last_peered":"2023-10-30T12:29:06.962486+0000","last_clean":"2023-10-30T12:29:06.962486+0000","last_became_active":"2023-10-30T12:28:50.386522+0000","last_became_peered":"2023-10-30T12:28:50.386522+0000","last_unstale":"2023-10-30T12:29:06.962486+0000","last_undegraded":"2023-10-30T12:29:06.962486+0000","last_fullsized":"2023-10-30T12:29:06.962486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:44:03.517641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6553,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":13106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113,"ondisk_log_size":113,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542156,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27836,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344004,"statfs":{"total":96636764160,"available":96608260096,"internally_reserved":0,"allocated":839680,"data_stored":670548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0549999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]}]},{"osd":6,"up_from":14,"seq":60129542156,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":245760,"data_stored":81670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0920000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542156,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":241664,"data_stored":79623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94599999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2250000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0760000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542155,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":241664,"data_stored":79836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1180000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.093}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]}]},{"osd":3,"up_from":14,"seq":60129542156,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":249856,"data_stored":82261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.012}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93300000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0680000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542156,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":241664,"data_stored":79063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92400000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542156,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":249856,"data_stored":79808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]}]},{"osd":0,"up_from":14,"seq":60129542156,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":831488,"data_stored":669273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85599999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:29:19.143 INFO:tasks.mds_thrash.ceph_manager:clean! 2023-10-30T12:29:19.143 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:29:19.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.315+0000 7f40e9054640 1 Processor -- start 2023-10-30T12:29:19.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.315+0000 7f40e9054640 1 -- start start 2023-10-30T12:29:19.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.315+0000 7f40e9054640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f40e41538a0 0x7f40e415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:19.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.315+0000 7f40e9054640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f40e40cf160 0x7f40e40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:19.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.315+0000 7f40e9054640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40e40cb800 con 0x7f40e40ca0e0 2023-10-30T12:29:19.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.315+0000 7f40e9054640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40e415dc30 con 0x7f40e40cf160 2023-10-30T12:29:19.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.315+0000 7f40e9054640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40e40cb330 con 0x7f40e41538a0 2023-10-30T12:29:19.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.315+0000 7f40e359e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f40e40cf160 0x7f40e40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:19.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.315+0000 7f40e359e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f40e40cf160 0x7f40e40cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57956/0 (socket says 172.21.15.89:57956) 2023-10-30T12:29:19.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.315+0000 7f40e359e640 1 -- 172.21.15.89:0/1739327258 learned_addr learned my addr 172.21.15.89:0/1739327258 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:19.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.315+0000 7f40e2d9d640 1 --2- 172.21.15.89:0/1739327258 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f40e41538a0 0x7f40e415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:19.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.315+0000 7f40e359e640 1 -- 172.21.15.89:0/1739327258 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f40e41538a0 msgr2=0x7f40e415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:19.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.315+0000 7f40e359e640 1 --2- 172.21.15.89:0/1739327258 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f40e41538a0 0x7f40e415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:19.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.315+0000 7f40e359e640 1 -- 172.21.15.89:0/1739327258 >> v1:172.21.15.89:6789/0 conn(0x7f40e40ca0e0 legacy=0x7f40e415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:19.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.315+0000 7f40e359e640 1 -- 172.21.15.89:0/1739327258 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e40cad70 con 0x7f40e40cf160 2023-10-30T12:29:19.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.315+0000 7f40e1d9b640 1 -- 172.21.15.89:0/1739327258 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855687536 0 0) 0x7f40e40cb800 con 0x7f40e40ca0e0 2023-10-30T12:29:19.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.319+0000 7f40e359e640 1 --2- 172.21.15.89:0/1739327258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40e40cf160 0x7f40e40cf590 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f40dc009e50 tx=0x7f40dc01f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=230675216dce65c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:19.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.319+0000 7f40e1d9b640 1 -- 172.21.15.89:0/1739327258 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40dc0af400 con 0x7f40e40cf160 2023-10-30T12:29:19.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.319+0000 7f40e1d9b640 1 -- 172.21.15.89:0/1739327258 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40dc0ab050 con 0x7f40e40cf160 2023-10-30T12:29:19.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.319+0000 7f40e1d9b640 1 -- 172.21.15.89:0/1739327258 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40dc0af780 con 0x7f40e40cf160 2023-10-30T12:29:19.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.319+0000 7f40e9054640 1 -- 172.21.15.89:0/1739327258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40e40cf160 msgr2=0x7f40e40cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:19.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.319+0000 7f40e9054640 1 --2- 172.21.15.89:0/1739327258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40e40cf160 0x7f40e40cf590 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f40dc009e50 tx=0x7f40dc01f1f0 comp rx=0 tx=0).stop 2023-10-30T12:29:19.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.319+0000 7f40e9054640 1 -- 172.21.15.89:0/1739327258 shutdown_connections 2023-10-30T12:29:19.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.319+0000 7f40e9054640 1 --2- 172.21.15.89:0/1739327258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40e40cf160 0x7f40e40cf590 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:19.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.319+0000 7f40e9054640 1 --2- 172.21.15.89:0/1739327258 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f40e41538a0 0x7f40e415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:19.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.319+0000 7f40e9054640 1 -- 172.21.15.89:0/1739327258 >> 172.21.15.89:0/1739327258 conn(0x7f40e406bfe0 msgr2=0x7f40e406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:19.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.319+0000 7f40e9054640 1 -- 172.21.15.89:0/1739327258 shutdown_connections 2023-10-30T12:29:19.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.319+0000 7f40e9054640 1 -- 172.21.15.89:0/1739327258 wait complete. 2023-10-30T12:29:19.326 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.323+0000 7f40e9054640 1 Processor -- start 2023-10-30T12:29:19.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.323+0000 7f40e9054640 1 -- start start 2023-10-30T12:29:19.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.323+0000 7f40e9054640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40e40ca0e0 0x7f40e40c57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:19.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.323+0000 7f40e9054640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40e40cf160 0x7f40e415c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:19.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.323+0000 7f40e9054640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f40e41538a0 0x7f40e415ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:19.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.323+0000 7f40e9054640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e415ac80 con 0x7f40e40cf160 2023-10-30T12:29:19.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.323+0000 7f40e9054640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e415aee0 con 0x7f40e40ca0e0 2023-10-30T12:29:19.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.323+0000 7f40e9054640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40e41f2530 con 0x7f40e41538a0 2023-10-30T12:29:19.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.323+0000 7f40e259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40e40cf160 0x7f40e415c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:19.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.323+0000 7f40e259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40e40cf160 0x7f40e415c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57962/0 (socket says 172.21.15.89:57962) 2023-10-30T12:29:19.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.323+0000 7f40e259c640 1 -- 172.21.15.89:0/1872681205 learned_addr learned my addr 172.21.15.89:0/1872681205 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:19.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.323+0000 7f40e359e640 1 --2- 172.21.15.89:0/1872681205 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f40e41538a0 0x7f40e415ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:19.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.323+0000 7f40e2d9d640 1 --2- 172.21.15.89:0/1872681205 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40e40ca0e0 0x7f40e40c57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:19.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.323+0000 7f40e259c640 1 -- 172.21.15.89:0/1872681205 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f40e41538a0 msgr2=0x7f40e415ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:19.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.323+0000 7f40e259c640 1 --2- 172.21.15.89:0/1872681205 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f40e41538a0 0x7f40e415ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:19.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.323+0000 7f40e259c640 1 -- 172.21.15.89:0/1872681205 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40e40ca0e0 msgr2=0x7f40e40c57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:19.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.323+0000 7f40e259c640 1 --2- 172.21.15.89:0/1872681205 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40e40ca0e0 0x7f40e40c57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:19.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.323+0000 7f40e259c640 1 -- 172.21.15.89:0/1872681205 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40dc009b30 con 0x7f40e40cf160 2023-10-30T12:29:19.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.323+0000 7f40e259c640 1 --2- 172.21.15.89:0/1872681205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40e40cf160 0x7f40e415c510 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f40d800c4d0 tx=0x7f40d800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:19.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.323+0000 7f40cb7fe640 1 -- 172.21.15.89:0/1872681205 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d8005480 con 0x7f40e40cf160 2023-10-30T12:29:19.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.323+0000 7f40cb7fe640 1 -- 172.21.15.89:0/1872681205 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40d8003600 con 0x7f40e40cf160 2023-10-30T12:29:19.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.323+0000 7f40cb7fe640 1 -- 172.21.15.89:0/1872681205 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d80039b0 con 0x7f40e40cf160 2023-10-30T12:29:19.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.323+0000 7f40e9054640 1 -- 172.21.15.89:0/1872681205 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40e41f2810 con 0x7f40e40cf160 2023-10-30T12:29:19.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.323+0000 7f40e9054640 1 -- 172.21.15.89:0/1872681205 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40e41f2dd0 con 0x7f40e40cf160 2023-10-30T12:29:19.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.323+0000 7f40e9054640 1 -- 172.21.15.89:0/1872681205 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40e40b2a50 con 0x7f40e40cf160 2023-10-30T12:29:19.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.327+0000 7f40cb7fe640 1 -- 172.21.15.89:0/1872681205 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f40d8003b10 con 0x7f40e40cf160 2023-10-30T12:29:19.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.327+0000 7f40cb7fe640 1 --2- 172.21.15.89:0/1872681205 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f40d00664d0 0x7f40d0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:19.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.327+0000 7f40cb7fe640 1 -- 172.21.15.89:0/1872681205 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f40d8083410 con 0x7f40e40cf160 2023-10-30T12:29:19.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.327+0000 7f40e2d9d640 1 --2- 172.21.15.89:0/1872681205 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f40d00664d0 0x7f40d0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:19.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.327+0000 7f40e2d9d640 1 --2- 172.21.15.89:0/1872681205 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f40d00664d0 0x7f40d0068990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f40d4009a50 tx=0x7f40d4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:19.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.331+0000 7f40cb7fe640 1 -- 172.21.15.89:0/1872681205 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f40d8050940 con 0x7f40e40cf160 2023-10-30T12:29:19.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.475+0000 7f40e9054640 1 -- 172.21.15.89:0/1872681205 --> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f40e4159890 con 0x7f40d00664d0 2023-10-30T12:29:19.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.495+0000 7f40cb7fe640 1 -- 172.21.15.89:0/1872681205 <== mgr.4105 v2:172.21.15.192:6832/4076276638 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264666 (secure 0 0 0) 0x7f40d40119a0 con 0x7f40d00664d0 2023-10-30T12:29:19.503 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:29:19.504 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-10-30T12:29:19.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.499+0000 7f40e9054640 1 -- 172.21.15.89:0/1872681205 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f40d00664d0 msgr2=0x7f40d0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:19.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.499+0000 7f40e9054640 1 --2- 172.21.15.89:0/1872681205 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f40d00664d0 0x7f40d0068990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f40d4009a50 tx=0x7f40d4008040 comp rx=0 tx=0).stop 2023-10-30T12:29:19.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.499+0000 7f40e9054640 1 -- 172.21.15.89:0/1872681205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40e40cf160 msgr2=0x7f40e415c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:19.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.499+0000 7f40e9054640 1 --2- 172.21.15.89:0/1872681205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40e40cf160 0x7f40e415c510 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f40d800c4d0 tx=0x7f40d800c990 comp rx=0 tx=0).stop 2023-10-30T12:29:19.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.499+0000 7f40e9054640 1 -- 172.21.15.89:0/1872681205 shutdown_connections 2023-10-30T12:29:19.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.499+0000 7f40e9054640 1 --2- 172.21.15.89:0/1872681205 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f40d00664d0 0x7f40d0068990 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:19.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.499+0000 7f40e9054640 1 --2- 172.21.15.89:0/1872681205 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f40e41538a0 0x7f40e415ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:19.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.499+0000 7f40e9054640 1 --2- 172.21.15.89:0/1872681205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40e40cf160 0x7f40e415c510 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:19.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.499+0000 7f40e9054640 1 --2- 172.21.15.89:0/1872681205 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40e40ca0e0 0x7f40e40c57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:19.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.499+0000 7f40e9054640 1 -- 172.21.15.89:0/1872681205 >> 172.21.15.89:0/1872681205 conn(0x7f40e406bfe0 msgr2=0x7f40e41523a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:19.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.499+0000 7f40e9054640 1 -- 172.21.15.89:0/1872681205 shutdown_connections 2023-10-30T12:29:19.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.499+0000 7f40e9054640 1 -- 172.21.15.89:0/1872681205 wait complete. 2023-10-30T12:29:19.521 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2023-10-30T12:29:18.928978+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596921,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":288,"num_write_kb":2458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":230,"ondisk_log_size":230,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219196,"kb_used_data":3068,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755524,"statfs":{"total":773094113280,"available":772869656576,"internally_reserved":0,"allocated":3141632,"data_stored":1822082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12741,"internal_metadata":221236795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2737,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004369"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028790+0000","last_change":"2023-10-30T12:28:50.979797+0000","last_active":"2023-10-30T12:28:53.028790+0000","last_peered":"2023-10-30T12:28:53.028790+0000","last_clean":"2023-10-30T12:28:53.028790+0000","last_became_active":"2023-10-30T12:28:50.979351+0000","last_became_peered":"2023-10-30T12:28:50.979351+0000","last_unstale":"2023-10-30T12:28:53.028790+0000","last_undegraded":"2023-10-30T12:28:53.028790+0000","last_fullsized":"2023-10-30T12:28:53.028790+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:53:00.556539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.932043+0000","last_change":"2023-10-30T12:28:50.929403+0000","last_active":"2023-10-30T12:28:51.932043+0000","last_peered":"2023-10-30T12:28:51.932043+0000","last_clean":"2023-10-30T12:28:51.932043+0000","last_became_active":"2023-10-30T12:28:50.929133+0000","last_became_peered":"2023-10-30T12:28:50.929133+0000","last_unstale":"2023-10-30T12:28:51.932043+0000","last_undegraded":"2023-10-30T12:28:51.932043+0000","last_fullsized":"2023-10-30T12:28:51.932043+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:12:57.327440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579508+0000","last_change":"2023-10-30T12:28:50.934201+0000","last_active":"2023-10-30T12:28:53.579508+0000","last_peered":"2023-10-30T12:28:53.579508+0000","last_clean":"2023-10-30T12:28:53.579508+0000","last_became_active":"2023-10-30T12:28:50.933896+0000","last_became_peered":"2023-10-30T12:28:50.933896+0000","last_unstale":"2023-10-30T12:28:53.579508+0000","last_undegraded":"2023-10-30T12:28:53.579508+0000","last_fullsized":"2023-10-30T12:28:53.579508+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:20:46.597550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581831+0000","last_change":"2023-10-30T12:28:50.969193+0000","last_active":"2023-10-30T12:28:53.581831+0000","last_peered":"2023-10-30T12:28:53.581831+0000","last_clean":"2023-10-30T12:28:53.581831+0000","last_became_active":"2023-10-30T12:28:50.967774+0000","last_became_peered":"2023-10-30T12:28:50.967774+0000","last_unstale":"2023-10-30T12:28:53.581831+0000","last_undegraded":"2023-10-30T12:28:53.581831+0000","last_fullsized":"2023-10-30T12:28:53.581831+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:23:07.329372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028482+0000","last_change":"2023-10-30T12:28:50.973751+0000","last_active":"2023-10-30T12:28:53.028482+0000","last_peered":"2023-10-30T12:28:53.028482+0000","last_clean":"2023-10-30T12:28:53.028482+0000","last_became_active":"2023-10-30T12:28:50.973132+0000","last_became_peered":"2023-10-30T12:28:50.973132+0000","last_unstale":"2023-10-30T12:28:53.028482+0000","last_undegraded":"2023-10-30T12:28:53.028482+0000","last_fullsized":"2023-10-30T12:28:53.028482+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:43:58.223555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581257+0000","last_change":"2023-10-30T12:28:50.968823+0000","last_active":"2023-10-30T12:28:53.581257+0000","last_peered":"2023-10-30T12:28:53.581257+0000","last_clean":"2023-10-30T12:28:53.581257+0000","last_became_active":"2023-10-30T12:28:50.967910+0000","last_became_peered":"2023-10-30T12:28:50.967910+0000","last_unstale":"2023-10-30T12:28:53.581257+0000","last_undegraded":"2023-10-30T12:28:53.581257+0000","last_fullsized":"2023-10-30T12:28:53.581257+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:34:36.882757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028044+0000","last_change":"2023-10-30T12:28:50.974164+0000","last_active":"2023-10-30T12:28:53.028044+0000","last_peered":"2023-10-30T12:28:53.028044+0000","last_clean":"2023-10-30T12:28:53.028044+0000","last_became_active":"2023-10-30T12:28:50.973288+0000","last_became_peered":"2023-10-30T12:28:50.973288+0000","last_unstale":"2023-10-30T12:28:53.028044+0000","last_undegraded":"2023-10-30T12:28:53.028044+0000","last_fullsized":"2023-10-30T12:28:53.028044+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:09:10.450492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.578880+0000","last_change":"2023-10-30T12:28:50.955287+0000","last_active":"2023-10-30T12:28:53.578880+0000","last_peered":"2023-10-30T12:28:53.578880+0000","last_clean":"2023-10-30T12:28:53.578880+0000","last_became_active":"2023-10-30T12:28:50.954988+0000","last_became_peered":"2023-10-30T12:28:50.954988+0000","last_unstale":"2023-10-30T12:28:53.578880+0000","last_undegraded":"2023-10-30T12:28:53.578880+0000","last_fullsized":"2023-10-30T12:28:53.578880+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:36:30.846920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.029019+0000","last_change":"2023-10-30T12:28:50.977144+0000","last_active":"2023-10-30T12:28:53.029019+0000","last_peered":"2023-10-30T12:28:53.029019+0000","last_clean":"2023-10-30T12:28:53.029019+0000","last_became_active":"2023-10-30T12:28:50.976564+0000","last_became_peered":"2023-10-30T12:28:50.976564+0000","last_unstale":"2023-10-30T12:28:53.029019+0000","last_undegraded":"2023-10-30T12:28:53.029019+0000","last_fullsized":"2023-10-30T12:28:53.029019+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:37:51.341935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.938542+0000","last_change":"2023-10-30T12:28:50.952564+0000","last_active":"2023-10-30T12:28:51.938542+0000","last_peered":"2023-10-30T12:28:51.938542+0000","last_clean":"2023-10-30T12:28:51.938542+0000","last_became_active":"2023-10-30T12:28:50.951809+0000","last_became_peered":"2023-10-30T12:28:50.951809+0000","last_unstale":"2023-10-30T12:28:51.938542+0000","last_undegraded":"2023-10-30T12:28:51.938542+0000","last_fullsized":"2023-10-30T12:28:51.938542+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:34:29.771577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.937215+0000","last_change":"2023-10-30T12:28:50.970779+0000","last_active":"2023-10-30T12:28:51.937215+0000","last_peered":"2023-10-30T12:28:51.937215+0000","last_clean":"2023-10-30T12:28:51.937215+0000","last_became_active":"2023-10-30T12:28:50.969877+0000","last_became_peered":"2023-10-30T12:28:50.969877+0000","last_unstale":"2023-10-30T12:28:51.937215+0000","last_undegraded":"2023-10-30T12:28:51.937215+0000","last_fullsized":"2023-10-30T12:28:51.937215+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:10.539915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.940164+0000","last_change":"2023-10-30T12:28:50.944452+0000","last_active":"2023-10-30T12:28:51.940164+0000","last_peered":"2023-10-30T12:28:51.940164+0000","last_clean":"2023-10-30T12:28:51.940164+0000","last_became_active":"2023-10-30T12:28:50.943787+0000","last_became_peered":"2023-10-30T12:28:50.943787+0000","last_unstale":"2023-10-30T12:28:51.940164+0000","last_undegraded":"2023-10-30T12:28:51.940164+0000","last_fullsized":"2023-10-30T12:28:51.940164+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:30:34.585298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579240+0000","last_change":"2023-10-30T12:28:50.938181+0000","last_active":"2023-10-30T12:28:53.579240+0000","last_peered":"2023-10-30T12:28:53.579240+0000","last_clean":"2023-10-30T12:28:53.579240+0000","last_became_active":"2023-10-30T12:28:50.937906+0000","last_became_peered":"2023-10-30T12:28:50.937906+0000","last_unstale":"2023-10-30T12:28:53.579240+0000","last_undegraded":"2023-10-30T12:28:53.579240+0000","last_fullsized":"2023-10-30T12:28:53.579240+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:41:19.990452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579303+0000","last_change":"2023-10-30T12:28:50.934085+0000","last_active":"2023-10-30T12:28:53.579303+0000","last_peered":"2023-10-30T12:28:53.579303+0000","last_clean":"2023-10-30T12:28:53.579303+0000","last_became_active":"2023-10-30T12:28:50.932753+0000","last_became_peered":"2023-10-30T12:28:50.932753+0000","last_unstale":"2023-10-30T12:28:53.579303+0000","last_undegraded":"2023-10-30T12:28:53.579303+0000","last_fullsized":"2023-10-30T12:28:53.579303+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:23:26.736283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931710+0000","last_change":"2023-10-30T12:28:50.943343+0000","last_active":"2023-10-30T12:28:51.931710+0000","last_peered":"2023-10-30T12:28:51.931710+0000","last_clean":"2023-10-30T12:28:51.931710+0000","last_became_active":"2023-10-30T12:28:50.943011+0000","last_became_peered":"2023-10-30T12:28:50.943011+0000","last_unstale":"2023-10-30T12:28:51.931710+0000","last_undegraded":"2023-10-30T12:28:51.931710+0000","last_fullsized":"2023-10-30T12:28:51.931710+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:59:35.881115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581157+0000","last_change":"2023-10-30T12:28:50.953573+0000","last_active":"2023-10-30T12:28:53.581157+0000","last_peered":"2023-10-30T12:28:53.581157+0000","last_clean":"2023-10-30T12:28:53.581157+0000","last_became_active":"2023-10-30T12:28:50.952744+0000","last_became_peered":"2023-10-30T12:28:50.952744+0000","last_unstale":"2023-10-30T12:28:53.581157+0000","last_undegraded":"2023-10-30T12:28:53.581157+0000","last_fullsized":"2023-10-30T12:28:53.581157+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:41:16.209759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.582342+0000","last_change":"2023-10-30T12:28:50.975156+0000","last_active":"2023-10-30T12:28:53.582342+0000","last_peered":"2023-10-30T12:28:53.582342+0000","last_clean":"2023-10-30T12:28:53.582342+0000","last_became_active":"2023-10-30T12:28:50.974576+0000","last_became_peered":"2023-10-30T12:28:50.974576+0000","last_unstale":"2023-10-30T12:28:53.582342+0000","last_undegraded":"2023-10-30T12:28:53.582342+0000","last_fullsized":"2023-10-30T12:28:53.582342+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:46.159555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.131817+0000","last_change":"2023-10-30T12:28:50.937685+0000","last_active":"2023-10-30T12:28:53.131817+0000","last_peered":"2023-10-30T12:28:53.131817+0000","last_clean":"2023-10-30T12:28:53.131817+0000","last_became_active":"2023-10-30T12:28:50.937384+0000","last_became_peered":"2023-10-30T12:28:50.937384+0000","last_unstale":"2023-10-30T12:28:53.131817+0000","last_undegraded":"2023-10-30T12:28:53.131817+0000","last_fullsized":"2023-10-30T12:28:53.131817+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:16:29.181523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.029181+0000","last_change":"2023-10-30T12:28:50.974918+0000","last_active":"2023-10-30T12:28:53.029181+0000","last_peered":"2023-10-30T12:28:53.029181+0000","last_clean":"2023-10-30T12:28:53.029181+0000","last_became_active":"2023-10-30T12:28:50.973369+0000","last_became_peered":"2023-10-30T12:28:50.973369+0000","last_unstale":"2023-10-30T12:28:53.029181+0000","last_undegraded":"2023-10-30T12:28:53.029181+0000","last_fullsized":"2023-10-30T12:28:53.029181+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:44:04.468611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.937226+0000","last_change":"2023-10-30T12:28:50.955835+0000","last_active":"2023-10-30T12:28:51.937226+0000","last_peered":"2023-10-30T12:28:51.937226+0000","last_clean":"2023-10-30T12:28:51.937226+0000","last_became_active":"2023-10-30T12:28:50.954831+0000","last_became_peered":"2023-10-30T12:28:50.954831+0000","last_unstale":"2023-10-30T12:28:51.937226+0000","last_undegraded":"2023-10-30T12:28:51.937226+0000","last_fullsized":"2023-10-30T12:28:51.937226+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:56:25.554582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.580290+0000","last_change":"2023-10-30T12:28:50.962417+0000","last_active":"2023-10-30T12:28:53.580290+0000","last_peered":"2023-10-30T12:28:53.580290+0000","last_clean":"2023-10-30T12:28:53.580290+0000","last_became_active":"2023-10-30T12:28:50.962065+0000","last_became_peered":"2023-10-30T12:28:50.962065+0000","last_unstale":"2023-10-30T12:28:53.580290+0000","last_undegraded":"2023-10-30T12:28:53.580290+0000","last_fullsized":"2023-10-30T12:28:53.580290+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:13:52.455824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581454+0000","last_change":"2023-10-30T12:28:50.968992+0000","last_active":"2023-10-30T12:28:53.581454+0000","last_peered":"2023-10-30T12:28:53.581454+0000","last_clean":"2023-10-30T12:28:53.581454+0000","last_became_active":"2023-10-30T12:28:50.967651+0000","last_became_peered":"2023-10-30T12:28:50.967651+0000","last_unstale":"2023-10-30T12:28:53.581454+0000","last_undegraded":"2023-10-30T12:28:53.581454+0000","last_fullsized":"2023-10-30T12:28:53.581454+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:39:31.893024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579014+0000","last_change":"2023-10-30T12:28:50.962609+0000","last_active":"2023-10-30T12:28:53.579014+0000","last_peered":"2023-10-30T12:28:53.579014+0000","last_clean":"2023-10-30T12:28:53.579014+0000","last_became_active":"2023-10-30T12:28:50.962290+0000","last_became_peered":"2023-10-30T12:28:50.962290+0000","last_unstale":"2023-10-30T12:28:53.579014+0000","last_undegraded":"2023-10-30T12:28:53.579014+0000","last_fullsized":"2023-10-30T12:28:53.579014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:44:35.484971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581381+0000","last_change":"2023-10-30T12:28:50.966653+0000","last_active":"2023-10-30T12:28:53.581381+0000","last_peered":"2023-10-30T12:28:53.581381+0000","last_clean":"2023-10-30T12:28:53.581381+0000","last_became_active":"2023-10-30T12:28:50.965713+0000","last_became_peered":"2023-10-30T12:28:50.965713+0000","last_unstale":"2023-10-30T12:28:53.581381+0000","last_undegraded":"2023-10-30T12:28:53.581381+0000","last_fullsized":"2023-10-30T12:28:53.581381+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:10:31.600981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579351+0000","last_change":"2023-10-30T12:28:50.951300+0000","last_active":"2023-10-30T12:28:53.579351+0000","last_peered":"2023-10-30T12:28:53.579351+0000","last_clean":"2023-10-30T12:28:53.579351+0000","last_became_active":"2023-10-30T12:28:50.951026+0000","last_became_peered":"2023-10-30T12:28:50.951026+0000","last_unstale":"2023-10-30T12:28:53.579351+0000","last_undegraded":"2023-10-30T12:28:53.579351+0000","last_fullsized":"2023-10-30T12:28:53.579351+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:44:58.872457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.938879+0000","last_change":"2023-10-30T12:28:50.963507+0000","last_active":"2023-10-30T12:28:51.938879+0000","last_peered":"2023-10-30T12:28:51.938879+0000","last_clean":"2023-10-30T12:28:51.938879+0000","last_became_active":"2023-10-30T12:28:50.962484+0000","last_became_peered":"2023-10-30T12:28:50.962484+0000","last_unstale":"2023-10-30T12:28:51.938879+0000","last_undegraded":"2023-10-30T12:28:51.938879+0000","last_fullsized":"2023-10-30T12:28:51.938879+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:11:24.160220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931933+0000","last_change":"2023-10-30T12:28:50.933362+0000","last_active":"2023-10-30T12:28:51.931933+0000","last_peered":"2023-10-30T12:28:51.931933+0000","last_clean":"2023-10-30T12:28:51.931933+0000","last_became_active":"2023-10-30T12:28:50.933093+0000","last_became_peered":"2023-10-30T12:28:50.933093+0000","last_unstale":"2023-10-30T12:28:51.931933+0000","last_undegraded":"2023-10-30T12:28:51.931933+0000","last_fullsized":"2023-10-30T12:28:51.931933+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:54:00.852482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579508+0000","last_change":"2023-10-30T12:28:50.952857+0000","last_active":"2023-10-30T12:28:53.579508+0000","last_peered":"2023-10-30T12:28:53.579508+0000","last_clean":"2023-10-30T12:28:53.579508+0000","last_became_active":"2023-10-30T12:28:50.952531+0000","last_became_peered":"2023-10-30T12:28:50.952531+0000","last_unstale":"2023-10-30T12:28:53.579508+0000","last_undegraded":"2023-10-30T12:28:53.579508+0000","last_fullsized":"2023-10-30T12:28:53.579508+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:01:03.980850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.580366+0000","last_change":"2023-10-30T12:28:50.933408+0000","last_active":"2023-10-30T12:28:53.580366+0000","last_peered":"2023-10-30T12:28:53.580366+0000","last_clean":"2023-10-30T12:28:53.580366+0000","last_became_active":"2023-10-30T12:28:50.933124+0000","last_became_peered":"2023-10-30T12:28:50.933124+0000","last_unstale":"2023-10-30T12:28:53.580366+0000","last_undegraded":"2023-10-30T12:28:53.580366+0000","last_fullsized":"2023-10-30T12:28:53.580366+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:53:50.772420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581630+0000","last_change":"2023-10-30T12:28:50.966388+0000","last_active":"2023-10-30T12:28:53.581630+0000","last_peered":"2023-10-30T12:28:53.581630+0000","last_clean":"2023-10-30T12:28:53.581630+0000","last_became_active":"2023-10-30T12:28:50.965639+0000","last_became_peered":"2023-10-30T12:28:50.965639+0000","last_unstale":"2023-10-30T12:28:53.581630+0000","last_undegraded":"2023-10-30T12:28:53.581630+0000","last_fullsized":"2023-10-30T12:28:53.581630+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:55:46.506378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.131583+0000","last_change":"2023-10-30T12:28:50.955391+0000","last_active":"2023-10-30T12:28:53.131583+0000","last_peered":"2023-10-30T12:28:53.131583+0000","last_clean":"2023-10-30T12:28:53.131583+0000","last_became_active":"2023-10-30T12:28:50.955084+0000","last_became_peered":"2023-10-30T12:28:50.955084+0000","last_unstale":"2023-10-30T12:28:53.131583+0000","last_undegraded":"2023-10-30T12:28:53.131583+0000","last_fullsized":"2023-10-30T12:28:53.131583+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:08:52.389085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.936598+0000","last_change":"2023-10-30T12:28:50.945609+0000","last_active":"2023-10-30T12:28:51.936598+0000","last_peered":"2023-10-30T12:28:51.936598+0000","last_clean":"2023-10-30T12:28:51.936598+0000","last_became_active":"2023-10-30T12:28:50.944775+0000","last_became_peered":"2023-10-30T12:28:50.944775+0000","last_unstale":"2023-10-30T12:28:51.936598+0000","last_undegraded":"2023-10-30T12:28:51.936598+0000","last_fullsized":"2023-10-30T12:28:51.936598+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:08:55.718193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.939146+0000","last_change":"2023-10-30T12:28:50.941378+0000","last_active":"2023-10-30T12:28:51.939146+0000","last_peered":"2023-10-30T12:28:51.939146+0000","last_clean":"2023-10-30T12:28:51.939146+0000","last_became_active":"2023-10-30T12:28:50.940567+0000","last_became_peered":"2023-10-30T12:28:50.940567+0000","last_unstale":"2023-10-30T12:28:51.939146+0000","last_undegraded":"2023-10-30T12:28:51.939146+0000","last_fullsized":"2023-10-30T12:28:51.939146+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:31:23.772343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'5","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.435783+0000","last_change":"2023-10-30T12:28:50.395670+0000","last_active":"2023-10-30T12:29:01.435783+0000","last_peered":"2023-10-30T12:29:01.435783+0000","last_clean":"2023-10-30T12:29:01.435783+0000","last_became_active":"2023-10-30T12:28:50.395382+0000","last_became_peered":"2023-10-30T12:28:50.395382+0000","last_unstale":"2023-10-30T12:29:01.435783+0000","last_undegraded":"2023-10-30T12:29:01.435783+0000","last_fullsized":"2023-10-30T12:29:01.435783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:14.858121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.580427+0000","last_change":"2023-10-30T12:28:50.953202+0000","last_active":"2023-10-30T12:28:53.580427+0000","last_peered":"2023-10-30T12:28:53.580427+0000","last_clean":"2023-10-30T12:28:53.580427+0000","last_became_active":"2023-10-30T12:28:50.952613+0000","last_became_peered":"2023-10-30T12:28:50.952613+0000","last_unstale":"2023-10-30T12:28:53.580427+0000","last_undegraded":"2023-10-30T12:28:53.580427+0000","last_fullsized":"2023-10-30T12:28:53.580427+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:38:56.947997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.959082+0000","last_change":"2023-10-30T12:28:50.384598+0000","last_active":"2023-10-30T12:29:06.959082+0000","last_peered":"2023-10-30T12:29:06.959082+0000","last_clean":"2023-10-30T12:29:06.959082+0000","last_became_active":"2023-10-30T12:28:50.383898+0000","last_became_peered":"2023-10-30T12:28:50.383898+0000","last_unstale":"2023-10-30T12:29:06.959082+0000","last_undegraded":"2023-10-30T12:29:06.959082+0000","last_fullsized":"2023-10-30T12:29:06.959082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:09:50.290004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.954822+0000","last_change":"2023-10-30T12:28:50.386193+0000","last_active":"2023-10-30T12:29:06.954822+0000","last_peered":"2023-10-30T12:29:06.954822+0000","last_clean":"2023-10-30T12:29:06.954822+0000","last_became_active":"2023-10-30T12:28:50.385963+0000","last_became_peered":"2023-10-30T12:28:50.385963+0000","last_unstale":"2023-10-30T12:29:06.954822+0000","last_undegraded":"2023-10-30T12:29:06.954822+0000","last_fullsized":"2023-10-30T12:29:06.954822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:00:55.603212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.578966+0000","last_change":"2023-10-30T12:28:50.936038+0000","last_active":"2023-10-30T12:28:53.578966+0000","last_peered":"2023-10-30T12:28:53.578966+0000","last_clean":"2023-10-30T12:28:53.578966+0000","last_became_active":"2023-10-30T12:28:50.935764+0000","last_became_peered":"2023-10-30T12:28:50.935764+0000","last_unstale":"2023-10-30T12:28:53.578966+0000","last_undegraded":"2023-10-30T12:28:53.578966+0000","last_fullsized":"2023-10-30T12:28:53.578966+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:23:01.597158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581945+0000","last_change":"2023-10-30T12:28:50.392213+0000","last_active":"2023-10-30T12:28:53.581945+0000","last_peered":"2023-10-30T12:28:53.581945+0000","last_clean":"2023-10-30T12:28:53.581945+0000","last_became_active":"2023-10-30T12:28:50.391494+0000","last_became_peered":"2023-10-30T12:28:50.391494+0000","last_unstale":"2023-10-30T12:28:53.581945+0000","last_undegraded":"2023-10-30T12:28:53.581945+0000","last_fullsized":"2023-10-30T12:28:53.581945+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:38:37.301840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931619+0000","last_change":"2023-10-30T12:28:50.933319+0000","last_active":"2023-10-30T12:28:51.931619+0000","last_peered":"2023-10-30T12:28:51.931619+0000","last_clean":"2023-10-30T12:28:51.931619+0000","last_became_active":"2023-10-30T12:28:50.933062+0000","last_became_peered":"2023-10-30T12:28:50.933062+0000","last_unstale":"2023-10-30T12:28:51.931619+0000","last_undegraded":"2023-10-30T12:28:51.931619+0000","last_fullsized":"2023-10-30T12:28:51.931619+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:26:48.475038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.424753+0000","last_change":"2023-10-30T12:28:50.394368+0000","last_active":"2023-10-30T12:29:01.424753+0000","last_peered":"2023-10-30T12:29:01.424753+0000","last_clean":"2023-10-30T12:29:01.424753+0000","last_became_active":"2023-10-30T12:28:50.393686+0000","last_became_peered":"2023-10-30T12:28:50.393686+0000","last_unstale":"2023-10-30T12:29:01.424753+0000","last_undegraded":"2023-10-30T12:29:01.424753+0000","last_fullsized":"2023-10-30T12:29:01.424753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:43:17.726462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.582247+0000","last_change":"2023-10-30T12:28:50.963703+0000","last_active":"2023-10-30T12:28:53.582247+0000","last_peered":"2023-10-30T12:28:53.582247+0000","last_clean":"2023-10-30T12:28:53.582247+0000","last_became_active":"2023-10-30T12:28:50.963105+0000","last_became_peered":"2023-10-30T12:28:50.963105+0000","last_unstale":"2023-10-30T12:28:53.582247+0000","last_undegraded":"2023-10-30T12:28:53.582247+0000","last_fullsized":"2023-10-30T12:28:53.582247+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:32.423943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.420755+0000","last_change":"2023-10-30T12:28:50.397679+0000","last_active":"2023-10-30T12:29:01.420755+0000","last_peered":"2023-10-30T12:29:01.420755+0000","last_clean":"2023-10-30T12:29:01.420755+0000","last_became_active":"2023-10-30T12:28:50.397041+0000","last_became_peered":"2023-10-30T12:28:50.397041+0000","last_unstale":"2023-10-30T12:29:01.420755+0000","last_undegraded":"2023-10-30T12:29:01.420755+0000","last_fullsized":"2023-10-30T12:29:01.420755+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:52:22.564616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.580443+0000","last_change":"2023-10-30T12:28:50.966838+0000","last_active":"2023-10-30T12:28:53.580443+0000","last_peered":"2023-10-30T12:28:53.580443+0000","last_clean":"2023-10-30T12:28:53.580443+0000","last_became_active":"2023-10-30T12:28:50.966569+0000","last_became_peered":"2023-10-30T12:28:50.966569+0000","last_unstale":"2023-10-30T12:28:53.580443+0000","last_undegraded":"2023-10-30T12:28:53.580443+0000","last_fullsized":"2023-10-30T12:28:53.580443+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:44:33.875449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:56.364393+0000","last_change":"2023-10-30T12:28:50.391527+0000","last_active":"2023-10-30T12:28:56.364393+0000","last_peered":"2023-10-30T12:28:56.364393+0000","last_clean":"2023-10-30T12:28:56.364393+0000","last_became_active":"2023-10-30T12:28:50.391260+0000","last_became_peered":"2023-10-30T12:28:50.391260+0000","last_unstale":"2023-10-30T12:28:56.364393+0000","last_undegraded":"2023-10-30T12:28:56.364393+0000","last_fullsized":"2023-10-30T12:28:56.364393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:29:34.772490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.940211+0000","last_change":"2023-10-30T12:28:50.975441+0000","last_active":"2023-10-30T12:28:51.940211+0000","last_peered":"2023-10-30T12:28:51.940211+0000","last_clean":"2023-10-30T12:28:51.940211+0000","last_became_active":"2023-10-30T12:28:50.974843+0000","last_became_peered":"2023-10-30T12:28:50.974843+0000","last_unstale":"2023-10-30T12:28:51.940211+0000","last_undegraded":"2023-10-30T12:28:51.940211+0000","last_fullsized":"2023-10-30T12:28:51.940211+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:06:12.060752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"22'11","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:15.025529+0000","last_change":"2023-10-30T12:28:50.374838+0000","last_active":"2023-10-30T12:29:15.025529+0000","last_peered":"2023-10-30T12:29:15.025529+0000","last_clean":"2023-10-30T12:29:15.025529+0000","last_became_active":"2023-10-30T12:28:50.373581+0000","last_became_peered":"2023-10-30T12:28:50.373581+0000","last_unstale":"2023-10-30T12:29:15.025529+0000","last_undegraded":"2023-10-30T12:29:15.025529+0000","last_fullsized":"2023-10-30T12:29:15.025529+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:14:51.506745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2685,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.029046+0000","last_change":"2023-10-30T12:28:50.974012+0000","last_active":"2023-10-30T12:28:53.029046+0000","last_peered":"2023-10-30T12:28:53.029046+0000","last_clean":"2023-10-30T12:28:53.029046+0000","last_became_active":"2023-10-30T12:28:50.973269+0000","last_became_peered":"2023-10-30T12:28:50.973269+0000","last_unstale":"2023-10-30T12:28:53.029046+0000","last_undegraded":"2023-10-30T12:28:53.029046+0000","last_fullsized":"2023-10-30T12:28:53.029046+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:19:21.993898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"22'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.957225+0000","last_change":"2023-10-30T12:28:50.366590+0000","last_active":"2023-10-30T12:29:06.957225+0000","last_peered":"2023-10-30T12:29:06.957225+0000","last_clean":"2023-10-30T12:29:06.957225+0000","last_became_active":"2023-10-30T12:28:50.366269+0000","last_became_peered":"2023-10-30T12:28:50.366269+0000","last_unstale":"2023-10-30T12:29:06.957225+0000","last_undegraded":"2023-10-30T12:29:06.957225+0000","last_fullsized":"2023-10-30T12:29:06.957225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:15:11.963438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579425+0000","last_change":"2023-10-30T12:28:50.962058+0000","last_active":"2023-10-30T12:28:53.579425+0000","last_peered":"2023-10-30T12:28:53.579425+0000","last_clean":"2023-10-30T12:28:53.579425+0000","last_became_active":"2023-10-30T12:28:50.961805+0000","last_became_peered":"2023-10-30T12:28:50.961805+0000","last_unstale":"2023-10-30T12:28:53.579425+0000","last_undegraded":"2023-10-30T12:28:53.579425+0000","last_fullsized":"2023-10-30T12:28:53.579425+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:45:16.373386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.427282+0000","last_change":"2023-10-30T12:28:50.397244+0000","last_active":"2023-10-30T12:29:01.427282+0000","last_peered":"2023-10-30T12:29:01.427282+0000","last_clean":"2023-10-30T12:29:01.427282+0000","last_became_active":"2023-10-30T12:28:50.396727+0000","last_became_peered":"2023-10-30T12:28:50.396727+0000","last_unstale":"2023-10-30T12:29:01.427282+0000","last_undegraded":"2023-10-30T12:29:01.427282+0000","last_fullsized":"2023-10-30T12:29:01.427282+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:33.493343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028618+0000","last_change":"2023-10-30T12:28:50.981498+0000","last_active":"2023-10-30T12:28:53.028618+0000","last_peered":"2023-10-30T12:28:53.028618+0000","last_clean":"2023-10-30T12:28:53.028618+0000","last_became_active":"2023-10-30T12:28:50.981063+0000","last_became_peered":"2023-10-30T12:28:50.981063+0000","last_unstale":"2023-10-30T12:28:53.028618+0000","last_undegraded":"2023-10-30T12:28:53.028618+0000","last_fullsized":"2023-10-30T12:28:53.028618+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:55:17.971880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.963531+0000","last_change":"2023-10-30T12:28:50.383880+0000","last_active":"2023-10-30T12:29:06.963531+0000","last_peered":"2023-10-30T12:29:06.963531+0000","last_clean":"2023-10-30T12:29:06.963531+0000","last_became_active":"2023-10-30T12:28:50.383570+0000","last_became_peered":"2023-10-30T12:28:50.383570+0000","last_unstale":"2023-10-30T12:29:06.963531+0000","last_undegraded":"2023-10-30T12:29:06.963531+0000","last_fullsized":"2023-10-30T12:29:06.963531+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:19:42.047520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":846,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.936983+0000","last_change":"2023-10-30T12:28:50.950312+0000","last_active":"2023-10-30T12:28:51.936983+0000","last_peered":"2023-10-30T12:28:51.936983+0000","last_clean":"2023-10-30T12:28:51.936983+0000","last_became_active":"2023-10-30T12:28:50.949485+0000","last_became_peered":"2023-10-30T12:28:50.949485+0000","last_unstale":"2023-10-30T12:28:51.936983+0000","last_undegraded":"2023-10-30T12:28:51.936983+0000","last_fullsized":"2023-10-30T12:28:51.936983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:13:04.669915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"22'15","reported_seq":27,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.968260+0000","last_change":"2023-10-30T12:28:50.362979+0000","last_active":"2023-10-30T12:29:06.968260+0000","last_peered":"2023-10-30T12:29:06.968260+0000","last_clean":"2023-10-30T12:29:06.968260+0000","last_became_active":"2023-10-30T12:28:50.362624+0000","last_became_peered":"2023-10-30T12:28:50.362624+0000","last_unstale":"2023-10-30T12:29:06.968260+0000","last_undegraded":"2023-10-30T12:29:06.968260+0000","last_fullsized":"2023-10-30T12:29:06.968260+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:01:13.111816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028233+0000","last_change":"2023-10-30T12:28:50.979815+0000","last_active":"2023-10-30T12:28:53.028233+0000","last_peered":"2023-10-30T12:28:53.028233+0000","last_clean":"2023-10-30T12:28:53.028233+0000","last_became_active":"2023-10-30T12:28:50.979344+0000","last_became_peered":"2023-10-30T12:28:50.979344+0000","last_unstale":"2023-10-30T12:28:53.028233+0000","last_undegraded":"2023-10-30T12:28:53.028233+0000","last_fullsized":"2023-10-30T12:28:53.028233+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:35:37.018282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:56.363098+0000","last_change":"2023-10-30T12:28:50.382175+0000","last_active":"2023-10-30T12:28:56.363098+0000","last_peered":"2023-10-30T12:28:56.363098+0000","last_clean":"2023-10-30T12:28:56.363098+0000","last_became_active":"2023-10-30T12:28:50.381125+0000","last_became_peered":"2023-10-30T12:28:50.381125+0000","last_unstale":"2023-10-30T12:28:56.363098+0000","last_undegraded":"2023-10-30T12:28:56.363098+0000","last_fullsized":"2023-10-30T12:28:56.363098+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:36:02.652665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028807+0000","last_change":"2023-10-30T12:28:50.978034+0000","last_active":"2023-10-30T12:28:53.028807+0000","last_peered":"2023-10-30T12:28:53.028807+0000","last_clean":"2023-10-30T12:28:53.028807+0000","last_became_active":"2023-10-30T12:28:50.977585+0000","last_became_peered":"2023-10-30T12:28:50.977585+0000","last_unstale":"2023-10-30T12:28:53.028807+0000","last_undegraded":"2023-10-30T12:28:53.028807+0000","last_fullsized":"2023-10-30T12:28:53.028807+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:42:56.806184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:56.364009+0000","last_change":"2023-10-30T12:28:50.367258+0000","last_active":"2023-10-30T12:28:56.364009+0000","last_peered":"2023-10-30T12:28:56.364009+0000","last_clean":"2023-10-30T12:28:56.364009+0000","last_became_active":"2023-10-30T12:28:50.366974+0000","last_became_peered":"2023-10-30T12:28:50.366974+0000","last_unstale":"2023-10-30T12:28:56.364009+0000","last_undegraded":"2023-10-30T12:28:56.364009+0000","last_fullsized":"2023-10-30T12:28:56.364009+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:14:48.823902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.029189+0000","last_change":"2023-10-30T12:28:50.981784+0000","last_active":"2023-10-30T12:28:53.029189+0000","last_peered":"2023-10-30T12:28:53.029189+0000","last_clean":"2023-10-30T12:28:53.029189+0000","last_became_active":"2023-10-30T12:28:50.981193+0000","last_became_peered":"2023-10-30T12:28:50.981193+0000","last_unstale":"2023-10-30T12:28:53.029189+0000","last_undegraded":"2023-10-30T12:28:53.029189+0000","last_fullsized":"2023-10-30T12:28:53.029189+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:36:17.384732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:52.280239+0000","last_change":"2023-10-30T12:28:50.368011+0000","last_active":"2023-10-30T12:28:52.280239+0000","last_peered":"2023-10-30T12:28:52.280239+0000","last_clean":"2023-10-30T12:28:52.280239+0000","last_became_active":"2023-10-30T12:28:50.367730+0000","last_became_peered":"2023-10-30T12:28:50.367730+0000","last_unstale":"2023-10-30T12:28:52.280239+0000","last_undegraded":"2023-10-30T12:28:52.280239+0000","last_fullsized":"2023-10-30T12:28:52.280239+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:59:41.946550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.582218+0000","last_change":"2023-10-30T12:28:50.968262+0000","last_active":"2023-10-30T12:28:53.582218+0000","last_peered":"2023-10-30T12:28:53.582218+0000","last_clean":"2023-10-30T12:28:53.582218+0000","last_became_active":"2023-10-30T12:28:50.966246+0000","last_became_peered":"2023-10-30T12:28:50.966246+0000","last_unstale":"2023-10-30T12:28:53.582218+0000","last_undegraded":"2023-10-30T12:28:53.582218+0000","last_fullsized":"2023-10-30T12:28:53.582218+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:00:01.991935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.962658+0000","last_change":"2023-10-30T12:28:50.394060+0000","last_active":"2023-10-30T12:29:06.962658+0000","last_peered":"2023-10-30T12:29:06.962658+0000","last_clean":"2023-10-30T12:29:06.962658+0000","last_became_active":"2023-10-30T12:28:50.393642+0000","last_became_peered":"2023-10-30T12:28:50.393642+0000","last_unstale":"2023-10-30T12:29:06.962658+0000","last_undegraded":"2023-10-30T12:29:06.962658+0000","last_fullsized":"2023-10-30T12:29:06.962658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:21.643882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.028738+0000","last_change":"2023-10-30T12:28:50.976814+0000","last_active":"2023-10-30T12:28:53.028738+0000","last_peered":"2023-10-30T12:28:53.028738+0000","last_clean":"2023-10-30T12:28:53.028738+0000","last_became_active":"2023-10-30T12:28:50.976343+0000","last_became_peered":"2023-10-30T12:28:50.976343+0000","last_unstale":"2023-10-30T12:28:53.028738+0000","last_undegraded":"2023-10-30T12:28:53.028738+0000","last_fullsized":"2023-10-30T12:28:53.028738+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:33:23.491998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.946067+0000","last_change":"2023-10-30T12:28:41.203990+0000","last_active":"2023-10-30T12:28:50.946067+0000","last_peered":"2023-10-30T12:28:50.946067+0000","last_clean":"2023-10-30T12:28:50.946067+0000","last_became_active":"2023-10-30T12:28:41.201796+0000","last_became_peered":"2023-10-30T12:28:41.201796+0000","last_unstale":"2023-10-30T12:28:50.946067+0000","last_undegraded":"2023-10-30T12:28:50.946067+0000","last_fullsized":"2023-10-30T12:28:50.946067+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:39.218257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:39.218257+0000","last_clean_scrub_stamp":"2023-10-30T12:28:39.218257+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:21:00.605462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.426165+0000","last_change":"2023-10-30T12:28:50.372651+0000","last_active":"2023-10-30T12:29:01.426165+0000","last_peered":"2023-10-30T12:29:01.426165+0000","last_clean":"2023-10-30T12:29:01.426165+0000","last_became_active":"2023-10-30T12:28:50.367593+0000","last_became_peered":"2023-10-30T12:28:50.367593+0000","last_unstale":"2023-10-30T12:29:01.426165+0000","last_undegraded":"2023-10-30T12:29:01.426165+0000","last_fullsized":"2023-10-30T12:29:01.426165+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:32:10.492061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579128+0000","last_change":"2023-10-30T12:28:50.970536+0000","last_active":"2023-10-30T12:28:53.579128+0000","last_peered":"2023-10-30T12:28:53.579128+0000","last_clean":"2023-10-30T12:28:53.579128+0000","last_became_active":"2023-10-30T12:28:50.970326+0000","last_became_peered":"2023-10-30T12:28:50.970326+0000","last_unstale":"2023-10-30T12:28:53.579128+0000","last_undegraded":"2023-10-30T12:28:53.579128+0000","last_fullsized":"2023-10-30T12:28:53.579128+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:53:54.589174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.958451+0000","last_change":"2023-10-30T12:28:50.392461+0000","last_active":"2023-10-30T12:29:06.958451+0000","last_peered":"2023-10-30T12:29:06.958451+0000","last_clean":"2023-10-30T12:29:06.958451+0000","last_became_active":"2023-10-30T12:28:50.391824+0000","last_became_peered":"2023-10-30T12:28:50.391824+0000","last_unstale":"2023-10-30T12:29:06.958451+0000","last_undegraded":"2023-10-30T12:29:06.958451+0000","last_fullsized":"2023-10-30T12:29:06.958451+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:56:10.373046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579229+0000","last_change":"2023-10-30T12:28:50.953046+0000","last_active":"2023-10-30T12:28:53.579229+0000","last_peered":"2023-10-30T12:28:53.579229+0000","last_clean":"2023-10-30T12:28:53.579229+0000","last_became_active":"2023-10-30T12:28:50.952625+0000","last_became_peered":"2023-10-30T12:28:50.952625+0000","last_unstale":"2023-10-30T12:28:53.579229+0000","last_undegraded":"2023-10-30T12:28:53.579229+0000","last_fullsized":"2023-10-30T12:28:53.579229+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:00:14.643989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.962549+0000","last_change":"2023-10-30T12:28:50.367547+0000","last_active":"2023-10-30T12:29:06.962549+0000","last_peered":"2023-10-30T12:29:06.962549+0000","last_clean":"2023-10-30T12:29:06.962549+0000","last_became_active":"2023-10-30T12:28:50.367230+0000","last_became_peered":"2023-10-30T12:28:50.367230+0000","last_unstale":"2023-10-30T12:29:06.962549+0000","last_undegraded":"2023-10-30T12:29:06.962549+0000","last_fullsized":"2023-10-30T12:29:06.962549+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:42.570786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931639+0000","last_change":"2023-10-30T12:28:50.928888+0000","last_active":"2023-10-30T12:28:51.931639+0000","last_peered":"2023-10-30T12:28:51.931639+0000","last_clean":"2023-10-30T12:28:51.931639+0000","last_became_active":"2023-10-30T12:28:50.928608+0000","last_became_peered":"2023-10-30T12:28:50.928608+0000","last_unstale":"2023-10-30T12:28:51.931639+0000","last_undegraded":"2023-10-30T12:28:51.931639+0000","last_fullsized":"2023-10-30T12:28:51.931639+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:40:51.042396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.582018+0000","last_change":"2023-10-30T12:28:50.391240+0000","last_active":"2023-10-30T12:28:53.582018+0000","last_peered":"2023-10-30T12:28:53.582018+0000","last_clean":"2023-10-30T12:28:53.582018+0000","last_became_active":"2023-10-30T12:28:50.390864+0000","last_became_peered":"2023-10-30T12:28:50.390864+0000","last_unstale":"2023-10-30T12:28:53.582018+0000","last_undegraded":"2023-10-30T12:28:53.582018+0000","last_fullsized":"2023-10-30T12:28:53.582018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:59:40.285830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.939756+0000","last_change":"2023-10-30T12:28:50.949990+0000","last_active":"2023-10-30T12:28:51.939756+0000","last_peered":"2023-10-30T12:28:51.939756+0000","last_clean":"2023-10-30T12:28:51.939756+0000","last_became_active":"2023-10-30T12:28:50.949122+0000","last_became_peered":"2023-10-30T12:28:50.949122+0000","last_unstale":"2023-10-30T12:28:51.939756+0000","last_undegraded":"2023-10-30T12:28:51.939756+0000","last_fullsized":"2023-10-30T12:28:51.939756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:30:56.099810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931803+0000","last_change":"2023-10-30T12:28:50.358700+0000","last_active":"2023-10-30T12:28:51.931803+0000","last_peered":"2023-10-30T12:28:51.931803+0000","last_clean":"2023-10-30T12:28:51.931803+0000","last_became_active":"2023-10-30T12:28:50.358310+0000","last_became_peered":"2023-10-30T12:28:50.358310+0000","last_unstale":"2023-10-30T12:28:51.931803+0000","last_undegraded":"2023-10-30T12:28:51.931803+0000","last_fullsized":"2023-10-30T12:28:51.931803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:23:30.910622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.582250+0000","last_change":"2023-10-30T12:28:50.967650+0000","last_active":"2023-10-30T12:28:53.582250+0000","last_peered":"2023-10-30T12:28:53.582250+0000","last_clean":"2023-10-30T12:28:53.582250+0000","last_became_active":"2023-10-30T12:28:50.965626+0000","last_became_peered":"2023-10-30T12:28:50.965626+0000","last_unstale":"2023-10-30T12:28:53.582250+0000","last_undegraded":"2023-10-30T12:28:53.582250+0000","last_fullsized":"2023-10-30T12:28:53.582250+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:49.050703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.029231+0000","last_change":"2023-10-30T12:28:50.395645+0000","last_active":"2023-10-30T12:28:53.029231+0000","last_peered":"2023-10-30T12:28:53.029231+0000","last_clean":"2023-10-30T12:28:53.029231+0000","last_became_active":"2023-10-30T12:28:50.395090+0000","last_became_peered":"2023-10-30T12:28:50.395090+0000","last_unstale":"2023-10-30T12:28:53.029231+0000","last_undegraded":"2023-10-30T12:28:53.029231+0000","last_fullsized":"2023-10-30T12:28:53.029231+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:45:44.678383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.931844+0000","last_change":"2023-10-30T12:28:50.929586+0000","last_active":"2023-10-30T12:28:51.931844+0000","last_peered":"2023-10-30T12:28:51.931844+0000","last_clean":"2023-10-30T12:28:51.931844+0000","last_became_active":"2023-10-30T12:28:50.929337+0000","last_became_peered":"2023-10-30T12:28:50.929337+0000","last_unstale":"2023-10-30T12:28:51.931844+0000","last_undegraded":"2023-10-30T12:28:51.931844+0000","last_fullsized":"2023-10-30T12:28:51.931844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:11:13.067698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.958476+0000","last_change":"2023-10-30T12:28:50.380474+0000","last_active":"2023-10-30T12:29:06.958476+0000","last_peered":"2023-10-30T12:29:06.958476+0000","last_clean":"2023-10-30T12:29:06.958476+0000","last_became_active":"2023-10-30T12:28:50.380247+0000","last_became_peered":"2023-10-30T12:28:50.380247+0000","last_unstale":"2023-10-30T12:29:06.958476+0000","last_undegraded":"2023-10-30T12:29:06.958476+0000","last_fullsized":"2023-10-30T12:29:06.958476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:52:03.952617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579476+0000","last_change":"2023-10-30T12:28:50.973361+0000","last_active":"2023-10-30T12:28:53.579476+0000","last_peered":"2023-10-30T12:28:53.579476+0000","last_clean":"2023-10-30T12:28:53.579476+0000","last_became_active":"2023-10-30T12:28:50.973086+0000","last_became_peered":"2023-10-30T12:28:50.973086+0000","last_unstale":"2023-10-30T12:28:53.579476+0000","last_undegraded":"2023-10-30T12:28:53.579476+0000","last_fullsized":"2023-10-30T12:28:53.579476+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:49:48.589889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.421918+0000","last_change":"2023-10-30T12:28:50.386277+0000","last_active":"2023-10-30T12:29:01.421918+0000","last_peered":"2023-10-30T12:29:01.421918+0000","last_clean":"2023-10-30T12:29:01.421918+0000","last_became_active":"2023-10-30T12:28:50.385924+0000","last_became_peered":"2023-10-30T12:28:50.385924+0000","last_unstale":"2023-10-30T12:29:01.421918+0000","last_undegraded":"2023-10-30T12:29:01.421918+0000","last_fullsized":"2023-10-30T12:29:01.421918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:35:03.837523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.580322+0000","last_change":"2023-10-30T12:28:50.929670+0000","last_active":"2023-10-30T12:28:53.580322+0000","last_peered":"2023-10-30T12:28:53.580322+0000","last_clean":"2023-10-30T12:28:53.580322+0000","last_became_active":"2023-10-30T12:28:50.929259+0000","last_became_peered":"2023-10-30T12:28:50.929259+0000","last_unstale":"2023-10-30T12:28:53.580322+0000","last_undegraded":"2023-10-30T12:28:53.580322+0000","last_fullsized":"2023-10-30T12:28:53.580322+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:50:16.507660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.420373+0000","last_change":"2023-10-30T12:28:50.386426+0000","last_active":"2023-10-30T12:29:01.420373+0000","last_peered":"2023-10-30T12:29:01.420373+0000","last_clean":"2023-10-30T12:29:01.420373+0000","last_became_active":"2023-10-30T12:28:50.385305+0000","last_became_peered":"2023-10-30T12:28:50.385305+0000","last_unstale":"2023-10-30T12:29:01.420373+0000","last_undegraded":"2023-10-30T12:29:01.420373+0000","last_fullsized":"2023-10-30T12:29:01.420373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:13:05.376581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.938368+0000","last_change":"2023-10-30T12:28:50.941607+0000","last_active":"2023-10-30T12:28:51.938368+0000","last_peered":"2023-10-30T12:28:51.938368+0000","last_clean":"2023-10-30T12:28:51.938368+0000","last_became_active":"2023-10-30T12:28:50.940846+0000","last_became_peered":"2023-10-30T12:28:50.940846+0000","last_unstale":"2023-10-30T12:28:51.938368+0000","last_undegraded":"2023-10-30T12:28:51.938368+0000","last_fullsized":"2023-10-30T12:28:51.938368+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:29.837077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.433290+0000","last_change":"2023-10-30T12:28:50.392333+0000","last_active":"2023-10-30T12:29:01.433290+0000","last_peered":"2023-10-30T12:29:01.433290+0000","last_clean":"2023-10-30T12:29:01.433290+0000","last_became_active":"2023-10-30T12:28:50.392068+0000","last_became_peered":"2023-10-30T12:28:50.392068+0000","last_unstale":"2023-10-30T12:29:01.433290+0000","last_undegraded":"2023-10-30T12:29:01.433290+0000","last_fullsized":"2023-10-30T12:29:01.433290+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:47:24.674925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579044+0000","last_change":"2023-10-30T12:28:50.932167+0000","last_active":"2023-10-30T12:28:53.579044+0000","last_peered":"2023-10-30T12:28:53.579044+0000","last_clean":"2023-10-30T12:28:53.579044+0000","last_became_active":"2023-10-30T12:28:50.931898+0000","last_became_peered":"2023-10-30T12:28:50.931898+0000","last_unstale":"2023-10-30T12:28:53.579044+0000","last_undegraded":"2023-10-30T12:28:53.579044+0000","last_fullsized":"2023-10-30T12:28:53.579044+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:44.429045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:52.280154+0000","last_change":"2023-10-30T12:28:50.365055+0000","last_active":"2023-10-30T12:28:52.280154+0000","last_peered":"2023-10-30T12:28:52.280154+0000","last_clean":"2023-10-30T12:28:52.280154+0000","last_became_active":"2023-10-30T12:28:50.364797+0000","last_became_peered":"2023-10-30T12:28:50.364797+0000","last_unstale":"2023-10-30T12:28:52.280154+0000","last_undegraded":"2023-10-30T12:28:52.280154+0000","last_fullsized":"2023-10-30T12:28:52.280154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:06:23.067895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579266+0000","last_change":"2023-10-30T12:28:50.935908+0000","last_active":"2023-10-30T12:28:53.579266+0000","last_peered":"2023-10-30T12:28:53.579266+0000","last_clean":"2023-10-30T12:28:53.579266+0000","last_became_active":"2023-10-30T12:28:50.935657+0000","last_became_peered":"2023-10-30T12:28:50.935657+0000","last_unstale":"2023-10-30T12:28:53.579266+0000","last_undegraded":"2023-10-30T12:28:53.579266+0000","last_fullsized":"2023-10-30T12:28:53.579266+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:36:13.242429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:01.416976+0000","last_change":"2023-10-30T12:28:50.393465+0000","last_active":"2023-10-30T12:29:01.416976+0000","last_peered":"2023-10-30T12:29:01.416976+0000","last_clean":"2023-10-30T12:29:01.416976+0000","last_became_active":"2023-10-30T12:28:50.392807+0000","last_became_peered":"2023-10-30T12:28:50.392807+0000","last_unstale":"2023-10-30T12:29:01.416976+0000","last_undegraded":"2023-10-30T12:29:01.416976+0000","last_fullsized":"2023-10-30T12:29:01.416976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:35:37.976956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:51.938767+0000","last_change":"2023-10-30T12:28:50.954834+0000","last_active":"2023-10-30T12:28:51.938767+0000","last_peered":"2023-10-30T12:28:51.938767+0000","last_clean":"2023-10-30T12:28:51.938767+0000","last_became_active":"2023-10-30T12:28:50.954291+0000","last_became_peered":"2023-10-30T12:28:50.954291+0000","last_unstale":"2023-10-30T12:28:51.938767+0000","last_undegraded":"2023-10-30T12:28:51.938767+0000","last_fullsized":"2023-10-30T12:28:51.938767+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:19:44.345918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581732+0000","last_change":"2023-10-30T12:28:50.963620+0000","last_active":"2023-10-30T12:28:53.581732+0000","last_peered":"2023-10-30T12:28:53.581732+0000","last_clean":"2023-10-30T12:28:53.581732+0000","last_became_active":"2023-10-30T12:28:50.962817+0000","last_became_peered":"2023-10-30T12:28:50.962817+0000","last_unstale":"2023-10-30T12:28:53.581732+0000","last_undegraded":"2023-10-30T12:28:53.581732+0000","last_fullsized":"2023-10-30T12:28:53.581732+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:23:23.276086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:56.367962+0000","last_change":"2023-10-30T12:28:50.402179+0000","last_active":"2023-10-30T12:28:56.367962+0000","last_peered":"2023-10-30T12:28:56.367962+0000","last_clean":"2023-10-30T12:28:56.367962+0000","last_became_active":"2023-10-30T12:28:50.401677+0000","last_became_peered":"2023-10-30T12:28:50.401677+0000","last_unstale":"2023-10-30T12:28:56.367962+0000","last_undegraded":"2023-10-30T12:28:56.367962+0000","last_fullsized":"2023-10-30T12:28:56.367962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:39:44.432214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.581874+0000","last_change":"2023-10-30T12:28:50.968805+0000","last_active":"2023-10-30T12:28:53.581874+0000","last_peered":"2023-10-30T12:28:53.581874+0000","last_clean":"2023-10-30T12:28:53.581874+0000","last_became_active":"2023-10-30T12:28:50.967591+0000","last_became_peered":"2023-10-30T12:28:50.967591+0000","last_unstale":"2023-10-30T12:28:53.581874+0000","last_undegraded":"2023-10-30T12:28:53.581874+0000","last_fullsized":"2023-10-30T12:28:53.581874+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:11:41.973858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.957870+0000","last_change":"2023-10-30T12:28:50.392617+0000","last_active":"2023-10-30T12:29:06.957870+0000","last_peered":"2023-10-30T12:29:06.957870+0000","last_clean":"2023-10-30T12:29:06.957870+0000","last_became_active":"2023-10-30T12:28:50.392021+0000","last_became_peered":"2023-10-30T12:28:50.392021+0000","last_unstale":"2023-10-30T12:29:06.957870+0000","last_undegraded":"2023-10-30T12:29:06.957870+0000","last_fullsized":"2023-10-30T12:29:06.957870+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:52:30.621125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.580378+0000","last_change":"2023-10-30T12:28:50.967491+0000","last_active":"2023-10-30T12:28:53.580378+0000","last_peered":"2023-10-30T12:28:53.580378+0000","last_clean":"2023-10-30T12:28:53.580378+0000","last_became_active":"2023-10-30T12:28:50.967186+0000","last_became_peered":"2023-10-30T12:28:50.967186+0000","last_unstale":"2023-10-30T12:28:53.580378+0000","last_undegraded":"2023-10-30T12:28:53.580378+0000","last_fullsized":"2023-10-30T12:28:53.580378+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:05:23.759940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:56.365702+0000","last_change":"2023-10-30T12:28:50.363230+0000","last_active":"2023-10-30T12:28:56.365702+0000","last_peered":"2023-10-30T12:28:56.365702+0000","last_clean":"2023-10-30T12:28:56.365702+0000","last_became_active":"2023-10-30T12:28:50.362929+0000","last_became_peered":"2023-10-30T12:28:50.362929+0000","last_unstale":"2023-10-30T12:28:56.365702+0000","last_undegraded":"2023-10-30T12:28:56.365702+0000","last_fullsized":"2023-10-30T12:28:56.365702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:41:27.760578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:53.579661+0000","last_change":"2023-10-30T12:28:50.937780+0000","last_active":"2023-10-30T12:28:53.579661+0000","last_peered":"2023-10-30T12:28:53.579661+0000","last_clean":"2023-10-30T12:28:53.579661+0000","last_became_active":"2023-10-30T12:28:50.937458+0000","last_became_peered":"2023-10-30T12:28:50.937458+0000","last_unstale":"2023-10-30T12:28:53.579661+0000","last_undegraded":"2023-10-30T12:28:53.579661+0000","last_fullsized":"2023-10-30T12:28:53.579661+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:02.083169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"22'9","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:06.962486+0000","last_change":"2023-10-30T12:28:50.386782+0000","last_active":"2023-10-30T12:29:06.962486+0000","last_peered":"2023-10-30T12:29:06.962486+0000","last_clean":"2023-10-30T12:29:06.962486+0000","last_became_active":"2023-10-30T12:28:50.386522+0000","last_became_peered":"2023-10-30T12:28:50.386522+0000","last_unstale":"2023-10-30T12:29:06.962486+0000","last_undegraded":"2023-10-30T12:29:06.962486+0000","last_fullsized":"2023-10-30T12:29:06.962486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:44:03.517641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6553,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":13106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113,"ondisk_log_size":113,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542156,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27836,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344004,"statfs":{"total":96636764160,"available":96608260096,"internally_reserved":0,"allocated":839680,"data_stored":670548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0549999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]}]},{"osd":6,"up_from":14,"seq":60129542156,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":245760,"data_stored":81670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0920000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542156,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":241664,"data_stored":79623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94599999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2250000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0760000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542155,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":241664,"data_stored":79836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1180000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.093}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]}]},{"osd":3,"up_from":14,"seq":60129542156,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":249856,"data_stored":82261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.012}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93300000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0680000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542156,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":241664,"data_stored":79063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92400000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542156,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":249856,"data_stored":79808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]}]},{"osd":0,"up_from":14,"seq":60129542156,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":831488,"data_stored":669273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85599999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:29:19.527 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:29:19.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.699+0000 7f9c694d6640 1 Processor -- start 2023-10-30T12:29:19.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.699+0000 7f9c694d6640 1 -- start start 2023-10-30T12:29:19.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.699+0000 7f9c694d6640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9c6415d6f0 0x7f9c6415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:19.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.699+0000 7f9c694d6640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9c640cf160 0x7f9c640cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:19.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.699+0000 7f9c694d6640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c640cb800 con 0x7f9c641538a0 2023-10-30T12:29:19.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.699+0000 7f9c694d6640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c640cb330 con 0x7f9c640cf160 2023-10-30T12:29:19.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.699+0000 7f9c694d6640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c640cad70 con 0x7f9c6415d6f0 2023-10-30T12:29:19.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.699+0000 7f9c62ffd640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9c641538a0 0x7f9c640ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58404/0 (socket says 172.21.15.89:58404) 2023-10-30T12:29:19.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.699+0000 7f9c62ffd640 1 -- 172.21.15.89:0/2654339477 learned_addr learned my addr 172.21.15.89:0/2654339477 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:29:19.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.699+0000 7f9c637fe640 1 --2- 172.21.15.89:0/2654339477 >> v2:172.21.15.89:3300/0 conn(0x7f9c640cf160 0x7f9c640cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:19.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.699+0000 7f9c5a7fc640 1 --2- 172.21.15.89:0/2654339477 >> v2:172.21.15.192:3300/0 conn(0x7f9c6415d6f0 0x7f9c6415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:19.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.699+0000 7f9c627fc640 1 -- 172.21.15.89:0/2654339477 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3587427100 0 0) 0x7f9c640cb800 con 0x7f9c641538a0 2023-10-30T12:29:19.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.699+0000 7f9c627fc640 1 -- 172.21.15.89:0/2654339477 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c50003530 con 0x7f9c641538a0 2023-10-30T12:29:19.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.699+0000 7f9c637fe640 1 -- 172.21.15.89:0/2654339477 >> v2:172.21.15.192:3300/0 conn(0x7f9c6415d6f0 msgr2=0x7f9c6415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:19.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.699+0000 7f9c637fe640 1 --2- 172.21.15.89:0/2654339477 >> v2:172.21.15.192:3300/0 conn(0x7f9c6415d6f0 0x7f9c6415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:19.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.699+0000 7f9c637fe640 1 -- 172.21.15.89:0/2654339477 >> v1:172.21.15.89:6789/0 conn(0x7f9c641538a0 legacy=0x7f9c640ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:19.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.699+0000 7f9c637fe640 1 -- 172.21.15.89:0/2654339477 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c640cb050 con 0x7f9c640cf160 2023-10-30T12:29:19.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.699+0000 7f9c5a7fc640 1 --2- 172.21.15.89:0/2654339477 >> v2:172.21.15.192:3300/0 conn(0x7f9c6415d6f0 0x7f9c6415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:19.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.703+0000 7f9c637fe640 1 --2- 172.21.15.89:0/2654339477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c640cf160 0x7f9c640cf590 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9c640b30a0 tx=0x7f9c5c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba0c2309fd730aeb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:19.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.703+0000 7f9c627fc640 1 -- 172.21.15.89:0/2654339477 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c0a8950 con 0x7f9c640cf160 2023-10-30T12:29:19.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.703+0000 7f9c627fc640 1 -- 172.21.15.89:0/2654339477 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c5c0a8ab0 con 0x7f9c640cf160 2023-10-30T12:29:19.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.703+0000 7f9c627fc640 1 -- 172.21.15.89:0/2654339477 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c0a8e40 con 0x7f9c640cf160 2023-10-30T12:29:19.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.703+0000 7f9c694d6640 1 -- 172.21.15.89:0/2654339477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c640cf160 msgr2=0x7f9c640cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:19.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.703+0000 7f9c694d6640 1 --2- 172.21.15.89:0/2654339477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c640cf160 0x7f9c640cf590 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9c640b30a0 tx=0x7f9c5c01ec50 comp rx=0 tx=0).stop 2023-10-30T12:29:19.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.703+0000 7f9c694d6640 1 -- 172.21.15.89:0/2654339477 shutdown_connections 2023-10-30T12:29:19.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.703+0000 7f9c694d6640 1 --2- 172.21.15.89:0/2654339477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c640cf160 0x7f9c640cf590 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:19.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.703+0000 7f9c694d6640 1 --2- 172.21.15.89:0/2654339477 >> v2:172.21.15.192:3300/0 conn(0x7f9c6415d6f0 0x7f9c6415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:19.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.703+0000 7f9c694d6640 1 -- 172.21.15.89:0/2654339477 >> 172.21.15.89:0/2654339477 conn(0x7f9c6406bfe0 msgr2=0x7f9c6406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:19.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.703+0000 7f9c694d6640 1 -- 172.21.15.89:0/2654339477 shutdown_connections 2023-10-30T12:29:19.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.703+0000 7f9c694d6640 1 -- 172.21.15.89:0/2654339477 wait complete. 2023-10-30T12:29:19.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.707+0000 7f9c694d6640 1 Processor -- start 2023-10-30T12:29:19.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.707+0000 7f9c694d6640 1 -- start start 2023-10-30T12:29:19.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.707+0000 7f9c694d6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c640cf160 0x7f9c64159100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:19.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.707+0000 7f9c694d6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c641538a0 0x7f9c640b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:19.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.707+0000 7f9c694d6640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c6415d6f0 0x7f9c6414f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:19.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.707+0000 7f9c694d6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6414eba0 con 0x7f9c641538a0 2023-10-30T12:29:19.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.707+0000 7f9c694d6640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6414ee80 con 0x7f9c640cf160 2023-10-30T12:29:19.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.707+0000 7f9c694d6640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c641501a0 con 0x7f9c6415d6f0 2023-10-30T12:29:19.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.707+0000 7f9c5a7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c641538a0 0x7f9c640b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:19.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.707+0000 7f9c5a7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c641538a0 0x7f9c640b4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57990/0 (socket says 172.21.15.89:57990) 2023-10-30T12:29:19.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.707+0000 7f9c5a7fc640 1 -- 172.21.15.89:0/3674687480 learned_addr learned my addr 172.21.15.89:0/3674687480 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:19.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.707+0000 7f9c62ffd640 1 --2- 172.21.15.89:0/3674687480 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c640cf160 0x7f9c64159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:19.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.707+0000 7f9c637fe640 1 --2- 172.21.15.89:0/3674687480 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c6415d6f0 0x7f9c6414f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:19.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.707+0000 7f9c5a7fc640 1 -- 172.21.15.89:0/3674687480 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c6415d6f0 msgr2=0x7f9c6414f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:19.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.707+0000 7f9c5a7fc640 1 --2- 172.21.15.89:0/3674687480 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c6415d6f0 0x7f9c6414f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:19.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.707+0000 7f9c5a7fc640 1 -- 172.21.15.89:0/3674687480 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c640cf160 msgr2=0x7f9c64159100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:19.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.707+0000 7f9c5a7fc640 1 --2- 172.21.15.89:0/3674687480 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c640cf160 0x7f9c64159100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:19.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.707+0000 7f9c5a7fc640 1 -- 172.21.15.89:0/3674687480 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5c009420 con 0x7f9c641538a0 2023-10-30T12:29:19.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.707+0000 7f9c5a7fc640 1 --2- 172.21.15.89:0/3674687480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c641538a0 0x7f9c640b4580 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c007f20 tx=0x7f9c4c00d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:19.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.707+0000 7f9c5bfff640 1 -- 172.21.15.89:0/3674687480 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4c00dee0 con 0x7f9c641538a0 2023-10-30T12:29:19.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.707+0000 7f9c5bfff640 1 -- 172.21.15.89:0/3674687480 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c4c00f040 con 0x7f9c641538a0 2023-10-30T12:29:19.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.707+0000 7f9c694d6640 1 -- 172.21.15.89:0/3674687480 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c64150460 con 0x7f9c641538a0 2023-10-30T12:29:19.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.707+0000 7f9c5bfff640 1 -- 172.21.15.89:0/3674687480 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4c003dc0 con 0x7f9c641538a0 2023-10-30T12:29:19.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.707+0000 7f9c694d6640 1 -- 172.21.15.89:0/3674687480 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c64150a20 con 0x7f9c641538a0 2023-10-30T12:29:19.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.707+0000 7f9c694d6640 1 -- 172.21.15.89:0/3674687480 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c64150c40 con 0x7f9c641538a0 2023-10-30T12:29:19.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.711+0000 7f9c5bfff640 1 -- 172.21.15.89:0/3674687480 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9c4c004030 con 0x7f9c641538a0 2023-10-30T12:29:19.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.711+0000 7f9c5bfff640 1 --2- 172.21.15.89:0/3674687480 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9c440664a0 0x7f9c44068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:19.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.711+0000 7f9c5bfff640 1 -- 172.21.15.89:0/3674687480 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9c4c084180 con 0x7f9c641538a0 2023-10-30T12:29:19.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.711+0000 7f9c62ffd640 1 --2- 172.21.15.89:0/3674687480 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9c440664a0 0x7f9c44068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:19.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.711+0000 7f9c62ffd640 1 --2- 172.21.15.89:0/3674687480 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9c440664a0 0x7f9c44068960 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9c540011f0 tx=0x7f9c54009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:19.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.715+0000 7f9c5bfff640 1 -- 172.21.15.89:0/3674687480 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9c4c051610 con 0x7f9c641538a0 2023-10-30T12:29:19.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.863+0000 7f9c694d6640 1 -- 172.21.15.89:0/3674687480 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c640b2a50 con 0x7f9c641538a0 2023-10-30T12:29:19.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.863+0000 7f9c5bfff640 1 -- 172.21.15.89:0/3674687480 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f9c4c054c30 con 0x7f9c641538a0 2023-10-30T12:29:19.868 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:29:19.868 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":22,"fsid":"401641be-db6e-4c16-b3ac-fd84a86c4de4","created":"2023-10-30T12:28:26.198532+0000","modified":"2023-10-30T12:28:51.918285+0000","last_up_change":"2023-10-30T12:28:37.212205+0000","last_in_change":"2023-10-30T12:28:28.517734+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:39.004943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T12:28:48.935909+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T12:28:49.716098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72595c54-fd6e-4a5a-bb14-ff27e9377b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6801","nonce":1673955622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6803","nonce":1673955622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6807","nonce":1673955622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6805","nonce":1673955622}]},"public_addr":"172.21.15.89:6801/1673955622","cluster_addr":"172.21.15.89:6803/1673955622","heartbeat_back_addr":"172.21.15.89:6807/1673955622","heartbeat_front_addr":"172.21.15.89:6805/1673955622","state":["exists","up"]},{"osd":1,"uuid":"0305670b-d7c4-4193-bec2-8b7351ff590a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6817","nonce":3340572039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6819","nonce":3340572039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6823","nonce":3340572039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6821","nonce":3340572039}]},"public_addr":"172.21.15.89:6817/3340572039","cluster_addr":"172.21.15.89:6819/3340572039","heartbeat_back_addr":"172.21.15.89:6823/3340572039","heartbeat_front_addr":"172.21.15.89:6821/3340572039","state":["exists","up"]},{"osd":2,"uuid":"4bab5426-a3c3-46b0-91fc-546c254006c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6809","nonce":1929940799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6811","nonce":1929940799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6815","nonce":1929940799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6813","nonce":1929940799}]},"public_addr":"172.21.15.89:6809/1929940799","cluster_addr":"172.21.15.89:6811/1929940799","heartbeat_back_addr":"172.21.15.89:6815/1929940799","heartbeat_front_addr":"172.21.15.89:6813/1929940799","state":["exists","up"]},{"osd":3,"uuid":"7d47aea8-27cd-42be-8591-d63ad68c90e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6825","nonce":345087939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6827","nonce":345087939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6831","nonce":345087939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6829","nonce":345087939}]},"public_addr":"172.21.15.89:6825/345087939","cluster_addr":"172.21.15.89:6827/345087939","heartbeat_back_addr":"172.21.15.89:6831/345087939","heartbeat_front_addr":"172.21.15.89:6829/345087939","state":["exists","up"]},{"osd":4,"uuid":"2e2ae8f8-d097-4bba-9448-95047fbfacad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6801","nonce":2138125545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6803","nonce":2138125545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6809","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6811","nonce":2138125545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6805","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6807","nonce":2138125545}]},"public_addr":"172.21.15.192:6801/2138125545","cluster_addr":"172.21.15.192:6803/2138125545","heartbeat_back_addr":"172.21.15.192:6811/2138125545","heartbeat_front_addr":"172.21.15.192:6807/2138125545","state":["exists","up"]},{"osd":5,"uuid":"49be5ce5-12f4-4c26-8d39-864b6ff67313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6806","nonce":2360218671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6810","nonce":2360218671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6815","nonce":2360218671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6813","nonce":2360218671}]},"public_addr":"172.21.15.192:6806/2360218671","cluster_addr":"172.21.15.192:6810/2360218671","heartbeat_back_addr":"172.21.15.192:6815/2360218671","heartbeat_front_addr":"172.21.15.192:6813/2360218671","state":["exists","up"]},{"osd":6,"uuid":"48623452-d456-4ea2-a0de-5fa4b37e6d10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6825","nonce":3707671287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6827","nonce":3707671287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6831","nonce":3707671287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6829","nonce":3707671287}]},"public_addr":"172.21.15.192:6825/3707671287","cluster_addr":"172.21.15.192:6827/3707671287","heartbeat_back_addr":"172.21.15.192:6831/3707671287","heartbeat_front_addr":"172.21.15.192:6829/3707671287","state":["exists","up"]},{"osd":7,"uuid":"ff0a4384-54f8-43ec-a3ce-9c8ae3bf8a94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6817","nonce":3111945871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6819","nonce":3111945871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6823","nonce":3111945871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6821","nonce":3111945871}]},"public_addr":"172.21.15.192:6817/3111945871","cluster_addr":"172.21.15.192:6819/3111945871","heartbeat_back_addr":"172.21.15.192:6823/3111945871","heartbeat_front_addr":"172.21.15.192:6821/3111945871","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.183065+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.248650+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.194576+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.397483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.043648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.186624+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.159674+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.100851+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:29:19.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.867+0000 7f9c694d6640 1 -- 172.21.15.89:0/3674687480 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9c440664a0 msgr2=0x7f9c44068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:19.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.867+0000 7f9c694d6640 1 --2- 172.21.15.89:0/3674687480 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9c440664a0 0x7f9c44068960 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9c540011f0 tx=0x7f9c54009040 comp rx=0 tx=0).stop 2023-10-30T12:29:19.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.867+0000 7f9c694d6640 1 -- 172.21.15.89:0/3674687480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c641538a0 msgr2=0x7f9c640b4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:19.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.867+0000 7f9c694d6640 1 --2- 172.21.15.89:0/3674687480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c641538a0 0x7f9c640b4580 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c007f20 tx=0x7f9c4c00d560 comp rx=0 tx=0).stop 2023-10-30T12:29:19.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.867+0000 7f9c694d6640 1 -- 172.21.15.89:0/3674687480 shutdown_connections 2023-10-30T12:29:19.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.867+0000 7f9c694d6640 1 --2- 172.21.15.89:0/3674687480 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9c440664a0 0x7f9c44068960 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:19.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.867+0000 7f9c694d6640 1 --2- 172.21.15.89:0/3674687480 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c6415d6f0 0x7f9c6414f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:19.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.867+0000 7f9c694d6640 1 --2- 172.21.15.89:0/3674687480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c641538a0 0x7f9c640b4580 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:19.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.867+0000 7f9c694d6640 1 --2- 172.21.15.89:0/3674687480 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c640cf160 0x7f9c64159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:19.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.867+0000 7f9c694d6640 1 -- 172.21.15.89:0/3674687480 >> 172.21.15.89:0/3674687480 conn(0x7f9c6406bfe0 msgr2=0x7f9c64152110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:19.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.867+0000 7f9c694d6640 1 -- 172.21.15.89:0/3674687480 shutdown_connections 2023-10-30T12:29:19.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:19.867+0000 7f9c694d6640 1 -- 172.21.15.89:0/3674687480 wait complete. 2023-10-30T12:29:19.889 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T12:28:39.004943+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T12:28:48.935909+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T12:28:49.716098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T12:29:19.889 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T12:29:20.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.027+0000 7fe5d733b640 1 Processor -- start 2023-10-30T12:29:20.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.027+0000 7fe5d733b640 1 -- start start 2023-10-30T12:29:20.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.031+0000 7fe5d733b640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe5d0153840 0x7fe5d015a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:20.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.031+0000 7fe5d733b640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe5d0153c20 0x7fe5d00c3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:20.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.031+0000 7fe5d733b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5d00cb4c0 0x7fe5d00cef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:20.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.031+0000 7fe5d733b640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5d015dbc0 con 0x7fe5d0153840 2023-10-30T12:29:20.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.031+0000 7fe5d733b640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5d015a6b0 con 0x7fe5d0153c20 2023-10-30T12:29:20.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.031+0000 7fe5d733b640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d015d730 con 0x7fe5d00cb4c0 2023-10-30T12:29:20.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.031+0000 7fe5d6339640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe5d0153840 0x7fe5d015a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:20.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.031+0000 7fe5d6339640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe5d0153840 0x7fe5d015a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53372/0 (socket says 172.21.15.89:53372) 2023-10-30T12:29:20.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.031+0000 7fe5d6339640 1 -- 172.21.15.89:0/1948423906 learned_addr learned my addr 172.21.15.89:0/1948423906 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:20.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.031+0000 7fe5d5b38640 1 --2- 172.21.15.89:0/1948423906 >> v2:172.21.15.192:3300/0 conn(0x7fe5d0153c20 0x7fe5d00c3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:20.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.031+0000 7fe5d6b3a640 1 --2- 172.21.15.89:0/1948423906 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5d00cb4c0 0x7fe5d00cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:20.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.031+0000 7fe5d6339640 1 -- 172.21.15.89:0/1948423906 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5d00cb4c0 msgr2=0x7fe5d00cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:20.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.031+0000 7fe5d6339640 1 --2- 172.21.15.89:0/1948423906 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5d00cb4c0 0x7fe5d00cef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.031+0000 7fe5d6339640 1 -- 172.21.15.89:0/1948423906 >> v2:172.21.15.192:3300/0 conn(0x7fe5d0153c20 msgr2=0x7fe5d00c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:20.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.031+0000 7fe5d6339640 1 --2- 172.21.15.89:0/1948423906 >> v2:172.21.15.192:3300/0 conn(0x7fe5d0153c20 0x7fe5d00c3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.031+0000 7fe5d6339640 1 -- 172.21.15.89:0/1948423906 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d00ca8f0 con 0x7fe5d0153840 2023-10-30T12:29:20.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.031+0000 7fe5d6b3a640 1 --2- 172.21.15.89:0/1948423906 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5d00cb4c0 0x7fe5d00cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:20.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.031+0000 7fe5d6339640 1 --2- 172.21.15.89:0/1948423906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5d0153840 0x7fe5d015a170 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe5d008cfe0 tx=0x7fe5cc01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56cf45f0a8b5ea4f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:20.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.035+0000 7fe5d5337640 1 -- 172.21.15.89:0/1948423906 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc0a89b0 con 0x7fe5d0153840 2023-10-30T12:29:20.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.035+0000 7fe5d5337640 1 -- 172.21.15.89:0/1948423906 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5cc0a8b10 con 0x7fe5d0153840 2023-10-30T12:29:20.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.035+0000 7fe5d5337640 1 -- 172.21.15.89:0/1948423906 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc0a8ea0 con 0x7fe5d0153840 2023-10-30T12:29:20.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.035+0000 7fe5d733b640 1 -- 172.21.15.89:0/1948423906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5d0153840 msgr2=0x7fe5d015a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:20.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.035+0000 7fe5d733b640 1 --2- 172.21.15.89:0/1948423906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5d0153840 0x7fe5d015a170 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe5d008cfe0 tx=0x7fe5cc01ecf0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.035+0000 7fe5d733b640 1 -- 172.21.15.89:0/1948423906 shutdown_connections 2023-10-30T12:29:20.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.035+0000 7fe5d733b640 1 --2- 172.21.15.89:0/1948423906 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5d00cb4c0 0x7fe5d00cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.035+0000 7fe5d733b640 1 --2- 172.21.15.89:0/1948423906 >> v2:172.21.15.192:3300/0 conn(0x7fe5d0153c20 0x7fe5d00c3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.035+0000 7fe5d733b640 1 --2- 172.21.15.89:0/1948423906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5d0153840 0x7fe5d015a170 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.035+0000 7fe5d733b640 1 -- 172.21.15.89:0/1948423906 >> 172.21.15.89:0/1948423906 conn(0x7fe5d006be40 msgr2=0x7fe5d006c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:20.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.035+0000 7fe5d733b640 1 -- 172.21.15.89:0/1948423906 shutdown_connections 2023-10-30T12:29:20.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.035+0000 7fe5d733b640 1 -- 172.21.15.89:0/1948423906 wait complete. 2023-10-30T12:29:20.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.035+0000 7fe5d733b640 1 Processor -- start 2023-10-30T12:29:20.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.035+0000 7fe5d733b640 1 -- start start 2023-10-30T12:29:20.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.035+0000 7fe5d733b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5d00cb4c0 0x7fe5d01e4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:20.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.035+0000 7fe5d733b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5d0153840 0x7fe5d01e4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:20.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.035+0000 7fe5d733b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5d0153c20 0x7fe5d00b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:20.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.035+0000 7fe5d733b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d01495d0 con 0x7fe5d00cb4c0 2023-10-30T12:29:20.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.035+0000 7fe5d733b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d01e2ef0 con 0x7fe5d0153c20 2023-10-30T12:29:20.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.035+0000 7fe5d733b640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d0157a10 con 0x7fe5d0153840 2023-10-30T12:29:20.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.035+0000 7fe5d6339640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5d00cb4c0 0x7fe5d01e4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:20.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.035+0000 7fe5d6339640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5d00cb4c0 0x7fe5d01e4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53386/0 (socket says 172.21.15.89:53386) 2023-10-30T12:29:20.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.039+0000 7fe5d6339640 1 -- 172.21.15.89:0/1571964906 learned_addr learned my addr 172.21.15.89:0/1571964906 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:20.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.039+0000 7fe5d5b38640 1 --2- 172.21.15.89:0/1571964906 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5d0153840 0x7fe5d01e4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:20.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.039+0000 7fe5d6b3a640 1 --2- 172.21.15.89:0/1571964906 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5d0153c20 0x7fe5d00b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:20.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.039+0000 7fe5d6339640 1 -- 172.21.15.89:0/1571964906 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5d0153840 msgr2=0x7fe5d01e4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:20.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.039+0000 7fe5d6339640 1 --2- 172.21.15.89:0/1571964906 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5d0153840 0x7fe5d01e4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.039+0000 7fe5d6339640 1 -- 172.21.15.89:0/1571964906 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5d0153c20 msgr2=0x7fe5d00b38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:20.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.039+0000 7fe5d6339640 1 --2- 172.21.15.89:0/1571964906 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5d0153c20 0x7fe5d00b38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.039+0000 7fe5d6339640 1 -- 172.21.15.89:0/1571964906 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5cc0094d0 con 0x7fe5d00cb4c0 2023-10-30T12:29:20.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.039+0000 7fe5d5b38640 1 --2- 172.21.15.89:0/1571964906 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5d0153840 0x7fe5d01e4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:20.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.039+0000 7fe5d6b3a640 1 --2- 172.21.15.89:0/1571964906 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5d0153c20 0x7fe5d00b38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:20.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.039+0000 7fe5d6339640 1 --2- 172.21.15.89:0/1571964906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5d00cb4c0 0x7fe5d01e4a20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc01d3a0 tx=0x7fe5cc01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:20.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.039+0000 7fe5c6ffd640 1 -- 172.21.15.89:0/1571964906 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc0ba600 con 0x7fe5d00cb4c0 2023-10-30T12:29:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.039+0000 7fe5c6ffd640 1 -- 172.21.15.89:0/1571964906 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5cc0ba760 con 0x7fe5d00cb4c0 2023-10-30T12:29:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.039+0000 7fe5d733b640 1 -- 172.21.15.89:0/1571964906 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5d01574c0 con 0x7fe5d00cb4c0 2023-10-30T12:29:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.039+0000 7fe5c6ffd640 1 -- 172.21.15.89:0/1571964906 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc0bab10 con 0x7fe5d00cb4c0 2023-10-30T12:29:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.039+0000 7fe5d733b640 1 -- 172.21.15.89:0/1571964906 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5d01e2b80 con 0x7fe5d00cb4c0 2023-10-30T12:29:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.039+0000 7fe5d733b640 1 -- 172.21.15.89:0/1571964906 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5d004e9c0 con 0x7fe5d00cb4c0 2023-10-30T12:29:20.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.039+0000 7fe5c6ffd640 1 -- 172.21.15.89:0/1571964906 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe5cc0bac70 con 0x7fe5d00cb4c0 2023-10-30T12:29:20.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.039+0000 7fe5c6ffd640 1 --2- 172.21.15.89:0/1571964906 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe5b40664d0 0x7fe5b4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:20.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.039+0000 7fe5c6ffd640 1 -- 172.21.15.89:0/1571964906 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe5cc1224a0 con 0x7fe5d00cb4c0 2023-10-30T12:29:20.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.043+0000 7fe5d5b38640 1 --2- 172.21.15.89:0/1571964906 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe5b40664d0 0x7fe5b4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:20.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.043+0000 7fe5d5b38640 1 --2- 172.21.15.89:0/1571964906 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe5b40664d0 0x7fe5b4068990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe5d006c790 tx=0x7fe5c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:20.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.043+0000 7fe5c6ffd640 1 -- 172.21.15.89:0/1571964906 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe5cc0ef880 con 0x7fe5d00cb4c0 2023-10-30T12:29:20.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.195+0000 7fe5d733b640 1 -- 172.21.15.89:0/1571964906 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe5d00cb000 con 0x7fe5d00cb4c0 2023-10-30T12:29:20.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.195+0000 7fe5c6ffd640 1 -- 172.21.15.89:0/1571964906 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe5cc0f2ea0 con 0x7fe5d00cb4c0 2023-10-30T12:29:20.197 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-10-30T12:29:20.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.195+0000 7fe5d733b640 1 -- 172.21.15.89:0/1571964906 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe5b40664d0 msgr2=0x7fe5b4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:20.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.195+0000 7fe5d733b640 1 --2- 172.21.15.89:0/1571964906 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe5b40664d0 0x7fe5b4068990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe5d006c790 tx=0x7fe5c0008040 comp rx=0 tx=0).stop 2023-10-30T12:29:20.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.195+0000 7fe5d733b640 1 -- 172.21.15.89:0/1571964906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5d00cb4c0 msgr2=0x7fe5d01e4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:20.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.195+0000 7fe5d733b640 1 --2- 172.21.15.89:0/1571964906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5d00cb4c0 0x7fe5d01e4a20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc01d3a0 tx=0x7fe5cc01ba90 comp rx=0 tx=0).stop 2023-10-30T12:29:20.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.195+0000 7fe5d733b640 1 -- 172.21.15.89:0/1571964906 shutdown_connections 2023-10-30T12:29:20.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.195+0000 7fe5d733b640 1 --2- 172.21.15.89:0/1571964906 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe5b40664d0 0x7fe5b4068990 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.195+0000 7fe5d733b640 1 --2- 172.21.15.89:0/1571964906 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5d0153c20 0x7fe5d00b38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.195+0000 7fe5d733b640 1 --2- 172.21.15.89:0/1571964906 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5d0153840 0x7fe5d01e4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.195+0000 7fe5d733b640 1 --2- 172.21.15.89:0/1571964906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5d00cb4c0 0x7fe5d01e4a20 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.195+0000 7fe5d733b640 1 -- 172.21.15.89:0/1571964906 >> 172.21.15.89:0/1571964906 conn(0x7fe5d006be40 msgr2=0x7fe5d0151070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:20.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.195+0000 7fe5d733b640 1 -- 172.21.15.89:0/1571964906 shutdown_connections 2023-10-30T12:29:20.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.195+0000 7fe5d733b640 1 -- 172.21.15.89:0/1571964906 wait complete. 2023-10-30T12:29:20.217 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T12:29:20.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.355+0000 7f249d23a640 1 Processor -- start 2023-10-30T12:29:20.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.355+0000 7f249d23a640 1 -- start start 2023-10-30T12:29:20.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.359+0000 7f249d23a640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2498153840 0x7f249815a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:20.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.359+0000 7f249d23a640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2498153c20 0x7f24980c3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:20.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.359+0000 7f249d23a640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f24980cb4c0 0x7f24980cef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:20.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.359+0000 7f249d23a640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f249815dbc0 con 0x7f2498153c20 2023-10-30T12:29:20.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.359+0000 7f249d23a640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f249815a6b0 con 0x7f24980cb4c0 2023-10-30T12:29:20.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.359+0000 7f249d23a640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f249815d730 con 0x7f2498153840 2023-10-30T12:29:20.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.359+0000 7f24977fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2498153c20 0x7f24980c3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:20.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.359+0000 7f24977fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2498153c20 0x7f24980c3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53402/0 (socket says 172.21.15.89:53402) 2023-10-30T12:29:20.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.359+0000 7f24977fe640 1 -- 172.21.15.89:0/1438382965 learned_addr learned my addr 172.21.15.89:0/1438382965 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:20.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.359+0000 7f2497fff640 1 --2- 172.21.15.89:0/1438382965 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2498153840 0x7f249815a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:20.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.359+0000 7f249ca39640 1 --2- 172.21.15.89:0/1438382965 >> v2:172.21.15.192:3300/0 conn(0x7f24980cb4c0 0x7f24980cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:20.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.359+0000 7f24977fe640 1 -- 172.21.15.89:0/1438382965 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2498153840 msgr2=0x7f249815a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:20.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.359+0000 7f24977fe640 1 --2- 172.21.15.89:0/1438382965 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2498153840 0x7f249815a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.359+0000 7f24977fe640 1 -- 172.21.15.89:0/1438382965 >> v2:172.21.15.192:3300/0 conn(0x7f24980cb4c0 msgr2=0x7f24980cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:20.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.359+0000 7f24977fe640 1 --2- 172.21.15.89:0/1438382965 >> v2:172.21.15.192:3300/0 conn(0x7f24980cb4c0 0x7f24980cef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.359+0000 7f24977fe640 1 -- 172.21.15.89:0/1438382965 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24980ca8f0 con 0x7f2498153c20 2023-10-30T12:29:20.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.359+0000 7f2497fff640 1 --2- 172.21.15.89:0/1438382965 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2498153840 0x7f249815a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:20.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.363+0000 7f24977fe640 1 --2- 172.21.15.89:0/1438382965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2498153c20 0x7f24980c3eb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2498072ca0 tx=0x7f248c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f587cf47786f6a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:20.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.363+0000 7f2496ffd640 1 -- 172.21.15.89:0/1438382965 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248c0a89b0 con 0x7f2498153c20 2023-10-30T12:29:20.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.363+0000 7f2496ffd640 1 -- 172.21.15.89:0/1438382965 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f248c0a8b10 con 0x7f2498153c20 2023-10-30T12:29:20.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.363+0000 7f2496ffd640 1 -- 172.21.15.89:0/1438382965 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248c0a8ea0 con 0x7f2498153c20 2023-10-30T12:29:20.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.363+0000 7f249d23a640 1 -- 172.21.15.89:0/1438382965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2498153c20 msgr2=0x7f24980c3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:20.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.363+0000 7f249d23a640 1 --2- 172.21.15.89:0/1438382965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2498153c20 0x7f24980c3eb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2498072ca0 tx=0x7f248c01ecf0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.363+0000 7f249d23a640 1 -- 172.21.15.89:0/1438382965 shutdown_connections 2023-10-30T12:29:20.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.363+0000 7f249d23a640 1 --2- 172.21.15.89:0/1438382965 >> v2:172.21.15.192:3300/0 conn(0x7f24980cb4c0 0x7f24980cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.363+0000 7f249d23a640 1 --2- 172.21.15.89:0/1438382965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2498153c20 0x7f24980c3eb0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.363+0000 7f249d23a640 1 --2- 172.21.15.89:0/1438382965 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2498153840 0x7f249815a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.363+0000 7f249d23a640 1 -- 172.21.15.89:0/1438382965 >> 172.21.15.89:0/1438382965 conn(0x7f249806be40 msgr2=0x7f249806c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:20.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.363+0000 7f249d23a640 1 -- 172.21.15.89:0/1438382965 shutdown_connections 2023-10-30T12:29:20.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.363+0000 7f249d23a640 1 -- 172.21.15.89:0/1438382965 wait complete. 2023-10-30T12:29:20.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.363+0000 7f249d23a640 1 Processor -- start 2023-10-30T12:29:20.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.363+0000 7f249d23a640 1 -- start start 2023-10-30T12:29:20.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.363+0000 7f249d23a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24980cb4c0 0x7f2498152030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:20.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.363+0000 7f249d23a640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2498153840 0x7f2498154840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:20.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.363+0000 7f249d23a640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2498153c20 0x7f2498154d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:20.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.363+0000 7f249d23a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24981f2fc0 con 0x7f24980cb4c0 2023-10-30T12:29:20.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.363+0000 7f249d23a640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24981552c0 con 0x7f2498153840 2023-10-30T12:29:20.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.363+0000 7f249d23a640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24981f2a20 con 0x7f2498153c20 2023-10-30T12:29:20.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.367+0000 7f2497fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24980cb4c0 0x7f2498152030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:20.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.367+0000 7f2497fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24980cb4c0 0x7f2498152030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53414/0 (socket says 172.21.15.89:53414) 2023-10-30T12:29:20.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.367+0000 7f2497fff640 1 -- 172.21.15.89:0/852580346 learned_addr learned my addr 172.21.15.89:0/852580346 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:20.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.367+0000 7f24977fe640 1 --2- 172.21.15.89:0/852580346 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2498153840 0x7f2498154840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:20.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.367+0000 7f249ca39640 1 --2- 172.21.15.89:0/852580346 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2498153c20 0x7f2498154d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:20.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.367+0000 7f2497fff640 1 -- 172.21.15.89:0/852580346 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2498153c20 msgr2=0x7f2498154d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:20.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.367+0000 7f2497fff640 1 --2- 172.21.15.89:0/852580346 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2498153c20 0x7f2498154d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.367+0000 7f2497fff640 1 -- 172.21.15.89:0/852580346 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2498153840 msgr2=0x7f2498154840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:20.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.367+0000 7f2497fff640 1 --2- 172.21.15.89:0/852580346 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2498153840 0x7f2498154840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.367+0000 7f2497fff640 1 -- 172.21.15.89:0/852580346 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f248c0094d0 con 0x7f24980cb4c0 2023-10-30T12:29:20.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.367+0000 7f2497fff640 1 --2- 172.21.15.89:0/852580346 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24980cb4c0 0x7f2498152030 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f248800e840 tx=0x7f248800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:20.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.367+0000 7f2494ff9640 1 -- 172.21.15.89:0/852580346 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248800c8d0 con 0x7f24980cb4c0 2023-10-30T12:29:20.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.367+0000 7f2494ff9640 1 -- 172.21.15.89:0/852580346 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f248800ca30 con 0x7f24980cb4c0 2023-10-30T12:29:20.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.367+0000 7f2494ff9640 1 -- 172.21.15.89:0/852580346 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248800cdc0 con 0x7f24980cb4c0 2023-10-30T12:29:20.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.367+0000 7f249d23a640 1 -- 172.21.15.89:0/852580346 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24981f2ce0 con 0x7f24980cb4c0 2023-10-30T12:29:20.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.367+0000 7f249d23a640 1 -- 172.21.15.89:0/852580346 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2498158d80 con 0x7f24980cb4c0 2023-10-30T12:29:20.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.367+0000 7f249d23a640 1 -- 172.21.15.89:0/852580346 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f249804e9c0 con 0x7f24980cb4c0 2023-10-30T12:29:20.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.367+0000 7f2494ff9640 1 -- 172.21.15.89:0/852580346 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2488013410 con 0x7f24980cb4c0 2023-10-30T12:29:20.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.367+0000 7f2494ff9640 1 --2- 172.21.15.89:0/852580346 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f24840664d0 0x7f2484068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:20.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.371+0000 7f2494ff9640 1 -- 172.21.15.89:0/852580346 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2488083c60 con 0x7f24980cb4c0 2023-10-30T12:29:20.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.371+0000 7f24977fe640 1 --2- 172.21.15.89:0/852580346 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f24840664d0 0x7f2484068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:20.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.371+0000 7f24977fe640 1 --2- 172.21.15.89:0/852580346 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f24840664d0 0x7f2484068990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f248c0b2000 tx=0x7f248c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:20.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.371+0000 7f2494ff9640 1 -- 172.21.15.89:0/852580346 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2488051060 con 0x7f24980cb4c0 2023-10-30T12:29:20.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.523+0000 7f249d23a640 1 -- 172.21.15.89:0/852580346 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f249815cdd0 con 0x7f24980cb4c0 2023-10-30T12:29:20.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.523+0000 7f2494ff9640 1 -- 172.21.15.89:0/852580346 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2488021020 con 0x7f24980cb4c0 2023-10-30T12:29:20.525 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-10-30T12:29:20.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.523+0000 7f249d23a640 1 -- 172.21.15.89:0/852580346 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f24840664d0 msgr2=0x7f2484068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:20.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.523+0000 7f249d23a640 1 --2- 172.21.15.89:0/852580346 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f24840664d0 0x7f2484068990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f248c0b2000 tx=0x7f248c009040 comp rx=0 tx=0).stop 2023-10-30T12:29:20.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.523+0000 7f249d23a640 1 -- 172.21.15.89:0/852580346 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24980cb4c0 msgr2=0x7f2498152030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:20.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.523+0000 7f249d23a640 1 --2- 172.21.15.89:0/852580346 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24980cb4c0 0x7f2498152030 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f248800e840 tx=0x7f248800ed00 comp rx=0 tx=0).stop 2023-10-30T12:29:20.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.523+0000 7f249d23a640 1 -- 172.21.15.89:0/852580346 shutdown_connections 2023-10-30T12:29:20.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.523+0000 7f249d23a640 1 --2- 172.21.15.89:0/852580346 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f24840664d0 0x7f2484068990 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.523+0000 7f249d23a640 1 --2- 172.21.15.89:0/852580346 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2498153840 0x7f2498154840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.523+0000 7f249d23a640 1 --2- 172.21.15.89:0/852580346 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2498153c20 0x7f2498154d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.523+0000 7f249d23a640 1 --2- 172.21.15.89:0/852580346 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24980cb4c0 0x7f2498152030 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.523+0000 7f249d23a640 1 -- 172.21.15.89:0/852580346 >> 172.21.15.89:0/852580346 conn(0x7f249806be40 msgr2=0x7f24980b4190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:20.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.523+0000 7f249d23a640 1 -- 172.21.15.89:0/852580346 shutdown_connections 2023-10-30T12:29:20.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.523+0000 7f249d23a640 1 -- 172.21.15.89:0/852580346 wait complete. 2023-10-30T12:29:20.545 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T12:29:20.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.683+0000 7f2bc4765640 1 Processor -- start 2023-10-30T12:29:20.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.683+0000 7f2bc4765640 1 -- start start 2023-10-30T12:29:20.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.683+0000 7f2bc4765640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2bbc153c90 0x7f2bbc0be770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:20.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.683+0000 7f2bc4765640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bbc06b540 0x7f2bbc06b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:20.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.683+0000 7f2bc4765640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2bbc1597a0 0x7f2bbc159c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:20.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.683+0000 7f2bc4765640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bbc070850 con 0x7f2bbc1597a0 2023-10-30T12:29:20.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.683+0000 7f2bc4765640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bbc06e8c0 con 0x7f2bbc153c90 2023-10-30T12:29:20.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.683+0000 7f2bc4765640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bbc0becb0 con 0x7f2bbc06b540 2023-10-30T12:29:20.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.683+0000 7f2bc2d02640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2bbc1597a0 0x7f2bbc159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:20.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.683+0000 7f2bc2d02640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2bbc1597a0 0x7f2bbc159c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53416/0 (socket says 172.21.15.89:53416) 2023-10-30T12:29:20.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.683+0000 7f2bc2d02640 1 -- 172.21.15.89:0/1059238827 learned_addr learned my addr 172.21.15.89:0/1059238827 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:20.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.683+0000 7f2bc2501640 1 --2- 172.21.15.89:0/1059238827 >> v2:172.21.15.192:3300/0 conn(0x7f2bbc153c90 0x7f2bbc0be770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:20.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.683+0000 7f2bc1d00640 1 --2- 172.21.15.89:0/1059238827 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bbc06b540 0x7f2bbc06b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:20.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.687+0000 7f2bc2d02640 1 -- 172.21.15.89:0/1059238827 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bbc06b540 msgr2=0x7f2bbc06b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:20.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.687+0000 7f2bc2d02640 1 --2- 172.21.15.89:0/1059238827 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bbc06b540 0x7f2bbc06b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.687+0000 7f2bc2d02640 1 -- 172.21.15.89:0/1059238827 >> v2:172.21.15.192:3300/0 conn(0x7f2bbc153c90 msgr2=0x7f2bbc0be770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:20.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.687+0000 7f2bc2d02640 1 --2- 172.21.15.89:0/1059238827 >> v2:172.21.15.192:3300/0 conn(0x7f2bbc153c90 0x7f2bbc0be770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.687+0000 7f2bc2d02640 1 -- 172.21.15.89:0/1059238827 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bbc0be140 con 0x7f2bbc1597a0 2023-10-30T12:29:20.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.687+0000 7f2bc2d02640 1 --2- 172.21.15.89:0/1059238827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bbc1597a0 0x7f2bbc159c10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc066080 tx=0x7f2bb401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83272c51f50317c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:20.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.687+0000 7f2bc14ff640 1 -- 172.21.15.89:0/1059238827 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb40af400 con 0x7f2bbc1597a0 2023-10-30T12:29:20.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.687+0000 7f2bc14ff640 1 -- 172.21.15.89:0/1059238827 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bb40ab050 con 0x7f2bbc1597a0 2023-10-30T12:29:20.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.687+0000 7f2bc14ff640 1 -- 172.21.15.89:0/1059238827 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb40af780 con 0x7f2bbc1597a0 2023-10-30T12:29:20.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.687+0000 7f2bc4765640 1 -- 172.21.15.89:0/1059238827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bbc1597a0 msgr2=0x7f2bbc159c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:20.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.687+0000 7f2bc4765640 1 --2- 172.21.15.89:0/1059238827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bbc1597a0 0x7f2bbc159c10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc066080 tx=0x7f2bb401f1c0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.687+0000 7f2bc4765640 1 -- 172.21.15.89:0/1059238827 shutdown_connections 2023-10-30T12:29:20.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.687+0000 7f2bc4765640 1 --2- 172.21.15.89:0/1059238827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bbc1597a0 0x7f2bbc159c10 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.687+0000 7f2bc4765640 1 --2- 172.21.15.89:0/1059238827 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bbc06b540 0x7f2bbc06b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.687+0000 7f2bc4765640 1 --2- 172.21.15.89:0/1059238827 >> v2:172.21.15.192:3300/0 conn(0x7f2bbc153c90 0x7f2bbc0be770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.687+0000 7f2bc4765640 1 -- 172.21.15.89:0/1059238827 >> 172.21.15.89:0/1059238827 conn(0x7f2bbc0c7fe0 msgr2=0x7f2bbc0c83f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:20.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.691+0000 7f2bc4765640 1 -- 172.21.15.89:0/1059238827 shutdown_connections 2023-10-30T12:29:20.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.691+0000 7f2bc4765640 1 -- 172.21.15.89:0/1059238827 wait complete. 2023-10-30T12:29:20.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.691+0000 7f2bc4765640 1 Processor -- start 2023-10-30T12:29:20.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.691+0000 7f2bc4765640 1 -- start start 2023-10-30T12:29:20.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.691+0000 7f2bc4765640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bbc06b540 0x7f2bbc0c2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:20.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.691+0000 7f2bc4765640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bbc153c90 0x7f2bbc0ca720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:20.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.691+0000 7f2bc4765640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bbc1597a0 0x7f2bbc1f2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:20.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.691+0000 7f2bc4765640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bbc0c4b60 con 0x7f2bbc1597a0 2023-10-30T12:29:20.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.691+0000 7f2bc4765640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bbc150cd0 con 0x7f2bbc06b540 2023-10-30T12:29:20.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.691+0000 7f2bc4765640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bbc0c6b10 con 0x7f2bbc153c90 2023-10-30T12:29:20.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.691+0000 7f2bc2d02640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bbc1597a0 0x7f2bbc1f2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:20.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.691+0000 7f2bc2d02640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bbc1597a0 0x7f2bbc1f2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53422/0 (socket says 172.21.15.89:53422) 2023-10-30T12:29:20.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.691+0000 7f2bc2d02640 1 -- 172.21.15.89:0/3057787404 learned_addr learned my addr 172.21.15.89:0/3057787404 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:20.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.691+0000 7f2bc2501640 1 --2- 172.21.15.89:0/3057787404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bbc06b540 0x7f2bbc0c2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:20.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.691+0000 7f2bc1d00640 1 --2- 172.21.15.89:0/3057787404 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bbc153c90 0x7f2bbc0ca720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:20.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.691+0000 7f2bc2d02640 1 -- 172.21.15.89:0/3057787404 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bbc153c90 msgr2=0x7f2bbc0ca720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:20.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.691+0000 7f2bc2d02640 1 --2- 172.21.15.89:0/3057787404 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bbc153c90 0x7f2bbc0ca720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.691+0000 7f2bc2d02640 1 -- 172.21.15.89:0/3057787404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bbc06b540 msgr2=0x7f2bbc0c2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:20.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.691+0000 7f2bc2d02640 1 --2- 172.21.15.89:0/3057787404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bbc06b540 0x7f2bbc0c2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.691+0000 7f2bc2d02640 1 -- 172.21.15.89:0/3057787404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb4009b30 con 0x7f2bbc1597a0 2023-10-30T12:29:20.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.691+0000 7f2bc2501640 1 --2- 172.21.15.89:0/3057787404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bbc06b540 0x7f2bbc0c2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:20.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.691+0000 7f2bc2d02640 1 --2- 172.21.15.89:0/3057787404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bbc1597a0 0x7f2bbc1f2200 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2bb401d870 tx=0x7f2bb401bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:20.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.691+0000 7f2bb2ffd640 1 -- 172.21.15.89:0/3057787404 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb40c0070 con 0x7f2bbc1597a0 2023-10-30T12:29:20.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.691+0000 7f2bb2ffd640 1 -- 172.21.15.89:0/3057787404 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bb4009510 con 0x7f2bbc1597a0 2023-10-30T12:29:20.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.691+0000 7f2bc4765640 1 -- 172.21.15.89:0/3057787404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bbc0c5840 con 0x7f2bbc1597a0 2023-10-30T12:29:20.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.691+0000 7f2bb2ffd640 1 -- 172.21.15.89:0/3057787404 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb40afdc0 con 0x7f2bbc1597a0 2023-10-30T12:29:20.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.695+0000 7f2bc4765640 1 -- 172.21.15.89:0/3057787404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bbc1503a0 con 0x7f2bbc1597a0 2023-10-30T12:29:20.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.695+0000 7f2bc4765640 1 -- 172.21.15.89:0/3057787404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bbc150e80 con 0x7f2bbc1597a0 2023-10-30T12:29:20.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.695+0000 7f2bb2ffd640 1 -- 172.21.15.89:0/3057787404 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2bb40b6040 con 0x7f2bbc1597a0 2023-10-30T12:29:20.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.695+0000 7f2bb2ffd640 1 --2- 172.21.15.89:0/3057787404 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2ba00664a0 0x7f2ba0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:20.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.695+0000 7f2bb2ffd640 1 -- 172.21.15.89:0/3057787404 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2bb4121770 con 0x7f2bbc1597a0 2023-10-30T12:29:20.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.695+0000 7f2bc2501640 1 --2- 172.21.15.89:0/3057787404 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2ba00664a0 0x7f2ba0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:20.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.695+0000 7f2bc2501640 1 --2- 172.21.15.89:0/3057787404 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2ba00664a0 0x7f2ba0068960 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2bb80079f0 tx=0x7f2bb8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:20.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.699+0000 7f2bb2ffd640 1 -- 172.21.15.89:0/3057787404 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2bbc150e80 con 0x7f2bbc1597a0 2023-10-30T12:29:20.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.847+0000 7f2bc4765640 1 -- 172.21.15.89:0/3057787404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2bbc150e80 con 0x7f2bbc1597a0 2023-10-30T12:29:20.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.847+0000 7f2bb2ffd640 1 -- 172.21.15.89:0/3057787404 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2bbc150e80 con 0x7f2bbc1597a0 2023-10-30T12:29:20.851 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-10-30T12:29:20.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.851+0000 7f2bc4765640 1 -- 172.21.15.89:0/3057787404 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2ba00664a0 msgr2=0x7f2ba0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:20.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.851+0000 7f2bc4765640 1 --2- 172.21.15.89:0/3057787404 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2ba00664a0 0x7f2ba0068960 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2bb80079f0 tx=0x7f2bb8008040 comp rx=0 tx=0).stop 2023-10-30T12:29:20.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.851+0000 7f2bc4765640 1 -- 172.21.15.89:0/3057787404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bbc1597a0 msgr2=0x7f2bbc1f2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:20.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.851+0000 7f2bc4765640 1 --2- 172.21.15.89:0/3057787404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bbc1597a0 0x7f2bbc1f2200 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2bb401d870 tx=0x7f2bb401bf60 comp rx=0 tx=0).stop 2023-10-30T12:29:20.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.851+0000 7f2bc4765640 1 -- 172.21.15.89:0/3057787404 shutdown_connections 2023-10-30T12:29:20.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.851+0000 7f2bc4765640 1 --2- 172.21.15.89:0/3057787404 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2ba00664a0 0x7f2ba0068960 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.851+0000 7f2bc4765640 1 --2- 172.21.15.89:0/3057787404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bbc1597a0 0x7f2bbc1f2200 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.851+0000 7f2bc4765640 1 --2- 172.21.15.89:0/3057787404 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2bbc153c90 0x7f2bbc0ca720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.855 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.851+0000 7f2bc4765640 1 --2- 172.21.15.89:0/3057787404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2bbc06b540 0x7f2bbc0c2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:20.855 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.851+0000 7f2bc4765640 1 -- 172.21.15.89:0/3057787404 >> 172.21.15.89:0/3057787404 conn(0x7f2bbc0c7fe0 msgr2=0x7f2bbc0be360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:20.855 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.851+0000 7f2bc4765640 1 -- 172.21.15.89:0/3057787404 shutdown_connections 2023-10-30T12:29:20.855 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:20.851+0000 7f2bc4765640 1 -- 172.21.15.89:0/3057787404 wait complete. 2023-10-30T12:29:20.872 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:29:21.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.011+0000 7f09b2f0f640 1 Processor -- start 2023-10-30T12:29:21.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.011+0000 7f09b2f0f640 1 -- start start 2023-10-30T12:29:21.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.011+0000 7f09b2f0f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09ac06f7a0 0x7f09ac06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:21.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.011+0000 7f09b2f0f640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f09ac071110 0x7f09ac158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:21.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.011+0000 7f09b2f0f640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09ac066cf0 con 0x7f09ac0c6bd0 2023-10-30T12:29:21.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.011+0000 7f09b2f0f640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09ac0700c0 con 0x7f09ac071110 2023-10-30T12:29:21.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.011+0000 7f09b2f0f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac0c70d0 con 0x7f09ac06f7a0 2023-10-30T12:29:21.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.011+0000 7f09b14ac640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f09ac071110 0x7f09ac158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:21.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.011+0000 7f09b14ac640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f09ac071110 0x7f09ac158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53426/0 (socket says 172.21.15.89:53426) 2023-10-30T12:29:21.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.011+0000 7f09b14ac640 1 -- 172.21.15.89:0/1746291138 learned_addr learned my addr 172.21.15.89:0/1746291138 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:21.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.011+0000 7f09b0cab640 1 --2- 172.21.15.89:0/1746291138 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09ac06f7a0 0x7f09ac06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:21.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.015+0000 7f09b14ac640 1 -- 172.21.15.89:0/1746291138 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09ac06f7a0 msgr2=0x7f09ac06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:21.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.015+0000 7f09b14ac640 1 --2- 172.21.15.89:0/1746291138 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09ac06f7a0 0x7f09ac06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:21.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.015+0000 7f09b14ac640 1 -- 172.21.15.89:0/1746291138 >> v1:172.21.15.192:6789/0 conn(0x7f09ac0c6bd0 legacy=0x7f09ac069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:21.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.015+0000 7f09b14ac640 1 -- 172.21.15.89:0/1746291138 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ac0c7390 con 0x7f09ac071110 2023-10-30T12:29:21.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.015+0000 7f09b0cab640 1 --2- 172.21.15.89:0/1746291138 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09ac06f7a0 0x7f09ac06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:21.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.015+0000 7f09b14ac640 1 --2- 172.21.15.89:0/1746291138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09ac071110 0x7f09ac158110 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f09a4009e50 tx=0x7f09a401f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7902d09e6b932db0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:21.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.015+0000 7f09a37fe640 1 -- 172.21.15.89:0/1746291138 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a40af400 con 0x7f09ac071110 2023-10-30T12:29:21.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.015+0000 7f09a37fe640 1 -- 172.21.15.89:0/1746291138 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a40ab050 con 0x7f09ac071110 2023-10-30T12:29:21.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.015+0000 7f09a37fe640 1 -- 172.21.15.89:0/1746291138 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a40af780 con 0x7f09ac071110 2023-10-30T12:29:21.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.015+0000 7f09b2f0f640 1 -- 172.21.15.89:0/1746291138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09ac071110 msgr2=0x7f09ac158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:21.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.015+0000 7f09b2f0f640 1 --2- 172.21.15.89:0/1746291138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09ac071110 0x7f09ac158110 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f09a4009e50 tx=0x7f09a401f1f0 comp rx=0 tx=0).stop 2023-10-30T12:29:21.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.015+0000 7f09b2f0f640 1 -- 172.21.15.89:0/1746291138 shutdown_connections 2023-10-30T12:29:21.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.015+0000 7f09b2f0f640 1 --2- 172.21.15.89:0/1746291138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09ac071110 0x7f09ac158110 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:21.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.015+0000 7f09b2f0f640 1 --2- 172.21.15.89:0/1746291138 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09ac06f7a0 0x7f09ac06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:21.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.015+0000 7f09b2f0f640 1 -- 172.21.15.89:0/1746291138 >> 172.21.15.89:0/1746291138 conn(0x7f09ac0c7910 msgr2=0x7f09ac0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:21.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.019+0000 7f09b2f0f640 1 -- 172.21.15.89:0/1746291138 shutdown_connections 2023-10-30T12:29:21.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.019+0000 7f09b2f0f640 1 -- 172.21.15.89:0/1746291138 wait complete. 2023-10-30T12:29:21.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.019+0000 7f09b2f0f640 1 Processor -- start 2023-10-30T12:29:21.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.019+0000 7f09b2f0f640 1 -- start start 2023-10-30T12:29:21.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.019+0000 7f09b2f0f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09ac06f7a0 0x7f09ac15a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:21.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.019+0000 7f09b2f0f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09ac071110 0x7f09ac14db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:21.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.019+0000 7f09b2f0f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09ac0c6bd0 0x7f09ac159240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:21.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.019+0000 7f09b2f0f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac155170 con 0x7f09ac0c6bd0 2023-10-30T12:29:21.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.019+0000 7f09b2f0f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac1547e0 con 0x7f09ac071110 2023-10-30T12:29:21.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.019+0000 7f09b2f0f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac15a8f0 con 0x7f09ac06f7a0 2023-10-30T12:29:21.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.019+0000 7f09b14ac640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09ac0c6bd0 0x7f09ac159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:21.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.019+0000 7f09b14ac640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09ac0c6bd0 0x7f09ac159240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53438/0 (socket says 172.21.15.89:53438) 2023-10-30T12:29:21.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.019+0000 7f09b14ac640 1 -- 172.21.15.89:0/899297469 learned_addr learned my addr 172.21.15.89:0/899297469 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:21.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.019+0000 7f09b0cab640 1 --2- 172.21.15.89:0/899297469 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09ac06f7a0 0x7f09ac15a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:21.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.019+0000 7f09a3fff640 1 --2- 172.21.15.89:0/899297469 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09ac071110 0x7f09ac14db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:21.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.019+0000 7f09b14ac640 1 -- 172.21.15.89:0/899297469 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09ac06f7a0 msgr2=0x7f09ac15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:21.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.019+0000 7f09b14ac640 1 --2- 172.21.15.89:0/899297469 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09ac06f7a0 0x7f09ac15a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:21.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.019+0000 7f09b14ac640 1 -- 172.21.15.89:0/899297469 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09ac071110 msgr2=0x7f09ac14db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:21.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.019+0000 7f09b14ac640 1 --2- 172.21.15.89:0/899297469 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09ac071110 0x7f09ac14db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:21.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.019+0000 7f09b14ac640 1 -- 172.21.15.89:0/899297469 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a4009b30 con 0x7f09ac0c6bd0 2023-10-30T12:29:21.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.019+0000 7f09b14ac640 1 --2- 172.21.15.89:0/899297469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09ac0c6bd0 0x7f09ac159240 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f09a401ce20 tx=0x7f09a401bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:21.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.023+0000 7f09a17fa640 1 -- 172.21.15.89:0/899297469 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a40afde0 con 0x7f09ac0c6bd0 2023-10-30T12:29:21.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.023+0000 7f09a17fa640 1 -- 172.21.15.89:0/899297469 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a4009510 con 0x7f09ac0c6bd0 2023-10-30T12:29:21.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.023+0000 7f09b2f0f640 1 -- 172.21.15.89:0/899297469 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09ac154bf0 con 0x7f09ac0c6bd0 2023-10-30T12:29:21.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.023+0000 7f09a17fa640 1 -- 172.21.15.89:0/899297469 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a40ba660 con 0x7f09ac0c6bd0 2023-10-30T12:29:21.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.023+0000 7f09b2f0f640 1 -- 172.21.15.89:0/899297469 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09ac158a40 con 0x7f09ac0c6bd0 2023-10-30T12:29:21.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.023+0000 7f09b2f0f640 1 -- 172.21.15.89:0/899297469 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09ac1588a0 con 0x7f09ac0c6bd0 2023-10-30T12:29:21.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.023+0000 7f09a17fa640 1 -- 172.21.15.89:0/899297469 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f09a40b6040 con 0x7f09ac0c6bd0 2023-10-30T12:29:21.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.023+0000 7f09a17fa640 1 --2- 172.21.15.89:0/899297469 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f09980664a0 0x7f0998068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:21.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.023+0000 7f09a17fa640 1 -- 172.21.15.89:0/899297469 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f09a4121fe0 con 0x7f09ac0c6bd0 2023-10-30T12:29:21.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.023+0000 7f09b0cab640 1 --2- 172.21.15.89:0/899297469 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f09980664a0 0x7f0998068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:21.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.027+0000 7f09b0cab640 1 --2- 172.21.15.89:0/899297469 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f09980664a0 0x7f0998068960 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f09ac0c8f20 tx=0x7f09a8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:21.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.027+0000 7f09a17fa640 1 -- 172.21.15.89:0/899297469 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f09ac1588a0 con 0x7f09ac0c6bd0 2023-10-30T12:29:21.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.207+0000 7f09b2f0f640 1 -- 172.21.15.89:0/899297469 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09ac1588a0 con 0x7f09ac0c6bd0 2023-10-30T12:29:21.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.211+0000 7f09a17fa640 1 -- 172.21.15.89:0/899297469 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6528 (secure 0 0 0) 0x7f09ac1588a0 con 0x7f09ac0c6bd0 2023-10-30T12:29:21.212 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:29:21.212 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:29:06.969397+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4253":{"gid":4253,"name":"b","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6835/1289220491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1289220491},{"type":"v1","addr":"172.21.15.192:6835","nonce":1289220491}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:29:21.213 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 12 2023-10-30T12:29:21.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.211+0000 7f09b2f0f640 1 -- 172.21.15.89:0/899297469 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f09980664a0 msgr2=0x7f0998068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:21.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.211+0000 7f09b2f0f640 1 --2- 172.21.15.89:0/899297469 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f09980664a0 0x7f0998068960 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f09ac0c8f20 tx=0x7f09a8008040 comp rx=0 tx=0).stop 2023-10-30T12:29:21.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.211+0000 7f09b2f0f640 1 -- 172.21.15.89:0/899297469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09ac0c6bd0 msgr2=0x7f09ac159240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:21.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.211+0000 7f09b2f0f640 1 --2- 172.21.15.89:0/899297469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09ac0c6bd0 0x7f09ac159240 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f09a401ce20 tx=0x7f09a401bf90 comp rx=0 tx=0).stop 2023-10-30T12:29:21.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.211+0000 7f09b2f0f640 1 -- 172.21.15.89:0/899297469 shutdown_connections 2023-10-30T12:29:21.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.211+0000 7f09b2f0f640 1 --2- 172.21.15.89:0/899297469 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f09980664a0 0x7f0998068960 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:21.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.211+0000 7f09b2f0f640 1 --2- 172.21.15.89:0/899297469 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09ac071110 0x7f09ac14db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:21.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.211+0000 7f09b2f0f640 1 --2- 172.21.15.89:0/899297469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09ac0c6bd0 0x7f09ac159240 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:21.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.211+0000 7f09b2f0f640 1 --2- 172.21.15.89:0/899297469 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f09ac06f7a0 0x7f09ac15a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:21.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.211+0000 7f09b2f0f640 1 -- 172.21.15.89:0/899297469 >> 172.21.15.89:0/899297469 conn(0x7f09ac0c7910 msgr2=0x7f09ac0ca670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:21.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.211+0000 7f09b2f0f640 1 -- 172.21.15.89:0/899297469 shutdown_connections 2023-10-30T12:29:21.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.211+0000 7f09b2f0f640 1 -- 172.21.15.89:0/899297469 wait complete. 2023-10-30T12:29:21.233 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:29:21.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.367+0000 7f99855a0640 1 Processor -- start 2023-10-30T12:29:21.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.367+0000 7f99855a0640 1 -- start start 2023-10-30T12:29:21.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.371+0000 7f99855a0640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99801538f0 0x7f9980159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:21.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.371+0000 7f99855a0640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f998015cd50 0x7f99800c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:21.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.371+0000 7f99855a0640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f998015d300 con 0x7f99800cefa0 2023-10-30T12:29:21.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.371+0000 7f99855a0640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f998015d5e0 con 0x7f998015cd50 2023-10-30T12:29:21.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.371+0000 7f99855a0640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f998015d8c0 con 0x7f99801538f0 2023-10-30T12:29:21.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.371+0000 7f997f7fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f998015cd50 0x7f99800c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:21.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.371+0000 7f997f7fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f998015cd50 0x7f99800c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53448/0 (socket says 172.21.15.89:53448) 2023-10-30T12:29:21.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.371+0000 7f997f7fe640 1 -- 172.21.15.89:0/980690442 learned_addr learned my addr 172.21.15.89:0/980690442 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:21.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.371+0000 7f997ffff640 1 --2- 172.21.15.89:0/980690442 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99801538f0 0x7f9980159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:21.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.371+0000 7f997f7fe640 1 -- 172.21.15.89:0/980690442 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99801538f0 msgr2=0x7f9980159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:21.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.371+0000 7f997f7fe640 1 --2- 172.21.15.89:0/980690442 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99801538f0 0x7f9980159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:21.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.371+0000 7f997f7fe640 1 -- 172.21.15.89:0/980690442 >> v1:172.21.15.192:6789/0 conn(0x7f99800cefa0 legacy=0x7f99800cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:21.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.371+0000 7f997f7fe640 1 -- 172.21.15.89:0/980690442 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f998015dba0 con 0x7f998015cd50 2023-10-30T12:29:21.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.375+0000 7f997f7fe640 1 --2- 172.21.15.89:0/980690442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f998015cd50 0x7f99800c9f90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9974009870 tx=0x7f997401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e1fb3762e014d5f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:21.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.375+0000 7f997effd640 1 -- 172.21.15.89:0/980690442 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99740a8a30 con 0x7f998015cd50 2023-10-30T12:29:21.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.375+0000 7f997effd640 1 -- 172.21.15.89:0/980690442 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99740a8b90 con 0x7f998015cd50 2023-10-30T12:29:21.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.375+0000 7f997effd640 1 -- 172.21.15.89:0/980690442 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99740b0440 con 0x7f998015cd50 2023-10-30T12:29:21.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.375+0000 7f99855a0640 1 -- 172.21.15.89:0/980690442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f998015cd50 msgr2=0x7f99800c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:21.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.375+0000 7f99855a0640 1 --2- 172.21.15.89:0/980690442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f998015cd50 0x7f99800c9f90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9974009870 tx=0x7f997401ed30 comp rx=0 tx=0).stop 2023-10-30T12:29:21.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.375+0000 7f99855a0640 1 -- 172.21.15.89:0/980690442 shutdown_connections 2023-10-30T12:29:21.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.375+0000 7f99855a0640 1 --2- 172.21.15.89:0/980690442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f998015cd50 0x7f99800c9f90 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:21.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.375+0000 7f99855a0640 1 --2- 172.21.15.89:0/980690442 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99801538f0 0x7f9980159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:21.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.375+0000 7f99855a0640 1 -- 172.21.15.89:0/980690442 >> 172.21.15.89:0/980690442 conn(0x7f998006a640 msgr2=0x7f9980065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:21.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.375+0000 7f99855a0640 1 -- 172.21.15.89:0/980690442 shutdown_connections 2023-10-30T12:29:21.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.375+0000 7f99855a0640 1 -- 172.21.15.89:0/980690442 wait complete. 2023-10-30T12:29:21.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.375+0000 7f99855a0640 1 Processor -- start 2023-10-30T12:29:21.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.375+0000 7f99855a0640 1 -- start start 2023-10-30T12:29:21.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.379+0000 7f99855a0640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99800cefa0 0x7f9980144ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:21.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.379+0000 7f99855a0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99801538f0 0x7f99801eef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:21.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.379+0000 7f99855a0640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f998015cd50 0x7f998013f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:21.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.379+0000 7f99855a0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99801f2e20 con 0x7f99801538f0 2023-10-30T12:29:21.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.379+0000 7f99855a0640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99800b3290 con 0x7f99800cefa0 2023-10-30T12:29:21.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.379+0000 7f99855a0640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99800b3570 con 0x7f998015cd50 2023-10-30T12:29:21.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.379+0000 7f997f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99801538f0 0x7f99801eef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:21.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.379+0000 7f997f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99801538f0 0x7f99801eef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53458/0 (socket says 172.21.15.89:53458) 2023-10-30T12:29:21.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.379+0000 7f997f7fe640 1 -- 172.21.15.89:0/1862657775 learned_addr learned my addr 172.21.15.89:0/1862657775 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:29:21.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.379+0000 7f997ffff640 1 --2- 172.21.15.89:0/1862657775 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99800cefa0 0x7f9980144ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:21.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.379+0000 7f9984d9f640 1 --2- 172.21.15.89:0/1862657775 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f998015cd50 0x7f998013f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:21.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.379+0000 7f997f7fe640 1 -- 172.21.15.89:0/1862657775 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f998015cd50 msgr2=0x7f998013f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:21.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.379+0000 7f997f7fe640 1 --2- 172.21.15.89:0/1862657775 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f998015cd50 0x7f998013f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:21.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.379+0000 7f997f7fe640 1 -- 172.21.15.89:0/1862657775 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99800cefa0 msgr2=0x7f9980144ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:21.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.379+0000 7f997f7fe640 1 --2- 172.21.15.89:0/1862657775 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99800cefa0 0x7f9980144ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:21.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.379+0000 7f997f7fe640 1 -- 172.21.15.89:0/1862657775 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99740094d0 con 0x7f99801538f0 2023-10-30T12:29:21.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.379+0000 7f997ffff640 1 --2- 172.21.15.89:0/1862657775 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99800cefa0 0x7f9980144ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:21.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.379+0000 7f997f7fe640 1 --2- 172.21.15.89:0/1862657775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99801538f0 0x7f99801eef80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f99740b3040 tx=0x7f997401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:21.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.379+0000 7f997cff9640 1 -- 172.21.15.89:0/1862657775 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99740b9620 con 0x7f99801538f0 2023-10-30T12:29:21.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.379+0000 7f997cff9640 1 -- 172.21.15.89:0/1862657775 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99740b97a0 con 0x7f99801538f0 2023-10-30T12:29:21.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.379+0000 7f997cff9640 1 -- 172.21.15.89:0/1862657775 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99740b9b50 con 0x7f99801538f0 2023-10-30T12:29:21.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.379+0000 7f99855a0640 1 -- 172.21.15.89:0/1862657775 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99801f24f0 con 0x7f99801538f0 2023-10-30T12:29:21.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.379+0000 7f99855a0640 1 -- 172.21.15.89:0/1862657775 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99801f2ab0 con 0x7f99801538f0 2023-10-30T12:29:21.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.379+0000 7f99855a0640 1 -- 172.21.15.89:0/1862657775 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99801f2c40 con 0x7f99801538f0 2023-10-30T12:29:21.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.379+0000 7f997cff9640 1 -- 172.21.15.89:0/1862657775 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f99740b7070 con 0x7f99801538f0 2023-10-30T12:29:21.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.383+0000 7f997cff9640 1 --2- 172.21.15.89:0/1862657775 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f996c0664d0 0x7f996c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:21.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.383+0000 7f997cff9640 1 -- 172.21.15.89:0/1862657775 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9974121cc0 con 0x7f99801538f0 2023-10-30T12:29:21.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.383+0000 7f997ffff640 1 --2- 172.21.15.89:0/1862657775 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f996c0664d0 0x7f996c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:21.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.383+0000 7f997ffff640 1 --2- 172.21.15.89:0/1862657775 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f996c0664d0 0x7f996c068990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9970009bb0 tx=0x7f9970008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:21.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.387+0000 7f997cff9640 1 -- 172.21.15.89:0/1862657775 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9974128080 con 0x7f99801538f0 2023-10-30T12:29:21.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.535+0000 7f99855a0640 1 -- 172.21.15.89:0/1862657775 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f998004e170 con 0x7f99801538f0 2023-10-30T12:29:21.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.539+0000 7f997cff9640 1 -- 172.21.15.89:0/1862657775 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f99740ef130 con 0x7f99801538f0 2023-10-30T12:29:21.540 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:29:21.540 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":22,"fsid":"401641be-db6e-4c16-b3ac-fd84a86c4de4","created":"2023-10-30T12:28:26.198532+0000","modified":"2023-10-30T12:28:51.918285+0000","last_up_change":"2023-10-30T12:28:37.212205+0000","last_in_change":"2023-10-30T12:28:28.517734+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:39.004943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T12:28:48.935909+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T12:28:49.716098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72595c54-fd6e-4a5a-bb14-ff27e9377b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6801","nonce":1673955622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6803","nonce":1673955622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6807","nonce":1673955622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6805","nonce":1673955622}]},"public_addr":"172.21.15.89:6801/1673955622","cluster_addr":"172.21.15.89:6803/1673955622","heartbeat_back_addr":"172.21.15.89:6807/1673955622","heartbeat_front_addr":"172.21.15.89:6805/1673955622","state":["exists","up"]},{"osd":1,"uuid":"0305670b-d7c4-4193-bec2-8b7351ff590a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6817","nonce":3340572039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6819","nonce":3340572039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6823","nonce":3340572039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6821","nonce":3340572039}]},"public_addr":"172.21.15.89:6817/3340572039","cluster_addr":"172.21.15.89:6819/3340572039","heartbeat_back_addr":"172.21.15.89:6823/3340572039","heartbeat_front_addr":"172.21.15.89:6821/3340572039","state":["exists","up"]},{"osd":2,"uuid":"4bab5426-a3c3-46b0-91fc-546c254006c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6809","nonce":1929940799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6811","nonce":1929940799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6815","nonce":1929940799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6813","nonce":1929940799}]},"public_addr":"172.21.15.89:6809/1929940799","cluster_addr":"172.21.15.89:6811/1929940799","heartbeat_back_addr":"172.21.15.89:6815/1929940799","heartbeat_front_addr":"172.21.15.89:6813/1929940799","state":["exists","up"]},{"osd":3,"uuid":"7d47aea8-27cd-42be-8591-d63ad68c90e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6825","nonce":345087939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6827","nonce":345087939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6831","nonce":345087939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6829","nonce":345087939}]},"public_addr":"172.21.15.89:6825/345087939","cluster_addr":"172.21.15.89:6827/345087939","heartbeat_back_addr":"172.21.15.89:6831/345087939","heartbeat_front_addr":"172.21.15.89:6829/345087939","state":["exists","up"]},{"osd":4,"uuid":"2e2ae8f8-d097-4bba-9448-95047fbfacad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6801","nonce":2138125545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6803","nonce":2138125545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6809","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6811","nonce":2138125545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6805","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6807","nonce":2138125545}]},"public_addr":"172.21.15.192:6801/2138125545","cluster_addr":"172.21.15.192:6803/2138125545","heartbeat_back_addr":"172.21.15.192:6811/2138125545","heartbeat_front_addr":"172.21.15.192:6807/2138125545","state":["exists","up"]},{"osd":5,"uuid":"49be5ce5-12f4-4c26-8d39-864b6ff67313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6806","nonce":2360218671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6810","nonce":2360218671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6815","nonce":2360218671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6813","nonce":2360218671}]},"public_addr":"172.21.15.192:6806/2360218671","cluster_addr":"172.21.15.192:6810/2360218671","heartbeat_back_addr":"172.21.15.192:6815/2360218671","heartbeat_front_addr":"172.21.15.192:6813/2360218671","state":["exists","up"]},{"osd":6,"uuid":"48623452-d456-4ea2-a0de-5fa4b37e6d10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6825","nonce":3707671287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6827","nonce":3707671287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6831","nonce":3707671287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6829","nonce":3707671287}]},"public_addr":"172.21.15.192:6825/3707671287","cluster_addr":"172.21.15.192:6827/3707671287","heartbeat_back_addr":"172.21.15.192:6831/3707671287","heartbeat_front_addr":"172.21.15.192:6829/3707671287","state":["exists","up"]},{"osd":7,"uuid":"ff0a4384-54f8-43ec-a3ce-9c8ae3bf8a94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6817","nonce":3111945871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6819","nonce":3111945871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6823","nonce":3111945871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6821","nonce":3111945871}]},"public_addr":"172.21.15.192:6817/3111945871","cluster_addr":"172.21.15.192:6819/3111945871","heartbeat_back_addr":"172.21.15.192:6823/3111945871","heartbeat_front_addr":"172.21.15.192:6821/3111945871","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.183065+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.248650+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.194576+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.397483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.043648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.186624+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.159674+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.100851+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:29:21.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.539+0000 7f99855a0640 1 -- 172.21.15.89:0/1862657775 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f996c0664d0 msgr2=0x7f996c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:21.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.539+0000 7f99855a0640 1 --2- 172.21.15.89:0/1862657775 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f996c0664d0 0x7f996c068990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9970009bb0 tx=0x7f9970008040 comp rx=0 tx=0).stop 2023-10-30T12:29:21.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.539+0000 7f99855a0640 1 -- 172.21.15.89:0/1862657775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99801538f0 msgr2=0x7f99801eef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:21.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.539+0000 7f99855a0640 1 --2- 172.21.15.89:0/1862657775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99801538f0 0x7f99801eef80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f99740b3040 tx=0x7f997401bad0 comp rx=0 tx=0).stop 2023-10-30T12:29:21.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.539+0000 7f99855a0640 1 -- 172.21.15.89:0/1862657775 shutdown_connections 2023-10-30T12:29:21.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.539+0000 7f99855a0640 1 --2- 172.21.15.89:0/1862657775 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f996c0664d0 0x7f996c068990 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:21.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.539+0000 7f99855a0640 1 --2- 172.21.15.89:0/1862657775 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f998015cd50 0x7f998013f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:21.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.539+0000 7f99855a0640 1 --2- 172.21.15.89:0/1862657775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99801538f0 0x7f99801eef80 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:21.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.539+0000 7f99855a0640 1 --2- 172.21.15.89:0/1862657775 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99800cefa0 0x7f9980144ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:21.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.539+0000 7f99855a0640 1 -- 172.21.15.89:0/1862657775 >> 172.21.15.89:0/1862657775 conn(0x7f998006a640 msgr2=0x7f99800cf3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:21.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.539+0000 7f99855a0640 1 -- 172.21.15.89:0/1862657775 shutdown_connections 2023-10-30T12:29:21.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:29:21.539+0000 7f99855a0640 1 -- 172.21.15.89:0/1862657775 wait complete. 2023-10-30T12:29:21.561 DEBUG:tasks.mds_thrash:Yielding 2023-10-30T12:29:21.562 INFO:teuthology.run_tasks:Running task workunit... 2023-10-30T12:29:21.577 INFO:tasks.workunit:Pulling workunits from ref 00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T12:29:21.577 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-30T12:29:21.578 INFO:tasks.workunit:timeout=6h 2023-10-30T12:29:21.578 INFO:tasks.workunit:cleanup=True 2023-10-30T12:29:21.578 DEBUG:teuthology.orchestra.run.smithi089:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-30T12:29:21.579 INFO:tasks.mds_thrash.fs.[cephfs]:starting mds_do_thrash for fs cephfs 2023-10-30T12:29:21.579 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 102 secs before thrashing 2023-10-30T12:29:21.590 INFO:teuthology.orchestra.run.smithi089.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-10-30T12:29:21.590 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-10-30T12:29:21.590 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 2bh/43d Inode: 1 Links: 2 2023-10-30T12:29:21.591 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:29:21.591 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2023-10-30T12:29:21.591 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2023-10-30 12:28:56.349807609 +0000 2023-10-30T12:29:21.591 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2023-10-30 12:29:15.015794012 +0000 2023-10-30T12:29:21.591 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2023-10-30 12:28:56.349807609 +0000 2023-10-30T12:29:21.592 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-10-30T12:29:21.592 DEBUG:teuthology.orchestra.run.smithi089:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-10-30T12:29:21.665 DEBUG:teuthology.orchestra.run.smithi089:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T12:29:21.721 INFO:tasks.workunit.client.0.smithi089.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-30T12:29:44.431 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:29:44.436 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:30:14.737 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:30:14.743 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:30:24.509 INFO:tasks.workunit.client.0.smithi089.stderr:Updating files: 69% (8393/12071) Updating files: 70% (8450/12071) Updating files: 71% (8571/12071) Updating files: 72% (8692/12071) Updating files: 73% (8812/12071) Updating files: 74% (8933/12071) Updating files: 75% (9054/12071) Updating files: 76% (9174/12071) Updating files: 77% (9295/12071) Updating files: 78% (9416/12071) Updating files: 79% (9537/12071) Updating files: 80% (9657/12071) Updating files: 81% (9778/12071) Updating files: 82% (9899/12071) Updating files: 83% (10019/12071) Updating files: 84% (10140/12071) Updating files: 85% (10261/12071) Updating files: 86% (10382/12071) Updating files: 87% (10502/12071) Updating files: 88% (10623/12071) Updating files: 89% (10744/12071) Updating files: 90% (10864/12071) Updating files: 91% (10985/12071) Updating files: 92% (11106/12071) Updating files: 93% (11227/12071) Updating files: 94% (11347/12071) Updating files: 95% (11468/12071) Updating files: 96% (11589/12071) Updating files: 97% (11709/12071) Updating files: 98% (11830/12071) Updating files: 99% (11951/12071) Updating files: 100% (12071/12071) Updating files: 100% (12071/12071), done. 2023-10-30T12:30:24.920 INFO:tasks.workunit.client.0.smithi089.stderr:Note: switching to '00c1c13cea21a686b046584596fbeac5315a7d71'. 2023-10-30T12:30:24.921 INFO:tasks.workunit.client.0.smithi089.stderr: 2023-10-30T12:30:24.921 INFO:tasks.workunit.client.0.smithi089.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-30T12:30:24.921 INFO:tasks.workunit.client.0.smithi089.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-30T12:30:24.921 INFO:tasks.workunit.client.0.smithi089.stderr:state without impacting any branches by switching back to a branch. 2023-10-30T12:30:24.922 INFO:tasks.workunit.client.0.smithi089.stderr: 2023-10-30T12:30:24.922 INFO:tasks.workunit.client.0.smithi089.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-30T12:30:24.922 INFO:tasks.workunit.client.0.smithi089.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-30T12:30:24.922 INFO:tasks.workunit.client.0.smithi089.stderr: 2023-10-30T12:30:24.922 INFO:tasks.workunit.client.0.smithi089.stderr: git switch -c 2023-10-30T12:30:24.923 INFO:tasks.workunit.client.0.smithi089.stderr: 2023-10-30T12:30:24.923 INFO:tasks.workunit.client.0.smithi089.stderr:Or undo this operation with: 2023-10-30T12:30:24.923 INFO:tasks.workunit.client.0.smithi089.stderr: 2023-10-30T12:30:24.923 INFO:tasks.workunit.client.0.smithi089.stderr: git switch - 2023-10-30T12:30:24.923 INFO:tasks.workunit.client.0.smithi089.stderr: 2023-10-30T12:30:24.923 INFO:tasks.workunit.client.0.smithi089.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-30T12:30:24.924 INFO:tasks.workunit.client.0.smithi089.stderr: 2023-10-30T12:30:24.924 INFO:tasks.workunit.client.0.smithi089.stderr:HEAD is now at 00c1c13cea2 mds/MDBalancer: ignore queued callbacks if MDS is not active 2023-10-30T12:30:24.926 DEBUG:teuthology.orchestra.run.smithi089:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-30T12:30:24.935 INFO:tasks.workunit.client.0.smithi089.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-30T12:30:24.938 INFO:tasks.workunit.client.0.smithi089.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-30T12:30:24.938 INFO:tasks.workunit.client.0.smithi089.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-30T12:30:25.733 INFO:tasks.workunit.client.0.smithi089.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-30T12:30:25.839 INFO:tasks.workunit.client.0.smithi089.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-30T12:30:25.894 INFO:tasks.workunit.client.0.smithi089.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-30T12:30:25.896 INFO:tasks.workunit.client.0.smithi089.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-30T12:30:25.896 INFO:tasks.workunit.client.0.smithi089.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-30T12:30:25.950 INFO:tasks.workunit.client.0.smithi089.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-30T12:30:25.956 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-10-30T12:30:25.956 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-30T12:30:26.013 INFO:tasks.workunit:Running workunits matching fs/snaps on client.0... 2023-10-30T12:30:26.015 INFO:tasks.workunit:Running workunit fs/snaps/snap-rm-diff.sh... 2023-10-30T12:30:26.016 DEBUG:teuthology.orchestra.run.smithi089:workunit test fs/snaps/snap-rm-diff.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snap-rm-diff.sh 2023-10-30T12:30:26.078 INFO:tasks.workunit.client.0.smithi089.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2023-10-30T12:30:29.311 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir foo 2023-10-30T12:30:29.322 INFO:tasks.workunit.client.0.smithi089.stderr:+ cp linux-2.6.33.tar.bz2 foo 2023-10-30T12:30:29.417 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir foo/.snap/barsnap 2023-10-30T12:30:29.438 INFO:tasks.workunit.client.0.smithi089.stderr:+ rm foo/linux-2.6.33.tar.bz2 2023-10-30T12:30:29.452 INFO:tasks.workunit.client.0.smithi089.stderr:+ diff -q foo/.snap/barsnap/linux-2.6.33.tar.bz2 linux-2.6.33.tar.bz2 2023-10-30T12:30:29.891 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo passed: files are identical 2023-10-30T12:30:29.891 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir foo/.snap/barsnap 2023-10-30T12:30:29.891 INFO:tasks.workunit.client.0.smithi089.stdout:passed: files are identical 2023-10-30T12:30:29.930 INFO:tasks.workunit.client.0.smithi089.stdout:OK 2023-10-30T12:30:29.931 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo OK 2023-10-30T12:30:29.931 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T12:30:29.931 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T12:30:30.713 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-1.sh... 2023-10-30T12:30:30.714 DEBUG:teuthology.orchestra.run.smithi089:workunit test fs/snaps/snaptest-1.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-1.sh 2023-10-30T12:30:31.426 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 1 2023-10-30T12:30:31.428 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 2 2023-10-30T12:30:31.428 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 3 2023-10-30T12:30:31.429 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' -e file4 ']' 2023-10-30T12:30:31.429 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir .snap/snap1 2023-10-30T12:30:31.454 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 4 2023-10-30T12:30:31.457 INFO:tasks.workunit.client.0.smithi089.stderr:++ ls 2023-10-30T12:30:31.465 INFO:tasks.workunit.client.0.smithi089.stderr:+ now='file1 2023-10-30T12:30:31.465 INFO:tasks.workunit.client.0.smithi089.stderr:file2 2023-10-30T12:30:31.465 INFO:tasks.workunit.client.0.smithi089.stderr:file3 2023-10-30T12:30:31.466 INFO:tasks.workunit.client.0.smithi089.stderr:file4' 2023-10-30T12:30:31.466 INFO:tasks.workunit.client.0.smithi089.stderr:++ ls .snap/snap1 2023-10-30T12:30:31.473 INFO:tasks.workunit.client.0.smithi089.stderr:+ then='file1 2023-10-30T12:30:31.474 INFO:tasks.workunit.client.0.smithi089.stderr:file2 2023-10-30T12:30:31.474 INFO:tasks.workunit.client.0.smithi089.stderr:file3' 2023-10-30T12:30:31.474 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir .snap/snap1 2023-10-30T12:30:31.486 INFO:tasks.workunit.client.0.smithi089.stderr:+ '[' 'file1 2023-10-30T12:30:31.486 INFO:tasks.workunit.client.0.smithi089.stderr:file2 2023-10-30T12:30:31.487 INFO:tasks.workunit.client.0.smithi089.stderr:file3 2023-10-30T12:30:31.487 INFO:tasks.workunit.client.0.smithi089.stderr:file4' = 'file1 2023-10-30T12:30:31.487 INFO:tasks.workunit.client.0.smithi089.stderr:file2 2023-10-30T12:30:31.487 INFO:tasks.workunit.client.0.smithi089.stderr:file3' ']' 2023-10-30T12:30:31.487 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 1 2023-10-30T12:30:31.499 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 2 2023-10-30T12:30:31.512 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 3 2023-10-30T12:30:31.524 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir .snap/snap1 2023-10-30T12:30:31.538 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo 4 2023-10-30T12:30:31.548 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir .snap/snap1 2023-10-30T12:30:31.562 INFO:tasks.workunit.client.0.smithi089.stderr:+ rm file1 file2 file3 file4 2023-10-30T12:30:31.578 INFO:tasks.workunit.client.0.smithi089.stdout:OK 2023-10-30T12:30:31.579 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo OK 2023-10-30T12:30:31.579 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T12:30:31.580 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T12:30:31.613 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-2.sh... 2023-10-30T12:30:31.613 DEBUG:teuthology.orchestra.run.smithi089:workunit test fs/snaps/snaptest-2.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-2.sh 2023-10-30T12:30:36.427 INFO:tasks.workunit.client.0.smithi089.stdout:Create dir 100 to 199 ... 2023-10-30T12:30:36.428 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 100 2023-10-30T12:30:36.438 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 101 2023-10-30T12:30:36.459 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 102 2023-10-30T12:30:36.481 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 103 2023-10-30T12:30:36.989 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 104 2023-10-30T12:30:37.010 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 105 2023-10-30T12:30:37.031 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 106 2023-10-30T12:30:37.052 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 107 2023-10-30T12:30:37.073 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 108 2023-10-30T12:30:37.093 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 109 2023-10-30T12:30:37.228 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 110 2023-10-30T12:30:37.248 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 111 2023-10-30T12:30:37.269 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 112 2023-10-30T12:30:37.290 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 113 2023-10-30T12:30:37.310 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 114 2023-10-30T12:30:37.329 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 115 2023-10-30T12:30:37.346 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 116 2023-10-30T12:30:37.380 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 117 2023-10-30T12:30:37.397 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 118 2023-10-30T12:30:37.414 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 119 2023-10-30T12:30:37.643 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 120 2023-10-30T12:30:37.823 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 121 2023-10-30T12:30:37.844 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 122 2023-10-30T12:30:37.865 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 123 2023-10-30T12:30:37.886 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 124 2023-10-30T12:30:37.907 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 125 2023-10-30T12:30:37.924 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 126 2023-10-30T12:30:37.940 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 127 2023-10-30T12:30:37.957 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 128 2023-10-30T12:30:37.973 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 129 2023-10-30T12:30:37.992 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 130 2023-10-30T12:30:38.006 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 131 2023-10-30T12:30:38.019 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 132 2023-10-30T12:30:38.031 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 133 2023-10-30T12:30:38.052 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 134 2023-10-30T12:30:38.064 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 135 2023-10-30T12:30:38.077 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 136 2023-10-30T12:30:38.090 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 137 2023-10-30T12:30:38.102 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 138 2023-10-30T12:30:38.115 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 139 2023-10-30T12:30:38.126 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 140 2023-10-30T12:30:38.138 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 141 2023-10-30T12:30:38.149 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 142 2023-10-30T12:30:38.161 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 143 2023-10-30T12:30:38.172 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 144 2023-10-30T12:30:38.183 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 145 2023-10-30T12:30:38.194 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 146 2023-10-30T12:30:38.666 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 147 2023-10-30T12:30:38.808 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 148 2023-10-30T12:30:38.831 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 149 2023-10-30T12:30:38.853 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 150 2023-10-30T12:30:38.875 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 151 2023-10-30T12:30:39.658 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 152 2023-10-30T12:30:39.833 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 153 2023-10-30T12:30:39.856 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 154 2023-10-30T12:30:39.878 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 155 2023-10-30T12:30:39.899 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 156 2023-10-30T12:30:39.919 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 157 2023-10-30T12:30:39.938 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 158 2023-10-30T12:30:39.954 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 159 2023-10-30T12:30:39.971 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 160 2023-10-30T12:30:39.986 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 161 2023-10-30T12:30:40.001 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 162 2023-10-30T12:30:40.015 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 163 2023-10-30T12:30:40.030 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 164 2023-10-30T12:30:40.044 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 165 2023-10-30T12:30:40.057 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 166 2023-10-30T12:30:40.070 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 167 2023-10-30T12:30:40.082 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 168 2023-10-30T12:30:40.094 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 169 2023-10-30T12:30:40.106 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 170 2023-10-30T12:30:40.371 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 171 2023-10-30T12:30:40.389 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 172 2023-10-30T12:30:40.408 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 173 2023-10-30T12:30:40.428 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 174 2023-10-30T12:30:40.691 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 175 2023-10-30T12:30:40.965 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 176 2023-10-30T12:30:41.643 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 177 2023-10-30T12:30:42.667 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 178 2023-10-30T12:30:42.694 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 179 2023-10-30T12:30:42.963 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 180 2023-10-30T12:30:42.992 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 181 2023-10-30T12:30:43.257 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 182 2023-10-30T12:30:43.278 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 183 2023-10-30T12:30:43.299 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 184 2023-10-30T12:30:43.321 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 185 2023-10-30T12:30:43.339 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 186 2023-10-30T12:30:43.357 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 187 2023-10-30T12:30:43.637 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 188 2023-10-30T12:30:43.657 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 189 2023-10-30T12:30:43.678 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 190 2023-10-30T12:30:43.700 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 191 2023-10-30T12:30:43.721 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 192 2023-10-30T12:30:43.752 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 193 2023-10-30T12:30:43.792 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 194 2023-10-30T12:30:43.830 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 195 2023-10-30T12:30:43.868 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 196 2023-10-30T12:30:43.904 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 197 2023-10-30T12:30:43.950 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 198 2023-10-30T12:30:43.990 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 199 2023-10-30T12:30:44.025 INFO:tasks.workunit.client.0.smithi089.stdout:Take first snapshot .snap/test1 2023-10-30T12:30:44.656 INFO:tasks.workunit.client.0.smithi089.stdout:Create dir 200 to 299 ... 2023-10-30T12:30:44.658 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 200 2023-10-30T12:30:44.802 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:30:44.807 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:30:44.920 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 201 2023-10-30T12:30:44.956 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 202 2023-10-30T12:30:44.982 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 203 2023-10-30T12:30:45.002 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 204 2023-10-30T12:30:45.021 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 205 2023-10-30T12:30:45.038 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 206 2023-10-30T12:30:45.054 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 207 2023-10-30T12:30:45.070 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 208 2023-10-30T12:30:45.084 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 209 2023-10-30T12:30:45.098 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 210 2023-10-30T12:30:45.112 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 211 2023-10-30T12:30:45.125 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 212 2023-10-30T12:30:45.138 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 213 2023-10-30T12:30:45.159 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 214 2023-10-30T12:30:45.642 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 215 2023-10-30T12:30:45.964 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 216 2023-10-30T12:30:45.981 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 217 2023-10-30T12:30:46.001 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 218 2023-10-30T12:30:46.022 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 219 2023-10-30T12:30:46.042 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 220 2023-10-30T12:30:46.060 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 221 2023-10-30T12:30:46.093 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 222 2023-10-30T12:30:46.111 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 223 2023-10-30T12:30:46.130 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 224 2023-10-30T12:30:46.564 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 225 2023-10-30T12:30:46.610 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 226 2023-10-30T12:30:46.630 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 227 2023-10-30T12:30:46.651 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 228 2023-10-30T12:30:46.671 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 229 2023-10-30T12:30:46.691 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 230 2023-10-30T12:30:46.709 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 231 2023-10-30T12:30:46.726 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 232 2023-10-30T12:30:46.741 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 233 2023-10-30T12:30:46.756 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 234 2023-10-30T12:30:47.310 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 235 2023-10-30T12:30:47.327 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 236 2023-10-30T12:30:47.347 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 237 2023-10-30T12:30:47.366 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 238 2023-10-30T12:30:47.385 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 239 2023-10-30T12:30:47.404 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 240 2023-10-30T12:30:47.421 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 241 2023-10-30T12:30:47.438 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 242 2023-10-30T12:30:47.452 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 243 2023-10-30T12:30:47.467 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 244 2023-10-30T12:30:47.481 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 245 2023-10-30T12:30:47.494 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 246 2023-10-30T12:30:47.756 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 247 2023-10-30T12:30:47.776 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 248 2023-10-30T12:30:47.796 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 249 2023-10-30T12:30:47.816 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 250 2023-10-30T12:30:47.893 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 251 2023-10-30T12:30:51.452 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 252 2023-10-30T12:30:51.468 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 253 2023-10-30T12:30:51.486 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 254 2023-10-30T12:30:51.659 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 255 2023-10-30T12:30:51.832 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 256 2023-10-30T12:30:51.852 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 257 2023-10-30T12:30:51.872 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 258 2023-10-30T12:30:51.892 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 259 2023-10-30T12:30:51.912 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 260 2023-10-30T12:30:51.929 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 261 2023-10-30T12:30:51.946 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 262 2023-10-30T12:30:51.971 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 263 2023-10-30T12:30:51.987 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 264 2023-10-30T12:30:52.011 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 265 2023-10-30T12:30:52.037 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 266 2023-10-30T12:30:52.063 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 267 2023-10-30T12:30:52.089 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 268 2023-10-30T12:30:52.116 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 269 2023-10-30T12:30:52.145 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 270 2023-10-30T12:30:52.181 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 271 2023-10-30T12:30:52.459 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 272 2023-10-30T12:30:52.493 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 273 2023-10-30T12:30:52.523 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 274 2023-10-30T12:30:52.551 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 275 2023-10-30T12:30:52.582 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 276 2023-10-30T12:30:52.614 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 277 2023-10-30T12:30:52.648 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 278 2023-10-30T12:30:52.685 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 279 2023-10-30T12:30:52.721 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 280 2023-10-30T12:30:52.754 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 281 2023-10-30T12:30:52.795 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 282 2023-10-30T12:30:52.837 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 283 2023-10-30T12:30:52.875 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 284 2023-10-30T12:30:52.903 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 285 2023-10-30T12:30:52.937 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 286 2023-10-30T12:30:52.972 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 287 2023-10-30T12:30:53.004 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 288 2023-10-30T12:30:53.021 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 289 2023-10-30T12:30:53.042 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 290 2023-10-30T12:30:53.076 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 291 2023-10-30T12:30:53.120 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 292 2023-10-30T12:30:53.162 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 293 2023-10-30T12:30:53.191 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 294 2023-10-30T12:30:53.221 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 295 2023-10-30T12:30:53.249 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 296 2023-10-30T12:30:53.281 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 297 2023-10-30T12:30:53.305 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 298 2023-10-30T12:30:53.325 INFO:tasks.workunit.client.0.smithi089.stdout: create dir 299 2023-10-30T12:30:53.591 INFO:tasks.workunit.client.0.smithi089.stdout:Create a snapshot in every first level dir ... 2023-10-30T12:30:54.422 INFO:tasks.workunit.client.0.smithi089.stdout: create 100/.snap/snap-subdir-test 2023-10-30T12:30:54.464 INFO:tasks.workunit.client.0.smithi089.stdout: create 100/30 file after the snapshot 2023-10-30T12:30:54.467 INFO:tasks.workunit.client.0.smithi089.stdout: create 100/31 file after the snapshot 2023-10-30T12:30:54.467 INFO:tasks.workunit.client.0.smithi089.stdout: create 100/32 file after the snapshot 2023-10-30T12:30:54.467 INFO:tasks.workunit.client.0.smithi089.stdout: create 100/33 file after the snapshot 2023-10-30T12:30:54.467 INFO:tasks.workunit.client.0.smithi089.stdout: create 100/34 file after the snapshot 2023-10-30T12:30:54.468 INFO:tasks.workunit.client.0.smithi089.stdout: create 100/35 file after the snapshot 2023-10-30T12:30:54.468 INFO:tasks.workunit.client.0.smithi089.stdout: create 100/36 file after the snapshot 2023-10-30T12:30:54.468 INFO:tasks.workunit.client.0.smithi089.stdout: create 100/37 file after the snapshot 2023-10-30T12:30:54.468 INFO:tasks.workunit.client.0.smithi089.stdout: create 100/38 file after the snapshot 2023-10-30T12:30:54.469 INFO:tasks.workunit.client.0.smithi089.stdout: create 100/39 file after the snapshot 2023-10-30T12:30:54.469 INFO:tasks.workunit.client.0.smithi089.stdout: create 101/.snap/snap-subdir-test 2023-10-30T12:30:55.497 INFO:tasks.workunit.client.0.smithi089.stdout: create 101/30 file after the snapshot 2023-10-30T12:30:55.505 INFO:tasks.workunit.client.0.smithi089.stdout: create 101/31 file after the snapshot 2023-10-30T12:30:55.506 INFO:tasks.workunit.client.0.smithi089.stdout: create 101/32 file after the snapshot 2023-10-30T12:30:55.506 INFO:tasks.workunit.client.0.smithi089.stdout: create 101/33 file after the snapshot 2023-10-30T12:30:55.506 INFO:tasks.workunit.client.0.smithi089.stdout: create 101/34 file after the snapshot 2023-10-30T12:30:55.506 INFO:tasks.workunit.client.0.smithi089.stdout: create 101/35 file after the snapshot 2023-10-30T12:30:55.507 INFO:tasks.workunit.client.0.smithi089.stdout: create 101/36 file after the snapshot 2023-10-30T12:30:55.507 INFO:tasks.workunit.client.0.smithi089.stdout: create 101/37 file after the snapshot 2023-10-30T12:30:55.507 INFO:tasks.workunit.client.0.smithi089.stdout: create 101/38 file after the snapshot 2023-10-30T12:30:55.507 INFO:tasks.workunit.client.0.smithi089.stdout: create 101/39 file after the snapshot 2023-10-30T12:30:55.507 INFO:tasks.workunit.client.0.smithi089.stdout: create 102/.snap/snap-subdir-test 2023-10-30T12:30:55.586 INFO:tasks.workunit.client.0.smithi089.stdout: create 102/30 file after the snapshot 2023-10-30T12:30:55.589 INFO:tasks.workunit.client.0.smithi089.stdout: create 102/31 file after the snapshot 2023-10-30T12:30:55.589 INFO:tasks.workunit.client.0.smithi089.stdout: create 102/32 file after the snapshot 2023-10-30T12:30:55.590 INFO:tasks.workunit.client.0.smithi089.stdout: create 102/33 file after the snapshot 2023-10-30T12:30:55.590 INFO:tasks.workunit.client.0.smithi089.stdout: create 102/34 file after the snapshot 2023-10-30T12:30:55.590 INFO:tasks.workunit.client.0.smithi089.stdout: create 102/35 file after the snapshot 2023-10-30T12:30:55.590 INFO:tasks.workunit.client.0.smithi089.stdout: create 102/36 file after the snapshot 2023-10-30T12:30:55.591 INFO:tasks.workunit.client.0.smithi089.stdout: create 102/37 file after the snapshot 2023-10-30T12:30:55.591 INFO:tasks.workunit.client.0.smithi089.stdout: create 102/38 file after the snapshot 2023-10-30T12:30:55.591 INFO:tasks.workunit.client.0.smithi089.stdout: create 102/39 file after the snapshot 2023-10-30T12:30:55.591 INFO:tasks.workunit.client.0.smithi089.stdout: create 103/.snap/snap-subdir-test 2023-10-30T12:30:55.657 INFO:tasks.workunit.client.0.smithi089.stdout: create 103/30 file after the snapshot 2023-10-30T12:30:55.662 INFO:tasks.workunit.client.0.smithi089.stdout: create 103/31 file after the snapshot 2023-10-30T12:30:55.663 INFO:tasks.workunit.client.0.smithi089.stdout: create 103/32 file after the snapshot 2023-10-30T12:30:55.663 INFO:tasks.workunit.client.0.smithi089.stdout: create 103/33 file after the snapshot 2023-10-30T12:30:55.663 INFO:tasks.workunit.client.0.smithi089.stdout: create 103/34 file after the snapshot 2023-10-30T12:30:55.663 INFO:tasks.workunit.client.0.smithi089.stdout: create 103/35 file after the snapshot 2023-10-30T12:30:55.664 INFO:tasks.workunit.client.0.smithi089.stdout: create 103/36 file after the snapshot 2023-10-30T12:30:55.664 INFO:tasks.workunit.client.0.smithi089.stdout: create 103/37 file after the snapshot 2023-10-30T12:30:55.664 INFO:tasks.workunit.client.0.smithi089.stdout: create 103/38 file after the snapshot 2023-10-30T12:30:55.664 INFO:tasks.workunit.client.0.smithi089.stdout: create 103/39 file after the snapshot 2023-10-30T12:30:55.664 INFO:tasks.workunit.client.0.smithi089.stdout: create 104/.snap/snap-subdir-test 2023-10-30T12:30:55.731 INFO:tasks.workunit.client.0.smithi089.stdout: create 104/30 file after the snapshot 2023-10-30T12:30:55.735 INFO:tasks.workunit.client.0.smithi089.stdout: create 104/31 file after the snapshot 2023-10-30T12:30:55.735 INFO:tasks.workunit.client.0.smithi089.stdout: create 104/32 file after the snapshot 2023-10-30T12:30:55.736 INFO:tasks.workunit.client.0.smithi089.stdout: create 104/33 file after the snapshot 2023-10-30T12:30:55.736 INFO:tasks.workunit.client.0.smithi089.stdout: create 104/34 file after the snapshot 2023-10-30T12:30:55.736 INFO:tasks.workunit.client.0.smithi089.stdout: create 104/35 file after the snapshot 2023-10-30T12:30:55.736 INFO:tasks.workunit.client.0.smithi089.stdout: create 104/36 file after the snapshot 2023-10-30T12:30:55.736 INFO:tasks.workunit.client.0.smithi089.stdout: create 104/37 file after the snapshot 2023-10-30T12:30:55.737 INFO:tasks.workunit.client.0.smithi089.stdout: create 104/38 file after the snapshot 2023-10-30T12:30:55.737 INFO:tasks.workunit.client.0.smithi089.stdout: create 104/39 file after the snapshot 2023-10-30T12:30:55.737 INFO:tasks.workunit.client.0.smithi089.stdout: create 105/.snap/snap-subdir-test 2023-10-30T12:30:55.802 INFO:tasks.workunit.client.0.smithi089.stdout: create 105/30 file after the snapshot 2023-10-30T12:30:56.054 INFO:tasks.workunit.client.0.smithi089.stdout: create 105/31 file after the snapshot 2023-10-30T12:30:56.055 INFO:tasks.workunit.client.0.smithi089.stdout: create 105/32 file after the snapshot 2023-10-30T12:30:56.055 INFO:tasks.workunit.client.0.smithi089.stdout: create 105/33 file after the snapshot 2023-10-30T12:30:56.055 INFO:tasks.workunit.client.0.smithi089.stdout: create 105/34 file after the snapshot 2023-10-30T12:30:56.055 INFO:tasks.workunit.client.0.smithi089.stdout: create 105/35 file after the snapshot 2023-10-30T12:30:56.056 INFO:tasks.workunit.client.0.smithi089.stdout: create 105/36 file after the snapshot 2023-10-30T12:30:56.056 INFO:tasks.workunit.client.0.smithi089.stdout: create 105/37 file after the snapshot 2023-10-30T12:30:56.056 INFO:tasks.workunit.client.0.smithi089.stdout: create 105/38 file after the snapshot 2023-10-30T12:30:56.056 INFO:tasks.workunit.client.0.smithi089.stdout: create 105/39 file after the snapshot 2023-10-30T12:30:56.056 INFO:tasks.workunit.client.0.smithi089.stdout: create 106/.snap/snap-subdir-test 2023-10-30T12:30:56.121 INFO:tasks.workunit.client.0.smithi089.stdout: create 106/30 file after the snapshot 2023-10-30T12:30:56.125 INFO:tasks.workunit.client.0.smithi089.stdout: create 106/31 file after the snapshot 2023-10-30T12:30:56.125 INFO:tasks.workunit.client.0.smithi089.stdout: create 106/32 file after the snapshot 2023-10-30T12:30:56.125 INFO:tasks.workunit.client.0.smithi089.stdout: create 106/33 file after the snapshot 2023-10-30T12:30:56.125 INFO:tasks.workunit.client.0.smithi089.stdout: create 106/34 file after the snapshot 2023-10-30T12:30:56.125 INFO:tasks.workunit.client.0.smithi089.stdout: create 106/35 file after the snapshot 2023-10-30T12:30:56.126 INFO:tasks.workunit.client.0.smithi089.stdout: create 106/36 file after the snapshot 2023-10-30T12:30:56.126 INFO:tasks.workunit.client.0.smithi089.stdout: create 106/37 file after the snapshot 2023-10-30T12:30:56.126 INFO:tasks.workunit.client.0.smithi089.stdout: create 106/38 file after the snapshot 2023-10-30T12:30:56.126 INFO:tasks.workunit.client.0.smithi089.stdout: create 106/39 file after the snapshot 2023-10-30T12:30:56.127 INFO:tasks.workunit.client.0.smithi089.stdout: create 107/.snap/snap-subdir-test 2023-10-30T12:30:56.187 INFO:tasks.workunit.client.0.smithi089.stdout: create 107/30 file after the snapshot 2023-10-30T12:30:56.192 INFO:tasks.workunit.client.0.smithi089.stdout: create 107/31 file after the snapshot 2023-10-30T12:30:56.193 INFO:tasks.workunit.client.0.smithi089.stdout: create 107/32 file after the snapshot 2023-10-30T12:30:56.193 INFO:tasks.workunit.client.0.smithi089.stdout: create 107/33 file after the snapshot 2023-10-30T12:30:56.193 INFO:tasks.workunit.client.0.smithi089.stdout: create 107/34 file after the snapshot 2023-10-30T12:30:56.193 INFO:tasks.workunit.client.0.smithi089.stdout: create 107/35 file after the snapshot 2023-10-30T12:30:56.193 INFO:tasks.workunit.client.0.smithi089.stdout: create 107/36 file after the snapshot 2023-10-30T12:30:56.194 INFO:tasks.workunit.client.0.smithi089.stdout: create 107/37 file after the snapshot 2023-10-30T12:30:56.194 INFO:tasks.workunit.client.0.smithi089.stdout: create 107/38 file after the snapshot 2023-10-30T12:30:56.194 INFO:tasks.workunit.client.0.smithi089.stdout: create 107/39 file after the snapshot 2023-10-30T12:30:56.194 INFO:tasks.workunit.client.0.smithi089.stdout: create 108/.snap/snap-subdir-test 2023-10-30T12:30:56.261 INFO:tasks.workunit.client.0.smithi089.stdout: create 108/30 file after the snapshot 2023-10-30T12:30:56.264 INFO:tasks.workunit.client.0.smithi089.stdout: create 108/31 file after the snapshot 2023-10-30T12:30:56.264 INFO:tasks.workunit.client.0.smithi089.stdout: create 108/32 file after the snapshot 2023-10-30T12:30:56.264 INFO:tasks.workunit.client.0.smithi089.stdout: create 108/33 file after the snapshot 2023-10-30T12:30:56.265 INFO:tasks.workunit.client.0.smithi089.stdout: create 108/34 file after the snapshot 2023-10-30T12:30:56.265 INFO:tasks.workunit.client.0.smithi089.stdout: create 108/35 file after the snapshot 2023-10-30T12:30:56.265 INFO:tasks.workunit.client.0.smithi089.stdout: create 108/36 file after the snapshot 2023-10-30T12:30:56.265 INFO:tasks.workunit.client.0.smithi089.stdout: create 108/37 file after the snapshot 2023-10-30T12:30:56.265 INFO:tasks.workunit.client.0.smithi089.stdout: create 108/38 file after the snapshot 2023-10-30T12:30:56.266 INFO:tasks.workunit.client.0.smithi089.stdout: create 108/39 file after the snapshot 2023-10-30T12:30:56.266 INFO:tasks.workunit.client.0.smithi089.stdout: create 109/.snap/snap-subdir-test 2023-10-30T12:30:56.329 INFO:tasks.workunit.client.0.smithi089.stdout: create 109/30 file after the snapshot 2023-10-30T12:30:56.331 INFO:tasks.workunit.client.0.smithi089.stdout: create 109/31 file after the snapshot 2023-10-30T12:30:56.332 INFO:tasks.workunit.client.0.smithi089.stdout: create 109/32 file after the snapshot 2023-10-30T12:30:56.332 INFO:tasks.workunit.client.0.smithi089.stdout: create 109/33 file after the snapshot 2023-10-30T12:30:56.332 INFO:tasks.workunit.client.0.smithi089.stdout: create 109/34 file after the snapshot 2023-10-30T12:30:56.332 INFO:tasks.workunit.client.0.smithi089.stdout: create 109/35 file after the snapshot 2023-10-30T12:30:56.332 INFO:tasks.workunit.client.0.smithi089.stdout: create 109/36 file after the snapshot 2023-10-30T12:30:56.333 INFO:tasks.workunit.client.0.smithi089.stdout: create 109/37 file after the snapshot 2023-10-30T12:30:56.333 INFO:tasks.workunit.client.0.smithi089.stdout: create 109/38 file after the snapshot 2023-10-30T12:30:56.333 INFO:tasks.workunit.client.0.smithi089.stdout: create 109/39 file after the snapshot 2023-10-30T12:30:56.333 INFO:tasks.workunit.client.0.smithi089.stdout: create 110/.snap/snap-subdir-test 2023-10-30T12:30:56.393 INFO:tasks.workunit.client.0.smithi089.stdout: create 110/30 file after the snapshot 2023-10-30T12:30:56.397 INFO:tasks.workunit.client.0.smithi089.stdout: create 110/31 file after the snapshot 2023-10-30T12:30:56.398 INFO:tasks.workunit.client.0.smithi089.stdout: create 110/32 file after the snapshot 2023-10-30T12:30:56.398 INFO:tasks.workunit.client.0.smithi089.stdout: create 110/33 file after the snapshot 2023-10-30T12:30:56.398 INFO:tasks.workunit.client.0.smithi089.stdout: create 110/34 file after the snapshot 2023-10-30T12:30:56.398 INFO:tasks.workunit.client.0.smithi089.stdout: create 110/35 file after the snapshot 2023-10-30T12:30:56.398 INFO:tasks.workunit.client.0.smithi089.stdout: create 110/36 file after the snapshot 2023-10-30T12:30:56.399 INFO:tasks.workunit.client.0.smithi089.stdout: create 110/37 file after the snapshot 2023-10-30T12:30:56.399 INFO:tasks.workunit.client.0.smithi089.stdout: create 110/38 file after the snapshot 2023-10-30T12:30:56.399 INFO:tasks.workunit.client.0.smithi089.stdout: create 110/39 file after the snapshot 2023-10-30T12:30:56.399 INFO:tasks.workunit.client.0.smithi089.stdout: create 111/.snap/snap-subdir-test 2023-10-30T12:30:56.459 INFO:tasks.workunit.client.0.smithi089.stdout: create 111/30 file after the snapshot 2023-10-30T12:30:56.464 INFO:tasks.workunit.client.0.smithi089.stdout: create 111/31 file after the snapshot 2023-10-30T12:30:56.464 INFO:tasks.workunit.client.0.smithi089.stdout: create 111/32 file after the snapshot 2023-10-30T12:30:56.464 INFO:tasks.workunit.client.0.smithi089.stdout: create 111/33 file after the snapshot 2023-10-30T12:30:56.465 INFO:tasks.workunit.client.0.smithi089.stdout: create 111/34 file after the snapshot 2023-10-30T12:30:56.465 INFO:tasks.workunit.client.0.smithi089.stdout: create 111/35 file after the snapshot 2023-10-30T12:30:56.465 INFO:tasks.workunit.client.0.smithi089.stdout: create 111/36 file after the snapshot 2023-10-30T12:30:56.465 INFO:tasks.workunit.client.0.smithi089.stdout: create 111/37 file after the snapshot 2023-10-30T12:30:56.465 INFO:tasks.workunit.client.0.smithi089.stdout: create 111/38 file after the snapshot 2023-10-30T12:30:56.465 INFO:tasks.workunit.client.0.smithi089.stdout: create 111/39 file after the snapshot 2023-10-30T12:30:56.466 INFO:tasks.workunit.client.0.smithi089.stdout: create 112/.snap/snap-subdir-test 2023-10-30T12:30:56.527 INFO:tasks.workunit.client.0.smithi089.stdout: create 112/30 file after the snapshot 2023-10-30T12:30:56.531 INFO:tasks.workunit.client.0.smithi089.stdout: create 112/31 file after the snapshot 2023-10-30T12:30:56.531 INFO:tasks.workunit.client.0.smithi089.stdout: create 112/32 file after the snapshot 2023-10-30T12:30:56.531 INFO:tasks.workunit.client.0.smithi089.stdout: create 112/33 file after the snapshot 2023-10-30T12:30:56.532 INFO:tasks.workunit.client.0.smithi089.stdout: create 112/34 file after the snapshot 2023-10-30T12:30:56.532 INFO:tasks.workunit.client.0.smithi089.stdout: create 112/35 file after the snapshot 2023-10-30T12:30:56.532 INFO:tasks.workunit.client.0.smithi089.stdout: create 112/36 file after the snapshot 2023-10-30T12:30:56.532 INFO:tasks.workunit.client.0.smithi089.stdout: create 112/37 file after the snapshot 2023-10-30T12:30:56.532 INFO:tasks.workunit.client.0.smithi089.stdout: create 112/38 file after the snapshot 2023-10-30T12:30:56.532 INFO:tasks.workunit.client.0.smithi089.stdout: create 112/39 file after the snapshot 2023-10-30T12:30:56.533 INFO:tasks.workunit.client.0.smithi089.stdout: create 113/.snap/snap-subdir-test 2023-10-30T12:30:56.782 INFO:tasks.workunit.client.0.smithi089.stdout: create 113/30 file after the snapshot 2023-10-30T12:30:56.784 INFO:tasks.workunit.client.0.smithi089.stdout: create 113/31 file after the snapshot 2023-10-30T12:30:56.785 INFO:tasks.workunit.client.0.smithi089.stdout: create 113/32 file after the snapshot 2023-10-30T12:30:56.785 INFO:tasks.workunit.client.0.smithi089.stdout: create 113/33 file after the snapshot 2023-10-30T12:30:56.785 INFO:tasks.workunit.client.0.smithi089.stdout: create 113/34 file after the snapshot 2023-10-30T12:30:56.785 INFO:tasks.workunit.client.0.smithi089.stdout: create 113/35 file after the snapshot 2023-10-30T12:30:56.786 INFO:tasks.workunit.client.0.smithi089.stdout: create 113/36 file after the snapshot 2023-10-30T12:30:56.786 INFO:tasks.workunit.client.0.smithi089.stdout: create 113/37 file after the snapshot 2023-10-30T12:30:56.786 INFO:tasks.workunit.client.0.smithi089.stdout: create 113/38 file after the snapshot 2023-10-30T12:30:56.786 INFO:tasks.workunit.client.0.smithi089.stdout: create 113/39 file after the snapshot 2023-10-30T12:30:56.786 INFO:tasks.workunit.client.0.smithi089.stdout: create 114/.snap/snap-subdir-test 2023-10-30T12:30:56.860 INFO:tasks.workunit.client.0.smithi089.stdout: create 114/30 file after the snapshot 2023-10-30T12:30:56.863 INFO:tasks.workunit.client.0.smithi089.stdout: create 114/31 file after the snapshot 2023-10-30T12:30:56.863 INFO:tasks.workunit.client.0.smithi089.stdout: create 114/32 file after the snapshot 2023-10-30T12:30:56.863 INFO:tasks.workunit.client.0.smithi089.stdout: create 114/33 file after the snapshot 2023-10-30T12:30:56.864 INFO:tasks.workunit.client.0.smithi089.stdout: create 114/34 file after the snapshot 2023-10-30T12:30:56.864 INFO:tasks.workunit.client.0.smithi089.stdout: create 114/35 file after the snapshot 2023-10-30T12:30:56.864 INFO:tasks.workunit.client.0.smithi089.stdout: create 114/36 file after the snapshot 2023-10-30T12:30:56.864 INFO:tasks.workunit.client.0.smithi089.stdout: create 114/37 file after the snapshot 2023-10-30T12:30:56.864 INFO:tasks.workunit.client.0.smithi089.stdout: create 114/38 file after the snapshot 2023-10-30T12:30:56.865 INFO:tasks.workunit.client.0.smithi089.stdout: create 114/39 file after the snapshot 2023-10-30T12:30:56.865 INFO:tasks.workunit.client.0.smithi089.stdout: create 115/.snap/snap-subdir-test 2023-10-30T12:30:56.934 INFO:tasks.workunit.client.0.smithi089.stdout: create 115/30 file after the snapshot 2023-10-30T12:30:56.937 INFO:tasks.workunit.client.0.smithi089.stdout: create 115/31 file after the snapshot 2023-10-30T12:30:56.938 INFO:tasks.workunit.client.0.smithi089.stdout: create 115/32 file after the snapshot 2023-10-30T12:30:56.938 INFO:tasks.workunit.client.0.smithi089.stdout: create 115/33 file after the snapshot 2023-10-30T12:30:56.938 INFO:tasks.workunit.client.0.smithi089.stdout: create 115/34 file after the snapshot 2023-10-30T12:30:56.938 INFO:tasks.workunit.client.0.smithi089.stdout: create 115/35 file after the snapshot 2023-10-30T12:30:56.938 INFO:tasks.workunit.client.0.smithi089.stdout: create 115/36 file after the snapshot 2023-10-30T12:30:56.939 INFO:tasks.workunit.client.0.smithi089.stdout: create 115/37 file after the snapshot 2023-10-30T12:30:56.939 INFO:tasks.workunit.client.0.smithi089.stdout: create 115/38 file after the snapshot 2023-10-30T12:30:56.939 INFO:tasks.workunit.client.0.smithi089.stdout: create 115/39 file after the snapshot 2023-10-30T12:30:56.940 INFO:tasks.workunit.client.0.smithi089.stdout: create 116/.snap/snap-subdir-test 2023-10-30T12:30:57.022 INFO:tasks.workunit.client.0.smithi089.stdout: create 116/30 file after the snapshot 2023-10-30T12:30:57.029 INFO:tasks.workunit.client.0.smithi089.stdout: create 116/31 file after the snapshot 2023-10-30T12:30:57.029 INFO:tasks.workunit.client.0.smithi089.stdout: create 116/32 file after the snapshot 2023-10-30T12:30:57.029 INFO:tasks.workunit.client.0.smithi089.stdout: create 116/33 file after the snapshot 2023-10-30T12:30:57.029 INFO:tasks.workunit.client.0.smithi089.stdout: create 116/34 file after the snapshot 2023-10-30T12:30:57.029 INFO:tasks.workunit.client.0.smithi089.stdout: create 116/35 file after the snapshot 2023-10-30T12:30:57.030 INFO:tasks.workunit.client.0.smithi089.stdout: create 116/36 file after the snapshot 2023-10-30T12:30:57.030 INFO:tasks.workunit.client.0.smithi089.stdout: create 116/37 file after the snapshot 2023-10-30T12:30:57.030 INFO:tasks.workunit.client.0.smithi089.stdout: create 116/38 file after the snapshot 2023-10-30T12:30:57.030 INFO:tasks.workunit.client.0.smithi089.stdout: create 116/39 file after the snapshot 2023-10-30T12:30:57.031 INFO:tasks.workunit.client.0.smithi089.stdout: create 117/.snap/snap-subdir-test 2023-10-30T12:30:57.135 INFO:tasks.workunit.client.0.smithi089.stdout: create 117/30 file after the snapshot 2023-10-30T12:30:57.140 INFO:tasks.workunit.client.0.smithi089.stdout: create 117/31 file after the snapshot 2023-10-30T12:30:57.140 INFO:tasks.workunit.client.0.smithi089.stdout: create 117/32 file after the snapshot 2023-10-30T12:30:57.140 INFO:tasks.workunit.client.0.smithi089.stdout: create 117/33 file after the snapshot 2023-10-30T12:30:57.140 INFO:tasks.workunit.client.0.smithi089.stdout: create 117/34 file after the snapshot 2023-10-30T12:30:57.140 INFO:tasks.workunit.client.0.smithi089.stdout: create 117/35 file after the snapshot 2023-10-30T12:30:57.141 INFO:tasks.workunit.client.0.smithi089.stdout: create 117/36 file after the snapshot 2023-10-30T12:30:57.141 INFO:tasks.workunit.client.0.smithi089.stdout: create 117/37 file after the snapshot 2023-10-30T12:30:57.141 INFO:tasks.workunit.client.0.smithi089.stdout: create 117/38 file after the snapshot 2023-10-30T12:30:57.141 INFO:tasks.workunit.client.0.smithi089.stdout: create 117/39 file after the snapshot 2023-10-30T12:30:57.141 INFO:tasks.workunit.client.0.smithi089.stdout: create 118/.snap/snap-subdir-test 2023-10-30T12:30:57.190 INFO:tasks.workunit.client.0.smithi089.stdout: create 118/30 file after the snapshot 2023-10-30T12:30:57.195 INFO:tasks.workunit.client.0.smithi089.stdout: create 118/31 file after the snapshot 2023-10-30T12:30:57.196 INFO:tasks.workunit.client.0.smithi089.stdout: create 118/32 file after the snapshot 2023-10-30T12:30:57.196 INFO:tasks.workunit.client.0.smithi089.stdout: create 118/33 file after the snapshot 2023-10-30T12:30:57.196 INFO:tasks.workunit.client.0.smithi089.stdout: create 118/34 file after the snapshot 2023-10-30T12:30:57.196 INFO:tasks.workunit.client.0.smithi089.stdout: create 118/35 file after the snapshot 2023-10-30T12:30:57.196 INFO:tasks.workunit.client.0.smithi089.stdout: create 118/36 file after the snapshot 2023-10-30T12:30:57.196 INFO:tasks.workunit.client.0.smithi089.stdout: create 118/37 file after the snapshot 2023-10-30T12:30:57.197 INFO:tasks.workunit.client.0.smithi089.stdout: create 118/38 file after the snapshot 2023-10-30T12:30:57.197 INFO:tasks.workunit.client.0.smithi089.stdout: create 118/39 file after the snapshot 2023-10-30T12:30:57.197 INFO:tasks.workunit.client.0.smithi089.stdout: create 119/.snap/snap-subdir-test 2023-10-30T12:30:57.262 INFO:tasks.workunit.client.0.smithi089.stdout: create 119/30 file after the snapshot 2023-10-30T12:30:57.271 INFO:tasks.workunit.client.0.smithi089.stdout: create 119/31 file after the snapshot 2023-10-30T12:30:57.271 INFO:tasks.workunit.client.0.smithi089.stdout: create 119/32 file after the snapshot 2023-10-30T12:30:57.271 INFO:tasks.workunit.client.0.smithi089.stdout: create 119/33 file after the snapshot 2023-10-30T12:30:57.272 INFO:tasks.workunit.client.0.smithi089.stdout: create 119/34 file after the snapshot 2023-10-30T12:30:57.272 INFO:tasks.workunit.client.0.smithi089.stdout: create 119/35 file after the snapshot 2023-10-30T12:30:57.272 INFO:tasks.workunit.client.0.smithi089.stdout: create 119/36 file after the snapshot 2023-10-30T12:30:57.272 INFO:tasks.workunit.client.0.smithi089.stdout: create 119/37 file after the snapshot 2023-10-30T12:30:57.272 INFO:tasks.workunit.client.0.smithi089.stdout: create 119/38 file after the snapshot 2023-10-30T12:30:57.273 INFO:tasks.workunit.client.0.smithi089.stdout: create 119/39 file after the snapshot 2023-10-30T12:30:57.273 INFO:tasks.workunit.client.0.smithi089.stdout: create 120/.snap/snap-subdir-test 2023-10-30T12:30:57.321 INFO:tasks.workunit.client.0.smithi089.stdout: create 120/30 file after the snapshot 2023-10-30T12:30:57.330 INFO:tasks.workunit.client.0.smithi089.stdout: create 120/31 file after the snapshot 2023-10-30T12:30:57.330 INFO:tasks.workunit.client.0.smithi089.stdout: create 120/32 file after the snapshot 2023-10-30T12:30:57.330 INFO:tasks.workunit.client.0.smithi089.stdout: create 120/33 file after the snapshot 2023-10-30T12:30:57.331 INFO:tasks.workunit.client.0.smithi089.stdout: create 120/34 file after the snapshot 2023-10-30T12:30:57.331 INFO:tasks.workunit.client.0.smithi089.stdout: create 120/35 file after the snapshot 2023-10-30T12:30:57.331 INFO:tasks.workunit.client.0.smithi089.stdout: create 120/36 file after the snapshot 2023-10-30T12:30:57.331 INFO:tasks.workunit.client.0.smithi089.stdout: create 120/37 file after the snapshot 2023-10-30T12:30:57.331 INFO:tasks.workunit.client.0.smithi089.stdout: create 120/38 file after the snapshot 2023-10-30T12:30:57.331 INFO:tasks.workunit.client.0.smithi089.stdout: create 120/39 file after the snapshot 2023-10-30T12:30:57.332 INFO:tasks.workunit.client.0.smithi089.stdout: create 121/.snap/snap-subdir-test 2023-10-30T12:30:57.378 INFO:tasks.workunit.client.0.smithi089.stdout: create 121/30 file after the snapshot 2023-10-30T12:30:57.387 INFO:tasks.workunit.client.0.smithi089.stdout: create 121/31 file after the snapshot 2023-10-30T12:30:57.388 INFO:tasks.workunit.client.0.smithi089.stdout: create 121/32 file after the snapshot 2023-10-30T12:30:57.388 INFO:tasks.workunit.client.0.smithi089.stdout: create 121/33 file after the snapshot 2023-10-30T12:30:57.388 INFO:tasks.workunit.client.0.smithi089.stdout: create 121/34 file after the snapshot 2023-10-30T12:30:57.388 INFO:tasks.workunit.client.0.smithi089.stdout: create 121/35 file after the snapshot 2023-10-30T12:30:57.388 INFO:tasks.workunit.client.0.smithi089.stdout: create 121/36 file after the snapshot 2023-10-30T12:30:57.388 INFO:tasks.workunit.client.0.smithi089.stdout: create 121/37 file after the snapshot 2023-10-30T12:30:57.389 INFO:tasks.workunit.client.0.smithi089.stdout: create 121/38 file after the snapshot 2023-10-30T12:30:57.389 INFO:tasks.workunit.client.0.smithi089.stdout: create 121/39 file after the snapshot 2023-10-30T12:30:57.389 INFO:tasks.workunit.client.0.smithi089.stdout: create 122/.snap/snap-subdir-test 2023-10-30T12:30:57.448 INFO:tasks.workunit.client.0.smithi089.stdout: create 122/30 file after the snapshot 2023-10-30T12:30:57.455 INFO:tasks.workunit.client.0.smithi089.stdout: create 122/31 file after the snapshot 2023-10-30T12:30:57.455 INFO:tasks.workunit.client.0.smithi089.stdout: create 122/32 file after the snapshot 2023-10-30T12:30:57.455 INFO:tasks.workunit.client.0.smithi089.stdout: create 122/33 file after the snapshot 2023-10-30T12:30:57.456 INFO:tasks.workunit.client.0.smithi089.stdout: create 122/34 file after the snapshot 2023-10-30T12:30:57.456 INFO:tasks.workunit.client.0.smithi089.stdout: create 122/35 file after the snapshot 2023-10-30T12:30:57.456 INFO:tasks.workunit.client.0.smithi089.stdout: create 122/36 file after the snapshot 2023-10-30T12:30:57.456 INFO:tasks.workunit.client.0.smithi089.stdout: create 122/37 file after the snapshot 2023-10-30T12:30:57.456 INFO:tasks.workunit.client.0.smithi089.stdout: create 122/38 file after the snapshot 2023-10-30T12:30:57.456 INFO:tasks.workunit.client.0.smithi089.stdout: create 122/39 file after the snapshot 2023-10-30T12:30:57.457 INFO:tasks.workunit.client.0.smithi089.stdout: create 123/.snap/snap-subdir-test 2023-10-30T12:30:57.682 INFO:tasks.workunit.client.0.smithi089.stdout: create 123/30 file after the snapshot 2023-10-30T12:30:57.686 INFO:tasks.workunit.client.0.smithi089.stdout: create 123/31 file after the snapshot 2023-10-30T12:30:57.687 INFO:tasks.workunit.client.0.smithi089.stdout: create 123/32 file after the snapshot 2023-10-30T12:30:57.687 INFO:tasks.workunit.client.0.smithi089.stdout: create 123/33 file after the snapshot 2023-10-30T12:30:57.687 INFO:tasks.workunit.client.0.smithi089.stdout: create 123/34 file after the snapshot 2023-10-30T12:30:57.687 INFO:tasks.workunit.client.0.smithi089.stdout: create 123/35 file after the snapshot 2023-10-30T12:30:57.687 INFO:tasks.workunit.client.0.smithi089.stdout: create 123/36 file after the snapshot 2023-10-30T12:30:57.688 INFO:tasks.workunit.client.0.smithi089.stdout: create 123/37 file after the snapshot 2023-10-30T12:30:57.688 INFO:tasks.workunit.client.0.smithi089.stdout: create 123/38 file after the snapshot 2023-10-30T12:30:57.688 INFO:tasks.workunit.client.0.smithi089.stdout: create 123/39 file after the snapshot 2023-10-30T12:30:57.688 INFO:tasks.workunit.client.0.smithi089.stdout: create 124/.snap/snap-subdir-test 2023-10-30T12:30:57.765 INFO:tasks.workunit.client.0.smithi089.stdout: create 124/30 file after the snapshot 2023-10-30T12:30:57.769 INFO:tasks.workunit.client.0.smithi089.stdout: create 124/31 file after the snapshot 2023-10-30T12:30:57.769 INFO:tasks.workunit.client.0.smithi089.stdout: create 124/32 file after the snapshot 2023-10-30T12:30:57.770 INFO:tasks.workunit.client.0.smithi089.stdout: create 124/33 file after the snapshot 2023-10-30T12:30:57.770 INFO:tasks.workunit.client.0.smithi089.stdout: create 124/34 file after the snapshot 2023-10-30T12:30:57.770 INFO:tasks.workunit.client.0.smithi089.stdout: create 124/35 file after the snapshot 2023-10-30T12:30:57.770 INFO:tasks.workunit.client.0.smithi089.stdout: create 124/36 file after the snapshot 2023-10-30T12:30:57.770 INFO:tasks.workunit.client.0.smithi089.stdout: create 124/37 file after the snapshot 2023-10-30T12:30:57.771 INFO:tasks.workunit.client.0.smithi089.stdout: create 124/38 file after the snapshot 2023-10-30T12:30:57.771 INFO:tasks.workunit.client.0.smithi089.stdout: create 124/39 file after the snapshot 2023-10-30T12:30:57.771 INFO:tasks.workunit.client.0.smithi089.stdout: create 125/.snap/snap-subdir-test 2023-10-30T12:30:57.835 INFO:tasks.workunit.client.0.smithi089.stdout: create 125/30 file after the snapshot 2023-10-30T12:30:57.840 INFO:tasks.workunit.client.0.smithi089.stdout: create 125/31 file after the snapshot 2023-10-30T12:30:57.840 INFO:tasks.workunit.client.0.smithi089.stdout: create 125/32 file after the snapshot 2023-10-30T12:30:57.840 INFO:tasks.workunit.client.0.smithi089.stdout: create 125/33 file after the snapshot 2023-10-30T12:30:57.840 INFO:tasks.workunit.client.0.smithi089.stdout: create 125/34 file after the snapshot 2023-10-30T12:30:57.840 INFO:tasks.workunit.client.0.smithi089.stdout: create 125/35 file after the snapshot 2023-10-30T12:30:57.841 INFO:tasks.workunit.client.0.smithi089.stdout: create 125/36 file after the snapshot 2023-10-30T12:30:57.841 INFO:tasks.workunit.client.0.smithi089.stdout: create 125/37 file after the snapshot 2023-10-30T12:30:57.841 INFO:tasks.workunit.client.0.smithi089.stdout: create 125/38 file after the snapshot 2023-10-30T12:30:57.841 INFO:tasks.workunit.client.0.smithi089.stdout: create 125/39 file after the snapshot 2023-10-30T12:30:57.841 INFO:tasks.workunit.client.0.smithi089.stdout: create 126/.snap/snap-subdir-test 2023-10-30T12:30:57.886 INFO:tasks.workunit.client.0.smithi089.stdout: create 126/30 file after the snapshot 2023-10-30T12:30:57.891 INFO:tasks.workunit.client.0.smithi089.stdout: create 126/31 file after the snapshot 2023-10-30T12:30:57.892 INFO:tasks.workunit.client.0.smithi089.stdout: create 126/32 file after the snapshot 2023-10-30T12:30:57.892 INFO:tasks.workunit.client.0.smithi089.stdout: create 126/33 file after the snapshot 2023-10-30T12:30:57.892 INFO:tasks.workunit.client.0.smithi089.stdout: create 126/34 file after the snapshot 2023-10-30T12:30:57.892 INFO:tasks.workunit.client.0.smithi089.stdout: create 126/35 file after the snapshot 2023-10-30T12:30:57.892 INFO:tasks.workunit.client.0.smithi089.stdout: create 126/36 file after the snapshot 2023-10-30T12:30:57.893 INFO:tasks.workunit.client.0.smithi089.stdout: create 126/37 file after the snapshot 2023-10-30T12:30:57.893 INFO:tasks.workunit.client.0.smithi089.stdout: create 126/38 file after the snapshot 2023-10-30T12:30:57.893 INFO:tasks.workunit.client.0.smithi089.stdout: create 126/39 file after the snapshot 2023-10-30T12:30:57.893 INFO:tasks.workunit.client.0.smithi089.stdout: create 127/.snap/snap-subdir-test 2023-10-30T12:30:57.948 INFO:tasks.workunit.client.0.smithi089.stdout: create 127/30 file after the snapshot 2023-10-30T12:30:57.953 INFO:tasks.workunit.client.0.smithi089.stdout: create 127/31 file after the snapshot 2023-10-30T12:30:57.954 INFO:tasks.workunit.client.0.smithi089.stdout: create 127/32 file after the snapshot 2023-10-30T12:30:57.954 INFO:tasks.workunit.client.0.smithi089.stdout: create 127/33 file after the snapshot 2023-10-30T12:30:57.954 INFO:tasks.workunit.client.0.smithi089.stdout: create 127/34 file after the snapshot 2023-10-30T12:30:57.954 INFO:tasks.workunit.client.0.smithi089.stdout: create 127/35 file after the snapshot 2023-10-30T12:30:57.954 INFO:tasks.workunit.client.0.smithi089.stdout: create 127/36 file after the snapshot 2023-10-30T12:30:57.955 INFO:tasks.workunit.client.0.smithi089.stdout: create 127/37 file after the snapshot 2023-10-30T12:30:57.955 INFO:tasks.workunit.client.0.smithi089.stdout: create 127/38 file after the snapshot 2023-10-30T12:30:57.955 INFO:tasks.workunit.client.0.smithi089.stdout: create 127/39 file after the snapshot 2023-10-30T12:30:57.955 INFO:tasks.workunit.client.0.smithi089.stdout: create 128/.snap/snap-subdir-test 2023-10-30T12:30:57.999 INFO:tasks.workunit.client.0.smithi089.stdout: create 128/30 file after the snapshot 2023-10-30T12:30:58.005 INFO:tasks.workunit.client.0.smithi089.stdout: create 128/31 file after the snapshot 2023-10-30T12:30:58.005 INFO:tasks.workunit.client.0.smithi089.stdout: create 128/32 file after the snapshot 2023-10-30T12:30:58.005 INFO:tasks.workunit.client.0.smithi089.stdout: create 128/33 file after the snapshot 2023-10-30T12:30:58.006 INFO:tasks.workunit.client.0.smithi089.stdout: create 128/34 file after the snapshot 2023-10-30T12:30:58.006 INFO:tasks.workunit.client.0.smithi089.stdout: create 128/35 file after the snapshot 2023-10-30T12:30:58.006 INFO:tasks.workunit.client.0.smithi089.stdout: create 128/36 file after the snapshot 2023-10-30T12:30:58.006 INFO:tasks.workunit.client.0.smithi089.stdout: create 128/37 file after the snapshot 2023-10-30T12:30:58.006 INFO:tasks.workunit.client.0.smithi089.stdout: create 128/38 file after the snapshot 2023-10-30T12:30:58.007 INFO:tasks.workunit.client.0.smithi089.stdout: create 128/39 file after the snapshot 2023-10-30T12:30:58.007 INFO:tasks.workunit.client.0.smithi089.stdout: create 129/.snap/snap-subdir-test 2023-10-30T12:30:58.042 INFO:tasks.workunit.client.0.smithi089.stdout: create 129/30 file after the snapshot 2023-10-30T12:30:58.045 INFO:tasks.workunit.client.0.smithi089.stdout: create 129/31 file after the snapshot 2023-10-30T12:30:58.045 INFO:tasks.workunit.client.0.smithi089.stdout: create 129/32 file after the snapshot 2023-10-30T12:30:58.045 INFO:tasks.workunit.client.0.smithi089.stdout: create 129/33 file after the snapshot 2023-10-30T12:30:58.046 INFO:tasks.workunit.client.0.smithi089.stdout: create 129/34 file after the snapshot 2023-10-30T12:30:58.046 INFO:tasks.workunit.client.0.smithi089.stdout: create 129/35 file after the snapshot 2023-10-30T12:30:58.046 INFO:tasks.workunit.client.0.smithi089.stdout: create 129/36 file after the snapshot 2023-10-30T12:30:58.046 INFO:tasks.workunit.client.0.smithi089.stdout: create 129/37 file after the snapshot 2023-10-30T12:30:58.046 INFO:tasks.workunit.client.0.smithi089.stdout: create 129/38 file after the snapshot 2023-10-30T12:30:58.046 INFO:tasks.workunit.client.0.smithi089.stdout: create 129/39 file after the snapshot 2023-10-30T12:30:58.047 INFO:tasks.workunit.client.0.smithi089.stdout: create 130/.snap/snap-subdir-test 2023-10-30T12:30:58.429 INFO:tasks.workunit.client.0.smithi089.stdout: create 130/30 file after the snapshot 2023-10-30T12:30:58.433 INFO:tasks.workunit.client.0.smithi089.stdout: create 130/31 file after the snapshot 2023-10-30T12:30:58.433 INFO:tasks.workunit.client.0.smithi089.stdout: create 130/32 file after the snapshot 2023-10-30T12:30:58.434 INFO:tasks.workunit.client.0.smithi089.stdout: create 130/33 file after the snapshot 2023-10-30T12:30:58.434 INFO:tasks.workunit.client.0.smithi089.stdout: create 130/34 file after the snapshot 2023-10-30T12:30:58.434 INFO:tasks.workunit.client.0.smithi089.stdout: create 130/35 file after the snapshot 2023-10-30T12:30:58.434 INFO:tasks.workunit.client.0.smithi089.stdout: create 130/36 file after the snapshot 2023-10-30T12:30:58.434 INFO:tasks.workunit.client.0.smithi089.stdout: create 130/37 file after the snapshot 2023-10-30T12:30:58.435 INFO:tasks.workunit.client.0.smithi089.stdout: create 130/38 file after the snapshot 2023-10-30T12:30:58.435 INFO:tasks.workunit.client.0.smithi089.stdout: create 130/39 file after the snapshot 2023-10-30T12:30:58.435 INFO:tasks.workunit.client.0.smithi089.stdout: create 131/.snap/snap-subdir-test 2023-10-30T12:30:58.491 INFO:tasks.workunit.client.0.smithi089.stdout: create 131/30 file after the snapshot 2023-10-30T12:30:58.494 INFO:tasks.workunit.client.0.smithi089.stdout: create 131/31 file after the snapshot 2023-10-30T12:30:58.494 INFO:tasks.workunit.client.0.smithi089.stdout: create 131/32 file after the snapshot 2023-10-30T12:30:58.494 INFO:tasks.workunit.client.0.smithi089.stdout: create 131/33 file after the snapshot 2023-10-30T12:30:58.495 INFO:tasks.workunit.client.0.smithi089.stdout: create 131/34 file after the snapshot 2023-10-30T12:30:58.495 INFO:tasks.workunit.client.0.smithi089.stdout: create 131/35 file after the snapshot 2023-10-30T12:30:58.495 INFO:tasks.workunit.client.0.smithi089.stdout: create 131/36 file after the snapshot 2023-10-30T12:30:58.495 INFO:tasks.workunit.client.0.smithi089.stdout: create 131/37 file after the snapshot 2023-10-30T12:30:58.495 INFO:tasks.workunit.client.0.smithi089.stdout: create 131/38 file after the snapshot 2023-10-30T12:30:58.496 INFO:tasks.workunit.client.0.smithi089.stdout: create 131/39 file after the snapshot 2023-10-30T12:30:58.496 INFO:tasks.workunit.client.0.smithi089.stdout: create 132/.snap/snap-subdir-test 2023-10-30T12:30:58.557 INFO:tasks.workunit.client.0.smithi089.stdout: create 132/30 file after the snapshot 2023-10-30T12:30:58.562 INFO:tasks.workunit.client.0.smithi089.stdout: create 132/31 file after the snapshot 2023-10-30T12:30:58.562 INFO:tasks.workunit.client.0.smithi089.stdout: create 132/32 file after the snapshot 2023-10-30T12:30:58.562 INFO:tasks.workunit.client.0.smithi089.stdout: create 132/33 file after the snapshot 2023-10-30T12:30:58.562 INFO:tasks.workunit.client.0.smithi089.stdout: create 132/34 file after the snapshot 2023-10-30T12:30:58.562 INFO:tasks.workunit.client.0.smithi089.stdout: create 132/35 file after the snapshot 2023-10-30T12:30:58.563 INFO:tasks.workunit.client.0.smithi089.stdout: create 132/36 file after the snapshot 2023-10-30T12:30:58.563 INFO:tasks.workunit.client.0.smithi089.stdout: create 132/37 file after the snapshot 2023-10-30T12:30:58.563 INFO:tasks.workunit.client.0.smithi089.stdout: create 132/38 file after the snapshot 2023-10-30T12:30:58.563 INFO:tasks.workunit.client.0.smithi089.stdout: create 132/39 file after the snapshot 2023-10-30T12:30:58.563 INFO:tasks.workunit.client.0.smithi089.stdout: create 133/.snap/snap-subdir-test 2023-10-30T12:30:58.623 INFO:tasks.workunit.client.0.smithi089.stdout: create 133/30 file after the snapshot 2023-10-30T12:30:58.626 INFO:tasks.workunit.client.0.smithi089.stdout: create 133/31 file after the snapshot 2023-10-30T12:30:58.626 INFO:tasks.workunit.client.0.smithi089.stdout: create 133/32 file after the snapshot 2023-10-30T12:30:58.626 INFO:tasks.workunit.client.0.smithi089.stdout: create 133/33 file after the snapshot 2023-10-30T12:30:58.627 INFO:tasks.workunit.client.0.smithi089.stdout: create 133/34 file after the snapshot 2023-10-30T12:30:58.627 INFO:tasks.workunit.client.0.smithi089.stdout: create 133/35 file after the snapshot 2023-10-30T12:30:58.627 INFO:tasks.workunit.client.0.smithi089.stdout: create 133/36 file after the snapshot 2023-10-30T12:30:58.627 INFO:tasks.workunit.client.0.smithi089.stdout: create 133/37 file after the snapshot 2023-10-30T12:30:58.628 INFO:tasks.workunit.client.0.smithi089.stdout: create 133/38 file after the snapshot 2023-10-30T12:30:58.628 INFO:tasks.workunit.client.0.smithi089.stdout: create 133/39 file after the snapshot 2023-10-30T12:30:58.628 INFO:tasks.workunit.client.0.smithi089.stdout: create 134/.snap/snap-subdir-test 2023-10-30T12:30:58.712 INFO:tasks.workunit.client.0.smithi089.stdout: create 134/30 file after the snapshot 2023-10-30T12:30:58.717 INFO:tasks.workunit.client.0.smithi089.stdout: create 134/31 file after the snapshot 2023-10-30T12:30:58.718 INFO:tasks.workunit.client.0.smithi089.stdout: create 134/32 file after the snapshot 2023-10-30T12:30:58.718 INFO:tasks.workunit.client.0.smithi089.stdout: create 134/33 file after the snapshot 2023-10-30T12:30:58.718 INFO:tasks.workunit.client.0.smithi089.stdout: create 134/34 file after the snapshot 2023-10-30T12:30:58.718 INFO:tasks.workunit.client.0.smithi089.stdout: create 134/35 file after the snapshot 2023-10-30T12:30:58.719 INFO:tasks.workunit.client.0.smithi089.stdout: create 134/36 file after the snapshot 2023-10-30T12:30:58.719 INFO:tasks.workunit.client.0.smithi089.stdout: create 134/37 file after the snapshot 2023-10-30T12:30:58.719 INFO:tasks.workunit.client.0.smithi089.stdout: create 134/38 file after the snapshot 2023-10-30T12:30:58.719 INFO:tasks.workunit.client.0.smithi089.stdout: create 134/39 file after the snapshot 2023-10-30T12:30:58.719 INFO:tasks.workunit.client.0.smithi089.stdout: create 135/.snap/snap-subdir-test 2023-10-30T12:30:58.776 INFO:tasks.workunit.client.0.smithi089.stdout: create 135/30 file after the snapshot 2023-10-30T12:30:58.779 INFO:tasks.workunit.client.0.smithi089.stdout: create 135/31 file after the snapshot 2023-10-30T12:30:58.779 INFO:tasks.workunit.client.0.smithi089.stdout: create 135/32 file after the snapshot 2023-10-30T12:30:58.779 INFO:tasks.workunit.client.0.smithi089.stdout: create 135/33 file after the snapshot 2023-10-30T12:30:58.780 INFO:tasks.workunit.client.0.smithi089.stdout: create 135/34 file after the snapshot 2023-10-30T12:30:58.780 INFO:tasks.workunit.client.0.smithi089.stdout: create 135/35 file after the snapshot 2023-10-30T12:30:58.780 INFO:tasks.workunit.client.0.smithi089.stdout: create 135/36 file after the snapshot 2023-10-30T12:30:58.780 INFO:tasks.workunit.client.0.smithi089.stdout: create 135/37 file after the snapshot 2023-10-30T12:30:58.780 INFO:tasks.workunit.client.0.smithi089.stdout: create 135/38 file after the snapshot 2023-10-30T12:30:58.780 INFO:tasks.workunit.client.0.smithi089.stdout: create 135/39 file after the snapshot 2023-10-30T12:30:58.781 INFO:tasks.workunit.client.0.smithi089.stdout: create 136/.snap/snap-subdir-test 2023-10-30T12:30:59.531 INFO:tasks.workunit.client.0.smithi089.stdout: create 136/30 file after the snapshot 2023-10-30T12:30:59.536 INFO:tasks.workunit.client.0.smithi089.stdout: create 136/31 file after the snapshot 2023-10-30T12:30:59.536 INFO:tasks.workunit.client.0.smithi089.stdout: create 136/32 file after the snapshot 2023-10-30T12:30:59.536 INFO:tasks.workunit.client.0.smithi089.stdout: create 136/33 file after the snapshot 2023-10-30T12:30:59.536 INFO:tasks.workunit.client.0.smithi089.stdout: create 136/34 file after the snapshot 2023-10-30T12:30:59.537 INFO:tasks.workunit.client.0.smithi089.stdout: create 136/35 file after the snapshot 2023-10-30T12:30:59.537 INFO:tasks.workunit.client.0.smithi089.stdout: create 136/36 file after the snapshot 2023-10-30T12:30:59.537 INFO:tasks.workunit.client.0.smithi089.stdout: create 136/37 file after the snapshot 2023-10-30T12:30:59.537 INFO:tasks.workunit.client.0.smithi089.stdout: create 136/38 file after the snapshot 2023-10-30T12:30:59.537 INFO:tasks.workunit.client.0.smithi089.stdout: create 136/39 file after the snapshot 2023-10-30T12:30:59.538 INFO:tasks.workunit.client.0.smithi089.stdout: create 137/.snap/snap-subdir-test 2023-10-30T12:30:59.595 INFO:tasks.workunit.client.0.smithi089.stdout: create 137/30 file after the snapshot 2023-10-30T12:30:59.598 INFO:tasks.workunit.client.0.smithi089.stdout: create 137/31 file after the snapshot 2023-10-30T12:30:59.598 INFO:tasks.workunit.client.0.smithi089.stdout: create 137/32 file after the snapshot 2023-10-30T12:30:59.599 INFO:tasks.workunit.client.0.smithi089.stdout: create 137/33 file after the snapshot 2023-10-30T12:30:59.599 INFO:tasks.workunit.client.0.smithi089.stdout: create 137/34 file after the snapshot 2023-10-30T12:30:59.599 INFO:tasks.workunit.client.0.smithi089.stdout: create 137/35 file after the snapshot 2023-10-30T12:30:59.599 INFO:tasks.workunit.client.0.smithi089.stdout: create 137/36 file after the snapshot 2023-10-30T12:30:59.599 INFO:tasks.workunit.client.0.smithi089.stdout: create 137/37 file after the snapshot 2023-10-30T12:30:59.600 INFO:tasks.workunit.client.0.smithi089.stdout: create 137/38 file after the snapshot 2023-10-30T12:30:59.600 INFO:tasks.workunit.client.0.smithi089.stdout: create 137/39 file after the snapshot 2023-10-30T12:30:59.600 INFO:tasks.workunit.client.0.smithi089.stdout: create 138/.snap/snap-subdir-test 2023-10-30T12:30:59.659 INFO:tasks.workunit.client.0.smithi089.stdout: create 138/30 file after the snapshot 2023-10-30T12:30:59.664 INFO:tasks.workunit.client.0.smithi089.stdout: create 138/31 file after the snapshot 2023-10-30T12:30:59.664 INFO:tasks.workunit.client.0.smithi089.stdout: create 138/32 file after the snapshot 2023-10-30T12:30:59.664 INFO:tasks.workunit.client.0.smithi089.stdout: create 138/33 file after the snapshot 2023-10-30T12:30:59.665 INFO:tasks.workunit.client.0.smithi089.stdout: create 138/34 file after the snapshot 2023-10-30T12:30:59.665 INFO:tasks.workunit.client.0.smithi089.stdout: create 138/35 file after the snapshot 2023-10-30T12:30:59.665 INFO:tasks.workunit.client.0.smithi089.stdout: create 138/36 file after the snapshot 2023-10-30T12:30:59.665 INFO:tasks.workunit.client.0.smithi089.stdout: create 138/37 file after the snapshot 2023-10-30T12:30:59.665 INFO:tasks.workunit.client.0.smithi089.stdout: create 138/38 file after the snapshot 2023-10-30T12:30:59.666 INFO:tasks.workunit.client.0.smithi089.stdout: create 138/39 file after the snapshot 2023-10-30T12:30:59.666 INFO:tasks.workunit.client.0.smithi089.stdout: create 139/.snap/snap-subdir-test 2023-10-30T12:30:59.724 INFO:tasks.workunit.client.0.smithi089.stdout: create 139/30 file after the snapshot 2023-10-30T12:30:59.727 INFO:tasks.workunit.client.0.smithi089.stdout: create 139/31 file after the snapshot 2023-10-30T12:30:59.727 INFO:tasks.workunit.client.0.smithi089.stdout: create 139/32 file after the snapshot 2023-10-30T12:30:59.728 INFO:tasks.workunit.client.0.smithi089.stdout: create 139/33 file after the snapshot 2023-10-30T12:30:59.728 INFO:tasks.workunit.client.0.smithi089.stdout: create 139/34 file after the snapshot 2023-10-30T12:30:59.728 INFO:tasks.workunit.client.0.smithi089.stdout: create 139/35 file after the snapshot 2023-10-30T12:30:59.728 INFO:tasks.workunit.client.0.smithi089.stdout: create 139/36 file after the snapshot 2023-10-30T12:30:59.729 INFO:tasks.workunit.client.0.smithi089.stdout: create 139/37 file after the snapshot 2023-10-30T12:30:59.729 INFO:tasks.workunit.client.0.smithi089.stdout: create 139/38 file after the snapshot 2023-10-30T12:30:59.729 INFO:tasks.workunit.client.0.smithi089.stdout: create 139/39 file after the snapshot 2023-10-30T12:30:59.729 INFO:tasks.workunit.client.0.smithi089.stdout: create 140/.snap/snap-subdir-test 2023-10-30T12:30:59.785 INFO:tasks.workunit.client.0.smithi089.stdout: create 140/30 file after the snapshot 2023-10-30T12:30:59.789 INFO:tasks.workunit.client.0.smithi089.stdout: create 140/31 file after the snapshot 2023-10-30T12:30:59.789 INFO:tasks.workunit.client.0.smithi089.stdout: create 140/32 file after the snapshot 2023-10-30T12:30:59.790 INFO:tasks.workunit.client.0.smithi089.stdout: create 140/33 file after the snapshot 2023-10-30T12:30:59.790 INFO:tasks.workunit.client.0.smithi089.stdout: create 140/34 file after the snapshot 2023-10-30T12:30:59.790 INFO:tasks.workunit.client.0.smithi089.stdout: create 140/35 file after the snapshot 2023-10-30T12:30:59.790 INFO:tasks.workunit.client.0.smithi089.stdout: create 140/36 file after the snapshot 2023-10-30T12:30:59.790 INFO:tasks.workunit.client.0.smithi089.stdout: create 140/37 file after the snapshot 2023-10-30T12:30:59.791 INFO:tasks.workunit.client.0.smithi089.stdout: create 140/38 file after the snapshot 2023-10-30T12:30:59.791 INFO:tasks.workunit.client.0.smithi089.stdout: create 140/39 file after the snapshot 2023-10-30T12:30:59.791 INFO:tasks.workunit.client.0.smithi089.stdout: create 141/.snap/snap-subdir-test 2023-10-30T12:31:00.101 INFO:tasks.workunit.client.0.smithi089.stdout: create 141/30 file after the snapshot 2023-10-30T12:31:00.358 INFO:tasks.workunit.client.0.smithi089.stdout: create 141/31 file after the snapshot 2023-10-30T12:31:00.359 INFO:tasks.workunit.client.0.smithi089.stdout: create 141/32 file after the snapshot 2023-10-30T12:31:00.359 INFO:tasks.workunit.client.0.smithi089.stdout: create 141/33 file after the snapshot 2023-10-30T12:31:00.359 INFO:tasks.workunit.client.0.smithi089.stdout: create 141/34 file after the snapshot 2023-10-30T12:31:00.359 INFO:tasks.workunit.client.0.smithi089.stdout: create 141/35 file after the snapshot 2023-10-30T12:31:00.359 INFO:tasks.workunit.client.0.smithi089.stdout: create 141/36 file after the snapshot 2023-10-30T12:31:00.360 INFO:tasks.workunit.client.0.smithi089.stdout: create 141/37 file after the snapshot 2023-10-30T12:31:00.360 INFO:tasks.workunit.client.0.smithi089.stdout: create 141/38 file after the snapshot 2023-10-30T12:31:00.360 INFO:tasks.workunit.client.0.smithi089.stdout: create 141/39 file after the snapshot 2023-10-30T12:31:00.360 INFO:tasks.workunit.client.0.smithi089.stdout: create 142/.snap/snap-subdir-test 2023-10-30T12:31:00.652 INFO:tasks.workunit.client.0.smithi089.stdout: create 142/30 file after the snapshot 2023-10-30T12:31:00.661 INFO:tasks.workunit.client.0.smithi089.stdout: create 142/31 file after the snapshot 2023-10-30T12:31:00.661 INFO:tasks.workunit.client.0.smithi089.stdout: create 142/32 file after the snapshot 2023-10-30T12:31:00.661 INFO:tasks.workunit.client.0.smithi089.stdout: create 142/33 file after the snapshot 2023-10-30T12:31:00.661 INFO:tasks.workunit.client.0.smithi089.stdout: create 142/34 file after the snapshot 2023-10-30T12:31:00.661 INFO:tasks.workunit.client.0.smithi089.stdout: create 142/35 file after the snapshot 2023-10-30T12:31:00.662 INFO:tasks.workunit.client.0.smithi089.stdout: create 142/36 file after the snapshot 2023-10-30T12:31:00.662 INFO:tasks.workunit.client.0.smithi089.stdout: create 142/37 file after the snapshot 2023-10-30T12:31:00.662 INFO:tasks.workunit.client.0.smithi089.stdout: create 142/38 file after the snapshot 2023-10-30T12:31:00.662 INFO:tasks.workunit.client.0.smithi089.stdout: create 142/39 file after the snapshot 2023-10-30T12:31:00.662 INFO:tasks.workunit.client.0.smithi089.stdout: create 143/.snap/snap-subdir-test 2023-10-30T12:31:00.736 INFO:tasks.workunit.client.0.smithi089.stdout: create 143/30 file after the snapshot 2023-10-30T12:31:00.739 INFO:tasks.workunit.client.0.smithi089.stdout: create 143/31 file after the snapshot 2023-10-30T12:31:00.739 INFO:tasks.workunit.client.0.smithi089.stdout: create 143/32 file after the snapshot 2023-10-30T12:31:00.739 INFO:tasks.workunit.client.0.smithi089.stdout: create 143/33 file after the snapshot 2023-10-30T12:31:00.740 INFO:tasks.workunit.client.0.smithi089.stdout: create 143/34 file after the snapshot 2023-10-30T12:31:00.740 INFO:tasks.workunit.client.0.smithi089.stdout: create 143/35 file after the snapshot 2023-10-30T12:31:00.740 INFO:tasks.workunit.client.0.smithi089.stdout: create 143/36 file after the snapshot 2023-10-30T12:31:00.740 INFO:tasks.workunit.client.0.smithi089.stdout: create 143/37 file after the snapshot 2023-10-30T12:31:00.740 INFO:tasks.workunit.client.0.smithi089.stdout: create 143/38 file after the snapshot 2023-10-30T12:31:00.741 INFO:tasks.workunit.client.0.smithi089.stdout: create 143/39 file after the snapshot 2023-10-30T12:31:00.741 INFO:tasks.workunit.client.0.smithi089.stdout: create 144/.snap/snap-subdir-test 2023-10-30T12:31:01.944 INFO:tasks.workunit.client.0.smithi089.stdout: create 144/30 file after the snapshot 2023-10-30T12:31:01.949 INFO:tasks.workunit.client.0.smithi089.stdout: create 144/31 file after the snapshot 2023-10-30T12:31:01.951 INFO:tasks.workunit.client.0.smithi089.stdout: create 144/32 file after the snapshot 2023-10-30T12:31:01.952 INFO:tasks.workunit.client.0.smithi089.stdout: create 144/33 file after the snapshot 2023-10-30T12:31:01.952 INFO:tasks.workunit.client.0.smithi089.stdout: create 144/34 file after the snapshot 2023-10-30T12:31:01.952 INFO:tasks.workunit.client.0.smithi089.stdout: create 144/35 file after the snapshot 2023-10-30T12:31:01.952 INFO:tasks.workunit.client.0.smithi089.stdout: create 144/36 file after the snapshot 2023-10-30T12:31:01.952 INFO:tasks.workunit.client.0.smithi089.stdout: create 144/37 file after the snapshot 2023-10-30T12:31:01.953 INFO:tasks.workunit.client.0.smithi089.stdout: create 144/38 file after the snapshot 2023-10-30T12:31:01.953 INFO:tasks.workunit.client.0.smithi089.stdout: create 144/39 file after the snapshot 2023-10-30T12:31:01.953 INFO:tasks.workunit.client.0.smithi089.stdout: create 145/.snap/snap-subdir-test 2023-10-30T12:31:02.668 INFO:tasks.workunit.client.0.smithi089.stdout: create 145/30 file after the snapshot 2023-10-30T12:31:03.209 INFO:tasks.workunit.client.0.smithi089.stdout: create 145/31 file after the snapshot 2023-10-30T12:31:03.210 INFO:tasks.workunit.client.0.smithi089.stdout: create 145/32 file after the snapshot 2023-10-30T12:31:03.210 INFO:tasks.workunit.client.0.smithi089.stdout: create 145/33 file after the snapshot 2023-10-30T12:31:03.210 INFO:tasks.workunit.client.0.smithi089.stdout: create 145/34 file after the snapshot 2023-10-30T12:31:03.210 INFO:tasks.workunit.client.0.smithi089.stdout: create 145/35 file after the snapshot 2023-10-30T12:31:03.210 INFO:tasks.workunit.client.0.smithi089.stdout: create 145/36 file after the snapshot 2023-10-30T12:31:03.211 INFO:tasks.workunit.client.0.smithi089.stdout: create 145/37 file after the snapshot 2023-10-30T12:31:03.211 INFO:tasks.workunit.client.0.smithi089.stdout: create 145/38 file after the snapshot 2023-10-30T12:31:03.211 INFO:tasks.workunit.client.0.smithi089.stdout: create 145/39 file after the snapshot 2023-10-30T12:31:03.212 INFO:tasks.workunit.client.0.smithi089.stdout: create 146/.snap/snap-subdir-test 2023-10-30T12:31:03.262 INFO:tasks.workunit.client.0.smithi089.stdout: create 146/30 file after the snapshot 2023-10-30T12:31:03.265 INFO:tasks.workunit.client.0.smithi089.stdout: create 146/31 file after the snapshot 2023-10-30T12:31:03.265 INFO:tasks.workunit.client.0.smithi089.stdout: create 146/32 file after the snapshot 2023-10-30T12:31:03.265 INFO:tasks.workunit.client.0.smithi089.stdout: create 146/33 file after the snapshot 2023-10-30T12:31:03.265 INFO:tasks.workunit.client.0.smithi089.stdout: create 146/34 file after the snapshot 2023-10-30T12:31:03.266 INFO:tasks.workunit.client.0.smithi089.stdout: create 146/35 file after the snapshot 2023-10-30T12:31:03.267 INFO:tasks.workunit.client.0.smithi089.stdout: create 146/36 file after the snapshot 2023-10-30T12:31:03.267 INFO:tasks.workunit.client.0.smithi089.stdout: create 146/37 file after the snapshot 2023-10-30T12:31:03.267 INFO:tasks.workunit.client.0.smithi089.stdout: create 146/38 file after the snapshot 2023-10-30T12:31:03.267 INFO:tasks.workunit.client.0.smithi089.stdout: create 146/39 file after the snapshot 2023-10-30T12:31:03.267 INFO:tasks.workunit.client.0.smithi089.stdout: create 147/.snap/snap-subdir-test 2023-10-30T12:31:03.303 INFO:tasks.workunit.client.0.smithi089.stdout: create 147/30 file after the snapshot 2023-10-30T12:31:03.307 INFO:tasks.workunit.client.0.smithi089.stdout: create 147/31 file after the snapshot 2023-10-30T12:31:03.307 INFO:tasks.workunit.client.0.smithi089.stdout: create 147/32 file after the snapshot 2023-10-30T12:31:03.308 INFO:tasks.workunit.client.0.smithi089.stdout: create 147/33 file after the snapshot 2023-10-30T12:31:03.308 INFO:tasks.workunit.client.0.smithi089.stdout: create 147/34 file after the snapshot 2023-10-30T12:31:03.308 INFO:tasks.workunit.client.0.smithi089.stdout: create 147/35 file after the snapshot 2023-10-30T12:31:03.308 INFO:tasks.workunit.client.0.smithi089.stdout: create 147/36 file after the snapshot 2023-10-30T12:31:03.308 INFO:tasks.workunit.client.0.smithi089.stdout: create 147/37 file after the snapshot 2023-10-30T12:31:03.309 INFO:tasks.workunit.client.0.smithi089.stdout: create 147/38 file after the snapshot 2023-10-30T12:31:03.309 INFO:tasks.workunit.client.0.smithi089.stdout: create 147/39 file after the snapshot 2023-10-30T12:31:03.309 INFO:tasks.workunit.client.0.smithi089.stdout: create 148/.snap/snap-subdir-test 2023-10-30T12:31:03.364 INFO:tasks.workunit.client.0.smithi089.stdout: create 148/30 file after the snapshot 2023-10-30T12:31:03.373 INFO:tasks.workunit.client.0.smithi089.stdout: create 148/31 file after the snapshot 2023-10-30T12:31:03.375 INFO:tasks.workunit.client.0.smithi089.stdout: create 148/32 file after the snapshot 2023-10-30T12:31:03.377 INFO:tasks.workunit.client.0.smithi089.stdout: create 148/33 file after the snapshot 2023-10-30T12:31:03.379 INFO:tasks.workunit.client.0.smithi089.stdout: create 148/34 file after the snapshot 2023-10-30T12:31:03.381 INFO:tasks.workunit.client.0.smithi089.stdout: create 148/35 file after the snapshot 2023-10-30T12:31:03.383 INFO:tasks.workunit.client.0.smithi089.stdout: create 148/36 file after the snapshot 2023-10-30T12:31:03.385 INFO:tasks.workunit.client.0.smithi089.stdout: create 148/37 file after the snapshot 2023-10-30T12:31:03.387 INFO:tasks.workunit.client.0.smithi089.stdout: create 148/38 file after the snapshot 2023-10-30T12:31:03.389 INFO:tasks.workunit.client.0.smithi089.stdout: create 148/39 file after the snapshot 2023-10-30T12:31:03.391 INFO:tasks.workunit.client.0.smithi089.stdout: create 149/.snap/snap-subdir-test 2023-10-30T12:31:03.580 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:31:03.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.721+0000 7ff602c46640 1 Processor -- start 2023-10-30T12:31:03.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.721+0000 7ff602c46640 1 -- start start 2023-10-30T12:31:03.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.721+0000 7ff602c46640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff5fc1538a0 0x7ff5fc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:03.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.721+0000 7ff602c46640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff5fc0cf160 0x7ff5fc0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:03.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.721+0000 7ff602c46640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5fc0cb800 con 0x7ff5fc0ca0e0 2023-10-30T12:31:03.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.721+0000 7ff602c46640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5fc15dc30 con 0x7ff5fc0cf160 2023-10-30T12:31:03.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.721+0000 7ff602c46640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5fc0cb330 con 0x7ff5fc1538a0 2023-10-30T12:31:03.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.721+0000 7ff6011e3640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff5fc0cf160 0x7ff5fc0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:03.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.721+0000 7ff6011e3640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff5fc0cf160 0x7ff5fc0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41406/0 (socket says 172.21.15.89:41406) 2023-10-30T12:31:03.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.721+0000 7ff6011e3640 1 -- 172.21.15.89:0/285483416 learned_addr learned my addr 172.21.15.89:0/285483416 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:03.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.721+0000 7ff6009e2640 1 --2- 172.21.15.89:0/285483416 >> v2:172.21.15.192:3300/0 conn(0x7ff5fc1538a0 0x7ff5fc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:03.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.721+0000 7ff6011e3640 1 -- 172.21.15.89:0/285483416 >> v2:172.21.15.192:3300/0 conn(0x7ff5fc1538a0 msgr2=0x7ff5fc15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:03.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.721+0000 7ff6011e3640 1 --2- 172.21.15.89:0/285483416 >> v2:172.21.15.192:3300/0 conn(0x7ff5fc1538a0 0x7ff5fc15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:03.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.721+0000 7ff6011e3640 1 -- 172.21.15.89:0/285483416 >> v1:172.21.15.192:6789/0 conn(0x7ff5fc0ca0e0 legacy=0x7ff5fc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:03.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.721+0000 7ff6011e3640 1 -- 172.21.15.89:0/285483416 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5fc0cad70 con 0x7ff5fc0cf160 2023-10-30T12:31:03.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.725+0000 7ff6011e3640 1 --2- 172.21.15.89:0/285483416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5fc0cf160 0x7ff5fc0cf590 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc0cfd10 tx=0x7ff5f401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c5404ab2eae8161 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:03.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.725+0000 7ff5fb7fe640 1 -- 172.21.15.89:0/285483416 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f40af400 con 0x7ff5fc0cf160 2023-10-30T12:31:03.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.725+0000 7ff5fb7fe640 1 -- 172.21.15.89:0/285483416 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f40ab050 con 0x7ff5fc0cf160 2023-10-30T12:31:03.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.725+0000 7ff5fb7fe640 1 -- 172.21.15.89:0/285483416 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f40af780 con 0x7ff5fc0cf160 2023-10-30T12:31:03.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.725+0000 7ff602c46640 1 -- 172.21.15.89:0/285483416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5fc0cf160 msgr2=0x7ff5fc0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:03.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.725+0000 7ff602c46640 1 --2- 172.21.15.89:0/285483416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5fc0cf160 0x7ff5fc0cf590 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc0cfd10 tx=0x7ff5f401f1c0 comp rx=0 tx=0).stop 2023-10-30T12:31:03.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.725+0000 7ff602c46640 1 -- 172.21.15.89:0/285483416 shutdown_connections 2023-10-30T12:31:03.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.725+0000 7ff602c46640 1 --2- 172.21.15.89:0/285483416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5fc0cf160 0x7ff5fc0cf590 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:03.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.725+0000 7ff602c46640 1 --2- 172.21.15.89:0/285483416 >> v2:172.21.15.192:3300/0 conn(0x7ff5fc1538a0 0x7ff5fc15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:03.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.725+0000 7ff602c46640 1 -- 172.21.15.89:0/285483416 >> 172.21.15.89:0/285483416 conn(0x7ff5fc06bfe0 msgr2=0x7ff5fc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:03.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.725+0000 7ff602c46640 1 -- 172.21.15.89:0/285483416 shutdown_connections 2023-10-30T12:31:03.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.725+0000 7ff602c46640 1 -- 172.21.15.89:0/285483416 wait complete. 2023-10-30T12:31:03.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.729+0000 7ff602c46640 1 Processor -- start 2023-10-30T12:31:03.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.729+0000 7ff602c46640 1 -- start start 2023-10-30T12:31:03.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.729+0000 7ff602c46640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5fc0ca0e0 0x7ff5fc143fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:03.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.729+0000 7ff602c46640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5fc0cf160 0x7ff5fc1f32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:03.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.729+0000 7ff602c46640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5fc1538a0 0x7ff5fc0c4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:03.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.729+0000 7ff602c46640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc144510 con 0x7ff5fc0ca0e0 2023-10-30T12:31:03.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.729+0000 7ff602c46640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc1f3800 con 0x7ff5fc0cf160 2023-10-30T12:31:03.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.729+0000 7ff602c46640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc0b34d0 con 0x7ff5fc1538a0 2023-10-30T12:31:03.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.729+0000 7ff6009e2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5fc0ca0e0 0x7ff5fc143fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:03.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.729+0000 7ff6009e2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5fc0ca0e0 0x7ff5fc143fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41420/0 (socket says 172.21.15.89:41420) 2023-10-30T12:31:03.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.729+0000 7ff6009e2640 1 -- 172.21.15.89:0/4173280100 learned_addr learned my addr 172.21.15.89:0/4173280100 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:03.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.729+0000 7ff5fbfff640 1 --2- 172.21.15.89:0/4173280100 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5fc0cf160 0x7ff5fc1f32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:03.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.729+0000 7ff6011e3640 1 --2- 172.21.15.89:0/4173280100 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5fc1538a0 0x7ff5fc0c4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:03.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.729+0000 7ff6009e2640 1 -- 172.21.15.89:0/4173280100 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5fc1538a0 msgr2=0x7ff5fc0c4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:03.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.729+0000 7ff6009e2640 1 --2- 172.21.15.89:0/4173280100 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5fc1538a0 0x7ff5fc0c4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:03.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.729+0000 7ff6009e2640 1 -- 172.21.15.89:0/4173280100 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5fc0cf160 msgr2=0x7ff5fc1f32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:03.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.729+0000 7ff6009e2640 1 --2- 172.21.15.89:0/4173280100 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5fc0cf160 0x7ff5fc1f32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:03.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.729+0000 7ff6009e2640 1 -- 172.21.15.89:0/4173280100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f4009b30 con 0x7ff5fc0ca0e0 2023-10-30T12:31:03.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.729+0000 7ff5fbfff640 1 --2- 172.21.15.89:0/4173280100 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5fc0cf160 0x7ff5fc1f32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:31:03.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.729+0000 7ff6009e2640 1 --2- 172.21.15.89:0/4173280100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5fc0ca0e0 0x7ff5fc143fd0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff5f000d800 tx=0x7ff5f000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:03.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.729+0000 7ff5f97fa640 1 -- 172.21.15.89:0/4173280100 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f000b660 con 0x7ff5fc0ca0e0 2023-10-30T12:31:03.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.729+0000 7ff5f97fa640 1 -- 172.21.15.89:0/4173280100 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f000b7c0 con 0x7ff5fc0ca0e0 2023-10-30T12:31:03.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.729+0000 7ff602c46640 1 -- 172.21.15.89:0/4173280100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5fc1ef340 con 0x7ff5fc0ca0e0 2023-10-30T12:31:03.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.729+0000 7ff5f97fa640 1 -- 172.21.15.89:0/4173280100 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f000bb30 con 0x7ff5fc0ca0e0 2023-10-30T12:31:03.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.729+0000 7ff602c46640 1 -- 172.21.15.89:0/4173280100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5fc1ef8c0 con 0x7ff5fc0ca0e0 2023-10-30T12:31:03.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.733+0000 7ff602c46640 1 -- 172.21.15.89:0/4173280100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5fc0b2a50 con 0x7ff5fc0ca0e0 2023-10-30T12:31:03.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.733+0000 7ff5f97fa640 1 -- 172.21.15.89:0/4173280100 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff5f000bc90 con 0x7ff5fc0ca0e0 2023-10-30T12:31:03.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.733+0000 7ff5f97fa640 1 --2- 172.21.15.89:0/4173280100 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff5e00664a0 0x7ff5e0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:03.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.733+0000 7ff5f97fa640 1 -- 172.21.15.89:0/4173280100 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5832+0+0 (secure 0 0 0) 0x7ff5f0084530 con 0x7ff5fc0ca0e0 2023-10-30T12:31:03.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.733+0000 7ff5fbfff640 1 --2- 172.21.15.89:0/4173280100 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff5e00664a0 0x7ff5e0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:03.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.733+0000 7ff5fbfff640 1 --2- 172.21.15.89:0/4173280100 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff5e00664a0 0x7ff5e0068960 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc15dc00 tx=0x7ff5e8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:03.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.737+0000 7ff5f97fa640 1 -- 172.21.15.89:0/4173280100 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff5f0051800 con 0x7ff5fc0ca0e0 2023-10-30T12:31:03.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.917+0000 7ff602c46640 1 -- 172.21.15.89:0/4173280100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5fc04e170 con 0x7ff5fc0ca0e0 2023-10-30T12:31:03.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.917+0000 7ff5f97fa640 1 -- 172.21.15.89:0/4173280100 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+6531 (secure 0 0 0) 0x7ff5f0054e20 con 0x7ff5fc0ca0e0 2023-10-30T12:31:03.924 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:31:03.924 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:30:56.409108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4253":{"gid":4253,"name":"b","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6835/1289220491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1289220491},{"type":"v1","addr":"172.21.15.192:6835","nonce":1289220491}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:31:03.924 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 14 2023-10-30T12:31:03.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.921+0000 7ff602c46640 1 -- 172.21.15.89:0/4173280100 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff5e00664a0 msgr2=0x7ff5e0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:03.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.921+0000 7ff602c46640 1 --2- 172.21.15.89:0/4173280100 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff5e00664a0 0x7ff5e0068960 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc15dc00 tx=0x7ff5e8009040 comp rx=0 tx=0).stop 2023-10-30T12:31:03.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.921+0000 7ff602c46640 1 -- 172.21.15.89:0/4173280100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5fc0ca0e0 msgr2=0x7ff5fc143fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:03.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.921+0000 7ff602c46640 1 --2- 172.21.15.89:0/4173280100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5fc0ca0e0 0x7ff5fc143fd0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff5f000d800 tx=0x7ff5f000dcc0 comp rx=0 tx=0).stop 2023-10-30T12:31:03.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.921+0000 7ff602c46640 1 -- 172.21.15.89:0/4173280100 shutdown_connections 2023-10-30T12:31:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.921+0000 7ff602c46640 1 --2- 172.21.15.89:0/4173280100 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff5e00664a0 0x7ff5e0068960 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.921+0000 7ff602c46640 1 --2- 172.21.15.89:0/4173280100 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5fc0cf160 0x7ff5fc1f32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.921+0000 7ff602c46640 1 --2- 172.21.15.89:0/4173280100 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5fc1538a0 0x7ff5fc0c4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.921+0000 7ff602c46640 1 --2- 172.21.15.89:0/4173280100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5fc0ca0e0 0x7ff5fc143fd0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.921+0000 7ff602c46640 1 -- 172.21.15.89:0/4173280100 >> 172.21.15.89:0/4173280100 conn(0x7ff5fc06bfe0 msgr2=0x7ff5fc1513b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:03.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.921+0000 7ff602c46640 1 -- 172.21.15.89:0/4173280100 shutdown_connections 2023-10-30T12:31:03.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:03.921+0000 7ff602c46640 1 -- 172.21.15.89:0/4173280100 wait complete. 2023-10-30T12:31:03.944 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=2) 2023-10-30T12:31:03.945 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-30T12:31:03.945 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:31:03.964 INFO:tasks.ceph.mds.b:Stopped 2023-10-30T12:31:03.964 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:31:04.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.097+0000 7efebd29e640 1 Processor -- start 2023-10-30T12:31:04.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.097+0000 7efebd29e640 1 -- start start 2023-10-30T12:31:04.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efebd29e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7efeb80c6bd0 0x7efeb806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:04.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efebd29e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efeb8071110 0x7efeb8158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:04.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efebd29e640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb8066cf0 con 0x7efeb806f7a0 2023-10-30T12:31:04.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efebd29e640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeb80700c0 con 0x7efeb80c6bd0 2023-10-30T12:31:04.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efebd29e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeb80c70d0 con 0x7efeb8071110 2023-10-30T12:31:04.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efeb6ffd640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7efeb806f7a0 0x7efeb8069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43358/0 (socket says 172.21.15.89:43358) 2023-10-30T12:31:04.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efeb6ffd640 1 -- 172.21.15.89:0/4246745611 learned_addr learned my addr 172.21.15.89:0/4246745611 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:31:04.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efeb77fe640 1 --2- 172.21.15.89:0/4246745611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efeb8071110 0x7efeb8158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:04.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efeae7fc640 1 --2- 172.21.15.89:0/4246745611 >> v2:172.21.15.192:3300/0 conn(0x7efeb80c6bd0 0x7efeb806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:04.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efeb67fc640 1 -- 172.21.15.89:0/4246745611 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3541124480 0 0) 0x7efeb8066cf0 con 0x7efeb806f7a0 2023-10-30T12:31:04.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efeb67fc640 1 -- 172.21.15.89:0/4246745611 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efea4003530 con 0x7efeb806f7a0 2023-10-30T12:31:04.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efeb67fc640 1 -- 172.21.15.89:0/4246745611 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2474472257 0 0) 0x7efea4003530 con 0x7efeb806f7a0 2023-10-30T12:31:04.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efeb67fc640 1 -- 172.21.15.89:0/4246745611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efeb8071110 msgr2=0x7efeb8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:04.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efeb67fc640 1 --2- 172.21.15.89:0/4246745611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efeb8071110 0x7efeb8158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:04.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efeb67fc640 1 -- 172.21.15.89:0/4246745611 >> v2:172.21.15.192:3300/0 conn(0x7efeb80c6bd0 msgr2=0x7efeb806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:04.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efeb67fc640 1 --2- 172.21.15.89:0/4246745611 >> v2:172.21.15.192:3300/0 conn(0x7efeb80c6bd0 0x7efeb806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:04.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efeb67fc640 1 -- 172.21.15.89:0/4246745611 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb80c7390 con 0x7efeb806f7a0 2023-10-30T12:31:04.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efeb67fc640 1 -- 172.21.15.89:0/4246745611 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7efea8001680 con 0x7efeb806f7a0 2023-10-30T12:31:04.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efeb67fc640 1 -- 172.21.15.89:0/4246745611 >> v1:172.21.15.89:6789/0 conn(0x7efeb806f7a0 legacy=0x7efeb8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:04.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efeb67fc640 1 --2- 172.21.15.89:0/4246745611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efea4003980 0x7efea4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:04.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efeb67fc640 1 --2- 172.21.15.89:0/4246745611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efeb8071110 0x7efea40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:04.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efeb67fc640 1 --2- 172.21.15.89:0/4246745611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efea4004d10 0x7efea4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:04.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efeb67fc640 1 -- 172.21.15.89:0/4246745611 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb80700c0 con 0x7efea4003980 2023-10-30T12:31:04.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efeb67fc640 1 -- 172.21.15.89:0/4246745611 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb80c70d0 con 0x7efea4004d10 2023-10-30T12:31:04.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efeb67fc640 1 -- 172.21.15.89:0/4246745611 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efea400b990 con 0x7efeb8071110 2023-10-30T12:31:04.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efeb6ffd640 1 --2- 172.21.15.89:0/4246745611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efea4003980 0x7efea4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:04.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efeae7fc640 1 --2- 172.21.15.89:0/4246745611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efeb8071110 0x7efea40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:04.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efeb77fe640 1 --2- 172.21.15.89:0/4246745611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efea4004d10 0x7efea4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:04.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efeb6ffd640 1 -- 172.21.15.89:0/4246745611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efeb8071110 msgr2=0x7efea40047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:04.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efeb6ffd640 1 --2- 172.21.15.89:0/4246745611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efeb8071110 0x7efea40047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:04.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efeb6ffd640 1 -- 172.21.15.89:0/4246745611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efea4004d10 msgr2=0x7efea4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:04.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efeb6ffd640 1 --2- 172.21.15.89:0/4246745611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efea4004d10 0x7efea4005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:04.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.101+0000 7efeb6ffd640 1 -- 172.21.15.89:0/4246745611 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea4003530 con 0x7efea4003980 2023-10-30T12:31:04.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.105+0000 7efeb6ffd640 1 --2- 172.21.15.89:0/4246745611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efea4003980 0x7efea4003da0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7efea800c240 tx=0x7efea801f3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b072e997cec3101f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:04.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.105+0000 7efeb67fc640 1 -- 172.21.15.89:0/4246745611 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea8005ae0 con 0x7efea4003980 2023-10-30T12:31:04.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.105+0000 7efeb67fc640 1 -- 172.21.15.89:0/4246745611 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efea80ab070 con 0x7efea4003980 2023-10-30T12:31:04.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.105+0000 7efeb67fc640 1 -- 172.21.15.89:0/4246745611 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea8005ea0 con 0x7efea4003980 2023-10-30T12:31:04.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.105+0000 7efebd29e640 1 -- 172.21.15.89:0/4246745611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efea4003980 msgr2=0x7efea4003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:04.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.105+0000 7efebd29e640 1 --2- 172.21.15.89:0/4246745611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efea4003980 0x7efea4003da0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7efea800c240 tx=0x7efea801f3d0 comp rx=0 tx=0).stop 2023-10-30T12:31:04.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.105+0000 7efeb77fe640 1 -- 172.21.15.89:0/4246745611 reap_dead start 2023-10-30T12:31:04.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.105+0000 7efebd29e640 1 -- 172.21.15.89:0/4246745611 shutdown_connections 2023-10-30T12:31:04.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.105+0000 7efebd29e640 1 -- 172.21.15.89:0/4246745611 >> 172.21.15.89:0/4246745611 conn(0x7efeb80c7910 msgr2=0x7efeb80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:04.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.105+0000 7efebd29e640 1 -- 172.21.15.89:0/4246745611 shutdown_connections 2023-10-30T12:31:04.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.105+0000 7efebd29e640 1 -- 172.21.15.89:0/4246745611 wait complete. 2023-10-30T12:31:04.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.105+0000 7efebd29e640 1 Processor -- start 2023-10-30T12:31:04.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.109+0000 7efebd29e640 1 -- start start 2023-10-30T12:31:04.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.109+0000 7efebd29e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efea4003980 0x7efeb8155e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:04.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.109+0000 7efebd29e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efeb81577f0 0x7efeb8156360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:04.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.109+0000 7efebd29e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efeb81538c0 0x7efeb80c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:04.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.109+0000 7efebd29e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb8157de0 con 0x7efeb81538c0 2023-10-30T12:31:04.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.109+0000 7efebd29e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb81f27a0 con 0x7efeb81577f0 2023-10-30T12:31:04.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.109+0000 7efebd29e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeb81f2a80 con 0x7efea4003980 2023-10-30T12:31:04.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.109+0000 7efeb77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efeb81538c0 0x7efeb80c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:04.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.109+0000 7efeb77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efeb81538c0 0x7efeb80c9ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41446/0 (socket says 172.21.15.89:41446) 2023-10-30T12:31:04.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.109+0000 7efeb77fe640 1 -- 172.21.15.89:0/3415171354 learned_addr learned my addr 172.21.15.89:0/3415171354 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:04.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.109+0000 7efeb6ffd640 1 --2- 172.21.15.89:0/3415171354 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efea4003980 0x7efeb8155e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:04.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.109+0000 7efeae7fc640 1 --2- 172.21.15.89:0/3415171354 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efeb81577f0 0x7efeb8156360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:04.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.109+0000 7efeb77fe640 1 -- 172.21.15.89:0/3415171354 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efea4003980 msgr2=0x7efeb8155e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:04.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.109+0000 7efeb77fe640 1 --2- 172.21.15.89:0/3415171354 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efea4003980 0x7efeb8155e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:04.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.109+0000 7efeb77fe640 1 -- 172.21.15.89:0/3415171354 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efeb81577f0 msgr2=0x7efeb8156360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:04.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.109+0000 7efeb77fe640 1 --2- 172.21.15.89:0/3415171354 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efeb81577f0 0x7efeb8156360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:04.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.109+0000 7efeb77fe640 1 -- 172.21.15.89:0/3415171354 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb8158290 con 0x7efeb81538c0 2023-10-30T12:31:04.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.109+0000 7efeb77fe640 1 --2- 172.21.15.89:0/3415171354 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efeb81538c0 0x7efeb80c9ec0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7efeb80c6fb0 tx=0x7efeb00115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:04.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.109+0000 7efeaffff640 1 -- 172.21.15.89:0/3415171354 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb000e070 con 0x7efeb81538c0 2023-10-30T12:31:04.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.109+0000 7efeaffff640 1 -- 172.21.15.89:0/3415171354 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb0011e10 con 0x7efeb81538c0 2023-10-30T12:31:04.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.109+0000 7efeaffff640 1 -- 172.21.15.89:0/3415171354 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb00039a0 con 0x7efeb81538c0 2023-10-30T12:31:04.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.109+0000 7efebd29e640 1 -- 172.21.15.89:0/3415171354 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb8158570 con 0x7efeb81538c0 2023-10-30T12:31:04.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.109+0000 7efebd29e640 1 -- 172.21.15.89:0/3415171354 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb8153290 con 0x7efeb81538c0 2023-10-30T12:31:04.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.109+0000 7efebd29e640 1 -- 172.21.15.89:0/3415171354 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeb81580f0 con 0x7efeb81538c0 2023-10-30T12:31:04.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.113+0000 7efeaffff640 1 -- 172.21.15.89:0/3415171354 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efeb0003b00 con 0x7efeb81538c0 2023-10-30T12:31:04.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.113+0000 7efeaffff640 1 --2- 172.21.15.89:0/3415171354 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7efe980664a0 0x7efe98068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:04.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.113+0000 7efeaffff640 1 -- 172.21.15.89:0/3415171354 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5832+0+0 (secure 0 0 0) 0x7efeb00845c0 con 0x7efeb81538c0 2023-10-30T12:31:04.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.113+0000 7efeb6ffd640 1 --2- 172.21.15.89:0/3415171354 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7efe980664a0 0x7efe98068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:04.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.113+0000 7efeb6ffd640 1 --2- 172.21.15.89:0/3415171354 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7efe980664a0 0x7efe98068960 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7efea80ae000 tx=0x7efea8001a80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:04.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.117+0000 7efeaffff640 1 -- 172.21.15.89:0/3415171354 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efeb81580f0 con 0x7efeb81538c0 2023-10-30T12:31:04.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.297+0000 7efebd29e640 1 -- 172.21.15.89:0/3415171354 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efeb81580f0 con 0x7efeb81538c0 2023-10-30T12:31:04.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.301+0000 7efeaffff640 1 -- 172.21.15.89:0/3415171354 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+6531 (secure 0 0 0) 0x7efeb81580f0 con 0x7efeb81538c0 2023-10-30T12:31:04.304 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:31:04.305 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:30:56.409108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4253":{"gid":4253,"name":"b","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6835/1289220491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1289220491},{"type":"v1","addr":"172.21.15.192:6835","nonce":1289220491}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:31:04.305 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 14 2023-10-30T12:31:04.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.301+0000 7efebd29e640 1 -- 172.21.15.89:0/3415171354 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7efe980664a0 msgr2=0x7efe98068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:04.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.301+0000 7efebd29e640 1 --2- 172.21.15.89:0/3415171354 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7efe980664a0 0x7efe98068960 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7efea80ae000 tx=0x7efea8001a80 comp rx=0 tx=0).stop 2023-10-30T12:31:04.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.301+0000 7efebd29e640 1 -- 172.21.15.89:0/3415171354 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efeb81538c0 msgr2=0x7efeb80c9ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:04.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.301+0000 7efebd29e640 1 --2- 172.21.15.89:0/3415171354 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efeb81538c0 0x7efeb80c9ec0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7efeb80c6fb0 tx=0x7efeb00115a0 comp rx=0 tx=0).stop 2023-10-30T12:31:04.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.301+0000 7efebd29e640 1 -- 172.21.15.89:0/3415171354 shutdown_connections 2023-10-30T12:31:04.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.301+0000 7efebd29e640 1 --2- 172.21.15.89:0/3415171354 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7efe980664a0 0x7efe98068960 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:04.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.301+0000 7efebd29e640 1 --2- 172.21.15.89:0/3415171354 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efeb81577f0 0x7efeb8156360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:04.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.301+0000 7efebd29e640 1 --2- 172.21.15.89:0/3415171354 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efeb81538c0 0x7efeb80c9ec0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:04.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.301+0000 7efebd29e640 1 --2- 172.21.15.89:0/3415171354 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efea4003980 0x7efeb8155e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:04.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.301+0000 7efebd29e640 1 -- 172.21.15.89:0/3415171354 >> 172.21.15.89:0/3415171354 conn(0x7efeb80c7910 msgr2=0x7efeb80c07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:04.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.301+0000 7efebd29e640 1 -- 172.21.15.89:0/3415171354 shutdown_connections 2023-10-30T12:31:04.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:04.301+0000 7efebd29e640 1 -- 172.21.15.89:0/3415171354 wait complete. 2023-10-30T12:31:04.325 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T12:31:06.326 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:31:06.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e28dc640 1 Processor -- start 2023-10-30T12:31:06.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e28dc640 1 -- start start 2023-10-30T12:31:06.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e28dc640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70dc15cd50 0x7f70dc0c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:06.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e28dc640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f70dc0cefa0 0x7f70dc0cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:06.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e28dc640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70dc15a6c0 con 0x7f70dc1538f0 2023-10-30T12:31:06.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e28dc640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70dc15d300 con 0x7f70dc0cefa0 2023-10-30T12:31:06.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e28dc640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70dc15d5e0 con 0x7f70dc15cd50 2023-10-30T12:31:06.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e18da640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f70dc1538f0 0x7f70dc159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43362/0 (socket says 172.21.15.89:43362) 2023-10-30T12:31:06.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e18da640 1 -- 172.21.15.89:0/3702683192 learned_addr learned my addr 172.21.15.89:0/3702683192 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:31:06.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e10d9640 1 --2- 172.21.15.89:0/3702683192 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70dc15cd50 0x7f70dc0c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:06.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e20db640 1 --2- 172.21.15.89:0/3702683192 >> v2:172.21.15.192:3300/0 conn(0x7f70dc0cefa0 0x7f70dc0cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:06.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e08d8640 1 -- 172.21.15.89:0/3702683192 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2632901610 0 0) 0x7f70dc15a6c0 con 0x7f70dc1538f0 2023-10-30T12:31:06.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e08d8640 1 -- 172.21.15.89:0/3702683192 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70c8003530 con 0x7f70dc1538f0 2023-10-30T12:31:06.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e08d8640 1 -- 172.21.15.89:0/3702683192 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3237850199 0 0) 0x7f70c8003530 con 0x7f70dc1538f0 2023-10-30T12:31:06.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e08d8640 1 -- 172.21.15.89:0/3702683192 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70dc15cd50 msgr2=0x7f70dc0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:06.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e08d8640 1 --2- 172.21.15.89:0/3702683192 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70dc15cd50 0x7f70dc0c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:06.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e08d8640 1 -- 172.21.15.89:0/3702683192 >> v2:172.21.15.192:3300/0 conn(0x7f70dc0cefa0 msgr2=0x7f70dc0cf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:06.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e08d8640 1 --2- 172.21.15.89:0/3702683192 >> v2:172.21.15.192:3300/0 conn(0x7f70dc0cefa0 0x7f70dc0cf3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:06.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e08d8640 1 -- 172.21.15.89:0/3702683192 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70dc15d8c0 con 0x7f70dc1538f0 2023-10-30T12:31:06.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e08d8640 1 -- 172.21.15.89:0/3702683192 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f70d8001680 con 0x7f70dc1538f0 2023-10-30T12:31:06.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e08d8640 1 -- 172.21.15.89:0/3702683192 >> v1:172.21.15.89:6789/0 conn(0x7f70dc1538f0 legacy=0x7f70dc159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:06.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e08d8640 1 --2- 172.21.15.89:0/3702683192 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70c8003980 0x7f70c8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:06.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e08d8640 1 --2- 172.21.15.89:0/3702683192 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70dc15cd50 0x7f70c80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:06.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e08d8640 1 --2- 172.21.15.89:0/3702683192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70c8004d10 0x7f70c8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:06.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e08d8640 1 -- 172.21.15.89:0/3702683192 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70dc15d300 con 0x7f70c8004d10 2023-10-30T12:31:06.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e08d8640 1 -- 172.21.15.89:0/3702683192 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70dc15d5e0 con 0x7f70c8003980 2023-10-30T12:31:06.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e08d8640 1 -- 172.21.15.89:0/3702683192 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c800b990 con 0x7f70dc15cd50 2023-10-30T12:31:06.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e20db640 1 --2- 172.21.15.89:0/3702683192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70c8004d10 0x7f70c8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:06.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e10d9640 1 --2- 172.21.15.89:0/3702683192 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70dc15cd50 0x7f70c80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:06.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e18da640 1 --2- 172.21.15.89:0/3702683192 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70c8003980 0x7f70c8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:06.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e20db640 1 -- 172.21.15.89:0/3702683192 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70dc15cd50 msgr2=0x7f70c80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:06.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e20db640 1 --2- 172.21.15.89:0/3702683192 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70dc15cd50 0x7f70c80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:06.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e20db640 1 -- 172.21.15.89:0/3702683192 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70c8003980 msgr2=0x7f70c8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:06.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e20db640 1 --2- 172.21.15.89:0/3702683192 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70c8003980 0x7f70c8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:06.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e20db640 1 -- 172.21.15.89:0/3702683192 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c8003530 con 0x7f70c8004d10 2023-10-30T12:31:06.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.481+0000 7f70e18da640 1 --2- 172.21.15.89:0/3702683192 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70c8003980 0x7f70c8003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:31:06.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.485+0000 7f70e20db640 1 --2- 172.21.15.89:0/3702683192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70c8004d10 0x7f70c8005190 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f70d4009fc0 tx=0x7f70d401cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c10a904834e5e350 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:06.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.485+0000 7f70e08d8640 1 -- 172.21.15.89:0/3702683192 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d40a5cb0 con 0x7f70c8004d10 2023-10-30T12:31:06.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.485+0000 7f70e08d8640 1 -- 172.21.15.89:0/3702683192 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d400f040 con 0x7f70c8004d10 2023-10-30T12:31:06.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.485+0000 7f70e08d8640 1 -- 172.21.15.89:0/3702683192 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d40ad680 con 0x7f70c8004d10 2023-10-30T12:31:06.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.485+0000 7f70e28dc640 1 -- 172.21.15.89:0/3702683192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70c8004d10 msgr2=0x7f70c8005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:06.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.485+0000 7f70e28dc640 1 --2- 172.21.15.89:0/3702683192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70c8004d10 0x7f70c8005190 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f70d4009fc0 tx=0x7f70d401cca0 comp rx=0 tx=0).stop 2023-10-30T12:31:06.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.485+0000 7f70e20db640 1 -- 172.21.15.89:0/3702683192 reap_dead start 2023-10-30T12:31:06.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.485+0000 7f70e28dc640 1 -- 172.21.15.89:0/3702683192 shutdown_connections 2023-10-30T12:31:06.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.485+0000 7f70e28dc640 1 -- 172.21.15.89:0/3702683192 >> 172.21.15.89:0/3702683192 conn(0x7f70dc06a640 msgr2=0x7f70dc065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:06.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.485+0000 7f70e28dc640 1 -- 172.21.15.89:0/3702683192 shutdown_connections 2023-10-30T12:31:06.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.485+0000 7f70e28dc640 1 -- 172.21.15.89:0/3702683192 wait complete. 2023-10-30T12:31:06.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.489+0000 7f70e28dc640 1 Processor -- start 2023-10-30T12:31:06.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.489+0000 7f70e28dc640 1 -- start start 2023-10-30T12:31:06.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.489+0000 7f70e28dc640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70dc147890 0x7f70dc158020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:06.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.489+0000 7f70e28dc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70dc147c70 0x7f70dc14e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:06.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.489+0000 7f70e28dc640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70dc144b50 0x7f70dc144fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:06.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.489+0000 7f70e28dc640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70dc158a70 con 0x7f70dc147c70 2023-10-30T12:31:06.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.489+0000 7f70e28dc640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70dc14fbc0 con 0x7f70dc147890 2023-10-30T12:31:06.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.489+0000 7f70e28dc640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70dc14e320 con 0x7f70dc144b50 2023-10-30T12:31:06.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.489+0000 7f70e10d9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70dc147c70 0x7f70dc14e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:06.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.489+0000 7f70e10d9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70dc147c70 0x7f70dc14e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41460/0 (socket says 172.21.15.89:41460) 2023-10-30T12:31:06.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.489+0000 7f70e10d9640 1 -- 172.21.15.89:0/4173020814 learned_addr learned my addr 172.21.15.89:0/4173020814 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:06.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.489+0000 7f70e20db640 1 --2- 172.21.15.89:0/4173020814 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70dc144b50 0x7f70dc144fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:06.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.489+0000 7f70e18da640 1 --2- 172.21.15.89:0/4173020814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70dc147890 0x7f70dc158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:06.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.489+0000 7f70e10d9640 1 -- 172.21.15.89:0/4173020814 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70dc144b50 msgr2=0x7f70dc144fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:06.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.489+0000 7f70e10d9640 1 --2- 172.21.15.89:0/4173020814 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70dc144b50 0x7f70dc144fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:06.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.489+0000 7f70e10d9640 1 -- 172.21.15.89:0/4173020814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70dc147890 msgr2=0x7f70dc158020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:06.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.489+0000 7f70e10d9640 1 --2- 172.21.15.89:0/4173020814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70dc147890 0x7f70dc158020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:06.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.489+0000 7f70e10d9640 1 -- 172.21.15.89:0/4173020814 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70dc14f600 con 0x7f70dc147c70 2023-10-30T12:31:06.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.489+0000 7f70e18da640 1 --2- 172.21.15.89:0/4173020814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70dc147890 0x7f70dc158020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:31:06.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.489+0000 7f70e10d9640 1 --2- 172.21.15.89:0/4173020814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70dc147c70 0x7f70dc14e7b0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f70cc011a00 tx=0x7f70cc011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:06.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.489+0000 7f70d27fc640 1 -- 172.21.15.89:0/4173020814 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70cc00fb50 con 0x7f70dc147c70 2023-10-30T12:31:06.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.489+0000 7f70d27fc640 1 -- 172.21.15.89:0/4173020814 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70cc00fcb0 con 0x7f70dc147c70 2023-10-30T12:31:06.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.489+0000 7f70d27fc640 1 -- 172.21.15.89:0/4173020814 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70cc014440 con 0x7f70dc147c70 2023-10-30T12:31:06.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.489+0000 7f70e28dc640 1 -- 172.21.15.89:0/4173020814 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70dc14f8e0 con 0x7f70dc147c70 2023-10-30T12:31:06.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.489+0000 7f70e28dc640 1 -- 172.21.15.89:0/4173020814 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70dc14dfb0 con 0x7f70dc147c70 2023-10-30T12:31:06.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.493+0000 7f70e28dc640 1 -- 172.21.15.89:0/4173020814 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70dc04e170 con 0x7f70dc147c70 2023-10-30T12:31:06.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.493+0000 7f70d27fc640 1 -- 172.21.15.89:0/4173020814 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f70cc0145a0 con 0x7f70dc147c70 2023-10-30T12:31:06.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.493+0000 7f70d27fc640 1 --2- 172.21.15.89:0/4173020814 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f70c00664a0 0x7f70c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:06.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.493+0000 7f70d27fc640 1 -- 172.21.15.89:0/4173020814 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5832+0+0 (secure 0 0 0) 0x7f70cc084760 con 0x7f70dc147c70 2023-10-30T12:31:06.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.493+0000 7f70e18da640 1 --2- 172.21.15.89:0/4173020814 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f70c00664a0 0x7f70c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:06.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.493+0000 7f70e18da640 1 --2- 172.21.15.89:0/4173020814 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f70c00664a0 0x7f70c0068960 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f70d8000c10 tx=0x7f70d8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:06.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.497+0000 7f70d27fc640 1 -- 172.21.15.89:0/4173020814 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f70cc051c10 con 0x7f70dc147c70 2023-10-30T12:31:06.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.681+0000 7f70e28dc640 1 -- 172.21.15.89:0/4173020814 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70dc0b2290 con 0x7f70dc147c70 2023-10-30T12:31:06.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.681+0000 7f70d27fc640 1 -- 172.21.15.89:0/4173020814 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+6531 (secure 0 0 0) 0x7f70cc055230 con 0x7f70dc147c70 2023-10-30T12:31:06.685 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:31:06.686 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:30:56.409108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4253":{"gid":4253,"name":"b","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6835/1289220491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1289220491},{"type":"v1","addr":"172.21.15.192:6835","nonce":1289220491}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:31:06.686 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 14 2023-10-30T12:31:06.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.681+0000 7f70e28dc640 1 -- 172.21.15.89:0/4173020814 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f70c00664a0 msgr2=0x7f70c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:06.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.681+0000 7f70e28dc640 1 --2- 172.21.15.89:0/4173020814 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f70c00664a0 0x7f70c0068960 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f70d8000c10 tx=0x7f70d8009040 comp rx=0 tx=0).stop 2023-10-30T12:31:06.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.681+0000 7f70e28dc640 1 -- 172.21.15.89:0/4173020814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70dc147c70 msgr2=0x7f70dc14e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:06.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.681+0000 7f70e28dc640 1 --2- 172.21.15.89:0/4173020814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70dc147c70 0x7f70dc14e7b0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f70cc011a00 tx=0x7f70cc011ec0 comp rx=0 tx=0).stop 2023-10-30T12:31:06.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.681+0000 7f70e28dc640 1 -- 172.21.15.89:0/4173020814 shutdown_connections 2023-10-30T12:31:06.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.681+0000 7f70e28dc640 1 --2- 172.21.15.89:0/4173020814 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f70c00664a0 0x7f70c0068960 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:06.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.681+0000 7f70e28dc640 1 --2- 172.21.15.89:0/4173020814 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f70dc144b50 0x7f70dc144fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:06.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.681+0000 7f70e28dc640 1 --2- 172.21.15.89:0/4173020814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70dc147c70 0x7f70dc14e7b0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:06.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.681+0000 7f70e28dc640 1 --2- 172.21.15.89:0/4173020814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f70dc147890 0x7f70dc158020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:06.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.681+0000 7f70e28dc640 1 -- 172.21.15.89:0/4173020814 >> 172.21.15.89:0/4173020814 conn(0x7f70dc06a640 msgr2=0x7f70dc0caaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:06.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.681+0000 7f70e28dc640 1 -- 172.21.15.89:0/4173020814 shutdown_connections 2023-10-30T12:31:06.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:06.681+0000 7f70e28dc640 1 -- 172.21.15.89:0/4173020814 wait complete. 2023-10-30T12:31:06.706 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T12:31:08.708 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:31:08.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.849+0000 7fe36c11d640 1 Processor -- start 2023-10-30T12:31:08.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.849+0000 7fe36c11d640 1 -- start start 2023-10-30T12:31:08.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.849+0000 7fe36c11d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe36415d6f0 0x7fe36415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:08.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.849+0000 7fe36c11d640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3640cb800 con 0x7fe3641538a0 2023-10-30T12:31:08.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.849+0000 7fe36c11d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3640cb330 con 0x7fe3640cf160 2023-10-30T12:31:08.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.849+0000 7fe36c11d640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3640cad70 con 0x7fe36415d6f0 2023-10-30T12:31:08.855 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.849+0000 7fe369eb9640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe3641538a0 0x7fe3640ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43374/0 (socket says 172.21.15.89:43374) 2023-10-30T12:31:08.855 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.849+0000 7fe369eb9640 1 -- 172.21.15.89:0/725104183 learned_addr learned my addr 172.21.15.89:0/725104183 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:31:08.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.849+0000 7fe3696b8640 1 --2- 172.21.15.89:0/725104183 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe36415d6f0 0x7fe36415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:08.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.849+0000 7fe368eb7640 1 -- 172.21.15.89:0/725104183 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1387320983 0 0) 0x7fe3640cb800 con 0x7fe3641538a0 2023-10-30T12:31:08.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.849+0000 7fe368eb7640 1 -- 172.21.15.89:0/725104183 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe350003530 con 0x7fe3641538a0 2023-10-30T12:31:08.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.849+0000 7fe368eb7640 1 -- 172.21.15.89:0/725104183 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1824677576 0 0) 0x7fe350003530 con 0x7fe3641538a0 2023-10-30T12:31:08.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.849+0000 7fe368eb7640 1 -- 172.21.15.89:0/725104183 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe36415d6f0 msgr2=0x7fe36415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:08.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.849+0000 7fe368eb7640 1 --2- 172.21.15.89:0/725104183 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe36415d6f0 0x7fe36415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:08.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.849+0000 7fe368eb7640 1 -- 172.21.15.89:0/725104183 >> v1:172.21.15.192:6789/0 conn(0x7fe3640cf160 legacy=0x7fe364164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:08.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.849+0000 7fe368eb7640 1 -- 172.21.15.89:0/725104183 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3640cb050 con 0x7fe3641538a0 2023-10-30T12:31:08.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.849+0000 7fe368eb7640 1 -- 172.21.15.89:0/725104183 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7fe360001680 con 0x7fe3641538a0 2023-10-30T12:31:08.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.849+0000 7fe368eb7640 1 -- 172.21.15.89:0/725104183 >> v1:172.21.15.89:6789/0 conn(0x7fe3641538a0 legacy=0x7fe3640ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:08.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.849+0000 7fe368eb7640 1 --2- 172.21.15.89:0/725104183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3500039b0 0x7fe350003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:08.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.849+0000 7fe368eb7640 1 --2- 172.21.15.89:0/725104183 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3500047d0 0x7fe350004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:08.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.849+0000 7fe368eb7640 1 --2- 172.21.15.89:0/725104183 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe36415d6f0 0x7fe350005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:08.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.849+0000 7fe368eb7640 1 -- 172.21.15.89:0/725104183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3640cad70 con 0x7fe3500039b0 2023-10-30T12:31:08.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.849+0000 7fe368eb7640 1 -- 172.21.15.89:0/725104183 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35000bb00 con 0x7fe3500047d0 2023-10-30T12:31:08.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.849+0000 7fe368eb7640 1 -- 172.21.15.89:0/725104183 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe35000bcc0 con 0x7fe36415d6f0 2023-10-30T12:31:08.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.849+0000 7fe369eb9640 1 --2- 172.21.15.89:0/725104183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3500039b0 0x7fe350003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:08.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.849+0000 7fe3696b8640 1 --2- 172.21.15.89:0/725104183 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3500047d0 0x7fe350004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:08.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.849+0000 7fe36a6ba640 1 --2- 172.21.15.89:0/725104183 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe36415d6f0 0x7fe350005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:08.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.853+0000 7fe369eb9640 1 -- 172.21.15.89:0/725104183 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe36415d6f0 msgr2=0x7fe350005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:08.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.853+0000 7fe369eb9640 1 --2- 172.21.15.89:0/725104183 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe36415d6f0 0x7fe350005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:08.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.853+0000 7fe369eb9640 1 -- 172.21.15.89:0/725104183 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3500047d0 msgr2=0x7fe350004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:08.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.853+0000 7fe369eb9640 1 --2- 172.21.15.89:0/725104183 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3500047d0 0x7fe350004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:08.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.853+0000 7fe369eb9640 1 -- 172.21.15.89:0/725104183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe350003530 con 0x7fe3500039b0 2023-10-30T12:31:08.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.853+0000 7fe3696b8640 1 --2- 172.21.15.89:0/725104183 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3500047d0 0x7fe350004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:31:08.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.853+0000 7fe369eb9640 1 --2- 172.21.15.89:0/725104183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3500039b0 0x7fe350003dd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe36000c210 tx=0x7fe36001f350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f203c9ade5526f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:08.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.853+0000 7fe368eb7640 1 -- 172.21.15.89:0/725104183 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe360005ae0 con 0x7fe3500039b0 2023-10-30T12:31:08.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.853+0000 7fe368eb7640 1 -- 172.21.15.89:0/725104183 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3600ab070 con 0x7fe3500039b0 2023-10-30T12:31:08.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.853+0000 7fe368eb7640 1 -- 172.21.15.89:0/725104183 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe360005e80 con 0x7fe3500039b0 2023-10-30T12:31:08.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.853+0000 7fe36c11d640 1 -- 172.21.15.89:0/725104183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3500039b0 msgr2=0x7fe350003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:08.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.853+0000 7fe36c11d640 1 --2- 172.21.15.89:0/725104183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3500039b0 0x7fe350003dd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe36000c210 tx=0x7fe36001f350 comp rx=0 tx=0).stop 2023-10-30T12:31:08.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.853+0000 7fe36a6ba640 1 -- 172.21.15.89:0/725104183 reap_dead start 2023-10-30T12:31:08.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.853+0000 7fe36c11d640 1 -- 172.21.15.89:0/725104183 shutdown_connections 2023-10-30T12:31:08.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.853+0000 7fe36c11d640 1 -- 172.21.15.89:0/725104183 >> 172.21.15.89:0/725104183 conn(0x7fe36406bfe0 msgr2=0x7fe36406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:08.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.857+0000 7fe36c11d640 1 -- 172.21.15.89:0/725104183 shutdown_connections 2023-10-30T12:31:08.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.857+0000 7fe36c11d640 1 -- 172.21.15.89:0/725104183 wait complete. 2023-10-30T12:31:08.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.857+0000 7fe36c11d640 1 Processor -- start 2023-10-30T12:31:08.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.857+0000 7fe36c11d640 1 -- start start 2023-10-30T12:31:08.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.857+0000 7fe36c11d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3500039b0 0x7fe3641500b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:08.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.857+0000 7fe36c11d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3641e4440 0x7fe364148ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:08.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.857+0000 7fe36c11d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3641491e0 0x7fe3640cdce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:08.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.857+0000 7fe36c11d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3641416f0 con 0x7fe3500039b0 2023-10-30T12:31:08.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.857+0000 7fe36c11d640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3641e3860 con 0x7fe3641e4440 2023-10-30T12:31:08.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.857+0000 7fe36c11d640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe364149660 con 0x7fe3641491e0 2023-10-30T12:31:08.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.857+0000 7fe369eb9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3500039b0 0x7fe3641500b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:08.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.857+0000 7fe369eb9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3500039b0 0x7fe3641500b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41474/0 (socket says 172.21.15.89:41474) 2023-10-30T12:31:08.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.857+0000 7fe369eb9640 1 -- 172.21.15.89:0/568284551 learned_addr learned my addr 172.21.15.89:0/568284551 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:08.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.857+0000 7fe3696b8640 1 --2- 172.21.15.89:0/568284551 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3641e4440 0x7fe364148ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:08.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.857+0000 7fe36a6ba640 1 --2- 172.21.15.89:0/568284551 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3641491e0 0x7fe3640cdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:08.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.857+0000 7fe369eb9640 1 -- 172.21.15.89:0/568284551 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3641491e0 msgr2=0x7fe3640cdce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:08.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.857+0000 7fe369eb9640 1 --2- 172.21.15.89:0/568284551 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3641491e0 0x7fe3640cdce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:08.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.857+0000 7fe369eb9640 1 -- 172.21.15.89:0/568284551 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3641e4440 msgr2=0x7fe364148ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:08.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.857+0000 7fe369eb9640 1 --2- 172.21.15.89:0/568284551 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3641e4440 0x7fe364148ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:08.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.857+0000 7fe369eb9640 1 -- 172.21.15.89:0/568284551 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3641e35c0 con 0x7fe3500039b0 2023-10-30T12:31:08.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.857+0000 7fe369eb9640 1 --2- 172.21.15.89:0/568284551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3500039b0 0x7fe3641500b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe360001940 tx=0x7fe350005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:08.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.857+0000 7fe35a7fc640 1 -- 172.21.15.89:0/568284551 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe360005a80 con 0x7fe3500039b0 2023-10-30T12:31:08.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.857+0000 7fe35a7fc640 1 -- 172.21.15.89:0/568284551 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe360005be0 con 0x7fe3500039b0 2023-10-30T12:31:08.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.857+0000 7fe35a7fc640 1 -- 172.21.15.89:0/568284551 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe360007070 con 0x7fe3500039b0 2023-10-30T12:31:08.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.857+0000 7fe36c11d640 1 -- 172.21.15.89:0/568284551 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3641484a0 con 0x7fe3500039b0 2023-10-30T12:31:08.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.861+0000 7fe36c11d640 1 -- 172.21.15.89:0/568284551 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3641505f0 con 0x7fe3500039b0 2023-10-30T12:31:08.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.861+0000 7fe36c11d640 1 -- 172.21.15.89:0/568284551 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe36404e170 con 0x7fe3500039b0 2023-10-30T12:31:08.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.861+0000 7fe35a7fc640 1 -- 172.21.15.89:0/568284551 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe3600071d0 con 0x7fe3500039b0 2023-10-30T12:31:08.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.861+0000 7fe35a7fc640 1 --2- 172.21.15.89:0/568284551 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe3480664a0 0x7fe348068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:08.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.861+0000 7fe35a7fc640 1 -- 172.21.15.89:0/568284551 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5832+0+0 (secure 0 0 0) 0x7fe3600b7040 con 0x7fe3500039b0 2023-10-30T12:31:08.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.861+0000 7fe3696b8640 1 --2- 172.21.15.89:0/568284551 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe3480664a0 0x7fe348068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:08.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.861+0000 7fe3696b8640 1 --2- 172.21.15.89:0/568284551 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe3480664a0 0x7fe348068960 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe364150c40 tx=0x7fe354008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:08.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:08.865+0000 7fe35a7fc640 1 -- 172.21.15.89:0/568284551 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe3600ede50 con 0x7fe3500039b0 2023-10-30T12:31:09.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:09.045+0000 7fe36c11d640 1 -- 172.21.15.89:0/568284551 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3640cdac0 con 0x7fe3500039b0 2023-10-30T12:31:09.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:09.045+0000 7fe35a7fc640 1 -- 172.21.15.89:0/568284551 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+6531 (secure 0 0 0) 0x7fe3600c1070 con 0x7fe3500039b0 2023-10-30T12:31:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:31:09.051 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:30:56.409108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4253":{"gid":4253,"name":"b","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6835/1289220491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1289220491},{"type":"v1","addr":"172.21.15.192:6835","nonce":1289220491}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:31:09.052 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 14 2023-10-30T12:31:09.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:09.049+0000 7fe36c11d640 1 -- 172.21.15.89:0/568284551 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe3480664a0 msgr2=0x7fe348068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:09.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:09.049+0000 7fe36c11d640 1 --2- 172.21.15.89:0/568284551 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe3480664a0 0x7fe348068960 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe364150c40 tx=0x7fe354008040 comp rx=0 tx=0).stop 2023-10-30T12:31:09.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:09.049+0000 7fe36c11d640 1 -- 172.21.15.89:0/568284551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3500039b0 msgr2=0x7fe3641500b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:09.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:09.049+0000 7fe36c11d640 1 --2- 172.21.15.89:0/568284551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3500039b0 0x7fe3641500b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe360001940 tx=0x7fe350005800 comp rx=0 tx=0).stop 2023-10-30T12:31:09.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:09.049+0000 7fe36c11d640 1 -- 172.21.15.89:0/568284551 shutdown_connections 2023-10-30T12:31:09.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:09.049+0000 7fe36c11d640 1 --2- 172.21.15.89:0/568284551 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe3480664a0 0x7fe348068960 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:09.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:09.049+0000 7fe36c11d640 1 --2- 172.21.15.89:0/568284551 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3641e4440 0x7fe364148ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:09.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:09.049+0000 7fe36c11d640 1 --2- 172.21.15.89:0/568284551 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3641491e0 0x7fe3640cdce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:09.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:09.049+0000 7fe36c11d640 1 --2- 172.21.15.89:0/568284551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3500039b0 0x7fe3641500b0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:09.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:09.049+0000 7fe36c11d640 1 -- 172.21.15.89:0/568284551 >> 172.21.15.89:0/568284551 conn(0x7fe36406bfe0 msgr2=0x7fe364065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:09.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:09.049+0000 7fe36c11d640 1 -- 172.21.15.89:0/568284551 shutdown_connections 2023-10-30T12:31:09.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:09.049+0000 7fe36c11d640 1 -- 172.21.15.89:0/568284551 wait complete. 2023-10-30T12:31:09.072 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T12:31:11.074 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:31:11.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.217+0000 7ff30af3e640 1 Processor -- start 2023-10-30T12:31:11.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.217+0000 7ff30af3e640 1 -- start start 2023-10-30T12:31:11.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.217+0000 7ff30af3e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff3040c05d0 0x7ff30413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:11.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.217+0000 7ff30af3e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3040c6f60 0x7ff30413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:11.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.217+0000 7ff30af3e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3040c0a50 con 0x7ff3040c6b80 2023-10-30T12:31:11.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.217+0000 7ff30af3e640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff30406f6e0 con 0x7ff3040c05d0 2023-10-30T12:31:11.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.217+0000 7ff30af3e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff30406f9c0 con 0x7ff3040c6f60 2023-10-30T12:31:11.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.217+0000 7ff2fbfff640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff3040c05d0 0x7ff30413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:11.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.217+0000 7ff3094db640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3040c6f60 0x7ff30413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:11.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.217+0000 7ff308cda640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7ff3040c6b80 0x7ff3040bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:51158/0 (socket says 172.21.15.89:51158) 2023-10-30T12:31:11.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.217+0000 7ff308cda640 1 -- 172.21.15.89:0/4058577771 learned_addr learned my addr 172.21.15.89:0/4058577771 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:31:11.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.221+0000 7ff2fb7fe640 1 -- 172.21.15.89:0/4058577771 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916104390 0 0) 0x7ff3040c0a50 con 0x7ff3040c6b80 2023-10-30T12:31:11.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.221+0000 7ff2fb7fe640 1 -- 172.21.15.89:0/4058577771 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2ec003530 con 0x7ff3040c6b80 2023-10-30T12:31:11.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.221+0000 7ff3094db640 1 -- 172.21.15.89:0/4058577771 >> v2:172.21.15.192:3300/0 conn(0x7ff3040c05d0 msgr2=0x7ff30413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:11.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.221+0000 7ff3094db640 1 --2- 172.21.15.89:0/4058577771 >> v2:172.21.15.192:3300/0 conn(0x7ff3040c05d0 0x7ff30413ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:11.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.221+0000 7ff3094db640 1 -- 172.21.15.89:0/4058577771 >> v1:172.21.15.192:6789/0 conn(0x7ff3040c6b80 legacy=0x7ff3040bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:11.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.221+0000 7ff3094db640 1 -- 172.21.15.89:0/4058577771 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30406fca0 con 0x7ff3040c6f60 2023-10-30T12:31:11.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.221+0000 7ff2fbfff640 1 --2- 172.21.15.89:0/4058577771 >> v2:172.21.15.192:3300/0 conn(0x7ff3040c05d0 0x7ff30413ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:31:11.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.221+0000 7ff3094db640 1 --2- 172.21.15.89:0/4058577771 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3040c6f60 0x7ff30413f2d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc009d90 tx=0x7ff2fc01c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=440264ede2580182 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:11.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.221+0000 7ff2fb7fe640 1 -- 172.21.15.89:0/4058577771 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2fc0a5920 con 0x7ff3040c6f60 2023-10-30T12:31:11.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.221+0000 7ff2fb7fe640 1 -- 172.21.15.89:0/4058577771 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2fc0a5a80 con 0x7ff3040c6f60 2023-10-30T12:31:11.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.221+0000 7ff2fb7fe640 1 -- 172.21.15.89:0/4058577771 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2fc0a5df0 con 0x7ff3040c6f60 2023-10-30T12:31:11.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.225+0000 7ff30af3e640 1 -- 172.21.15.89:0/4058577771 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3040c6f60 msgr2=0x7ff30413f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:11.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.225+0000 7ff30af3e640 1 --2- 172.21.15.89:0/4058577771 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3040c6f60 0x7ff30413f2d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc009d90 tx=0x7ff2fc01c950 comp rx=0 tx=0).stop 2023-10-30T12:31:11.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.225+0000 7ff30af3e640 1 -- 172.21.15.89:0/4058577771 shutdown_connections 2023-10-30T12:31:11.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.225+0000 7ff30af3e640 1 --2- 172.21.15.89:0/4058577771 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3040c6f60 0x7ff30413f2d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:11.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.225+0000 7ff30af3e640 1 --2- 172.21.15.89:0/4058577771 >> v2:172.21.15.192:3300/0 conn(0x7ff3040c05d0 0x7ff30413ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:11.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.225+0000 7ff30af3e640 1 -- 172.21.15.89:0/4058577771 >> 172.21.15.89:0/4058577771 conn(0x7ff3040c7910 msgr2=0x7ff3040c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:11.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.225+0000 7ff30af3e640 1 -- 172.21.15.89:0/4058577771 shutdown_connections 2023-10-30T12:31:11.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.225+0000 7ff30af3e640 1 -- 172.21.15.89:0/4058577771 wait complete. 2023-10-30T12:31:11.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.225+0000 7ff30af3e640 1 Processor -- start 2023-10-30T12:31:11.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.225+0000 7ff30af3e640 1 -- start start 2023-10-30T12:31:11.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.225+0000 7ff30af3e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3040c05d0 0x7ff30412eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:11.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.225+0000 7ff30af3e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3040c6b80 0x7ff30406a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:11.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.225+0000 7ff30af3e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3040c6f60 0x7ff30414d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:11.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.225+0000 7ff30af3e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff304150780 con 0x7ff3040c05d0 2023-10-30T12:31:11.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.225+0000 7ff30af3e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30412efe0 con 0x7ff3040c6b80 2023-10-30T12:31:11.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.225+0000 7ff30af3e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff30414c3c0 con 0x7ff3040c6f60 2023-10-30T12:31:11.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.225+0000 7ff308cda640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3040c05d0 0x7ff30412eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:11.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.225+0000 7ff308cda640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3040c05d0 0x7ff30412eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38198/0 (socket says 172.21.15.89:38198) 2023-10-30T12:31:11.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.225+0000 7ff308cda640 1 -- 172.21.15.89:0/473240181 learned_addr learned my addr 172.21.15.89:0/473240181 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:11.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.225+0000 7ff2fbfff640 1 --2- 172.21.15.89:0/473240181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3040c6b80 0x7ff30406a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:11.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.225+0000 7ff3094db640 1 --2- 172.21.15.89:0/473240181 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3040c6f60 0x7ff30414d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:11.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.229+0000 7ff308cda640 1 -- 172.21.15.89:0/473240181 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3040c6f60 msgr2=0x7ff30414d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:11.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.229+0000 7ff308cda640 1 --2- 172.21.15.89:0/473240181 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3040c6f60 0x7ff30414d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:11.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.229+0000 7ff308cda640 1 -- 172.21.15.89:0/473240181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3040c6b80 msgr2=0x7ff30406a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:11.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.229+0000 7ff308cda640 1 --2- 172.21.15.89:0/473240181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3040c6b80 0x7ff30406a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:11.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.229+0000 7ff308cda640 1 -- 172.21.15.89:0/473240181 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f4009420 con 0x7ff3040c05d0 2023-10-30T12:31:11.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.229+0000 7ff2fbfff640 1 --2- 172.21.15.89:0/473240181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3040c6b80 0x7ff30406a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:31:11.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.229+0000 7ff308cda640 1 --2- 172.21.15.89:0/473240181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3040c05d0 0x7ff30412eaa0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff30000c450 tx=0x7ff30000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:11.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.229+0000 7ff2f97fa640 1 -- 172.21.15.89:0/473240181 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff300005cf0 con 0x7ff3040c05d0 2023-10-30T12:31:11.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.229+0000 7ff2f97fa640 1 -- 172.21.15.89:0/473240181 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff300005e50 con 0x7ff3040c05d0 2023-10-30T12:31:11.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.229+0000 7ff30af3e640 1 -- 172.21.15.89:0/473240181 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2fc009a70 con 0x7ff3040c05d0 2023-10-30T12:31:11.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.229+0000 7ff2f97fa640 1 -- 172.21.15.89:0/473240181 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff300007290 con 0x7ff3040c05d0 2023-10-30T12:31:11.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.229+0000 7ff30af3e640 1 -- 172.21.15.89:0/473240181 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff30414c090 con 0x7ff3040c05d0 2023-10-30T12:31:11.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.229+0000 7ff30af3e640 1 -- 172.21.15.89:0/473240181 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff30406ade0 con 0x7ff3040c05d0 2023-10-30T12:31:11.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.229+0000 7ff2f97fa640 1 -- 172.21.15.89:0/473240181 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff3000073f0 con 0x7ff3040c05d0 2023-10-30T12:31:11.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.229+0000 7ff2f97fa640 1 --2- 172.21.15.89:0/473240181 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff2f00664d0 0x7ff2f0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:11.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.229+0000 7ff2f97fa640 1 -- 172.21.15.89:0/473240181 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5832+0+0 (secure 0 0 0) 0x7ff300083530 con 0x7ff3040c05d0 2023-10-30T12:31:11.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.229+0000 7ff2fbfff640 1 --2- 172.21.15.89:0/473240181 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff2f00664d0 0x7ff2f0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:11.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.233+0000 7ff2fbfff640 1 --2- 172.21.15.89:0/473240181 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff2f00664d0 0x7ff2f0068990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff30413dad0 tx=0x7ff2f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:11.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.233+0000 7ff2f97fa640 1 -- 172.21.15.89:0/473240181 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff300050890 con 0x7ff3040c05d0 2023-10-30T12:31:11.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.417+0000 7ff30af3e640 1 -- 172.21.15.89:0/473240181 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff30404e170 con 0x7ff3040c05d0 2023-10-30T12:31:11.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.417+0000 7ff2f97fa640 1 -- 172.21.15.89:0/473240181 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+6531 (secure 0 0 0) 0x7ff300017090 con 0x7ff3040c05d0 2023-10-30T12:31:11.422 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:31:11.422 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:30:56.409108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4253":{"gid":4253,"name":"b","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6835/1289220491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1289220491},{"type":"v1","addr":"172.21.15.192:6835","nonce":1289220491}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:31:11.423 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 14 2023-10-30T12:31:11.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.421+0000 7ff30af3e640 1 -- 172.21.15.89:0/473240181 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff2f00664d0 msgr2=0x7ff2f0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:11.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.421+0000 7ff30af3e640 1 --2- 172.21.15.89:0/473240181 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff2f00664d0 0x7ff2f0068990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff30413dad0 tx=0x7ff2f4008040 comp rx=0 tx=0).stop 2023-10-30T12:31:11.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.421+0000 7ff30af3e640 1 -- 172.21.15.89:0/473240181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3040c05d0 msgr2=0x7ff30412eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:11.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.421+0000 7ff30af3e640 1 --2- 172.21.15.89:0/473240181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3040c05d0 0x7ff30412eaa0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff30000c450 tx=0x7ff30000c910 comp rx=0 tx=0).stop 2023-10-30T12:31:11.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.421+0000 7ff30af3e640 1 -- 172.21.15.89:0/473240181 shutdown_connections 2023-10-30T12:31:11.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.421+0000 7ff30af3e640 1 --2- 172.21.15.89:0/473240181 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff2f00664d0 0x7ff2f0068990 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:11.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.421+0000 7ff30af3e640 1 --2- 172.21.15.89:0/473240181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3040c6b80 0x7ff30406a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:11.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.421+0000 7ff30af3e640 1 --2- 172.21.15.89:0/473240181 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3040c6f60 0x7ff30414d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:11.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.421+0000 7ff30af3e640 1 --2- 172.21.15.89:0/473240181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3040c05d0 0x7ff30412eaa0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:11.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.421+0000 7ff30af3e640 1 -- 172.21.15.89:0/473240181 >> 172.21.15.89:0/473240181 conn(0x7ff3040c7910 msgr2=0x7ff30413a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:11.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.421+0000 7ff30af3e640 1 -- 172.21.15.89:0/473240181 shutdown_connections 2023-10-30T12:31:11.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:11.421+0000 7ff30af3e640 1 -- 172.21.15.89:0/473240181 wait complete. 2023-10-30T12:31:11.444 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T12:31:13.445 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:31:13.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.589+0000 7f6ca830f640 1 Processor -- start 2023-10-30T12:31:13.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.589+0000 7f6ca830f640 1 -- start start 2023-10-30T12:31:13.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.589+0000 7f6ca830f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca00c6b80 0x7f6ca00bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:13.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.589+0000 7f6ca830f640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6ca00c6f60 0x7f6ca00c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:13.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.589+0000 7f6ca830f640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ca00c0a50 con 0x7f6ca00c8430 2023-10-30T12:31:13.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.589+0000 7f6ca830f640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ca006f6e0 con 0x7f6ca00c6f60 2023-10-30T12:31:13.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.589+0000 7f6ca830f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca006f9c0 con 0x7f6ca00c6b80 2023-10-30T12:31:13.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.589+0000 7f6ca58aa640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6ca00c6f60 0x7f6ca00c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:13.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.589+0000 7f6ca58aa640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6ca00c6f60 0x7f6ca00c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38204/0 (socket says 172.21.15.89:38204) 2023-10-30T12:31:13.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.589+0000 7f6ca58aa640 1 -- 172.21.15.89:0/3224976047 learned_addr learned my addr 172.21.15.89:0/3224976047 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:13.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.589+0000 7f6ca60ab640 1 --2- 172.21.15.89:0/3224976047 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca00c6b80 0x7f6ca00bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:13.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.593+0000 7f6ca58aa640 1 -- 172.21.15.89:0/3224976047 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca00c6b80 msgr2=0x7f6ca00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:13.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.593+0000 7f6ca58aa640 1 --2- 172.21.15.89:0/3224976047 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca00c6b80 0x7f6ca00bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:13.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.593+0000 7f6ca58aa640 1 -- 172.21.15.89:0/3224976047 >> v1:172.21.15.192:6789/0 conn(0x7f6ca00c8430 legacy=0x7f6ca013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:13.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.593+0000 7f6ca58aa640 1 -- 172.21.15.89:0/3224976047 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca006fca0 con 0x7f6ca00c6f60 2023-10-30T12:31:13.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.593+0000 7f6ca58aa640 1 --2- 172.21.15.89:0/3224976047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ca00c6f60 0x7f6ca00c0400 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6c90009870 tx=0x7f6c9001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db06c1b9c5199eec server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:13.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.593+0000 7f6ca50a9640 1 -- 172.21.15.89:0/3224976047 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c900ab650 con 0x7f6ca00c6f60 2023-10-30T12:31:13.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.593+0000 7f6ca50a9640 1 -- 172.21.15.89:0/3224976047 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c900ab7b0 con 0x7f6ca00c6f60 2023-10-30T12:31:13.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.593+0000 7f6ca50a9640 1 -- 172.21.15.89:0/3224976047 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c900abb40 con 0x7f6ca00c6f60 2023-10-30T12:31:13.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.593+0000 7f6ca830f640 1 -- 172.21.15.89:0/3224976047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ca00c6f60 msgr2=0x7f6ca00c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:13.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.593+0000 7f6ca830f640 1 --2- 172.21.15.89:0/3224976047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ca00c6f60 0x7f6ca00c0400 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6c90009870 tx=0x7f6c9001c540 comp rx=0 tx=0).stop 2023-10-30T12:31:13.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.597+0000 7f6ca830f640 1 -- 172.21.15.89:0/3224976047 shutdown_connections 2023-10-30T12:31:13.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.597+0000 7f6ca830f640 1 --2- 172.21.15.89:0/3224976047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ca00c6f60 0x7f6ca00c0400 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:13.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.597+0000 7f6ca830f640 1 --2- 172.21.15.89:0/3224976047 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca00c6b80 0x7f6ca00bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:13.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.597+0000 7f6ca830f640 1 -- 172.21.15.89:0/3224976047 >> 172.21.15.89:0/3224976047 conn(0x7f6ca00c7910 msgr2=0x7f6ca00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:13.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.597+0000 7f6ca830f640 1 -- 172.21.15.89:0/3224976047 shutdown_connections 2023-10-30T12:31:13.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.597+0000 7f6ca830f640 1 -- 172.21.15.89:0/3224976047 wait complete. 2023-10-30T12:31:13.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.597+0000 7f6ca830f640 1 Processor -- start 2023-10-30T12:31:13.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.597+0000 7f6ca830f640 1 -- start start 2023-10-30T12:31:13.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.597+0000 7f6ca830f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca00c6b80 0x7f6ca01560f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:13.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.597+0000 7f6ca830f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca00c6f60 0x7f6ca012fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:13.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.597+0000 7f6ca830f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ca00c8430 0x7f6ca0150350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:13.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.597+0000 7f6ca830f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca0134d40 con 0x7f6ca00c8430 2023-10-30T12:31:13.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.597+0000 7f6ca830f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca0155500 con 0x7f6ca00c6f60 2023-10-30T12:31:13.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.597+0000 7f6ca830f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca0150890 con 0x7f6ca00c6b80 2023-10-30T12:31:13.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.597+0000 7f6ca68ac640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ca00c8430 0x7f6ca0150350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:13.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.597+0000 7f6ca68ac640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ca00c8430 0x7f6ca0150350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38216/0 (socket says 172.21.15.89:38216) 2023-10-30T12:31:13.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.597+0000 7f6ca68ac640 1 -- 172.21.15.89:0/1713442111 learned_addr learned my addr 172.21.15.89:0/1713442111 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:13.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.597+0000 7f6ca58aa640 1 --2- 172.21.15.89:0/1713442111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca00c6f60 0x7f6ca012fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:13.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.597+0000 7f6ca60ab640 1 --2- 172.21.15.89:0/1713442111 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca00c6b80 0x7f6ca01560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:13.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.597+0000 7f6ca68ac640 1 -- 172.21.15.89:0/1713442111 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca00c6b80 msgr2=0x7f6ca01560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:13.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.597+0000 7f6ca68ac640 1 --2- 172.21.15.89:0/1713442111 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca00c6b80 0x7f6ca01560f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:13.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.597+0000 7f6ca68ac640 1 -- 172.21.15.89:0/1713442111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca00c6f60 msgr2=0x7f6ca012fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:13.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.597+0000 7f6ca68ac640 1 --2- 172.21.15.89:0/1713442111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca00c6f60 0x7f6ca012fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:13.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.597+0000 7f6ca68ac640 1 -- 172.21.15.89:0/1713442111 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c900094d0 con 0x7f6ca00c8430 2023-10-30T12:31:13.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.597+0000 7f6ca68ac640 1 --2- 172.21.15.89:0/1713442111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ca00c8430 0x7f6ca0150350 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6c9800c730 tx=0x7f6c9800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:13.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.597+0000 7f6c96ffd640 1 -- 172.21.15.89:0/1713442111 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c98003b80 con 0x7f6ca00c8430 2023-10-30T12:31:13.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.601+0000 7f6c96ffd640 1 -- 172.21.15.89:0/1713442111 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c98003ce0 con 0x7f6ca00c8430 2023-10-30T12:31:13.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.601+0000 7f6ca830f640 1 -- 172.21.15.89:0/1713442111 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ca0133330 con 0x7f6ca00c8430 2023-10-30T12:31:13.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.601+0000 7f6c96ffd640 1 -- 172.21.15.89:0/1713442111 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9800aa30 con 0x7f6ca00c8430 2023-10-30T12:31:13.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.601+0000 7f6ca830f640 1 -- 172.21.15.89:0/1713442111 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ca0154c60 con 0x7f6ca00c8430 2023-10-30T12:31:13.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.601+0000 7f6ca830f640 1 -- 172.21.15.89:0/1713442111 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ca004e170 con 0x7f6ca00c8430 2023-10-30T12:31:13.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.601+0000 7f6c96ffd640 1 -- 172.21.15.89:0/1713442111 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6c9800ab90 con 0x7f6ca00c8430 2023-10-30T12:31:13.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.601+0000 7f6c96ffd640 1 --2- 172.21.15.89:0/1713442111 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6c840664a0 0x7f6c84068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:13.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.601+0000 7f6c96ffd640 1 -- 172.21.15.89:0/1713442111 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5832+0+0 (secure 0 0 0) 0x7f6c98083cc0 con 0x7f6ca00c8430 2023-10-30T12:31:13.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.601+0000 7f6ca60ab640 1 --2- 172.21.15.89:0/1713442111 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6c840664a0 0x7f6c84068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:13.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.605+0000 7f6ca60ab640 1 --2- 172.21.15.89:0/1713442111 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6c840664a0 0x7f6c84068960 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c009af0 tx=0x7f6c9c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:13.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.605+0000 7f6c96ffd640 1 -- 172.21.15.89:0/1713442111 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6c9808a080 con 0x7f6ca00c8430 2023-10-30T12:31:13.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.785+0000 7f6ca830f640 1 -- 172.21.15.89:0/1713442111 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ca01334c0 con 0x7f6ca00c8430 2023-10-30T12:31:13.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.785+0000 7f6c96ffd640 1 -- 172.21.15.89:0/1713442111 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+6531 (secure 0 0 0) 0x7f6c98051020 con 0x7f6ca00c8430 2023-10-30T12:31:13.790 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:31:13.790 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:30:56.409108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4253":{"gid":4253,"name":"b","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6835/1289220491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1289220491},{"type":"v1","addr":"172.21.15.192:6835","nonce":1289220491}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:31:13.791 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 14 2023-10-30T12:31:13.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.789+0000 7f6ca830f640 1 -- 172.21.15.89:0/1713442111 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6c840664a0 msgr2=0x7f6c84068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:13.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.789+0000 7f6ca830f640 1 --2- 172.21.15.89:0/1713442111 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6c840664a0 0x7f6c84068960 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c009af0 tx=0x7f6c9c008040 comp rx=0 tx=0).stop 2023-10-30T12:31:13.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.789+0000 7f6ca830f640 1 -- 172.21.15.89:0/1713442111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ca00c8430 msgr2=0x7f6ca0150350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:13.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.789+0000 7f6ca830f640 1 --2- 172.21.15.89:0/1713442111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ca00c8430 0x7f6ca0150350 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6c9800c730 tx=0x7f6c9800cbf0 comp rx=0 tx=0).stop 2023-10-30T12:31:13.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.789+0000 7f6ca830f640 1 -- 172.21.15.89:0/1713442111 shutdown_connections 2023-10-30T12:31:13.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.789+0000 7f6ca830f640 1 --2- 172.21.15.89:0/1713442111 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6c840664a0 0x7f6c84068960 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:13.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.789+0000 7f6ca830f640 1 --2- 172.21.15.89:0/1713442111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca00c6f60 0x7f6ca012fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:13.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.789+0000 7f6ca830f640 1 --2- 172.21.15.89:0/1713442111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ca00c8430 0x7f6ca0150350 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:13.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.789+0000 7f6ca830f640 1 --2- 172.21.15.89:0/1713442111 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ca00c6b80 0x7f6ca01560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:13.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.789+0000 7f6ca830f640 1 -- 172.21.15.89:0/1713442111 >> 172.21.15.89:0/1713442111 conn(0x7f6ca00c7910 msgr2=0x7f6ca01398b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:13.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.789+0000 7f6ca830f640 1 -- 172.21.15.89:0/1713442111 shutdown_connections 2023-10-30T12:31:13.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:13.789+0000 7f6ca830f640 1 -- 172.21.15.89:0/1713442111 wait complete. 2023-10-30T12:31:13.811 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T12:31:15.184 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:31:15.188 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:31:15.812 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:31:15.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.957+0000 7f1988417640 1 Processor -- start 2023-10-30T12:31:15.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.957+0000 7f1988417640 1 -- start start 2023-10-30T12:31:15.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.957+0000 7f1988417640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19800c05d0 0x7f198013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:15.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.957+0000 7f1988417640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f19800c6f60 0x7f198013f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:15.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.957+0000 7f1988417640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19800c0a50 con 0x7f19800c6b80 2023-10-30T12:31:15.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.957+0000 7f1988417640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f198006f6e0 con 0x7f19800c6f60 2023-10-30T12:31:15.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.957+0000 7f1988417640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f198006f9c0 con 0x7f19800c05d0 2023-10-30T12:31:15.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.957+0000 7f19861b3640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f19800c6b80 0x7f19800bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48728/0 (socket says 172.21.15.89:48728) 2023-10-30T12:31:15.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.957+0000 7f19861b3640 1 -- 172.21.15.89:0/881196328 learned_addr learned my addr 172.21.15.89:0/881196328 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:31:15.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.957+0000 7f19859b2640 1 --2- 172.21.15.89:0/881196328 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19800c05d0 0x7f198013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:15.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.957+0000 7f19869b4640 1 --2- 172.21.15.89:0/881196328 >> v2:172.21.15.192:3300/0 conn(0x7f19800c6f60 0x7f198013f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:15.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.957+0000 7f19851b1640 1 -- 172.21.15.89:0/881196328 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2305432642 0 0) 0x7f19800c0a50 con 0x7f19800c6b80 2023-10-30T12:31:15.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.957+0000 7f19851b1640 1 -- 172.21.15.89:0/881196328 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f196c003530 con 0x7f19800c6b80 2023-10-30T12:31:15.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.957+0000 7f19851b1640 1 -- 172.21.15.89:0/881196328 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3918326330 0 0) 0x7f196c003530 con 0x7f19800c6b80 2023-10-30T12:31:15.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.957+0000 7f19851b1640 1 -- 172.21.15.89:0/881196328 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19800c05d0 msgr2=0x7f198013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:15.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.957+0000 7f19851b1640 1 --2- 172.21.15.89:0/881196328 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19800c05d0 0x7f198013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:15.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.957+0000 7f19851b1640 1 -- 172.21.15.89:0/881196328 >> v2:172.21.15.192:3300/0 conn(0x7f19800c6f60 msgr2=0x7f198013f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:15.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.957+0000 7f19851b1640 1 --2- 172.21.15.89:0/881196328 >> v2:172.21.15.192:3300/0 conn(0x7f19800c6f60 0x7f198013f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:15.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.957+0000 7f19851b1640 1 -- 172.21.15.89:0/881196328 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f198006fca0 con 0x7f19800c6b80 2023-10-30T12:31:15.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.957+0000 7f19851b1640 1 -- 172.21.15.89:0/881196328 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f197c001680 con 0x7f19800c6b80 2023-10-30T12:31:15.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.957+0000 7f19851b1640 1 -- 172.21.15.89:0/881196328 >> v1:172.21.15.89:6789/0 conn(0x7f19800c6b80 legacy=0x7f19800bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:15.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.957+0000 7f19851b1640 1 --2- 172.21.15.89:0/881196328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f196c003980 0x7f196c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:15.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.957+0000 7f19851b1640 1 --2- 172.21.15.89:0/881196328 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19800c05d0 0x7f196c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:15.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.957+0000 7f19851b1640 1 --2- 172.21.15.89:0/881196328 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f196c004d10 0x7f196c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:15.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.957+0000 7f19851b1640 1 -- 172.21.15.89:0/881196328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f198006f6e0 con 0x7f196c003980 2023-10-30T12:31:15.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.957+0000 7f19851b1640 1 -- 172.21.15.89:0/881196328 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f198006f9c0 con 0x7f196c004d10 2023-10-30T12:31:15.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.957+0000 7f19851b1640 1 -- 172.21.15.89:0/881196328 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f196c00b990 con 0x7f19800c05d0 2023-10-30T12:31:15.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.961+0000 7f19861b3640 1 --2- 172.21.15.89:0/881196328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f196c003980 0x7f196c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:15.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.961+0000 7f19859b2640 1 --2- 172.21.15.89:0/881196328 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19800c05d0 0x7f196c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:15.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.961+0000 7f19869b4640 1 --2- 172.21.15.89:0/881196328 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f196c004d10 0x7f196c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:15.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.961+0000 7f19861b3640 1 -- 172.21.15.89:0/881196328 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19800c05d0 msgr2=0x7f196c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:15.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.961+0000 7f19861b3640 1 --2- 172.21.15.89:0/881196328 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19800c05d0 0x7f196c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:15.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.961+0000 7f19861b3640 1 -- 172.21.15.89:0/881196328 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f196c004d10 msgr2=0x7f196c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:15.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.961+0000 7f19861b3640 1 --2- 172.21.15.89:0/881196328 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f196c004d10 0x7f196c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:15.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.961+0000 7f19861b3640 1 -- 172.21.15.89:0/881196328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f196c003530 con 0x7f196c003980 2023-10-30T12:31:15.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.961+0000 7f19859b2640 1 --2- 172.21.15.89:0/881196328 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19800c05d0 0x7f196c0047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:31:15.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.961+0000 7f19861b3640 1 --2- 172.21.15.89:0/881196328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f196c003980 0x7f196c003da0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f197c00c2c0 tx=0x7f197c01ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aa10c93dbfe1e38 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:15.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.961+0000 7f19851b1640 1 -- 172.21.15.89:0/881196328 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f197c0058e0 con 0x7f196c003980 2023-10-30T12:31:15.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.961+0000 7f19851b1640 1 -- 172.21.15.89:0/881196328 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f197c005a40 con 0x7f196c003980 2023-10-30T12:31:15.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.961+0000 7f19851b1640 1 -- 172.21.15.89:0/881196328 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f197c005df0 con 0x7f196c003980 2023-10-30T12:31:15.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.965+0000 7f1988417640 1 -- 172.21.15.89:0/881196328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f196c003980 msgr2=0x7f196c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:15.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.965+0000 7f1988417640 1 --2- 172.21.15.89:0/881196328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f196c003980 0x7f196c003da0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f197c00c2c0 tx=0x7f197c01ccc0 comp rx=0 tx=0).stop 2023-10-30T12:31:15.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.965+0000 7f19869b4640 1 -- 172.21.15.89:0/881196328 reap_dead start 2023-10-30T12:31:15.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.965+0000 7f1988417640 1 -- 172.21.15.89:0/881196328 shutdown_connections 2023-10-30T12:31:15.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.965+0000 7f1988417640 1 -- 172.21.15.89:0/881196328 >> 172.21.15.89:0/881196328 conn(0x7f19800c7910 msgr2=0x7f19800c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:15.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.965+0000 7f1988417640 1 -- 172.21.15.89:0/881196328 shutdown_connections 2023-10-30T12:31:15.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.965+0000 7f1988417640 1 -- 172.21.15.89:0/881196328 wait complete. 2023-10-30T12:31:15.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.965+0000 7f1988417640 1 Processor -- start 2023-10-30T12:31:15.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.965+0000 7f1988417640 1 -- start start 2023-10-30T12:31:15.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.965+0000 7f1988417640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f196c003980 0x7f1980139b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:15.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.965+0000 7f1988417640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1980138e10 0x7f198015ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:15.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.965+0000 7f1988417640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f198015f480 0x7f198013cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:15.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.965+0000 7f1988417640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19801f2f10 con 0x7f1980138e10 2023-10-30T12:31:15.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.965+0000 7f1988417640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f198013a720 con 0x7f198015f480 2023-10-30T12:31:15.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.965+0000 7f1988417640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f198013aa00 con 0x7f196c003980 2023-10-30T12:31:15.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.965+0000 7f19859b2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1980138e10 0x7f198015ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:15.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.965+0000 7f19859b2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1980138e10 0x7f198015ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38238/0 (socket says 172.21.15.89:38238) 2023-10-30T12:31:15.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.965+0000 7f19859b2640 1 -- 172.21.15.89:0/1770191257 learned_addr learned my addr 172.21.15.89:0/1770191257 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:15.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.965+0000 7f19861b3640 1 --2- 172.21.15.89:0/1770191257 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f196c003980 0x7f1980139b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:15.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.965+0000 7f19869b4640 1 --2- 172.21.15.89:0/1770191257 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f198015f480 0x7f198013cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:15.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.965+0000 7f19859b2640 1 -- 172.21.15.89:0/1770191257 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f196c003980 msgr2=0x7f1980139b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:15.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.965+0000 7f19859b2640 1 --2- 172.21.15.89:0/1770191257 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f196c003980 0x7f1980139b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:15.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.965+0000 7f19859b2640 1 -- 172.21.15.89:0/1770191257 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f198015f480 msgr2=0x7f198013cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:15.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.965+0000 7f19859b2640 1 --2- 172.21.15.89:0/1770191257 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f198015f480 0x7f198013cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:15.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.965+0000 7f19859b2640 1 -- 172.21.15.89:0/1770191257 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f198013ace0 con 0x7f1980138e10 2023-10-30T12:31:15.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.965+0000 7f19859b2640 1 --2- 172.21.15.89:0/1770191257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1980138e10 0x7f198015ef40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1970009160 tx=0x7f1970011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:15.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.969+0000 7f1976ffd640 1 -- 172.21.15.89:0/1770191257 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1970004220 con 0x7f1980138e10 2023-10-30T12:31:15.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.969+0000 7f1976ffd640 1 -- 172.21.15.89:0/1770191257 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1970004380 con 0x7f1980138e10 2023-10-30T12:31:15.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.969+0000 7f1976ffd640 1 -- 172.21.15.89:0/1770191257 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1970015890 con 0x7f1980138e10 2023-10-30T12:31:15.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.969+0000 7f1988417640 1 -- 172.21.15.89:0/1770191257 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f198013afc0 con 0x7f1980138e10 2023-10-30T12:31:15.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.969+0000 7f1988417640 1 -- 172.21.15.89:0/1770191257 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f198015fa60 con 0x7f1980138e10 2023-10-30T12:31:15.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.969+0000 7f1988417640 1 -- 172.21.15.89:0/1770191257 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f198013b150 con 0x7f1980138e10 2023-10-30T12:31:15.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.969+0000 7f1976ffd640 1 -- 172.21.15.89:0/1770191257 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f19700159f0 con 0x7f1980138e10 2023-10-30T12:31:15.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.969+0000 7f1976ffd640 1 --2- 172.21.15.89:0/1770191257 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f19640664a0 0x7f1964068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:15.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.969+0000 7f1976ffd640 1 -- 172.21.15.89:0/1770191257 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5832+0+0 (secure 0 0 0) 0x7f19700842b0 con 0x7f1980138e10 2023-10-30T12:31:15.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.969+0000 7f19861b3640 1 --2- 172.21.15.89:0/1770191257 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f19640664a0 0x7f1964068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:15.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.973+0000 7f19861b3640 1 --2- 172.21.15.89:0/1770191257 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f19640664a0 0x7f1964068960 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f197c0aa000 tx=0x7f197c001a10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:15.982 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:15.973+0000 7f1976ffd640 1 -- 172.21.15.89:0/1770191257 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f19700517f0 con 0x7f1980138e10 2023-10-30T12:31:16.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:16.157+0000 7f1988417640 1 -- 172.21.15.89:0/1770191257 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f198004e170 con 0x7f1980138e10 2023-10-30T12:31:16.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:16.157+0000 7f1976ffd640 1 -- 172.21.15.89:0/1770191257 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+6531 (secure 0 0 0) 0x7f1970054e10 con 0x7f1980138e10 2023-10-30T12:31:16.162 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:31:16.162 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:30:56.409108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4253":{"gid":4253,"name":"b","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6835/1289220491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1289220491},{"type":"v1","addr":"172.21.15.192:6835","nonce":1289220491}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:31:16.162 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 14 2023-10-30T12:31:16.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:16.157+0000 7f1988417640 1 -- 172.21.15.89:0/1770191257 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f19640664a0 msgr2=0x7f1964068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:16.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:16.157+0000 7f1988417640 1 --2- 172.21.15.89:0/1770191257 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f19640664a0 0x7f1964068960 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f197c0aa000 tx=0x7f197c001a10 comp rx=0 tx=0).stop 2023-10-30T12:31:16.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:16.157+0000 7f1988417640 1 -- 172.21.15.89:0/1770191257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1980138e10 msgr2=0x7f198015ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:16.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:16.157+0000 7f1988417640 1 --2- 172.21.15.89:0/1770191257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1980138e10 0x7f198015ef40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1970009160 tx=0x7f1970011a60 comp rx=0 tx=0).stop 2023-10-30T12:31:16.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:16.157+0000 7f1988417640 1 -- 172.21.15.89:0/1770191257 shutdown_connections 2023-10-30T12:31:16.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:16.157+0000 7f1988417640 1 --2- 172.21.15.89:0/1770191257 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f19640664a0 0x7f1964068960 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:16.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:16.157+0000 7f1988417640 1 --2- 172.21.15.89:0/1770191257 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f198015f480 0x7f198013cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:16.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:16.157+0000 7f1988417640 1 --2- 172.21.15.89:0/1770191257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1980138e10 0x7f198015ef40 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:16.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:16.157+0000 7f1988417640 1 --2- 172.21.15.89:0/1770191257 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f196c003980 0x7f1980139b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:16.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:16.157+0000 7f1988417640 1 -- 172.21.15.89:0/1770191257 >> 172.21.15.89:0/1770191257 conn(0x7f19800c7910 msgr2=0x7f1980137580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:16.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:16.157+0000 7f1988417640 1 -- 172.21.15.89:0/1770191257 shutdown_connections 2023-10-30T12:31:16.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:16.157+0000 7f1988417640 1 -- 172.21.15.89:0/1770191257 wait complete. 2023-10-30T12:31:16.182 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T12:31:18.183 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:31:18.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.317+0000 7f2a6e29c640 1 Processor -- start 2023-10-30T12:31:18.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.317+0000 7f2a6e29c640 1 -- start start 2023-10-30T12:31:18.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.317+0000 7f2a6e29c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2a680c6fe0 0x7f2a68071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:18.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.317+0000 7f2a6e29c640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2a680c5f60 0x7f2a680cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:18.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.317+0000 7f2a6e29c640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a680700b0 con 0x7f2a68153bf0 2023-10-30T12:31:18.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.317+0000 7f2a6e29c640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a680c6810 con 0x7f2a680c6fe0 2023-10-30T12:31:18.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.317+0000 7f2a6e29c640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a68159900 con 0x7f2a680c5f60 2023-10-30T12:31:18.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.317+0000 7f2a67fff640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2a68153bf0 0x7f2a680c74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48740/0 (socket says 172.21.15.89:48740) 2023-10-30T12:31:18.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.317+0000 7f2a67fff640 1 -- 172.21.15.89:0/2669758442 learned_addr learned my addr 172.21.15.89:0/2669758442 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:31:18.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.317+0000 7f2a677fe640 1 --2- 172.21.15.89:0/2669758442 >> v2:172.21.15.89:3300/0 conn(0x7f2a680c6fe0 0x7f2a68071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:18.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.317+0000 7f2a6c839640 1 --2- 172.21.15.89:0/2669758442 >> v2:172.21.15.192:3300/0 conn(0x7f2a680c5f60 0x7f2a680cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:18.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.321+0000 7f2a66ffd640 1 -- 172.21.15.89:0/2669758442 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2674463941 0 0) 0x7f2a680700b0 con 0x7f2a68153bf0 2023-10-30T12:31:18.326 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.321+0000 7f2a677fe640 1 -- 172.21.15.89:0/2669758442 >> v2:172.21.15.192:3300/0 conn(0x7f2a680c5f60 msgr2=0x7f2a680cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:18.326 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.321+0000 7f2a677fe640 1 --2- 172.21.15.89:0/2669758442 >> v2:172.21.15.192:3300/0 conn(0x7f2a680c5f60 0x7f2a680cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:18.326 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.321+0000 7f2a677fe640 1 -- 172.21.15.89:0/2669758442 >> v1:172.21.15.89:6789/0 conn(0x7f2a68153bf0 legacy=0x7f2a680c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:18.326 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.321+0000 7f2a677fe640 1 -- 172.21.15.89:0/2669758442 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a680c6530 con 0x7f2a680c6fe0 2023-10-30T12:31:18.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.321+0000 7f2a677fe640 1 --2- 172.21.15.89:0/2669758442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a680c6fe0 0x7f2a68071260 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2a6806ce90 tx=0x7f2a5c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b890c5f3ef27df0a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:18.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.321+0000 7f2a66ffd640 1 -- 172.21.15.89:0/2669758442 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c0a8950 con 0x7f2a680c6fe0 2023-10-30T12:31:18.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.321+0000 7f2a66ffd640 1 -- 172.21.15.89:0/2669758442 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a5c0a8ab0 con 0x7f2a680c6fe0 2023-10-30T12:31:18.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.321+0000 7f2a66ffd640 1 -- 172.21.15.89:0/2669758442 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c0a8e40 con 0x7f2a680c6fe0 2023-10-30T12:31:18.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.321+0000 7f2a6e29c640 1 -- 172.21.15.89:0/2669758442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a680c6fe0 msgr2=0x7f2a68071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:18.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.321+0000 7f2a6e29c640 1 --2- 172.21.15.89:0/2669758442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a680c6fe0 0x7f2a68071260 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2a6806ce90 tx=0x7f2a5c01ec50 comp rx=0 tx=0).stop 2023-10-30T12:31:18.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.321+0000 7f2a6e29c640 1 -- 172.21.15.89:0/2669758442 shutdown_connections 2023-10-30T12:31:18.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.321+0000 7f2a6e29c640 1 --2- 172.21.15.89:0/2669758442 >> v2:172.21.15.192:3300/0 conn(0x7f2a680c5f60 0x7f2a680cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:18.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.321+0000 7f2a6e29c640 1 --2- 172.21.15.89:0/2669758442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a680c6fe0 0x7f2a68071260 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:18.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.321+0000 7f2a6e29c640 1 -- 172.21.15.89:0/2669758442 >> 172.21.15.89:0/2669758442 conn(0x7f2a6806c6e0 msgr2=0x7f2a68069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:18.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.325+0000 7f2a6e29c640 1 -- 172.21.15.89:0/2669758442 shutdown_connections 2023-10-30T12:31:18.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.325+0000 7f2a6e29c640 1 -- 172.21.15.89:0/2669758442 wait complete. 2023-10-30T12:31:18.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.325+0000 7f2a6e29c640 1 Processor -- start 2023-10-30T12:31:18.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.325+0000 7f2a6e29c640 1 -- start start 2023-10-30T12:31:18.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.325+0000 7f2a6e29c640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a680c5f60 0x7f2a68070eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:18.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.325+0000 7f2a6e29c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a680c6fe0 0x7f2a680c36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:18.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.325+0000 7f2a6e29c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a68153bf0 0x7f2a680c3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:18.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.325+0000 7f2a6e29c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a681f20f0 con 0x7f2a68153bf0 2023-10-30T12:31:18.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.325+0000 7f2a6e29c640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a681f23d0 con 0x7f2a680c6fe0 2023-10-30T12:31:18.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.325+0000 7f2a6e29c640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a681f26b0 con 0x7f2a680c5f60 2023-10-30T12:31:18.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.325+0000 7f2a6c839640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a68153bf0 0x7f2a680c3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:18.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.325+0000 7f2a6c839640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a68153bf0 0x7f2a680c3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38270/0 (socket says 172.21.15.89:38270) 2023-10-30T12:31:18.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.325+0000 7f2a6c839640 1 -- 172.21.15.89:0/2660966937 learned_addr learned my addr 172.21.15.89:0/2660966937 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:18.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.325+0000 7f2a677fe640 1 --2- 172.21.15.89:0/2660966937 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a680c6fe0 0x7f2a680c36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:18.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.325+0000 7f2a67fff640 1 --2- 172.21.15.89:0/2660966937 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a680c5f60 0x7f2a68070eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:18.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.325+0000 7f2a6c839640 1 -- 172.21.15.89:0/2660966937 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a680c5f60 msgr2=0x7f2a68070eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:18.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.325+0000 7f2a6c839640 1 --2- 172.21.15.89:0/2660966937 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a680c5f60 0x7f2a68070eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:18.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.325+0000 7f2a6c839640 1 -- 172.21.15.89:0/2660966937 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a680c6fe0 msgr2=0x7f2a680c36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:18.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.325+0000 7f2a6c839640 1 --2- 172.21.15.89:0/2660966937 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a680c6fe0 0x7f2a680c36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:18.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.325+0000 7f2a6c839640 1 -- 172.21.15.89:0/2660966937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a5c009420 con 0x7f2a68153bf0 2023-10-30T12:31:18.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.329+0000 7f2a6c839640 1 --2- 172.21.15.89:0/2660966937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a68153bf0 0x7f2a680c3c10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2a60007e40 tx=0x7f2a6000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:18.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.329+0000 7f2a64ff9640 1 -- 172.21.15.89:0/2660966937 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a60014620 con 0x7f2a68153bf0 2023-10-30T12:31:18.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.329+0000 7f2a64ff9640 1 -- 172.21.15.89:0/2660966937 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a6000ee90 con 0x7f2a68153bf0 2023-10-30T12:31:18.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.329+0000 7f2a64ff9640 1 -- 172.21.15.89:0/2660966937 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a600149e0 con 0x7f2a68153bf0 2023-10-30T12:31:18.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.329+0000 7f2a6e29c640 1 -- 172.21.15.89:0/2660966937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a681f2990 con 0x7f2a68153bf0 2023-10-30T12:31:18.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.329+0000 7f2a6e29c640 1 -- 172.21.15.89:0/2660966937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a6814dc70 con 0x7f2a68153bf0 2023-10-30T12:31:18.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.329+0000 7f2a6e29c640 1 -- 172.21.15.89:0/2660966937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a6804e170 con 0x7f2a68153bf0 2023-10-30T12:31:18.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.329+0000 7f2a64ff9640 1 -- 172.21.15.89:0/2660966937 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2a60014b40 con 0x7f2a68153bf0 2023-10-30T12:31:18.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.329+0000 7f2a64ff9640 1 --2- 172.21.15.89:0/2660966937 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2a540664a0 0x7f2a54068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:18.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.329+0000 7f2a64ff9640 1 -- 172.21.15.89:0/2660966937 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5832+0+0 (secure 0 0 0) 0x7f2a60084950 con 0x7f2a68153bf0 2023-10-30T12:31:18.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.329+0000 7f2a67fff640 1 --2- 172.21.15.89:0/2660966937 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2a540664a0 0x7f2a54068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:18.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.333+0000 7f2a67fff640 1 --2- 172.21.15.89:0/2660966937 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2a540664a0 0x7f2a54068960 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2a58000f70 tx=0x7f2a58007c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:18.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.333+0000 7f2a64ff9640 1 -- 172.21.15.89:0/2660966937 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2a60051c20 con 0x7f2a68153bf0 2023-10-30T12:31:18.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.513+0000 7f2a6e29c640 1 -- 172.21.15.89:0/2660966937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a681f2e40 con 0x7f2a68153bf0 2023-10-30T12:31:18.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.513+0000 7f2a64ff9640 1 -- 172.21.15.89:0/2660966937 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+6531 (secure 0 0 0) 0x7f2a60055240 con 0x7f2a68153bf0 2023-10-30T12:31:18.518 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:31:18.518 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:30:56.409108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4253":{"gid":4253,"name":"b","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6835/1289220491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1289220491},{"type":"v1","addr":"172.21.15.192:6835","nonce":1289220491}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:31:18.520 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 14 2023-10-30T12:31:18.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.513+0000 7f2a6e29c640 1 -- 172.21.15.89:0/2660966937 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2a540664a0 msgr2=0x7f2a54068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:18.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.513+0000 7f2a6e29c640 1 --2- 172.21.15.89:0/2660966937 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2a540664a0 0x7f2a54068960 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2a58000f70 tx=0x7f2a58007c10 comp rx=0 tx=0).stop 2023-10-30T12:31:18.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.513+0000 7f2a6e29c640 1 -- 172.21.15.89:0/2660966937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a68153bf0 msgr2=0x7f2a680c3c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:18.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.513+0000 7f2a6e29c640 1 --2- 172.21.15.89:0/2660966937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a68153bf0 0x7f2a680c3c10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2a60007e40 tx=0x7f2a6000e4b0 comp rx=0 tx=0).stop 2023-10-30T12:31:18.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.513+0000 7f2a6e29c640 1 -- 172.21.15.89:0/2660966937 shutdown_connections 2023-10-30T12:31:18.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.513+0000 7f2a6e29c640 1 --2- 172.21.15.89:0/2660966937 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2a540664a0 0x7f2a54068960 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:18.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.513+0000 7f2a6e29c640 1 --2- 172.21.15.89:0/2660966937 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a680c6fe0 0x7f2a680c36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:18.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.513+0000 7f2a6e29c640 1 --2- 172.21.15.89:0/2660966937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a68153bf0 0x7f2a680c3c10 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:18.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.513+0000 7f2a6e29c640 1 --2- 172.21.15.89:0/2660966937 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a680c5f60 0x7f2a68070eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:18.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.513+0000 7f2a6e29c640 1 -- 172.21.15.89:0/2660966937 >> 172.21.15.89:0/2660966937 conn(0x7f2a6806c6e0 msgr2=0x7f2a68158750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:18.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.513+0000 7f2a6e29c640 1 -- 172.21.15.89:0/2660966937 shutdown_connections 2023-10-30T12:31:18.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:18.517+0000 7f2a6e29c640 1 -- 172.21.15.89:0/2660966937 wait complete. 2023-10-30T12:31:18.538 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T12:31:20.540 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:31:20.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.685+0000 7fb616ed6640 1 Processor -- start 2023-10-30T12:31:20.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.685+0000 7fb616ed6640 1 -- start start 2023-10-30T12:31:20.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.685+0000 7fb616ed6640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb610153d70 0x7fb61006ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:20.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.685+0000 7fb616ed6640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb61006e470 0x7fb6100cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:20.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.685+0000 7fb616ed6640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb6100c8490 0x7fb6100cd340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:20.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.685+0000 7fb616ed6640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb61006d780 con 0x7fb61006e470 2023-10-30T12:31:20.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.685+0000 7fb616ed6640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb61006afa0 con 0x7fb6100c8490 2023-10-30T12:31:20.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.685+0000 7fb616ed6640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb610154150 con 0x7fb610153d70 2023-10-30T12:31:20.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.685+0000 7fb60ffff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb61006e470 0x7fb6100cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:20.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.685+0000 7fb60ffff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb61006e470 0x7fb6100cce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53494/0 (socket says 172.21.15.89:53494) 2023-10-30T12:31:20.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.685+0000 7fb60ffff640 1 -- 172.21.15.89:0/1029724573 learned_addr learned my addr 172.21.15.89:0/1029724573 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:20.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.685+0000 7fb614c72640 1 --2- 172.21.15.89:0/1029724573 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb610153d70 0x7fb61006ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:20.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.685+0000 7fb615473640 1 --2- 172.21.15.89:0/1029724573 >> v2:172.21.15.192:3300/0 conn(0x7fb6100c8490 0x7fb6100cd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:20.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.685+0000 7fb60ffff640 1 -- 172.21.15.89:0/1029724573 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb610153d70 msgr2=0x7fb61006ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:20.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.685+0000 7fb60ffff640 1 --2- 172.21.15.89:0/1029724573 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb610153d70 0x7fb61006ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:20.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.685+0000 7fb60ffff640 1 -- 172.21.15.89:0/1029724573 >> v2:172.21.15.192:3300/0 conn(0x7fb6100c8490 msgr2=0x7fb6100cd340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:20.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.685+0000 7fb60ffff640 1 --2- 172.21.15.89:0/1029724573 >> v2:172.21.15.192:3300/0 conn(0x7fb6100c8490 0x7fb6100cd340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:20.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.685+0000 7fb60ffff640 1 -- 172.21.15.89:0/1029724573 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb61006e030 con 0x7fb61006e470 2023-10-30T12:31:20.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.689+0000 7fb60ffff640 1 --2- 172.21.15.89:0/1029724573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb61006e470 0x7fb6100cce00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb6100b1c40 tx=0x7fb5fc01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c37d535223e064b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:20.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.689+0000 7fb60f7fe640 1 -- 172.21.15.89:0/1029724573 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc0aa5f0 con 0x7fb61006e470 2023-10-30T12:31:20.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.689+0000 7fb60f7fe640 1 -- 172.21.15.89:0/1029724573 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5fc0aa750 con 0x7fb61006e470 2023-10-30T12:31:20.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.689+0000 7fb60f7fe640 1 -- 172.21.15.89:0/1029724573 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc0aaae0 con 0x7fb61006e470 2023-10-30T12:31:20.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.689+0000 7fb616ed6640 1 -- 172.21.15.89:0/1029724573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb61006e470 msgr2=0x7fb6100cce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:20.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.689+0000 7fb616ed6640 1 --2- 172.21.15.89:0/1029724573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb61006e470 0x7fb6100cce00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb6100b1c40 tx=0x7fb5fc01c4d0 comp rx=0 tx=0).stop 2023-10-30T12:31:20.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.689+0000 7fb616ed6640 1 -- 172.21.15.89:0/1029724573 shutdown_connections 2023-10-30T12:31:20.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.689+0000 7fb616ed6640 1 --2- 172.21.15.89:0/1029724573 >> v2:172.21.15.192:3300/0 conn(0x7fb6100c8490 0x7fb6100cd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:20.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.689+0000 7fb616ed6640 1 --2- 172.21.15.89:0/1029724573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb61006e470 0x7fb6100cce00 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:20.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.689+0000 7fb616ed6640 1 --2- 172.21.15.89:0/1029724573 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb610153d70 0x7fb61006ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:20.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.689+0000 7fb616ed6640 1 -- 172.21.15.89:0/1029724573 >> 172.21.15.89:0/1029724573 conn(0x7fb610071ea0 msgr2=0x7fb610065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:20.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.689+0000 7fb616ed6640 1 -- 172.21.15.89:0/1029724573 shutdown_connections 2023-10-30T12:31:20.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.689+0000 7fb616ed6640 1 -- 172.21.15.89:0/1029724573 wait complete. 2023-10-30T12:31:20.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.693+0000 7fb616ed6640 1 Processor -- start 2023-10-30T12:31:20.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.693+0000 7fb616ed6640 1 -- start start 2023-10-30T12:31:20.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.693+0000 7fb616ed6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb61006e470 0x7fb61015b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:20.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.693+0000 7fb616ed6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6100c8490 0x7fb610156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:20.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.693+0000 7fb616ed6640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb610153d70 0x7fb6101f5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:20.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.693+0000 7fb616ed6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6101f6ce0 con 0x7fb6100c8490 2023-10-30T12:31:20.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.693+0000 7fb616ed6640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6100ccb60 con 0x7fb61006e470 2023-10-30T12:31:20.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.693+0000 7fb616ed6640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb610154730 con 0x7fb610153d70 2023-10-30T12:31:20.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.693+0000 7fb60ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6100c8490 0x7fb610156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:20.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.693+0000 7fb60ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6100c8490 0x7fb610156cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53506/0 (socket says 172.21.15.89:53506) 2023-10-30T12:31:20.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.693+0000 7fb60ffff640 1 -- 172.21.15.89:0/2047325258 learned_addr learned my addr 172.21.15.89:0/2047325258 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:20.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.693+0000 7fb614c72640 1 --2- 172.21.15.89:0/2047325258 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb61006e470 0x7fb61015b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:20.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.693+0000 7fb615473640 1 --2- 172.21.15.89:0/2047325258 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb610153d70 0x7fb6101f5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:20.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.693+0000 7fb60ffff640 1 -- 172.21.15.89:0/2047325258 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb610153d70 msgr2=0x7fb6101f5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:20.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.693+0000 7fb60ffff640 1 --2- 172.21.15.89:0/2047325258 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb610153d70 0x7fb6101f5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:20.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.693+0000 7fb60ffff640 1 -- 172.21.15.89:0/2047325258 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb61006e470 msgr2=0x7fb61015b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:20.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.693+0000 7fb60ffff640 1 --2- 172.21.15.89:0/2047325258 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb61006e470 0x7fb61015b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:20.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.693+0000 7fb60ffff640 1 -- 172.21.15.89:0/2047325258 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5fc0094d0 con 0x7fb6100c8490 2023-10-30T12:31:20.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.693+0000 7fb60ffff640 1 --2- 172.21.15.89:0/2047325258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6100c8490 0x7fb610156cf0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc0ac040 tx=0x7fb5fc01ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:20.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.693+0000 7fb60d7fa640 1 -- 172.21.15.89:0/2047325258 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc0aa5f0 con 0x7fb6100c8490 2023-10-30T12:31:20.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.693+0000 7fb60d7fa640 1 -- 172.21.15.89:0/2047325258 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5fc0b3800 con 0x7fb6100c8490 2023-10-30T12:31:20.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.693+0000 7fb616ed6640 1 -- 172.21.15.89:0/2047325258 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6101f6a00 con 0x7fb6100c8490 2023-10-30T12:31:20.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.693+0000 7fb60d7fa640 1 -- 172.21.15.89:0/2047325258 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc0b3b90 con 0x7fb6100c8490 2023-10-30T12:31:20.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.693+0000 7fb616ed6640 1 -- 172.21.15.89:0/2047325258 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb61015b530 con 0x7fb6100c8490 2023-10-30T12:31:20.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.693+0000 7fb616ed6640 1 -- 172.21.15.89:0/2047325258 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb61004e170 con 0x7fb6100c8490 2023-10-30T12:31:20.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.697+0000 7fb60d7fa640 1 -- 172.21.15.89:0/2047325258 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb5fc0b3cf0 con 0x7fb6100c8490 2023-10-30T12:31:20.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.697+0000 7fb60d7fa640 1 --2- 172.21.15.89:0/2047325258 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb5f40664a0 0x7fb5f4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:20.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.697+0000 7fb60d7fa640 1 -- 172.21.15.89:0/2047325258 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5832+0+0 (secure 0 0 0) 0x7fb5fc0a7070 con 0x7fb6100c8490 2023-10-30T12:31:20.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.697+0000 7fb614c72640 1 --2- 172.21.15.89:0/2047325258 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb5f40664a0 0x7fb5f4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:20.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.697+0000 7fb614c72640 1 --2- 172.21.15.89:0/2047325258 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb5f40664a0 0x7fb5f4068960 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb60400d8d0 tx=0x7fb604008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:20.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.701+0000 7fb60d7fa640 1 -- 172.21.15.89:0/2047325258 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb5fc0ebe20 con 0x7fb6100c8490 2023-10-30T12:31:20.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.877+0000 7fb616ed6640 1 -- 172.21.15.89:0/2047325258 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb610157360 con 0x7fb6100c8490 2023-10-30T12:31:20.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.881+0000 7fb60d7fa640 1 -- 172.21.15.89:0/2047325258 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+6531 (secure 0 0 0) 0x7fb5fc0ef440 con 0x7fb6100c8490 2023-10-30T12:31:20.884 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:31:20.885 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:30:56.409108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4253":{"gid":4253,"name":"b","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.192:6835/1289220491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1289220491},{"type":"v1","addr":"172.21.15.192:6835","nonce":1289220491}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:31:20.886 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 14 2023-10-30T12:31:20.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.881+0000 7fb616ed6640 1 -- 172.21.15.89:0/2047325258 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb5f40664a0 msgr2=0x7fb5f4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:20.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.881+0000 7fb616ed6640 1 --2- 172.21.15.89:0/2047325258 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb5f40664a0 0x7fb5f4068960 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb60400d8d0 tx=0x7fb604008040 comp rx=0 tx=0).stop 2023-10-30T12:31:20.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.881+0000 7fb616ed6640 1 -- 172.21.15.89:0/2047325258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6100c8490 msgr2=0x7fb610156cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:20.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.881+0000 7fb616ed6640 1 --2- 172.21.15.89:0/2047325258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6100c8490 0x7fb610156cf0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc0ac040 tx=0x7fb5fc01ad10 comp rx=0 tx=0).stop 2023-10-30T12:31:20.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.881+0000 7fb616ed6640 1 -- 172.21.15.89:0/2047325258 shutdown_connections 2023-10-30T12:31:20.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.881+0000 7fb616ed6640 1 --2- 172.21.15.89:0/2047325258 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb5f40664a0 0x7fb5f4068960 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:20.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.881+0000 7fb616ed6640 1 --2- 172.21.15.89:0/2047325258 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb610153d70 0x7fb6101f5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:20.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.881+0000 7fb616ed6640 1 --2- 172.21.15.89:0/2047325258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6100c8490 0x7fb610156cf0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:20.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.881+0000 7fb616ed6640 1 --2- 172.21.15.89:0/2047325258 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb61006e470 0x7fb61015b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:20.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.881+0000 7fb616ed6640 1 -- 172.21.15.89:0/2047325258 >> 172.21.15.89:0/2047325258 conn(0x7fb610071ea0 msgr2=0x7fb6100c7520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:20.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.881+0000 7fb616ed6640 1 -- 172.21.15.89:0/2047325258 shutdown_connections 2023-10-30T12:31:20.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:20.881+0000 7fb616ed6640 1 -- 172.21.15.89:0/2047325258 wait complete. 2023-10-30T12:31:20.905 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T12:31:22.905 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:31:23.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.045+0000 7f0f1306b640 1 Processor -- start 2023-10-30T12:31:23.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.045+0000 7f0f1306b640 1 -- start start 2023-10-30T12:31:23.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.045+0000 7f0f1306b640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0f0c0c6b80 0x7f0f0c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:23.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.045+0000 7f0f1306b640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0f0c0c6f60 0x7f0f0c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:23.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.045+0000 7f0f1306b640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f0c0c0a50 con 0x7f0f0c0c8430 2023-10-30T12:31:23.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.045+0000 7f0f1306b640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f0c06f6e0 con 0x7f0f0c0c6b80 2023-10-30T12:31:23.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.045+0000 7f0f1306b640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f0c06f9c0 con 0x7f0f0c0c6f60 2023-10-30T12:31:23.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.045+0000 7f0f10e07640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0f0c0c6b80 0x7f0f0c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:23.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.045+0000 7f0f10e07640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0f0c0c6b80 0x7f0f0c0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53508/0 (socket says 172.21.15.89:53508) 2023-10-30T12:31:23.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.045+0000 7f0f10e07640 1 -- 172.21.15.89:0/1473036179 learned_addr learned my addr 172.21.15.89:0/1473036179 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:23.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.045+0000 7f0f0bfff640 1 --2- 172.21.15.89:0/1473036179 >> v2:172.21.15.192:3300/0 conn(0x7f0f0c0c6f60 0x7f0f0c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:23.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.045+0000 7f0f10e07640 1 -- 172.21.15.89:0/1473036179 >> v2:172.21.15.192:3300/0 conn(0x7f0f0c0c6f60 msgr2=0x7f0f0c0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:23.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.045+0000 7f0f10e07640 1 --2- 172.21.15.89:0/1473036179 >> v2:172.21.15.192:3300/0 conn(0x7f0f0c0c6f60 0x7f0f0c0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:23.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.045+0000 7f0f10e07640 1 -- 172.21.15.89:0/1473036179 >> v1:172.21.15.192:6789/0 conn(0x7f0f0c0c8430 legacy=0x7f0f0c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:23.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.045+0000 7f0f10e07640 1 -- 172.21.15.89:0/1473036179 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f0c06fca0 con 0x7f0f0c0c6b80 2023-10-30T12:31:23.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.049+0000 7f0f10e07640 1 --2- 172.21.15.89:0/1473036179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f0c0c6b80 0x7f0f0c0bfec0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c159b90 tx=0x7f0f0001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd24535fc32e0f8c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:23.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.049+0000 7f0f0b7fe640 1 -- 172.21.15.89:0/1473036179 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f000ab610 con 0x7f0f0c0c6b80 2023-10-30T12:31:23.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.049+0000 7f0f0b7fe640 1 -- 172.21.15.89:0/1473036179 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f000ab770 con 0x7f0f0c0c6b80 2023-10-30T12:31:23.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.049+0000 7f0f0b7fe640 1 -- 172.21.15.89:0/1473036179 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f000abb00 con 0x7f0f0c0c6b80 2023-10-30T12:31:23.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.049+0000 7f0f1306b640 1 -- 172.21.15.89:0/1473036179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f0c0c6b80 msgr2=0x7f0f0c0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:23.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.049+0000 7f0f1306b640 1 --2- 172.21.15.89:0/1473036179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f0c0c6b80 0x7f0f0c0bfec0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c159b90 tx=0x7f0f0001c500 comp rx=0 tx=0).stop 2023-10-30T12:31:23.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.049+0000 7f0f1306b640 1 -- 172.21.15.89:0/1473036179 shutdown_connections 2023-10-30T12:31:23.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.049+0000 7f0f1306b640 1 --2- 172.21.15.89:0/1473036179 >> v2:172.21.15.192:3300/0 conn(0x7f0f0c0c6f60 0x7f0f0c0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:23.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.049+0000 7f0f1306b640 1 --2- 172.21.15.89:0/1473036179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f0c0c6b80 0x7f0f0c0bfec0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:23.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.049+0000 7f0f1306b640 1 -- 172.21.15.89:0/1473036179 >> 172.21.15.89:0/1473036179 conn(0x7f0f0c0c7910 msgr2=0x7f0f0c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:23.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.049+0000 7f0f1306b640 1 -- 172.21.15.89:0/1473036179 shutdown_connections 2023-10-30T12:31:23.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.049+0000 7f0f1306b640 1 -- 172.21.15.89:0/1473036179 wait complete. 2023-10-30T12:31:23.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.053+0000 7f0f1306b640 1 Processor -- start 2023-10-30T12:31:23.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.053+0000 7f0f1306b640 1 -- start start 2023-10-30T12:31:23.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.053+0000 7f0f1306b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f0c0c6b80 0x7f0f0c160610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:23.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.053+0000 7f0f1306b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f0c0c6f60 0x7f0f0c15f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:23.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.053+0000 7f0f1306b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f0c0c8430 0x7f0f0c15f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:23.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.053+0000 7f0f1306b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c0c92c0 con 0x7f0f0c0c8430 2023-10-30T12:31:23.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.053+0000 7f0f1306b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c0c8cb0 con 0x7f0f0c0c6b80 2023-10-30T12:31:23.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.053+0000 7f0f1306b640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c136c80 con 0x7f0f0c0c6f60 2023-10-30T12:31:23.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.053+0000 7f0f11608640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f0c0c8430 0x7f0f0c15f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:23.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.053+0000 7f0f11608640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f0c0c8430 0x7f0f0c15f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53516/0 (socket says 172.21.15.89:53516) 2023-10-30T12:31:23.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.053+0000 7f0f11608640 1 -- 172.21.15.89:0/1349635766 learned_addr learned my addr 172.21.15.89:0/1349635766 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:23.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.053+0000 7f0f10e07640 1 --2- 172.21.15.89:0/1349635766 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f0c0c6b80 0x7f0f0c160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:23.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.053+0000 7f0f0bfff640 1 --2- 172.21.15.89:0/1349635766 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f0c0c6f60 0x7f0f0c15f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:23.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.053+0000 7f0f11608640 1 -- 172.21.15.89:0/1349635766 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f0c0c6f60 msgr2=0x7f0f0c15f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:23.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.053+0000 7f0f11608640 1 --2- 172.21.15.89:0/1349635766 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f0c0c6f60 0x7f0f0c15f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:23.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.053+0000 7f0f11608640 1 -- 172.21.15.89:0/1349635766 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f0c0c6b80 msgr2=0x7f0f0c160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:23.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.053+0000 7f0f11608640 1 --2- 172.21.15.89:0/1349635766 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f0c0c6b80 0x7f0f0c160610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:23.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.053+0000 7f0f11608640 1 -- 172.21.15.89:0/1349635766 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f000094d0 con 0x7f0f0c0c8430 2023-10-30T12:31:23.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.053+0000 7f0f0bfff640 1 --2- 172.21.15.89:0/1349635766 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f0c0c6f60 0x7f0f0c15f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:31:23.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.053+0000 7f0f11608640 1 --2- 172.21.15.89:0/1349635766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f0c0c8430 0x7f0f0c15f8c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0f0400c730 tx=0x7f0f0400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:23.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.053+0000 7f0f097fa640 1 -- 172.21.15.89:0/1349635766 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f04003b80 con 0x7f0f0c0c8430 2023-10-30T12:31:23.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.053+0000 7f0f097fa640 1 -- 172.21.15.89:0/1349635766 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f04003ce0 con 0x7f0f0c0c8430 2023-10-30T12:31:23.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.053+0000 7f0f1306b640 1 -- 172.21.15.89:0/1349635766 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f0c136f60 con 0x7f0f0c0c8430 2023-10-30T12:31:23.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.053+0000 7f0f097fa640 1 -- 172.21.15.89:0/1349635766 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0400aa30 con 0x7f0f0c0c8430 2023-10-30T12:31:23.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.053+0000 7f0f1306b640 1 -- 172.21.15.89:0/1349635766 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f0c137240 con 0x7f0f0c0c8430 2023-10-30T12:31:23.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.053+0000 7f0f1306b640 1 -- 172.21.15.89:0/1349635766 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f0c0c9470 con 0x7f0f0c0c8430 2023-10-30T12:31:23.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.057+0000 7f0f097fa640 1 -- 172.21.15.89:0/1349635766 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0f0400ab90 con 0x7f0f0c0c8430 2023-10-30T12:31:23.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.057+0000 7f0f097fa640 1 --2- 172.21.15.89:0/1349635766 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0ef00664a0 0x7f0ef0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:23.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.057+0000 7f0f097fa640 1 -- 172.21.15.89:0/1349635766 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5890+0+0 (secure 0 0 0) 0x7f0f04083d10 con 0x7f0f0c0c8430 2023-10-30T12:31:23.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.057+0000 7f0f10e07640 1 --2- 172.21.15.89:0/1349635766 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0ef00664a0 0x7f0ef0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:23.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.057+0000 7f0f10e07640 1 --2- 172.21.15.89:0/1349635766 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0ef00664a0 0x7f0ef0068960 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0f000af000 tx=0x7f0f000b6040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:23.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.061+0000 7f0f097fa640 1 -- 172.21.15.89:0/1349635766 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0f0c0c9470 con 0x7f0f0c0c8430 2023-10-30T12:31:23.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.241+0000 7f0f1306b640 1 -- 172.21.15.89:0/1349635766 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f0c04e170 con 0x7f0f0c0c8430 2023-10-30T12:31:23.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.245+0000 7f0f097fa640 1 -- 172.21.15.89:0/1349635766 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f0f04089020 con 0x7f0f0c0c8430 2023-10-30T12:31:23.248 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:31:23.248 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:31:20.953518+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":2,"incarnation":15,"state":"up:replay","state_seq":1,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:31:23.249 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 15 2023-10-30T12:31:23.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.245+0000 7f0f1306b640 1 -- 172.21.15.89:0/1349635766 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0ef00664a0 msgr2=0x7f0ef0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:23.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.245+0000 7f0f1306b640 1 --2- 172.21.15.89:0/1349635766 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0ef00664a0 0x7f0ef0068960 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0f000af000 tx=0x7f0f000b6040 comp rx=0 tx=0).stop 2023-10-30T12:31:23.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.245+0000 7f0f1306b640 1 -- 172.21.15.89:0/1349635766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f0c0c8430 msgr2=0x7f0f0c15f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:23.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.245+0000 7f0f1306b640 1 --2- 172.21.15.89:0/1349635766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f0c0c8430 0x7f0f0c15f8c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0f0400c730 tx=0x7f0f0400cbf0 comp rx=0 tx=0).stop 2023-10-30T12:31:23.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.245+0000 7f0f1306b640 1 -- 172.21.15.89:0/1349635766 shutdown_connections 2023-10-30T12:31:23.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.245+0000 7f0f1306b640 1 --2- 172.21.15.89:0/1349635766 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0ef00664a0 0x7f0ef0068960 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:23.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.245+0000 7f0f1306b640 1 --2- 172.21.15.89:0/1349635766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f0c0c8430 0x7f0f0c15f8c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:23.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.245+0000 7f0f1306b640 1 --2- 172.21.15.89:0/1349635766 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f0c0c6f60 0x7f0f0c15f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:23.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.245+0000 7f0f1306b640 1 --2- 172.21.15.89:0/1349635766 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f0c0c6b80 0x7f0f0c160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:23.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.245+0000 7f0f1306b640 1 -- 172.21.15.89:0/1349635766 >> 172.21.15.89:0/1349635766 conn(0x7f0f0c0c7910 msgr2=0x7f0f0c0c0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:23.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.245+0000 7f0f1306b640 1 -- 172.21.15.89:0/1349635766 shutdown_connections 2023-10-30T12:31:23.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.245+0000 7f0f1306b640 1 -- 172.21.15.89:0/1349635766 wait complete. 2023-10-30T12:31:23.269 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2023-10-30T12:31:23.269 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:31:23.269 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:31:23.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.401+0000 7f87c8cc9640 1 Processor -- start 2023-10-30T12:31:23.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.401+0000 7f87c8cc9640 1 -- start start 2023-10-30T12:31:23.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c8cc9640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f87c40c6f60 0x7f87c41566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:23.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c8cc9640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87c40c0a50 con 0x7f87c40c6b80 2023-10-30T12:31:23.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c8cc9640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87c406f6e0 con 0x7f87c40c05d0 2023-10-30T12:31:23.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c8cc9640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87c406f9c0 con 0x7f87c40c6f60 2023-10-30T12:31:23.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c259c640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f87c40c6b80 0x7f87c40bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39068/0 (socket says 172.21.15.89:39068) 2023-10-30T12:31:23.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c259c640 1 -- 172.21.15.89:0/3006138525 learned_addr learned my addr 172.21.15.89:0/3006138525 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:31:23.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c2d9d640 1 --2- 172.21.15.89:0/3006138525 >> v2:172.21.15.192:3300/0 conn(0x7f87c40c6f60 0x7f87c41566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:23.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c159a640 1 -- 172.21.15.89:0/3006138525 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3047666062 0 0) 0x7f87c40c0a50 con 0x7f87c40c6b80 2023-10-30T12:31:23.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c159a640 1 -- 172.21.15.89:0/3006138525 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87ac003530 con 0x7f87c40c6b80 2023-10-30T12:31:23.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c159a640 1 -- 172.21.15.89:0/3006138525 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1061897248 0 0) 0x7f87c406f6e0 con 0x7f87c40c05d0 2023-10-30T12:31:23.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c159a640 1 -- 172.21.15.89:0/3006138525 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87c40c0a50 con 0x7f87c40c05d0 2023-10-30T12:31:23.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c159a640 1 -- 172.21.15.89:0/3006138525 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 633250032 0 0) 0x7f87ac003530 con 0x7f87c40c6b80 2023-10-30T12:31:23.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c159a640 1 -- 172.21.15.89:0/3006138525 >> v2:172.21.15.192:3300/0 conn(0x7f87c40c6f60 msgr2=0x7f87c41566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:23.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c159a640 1 --2- 172.21.15.89:0/3006138525 >> v2:172.21.15.192:3300/0 conn(0x7f87c40c6f60 0x7f87c41566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:23.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c159a640 1 -- 172.21.15.89:0/3006138525 >> v1:172.21.15.192:6789/0 conn(0x7f87c40c05d0 legacy=0x7f87c413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:23.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c159a640 1 -- 172.21.15.89:0/3006138525 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c406fca0 con 0x7f87c40c6b80 2023-10-30T12:31:23.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c159a640 1 -- 172.21.15.89:0/3006138525 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f87b4001680 con 0x7f87c40c6b80 2023-10-30T12:31:23.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c159a640 1 -- 172.21.15.89:0/3006138525 >> v1:172.21.15.89:6789/0 conn(0x7f87c40c6b80 legacy=0x7f87c40bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:23.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c159a640 1 --2- 172.21.15.89:0/3006138525 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87ac003b60 0x7f87ac003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:23.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c159a640 1 --2- 172.21.15.89:0/3006138525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87ac0047d0 0x7f87ac004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:23.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c159a640 1 --2- 172.21.15.89:0/3006138525 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87ac005190 0x7f87ac00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:23.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c159a640 1 -- 172.21.15.89:0/3006138525 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c406f9c0 con 0x7f87ac0047d0 2023-10-30T12:31:23.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c159a640 1 -- 172.21.15.89:0/3006138525 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87ac00c060 con 0x7f87ac003b60 2023-10-30T12:31:23.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c159a640 1 -- 172.21.15.89:0/3006138525 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87ac00e430 con 0x7f87ac005190 2023-10-30T12:31:23.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c1d9b640 1 --2- 172.21.15.89:0/3006138525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87ac0047d0 0x7f87ac004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:23.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c259c640 1 --2- 172.21.15.89:0/3006138525 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87ac003b60 0x7f87ac003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:23.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c2d9d640 1 --2- 172.21.15.89:0/3006138525 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87ac005190 0x7f87ac00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:23.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c1d9b640 1 -- 172.21.15.89:0/3006138525 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87ac005190 msgr2=0x7f87ac00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:23.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c1d9b640 1 --2- 172.21.15.89:0/3006138525 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87ac005190 0x7f87ac00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:23.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c1d9b640 1 -- 172.21.15.89:0/3006138525 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87ac003b60 msgr2=0x7f87ac003f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:23.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c1d9b640 1 --2- 172.21.15.89:0/3006138525 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87ac003b60 0x7f87ac003f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:23.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c1d9b640 1 -- 172.21.15.89:0/3006138525 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87ac003530 con 0x7f87ac0047d0 2023-10-30T12:31:23.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.405+0000 7f87c2d9d640 1 -- 172.21.15.89:0/3006138525 reap_dead start 2023-10-30T12:31:23.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.409+0000 7f87c1d9b640 1 --2- 172.21.15.89:0/3006138525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87ac0047d0 0x7f87ac004c50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f87c406c9a0 tx=0x7f87b801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b8ac1f6975eeb4f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:23.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.409+0000 7f87c159a640 1 -- 172.21.15.89:0/3006138525 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b80052d0 con 0x7f87ac0047d0 2023-10-30T12:31:23.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.409+0000 7f87c159a640 1 -- 172.21.15.89:0/3006138525 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b8005450 con 0x7f87ac0047d0 2023-10-30T12:31:23.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.409+0000 7f87c159a640 1 -- 172.21.15.89:0/3006138525 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b80a88b0 con 0x7f87ac0047d0 2023-10-30T12:31:23.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.409+0000 7f87c8cc9640 1 -- 172.21.15.89:0/3006138525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87ac0047d0 msgr2=0x7f87ac004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:23.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.409+0000 7f87c8cc9640 1 --2- 172.21.15.89:0/3006138525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87ac0047d0 0x7f87ac004c50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f87c406c9a0 tx=0x7f87b801ec50 comp rx=0 tx=0).stop 2023-10-30T12:31:23.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.409+0000 7f87c8cc9640 1 -- 172.21.15.89:0/3006138525 shutdown_connections 2023-10-30T12:31:23.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.409+0000 7f87c8cc9640 1 --2- 172.21.15.89:0/3006138525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87ac0047d0 0x7f87ac004c50 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:23.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.409+0000 7f87c8cc9640 1 -- 172.21.15.89:0/3006138525 >> 172.21.15.89:0/3006138525 conn(0x7f87c40c7910 msgr2=0x7f87c40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:23.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.409+0000 7f87c8cc9640 1 -- 172.21.15.89:0/3006138525 shutdown_connections 2023-10-30T12:31:23.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.409+0000 7f87c8cc9640 1 -- 172.21.15.89:0/3006138525 wait complete. 2023-10-30T12:31:23.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.409+0000 7f87c8cc9640 1 Processor -- start 2023-10-30T12:31:23.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.409+0000 7f87c8cc9640 1 -- start start 2023-10-30T12:31:23.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.413+0000 7f87c8cc9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87ac0047d0 0x7f87c40ca510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:23.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.413+0000 7f87c8cc9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87c413e250 0x7f87c406b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:23.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.413+0000 7f87c8cc9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87c41f20f0 0x7f87c41f2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:23.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.413+0000 7f87c8cc9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c40ae8c0 con 0x7f87c41f20f0 2023-10-30T12:31:23.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.413+0000 7f87c8cc9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c40cabc0 con 0x7f87ac0047d0 2023-10-30T12:31:23.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.413+0000 7f87c8cc9640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87c40b0d20 con 0x7f87c413e250 2023-10-30T12:31:23.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.413+0000 7f87c2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87c41f20f0 0x7f87c41f2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:23.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.413+0000 7f87c259c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87ac0047d0 0x7f87c40ca510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:23.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.413+0000 7f87c259c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87ac0047d0 0x7f87c40ca510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:32796/0 (socket says 172.21.15.89:32796) 2023-10-30T12:31:23.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.413+0000 7f87c259c640 1 -- 172.21.15.89:0/2506226030 learned_addr learned my addr 172.21.15.89:0/2506226030 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:23.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.413+0000 7f87c1d9b640 1 --2- 172.21.15.89:0/2506226030 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87c413e250 0x7f87c406b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:23.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.413+0000 7f87c2d9d640 1 -- 172.21.15.89:0/2506226030 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87c413e250 msgr2=0x7f87c406b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:23.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.413+0000 7f87c2d9d640 1 --2- 172.21.15.89:0/2506226030 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87c413e250 0x7f87c406b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:23.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.413+0000 7f87c2d9d640 1 -- 172.21.15.89:0/2506226030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87ac0047d0 msgr2=0x7f87c40ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:23.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.413+0000 7f87c2d9d640 1 --2- 172.21.15.89:0/2506226030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87ac0047d0 0x7f87c40ca510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:23.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.413+0000 7f87c2d9d640 1 -- 172.21.15.89:0/2506226030 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c41f2a90 con 0x7f87c41f20f0 2023-10-30T12:31:23.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.413+0000 7f87c2d9d640 1 --2- 172.21.15.89:0/2506226030 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87c41f20f0 0x7f87c41f2550 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f87c40b0760 tx=0x7f87bc0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:23.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.413+0000 7f87b2ffd640 1 -- 172.21.15.89:0/2506226030 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87bc00e070 con 0x7f87c41f20f0 2023-10-30T12:31:23.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.413+0000 7f87b2ffd640 1 -- 172.21.15.89:0/2506226030 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87bc011db0 con 0x7f87c41f20f0 2023-10-30T12:31:23.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.413+0000 7f87c8cc9640 1 -- 172.21.15.89:0/2506226030 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87c40b0410 con 0x7f87c41f20f0 2023-10-30T12:31:23.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.413+0000 7f87b2ffd640 1 -- 172.21.15.89:0/2506226030 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87bc0039a0 con 0x7f87c41f20f0 2023-10-30T12:31:23.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.413+0000 7f87c8cc9640 1 -- 172.21.15.89:0/2506226030 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87c40afb70 con 0x7f87c41f20f0 2023-10-30T12:31:23.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.413+0000 7f87c8cc9640 1 -- 172.21.15.89:0/2506226030 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87c41f1e10 con 0x7f87c41f20f0 2023-10-30T12:31:23.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.413+0000 7f87b2ffd640 1 -- 172.21.15.89:0/2506226030 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f87bc003b00 con 0x7f87c41f20f0 2023-10-30T12:31:23.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.417+0000 7f87b2ffd640 1 --2- 172.21.15.89:0/2506226030 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f87a40664d0 0x7f87a4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:23.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.417+0000 7f87b2ffd640 1 -- 172.21.15.89:0/2506226030 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5890+0+0 (secure 0 0 0) 0x7f87bc084460 con 0x7f87c41f20f0 2023-10-30T12:31:23.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.417+0000 7f87c259c640 1 --2- 172.21.15.89:0/2506226030 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f87a40664d0 0x7f87a4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:23.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.417+0000 7f87c259c640 1 --2- 172.21.15.89:0/2506226030 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f87a40664d0 0x7f87a4068990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f87b4001b70 tx=0x7f87b4001b00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:23.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.421+0000 7f87b2ffd640 1 -- 172.21.15.89:0/2506226030 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f87c41f1e10 con 0x7f87c41f20f0 2023-10-30T12:31:23.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.601+0000 7f87c8cc9640 1 -- 172.21.15.89:0/2506226030 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87c41f1e10 con 0x7f87c41f20f0 2023-10-30T12:31:23.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.601+0000 7f87b2ffd640 1 -- 172.21.15.89:0/2506226030 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f87c41f1e10 con 0x7f87c41f20f0 2023-10-30T12:31:23.605 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:31:23.605 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:31:20.953518+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":2,"incarnation":15,"state":"up:replay","state_seq":1,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:31:23.605 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 15 2023-10-30T12:31:23.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.601+0000 7f87c8cc9640 1 -- 172.21.15.89:0/2506226030 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f87a40664d0 msgr2=0x7f87a4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:23.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.601+0000 7f87c8cc9640 1 --2- 172.21.15.89:0/2506226030 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f87a40664d0 0x7f87a4068990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f87b4001b70 tx=0x7f87b4001b00 comp rx=0 tx=0).stop 2023-10-30T12:31:23.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.601+0000 7f87c8cc9640 1 -- 172.21.15.89:0/2506226030 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87c41f20f0 msgr2=0x7f87c41f2550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:23.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.601+0000 7f87c8cc9640 1 --2- 172.21.15.89:0/2506226030 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87c41f20f0 0x7f87c41f2550 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f87c40b0760 tx=0x7f87bc0115a0 comp rx=0 tx=0).stop 2023-10-30T12:31:23.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.601+0000 7f87c8cc9640 1 -- 172.21.15.89:0/2506226030 shutdown_connections 2023-10-30T12:31:23.607 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.601+0000 7f87c8cc9640 1 --2- 172.21.15.89:0/2506226030 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f87a40664d0 0x7f87a4068990 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:23.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.601+0000 7f87c8cc9640 1 --2- 172.21.15.89:0/2506226030 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87c41f20f0 0x7f87c41f2550 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:23.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.601+0000 7f87c8cc9640 1 --2- 172.21.15.89:0/2506226030 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87c413e250 0x7f87c406b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:23.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.601+0000 7f87c8cc9640 1 --2- 172.21.15.89:0/2506226030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87ac0047d0 0x7f87c40ca510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:23.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.601+0000 7f87c8cc9640 1 -- 172.21.15.89:0/2506226030 >> 172.21.15.89:0/2506226030 conn(0x7f87c40c7910 msgr2=0x7f87c40bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:23.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.601+0000 7f87c8cc9640 1 -- 172.21.15.89:0/2506226030 shutdown_connections 2023-10-30T12:31:23.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:23.601+0000 7f87c8cc9640 1 -- 172.21.15.89:0/2506226030 wait complete. 2023-10-30T12:31:23.626 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 15, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:50.729245+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 15, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4242": { "addr": "172.21.15.89:6837/2053475104", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6836", "nonce": 2053475104, "type": "v2" }, { "addr": "172.21.15.89:6837", "nonce": 2053475104, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4242, "incarnation": 15, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:replay", "state_seq": 1 }, "gid_4262": { "addr": "172.21.15.192:6839/3339589587", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6838", "nonce": 3339589587, "type": "v2" }, { "addr": "172.21.15.192:6839", "nonce": 3339589587, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 9, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4318": { "addr": "172.21.15.192:6837/1002124471", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6836", "nonce": 1002124471, "type": "v2" }, { "addr": "172.21.15.192:6837", "nonce": 1002124471, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4318, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:31:20.953518+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4318, "mds_1": 4262, "mds_2": 4242 } } } ], "standbys": [ { "addr": "172.21.15.89:6834/2909790173", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6832", "nonce": 2909790173, "type": "v2" }, { "addr": "172.21.15.89:6834", "nonce": 2909790173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.89:6835/2387357615", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6833", "nonce": 2387357615, "type": "v2" }, { "addr": "172.21.15.89:6835", "nonce": 2387357615, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4239, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:31:25.627 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:31:25.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.768+0000 7fd1f3dd3640 1 Processor -- start 2023-10-30T12:31:25.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.768+0000 7fd1f3dd3640 1 -- start start 2023-10-30T12:31:25.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.772+0000 7fd1f3dd3640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd1ec150920 0x7fd1ec06d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:25.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.772+0000 7fd1f3dd3640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1ec0c68b0 con 0x7fd1ec0c2570 2023-10-30T12:31:25.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.772+0000 7fd1f3dd3640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1ec150d00 con 0x7fd1ec0cb7e0 2023-10-30T12:31:25.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.772+0000 7fd1f3dd3640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1ec06d960 con 0x7fd1ec150920 2023-10-30T12:31:25.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.772+0000 7fd1f1b6f640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd1ec150920 0x7fd1ec06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:25.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.772+0000 7fd1f1b6f640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd1ec150920 0x7fd1ec06d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53536/0 (socket says 172.21.15.89:53536) 2023-10-30T12:31:25.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.772+0000 7fd1f1b6f640 1 -- 172.21.15.89:0/979755758 learned_addr learned my addr 172.21.15.89:0/979755758 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:25.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.772+0000 7fd1f136e640 1 --1- 172.21.15.89:0/979755758 >> v1:172.21.15.89:6789/0 conn(0x7fd1ec0c2570 0x7fd1ec0cb0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39076/0 (socket says 172.21.15.89:39076) 2023-10-30T12:31:25.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.772+0000 7fd1f1b6f640 1 -- 172.21.15.89:0/979755758 >> v1:172.21.15.192:6789/0 conn(0x7fd1ec0cb7e0 legacy=0x7fd1ec156680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:25.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.772+0000 7fd1f1b6f640 1 -- 172.21.15.89:0/979755758 >> v1:172.21.15.89:6789/0 conn(0x7fd1ec0c2570 legacy=0x7fd1ec0cb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:25.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.772+0000 7fd1f1b6f640 1 -- 172.21.15.89:0/979755758 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1ec0c2e90 con 0x7fd1ec150920 2023-10-30T12:31:25.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.772+0000 7fd1f0b6d640 1 -- 172.21.15.89:0/979755758 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1305365540 0 0) 0x7fd1ec0c68b0 con 0x7fd1ec0c2570 2023-10-30T12:31:25.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.776+0000 7fd1f1b6f640 1 --2- 172.21.15.89:0/979755758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1ec150920 0x7fd1ec06d420 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd1e8009870 tx=0x7fd1e801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=196f297ebc5e5774 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:25.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.776+0000 7fd1f0b6d640 1 -- 172.21.15.89:0/979755758 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e80ab5f0 con 0x7fd1ec150920 2023-10-30T12:31:25.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.776+0000 7fd1f0b6d640 1 -- 172.21.15.89:0/979755758 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1e80ab750 con 0x7fd1ec150920 2023-10-30T12:31:25.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.776+0000 7fd1f0b6d640 1 -- 172.21.15.89:0/979755758 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e80abae0 con 0x7fd1ec150920 2023-10-30T12:31:25.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.776+0000 7fd1f3dd3640 1 -- 172.21.15.89:0/979755758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1ec150920 msgr2=0x7fd1ec06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:25.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.776+0000 7fd1f3dd3640 1 --2- 172.21.15.89:0/979755758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1ec150920 0x7fd1ec06d420 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd1e8009870 tx=0x7fd1e801c500 comp rx=0 tx=0).stop 2023-10-30T12:31:25.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.776+0000 7fd1f3dd3640 1 -- 172.21.15.89:0/979755758 shutdown_connections 2023-10-30T12:31:25.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.776+0000 7fd1f3dd3640 1 --2- 172.21.15.89:0/979755758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1ec150920 0x7fd1ec06d420 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:25.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.776+0000 7fd1f3dd3640 1 -- 172.21.15.89:0/979755758 >> 172.21.15.89:0/979755758 conn(0x7fd1ec06dfd0 msgr2=0x7fd1ec06e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:25.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.776+0000 7fd1f3dd3640 1 -- 172.21.15.89:0/979755758 shutdown_connections 2023-10-30T12:31:25.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.776+0000 7fd1f3dd3640 1 -- 172.21.15.89:0/979755758 wait complete. 2023-10-30T12:31:25.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.776+0000 7fd1f3dd3640 1 Processor -- start 2023-10-30T12:31:25.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.776+0000 7fd1f3dd3640 1 -- start start 2023-10-30T12:31:25.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.776+0000 7fd1f3dd3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1ec0c2570 0x7fd1ec155350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:25.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.776+0000 7fd1f3dd3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1ec0cb7e0 0x7fd1ec14b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:25.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.776+0000 7fd1f3dd3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1ec150920 0x7fd1ec1f5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:25.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.776+0000 7fd1f3dd3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ec154f00 con 0x7fd1ec0c2570 2023-10-30T12:31:25.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.776+0000 7fd1f3dd3640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ec14c5e0 con 0x7fd1ec0cb7e0 2023-10-30T12:31:25.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.776+0000 7fd1f3dd3640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ec1588f0 con 0x7fd1ec150920 2023-10-30T12:31:25.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.780+0000 7fd1f1b6f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1ec0c2570 0x7fd1ec155350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:25.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.780+0000 7fd1f1b6f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1ec0c2570 0x7fd1ec155350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53552/0 (socket says 172.21.15.89:53552) 2023-10-30T12:31:25.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.780+0000 7fd1f1b6f640 1 -- 172.21.15.89:0/3141320482 learned_addr learned my addr 172.21.15.89:0/3141320482 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:25.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.780+0000 7fd1f136e640 1 --2- 172.21.15.89:0/3141320482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1ec0cb7e0 0x7fd1ec14b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:25.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.780+0000 7fd1f2370640 1 --2- 172.21.15.89:0/3141320482 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1ec150920 0x7fd1ec1f5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:25.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.780+0000 7fd1f1b6f640 1 -- 172.21.15.89:0/3141320482 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1ec150920 msgr2=0x7fd1ec1f5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:25.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.780+0000 7fd1f1b6f640 1 --2- 172.21.15.89:0/3141320482 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1ec150920 0x7fd1ec1f5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:25.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.780+0000 7fd1f1b6f640 1 -- 172.21.15.89:0/3141320482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1ec0cb7e0 msgr2=0x7fd1ec14b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:25.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.780+0000 7fd1f1b6f640 1 --2- 172.21.15.89:0/3141320482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1ec0cb7e0 0x7fd1ec14b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:25.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.780+0000 7fd1f1b6f640 1 -- 172.21.15.89:0/3141320482 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e80094d0 con 0x7fd1ec0c2570 2023-10-30T12:31:25.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.780+0000 7fd1f1b6f640 1 --2- 172.21.15.89:0/3141320482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1ec0c2570 0x7fd1ec155350 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd1e80ad040 tx=0x7fd1e801ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:25.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.780+0000 7fd1e27fc640 1 -- 172.21.15.89:0/3141320482 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e80b4800 con 0x7fd1ec0c2570 2023-10-30T12:31:25.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.780+0000 7fd1e27fc640 1 -- 172.21.15.89:0/3141320482 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1e80b4980 con 0x7fd1ec0c2570 2023-10-30T12:31:25.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.780+0000 7fd1e27fc640 1 -- 172.21.15.89:0/3141320482 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e80b4d30 con 0x7fd1ec0c2570 2023-10-30T12:31:25.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.780+0000 7fd1f3dd3640 1 -- 172.21.15.89:0/3141320482 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1ec14c880 con 0x7fd1ec0c2570 2023-10-30T12:31:25.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.780+0000 7fd1f3dd3640 1 -- 172.21.15.89:0/3141320482 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1ec143240 con 0x7fd1ec0c2570 2023-10-30T12:31:25.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.780+0000 7fd1f3dd3640 1 -- 172.21.15.89:0/3141320482 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1ec155070 con 0x7fd1ec0c2570 2023-10-30T12:31:25.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.780+0000 7fd1e27fc640 1 -- 172.21.15.89:0/3141320482 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd1e80b3380 con 0x7fd1ec0c2570 2023-10-30T12:31:25.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.784+0000 7fd1e27fc640 1 --2- 172.21.15.89:0/3141320482 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd1d00664a0 0x7fd1d0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:25.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.784+0000 7fd1e27fc640 1 -- 172.21.15.89:0/3141320482 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5890+0+0 (secure 0 0 0) 0x7fd1e80a8050 con 0x7fd1ec0c2570 2023-10-30T12:31:25.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.784+0000 7fd1f136e640 1 --2- 172.21.15.89:0/3141320482 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd1d00664a0 0x7fd1d0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:25.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.784+0000 7fd1f136e640 1 --2- 172.21.15.89:0/3141320482 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd1d00664a0 0x7fd1d0068960 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec0ca240 tx=0x7fd1dc007b60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:25.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.788+0000 7fd1e27fc640 1 -- 172.21.15.89:0/3141320482 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd1ec155070 con 0x7fd1ec0c2570 2023-10-30T12:31:25.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.968+0000 7fd1f3dd3640 1 -- 172.21.15.89:0/3141320482 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1ec155040 con 0x7fd1ec0c2570 2023-10-30T12:31:25.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.968+0000 7fd1e27fc640 1 -- 172.21.15.89:0/3141320482 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+5739 (secure 0 0 0) 0x7fd1e80eb820 con 0x7fd1ec0c2570 2023-10-30T12:31:25.976 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:31:25.976 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:31:20.953518+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":2,"incarnation":15,"state":"up:replay","state_seq":1,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:31:25.977 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 15 2023-10-30T12:31:25.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.972+0000 7fd1f3dd3640 1 -- 172.21.15.89:0/3141320482 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd1d00664a0 msgr2=0x7fd1d0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:25.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.972+0000 7fd1f3dd3640 1 --2- 172.21.15.89:0/3141320482 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd1d00664a0 0x7fd1d0068960 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec0ca240 tx=0x7fd1dc007b60 comp rx=0 tx=0).stop 2023-10-30T12:31:25.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.972+0000 7fd1f3dd3640 1 -- 172.21.15.89:0/3141320482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1ec0c2570 msgr2=0x7fd1ec155350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:25.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.972+0000 7fd1f3dd3640 1 --2- 172.21.15.89:0/3141320482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1ec0c2570 0x7fd1ec155350 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd1e80ad040 tx=0x7fd1e801ad40 comp rx=0 tx=0).stop 2023-10-30T12:31:25.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.972+0000 7fd1f3dd3640 1 -- 172.21.15.89:0/3141320482 shutdown_connections 2023-10-30T12:31:25.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.972+0000 7fd1f3dd3640 1 --2- 172.21.15.89:0/3141320482 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd1d00664a0 0x7fd1d0068960 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:25.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.972+0000 7fd1f3dd3640 1 --2- 172.21.15.89:0/3141320482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1ec0cb7e0 0x7fd1ec14b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:25.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.972+0000 7fd1f3dd3640 1 --2- 172.21.15.89:0/3141320482 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd1ec150920 0x7fd1ec1f5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:25.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.972+0000 7fd1f3dd3640 1 --2- 172.21.15.89:0/3141320482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1ec0c2570 0x7fd1ec155350 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:25.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.972+0000 7fd1f3dd3640 1 -- 172.21.15.89:0/3141320482 >> 172.21.15.89:0/3141320482 conn(0x7fd1ec06dfd0 msgr2=0x7fd1ec14fb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:25.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.972+0000 7fd1f3dd3640 1 -- 172.21.15.89:0/3141320482 shutdown_connections 2023-10-30T12:31:25.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:25.972+0000 7fd1f3dd3640 1 -- 172.21.15.89:0/3141320482 wait complete. 2023-10-30T12:31:25.999 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:31:28.000 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:31:28.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe81492a640 1 Processor -- start 2023-10-30T12:31:28.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe81492a640 1 -- start start 2023-10-30T12:31:28.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe81492a640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe8100c6bd0 0x7fe81006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:28.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe81492a640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe810066cf0 con 0x7fe81006f7a0 2023-10-30T12:31:28.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe81492a640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8100c70d0 con 0x7fe810071110 2023-10-30T12:31:28.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe81492a640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8100700c0 con 0x7fe8100c6bd0 2023-10-30T12:31:28.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe80e59c640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe81006f7a0 0x7fe810069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39090/0 (socket says 172.21.15.89:39090) 2023-10-30T12:31:28.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe80e59c640 1 -- 172.21.15.89:0/2704308712 learned_addr learned my addr 172.21.15.89:0/2704308712 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:31:28.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe80dd9b640 1 --2- 172.21.15.89:0/2704308712 >> v2:172.21.15.192:3300/0 conn(0x7fe8100c6bd0 0x7fe81006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:28.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe80d59a640 1 -- 172.21.15.89:0/2704308712 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398555232 0 0) 0x7fe810066cf0 con 0x7fe81006f7a0 2023-10-30T12:31:28.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe80d59a640 1 -- 172.21.15.89:0/2704308712 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7f8003530 con 0x7fe81006f7a0 2023-10-30T12:31:28.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe80d59a640 1 -- 172.21.15.89:0/2704308712 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 150671375 0 0) 0x7fe7f8003530 con 0x7fe81006f7a0 2023-10-30T12:31:28.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe80d59a640 1 -- 172.21.15.89:0/2704308712 >> v2:172.21.15.192:3300/0 conn(0x7fe8100c6bd0 msgr2=0x7fe81006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:28.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe80d59a640 1 --2- 172.21.15.89:0/2704308712 >> v2:172.21.15.192:3300/0 conn(0x7fe8100c6bd0 0x7fe81006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:28.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe80d59a640 1 -- 172.21.15.89:0/2704308712 >> v1:172.21.15.192:6789/0 conn(0x7fe810071110 legacy=0x7fe810158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:28.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe80d59a640 1 -- 172.21.15.89:0/2704308712 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8100c7390 con 0x7fe81006f7a0 2023-10-30T12:31:28.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe80d59a640 1 -- 172.21.15.89:0/2704308712 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7fe800001680 con 0x7fe81006f7a0 2023-10-30T12:31:28.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe80d59a640 1 -- 172.21.15.89:0/2704308712 >> v1:172.21.15.89:6789/0 conn(0x7fe81006f7a0 legacy=0x7fe810069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:28.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe80d59a640 1 --2- 172.21.15.89:0/2704308712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7f8003980 0x7fe7f8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:28.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe80d59a640 1 --2- 172.21.15.89:0/2704308712 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7f80047d0 0x7fe7f8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:28.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe80d59a640 1 --2- 172.21.15.89:0/2704308712 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7f8005190 0x7fe7f800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:28.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe80d59a640 1 -- 172.21.15.89:0/2704308712 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8100700c0 con 0x7fe7f8003980 2023-10-30T12:31:28.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe80d59a640 1 -- 172.21.15.89:0/2704308712 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f800c040 con 0x7fe7f8005190 2023-10-30T12:31:28.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe80d59a640 1 -- 172.21.15.89:0/2704308712 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f800c200 con 0x7fe7f80047d0 2023-10-30T12:31:28.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe80e59c640 1 --2- 172.21.15.89:0/2704308712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7f8003980 0x7fe7f8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:28.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe80ed9d640 1 --2- 172.21.15.89:0/2704308712 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7f8005190 0x7fe7f800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:28.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe80dd9b640 1 --2- 172.21.15.89:0/2704308712 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7f80047d0 0x7fe7f8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:28.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe80e59c640 1 -- 172.21.15.89:0/2704308712 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7f80047d0 msgr2=0x7fe7f8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:28.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe80e59c640 1 --2- 172.21.15.89:0/2704308712 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe7f80047d0 0x7fe7f8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:28.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe80e59c640 1 -- 172.21.15.89:0/2704308712 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7f8005190 msgr2=0x7fe7f800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:28.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe80e59c640 1 --2- 172.21.15.89:0/2704308712 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7f8005190 0x7fe7f800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:28.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe80e59c640 1 -- 172.21.15.89:0/2704308712 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f8003530 con 0x7fe7f8003980 2023-10-30T12:31:28.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe80ed9d640 1 --2- 172.21.15.89:0/2704308712 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7f8005190 0x7fe7f800b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:31:28.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.136+0000 7fe80ed9d640 1 -- 172.21.15.89:0/2704308712 reap_dead start 2023-10-30T12:31:28.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.140+0000 7fe80e59c640 1 --2- 172.21.15.89:0/2704308712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7f8003980 0x7fe7f8003da0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe80000c240 tx=0x7fe80001f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2716e14501e41857 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:28.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.140+0000 7fe80d59a640 1 -- 172.21.15.89:0/2704308712 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe800005ae0 con 0x7fe7f8003980 2023-10-30T12:31:28.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.140+0000 7fe80d59a640 1 -- 172.21.15.89:0/2704308712 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8000ab070 con 0x7fe7f8003980 2023-10-30T12:31:28.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.140+0000 7fe80d59a640 1 -- 172.21.15.89:0/2704308712 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe800007070 con 0x7fe7f8003980 2023-10-30T12:31:28.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.140+0000 7fe81492a640 1 -- 172.21.15.89:0/2704308712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7f8003980 msgr2=0x7fe7f8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:28.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.140+0000 7fe81492a640 1 --2- 172.21.15.89:0/2704308712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7f8003980 0x7fe7f8003da0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe80000c240 tx=0x7fe80001f3b0 comp rx=0 tx=0).stop 2023-10-30T12:31:28.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.140+0000 7fe81492a640 1 -- 172.21.15.89:0/2704308712 shutdown_connections 2023-10-30T12:31:28.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.140+0000 7fe81492a640 1 --2- 172.21.15.89:0/2704308712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7f8003980 0x7fe7f8003da0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:28.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.140+0000 7fe81492a640 1 -- 172.21.15.89:0/2704308712 >> 172.21.15.89:0/2704308712 conn(0x7fe8100c7910 msgr2=0x7fe8100c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:28.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.140+0000 7fe81492a640 1 -- 172.21.15.89:0/2704308712 shutdown_connections 2023-10-30T12:31:28.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.140+0000 7fe81492a640 1 -- 172.21.15.89:0/2704308712 wait complete. 2023-10-30T12:31:28.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.144+0000 7fe81492a640 1 Processor -- start 2023-10-30T12:31:28.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.144+0000 7fe81492a640 1 -- start start 2023-10-30T12:31:28.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.144+0000 7fe81492a640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7f8003980 0x7fe81015acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:28.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.144+0000 7fe81492a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8100ca3d0 0x7fe810155300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:28.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.144+0000 7fe81492a640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe810155840 0x7fe81014f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:28.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.144+0000 7fe81492a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe810157d10 con 0x7fe8100ca3d0 2023-10-30T12:31:28.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.144+0000 7fe81492a640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe810158300 con 0x7fe7f8003980 2023-10-30T12:31:28.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.144+0000 7fe81492a640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe810159940 con 0x7fe810155840 2023-10-30T12:31:28.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.144+0000 7fe80dd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8100ca3d0 0x7fe810155300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:28.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.144+0000 7fe80dd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8100ca3d0 0x7fe810155300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53564/0 (socket says 172.21.15.89:53564) 2023-10-30T12:31:28.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.144+0000 7fe80dd9b640 1 -- 172.21.15.89:0/813422467 learned_addr learned my addr 172.21.15.89:0/813422467 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:28.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.144+0000 7fe80e59c640 1 --2- 172.21.15.89:0/813422467 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7f8003980 0x7fe81015acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:28.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.144+0000 7fe80ed9d640 1 --2- 172.21.15.89:0/813422467 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe810155840 0x7fe81014f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:28.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.144+0000 7fe80dd9b640 1 -- 172.21.15.89:0/813422467 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe810155840 msgr2=0x7fe81014f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:28.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.144+0000 7fe80dd9b640 1 --2- 172.21.15.89:0/813422467 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe810155840 0x7fe81014f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:28.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.144+0000 7fe80dd9b640 1 -- 172.21.15.89:0/813422467 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7f8003980 msgr2=0x7fe81015acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:28.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.144+0000 7fe80dd9b640 1 --2- 172.21.15.89:0/813422467 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7f8003980 0x7fe81015acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:28.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.144+0000 7fe80dd9b640 1 -- 172.21.15.89:0/813422467 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe810154d00 con 0x7fe8100ca3d0 2023-10-30T12:31:28.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.144+0000 7fe80e59c640 1 --2- 172.21.15.89:0/813422467 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7f8003980 0x7fe81015acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:31:28.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.144+0000 7fe80ed9d640 1 --2- 172.21.15.89:0/813422467 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe810155840 0x7fe81014f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:31:28.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.144+0000 7fe80dd9b640 1 --2- 172.21.15.89:0/813422467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8100ca3d0 0x7fe810155300 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe804011530 tx=0x7fe8040119f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:28.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.144+0000 7fe7feffd640 1 -- 172.21.15.89:0/813422467 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe804004440 con 0x7fe8100ca3d0 2023-10-30T12:31:28.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.144+0000 7fe7feffd640 1 -- 172.21.15.89:0/813422467 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe804003620 con 0x7fe8100ca3d0 2023-10-30T12:31:28.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.144+0000 7fe7feffd640 1 -- 172.21.15.89:0/813422467 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8040039b0 con 0x7fe8100ca3d0 2023-10-30T12:31:28.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.144+0000 7fe81492a640 1 -- 172.21.15.89:0/813422467 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe810154a30 con 0x7fe8100ca3d0 2023-10-30T12:31:28.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.144+0000 7fe81492a640 1 -- 172.21.15.89:0/813422467 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8101588a0 con 0x7fe8100ca3d0 2023-10-30T12:31:28.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.148+0000 7fe81492a640 1 -- 172.21.15.89:0/813422467 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe810073cf0 con 0x7fe8100ca3d0 2023-10-30T12:31:28.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.148+0000 7fe7feffd640 1 -- 172.21.15.89:0/813422467 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe804003b10 con 0x7fe8100ca3d0 2023-10-30T12:31:28.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.148+0000 7fe7feffd640 1 --2- 172.21.15.89:0/813422467 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe7f00664a0 0x7fe7f0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:28.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.148+0000 7fe7feffd640 1 -- 172.21.15.89:0/813422467 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5890+0+0 (secure 0 0 0) 0x7fe8040841e0 con 0x7fe8100ca3d0 2023-10-30T12:31:28.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.148+0000 7fe80e59c640 1 --2- 172.21.15.89:0/813422467 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe7f00664a0 0x7fe7f0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:28.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.152+0000 7fe80e59c640 1 --2- 172.21.15.89:0/813422467 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe7f00664a0 0x7fe7f0068960 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe8000af000 tx=0x7fe8000aa000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:28.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.152+0000 7fe7feffd640 1 -- 172.21.15.89:0/813422467 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe804051470 con 0x7fe8100ca3d0 2023-10-30T12:31:28.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.336+0000 7fe81492a640 1 -- 172.21.15.89:0/813422467 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe81004e170 con 0x7fe8100ca3d0 2023-10-30T12:31:28.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.336+0000 7fe7feffd640 1 -- 172.21.15.89:0/813422467 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe804054a90 con 0x7fe8100ca3d0 2023-10-30T12:31:28.341 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:31:28.342 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:31:25.990848+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":2,"incarnation":15,"state":"up:resolve","state_seq":42,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:31:28.342 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 16 2023-10-30T12:31:28.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.336+0000 7fe81492a640 1 -- 172.21.15.89:0/813422467 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe7f00664a0 msgr2=0x7fe7f0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:28.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.336+0000 7fe81492a640 1 --2- 172.21.15.89:0/813422467 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe7f00664a0 0x7fe7f0068960 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe8000af000 tx=0x7fe8000aa000 comp rx=0 tx=0).stop 2023-10-30T12:31:28.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.336+0000 7fe81492a640 1 -- 172.21.15.89:0/813422467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8100ca3d0 msgr2=0x7fe810155300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:28.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.336+0000 7fe81492a640 1 --2- 172.21.15.89:0/813422467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8100ca3d0 0x7fe810155300 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe804011530 tx=0x7fe8040119f0 comp rx=0 tx=0).stop 2023-10-30T12:31:28.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.336+0000 7fe81492a640 1 -- 172.21.15.89:0/813422467 shutdown_connections 2023-10-30T12:31:28.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.336+0000 7fe81492a640 1 --2- 172.21.15.89:0/813422467 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe7f00664a0 0x7fe7f0068960 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:28.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.336+0000 7fe81492a640 1 --2- 172.21.15.89:0/813422467 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe810155840 0x7fe81014f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:28.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.336+0000 7fe81492a640 1 --2- 172.21.15.89:0/813422467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8100ca3d0 0x7fe810155300 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:28.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.336+0000 7fe81492a640 1 --2- 172.21.15.89:0/813422467 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe7f8003980 0x7fe81015acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:28.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.336+0000 7fe81492a640 1 -- 172.21.15.89:0/813422467 >> 172.21.15.89:0/813422467 conn(0x7fe8100c7910 msgr2=0x7fe8100c4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:28.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.336+0000 7fe81492a640 1 -- 172.21.15.89:0/813422467 shutdown_connections 2023-10-30T12:31:28.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:28.336+0000 7fe81492a640 1 -- 172.21.15.89:0/813422467 wait complete. 2023-10-30T12:31:28.362 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:31:30.363 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:31:30.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.508+0000 7fa09ba46640 1 Processor -- start 2023-10-30T12:31:30.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.508+0000 7fa09ba46640 1 -- start start 2023-10-30T12:31:30.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.508+0000 7fa09ba46640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa0941580e0 0x7fa094070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:30.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.508+0000 7fa09ba46640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0940c6540 0x7fa09415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:30.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.508+0000 7fa09ba46640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0940c5810 con 0x7fa094069e90 2023-10-30T12:31:30.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.508+0000 7fa09ba46640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0940c6960 con 0x7fa0941580e0 2023-10-30T12:31:30.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.508+0000 7fa09ba46640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0940c6bb0 con 0x7fa0940c6540 2023-10-30T12:31:30.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.508+0000 7fa0997e2640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa0941580e0 0x7fa094070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:30.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.508+0000 7fa0997e2640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa0941580e0 0x7fa094070740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53456/0 (socket says 172.21.15.89:53456) 2023-10-30T12:31:30.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.508+0000 7fa0997e2640 1 -- 172.21.15.89:0/75133328 learned_addr learned my addr 172.21.15.89:0/75133328 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:30.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.508+0000 7fa098fe1640 1 --2- 172.21.15.89:0/75133328 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0940c6540 0x7fa09415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:30.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.508+0000 7fa0997e2640 1 -- 172.21.15.89:0/75133328 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0940c6540 msgr2=0x7fa09415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:30.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.508+0000 7fa0997e2640 1 --2- 172.21.15.89:0/75133328 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0940c6540 0x7fa09415e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:30.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.508+0000 7fa0997e2640 1 -- 172.21.15.89:0/75133328 >> v1:172.21.15.192:6789/0 conn(0x7fa094069e90 legacy=0x7fa09406a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:30.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.508+0000 7fa0997e2640 1 -- 172.21.15.89:0/75133328 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0940c6e70 con 0x7fa0941580e0 2023-10-30T12:31:30.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.512+0000 7fa0997e2640 1 --2- 172.21.15.89:0/75133328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0941580e0 0x7fa094070740 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa090009870 tx=0x7fa09001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b358ace43832f59 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:30.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.512+0000 7fa08bfff640 1 -- 172.21.15.89:0/75133328 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0900aa620 con 0x7fa0941580e0 2023-10-30T12:31:30.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.512+0000 7fa08bfff640 1 -- 172.21.15.89:0/75133328 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0900aa780 con 0x7fa0941580e0 2023-10-30T12:31:30.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.512+0000 7fa08bfff640 1 -- 172.21.15.89:0/75133328 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0900aab10 con 0x7fa0941580e0 2023-10-30T12:31:30.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.512+0000 7fa09ba46640 1 -- 172.21.15.89:0/75133328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0941580e0 msgr2=0x7fa094070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:30.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.512+0000 7fa09ba46640 1 --2- 172.21.15.89:0/75133328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0941580e0 0x7fa094070740 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa090009870 tx=0x7fa09001c500 comp rx=0 tx=0).stop 2023-10-30T12:31:30.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.512+0000 7fa09ba46640 1 -- 172.21.15.89:0/75133328 shutdown_connections 2023-10-30T12:31:30.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.512+0000 7fa09ba46640 1 --2- 172.21.15.89:0/75133328 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0940c6540 0x7fa09415e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:30.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.512+0000 7fa09ba46640 1 --2- 172.21.15.89:0/75133328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0941580e0 0x7fa094070740 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:30.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.512+0000 7fa09ba46640 1 -- 172.21.15.89:0/75133328 >> 172.21.15.89:0/75133328 conn(0x7fa0940c7910 msgr2=0x7fa0940c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:30.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.516+0000 7fa09ba46640 1 -- 172.21.15.89:0/75133328 shutdown_connections 2023-10-30T12:31:30.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.516+0000 7fa09ba46640 1 -- 172.21.15.89:0/75133328 wait complete. 2023-10-30T12:31:30.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.516+0000 7fa09ba46640 1 Processor -- start 2023-10-30T12:31:30.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.516+0000 7fa09ba46640 1 -- start start 2023-10-30T12:31:30.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.516+0000 7fa09ba46640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa094069e90 0x7fa0940c20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:30.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.516+0000 7fa09ba46640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0940c6540 0x7fa094071130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:30.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.516+0000 7fa09ba46640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0941580e0 0x7fa0941527e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:30.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.516+0000 7fa09ba46640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa094070a00 con 0x7fa0941580e0 2023-10-30T12:31:30.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.516+0000 7fa09ba46640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0940c42f0 con 0x7fa094069e90 2023-10-30T12:31:30.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.516+0000 7fa09ba46640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa094071af0 con 0x7fa0940c6540 2023-10-30T12:31:30.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.516+0000 7fa099fe3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0941580e0 0x7fa0941527e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:30.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.516+0000 7fa099fe3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0941580e0 0x7fa0941527e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53458/0 (socket says 172.21.15.89:53458) 2023-10-30T12:31:30.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.516+0000 7fa099fe3640 1 -- 172.21.15.89:0/3721585948 learned_addr learned my addr 172.21.15.89:0/3721585948 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:30.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.516+0000 7fa0997e2640 1 --2- 172.21.15.89:0/3721585948 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa094069e90 0x7fa0940c20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:30.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.516+0000 7fa098fe1640 1 --2- 172.21.15.89:0/3721585948 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0940c6540 0x7fa094071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:30.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.516+0000 7fa099fe3640 1 -- 172.21.15.89:0/3721585948 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0940c6540 msgr2=0x7fa094071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:30.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.516+0000 7fa099fe3640 1 --2- 172.21.15.89:0/3721585948 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0940c6540 0x7fa094071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:30.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.516+0000 7fa099fe3640 1 -- 172.21.15.89:0/3721585948 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa094069e90 msgr2=0x7fa0940c20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:30.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.516+0000 7fa099fe3640 1 --2- 172.21.15.89:0/3721585948 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa094069e90 0x7fa0940c20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:30.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.516+0000 7fa099fe3640 1 -- 172.21.15.89:0/3721585948 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0900094d0 con 0x7fa0941580e0 2023-10-30T12:31:30.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.516+0000 7fa099fe3640 1 --2- 172.21.15.89:0/3721585948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0941580e0 0x7fa0941527e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa08c00c7e0 tx=0x7fa08c00cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:30.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.516+0000 7fa089ffb640 1 -- 172.21.15.89:0/3721585948 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08c004b70 con 0x7fa0941580e0 2023-10-30T12:31:30.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.516+0000 7fa089ffb640 1 -- 172.21.15.89:0/3721585948 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa08c004cd0 con 0x7fa0941580e0 2023-10-30T12:31:30.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.516+0000 7fa089ffb640 1 -- 172.21.15.89:0/3721585948 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08c00aa30 con 0x7fa0941580e0 2023-10-30T12:31:30.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.516+0000 7fa09ba46640 1 -- 172.21.15.89:0/3721585948 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa094071830 con 0x7fa0941580e0 2023-10-30T12:31:30.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.520+0000 7fa09ba46640 1 -- 172.21.15.89:0/3721585948 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0940705f0 con 0x7fa0941580e0 2023-10-30T12:31:30.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.520+0000 7fa09ba46640 1 -- 172.21.15.89:0/3721585948 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa09404e170 con 0x7fa0941580e0 2023-10-30T12:31:30.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.520+0000 7fa089ffb640 1 -- 172.21.15.89:0/3721585948 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa08c00ab90 con 0x7fa0941580e0 2023-10-30T12:31:30.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.520+0000 7fa089ffb640 1 --2- 172.21.15.89:0/3721585948 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa0780664a0 0x7fa078068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:30.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.520+0000 7fa089ffb640 1 -- 172.21.15.89:0/3721585948 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5890+0+0 (secure 0 0 0) 0x7fa08c083f10 con 0x7fa0941580e0 2023-10-30T12:31:30.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.520+0000 7fa0997e2640 1 --2- 172.21.15.89:0/3721585948 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa0780664a0 0x7fa078068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:30.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.524+0000 7fa0997e2640 1 --2- 172.21.15.89:0/3721585948 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa0780664a0 0x7fa078068960 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa0900ae000 tx=0x7fa0900b5040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:30.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.524+0000 7fa089ffb640 1 -- 172.21.15.89:0/3721585948 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa08c051380 con 0x7fa0941580e0 2023-10-30T12:31:30.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.708+0000 7fa09ba46640 1 -- 172.21.15.89:0/3721585948 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa094157520 con 0x7fa0941580e0 2023-10-30T12:31:30.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.708+0000 7fa089ffb640 1 -- 172.21.15.89:0/3721585948 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa08c0549a0 con 0x7fa0941580e0 2023-10-30T12:31:30.730 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:31:30.730 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:31:25.990848+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":2,"incarnation":15,"state":"up:resolve","state_seq":42,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:31:30.731 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 16 2023-10-30T12:31:30.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.708+0000 7fa09ba46640 1 -- 172.21.15.89:0/3721585948 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa0780664a0 msgr2=0x7fa078068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:30.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.708+0000 7fa09ba46640 1 --2- 172.21.15.89:0/3721585948 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa0780664a0 0x7fa078068960 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa0900ae000 tx=0x7fa0900b5040 comp rx=0 tx=0).stop 2023-10-30T12:31:30.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.708+0000 7fa09ba46640 1 -- 172.21.15.89:0/3721585948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0941580e0 msgr2=0x7fa0941527e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:30.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.708+0000 7fa09ba46640 1 --2- 172.21.15.89:0/3721585948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0941580e0 0x7fa0941527e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa08c00c7e0 tx=0x7fa08c00cca0 comp rx=0 tx=0).stop 2023-10-30T12:31:30.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.708+0000 7fa09ba46640 1 -- 172.21.15.89:0/3721585948 shutdown_connections 2023-10-30T12:31:30.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.708+0000 7fa09ba46640 1 --2- 172.21.15.89:0/3721585948 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa0780664a0 0x7fa078068960 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:30.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.708+0000 7fa09ba46640 1 --2- 172.21.15.89:0/3721585948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa0941580e0 0x7fa0941527e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:30.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.708+0000 7fa09ba46640 1 --2- 172.21.15.89:0/3721585948 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0940c6540 0x7fa094071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:30.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.708+0000 7fa09ba46640 1 --2- 172.21.15.89:0/3721585948 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa094069e90 0x7fa0940c20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:30.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.712+0000 7fa09ba46640 1 -- 172.21.15.89:0/3721585948 >> 172.21.15.89:0/3721585948 conn(0x7fa0940c7910 msgr2=0x7fa094153480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:30.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.712+0000 7fa09ba46640 1 -- 172.21.15.89:0/3721585948 shutdown_connections 2023-10-30T12:31:30.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:30.712+0000 7fa09ba46640 1 -- 172.21.15.89:0/3721585948 wait complete. 2023-10-30T12:31:30.736 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:31:32.736 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:31:32.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.880+0000 7fd21bfd3640 1 Processor -- start 2023-10-30T12:31:32.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.880+0000 7fd21bfd3640 1 -- start start 2023-10-30T12:31:32.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.880+0000 7fd21bfd3640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd214153d70 0x7fd21406ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:32.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.880+0000 7fd21bfd3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2140c8490 0x7fd2140c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:32.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.880+0000 7fd21bfd3640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd21406afa0 con 0x7fd21406e470 2023-10-30T12:31:32.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.880+0000 7fd21bfd3640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd21406d780 con 0x7fd214153d70 2023-10-30T12:31:32.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.880+0000 7fd21bfd3640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd214154150 con 0x7fd2140c8490 2023-10-30T12:31:32.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.880+0000 7fd219d6f640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd214153d70 0x7fd21406ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:32.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.880+0000 7fd219d6f640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd214153d70 0x7fd21406ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53472/0 (socket says 172.21.15.89:53472) 2023-10-30T12:31:32.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.880+0000 7fd219d6f640 1 -- 172.21.15.89:0/2619200186 learned_addr learned my addr 172.21.15.89:0/2619200186 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:32.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.880+0000 7fd21a570640 1 --2- 172.21.15.89:0/2619200186 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2140c8490 0x7fd2140c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:32.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.880+0000 7fd219d6f640 1 -- 172.21.15.89:0/2619200186 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2140c8490 msgr2=0x7fd2140c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:32.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.880+0000 7fd219d6f640 1 --2- 172.21.15.89:0/2619200186 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2140c8490 0x7fd2140c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:32.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.880+0000 7fd219d6f640 1 -- 172.21.15.89:0/2619200186 >> v1:172.21.15.192:6789/0 conn(0x7fd21406e470 legacy=0x7fd2140cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:32.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.880+0000 7fd219d6f640 1 -- 172.21.15.89:0/2619200186 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd21406e030 con 0x7fd214153d70 2023-10-30T12:31:32.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.884+0000 7fd219d6f640 1 --2- 172.21.15.89:0/2619200186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd214153d70 0x7fd21406ca80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd210009870 tx=0x7fd21001c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4edbc0793f04e576 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:32.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.884+0000 7fd218d6d640 1 -- 172.21.15.89:0/2619200186 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2100aa670 con 0x7fd214153d70 2023-10-30T12:31:32.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.884+0000 7fd218d6d640 1 -- 172.21.15.89:0/2619200186 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2100aa7d0 con 0x7fd214153d70 2023-10-30T12:31:32.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.884+0000 7fd218d6d640 1 -- 172.21.15.89:0/2619200186 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2100aab60 con 0x7fd214153d70 2023-10-30T12:31:32.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.884+0000 7fd21bfd3640 1 -- 172.21.15.89:0/2619200186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd214153d70 msgr2=0x7fd21406ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:32.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.884+0000 7fd21bfd3640 1 --2- 172.21.15.89:0/2619200186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd214153d70 0x7fd21406ca80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd210009870 tx=0x7fd21001c510 comp rx=0 tx=0).stop 2023-10-30T12:31:32.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.884+0000 7fd21bfd3640 1 -- 172.21.15.89:0/2619200186 shutdown_connections 2023-10-30T12:31:32.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.884+0000 7fd21bfd3640 1 --2- 172.21.15.89:0/2619200186 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2140c8490 0x7fd2140c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:32.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.884+0000 7fd21bfd3640 1 --2- 172.21.15.89:0/2619200186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd214153d70 0x7fd21406ca80 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:32.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.884+0000 7fd21bfd3640 1 -- 172.21.15.89:0/2619200186 >> 172.21.15.89:0/2619200186 conn(0x7fd214071ea0 msgr2=0x7fd214065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:32.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.884+0000 7fd21bfd3640 1 -- 172.21.15.89:0/2619200186 shutdown_connections 2023-10-30T12:31:32.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.884+0000 7fd21bfd3640 1 -- 172.21.15.89:0/2619200186 wait complete. 2023-10-30T12:31:32.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.888+0000 7fd21bfd3640 1 Processor -- start 2023-10-30T12:31:32.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.888+0000 7fd21bfd3640 1 -- start start 2023-10-30T12:31:32.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.888+0000 7fd21bfd3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd21406e470 0x7fd2141479e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:32.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.888+0000 7fd21bfd3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2140c8490 0x7fd21414f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:32.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.888+0000 7fd21bfd3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd214153d70 0x7fd2141f6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:32.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.888+0000 7fd21bfd3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd214148140 con 0x7fd21406e470 2023-10-30T12:31:32.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.888+0000 7fd21bfd3640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd214147f20 con 0x7fd214153d70 2023-10-30T12:31:32.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.888+0000 7fd21bfd3640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd214144520 con 0x7fd2140c8490 2023-10-30T12:31:32.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.888+0000 7fd219d6f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd21406e470 0x7fd2141479e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:32.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.888+0000 7fd219d6f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd21406e470 0x7fd2141479e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53484/0 (socket says 172.21.15.89:53484) 2023-10-30T12:31:32.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.888+0000 7fd219d6f640 1 -- 172.21.15.89:0/2414120412 learned_addr learned my addr 172.21.15.89:0/2414120412 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:32.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.888+0000 7fd21956e640 1 --2- 172.21.15.89:0/2414120412 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2140c8490 0x7fd21414f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:32.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.888+0000 7fd21a570640 1 --2- 172.21.15.89:0/2414120412 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd214153d70 0x7fd2141f6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:32.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.888+0000 7fd219d6f640 1 -- 172.21.15.89:0/2414120412 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2140c8490 msgr2=0x7fd21414f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:32.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.888+0000 7fd219d6f640 1 --2- 172.21.15.89:0/2414120412 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2140c8490 0x7fd21414f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:32.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.888+0000 7fd219d6f640 1 -- 172.21.15.89:0/2414120412 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd214153d70 msgr2=0x7fd2141f6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:32.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.888+0000 7fd219d6f640 1 --2- 172.21.15.89:0/2414120412 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd214153d70 0x7fd2141f6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:32.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.888+0000 7fd219d6f640 1 -- 172.21.15.89:0/2414120412 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2100094d0 con 0x7fd21406e470 2023-10-30T12:31:32.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.888+0000 7fd21956e640 1 --2- 172.21.15.89:0/2414120412 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2140c8490 0x7fd21414f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:31:32.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.888+0000 7fd219d6f640 1 --2- 172.21.15.89:0/2414120412 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd21406e470 0x7fd2141479e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd2100ac040 tx=0x7fd21001ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:32.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.888+0000 7fd20a7fc640 1 -- 172.21.15.89:0/2414120412 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2100b3800 con 0x7fd21406e470 2023-10-30T12:31:32.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.888+0000 7fd20a7fc640 1 -- 172.21.15.89:0/2414120412 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2100b3980 con 0x7fd21406e470 2023-10-30T12:31:32.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.888+0000 7fd20a7fc640 1 -- 172.21.15.89:0/2414120412 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2100b3d30 con 0x7fd21406e470 2023-10-30T12:31:32.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.888+0000 7fd21bfd3640 1 -- 172.21.15.89:0/2414120412 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd21413f860 con 0x7fd21406e470 2023-10-30T12:31:32.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.888+0000 7fd21bfd3640 1 -- 172.21.15.89:0/2414120412 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2141447c0 con 0x7fd21406e470 2023-10-30T12:31:32.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.888+0000 7fd21bfd3640 1 -- 172.21.15.89:0/2414120412 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd21404e170 con 0x7fd21406e470 2023-10-30T12:31:32.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.892+0000 7fd20a7fc640 1 -- 172.21.15.89:0/2414120412 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd2100c73a0 con 0x7fd21406e470 2023-10-30T12:31:32.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.892+0000 7fd20a7fc640 1 --2- 172.21.15.89:0/2414120412 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd1f80664a0 0x7fd1f8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:32.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.892+0000 7fd20a7fc640 1 -- 172.21.15.89:0/2414120412 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5890+0+0 (secure 0 0 0) 0x7fd21011ea90 con 0x7fd21406e470 2023-10-30T12:31:32.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.892+0000 7fd21956e640 1 --2- 172.21.15.89:0/2414120412 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd1f80664a0 0x7fd1f8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:32.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.892+0000 7fd21956e640 1 --2- 172.21.15.89:0/2414120412 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd1f80664a0 0x7fd1f8068960 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd214149380 tx=0x7fd204009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:32.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:32.896+0000 7fd20a7fc640 1 -- 172.21.15.89:0/2414120412 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd2100ebe60 con 0x7fd21406e470 2023-10-30T12:31:33.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:33.108+0000 7fd21bfd3640 1 -- 172.21.15.89:0/2414120412 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd21406c2a0 con 0x7fd21406e470 2023-10-30T12:31:33.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:33.108+0000 7fd20a7fc640 1 -- 172.21.15.89:0/2414120412 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5743 (secure 0 0 0) 0x7fd2100c3020 con 0x7fd21406e470 2023-10-30T12:31:33.114 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:31:33.114 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:31:31.965935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":2,"incarnation":15,"state":"up:reconnect","state_seq":44,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:31:33.114 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 17 2023-10-30T12:31:33.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:33.108+0000 7fd21bfd3640 1 -- 172.21.15.89:0/2414120412 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd1f80664a0 msgr2=0x7fd1f8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:33.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:33.108+0000 7fd21bfd3640 1 --2- 172.21.15.89:0/2414120412 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd1f80664a0 0x7fd1f8068960 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd214149380 tx=0x7fd204009040 comp rx=0 tx=0).stop 2023-10-30T12:31:33.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:33.108+0000 7fd21bfd3640 1 -- 172.21.15.89:0/2414120412 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd21406e470 msgr2=0x7fd2141479e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:33.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:33.108+0000 7fd21bfd3640 1 --2- 172.21.15.89:0/2414120412 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd21406e470 0x7fd2141479e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd2100ac040 tx=0x7fd21001ad50 comp rx=0 tx=0).stop 2023-10-30T12:31:33.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:33.108+0000 7fd21bfd3640 1 -- 172.21.15.89:0/2414120412 shutdown_connections 2023-10-30T12:31:33.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:33.108+0000 7fd21bfd3640 1 --2- 172.21.15.89:0/2414120412 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd1f80664a0 0x7fd1f8068960 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:33.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:33.108+0000 7fd21bfd3640 1 --2- 172.21.15.89:0/2414120412 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd214153d70 0x7fd2141f6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:33.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:33.108+0000 7fd21bfd3640 1 --2- 172.21.15.89:0/2414120412 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2140c8490 0x7fd21414f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:33.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:33.108+0000 7fd21bfd3640 1 --2- 172.21.15.89:0/2414120412 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd21406e470 0x7fd2141479e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:33.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:33.108+0000 7fd21bfd3640 1 -- 172.21.15.89:0/2414120412 >> 172.21.15.89:0/2414120412 conn(0x7fd214071ea0 msgr2=0x7fd2140c7520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:33.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:33.112+0000 7fd21bfd3640 1 -- 172.21.15.89:0/2414120412 shutdown_connections 2023-10-30T12:31:33.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:33.112+0000 7fd21bfd3640 1 -- 172.21.15.89:0/2414120412 wait complete. 2023-10-30T12:31:33.134 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:31:35.135 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:31:35.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.280+0000 7f4dbfe09640 1 Processor -- start 2023-10-30T12:31:35.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.280+0000 7f4dbfe09640 1 -- start start 2023-10-30T12:31:35.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.280+0000 7f4dbfe09640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4db815d6f0 0x7f4db815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:35.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.280+0000 7f4dbfe09640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4db80cf160 0x7f4db80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:35.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.280+0000 7f4dbfe09640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4db80cb800 con 0x7f4db81538a0 2023-10-30T12:31:35.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.280+0000 7f4dbfe09640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db80cb330 con 0x7f4db80cf160 2023-10-30T12:31:35.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.280+0000 7f4dbfe09640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db80cad70 con 0x7f4db815d6f0 2023-10-30T12:31:35.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.280+0000 7f4dbdba5640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f4db81538a0 0x7f4db80ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:35640/0 (socket says 172.21.15.89:35640) 2023-10-30T12:31:35.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.280+0000 7f4dbdba5640 1 -- 172.21.15.89:0/2866766365 learned_addr learned my addr 172.21.15.89:0/2866766365 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:31:35.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.280+0000 7f4dbe3a6640 1 --2- 172.21.15.89:0/2866766365 >> v2:172.21.15.89:3300/0 conn(0x7f4db80cf160 0x7f4db80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:35.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.280+0000 7f4dbd3a4640 1 --2- 172.21.15.89:0/2866766365 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4db815d6f0 0x7f4db815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:35.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.280+0000 7f4dbcba3640 1 -- 172.21.15.89:0/2866766365 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938841193 0 0) 0x7f4db80cb800 con 0x7f4db81538a0 2023-10-30T12:31:35.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.280+0000 7f4dbcba3640 1 -- 172.21.15.89:0/2866766365 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4da4003530 con 0x7f4db81538a0 2023-10-30T12:31:35.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.280+0000 7f4dbe3a6640 1 -- 172.21.15.89:0/2866766365 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4db815d6f0 msgr2=0x7f4db815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:35.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.280+0000 7f4dbe3a6640 1 --2- 172.21.15.89:0/2866766365 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4db815d6f0 0x7f4db815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:35.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.280+0000 7f4dbe3a6640 1 -- 172.21.15.89:0/2866766365 >> v1:172.21.15.89:6789/0 conn(0x7f4db81538a0 legacy=0x7f4db80ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:35.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.280+0000 7f4dbe3a6640 1 -- 172.21.15.89:0/2866766365 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db80cb050 con 0x7f4db80cf160 2023-10-30T12:31:35.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.284+0000 7f4dbe3a6640 1 --2- 172.21.15.89:0/2866766365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4db80cf160 0x7f4db80cf590 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4db0009d90 tx=0x7f4db001f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18fd9a61ff4f219d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:35.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.284+0000 7f4dbcba3640 1 -- 172.21.15.89:0/2866766365 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db00af600 con 0x7f4db80cf160 2023-10-30T12:31:35.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.284+0000 7f4dbcba3640 1 -- 172.21.15.89:0/2866766365 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db00ab050 con 0x7f4db80cf160 2023-10-30T12:31:35.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.284+0000 7f4dbcba3640 1 -- 172.21.15.89:0/2866766365 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db00af980 con 0x7f4db80cf160 2023-10-30T12:31:35.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.284+0000 7f4dbfe09640 1 -- 172.21.15.89:0/2866766365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4db80cf160 msgr2=0x7f4db80cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:35.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.284+0000 7f4dbfe09640 1 --2- 172.21.15.89:0/2866766365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4db80cf160 0x7f4db80cf590 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4db0009d90 tx=0x7f4db001f130 comp rx=0 tx=0).stop 2023-10-30T12:31:35.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.284+0000 7f4dbfe09640 1 -- 172.21.15.89:0/2866766365 shutdown_connections 2023-10-30T12:31:35.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.284+0000 7f4dbfe09640 1 --2- 172.21.15.89:0/2866766365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4db80cf160 0x7f4db80cf590 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:35.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.284+0000 7f4dbfe09640 1 --2- 172.21.15.89:0/2866766365 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4db815d6f0 0x7f4db815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:35.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.284+0000 7f4dbfe09640 1 -- 172.21.15.89:0/2866766365 >> 172.21.15.89:0/2866766365 conn(0x7f4db806bfe0 msgr2=0x7f4db806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:35.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.284+0000 7f4dbfe09640 1 -- 172.21.15.89:0/2866766365 shutdown_connections 2023-10-30T12:31:35.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.284+0000 7f4dbfe09640 1 -- 172.21.15.89:0/2866766365 wait complete. 2023-10-30T12:31:35.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.288+0000 7f4dbfe09640 1 Processor -- start 2023-10-30T12:31:35.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.288+0000 7f4dbfe09640 1 -- start start 2023-10-30T12:31:35.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.288+0000 7f4dbfe09640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4db80cf160 0x7f4db8065ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:35.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.288+0000 7f4dbfe09640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4db81538a0 0x7f4db80c71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:35.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.288+0000 7f4dbfe09640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4db815d6f0 0x7f4db80c7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:35.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.288+0000 7f4dbfe09640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db81f3270 con 0x7f4db81538a0 2023-10-30T12:31:35.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.288+0000 7f4dbfe09640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db81f2350 con 0x7f4db815d6f0 2023-10-30T12:31:35.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.288+0000 7f4dbfe09640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db81f25f0 con 0x7f4db80cf160 2023-10-30T12:31:35.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.288+0000 7f4dbd3a4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4db81538a0 0x7f4db80c71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:35.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.288+0000 7f4dbd3a4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4db81538a0 0x7f4db80c71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53510/0 (socket says 172.21.15.89:53510) 2023-10-30T12:31:35.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.288+0000 7f4dbd3a4640 1 -- 172.21.15.89:0/3375953141 learned_addr learned my addr 172.21.15.89:0/3375953141 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:35.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.288+0000 7f4dbdba5640 1 --2- 172.21.15.89:0/3375953141 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4db80cf160 0x7f4db8065ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:35.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.288+0000 7f4dbe3a6640 1 --2- 172.21.15.89:0/3375953141 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4db815d6f0 0x7f4db80c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:35.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.288+0000 7f4dbd3a4640 1 -- 172.21.15.89:0/3375953141 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4db80cf160 msgr2=0x7f4db8065ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:35.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.288+0000 7f4dbd3a4640 1 --2- 172.21.15.89:0/3375953141 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4db80cf160 0x7f4db8065ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:35.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.288+0000 7f4dbd3a4640 1 -- 172.21.15.89:0/3375953141 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4db815d6f0 msgr2=0x7f4db80c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:35.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.288+0000 7f4dbd3a4640 1 --2- 172.21.15.89:0/3375953141 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4db815d6f0 0x7f4db80c7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:35.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.288+0000 7f4dbd3a4640 1 -- 172.21.15.89:0/3375953141 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db0009a70 con 0x7f4db81538a0 2023-10-30T12:31:35.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.288+0000 7f4dbd3a4640 1 --2- 172.21.15.89:0/3375953141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4db81538a0 0x7f4db80c71e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4da800e870 tx=0x7f4da800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:35.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.288+0000 7f4dae7fc640 1 -- 172.21.15.89:0/3375953141 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da8009510 con 0x7f4db81538a0 2023-10-30T12:31:35.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.288+0000 7f4dae7fc640 1 -- 172.21.15.89:0/3375953141 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da800eec0 con 0x7f4db81538a0 2023-10-30T12:31:35.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.288+0000 7f4dbfe09640 1 -- 172.21.15.89:0/3375953141 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db81f28d0 con 0x7f4db81538a0 2023-10-30T12:31:35.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.288+0000 7f4dae7fc640 1 -- 172.21.15.89:0/3375953141 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da800cb40 con 0x7f4db81538a0 2023-10-30T12:31:35.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.288+0000 7f4dbfe09640 1 -- 172.21.15.89:0/3375953141 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db81f2e90 con 0x7f4db81538a0 2023-10-30T12:31:35.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.292+0000 7f4dbfe09640 1 -- 172.21.15.89:0/3375953141 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db81eef80 con 0x7f4db81538a0 2023-10-30T12:31:35.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.292+0000 7f4dae7fc640 1 -- 172.21.15.89:0/3375953141 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4da800cca0 con 0x7f4db81538a0 2023-10-30T12:31:35.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.292+0000 7f4dae7fc640 1 --2- 172.21.15.89:0/3375953141 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4d9c0664a0 0x7f4d9c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:35.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.292+0000 7f4dae7fc640 1 -- 172.21.15.89:0/3375953141 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5890+0+0 (secure 0 0 0) 0x7f4da8083fe0 con 0x7f4db81538a0 2023-10-30T12:31:35.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.292+0000 7f4dbdba5640 1 --2- 172.21.15.89:0/3375953141 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4d9c0664a0 0x7f4d9c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:35.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.292+0000 7f4dbdba5640 1 --2- 172.21.15.89:0/3375953141 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4d9c0664a0 0x7f4d9c068960 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4db40011f0 tx=0x7f4db4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:35.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.296+0000 7f4dae7fc640 1 -- 172.21.15.89:0/3375953141 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4da80513b0 con 0x7f4db81538a0 2023-10-30T12:31:35.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.476+0000 7f4dbfe09640 1 -- 172.21.15.89:0/3375953141 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4db80b2a50 con 0x7f4db81538a0 2023-10-30T12:31:35.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.476+0000 7f4dae7fc640 1 -- 172.21.15.89:0/3375953141 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5743 (secure 0 0 0) 0x7f4da80549d0 con 0x7f4db81538a0 2023-10-30T12:31:35.483 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:31:35.484 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:31:31.965935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":2,"incarnation":15,"state":"up:reconnect","state_seq":44,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:31:35.484 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 17 2023-10-30T12:31:35.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.480+0000 7f4dbfe09640 1 -- 172.21.15.89:0/3375953141 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4d9c0664a0 msgr2=0x7f4d9c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:35.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.480+0000 7f4dbfe09640 1 --2- 172.21.15.89:0/3375953141 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4d9c0664a0 0x7f4d9c068960 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4db40011f0 tx=0x7f4db4009040 comp rx=0 tx=0).stop 2023-10-30T12:31:35.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.480+0000 7f4dbfe09640 1 -- 172.21.15.89:0/3375953141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4db81538a0 msgr2=0x7f4db80c71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:35.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.480+0000 7f4dbfe09640 1 --2- 172.21.15.89:0/3375953141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4db81538a0 0x7f4db80c71e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4da800e870 tx=0x7f4da800ec70 comp rx=0 tx=0).stop 2023-10-30T12:31:35.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.480+0000 7f4dbfe09640 1 -- 172.21.15.89:0/3375953141 shutdown_connections 2023-10-30T12:31:35.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.480+0000 7f4dbfe09640 1 --2- 172.21.15.89:0/3375953141 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4d9c0664a0 0x7f4d9c068960 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:35.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.480+0000 7f4dbfe09640 1 --2- 172.21.15.89:0/3375953141 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4db815d6f0 0x7f4db80c7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:35.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.480+0000 7f4dbfe09640 1 --2- 172.21.15.89:0/3375953141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4db81538a0 0x7f4db80c71e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:35.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.480+0000 7f4dbfe09640 1 --2- 172.21.15.89:0/3375953141 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4db80cf160 0x7f4db8065ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:35.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.480+0000 7f4dbfe09640 1 -- 172.21.15.89:0/3375953141 >> 172.21.15.89:0/3375953141 conn(0x7f4db806bfe0 msgr2=0x7f4db806c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:35.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.480+0000 7f4dbfe09640 1 -- 172.21.15.89:0/3375953141 shutdown_connections 2023-10-30T12:31:35.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:35.480+0000 7f4dbfe09640 1 -- 172.21.15.89:0/3375953141 wait complete. 2023-10-30T12:31:35.504 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 17, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:50.729245+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 17, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4242": { "addr": "172.21.15.89:6837/2053475104", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6836", "nonce": 2053475104, "type": "v2" }, { "addr": "172.21.15.89:6837", "nonce": 2053475104, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4242, "incarnation": 15, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:reconnect", "state_seq": 44 }, "gid_4262": { "addr": "172.21.15.192:6839/3339589587", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6838", "nonce": 3339589587, "type": "v2" }, { "addr": "172.21.15.192:6839", "nonce": 3339589587, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 9, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4318": { "addr": "172.21.15.192:6837/1002124471", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6836", "nonce": 1002124471, "type": "v2" }, { "addr": "172.21.15.192:6837", "nonce": 1002124471, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4318, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:31:31.965935+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4318, "mds_1": 4262, "mds_2": 4242 } } } ], "standbys": [ { "addr": "172.21.15.89:6834/2909790173", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6832", "nonce": 2909790173, "type": "v2" }, { "addr": "172.21.15.89:6834", "nonce": 2909790173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.89:6835/2387357615", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6833", "nonce": 2387357615, "type": "v2" }, { "addr": "172.21.15.89:6835", "nonce": 2387357615, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4239, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:31:37.505 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:31:37.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.648+0000 7ff206646640 1 Processor -- start 2023-10-30T12:31:37.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.648+0000 7ff206646640 1 -- start start 2023-10-30T12:31:37.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.648+0000 7ff206646640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff2001538a0 0x7ff20015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:37.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.648+0000 7ff206646640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2000ca0e0 0x7ff20015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:37.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.648+0000 7ff206646640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2000cb800 con 0x7ff2000cf160 2023-10-30T12:31:37.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.648+0000 7ff206646640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff20015dc30 con 0x7ff2001538a0 2023-10-30T12:31:37.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.648+0000 7ff206646640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2000cb330 con 0x7ff2000ca0e0 2023-10-30T12:31:37.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.648+0000 7ff1fffff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff2001538a0 0x7ff20015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:37.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.648+0000 7ff1fffff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff2001538a0 0x7ff20015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53514/0 (socket says 172.21.15.89:53514) 2023-10-30T12:31:37.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.648+0000 7ff1fffff640 1 -- 172.21.15.89:0/2822421484 learned_addr learned my addr 172.21.15.89:0/2822421484 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:37.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.648+0000 7ff1ff7fe640 1 --2- 172.21.15.89:0/2822421484 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2000ca0e0 0x7ff20015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:37.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.648+0000 7ff1fffff640 1 -- 172.21.15.89:0/2822421484 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2000ca0e0 msgr2=0x7ff20015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:37.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.648+0000 7ff1fffff640 1 --2- 172.21.15.89:0/2822421484 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2000ca0e0 0x7ff20015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:37.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.648+0000 7ff1fffff640 1 -- 172.21.15.89:0/2822421484 >> v1:172.21.15.192:6789/0 conn(0x7ff2000cf160 legacy=0x7ff2000cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:37.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.648+0000 7ff1fffff640 1 -- 172.21.15.89:0/2822421484 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2000cad70 con 0x7ff2001538a0 2023-10-30T12:31:37.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.652+0000 7ff1fffff640 1 --2- 172.21.15.89:0/2822421484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2001538a0 0x7ff20015d6f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0009870 tx=0x7ff1f001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aacdebb3bf25b8c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:37.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.652+0000 7ff1feffd640 1 -- 172.21.15.89:0/2822421484 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f00a8a30 con 0x7ff2001538a0 2023-10-30T12:31:37.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.652+0000 7ff1feffd640 1 -- 172.21.15.89:0/2822421484 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f00a8b90 con 0x7ff2001538a0 2023-10-30T12:31:37.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.652+0000 7ff1feffd640 1 -- 172.21.15.89:0/2822421484 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f00b0440 con 0x7ff2001538a0 2023-10-30T12:31:37.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.652+0000 7ff206646640 1 -- 172.21.15.89:0/2822421484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2001538a0 msgr2=0x7ff20015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:37.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.652+0000 7ff206646640 1 --2- 172.21.15.89:0/2822421484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2001538a0 0x7ff20015d6f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0009870 tx=0x7ff1f001ed30 comp rx=0 tx=0).stop 2023-10-30T12:31:37.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.652+0000 7ff206646640 1 -- 172.21.15.89:0/2822421484 shutdown_connections 2023-10-30T12:31:37.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.652+0000 7ff206646640 1 --2- 172.21.15.89:0/2822421484 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2000ca0e0 0x7ff20015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:37.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.652+0000 7ff206646640 1 --2- 172.21.15.89:0/2822421484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2001538a0 0x7ff20015d6f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:37.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.652+0000 7ff206646640 1 -- 172.21.15.89:0/2822421484 >> 172.21.15.89:0/2822421484 conn(0x7ff20006bfe0 msgr2=0x7ff20006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:37.661 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.652+0000 7ff206646640 1 -- 172.21.15.89:0/2822421484 shutdown_connections 2023-10-30T12:31:37.661 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.652+0000 7ff206646640 1 -- 172.21.15.89:0/2822421484 wait complete. 2023-10-30T12:31:37.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.656+0000 7ff206646640 1 Processor -- start 2023-10-30T12:31:37.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.656+0000 7ff206646640 1 -- start start 2023-10-30T12:31:37.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.656+0000 7ff206646640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2000ca0e0 0x7ff200065de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:37.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.656+0000 7ff206646640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2000cf160 0x7ff20015b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:37.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.656+0000 7ff206646640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2001538a0 0x7ff20015bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:37.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.656+0000 7ff206646640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2000b3fe0 con 0x7ff2000cf160 2023-10-30T12:31:37.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.656+0000 7ff206646640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2000b4b60 con 0x7ff2000ca0e0 2023-10-30T12:31:37.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.656+0000 7ff206646640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2001e4df0 con 0x7ff2001538a0 2023-10-30T12:31:37.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.656+0000 7ff1ff7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2000cf160 0x7ff20015b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:37.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.656+0000 7ff1ff7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2000cf160 0x7ff20015b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53524/0 (socket says 172.21.15.89:53524) 2023-10-30T12:31:37.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.656+0000 7ff1ff7fe640 1 -- 172.21.15.89:0/2597444419 learned_addr learned my addr 172.21.15.89:0/2597444419 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:37.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.656+0000 7ff1fffff640 1 --2- 172.21.15.89:0/2597444419 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2000ca0e0 0x7ff200065de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:37.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.656+0000 7ff204be3640 1 --2- 172.21.15.89:0/2597444419 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2001538a0 0x7ff20015bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:37.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.656+0000 7ff1ff7fe640 1 -- 172.21.15.89:0/2597444419 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2001538a0 msgr2=0x7ff20015bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:37.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.656+0000 7ff1ff7fe640 1 --2- 172.21.15.89:0/2597444419 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2001538a0 0x7ff20015bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:37.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.656+0000 7ff1ff7fe640 1 -- 172.21.15.89:0/2597444419 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2000ca0e0 msgr2=0x7ff200065de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:37.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.656+0000 7ff1ff7fe640 1 --2- 172.21.15.89:0/2597444419 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2000ca0e0 0x7ff200065de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:37.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.656+0000 7ff1ff7fe640 1 -- 172.21.15.89:0/2597444419 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f00094d0 con 0x7ff2000cf160 2023-10-30T12:31:37.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.656+0000 7ff1ff7fe640 1 --2- 172.21.15.89:0/2597444419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2000cf160 0x7ff20015b7a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff1f400d800 tx=0x7ff1f400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:37.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.656+0000 7ff1fcff9640 1 -- 172.21.15.89:0/2597444419 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f400b6c0 con 0x7ff2000cf160 2023-10-30T12:31:37.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.656+0000 7ff1fcff9640 1 -- 172.21.15.89:0/2597444419 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f400b820 con 0x7ff2000cf160 2023-10-30T12:31:37.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.656+0000 7ff1fcff9640 1 -- 172.21.15.89:0/2597444419 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f400bb90 con 0x7ff2000cf160 2023-10-30T12:31:37.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.656+0000 7ff206646640 1 -- 172.21.15.89:0/2597444419 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2000b4230 con 0x7ff2000cf160 2023-10-30T12:31:37.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.656+0000 7ff206646640 1 -- 172.21.15.89:0/2597444419 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2000b47f0 con 0x7ff2000cf160 2023-10-30T12:31:37.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.660+0000 7ff206646640 1 -- 172.21.15.89:0/2597444419 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2000b49b0 con 0x7ff2000cf160 2023-10-30T12:31:37.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.660+0000 7ff1fcff9640 1 -- 172.21.15.89:0/2597444419 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff1f400bcf0 con 0x7ff2000cf160 2023-10-30T12:31:37.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.660+0000 7ff1fcff9640 1 --2- 172.21.15.89:0/2597444419 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff1ec0664a0 0x7ff1ec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:37.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.660+0000 7ff1fcff9640 1 -- 172.21.15.89:0/2597444419 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5890+0+0 (secure 0 0 0) 0x7ff1f4084450 con 0x7ff2000cf160 2023-10-30T12:31:37.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.660+0000 7ff1fffff640 1 --2- 172.21.15.89:0/2597444419 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff1ec0664a0 0x7ff1ec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:37.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.660+0000 7ff1fffff640 1 --2- 172.21.15.89:0/2597444419 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff1ec0664a0 0x7ff1ec068960 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff1f00b2000 tx=0x7ff1f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:37.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.664+0000 7ff1fcff9640 1 -- 172.21.15.89:0/2597444419 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff1f4051780 con 0x7ff2000cf160 2023-10-30T12:31:37.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.844+0000 7ff206646640 1 -- 172.21.15.89:0/2597444419 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2000b2a50 con 0x7ff2000cf160 2023-10-30T12:31:37.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.844+0000 7ff1fcff9640 1 -- 172.21.15.89:0/2597444419 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5743 (secure 0 0 0) 0x7ff1f4054da0 con 0x7ff2000cf160 2023-10-30T12:31:37.850 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:31:37.850 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:31:31.965935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":2,"incarnation":15,"state":"up:reconnect","state_seq":44,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:31:37.850 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 17 2023-10-30T12:31:37.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.844+0000 7ff206646640 1 -- 172.21.15.89:0/2597444419 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff1ec0664a0 msgr2=0x7ff1ec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:37.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.844+0000 7ff206646640 1 --2- 172.21.15.89:0/2597444419 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff1ec0664a0 0x7ff1ec068960 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff1f00b2000 tx=0x7ff1f0009040 comp rx=0 tx=0).stop 2023-10-30T12:31:37.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.844+0000 7ff206646640 1 -- 172.21.15.89:0/2597444419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2000cf160 msgr2=0x7ff20015b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:37.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.844+0000 7ff206646640 1 --2- 172.21.15.89:0/2597444419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2000cf160 0x7ff20015b7a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff1f400d800 tx=0x7ff1f400dcc0 comp rx=0 tx=0).stop 2023-10-30T12:31:37.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.844+0000 7ff206646640 1 -- 172.21.15.89:0/2597444419 shutdown_connections 2023-10-30T12:31:37.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.844+0000 7ff206646640 1 --2- 172.21.15.89:0/2597444419 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff1ec0664a0 0x7ff1ec068960 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:37.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.844+0000 7ff206646640 1 --2- 172.21.15.89:0/2597444419 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff2001538a0 0x7ff20015bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:37.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.844+0000 7ff206646640 1 --2- 172.21.15.89:0/2597444419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2000cf160 0x7ff20015b7a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:37.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.844+0000 7ff206646640 1 --2- 172.21.15.89:0/2597444419 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2000ca0e0 0x7ff200065de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:37.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.844+0000 7ff206646640 1 -- 172.21.15.89:0/2597444419 >> 172.21.15.89:0/2597444419 conn(0x7ff20006bfe0 msgr2=0x7ff20014de00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:37.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.844+0000 7ff206646640 1 -- 172.21.15.89:0/2597444419 shutdown_connections 2023-10-30T12:31:37.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:37.844+0000 7ff206646640 1 -- 172.21.15.89:0/2597444419 wait complete. 2023-10-30T12:31:37.871 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:31:39.872 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:31:40.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.016+0000 7f56e9017640 1 Processor -- start 2023-10-30T12:31:40.023 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.016+0000 7f56e9017640 1 -- start start 2023-10-30T12:31:40.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.016+0000 7f56e9017640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56e41539a0 0x7f56e415a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:40.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.016+0000 7f56e9017640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f56e4069e90 0x7f56e406a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:40.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.016+0000 7f56e9017640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56e415a8c0 con 0x7f56e4070a00 2023-10-30T12:31:40.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.016+0000 7f56e9017640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56e406fbb0 con 0x7f56e4069e90 2023-10-30T12:31:40.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.016+0000 7f56e9017640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e4153d80 con 0x7f56e41539a0 2023-10-30T12:31:40.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.020+0000 7f56e359e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f56e4069e90 0x7f56e406a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:40.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.020+0000 7f56e2d9d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56e41539a0 0x7f56e415a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:40.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.020+0000 7f56e359e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f56e4069e90 0x7f56e406a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:41716/0 (socket says 172.21.15.89:41716) 2023-10-30T12:31:40.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.020+0000 7f56e359e640 1 -- 172.21.15.89:0/3816368348 learned_addr learned my addr 172.21.15.89:0/3816368348 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:40.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.020+0000 7f56da59c640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f56e4070a00 0x7f56e4070e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:44128/0 (socket says 172.21.15.89:44128) 2023-10-30T12:31:40.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.020+0000 7f56e2d9d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56e41539a0 0x7f56e415a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:34600/0 (socket says 172.21.15.89:34600) 2023-10-30T12:31:40.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.020+0000 7f56e2d9d640 1 -- 172.21.15.89:0/3816368348 >> v2:172.21.15.192:3300/0 conn(0x7f56e4069e90 msgr2=0x7f56e406a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:40.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.020+0000 7f56e2d9d640 1 --2- 172.21.15.89:0/3816368348 >> v2:172.21.15.192:3300/0 conn(0x7f56e4069e90 0x7f56e406a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:40.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.020+0000 7f56e2d9d640 1 -- 172.21.15.89:0/3816368348 >> v1:172.21.15.192:6789/0 conn(0x7f56e4070a00 legacy=0x7f56e4070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:40.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.020+0000 7f56e2d9d640 1 -- 172.21.15.89:0/3816368348 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e40c5c50 con 0x7f56e41539a0 2023-10-30T12:31:40.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.020+0000 7f56e2d9d640 1 --2- 172.21.15.89:0/3816368348 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56e41539a0 0x7f56e415a380 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f56d40097c0 tx=0x7f56d401c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e51324968345541d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:40.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.024+0000 7f56e259c640 1 -- 172.21.15.89:0/3816368348 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d40ab600 con 0x7f56e41539a0 2023-10-30T12:31:40.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.024+0000 7f56e259c640 1 -- 172.21.15.89:0/3816368348 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d40a7050 con 0x7f56e41539a0 2023-10-30T12:31:40.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.024+0000 7f56e259c640 1 -- 172.21.15.89:0/3816368348 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d40ab980 con 0x7f56e41539a0 2023-10-30T12:31:40.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.024+0000 7f56e9017640 1 -- 172.21.15.89:0/3816368348 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56e41539a0 msgr2=0x7f56e415a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:40.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.024+0000 7f56e9017640 1 --2- 172.21.15.89:0/3816368348 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56e41539a0 0x7f56e415a380 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f56d40097c0 tx=0x7f56d401c3c0 comp rx=0 tx=0).stop 2023-10-30T12:31:40.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.024+0000 7f56e9017640 1 -- 172.21.15.89:0/3816368348 shutdown_connections 2023-10-30T12:31:40.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.024+0000 7f56e9017640 1 --2- 172.21.15.89:0/3816368348 >> v2:172.21.15.192:3300/0 conn(0x7f56e4069e90 0x7f56e406a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:40.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.024+0000 7f56e9017640 1 --2- 172.21.15.89:0/3816368348 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56e41539a0 0x7f56e415a380 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:40.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.024+0000 7f56e9017640 1 -- 172.21.15.89:0/3816368348 >> 172.21.15.89:0/3816368348 conn(0x7f56e40c7910 msgr2=0x7f56e40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:40.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.024+0000 7f56e9017640 1 -- 172.21.15.89:0/3816368348 shutdown_connections 2023-10-30T12:31:40.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.024+0000 7f56e9017640 1 -- 172.21.15.89:0/3816368348 wait complete. 2023-10-30T12:31:40.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.024+0000 7f56e9017640 1 Processor -- start 2023-10-30T12:31:40.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.024+0000 7f56e9017640 1 -- start start 2023-10-30T12:31:40.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.024+0000 7f56e9017640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56e4069e90 0x7f56e4145d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:40.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.028+0000 7f56e9017640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56e4070a00 0x7f56e414f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:40.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.028+0000 7f56e9017640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56e41539a0 0x7f56e41f5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:40.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.028+0000 7f56e9017640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e4158650 con 0x7f56e41539a0 2023-10-30T12:31:40.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.028+0000 7f56e9017640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e4159770 con 0x7f56e4070a00 2023-10-30T12:31:40.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.028+0000 7f56e9017640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e41f5570 con 0x7f56e4069e90 2023-10-30T12:31:40.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.028+0000 7f56e359e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56e41539a0 0x7f56e41f5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:40.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.028+0000 7f56e359e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56e41539a0 0x7f56e41f5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43508/0 (socket says 172.21.15.89:43508) 2023-10-30T12:31:40.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.028+0000 7f56e359e640 1 -- 172.21.15.89:0/2444271969 learned_addr learned my addr 172.21.15.89:0/2444271969 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:40.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.028+0000 7f56e2d9d640 1 --2- 172.21.15.89:0/2444271969 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56e4069e90 0x7f56e4145d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:40.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.028+0000 7f56da59c640 1 --2- 172.21.15.89:0/2444271969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56e4070a00 0x7f56e414f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:40.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.028+0000 7f56e359e640 1 -- 172.21.15.89:0/2444271969 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56e4069e90 msgr2=0x7f56e4145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:40.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.028+0000 7f56e359e640 1 --2- 172.21.15.89:0/2444271969 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56e4069e90 0x7f56e4145d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:40.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.028+0000 7f56e359e640 1 -- 172.21.15.89:0/2444271969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56e4070a00 msgr2=0x7f56e414f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:40.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.028+0000 7f56e359e640 1 --2- 172.21.15.89:0/2444271969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56e4070a00 0x7f56e414f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:40.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.028+0000 7f56e359e640 1 -- 172.21.15.89:0/2444271969 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d4009420 con 0x7f56e41539a0 2023-10-30T12:31:40.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.028+0000 7f56e359e640 1 --2- 172.21.15.89:0/2444271969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56e41539a0 0x7f56e41f5a50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f56dc00efc0 tx=0x7f56dc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:40.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.028+0000 7f56dbfff640 1 -- 172.21.15.89:0/2444271969 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56dc009b60 con 0x7f56e41539a0 2023-10-30T12:31:40.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.028+0000 7f56dbfff640 1 -- 172.21.15.89:0/2444271969 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56dc00f040 con 0x7f56e41539a0 2023-10-30T12:31:40.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.028+0000 7f56e9017640 1 -- 172.21.15.89:0/2444271969 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56e4159d50 con 0x7f56e41539a0 2023-10-30T12:31:40.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.028+0000 7f56dbfff640 1 -- 172.21.15.89:0/2444271969 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56dc0039a0 con 0x7f56e41539a0 2023-10-30T12:31:40.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.028+0000 7f56e9017640 1 -- 172.21.15.89:0/2444271969 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56e415a310 con 0x7f56e41539a0 2023-10-30T12:31:40.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.028+0000 7f56e9017640 1 -- 172.21.15.89:0/2444271969 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56e41f5770 con 0x7f56e41539a0 2023-10-30T12:31:40.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.028+0000 7f56dbfff640 1 -- 172.21.15.89:0/2444271969 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f56dc003b00 con 0x7f56e41539a0 2023-10-30T12:31:40.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.032+0000 7f56dbfff640 1 --2- 172.21.15.89:0/2444271969 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f56c40664a0 0x7f56c4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:40.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.032+0000 7f56dbfff640 1 -- 172.21.15.89:0/2444271969 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5890+0+0 (secure 0 0 0) 0x7f56dc0842d0 con 0x7f56e41539a0 2023-10-30T12:31:40.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.032+0000 7f56e2d9d640 1 --2- 172.21.15.89:0/2444271969 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f56c40664a0 0x7f56c4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:40.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.032+0000 7f56e2d9d640 1 --2- 172.21.15.89:0/2444271969 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f56c40664a0 0x7f56c4068960 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f56d40ad000 tx=0x7f56d40a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:40.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.036+0000 7f56dbfff640 1 -- 172.21.15.89:0/2444271969 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f56e41f5770 con 0x7f56e41539a0 2023-10-30T12:31:40.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.216+0000 7f56e9017640 1 -- 172.21.15.89:0/2444271969 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56e404e170 con 0x7f56e41539a0 2023-10-30T12:31:40.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.216+0000 7f56dbfff640 1 -- 172.21.15.89:0/2444271969 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+5740 (secure 0 0 0) 0x7f56dc0517b0 con 0x7f56e41539a0 2023-10-30T12:31:40.221 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:31:40.222 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:31:37.968020+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":2,"incarnation":15,"state":"up:rejoin","state_seq":46,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:31:40.223 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 18 2023-10-30T12:31:40.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.216+0000 7f56e9017640 1 -- 172.21.15.89:0/2444271969 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f56c40664a0 msgr2=0x7f56c4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:40.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.216+0000 7f56e9017640 1 --2- 172.21.15.89:0/2444271969 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f56c40664a0 0x7f56c4068960 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f56d40ad000 tx=0x7f56d40a6000 comp rx=0 tx=0).stop 2023-10-30T12:31:40.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.216+0000 7f56e9017640 1 -- 172.21.15.89:0/2444271969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56e41539a0 msgr2=0x7f56e41f5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:40.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.216+0000 7f56e9017640 1 --2- 172.21.15.89:0/2444271969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56e41539a0 0x7f56e41f5a50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f56dc00efc0 tx=0x7f56dc00c490 comp rx=0 tx=0).stop 2023-10-30T12:31:40.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.216+0000 7f56e9017640 1 -- 172.21.15.89:0/2444271969 shutdown_connections 2023-10-30T12:31:40.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.216+0000 7f56e9017640 1 --2- 172.21.15.89:0/2444271969 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f56c40664a0 0x7f56c4068960 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:40.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.216+0000 7f56e9017640 1 --2- 172.21.15.89:0/2444271969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56e4070a00 0x7f56e414f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:40.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.216+0000 7f56e9017640 1 --2- 172.21.15.89:0/2444271969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56e41539a0 0x7f56e41f5a50 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:40.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.216+0000 7f56e9017640 1 --2- 172.21.15.89:0/2444271969 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56e4069e90 0x7f56e4145d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:40.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.216+0000 7f56e9017640 1 -- 172.21.15.89:0/2444271969 >> 172.21.15.89:0/2444271969 conn(0x7f56e40c7910 msgr2=0x7f56e414e340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:40.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.216+0000 7f56e9017640 1 -- 172.21.15.89:0/2444271969 shutdown_connections 2023-10-30T12:31:40.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:40.216+0000 7f56e9017640 1 -- 172.21.15.89:0/2444271969 wait complete. 2023-10-30T12:31:40.243 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:31:42.244 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:31:42.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.388+0000 7fa8a2eea640 1 Processor -- start 2023-10-30T12:31:42.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.388+0000 7fa8a2eea640 1 -- start start 2023-10-30T12:31:42.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.388+0000 7fa8a2eea640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa89c153c30 0x7fa89c06a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:42.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.388+0000 7fa8a2eea640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa89c06ade0 con 0x7fa89c0b1ea0 2023-10-30T12:31:42.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.388+0000 7fa8a2eea640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa89c1598f0 con 0x7fa89c0cc0b0 2023-10-30T12:31:42.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.388+0000 7fa8a2eea640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c1596e0 con 0x7fa89c153c30 2023-10-30T12:31:42.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.388+0000 7fa893fff640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa89c0b1ea0 0x7fa89c0cb9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58118/0 (socket says 172.21.15.89:58118) 2023-10-30T12:31:42.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.388+0000 7fa893fff640 1 -- 172.21.15.89:0/2402498365 learned_addr learned my addr 172.21.15.89:0/2402498365 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:31:42.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.388+0000 7fa8a0c86640 1 --2- 172.21.15.89:0/2402498365 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa89c153c30 0x7fa89c06a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:42.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.388+0000 7fa8937fe640 1 -- 172.21.15.89:0/2402498365 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3690049229 0 0) 0x7fa88c0012a0 con 0x7fa89c0b1ea0 2023-10-30T12:31:42.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.388+0000 7fa8937fe640 1 -- 172.21.15.89:0/2402498365 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa884003530 con 0x7fa89c0b1ea0 2023-10-30T12:31:42.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.388+0000 7fa8937fe640 1 -- 172.21.15.89:0/2402498365 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1232905207 0 0) 0x7fa884003530 con 0x7fa89c0b1ea0 2023-10-30T12:31:42.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.388+0000 7fa8a0c86640 1 -- 172.21.15.89:0/2402498365 >> v1:172.21.15.192:6789/0 conn(0x7fa89c0cc0b0 legacy=0x7fa89c0cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:42.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.388+0000 7fa8a0c86640 1 -- 172.21.15.89:0/2402498365 >> v1:172.21.15.89:6789/0 conn(0x7fa89c0b1ea0 legacy=0x7fa89c0cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:42.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.388+0000 7fa8a0c86640 1 -- 172.21.15.89:0/2402498365 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa89c159b90 con 0x7fa89c153c30 2023-10-30T12:31:42.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.392+0000 7fa8a0c86640 1 --2- 172.21.15.89:0/2402498365 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa89c153c30 0x7fa89c06a8a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa8980097f0 tx=0x7fa89801ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=feb98f8a161b34f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:42.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.392+0000 7fa8937fe640 1 -- 172.21.15.89:0/2402498365 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88c001990 con 0x7fa89c153c30 2023-10-30T12:31:42.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.392+0000 7fa8937fe640 1 -- 172.21.15.89:0/2402498365 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8980a8650 con 0x7fa89c153c30 2023-10-30T12:31:42.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.392+0000 7fa8937fe640 1 -- 172.21.15.89:0/2402498365 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8980a89c0 con 0x7fa89c153c30 2023-10-30T12:31:42.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.392+0000 7fa8a2eea640 1 -- 172.21.15.89:0/2402498365 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa89c153c30 msgr2=0x7fa89c06a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:42.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.392+0000 7fa8a2eea640 1 --2- 172.21.15.89:0/2402498365 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa89c153c30 0x7fa89c06a8a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa8980097f0 tx=0x7fa89801ebd0 comp rx=0 tx=0).stop 2023-10-30T12:31:42.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.392+0000 7fa8a2eea640 1 -- 172.21.15.89:0/2402498365 shutdown_connections 2023-10-30T12:31:42.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.392+0000 7fa8a2eea640 1 --2- 172.21.15.89:0/2402498365 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa89c153c30 0x7fa89c06a8a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:42.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.392+0000 7fa8a2eea640 1 -- 172.21.15.89:0/2402498365 >> 172.21.15.89:0/2402498365 conn(0x7fa89c06bf40 msgr2=0x7fa89c06c350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:42.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.392+0000 7fa8a2eea640 1 -- 172.21.15.89:0/2402498365 shutdown_connections 2023-10-30T12:31:42.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.392+0000 7fa8a2eea640 1 -- 172.21.15.89:0/2402498365 wait complete. 2023-10-30T12:31:42.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.396+0000 7fa8a2eea640 1 Processor -- start 2023-10-30T12:31:42.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.396+0000 7fa8a2eea640 1 -- start start 2023-10-30T12:31:42.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.396+0000 7fa8a2eea640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa89c0b1ea0 0x7fa89c1e63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:42.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.396+0000 7fa8a2eea640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa89c0cc0b0 0x7fa89c0cced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:42.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.396+0000 7fa8a2eea640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa89c153c30 0x7fa89c0cd410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:42.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.396+0000 7fa8a2eea640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c0cec20 con 0x7fa89c153c30 2023-10-30T12:31:42.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.396+0000 7fa8a2eea640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c0ce940 con 0x7fa89c0cc0b0 2023-10-30T12:31:42.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.396+0000 7fa8a2eea640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c0b2530 con 0x7fa89c0b1ea0 2023-10-30T12:31:42.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.396+0000 7fa8a1487640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa89c153c30 0x7fa89c0cd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:42.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.396+0000 7fa8a1487640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa89c153c30 0x7fa89c0cd410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43514/0 (socket says 172.21.15.89:43514) 2023-10-30T12:31:42.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.396+0000 7fa8a0c86640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa89c0b1ea0 0x7fa89c1e63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:42.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.396+0000 7fa8a1487640 1 -- 172.21.15.89:0/2296047394 learned_addr learned my addr 172.21.15.89:0/2296047394 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:42.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.396+0000 7fa893fff640 1 --2- 172.21.15.89:0/2296047394 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa89c0cc0b0 0x7fa89c0cced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:42.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.396+0000 7fa8a1487640 1 -- 172.21.15.89:0/2296047394 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa89c0b1ea0 msgr2=0x7fa89c1e63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:42.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.396+0000 7fa8a1487640 1 --2- 172.21.15.89:0/2296047394 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa89c0b1ea0 0x7fa89c1e63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:42.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.396+0000 7fa8a1487640 1 -- 172.21.15.89:0/2296047394 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa89c0cc0b0 msgr2=0x7fa89c0cced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:42.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.396+0000 7fa8a1487640 1 --2- 172.21.15.89:0/2296047394 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa89c0cc0b0 0x7fa89c0cced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:42.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.396+0000 7fa8a1487640 1 -- 172.21.15.89:0/2296047394 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa898009420 con 0x7fa89c153c30 2023-10-30T12:31:42.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.396+0000 7fa8a1487640 1 --2- 172.21.15.89:0/2296047394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa89c153c30 0x7fa89c0cd410 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa89400c7d0 tx=0x7fa89400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:42.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.396+0000 7fa8917fa640 1 -- 172.21.15.89:0/2296047394 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa894003b80 con 0x7fa89c153c30 2023-10-30T12:31:42.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.396+0000 7fa8917fa640 1 -- 172.21.15.89:0/2296047394 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa894003ce0 con 0x7fa89c153c30 2023-10-30T12:31:42.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.396+0000 7fa8917fa640 1 -- 172.21.15.89:0/2296047394 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89400aa30 con 0x7fa89c153c30 2023-10-30T12:31:42.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.396+0000 7fa8a2eea640 1 -- 172.21.15.89:0/2296047394 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa89c0b27a0 con 0x7fa89c153c30 2023-10-30T12:31:42.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.396+0000 7fa8a2eea640 1 -- 172.21.15.89:0/2296047394 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa89c0ce5d0 con 0x7fa89c153c30 2023-10-30T12:31:42.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.400+0000 7fa8a2eea640 1 -- 172.21.15.89:0/2296047394 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa89c04e170 con 0x7fa89c153c30 2023-10-30T12:31:42.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.400+0000 7fa8917fa640 1 -- 172.21.15.89:0/2296047394 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa89400ab90 con 0x7fa89c153c30 2023-10-30T12:31:42.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.400+0000 7fa8917fa640 1 --2- 172.21.15.89:0/2296047394 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa8880664a0 0x7fa888068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:42.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.400+0000 7fa8917fa640 1 -- 172.21.15.89:0/2296047394 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5890+0+0 (secure 0 0 0) 0x7fa894083d00 con 0x7fa89c153c30 2023-10-30T12:31:42.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.400+0000 7fa8a0c86640 1 --2- 172.21.15.89:0/2296047394 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa8880664a0 0x7fa888068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:42.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.400+0000 7fa8a0c86640 1 --2- 172.21.15.89:0/2296047394 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa8880664a0 0x7fa888068960 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa8980b2000 tx=0x7fa8980aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:42.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.404+0000 7fa8917fa640 1 -- 172.21.15.89:0/2296047394 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa8940510d0 con 0x7fa89c153c30 2023-10-30T12:31:42.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.584+0000 7fa8a2eea640 1 -- 172.21.15.89:0/2296047394 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa89c147a60 con 0x7fa89c153c30 2023-10-30T12:31:42.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.584+0000 7fa8917fa640 1 -- 172.21.15.89:0/2296047394 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+5740 (secure 0 0 0) 0x7fa8940546f0 con 0x7fa89c153c30 2023-10-30T12:31:42.591 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:31:42.591 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:31:37.968020+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":2,"incarnation":15,"state":"up:rejoin","state_seq":46,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:31:42.592 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 18 2023-10-30T12:31:42.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.588+0000 7fa8a2eea640 1 -- 172.21.15.89:0/2296047394 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa8880664a0 msgr2=0x7fa888068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:42.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.588+0000 7fa8a2eea640 1 --2- 172.21.15.89:0/2296047394 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa8880664a0 0x7fa888068960 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa8980b2000 tx=0x7fa8980aa070 comp rx=0 tx=0).stop 2023-10-30T12:31:42.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.588+0000 7fa8a2eea640 1 -- 172.21.15.89:0/2296047394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa89c153c30 msgr2=0x7fa89c0cd410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:42.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.588+0000 7fa8a2eea640 1 --2- 172.21.15.89:0/2296047394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa89c153c30 0x7fa89c0cd410 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa89400c7d0 tx=0x7fa89400cc90 comp rx=0 tx=0).stop 2023-10-30T12:31:42.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.588+0000 7fa8a2eea640 1 -- 172.21.15.89:0/2296047394 shutdown_connections 2023-10-30T12:31:42.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.588+0000 7fa8a2eea640 1 --2- 172.21.15.89:0/2296047394 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa8880664a0 0x7fa888068960 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:42.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.588+0000 7fa8a2eea640 1 --2- 172.21.15.89:0/2296047394 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa89c0cc0b0 0x7fa89c0cced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:42.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.588+0000 7fa8a2eea640 1 --2- 172.21.15.89:0/2296047394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa89c153c30 0x7fa89c0cd410 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:42.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.588+0000 7fa8a2eea640 1 --2- 172.21.15.89:0/2296047394 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa89c0b1ea0 0x7fa89c1e63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:42.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.588+0000 7fa8a2eea640 1 -- 172.21.15.89:0/2296047394 >> 172.21.15.89:0/2296047394 conn(0x7fa89c06bf40 msgr2=0x7fa89c158fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:42.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.588+0000 7fa8a2eea640 1 -- 172.21.15.89:0/2296047394 shutdown_connections 2023-10-30T12:31:42.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:42.588+0000 7fa8a2eea640 1 -- 172.21.15.89:0/2296047394 wait complete. 2023-10-30T12:31:42.611 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:31:44.613 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:31:44.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.756+0000 7f51d991d640 1 Processor -- start 2023-10-30T12:31:44.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.756+0000 7f51d991d640 1 -- start start 2023-10-30T12:31:44.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.756+0000 7f51d991d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f51d41538a0 0x7f51d415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:44.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.756+0000 7f51d991d640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d40cb800 con 0x7f51d40cf160 2023-10-30T12:31:44.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.756+0000 7f51d991d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d415dc30 con 0x7f51d40ca0e0 2023-10-30T12:31:44.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.756+0000 7f51d991d640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51d40cb330 con 0x7f51d41538a0 2023-10-30T12:31:44.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.756+0000 7f51d2ffd640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f51d41538a0 0x7f51d415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:44.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.756+0000 7f51d2ffd640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f51d41538a0 0x7f51d415d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43518/0 (socket says 172.21.15.89:43518) 2023-10-30T12:31:44.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.756+0000 7f51d2ffd640 1 -- 172.21.15.89:0/1905016736 learned_addr learned my addr 172.21.15.89:0/1905016736 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:44.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.756+0000 7f51d37fe640 1 --1- 172.21.15.89:0/1905016736 >> v1:172.21.15.89:6789/0 conn(0x7f51d40cf160 0x7f51d4164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58130/0 (socket says 172.21.15.89:58130) 2023-10-30T12:31:44.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.756+0000 7f51d2ffd640 1 -- 172.21.15.89:0/1905016736 >> v1:172.21.15.192:6789/0 conn(0x7f51d40ca0e0 legacy=0x7f51d415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:44.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.756+0000 7f51d27fc640 1 -- 172.21.15.89:0/1905016736 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2658353832 0 0) 0x7f51d40cb800 con 0x7f51d40cf160 2023-10-30T12:31:44.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.756+0000 7f51d2ffd640 1 -- 172.21.15.89:0/1905016736 >> v1:172.21.15.89:6789/0 conn(0x7f51d40cf160 legacy=0x7f51d4164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:44.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.756+0000 7f51d2ffd640 1 -- 172.21.15.89:0/1905016736 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d40cad70 con 0x7f51d41538a0 2023-10-30T12:31:44.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.760+0000 7f51d2ffd640 1 --2- 172.21.15.89:0/1905016736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51d41538a0 0x7f51d415d6f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f51c4009870 tx=0x7f51c401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d54fd62183de85 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:44.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.760+0000 7f51d27fc640 1 -- 172.21.15.89:0/1905016736 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c40a8a30 con 0x7f51d41538a0 2023-10-30T12:31:44.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.760+0000 7f51d27fc640 1 -- 172.21.15.89:0/1905016736 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c40a8b90 con 0x7f51d41538a0 2023-10-30T12:31:44.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.760+0000 7f51d27fc640 1 -- 172.21.15.89:0/1905016736 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c40b0440 con 0x7f51d41538a0 2023-10-30T12:31:44.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.760+0000 7f51d991d640 1 -- 172.21.15.89:0/1905016736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51d41538a0 msgr2=0x7f51d415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:44.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.760+0000 7f51d991d640 1 --2- 172.21.15.89:0/1905016736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51d41538a0 0x7f51d415d6f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f51c4009870 tx=0x7f51c401ed30 comp rx=0 tx=0).stop 2023-10-30T12:31:44.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.760+0000 7f51d991d640 1 -- 172.21.15.89:0/1905016736 shutdown_connections 2023-10-30T12:31:44.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.760+0000 7f51d991d640 1 --2- 172.21.15.89:0/1905016736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51d41538a0 0x7f51d415d6f0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:44.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.760+0000 7f51d991d640 1 -- 172.21.15.89:0/1905016736 >> 172.21.15.89:0/1905016736 conn(0x7f51d406bfe0 msgr2=0x7f51d406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:44.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.760+0000 7f51d991d640 1 -- 172.21.15.89:0/1905016736 shutdown_connections 2023-10-30T12:31:44.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.760+0000 7f51d991d640 1 -- 172.21.15.89:0/1905016736 wait complete. 2023-10-30T12:31:44.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.764+0000 7f51d991d640 1 Processor -- start 2023-10-30T12:31:44.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.764+0000 7f51d991d640 1 -- start start 2023-10-30T12:31:44.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.764+0000 7f51d991d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51d40ca0e0 0x7f51d4066310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:44.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.764+0000 7f51d991d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51d40cf160 0x7f51d41f2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:44.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.764+0000 7f51d991d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51d41538a0 0x7f51d4145a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:44.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.764+0000 7f51d991d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d41f3150 con 0x7f51d41538a0 2023-10-30T12:31:44.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.764+0000 7f51d991d640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d40b50b0 con 0x7f51d40cf160 2023-10-30T12:31:44.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.764+0000 7f51d991d640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d4157650 con 0x7f51d40ca0e0 2023-10-30T12:31:44.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.764+0000 7f51d37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51d41538a0 0x7f51d4145a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:44.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.764+0000 7f51d37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51d41538a0 0x7f51d4145a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43534/0 (socket says 172.21.15.89:43534) 2023-10-30T12:31:44.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.764+0000 7f51d37fe640 1 -- 172.21.15.89:0/2701815159 learned_addr learned my addr 172.21.15.89:0/2701815159 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:44.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.764+0000 7f51d2ffd640 1 --2- 172.21.15.89:0/2701815159 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51d40ca0e0 0x7f51d4066310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:44.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.764+0000 7f51ca7fc640 1 --2- 172.21.15.89:0/2701815159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51d40cf160 0x7f51d41f2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:44.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.764+0000 7f51d37fe640 1 -- 172.21.15.89:0/2701815159 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51d40ca0e0 msgr2=0x7f51d4066310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:44.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.764+0000 7f51d37fe640 1 --2- 172.21.15.89:0/2701815159 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51d40ca0e0 0x7f51d4066310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:44.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.764+0000 7f51d37fe640 1 -- 172.21.15.89:0/2701815159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51d40cf160 msgr2=0x7f51d41f2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:44.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.764+0000 7f51d37fe640 1 --2- 172.21.15.89:0/2701815159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51d40cf160 0x7f51d41f2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:44.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.764+0000 7f51d37fe640 1 -- 172.21.15.89:0/2701815159 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c40094d0 con 0x7f51d41538a0 2023-10-30T12:31:44.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.764+0000 7f51d37fe640 1 --2- 172.21.15.89:0/2701815159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51d41538a0 0x7f51d4145a40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f51cc00c410 tx=0x7f51cc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:44.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.764+0000 7f51d0ff9640 1 -- 172.21.15.89:0/2701815159 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51cc0074e0 con 0x7f51d41538a0 2023-10-30T12:31:44.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.764+0000 7f51d0ff9640 1 -- 172.21.15.89:0/2701815159 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51cc007640 con 0x7f51d41538a0 2023-10-30T12:31:44.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.764+0000 7f51d991d640 1 -- 172.21.15.89:0/2701815159 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d40b4dd0 con 0x7f51d41538a0 2023-10-30T12:31:44.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.764+0000 7f51d0ff9640 1 -- 172.21.15.89:0/2701815159 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51cc004a30 con 0x7f51d41538a0 2023-10-30T12:31:44.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.764+0000 7f51d991d640 1 -- 172.21.15.89:0/2701815159 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51d4066850 con 0x7f51d41538a0 2023-10-30T12:31:44.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.768+0000 7f51d991d640 1 -- 172.21.15.89:0/2701815159 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51d40b3560 con 0x7f51d41538a0 2023-10-30T12:31:44.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.768+0000 7f51d0ff9640 1 -- 172.21.15.89:0/2701815159 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f51cc004b90 con 0x7f51d41538a0 2023-10-30T12:31:44.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.768+0000 7f51d0ff9640 1 --2- 172.21.15.89:0/2701815159 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f51b40664a0 0x7f51b4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:44.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.768+0000 7f51d0ff9640 1 -- 172.21.15.89:0/2701815159 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5890+0+0 (secure 0 0 0) 0x7f51cc083d30 con 0x7f51d41538a0 2023-10-30T12:31:44.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.768+0000 7f51d2ffd640 1 --2- 172.21.15.89:0/2701815159 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f51b40664a0 0x7f51b4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:44.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.768+0000 7f51d2ffd640 1 --2- 172.21.15.89:0/2701815159 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f51b40664a0 0x7f51b4068960 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f51c40b2000 tx=0x7f51c40aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:44.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.772+0000 7f51d0ff9640 1 -- 172.21.15.89:0/2701815159 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f51d40b3560 con 0x7f51d41538a0 2023-10-30T12:31:44.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.952+0000 7f51d991d640 1 -- 172.21.15.89:0/2701815159 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51d40b3560 con 0x7f51d41538a0 2023-10-30T12:31:44.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.952+0000 7f51d0ff9640 1 -- 172.21.15.89:0/2701815159 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+5746 (secure 0 0 0) 0x7f51d40b3560 con 0x7f51d41538a0 2023-10-30T12:31:44.959 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:31:44.959 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:31:43.062921+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":2,"incarnation":15,"state":"up:clientreplay","state_seq":48,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:31:44.960 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 19 2023-10-30T12:31:44.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.956+0000 7f51d991d640 1 -- 172.21.15.89:0/2701815159 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f51b40664a0 msgr2=0x7f51b4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:44.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.956+0000 7f51d991d640 1 --2- 172.21.15.89:0/2701815159 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f51b40664a0 0x7f51b4068960 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f51c40b2000 tx=0x7f51c40aa070 comp rx=0 tx=0).stop 2023-10-30T12:31:44.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.956+0000 7f51d991d640 1 -- 172.21.15.89:0/2701815159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51d41538a0 msgr2=0x7f51d4145a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:44.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.956+0000 7f51d991d640 1 --2- 172.21.15.89:0/2701815159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51d41538a0 0x7f51d4145a40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f51cc00c410 tx=0x7f51cc00c8d0 comp rx=0 tx=0).stop 2023-10-30T12:31:44.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.956+0000 7f51d991d640 1 -- 172.21.15.89:0/2701815159 shutdown_connections 2023-10-30T12:31:44.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.956+0000 7f51d991d640 1 --2- 172.21.15.89:0/2701815159 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f51b40664a0 0x7f51b4068960 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:44.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.956+0000 7f51d991d640 1 --2- 172.21.15.89:0/2701815159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51d40cf160 0x7f51d41f2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:44.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.956+0000 7f51d991d640 1 --2- 172.21.15.89:0/2701815159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51d41538a0 0x7f51d4145a40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:44.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.956+0000 7f51d991d640 1 --2- 172.21.15.89:0/2701815159 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51d40ca0e0 0x7f51d4066310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:44.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.956+0000 7f51d991d640 1 -- 172.21.15.89:0/2701815159 >> 172.21.15.89:0/2701815159 conn(0x7f51d406bfe0 msgr2=0x7f51d4152310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:44.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.956+0000 7f51d991d640 1 -- 172.21.15.89:0/2701815159 shutdown_connections 2023-10-30T12:31:44.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:44.956+0000 7f51d991d640 1 -- 172.21.15.89:0/2701815159 wait complete. 2023-10-30T12:31:44.979 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:31:45.236 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:31:45.240 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:31:46.980 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:31:47.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.124+0000 7f130ad3e640 1 Processor -- start 2023-10-30T12:31:47.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.124+0000 7f130ad3e640 1 -- start start 2023-10-30T12:31:47.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.124+0000 7f130ad3e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f130406f7a0 0x7f130406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:47.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.124+0000 7f130ad3e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1304071110 0x7f1304158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:47.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.124+0000 7f130ad3e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1304066cf0 con 0x7f13040c6bd0 2023-10-30T12:31:47.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.124+0000 7f130ad3e640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13040700c0 con 0x7f130406f7a0 2023-10-30T12:31:47.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.124+0000 7f130ad3e640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13040c70d0 con 0x7f1304071110 2023-10-30T12:31:47.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.124+0000 7f1308ada640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f130406f7a0 0x7f130406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:47.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.124+0000 7f1308ada640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f130406f7a0 0x7f130406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43548/0 (socket says 172.21.15.89:43548) 2023-10-30T12:31:47.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.124+0000 7f1308ada640 1 -- 172.21.15.89:0/3629556403 learned_addr learned my addr 172.21.15.89:0/3629556403 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:47.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.124+0000 7f13092db640 1 --2- 172.21.15.89:0/3629556403 >> v2:172.21.15.192:3300/0 conn(0x7f1304071110 0x7f1304158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:47.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.124+0000 7f1308ada640 1 -- 172.21.15.89:0/3629556403 >> v2:172.21.15.192:3300/0 conn(0x7f1304071110 msgr2=0x7f1304158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:47.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.124+0000 7f1308ada640 1 --2- 172.21.15.89:0/3629556403 >> v2:172.21.15.192:3300/0 conn(0x7f1304071110 0x7f1304158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:47.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.124+0000 7f1308ada640 1 -- 172.21.15.89:0/3629556403 >> v1:172.21.15.192:6789/0 conn(0x7f13040c6bd0 legacy=0x7f1304069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:47.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.124+0000 7f1308ada640 1 -- 172.21.15.89:0/3629556403 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13040c7390 con 0x7f130406f7a0 2023-10-30T12:31:47.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.128+0000 7f1308ada640 1 --2- 172.21.15.89:0/3629556403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f130406f7a0 0x7f130406fb80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f130406cdd0 tx=0x7f12f801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5aec15d517a399d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:47.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.128+0000 7f13037fe640 1 -- 172.21.15.89:0/3629556403 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f80a89f0 con 0x7f130406f7a0 2023-10-30T12:31:47.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.128+0000 7f13037fe640 1 -- 172.21.15.89:0/3629556403 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f80a8b50 con 0x7f130406f7a0 2023-10-30T12:31:47.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.128+0000 7f13037fe640 1 -- 172.21.15.89:0/3629556403 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f80b0440 con 0x7f130406f7a0 2023-10-30T12:31:47.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.128+0000 7f130ad3e640 1 -- 172.21.15.89:0/3629556403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f130406f7a0 msgr2=0x7f130406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:47.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.128+0000 7f130ad3e640 1 --2- 172.21.15.89:0/3629556403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f130406f7a0 0x7f130406fb80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f130406cdd0 tx=0x7f12f801ecf0 comp rx=0 tx=0).stop 2023-10-30T12:31:47.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.128+0000 7f130ad3e640 1 -- 172.21.15.89:0/3629556403 shutdown_connections 2023-10-30T12:31:47.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.128+0000 7f130ad3e640 1 --2- 172.21.15.89:0/3629556403 >> v2:172.21.15.192:3300/0 conn(0x7f1304071110 0x7f1304158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:47.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.128+0000 7f130ad3e640 1 --2- 172.21.15.89:0/3629556403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f130406f7a0 0x7f130406fb80 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:47.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.128+0000 7f130ad3e640 1 -- 172.21.15.89:0/3629556403 >> 172.21.15.89:0/3629556403 conn(0x7f13040c7910 msgr2=0x7f13040c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:47.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.128+0000 7f130ad3e640 1 -- 172.21.15.89:0/3629556403 shutdown_connections 2023-10-30T12:31:47.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.128+0000 7f130ad3e640 1 -- 172.21.15.89:0/3629556403 wait complete. 2023-10-30T12:31:47.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.132+0000 7f130ad3e640 1 Processor -- start 2023-10-30T12:31:47.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.132+0000 7f130ad3e640 1 -- start start 2023-10-30T12:31:47.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.132+0000 7f130ad3e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f130406f7a0 0x7f13041f0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:47.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.132+0000 7f130ad3e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1304071110 0x7f13041e2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:47.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.132+0000 7f130ad3e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13040c6bd0 0x7f13041de260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:47.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.132+0000 7f130ad3e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13040c5a80 con 0x7f1304071110 2023-10-30T12:31:47.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.132+0000 7f130ad3e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1304157440 con 0x7f130406f7a0 2023-10-30T12:31:47.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.132+0000 7f130ad3e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13041f09c0 con 0x7f13040c6bd0 2023-10-30T12:31:47.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.132+0000 7f1303fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1304071110 0x7f13041e2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:47.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.132+0000 7f1303fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1304071110 0x7f13041e2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43556/0 (socket says 172.21.15.89:43556) 2023-10-30T12:31:47.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.132+0000 7f1303fff640 1 -- 172.21.15.89:0/2142158605 learned_addr learned my addr 172.21.15.89:0/2142158605 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:47.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.132+0000 7f13092db640 1 --2- 172.21.15.89:0/2142158605 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13040c6bd0 0x7f13041de260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:47.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.132+0000 7f1308ada640 1 --2- 172.21.15.89:0/2142158605 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f130406f7a0 0x7f13041f0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:47.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.132+0000 7f1303fff640 1 -- 172.21.15.89:0/2142158605 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13040c6bd0 msgr2=0x7f13041de260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:47.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.132+0000 7f1303fff640 1 --2- 172.21.15.89:0/2142158605 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13040c6bd0 0x7f13041de260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:47.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.132+0000 7f1303fff640 1 -- 172.21.15.89:0/2142158605 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f130406f7a0 msgr2=0x7f13041f0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:47.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.132+0000 7f1303fff640 1 --2- 172.21.15.89:0/2142158605 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f130406f7a0 0x7f13041f0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:47.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.132+0000 7f1303fff640 1 -- 172.21.15.89:0/2142158605 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f80094d0 con 0x7f1304071110 2023-10-30T12:31:47.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.132+0000 7f1308ada640 1 --2- 172.21.15.89:0/2142158605 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f130406f7a0 0x7f13041f0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:31:47.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.132+0000 7f1303fff640 1 --2- 172.21.15.89:0/2142158605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1304071110 0x7f13041e2e00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f12f000c4d0 tx=0x7f12f000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:47.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.132+0000 7f13017fa640 1 -- 172.21.15.89:0/2142158605 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f00052d0 con 0x7f1304071110 2023-10-30T12:31:47.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.132+0000 7f13017fa640 1 -- 172.21.15.89:0/2142158605 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f0005430 con 0x7f1304071110 2023-10-30T12:31:47.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.132+0000 7f13017fa640 1 -- 172.21.15.89:0/2142158605 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f0012830 con 0x7f1304071110 2023-10-30T12:31:47.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.132+0000 7f130ad3e640 1 -- 172.21.15.89:0/2142158605 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1304151f30 con 0x7f1304071110 2023-10-30T12:31:47.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.132+0000 7f130ad3e640 1 -- 172.21.15.89:0/2142158605 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13040c5f40 con 0x7f1304071110 2023-10-30T12:31:47.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.132+0000 7f130ad3e640 1 -- 172.21.15.89:0/2142158605 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13041575b0 con 0x7f1304071110 2023-10-30T12:31:47.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.136+0000 7f13017fa640 1 -- 172.21.15.89:0/2142158605 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f12f0012990 con 0x7f1304071110 2023-10-30T12:31:47.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.136+0000 7f13017fa640 1 --2- 172.21.15.89:0/2142158605 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f12e80664a0 0x7f12e8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:47.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.136+0000 7f13017fa640 1 -- 172.21.15.89:0/2142158605 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5890+0+0 (secure 0 0 0) 0x7f12f0083920 con 0x7f1304071110 2023-10-30T12:31:47.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.136+0000 7f1308ada640 1 --2- 172.21.15.89:0/2142158605 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f12e80664a0 0x7f12e8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:47.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.136+0000 7f1308ada640 1 --2- 172.21.15.89:0/2142158605 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f12e80664a0 0x7f12e8068960 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f12f80b2000 tx=0x7f12f8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:47.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.140+0000 7f13017fa640 1 -- 172.21.15.89:0/2142158605 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f12f0050d70 con 0x7f1304071110 2023-10-30T12:31:47.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.320+0000 7f130ad3e640 1 -- 172.21.15.89:0/2142158605 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f130404e170 con 0x7f1304071110 2023-10-30T12:31:47.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.324+0000 7f13017fa640 1 -- 172.21.15.89:0/2142158605 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+5746 (secure 0 0 0) 0x7f12f0054390 con 0x7f1304071110 2023-10-30T12:31:47.329 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:31:47.329 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:31:43.062921+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":2,"incarnation":15,"state":"up:clientreplay","state_seq":48,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:31:47.329 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 19 2023-10-30T12:31:47.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.324+0000 7f130ad3e640 1 -- 172.21.15.89:0/2142158605 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f12e80664a0 msgr2=0x7f12e8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:47.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.324+0000 7f130ad3e640 1 --2- 172.21.15.89:0/2142158605 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f12e80664a0 0x7f12e8068960 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f12f80b2000 tx=0x7f12f8008040 comp rx=0 tx=0).stop 2023-10-30T12:31:47.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.324+0000 7f130ad3e640 1 -- 172.21.15.89:0/2142158605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1304071110 msgr2=0x7f13041e2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:47.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.324+0000 7f130ad3e640 1 --2- 172.21.15.89:0/2142158605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1304071110 0x7f13041e2e00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f12f000c4d0 tx=0x7f12f000c990 comp rx=0 tx=0).stop 2023-10-30T12:31:47.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.324+0000 7f130ad3e640 1 -- 172.21.15.89:0/2142158605 shutdown_connections 2023-10-30T12:31:47.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.324+0000 7f130ad3e640 1 --2- 172.21.15.89:0/2142158605 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f12e80664a0 0x7f12e8068960 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:47.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.324+0000 7f130ad3e640 1 --2- 172.21.15.89:0/2142158605 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13040c6bd0 0x7f13041de260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:47.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.324+0000 7f130ad3e640 1 --2- 172.21.15.89:0/2142158605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1304071110 0x7f13041e2e00 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:47.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.324+0000 7f130ad3e640 1 --2- 172.21.15.89:0/2142158605 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f130406f7a0 0x7f13041f0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:47.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.324+0000 7f130ad3e640 1 -- 172.21.15.89:0/2142158605 >> 172.21.15.89:0/2142158605 conn(0x7f13040c7910 msgr2=0x7f130406a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:47.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.324+0000 7f130ad3e640 1 -- 172.21.15.89:0/2142158605 shutdown_connections 2023-10-30T12:31:47.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:47.324+0000 7f130ad3e640 1 -- 172.21.15.89:0/2142158605 wait complete. 2023-10-30T12:31:47.350 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 19, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:50.729245+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 19, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4242": { "addr": "172.21.15.89:6837/2053475104", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6836", "nonce": 2053475104, "type": "v2" }, { "addr": "172.21.15.89:6837", "nonce": 2053475104, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4242, "incarnation": 15, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:clientreplay", "state_seq": 48 }, "gid_4262": { "addr": "172.21.15.192:6839/3339589587", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6838", "nonce": 3339589587, "type": "v2" }, { "addr": "172.21.15.192:6839", "nonce": 3339589587, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 9, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4318": { "addr": "172.21.15.192:6837/1002124471", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6836", "nonce": 1002124471, "type": "v2" }, { "addr": "172.21.15.192:6837", "nonce": 1002124471, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4318, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:31:43.062921+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4318, "mds_1": 4262, "mds_2": 4242 } } } ], "standbys": [ { "addr": "172.21.15.89:6834/2909790173", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6832", "nonce": 2909790173, "type": "v2" }, { "addr": "172.21.15.89:6834", "nonce": 2909790173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.89:6835/2387357615", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6833", "nonce": 2387357615, "type": "v2" }, { "addr": "172.21.15.89:6835", "nonce": 2387357615, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4239, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:31:49.353 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:31:49.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.496+0000 7efd795d3640 1 Processor -- start 2023-10-30T12:31:49.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.496+0000 7efd795d3640 1 -- start start 2023-10-30T12:31:49.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.496+0000 7efd795d3640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7efd740c6bd0 0x7efd7406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:49.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.496+0000 7efd795d3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd74071110 0x7efd74158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:49.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.496+0000 7efd795d3640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd74066cf0 con 0x7efd7406f7a0 2023-10-30T12:31:49.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.496+0000 7efd795d3640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd740700c0 con 0x7efd740c6bd0 2023-10-30T12:31:49.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.496+0000 7efd795d3640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd740c70d0 con 0x7efd74071110 2023-10-30T12:31:49.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.496+0000 7efd72ffd640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7efd7406f7a0 0x7efd74069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58138/0 (socket says 172.21.15.89:58138) 2023-10-30T12:31:49.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.496+0000 7efd72ffd640 1 -- 172.21.15.89:0/2279834957 learned_addr learned my addr 172.21.15.89:0/2279834957 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:31:49.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.496+0000 7efd737fe640 1 --2- 172.21.15.89:0/2279834957 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd74071110 0x7efd74158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:49.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.496+0000 7efd6a7fc640 1 --2- 172.21.15.89:0/2279834957 >> v2:172.21.15.192:3300/0 conn(0x7efd740c6bd0 0x7efd7406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:49.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.496+0000 7efd727fc640 1 -- 172.21.15.89:0/2279834957 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1438004799 0 0) 0x7efd74066cf0 con 0x7efd7406f7a0 2023-10-30T12:31:49.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.496+0000 7efd727fc640 1 -- 172.21.15.89:0/2279834957 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd60003530 con 0x7efd7406f7a0 2023-10-30T12:31:49.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.500+0000 7efd737fe640 1 -- 172.21.15.89:0/2279834957 >> v2:172.21.15.192:3300/0 conn(0x7efd740c6bd0 msgr2=0x7efd7406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:31:49.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.500+0000 7efd727fc640 1 -- 172.21.15.89:0/2279834957 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 552928375 0 0) 0x7efd60003530 con 0x7efd7406f7a0 2023-10-30T12:31:49.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.500+0000 7efd737fe640 1 --2- 172.21.15.89:0/2279834957 >> v2:172.21.15.192:3300/0 conn(0x7efd740c6bd0 0x7efd7406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:49.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.500+0000 7efd737fe640 1 -- 172.21.15.89:0/2279834957 >> v1:172.21.15.89:6789/0 conn(0x7efd7406f7a0 legacy=0x7efd74069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:49.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.500+0000 7efd737fe640 1 -- 172.21.15.89:0/2279834957 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd740c7390 con 0x7efd74071110 2023-10-30T12:31:49.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.500+0000 7efd6a7fc640 1 --2- 172.21.15.89:0/2279834957 >> v2:172.21.15.192:3300/0 conn(0x7efd740c6bd0 0x7efd7406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:31:49.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.500+0000 7efd737fe640 1 --2- 172.21.15.89:0/2279834957 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd74071110 0x7efd74158110 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efd6c0097c0 tx=0x7efd6c01eb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6e503e0ef31fa6c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:49.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.500+0000 7efd727fc640 1 -- 172.21.15.89:0/2279834957 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd64001a00 con 0x7efd74071110 2023-10-30T12:31:49.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.500+0000 7efd727fc640 1 -- 172.21.15.89:0/2279834957 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd6c0a85d0 con 0x7efd74071110 2023-10-30T12:31:49.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.500+0000 7efd727fc640 1 -- 172.21.15.89:0/2279834957 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6c0a8940 con 0x7efd74071110 2023-10-30T12:31:49.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.504+0000 7efd795d3640 1 -- 172.21.15.89:0/2279834957 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd74071110 msgr2=0x7efd74158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:49.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.504+0000 7efd795d3640 1 --2- 172.21.15.89:0/2279834957 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd74071110 0x7efd74158110 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efd6c0097c0 tx=0x7efd6c01eb90 comp rx=0 tx=0).stop 2023-10-30T12:31:49.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.504+0000 7efd795d3640 1 -- 172.21.15.89:0/2279834957 shutdown_connections 2023-10-30T12:31:49.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.504+0000 7efd795d3640 1 --2- 172.21.15.89:0/2279834957 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd74071110 0x7efd74158110 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:49.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.504+0000 7efd795d3640 1 --2- 172.21.15.89:0/2279834957 >> v2:172.21.15.192:3300/0 conn(0x7efd740c6bd0 0x7efd7406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:49.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.504+0000 7efd795d3640 1 -- 172.21.15.89:0/2279834957 >> 172.21.15.89:0/2279834957 conn(0x7efd740c7910 msgr2=0x7efd740c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:49.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.504+0000 7efd795d3640 1 -- 172.21.15.89:0/2279834957 shutdown_connections 2023-10-30T12:31:49.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.504+0000 7efd795d3640 1 -- 172.21.15.89:0/2279834957 wait complete. 2023-10-30T12:31:49.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.504+0000 7efd795d3640 1 Processor -- start 2023-10-30T12:31:49.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.504+0000 7efd795d3640 1 -- start start 2023-10-30T12:31:49.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.504+0000 7efd795d3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd7406f7a0 0x7efd740c5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:49.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.504+0000 7efd795d3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd74071110 0x7efd74155550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:49.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.504+0000 7efd795d3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd740c6bd0 0x7efd7414f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:49.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.504+0000 7efd795d3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd74153cc0 con 0x7efd74071110 2023-10-30T12:31:49.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.504+0000 7efd795d3640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd74153930 con 0x7efd740c6bd0 2023-10-30T12:31:49.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.504+0000 7efd795d3640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd741569d0 con 0x7efd7406f7a0 2023-10-30T12:31:49.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.504+0000 7efd6a7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd74071110 0x7efd74155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:49.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.504+0000 7efd6a7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd74071110 0x7efd74155550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43566/0 (socket says 172.21.15.89:43566) 2023-10-30T12:31:49.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.504+0000 7efd6a7fc640 1 -- 172.21.15.89:0/1935752274 learned_addr learned my addr 172.21.15.89:0/1935752274 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:49.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.504+0000 7efd72ffd640 1 --2- 172.21.15.89:0/1935752274 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd7406f7a0 0x7efd740c5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:49.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.504+0000 7efd737fe640 1 --2- 172.21.15.89:0/1935752274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd740c6bd0 0x7efd7414f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:49.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.504+0000 7efd6a7fc640 1 -- 172.21.15.89:0/1935752274 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd7406f7a0 msgr2=0x7efd740c5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:49.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.504+0000 7efd6a7fc640 1 --2- 172.21.15.89:0/1935752274 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd7406f7a0 0x7efd740c5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:49.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.504+0000 7efd6a7fc640 1 -- 172.21.15.89:0/1935752274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd740c6bd0 msgr2=0x7efd7414f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:49.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.504+0000 7efd6a7fc640 1 --2- 172.21.15.89:0/1935752274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd740c6bd0 0x7efd7414f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:49.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.504+0000 7efd6a7fc640 1 -- 172.21.15.89:0/1935752274 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd5c009420 con 0x7efd74071110 2023-10-30T12:31:49.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.508+0000 7efd6a7fc640 1 --2- 172.21.15.89:0/1935752274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd74071110 0x7efd74155550 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7efd5c00ebb0 tx=0x7efd5c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:49.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.508+0000 7efd6bfff640 1 -- 172.21.15.89:0/1935752274 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5c00eeb0 con 0x7efd74071110 2023-10-30T12:31:49.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.508+0000 7efd6bfff640 1 -- 172.21.15.89:0/1935752274 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd5c00cc30 con 0x7efd74071110 2023-10-30T12:31:49.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.508+0000 7efd6bfff640 1 -- 172.21.15.89:0/1935752274 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5c013470 con 0x7efd74071110 2023-10-30T12:31:49.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.508+0000 7efd795d3640 1 -- 172.21.15.89:0/1935752274 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd6c009420 con 0x7efd74071110 2023-10-30T12:31:49.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.508+0000 7efd795d3640 1 -- 172.21.15.89:0/1935752274 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd74158120 con 0x7efd74071110 2023-10-30T12:31:49.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.508+0000 7efd795d3640 1 -- 172.21.15.89:0/1935752274 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd7404e170 con 0x7efd74071110 2023-10-30T12:31:49.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.508+0000 7efd6bfff640 1 -- 172.21.15.89:0/1935752274 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efd5c0135d0 con 0x7efd74071110 2023-10-30T12:31:49.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.508+0000 7efd6bfff640 1 --2- 172.21.15.89:0/1935752274 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7efd540664a0 0x7efd54068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:49.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.508+0000 7efd6bfff640 1 -- 172.21.15.89:0/1935752274 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5890+0+0 (secure 0 0 0) 0x7efd5c084630 con 0x7efd74071110 2023-10-30T12:31:49.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.508+0000 7efd72ffd640 1 --2- 172.21.15.89:0/1935752274 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7efd540664a0 0x7efd54068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:49.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.512+0000 7efd72ffd640 1 --2- 172.21.15.89:0/1935752274 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7efd540664a0 0x7efd54068960 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7efd6400e270 tx=0x7efd64009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:49.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.512+0000 7efd6bfff640 1 -- 172.21.15.89:0/1935752274 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efd5c0519e0 con 0x7efd74071110 2023-10-30T12:31:49.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.692+0000 7efd795d3640 1 -- 172.21.15.89:0/1935752274 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd740c8df0 con 0x7efd74071110 2023-10-30T12:31:49.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.692+0000 7efd6bfff640 1 -- 172.21.15.89:0/1935752274 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+5740 (secure 0 0 0) 0x7efd5c055000 con 0x7efd74071110 2023-10-30T12:31:49.700 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:31:49.700 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:31:48.285333+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":2,"incarnation":15,"state":"up:active","state_seq":50,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:31:49.701 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 20 2023-10-30T12:31:49.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.696+0000 7efd795d3640 1 -- 172.21.15.89:0/1935752274 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7efd540664a0 msgr2=0x7efd54068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:49.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.696+0000 7efd795d3640 1 --2- 172.21.15.89:0/1935752274 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7efd540664a0 0x7efd54068960 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7efd6400e270 tx=0x7efd64009040 comp rx=0 tx=0).stop 2023-10-30T12:31:49.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.696+0000 7efd795d3640 1 -- 172.21.15.89:0/1935752274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd74071110 msgr2=0x7efd74155550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:49.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.696+0000 7efd795d3640 1 --2- 172.21.15.89:0/1935752274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd74071110 0x7efd74155550 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7efd5c00ebb0 tx=0x7efd5c00c520 comp rx=0 tx=0).stop 2023-10-30T12:31:49.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.696+0000 7efd795d3640 1 -- 172.21.15.89:0/1935752274 shutdown_connections 2023-10-30T12:31:49.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.696+0000 7efd795d3640 1 --2- 172.21.15.89:0/1935752274 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7efd540664a0 0x7efd54068960 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:49.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.696+0000 7efd795d3640 1 --2- 172.21.15.89:0/1935752274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd740c6bd0 0x7efd7414f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:49.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.696+0000 7efd795d3640 1 --2- 172.21.15.89:0/1935752274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd74071110 0x7efd74155550 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:49.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.696+0000 7efd795d3640 1 --2- 172.21.15.89:0/1935752274 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efd7406f7a0 0x7efd740c5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:49.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.696+0000 7efd795d3640 1 -- 172.21.15.89:0/1935752274 >> 172.21.15.89:0/1935752274 conn(0x7efd740c7910 msgr2=0x7efd740c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:49.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.696+0000 7efd795d3640 1 -- 172.21.15.89:0/1935752274 shutdown_connections 2023-10-30T12:31:49.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:49.696+0000 7efd795d3640 1 -- 172.21.15.89:0/1935752274 wait complete. 2023-10-30T12:31:49.721 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=2, replacing gid=4253 2023-10-30T12:31:49.721 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.b 2023-10-30T12:31:50.331 INFO:tasks.ceph.mds.f.smithi192.stderr:2023-10-30T12:31:50.327+0000 7f0036e29640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T12:31:54.160 INFO:tasks.workunit.client.0.smithi089.stdout: create 149/30 file after the snapshot 2023-10-30T12:31:54.186 INFO:tasks.workunit.client.0.smithi089.stdout: create 149/31 file after the snapshot 2023-10-30T12:31:54.186 INFO:tasks.workunit.client.0.smithi089.stdout: create 149/32 file after the snapshot 2023-10-30T12:31:54.187 INFO:tasks.workunit.client.0.smithi089.stdout: create 149/33 file after the snapshot 2023-10-30T12:31:54.188 INFO:tasks.workunit.client.0.smithi089.stdout: create 149/34 file after the snapshot 2023-10-30T12:31:54.188 INFO:tasks.workunit.client.0.smithi089.stdout: create 149/35 file after the snapshot 2023-10-30T12:31:54.189 INFO:tasks.workunit.client.0.smithi089.stdout: create 149/36 file after the snapshot 2023-10-30T12:31:54.190 INFO:tasks.workunit.client.0.smithi089.stdout: create 149/37 file after the snapshot 2023-10-30T12:31:54.191 INFO:tasks.workunit.client.0.smithi089.stdout: create 149/38 file after the snapshot 2023-10-30T12:31:54.191 INFO:tasks.workunit.client.0.smithi089.stdout: create 149/39 file after the snapshot 2023-10-30T12:31:54.192 INFO:tasks.workunit.client.0.smithi089.stdout: create 150/.snap/snap-subdir-test 2023-10-30T12:31:54.855 INFO:tasks.workunit.client.0.smithi089.stdout: create 150/30 file after the snapshot 2023-10-30T12:31:54.856 INFO:tasks.workunit.client.0.smithi089.stdout: create 150/31 file after the snapshot 2023-10-30T12:31:54.858 INFO:tasks.workunit.client.0.smithi089.stdout: create 150/32 file after the snapshot 2023-10-30T12:31:54.859 INFO:tasks.workunit.client.0.smithi089.stdout: create 150/33 file after the snapshot 2023-10-30T12:31:54.860 INFO:tasks.workunit.client.0.smithi089.stdout: create 150/34 file after the snapshot 2023-10-30T12:31:54.861 INFO:tasks.workunit.client.0.smithi089.stdout: create 150/35 file after the snapshot 2023-10-30T12:31:54.863 INFO:tasks.workunit.client.0.smithi089.stdout: create 150/36 file after the snapshot 2023-10-30T12:31:54.864 INFO:tasks.workunit.client.0.smithi089.stdout: create 150/37 file after the snapshot 2023-10-30T12:31:54.865 INFO:tasks.workunit.client.0.smithi089.stdout: create 150/38 file after the snapshot 2023-10-30T12:31:54.866 INFO:tasks.workunit.client.0.smithi089.stdout: create 150/39 file after the snapshot 2023-10-30T12:31:54.867 INFO:tasks.workunit.client.0.smithi089.stdout: create 151/.snap/snap-subdir-test 2023-10-30T12:31:54.887 INFO:tasks.workunit.client.0.smithi089.stdout: create 151/30 file after the snapshot 2023-10-30T12:31:54.887 INFO:tasks.workunit.client.0.smithi089.stdout: create 151/31 file after the snapshot 2023-10-30T12:31:54.889 INFO:tasks.workunit.client.0.smithi089.stdout: create 151/32 file after the snapshot 2023-10-30T12:31:54.890 INFO:tasks.workunit.client.0.smithi089.stdout: create 151/33 file after the snapshot 2023-10-30T12:31:54.891 INFO:tasks.workunit.client.0.smithi089.stdout: create 151/34 file after the snapshot 2023-10-30T12:31:54.892 INFO:tasks.workunit.client.0.smithi089.stdout: create 151/35 file after the snapshot 2023-10-30T12:31:54.893 INFO:tasks.workunit.client.0.smithi089.stdout: create 151/36 file after the snapshot 2023-10-30T12:31:54.894 INFO:tasks.workunit.client.0.smithi089.stdout: create 151/37 file after the snapshot 2023-10-30T12:31:54.895 INFO:tasks.workunit.client.0.smithi089.stdout: create 151/38 file after the snapshot 2023-10-30T12:31:54.896 INFO:tasks.workunit.client.0.smithi089.stdout: create 151/39 file after the snapshot 2023-10-30T12:31:54.897 INFO:tasks.workunit.client.0.smithi089.stdout: create 152/.snap/snap-subdir-test 2023-10-30T12:31:54.918 INFO:tasks.workunit.client.0.smithi089.stdout: create 152/30 file after the snapshot 2023-10-30T12:31:54.919 INFO:tasks.workunit.client.0.smithi089.stdout: create 152/31 file after the snapshot 2023-10-30T12:31:54.920 INFO:tasks.workunit.client.0.smithi089.stdout: create 152/32 file after the snapshot 2023-10-30T12:31:54.921 INFO:tasks.workunit.client.0.smithi089.stdout: create 152/33 file after the snapshot 2023-10-30T12:31:54.922 INFO:tasks.workunit.client.0.smithi089.stdout: create 152/34 file after the snapshot 2023-10-30T12:31:54.923 INFO:tasks.workunit.client.0.smithi089.stdout: create 152/35 file after the snapshot 2023-10-30T12:31:54.924 INFO:tasks.workunit.client.0.smithi089.stdout: create 152/36 file after the snapshot 2023-10-30T12:31:54.926 INFO:tasks.workunit.client.0.smithi089.stdout: create 152/37 file after the snapshot 2023-10-30T12:31:54.926 INFO:tasks.workunit.client.0.smithi089.stdout: create 152/38 file after the snapshot 2023-10-30T12:31:54.927 INFO:tasks.workunit.client.0.smithi089.stdout: create 152/39 file after the snapshot 2023-10-30T12:31:54.929 INFO:tasks.workunit.client.0.smithi089.stdout: create 153/.snap/snap-subdir-test 2023-10-30T12:31:54.949 INFO:tasks.workunit.client.0.smithi089.stdout: create 153/30 file after the snapshot 2023-10-30T12:31:54.950 INFO:tasks.workunit.client.0.smithi089.stdout: create 153/31 file after the snapshot 2023-10-30T12:31:54.950 INFO:tasks.workunit.client.0.smithi089.stdout: create 153/32 file after the snapshot 2023-10-30T12:31:54.951 INFO:tasks.workunit.client.0.smithi089.stdout: create 153/33 file after the snapshot 2023-10-30T12:31:54.952 INFO:tasks.workunit.client.0.smithi089.stdout: create 153/34 file after the snapshot 2023-10-30T12:31:54.952 INFO:tasks.workunit.client.0.smithi089.stdout: create 153/35 file after the snapshot 2023-10-30T12:31:54.953 INFO:tasks.workunit.client.0.smithi089.stdout: create 153/36 file after the snapshot 2023-10-30T12:31:54.954 INFO:tasks.workunit.client.0.smithi089.stdout: create 153/37 file after the snapshot 2023-10-30T12:31:54.955 INFO:tasks.workunit.client.0.smithi089.stdout: create 153/38 file after the snapshot 2023-10-30T12:31:54.955 INFO:tasks.workunit.client.0.smithi089.stdout: create 153/39 file after the snapshot 2023-10-30T12:31:54.956 INFO:tasks.workunit.client.0.smithi089.stdout: create 154/.snap/snap-subdir-test 2023-10-30T12:31:54.990 INFO:tasks.workunit.client.0.smithi089.stdout: create 154/30 file after the snapshot 2023-10-30T12:31:54.992 INFO:tasks.workunit.client.0.smithi089.stdout: create 154/31 file after the snapshot 2023-10-30T12:31:54.993 INFO:tasks.workunit.client.0.smithi089.stdout: create 154/32 file after the snapshot 2023-10-30T12:31:54.993 INFO:tasks.workunit.client.0.smithi089.stdout: create 154/33 file after the snapshot 2023-10-30T12:31:54.994 INFO:tasks.workunit.client.0.smithi089.stdout: create 154/34 file after the snapshot 2023-10-30T12:31:54.995 INFO:tasks.workunit.client.0.smithi089.stdout: create 154/35 file after the snapshot 2023-10-30T12:31:54.996 INFO:tasks.workunit.client.0.smithi089.stdout: create 154/36 file after the snapshot 2023-10-30T12:31:54.996 INFO:tasks.workunit.client.0.smithi089.stdout: create 154/37 file after the snapshot 2023-10-30T12:31:54.997 INFO:tasks.workunit.client.0.smithi089.stdout: create 154/38 file after the snapshot 2023-10-30T12:31:54.997 INFO:tasks.workunit.client.0.smithi089.stdout: create 154/39 file after the snapshot 2023-10-30T12:31:54.998 INFO:tasks.workunit.client.0.smithi089.stdout: create 155/.snap/snap-subdir-test 2023-10-30T12:31:55.017 INFO:tasks.workunit.client.0.smithi089.stdout: create 155/30 file after the snapshot 2023-10-30T12:31:55.017 INFO:tasks.workunit.client.0.smithi089.stdout: create 155/31 file after the snapshot 2023-10-30T12:31:55.018 INFO:tasks.workunit.client.0.smithi089.stdout: create 155/32 file after the snapshot 2023-10-30T12:31:55.019 INFO:tasks.workunit.client.0.smithi089.stdout: create 155/33 file after the snapshot 2023-10-30T12:31:55.019 INFO:tasks.workunit.client.0.smithi089.stdout: create 155/34 file after the snapshot 2023-10-30T12:31:55.021 INFO:tasks.workunit.client.0.smithi089.stdout: create 155/35 file after the snapshot 2023-10-30T12:31:55.022 INFO:tasks.workunit.client.0.smithi089.stdout: create 155/36 file after the snapshot 2023-10-30T12:31:55.024 INFO:tasks.workunit.client.0.smithi089.stdout: create 155/37 file after the snapshot 2023-10-30T12:31:55.024 INFO:tasks.workunit.client.0.smithi089.stdout: create 155/38 file after the snapshot 2023-10-30T12:31:55.024 INFO:tasks.workunit.client.0.smithi089.stdout: create 155/39 file after the snapshot 2023-10-30T12:31:55.024 INFO:tasks.workunit.client.0.smithi089.stdout: create 156/.snap/snap-subdir-test 2023-10-30T12:31:55.047 INFO:tasks.workunit.client.0.smithi089.stdout: create 156/30 file after the snapshot 2023-10-30T12:31:55.048 INFO:tasks.workunit.client.0.smithi089.stdout: create 156/31 file after the snapshot 2023-10-30T12:31:55.048 INFO:tasks.workunit.client.0.smithi089.stdout: create 156/32 file after the snapshot 2023-10-30T12:31:55.049 INFO:tasks.workunit.client.0.smithi089.stdout: create 156/33 file after the snapshot 2023-10-30T12:31:55.050 INFO:tasks.workunit.client.0.smithi089.stdout: create 156/34 file after the snapshot 2023-10-30T12:31:55.050 INFO:tasks.workunit.client.0.smithi089.stdout: create 156/35 file after the snapshot 2023-10-30T12:31:55.051 INFO:tasks.workunit.client.0.smithi089.stdout: create 156/36 file after the snapshot 2023-10-30T12:31:55.052 INFO:tasks.workunit.client.0.smithi089.stdout: create 156/37 file after the snapshot 2023-10-30T12:31:55.052 INFO:tasks.workunit.client.0.smithi089.stdout: create 156/38 file after the snapshot 2023-10-30T12:31:55.053 INFO:tasks.workunit.client.0.smithi089.stdout: create 156/39 file after the snapshot 2023-10-30T12:31:55.054 INFO:tasks.workunit.client.0.smithi089.stdout: create 157/.snap/snap-subdir-test 2023-10-30T12:31:55.077 INFO:tasks.workunit.client.0.smithi089.stdout: create 157/30 file after the snapshot 2023-10-30T12:31:55.078 INFO:tasks.workunit.client.0.smithi089.stdout: create 157/31 file after the snapshot 2023-10-30T12:31:55.079 INFO:tasks.workunit.client.0.smithi089.stdout: create 157/32 file after the snapshot 2023-10-30T12:31:55.080 INFO:tasks.workunit.client.0.smithi089.stdout: create 157/33 file after the snapshot 2023-10-30T12:31:55.081 INFO:tasks.workunit.client.0.smithi089.stdout: create 157/34 file after the snapshot 2023-10-30T12:31:55.082 INFO:tasks.workunit.client.0.smithi089.stdout: create 157/35 file after the snapshot 2023-10-30T12:31:55.083 INFO:tasks.workunit.client.0.smithi089.stdout: create 157/36 file after the snapshot 2023-10-30T12:31:55.083 INFO:tasks.workunit.client.0.smithi089.stdout: create 157/37 file after the snapshot 2023-10-30T12:31:55.084 INFO:tasks.workunit.client.0.smithi089.stdout: create 157/38 file after the snapshot 2023-10-30T12:31:55.084 INFO:tasks.workunit.client.0.smithi089.stdout: create 157/39 file after the snapshot 2023-10-30T12:31:55.085 INFO:tasks.workunit.client.0.smithi089.stdout: create 158/.snap/snap-subdir-test 2023-10-30T12:31:55.498 INFO:tasks.workunit.client.0.smithi089.stdout: create 158/30 file after the snapshot 2023-10-30T12:31:55.499 INFO:tasks.workunit.client.0.smithi089.stdout: create 158/31 file after the snapshot 2023-10-30T12:31:55.499 INFO:tasks.workunit.client.0.smithi089.stdout: create 158/32 file after the snapshot 2023-10-30T12:31:55.500 INFO:tasks.workunit.client.0.smithi089.stdout: create 158/33 file after the snapshot 2023-10-30T12:31:55.501 INFO:tasks.workunit.client.0.smithi089.stdout: create 158/34 file after the snapshot 2023-10-30T12:31:55.502 INFO:tasks.workunit.client.0.smithi089.stdout: create 158/35 file after the snapshot 2023-10-30T12:31:55.503 INFO:tasks.workunit.client.0.smithi089.stdout: create 158/36 file after the snapshot 2023-10-30T12:31:55.503 INFO:tasks.workunit.client.0.smithi089.stdout: create 158/37 file after the snapshot 2023-10-30T12:31:55.504 INFO:tasks.workunit.client.0.smithi089.stdout: create 158/38 file after the snapshot 2023-10-30T12:31:55.505 INFO:tasks.workunit.client.0.smithi089.stdout: create 158/39 file after the snapshot 2023-10-30T12:31:55.506 INFO:tasks.workunit.client.0.smithi089.stdout: create 159/.snap/snap-subdir-test 2023-10-30T12:31:55.522 INFO:tasks.workunit.client.0.smithi089.stdout: create 159/30 file after the snapshot 2023-10-30T12:31:55.523 INFO:tasks.workunit.client.0.smithi089.stdout: create 159/31 file after the snapshot 2023-10-30T12:31:55.524 INFO:tasks.workunit.client.0.smithi089.stdout: create 159/32 file after the snapshot 2023-10-30T12:31:55.525 INFO:tasks.workunit.client.0.smithi089.stdout: create 159/33 file after the snapshot 2023-10-30T12:31:55.525 INFO:tasks.workunit.client.0.smithi089.stdout: create 159/34 file after the snapshot 2023-10-30T12:31:55.526 INFO:tasks.workunit.client.0.smithi089.stdout: create 159/35 file after the snapshot 2023-10-30T12:31:55.527 INFO:tasks.workunit.client.0.smithi089.stdout: create 159/36 file after the snapshot 2023-10-30T12:31:55.528 INFO:tasks.workunit.client.0.smithi089.stdout: create 159/37 file after the snapshot 2023-10-30T12:31:55.528 INFO:tasks.workunit.client.0.smithi089.stdout: create 159/38 file after the snapshot 2023-10-30T12:31:55.529 INFO:tasks.workunit.client.0.smithi089.stdout: create 159/39 file after the snapshot 2023-10-30T12:31:55.530 INFO:tasks.workunit.client.0.smithi089.stdout: create 160/.snap/snap-subdir-test 2023-10-30T12:31:55.549 INFO:tasks.workunit.client.0.smithi089.stdout: create 160/30 file after the snapshot 2023-10-30T12:31:55.549 INFO:tasks.workunit.client.0.smithi089.stdout: create 160/31 file after the snapshot 2023-10-30T12:31:55.550 INFO:tasks.workunit.client.0.smithi089.stdout: create 160/32 file after the snapshot 2023-10-30T12:31:55.551 INFO:tasks.workunit.client.0.smithi089.stdout: create 160/33 file after the snapshot 2023-10-30T12:31:55.552 INFO:tasks.workunit.client.0.smithi089.stdout: create 160/34 file after the snapshot 2023-10-30T12:31:55.552 INFO:tasks.workunit.client.0.smithi089.stdout: create 160/35 file after the snapshot 2023-10-30T12:31:55.553 INFO:tasks.workunit.client.0.smithi089.stdout: create 160/36 file after the snapshot 2023-10-30T12:31:55.554 INFO:tasks.workunit.client.0.smithi089.stdout: create 160/37 file after the snapshot 2023-10-30T12:31:55.555 INFO:tasks.workunit.client.0.smithi089.stdout: create 160/38 file after the snapshot 2023-10-30T12:31:55.555 INFO:tasks.workunit.client.0.smithi089.stdout: create 160/39 file after the snapshot 2023-10-30T12:31:55.556 INFO:tasks.workunit.client.0.smithi089.stdout: create 161/.snap/snap-subdir-test 2023-10-30T12:31:55.574 INFO:tasks.workunit.client.0.smithi089.stdout: create 161/30 file after the snapshot 2023-10-30T12:31:55.575 INFO:tasks.workunit.client.0.smithi089.stdout: create 161/31 file after the snapshot 2023-10-30T12:31:55.576 INFO:tasks.workunit.client.0.smithi089.stdout: create 161/32 file after the snapshot 2023-10-30T12:31:55.577 INFO:tasks.workunit.client.0.smithi089.stdout: create 161/33 file after the snapshot 2023-10-30T12:31:55.577 INFO:tasks.workunit.client.0.smithi089.stdout: create 161/34 file after the snapshot 2023-10-30T12:31:55.578 INFO:tasks.workunit.client.0.smithi089.stdout: create 161/35 file after the snapshot 2023-10-30T12:31:55.579 INFO:tasks.workunit.client.0.smithi089.stdout: create 161/36 file after the snapshot 2023-10-30T12:31:55.580 INFO:tasks.workunit.client.0.smithi089.stdout: create 161/37 file after the snapshot 2023-10-30T12:31:55.580 INFO:tasks.workunit.client.0.smithi089.stdout: create 161/38 file after the snapshot 2023-10-30T12:31:55.581 INFO:tasks.workunit.client.0.smithi089.stdout: create 161/39 file after the snapshot 2023-10-30T12:31:55.582 INFO:tasks.workunit.client.0.smithi089.stdout: create 162/.snap/snap-subdir-test 2023-10-30T12:31:55.602 INFO:tasks.workunit.client.0.smithi089.stdout: create 162/30 file after the snapshot 2023-10-30T12:31:55.602 INFO:tasks.workunit.client.0.smithi089.stdout: create 162/31 file after the snapshot 2023-10-30T12:31:55.603 INFO:tasks.workunit.client.0.smithi089.stdout: create 162/32 file after the snapshot 2023-10-30T12:31:55.604 INFO:tasks.workunit.client.0.smithi089.stdout: create 162/33 file after the snapshot 2023-10-30T12:31:55.605 INFO:tasks.workunit.client.0.smithi089.stdout: create 162/34 file after the snapshot 2023-10-30T12:31:55.605 INFO:tasks.workunit.client.0.smithi089.stdout: create 162/35 file after the snapshot 2023-10-30T12:31:55.606 INFO:tasks.workunit.client.0.smithi089.stdout: create 162/36 file after the snapshot 2023-10-30T12:31:55.607 INFO:tasks.workunit.client.0.smithi089.stdout: create 162/37 file after the snapshot 2023-10-30T12:31:55.608 INFO:tasks.workunit.client.0.smithi089.stdout: create 162/38 file after the snapshot 2023-10-30T12:31:55.608 INFO:tasks.workunit.client.0.smithi089.stdout: create 162/39 file after the snapshot 2023-10-30T12:31:55.609 INFO:tasks.workunit.client.0.smithi089.stdout: create 163/.snap/snap-subdir-test 2023-10-30T12:31:55.632 INFO:tasks.workunit.client.0.smithi089.stdout: create 163/30 file after the snapshot 2023-10-30T12:31:55.633 INFO:tasks.workunit.client.0.smithi089.stdout: create 163/31 file after the snapshot 2023-10-30T12:31:55.634 INFO:tasks.workunit.client.0.smithi089.stdout: create 163/32 file after the snapshot 2023-10-30T12:31:55.634 INFO:tasks.workunit.client.0.smithi089.stdout: create 163/33 file after the snapshot 2023-10-30T12:31:55.635 INFO:tasks.workunit.client.0.smithi089.stdout: create 163/34 file after the snapshot 2023-10-30T12:31:55.636 INFO:tasks.workunit.client.0.smithi089.stdout: create 163/35 file after the snapshot 2023-10-30T12:31:55.785 INFO:tasks.workunit.client.0.smithi089.stdout: create 163/36 file after the snapshot 2023-10-30T12:31:56.577 INFO:tasks.workunit.client.0.smithi089.stdout: create 163/37 file after the snapshot 2023-10-30T12:31:56.579 INFO:tasks.workunit.client.0.smithi089.stdout: create 163/38 file after the snapshot 2023-10-30T12:31:56.579 INFO:tasks.workunit.client.0.smithi089.stdout: create 163/39 file after the snapshot 2023-10-30T12:31:56.580 INFO:tasks.workunit.client.0.smithi089.stdout: create 164/.snap/snap-subdir-test 2023-10-30T12:31:56.613 INFO:tasks.workunit.client.0.smithi089.stdout: create 164/30 file after the snapshot 2023-10-30T12:31:56.614 INFO:tasks.workunit.client.0.smithi089.stdout: create 164/31 file after the snapshot 2023-10-30T12:31:56.615 INFO:tasks.workunit.client.0.smithi089.stdout: create 164/32 file after the snapshot 2023-10-30T12:31:56.616 INFO:tasks.workunit.client.0.smithi089.stdout: create 164/33 file after the snapshot 2023-10-30T12:31:56.616 INFO:tasks.workunit.client.0.smithi089.stdout: create 164/34 file after the snapshot 2023-10-30T12:31:56.617 INFO:tasks.workunit.client.0.smithi089.stdout: create 164/35 file after the snapshot 2023-10-30T12:31:56.618 INFO:tasks.workunit.client.0.smithi089.stdout: create 164/36 file after the snapshot 2023-10-30T12:31:56.619 INFO:tasks.workunit.client.0.smithi089.stdout: create 164/37 file after the snapshot 2023-10-30T12:31:56.620 INFO:tasks.workunit.client.0.smithi089.stdout: create 164/38 file after the snapshot 2023-10-30T12:31:56.621 INFO:tasks.workunit.client.0.smithi089.stdout: create 164/39 file after the snapshot 2023-10-30T12:31:56.621 INFO:tasks.workunit.client.0.smithi089.stdout: create 165/.snap/snap-subdir-test 2023-10-30T12:31:56.641 INFO:tasks.workunit.client.0.smithi089.stdout: create 165/30 file after the snapshot 2023-10-30T12:31:56.642 INFO:tasks.workunit.client.0.smithi089.stdout: create 165/31 file after the snapshot 2023-10-30T12:31:56.643 INFO:tasks.workunit.client.0.smithi089.stdout: create 165/32 file after the snapshot 2023-10-30T12:31:56.643 INFO:tasks.workunit.client.0.smithi089.stdout: create 165/33 file after the snapshot 2023-10-30T12:31:56.644 INFO:tasks.workunit.client.0.smithi089.stdout: create 165/34 file after the snapshot 2023-10-30T12:31:56.645 INFO:tasks.workunit.client.0.smithi089.stdout: create 165/35 file after the snapshot 2023-10-30T12:31:56.646 INFO:tasks.workunit.client.0.smithi089.stdout: create 165/36 file after the snapshot 2023-10-30T12:31:56.647 INFO:tasks.workunit.client.0.smithi089.stdout: create 165/37 file after the snapshot 2023-10-30T12:31:56.647 INFO:tasks.workunit.client.0.smithi089.stdout: create 165/38 file after the snapshot 2023-10-30T12:31:56.648 INFO:tasks.workunit.client.0.smithi089.stdout: create 165/39 file after the snapshot 2023-10-30T12:31:56.649 INFO:tasks.workunit.client.0.smithi089.stdout: create 166/.snap/snap-subdir-test 2023-10-30T12:31:56.669 INFO:tasks.workunit.client.0.smithi089.stdout: create 166/30 file after the snapshot 2023-10-30T12:31:56.670 INFO:tasks.workunit.client.0.smithi089.stdout: create 166/31 file after the snapshot 2023-10-30T12:31:56.670 INFO:tasks.workunit.client.0.smithi089.stdout: create 166/32 file after the snapshot 2023-10-30T12:31:56.671 INFO:tasks.workunit.client.0.smithi089.stdout: create 166/33 file after the snapshot 2023-10-30T12:31:56.672 INFO:tasks.workunit.client.0.smithi089.stdout: create 166/34 file after the snapshot 2023-10-30T12:31:56.672 INFO:tasks.workunit.client.0.smithi089.stdout: create 166/35 file after the snapshot 2023-10-30T12:31:56.673 INFO:tasks.workunit.client.0.smithi089.stdout: create 166/36 file after the snapshot 2023-10-30T12:31:56.674 INFO:tasks.workunit.client.0.smithi089.stdout: create 166/37 file after the snapshot 2023-10-30T12:31:56.674 INFO:tasks.workunit.client.0.smithi089.stdout: create 166/38 file after the snapshot 2023-10-30T12:31:56.675 INFO:tasks.workunit.client.0.smithi089.stdout: create 166/39 file after the snapshot 2023-10-30T12:31:56.676 INFO:tasks.workunit.client.0.smithi089.stdout: create 167/.snap/snap-subdir-test 2023-10-30T12:31:56.695 INFO:tasks.workunit.client.0.smithi089.stdout: create 167/30 file after the snapshot 2023-10-30T12:31:56.695 INFO:tasks.workunit.client.0.smithi089.stdout: create 167/31 file after the snapshot 2023-10-30T12:31:56.696 INFO:tasks.workunit.client.0.smithi089.stdout: create 167/32 file after the snapshot 2023-10-30T12:31:56.697 INFO:tasks.workunit.client.0.smithi089.stdout: create 167/33 file after the snapshot 2023-10-30T12:31:56.698 INFO:tasks.workunit.client.0.smithi089.stdout: create 167/34 file after the snapshot 2023-10-30T12:31:56.698 INFO:tasks.workunit.client.0.smithi089.stdout: create 167/35 file after the snapshot 2023-10-30T12:31:56.699 INFO:tasks.workunit.client.0.smithi089.stdout: create 167/36 file after the snapshot 2023-10-30T12:31:56.700 INFO:tasks.workunit.client.0.smithi089.stdout: create 167/37 file after the snapshot 2023-10-30T12:31:56.700 INFO:tasks.workunit.client.0.smithi089.stdout: create 167/38 file after the snapshot 2023-10-30T12:31:56.701 INFO:tasks.workunit.client.0.smithi089.stdout: create 167/39 file after the snapshot 2023-10-30T12:31:56.702 INFO:tasks.workunit.client.0.smithi089.stdout: create 168/.snap/snap-subdir-test 2023-10-30T12:31:56.720 INFO:tasks.workunit.client.0.smithi089.stdout: create 168/30 file after the snapshot 2023-10-30T12:31:56.721 INFO:tasks.workunit.client.0.smithi089.stdout: create 168/31 file after the snapshot 2023-10-30T12:31:56.721 INFO:tasks.workunit.client.0.smithi089.stdout: create 168/32 file after the snapshot 2023-10-30T12:31:56.722 INFO:tasks.workunit.client.0.smithi089.stdout: create 168/33 file after the snapshot 2023-10-30T12:31:56.723 INFO:tasks.workunit.client.0.smithi089.stdout: create 168/34 file after the snapshot 2023-10-30T12:31:56.724 INFO:tasks.workunit.client.0.smithi089.stdout: create 168/35 file after the snapshot 2023-10-30T12:31:56.724 INFO:tasks.workunit.client.0.smithi089.stdout: create 168/36 file after the snapshot 2023-10-30T12:31:56.725 INFO:tasks.workunit.client.0.smithi089.stdout: create 168/37 file after the snapshot 2023-10-30T12:31:56.726 INFO:tasks.workunit.client.0.smithi089.stdout: create 168/38 file after the snapshot 2023-10-30T12:31:56.727 INFO:tasks.workunit.client.0.smithi089.stdout: create 168/39 file after the snapshot 2023-10-30T12:31:56.727 INFO:tasks.workunit.client.0.smithi089.stdout: create 169/.snap/snap-subdir-test 2023-10-30T12:31:56.755 INFO:tasks.workunit.client.0.smithi089.stdout: create 169/30 file after the snapshot 2023-10-30T12:31:56.757 INFO:tasks.workunit.client.0.smithi089.stdout: create 169/31 file after the snapshot 2023-10-30T12:31:56.757 INFO:tasks.workunit.client.0.smithi089.stdout: create 169/32 file after the snapshot 2023-10-30T12:31:56.758 INFO:tasks.workunit.client.0.smithi089.stdout: create 169/33 file after the snapshot 2023-10-30T12:31:56.759 INFO:tasks.workunit.client.0.smithi089.stdout: create 169/34 file after the snapshot 2023-10-30T12:31:56.760 INFO:tasks.workunit.client.0.smithi089.stdout: create 169/35 file after the snapshot 2023-10-30T12:31:56.761 INFO:tasks.workunit.client.0.smithi089.stdout: create 169/36 file after the snapshot 2023-10-30T12:31:56.761 INFO:tasks.workunit.client.0.smithi089.stdout: create 169/37 file after the snapshot 2023-10-30T12:31:56.762 INFO:tasks.workunit.client.0.smithi089.stdout: create 169/38 file after the snapshot 2023-10-30T12:31:56.763 INFO:tasks.workunit.client.0.smithi089.stdout: create 169/39 file after the snapshot 2023-10-30T12:31:56.764 INFO:tasks.workunit.client.0.smithi089.stdout: create 170/.snap/snap-subdir-test 2023-10-30T12:31:56.781 INFO:tasks.workunit.client.0.smithi089.stdout: create 170/30 file after the snapshot 2023-10-30T12:31:56.782 INFO:tasks.workunit.client.0.smithi089.stdout: create 170/31 file after the snapshot 2023-10-30T12:31:56.782 INFO:tasks.workunit.client.0.smithi089.stdout: create 170/32 file after the snapshot 2023-10-30T12:31:56.783 INFO:tasks.workunit.client.0.smithi089.stdout: create 170/33 file after the snapshot 2023-10-30T12:31:56.784 INFO:tasks.workunit.client.0.smithi089.stdout: create 170/34 file after the snapshot 2023-10-30T12:31:56.785 INFO:tasks.workunit.client.0.smithi089.stdout: create 170/35 file after the snapshot 2023-10-30T12:31:56.786 INFO:tasks.workunit.client.0.smithi089.stdout: create 170/36 file after the snapshot 2023-10-30T12:31:56.786 INFO:tasks.workunit.client.0.smithi089.stdout: create 170/37 file after the snapshot 2023-10-30T12:31:56.787 INFO:tasks.workunit.client.0.smithi089.stdout: create 170/38 file after the snapshot 2023-10-30T12:31:56.788 INFO:tasks.workunit.client.0.smithi089.stdout: create 170/39 file after the snapshot 2023-10-30T12:31:56.788 INFO:tasks.workunit.client.0.smithi089.stdout: create 171/.snap/snap-subdir-test 2023-10-30T12:31:56.808 INFO:tasks.workunit.client.0.smithi089.stdout: create 171/30 file after the snapshot 2023-10-30T12:31:56.809 INFO:tasks.workunit.client.0.smithi089.stdout: create 171/31 file after the snapshot 2023-10-30T12:31:56.810 INFO:tasks.workunit.client.0.smithi089.stdout: create 171/32 file after the snapshot 2023-10-30T12:31:56.810 INFO:tasks.workunit.client.0.smithi089.stdout: create 171/33 file after the snapshot 2023-10-30T12:31:56.811 INFO:tasks.workunit.client.0.smithi089.stdout: create 171/34 file after the snapshot 2023-10-30T12:31:56.812 INFO:tasks.workunit.client.0.smithi089.stdout: create 171/35 file after the snapshot 2023-10-30T12:31:56.812 INFO:tasks.workunit.client.0.smithi089.stdout: create 171/36 file after the snapshot 2023-10-30T12:31:56.813 INFO:tasks.workunit.client.0.smithi089.stdout: create 171/37 file after the snapshot 2023-10-30T12:31:56.814 INFO:tasks.workunit.client.0.smithi089.stdout: create 171/38 file after the snapshot 2023-10-30T12:31:56.815 INFO:tasks.workunit.client.0.smithi089.stdout: create 171/39 file after the snapshot 2023-10-30T12:31:56.815 INFO:tasks.workunit.client.0.smithi089.stdout: create 172/.snap/snap-subdir-test 2023-10-30T12:31:56.832 INFO:tasks.workunit.client.0.smithi089.stdout: create 172/30 file after the snapshot 2023-10-30T12:31:56.833 INFO:tasks.workunit.client.0.smithi089.stdout: create 172/31 file after the snapshot 2023-10-30T12:31:56.833 INFO:tasks.workunit.client.0.smithi089.stdout: create 172/32 file after the snapshot 2023-10-30T12:31:56.834 INFO:tasks.workunit.client.0.smithi089.stdout: create 172/33 file after the snapshot 2023-10-30T12:31:56.835 INFO:tasks.workunit.client.0.smithi089.stdout: create 172/34 file after the snapshot 2023-10-30T12:31:56.836 INFO:tasks.workunit.client.0.smithi089.stdout: create 172/35 file after the snapshot 2023-10-30T12:31:56.836 INFO:tasks.workunit.client.0.smithi089.stdout: create 172/36 file after the snapshot 2023-10-30T12:31:56.837 INFO:tasks.workunit.client.0.smithi089.stdout: create 172/37 file after the snapshot 2023-10-30T12:31:56.838 INFO:tasks.workunit.client.0.smithi089.stdout: create 172/38 file after the snapshot 2023-10-30T12:31:56.838 INFO:tasks.workunit.client.0.smithi089.stdout: create 172/39 file after the snapshot 2023-10-30T12:31:56.839 INFO:tasks.workunit.client.0.smithi089.stdout: create 173/.snap/snap-subdir-test 2023-10-30T12:31:56.858 INFO:tasks.workunit.client.0.smithi089.stdout: create 173/30 file after the snapshot 2023-10-30T12:31:56.859 INFO:tasks.workunit.client.0.smithi089.stdout: create 173/31 file after the snapshot 2023-10-30T12:31:56.860 INFO:tasks.workunit.client.0.smithi089.stdout: create 173/32 file after the snapshot 2023-10-30T12:31:56.860 INFO:tasks.workunit.client.0.smithi089.stdout: create 173/33 file after the snapshot 2023-10-30T12:31:56.861 INFO:tasks.workunit.client.0.smithi089.stdout: create 173/34 file after the snapshot 2023-10-30T12:31:56.862 INFO:tasks.workunit.client.0.smithi089.stdout: create 173/35 file after the snapshot 2023-10-30T12:31:56.862 INFO:tasks.workunit.client.0.smithi089.stdout: create 173/36 file after the snapshot 2023-10-30T12:31:56.863 INFO:tasks.workunit.client.0.smithi089.stdout: create 173/37 file after the snapshot 2023-10-30T12:31:56.864 INFO:tasks.workunit.client.0.smithi089.stdout: create 173/38 file after the snapshot 2023-10-30T12:31:56.864 INFO:tasks.workunit.client.0.smithi089.stdout: create 173/39 file after the snapshot 2023-10-30T12:31:56.865 INFO:tasks.workunit.client.0.smithi089.stdout: create 174/.snap/snap-subdir-test 2023-10-30T12:31:56.884 INFO:tasks.workunit.client.0.smithi089.stdout: create 174/30 file after the snapshot 2023-10-30T12:31:56.884 INFO:tasks.workunit.client.0.smithi089.stdout: create 174/31 file after the snapshot 2023-10-30T12:31:56.885 INFO:tasks.workunit.client.0.smithi089.stdout: create 174/32 file after the snapshot 2023-10-30T12:31:56.886 INFO:tasks.workunit.client.0.smithi089.stdout: create 174/33 file after the snapshot 2023-10-30T12:31:56.887 INFO:tasks.workunit.client.0.smithi089.stdout: create 174/34 file after the snapshot 2023-10-30T12:31:56.887 INFO:tasks.workunit.client.0.smithi089.stdout: create 174/35 file after the snapshot 2023-10-30T12:31:56.888 INFO:tasks.workunit.client.0.smithi089.stdout: create 174/36 file after the snapshot 2023-10-30T12:31:56.889 INFO:tasks.workunit.client.0.smithi089.stdout: create 174/37 file after the snapshot 2023-10-30T12:31:56.889 INFO:tasks.workunit.client.0.smithi089.stdout: create 174/38 file after the snapshot 2023-10-30T12:31:56.890 INFO:tasks.workunit.client.0.smithi089.stdout: create 174/39 file after the snapshot 2023-10-30T12:31:56.891 INFO:tasks.workunit.client.0.smithi089.stdout: create 175/.snap/snap-subdir-test 2023-10-30T12:31:56.912 INFO:tasks.workunit.client.0.smithi089.stdout: create 175/30 file after the snapshot 2023-10-30T12:31:56.913 INFO:tasks.workunit.client.0.smithi089.stdout: create 175/31 file after the snapshot 2023-10-30T12:31:56.914 INFO:tasks.workunit.client.0.smithi089.stdout: create 175/32 file after the snapshot 2023-10-30T12:31:56.914 INFO:tasks.workunit.client.0.smithi089.stdout: create 175/33 file after the snapshot 2023-10-30T12:31:56.915 INFO:tasks.workunit.client.0.smithi089.stdout: create 175/34 file after the snapshot 2023-10-30T12:31:56.916 INFO:tasks.workunit.client.0.smithi089.stdout: create 175/35 file after the snapshot 2023-10-30T12:31:56.916 INFO:tasks.workunit.client.0.smithi089.stdout: create 175/36 file after the snapshot 2023-10-30T12:31:56.917 INFO:tasks.workunit.client.0.smithi089.stdout: create 175/37 file after the snapshot 2023-10-30T12:31:56.918 INFO:tasks.workunit.client.0.smithi089.stdout: create 175/38 file after the snapshot 2023-10-30T12:31:56.919 INFO:tasks.workunit.client.0.smithi089.stdout: create 175/39 file after the snapshot 2023-10-30T12:31:56.919 INFO:tasks.workunit.client.0.smithi089.stdout: create 176/.snap/snap-subdir-test 2023-10-30T12:31:56.937 INFO:tasks.workunit.client.0.smithi089.stdout: create 176/30 file after the snapshot 2023-10-30T12:31:56.938 INFO:tasks.workunit.client.0.smithi089.stdout: create 176/31 file after the snapshot 2023-10-30T12:31:56.939 INFO:tasks.workunit.client.0.smithi089.stdout: create 176/32 file after the snapshot 2023-10-30T12:31:56.939 INFO:tasks.workunit.client.0.smithi089.stdout: create 176/33 file after the snapshot 2023-10-30T12:31:56.940 INFO:tasks.workunit.client.0.smithi089.stdout: create 176/34 file after the snapshot 2023-10-30T12:31:56.941 INFO:tasks.workunit.client.0.smithi089.stdout: create 176/35 file after the snapshot 2023-10-30T12:31:56.942 INFO:tasks.workunit.client.0.smithi089.stdout: create 176/36 file after the snapshot 2023-10-30T12:31:56.943 INFO:tasks.workunit.client.0.smithi089.stdout: create 176/37 file after the snapshot 2023-10-30T12:31:56.944 INFO:tasks.workunit.client.0.smithi089.stdout: create 176/38 file after the snapshot 2023-10-30T12:31:56.944 INFO:tasks.workunit.client.0.smithi089.stdout: create 176/39 file after the snapshot 2023-10-30T12:31:56.945 INFO:tasks.workunit.client.0.smithi089.stdout: create 177/.snap/snap-subdir-test 2023-10-30T12:31:56.964 INFO:tasks.workunit.client.0.smithi089.stdout: create 177/30 file after the snapshot 2023-10-30T12:31:56.966 INFO:tasks.workunit.client.0.smithi089.stdout: create 177/31 file after the snapshot 2023-10-30T12:31:56.966 INFO:tasks.workunit.client.0.smithi089.stdout: create 177/32 file after the snapshot 2023-10-30T12:31:56.967 INFO:tasks.workunit.client.0.smithi089.stdout: create 177/33 file after the snapshot 2023-10-30T12:31:56.968 INFO:tasks.workunit.client.0.smithi089.stdout: create 177/34 file after the snapshot 2023-10-30T12:31:56.968 INFO:tasks.workunit.client.0.smithi089.stdout: create 177/35 file after the snapshot 2023-10-30T12:31:56.969 INFO:tasks.workunit.client.0.smithi089.stdout: create 177/36 file after the snapshot 2023-10-30T12:31:56.970 INFO:tasks.workunit.client.0.smithi089.stdout: create 177/37 file after the snapshot 2023-10-30T12:31:56.970 INFO:tasks.workunit.client.0.smithi089.stdout: create 177/38 file after the snapshot 2023-10-30T12:31:56.971 INFO:tasks.workunit.client.0.smithi089.stdout: create 177/39 file after the snapshot 2023-10-30T12:31:56.972 INFO:tasks.workunit.client.0.smithi089.stdout: create 178/.snap/snap-subdir-test 2023-10-30T12:31:56.992 INFO:tasks.workunit.client.0.smithi089.stdout: create 178/30 file after the snapshot 2023-10-30T12:31:56.994 INFO:tasks.workunit.client.0.smithi089.stdout: create 178/31 file after the snapshot 2023-10-30T12:31:56.996 INFO:tasks.workunit.client.0.smithi089.stdout: create 178/32 file after the snapshot 2023-10-30T12:31:56.998 INFO:tasks.workunit.client.0.smithi089.stdout: create 178/33 file after the snapshot 2023-10-30T12:31:57.000 INFO:tasks.workunit.client.0.smithi089.stdout: create 178/34 file after the snapshot 2023-10-30T12:31:57.001 INFO:tasks.workunit.client.0.smithi089.stdout: create 178/35 file after the snapshot 2023-10-30T12:31:57.003 INFO:tasks.workunit.client.0.smithi089.stdout: create 178/36 file after the snapshot 2023-10-30T12:31:57.005 INFO:tasks.workunit.client.0.smithi089.stdout: create 178/37 file after the snapshot 2023-10-30T12:31:57.006 INFO:tasks.workunit.client.0.smithi089.stdout: create 178/38 file after the snapshot 2023-10-30T12:31:57.008 INFO:tasks.workunit.client.0.smithi089.stdout: create 178/39 file after the snapshot 2023-10-30T12:31:57.009 INFO:tasks.workunit.client.0.smithi089.stdout: create 179/.snap/snap-subdir-test 2023-10-30T12:31:57.029 INFO:tasks.workunit.client.0.smithi089.stdout: create 179/30 file after the snapshot 2023-10-30T12:31:57.031 INFO:tasks.workunit.client.0.smithi089.stdout: create 179/31 file after the snapshot 2023-10-30T12:31:57.033 INFO:tasks.workunit.client.0.smithi089.stdout: create 179/32 file after the snapshot 2023-10-30T12:31:57.035 INFO:tasks.workunit.client.0.smithi089.stdout: create 179/33 file after the snapshot 2023-10-30T12:31:57.037 INFO:tasks.workunit.client.0.smithi089.stdout: create 179/34 file after the snapshot 2023-10-30T12:31:57.038 INFO:tasks.workunit.client.0.smithi089.stdout: create 179/35 file after the snapshot 2023-10-30T12:31:57.040 INFO:tasks.workunit.client.0.smithi089.stdout: create 179/36 file after the snapshot 2023-10-30T12:31:57.042 INFO:tasks.workunit.client.0.smithi089.stdout: create 179/37 file after the snapshot 2023-10-30T12:31:57.044 INFO:tasks.workunit.client.0.smithi089.stdout: create 179/38 file after the snapshot 2023-10-30T12:31:57.046 INFO:tasks.workunit.client.0.smithi089.stdout: create 179/39 file after the snapshot 2023-10-30T12:31:57.048 INFO:tasks.workunit.client.0.smithi089.stdout: create 180/.snap/snap-subdir-test 2023-10-30T12:31:57.077 INFO:tasks.workunit.client.0.smithi089.stdout: create 180/30 file after the snapshot 2023-10-30T12:31:57.080 INFO:tasks.workunit.client.0.smithi089.stdout: create 180/31 file after the snapshot 2023-10-30T12:31:57.082 INFO:tasks.workunit.client.0.smithi089.stdout: create 180/32 file after the snapshot 2023-10-30T12:31:57.084 INFO:tasks.workunit.client.0.smithi089.stdout: create 180/33 file after the snapshot 2023-10-30T12:31:57.085 INFO:tasks.workunit.client.0.smithi089.stdout: create 180/34 file after the snapshot 2023-10-30T12:31:57.087 INFO:tasks.workunit.client.0.smithi089.stdout: create 180/35 file after the snapshot 2023-10-30T12:31:57.089 INFO:tasks.workunit.client.0.smithi089.stdout: create 180/36 file after the snapshot 2023-10-30T12:31:57.092 INFO:tasks.workunit.client.0.smithi089.stdout: create 180/37 file after the snapshot 2023-10-30T12:31:57.094 INFO:tasks.workunit.client.0.smithi089.stdout: create 180/38 file after the snapshot 2023-10-30T12:31:57.096 INFO:tasks.workunit.client.0.smithi089.stdout: create 180/39 file after the snapshot 2023-10-30T12:31:57.098 INFO:tasks.workunit.client.0.smithi089.stdout: create 181/.snap/snap-subdir-test 2023-10-30T12:31:57.119 INFO:tasks.workunit.client.0.smithi089.stdout: create 181/30 file after the snapshot 2023-10-30T12:31:57.120 INFO:tasks.workunit.client.0.smithi089.stdout: create 181/31 file after the snapshot 2023-10-30T12:31:57.122 INFO:tasks.workunit.client.0.smithi089.stdout: create 181/32 file after the snapshot 2023-10-30T12:31:57.124 INFO:tasks.workunit.client.0.smithi089.stdout: create 181/33 file after the snapshot 2023-10-30T12:31:57.125 INFO:tasks.workunit.client.0.smithi089.stdout: create 181/34 file after the snapshot 2023-10-30T12:31:57.130 INFO:tasks.workunit.client.0.smithi089.stdout: create 181/35 file after the snapshot 2023-10-30T12:31:57.130 INFO:tasks.workunit.client.0.smithi089.stdout: create 181/36 file after the snapshot 2023-10-30T12:31:57.131 INFO:tasks.workunit.client.0.smithi089.stdout: create 181/37 file after the snapshot 2023-10-30T12:31:57.132 INFO:tasks.workunit.client.0.smithi089.stdout: create 181/38 file after the snapshot 2023-10-30T12:31:57.134 INFO:tasks.workunit.client.0.smithi089.stdout: create 181/39 file after the snapshot 2023-10-30T12:31:57.136 INFO:tasks.workunit.client.0.smithi089.stdout: create 182/.snap/snap-subdir-test 2023-10-30T12:31:57.165 INFO:tasks.workunit.client.0.smithi089.stdout: create 182/30 file after the snapshot 2023-10-30T12:31:57.167 INFO:tasks.workunit.client.0.smithi089.stdout: create 182/31 file after the snapshot 2023-10-30T12:31:57.169 INFO:tasks.workunit.client.0.smithi089.stdout: create 182/32 file after the snapshot 2023-10-30T12:31:57.171 INFO:tasks.workunit.client.0.smithi089.stdout: create 182/33 file after the snapshot 2023-10-30T12:31:57.173 INFO:tasks.workunit.client.0.smithi089.stdout: create 182/34 file after the snapshot 2023-10-30T12:31:57.175 INFO:tasks.workunit.client.0.smithi089.stdout: create 182/35 file after the snapshot 2023-10-30T12:31:57.177 INFO:tasks.workunit.client.0.smithi089.stdout: create 182/36 file after the snapshot 2023-10-30T12:31:57.178 INFO:tasks.workunit.client.0.smithi089.stdout: create 182/37 file after the snapshot 2023-10-30T12:31:57.180 INFO:tasks.workunit.client.0.smithi089.stdout: create 182/38 file after the snapshot 2023-10-30T12:31:57.182 INFO:tasks.workunit.client.0.smithi089.stdout: create 182/39 file after the snapshot 2023-10-30T12:31:57.184 INFO:tasks.workunit.client.0.smithi089.stdout: create 183/.snap/snap-subdir-test 2023-10-30T12:31:57.201 INFO:tasks.workunit.client.0.smithi089.stdout: create 183/30 file after the snapshot 2023-10-30T12:31:57.202 INFO:tasks.workunit.client.0.smithi089.stdout: create 183/31 file after the snapshot 2023-10-30T12:31:57.204 INFO:tasks.workunit.client.0.smithi089.stdout: create 183/32 file after the snapshot 2023-10-30T12:31:57.206 INFO:tasks.workunit.client.0.smithi089.stdout: create 183/33 file after the snapshot 2023-10-30T12:31:57.207 INFO:tasks.workunit.client.0.smithi089.stdout: create 183/34 file after the snapshot 2023-10-30T12:31:57.209 INFO:tasks.workunit.client.0.smithi089.stdout: create 183/35 file after the snapshot 2023-10-30T12:31:57.210 INFO:tasks.workunit.client.0.smithi089.stdout: create 183/36 file after the snapshot 2023-10-30T12:31:57.212 INFO:tasks.workunit.client.0.smithi089.stdout: create 183/37 file after the snapshot 2023-10-30T12:31:57.214 INFO:tasks.workunit.client.0.smithi089.stdout: create 183/38 file after the snapshot 2023-10-30T12:31:57.216 INFO:tasks.workunit.client.0.smithi089.stdout: create 183/39 file after the snapshot 2023-10-30T12:31:57.217 INFO:tasks.workunit.client.0.smithi089.stdout: create 184/.snap/snap-subdir-test 2023-10-30T12:31:58.173 INFO:tasks.workunit.client.0.smithi089.stdout: create 184/30 file after the snapshot 2023-10-30T12:31:58.177 INFO:tasks.workunit.client.0.smithi089.stdout: create 184/31 file after the snapshot 2023-10-30T12:31:58.178 INFO:tasks.workunit.client.0.smithi089.stdout: create 184/32 file after the snapshot 2023-10-30T12:31:58.180 INFO:tasks.workunit.client.0.smithi089.stdout: create 184/33 file after the snapshot 2023-10-30T12:31:58.182 INFO:tasks.workunit.client.0.smithi089.stdout: create 184/34 file after the snapshot 2023-10-30T12:31:58.184 INFO:tasks.workunit.client.0.smithi089.stdout: create 184/35 file after the snapshot 2023-10-30T12:31:58.186 INFO:tasks.workunit.client.0.smithi089.stdout: create 184/36 file after the snapshot 2023-10-30T12:31:58.188 INFO:tasks.workunit.client.0.smithi089.stdout: create 184/37 file after the snapshot 2023-10-30T12:31:58.189 INFO:tasks.workunit.client.0.smithi089.stdout: create 184/38 file after the snapshot 2023-10-30T12:31:58.191 INFO:tasks.workunit.client.0.smithi089.stdout: create 184/39 file after the snapshot 2023-10-30T12:31:58.193 INFO:tasks.workunit.client.0.smithi089.stdout: create 185/.snap/snap-subdir-test 2023-10-30T12:31:58.217 INFO:tasks.workunit.client.0.smithi089.stdout: create 185/30 file after the snapshot 2023-10-30T12:31:58.219 INFO:tasks.workunit.client.0.smithi089.stdout: create 185/31 file after the snapshot 2023-10-30T12:31:58.221 INFO:tasks.workunit.client.0.smithi089.stdout: create 185/32 file after the snapshot 2023-10-30T12:31:58.223 INFO:tasks.workunit.client.0.smithi089.stdout: create 185/33 file after the snapshot 2023-10-30T12:31:58.225 INFO:tasks.workunit.client.0.smithi089.stdout: create 185/34 file after the snapshot 2023-10-30T12:31:58.227 INFO:tasks.workunit.client.0.smithi089.stdout: create 185/35 file after the snapshot 2023-10-30T12:31:58.229 INFO:tasks.workunit.client.0.smithi089.stdout: create 185/36 file after the snapshot 2023-10-30T12:31:58.230 INFO:tasks.workunit.client.0.smithi089.stdout: create 185/37 file after the snapshot 2023-10-30T12:31:58.232 INFO:tasks.workunit.client.0.smithi089.stdout: create 185/38 file after the snapshot 2023-10-30T12:31:58.234 INFO:tasks.workunit.client.0.smithi089.stdout: create 185/39 file after the snapshot 2023-10-30T12:31:58.236 INFO:tasks.workunit.client.0.smithi089.stdout: create 186/.snap/snap-subdir-test 2023-10-30T12:31:58.254 INFO:tasks.workunit.client.0.smithi089.stdout: create 186/30 file after the snapshot 2023-10-30T12:31:58.255 INFO:tasks.workunit.client.0.smithi089.stdout: create 186/31 file after the snapshot 2023-10-30T12:31:58.257 INFO:tasks.workunit.client.0.smithi089.stdout: create 186/32 file after the snapshot 2023-10-30T12:31:58.259 INFO:tasks.workunit.client.0.smithi089.stdout: create 186/33 file after the snapshot 2023-10-30T12:31:58.260 INFO:tasks.workunit.client.0.smithi089.stdout: create 186/34 file after the snapshot 2023-10-30T12:31:58.263 INFO:tasks.workunit.client.0.smithi089.stdout: create 186/35 file after the snapshot 2023-10-30T12:31:58.266 INFO:tasks.workunit.client.0.smithi089.stdout: create 186/36 file after the snapshot 2023-10-30T12:31:58.269 INFO:tasks.workunit.client.0.smithi089.stdout: create 186/37 file after the snapshot 2023-10-30T12:31:58.271 INFO:tasks.workunit.client.0.smithi089.stdout: create 186/38 file after the snapshot 2023-10-30T12:31:58.274 INFO:tasks.workunit.client.0.smithi089.stdout: create 186/39 file after the snapshot 2023-10-30T12:31:58.277 INFO:tasks.workunit.client.0.smithi089.stdout: create 187/.snap/snap-subdir-test 2023-10-30T12:31:58.722 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2023-10-30T12:31:58.723 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-30T12:31:58.723 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-30T12:31:58.726 INFO:tasks.ceph.mds.b:Started 2023-10-30T12:31:58.727 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:31:58.803 INFO:tasks.ceph.mds.b.smithi192.stdout:starting mds.b at 2023-10-30T12:31:58.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.871+0000 7f9096691640 1 Processor -- start 2023-10-30T12:31:58.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.871+0000 7f9096691640 1 -- start start 2023-10-30T12:31:58.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.871+0000 7f9096691640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f909006f7a0 0x7f909006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:58.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.871+0000 7f9096691640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9090066cf0 con 0x7f9090071110 2023-10-30T12:31:58.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.871+0000 7f9096691640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90900c70d0 con 0x7f90900c6bd0 2023-10-30T12:31:58.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.871+0000 7f9096691640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90900700c0 con 0x7f909006f7a0 2023-10-30T12:31:58.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.871+0000 7f908ffff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f909006f7a0 0x7f909006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:58.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.871+0000 7f908ffff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f909006f7a0 0x7f909006fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52914/0 (socket says 172.21.15.89:52914) 2023-10-30T12:31:58.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.871+0000 7f908ffff640 1 -- 172.21.15.89:0/2834273032 learned_addr learned my addr 172.21.15.89:0/2834273032 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:58.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.871+0000 7f908ffff640 1 -- 172.21.15.89:0/2834273032 >> v1:172.21.15.192:6789/0 conn(0x7f90900c6bd0 legacy=0x7f9090069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:58.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.871+0000 7f908ffff640 1 -- 172.21.15.89:0/2834273032 >> v1:172.21.15.89:6789/0 conn(0x7f9090071110 legacy=0x7f9090158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:58.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.871+0000 7f908ffff640 1 -- 172.21.15.89:0/2834273032 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90900c7390 con 0x7f909006f7a0 2023-10-30T12:31:58.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.875+0000 7f908ffff640 1 --2- 172.21.15.89:0/2834273032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f909006f7a0 0x7f909006fb80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9080009870 tx=0x7f908001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0244f211f00181e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:58.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.875+0000 7f908effd640 1 -- 172.21.15.89:0/2834273032 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90800a8a30 con 0x7f909006f7a0 2023-10-30T12:31:58.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.875+0000 7f908effd640 1 -- 172.21.15.89:0/2834273032 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90800a8b90 con 0x7f909006f7a0 2023-10-30T12:31:58.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.875+0000 7f908effd640 1 -- 172.21.15.89:0/2834273032 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90800b0440 con 0x7f909006f7a0 2023-10-30T12:31:58.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.875+0000 7f9096691640 1 -- 172.21.15.89:0/2834273032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f909006f7a0 msgr2=0x7f909006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:58.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.875+0000 7f9096691640 1 --2- 172.21.15.89:0/2834273032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f909006f7a0 0x7f909006fb80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9080009870 tx=0x7f908001ed30 comp rx=0 tx=0).stop 2023-10-30T12:31:58.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.875+0000 7f9096691640 1 -- 172.21.15.89:0/2834273032 shutdown_connections 2023-10-30T12:31:58.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.875+0000 7f9096691640 1 --2- 172.21.15.89:0/2834273032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f909006f7a0 0x7f909006fb80 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:58.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.875+0000 7f9096691640 1 -- 172.21.15.89:0/2834273032 >> 172.21.15.89:0/2834273032 conn(0x7f90900c7910 msgr2=0x7f90900c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:58.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.875+0000 7f9096691640 1 -- 172.21.15.89:0/2834273032 shutdown_connections 2023-10-30T12:31:58.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.875+0000 7f9096691640 1 -- 172.21.15.89:0/2834273032 wait complete. 2023-10-30T12:31:58.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.879+0000 7f9096691640 1 Processor -- start 2023-10-30T12:31:58.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.879+0000 7f9096691640 1 -- start start 2023-10-30T12:31:58.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.879+0000 7f9096691640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f909006f7a0 0x7f90901464b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:58.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.879+0000 7f9096691640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9090071110 0x7f90901e2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:58.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.879+0000 7f9096691640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90900c6bd0 0x7f9090140af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:58.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.879+0000 7f9096691640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90901f2620 con 0x7f90900c6bd0 2023-10-30T12:31:58.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.879+0000 7f9096691640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9090153ee0 con 0x7f909006f7a0 2023-10-30T12:31:58.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.879+0000 7f9096691640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9090154150 con 0x7f9090071110 2023-10-30T12:31:58.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.879+0000 7f9094c2e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90900c6bd0 0x7f9090140af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:58.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.879+0000 7f9094c2e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90900c6bd0 0x7f9090140af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52930/0 (socket says 172.21.15.89:52930) 2023-10-30T12:31:58.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.879+0000 7f9094c2e640 1 -- 172.21.15.89:0/3438297488 learned_addr learned my addr 172.21.15.89:0/3438297488 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:31:58.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.879+0000 7f908ffff640 1 --2- 172.21.15.89:0/3438297488 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f909006f7a0 0x7f90901464b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:58.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.879+0000 7f908f7fe640 1 --2- 172.21.15.89:0/3438297488 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9090071110 0x7f90901e2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:58.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.879+0000 7f9094c2e640 1 -- 172.21.15.89:0/3438297488 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9090071110 msgr2=0x7f90901e2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:58.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.879+0000 7f9094c2e640 1 --2- 172.21.15.89:0/3438297488 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9090071110 0x7f90901e2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:58.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.879+0000 7f9094c2e640 1 -- 172.21.15.89:0/3438297488 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f909006f7a0 msgr2=0x7f90901464b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:58.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.879+0000 7f9094c2e640 1 --2- 172.21.15.89:0/3438297488 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f909006f7a0 0x7f90901464b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:58.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.879+0000 7f9094c2e640 1 -- 172.21.15.89:0/3438297488 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90800094d0 con 0x7f90900c6bd0 2023-10-30T12:31:58.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.879+0000 7f9094c2e640 1 --2- 172.21.15.89:0/3438297488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90900c6bd0 0x7f9090140af0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f908800c730 tx=0x7f908800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:58.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.879+0000 7f908cff9640 1 -- 172.21.15.89:0/3438297488 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9088003b80 con 0x7f90900c6bd0 2023-10-30T12:31:58.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.879+0000 7f908cff9640 1 -- 172.21.15.89:0/3438297488 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9088003ce0 con 0x7f90900c6bd0 2023-10-30T12:31:58.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.879+0000 7f908cff9640 1 -- 172.21.15.89:0/3438297488 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908800aa30 con 0x7f90900c6bd0 2023-10-30T12:31:58.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.879+0000 7f9096691640 1 -- 172.21.15.89:0/3438297488 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9090154430 con 0x7f90900c6bd0 2023-10-30T12:31:58.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.879+0000 7f9096691640 1 -- 172.21.15.89:0/3438297488 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90901e3570 con 0x7f90900c6bd0 2023-10-30T12:31:58.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.883+0000 7f9096691640 1 -- 172.21.15.89:0/3438297488 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90901f2790 con 0x7f90900c6bd0 2023-10-30T12:31:58.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.883+0000 7f908cff9640 1 -- 172.21.15.89:0/3438297488 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f908800ab90 con 0x7f90900c6bd0 2023-10-30T12:31:58.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.883+0000 7f908cff9640 1 --2- 172.21.15.89:0/3438297488 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f907c0664a0 0x7f907c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:31:58.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.883+0000 7f908cff9640 1 -- 172.21.15.89:0/3438297488 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5890+0+0 (secure 0 0 0) 0x7f9088083d90 con 0x7f90900c6bd0 2023-10-30T12:31:58.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.883+0000 7f908ffff640 1 --2- 172.21.15.89:0/3438297488 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f907c0664a0 0x7f907c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:31:58.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.887+0000 7f908ffff640 1 --2- 172.21.15.89:0/3438297488 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f907c0664a0 0x7f907c068960 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f90800b2000 tx=0x7f9080009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:31:58.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:58.887+0000 7f908cff9640 1 -- 172.21.15.89:0/3438297488 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9088051020 con 0x7f90900c6bd0 2023-10-30T12:31:59.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:59.071+0000 7f9096691640 1 -- 172.21.15.89:0/3438297488 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f909004e170 con 0x7f90900c6bd0 2023-10-30T12:31:59.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:59.071+0000 7f908cff9640 1 -- 172.21.15.89:0/3438297488 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+6531 (secure 0 0 0) 0x7f9088054640 con 0x7f90900c6bd0 2023-10-30T12:31:59.076 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:31:59.076 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4781,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:31:48.285333+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":2,"incarnation":15,"state":"up:active","state_seq":50,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:31:59.077 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 21 2023-10-30T12:31:59.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:59.075+0000 7f9096691640 1 -- 172.21.15.89:0/3438297488 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f907c0664a0 msgr2=0x7f907c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:59.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:59.075+0000 7f9096691640 1 --2- 172.21.15.89:0/3438297488 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f907c0664a0 0x7f907c068960 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f90800b2000 tx=0x7f9080009040 comp rx=0 tx=0).stop 2023-10-30T12:31:59.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:59.075+0000 7f9096691640 1 -- 172.21.15.89:0/3438297488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90900c6bd0 msgr2=0x7f9090140af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:31:59.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:59.075+0000 7f9096691640 1 --2- 172.21.15.89:0/3438297488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90900c6bd0 0x7f9090140af0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f908800c730 tx=0x7f908800cbf0 comp rx=0 tx=0).stop 2023-10-30T12:31:59.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:59.075+0000 7f9096691640 1 -- 172.21.15.89:0/3438297488 shutdown_connections 2023-10-30T12:31:59.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:59.075+0000 7f9096691640 1 --2- 172.21.15.89:0/3438297488 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f907c0664a0 0x7f907c068960 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:59.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:59.075+0000 7f9096691640 1 --2- 172.21.15.89:0/3438297488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90900c6bd0 0x7f9090140af0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:59.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:59.075+0000 7f9096691640 1 --2- 172.21.15.89:0/3438297488 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9090071110 0x7f90901e2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:59.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:59.075+0000 7f9096691640 1 --2- 172.21.15.89:0/3438297488 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f909006f7a0 0x7f90901464b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:31:59.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:59.075+0000 7f9096691640 1 -- 172.21.15.89:0/3438297488 >> 172.21.15.89:0/3438297488 conn(0x7f90900c7910 msgr2=0x7f90900c6380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:31:59.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:59.075+0000 7f9096691640 1 -- 172.21.15.89:0/3438297488 shutdown_connections 2023-10-30T12:31:59.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:31:59.075+0000 7f9096691640 1 -- 172.21.15.89:0/3438297488 wait complete. 2023-10-30T12:31:59.081 INFO:tasks.workunit.client.0.smithi089.stdout: create 187/30 file after the snapshot 2023-10-30T12:31:59.081 INFO:tasks.workunit.client.0.smithi089.stdout: create 187/31 file after the snapshot 2023-10-30T12:31:59.083 INFO:tasks.workunit.client.0.smithi089.stdout: create 187/32 file after the snapshot 2023-10-30T12:31:59.084 INFO:tasks.workunit.client.0.smithi089.stdout: create 187/33 file after the snapshot 2023-10-30T12:31:59.086 INFO:tasks.workunit.client.0.smithi089.stdout: create 187/34 file after the snapshot 2023-10-30T12:31:59.088 INFO:tasks.workunit.client.0.smithi089.stdout: create 187/35 file after the snapshot 2023-10-30T12:31:59.090 INFO:tasks.workunit.client.0.smithi089.stdout: create 187/36 file after the snapshot 2023-10-30T12:31:59.091 INFO:tasks.workunit.client.0.smithi089.stdout: create 187/37 file after the snapshot 2023-10-30T12:31:59.094 INFO:tasks.workunit.client.0.smithi089.stdout: create 187/38 file after the snapshot 2023-10-30T12:31:59.095 INFO:tasks.workunit.client.0.smithi089.stdout: create 187/39 file after the snapshot 2023-10-30T12:31:59.098 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2023-10-30T12:31:59.098 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 53 secs before thrashing 2023-10-30T12:31:59.098 INFO:tasks.workunit.client.0.smithi089.stdout: create 188/.snap/snap-subdir-test 2023-10-30T12:32:00.114 INFO:tasks.workunit.client.0.smithi089.stdout: create 188/30 file after the snapshot 2023-10-30T12:32:00.116 INFO:tasks.workunit.client.0.smithi089.stdout: create 188/31 file after the snapshot 2023-10-30T12:32:00.119 INFO:tasks.workunit.client.0.smithi089.stdout: create 188/32 file after the snapshot 2023-10-30T12:32:00.122 INFO:tasks.workunit.client.0.smithi089.stdout: create 188/33 file after the snapshot 2023-10-30T12:32:00.124 INFO:tasks.workunit.client.0.smithi089.stdout: create 188/34 file after the snapshot 2023-10-30T12:32:00.127 INFO:tasks.workunit.client.0.smithi089.stdout: create 188/35 file after the snapshot 2023-10-30T12:32:00.129 INFO:tasks.workunit.client.0.smithi089.stdout: create 188/36 file after the snapshot 2023-10-30T12:32:00.132 INFO:tasks.workunit.client.0.smithi089.stdout: create 188/37 file after the snapshot 2023-10-30T12:32:00.386 INFO:tasks.workunit.client.0.smithi089.stdout: create 188/38 file after the snapshot 2023-10-30T12:32:00.388 INFO:tasks.workunit.client.0.smithi089.stdout: create 188/39 file after the snapshot 2023-10-30T12:32:00.391 INFO:tasks.workunit.client.0.smithi089.stdout: create 189/.snap/snap-subdir-test 2023-10-30T12:32:00.414 INFO:tasks.workunit.client.0.smithi089.stdout: create 189/30 file after the snapshot 2023-10-30T12:32:00.416 INFO:tasks.workunit.client.0.smithi089.stdout: create 189/31 file after the snapshot 2023-10-30T12:32:00.419 INFO:tasks.workunit.client.0.smithi089.stdout: create 189/32 file after the snapshot 2023-10-30T12:32:00.422 INFO:tasks.workunit.client.0.smithi089.stdout: create 189/33 file after the snapshot 2023-10-30T12:32:00.424 INFO:tasks.workunit.client.0.smithi089.stdout: create 189/34 file after the snapshot 2023-10-30T12:32:00.427 INFO:tasks.workunit.client.0.smithi089.stdout: create 189/35 file after the snapshot 2023-10-30T12:32:00.429 INFO:tasks.workunit.client.0.smithi089.stdout: create 189/36 file after the snapshot 2023-10-30T12:32:00.432 INFO:tasks.workunit.client.0.smithi089.stdout: create 189/37 file after the snapshot 2023-10-30T12:32:00.905 INFO:tasks.workunit.client.0.smithi089.stdout: create 189/38 file after the snapshot 2023-10-30T12:32:00.907 INFO:tasks.workunit.client.0.smithi089.stdout: create 189/39 file after the snapshot 2023-10-30T12:32:00.909 INFO:tasks.workunit.client.0.smithi089.stdout: create 190/.snap/snap-subdir-test 2023-10-30T12:32:00.944 INFO:tasks.workunit.client.0.smithi089.stdout: create 190/30 file after the snapshot 2023-10-30T12:32:00.946 INFO:tasks.workunit.client.0.smithi089.stdout: create 190/31 file after the snapshot 2023-10-30T12:32:00.947 INFO:tasks.workunit.client.0.smithi089.stdout: create 190/32 file after the snapshot 2023-10-30T12:32:00.949 INFO:tasks.workunit.client.0.smithi089.stdout: create 190/33 file after the snapshot 2023-10-30T12:32:00.951 INFO:tasks.workunit.client.0.smithi089.stdout: create 190/34 file after the snapshot 2023-10-30T12:32:00.953 INFO:tasks.workunit.client.0.smithi089.stdout: create 190/35 file after the snapshot 2023-10-30T12:32:00.954 INFO:tasks.workunit.client.0.smithi089.stdout: create 190/36 file after the snapshot 2023-10-30T12:32:00.956 INFO:tasks.workunit.client.0.smithi089.stdout: create 190/37 file after the snapshot 2023-10-30T12:32:00.959 INFO:tasks.workunit.client.0.smithi089.stdout: create 190/38 file after the snapshot 2023-10-30T12:32:00.961 INFO:tasks.workunit.client.0.smithi089.stdout: create 190/39 file after the snapshot 2023-10-30T12:32:00.963 INFO:tasks.workunit.client.0.smithi089.stdout: create 191/.snap/snap-subdir-test 2023-10-30T12:32:01.962 INFO:tasks.workunit.client.0.smithi089.stdout: create 191/30 file after the snapshot 2023-10-30T12:32:01.969 INFO:tasks.workunit.client.0.smithi089.stdout: create 191/31 file after the snapshot 2023-10-30T12:32:01.971 INFO:tasks.workunit.client.0.smithi089.stdout: create 191/32 file after the snapshot 2023-10-30T12:32:01.972 INFO:tasks.workunit.client.0.smithi089.stdout: create 191/33 file after the snapshot 2023-10-30T12:32:01.975 INFO:tasks.workunit.client.0.smithi089.stdout: create 191/34 file after the snapshot 2023-10-30T12:32:01.977 INFO:tasks.workunit.client.0.smithi089.stdout: create 191/35 file after the snapshot 2023-10-30T12:32:01.979 INFO:tasks.workunit.client.0.smithi089.stdout: create 191/36 file after the snapshot 2023-10-30T12:32:01.980 INFO:tasks.workunit.client.0.smithi089.stdout: create 191/37 file after the snapshot 2023-10-30T12:32:01.983 INFO:tasks.workunit.client.0.smithi089.stdout: create 191/38 file after the snapshot 2023-10-30T12:32:01.985 INFO:tasks.workunit.client.0.smithi089.stdout: create 191/39 file after the snapshot 2023-10-30T12:32:01.987 INFO:tasks.workunit.client.0.smithi089.stdout: create 192/.snap/snap-subdir-test 2023-10-30T12:32:02.014 INFO:tasks.workunit.client.0.smithi089.stdout: create 192/30 file after the snapshot 2023-10-30T12:32:02.017 INFO:tasks.workunit.client.0.smithi089.stdout: create 192/31 file after the snapshot 2023-10-30T12:32:02.019 INFO:tasks.workunit.client.0.smithi089.stdout: create 192/32 file after the snapshot 2023-10-30T12:32:02.021 INFO:tasks.workunit.client.0.smithi089.stdout: create 192/33 file after the snapshot 2023-10-30T12:32:02.023 INFO:tasks.workunit.client.0.smithi089.stdout: create 192/34 file after the snapshot 2023-10-30T12:32:02.025 INFO:tasks.workunit.client.0.smithi089.stdout: create 192/35 file after the snapshot 2023-10-30T12:32:02.027 INFO:tasks.workunit.client.0.smithi089.stdout: create 192/36 file after the snapshot 2023-10-30T12:32:02.029 INFO:tasks.workunit.client.0.smithi089.stdout: create 192/37 file after the snapshot 2023-10-30T12:32:02.031 INFO:tasks.workunit.client.0.smithi089.stdout: create 192/38 file after the snapshot 2023-10-30T12:32:02.032 INFO:tasks.workunit.client.0.smithi089.stdout: create 192/39 file after the snapshot 2023-10-30T12:32:02.034 INFO:tasks.workunit.client.0.smithi089.stdout: create 193/.snap/snap-subdir-test 2023-10-30T12:32:02.050 INFO:tasks.workunit.client.0.smithi089.stdout: create 193/30 file after the snapshot 2023-10-30T12:32:02.052 INFO:tasks.workunit.client.0.smithi089.stdout: create 193/31 file after the snapshot 2023-10-30T12:32:02.054 INFO:tasks.workunit.client.0.smithi089.stdout: create 193/32 file after the snapshot 2023-10-30T12:32:02.055 INFO:tasks.workunit.client.0.smithi089.stdout: create 193/33 file after the snapshot 2023-10-30T12:32:02.057 INFO:tasks.workunit.client.0.smithi089.stdout: create 193/34 file after the snapshot 2023-10-30T12:32:02.058 INFO:tasks.workunit.client.0.smithi089.stdout: create 193/35 file after the snapshot 2023-10-30T12:32:02.060 INFO:tasks.workunit.client.0.smithi089.stdout: create 193/36 file after the snapshot 2023-10-30T12:32:02.061 INFO:tasks.workunit.client.0.smithi089.stdout: create 193/37 file after the snapshot 2023-10-30T12:32:02.063 INFO:tasks.workunit.client.0.smithi089.stdout: create 193/38 file after the snapshot 2023-10-30T12:32:02.064 INFO:tasks.workunit.client.0.smithi089.stdout: create 193/39 file after the snapshot 2023-10-30T12:32:02.066 INFO:tasks.workunit.client.0.smithi089.stdout: create 194/.snap/snap-subdir-test 2023-10-30T12:32:02.100 INFO:tasks.workunit.client.0.smithi089.stdout: create 194/30 file after the snapshot 2023-10-30T12:32:02.102 INFO:tasks.workunit.client.0.smithi089.stdout: create 194/31 file after the snapshot 2023-10-30T12:32:02.104 INFO:tasks.workunit.client.0.smithi089.stdout: create 194/32 file after the snapshot 2023-10-30T12:32:02.106 INFO:tasks.workunit.client.0.smithi089.stdout: create 194/33 file after the snapshot 2023-10-30T12:32:02.108 INFO:tasks.workunit.client.0.smithi089.stdout: create 194/34 file after the snapshot 2023-10-30T12:32:02.109 INFO:tasks.workunit.client.0.smithi089.stdout: create 194/35 file after the snapshot 2023-10-30T12:32:02.111 INFO:tasks.workunit.client.0.smithi089.stdout: create 194/36 file after the snapshot 2023-10-30T12:32:02.113 INFO:tasks.workunit.client.0.smithi089.stdout: create 194/37 file after the snapshot 2023-10-30T12:32:02.114 INFO:tasks.workunit.client.0.smithi089.stdout: create 194/38 file after the snapshot 2023-10-30T12:32:02.116 INFO:tasks.workunit.client.0.smithi089.stdout: create 194/39 file after the snapshot 2023-10-30T12:32:02.118 INFO:tasks.workunit.client.0.smithi089.stdout: create 195/.snap/snap-subdir-test 2023-10-30T12:32:02.137 INFO:tasks.workunit.client.0.smithi089.stdout: create 195/30 file after the snapshot 2023-10-30T12:32:02.140 INFO:tasks.workunit.client.0.smithi089.stdout: create 195/31 file after the snapshot 2023-10-30T12:32:02.142 INFO:tasks.workunit.client.0.smithi089.stdout: create 195/32 file after the snapshot 2023-10-30T12:32:02.145 INFO:tasks.workunit.client.0.smithi089.stdout: create 195/33 file after the snapshot 2023-10-30T12:32:02.147 INFO:tasks.workunit.client.0.smithi089.stdout: create 195/34 file after the snapshot 2023-10-30T12:32:02.150 INFO:tasks.workunit.client.0.smithi089.stdout: create 195/35 file after the snapshot 2023-10-30T12:32:02.152 INFO:tasks.workunit.client.0.smithi089.stdout: create 195/36 file after the snapshot 2023-10-30T12:32:02.155 INFO:tasks.workunit.client.0.smithi089.stdout: create 195/37 file after the snapshot 2023-10-30T12:32:02.158 INFO:tasks.workunit.client.0.smithi089.stdout: create 195/38 file after the snapshot 2023-10-30T12:32:02.160 INFO:tasks.workunit.client.0.smithi089.stdout: create 195/39 file after the snapshot 2023-10-30T12:32:02.161 INFO:tasks.workunit.client.0.smithi089.stdout: create 196/.snap/snap-subdir-test 2023-10-30T12:32:02.185 INFO:tasks.workunit.client.0.smithi089.stdout: create 196/30 file after the snapshot 2023-10-30T12:32:02.187 INFO:tasks.workunit.client.0.smithi089.stdout: create 196/31 file after the snapshot 2023-10-30T12:32:02.189 INFO:tasks.workunit.client.0.smithi089.stdout: create 196/32 file after the snapshot 2023-10-30T12:32:02.191 INFO:tasks.workunit.client.0.smithi089.stdout: create 196/33 file after the snapshot 2023-10-30T12:32:02.192 INFO:tasks.workunit.client.0.smithi089.stdout: create 196/34 file after the snapshot 2023-10-30T12:32:02.194 INFO:tasks.workunit.client.0.smithi089.stdout: create 196/35 file after the snapshot 2023-10-30T12:32:02.196 INFO:tasks.workunit.client.0.smithi089.stdout: create 196/36 file after the snapshot 2023-10-30T12:32:02.197 INFO:tasks.workunit.client.0.smithi089.stdout: create 196/37 file after the snapshot 2023-10-30T12:32:02.199 INFO:tasks.workunit.client.0.smithi089.stdout: create 196/38 file after the snapshot 2023-10-30T12:32:02.201 INFO:tasks.workunit.client.0.smithi089.stdout: create 196/39 file after the snapshot 2023-10-30T12:32:02.203 INFO:tasks.workunit.client.0.smithi089.stdout: create 197/.snap/snap-subdir-test 2023-10-30T12:32:02.224 INFO:tasks.workunit.client.0.smithi089.stdout: create 197/30 file after the snapshot 2023-10-30T12:32:02.226 INFO:tasks.workunit.client.0.smithi089.stdout: create 197/31 file after the snapshot 2023-10-30T12:32:02.227 INFO:tasks.workunit.client.0.smithi089.stdout: create 197/32 file after the snapshot 2023-10-30T12:32:02.229 INFO:tasks.workunit.client.0.smithi089.stdout: create 197/33 file after the snapshot 2023-10-30T12:32:02.231 INFO:tasks.workunit.client.0.smithi089.stdout: create 197/34 file after the snapshot 2023-10-30T12:32:02.233 INFO:tasks.workunit.client.0.smithi089.stdout: create 197/35 file after the snapshot 2023-10-30T12:32:02.235 INFO:tasks.workunit.client.0.smithi089.stdout: create 197/36 file after the snapshot 2023-10-30T12:32:02.237 INFO:tasks.workunit.client.0.smithi089.stdout: create 197/37 file after the snapshot 2023-10-30T12:32:02.239 INFO:tasks.workunit.client.0.smithi089.stdout: create 197/38 file after the snapshot 2023-10-30T12:32:02.241 INFO:tasks.workunit.client.0.smithi089.stdout: create 197/39 file after the snapshot 2023-10-30T12:32:02.242 INFO:tasks.workunit.client.0.smithi089.stdout: create 198/.snap/snap-subdir-test 2023-10-30T12:32:02.285 INFO:tasks.workunit.client.0.smithi089.stdout: create 198/30 file after the snapshot 2023-10-30T12:32:02.287 INFO:tasks.workunit.client.0.smithi089.stdout: create 198/31 file after the snapshot 2023-10-30T12:32:02.289 INFO:tasks.workunit.client.0.smithi089.stdout: create 198/32 file after the snapshot 2023-10-30T12:32:02.291 INFO:tasks.workunit.client.0.smithi089.stdout: create 198/33 file after the snapshot 2023-10-30T12:32:02.293 INFO:tasks.workunit.client.0.smithi089.stdout: create 198/34 file after the snapshot 2023-10-30T12:32:02.295 INFO:tasks.workunit.client.0.smithi089.stdout: create 198/35 file after the snapshot 2023-10-30T12:32:02.297 INFO:tasks.workunit.client.0.smithi089.stdout: create 198/36 file after the snapshot 2023-10-30T12:32:02.298 INFO:tasks.workunit.client.0.smithi089.stdout: create 198/37 file after the snapshot 2023-10-30T12:32:02.300 INFO:tasks.workunit.client.0.smithi089.stdout: create 198/38 file after the snapshot 2023-10-30T12:32:02.302 INFO:tasks.workunit.client.0.smithi089.stdout: create 198/39 file after the snapshot 2023-10-30T12:32:02.304 INFO:tasks.workunit.client.0.smithi089.stdout: create 199/.snap/snap-subdir-test 2023-10-30T12:32:02.326 INFO:tasks.workunit.client.0.smithi089.stdout: create 199/30 file after the snapshot 2023-10-30T12:32:02.329 INFO:tasks.workunit.client.0.smithi089.stdout: create 199/31 file after the snapshot 2023-10-30T12:32:02.332 INFO:tasks.workunit.client.0.smithi089.stdout: create 199/32 file after the snapshot 2023-10-30T12:32:02.335 INFO:tasks.workunit.client.0.smithi089.stdout: create 199/33 file after the snapshot 2023-10-30T12:32:02.337 INFO:tasks.workunit.client.0.smithi089.stdout: create 199/34 file after the snapshot 2023-10-30T12:32:02.340 INFO:tasks.workunit.client.0.smithi089.stdout: create 199/35 file after the snapshot 2023-10-30T12:32:02.342 INFO:tasks.workunit.client.0.smithi089.stdout: create 199/36 file after the snapshot 2023-10-30T12:32:02.344 INFO:tasks.workunit.client.0.smithi089.stdout: create 199/37 file after the snapshot 2023-10-30T12:32:02.346 INFO:tasks.workunit.client.0.smithi089.stdout: create 199/38 file after the snapshot 2023-10-30T12:32:02.348 INFO:tasks.workunit.client.0.smithi089.stdout: create 199/39 file after the snapshot 2023-10-30T12:32:02.350 INFO:tasks.workunit.client.0.smithi089.stdout: create 200/.snap/snap-subdir-test 2023-10-30T12:32:02.363 INFO:tasks.workunit.client.0.smithi089.stdout: create 200/30 file after the snapshot 2023-10-30T12:32:02.364 INFO:tasks.workunit.client.0.smithi089.stdout: create 200/31 file after the snapshot 2023-10-30T12:32:02.364 INFO:tasks.workunit.client.0.smithi089.stdout: create 200/32 file after the snapshot 2023-10-30T12:32:03.078 INFO:tasks.workunit.client.0.smithi089.stdout: create 200/33 file after the snapshot 2023-10-30T12:32:03.078 INFO:tasks.workunit.client.0.smithi089.stdout: create 200/34 file after the snapshot 2023-10-30T12:32:03.079 INFO:tasks.workunit.client.0.smithi089.stdout: create 200/35 file after the snapshot 2023-10-30T12:32:03.080 INFO:tasks.workunit.client.0.smithi089.stdout: create 200/36 file after the snapshot 2023-10-30T12:32:03.081 INFO:tasks.workunit.client.0.smithi089.stdout: create 200/37 file after the snapshot 2023-10-30T12:32:03.082 INFO:tasks.workunit.client.0.smithi089.stdout: create 200/38 file after the snapshot 2023-10-30T12:32:03.083 INFO:tasks.workunit.client.0.smithi089.stdout: create 200/39 file after the snapshot 2023-10-30T12:32:03.083 INFO:tasks.workunit.client.0.smithi089.stdout: create 201/.snap/snap-subdir-test 2023-10-30T12:32:03.108 INFO:tasks.workunit.client.0.smithi089.stdout: create 201/30 file after the snapshot 2023-10-30T12:32:03.386 INFO:tasks.workunit.client.0.smithi089.stdout: create 201/31 file after the snapshot 2023-10-30T12:32:03.388 INFO:tasks.workunit.client.0.smithi089.stdout: create 201/32 file after the snapshot 2023-10-30T12:32:03.390 INFO:tasks.workunit.client.0.smithi089.stdout: create 201/33 file after the snapshot 2023-10-30T12:32:03.392 INFO:tasks.workunit.client.0.smithi089.stdout: create 201/34 file after the snapshot 2023-10-30T12:32:03.394 INFO:tasks.workunit.client.0.smithi089.stdout: create 201/35 file after the snapshot 2023-10-30T12:32:03.396 INFO:tasks.workunit.client.0.smithi089.stdout: create 201/36 file after the snapshot 2023-10-30T12:32:03.398 INFO:tasks.workunit.client.0.smithi089.stdout: create 201/37 file after the snapshot 2023-10-30T12:32:03.400 INFO:tasks.workunit.client.0.smithi089.stdout: create 201/38 file after the snapshot 2023-10-30T12:32:03.401 INFO:tasks.workunit.client.0.smithi089.stdout: create 201/39 file after the snapshot 2023-10-30T12:32:03.403 INFO:tasks.workunit.client.0.smithi089.stdout: create 202/.snap/snap-subdir-test 2023-10-30T12:32:03.446 INFO:tasks.workunit.client.0.smithi089.stdout: create 202/30 file after the snapshot 2023-10-30T12:32:03.450 INFO:tasks.workunit.client.0.smithi089.stdout: create 202/31 file after the snapshot 2023-10-30T12:32:03.453 INFO:tasks.workunit.client.0.smithi089.stdout: create 202/32 file after the snapshot 2023-10-30T12:32:03.456 INFO:tasks.workunit.client.0.smithi089.stdout: create 202/33 file after the snapshot 2023-10-30T12:32:03.458 INFO:tasks.workunit.client.0.smithi089.stdout: create 202/34 file after the snapshot 2023-10-30T12:32:03.461 INFO:tasks.workunit.client.0.smithi089.stdout: create 202/35 file after the snapshot 2023-10-30T12:32:03.464 INFO:tasks.workunit.client.0.smithi089.stdout: create 202/36 file after the snapshot 2023-10-30T12:32:03.466 INFO:tasks.workunit.client.0.smithi089.stdout: create 202/37 file after the snapshot 2023-10-30T12:32:03.469 INFO:tasks.workunit.client.0.smithi089.stdout: create 202/38 file after the snapshot 2023-10-30T12:32:03.472 INFO:tasks.workunit.client.0.smithi089.stdout: create 202/39 file after the snapshot 2023-10-30T12:32:03.479 INFO:tasks.workunit.client.0.smithi089.stdout: create 203/.snap/snap-subdir-test 2023-10-30T12:32:03.495 INFO:tasks.workunit.client.0.smithi089.stdout: create 203/30 file after the snapshot 2023-10-30T12:32:03.496 INFO:tasks.workunit.client.0.smithi089.stdout: create 203/31 file after the snapshot 2023-10-30T12:32:03.497 INFO:tasks.workunit.client.0.smithi089.stdout: create 203/32 file after the snapshot 2023-10-30T12:32:03.498 INFO:tasks.workunit.client.0.smithi089.stdout: create 203/33 file after the snapshot 2023-10-30T12:32:03.499 INFO:tasks.workunit.client.0.smithi089.stdout: create 203/34 file after the snapshot 2023-10-30T12:32:03.500 INFO:tasks.workunit.client.0.smithi089.stdout: create 203/35 file after the snapshot 2023-10-30T12:32:03.501 INFO:tasks.workunit.client.0.smithi089.stdout: create 203/36 file after the snapshot 2023-10-30T12:32:03.502 INFO:tasks.workunit.client.0.smithi089.stdout: create 203/37 file after the snapshot 2023-10-30T12:32:03.503 INFO:tasks.workunit.client.0.smithi089.stdout: create 203/38 file after the snapshot 2023-10-30T12:32:03.503 INFO:tasks.workunit.client.0.smithi089.stdout: create 203/39 file after the snapshot 2023-10-30T12:32:03.504 INFO:tasks.workunit.client.0.smithi089.stdout: create 204/.snap/snap-subdir-test 2023-10-30T12:32:04.079 INFO:tasks.workunit.client.0.smithi089.stdout: create 204/30 file after the snapshot 2023-10-30T12:32:04.082 INFO:tasks.workunit.client.0.smithi089.stdout: create 204/31 file after the snapshot 2023-10-30T12:32:04.084 INFO:tasks.workunit.client.0.smithi089.stdout: create 204/32 file after the snapshot 2023-10-30T12:32:04.086 INFO:tasks.workunit.client.0.smithi089.stdout: create 204/33 file after the snapshot 2023-10-30T12:32:04.088 INFO:tasks.workunit.client.0.smithi089.stdout: create 204/34 file after the snapshot 2023-10-30T12:32:04.089 INFO:tasks.workunit.client.0.smithi089.stdout: create 204/35 file after the snapshot 2023-10-30T12:32:04.093 INFO:tasks.workunit.client.0.smithi089.stdout: create 204/36 file after the snapshot 2023-10-30T12:32:04.094 INFO:tasks.workunit.client.0.smithi089.stdout: create 204/37 file after the snapshot 2023-10-30T12:32:04.095 INFO:tasks.workunit.client.0.smithi089.stdout: create 204/38 file after the snapshot 2023-10-30T12:32:04.097 INFO:tasks.workunit.client.0.smithi089.stdout: create 204/39 file after the snapshot 2023-10-30T12:32:04.099 INFO:tasks.workunit.client.0.smithi089.stdout: create 205/.snap/snap-subdir-test 2023-10-30T12:32:04.132 INFO:tasks.workunit.client.0.smithi089.stdout: create 205/30 file after the snapshot 2023-10-30T12:32:04.136 INFO:tasks.workunit.client.0.smithi089.stdout: create 205/31 file after the snapshot 2023-10-30T12:32:04.139 INFO:tasks.workunit.client.0.smithi089.stdout: create 205/32 file after the snapshot 2023-10-30T12:32:04.142 INFO:tasks.workunit.client.0.smithi089.stdout: create 205/33 file after the snapshot 2023-10-30T12:32:04.144 INFO:tasks.workunit.client.0.smithi089.stdout: create 205/34 file after the snapshot 2023-10-30T12:32:04.147 INFO:tasks.workunit.client.0.smithi089.stdout: create 205/35 file after the snapshot 2023-10-30T12:32:04.150 INFO:tasks.workunit.client.0.smithi089.stdout: create 205/36 file after the snapshot 2023-10-30T12:32:04.153 INFO:tasks.workunit.client.0.smithi089.stdout: create 205/37 file after the snapshot 2023-10-30T12:32:04.155 INFO:tasks.workunit.client.0.smithi089.stdout: create 205/38 file after the snapshot 2023-10-30T12:32:04.158 INFO:tasks.workunit.client.0.smithi089.stdout: create 205/39 file after the snapshot 2023-10-30T12:32:04.161 INFO:tasks.workunit.client.0.smithi089.stdout: create 206/.snap/snap-subdir-test 2023-10-30T12:32:04.478 INFO:tasks.workunit.client.0.smithi089.stdout: create 206/30 file after the snapshot 2023-10-30T12:32:04.481 INFO:tasks.workunit.client.0.smithi089.stdout: create 206/31 file after the snapshot 2023-10-30T12:32:04.482 INFO:tasks.workunit.client.0.smithi089.stdout: create 206/32 file after the snapshot 2023-10-30T12:32:04.484 INFO:tasks.workunit.client.0.smithi089.stdout: create 206/33 file after the snapshot 2023-10-30T12:32:04.487 INFO:tasks.workunit.client.0.smithi089.stdout: create 206/34 file after the snapshot 2023-10-30T12:32:04.488 INFO:tasks.workunit.client.0.smithi089.stdout: create 206/35 file after the snapshot 2023-10-30T12:32:04.490 INFO:tasks.workunit.client.0.smithi089.stdout: create 206/36 file after the snapshot 2023-10-30T12:32:04.492 INFO:tasks.workunit.client.0.smithi089.stdout: create 206/37 file after the snapshot 2023-10-30T12:32:04.494 INFO:tasks.workunit.client.0.smithi089.stdout: create 206/38 file after the snapshot 2023-10-30T12:32:04.496 INFO:tasks.workunit.client.0.smithi089.stdout: create 206/39 file after the snapshot 2023-10-30T12:32:04.498 INFO:tasks.workunit.client.0.smithi089.stdout: create 207/.snap/snap-subdir-test 2023-10-30T12:32:04.520 INFO:tasks.workunit.client.0.smithi089.stdout: create 207/30 file after the snapshot 2023-10-30T12:32:04.521 INFO:tasks.workunit.client.0.smithi089.stdout: create 207/31 file after the snapshot 2023-10-30T12:32:04.522 INFO:tasks.workunit.client.0.smithi089.stdout: create 207/32 file after the snapshot 2023-10-30T12:32:04.523 INFO:tasks.workunit.client.0.smithi089.stdout: create 207/33 file after the snapshot 2023-10-30T12:32:04.524 INFO:tasks.workunit.client.0.smithi089.stdout: create 207/34 file after the snapshot 2023-10-30T12:32:04.524 INFO:tasks.workunit.client.0.smithi089.stdout: create 207/35 file after the snapshot 2023-10-30T12:32:04.525 INFO:tasks.workunit.client.0.smithi089.stdout: create 207/36 file after the snapshot 2023-10-30T12:32:04.526 INFO:tasks.workunit.client.0.smithi089.stdout: create 207/37 file after the snapshot 2023-10-30T12:32:04.527 INFO:tasks.workunit.client.0.smithi089.stdout: create 207/38 file after the snapshot 2023-10-30T12:32:04.528 INFO:tasks.workunit.client.0.smithi089.stdout: create 207/39 file after the snapshot 2023-10-30T12:32:04.529 INFO:tasks.workunit.client.0.smithi089.stdout: create 208/.snap/snap-subdir-test 2023-10-30T12:32:05.341 INFO:tasks.workunit.client.0.smithi089.stdout: create 208/30 file after the snapshot 2023-10-30T12:32:05.356 INFO:tasks.workunit.client.0.smithi089.stdout: create 208/31 file after the snapshot 2023-10-30T12:32:05.358 INFO:tasks.workunit.client.0.smithi089.stdout: create 208/32 file after the snapshot 2023-10-30T12:32:05.360 INFO:tasks.workunit.client.0.smithi089.stdout: create 208/33 file after the snapshot 2023-10-30T12:32:05.362 INFO:tasks.workunit.client.0.smithi089.stdout: create 208/34 file after the snapshot 2023-10-30T12:32:05.364 INFO:tasks.workunit.client.0.smithi089.stdout: create 208/35 file after the snapshot 2023-10-30T12:32:05.366 INFO:tasks.workunit.client.0.smithi089.stdout: create 208/36 file after the snapshot 2023-10-30T12:32:05.368 INFO:tasks.workunit.client.0.smithi089.stdout: create 208/37 file after the snapshot 2023-10-30T12:32:05.370 INFO:tasks.workunit.client.0.smithi089.stdout: create 208/38 file after the snapshot 2023-10-30T12:32:05.372 INFO:tasks.workunit.client.0.smithi089.stdout: create 208/39 file after the snapshot 2023-10-30T12:32:05.374 INFO:tasks.workunit.client.0.smithi089.stdout: create 209/.snap/snap-subdir-test 2023-10-30T12:32:05.454 INFO:tasks.workunit.client.0.smithi089.stdout: create 209/30 file after the snapshot 2023-10-30T12:32:05.456 INFO:tasks.workunit.client.0.smithi089.stdout: create 209/31 file after the snapshot 2023-10-30T12:32:05.458 INFO:tasks.workunit.client.0.smithi089.stdout: create 209/32 file after the snapshot 2023-10-30T12:32:05.459 INFO:tasks.workunit.client.0.smithi089.stdout: create 209/33 file after the snapshot 2023-10-30T12:32:05.460 INFO:tasks.workunit.client.0.smithi089.stdout: create 209/34 file after the snapshot 2023-10-30T12:32:05.462 INFO:tasks.workunit.client.0.smithi089.stdout: create 209/35 file after the snapshot 2023-10-30T12:32:05.463 INFO:tasks.workunit.client.0.smithi089.stdout: create 209/36 file after the snapshot 2023-10-30T12:32:05.465 INFO:tasks.workunit.client.0.smithi089.stdout: create 209/37 file after the snapshot 2023-10-30T12:32:05.466 INFO:tasks.workunit.client.0.smithi089.stdout: create 209/38 file after the snapshot 2023-10-30T12:32:05.467 INFO:tasks.workunit.client.0.smithi089.stdout: create 209/39 file after the snapshot 2023-10-30T12:32:05.469 INFO:tasks.workunit.client.0.smithi089.stdout: create 210/.snap/snap-subdir-test 2023-10-30T12:32:05.489 INFO:tasks.workunit.client.0.smithi089.stdout: create 210/30 file after the snapshot 2023-10-30T12:32:05.490 INFO:tasks.workunit.client.0.smithi089.stdout: create 210/31 file after the snapshot 2023-10-30T12:32:05.492 INFO:tasks.workunit.client.0.smithi089.stdout: create 210/32 file after the snapshot 2023-10-30T12:32:05.494 INFO:tasks.workunit.client.0.smithi089.stdout: create 210/33 file after the snapshot 2023-10-30T12:32:05.496 INFO:tasks.workunit.client.0.smithi089.stdout: create 210/34 file after the snapshot 2023-10-30T12:32:05.498 INFO:tasks.workunit.client.0.smithi089.stdout: create 210/35 file after the snapshot 2023-10-30T12:32:05.500 INFO:tasks.workunit.client.0.smithi089.stdout: create 210/36 file after the snapshot 2023-10-30T12:32:05.501 INFO:tasks.workunit.client.0.smithi089.stdout: create 210/37 file after the snapshot 2023-10-30T12:32:05.503 INFO:tasks.workunit.client.0.smithi089.stdout: create 210/38 file after the snapshot 2023-10-30T12:32:05.505 INFO:tasks.workunit.client.0.smithi089.stdout: create 210/39 file after the snapshot 2023-10-30T12:32:05.507 INFO:tasks.workunit.client.0.smithi089.stdout: create 211/.snap/snap-subdir-test 2023-10-30T12:32:05.563 INFO:tasks.workunit.client.0.smithi089.stdout: create 211/30 file after the snapshot 2023-10-30T12:32:05.565 INFO:tasks.workunit.client.0.smithi089.stdout: create 211/31 file after the snapshot 2023-10-30T12:32:05.567 INFO:tasks.workunit.client.0.smithi089.stdout: create 211/32 file after the snapshot 2023-10-30T12:32:05.569 INFO:tasks.workunit.client.0.smithi089.stdout: create 211/33 file after the snapshot 2023-10-30T12:32:05.571 INFO:tasks.workunit.client.0.smithi089.stdout: create 211/34 file after the snapshot 2023-10-30T12:32:05.573 INFO:tasks.workunit.client.0.smithi089.stdout: create 211/35 file after the snapshot 2023-10-30T12:32:05.575 INFO:tasks.workunit.client.0.smithi089.stdout: create 211/36 file after the snapshot 2023-10-30T12:32:05.577 INFO:tasks.workunit.client.0.smithi089.stdout: create 211/37 file after the snapshot 2023-10-30T12:32:05.579 INFO:tasks.workunit.client.0.smithi089.stdout: create 211/38 file after the snapshot 2023-10-30T12:32:05.580 INFO:tasks.workunit.client.0.smithi089.stdout: create 211/39 file after the snapshot 2023-10-30T12:32:05.583 INFO:tasks.workunit.client.0.smithi089.stdout: create 212/.snap/snap-subdir-test 2023-10-30T12:32:05.706 INFO:tasks.workunit.client.0.smithi089.stdout: create 212/30 file after the snapshot 2023-10-30T12:32:05.707 INFO:tasks.workunit.client.0.smithi089.stdout: create 212/31 file after the snapshot 2023-10-30T12:32:05.708 INFO:tasks.workunit.client.0.smithi089.stdout: create 212/32 file after the snapshot 2023-10-30T12:32:05.709 INFO:tasks.workunit.client.0.smithi089.stdout: create 212/33 file after the snapshot 2023-10-30T12:32:05.709 INFO:tasks.workunit.client.0.smithi089.stdout: create 212/34 file after the snapshot 2023-10-30T12:32:05.710 INFO:tasks.workunit.client.0.smithi089.stdout: create 212/35 file after the snapshot 2023-10-30T12:32:05.710 INFO:tasks.workunit.client.0.smithi089.stdout: create 212/36 file after the snapshot 2023-10-30T12:32:05.711 INFO:tasks.workunit.client.0.smithi089.stdout: create 212/37 file after the snapshot 2023-10-30T12:32:05.712 INFO:tasks.workunit.client.0.smithi089.stdout: create 212/38 file after the snapshot 2023-10-30T12:32:05.712 INFO:tasks.workunit.client.0.smithi089.stdout: create 212/39 file after the snapshot 2023-10-30T12:32:05.713 INFO:tasks.workunit.client.0.smithi089.stdout: create 213/.snap/snap-subdir-test 2023-10-30T12:32:05.729 INFO:tasks.workunit.client.0.smithi089.stdout: create 213/30 file after the snapshot 2023-10-30T12:32:05.730 INFO:tasks.workunit.client.0.smithi089.stdout: create 213/31 file after the snapshot 2023-10-30T12:32:05.730 INFO:tasks.workunit.client.0.smithi089.stdout: create 213/32 file after the snapshot 2023-10-30T12:32:05.731 INFO:tasks.workunit.client.0.smithi089.stdout: create 213/33 file after the snapshot 2023-10-30T12:32:05.731 INFO:tasks.workunit.client.0.smithi089.stdout: create 213/34 file after the snapshot 2023-10-30T12:32:05.732 INFO:tasks.workunit.client.0.smithi089.stdout: create 213/35 file after the snapshot 2023-10-30T12:32:05.985 INFO:tasks.workunit.client.0.smithi089.stdout: create 213/36 file after the snapshot 2023-10-30T12:32:05.985 INFO:tasks.workunit.client.0.smithi089.stdout: create 213/37 file after the snapshot 2023-10-30T12:32:05.986 INFO:tasks.workunit.client.0.smithi089.stdout: create 213/38 file after the snapshot 2023-10-30T12:32:05.987 INFO:tasks.workunit.client.0.smithi089.stdout: create 213/39 file after the snapshot 2023-10-30T12:32:05.987 INFO:tasks.workunit.client.0.smithi089.stdout: create 214/.snap/snap-subdir-test 2023-10-30T12:32:06.010 INFO:tasks.workunit.client.0.smithi089.stdout: create 214/30 file after the snapshot 2023-10-30T12:32:06.013 INFO:tasks.workunit.client.0.smithi089.stdout: create 214/31 file after the snapshot 2023-10-30T12:32:06.014 INFO:tasks.workunit.client.0.smithi089.stdout: create 214/32 file after the snapshot 2023-10-30T12:32:06.016 INFO:tasks.workunit.client.0.smithi089.stdout: create 214/33 file after the snapshot 2023-10-30T12:32:06.018 INFO:tasks.workunit.client.0.smithi089.stdout: create 214/34 file after the snapshot 2023-10-30T12:32:06.020 INFO:tasks.workunit.client.0.smithi089.stdout: create 214/35 file after the snapshot 2023-10-30T12:32:06.022 INFO:tasks.workunit.client.0.smithi089.stdout: create 214/36 file after the snapshot 2023-10-30T12:32:06.024 INFO:tasks.workunit.client.0.smithi089.stdout: create 214/37 file after the snapshot 2023-10-30T12:32:06.026 INFO:tasks.workunit.client.0.smithi089.stdout: create 214/38 file after the snapshot 2023-10-30T12:32:06.028 INFO:tasks.workunit.client.0.smithi089.stdout: create 214/39 file after the snapshot 2023-10-30T12:32:06.030 INFO:tasks.workunit.client.0.smithi089.stdout: create 215/.snap/snap-subdir-test 2023-10-30T12:32:06.059 INFO:tasks.workunit.client.0.smithi089.stdout: create 215/30 file after the snapshot 2023-10-30T12:32:06.062 INFO:tasks.workunit.client.0.smithi089.stdout: create 215/31 file after the snapshot 2023-10-30T12:32:06.064 INFO:tasks.workunit.client.0.smithi089.stdout: create 215/32 file after the snapshot 2023-10-30T12:32:06.066 INFO:tasks.workunit.client.0.smithi089.stdout: create 215/33 file after the snapshot 2023-10-30T12:32:06.067 INFO:tasks.workunit.client.0.smithi089.stdout: create 215/34 file after the snapshot 2023-10-30T12:32:06.069 INFO:tasks.workunit.client.0.smithi089.stdout: create 215/35 file after the snapshot 2023-10-30T12:32:06.071 INFO:tasks.workunit.client.0.smithi089.stdout: create 215/36 file after the snapshot 2023-10-30T12:32:06.073 INFO:tasks.workunit.client.0.smithi089.stdout: create 215/37 file after the snapshot 2023-10-30T12:32:06.075 INFO:tasks.workunit.client.0.smithi089.stdout: create 215/38 file after the snapshot 2023-10-30T12:32:06.077 INFO:tasks.workunit.client.0.smithi089.stdout: create 215/39 file after the snapshot 2023-10-30T12:32:06.079 INFO:tasks.workunit.client.0.smithi089.stdout: create 216/.snap/snap-subdir-test 2023-10-30T12:32:06.101 INFO:tasks.workunit.client.0.smithi089.stdout: create 216/30 file after the snapshot 2023-10-30T12:32:06.104 INFO:tasks.workunit.client.0.smithi089.stdout: create 216/31 file after the snapshot 2023-10-30T12:32:06.105 INFO:tasks.workunit.client.0.smithi089.stdout: create 216/32 file after the snapshot 2023-10-30T12:32:06.107 INFO:tasks.workunit.client.0.smithi089.stdout: create 216/33 file after the snapshot 2023-10-30T12:32:06.109 INFO:tasks.workunit.client.0.smithi089.stdout: create 216/34 file after the snapshot 2023-10-30T12:32:06.110 INFO:tasks.workunit.client.0.smithi089.stdout: create 216/35 file after the snapshot 2023-10-30T12:32:06.112 INFO:tasks.workunit.client.0.smithi089.stdout: create 216/36 file after the snapshot 2023-10-30T12:32:06.114 INFO:tasks.workunit.client.0.smithi089.stdout: create 216/37 file after the snapshot 2023-10-30T12:32:06.116 INFO:tasks.workunit.client.0.smithi089.stdout: create 216/38 file after the snapshot 2023-10-30T12:32:06.117 INFO:tasks.workunit.client.0.smithi089.stdout: create 216/39 file after the snapshot 2023-10-30T12:32:06.119 INFO:tasks.workunit.client.0.smithi089.stdout: create 217/.snap/snap-subdir-test 2023-10-30T12:32:06.153 INFO:tasks.workunit.client.0.smithi089.stdout: create 217/30 file after the snapshot 2023-10-30T12:32:06.155 INFO:tasks.workunit.client.0.smithi089.stdout: create 217/31 file after the snapshot 2023-10-30T12:32:06.157 INFO:tasks.workunit.client.0.smithi089.stdout: create 217/32 file after the snapshot 2023-10-30T12:32:06.159 INFO:tasks.workunit.client.0.smithi089.stdout: create 217/33 file after the snapshot 2023-10-30T12:32:06.161 INFO:tasks.workunit.client.0.smithi089.stdout: create 217/34 file after the snapshot 2023-10-30T12:32:06.163 INFO:tasks.workunit.client.0.smithi089.stdout: create 217/35 file after the snapshot 2023-10-30T12:32:06.165 INFO:tasks.workunit.client.0.smithi089.stdout: create 217/36 file after the snapshot 2023-10-30T12:32:06.167 INFO:tasks.workunit.client.0.smithi089.stdout: create 217/37 file after the snapshot 2023-10-30T12:32:06.169 INFO:tasks.workunit.client.0.smithi089.stdout: create 217/38 file after the snapshot 2023-10-30T12:32:06.171 INFO:tasks.workunit.client.0.smithi089.stdout: create 217/39 file after the snapshot 2023-10-30T12:32:06.172 INFO:tasks.workunit.client.0.smithi089.stdout: create 218/.snap/snap-subdir-test 2023-10-30T12:32:06.204 INFO:tasks.workunit.client.0.smithi089.stdout: create 218/30 file after the snapshot 2023-10-30T12:32:06.206 INFO:tasks.workunit.client.0.smithi089.stdout: create 218/31 file after the snapshot 2023-10-30T12:32:06.208 INFO:tasks.workunit.client.0.smithi089.stdout: create 218/32 file after the snapshot 2023-10-30T12:32:06.210 INFO:tasks.workunit.client.0.smithi089.stdout: create 218/33 file after the snapshot 2023-10-30T12:32:06.211 INFO:tasks.workunit.client.0.smithi089.stdout: create 218/34 file after the snapshot 2023-10-30T12:32:06.213 INFO:tasks.workunit.client.0.smithi089.stdout: create 218/35 file after the snapshot 2023-10-30T12:32:06.215 INFO:tasks.workunit.client.0.smithi089.stdout: create 218/36 file after the snapshot 2023-10-30T12:32:06.217 INFO:tasks.workunit.client.0.smithi089.stdout: create 218/37 file after the snapshot 2023-10-30T12:32:06.219 INFO:tasks.workunit.client.0.smithi089.stdout: create 218/38 file after the snapshot 2023-10-30T12:32:06.221 INFO:tasks.workunit.client.0.smithi089.stdout: create 218/39 file after the snapshot 2023-10-30T12:32:06.222 INFO:tasks.workunit.client.0.smithi089.stdout: create 219/.snap/snap-subdir-test 2023-10-30T12:32:06.255 INFO:tasks.workunit.client.0.smithi089.stdout: create 219/30 file after the snapshot 2023-10-30T12:32:06.257 INFO:tasks.workunit.client.0.smithi089.stdout: create 219/31 file after the snapshot 2023-10-30T12:32:06.259 INFO:tasks.workunit.client.0.smithi089.stdout: create 219/32 file after the snapshot 2023-10-30T12:32:06.261 INFO:tasks.workunit.client.0.smithi089.stdout: create 219/33 file after the snapshot 2023-10-30T12:32:06.263 INFO:tasks.workunit.client.0.smithi089.stdout: create 219/34 file after the snapshot 2023-10-30T12:32:06.265 INFO:tasks.workunit.client.0.smithi089.stdout: create 219/35 file after the snapshot 2023-10-30T12:32:06.267 INFO:tasks.workunit.client.0.smithi089.stdout: create 219/36 file after the snapshot 2023-10-30T12:32:06.269 INFO:tasks.workunit.client.0.smithi089.stdout: create 219/37 file after the snapshot 2023-10-30T12:32:06.271 INFO:tasks.workunit.client.0.smithi089.stdout: create 219/38 file after the snapshot 2023-10-30T12:32:06.273 INFO:tasks.workunit.client.0.smithi089.stdout: create 219/39 file after the snapshot 2023-10-30T12:32:06.275 INFO:tasks.workunit.client.0.smithi089.stdout: create 220/.snap/snap-subdir-test 2023-10-30T12:32:06.290 INFO:tasks.workunit.client.0.smithi089.stdout: create 220/30 file after the snapshot 2023-10-30T12:32:06.291 INFO:tasks.workunit.client.0.smithi089.stdout: create 220/31 file after the snapshot 2023-10-30T12:32:06.291 INFO:tasks.workunit.client.0.smithi089.stdout: create 220/32 file after the snapshot 2023-10-30T12:32:06.292 INFO:tasks.workunit.client.0.smithi089.stdout: create 220/33 file after the snapshot 2023-10-30T12:32:06.293 INFO:tasks.workunit.client.0.smithi089.stdout: create 220/34 file after the snapshot 2023-10-30T12:32:06.293 INFO:tasks.workunit.client.0.smithi089.stdout: create 220/35 file after the snapshot 2023-10-30T12:32:06.294 INFO:tasks.workunit.client.0.smithi089.stdout: create 220/36 file after the snapshot 2023-10-30T12:32:06.294 INFO:tasks.workunit.client.0.smithi089.stdout: create 220/37 file after the snapshot 2023-10-30T12:32:06.295 INFO:tasks.workunit.client.0.smithi089.stdout: create 220/38 file after the snapshot 2023-10-30T12:32:06.296 INFO:tasks.workunit.client.0.smithi089.stdout: create 220/39 file after the snapshot 2023-10-30T12:32:06.296 INFO:tasks.workunit.client.0.smithi089.stdout: create 221/.snap/snap-subdir-test 2023-10-30T12:32:06.319 INFO:tasks.workunit.client.0.smithi089.stdout: create 221/30 file after the snapshot 2023-10-30T12:32:06.322 INFO:tasks.workunit.client.0.smithi089.stdout: create 221/31 file after the snapshot 2023-10-30T12:32:06.324 INFO:tasks.workunit.client.0.smithi089.stdout: create 221/32 file after the snapshot 2023-10-30T12:32:06.326 INFO:tasks.workunit.client.0.smithi089.stdout: create 221/33 file after the snapshot 2023-10-30T12:32:06.328 INFO:tasks.workunit.client.0.smithi089.stdout: create 221/34 file after the snapshot 2023-10-30T12:32:06.329 INFO:tasks.workunit.client.0.smithi089.stdout: create 221/35 file after the snapshot 2023-10-30T12:32:06.331 INFO:tasks.workunit.client.0.smithi089.stdout: create 221/36 file after the snapshot 2023-10-30T12:32:06.333 INFO:tasks.workunit.client.0.smithi089.stdout: create 221/37 file after the snapshot 2023-10-30T12:32:06.335 INFO:tasks.workunit.client.0.smithi089.stdout: create 221/38 file after the snapshot 2023-10-30T12:32:06.337 INFO:tasks.workunit.client.0.smithi089.stdout: create 221/39 file after the snapshot 2023-10-30T12:32:06.339 INFO:tasks.workunit.client.0.smithi089.stdout: create 222/.snap/snap-subdir-test 2023-10-30T12:32:06.361 INFO:tasks.workunit.client.0.smithi089.stdout: create 222/30 file after the snapshot 2023-10-30T12:32:06.362 INFO:tasks.workunit.client.0.smithi089.stdout: create 222/31 file after the snapshot 2023-10-30T12:32:06.363 INFO:tasks.workunit.client.0.smithi089.stdout: create 222/32 file after the snapshot 2023-10-30T12:32:06.364 INFO:tasks.workunit.client.0.smithi089.stdout: create 222/33 file after the snapshot 2023-10-30T12:32:06.364 INFO:tasks.workunit.client.0.smithi089.stdout: create 222/34 file after the snapshot 2023-10-30T12:32:06.365 INFO:tasks.workunit.client.0.smithi089.stdout: create 222/35 file after the snapshot 2023-10-30T12:32:06.366 INFO:tasks.workunit.client.0.smithi089.stdout: create 222/36 file after the snapshot 2023-10-30T12:32:06.367 INFO:tasks.workunit.client.0.smithi089.stdout: create 222/37 file after the snapshot 2023-10-30T12:32:06.368 INFO:tasks.workunit.client.0.smithi089.stdout: create 222/38 file after the snapshot 2023-10-30T12:32:06.368 INFO:tasks.workunit.client.0.smithi089.stdout: create 222/39 file after the snapshot 2023-10-30T12:32:06.369 INFO:tasks.workunit.client.0.smithi089.stdout: create 223/.snap/snap-subdir-test 2023-10-30T12:32:06.385 INFO:tasks.workunit.client.0.smithi089.stdout: create 223/30 file after the snapshot 2023-10-30T12:32:06.637 INFO:tasks.workunit.client.0.smithi089.stdout: create 223/31 file after the snapshot 2023-10-30T12:32:06.638 INFO:tasks.workunit.client.0.smithi089.stdout: create 223/32 file after the snapshot 2023-10-30T12:32:06.639 INFO:tasks.workunit.client.0.smithi089.stdout: create 223/33 file after the snapshot 2023-10-30T12:32:06.639 INFO:tasks.workunit.client.0.smithi089.stdout: create 223/34 file after the snapshot 2023-10-30T12:32:06.640 INFO:tasks.workunit.client.0.smithi089.stdout: create 223/35 file after the snapshot 2023-10-30T12:32:06.641 INFO:tasks.workunit.client.0.smithi089.stdout: create 223/36 file after the snapshot 2023-10-30T12:32:06.642 INFO:tasks.workunit.client.0.smithi089.stdout: create 223/37 file after the snapshot 2023-10-30T12:32:06.642 INFO:tasks.workunit.client.0.smithi089.stdout: create 223/38 file after the snapshot 2023-10-30T12:32:06.643 INFO:tasks.workunit.client.0.smithi089.stdout: create 223/39 file after the snapshot 2023-10-30T12:32:06.644 INFO:tasks.workunit.client.0.smithi089.stdout: create 224/.snap/snap-subdir-test 2023-10-30T12:32:06.664 INFO:tasks.workunit.client.0.smithi089.stdout: create 224/30 file after the snapshot 2023-10-30T12:32:06.666 INFO:tasks.workunit.client.0.smithi089.stdout: create 224/31 file after the snapshot 2023-10-30T12:32:06.668 INFO:tasks.workunit.client.0.smithi089.stdout: create 224/32 file after the snapshot 2023-10-30T12:32:06.670 INFO:tasks.workunit.client.0.smithi089.stdout: create 224/33 file after the snapshot 2023-10-30T12:32:06.672 INFO:tasks.workunit.client.0.smithi089.stdout: create 224/34 file after the snapshot 2023-10-30T12:32:06.674 INFO:tasks.workunit.client.0.smithi089.stdout: create 224/35 file after the snapshot 2023-10-30T12:32:06.676 INFO:tasks.workunit.client.0.smithi089.stdout: create 224/36 file after the snapshot 2023-10-30T12:32:06.677 INFO:tasks.workunit.client.0.smithi089.stdout: create 224/37 file after the snapshot 2023-10-30T12:32:06.679 INFO:tasks.workunit.client.0.smithi089.stdout: create 224/38 file after the snapshot 2023-10-30T12:32:06.681 INFO:tasks.workunit.client.0.smithi089.stdout: create 224/39 file after the snapshot 2023-10-30T12:32:06.683 INFO:tasks.workunit.client.0.smithi089.stdout: create 225/.snap/snap-subdir-test 2023-10-30T12:32:06.719 INFO:tasks.workunit.client.0.smithi089.stdout: create 225/30 file after the snapshot 2023-10-30T12:32:06.721 INFO:tasks.workunit.client.0.smithi089.stdout: create 225/31 file after the snapshot 2023-10-30T12:32:06.723 INFO:tasks.workunit.client.0.smithi089.stdout: create 225/32 file after the snapshot 2023-10-30T12:32:06.725 INFO:tasks.workunit.client.0.smithi089.stdout: create 225/33 file after the snapshot 2023-10-30T12:32:06.726 INFO:tasks.workunit.client.0.smithi089.stdout: create 225/34 file after the snapshot 2023-10-30T12:32:06.728 INFO:tasks.workunit.client.0.smithi089.stdout: create 225/35 file after the snapshot 2023-10-30T12:32:06.730 INFO:tasks.workunit.client.0.smithi089.stdout: create 225/36 file after the snapshot 2023-10-30T12:32:06.731 INFO:tasks.workunit.client.0.smithi089.stdout: create 225/37 file after the snapshot 2023-10-30T12:32:06.733 INFO:tasks.workunit.client.0.smithi089.stdout: create 225/38 file after the snapshot 2023-10-30T12:32:06.735 INFO:tasks.workunit.client.0.smithi089.stdout: create 225/39 file after the snapshot 2023-10-30T12:32:06.737 INFO:tasks.workunit.client.0.smithi089.stdout: create 226/.snap/snap-subdir-test 2023-10-30T12:32:06.758 INFO:tasks.workunit.client.0.smithi089.stdout: create 226/30 file after the snapshot 2023-10-30T12:32:06.760 INFO:tasks.workunit.client.0.smithi089.stdout: create 226/31 file after the snapshot 2023-10-30T12:32:06.762 INFO:tasks.workunit.client.0.smithi089.stdout: create 226/32 file after the snapshot 2023-10-30T12:32:06.764 INFO:tasks.workunit.client.0.smithi089.stdout: create 226/33 file after the snapshot 2023-10-30T12:32:06.766 INFO:tasks.workunit.client.0.smithi089.stdout: create 226/34 file after the snapshot 2023-10-30T12:32:06.768 INFO:tasks.workunit.client.0.smithi089.stdout: create 226/35 file after the snapshot 2023-10-30T12:32:06.770 INFO:tasks.workunit.client.0.smithi089.stdout: create 226/36 file after the snapshot 2023-10-30T12:32:06.772 INFO:tasks.workunit.client.0.smithi089.stdout: create 226/37 file after the snapshot 2023-10-30T12:32:06.774 INFO:tasks.workunit.client.0.smithi089.stdout: create 226/38 file after the snapshot 2023-10-30T12:32:06.776 INFO:tasks.workunit.client.0.smithi089.stdout: create 226/39 file after the snapshot 2023-10-30T12:32:06.777 INFO:tasks.workunit.client.0.smithi089.stdout: create 227/.snap/snap-subdir-test 2023-10-30T12:32:06.808 INFO:tasks.workunit.client.0.smithi089.stdout: create 227/30 file after the snapshot 2023-10-30T12:32:06.810 INFO:tasks.workunit.client.0.smithi089.stdout: create 227/31 file after the snapshot 2023-10-30T12:32:06.812 INFO:tasks.workunit.client.0.smithi089.stdout: create 227/32 file after the snapshot 2023-10-30T12:32:06.813 INFO:tasks.workunit.client.0.smithi089.stdout: create 227/33 file after the snapshot 2023-10-30T12:32:06.815 INFO:tasks.workunit.client.0.smithi089.stdout: create 227/34 file after the snapshot 2023-10-30T12:32:06.817 INFO:tasks.workunit.client.0.smithi089.stdout: create 227/35 file after the snapshot 2023-10-30T12:32:06.819 INFO:tasks.workunit.client.0.smithi089.stdout: create 227/36 file after the snapshot 2023-10-30T12:32:06.821 INFO:tasks.workunit.client.0.smithi089.stdout: create 227/37 file after the snapshot 2023-10-30T12:32:06.823 INFO:tasks.workunit.client.0.smithi089.stdout: create 227/38 file after the snapshot 2023-10-30T12:32:06.824 INFO:tasks.workunit.client.0.smithi089.stdout: create 227/39 file after the snapshot 2023-10-30T12:32:06.826 INFO:tasks.workunit.client.0.smithi089.stdout: create 228/.snap/snap-subdir-test 2023-10-30T12:32:06.851 INFO:tasks.workunit.client.0.smithi089.stdout: create 228/30 file after the snapshot 2023-10-30T12:32:06.853 INFO:tasks.workunit.client.0.smithi089.stdout: create 228/31 file after the snapshot 2023-10-30T12:32:06.855 INFO:tasks.workunit.client.0.smithi089.stdout: create 228/32 file after the snapshot 2023-10-30T12:32:06.857 INFO:tasks.workunit.client.0.smithi089.stdout: create 228/33 file after the snapshot 2023-10-30T12:32:06.859 INFO:tasks.workunit.client.0.smithi089.stdout: create 228/34 file after the snapshot 2023-10-30T12:32:06.860 INFO:tasks.workunit.client.0.smithi089.stdout: create 228/35 file after the snapshot 2023-10-30T12:32:06.862 INFO:tasks.workunit.client.0.smithi089.stdout: create 228/36 file after the snapshot 2023-10-30T12:32:06.864 INFO:tasks.workunit.client.0.smithi089.stdout: create 228/37 file after the snapshot 2023-10-30T12:32:06.866 INFO:tasks.workunit.client.0.smithi089.stdout: create 228/38 file after the snapshot 2023-10-30T12:32:06.867 INFO:tasks.workunit.client.0.smithi089.stdout: create 228/39 file after the snapshot 2023-10-30T12:32:06.869 INFO:tasks.workunit.client.0.smithi089.stdout: create 229/.snap/snap-subdir-test 2023-10-30T12:32:06.893 INFO:tasks.workunit.client.0.smithi089.stdout: create 229/30 file after the snapshot 2023-10-30T12:32:06.895 INFO:tasks.workunit.client.0.smithi089.stdout: create 229/31 file after the snapshot 2023-10-30T12:32:06.897 INFO:tasks.workunit.client.0.smithi089.stdout: create 229/32 file after the snapshot 2023-10-30T12:32:06.899 INFO:tasks.workunit.client.0.smithi089.stdout: create 229/33 file after the snapshot 2023-10-30T12:32:06.901 INFO:tasks.workunit.client.0.smithi089.stdout: create 229/34 file after the snapshot 2023-10-30T12:32:06.902 INFO:tasks.workunit.client.0.smithi089.stdout: create 229/35 file after the snapshot 2023-10-30T12:32:06.904 INFO:tasks.workunit.client.0.smithi089.stdout: create 229/36 file after the snapshot 2023-10-30T12:32:06.906 INFO:tasks.workunit.client.0.smithi089.stdout: create 229/37 file after the snapshot 2023-10-30T12:32:06.907 INFO:tasks.workunit.client.0.smithi089.stdout: create 229/38 file after the snapshot 2023-10-30T12:32:06.909 INFO:tasks.workunit.client.0.smithi089.stdout: create 229/39 file after the snapshot 2023-10-30T12:32:06.911 INFO:tasks.workunit.client.0.smithi089.stdout: create 230/.snap/snap-subdir-test 2023-10-30T12:32:06.935 INFO:tasks.workunit.client.0.smithi089.stdout: create 230/30 file after the snapshot 2023-10-30T12:32:06.937 INFO:tasks.workunit.client.0.smithi089.stdout: create 230/31 file after the snapshot 2023-10-30T12:32:06.939 INFO:tasks.workunit.client.0.smithi089.stdout: create 230/32 file after the snapshot 2023-10-30T12:32:06.941 INFO:tasks.workunit.client.0.smithi089.stdout: create 230/33 file after the snapshot 2023-10-30T12:32:06.946 INFO:tasks.workunit.client.0.smithi089.stdout: create 230/34 file after the snapshot 2023-10-30T12:32:06.950 INFO:tasks.workunit.client.0.smithi089.stdout: create 230/35 file after the snapshot 2023-10-30T12:32:06.952 INFO:tasks.workunit.client.0.smithi089.stdout: create 230/36 file after the snapshot 2023-10-30T12:32:06.954 INFO:tasks.workunit.client.0.smithi089.stdout: create 230/37 file after the snapshot 2023-10-30T12:32:06.958 INFO:tasks.workunit.client.0.smithi089.stdout: create 230/38 file after the snapshot 2023-10-30T12:32:06.961 INFO:tasks.workunit.client.0.smithi089.stdout: create 230/39 file after the snapshot 2023-10-30T12:32:06.966 INFO:tasks.workunit.client.0.smithi089.stdout: create 231/.snap/snap-subdir-test 2023-10-30T12:32:07.805 INFO:tasks.workunit.client.0.smithi089.stdout: create 231/30 file after the snapshot 2023-10-30T12:32:07.807 INFO:tasks.workunit.client.0.smithi089.stdout: create 231/31 file after the snapshot 2023-10-30T12:32:07.809 INFO:tasks.workunit.client.0.smithi089.stdout: create 231/32 file after the snapshot 2023-10-30T12:32:07.811 INFO:tasks.workunit.client.0.smithi089.stdout: create 231/33 file after the snapshot 2023-10-30T12:32:07.813 INFO:tasks.workunit.client.0.smithi089.stdout: create 231/34 file after the snapshot 2023-10-30T12:32:07.815 INFO:tasks.workunit.client.0.smithi089.stdout: create 231/35 file after the snapshot 2023-10-30T12:32:07.816 INFO:tasks.workunit.client.0.smithi089.stdout: create 231/36 file after the snapshot 2023-10-30T12:32:07.818 INFO:tasks.workunit.client.0.smithi089.stdout: create 231/37 file after the snapshot 2023-10-30T12:32:07.820 INFO:tasks.workunit.client.0.smithi089.stdout: create 231/38 file after the snapshot 2023-10-30T12:32:07.822 INFO:tasks.workunit.client.0.smithi089.stdout: create 231/39 file after the snapshot 2023-10-30T12:32:07.824 INFO:tasks.workunit.client.0.smithi089.stdout: create 232/.snap/snap-subdir-test 2023-10-30T12:32:07.860 INFO:tasks.workunit.client.0.smithi089.stdout: create 232/30 file after the snapshot 2023-10-30T12:32:07.862 INFO:tasks.workunit.client.0.smithi089.stdout: create 232/31 file after the snapshot 2023-10-30T12:32:07.863 INFO:tasks.workunit.client.0.smithi089.stdout: create 232/32 file after the snapshot 2023-10-30T12:32:07.865 INFO:tasks.workunit.client.0.smithi089.stdout: create 232/33 file after the snapshot 2023-10-30T12:32:07.867 INFO:tasks.workunit.client.0.smithi089.stdout: create 232/34 file after the snapshot 2023-10-30T12:32:07.869 INFO:tasks.workunit.client.0.smithi089.stdout: create 232/35 file after the snapshot 2023-10-30T12:32:07.871 INFO:tasks.workunit.client.0.smithi089.stdout: create 232/36 file after the snapshot 2023-10-30T12:32:07.873 INFO:tasks.workunit.client.0.smithi089.stdout: create 232/37 file after the snapshot 2023-10-30T12:32:07.875 INFO:tasks.workunit.client.0.smithi089.stdout: create 232/38 file after the snapshot 2023-10-30T12:32:07.877 INFO:tasks.workunit.client.0.smithi089.stdout: create 232/39 file after the snapshot 2023-10-30T12:32:07.878 INFO:tasks.workunit.client.0.smithi089.stdout: create 233/.snap/snap-subdir-test 2023-10-30T12:32:07.904 INFO:tasks.workunit.client.0.smithi089.stdout: create 233/30 file after the snapshot 2023-10-30T12:32:07.906 INFO:tasks.workunit.client.0.smithi089.stdout: create 233/31 file after the snapshot 2023-10-30T12:32:07.908 INFO:tasks.workunit.client.0.smithi089.stdout: create 233/32 file after the snapshot 2023-10-30T12:32:07.910 INFO:tasks.workunit.client.0.smithi089.stdout: create 233/33 file after the snapshot 2023-10-30T12:32:07.912 INFO:tasks.workunit.client.0.smithi089.stdout: create 233/34 file after the snapshot 2023-10-30T12:32:07.914 INFO:tasks.workunit.client.0.smithi089.stdout: create 233/35 file after the snapshot 2023-10-30T12:32:07.916 INFO:tasks.workunit.client.0.smithi089.stdout: create 233/36 file after the snapshot 2023-10-30T12:32:07.918 INFO:tasks.workunit.client.0.smithi089.stdout: create 233/37 file after the snapshot 2023-10-30T12:32:07.920 INFO:tasks.workunit.client.0.smithi089.stdout: create 233/38 file after the snapshot 2023-10-30T12:32:07.922 INFO:tasks.workunit.client.0.smithi089.stdout: create 233/39 file after the snapshot 2023-10-30T12:32:07.924 INFO:tasks.workunit.client.0.smithi089.stdout: create 234/.snap/snap-subdir-test 2023-10-30T12:32:07.947 INFO:tasks.workunit.client.0.smithi089.stdout: create 234/30 file after the snapshot 2023-10-30T12:32:07.950 INFO:tasks.workunit.client.0.smithi089.stdout: create 234/31 file after the snapshot 2023-10-30T12:32:07.952 INFO:tasks.workunit.client.0.smithi089.stdout: create 234/32 file after the snapshot 2023-10-30T12:32:07.954 INFO:tasks.workunit.client.0.smithi089.stdout: create 234/33 file after the snapshot 2023-10-30T12:32:07.956 INFO:tasks.workunit.client.0.smithi089.stdout: create 234/34 file after the snapshot 2023-10-30T12:32:07.958 INFO:tasks.workunit.client.0.smithi089.stdout: create 234/35 file after the snapshot 2023-10-30T12:32:07.960 INFO:tasks.workunit.client.0.smithi089.stdout: create 234/36 file after the snapshot 2023-10-30T12:32:07.962 INFO:tasks.workunit.client.0.smithi089.stdout: create 234/37 file after the snapshot 2023-10-30T12:32:07.964 INFO:tasks.workunit.client.0.smithi089.stdout: create 234/38 file after the snapshot 2023-10-30T12:32:07.966 INFO:tasks.workunit.client.0.smithi089.stdout: create 234/39 file after the snapshot 2023-10-30T12:32:07.968 INFO:tasks.workunit.client.0.smithi089.stdout: create 235/.snap/snap-subdir-test 2023-10-30T12:32:07.982 INFO:tasks.workunit.client.0.smithi089.stdout: create 235/30 file after the snapshot 2023-10-30T12:32:07.983 INFO:tasks.workunit.client.0.smithi089.stdout: create 235/31 file after the snapshot 2023-10-30T12:32:07.984 INFO:tasks.workunit.client.0.smithi089.stdout: create 235/32 file after the snapshot 2023-10-30T12:32:07.984 INFO:tasks.workunit.client.0.smithi089.stdout: create 235/33 file after the snapshot 2023-10-30T12:32:07.985 INFO:tasks.workunit.client.0.smithi089.stdout: create 235/34 file after the snapshot 2023-10-30T12:32:07.986 INFO:tasks.workunit.client.0.smithi089.stdout: create 235/35 file after the snapshot 2023-10-30T12:32:07.987 INFO:tasks.workunit.client.0.smithi089.stdout: create 235/36 file after the snapshot 2023-10-30T12:32:07.987 INFO:tasks.workunit.client.0.smithi089.stdout: create 235/37 file after the snapshot 2023-10-30T12:32:07.988 INFO:tasks.workunit.client.0.smithi089.stdout: create 235/38 file after the snapshot 2023-10-30T12:32:07.989 INFO:tasks.workunit.client.0.smithi089.stdout: create 235/39 file after the snapshot 2023-10-30T12:32:07.990 INFO:tasks.workunit.client.0.smithi089.stdout: create 236/.snap/snap-subdir-test 2023-10-30T12:32:08.013 INFO:tasks.workunit.client.0.smithi089.stdout: create 236/30 file after the snapshot 2023-10-30T12:32:08.016 INFO:tasks.workunit.client.0.smithi089.stdout: create 236/31 file after the snapshot 2023-10-30T12:32:08.017 INFO:tasks.workunit.client.0.smithi089.stdout: create 236/32 file after the snapshot 2023-10-30T12:32:08.019 INFO:tasks.workunit.client.0.smithi089.stdout: create 236/33 file after the snapshot 2023-10-30T12:32:08.021 INFO:tasks.workunit.client.0.smithi089.stdout: create 236/34 file after the snapshot 2023-10-30T12:32:08.023 INFO:tasks.workunit.client.0.smithi089.stdout: create 236/35 file after the snapshot 2023-10-30T12:32:08.025 INFO:tasks.workunit.client.0.smithi089.stdout: create 236/36 file after the snapshot 2023-10-30T12:32:08.027 INFO:tasks.workunit.client.0.smithi089.stdout: create 236/37 file after the snapshot 2023-10-30T12:32:08.029 INFO:tasks.workunit.client.0.smithi089.stdout: create 236/38 file after the snapshot 2023-10-30T12:32:08.031 INFO:tasks.workunit.client.0.smithi089.stdout: create 236/39 file after the snapshot 2023-10-30T12:32:08.033 INFO:tasks.workunit.client.0.smithi089.stdout: create 237/.snap/snap-subdir-test 2023-10-30T12:32:08.056 INFO:tasks.workunit.client.0.smithi089.stdout: create 237/30 file after the snapshot 2023-10-30T12:32:08.057 INFO:tasks.workunit.client.0.smithi089.stdout: create 237/31 file after the snapshot 2023-10-30T12:32:08.058 INFO:tasks.workunit.client.0.smithi089.stdout: create 237/32 file after the snapshot 2023-10-30T12:32:08.059 INFO:tasks.workunit.client.0.smithi089.stdout: create 237/33 file after the snapshot 2023-10-30T12:32:08.060 INFO:tasks.workunit.client.0.smithi089.stdout: create 237/34 file after the snapshot 2023-10-30T12:32:08.061 INFO:tasks.workunit.client.0.smithi089.stdout: create 237/35 file after the snapshot 2023-10-30T12:32:08.062 INFO:tasks.workunit.client.0.smithi089.stdout: create 237/36 file after the snapshot 2023-10-30T12:32:08.063 INFO:tasks.workunit.client.0.smithi089.stdout: create 237/37 file after the snapshot 2023-10-30T12:32:08.065 INFO:tasks.workunit.client.0.smithi089.stdout: create 237/38 file after the snapshot 2023-10-30T12:32:08.066 INFO:tasks.workunit.client.0.smithi089.stdout: create 237/39 file after the snapshot 2023-10-30T12:32:08.067 INFO:tasks.workunit.client.0.smithi089.stdout: create 238/.snap/snap-subdir-test 2023-10-30T12:32:08.091 INFO:tasks.workunit.client.0.smithi089.stdout: create 238/30 file after the snapshot 2023-10-30T12:32:08.094 INFO:tasks.workunit.client.0.smithi089.stdout: create 238/31 file after the snapshot 2023-10-30T12:32:08.096 INFO:tasks.workunit.client.0.smithi089.stdout: create 238/32 file after the snapshot 2023-10-30T12:32:08.098 INFO:tasks.workunit.client.0.smithi089.stdout: create 238/33 file after the snapshot 2023-10-30T12:32:08.100 INFO:tasks.workunit.client.0.smithi089.stdout: create 238/34 file after the snapshot 2023-10-30T12:32:08.102 INFO:tasks.workunit.client.0.smithi089.stdout: create 238/35 file after the snapshot 2023-10-30T12:32:08.104 INFO:tasks.workunit.client.0.smithi089.stdout: create 238/36 file after the snapshot 2023-10-30T12:32:08.106 INFO:tasks.workunit.client.0.smithi089.stdout: create 238/37 file after the snapshot 2023-10-30T12:32:08.107 INFO:tasks.workunit.client.0.smithi089.stdout: create 238/38 file after the snapshot 2023-10-30T12:32:08.109 INFO:tasks.workunit.client.0.smithi089.stdout: create 238/39 file after the snapshot 2023-10-30T12:32:08.111 INFO:tasks.workunit.client.0.smithi089.stdout: create 239/.snap/snap-subdir-test 2023-10-30T12:32:08.134 INFO:tasks.workunit.client.0.smithi089.stdout: create 239/30 file after the snapshot 2023-10-30T12:32:08.136 INFO:tasks.workunit.client.0.smithi089.stdout: create 239/31 file after the snapshot 2023-10-30T12:32:08.138 INFO:tasks.workunit.client.0.smithi089.stdout: create 239/32 file after the snapshot 2023-10-30T12:32:08.140 INFO:tasks.workunit.client.0.smithi089.stdout: create 239/33 file after the snapshot 2023-10-30T12:32:08.142 INFO:tasks.workunit.client.0.smithi089.stdout: create 239/34 file after the snapshot 2023-10-30T12:32:08.144 INFO:tasks.workunit.client.0.smithi089.stdout: create 239/35 file after the snapshot 2023-10-30T12:32:08.145 INFO:tasks.workunit.client.0.smithi089.stdout: create 239/36 file after the snapshot 2023-10-30T12:32:08.148 INFO:tasks.workunit.client.0.smithi089.stdout: create 239/37 file after the snapshot 2023-10-30T12:32:08.149 INFO:tasks.workunit.client.0.smithi089.stdout: create 239/38 file after the snapshot 2023-10-30T12:32:08.151 INFO:tasks.workunit.client.0.smithi089.stdout: create 239/39 file after the snapshot 2023-10-30T12:32:08.153 INFO:tasks.workunit.client.0.smithi089.stdout: create 240/.snap/snap-subdir-test 2023-10-30T12:32:08.180 INFO:tasks.workunit.client.0.smithi089.stdout: create 240/30 file after the snapshot 2023-10-30T12:32:08.182 INFO:tasks.workunit.client.0.smithi089.stdout: create 240/31 file after the snapshot 2023-10-30T12:32:08.184 INFO:tasks.workunit.client.0.smithi089.stdout: create 240/32 file after the snapshot 2023-10-30T12:32:08.186 INFO:tasks.workunit.client.0.smithi089.stdout: create 240/33 file after the snapshot 2023-10-30T12:32:08.188 INFO:tasks.workunit.client.0.smithi089.stdout: create 240/34 file after the snapshot 2023-10-30T12:32:08.190 INFO:tasks.workunit.client.0.smithi089.stdout: create 240/35 file after the snapshot 2023-10-30T12:32:08.192 INFO:tasks.workunit.client.0.smithi089.stdout: create 240/36 file after the snapshot 2023-10-30T12:32:08.193 INFO:tasks.workunit.client.0.smithi089.stdout: create 240/37 file after the snapshot 2023-10-30T12:32:08.195 INFO:tasks.workunit.client.0.smithi089.stdout: create 240/38 file after the snapshot 2023-10-30T12:32:08.197 INFO:tasks.workunit.client.0.smithi089.stdout: create 240/39 file after the snapshot 2023-10-30T12:32:08.199 INFO:tasks.workunit.client.0.smithi089.stdout: create 241/.snap/snap-subdir-test 2023-10-30T12:32:08.229 INFO:tasks.workunit.client.0.smithi089.stdout: create 241/30 file after the snapshot 2023-10-30T12:32:08.232 INFO:tasks.workunit.client.0.smithi089.stdout: create 241/31 file after the snapshot 2023-10-30T12:32:08.234 INFO:tasks.workunit.client.0.smithi089.stdout: create 241/32 file after the snapshot 2023-10-30T12:32:08.235 INFO:tasks.workunit.client.0.smithi089.stdout: create 241/33 file after the snapshot 2023-10-30T12:32:08.237 INFO:tasks.workunit.client.0.smithi089.stdout: create 241/34 file after the snapshot 2023-10-30T12:32:08.439 INFO:tasks.workunit.client.0.smithi089.stdout: create 241/35 file after the snapshot 2023-10-30T12:32:08.441 INFO:tasks.workunit.client.0.smithi089.stdout: create 241/36 file after the snapshot 2023-10-30T12:32:08.443 INFO:tasks.workunit.client.0.smithi089.stdout: create 241/37 file after the snapshot 2023-10-30T12:32:08.445 INFO:tasks.workunit.client.0.smithi089.stdout: create 241/38 file after the snapshot 2023-10-30T12:32:08.447 INFO:tasks.workunit.client.0.smithi089.stdout: create 241/39 file after the snapshot 2023-10-30T12:32:08.448 INFO:tasks.workunit.client.0.smithi089.stdout: create 242/.snap/snap-subdir-test 2023-10-30T12:32:08.469 INFO:tasks.workunit.client.0.smithi089.stdout: create 242/30 file after the snapshot 2023-10-30T12:32:08.471 INFO:tasks.workunit.client.0.smithi089.stdout: create 242/31 file after the snapshot 2023-10-30T12:32:08.473 INFO:tasks.workunit.client.0.smithi089.stdout: create 242/32 file after the snapshot 2023-10-30T12:32:08.475 INFO:tasks.workunit.client.0.smithi089.stdout: create 242/33 file after the snapshot 2023-10-30T12:32:08.477 INFO:tasks.workunit.client.0.smithi089.stdout: create 242/34 file after the snapshot 2023-10-30T12:32:08.479 INFO:tasks.workunit.client.0.smithi089.stdout: create 242/35 file after the snapshot 2023-10-30T12:32:08.481 INFO:tasks.workunit.client.0.smithi089.stdout: create 242/36 file after the snapshot 2023-10-30T12:32:08.483 INFO:tasks.workunit.client.0.smithi089.stdout: create 242/37 file after the snapshot 2023-10-30T12:32:08.485 INFO:tasks.workunit.client.0.smithi089.stdout: create 242/38 file after the snapshot 2023-10-30T12:32:08.488 INFO:tasks.workunit.client.0.smithi089.stdout: create 242/39 file after the snapshot 2023-10-30T12:32:08.490 INFO:tasks.workunit.client.0.smithi089.stdout: create 243/.snap/snap-subdir-test 2023-10-30T12:32:08.516 INFO:tasks.workunit.client.0.smithi089.stdout: create 243/30 file after the snapshot 2023-10-30T12:32:08.519 INFO:tasks.workunit.client.0.smithi089.stdout: create 243/31 file after the snapshot 2023-10-30T12:32:08.520 INFO:tasks.workunit.client.0.smithi089.stdout: create 243/32 file after the snapshot 2023-10-30T12:32:08.522 INFO:tasks.workunit.client.0.smithi089.stdout: create 243/33 file after the snapshot 2023-10-30T12:32:08.524 INFO:tasks.workunit.client.0.smithi089.stdout: create 243/34 file after the snapshot 2023-10-30T12:32:08.526 INFO:tasks.workunit.client.0.smithi089.stdout: create 243/35 file after the snapshot 2023-10-30T12:32:08.528 INFO:tasks.workunit.client.0.smithi089.stdout: create 243/36 file after the snapshot 2023-10-30T12:32:08.530 INFO:tasks.workunit.client.0.smithi089.stdout: create 243/37 file after the snapshot 2023-10-30T12:32:08.532 INFO:tasks.workunit.client.0.smithi089.stdout: create 243/38 file after the snapshot 2023-10-30T12:32:08.534 INFO:tasks.workunit.client.0.smithi089.stdout: create 243/39 file after the snapshot 2023-10-30T12:32:08.536 INFO:tasks.workunit.client.0.smithi089.stdout: create 244/.snap/snap-subdir-test 2023-10-30T12:32:09.271 INFO:tasks.workunit.client.0.smithi089.stdout: create 244/30 file after the snapshot 2023-10-30T12:32:09.274 INFO:tasks.workunit.client.0.smithi089.stdout: create 244/31 file after the snapshot 2023-10-30T12:32:09.276 INFO:tasks.workunit.client.0.smithi089.stdout: create 244/32 file after the snapshot 2023-10-30T12:32:09.278 INFO:tasks.workunit.client.0.smithi089.stdout: create 244/33 file after the snapshot 2023-10-30T12:32:09.280 INFO:tasks.workunit.client.0.smithi089.stdout: create 244/34 file after the snapshot 2023-10-30T12:32:09.282 INFO:tasks.workunit.client.0.smithi089.stdout: create 244/35 file after the snapshot 2023-10-30T12:32:09.284 INFO:tasks.workunit.client.0.smithi089.stdout: create 244/36 file after the snapshot 2023-10-30T12:32:09.286 INFO:tasks.workunit.client.0.smithi089.stdout: create 244/37 file after the snapshot 2023-10-30T12:32:09.288 INFO:tasks.workunit.client.0.smithi089.stdout: create 244/38 file after the snapshot 2023-10-30T12:32:09.290 INFO:tasks.workunit.client.0.smithi089.stdout: create 244/39 file after the snapshot 2023-10-30T12:32:09.292 INFO:tasks.workunit.client.0.smithi089.stdout: create 245/.snap/snap-subdir-test 2023-10-30T12:32:09.313 INFO:tasks.workunit.client.0.smithi089.stdout: create 245/30 file after the snapshot 2023-10-30T12:32:09.314 INFO:tasks.workunit.client.0.smithi089.stdout: create 245/31 file after the snapshot 2023-10-30T12:32:09.315 INFO:tasks.workunit.client.0.smithi089.stdout: create 245/32 file after the snapshot 2023-10-30T12:32:09.316 INFO:tasks.workunit.client.0.smithi089.stdout: create 245/33 file after the snapshot 2023-10-30T12:32:09.317 INFO:tasks.workunit.client.0.smithi089.stdout: create 245/34 file after the snapshot 2023-10-30T12:32:09.318 INFO:tasks.workunit.client.0.smithi089.stdout: create 245/35 file after the snapshot 2023-10-30T12:32:09.319 INFO:tasks.workunit.client.0.smithi089.stdout: create 245/36 file after the snapshot 2023-10-30T12:32:09.319 INFO:tasks.workunit.client.0.smithi089.stdout: create 245/37 file after the snapshot 2023-10-30T12:32:09.320 INFO:tasks.workunit.client.0.smithi089.stdout: create 245/38 file after the snapshot 2023-10-30T12:32:09.321 INFO:tasks.workunit.client.0.smithi089.stdout: create 245/39 file after the snapshot 2023-10-30T12:32:09.322 INFO:tasks.workunit.client.0.smithi089.stdout: create 246/.snap/snap-subdir-test 2023-10-30T12:32:09.339 INFO:tasks.workunit.client.0.smithi089.stdout: create 246/30 file after the snapshot 2023-10-30T12:32:09.342 INFO:tasks.workunit.client.0.smithi089.stdout: create 246/31 file after the snapshot 2023-10-30T12:32:09.343 INFO:tasks.workunit.client.0.smithi089.stdout: create 246/32 file after the snapshot 2023-10-30T12:32:09.345 INFO:tasks.workunit.client.0.smithi089.stdout: create 246/33 file after the snapshot 2023-10-30T12:32:09.347 INFO:tasks.workunit.client.0.smithi089.stdout: create 246/34 file after the snapshot 2023-10-30T12:32:09.349 INFO:tasks.workunit.client.0.smithi089.stdout: create 246/35 file after the snapshot 2023-10-30T12:32:09.351 INFO:tasks.workunit.client.0.smithi089.stdout: create 246/36 file after the snapshot 2023-10-30T12:32:09.354 INFO:tasks.workunit.client.0.smithi089.stdout: create 246/37 file after the snapshot 2023-10-30T12:32:09.355 INFO:tasks.workunit.client.0.smithi089.stdout: create 246/38 file after the snapshot 2023-10-30T12:32:09.357 INFO:tasks.workunit.client.0.smithi089.stdout: create 246/39 file after the snapshot 2023-10-30T12:32:09.359 INFO:tasks.workunit.client.0.smithi089.stdout: create 247/.snap/snap-subdir-test 2023-10-30T12:32:09.378 INFO:tasks.workunit.client.0.smithi089.stdout: create 247/30 file after the snapshot 2023-10-30T12:32:09.381 INFO:tasks.workunit.client.0.smithi089.stdout: create 247/31 file after the snapshot 2023-10-30T12:32:09.383 INFO:tasks.workunit.client.0.smithi089.stdout: create 247/32 file after the snapshot 2023-10-30T12:32:09.385 INFO:tasks.workunit.client.0.smithi089.stdout: create 247/33 file after the snapshot 2023-10-30T12:32:09.386 INFO:tasks.workunit.client.0.smithi089.stdout: create 247/34 file after the snapshot 2023-10-30T12:32:09.388 INFO:tasks.workunit.client.0.smithi089.stdout: create 247/35 file after the snapshot 2023-10-30T12:32:09.390 INFO:tasks.workunit.client.0.smithi089.stdout: create 247/36 file after the snapshot 2023-10-30T12:32:09.392 INFO:tasks.workunit.client.0.smithi089.stdout: create 247/37 file after the snapshot 2023-10-30T12:32:09.394 INFO:tasks.workunit.client.0.smithi089.stdout: create 247/38 file after the snapshot 2023-10-30T12:32:09.396 INFO:tasks.workunit.client.0.smithi089.stdout: create 247/39 file after the snapshot 2023-10-30T12:32:09.398 INFO:tasks.workunit.client.0.smithi089.stdout: create 248/.snap/snap-subdir-test 2023-10-30T12:32:09.438 INFO:tasks.workunit.client.0.smithi089.stdout: create 248/30 file after the snapshot 2023-10-30T12:32:09.441 INFO:tasks.workunit.client.0.smithi089.stdout: create 248/31 file after the snapshot 2023-10-30T12:32:09.443 INFO:tasks.workunit.client.0.smithi089.stdout: create 248/32 file after the snapshot 2023-10-30T12:32:09.445 INFO:tasks.workunit.client.0.smithi089.stdout: create 248/33 file after the snapshot 2023-10-30T12:32:09.446 INFO:tasks.workunit.client.0.smithi089.stdout: create 248/34 file after the snapshot 2023-10-30T12:32:09.448 INFO:tasks.workunit.client.0.smithi089.stdout: create 248/35 file after the snapshot 2023-10-30T12:32:09.450 INFO:tasks.workunit.client.0.smithi089.stdout: create 248/36 file after the snapshot 2023-10-30T12:32:09.452 INFO:tasks.workunit.client.0.smithi089.stdout: create 248/37 file after the snapshot 2023-10-30T12:32:09.454 INFO:tasks.workunit.client.0.smithi089.stdout: create 248/38 file after the snapshot 2023-10-30T12:32:09.456 INFO:tasks.workunit.client.0.smithi089.stdout: create 248/39 file after the snapshot 2023-10-30T12:32:09.458 INFO:tasks.workunit.client.0.smithi089.stdout: create 249/.snap/snap-subdir-test 2023-10-30T12:32:09.482 INFO:tasks.workunit.client.0.smithi089.stdout: create 249/30 file after the snapshot 2023-10-30T12:32:09.485 INFO:tasks.workunit.client.0.smithi089.stdout: create 249/31 file after the snapshot 2023-10-30T12:32:09.487 INFO:tasks.workunit.client.0.smithi089.stdout: create 249/32 file after the snapshot 2023-10-30T12:32:09.489 INFO:tasks.workunit.client.0.smithi089.stdout: create 249/33 file after the snapshot 2023-10-30T12:32:09.491 INFO:tasks.workunit.client.0.smithi089.stdout: create 249/34 file after the snapshot 2023-10-30T12:32:09.493 INFO:tasks.workunit.client.0.smithi089.stdout: create 249/35 file after the snapshot 2023-10-30T12:32:09.495 INFO:tasks.workunit.client.0.smithi089.stdout: create 249/36 file after the snapshot 2023-10-30T12:32:09.497 INFO:tasks.workunit.client.0.smithi089.stdout: create 249/37 file after the snapshot 2023-10-30T12:32:09.499 INFO:tasks.workunit.client.0.smithi089.stdout: create 249/38 file after the snapshot 2023-10-30T12:32:09.501 INFO:tasks.workunit.client.0.smithi089.stdout: create 249/39 file after the snapshot 2023-10-30T12:32:09.503 INFO:tasks.workunit.client.0.smithi089.stdout: create 250/.snap/snap-subdir-test 2023-10-30T12:32:09.526 INFO:tasks.workunit.client.0.smithi089.stdout: create 250/30 file after the snapshot 2023-10-30T12:32:09.528 INFO:tasks.workunit.client.0.smithi089.stdout: create 250/31 file after the snapshot 2023-10-30T12:32:09.530 INFO:tasks.workunit.client.0.smithi089.stdout: create 250/32 file after the snapshot 2023-10-30T12:32:09.532 INFO:tasks.workunit.client.0.smithi089.stdout: create 250/33 file after the snapshot 2023-10-30T12:32:09.534 INFO:tasks.workunit.client.0.smithi089.stdout: create 250/34 file after the snapshot 2023-10-30T12:32:09.536 INFO:tasks.workunit.client.0.smithi089.stdout: create 250/35 file after the snapshot 2023-10-30T12:32:09.538 INFO:tasks.workunit.client.0.smithi089.stdout: create 250/36 file after the snapshot 2023-10-30T12:32:09.540 INFO:tasks.workunit.client.0.smithi089.stdout: create 250/37 file after the snapshot 2023-10-30T12:32:09.542 INFO:tasks.workunit.client.0.smithi089.stdout: create 250/38 file after the snapshot 2023-10-30T12:32:09.544 INFO:tasks.workunit.client.0.smithi089.stdout: create 250/39 file after the snapshot 2023-10-30T12:32:09.546 INFO:tasks.workunit.client.0.smithi089.stdout: create 251/.snap/snap-subdir-test 2023-10-30T12:32:09.566 INFO:tasks.workunit.client.0.smithi089.stdout: create 251/30 file after the snapshot 2023-10-30T12:32:09.567 INFO:tasks.workunit.client.0.smithi089.stdout: create 251/31 file after the snapshot 2023-10-30T12:32:09.568 INFO:tasks.workunit.client.0.smithi089.stdout: create 251/32 file after the snapshot 2023-10-30T12:32:09.569 INFO:tasks.workunit.client.0.smithi089.stdout: create 251/33 file after the snapshot 2023-10-30T12:32:09.569 INFO:tasks.workunit.client.0.smithi089.stdout: create 251/34 file after the snapshot 2023-10-30T12:32:09.570 INFO:tasks.workunit.client.0.smithi089.stdout: create 251/35 file after the snapshot 2023-10-30T12:32:09.571 INFO:tasks.workunit.client.0.smithi089.stdout: create 251/36 file after the snapshot 2023-10-30T12:32:09.571 INFO:tasks.workunit.client.0.smithi089.stdout: create 251/37 file after the snapshot 2023-10-30T12:32:09.572 INFO:tasks.workunit.client.0.smithi089.stdout: create 251/38 file after the snapshot 2023-10-30T12:32:09.573 INFO:tasks.workunit.client.0.smithi089.stdout: create 251/39 file after the snapshot 2023-10-30T12:32:09.574 INFO:tasks.workunit.client.0.smithi089.stdout: create 252/.snap/snap-subdir-test 2023-10-30T12:32:09.590 INFO:tasks.workunit.client.0.smithi089.stdout: create 252/30 file after the snapshot 2023-10-30T12:32:09.593 INFO:tasks.workunit.client.0.smithi089.stdout: create 252/31 file after the snapshot 2023-10-30T12:32:09.595 INFO:tasks.workunit.client.0.smithi089.stdout: create 252/32 file after the snapshot 2023-10-30T12:32:09.597 INFO:tasks.workunit.client.0.smithi089.stdout: create 252/33 file after the snapshot 2023-10-30T12:32:09.599 INFO:tasks.workunit.client.0.smithi089.stdout: create 252/34 file after the snapshot 2023-10-30T12:32:09.601 INFO:tasks.workunit.client.0.smithi089.stdout: create 252/35 file after the snapshot 2023-10-30T12:32:09.603 INFO:tasks.workunit.client.0.smithi089.stdout: create 252/36 file after the snapshot 2023-10-30T12:32:09.605 INFO:tasks.workunit.client.0.smithi089.stdout: create 252/37 file after the snapshot 2023-10-30T12:32:09.607 INFO:tasks.workunit.client.0.smithi089.stdout: create 252/38 file after the snapshot 2023-10-30T12:32:09.609 INFO:tasks.workunit.client.0.smithi089.stdout: create 252/39 file after the snapshot 2023-10-30T12:32:09.611 INFO:tasks.workunit.client.0.smithi089.stdout: create 253/.snap/snap-subdir-test 2023-10-30T12:32:09.635 INFO:tasks.workunit.client.0.smithi089.stdout: create 253/30 file after the snapshot 2023-10-30T12:32:09.637 INFO:tasks.workunit.client.0.smithi089.stdout: create 253/31 file after the snapshot 2023-10-30T12:32:09.639 INFO:tasks.workunit.client.0.smithi089.stdout: create 253/32 file after the snapshot 2023-10-30T12:32:09.641 INFO:tasks.workunit.client.0.smithi089.stdout: create 253/33 file after the snapshot 2023-10-30T12:32:09.643 INFO:tasks.workunit.client.0.smithi089.stdout: create 253/34 file after the snapshot 2023-10-30T12:32:09.645 INFO:tasks.workunit.client.0.smithi089.stdout: create 253/35 file after the snapshot 2023-10-30T12:32:09.647 INFO:tasks.workunit.client.0.smithi089.stdout: create 253/36 file after the snapshot 2023-10-30T12:32:09.649 INFO:tasks.workunit.client.0.smithi089.stdout: create 253/37 file after the snapshot 2023-10-30T12:32:09.651 INFO:tasks.workunit.client.0.smithi089.stdout: create 253/38 file after the snapshot 2023-10-30T12:32:09.653 INFO:tasks.workunit.client.0.smithi089.stdout: create 253/39 file after the snapshot 2023-10-30T12:32:09.656 INFO:tasks.workunit.client.0.smithi089.stdout: create 254/.snap/snap-subdir-test 2023-10-30T12:32:10.681 INFO:tasks.workunit.client.0.smithi089.stdout: create 254/30 file after the snapshot 2023-10-30T12:32:10.684 INFO:tasks.workunit.client.0.smithi089.stdout: create 254/31 file after the snapshot 2023-10-30T12:32:10.686 INFO:tasks.workunit.client.0.smithi089.stdout: create 254/32 file after the snapshot 2023-10-30T12:32:10.687 INFO:tasks.workunit.client.0.smithi089.stdout: create 254/33 file after the snapshot 2023-10-30T12:32:10.690 INFO:tasks.workunit.client.0.smithi089.stdout: create 254/34 file after the snapshot 2023-10-30T12:32:10.692 INFO:tasks.workunit.client.0.smithi089.stdout: create 254/35 file after the snapshot 2023-10-30T12:32:10.694 INFO:tasks.workunit.client.0.smithi089.stdout: create 254/36 file after the snapshot 2023-10-30T12:32:10.696 INFO:tasks.workunit.client.0.smithi089.stdout: create 254/37 file after the snapshot 2023-10-30T12:32:10.698 INFO:tasks.workunit.client.0.smithi089.stdout: create 254/38 file after the snapshot 2023-10-30T12:32:10.700 INFO:tasks.workunit.client.0.smithi089.stdout: create 254/39 file after the snapshot 2023-10-30T12:32:10.702 INFO:tasks.workunit.client.0.smithi089.stdout: create 255/.snap/snap-subdir-test 2023-10-30T12:32:10.733 INFO:tasks.workunit.client.0.smithi089.stdout: create 255/30 file after the snapshot 2023-10-30T12:32:10.736 INFO:tasks.workunit.client.0.smithi089.stdout: create 255/31 file after the snapshot 2023-10-30T12:32:10.738 INFO:tasks.workunit.client.0.smithi089.stdout: create 255/32 file after the snapshot 2023-10-30T12:32:10.739 INFO:tasks.workunit.client.0.smithi089.stdout: create 255/33 file after the snapshot 2023-10-30T12:32:10.741 INFO:tasks.workunit.client.0.smithi089.stdout: create 255/34 file after the snapshot 2023-10-30T12:32:10.743 INFO:tasks.workunit.client.0.smithi089.stdout: create 255/35 file after the snapshot 2023-10-30T12:32:10.745 INFO:tasks.workunit.client.0.smithi089.stdout: create 255/36 file after the snapshot 2023-10-30T12:32:10.747 INFO:tasks.workunit.client.0.smithi089.stdout: create 255/37 file after the snapshot 2023-10-30T12:32:10.749 INFO:tasks.workunit.client.0.smithi089.stdout: create 255/38 file after the snapshot 2023-10-30T12:32:10.751 INFO:tasks.workunit.client.0.smithi089.stdout: create 255/39 file after the snapshot 2023-10-30T12:32:10.753 INFO:tasks.workunit.client.0.smithi089.stdout: create 256/.snap/snap-subdir-test 2023-10-30T12:32:10.772 INFO:tasks.workunit.client.0.smithi089.stdout: create 256/30 file after the snapshot 2023-10-30T12:32:10.774 INFO:tasks.workunit.client.0.smithi089.stdout: create 256/31 file after the snapshot 2023-10-30T12:32:10.776 INFO:tasks.workunit.client.0.smithi089.stdout: create 256/32 file after the snapshot 2023-10-30T12:32:10.778 INFO:tasks.workunit.client.0.smithi089.stdout: create 256/33 file after the snapshot 2023-10-30T12:32:10.780 INFO:tasks.workunit.client.0.smithi089.stdout: create 256/34 file after the snapshot 2023-10-30T12:32:10.782 INFO:tasks.workunit.client.0.smithi089.stdout: create 256/35 file after the snapshot 2023-10-30T12:32:10.783 INFO:tasks.workunit.client.0.smithi089.stdout: create 256/36 file after the snapshot 2023-10-30T12:32:10.785 INFO:tasks.workunit.client.0.smithi089.stdout: create 256/37 file after the snapshot 2023-10-30T12:32:10.787 INFO:tasks.workunit.client.0.smithi089.stdout: create 256/38 file after the snapshot 2023-10-30T12:32:10.789 INFO:tasks.workunit.client.0.smithi089.stdout: create 256/39 file after the snapshot 2023-10-30T12:32:10.791 INFO:tasks.workunit.client.0.smithi089.stdout: create 257/.snap/snap-subdir-test 2023-10-30T12:32:10.822 INFO:tasks.workunit.client.0.smithi089.stdout: create 257/30 file after the snapshot 2023-10-30T12:32:10.824 INFO:tasks.workunit.client.0.smithi089.stdout: create 257/31 file after the snapshot 2023-10-30T12:32:10.826 INFO:tasks.workunit.client.0.smithi089.stdout: create 257/32 file after the snapshot 2023-10-30T12:32:10.828 INFO:tasks.workunit.client.0.smithi089.stdout: create 257/33 file after the snapshot 2023-10-30T12:32:10.830 INFO:tasks.workunit.client.0.smithi089.stdout: create 257/34 file after the snapshot 2023-10-30T12:32:10.832 INFO:tasks.workunit.client.0.smithi089.stdout: create 257/35 file after the snapshot 2023-10-30T12:32:10.834 INFO:tasks.workunit.client.0.smithi089.stdout: create 257/36 file after the snapshot 2023-10-30T12:32:10.836 INFO:tasks.workunit.client.0.smithi089.stdout: create 257/37 file after the snapshot 2023-10-30T12:32:10.838 INFO:tasks.workunit.client.0.smithi089.stdout: create 257/38 file after the snapshot 2023-10-30T12:32:10.839 INFO:tasks.workunit.client.0.smithi089.stdout: create 257/39 file after the snapshot 2023-10-30T12:32:10.841 INFO:tasks.workunit.client.0.smithi089.stdout: create 258/.snap/snap-subdir-test 2023-10-30T12:32:10.872 INFO:tasks.workunit.client.0.smithi089.stdout: create 258/30 file after the snapshot 2023-10-30T12:32:10.874 INFO:tasks.workunit.client.0.smithi089.stdout: create 258/31 file after the snapshot 2023-10-30T12:32:10.876 INFO:tasks.workunit.client.0.smithi089.stdout: create 258/32 file after the snapshot 2023-10-30T12:32:10.878 INFO:tasks.workunit.client.0.smithi089.stdout: create 258/33 file after the snapshot 2023-10-30T12:32:11.658 INFO:tasks.workunit.client.0.smithi089.stdout: create 258/34 file after the snapshot 2023-10-30T12:32:11.707 INFO:tasks.workunit.client.0.smithi089.stdout: create 258/35 file after the snapshot 2023-10-30T12:32:11.709 INFO:tasks.workunit.client.0.smithi089.stdout: create 258/36 file after the snapshot 2023-10-30T12:32:11.711 INFO:tasks.workunit.client.0.smithi089.stdout: create 258/37 file after the snapshot 2023-10-30T12:32:11.714 INFO:tasks.workunit.client.0.smithi089.stdout: create 258/38 file after the snapshot 2023-10-30T12:32:11.716 INFO:tasks.workunit.client.0.smithi089.stdout: create 258/39 file after the snapshot 2023-10-30T12:32:11.718 INFO:tasks.workunit.client.0.smithi089.stdout: create 259/.snap/snap-subdir-test 2023-10-30T12:32:11.743 INFO:tasks.workunit.client.0.smithi089.stdout: create 259/30 file after the snapshot 2023-10-30T12:32:11.746 INFO:tasks.workunit.client.0.smithi089.stdout: create 259/31 file after the snapshot 2023-10-30T12:32:11.748 INFO:tasks.workunit.client.0.smithi089.stdout: create 259/32 file after the snapshot 2023-10-30T12:32:11.750 INFO:tasks.workunit.client.0.smithi089.stdout: create 259/33 file after the snapshot 2023-10-30T12:32:11.752 INFO:tasks.workunit.client.0.smithi089.stdout: create 259/34 file after the snapshot 2023-10-30T12:32:11.754 INFO:tasks.workunit.client.0.smithi089.stdout: create 259/35 file after the snapshot 2023-10-30T12:32:11.756 INFO:tasks.workunit.client.0.smithi089.stdout: create 259/36 file after the snapshot 2023-10-30T12:32:11.758 INFO:tasks.workunit.client.0.smithi089.stdout: create 259/37 file after the snapshot 2023-10-30T12:32:11.760 INFO:tasks.workunit.client.0.smithi089.stdout: create 259/38 file after the snapshot 2023-10-30T12:32:11.762 INFO:tasks.workunit.client.0.smithi089.stdout: create 259/39 file after the snapshot 2023-10-30T12:32:11.764 INFO:tasks.workunit.client.0.smithi089.stdout: create 260/.snap/snap-subdir-test 2023-10-30T12:32:11.794 INFO:tasks.workunit.client.0.smithi089.stdout: create 260/30 file after the snapshot 2023-10-30T12:32:11.797 INFO:tasks.workunit.client.0.smithi089.stdout: create 260/31 file after the snapshot 2023-10-30T12:32:11.798 INFO:tasks.workunit.client.0.smithi089.stdout: create 260/32 file after the snapshot 2023-10-30T12:32:11.800 INFO:tasks.workunit.client.0.smithi089.stdout: create 260/33 file after the snapshot 2023-10-30T12:32:11.802 INFO:tasks.workunit.client.0.smithi089.stdout: create 260/34 file after the snapshot 2023-10-30T12:32:11.804 INFO:tasks.workunit.client.0.smithi089.stdout: create 260/35 file after the snapshot 2023-10-30T12:32:11.806 INFO:tasks.workunit.client.0.smithi089.stdout: create 260/36 file after the snapshot 2023-10-30T12:32:11.808 INFO:tasks.workunit.client.0.smithi089.stdout: create 260/37 file after the snapshot 2023-10-30T12:32:11.810 INFO:tasks.workunit.client.0.smithi089.stdout: create 260/38 file after the snapshot 2023-10-30T12:32:11.812 INFO:tasks.workunit.client.0.smithi089.stdout: create 260/39 file after the snapshot 2023-10-30T12:32:11.814 INFO:tasks.workunit.client.0.smithi089.stdout: create 261/.snap/snap-subdir-test 2023-10-30T12:32:11.835 INFO:tasks.workunit.client.0.smithi089.stdout: create 261/30 file after the snapshot 2023-10-30T12:32:11.839 INFO:tasks.workunit.client.0.smithi089.stdout: create 261/31 file after the snapshot 2023-10-30T12:32:11.842 INFO:tasks.workunit.client.0.smithi089.stdout: create 261/32 file after the snapshot 2023-10-30T12:32:11.844 INFO:tasks.workunit.client.0.smithi089.stdout: create 261/33 file after the snapshot 2023-10-30T12:32:11.847 INFO:tasks.workunit.client.0.smithi089.stdout: create 261/34 file after the snapshot 2023-10-30T12:32:11.850 INFO:tasks.workunit.client.0.smithi089.stdout: create 261/35 file after the snapshot 2023-10-30T12:32:11.853 INFO:tasks.workunit.client.0.smithi089.stdout: create 261/36 file after the snapshot 2023-10-30T12:32:11.855 INFO:tasks.workunit.client.0.smithi089.stdout: create 261/37 file after the snapshot 2023-10-30T12:32:11.858 INFO:tasks.workunit.client.0.smithi089.stdout: create 261/38 file after the snapshot 2023-10-30T12:32:11.861 INFO:tasks.workunit.client.0.smithi089.stdout: create 261/39 file after the snapshot 2023-10-30T12:32:11.863 INFO:tasks.workunit.client.0.smithi089.stdout: create 262/.snap/snap-subdir-test 2023-10-30T12:32:12.125 INFO:tasks.workunit.client.0.smithi089.stdout: create 262/30 file after the snapshot 2023-10-30T12:32:12.127 INFO:tasks.workunit.client.0.smithi089.stdout: create 262/31 file after the snapshot 2023-10-30T12:32:12.129 INFO:tasks.workunit.client.0.smithi089.stdout: create 262/32 file after the snapshot 2023-10-30T12:32:12.130 INFO:tasks.workunit.client.0.smithi089.stdout: create 262/33 file after the snapshot 2023-10-30T12:32:12.132 INFO:tasks.workunit.client.0.smithi089.stdout: create 262/34 file after the snapshot 2023-10-30T12:32:12.134 INFO:tasks.workunit.client.0.smithi089.stdout: create 262/35 file after the snapshot 2023-10-30T12:32:12.136 INFO:tasks.workunit.client.0.smithi089.stdout: create 262/36 file after the snapshot 2023-10-30T12:32:12.138 INFO:tasks.workunit.client.0.smithi089.stdout: create 262/37 file after the snapshot 2023-10-30T12:32:12.390 INFO:tasks.workunit.client.0.smithi089.stdout: create 262/38 file after the snapshot 2023-10-30T12:32:12.392 INFO:tasks.workunit.client.0.smithi089.stdout: create 262/39 file after the snapshot 2023-10-30T12:32:12.393 INFO:tasks.workunit.client.0.smithi089.stdout: create 263/.snap/snap-subdir-test 2023-10-30T12:32:12.418 INFO:tasks.workunit.client.0.smithi089.stdout: create 263/30 file after the snapshot 2023-10-30T12:32:12.420 INFO:tasks.workunit.client.0.smithi089.stdout: create 263/31 file after the snapshot 2023-10-30T12:32:12.421 INFO:tasks.workunit.client.0.smithi089.stdout: create 263/32 file after the snapshot 2023-10-30T12:32:12.422 INFO:tasks.workunit.client.0.smithi089.stdout: create 263/33 file after the snapshot 2023-10-30T12:32:12.422 INFO:tasks.workunit.client.0.smithi089.stdout: create 263/34 file after the snapshot 2023-10-30T12:32:12.423 INFO:tasks.workunit.client.0.smithi089.stdout: create 263/35 file after the snapshot 2023-10-30T12:32:12.424 INFO:tasks.workunit.client.0.smithi089.stdout: create 263/36 file after the snapshot 2023-10-30T12:32:12.425 INFO:tasks.workunit.client.0.smithi089.stdout: create 263/37 file after the snapshot 2023-10-30T12:32:12.425 INFO:tasks.workunit.client.0.smithi089.stdout: create 263/38 file after the snapshot 2023-10-30T12:32:12.426 INFO:tasks.workunit.client.0.smithi089.stdout: create 263/39 file after the snapshot 2023-10-30T12:32:12.426 INFO:tasks.workunit.client.0.smithi089.stdout: create 264/.snap/snap-subdir-test 2023-10-30T12:32:12.444 INFO:tasks.workunit.client.0.smithi089.stdout: create 264/30 file after the snapshot 2023-10-30T12:32:12.446 INFO:tasks.workunit.client.0.smithi089.stdout: create 264/31 file after the snapshot 2023-10-30T12:32:12.448 INFO:tasks.workunit.client.0.smithi089.stdout: create 264/32 file after the snapshot 2023-10-30T12:32:12.450 INFO:tasks.workunit.client.0.smithi089.stdout: create 264/33 file after the snapshot 2023-10-30T12:32:12.452 INFO:tasks.workunit.client.0.smithi089.stdout: create 264/34 file after the snapshot 2023-10-30T12:32:12.454 INFO:tasks.workunit.client.0.smithi089.stdout: create 264/35 file after the snapshot 2023-10-30T12:32:12.456 INFO:tasks.workunit.client.0.smithi089.stdout: create 264/36 file after the snapshot 2023-10-30T12:32:12.458 INFO:tasks.workunit.client.0.smithi089.stdout: create 264/37 file after the snapshot 2023-10-30T12:32:12.459 INFO:tasks.workunit.client.0.smithi089.stdout: create 264/38 file after the snapshot 2023-10-30T12:32:12.461 INFO:tasks.workunit.client.0.smithi089.stdout: create 264/39 file after the snapshot 2023-10-30T12:32:12.463 INFO:tasks.workunit.client.0.smithi089.stdout: create 265/.snap/snap-subdir-test 2023-10-30T12:32:12.488 INFO:tasks.workunit.client.0.smithi089.stdout: create 265/30 file after the snapshot 2023-10-30T12:32:12.491 INFO:tasks.workunit.client.0.smithi089.stdout: create 265/31 file after the snapshot 2023-10-30T12:32:12.493 INFO:tasks.workunit.client.0.smithi089.stdout: create 265/32 file after the snapshot 2023-10-30T12:32:12.495 INFO:tasks.workunit.client.0.smithi089.stdout: create 265/33 file after the snapshot 2023-10-30T12:32:12.497 INFO:tasks.workunit.client.0.smithi089.stdout: create 265/34 file after the snapshot 2023-10-30T12:32:12.499 INFO:tasks.workunit.client.0.smithi089.stdout: create 265/35 file after the snapshot 2023-10-30T12:32:12.500 INFO:tasks.workunit.client.0.smithi089.stdout: create 265/36 file after the snapshot 2023-10-30T12:32:12.502 INFO:tasks.workunit.client.0.smithi089.stdout: create 265/37 file after the snapshot 2023-10-30T12:32:12.505 INFO:tasks.workunit.client.0.smithi089.stdout: create 265/38 file after the snapshot 2023-10-30T12:32:12.506 INFO:tasks.workunit.client.0.smithi089.stdout: create 265/39 file after the snapshot 2023-10-30T12:32:12.508 INFO:tasks.workunit.client.0.smithi089.stdout: create 266/.snap/snap-subdir-test 2023-10-30T12:32:12.532 INFO:tasks.workunit.client.0.smithi089.stdout: create 266/30 file after the snapshot 2023-10-30T12:32:12.535 INFO:tasks.workunit.client.0.smithi089.stdout: create 266/31 file after the snapshot 2023-10-30T12:32:12.537 INFO:tasks.workunit.client.0.smithi089.stdout: create 266/32 file after the snapshot 2023-10-30T12:32:12.539 INFO:tasks.workunit.client.0.smithi089.stdout: create 266/33 file after the snapshot 2023-10-30T12:32:12.540 INFO:tasks.workunit.client.0.smithi089.stdout: create 266/34 file after the snapshot 2023-10-30T12:32:12.542 INFO:tasks.workunit.client.0.smithi089.stdout: create 266/35 file after the snapshot 2023-10-30T12:32:12.544 INFO:tasks.workunit.client.0.smithi089.stdout: create 266/36 file after the snapshot 2023-10-30T12:32:12.546 INFO:tasks.workunit.client.0.smithi089.stdout: create 266/37 file after the snapshot 2023-10-30T12:32:12.548 INFO:tasks.workunit.client.0.smithi089.stdout: create 266/38 file after the snapshot 2023-10-30T12:32:12.550 INFO:tasks.workunit.client.0.smithi089.stdout: create 266/39 file after the snapshot 2023-10-30T12:32:12.552 INFO:tasks.workunit.client.0.smithi089.stdout: create 267/.snap/snap-subdir-test 2023-10-30T12:32:12.576 INFO:tasks.workunit.client.0.smithi089.stdout: create 267/30 file after the snapshot 2023-10-30T12:32:12.578 INFO:tasks.workunit.client.0.smithi089.stdout: create 267/31 file after the snapshot 2023-10-30T12:32:12.581 INFO:tasks.workunit.client.0.smithi089.stdout: create 267/32 file after the snapshot 2023-10-30T12:32:12.582 INFO:tasks.workunit.client.0.smithi089.stdout: create 267/33 file after the snapshot 2023-10-30T12:32:12.584 INFO:tasks.workunit.client.0.smithi089.stdout: create 267/34 file after the snapshot 2023-10-30T12:32:12.586 INFO:tasks.workunit.client.0.smithi089.stdout: create 267/35 file after the snapshot 2023-10-30T12:32:12.588 INFO:tasks.workunit.client.0.smithi089.stdout: create 267/36 file after the snapshot 2023-10-30T12:32:12.590 INFO:tasks.workunit.client.0.smithi089.stdout: create 267/37 file after the snapshot 2023-10-30T12:32:12.592 INFO:tasks.workunit.client.0.smithi089.stdout: create 267/38 file after the snapshot 2023-10-30T12:32:12.594 INFO:tasks.workunit.client.0.smithi089.stdout: create 267/39 file after the snapshot 2023-10-30T12:32:12.595 INFO:tasks.workunit.client.0.smithi089.stdout: create 268/.snap/snap-subdir-test 2023-10-30T12:32:12.614 INFO:tasks.workunit.client.0.smithi089.stdout: create 268/30 file after the snapshot 2023-10-30T12:32:12.617 INFO:tasks.workunit.client.0.smithi089.stdout: create 268/31 file after the snapshot 2023-10-30T12:32:12.618 INFO:tasks.workunit.client.0.smithi089.stdout: create 268/32 file after the snapshot 2023-10-30T12:32:12.620 INFO:tasks.workunit.client.0.smithi089.stdout: create 268/33 file after the snapshot 2023-10-30T12:32:12.622 INFO:tasks.workunit.client.0.smithi089.stdout: create 268/34 file after the snapshot 2023-10-30T12:32:12.623 INFO:tasks.workunit.client.0.smithi089.stdout: create 268/35 file after the snapshot 2023-10-30T12:32:12.625 INFO:tasks.workunit.client.0.smithi089.stdout: create 268/36 file after the snapshot 2023-10-30T12:32:12.627 INFO:tasks.workunit.client.0.smithi089.stdout: create 268/37 file after the snapshot 2023-10-30T12:32:12.629 INFO:tasks.workunit.client.0.smithi089.stdout: create 268/38 file after the snapshot 2023-10-30T12:32:12.631 INFO:tasks.workunit.client.0.smithi089.stdout: create 268/39 file after the snapshot 2023-10-30T12:32:12.633 INFO:tasks.workunit.client.0.smithi089.stdout: create 269/.snap/snap-subdir-test 2023-10-30T12:32:12.658 INFO:tasks.workunit.client.0.smithi089.stdout: create 269/30 file after the snapshot 2023-10-30T12:32:12.661 INFO:tasks.workunit.client.0.smithi089.stdout: create 269/31 file after the snapshot 2023-10-30T12:32:12.662 INFO:tasks.workunit.client.0.smithi089.stdout: create 269/32 file after the snapshot 2023-10-30T12:32:12.664 INFO:tasks.workunit.client.0.smithi089.stdout: create 269/33 file after the snapshot 2023-10-30T12:32:12.666 INFO:tasks.workunit.client.0.smithi089.stdout: create 269/34 file after the snapshot 2023-10-30T12:32:12.668 INFO:tasks.workunit.client.0.smithi089.stdout: create 269/35 file after the snapshot 2023-10-30T12:32:12.670 INFO:tasks.workunit.client.0.smithi089.stdout: create 269/36 file after the snapshot 2023-10-30T12:32:12.672 INFO:tasks.workunit.client.0.smithi089.stdout: create 269/37 file after the snapshot 2023-10-30T12:32:12.674 INFO:tasks.workunit.client.0.smithi089.stdout: create 269/38 file after the snapshot 2023-10-30T12:32:12.676 INFO:tasks.workunit.client.0.smithi089.stdout: create 269/39 file after the snapshot 2023-10-30T12:32:12.677 INFO:tasks.workunit.client.0.smithi089.stdout: create 270/.snap/snap-subdir-test 2023-10-30T12:32:12.715 INFO:tasks.workunit.client.0.smithi089.stdout: create 270/30 file after the snapshot 2023-10-30T12:32:12.718 INFO:tasks.workunit.client.0.smithi089.stdout: create 270/31 file after the snapshot 2023-10-30T12:32:12.721 INFO:tasks.workunit.client.0.smithi089.stdout: create 270/32 file after the snapshot 2023-10-30T12:32:12.723 INFO:tasks.workunit.client.0.smithi089.stdout: create 270/33 file after the snapshot 2023-10-30T12:32:12.725 INFO:tasks.workunit.client.0.smithi089.stdout: create 270/34 file after the snapshot 2023-10-30T12:32:12.727 INFO:tasks.workunit.client.0.smithi089.stdout: create 270/35 file after the snapshot 2023-10-30T12:32:12.729 INFO:tasks.workunit.client.0.smithi089.stdout: create 270/36 file after the snapshot 2023-10-30T12:32:12.731 INFO:tasks.workunit.client.0.smithi089.stdout: create 270/37 file after the snapshot 2023-10-30T12:32:12.733 INFO:tasks.workunit.client.0.smithi089.stdout: create 270/38 file after the snapshot 2023-10-30T12:32:12.735 INFO:tasks.workunit.client.0.smithi089.stdout: create 270/39 file after the snapshot 2023-10-30T12:32:12.737 INFO:tasks.workunit.client.0.smithi089.stdout: create 271/.snap/snap-subdir-test 2023-10-30T12:32:12.751 INFO:tasks.workunit.client.0.smithi089.stdout: create 271/30 file after the snapshot 2023-10-30T12:32:12.753 INFO:tasks.workunit.client.0.smithi089.stdout: create 271/31 file after the snapshot 2023-10-30T12:32:12.753 INFO:tasks.workunit.client.0.smithi089.stdout: create 271/32 file after the snapshot 2023-10-30T12:32:12.754 INFO:tasks.workunit.client.0.smithi089.stdout: create 271/33 file after the snapshot 2023-10-30T12:32:12.755 INFO:tasks.workunit.client.0.smithi089.stdout: create 271/34 file after the snapshot 2023-10-30T12:32:12.755 INFO:tasks.workunit.client.0.smithi089.stdout: create 271/35 file after the snapshot 2023-10-30T12:32:12.756 INFO:tasks.workunit.client.0.smithi089.stdout: create 271/36 file after the snapshot 2023-10-30T12:32:12.757 INFO:tasks.workunit.client.0.smithi089.stdout: create 271/37 file after the snapshot 2023-10-30T12:32:12.758 INFO:tasks.workunit.client.0.smithi089.stdout: create 271/38 file after the snapshot 2023-10-30T12:32:12.758 INFO:tasks.workunit.client.0.smithi089.stdout: create 271/39 file after the snapshot 2023-10-30T12:32:12.759 INFO:tasks.workunit.client.0.smithi089.stdout: create 272/.snap/snap-subdir-test 2023-10-30T12:32:12.773 INFO:tasks.workunit.client.0.smithi089.stdout: create 272/30 file after the snapshot 2023-10-30T12:32:12.775 INFO:tasks.workunit.client.0.smithi089.stdout: create 272/31 file after the snapshot 2023-10-30T12:32:12.776 INFO:tasks.workunit.client.0.smithi089.stdout: create 272/32 file after the snapshot 2023-10-30T12:32:12.777 INFO:tasks.workunit.client.0.smithi089.stdout: create 272/33 file after the snapshot 2023-10-30T12:32:12.778 INFO:tasks.workunit.client.0.smithi089.stdout: create 272/34 file after the snapshot 2023-10-30T12:32:12.779 INFO:tasks.workunit.client.0.smithi089.stdout: create 272/35 file after the snapshot 2023-10-30T12:32:12.780 INFO:tasks.workunit.client.0.smithi089.stdout: create 272/36 file after the snapshot 2023-10-30T12:32:12.781 INFO:tasks.workunit.client.0.smithi089.stdout: create 272/37 file after the snapshot 2023-10-30T12:32:12.782 INFO:tasks.workunit.client.0.smithi089.stdout: create 272/38 file after the snapshot 2023-10-30T12:32:12.783 INFO:tasks.workunit.client.0.smithi089.stdout: create 272/39 file after the snapshot 2023-10-30T12:32:12.784 INFO:tasks.workunit.client.0.smithi089.stdout: create 273/.snap/snap-subdir-test 2023-10-30T12:32:12.964 INFO:tasks.workunit.client.0.smithi089.stdout: create 273/30 file after the snapshot 2023-10-30T12:32:12.967 INFO:tasks.workunit.client.0.smithi089.stdout: create 273/31 file after the snapshot 2023-10-30T12:32:12.969 INFO:tasks.workunit.client.0.smithi089.stdout: create 273/32 file after the snapshot 2023-10-30T12:32:12.970 INFO:tasks.workunit.client.0.smithi089.stdout: create 273/33 file after the snapshot 2023-10-30T12:32:12.972 INFO:tasks.workunit.client.0.smithi089.stdout: create 273/34 file after the snapshot 2023-10-30T12:32:12.974 INFO:tasks.workunit.client.0.smithi089.stdout: create 273/35 file after the snapshot 2023-10-30T12:32:13.203 INFO:tasks.workunit.client.0.smithi089.stdout: create 273/36 file after the snapshot 2023-10-30T12:32:13.204 INFO:tasks.workunit.client.0.smithi089.stdout: create 273/37 file after the snapshot 2023-10-30T12:32:13.207 INFO:tasks.workunit.client.0.smithi089.stdout: create 273/38 file after the snapshot 2023-10-30T12:32:13.208 INFO:tasks.workunit.client.0.smithi089.stdout: create 273/39 file after the snapshot 2023-10-30T12:32:13.210 INFO:tasks.workunit.client.0.smithi089.stdout: create 274/.snap/snap-subdir-test 2023-10-30T12:32:13.262 INFO:tasks.workunit.client.0.smithi089.stdout: create 274/30 file after the snapshot 2023-10-30T12:32:13.265 INFO:tasks.workunit.client.0.smithi089.stdout: create 274/31 file after the snapshot 2023-10-30T12:32:13.266 INFO:tasks.workunit.client.0.smithi089.stdout: create 274/32 file after the snapshot 2023-10-30T12:32:13.268 INFO:tasks.workunit.client.0.smithi089.stdout: create 274/33 file after the snapshot 2023-10-30T12:32:13.270 INFO:tasks.workunit.client.0.smithi089.stdout: create 274/34 file after the snapshot 2023-10-30T12:32:13.272 INFO:tasks.workunit.client.0.smithi089.stdout: create 274/35 file after the snapshot 2023-10-30T12:32:13.274 INFO:tasks.workunit.client.0.smithi089.stdout: create 274/36 file after the snapshot 2023-10-30T12:32:13.276 INFO:tasks.workunit.client.0.smithi089.stdout: create 274/37 file after the snapshot 2023-10-30T12:32:13.278 INFO:tasks.workunit.client.0.smithi089.stdout: create 274/38 file after the snapshot 2023-10-30T12:32:13.280 INFO:tasks.workunit.client.0.smithi089.stdout: create 274/39 file after the snapshot 2023-10-30T12:32:13.281 INFO:tasks.workunit.client.0.smithi089.stdout: create 275/.snap/snap-subdir-test 2023-10-30T12:32:13.306 INFO:tasks.workunit.client.0.smithi089.stdout: create 275/30 file after the snapshot 2023-10-30T12:32:13.309 INFO:tasks.workunit.client.0.smithi089.stdout: create 275/31 file after the snapshot 2023-10-30T12:32:13.310 INFO:tasks.workunit.client.0.smithi089.stdout: create 275/32 file after the snapshot 2023-10-30T12:32:13.312 INFO:tasks.workunit.client.0.smithi089.stdout: create 275/33 file after the snapshot 2023-10-30T12:32:13.314 INFO:tasks.workunit.client.0.smithi089.stdout: create 275/34 file after the snapshot 2023-10-30T12:32:13.316 INFO:tasks.workunit.client.0.smithi089.stdout: create 275/35 file after the snapshot 2023-10-30T12:32:13.318 INFO:tasks.workunit.client.0.smithi089.stdout: create 275/36 file after the snapshot 2023-10-30T12:32:13.320 INFO:tasks.workunit.client.0.smithi089.stdout: create 275/37 file after the snapshot 2023-10-30T12:32:13.322 INFO:tasks.workunit.client.0.smithi089.stdout: create 275/38 file after the snapshot 2023-10-30T12:32:13.323 INFO:tasks.workunit.client.0.smithi089.stdout: create 275/39 file after the snapshot 2023-10-30T12:32:13.325 INFO:tasks.workunit.client.0.smithi089.stdout: create 276/.snap/snap-subdir-test 2023-10-30T12:32:13.346 INFO:tasks.workunit.client.0.smithi089.stdout: create 276/30 file after the snapshot 2023-10-30T12:32:13.347 INFO:tasks.workunit.client.0.smithi089.stdout: create 276/31 file after the snapshot 2023-10-30T12:32:13.347 INFO:tasks.workunit.client.0.smithi089.stdout: create 276/32 file after the snapshot 2023-10-30T12:32:13.348 INFO:tasks.workunit.client.0.smithi089.stdout: create 276/33 file after the snapshot 2023-10-30T12:32:13.349 INFO:tasks.workunit.client.0.smithi089.stdout: create 276/34 file after the snapshot 2023-10-30T12:32:13.350 INFO:tasks.workunit.client.0.smithi089.stdout: create 276/35 file after the snapshot 2023-10-30T12:32:13.350 INFO:tasks.workunit.client.0.smithi089.stdout: create 276/36 file after the snapshot 2023-10-30T12:32:13.351 INFO:tasks.workunit.client.0.smithi089.stdout: create 276/37 file after the snapshot 2023-10-30T12:32:13.351 INFO:tasks.workunit.client.0.smithi089.stdout: create 276/38 file after the snapshot 2023-10-30T12:32:13.352 INFO:tasks.workunit.client.0.smithi089.stdout: create 276/39 file after the snapshot 2023-10-30T12:32:13.353 INFO:tasks.workunit.client.0.smithi089.stdout: create 277/.snap/snap-subdir-test 2023-10-30T12:32:13.368 INFO:tasks.workunit.client.0.smithi089.stdout: create 277/30 file after the snapshot 2023-10-30T12:32:13.368 INFO:tasks.workunit.client.0.smithi089.stdout: create 277/31 file after the snapshot 2023-10-30T12:32:13.369 INFO:tasks.workunit.client.0.smithi089.stdout: create 277/32 file after the snapshot 2023-10-30T12:32:13.370 INFO:tasks.workunit.client.0.smithi089.stdout: create 277/33 file after the snapshot 2023-10-30T12:32:13.370 INFO:tasks.workunit.client.0.smithi089.stdout: create 277/34 file after the snapshot 2023-10-30T12:32:13.371 INFO:tasks.workunit.client.0.smithi089.stdout: create 277/35 file after the snapshot 2023-10-30T12:32:13.371 INFO:tasks.workunit.client.0.smithi089.stdout: create 277/36 file after the snapshot 2023-10-30T12:32:13.372 INFO:tasks.workunit.client.0.smithi089.stdout: create 277/37 file after the snapshot 2023-10-30T12:32:13.373 INFO:tasks.workunit.client.0.smithi089.stdout: create 277/38 file after the snapshot 2023-10-30T12:32:13.373 INFO:tasks.workunit.client.0.smithi089.stdout: create 277/39 file after the snapshot 2023-10-30T12:32:13.374 INFO:tasks.workunit.client.0.smithi089.stdout: create 278/.snap/snap-subdir-test 2023-10-30T12:32:13.404 INFO:tasks.workunit.client.0.smithi089.stdout: create 278/30 file after the snapshot 2023-10-30T12:32:13.406 INFO:tasks.workunit.client.0.smithi089.stdout: create 278/31 file after the snapshot 2023-10-30T12:32:13.408 INFO:tasks.workunit.client.0.smithi089.stdout: create 278/32 file after the snapshot 2023-10-30T12:32:13.410 INFO:tasks.workunit.client.0.smithi089.stdout: create 278/33 file after the snapshot 2023-10-30T12:32:13.412 INFO:tasks.workunit.client.0.smithi089.stdout: create 278/34 file after the snapshot 2023-10-30T12:32:13.414 INFO:tasks.workunit.client.0.smithi089.stdout: create 278/35 file after the snapshot 2023-10-30T12:32:13.415 INFO:tasks.workunit.client.0.smithi089.stdout: create 278/36 file after the snapshot 2023-10-30T12:32:13.417 INFO:tasks.workunit.client.0.smithi089.stdout: create 278/37 file after the snapshot 2023-10-30T12:32:13.419 INFO:tasks.workunit.client.0.smithi089.stdout: create 278/38 file after the snapshot 2023-10-30T12:32:13.421 INFO:tasks.workunit.client.0.smithi089.stdout: create 278/39 file after the snapshot 2023-10-30T12:32:13.423 INFO:tasks.workunit.client.0.smithi089.stdout: create 279/.snap/snap-subdir-test 2023-10-30T12:32:13.441 INFO:tasks.workunit.client.0.smithi089.stdout: create 279/30 file after the snapshot 2023-10-30T12:32:13.444 INFO:tasks.workunit.client.0.smithi089.stdout: create 279/31 file after the snapshot 2023-10-30T12:32:13.446 INFO:tasks.workunit.client.0.smithi089.stdout: create 279/32 file after the snapshot 2023-10-30T12:32:13.447 INFO:tasks.workunit.client.0.smithi089.stdout: create 279/33 file after the snapshot 2023-10-30T12:32:13.449 INFO:tasks.workunit.client.0.smithi089.stdout: create 279/34 file after the snapshot 2023-10-30T12:32:13.451 INFO:tasks.workunit.client.0.smithi089.stdout: create 279/35 file after the snapshot 2023-10-30T12:32:13.453 INFO:tasks.workunit.client.0.smithi089.stdout: create 279/36 file after the snapshot 2023-10-30T12:32:13.455 INFO:tasks.workunit.client.0.smithi089.stdout: create 279/37 file after the snapshot 2023-10-30T12:32:13.457 INFO:tasks.workunit.client.0.smithi089.stdout: create 279/38 file after the snapshot 2023-10-30T12:32:13.459 INFO:tasks.workunit.client.0.smithi089.stdout: create 279/39 file after the snapshot 2023-10-30T12:32:13.461 INFO:tasks.workunit.client.0.smithi089.stdout: create 280/.snap/snap-subdir-test 2023-10-30T12:32:13.481 INFO:tasks.workunit.client.0.smithi089.stdout: create 280/30 file after the snapshot 2023-10-30T12:32:13.482 INFO:tasks.workunit.client.0.smithi089.stdout: create 280/31 file after the snapshot 2023-10-30T12:32:13.482 INFO:tasks.workunit.client.0.smithi089.stdout: create 280/32 file after the snapshot 2023-10-30T12:32:13.483 INFO:tasks.workunit.client.0.smithi089.stdout: create 280/33 file after the snapshot 2023-10-30T12:32:13.484 INFO:tasks.workunit.client.0.smithi089.stdout: create 280/34 file after the snapshot 2023-10-30T12:32:13.484 INFO:tasks.workunit.client.0.smithi089.stdout: create 280/35 file after the snapshot 2023-10-30T12:32:13.485 INFO:tasks.workunit.client.0.smithi089.stdout: create 280/36 file after the snapshot 2023-10-30T12:32:13.486 INFO:tasks.workunit.client.0.smithi089.stdout: create 280/37 file after the snapshot 2023-10-30T12:32:13.486 INFO:tasks.workunit.client.0.smithi089.stdout: create 280/38 file after the snapshot 2023-10-30T12:32:13.487 INFO:tasks.workunit.client.0.smithi089.stdout: create 280/39 file after the snapshot 2023-10-30T12:32:13.488 INFO:tasks.workunit.client.0.smithi089.stdout: create 281/.snap/snap-subdir-test 2023-10-30T12:32:13.507 INFO:tasks.workunit.client.0.smithi089.stdout: create 281/30 file after the snapshot 2023-10-30T12:32:13.509 INFO:tasks.workunit.client.0.smithi089.stdout: create 281/31 file after the snapshot 2023-10-30T12:32:13.511 INFO:tasks.workunit.client.0.smithi089.stdout: create 281/32 file after the snapshot 2023-10-30T12:32:13.513 INFO:tasks.workunit.client.0.smithi089.stdout: create 281/33 file after the snapshot 2023-10-30T12:32:13.514 INFO:tasks.workunit.client.0.smithi089.stdout: create 281/34 file after the snapshot 2023-10-30T12:32:13.516 INFO:tasks.workunit.client.0.smithi089.stdout: create 281/35 file after the snapshot 2023-10-30T12:32:13.518 INFO:tasks.workunit.client.0.smithi089.stdout: create 281/36 file after the snapshot 2023-10-30T12:32:13.520 INFO:tasks.workunit.client.0.smithi089.stdout: create 281/37 file after the snapshot 2023-10-30T12:32:13.522 INFO:tasks.workunit.client.0.smithi089.stdout: create 281/38 file after the snapshot 2023-10-30T12:32:13.524 INFO:tasks.workunit.client.0.smithi089.stdout: create 281/39 file after the snapshot 2023-10-30T12:32:13.526 INFO:tasks.workunit.client.0.smithi089.stdout: create 282/.snap/snap-subdir-test 2023-10-30T12:32:13.549 INFO:tasks.workunit.client.0.smithi089.stdout: create 282/30 file after the snapshot 2023-10-30T12:32:13.551 INFO:tasks.workunit.client.0.smithi089.stdout: create 282/31 file after the snapshot 2023-10-30T12:32:13.554 INFO:tasks.workunit.client.0.smithi089.stdout: create 282/32 file after the snapshot 2023-10-30T12:32:13.555 INFO:tasks.workunit.client.0.smithi089.stdout: create 282/33 file after the snapshot 2023-10-30T12:32:13.641 INFO:tasks.workunit.client.0.smithi089.stdout: create 282/34 file after the snapshot 2023-10-30T12:32:14.500 INFO:tasks.workunit.client.0.smithi089.stdout: create 282/35 file after the snapshot 2023-10-30T12:32:14.502 INFO:tasks.workunit.client.0.smithi089.stdout: create 282/36 file after the snapshot 2023-10-30T12:32:14.504 INFO:tasks.workunit.client.0.smithi089.stdout: create 282/37 file after the snapshot 2023-10-30T12:32:14.506 INFO:tasks.workunit.client.0.smithi089.stdout: create 282/38 file after the snapshot 2023-10-30T12:32:14.508 INFO:tasks.workunit.client.0.smithi089.stdout: create 282/39 file after the snapshot 2023-10-30T12:32:14.510 INFO:tasks.workunit.client.0.smithi089.stdout: create 283/.snap/snap-subdir-test 2023-10-30T12:32:14.550 INFO:tasks.workunit.client.0.smithi089.stdout: create 283/30 file after the snapshot 2023-10-30T12:32:14.554 INFO:tasks.workunit.client.0.smithi089.stdout: create 283/31 file after the snapshot 2023-10-30T12:32:14.557 INFO:tasks.workunit.client.0.smithi089.stdout: create 283/32 file after the snapshot 2023-10-30T12:32:14.560 INFO:tasks.workunit.client.0.smithi089.stdout: create 283/33 file after the snapshot 2023-10-30T12:32:14.563 INFO:tasks.workunit.client.0.smithi089.stdout: create 283/34 file after the snapshot 2023-10-30T12:32:14.565 INFO:tasks.workunit.client.0.smithi089.stdout: create 283/35 file after the snapshot 2023-10-30T12:32:14.568 INFO:tasks.workunit.client.0.smithi089.stdout: create 283/36 file after the snapshot 2023-10-30T12:32:14.571 INFO:tasks.workunit.client.0.smithi089.stdout: create 283/37 file after the snapshot 2023-10-30T12:32:14.574 INFO:tasks.workunit.client.0.smithi089.stdout: create 283/38 file after the snapshot 2023-10-30T12:32:14.577 INFO:tasks.workunit.client.0.smithi089.stdout: create 283/39 file after the snapshot 2023-10-30T12:32:14.580 INFO:tasks.workunit.client.0.smithi089.stdout: create 284/.snap/snap-subdir-test 2023-10-30T12:32:14.594 INFO:tasks.workunit.client.0.smithi089.stdout: create 284/30 file after the snapshot 2023-10-30T12:32:14.595 INFO:tasks.workunit.client.0.smithi089.stdout: create 284/31 file after the snapshot 2023-10-30T12:32:14.595 INFO:tasks.workunit.client.0.smithi089.stdout: create 284/32 file after the snapshot 2023-10-30T12:32:14.596 INFO:tasks.workunit.client.0.smithi089.stdout: create 284/33 file after the snapshot 2023-10-30T12:32:14.597 INFO:tasks.workunit.client.0.smithi089.stdout: create 284/34 file after the snapshot 2023-10-30T12:32:14.598 INFO:tasks.workunit.client.0.smithi089.stdout: create 284/35 file after the snapshot 2023-10-30T12:32:14.599 INFO:tasks.workunit.client.0.smithi089.stdout: create 284/36 file after the snapshot 2023-10-30T12:32:14.599 INFO:tasks.workunit.client.0.smithi089.stdout: create 284/37 file after the snapshot 2023-10-30T12:32:14.600 INFO:tasks.workunit.client.0.smithi089.stdout: create 284/38 file after the snapshot 2023-10-30T12:32:14.601 INFO:tasks.workunit.client.0.smithi089.stdout: create 284/39 file after the snapshot 2023-10-30T12:32:14.602 INFO:tasks.workunit.client.0.smithi089.stdout: create 285/.snap/snap-subdir-test 2023-10-30T12:32:14.625 INFO:tasks.workunit.client.0.smithi089.stdout: create 285/30 file after the snapshot 2023-10-30T12:32:14.628 INFO:tasks.workunit.client.0.smithi089.stdout: create 285/31 file after the snapshot 2023-10-30T12:32:14.666 INFO:tasks.workunit.client.0.smithi089.stdout: create 285/32 file after the snapshot 2023-10-30T12:32:15.309 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:32:15.313 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:32:15.411 INFO:tasks.workunit.client.0.smithi089.stdout: create 285/33 file after the snapshot 2023-10-30T12:32:15.413 INFO:tasks.workunit.client.0.smithi089.stdout: create 285/34 file after the snapshot 2023-10-30T12:32:15.415 INFO:tasks.workunit.client.0.smithi089.stdout: create 285/35 file after the snapshot 2023-10-30T12:32:15.417 INFO:tasks.workunit.client.0.smithi089.stdout: create 285/36 file after the snapshot 2023-10-30T12:32:15.419 INFO:tasks.workunit.client.0.smithi089.stdout: create 285/37 file after the snapshot 2023-10-30T12:32:15.421 INFO:tasks.workunit.client.0.smithi089.stdout: create 285/38 file after the snapshot 2023-10-30T12:32:15.422 INFO:tasks.workunit.client.0.smithi089.stdout: create 285/39 file after the snapshot 2023-10-30T12:32:15.424 INFO:tasks.workunit.client.0.smithi089.stdout: create 286/.snap/snap-subdir-test 2023-10-30T12:32:15.448 INFO:tasks.workunit.client.0.smithi089.stdout: create 286/30 file after the snapshot 2023-10-30T12:32:15.449 INFO:tasks.workunit.client.0.smithi089.stdout: create 286/31 file after the snapshot 2023-10-30T12:32:15.450 INFO:tasks.workunit.client.0.smithi089.stdout: create 286/32 file after the snapshot 2023-10-30T12:32:15.451 INFO:tasks.workunit.client.0.smithi089.stdout: create 286/33 file after the snapshot 2023-10-30T12:32:15.451 INFO:tasks.workunit.client.0.smithi089.stdout: create 286/34 file after the snapshot 2023-10-30T12:32:15.452 INFO:tasks.workunit.client.0.smithi089.stdout: create 286/35 file after the snapshot 2023-10-30T12:32:15.453 INFO:tasks.workunit.client.0.smithi089.stdout: create 286/36 file after the snapshot 2023-10-30T12:32:15.454 INFO:tasks.workunit.client.0.smithi089.stdout: create 286/37 file after the snapshot 2023-10-30T12:32:15.455 INFO:tasks.workunit.client.0.smithi089.stdout: create 286/38 file after the snapshot 2023-10-30T12:32:15.455 INFO:tasks.workunit.client.0.smithi089.stdout: create 286/39 file after the snapshot 2023-10-30T12:32:15.456 INFO:tasks.workunit.client.0.smithi089.stdout: create 287/.snap/snap-subdir-test 2023-10-30T12:32:15.477 INFO:tasks.workunit.client.0.smithi089.stdout: create 287/30 file after the snapshot 2023-10-30T12:32:15.480 INFO:tasks.workunit.client.0.smithi089.stdout: create 287/31 file after the snapshot 2023-10-30T12:32:15.482 INFO:tasks.workunit.client.0.smithi089.stdout: create 287/32 file after the snapshot 2023-10-30T12:32:15.484 INFO:tasks.workunit.client.0.smithi089.stdout: create 287/33 file after the snapshot 2023-10-30T12:32:15.486 INFO:tasks.workunit.client.0.smithi089.stdout: create 287/34 file after the snapshot 2023-10-30T12:32:15.487 INFO:tasks.workunit.client.0.smithi089.stdout: create 287/35 file after the snapshot 2023-10-30T12:32:15.490 INFO:tasks.workunit.client.0.smithi089.stdout: create 287/36 file after the snapshot 2023-10-30T12:32:15.492 INFO:tasks.workunit.client.0.smithi089.stdout: create 287/37 file after the snapshot 2023-10-30T12:32:15.494 INFO:tasks.workunit.client.0.smithi089.stdout: create 287/38 file after the snapshot 2023-10-30T12:32:15.496 INFO:tasks.workunit.client.0.smithi089.stdout: create 287/39 file after the snapshot 2023-10-30T12:32:15.498 INFO:tasks.workunit.client.0.smithi089.stdout: create 288/.snap/snap-subdir-test 2023-10-30T12:32:15.866 INFO:tasks.workunit.client.0.smithi089.stdout: create 288/30 file after the snapshot 2023-10-30T12:32:16.032 INFO:tasks.workunit.client.0.smithi089.stdout: create 288/31 file after the snapshot 2023-10-30T12:32:16.034 INFO:tasks.workunit.client.0.smithi089.stdout: create 288/32 file after the snapshot 2023-10-30T12:32:16.035 INFO:tasks.workunit.client.0.smithi089.stdout: create 288/33 file after the snapshot 2023-10-30T12:32:16.036 INFO:tasks.workunit.client.0.smithi089.stdout: create 288/34 file after the snapshot 2023-10-30T12:32:16.038 INFO:tasks.workunit.client.0.smithi089.stdout: create 288/35 file after the snapshot 2023-10-30T12:32:16.039 INFO:tasks.workunit.client.0.smithi089.stdout: create 288/36 file after the snapshot 2023-10-30T12:32:16.041 INFO:tasks.workunit.client.0.smithi089.stdout: create 288/37 file after the snapshot 2023-10-30T12:32:16.042 INFO:tasks.workunit.client.0.smithi089.stdout: create 288/38 file after the snapshot 2023-10-30T12:32:16.043 INFO:tasks.workunit.client.0.smithi089.stdout: create 288/39 file after the snapshot 2023-10-30T12:32:16.044 INFO:tasks.workunit.client.0.smithi089.stdout: create 289/.snap/snap-subdir-test 2023-10-30T12:32:16.180 INFO:tasks.workunit.client.0.smithi089.stdout: create 289/30 file after the snapshot 2023-10-30T12:32:16.183 INFO:tasks.workunit.client.0.smithi089.stdout: create 289/31 file after the snapshot 2023-10-30T12:32:16.185 INFO:tasks.workunit.client.0.smithi089.stdout: create 289/32 file after the snapshot 2023-10-30T12:32:16.438 INFO:tasks.workunit.client.0.smithi089.stdout: create 289/33 file after the snapshot 2023-10-30T12:32:16.439 INFO:tasks.workunit.client.0.smithi089.stdout: create 289/34 file after the snapshot 2023-10-30T12:32:16.690 INFO:tasks.workunit.client.0.smithi089.stdout: create 289/35 file after the snapshot 2023-10-30T12:32:16.692 INFO:tasks.workunit.client.0.smithi089.stdout: create 289/36 file after the snapshot 2023-10-30T12:32:16.693 INFO:tasks.workunit.client.0.smithi089.stdout: create 289/37 file after the snapshot 2023-10-30T12:32:16.695 INFO:tasks.workunit.client.0.smithi089.stdout: create 289/38 file after the snapshot 2023-10-30T12:32:16.697 INFO:tasks.workunit.client.0.smithi089.stdout: create 289/39 file after the snapshot 2023-10-30T12:32:16.699 INFO:tasks.workunit.client.0.smithi089.stdout: create 290/.snap/snap-subdir-test 2023-10-30T12:32:16.724 INFO:tasks.workunit.client.0.smithi089.stdout: create 290/30 file after the snapshot 2023-10-30T12:32:16.726 INFO:tasks.workunit.client.0.smithi089.stdout: create 290/31 file after the snapshot 2023-10-30T12:32:16.728 INFO:tasks.workunit.client.0.smithi089.stdout: create 290/32 file after the snapshot 2023-10-30T12:32:16.730 INFO:tasks.workunit.client.0.smithi089.stdout: create 290/33 file after the snapshot 2023-10-30T12:32:16.732 INFO:tasks.workunit.client.0.smithi089.stdout: create 290/34 file after the snapshot 2023-10-30T12:32:16.733 INFO:tasks.workunit.client.0.smithi089.stdout: create 290/35 file after the snapshot 2023-10-30T12:32:16.736 INFO:tasks.workunit.client.0.smithi089.stdout: create 290/36 file after the snapshot 2023-10-30T12:32:16.737 INFO:tasks.workunit.client.0.smithi089.stdout: create 290/37 file after the snapshot 2023-10-30T12:32:16.739 INFO:tasks.workunit.client.0.smithi089.stdout: create 290/38 file after the snapshot 2023-10-30T12:32:16.741 INFO:tasks.workunit.client.0.smithi089.stdout: create 290/39 file after the snapshot 2023-10-30T12:32:16.743 INFO:tasks.workunit.client.0.smithi089.stdout: create 291/.snap/snap-subdir-test 2023-10-30T12:32:16.770 INFO:tasks.workunit.client.0.smithi089.stdout: create 291/30 file after the snapshot 2023-10-30T12:32:16.772 INFO:tasks.workunit.client.0.smithi089.stdout: create 291/31 file after the snapshot 2023-10-30T12:32:16.774 INFO:tasks.workunit.client.0.smithi089.stdout: create 291/32 file after the snapshot 2023-10-30T12:32:16.776 INFO:tasks.workunit.client.0.smithi089.stdout: create 291/33 file after the snapshot 2023-10-30T12:32:17.030 INFO:tasks.workunit.client.0.smithi089.stdout: create 291/34 file after the snapshot 2023-10-30T12:32:17.031 INFO:tasks.workunit.client.0.smithi089.stdout: create 291/35 file after the snapshot 2023-10-30T12:32:17.032 INFO:tasks.workunit.client.0.smithi089.stdout: create 291/36 file after the snapshot 2023-10-30T12:32:17.034 INFO:tasks.workunit.client.0.smithi089.stdout: create 291/37 file after the snapshot 2023-10-30T12:32:17.036 INFO:tasks.workunit.client.0.smithi089.stdout: create 291/38 file after the snapshot 2023-10-30T12:32:17.038 INFO:tasks.workunit.client.0.smithi089.stdout: create 291/39 file after the snapshot 2023-10-30T12:32:17.040 INFO:tasks.workunit.client.0.smithi089.stdout: create 292/.snap/snap-subdir-test 2023-10-30T12:32:17.067 INFO:tasks.workunit.client.0.smithi089.stdout: create 292/30 file after the snapshot 2023-10-30T12:32:17.069 INFO:tasks.workunit.client.0.smithi089.stdout: create 292/31 file after the snapshot 2023-10-30T12:32:17.071 INFO:tasks.workunit.client.0.smithi089.stdout: create 292/32 file after the snapshot 2023-10-30T12:32:17.073 INFO:tasks.workunit.client.0.smithi089.stdout: create 292/33 file after the snapshot 2023-10-30T12:32:17.075 INFO:tasks.workunit.client.0.smithi089.stdout: create 292/34 file after the snapshot 2023-10-30T12:32:17.077 INFO:tasks.workunit.client.0.smithi089.stdout: create 292/35 file after the snapshot 2023-10-30T12:32:17.079 INFO:tasks.workunit.client.0.smithi089.stdout: create 292/36 file after the snapshot 2023-10-30T12:32:17.081 INFO:tasks.workunit.client.0.smithi089.stdout: create 292/37 file after the snapshot 2023-10-30T12:32:17.083 INFO:tasks.workunit.client.0.smithi089.stdout: create 292/38 file after the snapshot 2023-10-30T12:32:17.085 INFO:tasks.workunit.client.0.smithi089.stdout: create 292/39 file after the snapshot 2023-10-30T12:32:17.087 INFO:tasks.workunit.client.0.smithi089.stdout: create 293/.snap/snap-subdir-test 2023-10-30T12:32:17.891 INFO:tasks.workunit.client.0.smithi089.stdout: create 293/30 file after the snapshot 2023-10-30T12:32:17.894 INFO:tasks.workunit.client.0.smithi089.stdout: create 293/31 file after the snapshot 2023-10-30T12:32:17.896 INFO:tasks.workunit.client.0.smithi089.stdout: create 293/32 file after the snapshot 2023-10-30T12:32:17.898 INFO:tasks.workunit.client.0.smithi089.stdout: create 293/33 file after the snapshot 2023-10-30T12:32:17.900 INFO:tasks.workunit.client.0.smithi089.stdout: create 293/34 file after the snapshot 2023-10-30T12:32:17.902 INFO:tasks.workunit.client.0.smithi089.stdout: create 293/35 file after the snapshot 2023-10-30T12:32:17.904 INFO:tasks.workunit.client.0.smithi089.stdout: create 293/36 file after the snapshot 2023-10-30T12:32:17.906 INFO:tasks.workunit.client.0.smithi089.stdout: create 293/37 file after the snapshot 2023-10-30T12:32:17.908 INFO:tasks.workunit.client.0.smithi089.stdout: create 293/38 file after the snapshot 2023-10-30T12:32:17.910 INFO:tasks.workunit.client.0.smithi089.stdout: create 293/39 file after the snapshot 2023-10-30T12:32:17.912 INFO:tasks.workunit.client.0.smithi089.stdout: create 294/.snap/snap-subdir-test 2023-10-30T12:32:17.929 INFO:tasks.workunit.client.0.smithi089.stdout: create 294/30 file after the snapshot 2023-10-30T12:32:17.931 INFO:tasks.workunit.client.0.smithi089.stdout: create 294/31 file after the snapshot 2023-10-30T12:32:17.932 INFO:tasks.workunit.client.0.smithi089.stdout: create 294/32 file after the snapshot 2023-10-30T12:32:17.933 INFO:tasks.workunit.client.0.smithi089.stdout: create 294/33 file after the snapshot 2023-10-30T12:32:17.933 INFO:tasks.workunit.client.0.smithi089.stdout: create 294/34 file after the snapshot 2023-10-30T12:32:17.935 INFO:tasks.workunit.client.0.smithi089.stdout: create 294/35 file after the snapshot 2023-10-30T12:32:17.935 INFO:tasks.workunit.client.0.smithi089.stdout: create 294/36 file after the snapshot 2023-10-30T12:32:17.936 INFO:tasks.workunit.client.0.smithi089.stdout: create 294/37 file after the snapshot 2023-10-30T12:32:17.937 INFO:tasks.workunit.client.0.smithi089.stdout: create 294/38 file after the snapshot 2023-10-30T12:32:17.937 INFO:tasks.workunit.client.0.smithi089.stdout: create 294/39 file after the snapshot 2023-10-30T12:32:17.938 INFO:tasks.workunit.client.0.smithi089.stdout: create 295/.snap/snap-subdir-test 2023-10-30T12:32:17.955 INFO:tasks.workunit.client.0.smithi089.stdout: create 295/30 file after the snapshot 2023-10-30T12:32:17.957 INFO:tasks.workunit.client.0.smithi089.stdout: create 295/31 file after the snapshot 2023-10-30T12:32:17.957 INFO:tasks.workunit.client.0.smithi089.stdout: create 295/32 file after the snapshot 2023-10-30T12:32:17.958 INFO:tasks.workunit.client.0.smithi089.stdout: create 295/33 file after the snapshot 2023-10-30T12:32:17.959 INFO:tasks.workunit.client.0.smithi089.stdout: create 295/34 file after the snapshot 2023-10-30T12:32:17.960 INFO:tasks.workunit.client.0.smithi089.stdout: create 295/35 file after the snapshot 2023-10-30T12:32:17.960 INFO:tasks.workunit.client.0.smithi089.stdout: create 295/36 file after the snapshot 2023-10-30T12:32:17.961 INFO:tasks.workunit.client.0.smithi089.stdout: create 295/37 file after the snapshot 2023-10-30T12:32:17.962 INFO:tasks.workunit.client.0.smithi089.stdout: create 295/38 file after the snapshot 2023-10-30T12:32:17.963 INFO:tasks.workunit.client.0.smithi089.stdout: create 295/39 file after the snapshot 2023-10-30T12:32:17.964 INFO:tasks.workunit.client.0.smithi089.stdout: create 296/.snap/snap-subdir-test 2023-10-30T12:32:17.987 INFO:tasks.workunit.client.0.smithi089.stdout: create 296/30 file after the snapshot 2023-10-30T12:32:17.990 INFO:tasks.workunit.client.0.smithi089.stdout: create 296/31 file after the snapshot 2023-10-30T12:32:17.992 INFO:tasks.workunit.client.0.smithi089.stdout: create 296/32 file after the snapshot 2023-10-30T12:32:17.993 INFO:tasks.workunit.client.0.smithi089.stdout: create 296/33 file after the snapshot 2023-10-30T12:32:17.995 INFO:tasks.workunit.client.0.smithi089.stdout: create 296/34 file after the snapshot 2023-10-30T12:32:17.997 INFO:tasks.workunit.client.0.smithi089.stdout: create 296/35 file after the snapshot 2023-10-30T12:32:17.999 INFO:tasks.workunit.client.0.smithi089.stdout: create 296/36 file after the snapshot 2023-10-30T12:32:18.001 INFO:tasks.workunit.client.0.smithi089.stdout: create 296/37 file after the snapshot 2023-10-30T12:32:18.002 INFO:tasks.workunit.client.0.smithi089.stdout: create 296/38 file after the snapshot 2023-10-30T12:32:18.004 INFO:tasks.workunit.client.0.smithi089.stdout: create 296/39 file after the snapshot 2023-10-30T12:32:18.006 INFO:tasks.workunit.client.0.smithi089.stdout: create 297/.snap/snap-subdir-test 2023-10-30T12:32:18.037 INFO:tasks.workunit.client.0.smithi089.stdout: create 297/30 file after the snapshot 2023-10-30T12:32:18.039 INFO:tasks.workunit.client.0.smithi089.stdout: create 297/31 file after the snapshot 2023-10-30T12:32:18.041 INFO:tasks.workunit.client.0.smithi089.stdout: create 297/32 file after the snapshot 2023-10-30T12:32:18.045 INFO:tasks.workunit.client.0.smithi089.stdout: create 297/33 file after the snapshot 2023-10-30T12:32:18.046 INFO:tasks.workunit.client.0.smithi089.stdout: create 297/34 file after the snapshot 2023-10-30T12:32:18.046 INFO:tasks.workunit.client.0.smithi089.stdout: create 297/35 file after the snapshot 2023-10-30T12:32:18.048 INFO:tasks.workunit.client.0.smithi089.stdout: create 297/36 file after the snapshot 2023-10-30T12:32:18.050 INFO:tasks.workunit.client.0.smithi089.stdout: create 297/37 file after the snapshot 2023-10-30T12:32:18.052 INFO:tasks.workunit.client.0.smithi089.stdout: create 297/38 file after the snapshot 2023-10-30T12:32:18.054 INFO:tasks.workunit.client.0.smithi089.stdout: create 297/39 file after the snapshot 2023-10-30T12:32:18.056 INFO:tasks.workunit.client.0.smithi089.stdout: create 298/.snap/snap-subdir-test 2023-10-30T12:32:18.082 INFO:tasks.workunit.client.0.smithi089.stdout: create 298/30 file after the snapshot 2023-10-30T12:32:18.084 INFO:tasks.workunit.client.0.smithi089.stdout: create 298/31 file after the snapshot 2023-10-30T12:32:18.086 INFO:tasks.workunit.client.0.smithi089.stdout: create 298/32 file after the snapshot 2023-10-30T12:32:18.088 INFO:tasks.workunit.client.0.smithi089.stdout: create 298/33 file after the snapshot 2023-10-30T12:32:18.090 INFO:tasks.workunit.client.0.smithi089.stdout: create 298/34 file after the snapshot 2023-10-30T12:32:18.092 INFO:tasks.workunit.client.0.smithi089.stdout: create 298/35 file after the snapshot 2023-10-30T12:32:18.094 INFO:tasks.workunit.client.0.smithi089.stdout: create 298/36 file after the snapshot 2023-10-30T12:32:18.096 INFO:tasks.workunit.client.0.smithi089.stdout: create 298/37 file after the snapshot 2023-10-30T12:32:18.098 INFO:tasks.workunit.client.0.smithi089.stdout: create 298/38 file after the snapshot 2023-10-30T12:32:18.099 INFO:tasks.workunit.client.0.smithi089.stdout: create 298/39 file after the snapshot 2023-10-30T12:32:18.101 INFO:tasks.workunit.client.0.smithi089.stdout: create 299/.snap/snap-subdir-test 2023-10-30T12:32:18.124 INFO:tasks.workunit.client.0.smithi089.stdout: create 299/30 file after the snapshot 2023-10-30T12:32:18.127 INFO:tasks.workunit.client.0.smithi089.stdout: create 299/31 file after the snapshot 2023-10-30T12:32:18.128 INFO:tasks.workunit.client.0.smithi089.stdout: create 299/32 file after the snapshot 2023-10-30T12:32:18.130 INFO:tasks.workunit.client.0.smithi089.stdout: create 299/33 file after the snapshot 2023-10-30T12:32:18.132 INFO:tasks.workunit.client.0.smithi089.stdout: create 299/34 file after the snapshot 2023-10-30T12:32:18.134 INFO:tasks.workunit.client.0.smithi089.stdout: create 299/35 file after the snapshot 2023-10-30T12:32:18.135 INFO:tasks.workunit.client.0.smithi089.stdout: create 299/36 file after the snapshot 2023-10-30T12:32:18.137 INFO:tasks.workunit.client.0.smithi089.stdout: create 299/37 file after the snapshot 2023-10-30T12:32:18.139 INFO:tasks.workunit.client.0.smithi089.stdout: create 299/38 file after the snapshot 2023-10-30T12:32:18.141 INFO:tasks.workunit.client.0.smithi089.stdout: create 299/39 file after the snapshot 2023-10-30T12:32:18.143 INFO:tasks.workunit.client.0.smithi089.stdout:Take second snapshot .snap/test2 2023-10-30T12:32:18.396 INFO:tasks.workunit.client.0.smithi089.stdout:Copy content of .snap/test1 to copyofsnap1 ... 2023-10-30T12:32:19.383 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1' -> 'copyofsnap1/test1' 2023-10-30T12:32:19.383 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/100' -> 'copyofsnap1/test1/100' 2023-10-30T12:32:19.383 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/100/11' -> 'copyofsnap1/test1/100/11' 2023-10-30T12:32:19.383 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/100/12' -> 'copyofsnap1/test1/100/12' 2023-10-30T12:32:19.384 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/100/13' -> 'copyofsnap1/test1/100/13' 2023-10-30T12:32:19.384 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/100/14' -> 'copyofsnap1/test1/100/14' 2023-10-30T12:32:19.384 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/100/15' -> 'copyofsnap1/test1/100/15' 2023-10-30T12:32:19.384 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/100/16' -> 'copyofsnap1/test1/100/16' 2023-10-30T12:32:19.384 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/100/17' -> 'copyofsnap1/test1/100/17' 2023-10-30T12:32:19.384 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/100/18' -> 'copyofsnap1/test1/100/18' 2023-10-30T12:32:19.385 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/100/19' -> 'copyofsnap1/test1/100/19' 2023-10-30T12:32:19.385 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/100/20' -> 'copyofsnap1/test1/100/20' 2023-10-30T12:32:19.385 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/100/10' -> 'copyofsnap1/test1/100/10' 2023-10-30T12:32:19.385 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/101' -> 'copyofsnap1/test1/101' 2023-10-30T12:32:19.385 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/101/11' -> 'copyofsnap1/test1/101/11' 2023-10-30T12:32:19.386 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/101/12' -> 'copyofsnap1/test1/101/12' 2023-10-30T12:32:19.386 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/101/13' -> 'copyofsnap1/test1/101/13' 2023-10-30T12:32:19.386 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/101/14' -> 'copyofsnap1/test1/101/14' 2023-10-30T12:32:19.386 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/101/15' -> 'copyofsnap1/test1/101/15' 2023-10-30T12:32:19.386 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/101/16' -> 'copyofsnap1/test1/101/16' 2023-10-30T12:32:19.387 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/101/17' -> 'copyofsnap1/test1/101/17' 2023-10-30T12:32:19.387 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/101/18' -> 'copyofsnap1/test1/101/18' 2023-10-30T12:32:19.387 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/101/19' -> 'copyofsnap1/test1/101/19' 2023-10-30T12:32:19.387 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/101/20' -> 'copyofsnap1/test1/101/20' 2023-10-30T12:32:19.387 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/101/10' -> 'copyofsnap1/test1/101/10' 2023-10-30T12:32:19.388 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/102' -> 'copyofsnap1/test1/102' 2023-10-30T12:32:19.388 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/102/11' -> 'copyofsnap1/test1/102/11' 2023-10-30T12:32:19.388 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/102/12' -> 'copyofsnap1/test1/102/12' 2023-10-30T12:32:19.388 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/102/13' -> 'copyofsnap1/test1/102/13' 2023-10-30T12:32:19.388 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/102/14' -> 'copyofsnap1/test1/102/14' 2023-10-30T12:32:19.388 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/102/15' -> 'copyofsnap1/test1/102/15' 2023-10-30T12:32:19.389 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/102/16' -> 'copyofsnap1/test1/102/16' 2023-10-30T12:32:19.389 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/102/17' -> 'copyofsnap1/test1/102/17' 2023-10-30T12:32:19.389 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/102/18' -> 'copyofsnap1/test1/102/18' 2023-10-30T12:32:19.389 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/102/19' -> 'copyofsnap1/test1/102/19' 2023-10-30T12:32:19.389 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/102/20' -> 'copyofsnap1/test1/102/20' 2023-10-30T12:32:19.389 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/102/10' -> 'copyofsnap1/test1/102/10' 2023-10-30T12:32:19.390 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/103' -> 'copyofsnap1/test1/103' 2023-10-30T12:32:19.390 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/103/11' -> 'copyofsnap1/test1/103/11' 2023-10-30T12:32:19.390 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/103/12' -> 'copyofsnap1/test1/103/12' 2023-10-30T12:32:19.390 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/103/13' -> 'copyofsnap1/test1/103/13' 2023-10-30T12:32:19.390 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/103/14' -> 'copyofsnap1/test1/103/14' 2023-10-30T12:32:19.391 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/103/15' -> 'copyofsnap1/test1/103/15' 2023-10-30T12:32:19.391 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/103/16' -> 'copyofsnap1/test1/103/16' 2023-10-30T12:32:19.391 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/103/17' -> 'copyofsnap1/test1/103/17' 2023-10-30T12:32:19.391 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/103/18' -> 'copyofsnap1/test1/103/18' 2023-10-30T12:32:19.391 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/103/19' -> 'copyofsnap1/test1/103/19' 2023-10-30T12:32:19.391 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/103/20' -> 'copyofsnap1/test1/103/20' 2023-10-30T12:32:19.392 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/103/10' -> 'copyofsnap1/test1/103/10' 2023-10-30T12:32:19.392 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/104' -> 'copyofsnap1/test1/104' 2023-10-30T12:32:19.392 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/104/11' -> 'copyofsnap1/test1/104/11' 2023-10-30T12:32:19.392 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/104/12' -> 'copyofsnap1/test1/104/12' 2023-10-30T12:32:19.392 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/104/13' -> 'copyofsnap1/test1/104/13' 2023-10-30T12:32:19.392 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/104/14' -> 'copyofsnap1/test1/104/14' 2023-10-30T12:32:19.393 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/104/15' -> 'copyofsnap1/test1/104/15' 2023-10-30T12:32:19.393 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/104/16' -> 'copyofsnap1/test1/104/16' 2023-10-30T12:32:19.393 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/104/17' -> 'copyofsnap1/test1/104/17' 2023-10-30T12:32:19.393 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/104/18' -> 'copyofsnap1/test1/104/18' 2023-10-30T12:32:19.393 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/104/19' -> 'copyofsnap1/test1/104/19' 2023-10-30T12:32:19.393 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/104/20' -> 'copyofsnap1/test1/104/20' 2023-10-30T12:32:19.394 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/104/10' -> 'copyofsnap1/test1/104/10' 2023-10-30T12:32:19.394 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/105' -> 'copyofsnap1/test1/105' 2023-10-30T12:32:19.394 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/105/11' -> 'copyofsnap1/test1/105/11' 2023-10-30T12:32:19.394 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/105/12' -> 'copyofsnap1/test1/105/12' 2023-10-30T12:32:19.395 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/105/13' -> 'copyofsnap1/test1/105/13' 2023-10-30T12:32:19.395 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/105/14' -> 'copyofsnap1/test1/105/14' 2023-10-30T12:32:19.395 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/105/15' -> 'copyofsnap1/test1/105/15' 2023-10-30T12:32:19.395 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/105/16' -> 'copyofsnap1/test1/105/16' 2023-10-30T12:32:19.395 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/105/17' -> 'copyofsnap1/test1/105/17' 2023-10-30T12:32:19.396 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/105/18' -> 'copyofsnap1/test1/105/18' 2023-10-30T12:32:19.396 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/105/19' -> 'copyofsnap1/test1/105/19' 2023-10-30T12:32:19.396 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/105/20' -> 'copyofsnap1/test1/105/20' 2023-10-30T12:32:19.396 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/105/10' -> 'copyofsnap1/test1/105/10' 2023-10-30T12:32:19.396 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/106' -> 'copyofsnap1/test1/106' 2023-10-30T12:32:19.396 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/106/11' -> 'copyofsnap1/test1/106/11' 2023-10-30T12:32:19.397 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/106/12' -> 'copyofsnap1/test1/106/12' 2023-10-30T12:32:19.397 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/106/13' -> 'copyofsnap1/test1/106/13' 2023-10-30T12:32:19.397 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/106/14' -> 'copyofsnap1/test1/106/14' 2023-10-30T12:32:19.397 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/106/15' -> 'copyofsnap1/test1/106/15' 2023-10-30T12:32:19.397 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/106/16' -> 'copyofsnap1/test1/106/16' 2023-10-30T12:32:19.398 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/106/17' -> 'copyofsnap1/test1/106/17' 2023-10-30T12:32:21.098 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/106/18' -> 'copyofsnap1/test1/106/18' 2023-10-30T12:32:21.099 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/106/19' -> 'copyofsnap1/test1/106/19' 2023-10-30T12:32:21.099 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/106/20' -> 'copyofsnap1/test1/106/20' 2023-10-30T12:32:21.099 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/106/10' -> 'copyofsnap1/test1/106/10' 2023-10-30T12:32:21.099 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/107' -> 'copyofsnap1/test1/107' 2023-10-30T12:32:21.100 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/107/11' -> 'copyofsnap1/test1/107/11' 2023-10-30T12:32:21.100 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/107/12' -> 'copyofsnap1/test1/107/12' 2023-10-30T12:32:21.100 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/107/13' -> 'copyofsnap1/test1/107/13' 2023-10-30T12:32:21.100 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/107/14' -> 'copyofsnap1/test1/107/14' 2023-10-30T12:32:21.100 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/107/15' -> 'copyofsnap1/test1/107/15' 2023-10-30T12:32:21.101 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/107/16' -> 'copyofsnap1/test1/107/16' 2023-10-30T12:32:21.101 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/107/17' -> 'copyofsnap1/test1/107/17' 2023-10-30T12:32:21.101 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/107/18' -> 'copyofsnap1/test1/107/18' 2023-10-30T12:32:21.101 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/107/19' -> 'copyofsnap1/test1/107/19' 2023-10-30T12:32:21.101 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/107/20' -> 'copyofsnap1/test1/107/20' 2023-10-30T12:32:21.102 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/107/10' -> 'copyofsnap1/test1/107/10' 2023-10-30T12:32:21.102 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/108' -> 'copyofsnap1/test1/108' 2023-10-30T12:32:21.102 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/108/11' -> 'copyofsnap1/test1/108/11' 2023-10-30T12:32:21.102 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/108/12' -> 'copyofsnap1/test1/108/12' 2023-10-30T12:32:21.102 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/108/13' -> 'copyofsnap1/test1/108/13' 2023-10-30T12:32:21.103 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/108/14' -> 'copyofsnap1/test1/108/14' 2023-10-30T12:32:21.103 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/108/15' -> 'copyofsnap1/test1/108/15' 2023-10-30T12:32:21.103 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/108/16' -> 'copyofsnap1/test1/108/16' 2023-10-30T12:32:21.103 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/108/17' -> 'copyofsnap1/test1/108/17' 2023-10-30T12:32:21.103 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/108/18' -> 'copyofsnap1/test1/108/18' 2023-10-30T12:32:21.103 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/108/19' -> 'copyofsnap1/test1/108/19' 2023-10-30T12:32:21.104 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/108/20' -> 'copyofsnap1/test1/108/20' 2023-10-30T12:32:21.104 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/108/10' -> 'copyofsnap1/test1/108/10' 2023-10-30T12:32:21.104 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/109' -> 'copyofsnap1/test1/109' 2023-10-30T12:32:21.104 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/109/11' -> 'copyofsnap1/test1/109/11' 2023-10-30T12:32:21.104 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/109/12' -> 'copyofsnap1/test1/109/12' 2023-10-30T12:32:21.105 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/109/13' -> 'copyofsnap1/test1/109/13' 2023-10-30T12:32:21.105 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/109/14' -> 'copyofsnap1/test1/109/14' 2023-10-30T12:32:21.105 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/109/15' -> 'copyofsnap1/test1/109/15' 2023-10-30T12:32:21.105 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/109/16' -> 'copyofsnap1/test1/109/16' 2023-10-30T12:32:21.105 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/109/17' -> 'copyofsnap1/test1/109/17' 2023-10-30T12:32:21.106 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/109/18' -> 'copyofsnap1/test1/109/18' 2023-10-30T12:32:21.106 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/109/19' -> 'copyofsnap1/test1/109/19' 2023-10-30T12:32:21.106 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/109/20' -> 'copyofsnap1/test1/109/20' 2023-10-30T12:32:21.106 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/109/10' -> 'copyofsnap1/test1/109/10' 2023-10-30T12:32:21.106 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/110' -> 'copyofsnap1/test1/110' 2023-10-30T12:32:21.107 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/110/11' -> 'copyofsnap1/test1/110/11' 2023-10-30T12:32:21.107 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/110/12' -> 'copyofsnap1/test1/110/12' 2023-10-30T12:32:21.107 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/110/13' -> 'copyofsnap1/test1/110/13' 2023-10-30T12:32:21.107 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/110/14' -> 'copyofsnap1/test1/110/14' 2023-10-30T12:32:21.107 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/110/15' -> 'copyofsnap1/test1/110/15' 2023-10-30T12:32:21.107 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/110/16' -> 'copyofsnap1/test1/110/16' 2023-10-30T12:32:21.108 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/110/17' -> 'copyofsnap1/test1/110/17' 2023-10-30T12:32:21.108 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/110/18' -> 'copyofsnap1/test1/110/18' 2023-10-30T12:32:21.108 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/110/19' -> 'copyofsnap1/test1/110/19' 2023-10-30T12:32:21.108 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/110/20' -> 'copyofsnap1/test1/110/20' 2023-10-30T12:32:21.108 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/110/10' -> 'copyofsnap1/test1/110/10' 2023-10-30T12:32:21.109 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/111' -> 'copyofsnap1/test1/111' 2023-10-30T12:32:21.109 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/111/11' -> 'copyofsnap1/test1/111/11' 2023-10-30T12:32:21.109 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/111/12' -> 'copyofsnap1/test1/111/12' 2023-10-30T12:32:21.109 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/111/13' -> 'copyofsnap1/test1/111/13' 2023-10-30T12:32:21.109 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/111/14' -> 'copyofsnap1/test1/111/14' 2023-10-30T12:32:21.110 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/111/15' -> 'copyofsnap1/test1/111/15' 2023-10-30T12:32:21.110 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/111/16' -> 'copyofsnap1/test1/111/16' 2023-10-30T12:32:21.110 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/111/17' -> 'copyofsnap1/test1/111/17' 2023-10-30T12:32:21.110 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/111/18' -> 'copyofsnap1/test1/111/18' 2023-10-30T12:32:21.110 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/111/19' -> 'copyofsnap1/test1/111/19' 2023-10-30T12:32:21.111 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/111/20' -> 'copyofsnap1/test1/111/20' 2023-10-30T12:32:21.111 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/111/10' -> 'copyofsnap1/test1/111/10' 2023-10-30T12:32:21.111 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/112' -> 'copyofsnap1/test1/112' 2023-10-30T12:32:21.111 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/112/11' -> 'copyofsnap1/test1/112/11' 2023-10-30T12:32:21.111 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/112/12' -> 'copyofsnap1/test1/112/12' 2023-10-30T12:32:21.112 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/112/13' -> 'copyofsnap1/test1/112/13' 2023-10-30T12:32:21.112 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/112/14' -> 'copyofsnap1/test1/112/14' 2023-10-30T12:32:21.112 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/112/15' -> 'copyofsnap1/test1/112/15' 2023-10-30T12:32:21.112 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/112/16' -> 'copyofsnap1/test1/112/16' 2023-10-30T12:32:21.112 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/112/17' -> 'copyofsnap1/test1/112/17' 2023-10-30T12:32:21.112 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/112/18' -> 'copyofsnap1/test1/112/18' 2023-10-30T12:32:21.113 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/112/19' -> 'copyofsnap1/test1/112/19' 2023-10-30T12:32:21.113 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/112/20' -> 'copyofsnap1/test1/112/20' 2023-10-30T12:32:21.113 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/112/10' -> 'copyofsnap1/test1/112/10' 2023-10-30T12:32:21.113 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/113' -> 'copyofsnap1/test1/113' 2023-10-30T12:32:21.113 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/113/11' -> 'copyofsnap1/test1/113/11' 2023-10-30T12:32:21.114 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/113/12' -> 'copyofsnap1/test1/113/12' 2023-10-30T12:32:21.114 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/113/13' -> 'copyofsnap1/test1/113/13' 2023-10-30T12:32:21.114 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/113/14' -> 'copyofsnap1/test1/113/14' 2023-10-30T12:32:21.996 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/113/15' -> 'copyofsnap1/test1/113/15' 2023-10-30T12:32:21.996 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/113/16' -> 'copyofsnap1/test1/113/16' 2023-10-30T12:32:21.996 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/113/17' -> 'copyofsnap1/test1/113/17' 2023-10-30T12:32:21.997 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/113/18' -> 'copyofsnap1/test1/113/18' 2023-10-30T12:32:21.997 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/113/19' -> 'copyofsnap1/test1/113/19' 2023-10-30T12:32:21.997 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/113/20' -> 'copyofsnap1/test1/113/20' 2023-10-30T12:32:21.997 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/113/10' -> 'copyofsnap1/test1/113/10' 2023-10-30T12:32:21.997 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/114' -> 'copyofsnap1/test1/114' 2023-10-30T12:32:21.998 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/114/11' -> 'copyofsnap1/test1/114/11' 2023-10-30T12:32:21.998 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/114/12' -> 'copyofsnap1/test1/114/12' 2023-10-30T12:32:21.998 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/114/13' -> 'copyofsnap1/test1/114/13' 2023-10-30T12:32:21.998 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/114/14' -> 'copyofsnap1/test1/114/14' 2023-10-30T12:32:21.998 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/114/15' -> 'copyofsnap1/test1/114/15' 2023-10-30T12:32:21.999 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/114/16' -> 'copyofsnap1/test1/114/16' 2023-10-30T12:32:21.999 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/114/17' -> 'copyofsnap1/test1/114/17' 2023-10-30T12:32:21.999 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/114/18' -> 'copyofsnap1/test1/114/18' 2023-10-30T12:32:21.999 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/114/19' -> 'copyofsnap1/test1/114/19' 2023-10-30T12:32:21.999 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/114/20' -> 'copyofsnap1/test1/114/20' 2023-10-30T12:32:21.999 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/114/10' -> 'copyofsnap1/test1/114/10' 2023-10-30T12:32:22.000 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/115' -> 'copyofsnap1/test1/115' 2023-10-30T12:32:22.000 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/115/11' -> 'copyofsnap1/test1/115/11' 2023-10-30T12:32:22.000 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/115/12' -> 'copyofsnap1/test1/115/12' 2023-10-30T12:32:22.000 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/115/13' -> 'copyofsnap1/test1/115/13' 2023-10-30T12:32:22.000 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/115/14' -> 'copyofsnap1/test1/115/14' 2023-10-30T12:32:22.001 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/115/15' -> 'copyofsnap1/test1/115/15' 2023-10-30T12:32:22.001 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/115/16' -> 'copyofsnap1/test1/115/16' 2023-10-30T12:32:22.001 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/115/17' -> 'copyofsnap1/test1/115/17' 2023-10-30T12:32:22.002 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/115/18' -> 'copyofsnap1/test1/115/18' 2023-10-30T12:32:22.002 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/115/19' -> 'copyofsnap1/test1/115/19' 2023-10-30T12:32:22.002 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/115/20' -> 'copyofsnap1/test1/115/20' 2023-10-30T12:32:22.002 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/115/10' -> 'copyofsnap1/test1/115/10' 2023-10-30T12:32:22.002 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/116' -> 'copyofsnap1/test1/116' 2023-10-30T12:32:22.003 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/116/11' -> 'copyofsnap1/test1/116/11' 2023-10-30T12:32:22.003 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/116/12' -> 'copyofsnap1/test1/116/12' 2023-10-30T12:32:22.003 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/116/13' -> 'copyofsnap1/test1/116/13' 2023-10-30T12:32:22.003 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/116/14' -> 'copyofsnap1/test1/116/14' 2023-10-30T12:32:22.003 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/116/15' -> 'copyofsnap1/test1/116/15' 2023-10-30T12:32:22.003 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/116/16' -> 'copyofsnap1/test1/116/16' 2023-10-30T12:32:22.004 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/116/17' -> 'copyofsnap1/test1/116/17' 2023-10-30T12:32:22.004 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/116/18' -> 'copyofsnap1/test1/116/18' 2023-10-30T12:32:22.004 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/116/19' -> 'copyofsnap1/test1/116/19' 2023-10-30T12:32:22.004 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/116/20' -> 'copyofsnap1/test1/116/20' 2023-10-30T12:32:22.004 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/116/10' -> 'copyofsnap1/test1/116/10' 2023-10-30T12:32:22.005 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/117' -> 'copyofsnap1/test1/117' 2023-10-30T12:32:22.005 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/117/11' -> 'copyofsnap1/test1/117/11' 2023-10-30T12:32:22.005 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/117/12' -> 'copyofsnap1/test1/117/12' 2023-10-30T12:32:22.005 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/117/13' -> 'copyofsnap1/test1/117/13' 2023-10-30T12:32:22.005 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/117/14' -> 'copyofsnap1/test1/117/14' 2023-10-30T12:32:22.005 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/117/15' -> 'copyofsnap1/test1/117/15' 2023-10-30T12:32:22.006 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/117/16' -> 'copyofsnap1/test1/117/16' 2023-10-30T12:32:22.006 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/117/17' -> 'copyofsnap1/test1/117/17' 2023-10-30T12:32:22.006 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/117/18' -> 'copyofsnap1/test1/117/18' 2023-10-30T12:32:22.006 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/117/19' -> 'copyofsnap1/test1/117/19' 2023-10-30T12:32:22.006 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/117/20' -> 'copyofsnap1/test1/117/20' 2023-10-30T12:32:22.007 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/117/10' -> 'copyofsnap1/test1/117/10' 2023-10-30T12:32:22.007 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/118' -> 'copyofsnap1/test1/118' 2023-10-30T12:32:22.007 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/118/11' -> 'copyofsnap1/test1/118/11' 2023-10-30T12:32:22.007 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/118/12' -> 'copyofsnap1/test1/118/12' 2023-10-30T12:32:22.007 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/118/13' -> 'copyofsnap1/test1/118/13' 2023-10-30T12:32:22.008 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/118/14' -> 'copyofsnap1/test1/118/14' 2023-10-30T12:32:22.008 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/118/15' -> 'copyofsnap1/test1/118/15' 2023-10-30T12:32:22.008 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/118/16' -> 'copyofsnap1/test1/118/16' 2023-10-30T12:32:22.008 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/118/17' -> 'copyofsnap1/test1/118/17' 2023-10-30T12:32:22.008 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/118/18' -> 'copyofsnap1/test1/118/18' 2023-10-30T12:32:22.009 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/118/19' -> 'copyofsnap1/test1/118/19' 2023-10-30T12:32:22.009 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/118/20' -> 'copyofsnap1/test1/118/20' 2023-10-30T12:32:22.009 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/118/10' -> 'copyofsnap1/test1/118/10' 2023-10-30T12:32:22.009 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/119' -> 'copyofsnap1/test1/119' 2023-10-30T12:32:22.009 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/119/11' -> 'copyofsnap1/test1/119/11' 2023-10-30T12:32:22.010 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/119/12' -> 'copyofsnap1/test1/119/12' 2023-10-30T12:32:22.010 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/119/13' -> 'copyofsnap1/test1/119/13' 2023-10-30T12:32:22.010 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/119/14' -> 'copyofsnap1/test1/119/14' 2023-10-30T12:32:22.010 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/119/15' -> 'copyofsnap1/test1/119/15' 2023-10-30T12:32:22.010 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/119/16' -> 'copyofsnap1/test1/119/16' 2023-10-30T12:32:22.011 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/119/17' -> 'copyofsnap1/test1/119/17' 2023-10-30T12:32:22.011 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/119/18' -> 'copyofsnap1/test1/119/18' 2023-10-30T12:32:22.011 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/119/19' -> 'copyofsnap1/test1/119/19' 2023-10-30T12:32:22.011 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/119/20' -> 'copyofsnap1/test1/119/20' 2023-10-30T12:32:22.011 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/119/10' -> 'copyofsnap1/test1/119/10' 2023-10-30T12:32:22.011 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/120' -> 'copyofsnap1/test1/120' 2023-10-30T12:32:22.012 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/120/11' -> 'copyofsnap1/test1/120/11' 2023-10-30T12:32:22.576 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/120/12' -> 'copyofsnap1/test1/120/12' 2023-10-30T12:32:22.577 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/120/13' -> 'copyofsnap1/test1/120/13' 2023-10-30T12:32:22.577 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/120/14' -> 'copyofsnap1/test1/120/14' 2023-10-30T12:32:22.577 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/120/15' -> 'copyofsnap1/test1/120/15' 2023-10-30T12:32:22.577 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/120/16' -> 'copyofsnap1/test1/120/16' 2023-10-30T12:32:22.578 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/120/17' -> 'copyofsnap1/test1/120/17' 2023-10-30T12:32:22.578 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/120/18' -> 'copyofsnap1/test1/120/18' 2023-10-30T12:32:22.578 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/120/19' -> 'copyofsnap1/test1/120/19' 2023-10-30T12:32:22.578 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/120/20' -> 'copyofsnap1/test1/120/20' 2023-10-30T12:32:22.578 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/120/10' -> 'copyofsnap1/test1/120/10' 2023-10-30T12:32:22.579 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/121' -> 'copyofsnap1/test1/121' 2023-10-30T12:32:22.579 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/121/11' -> 'copyofsnap1/test1/121/11' 2023-10-30T12:32:22.579 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/121/12' -> 'copyofsnap1/test1/121/12' 2023-10-30T12:32:22.579 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/121/13' -> 'copyofsnap1/test1/121/13' 2023-10-30T12:32:22.579 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/121/14' -> 'copyofsnap1/test1/121/14' 2023-10-30T12:32:22.579 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/121/15' -> 'copyofsnap1/test1/121/15' 2023-10-30T12:32:22.580 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/121/16' -> 'copyofsnap1/test1/121/16' 2023-10-30T12:32:22.580 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/121/17' -> 'copyofsnap1/test1/121/17' 2023-10-30T12:32:22.580 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/121/18' -> 'copyofsnap1/test1/121/18' 2023-10-30T12:32:22.580 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/121/19' -> 'copyofsnap1/test1/121/19' 2023-10-30T12:32:22.580 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/121/20' -> 'copyofsnap1/test1/121/20' 2023-10-30T12:32:22.581 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/121/10' -> 'copyofsnap1/test1/121/10' 2023-10-30T12:32:22.581 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/122' -> 'copyofsnap1/test1/122' 2023-10-30T12:32:22.581 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/122/11' -> 'copyofsnap1/test1/122/11' 2023-10-30T12:32:22.581 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/122/12' -> 'copyofsnap1/test1/122/12' 2023-10-30T12:32:22.581 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/122/13' -> 'copyofsnap1/test1/122/13' 2023-10-30T12:32:22.581 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/122/14' -> 'copyofsnap1/test1/122/14' 2023-10-30T12:32:22.582 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/122/15' -> 'copyofsnap1/test1/122/15' 2023-10-30T12:32:22.582 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/122/16' -> 'copyofsnap1/test1/122/16' 2023-10-30T12:32:22.582 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/122/17' -> 'copyofsnap1/test1/122/17' 2023-10-30T12:32:22.582 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/122/18' -> 'copyofsnap1/test1/122/18' 2023-10-30T12:32:22.582 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/122/19' -> 'copyofsnap1/test1/122/19' 2023-10-30T12:32:22.583 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/122/20' -> 'copyofsnap1/test1/122/20' 2023-10-30T12:32:22.583 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/122/10' -> 'copyofsnap1/test1/122/10' 2023-10-30T12:32:22.583 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/123' -> 'copyofsnap1/test1/123' 2023-10-30T12:32:22.583 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/123/11' -> 'copyofsnap1/test1/123/11' 2023-10-30T12:32:22.583 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/123/12' -> 'copyofsnap1/test1/123/12' 2023-10-30T12:32:22.584 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/123/13' -> 'copyofsnap1/test1/123/13' 2023-10-30T12:32:22.584 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/123/14' -> 'copyofsnap1/test1/123/14' 2023-10-30T12:32:22.584 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/123/15' -> 'copyofsnap1/test1/123/15' 2023-10-30T12:32:22.584 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/123/16' -> 'copyofsnap1/test1/123/16' 2023-10-30T12:32:22.584 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/123/17' -> 'copyofsnap1/test1/123/17' 2023-10-30T12:32:22.585 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/123/18' -> 'copyofsnap1/test1/123/18' 2023-10-30T12:32:22.585 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/123/19' -> 'copyofsnap1/test1/123/19' 2023-10-30T12:32:22.585 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/123/20' -> 'copyofsnap1/test1/123/20' 2023-10-30T12:32:22.585 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/123/10' -> 'copyofsnap1/test1/123/10' 2023-10-30T12:32:22.585 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/124' -> 'copyofsnap1/test1/124' 2023-10-30T12:32:22.586 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/124/11' -> 'copyofsnap1/test1/124/11' 2023-10-30T12:32:22.586 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/124/12' -> 'copyofsnap1/test1/124/12' 2023-10-30T12:32:22.586 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/124/13' -> 'copyofsnap1/test1/124/13' 2023-10-30T12:32:22.586 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/124/14' -> 'copyofsnap1/test1/124/14' 2023-10-30T12:32:22.586 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/124/15' -> 'copyofsnap1/test1/124/15' 2023-10-30T12:32:22.587 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/124/16' -> 'copyofsnap1/test1/124/16' 2023-10-30T12:32:22.587 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/124/17' -> 'copyofsnap1/test1/124/17' 2023-10-30T12:32:22.587 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/124/18' -> 'copyofsnap1/test1/124/18' 2023-10-30T12:32:22.587 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/124/19' -> 'copyofsnap1/test1/124/19' 2023-10-30T12:32:22.587 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/124/20' -> 'copyofsnap1/test1/124/20' 2023-10-30T12:32:22.588 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/124/10' -> 'copyofsnap1/test1/124/10' 2023-10-30T12:32:22.588 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/125' -> 'copyofsnap1/test1/125' 2023-10-30T12:32:22.588 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/125/11' -> 'copyofsnap1/test1/125/11' 2023-10-30T12:32:22.588 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/125/12' -> 'copyofsnap1/test1/125/12' 2023-10-30T12:32:22.588 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/125/13' -> 'copyofsnap1/test1/125/13' 2023-10-30T12:32:22.589 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/125/14' -> 'copyofsnap1/test1/125/14' 2023-10-30T12:32:22.589 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/125/15' -> 'copyofsnap1/test1/125/15' 2023-10-30T12:32:22.589 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/125/16' -> 'copyofsnap1/test1/125/16' 2023-10-30T12:32:22.589 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/125/17' -> 'copyofsnap1/test1/125/17' 2023-10-30T12:32:22.589 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/125/18' -> 'copyofsnap1/test1/125/18' 2023-10-30T12:32:22.590 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/125/19' -> 'copyofsnap1/test1/125/19' 2023-10-30T12:32:22.590 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/125/20' -> 'copyofsnap1/test1/125/20' 2023-10-30T12:32:22.590 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/125/10' -> 'copyofsnap1/test1/125/10' 2023-10-30T12:32:22.590 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/126' -> 'copyofsnap1/test1/126' 2023-10-30T12:32:22.590 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/126/11' -> 'copyofsnap1/test1/126/11' 2023-10-30T12:32:22.591 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/126/12' -> 'copyofsnap1/test1/126/12' 2023-10-30T12:32:22.591 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/126/13' -> 'copyofsnap1/test1/126/13' 2023-10-30T12:32:22.591 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/126/14' -> 'copyofsnap1/test1/126/14' 2023-10-30T12:32:22.591 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/126/15' -> 'copyofsnap1/test1/126/15' 2023-10-30T12:32:22.591 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/126/16' -> 'copyofsnap1/test1/126/16' 2023-10-30T12:32:22.592 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/126/17' -> 'copyofsnap1/test1/126/17' 2023-10-30T12:32:22.592 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/126/18' -> 'copyofsnap1/test1/126/18' 2023-10-30T12:32:22.592 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/126/19' -> 'copyofsnap1/test1/126/19' 2023-10-30T12:32:22.592 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/126/20' -> 'copyofsnap1/test1/126/20' 2023-10-30T12:32:23.681 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/126/10' -> 'copyofsnap1/test1/126/10' 2023-10-30T12:32:23.682 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/127' -> 'copyofsnap1/test1/127' 2023-10-30T12:32:23.682 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/127/11' -> 'copyofsnap1/test1/127/11' 2023-10-30T12:32:23.682 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/127/12' -> 'copyofsnap1/test1/127/12' 2023-10-30T12:32:23.682 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/127/13' -> 'copyofsnap1/test1/127/13' 2023-10-30T12:32:23.682 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/127/14' -> 'copyofsnap1/test1/127/14' 2023-10-30T12:32:23.683 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/127/15' -> 'copyofsnap1/test1/127/15' 2023-10-30T12:32:23.683 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/127/16' -> 'copyofsnap1/test1/127/16' 2023-10-30T12:32:23.683 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/127/17' -> 'copyofsnap1/test1/127/17' 2023-10-30T12:32:23.683 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/127/18' -> 'copyofsnap1/test1/127/18' 2023-10-30T12:32:23.683 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/127/19' -> 'copyofsnap1/test1/127/19' 2023-10-30T12:32:23.684 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/127/20' -> 'copyofsnap1/test1/127/20' 2023-10-30T12:32:23.684 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/127/10' -> 'copyofsnap1/test1/127/10' 2023-10-30T12:32:23.684 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/128' -> 'copyofsnap1/test1/128' 2023-10-30T12:32:23.684 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/128/11' -> 'copyofsnap1/test1/128/11' 2023-10-30T12:32:23.684 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/128/12' -> 'copyofsnap1/test1/128/12' 2023-10-30T12:32:23.685 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/128/13' -> 'copyofsnap1/test1/128/13' 2023-10-30T12:32:23.685 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/128/14' -> 'copyofsnap1/test1/128/14' 2023-10-30T12:32:23.685 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/128/15' -> 'copyofsnap1/test1/128/15' 2023-10-30T12:32:23.685 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/128/16' -> 'copyofsnap1/test1/128/16' 2023-10-30T12:32:23.685 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/128/17' -> 'copyofsnap1/test1/128/17' 2023-10-30T12:32:23.686 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/128/18' -> 'copyofsnap1/test1/128/18' 2023-10-30T12:32:23.686 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/128/19' -> 'copyofsnap1/test1/128/19' 2023-10-30T12:32:23.686 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/128/20' -> 'copyofsnap1/test1/128/20' 2023-10-30T12:32:23.687 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/128/10' -> 'copyofsnap1/test1/128/10' 2023-10-30T12:32:23.687 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/129' -> 'copyofsnap1/test1/129' 2023-10-30T12:32:23.687 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/129/11' -> 'copyofsnap1/test1/129/11' 2023-10-30T12:32:23.687 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/129/12' -> 'copyofsnap1/test1/129/12' 2023-10-30T12:32:23.687 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/129/13' -> 'copyofsnap1/test1/129/13' 2023-10-30T12:32:23.687 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/129/14' -> 'copyofsnap1/test1/129/14' 2023-10-30T12:32:23.688 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/129/15' -> 'copyofsnap1/test1/129/15' 2023-10-30T12:32:23.688 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/129/16' -> 'copyofsnap1/test1/129/16' 2023-10-30T12:32:23.688 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/129/17' -> 'copyofsnap1/test1/129/17' 2023-10-30T12:32:23.688 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/129/18' -> 'copyofsnap1/test1/129/18' 2023-10-30T12:32:23.688 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/129/19' -> 'copyofsnap1/test1/129/19' 2023-10-30T12:32:23.689 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/129/20' -> 'copyofsnap1/test1/129/20' 2023-10-30T12:32:23.689 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/129/10' -> 'copyofsnap1/test1/129/10' 2023-10-30T12:32:23.689 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/130' -> 'copyofsnap1/test1/130' 2023-10-30T12:32:23.689 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/130/11' -> 'copyofsnap1/test1/130/11' 2023-10-30T12:32:23.689 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/130/12' -> 'copyofsnap1/test1/130/12' 2023-10-30T12:32:23.690 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/130/13' -> 'copyofsnap1/test1/130/13' 2023-10-30T12:32:23.690 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/130/14' -> 'copyofsnap1/test1/130/14' 2023-10-30T12:32:23.690 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/130/15' -> 'copyofsnap1/test1/130/15' 2023-10-30T12:32:23.690 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/130/16' -> 'copyofsnap1/test1/130/16' 2023-10-30T12:32:23.691 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/130/17' -> 'copyofsnap1/test1/130/17' 2023-10-30T12:32:23.691 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/130/18' -> 'copyofsnap1/test1/130/18' 2023-10-30T12:32:23.691 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/130/19' -> 'copyofsnap1/test1/130/19' 2023-10-30T12:32:23.691 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/130/20' -> 'copyofsnap1/test1/130/20' 2023-10-30T12:32:23.691 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/130/10' -> 'copyofsnap1/test1/130/10' 2023-10-30T12:32:23.692 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/131' -> 'copyofsnap1/test1/131' 2023-10-30T12:32:23.692 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/131/11' -> 'copyofsnap1/test1/131/11' 2023-10-30T12:32:23.692 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/131/12' -> 'copyofsnap1/test1/131/12' 2023-10-30T12:32:23.692 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/131/13' -> 'copyofsnap1/test1/131/13' 2023-10-30T12:32:23.692 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/131/14' -> 'copyofsnap1/test1/131/14' 2023-10-30T12:32:23.693 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/131/15' -> 'copyofsnap1/test1/131/15' 2023-10-30T12:32:23.693 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/131/16' -> 'copyofsnap1/test1/131/16' 2023-10-30T12:32:23.693 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/131/17' -> 'copyofsnap1/test1/131/17' 2023-10-30T12:32:23.693 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/131/18' -> 'copyofsnap1/test1/131/18' 2023-10-30T12:32:23.693 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/131/19' -> 'copyofsnap1/test1/131/19' 2023-10-30T12:32:23.694 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/131/20' -> 'copyofsnap1/test1/131/20' 2023-10-30T12:32:23.694 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/131/10' -> 'copyofsnap1/test1/131/10' 2023-10-30T12:32:23.694 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/132' -> 'copyofsnap1/test1/132' 2023-10-30T12:32:23.694 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/132/11' -> 'copyofsnap1/test1/132/11' 2023-10-30T12:32:23.694 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/132/12' -> 'copyofsnap1/test1/132/12' 2023-10-30T12:32:23.695 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/132/13' -> 'copyofsnap1/test1/132/13' 2023-10-30T12:32:23.695 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/132/14' -> 'copyofsnap1/test1/132/14' 2023-10-30T12:32:23.695 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/132/15' -> 'copyofsnap1/test1/132/15' 2023-10-30T12:32:23.695 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/132/16' -> 'copyofsnap1/test1/132/16' 2023-10-30T12:32:23.695 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/132/17' -> 'copyofsnap1/test1/132/17' 2023-10-30T12:32:23.696 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/132/18' -> 'copyofsnap1/test1/132/18' 2023-10-30T12:32:23.696 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/132/19' -> 'copyofsnap1/test1/132/19' 2023-10-30T12:32:23.697 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/132/20' -> 'copyofsnap1/test1/132/20' 2023-10-30T12:32:23.697 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/132/10' -> 'copyofsnap1/test1/132/10' 2023-10-30T12:32:23.697 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/133' -> 'copyofsnap1/test1/133' 2023-10-30T12:32:23.698 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/133/11' -> 'copyofsnap1/test1/133/11' 2023-10-30T12:32:23.698 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/133/12' -> 'copyofsnap1/test1/133/12' 2023-10-30T12:32:23.698 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/133/13' -> 'copyofsnap1/test1/133/13' 2023-10-30T12:32:23.699 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/133/14' -> 'copyofsnap1/test1/133/14' 2023-10-30T12:32:23.699 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/133/15' -> 'copyofsnap1/test1/133/15' 2023-10-30T12:32:23.699 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/133/16' -> 'copyofsnap1/test1/133/16' 2023-10-30T12:32:23.700 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/133/17' -> 'copyofsnap1/test1/133/17' 2023-10-30T12:32:23.961 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/133/18' -> 'copyofsnap1/test1/133/18' 2023-10-30T12:32:23.962 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/133/19' -> 'copyofsnap1/test1/133/19' 2023-10-30T12:32:23.962 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/133/20' -> 'copyofsnap1/test1/133/20' 2023-10-30T12:32:23.962 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/133/10' -> 'copyofsnap1/test1/133/10' 2023-10-30T12:32:23.962 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/134' -> 'copyofsnap1/test1/134' 2023-10-30T12:32:23.962 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/134/11' -> 'copyofsnap1/test1/134/11' 2023-10-30T12:32:23.963 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/134/12' -> 'copyofsnap1/test1/134/12' 2023-10-30T12:32:23.963 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/134/13' -> 'copyofsnap1/test1/134/13' 2023-10-30T12:32:23.963 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/134/14' -> 'copyofsnap1/test1/134/14' 2023-10-30T12:32:23.963 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/134/15' -> 'copyofsnap1/test1/134/15' 2023-10-30T12:32:23.963 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/134/16' -> 'copyofsnap1/test1/134/16' 2023-10-30T12:32:23.964 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/134/17' -> 'copyofsnap1/test1/134/17' 2023-10-30T12:32:23.964 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/134/18' -> 'copyofsnap1/test1/134/18' 2023-10-30T12:32:23.964 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/134/19' -> 'copyofsnap1/test1/134/19' 2023-10-30T12:32:23.964 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/134/20' -> 'copyofsnap1/test1/134/20' 2023-10-30T12:32:23.964 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/134/10' -> 'copyofsnap1/test1/134/10' 2023-10-30T12:32:23.965 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/135' -> 'copyofsnap1/test1/135' 2023-10-30T12:32:23.965 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/135/11' -> 'copyofsnap1/test1/135/11' 2023-10-30T12:32:23.965 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/135/12' -> 'copyofsnap1/test1/135/12' 2023-10-30T12:32:23.965 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/135/13' -> 'copyofsnap1/test1/135/13' 2023-10-30T12:32:23.965 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/135/14' -> 'copyofsnap1/test1/135/14' 2023-10-30T12:32:23.966 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/135/15' -> 'copyofsnap1/test1/135/15' 2023-10-30T12:32:23.966 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/135/16' -> 'copyofsnap1/test1/135/16' 2023-10-30T12:32:23.966 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/135/17' -> 'copyofsnap1/test1/135/17' 2023-10-30T12:32:23.966 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/135/18' -> 'copyofsnap1/test1/135/18' 2023-10-30T12:32:23.966 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/135/19' -> 'copyofsnap1/test1/135/19' 2023-10-30T12:32:23.967 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/135/20' -> 'copyofsnap1/test1/135/20' 2023-10-30T12:32:23.967 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/135/10' -> 'copyofsnap1/test1/135/10' 2023-10-30T12:32:23.967 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/136' -> 'copyofsnap1/test1/136' 2023-10-30T12:32:23.967 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/136/11' -> 'copyofsnap1/test1/136/11' 2023-10-30T12:32:23.967 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/136/12' -> 'copyofsnap1/test1/136/12' 2023-10-30T12:32:23.968 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/136/13' -> 'copyofsnap1/test1/136/13' 2023-10-30T12:32:23.968 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/136/14' -> 'copyofsnap1/test1/136/14' 2023-10-30T12:32:23.968 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/136/15' -> 'copyofsnap1/test1/136/15' 2023-10-30T12:32:23.968 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/136/16' -> 'copyofsnap1/test1/136/16' 2023-10-30T12:32:23.968 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/136/17' -> 'copyofsnap1/test1/136/17' 2023-10-30T12:32:23.969 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/136/18' -> 'copyofsnap1/test1/136/18' 2023-10-30T12:32:23.969 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/136/19' -> 'copyofsnap1/test1/136/19' 2023-10-30T12:32:23.969 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/136/20' -> 'copyofsnap1/test1/136/20' 2023-10-30T12:32:23.969 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/136/10' -> 'copyofsnap1/test1/136/10' 2023-10-30T12:32:23.969 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/137' -> 'copyofsnap1/test1/137' 2023-10-30T12:32:23.970 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/137/11' -> 'copyofsnap1/test1/137/11' 2023-10-30T12:32:23.970 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/137/12' -> 'copyofsnap1/test1/137/12' 2023-10-30T12:32:23.970 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/137/13' -> 'copyofsnap1/test1/137/13' 2023-10-30T12:32:23.970 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/137/14' -> 'copyofsnap1/test1/137/14' 2023-10-30T12:32:23.970 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/137/15' -> 'copyofsnap1/test1/137/15' 2023-10-30T12:32:23.971 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/137/16' -> 'copyofsnap1/test1/137/16' 2023-10-30T12:32:23.971 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/137/17' -> 'copyofsnap1/test1/137/17' 2023-10-30T12:32:23.971 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/137/18' -> 'copyofsnap1/test1/137/18' 2023-10-30T12:32:23.971 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/137/19' -> 'copyofsnap1/test1/137/19' 2023-10-30T12:32:23.971 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/137/20' -> 'copyofsnap1/test1/137/20' 2023-10-30T12:32:23.972 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/137/10' -> 'copyofsnap1/test1/137/10' 2023-10-30T12:32:23.972 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/138' -> 'copyofsnap1/test1/138' 2023-10-30T12:32:23.972 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/138/11' -> 'copyofsnap1/test1/138/11' 2023-10-30T12:32:23.972 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/138/12' -> 'copyofsnap1/test1/138/12' 2023-10-30T12:32:23.972 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/138/13' -> 'copyofsnap1/test1/138/13' 2023-10-30T12:32:23.972 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/138/14' -> 'copyofsnap1/test1/138/14' 2023-10-30T12:32:23.973 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/138/15' -> 'copyofsnap1/test1/138/15' 2023-10-30T12:32:23.973 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/138/16' -> 'copyofsnap1/test1/138/16' 2023-10-30T12:32:23.973 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/138/17' -> 'copyofsnap1/test1/138/17' 2023-10-30T12:32:23.973 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/138/18' -> 'copyofsnap1/test1/138/18' 2023-10-30T12:32:23.973 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/138/19' -> 'copyofsnap1/test1/138/19' 2023-10-30T12:32:23.974 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/138/20' -> 'copyofsnap1/test1/138/20' 2023-10-30T12:32:23.974 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/138/10' -> 'copyofsnap1/test1/138/10' 2023-10-30T12:32:23.974 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/139' -> 'copyofsnap1/test1/139' 2023-10-30T12:32:23.974 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/139/11' -> 'copyofsnap1/test1/139/11' 2023-10-30T12:32:23.974 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/139/12' -> 'copyofsnap1/test1/139/12' 2023-10-30T12:32:23.975 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/139/13' -> 'copyofsnap1/test1/139/13' 2023-10-30T12:32:23.975 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/139/14' -> 'copyofsnap1/test1/139/14' 2023-10-30T12:32:23.975 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/139/15' -> 'copyofsnap1/test1/139/15' 2023-10-30T12:32:23.975 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/139/16' -> 'copyofsnap1/test1/139/16' 2023-10-30T12:32:23.975 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/139/17' -> 'copyofsnap1/test1/139/17' 2023-10-30T12:32:23.976 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/139/18' -> 'copyofsnap1/test1/139/18' 2023-10-30T12:32:23.976 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/139/19' -> 'copyofsnap1/test1/139/19' 2023-10-30T12:32:23.976 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/139/20' -> 'copyofsnap1/test1/139/20' 2023-10-30T12:32:23.976 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/139/10' -> 'copyofsnap1/test1/139/10' 2023-10-30T12:32:23.976 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/140' -> 'copyofsnap1/test1/140' 2023-10-30T12:32:23.976 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/140/11' -> 'copyofsnap1/test1/140/11' 2023-10-30T12:32:23.977 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/140/12' -> 'copyofsnap1/test1/140/12' 2023-10-30T12:32:23.977 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/140/13' -> 'copyofsnap1/test1/140/13' 2023-10-30T12:32:23.977 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/140/14' -> 'copyofsnap1/test1/140/14' 2023-10-30T12:32:25.268 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/140/15' -> 'copyofsnap1/test1/140/15' 2023-10-30T12:32:25.269 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/140/16' -> 'copyofsnap1/test1/140/16' 2023-10-30T12:32:25.269 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/140/17' -> 'copyofsnap1/test1/140/17' 2023-10-30T12:32:25.269 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/140/18' -> 'copyofsnap1/test1/140/18' 2023-10-30T12:32:25.270 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/140/19' -> 'copyofsnap1/test1/140/19' 2023-10-30T12:32:25.270 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/140/20' -> 'copyofsnap1/test1/140/20' 2023-10-30T12:32:25.270 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/140/10' -> 'copyofsnap1/test1/140/10' 2023-10-30T12:32:25.271 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/141' -> 'copyofsnap1/test1/141' 2023-10-30T12:32:25.271 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/141/11' -> 'copyofsnap1/test1/141/11' 2023-10-30T12:32:25.271 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/141/12' -> 'copyofsnap1/test1/141/12' 2023-10-30T12:32:25.272 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/141/13' -> 'copyofsnap1/test1/141/13' 2023-10-30T12:32:25.272 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/141/14' -> 'copyofsnap1/test1/141/14' 2023-10-30T12:32:25.272 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/141/15' -> 'copyofsnap1/test1/141/15' 2023-10-30T12:32:25.273 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/141/16' -> 'copyofsnap1/test1/141/16' 2023-10-30T12:32:25.273 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/141/17' -> 'copyofsnap1/test1/141/17' 2023-10-30T12:32:25.273 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/141/18' -> 'copyofsnap1/test1/141/18' 2023-10-30T12:32:25.274 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/141/19' -> 'copyofsnap1/test1/141/19' 2023-10-30T12:32:25.274 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/141/20' -> 'copyofsnap1/test1/141/20' 2023-10-30T12:32:25.274 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/141/10' -> 'copyofsnap1/test1/141/10' 2023-10-30T12:32:25.275 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/142' -> 'copyofsnap1/test1/142' 2023-10-30T12:32:25.275 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/142/11' -> 'copyofsnap1/test1/142/11' 2023-10-30T12:32:25.276 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/142/12' -> 'copyofsnap1/test1/142/12' 2023-10-30T12:32:25.276 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/142/13' -> 'copyofsnap1/test1/142/13' 2023-10-30T12:32:25.276 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/142/14' -> 'copyofsnap1/test1/142/14' 2023-10-30T12:32:25.277 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/142/15' -> 'copyofsnap1/test1/142/15' 2023-10-30T12:32:25.277 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/142/16' -> 'copyofsnap1/test1/142/16' 2023-10-30T12:32:25.277 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/142/17' -> 'copyofsnap1/test1/142/17' 2023-10-30T12:32:25.278 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/142/18' -> 'copyofsnap1/test1/142/18' 2023-10-30T12:32:25.278 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/142/19' -> 'copyofsnap1/test1/142/19' 2023-10-30T12:32:25.278 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/142/20' -> 'copyofsnap1/test1/142/20' 2023-10-30T12:32:25.278 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/142/10' -> 'copyofsnap1/test1/142/10' 2023-10-30T12:32:25.279 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/143' -> 'copyofsnap1/test1/143' 2023-10-30T12:32:25.279 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/143/11' -> 'copyofsnap1/test1/143/11' 2023-10-30T12:32:25.279 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/143/12' -> 'copyofsnap1/test1/143/12' 2023-10-30T12:32:25.279 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/143/13' -> 'copyofsnap1/test1/143/13' 2023-10-30T12:32:25.279 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/143/14' -> 'copyofsnap1/test1/143/14' 2023-10-30T12:32:25.280 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/143/15' -> 'copyofsnap1/test1/143/15' 2023-10-30T12:32:25.280 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/143/16' -> 'copyofsnap1/test1/143/16' 2023-10-30T12:32:25.280 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/143/17' -> 'copyofsnap1/test1/143/17' 2023-10-30T12:32:25.280 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/143/18' -> 'copyofsnap1/test1/143/18' 2023-10-30T12:32:25.280 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/143/19' -> 'copyofsnap1/test1/143/19' 2023-10-30T12:32:25.281 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/143/20' -> 'copyofsnap1/test1/143/20' 2023-10-30T12:32:25.281 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/143/10' -> 'copyofsnap1/test1/143/10' 2023-10-30T12:32:25.281 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/144' -> 'copyofsnap1/test1/144' 2023-10-30T12:32:25.281 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/144/11' -> 'copyofsnap1/test1/144/11' 2023-10-30T12:32:25.281 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/144/12' -> 'copyofsnap1/test1/144/12' 2023-10-30T12:32:25.282 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/144/13' -> 'copyofsnap1/test1/144/13' 2023-10-30T12:32:25.282 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/144/14' -> 'copyofsnap1/test1/144/14' 2023-10-30T12:32:25.282 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/144/15' -> 'copyofsnap1/test1/144/15' 2023-10-30T12:32:25.282 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/144/16' -> 'copyofsnap1/test1/144/16' 2023-10-30T12:32:25.282 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/144/17' -> 'copyofsnap1/test1/144/17' 2023-10-30T12:32:25.283 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/144/18' -> 'copyofsnap1/test1/144/18' 2023-10-30T12:32:25.283 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/144/19' -> 'copyofsnap1/test1/144/19' 2023-10-30T12:32:25.283 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/144/20' -> 'copyofsnap1/test1/144/20' 2023-10-30T12:32:25.283 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/144/10' -> 'copyofsnap1/test1/144/10' 2023-10-30T12:32:25.283 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/145' -> 'copyofsnap1/test1/145' 2023-10-30T12:32:25.284 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/145/11' -> 'copyofsnap1/test1/145/11' 2023-10-30T12:32:25.284 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/145/12' -> 'copyofsnap1/test1/145/12' 2023-10-30T12:32:25.284 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/145/13' -> 'copyofsnap1/test1/145/13' 2023-10-30T12:32:25.284 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/145/14' -> 'copyofsnap1/test1/145/14' 2023-10-30T12:32:25.284 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/145/15' -> 'copyofsnap1/test1/145/15' 2023-10-30T12:32:25.285 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/145/16' -> 'copyofsnap1/test1/145/16' 2023-10-30T12:32:25.285 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/145/17' -> 'copyofsnap1/test1/145/17' 2023-10-30T12:32:25.285 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/145/18' -> 'copyofsnap1/test1/145/18' 2023-10-30T12:32:25.285 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/145/19' -> 'copyofsnap1/test1/145/19' 2023-10-30T12:32:25.285 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/145/20' -> 'copyofsnap1/test1/145/20' 2023-10-30T12:32:25.286 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/145/10' -> 'copyofsnap1/test1/145/10' 2023-10-30T12:32:25.286 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/146' -> 'copyofsnap1/test1/146' 2023-10-30T12:32:25.286 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/146/11' -> 'copyofsnap1/test1/146/11' 2023-10-30T12:32:25.286 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/146/12' -> 'copyofsnap1/test1/146/12' 2023-10-30T12:32:25.287 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/146/13' -> 'copyofsnap1/test1/146/13' 2023-10-30T12:32:25.287 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/146/14' -> 'copyofsnap1/test1/146/14' 2023-10-30T12:32:25.287 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/146/15' -> 'copyofsnap1/test1/146/15' 2023-10-30T12:32:25.287 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/146/16' -> 'copyofsnap1/test1/146/16' 2023-10-30T12:32:25.287 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/146/17' -> 'copyofsnap1/test1/146/17' 2023-10-30T12:32:25.287 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/146/18' -> 'copyofsnap1/test1/146/18' 2023-10-30T12:32:25.288 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/146/19' -> 'copyofsnap1/test1/146/19' 2023-10-30T12:32:25.288 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/146/20' -> 'copyofsnap1/test1/146/20' 2023-10-30T12:32:25.288 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/146/10' -> 'copyofsnap1/test1/146/10' 2023-10-30T12:32:25.288 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/147' -> 'copyofsnap1/test1/147' 2023-10-30T12:32:25.288 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/147/11' -> 'copyofsnap1/test1/147/11' 2023-10-30T12:32:25.289 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/147/12' -> 'copyofsnap1/test1/147/12' 2023-10-30T12:32:27.340 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/147/13' -> 'copyofsnap1/test1/147/13' 2023-10-30T12:32:27.340 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/147/14' -> 'copyofsnap1/test1/147/14' 2023-10-30T12:32:27.340 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/147/15' -> 'copyofsnap1/test1/147/15' 2023-10-30T12:32:27.341 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/147/16' -> 'copyofsnap1/test1/147/16' 2023-10-30T12:32:27.341 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/147/17' -> 'copyofsnap1/test1/147/17' 2023-10-30T12:32:27.341 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/147/18' -> 'copyofsnap1/test1/147/18' 2023-10-30T12:32:27.342 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/147/19' -> 'copyofsnap1/test1/147/19' 2023-10-30T12:32:27.342 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/147/20' -> 'copyofsnap1/test1/147/20' 2023-10-30T12:32:27.342 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/147/10' -> 'copyofsnap1/test1/147/10' 2023-10-30T12:32:27.342 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/148' -> 'copyofsnap1/test1/148' 2023-10-30T12:32:27.343 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/148/11' -> 'copyofsnap1/test1/148/11' 2023-10-30T12:32:27.343 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/148/12' -> 'copyofsnap1/test1/148/12' 2023-10-30T12:32:27.343 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/148/13' -> 'copyofsnap1/test1/148/13' 2023-10-30T12:32:27.343 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/148/14' -> 'copyofsnap1/test1/148/14' 2023-10-30T12:32:27.343 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/148/15' -> 'copyofsnap1/test1/148/15' 2023-10-30T12:32:27.344 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/148/16' -> 'copyofsnap1/test1/148/16' 2023-10-30T12:32:27.344 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/148/17' -> 'copyofsnap1/test1/148/17' 2023-10-30T12:32:27.344 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/148/18' -> 'copyofsnap1/test1/148/18' 2023-10-30T12:32:27.344 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/148/19' -> 'copyofsnap1/test1/148/19' 2023-10-30T12:32:27.344 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/148/20' -> 'copyofsnap1/test1/148/20' 2023-10-30T12:32:27.345 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/148/10' -> 'copyofsnap1/test1/148/10' 2023-10-30T12:32:27.345 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/149' -> 'copyofsnap1/test1/149' 2023-10-30T12:32:27.345 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/149/11' -> 'copyofsnap1/test1/149/11' 2023-10-30T12:32:27.345 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/149/12' -> 'copyofsnap1/test1/149/12' 2023-10-30T12:32:27.345 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/149/13' -> 'copyofsnap1/test1/149/13' 2023-10-30T12:32:27.346 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/149/14' -> 'copyofsnap1/test1/149/14' 2023-10-30T12:32:27.346 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/149/15' -> 'copyofsnap1/test1/149/15' 2023-10-30T12:32:27.346 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/149/16' -> 'copyofsnap1/test1/149/16' 2023-10-30T12:32:27.346 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/149/17' -> 'copyofsnap1/test1/149/17' 2023-10-30T12:32:27.347 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/149/18' -> 'copyofsnap1/test1/149/18' 2023-10-30T12:32:27.347 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/149/19' -> 'copyofsnap1/test1/149/19' 2023-10-30T12:32:27.347 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/149/20' -> 'copyofsnap1/test1/149/20' 2023-10-30T12:32:27.347 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/149/10' -> 'copyofsnap1/test1/149/10' 2023-10-30T12:32:27.347 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/150' -> 'copyofsnap1/test1/150' 2023-10-30T12:32:27.348 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/150/11' -> 'copyofsnap1/test1/150/11' 2023-10-30T12:32:27.348 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/150/12' -> 'copyofsnap1/test1/150/12' 2023-10-30T12:32:27.348 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/150/13' -> 'copyofsnap1/test1/150/13' 2023-10-30T12:32:27.348 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/150/14' -> 'copyofsnap1/test1/150/14' 2023-10-30T12:32:27.348 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/150/15' -> 'copyofsnap1/test1/150/15' 2023-10-30T12:32:27.349 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/150/16' -> 'copyofsnap1/test1/150/16' 2023-10-30T12:32:27.349 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/150/17' -> 'copyofsnap1/test1/150/17' 2023-10-30T12:32:27.349 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/150/18' -> 'copyofsnap1/test1/150/18' 2023-10-30T12:32:27.349 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/150/19' -> 'copyofsnap1/test1/150/19' 2023-10-30T12:32:27.349 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/150/20' -> 'copyofsnap1/test1/150/20' 2023-10-30T12:32:27.350 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/150/10' -> 'copyofsnap1/test1/150/10' 2023-10-30T12:32:27.350 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/151' -> 'copyofsnap1/test1/151' 2023-10-30T12:32:27.350 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/151/11' -> 'copyofsnap1/test1/151/11' 2023-10-30T12:32:27.350 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/151/12' -> 'copyofsnap1/test1/151/12' 2023-10-30T12:32:27.351 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/151/13' -> 'copyofsnap1/test1/151/13' 2023-10-30T12:32:27.351 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/151/14' -> 'copyofsnap1/test1/151/14' 2023-10-30T12:32:27.351 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/151/15' -> 'copyofsnap1/test1/151/15' 2023-10-30T12:32:27.351 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/151/16' -> 'copyofsnap1/test1/151/16' 2023-10-30T12:32:27.351 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/151/17' -> 'copyofsnap1/test1/151/17' 2023-10-30T12:32:27.352 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/151/18' -> 'copyofsnap1/test1/151/18' 2023-10-30T12:32:27.352 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/151/19' -> 'copyofsnap1/test1/151/19' 2023-10-30T12:32:27.352 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/151/20' -> 'copyofsnap1/test1/151/20' 2023-10-30T12:32:27.352 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/151/10' -> 'copyofsnap1/test1/151/10' 2023-10-30T12:32:27.352 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/152' -> 'copyofsnap1/test1/152' 2023-10-30T12:32:27.353 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/152/11' -> 'copyofsnap1/test1/152/11' 2023-10-30T12:32:27.353 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/152/12' -> 'copyofsnap1/test1/152/12' 2023-10-30T12:32:27.353 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/152/13' -> 'copyofsnap1/test1/152/13' 2023-10-30T12:32:27.353 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/152/14' -> 'copyofsnap1/test1/152/14' 2023-10-30T12:32:27.353 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/152/15' -> 'copyofsnap1/test1/152/15' 2023-10-30T12:32:27.354 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/152/16' -> 'copyofsnap1/test1/152/16' 2023-10-30T12:32:27.354 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/152/17' -> 'copyofsnap1/test1/152/17' 2023-10-30T12:32:27.354 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/152/18' -> 'copyofsnap1/test1/152/18' 2023-10-30T12:32:27.354 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/152/19' -> 'copyofsnap1/test1/152/19' 2023-10-30T12:32:27.355 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/152/20' -> 'copyofsnap1/test1/152/20' 2023-10-30T12:32:27.355 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/152/10' -> 'copyofsnap1/test1/152/10' 2023-10-30T12:32:27.355 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/153' -> 'copyofsnap1/test1/153' 2023-10-30T12:32:27.356 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/153/11' -> 'copyofsnap1/test1/153/11' 2023-10-30T12:32:27.356 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/153/12' -> 'copyofsnap1/test1/153/12' 2023-10-30T12:32:27.356 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/153/13' -> 'copyofsnap1/test1/153/13' 2023-10-30T12:32:27.356 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/153/14' -> 'copyofsnap1/test1/153/14' 2023-10-30T12:32:27.357 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/153/15' -> 'copyofsnap1/test1/153/15' 2023-10-30T12:32:27.357 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/153/16' -> 'copyofsnap1/test1/153/16' 2023-10-30T12:32:27.357 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/153/17' -> 'copyofsnap1/test1/153/17' 2023-10-30T12:32:27.358 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/153/18' -> 'copyofsnap1/test1/153/18' 2023-10-30T12:32:27.358 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/153/19' -> 'copyofsnap1/test1/153/19' 2023-10-30T12:32:27.358 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/153/20' -> 'copyofsnap1/test1/153/20' 2023-10-30T12:32:27.359 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/153/10' -> 'copyofsnap1/test1/153/10' 2023-10-30T12:32:28.634 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/154' -> 'copyofsnap1/test1/154' 2023-10-30T12:32:28.635 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/154/11' -> 'copyofsnap1/test1/154/11' 2023-10-30T12:32:28.635 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/154/12' -> 'copyofsnap1/test1/154/12' 2023-10-30T12:32:28.635 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/154/13' -> 'copyofsnap1/test1/154/13' 2023-10-30T12:32:28.635 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/154/14' -> 'copyofsnap1/test1/154/14' 2023-10-30T12:32:28.636 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/154/15' -> 'copyofsnap1/test1/154/15' 2023-10-30T12:32:28.636 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/154/16' -> 'copyofsnap1/test1/154/16' 2023-10-30T12:32:28.636 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/154/17' -> 'copyofsnap1/test1/154/17' 2023-10-30T12:32:28.636 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/154/18' -> 'copyofsnap1/test1/154/18' 2023-10-30T12:32:28.636 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/154/19' -> 'copyofsnap1/test1/154/19' 2023-10-30T12:32:28.637 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/154/20' -> 'copyofsnap1/test1/154/20' 2023-10-30T12:32:28.637 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/154/10' -> 'copyofsnap1/test1/154/10' 2023-10-30T12:32:28.637 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/155' -> 'copyofsnap1/test1/155' 2023-10-30T12:32:28.637 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/155/11' -> 'copyofsnap1/test1/155/11' 2023-10-30T12:32:28.637 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/155/12' -> 'copyofsnap1/test1/155/12' 2023-10-30T12:32:28.637 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/155/13' -> 'copyofsnap1/test1/155/13' 2023-10-30T12:32:28.638 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/155/14' -> 'copyofsnap1/test1/155/14' 2023-10-30T12:32:28.638 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/155/15' -> 'copyofsnap1/test1/155/15' 2023-10-30T12:32:28.638 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/155/16' -> 'copyofsnap1/test1/155/16' 2023-10-30T12:32:28.638 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/155/17' -> 'copyofsnap1/test1/155/17' 2023-10-30T12:32:28.639 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/155/18' -> 'copyofsnap1/test1/155/18' 2023-10-30T12:32:28.639 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/155/19' -> 'copyofsnap1/test1/155/19' 2023-10-30T12:32:28.639 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/155/20' -> 'copyofsnap1/test1/155/20' 2023-10-30T12:32:28.639 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/155/10' -> 'copyofsnap1/test1/155/10' 2023-10-30T12:32:28.639 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/156' -> 'copyofsnap1/test1/156' 2023-10-30T12:32:28.639 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/156/11' -> 'copyofsnap1/test1/156/11' 2023-10-30T12:32:28.640 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/156/12' -> 'copyofsnap1/test1/156/12' 2023-10-30T12:32:28.640 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/156/13' -> 'copyofsnap1/test1/156/13' 2023-10-30T12:32:28.640 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/156/14' -> 'copyofsnap1/test1/156/14' 2023-10-30T12:32:28.640 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/156/15' -> 'copyofsnap1/test1/156/15' 2023-10-30T12:32:28.640 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/156/16' -> 'copyofsnap1/test1/156/16' 2023-10-30T12:32:28.641 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/156/17' -> 'copyofsnap1/test1/156/17' 2023-10-30T12:32:28.641 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/156/18' -> 'copyofsnap1/test1/156/18' 2023-10-30T12:32:28.641 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/156/19' -> 'copyofsnap1/test1/156/19' 2023-10-30T12:32:28.641 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/156/20' -> 'copyofsnap1/test1/156/20' 2023-10-30T12:32:28.641 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/156/10' -> 'copyofsnap1/test1/156/10' 2023-10-30T12:32:28.642 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/157' -> 'copyofsnap1/test1/157' 2023-10-30T12:32:28.642 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/157/11' -> 'copyofsnap1/test1/157/11' 2023-10-30T12:32:28.642 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/157/12' -> 'copyofsnap1/test1/157/12' 2023-10-30T12:32:28.642 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/157/13' -> 'copyofsnap1/test1/157/13' 2023-10-30T12:32:28.642 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/157/14' -> 'copyofsnap1/test1/157/14' 2023-10-30T12:32:28.643 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/157/15' -> 'copyofsnap1/test1/157/15' 2023-10-30T12:32:28.643 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/157/16' -> 'copyofsnap1/test1/157/16' 2023-10-30T12:32:28.643 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/157/17' -> 'copyofsnap1/test1/157/17' 2023-10-30T12:32:28.643 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/157/18' -> 'copyofsnap1/test1/157/18' 2023-10-30T12:32:28.643 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/157/19' -> 'copyofsnap1/test1/157/19' 2023-10-30T12:32:28.643 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/157/20' -> 'copyofsnap1/test1/157/20' 2023-10-30T12:32:28.644 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/157/10' -> 'copyofsnap1/test1/157/10' 2023-10-30T12:32:28.644 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/158' -> 'copyofsnap1/test1/158' 2023-10-30T12:32:28.644 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/158/11' -> 'copyofsnap1/test1/158/11' 2023-10-30T12:32:28.644 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/158/12' -> 'copyofsnap1/test1/158/12' 2023-10-30T12:32:28.644 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/158/13' -> 'copyofsnap1/test1/158/13' 2023-10-30T12:32:28.645 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/158/14' -> 'copyofsnap1/test1/158/14' 2023-10-30T12:32:28.646 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/158/15' -> 'copyofsnap1/test1/158/15' 2023-10-30T12:32:28.646 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/158/16' -> 'copyofsnap1/test1/158/16' 2023-10-30T12:32:28.647 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/158/17' -> 'copyofsnap1/test1/158/17' 2023-10-30T12:32:28.647 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/158/18' -> 'copyofsnap1/test1/158/18' 2023-10-30T12:32:28.647 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/158/19' -> 'copyofsnap1/test1/158/19' 2023-10-30T12:32:28.647 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/158/20' -> 'copyofsnap1/test1/158/20' 2023-10-30T12:32:28.647 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/158/10' -> 'copyofsnap1/test1/158/10' 2023-10-30T12:32:28.648 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/159' -> 'copyofsnap1/test1/159' 2023-10-30T12:32:28.648 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/159/11' -> 'copyofsnap1/test1/159/11' 2023-10-30T12:32:28.648 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/159/12' -> 'copyofsnap1/test1/159/12' 2023-10-30T12:32:28.648 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/159/13' -> 'copyofsnap1/test1/159/13' 2023-10-30T12:32:28.648 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/159/14' -> 'copyofsnap1/test1/159/14' 2023-10-30T12:32:28.648 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/159/15' -> 'copyofsnap1/test1/159/15' 2023-10-30T12:32:28.649 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/159/16' -> 'copyofsnap1/test1/159/16' 2023-10-30T12:32:28.649 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/159/17' -> 'copyofsnap1/test1/159/17' 2023-10-30T12:32:28.649 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/159/18' -> 'copyofsnap1/test1/159/18' 2023-10-30T12:32:28.649 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/159/19' -> 'copyofsnap1/test1/159/19' 2023-10-30T12:32:28.649 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/159/20' -> 'copyofsnap1/test1/159/20' 2023-10-30T12:32:28.650 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/159/10' -> 'copyofsnap1/test1/159/10' 2023-10-30T12:32:28.650 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/160' -> 'copyofsnap1/test1/160' 2023-10-30T12:32:28.650 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/160/11' -> 'copyofsnap1/test1/160/11' 2023-10-30T12:32:28.650 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/160/12' -> 'copyofsnap1/test1/160/12' 2023-10-30T12:32:28.650 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/160/13' -> 'copyofsnap1/test1/160/13' 2023-10-30T12:32:28.651 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/160/14' -> 'copyofsnap1/test1/160/14' 2023-10-30T12:32:28.651 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/160/15' -> 'copyofsnap1/test1/160/15' 2023-10-30T12:32:28.651 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/160/16' -> 'copyofsnap1/test1/160/16' 2023-10-30T12:32:28.651 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/160/17' -> 'copyofsnap1/test1/160/17' 2023-10-30T12:32:28.651 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/160/18' -> 'copyofsnap1/test1/160/18' 2023-10-30T12:32:29.209 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/160/19' -> 'copyofsnap1/test1/160/19' 2023-10-30T12:32:29.209 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/160/20' -> 'copyofsnap1/test1/160/20' 2023-10-30T12:32:29.209 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/160/10' -> 'copyofsnap1/test1/160/10' 2023-10-30T12:32:29.210 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/161' -> 'copyofsnap1/test1/161' 2023-10-30T12:32:29.210 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/161/11' -> 'copyofsnap1/test1/161/11' 2023-10-30T12:32:29.210 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/161/12' -> 'copyofsnap1/test1/161/12' 2023-10-30T12:32:29.210 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/161/13' -> 'copyofsnap1/test1/161/13' 2023-10-30T12:32:29.210 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/161/14' -> 'copyofsnap1/test1/161/14' 2023-10-30T12:32:29.211 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/161/15' -> 'copyofsnap1/test1/161/15' 2023-10-30T12:32:29.211 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/161/16' -> 'copyofsnap1/test1/161/16' 2023-10-30T12:32:29.211 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/161/17' -> 'copyofsnap1/test1/161/17' 2023-10-30T12:32:29.211 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/161/18' -> 'copyofsnap1/test1/161/18' 2023-10-30T12:32:29.211 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/161/19' -> 'copyofsnap1/test1/161/19' 2023-10-30T12:32:29.212 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/161/20' -> 'copyofsnap1/test1/161/20' 2023-10-30T12:32:29.212 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/161/10' -> 'copyofsnap1/test1/161/10' 2023-10-30T12:32:29.212 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/162' -> 'copyofsnap1/test1/162' 2023-10-30T12:32:29.212 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/162/11' -> 'copyofsnap1/test1/162/11' 2023-10-30T12:32:29.212 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/162/12' -> 'copyofsnap1/test1/162/12' 2023-10-30T12:32:29.213 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/162/13' -> 'copyofsnap1/test1/162/13' 2023-10-30T12:32:29.213 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/162/14' -> 'copyofsnap1/test1/162/14' 2023-10-30T12:32:29.213 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/162/15' -> 'copyofsnap1/test1/162/15' 2023-10-30T12:32:29.213 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/162/16' -> 'copyofsnap1/test1/162/16' 2023-10-30T12:32:29.213 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/162/17' -> 'copyofsnap1/test1/162/17' 2023-10-30T12:32:29.214 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/162/18' -> 'copyofsnap1/test1/162/18' 2023-10-30T12:32:29.214 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/162/19' -> 'copyofsnap1/test1/162/19' 2023-10-30T12:32:29.214 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/162/20' -> 'copyofsnap1/test1/162/20' 2023-10-30T12:32:29.214 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/162/10' -> 'copyofsnap1/test1/162/10' 2023-10-30T12:32:29.214 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/163' -> 'copyofsnap1/test1/163' 2023-10-30T12:32:29.215 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/163/11' -> 'copyofsnap1/test1/163/11' 2023-10-30T12:32:29.215 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/163/12' -> 'copyofsnap1/test1/163/12' 2023-10-30T12:32:29.215 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/163/13' -> 'copyofsnap1/test1/163/13' 2023-10-30T12:32:29.215 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/163/14' -> 'copyofsnap1/test1/163/14' 2023-10-30T12:32:29.215 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/163/15' -> 'copyofsnap1/test1/163/15' 2023-10-30T12:32:29.216 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/163/16' -> 'copyofsnap1/test1/163/16' 2023-10-30T12:32:29.216 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/163/17' -> 'copyofsnap1/test1/163/17' 2023-10-30T12:32:29.216 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/163/18' -> 'copyofsnap1/test1/163/18' 2023-10-30T12:32:29.216 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/163/19' -> 'copyofsnap1/test1/163/19' 2023-10-30T12:32:29.216 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/163/20' -> 'copyofsnap1/test1/163/20' 2023-10-30T12:32:29.217 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/163/10' -> 'copyofsnap1/test1/163/10' 2023-10-30T12:32:29.217 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/164' -> 'copyofsnap1/test1/164' 2023-10-30T12:32:29.217 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/164/11' -> 'copyofsnap1/test1/164/11' 2023-10-30T12:32:29.217 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/164/12' -> 'copyofsnap1/test1/164/12' 2023-10-30T12:32:29.218 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/164/13' -> 'copyofsnap1/test1/164/13' 2023-10-30T12:32:29.218 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/164/14' -> 'copyofsnap1/test1/164/14' 2023-10-30T12:32:29.218 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/164/15' -> 'copyofsnap1/test1/164/15' 2023-10-30T12:32:29.218 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/164/16' -> 'copyofsnap1/test1/164/16' 2023-10-30T12:32:29.218 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/164/17' -> 'copyofsnap1/test1/164/17' 2023-10-30T12:32:29.218 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/164/18' -> 'copyofsnap1/test1/164/18' 2023-10-30T12:32:29.219 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/164/19' -> 'copyofsnap1/test1/164/19' 2023-10-30T12:32:29.219 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/164/20' -> 'copyofsnap1/test1/164/20' 2023-10-30T12:32:29.219 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/164/10' -> 'copyofsnap1/test1/164/10' 2023-10-30T12:32:29.219 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/165' -> 'copyofsnap1/test1/165' 2023-10-30T12:32:29.220 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/165/11' -> 'copyofsnap1/test1/165/11' 2023-10-30T12:32:29.220 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/165/12' -> 'copyofsnap1/test1/165/12' 2023-10-30T12:32:29.220 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/165/13' -> 'copyofsnap1/test1/165/13' 2023-10-30T12:32:29.220 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/165/14' -> 'copyofsnap1/test1/165/14' 2023-10-30T12:32:29.220 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/165/15' -> 'copyofsnap1/test1/165/15' 2023-10-30T12:32:29.221 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/165/16' -> 'copyofsnap1/test1/165/16' 2023-10-30T12:32:29.221 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/165/17' -> 'copyofsnap1/test1/165/17' 2023-10-30T12:32:29.221 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/165/18' -> 'copyofsnap1/test1/165/18' 2023-10-30T12:32:29.221 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/165/19' -> 'copyofsnap1/test1/165/19' 2023-10-30T12:32:29.221 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/165/20' -> 'copyofsnap1/test1/165/20' 2023-10-30T12:32:29.221 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/165/10' -> 'copyofsnap1/test1/165/10' 2023-10-30T12:32:29.222 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/166' -> 'copyofsnap1/test1/166' 2023-10-30T12:32:29.222 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/166/11' -> 'copyofsnap1/test1/166/11' 2023-10-30T12:32:29.222 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/166/12' -> 'copyofsnap1/test1/166/12' 2023-10-30T12:32:29.222 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/166/13' -> 'copyofsnap1/test1/166/13' 2023-10-30T12:32:29.222 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/166/14' -> 'copyofsnap1/test1/166/14' 2023-10-30T12:32:29.223 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/166/15' -> 'copyofsnap1/test1/166/15' 2023-10-30T12:32:29.223 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/166/16' -> 'copyofsnap1/test1/166/16' 2023-10-30T12:32:29.223 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/166/17' -> 'copyofsnap1/test1/166/17' 2023-10-30T12:32:29.223 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/166/18' -> 'copyofsnap1/test1/166/18' 2023-10-30T12:32:29.223 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/166/19' -> 'copyofsnap1/test1/166/19' 2023-10-30T12:32:29.224 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/166/20' -> 'copyofsnap1/test1/166/20' 2023-10-30T12:32:29.224 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/166/10' -> 'copyofsnap1/test1/166/10' 2023-10-30T12:32:29.224 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/167' -> 'copyofsnap1/test1/167' 2023-10-30T12:32:29.224 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/167/11' -> 'copyofsnap1/test1/167/11' 2023-10-30T12:32:29.224 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/167/12' -> 'copyofsnap1/test1/167/12' 2023-10-30T12:32:29.225 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/167/13' -> 'copyofsnap1/test1/167/13' 2023-10-30T12:32:29.225 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/167/14' -> 'copyofsnap1/test1/167/14' 2023-10-30T12:32:29.225 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/167/15' -> 'copyofsnap1/test1/167/15' 2023-10-30T12:32:30.176 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/167/16' -> 'copyofsnap1/test1/167/16' 2023-10-30T12:32:30.176 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/167/17' -> 'copyofsnap1/test1/167/17' 2023-10-30T12:32:30.176 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/167/18' -> 'copyofsnap1/test1/167/18' 2023-10-30T12:32:30.177 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/167/19' -> 'copyofsnap1/test1/167/19' 2023-10-30T12:32:30.177 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/167/20' -> 'copyofsnap1/test1/167/20' 2023-10-30T12:32:30.177 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/167/10' -> 'copyofsnap1/test1/167/10' 2023-10-30T12:32:30.177 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/168' -> 'copyofsnap1/test1/168' 2023-10-30T12:32:30.177 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/168/11' -> 'copyofsnap1/test1/168/11' 2023-10-30T12:32:30.178 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/168/12' -> 'copyofsnap1/test1/168/12' 2023-10-30T12:32:30.178 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/168/13' -> 'copyofsnap1/test1/168/13' 2023-10-30T12:32:30.178 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/168/14' -> 'copyofsnap1/test1/168/14' 2023-10-30T12:32:30.178 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/168/15' -> 'copyofsnap1/test1/168/15' 2023-10-30T12:32:30.178 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/168/16' -> 'copyofsnap1/test1/168/16' 2023-10-30T12:32:30.179 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/168/17' -> 'copyofsnap1/test1/168/17' 2023-10-30T12:32:30.179 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/168/18' -> 'copyofsnap1/test1/168/18' 2023-10-30T12:32:30.179 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/168/19' -> 'copyofsnap1/test1/168/19' 2023-10-30T12:32:30.179 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/168/20' -> 'copyofsnap1/test1/168/20' 2023-10-30T12:32:30.179 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/168/10' -> 'copyofsnap1/test1/168/10' 2023-10-30T12:32:30.180 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/169' -> 'copyofsnap1/test1/169' 2023-10-30T12:32:30.180 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/169/11' -> 'copyofsnap1/test1/169/11' 2023-10-30T12:32:30.180 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/169/12' -> 'copyofsnap1/test1/169/12' 2023-10-30T12:32:30.180 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/169/13' -> 'copyofsnap1/test1/169/13' 2023-10-30T12:32:30.180 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/169/14' -> 'copyofsnap1/test1/169/14' 2023-10-30T12:32:30.181 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/169/15' -> 'copyofsnap1/test1/169/15' 2023-10-30T12:32:30.181 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/169/16' -> 'copyofsnap1/test1/169/16' 2023-10-30T12:32:30.181 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/169/17' -> 'copyofsnap1/test1/169/17' 2023-10-30T12:32:30.181 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/169/18' -> 'copyofsnap1/test1/169/18' 2023-10-30T12:32:30.181 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/169/19' -> 'copyofsnap1/test1/169/19' 2023-10-30T12:32:30.182 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/169/20' -> 'copyofsnap1/test1/169/20' 2023-10-30T12:32:30.182 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/169/10' -> 'copyofsnap1/test1/169/10' 2023-10-30T12:32:30.182 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/170' -> 'copyofsnap1/test1/170' 2023-10-30T12:32:30.183 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/170/11' -> 'copyofsnap1/test1/170/11' 2023-10-30T12:32:30.183 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/170/12' -> 'copyofsnap1/test1/170/12' 2023-10-30T12:32:30.183 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/170/13' -> 'copyofsnap1/test1/170/13' 2023-10-30T12:32:30.183 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/170/14' -> 'copyofsnap1/test1/170/14' 2023-10-30T12:32:30.183 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/170/15' -> 'copyofsnap1/test1/170/15' 2023-10-30T12:32:30.184 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/170/16' -> 'copyofsnap1/test1/170/16' 2023-10-30T12:32:30.184 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/170/17' -> 'copyofsnap1/test1/170/17' 2023-10-30T12:32:30.184 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/170/18' -> 'copyofsnap1/test1/170/18' 2023-10-30T12:32:30.184 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/170/19' -> 'copyofsnap1/test1/170/19' 2023-10-30T12:32:30.184 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/170/20' -> 'copyofsnap1/test1/170/20' 2023-10-30T12:32:30.184 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/170/10' -> 'copyofsnap1/test1/170/10' 2023-10-30T12:32:30.185 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/171' -> 'copyofsnap1/test1/171' 2023-10-30T12:32:30.185 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/171/11' -> 'copyofsnap1/test1/171/11' 2023-10-30T12:32:30.185 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/171/12' -> 'copyofsnap1/test1/171/12' 2023-10-30T12:32:30.185 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/171/13' -> 'copyofsnap1/test1/171/13' 2023-10-30T12:32:30.185 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/171/14' -> 'copyofsnap1/test1/171/14' 2023-10-30T12:32:30.186 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/171/15' -> 'copyofsnap1/test1/171/15' 2023-10-30T12:32:30.186 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/171/16' -> 'copyofsnap1/test1/171/16' 2023-10-30T12:32:30.186 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/171/17' -> 'copyofsnap1/test1/171/17' 2023-10-30T12:32:30.186 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/171/18' -> 'copyofsnap1/test1/171/18' 2023-10-30T12:32:30.186 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/171/19' -> 'copyofsnap1/test1/171/19' 2023-10-30T12:32:30.187 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/171/20' -> 'copyofsnap1/test1/171/20' 2023-10-30T12:32:30.187 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/171/10' -> 'copyofsnap1/test1/171/10' 2023-10-30T12:32:30.187 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/172' -> 'copyofsnap1/test1/172' 2023-10-30T12:32:30.187 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/172/11' -> 'copyofsnap1/test1/172/11' 2023-10-30T12:32:30.187 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/172/12' -> 'copyofsnap1/test1/172/12' 2023-10-30T12:32:30.188 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/172/13' -> 'copyofsnap1/test1/172/13' 2023-10-30T12:32:30.188 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/172/14' -> 'copyofsnap1/test1/172/14' 2023-10-30T12:32:30.188 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/172/15' -> 'copyofsnap1/test1/172/15' 2023-10-30T12:32:30.188 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/172/16' -> 'copyofsnap1/test1/172/16' 2023-10-30T12:32:30.188 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/172/17' -> 'copyofsnap1/test1/172/17' 2023-10-30T12:32:30.188 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/172/18' -> 'copyofsnap1/test1/172/18' 2023-10-30T12:32:30.189 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/172/19' -> 'copyofsnap1/test1/172/19' 2023-10-30T12:32:30.189 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/172/20' -> 'copyofsnap1/test1/172/20' 2023-10-30T12:32:30.189 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/172/10' -> 'copyofsnap1/test1/172/10' 2023-10-30T12:32:30.189 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/173' -> 'copyofsnap1/test1/173' 2023-10-30T12:32:30.189 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/173/11' -> 'copyofsnap1/test1/173/11' 2023-10-30T12:32:30.190 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/173/12' -> 'copyofsnap1/test1/173/12' 2023-10-30T12:32:30.190 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/173/13' -> 'copyofsnap1/test1/173/13' 2023-10-30T12:32:30.190 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/173/14' -> 'copyofsnap1/test1/173/14' 2023-10-30T12:32:30.190 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/173/15' -> 'copyofsnap1/test1/173/15' 2023-10-30T12:32:30.190 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/173/16' -> 'copyofsnap1/test1/173/16' 2023-10-30T12:32:30.191 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/173/17' -> 'copyofsnap1/test1/173/17' 2023-10-30T12:32:30.191 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/173/18' -> 'copyofsnap1/test1/173/18' 2023-10-30T12:32:30.191 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/173/19' -> 'copyofsnap1/test1/173/19' 2023-10-30T12:32:30.191 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/173/20' -> 'copyofsnap1/test1/173/20' 2023-10-30T12:32:30.191 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/173/10' -> 'copyofsnap1/test1/173/10' 2023-10-30T12:32:30.191 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/174' -> 'copyofsnap1/test1/174' 2023-10-30T12:32:30.192 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/174/11' -> 'copyofsnap1/test1/174/11' 2023-10-30T12:32:30.192 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/174/12' -> 'copyofsnap1/test1/174/12' 2023-10-30T12:32:31.704 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/174/13' -> 'copyofsnap1/test1/174/13' 2023-10-30T12:32:31.705 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/174/14' -> 'copyofsnap1/test1/174/14' 2023-10-30T12:32:31.705 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/174/15' -> 'copyofsnap1/test1/174/15' 2023-10-30T12:32:31.706 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/174/16' -> 'copyofsnap1/test1/174/16' 2023-10-30T12:32:31.706 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/174/17' -> 'copyofsnap1/test1/174/17' 2023-10-30T12:32:31.706 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/174/18' -> 'copyofsnap1/test1/174/18' 2023-10-30T12:32:31.707 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/174/19' -> 'copyofsnap1/test1/174/19' 2023-10-30T12:32:31.707 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/174/20' -> 'copyofsnap1/test1/174/20' 2023-10-30T12:32:31.707 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/174/10' -> 'copyofsnap1/test1/174/10' 2023-10-30T12:32:31.708 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/175' -> 'copyofsnap1/test1/175' 2023-10-30T12:32:31.708 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/175/11' -> 'copyofsnap1/test1/175/11' 2023-10-30T12:32:31.708 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/175/12' -> 'copyofsnap1/test1/175/12' 2023-10-30T12:32:31.709 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/175/13' -> 'copyofsnap1/test1/175/13' 2023-10-30T12:32:31.709 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/175/14' -> 'copyofsnap1/test1/175/14' 2023-10-30T12:32:31.709 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/175/15' -> 'copyofsnap1/test1/175/15' 2023-10-30T12:32:31.710 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/175/16' -> 'copyofsnap1/test1/175/16' 2023-10-30T12:32:31.710 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/175/17' -> 'copyofsnap1/test1/175/17' 2023-10-30T12:32:31.710 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/175/18' -> 'copyofsnap1/test1/175/18' 2023-10-30T12:32:31.710 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/175/19' -> 'copyofsnap1/test1/175/19' 2023-10-30T12:32:31.711 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/175/20' -> 'copyofsnap1/test1/175/20' 2023-10-30T12:32:31.711 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/175/10' -> 'copyofsnap1/test1/175/10' 2023-10-30T12:32:31.712 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/176' -> 'copyofsnap1/test1/176' 2023-10-30T12:32:31.712 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/176/11' -> 'copyofsnap1/test1/176/11' 2023-10-30T12:32:31.712 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/176/12' -> 'copyofsnap1/test1/176/12' 2023-10-30T12:32:31.712 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/176/13' -> 'copyofsnap1/test1/176/13' 2023-10-30T12:32:31.712 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/176/14' -> 'copyofsnap1/test1/176/14' 2023-10-30T12:32:31.713 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/176/15' -> 'copyofsnap1/test1/176/15' 2023-10-30T12:32:31.713 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/176/16' -> 'copyofsnap1/test1/176/16' 2023-10-30T12:32:31.713 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/176/17' -> 'copyofsnap1/test1/176/17' 2023-10-30T12:32:31.713 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/176/18' -> 'copyofsnap1/test1/176/18' 2023-10-30T12:32:31.713 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/176/19' -> 'copyofsnap1/test1/176/19' 2023-10-30T12:32:31.714 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/176/20' -> 'copyofsnap1/test1/176/20' 2023-10-30T12:32:31.714 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/176/10' -> 'copyofsnap1/test1/176/10' 2023-10-30T12:32:31.714 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/177' -> 'copyofsnap1/test1/177' 2023-10-30T12:32:31.714 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/177/11' -> 'copyofsnap1/test1/177/11' 2023-10-30T12:32:31.714 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/177/12' -> 'copyofsnap1/test1/177/12' 2023-10-30T12:32:31.715 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/177/13' -> 'copyofsnap1/test1/177/13' 2023-10-30T12:32:31.715 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/177/14' -> 'copyofsnap1/test1/177/14' 2023-10-30T12:32:31.715 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/177/15' -> 'copyofsnap1/test1/177/15' 2023-10-30T12:32:31.715 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/177/16' -> 'copyofsnap1/test1/177/16' 2023-10-30T12:32:31.715 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/177/17' -> 'copyofsnap1/test1/177/17' 2023-10-30T12:32:31.716 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/177/18' -> 'copyofsnap1/test1/177/18' 2023-10-30T12:32:31.716 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/177/19' -> 'copyofsnap1/test1/177/19' 2023-10-30T12:32:31.716 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/177/20' -> 'copyofsnap1/test1/177/20' 2023-10-30T12:32:31.716 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/177/10' -> 'copyofsnap1/test1/177/10' 2023-10-30T12:32:31.716 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/178' -> 'copyofsnap1/test1/178' 2023-10-30T12:32:31.716 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/178/11' -> 'copyofsnap1/test1/178/11' 2023-10-30T12:32:31.717 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/178/12' -> 'copyofsnap1/test1/178/12' 2023-10-30T12:32:31.717 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/178/13' -> 'copyofsnap1/test1/178/13' 2023-10-30T12:32:31.717 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/178/14' -> 'copyofsnap1/test1/178/14' 2023-10-30T12:32:31.717 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/178/15' -> 'copyofsnap1/test1/178/15' 2023-10-30T12:32:31.717 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/178/16' -> 'copyofsnap1/test1/178/16' 2023-10-30T12:32:31.718 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/178/17' -> 'copyofsnap1/test1/178/17' 2023-10-30T12:32:31.718 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/178/18' -> 'copyofsnap1/test1/178/18' 2023-10-30T12:32:31.718 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/178/19' -> 'copyofsnap1/test1/178/19' 2023-10-30T12:32:31.718 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/178/20' -> 'copyofsnap1/test1/178/20' 2023-10-30T12:32:31.718 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/178/10' -> 'copyofsnap1/test1/178/10' 2023-10-30T12:32:31.719 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/179' -> 'copyofsnap1/test1/179' 2023-10-30T12:32:31.719 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/179/11' -> 'copyofsnap1/test1/179/11' 2023-10-30T12:32:31.719 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/179/12' -> 'copyofsnap1/test1/179/12' 2023-10-30T12:32:31.719 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/179/13' -> 'copyofsnap1/test1/179/13' 2023-10-30T12:32:31.719 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/179/14' -> 'copyofsnap1/test1/179/14' 2023-10-30T12:32:31.720 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/179/15' -> 'copyofsnap1/test1/179/15' 2023-10-30T12:32:31.720 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/179/16' -> 'copyofsnap1/test1/179/16' 2023-10-30T12:32:31.720 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/179/17' -> 'copyofsnap1/test1/179/17' 2023-10-30T12:32:31.720 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/179/18' -> 'copyofsnap1/test1/179/18' 2023-10-30T12:32:31.720 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/179/19' -> 'copyofsnap1/test1/179/19' 2023-10-30T12:32:31.721 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/179/20' -> 'copyofsnap1/test1/179/20' 2023-10-30T12:32:31.721 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/179/10' -> 'copyofsnap1/test1/179/10' 2023-10-30T12:32:31.721 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/180' -> 'copyofsnap1/test1/180' 2023-10-30T12:32:31.721 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/180/11' -> 'copyofsnap1/test1/180/11' 2023-10-30T12:32:31.721 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/180/12' -> 'copyofsnap1/test1/180/12' 2023-10-30T12:32:31.722 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/180/13' -> 'copyofsnap1/test1/180/13' 2023-10-30T12:32:31.722 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/180/14' -> 'copyofsnap1/test1/180/14' 2023-10-30T12:32:31.722 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/180/15' -> 'copyofsnap1/test1/180/15' 2023-10-30T12:32:31.722 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/180/16' -> 'copyofsnap1/test1/180/16' 2023-10-30T12:32:31.723 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/180/17' -> 'copyofsnap1/test1/180/17' 2023-10-30T12:32:31.723 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/180/18' -> 'copyofsnap1/test1/180/18' 2023-10-30T12:32:31.723 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/180/19' -> 'copyofsnap1/test1/180/19' 2023-10-30T12:32:31.723 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/180/20' -> 'copyofsnap1/test1/180/20' 2023-10-30T12:32:31.723 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/180/10' -> 'copyofsnap1/test1/180/10' 2023-10-30T12:32:34.406 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/181' -> 'copyofsnap1/test1/181' 2023-10-30T12:32:34.406 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/181/11' -> 'copyofsnap1/test1/181/11' 2023-10-30T12:32:34.406 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/181/12' -> 'copyofsnap1/test1/181/12' 2023-10-30T12:32:34.407 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/181/13' -> 'copyofsnap1/test1/181/13' 2023-10-30T12:32:34.407 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/181/14' -> 'copyofsnap1/test1/181/14' 2023-10-30T12:32:34.407 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/181/15' -> 'copyofsnap1/test1/181/15' 2023-10-30T12:32:34.407 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/181/16' -> 'copyofsnap1/test1/181/16' 2023-10-30T12:32:34.407 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/181/17' -> 'copyofsnap1/test1/181/17' 2023-10-30T12:32:34.408 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/181/18' -> 'copyofsnap1/test1/181/18' 2023-10-30T12:32:34.408 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/181/19' -> 'copyofsnap1/test1/181/19' 2023-10-30T12:32:34.408 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/181/20' -> 'copyofsnap1/test1/181/20' 2023-10-30T12:32:34.408 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/181/10' -> 'copyofsnap1/test1/181/10' 2023-10-30T12:32:34.408 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/182' -> 'copyofsnap1/test1/182' 2023-10-30T12:32:34.408 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/182/11' -> 'copyofsnap1/test1/182/11' 2023-10-30T12:32:34.409 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/182/12' -> 'copyofsnap1/test1/182/12' 2023-10-30T12:32:34.409 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/182/13' -> 'copyofsnap1/test1/182/13' 2023-10-30T12:32:34.409 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/182/14' -> 'copyofsnap1/test1/182/14' 2023-10-30T12:32:34.409 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/182/15' -> 'copyofsnap1/test1/182/15' 2023-10-30T12:32:34.409 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/182/16' -> 'copyofsnap1/test1/182/16' 2023-10-30T12:32:34.410 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/182/17' -> 'copyofsnap1/test1/182/17' 2023-10-30T12:32:34.410 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/182/18' -> 'copyofsnap1/test1/182/18' 2023-10-30T12:32:34.410 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/182/19' -> 'copyofsnap1/test1/182/19' 2023-10-30T12:32:34.410 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/182/20' -> 'copyofsnap1/test1/182/20' 2023-10-30T12:32:34.410 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/182/10' -> 'copyofsnap1/test1/182/10' 2023-10-30T12:32:34.411 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/183' -> 'copyofsnap1/test1/183' 2023-10-30T12:32:34.411 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/183/11' -> 'copyofsnap1/test1/183/11' 2023-10-30T12:32:34.411 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/183/12' -> 'copyofsnap1/test1/183/12' 2023-10-30T12:32:34.411 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/183/13' -> 'copyofsnap1/test1/183/13' 2023-10-30T12:32:34.411 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/183/14' -> 'copyofsnap1/test1/183/14' 2023-10-30T12:32:34.411 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/183/15' -> 'copyofsnap1/test1/183/15' 2023-10-30T12:32:34.412 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/183/16' -> 'copyofsnap1/test1/183/16' 2023-10-30T12:32:34.412 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/183/17' -> 'copyofsnap1/test1/183/17' 2023-10-30T12:32:34.412 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/183/18' -> 'copyofsnap1/test1/183/18' 2023-10-30T12:32:34.412 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/183/19' -> 'copyofsnap1/test1/183/19' 2023-10-30T12:32:34.412 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/183/20' -> 'copyofsnap1/test1/183/20' 2023-10-30T12:32:34.413 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/183/10' -> 'copyofsnap1/test1/183/10' 2023-10-30T12:32:34.413 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/184' -> 'copyofsnap1/test1/184' 2023-10-30T12:32:34.413 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/184/11' -> 'copyofsnap1/test1/184/11' 2023-10-30T12:32:34.413 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/184/12' -> 'copyofsnap1/test1/184/12' 2023-10-30T12:32:34.413 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/184/13' -> 'copyofsnap1/test1/184/13' 2023-10-30T12:32:34.413 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/184/14' -> 'copyofsnap1/test1/184/14' 2023-10-30T12:32:34.414 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/184/15' -> 'copyofsnap1/test1/184/15' 2023-10-30T12:32:34.414 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/184/16' -> 'copyofsnap1/test1/184/16' 2023-10-30T12:32:34.414 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/184/17' -> 'copyofsnap1/test1/184/17' 2023-10-30T12:32:34.414 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/184/18' -> 'copyofsnap1/test1/184/18' 2023-10-30T12:32:34.414 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/184/19' -> 'copyofsnap1/test1/184/19' 2023-10-30T12:32:34.415 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/184/20' -> 'copyofsnap1/test1/184/20' 2023-10-30T12:32:34.415 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/184/10' -> 'copyofsnap1/test1/184/10' 2023-10-30T12:32:34.415 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/185' -> 'copyofsnap1/test1/185' 2023-10-30T12:32:34.415 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/185/11' -> 'copyofsnap1/test1/185/11' 2023-10-30T12:32:34.415 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/185/12' -> 'copyofsnap1/test1/185/12' 2023-10-30T12:32:34.416 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/185/13' -> 'copyofsnap1/test1/185/13' 2023-10-30T12:32:34.416 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/185/14' -> 'copyofsnap1/test1/185/14' 2023-10-30T12:32:34.416 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/185/15' -> 'copyofsnap1/test1/185/15' 2023-10-30T12:32:34.416 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/185/16' -> 'copyofsnap1/test1/185/16' 2023-10-30T12:32:34.416 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/185/17' -> 'copyofsnap1/test1/185/17' 2023-10-30T12:32:34.417 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/185/18' -> 'copyofsnap1/test1/185/18' 2023-10-30T12:32:34.417 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/185/19' -> 'copyofsnap1/test1/185/19' 2023-10-30T12:32:34.417 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/185/20' -> 'copyofsnap1/test1/185/20' 2023-10-30T12:32:34.417 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/185/10' -> 'copyofsnap1/test1/185/10' 2023-10-30T12:32:34.417 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/186' -> 'copyofsnap1/test1/186' 2023-10-30T12:32:34.417 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/186/11' -> 'copyofsnap1/test1/186/11' 2023-10-30T12:32:34.418 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/186/12' -> 'copyofsnap1/test1/186/12' 2023-10-30T12:32:34.418 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/186/13' -> 'copyofsnap1/test1/186/13' 2023-10-30T12:32:34.418 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/186/14' -> 'copyofsnap1/test1/186/14' 2023-10-30T12:32:34.418 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/186/15' -> 'copyofsnap1/test1/186/15' 2023-10-30T12:32:34.418 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/186/16' -> 'copyofsnap1/test1/186/16' 2023-10-30T12:32:34.419 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/186/17' -> 'copyofsnap1/test1/186/17' 2023-10-30T12:32:34.419 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/186/18' -> 'copyofsnap1/test1/186/18' 2023-10-30T12:32:34.419 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/186/19' -> 'copyofsnap1/test1/186/19' 2023-10-30T12:32:34.419 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/186/20' -> 'copyofsnap1/test1/186/20' 2023-10-30T12:32:34.419 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/186/10' -> 'copyofsnap1/test1/186/10' 2023-10-30T12:32:34.420 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/187' -> 'copyofsnap1/test1/187' 2023-10-30T12:32:34.420 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/187/11' -> 'copyofsnap1/test1/187/11' 2023-10-30T12:32:34.420 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/187/12' -> 'copyofsnap1/test1/187/12' 2023-10-30T12:32:34.420 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/187/13' -> 'copyofsnap1/test1/187/13' 2023-10-30T12:32:34.420 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/187/14' -> 'copyofsnap1/test1/187/14' 2023-10-30T12:32:34.420 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/187/15' -> 'copyofsnap1/test1/187/15' 2023-10-30T12:32:34.421 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/187/16' -> 'copyofsnap1/test1/187/16' 2023-10-30T12:32:34.421 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/187/17' -> 'copyofsnap1/test1/187/17' 2023-10-30T12:32:34.421 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/187/18' -> 'copyofsnap1/test1/187/18' 2023-10-30T12:32:35.126 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/187/19' -> 'copyofsnap1/test1/187/19' 2023-10-30T12:32:35.126 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/187/20' -> 'copyofsnap1/test1/187/20' 2023-10-30T12:32:35.127 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/187/10' -> 'copyofsnap1/test1/187/10' 2023-10-30T12:32:35.127 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/188' -> 'copyofsnap1/test1/188' 2023-10-30T12:32:35.127 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/188/11' -> 'copyofsnap1/test1/188/11' 2023-10-30T12:32:35.127 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/188/12' -> 'copyofsnap1/test1/188/12' 2023-10-30T12:32:35.128 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/188/13' -> 'copyofsnap1/test1/188/13' 2023-10-30T12:32:35.128 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/188/14' -> 'copyofsnap1/test1/188/14' 2023-10-30T12:32:35.128 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/188/15' -> 'copyofsnap1/test1/188/15' 2023-10-30T12:32:35.128 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/188/16' -> 'copyofsnap1/test1/188/16' 2023-10-30T12:32:35.129 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/188/17' -> 'copyofsnap1/test1/188/17' 2023-10-30T12:32:35.129 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/188/18' -> 'copyofsnap1/test1/188/18' 2023-10-30T12:32:35.129 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/188/19' -> 'copyofsnap1/test1/188/19' 2023-10-30T12:32:35.129 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/188/20' -> 'copyofsnap1/test1/188/20' 2023-10-30T12:32:35.130 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/188/10' -> 'copyofsnap1/test1/188/10' 2023-10-30T12:32:35.130 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/189' -> 'copyofsnap1/test1/189' 2023-10-30T12:32:35.130 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/189/11' -> 'copyofsnap1/test1/189/11' 2023-10-30T12:32:35.130 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/189/12' -> 'copyofsnap1/test1/189/12' 2023-10-30T12:32:35.130 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/189/13' -> 'copyofsnap1/test1/189/13' 2023-10-30T12:32:35.131 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/189/14' -> 'copyofsnap1/test1/189/14' 2023-10-30T12:32:35.131 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/189/15' -> 'copyofsnap1/test1/189/15' 2023-10-30T12:32:35.131 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/189/16' -> 'copyofsnap1/test1/189/16' 2023-10-30T12:32:35.131 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/189/17' -> 'copyofsnap1/test1/189/17' 2023-10-30T12:32:35.131 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/189/18' -> 'copyofsnap1/test1/189/18' 2023-10-30T12:32:35.132 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/189/19' -> 'copyofsnap1/test1/189/19' 2023-10-30T12:32:35.132 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/189/20' -> 'copyofsnap1/test1/189/20' 2023-10-30T12:32:35.132 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/189/10' -> 'copyofsnap1/test1/189/10' 2023-10-30T12:32:35.132 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/190' -> 'copyofsnap1/test1/190' 2023-10-30T12:32:35.133 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/190/11' -> 'copyofsnap1/test1/190/11' 2023-10-30T12:32:35.133 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/190/12' -> 'copyofsnap1/test1/190/12' 2023-10-30T12:32:35.133 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/190/13' -> 'copyofsnap1/test1/190/13' 2023-10-30T12:32:35.133 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/190/14' -> 'copyofsnap1/test1/190/14' 2023-10-30T12:32:35.133 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/190/15' -> 'copyofsnap1/test1/190/15' 2023-10-30T12:32:35.134 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/190/16' -> 'copyofsnap1/test1/190/16' 2023-10-30T12:32:35.134 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/190/17' -> 'copyofsnap1/test1/190/17' 2023-10-30T12:32:35.134 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/190/18' -> 'copyofsnap1/test1/190/18' 2023-10-30T12:32:35.134 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/190/19' -> 'copyofsnap1/test1/190/19' 2023-10-30T12:32:35.135 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/190/20' -> 'copyofsnap1/test1/190/20' 2023-10-30T12:32:35.135 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/190/10' -> 'copyofsnap1/test1/190/10' 2023-10-30T12:32:35.135 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/191' -> 'copyofsnap1/test1/191' 2023-10-30T12:32:35.135 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/191/11' -> 'copyofsnap1/test1/191/11' 2023-10-30T12:32:35.135 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/191/12' -> 'copyofsnap1/test1/191/12' 2023-10-30T12:32:35.136 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/191/13' -> 'copyofsnap1/test1/191/13' 2023-10-30T12:32:35.136 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/191/14' -> 'copyofsnap1/test1/191/14' 2023-10-30T12:32:35.136 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/191/15' -> 'copyofsnap1/test1/191/15' 2023-10-30T12:32:35.136 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/191/16' -> 'copyofsnap1/test1/191/16' 2023-10-30T12:32:35.137 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/191/17' -> 'copyofsnap1/test1/191/17' 2023-10-30T12:32:35.137 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/191/18' -> 'copyofsnap1/test1/191/18' 2023-10-30T12:32:35.137 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/191/19' -> 'copyofsnap1/test1/191/19' 2023-10-30T12:32:35.137 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/191/20' -> 'copyofsnap1/test1/191/20' 2023-10-30T12:32:35.137 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/191/10' -> 'copyofsnap1/test1/191/10' 2023-10-30T12:32:35.138 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/192' -> 'copyofsnap1/test1/192' 2023-10-30T12:32:35.138 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/192/11' -> 'copyofsnap1/test1/192/11' 2023-10-30T12:32:35.138 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/192/12' -> 'copyofsnap1/test1/192/12' 2023-10-30T12:32:35.138 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/192/13' -> 'copyofsnap1/test1/192/13' 2023-10-30T12:32:35.138 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/192/14' -> 'copyofsnap1/test1/192/14' 2023-10-30T12:32:35.139 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/192/15' -> 'copyofsnap1/test1/192/15' 2023-10-30T12:32:35.139 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/192/16' -> 'copyofsnap1/test1/192/16' 2023-10-30T12:32:35.139 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/192/17' -> 'copyofsnap1/test1/192/17' 2023-10-30T12:32:35.139 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/192/18' -> 'copyofsnap1/test1/192/18' 2023-10-30T12:32:35.139 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/192/19' -> 'copyofsnap1/test1/192/19' 2023-10-30T12:32:35.140 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/192/20' -> 'copyofsnap1/test1/192/20' 2023-10-30T12:32:35.140 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/192/10' -> 'copyofsnap1/test1/192/10' 2023-10-30T12:32:35.140 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/193' -> 'copyofsnap1/test1/193' 2023-10-30T12:32:35.140 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/193/11' -> 'copyofsnap1/test1/193/11' 2023-10-30T12:32:35.140 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/193/12' -> 'copyofsnap1/test1/193/12' 2023-10-30T12:32:35.141 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/193/13' -> 'copyofsnap1/test1/193/13' 2023-10-30T12:32:35.141 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/193/14' -> 'copyofsnap1/test1/193/14' 2023-10-30T12:32:35.141 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/193/15' -> 'copyofsnap1/test1/193/15' 2023-10-30T12:32:35.141 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/193/16' -> 'copyofsnap1/test1/193/16' 2023-10-30T12:32:35.142 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/193/17' -> 'copyofsnap1/test1/193/17' 2023-10-30T12:32:35.142 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/193/18' -> 'copyofsnap1/test1/193/18' 2023-10-30T12:32:35.142 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/193/19' -> 'copyofsnap1/test1/193/19' 2023-10-30T12:32:35.142 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/193/20' -> 'copyofsnap1/test1/193/20' 2023-10-30T12:32:35.142 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/193/10' -> 'copyofsnap1/test1/193/10' 2023-10-30T12:32:35.143 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/194' -> 'copyofsnap1/test1/194' 2023-10-30T12:32:35.143 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/194/11' -> 'copyofsnap1/test1/194/11' 2023-10-30T12:32:35.143 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/194/12' -> 'copyofsnap1/test1/194/12' 2023-10-30T12:32:35.143 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/194/13' -> 'copyofsnap1/test1/194/13' 2023-10-30T12:32:35.144 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/194/14' -> 'copyofsnap1/test1/194/14' 2023-10-30T12:32:35.144 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/194/15' -> 'copyofsnap1/test1/194/15' 2023-10-30T12:32:36.274 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/194/16' -> 'copyofsnap1/test1/194/16' 2023-10-30T12:32:36.274 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/194/17' -> 'copyofsnap1/test1/194/17' 2023-10-30T12:32:36.275 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/194/18' -> 'copyofsnap1/test1/194/18' 2023-10-30T12:32:36.275 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/194/19' -> 'copyofsnap1/test1/194/19' 2023-10-30T12:32:36.275 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/194/20' -> 'copyofsnap1/test1/194/20' 2023-10-30T12:32:36.275 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/194/10' -> 'copyofsnap1/test1/194/10' 2023-10-30T12:32:36.275 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/195' -> 'copyofsnap1/test1/195' 2023-10-30T12:32:36.276 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/195/11' -> 'copyofsnap1/test1/195/11' 2023-10-30T12:32:36.276 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/195/12' -> 'copyofsnap1/test1/195/12' 2023-10-30T12:32:36.276 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/195/13' -> 'copyofsnap1/test1/195/13' 2023-10-30T12:32:36.276 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/195/14' -> 'copyofsnap1/test1/195/14' 2023-10-30T12:32:36.276 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/195/15' -> 'copyofsnap1/test1/195/15' 2023-10-30T12:32:36.276 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/195/16' -> 'copyofsnap1/test1/195/16' 2023-10-30T12:32:36.277 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/195/17' -> 'copyofsnap1/test1/195/17' 2023-10-30T12:32:36.277 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/195/18' -> 'copyofsnap1/test1/195/18' 2023-10-30T12:32:36.277 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/195/19' -> 'copyofsnap1/test1/195/19' 2023-10-30T12:32:36.277 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/195/20' -> 'copyofsnap1/test1/195/20' 2023-10-30T12:32:36.277 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/195/10' -> 'copyofsnap1/test1/195/10' 2023-10-30T12:32:36.278 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/196' -> 'copyofsnap1/test1/196' 2023-10-30T12:32:36.278 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/196/11' -> 'copyofsnap1/test1/196/11' 2023-10-30T12:32:36.278 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/196/12' -> 'copyofsnap1/test1/196/12' 2023-10-30T12:32:36.278 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/196/13' -> 'copyofsnap1/test1/196/13' 2023-10-30T12:32:36.278 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/196/14' -> 'copyofsnap1/test1/196/14' 2023-10-30T12:32:36.279 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/196/15' -> 'copyofsnap1/test1/196/15' 2023-10-30T12:32:36.279 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/196/16' -> 'copyofsnap1/test1/196/16' 2023-10-30T12:32:36.279 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/196/17' -> 'copyofsnap1/test1/196/17' 2023-10-30T12:32:36.279 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/196/18' -> 'copyofsnap1/test1/196/18' 2023-10-30T12:32:36.279 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/196/19' -> 'copyofsnap1/test1/196/19' 2023-10-30T12:32:36.280 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/196/20' -> 'copyofsnap1/test1/196/20' 2023-10-30T12:32:36.280 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/196/10' -> 'copyofsnap1/test1/196/10' 2023-10-30T12:32:36.280 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/197' -> 'copyofsnap1/test1/197' 2023-10-30T12:32:36.280 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/197/11' -> 'copyofsnap1/test1/197/11' 2023-10-30T12:32:36.280 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/197/12' -> 'copyofsnap1/test1/197/12' 2023-10-30T12:32:36.280 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/197/13' -> 'copyofsnap1/test1/197/13' 2023-10-30T12:32:36.281 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/197/14' -> 'copyofsnap1/test1/197/14' 2023-10-30T12:32:36.281 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/197/15' -> 'copyofsnap1/test1/197/15' 2023-10-30T12:32:36.281 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/197/16' -> 'copyofsnap1/test1/197/16' 2023-10-30T12:32:36.281 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/197/17' -> 'copyofsnap1/test1/197/17' 2023-10-30T12:32:36.281 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/197/18' -> 'copyofsnap1/test1/197/18' 2023-10-30T12:32:36.282 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/197/19' -> 'copyofsnap1/test1/197/19' 2023-10-30T12:32:36.282 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/197/20' -> 'copyofsnap1/test1/197/20' 2023-10-30T12:32:36.282 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/197/10' -> 'copyofsnap1/test1/197/10' 2023-10-30T12:32:36.282 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/198' -> 'copyofsnap1/test1/198' 2023-10-30T12:32:36.282 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/198/11' -> 'copyofsnap1/test1/198/11' 2023-10-30T12:32:36.283 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/198/12' -> 'copyofsnap1/test1/198/12' 2023-10-30T12:32:36.283 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/198/13' -> 'copyofsnap1/test1/198/13' 2023-10-30T12:32:36.283 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/198/14' -> 'copyofsnap1/test1/198/14' 2023-10-30T12:32:36.283 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/198/15' -> 'copyofsnap1/test1/198/15' 2023-10-30T12:32:36.283 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/198/16' -> 'copyofsnap1/test1/198/16' 2023-10-30T12:32:36.284 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/198/17' -> 'copyofsnap1/test1/198/17' 2023-10-30T12:32:36.284 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/198/18' -> 'copyofsnap1/test1/198/18' 2023-10-30T12:32:36.284 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/198/19' -> 'copyofsnap1/test1/198/19' 2023-10-30T12:32:36.284 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/198/20' -> 'copyofsnap1/test1/198/20' 2023-10-30T12:32:36.284 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/198/10' -> 'copyofsnap1/test1/198/10' 2023-10-30T12:32:36.284 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/199' -> 'copyofsnap1/test1/199' 2023-10-30T12:32:36.285 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/199/11' -> 'copyofsnap1/test1/199/11' 2023-10-30T12:32:36.285 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/199/12' -> 'copyofsnap1/test1/199/12' 2023-10-30T12:32:36.285 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/199/13' -> 'copyofsnap1/test1/199/13' 2023-10-30T12:32:36.285 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/199/14' -> 'copyofsnap1/test1/199/14' 2023-10-30T12:32:36.285 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/199/15' -> 'copyofsnap1/test1/199/15' 2023-10-30T12:32:36.286 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/199/16' -> 'copyofsnap1/test1/199/16' 2023-10-30T12:32:36.286 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/199/17' -> 'copyofsnap1/test1/199/17' 2023-10-30T12:32:36.286 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/199/18' -> 'copyofsnap1/test1/199/18' 2023-10-30T12:32:36.286 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/199/19' -> 'copyofsnap1/test1/199/19' 2023-10-30T12:32:36.286 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/199/20' -> 'copyofsnap1/test1/199/20' 2023-10-30T12:32:36.287 INFO:tasks.workunit.client.0.smithi089.stdout:'.snap/test1/199/10' -> 'copyofsnap1/test1/199/10' 2023-10-30T12:32:36.287 INFO:tasks.workunit.client.0.smithi089.stdout:Take third snapshot .snap/test3 2023-10-30T12:32:36.296 INFO:tasks.workunit.client.0.smithi089.stdout:Delete the snapshots... 2023-10-30T12:32:39.175 INFO:tasks.workunit.client.0.smithi089.stdout:.//.snap/test1 2023-10-30T12:32:39.176 INFO:tasks.workunit.client.0.smithi089.stdout:.//.snap/test2 2023-10-30T12:32:39.176 INFO:tasks.workunit.client.0.smithi089.stdout:.//.snap/test3 2023-10-30T12:32:39.201 INFO:tasks.workunit.client.0.smithi089.stdout:./274/.snap/snap-subdir-test 2023-10-30T12:32:39.858 INFO:tasks.workunit.client.0.smithi089.stdout:./166/.snap/snap-subdir-test 2023-10-30T12:32:40.379 INFO:tasks.workunit.client.0.smithi089.stdout:./256/.snap/snap-subdir-test 2023-10-30T12:32:40.387 INFO:tasks.workunit.client.0.smithi089.stdout:./247/.snap/snap-subdir-test 2023-10-30T12:32:40.397 INFO:tasks.workunit.client.0.smithi089.stdout:./248/.snap/snap-subdir-test 2023-10-30T12:32:40.408 INFO:tasks.workunit.client.0.smithi089.stdout:./147/.snap/snap-subdir-test 2023-10-30T12:32:40.417 INFO:tasks.workunit.client.0.smithi089.stdout:./151/.snap/snap-subdir-test 2023-10-30T12:32:40.427 INFO:tasks.workunit.client.0.smithi089.stdout:./170/.snap/snap-subdir-test 2023-10-30T12:32:40.438 INFO:tasks.workunit.client.0.smithi089.stdout:./167/.snap/snap-subdir-test 2023-10-30T12:32:40.448 INFO:tasks.workunit.client.0.smithi089.stdout:./140/.snap/snap-subdir-test 2023-10-30T12:32:40.459 INFO:tasks.workunit.client.0.smithi089.stdout:./142/.snap/snap-subdir-test 2023-10-30T12:32:40.467 INFO:tasks.workunit.client.0.smithi089.stdout:./150/.snap/snap-subdir-test 2023-10-30T12:32:40.471 INFO:tasks.workunit.client.0.smithi089.stdout:./273/.snap/snap-subdir-test 2023-10-30T12:32:41.268 INFO:tasks.workunit.client.0.smithi089.stdout:./269/.snap/snap-subdir-test 2023-10-30T12:32:41.280 INFO:tasks.workunit.client.0.smithi089.stdout:./105/.snap/snap-subdir-test 2023-10-30T12:32:41.288 INFO:tasks.workunit.client.0.smithi089.stdout:./210/.snap/snap-subdir-test 2023-10-30T12:32:41.293 INFO:tasks.workunit.client.0.smithi089.stdout:./265/.snap/snap-subdir-test 2023-10-30T12:32:41.301 INFO:tasks.workunit.client.0.smithi089.stdout:./218/.snap/snap-subdir-test 2023-10-30T12:32:41.308 INFO:tasks.workunit.client.0.smithi089.stdout:./290/.snap/snap-subdir-test 2023-10-30T12:32:41.318 INFO:tasks.workunit.client.0.smithi089.stdout:./149/.snap/snap-subdir-test 2023-10-30T12:32:41.329 INFO:tasks.workunit.client.0.smithi089.stdout:./109/.snap/snap-subdir-test 2023-10-30T12:32:41.731 INFO:tasks.workunit.client.0.smithi089.stdout:./292/.snap/snap-subdir-test 2023-10-30T12:32:41.736 INFO:tasks.workunit.client.0.smithi089.stdout:./242/.snap/snap-subdir-test 2023-10-30T12:32:41.747 INFO:tasks.workunit.client.0.smithi089.stdout:./158/.snap/snap-subdir-test 2023-10-30T12:32:41.755 INFO:tasks.workunit.client.0.smithi089.stdout:./243/.snap/snap-subdir-test 2023-10-30T12:32:41.759 INFO:tasks.workunit.client.0.smithi089.stdout:./106/.snap/snap-subdir-test 2023-10-30T12:32:41.763 INFO:tasks.workunit.client.0.smithi089.stdout:./250/.snap/snap-subdir-test 2023-10-30T12:32:41.768 INFO:tasks.workunit.client.0.smithi089.stdout:./221/.snap/snap-subdir-test 2023-10-30T12:32:41.777 INFO:tasks.workunit.client.0.smithi089.stdout:./244/.snap/snap-subdir-test 2023-10-30T12:32:41.782 INFO:tasks.workunit.client.0.smithi089.stdout:./258/.snap/snap-subdir-test 2023-10-30T12:32:41.789 INFO:tasks.workunit.client.0.smithi089.stdout:./268/.snap/snap-subdir-test 2023-10-30T12:32:41.792 INFO:tasks.workunit.client.0.smithi089.stdout:./189/.snap/snap-subdir-test 2023-10-30T12:32:41.799 INFO:tasks.workunit.client.0.smithi089.stdout:./285/.snap/snap-subdir-test 2023-10-30T12:32:41.806 INFO:tasks.workunit.client.0.smithi089.stdout:./208/.snap/snap-subdir-test 2023-10-30T12:32:41.813 INFO:tasks.workunit.client.0.smithi089.stdout:./288/.snap/snap-subdir-test 2023-10-30T12:32:41.822 INFO:tasks.workunit.client.0.smithi089.stdout:./209/.snap/snap-subdir-test 2023-10-30T12:32:41.829 INFO:tasks.workunit.client.0.smithi089.stdout:./253/.snap/snap-subdir-test 2023-10-30T12:32:41.838 INFO:tasks.workunit.client.0.smithi089.stdout:./249/.snap/snap-subdir-test 2023-10-30T12:32:42.536 INFO:tasks.workunit.client.0.smithi089.stdout:./133/.snap/snap-subdir-test 2023-10-30T12:32:42.544 INFO:tasks.workunit.client.0.smithi089.stdout:./291/.snap/snap-subdir-test 2023-10-30T12:32:42.551 INFO:tasks.workunit.client.0.smithi089.stdout:./226/.snap/snap-subdir-test 2023-10-30T12:32:42.560 INFO:tasks.workunit.client.0.smithi089.stdout:./173/.snap/snap-subdir-test 2023-10-30T12:32:42.571 INFO:tasks.workunit.client.0.smithi089.stdout:./174/.snap/snap-subdir-test 2023-10-30T12:32:42.580 INFO:tasks.workunit.client.0.smithi089.stdout:./299/.snap/snap-subdir-test 2023-10-30T12:32:42.586 INFO:tasks.workunit.client.0.smithi089.stdout:./262/.snap/snap-subdir-test 2023-10-30T12:32:42.593 INFO:tasks.workunit.client.0.smithi089.stdout:./231/.snap/snap-subdir-test 2023-10-30T12:32:42.599 INFO:tasks.workunit.client.0.smithi089.stdout:./214/.snap/snap-subdir-test 2023-10-30T12:32:42.606 INFO:tasks.workunit.client.0.smithi089.stdout:./281/.snap/snap-subdir-test 2023-10-30T12:32:42.614 INFO:tasks.workunit.client.0.smithi089.stdout:./289/.snap/snap-subdir-test 2023-10-30T12:32:42.625 INFO:tasks.workunit.client.0.smithi089.stdout:./283/.snap/snap-subdir-test 2023-10-30T12:32:42.636 INFO:tasks.workunit.client.0.smithi089.stdout:./232/.snap/snap-subdir-test 2023-10-30T12:32:42.945 INFO:tasks.workunit.client.0.smithi089.stdout:./298/.snap/snap-subdir-test 2023-10-30T12:32:42.954 INFO:tasks.workunit.client.0.smithi089.stdout:./233/.snap/snap-subdir-test 2023-10-30T12:32:42.957 INFO:tasks.workunit.client.0.smithi089.stdout:./155/.snap/snap-subdir-test 2023-10-30T12:32:42.968 INFO:tasks.workunit.client.0.smithi089.stdout:./114/.snap/snap-subdir-test 2023-10-30T12:32:42.978 INFO:tasks.workunit.client.0.smithi089.stdout:./159/.snap/snap-subdir-test 2023-10-30T12:32:42.986 INFO:tasks.workunit.client.0.smithi089.stdout:./187/.snap/snap-subdir-test 2023-10-30T12:32:42.993 INFO:tasks.workunit.client.0.smithi089.stdout:./185/.snap/snap-subdir-test 2023-10-30T12:32:43.004 INFO:tasks.workunit.client.0.smithi089.stdout:./293/.snap/snap-subdir-test 2023-10-30T12:32:43.012 INFO:tasks.workunit.client.0.smithi089.stdout:./228/.snap/snap-subdir-test 2023-10-30T12:32:43.025 INFO:tasks.workunit.client.0.smithi089.stdout:./120/.snap/snap-subdir-test 2023-10-30T12:32:43.035 INFO:tasks.workunit.client.0.smithi089.stdout:./110/.snap/snap-subdir-test 2023-10-30T12:32:43.044 INFO:tasks.workunit.client.0.smithi089.stdout:./267/.snap/snap-subdir-test 2023-10-30T12:32:43.051 INFO:tasks.workunit.client.0.smithi089.stdout:./202/.snap/snap-subdir-test 2023-10-30T12:32:43.062 INFO:tasks.workunit.client.0.smithi089.stdout:./137/.snap/snap-subdir-test 2023-10-30T12:32:43.071 INFO:tasks.workunit.client.0.smithi089.stdout:./246/.snap/snap-subdir-test 2023-10-30T12:32:43.080 INFO:tasks.workunit.client.0.smithi089.stdout:./219/.snap/snap-subdir-test 2023-10-30T12:32:43.088 INFO:tasks.workunit.client.0.smithi089.stdout:./198/.snap/snap-subdir-test 2023-10-30T12:32:43.103 INFO:tasks.workunit.client.0.smithi089.stdout:./136/.snap/snap-subdir-test 2023-10-30T12:32:43.109 INFO:tasks.workunit.client.0.smithi089.stdout:./270/.snap/snap-subdir-test 2023-10-30T12:32:43.121 INFO:tasks.workunit.client.0.smithi089.stdout:./125/.snap/snap-subdir-test 2023-10-30T12:32:43.129 INFO:tasks.workunit.client.0.smithi089.stdout:./194/.snap/snap-subdir-test 2023-10-30T12:32:43.144 INFO:tasks.workunit.client.0.smithi089.stdout:./119/.snap/snap-subdir-test 2023-10-30T12:32:43.153 INFO:tasks.workunit.client.0.smithi089.stdout:./297/.snap/snap-subdir-test 2023-10-30T12:32:43.398 INFO:tasks.workunit.client.0.smithi089.stdout:./230/.snap/snap-subdir-test 2023-10-30T12:32:43.407 INFO:tasks.workunit.client.0.smithi089.stdout:./260/.snap/snap-subdir-test 2023-10-30T12:32:43.415 INFO:tasks.workunit.client.0.smithi089.stdout:./157/.snap/snap-subdir-test 2023-10-30T12:32:43.423 INFO:tasks.workunit.client.0.smithi089.stdout:./172/.snap/snap-subdir-test 2023-10-30T12:32:43.429 INFO:tasks.workunit.client.0.smithi089.stdout:./277/.snap/snap-subdir-test 2023-10-30T12:32:43.432 INFO:tasks.workunit.client.0.smithi089.stdout:./186/.snap/snap-subdir-test 2023-10-30T12:32:43.436 INFO:tasks.workunit.client.0.smithi089.stdout:./237/.snap/snap-subdir-test 2023-10-30T12:32:43.439 INFO:tasks.workunit.client.0.smithi089.stdout:./190/.snap/snap-subdir-test 2023-10-30T12:32:43.445 INFO:tasks.workunit.client.0.smithi089.stdout:./213/.snap/snap-subdir-test 2023-10-30T12:32:44.453 INFO:tasks.workunit.client.0.smithi089.stdout:./104/.snap/snap-subdir-test 2023-10-30T12:32:44.459 INFO:tasks.workunit.client.0.smithi089.stdout:./164/.snap/snap-subdir-test 2023-10-30T12:32:44.467 INFO:tasks.workunit.client.0.smithi089.stdout:./112/.snap/snap-subdir-test 2023-10-30T12:32:44.477 INFO:tasks.workunit.client.0.smithi089.stdout:./295/.snap/snap-subdir-test 2023-10-30T12:32:44.486 INFO:tasks.workunit.client.0.smithi089.stdout:./272/.snap/snap-subdir-test 2023-10-30T12:32:44.498 INFO:tasks.workunit.client.0.smithi089.stdout:./100/.snap/snap-subdir-test 2023-10-30T12:32:44.507 INFO:tasks.workunit.client.0.smithi089.stdout:./134/.snap/snap-subdir-test 2023-10-30T12:32:44.517 INFO:tasks.workunit.client.0.smithi089.stdout:./200/.snap/snap-subdir-test 2023-10-30T12:32:44.525 INFO:tasks.workunit.client.0.smithi089.stdout:./115/.snap/snap-subdir-test 2023-10-30T12:32:44.654 INFO:tasks.workunit.client.0.smithi089.stdout:./284/.snap/snap-subdir-test 2023-10-30T12:32:44.660 INFO:tasks.workunit.client.0.smithi089.stdout:./280/.snap/snap-subdir-test 2023-10-30T12:32:44.670 INFO:tasks.workunit.client.0.smithi089.stdout:./116/.snap/snap-subdir-test 2023-10-30T12:32:44.676 INFO:tasks.workunit.client.0.smithi089.stdout:./245/.snap/snap-subdir-test 2023-10-30T12:32:44.679 INFO:tasks.workunit.client.0.smithi089.stdout:./195/.snap/snap-subdir-test 2023-10-30T12:32:44.684 INFO:tasks.workunit.client.0.smithi089.stdout:./138/.snap/snap-subdir-test 2023-10-30T12:32:44.696 INFO:tasks.workunit.client.0.smithi089.stdout:./135/.snap/snap-subdir-test 2023-10-30T12:32:44.711 INFO:tasks.workunit.client.0.smithi089.stdout:./117/.snap/snap-subdir-test 2023-10-30T12:32:45.343 INFO:tasks.workunit.client.0.smithi089.stdout:./241/.snap/snap-subdir-test 2023-10-30T12:32:45.356 INFO:tasks.workunit.client.0.smithi089.stdout:./143/.snap/snap-subdir-test 2023-10-30T12:32:45.359 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:32:45.363 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:32:46.112 INFO:tasks.workunit.client.0.smithi089.stdout:./180/.snap/snap-subdir-test 2023-10-30T12:32:46.423 INFO:tasks.workunit.client.0.smithi089.stdout:./126/.snap/snap-subdir-test 2023-10-30T12:32:46.433 INFO:tasks.workunit.client.0.smithi089.stdout:./161/.snap/snap-subdir-test 2023-10-30T12:32:46.443 INFO:tasks.workunit.client.0.smithi089.stdout:./204/.snap/snap-subdir-test 2023-10-30T12:32:46.447 INFO:tasks.workunit.client.0.smithi089.stdout:./176/.snap/snap-subdir-test 2023-10-30T12:32:47.413 INFO:tasks.workunit.client.0.smithi089.stdout:./255/.snap/snap-subdir-test 2023-10-30T12:32:47.422 INFO:tasks.workunit.client.0.smithi089.stdout:./215/.snap/snap-subdir-test 2023-10-30T12:32:47.441 INFO:tasks.workunit.client.0.smithi089.stdout:./111/.snap/snap-subdir-test 2023-10-30T12:32:47.452 INFO:tasks.workunit.client.0.smithi089.stdout:./238/.snap/snap-subdir-test 2023-10-30T12:32:47.462 INFO:tasks.workunit.client.0.smithi089.stdout:./191/.snap/snap-subdir-test 2023-10-30T12:32:47.466 INFO:tasks.workunit.client.0.smithi089.stdout:./184/.snap/snap-subdir-test 2023-10-30T12:32:47.655 INFO:tasks.workunit.client.0.smithi089.stdout:./192/.snap/snap-subdir-test 2023-10-30T12:32:47.668 INFO:tasks.workunit.client.0.smithi089.stdout:./103/.snap/snap-subdir-test 2023-10-30T12:32:47.677 INFO:tasks.workunit.client.0.smithi089.stdout:./128/.snap/snap-subdir-test 2023-10-30T12:32:47.689 INFO:tasks.workunit.client.0.smithi089.stdout:./229/.snap/snap-subdir-test 2023-10-30T12:32:48.464 INFO:tasks.workunit.client.0.smithi089.stdout:./144/.snap/snap-subdir-test 2023-10-30T12:32:48.480 INFO:tasks.workunit.client.0.smithi089.stdout:./146/.snap/snap-subdir-test 2023-10-30T12:32:48.490 INFO:tasks.workunit.client.0.smithi089.stdout:./227/.snap/snap-subdir-test 2023-10-30T12:32:48.502 INFO:tasks.workunit.client.0.smithi089.stdout:./278/.snap/snap-subdir-test 2023-10-30T12:32:48.515 INFO:tasks.workunit.client.0.smithi089.stdout:./141/.snap/snap-subdir-test 2023-10-30T12:32:48.524 INFO:tasks.workunit.client.0.smithi089.stdout:./234/.snap/snap-subdir-test 2023-10-30T12:32:48.534 INFO:tasks.workunit.client.0.smithi089.stdout:./225/.snap/snap-subdir-test 2023-10-30T12:32:48.546 INFO:tasks.workunit.client.0.smithi089.stdout:./182/.snap/snap-subdir-test 2023-10-30T12:32:48.558 INFO:tasks.workunit.client.0.smithi089.stdout:./107/.snap/snap-subdir-test 2023-10-30T12:32:48.561 INFO:tasks.workunit.client.0.smithi089.stdout:./178/.snap/snap-subdir-test 2023-10-30T12:32:48.569 INFO:tasks.workunit.client.0.smithi089.stdout:./266/.snap/snap-subdir-test 2023-10-30T12:32:48.575 INFO:tasks.workunit.client.0.smithi089.stdout:./240/.snap/snap-subdir-test 2023-10-30T12:32:48.583 INFO:tasks.workunit.client.0.smithi089.stdout:./156/.snap/snap-subdir-test 2023-10-30T12:32:48.780 INFO:tasks.workunit.client.0.smithi089.stdout:./131/.snap/snap-subdir-test 2023-10-30T12:32:48.792 INFO:tasks.workunit.client.0.smithi089.stdout:./123/.snap/snap-subdir-test 2023-10-30T12:32:48.802 INFO:tasks.workunit.client.0.smithi089.stdout:./160/.snap/snap-subdir-test 2023-10-30T12:32:49.479 INFO:tasks.workunit.client.0.smithi089.stdout:./165/.snap/snap-subdir-test 2023-10-30T12:32:49.491 INFO:tasks.workunit.client.0.smithi089.stdout:./101/.snap/snap-subdir-test 2023-10-30T12:32:49.500 INFO:tasks.workunit.client.0.smithi089.stdout:./148/.snap/snap-subdir-test 2023-10-30T12:32:49.508 INFO:tasks.workunit.client.0.smithi089.stdout:./168/.snap/snap-subdir-test 2023-10-30T12:32:49.511 INFO:tasks.workunit.client.0.smithi089.stdout:./179/.snap/snap-subdir-test 2023-10-30T12:32:49.518 INFO:tasks.workunit.client.0.smithi089.stdout:./257/.snap/snap-subdir-test 2023-10-30T12:32:49.526 INFO:tasks.workunit.client.0.smithi089.stdout:./252/.snap/snap-subdir-test 2023-10-30T12:32:49.535 INFO:tasks.workunit.client.0.smithi089.stdout:./132/.snap/snap-subdir-test 2023-10-30T12:32:49.542 INFO:tasks.workunit.client.0.smithi089.stdout:./236/.snap/snap-subdir-test 2023-10-30T12:32:49.550 INFO:tasks.workunit.client.0.smithi089.stdout:./211/.snap/snap-subdir-test 2023-10-30T12:32:49.553 INFO:tasks.workunit.client.0.smithi089.stdout:./199/.snap/snap-subdir-test 2023-10-30T12:32:50.113 INFO:tasks.workunit.client.0.smithi089.stdout:./205/.snap/snap-subdir-test 2023-10-30T12:32:50.202 INFO:tasks.workunit.client.0.smithi089.stdout:./163/.snap/snap-subdir-test 2023-10-30T12:32:50.210 INFO:tasks.workunit.client.0.smithi089.stdout:./261/.snap/snap-subdir-test 2023-10-30T12:32:50.217 INFO:tasks.workunit.client.0.smithi089.stdout:./201/.snap/snap-subdir-test 2023-10-30T12:32:50.224 INFO:tasks.workunit.client.0.smithi089.stdout:./282/.snap/snap-subdir-test 2023-10-30T12:32:50.235 INFO:tasks.workunit.client.0.smithi089.stdout:./296/.snap/snap-subdir-test 2023-10-30T12:32:51.017 INFO:tasks.workunit.client.0.smithi089.stdout:./217/.snap/snap-subdir-test 2023-10-30T12:32:51.021 INFO:tasks.workunit.client.0.smithi089.stdout:./153/.snap/snap-subdir-test 2023-10-30T12:32:51.031 INFO:tasks.workunit.client.0.smithi089.stdout:./206/.snap/snap-subdir-test 2023-10-30T12:32:51.046 INFO:tasks.workunit.client.0.smithi089.stdout:./122/.snap/snap-subdir-test 2023-10-30T12:32:51.056 INFO:tasks.workunit.client.0.smithi089.stdout:./162/.snap/snap-subdir-test 2023-10-30T12:32:51.407 INFO:tasks.workunit.client.0.smithi089.stdout:./197/.snap/snap-subdir-test 2023-10-30T12:32:51.420 INFO:tasks.workunit.client.0.smithi089.stdout:./216/.snap/snap-subdir-test 2023-10-30T12:32:52.099 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:32:52.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a8411d640 1 Processor -- start 2023-10-30T12:32:52.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a8411d640 1 -- start start 2023-10-30T12:32:52.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a8411d640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4a7c153bf0 0x7f4a7c071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:52.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a8411d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a7c0c6fe0 0x7f4a7c0c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:52.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a8411d640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a7c0700b0 con 0x7f4a7c0c79e0 2023-10-30T12:32:52.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a8411d640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a7c0c6810 con 0x7f4a7c153bf0 2023-10-30T12:32:52.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a8411d640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7c159900 con 0x7f4a7c0c6fe0 2023-10-30T12:32:52.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a826ba640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f4a7c0c79e0 0x7f4a7c0cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:32846/0 (socket says 172.21.15.89:32846) 2023-10-30T12:32:52.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a826ba640 1 -- 172.21.15.89:0/564813961 learned_addr learned my addr 172.21.15.89:0/564813961 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:32:52.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a81eb9640 1 --2- 172.21.15.89:0/564813961 >> v2:172.21.15.192:3300/0 conn(0x7f4a7c153bf0 0x7f4a7c071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:52.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a816b8640 1 --2- 172.21.15.89:0/564813961 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a7c0c6fe0 0x7f4a7c0c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:52.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a80eb7640 1 -- 172.21.15.89:0/564813961 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2671149040 0 0) 0x7f4a7c0700b0 con 0x7f4a7c0c79e0 2023-10-30T12:32:52.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a80eb7640 1 -- 172.21.15.89:0/564813961 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a68003530 con 0x7f4a7c0c79e0 2023-10-30T12:32:52.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a80eb7640 1 -- 172.21.15.89:0/564813961 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3330659171 0 0) 0x7f4a68003530 con 0x7f4a7c0c79e0 2023-10-30T12:32:52.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a80eb7640 1 -- 172.21.15.89:0/564813961 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a7c0c6fe0 msgr2=0x7f4a7c0c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:52.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a80eb7640 1 --2- 172.21.15.89:0/564813961 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a7c0c6fe0 0x7f4a7c0c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:52.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a80eb7640 1 -- 172.21.15.89:0/564813961 >> v2:172.21.15.192:3300/0 conn(0x7f4a7c153bf0 msgr2=0x7f4a7c071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:52.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a80eb7640 1 --2- 172.21.15.89:0/564813961 >> v2:172.21.15.192:3300/0 conn(0x7f4a7c153bf0 0x7f4a7c071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:52.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a80eb7640 1 -- 172.21.15.89:0/564813961 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a7c0c6530 con 0x7f4a7c0c79e0 2023-10-30T12:32:52.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a80eb7640 1 -- 172.21.15.89:0/564813961 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f4a74001a00 con 0x7f4a7c0c79e0 2023-10-30T12:32:52.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a80eb7640 1 -- 172.21.15.89:0/564813961 >> v1:172.21.15.89:6789/0 conn(0x7f4a7c0c79e0 legacy=0x7f4a7c0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:52.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a80eb7640 1 --2- 172.21.15.89:0/564813961 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a68003980 0x7f4a68003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:52.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a80eb7640 1 --2- 172.21.15.89:0/564813961 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a7c0c6fe0 0x7f4a680047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:52.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a80eb7640 1 --2- 172.21.15.89:0/564813961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a68004d10 0x7f4a68005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:52.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a80eb7640 1 -- 172.21.15.89:0/564813961 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7c0c6810 con 0x7f4a68004d10 2023-10-30T12:32:52.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a80eb7640 1 -- 172.21.15.89:0/564813961 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7c159900 con 0x7f4a68003980 2023-10-30T12:32:52.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a80eb7640 1 -- 172.21.15.89:0/564813961 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6800b990 con 0x7f4a7c0c6fe0 2023-10-30T12:32:52.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a826ba640 1 --2- 172.21.15.89:0/564813961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a68004d10 0x7f4a68005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:52.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a81eb9640 1 --2- 172.21.15.89:0/564813961 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a68003980 0x7f4a68003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:52.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a816b8640 1 --2- 172.21.15.89:0/564813961 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a7c0c6fe0 0x7f4a680047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:52.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a826ba640 1 -- 172.21.15.89:0/564813961 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a7c0c6fe0 msgr2=0x7f4a680047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:52.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a826ba640 1 --2- 172.21.15.89:0/564813961 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a7c0c6fe0 0x7f4a680047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:52.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a826ba640 1 -- 172.21.15.89:0/564813961 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a68003980 msgr2=0x7f4a68003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:52.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a826ba640 1 --2- 172.21.15.89:0/564813961 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a68003980 0x7f4a68003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:52.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a826ba640 1 -- 172.21.15.89:0/564813961 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a68003530 con 0x7f4a68004d10 2023-10-30T12:32:52.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.242+0000 7f4a816b8640 1 --2- 172.21.15.89:0/564813961 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a7c0c6fe0 0x7f4a680047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:32:52.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.246+0000 7f4a826ba640 1 --2- 172.21.15.89:0/564813961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a68004d10 0x7f4a68005190 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4a7400a860 tx=0x7f4a7401d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1689f03519f84ae server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:52.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.246+0000 7f4a80eb7640 1 -- 172.21.15.89:0/564813961 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a740ad630 con 0x7f4a68004d10 2023-10-30T12:32:52.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.246+0000 7f4a80eb7640 1 -- 172.21.15.89:0/564813961 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a740a8070 con 0x7f4a68004d10 2023-10-30T12:32:52.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.246+0000 7f4a80eb7640 1 -- 172.21.15.89:0/564813961 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a740ad9f0 con 0x7f4a68004d10 2023-10-30T12:32:52.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.246+0000 7f4a8411d640 1 -- 172.21.15.89:0/564813961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a68004d10 msgr2=0x7f4a68005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:52.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.246+0000 7f4a8411d640 1 --2- 172.21.15.89:0/564813961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a68004d10 0x7f4a68005190 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4a7400a860 tx=0x7f4a7401d360 comp rx=0 tx=0).stop 2023-10-30T12:32:52.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.246+0000 7f4a826ba640 1 -- 172.21.15.89:0/564813961 reap_dead start 2023-10-30T12:32:52.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.246+0000 7f4a8411d640 1 -- 172.21.15.89:0/564813961 shutdown_connections 2023-10-30T12:32:52.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.246+0000 7f4a8411d640 1 -- 172.21.15.89:0/564813961 >> 172.21.15.89:0/564813961 conn(0x7f4a7c06c6e0 msgr2=0x7f4a7c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:52.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.246+0000 7f4a8411d640 1 -- 172.21.15.89:0/564813961 shutdown_connections 2023-10-30T12:32:52.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.246+0000 7f4a8411d640 1 -- 172.21.15.89:0/564813961 wait complete. 2023-10-30T12:32:52.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.250+0000 7f4a8411d640 1 Processor -- start 2023-10-30T12:32:52.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.250+0000 7f4a8411d640 1 -- start start 2023-10-30T12:32:52.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.250+0000 7f4a8411d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a68004d10 0x7f4a7c152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:52.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.250+0000 7f4a8411d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a7c14dea0 0x7f4a7c155e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:52.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.250+0000 7f4a8411d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a7c156350 0x7f4a7c0c3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:52.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.250+0000 7f4a8411d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7c155980 con 0x7f4a7c14dea0 2023-10-30T12:32:52.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.250+0000 7f4a8411d640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7c1553a0 con 0x7f4a68004d10 2023-10-30T12:32:52.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.250+0000 7f4a8411d640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7c1567d0 con 0x7f4a7c156350 2023-10-30T12:32:52.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.250+0000 7f4a816b8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a7c14dea0 0x7f4a7c155e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:52.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.250+0000 7f4a816b8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a7c14dea0 0x7f4a7c155e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48578/0 (socket says 172.21.15.89:48578) 2023-10-30T12:32:52.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.250+0000 7f4a816b8640 1 -- 172.21.15.89:0/913979155 learned_addr learned my addr 172.21.15.89:0/913979155 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:32:52.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.250+0000 7f4a81eb9640 1 --2- 172.21.15.89:0/913979155 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a68004d10 0x7f4a7c152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:52.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.250+0000 7f4a826ba640 1 --2- 172.21.15.89:0/913979155 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a7c156350 0x7f4a7c0c3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:52.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.250+0000 7f4a816b8640 1 -- 172.21.15.89:0/913979155 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a7c156350 msgr2=0x7f4a7c0c3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:52.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.250+0000 7f4a816b8640 1 --2- 172.21.15.89:0/913979155 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a7c156350 0x7f4a7c0c3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:52.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.250+0000 7f4a816b8640 1 -- 172.21.15.89:0/913979155 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a68004d10 msgr2=0x7f4a7c152a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:52.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.250+0000 7f4a816b8640 1 --2- 172.21.15.89:0/913979155 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a68004d10 0x7f4a7c152a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:52.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.250+0000 7f4a816b8640 1 -- 172.21.15.89:0/913979155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a7c1f2d10 con 0x7f4a7c14dea0 2023-10-30T12:32:52.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.250+0000 7f4a816b8640 1 --2- 172.21.15.89:0/913979155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a7c14dea0 0x7f4a7c155e10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c0115f0 tx=0x7f4a6c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:52.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.250+0000 7f4a727fc640 1 -- 172.21.15.89:0/913979155 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c004220 con 0x7f4a7c14dea0 2023-10-30T12:32:52.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.250+0000 7f4a727fc640 1 -- 172.21.15.89:0/913979155 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a6c004380 con 0x7f4a7c14dea0 2023-10-30T12:32:52.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.250+0000 7f4a8411d640 1 -- 172.21.15.89:0/913979155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a7c1f2fb0 con 0x7f4a7c14dea0 2023-10-30T12:32:52.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.250+0000 7f4a727fc640 1 -- 172.21.15.89:0/913979155 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c00f8f0 con 0x7f4a7c14dea0 2023-10-30T12:32:52.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.250+0000 7f4a8411d640 1 -- 172.21.15.89:0/913979155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a7c0c4f10 con 0x7f4a7c14dea0 2023-10-30T12:32:52.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.254+0000 7f4a8411d640 1 -- 172.21.15.89:0/913979155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a7c04e170 con 0x7f4a7c14dea0 2023-10-30T12:32:52.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.254+0000 7f4a727fc640 1 -- 172.21.15.89:0/913979155 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4a6c00fa50 con 0x7f4a7c14dea0 2023-10-30T12:32:52.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.254+0000 7f4a727fc640 1 --2- 172.21.15.89:0/913979155 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4a600664a0 0x7f4a60068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:52.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.254+0000 7f4a727fc640 1 -- 172.21.15.89:0/913979155 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5890+0+0 (secure 0 0 0) 0x7f4a6c0842c0 con 0x7f4a7c14dea0 2023-10-30T12:32:52.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.254+0000 7f4a81eb9640 1 --2- 172.21.15.89:0/913979155 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4a600664a0 0x7f4a60068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:52.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.254+0000 7f4a81eb9640 1 --2- 172.21.15.89:0/913979155 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4a600664a0 0x7f4a60068960 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4a78009b20 tx=0x7f4a78008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:52.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.258+0000 7f4a727fc640 1 -- 172.21.15.89:0/913979155 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4a6c051560 con 0x7f4a7c14dea0 2023-10-30T12:32:52.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.438+0000 7f4a8411d640 1 -- 172.21.15.89:0/913979155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a7c06fe90 con 0x7f4a7c14dea0 2023-10-30T12:32:52.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.438+0000 7f4a727fc640 1 -- 172.21.15.89:0/913979155 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6534 (secure 0 0 0) 0x7f4a6c054b80 con 0x7f4a7c14dea0 2023-10-30T12:32:52.444 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:32:52.445 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4781,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:32:04.109710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":2,"incarnation":15,"state":"up:active","state_seq":50,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:32:52.445 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 23 2023-10-30T12:32:52.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.442+0000 7f4a8411d640 1 -- 172.21.15.89:0/913979155 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4a600664a0 msgr2=0x7f4a60068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:52.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.442+0000 7f4a8411d640 1 --2- 172.21.15.89:0/913979155 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4a600664a0 0x7f4a60068960 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4a78009b20 tx=0x7f4a78008040 comp rx=0 tx=0).stop 2023-10-30T12:32:52.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.442+0000 7f4a8411d640 1 -- 172.21.15.89:0/913979155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a7c14dea0 msgr2=0x7f4a7c155e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:52.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.442+0000 7f4a8411d640 1 --2- 172.21.15.89:0/913979155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a7c14dea0 0x7f4a7c155e10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c0115f0 tx=0x7f4a6c011ab0 comp rx=0 tx=0).stop 2023-10-30T12:32:52.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.442+0000 7f4a8411d640 1 -- 172.21.15.89:0/913979155 shutdown_connections 2023-10-30T12:32:52.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.442+0000 7f4a8411d640 1 --2- 172.21.15.89:0/913979155 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4a600664a0 0x7f4a60068960 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:52.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.442+0000 7f4a8411d640 1 --2- 172.21.15.89:0/913979155 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a7c156350 0x7f4a7c0c3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:52.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.442+0000 7f4a8411d640 1 --2- 172.21.15.89:0/913979155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a7c14dea0 0x7f4a7c155e10 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:52.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.442+0000 7f4a8411d640 1 --2- 172.21.15.89:0/913979155 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a68004d10 0x7f4a7c152a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:52.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.442+0000 7f4a8411d640 1 -- 172.21.15.89:0/913979155 >> 172.21.15.89:0/913979155 conn(0x7f4a7c06c6e0 msgr2=0x7f4a7c070520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:52.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.442+0000 7f4a8411d640 1 -- 172.21.15.89:0/913979155 shutdown_connections 2023-10-30T12:32:52.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.442+0000 7f4a8411d640 1 -- 172.21.15.89:0/913979155 wait complete. 2023-10-30T12:32:52.465 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=2) 2023-10-30T12:32:52.466 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T12:32:52.466 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:32:52.483 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T12:32:52.484 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:32:52.661 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.658+0000 7fb65269a640 1 Processor -- start 2023-10-30T12:32:52.661 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.658+0000 7fb65269a640 1 -- start start 2023-10-30T12:32:52.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.658+0000 7fb65269a640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb64c153d70 0x7fb64c06ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:52.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.658+0000 7fb65269a640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb64c06e470 0x7fb64c0cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:52.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.658+0000 7fb65269a640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb64c0c8490 0x7fb64c0cd340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:52.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.658+0000 7fb65269a640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb64c06d780 con 0x7fb64c153d70 2023-10-30T12:32:52.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.658+0000 7fb65269a640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb64c06afa0 con 0x7fb64c0c8490 2023-10-30T12:32:52.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.658+0000 7fb65269a640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb64c154150 con 0x7fb64c06e470 2023-10-30T12:32:52.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.658+0000 7fb64bfff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb64c153d70 0x7fb64c06ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:52.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.658+0000 7fb64bfff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb64c153d70 0x7fb64c06ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48580/0 (socket says 172.21.15.89:48580) 2023-10-30T12:32:52.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.658+0000 7fb64bfff640 1 -- 172.21.15.89:0/1140633262 learned_addr learned my addr 172.21.15.89:0/1140633262 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:32:52.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.658+0000 7fb650c37640 1 --2- 172.21.15.89:0/1140633262 >> v2:172.21.15.192:3300/0 conn(0x7fb64c0c8490 0x7fb64c0cd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:52.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.658+0000 7fb64b7fe640 1 --2- 172.21.15.89:0/1140633262 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb64c06e470 0x7fb64c0cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:52.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.658+0000 7fb64bfff640 1 -- 172.21.15.89:0/1140633262 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb64c06e470 msgr2=0x7fb64c0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:52.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.658+0000 7fb64bfff640 1 --2- 172.21.15.89:0/1140633262 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb64c06e470 0x7fb64c0cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:52.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.658+0000 7fb64bfff640 1 -- 172.21.15.89:0/1140633262 >> v2:172.21.15.192:3300/0 conn(0x7fb64c0c8490 msgr2=0x7fb64c0cd340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:52.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.658+0000 7fb64bfff640 1 --2- 172.21.15.89:0/1140633262 >> v2:172.21.15.192:3300/0 conn(0x7fb64c0c8490 0x7fb64c0cd340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:52.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.658+0000 7fb64bfff640 1 -- 172.21.15.89:0/1140633262 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb64c06e030 con 0x7fb64c153d70 2023-10-30T12:32:52.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.658+0000 7fb650c37640 1 --2- 172.21.15.89:0/1140633262 >> v2:172.21.15.192:3300/0 conn(0x7fb64c0c8490 0x7fb64c0cd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:32:52.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.662+0000 7fb64bfff640 1 --2- 172.21.15.89:0/1140633262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb64c153d70 0x7fb64c06ca80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb64c0b1c40 tx=0x7fb63c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7727b2b2c1ed0336 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:52.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.662+0000 7fb64affd640 1 -- 172.21.15.89:0/1140633262 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63c0aa5f0 con 0x7fb64c153d70 2023-10-30T12:32:52.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.662+0000 7fb64affd640 1 -- 172.21.15.89:0/1140633262 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb63c0aa750 con 0x7fb64c153d70 2023-10-30T12:32:52.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.662+0000 7fb64affd640 1 -- 172.21.15.89:0/1140633262 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63c0aaae0 con 0x7fb64c153d70 2023-10-30T12:32:52.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.662+0000 7fb65269a640 1 -- 172.21.15.89:0/1140633262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb64c153d70 msgr2=0x7fb64c06ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:52.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.662+0000 7fb65269a640 1 --2- 172.21.15.89:0/1140633262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb64c153d70 0x7fb64c06ca80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb64c0b1c40 tx=0x7fb63c01c4d0 comp rx=0 tx=0).stop 2023-10-30T12:32:52.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.662+0000 7fb65269a640 1 -- 172.21.15.89:0/1140633262 shutdown_connections 2023-10-30T12:32:52.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.662+0000 7fb65269a640 1 --2- 172.21.15.89:0/1140633262 >> v2:172.21.15.192:3300/0 conn(0x7fb64c0c8490 0x7fb64c0cd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:52.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.662+0000 7fb65269a640 1 --2- 172.21.15.89:0/1140633262 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb64c06e470 0x7fb64c0cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:52.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.662+0000 7fb65269a640 1 --2- 172.21.15.89:0/1140633262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb64c153d70 0x7fb64c06ca80 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:52.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.662+0000 7fb65269a640 1 -- 172.21.15.89:0/1140633262 >> 172.21.15.89:0/1140633262 conn(0x7fb64c071ea0 msgr2=0x7fb64c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:52.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.662+0000 7fb65269a640 1 -- 172.21.15.89:0/1140633262 shutdown_connections 2023-10-30T12:32:52.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.662+0000 7fb65269a640 1 -- 172.21.15.89:0/1140633262 wait complete. 2023-10-30T12:32:52.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.666+0000 7fb65269a640 1 Processor -- start 2023-10-30T12:32:52.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.666+0000 7fb65269a640 1 -- start start 2023-10-30T12:32:52.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.666+0000 7fb65269a640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb64c06e470 0x7fb64c15bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:52.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.666+0000 7fb65269a640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb64c0c8490 0x7fb64c15b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:52.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.666+0000 7fb65269a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb64c153d70 0x7fb64c155150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:52.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.666+0000 7fb65269a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb64c156360 con 0x7fb64c153d70 2023-10-30T12:32:52.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.666+0000 7fb65269a640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb64c154cc0 con 0x7fb64c06e470 2023-10-30T12:32:52.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.666+0000 7fb65269a640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb64c1543d0 con 0x7fb64c0c8490 2023-10-30T12:32:52.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.666+0000 7fb650c37640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb64c153d70 0x7fb64c155150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:52.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.666+0000 7fb650c37640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb64c153d70 0x7fb64c155150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48592/0 (socket says 172.21.15.89:48592) 2023-10-30T12:32:52.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.666+0000 7fb650c37640 1 -- 172.21.15.89:0/3931262053 learned_addr learned my addr 172.21.15.89:0/3931262053 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:32:52.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.666+0000 7fb64bfff640 1 --2- 172.21.15.89:0/3931262053 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb64c06e470 0x7fb64c15bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:52.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.666+0000 7fb64b7fe640 1 --2- 172.21.15.89:0/3931262053 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb64c0c8490 0x7fb64c15b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:52.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.666+0000 7fb650c37640 1 -- 172.21.15.89:0/3931262053 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb64c0c8490 msgr2=0x7fb64c15b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:52.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.666+0000 7fb650c37640 1 --2- 172.21.15.89:0/3931262053 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb64c0c8490 0x7fb64c15b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:52.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.666+0000 7fb650c37640 1 -- 172.21.15.89:0/3931262053 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb64c06e470 msgr2=0x7fb64c15bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:52.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.666+0000 7fb650c37640 1 --2- 172.21.15.89:0/3931262053 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb64c06e470 0x7fb64c15bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:52.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.666+0000 7fb650c37640 1 -- 172.21.15.89:0/3931262053 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb63c0094d0 con 0x7fb64c153d70 2023-10-30T12:32:52.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.666+0000 7fb64bfff640 1 --2- 172.21.15.89:0/3931262053 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb64c06e470 0x7fb64c15bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:32:52.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.666+0000 7fb650c37640 1 --2- 172.21.15.89:0/3931262053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb64c153d70 0x7fb64c155150 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb644009fa0 tx=0x7fb64400d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:52.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.666+0000 7fb648ff9640 1 -- 172.21.15.89:0/3931262053 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb644003ba0 con 0x7fb64c153d70 2023-10-30T12:32:52.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.666+0000 7fb648ff9640 1 -- 172.21.15.89:0/3931262053 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb64400f040 con 0x7fb64c153d70 2023-10-30T12:32:52.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.666+0000 7fb648ff9640 1 -- 172.21.15.89:0/3931262053 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb644004030 con 0x7fb64c153d70 2023-10-30T12:32:52.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.666+0000 7fb65269a640 1 -- 172.21.15.89:0/3931262053 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb64c1f3260 con 0x7fb64c153d70 2023-10-30T12:32:52.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.666+0000 7fb65269a640 1 -- 172.21.15.89:0/3931262053 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb64c154950 con 0x7fb64c153d70 2023-10-30T12:32:52.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.666+0000 7fb65269a640 1 -- 172.21.15.89:0/3931262053 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb64c154e70 con 0x7fb64c153d70 2023-10-30T12:32:52.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.670+0000 7fb648ff9640 1 -- 172.21.15.89:0/3931262053 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb644004190 con 0x7fb64c153d70 2023-10-30T12:32:52.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.670+0000 7fb648ff9640 1 --2- 172.21.15.89:0/3931262053 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb6380664a0 0x7fb638068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:52.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.670+0000 7fb648ff9640 1 -- 172.21.15.89:0/3931262053 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5890+0+0 (secure 0 0 0) 0x7fb644084280 con 0x7fb64c153d70 2023-10-30T12:32:52.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.670+0000 7fb64bfff640 1 --2- 172.21.15.89:0/3931262053 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb6380664a0 0x7fb638068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:52.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.670+0000 7fb64bfff640 1 --2- 172.21.15.89:0/3931262053 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb6380664a0 0x7fb638068960 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb63c0ae000 tx=0x7fb63c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:52.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.674+0000 7fb648ff9640 1 -- 172.21.15.89:0/3931262053 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb64c154e70 con 0x7fb64c153d70 2023-10-30T12:32:52.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.854+0000 7fb65269a640 1 -- 172.21.15.89:0/3931262053 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb64c154e70 con 0x7fb64c153d70 2023-10-30T12:32:52.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.854+0000 7fb648ff9640 1 -- 172.21.15.89:0/3931262053 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6534 (secure 0 0 0) 0x7fb64c154e70 con 0x7fb64c153d70 2023-10-30T12:32:52.860 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:32:52.860 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4781,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:32:04.109710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":2,"incarnation":15,"state":"up:active","state_seq":50,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:32:52.861 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 23 2023-10-30T12:32:52.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.858+0000 7fb65269a640 1 -- 172.21.15.89:0/3931262053 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb6380664a0 msgr2=0x7fb638068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:52.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.858+0000 7fb65269a640 1 --2- 172.21.15.89:0/3931262053 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb6380664a0 0x7fb638068960 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb63c0ae000 tx=0x7fb63c0a7000 comp rx=0 tx=0).stop 2023-10-30T12:32:52.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.858+0000 7fb65269a640 1 -- 172.21.15.89:0/3931262053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb64c153d70 msgr2=0x7fb64c155150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:52.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.858+0000 7fb65269a640 1 --2- 172.21.15.89:0/3931262053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb64c153d70 0x7fb64c155150 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb644009fa0 tx=0x7fb64400d560 comp rx=0 tx=0).stop 2023-10-30T12:32:52.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.858+0000 7fb65269a640 1 -- 172.21.15.89:0/3931262053 shutdown_connections 2023-10-30T12:32:52.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.858+0000 7fb65269a640 1 --2- 172.21.15.89:0/3931262053 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb6380664a0 0x7fb638068960 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:52.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.858+0000 7fb65269a640 1 --2- 172.21.15.89:0/3931262053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb64c153d70 0x7fb64c155150 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:52.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.858+0000 7fb65269a640 1 --2- 172.21.15.89:0/3931262053 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb64c0c8490 0x7fb64c15b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:52.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.858+0000 7fb65269a640 1 --2- 172.21.15.89:0/3931262053 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb64c06e470 0x7fb64c15bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:52.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.858+0000 7fb65269a640 1 -- 172.21.15.89:0/3931262053 >> 172.21.15.89:0/3931262053 conn(0x7fb64c071ea0 msgr2=0x7fb64c149f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:52.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.858+0000 7fb65269a640 1 -- 172.21.15.89:0/3931262053 shutdown_connections 2023-10-30T12:32:52.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:52.858+0000 7fb65269a640 1 -- 172.21.15.89:0/3931262053 wait complete. 2023-10-30T12:32:52.880 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:32:54.882 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:32:55.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.026+0000 7fa9b3654640 1 Processor -- start 2023-10-30T12:32:55.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.026+0000 7fa9b3654640 1 -- start start 2023-10-30T12:32:55.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.026+0000 7fa9b3654640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa9ac0c05d0 0x7fa9ac13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:55.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.026+0000 7fa9b3654640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa9ac0c6f60 0x7fa9ac13f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:55.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.026+0000 7fa9b3654640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9ac0c0a50 con 0x7fa9ac0c6b80 2023-10-30T12:32:55.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.026+0000 7fa9b3654640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9ac06f6e0 con 0x7fa9ac0c6f60 2023-10-30T12:32:55.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.026+0000 7fa9b3654640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9ac06f9c0 con 0x7fa9ac0c05d0 2023-10-30T12:32:55.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.026+0000 7fa9b1bf1640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa9ac0c6f60 0x7fa9ac13f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:55.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.026+0000 7fa9b13f0640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa9ac0c6b80 0x7fa9ac0bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:32856/0 (socket says 172.21.15.89:32856) 2023-10-30T12:32:55.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.026+0000 7fa9b13f0640 1 -- 172.21.15.89:0/3901130112 learned_addr learned my addr 172.21.15.89:0/3901130112 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:32:55.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.026+0000 7fa9b0bef640 1 --2- 172.21.15.89:0/3901130112 >> v2:172.21.15.192:3300/0 conn(0x7fa9ac0c05d0 0x7fa9ac13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:55.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.026+0000 7fa9b1bf1640 1 -- 172.21.15.89:0/3901130112 >> v2:172.21.15.192:3300/0 conn(0x7fa9ac0c05d0 msgr2=0x7fa9ac13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:55.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.026+0000 7fa9b1bf1640 1 --2- 172.21.15.89:0/3901130112 >> v2:172.21.15.192:3300/0 conn(0x7fa9ac0c05d0 0x7fa9ac13ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:55.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.026+0000 7fa9b1bf1640 1 -- 172.21.15.89:0/3901130112 >> v1:172.21.15.89:6789/0 conn(0x7fa9ac0c6b80 legacy=0x7fa9ac0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:55.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.030+0000 7fa9b1bf1640 1 -- 172.21.15.89:0/3901130112 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ac06fca0 con 0x7fa9ac0c6f60 2023-10-30T12:32:55.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.026+0000 7fa9a3fff640 1 -- 172.21.15.89:0/3901130112 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2172839191 0 0) 0x7fa9ac0c0a50 con 0x7fa9ac0c6b80 2023-10-30T12:32:55.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.030+0000 7fa9b1bf1640 1 --2- 172.21.15.89:0/3901130112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9ac0c6f60 0x7fa9ac13f2d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac06c7e0 tx=0x7fa9a401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b763bd8b0cd5b47b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:55.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.030+0000 7fa9a3fff640 1 -- 172.21.15.89:0/3901130112 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a40a58e0 con 0x7fa9ac0c6f60 2023-10-30T12:32:55.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.030+0000 7fa9a3fff640 1 -- 172.21.15.89:0/3901130112 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a40a5a40 con 0x7fa9ac0c6f60 2023-10-30T12:32:55.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.030+0000 7fa9a3fff640 1 -- 172.21.15.89:0/3901130112 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a40a5dd0 con 0x7fa9ac0c6f60 2023-10-30T12:32:55.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.030+0000 7fa9b3654640 1 -- 172.21.15.89:0/3901130112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9ac0c6f60 msgr2=0x7fa9ac13f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:55.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.030+0000 7fa9b3654640 1 --2- 172.21.15.89:0/3901130112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9ac0c6f60 0x7fa9ac13f2d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac06c7e0 tx=0x7fa9a401c910 comp rx=0 tx=0).stop 2023-10-30T12:32:55.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.030+0000 7fa9b3654640 1 -- 172.21.15.89:0/3901130112 shutdown_connections 2023-10-30T12:32:55.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.030+0000 7fa9b3654640 1 --2- 172.21.15.89:0/3901130112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9ac0c6f60 0x7fa9ac13f2d0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:55.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.030+0000 7fa9b3654640 1 --2- 172.21.15.89:0/3901130112 >> v2:172.21.15.192:3300/0 conn(0x7fa9ac0c05d0 0x7fa9ac13ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:55.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.030+0000 7fa9b3654640 1 -- 172.21.15.89:0/3901130112 >> 172.21.15.89:0/3901130112 conn(0x7fa9ac0c7910 msgr2=0x7fa9ac0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:55.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.034+0000 7fa9b3654640 1 -- 172.21.15.89:0/3901130112 shutdown_connections 2023-10-30T12:32:55.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.034+0000 7fa9b3654640 1 -- 172.21.15.89:0/3901130112 wait complete. 2023-10-30T12:32:55.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.034+0000 7fa9b3654640 1 Processor -- start 2023-10-30T12:32:55.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.034+0000 7fa9b3654640 1 -- start start 2023-10-30T12:32:55.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.034+0000 7fa9b3654640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9ac0c05d0 0x7fa9ac13da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:55.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.034+0000 7fa9b3654640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9ac0c6b80 0x7fa9ac150150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:55.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.034+0000 7fa9b3654640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9ac0c6f60 0x7fa9ac151240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:55.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.034+0000 7fa9b3654640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac14bd40 con 0x7fa9ac0c6f60 2023-10-30T12:32:55.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.034+0000 7fa9b3654640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac1604b0 con 0x7fa9ac0c05d0 2023-10-30T12:32:55.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.034+0000 7fa9b3654640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac160790 con 0x7fa9ac0c6b80 2023-10-30T12:32:55.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.034+0000 7fa9b1bf1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9ac0c6f60 0x7fa9ac151240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:55.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.034+0000 7fa9b1bf1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9ac0c6f60 0x7fa9ac151240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48618/0 (socket says 172.21.15.89:48618) 2023-10-30T12:32:55.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.034+0000 7fa9b1bf1640 1 -- 172.21.15.89:0/4108713484 learned_addr learned my addr 172.21.15.89:0/4108713484 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:32:55.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.034+0000 7fa9b13f0640 1 --2- 172.21.15.89:0/4108713484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9ac0c05d0 0x7fa9ac13da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:55.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.034+0000 7fa9b0bef640 1 --2- 172.21.15.89:0/4108713484 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9ac0c6b80 0x7fa9ac150150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:55.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.034+0000 7fa9b1bf1640 1 -- 172.21.15.89:0/4108713484 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9ac0c6b80 msgr2=0x7fa9ac150150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:55.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.034+0000 7fa9b1bf1640 1 --2- 172.21.15.89:0/4108713484 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9ac0c6b80 0x7fa9ac150150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:55.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.034+0000 7fa9b1bf1640 1 -- 172.21.15.89:0/4108713484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9ac0c05d0 msgr2=0x7fa9ac13da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:55.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.034+0000 7fa9b1bf1640 1 --2- 172.21.15.89:0/4108713484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9ac0c05d0 0x7fa9ac13da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:55.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.034+0000 7fa9b1bf1640 1 -- 172.21.15.89:0/4108713484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a4009a70 con 0x7fa9ac0c6f60 2023-10-30T12:32:55.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.034+0000 7fa9b1bf1640 1 --2- 172.21.15.89:0/4108713484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9ac0c6f60 0x7fa9ac151240 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa9a40b0040 tx=0x7fa9a40196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:55.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.034+0000 7fa9a1ffb640 1 -- 172.21.15.89:0/4108713484 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a40ad9d0 con 0x7fa9ac0c6f60 2023-10-30T12:32:55.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.034+0000 7fa9a1ffb640 1 -- 172.21.15.89:0/4108713484 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a40adb30 con 0x7fa9ac0c6f60 2023-10-30T12:32:55.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.034+0000 7fa9a1ffb640 1 -- 172.21.15.89:0/4108713484 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a40b9450 con 0x7fa9ac0c6f60 2023-10-30T12:32:55.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.034+0000 7fa9b3654640 1 -- 172.21.15.89:0/4108713484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9ac160a70 con 0x7fa9ac0c6f60 2023-10-30T12:32:55.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.038+0000 7fa9b3654640 1 -- 172.21.15.89:0/4108713484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9ac13d4e0 con 0x7fa9ac0c6f60 2023-10-30T12:32:55.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.038+0000 7fa9b3654640 1 -- 172.21.15.89:0/4108713484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9ac04e170 con 0x7fa9ac0c6f60 2023-10-30T12:32:55.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.038+0000 7fa9a1ffb640 1 -- 172.21.15.89:0/4108713484 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa9a40b95b0 con 0x7fa9ac0c6f60 2023-10-30T12:32:55.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.038+0000 7fa9a1ffb640 1 --2- 172.21.15.89:0/4108713484 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa9900664a0 0x7fa990068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:55.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.038+0000 7fa9a1ffb640 1 -- 172.21.15.89:0/4108713484 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5890+0+0 (secure 0 0 0) 0x7fa9a411f6c0 con 0x7fa9ac0c6f60 2023-10-30T12:32:55.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.038+0000 7fa9b13f0640 1 --2- 172.21.15.89:0/4108713484 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa9900664a0 0x7fa990068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:55.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.042+0000 7fa9b13f0640 1 --2- 172.21.15.89:0/4108713484 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa9900664a0 0x7fa990068960 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8000f70 tx=0x7fa9a8007b80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:55.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.042+0000 7fa9a1ffb640 1 -- 172.21.15.89:0/4108713484 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa9a40ec950 con 0x7fa9ac0c6f60 2023-10-30T12:32:55.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.222+0000 7fa9b3654640 1 -- 172.21.15.89:0/4108713484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9ac13d6d0 con 0x7fa9ac0c6f60 2023-10-30T12:32:55.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.222+0000 7fa9a1ffb640 1 -- 172.21.15.89:0/4108713484 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6534 (secure 0 0 0) 0x7fa9a40eff70 con 0x7fa9ac0c6f60 2023-10-30T12:32:55.228 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:32:55.229 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4781,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:32:04.109710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":2,"incarnation":15,"state":"up:active","state_seq":50,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:32:55.229 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 23 2023-10-30T12:32:55.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.226+0000 7fa9b3654640 1 -- 172.21.15.89:0/4108713484 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa9900664a0 msgr2=0x7fa990068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:55.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.226+0000 7fa9b3654640 1 --2- 172.21.15.89:0/4108713484 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa9900664a0 0x7fa990068960 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8000f70 tx=0x7fa9a8007b80 comp rx=0 tx=0).stop 2023-10-30T12:32:55.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.226+0000 7fa9b3654640 1 -- 172.21.15.89:0/4108713484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9ac0c6f60 msgr2=0x7fa9ac151240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:55.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.226+0000 7fa9b3654640 1 --2- 172.21.15.89:0/4108713484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9ac0c6f60 0x7fa9ac151240 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa9a40b0040 tx=0x7fa9a40196b0 comp rx=0 tx=0).stop 2023-10-30T12:32:55.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.226+0000 7fa9b3654640 1 -- 172.21.15.89:0/4108713484 shutdown_connections 2023-10-30T12:32:55.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.226+0000 7fa9b3654640 1 --2- 172.21.15.89:0/4108713484 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa9900664a0 0x7fa990068960 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:55.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.226+0000 7fa9b3654640 1 --2- 172.21.15.89:0/4108713484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9ac0c6f60 0x7fa9ac151240 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:55.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.226+0000 7fa9b3654640 1 --2- 172.21.15.89:0/4108713484 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9ac0c6b80 0x7fa9ac150150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:55.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.226+0000 7fa9b3654640 1 --2- 172.21.15.89:0/4108713484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9ac0c05d0 0x7fa9ac13da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:55.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.226+0000 7fa9b3654640 1 -- 172.21.15.89:0/4108713484 >> 172.21.15.89:0/4108713484 conn(0x7fa9ac0c7910 msgr2=0x7fa9ac134750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:55.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.226+0000 7fa9b3654640 1 -- 172.21.15.89:0/4108713484 shutdown_connections 2023-10-30T12:32:55.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:55.226+0000 7fa9b3654640 1 -- 172.21.15.89:0/4108713484 wait complete. 2023-10-30T12:32:55.249 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:32:57.250 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:32:57.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.390+0000 7f6781032640 1 Processor -- start 2023-10-30T12:32:57.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.390+0000 7f6781032640 1 -- start start 2023-10-30T12:32:57.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.394+0000 7f6781032640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f677c0c05d0 0x7f677c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:57.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.394+0000 7f6781032640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f677c0c6f60 0x7f677c13f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:57.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.394+0000 7f6781032640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f677c0c0a50 con 0x7f677c0c6b80 2023-10-30T12:32:57.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.394+0000 7f6781032640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f677c06f6e0 con 0x7f677c0c6f60 2023-10-30T12:32:57.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.394+0000 7f6781032640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f677c06f9c0 con 0x7f677c0c05d0 2023-10-30T12:32:57.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.394+0000 7f677ad9d640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f677c0c6b80 0x7f677c0bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:32866/0 (socket says 172.21.15.89:32866) 2023-10-30T12:32:57.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.394+0000 7f677ad9d640 1 -- 172.21.15.89:0/3054108897 learned_addr learned my addr 172.21.15.89:0/3054108897 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:32:57.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.394+0000 7f677b59e640 1 --2- 172.21.15.89:0/3054108897 >> v2:172.21.15.89:3300/0 conn(0x7f677c0c6f60 0x7f677c13f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:57.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.394+0000 7f6773fff640 1 --2- 172.21.15.89:0/3054108897 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f677c0c05d0 0x7f677c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:57.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.394+0000 7f677a59c640 1 -- 172.21.15.89:0/3054108897 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373559519 0 0) 0x7f677c0c0a50 con 0x7f677c0c6b80 2023-10-30T12:32:57.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.394+0000 7f677a59c640 1 -- 172.21.15.89:0/3054108897 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6768003530 con 0x7f677c0c6b80 2023-10-30T12:32:57.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.394+0000 7f677b59e640 1 -- 172.21.15.89:0/3054108897 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f677c0c05d0 msgr2=0x7f677c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:57.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.394+0000 7f677b59e640 1 --2- 172.21.15.89:0/3054108897 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f677c0c05d0 0x7f677c13ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:57.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.394+0000 7f677b59e640 1 -- 172.21.15.89:0/3054108897 >> v1:172.21.15.89:6789/0 conn(0x7f677c0c6b80 legacy=0x7f677c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:57.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.394+0000 7f677b59e640 1 -- 172.21.15.89:0/3054108897 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f677c06fca0 con 0x7f677c0c6f60 2023-10-30T12:32:57.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.394+0000 7f677b59e640 1 --2- 172.21.15.89:0/3054108897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f677c0c6f60 0x7f677c13f2d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6774009d90 tx=0x7f677401c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7651e9db9e5aad8f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:57.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.398+0000 7f677a59c640 1 -- 172.21.15.89:0/3054108897 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67740a5920 con 0x7f677c0c6f60 2023-10-30T12:32:57.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.398+0000 7f677a59c640 1 -- 172.21.15.89:0/3054108897 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67740a5a80 con 0x7f677c0c6f60 2023-10-30T12:32:57.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.398+0000 7f677a59c640 1 -- 172.21.15.89:0/3054108897 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67740a5e10 con 0x7f677c0c6f60 2023-10-30T12:32:57.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.398+0000 7f6781032640 1 -- 172.21.15.89:0/3054108897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f677c0c6f60 msgr2=0x7f677c13f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:57.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.398+0000 7f6781032640 1 --2- 172.21.15.89:0/3054108897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f677c0c6f60 0x7f677c13f2d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6774009d90 tx=0x7f677401c950 comp rx=0 tx=0).stop 2023-10-30T12:32:57.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.398+0000 7f6781032640 1 -- 172.21.15.89:0/3054108897 shutdown_connections 2023-10-30T12:32:57.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.398+0000 7f6781032640 1 --2- 172.21.15.89:0/3054108897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f677c0c6f60 0x7f677c13f2d0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:57.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.398+0000 7f6781032640 1 --2- 172.21.15.89:0/3054108897 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f677c0c05d0 0x7f677c13ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:57.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.398+0000 7f6781032640 1 -- 172.21.15.89:0/3054108897 >> 172.21.15.89:0/3054108897 conn(0x7f677c0c7910 msgr2=0x7f677c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:57.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.398+0000 7f6781032640 1 -- 172.21.15.89:0/3054108897 shutdown_connections 2023-10-30T12:32:57.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.398+0000 7f6781032640 1 -- 172.21.15.89:0/3054108897 wait complete. 2023-10-30T12:32:57.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.398+0000 7f6781032640 1 Processor -- start 2023-10-30T12:32:57.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.398+0000 7f6781032640 1 -- start start 2023-10-30T12:32:57.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.398+0000 7f6781032640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f677c0c05d0 0x7f677c1360b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:57.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.398+0000 7f6781032640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f677c0c6b80 0x7f677c155000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:57.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.398+0000 7f6781032640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f677c0c6f60 0x7f677c155540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:57.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.398+0000 7f6781032640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677c13ab60 con 0x7f677c0c6f60 2023-10-30T12:32:57.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.398+0000 7f6781032640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677c13a5a0 con 0x7f677c0c6b80 2023-10-30T12:32:57.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.398+0000 7f6781032640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f677c13a880 con 0x7f677c0c05d0 2023-10-30T12:32:57.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.398+0000 7f677b59e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f677c0c6f60 0x7f677c155540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:57.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.398+0000 7f677b59e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f677c0c6f60 0x7f677c155540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48634/0 (socket says 172.21.15.89:48634) 2023-10-30T12:32:57.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.398+0000 7f677b59e640 1 -- 172.21.15.89:0/3393774349 learned_addr learned my addr 172.21.15.89:0/3393774349 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:32:57.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.398+0000 7f677ad9d640 1 --2- 172.21.15.89:0/3393774349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f677c0c05d0 0x7f677c1360b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:57.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.398+0000 7f6773fff640 1 --2- 172.21.15.89:0/3393774349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f677c0c6b80 0x7f677c155000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:57.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.402+0000 7f677b59e640 1 -- 172.21.15.89:0/3393774349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f677c0c05d0 msgr2=0x7f677c1360b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:57.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.402+0000 7f677b59e640 1 --2- 172.21.15.89:0/3393774349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f677c0c05d0 0x7f677c1360b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:57.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.402+0000 7f677b59e640 1 -- 172.21.15.89:0/3393774349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f677c0c6b80 msgr2=0x7f677c155000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:57.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.402+0000 7f677b59e640 1 --2- 172.21.15.89:0/3393774349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f677c0c6b80 0x7f677c155000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:57.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.402+0000 7f677b59e640 1 -- 172.21.15.89:0/3393774349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6774009a70 con 0x7f677c0c6f60 2023-10-30T12:32:57.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.402+0000 7f677b59e640 1 --2- 172.21.15.89:0/3393774349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f677c0c6f60 0x7f677c155540 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f677401a580 tx=0x7f67740196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:57.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.402+0000 7f67737fe640 1 -- 172.21.15.89:0/3393774349 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67740a5920 con 0x7f677c0c6f60 2023-10-30T12:32:57.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.402+0000 7f67737fe640 1 -- 172.21.15.89:0/3393774349 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6774009450 con 0x7f677c0c6f60 2023-10-30T12:32:57.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.402+0000 7f67737fe640 1 -- 172.21.15.89:0/3393774349 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67740b7650 con 0x7f677c0c6f60 2023-10-30T12:32:57.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.402+0000 7f6781032640 1 -- 172.21.15.89:0/3393774349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f677c154520 con 0x7f677c0c6f60 2023-10-30T12:32:57.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.402+0000 7f6781032640 1 -- 172.21.15.89:0/3393774349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f677c154ae0 con 0x7f677c0c6f60 2023-10-30T12:32:57.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.402+0000 7f6781032640 1 -- 172.21.15.89:0/3393774349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f677c154cd0 con 0x7f677c0c6f60 2023-10-30T12:32:57.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.402+0000 7f67737fe640 1 -- 172.21.15.89:0/3393774349 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f67740b4050 con 0x7f677c0c6f60 2023-10-30T12:32:57.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.402+0000 7f67737fe640 1 --2- 172.21.15.89:0/3393774349 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f675c0664a0 0x7f675c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:57.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.402+0000 7f67737fe640 1 -- 172.21.15.89:0/3393774349 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5890+0+0 (secure 0 0 0) 0x7f677411f3c0 con 0x7f677c0c6f60 2023-10-30T12:32:57.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.406+0000 7f677ad9d640 1 --2- 172.21.15.89:0/3393774349 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f675c0664a0 0x7f675c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:57.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.406+0000 7f677ad9d640 1 --2- 172.21.15.89:0/3393774349 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f675c0664a0 0x7f675c068960 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f676c007fd0 tx=0x7f676c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:57.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.406+0000 7f67737fe640 1 -- 172.21.15.89:0/3393774349 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f67740ec940 con 0x7f677c0c6f60 2023-10-30T12:32:57.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.590+0000 7f6781032640 1 -- 172.21.15.89:0/3393774349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f677c04e170 con 0x7f677c0c6f60 2023-10-30T12:32:57.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.590+0000 7f67737fe640 1 -- 172.21.15.89:0/3393774349 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6534 (secure 0 0 0) 0x7f67740eff60 con 0x7f677c0c6f60 2023-10-30T12:32:57.594 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:32:57.594 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4781,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:32:04.109710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":2,"incarnation":15,"state":"up:active","state_seq":50,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:32:57.595 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 23 2023-10-30T12:32:57.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.590+0000 7f6781032640 1 -- 172.21.15.89:0/3393774349 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f675c0664a0 msgr2=0x7f675c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:57.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.590+0000 7f6781032640 1 --2- 172.21.15.89:0/3393774349 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f675c0664a0 0x7f675c068960 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f676c007fd0 tx=0x7f676c009040 comp rx=0 tx=0).stop 2023-10-30T12:32:57.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.590+0000 7f6781032640 1 -- 172.21.15.89:0/3393774349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f677c0c6f60 msgr2=0x7f677c155540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:57.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.590+0000 7f6781032640 1 --2- 172.21.15.89:0/3393774349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f677c0c6f60 0x7f677c155540 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f677401a580 tx=0x7f67740196f0 comp rx=0 tx=0).stop 2023-10-30T12:32:57.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.590+0000 7f6781032640 1 -- 172.21.15.89:0/3393774349 shutdown_connections 2023-10-30T12:32:57.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.590+0000 7f6781032640 1 --2- 172.21.15.89:0/3393774349 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f675c0664a0 0x7f675c068960 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:57.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.590+0000 7f6781032640 1 --2- 172.21.15.89:0/3393774349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f677c0c6b80 0x7f677c155000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:57.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.590+0000 7f6781032640 1 --2- 172.21.15.89:0/3393774349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f677c0c6f60 0x7f677c155540 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:57.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.590+0000 7f6781032640 1 --2- 172.21.15.89:0/3393774349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f677c0c05d0 0x7f677c1360b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:57.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.590+0000 7f6781032640 1 -- 172.21.15.89:0/3393774349 >> 172.21.15.89:0/3393774349 conn(0x7f677c0c7910 msgr2=0x7f677c0afd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:57.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.590+0000 7f6781032640 1 -- 172.21.15.89:0/3393774349 shutdown_connections 2023-10-30T12:32:57.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:57.590+0000 7f6781032640 1 -- 172.21.15.89:0/3393774349 wait complete. 2023-10-30T12:32:57.616 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:32:59.617 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:32:59.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.762+0000 7f1b44fa1640 1 Processor -- start 2023-10-30T12:32:59.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.762+0000 7f1b44fa1640 1 -- start start 2023-10-30T12:32:59.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.762+0000 7f1b44fa1640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1b400cf160 0x7f1b400cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:59.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.762+0000 7f1b44fa1640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b400cb800 con 0x7f1b401538a0 2023-10-30T12:32:59.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.762+0000 7f1b44fa1640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b400cb330 con 0x7f1b4015d6f0 2023-10-30T12:32:59.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.762+0000 7f1b44fa1640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b400cad70 con 0x7f1b400cf160 2023-10-30T12:32:59.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.762+0000 7f1b3e59c640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f1b401538a0 0x7f1b400ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:37162/0 (socket says 172.21.15.89:37162) 2023-10-30T12:32:59.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.762+0000 7f1b3e59c640 1 -- 172.21.15.89:0/3102008673 learned_addr learned my addr 172.21.15.89:0/3102008673 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:32:59.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.762+0000 7f1b3ed9d640 1 --2- 172.21.15.89:0/3102008673 >> v2:172.21.15.89:3300/0 conn(0x7f1b400cf160 0x7f1b400cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:59.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.762+0000 7f1b3dd9b640 1 -- 172.21.15.89:0/3102008673 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662456615 0 0) 0x7f1b400cb800 con 0x7f1b401538a0 2023-10-30T12:32:59.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.762+0000 7f1b3dd9b640 1 -- 172.21.15.89:0/3102008673 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b2c003530 con 0x7f1b401538a0 2023-10-30T12:32:59.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.762+0000 7f1b3ed9d640 1 -- 172.21.15.89:0/3102008673 >> v1:172.21.15.192:6789/0 conn(0x7f1b4015d6f0 legacy=0x7f1b4015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:59.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.762+0000 7f1b3ed9d640 1 -- 172.21.15.89:0/3102008673 >> v1:172.21.15.89:6789/0 conn(0x7f1b401538a0 legacy=0x7f1b400ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:59.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.762+0000 7f1b3ed9d640 1 -- 172.21.15.89:0/3102008673 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b400cb050 con 0x7f1b400cf160 2023-10-30T12:32:59.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.766+0000 7f1b3ed9d640 1 --2- 172.21.15.89:0/3102008673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b400cf160 0x7f1b400cf590 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1b38009d90 tx=0x7f1b3801f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5c3c020b5e9342d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:59.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.766+0000 7f1b3dd9b640 1 -- 172.21.15.89:0/3102008673 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b380af600 con 0x7f1b400cf160 2023-10-30T12:32:59.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.766+0000 7f1b3dd9b640 1 -- 172.21.15.89:0/3102008673 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b380ab050 con 0x7f1b400cf160 2023-10-30T12:32:59.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.766+0000 7f1b3dd9b640 1 -- 172.21.15.89:0/3102008673 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b380af980 con 0x7f1b400cf160 2023-10-30T12:32:59.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.766+0000 7f1b44fa1640 1 -- 172.21.15.89:0/3102008673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b400cf160 msgr2=0x7f1b400cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:59.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.766+0000 7f1b44fa1640 1 --2- 172.21.15.89:0/3102008673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b400cf160 0x7f1b400cf590 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1b38009d90 tx=0x7f1b3801f130 comp rx=0 tx=0).stop 2023-10-30T12:32:59.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.766+0000 7f1b44fa1640 1 -- 172.21.15.89:0/3102008673 shutdown_connections 2023-10-30T12:32:59.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.766+0000 7f1b44fa1640 1 --2- 172.21.15.89:0/3102008673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b400cf160 0x7f1b400cf590 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:59.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.766+0000 7f1b44fa1640 1 -- 172.21.15.89:0/3102008673 >> 172.21.15.89:0/3102008673 conn(0x7f1b4006bfe0 msgr2=0x7f1b4006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:59.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.766+0000 7f1b44fa1640 1 -- 172.21.15.89:0/3102008673 shutdown_connections 2023-10-30T12:32:59.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.766+0000 7f1b44fa1640 1 -- 172.21.15.89:0/3102008673 wait complete. 2023-10-30T12:32:59.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.770+0000 7f1b44fa1640 1 Processor -- start 2023-10-30T12:32:59.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.770+0000 7f1b44fa1640 1 -- start start 2023-10-30T12:32:59.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.770+0000 7f1b44fa1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b400cf160 0x7f1b400ce9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:59.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.770+0000 7f1b44fa1640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b401538a0 0x7f1b400b3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:59.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.770+0000 7f1b44fa1640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b4015d6f0 0x7f1b40148690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:59.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.770+0000 7f1b44fa1640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b40149a30 con 0x7f1b400cf160 2023-10-30T12:32:59.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.770+0000 7f1b44fa1640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b400ce3a0 con 0x7f1b4015d6f0 2023-10-30T12:32:59.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.770+0000 7f1b44fa1640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b40149cd0 con 0x7f1b401538a0 2023-10-30T12:32:59.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.770+0000 7f1b3e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b400cf160 0x7f1b400ce9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:59.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.770+0000 7f1b3e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b400cf160 0x7f1b400ce9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54628/0 (socket says 172.21.15.89:54628) 2023-10-30T12:32:59.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.770+0000 7f1b3e59c640 1 -- 172.21.15.89:0/887830597 learned_addr learned my addr 172.21.15.89:0/887830597 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:32:59.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.770+0000 7f1b3ed9d640 1 --2- 172.21.15.89:0/887830597 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b4015d6f0 0x7f1b40148690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:59.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.770+0000 7f1b35d9b640 1 --2- 172.21.15.89:0/887830597 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b401538a0 0x7f1b400b3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:59.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.770+0000 7f1b3e59c640 1 -- 172.21.15.89:0/887830597 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b401538a0 msgr2=0x7f1b400b3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:59.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.770+0000 7f1b3e59c640 1 --2- 172.21.15.89:0/887830597 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b401538a0 0x7f1b400b3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:59.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.770+0000 7f1b3e59c640 1 -- 172.21.15.89:0/887830597 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b4015d6f0 msgr2=0x7f1b40148690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:59.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.770+0000 7f1b3e59c640 1 --2- 172.21.15.89:0/887830597 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b4015d6f0 0x7f1b40148690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:59.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.770+0000 7f1b3e59c640 1 -- 172.21.15.89:0/887830597 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b38009a70 con 0x7f1b400cf160 2023-10-30T12:32:59.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.770+0000 7f1b35d9b640 1 --2- 172.21.15.89:0/887830597 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b401538a0 0x7f1b400b3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:32:59.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.770+0000 7f1b3e59c640 1 --2- 172.21.15.89:0/887830597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b400cf160 0x7f1b400ce9e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1b2800c4d0 tx=0x7f1b2800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:59.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.770+0000 7f1b377fe640 1 -- 172.21.15.89:0/887830597 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b28005cf0 con 0x7f1b400cf160 2023-10-30T12:32:59.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.770+0000 7f1b377fe640 1 -- 172.21.15.89:0/887830597 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b28005e50 con 0x7f1b400cf160 2023-10-30T12:32:59.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.770+0000 7f1b44fa1640 1 -- 172.21.15.89:0/887830597 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b40149fb0 con 0x7f1b400cf160 2023-10-30T12:32:59.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.770+0000 7f1b377fe640 1 -- 172.21.15.89:0/887830597 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b28007270 con 0x7f1b400cf160 2023-10-30T12:32:59.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.770+0000 7f1b44fa1640 1 -- 172.21.15.89:0/887830597 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b400ce680 con 0x7f1b400cf160 2023-10-30T12:32:59.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.774+0000 7f1b44fa1640 1 -- 172.21.15.89:0/887830597 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b4004e170 con 0x7f1b400cf160 2023-10-30T12:32:59.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.774+0000 7f1b377fe640 1 -- 172.21.15.89:0/887830597 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1b280073d0 con 0x7f1b400cf160 2023-10-30T12:32:59.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.774+0000 7f1b377fe640 1 --2- 172.21.15.89:0/887830597 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1b200664a0 0x7f1b20068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:59.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.774+0000 7f1b377fe640 1 -- 172.21.15.89:0/887830597 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5890+0+0 (secure 0 0 0) 0x7f1b280833f0 con 0x7f1b400cf160 2023-10-30T12:32:59.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.774+0000 7f1b35d9b640 1 --2- 172.21.15.89:0/887830597 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1b200664a0 0x7f1b20068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:59.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.774+0000 7f1b35d9b640 1 --2- 172.21.15.89:0/887830597 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1b200664a0 0x7f1b20068960 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1b400ce1b0 tx=0x7f1b30009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:59.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.778+0000 7f1b377fe640 1 -- 172.21.15.89:0/887830597 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1b28050860 con 0x7f1b400cf160 2023-10-30T12:32:59.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.966+0000 7f1b44fa1640 1 -- 172.21.15.89:0/887830597 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b400b1dd0 con 0x7f1b400cf160 2023-10-30T12:32:59.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.966+0000 7f1b377fe640 1 -- 172.21.15.89:0/887830597 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6534 (secure 0 0 0) 0x7f1b28053e80 con 0x7f1b400cf160 2023-10-30T12:32:59.970 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:32:59.970 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4781,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:32:04.109710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":2,"incarnation":15,"state":"up:active","state_seq":50,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:32:59.971 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 23 2023-10-30T12:32:59.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.966+0000 7f1b44fa1640 1 -- 172.21.15.89:0/887830597 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1b200664a0 msgr2=0x7f1b20068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:59.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.966+0000 7f1b44fa1640 1 --2- 172.21.15.89:0/887830597 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1b200664a0 0x7f1b20068960 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1b400ce1b0 tx=0x7f1b30009040 comp rx=0 tx=0).stop 2023-10-30T12:32:59.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.966+0000 7f1b44fa1640 1 -- 172.21.15.89:0/887830597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b400cf160 msgr2=0x7f1b400ce9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:59.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.966+0000 7f1b44fa1640 1 --2- 172.21.15.89:0/887830597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b400cf160 0x7f1b400ce9e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1b2800c4d0 tx=0x7f1b2800c990 comp rx=0 tx=0).stop 2023-10-30T12:32:59.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.966+0000 7f1b44fa1640 1 -- 172.21.15.89:0/887830597 shutdown_connections 2023-10-30T12:32:59.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.966+0000 7f1b44fa1640 1 --2- 172.21.15.89:0/887830597 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1b200664a0 0x7f1b20068960 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:59.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.966+0000 7f1b44fa1640 1 --2- 172.21.15.89:0/887830597 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b4015d6f0 0x7f1b40148690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:59.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.966+0000 7f1b44fa1640 1 --2- 172.21.15.89:0/887830597 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b401538a0 0x7f1b400b3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:59.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.966+0000 7f1b44fa1640 1 --2- 172.21.15.89:0/887830597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b400cf160 0x7f1b400ce9e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:59.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.966+0000 7f1b44fa1640 1 -- 172.21.15.89:0/887830597 >> 172.21.15.89:0/887830597 conn(0x7f1b4006bfe0 msgr2=0x7f1b400c5610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:59.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.966+0000 7f1b44fa1640 1 -- 172.21.15.89:0/887830597 shutdown_connections 2023-10-30T12:32:59.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:32:59.966+0000 7f1b44fa1640 1 -- 172.21.15.89:0/887830597 wait complete. 2023-10-30T12:32:59.992 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:33:01.993 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:33:02.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.134+0000 7f6172b0f640 1 Processor -- start 2023-10-30T12:33:02.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.134+0000 7f6172b0f640 1 -- start start 2023-10-30T12:33:02.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.134+0000 7f6172b0f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f616c0c6bd0 0x7f616c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:02.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.134+0000 7f6172b0f640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f616c066cf0 con 0x7f616c06f7a0 2023-10-30T12:33:02.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.138+0000 7f6172b0f640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f616c0c70d0 con 0x7f616c071110 2023-10-30T12:33:02.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.138+0000 7f6172b0f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f616c0700c0 con 0x7f616c0c6bd0 2023-10-30T12:33:02.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.138+0000 7f61708ab640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f616c06f7a0 0x7f616c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:37166/0 (socket says 172.21.15.89:37166) 2023-10-30T12:33:02.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.138+0000 7f61708ab640 1 -- 172.21.15.89:0/1205945624 learned_addr learned my addr 172.21.15.89:0/1205945624 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:33:02.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.138+0000 7f616bfff640 1 --2- 172.21.15.89:0/1205945624 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f616c0c6bd0 0x7f616c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:02.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.138+0000 7f616b7fe640 1 -- 172.21.15.89:0/1205945624 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176102818 0 0) 0x7f616c066cf0 con 0x7f616c06f7a0 2023-10-30T12:33:02.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.138+0000 7f616b7fe640 1 -- 172.21.15.89:0/1205945624 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f615c003530 con 0x7f616c06f7a0 2023-10-30T12:33:02.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.138+0000 7f616b7fe640 1 -- 172.21.15.89:0/1205945624 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3199901950 0 0) 0x7f615c003530 con 0x7f616c06f7a0 2023-10-30T12:33:02.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.138+0000 7f616b7fe640 1 -- 172.21.15.89:0/1205945624 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f616c0c6bd0 msgr2=0x7f616c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:02.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.138+0000 7f616b7fe640 1 --2- 172.21.15.89:0/1205945624 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f616c0c6bd0 0x7f616c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:02.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.138+0000 7f616b7fe640 1 -- 172.21.15.89:0/1205945624 >> v1:172.21.15.192:6789/0 conn(0x7f616c071110 legacy=0x7f616c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:02.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.138+0000 7f616b7fe640 1 -- 172.21.15.89:0/1205945624 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616c0c7390 con 0x7f616c06f7a0 2023-10-30T12:33:02.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.138+0000 7f616b7fe640 1 -- 172.21.15.89:0/1205945624 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f6160001680 con 0x7f616c06f7a0 2023-10-30T12:33:02.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.138+0000 7f616b7fe640 1 -- 172.21.15.89:0/1205945624 >> v1:172.21.15.89:6789/0 conn(0x7f616c06f7a0 legacy=0x7f616c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:02.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.138+0000 7f616b7fe640 1 --2- 172.21.15.89:0/1205945624 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f615c0039b0 0x7f615c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:02.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.138+0000 7f616b7fe640 1 --2- 172.21.15.89:0/1205945624 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f616c0c6bd0 0x7f615c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:02.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.138+0000 7f616b7fe640 1 --2- 172.21.15.89:0/1205945624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f615c004d10 0x7f615c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:02.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.138+0000 7f616b7fe640 1 -- 172.21.15.89:0/1205945624 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c0700c0 con 0x7f615c004d10 2023-10-30T12:33:02.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.138+0000 7f616b7fe640 1 -- 172.21.15.89:0/1205945624 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f615c00bad0 con 0x7f615c0039b0 2023-10-30T12:33:02.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.138+0000 7f616b7fe640 1 -- 172.21.15.89:0/1205945624 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f615c00bc90 con 0x7f616c0c6bd0 2023-10-30T12:33:02.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.138+0000 7f61710ac640 1 --2- 172.21.15.89:0/1205945624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f615c004d10 0x7f615c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:02.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.138+0000 7f61708ab640 1 --2- 172.21.15.89:0/1205945624 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f615c0039b0 0x7f615c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:02.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.138+0000 7f616bfff640 1 --2- 172.21.15.89:0/1205945624 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f616c0c6bd0 0x7f615c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:02.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.138+0000 7f61710ac640 1 -- 172.21.15.89:0/1205945624 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f616c0c6bd0 msgr2=0x7f615c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:02.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.138+0000 7f61710ac640 1 --2- 172.21.15.89:0/1205945624 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f616c0c6bd0 0x7f615c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:02.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.138+0000 7f61710ac640 1 -- 172.21.15.89:0/1205945624 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f615c0039b0 msgr2=0x7f615c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:02.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.138+0000 7f61710ac640 1 --2- 172.21.15.89:0/1205945624 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f615c0039b0 0x7f615c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:02.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.138+0000 7f61710ac640 1 -- 172.21.15.89:0/1205945624 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f615c003530 con 0x7f615c004d10 2023-10-30T12:33:02.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.142+0000 7f61710ac640 1 --2- 172.21.15.89:0/1205945624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f615c004d10 0x7f615c005190 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6164001e40 tx=0x7f616401c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=572419f029b6f101 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:02.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.142+0000 7f616b7fe640 1 -- 172.21.15.89:0/1205945624 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6164003b80 con 0x7f615c004d10 2023-10-30T12:33:02.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.142+0000 7f616b7fe640 1 -- 172.21.15.89:0/1205945624 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6164003ce0 con 0x7f615c004d10 2023-10-30T12:33:02.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.142+0000 7f616b7fe640 1 -- 172.21.15.89:0/1205945624 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61640052c0 con 0x7f615c004d10 2023-10-30T12:33:02.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.142+0000 7f6172b0f640 1 -- 172.21.15.89:0/1205945624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f615c004d10 msgr2=0x7f615c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:02.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.142+0000 7f6172b0f640 1 --2- 172.21.15.89:0/1205945624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f615c004d10 0x7f615c005190 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6164001e40 tx=0x7f616401c740 comp rx=0 tx=0).stop 2023-10-30T12:33:02.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.142+0000 7f61710ac640 1 -- 172.21.15.89:0/1205945624 reap_dead start 2023-10-30T12:33:02.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.142+0000 7f6172b0f640 1 -- 172.21.15.89:0/1205945624 shutdown_connections 2023-10-30T12:33:02.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.142+0000 7f6172b0f640 1 -- 172.21.15.89:0/1205945624 >> 172.21.15.89:0/1205945624 conn(0x7f616c0c7910 msgr2=0x7f616c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:02.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.142+0000 7f6172b0f640 1 -- 172.21.15.89:0/1205945624 shutdown_connections 2023-10-30T12:33:02.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.142+0000 7f6172b0f640 1 -- 172.21.15.89:0/1205945624 wait complete. 2023-10-30T12:33:02.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.142+0000 7f6172b0f640 1 Processor -- start 2023-10-30T12:33:02.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.142+0000 7f6172b0f640 1 -- start start 2023-10-30T12:33:02.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.146+0000 7f6172b0f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f615c004d10 0x7f616c0c1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:02.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.146+0000 7f6172b0f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f616c1ef760 0x7f616c0af770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:02.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.146+0000 7f6172b0f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f616c148cf0 0x7f616c1436d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:02.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.146+0000 7f6172b0f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c0af350 con 0x7f616c1ef760 2023-10-30T12:33:02.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.146+0000 7f6172b0f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c06c5d0 con 0x7f616c148cf0 2023-10-30T12:33:02.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.146+0000 7f6172b0f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f616c1efba0 con 0x7f615c004d10 2023-10-30T12:33:02.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.146+0000 7f616bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f616c1ef760 0x7f616c0af770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:02.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.146+0000 7f616bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f616c1ef760 0x7f616c0af770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54640/0 (socket says 172.21.15.89:54640) 2023-10-30T12:33:02.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.146+0000 7f616bfff640 1 -- 172.21.15.89:0/4229410733 learned_addr learned my addr 172.21.15.89:0/4229410733 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:33:02.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.146+0000 7f61708ab640 1 --2- 172.21.15.89:0/4229410733 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f615c004d10 0x7f616c0c1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:02.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.146+0000 7f61710ac640 1 --2- 172.21.15.89:0/4229410733 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f616c148cf0 0x7f616c1436d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:02.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.146+0000 7f616bfff640 1 -- 172.21.15.89:0/4229410733 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f615c004d10 msgr2=0x7f616c0c1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:02.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.146+0000 7f616bfff640 1 --2- 172.21.15.89:0/4229410733 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f615c004d10 0x7f616c0c1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:02.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.146+0000 7f616bfff640 1 -- 172.21.15.89:0/4229410733 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f616c148cf0 msgr2=0x7f616c1436d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:02.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.146+0000 7f616bfff640 1 --2- 172.21.15.89:0/4229410733 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f616c148cf0 0x7f616c1436d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:02.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.146+0000 7f616bfff640 1 -- 172.21.15.89:0/4229410733 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616c0aecb0 con 0x7f616c1ef760 2023-10-30T12:33:02.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.146+0000 7f616bfff640 1 --2- 172.21.15.89:0/4229410733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f616c1ef760 0x7f616c0af770 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f61580119d0 tx=0x7f6158011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:02.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.146+0000 7f61697fa640 1 -- 172.21.15.89:0/4229410733 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615800faf0 con 0x7f616c1ef760 2023-10-30T12:33:02.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.146+0000 7f61697fa640 1 -- 172.21.15.89:0/4229410733 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f615800fc50 con 0x7f616c1ef760 2023-10-30T12:33:02.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.146+0000 7f6172b0f640 1 -- 172.21.15.89:0/4229410733 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f616c0adc70 con 0x7f616c1ef760 2023-10-30T12:33:02.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.146+0000 7f61697fa640 1 -- 172.21.15.89:0/4229410733 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6158014440 con 0x7f616c1ef760 2023-10-30T12:33:02.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.146+0000 7f6172b0f640 1 -- 172.21.15.89:0/4229410733 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f616c0ae230 con 0x7f616c1ef760 2023-10-30T12:33:02.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.146+0000 7f6172b0f640 1 -- 172.21.15.89:0/4229410733 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f616c0af5c0 con 0x7f616c1ef760 2023-10-30T12:33:02.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.146+0000 7f61697fa640 1 -- 172.21.15.89:0/4229410733 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f615801d400 con 0x7f616c1ef760 2023-10-30T12:33:02.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.150+0000 7f61697fa640 1 --2- 172.21.15.89:0/4229410733 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f61500664a0 0x7f6150068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:02.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.150+0000 7f61697fa640 1 -- 172.21.15.89:0/4229410733 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5890+0+0 (secure 0 0 0) 0x7f6158084750 con 0x7f616c1ef760 2023-10-30T12:33:02.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.150+0000 7f61708ab640 1 --2- 172.21.15.89:0/4229410733 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f61500664a0 0x7f6150068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:02.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.150+0000 7f61708ab640 1 --2- 172.21.15.89:0/4229410733 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f61500664a0 0x7f6150068960 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6160000c10 tx=0x7f6160000ee0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:02.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.154+0000 7f61697fa640 1 -- 172.21.15.89:0/4229410733 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6158051b30 con 0x7f616c1ef760 2023-10-30T12:33:02.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.334+0000 7f6172b0f640 1 -- 172.21.15.89:0/4229410733 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f616c04e170 con 0x7f616c1ef760 2023-10-30T12:33:02.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.334+0000 7f61697fa640 1 -- 172.21.15.89:0/4229410733 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6534 (secure 0 0 0) 0x7f6158055150 con 0x7f616c1ef760 2023-10-30T12:33:02.340 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:33:02.340 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4781,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:32:04.109710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":2,"incarnation":15,"state":"up:active","state_seq":50,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:33:02.341 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 23 2023-10-30T12:33:02.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.338+0000 7f6172b0f640 1 -- 172.21.15.89:0/4229410733 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f61500664a0 msgr2=0x7f6150068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:02.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.338+0000 7f6172b0f640 1 --2- 172.21.15.89:0/4229410733 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f61500664a0 0x7f6150068960 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6160000c10 tx=0x7f6160000ee0 comp rx=0 tx=0).stop 2023-10-30T12:33:02.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.338+0000 7f6172b0f640 1 -- 172.21.15.89:0/4229410733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f616c1ef760 msgr2=0x7f616c0af770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:02.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.338+0000 7f6172b0f640 1 --2- 172.21.15.89:0/4229410733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f616c1ef760 0x7f616c0af770 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f61580119d0 tx=0x7f6158011dd0 comp rx=0 tx=0).stop 2023-10-30T12:33:02.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.338+0000 7f6172b0f640 1 -- 172.21.15.89:0/4229410733 shutdown_connections 2023-10-30T12:33:02.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.338+0000 7f6172b0f640 1 --2- 172.21.15.89:0/4229410733 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f61500664a0 0x7f6150068960 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:02.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.338+0000 7f6172b0f640 1 --2- 172.21.15.89:0/4229410733 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f616c148cf0 0x7f616c1436d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:02.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.338+0000 7f6172b0f640 1 --2- 172.21.15.89:0/4229410733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f616c1ef760 0x7f616c0af770 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:02.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.338+0000 7f6172b0f640 1 --2- 172.21.15.89:0/4229410733 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f615c004d10 0x7f616c0c1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:02.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.338+0000 7f6172b0f640 1 -- 172.21.15.89:0/4229410733 >> 172.21.15.89:0/4229410733 conn(0x7f616c0c7910 msgr2=0x7f616c06b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:02.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.338+0000 7f6172b0f640 1 -- 172.21.15.89:0/4229410733 shutdown_connections 2023-10-30T12:33:02.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:02.338+0000 7f6172b0f640 1 -- 172.21.15.89:0/4229410733 wait complete. 2023-10-30T12:33:02.361 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:33:04.363 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:33:04.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.506+0000 7f7156365640 1 Processor -- start 2023-10-30T12:33:04.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.506+0000 7f7156365640 1 -- start start 2023-10-30T12:33:04.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f7156365640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f71500c8460 0x7f71500715c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:04.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f7156365640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7150070650 0x7f7150159810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:04.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f7156365640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7150159d50 con 0x7f7150157010 2023-10-30T12:33:04.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f7156365640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71500c21b0 con 0x7f71500c8460 2023-10-30T12:33:04.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f7156365640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71500c1530 con 0x7f7150070650 2023-10-30T12:33:04.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f714ffff640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f7150157010 0x7f7150071bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:37180/0 (socket says 172.21.15.89:37180) 2023-10-30T12:33:04.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f714ffff640 1 -- 172.21.15.89:0/3872209592 learned_addr learned my addr 172.21.15.89:0/3872209592 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:33:04.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f714f7fe640 1 --2- 172.21.15.89:0/3872209592 >> v2:172.21.15.192:3300/0 conn(0x7f71500c8460 0x7f71500715c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:04.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f7154902640 1 --2- 172.21.15.89:0/3872209592 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7150070650 0x7f7150159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:04.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f714effd640 1 -- 172.21.15.89:0/3872209592 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 783819026 0 0) 0x7f7150159d50 con 0x7f7150157010 2023-10-30T12:33:04.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f714effd640 1 -- 172.21.15.89:0/3872209592 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7138003530 con 0x7f7150157010 2023-10-30T12:33:04.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f714effd640 1 -- 172.21.15.89:0/3872209592 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1599414561 0 0) 0x7f7138003530 con 0x7f7150157010 2023-10-30T12:33:04.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f714effd640 1 -- 172.21.15.89:0/3872209592 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7150070650 msgr2=0x7f7150159810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:04.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f714effd640 1 --2- 172.21.15.89:0/3872209592 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7150070650 0x7f7150159810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:04.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f714effd640 1 -- 172.21.15.89:0/3872209592 >> v2:172.21.15.192:3300/0 conn(0x7f71500c8460 msgr2=0x7f71500715c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:04.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f714effd640 1 --2- 172.21.15.89:0/3872209592 >> v2:172.21.15.192:3300/0 conn(0x7f71500c8460 0x7f71500715c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:04.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f714effd640 1 -- 172.21.15.89:0/3872209592 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71500c0f70 con 0x7f7150157010 2023-10-30T12:33:04.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f714effd640 1 -- 172.21.15.89:0/3872209592 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f7140001680 con 0x7f7150157010 2023-10-30T12:33:04.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f714effd640 1 -- 172.21.15.89:0/3872209592 >> v1:172.21.15.89:6789/0 conn(0x7f7150157010 legacy=0x7f7150071bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:04.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f714effd640 1 --2- 172.21.15.89:0/3872209592 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7138003980 0x7f7138003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:04.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f714effd640 1 --2- 172.21.15.89:0/3872209592 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7150070650 0x7f71380047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:04.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f714effd640 1 --2- 172.21.15.89:0/3872209592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7138004d10 0x7f7138005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:04.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f714effd640 1 -- 172.21.15.89:0/3872209592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71500c21b0 con 0x7f7138004d10 2023-10-30T12:33:04.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f714effd640 1 -- 172.21.15.89:0/3872209592 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71500c1530 con 0x7f7138003980 2023-10-30T12:33:04.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f714effd640 1 -- 172.21.15.89:0/3872209592 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f713800b990 con 0x7f7150070650 2023-10-30T12:33:04.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f7154902640 1 --2- 172.21.15.89:0/3872209592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7138004d10 0x7f7138005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:04.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f714f7fe640 1 --2- 172.21.15.89:0/3872209592 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7150070650 0x7f71380047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:04.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f714ffff640 1 --2- 172.21.15.89:0/3872209592 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7138003980 0x7f7138003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:04.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f7154902640 1 -- 172.21.15.89:0/3872209592 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7150070650 msgr2=0x7f71380047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:04.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f7154902640 1 --2- 172.21.15.89:0/3872209592 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7150070650 0x7f71380047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:04.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f7154902640 1 -- 172.21.15.89:0/3872209592 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7138003980 msgr2=0x7f7138003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:04.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f7154902640 1 --2- 172.21.15.89:0/3872209592 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7138003980 0x7f7138003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:04.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f7154902640 1 -- 172.21.15.89:0/3872209592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7138003530 con 0x7f7138004d10 2023-10-30T12:33:04.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.510+0000 7f714ffff640 1 --2- 172.21.15.89:0/3872209592 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7138003980 0x7f7138003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:33:04.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.514+0000 7f7154902640 1 --2- 172.21.15.89:0/3872209592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7138004d10 0x7f7138005190 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f714800e170 tx=0x7f714801cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c20c495e611f3dd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:04.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.514+0000 7f714effd640 1 -- 172.21.15.89:0/3872209592 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71480a5c50 con 0x7f7138004d10 2023-10-30T12:33:04.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.514+0000 7f714effd640 1 -- 172.21.15.89:0/3872209592 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714800f040 con 0x7f7138004d10 2023-10-30T12:33:04.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.514+0000 7f714effd640 1 -- 172.21.15.89:0/3872209592 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71480ad440 con 0x7f7138004d10 2023-10-30T12:33:04.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.514+0000 7f7156365640 1 -- 172.21.15.89:0/3872209592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7138004d10 msgr2=0x7f7138005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:04.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.514+0000 7f7156365640 1 --2- 172.21.15.89:0/3872209592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7138004d10 0x7f7138005190 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f714800e170 tx=0x7f714801cc40 comp rx=0 tx=0).stop 2023-10-30T12:33:04.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.514+0000 7f7154902640 1 -- 172.21.15.89:0/3872209592 reap_dead start 2023-10-30T12:33:04.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.514+0000 7f7156365640 1 -- 172.21.15.89:0/3872209592 shutdown_connections 2023-10-30T12:33:04.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.514+0000 7f7156365640 1 -- 172.21.15.89:0/3872209592 >> 172.21.15.89:0/3872209592 conn(0x7f71500cbeb0 msgr2=0x7f71500cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:04.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.514+0000 7f7156365640 1 -- 172.21.15.89:0/3872209592 shutdown_connections 2023-10-30T12:33:04.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.514+0000 7f7156365640 1 -- 172.21.15.89:0/3872209592 wait complete. 2023-10-30T12:33:04.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.518+0000 7f7156365640 1 Processor -- start 2023-10-30T12:33:04.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.518+0000 7f7156365640 1 -- start start 2023-10-30T12:33:04.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.518+0000 7f7156365640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7138004d10 0x7f715006b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:04.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.518+0000 7f7156365640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7150155020 0x7f715014bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:04.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.518+0000 7f7156365640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f715014c4e0 0x7f715014ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:04.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.518+0000 7f7156365640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71501562d0 con 0x7f7138004d10 2023-10-30T12:33:04.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.518+0000 7f7156365640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71501efa20 con 0x7f7150155020 2023-10-30T12:33:04.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.518+0000 7f7156365640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71501ef740 con 0x7f715014c4e0 2023-10-30T12:33:04.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.518+0000 7f714ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7138004d10 0x7f715006b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:04.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.518+0000 7f714ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7138004d10 0x7f715006b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54650/0 (socket says 172.21.15.89:54650) 2023-10-30T12:33:04.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.518+0000 7f714ffff640 1 -- 172.21.15.89:0/4036569110 learned_addr learned my addr 172.21.15.89:0/4036569110 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:33:04.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.518+0000 7f714f7fe640 1 --2- 172.21.15.89:0/4036569110 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7150155020 0x7f715014bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:04.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.518+0000 7f7154902640 1 --2- 172.21.15.89:0/4036569110 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f715014c4e0 0x7f715014ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:04.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.518+0000 7f714ffff640 1 -- 172.21.15.89:0/4036569110 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f715014c4e0 msgr2=0x7f715014ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:04.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.518+0000 7f714ffff640 1 --2- 172.21.15.89:0/4036569110 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f715014c4e0 0x7f715014ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:04.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.518+0000 7f714ffff640 1 -- 172.21.15.89:0/4036569110 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7150155020 msgr2=0x7f715014bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:04.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.518+0000 7f714ffff640 1 --2- 172.21.15.89:0/4036569110 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7150155020 0x7f715014bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:04.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.518+0000 7f714ffff640 1 -- 172.21.15.89:0/4036569110 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71500c9e50 con 0x7f7138004d10 2023-10-30T12:33:04.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.518+0000 7f7154902640 1 --2- 172.21.15.89:0/4036569110 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f715014c4e0 0x7f715014ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:33:04.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.518+0000 7f714f7fe640 1 --2- 172.21.15.89:0/4036569110 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7150155020 0x7f715014bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:33:04.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.518+0000 7f714ffff640 1 --2- 172.21.15.89:0/4036569110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7138004d10 0x7f715006b140 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f714000e870 tx=0x7f714000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:04.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.518+0000 7f714cff9640 1 -- 172.21.15.89:0/4036569110 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714000c870 con 0x7f7138004d10 2023-10-30T12:33:04.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.518+0000 7f714cff9640 1 -- 172.21.15.89:0/4036569110 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714000c9d0 con 0x7f7138004d10 2023-10-30T12:33:04.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.518+0000 7f7156365640 1 -- 172.21.15.89:0/4036569110 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71501eeb30 con 0x7f7138004d10 2023-10-30T12:33:04.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.518+0000 7f714cff9640 1 -- 172.21.15.89:0/4036569110 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714000cd40 con 0x7f7138004d10 2023-10-30T12:33:04.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.518+0000 7f7156365640 1 -- 172.21.15.89:0/4036569110 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71501ef0f0 con 0x7f7138004d10 2023-10-30T12:33:04.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.518+0000 7f7156365640 1 -- 172.21.15.89:0/4036569110 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f715004e170 con 0x7f7138004d10 2023-10-30T12:33:04.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.522+0000 7f714cff9640 1 -- 172.21.15.89:0/4036569110 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7140014480 con 0x7f7138004d10 2023-10-30T12:33:04.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.522+0000 7f714cff9640 1 --2- 172.21.15.89:0/4036569110 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f713c0664a0 0x7f713c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:04.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.522+0000 7f714cff9640 1 -- 172.21.15.89:0/4036569110 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5890+0+0 (secure 0 0 0) 0x7f7140083820 con 0x7f7138004d10 2023-10-30T12:33:04.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.522+0000 7f714f7fe640 1 --2- 172.21.15.89:0/4036569110 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f713c0664a0 0x7f713c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:04.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.522+0000 7f714f7fe640 1 --2- 172.21.15.89:0/4036569110 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f713c0664a0 0x7f713c068960 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f715006bab0 tx=0x7f7144008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:04.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.526+0000 7f714cff9640 1 -- 172.21.15.89:0/4036569110 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7140050c70 con 0x7f7138004d10 2023-10-30T12:33:04.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.702+0000 7f7156365640 1 -- 172.21.15.89:0/4036569110 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71500c8240 con 0x7f7138004d10 2023-10-30T12:33:04.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.706+0000 7f714cff9640 1 -- 172.21.15.89:0/4036569110 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6534 (secure 0 0 0) 0x7f7140019070 con 0x7f7138004d10 2023-10-30T12:33:04.709 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:33:04.709 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4781,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:32:04.109710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":2,"incarnation":15,"state":"up:active","state_seq":50,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:33:04.709 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 23 2023-10-30T12:33:04.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.706+0000 7f7156365640 1 -- 172.21.15.89:0/4036569110 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f713c0664a0 msgr2=0x7f713c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:04.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.706+0000 7f7156365640 1 --2- 172.21.15.89:0/4036569110 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f713c0664a0 0x7f713c068960 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f715006bab0 tx=0x7f7144008040 comp rx=0 tx=0).stop 2023-10-30T12:33:04.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.706+0000 7f7156365640 1 -- 172.21.15.89:0/4036569110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7138004d10 msgr2=0x7f715006b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:04.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.706+0000 7f7156365640 1 --2- 172.21.15.89:0/4036569110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7138004d10 0x7f715006b140 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f714000e870 tx=0x7f714000ed30 comp rx=0 tx=0).stop 2023-10-30T12:33:04.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.706+0000 7f7156365640 1 -- 172.21.15.89:0/4036569110 shutdown_connections 2023-10-30T12:33:04.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.706+0000 7f7156365640 1 --2- 172.21.15.89:0/4036569110 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f713c0664a0 0x7f713c068960 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:04.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.706+0000 7f7156365640 1 --2- 172.21.15.89:0/4036569110 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7150155020 0x7f715014bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:04.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.706+0000 7f7156365640 1 --2- 172.21.15.89:0/4036569110 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f715014c4e0 0x7f715014ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:04.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.706+0000 7f7156365640 1 --2- 172.21.15.89:0/4036569110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7138004d10 0x7f715006b140 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:04.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.706+0000 7f7156365640 1 -- 172.21.15.89:0/4036569110 >> 172.21.15.89:0/4036569110 conn(0x7f71500cbeb0 msgr2=0x7f715006c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:04.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.706+0000 7f7156365640 1 -- 172.21.15.89:0/4036569110 shutdown_connections 2023-10-30T12:33:04.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:04.706+0000 7f7156365640 1 -- 172.21.15.89:0/4036569110 wait complete. 2023-10-30T12:33:04.729 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:33:06.731 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:33:06.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.874+0000 7f3062217640 1 Processor -- start 2023-10-30T12:33:06.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.874+0000 7f3062217640 1 -- start start 2023-10-30T12:33:06.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.878+0000 7f3062217640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f305c0c05d0 0x7f305c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:06.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.878+0000 7f3062217640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f305c0c6f60 0x7f305c13f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:06.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.878+0000 7f3062217640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f305c0c0a50 con 0x7f305c0c6b80 2023-10-30T12:33:06.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.878+0000 7f3062217640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f305c06f6e0 con 0x7f305c0c05d0 2023-10-30T12:33:06.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.878+0000 7f3062217640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f305c06f9c0 con 0x7f305c0c6f60 2023-10-30T12:33:06.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.878+0000 7f305affd640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f305c0c05d0 0x7f305c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:06.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.878+0000 7f305b7fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f305c0c6b80 0x7f305c0bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:37192/0 (socket says 172.21.15.89:37192) 2023-10-30T12:33:06.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.878+0000 7f305b7fe640 1 -- 172.21.15.89:0/276958409 learned_addr learned my addr 172.21.15.89:0/276958409 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:33:06.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.878+0000 7f305bfff640 1 --2- 172.21.15.89:0/276958409 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f305c0c6f60 0x7f305c13f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:06.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.878+0000 7f305a7fc640 1 -- 172.21.15.89:0/276958409 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3831777118 0 0) 0x7f305c0c0a50 con 0x7f305c0c6b80 2023-10-30T12:33:06.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.878+0000 7f305a7fc640 1 -- 172.21.15.89:0/276958409 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3044005740 con 0x7f305c0c6b80 2023-10-30T12:33:06.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.878+0000 7f305affd640 1 -- 172.21.15.89:0/276958409 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f305c0c6f60 msgr2=0x7f305c13f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:06.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.878+0000 7f305affd640 1 --2- 172.21.15.89:0/276958409 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f305c0c6f60 0x7f305c13f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:06.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.878+0000 7f305affd640 1 -- 172.21.15.89:0/276958409 >> v1:172.21.15.89:6789/0 conn(0x7f305c0c6b80 legacy=0x7f305c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:06.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.878+0000 7f305affd640 1 -- 172.21.15.89:0/276958409 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f305c06fca0 con 0x7f305c0c05d0 2023-10-30T12:33:06.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.878+0000 7f305affd640 1 --2- 172.21.15.89:0/276958409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f305c0c05d0 0x7f305c13ed90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f30500097c0 tx=0x7f305001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=212174f432854fd0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:06.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.878+0000 7f305a7fc640 1 -- 172.21.15.89:0/276958409 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30500ad640 con 0x7f305c0c05d0 2023-10-30T12:33:06.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.882+0000 7f305a7fc640 1 -- 172.21.15.89:0/276958409 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30500ad7a0 con 0x7f305c0c05d0 2023-10-30T12:33:06.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.882+0000 7f305a7fc640 1 -- 172.21.15.89:0/276958409 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30500adb10 con 0x7f305c0c05d0 2023-10-30T12:33:06.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.882+0000 7f3062217640 1 -- 172.21.15.89:0/276958409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f305c0c05d0 msgr2=0x7f305c13ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:06.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.882+0000 7f3062217640 1 --2- 172.21.15.89:0/276958409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f305c0c05d0 0x7f305c13ed90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f30500097c0 tx=0x7f305001c480 comp rx=0 tx=0).stop 2023-10-30T12:33:06.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.882+0000 7f3062217640 1 -- 172.21.15.89:0/276958409 shutdown_connections 2023-10-30T12:33:06.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.882+0000 7f3062217640 1 --2- 172.21.15.89:0/276958409 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f305c0c6f60 0x7f305c13f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:06.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.882+0000 7f3062217640 1 --2- 172.21.15.89:0/276958409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f305c0c05d0 0x7f305c13ed90 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:06.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.882+0000 7f3062217640 1 -- 172.21.15.89:0/276958409 >> 172.21.15.89:0/276958409 conn(0x7f305c0c7910 msgr2=0x7f305c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:06.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.882+0000 7f3062217640 1 -- 172.21.15.89:0/276958409 shutdown_connections 2023-10-30T12:33:06.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.882+0000 7f3062217640 1 -- 172.21.15.89:0/276958409 wait complete. 2023-10-30T12:33:06.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.882+0000 7f3062217640 1 Processor -- start 2023-10-30T12:33:06.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.882+0000 7f3062217640 1 -- start start 2023-10-30T12:33:06.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.882+0000 7f3062217640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f305c0c05d0 0x7f305c0c9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:06.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.882+0000 7f3062217640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f305c0c6b80 0x7f305c1349d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:06.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.882+0000 7f3062217640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f305c0c6f60 0x7f305c1501c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:06.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.882+0000 7f3062217640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305c1f1e80 con 0x7f305c0c05d0 2023-10-30T12:33:06.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.882+0000 7f3062217640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305c132dc0 con 0x7f305c0c6b80 2023-10-30T12:33:06.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.882+0000 7f3062217640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f305c1337f0 con 0x7f305c0c6f60 2023-10-30T12:33:06.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.882+0000 7f305b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f305c0c05d0 0x7f305c0c9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:06.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.882+0000 7f305b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f305c0c05d0 0x7f305c0c9bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54664/0 (socket says 172.21.15.89:54664) 2023-10-30T12:33:06.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.882+0000 7f305b7fe640 1 -- 172.21.15.89:0/4256867379 learned_addr learned my addr 172.21.15.89:0/4256867379 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:33:06.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.882+0000 7f305bfff640 1 --2- 172.21.15.89:0/4256867379 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f305c0c6f60 0x7f305c1501c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:06.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.886+0000 7f305affd640 1 --2- 172.21.15.89:0/4256867379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f305c0c6b80 0x7f305c1349d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:06.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.886+0000 7f305b7fe640 1 -- 172.21.15.89:0/4256867379 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f305c0c6f60 msgr2=0x7f305c1501c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:06.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.886+0000 7f305b7fe640 1 --2- 172.21.15.89:0/4256867379 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f305c0c6f60 0x7f305c1501c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:06.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.886+0000 7f305b7fe640 1 -- 172.21.15.89:0/4256867379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f305c0c6b80 msgr2=0x7f305c1349d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:06.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.886+0000 7f305b7fe640 1 --2- 172.21.15.89:0/4256867379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f305c0c6b80 0x7f305c1349d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:06.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.886+0000 7f305b7fe640 1 -- 172.21.15.89:0/4256867379 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3050009420 con 0x7f305c0c05d0 2023-10-30T12:33:06.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.886+0000 7f305bfff640 1 --2- 172.21.15.89:0/4256867379 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f305c0c6f60 0x7f305c1501c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:33:06.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.886+0000 7f305affd640 1 --2- 172.21.15.89:0/4256867379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f305c0c6b80 0x7f305c1349d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:33:06.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.886+0000 7f305b7fe640 1 --2- 172.21.15.89:0/4256867379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f305c0c05d0 0x7f305c0c9bb0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f304c00c450 tx=0x7f304c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:06.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.886+0000 7f3043fff640 1 -- 172.21.15.89:0/4256867379 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304c005cf0 con 0x7f305c0c05d0 2023-10-30T12:33:06.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.886+0000 7f3043fff640 1 -- 172.21.15.89:0/4256867379 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f304c005e50 con 0x7f305c0c05d0 2023-10-30T12:33:06.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.886+0000 7f3043fff640 1 -- 172.21.15.89:0/4256867379 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304c007290 con 0x7f305c0c05d0 2023-10-30T12:33:06.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.886+0000 7f3062217640 1 -- 172.21.15.89:0/4256867379 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f305c133ad0 con 0x7f305c0c05d0 2023-10-30T12:33:06.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.886+0000 7f3062217640 1 -- 172.21.15.89:0/4256867379 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f305c134090 con 0x7f305c0c05d0 2023-10-30T12:33:06.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.886+0000 7f3062217640 1 -- 172.21.15.89:0/4256867379 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f305c06c440 con 0x7f305c0c05d0 2023-10-30T12:33:06.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.886+0000 7f3043fff640 1 -- 172.21.15.89:0/4256867379 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f304c0073f0 con 0x7f305c0c05d0 2023-10-30T12:33:06.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.886+0000 7f3043fff640 1 --2- 172.21.15.89:0/4256867379 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f30480664a0 0x7f3048068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:06.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.886+0000 7f3043fff640 1 -- 172.21.15.89:0/4256867379 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5890+0+0 (secure 0 0 0) 0x7f304c083530 con 0x7f305c0c05d0 2023-10-30T12:33:06.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.890+0000 7f305affd640 1 --2- 172.21.15.89:0/4256867379 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f30480664a0 0x7f3048068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:06.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.890+0000 7f305affd640 1 --2- 172.21.15.89:0/4256867379 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f30480664a0 0x7f3048068960 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f30500af000 tx=0x7f30500a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:06.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:06.890+0000 7f3043fff640 1 -- 172.21.15.89:0/4256867379 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f304c0508e0 con 0x7f305c0c05d0 2023-10-30T12:33:07.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:07.078+0000 7f3062217640 1 -- 172.21.15.89:0/4256867379 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f305c04e170 con 0x7f305c0c05d0 2023-10-30T12:33:07.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:07.078+0000 7f3043fff640 1 -- 172.21.15.89:0/4256867379 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6534 (secure 0 0 0) 0x7f304c017090 con 0x7f305c0c05d0 2023-10-30T12:33:07.084 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:33:07.084 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4781,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:32:04.109710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":2,"incarnation":15,"state":"up:active","state_seq":50,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:33:07.085 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 23 2023-10-30T12:33:07.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:07.082+0000 7f3062217640 1 -- 172.21.15.89:0/4256867379 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f30480664a0 msgr2=0x7f3048068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:07.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:07.082+0000 7f3062217640 1 --2- 172.21.15.89:0/4256867379 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f30480664a0 0x7f3048068960 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f30500af000 tx=0x7f30500a7070 comp rx=0 tx=0).stop 2023-10-30T12:33:07.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:07.082+0000 7f3062217640 1 -- 172.21.15.89:0/4256867379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f305c0c05d0 msgr2=0x7f305c0c9bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:07.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:07.082+0000 7f3062217640 1 --2- 172.21.15.89:0/4256867379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f305c0c05d0 0x7f305c0c9bb0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f304c00c450 tx=0x7f304c00c910 comp rx=0 tx=0).stop 2023-10-30T12:33:07.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:07.082+0000 7f3062217640 1 -- 172.21.15.89:0/4256867379 shutdown_connections 2023-10-30T12:33:07.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:07.082+0000 7f3062217640 1 --2- 172.21.15.89:0/4256867379 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f30480664a0 0x7f3048068960 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:07.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:07.082+0000 7f3062217640 1 --2- 172.21.15.89:0/4256867379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f305c0c6b80 0x7f305c1349d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:07.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:07.082+0000 7f3062217640 1 --2- 172.21.15.89:0/4256867379 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f305c0c6f60 0x7f305c1501c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:07.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:07.082+0000 7f3062217640 1 --2- 172.21.15.89:0/4256867379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f305c0c05d0 0x7f305c0c9bb0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:07.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:07.082+0000 7f3062217640 1 -- 172.21.15.89:0/4256867379 >> 172.21.15.89:0/4256867379 conn(0x7f305c0c7910 msgr2=0x7f305c0b0880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:07.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:07.082+0000 7f3062217640 1 -- 172.21.15.89:0/4256867379 shutdown_connections 2023-10-30T12:33:07.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:07.082+0000 7f3062217640 1 -- 172.21.15.89:0/4256867379 wait complete. 2023-10-30T12:33:07.105 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:33:09.106 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:33:09.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.245+0000 7f0e87baa640 1 Processor -- start 2023-10-30T12:33:09.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.245+0000 7f0e87baa640 1 -- start start 2023-10-30T12:33:09.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.249+0000 7f0e87baa640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0e8006f7a0 0x7f0e8006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:09.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.249+0000 7f0e87baa640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e800c6bd0 0x7f0e80069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:09.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.249+0000 7f0e87baa640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e8006a3d0 con 0x7f0e80071110 2023-10-30T12:33:09.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.249+0000 7f0e87baa640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e800700c0 con 0x7f0e8006f7a0 2023-10-30T12:33:09.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.249+0000 7f0e87baa640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e80066cf0 con 0x7f0e800c6bd0 2023-10-30T12:33:09.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.249+0000 7f0e85145640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e800c6bd0 0x7f0e80069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:09.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.249+0000 7f0e85946640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0e8006f7a0 0x7f0e8006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:09.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.249+0000 7f0e85145640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e800c6bd0 0x7f0e80069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:50220/0 (socket says 172.21.15.89:50220) 2023-10-30T12:33:09.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.249+0000 7f0e85145640 1 -- 172.21.15.89:0/1109476767 learned_addr learned my addr 172.21.15.89:0/1109476767 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:33:09.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.249+0000 7f0e86147640 1 --1- 172.21.15.89:0/1109476767 >> v1:172.21.15.192:6789/0 conn(0x7f0e80071110 0x7f0e80158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:47624/0 (socket says 172.21.15.89:47624) 2023-10-30T12:33:09.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.249+0000 7f0e85145640 1 -- 172.21.15.89:0/1109476767 >> v2:172.21.15.192:3300/0 conn(0x7f0e8006f7a0 msgr2=0x7f0e8006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:09.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.249+0000 7f0e85145640 1 --2- 172.21.15.89:0/1109476767 >> v2:172.21.15.192:3300/0 conn(0x7f0e8006f7a0 0x7f0e8006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:09.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.249+0000 7f0e85145640 1 -- 172.21.15.89:0/1109476767 >> v1:172.21.15.192:6789/0 conn(0x7f0e80071110 legacy=0x7f0e80158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:09.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.249+0000 7f0e85145640 1 -- 172.21.15.89:0/1109476767 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e800c70d0 con 0x7f0e800c6bd0 2023-10-30T12:33:09.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.253+0000 7f0e85145640 1 --2- 172.21.15.89:0/1109476767 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e800c6bd0 0x7f0e80069e90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0e70009870 tx=0x7f0e7001ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abc669902f61f103 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:09.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.253+0000 7f0e84944640 1 -- 172.21.15.89:0/1109476767 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e700a8a30 con 0x7f0e800c6bd0 2023-10-30T12:33:09.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.253+0000 7f0e84944640 1 -- 172.21.15.89:0/1109476767 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e700a8b90 con 0x7f0e800c6bd0 2023-10-30T12:33:09.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.253+0000 7f0e84944640 1 -- 172.21.15.89:0/1109476767 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e700b0440 con 0x7f0e800c6bd0 2023-10-30T12:33:09.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.253+0000 7f0e87baa640 1 -- 172.21.15.89:0/1109476767 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e800c6bd0 msgr2=0x7f0e80069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:09.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.253+0000 7f0e87baa640 1 --2- 172.21.15.89:0/1109476767 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e800c6bd0 0x7f0e80069e90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0e70009870 tx=0x7f0e7001ed30 comp rx=0 tx=0).stop 2023-10-30T12:33:09.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.253+0000 7f0e87baa640 1 -- 172.21.15.89:0/1109476767 shutdown_connections 2023-10-30T12:33:09.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.253+0000 7f0e87baa640 1 --2- 172.21.15.89:0/1109476767 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e800c6bd0 0x7f0e80069e90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:09.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.253+0000 7f0e87baa640 1 --2- 172.21.15.89:0/1109476767 >> v2:172.21.15.192:3300/0 conn(0x7f0e8006f7a0 0x7f0e8006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:09.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.253+0000 7f0e87baa640 1 -- 172.21.15.89:0/1109476767 >> 172.21.15.89:0/1109476767 conn(0x7f0e800c7910 msgr2=0x7f0e800c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:09.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.253+0000 7f0e87baa640 1 -- 172.21.15.89:0/1109476767 shutdown_connections 2023-10-30T12:33:09.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.253+0000 7f0e87baa640 1 -- 172.21.15.89:0/1109476767 wait complete. 2023-10-30T12:33:09.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.257+0000 7f0e87baa640 1 Processor -- start 2023-10-30T12:33:09.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.257+0000 7f0e87baa640 1 -- start start 2023-10-30T12:33:09.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.257+0000 7f0e87baa640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e8006f7a0 0x7f0e8014bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:09.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.257+0000 7f0e87baa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e80071110 0x7f0e80149f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:09.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.257+0000 7f0e87baa640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e800c6bd0 0x7f0e80147430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:09.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.257+0000 7f0e87baa640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e80148e30 con 0x7f0e80071110 2023-10-30T12:33:09.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.257+0000 7f0e87baa640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e80149930 con 0x7f0e8006f7a0 2023-10-30T12:33:09.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.257+0000 7f0e87baa640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8014c260 con 0x7f0e800c6bd0 2023-10-30T12:33:09.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.257+0000 7f0e85145640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e80071110 0x7f0e80149f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:09.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.257+0000 7f0e85145640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e80071110 0x7f0e80149f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54672/0 (socket says 172.21.15.89:54672) 2023-10-30T12:33:09.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.257+0000 7f0e85145640 1 -- 172.21.15.89:0/2280714571 learned_addr learned my addr 172.21.15.89:0/2280714571 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:33:09.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.257+0000 7f0e85946640 1 --2- 172.21.15.89:0/2280714571 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e8006f7a0 0x7f0e8014bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:09.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.257+0000 7f0e86147640 1 --2- 172.21.15.89:0/2280714571 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e800c6bd0 0x7f0e80147430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:09.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.257+0000 7f0e85145640 1 -- 172.21.15.89:0/2280714571 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e800c6bd0 msgr2=0x7f0e80147430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:09.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.257+0000 7f0e85145640 1 --2- 172.21.15.89:0/2280714571 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e800c6bd0 0x7f0e80147430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:09.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.257+0000 7f0e85145640 1 -- 172.21.15.89:0/2280714571 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e8006f7a0 msgr2=0x7f0e8014bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:09.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.257+0000 7f0e85145640 1 --2- 172.21.15.89:0/2280714571 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e8006f7a0 0x7f0e8014bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:09.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.257+0000 7f0e85145640 1 -- 172.21.15.89:0/2280714571 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e700094d0 con 0x7f0e80071110 2023-10-30T12:33:09.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.257+0000 7f0e85145640 1 --2- 172.21.15.89:0/2280714571 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e80071110 0x7f0e80149f20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f0e7001d3e0 tx=0x7f0e7001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:09.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.257+0000 7f0e767fc640 1 -- 172.21.15.89:0/2280714571 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e700b1de0 con 0x7f0e80071110 2023-10-30T12:33:09.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.257+0000 7f0e767fc640 1 -- 172.21.15.89:0/2280714571 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e700b9410 con 0x7f0e80071110 2023-10-30T12:33:09.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.257+0000 7f0e87baa640 1 -- 172.21.15.89:0/2280714571 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e8014ab30 con 0x7f0e80071110 2023-10-30T12:33:09.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.257+0000 7f0e767fc640 1 -- 172.21.15.89:0/2280714571 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e700b97c0 con 0x7f0e80071110 2023-10-30T12:33:09.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.257+0000 7f0e87baa640 1 -- 172.21.15.89:0/2280714571 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e8014b0f0 con 0x7f0e80071110 2023-10-30T12:33:09.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.257+0000 7f0e87baa640 1 -- 172.21.15.89:0/2280714571 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e8004e170 con 0x7f0e80071110 2023-10-30T12:33:09.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.261+0000 7f0e767fc640 1 -- 172.21.15.89:0/2280714571 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0e700b7040 con 0x7f0e80071110 2023-10-30T12:33:09.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.261+0000 7f0e767fc640 1 --2- 172.21.15.89:0/2280714571 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0e640664a0 0x7f0e64068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:09.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.261+0000 7f0e767fc640 1 -- 172.21.15.89:0/2280714571 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5890+0+0 (secure 0 0 0) 0x7f0e70121ab0 con 0x7f0e80071110 2023-10-30T12:33:09.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.261+0000 7f0e85946640 1 --2- 172.21.15.89:0/2280714571 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0e640664a0 0x7f0e64068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:09.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.261+0000 7f0e85946640 1 --2- 172.21.15.89:0/2280714571 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0e640664a0 0x7f0e64068960 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c009a50 tx=0x7f0e7c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:09.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.265+0000 7f0e767fc640 1 -- 172.21.15.89:0/2280714571 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0e700eedd0 con 0x7f0e80071110 2023-10-30T12:33:09.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.449+0000 7f0e87baa640 1 -- 172.21.15.89:0/2280714571 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e8006a820 con 0x7f0e80071110 2023-10-30T12:33:09.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.449+0000 7f0e767fc640 1 -- 172.21.15.89:0/2280714571 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6534 (secure 0 0 0) 0x7f0e700b5090 con 0x7f0e80071110 2023-10-30T12:33:09.455 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:33:09.456 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4781,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:32:04.109710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":2,"incarnation":15,"state":"up:active","state_seq":50,"addr":"172.21.15.89:6837/2053475104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2053475104},{"type":"v1","addr":"172.21.15.89:6837","nonce":2053475104}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:33:09.457 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 23 2023-10-30T12:33:09.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.449+0000 7f0e87baa640 1 -- 172.21.15.89:0/2280714571 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0e640664a0 msgr2=0x7f0e64068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:09.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.449+0000 7f0e87baa640 1 --2- 172.21.15.89:0/2280714571 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0e640664a0 0x7f0e64068960 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c009a50 tx=0x7f0e7c008040 comp rx=0 tx=0).stop 2023-10-30T12:33:09.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.449+0000 7f0e87baa640 1 -- 172.21.15.89:0/2280714571 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e80071110 msgr2=0x7f0e80149f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:09.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.449+0000 7f0e87baa640 1 --2- 172.21.15.89:0/2280714571 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e80071110 0x7f0e80149f20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f0e7001d3e0 tx=0x7f0e7001bad0 comp rx=0 tx=0).stop 2023-10-30T12:33:09.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.449+0000 7f0e87baa640 1 -- 172.21.15.89:0/2280714571 shutdown_connections 2023-10-30T12:33:09.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.449+0000 7f0e87baa640 1 --2- 172.21.15.89:0/2280714571 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0e640664a0 0x7f0e64068960 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:09.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.449+0000 7f0e87baa640 1 --2- 172.21.15.89:0/2280714571 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e800c6bd0 0x7f0e80147430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:09.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.449+0000 7f0e87baa640 1 --2- 172.21.15.89:0/2280714571 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e80071110 0x7f0e80149f20 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:09.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.449+0000 7f0e87baa640 1 --2- 172.21.15.89:0/2280714571 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e8006f7a0 0x7f0e8014bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:09.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.449+0000 7f0e87baa640 1 -- 172.21.15.89:0/2280714571 >> 172.21.15.89:0/2280714571 conn(0x7f0e800c7910 msgr2=0x7f0e800b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:09.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.453+0000 7f0e87baa640 1 -- 172.21.15.89:0/2280714571 shutdown_connections 2023-10-30T12:33:09.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:09.453+0000 7f0e87baa640 1 -- 172.21.15.89:0/2280714571 wait complete. 2023-10-30T12:33:09.475 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:33:11.477 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:33:11.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.617+0000 7fa96ab65640 1 Processor -- start 2023-10-30T12:33:11.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.621+0000 7fa96ab65640 1 -- start start 2023-10-30T12:33:11.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.621+0000 7fa96ab65640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa9641539a0 0x7fa96415a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:11.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.621+0000 7fa96ab65640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa964069e90 0x7fa96406a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:11.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.621+0000 7fa96ab65640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa96415a8c0 con 0x7fa964070a00 2023-10-30T12:33:11.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.621+0000 7fa96ab65640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa96406fbb0 con 0x7fa9641539a0 2023-10-30T12:33:11.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.621+0000 7fa96ab65640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa964153d80 con 0x7fa964069e90 2023-10-30T12:33:11.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.621+0000 7fa968901640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa9641539a0 0x7fa96415a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:11.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.621+0000 7fa968901640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa9641539a0 0x7fa96415a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46562/0 (socket says 172.21.15.89:46562) 2023-10-30T12:33:11.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.621+0000 7fa968901640 1 -- 172.21.15.89:0/3019343642 learned_addr learned my addr 172.21.15.89:0/3019343642 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:33:11.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.621+0000 7fa969102640 1 --2- 172.21.15.89:0/3019343642 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa964069e90 0x7fa96406a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:11.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.621+0000 7fa968901640 1 -- 172.21.15.89:0/3019343642 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa964069e90 msgr2=0x7fa96406a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:11.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.621+0000 7fa968901640 1 --2- 172.21.15.89:0/3019343642 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa964069e90 0x7fa96406a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:11.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.621+0000 7fa968901640 1 -- 172.21.15.89:0/3019343642 >> v1:172.21.15.192:6789/0 conn(0x7fa964070a00 legacy=0x7fa964070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:11.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.621+0000 7fa968901640 1 -- 172.21.15.89:0/3019343642 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9640c5c50 con 0x7fa9641539a0 2023-10-30T12:33:11.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.625+0000 7fa968901640 1 --2- 172.21.15.89:0/3019343642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9641539a0 0x7fa96415a380 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa954009870 tx=0x7fa95401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e460b46800491a75 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:11.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.625+0000 7fa9637fe640 1 -- 172.21.15.89:0/3019343642 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9540ad640 con 0x7fa9641539a0 2023-10-30T12:33:11.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.625+0000 7fa9637fe640 1 -- 172.21.15.89:0/3019343642 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9540ad7a0 con 0x7fa9641539a0 2023-10-30T12:33:11.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.625+0000 7fa9637fe640 1 -- 172.21.15.89:0/3019343642 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9540adb30 con 0x7fa9641539a0 2023-10-30T12:33:11.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.625+0000 7fa96ab65640 1 -- 172.21.15.89:0/3019343642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9641539a0 msgr2=0x7fa96415a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:11.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.625+0000 7fa96ab65640 1 --2- 172.21.15.89:0/3019343642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9641539a0 0x7fa96415a380 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa954009870 tx=0x7fa95401c480 comp rx=0 tx=0).stop 2023-10-30T12:33:11.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.625+0000 7fa96ab65640 1 -- 172.21.15.89:0/3019343642 shutdown_connections 2023-10-30T12:33:11.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.625+0000 7fa96ab65640 1 --2- 172.21.15.89:0/3019343642 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa964069e90 0x7fa96406a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:11.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.625+0000 7fa96ab65640 1 --2- 172.21.15.89:0/3019343642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9641539a0 0x7fa96415a380 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:11.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.625+0000 7fa96ab65640 1 -- 172.21.15.89:0/3019343642 >> 172.21.15.89:0/3019343642 conn(0x7fa9640c7910 msgr2=0x7fa9640c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:11.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.625+0000 7fa96ab65640 1 -- 172.21.15.89:0/3019343642 shutdown_connections 2023-10-30T12:33:11.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.625+0000 7fa96ab65640 1 -- 172.21.15.89:0/3019343642 wait complete. 2023-10-30T12:33:11.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.625+0000 7fa96ab65640 1 Processor -- start 2023-10-30T12:33:11.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.625+0000 7fa96ab65640 1 -- start start 2023-10-30T12:33:11.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.629+0000 7fa96ab65640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa964069e90 0x7fa96414c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:11.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.629+0000 7fa96ab65640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa964070a00 0x7fa9640ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:11.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.629+0000 7fa96ab65640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9641539a0 0x7fa96414cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:11.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.629+0000 7fa96ab65640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa96414ff00 con 0x7fa9641539a0 2023-10-30T12:33:11.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.629+0000 7fa96ab65640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa96414b210 con 0x7fa964070a00 2023-10-30T12:33:11.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.629+0000 7fa96ab65640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa96414c600 con 0x7fa964069e90 2023-10-30T12:33:11.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.629+0000 7fa969102640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9641539a0 0x7fa96414cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:11.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.629+0000 7fa969102640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9641539a0 0x7fa96414cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46576/0 (socket says 172.21.15.89:46576) 2023-10-30T12:33:11.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.629+0000 7fa969102640 1 -- 172.21.15.89:0/2499926720 learned_addr learned my addr 172.21.15.89:0/2499926720 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:33:11.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.629+0000 7fa963fff640 1 --2- 172.21.15.89:0/2499926720 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa964070a00 0x7fa9640ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:11.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.629+0000 7fa968901640 1 --2- 172.21.15.89:0/2499926720 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa964069e90 0x7fa96414c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:11.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.629+0000 7fa969102640 1 -- 172.21.15.89:0/2499926720 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa964069e90 msgr2=0x7fa96414c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:11.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.629+0000 7fa969102640 1 --2- 172.21.15.89:0/2499926720 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa964069e90 0x7fa96414c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:11.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.629+0000 7fa969102640 1 -- 172.21.15.89:0/2499926720 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa964070a00 msgr2=0x7fa9640ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:11.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.629+0000 7fa969102640 1 --2- 172.21.15.89:0/2499926720 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa964070a00 0x7fa9640ad8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:11.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.629+0000 7fa969102640 1 -- 172.21.15.89:0/2499926720 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9540094d0 con 0x7fa9641539a0 2023-10-30T12:33:11.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.629+0000 7fa963fff640 1 --2- 172.21.15.89:0/2499926720 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa964070a00 0x7fa9640ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:33:11.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.629+0000 7fa968901640 1 --2- 172.21.15.89:0/2499926720 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa964069e90 0x7fa96414c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:33:11.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.629+0000 7fa969102640 1 --2- 172.21.15.89:0/2499926720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9641539a0 0x7fa96414cf90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa95c00d560 tx=0x7fa95c00da20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:11.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.629+0000 7fa9617fa640 1 -- 172.21.15.89:0/2499926720 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa95c0125f0 con 0x7fa9641539a0 2023-10-30T12:33:11.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.629+0000 7fa9617fa640 1 -- 172.21.15.89:0/2499926720 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa95c012770 con 0x7fa9641539a0 2023-10-30T12:33:11.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.629+0000 7fa9617fa640 1 -- 172.21.15.89:0/2499926720 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa95c012b20 con 0x7fa9641539a0 2023-10-30T12:33:11.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.629+0000 7fa96ab65640 1 -- 172.21.15.89:0/2499926720 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9641500f0 con 0x7fa9641539a0 2023-10-30T12:33:11.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.629+0000 7fa96ab65640 1 -- 172.21.15.89:0/2499926720 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9640c0e70 con 0x7fa9641539a0 2023-10-30T12:33:11.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.629+0000 7fa96ab65640 1 -- 172.21.15.89:0/2499926720 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa96414b380 con 0x7fa9641539a0 2023-10-30T12:33:11.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.633+0000 7fa9617fa640 1 -- 172.21.15.89:0/2499926720 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa95c012c80 con 0x7fa9641539a0 2023-10-30T12:33:11.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.633+0000 7fa9617fa640 1 --2- 172.21.15.89:0/2499926720 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa9500664a0 0x7fa950068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:11.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.633+0000 7fa9617fa640 1 -- 172.21.15.89:0/2499926720 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5976+0+0 (secure 0 0 0) 0x7fa95c0844a0 con 0x7fa9641539a0 2023-10-30T12:33:11.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.633+0000 7fa968901640 1 --2- 172.21.15.89:0/2499926720 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa9500664a0 0x7fa950068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:11.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.633+0000 7fa968901640 1 --2- 172.21.15.89:0/2499926720 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa9500664a0 0x7fa950068960 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa9540af000 tx=0x7fa954008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:11.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.637+0000 7fa9617fa640 1 -- 172.21.15.89:0/2499926720 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa96414b380 con 0x7fa9641539a0 2023-10-30T12:33:11.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.817+0000 7fa96ab65640 1 -- 172.21.15.89:0/2499926720 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa96414b380 con 0x7fa9641539a0 2023-10-30T12:33:11.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.817+0000 7fa9617fa640 1 -- 172.21.15.89:0/2499926720 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5739 (secure 0 0 0) 0x7fa96414b380 con 0x7fa9641539a0 2023-10-30T12:33:11.821 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:33:11.822 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:33:10.981346+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:33:11.822 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 24 2023-10-30T12:33:11.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.817+0000 7fa96ab65640 1 -- 172.21.15.89:0/2499926720 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa9500664a0 msgr2=0x7fa950068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:11.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.817+0000 7fa96ab65640 1 --2- 172.21.15.89:0/2499926720 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa9500664a0 0x7fa950068960 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa9540af000 tx=0x7fa954008040 comp rx=0 tx=0).stop 2023-10-30T12:33:11.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.817+0000 7fa96ab65640 1 -- 172.21.15.89:0/2499926720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9641539a0 msgr2=0x7fa96414cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:11.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.817+0000 7fa96ab65640 1 --2- 172.21.15.89:0/2499926720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9641539a0 0x7fa96414cf90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa95c00d560 tx=0x7fa95c00da20 comp rx=0 tx=0).stop 2023-10-30T12:33:11.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.817+0000 7fa96ab65640 1 -- 172.21.15.89:0/2499926720 shutdown_connections 2023-10-30T12:33:11.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.817+0000 7fa96ab65640 1 --2- 172.21.15.89:0/2499926720 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa9500664a0 0x7fa950068960 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:11.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.817+0000 7fa96ab65640 1 --2- 172.21.15.89:0/2499926720 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa964070a00 0x7fa9640ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:11.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.817+0000 7fa96ab65640 1 --2- 172.21.15.89:0/2499926720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9641539a0 0x7fa96414cf90 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:11.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.817+0000 7fa96ab65640 1 --2- 172.21.15.89:0/2499926720 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa964069e90 0x7fa96414c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:11.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.817+0000 7fa96ab65640 1 -- 172.21.15.89:0/2499926720 >> 172.21.15.89:0/2499926720 conn(0x7fa9640c7910 msgr2=0x7fa964151890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:11.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.817+0000 7fa96ab65640 1 -- 172.21.15.89:0/2499926720 shutdown_connections 2023-10-30T12:33:11.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.817+0000 7fa96ab65640 1 -- 172.21.15.89:0/2499926720 wait complete. 2023-10-30T12:33:11.842 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T12:33:11.842 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:33:11.843 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:33:11.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.973+0000 7f17ac8e9640 1 Processor -- start 2023-10-30T12:33:11.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17ac8e9640 1 -- start start 2023-10-30T12:33:11.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17ac8e9640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f17a815cd50 0x7f17a80c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:11.982 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17ac8e9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17a80cefa0 0x7f17a80cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:11.982 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17ac8e9640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a815a6c0 con 0x7f17a81538f0 2023-10-30T12:33:11.982 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17ac8e9640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17a815d300 con 0x7f17a815cd50 2023-10-30T12:33:11.982 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17ac8e9640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17a815d5e0 con 0x7f17a80cefa0 2023-10-30T12:33:11.982 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17a6d9d640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f17a81538f0 0x7f17a8159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:35870/0 (socket says 172.21.15.89:35870) 2023-10-30T12:33:11.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17a6d9d640 1 -- 172.21.15.89:0/115471681 learned_addr learned my addr 172.21.15.89:0/115471681 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:33:11.985 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17a659c640 1 --2- 172.21.15.89:0/115471681 >> v2:172.21.15.192:3300/0 conn(0x7f17a815cd50 0x7f17a80c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:11.985 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17a759e640 1 --2- 172.21.15.89:0/115471681 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17a80cefa0 0x7f17a80cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:11.985 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17a5d9b640 1 -- 172.21.15.89:0/115471681 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2073798016 0 0) 0x7f17a815a6c0 con 0x7f17a81538f0 2023-10-30T12:33:11.985 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17a5d9b640 1 -- 172.21.15.89:0/115471681 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1790003530 con 0x7f17a81538f0 2023-10-30T12:33:11.985 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17a5d9b640 1 -- 172.21.15.89:0/115471681 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3381930791 0 0) 0x7f1790003530 con 0x7f17a81538f0 2023-10-30T12:33:11.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17a5d9b640 1 -- 172.21.15.89:0/115471681 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17a80cefa0 msgr2=0x7f17a80cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:11.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17a5d9b640 1 --2- 172.21.15.89:0/115471681 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17a80cefa0 0x7f17a80cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:11.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17a5d9b640 1 -- 172.21.15.89:0/115471681 >> v2:172.21.15.192:3300/0 conn(0x7f17a815cd50 msgr2=0x7f17a80c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:11.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17a5d9b640 1 --2- 172.21.15.89:0/115471681 >> v2:172.21.15.192:3300/0 conn(0x7f17a815cd50 0x7f17a80c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:11.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17a5d9b640 1 -- 172.21.15.89:0/115471681 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a815d8c0 con 0x7f17a81538f0 2023-10-30T12:33:11.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17a5d9b640 1 -- 172.21.15.89:0/115471681 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f1798001a00 con 0x7f17a81538f0 2023-10-30T12:33:11.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17a5d9b640 1 -- 172.21.15.89:0/115471681 >> v1:172.21.15.89:6789/0 conn(0x7f17a81538f0 legacy=0x7f17a8159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:11.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17a5d9b640 1 --2- 172.21.15.89:0/115471681 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17a80cefa0 0x7f1790003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:11.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17a5d9b640 1 --2- 172.21.15.89:0/115471681 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1790003f00 0x7f17900047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:11.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17a5d9b640 1 --2- 172.21.15.89:0/115471681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1790004d10 0x7f1790005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:11.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17a5d9b640 1 -- 172.21.15.89:0/115471681 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a815d300 con 0x7f1790004d10 2023-10-30T12:33:11.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17a5d9b640 1 -- 172.21.15.89:0/115471681 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a815d5e0 con 0x7f1790003f00 2023-10-30T12:33:11.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17a5d9b640 1 -- 172.21.15.89:0/115471681 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f179000b9c0 con 0x7f17a80cefa0 2023-10-30T12:33:11.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17a759e640 1 --2- 172.21.15.89:0/115471681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1790004d10 0x7f1790005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:11.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17a659c640 1 --2- 172.21.15.89:0/115471681 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1790003f00 0x7f17900047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:11.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17a6d9d640 1 --2- 172.21.15.89:0/115471681 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17a80cefa0 0x7f1790003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:11.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17a759e640 1 -- 172.21.15.89:0/115471681 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17a80cefa0 msgr2=0x7f1790003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:11.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17a759e640 1 --2- 172.21.15.89:0/115471681 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17a80cefa0 0x7f1790003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:11.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17a759e640 1 -- 172.21.15.89:0/115471681 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1790003f00 msgr2=0x7f17900047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:11.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17a759e640 1 --2- 172.21.15.89:0/115471681 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1790003f00 0x7f17900047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:11.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.977+0000 7f17a759e640 1 -- 172.21.15.89:0/115471681 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1790003530 con 0x7f1790004d10 2023-10-30T12:33:11.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.981+0000 7f17a759e640 1 --2- 172.21.15.89:0/115471681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1790004d10 0x7f1790005190 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f17a0009ae0 tx=0x7f17a001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89a5a4020cd0c750 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:11.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.981+0000 7f17a5d9b640 1 -- 172.21.15.89:0/115471681 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a00a5640 con 0x7f1790004d10 2023-10-30T12:33:11.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.981+0000 7f17a5d9b640 1 -- 172.21.15.89:0/115471681 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17a00a57a0 con 0x7f1790004d10 2023-10-30T12:33:11.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.981+0000 7f17a5d9b640 1 -- 172.21.15.89:0/115471681 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a00a5b50 con 0x7f1790004d10 2023-10-30T12:33:11.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.981+0000 7f17ac8e9640 1 -- 172.21.15.89:0/115471681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1790004d10 msgr2=0x7f1790005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:11.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.981+0000 7f17ac8e9640 1 --2- 172.21.15.89:0/115471681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1790004d10 0x7f1790005190 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f17a0009ae0 tx=0x7f17a001ca50 comp rx=0 tx=0).stop 2023-10-30T12:33:11.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.981+0000 7f17a759e640 1 -- 172.21.15.89:0/115471681 reap_dead start 2023-10-30T12:33:11.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.981+0000 7f17ac8e9640 1 -- 172.21.15.89:0/115471681 shutdown_connections 2023-10-30T12:33:11.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.981+0000 7f17ac8e9640 1 -- 172.21.15.89:0/115471681 >> 172.21.15.89:0/115471681 conn(0x7f17a806a640 msgr2=0x7f17a8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:11.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.981+0000 7f17ac8e9640 1 -- 172.21.15.89:0/115471681 shutdown_connections 2023-10-30T12:33:11.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.981+0000 7f17ac8e9640 1 -- 172.21.15.89:0/115471681 wait complete. 2023-10-30T12:33:11.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.985+0000 7f17ac8e9640 1 Processor -- start 2023-10-30T12:33:11.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.985+0000 7f17ac8e9640 1 -- start start 2023-10-30T12:33:11.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.985+0000 7f17ac8e9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1790004d10 0x7f17a80b4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:11.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.985+0000 7f17ac8e9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17a815b550 0x7f17a8147760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:11.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.985+0000 7f17ac8e9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17a815bdb0 0x7f17a80b2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:11.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.985+0000 7f17ac8e9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a8158470 con 0x7f17a815b550 2023-10-30T12:33:11.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.985+0000 7f17ac8e9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a8158890 con 0x7f17a815bdb0 2023-10-30T12:33:11.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.985+0000 7f17ac8e9640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17a80b3f40 con 0x7f1790004d10 2023-10-30T12:33:11.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.985+0000 7f17a659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17a815b550 0x7f17a8147760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:11.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.985+0000 7f17a659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17a815b550 0x7f17a8147760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46590/0 (socket says 172.21.15.89:46590) 2023-10-30T12:33:11.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.985+0000 7f17a659c640 1 -- 172.21.15.89:0/2561465161 learned_addr learned my addr 172.21.15.89:0/2561465161 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:33:11.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.985+0000 7f17a6d9d640 1 --2- 172.21.15.89:0/2561465161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1790004d10 0x7f17a80b4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:11.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.985+0000 7f17a759e640 1 --2- 172.21.15.89:0/2561465161 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17a815bdb0 0x7f17a80b2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:11.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.985+0000 7f17a659c640 1 -- 172.21.15.89:0/2561465161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1790004d10 msgr2=0x7f17a80b4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:11.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.985+0000 7f17a659c640 1 --2- 172.21.15.89:0/2561465161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1790004d10 0x7f17a80b4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:11.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.985+0000 7f17a659c640 1 -- 172.21.15.89:0/2561465161 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17a815bdb0 msgr2=0x7f17a80b2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:11.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.985+0000 7f17a659c640 1 --2- 172.21.15.89:0/2561465161 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17a815bdb0 0x7f17a80b2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:11.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.985+0000 7f17a659c640 1 -- 172.21.15.89:0/2561465161 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a8157880 con 0x7f17a815b550 2023-10-30T12:33:11.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.985+0000 7f17a659c640 1 --2- 172.21.15.89:0/2561465161 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17a815b550 0x7f17a8147760 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f179c011a30 tx=0x7f179c011ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:11.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.985+0000 7f178f7fe640 1 -- 172.21.15.89:0/2561465161 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179c00fb50 con 0x7f17a815b550 2023-10-30T12:33:11.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.985+0000 7f178f7fe640 1 -- 172.21.15.89:0/2561465161 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f179c00fcb0 con 0x7f17a815b550 2023-10-30T12:33:11.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.985+0000 7f178f7fe640 1 -- 172.21.15.89:0/2561465161 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179c014440 con 0x7f17a815b550 2023-10-30T12:33:11.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.985+0000 7f17ac8e9640 1 -- 172.21.15.89:0/2561465161 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a815ba40 con 0x7f17a815b550 2023-10-30T12:33:11.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.985+0000 7f17ac8e9640 1 -- 172.21.15.89:0/2561465161 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a8157db0 con 0x7f17a815b550 2023-10-30T12:33:11.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.989+0000 7f17ac8e9640 1 -- 172.21.15.89:0/2561465161 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a804e170 con 0x7f17a815b550 2023-10-30T12:33:12.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.989+0000 7f178f7fe640 1 -- 172.21.15.89:0/2561465161 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f179c0145a0 con 0x7f17a815b550 2023-10-30T12:33:12.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.989+0000 7f178f7fe640 1 --2- 172.21.15.89:0/2561465161 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f17940664a0 0x7f1794068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:12.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.989+0000 7f178f7fe640 1 -- 172.21.15.89:0/2561465161 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5976+0+0 (secure 0 0 0) 0x7f179c0847e0 con 0x7f17a815b550 2023-10-30T12:33:12.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.989+0000 7f17a6d9d640 1 --2- 172.21.15.89:0/2561465161 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f17940664a0 0x7f1794068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:12.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.989+0000 7f17a6d9d640 1 --2- 172.21.15.89:0/2561465161 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f17940664a0 0x7f1794068960 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1798004d50 tx=0x7f1798006040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:12.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:11.993+0000 7f178f7fe640 1 -- 172.21.15.89:0/2561465161 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f179c051c00 con 0x7f17a815b550 2023-10-30T12:33:12.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:12.185+0000 7f17ac8e9640 1 -- 172.21.15.89:0/2561465161 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17a8157180 con 0x7f17a815b550 2023-10-30T12:33:12.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:12.185+0000 7f178f7fe640 1 -- 172.21.15.89:0/2561465161 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f179c055220 con 0x7f17a815b550 2023-10-30T12:33:12.190 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:33:12.191 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:33:10.981346+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:33:12.191 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 24 2023-10-30T12:33:12.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:12.185+0000 7f17ac8e9640 1 -- 172.21.15.89:0/2561465161 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f17940664a0 msgr2=0x7f1794068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:12.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:12.185+0000 7f17ac8e9640 1 --2- 172.21.15.89:0/2561465161 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f17940664a0 0x7f1794068960 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1798004d50 tx=0x7f1798006040 comp rx=0 tx=0).stop 2023-10-30T12:33:12.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:12.185+0000 7f17ac8e9640 1 -- 172.21.15.89:0/2561465161 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17a815b550 msgr2=0x7f17a8147760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:12.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:12.185+0000 7f17ac8e9640 1 --2- 172.21.15.89:0/2561465161 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17a815b550 0x7f17a8147760 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f179c011a30 tx=0x7f179c011ef0 comp rx=0 tx=0).stop 2023-10-30T12:33:12.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:12.185+0000 7f17ac8e9640 1 -- 172.21.15.89:0/2561465161 shutdown_connections 2023-10-30T12:33:12.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:12.185+0000 7f17ac8e9640 1 --2- 172.21.15.89:0/2561465161 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f17940664a0 0x7f1794068960 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:12.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:12.185+0000 7f17ac8e9640 1 --2- 172.21.15.89:0/2561465161 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17a815bdb0 0x7f17a80b2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:12.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:12.185+0000 7f17ac8e9640 1 --2- 172.21.15.89:0/2561465161 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17a815b550 0x7f17a8147760 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:12.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:12.185+0000 7f17ac8e9640 1 --2- 172.21.15.89:0/2561465161 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1790004d10 0x7f17a80b4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:12.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:12.185+0000 7f17ac8e9640 1 -- 172.21.15.89:0/2561465161 >> 172.21.15.89:0/2561465161 conn(0x7f17a806a640 msgr2=0x7f17a806c530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:12.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:12.185+0000 7f17ac8e9640 1 -- 172.21.15.89:0/2561465161 shutdown_connections 2023-10-30T12:33:12.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:12.185+0000 7f17ac8e9640 1 -- 172.21.15.89:0/2561465161 wait complete. 2023-10-30T12:33:12.212 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 24, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:50.729245+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 24, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4262": { "addr": "172.21.15.192:6839/3339589587", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6838", "nonce": 3339589587, "type": "v2" }, { "addr": "172.21.15.192:6839", "nonce": 3339589587, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 9, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4318": { "addr": "172.21.15.192:6837/1002124471", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6836", "nonce": 1002124471, "type": "v2" }, { "addr": "172.21.15.192:6837", "nonce": 1002124471, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4318, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4781": { "addr": "172.21.15.192:6835/654256299", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6834", "nonce": 654256299, "type": "v2" }, { "addr": "172.21.15.192:6835", "nonce": 654256299, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4781, "incarnation": 24, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 29, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:33:10.981346+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4318, "mds_1": 4262, "mds_2": 4781 } } } ], "standbys": [ { "addr": "172.21.15.89:6834/2909790173", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6832", "nonce": 2909790173, "type": "v2" }, { "addr": "172.21.15.89:6834", "nonce": 2909790173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.89:6835/2387357615", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6833", "nonce": 2387357615, "type": "v2" }, { "addr": "172.21.15.89:6835", "nonce": 2387357615, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4239, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:33:14.214 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:33:14.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.353+0000 7f87e7772640 1 Processor -- start 2023-10-30T12:33:14.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.353+0000 7f87e7772640 1 -- start start 2023-10-30T12:33:14.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.357+0000 7f87e7772640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f87e006f7a0 0x7f87e006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:14.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.357+0000 7f87e7772640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87e0066cf0 con 0x7f87e00c6bd0 2023-10-30T12:33:14.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.357+0000 7f87e7772640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87e00c70d0 con 0x7f87e0071110 2023-10-30T12:33:14.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.357+0000 7f87e7772640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87e00700c0 con 0x7f87e006f7a0 2023-10-30T12:33:14.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.357+0000 7f87e4d0d640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f87e00c6bd0 0x7f87e0069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:35886/0 (socket says 172.21.15.89:35886) 2023-10-30T12:33:14.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.357+0000 7f87e4d0d640 1 -- 172.21.15.89:0/3700080496 learned_addr learned my addr 172.21.15.89:0/3700080496 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:33:14.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.357+0000 7f87e550e640 1 --2- 172.21.15.89:0/3700080496 >> v2:172.21.15.192:3300/0 conn(0x7f87e006f7a0 0x7f87e006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:14.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.357+0000 7f87d7fff640 1 -- 172.21.15.89:0/3700080496 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1886323915 0 0) 0x7f87e0066cf0 con 0x7f87e00c6bd0 2023-10-30T12:33:14.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.357+0000 7f87d7fff640 1 -- 172.21.15.89:0/3700080496 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87cc003530 con 0x7f87e00c6bd0 2023-10-30T12:33:14.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.357+0000 7f87d7fff640 1 -- 172.21.15.89:0/3700080496 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2002786766 0 0) 0x7f87cc003530 con 0x7f87e00c6bd0 2023-10-30T12:33:14.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.357+0000 7f87d7fff640 1 -- 172.21.15.89:0/3700080496 >> v2:172.21.15.192:3300/0 conn(0x7f87e006f7a0 msgr2=0x7f87e006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:14.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.357+0000 7f87d7fff640 1 --2- 172.21.15.89:0/3700080496 >> v2:172.21.15.192:3300/0 conn(0x7f87e006f7a0 0x7f87e006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:14.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.357+0000 7f87d7fff640 1 -- 172.21.15.89:0/3700080496 >> v1:172.21.15.192:6789/0 conn(0x7f87e0071110 legacy=0x7f87e0158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:14.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.357+0000 7f87d7fff640 1 -- 172.21.15.89:0/3700080496 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e00c7390 con 0x7f87e00c6bd0 2023-10-30T12:33:14.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.357+0000 7f87d7fff640 1 -- 172.21.15.89:0/3700080496 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f87d0001680 con 0x7f87e00c6bd0 2023-10-30T12:33:14.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.357+0000 7f87d7fff640 1 -- 172.21.15.89:0/3700080496 >> v1:172.21.15.89:6789/0 conn(0x7f87e00c6bd0 legacy=0x7f87e0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:14.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.357+0000 7f87d7fff640 1 --2- 172.21.15.89:0/3700080496 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87cc003980 0x7f87cc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:14.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.357+0000 7f87d7fff640 1 --2- 172.21.15.89:0/3700080496 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87cc0047d0 0x7f87cc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:14.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.357+0000 7f87d7fff640 1 --2- 172.21.15.89:0/3700080496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87cc005190 0x7f87cc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:14.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.357+0000 7f87d7fff640 1 -- 172.21.15.89:0/3700080496 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e00700c0 con 0x7f87cc005190 2023-10-30T12:33:14.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.357+0000 7f87d7fff640 1 -- 172.21.15.89:0/3700080496 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87cc00c040 con 0x7f87cc0047d0 2023-10-30T12:33:14.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.357+0000 7f87d7fff640 1 -- 172.21.15.89:0/3700080496 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87cc00c200 con 0x7f87cc003980 2023-10-30T12:33:14.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.357+0000 7f87e5d0f640 1 --2- 172.21.15.89:0/3700080496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87cc005190 0x7f87cc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:14.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.357+0000 7f87e4d0d640 1 --2- 172.21.15.89:0/3700080496 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87cc0047d0 0x7f87cc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:14.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.357+0000 7f87e550e640 1 --2- 172.21.15.89:0/3700080496 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87cc003980 0x7f87cc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:14.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.357+0000 7f87e5d0f640 1 -- 172.21.15.89:0/3700080496 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87cc003980 msgr2=0x7f87cc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:14.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.357+0000 7f87e5d0f640 1 --2- 172.21.15.89:0/3700080496 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87cc003980 0x7f87cc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:14.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.357+0000 7f87e5d0f640 1 -- 172.21.15.89:0/3700080496 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87cc0047d0 msgr2=0x7f87cc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:14.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.357+0000 7f87e5d0f640 1 --2- 172.21.15.89:0/3700080496 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87cc0047d0 0x7f87cc004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:14.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.357+0000 7f87e5d0f640 1 -- 172.21.15.89:0/3700080496 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87cc003530 con 0x7f87cc005190 2023-10-30T12:33:14.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.361+0000 7f87e5d0f640 1 -- 172.21.15.89:0/3700080496 reap_dead start 2023-10-30T12:33:14.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.361+0000 7f87e5d0f640 1 --2- 172.21.15.89:0/3700080496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87cc005190 0x7f87cc00b990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f87d8001e70 tx=0x7f87d801c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1fc11b4cff0359d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:14.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.361+0000 7f87d7fff640 1 -- 172.21.15.89:0/3700080496 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d8003b80 con 0x7f87cc005190 2023-10-30T12:33:14.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.361+0000 7f87d7fff640 1 -- 172.21.15.89:0/3700080496 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d8003d20 con 0x7f87cc005190 2023-10-30T12:33:14.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.361+0000 7f87d7fff640 1 -- 172.21.15.89:0/3700080496 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d8005260 con 0x7f87cc005190 2023-10-30T12:33:14.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.361+0000 7f87e7772640 1 -- 172.21.15.89:0/3700080496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87cc005190 msgr2=0x7f87cc00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:14.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.361+0000 7f87e7772640 1 --2- 172.21.15.89:0/3700080496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87cc005190 0x7f87cc00b990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f87d8001e70 tx=0x7f87d801c760 comp rx=0 tx=0).stop 2023-10-30T12:33:14.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.361+0000 7f87e7772640 1 -- 172.21.15.89:0/3700080496 shutdown_connections 2023-10-30T12:33:14.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.361+0000 7f87e7772640 1 --2- 172.21.15.89:0/3700080496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87cc005190 0x7f87cc00b990 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:14.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.361+0000 7f87e7772640 1 -- 172.21.15.89:0/3700080496 >> 172.21.15.89:0/3700080496 conn(0x7f87e00c7910 msgr2=0x7f87e00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:14.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.361+0000 7f87e7772640 1 -- 172.21.15.89:0/3700080496 shutdown_connections 2023-10-30T12:33:14.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.361+0000 7f87e7772640 1 -- 172.21.15.89:0/3700080496 wait complete. 2023-10-30T12:33:14.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.361+0000 7f87e7772640 1 Processor -- start 2023-10-30T12:33:14.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.365+0000 7f87e7772640 1 -- start start 2023-10-30T12:33:14.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.365+0000 7f87e7772640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87cc005190 0x7f87e015b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:14.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.365+0000 7f87e7772640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87e0156810 0x7f87e00c8f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:14.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.365+0000 7f87e7772640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87e014b750 0x7f87e014bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:14.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.365+0000 7f87e7772640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e0155550 con 0x7f87cc005190 2023-10-30T12:33:14.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.365+0000 7f87e7772640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e01eed30 con 0x7f87e014b750 2023-10-30T12:33:14.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.365+0000 7f87e7772640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e01ef010 con 0x7f87e0156810 2023-10-30T12:33:14.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.365+0000 7f87e550e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87cc005190 0x7f87e015b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:14.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.365+0000 7f87e550e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87cc005190 0x7f87e015b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46600/0 (socket says 172.21.15.89:46600) 2023-10-30T12:33:14.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.365+0000 7f87e550e640 1 -- 172.21.15.89:0/2278098739 learned_addr learned my addr 172.21.15.89:0/2278098739 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:33:14.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.365+0000 7f87e5d0f640 1 --2- 172.21.15.89:0/2278098739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87e014b750 0x7f87e014bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:14.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.365+0000 7f87e4d0d640 1 --2- 172.21.15.89:0/2278098739 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87e0156810 0x7f87e00c8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:14.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.365+0000 7f87e550e640 1 -- 172.21.15.89:0/2278098739 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87e0156810 msgr2=0x7f87e00c8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:14.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.365+0000 7f87e550e640 1 --2- 172.21.15.89:0/2278098739 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87e0156810 0x7f87e00c8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:14.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.365+0000 7f87e550e640 1 -- 172.21.15.89:0/2278098739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87e014b750 msgr2=0x7f87e014bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:14.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.365+0000 7f87e550e640 1 --2- 172.21.15.89:0/2278098739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87e014b750 0x7f87e014bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:14.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.365+0000 7f87e550e640 1 -- 172.21.15.89:0/2278098739 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e01ef2f0 con 0x7f87cc005190 2023-10-30T12:33:14.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.365+0000 7f87e550e640 1 --2- 172.21.15.89:0/2278098739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87cc005190 0x7f87e015b2e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f87dc011960 tx=0x7f87dc011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:14.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.365+0000 7f87d5ffb640 1 -- 172.21.15.89:0/2278098739 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc00faf0 con 0x7f87cc005190 2023-10-30T12:33:14.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.365+0000 7f87d5ffb640 1 -- 172.21.15.89:0/2278098739 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87dc00fc50 con 0x7f87cc005190 2023-10-30T12:33:14.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.365+0000 7f87e7772640 1 -- 172.21.15.89:0/2278098739 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87e01ef5d0 con 0x7f87cc005190 2023-10-30T12:33:14.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.365+0000 7f87d5ffb640 1 -- 172.21.15.89:0/2278098739 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc014440 con 0x7f87cc005190 2023-10-30T12:33:14.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.365+0000 7f87e7772640 1 -- 172.21.15.89:0/2278098739 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87e01ef8b0 con 0x7f87cc005190 2023-10-30T12:33:14.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.365+0000 7f87e7772640 1 -- 172.21.15.89:0/2278098739 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87e004e170 con 0x7f87cc005190 2023-10-30T12:33:14.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.369+0000 7f87d5ffb640 1 -- 172.21.15.89:0/2278098739 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f87dc0145a0 con 0x7f87cc005190 2023-10-30T12:33:14.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.369+0000 7f87d5ffb640 1 --2- 172.21.15.89:0/2278098739 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f87c40664a0 0x7f87c4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:14.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.369+0000 7f87d5ffb640 1 -- 172.21.15.89:0/2278098739 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5976+0+0 (secure 0 0 0) 0x7f87dc084700 con 0x7f87cc005190 2023-10-30T12:33:14.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.369+0000 7f87e4d0d640 1 --2- 172.21.15.89:0/2278098739 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f87c40664a0 0x7f87c4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:14.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.369+0000 7f87e4d0d640 1 --2- 172.21.15.89:0/2278098739 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f87c40664a0 0x7f87c4068960 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f87d0001060 tx=0x7f87d0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:14.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.373+0000 7f87d5ffb640 1 -- 172.21.15.89:0/2278098739 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f87dc051b00 con 0x7f87cc005190 2023-10-30T12:33:14.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.557+0000 7f87e7772640 1 -- 172.21.15.89:0/2278098739 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87e0070600 con 0x7f87cc005190 2023-10-30T12:33:14.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.557+0000 7f87d5ffb640 1 -- 172.21.15.89:0/2278098739 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f87dc055120 con 0x7f87cc005190 2023-10-30T12:33:14.564 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:33:14.565 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:33:10.981346+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:33:14.565 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 24 2023-10-30T12:33:14.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.561+0000 7f87e7772640 1 -- 172.21.15.89:0/2278098739 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f87c40664a0 msgr2=0x7f87c4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:14.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.561+0000 7f87e7772640 1 --2- 172.21.15.89:0/2278098739 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f87c40664a0 0x7f87c4068960 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f87d0001060 tx=0x7f87d0009040 comp rx=0 tx=0).stop 2023-10-30T12:33:14.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.561+0000 7f87e7772640 1 -- 172.21.15.89:0/2278098739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87cc005190 msgr2=0x7f87e015b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:14.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.561+0000 7f87e7772640 1 --2- 172.21.15.89:0/2278098739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87cc005190 0x7f87e015b2e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f87dc011960 tx=0x7f87dc011e20 comp rx=0 tx=0).stop 2023-10-30T12:33:14.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.561+0000 7f87e7772640 1 -- 172.21.15.89:0/2278098739 shutdown_connections 2023-10-30T12:33:14.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.561+0000 7f87e7772640 1 --2- 172.21.15.89:0/2278098739 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f87c40664a0 0x7f87c4068960 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:14.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.561+0000 7f87e7772640 1 --2- 172.21.15.89:0/2278098739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87e014b750 0x7f87e014bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:14.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.561+0000 7f87e7772640 1 --2- 172.21.15.89:0/2278098739 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87e0156810 0x7f87e00c8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:14.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.561+0000 7f87e7772640 1 --2- 172.21.15.89:0/2278098739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87cc005190 0x7f87e015b2e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:14.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.561+0000 7f87e7772640 1 -- 172.21.15.89:0/2278098739 >> 172.21.15.89:0/2278098739 conn(0x7f87e00c7910 msgr2=0x7f87e00c5500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:14.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.561+0000 7f87e7772640 1 -- 172.21.15.89:0/2278098739 shutdown_connections 2023-10-30T12:33:14.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:14.561+0000 7f87e7772640 1 -- 172.21.15.89:0/2278098739 wait complete. 2023-10-30T12:33:14.585 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:33:15.517 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:33:15.521 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:33:16.586 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:33:16.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.729+0000 7f64e1691640 1 Processor -- start 2023-10-30T12:33:16.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.729+0000 7f64e1691640 1 -- start start 2023-10-30T12:33:16.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.729+0000 7f64e1691640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64dc0c6b80 0x7f64dc0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:16.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.729+0000 7f64e1691640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f64dc13f4a0 0x7f64dc0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:16.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.729+0000 7f64e1691640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64dc0c0a50 con 0x7f64dc0c6f60 2023-10-30T12:33:16.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.729+0000 7f64e1691640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64dc06f6e0 con 0x7f64dc13f4a0 2023-10-30T12:33:16.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.729+0000 7f64e1691640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc06f9c0 con 0x7f64dc0c6b80 2023-10-30T12:33:16.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.729+0000 7f64daffd640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64dc0c6b80 0x7f64dc0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:16.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.729+0000 7f64db7fe640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f64dc13f4a0 0x7f64dc0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:16.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.729+0000 7f64daffd640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64dc0c6b80 0x7f64dc0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:37992/0 (socket says 172.21.15.89:37992) 2023-10-30T12:33:16.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.729+0000 7f64db7fe640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f64dc13f4a0 0x7f64dc0c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:48010/0 (socket says 172.21.15.89:48010) 2023-10-30T12:33:16.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.729+0000 7f64daffd640 1 -- 172.21.15.89:0/1318424710 learned_addr learned my addr 172.21.15.89:0/1318424710 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:33:16.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.729+0000 7f64d3fff640 1 --1- 172.21.15.89:0/1318424710 >> v1:172.21.15.192:6789/0 conn(0x7f64dc0c6f60 0x7f64dc13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:48930/0 (socket says 172.21.15.89:48930) 2023-10-30T12:33:16.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.733+0000 7f64db7fe640 1 -- 172.21.15.89:0/1318424710 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64dc0c6b80 msgr2=0x7f64dc0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:16.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.733+0000 7f64db7fe640 1 --2- 172.21.15.89:0/1318424710 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64dc0c6b80 0x7f64dc0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:16.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.733+0000 7f64db7fe640 1 -- 172.21.15.89:0/1318424710 >> v1:172.21.15.192:6789/0 conn(0x7f64dc0c6f60 legacy=0x7f64dc13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:16.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.733+0000 7f64db7fe640 1 -- 172.21.15.89:0/1318424710 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64dc06fca0 con 0x7f64dc13f4a0 2023-10-30T12:33:16.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.733+0000 7f64db7fe640 1 --2- 172.21.15.89:0/1318424710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64dc13f4a0 0x7f64dc0c0400 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f64d4009d90 tx=0x7f64d401eb60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4927a55eef16bc19 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:16.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.733+0000 7f64da7fc640 1 -- 172.21.15.89:0/1318424710 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d40a7b40 con 0x7f64dc13f4a0 2023-10-30T12:33:16.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.733+0000 7f64da7fc640 1 -- 172.21.15.89:0/1318424710 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d40a7ca0 con 0x7f64dc13f4a0 2023-10-30T12:33:16.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.733+0000 7f64da7fc640 1 -- 172.21.15.89:0/1318424710 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d40af660 con 0x7f64dc13f4a0 2023-10-30T12:33:16.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.737+0000 7f64e1691640 1 -- 172.21.15.89:0/1318424710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64dc13f4a0 msgr2=0x7f64dc0c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:16.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.737+0000 7f64e1691640 1 --2- 172.21.15.89:0/1318424710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64dc13f4a0 0x7f64dc0c0400 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f64d4009d90 tx=0x7f64d401eb60 comp rx=0 tx=0).stop 2023-10-30T12:33:16.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.737+0000 7f64e1691640 1 -- 172.21.15.89:0/1318424710 shutdown_connections 2023-10-30T12:33:16.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.737+0000 7f64e1691640 1 --2- 172.21.15.89:0/1318424710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64dc13f4a0 0x7f64dc0c0400 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:16.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.737+0000 7f64e1691640 1 --2- 172.21.15.89:0/1318424710 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64dc0c6b80 0x7f64dc0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:16.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.737+0000 7f64e1691640 1 -- 172.21.15.89:0/1318424710 >> 172.21.15.89:0/1318424710 conn(0x7f64dc0c7910 msgr2=0x7f64dc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:16.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.737+0000 7f64e1691640 1 -- 172.21.15.89:0/1318424710 shutdown_connections 2023-10-30T12:33:16.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.737+0000 7f64e1691640 1 -- 172.21.15.89:0/1318424710 wait complete. 2023-10-30T12:33:16.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.737+0000 7f64e1691640 1 Processor -- start 2023-10-30T12:33:16.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.737+0000 7f64e1691640 1 -- start start 2023-10-30T12:33:16.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.737+0000 7f64e1691640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64dc0c6b80 0x7f64dc157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:16.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.737+0000 7f64e1691640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64dc0c6f60 0x7f64dc1f5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:16.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.737+0000 7f64e1691640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64dc13f4a0 0x7f64dc0c9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:16.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.737+0000 7f64e1691640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc1595d0 con 0x7f64dc13f4a0 2023-10-30T12:33:16.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.737+0000 7f64e1691640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc1f5240 con 0x7f64dc0c6b80 2023-10-30T12:33:16.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.737+0000 7f64e1691640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc15f2f0 con 0x7f64dc0c6f60 2023-10-30T12:33:16.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.737+0000 7f64db7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64dc13f4a0 0x7f64dc0c9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:16.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.737+0000 7f64db7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64dc13f4a0 0x7f64dc0c9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46612/0 (socket says 172.21.15.89:46612) 2023-10-30T12:33:16.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.737+0000 7f64db7fe640 1 -- 172.21.15.89:0/1122333550 learned_addr learned my addr 172.21.15.89:0/1122333550 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:33:16.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.737+0000 7f64daffd640 1 --2- 172.21.15.89:0/1122333550 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64dc0c6b80 0x7f64dc157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:16.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.737+0000 7f64d3fff640 1 --2- 172.21.15.89:0/1122333550 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64dc0c6f60 0x7f64dc1f5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:16.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.737+0000 7f64db7fe640 1 -- 172.21.15.89:0/1122333550 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64dc0c6f60 msgr2=0x7f64dc1f5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:16.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.737+0000 7f64db7fe640 1 --2- 172.21.15.89:0/1122333550 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64dc0c6f60 0x7f64dc1f5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:16.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.737+0000 7f64db7fe640 1 -- 172.21.15.89:0/1122333550 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64dc0c6b80 msgr2=0x7f64dc157790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.737+0000 7f64db7fe640 1 --2- 172.21.15.89:0/1122333550 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64dc0c6b80 0x7f64dc157790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.737+0000 7f64db7fe640 1 -- 172.21.15.89:0/1122333550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d4009a70 con 0x7f64dc13f4a0 2023-10-30T12:33:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.737+0000 7f64daffd640 1 --2- 172.21.15.89:0/1122333550 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64dc0c6b80 0x7f64dc157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:33:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.741+0000 7f64db7fe640 1 --2- 172.21.15.89:0/1122333550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64dc13f4a0 0x7f64dc0c9ff0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f64d40b2040 tx=0x7f64d401b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.741+0000 7f64d37fe640 1 -- 172.21.15.89:0/1122333550 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d4009450 con 0x7f64dc13f4a0 2023-10-30T12:33:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.741+0000 7f64d37fe640 1 -- 172.21.15.89:0/1122333550 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d40a7940 con 0x7f64dc13f4a0 2023-10-30T12:33:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.741+0000 7f64d37fe640 1 -- 172.21.15.89:0/1122333550 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d40b8660 con 0x7f64dc13f4a0 2023-10-30T12:33:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.741+0000 7f64e1691640 1 -- 172.21.15.89:0/1122333550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64dc159870 con 0x7f64dc13f4a0 2023-10-30T12:33:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.741+0000 7f64e1691640 1 -- 172.21.15.89:0/1122333550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64dc1f57c0 con 0x7f64dc13f4a0 2023-10-30T12:33:16.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.741+0000 7f64e1691640 1 -- 172.21.15.89:0/1122333550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64dc1f59b0 con 0x7f64dc13f4a0 2023-10-30T12:33:16.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.741+0000 7f64d37fe640 1 -- 172.21.15.89:0/1122333550 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f64d40b6070 con 0x7f64dc13f4a0 2023-10-30T12:33:16.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.741+0000 7f64d37fe640 1 --2- 172.21.15.89:0/1122333550 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f64bc0664a0 0x7f64bc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:16.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.741+0000 7f64d37fe640 1 -- 172.21.15.89:0/1122333550 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5976+0+0 (secure 0 0 0) 0x7f64d411ef90 con 0x7f64dc13f4a0 2023-10-30T12:33:16.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.745+0000 7f64daffd640 1 --2- 172.21.15.89:0/1122333550 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f64bc0664a0 0x7f64bc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:16.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.745+0000 7f64daffd640 1 --2- 172.21.15.89:0/1122333550 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f64bc0664a0 0x7f64bc068960 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f64cc009ba0 tx=0x7f64cc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:16.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.749+0000 7f64d37fe640 1 -- 172.21.15.89:0/1122333550 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f64d40ec1d0 con 0x7f64dc13f4a0 2023-10-30T12:33:16.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.925+0000 7f64e1691640 1 -- 172.21.15.89:0/1122333550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64dc04e170 con 0x7f64dc13f4a0 2023-10-30T12:33:16.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.929+0000 7f64d37fe640 1 -- 172.21.15.89:0/1122333550 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f64d40ef7f0 con 0x7f64dc13f4a0 2023-10-30T12:33:16.933 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:33:16.933 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:33:10.981346+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:33:16.935 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 24 2023-10-30T12:33:16.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.929+0000 7f64e1691640 1 -- 172.21.15.89:0/1122333550 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f64bc0664a0 msgr2=0x7f64bc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:16.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.929+0000 7f64e1691640 1 --2- 172.21.15.89:0/1122333550 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f64bc0664a0 0x7f64bc068960 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f64cc009ba0 tx=0x7f64cc008040 comp rx=0 tx=0).stop 2023-10-30T12:33:16.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.929+0000 7f64e1691640 1 -- 172.21.15.89:0/1122333550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64dc13f4a0 msgr2=0x7f64dc0c9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:16.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.929+0000 7f64e1691640 1 --2- 172.21.15.89:0/1122333550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64dc13f4a0 0x7f64dc0c9ff0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f64d40b2040 tx=0x7f64d401b900 comp rx=0 tx=0).stop 2023-10-30T12:33:16.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.929+0000 7f64e1691640 1 -- 172.21.15.89:0/1122333550 shutdown_connections 2023-10-30T12:33:16.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.929+0000 7f64e1691640 1 --2- 172.21.15.89:0/1122333550 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f64bc0664a0 0x7f64bc068960 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:16.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.929+0000 7f64e1691640 1 --2- 172.21.15.89:0/1122333550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64dc13f4a0 0x7f64dc0c9ff0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:16.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.929+0000 7f64e1691640 1 --2- 172.21.15.89:0/1122333550 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64dc0c6f60 0x7f64dc1f5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:16.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.929+0000 7f64e1691640 1 --2- 172.21.15.89:0/1122333550 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64dc0c6b80 0x7f64dc157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:16.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.929+0000 7f64e1691640 1 -- 172.21.15.89:0/1122333550 >> 172.21.15.89:0/1122333550 conn(0x7f64dc0c7910 msgr2=0x7f64dc1391e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:16.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.929+0000 7f64e1691640 1 -- 172.21.15.89:0/1122333550 shutdown_connections 2023-10-30T12:33:16.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:16.929+0000 7f64e1691640 1 -- 172.21.15.89:0/1122333550 wait complete. 2023-10-30T12:33:16.954 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:33:18.955 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:33:19.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.097+0000 7facca5c4640 1 Processor -- start 2023-10-30T12:33:19.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.097+0000 7facca5c4640 1 -- start start 2023-10-30T12:33:19.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.101+0000 7facca5c4640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facc4159e60 0x7facc406f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:19.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.101+0000 7facca5c4640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7facc40bfec0 0x7facc40c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:19.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.101+0000 7facca5c4640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc406fa80 con 0x7facc406a720 2023-10-30T12:33:19.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.101+0000 7facca5c4640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facc4071290 con 0x7facc40bfec0 2023-10-30T12:33:19.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.101+0000 7facca5c4640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facc4071570 con 0x7facc4159e60 2023-10-30T12:33:19.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.101+0000 7facc37fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7facc40bfec0 0x7facc40c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:19.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.101+0000 7facc37fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7facc40bfec0 0x7facc40c1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46628/0 (socket says 172.21.15.89:46628) 2023-10-30T12:33:19.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.101+0000 7facc37fe640 1 -- 172.21.15.89:0/714046950 learned_addr learned my addr 172.21.15.89:0/714046950 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:33:19.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.101+0000 7facc3fff640 1 --2- 172.21.15.89:0/714046950 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facc4159e60 0x7facc406f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:19.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.101+0000 7facc37fe640 1 -- 172.21.15.89:0/714046950 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facc4159e60 msgr2=0x7facc406f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:19.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.101+0000 7facc37fe640 1 --2- 172.21.15.89:0/714046950 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facc4159e60 0x7facc406f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:19.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.101+0000 7facc37fe640 1 -- 172.21.15.89:0/714046950 >> v1:172.21.15.192:6789/0 conn(0x7facc406a720 legacy=0x7facc4155d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:19.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.101+0000 7facc37fe640 1 -- 172.21.15.89:0/714046950 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc4071850 con 0x7facc40bfec0 2023-10-30T12:33:19.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.105+0000 7facc37fe640 1 --2- 172.21.15.89:0/714046950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facc40bfec0 0x7facc40c1a40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7facb800aa10 tx=0x7facb8020100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd90d9ae6b460794 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:19.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.105+0000 7facc2ffd640 1 -- 172.21.15.89:0/714046950 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb80b0600 con 0x7facc40bfec0 2023-10-30T12:33:19.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.105+0000 7facc2ffd640 1 -- 172.21.15.89:0/714046950 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb80ac050 con 0x7facc40bfec0 2023-10-30T12:33:19.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.105+0000 7facc2ffd640 1 -- 172.21.15.89:0/714046950 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb80b0980 con 0x7facc40bfec0 2023-10-30T12:33:19.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.105+0000 7facca5c4640 1 -- 172.21.15.89:0/714046950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facc40bfec0 msgr2=0x7facc40c1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:19.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.105+0000 7facca5c4640 1 --2- 172.21.15.89:0/714046950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facc40bfec0 0x7facc40c1a40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7facb800aa10 tx=0x7facb8020100 comp rx=0 tx=0).stop 2023-10-30T12:33:19.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.105+0000 7facca5c4640 1 -- 172.21.15.89:0/714046950 shutdown_connections 2023-10-30T12:33:19.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.105+0000 7facca5c4640 1 --2- 172.21.15.89:0/714046950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facc40bfec0 0x7facc40c1a40 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:19.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.105+0000 7facca5c4640 1 --2- 172.21.15.89:0/714046950 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facc4159e60 0x7facc406f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:19.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.105+0000 7facca5c4640 1 -- 172.21.15.89:0/714046950 >> 172.21.15.89:0/714046950 conn(0x7facc40c7910 msgr2=0x7facc40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:19.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.105+0000 7facca5c4640 1 -- 172.21.15.89:0/714046950 shutdown_connections 2023-10-30T12:33:19.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.105+0000 7facca5c4640 1 -- 172.21.15.89:0/714046950 wait complete. 2023-10-30T12:33:19.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.105+0000 7facca5c4640 1 Processor -- start 2023-10-30T12:33:19.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.105+0000 7facca5c4640 1 -- start start 2023-10-30T12:33:19.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.109+0000 7facca5c4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7facc406a720 0x7facc41ec8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:19.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.109+0000 7facca5c4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facc40bfec0 0x7facc40c66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:19.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.109+0000 7facca5c4640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facc4159e60 0x7facc41e17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:19.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.109+0000 7facca5c4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc40c4350 con 0x7facc40bfec0 2023-10-30T12:33:19.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.109+0000 7facca5c4640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc41e1320 con 0x7facc406a720 2023-10-30T12:33:19.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.109+0000 7facca5c4640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facc41ecee0 con 0x7facc4159e60 2023-10-30T12:33:19.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.109+0000 7facc37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facc40bfec0 0x7facc40c66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:19.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.109+0000 7facc37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facc40bfec0 0x7facc40c66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46638/0 (socket says 172.21.15.89:46638) 2023-10-30T12:33:19.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.109+0000 7facc37fe640 1 -- 172.21.15.89:0/297762078 learned_addr learned my addr 172.21.15.89:0/297762078 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:33:19.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.109+0000 7facc3fff640 1 --2- 172.21.15.89:0/297762078 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7facc406a720 0x7facc41ec8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:19.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.109+0000 7facc8b61640 1 --2- 172.21.15.89:0/297762078 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facc4159e60 0x7facc41e17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:19.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.109+0000 7facc37fe640 1 -- 172.21.15.89:0/297762078 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facc4159e60 msgr2=0x7facc41e17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:19.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.109+0000 7facc37fe640 1 --2- 172.21.15.89:0/297762078 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facc4159e60 0x7facc41e17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:19.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.109+0000 7facc37fe640 1 -- 172.21.15.89:0/297762078 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7facc406a720 msgr2=0x7facc41ec8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:19.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.109+0000 7facc37fe640 1 --2- 172.21.15.89:0/297762078 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7facc406a720 0x7facc41ec8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:19.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.109+0000 7facc37fe640 1 -- 172.21.15.89:0/297762078 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb800a670 con 0x7facc40bfec0 2023-10-30T12:33:19.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.109+0000 7facc8b61640 1 --2- 172.21.15.89:0/297762078 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facc4159e60 0x7facc41e17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:33:19.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.109+0000 7facc37fe640 1 --2- 172.21.15.89:0/297762078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facc40bfec0 0x7facc40c66a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7facb800a1c0 tx=0x7facb801e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:19.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.109+0000 7facc0ff9640 1 -- 172.21.15.89:0/297762078 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb80bf070 con 0x7facc40bfec0 2023-10-30T12:33:19.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.109+0000 7facc0ff9640 1 -- 172.21.15.89:0/297762078 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb80b9630 con 0x7facc40bfec0 2023-10-30T12:33:19.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.109+0000 7facc0ff9640 1 -- 172.21.15.89:0/297762078 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb80b99c0 con 0x7facc40bfec0 2023-10-30T12:33:19.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.109+0000 7facca5c4640 1 -- 172.21.15.89:0/297762078 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facc41ed1a0 con 0x7facc40bfec0 2023-10-30T12:33:19.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.109+0000 7facca5c4640 1 -- 172.21.15.89:0/297762078 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facc41e0fb0 con 0x7facc40bfec0 2023-10-30T12:33:19.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.109+0000 7facca5c4640 1 -- 172.21.15.89:0/297762078 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facc404e170 con 0x7facc40bfec0 2023-10-30T12:33:19.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.109+0000 7facc0ff9640 1 -- 172.21.15.89:0/297762078 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7facb80b7070 con 0x7facc40bfec0 2023-10-30T12:33:19.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.113+0000 7facc0ff9640 1 --2- 172.21.15.89:0/297762078 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7facb00664a0 0x7facb0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:19.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.113+0000 7facc0ff9640 1 -- 172.21.15.89:0/297762078 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5976+0+0 (secure 0 0 0) 0x7facb8121e90 con 0x7facc40bfec0 2023-10-30T12:33:19.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.113+0000 7facc3fff640 1 --2- 172.21.15.89:0/297762078 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7facb00664a0 0x7facb0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:19.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.113+0000 7facc3fff640 1 --2- 172.21.15.89:0/297762078 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7facb00664a0 0x7facb0068960 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7facb400b930 tx=0x7facb4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:19.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.113+0000 7facc0ff9640 1 -- 172.21.15.89:0/297762078 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7facb80ef1f0 con 0x7facc40bfec0 2023-10-30T12:33:19.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.297+0000 7facca5c4640 1 -- 172.21.15.89:0/297762078 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7facc41e1460 con 0x7facc40bfec0 2023-10-30T12:33:19.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.301+0000 7facc0ff9640 1 -- 172.21.15.89:0/297762078 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+5741 (secure 0 0 0) 0x7facb80f2810 con 0x7facc40bfec0 2023-10-30T12:33:19.305 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:33:19.306 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:33:16.428171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:resolve","state_seq":21,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:33:19.307 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 25 2023-10-30T12:33:19.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.301+0000 7facca5c4640 1 -- 172.21.15.89:0/297762078 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7facb00664a0 msgr2=0x7facb0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:19.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.301+0000 7facca5c4640 1 --2- 172.21.15.89:0/297762078 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7facb00664a0 0x7facb0068960 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7facb400b930 tx=0x7facb4008040 comp rx=0 tx=0).stop 2023-10-30T12:33:19.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.301+0000 7facca5c4640 1 -- 172.21.15.89:0/297762078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facc40bfec0 msgr2=0x7facc40c66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:19.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.301+0000 7facca5c4640 1 --2- 172.21.15.89:0/297762078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facc40bfec0 0x7facc40c66a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7facb800a1c0 tx=0x7facb801e940 comp rx=0 tx=0).stop 2023-10-30T12:33:19.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.301+0000 7facca5c4640 1 -- 172.21.15.89:0/297762078 shutdown_connections 2023-10-30T12:33:19.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.301+0000 7facca5c4640 1 --2- 172.21.15.89:0/297762078 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7facb00664a0 0x7facb0068960 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:19.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.301+0000 7facca5c4640 1 --2- 172.21.15.89:0/297762078 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facc4159e60 0x7facc41e17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:19.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.301+0000 7facca5c4640 1 --2- 172.21.15.89:0/297762078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facc40bfec0 0x7facc40c66a0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:19.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.301+0000 7facca5c4640 1 --2- 172.21.15.89:0/297762078 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7facc406a720 0x7facc41ec8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:19.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.301+0000 7facca5c4640 1 -- 172.21.15.89:0/297762078 >> 172.21.15.89:0/297762078 conn(0x7facc40c7910 msgr2=0x7facc40ca270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:19.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.301+0000 7facca5c4640 1 -- 172.21.15.89:0/297762078 shutdown_connections 2023-10-30T12:33:19.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:19.301+0000 7facca5c4640 1 -- 172.21.15.89:0/297762078 wait complete. 2023-10-30T12:33:19.326 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:33:21.327 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:33:21.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.469+0000 7f598ba54640 1 Processor -- start 2023-10-30T12:33:21.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f598ba54640 1 -- start start 2023-10-30T12:33:21.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f598ba54640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f59840c6f60 0x7f59841566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:21.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f598ba54640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59840c0a50 con 0x7f59840c6b80 2023-10-30T12:33:21.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f598ba54640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f598406f6e0 con 0x7f59840c05d0 2023-10-30T12:33:21.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f598ba54640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f598406f9c0 con 0x7f59840c6f60 2023-10-30T12:33:21.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f59897f0640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f59840c6b80 0x7f59840bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38078/0 (socket says 172.21.15.89:38078) 2023-10-30T12:33:21.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f59897f0640 1 -- 172.21.15.89:0/1178049977 learned_addr learned my addr 172.21.15.89:0/1178049977 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:33:21.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f5989ff1640 1 --2- 172.21.15.89:0/1178049977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f59840c6f60 0x7f59841566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:21.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f597bfff640 1 -- 172.21.15.89:0/1178049977 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3309516043 0 0) 0x7f59840c0a50 con 0x7f59840c6b80 2023-10-30T12:33:21.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f597bfff640 1 -- 172.21.15.89:0/1178049977 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5970005740 con 0x7f59840c6b80 2023-10-30T12:33:21.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f597bfff640 1 -- 172.21.15.89:0/1178049977 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3247825646 0 0) 0x7f5970005740 con 0x7f59840c6b80 2023-10-30T12:33:21.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f597bfff640 1 -- 172.21.15.89:0/1178049977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f59840c6f60 msgr2=0x7f59841566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:21.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f597bfff640 1 --2- 172.21.15.89:0/1178049977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f59840c6f60 0x7f59841566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:21.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f597bfff640 1 -- 172.21.15.89:0/1178049977 >> v1:172.21.15.192:6789/0 conn(0x7f59840c05d0 legacy=0x7f598413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:21.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f597bfff640 1 -- 172.21.15.89:0/1178049977 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598406fca0 con 0x7f59840c6b80 2023-10-30T12:33:21.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f597bfff640 1 -- 172.21.15.89:0/1178049977 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f5980001680 con 0x7f59840c6b80 2023-10-30T12:33:21.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f597bfff640 1 -- 172.21.15.89:0/1178049977 >> v1:172.21.15.89:6789/0 conn(0x7f59840c6b80 legacy=0x7f59840bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:21.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f597bfff640 1 --2- 172.21.15.89:0/1178049977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5970005bc0 0x7f5970005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:21.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f597bfff640 1 --2- 172.21.15.89:0/1178049977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59700069e0 0x7f5970006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:21.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f597bfff640 1 --2- 172.21.15.89:0/1178049977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f59840c6f60 0x7f59700073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:21.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f597bfff640 1 -- 172.21.15.89:0/1178049977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598406f9c0 con 0x7f59700069e0 2023-10-30T12:33:21.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f597bfff640 1 -- 172.21.15.89:0/1178049977 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597000dc50 con 0x7f5970005bc0 2023-10-30T12:33:21.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f597bfff640 1 -- 172.21.15.89:0/1178049977 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f597000de10 con 0x7f59840c6f60 2023-10-30T12:33:21.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f5988fef640 1 --2- 172.21.15.89:0/1178049977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59700069e0 0x7f5970006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:21.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f59897f0640 1 --2- 172.21.15.89:0/1178049977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5970005bc0 0x7f5970005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:21.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f5989ff1640 1 --2- 172.21.15.89:0/1178049977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f59840c6f60 0x7f59700073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:21.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f5988fef640 1 -- 172.21.15.89:0/1178049977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f59840c6f60 msgr2=0x7f59700073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:21.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f5988fef640 1 --2- 172.21.15.89:0/1178049977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f59840c6f60 0x7f59700073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:21.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f5988fef640 1 -- 172.21.15.89:0/1178049977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5970005bc0 msgr2=0x7f5970005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:21.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f5988fef640 1 --2- 172.21.15.89:0/1178049977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5970005bc0 0x7f5970005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:21.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.473+0000 7f5988fef640 1 -- 172.21.15.89:0/1178049977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5970005740 con 0x7f59700069e0 2023-10-30T12:33:21.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.477+0000 7f5988fef640 1 --2- 172.21.15.89:0/1178049977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59700069e0 0x7f5970006e60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f598406c9a0 tx=0x7f597401c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf70e86fbd97a400 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:21.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.477+0000 7f597bfff640 1 -- 172.21.15.89:0/1178049977 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59740052d0 con 0x7f59700069e0 2023-10-30T12:33:21.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.477+0000 7f597bfff640 1 -- 172.21.15.89:0/1178049977 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59740a7050 con 0x7f59700069e0 2023-10-30T12:33:21.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.477+0000 7f597bfff640 1 -- 172.21.15.89:0/1178049977 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5974003600 con 0x7f59700069e0 2023-10-30T12:33:21.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.477+0000 7f598ba54640 1 -- 172.21.15.89:0/1178049977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59700069e0 msgr2=0x7f5970006e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:21.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.477+0000 7f598ba54640 1 --2- 172.21.15.89:0/1178049977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59700069e0 0x7f5970006e60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f598406c9a0 tx=0x7f597401c380 comp rx=0 tx=0).stop 2023-10-30T12:33:21.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.477+0000 7f5989ff1640 1 -- 172.21.15.89:0/1178049977 reap_dead start 2023-10-30T12:33:21.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.477+0000 7f598ba54640 1 -- 172.21.15.89:0/1178049977 shutdown_connections 2023-10-30T12:33:21.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.477+0000 7f598ba54640 1 -- 172.21.15.89:0/1178049977 >> 172.21.15.89:0/1178049977 conn(0x7f59840c7910 msgr2=0x7f59840c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:21.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.477+0000 7f598ba54640 1 -- 172.21.15.89:0/1178049977 shutdown_connections 2023-10-30T12:33:21.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.477+0000 7f598ba54640 1 -- 172.21.15.89:0/1178049977 wait complete. 2023-10-30T12:33:21.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.481+0000 7f598ba54640 1 Processor -- start 2023-10-30T12:33:21.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.481+0000 7f598ba54640 1 -- start start 2023-10-30T12:33:21.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.481+0000 7f598ba54640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59700069e0 0x7f59841e4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:21.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.481+0000 7f598ba54640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59840c8e20 0x7f59841eee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:21.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.481+0000 7f598ba54640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f59841ef370 0x7f5984136950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:21.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.481+0000 7f598ba54640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598413b330 con 0x7f59840c8e20 2023-10-30T12:33:21.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.481+0000 7f598ba54640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59840c95f0 con 0x7f59700069e0 2023-10-30T12:33:21.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.481+0000 7f598ba54640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f598413aea0 con 0x7f59841ef370 2023-10-30T12:33:21.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.481+0000 7f5988fef640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59840c8e20 0x7f59841eee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:21.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.481+0000 7f5988fef640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59840c8e20 0x7f59841eee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37406/0 (socket says 172.21.15.89:37406) 2023-10-30T12:33:21.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.481+0000 7f5988fef640 1 -- 172.21.15.89:0/4078725465 learned_addr learned my addr 172.21.15.89:0/4078725465 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:33:21.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.481+0000 7f59897f0640 1 --2- 172.21.15.89:0/4078725465 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59700069e0 0x7f59841e4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:21.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.481+0000 7f5989ff1640 1 --2- 172.21.15.89:0/4078725465 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f59841ef370 0x7f5984136950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:21.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.481+0000 7f5988fef640 1 -- 172.21.15.89:0/4078725465 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f59841ef370 msgr2=0x7f5984136950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:21.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.481+0000 7f5988fef640 1 --2- 172.21.15.89:0/4078725465 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f59841ef370 0x7f5984136950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:21.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.481+0000 7f5988fef640 1 -- 172.21.15.89:0/4078725465 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59700069e0 msgr2=0x7f59841e4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:21.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.481+0000 7f5988fef640 1 --2- 172.21.15.89:0/4078725465 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59700069e0 0x7f59841e4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:21.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.481+0000 7f5988fef640 1 -- 172.21.15.89:0/4078725465 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5984159310 con 0x7f59840c8e20 2023-10-30T12:33:21.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.481+0000 7f59897f0640 1 --2- 172.21.15.89:0/4078725465 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59700069e0 0x7f59841e4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:33:21.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.481+0000 7f5988fef640 1 --2- 172.21.15.89:0/4078725465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59840c8e20 0x7f59841eee30 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f5974005040 tx=0x7f5970007a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:21.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.481+0000 7f5979ffb640 1 -- 172.21.15.89:0/4078725465 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59740072d0 con 0x7f59840c8e20 2023-10-30T12:33:21.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.481+0000 7f5979ffb640 1 -- 172.21.15.89:0/4078725465 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5974007430 con 0x7f59840c8e20 2023-10-30T12:33:21.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.481+0000 7f598ba54640 1 -- 172.21.15.89:0/4078725465 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5984159030 con 0x7f59840c8e20 2023-10-30T12:33:21.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.481+0000 7f5979ffb640 1 -- 172.21.15.89:0/4078725465 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59740067e0 con 0x7f59840c8e20 2023-10-30T12:33:21.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.481+0000 7f598ba54640 1 -- 172.21.15.89:0/4078725465 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f598413a5e0 con 0x7f59840c8e20 2023-10-30T12:33:21.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.485+0000 7f598ba54640 1 -- 172.21.15.89:0/4078725465 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f598406ff60 con 0x7f59840c8e20 2023-10-30T12:33:21.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.485+0000 7f5979ffb640 1 -- 172.21.15.89:0/4078725465 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5974006940 con 0x7f59840c8e20 2023-10-30T12:33:21.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.485+0000 7f5979ffb640 1 --2- 172.21.15.89:0/4078725465 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f59680664d0 0x7f5968068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:21.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.485+0000 7f5979ffb640 1 -- 172.21.15.89:0/4078725465 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5976+0+0 (secure 0 0 0) 0x7f597411e370 con 0x7f59840c8e20 2023-10-30T12:33:21.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.485+0000 7f59897f0640 1 --2- 172.21.15.89:0/4078725465 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f59680664d0 0x7f5968068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:21.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.485+0000 7f59897f0640 1 --2- 172.21.15.89:0/4078725465 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f59680664d0 0x7f5968068990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5980005180 tx=0x7f5980009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:21.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.489+0000 7f5979ffb640 1 -- 172.21.15.89:0/4078725465 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f598406ff60 con 0x7f59840c8e20 2023-10-30T12:33:21.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.669+0000 7f598ba54640 1 -- 172.21.15.89:0/4078725465 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f598406ff60 con 0x7f59840c8e20 2023-10-30T12:33:21.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.669+0000 7f5979ffb640 1 -- 172.21.15.89:0/4078725465 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f598406ff60 con 0x7f59840c8e20 2023-10-30T12:33:21.675 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:33:21.676 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:33:16.428171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:resolve","state_seq":21,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:33:21.677 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 25 2023-10-30T12:33:21.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.673+0000 7f598ba54640 1 -- 172.21.15.89:0/4078725465 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f59680664d0 msgr2=0x7f5968068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:21.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.673+0000 7f598ba54640 1 --2- 172.21.15.89:0/4078725465 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f59680664d0 0x7f5968068990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5980005180 tx=0x7f5980009040 comp rx=0 tx=0).stop 2023-10-30T12:33:21.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.673+0000 7f598ba54640 1 -- 172.21.15.89:0/4078725465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59840c8e20 msgr2=0x7f59841eee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:21.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.673+0000 7f598ba54640 1 --2- 172.21.15.89:0/4078725465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59840c8e20 0x7f59841eee30 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f5974005040 tx=0x7f5970007a10 comp rx=0 tx=0).stop 2023-10-30T12:33:21.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.673+0000 7f598ba54640 1 -- 172.21.15.89:0/4078725465 shutdown_connections 2023-10-30T12:33:21.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.673+0000 7f598ba54640 1 --2- 172.21.15.89:0/4078725465 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f59680664d0 0x7f5968068990 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:21.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.673+0000 7f598ba54640 1 --2- 172.21.15.89:0/4078725465 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f59841ef370 0x7f5984136950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:21.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.673+0000 7f598ba54640 1 --2- 172.21.15.89:0/4078725465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59840c8e20 0x7f59841eee30 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:21.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.673+0000 7f598ba54640 1 --2- 172.21.15.89:0/4078725465 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59700069e0 0x7f59841e4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:21.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.673+0000 7f598ba54640 1 -- 172.21.15.89:0/4078725465 >> 172.21.15.89:0/4078725465 conn(0x7f59840c7910 msgr2=0x7f59840c0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:21.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.673+0000 7f598ba54640 1 -- 172.21.15.89:0/4078725465 shutdown_connections 2023-10-30T12:33:21.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:21.673+0000 7f598ba54640 1 -- 172.21.15.89:0/4078725465 wait complete. 2023-10-30T12:33:21.696 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:33:23.697 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:33:23.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.841+0000 7f9aea69c640 1 Processor -- start 2023-10-30T12:33:23.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.841+0000 7f9aea69c640 1 -- start start 2023-10-30T12:33:23.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.841+0000 7f9aea69c640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9ae4156f60 0x7f9ae4071b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:23.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.841+0000 7f9aea69c640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ae406fd50 0x7f9ae4072050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:23.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.841+0000 7f9aea69c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9ae406afa0 0x7f9ae415ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:23.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.841+0000 7f9aea69c640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ae406bea0 con 0x7f9ae406afa0 2023-10-30T12:33:23.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.841+0000 7f9aea69c640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ae40caa80 con 0x7f9ae4156f60 2023-10-30T12:33:23.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.841+0000 7f9aea69c640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae4157340 con 0x7f9ae406fd50 2023-10-30T12:33:23.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.841+0000 7f9ae8c39640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9ae406afa0 0x7f9ae415ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:23.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.841+0000 7f9ae8c39640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9ae406afa0 0x7f9ae415ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37410/0 (socket says 172.21.15.89:37410) 2023-10-30T12:33:23.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.841+0000 7f9ae8c39640 1 -- 172.21.15.89:0/722758342 learned_addr learned my addr 172.21.15.89:0/722758342 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:33:23.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.841+0000 7f9ae3fff640 1 --2- 172.21.15.89:0/722758342 >> v2:172.21.15.192:3300/0 conn(0x7f9ae4156f60 0x7f9ae4071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:23.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.841+0000 7f9ae37fe640 1 --2- 172.21.15.89:0/722758342 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ae406fd50 0x7f9ae4072050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:23.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.841+0000 7f9ae8c39640 1 -- 172.21.15.89:0/722758342 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ae406fd50 msgr2=0x7f9ae4072050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:23.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.841+0000 7f9ae8c39640 1 --2- 172.21.15.89:0/722758342 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ae406fd50 0x7f9ae4072050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:23.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.841+0000 7f9ae8c39640 1 -- 172.21.15.89:0/722758342 >> v2:172.21.15.192:3300/0 conn(0x7f9ae4156f60 msgr2=0x7f9ae4071b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:23.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.841+0000 7f9ae8c39640 1 --2- 172.21.15.89:0/722758342 >> v2:172.21.15.192:3300/0 conn(0x7f9ae4156f60 0x7f9ae4071b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:23.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.841+0000 7f9ae8c39640 1 -- 172.21.15.89:0/722758342 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae406ca30 con 0x7f9ae406afa0 2023-10-30T12:33:23.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.841+0000 7f9ae3fff640 1 --2- 172.21.15.89:0/722758342 >> v2:172.21.15.192:3300/0 conn(0x7f9ae4156f60 0x7f9ae4071b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:33:23.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.845+0000 7f9ae8c39640 1 --2- 172.21.15.89:0/722758342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ae406afa0 0x7f9ae415ce10 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9ae40ce070 tx=0x7f9adc01c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37c42ada80a2ef29 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:23.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.845+0000 7f9ae2ffd640 1 -- 172.21.15.89:0/722758342 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9adc0a5740 con 0x7f9ae406afa0 2023-10-30T12:33:23.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.845+0000 7f9ae2ffd640 1 -- 172.21.15.89:0/722758342 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9adc0a58a0 con 0x7f9ae406afa0 2023-10-30T12:33:23.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.845+0000 7f9ae2ffd640 1 -- 172.21.15.89:0/722758342 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9adc0a5c30 con 0x7f9ae406afa0 2023-10-30T12:33:23.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.845+0000 7f9aea69c640 1 -- 172.21.15.89:0/722758342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ae406afa0 msgr2=0x7f9ae415ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:23.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.845+0000 7f9aea69c640 1 --2- 172.21.15.89:0/722758342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ae406afa0 0x7f9ae415ce10 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9ae40ce070 tx=0x7f9adc01c9b0 comp rx=0 tx=0).stop 2023-10-30T12:33:23.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.845+0000 7f9aea69c640 1 -- 172.21.15.89:0/722758342 shutdown_connections 2023-10-30T12:33:23.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.845+0000 7f9aea69c640 1 --2- 172.21.15.89:0/722758342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ae406afa0 0x7f9ae415ce10 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:23.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.845+0000 7f9aea69c640 1 --2- 172.21.15.89:0/722758342 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ae406fd50 0x7f9ae4072050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:23.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.845+0000 7f9aea69c640 1 --2- 172.21.15.89:0/722758342 >> v2:172.21.15.192:3300/0 conn(0x7f9ae4156f60 0x7f9ae4071b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:23.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.845+0000 7f9aea69c640 1 -- 172.21.15.89:0/722758342 >> 172.21.15.89:0/722758342 conn(0x7f9ae4069e90 msgr2=0x7f9ae406a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:23.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.845+0000 7f9aea69c640 1 -- 172.21.15.89:0/722758342 shutdown_connections 2023-10-30T12:33:23.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.845+0000 7f9aea69c640 1 -- 172.21.15.89:0/722758342 wait complete. 2023-10-30T12:33:23.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.849+0000 7f9aea69c640 1 Processor -- start 2023-10-30T12:33:23.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.849+0000 7f9aea69c640 1 -- start start 2023-10-30T12:33:23.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.849+0000 7f9aea69c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ae406afa0 0x7f9ae415da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:23.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.849+0000 7f9aea69c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ae406fd50 0x7f9ae415b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:23.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.849+0000 7f9aea69c640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ae4156f60 0x7f9ae415b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:23.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.849+0000 7f9aea69c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae40c4fd0 con 0x7f9ae406fd50 2023-10-30T12:33:23.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.849+0000 7f9aea69c640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae40c6c90 con 0x7f9ae406afa0 2023-10-30T12:33:23.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.849+0000 7f9aea69c640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae415aba0 con 0x7f9ae4156f60 2023-10-30T12:33:23.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.849+0000 7f9ae37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ae406fd50 0x7f9ae415b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:23.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.849+0000 7f9ae37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ae406fd50 0x7f9ae415b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37422/0 (socket says 172.21.15.89:37422) 2023-10-30T12:33:23.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.849+0000 7f9ae37fe640 1 -- 172.21.15.89:0/3545866080 learned_addr learned my addr 172.21.15.89:0/3545866080 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:33:23.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.849+0000 7f9ae3fff640 1 --2- 172.21.15.89:0/3545866080 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ae406afa0 0x7f9ae415da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:23.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.849+0000 7f9ae8c39640 1 --2- 172.21.15.89:0/3545866080 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ae4156f60 0x7f9ae415b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:23.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.849+0000 7f9ae37fe640 1 -- 172.21.15.89:0/3545866080 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ae4156f60 msgr2=0x7f9ae415b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:23.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.849+0000 7f9ae37fe640 1 --2- 172.21.15.89:0/3545866080 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ae4156f60 0x7f9ae415b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:23.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.849+0000 7f9ae37fe640 1 -- 172.21.15.89:0/3545866080 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ae406afa0 msgr2=0x7f9ae415da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:23.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.849+0000 7f9ae37fe640 1 --2- 172.21.15.89:0/3545866080 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ae406afa0 0x7f9ae415da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:23.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.849+0000 7f9ae37fe640 1 -- 172.21.15.89:0/3545866080 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9adc009b30 con 0x7f9ae406fd50 2023-10-30T12:33:23.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.849+0000 7f9ae37fe640 1 --2- 172.21.15.89:0/3545866080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ae406fd50 0x7f9ae415b080 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9ad800d800 tx=0x7f9ad800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:23.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.849+0000 7f9ae0ff9640 1 -- 172.21.15.89:0/3545866080 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad80096a0 con 0x7f9ae406fd50 2023-10-30T12:33:23.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.849+0000 7f9ae0ff9640 1 -- 172.21.15.89:0/3545866080 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad8009800 con 0x7f9ae406fd50 2023-10-30T12:33:23.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.849+0000 7f9ae0ff9640 1 -- 172.21.15.89:0/3545866080 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad8009990 con 0x7f9ae406fd50 2023-10-30T12:33:23.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.849+0000 7f9aea69c640 1 -- 172.21.15.89:0/3545866080 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ae40c69d0 con 0x7f9ae406fd50 2023-10-30T12:33:23.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.849+0000 7f9aea69c640 1 -- 172.21.15.89:0/3545866080 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ae40c4a10 con 0x7f9ae406fd50 2023-10-30T12:33:23.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.853+0000 7f9aea69c640 1 -- 172.21.15.89:0/3545866080 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ae404e170 con 0x7f9ae406fd50 2023-10-30T12:33:23.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.853+0000 7f9ae0ff9640 1 -- 172.21.15.89:0/3545866080 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9ad8009af0 con 0x7f9ae406fd50 2023-10-30T12:33:23.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.853+0000 7f9ae0ff9640 1 --2- 172.21.15.89:0/3545866080 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9ad00664a0 0x7f9ad0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:23.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.853+0000 7f9ae0ff9640 1 -- 172.21.15.89:0/3545866080 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5976+0+0 (secure 0 0 0) 0x7f9ad8084490 con 0x7f9ae406fd50 2023-10-30T12:33:23.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.853+0000 7f9ae3fff640 1 --2- 172.21.15.89:0/3545866080 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9ad00664a0 0x7f9ad0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:23.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.853+0000 7f9ae3fff640 1 --2- 172.21.15.89:0/3545866080 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9ad00664a0 0x7f9ad0068960 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9ad400c900 tx=0x7f9ad4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:23.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:23.857+0000 7f9ae0ff9640 1 -- 172.21.15.89:0/3545866080 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9ad8051760 con 0x7f9ae406fd50 2023-10-30T12:33:24.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:24.041+0000 7f9aea69c640 1 -- 172.21.15.89:0/3545866080 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ae406fb30 con 0x7f9ae406fd50 2023-10-30T12:33:24.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:24.041+0000 7f9ae0ff9640 1 -- 172.21.15.89:0/3545866080 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5743 (secure 0 0 0) 0x7f9ad8009da0 con 0x7f9ae406fd50 2023-10-30T12:33:24.045 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:33:24.046 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:33:22.102214+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:reconnect","state_seq":23,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:33:24.046 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 26 2023-10-30T12:33:24.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:24.041+0000 7f9aea69c640 1 -- 172.21.15.89:0/3545866080 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9ad00664a0 msgr2=0x7f9ad0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:24.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:24.041+0000 7f9aea69c640 1 --2- 172.21.15.89:0/3545866080 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9ad00664a0 0x7f9ad0068960 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9ad400c900 tx=0x7f9ad4008040 comp rx=0 tx=0).stop 2023-10-30T12:33:24.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:24.041+0000 7f9aea69c640 1 -- 172.21.15.89:0/3545866080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ae406fd50 msgr2=0x7f9ae415b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:24.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:24.041+0000 7f9aea69c640 1 --2- 172.21.15.89:0/3545866080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ae406fd50 0x7f9ae415b080 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9ad800d800 tx=0x7f9ad800dcc0 comp rx=0 tx=0).stop 2023-10-30T12:33:24.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:24.041+0000 7f9aea69c640 1 -- 172.21.15.89:0/3545866080 shutdown_connections 2023-10-30T12:33:24.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:24.041+0000 7f9aea69c640 1 --2- 172.21.15.89:0/3545866080 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9ad00664a0 0x7f9ad0068960 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:24.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:24.041+0000 7f9aea69c640 1 --2- 172.21.15.89:0/3545866080 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9ae4156f60 0x7f9ae415b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:24.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:24.041+0000 7f9aea69c640 1 --2- 172.21.15.89:0/3545866080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ae406fd50 0x7f9ae415b080 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:24.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:24.041+0000 7f9aea69c640 1 --2- 172.21.15.89:0/3545866080 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ae406afa0 0x7f9ae415da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:24.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:24.041+0000 7f9aea69c640 1 -- 172.21.15.89:0/3545866080 >> 172.21.15.89:0/3545866080 conn(0x7f9ae4069e90 msgr2=0x7f9ae406a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:24.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:24.041+0000 7f9aea69c640 1 -- 172.21.15.89:0/3545866080 shutdown_connections 2023-10-30T12:33:24.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:24.041+0000 7f9aea69c640 1 -- 172.21.15.89:0/3545866080 wait complete. 2023-10-30T12:33:24.068 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 26, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:50.729245+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 26, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4262": { "addr": "172.21.15.192:6839/3339589587", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6838", "nonce": 3339589587, "type": "v2" }, { "addr": "172.21.15.192:6839", "nonce": 3339589587, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 9, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4318": { "addr": "172.21.15.192:6837/1002124471", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6836", "nonce": 1002124471, "type": "v2" }, { "addr": "172.21.15.192:6837", "nonce": 1002124471, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4318, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4781": { "addr": "172.21.15.192:6835/654256299", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6834", "nonce": 654256299, "type": "v2" }, { "addr": "172.21.15.192:6835", "nonce": 654256299, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4781, "incarnation": 24, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:reconnect", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 29, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:33:22.102214+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4318, "mds_1": 4262, "mds_2": 4781 } } } ], "standbys": [ { "addr": "172.21.15.89:6834/2909790173", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6832", "nonce": 2909790173, "type": "v2" }, { "addr": "172.21.15.89:6834", "nonce": 2909790173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.89:6835/2387357615", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6833", "nonce": 2387357615, "type": "v2" }, { "addr": "172.21.15.89:6835", "nonce": 2387357615, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4239, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:33:26.068 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:33:26.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.209+0000 7f9c2bd1d640 1 Processor -- start 2023-10-30T12:33:26.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c2bd1d640 1 -- start start 2023-10-30T12:33:26.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c2bd1d640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9c241538a0 0x7f9c2415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:26.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c2bd1d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c240ca0e0 0x7f9c2415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:26.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c2bd1d640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c240cb800 con 0x7f9c240cf160 2023-10-30T12:33:26.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c2bd1d640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c2415dc30 con 0x7f9c241538a0 2023-10-30T12:33:26.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c2bd1d640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c240cb330 con 0x7f9c240ca0e0 2023-10-30T12:33:26.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c2a2ba640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9c240cf160 0x7f9c240cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38088/0 (socket says 172.21.15.89:38088) 2023-10-30T12:33:26.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c2a2ba640 1 -- 172.21.15.89:0/2592053795 learned_addr learned my addr 172.21.15.89:0/2592053795 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:33:26.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c29ab9640 1 --2- 172.21.15.89:0/2592053795 >> v2:172.21.15.192:3300/0 conn(0x7f9c241538a0 0x7f9c2415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:26.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c292b8640 1 --2- 172.21.15.89:0/2592053795 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c240ca0e0 0x7f9c2415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:26.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c28ab7640 1 -- 172.21.15.89:0/2592053795 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1229712897 0 0) 0x7f9c240cb800 con 0x7f9c240cf160 2023-10-30T12:33:26.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c28ab7640 1 -- 172.21.15.89:0/2592053795 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c10003530 con 0x7f9c240cf160 2023-10-30T12:33:26.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c28ab7640 1 -- 172.21.15.89:0/2592053795 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 969376358 0 0) 0x7f9c10003530 con 0x7f9c240cf160 2023-10-30T12:33:26.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c28ab7640 1 -- 172.21.15.89:0/2592053795 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c240ca0e0 msgr2=0x7f9c2415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:26.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c28ab7640 1 --2- 172.21.15.89:0/2592053795 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c240ca0e0 0x7f9c2415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:26.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c28ab7640 1 -- 172.21.15.89:0/2592053795 >> v2:172.21.15.192:3300/0 conn(0x7f9c241538a0 msgr2=0x7f9c2415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:26.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c28ab7640 1 --2- 172.21.15.89:0/2592053795 >> v2:172.21.15.192:3300/0 conn(0x7f9c241538a0 0x7f9c2415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:26.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c28ab7640 1 -- 172.21.15.89:0/2592053795 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c240cad70 con 0x7f9c240cf160 2023-10-30T12:33:26.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c28ab7640 1 -- 172.21.15.89:0/2592053795 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f9c1c001a00 con 0x7f9c240cf160 2023-10-30T12:33:26.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c29ab9640 1 --2- 172.21.15.89:0/2592053795 >> v2:172.21.15.192:3300/0 conn(0x7f9c241538a0 0x7f9c2415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:33:26.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c28ab7640 1 -- 172.21.15.89:0/2592053795 >> v1:172.21.15.89:6789/0 conn(0x7f9c240cf160 legacy=0x7f9c240cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:26.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c28ab7640 1 --2- 172.21.15.89:0/2592053795 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c10003980 0x7f9c10003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:26.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c28ab7640 1 --2- 172.21.15.89:0/2592053795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c100047d0 0x7f9c10004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:26.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c28ab7640 1 --2- 172.21.15.89:0/2592053795 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c240ca0e0 0x7f9c10005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:26.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c28ab7640 1 -- 172.21.15.89:0/2592053795 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2415dc30 con 0x7f9c100047d0 2023-10-30T12:33:26.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c28ab7640 1 -- 172.21.15.89:0/2592053795 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c240cb330 con 0x7f9c10003980 2023-10-30T12:33:26.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c28ab7640 1 -- 172.21.15.89:0/2592053795 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1000b9c0 con 0x7f9c240ca0e0 2023-10-30T12:33:26.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c292b8640 1 --2- 172.21.15.89:0/2592053795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c100047d0 0x7f9c10004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:26.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c29ab9640 1 --2- 172.21.15.89:0/2592053795 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c10003980 0x7f9c10003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:26.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c2a2ba640 1 --2- 172.21.15.89:0/2592053795 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c240ca0e0 0x7f9c10005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:26.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c292b8640 1 -- 172.21.15.89:0/2592053795 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c240ca0e0 msgr2=0x7f9c10005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:26.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c292b8640 1 --2- 172.21.15.89:0/2592053795 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c240ca0e0 0x7f9c10005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:26.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c292b8640 1 -- 172.21.15.89:0/2592053795 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c10003980 msgr2=0x7f9c10003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:26.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c292b8640 1 --2- 172.21.15.89:0/2592053795 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c10003980 0x7f9c10003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:26.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c292b8640 1 -- 172.21.15.89:0/2592053795 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c10003530 con 0x7f9c100047d0 2023-10-30T12:33:26.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.213+0000 7f9c2a2ba640 1 --2- 172.21.15.89:0/2592053795 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c240ca0e0 0x7f9c10005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:33:26.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.217+0000 7f9c292b8640 1 --2- 172.21.15.89:0/2592053795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c100047d0 0x7f9c10004c50 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f9c14009ae0 tx=0x7f9c1401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db7a4ed487b4f88b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:26.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.217+0000 7f9c28ab7640 1 -- 172.21.15.89:0/2592053795 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c140a5560 con 0x7f9c100047d0 2023-10-30T12:33:26.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.217+0000 7f9c28ab7640 1 -- 172.21.15.89:0/2592053795 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c140a56c0 con 0x7f9c100047d0 2023-10-30T12:33:26.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.217+0000 7f9c28ab7640 1 -- 172.21.15.89:0/2592053795 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c140a5a70 con 0x7f9c100047d0 2023-10-30T12:33:26.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.217+0000 7f9c2bd1d640 1 -- 172.21.15.89:0/2592053795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c100047d0 msgr2=0x7f9c10004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:26.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.217+0000 7f9c2bd1d640 1 --2- 172.21.15.89:0/2592053795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c100047d0 0x7f9c10004c50 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f9c14009ae0 tx=0x7f9c1401ca50 comp rx=0 tx=0).stop 2023-10-30T12:33:26.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.217+0000 7f9c2a2ba640 1 -- 172.21.15.89:0/2592053795 reap_dead start 2023-10-30T12:33:26.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.217+0000 7f9c2bd1d640 1 -- 172.21.15.89:0/2592053795 shutdown_connections 2023-10-30T12:33:26.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.217+0000 7f9c2bd1d640 1 -- 172.21.15.89:0/2592053795 >> 172.21.15.89:0/2592053795 conn(0x7f9c2406bfe0 msgr2=0x7f9c2406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:26.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.217+0000 7f9c2bd1d640 1 -- 172.21.15.89:0/2592053795 shutdown_connections 2023-10-30T12:33:26.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.217+0000 7f9c2bd1d640 1 -- 172.21.15.89:0/2592053795 wait complete. 2023-10-30T12:33:26.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.221+0000 7f9c2bd1d640 1 Processor -- start 2023-10-30T12:33:26.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.221+0000 7f9c2bd1d640 1 -- start start 2023-10-30T12:33:26.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.221+0000 7f9c2bd1d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c100047d0 0x7f9c24146ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:26.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.221+0000 7f9c2bd1d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c24147df0 0x7f9c240c5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:26.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.221+0000 7f9c2bd1d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c24158d60 0x7f9c24145be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:26.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.221+0000 7f9c2bd1d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c241592c0 con 0x7f9c24147df0 2023-10-30T12:33:26.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.221+0000 7f9c2bd1d640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c24149650 con 0x7f9c24158d60 2023-10-30T12:33:26.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.221+0000 7f9c2bd1d640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c24159560 con 0x7f9c100047d0 2023-10-30T12:33:26.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.221+0000 7f9c292b8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c24147df0 0x7f9c240c5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:26.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.221+0000 7f9c292b8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c24147df0 0x7f9c240c5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37436/0 (socket says 172.21.15.89:37436) 2023-10-30T12:33:26.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.221+0000 7f9c292b8640 1 -- 172.21.15.89:0/3925918919 learned_addr learned my addr 172.21.15.89:0/3925918919 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:33:26.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.221+0000 7f9c29ab9640 1 --2- 172.21.15.89:0/3925918919 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c100047d0 0x7f9c24146ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:26.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.221+0000 7f9c2a2ba640 1 --2- 172.21.15.89:0/3925918919 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c24158d60 0x7f9c24145be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:26.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.221+0000 7f9c292b8640 1 -- 172.21.15.89:0/3925918919 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c100047d0 msgr2=0x7f9c24146ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:26.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.221+0000 7f9c292b8640 1 --2- 172.21.15.89:0/3925918919 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c100047d0 0x7f9c24146ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:26.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.221+0000 7f9c292b8640 1 -- 172.21.15.89:0/3925918919 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c24158d60 msgr2=0x7f9c24145be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:26.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.221+0000 7f9c292b8640 1 --2- 172.21.15.89:0/3925918919 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c24158d60 0x7f9c24145be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:26.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.221+0000 7f9c292b8640 1 -- 172.21.15.89:0/3925918919 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c241581d0 con 0x7f9c24147df0 2023-10-30T12:33:26.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.221+0000 7f9c292b8640 1 --2- 172.21.15.89:0/3925918919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c24147df0 0x7f9c240c5320 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f9c14009ae0 tx=0x7f9c10005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:26.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.221+0000 7f9c1a7fc640 1 -- 172.21.15.89:0/3925918919 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c140be070 con 0x7f9c24147df0 2023-10-30T12:33:26.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.221+0000 7f9c1a7fc640 1 -- 172.21.15.89:0/3925918919 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c14004020 con 0x7f9c24147df0 2023-10-30T12:33:26.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.221+0000 7f9c1a7fc640 1 -- 172.21.15.89:0/3925918919 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c14004390 con 0x7f9c24147df0 2023-10-30T12:33:26.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.221+0000 7f9c2bd1d640 1 -- 172.21.15.89:0/3925918919 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c24149880 con 0x7f9c24147df0 2023-10-30T12:33:26.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.221+0000 7f9c2bd1d640 1 -- 172.21.15.89:0/3925918919 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c240b1960 con 0x7f9c24147df0 2023-10-30T12:33:26.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.225+0000 7f9c2bd1d640 1 -- 172.21.15.89:0/3925918919 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c2404e170 con 0x7f9c24147df0 2023-10-30T12:33:26.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.225+0000 7f9c1a7fc640 1 -- 172.21.15.89:0/3925918919 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9c140b4040 con 0x7f9c24147df0 2023-10-30T12:33:26.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.225+0000 7f9c1a7fc640 1 --2- 172.21.15.89:0/3925918919 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9c080664d0 0x7f9c08068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:26.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.225+0000 7f9c1a7fc640 1 -- 172.21.15.89:0/3925918919 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5976+0+0 (secure 0 0 0) 0x7f9c1411db00 con 0x7f9c24147df0 2023-10-30T12:33:26.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.225+0000 7f9c29ab9640 1 --2- 172.21.15.89:0/3925918919 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9c080664d0 0x7f9c08068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:26.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.225+0000 7f9c29ab9640 1 --2- 172.21.15.89:0/3925918919 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9c080664d0 0x7f9c08068990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9c20009d00 tx=0x7f9c20008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:26.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.229+0000 7f9c1a7fc640 1 -- 172.21.15.89:0/3925918919 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9c1411fe60 con 0x7f9c24147df0 2023-10-30T12:33:26.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.409+0000 7f9c2bd1d640 1 -- 172.21.15.89:0/3925918919 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c240ca9b0 con 0x7f9c24147df0 2023-10-30T12:33:26.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.409+0000 7f9c1a7fc640 1 -- 172.21.15.89:0/3925918919 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5743 (secure 0 0 0) 0x7f9c140eae80 con 0x7f9c24147df0 2023-10-30T12:33:26.416 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:33:26.416 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:33:22.102214+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:reconnect","state_seq":23,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:33:26.417 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 26 2023-10-30T12:33:26.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.413+0000 7f9c2bd1d640 1 -- 172.21.15.89:0/3925918919 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9c080664d0 msgr2=0x7f9c08068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:26.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.413+0000 7f9c2bd1d640 1 --2- 172.21.15.89:0/3925918919 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9c080664d0 0x7f9c08068990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9c20009d00 tx=0x7f9c20008040 comp rx=0 tx=0).stop 2023-10-30T12:33:26.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.413+0000 7f9c2bd1d640 1 -- 172.21.15.89:0/3925918919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c24147df0 msgr2=0x7f9c240c5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:26.419 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.413+0000 7f9c2bd1d640 1 --2- 172.21.15.89:0/3925918919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c24147df0 0x7f9c240c5320 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f9c14009ae0 tx=0x7f9c10005800 comp rx=0 tx=0).stop 2023-10-30T12:33:26.419 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.413+0000 7f9c2bd1d640 1 -- 172.21.15.89:0/3925918919 shutdown_connections 2023-10-30T12:33:26.419 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.413+0000 7f9c2bd1d640 1 --2- 172.21.15.89:0/3925918919 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9c080664d0 0x7f9c08068990 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:26.419 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.413+0000 7f9c2bd1d640 1 --2- 172.21.15.89:0/3925918919 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9c24158d60 0x7f9c24145be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:26.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.413+0000 7f9c2bd1d640 1 --2- 172.21.15.89:0/3925918919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c24147df0 0x7f9c240c5320 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:26.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.413+0000 7f9c2bd1d640 1 --2- 172.21.15.89:0/3925918919 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9c100047d0 0x7f9c24146ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:26.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.413+0000 7f9c2bd1d640 1 -- 172.21.15.89:0/3925918919 >> 172.21.15.89:0/3925918919 conn(0x7f9c2406bfe0 msgr2=0x7f9c24065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:26.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.413+0000 7f9c2bd1d640 1 -- 172.21.15.89:0/3925918919 shutdown_connections 2023-10-30T12:33:26.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:26.413+0000 7f9c2bd1d640 1 -- 172.21.15.89:0/3925918919 wait complete. 2023-10-30T12:33:26.436 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:33:28.438 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:33:28.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.585+0000 7f5a36c68640 1 Processor -- start 2023-10-30T12:33:28.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.585+0000 7f5a36c68640 1 -- start start 2023-10-30T12:33:28.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.585+0000 7f5a36c68640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5a30153bf0 0x7f5a30071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:28.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.585+0000 7f5a36c68640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5a300c6fe0 0x7f5a300c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:28.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.585+0000 7f5a36c68640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a300c79e0 0x7f5a300cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:28.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.585+0000 7f5a36c68640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a300c6810 con 0x7f5a30153bf0 2023-10-30T12:33:28.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.585+0000 7f5a36c68640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a30159900 con 0x7f5a300c6fe0 2023-10-30T12:33:28.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.585+0000 7f5a36c68640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a300700b0 con 0x7f5a300c79e0 2023-10-30T12:33:28.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.585+0000 7f5a34a04640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5a30153bf0 0x7f5a30071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:28.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.585+0000 7f5a34a04640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5a30153bf0 0x7f5a30071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37448/0 (socket says 172.21.15.89:37448) 2023-10-30T12:33:28.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.585+0000 7f5a2ffff640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5a300c6fe0 0x7f5a300c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:28.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.585+0000 7f5a34a04640 1 -- 172.21.15.89:0/3713929981 learned_addr learned my addr 172.21.15.89:0/3713929981 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:33:28.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.585+0000 7f5a35205640 1 --2- 172.21.15.89:0/3713929981 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a300c79e0 0x7f5a300cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:28.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.585+0000 7f5a34a04640 1 -- 172.21.15.89:0/3713929981 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a300c79e0 msgr2=0x7f5a300cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:28.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.585+0000 7f5a34a04640 1 --2- 172.21.15.89:0/3713929981 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a300c79e0 0x7f5a300cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:28.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.585+0000 7f5a34a04640 1 -- 172.21.15.89:0/3713929981 >> v2:172.21.15.192:3300/0 conn(0x7f5a300c6fe0 msgr2=0x7f5a300c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:28.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.585+0000 7f5a34a04640 1 --2- 172.21.15.89:0/3713929981 >> v2:172.21.15.192:3300/0 conn(0x7f5a300c6fe0 0x7f5a300c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:28.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.585+0000 7f5a34a04640 1 -- 172.21.15.89:0/3713929981 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a300c6530 con 0x7f5a30153bf0 2023-10-30T12:33:28.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.589+0000 7f5a34a04640 1 --2- 172.21.15.89:0/3713929981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a30153bf0 0x7f5a30071260 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f5a300723d0 tx=0x7f5a2401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24119cfd706dc213 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:28.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.589+0000 7f5a2f7fe640 1 -- 172.21.15.89:0/3713929981 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a240a89b0 con 0x7f5a30153bf0 2023-10-30T12:33:28.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.589+0000 7f5a2f7fe640 1 -- 172.21.15.89:0/3713929981 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a240a8b10 con 0x7f5a30153bf0 2023-10-30T12:33:28.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.589+0000 7f5a2f7fe640 1 -- 172.21.15.89:0/3713929981 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a240a8ea0 con 0x7f5a30153bf0 2023-10-30T12:33:28.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.589+0000 7f5a36c68640 1 -- 172.21.15.89:0/3713929981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a30153bf0 msgr2=0x7f5a30071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:28.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.589+0000 7f5a36c68640 1 --2- 172.21.15.89:0/3713929981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a30153bf0 0x7f5a30071260 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f5a300723d0 tx=0x7f5a2401ecf0 comp rx=0 tx=0).stop 2023-10-30T12:33:28.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.589+0000 7f5a36c68640 1 -- 172.21.15.89:0/3713929981 shutdown_connections 2023-10-30T12:33:28.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.589+0000 7f5a36c68640 1 --2- 172.21.15.89:0/3713929981 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a300c79e0 0x7f5a300cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:28.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.589+0000 7f5a36c68640 1 --2- 172.21.15.89:0/3713929981 >> v2:172.21.15.192:3300/0 conn(0x7f5a300c6fe0 0x7f5a300c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:28.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.589+0000 7f5a36c68640 1 --2- 172.21.15.89:0/3713929981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a30153bf0 0x7f5a30071260 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:28.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.589+0000 7f5a36c68640 1 -- 172.21.15.89:0/3713929981 >> 172.21.15.89:0/3713929981 conn(0x7f5a3006c6e0 msgr2=0x7f5a30069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:28.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.589+0000 7f5a36c68640 1 -- 172.21.15.89:0/3713929981 shutdown_connections 2023-10-30T12:33:28.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.589+0000 7f5a36c68640 1 -- 172.21.15.89:0/3713929981 wait complete. 2023-10-30T12:33:28.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.593+0000 7f5a36c68640 1 Processor -- start 2023-10-30T12:33:28.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.593+0000 7f5a36c68640 1 -- start start 2023-10-30T12:33:28.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.593+0000 7f5a36c68640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a300c6fe0 0x7f5a301f70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:28.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.593+0000 7f5a36c68640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a300c79e0 0x7f5a300c40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:28.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.593+0000 7f5a36c68640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a30153bf0 0x7f5a30152ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:28.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.593+0000 7f5a36c68640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3014a560 con 0x7f5a30153bf0 2023-10-30T12:33:28.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.593+0000 7f5a36c68640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a301f75e0 con 0x7f5a300c6fe0 2023-10-30T12:33:28.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.593+0000 7f5a36c68640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3014cd60 con 0x7f5a300c79e0 2023-10-30T12:33:28.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.593+0000 7f5a35205640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a30153bf0 0x7f5a30152ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:28.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.593+0000 7f5a35205640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a30153bf0 0x7f5a30152ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37464/0 (socket says 172.21.15.89:37464) 2023-10-30T12:33:28.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.593+0000 7f5a35205640 1 -- 172.21.15.89:0/3277904602 learned_addr learned my addr 172.21.15.89:0/3277904602 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:33:28.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.593+0000 7f5a2ffff640 1 --2- 172.21.15.89:0/3277904602 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a300c79e0 0x7f5a300c40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:28.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.593+0000 7f5a34a04640 1 --2- 172.21.15.89:0/3277904602 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a300c6fe0 0x7f5a301f70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:28.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.593+0000 7f5a35205640 1 -- 172.21.15.89:0/3277904602 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a300c79e0 msgr2=0x7f5a300c40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:28.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.593+0000 7f5a35205640 1 --2- 172.21.15.89:0/3277904602 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a300c79e0 0x7f5a300c40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:28.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.593+0000 7f5a35205640 1 -- 172.21.15.89:0/3277904602 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a300c6fe0 msgr2=0x7f5a301f70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:28.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.593+0000 7f5a35205640 1 --2- 172.21.15.89:0/3277904602 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a300c6fe0 0x7f5a301f70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:28.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.593+0000 7f5a35205640 1 -- 172.21.15.89:0/3277904602 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a240094d0 con 0x7f5a30153bf0 2023-10-30T12:33:28.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.593+0000 7f5a35205640 1 --2- 172.21.15.89:0/3277904602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a30153bf0 0x7f5a30152ca0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f5a2800efc0 tx=0x7f5a2800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:28.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.593+0000 7f5a2d7fa640 1 -- 172.21.15.89:0/3277904602 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a28014480 con 0x7f5a30153bf0 2023-10-30T12:33:28.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.593+0000 7f5a2d7fa640 1 -- 172.21.15.89:0/3277904602 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a2800f040 con 0x7f5a30153bf0 2023-10-30T12:33:28.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.593+0000 7f5a36c68640 1 -- 172.21.15.89:0/3277904602 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a301f3120 con 0x7f5a30153bf0 2023-10-30T12:33:28.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.593+0000 7f5a2d7fa640 1 -- 172.21.15.89:0/3277904602 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a28014820 con 0x7f5a30153bf0 2023-10-30T12:33:28.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.593+0000 7f5a36c68640 1 -- 172.21.15.89:0/3277904602 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a301f33e0 con 0x7f5a30153bf0 2023-10-30T12:33:28.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.593+0000 7f5a36c68640 1 -- 172.21.15.89:0/3277904602 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a3004e170 con 0x7f5a30153bf0 2023-10-30T12:33:28.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.597+0000 7f5a2d7fa640 1 -- 172.21.15.89:0/3277904602 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5a28014980 con 0x7f5a30153bf0 2023-10-30T12:33:28.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.597+0000 7f5a2d7fa640 1 --2- 172.21.15.89:0/3277904602 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5a140664a0 0x7f5a14068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:28.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.597+0000 7f5a2d7fa640 1 -- 172.21.15.89:0/3277904602 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5976+0+0 (secure 0 0 0) 0x7f5a28084c70 con 0x7f5a30153bf0 2023-10-30T12:33:28.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.597+0000 7f5a34a04640 1 --2- 172.21.15.89:0/3277904602 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5a140664a0 0x7f5a14068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:28.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.597+0000 7f5a34a04640 1 --2- 172.21.15.89:0/3277904602 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5a140664a0 0x7f5a14068960 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5a240b2000 tx=0x7f5a24008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:28.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.601+0000 7f5a2d7fa640 1 -- 172.21.15.89:0/3277904602 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5a28051f50 con 0x7f5a30153bf0 2023-10-30T12:33:28.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.781+0000 7f5a36c68640 1 -- 172.21.15.89:0/3277904602 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a3006fe90 con 0x7f5a30153bf0 2023-10-30T12:33:28.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.781+0000 7f5a2d7fa640 1 -- 172.21.15.89:0/3277904602 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5740 (secure 0 0 0) 0x7f5a28018090 con 0x7f5a30153bf0 2023-10-30T12:33:28.786 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:33:28.786 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:33:28.002806+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:rejoin","state_seq":25,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:33:28.786 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 27 2023-10-30T12:33:28.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.781+0000 7f5a36c68640 1 -- 172.21.15.89:0/3277904602 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5a140664a0 msgr2=0x7f5a14068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:28.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.781+0000 7f5a36c68640 1 --2- 172.21.15.89:0/3277904602 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5a140664a0 0x7f5a14068960 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5a240b2000 tx=0x7f5a24008040 comp rx=0 tx=0).stop 2023-10-30T12:33:28.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.781+0000 7f5a36c68640 1 -- 172.21.15.89:0/3277904602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a30153bf0 msgr2=0x7f5a30152ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:28.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.781+0000 7f5a36c68640 1 --2- 172.21.15.89:0/3277904602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a30153bf0 0x7f5a30152ca0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f5a2800efc0 tx=0x7f5a2800c490 comp rx=0 tx=0).stop 2023-10-30T12:33:28.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.781+0000 7f5a36c68640 1 -- 172.21.15.89:0/3277904602 shutdown_connections 2023-10-30T12:33:28.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.781+0000 7f5a36c68640 1 --2- 172.21.15.89:0/3277904602 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5a140664a0 0x7f5a14068960 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:28.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.781+0000 7f5a36c68640 1 --2- 172.21.15.89:0/3277904602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a30153bf0 0x7f5a30152ca0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:28.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.781+0000 7f5a36c68640 1 --2- 172.21.15.89:0/3277904602 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a300c79e0 0x7f5a300c40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:28.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.781+0000 7f5a36c68640 1 --2- 172.21.15.89:0/3277904602 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a300c6fe0 0x7f5a301f70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:28.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.781+0000 7f5a36c68640 1 -- 172.21.15.89:0/3277904602 >> 172.21.15.89:0/3277904602 conn(0x7f5a3006c6e0 msgr2=0x7f5a30151be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:28.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.781+0000 7f5a36c68640 1 -- 172.21.15.89:0/3277904602 shutdown_connections 2023-10-30T12:33:28.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:28.781+0000 7f5a36c68640 1 -- 172.21.15.89:0/3277904602 wait complete. 2023-10-30T12:33:28.806 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:33:30.808 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:33:30.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.953+0000 7f237db80640 1 Processor -- start 2023-10-30T12:33:30.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.953+0000 7f237db80640 1 -- start start 2023-10-30T12:33:30.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.953+0000 7f237db80640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f237806f7a0 0x7f237806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:30.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.953+0000 7f237db80640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2378071110 0x7f2378158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:30.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.953+0000 7f237db80640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2378066cf0 con 0x7f23780c6bd0 2023-10-30T12:33:30.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.953+0000 7f237db80640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23780700c0 con 0x7f2378071110 2023-10-30T12:33:30.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.953+0000 7f237db80640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23780c70d0 con 0x7f237806f7a0 2023-10-30T12:33:30.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.953+0000 7f2376ffd640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f23780c6bd0 0x7f2378069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44776/0 (socket says 172.21.15.89:44776) 2023-10-30T12:33:30.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.953+0000 7f2376ffd640 1 -- 172.21.15.89:0/2403486739 learned_addr learned my addr 172.21.15.89:0/2403486739 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:33:30.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.953+0000 7f2377fff640 1 --2- 172.21.15.89:0/2403486739 >> v2:172.21.15.192:3300/0 conn(0x7f2378071110 0x7f2378158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:30.960 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.953+0000 7f23777fe640 1 --2- 172.21.15.89:0/2403486739 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f237806f7a0 0x7f237806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:30.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.953+0000 7f23767fc640 1 -- 172.21.15.89:0/2403486739 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2970703027 0 0) 0x7f2378066cf0 con 0x7f23780c6bd0 2023-10-30T12:33:30.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.953+0000 7f23767fc640 1 -- 172.21.15.89:0/2403486739 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2360003530 con 0x7f23780c6bd0 2023-10-30T12:33:30.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.953+0000 7f23767fc640 1 -- 172.21.15.89:0/2403486739 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1404847795 0 0) 0x7f2360003530 con 0x7f23780c6bd0 2023-10-30T12:33:30.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.953+0000 7f23767fc640 1 -- 172.21.15.89:0/2403486739 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f237806f7a0 msgr2=0x7f237806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:30.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.953+0000 7f23767fc640 1 --2- 172.21.15.89:0/2403486739 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f237806f7a0 0x7f237806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:30.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.953+0000 7f23767fc640 1 -- 172.21.15.89:0/2403486739 >> v2:172.21.15.192:3300/0 conn(0x7f2378071110 msgr2=0x7f2378158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:30.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.953+0000 7f23767fc640 1 --2- 172.21.15.89:0/2403486739 >> v2:172.21.15.192:3300/0 conn(0x7f2378071110 0x7f2378158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:30.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.953+0000 7f23767fc640 1 -- 172.21.15.89:0/2403486739 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23780c7390 con 0x7f23780c6bd0 2023-10-30T12:33:30.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.953+0000 7f23767fc640 1 -- 172.21.15.89:0/2403486739 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f236c001680 con 0x7f23780c6bd0 2023-10-30T12:33:30.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.953+0000 7f23767fc640 1 -- 172.21.15.89:0/2403486739 >> v1:172.21.15.89:6789/0 conn(0x7f23780c6bd0 legacy=0x7f2378069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:30.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.953+0000 7f23767fc640 1 --2- 172.21.15.89:0/2403486739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2360003980 0x7f2360003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:30.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.953+0000 7f23767fc640 1 --2- 172.21.15.89:0/2403486739 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f237806f7a0 0x7f23600047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:30.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.953+0000 7f23767fc640 1 --2- 172.21.15.89:0/2403486739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2360004d10 0x7f2360005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:30.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.953+0000 7f23767fc640 1 -- 172.21.15.89:0/2403486739 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23780700c0 con 0x7f2360004d10 2023-10-30T12:33:30.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.953+0000 7f23767fc640 1 -- 172.21.15.89:0/2403486739 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23780c70d0 con 0x7f2360003980 2023-10-30T12:33:30.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.953+0000 7f23767fc640 1 -- 172.21.15.89:0/2403486739 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f236000b990 con 0x7f237806f7a0 2023-10-30T12:33:30.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.953+0000 7f2377fff640 1 --2- 172.21.15.89:0/2403486739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2360004d10 0x7f2360005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:30.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.953+0000 7f23777fe640 1 --2- 172.21.15.89:0/2403486739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2360003980 0x7f2360003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:30.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.953+0000 7f2376ffd640 1 --2- 172.21.15.89:0/2403486739 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f237806f7a0 0x7f23600047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:30.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.957+0000 7f2377fff640 1 -- 172.21.15.89:0/2403486739 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f237806f7a0 msgr2=0x7f23600047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:30.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.957+0000 7f2377fff640 1 --2- 172.21.15.89:0/2403486739 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f237806f7a0 0x7f23600047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:30.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.957+0000 7f2377fff640 1 -- 172.21.15.89:0/2403486739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2360003980 msgr2=0x7f2360003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:30.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.957+0000 7f2377fff640 1 --2- 172.21.15.89:0/2403486739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2360003980 0x7f2360003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:30.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.957+0000 7f2377fff640 1 -- 172.21.15.89:0/2403486739 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2360003530 con 0x7f2360004d10 2023-10-30T12:33:30.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.957+0000 7f2377fff640 1 --2- 172.21.15.89:0/2403486739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2360004d10 0x7f2360005190 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2370009fa0 tx=0x7f237001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4693cfe98a1fe4fd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:30.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.957+0000 7f23767fc640 1 -- 172.21.15.89:0/2403486739 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23700a5d30 con 0x7f2360004d10 2023-10-30T12:33:30.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.957+0000 7f23767fc640 1 -- 172.21.15.89:0/2403486739 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f237000f040 con 0x7f2360004d10 2023-10-30T12:33:30.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.957+0000 7f23767fc640 1 -- 172.21.15.89:0/2403486739 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23700ad660 con 0x7f2360004d10 2023-10-30T12:33:30.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.957+0000 7f237db80640 1 -- 172.21.15.89:0/2403486739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2360004d10 msgr2=0x7f2360005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:30.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.957+0000 7f237db80640 1 --2- 172.21.15.89:0/2403486739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2360004d10 0x7f2360005190 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2370009fa0 tx=0x7f237001cc60 comp rx=0 tx=0).stop 2023-10-30T12:33:30.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.957+0000 7f2377fff640 1 -- 172.21.15.89:0/2403486739 reap_dead start 2023-10-30T12:33:30.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.957+0000 7f237db80640 1 -- 172.21.15.89:0/2403486739 shutdown_connections 2023-10-30T12:33:30.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.957+0000 7f237db80640 1 -- 172.21.15.89:0/2403486739 >> 172.21.15.89:0/2403486739 conn(0x7f23780c7910 msgr2=0x7f23780c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:30.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.961+0000 7f237db80640 1 -- 172.21.15.89:0/2403486739 shutdown_connections 2023-10-30T12:33:30.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.961+0000 7f237db80640 1 -- 172.21.15.89:0/2403486739 wait complete. 2023-10-30T12:33:30.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.961+0000 7f237db80640 1 Processor -- start 2023-10-30T12:33:30.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.961+0000 7f237db80640 1 -- start start 2023-10-30T12:33:30.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.961+0000 7f237db80640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2360004d10 0x7f23780c1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:30.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.961+0000 7f237db80640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23780b0200 0x7f2378150840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:30.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.961+0000 7f237db80640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2378157430 0x7f2378150d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:30.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.961+0000 7f237db80640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2378158e00 con 0x7f2378157430 2023-10-30T12:33:30.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.961+0000 7f237db80640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237815b1d0 con 0x7f23780b0200 2023-10-30T12:33:30.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.961+0000 7f237db80640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23781588a0 con 0x7f2360004d10 2023-10-30T12:33:30.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.961+0000 7f2377fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2378157430 0x7f2378150d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:30.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.961+0000 7f23777fe640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2360004d10 0x7f23780c1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:30.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.961+0000 7f2377fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2378157430 0x7f2378150d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44152/0 (socket says 172.21.15.89:44152) 2023-10-30T12:33:30.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.961+0000 7f2377fff640 1 -- 172.21.15.89:0/504895359 learned_addr learned my addr 172.21.15.89:0/504895359 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:33:30.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.961+0000 7f2376ffd640 1 --2- 172.21.15.89:0/504895359 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23780b0200 0x7f2378150840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:30.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.961+0000 7f2377fff640 1 -- 172.21.15.89:0/504895359 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2360004d10 msgr2=0x7f23780c1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:30.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.961+0000 7f2377fff640 1 --2- 172.21.15.89:0/504895359 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2360004d10 0x7f23780c1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:30.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.961+0000 7f2377fff640 1 -- 172.21.15.89:0/504895359 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23780b0200 msgr2=0x7f2378150840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:30.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.961+0000 7f2377fff640 1 --2- 172.21.15.89:0/504895359 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23780b0200 0x7f2378150840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:30.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.961+0000 7f2377fff640 1 -- 172.21.15.89:0/504895359 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2378158b20 con 0x7f2378157430 2023-10-30T12:33:30.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.961+0000 7f2377fff640 1 --2- 172.21.15.89:0/504895359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2378157430 0x7f2378150d80 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2370002240 tx=0x7f2370009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:30.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.965+0000 7f235ffff640 1 -- 172.21.15.89:0/504895359 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23700bc070 con 0x7f2378157430 2023-10-30T12:33:30.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.965+0000 7f235ffff640 1 -- 172.21.15.89:0/504895359 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23700ae990 con 0x7f2378157430 2023-10-30T12:33:30.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.965+0000 7f235ffff640 1 -- 172.21.15.89:0/504895359 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2370009490 con 0x7f2378157430 2023-10-30T12:33:30.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.965+0000 7f237db80640 1 -- 172.21.15.89:0/504895359 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23781eebb0 con 0x7f2378157430 2023-10-30T12:33:30.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.965+0000 7f237db80640 1 -- 172.21.15.89:0/504895359 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23781555f0 con 0x7f2378157430 2023-10-30T12:33:30.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.965+0000 7f237db80640 1 -- 172.21.15.89:0/504895359 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f237804e170 con 0x7f2378157430 2023-10-30T12:33:30.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.965+0000 7f235ffff640 1 -- 172.21.15.89:0/504895359 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f23700b4040 con 0x7f2378157430 2023-10-30T12:33:30.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.965+0000 7f235ffff640 1 --2- 172.21.15.89:0/504895359 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23640664d0 0x7f2364068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:30.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.965+0000 7f235ffff640 1 -- 172.21.15.89:0/504895359 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5976+0+0 (secure 0 0 0) 0x7f237011e530 con 0x7f2378157430 2023-10-30T12:33:30.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.965+0000 7f23777fe640 1 --2- 172.21.15.89:0/504895359 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23640664d0 0x7f2364068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:30.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.969+0000 7f23777fe640 1 --2- 172.21.15.89:0/504895359 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23640664d0 0x7f2364068990 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2368009b60 tx=0x7f2368009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:30.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:30.969+0000 7f235ffff640 1 -- 172.21.15.89:0/504895359 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f23700eb8c0 con 0x7f2378157430 2023-10-30T12:33:31.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:31.157+0000 7f237db80640 1 -- 172.21.15.89:0/504895359 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2378066510 con 0x7f2378157430 2023-10-30T12:33:31.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:31.157+0000 7f235ffff640 1 -- 172.21.15.89:0/504895359 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5740 (secure 0 0 0) 0x7f23700b2020 con 0x7f2378157430 2023-10-30T12:33:31.161 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:33:31.162 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:33:28.002806+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:rejoin","state_seq":25,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:33:31.163 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 27 2023-10-30T12:33:31.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:31.157+0000 7f237db80640 1 -- 172.21.15.89:0/504895359 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23640664d0 msgr2=0x7f2364068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:31.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:31.157+0000 7f237db80640 1 --2- 172.21.15.89:0/504895359 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23640664d0 0x7f2364068990 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2368009b60 tx=0x7f2368009190 comp rx=0 tx=0).stop 2023-10-30T12:33:31.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:31.157+0000 7f237db80640 1 -- 172.21.15.89:0/504895359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2378157430 msgr2=0x7f2378150d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:31.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:31.157+0000 7f237db80640 1 --2- 172.21.15.89:0/504895359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2378157430 0x7f2378150d80 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2370002240 tx=0x7f2370009750 comp rx=0 tx=0).stop 2023-10-30T12:33:31.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:31.157+0000 7f237db80640 1 -- 172.21.15.89:0/504895359 shutdown_connections 2023-10-30T12:33:31.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:31.157+0000 7f237db80640 1 --2- 172.21.15.89:0/504895359 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23640664d0 0x7f2364068990 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:31.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:31.157+0000 7f237db80640 1 --2- 172.21.15.89:0/504895359 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23780b0200 0x7f2378150840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:31.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:31.157+0000 7f237db80640 1 --2- 172.21.15.89:0/504895359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2378157430 0x7f2378150d80 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:31.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:31.157+0000 7f237db80640 1 --2- 172.21.15.89:0/504895359 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2360004d10 0x7f23780c1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:31.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:31.157+0000 7f237db80640 1 -- 172.21.15.89:0/504895359 >> 172.21.15.89:0/504895359 conn(0x7f23780c7910 msgr2=0x7f237806be40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:31.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:31.157+0000 7f237db80640 1 -- 172.21.15.89:0/504895359 shutdown_connections 2023-10-30T12:33:31.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:31.157+0000 7f237db80640 1 -- 172.21.15.89:0/504895359 wait complete. 2023-10-30T12:33:31.182 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:33:33.183 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:33:33.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.325+0000 7f8f19c9e640 1 Processor -- start 2023-10-30T12:33:33.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f19c9e640 1 -- start start 2023-10-30T12:33:33.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f19c9e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f140c05d0 0x7f8f1413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:33.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f19c9e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8f140c6f60 0x7f8f1413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:33.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f19c9e640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f140c0a50 con 0x7f8f140c6b80 2023-10-30T12:33:33.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f19c9e640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f1406f6e0 con 0x7f8f140c6f60 2023-10-30T12:33:33.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f19c9e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1406f9c0 con 0x7f8f140c05d0 2023-10-30T12:33:33.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f137fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f8f140c6b80 0x7f8f140bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44792/0 (socket says 172.21.15.89:44792) 2023-10-30T12:33:33.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f137fe640 1 -- 172.21.15.89:0/343361814 learned_addr learned my addr 172.21.15.89:0/343361814 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:33:33.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f13fff640 1 --2- 172.21.15.89:0/343361814 >> v2:172.21.15.192:3300/0 conn(0x7f8f140c6f60 0x7f8f1413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:33.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f12ffd640 1 --2- 172.21.15.89:0/343361814 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f140c05d0 0x7f8f1413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:33.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f127fc640 1 -- 172.21.15.89:0/343361814 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1016356015 0 0) 0x7f8f140c0a50 con 0x7f8f140c6b80 2023-10-30T12:33:33.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f127fc640 1 -- 172.21.15.89:0/343361814 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8efc003530 con 0x7f8f140c6b80 2023-10-30T12:33:33.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f127fc640 1 -- 172.21.15.89:0/343361814 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3495281891 0 0) 0x7f8efc003530 con 0x7f8f140c6b80 2023-10-30T12:33:33.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f127fc640 1 -- 172.21.15.89:0/343361814 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f140c05d0 msgr2=0x7f8f1413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:33.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f127fc640 1 --2- 172.21.15.89:0/343361814 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f140c05d0 0x7f8f1413ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:33.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f127fc640 1 -- 172.21.15.89:0/343361814 >> v2:172.21.15.192:3300/0 conn(0x7f8f140c6f60 msgr2=0x7f8f1413f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:33.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f127fc640 1 --2- 172.21.15.89:0/343361814 >> v2:172.21.15.192:3300/0 conn(0x7f8f140c6f60 0x7f8f1413f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:33.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f127fc640 1 -- 172.21.15.89:0/343361814 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1406fca0 con 0x7f8f140c6b80 2023-10-30T12:33:33.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f127fc640 1 -- 172.21.15.89:0/343361814 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f8f04001680 con 0x7f8f140c6b80 2023-10-30T12:33:33.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f127fc640 1 -- 172.21.15.89:0/343361814 >> v1:172.21.15.89:6789/0 conn(0x7f8f140c6b80 legacy=0x7f8f140bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:33.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f127fc640 1 --2- 172.21.15.89:0/343361814 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f140c05d0 0x7f8efc003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:33.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f127fc640 1 --2- 172.21.15.89:0/343361814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8efc003f00 0x7f8efc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:33.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f127fc640 1 --2- 172.21.15.89:0/343361814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8efc004d10 0x7f8efc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:33.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f127fc640 1 -- 172.21.15.89:0/343361814 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1406f6e0 con 0x7f8efc003f00 2023-10-30T12:33:33.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f127fc640 1 -- 172.21.15.89:0/343361814 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1406f9c0 con 0x7f8efc004d10 2023-10-30T12:33:33.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f127fc640 1 -- 172.21.15.89:0/343361814 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc00b990 con 0x7f8f140c05d0 2023-10-30T12:33:33.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f12ffd640 1 --2- 172.21.15.89:0/343361814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8efc003f00 0x7f8efc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:33.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f13fff640 1 --2- 172.21.15.89:0/343361814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8efc004d10 0x7f8efc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:33.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f137fe640 1 --2- 172.21.15.89:0/343361814 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f140c05d0 0x7f8efc003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:33.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f12ffd640 1 -- 172.21.15.89:0/343361814 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f140c05d0 msgr2=0x7f8efc003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:33.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f12ffd640 1 --2- 172.21.15.89:0/343361814 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f140c05d0 0x7f8efc003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:33.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f12ffd640 1 -- 172.21.15.89:0/343361814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8efc004d10 msgr2=0x7f8efc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:33.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f12ffd640 1 --2- 172.21.15.89:0/343361814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8efc004d10 0x7f8efc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:33.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.329+0000 7f8f12ffd640 1 -- 172.21.15.89:0/343361814 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8efc003530 con 0x7f8efc003f00 2023-10-30T12:33:33.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.333+0000 7f8f12ffd640 1 --2- 172.21.15.89:0/343361814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8efc003f00 0x7f8efc0047d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f8f08009ae0 tx=0x7f8f0801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c010c5d21df973 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:33.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.333+0000 7f8f127fc640 1 -- 172.21.15.89:0/343361814 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f080a5530 con 0x7f8efc003f00 2023-10-30T12:33:33.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.333+0000 7f8f127fc640 1 -- 172.21.15.89:0/343361814 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f080a5690 con 0x7f8efc003f00 2023-10-30T12:33:33.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.333+0000 7f8f127fc640 1 -- 172.21.15.89:0/343361814 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f080a5a40 con 0x7f8efc003f00 2023-10-30T12:33:33.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.333+0000 7f8f19c9e640 1 -- 172.21.15.89:0/343361814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8efc003f00 msgr2=0x7f8efc0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:33.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.333+0000 7f8f19c9e640 1 --2- 172.21.15.89:0/343361814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8efc003f00 0x7f8efc0047d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f8f08009ae0 tx=0x7f8f0801c9f0 comp rx=0 tx=0).stop 2023-10-30T12:33:33.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.333+0000 7f8f13fff640 1 -- 172.21.15.89:0/343361814 reap_dead start 2023-10-30T12:33:33.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.333+0000 7f8f19c9e640 1 -- 172.21.15.89:0/343361814 shutdown_connections 2023-10-30T12:33:33.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.333+0000 7f8f19c9e640 1 -- 172.21.15.89:0/343361814 >> 172.21.15.89:0/343361814 conn(0x7f8f140c7910 msgr2=0x7f8f140c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:33.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.333+0000 7f8f19c9e640 1 -- 172.21.15.89:0/343361814 shutdown_connections 2023-10-30T12:33:33.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.333+0000 7f8f19c9e640 1 -- 172.21.15.89:0/343361814 wait complete. 2023-10-30T12:33:33.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.337+0000 7f8f19c9e640 1 Processor -- start 2023-10-30T12:33:33.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.337+0000 7f8f19c9e640 1 -- start start 2023-10-30T12:33:33.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.337+0000 7f8f19c9e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f1413cf90 0x7f8f140c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:33.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.337+0000 7f8f19c9e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f1413d370 0x7f8f1413e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:33.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.337+0000 7f8f19c9e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f1413d870 0x7f8f14151300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:33.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.337+0000 7f8f19c9e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f140c44b0 con 0x7f8f1413d870 2023-10-30T12:33:33.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.337+0000 7f8f19c9e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f140c4e10 con 0x7f8f1413d370 2023-10-30T12:33:33.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.337+0000 7f8f19c9e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f140c4140 con 0x7f8f1413cf90 2023-10-30T12:33:33.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.337+0000 7f8f13fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f1413d870 0x7f8f14151300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:33.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.337+0000 7f8f13fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f1413d870 0x7f8f14151300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44174/0 (socket says 172.21.15.89:44174) 2023-10-30T12:33:33.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.337+0000 7f8f13fff640 1 -- 172.21.15.89:0/1161547812 learned_addr learned my addr 172.21.15.89:0/1161547812 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:33:33.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.337+0000 7f8f137fe640 1 --2- 172.21.15.89:0/1161547812 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f1413cf90 0x7f8f140c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:33.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.337+0000 7f8f12ffd640 1 --2- 172.21.15.89:0/1161547812 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f1413d370 0x7f8f1413e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:33.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.337+0000 7f8f13fff640 1 -- 172.21.15.89:0/1161547812 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f1413cf90 msgr2=0x7f8f140c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:33.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.337+0000 7f8f13fff640 1 --2- 172.21.15.89:0/1161547812 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f1413cf90 0x7f8f140c48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:33.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.337+0000 7f8f13fff640 1 -- 172.21.15.89:0/1161547812 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f1413d370 msgr2=0x7f8f1413e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:33.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.337+0000 7f8f13fff640 1 --2- 172.21.15.89:0/1161547812 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f1413d370 0x7f8f1413e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:33.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.337+0000 7f8f13fff640 1 -- 172.21.15.89:0/1161547812 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f140ca2c0 con 0x7f8f1413d870 2023-10-30T12:33:33.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.337+0000 7f8f12ffd640 1 --2- 172.21.15.89:0/1161547812 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f1413d370 0x7f8f1413e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:33:33.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.337+0000 7f8f13fff640 1 --2- 172.21.15.89:0/1161547812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f1413d870 0x7f8f14151300 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f8f140b1190 tx=0x7f8f0c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:33.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.337+0000 7f8efbfff640 1 -- 172.21.15.89:0/1161547812 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c00e070 con 0x7f8f1413d870 2023-10-30T12:33:33.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.337+0000 7f8efbfff640 1 -- 172.21.15.89:0/1161547812 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f0c011e40 con 0x7f8f1413d870 2023-10-30T12:33:33.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.337+0000 7f8f19c9e640 1 -- 172.21.15.89:0/1161547812 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f140ca5a0 con 0x7f8f1413d870 2023-10-30T12:33:33.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.337+0000 7f8efbfff640 1 -- 172.21.15.89:0/1161547812 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c0039a0 con 0x7f8f1413d870 2023-10-30T12:33:33.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.337+0000 7f8f19c9e640 1 -- 172.21.15.89:0/1161547812 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f1413e7c0 con 0x7f8f1413d870 2023-10-30T12:33:33.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.337+0000 7f8f19c9e640 1 -- 172.21.15.89:0/1161547812 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f140c45f0 con 0x7f8f1413d870 2023-10-30T12:33:33.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.341+0000 7f8efbfff640 1 -- 172.21.15.89:0/1161547812 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8f0c003b20 con 0x7f8f1413d870 2023-10-30T12:33:33.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.341+0000 7f8efbfff640 1 --2- 172.21.15.89:0/1161547812 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8f000664d0 0x7f8f00068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:33.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.341+0000 7f8efbfff640 1 -- 172.21.15.89:0/1161547812 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5976+0+0 (secure 0 0 0) 0x7f8f0c084690 con 0x7f8f1413d870 2023-10-30T12:33:33.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.341+0000 7f8f137fe640 1 --2- 172.21.15.89:0/1161547812 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8f000664d0 0x7f8f00068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:33.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.341+0000 7f8f137fe640 1 --2- 172.21.15.89:0/1161547812 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8f000664d0 0x7f8f00068990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8f040054e0 tx=0x7f8f040014f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:33.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.345+0000 7f8efbfff640 1 -- 172.21.15.89:0/1161547812 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8f140c45f0 con 0x7f8f1413d870 2023-10-30T12:33:33.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.525+0000 7f8f19c9e640 1 -- 172.21.15.89:0/1161547812 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f140c45f0 con 0x7f8f1413d870 2023-10-30T12:33:33.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.525+0000 7f8efbfff640 1 -- 172.21.15.89:0/1161547812 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5740 (secure 0 0 0) 0x7f8f140c45f0 con 0x7f8f1413d870 2023-10-30T12:33:33.531 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:33:33.532 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:33:28.002806+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:rejoin","state_seq":25,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:33:33.532 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 27 2023-10-30T12:33:33.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.529+0000 7f8f19c9e640 1 -- 172.21.15.89:0/1161547812 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8f000664d0 msgr2=0x7f8f00068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:33.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.529+0000 7f8f19c9e640 1 --2- 172.21.15.89:0/1161547812 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8f000664d0 0x7f8f00068990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8f040054e0 tx=0x7f8f040014f0 comp rx=0 tx=0).stop 2023-10-30T12:33:33.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.529+0000 7f8f19c9e640 1 -- 172.21.15.89:0/1161547812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f1413d870 msgr2=0x7f8f14151300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:33.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.529+0000 7f8f19c9e640 1 --2- 172.21.15.89:0/1161547812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f1413d870 0x7f8f14151300 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f8f140b1190 tx=0x7f8f0c0115a0 comp rx=0 tx=0).stop 2023-10-30T12:33:33.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.529+0000 7f8f19c9e640 1 -- 172.21.15.89:0/1161547812 shutdown_connections 2023-10-30T12:33:33.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.529+0000 7f8f19c9e640 1 --2- 172.21.15.89:0/1161547812 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8f000664d0 0x7f8f00068990 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:33.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.529+0000 7f8f19c9e640 1 --2- 172.21.15.89:0/1161547812 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f1413d370 0x7f8f1413e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:33.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.529+0000 7f8f19c9e640 1 --2- 172.21.15.89:0/1161547812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f1413d870 0x7f8f14151300 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:33.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.529+0000 7f8f19c9e640 1 --2- 172.21.15.89:0/1161547812 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f1413cf90 0x7f8f140c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:33.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.529+0000 7f8f19c9e640 1 -- 172.21.15.89:0/1161547812 >> 172.21.15.89:0/1161547812 conn(0x7f8f140c7910 msgr2=0x7f8f140bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:33.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.529+0000 7f8f19c9e640 1 -- 172.21.15.89:0/1161547812 shutdown_connections 2023-10-30T12:33:33.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:33.529+0000 7f8f19c9e640 1 -- 172.21.15.89:0/1161547812 wait complete. 2023-10-30T12:33:33.553 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:33:35.338 INFO:tasks.ceph.mds.f.smithi192.stderr:2023-10-30T12:33:35.334+0000 7f0036e29640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T12:33:35.554 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:33:35.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.697+0000 7f6b926d6640 1 Processor -- start 2023-10-30T12:33:35.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.697+0000 7f6b926d6640 1 -- start start 2023-10-30T12:33:35.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.697+0000 7f6b926d6640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6b8c0c6b80 0x7f6b8c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:35.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.697+0000 7f6b926d6640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b8c0c6f60 0x7f6b8c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:35.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.697+0000 7f6b926d6640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6b8c0c8430 0x7f6b8c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:35.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.697+0000 7f6b926d6640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b8c0c0a50 con 0x7f6b8c0c8430 2023-10-30T12:33:35.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.697+0000 7f6b926d6640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b8c06f6e0 con 0x7f6b8c0c6b80 2023-10-30T12:33:35.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.697+0000 7f6b926d6640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8c06f9c0 con 0x7f6b8c0c6f60 2023-10-30T12:33:35.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.701+0000 7f6b90c73640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6b8c0c8430 0x7f6b8c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:35.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.701+0000 7f6b90c73640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6b8c0c8430 0x7f6b8c13ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44178/0 (socket says 172.21.15.89:44178) 2023-10-30T12:33:35.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.701+0000 7f6b90c73640 1 -- 172.21.15.89:0/1228968029 learned_addr learned my addr 172.21.15.89:0/1228968029 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:33:35.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.701+0000 7f6b8bfff640 1 --2- 172.21.15.89:0/1228968029 >> v2:172.21.15.192:3300/0 conn(0x7f6b8c0c6b80 0x7f6b8c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:35.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.701+0000 7f6b8b7fe640 1 --2- 172.21.15.89:0/1228968029 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b8c0c6f60 0x7f6b8c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:35.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.701+0000 7f6b90c73640 1 -- 172.21.15.89:0/1228968029 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b8c0c6f60 msgr2=0x7f6b8c0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:35.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.701+0000 7f6b90c73640 1 --2- 172.21.15.89:0/1228968029 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b8c0c6f60 0x7f6b8c0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:35.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.701+0000 7f6b90c73640 1 -- 172.21.15.89:0/1228968029 >> v2:172.21.15.192:3300/0 conn(0x7f6b8c0c6b80 msgr2=0x7f6b8c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:35.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.701+0000 7f6b90c73640 1 --2- 172.21.15.89:0/1228968029 >> v2:172.21.15.192:3300/0 conn(0x7f6b8c0c6b80 0x7f6b8c0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:35.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.701+0000 7f6b90c73640 1 -- 172.21.15.89:0/1228968029 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b8c06fca0 con 0x7f6b8c0c8430 2023-10-30T12:33:35.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.701+0000 7f6b90c73640 1 --2- 172.21.15.89:0/1228968029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b8c0c8430 0x7f6b8c13ed90 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c06f1b0 tx=0x7f6b8401c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77f60e3a16698e0f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:35.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.701+0000 7f6b8affd640 1 -- 172.21.15.89:0/1228968029 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b840a56d0 con 0x7f6b8c0c8430 2023-10-30T12:33:35.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.701+0000 7f6b8affd640 1 -- 172.21.15.89:0/1228968029 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b840a5830 con 0x7f6b8c0c8430 2023-10-30T12:33:35.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.701+0000 7f6b8affd640 1 -- 172.21.15.89:0/1228968029 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b840a5bc0 con 0x7f6b8c0c8430 2023-10-30T12:33:35.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.705+0000 7f6b926d6640 1 -- 172.21.15.89:0/1228968029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b8c0c8430 msgr2=0x7f6b8c13ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:35.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.705+0000 7f6b926d6640 1 --2- 172.21.15.89:0/1228968029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b8c0c8430 0x7f6b8c13ed90 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c06f1b0 tx=0x7f6b8401c940 comp rx=0 tx=0).stop 2023-10-30T12:33:35.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.705+0000 7f6b926d6640 1 -- 172.21.15.89:0/1228968029 shutdown_connections 2023-10-30T12:33:35.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.705+0000 7f6b926d6640 1 --2- 172.21.15.89:0/1228968029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b8c0c8430 0x7f6b8c13ed90 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:35.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.705+0000 7f6b926d6640 1 --2- 172.21.15.89:0/1228968029 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b8c0c6f60 0x7f6b8c0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:35.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.705+0000 7f6b926d6640 1 --2- 172.21.15.89:0/1228968029 >> v2:172.21.15.192:3300/0 conn(0x7f6b8c0c6b80 0x7f6b8c0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:35.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.705+0000 7f6b926d6640 1 -- 172.21.15.89:0/1228968029 >> 172.21.15.89:0/1228968029 conn(0x7f6b8c0c7910 msgr2=0x7f6b8c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:35.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.705+0000 7f6b926d6640 1 -- 172.21.15.89:0/1228968029 shutdown_connections 2023-10-30T12:33:35.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.705+0000 7f6b926d6640 1 -- 172.21.15.89:0/1228968029 wait complete. 2023-10-30T12:33:35.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.705+0000 7f6b926d6640 1 Processor -- start 2023-10-30T12:33:35.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.705+0000 7f6b926d6640 1 -- start start 2023-10-30T12:33:35.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.705+0000 7f6b926d6640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b8c0c6b80 0x7f6b8c06a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:35.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.705+0000 7f6b926d6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b8c0c6f60 0x7f6b8c13c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:35.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.705+0000 7f6b926d6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b8c0c8430 0x7f6b8c153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:35.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.705+0000 7f6b926d6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8c13d190 con 0x7f6b8c0c6f60 2023-10-30T12:33:35.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.705+0000 7f6b926d6640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8c1588d0 con 0x7f6b8c0c8430 2023-10-30T12:33:35.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.705+0000 7f6b926d6640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8c13e2e0 con 0x7f6b8c0c6b80 2023-10-30T12:33:35.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.705+0000 7f6b8b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b8c0c6f60 0x7f6b8c13c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:35.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.705+0000 7f6b8b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b8c0c6f60 0x7f6b8c13c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44190/0 (socket says 172.21.15.89:44190) 2023-10-30T12:33:35.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.705+0000 7f6b8b7fe640 1 -- 172.21.15.89:0/2594764550 learned_addr learned my addr 172.21.15.89:0/2594764550 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:33:35.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.705+0000 7f6b8bfff640 1 --2- 172.21.15.89:0/2594764550 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b8c0c6b80 0x7f6b8c06a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:35.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.705+0000 7f6b90c73640 1 --2- 172.21.15.89:0/2594764550 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b8c0c8430 0x7f6b8c153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:35.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.705+0000 7f6b8b7fe640 1 -- 172.21.15.89:0/2594764550 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b8c0c6b80 msgr2=0x7f6b8c06a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:35.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.705+0000 7f6b8b7fe640 1 --2- 172.21.15.89:0/2594764550 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b8c0c6b80 0x7f6b8c06a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:35.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.705+0000 7f6b8b7fe640 1 -- 172.21.15.89:0/2594764550 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b8c0c8430 msgr2=0x7f6b8c153120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:35.718 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.705+0000 7f6b8b7fe640 1 --2- 172.21.15.89:0/2594764550 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b8c0c8430 0x7f6b8c153120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:35.718 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.705+0000 7f6b8b7fe640 1 -- 172.21.15.89:0/2594764550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b84009b30 con 0x7f6b8c0c6f60 2023-10-30T12:33:35.718 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.709+0000 7f6b8b7fe640 1 --2- 172.21.15.89:0/2594764550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b8c0c6f60 0x7f6b8c13c6a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f6b8000e960 tx=0x7f6b8000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:35.718 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.709+0000 7f6b88ff9640 1 -- 172.21.15.89:0/2594764550 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b80009690 con 0x7f6b8c0c6f60 2023-10-30T12:33:35.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.709+0000 7f6b88ff9640 1 -- 172.21.15.89:0/2594764550 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b8000ca00 con 0x7f6b8c0c6f60 2023-10-30T12:33:35.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.709+0000 7f6b88ff9640 1 -- 172.21.15.89:0/2594764550 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8000cd70 con 0x7f6b8c0c6f60 2023-10-30T12:33:35.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.709+0000 7f6b926d6640 1 -- 172.21.15.89:0/2594764550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b8c0cab80 con 0x7f6b8c0c6f60 2023-10-30T12:33:35.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.709+0000 7f6b926d6640 1 -- 172.21.15.89:0/2594764550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b8c159810 con 0x7f6b8c0c6f60 2023-10-30T12:33:35.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.709+0000 7f6b926d6640 1 -- 172.21.15.89:0/2594764550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b8c04e170 con 0x7f6b8c0c6f60 2023-10-30T12:33:35.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.709+0000 7f6b88ff9640 1 -- 172.21.15.89:0/2594764550 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6b80013470 con 0x7f6b8c0c6f60 2023-10-30T12:33:35.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.709+0000 7f6b88ff9640 1 --2- 172.21.15.89:0/2594764550 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6b780664a0 0x7f6b78068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:35.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.713+0000 7f6b88ff9640 1 -- 172.21.15.89:0/2594764550 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5976+0+0 (secure 0 0 0) 0x7f6b80084180 con 0x7f6b8c0c6f60 2023-10-30T12:33:35.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.713+0000 7f6b8bfff640 1 --2- 172.21.15.89:0/2594764550 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6b780664a0 0x7f6b78068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:35.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.713+0000 7f6b8bfff640 1 --2- 172.21.15.89:0/2594764550 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6b780664a0 0x7f6b78068960 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c009a70 tx=0x7f6b7c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:35.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.717+0000 7f6b88ff9640 1 -- 172.21.15.89:0/2594764550 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6b800515a0 con 0x7f6b8c0c6f60 2023-10-30T12:33:35.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.897+0000 7f6b926d6640 1 -- 172.21.15.89:0/2594764550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b8c0c1040 con 0x7f6b8c0c6f60 2023-10-30T12:33:35.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.897+0000 7f6b88ff9640 1 -- 172.21.15.89:0/2594764550 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5740 (secure 0 0 0) 0x7f6b80054bc0 con 0x7f6b8c0c6f60 2023-10-30T12:33:35.904 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:33:35.904 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:33:33.100889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:33:35.905 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 28 2023-10-30T12:33:35.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.901+0000 7f6b926d6640 1 -- 172.21.15.89:0/2594764550 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6b780664a0 msgr2=0x7f6b78068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:35.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.901+0000 7f6b926d6640 1 --2- 172.21.15.89:0/2594764550 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6b780664a0 0x7f6b78068960 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c009a70 tx=0x7f6b7c008040 comp rx=0 tx=0).stop 2023-10-30T12:33:35.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.901+0000 7f6b926d6640 1 -- 172.21.15.89:0/2594764550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b8c0c6f60 msgr2=0x7f6b8c13c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:35.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.901+0000 7f6b926d6640 1 --2- 172.21.15.89:0/2594764550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b8c0c6f60 0x7f6b8c13c6a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f6b8000e960 tx=0x7f6b8000ee20 comp rx=0 tx=0).stop 2023-10-30T12:33:35.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.901+0000 7f6b926d6640 1 -- 172.21.15.89:0/2594764550 shutdown_connections 2023-10-30T12:33:35.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.901+0000 7f6b926d6640 1 --2- 172.21.15.89:0/2594764550 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6b780664a0 0x7f6b78068960 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:35.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.901+0000 7f6b926d6640 1 --2- 172.21.15.89:0/2594764550 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b8c0c8430 0x7f6b8c153120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:35.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.901+0000 7f6b926d6640 1 --2- 172.21.15.89:0/2594764550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b8c0c6f60 0x7f6b8c13c6a0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:35.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.901+0000 7f6b926d6640 1 --2- 172.21.15.89:0/2594764550 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6b8c0c6b80 0x7f6b8c06a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:35.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.901+0000 7f6b926d6640 1 -- 172.21.15.89:0/2594764550 >> 172.21.15.89:0/2594764550 conn(0x7f6b8c0c7910 msgr2=0x7f6b8c139650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:35.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.901+0000 7f6b926d6640 1 -- 172.21.15.89:0/2594764550 shutdown_connections 2023-10-30T12:33:35.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:35.901+0000 7f6b926d6640 1 -- 172.21.15.89:0/2594764550 wait complete. 2023-10-30T12:33:35.924 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=2, replacing gid=4242 2023-10-30T12:33:35.925 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.c 2023-10-30T12:33:39.512 INFO:tasks.workunit.client.0.smithi089.stdout:./254/.snap/snap-subdir-test 2023-10-30T12:33:39.524 INFO:tasks.workunit.client.0.smithi089.stdout:./113/.snap/snap-subdir-test 2023-10-30T12:33:39.530 INFO:tasks.workunit.client.0.smithi089.stdout:./152/.snap/snap-subdir-test 2023-10-30T12:33:39.646 INFO:tasks.workunit.client.0.smithi089.stdout:./121/.snap/snap-subdir-test 2023-10-30T12:33:39.659 INFO:tasks.workunit.client.0.smithi089.stdout:./118/.snap/snap-subdir-test 2023-10-30T12:33:39.666 INFO:tasks.workunit.client.0.smithi089.stdout:./196/.snap/snap-subdir-test 2023-10-30T12:33:39.679 INFO:tasks.workunit.client.0.smithi089.stdout:./124/.snap/snap-subdir-test 2023-10-30T12:33:39.685 INFO:tasks.workunit.client.0.smithi089.stdout:./239/.snap/snap-subdir-test 2023-10-30T12:33:39.692 INFO:tasks.workunit.client.0.smithi089.stdout:./188/.snap/snap-subdir-test 2023-10-30T12:33:39.699 INFO:tasks.workunit.client.0.smithi089.stdout:./287/.snap/snap-subdir-test 2023-10-30T12:33:39.704 INFO:tasks.workunit.client.0.smithi089.stdout:./275/.snap/snap-subdir-test 2023-10-30T12:33:39.710 INFO:tasks.workunit.client.0.smithi089.stdout:./259/.snap/snap-subdir-test 2023-10-30T12:33:39.717 INFO:tasks.workunit.client.0.smithi089.stdout:./224/.snap/snap-subdir-test 2023-10-30T12:33:39.722 INFO:tasks.workunit.client.0.smithi089.stdout:./264/.snap/snap-subdir-test 2023-10-30T12:33:39.737 INFO:tasks.workunit.client.0.smithi089.stdout:./145/.snap/snap-subdir-test 2023-10-30T12:33:39.744 INFO:tasks.workunit.client.0.smithi089.stdout:./279/.snap/snap-subdir-test 2023-10-30T12:33:39.750 INFO:tasks.workunit.client.0.smithi089.stdout:./263/.snap/snap-subdir-test 2023-10-30T12:33:39.756 INFO:tasks.workunit.client.0.smithi089.stdout:./130/.snap/snap-subdir-test 2023-10-30T12:33:39.762 INFO:tasks.workunit.client.0.smithi089.stdout:./193/.snap/snap-subdir-test 2023-10-30T12:33:39.769 INFO:tasks.workunit.client.0.smithi089.stdout:./223/.snap/snap-subdir-test 2023-10-30T12:33:39.775 INFO:tasks.workunit.client.0.smithi089.stdout:./207/.snap/snap-subdir-test 2023-10-30T12:33:39.781 INFO:tasks.workunit.client.0.smithi089.stdout:./276/.snap/snap-subdir-test 2023-10-30T12:33:39.788 INFO:tasks.workunit.client.0.smithi089.stdout:./271/.snap/snap-subdir-test 2023-10-30T12:33:39.793 INFO:tasks.workunit.client.0.smithi089.stdout:./251/.snap/snap-subdir-test 2023-10-30T12:33:39.799 INFO:tasks.workunit.client.0.smithi089.stdout:./175/.snap/snap-subdir-test 2023-10-30T12:33:39.805 INFO:tasks.workunit.client.0.smithi089.stdout:./222/.snap/snap-subdir-test 2023-10-30T12:33:39.811 INFO:tasks.workunit.client.0.smithi089.stdout:./220/.snap/snap-subdir-test 2023-10-30T12:33:39.817 INFO:tasks.workunit.client.0.smithi089.stdout:./203/.snap/snap-subdir-test 2023-10-30T12:33:39.824 INFO:tasks.workunit.client.0.smithi089.stdout:./181/.snap/snap-subdir-test 2023-10-30T12:33:39.831 INFO:tasks.workunit.client.0.smithi089.stdout:./286/.snap/snap-subdir-test 2023-10-30T12:33:39.836 INFO:tasks.workunit.client.0.smithi089.stdout:./127/.snap/snap-subdir-test 2023-10-30T12:33:39.842 INFO:tasks.workunit.client.0.smithi089.stdout:./108/.snap/snap-subdir-test 2023-10-30T12:33:39.848 INFO:tasks.workunit.client.0.smithi089.stdout:./183/.snap/snap-subdir-test 2023-10-30T12:33:39.853 INFO:tasks.workunit.client.0.smithi089.stdout:./294/.snap/snap-subdir-test 2023-10-30T12:33:40.086 INFO:tasks.workunit.client.0.smithi089.stdout:./235/.snap/snap-subdir-test 2023-10-30T12:33:40.092 INFO:tasks.workunit.client.0.smithi089.stdout:./177/.snap/snap-subdir-test 2023-10-30T12:33:40.098 INFO:tasks.workunit.client.0.smithi089.stdout:./169/.snap/snap-subdir-test 2023-10-30T12:33:40.104 INFO:tasks.workunit.client.0.smithi089.stdout:./212/.snap/snap-subdir-test 2023-10-30T12:33:40.109 INFO:tasks.workunit.client.0.smithi089.stdout:./154/.snap/snap-subdir-test 2023-10-30T12:33:40.115 INFO:tasks.workunit.client.0.smithi089.stdout:./171/.snap/snap-subdir-test 2023-10-30T12:33:40.120 INFO:tasks.workunit.client.0.smithi089.stdout:./139/.snap/snap-subdir-test 2023-10-30T12:33:40.126 INFO:tasks.workunit.client.0.smithi089.stdout:./102/.snap/snap-subdir-test 2023-10-30T12:33:40.132 INFO:tasks.workunit.client.0.smithi089.stdout:./129/.snap/snap-subdir-test 2023-10-30T12:33:44.926 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T12:33:44.926 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T12:33:44.927 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T12:33:44.930 INFO:tasks.ceph.mds.c:Started 2023-10-30T12:33:44.931 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:33:45.017 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2023-10-30T12:33:45.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.065+0000 7f46d04d6640 1 Processor -- start 2023-10-30T12:33:45.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.065+0000 7f46d04d6640 1 -- start start 2023-10-30T12:33:45.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.069+0000 7f46d04d6640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f46c80c6b80 0x7f46c80bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:45.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.069+0000 7f46d04d6640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f46c80c6f60 0x7f46c80c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:45.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.069+0000 7f46d04d6640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46c80c0a50 con 0x7f46c80c8430 2023-10-30T12:33:45.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.069+0000 7f46d04d6640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46c806f6e0 con 0x7f46c80c6b80 2023-10-30T12:33:45.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.069+0000 7f46d04d6640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46c806f9c0 con 0x7f46c80c6f60 2023-10-30T12:33:45.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.069+0000 7f46ce272640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f46c80c6b80 0x7f46c80bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:45.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.069+0000 7f46ce272640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f46c80c6b80 0x7f46c80bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59502/0 (socket says 172.21.15.89:59502) 2023-10-30T12:33:45.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.069+0000 7f46ce272640 1 -- 172.21.15.89:0/1974415321 learned_addr learned my addr 172.21.15.89:0/1974415321 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:33:45.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.069+0000 7f46cea73640 1 --1- 172.21.15.89:0/1974415321 >> v1:172.21.15.89:6789/0 conn(0x7f46c80c8430 0x7f46c813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:55746/0 (socket says 172.21.15.89:55746) 2023-10-30T12:33:45.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.069+0000 7f46cda71640 1 --2- 172.21.15.89:0/1974415321 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f46c80c6f60 0x7f46c80c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:45.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.069+0000 7f46ce272640 1 -- 172.21.15.89:0/1974415321 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f46c80c6f60 msgr2=0x7f46c80c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:45.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.069+0000 7f46ce272640 1 --2- 172.21.15.89:0/1974415321 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f46c80c6f60 0x7f46c80c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:45.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.069+0000 7f46ce272640 1 -- 172.21.15.89:0/1974415321 >> v1:172.21.15.89:6789/0 conn(0x7f46c80c8430 legacy=0x7f46c813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:45.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.069+0000 7f46ce272640 1 -- 172.21.15.89:0/1974415321 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c806fca0 con 0x7f46c80c6b80 2023-10-30T12:33:45.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.073+0000 7f46ce272640 1 --2- 172.21.15.89:0/1974415321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46c80c6b80 0x7f46c80bfec0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f46c4009870 tx=0x7f46c401c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adc2bf2be2218625 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:45.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.073+0000 7f46cd270640 1 -- 172.21.15.89:0/1974415321 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c40ac620 con 0x7f46c80c6b80 2023-10-30T12:33:45.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.073+0000 7f46cd270640 1 -- 172.21.15.89:0/1974415321 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c40ac780 con 0x7f46c80c6b80 2023-10-30T12:33:45.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.073+0000 7f46cd270640 1 -- 172.21.15.89:0/1974415321 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c40acb10 con 0x7f46c80c6b80 2023-10-30T12:33:45.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.073+0000 7f46d04d6640 1 -- 172.21.15.89:0/1974415321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46c80c6b80 msgr2=0x7f46c80bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:45.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.073+0000 7f46d04d6640 1 --2- 172.21.15.89:0/1974415321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46c80c6b80 0x7f46c80bfec0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f46c4009870 tx=0x7f46c401c4b0 comp rx=0 tx=0).stop 2023-10-30T12:33:45.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.073+0000 7f46d04d6640 1 -- 172.21.15.89:0/1974415321 shutdown_connections 2023-10-30T12:33:45.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.073+0000 7f46d04d6640 1 --2- 172.21.15.89:0/1974415321 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f46c80c6f60 0x7f46c80c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:45.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.073+0000 7f46d04d6640 1 --2- 172.21.15.89:0/1974415321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46c80c6b80 0x7f46c80bfec0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:45.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.073+0000 7f46d04d6640 1 -- 172.21.15.89:0/1974415321 >> 172.21.15.89:0/1974415321 conn(0x7f46c80c7910 msgr2=0x7f46c80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:45.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.073+0000 7f46d04d6640 1 -- 172.21.15.89:0/1974415321 shutdown_connections 2023-10-30T12:33:45.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.073+0000 7f46d04d6640 1 -- 172.21.15.89:0/1974415321 wait complete. 2023-10-30T12:33:45.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.073+0000 7f46d04d6640 1 Processor -- start 2023-10-30T12:33:45.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.073+0000 7f46d04d6640 1 -- start start 2023-10-30T12:33:45.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.073+0000 7f46d04d6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46c80c6b80 0x7f46c813aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:45.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.073+0000 7f46d04d6640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f46c80c6f60 0x7f46c8133100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:45.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.073+0000 7f46d04d6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46c80c8430 0x7f46c8133640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:45.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.073+0000 7f46d04d6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c8132cb0 con 0x7f46c80c8430 2023-10-30T12:33:45.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.073+0000 7f46d04d6640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c8132680 con 0x7f46c80c6b80 2023-10-30T12:33:45.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.073+0000 7f46d04d6640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46c8158500 con 0x7f46c80c6f60 2023-10-30T12:33:45.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.077+0000 7f46cea73640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46c80c8430 0x7f46c8133640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:45.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.077+0000 7f46cea73640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46c80c8430 0x7f46c8133640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59504/0 (socket says 172.21.15.89:59504) 2023-10-30T12:33:45.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.077+0000 7f46cea73640 1 -- 172.21.15.89:0/3182093392 learned_addr learned my addr 172.21.15.89:0/3182093392 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:33:45.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.077+0000 7f46ce272640 1 --2- 172.21.15.89:0/3182093392 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46c80c6b80 0x7f46c813aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:45.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.077+0000 7f46cda71640 1 --2- 172.21.15.89:0/3182093392 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f46c80c6f60 0x7f46c8133100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:45.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.077+0000 7f46cea73640 1 -- 172.21.15.89:0/3182093392 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f46c80c6f60 msgr2=0x7f46c8133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:45.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.077+0000 7f46cea73640 1 --2- 172.21.15.89:0/3182093392 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f46c80c6f60 0x7f46c8133100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:45.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.077+0000 7f46cea73640 1 -- 172.21.15.89:0/3182093392 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46c80c6b80 msgr2=0x7f46c813aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:45.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.077+0000 7f46cea73640 1 --2- 172.21.15.89:0/3182093392 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46c80c6b80 0x7f46c813aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:45.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.077+0000 7f46cea73640 1 -- 172.21.15.89:0/3182093392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c40094d0 con 0x7f46c80c8430 2023-10-30T12:33:45.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.077+0000 7f46cea73640 1 --2- 172.21.15.89:0/3182093392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46c80c8430 0x7f46c8133640 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f46c000c700 tx=0x7f46c000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:45.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.077+0000 7f46beffd640 1 -- 172.21.15.89:0/3182093392 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c000ce10 con 0x7f46c80c8430 2023-10-30T12:33:45.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.077+0000 7f46beffd640 1 -- 172.21.15.89:0/3182093392 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c000a640 con 0x7f46c80c8430 2023-10-30T12:33:45.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.077+0000 7f46d04d6640 1 -- 172.21.15.89:0/3182093392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46c81587a0 con 0x7f46c80c8430 2023-10-30T12:33:45.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.077+0000 7f46beffd640 1 -- 172.21.15.89:0/3182093392 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c000a9b0 con 0x7f46c80c8430 2023-10-30T12:33:45.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.077+0000 7f46d04d6640 1 -- 172.21.15.89:0/3182093392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46c81f1c00 con 0x7f46c80c8430 2023-10-30T12:33:45.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.077+0000 7f46d04d6640 1 -- 172.21.15.89:0/3182093392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46c804e170 con 0x7f46c80c8430 2023-10-30T12:33:45.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.077+0000 7f46beffd640 1 -- 172.21.15.89:0/3182093392 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f46c000ab10 con 0x7f46c80c8430 2023-10-30T12:33:45.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.077+0000 7f46beffd640 1 --2- 172.21.15.89:0/3182093392 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f46ac0664a0 0x7f46ac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:45.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.081+0000 7f46beffd640 1 -- 172.21.15.89:0/3182093392 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5976+0+0 (secure 0 0 0) 0x7f46c00836f0 con 0x7f46c80c8430 2023-10-30T12:33:45.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.081+0000 7f46ce272640 1 --2- 172.21.15.89:0/3182093392 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f46ac0664a0 0x7f46ac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:45.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.081+0000 7f46ce272640 1 --2- 172.21.15.89:0/3182093392 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f46ac0664a0 0x7f46ac068960 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f46c40ae000 tx=0x7f46c40a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:45.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.085+0000 7f46beffd640 1 -- 172.21.15.89:0/3182093392 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f46c0050a70 con 0x7f46c80c8430 2023-10-30T12:33:45.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.265+0000 7f46d04d6640 1 -- 172.21.15.89:0/3182093392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46c8135450 con 0x7f46c80c8430 2023-10-30T12:33:45.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.265+0000 7f46beffd640 1 -- 172.21.15.89:0/3182093392 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6529 (secure 0 0 0) 0x7f46c0054090 con 0x7f46c80c8430 2023-10-30T12:33:45.272 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:33:45.272 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5139,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:33:44.123506+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:33:45.273 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2023-10-30T12:33:45.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.269+0000 7f46d04d6640 1 -- 172.21.15.89:0/3182093392 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f46ac0664a0 msgr2=0x7f46ac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:45.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.269+0000 7f46d04d6640 1 --2- 172.21.15.89:0/3182093392 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f46ac0664a0 0x7f46ac068960 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f46c40ae000 tx=0x7f46c40a7000 comp rx=0 tx=0).stop 2023-10-30T12:33:45.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.269+0000 7f46d04d6640 1 -- 172.21.15.89:0/3182093392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46c80c8430 msgr2=0x7f46c8133640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:45.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.269+0000 7f46d04d6640 1 --2- 172.21.15.89:0/3182093392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46c80c8430 0x7f46c8133640 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f46c000c700 tx=0x7f46c000cbc0 comp rx=0 tx=0).stop 2023-10-30T12:33:45.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.269+0000 7f46d04d6640 1 -- 172.21.15.89:0/3182093392 shutdown_connections 2023-10-30T12:33:45.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.269+0000 7f46d04d6640 1 --2- 172.21.15.89:0/3182093392 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f46ac0664a0 0x7f46ac068960 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:45.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.269+0000 7f46d04d6640 1 --2- 172.21.15.89:0/3182093392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46c80c8430 0x7f46c8133640 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:45.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.269+0000 7f46d04d6640 1 --2- 172.21.15.89:0/3182093392 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f46c80c6f60 0x7f46c8133100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:45.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.269+0000 7f46d04d6640 1 --2- 172.21.15.89:0/3182093392 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46c80c6b80 0x7f46c813aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:45.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.269+0000 7f46d04d6640 1 -- 172.21.15.89:0/3182093392 >> 172.21.15.89:0/3182093392 conn(0x7f46c80c7910 msgr2=0x7f46c80bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:45.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.269+0000 7f46d04d6640 1 -- 172.21.15.89:0/3182093392 shutdown_connections 2023-10-30T12:33:45.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:33:45.269+0000 7f46d04d6640 1 -- 172.21.15.89:0/3182093392 wait complete. 2023-10-30T12:33:45.292 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T12:33:45.293 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 55 secs before thrashing 2023-10-30T12:33:45.577 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:33:45.582 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:34:03.308 INFO:tasks.workunit.client.0.smithi089.stdout:Delete all the files and directories ... 2023-10-30T12:34:09.809 INFO:tasks.workunit.client.0.smithi089.stdout:removed './100/19' 2023-10-30T12:34:09.810 INFO:tasks.workunit.client.0.smithi089.stdout:removed './100/15' 2023-10-30T12:34:09.810 INFO:tasks.workunit.client.0.smithi089.stdout:removed './100/31' 2023-10-30T12:34:09.810 INFO:tasks.workunit.client.0.smithi089.stdout:removed './100/36' 2023-10-30T12:34:09.811 INFO:tasks.workunit.client.0.smithi089.stdout:removed './100/20' 2023-10-30T12:34:09.811 INFO:tasks.workunit.client.0.smithi089.stdout:removed './100/12' 2023-10-30T12:34:09.811 INFO:tasks.workunit.client.0.smithi089.stdout:removed './100/39' 2023-10-30T12:34:09.812 INFO:tasks.workunit.client.0.smithi089.stdout:removed './100/10' 2023-10-30T12:34:09.812 INFO:tasks.workunit.client.0.smithi089.stdout:removed './100/34' 2023-10-30T12:34:09.813 INFO:tasks.workunit.client.0.smithi089.stdout:removed './100/38' 2023-10-30T12:34:09.813 INFO:tasks.workunit.client.0.smithi089.stdout:removed './100/11' 2023-10-30T12:34:09.813 INFO:tasks.workunit.client.0.smithi089.stdout:removed './100/16' 2023-10-30T12:34:09.814 INFO:tasks.workunit.client.0.smithi089.stdout:removed './100/32' 2023-10-30T12:34:09.814 INFO:tasks.workunit.client.0.smithi089.stdout:removed './100/30' 2023-10-30T12:34:09.814 INFO:tasks.workunit.client.0.smithi089.stdout:removed './100/14' 2023-10-30T12:34:09.815 INFO:tasks.workunit.client.0.smithi089.stdout:removed './100/18' 2023-10-30T12:34:09.815 INFO:tasks.workunit.client.0.smithi089.stdout:removed './100/13' 2023-10-30T12:34:09.816 INFO:tasks.workunit.client.0.smithi089.stdout:removed './100/33' 2023-10-30T12:34:09.816 INFO:tasks.workunit.client.0.smithi089.stdout:removed './100/17' 2023-10-30T12:34:09.816 INFO:tasks.workunit.client.0.smithi089.stdout:removed './100/35' 2023-10-30T12:34:09.817 INFO:tasks.workunit.client.0.smithi089.stdout:removed './100/37' 2023-10-30T12:34:09.817 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './100' 2023-10-30T12:34:09.817 INFO:tasks.workunit.client.0.smithi089.stdout:removed './101/19' 2023-10-30T12:34:09.818 INFO:tasks.workunit.client.0.smithi089.stdout:removed './101/15' 2023-10-30T12:34:09.818 INFO:tasks.workunit.client.0.smithi089.stdout:removed './101/31' 2023-10-30T12:34:09.818 INFO:tasks.workunit.client.0.smithi089.stdout:removed './101/36' 2023-10-30T12:34:09.819 INFO:tasks.workunit.client.0.smithi089.stdout:removed './101/20' 2023-10-30T12:34:09.819 INFO:tasks.workunit.client.0.smithi089.stdout:removed './101/12' 2023-10-30T12:34:09.820 INFO:tasks.workunit.client.0.smithi089.stdout:removed './101/39' 2023-10-30T12:34:09.820 INFO:tasks.workunit.client.0.smithi089.stdout:removed './101/10' 2023-10-30T12:34:09.820 INFO:tasks.workunit.client.0.smithi089.stdout:removed './101/34' 2023-10-30T12:34:09.821 INFO:tasks.workunit.client.0.smithi089.stdout:removed './101/38' 2023-10-30T12:34:09.821 INFO:tasks.workunit.client.0.smithi089.stdout:removed './101/11' 2023-10-30T12:34:09.822 INFO:tasks.workunit.client.0.smithi089.stdout:removed './101/16' 2023-10-30T12:34:09.822 INFO:tasks.workunit.client.0.smithi089.stdout:removed './101/32' 2023-10-30T12:34:09.822 INFO:tasks.workunit.client.0.smithi089.stdout:removed './101/30' 2023-10-30T12:34:09.823 INFO:tasks.workunit.client.0.smithi089.stdout:removed './101/14' 2023-10-30T12:34:09.823 INFO:tasks.workunit.client.0.smithi089.stdout:removed './101/18' 2023-10-30T12:34:09.823 INFO:tasks.workunit.client.0.smithi089.stdout:removed './101/13' 2023-10-30T12:34:09.824 INFO:tasks.workunit.client.0.smithi089.stdout:removed './101/33' 2023-10-30T12:34:09.824 INFO:tasks.workunit.client.0.smithi089.stdout:removed './101/17' 2023-10-30T12:34:09.825 INFO:tasks.workunit.client.0.smithi089.stdout:removed './101/35' 2023-10-30T12:34:09.825 INFO:tasks.workunit.client.0.smithi089.stdout:removed './101/37' 2023-10-30T12:34:09.825 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './101' 2023-10-30T12:34:09.826 INFO:tasks.workunit.client.0.smithi089.stdout:removed './102/19' 2023-10-30T12:34:09.826 INFO:tasks.workunit.client.0.smithi089.stdout:removed './102/15' 2023-10-30T12:34:09.826 INFO:tasks.workunit.client.0.smithi089.stdout:removed './102/31' 2023-10-30T12:34:09.827 INFO:tasks.workunit.client.0.smithi089.stdout:removed './102/36' 2023-10-30T12:34:09.827 INFO:tasks.workunit.client.0.smithi089.stdout:removed './102/20' 2023-10-30T12:34:09.828 INFO:tasks.workunit.client.0.smithi089.stdout:removed './102/12' 2023-10-30T12:34:09.828 INFO:tasks.workunit.client.0.smithi089.stdout:removed './102/39' 2023-10-30T12:34:09.828 INFO:tasks.workunit.client.0.smithi089.stdout:removed './102/10' 2023-10-30T12:34:09.829 INFO:tasks.workunit.client.0.smithi089.stdout:removed './102/34' 2023-10-30T12:34:09.829 INFO:tasks.workunit.client.0.smithi089.stdout:removed './102/38' 2023-10-30T12:34:09.829 INFO:tasks.workunit.client.0.smithi089.stdout:removed './102/11' 2023-10-30T12:34:09.830 INFO:tasks.workunit.client.0.smithi089.stdout:removed './102/16' 2023-10-30T12:34:09.830 INFO:tasks.workunit.client.0.smithi089.stdout:removed './102/32' 2023-10-30T12:34:09.831 INFO:tasks.workunit.client.0.smithi089.stdout:removed './102/30' 2023-10-30T12:34:09.831 INFO:tasks.workunit.client.0.smithi089.stdout:removed './102/14' 2023-10-30T12:34:09.831 INFO:tasks.workunit.client.0.smithi089.stdout:removed './102/18' 2023-10-30T12:34:09.832 INFO:tasks.workunit.client.0.smithi089.stdout:removed './102/13' 2023-10-30T12:34:09.832 INFO:tasks.workunit.client.0.smithi089.stdout:removed './102/33' 2023-10-30T12:34:09.832 INFO:tasks.workunit.client.0.smithi089.stdout:removed './102/17' 2023-10-30T12:34:09.833 INFO:tasks.workunit.client.0.smithi089.stdout:removed './102/35' 2023-10-30T12:34:09.833 INFO:tasks.workunit.client.0.smithi089.stdout:removed './102/37' 2023-10-30T12:34:09.834 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './102' 2023-10-30T12:34:09.834 INFO:tasks.workunit.client.0.smithi089.stdout:removed './103/19' 2023-10-30T12:34:09.834 INFO:tasks.workunit.client.0.smithi089.stdout:removed './103/15' 2023-10-30T12:34:09.835 INFO:tasks.workunit.client.0.smithi089.stdout:removed './103/31' 2023-10-30T12:34:09.835 INFO:tasks.workunit.client.0.smithi089.stdout:removed './103/36' 2023-10-30T12:34:09.836 INFO:tasks.workunit.client.0.smithi089.stdout:removed './103/20' 2023-10-30T12:34:09.836 INFO:tasks.workunit.client.0.smithi089.stdout:removed './103/12' 2023-10-30T12:34:09.836 INFO:tasks.workunit.client.0.smithi089.stdout:removed './103/39' 2023-10-30T12:34:09.837 INFO:tasks.workunit.client.0.smithi089.stdout:removed './103/10' 2023-10-30T12:34:09.837 INFO:tasks.workunit.client.0.smithi089.stdout:removed './103/34' 2023-10-30T12:34:09.837 INFO:tasks.workunit.client.0.smithi089.stdout:removed './103/38' 2023-10-30T12:34:09.837 INFO:tasks.workunit.client.0.smithi089.stdout:removed './103/11' 2023-10-30T12:34:09.838 INFO:tasks.workunit.client.0.smithi089.stdout:removed './103/16' 2023-10-30T12:34:09.838 INFO:tasks.workunit.client.0.smithi089.stdout:removed './103/32' 2023-10-30T12:34:09.838 INFO:tasks.workunit.client.0.smithi089.stdout:removed './103/30' 2023-10-30T12:34:09.839 INFO:tasks.workunit.client.0.smithi089.stdout:removed './103/14' 2023-10-30T12:34:09.839 INFO:tasks.workunit.client.0.smithi089.stdout:removed './103/18' 2023-10-30T12:34:09.839 INFO:tasks.workunit.client.0.smithi089.stdout:removed './103/13' 2023-10-30T12:34:09.840 INFO:tasks.workunit.client.0.smithi089.stdout:removed './103/33' 2023-10-30T12:34:09.840 INFO:tasks.workunit.client.0.smithi089.stdout:removed './103/17' 2023-10-30T12:34:09.840 INFO:tasks.workunit.client.0.smithi089.stdout:removed './103/35' 2023-10-30T12:34:09.840 INFO:tasks.workunit.client.0.smithi089.stdout:removed './103/37' 2023-10-30T12:34:09.841 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './103' 2023-10-30T12:34:09.841 INFO:tasks.workunit.client.0.smithi089.stdout:removed './104/19' 2023-10-30T12:34:09.841 INFO:tasks.workunit.client.0.smithi089.stdout:removed './104/15' 2023-10-30T12:34:09.841 INFO:tasks.workunit.client.0.smithi089.stdout:removed './104/31' 2023-10-30T12:34:09.842 INFO:tasks.workunit.client.0.smithi089.stdout:removed './104/36' 2023-10-30T12:34:09.842 INFO:tasks.workunit.client.0.smithi089.stdout:removed './104/20' 2023-10-30T12:34:09.842 INFO:tasks.workunit.client.0.smithi089.stdout:removed './104/12' 2023-10-30T12:34:09.843 INFO:tasks.workunit.client.0.smithi089.stdout:removed './104/39' 2023-10-30T12:34:09.843 INFO:tasks.workunit.client.0.smithi089.stdout:removed './104/10' 2023-10-30T12:34:09.843 INFO:tasks.workunit.client.0.smithi089.stdout:removed './104/34' 2023-10-30T12:34:09.843 INFO:tasks.workunit.client.0.smithi089.stdout:removed './104/38' 2023-10-30T12:34:09.844 INFO:tasks.workunit.client.0.smithi089.stdout:removed './104/11' 2023-10-30T12:34:09.844 INFO:tasks.workunit.client.0.smithi089.stdout:removed './104/16' 2023-10-30T12:34:09.844 INFO:tasks.workunit.client.0.smithi089.stdout:removed './104/32' 2023-10-30T12:34:09.844 INFO:tasks.workunit.client.0.smithi089.stdout:removed './104/30' 2023-10-30T12:34:09.845 INFO:tasks.workunit.client.0.smithi089.stdout:removed './104/14' 2023-10-30T12:34:09.845 INFO:tasks.workunit.client.0.smithi089.stdout:removed './104/18' 2023-10-30T12:34:09.845 INFO:tasks.workunit.client.0.smithi089.stdout:removed './104/13' 2023-10-30T12:34:09.846 INFO:tasks.workunit.client.0.smithi089.stdout:removed './104/33' 2023-10-30T12:34:09.846 INFO:tasks.workunit.client.0.smithi089.stdout:removed './104/17' 2023-10-30T12:34:09.846 INFO:tasks.workunit.client.0.smithi089.stdout:removed './104/35' 2023-10-30T12:34:09.846 INFO:tasks.workunit.client.0.smithi089.stdout:removed './104/37' 2023-10-30T12:34:09.847 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './104' 2023-10-30T12:34:09.847 INFO:tasks.workunit.client.0.smithi089.stdout:removed './105/19' 2023-10-30T12:34:09.847 INFO:tasks.workunit.client.0.smithi089.stdout:removed './105/15' 2023-10-30T12:34:09.847 INFO:tasks.workunit.client.0.smithi089.stdout:removed './105/31' 2023-10-30T12:34:09.848 INFO:tasks.workunit.client.0.smithi089.stdout:removed './105/36' 2023-10-30T12:34:09.848 INFO:tasks.workunit.client.0.smithi089.stdout:removed './105/20' 2023-10-30T12:34:09.848 INFO:tasks.workunit.client.0.smithi089.stdout:removed './105/12' 2023-10-30T12:34:09.848 INFO:tasks.workunit.client.0.smithi089.stdout:removed './105/39' 2023-10-30T12:34:09.848 INFO:tasks.workunit.client.0.smithi089.stdout:removed './105/10' 2023-10-30T12:34:09.849 INFO:tasks.workunit.client.0.smithi089.stdout:removed './105/34' 2023-10-30T12:34:09.849 INFO:tasks.workunit.client.0.smithi089.stdout:removed './105/38' 2023-10-30T12:34:09.849 INFO:tasks.workunit.client.0.smithi089.stdout:removed './105/11' 2023-10-30T12:34:09.849 INFO:tasks.workunit.client.0.smithi089.stdout:removed './105/16' 2023-10-30T12:34:09.849 INFO:tasks.workunit.client.0.smithi089.stdout:removed './105/32' 2023-10-30T12:34:09.850 INFO:tasks.workunit.client.0.smithi089.stdout:removed './105/30' 2023-10-30T12:34:09.850 INFO:tasks.workunit.client.0.smithi089.stdout:removed './105/14' 2023-10-30T12:34:09.850 INFO:tasks.workunit.client.0.smithi089.stdout:removed './105/18' 2023-10-30T12:34:09.850 INFO:tasks.workunit.client.0.smithi089.stdout:removed './105/13' 2023-10-30T12:34:09.850 INFO:tasks.workunit.client.0.smithi089.stdout:removed './105/33' 2023-10-30T12:34:09.850 INFO:tasks.workunit.client.0.smithi089.stdout:removed './105/17' 2023-10-30T12:34:09.851 INFO:tasks.workunit.client.0.smithi089.stdout:removed './105/35' 2023-10-30T12:34:09.851 INFO:tasks.workunit.client.0.smithi089.stdout:removed './105/37' 2023-10-30T12:34:09.851 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './105' 2023-10-30T12:34:09.851 INFO:tasks.workunit.client.0.smithi089.stdout:removed './106/19' 2023-10-30T12:34:09.851 INFO:tasks.workunit.client.0.smithi089.stdout:removed './106/15' 2023-10-30T12:34:09.852 INFO:tasks.workunit.client.0.smithi089.stdout:removed './106/31' 2023-10-30T12:34:09.852 INFO:tasks.workunit.client.0.smithi089.stdout:removed './106/36' 2023-10-30T12:34:09.852 INFO:tasks.workunit.client.0.smithi089.stdout:removed './106/20' 2023-10-30T12:34:09.852 INFO:tasks.workunit.client.0.smithi089.stdout:removed './106/12' 2023-10-30T12:34:09.852 INFO:tasks.workunit.client.0.smithi089.stdout:removed './106/39' 2023-10-30T12:34:09.853 INFO:tasks.workunit.client.0.smithi089.stdout:removed './106/10' 2023-10-30T12:34:09.853 INFO:tasks.workunit.client.0.smithi089.stdout:removed './106/34' 2023-10-30T12:34:09.853 INFO:tasks.workunit.client.0.smithi089.stdout:removed './106/38' 2023-10-30T12:34:09.853 INFO:tasks.workunit.client.0.smithi089.stdout:removed './106/11' 2023-10-30T12:34:09.853 INFO:tasks.workunit.client.0.smithi089.stdout:removed './106/16' 2023-10-30T12:34:09.853 INFO:tasks.workunit.client.0.smithi089.stdout:removed './106/32' 2023-10-30T12:34:09.854 INFO:tasks.workunit.client.0.smithi089.stdout:removed './106/30' 2023-10-30T12:34:09.854 INFO:tasks.workunit.client.0.smithi089.stdout:removed './106/14' 2023-10-30T12:34:09.854 INFO:tasks.workunit.client.0.smithi089.stdout:removed './106/18' 2023-10-30T12:34:09.854 INFO:tasks.workunit.client.0.smithi089.stdout:removed './106/13' 2023-10-30T12:34:09.854 INFO:tasks.workunit.client.0.smithi089.stdout:removed './106/33' 2023-10-30T12:34:09.855 INFO:tasks.workunit.client.0.smithi089.stdout:removed './106/17' 2023-10-30T12:34:09.855 INFO:tasks.workunit.client.0.smithi089.stdout:removed './106/35' 2023-10-30T12:34:09.855 INFO:tasks.workunit.client.0.smithi089.stdout:removed './106/37' 2023-10-30T12:34:09.855 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './106' 2023-10-30T12:34:09.855 INFO:tasks.workunit.client.0.smithi089.stdout:removed './107/19' 2023-10-30T12:34:09.856 INFO:tasks.workunit.client.0.smithi089.stdout:removed './107/15' 2023-10-30T12:34:09.856 INFO:tasks.workunit.client.0.smithi089.stdout:removed './107/31' 2023-10-30T12:34:09.856 INFO:tasks.workunit.client.0.smithi089.stdout:removed './107/36' 2023-10-30T12:34:09.856 INFO:tasks.workunit.client.0.smithi089.stdout:removed './107/20' 2023-10-30T12:34:09.856 INFO:tasks.workunit.client.0.smithi089.stdout:removed './107/12' 2023-10-30T12:34:09.856 INFO:tasks.workunit.client.0.smithi089.stdout:removed './107/39' 2023-10-30T12:34:09.857 INFO:tasks.workunit.client.0.smithi089.stdout:removed './107/10' 2023-10-30T12:34:09.857 INFO:tasks.workunit.client.0.smithi089.stdout:removed './107/34' 2023-10-30T12:34:09.857 INFO:tasks.workunit.client.0.smithi089.stdout:removed './107/38' 2023-10-30T12:34:09.857 INFO:tasks.workunit.client.0.smithi089.stdout:removed './107/11' 2023-10-30T12:34:09.857 INFO:tasks.workunit.client.0.smithi089.stdout:removed './107/16' 2023-10-30T12:34:09.858 INFO:tasks.workunit.client.0.smithi089.stdout:removed './107/32' 2023-10-30T12:34:09.858 INFO:tasks.workunit.client.0.smithi089.stdout:removed './107/30' 2023-10-30T12:34:09.858 INFO:tasks.workunit.client.0.smithi089.stdout:removed './107/14' 2023-10-30T12:34:09.858 INFO:tasks.workunit.client.0.smithi089.stdout:removed './107/18' 2023-10-30T12:34:09.858 INFO:tasks.workunit.client.0.smithi089.stdout:removed './107/13' 2023-10-30T12:34:09.859 INFO:tasks.workunit.client.0.smithi089.stdout:removed './107/33' 2023-10-30T12:34:09.859 INFO:tasks.workunit.client.0.smithi089.stdout:removed './107/17' 2023-10-30T12:34:09.859 INFO:tasks.workunit.client.0.smithi089.stdout:removed './107/35' 2023-10-30T12:34:09.859 INFO:tasks.workunit.client.0.smithi089.stdout:removed './107/37' 2023-10-30T12:34:09.859 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './107' 2023-10-30T12:34:09.859 INFO:tasks.workunit.client.0.smithi089.stdout:removed './108/19' 2023-10-30T12:34:09.860 INFO:tasks.workunit.client.0.smithi089.stdout:removed './108/15' 2023-10-30T12:34:09.860 INFO:tasks.workunit.client.0.smithi089.stdout:removed './108/31' 2023-10-30T12:34:09.860 INFO:tasks.workunit.client.0.smithi089.stdout:removed './108/36' 2023-10-30T12:34:09.860 INFO:tasks.workunit.client.0.smithi089.stdout:removed './108/20' 2023-10-30T12:34:09.860 INFO:tasks.workunit.client.0.smithi089.stdout:removed './108/12' 2023-10-30T12:34:09.861 INFO:tasks.workunit.client.0.smithi089.stdout:removed './108/39' 2023-10-30T12:34:09.861 INFO:tasks.workunit.client.0.smithi089.stdout:removed './108/10' 2023-10-30T12:34:09.861 INFO:tasks.workunit.client.0.smithi089.stdout:removed './108/34' 2023-10-30T12:34:09.861 INFO:tasks.workunit.client.0.smithi089.stdout:removed './108/38' 2023-10-30T12:34:09.861 INFO:tasks.workunit.client.0.smithi089.stdout:removed './108/11' 2023-10-30T12:34:09.862 INFO:tasks.workunit.client.0.smithi089.stdout:removed './108/16' 2023-10-30T12:34:09.862 INFO:tasks.workunit.client.0.smithi089.stdout:removed './108/32' 2023-10-30T12:34:09.862 INFO:tasks.workunit.client.0.smithi089.stdout:removed './108/30' 2023-10-30T12:34:09.862 INFO:tasks.workunit.client.0.smithi089.stdout:removed './108/14' 2023-10-30T12:34:09.862 INFO:tasks.workunit.client.0.smithi089.stdout:removed './108/18' 2023-10-30T12:34:09.862 INFO:tasks.workunit.client.0.smithi089.stdout:removed './108/13' 2023-10-30T12:34:09.863 INFO:tasks.workunit.client.0.smithi089.stdout:removed './108/33' 2023-10-30T12:34:09.863 INFO:tasks.workunit.client.0.smithi089.stdout:removed './108/17' 2023-10-30T12:34:09.863 INFO:tasks.workunit.client.0.smithi089.stdout:removed './108/35' 2023-10-30T12:34:09.863 INFO:tasks.workunit.client.0.smithi089.stdout:removed './108/37' 2023-10-30T12:34:09.863 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './108' 2023-10-30T12:34:09.864 INFO:tasks.workunit.client.0.smithi089.stdout:removed './109/19' 2023-10-30T12:34:09.864 INFO:tasks.workunit.client.0.smithi089.stdout:removed './109/15' 2023-10-30T12:34:09.864 INFO:tasks.workunit.client.0.smithi089.stdout:removed './109/31' 2023-10-30T12:34:09.864 INFO:tasks.workunit.client.0.smithi089.stdout:removed './109/36' 2023-10-30T12:34:09.864 INFO:tasks.workunit.client.0.smithi089.stdout:removed './109/20' 2023-10-30T12:34:09.865 INFO:tasks.workunit.client.0.smithi089.stdout:removed './109/12' 2023-10-30T12:34:09.865 INFO:tasks.workunit.client.0.smithi089.stdout:removed './109/39' 2023-10-30T12:34:09.865 INFO:tasks.workunit.client.0.smithi089.stdout:removed './109/10' 2023-10-30T12:34:09.865 INFO:tasks.workunit.client.0.smithi089.stdout:removed './109/34' 2023-10-30T12:34:09.865 INFO:tasks.workunit.client.0.smithi089.stdout:removed './109/38' 2023-10-30T12:34:09.865 INFO:tasks.workunit.client.0.smithi089.stdout:removed './109/11' 2023-10-30T12:34:09.866 INFO:tasks.workunit.client.0.smithi089.stdout:removed './109/16' 2023-10-30T12:34:09.866 INFO:tasks.workunit.client.0.smithi089.stdout:removed './109/32' 2023-10-30T12:34:09.866 INFO:tasks.workunit.client.0.smithi089.stdout:removed './109/30' 2023-10-30T12:34:14.169 INFO:tasks.workunit.client.0.smithi089.stdout:removed './109/14' 2023-10-30T12:34:14.169 INFO:tasks.workunit.client.0.smithi089.stdout:removed './109/18' 2023-10-30T12:34:14.170 INFO:tasks.workunit.client.0.smithi089.stdout:removed './109/13' 2023-10-30T12:34:14.170 INFO:tasks.workunit.client.0.smithi089.stdout:removed './109/33' 2023-10-30T12:34:14.170 INFO:tasks.workunit.client.0.smithi089.stdout:removed './109/17' 2023-10-30T12:34:14.170 INFO:tasks.workunit.client.0.smithi089.stdout:removed './109/35' 2023-10-30T12:34:14.171 INFO:tasks.workunit.client.0.smithi089.stdout:removed './109/37' 2023-10-30T12:34:14.171 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './109' 2023-10-30T12:34:14.171 INFO:tasks.workunit.client.0.smithi089.stdout:removed './110/19' 2023-10-30T12:34:14.171 INFO:tasks.workunit.client.0.smithi089.stdout:removed './110/15' 2023-10-30T12:34:14.171 INFO:tasks.workunit.client.0.smithi089.stdout:removed './110/31' 2023-10-30T12:34:14.172 INFO:tasks.workunit.client.0.smithi089.stdout:removed './110/36' 2023-10-30T12:34:14.172 INFO:tasks.workunit.client.0.smithi089.stdout:removed './110/20' 2023-10-30T12:34:14.172 INFO:tasks.workunit.client.0.smithi089.stdout:removed './110/12' 2023-10-30T12:34:14.172 INFO:tasks.workunit.client.0.smithi089.stdout:removed './110/39' 2023-10-30T12:34:14.172 INFO:tasks.workunit.client.0.smithi089.stdout:removed './110/10' 2023-10-30T12:34:14.173 INFO:tasks.workunit.client.0.smithi089.stdout:removed './110/34' 2023-10-30T12:34:14.173 INFO:tasks.workunit.client.0.smithi089.stdout:removed './110/38' 2023-10-30T12:34:14.173 INFO:tasks.workunit.client.0.smithi089.stdout:removed './110/11' 2023-10-30T12:34:14.173 INFO:tasks.workunit.client.0.smithi089.stdout:removed './110/16' 2023-10-30T12:34:14.173 INFO:tasks.workunit.client.0.smithi089.stdout:removed './110/32' 2023-10-30T12:34:14.174 INFO:tasks.workunit.client.0.smithi089.stdout:removed './110/30' 2023-10-30T12:34:14.174 INFO:tasks.workunit.client.0.smithi089.stdout:removed './110/14' 2023-10-30T12:34:14.174 INFO:tasks.workunit.client.0.smithi089.stdout:removed './110/18' 2023-10-30T12:34:14.174 INFO:tasks.workunit.client.0.smithi089.stdout:removed './110/13' 2023-10-30T12:34:14.174 INFO:tasks.workunit.client.0.smithi089.stdout:removed './110/33' 2023-10-30T12:34:14.175 INFO:tasks.workunit.client.0.smithi089.stdout:removed './110/17' 2023-10-30T12:34:14.175 INFO:tasks.workunit.client.0.smithi089.stdout:removed './110/35' 2023-10-30T12:34:14.175 INFO:tasks.workunit.client.0.smithi089.stdout:removed './110/37' 2023-10-30T12:34:14.175 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './110' 2023-10-30T12:34:14.175 INFO:tasks.workunit.client.0.smithi089.stdout:removed './111/19' 2023-10-30T12:34:14.176 INFO:tasks.workunit.client.0.smithi089.stdout:removed './111/15' 2023-10-30T12:34:14.176 INFO:tasks.workunit.client.0.smithi089.stdout:removed './111/31' 2023-10-30T12:34:14.176 INFO:tasks.workunit.client.0.smithi089.stdout:removed './111/36' 2023-10-30T12:34:14.176 INFO:tasks.workunit.client.0.smithi089.stdout:removed './111/20' 2023-10-30T12:34:14.176 INFO:tasks.workunit.client.0.smithi089.stdout:removed './111/12' 2023-10-30T12:34:14.177 INFO:tasks.workunit.client.0.smithi089.stdout:removed './111/39' 2023-10-30T12:34:14.177 INFO:tasks.workunit.client.0.smithi089.stdout:removed './111/10' 2023-10-30T12:34:14.177 INFO:tasks.workunit.client.0.smithi089.stdout:removed './111/34' 2023-10-30T12:34:14.178 INFO:tasks.workunit.client.0.smithi089.stdout:removed './111/38' 2023-10-30T12:34:14.178 INFO:tasks.workunit.client.0.smithi089.stdout:removed './111/11' 2023-10-30T12:34:14.178 INFO:tasks.workunit.client.0.smithi089.stdout:removed './111/16' 2023-10-30T12:34:14.178 INFO:tasks.workunit.client.0.smithi089.stdout:removed './111/32' 2023-10-30T12:34:14.178 INFO:tasks.workunit.client.0.smithi089.stdout:removed './111/30' 2023-10-30T12:34:14.179 INFO:tasks.workunit.client.0.smithi089.stdout:removed './111/14' 2023-10-30T12:34:14.179 INFO:tasks.workunit.client.0.smithi089.stdout:removed './111/18' 2023-10-30T12:34:14.179 INFO:tasks.workunit.client.0.smithi089.stdout:removed './111/13' 2023-10-30T12:34:14.179 INFO:tasks.workunit.client.0.smithi089.stdout:removed './111/33' 2023-10-30T12:34:14.179 INFO:tasks.workunit.client.0.smithi089.stdout:removed './111/17' 2023-10-30T12:34:14.180 INFO:tasks.workunit.client.0.smithi089.stdout:removed './111/35' 2023-10-30T12:34:14.180 INFO:tasks.workunit.client.0.smithi089.stdout:removed './111/37' 2023-10-30T12:34:14.180 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './111' 2023-10-30T12:34:14.180 INFO:tasks.workunit.client.0.smithi089.stdout:removed './112/19' 2023-10-30T12:34:14.180 INFO:tasks.workunit.client.0.smithi089.stdout:removed './112/15' 2023-10-30T12:34:14.181 INFO:tasks.workunit.client.0.smithi089.stdout:removed './112/31' 2023-10-30T12:34:14.181 INFO:tasks.workunit.client.0.smithi089.stdout:removed './112/36' 2023-10-30T12:34:14.181 INFO:tasks.workunit.client.0.smithi089.stdout:removed './112/20' 2023-10-30T12:34:14.181 INFO:tasks.workunit.client.0.smithi089.stdout:removed './112/12' 2023-10-30T12:34:14.181 INFO:tasks.workunit.client.0.smithi089.stdout:removed './112/39' 2023-10-30T12:34:14.182 INFO:tasks.workunit.client.0.smithi089.stdout:removed './112/10' 2023-10-30T12:34:14.182 INFO:tasks.workunit.client.0.smithi089.stdout:removed './112/34' 2023-10-30T12:34:14.182 INFO:tasks.workunit.client.0.smithi089.stdout:removed './112/38' 2023-10-30T12:34:14.182 INFO:tasks.workunit.client.0.smithi089.stdout:removed './112/11' 2023-10-30T12:34:14.182 INFO:tasks.workunit.client.0.smithi089.stdout:removed './112/16' 2023-10-30T12:34:14.182 INFO:tasks.workunit.client.0.smithi089.stdout:removed './112/32' 2023-10-30T12:34:14.183 INFO:tasks.workunit.client.0.smithi089.stdout:removed './112/30' 2023-10-30T12:34:14.183 INFO:tasks.workunit.client.0.smithi089.stdout:removed './112/14' 2023-10-30T12:34:14.183 INFO:tasks.workunit.client.0.smithi089.stdout:removed './112/18' 2023-10-30T12:34:14.183 INFO:tasks.workunit.client.0.smithi089.stdout:removed './112/13' 2023-10-30T12:34:14.183 INFO:tasks.workunit.client.0.smithi089.stdout:removed './112/33' 2023-10-30T12:34:14.184 INFO:tasks.workunit.client.0.smithi089.stdout:removed './112/17' 2023-10-30T12:34:14.184 INFO:tasks.workunit.client.0.smithi089.stdout:removed './112/35' 2023-10-30T12:34:14.184 INFO:tasks.workunit.client.0.smithi089.stdout:removed './112/37' 2023-10-30T12:34:14.184 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './112' 2023-10-30T12:34:14.184 INFO:tasks.workunit.client.0.smithi089.stdout:removed './113/19' 2023-10-30T12:34:14.185 INFO:tasks.workunit.client.0.smithi089.stdout:removed './113/15' 2023-10-30T12:34:14.185 INFO:tasks.workunit.client.0.smithi089.stdout:removed './113/31' 2023-10-30T12:34:14.185 INFO:tasks.workunit.client.0.smithi089.stdout:removed './113/36' 2023-10-30T12:34:14.185 INFO:tasks.workunit.client.0.smithi089.stdout:removed './113/20' 2023-10-30T12:34:14.185 INFO:tasks.workunit.client.0.smithi089.stdout:removed './113/12' 2023-10-30T12:34:14.186 INFO:tasks.workunit.client.0.smithi089.stdout:removed './113/39' 2023-10-30T12:34:14.186 INFO:tasks.workunit.client.0.smithi089.stdout:removed './113/10' 2023-10-30T12:34:14.186 INFO:tasks.workunit.client.0.smithi089.stdout:removed './113/34' 2023-10-30T12:34:14.186 INFO:tasks.workunit.client.0.smithi089.stdout:removed './113/38' 2023-10-30T12:34:14.186 INFO:tasks.workunit.client.0.smithi089.stdout:removed './113/11' 2023-10-30T12:34:14.187 INFO:tasks.workunit.client.0.smithi089.stdout:removed './113/16' 2023-10-30T12:34:14.187 INFO:tasks.workunit.client.0.smithi089.stdout:removed './113/32' 2023-10-30T12:34:14.187 INFO:tasks.workunit.client.0.smithi089.stdout:removed './113/30' 2023-10-30T12:34:14.187 INFO:tasks.workunit.client.0.smithi089.stdout:removed './113/14' 2023-10-30T12:34:14.187 INFO:tasks.workunit.client.0.smithi089.stdout:removed './113/18' 2023-10-30T12:34:14.187 INFO:tasks.workunit.client.0.smithi089.stdout:removed './113/13' 2023-10-30T12:34:14.188 INFO:tasks.workunit.client.0.smithi089.stdout:removed './113/33' 2023-10-30T12:34:14.188 INFO:tasks.workunit.client.0.smithi089.stdout:removed './113/17' 2023-10-30T12:34:14.188 INFO:tasks.workunit.client.0.smithi089.stdout:removed './113/35' 2023-10-30T12:34:14.188 INFO:tasks.workunit.client.0.smithi089.stdout:removed './113/37' 2023-10-30T12:34:14.188 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './113' 2023-10-30T12:34:14.189 INFO:tasks.workunit.client.0.smithi089.stdout:removed './114/19' 2023-10-30T12:34:14.189 INFO:tasks.workunit.client.0.smithi089.stdout:removed './114/15' 2023-10-30T12:34:14.189 INFO:tasks.workunit.client.0.smithi089.stdout:removed './114/31' 2023-10-30T12:34:14.189 INFO:tasks.workunit.client.0.smithi089.stdout:removed './114/36' 2023-10-30T12:34:14.189 INFO:tasks.workunit.client.0.smithi089.stdout:removed './114/20' 2023-10-30T12:34:14.190 INFO:tasks.workunit.client.0.smithi089.stdout:removed './114/12' 2023-10-30T12:34:14.190 INFO:tasks.workunit.client.0.smithi089.stdout:removed './114/39' 2023-10-30T12:34:14.190 INFO:tasks.workunit.client.0.smithi089.stdout:removed './114/10' 2023-10-30T12:34:14.190 INFO:tasks.workunit.client.0.smithi089.stdout:removed './114/34' 2023-10-30T12:34:14.190 INFO:tasks.workunit.client.0.smithi089.stdout:removed './114/38' 2023-10-30T12:34:14.190 INFO:tasks.workunit.client.0.smithi089.stdout:removed './114/11' 2023-10-30T12:34:14.191 INFO:tasks.workunit.client.0.smithi089.stdout:removed './114/16' 2023-10-30T12:34:14.191 INFO:tasks.workunit.client.0.smithi089.stdout:removed './114/32' 2023-10-30T12:34:14.191 INFO:tasks.workunit.client.0.smithi089.stdout:removed './114/30' 2023-10-30T12:34:14.191 INFO:tasks.workunit.client.0.smithi089.stdout:removed './114/14' 2023-10-30T12:34:14.191 INFO:tasks.workunit.client.0.smithi089.stdout:removed './114/18' 2023-10-30T12:34:14.192 INFO:tasks.workunit.client.0.smithi089.stdout:removed './114/13' 2023-10-30T12:34:14.192 INFO:tasks.workunit.client.0.smithi089.stdout:removed './114/33' 2023-10-30T12:34:14.192 INFO:tasks.workunit.client.0.smithi089.stdout:removed './114/17' 2023-10-30T12:34:14.192 INFO:tasks.workunit.client.0.smithi089.stdout:removed './114/35' 2023-10-30T12:34:14.192 INFO:tasks.workunit.client.0.smithi089.stdout:removed './114/37' 2023-10-30T12:34:14.193 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './114' 2023-10-30T12:34:14.193 INFO:tasks.workunit.client.0.smithi089.stdout:removed './115/19' 2023-10-30T12:34:14.193 INFO:tasks.workunit.client.0.smithi089.stdout:removed './115/15' 2023-10-30T12:34:14.193 INFO:tasks.workunit.client.0.smithi089.stdout:removed './115/31' 2023-10-30T12:34:14.193 INFO:tasks.workunit.client.0.smithi089.stdout:removed './115/36' 2023-10-30T12:34:14.193 INFO:tasks.workunit.client.0.smithi089.stdout:removed './115/20' 2023-10-30T12:34:14.194 INFO:tasks.workunit.client.0.smithi089.stdout:removed './115/12' 2023-10-30T12:34:14.194 INFO:tasks.workunit.client.0.smithi089.stdout:removed './115/39' 2023-10-30T12:34:14.194 INFO:tasks.workunit.client.0.smithi089.stdout:removed './115/10' 2023-10-30T12:34:14.194 INFO:tasks.workunit.client.0.smithi089.stdout:removed './115/34' 2023-10-30T12:34:14.194 INFO:tasks.workunit.client.0.smithi089.stdout:removed './115/38' 2023-10-30T12:34:14.195 INFO:tasks.workunit.client.0.smithi089.stdout:removed './115/11' 2023-10-30T12:34:14.195 INFO:tasks.workunit.client.0.smithi089.stdout:removed './115/16' 2023-10-30T12:34:14.195 INFO:tasks.workunit.client.0.smithi089.stdout:removed './115/32' 2023-10-30T12:34:14.195 INFO:tasks.workunit.client.0.smithi089.stdout:removed './115/30' 2023-10-30T12:34:14.195 INFO:tasks.workunit.client.0.smithi089.stdout:removed './115/14' 2023-10-30T12:34:14.195 INFO:tasks.workunit.client.0.smithi089.stdout:removed './115/18' 2023-10-30T12:34:14.196 INFO:tasks.workunit.client.0.smithi089.stdout:removed './115/13' 2023-10-30T12:34:14.196 INFO:tasks.workunit.client.0.smithi089.stdout:removed './115/33' 2023-10-30T12:34:14.196 INFO:tasks.workunit.client.0.smithi089.stdout:removed './115/17' 2023-10-30T12:34:14.196 INFO:tasks.workunit.client.0.smithi089.stdout:removed './115/35' 2023-10-30T12:34:14.196 INFO:tasks.workunit.client.0.smithi089.stdout:removed './115/37' 2023-10-30T12:34:14.197 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './115' 2023-10-30T12:34:14.197 INFO:tasks.workunit.client.0.smithi089.stdout:removed './116/19' 2023-10-30T12:34:14.197 INFO:tasks.workunit.client.0.smithi089.stdout:removed './116/15' 2023-10-30T12:34:14.197 INFO:tasks.workunit.client.0.smithi089.stdout:removed './116/31' 2023-10-30T12:34:14.197 INFO:tasks.workunit.client.0.smithi089.stdout:removed './116/36' 2023-10-30T12:34:14.197 INFO:tasks.workunit.client.0.smithi089.stdout:removed './116/20' 2023-10-30T12:34:14.198 INFO:tasks.workunit.client.0.smithi089.stdout:removed './116/12' 2023-10-30T12:34:14.198 INFO:tasks.workunit.client.0.smithi089.stdout:removed './116/39' 2023-10-30T12:34:14.198 INFO:tasks.workunit.client.0.smithi089.stdout:removed './116/10' 2023-10-30T12:34:14.198 INFO:tasks.workunit.client.0.smithi089.stdout:removed './116/34' 2023-10-30T12:34:14.198 INFO:tasks.workunit.client.0.smithi089.stdout:removed './116/38' 2023-10-30T12:34:14.199 INFO:tasks.workunit.client.0.smithi089.stdout:removed './116/11' 2023-10-30T12:34:14.199 INFO:tasks.workunit.client.0.smithi089.stdout:removed './116/16' 2023-10-30T12:34:14.199 INFO:tasks.workunit.client.0.smithi089.stdout:removed './116/32' 2023-10-30T12:34:14.199 INFO:tasks.workunit.client.0.smithi089.stdout:removed './116/30' 2023-10-30T12:34:14.199 INFO:tasks.workunit.client.0.smithi089.stdout:removed './116/14' 2023-10-30T12:34:14.200 INFO:tasks.workunit.client.0.smithi089.stdout:removed './116/18' 2023-10-30T12:34:14.200 INFO:tasks.workunit.client.0.smithi089.stdout:removed './116/13' 2023-10-30T12:34:14.200 INFO:tasks.workunit.client.0.smithi089.stdout:removed './116/33' 2023-10-30T12:34:14.200 INFO:tasks.workunit.client.0.smithi089.stdout:removed './116/17' 2023-10-30T12:34:14.200 INFO:tasks.workunit.client.0.smithi089.stdout:removed './116/35' 2023-10-30T12:34:14.201 INFO:tasks.workunit.client.0.smithi089.stdout:removed './116/37' 2023-10-30T12:34:14.201 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './116' 2023-10-30T12:34:14.201 INFO:tasks.workunit.client.0.smithi089.stdout:removed './117/19' 2023-10-30T12:34:14.201 INFO:tasks.workunit.client.0.smithi089.stdout:removed './117/15' 2023-10-30T12:34:14.201 INFO:tasks.workunit.client.0.smithi089.stdout:removed './117/31' 2023-10-30T12:34:14.201 INFO:tasks.workunit.client.0.smithi089.stdout:removed './117/36' 2023-10-30T12:34:14.202 INFO:tasks.workunit.client.0.smithi089.stdout:removed './117/20' 2023-10-30T12:34:14.202 INFO:tasks.workunit.client.0.smithi089.stdout:removed './117/12' 2023-10-30T12:34:14.202 INFO:tasks.workunit.client.0.smithi089.stdout:removed './117/39' 2023-10-30T12:34:14.202 INFO:tasks.workunit.client.0.smithi089.stdout:removed './117/10' 2023-10-30T12:34:14.202 INFO:tasks.workunit.client.0.smithi089.stdout:removed './117/34' 2023-10-30T12:34:14.203 INFO:tasks.workunit.client.0.smithi089.stdout:removed './117/38' 2023-10-30T12:34:14.203 INFO:tasks.workunit.client.0.smithi089.stdout:removed './117/11' 2023-10-30T12:34:14.203 INFO:tasks.workunit.client.0.smithi089.stdout:removed './117/16' 2023-10-30T12:34:14.203 INFO:tasks.workunit.client.0.smithi089.stdout:removed './117/32' 2023-10-30T12:34:14.203 INFO:tasks.workunit.client.0.smithi089.stdout:removed './117/30' 2023-10-30T12:34:14.204 INFO:tasks.workunit.client.0.smithi089.stdout:removed './117/14' 2023-10-30T12:34:14.204 INFO:tasks.workunit.client.0.smithi089.stdout:removed './117/18' 2023-10-30T12:34:14.204 INFO:tasks.workunit.client.0.smithi089.stdout:removed './117/13' 2023-10-30T12:34:14.204 INFO:tasks.workunit.client.0.smithi089.stdout:removed './117/33' 2023-10-30T12:34:14.204 INFO:tasks.workunit.client.0.smithi089.stdout:removed './117/17' 2023-10-30T12:34:14.204 INFO:tasks.workunit.client.0.smithi089.stdout:removed './117/35' 2023-10-30T12:34:14.205 INFO:tasks.workunit.client.0.smithi089.stdout:removed './117/37' 2023-10-30T12:34:14.205 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './117' 2023-10-30T12:34:14.205 INFO:tasks.workunit.client.0.smithi089.stdout:removed './118/19' 2023-10-30T12:34:14.205 INFO:tasks.workunit.client.0.smithi089.stdout:removed './118/15' 2023-10-30T12:34:14.205 INFO:tasks.workunit.client.0.smithi089.stdout:removed './118/31' 2023-10-30T12:34:14.206 INFO:tasks.workunit.client.0.smithi089.stdout:removed './118/36' 2023-10-30T12:34:14.206 INFO:tasks.workunit.client.0.smithi089.stdout:removed './118/20' 2023-10-30T12:34:14.206 INFO:tasks.workunit.client.0.smithi089.stdout:removed './118/12' 2023-10-30T12:34:14.206 INFO:tasks.workunit.client.0.smithi089.stdout:removed './118/39' 2023-10-30T12:34:14.206 INFO:tasks.workunit.client.0.smithi089.stdout:removed './118/10' 2023-10-30T12:34:14.207 INFO:tasks.workunit.client.0.smithi089.stdout:removed './118/34' 2023-10-30T12:34:14.207 INFO:tasks.workunit.client.0.smithi089.stdout:removed './118/38' 2023-10-30T12:34:14.207 INFO:tasks.workunit.client.0.smithi089.stdout:removed './118/11' 2023-10-30T12:34:14.207 INFO:tasks.workunit.client.0.smithi089.stdout:removed './118/16' 2023-10-30T12:34:14.207 INFO:tasks.workunit.client.0.smithi089.stdout:removed './118/32' 2023-10-30T12:34:14.209 INFO:tasks.workunit.client.0.smithi089.stdout:removed './118/30' 2023-10-30T12:34:14.210 INFO:tasks.workunit.client.0.smithi089.stdout:removed './118/14' 2023-10-30T12:34:14.210 INFO:tasks.workunit.client.0.smithi089.stdout:removed './118/18' 2023-10-30T12:34:14.210 INFO:tasks.workunit.client.0.smithi089.stdout:removed './118/13' 2023-10-30T12:34:14.211 INFO:tasks.workunit.client.0.smithi089.stdout:removed './118/33' 2023-10-30T12:34:14.211 INFO:tasks.workunit.client.0.smithi089.stdout:removed './118/17' 2023-10-30T12:34:14.211 INFO:tasks.workunit.client.0.smithi089.stdout:removed './118/35' 2023-10-30T12:34:14.211 INFO:tasks.workunit.client.0.smithi089.stdout:removed './118/37' 2023-10-30T12:34:14.212 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './118' 2023-10-30T12:34:14.212 INFO:tasks.workunit.client.0.smithi089.stdout:removed './119/19' 2023-10-30T12:34:14.212 INFO:tasks.workunit.client.0.smithi089.stdout:removed './119/15' 2023-10-30T12:34:14.212 INFO:tasks.workunit.client.0.smithi089.stdout:removed './119/31' 2023-10-30T12:34:14.213 INFO:tasks.workunit.client.0.smithi089.stdout:removed './119/36' 2023-10-30T12:34:14.213 INFO:tasks.workunit.client.0.smithi089.stdout:removed './119/20' 2023-10-30T12:34:14.213 INFO:tasks.workunit.client.0.smithi089.stdout:removed './119/12' 2023-10-30T12:34:15.652 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:34:15.657 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:34:23.102 INFO:tasks.workunit.client.0.smithi089.stdout:removed './119/39' 2023-10-30T12:34:23.103 INFO:tasks.workunit.client.0.smithi089.stdout:removed './119/10' 2023-10-30T12:34:23.103 INFO:tasks.workunit.client.0.smithi089.stdout:removed './119/34' 2023-10-30T12:34:23.103 INFO:tasks.workunit.client.0.smithi089.stdout:removed './119/38' 2023-10-30T12:34:23.104 INFO:tasks.workunit.client.0.smithi089.stdout:removed './119/11' 2023-10-30T12:34:23.104 INFO:tasks.workunit.client.0.smithi089.stdout:removed './119/16' 2023-10-30T12:34:23.104 INFO:tasks.workunit.client.0.smithi089.stdout:removed './119/32' 2023-10-30T12:34:23.104 INFO:tasks.workunit.client.0.smithi089.stdout:removed './119/30' 2023-10-30T12:34:23.104 INFO:tasks.workunit.client.0.smithi089.stdout:removed './119/14' 2023-10-30T12:34:23.105 INFO:tasks.workunit.client.0.smithi089.stdout:removed './119/18' 2023-10-30T12:34:23.105 INFO:tasks.workunit.client.0.smithi089.stdout:removed './119/13' 2023-10-30T12:34:23.105 INFO:tasks.workunit.client.0.smithi089.stdout:removed './119/33' 2023-10-30T12:34:23.105 INFO:tasks.workunit.client.0.smithi089.stdout:removed './119/17' 2023-10-30T12:34:23.105 INFO:tasks.workunit.client.0.smithi089.stdout:removed './119/35' 2023-10-30T12:34:23.105 INFO:tasks.workunit.client.0.smithi089.stdout:removed './119/37' 2023-10-30T12:34:23.106 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './119' 2023-10-30T12:34:23.106 INFO:tasks.workunit.client.0.smithi089.stdout:removed './120/19' 2023-10-30T12:34:23.106 INFO:tasks.workunit.client.0.smithi089.stdout:removed './120/15' 2023-10-30T12:34:23.106 INFO:tasks.workunit.client.0.smithi089.stdout:removed './120/31' 2023-10-30T12:34:23.107 INFO:tasks.workunit.client.0.smithi089.stdout:removed './120/36' 2023-10-30T12:34:23.107 INFO:tasks.workunit.client.0.smithi089.stdout:removed './120/20' 2023-10-30T12:34:23.107 INFO:tasks.workunit.client.0.smithi089.stdout:removed './120/12' 2023-10-30T12:34:23.107 INFO:tasks.workunit.client.0.smithi089.stdout:removed './120/39' 2023-10-30T12:34:23.107 INFO:tasks.workunit.client.0.smithi089.stdout:removed './120/10' 2023-10-30T12:34:23.107 INFO:tasks.workunit.client.0.smithi089.stdout:removed './120/34' 2023-10-30T12:34:23.108 INFO:tasks.workunit.client.0.smithi089.stdout:removed './120/38' 2023-10-30T12:34:23.108 INFO:tasks.workunit.client.0.smithi089.stdout:removed './120/11' 2023-10-30T12:34:23.108 INFO:tasks.workunit.client.0.smithi089.stdout:removed './120/16' 2023-10-30T12:34:23.108 INFO:tasks.workunit.client.0.smithi089.stdout:removed './120/32' 2023-10-30T12:34:23.108 INFO:tasks.workunit.client.0.smithi089.stdout:removed './120/30' 2023-10-30T12:34:23.109 INFO:tasks.workunit.client.0.smithi089.stdout:removed './120/14' 2023-10-30T12:34:23.109 INFO:tasks.workunit.client.0.smithi089.stdout:removed './120/18' 2023-10-30T12:34:23.109 INFO:tasks.workunit.client.0.smithi089.stdout:removed './120/13' 2023-10-30T12:34:23.109 INFO:tasks.workunit.client.0.smithi089.stdout:removed './120/33' 2023-10-30T12:34:23.110 INFO:tasks.workunit.client.0.smithi089.stdout:removed './120/17' 2023-10-30T12:34:23.110 INFO:tasks.workunit.client.0.smithi089.stdout:removed './120/35' 2023-10-30T12:34:23.110 INFO:tasks.workunit.client.0.smithi089.stdout:removed './120/37' 2023-10-30T12:34:23.110 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './120' 2023-10-30T12:34:23.110 INFO:tasks.workunit.client.0.smithi089.stdout:removed './121/19' 2023-10-30T12:34:23.111 INFO:tasks.workunit.client.0.smithi089.stdout:removed './121/15' 2023-10-30T12:34:23.111 INFO:tasks.workunit.client.0.smithi089.stdout:removed './121/31' 2023-10-30T12:34:23.111 INFO:tasks.workunit.client.0.smithi089.stdout:removed './121/36' 2023-10-30T12:34:23.111 INFO:tasks.workunit.client.0.smithi089.stdout:removed './121/20' 2023-10-30T12:34:23.111 INFO:tasks.workunit.client.0.smithi089.stdout:removed './121/12' 2023-10-30T12:34:23.112 INFO:tasks.workunit.client.0.smithi089.stdout:removed './121/39' 2023-10-30T12:34:23.112 INFO:tasks.workunit.client.0.smithi089.stdout:removed './121/10' 2023-10-30T12:34:23.112 INFO:tasks.workunit.client.0.smithi089.stdout:removed './121/34' 2023-10-30T12:34:23.112 INFO:tasks.workunit.client.0.smithi089.stdout:removed './121/38' 2023-10-30T12:34:23.112 INFO:tasks.workunit.client.0.smithi089.stdout:removed './121/11' 2023-10-30T12:34:23.113 INFO:tasks.workunit.client.0.smithi089.stdout:removed './121/16' 2023-10-30T12:34:23.113 INFO:tasks.workunit.client.0.smithi089.stdout:removed './121/32' 2023-10-30T12:34:23.113 INFO:tasks.workunit.client.0.smithi089.stdout:removed './121/30' 2023-10-30T12:34:23.113 INFO:tasks.workunit.client.0.smithi089.stdout:removed './121/14' 2023-10-30T12:34:23.113 INFO:tasks.workunit.client.0.smithi089.stdout:removed './121/18' 2023-10-30T12:34:23.114 INFO:tasks.workunit.client.0.smithi089.stdout:removed './121/13' 2023-10-30T12:34:23.114 INFO:tasks.workunit.client.0.smithi089.stdout:removed './121/33' 2023-10-30T12:34:23.114 INFO:tasks.workunit.client.0.smithi089.stdout:removed './121/17' 2023-10-30T12:34:23.114 INFO:tasks.workunit.client.0.smithi089.stdout:removed './121/35' 2023-10-30T12:34:23.114 INFO:tasks.workunit.client.0.smithi089.stdout:removed './121/37' 2023-10-30T12:34:23.115 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './121' 2023-10-30T12:34:23.115 INFO:tasks.workunit.client.0.smithi089.stdout:removed './122/19' 2023-10-30T12:34:23.115 INFO:tasks.workunit.client.0.smithi089.stdout:removed './122/15' 2023-10-30T12:34:23.115 INFO:tasks.workunit.client.0.smithi089.stdout:removed './122/31' 2023-10-30T12:34:23.115 INFO:tasks.workunit.client.0.smithi089.stdout:removed './122/36' 2023-10-30T12:34:23.115 INFO:tasks.workunit.client.0.smithi089.stdout:removed './122/20' 2023-10-30T12:34:23.116 INFO:tasks.workunit.client.0.smithi089.stdout:removed './122/12' 2023-10-30T12:34:23.116 INFO:tasks.workunit.client.0.smithi089.stdout:removed './122/39' 2023-10-30T12:34:23.116 INFO:tasks.workunit.client.0.smithi089.stdout:removed './122/10' 2023-10-30T12:34:23.116 INFO:tasks.workunit.client.0.smithi089.stdout:removed './122/34' 2023-10-30T12:34:23.116 INFO:tasks.workunit.client.0.smithi089.stdout:removed './122/38' 2023-10-30T12:34:23.117 INFO:tasks.workunit.client.0.smithi089.stdout:removed './122/11' 2023-10-30T12:34:23.117 INFO:tasks.workunit.client.0.smithi089.stdout:removed './122/16' 2023-10-30T12:34:23.117 INFO:tasks.workunit.client.0.smithi089.stdout:removed './122/32' 2023-10-30T12:34:23.117 INFO:tasks.workunit.client.0.smithi089.stdout:removed './122/30' 2023-10-30T12:34:23.117 INFO:tasks.workunit.client.0.smithi089.stdout:removed './122/14' 2023-10-30T12:34:23.118 INFO:tasks.workunit.client.0.smithi089.stdout:removed './122/18' 2023-10-30T12:34:23.118 INFO:tasks.workunit.client.0.smithi089.stdout:removed './122/13' 2023-10-30T12:34:23.118 INFO:tasks.workunit.client.0.smithi089.stdout:removed './122/33' 2023-10-30T12:34:23.118 INFO:tasks.workunit.client.0.smithi089.stdout:removed './122/17' 2023-10-30T12:34:23.119 INFO:tasks.workunit.client.0.smithi089.stdout:removed './122/35' 2023-10-30T12:34:23.119 INFO:tasks.workunit.client.0.smithi089.stdout:removed './122/37' 2023-10-30T12:34:23.119 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './122' 2023-10-30T12:34:23.119 INFO:tasks.workunit.client.0.smithi089.stdout:removed './123/19' 2023-10-30T12:34:23.119 INFO:tasks.workunit.client.0.smithi089.stdout:removed './123/15' 2023-10-30T12:34:23.119 INFO:tasks.workunit.client.0.smithi089.stdout:removed './123/31' 2023-10-30T12:34:23.120 INFO:tasks.workunit.client.0.smithi089.stdout:removed './123/36' 2023-10-30T12:34:23.120 INFO:tasks.workunit.client.0.smithi089.stdout:removed './123/20' 2023-10-30T12:34:23.120 INFO:tasks.workunit.client.0.smithi089.stdout:removed './123/12' 2023-10-30T12:34:23.120 INFO:tasks.workunit.client.0.smithi089.stdout:removed './123/39' 2023-10-30T12:34:23.120 INFO:tasks.workunit.client.0.smithi089.stdout:removed './123/10' 2023-10-30T12:34:23.121 INFO:tasks.workunit.client.0.smithi089.stdout:removed './123/34' 2023-10-30T12:34:23.121 INFO:tasks.workunit.client.0.smithi089.stdout:removed './123/38' 2023-10-30T12:34:23.121 INFO:tasks.workunit.client.0.smithi089.stdout:removed './123/11' 2023-10-30T12:34:23.121 INFO:tasks.workunit.client.0.smithi089.stdout:removed './123/16' 2023-10-30T12:34:23.121 INFO:tasks.workunit.client.0.smithi089.stdout:removed './123/32' 2023-10-30T12:34:23.122 INFO:tasks.workunit.client.0.smithi089.stdout:removed './123/30' 2023-10-30T12:34:23.122 INFO:tasks.workunit.client.0.smithi089.stdout:removed './123/14' 2023-10-30T12:34:23.122 INFO:tasks.workunit.client.0.smithi089.stdout:removed './123/18' 2023-10-30T12:34:23.122 INFO:tasks.workunit.client.0.smithi089.stdout:removed './123/13' 2023-10-30T12:34:23.122 INFO:tasks.workunit.client.0.smithi089.stdout:removed './123/33' 2023-10-30T12:34:23.122 INFO:tasks.workunit.client.0.smithi089.stdout:removed './123/17' 2023-10-30T12:34:23.123 INFO:tasks.workunit.client.0.smithi089.stdout:removed './123/35' 2023-10-30T12:34:23.123 INFO:tasks.workunit.client.0.smithi089.stdout:removed './123/37' 2023-10-30T12:34:23.123 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './123' 2023-10-30T12:34:23.123 INFO:tasks.workunit.client.0.smithi089.stdout:removed './124/19' 2023-10-30T12:34:23.123 INFO:tasks.workunit.client.0.smithi089.stdout:removed './124/15' 2023-10-30T12:34:23.124 INFO:tasks.workunit.client.0.smithi089.stdout:removed './124/31' 2023-10-30T12:34:23.124 INFO:tasks.workunit.client.0.smithi089.stdout:removed './124/36' 2023-10-30T12:34:23.124 INFO:tasks.workunit.client.0.smithi089.stdout:removed './124/20' 2023-10-30T12:34:23.124 INFO:tasks.workunit.client.0.smithi089.stdout:removed './124/12' 2023-10-30T12:34:23.124 INFO:tasks.workunit.client.0.smithi089.stdout:removed './124/39' 2023-10-30T12:34:23.125 INFO:tasks.workunit.client.0.smithi089.stdout:removed './124/10' 2023-10-30T12:34:23.125 INFO:tasks.workunit.client.0.smithi089.stdout:removed './124/34' 2023-10-30T12:34:23.125 INFO:tasks.workunit.client.0.smithi089.stdout:removed './124/38' 2023-10-30T12:34:23.125 INFO:tasks.workunit.client.0.smithi089.stdout:removed './124/11' 2023-10-30T12:34:23.125 INFO:tasks.workunit.client.0.smithi089.stdout:removed './124/16' 2023-10-30T12:34:23.125 INFO:tasks.workunit.client.0.smithi089.stdout:removed './124/32' 2023-10-30T12:34:23.126 INFO:tasks.workunit.client.0.smithi089.stdout:removed './124/30' 2023-10-30T12:34:23.126 INFO:tasks.workunit.client.0.smithi089.stdout:removed './124/14' 2023-10-30T12:34:23.126 INFO:tasks.workunit.client.0.smithi089.stdout:removed './124/18' 2023-10-30T12:34:23.126 INFO:tasks.workunit.client.0.smithi089.stdout:removed './124/13' 2023-10-30T12:34:23.126 INFO:tasks.workunit.client.0.smithi089.stdout:removed './124/33' 2023-10-30T12:34:23.127 INFO:tasks.workunit.client.0.smithi089.stdout:removed './124/17' 2023-10-30T12:34:23.127 INFO:tasks.workunit.client.0.smithi089.stdout:removed './124/35' 2023-10-30T12:34:23.127 INFO:tasks.workunit.client.0.smithi089.stdout:removed './124/37' 2023-10-30T12:34:23.127 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './124' 2023-10-30T12:34:23.127 INFO:tasks.workunit.client.0.smithi089.stdout:removed './125/19' 2023-10-30T12:34:23.128 INFO:tasks.workunit.client.0.smithi089.stdout:removed './125/15' 2023-10-30T12:34:23.128 INFO:tasks.workunit.client.0.smithi089.stdout:removed './125/31' 2023-10-30T12:34:23.128 INFO:tasks.workunit.client.0.smithi089.stdout:removed './125/36' 2023-10-30T12:34:23.128 INFO:tasks.workunit.client.0.smithi089.stdout:removed './125/20' 2023-10-30T12:34:23.128 INFO:tasks.workunit.client.0.smithi089.stdout:removed './125/12' 2023-10-30T12:34:23.129 INFO:tasks.workunit.client.0.smithi089.stdout:removed './125/39' 2023-10-30T12:34:23.129 INFO:tasks.workunit.client.0.smithi089.stdout:removed './125/10' 2023-10-30T12:34:23.129 INFO:tasks.workunit.client.0.smithi089.stdout:removed './125/34' 2023-10-30T12:34:23.129 INFO:tasks.workunit.client.0.smithi089.stdout:removed './125/38' 2023-10-30T12:34:23.129 INFO:tasks.workunit.client.0.smithi089.stdout:removed './125/11' 2023-10-30T12:34:23.129 INFO:tasks.workunit.client.0.smithi089.stdout:removed './125/16' 2023-10-30T12:34:23.130 INFO:tasks.workunit.client.0.smithi089.stdout:removed './125/32' 2023-10-30T12:34:23.130 INFO:tasks.workunit.client.0.smithi089.stdout:removed './125/30' 2023-10-30T12:34:23.130 INFO:tasks.workunit.client.0.smithi089.stdout:removed './125/14' 2023-10-30T12:34:23.130 INFO:tasks.workunit.client.0.smithi089.stdout:removed './125/18' 2023-10-30T12:34:23.130 INFO:tasks.workunit.client.0.smithi089.stdout:removed './125/13' 2023-10-30T12:34:23.131 INFO:tasks.workunit.client.0.smithi089.stdout:removed './125/33' 2023-10-30T12:34:23.131 INFO:tasks.workunit.client.0.smithi089.stdout:removed './125/17' 2023-10-30T12:34:23.131 INFO:tasks.workunit.client.0.smithi089.stdout:removed './125/35' 2023-10-30T12:34:23.131 INFO:tasks.workunit.client.0.smithi089.stdout:removed './125/37' 2023-10-30T12:34:23.131 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './125' 2023-10-30T12:34:23.132 INFO:tasks.workunit.client.0.smithi089.stdout:removed './126/19' 2023-10-30T12:34:23.132 INFO:tasks.workunit.client.0.smithi089.stdout:removed './126/15' 2023-10-30T12:34:23.132 INFO:tasks.workunit.client.0.smithi089.stdout:removed './126/31' 2023-10-30T12:34:23.132 INFO:tasks.workunit.client.0.smithi089.stdout:removed './126/36' 2023-10-30T12:34:23.132 INFO:tasks.workunit.client.0.smithi089.stdout:removed './126/20' 2023-10-30T12:34:23.132 INFO:tasks.workunit.client.0.smithi089.stdout:removed './126/12' 2023-10-30T12:34:23.133 INFO:tasks.workunit.client.0.smithi089.stdout:removed './126/39' 2023-10-30T12:34:23.133 INFO:tasks.workunit.client.0.smithi089.stdout:removed './126/10' 2023-10-30T12:34:23.133 INFO:tasks.workunit.client.0.smithi089.stdout:removed './126/34' 2023-10-30T12:34:23.133 INFO:tasks.workunit.client.0.smithi089.stdout:removed './126/38' 2023-10-30T12:34:23.133 INFO:tasks.workunit.client.0.smithi089.stdout:removed './126/11' 2023-10-30T12:34:23.134 INFO:tasks.workunit.client.0.smithi089.stdout:removed './126/16' 2023-10-30T12:34:23.134 INFO:tasks.workunit.client.0.smithi089.stdout:removed './126/32' 2023-10-30T12:34:23.134 INFO:tasks.workunit.client.0.smithi089.stdout:removed './126/30' 2023-10-30T12:34:23.134 INFO:tasks.workunit.client.0.smithi089.stdout:removed './126/14' 2023-10-30T12:34:23.134 INFO:tasks.workunit.client.0.smithi089.stdout:removed './126/18' 2023-10-30T12:34:23.135 INFO:tasks.workunit.client.0.smithi089.stdout:removed './126/13' 2023-10-30T12:34:23.135 INFO:tasks.workunit.client.0.smithi089.stdout:removed './126/33' 2023-10-30T12:34:23.135 INFO:tasks.workunit.client.0.smithi089.stdout:removed './126/17' 2023-10-30T12:34:23.135 INFO:tasks.workunit.client.0.smithi089.stdout:removed './126/35' 2023-10-30T12:34:23.135 INFO:tasks.workunit.client.0.smithi089.stdout:removed './126/37' 2023-10-30T12:34:23.136 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './126' 2023-10-30T12:34:23.136 INFO:tasks.workunit.client.0.smithi089.stdout:removed './127/19' 2023-10-30T12:34:23.136 INFO:tasks.workunit.client.0.smithi089.stdout:removed './127/15' 2023-10-30T12:34:23.136 INFO:tasks.workunit.client.0.smithi089.stdout:removed './127/31' 2023-10-30T12:34:23.136 INFO:tasks.workunit.client.0.smithi089.stdout:removed './127/36' 2023-10-30T12:34:23.136 INFO:tasks.workunit.client.0.smithi089.stdout:removed './127/20' 2023-10-30T12:34:23.137 INFO:tasks.workunit.client.0.smithi089.stdout:removed './127/12' 2023-10-30T12:34:23.137 INFO:tasks.workunit.client.0.smithi089.stdout:removed './127/39' 2023-10-30T12:34:23.137 INFO:tasks.workunit.client.0.smithi089.stdout:removed './127/10' 2023-10-30T12:34:23.137 INFO:tasks.workunit.client.0.smithi089.stdout:removed './127/34' 2023-10-30T12:34:23.137 INFO:tasks.workunit.client.0.smithi089.stdout:removed './127/38' 2023-10-30T12:34:23.138 INFO:tasks.workunit.client.0.smithi089.stdout:removed './127/11' 2023-10-30T12:34:23.138 INFO:tasks.workunit.client.0.smithi089.stdout:removed './127/16' 2023-10-30T12:34:23.138 INFO:tasks.workunit.client.0.smithi089.stdout:removed './127/32' 2023-10-30T12:34:23.138 INFO:tasks.workunit.client.0.smithi089.stdout:removed './127/30' 2023-10-30T12:34:23.138 INFO:tasks.workunit.client.0.smithi089.stdout:removed './127/14' 2023-10-30T12:34:23.139 INFO:tasks.workunit.client.0.smithi089.stdout:removed './127/18' 2023-10-30T12:34:23.139 INFO:tasks.workunit.client.0.smithi089.stdout:removed './127/13' 2023-10-30T12:34:23.139 INFO:tasks.workunit.client.0.smithi089.stdout:removed './127/33' 2023-10-30T12:34:23.139 INFO:tasks.workunit.client.0.smithi089.stdout:removed './127/17' 2023-10-30T12:34:23.139 INFO:tasks.workunit.client.0.smithi089.stdout:removed './127/35' 2023-10-30T12:34:23.140 INFO:tasks.workunit.client.0.smithi089.stdout:removed './127/37' 2023-10-30T12:34:23.140 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './127' 2023-10-30T12:34:23.140 INFO:tasks.workunit.client.0.smithi089.stdout:removed './128/19' 2023-10-30T12:34:23.140 INFO:tasks.workunit.client.0.smithi089.stdout:removed './128/15' 2023-10-30T12:34:23.140 INFO:tasks.workunit.client.0.smithi089.stdout:removed './128/31' 2023-10-30T12:34:23.140 INFO:tasks.workunit.client.0.smithi089.stdout:removed './128/36' 2023-10-30T12:34:23.141 INFO:tasks.workunit.client.0.smithi089.stdout:removed './128/20' 2023-10-30T12:34:23.141 INFO:tasks.workunit.client.0.smithi089.stdout:removed './128/12' 2023-10-30T12:34:23.141 INFO:tasks.workunit.client.0.smithi089.stdout:removed './128/39' 2023-10-30T12:34:23.141 INFO:tasks.workunit.client.0.smithi089.stdout:removed './128/10' 2023-10-30T12:34:23.141 INFO:tasks.workunit.client.0.smithi089.stdout:removed './128/34' 2023-10-30T12:34:23.142 INFO:tasks.workunit.client.0.smithi089.stdout:removed './128/38' 2023-10-30T12:34:23.142 INFO:tasks.workunit.client.0.smithi089.stdout:removed './128/11' 2023-10-30T12:34:23.142 INFO:tasks.workunit.client.0.smithi089.stdout:removed './128/16' 2023-10-30T12:34:23.142 INFO:tasks.workunit.client.0.smithi089.stdout:removed './128/32' 2023-10-30T12:34:23.142 INFO:tasks.workunit.client.0.smithi089.stdout:removed './128/30' 2023-10-30T12:34:23.142 INFO:tasks.workunit.client.0.smithi089.stdout:removed './128/14' 2023-10-30T12:34:23.143 INFO:tasks.workunit.client.0.smithi089.stdout:removed './128/18' 2023-10-30T12:34:23.143 INFO:tasks.workunit.client.0.smithi089.stdout:removed './128/13' 2023-10-30T12:34:23.143 INFO:tasks.workunit.client.0.smithi089.stdout:removed './128/33' 2023-10-30T12:34:23.143 INFO:tasks.workunit.client.0.smithi089.stdout:removed './128/17' 2023-10-30T12:34:23.143 INFO:tasks.workunit.client.0.smithi089.stdout:removed './128/35' 2023-10-30T12:34:26.982 INFO:tasks.workunit.client.0.smithi089.stdout:removed './128/37' 2023-10-30T12:34:26.983 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './128' 2023-10-30T12:34:26.983 INFO:tasks.workunit.client.0.smithi089.stdout:removed './129/19' 2023-10-30T12:34:26.983 INFO:tasks.workunit.client.0.smithi089.stdout:removed './129/15' 2023-10-30T12:34:26.984 INFO:tasks.workunit.client.0.smithi089.stdout:removed './129/31' 2023-10-30T12:34:26.984 INFO:tasks.workunit.client.0.smithi089.stdout:removed './129/36' 2023-10-30T12:34:26.984 INFO:tasks.workunit.client.0.smithi089.stdout:removed './129/20' 2023-10-30T12:34:26.984 INFO:tasks.workunit.client.0.smithi089.stdout:removed './129/12' 2023-10-30T12:34:26.984 INFO:tasks.workunit.client.0.smithi089.stdout:removed './129/39' 2023-10-30T12:34:26.985 INFO:tasks.workunit.client.0.smithi089.stdout:removed './129/10' 2023-10-30T12:34:26.985 INFO:tasks.workunit.client.0.smithi089.stdout:removed './129/34' 2023-10-30T12:34:26.985 INFO:tasks.workunit.client.0.smithi089.stdout:removed './129/38' 2023-10-30T12:34:26.985 INFO:tasks.workunit.client.0.smithi089.stdout:removed './129/11' 2023-10-30T12:34:26.985 INFO:tasks.workunit.client.0.smithi089.stdout:removed './129/16' 2023-10-30T12:34:26.986 INFO:tasks.workunit.client.0.smithi089.stdout:removed './129/32' 2023-10-30T12:34:26.986 INFO:tasks.workunit.client.0.smithi089.stdout:removed './129/30' 2023-10-30T12:34:26.986 INFO:tasks.workunit.client.0.smithi089.stdout:removed './129/14' 2023-10-30T12:34:26.986 INFO:tasks.workunit.client.0.smithi089.stdout:removed './129/18' 2023-10-30T12:34:26.987 INFO:tasks.workunit.client.0.smithi089.stdout:removed './129/13' 2023-10-30T12:34:26.987 INFO:tasks.workunit.client.0.smithi089.stdout:removed './129/33' 2023-10-30T12:34:26.987 INFO:tasks.workunit.client.0.smithi089.stdout:removed './129/17' 2023-10-30T12:34:26.987 INFO:tasks.workunit.client.0.smithi089.stdout:removed './129/35' 2023-10-30T12:34:26.988 INFO:tasks.workunit.client.0.smithi089.stdout:removed './129/37' 2023-10-30T12:34:26.988 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './129' 2023-10-30T12:34:26.988 INFO:tasks.workunit.client.0.smithi089.stdout:removed './130/19' 2023-10-30T12:34:26.989 INFO:tasks.workunit.client.0.smithi089.stdout:removed './130/15' 2023-10-30T12:34:26.989 INFO:tasks.workunit.client.0.smithi089.stdout:removed './130/31' 2023-10-30T12:34:26.989 INFO:tasks.workunit.client.0.smithi089.stdout:removed './130/36' 2023-10-30T12:34:26.989 INFO:tasks.workunit.client.0.smithi089.stdout:removed './130/20' 2023-10-30T12:34:26.989 INFO:tasks.workunit.client.0.smithi089.stdout:removed './130/12' 2023-10-30T12:34:26.990 INFO:tasks.workunit.client.0.smithi089.stdout:removed './130/39' 2023-10-30T12:34:26.990 INFO:tasks.workunit.client.0.smithi089.stdout:removed './130/10' 2023-10-30T12:34:26.990 INFO:tasks.workunit.client.0.smithi089.stdout:removed './130/34' 2023-10-30T12:34:26.990 INFO:tasks.workunit.client.0.smithi089.stdout:removed './130/38' 2023-10-30T12:34:26.991 INFO:tasks.workunit.client.0.smithi089.stdout:removed './130/11' 2023-10-30T12:34:26.991 INFO:tasks.workunit.client.0.smithi089.stdout:removed './130/16' 2023-10-30T12:34:26.991 INFO:tasks.workunit.client.0.smithi089.stdout:removed './130/32' 2023-10-30T12:34:26.991 INFO:tasks.workunit.client.0.smithi089.stdout:removed './130/30' 2023-10-30T12:34:26.991 INFO:tasks.workunit.client.0.smithi089.stdout:removed './130/14' 2023-10-30T12:34:26.992 INFO:tasks.workunit.client.0.smithi089.stdout:removed './130/18' 2023-10-30T12:34:26.992 INFO:tasks.workunit.client.0.smithi089.stdout:removed './130/13' 2023-10-30T12:34:26.992 INFO:tasks.workunit.client.0.smithi089.stdout:removed './130/33' 2023-10-30T12:34:26.992 INFO:tasks.workunit.client.0.smithi089.stdout:removed './130/17' 2023-10-30T12:34:26.992 INFO:tasks.workunit.client.0.smithi089.stdout:removed './130/35' 2023-10-30T12:34:26.993 INFO:tasks.workunit.client.0.smithi089.stdout:removed './130/37' 2023-10-30T12:34:26.993 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './130' 2023-10-30T12:34:26.993 INFO:tasks.workunit.client.0.smithi089.stdout:removed './131/19' 2023-10-30T12:34:26.993 INFO:tasks.workunit.client.0.smithi089.stdout:removed './131/15' 2023-10-30T12:34:26.993 INFO:tasks.workunit.client.0.smithi089.stdout:removed './131/31' 2023-10-30T12:34:26.993 INFO:tasks.workunit.client.0.smithi089.stdout:removed './131/36' 2023-10-30T12:34:26.994 INFO:tasks.workunit.client.0.smithi089.stdout:removed './131/20' 2023-10-30T12:34:26.994 INFO:tasks.workunit.client.0.smithi089.stdout:removed './131/12' 2023-10-30T12:34:26.994 INFO:tasks.workunit.client.0.smithi089.stdout:removed './131/39' 2023-10-30T12:34:26.994 INFO:tasks.workunit.client.0.smithi089.stdout:removed './131/10' 2023-10-30T12:34:26.995 INFO:tasks.workunit.client.0.smithi089.stdout:removed './131/34' 2023-10-30T12:34:26.995 INFO:tasks.workunit.client.0.smithi089.stdout:removed './131/38' 2023-10-30T12:34:26.995 INFO:tasks.workunit.client.0.smithi089.stdout:removed './131/11' 2023-10-30T12:34:26.995 INFO:tasks.workunit.client.0.smithi089.stdout:removed './131/16' 2023-10-30T12:34:26.995 INFO:tasks.workunit.client.0.smithi089.stdout:removed './131/32' 2023-10-30T12:34:26.996 INFO:tasks.workunit.client.0.smithi089.stdout:removed './131/30' 2023-10-30T12:34:26.996 INFO:tasks.workunit.client.0.smithi089.stdout:removed './131/14' 2023-10-30T12:34:26.996 INFO:tasks.workunit.client.0.smithi089.stdout:removed './131/18' 2023-10-30T12:34:26.996 INFO:tasks.workunit.client.0.smithi089.stdout:removed './131/13' 2023-10-30T12:34:26.996 INFO:tasks.workunit.client.0.smithi089.stdout:removed './131/33' 2023-10-30T12:34:26.997 INFO:tasks.workunit.client.0.smithi089.stdout:removed './131/17' 2023-10-30T12:34:26.997 INFO:tasks.workunit.client.0.smithi089.stdout:removed './131/35' 2023-10-30T12:34:26.997 INFO:tasks.workunit.client.0.smithi089.stdout:removed './131/37' 2023-10-30T12:34:26.997 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './131' 2023-10-30T12:34:26.997 INFO:tasks.workunit.client.0.smithi089.stdout:removed './132/19' 2023-10-30T12:34:26.998 INFO:tasks.workunit.client.0.smithi089.stdout:removed './132/15' 2023-10-30T12:34:26.998 INFO:tasks.workunit.client.0.smithi089.stdout:removed './132/31' 2023-10-30T12:34:26.998 INFO:tasks.workunit.client.0.smithi089.stdout:removed './132/36' 2023-10-30T12:34:26.998 INFO:tasks.workunit.client.0.smithi089.stdout:removed './132/20' 2023-10-30T12:34:26.998 INFO:tasks.workunit.client.0.smithi089.stdout:removed './132/12' 2023-10-30T12:34:26.999 INFO:tasks.workunit.client.0.smithi089.stdout:removed './132/39' 2023-10-30T12:34:26.999 INFO:tasks.workunit.client.0.smithi089.stdout:removed './132/10' 2023-10-30T12:34:26.999 INFO:tasks.workunit.client.0.smithi089.stdout:removed './132/34' 2023-10-30T12:34:26.999 INFO:tasks.workunit.client.0.smithi089.stdout:removed './132/38' 2023-10-30T12:34:26.999 INFO:tasks.workunit.client.0.smithi089.stdout:removed './132/11' 2023-10-30T12:34:27.000 INFO:tasks.workunit.client.0.smithi089.stdout:removed './132/16' 2023-10-30T12:34:27.000 INFO:tasks.workunit.client.0.smithi089.stdout:removed './132/32' 2023-10-30T12:34:27.000 INFO:tasks.workunit.client.0.smithi089.stdout:removed './132/30' 2023-10-30T12:34:27.000 INFO:tasks.workunit.client.0.smithi089.stdout:removed './132/14' 2023-10-30T12:34:27.000 INFO:tasks.workunit.client.0.smithi089.stdout:removed './132/18' 2023-10-30T12:34:27.001 INFO:tasks.workunit.client.0.smithi089.stdout:removed './132/13' 2023-10-30T12:34:27.001 INFO:tasks.workunit.client.0.smithi089.stdout:removed './132/33' 2023-10-30T12:34:27.001 INFO:tasks.workunit.client.0.smithi089.stdout:removed './132/17' 2023-10-30T12:34:27.001 INFO:tasks.workunit.client.0.smithi089.stdout:removed './132/35' 2023-10-30T12:34:27.001 INFO:tasks.workunit.client.0.smithi089.stdout:removed './132/37' 2023-10-30T12:34:27.001 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './132' 2023-10-30T12:34:27.002 INFO:tasks.workunit.client.0.smithi089.stdout:removed './133/19' 2023-10-30T12:34:27.002 INFO:tasks.workunit.client.0.smithi089.stdout:removed './133/15' 2023-10-30T12:34:27.002 INFO:tasks.workunit.client.0.smithi089.stdout:removed './133/31' 2023-10-30T12:34:27.002 INFO:tasks.workunit.client.0.smithi089.stdout:removed './133/36' 2023-10-30T12:34:27.002 INFO:tasks.workunit.client.0.smithi089.stdout:removed './133/20' 2023-10-30T12:34:27.003 INFO:tasks.workunit.client.0.smithi089.stdout:removed './133/12' 2023-10-30T12:34:27.003 INFO:tasks.workunit.client.0.smithi089.stdout:removed './133/39' 2023-10-30T12:34:27.003 INFO:tasks.workunit.client.0.smithi089.stdout:removed './133/10' 2023-10-30T12:34:27.003 INFO:tasks.workunit.client.0.smithi089.stdout:removed './133/34' 2023-10-30T12:34:27.003 INFO:tasks.workunit.client.0.smithi089.stdout:removed './133/38' 2023-10-30T12:34:27.004 INFO:tasks.workunit.client.0.smithi089.stdout:removed './133/11' 2023-10-30T12:34:27.004 INFO:tasks.workunit.client.0.smithi089.stdout:removed './133/16' 2023-10-30T12:34:27.004 INFO:tasks.workunit.client.0.smithi089.stdout:removed './133/32' 2023-10-30T12:34:27.004 INFO:tasks.workunit.client.0.smithi089.stdout:removed './133/30' 2023-10-30T12:34:27.004 INFO:tasks.workunit.client.0.smithi089.stdout:removed './133/14' 2023-10-30T12:34:27.005 INFO:tasks.workunit.client.0.smithi089.stdout:removed './133/18' 2023-10-30T12:34:27.005 INFO:tasks.workunit.client.0.smithi089.stdout:removed './133/13' 2023-10-30T12:34:27.005 INFO:tasks.workunit.client.0.smithi089.stdout:removed './133/33' 2023-10-30T12:34:27.005 INFO:tasks.workunit.client.0.smithi089.stdout:removed './133/17' 2023-10-30T12:34:27.005 INFO:tasks.workunit.client.0.smithi089.stdout:removed './133/35' 2023-10-30T12:34:27.006 INFO:tasks.workunit.client.0.smithi089.stdout:removed './133/37' 2023-10-30T12:34:27.006 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './133' 2023-10-30T12:34:27.006 INFO:tasks.workunit.client.0.smithi089.stdout:removed './134/19' 2023-10-30T12:34:27.006 INFO:tasks.workunit.client.0.smithi089.stdout:removed './134/15' 2023-10-30T12:34:27.006 INFO:tasks.workunit.client.0.smithi089.stdout:removed './134/31' 2023-10-30T12:34:27.006 INFO:tasks.workunit.client.0.smithi089.stdout:removed './134/36' 2023-10-30T12:34:27.007 INFO:tasks.workunit.client.0.smithi089.stdout:removed './134/20' 2023-10-30T12:34:27.007 INFO:tasks.workunit.client.0.smithi089.stdout:removed './134/12' 2023-10-30T12:34:27.007 INFO:tasks.workunit.client.0.smithi089.stdout:removed './134/39' 2023-10-30T12:34:27.007 INFO:tasks.workunit.client.0.smithi089.stdout:removed './134/10' 2023-10-30T12:34:27.008 INFO:tasks.workunit.client.0.smithi089.stdout:removed './134/34' 2023-10-30T12:34:27.008 INFO:tasks.workunit.client.0.smithi089.stdout:removed './134/38' 2023-10-30T12:34:27.008 INFO:tasks.workunit.client.0.smithi089.stdout:removed './134/11' 2023-10-30T12:34:27.008 INFO:tasks.workunit.client.0.smithi089.stdout:removed './134/16' 2023-10-30T12:34:27.008 INFO:tasks.workunit.client.0.smithi089.stdout:removed './134/32' 2023-10-30T12:34:27.008 INFO:tasks.workunit.client.0.smithi089.stdout:removed './134/30' 2023-10-30T12:34:27.009 INFO:tasks.workunit.client.0.smithi089.stdout:removed './134/14' 2023-10-30T12:34:27.009 INFO:tasks.workunit.client.0.smithi089.stdout:removed './134/18' 2023-10-30T12:34:27.009 INFO:tasks.workunit.client.0.smithi089.stdout:removed './134/13' 2023-10-30T12:34:27.009 INFO:tasks.workunit.client.0.smithi089.stdout:removed './134/33' 2023-10-30T12:34:27.009 INFO:tasks.workunit.client.0.smithi089.stdout:removed './134/17' 2023-10-30T12:34:27.010 INFO:tasks.workunit.client.0.smithi089.stdout:removed './134/35' 2023-10-30T12:34:27.010 INFO:tasks.workunit.client.0.smithi089.stdout:removed './134/37' 2023-10-30T12:34:27.010 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './134' 2023-10-30T12:34:27.010 INFO:tasks.workunit.client.0.smithi089.stdout:removed './135/19' 2023-10-30T12:34:27.010 INFO:tasks.workunit.client.0.smithi089.stdout:removed './135/15' 2023-10-30T12:34:27.011 INFO:tasks.workunit.client.0.smithi089.stdout:removed './135/31' 2023-10-30T12:34:27.011 INFO:tasks.workunit.client.0.smithi089.stdout:removed './135/36' 2023-10-30T12:34:27.011 INFO:tasks.workunit.client.0.smithi089.stdout:removed './135/20' 2023-10-30T12:34:27.012 INFO:tasks.workunit.client.0.smithi089.stdout:removed './135/12' 2023-10-30T12:34:27.012 INFO:tasks.workunit.client.0.smithi089.stdout:removed './135/39' 2023-10-30T12:34:27.012 INFO:tasks.workunit.client.0.smithi089.stdout:removed './135/10' 2023-10-30T12:34:27.012 INFO:tasks.workunit.client.0.smithi089.stdout:removed './135/34' 2023-10-30T12:34:27.012 INFO:tasks.workunit.client.0.smithi089.stdout:removed './135/38' 2023-10-30T12:34:27.013 INFO:tasks.workunit.client.0.smithi089.stdout:removed './135/11' 2023-10-30T12:34:27.013 INFO:tasks.workunit.client.0.smithi089.stdout:removed './135/16' 2023-10-30T12:34:27.013 INFO:tasks.workunit.client.0.smithi089.stdout:removed './135/32' 2023-10-30T12:34:27.013 INFO:tasks.workunit.client.0.smithi089.stdout:removed './135/30' 2023-10-30T12:34:27.013 INFO:tasks.workunit.client.0.smithi089.stdout:removed './135/14' 2023-10-30T12:34:27.014 INFO:tasks.workunit.client.0.smithi089.stdout:removed './135/18' 2023-10-30T12:34:27.014 INFO:tasks.workunit.client.0.smithi089.stdout:removed './135/13' 2023-10-30T12:34:27.014 INFO:tasks.workunit.client.0.smithi089.stdout:removed './135/33' 2023-10-30T12:34:27.014 INFO:tasks.workunit.client.0.smithi089.stdout:removed './135/17' 2023-10-30T12:34:27.014 INFO:tasks.workunit.client.0.smithi089.stdout:removed './135/35' 2023-10-30T12:34:27.014 INFO:tasks.workunit.client.0.smithi089.stdout:removed './135/37' 2023-10-30T12:34:27.015 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './135' 2023-10-30T12:34:27.015 INFO:tasks.workunit.client.0.smithi089.stdout:removed './136/19' 2023-10-30T12:34:27.015 INFO:tasks.workunit.client.0.smithi089.stdout:removed './136/15' 2023-10-30T12:34:27.015 INFO:tasks.workunit.client.0.smithi089.stdout:removed './136/31' 2023-10-30T12:34:27.015 INFO:tasks.workunit.client.0.smithi089.stdout:removed './136/36' 2023-10-30T12:34:27.016 INFO:tasks.workunit.client.0.smithi089.stdout:removed './136/20' 2023-10-30T12:34:27.016 INFO:tasks.workunit.client.0.smithi089.stdout:removed './136/12' 2023-10-30T12:34:27.016 INFO:tasks.workunit.client.0.smithi089.stdout:removed './136/39' 2023-10-30T12:34:27.016 INFO:tasks.workunit.client.0.smithi089.stdout:removed './136/10' 2023-10-30T12:34:27.016 INFO:tasks.workunit.client.0.smithi089.stdout:removed './136/34' 2023-10-30T12:34:27.017 INFO:tasks.workunit.client.0.smithi089.stdout:removed './136/38' 2023-10-30T12:34:27.017 INFO:tasks.workunit.client.0.smithi089.stdout:removed './136/11' 2023-10-30T12:34:27.017 INFO:tasks.workunit.client.0.smithi089.stdout:removed './136/16' 2023-10-30T12:34:27.017 INFO:tasks.workunit.client.0.smithi089.stdout:removed './136/32' 2023-10-30T12:34:27.017 INFO:tasks.workunit.client.0.smithi089.stdout:removed './136/30' 2023-10-30T12:34:27.018 INFO:tasks.workunit.client.0.smithi089.stdout:removed './136/14' 2023-10-30T12:34:27.018 INFO:tasks.workunit.client.0.smithi089.stdout:removed './136/18' 2023-10-30T12:34:27.018 INFO:tasks.workunit.client.0.smithi089.stdout:removed './136/13' 2023-10-30T12:34:27.018 INFO:tasks.workunit.client.0.smithi089.stdout:removed './136/33' 2023-10-30T12:34:27.018 INFO:tasks.workunit.client.0.smithi089.stdout:removed './136/17' 2023-10-30T12:34:27.018 INFO:tasks.workunit.client.0.smithi089.stdout:removed './136/35' 2023-10-30T12:34:27.019 INFO:tasks.workunit.client.0.smithi089.stdout:removed './136/37' 2023-10-30T12:34:27.019 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './136' 2023-10-30T12:34:27.019 INFO:tasks.workunit.client.0.smithi089.stdout:removed './137/19' 2023-10-30T12:34:27.019 INFO:tasks.workunit.client.0.smithi089.stdout:removed './137/15' 2023-10-30T12:34:27.019 INFO:tasks.workunit.client.0.smithi089.stdout:removed './137/31' 2023-10-30T12:34:27.020 INFO:tasks.workunit.client.0.smithi089.stdout:removed './137/36' 2023-10-30T12:34:27.020 INFO:tasks.workunit.client.0.smithi089.stdout:removed './137/20' 2023-10-30T12:34:27.020 INFO:tasks.workunit.client.0.smithi089.stdout:removed './137/12' 2023-10-30T12:34:27.020 INFO:tasks.workunit.client.0.smithi089.stdout:removed './137/39' 2023-10-30T12:34:27.020 INFO:tasks.workunit.client.0.smithi089.stdout:removed './137/10' 2023-10-30T12:34:27.021 INFO:tasks.workunit.client.0.smithi089.stdout:removed './137/34' 2023-10-30T12:34:27.021 INFO:tasks.workunit.client.0.smithi089.stdout:removed './137/38' 2023-10-30T12:34:27.021 INFO:tasks.workunit.client.0.smithi089.stdout:removed './137/11' 2023-10-30T12:34:27.021 INFO:tasks.workunit.client.0.smithi089.stdout:removed './137/16' 2023-10-30T12:34:27.021 INFO:tasks.workunit.client.0.smithi089.stdout:removed './137/32' 2023-10-30T12:34:27.021 INFO:tasks.workunit.client.0.smithi089.stdout:removed './137/30' 2023-10-30T12:34:27.022 INFO:tasks.workunit.client.0.smithi089.stdout:removed './137/14' 2023-10-30T12:34:27.022 INFO:tasks.workunit.client.0.smithi089.stdout:removed './137/18' 2023-10-30T12:34:27.022 INFO:tasks.workunit.client.0.smithi089.stdout:removed './137/13' 2023-10-30T12:34:27.022 INFO:tasks.workunit.client.0.smithi089.stdout:removed './137/33' 2023-10-30T12:34:27.022 INFO:tasks.workunit.client.0.smithi089.stdout:removed './137/17' 2023-10-30T12:34:27.023 INFO:tasks.workunit.client.0.smithi089.stdout:removed './137/35' 2023-10-30T12:34:27.023 INFO:tasks.workunit.client.0.smithi089.stdout:removed './137/37' 2023-10-30T12:34:27.023 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './137' 2023-10-30T12:34:27.023 INFO:tasks.workunit.client.0.smithi089.stdout:removed './138/19' 2023-10-30T12:34:27.023 INFO:tasks.workunit.client.0.smithi089.stdout:removed './138/15' 2023-10-30T12:34:27.024 INFO:tasks.workunit.client.0.smithi089.stdout:removed './138/31' 2023-10-30T12:34:27.024 INFO:tasks.workunit.client.0.smithi089.stdout:removed './138/36' 2023-10-30T12:34:27.024 INFO:tasks.workunit.client.0.smithi089.stdout:removed './138/20' 2023-10-30T12:34:27.024 INFO:tasks.workunit.client.0.smithi089.stdout:removed './138/12' 2023-10-30T12:34:27.024 INFO:tasks.workunit.client.0.smithi089.stdout:removed './138/39' 2023-10-30T12:34:27.024 INFO:tasks.workunit.client.0.smithi089.stdout:removed './138/10' 2023-10-30T12:34:27.025 INFO:tasks.workunit.client.0.smithi089.stdout:removed './138/34' 2023-10-30T12:34:27.025 INFO:tasks.workunit.client.0.smithi089.stdout:removed './138/38' 2023-10-30T12:34:27.025 INFO:tasks.workunit.client.0.smithi089.stdout:removed './138/11' 2023-10-30T12:34:27.025 INFO:tasks.workunit.client.0.smithi089.stdout:removed './138/16' 2023-10-30T12:34:31.073 INFO:tasks.workunit.client.0.smithi089.stdout:removed './138/32' 2023-10-30T12:34:31.074 INFO:tasks.workunit.client.0.smithi089.stdout:removed './138/30' 2023-10-30T12:34:31.074 INFO:tasks.workunit.client.0.smithi089.stdout:removed './138/14' 2023-10-30T12:34:31.074 INFO:tasks.workunit.client.0.smithi089.stdout:removed './138/18' 2023-10-30T12:34:31.074 INFO:tasks.workunit.client.0.smithi089.stdout:removed './138/13' 2023-10-30T12:34:31.074 INFO:tasks.workunit.client.0.smithi089.stdout:removed './138/33' 2023-10-30T12:34:31.075 INFO:tasks.workunit.client.0.smithi089.stdout:removed './138/17' 2023-10-30T12:34:31.075 INFO:tasks.workunit.client.0.smithi089.stdout:removed './138/35' 2023-10-30T12:34:31.075 INFO:tasks.workunit.client.0.smithi089.stdout:removed './138/37' 2023-10-30T12:34:31.075 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './138' 2023-10-30T12:34:31.075 INFO:tasks.workunit.client.0.smithi089.stdout:removed './139/19' 2023-10-30T12:34:31.076 INFO:tasks.workunit.client.0.smithi089.stdout:removed './139/15' 2023-10-30T12:34:31.076 INFO:tasks.workunit.client.0.smithi089.stdout:removed './139/31' 2023-10-30T12:34:31.076 INFO:tasks.workunit.client.0.smithi089.stdout:removed './139/36' 2023-10-30T12:34:31.076 INFO:tasks.workunit.client.0.smithi089.stdout:removed './139/20' 2023-10-30T12:34:31.076 INFO:tasks.workunit.client.0.smithi089.stdout:removed './139/12' 2023-10-30T12:34:31.077 INFO:tasks.workunit.client.0.smithi089.stdout:removed './139/39' 2023-10-30T12:34:31.077 INFO:tasks.workunit.client.0.smithi089.stdout:removed './139/10' 2023-10-30T12:34:31.077 INFO:tasks.workunit.client.0.smithi089.stdout:removed './139/34' 2023-10-30T12:34:31.077 INFO:tasks.workunit.client.0.smithi089.stdout:removed './139/38' 2023-10-30T12:34:31.077 INFO:tasks.workunit.client.0.smithi089.stdout:removed './139/11' 2023-10-30T12:34:31.078 INFO:tasks.workunit.client.0.smithi089.stdout:removed './139/16' 2023-10-30T12:34:31.078 INFO:tasks.workunit.client.0.smithi089.stdout:removed './139/32' 2023-10-30T12:34:31.078 INFO:tasks.workunit.client.0.smithi089.stdout:removed './139/30' 2023-10-30T12:34:31.078 INFO:tasks.workunit.client.0.smithi089.stdout:removed './139/14' 2023-10-30T12:34:31.078 INFO:tasks.workunit.client.0.smithi089.stdout:removed './139/18' 2023-10-30T12:34:31.079 INFO:tasks.workunit.client.0.smithi089.stdout:removed './139/13' 2023-10-30T12:34:31.079 INFO:tasks.workunit.client.0.smithi089.stdout:removed './139/33' 2023-10-30T12:34:31.079 INFO:tasks.workunit.client.0.smithi089.stdout:removed './139/17' 2023-10-30T12:34:31.079 INFO:tasks.workunit.client.0.smithi089.stdout:removed './139/35' 2023-10-30T12:34:31.079 INFO:tasks.workunit.client.0.smithi089.stdout:removed './139/37' 2023-10-30T12:34:31.079 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './139' 2023-10-30T12:34:31.080 INFO:tasks.workunit.client.0.smithi089.stdout:removed './140/19' 2023-10-30T12:34:31.080 INFO:tasks.workunit.client.0.smithi089.stdout:removed './140/15' 2023-10-30T12:34:31.080 INFO:tasks.workunit.client.0.smithi089.stdout:removed './140/31' 2023-10-30T12:34:31.080 INFO:tasks.workunit.client.0.smithi089.stdout:removed './140/36' 2023-10-30T12:34:31.080 INFO:tasks.workunit.client.0.smithi089.stdout:removed './140/20' 2023-10-30T12:34:31.081 INFO:tasks.workunit.client.0.smithi089.stdout:removed './140/12' 2023-10-30T12:34:31.081 INFO:tasks.workunit.client.0.smithi089.stdout:removed './140/39' 2023-10-30T12:34:31.081 INFO:tasks.workunit.client.0.smithi089.stdout:removed './140/10' 2023-10-30T12:34:31.081 INFO:tasks.workunit.client.0.smithi089.stdout:removed './140/34' 2023-10-30T12:34:31.081 INFO:tasks.workunit.client.0.smithi089.stdout:removed './140/38' 2023-10-30T12:34:31.082 INFO:tasks.workunit.client.0.smithi089.stdout:removed './140/11' 2023-10-30T12:34:31.082 INFO:tasks.workunit.client.0.smithi089.stdout:removed './140/16' 2023-10-30T12:34:31.082 INFO:tasks.workunit.client.0.smithi089.stdout:removed './140/32' 2023-10-30T12:34:31.082 INFO:tasks.workunit.client.0.smithi089.stdout:removed './140/30' 2023-10-30T12:34:31.082 INFO:tasks.workunit.client.0.smithi089.stdout:removed './140/14' 2023-10-30T12:34:31.083 INFO:tasks.workunit.client.0.smithi089.stdout:removed './140/18' 2023-10-30T12:34:31.083 INFO:tasks.workunit.client.0.smithi089.stdout:removed './140/13' 2023-10-30T12:34:31.083 INFO:tasks.workunit.client.0.smithi089.stdout:removed './140/33' 2023-10-30T12:34:31.083 INFO:tasks.workunit.client.0.smithi089.stdout:removed './140/17' 2023-10-30T12:34:31.083 INFO:tasks.workunit.client.0.smithi089.stdout:removed './140/35' 2023-10-30T12:34:31.084 INFO:tasks.workunit.client.0.smithi089.stdout:removed './140/37' 2023-10-30T12:34:31.084 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './140' 2023-10-30T12:34:31.084 INFO:tasks.workunit.client.0.smithi089.stdout:removed './141/19' 2023-10-30T12:34:31.084 INFO:tasks.workunit.client.0.smithi089.stdout:removed './141/15' 2023-10-30T12:34:31.084 INFO:tasks.workunit.client.0.smithi089.stdout:removed './141/31' 2023-10-30T12:34:31.085 INFO:tasks.workunit.client.0.smithi089.stdout:removed './141/36' 2023-10-30T12:34:31.085 INFO:tasks.workunit.client.0.smithi089.stdout:removed './141/20' 2023-10-30T12:34:31.085 INFO:tasks.workunit.client.0.smithi089.stdout:removed './141/12' 2023-10-30T12:34:31.085 INFO:tasks.workunit.client.0.smithi089.stdout:removed './141/39' 2023-10-30T12:34:31.085 INFO:tasks.workunit.client.0.smithi089.stdout:removed './141/10' 2023-10-30T12:34:31.086 INFO:tasks.workunit.client.0.smithi089.stdout:removed './141/34' 2023-10-30T12:34:31.086 INFO:tasks.workunit.client.0.smithi089.stdout:removed './141/38' 2023-10-30T12:34:31.086 INFO:tasks.workunit.client.0.smithi089.stdout:removed './141/11' 2023-10-30T12:34:31.086 INFO:tasks.workunit.client.0.smithi089.stdout:removed './141/16' 2023-10-30T12:34:31.086 INFO:tasks.workunit.client.0.smithi089.stdout:removed './141/32' 2023-10-30T12:34:31.087 INFO:tasks.workunit.client.0.smithi089.stdout:removed './141/30' 2023-10-30T12:34:31.087 INFO:tasks.workunit.client.0.smithi089.stdout:removed './141/14' 2023-10-30T12:34:31.087 INFO:tasks.workunit.client.0.smithi089.stdout:removed './141/18' 2023-10-30T12:34:31.087 INFO:tasks.workunit.client.0.smithi089.stdout:removed './141/13' 2023-10-30T12:34:31.087 INFO:tasks.workunit.client.0.smithi089.stdout:removed './141/33' 2023-10-30T12:34:31.088 INFO:tasks.workunit.client.0.smithi089.stdout:removed './141/17' 2023-10-30T12:34:31.088 INFO:tasks.workunit.client.0.smithi089.stdout:removed './141/35' 2023-10-30T12:34:31.088 INFO:tasks.workunit.client.0.smithi089.stdout:removed './141/37' 2023-10-30T12:34:31.088 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './141' 2023-10-30T12:34:31.088 INFO:tasks.workunit.client.0.smithi089.stdout:removed './142/19' 2023-10-30T12:34:31.089 INFO:tasks.workunit.client.0.smithi089.stdout:removed './142/15' 2023-10-30T12:34:31.089 INFO:tasks.workunit.client.0.smithi089.stdout:removed './142/31' 2023-10-30T12:34:31.089 INFO:tasks.workunit.client.0.smithi089.stdout:removed './142/36' 2023-10-30T12:34:31.089 INFO:tasks.workunit.client.0.smithi089.stdout:removed './142/20' 2023-10-30T12:34:31.089 INFO:tasks.workunit.client.0.smithi089.stdout:removed './142/12' 2023-10-30T12:34:31.089 INFO:tasks.workunit.client.0.smithi089.stdout:removed './142/39' 2023-10-30T12:34:31.090 INFO:tasks.workunit.client.0.smithi089.stdout:removed './142/10' 2023-10-30T12:34:31.090 INFO:tasks.workunit.client.0.smithi089.stdout:removed './142/34' 2023-10-30T12:34:31.090 INFO:tasks.workunit.client.0.smithi089.stdout:removed './142/38' 2023-10-30T12:34:31.090 INFO:tasks.workunit.client.0.smithi089.stdout:removed './142/11' 2023-10-30T12:34:31.090 INFO:tasks.workunit.client.0.smithi089.stdout:removed './142/16' 2023-10-30T12:34:31.091 INFO:tasks.workunit.client.0.smithi089.stdout:removed './142/32' 2023-10-30T12:34:31.091 INFO:tasks.workunit.client.0.smithi089.stdout:removed './142/30' 2023-10-30T12:34:31.091 INFO:tasks.workunit.client.0.smithi089.stdout:removed './142/14' 2023-10-30T12:34:31.091 INFO:tasks.workunit.client.0.smithi089.stdout:removed './142/18' 2023-10-30T12:34:31.091 INFO:tasks.workunit.client.0.smithi089.stdout:removed './142/13' 2023-10-30T12:34:31.092 INFO:tasks.workunit.client.0.smithi089.stdout:removed './142/33' 2023-10-30T12:34:31.092 INFO:tasks.workunit.client.0.smithi089.stdout:removed './142/17' 2023-10-30T12:34:31.092 INFO:tasks.workunit.client.0.smithi089.stdout:removed './142/35' 2023-10-30T12:34:31.092 INFO:tasks.workunit.client.0.smithi089.stdout:removed './142/37' 2023-10-30T12:34:31.092 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './142' 2023-10-30T12:34:31.093 INFO:tasks.workunit.client.0.smithi089.stdout:removed './143/19' 2023-10-30T12:34:31.093 INFO:tasks.workunit.client.0.smithi089.stdout:removed './143/15' 2023-10-30T12:34:31.093 INFO:tasks.workunit.client.0.smithi089.stdout:removed './143/31' 2023-10-30T12:34:31.093 INFO:tasks.workunit.client.0.smithi089.stdout:removed './143/36' 2023-10-30T12:34:31.093 INFO:tasks.workunit.client.0.smithi089.stdout:removed './143/20' 2023-10-30T12:34:31.094 INFO:tasks.workunit.client.0.smithi089.stdout:removed './143/12' 2023-10-30T12:34:31.094 INFO:tasks.workunit.client.0.smithi089.stdout:removed './143/39' 2023-10-30T12:34:31.094 INFO:tasks.workunit.client.0.smithi089.stdout:removed './143/10' 2023-10-30T12:34:31.094 INFO:tasks.workunit.client.0.smithi089.stdout:removed './143/34' 2023-10-30T12:34:31.094 INFO:tasks.workunit.client.0.smithi089.stdout:removed './143/38' 2023-10-30T12:34:31.095 INFO:tasks.workunit.client.0.smithi089.stdout:removed './143/11' 2023-10-30T12:34:31.095 INFO:tasks.workunit.client.0.smithi089.stdout:removed './143/16' 2023-10-30T12:34:31.095 INFO:tasks.workunit.client.0.smithi089.stdout:removed './143/32' 2023-10-30T12:34:31.095 INFO:tasks.workunit.client.0.smithi089.stdout:removed './143/30' 2023-10-30T12:34:31.095 INFO:tasks.workunit.client.0.smithi089.stdout:removed './143/14' 2023-10-30T12:34:31.095 INFO:tasks.workunit.client.0.smithi089.stdout:removed './143/18' 2023-10-30T12:34:31.096 INFO:tasks.workunit.client.0.smithi089.stdout:removed './143/13' 2023-10-30T12:34:31.096 INFO:tasks.workunit.client.0.smithi089.stdout:removed './143/33' 2023-10-30T12:34:31.096 INFO:tasks.workunit.client.0.smithi089.stdout:removed './143/17' 2023-10-30T12:34:31.096 INFO:tasks.workunit.client.0.smithi089.stdout:removed './143/35' 2023-10-30T12:34:31.096 INFO:tasks.workunit.client.0.smithi089.stdout:removed './143/37' 2023-10-30T12:34:31.097 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './143' 2023-10-30T12:34:31.097 INFO:tasks.workunit.client.0.smithi089.stdout:removed './144/19' 2023-10-30T12:34:31.097 INFO:tasks.workunit.client.0.smithi089.stdout:removed './144/15' 2023-10-30T12:34:31.097 INFO:tasks.workunit.client.0.smithi089.stdout:removed './144/31' 2023-10-30T12:34:31.097 INFO:tasks.workunit.client.0.smithi089.stdout:removed './144/36' 2023-10-30T12:34:31.098 INFO:tasks.workunit.client.0.smithi089.stdout:removed './144/20' 2023-10-30T12:34:31.098 INFO:tasks.workunit.client.0.smithi089.stdout:removed './144/12' 2023-10-30T12:34:31.098 INFO:tasks.workunit.client.0.smithi089.stdout:removed './144/39' 2023-10-30T12:34:31.098 INFO:tasks.workunit.client.0.smithi089.stdout:removed './144/10' 2023-10-30T12:34:31.098 INFO:tasks.workunit.client.0.smithi089.stdout:removed './144/34' 2023-10-30T12:34:31.099 INFO:tasks.workunit.client.0.smithi089.stdout:removed './144/38' 2023-10-30T12:34:31.101 INFO:tasks.workunit.client.0.smithi089.stdout:removed './144/11' 2023-10-30T12:34:31.101 INFO:tasks.workunit.client.0.smithi089.stdout:removed './144/16' 2023-10-30T12:34:31.101 INFO:tasks.workunit.client.0.smithi089.stdout:removed './144/32' 2023-10-30T12:34:31.101 INFO:tasks.workunit.client.0.smithi089.stdout:removed './144/30' 2023-10-30T12:34:31.102 INFO:tasks.workunit.client.0.smithi089.stdout:removed './144/14' 2023-10-30T12:34:31.102 INFO:tasks.workunit.client.0.smithi089.stdout:removed './144/18' 2023-10-30T12:34:31.102 INFO:tasks.workunit.client.0.smithi089.stdout:removed './144/13' 2023-10-30T12:34:31.102 INFO:tasks.workunit.client.0.smithi089.stdout:removed './144/33' 2023-10-30T12:34:31.102 INFO:tasks.workunit.client.0.smithi089.stdout:removed './144/17' 2023-10-30T12:34:31.103 INFO:tasks.workunit.client.0.smithi089.stdout:removed './144/35' 2023-10-30T12:34:31.103 INFO:tasks.workunit.client.0.smithi089.stdout:removed './144/37' 2023-10-30T12:34:31.103 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './144' 2023-10-30T12:34:31.103 INFO:tasks.workunit.client.0.smithi089.stdout:removed './145/19' 2023-10-30T12:34:31.103 INFO:tasks.workunit.client.0.smithi089.stdout:removed './145/15' 2023-10-30T12:34:31.104 INFO:tasks.workunit.client.0.smithi089.stdout:removed './145/31' 2023-10-30T12:34:31.104 INFO:tasks.workunit.client.0.smithi089.stdout:removed './145/36' 2023-10-30T12:34:31.104 INFO:tasks.workunit.client.0.smithi089.stdout:removed './145/20' 2023-10-30T12:34:31.104 INFO:tasks.workunit.client.0.smithi089.stdout:removed './145/12' 2023-10-30T12:34:31.104 INFO:tasks.workunit.client.0.smithi089.stdout:removed './145/39' 2023-10-30T12:34:31.105 INFO:tasks.workunit.client.0.smithi089.stdout:removed './145/10' 2023-10-30T12:34:31.105 INFO:tasks.workunit.client.0.smithi089.stdout:removed './145/34' 2023-10-30T12:34:31.105 INFO:tasks.workunit.client.0.smithi089.stdout:removed './145/38' 2023-10-30T12:34:31.105 INFO:tasks.workunit.client.0.smithi089.stdout:removed './145/11' 2023-10-30T12:34:31.105 INFO:tasks.workunit.client.0.smithi089.stdout:removed './145/16' 2023-10-30T12:34:31.105 INFO:tasks.workunit.client.0.smithi089.stdout:removed './145/32' 2023-10-30T12:34:31.106 INFO:tasks.workunit.client.0.smithi089.stdout:removed './145/30' 2023-10-30T12:34:31.106 INFO:tasks.workunit.client.0.smithi089.stdout:removed './145/14' 2023-10-30T12:34:31.106 INFO:tasks.workunit.client.0.smithi089.stdout:removed './145/18' 2023-10-30T12:34:31.106 INFO:tasks.workunit.client.0.smithi089.stdout:removed './145/13' 2023-10-30T12:34:31.106 INFO:tasks.workunit.client.0.smithi089.stdout:removed './145/33' 2023-10-30T12:34:31.107 INFO:tasks.workunit.client.0.smithi089.stdout:removed './145/17' 2023-10-30T12:34:31.107 INFO:tasks.workunit.client.0.smithi089.stdout:removed './145/35' 2023-10-30T12:34:31.107 INFO:tasks.workunit.client.0.smithi089.stdout:removed './145/37' 2023-10-30T12:34:31.107 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './145' 2023-10-30T12:34:31.107 INFO:tasks.workunit.client.0.smithi089.stdout:removed './146/19' 2023-10-30T12:34:31.108 INFO:tasks.workunit.client.0.smithi089.stdout:removed './146/15' 2023-10-30T12:34:31.108 INFO:tasks.workunit.client.0.smithi089.stdout:removed './146/31' 2023-10-30T12:34:31.108 INFO:tasks.workunit.client.0.smithi089.stdout:removed './146/36' 2023-10-30T12:34:31.108 INFO:tasks.workunit.client.0.smithi089.stdout:removed './146/20' 2023-10-30T12:34:31.108 INFO:tasks.workunit.client.0.smithi089.stdout:removed './146/12' 2023-10-30T12:34:31.109 INFO:tasks.workunit.client.0.smithi089.stdout:removed './146/39' 2023-10-30T12:34:31.109 INFO:tasks.workunit.client.0.smithi089.stdout:removed './146/10' 2023-10-30T12:34:31.109 INFO:tasks.workunit.client.0.smithi089.stdout:removed './146/34' 2023-10-30T12:34:31.109 INFO:tasks.workunit.client.0.smithi089.stdout:removed './146/38' 2023-10-30T12:34:31.109 INFO:tasks.workunit.client.0.smithi089.stdout:removed './146/11' 2023-10-30T12:34:31.110 INFO:tasks.workunit.client.0.smithi089.stdout:removed './146/16' 2023-10-30T12:34:31.110 INFO:tasks.workunit.client.0.smithi089.stdout:removed './146/32' 2023-10-30T12:34:31.110 INFO:tasks.workunit.client.0.smithi089.stdout:removed './146/30' 2023-10-30T12:34:31.110 INFO:tasks.workunit.client.0.smithi089.stdout:removed './146/14' 2023-10-30T12:34:31.110 INFO:tasks.workunit.client.0.smithi089.stdout:removed './146/18' 2023-10-30T12:34:31.111 INFO:tasks.workunit.client.0.smithi089.stdout:removed './146/13' 2023-10-30T12:34:31.111 INFO:tasks.workunit.client.0.smithi089.stdout:removed './146/33' 2023-10-30T12:34:31.111 INFO:tasks.workunit.client.0.smithi089.stdout:removed './146/17' 2023-10-30T12:34:31.111 INFO:tasks.workunit.client.0.smithi089.stdout:removed './146/35' 2023-10-30T12:34:31.111 INFO:tasks.workunit.client.0.smithi089.stdout:removed './146/37' 2023-10-30T12:34:31.111 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './146' 2023-10-30T12:34:31.112 INFO:tasks.workunit.client.0.smithi089.stdout:removed './147/19' 2023-10-30T12:34:31.112 INFO:tasks.workunit.client.0.smithi089.stdout:removed './147/15' 2023-10-30T12:34:31.112 INFO:tasks.workunit.client.0.smithi089.stdout:removed './147/31' 2023-10-30T12:34:31.112 INFO:tasks.workunit.client.0.smithi089.stdout:removed './147/36' 2023-10-30T12:34:31.112 INFO:tasks.workunit.client.0.smithi089.stdout:removed './147/20' 2023-10-30T12:34:31.113 INFO:tasks.workunit.client.0.smithi089.stdout:removed './147/12' 2023-10-30T12:34:31.113 INFO:tasks.workunit.client.0.smithi089.stdout:removed './147/39' 2023-10-30T12:34:31.113 INFO:tasks.workunit.client.0.smithi089.stdout:removed './147/10' 2023-10-30T12:34:31.113 INFO:tasks.workunit.client.0.smithi089.stdout:removed './147/34' 2023-10-30T12:34:31.113 INFO:tasks.workunit.client.0.smithi089.stdout:removed './147/38' 2023-10-30T12:34:31.114 INFO:tasks.workunit.client.0.smithi089.stdout:removed './147/11' 2023-10-30T12:34:31.114 INFO:tasks.workunit.client.0.smithi089.stdout:removed './147/16' 2023-10-30T12:34:31.114 INFO:tasks.workunit.client.0.smithi089.stdout:removed './147/32' 2023-10-30T12:34:31.114 INFO:tasks.workunit.client.0.smithi089.stdout:removed './147/30' 2023-10-30T12:34:31.114 INFO:tasks.workunit.client.0.smithi089.stdout:removed './147/14' 2023-10-30T12:34:31.114 INFO:tasks.workunit.client.0.smithi089.stdout:removed './147/18' 2023-10-30T12:34:31.115 INFO:tasks.workunit.client.0.smithi089.stdout:removed './147/13' 2023-10-30T12:34:31.115 INFO:tasks.workunit.client.0.smithi089.stdout:removed './147/33' 2023-10-30T12:34:31.115 INFO:tasks.workunit.client.0.smithi089.stdout:removed './147/17' 2023-10-30T12:34:31.115 INFO:tasks.workunit.client.0.smithi089.stdout:removed './147/35' 2023-10-30T12:34:31.115 INFO:tasks.workunit.client.0.smithi089.stdout:removed './147/37' 2023-10-30T12:34:31.116 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './147' 2023-10-30T12:34:31.116 INFO:tasks.workunit.client.0.smithi089.stdout:removed './148/19' 2023-10-30T12:34:31.116 INFO:tasks.workunit.client.0.smithi089.stdout:removed './148/15' 2023-10-30T12:34:31.116 INFO:tasks.workunit.client.0.smithi089.stdout:removed './148/31' 2023-10-30T12:34:31.116 INFO:tasks.workunit.client.0.smithi089.stdout:removed './148/36' 2023-10-30T12:34:36.347 INFO:tasks.workunit.client.0.smithi089.stdout:removed './148/20' 2023-10-30T12:34:36.348 INFO:tasks.workunit.client.0.smithi089.stdout:removed './148/12' 2023-10-30T12:34:36.348 INFO:tasks.workunit.client.0.smithi089.stdout:removed './148/39' 2023-10-30T12:34:36.348 INFO:tasks.workunit.client.0.smithi089.stdout:removed './148/10' 2023-10-30T12:34:36.349 INFO:tasks.workunit.client.0.smithi089.stdout:removed './148/34' 2023-10-30T12:34:36.349 INFO:tasks.workunit.client.0.smithi089.stdout:removed './148/38' 2023-10-30T12:34:36.349 INFO:tasks.workunit.client.0.smithi089.stdout:removed './148/11' 2023-10-30T12:34:36.349 INFO:tasks.workunit.client.0.smithi089.stdout:removed './148/16' 2023-10-30T12:34:36.349 INFO:tasks.workunit.client.0.smithi089.stdout:removed './148/32' 2023-10-30T12:34:36.350 INFO:tasks.workunit.client.0.smithi089.stdout:removed './148/30' 2023-10-30T12:34:36.353 INFO:tasks.workunit.client.0.smithi089.stdout:removed './148/14' 2023-10-30T12:34:36.353 INFO:tasks.workunit.client.0.smithi089.stdout:removed './148/18' 2023-10-30T12:34:36.353 INFO:tasks.workunit.client.0.smithi089.stdout:removed './148/13' 2023-10-30T12:34:36.354 INFO:tasks.workunit.client.0.smithi089.stdout:removed './148/33' 2023-10-30T12:34:36.354 INFO:tasks.workunit.client.0.smithi089.stdout:removed './148/17' 2023-10-30T12:34:36.354 INFO:tasks.workunit.client.0.smithi089.stdout:removed './148/35' 2023-10-30T12:34:36.354 INFO:tasks.workunit.client.0.smithi089.stdout:removed './148/37' 2023-10-30T12:34:36.355 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './148' 2023-10-30T12:34:36.355 INFO:tasks.workunit.client.0.smithi089.stdout:removed './149/19' 2023-10-30T12:34:36.355 INFO:tasks.workunit.client.0.smithi089.stdout:removed './149/15' 2023-10-30T12:34:36.355 INFO:tasks.workunit.client.0.smithi089.stdout:removed './149/31' 2023-10-30T12:34:36.356 INFO:tasks.workunit.client.0.smithi089.stdout:removed './149/36' 2023-10-30T12:34:36.356 INFO:tasks.workunit.client.0.smithi089.stdout:removed './149/20' 2023-10-30T12:34:36.356 INFO:tasks.workunit.client.0.smithi089.stdout:removed './149/12' 2023-10-30T12:34:36.356 INFO:tasks.workunit.client.0.smithi089.stdout:removed './149/39' 2023-10-30T12:34:36.356 INFO:tasks.workunit.client.0.smithi089.stdout:removed './149/10' 2023-10-30T12:34:36.357 INFO:tasks.workunit.client.0.smithi089.stdout:removed './149/34' 2023-10-30T12:34:36.357 INFO:tasks.workunit.client.0.smithi089.stdout:removed './149/38' 2023-10-30T12:34:36.357 INFO:tasks.workunit.client.0.smithi089.stdout:removed './149/11' 2023-10-30T12:34:36.357 INFO:tasks.workunit.client.0.smithi089.stdout:removed './149/16' 2023-10-30T12:34:36.357 INFO:tasks.workunit.client.0.smithi089.stdout:removed './149/32' 2023-10-30T12:34:36.358 INFO:tasks.workunit.client.0.smithi089.stdout:removed './149/30' 2023-10-30T12:34:36.358 INFO:tasks.workunit.client.0.smithi089.stdout:removed './149/14' 2023-10-30T12:34:36.358 INFO:tasks.workunit.client.0.smithi089.stdout:removed './149/18' 2023-10-30T12:34:36.358 INFO:tasks.workunit.client.0.smithi089.stdout:removed './149/13' 2023-10-30T12:34:36.359 INFO:tasks.workunit.client.0.smithi089.stdout:removed './149/33' 2023-10-30T12:34:36.359 INFO:tasks.workunit.client.0.smithi089.stdout:removed './149/17' 2023-10-30T12:34:36.359 INFO:tasks.workunit.client.0.smithi089.stdout:removed './149/35' 2023-10-30T12:34:36.359 INFO:tasks.workunit.client.0.smithi089.stdout:removed './149/37' 2023-10-30T12:34:36.359 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './149' 2023-10-30T12:34:36.360 INFO:tasks.workunit.client.0.smithi089.stdout:removed './150/19' 2023-10-30T12:34:36.360 INFO:tasks.workunit.client.0.smithi089.stdout:removed './150/15' 2023-10-30T12:34:36.360 INFO:tasks.workunit.client.0.smithi089.stdout:removed './150/31' 2023-10-30T12:34:36.360 INFO:tasks.workunit.client.0.smithi089.stdout:removed './150/36' 2023-10-30T12:34:36.361 INFO:tasks.workunit.client.0.smithi089.stdout:removed './150/20' 2023-10-30T12:34:36.361 INFO:tasks.workunit.client.0.smithi089.stdout:removed './150/12' 2023-10-30T12:34:36.361 INFO:tasks.workunit.client.0.smithi089.stdout:removed './150/39' 2023-10-30T12:34:36.361 INFO:tasks.workunit.client.0.smithi089.stdout:removed './150/10' 2023-10-30T12:34:36.362 INFO:tasks.workunit.client.0.smithi089.stdout:removed './150/34' 2023-10-30T12:34:36.362 INFO:tasks.workunit.client.0.smithi089.stdout:removed './150/38' 2023-10-30T12:34:36.362 INFO:tasks.workunit.client.0.smithi089.stdout:removed './150/11' 2023-10-30T12:34:36.362 INFO:tasks.workunit.client.0.smithi089.stdout:removed './150/16' 2023-10-30T12:34:36.362 INFO:tasks.workunit.client.0.smithi089.stdout:removed './150/32' 2023-10-30T12:34:36.363 INFO:tasks.workunit.client.0.smithi089.stdout:removed './150/30' 2023-10-30T12:34:36.363 INFO:tasks.workunit.client.0.smithi089.stdout:removed './150/14' 2023-10-30T12:34:36.363 INFO:tasks.workunit.client.0.smithi089.stdout:removed './150/18' 2023-10-30T12:34:36.363 INFO:tasks.workunit.client.0.smithi089.stdout:removed './150/13' 2023-10-30T12:34:36.363 INFO:tasks.workunit.client.0.smithi089.stdout:removed './150/33' 2023-10-30T12:34:36.364 INFO:tasks.workunit.client.0.smithi089.stdout:removed './150/17' 2023-10-30T12:34:36.364 INFO:tasks.workunit.client.0.smithi089.stdout:removed './150/35' 2023-10-30T12:34:36.364 INFO:tasks.workunit.client.0.smithi089.stdout:removed './150/37' 2023-10-30T12:34:36.364 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './150' 2023-10-30T12:34:36.364 INFO:tasks.workunit.client.0.smithi089.stdout:removed './151/19' 2023-10-30T12:34:36.365 INFO:tasks.workunit.client.0.smithi089.stdout:removed './151/15' 2023-10-30T12:34:36.365 INFO:tasks.workunit.client.0.smithi089.stdout:removed './151/31' 2023-10-30T12:34:36.365 INFO:tasks.workunit.client.0.smithi089.stdout:removed './151/36' 2023-10-30T12:34:36.365 INFO:tasks.workunit.client.0.smithi089.stdout:removed './151/20' 2023-10-30T12:34:36.365 INFO:tasks.workunit.client.0.smithi089.stdout:removed './151/12' 2023-10-30T12:34:36.366 INFO:tasks.workunit.client.0.smithi089.stdout:removed './151/39' 2023-10-30T12:34:36.366 INFO:tasks.workunit.client.0.smithi089.stdout:removed './151/10' 2023-10-30T12:34:36.366 INFO:tasks.workunit.client.0.smithi089.stdout:removed './151/34' 2023-10-30T12:34:36.366 INFO:tasks.workunit.client.0.smithi089.stdout:removed './151/38' 2023-10-30T12:34:36.367 INFO:tasks.workunit.client.0.smithi089.stdout:removed './151/11' 2023-10-30T12:34:36.367 INFO:tasks.workunit.client.0.smithi089.stdout:removed './151/16' 2023-10-30T12:34:36.367 INFO:tasks.workunit.client.0.smithi089.stdout:removed './151/32' 2023-10-30T12:34:36.367 INFO:tasks.workunit.client.0.smithi089.stdout:removed './151/30' 2023-10-30T12:34:36.367 INFO:tasks.workunit.client.0.smithi089.stdout:removed './151/14' 2023-10-30T12:34:36.368 INFO:tasks.workunit.client.0.smithi089.stdout:removed './151/18' 2023-10-30T12:34:36.368 INFO:tasks.workunit.client.0.smithi089.stdout:removed './151/13' 2023-10-30T12:34:36.368 INFO:tasks.workunit.client.0.smithi089.stdout:removed './151/33' 2023-10-30T12:34:36.368 INFO:tasks.workunit.client.0.smithi089.stdout:removed './151/17' 2023-10-30T12:34:36.369 INFO:tasks.workunit.client.0.smithi089.stdout:removed './151/35' 2023-10-30T12:34:36.369 INFO:tasks.workunit.client.0.smithi089.stdout:removed './151/37' 2023-10-30T12:34:36.369 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './151' 2023-10-30T12:34:36.369 INFO:tasks.workunit.client.0.smithi089.stdout:removed './152/19' 2023-10-30T12:34:36.369 INFO:tasks.workunit.client.0.smithi089.stdout:removed './152/15' 2023-10-30T12:34:36.370 INFO:tasks.workunit.client.0.smithi089.stdout:removed './152/31' 2023-10-30T12:34:36.370 INFO:tasks.workunit.client.0.smithi089.stdout:removed './152/36' 2023-10-30T12:34:36.370 INFO:tasks.workunit.client.0.smithi089.stdout:removed './152/20' 2023-10-30T12:34:36.370 INFO:tasks.workunit.client.0.smithi089.stdout:removed './152/12' 2023-10-30T12:34:36.371 INFO:tasks.workunit.client.0.smithi089.stdout:removed './152/39' 2023-10-30T12:34:36.371 INFO:tasks.workunit.client.0.smithi089.stdout:removed './152/10' 2023-10-30T12:34:36.371 INFO:tasks.workunit.client.0.smithi089.stdout:removed './152/34' 2023-10-30T12:34:36.371 INFO:tasks.workunit.client.0.smithi089.stdout:removed './152/38' 2023-10-30T12:34:36.371 INFO:tasks.workunit.client.0.smithi089.stdout:removed './152/11' 2023-10-30T12:34:36.372 INFO:tasks.workunit.client.0.smithi089.stdout:removed './152/16' 2023-10-30T12:34:36.372 INFO:tasks.workunit.client.0.smithi089.stdout:removed './152/32' 2023-10-30T12:34:36.372 INFO:tasks.workunit.client.0.smithi089.stdout:removed './152/30' 2023-10-30T12:34:36.372 INFO:tasks.workunit.client.0.smithi089.stdout:removed './152/14' 2023-10-30T12:34:36.373 INFO:tasks.workunit.client.0.smithi089.stdout:removed './152/18' 2023-10-30T12:34:36.373 INFO:tasks.workunit.client.0.smithi089.stdout:removed './152/13' 2023-10-30T12:34:36.373 INFO:tasks.workunit.client.0.smithi089.stdout:removed './152/33' 2023-10-30T12:34:36.373 INFO:tasks.workunit.client.0.smithi089.stdout:removed './152/17' 2023-10-30T12:34:36.374 INFO:tasks.workunit.client.0.smithi089.stdout:removed './152/35' 2023-10-30T12:34:36.374 INFO:tasks.workunit.client.0.smithi089.stdout:removed './152/37' 2023-10-30T12:34:36.374 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './152' 2023-10-30T12:34:36.374 INFO:tasks.workunit.client.0.smithi089.stdout:removed './153/19' 2023-10-30T12:34:36.375 INFO:tasks.workunit.client.0.smithi089.stdout:removed './153/15' 2023-10-30T12:34:36.375 INFO:tasks.workunit.client.0.smithi089.stdout:removed './153/31' 2023-10-30T12:34:36.375 INFO:tasks.workunit.client.0.smithi089.stdout:removed './153/36' 2023-10-30T12:34:36.376 INFO:tasks.workunit.client.0.smithi089.stdout:removed './153/20' 2023-10-30T12:34:36.376 INFO:tasks.workunit.client.0.smithi089.stdout:removed './153/12' 2023-10-30T12:34:36.376 INFO:tasks.workunit.client.0.smithi089.stdout:removed './153/39' 2023-10-30T12:34:36.376 INFO:tasks.workunit.client.0.smithi089.stdout:removed './153/10' 2023-10-30T12:34:36.377 INFO:tasks.workunit.client.0.smithi089.stdout:removed './153/34' 2023-10-30T12:34:36.377 INFO:tasks.workunit.client.0.smithi089.stdout:removed './153/38' 2023-10-30T12:34:36.377 INFO:tasks.workunit.client.0.smithi089.stdout:removed './153/11' 2023-10-30T12:34:36.377 INFO:tasks.workunit.client.0.smithi089.stdout:removed './153/16' 2023-10-30T12:34:36.377 INFO:tasks.workunit.client.0.smithi089.stdout:removed './153/32' 2023-10-30T12:34:36.378 INFO:tasks.workunit.client.0.smithi089.stdout:removed './153/30' 2023-10-30T12:34:36.378 INFO:tasks.workunit.client.0.smithi089.stdout:removed './153/14' 2023-10-30T12:34:36.378 INFO:tasks.workunit.client.0.smithi089.stdout:removed './153/18' 2023-10-30T12:34:36.378 INFO:tasks.workunit.client.0.smithi089.stdout:removed './153/13' 2023-10-30T12:34:36.379 INFO:tasks.workunit.client.0.smithi089.stdout:removed './153/33' 2023-10-30T12:34:36.379 INFO:tasks.workunit.client.0.smithi089.stdout:removed './153/17' 2023-10-30T12:34:36.379 INFO:tasks.workunit.client.0.smithi089.stdout:removed './153/35' 2023-10-30T12:34:36.379 INFO:tasks.workunit.client.0.smithi089.stdout:removed './153/37' 2023-10-30T12:34:36.379 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './153' 2023-10-30T12:34:36.380 INFO:tasks.workunit.client.0.smithi089.stdout:removed './154/19' 2023-10-30T12:34:36.380 INFO:tasks.workunit.client.0.smithi089.stdout:removed './154/15' 2023-10-30T12:34:36.380 INFO:tasks.workunit.client.0.smithi089.stdout:removed './154/31' 2023-10-30T12:34:36.380 INFO:tasks.workunit.client.0.smithi089.stdout:removed './154/36' 2023-10-30T12:34:36.380 INFO:tasks.workunit.client.0.smithi089.stdout:removed './154/20' 2023-10-30T12:34:36.381 INFO:tasks.workunit.client.0.smithi089.stdout:removed './154/12' 2023-10-30T12:34:36.381 INFO:tasks.workunit.client.0.smithi089.stdout:removed './154/39' 2023-10-30T12:34:36.381 INFO:tasks.workunit.client.0.smithi089.stdout:removed './154/10' 2023-10-30T12:34:36.381 INFO:tasks.workunit.client.0.smithi089.stdout:removed './154/34' 2023-10-30T12:34:36.382 INFO:tasks.workunit.client.0.smithi089.stdout:removed './154/38' 2023-10-30T12:34:36.382 INFO:tasks.workunit.client.0.smithi089.stdout:removed './154/11' 2023-10-30T12:34:36.382 INFO:tasks.workunit.client.0.smithi089.stdout:removed './154/16' 2023-10-30T12:34:36.382 INFO:tasks.workunit.client.0.smithi089.stdout:removed './154/32' 2023-10-30T12:34:36.382 INFO:tasks.workunit.client.0.smithi089.stdout:removed './154/30' 2023-10-30T12:34:36.383 INFO:tasks.workunit.client.0.smithi089.stdout:removed './154/14' 2023-10-30T12:34:36.383 INFO:tasks.workunit.client.0.smithi089.stdout:removed './154/18' 2023-10-30T12:34:36.383 INFO:tasks.workunit.client.0.smithi089.stdout:removed './154/13' 2023-10-30T12:34:36.383 INFO:tasks.workunit.client.0.smithi089.stdout:removed './154/33' 2023-10-30T12:34:36.383 INFO:tasks.workunit.client.0.smithi089.stdout:removed './154/17' 2023-10-30T12:34:36.384 INFO:tasks.workunit.client.0.smithi089.stdout:removed './154/35' 2023-10-30T12:34:36.384 INFO:tasks.workunit.client.0.smithi089.stdout:removed './154/37' 2023-10-30T12:34:36.384 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './154' 2023-10-30T12:34:36.384 INFO:tasks.workunit.client.0.smithi089.stdout:removed './155/19' 2023-10-30T12:34:36.385 INFO:tasks.workunit.client.0.smithi089.stdout:removed './155/15' 2023-10-30T12:34:36.385 INFO:tasks.workunit.client.0.smithi089.stdout:removed './155/31' 2023-10-30T12:34:36.385 INFO:tasks.workunit.client.0.smithi089.stdout:removed './155/36' 2023-10-30T12:34:36.385 INFO:tasks.workunit.client.0.smithi089.stdout:removed './155/20' 2023-10-30T12:34:36.385 INFO:tasks.workunit.client.0.smithi089.stdout:removed './155/12' 2023-10-30T12:34:36.386 INFO:tasks.workunit.client.0.smithi089.stdout:removed './155/39' 2023-10-30T12:34:36.386 INFO:tasks.workunit.client.0.smithi089.stdout:removed './155/10' 2023-10-30T12:34:36.386 INFO:tasks.workunit.client.0.smithi089.stdout:removed './155/34' 2023-10-30T12:34:36.386 INFO:tasks.workunit.client.0.smithi089.stdout:removed './155/38' 2023-10-30T12:34:36.387 INFO:tasks.workunit.client.0.smithi089.stdout:removed './155/11' 2023-10-30T12:34:36.387 INFO:tasks.workunit.client.0.smithi089.stdout:removed './155/16' 2023-10-30T12:34:36.387 INFO:tasks.workunit.client.0.smithi089.stdout:removed './155/32' 2023-10-30T12:34:36.387 INFO:tasks.workunit.client.0.smithi089.stdout:removed './155/30' 2023-10-30T12:34:36.388 INFO:tasks.workunit.client.0.smithi089.stdout:removed './155/14' 2023-10-30T12:34:36.388 INFO:tasks.workunit.client.0.smithi089.stdout:removed './155/18' 2023-10-30T12:34:36.388 INFO:tasks.workunit.client.0.smithi089.stdout:removed './155/13' 2023-10-30T12:34:36.388 INFO:tasks.workunit.client.0.smithi089.stdout:removed './155/33' 2023-10-30T12:34:36.389 INFO:tasks.workunit.client.0.smithi089.stdout:removed './155/17' 2023-10-30T12:34:36.389 INFO:tasks.workunit.client.0.smithi089.stdout:removed './155/35' 2023-10-30T12:34:36.389 INFO:tasks.workunit.client.0.smithi089.stdout:removed './155/37' 2023-10-30T12:34:36.389 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './155' 2023-10-30T12:34:36.389 INFO:tasks.workunit.client.0.smithi089.stdout:removed './156/19' 2023-10-30T12:34:36.390 INFO:tasks.workunit.client.0.smithi089.stdout:removed './156/15' 2023-10-30T12:34:36.390 INFO:tasks.workunit.client.0.smithi089.stdout:removed './156/31' 2023-10-30T12:34:36.390 INFO:tasks.workunit.client.0.smithi089.stdout:removed './156/36' 2023-10-30T12:34:36.390 INFO:tasks.workunit.client.0.smithi089.stdout:removed './156/20' 2023-10-30T12:34:36.390 INFO:tasks.workunit.client.0.smithi089.stdout:removed './156/12' 2023-10-30T12:34:36.391 INFO:tasks.workunit.client.0.smithi089.stdout:removed './156/39' 2023-10-30T12:34:36.391 INFO:tasks.workunit.client.0.smithi089.stdout:removed './156/10' 2023-10-30T12:34:36.391 INFO:tasks.workunit.client.0.smithi089.stdout:removed './156/34' 2023-10-30T12:34:36.391 INFO:tasks.workunit.client.0.smithi089.stdout:removed './156/38' 2023-10-30T12:34:36.392 INFO:tasks.workunit.client.0.smithi089.stdout:removed './156/11' 2023-10-30T12:34:36.392 INFO:tasks.workunit.client.0.smithi089.stdout:removed './156/16' 2023-10-30T12:34:36.392 INFO:tasks.workunit.client.0.smithi089.stdout:removed './156/32' 2023-10-30T12:34:36.393 INFO:tasks.workunit.client.0.smithi089.stdout:removed './156/30' 2023-10-30T12:34:36.393 INFO:tasks.workunit.client.0.smithi089.stdout:removed './156/14' 2023-10-30T12:34:36.393 INFO:tasks.workunit.client.0.smithi089.stdout:removed './156/18' 2023-10-30T12:34:36.393 INFO:tasks.workunit.client.0.smithi089.stdout:removed './156/13' 2023-10-30T12:34:36.394 INFO:tasks.workunit.client.0.smithi089.stdout:removed './156/33' 2023-10-30T12:34:36.394 INFO:tasks.workunit.client.0.smithi089.stdout:removed './156/17' 2023-10-30T12:34:36.394 INFO:tasks.workunit.client.0.smithi089.stdout:removed './156/35' 2023-10-30T12:34:36.394 INFO:tasks.workunit.client.0.smithi089.stdout:removed './156/37' 2023-10-30T12:34:36.394 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './156' 2023-10-30T12:34:36.395 INFO:tasks.workunit.client.0.smithi089.stdout:removed './157/19' 2023-10-30T12:34:36.395 INFO:tasks.workunit.client.0.smithi089.stdout:removed './157/15' 2023-10-30T12:34:36.395 INFO:tasks.workunit.client.0.smithi089.stdout:removed './157/31' 2023-10-30T12:34:36.395 INFO:tasks.workunit.client.0.smithi089.stdout:removed './157/36' 2023-10-30T12:34:36.396 INFO:tasks.workunit.client.0.smithi089.stdout:removed './157/20' 2023-10-30T12:34:36.396 INFO:tasks.workunit.client.0.smithi089.stdout:removed './157/12' 2023-10-30T12:34:36.396 INFO:tasks.workunit.client.0.smithi089.stdout:removed './157/39' 2023-10-30T12:34:36.396 INFO:tasks.workunit.client.0.smithi089.stdout:removed './157/10' 2023-10-30T12:34:36.396 INFO:tasks.workunit.client.0.smithi089.stdout:removed './157/34' 2023-10-30T12:34:36.397 INFO:tasks.workunit.client.0.smithi089.stdout:removed './157/38' 2023-10-30T12:34:36.397 INFO:tasks.workunit.client.0.smithi089.stdout:removed './157/11' 2023-10-30T12:34:36.397 INFO:tasks.workunit.client.0.smithi089.stdout:removed './157/16' 2023-10-30T12:34:36.397 INFO:tasks.workunit.client.0.smithi089.stdout:removed './157/32' 2023-10-30T12:34:36.397 INFO:tasks.workunit.client.0.smithi089.stdout:removed './157/30' 2023-10-30T12:34:36.398 INFO:tasks.workunit.client.0.smithi089.stdout:removed './157/14' 2023-10-30T12:34:36.398 INFO:tasks.workunit.client.0.smithi089.stdout:removed './157/18' 2023-10-30T12:34:36.398 INFO:tasks.workunit.client.0.smithi089.stdout:removed './157/13' 2023-10-30T12:34:36.398 INFO:tasks.workunit.client.0.smithi089.stdout:removed './157/33' 2023-10-30T12:34:40.294 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:34:40.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.436+0000 7f9a1b99d640 1 Processor -- start 2023-10-30T12:34:40.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.436+0000 7f9a1b99d640 1 -- start start 2023-10-30T12:34:40.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a1b99d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a1406f7a0 0x7f9a1406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:40.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a1b99d640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a14066cf0 con 0x7f9a140c6bd0 2023-10-30T12:34:40.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a1b99d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a140c70d0 con 0x7f9a14071110 2023-10-30T12:34:40.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a1b99d640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a140700c0 con 0x7f9a1406f7a0 2023-10-30T12:34:40.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a18f38640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9a140c6bd0 0x7f9a14069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51954/0 (socket says 172.21.15.89:51954) 2023-10-30T12:34:40.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a18f38640 1 -- 172.21.15.89:0/1058333953 learned_addr learned my addr 172.21.15.89:0/1058333953 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:34:40.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a19739640 1 --2- 172.21.15.89:0/1058333953 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a1406f7a0 0x7f9a1406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:40.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a0bfff640 1 -- 172.21.15.89:0/1058333953 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267615960 0 0) 0x7f9a14066cf0 con 0x7f9a140c6bd0 2023-10-30T12:34:40.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a0bfff640 1 -- 172.21.15.89:0/1058333953 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a00003530 con 0x7f9a140c6bd0 2023-10-30T12:34:40.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a0bfff640 1 -- 172.21.15.89:0/1058333953 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3686897806 0 0) 0x7f9a140c70d0 con 0x7f9a14071110 2023-10-30T12:34:40.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a0bfff640 1 -- 172.21.15.89:0/1058333953 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a14066cf0 con 0x7f9a14071110 2023-10-30T12:34:40.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a0bfff640 1 -- 172.21.15.89:0/1058333953 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2481274300 0 0) 0x7f9a00003530 con 0x7f9a140c6bd0 2023-10-30T12:34:40.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a0bfff640 1 -- 172.21.15.89:0/1058333953 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a1406f7a0 msgr2=0x7f9a1406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:34:40.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a0bfff640 1 --2- 172.21.15.89:0/1058333953 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a1406f7a0 0x7f9a1406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:40.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a0bfff640 1 -- 172.21.15.89:0/1058333953 >> v1:172.21.15.192:6789/0 conn(0x7f9a14071110 legacy=0x7f9a14158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:40.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a0bfff640 1 -- 172.21.15.89:0/1058333953 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a140c7390 con 0x7f9a140c6bd0 2023-10-30T12:34:40.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a0bfff640 1 -- 172.21.15.89:0/1058333953 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f9a04001680 con 0x7f9a140c6bd0 2023-10-30T12:34:40.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a0bfff640 1 -- 172.21.15.89:0/1058333953 >> v1:172.21.15.89:6789/0 conn(0x7f9a140c6bd0 legacy=0x7f9a14069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:40.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a0bfff640 1 --2- 172.21.15.89:0/1058333953 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a1406f7a0 0x7f9a00003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:40.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a0bfff640 1 --2- 172.21.15.89:0/1058333953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a000040a0 0x7f9a000047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:40.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a0bfff640 1 --2- 172.21.15.89:0/1058333953 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a00004d10 0x7f9a00005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:40.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a0bfff640 1 -- 172.21.15.89:0/1058333953 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a140700c0 con 0x7f9a000040a0 2023-10-30T12:34:40.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a0bfff640 1 -- 172.21.15.89:0/1058333953 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0000ba70 con 0x7f9a00004d10 2023-10-30T12:34:40.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a0bfff640 1 -- 172.21.15.89:0/1058333953 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0000bc30 con 0x7f9a1406f7a0 2023-10-30T12:34:40.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a18f38640 1 --2- 172.21.15.89:0/1058333953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a000040a0 0x7f9a000047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:40.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a19739640 1 --2- 172.21.15.89:0/1058333953 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a1406f7a0 0x7f9a00003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:40.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a19f3a640 1 --2- 172.21.15.89:0/1058333953 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a00004d10 0x7f9a00005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:40.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a18f38640 1 -- 172.21.15.89:0/1058333953 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a1406f7a0 msgr2=0x7f9a00003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:34:40.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a18f38640 1 --2- 172.21.15.89:0/1058333953 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a1406f7a0 0x7f9a00003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:40.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a18f38640 1 -- 172.21.15.89:0/1058333953 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a00004d10 msgr2=0x7f9a00005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:34:40.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a18f38640 1 --2- 172.21.15.89:0/1058333953 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a00004d10 0x7f9a00005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:40.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.440+0000 7f9a18f38640 1 -- 172.21.15.89:0/1058333953 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a00003530 con 0x7f9a000040a0 2023-10-30T12:34:40.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.444+0000 7f9a18f38640 1 --2- 172.21.15.89:0/1058333953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a000040a0 0x7f9a000047d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f9a0400c160 tx=0x7f9a0401f310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b811a1cb3ae6b153 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:40.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.444+0000 7f9a0bfff640 1 -- 172.21.15.89:0/1058333953 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a04005ae0 con 0x7f9a000040a0 2023-10-30T12:34:40.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.444+0000 7f9a0bfff640 1 -- 172.21.15.89:0/1058333953 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a040ab070 con 0x7f9a000040a0 2023-10-30T12:34:40.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.444+0000 7f9a0bfff640 1 -- 172.21.15.89:0/1058333953 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a04007070 con 0x7f9a000040a0 2023-10-30T12:34:40.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.444+0000 7f9a1b99d640 1 -- 172.21.15.89:0/1058333953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a000040a0 msgr2=0x7f9a000047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:40.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.444+0000 7f9a1b99d640 1 --2- 172.21.15.89:0/1058333953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a000040a0 0x7f9a000047d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f9a0400c160 tx=0x7f9a0401f310 comp rx=0 tx=0).stop 2023-10-30T12:34:40.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.444+0000 7f9a19f3a640 1 -- 172.21.15.89:0/1058333953 reap_dead start 2023-10-30T12:34:40.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.444+0000 7f9a1b99d640 1 -- 172.21.15.89:0/1058333953 shutdown_connections 2023-10-30T12:34:40.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.444+0000 7f9a1b99d640 1 -- 172.21.15.89:0/1058333953 >> 172.21.15.89:0/1058333953 conn(0x7f9a140c7910 msgr2=0x7f9a140c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:34:40.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.444+0000 7f9a1b99d640 1 -- 172.21.15.89:0/1058333953 shutdown_connections 2023-10-30T12:34:40.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.444+0000 7f9a1b99d640 1 -- 172.21.15.89:0/1058333953 wait complete. 2023-10-30T12:34:40.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.448+0000 7f9a1b99d640 1 Processor -- start 2023-10-30T12:34:40.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.448+0000 7f9a1b99d640 1 -- start start 2023-10-30T12:34:40.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.448+0000 7f9a1b99d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a000040a0 0x7f9a1414d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:40.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.448+0000 7f9a1b99d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a141f09b0 0x7f9a140c5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:40.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.448+0000 7f9a1b99d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a1414aea0 0x7f9a1414b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:40.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.448+0000 7f9a1b99d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a14144220 con 0x7f9a000040a0 2023-10-30T12:34:40.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.448+0000 7f9a1b99d640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1414aa10 con 0x7f9a141f09b0 2023-10-30T12:34:40.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.448+0000 7f9a1b99d640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1414cab0 con 0x7f9a1414aea0 2023-10-30T12:34:40.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.448+0000 7f9a19739640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a000040a0 0x7f9a1414d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:40.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.448+0000 7f9a19739640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a000040a0 0x7f9a1414d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55920/0 (socket says 172.21.15.89:55920) 2023-10-30T12:34:40.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.448+0000 7f9a19739640 1 -- 172.21.15.89:0/1554002844 learned_addr learned my addr 172.21.15.89:0/1554002844 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:34:40.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.448+0000 7f9a18f38640 1 --2- 172.21.15.89:0/1554002844 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a141f09b0 0x7f9a140c5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:40.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.448+0000 7f9a19f3a640 1 --2- 172.21.15.89:0/1554002844 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a1414aea0 0x7f9a1414b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:40.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.448+0000 7f9a19739640 1 -- 172.21.15.89:0/1554002844 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a1414aea0 msgr2=0x7f9a1414b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:40.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.448+0000 7f9a19739640 1 --2- 172.21.15.89:0/1554002844 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a1414aea0 0x7f9a1414b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:40.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.448+0000 7f9a19739640 1 -- 172.21.15.89:0/1554002844 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a141f09b0 msgr2=0x7f9a140c5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:40.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.448+0000 7f9a19739640 1 --2- 172.21.15.89:0/1554002844 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a141f09b0 0x7f9a140c5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:40.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.448+0000 7f9a19739640 1 -- 172.21.15.89:0/1554002844 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1414b840 con 0x7f9a000040a0 2023-10-30T12:34:40.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.448+0000 7f9a18f38640 1 --2- 172.21.15.89:0/1554002844 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a141f09b0 0x7f9a140c5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:34:40.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.448+0000 7f9a19739640 1 --2- 172.21.15.89:0/1554002844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a000040a0 0x7f9a1414d1b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9a100115e0 tx=0x7f9a10011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:40.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.448+0000 7f9a09ffb640 1 -- 172.21.15.89:0/1554002844 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1000f8f0 con 0x7f9a000040a0 2023-10-30T12:34:40.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.448+0000 7f9a09ffb640 1 -- 172.21.15.89:0/1554002844 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a1000fa50 con 0x7f9a000040a0 2023-10-30T12:34:40.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.448+0000 7f9a09ffb640 1 -- 172.21.15.89:0/1554002844 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1000fdc0 con 0x7f9a000040a0 2023-10-30T12:34:40.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.448+0000 7f9a1b99d640 1 -- 172.21.15.89:0/1554002844 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a1414d810 con 0x7f9a000040a0 2023-10-30T12:34:40.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.448+0000 7f9a1b99d640 1 -- 172.21.15.89:0/1554002844 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a141583a0 con 0x7f9a000040a0 2023-10-30T12:34:40.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.448+0000 7f9a1b99d640 1 -- 172.21.15.89:0/1554002844 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a141585c0 con 0x7f9a000040a0 2023-10-30T12:34:40.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.452+0000 7f9a09ffb640 1 -- 172.21.15.89:0/1554002844 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9a10015460 con 0x7f9a000040a0 2023-10-30T12:34:40.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.452+0000 7f9a09ffb640 1 --2- 172.21.15.89:0/1554002844 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f99f80664d0 0x7f99f8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:40.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.452+0000 7f9a09ffb640 1 -- 172.21.15.89:0/1554002844 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f9a100846f0 con 0x7f9a000040a0 2023-10-30T12:34:40.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.452+0000 7f9a18f38640 1 --2- 172.21.15.89:0/1554002844 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f99f80664d0 0x7f99f8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:40.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.452+0000 7f9a18f38640 1 --2- 172.21.15.89:0/1554002844 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f99f80664d0 0x7f99f8068990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9a040ae000 tx=0x7f9a04001a00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:40.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.456+0000 7f9a09ffb640 1 -- 172.21.15.89:0/1554002844 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9a10051740 con 0x7f9a000040a0 2023-10-30T12:34:40.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.640+0000 7f9a1b99d640 1 -- 172.21.15.89:0/1554002844 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a140708e0 con 0x7f9a000040a0 2023-10-30T12:34:40.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.640+0000 7f9a09ffb640 1 -- 172.21.15.89:0/1554002844 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+6534 (secure 0 0 0) 0x7f9a10054d60 con 0x7f9a000040a0 2023-10-30T12:34:40.646 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:34:40.646 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5139,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:34:11.946671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:34:40.647 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 32 2023-10-30T12:34:40.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.644+0000 7f9a1b99d640 1 -- 172.21.15.89:0/1554002844 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f99f80664d0 msgr2=0x7f99f8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:40.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.644+0000 7f9a1b99d640 1 --2- 172.21.15.89:0/1554002844 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f99f80664d0 0x7f99f8068990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9a040ae000 tx=0x7f9a04001a00 comp rx=0 tx=0).stop 2023-10-30T12:34:40.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.644+0000 7f9a1b99d640 1 -- 172.21.15.89:0/1554002844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a000040a0 msgr2=0x7f9a1414d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:40.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.644+0000 7f9a1b99d640 1 --2- 172.21.15.89:0/1554002844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a000040a0 0x7f9a1414d1b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9a100115e0 tx=0x7f9a10011aa0 comp rx=0 tx=0).stop 2023-10-30T12:34:40.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.644+0000 7f9a1b99d640 1 -- 172.21.15.89:0/1554002844 shutdown_connections 2023-10-30T12:34:40.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.644+0000 7f9a1b99d640 1 --2- 172.21.15.89:0/1554002844 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f99f80664d0 0x7f99f8068990 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:40.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.644+0000 7f9a1b99d640 1 --2- 172.21.15.89:0/1554002844 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a141f09b0 0x7f9a140c5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:40.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.644+0000 7f9a1b99d640 1 --2- 172.21.15.89:0/1554002844 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a1414aea0 0x7f9a1414b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:40.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.644+0000 7f9a1b99d640 1 --2- 172.21.15.89:0/1554002844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a000040a0 0x7f9a1414d1b0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:40.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.644+0000 7f9a1b99d640 1 -- 172.21.15.89:0/1554002844 >> 172.21.15.89:0/1554002844 conn(0x7f9a140c7910 msgr2=0x7f9a140c2840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:34:40.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.644+0000 7f9a1b99d640 1 -- 172.21.15.89:0/1554002844 shutdown_connections 2023-10-30T12:34:40.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.644+0000 7f9a1b99d640 1 -- 172.21.15.89:0/1554002844 wait complete. 2023-10-30T12:34:40.671 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=1) 2023-10-30T12:34:40.672 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-30T12:34:40.672 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:34:40.692 INFO:tasks.ceph.mds.d:Stopped 2023-10-30T12:34:40.692 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:34:40.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.824+0000 7fcf03165640 1 Processor -- start 2023-10-30T12:34:40.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.824+0000 7fcf03165640 1 -- start start 2023-10-30T12:34:40.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.824+0000 7fcf03165640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcefc0c6b80 0x7fcefc0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:40.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.824+0000 7fcf03165640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcefc0c0a50 con 0x7fcefc0c6f60 2023-10-30T12:34:40.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.824+0000 7fcf03165640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcefc06f6e0 con 0x7fcefc13f4a0 2023-10-30T12:34:40.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.824+0000 7fcf03165640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcefc06f9c0 con 0x7fcefc0c6b80 2023-10-30T12:34:40.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.824+0000 7fcf00f01640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcefc0c6b80 0x7fcefc0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:40.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.824+0000 7fcef3fff640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fcefc0c6f60 0x7fcefc13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51966/0 (socket says 172.21.15.89:51966) 2023-10-30T12:34:40.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.824+0000 7fcef3fff640 1 -- 172.21.15.89:0/1604807609 learned_addr learned my addr 172.21.15.89:0/1604807609 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:34:40.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.824+0000 7fcef37fe640 1 -- 172.21.15.89:0/1604807609 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839152365 0 0) 0x7fcefc0c0a50 con 0x7fcefc0c6f60 2023-10-30T12:34:40.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.824+0000 7fcef37fe640 1 -- 172.21.15.89:0/1604807609 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcee4005740 con 0x7fcefc0c6f60 2023-10-30T12:34:40.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.824+0000 7fcf00f01640 1 -- 172.21.15.89:0/1604807609 >> v1:172.21.15.192:6789/0 conn(0x7fcefc13f4a0 legacy=0x7fcefc1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:40.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.824+0000 7fcf00f01640 1 -- 172.21.15.89:0/1604807609 >> v1:172.21.15.89:6789/0 conn(0x7fcefc0c6f60 legacy=0x7fcefc13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:40.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.824+0000 7fcf00f01640 1 -- 172.21.15.89:0/1604807609 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcefc06fca0 con 0x7fcefc0c6b80 2023-10-30T12:34:40.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.828+0000 7fcf00f01640 1 --2- 172.21.15.89:0/1604807609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcefc0c6b80 0x7fcefc0bfec0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fcef80097c0 tx=0x7fcef801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a8ef03a72400f5d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:40.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.828+0000 7fcef37fe640 1 -- 172.21.15.89:0/1604807609 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef80ab600 con 0x7fcefc0c6b80 2023-10-30T12:34:40.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.828+0000 7fcef37fe640 1 -- 172.21.15.89:0/1604807609 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef80a7050 con 0x7fcefc0c6b80 2023-10-30T12:34:40.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.828+0000 7fcef37fe640 1 -- 172.21.15.89:0/1604807609 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef80ab980 con 0x7fcefc0c6b80 2023-10-30T12:34:40.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.828+0000 7fcf03165640 1 -- 172.21.15.89:0/1604807609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcefc0c6b80 msgr2=0x7fcefc0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:40.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.828+0000 7fcf03165640 1 --2- 172.21.15.89:0/1604807609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcefc0c6b80 0x7fcefc0bfec0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fcef80097c0 tx=0x7fcef801c480 comp rx=0 tx=0).stop 2023-10-30T12:34:40.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.828+0000 7fcf03165640 1 -- 172.21.15.89:0/1604807609 shutdown_connections 2023-10-30T12:34:40.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.828+0000 7fcf03165640 1 --2- 172.21.15.89:0/1604807609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcefc0c6b80 0x7fcefc0bfec0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:40.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.828+0000 7fcf03165640 1 -- 172.21.15.89:0/1604807609 >> 172.21.15.89:0/1604807609 conn(0x7fcefc0c7910 msgr2=0x7fcefc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:34:40.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.828+0000 7fcf03165640 1 -- 172.21.15.89:0/1604807609 shutdown_connections 2023-10-30T12:34:40.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.828+0000 7fcf03165640 1 -- 172.21.15.89:0/1604807609 wait complete. 2023-10-30T12:34:40.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.832+0000 7fcf03165640 1 Processor -- start 2023-10-30T12:34:40.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.832+0000 7fcf03165640 1 -- start start 2023-10-30T12:34:40.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.832+0000 7fcf03165640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcefc0c6b80 0x7fcefc150d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:40.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.832+0000 7fcf03165640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcefc0c6f60 0x7fcefc0c9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:40.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.832+0000 7fcf03165640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcefc13f4a0 0x7fcefc0ca3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:40.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.832+0000 7fcf03165640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc155d80 con 0x7fcefc0c6f60 2023-10-30T12:34:40.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.832+0000 7fcf03165640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc0c90c0 con 0x7fcefc0c6b80 2023-10-30T12:34:40.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.832+0000 7fcf03165640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc0c93a0 con 0x7fcefc13f4a0 2023-10-30T12:34:40.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.832+0000 7fcef3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcefc0c6f60 0x7fcefc0c9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:40.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.832+0000 7fcef3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcefc0c6f60 0x7fcefc0c9e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55930/0 (socket says 172.21.15.89:55930) 2023-10-30T12:34:40.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.832+0000 7fcef3fff640 1 -- 172.21.15.89:0/413755547 learned_addr learned my addr 172.21.15.89:0/413755547 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:34:40.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.832+0000 7fcf00f01640 1 --2- 172.21.15.89:0/413755547 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcefc0c6b80 0x7fcefc150d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:40.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.832+0000 7fcf01702640 1 --2- 172.21.15.89:0/413755547 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcefc13f4a0 0x7fcefc0ca3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:40.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.832+0000 7fcef3fff640 1 -- 172.21.15.89:0/413755547 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcefc13f4a0 msgr2=0x7fcefc0ca3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:40.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.832+0000 7fcef3fff640 1 --2- 172.21.15.89:0/413755547 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcefc13f4a0 0x7fcefc0ca3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:40.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.832+0000 7fcef3fff640 1 -- 172.21.15.89:0/413755547 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcefc0c6b80 msgr2=0x7fcefc150d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:40.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.832+0000 7fcef3fff640 1 --2- 172.21.15.89:0/413755547 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcefc0c6b80 0x7fcefc150d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:40.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.832+0000 7fcef3fff640 1 -- 172.21.15.89:0/413755547 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef8009420 con 0x7fcefc0c6f60 2023-10-30T12:34:40.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.832+0000 7fcf01702640 1 --2- 172.21.15.89:0/413755547 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcefc13f4a0 0x7fcefc0ca3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:34:40.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.832+0000 7fcef3fff640 1 --2- 172.21.15.89:0/413755547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcefc0c6f60 0x7fcefc0c9e80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fceec00c4d0 tx=0x7fceec00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:40.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.832+0000 7fcef17fa640 1 -- 172.21.15.89:0/413755547 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceec005cf0 con 0x7fcefc0c6f60 2023-10-30T12:34:40.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.832+0000 7fcef17fa640 1 -- 172.21.15.89:0/413755547 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceec005e50 con 0x7fcefc0c6f60 2023-10-30T12:34:40.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.832+0000 7fcef17fa640 1 -- 172.21.15.89:0/413755547 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceec007270 con 0x7fcefc0c6f60 2023-10-30T12:34:40.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.832+0000 7fcf03165640 1 -- 172.21.15.89:0/413755547 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcefc0c9680 con 0x7fcefc0c6f60 2023-10-30T12:34:40.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.832+0000 7fcf03165640 1 -- 172.21.15.89:0/413755547 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcefc151a10 con 0x7fcefc0c6f60 2023-10-30T12:34:40.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.836+0000 7fcf03165640 1 -- 172.21.15.89:0/413755547 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcefc04e170 con 0x7fcefc0c6f60 2023-10-30T12:34:40.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.836+0000 7fcef17fa640 1 -- 172.21.15.89:0/413755547 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fceec0073d0 con 0x7fcefc0c6f60 2023-10-30T12:34:40.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.836+0000 7fcef17fa640 1 --2- 172.21.15.89:0/413755547 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fcee80664a0 0x7fcee8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:40.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.836+0000 7fcef17fa640 1 -- 172.21.15.89:0/413755547 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6468+0+0 (secure 0 0 0) 0x7fceec083770 con 0x7fcefc0c6f60 2023-10-30T12:34:40.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.836+0000 7fcf00f01640 1 --2- 172.21.15.89:0/413755547 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fcee80664a0 0x7fcee8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:40.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.836+0000 7fcf00f01640 1 --2- 172.21.15.89:0/413755547 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fcee80664a0 0x7fcee8068960 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fcef80ad000 tx=0x7fcef8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:40.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:40.840+0000 7fcef17fa640 1 -- 172.21.15.89:0/413755547 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fceec050870 con 0x7fcefc0c6f60 2023-10-30T12:34:41.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:41.020+0000 7fcf03165640 1 -- 172.21.15.89:0/413755547 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcefc0c9ba0 con 0x7fcefc0c6f60 2023-10-30T12:34:41.027 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:34:41.027 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5139,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:34:11.946671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:34:41.027 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 32 2023-10-30T12:34:41.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:41.020+0000 7fcef17fa640 1 -- 172.21.15.89:0/413755547 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+6534 (secure 0 0 0) 0x7fceec017090 con 0x7fcefc0c6f60 2023-10-30T12:34:41.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:41.024+0000 7fcf03165640 1 -- 172.21.15.89:0/413755547 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fcee80664a0 msgr2=0x7fcee8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:41.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:41.024+0000 7fcf03165640 1 --2- 172.21.15.89:0/413755547 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fcee80664a0 0x7fcee8068960 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fcef80ad000 tx=0x7fcef8008040 comp rx=0 tx=0).stop 2023-10-30T12:34:41.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:41.024+0000 7fcf03165640 1 -- 172.21.15.89:0/413755547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcefc0c6f60 msgr2=0x7fcefc0c9e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:41.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:41.024+0000 7fcf03165640 1 --2- 172.21.15.89:0/413755547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcefc0c6f60 0x7fcefc0c9e80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fceec00c4d0 tx=0x7fceec00c990 comp rx=0 tx=0).stop 2023-10-30T12:34:41.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:41.024+0000 7fcf03165640 1 -- 172.21.15.89:0/413755547 shutdown_connections 2023-10-30T12:34:41.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:41.024+0000 7fcf03165640 1 --2- 172.21.15.89:0/413755547 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fcee80664a0 0x7fcee8068960 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:41.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:41.024+0000 7fcf03165640 1 --2- 172.21.15.89:0/413755547 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcefc13f4a0 0x7fcefc0ca3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:41.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:41.024+0000 7fcf03165640 1 --2- 172.21.15.89:0/413755547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcefc0c6f60 0x7fcefc0c9e80 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:41.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:41.024+0000 7fcf03165640 1 --2- 172.21.15.89:0/413755547 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcefc0c6b80 0x7fcefc150d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:41.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:41.024+0000 7fcf03165640 1 -- 172.21.15.89:0/413755547 >> 172.21.15.89:0/413755547 conn(0x7fcefc0c7910 msgr2=0x7fcefc139760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:34:41.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:41.024+0000 7fcf03165640 1 -- 172.21.15.89:0/413755547 shutdown_connections 2023-10-30T12:34:41.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:41.024+0000 7fcf03165640 1 -- 172.21.15.89:0/413755547 wait complete. 2023-10-30T12:34:41.116 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T12:34:43.118 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:34:43.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.264+0000 7f033f980640 1 Processor -- start 2023-10-30T12:34:43.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.264+0000 7f033f980640 1 -- start start 2023-10-30T12:34:43.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.264+0000 7f033f980640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0338153bf0 0x7f0338071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:43.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.264+0000 7f033f980640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f03380c6fe0 0x7f03380c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:43.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.264+0000 7f033f980640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03380700b0 con 0x7f03380c79e0 2023-10-30T12:34:43.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.264+0000 7f033f980640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03380c6810 con 0x7f0338153bf0 2023-10-30T12:34:43.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.264+0000 7f033f980640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0338159900 con 0x7f03380c6fe0 2023-10-30T12:34:43.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.264+0000 7f033d71c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0338153bf0 0x7f0338071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:43.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.264+0000 7f033d71c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0338153bf0 0x7f0338071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55932/0 (socket says 172.21.15.89:55932) 2023-10-30T12:34:43.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.264+0000 7f033d71c640 1 -- 172.21.15.89:0/1435658351 learned_addr learned my addr 172.21.15.89:0/1435658351 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:34:43.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.264+0000 7f033df1d640 1 --1- 172.21.15.89:0/1435658351 >> v1:172.21.15.89:6789/0 conn(0x7f03380c79e0 0x7f03380cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51980/0 (socket says 172.21.15.89:51980) 2023-10-30T12:34:43.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.264+0000 7f033cf1b640 1 --2- 172.21.15.89:0/1435658351 >> v2:172.21.15.192:3300/0 conn(0x7f03380c6fe0 0x7f03380c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:43.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.264+0000 7f033d71c640 1 -- 172.21.15.89:0/1435658351 >> v2:172.21.15.192:3300/0 conn(0x7f03380c6fe0 msgr2=0x7f03380c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:34:43.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.264+0000 7f033d71c640 1 --2- 172.21.15.89:0/1435658351 >> v2:172.21.15.192:3300/0 conn(0x7f03380c6fe0 0x7f03380c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:43.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.264+0000 7f033d71c640 1 -- 172.21.15.89:0/1435658351 >> v1:172.21.15.89:6789/0 conn(0x7f03380c79e0 legacy=0x7f03380cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:43.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.264+0000 7f033d71c640 1 -- 172.21.15.89:0/1435658351 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03380c6530 con 0x7f0338153bf0 2023-10-30T12:34:43.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.264+0000 7f032ffff640 1 -- 172.21.15.89:0/1435658351 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2249903800 0 0) 0x7f03380700b0 con 0x7f03380c79e0 2023-10-30T12:34:43.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.268+0000 7f033d71c640 1 --2- 172.21.15.89:0/1435658351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0338153bf0 0x7f0338071260 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f03380723d0 tx=0x7f033401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4471ceca05b87132 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:43.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.268+0000 7f032ffff640 1 -- 172.21.15.89:0/1435658351 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03340a89f0 con 0x7f0338153bf0 2023-10-30T12:34:43.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.268+0000 7f032ffff640 1 -- 172.21.15.89:0/1435658351 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03340a8b50 con 0x7f0338153bf0 2023-10-30T12:34:43.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.268+0000 7f032ffff640 1 -- 172.21.15.89:0/1435658351 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03340b0440 con 0x7f0338153bf0 2023-10-30T12:34:43.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.268+0000 7f033f980640 1 -- 172.21.15.89:0/1435658351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0338153bf0 msgr2=0x7f0338071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:43.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.268+0000 7f033f980640 1 --2- 172.21.15.89:0/1435658351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0338153bf0 0x7f0338071260 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f03380723d0 tx=0x7f033401ecf0 comp rx=0 tx=0).stop 2023-10-30T12:34:43.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.268+0000 7f033f980640 1 -- 172.21.15.89:0/1435658351 shutdown_connections 2023-10-30T12:34:43.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.268+0000 7f033f980640 1 --2- 172.21.15.89:0/1435658351 >> v2:172.21.15.192:3300/0 conn(0x7f03380c6fe0 0x7f03380c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:43.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.268+0000 7f033f980640 1 --2- 172.21.15.89:0/1435658351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0338153bf0 0x7f0338071260 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:43.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.268+0000 7f033f980640 1 -- 172.21.15.89:0/1435658351 >> 172.21.15.89:0/1435658351 conn(0x7f033806c6e0 msgr2=0x7f0338069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:34:43.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.268+0000 7f033f980640 1 -- 172.21.15.89:0/1435658351 shutdown_connections 2023-10-30T12:34:43.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.268+0000 7f033f980640 1 -- 172.21.15.89:0/1435658351 wait complete. 2023-10-30T12:34:43.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.272+0000 7f033f980640 1 Processor -- start 2023-10-30T12:34:43.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.272+0000 7f033f980640 1 -- start start 2023-10-30T12:34:43.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.272+0000 7f033f980640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03380c6fe0 0x7f03380c5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:43.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.272+0000 7f033f980640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03380c79e0 0x7f03380c56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:43.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.272+0000 7f033f980640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0338153bf0 0x7f0338155540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:43.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.272+0000 7f033f980640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03380c3150 con 0x7f0338153bf0 2023-10-30T12:34:43.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.272+0000 7f033f980640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0338154ce0 con 0x7f03380c79e0 2023-10-30T12:34:43.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.272+0000 7f033f980640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0338155090 con 0x7f03380c6fe0 2023-10-30T12:34:43.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.272+0000 7f033df1d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0338153bf0 0x7f0338155540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:43.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.272+0000 7f033df1d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0338153bf0 0x7f0338155540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55934/0 (socket says 172.21.15.89:55934) 2023-10-30T12:34:43.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.272+0000 7f033df1d640 1 -- 172.21.15.89:0/3144873620 learned_addr learned my addr 172.21.15.89:0/3144873620 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:34:43.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.272+0000 7f033d71c640 1 --2- 172.21.15.89:0/3144873620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03380c6fe0 0x7f03380c5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:43.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.272+0000 7f033cf1b640 1 --2- 172.21.15.89:0/3144873620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03380c79e0 0x7f03380c56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:43.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.272+0000 7f033df1d640 1 -- 172.21.15.89:0/3144873620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03380c6fe0 msgr2=0x7f03380c5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:43.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.272+0000 7f033df1d640 1 --2- 172.21.15.89:0/3144873620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03380c6fe0 0x7f03380c5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:43.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.272+0000 7f033df1d640 1 -- 172.21.15.89:0/3144873620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03380c79e0 msgr2=0x7f03380c56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:43.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.272+0000 7f033df1d640 1 --2- 172.21.15.89:0/3144873620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03380c79e0 0x7f03380c56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:43.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.272+0000 7f033df1d640 1 -- 172.21.15.89:0/3144873620 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03340094d0 con 0x7f0338153bf0 2023-10-30T12:34:43.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.272+0000 7f033cf1b640 1 --2- 172.21.15.89:0/3144873620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03380c79e0 0x7f03380c56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:34:43.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.272+0000 7f033df1d640 1 --2- 172.21.15.89:0/3144873620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0338153bf0 0x7f0338155540 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f033000c810 tx=0x7f033000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:43.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.272+0000 7f032dffb640 1 -- 172.21.15.89:0/3144873620 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f033000a640 con 0x7f0338153bf0 2023-10-30T12:34:43.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.272+0000 7f032dffb640 1 -- 172.21.15.89:0/3144873620 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f033000a7c0 con 0x7f0338153bf0 2023-10-30T12:34:43.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.272+0000 7f033f980640 1 -- 172.21.15.89:0/3144873620 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03380c3380 con 0x7f0338153bf0 2023-10-30T12:34:43.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.272+0000 7f032dffb640 1 -- 172.21.15.89:0/3144873620 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f033000ab70 con 0x7f0338153bf0 2023-10-30T12:34:43.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.272+0000 7f033f980640 1 -- 172.21.15.89:0/3144873620 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03380c4330 con 0x7f0338153bf0 2023-10-30T12:34:43.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.272+0000 7f033f980640 1 -- 172.21.15.89:0/3144873620 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0338155260 con 0x7f0338153bf0 2023-10-30T12:34:43.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.276+0000 7f032dffb640 1 -- 172.21.15.89:0/3144873620 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f033000acd0 con 0x7f0338153bf0 2023-10-30T12:34:43.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.276+0000 7f032dffb640 1 --2- 172.21.15.89:0/3144873620 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f031c0664a0 0x7f031c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:43.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.276+0000 7f032dffb640 1 -- 172.21.15.89:0/3144873620 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f0330083830 con 0x7f0338153bf0 2023-10-30T12:34:43.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.276+0000 7f033d71c640 1 --2- 172.21.15.89:0/3144873620 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f031c0664a0 0x7f031c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:43.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.276+0000 7f033d71c640 1 --2- 172.21.15.89:0/3144873620 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f031c0664a0 0x7f031c068960 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f03340b2000 tx=0x7f0334008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:43.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.280+0000 7f032dffb640 1 -- 172.21.15.89:0/3144873620 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0330050880 con 0x7f0338153bf0 2023-10-30T12:34:43.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.460+0000 7f033f980640 1 -- 172.21.15.89:0/3144873620 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0338159dc0 con 0x7f0338153bf0 2023-10-30T12:34:43.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.460+0000 7f032dffb640 1 -- 172.21.15.89:0/3144873620 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+6534 (secure 0 0 0) 0x7f0330018090 con 0x7f0338153bf0 2023-10-30T12:34:43.466 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:34:43.466 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5139,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:34:11.946671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:34:43.468 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 32 2023-10-30T12:34:43.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.464+0000 7f033f980640 1 -- 172.21.15.89:0/3144873620 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f031c0664a0 msgr2=0x7f031c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:43.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.464+0000 7f033f980640 1 --2- 172.21.15.89:0/3144873620 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f031c0664a0 0x7f031c068960 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f03340b2000 tx=0x7f0334008040 comp rx=0 tx=0).stop 2023-10-30T12:34:43.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.464+0000 7f033f980640 1 -- 172.21.15.89:0/3144873620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0338153bf0 msgr2=0x7f0338155540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:43.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.464+0000 7f033f980640 1 --2- 172.21.15.89:0/3144873620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0338153bf0 0x7f0338155540 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f033000c810 tx=0x7f033000ccd0 comp rx=0 tx=0).stop 2023-10-30T12:34:43.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.464+0000 7f033f980640 1 -- 172.21.15.89:0/3144873620 shutdown_connections 2023-10-30T12:34:43.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.464+0000 7f033f980640 1 --2- 172.21.15.89:0/3144873620 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f031c0664a0 0x7f031c068960 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:43.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.464+0000 7f033f980640 1 --2- 172.21.15.89:0/3144873620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03380c79e0 0x7f03380c56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:43.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.464+0000 7f033f980640 1 --2- 172.21.15.89:0/3144873620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0338153bf0 0x7f0338155540 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:43.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.464+0000 7f033f980640 1 --2- 172.21.15.89:0/3144873620 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03380c6fe0 0x7f03380c5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:43.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.464+0000 7f033f980640 1 -- 172.21.15.89:0/3144873620 >> 172.21.15.89:0/3144873620 conn(0x7f033806c6e0 msgr2=0x7f0338069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:34:43.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.464+0000 7f033f980640 1 -- 172.21.15.89:0/3144873620 shutdown_connections 2023-10-30T12:34:43.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:43.464+0000 7f033f980640 1 -- 172.21.15.89:0/3144873620 wait complete. 2023-10-30T12:34:43.486 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T12:34:45.487 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:34:45.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.628+0000 7f912cb94640 1 Processor -- start 2023-10-30T12:34:45.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.632+0000 7f912cb94640 1 -- start start 2023-10-30T12:34:45.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.632+0000 7f912cb94640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f91280bfec0 0x7f912806f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:45.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.632+0000 7f912cb94640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9128069e90 0x7f912806a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:45.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.632+0000 7f912cb94640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f912806fa80 con 0x7f9128159e60 2023-10-30T12:34:45.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.632+0000 7f912cb94640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9128071290 con 0x7f91280bfec0 2023-10-30T12:34:45.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.632+0000 7f912cb94640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9128071570 con 0x7f9128069e90 2023-10-30T12:34:45.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.632+0000 7f912659c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f91280bfec0 0x7f912806f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:45.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.632+0000 7f912659c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f91280bfec0 0x7f912806f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55950/0 (socket says 172.21.15.89:55950) 2023-10-30T12:34:45.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.632+0000 7f912659c640 1 -- 172.21.15.89:0/3477062862 learned_addr learned my addr 172.21.15.89:0/3477062862 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:34:45.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.632+0000 7f911ffff640 1 --1- 172.21.15.89:0/3477062862 >> v1:172.21.15.89:6789/0 conn(0x7f9128159e60 0x7f91280c1a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51988/0 (socket says 172.21.15.89:51988) 2023-10-30T12:34:45.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.632+0000 7f9126d9d640 1 --2- 172.21.15.89:0/3477062862 >> v2:172.21.15.192:3300/0 conn(0x7f9128069e90 0x7f912806a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:45.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.632+0000 7f912659c640 1 -- 172.21.15.89:0/3477062862 >> v2:172.21.15.192:3300/0 conn(0x7f9128069e90 msgr2=0x7f912806a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:34:45.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.632+0000 7f912659c640 1 --2- 172.21.15.89:0/3477062862 >> v2:172.21.15.192:3300/0 conn(0x7f9128069e90 0x7f912806a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:45.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.632+0000 7f912659c640 1 -- 172.21.15.89:0/3477062862 >> v1:172.21.15.89:6789/0 conn(0x7f9128159e60 legacy=0x7f91280c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:45.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.632+0000 7f912659c640 1 -- 172.21.15.89:0/3477062862 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9128071850 con 0x7f91280bfec0 2023-10-30T12:34:45.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.636+0000 7f912659c640 1 --2- 172.21.15.89:0/3477062862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91280bfec0 0x7f912806f540 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f912806f4d0 tx=0x7f91180213c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7bc47cb952d40b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:45.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.636+0000 7f9125d9b640 1 -- 172.21.15.89:0/3477062862 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91180b1600 con 0x7f91280bfec0 2023-10-30T12:34:45.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.636+0000 7f9125d9b640 1 -- 172.21.15.89:0/3477062862 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91180ad050 con 0x7f91280bfec0 2023-10-30T12:34:45.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.636+0000 7f9125d9b640 1 -- 172.21.15.89:0/3477062862 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91180b1980 con 0x7f91280bfec0 2023-10-30T12:34:45.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.636+0000 7f912cb94640 1 -- 172.21.15.89:0/3477062862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91280bfec0 msgr2=0x7f912806f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:45.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.636+0000 7f912cb94640 1 --2- 172.21.15.89:0/3477062862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91280bfec0 0x7f912806f540 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f912806f4d0 tx=0x7f91180213c0 comp rx=0 tx=0).stop 2023-10-30T12:34:45.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.636+0000 7f912cb94640 1 -- 172.21.15.89:0/3477062862 shutdown_connections 2023-10-30T12:34:45.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.636+0000 7f912cb94640 1 --2- 172.21.15.89:0/3477062862 >> v2:172.21.15.192:3300/0 conn(0x7f9128069e90 0x7f912806a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:45.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.636+0000 7f912cb94640 1 --2- 172.21.15.89:0/3477062862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91280bfec0 0x7f912806f540 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:45.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.636+0000 7f912cb94640 1 -- 172.21.15.89:0/3477062862 >> 172.21.15.89:0/3477062862 conn(0x7f91280c7910 msgr2=0x7f91280c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:34:45.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.636+0000 7f912cb94640 1 -- 172.21.15.89:0/3477062862 shutdown_connections 2023-10-30T12:34:45.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.636+0000 7f912cb94640 1 -- 172.21.15.89:0/3477062862 wait complete. 2023-10-30T12:34:45.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.636+0000 7f912cb94640 1 Processor -- start 2023-10-30T12:34:45.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.636+0000 7f912cb94640 1 -- start start 2023-10-30T12:34:45.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.640+0000 7f912cb94640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9128069e90 0x7f91280a9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:45.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.640+0000 7f912cb94640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91280bfec0 0x7f91280c9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:45.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.640+0000 7f912cb94640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9128159e60 0x7f912814d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:45.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.640+0000 7f912cb94640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91281efdb0 con 0x7f9128159e60 2023-10-30T12:34:45.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.640+0000 7f912cb94640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91281ef130 con 0x7f9128069e90 2023-10-30T12:34:45.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.640+0000 7f912cb94640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91281ef410 con 0x7f91280bfec0 2023-10-30T12:34:45.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.640+0000 7f9126d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9128159e60 0x7f912814d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:45.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.640+0000 7f9126d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9128159e60 0x7f912814d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55962/0 (socket says 172.21.15.89:55962) 2023-10-30T12:34:45.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.640+0000 7f9126d9d640 1 -- 172.21.15.89:0/4009520311 learned_addr learned my addr 172.21.15.89:0/4009520311 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:34:45.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.640+0000 7f912659c640 1 --2- 172.21.15.89:0/4009520311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9128069e90 0x7f91280a9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:45.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.640+0000 7f911ffff640 1 --2- 172.21.15.89:0/4009520311 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91280bfec0 0x7f91280c9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:45.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.640+0000 7f9126d9d640 1 -- 172.21.15.89:0/4009520311 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91280bfec0 msgr2=0x7f91280c9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:45.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.640+0000 7f9126d9d640 1 --2- 172.21.15.89:0/4009520311 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91280bfec0 0x7f91280c9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:45.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.640+0000 7f9126d9d640 1 -- 172.21.15.89:0/4009520311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9128069e90 msgr2=0x7f91280a9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:45.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.640+0000 7f9126d9d640 1 --2- 172.21.15.89:0/4009520311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9128069e90 0x7f91280a9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:45.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.640+0000 7f9126d9d640 1 -- 172.21.15.89:0/4009520311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f911800b770 con 0x7f9128159e60 2023-10-30T12:34:45.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.640+0000 7f9126d9d640 1 --2- 172.21.15.89:0/4009520311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9128159e60 0x7f912814d450 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f9120007e40 tx=0x7f912000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:45.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.640+0000 7f911effd640 1 -- 172.21.15.89:0/4009520311 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9120014620 con 0x7f9128159e60 2023-10-30T12:34:45.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.640+0000 7f911effd640 1 -- 172.21.15.89:0/4009520311 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f912000ee90 con 0x7f9128159e60 2023-10-30T12:34:45.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.640+0000 7f912cb94640 1 -- 172.21.15.89:0/4009520311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91281ef6f0 con 0x7f9128159e60 2023-10-30T12:34:45.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.640+0000 7f911effd640 1 -- 172.21.15.89:0/4009520311 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91200149e0 con 0x7f9128159e60 2023-10-30T12:34:45.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.640+0000 7f912cb94640 1 -- 172.21.15.89:0/4009520311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91281eee40 con 0x7f9128159e60 2023-10-30T12:34:45.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.640+0000 7f912cb94640 1 -- 172.21.15.89:0/4009520311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f912804e170 con 0x7f9128159e60 2023-10-30T12:34:45.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.644+0000 7f911effd640 1 -- 172.21.15.89:0/4009520311 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9120014b40 con 0x7f9128159e60 2023-10-30T12:34:45.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.644+0000 7f911effd640 1 --2- 172.21.15.89:0/4009520311 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f91080664a0 0x7f9108068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:45.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.644+0000 7f911effd640 1 -- 172.21.15.89:0/4009520311 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f9120084b40 con 0x7f9128159e60 2023-10-30T12:34:45.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.644+0000 7f912659c640 1 --2- 172.21.15.89:0/4009520311 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f91080664a0 0x7f9108068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:45.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.644+0000 7f912659c640 1 --2- 172.21.15.89:0/4009520311 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f91080664a0 0x7f9108068960 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f91180b3000 tx=0x7f91180ac000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:45.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.648+0000 7f911effd640 1 -- 172.21.15.89:0/4009520311 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9120051c20 con 0x7f9128159e60 2023-10-30T12:34:45.720 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:34:45.724 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:34:45.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.828+0000 7f912cb94640 1 -- 172.21.15.89:0/4009520311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9128158690 con 0x7f9128159e60 2023-10-30T12:34:45.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.828+0000 7f911effd640 1 -- 172.21.15.89:0/4009520311 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+6534 (secure 0 0 0) 0x7f9120055240 con 0x7f9128159e60 2023-10-30T12:34:45.835 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:34:45.836 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5139,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:34:11.946671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:34:45.836 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 32 2023-10-30T12:34:45.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.832+0000 7f912cb94640 1 -- 172.21.15.89:0/4009520311 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f91080664a0 msgr2=0x7f9108068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:45.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.832+0000 7f912cb94640 1 --2- 172.21.15.89:0/4009520311 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f91080664a0 0x7f9108068960 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f91180b3000 tx=0x7f91180ac000 comp rx=0 tx=0).stop 2023-10-30T12:34:45.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.832+0000 7f912cb94640 1 -- 172.21.15.89:0/4009520311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9128159e60 msgr2=0x7f912814d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:45.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.832+0000 7f912cb94640 1 --2- 172.21.15.89:0/4009520311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9128159e60 0x7f912814d450 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f9120007e40 tx=0x7f912000e4b0 comp rx=0 tx=0).stop 2023-10-30T12:34:45.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.832+0000 7f912cb94640 1 -- 172.21.15.89:0/4009520311 shutdown_connections 2023-10-30T12:34:45.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.832+0000 7f912cb94640 1 --2- 172.21.15.89:0/4009520311 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f91080664a0 0x7f9108068960 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:45.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.832+0000 7f912cb94640 1 --2- 172.21.15.89:0/4009520311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9128159e60 0x7f912814d450 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:45.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.832+0000 7f912cb94640 1 --2- 172.21.15.89:0/4009520311 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91280bfec0 0x7f91280c9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:45.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.832+0000 7f912cb94640 1 --2- 172.21.15.89:0/4009520311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9128069e90 0x7f91280a9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:45.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.832+0000 7f912cb94640 1 -- 172.21.15.89:0/4009520311 >> 172.21.15.89:0/4009520311 conn(0x7f91280c7910 msgr2=0x7f9128157060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:34:45.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.832+0000 7f912cb94640 1 -- 172.21.15.89:0/4009520311 shutdown_connections 2023-10-30T12:34:45.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:45.832+0000 7f912cb94640 1 -- 172.21.15.89:0/4009520311 wait complete. 2023-10-30T12:34:45.856 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T12:34:47.857 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:34:48.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.000+0000 7f908edf8640 1 Processor -- start 2023-10-30T12:34:48.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.000+0000 7f908edf8640 1 -- start start 2023-10-30T12:34:48.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.000+0000 7f908edf8640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f908815d6f0 0x7f908815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:48.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.000+0000 7f908edf8640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f90880cf160 0x7f90880cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:48.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.000+0000 7f908edf8640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90880cb800 con 0x7f90881538a0 2023-10-30T12:34:48.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.000+0000 7f908edf8640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90880cb330 con 0x7f90880cf160 2023-10-30T12:34:48.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.000+0000 7f908edf8640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90880cad70 con 0x7f908815d6f0 2023-10-30T12:34:48.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.000+0000 7f907ffff640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f908815d6f0 0x7f908815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:48.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.000+0000 7f908d395640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f90880cf160 0x7f90880cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:48.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.000+0000 7f907ffff640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f908815d6f0 0x7f908815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:46192/0 (socket says 172.21.15.89:46192) 2023-10-30T12:34:48.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.000+0000 7f907ffff640 1 -- 172.21.15.89:0/3109314450 learned_addr learned my addr 172.21.15.89:0/3109314450 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:34:48.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.000+0000 7f908cb94640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f90881538a0 0x7f90880ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:47706/0 (socket says 172.21.15.89:47706) 2023-10-30T12:34:48.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.000+0000 7f907ffff640 1 -- 172.21.15.89:0/3109314450 >> v2:172.21.15.192:3300/0 conn(0x7f90880cf160 msgr2=0x7f90880cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:34:48.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.000+0000 7f907ffff640 1 --2- 172.21.15.89:0/3109314450 >> v2:172.21.15.192:3300/0 conn(0x7f90880cf160 0x7f90880cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:48.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.000+0000 7f907ffff640 1 -- 172.21.15.89:0/3109314450 >> v1:172.21.15.192:6789/0 conn(0x7f90881538a0 legacy=0x7f90880ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:48.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.000+0000 7f907ffff640 1 -- 172.21.15.89:0/3109314450 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90880cb050 con 0x7f908815d6f0 2023-10-30T12:34:48.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.000+0000 7f908d395640 1 --2- 172.21.15.89:0/3109314450 >> v2:172.21.15.192:3300/0 conn(0x7f90880cf160 0x7f90880cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:34:48.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.004+0000 7f907ffff640 1 --2- 172.21.15.89:0/3109314450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f908815d6f0 0x7f908815a090 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9078009870 tx=0x7f907801ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a6aa9bf37b37333 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:48.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.004+0000 7f907f7fe640 1 -- 172.21.15.89:0/3109314450 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90780a8a30 con 0x7f908815d6f0 2023-10-30T12:34:48.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.004+0000 7f907f7fe640 1 -- 172.21.15.89:0/3109314450 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90780a8b90 con 0x7f908815d6f0 2023-10-30T12:34:48.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.004+0000 7f907f7fe640 1 -- 172.21.15.89:0/3109314450 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90780b0440 con 0x7f908815d6f0 2023-10-30T12:34:48.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.004+0000 7f908edf8640 1 -- 172.21.15.89:0/3109314450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f908815d6f0 msgr2=0x7f908815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:48.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.004+0000 7f908edf8640 1 --2- 172.21.15.89:0/3109314450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f908815d6f0 0x7f908815a090 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9078009870 tx=0x7f907801ed30 comp rx=0 tx=0).stop 2023-10-30T12:34:48.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.004+0000 7f908edf8640 1 -- 172.21.15.89:0/3109314450 shutdown_connections 2023-10-30T12:34:48.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.004+0000 7f908edf8640 1 --2- 172.21.15.89:0/3109314450 >> v2:172.21.15.192:3300/0 conn(0x7f90880cf160 0x7f90880cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:48.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.004+0000 7f908edf8640 1 --2- 172.21.15.89:0/3109314450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f908815d6f0 0x7f908815a090 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:48.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.004+0000 7f908edf8640 1 -- 172.21.15.89:0/3109314450 >> 172.21.15.89:0/3109314450 conn(0x7f908806bfe0 msgr2=0x7f908806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:34:48.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.004+0000 7f908edf8640 1 -- 172.21.15.89:0/3109314450 shutdown_connections 2023-10-30T12:34:48.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.004+0000 7f908edf8640 1 -- 172.21.15.89:0/3109314450 wait complete. 2023-10-30T12:34:48.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.008+0000 7f908edf8640 1 Processor -- start 2023-10-30T12:34:48.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.008+0000 7f908edf8640 1 -- start start 2023-10-30T12:34:48.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.008+0000 7f908edf8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90880cf160 0x7f90881f2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:48.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.008+0000 7f908edf8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90881538a0 0x7f90881e3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:48.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.008+0000 7f908edf8640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f908815d6f0 0x7f90881e4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:48.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.008+0000 7f908edf8640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90881e4150 con 0x7f90880cf160 2023-10-30T12:34:48.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.008+0000 7f908edf8640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90881e4710 con 0x7f90881538a0 2023-10-30T12:34:48.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.008+0000 7f908edf8640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90880c6b20 con 0x7f908815d6f0 2023-10-30T12:34:48.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.008+0000 7f908cb94640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90880cf160 0x7f90881f2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:48.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.008+0000 7f908cb94640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90880cf160 0x7f90881f2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55972/0 (socket says 172.21.15.89:55972) 2023-10-30T12:34:48.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.008+0000 7f908cb94640 1 -- 172.21.15.89:0/3764710541 learned_addr learned my addr 172.21.15.89:0/3764710541 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:34:48.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.008+0000 7f907ffff640 1 --2- 172.21.15.89:0/3764710541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90881538a0 0x7f90881e3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:48.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.008+0000 7f908d395640 1 --2- 172.21.15.89:0/3764710541 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f908815d6f0 0x7f90881e4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:48.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.008+0000 7f908cb94640 1 -- 172.21.15.89:0/3764710541 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f908815d6f0 msgr2=0x7f90881e4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:48.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.008+0000 7f908cb94640 1 --2- 172.21.15.89:0/3764710541 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f908815d6f0 0x7f90881e4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:48.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.008+0000 7f908cb94640 1 -- 172.21.15.89:0/3764710541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90881538a0 msgr2=0x7f90881e3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:48.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.008+0000 7f908cb94640 1 --2- 172.21.15.89:0/3764710541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90881538a0 0x7f90881e3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:48.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.008+0000 7f908cb94640 1 -- 172.21.15.89:0/3764710541 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9080009a70 con 0x7f90880cf160 2023-10-30T12:34:48.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.008+0000 7f908cb94640 1 --2- 172.21.15.89:0/3764710541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90880cf160 0x7f90881f2460 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f908400c4c0 tx=0x7f908400c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:48.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.008+0000 7f907d7fa640 1 -- 172.21.15.89:0/3764710541 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084007480 con 0x7f90880cf160 2023-10-30T12:34:48.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.008+0000 7f907d7fa640 1 -- 172.21.15.89:0/3764710541 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90840075e0 con 0x7f90880cf160 2023-10-30T12:34:48.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.008+0000 7f908edf8640 1 -- 172.21.15.89:0/3764710541 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90780094d0 con 0x7f90880cf160 2023-10-30T12:34:48.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.008+0000 7f907d7fa640 1 -- 172.21.15.89:0/3764710541 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084004a10 con 0x7f90880cf160 2023-10-30T12:34:48.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.008+0000 7f908edf8640 1 -- 172.21.15.89:0/3764710541 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90881f2c10 con 0x7f90880cf160 2023-10-30T12:34:48.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.012+0000 7f908edf8640 1 -- 172.21.15.89:0/3764710541 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90880b2a50 con 0x7f90880cf160 2023-10-30T12:34:48.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.012+0000 7f907d7fa640 1 -- 172.21.15.89:0/3764710541 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9084004b70 con 0x7f90880cf160 2023-10-30T12:34:48.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.012+0000 7f907d7fa640 1 --2- 172.21.15.89:0/3764710541 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f90740664a0 0x7f9074068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:48.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.012+0000 7f907d7fa640 1 -- 172.21.15.89:0/3764710541 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f9084083630 con 0x7f90880cf160 2023-10-30T12:34:48.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.012+0000 7f907ffff640 1 --2- 172.21.15.89:0/3764710541 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f90740664a0 0x7f9074068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:48.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.012+0000 7f907ffff640 1 --2- 172.21.15.89:0/3764710541 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f90740664a0 0x7f9074068960 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f90780b2000 tx=0x7f90780aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:48.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.016+0000 7f907d7fa640 1 -- 172.21.15.89:0/3764710541 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f90840507a0 con 0x7f90880cf160 2023-10-30T12:34:48.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.200+0000 7f908edf8640 1 -- 172.21.15.89:0/3764710541 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f908804e170 con 0x7f90880cf160 2023-10-30T12:34:48.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.200+0000 7f907d7fa640 1 -- 172.21.15.89:0/3764710541 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+6534 (secure 0 0 0) 0x7f9084053dc0 con 0x7f90880cf160 2023-10-30T12:34:48.204 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:34:48.205 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5139,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:34:11.946671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:34:48.205 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 32 2023-10-30T12:34:48.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.200+0000 7f908edf8640 1 -- 172.21.15.89:0/3764710541 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f90740664a0 msgr2=0x7f9074068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:48.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.200+0000 7f908edf8640 1 --2- 172.21.15.89:0/3764710541 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f90740664a0 0x7f9074068960 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f90780b2000 tx=0x7f90780aa070 comp rx=0 tx=0).stop 2023-10-30T12:34:48.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.200+0000 7f908edf8640 1 -- 172.21.15.89:0/3764710541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90880cf160 msgr2=0x7f90881f2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:48.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.200+0000 7f908edf8640 1 --2- 172.21.15.89:0/3764710541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90880cf160 0x7f90881f2460 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f908400c4c0 tx=0x7f908400c980 comp rx=0 tx=0).stop 2023-10-30T12:34:48.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.200+0000 7f908edf8640 1 -- 172.21.15.89:0/3764710541 shutdown_connections 2023-10-30T12:34:48.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.200+0000 7f908edf8640 1 --2- 172.21.15.89:0/3764710541 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f90740664a0 0x7f9074068960 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:48.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.200+0000 7f908edf8640 1 --2- 172.21.15.89:0/3764710541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f90881538a0 0x7f90881e3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:48.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.200+0000 7f908edf8640 1 --2- 172.21.15.89:0/3764710541 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f908815d6f0 0x7f90881e4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:48.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.200+0000 7f908edf8640 1 --2- 172.21.15.89:0/3764710541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90880cf160 0x7f90881f2460 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:48.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.200+0000 7f908edf8640 1 -- 172.21.15.89:0/3764710541 >> 172.21.15.89:0/3764710541 conn(0x7f908806bfe0 msgr2=0x7f90881510d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:34:48.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.200+0000 7f908edf8640 1 -- 172.21.15.89:0/3764710541 shutdown_connections 2023-10-30T12:34:48.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:48.200+0000 7f908edf8640 1 -- 172.21.15.89:0/3764710541 wait complete. 2023-10-30T12:34:48.225 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T12:34:50.227 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:34:50.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.372+0000 7ff2637d3640 1 Processor -- start 2023-10-30T12:34:50.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.372+0000 7ff2637d3640 1 -- start start 2023-10-30T12:34:50.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.372+0000 7ff2637d3640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff25c06f7a0 0x7ff25c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:50.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.372+0000 7ff2637d3640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff25c0c6bd0 0x7ff25c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:50.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.372+0000 7ff2637d3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff25c071110 0x7ff25c158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:50.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.372+0000 7ff2637d3640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff25c0700c0 con 0x7ff25c0c6bd0 2023-10-30T12:34:50.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.372+0000 7ff2637d3640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff25c06a3d0 con 0x7ff25c06f7a0 2023-10-30T12:34:50.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.372+0000 7ff2637d3640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c066cf0 con 0x7ff25c071110 2023-10-30T12:34:50.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.372+0000 7ff260d6e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff25c0c6bd0 0x7ff25c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:50.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.372+0000 7ff260d6e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff25c0c6bd0 0x7ff25c069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60774/0 (socket says 172.21.15.89:60774) 2023-10-30T12:34:50.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.372+0000 7ff260d6e640 1 -- 172.21.15.89:0/1552574935 learned_addr learned my addr 172.21.15.89:0/1552574935 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:34:50.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.372+0000 7ff26156f640 1 --2- 172.21.15.89:0/1552574935 >> v2:172.21.15.192:3300/0 conn(0x7ff25c06f7a0 0x7ff25c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:50.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.372+0000 7ff261d70640 1 --2- 172.21.15.89:0/1552574935 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff25c071110 0x7ff25c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:50.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.372+0000 7ff260d6e640 1 -- 172.21.15.89:0/1552574935 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff25c071110 msgr2=0x7ff25c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:34:50.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.372+0000 7ff260d6e640 1 --2- 172.21.15.89:0/1552574935 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff25c071110 0x7ff25c158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:50.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.372+0000 7ff260d6e640 1 -- 172.21.15.89:0/1552574935 >> v2:172.21.15.192:3300/0 conn(0x7ff25c06f7a0 msgr2=0x7ff25c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:34:50.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.372+0000 7ff260d6e640 1 --2- 172.21.15.89:0/1552574935 >> v2:172.21.15.192:3300/0 conn(0x7ff25c06f7a0 0x7ff25c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:50.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.372+0000 7ff260d6e640 1 -- 172.21.15.89:0/1552574935 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff25c0c70d0 con 0x7ff25c0c6bd0 2023-10-30T12:34:50.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.376+0000 7ff260d6e640 1 --2- 172.21.15.89:0/1552574935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff25c0c6bd0 0x7ff25c069e90 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff25c06c830 tx=0x7ff24c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=571f9c365c146b38 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:50.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.376+0000 7ff253fff640 1 -- 172.21.15.89:0/1552574935 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24c0a89b0 con 0x7ff25c0c6bd0 2023-10-30T12:34:50.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.376+0000 7ff253fff640 1 -- 172.21.15.89:0/1552574935 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff24c0a8b10 con 0x7ff25c0c6bd0 2023-10-30T12:34:50.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.376+0000 7ff253fff640 1 -- 172.21.15.89:0/1552574935 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24c0a8ea0 con 0x7ff25c0c6bd0 2023-10-30T12:34:50.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.376+0000 7ff2637d3640 1 -- 172.21.15.89:0/1552574935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff25c0c6bd0 msgr2=0x7ff25c069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:50.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.376+0000 7ff2637d3640 1 --2- 172.21.15.89:0/1552574935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff25c0c6bd0 0x7ff25c069e90 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff25c06c830 tx=0x7ff24c01ecf0 comp rx=0 tx=0).stop 2023-10-30T12:34:50.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.376+0000 7ff2637d3640 1 -- 172.21.15.89:0/1552574935 shutdown_connections 2023-10-30T12:34:50.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.376+0000 7ff2637d3640 1 --2- 172.21.15.89:0/1552574935 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff25c071110 0x7ff25c158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:50.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.376+0000 7ff2637d3640 1 --2- 172.21.15.89:0/1552574935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff25c0c6bd0 0x7ff25c069e90 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:50.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.376+0000 7ff2637d3640 1 --2- 172.21.15.89:0/1552574935 >> v2:172.21.15.192:3300/0 conn(0x7ff25c06f7a0 0x7ff25c06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:50.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.376+0000 7ff2637d3640 1 -- 172.21.15.89:0/1552574935 >> 172.21.15.89:0/1552574935 conn(0x7ff25c0c7910 msgr2=0x7ff25c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:34:50.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.376+0000 7ff2637d3640 1 -- 172.21.15.89:0/1552574935 shutdown_connections 2023-10-30T12:34:50.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.376+0000 7ff2637d3640 1 -- 172.21.15.89:0/1552574935 wait complete. 2023-10-30T12:34:50.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.380+0000 7ff2637d3640 1 Processor -- start 2023-10-30T12:34:50.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.380+0000 7ff2637d3640 1 -- start start 2023-10-30T12:34:50.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.380+0000 7ff2637d3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff25c06f7a0 0x7ff25c14aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:50.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.380+0000 7ff2637d3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff25c071110 0x7ff25c14bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:50.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.380+0000 7ff2637d3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff25c0c6bd0 0x7ff25c148930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:50.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.380+0000 7ff2637d3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c1efee0 con 0x7ff25c0c6bd0 2023-10-30T12:34:50.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.380+0000 7ff2637d3640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c06c1d0 con 0x7ff25c071110 2023-10-30T12:34:50.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.380+0000 7ff2637d3640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c06c4b0 con 0x7ff25c06f7a0 2023-10-30T12:34:50.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.380+0000 7ff261d70640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff25c0c6bd0 0x7ff25c148930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:50.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.380+0000 7ff261d70640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff25c0c6bd0 0x7ff25c148930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60788/0 (socket says 172.21.15.89:60788) 2023-10-30T12:34:50.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.380+0000 7ff261d70640 1 -- 172.21.15.89:0/3609810990 learned_addr learned my addr 172.21.15.89:0/3609810990 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:34:50.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.380+0000 7ff26156f640 1 --2- 172.21.15.89:0/3609810990 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff25c06f7a0 0x7ff25c14aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:50.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.380+0000 7ff260d6e640 1 --2- 172.21.15.89:0/3609810990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff25c071110 0x7ff25c14bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:50.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.380+0000 7ff261d70640 1 -- 172.21.15.89:0/3609810990 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff25c06f7a0 msgr2=0x7ff25c14aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:50.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.380+0000 7ff261d70640 1 --2- 172.21.15.89:0/3609810990 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff25c06f7a0 0x7ff25c14aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:50.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.380+0000 7ff261d70640 1 -- 172.21.15.89:0/3609810990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff25c071110 msgr2=0x7ff25c14bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:50.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.380+0000 7ff261d70640 1 --2- 172.21.15.89:0/3609810990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff25c071110 0x7ff25c14bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:50.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.380+0000 7ff261d70640 1 -- 172.21.15.89:0/3609810990 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24c0094d0 con 0x7ff25c0c6bd0 2023-10-30T12:34:50.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.380+0000 7ff261d70640 1 --2- 172.21.15.89:0/3609810990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff25c0c6bd0 0x7ff25c148930 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff254007e60 tx=0x7ff25400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:50.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.380+0000 7ff251ffb640 1 -- 172.21.15.89:0/3609810990 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff254014420 con 0x7ff25c0c6bd0 2023-10-30T12:34:50.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.380+0000 7ff251ffb640 1 -- 172.21.15.89:0/3609810990 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2540145a0 con 0x7ff25c0c6bd0 2023-10-30T12:34:50.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.380+0000 7ff251ffb640 1 -- 172.21.15.89:0/3609810990 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff254014950 con 0x7ff25c0c6bd0 2023-10-30T12:34:50.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.380+0000 7ff2637d3640 1 -- 172.21.15.89:0/3609810990 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff25c1f0180 con 0x7ff25c0c6bd0 2023-10-30T12:34:50.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.380+0000 7ff2637d3640 1 -- 172.21.15.89:0/3609810990 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff25c1f0740 con 0x7ff25c0c6bd0 2023-10-30T12:34:50.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.380+0000 7ff2637d3640 1 -- 172.21.15.89:0/3609810990 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff25c1f0930 con 0x7ff25c0c6bd0 2023-10-30T12:34:50.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.384+0000 7ff251ffb640 1 -- 172.21.15.89:0/3609810990 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff254014ab0 con 0x7ff25c0c6bd0 2023-10-30T12:34:50.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.384+0000 7ff251ffb640 1 --2- 172.21.15.89:0/3609810990 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff2400664a0 0x7ff240068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:50.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.384+0000 7ff251ffb640 1 -- 172.21.15.89:0/3609810990 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6468+0+0 (secure 0 0 0) 0x7ff254084aa0 con 0x7ff25c0c6bd0 2023-10-30T12:34:50.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.384+0000 7ff26156f640 1 --2- 172.21.15.89:0/3609810990 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff2400664a0 0x7ff240068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:50.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.384+0000 7ff26156f640 1 --2- 172.21.15.89:0/3609810990 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff2400664a0 0x7ff240068960 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff2580045f0 tx=0x7ff258008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:50.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.388+0000 7ff251ffb640 1 -- 172.21.15.89:0/3609810990 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff254051ba0 con 0x7ff25c0c6bd0 2023-10-30T12:34:50.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.568+0000 7ff2637d3640 1 -- 172.21.15.89:0/3609810990 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff25c04e170 con 0x7ff25c0c6bd0 2023-10-30T12:34:50.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.568+0000 7ff251ffb640 1 -- 172.21.15.89:0/3609810990 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+6534 (secure 0 0 0) 0x7ff2540551c0 con 0x7ff25c0c6bd0 2023-10-30T12:34:50.574 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:34:50.574 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5139,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:34:11.946671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:34:50.575 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 32 2023-10-30T12:34:50.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.568+0000 7ff2637d3640 1 -- 172.21.15.89:0/3609810990 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff2400664a0 msgr2=0x7ff240068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:50.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.568+0000 7ff2637d3640 1 --2- 172.21.15.89:0/3609810990 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff2400664a0 0x7ff240068960 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff2580045f0 tx=0x7ff258008040 comp rx=0 tx=0).stop 2023-10-30T12:34:50.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.568+0000 7ff2637d3640 1 -- 172.21.15.89:0/3609810990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff25c0c6bd0 msgr2=0x7ff25c148930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:50.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.568+0000 7ff2637d3640 1 --2- 172.21.15.89:0/3609810990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff25c0c6bd0 0x7ff25c148930 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff254007e60 tx=0x7ff25400e4b0 comp rx=0 tx=0).stop 2023-10-30T12:34:50.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.568+0000 7ff2637d3640 1 -- 172.21.15.89:0/3609810990 shutdown_connections 2023-10-30T12:34:50.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.568+0000 7ff2637d3640 1 --2- 172.21.15.89:0/3609810990 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff2400664a0 0x7ff240068960 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:50.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.568+0000 7ff2637d3640 1 --2- 172.21.15.89:0/3609810990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff25c071110 0x7ff25c14bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:50.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.568+0000 7ff2637d3640 1 --2- 172.21.15.89:0/3609810990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff25c0c6bd0 0x7ff25c148930 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:50.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.568+0000 7ff2637d3640 1 --2- 172.21.15.89:0/3609810990 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff25c06f7a0 0x7ff25c14aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:50.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.568+0000 7ff2637d3640 1 -- 172.21.15.89:0/3609810990 >> 172.21.15.89:0/3609810990 conn(0x7ff25c0c7910 msgr2=0x7ff25c0c3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:34:50.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.572+0000 7ff2637d3640 1 -- 172.21.15.89:0/3609810990 shutdown_connections 2023-10-30T12:34:50.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:50.572+0000 7ff2637d3640 1 -- 172.21.15.89:0/3609810990 wait complete. 2023-10-30T12:34:50.595 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T12:34:52.596 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:34:52.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.740+0000 7effc8808640 1 Processor -- start 2023-10-30T12:34:52.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.740+0000 7effc8808640 1 -- start start 2023-10-30T12:34:52.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.740+0000 7effc8808640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7effc40717b0 0x7effc415ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:52.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.740+0000 7effc8808640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7effc40c6290 0x7effc4161eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:52.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.740+0000 7effc8808640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc415b060 con 0x7effc4071b90 2023-10-30T12:34:52.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.740+0000 7effc8808640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effc4157530 con 0x7effc40717b0 2023-10-30T12:34:52.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.740+0000 7effc8808640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc40c4ab0 con 0x7effc40c6290 2023-10-30T12:34:52.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.740+0000 7effc2d9d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7effc40717b0 0x7effc415ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:52.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.740+0000 7effc2d9d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7effc40717b0 0x7effc415ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60802/0 (socket says 172.21.15.89:60802) 2023-10-30T12:34:52.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.740+0000 7effc2d9d640 1 -- 172.21.15.89:0/1817259460 learned_addr learned my addr 172.21.15.89:0/1817259460 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:34:52.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.740+0000 7effc359e640 1 --2- 172.21.15.89:0/1817259460 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7effc40c6290 0x7effc4161eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:52.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.740+0000 7effc2d9d640 1 -- 172.21.15.89:0/1817259460 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7effc40c6290 msgr2=0x7effc4161eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:34:52.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.744+0000 7effc2d9d640 1 --2- 172.21.15.89:0/1817259460 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7effc40c6290 0x7effc4161eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:52.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.744+0000 7effc2d9d640 1 -- 172.21.15.89:0/1817259460 >> v1:172.21.15.89:6789/0 conn(0x7effc4071b90 legacy=0x7effc41513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:52.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.744+0000 7effc2d9d640 1 -- 172.21.15.89:0/1817259460 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc40c4d90 con 0x7effc40717b0 2023-10-30T12:34:52.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.744+0000 7effc259c640 1 -- 172.21.15.89:0/1817259460 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2178293838 0 0) 0x7effc415b060 con 0x7effc4071b90 2023-10-30T12:34:52.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.744+0000 7effc2d9d640 1 --2- 172.21.15.89:0/1817259460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effc40717b0 0x7effc415ab20 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7effb4006fd0 tx=0x7effb401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9f84965af5a7a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:52.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.744+0000 7effc259c640 1 -- 172.21.15.89:0/1817259460 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb40a8a10 con 0x7effc40717b0 2023-10-30T12:34:52.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.744+0000 7effc259c640 1 -- 172.21.15.89:0/1817259460 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb40a8b70 con 0x7effc40717b0 2023-10-30T12:34:52.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.744+0000 7effc259c640 1 -- 172.21.15.89:0/1817259460 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb40b0440 con 0x7effc40717b0 2023-10-30T12:34:52.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.744+0000 7effc8808640 1 -- 172.21.15.89:0/1817259460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effc40717b0 msgr2=0x7effc415ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:52.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.744+0000 7effc8808640 1 --2- 172.21.15.89:0/1817259460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effc40717b0 0x7effc415ab20 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7effb4006fd0 tx=0x7effb401ed10 comp rx=0 tx=0).stop 2023-10-30T12:34:52.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.744+0000 7effc8808640 1 -- 172.21.15.89:0/1817259460 shutdown_connections 2023-10-30T12:34:52.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.744+0000 7effc8808640 1 --2- 172.21.15.89:0/1817259460 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7effc40c6290 0x7effc4161eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:52.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.744+0000 7effc8808640 1 --2- 172.21.15.89:0/1817259460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effc40717b0 0x7effc415ab20 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:52.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.744+0000 7effc8808640 1 -- 172.21.15.89:0/1817259460 >> 172.21.15.89:0/1817259460 conn(0x7effc40c75d0 msgr2=0x7effc40c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:34:52.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.744+0000 7effc8808640 1 -- 172.21.15.89:0/1817259460 shutdown_connections 2023-10-30T12:34:52.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.744+0000 7effc8808640 1 -- 172.21.15.89:0/1817259460 wait complete. 2023-10-30T12:34:52.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.748+0000 7effc8808640 1 Processor -- start 2023-10-30T12:34:52.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.748+0000 7effc8808640 1 -- start start 2023-10-30T12:34:52.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.748+0000 7effc8808640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effc40717b0 0x7effc406f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:52.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.748+0000 7effc8808640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7effc4071b90 0x7effc4147440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:52.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.748+0000 7effc8808640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7effc40c6290 0x7effc4146410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:52.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.748+0000 7effc8808640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc4071260 con 0x7effc40717b0 2023-10-30T12:34:52.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.748+0000 7effc8808640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc41f2680 con 0x7effc4071b90 2023-10-30T12:34:52.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.748+0000 7effc8808640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc40c2a20 con 0x7effc40c6290 2023-10-30T12:34:52.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.748+0000 7effc2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effc40717b0 0x7effc406f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:52.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.748+0000 7effc2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effc40717b0 0x7effc406f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60818/0 (socket says 172.21.15.89:60818) 2023-10-30T12:34:52.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.748+0000 7effc2d9d640 1 -- 172.21.15.89:0/3212710373 learned_addr learned my addr 172.21.15.89:0/3212710373 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:34:52.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.748+0000 7effbbfff640 1 --2- 172.21.15.89:0/3212710373 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7effc4071b90 0x7effc4147440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:52.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.748+0000 7effc359e640 1 --2- 172.21.15.89:0/3212710373 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7effc40c6290 0x7effc4146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:52.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.748+0000 7effc2d9d640 1 -- 172.21.15.89:0/3212710373 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7effc40c6290 msgr2=0x7effc4146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:52.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.748+0000 7effc2d9d640 1 --2- 172.21.15.89:0/3212710373 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7effc40c6290 0x7effc4146410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:52.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.748+0000 7effc2d9d640 1 -- 172.21.15.89:0/3212710373 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7effc4071b90 msgr2=0x7effc4147440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:52.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.748+0000 7effc2d9d640 1 --2- 172.21.15.89:0/3212710373 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7effc4071b90 0x7effc4147440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:52.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.748+0000 7effc2d9d640 1 -- 172.21.15.89:0/3212710373 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb4009480 con 0x7effc40717b0 2023-10-30T12:34:52.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.748+0000 7effbbfff640 1 --2- 172.21.15.89:0/3212710373 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7effc4071b90 0x7effc4147440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:34:52.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.748+0000 7effc2d9d640 1 --2- 172.21.15.89:0/3212710373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effc40717b0 0x7effc406f340 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7effb401d3c0 tx=0x7effb401bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:52.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.748+0000 7effbb7fe640 1 -- 172.21.15.89:0/3212710373 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb4003820 con 0x7effc40717b0 2023-10-30T12:34:52.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.748+0000 7effbb7fe640 1 -- 172.21.15.89:0/3212710373 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb4003980 con 0x7effc40717b0 2023-10-30T12:34:52.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.748+0000 7effbb7fe640 1 -- 172.21.15.89:0/3212710373 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb4003d30 con 0x7effc40717b0 2023-10-30T12:34:52.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.748+0000 7effc8808640 1 -- 172.21.15.89:0/3212710373 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc41f2b70 con 0x7effc40717b0 2023-10-30T12:34:52.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.748+0000 7effc8808640 1 -- 172.21.15.89:0/3212710373 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effc41f2890 con 0x7effc40717b0 2023-10-30T12:34:52.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.752+0000 7effc8808640 1 -- 172.21.15.89:0/3212710373 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc404e170 con 0x7effc40717b0 2023-10-30T12:34:52.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.752+0000 7effbb7fe640 1 -- 172.21.15.89:0/3212710373 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7effb40b0440 con 0x7effc40717b0 2023-10-30T12:34:52.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.752+0000 7effbb7fe640 1 --2- 172.21.15.89:0/3212710373 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7effa40664a0 0x7effa4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:52.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.752+0000 7effbb7fe640 1 -- 172.21.15.89:0/3212710373 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6468+0+0 (secure 0 0 0) 0x7effb4120fb0 con 0x7effc40717b0 2023-10-30T12:34:52.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.752+0000 7effbbfff640 1 --2- 172.21.15.89:0/3212710373 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7effa40664a0 0x7effa4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:52.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.752+0000 7effbbfff640 1 --2- 172.21.15.89:0/3212710373 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7effa40664a0 0x7effa4068960 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7effc40c2bc0 tx=0x7effac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:52.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.756+0000 7effbb7fe640 1 -- 172.21.15.89:0/3212710373 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7effb40ee090 con 0x7effc40717b0 2023-10-30T12:34:52.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.936+0000 7effc8808640 1 -- 172.21.15.89:0/3212710373 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effc40c3410 con 0x7effc40717b0 2023-10-30T12:34:52.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.936+0000 7effbb7fe640 1 -- 172.21.15.89:0/3212710373 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+6534 (secure 0 0 0) 0x7effb40b5090 con 0x7effc40717b0 2023-10-30T12:34:52.942 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:34:52.942 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5139,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:34:11.946671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:34:52.943 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 32 2023-10-30T12:34:52.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.940+0000 7effc8808640 1 -- 172.21.15.89:0/3212710373 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7effa40664a0 msgr2=0x7effa4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:52.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.940+0000 7effc8808640 1 --2- 172.21.15.89:0/3212710373 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7effa40664a0 0x7effa4068960 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7effc40c2bc0 tx=0x7effac009040 comp rx=0 tx=0).stop 2023-10-30T12:34:52.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.940+0000 7effc8808640 1 -- 172.21.15.89:0/3212710373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effc40717b0 msgr2=0x7effc406f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:52.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.940+0000 7effc8808640 1 --2- 172.21.15.89:0/3212710373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effc40717b0 0x7effc406f340 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7effb401d3c0 tx=0x7effb401bab0 comp rx=0 tx=0).stop 2023-10-30T12:34:52.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.940+0000 7effc8808640 1 -- 172.21.15.89:0/3212710373 shutdown_connections 2023-10-30T12:34:52.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.940+0000 7effc8808640 1 --2- 172.21.15.89:0/3212710373 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7effa40664a0 0x7effa4068960 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:52.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.940+0000 7effc8808640 1 --2- 172.21.15.89:0/3212710373 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7effc4071b90 0x7effc4147440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:52.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.940+0000 7effc8808640 1 --2- 172.21.15.89:0/3212710373 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7effc40c6290 0x7effc4146410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:52.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.940+0000 7effc8808640 1 --2- 172.21.15.89:0/3212710373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effc40717b0 0x7effc406f340 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:52.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.940+0000 7effc8808640 1 -- 172.21.15.89:0/3212710373 >> 172.21.15.89:0/3212710373 conn(0x7effc40c75d0 msgr2=0x7effc4151190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:34:52.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.940+0000 7effc8808640 1 -- 172.21.15.89:0/3212710373 shutdown_connections 2023-10-30T12:34:52.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:52.940+0000 7effc8808640 1 -- 172.21.15.89:0/3212710373 wait complete. 2023-10-30T12:34:52.962 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T12:34:54.963 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:34:55.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.104+0000 7ff680890640 1 Processor -- start 2023-10-30T12:34:55.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff680890640 1 -- start start 2023-10-30T12:34:55.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff680890640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff67c1538f0 0x7ff67c159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:55.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff680890640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67c0ca6a0 con 0x7ff67c15cd50 2023-10-30T12:34:55.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff680890640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67c15d300 con 0x7ff67c0cefa0 2023-10-30T12:34:55.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff680890640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff67c15d5e0 con 0x7ff67c1538f0 2023-10-30T12:34:55.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff67a59c640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ff67c15cd50 0x7ff67c0c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56594/0 (socket says 172.21.15.89:56594) 2023-10-30T12:34:55.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff67a59c640 1 -- 172.21.15.89:0/2876578239 learned_addr learned my addr 172.21.15.89:0/2876578239 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:34:55.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff67ad9d640 1 --2- 172.21.15.89:0/2876578239 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff67c1538f0 0x7ff67c159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:55.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff679d9b640 1 -- 172.21.15.89:0/2876578239 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054711988 0 0) 0x7ff67c0ca6a0 con 0x7ff67c15cd50 2023-10-30T12:34:55.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff679d9b640 1 -- 172.21.15.89:0/2876578239 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff664003530 con 0x7ff67c15cd50 2023-10-30T12:34:55.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff679d9b640 1 -- 172.21.15.89:0/2876578239 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1228568162 0 0) 0x7ff664003530 con 0x7ff67c15cd50 2023-10-30T12:34:55.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff679d9b640 1 -- 172.21.15.89:0/2876578239 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff67c1538f0 msgr2=0x7ff67c159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:34:55.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff679d9b640 1 --2- 172.21.15.89:0/2876578239 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff67c1538f0 0x7ff67c159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:55.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff679d9b640 1 -- 172.21.15.89:0/2876578239 >> v1:172.21.15.192:6789/0 conn(0x7ff67c0cefa0 legacy=0x7ff67c1640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:55.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff679d9b640 1 -- 172.21.15.89:0/2876578239 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff67c15d8c0 con 0x7ff67c15cd50 2023-10-30T12:34:55.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff679d9b640 1 -- 172.21.15.89:0/2876578239 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7ff670001680 con 0x7ff67c15cd50 2023-10-30T12:34:55.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff679d9b640 1 -- 172.21.15.89:0/2876578239 >> v1:172.21.15.89:6789/0 conn(0x7ff67c15cd50 legacy=0x7ff67c0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:55.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff679d9b640 1 --2- 172.21.15.89:0/2876578239 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6640039b0 0x7ff664003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:55.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff679d9b640 1 --2- 172.21.15.89:0/2876578239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6640047d0 0x7ff664004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:55.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff679d9b640 1 --2- 172.21.15.89:0/2876578239 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff67c1538f0 0x7ff664005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:55.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff679d9b640 1 -- 172.21.15.89:0/2876578239 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67c15d5e0 con 0x7ff6640047d0 2023-10-30T12:34:55.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff679d9b640 1 -- 172.21.15.89:0/2876578239 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66400bad0 con 0x7ff6640039b0 2023-10-30T12:34:55.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff679d9b640 1 -- 172.21.15.89:0/2876578239 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff66400bc90 con 0x7ff67c1538f0 2023-10-30T12:34:55.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff67a59c640 1 --2- 172.21.15.89:0/2876578239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6640047d0 0x7ff664004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:55.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff67ad9d640 1 --2- 172.21.15.89:0/2876578239 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6640039b0 0x7ff664003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:55.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff67b59e640 1 --2- 172.21.15.89:0/2876578239 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff67c1538f0 0x7ff664005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:55.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff67a59c640 1 -- 172.21.15.89:0/2876578239 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff67c1538f0 msgr2=0x7ff664005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:34:55.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff67a59c640 1 --2- 172.21.15.89:0/2876578239 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff67c1538f0 0x7ff664005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:55.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff67a59c640 1 -- 172.21.15.89:0/2876578239 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6640039b0 msgr2=0x7ff664003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:34:55.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff67a59c640 1 --2- 172.21.15.89:0/2876578239 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6640039b0 0x7ff664003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:55.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.108+0000 7ff67a59c640 1 -- 172.21.15.89:0/2876578239 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff664003530 con 0x7ff6640047d0 2023-10-30T12:34:55.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.112+0000 7ff67a59c640 1 --2- 172.21.15.89:0/2876578239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6640047d0 0x7ff664004c50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff67000c210 tx=0x7ff67001cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8439f373e056cd0f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:55.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.112+0000 7ff679d9b640 1 -- 172.21.15.89:0/2876578239 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6700a5b00 con 0x7ff6640047d0 2023-10-30T12:34:55.126 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.112+0000 7ff679d9b640 1 -- 172.21.15.89:0/2876578239 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6700a5c60 con 0x7ff6640047d0 2023-10-30T12:34:55.126 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.112+0000 7ff679d9b640 1 -- 172.21.15.89:0/2876578239 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff670005b20 con 0x7ff6640047d0 2023-10-30T12:34:55.126 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.112+0000 7ff680890640 1 -- 172.21.15.89:0/2876578239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6640047d0 msgr2=0x7ff664004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:55.126 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.112+0000 7ff680890640 1 --2- 172.21.15.89:0/2876578239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6640047d0 0x7ff664004c50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff67000c210 tx=0x7ff67001cc00 comp rx=0 tx=0).stop 2023-10-30T12:34:55.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.112+0000 7ff67b59e640 1 -- 172.21.15.89:0/2876578239 reap_dead start 2023-10-30T12:34:55.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.112+0000 7ff680890640 1 -- 172.21.15.89:0/2876578239 shutdown_connections 2023-10-30T12:34:55.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.112+0000 7ff680890640 1 -- 172.21.15.89:0/2876578239 >> 172.21.15.89:0/2876578239 conn(0x7ff67c06a640 msgr2=0x7ff67c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:34:55.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.112+0000 7ff680890640 1 -- 172.21.15.89:0/2876578239 shutdown_connections 2023-10-30T12:34:55.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.112+0000 7ff680890640 1 -- 172.21.15.89:0/2876578239 wait complete. 2023-10-30T12:34:55.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.116+0000 7ff680890640 1 Processor -- start 2023-10-30T12:34:55.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.116+0000 7ff680890640 1 -- start start 2023-10-30T12:34:55.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.116+0000 7ff680890640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6640047d0 0x7ff67c158b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:55.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.116+0000 7ff680890640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff67c0b4bf0 0x7ff67c1f2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:55.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.116+0000 7ff680890640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff67c0b5050 0x7ff67c1f2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:55.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.116+0000 7ff680890640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67c142ff0 con 0x7ff6640047d0 2023-10-30T12:34:55.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.116+0000 7ff680890640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67c143890 con 0x7ff67c0b4bf0 2023-10-30T12:34:55.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.116+0000 7ff680890640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff67c15c750 con 0x7ff67c0b5050 2023-10-30T12:34:55.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.116+0000 7ff67ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6640047d0 0x7ff67c158b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:55.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.116+0000 7ff67ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6640047d0 0x7ff67c158b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60838/0 (socket says 172.21.15.89:60838) 2023-10-30T12:34:55.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.116+0000 7ff67ad9d640 1 -- 172.21.15.89:0/2167092627 learned_addr learned my addr 172.21.15.89:0/2167092627 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:34:55.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.116+0000 7ff67a59c640 1 --2- 172.21.15.89:0/2167092627 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff67c0b4bf0 0x7ff67c1f2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:55.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.116+0000 7ff67b59e640 1 --2- 172.21.15.89:0/2167092627 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff67c0b5050 0x7ff67c1f2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:55.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.116+0000 7ff67ad9d640 1 -- 172.21.15.89:0/2167092627 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff67c0b5050 msgr2=0x7ff67c1f2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:55.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.116+0000 7ff67ad9d640 1 --2- 172.21.15.89:0/2167092627 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff67c0b5050 0x7ff67c1f2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:55.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.116+0000 7ff67ad9d640 1 -- 172.21.15.89:0/2167092627 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff67c0b4bf0 msgr2=0x7ff67c1f2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:55.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.116+0000 7ff67ad9d640 1 --2- 172.21.15.89:0/2167092627 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff67c0b4bf0 0x7ff67c1f2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:55.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.116+0000 7ff67ad9d640 1 -- 172.21.15.89:0/2167092627 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff67c15c9f0 con 0x7ff6640047d0 2023-10-30T12:34:55.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.116+0000 7ff67b59e640 1 --2- 172.21.15.89:0/2167092627 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff67c0b5050 0x7ff67c1f2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:34:55.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.116+0000 7ff67ad9d640 1 --2- 172.21.15.89:0/2167092627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6640047d0 0x7ff67c158b30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff66c0119d0 tx=0x7ff66c011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:55.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.116+0000 7ff6637fe640 1 -- 172.21.15.89:0/2167092627 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff66c00fb50 con 0x7ff6640047d0 2023-10-30T12:34:55.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.116+0000 7ff6637fe640 1 -- 172.21.15.89:0/2167092627 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff66c00fcb0 con 0x7ff6640047d0 2023-10-30T12:34:55.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.116+0000 7ff6637fe640 1 -- 172.21.15.89:0/2167092627 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff66c014440 con 0x7ff6640047d0 2023-10-30T12:34:55.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.116+0000 7ff680890640 1 -- 172.21.15.89:0/2167092627 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff67c143240 con 0x7ff6640047d0 2023-10-30T12:34:55.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.116+0000 7ff680890640 1 -- 172.21.15.89:0/2167092627 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff67c1596a0 con 0x7ff6640047d0 2023-10-30T12:34:55.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.120+0000 7ff680890640 1 -- 172.21.15.89:0/2167092627 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff67c04e170 con 0x7ff6640047d0 2023-10-30T12:34:55.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.120+0000 7ff6637fe640 1 -- 172.21.15.89:0/2167092627 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff66c0145a0 con 0x7ff6640047d0 2023-10-30T12:34:55.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.120+0000 7ff6637fe640 1 --2- 172.21.15.89:0/2167092627 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff6680664a0 0x7ff668068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:55.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.120+0000 7ff6637fe640 1 -- 172.21.15.89:0/2167092627 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6468+0+0 (secure 0 0 0) 0x7ff66c084b00 con 0x7ff6640047d0 2023-10-30T12:34:55.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.120+0000 7ff67a59c640 1 --2- 172.21.15.89:0/2167092627 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff6680664a0 0x7ff668068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:55.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.120+0000 7ff67a59c640 1 --2- 172.21.15.89:0/2167092627 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff6680664a0 0x7ff668068960 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff6700ab000 tx=0x7ff6700a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:55.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.124+0000 7ff6637fe640 1 -- 172.21.15.89:0/2167092627 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff66c086e60 con 0x7ff6640047d0 2023-10-30T12:34:55.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.304+0000 7ff680890640 1 -- 172.21.15.89:0/2167092627 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff67c157180 con 0x7ff6640047d0 2023-10-30T12:34:55.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.304+0000 7ff6637fe640 1 -- 172.21.15.89:0/2167092627 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+6534 (secure 0 0 0) 0x7ff66c022070 con 0x7ff6640047d0 2023-10-30T12:34:55.312 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:34:55.312 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5139,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:34:11.946671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":4262,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4262":{"gid":4262,"name":"d","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.192:6839/3339589587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3339589587},{"type":"v1","addr":"172.21.15.192:6839","nonce":3339589587}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:34:55.314 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 32 2023-10-30T12:34:55.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.308+0000 7ff680890640 1 -- 172.21.15.89:0/2167092627 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff6680664a0 msgr2=0x7ff668068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:55.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.308+0000 7ff680890640 1 --2- 172.21.15.89:0/2167092627 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff6680664a0 0x7ff668068960 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff6700ab000 tx=0x7ff6700a7070 comp rx=0 tx=0).stop 2023-10-30T12:34:55.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.308+0000 7ff680890640 1 -- 172.21.15.89:0/2167092627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6640047d0 msgr2=0x7ff67c158b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:55.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.308+0000 7ff680890640 1 --2- 172.21.15.89:0/2167092627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6640047d0 0x7ff67c158b30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff66c0119d0 tx=0x7ff66c011e90 comp rx=0 tx=0).stop 2023-10-30T12:34:55.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.308+0000 7ff680890640 1 -- 172.21.15.89:0/2167092627 shutdown_connections 2023-10-30T12:34:55.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.308+0000 7ff680890640 1 --2- 172.21.15.89:0/2167092627 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff6680664a0 0x7ff668068960 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:55.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.308+0000 7ff680890640 1 --2- 172.21.15.89:0/2167092627 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff67c0b4bf0 0x7ff67c1f2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:55.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.308+0000 7ff680890640 1 --2- 172.21.15.89:0/2167092627 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff67c0b5050 0x7ff67c1f2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:55.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.308+0000 7ff680890640 1 --2- 172.21.15.89:0/2167092627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6640047d0 0x7ff67c158b30 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:55.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.308+0000 7ff680890640 1 -- 172.21.15.89:0/2167092627 >> 172.21.15.89:0/2167092627 conn(0x7ff67c06a640 msgr2=0x7ff67c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:34:55.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.308+0000 7ff680890640 1 -- 172.21.15.89:0/2167092627 shutdown_connections 2023-10-30T12:34:55.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:55.308+0000 7ff680890640 1 -- 172.21.15.89:0/2167092627 wait complete. 2023-10-30T12:34:55.331 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T12:34:57.332 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:34:57.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.476+0000 7f0076e6b640 1 Processor -- start 2023-10-30T12:34:57.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.476+0000 7f0076e6b640 1 -- start start 2023-10-30T12:34:57.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.476+0000 7f0076e6b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00700c05d0 0x7f007013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:57.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.476+0000 7f0076e6b640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00700c0a50 con 0x7f00700c6b80 2023-10-30T12:34:57.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.476+0000 7f0076e6b640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f007006f6e0 con 0x7f00700c6f60 2023-10-30T12:34:57.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.476+0000 7f0076e6b640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f007006f9c0 con 0x7f00700c05d0 2023-10-30T12:34:57.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.476+0000 7f0074c07640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f00700c6b80 0x7f00700bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56598/0 (socket says 172.21.15.89:56598) 2023-10-30T12:34:57.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.476+0000 7f0074c07640 1 -- 172.21.15.89:0/1087547584 learned_addr learned my addr 172.21.15.89:0/1087547584 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:34:57.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.480+0000 7f0067fff640 1 --2- 172.21.15.89:0/1087547584 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00700c05d0 0x7f007013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:57.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.480+0000 7f00677fe640 1 -- 172.21.15.89:0/1087547584 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054514348 0 0) 0x7f00700c0a50 con 0x7f00700c6b80 2023-10-30T12:34:57.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.480+0000 7f00677fe640 1 -- 172.21.15.89:0/1087547584 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0058005740 con 0x7f00700c6b80 2023-10-30T12:34:57.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.480+0000 7f00677fe640 1 -- 172.21.15.89:0/1087547584 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1119893024 0 0) 0x7f0058005740 con 0x7f00700c6b80 2023-10-30T12:34:57.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.480+0000 7f00677fe640 1 -- 172.21.15.89:0/1087547584 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00700c05d0 msgr2=0x7f007013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:34:57.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.480+0000 7f00677fe640 1 --2- 172.21.15.89:0/1087547584 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00700c05d0 0x7f007013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:57.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.480+0000 7f00677fe640 1 -- 172.21.15.89:0/1087547584 >> v1:172.21.15.192:6789/0 conn(0x7f00700c6f60 legacy=0x7f00701566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:57.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.480+0000 7f00677fe640 1 -- 172.21.15.89:0/1087547584 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f007006fca0 con 0x7f00700c6b80 2023-10-30T12:34:57.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.480+0000 7f00677fe640 1 -- 172.21.15.89:0/1087547584 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f006c001680 con 0x7f00700c6b80 2023-10-30T12:34:57.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.480+0000 7f00677fe640 1 -- 172.21.15.89:0/1087547584 >> v1:172.21.15.89:6789/0 conn(0x7f00700c6b80 legacy=0x7f00700bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:57.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.480+0000 7f00677fe640 1 --2- 172.21.15.89:0/1087547584 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00700c05d0 0x7f0058005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:57.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.480+0000 7f00677fe640 1 --2- 172.21.15.89:0/1087547584 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0058006140 0x7f00580069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:57.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.480+0000 7f00677fe640 1 --2- 172.21.15.89:0/1087547584 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0058006f20 0x7f00580073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:57.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.480+0000 7f00677fe640 1 -- 172.21.15.89:0/1087547584 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007006f9c0 con 0x7f0058006140 2023-10-30T12:34:57.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.480+0000 7f00677fe640 1 -- 172.21.15.89:0/1087547584 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005800dc80 con 0x7f0058006f20 2023-10-30T12:34:57.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.480+0000 7f00677fe640 1 -- 172.21.15.89:0/1087547584 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f005800de40 con 0x7f00700c05d0 2023-10-30T12:34:57.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.480+0000 7f0067fff640 1 --2- 172.21.15.89:0/1087547584 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0058006140 0x7f00580069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:57.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.480+0000 7f0074c07640 1 --2- 172.21.15.89:0/1087547584 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00700c05d0 0x7f0058005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:57.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.480+0000 7f0075408640 1 --2- 172.21.15.89:0/1087547584 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0058006f20 0x7f00580073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:57.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.480+0000 7f0067fff640 1 -- 172.21.15.89:0/1087547584 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00700c05d0 msgr2=0x7f0058005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:34:57.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.480+0000 7f0067fff640 1 --2- 172.21.15.89:0/1087547584 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00700c05d0 0x7f0058005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:57.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.480+0000 7f0067fff640 1 -- 172.21.15.89:0/1087547584 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0058006f20 msgr2=0x7f00580073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:34:57.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.480+0000 7f0067fff640 1 --2- 172.21.15.89:0/1087547584 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0058006f20 0x7f00580073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:57.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.480+0000 7f0067fff640 1 -- 172.21.15.89:0/1087547584 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0058005740 con 0x7f0058006140 2023-10-30T12:34:57.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.484+0000 7f0067fff640 1 --2- 172.21.15.89:0/1087547584 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0058006140 0x7f00580069e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0060009ae0 tx=0x7f006001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9837f8b324742612 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:57.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.484+0000 7f00677fe640 1 -- 172.21.15.89:0/1087547584 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00600a5540 con 0x7f0058006140 2023-10-30T12:34:57.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.484+0000 7f00677fe640 1 -- 172.21.15.89:0/1087547584 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00600a56a0 con 0x7f0058006140 2023-10-30T12:34:57.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.484+0000 7f00677fe640 1 -- 172.21.15.89:0/1087547584 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00600a5a50 con 0x7f0058006140 2023-10-30T12:34:57.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.484+0000 7f0076e6b640 1 -- 172.21.15.89:0/1087547584 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0058006140 msgr2=0x7f00580069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:57.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.484+0000 7f0076e6b640 1 --2- 172.21.15.89:0/1087547584 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0058006140 0x7f00580069e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0060009ae0 tx=0x7f006001ca50 comp rx=0 tx=0).stop 2023-10-30T12:34:57.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.484+0000 7f0075408640 1 -- 172.21.15.89:0/1087547584 reap_dead start 2023-10-30T12:34:57.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.484+0000 7f0076e6b640 1 -- 172.21.15.89:0/1087547584 shutdown_connections 2023-10-30T12:34:57.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.484+0000 7f0076e6b640 1 -- 172.21.15.89:0/1087547584 >> 172.21.15.89:0/1087547584 conn(0x7f00700c7910 msgr2=0x7f00700c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:34:57.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.484+0000 7f0076e6b640 1 -- 172.21.15.89:0/1087547584 shutdown_connections 2023-10-30T12:34:57.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.484+0000 7f0076e6b640 1 -- 172.21.15.89:0/1087547584 wait complete. 2023-10-30T12:34:57.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.484+0000 7f0076e6b640 1 Processor -- start 2023-10-30T12:34:57.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.484+0000 7f0076e6b640 1 -- start start 2023-10-30T12:34:57.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.484+0000 7f0076e6b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0058006140 0x7f00701ddb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:57.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.484+0000 7f0076e6b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00700ca430 0x7f00701dc910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:57.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.484+0000 7f0076e6b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00700c38b0 0x7f00701edd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:57.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.484+0000 7f0076e6b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00700c2ec0 con 0x7f00700c38b0 2023-10-30T12:34:57.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.484+0000 7f0076e6b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00701de300 con 0x7f0058006140 2023-10-30T12:34:57.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.484+0000 7f0076e6b640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00700cabe0 con 0x7f00700ca430 2023-10-30T12:34:57.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.488+0000 7f0075408640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00700c38b0 0x7f00701edd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:57.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.488+0000 7f0075408640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00700c38b0 0x7f00701edd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60866/0 (socket says 172.21.15.89:60866) 2023-10-30T12:34:57.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.488+0000 7f0075408640 1 -- 172.21.15.89:0/1836409146 learned_addr learned my addr 172.21.15.89:0/1836409146 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:34:57.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.488+0000 7f0074c07640 1 --2- 172.21.15.89:0/1836409146 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0058006140 0x7f00701ddb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:57.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.488+0000 7f0067fff640 1 --2- 172.21.15.89:0/1836409146 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00700ca430 0x7f00701dc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:57.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.488+0000 7f0075408640 1 -- 172.21.15.89:0/1836409146 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00700ca430 msgr2=0x7f00701dc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:57.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.488+0000 7f0075408640 1 --2- 172.21.15.89:0/1836409146 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00700ca430 0x7f00701dc910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:57.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.488+0000 7f0075408640 1 -- 172.21.15.89:0/1836409146 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0058006140 msgr2=0x7f00701ddb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:57.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.488+0000 7f0075408640 1 --2- 172.21.15.89:0/1836409146 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0058006140 0x7f00701ddb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:57.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.488+0000 7f0075408640 1 -- 172.21.15.89:0/1836409146 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f007013e280 con 0x7f00700c38b0 2023-10-30T12:34:57.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.488+0000 7f0067fff640 1 --2- 172.21.15.89:0/1836409146 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00700ca430 0x7f00701dc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:34:57.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.488+0000 7f0074c07640 1 --2- 172.21.15.89:0/1836409146 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0058006140 0x7f00701ddb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:34:57.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.488+0000 7f0075408640 1 --2- 172.21.15.89:0/1836409146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00700c38b0 0x7f00701edd50 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f00700c8980 tx=0x7f006800ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:57.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.488+0000 7f00657fa640 1 -- 172.21.15.89:0/1836409146 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0068003980 con 0x7f00700c38b0 2023-10-30T12:34:57.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.488+0000 7f00657fa640 1 -- 172.21.15.89:0/1836409146 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0068003ae0 con 0x7f00700c38b0 2023-10-30T12:34:57.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.488+0000 7f0076e6b640 1 -- 172.21.15.89:0/1836409146 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00700c22b0 con 0x7f00700c38b0 2023-10-30T12:34:57.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.488+0000 7f00657fa640 1 -- 172.21.15.89:0/1836409146 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0068003e90 con 0x7f00700c38b0 2023-10-30T12:34:57.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.488+0000 7f0076e6b640 1 -- 172.21.15.89:0/1836409146 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00700c2870 con 0x7f00700c38b0 2023-10-30T12:34:57.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.488+0000 7f0076e6b640 1 -- 172.21.15.89:0/1836409146 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00700c3070 con 0x7f00700c38b0 2023-10-30T12:34:57.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.488+0000 7f00657fa640 1 -- 172.21.15.89:0/1836409146 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f00680050d0 con 0x7f00700c38b0 2023-10-30T12:34:57.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.488+0000 7f00657fa640 1 --2- 172.21.15.89:0/1836409146 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f005c0664a0 0x7f005c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:57.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.492+0000 7f00657fa640 1 -- 172.21.15.89:0/1836409146 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0068083e80 con 0x7f00700c38b0 2023-10-30T12:34:57.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.492+0000 7f0074c07640 1 --2- 172.21.15.89:0/1836409146 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f005c0664a0 0x7f005c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:57.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.492+0000 7f0074c07640 1 --2- 172.21.15.89:0/1836409146 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f005c0664a0 0x7f005c068960 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f006c005180 tx=0x7f006c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:57.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.492+0000 7f00657fa640 1 -- 172.21.15.89:0/1836409146 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f00700c3070 con 0x7f00700c38b0 2023-10-30T12:34:57.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.676+0000 7f0076e6b640 1 -- 172.21.15.89:0/1836409146 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00700c3070 con 0x7f00700c38b0 2023-10-30T12:34:57.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.676+0000 7f00657fa640 1 -- 172.21.15.89:0/1836409146 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f00700c3070 con 0x7f00700c38b0 2023-10-30T12:34:57.681 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:34:57.682 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:34:56.026194+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:replay","state_seq":1,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:34:57.683 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 33 2023-10-30T12:34:57.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.676+0000 7f0076e6b640 1 -- 172.21.15.89:0/1836409146 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f005c0664a0 msgr2=0x7f005c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:57.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.676+0000 7f0076e6b640 1 --2- 172.21.15.89:0/1836409146 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f005c0664a0 0x7f005c068960 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f006c005180 tx=0x7f006c009040 comp rx=0 tx=0).stop 2023-10-30T12:34:57.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.676+0000 7f0076e6b640 1 -- 172.21.15.89:0/1836409146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00700c38b0 msgr2=0x7f00701edd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:57.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.676+0000 7f0076e6b640 1 --2- 172.21.15.89:0/1836409146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00700c38b0 0x7f00701edd50 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f00700c8980 tx=0x7f006800ec10 comp rx=0 tx=0).stop 2023-10-30T12:34:57.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.676+0000 7f0076e6b640 1 -- 172.21.15.89:0/1836409146 shutdown_connections 2023-10-30T12:34:57.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.676+0000 7f0076e6b640 1 --2- 172.21.15.89:0/1836409146 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f005c0664a0 0x7f005c068960 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:57.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.676+0000 7f0076e6b640 1 --2- 172.21.15.89:0/1836409146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00700c38b0 0x7f00701edd50 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:57.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.676+0000 7f0076e6b640 1 --2- 172.21.15.89:0/1836409146 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00700ca430 0x7f00701dc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:57.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.676+0000 7f0076e6b640 1 --2- 172.21.15.89:0/1836409146 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0058006140 0x7f00701ddb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:57.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.676+0000 7f0076e6b640 1 -- 172.21.15.89:0/1836409146 >> 172.21.15.89:0/1836409146 conn(0x7f00700c7910 msgr2=0x7f00700c19b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:34:57.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.676+0000 7f0076e6b640 1 -- 172.21.15.89:0/1836409146 shutdown_connections 2023-10-30T12:34:57.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.676+0000 7f0076e6b640 1 -- 172.21.15.89:0/1836409146 wait complete. 2023-10-30T12:34:57.701 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-10-30T12:34:57.702 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:34:57.702 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:34:57.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.832+0000 7fe348daa640 1 Processor -- start 2023-10-30T12:34:57.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.832+0000 7fe348daa640 1 -- start start 2023-10-30T12:34:57.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.836+0000 7fe348daa640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe3440c6b80 0x7fe3440bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:57.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.836+0000 7fe348daa640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe34413f4a0 0x7fe3440c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:57.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.836+0000 7fe348daa640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3440c0a50 con 0x7fe3440c6f60 2023-10-30T12:34:57.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.836+0000 7fe348daa640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe34406f6e0 con 0x7fe3440c6b80 2023-10-30T12:34:57.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.836+0000 7fe348daa640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe34406f9c0 con 0x7fe34413f4a0 2023-10-30T12:34:57.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.836+0000 7fe34259c640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe3440c6b80 0x7fe3440bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:57.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.836+0000 7fe342d9d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe34413f4a0 0x7fe3440c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:57.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.836+0000 7fe34259c640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe3440c6b80 0x7fe3440bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:59676/0 (socket says 172.21.15.89:59676) 2023-10-30T12:34:57.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.836+0000 7fe342d9d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe34413f4a0 0x7fe3440c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:47924/0 (socket says 172.21.15.89:47924) 2023-10-30T12:34:57.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.836+0000 7fe34259c640 1 -- 172.21.15.89:0/3279853433 learned_addr learned my addr 172.21.15.89:0/3279853433 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:34:57.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.836+0000 7fe34259c640 1 -- 172.21.15.89:0/3279853433 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe34413f4a0 msgr2=0x7fe3440c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:34:57.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.836+0000 7fe34259c640 1 --2- 172.21.15.89:0/3279853433 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe34413f4a0 0x7fe3440c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:57.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.836+0000 7fe34259c640 1 -- 172.21.15.89:0/3279853433 >> v1:172.21.15.192:6789/0 conn(0x7fe3440c6f60 legacy=0x7fe34413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:57.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.836+0000 7fe34259c640 1 -- 172.21.15.89:0/3279853433 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe34406fca0 con 0x7fe3440c6b80 2023-10-30T12:34:57.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.836+0000 7fe342d9d640 1 --2- 172.21.15.89:0/3279853433 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe34413f4a0 0x7fe3440c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:34:57.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.840+0000 7fe34259c640 1 --2- 172.21.15.89:0/3279853433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3440c6b80 0x7fe3440bfec0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe334009870 tx=0x7fe33401c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b3e1aa507987d11 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:57.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.840+0000 7fe341d9b640 1 -- 172.21.15.89:0/3279853433 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3340ab650 con 0x7fe3440c6b80 2023-10-30T12:34:57.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.840+0000 7fe341d9b640 1 -- 172.21.15.89:0/3279853433 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3340ab7b0 con 0x7fe3440c6b80 2023-10-30T12:34:57.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.840+0000 7fe341d9b640 1 -- 172.21.15.89:0/3279853433 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3340abb40 con 0x7fe3440c6b80 2023-10-30T12:34:57.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.840+0000 7fe348daa640 1 -- 172.21.15.89:0/3279853433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3440c6b80 msgr2=0x7fe3440bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:57.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.840+0000 7fe348daa640 1 --2- 172.21.15.89:0/3279853433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3440c6b80 0x7fe3440bfec0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe334009870 tx=0x7fe33401c540 comp rx=0 tx=0).stop 2023-10-30T12:34:57.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.840+0000 7fe348daa640 1 -- 172.21.15.89:0/3279853433 shutdown_connections 2023-10-30T12:34:57.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.840+0000 7fe348daa640 1 --2- 172.21.15.89:0/3279853433 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe34413f4a0 0x7fe3440c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:57.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.840+0000 7fe348daa640 1 --2- 172.21.15.89:0/3279853433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3440c6b80 0x7fe3440bfec0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:57.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.840+0000 7fe348daa640 1 -- 172.21.15.89:0/3279853433 >> 172.21.15.89:0/3279853433 conn(0x7fe3440c7910 msgr2=0x7fe3440c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:34:57.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.840+0000 7fe348daa640 1 -- 172.21.15.89:0/3279853433 shutdown_connections 2023-10-30T12:34:57.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.840+0000 7fe348daa640 1 -- 172.21.15.89:0/3279853433 wait complete. 2023-10-30T12:34:57.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.840+0000 7fe348daa640 1 Processor -- start 2023-10-30T12:34:57.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.840+0000 7fe348daa640 1 -- start start 2023-10-30T12:34:57.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.844+0000 7fe348daa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3440c6b80 0x7fe3441f5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:57.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.844+0000 7fe348daa640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3440c6f60 0x7fe34413d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:57.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.844+0000 7fe348daa640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe34413f4a0 0x7fe34415f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:57.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.844+0000 7fe348daa640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3440ca3e0 con 0x7fe3440c6b80 2023-10-30T12:34:57.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.844+0000 7fe348daa640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3441590f0 con 0x7fe3440c6f60 2023-10-30T12:34:57.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.844+0000 7fe348daa640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3441585b0 con 0x7fe34413f4a0 2023-10-30T12:34:57.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.844+0000 7fe34259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3440c6b80 0x7fe3441f5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:57.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.844+0000 7fe34259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3440c6b80 0x7fe3441f5ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60874/0 (socket says 172.21.15.89:60874) 2023-10-30T12:34:57.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.844+0000 7fe34259c640 1 -- 172.21.15.89:0/4051997979 learned_addr learned my addr 172.21.15.89:0/4051997979 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:34:57.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.844+0000 7fe33bfff640 1 --2- 172.21.15.89:0/4051997979 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3440c6f60 0x7fe34413d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:57.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.844+0000 7fe342d9d640 1 --2- 172.21.15.89:0/4051997979 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe34413f4a0 0x7fe34415f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:57.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.844+0000 7fe34259c640 1 -- 172.21.15.89:0/4051997979 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe34413f4a0 msgr2=0x7fe34415f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:57.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.844+0000 7fe34259c640 1 --2- 172.21.15.89:0/4051997979 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe34413f4a0 0x7fe34415f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:57.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.844+0000 7fe34259c640 1 -- 172.21.15.89:0/4051997979 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3440c6f60 msgr2=0x7fe34413d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:57.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.844+0000 7fe34259c640 1 --2- 172.21.15.89:0/4051997979 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3440c6f60 0x7fe34413d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:57.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.844+0000 7fe34259c640 1 -- 172.21.15.89:0/4051997979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33c009a70 con 0x7fe3440c6b80 2023-10-30T12:34:57.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.844+0000 7fe34259c640 1 --2- 172.21.15.89:0/4051997979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3440c6b80 0x7fe3441f5ec0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe33401abf0 tx=0x7fe33401ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:57.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.844+0000 7fe33affd640 1 -- 172.21.15.89:0/4051997979 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3340b4800 con 0x7fe3440c6b80 2023-10-30T12:34:57.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.844+0000 7fe33affd640 1 -- 172.21.15.89:0/4051997979 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3340b4960 con 0x7fe3440c6b80 2023-10-30T12:34:57.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.844+0000 7fe348daa640 1 -- 172.21.15.89:0/4051997979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3340094d0 con 0x7fe3440c6b80 2023-10-30T12:34:57.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.844+0000 7fe33affd640 1 -- 172.21.15.89:0/4051997979 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3340b4cf0 con 0x7fe3440c6b80 2023-10-30T12:34:57.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.844+0000 7fe348daa640 1 -- 172.21.15.89:0/4051997979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3441f5a30 con 0x7fe3440c6b80 2023-10-30T12:34:57.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.844+0000 7fe348daa640 1 -- 172.21.15.89:0/4051997979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe34404e170 con 0x7fe3440c6b80 2023-10-30T12:34:57.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.844+0000 7fe33affd640 1 -- 172.21.15.89:0/4051997979 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe3340c7390 con 0x7fe3440c6b80 2023-10-30T12:34:57.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.848+0000 7fe33affd640 1 --2- 172.21.15.89:0/4051997979 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe3240664d0 0x7fe324068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:57.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.848+0000 7fe33affd640 1 -- 172.21.15.89:0/4051997979 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fe33411eca0 con 0x7fe3440c6b80 2023-10-30T12:34:57.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.848+0000 7fe33bfff640 1 --2- 172.21.15.89:0/4051997979 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe3240664d0 0x7fe324068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:57.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.848+0000 7fe33bfff640 1 --2- 172.21.15.89:0/4051997979 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe3240664d0 0x7fe324068990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe344159b60 tx=0x7fe32c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:57.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:57.852+0000 7fe33affd640 1 -- 172.21.15.89:0/4051997979 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe3340ebf20 con 0x7fe3440c6b80 2023-10-30T12:34:58.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:58.032+0000 7fe348daa640 1 -- 172.21.15.89:0/4051997979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe34413f1a0 con 0x7fe3440c6b80 2023-10-30T12:34:58.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:58.032+0000 7fe33affd640 1 -- 172.21.15.89:0/4051997979 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5738 (secure 0 0 0) 0x7fe3340ef540 con 0x7fe3440c6b80 2023-10-30T12:34:58.038 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:34:58.039 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:34:56.026194+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:replay","state_seq":1,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:34:58.039 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 33 2023-10-30T12:34:58.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:58.036+0000 7fe348daa640 1 -- 172.21.15.89:0/4051997979 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe3240664d0 msgr2=0x7fe324068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:58.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:58.036+0000 7fe348daa640 1 --2- 172.21.15.89:0/4051997979 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe3240664d0 0x7fe324068990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe344159b60 tx=0x7fe32c009040 comp rx=0 tx=0).stop 2023-10-30T12:34:58.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:58.036+0000 7fe348daa640 1 -- 172.21.15.89:0/4051997979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3440c6b80 msgr2=0x7fe3441f5ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:58.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:58.036+0000 7fe348daa640 1 --2- 172.21.15.89:0/4051997979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3440c6b80 0x7fe3441f5ec0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe33401abf0 tx=0x7fe33401ad80 comp rx=0 tx=0).stop 2023-10-30T12:34:58.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:58.036+0000 7fe348daa640 1 -- 172.21.15.89:0/4051997979 shutdown_connections 2023-10-30T12:34:58.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:58.036+0000 7fe348daa640 1 --2- 172.21.15.89:0/4051997979 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe3240664d0 0x7fe324068990 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:58.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:58.036+0000 7fe348daa640 1 --2- 172.21.15.89:0/4051997979 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3440c6f60 0x7fe34413d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:58.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:58.036+0000 7fe348daa640 1 --2- 172.21.15.89:0/4051997979 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe34413f4a0 0x7fe34415f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:58.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:58.036+0000 7fe348daa640 1 --2- 172.21.15.89:0/4051997979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3440c6b80 0x7fe3441f5ec0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:58.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:58.036+0000 7fe348daa640 1 -- 172.21.15.89:0/4051997979 >> 172.21.15.89:0/4051997979 conn(0x7fe3440c7910 msgr2=0x7fe3440bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:34:58.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:58.036+0000 7fe348daa640 1 -- 172.21.15.89:0/4051997979 shutdown_connections 2023-10-30T12:34:58.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:34:58.036+0000 7fe348daa640 1 -- 172.21.15.89:0/4051997979 wait complete. 2023-10-30T12:34:58.059 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 33, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:50.729245+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 33, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4318": { "addr": "172.21.15.192:6837/1002124471", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6836", "nonce": 1002124471, "type": "v2" }, { "addr": "172.21.15.192:6837", "nonce": 1002124471, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4318, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4781": { "addr": "172.21.15.192:6835/654256299", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6834", "nonce": 654256299, "type": "v2" }, { "addr": "172.21.15.192:6835", "nonce": 654256299, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4781, "incarnation": 24, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 27 }, "gid_5139": { "addr": "172.21.15.89:6837/951276252", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6836", "nonce": 951276252, "type": "v2" }, { "addr": "172.21.15.89:6837", "nonce": 951276252, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5139, "incarnation": 33, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 44, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:34:56.026194+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4318, "mds_1": 5139, "mds_2": 4781 } } } ], "standbys": [ { "addr": "172.21.15.89:6834/2909790173", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6832", "nonce": 2909790173, "type": "v2" }, { "addr": "172.21.15.89:6834", "nonce": 2909790173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.89:6835/2387357615", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6833", "nonce": 2387357615, "type": "v2" }, { "addr": "172.21.15.89:6835", "nonce": 2387357615, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4239, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:35:00.061 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:00.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.200+0000 7fac738ea640 1 Processor -- start 2023-10-30T12:35:00.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.200+0000 7fac738ea640 1 -- start start 2023-10-30T12:35:00.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.200+0000 7fac738ea640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fac6c1538a0 0x7fac6c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:00.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.200+0000 7fac738ea640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac6c0ca0e0 0x7fac6c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:00.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.200+0000 7fac738ea640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac6c0cb800 con 0x7fac6c0cf160 2023-10-30T12:35:00.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.200+0000 7fac738ea640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac6c15dc30 con 0x7fac6c1538a0 2023-10-30T12:35:00.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.200+0000 7fac738ea640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c0cb330 con 0x7fac6c0ca0e0 2023-10-30T12:35:00.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.200+0000 7fac71686640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fac6c1538a0 0x7fac6c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:00.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.200+0000 7fac71686640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fac6c1538a0 0x7fac6c15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58494/0 (socket says 172.21.15.89:58494) 2023-10-30T12:35:00.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.200+0000 7fac71686640 1 -- 172.21.15.89:0/312509328 learned_addr learned my addr 172.21.15.89:0/312509328 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:35:00.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.200+0000 7fac70e85640 1 --2- 172.21.15.89:0/312509328 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac6c0ca0e0 0x7fac6c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:00.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.200+0000 7fac71686640 1 -- 172.21.15.89:0/312509328 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac6c0ca0e0 msgr2=0x7fac6c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:00.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.200+0000 7fac71686640 1 --2- 172.21.15.89:0/312509328 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac6c0ca0e0 0x7fac6c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:00.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.200+0000 7fac71686640 1 -- 172.21.15.89:0/312509328 >> v1:172.21.15.192:6789/0 conn(0x7fac6c0cf160 legacy=0x7fac6c0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:00.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.200+0000 7fac71686640 1 -- 172.21.15.89:0/312509328 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6c0cad70 con 0x7fac6c1538a0 2023-10-30T12:35:00.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.204+0000 7fac71686640 1 --2- 172.21.15.89:0/312509328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac6c1538a0 0x7fac6c15d6f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fac68009870 tx=0x7fac6801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2455da646fcadce server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:00.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.204+0000 7fac63fff640 1 -- 172.21.15.89:0/312509328 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac680a8a30 con 0x7fac6c1538a0 2023-10-30T12:35:00.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.204+0000 7fac63fff640 1 -- 172.21.15.89:0/312509328 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac680a8b90 con 0x7fac6c1538a0 2023-10-30T12:35:00.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.204+0000 7fac63fff640 1 -- 172.21.15.89:0/312509328 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac680b0440 con 0x7fac6c1538a0 2023-10-30T12:35:00.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.204+0000 7fac738ea640 1 -- 172.21.15.89:0/312509328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac6c1538a0 msgr2=0x7fac6c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:00.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.204+0000 7fac738ea640 1 --2- 172.21.15.89:0/312509328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac6c1538a0 0x7fac6c15d6f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fac68009870 tx=0x7fac6801ed30 comp rx=0 tx=0).stop 2023-10-30T12:35:00.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.204+0000 7fac738ea640 1 -- 172.21.15.89:0/312509328 shutdown_connections 2023-10-30T12:35:00.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.204+0000 7fac738ea640 1 --2- 172.21.15.89:0/312509328 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac6c0ca0e0 0x7fac6c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:00.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.204+0000 7fac738ea640 1 --2- 172.21.15.89:0/312509328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac6c1538a0 0x7fac6c15d6f0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:00.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.204+0000 7fac738ea640 1 -- 172.21.15.89:0/312509328 >> 172.21.15.89:0/312509328 conn(0x7fac6c06bfe0 msgr2=0x7fac6c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:00.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.204+0000 7fac738ea640 1 -- 172.21.15.89:0/312509328 shutdown_connections 2023-10-30T12:35:00.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.204+0000 7fac738ea640 1 -- 172.21.15.89:0/312509328 wait complete. 2023-10-30T12:35:00.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.208+0000 7fac738ea640 1 Processor -- start 2023-10-30T12:35:00.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.208+0000 7fac738ea640 1 -- start start 2023-10-30T12:35:00.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.208+0000 7fac738ea640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac6c0ca0e0 0x7fac6c065de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:00.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.208+0000 7fac738ea640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac6c0cf160 0x7fac6c15b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:00.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.208+0000 7fac738ea640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac6c1538a0 0x7fac6c15bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:00.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.208+0000 7fac738ea640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c0b3fe0 con 0x7fac6c0ca0e0 2023-10-30T12:35:00.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.208+0000 7fac738ea640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c0b4b60 con 0x7fac6c0cf160 2023-10-30T12:35:00.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.208+0000 7fac738ea640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c1e4df0 con 0x7fac6c1538a0 2023-10-30T12:35:00.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.208+0000 7fac71686640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac6c0ca0e0 0x7fac6c065de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:00.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.208+0000 7fac71686640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac6c0ca0e0 0x7fac6c065de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58502/0 (socket says 172.21.15.89:58502) 2023-10-30T12:35:00.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.208+0000 7fac71686640 1 -- 172.21.15.89:0/35264024 learned_addr learned my addr 172.21.15.89:0/35264024 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:35:00.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.208+0000 7fac70e85640 1 --2- 172.21.15.89:0/35264024 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac6c0cf160 0x7fac6c15b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:00.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.208+0000 7fac71e87640 1 --2- 172.21.15.89:0/35264024 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac6c1538a0 0x7fac6c15bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:00.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.208+0000 7fac71686640 1 -- 172.21.15.89:0/35264024 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac6c1538a0 msgr2=0x7fac6c15bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:00.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.208+0000 7fac71686640 1 --2- 172.21.15.89:0/35264024 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac6c1538a0 0x7fac6c15bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:00.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.208+0000 7fac71686640 1 -- 172.21.15.89:0/35264024 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac6c0cf160 msgr2=0x7fac6c15b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:00.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.208+0000 7fac71686640 1 --2- 172.21.15.89:0/35264024 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac6c0cf160 0x7fac6c15b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:00.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.208+0000 7fac71686640 1 -- 172.21.15.89:0/35264024 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac680094d0 con 0x7fac6c0ca0e0 2023-10-30T12:35:00.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.208+0000 7fac71e87640 1 --2- 172.21.15.89:0/35264024 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac6c1538a0 0x7fac6c15bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:35:00.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.208+0000 7fac71686640 1 --2- 172.21.15.89:0/35264024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac6c0ca0e0 0x7fac6c065de0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fac680b3040 tx=0x7fac6801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:00.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.208+0000 7fac61ffb640 1 -- 172.21.15.89:0/35264024 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac680be070 con 0x7fac6c0ca0e0 2023-10-30T12:35:00.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.208+0000 7fac61ffb640 1 -- 172.21.15.89:0/35264024 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac680ba400 con 0x7fac6c0ca0e0 2023-10-30T12:35:00.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.208+0000 7fac61ffb640 1 -- 172.21.15.89:0/35264024 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac680ba7d0 con 0x7fac6c0ca0e0 2023-10-30T12:35:00.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.208+0000 7fac738ea640 1 -- 172.21.15.89:0/35264024 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac6c0b4230 con 0x7fac6c0ca0e0 2023-10-30T12:35:00.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.212+0000 7fac738ea640 1 -- 172.21.15.89:0/35264024 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac6c0b47f0 con 0x7fac6c0ca0e0 2023-10-30T12:35:00.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.212+0000 7fac738ea640 1 -- 172.21.15.89:0/35264024 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac6c0b2a50 con 0x7fac6c0ca0e0 2023-10-30T12:35:00.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.212+0000 7fac61ffb640 1 -- 172.21.15.89:0/35264024 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fac680c3410 con 0x7fac6c0ca0e0 2023-10-30T12:35:00.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.212+0000 7fac61ffb640 1 --2- 172.21.15.89:0/35264024 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fac500664a0 0x7fac50068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:00.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.212+0000 7fac61ffb640 1 -- 172.21.15.89:0/35264024 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fac68121c20 con 0x7fac6c0ca0e0 2023-10-30T12:35:00.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.212+0000 7fac70e85640 1 --2- 172.21.15.89:0/35264024 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fac500664a0 0x7fac50068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:00.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.216+0000 7fac70e85640 1 --2- 172.21.15.89:0/35264024 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fac500664a0 0x7fac50068960 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fac6c06c710 tx=0x7fac5c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:00.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.216+0000 7fac61ffb640 1 -- 172.21.15.89:0/35264024 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fac680eeea0 con 0x7fac6c0ca0e0 2023-10-30T12:35:00.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.396+0000 7fac738ea640 1 -- 172.21.15.89:0/35264024 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac6c04e170 con 0x7fac6c0ca0e0 2023-10-30T12:35:00.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.400+0000 7fac61ffb640 1 -- 172.21.15.89:0/35264024 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5738 (secure 0 0 0) 0x7fac680f24c0 con 0x7fac6c0ca0e0 2023-10-30T12:35:00.404 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:35:00.404 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:34:56.026194+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:replay","state_seq":1,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:00.405 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 33 2023-10-30T12:35:00.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.400+0000 7fac738ea640 1 -- 172.21.15.89:0/35264024 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fac500664a0 msgr2=0x7fac50068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:00.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.400+0000 7fac738ea640 1 --2- 172.21.15.89:0/35264024 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fac500664a0 0x7fac50068960 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fac6c06c710 tx=0x7fac5c008040 comp rx=0 tx=0).stop 2023-10-30T12:35:00.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.400+0000 7fac738ea640 1 -- 172.21.15.89:0/35264024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac6c0ca0e0 msgr2=0x7fac6c065de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:00.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.400+0000 7fac738ea640 1 --2- 172.21.15.89:0/35264024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac6c0ca0e0 0x7fac6c065de0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fac680b3040 tx=0x7fac6801bad0 comp rx=0 tx=0).stop 2023-10-30T12:35:00.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.400+0000 7fac738ea640 1 -- 172.21.15.89:0/35264024 shutdown_connections 2023-10-30T12:35:00.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.400+0000 7fac738ea640 1 --2- 172.21.15.89:0/35264024 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fac500664a0 0x7fac50068960 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:00.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.400+0000 7fac738ea640 1 --2- 172.21.15.89:0/35264024 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac6c0cf160 0x7fac6c15b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:00.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.400+0000 7fac738ea640 1 --2- 172.21.15.89:0/35264024 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac6c1538a0 0x7fac6c15bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:00.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.400+0000 7fac738ea640 1 --2- 172.21.15.89:0/35264024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac6c0ca0e0 0x7fac6c065de0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:00.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.400+0000 7fac738ea640 1 -- 172.21.15.89:0/35264024 >> 172.21.15.89:0/35264024 conn(0x7fac6c06bfe0 msgr2=0x7fac6c14de00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:00.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.400+0000 7fac738ea640 1 -- 172.21.15.89:0/35264024 shutdown_connections 2023-10-30T12:35:00.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:00.400+0000 7fac738ea640 1 -- 172.21.15.89:0/35264024 wait complete. 2023-10-30T12:35:00.424 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:35:02.425 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:02.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.568+0000 7f05949d3640 1 Processor -- start 2023-10-30T12:35:02.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.568+0000 7f05949d3640 1 -- start start 2023-10-30T12:35:02.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.568+0000 7f05949d3640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f05901538a0 0x7f059015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:02.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.568+0000 7f05949d3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05900cf160 0x7f05900cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:02.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.568+0000 7f05949d3640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05900cb800 con 0x7f05900ca0e0 2023-10-30T12:35:02.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.568+0000 7f05949d3640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f059015dc30 con 0x7f05901538a0 2023-10-30T12:35:02.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.568+0000 7f05949d3640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05900cb330 con 0x7f05900cf160 2023-10-30T12:35:02.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.568+0000 7f058e59c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f05901538a0 0x7f059015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:02.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.568+0000 7f058e59c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f05901538a0 0x7f059015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58512/0 (socket says 172.21.15.89:58512) 2023-10-30T12:35:02.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.568+0000 7f0585d9b640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f05900ca0e0 0x7f059015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51916/0 (socket says 172.21.15.89:51916) 2023-10-30T12:35:02.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.568+0000 7f058e59c640 1 -- 172.21.15.89:0/4002150786 learned_addr learned my addr 172.21.15.89:0/4002150786 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:35:02.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.568+0000 7f058ed9d640 1 --2- 172.21.15.89:0/4002150786 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05900cf160 0x7f05900cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:02.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.568+0000 7f058e59c640 1 -- 172.21.15.89:0/4002150786 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05900cf160 msgr2=0x7f05900cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:02.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.568+0000 7f058e59c640 1 --2- 172.21.15.89:0/4002150786 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05900cf160 0x7f05900cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:02.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.568+0000 7f058e59c640 1 -- 172.21.15.89:0/4002150786 >> v1:172.21.15.89:6789/0 conn(0x7f05900ca0e0 legacy=0x7f059015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:02.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.568+0000 7f058e59c640 1 -- 172.21.15.89:0/4002150786 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05900cad70 con 0x7f05901538a0 2023-10-30T12:35:02.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.568+0000 7f058dd9b640 1 -- 172.21.15.89:0/4002150786 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154268030 0 0) 0x7f05900cb800 con 0x7f05900ca0e0 2023-10-30T12:35:02.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.572+0000 7f058e59c640 1 --2- 172.21.15.89:0/4002150786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05901538a0 0x7f059015d6f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0580009870 tx=0x7f058001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed174d957573b30c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:02.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.572+0000 7f058dd9b640 1 -- 172.21.15.89:0/4002150786 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05800a8a30 con 0x7f05901538a0 2023-10-30T12:35:02.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.572+0000 7f058dd9b640 1 -- 172.21.15.89:0/4002150786 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05800a8b90 con 0x7f05901538a0 2023-10-30T12:35:02.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.572+0000 7f058dd9b640 1 -- 172.21.15.89:0/4002150786 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05800b0440 con 0x7f05901538a0 2023-10-30T12:35:02.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.572+0000 7f05949d3640 1 -- 172.21.15.89:0/4002150786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05901538a0 msgr2=0x7f059015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:02.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.572+0000 7f05949d3640 1 --2- 172.21.15.89:0/4002150786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05901538a0 0x7f059015d6f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0580009870 tx=0x7f058001ed30 comp rx=0 tx=0).stop 2023-10-30T12:35:02.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.572+0000 7f05949d3640 1 -- 172.21.15.89:0/4002150786 shutdown_connections 2023-10-30T12:35:02.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.572+0000 7f05949d3640 1 --2- 172.21.15.89:0/4002150786 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05900cf160 0x7f05900cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:02.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.572+0000 7f05949d3640 1 --2- 172.21.15.89:0/4002150786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05901538a0 0x7f059015d6f0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:02.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.572+0000 7f05949d3640 1 -- 172.21.15.89:0/4002150786 >> 172.21.15.89:0/4002150786 conn(0x7f059006bfe0 msgr2=0x7f059006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:02.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.572+0000 7f05949d3640 1 -- 172.21.15.89:0/4002150786 shutdown_connections 2023-10-30T12:35:02.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.572+0000 7f05949d3640 1 -- 172.21.15.89:0/4002150786 wait complete. 2023-10-30T12:35:02.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.576+0000 7f05949d3640 1 Processor -- start 2023-10-30T12:35:02.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.576+0000 7f05949d3640 1 -- start start 2023-10-30T12:35:02.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.576+0000 7f05949d3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05900ca0e0 0x7f059015aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:02.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.576+0000 7f05949d3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05900cf160 0x7f05900ce930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:02.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.576+0000 7f05949d3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05901538a0 0x7f05901ef320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:02.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.576+0000 7f05949d3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05901f3160 con 0x7f05900cf160 2023-10-30T12:35:02.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.576+0000 7f05949d3640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059015b3e0 con 0x7f05901538a0 2023-10-30T12:35:02.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.576+0000 7f05949d3640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05901f2890 con 0x7f05900ca0e0 2023-10-30T12:35:02.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.576+0000 7f0585d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05900cf160 0x7f05900ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:02.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.576+0000 7f058e59c640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05900ca0e0 0x7f059015aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:02.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.576+0000 7f0585d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05900cf160 0x7f05900ce930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58514/0 (socket says 172.21.15.89:58514) 2023-10-30T12:35:02.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.576+0000 7f0585d9b640 1 -- 172.21.15.89:0/3998226956 learned_addr learned my addr 172.21.15.89:0/3998226956 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:35:02.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.576+0000 7f058ed9d640 1 --2- 172.21.15.89:0/3998226956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05901538a0 0x7f05901ef320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:02.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.576+0000 7f0585d9b640 1 -- 172.21.15.89:0/3998226956 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05900ca0e0 msgr2=0x7f059015aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:02.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.576+0000 7f0585d9b640 1 --2- 172.21.15.89:0/3998226956 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05900ca0e0 0x7f059015aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:02.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.576+0000 7f0585d9b640 1 -- 172.21.15.89:0/3998226956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05901538a0 msgr2=0x7f05901ef320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:02.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.576+0000 7f0585d9b640 1 --2- 172.21.15.89:0/3998226956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05901538a0 0x7f05901ef320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:02.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.576+0000 7f0585d9b640 1 -- 172.21.15.89:0/3998226956 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05800094d0 con 0x7f05900cf160 2023-10-30T12:35:02.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.576+0000 7f058ed9d640 1 --2- 172.21.15.89:0/3998226956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05901538a0 0x7f05901ef320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:35:02.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.576+0000 7f0585d9b640 1 --2- 172.21.15.89:0/3998226956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05900cf160 0x7f05900ce930 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f05780011f0 tx=0x7f057800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:02.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.580+0000 7f05877fe640 1 -- 172.21.15.89:0/3998226956 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05780074e0 con 0x7f05900cf160 2023-10-30T12:35:02.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.580+0000 7f05877fe640 1 -- 172.21.15.89:0/3998226956 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0578007640 con 0x7f05900cf160 2023-10-30T12:35:02.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.580+0000 7f05877fe640 1 -- 172.21.15.89:0/3998226956 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0578004a10 con 0x7f05900cf160 2023-10-30T12:35:02.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.580+0000 7f05949d3640 1 -- 172.21.15.89:0/3998226956 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05900b51f0 con 0x7f05900cf160 2023-10-30T12:35:02.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.580+0000 7f05949d3640 1 -- 172.21.15.89:0/3998226956 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05901f2aa0 con 0x7f05900cf160 2023-10-30T12:35:02.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.580+0000 7f05949d3640 1 -- 172.21.15.89:0/3998226956 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05900b2a50 con 0x7f05900cf160 2023-10-30T12:35:02.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.580+0000 7f05877fe640 1 -- 172.21.15.89:0/3998226956 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0578004b70 con 0x7f05900cf160 2023-10-30T12:35:02.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.580+0000 7f05877fe640 1 --2- 172.21.15.89:0/3998226956 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f05700664a0 0x7f0570068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:02.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.580+0000 7f05877fe640 1 -- 172.21.15.89:0/3998226956 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0578083160 con 0x7f05900cf160 2023-10-30T12:35:02.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.580+0000 7f058e59c640 1 --2- 172.21.15.89:0/3998226956 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f05700664a0 0x7f0570068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:02.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.584+0000 7f058e59c640 1 --2- 172.21.15.89:0/3998226956 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f05700664a0 0x7f0570068960 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f05800b2000 tx=0x7f0580009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:02.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.584+0000 7f05877fe640 1 -- 172.21.15.89:0/3998226956 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0578050490 con 0x7f05900cf160 2023-10-30T12:35:02.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.764+0000 7f05949d3640 1 -- 172.21.15.89:0/3998226956 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05900b37a0 con 0x7f05900cf160 2023-10-30T12:35:02.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.764+0000 7f05877fe640 1 -- 172.21.15.89:0/3998226956 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+5740 (secure 0 0 0) 0x7f05900b37a0 con 0x7f05900cf160 2023-10-30T12:35:02.769 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:35:02.769 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:35:01.355331+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:resolve","state_seq":21,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:02.770 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 34 2023-10-30T12:35:02.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.764+0000 7f05949d3640 1 -- 172.21.15.89:0/3998226956 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f05700664a0 msgr2=0x7f0570068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:02.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.764+0000 7f05949d3640 1 --2- 172.21.15.89:0/3998226956 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f05700664a0 0x7f0570068960 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f05800b2000 tx=0x7f0580009040 comp rx=0 tx=0).stop 2023-10-30T12:35:02.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.764+0000 7f05949d3640 1 -- 172.21.15.89:0/3998226956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05900cf160 msgr2=0x7f05900ce930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:02.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.764+0000 7f05949d3640 1 --2- 172.21.15.89:0/3998226956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05900cf160 0x7f05900ce930 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f05780011f0 tx=0x7f057800c9b0 comp rx=0 tx=0).stop 2023-10-30T12:35:02.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.764+0000 7f05949d3640 1 -- 172.21.15.89:0/3998226956 shutdown_connections 2023-10-30T12:35:02.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.764+0000 7f05949d3640 1 --2- 172.21.15.89:0/3998226956 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f05700664a0 0x7f0570068960 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:02.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.764+0000 7f05949d3640 1 --2- 172.21.15.89:0/3998226956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05901538a0 0x7f05901ef320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:02.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.764+0000 7f05949d3640 1 --2- 172.21.15.89:0/3998226956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05900cf160 0x7f05900ce930 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:02.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.764+0000 7f05949d3640 1 --2- 172.21.15.89:0/3998226956 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f05900ca0e0 0x7f059015aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:02.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.764+0000 7f05949d3640 1 -- 172.21.15.89:0/3998226956 >> 172.21.15.89:0/3998226956 conn(0x7f059006bfe0 msgr2=0x7f059014db50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:02.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.764+0000 7f05949d3640 1 -- 172.21.15.89:0/3998226956 shutdown_connections 2023-10-30T12:35:02.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:02.764+0000 7f05949d3640 1 -- 172.21.15.89:0/3998226956 wait complete. 2023-10-30T12:35:02.791 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:35:04.792 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:04.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.936+0000 7f1bf229e640 1 Processor -- start 2023-10-30T12:35:04.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.936+0000 7f1bf229e640 1 -- start start 2023-10-30T12:35:04.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1bf229e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1bec070a00 0x7f1bec070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:04.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1bf229e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bec069e90 0x7f1bec06a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:04.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1bf229e640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bec0713b0 con 0x7f1bec1539a0 2023-10-30T12:35:04.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1bf229e640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bec06fbb0 con 0x7f1bec070a00 2023-10-30T12:35:04.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1bf229e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bec153d80 con 0x7f1bec069e90 2023-10-30T12:35:04.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1bebfff640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f1bec1539a0 0x7f1bec15a380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51922/0 (socket says 172.21.15.89:51922) 2023-10-30T12:35:04.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1bebfff640 1 -- 172.21.15.89:0/3585401462 learned_addr learned my addr 172.21.15.89:0/3585401462 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:35:04.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1beb7fe640 1 --2- 172.21.15.89:0/3585401462 >> v2:172.21.15.192:3300/0 conn(0x7f1bec070a00 0x7f1bec070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:04.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1bf083b640 1 --2- 172.21.15.89:0/3585401462 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bec069e90 0x7f1bec06a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:04.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1beaffd640 1 -- 172.21.15.89:0/3585401462 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2737893774 0 0) 0x7f1bec0713b0 con 0x7f1bec1539a0 2023-10-30T12:35:04.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1beaffd640 1 -- 172.21.15.89:0/3585401462 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bd4003530 con 0x7f1bec1539a0 2023-10-30T12:35:04.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1beaffd640 1 -- 172.21.15.89:0/3585401462 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1636309012 0 0) 0x7f1bd4003530 con 0x7f1bec1539a0 2023-10-30T12:35:04.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1beaffd640 1 -- 172.21.15.89:0/3585401462 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bec069e90 msgr2=0x7f1bec06a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:04.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1beaffd640 1 --2- 172.21.15.89:0/3585401462 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bec069e90 0x7f1bec06a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:04.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1beaffd640 1 -- 172.21.15.89:0/3585401462 >> v2:172.21.15.192:3300/0 conn(0x7f1bec070a00 msgr2=0x7f1bec070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:04.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1beaffd640 1 --2- 172.21.15.89:0/3585401462 >> v2:172.21.15.192:3300/0 conn(0x7f1bec070a00 0x7f1bec070e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:04.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1beaffd640 1 -- 172.21.15.89:0/3585401462 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bec0c5c50 con 0x7f1bec1539a0 2023-10-30T12:35:04.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1beaffd640 1 -- 172.21.15.89:0/3585401462 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f1bdc001680 con 0x7f1bec1539a0 2023-10-30T12:35:04.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1beaffd640 1 -- 172.21.15.89:0/3585401462 >> v1:172.21.15.89:6789/0 conn(0x7f1bec1539a0 legacy=0x7f1bec15a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:04.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1beaffd640 1 --2- 172.21.15.89:0/3585401462 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bec069e90 0x7f1bd4003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:04.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1beaffd640 1 --2- 172.21.15.89:0/3585401462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bd4003f00 0x7f1bd40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:04.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1beaffd640 1 --2- 172.21.15.89:0/3585401462 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bd4004d10 0x7f1bd4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:04.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1beaffd640 1 -- 172.21.15.89:0/3585401462 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bec06fbb0 con 0x7f1bd4003f00 2023-10-30T12:35:04.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1beaffd640 1 -- 172.21.15.89:0/3585401462 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bec153d80 con 0x7f1bd4004d10 2023-10-30T12:35:04.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1beaffd640 1 -- 172.21.15.89:0/3585401462 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd400b9e0 con 0x7f1bec069e90 2023-10-30T12:35:04.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1beb7fe640 1 --2- 172.21.15.89:0/3585401462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bd4003f00 0x7f1bd40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:04.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1bf083b640 1 --2- 172.21.15.89:0/3585401462 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bd4004d10 0x7f1bd4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:04.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1bebfff640 1 --2- 172.21.15.89:0/3585401462 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bec069e90 0x7f1bd4003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:04.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1beb7fe640 1 -- 172.21.15.89:0/3585401462 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bec069e90 msgr2=0x7f1bd4003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:04.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1beb7fe640 1 --2- 172.21.15.89:0/3585401462 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bec069e90 0x7f1bd4003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:04.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1beb7fe640 1 -- 172.21.15.89:0/3585401462 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bd4004d10 msgr2=0x7f1bd4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:04.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1beb7fe640 1 --2- 172.21.15.89:0/3585401462 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bd4004d10 0x7f1bd4005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:04.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.940+0000 7f1beb7fe640 1 -- 172.21.15.89:0/3585401462 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd4003530 con 0x7f1bd4003f00 2023-10-30T12:35:04.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.944+0000 7f1beb7fe640 1 --2- 172.21.15.89:0/3585401462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bd4003f00 0x7f1bd40047d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f1be0009b10 tx=0x7f1be001ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe70cfb746eac8ac server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:04.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.944+0000 7f1beaffd640 1 -- 172.21.15.89:0/3585401462 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be00a5630 con 0x7f1bd4003f00 2023-10-30T12:35:04.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.944+0000 7f1beaffd640 1 -- 172.21.15.89:0/3585401462 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1be00a5790 con 0x7f1bd4003f00 2023-10-30T12:35:04.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.944+0000 7f1beaffd640 1 -- 172.21.15.89:0/3585401462 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be00a5b40 con 0x7f1bd4003f00 2023-10-30T12:35:04.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.944+0000 7f1bf229e640 1 -- 172.21.15.89:0/3585401462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bd4003f00 msgr2=0x7f1bd40047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:04.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.944+0000 7f1bf229e640 1 --2- 172.21.15.89:0/3585401462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bd4003f00 0x7f1bd40047d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f1be0009b10 tx=0x7f1be001ca60 comp rx=0 tx=0).stop 2023-10-30T12:35:04.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.944+0000 7f1bf083b640 1 -- 172.21.15.89:0/3585401462 reap_dead start 2023-10-30T12:35:04.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.944+0000 7f1bf229e640 1 -- 172.21.15.89:0/3585401462 shutdown_connections 2023-10-30T12:35:04.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.944+0000 7f1bf229e640 1 -- 172.21.15.89:0/3585401462 >> 172.21.15.89:0/3585401462 conn(0x7f1bec0c7910 msgr2=0x7f1bec0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:04.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.944+0000 7f1bf229e640 1 -- 172.21.15.89:0/3585401462 shutdown_connections 2023-10-30T12:35:04.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.944+0000 7f1bf229e640 1 -- 172.21.15.89:0/3585401462 wait complete. 2023-10-30T12:35:04.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.944+0000 7f1bf229e640 1 Processor -- start 2023-10-30T12:35:04.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.944+0000 7f1bf229e640 1 -- start start 2023-10-30T12:35:04.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.948+0000 7f1bf229e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bd4003f00 0x7f1bec158af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:04.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.948+0000 7f1bf229e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bec0ca240 0x7f1bec14ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:04.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.948+0000 7f1bf229e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bec159640 0x7f1bec143990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:04.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.948+0000 7f1bf229e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bec1f2e20 con 0x7f1bec0ca240 2023-10-30T12:35:04.960 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.948+0000 7f1bf229e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bec0c9c70 con 0x7f1bd4003f00 2023-10-30T12:35:04.960 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.948+0000 7f1bf229e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bec0c9f10 con 0x7f1bec159640 2023-10-30T12:35:04.960 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.948+0000 7f1beb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bec0ca240 0x7f1bec14ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:04.960 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.948+0000 7f1bebfff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bd4003f00 0x7f1bec158af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:04.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.948+0000 7f1beb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bec0ca240 0x7f1bec14ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58528/0 (socket says 172.21.15.89:58528) 2023-10-30T12:35:04.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.948+0000 7f1beb7fe640 1 -- 172.21.15.89:0/1138350531 learned_addr learned my addr 172.21.15.89:0/1138350531 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:35:04.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.948+0000 7f1bf083b640 1 --2- 172.21.15.89:0/1138350531 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bec159640 0x7f1bec143990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:04.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.948+0000 7f1beb7fe640 1 -- 172.21.15.89:0/1138350531 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bec159640 msgr2=0x7f1bec143990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:04.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.948+0000 7f1beb7fe640 1 --2- 172.21.15.89:0/1138350531 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bec159640 0x7f1bec143990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:04.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.948+0000 7f1beb7fe640 1 -- 172.21.15.89:0/1138350531 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bd4003f00 msgr2=0x7f1bec158af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:04.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.948+0000 7f1beb7fe640 1 --2- 172.21.15.89:0/1138350531 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bd4003f00 0x7f1bec158af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:04.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.948+0000 7f1beb7fe640 1 -- 172.21.15.89:0/1138350531 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bec0ca790 con 0x7f1bec0ca240 2023-10-30T12:35:04.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.948+0000 7f1beb7fe640 1 --2- 172.21.15.89:0/1138350531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bec0ca240 0x7f1bec14ec40 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1be001c890 tx=0x7f1bd4005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:04.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.948+0000 7f1be8ff9640 1 -- 172.21.15.89:0/1138350531 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be00be070 con 0x7f1bec0ca240 2023-10-30T12:35:04.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.948+0000 7f1be8ff9640 1 -- 172.21.15.89:0/1138350531 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1be000f000 con 0x7f1bec0ca240 2023-10-30T12:35:04.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.948+0000 7f1bf229e640 1 -- 172.21.15.89:0/1138350531 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bec0caa70 con 0x7f1bec0ca240 2023-10-30T12:35:04.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.948+0000 7f1be8ff9640 1 -- 172.21.15.89:0/1138350531 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be00a5630 con 0x7f1bec0ca240 2023-10-30T12:35:04.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.948+0000 7f1bf229e640 1 -- 172.21.15.89:0/1138350531 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bec0c97e0 con 0x7f1bec0ca240 2023-10-30T12:35:04.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.948+0000 7f1bf229e640 1 -- 172.21.15.89:0/1138350531 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bec0cac00 con 0x7f1bec0ca240 2023-10-30T12:35:04.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.952+0000 7f1be8ff9640 1 -- 172.21.15.89:0/1138350531 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1be00b4040 con 0x7f1bec0ca240 2023-10-30T12:35:04.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.952+0000 7f1be8ff9640 1 --2- 172.21.15.89:0/1138350531 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1bd80664a0 0x7f1bd8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:04.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.952+0000 7f1be8ff9640 1 -- 172.21.15.89:0/1138350531 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1be011d950 con 0x7f1bec0ca240 2023-10-30T12:35:04.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.952+0000 7f1bebfff640 1 --2- 172.21.15.89:0/1138350531 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1bd80664a0 0x7f1bd8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:04.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.952+0000 7f1bebfff640 1 --2- 172.21.15.89:0/1138350531 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1bd80664a0 0x7f1bd8068960 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc001830 tx=0x7f1bdc007040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:04.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:04.956+0000 7f1be8ff9640 1 -- 172.21.15.89:0/1138350531 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1be00ead00 con 0x7f1bec0ca240 2023-10-30T12:35:05.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:05.140+0000 7f1bf229e640 1 -- 172.21.15.89:0/1138350531 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1bec0c6db0 con 0x7f1bec0ca240 2023-10-30T12:35:05.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:05.140+0000 7f1be8ff9640 1 -- 172.21.15.89:0/1138350531 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+5740 (secure 0 0 0) 0x7f1be00ee320 con 0x7f1bec0ca240 2023-10-30T12:35:05.146 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:35:05.147 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:35:01.355331+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:resolve","state_seq":21,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:05.148 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 34 2023-10-30T12:35:05.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:05.144+0000 7f1bf229e640 1 -- 172.21.15.89:0/1138350531 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1bd80664a0 msgr2=0x7f1bd8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:05.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:05.144+0000 7f1bf229e640 1 --2- 172.21.15.89:0/1138350531 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1bd80664a0 0x7f1bd8068960 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc001830 tx=0x7f1bdc007040 comp rx=0 tx=0).stop 2023-10-30T12:35:05.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:05.144+0000 7f1bf229e640 1 -- 172.21.15.89:0/1138350531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bec0ca240 msgr2=0x7f1bec14ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:05.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:05.144+0000 7f1bf229e640 1 --2- 172.21.15.89:0/1138350531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bec0ca240 0x7f1bec14ec40 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1be001c890 tx=0x7f1bd4005800 comp rx=0 tx=0).stop 2023-10-30T12:35:05.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:05.144+0000 7f1bf229e640 1 -- 172.21.15.89:0/1138350531 shutdown_connections 2023-10-30T12:35:05.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:05.144+0000 7f1bf229e640 1 --2- 172.21.15.89:0/1138350531 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1bd80664a0 0x7f1bd8068960 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:05.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:05.144+0000 7f1bf229e640 1 --2- 172.21.15.89:0/1138350531 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bec159640 0x7f1bec143990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:05.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:05.144+0000 7f1bf229e640 1 --2- 172.21.15.89:0/1138350531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bec0ca240 0x7f1bec14ec40 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:05.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:05.144+0000 7f1bf229e640 1 --2- 172.21.15.89:0/1138350531 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bd4003f00 0x7f1bec158af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:05.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:05.144+0000 7f1bf229e640 1 -- 172.21.15.89:0/1138350531 >> 172.21.15.89:0/1138350531 conn(0x7f1bec0c7910 msgr2=0x7f1bec06b560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:05.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:05.144+0000 7f1bf229e640 1 -- 172.21.15.89:0/1138350531 shutdown_connections 2023-10-30T12:35:05.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:05.144+0000 7f1bf229e640 1 -- 172.21.15.89:0/1138350531 wait complete. 2023-10-30T12:35:05.167 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:35:07.168 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:07.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.316+0000 7f1bd9654640 1 Processor -- start 2023-10-30T12:35:07.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.316+0000 7f1bd9654640 1 -- start start 2023-10-30T12:35:07.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.316+0000 7f1bd9654640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1bd41538a0 0x7f1bd415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:07.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.316+0000 7f1bd9654640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bd40cf160 0x7f1bd40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:07.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.316+0000 7f1bd9654640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd40cb800 con 0x7f1bd40ca0e0 2023-10-30T12:35:07.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.316+0000 7f1bd9654640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bd415dc30 con 0x7f1bd41538a0 2023-10-30T12:35:07.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.316+0000 7f1bd9654640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd40cb330 con 0x7f1bd40cf160 2023-10-30T12:35:07.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.316+0000 7f1bca7fc640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f1bd40ca0e0 0x7f1bd415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51934/0 (socket says 172.21.15.89:51934) 2023-10-30T12:35:07.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.316+0000 7f1bca7fc640 1 -- 172.21.15.89:0/3537266580 learned_addr learned my addr 172.21.15.89:0/3537266580 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:35:07.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.316+0000 7f1bd37fe640 1 --2- 172.21.15.89:0/3537266580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bd40cf160 0x7f1bd40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:07.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.316+0000 7f1bd2ffd640 1 --2- 172.21.15.89:0/3537266580 >> v2:172.21.15.192:3300/0 conn(0x7f1bd41538a0 0x7f1bd415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:07.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.316+0000 7f1bd27fc640 1 -- 172.21.15.89:0/3537266580 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2536700693 0 0) 0x7f1bd40cb800 con 0x7f1bd40ca0e0 2023-10-30T12:35:07.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.316+0000 7f1bd27fc640 1 -- 172.21.15.89:0/3537266580 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bc0003530 con 0x7f1bd40ca0e0 2023-10-30T12:35:07.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.316+0000 7f1bd27fc640 1 -- 172.21.15.89:0/3537266580 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2273176457 0 0) 0x7f1bc0003530 con 0x7f1bd40ca0e0 2023-10-30T12:35:07.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.316+0000 7f1bd27fc640 1 -- 172.21.15.89:0/3537266580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bd40cf160 msgr2=0x7f1bd40cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:07.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.316+0000 7f1bd27fc640 1 --2- 172.21.15.89:0/3537266580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bd40cf160 0x7f1bd40cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:07.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.316+0000 7f1bd27fc640 1 -- 172.21.15.89:0/3537266580 >> v2:172.21.15.192:3300/0 conn(0x7f1bd41538a0 msgr2=0x7f1bd415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:07.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.316+0000 7f1bd27fc640 1 --2- 172.21.15.89:0/3537266580 >> v2:172.21.15.192:3300/0 conn(0x7f1bd41538a0 0x7f1bd415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:07.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.320+0000 7f1bd27fc640 1 -- 172.21.15.89:0/3537266580 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd40cad70 con 0x7f1bd40ca0e0 2023-10-30T12:35:07.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.320+0000 7f1bd27fc640 1 -- 172.21.15.89:0/3537266580 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f1bc4001680 con 0x7f1bd40ca0e0 2023-10-30T12:35:07.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.320+0000 7f1bd27fc640 1 -- 172.21.15.89:0/3537266580 >> v1:172.21.15.89:6789/0 conn(0x7f1bd40ca0e0 legacy=0x7f1bd415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:07.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.320+0000 7f1bd27fc640 1 --2- 172.21.15.89:0/3537266580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bc0003980 0x7f1bc0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:07.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.320+0000 7f1bd27fc640 1 --2- 172.21.15.89:0/3537266580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bc00047d0 0x7f1bc0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:07.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.320+0000 7f1bd27fc640 1 --2- 172.21.15.89:0/3537266580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bd40cf160 0x7f1bc0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:07.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.320+0000 7f1bd27fc640 1 -- 172.21.15.89:0/3537266580 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd415dc30 con 0x7f1bc0003980 2023-10-30T12:35:07.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.320+0000 7f1bd27fc640 1 -- 172.21.15.89:0/3537266580 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd40cb330 con 0x7f1bc00047d0 2023-10-30T12:35:07.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.320+0000 7f1bd27fc640 1 -- 172.21.15.89:0/3537266580 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc000b990 con 0x7f1bd40cf160 2023-10-30T12:35:07.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.320+0000 7f1bd2ffd640 1 --2- 172.21.15.89:0/3537266580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bc0003980 0x7f1bc0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:07.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.320+0000 7f1bca7fc640 1 --2- 172.21.15.89:0/3537266580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bc00047d0 0x7f1bc0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:07.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.320+0000 7f1bd37fe640 1 --2- 172.21.15.89:0/3537266580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bd40cf160 0x7f1bc0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:07.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.320+0000 7f1bd2ffd640 1 -- 172.21.15.89:0/3537266580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bd40cf160 msgr2=0x7f1bc0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:07.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.320+0000 7f1bd2ffd640 1 --2- 172.21.15.89:0/3537266580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bd40cf160 0x7f1bc0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:07.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.320+0000 7f1bd2ffd640 1 -- 172.21.15.89:0/3537266580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bc00047d0 msgr2=0x7f1bc0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:07.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.320+0000 7f1bd2ffd640 1 --2- 172.21.15.89:0/3537266580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bc00047d0 0x7f1bc0004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:07.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.320+0000 7f1bd2ffd640 1 -- 172.21.15.89:0/3537266580 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc0003530 con 0x7f1bc0003980 2023-10-30T12:35:07.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.320+0000 7f1bd37fe640 1 --2- 172.21.15.89:0/3537266580 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bd40cf160 0x7f1bc0005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:35:07.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.320+0000 7f1bca7fc640 1 --2- 172.21.15.89:0/3537266580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bc00047d0 0x7f1bc0004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:35:07.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.320+0000 7f1bd2ffd640 1 --2- 172.21.15.89:0/3537266580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bc0003980 0x7f1bc0003da0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc009ae0 tx=0x7f1bbc01ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=708c4e17548654fc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:07.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.320+0000 7f1bd27fc640 1 -- 172.21.15.89:0/3537266580 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc0a55a0 con 0x7f1bc0003980 2023-10-30T12:35:07.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.320+0000 7f1bd27fc640 1 -- 172.21.15.89:0/3537266580 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bbc0a5700 con 0x7f1bc0003980 2023-10-30T12:35:07.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.320+0000 7f1bd27fc640 1 -- 172.21.15.89:0/3537266580 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc0a5ab0 con 0x7f1bc0003980 2023-10-30T12:35:07.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.324+0000 7f1bd9654640 1 -- 172.21.15.89:0/3537266580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bc0003980 msgr2=0x7f1bc0003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:07.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.324+0000 7f1bd9654640 1 --2- 172.21.15.89:0/3537266580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bc0003980 0x7f1bc0003da0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc009ae0 tx=0x7f1bbc01ca90 comp rx=0 tx=0).stop 2023-10-30T12:35:07.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.324+0000 7f1bd37fe640 1 -- 172.21.15.89:0/3537266580 reap_dead start 2023-10-30T12:35:07.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.324+0000 7f1bd9654640 1 -- 172.21.15.89:0/3537266580 shutdown_connections 2023-10-30T12:35:07.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.324+0000 7f1bd9654640 1 -- 172.21.15.89:0/3537266580 >> 172.21.15.89:0/3537266580 conn(0x7f1bd406bfe0 msgr2=0x7f1bd406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:07.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.324+0000 7f1bd9654640 1 -- 172.21.15.89:0/3537266580 shutdown_connections 2023-10-30T12:35:07.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.324+0000 7f1bd9654640 1 -- 172.21.15.89:0/3537266580 wait complete. 2023-10-30T12:35:07.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.324+0000 7f1bd9654640 1 Processor -- start 2023-10-30T12:35:07.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.324+0000 7f1bd9654640 1 -- start start 2023-10-30T12:35:07.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.324+0000 7f1bd9654640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bc0003980 0x7f1bd4145af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:07.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.324+0000 7f1bd9654640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bd40b31a0 0x7f1bd413cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:07.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.324+0000 7f1bd9654640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bd40cde60 0x7f1bd413d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:07.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.324+0000 7f1bd9654640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd4142420 con 0x7f1bc0003980 2023-10-30T12:35:07.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.324+0000 7f1bd9654640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd40b36e0 con 0x7f1bd40b31a0 2023-10-30T12:35:07.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.324+0000 7f1bd9654640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd4142670 con 0x7f1bd40cde60 2023-10-30T12:35:07.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.324+0000 7f1bd2ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bc0003980 0x7f1bd4145af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:07.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.324+0000 7f1bd2ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bc0003980 0x7f1bd4145af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58548/0 (socket says 172.21.15.89:58548) 2023-10-30T12:35:07.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.324+0000 7f1bd2ffd640 1 -- 172.21.15.89:0/188165112 learned_addr learned my addr 172.21.15.89:0/188165112 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:35:07.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.324+0000 7f1bca7fc640 1 --2- 172.21.15.89:0/188165112 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bd40b31a0 0x7f1bd413cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:07.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.324+0000 7f1bd37fe640 1 --2- 172.21.15.89:0/188165112 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bd40cde60 0x7f1bd413d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:07.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.328+0000 7f1bd2ffd640 1 -- 172.21.15.89:0/188165112 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bd40cde60 msgr2=0x7f1bd413d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:07.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.328+0000 7f1bd2ffd640 1 --2- 172.21.15.89:0/188165112 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bd40cde60 0x7f1bd413d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:07.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.328+0000 7f1bd2ffd640 1 -- 172.21.15.89:0/188165112 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bd40b31a0 msgr2=0x7f1bd413cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:07.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.328+0000 7f1bd2ffd640 1 --2- 172.21.15.89:0/188165112 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bd40b31a0 0x7f1bd413cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:07.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.328+0000 7f1bd2ffd640 1 -- 172.21.15.89:0/188165112 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd4142950 con 0x7f1bc0003980 2023-10-30T12:35:07.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.328+0000 7f1bd2ffd640 1 --2- 172.21.15.89:0/188165112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bc0003980 0x7f1bd4145af0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc009ae0 tx=0x7f1bc0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:07.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.328+0000 7f1bcbfff640 1 -- 172.21.15.89:0/188165112 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc0be070 con 0x7f1bc0003980 2023-10-30T12:35:07.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.328+0000 7f1bcbfff640 1 -- 172.21.15.89:0/188165112 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bbc0a5e70 con 0x7f1bc0003980 2023-10-30T12:35:07.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.328+0000 7f1bcbfff640 1 -- 172.21.15.89:0/188165112 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc004220 con 0x7f1bc0003980 2023-10-30T12:35:07.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.328+0000 7f1bd9654640 1 -- 172.21.15.89:0/188165112 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bd4142c30 con 0x7f1bc0003980 2023-10-30T12:35:07.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.328+0000 7f1bd9654640 1 -- 172.21.15.89:0/188165112 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bd4146030 con 0x7f1bc0003980 2023-10-30T12:35:07.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.328+0000 7f1bd9654640 1 -- 172.21.15.89:0/188165112 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bd40b4290 con 0x7f1bc0003980 2023-10-30T12:35:07.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.328+0000 7f1bcbfff640 1 -- 172.21.15.89:0/188165112 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1bbc0b4040 con 0x7f1bc0003980 2023-10-30T12:35:07.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.328+0000 7f1bcbfff640 1 --2- 172.21.15.89:0/188165112 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1bb40664a0 0x7f1bb4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:07.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.328+0000 7f1bcbfff640 1 -- 172.21.15.89:0/188165112 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1bbc11da60 con 0x7f1bc0003980 2023-10-30T12:35:07.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.332+0000 7f1bca7fc640 1 --2- 172.21.15.89:0/188165112 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1bb40664a0 0x7f1bb4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:07.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.332+0000 7f1bca7fc640 1 --2- 172.21.15.89:0/188165112 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1bb40664a0 0x7f1bb4068960 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1bd40b3c60 tx=0x7f1bc4001580 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:07.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.336+0000 7f1bcbfff640 1 -- 172.21.15.89:0/188165112 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1bbc0ead80 con 0x7f1bc0003980 2023-10-30T12:35:07.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.512+0000 7f1bd9654640 1 -- 172.21.15.89:0/188165112 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1bd41427b0 con 0x7f1bc0003980 2023-10-30T12:35:07.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.516+0000 7f1bcbfff640 1 -- 172.21.15.89:0/188165112 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+5742 (secure 0 0 0) 0x7f1bd41427b0 con 0x7f1bc0003980 2023-10-30T12:35:07.520 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:35:07.520 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:35:07.217609+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:reconnect","state_seq":23,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:07.520 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 35 2023-10-30T12:35:07.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.516+0000 7f1bd9654640 1 -- 172.21.15.89:0/188165112 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1bb40664a0 msgr2=0x7f1bb4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:07.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.516+0000 7f1bd9654640 1 --2- 172.21.15.89:0/188165112 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1bb40664a0 0x7f1bb4068960 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1bd40b3c60 tx=0x7f1bc4001580 comp rx=0 tx=0).stop 2023-10-30T12:35:07.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.516+0000 7f1bd9654640 1 -- 172.21.15.89:0/188165112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bc0003980 msgr2=0x7f1bd4145af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:07.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.516+0000 7f1bd9654640 1 --2- 172.21.15.89:0/188165112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bc0003980 0x7f1bd4145af0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc009ae0 tx=0x7f1bc0005800 comp rx=0 tx=0).stop 2023-10-30T12:35:07.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.516+0000 7f1bd9654640 1 -- 172.21.15.89:0/188165112 shutdown_connections 2023-10-30T12:35:07.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.516+0000 7f1bd9654640 1 --2- 172.21.15.89:0/188165112 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1bb40664a0 0x7f1bb4068960 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:07.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.516+0000 7f1bd9654640 1 --2- 172.21.15.89:0/188165112 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1bd40b31a0 0x7f1bd413cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:07.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.516+0000 7f1bd9654640 1 --2- 172.21.15.89:0/188165112 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1bd40cde60 0x7f1bd413d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:07.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.516+0000 7f1bd9654640 1 --2- 172.21.15.89:0/188165112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1bc0003980 0x7f1bd4145af0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:07.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.516+0000 7f1bd9654640 1 -- 172.21.15.89:0/188165112 >> 172.21.15.89:0/188165112 conn(0x7f1bd406bfe0 msgr2=0x7f1bd41525f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:07.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.516+0000 7f1bd9654640 1 -- 172.21.15.89:0/188165112 shutdown_connections 2023-10-30T12:35:07.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:07.516+0000 7f1bd9654640 1 -- 172.21.15.89:0/188165112 wait complete. 2023-10-30T12:35:07.541 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:35:09.542 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:09.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.688+0000 7f9e9d69e640 1 Processor -- start 2023-10-30T12:35:09.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.688+0000 7f9e9d69e640 1 -- start start 2023-10-30T12:35:09.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.688+0000 7f9e9d69e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e980c05d0 0x7f9e9813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:09.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.688+0000 7f9e9d69e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9e980c6f60 0x7f9e9813f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:09.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.688+0000 7f9e9d69e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e980c0a50 con 0x7f9e980c6b80 2023-10-30T12:35:09.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.688+0000 7f9e9d69e640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e9806f6e0 con 0x7f9e980c6f60 2023-10-30T12:35:09.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.688+0000 7f9e9d69e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9806f9c0 con 0x7f9e980c05d0 2023-10-30T12:35:09.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.688+0000 7f9e967fc640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e980c05d0 0x7f9e9813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:09.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.688+0000 7f9e977fe640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9e980c6f60 0x7f9e9813f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:09.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.688+0000 7f9e977fe640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9e980c6f60 0x7f9e9813f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:39040/0 (socket says 172.21.15.89:39040) 2023-10-30T12:35:09.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.688+0000 7f9e977fe640 1 -- 172.21.15.89:0/328819673 learned_addr learned my addr 172.21.15.89:0/328819673 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:35:09.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.688+0000 7f9e967fc640 1 -- 172.21.15.89:0/328819673 >> v2:172.21.15.192:3300/0 conn(0x7f9e980c6f60 msgr2=0x7f9e9813f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:09.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.688+0000 7f9e967fc640 1 --2- 172.21.15.89:0/328819673 >> v2:172.21.15.192:3300/0 conn(0x7f9e980c6f60 0x7f9e9813f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:09.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.688+0000 7f9e967fc640 1 -- 172.21.15.89:0/328819673 >> v1:172.21.15.192:6789/0 conn(0x7f9e980c6b80 legacy=0x7f9e980bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:09.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.688+0000 7f9e967fc640 1 -- 172.21.15.89:0/328819673 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e9806fca0 con 0x7f9e980c05d0 2023-10-30T12:35:09.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.688+0000 7f9e977fe640 1 --2- 172.21.15.89:0/328819673 >> v2:172.21.15.192:3300/0 conn(0x7f9e980c6f60 0x7f9e9813f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:35:09.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.692+0000 7f9e967fc640 1 --2- 172.21.15.89:0/328819673 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e980c05d0 0x7f9e9813ed90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c0097c0 tx=0x7f9e8c01c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b127a208fd623482 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:09.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.692+0000 7f9e95ffb640 1 -- 172.21.15.89:0/328819673 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c0ad640 con 0x7f9e980c05d0 2023-10-30T12:35:09.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.692+0000 7f9e95ffb640 1 -- 172.21.15.89:0/328819673 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e8c0ad7a0 con 0x7f9e980c05d0 2023-10-30T12:35:09.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.692+0000 7f9e95ffb640 1 -- 172.21.15.89:0/328819673 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c0adb30 con 0x7f9e980c05d0 2023-10-30T12:35:09.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.692+0000 7f9e9d69e640 1 -- 172.21.15.89:0/328819673 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e980c05d0 msgr2=0x7f9e9813ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:09.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.692+0000 7f9e9d69e640 1 --2- 172.21.15.89:0/328819673 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e980c05d0 0x7f9e9813ed90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c0097c0 tx=0x7f9e8c01c480 comp rx=0 tx=0).stop 2023-10-30T12:35:09.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.692+0000 7f9e9d69e640 1 -- 172.21.15.89:0/328819673 shutdown_connections 2023-10-30T12:35:09.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.692+0000 7f9e9d69e640 1 --2- 172.21.15.89:0/328819673 >> v2:172.21.15.192:3300/0 conn(0x7f9e980c6f60 0x7f9e9813f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:09.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.692+0000 7f9e9d69e640 1 --2- 172.21.15.89:0/328819673 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e980c05d0 0x7f9e9813ed90 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:09.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.692+0000 7f9e9d69e640 1 -- 172.21.15.89:0/328819673 >> 172.21.15.89:0/328819673 conn(0x7f9e980c7910 msgr2=0x7f9e980c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:09.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.692+0000 7f9e9d69e640 1 -- 172.21.15.89:0/328819673 shutdown_connections 2023-10-30T12:35:09.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.692+0000 7f9e9d69e640 1 -- 172.21.15.89:0/328819673 wait complete. 2023-10-30T12:35:09.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.696+0000 7f9e9d69e640 1 Processor -- start 2023-10-30T12:35:09.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.696+0000 7f9e9d69e640 1 -- start start 2023-10-30T12:35:09.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.696+0000 7f9e9d69e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e980c05d0 0x7f9e98154350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:09.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.696+0000 7f9e9d69e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e980c6b80 0x7f9e980c4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:09.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.696+0000 7f9e9d69e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e980c6f60 0x7f9e981f2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:09.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.696+0000 7f9e9d69e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e98164000 con 0x7f9e980c6f60 2023-10-30T12:35:09.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.696+0000 7f9e9d69e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e98161f60 con 0x7f9e980c05d0 2023-10-30T12:35:09.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.696+0000 7f9e9d69e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e981f2360 con 0x7f9e980c6b80 2023-10-30T12:35:09.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.696+0000 7f9e977fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e980c6f60 0x7f9e981f2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:09.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.696+0000 7f9e977fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e980c6f60 0x7f9e981f2a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58562/0 (socket says 172.21.15.89:58562) 2023-10-30T12:35:09.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.696+0000 7f9e977fe640 1 -- 172.21.15.89:0/713483252 learned_addr learned my addr 172.21.15.89:0/713483252 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:35:09.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.696+0000 7f9e96ffd640 1 --2- 172.21.15.89:0/713483252 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e980c05d0 0x7f9e98154350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:09.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.696+0000 7f9e967fc640 1 --2- 172.21.15.89:0/713483252 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e980c6b80 0x7f9e980c4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:09.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.696+0000 7f9e977fe640 1 -- 172.21.15.89:0/713483252 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e980c6b80 msgr2=0x7f9e980c4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:09.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.696+0000 7f9e977fe640 1 --2- 172.21.15.89:0/713483252 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e980c6b80 0x7f9e980c4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:09.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.696+0000 7f9e977fe640 1 -- 172.21.15.89:0/713483252 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e980c05d0 msgr2=0x7f9e98154350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:09.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.696+0000 7f9e977fe640 1 --2- 172.21.15.89:0/713483252 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e980c05d0 0x7f9e98154350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:09.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.696+0000 7f9e977fe640 1 -- 172.21.15.89:0/713483252 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e90009b30 con 0x7f9e980c6f60 2023-10-30T12:35:09.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.696+0000 7f9e96ffd640 1 --2- 172.21.15.89:0/713483252 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e980c05d0 0x7f9e98154350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:35:09.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.696+0000 7f9e967fc640 1 --2- 172.21.15.89:0/713483252 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e980c6b80 0x7f9e980c4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:35:09.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.696+0000 7f9e977fe640 1 --2- 172.21.15.89:0/713483252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e980c6f60 0x7f9e981f2a90 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f9e90009e70 tx=0x7f9e9000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:09.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.696+0000 7f9e7f7fe640 1 -- 172.21.15.89:0/713483252 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e90018070 con 0x7f9e980c6f60 2023-10-30T12:35:09.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.696+0000 7f9e7f7fe640 1 -- 172.21.15.89:0/713483252 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e9000f040 con 0x7f9e980c6f60 2023-10-30T12:35:09.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.696+0000 7f9e9d69e640 1 -- 172.21.15.89:0/713483252 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e8c009420 con 0x7f9e980c6f60 2023-10-30T12:35:09.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.696+0000 7f9e7f7fe640 1 -- 172.21.15.89:0/713483252 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9000ce00 con 0x7f9e980c6f60 2023-10-30T12:35:09.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.696+0000 7f9e9d69e640 1 -- 172.21.15.89:0/713483252 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e98156500 con 0x7f9e980c6f60 2023-10-30T12:35:09.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.700+0000 7f9e9d69e640 1 -- 172.21.15.89:0/713483252 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e98154890 con 0x7f9e980c6f60 2023-10-30T12:35:09.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.700+0000 7f9e7f7fe640 1 -- 172.21.15.89:0/713483252 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9e90014480 con 0x7f9e980c6f60 2023-10-30T12:35:09.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.700+0000 7f9e7f7fe640 1 --2- 172.21.15.89:0/713483252 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9e840664a0 0x7f9e84068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:09.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.700+0000 7f9e7f7fe640 1 -- 172.21.15.89:0/713483252 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9e90084180 con 0x7f9e980c6f60 2023-10-30T12:35:09.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.700+0000 7f9e96ffd640 1 --2- 172.21.15.89:0/713483252 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9e840664a0 0x7f9e84068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:09.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.700+0000 7f9e96ffd640 1 --2- 172.21.15.89:0/713483252 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9e840664a0 0x7f9e84068960 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9e88000f70 tx=0x7f9e88009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:09.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.704+0000 7f9e7f7fe640 1 -- 172.21.15.89:0/713483252 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9e90051400 con 0x7f9e980c6f60 2023-10-30T12:35:09.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.884+0000 7f9e9d69e640 1 -- 172.21.15.89:0/713483252 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e98156500 con 0x7f9e980c6f60 2023-10-30T12:35:09.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.884+0000 7f9e7f7fe640 1 -- 172.21.15.89:0/713483252 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+5742 (secure 0 0 0) 0x7f9e98156500 con 0x7f9e980c6f60 2023-10-30T12:35:09.888 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:35:09.889 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:35:07.217609+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:reconnect","state_seq":23,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:09.889 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 35 2023-10-30T12:35:09.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.884+0000 7f9e9d69e640 1 -- 172.21.15.89:0/713483252 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9e840664a0 msgr2=0x7f9e84068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:09.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.884+0000 7f9e9d69e640 1 --2- 172.21.15.89:0/713483252 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9e840664a0 0x7f9e84068960 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9e88000f70 tx=0x7f9e88009040 comp rx=0 tx=0).stop 2023-10-30T12:35:09.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.884+0000 7f9e9d69e640 1 -- 172.21.15.89:0/713483252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e980c6f60 msgr2=0x7f9e981f2a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:09.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.884+0000 7f9e9d69e640 1 --2- 172.21.15.89:0/713483252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e980c6f60 0x7f9e981f2a90 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f9e90009e70 tx=0x7f9e9000c490 comp rx=0 tx=0).stop 2023-10-30T12:35:09.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.884+0000 7f9e9d69e640 1 -- 172.21.15.89:0/713483252 shutdown_connections 2023-10-30T12:35:09.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.884+0000 7f9e9d69e640 1 --2- 172.21.15.89:0/713483252 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9e840664a0 0x7f9e84068960 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:09.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.884+0000 7f9e9d69e640 1 --2- 172.21.15.89:0/713483252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e980c6f60 0x7f9e981f2a90 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:09.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.884+0000 7f9e9d69e640 1 --2- 172.21.15.89:0/713483252 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e980c6b80 0x7f9e980c4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:09.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.884+0000 7f9e9d69e640 1 --2- 172.21.15.89:0/713483252 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e980c05d0 0x7f9e98154350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:09.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.884+0000 7f9e9d69e640 1 -- 172.21.15.89:0/713483252 >> 172.21.15.89:0/713483252 conn(0x7f9e980c7910 msgr2=0x7f9e9806a720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:09.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.884+0000 7f9e9d69e640 1 -- 172.21.15.89:0/713483252 shutdown_connections 2023-10-30T12:35:09.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:09.884+0000 7f9e9d69e640 1 -- 172.21.15.89:0/713483252 wait complete. 2023-10-30T12:35:09.909 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 35, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:50.729245+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 35, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4318": { "addr": "172.21.15.192:6837/1002124471", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6836", "nonce": 1002124471, "type": "v2" }, { "addr": "172.21.15.192:6837", "nonce": 1002124471, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4318, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4781": { "addr": "172.21.15.192:6835/654256299", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6834", "nonce": 654256299, "type": "v2" }, { "addr": "172.21.15.192:6835", "nonce": 654256299, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4781, "incarnation": 24, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 27 }, "gid_5139": { "addr": "172.21.15.89:6837/951276252", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6836", "nonce": 951276252, "type": "v2" }, { "addr": "172.21.15.89:6837", "nonce": 951276252, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5139, "incarnation": 33, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:reconnect", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 44, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:35:07.217609+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4318, "mds_1": 5139, "mds_2": 4781 } } } ], "standbys": [ { "addr": "172.21.15.89:6834/2909790173", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6832", "nonce": 2909790173, "type": "v2" }, { "addr": "172.21.15.89:6834", "nonce": 2909790173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.89:6835/2387357615", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6833", "nonce": 2387357615, "type": "v2" }, { "addr": "172.21.15.89:6835", "nonce": 2387357615, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4239, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:35:11.911 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:12.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.056+0000 7fa2fda05640 1 Processor -- start 2023-10-30T12:35:12.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.056+0000 7fa2fda05640 1 -- start start 2023-10-30T12:35:12.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.060+0000 7fa2fda05640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa2f80c6b80 0x7fa2f80bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:12.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.060+0000 7fa2fda05640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa2f80c6f60 0x7fa2f80c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:12.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.060+0000 7fa2fda05640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f80c0a50 con 0x7fa2f80c8430 2023-10-30T12:35:12.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.060+0000 7fa2fda05640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2f806f6e0 con 0x7fa2f80c6b80 2023-10-30T12:35:12.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.060+0000 7fa2fda05640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2f806f9c0 con 0x7fa2f80c6f60 2023-10-30T12:35:12.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.060+0000 7fa2f6ffd640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa2f80c6b80 0x7fa2f80bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:12.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.060+0000 7fa2f77fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa2f80c8430 0x7fa2f813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44356/0 (socket says 172.21.15.89:44356) 2023-10-30T12:35:12.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.060+0000 7fa2f77fe640 1 -- 172.21.15.89:0/2204509931 learned_addr learned my addr 172.21.15.89:0/2204509931 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:35:12.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.060+0000 7fa2ee7fc640 1 --2- 172.21.15.89:0/2204509931 >> v2:172.21.15.192:3300/0 conn(0x7fa2f80c6f60 0x7fa2f80c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:12.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.060+0000 7fa2f67fc640 1 -- 172.21.15.89:0/2204509931 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1024548773 0 0) 0x7fa2f80c0a50 con 0x7fa2f80c8430 2023-10-30T12:35:12.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.060+0000 7fa2f67fc640 1 -- 172.21.15.89:0/2204509931 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2e4003530 con 0x7fa2f80c8430 2023-10-30T12:35:12.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.060+0000 7fa2f6ffd640 1 -- 172.21.15.89:0/2204509931 >> v2:172.21.15.192:3300/0 conn(0x7fa2f80c6f60 msgr2=0x7fa2f80c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:12.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.060+0000 7fa2f6ffd640 1 --2- 172.21.15.89:0/2204509931 >> v2:172.21.15.192:3300/0 conn(0x7fa2f80c6f60 0x7fa2f80c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:12.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.060+0000 7fa2f6ffd640 1 -- 172.21.15.89:0/2204509931 >> v1:172.21.15.89:6789/0 conn(0x7fa2f80c8430 legacy=0x7fa2f813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:12.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.060+0000 7fa2f6ffd640 1 -- 172.21.15.89:0/2204509931 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f806fca0 con 0x7fa2f80c6b80 2023-10-30T12:35:12.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.060+0000 7fa2ee7fc640 1 --2- 172.21.15.89:0/2204509931 >> v2:172.21.15.192:3300/0 conn(0x7fa2f80c6f60 0x7fa2f80c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:35:12.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.064+0000 7fa2f6ffd640 1 --2- 172.21.15.89:0/2204509931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f80c6b80 0x7fa2f80bfec0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8159b90 tx=0x7fa2e801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92a6f96dd5e9c5c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:12.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.064+0000 7fa2f67fc640 1 -- 172.21.15.89:0/2204509931 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e80ab600 con 0x7fa2f80c6b80 2023-10-30T12:35:12.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.064+0000 7fa2f67fc640 1 -- 172.21.15.89:0/2204509931 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e80a7050 con 0x7fa2f80c6b80 2023-10-30T12:35:12.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.064+0000 7fa2f67fc640 1 -- 172.21.15.89:0/2204509931 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e80ab980 con 0x7fa2f80c6b80 2023-10-30T12:35:12.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.064+0000 7fa2fda05640 1 -- 172.21.15.89:0/2204509931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f80c6b80 msgr2=0x7fa2f80bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:12.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.064+0000 7fa2fda05640 1 --2- 172.21.15.89:0/2204509931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f80c6b80 0x7fa2f80bfec0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8159b90 tx=0x7fa2e801c460 comp rx=0 tx=0).stop 2023-10-30T12:35:12.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.064+0000 7fa2fda05640 1 -- 172.21.15.89:0/2204509931 shutdown_connections 2023-10-30T12:35:12.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.064+0000 7fa2fda05640 1 --2- 172.21.15.89:0/2204509931 >> v2:172.21.15.192:3300/0 conn(0x7fa2f80c6f60 0x7fa2f80c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:12.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.064+0000 7fa2fda05640 1 --2- 172.21.15.89:0/2204509931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f80c6b80 0x7fa2f80bfec0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:12.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.064+0000 7fa2fda05640 1 -- 172.21.15.89:0/2204509931 >> 172.21.15.89:0/2204509931 conn(0x7fa2f80c7910 msgr2=0x7fa2f80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:12.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.064+0000 7fa2fda05640 1 -- 172.21.15.89:0/2204509931 shutdown_connections 2023-10-30T12:35:12.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.064+0000 7fa2fda05640 1 -- 172.21.15.89:0/2204509931 wait complete. 2023-10-30T12:35:12.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.064+0000 7fa2fda05640 1 Processor -- start 2023-10-30T12:35:12.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.064+0000 7fa2fda05640 1 -- start start 2023-10-30T12:35:12.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.064+0000 7fa2fda05640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2f80c6b80 0x7fa2f80c47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:12.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.064+0000 7fa2fda05640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2f80c6f60 0x7fa2f80c4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:12.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.064+0000 7fa2fda05640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f80c8430 0x7fa2f81394c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:12.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.064+0000 7fa2fda05640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f8156280 con 0x7fa2f80c8430 2023-10-30T12:35:12.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.064+0000 7fa2fda05640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f8160300 con 0x7fa2f80c6b80 2023-10-30T12:35:12.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.064+0000 7fa2fda05640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f81f2ec0 con 0x7fa2f80c6f60 2023-10-30T12:35:12.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.064+0000 7fa2f77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f80c8430 0x7fa2f81394c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:12.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.068+0000 7fa2f77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f80c8430 0x7fa2f81394c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41536/0 (socket says 172.21.15.89:41536) 2023-10-30T12:35:12.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.068+0000 7fa2f77fe640 1 -- 172.21.15.89:0/3854847463 learned_addr learned my addr 172.21.15.89:0/3854847463 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:35:12.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.068+0000 7fa2f6ffd640 1 --2- 172.21.15.89:0/3854847463 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2f80c6b80 0x7fa2f80c47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:12.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.068+0000 7fa2ee7fc640 1 --2- 172.21.15.89:0/3854847463 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2f80c6f60 0x7fa2f80c4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:12.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.068+0000 7fa2f77fe640 1 -- 172.21.15.89:0/3854847463 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2f80c6f60 msgr2=0x7fa2f80c4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:12.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.068+0000 7fa2f77fe640 1 --2- 172.21.15.89:0/3854847463 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2f80c6f60 0x7fa2f80c4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:12.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.068+0000 7fa2f77fe640 1 -- 172.21.15.89:0/3854847463 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2f80c6b80 msgr2=0x7fa2f80c47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:12.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.068+0000 7fa2f77fe640 1 --2- 172.21.15.89:0/3854847463 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2f80c6b80 0x7fa2f80c47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:12.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.068+0000 7fa2f77fe640 1 -- 172.21.15.89:0/3854847463 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e8009420 con 0x7fa2f80c8430 2023-10-30T12:35:12.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.068+0000 7fa2f77fe640 1 --2- 172.21.15.89:0/3854847463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f80c8430 0x7fa2f81394c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa2f000c3c0 tx=0x7fa2f000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:12.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.068+0000 7fa2f4ff9640 1 -- 172.21.15.89:0/3854847463 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f0005cf0 con 0x7fa2f80c8430 2023-10-30T12:35:12.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.068+0000 7fa2f4ff9640 1 -- 172.21.15.89:0/3854847463 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f0005e50 con 0x7fa2f80c8430 2023-10-30T12:35:12.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.068+0000 7fa2fda05640 1 -- 172.21.15.89:0/3854847463 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f8157600 con 0x7fa2f80c8430 2023-10-30T12:35:12.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.068+0000 7fa2f4ff9640 1 -- 172.21.15.89:0/3854847463 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f0007310 con 0x7fa2f80c8430 2023-10-30T12:35:12.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.068+0000 7fa2fda05640 1 -- 172.21.15.89:0/3854847463 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f81590f0 con 0x7fa2f80c8430 2023-10-30T12:35:12.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.068+0000 7fa2fda05640 1 -- 172.21.15.89:0/3854847463 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f8157b20 con 0x7fa2f80c8430 2023-10-30T12:35:12.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.068+0000 7fa2f4ff9640 1 -- 172.21.15.89:0/3854847463 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa2f0007470 con 0x7fa2f80c8430 2023-10-30T12:35:12.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.068+0000 7fa2f4ff9640 1 --2- 172.21.15.89:0/3854847463 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa2d80664a0 0x7fa2d8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:12.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.068+0000 7fa2f4ff9640 1 -- 172.21.15.89:0/3854847463 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa2f00834b0 con 0x7fa2f80c8430 2023-10-30T12:35:12.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.072+0000 7fa2f6ffd640 1 --2- 172.21.15.89:0/3854847463 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa2d80664a0 0x7fa2d8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:12.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.072+0000 7fa2f6ffd640 1 --2- 172.21.15.89:0/3854847463 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa2d80664a0 0x7fa2d8068960 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa2e80ad000 tx=0x7fa2e8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:12.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.072+0000 7fa2f4ff9640 1 -- 172.21.15.89:0/3854847463 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa2f8157b20 con 0x7fa2f80c8430 2023-10-30T12:35:12.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.256+0000 7fa2fda05640 1 -- 172.21.15.89:0/3854847463 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2f8158f50 con 0x7fa2f80c8430 2023-10-30T12:35:12.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.256+0000 7fa2f4ff9640 1 -- 172.21.15.89:0/3854847463 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+5742 (secure 0 0 0) 0x7fa2f8158f50 con 0x7fa2f80c8430 2023-10-30T12:35:12.261 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:35:12.261 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:35:07.217609+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:reconnect","state_seq":23,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:12.262 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 35 2023-10-30T12:35:12.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.256+0000 7fa2fda05640 1 -- 172.21.15.89:0/3854847463 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa2d80664a0 msgr2=0x7fa2d8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:12.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.256+0000 7fa2fda05640 1 --2- 172.21.15.89:0/3854847463 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa2d80664a0 0x7fa2d8068960 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa2e80ad000 tx=0x7fa2e8008040 comp rx=0 tx=0).stop 2023-10-30T12:35:12.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.256+0000 7fa2fda05640 1 -- 172.21.15.89:0/3854847463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f80c8430 msgr2=0x7fa2f81394c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:12.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.256+0000 7fa2fda05640 1 --2- 172.21.15.89:0/3854847463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f80c8430 0x7fa2f81394c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa2f000c3c0 tx=0x7fa2f000c880 comp rx=0 tx=0).stop 2023-10-30T12:35:12.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.256+0000 7fa2fda05640 1 -- 172.21.15.89:0/3854847463 shutdown_connections 2023-10-30T12:35:12.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.256+0000 7fa2fda05640 1 --2- 172.21.15.89:0/3854847463 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa2d80664a0 0x7fa2d8068960 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:12.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.256+0000 7fa2fda05640 1 --2- 172.21.15.89:0/3854847463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f80c8430 0x7fa2f81394c0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:12.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.256+0000 7fa2fda05640 1 --2- 172.21.15.89:0/3854847463 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2f80c6f60 0x7fa2f80c4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:12.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.256+0000 7fa2fda05640 1 --2- 172.21.15.89:0/3854847463 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2f80c6b80 0x7fa2f80c47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:12.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.256+0000 7fa2fda05640 1 -- 172.21.15.89:0/3854847463 >> 172.21.15.89:0/3854847463 conn(0x7fa2f80c7910 msgr2=0x7fa2f80c1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:12.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.256+0000 7fa2fda05640 1 -- 172.21.15.89:0/3854847463 shutdown_connections 2023-10-30T12:35:12.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:12.256+0000 7fa2fda05640 1 -- 172.21.15.89:0/3854847463 wait complete. 2023-10-30T12:35:12.283 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:35:14.284 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:14.435 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.432+0000 7fb0da380640 1 Processor -- start 2023-10-30T12:35:14.436 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.432+0000 7fb0da380640 1 -- start start 2023-10-30T12:35:14.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.432+0000 7fb0da380640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb0d4071110 0x7fb0d406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:14.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.432+0000 7fb0da380640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d4066cf0 con 0x7fb0d40c6bd0 2023-10-30T12:35:14.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.432+0000 7fb0da380640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d40c70d0 con 0x7fb0d406f7a0 2023-10-30T12:35:14.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.432+0000 7fb0da380640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0d40700c0 con 0x7fb0d4071110 2023-10-30T12:35:14.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.432+0000 7fb0d891d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb0d4071110 0x7fb0d406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:14.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.432+0000 7fb0d37fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb0d40c6bd0 0x7fb0d4158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44372/0 (socket says 172.21.15.89:44372) 2023-10-30T12:35:14.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.432+0000 7fb0d37fe640 1 -- 172.21.15.89:0/2153865296 learned_addr learned my addr 172.21.15.89:0/2153865296 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:35:14.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.432+0000 7fb0d891d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb0d4071110 0x7fb0d406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41550/0 (socket says 172.21.15.89:41550) 2023-10-30T12:35:14.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.432+0000 7fb0d891d640 1 -- 172.21.15.89:0/2153865296 >> v1:172.21.15.192:6789/0 conn(0x7fb0d406f7a0 legacy=0x7fb0d4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:14.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.432+0000 7fb0d891d640 1 -- 172.21.15.89:0/2153865296 >> v1:172.21.15.89:6789/0 conn(0x7fb0d40c6bd0 legacy=0x7fb0d4158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:14.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.432+0000 7fb0d891d640 1 -- 172.21.15.89:0/2153865296 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d40c7390 con 0x7fb0d4071110 2023-10-30T12:35:14.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.432+0000 7fb0d2ffd640 1 -- 172.21.15.89:0/2153865296 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2866953551 0 0) 0x7fb0d4066cf0 con 0x7fb0d40c6bd0 2023-10-30T12:35:14.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.436+0000 7fb0d891d640 1 --2- 172.21.15.89:0/2153865296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0d4071110 0x7fb0d406fb80 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc009d90 tx=0x7fb0cc01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50a10909362d90e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:14.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.436+0000 7fb0d2ffd640 1 -- 172.21.15.89:0/2153865296 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc0af600 con 0x7fb0d4071110 2023-10-30T12:35:14.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.436+0000 7fb0d2ffd640 1 -- 172.21.15.89:0/2153865296 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0cc0ab050 con 0x7fb0d4071110 2023-10-30T12:35:14.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.436+0000 7fb0d2ffd640 1 -- 172.21.15.89:0/2153865296 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc0af980 con 0x7fb0d4071110 2023-10-30T12:35:14.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.436+0000 7fb0da380640 1 -- 172.21.15.89:0/2153865296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0d4071110 msgr2=0x7fb0d406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:14.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.436+0000 7fb0da380640 1 --2- 172.21.15.89:0/2153865296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0d4071110 0x7fb0d406fb80 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc009d90 tx=0x7fb0cc01f130 comp rx=0 tx=0).stop 2023-10-30T12:35:14.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.436+0000 7fb0da380640 1 -- 172.21.15.89:0/2153865296 shutdown_connections 2023-10-30T12:35:14.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.436+0000 7fb0da380640 1 --2- 172.21.15.89:0/2153865296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0d4071110 0x7fb0d406fb80 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:14.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.436+0000 7fb0da380640 1 -- 172.21.15.89:0/2153865296 >> 172.21.15.89:0/2153865296 conn(0x7fb0d40c7910 msgr2=0x7fb0d40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:14.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.436+0000 7fb0da380640 1 -- 172.21.15.89:0/2153865296 shutdown_connections 2023-10-30T12:35:14.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.436+0000 7fb0da380640 1 -- 172.21.15.89:0/2153865296 wait complete. 2023-10-30T12:35:14.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.440+0000 7fb0da380640 1 Processor -- start 2023-10-30T12:35:14.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.440+0000 7fb0da380640 1 -- start start 2023-10-30T12:35:14.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.440+0000 7fb0da380640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb0d406f7a0 0x7fb0d41f00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:14.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.440+0000 7fb0da380640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0d4071110 0x7fb0d41569f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:14.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.440+0000 7fb0da380640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0d40c6bd0 0x7fb0d4156f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:14.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.440+0000 7fb0da380640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d41543c0 con 0x7fb0d4071110 2023-10-30T12:35:14.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.440+0000 7fb0da380640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d4154f00 con 0x7fb0d40c6bd0 2023-10-30T12:35:14.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.440+0000 7fb0da380640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d40c83b0 con 0x7fb0d406f7a0 2023-10-30T12:35:14.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.440+0000 7fb0d37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0d4071110 0x7fb0d41569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:14.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.440+0000 7fb0d37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0d4071110 0x7fb0d41569f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41552/0 (socket says 172.21.15.89:41552) 2023-10-30T12:35:14.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.440+0000 7fb0d37fe640 1 -- 172.21.15.89:0/3611991582 learned_addr learned my addr 172.21.15.89:0/3611991582 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:35:14.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.440+0000 7fb0d891d640 1 --2- 172.21.15.89:0/3611991582 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0d40c6bd0 0x7fb0d4156f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:14.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.440+0000 7fb0d3fff640 1 --2- 172.21.15.89:0/3611991582 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb0d406f7a0 0x7fb0d41f00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:14.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.440+0000 7fb0d37fe640 1 -- 172.21.15.89:0/3611991582 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb0d406f7a0 msgr2=0x7fb0d41f00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:14.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.440+0000 7fb0d37fe640 1 --2- 172.21.15.89:0/3611991582 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb0d406f7a0 0x7fb0d41f00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:14.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.440+0000 7fb0d37fe640 1 -- 172.21.15.89:0/3611991582 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0d40c6bd0 msgr2=0x7fb0d4156f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:14.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.440+0000 7fb0d37fe640 1 --2- 172.21.15.89:0/3611991582 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0d40c6bd0 0x7fb0d4156f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:14.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.440+0000 7fb0d37fe640 1 -- 172.21.15.89:0/3611991582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0cc009a70 con 0x7fb0d4071110 2023-10-30T12:35:14.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.440+0000 7fb0d891d640 1 --2- 172.21.15.89:0/3611991582 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0d40c6bd0 0x7fb0d4156f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:35:14.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.440+0000 7fb0d37fe640 1 --2- 172.21.15.89:0/3611991582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0d4071110 0x7fb0d41569f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800c480 tx=0x7fb0c800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:14.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.440+0000 7fb0d0ff9640 1 -- 172.21.15.89:0/3611991582 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c80074e0 con 0x7fb0d4071110 2023-10-30T12:35:14.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.440+0000 7fb0d0ff9640 1 -- 172.21.15.89:0/3611991582 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c8007640 con 0x7fb0d4071110 2023-10-30T12:35:14.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.440+0000 7fb0d0ff9640 1 -- 172.21.15.89:0/3611991582 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c8004a10 con 0x7fb0d4071110 2023-10-30T12:35:14.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.440+0000 7fb0da380640 1 -- 172.21.15.89:0/3611991582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0d41f0a80 con 0x7fb0d4071110 2023-10-30T12:35:14.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.440+0000 7fb0da380640 1 -- 172.21.15.89:0/3611991582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0d41548b0 con 0x7fb0d4071110 2023-10-30T12:35:14.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.440+0000 7fb0da380640 1 -- 172.21.15.89:0/3611991582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0d41550b0 con 0x7fb0d4071110 2023-10-30T12:35:14.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.444+0000 7fb0d0ff9640 1 -- 172.21.15.89:0/3611991582 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb0c8004b70 con 0x7fb0d4071110 2023-10-30T12:35:14.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.444+0000 7fb0d0ff9640 1 --2- 172.21.15.89:0/3611991582 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb0c00664d0 0x7fb0c0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:14.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.444+0000 7fb0d0ff9640 1 -- 172.21.15.89:0/3611991582 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fb0c8083250 con 0x7fb0d4071110 2023-10-30T12:35:14.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.444+0000 7fb0d3fff640 1 --2- 172.21.15.89:0/3611991582 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb0c00664d0 0x7fb0c0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:14.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.444+0000 7fb0d3fff640 1 --2- 172.21.15.89:0/3611991582 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb0c00664d0 0x7fb0c0068990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4000f70 tx=0x7fb0c4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:14.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.448+0000 7fb0d0ff9640 1 -- 172.21.15.89:0/3611991582 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb0d41550b0 con 0x7fb0d4071110 2023-10-30T12:35:14.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.628+0000 7fb0da380640 1 -- 172.21.15.89:0/3611991582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0d404e170 con 0x7fb0d4071110 2023-10-30T12:35:14.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.628+0000 7fb0d0ff9640 1 -- 172.21.15.89:0/3611991582 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+5739 (secure 0 0 0) 0x7fb0c8018090 con 0x7fb0d4071110 2023-10-30T12:35:14.633 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:35:14.634 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:35:12.272846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:rejoin","state_seq":25,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:14.634 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 36 2023-10-30T12:35:14.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.628+0000 7fb0da380640 1 -- 172.21.15.89:0/3611991582 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb0c00664d0 msgr2=0x7fb0c0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:14.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.628+0000 7fb0da380640 1 --2- 172.21.15.89:0/3611991582 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb0c00664d0 0x7fb0c0068990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4000f70 tx=0x7fb0c4009040 comp rx=0 tx=0).stop 2023-10-30T12:35:14.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.628+0000 7fb0da380640 1 -- 172.21.15.89:0/3611991582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0d4071110 msgr2=0x7fb0d41569f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:14.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.628+0000 7fb0da380640 1 --2- 172.21.15.89:0/3611991582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0d4071110 0x7fb0d41569f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800c480 tx=0x7fb0c800c940 comp rx=0 tx=0).stop 2023-10-30T12:35:14.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.632+0000 7fb0da380640 1 -- 172.21.15.89:0/3611991582 shutdown_connections 2023-10-30T12:35:14.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.632+0000 7fb0da380640 1 --2- 172.21.15.89:0/3611991582 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb0c00664d0 0x7fb0c0068990 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:14.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.632+0000 7fb0da380640 1 --2- 172.21.15.89:0/3611991582 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0d40c6bd0 0x7fb0d4156f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:14.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.632+0000 7fb0da380640 1 --2- 172.21.15.89:0/3611991582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0d4071110 0x7fb0d41569f0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:14.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.632+0000 7fb0da380640 1 --2- 172.21.15.89:0/3611991582 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb0d406f7a0 0x7fb0d41f00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:14.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.632+0000 7fb0da380640 1 -- 172.21.15.89:0/3611991582 >> 172.21.15.89:0/3611991582 conn(0x7fb0d40c7910 msgr2=0x7fb0d40c3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:14.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.632+0000 7fb0da380640 1 -- 172.21.15.89:0/3611991582 shutdown_connections 2023-10-30T12:35:14.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:14.632+0000 7fb0da380640 1 -- 172.21.15.89:0/3611991582 wait complete. 2023-10-30T12:35:14.654 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:35:15.774 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:35:15.779 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:35:16.656 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:16.805 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.800+0000 7f463e09a640 1 Processor -- start 2023-10-30T12:35:16.805 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.800+0000 7f463e09a640 1 -- start start 2023-10-30T12:35:16.806 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.800+0000 7f463e09a640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f463806f7a0 0x7f463806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:16.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.800+0000 7f463e09a640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4638066cf0 con 0x7f46380c6bd0 2023-10-30T12:35:16.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.800+0000 7f463e09a640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46380c70d0 con 0x7f4638071110 2023-10-30T12:35:16.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.800+0000 7f463e09a640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46380700c0 con 0x7f463806f7a0 2023-10-30T12:35:16.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.800+0000 7f4636ffd640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f46380c6bd0 0x7f4638069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44374/0 (socket says 172.21.15.89:44374) 2023-10-30T12:35:16.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.800+0000 7f4636ffd640 1 -- 172.21.15.89:0/3083632335 learned_addr learned my addr 172.21.15.89:0/3083632335 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:35:16.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.800+0000 7f46377fe640 1 --2- 172.21.15.89:0/3083632335 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f463806f7a0 0x7f463806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:16.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.804+0000 7f46367fc640 1 -- 172.21.15.89:0/3083632335 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2136353693 0 0) 0x7f4638066cf0 con 0x7f46380c6bd0 2023-10-30T12:35:16.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.804+0000 7f46367fc640 1 -- 172.21.15.89:0/3083632335 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4620003530 con 0x7f46380c6bd0 2023-10-30T12:35:16.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.804+0000 7f46367fc640 1 -- 172.21.15.89:0/3083632335 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3324594322 0 0) 0x7f46380c70d0 con 0x7f4638071110 2023-10-30T12:35:16.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.804+0000 7f46377fe640 1 -- 172.21.15.89:0/3083632335 >> v1:172.21.15.192:6789/0 conn(0x7f4638071110 legacy=0x7f4638158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:16.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.804+0000 7f46377fe640 1 -- 172.21.15.89:0/3083632335 >> v1:172.21.15.89:6789/0 conn(0x7f46380c6bd0 legacy=0x7f4638069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:16.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.804+0000 7f46377fe640 1 -- 172.21.15.89:0/3083632335 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46380c7390 con 0x7f463806f7a0 2023-10-30T12:35:16.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.804+0000 7f46377fe640 1 --2- 172.21.15.89:0/3083632335 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f463806f7a0 0x7f463806fb80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f46280097f0 tx=0x7f462801eaf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8ba75f85048149a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:16.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.808+0000 7f46367fc640 1 -- 172.21.15.89:0/3083632335 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462c001680 con 0x7f463806f7a0 2023-10-30T12:35:16.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.808+0000 7f46367fc640 1 -- 172.21.15.89:0/3083632335 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46280a85b0 con 0x7f463806f7a0 2023-10-30T12:35:16.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.808+0000 7f46367fc640 1 -- 172.21.15.89:0/3083632335 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46280a8920 con 0x7f463806f7a0 2023-10-30T12:35:16.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.808+0000 7f463e09a640 1 -- 172.21.15.89:0/3083632335 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f463806f7a0 msgr2=0x7f463806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:16.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.808+0000 7f463e09a640 1 --2- 172.21.15.89:0/3083632335 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f463806f7a0 0x7f463806fb80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f46280097f0 tx=0x7f462801eaf0 comp rx=0 tx=0).stop 2023-10-30T12:35:16.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.808+0000 7f463e09a640 1 -- 172.21.15.89:0/3083632335 shutdown_connections 2023-10-30T12:35:16.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.808+0000 7f463e09a640 1 --2- 172.21.15.89:0/3083632335 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f463806f7a0 0x7f463806fb80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:16.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.808+0000 7f463e09a640 1 -- 172.21.15.89:0/3083632335 >> 172.21.15.89:0/3083632335 conn(0x7f46380c7910 msgr2=0x7f46380c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:16.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.808+0000 7f463e09a640 1 -- 172.21.15.89:0/3083632335 shutdown_connections 2023-10-30T12:35:16.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.808+0000 7f463e09a640 1 -- 172.21.15.89:0/3083632335 wait complete. 2023-10-30T12:35:16.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.808+0000 7f463e09a640 1 Processor -- start 2023-10-30T12:35:16.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.808+0000 7f463e09a640 1 -- start start 2023-10-30T12:35:16.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.808+0000 7f463e09a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f463806f7a0 0x7f46381ec920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:16.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.808+0000 7f463e09a640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4638071110 0x7f4638157780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:16.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.808+0000 7f463e09a640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46380c6bd0 0x7f4638157cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:16.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.808+0000 7f463e09a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46380c48b0 con 0x7f463806f7a0 2023-10-30T12:35:16.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.808+0000 7f463e09a640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46381f2ed0 con 0x7f46380c6bd0 2023-10-30T12:35:16.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.808+0000 7f463e09a640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46381f09c0 con 0x7f4638071110 2023-10-30T12:35:16.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.812+0000 7f46377fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f463806f7a0 0x7f46381ec920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:16.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.812+0000 7f46377fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f463806f7a0 0x7f46381ec920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41564/0 (socket says 172.21.15.89:41564) 2023-10-30T12:35:16.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.812+0000 7f46377fe640 1 -- 172.21.15.89:0/3400360130 learned_addr learned my addr 172.21.15.89:0/3400360130 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:35:16.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.812+0000 7f4636ffd640 1 --2- 172.21.15.89:0/3400360130 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4638071110 0x7f4638157780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:16.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.812+0000 7f4637fff640 1 --2- 172.21.15.89:0/3400360130 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46380c6bd0 0x7f4638157cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:16.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.812+0000 7f46377fe640 1 -- 172.21.15.89:0/3400360130 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4638071110 msgr2=0x7f4638157780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:16.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.812+0000 7f46377fe640 1 --2- 172.21.15.89:0/3400360130 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4638071110 0x7f4638157780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:16.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.812+0000 7f46377fe640 1 -- 172.21.15.89:0/3400360130 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46380c6bd0 msgr2=0x7f4638157cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:16.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.812+0000 7f46377fe640 1 --2- 172.21.15.89:0/3400360130 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46380c6bd0 0x7f4638157cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:16.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.812+0000 7f46377fe640 1 -- 172.21.15.89:0/3400360130 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4628009420 con 0x7f463806f7a0 2023-10-30T12:35:16.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.812+0000 7f46377fe640 1 --2- 172.21.15.89:0/3400360130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f463806f7a0 0x7f46381ec920 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f4628000c10 tx=0x7f462801b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:16.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.812+0000 7f461ffff640 1 -- 172.21.15.89:0/3400360130 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46280b0850 con 0x7f463806f7a0 2023-10-30T12:35:16.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.812+0000 7f461ffff640 1 -- 172.21.15.89:0/3400360130 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46280b09b0 con 0x7f463806f7a0 2023-10-30T12:35:16.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.812+0000 7f463e09a640 1 -- 172.21.15.89:0/3400360130 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46381f1d00 con 0x7f463806f7a0 2023-10-30T12:35:16.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.812+0000 7f461ffff640 1 -- 172.21.15.89:0/3400360130 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46280b0d20 con 0x7f463806f7a0 2023-10-30T12:35:16.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.812+0000 7f463e09a640 1 -- 172.21.15.89:0/3400360130 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46381f0f40 con 0x7f463806f7a0 2023-10-30T12:35:16.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.812+0000 7f463e09a640 1 -- 172.21.15.89:0/3400360130 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f463804e170 con 0x7f463806f7a0 2023-10-30T12:35:16.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.812+0000 7f461ffff640 1 -- 172.21.15.89:0/3400360130 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f46280b7040 con 0x7f463806f7a0 2023-10-30T12:35:16.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.812+0000 7f461ffff640 1 --2- 172.21.15.89:0/3400360130 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f46240664a0 0x7f4624068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:16.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.816+0000 7f461ffff640 1 -- 172.21.15.89:0/3400360130 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f4628121650 con 0x7f463806f7a0 2023-10-30T12:35:16.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.816+0000 7f4636ffd640 1 --2- 172.21.15.89:0/3400360130 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f46240664a0 0x7f4624068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:16.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.816+0000 7f4636ffd640 1 --2- 172.21.15.89:0/3400360130 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f46240664a0 0x7f4624068960 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f46380c91d0 tx=0x7f462c001940 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:16.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:16.816+0000 7f461ffff640 1 -- 172.21.15.89:0/3400360130 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f46280ee8d0 con 0x7f463806f7a0 2023-10-30T12:35:17.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:17.000+0000 7f463e09a640 1 -- 172.21.15.89:0/3400360130 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f463804e170 con 0x7f463806f7a0 2023-10-30T12:35:17.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:17.000+0000 7f461ffff640 1 -- 172.21.15.89:0/3400360130 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f46280b5090 con 0x7f463806f7a0 2023-10-30T12:35:17.005 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:35:17.005 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:35:12.272846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:rejoin","state_seq":25,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:17.006 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 36 2023-10-30T12:35:17.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:17.000+0000 7f463e09a640 1 -- 172.21.15.89:0/3400360130 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f46240664a0 msgr2=0x7f4624068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:17.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:17.000+0000 7f463e09a640 1 --2- 172.21.15.89:0/3400360130 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f46240664a0 0x7f4624068960 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f46380c91d0 tx=0x7f462c001940 comp rx=0 tx=0).stop 2023-10-30T12:35:17.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:17.000+0000 7f463e09a640 1 -- 172.21.15.89:0/3400360130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f463806f7a0 msgr2=0x7f46381ec920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:17.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:17.000+0000 7f463e09a640 1 --2- 172.21.15.89:0/3400360130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f463806f7a0 0x7f46381ec920 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f4628000c10 tx=0x7f462801b890 comp rx=0 tx=0).stop 2023-10-30T12:35:17.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:17.000+0000 7f463e09a640 1 -- 172.21.15.89:0/3400360130 shutdown_connections 2023-10-30T12:35:17.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:17.000+0000 7f463e09a640 1 --2- 172.21.15.89:0/3400360130 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f46240664a0 0x7f4624068960 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:17.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:17.000+0000 7f463e09a640 1 --2- 172.21.15.89:0/3400360130 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f46380c6bd0 0x7f4638157cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:17.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:17.000+0000 7f463e09a640 1 --2- 172.21.15.89:0/3400360130 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4638071110 0x7f4638157780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:17.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:17.000+0000 7f463e09a640 1 --2- 172.21.15.89:0/3400360130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f463806f7a0 0x7f46381ec920 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:17.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:17.000+0000 7f463e09a640 1 -- 172.21.15.89:0/3400360130 >> 172.21.15.89:0/3400360130 conn(0x7f46380c7910 msgr2=0x7f46380ca5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:17.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:17.000+0000 7f463e09a640 1 -- 172.21.15.89:0/3400360130 shutdown_connections 2023-10-30T12:35:17.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:17.000+0000 7f463e09a640 1 -- 172.21.15.89:0/3400360130 wait complete. 2023-10-30T12:35:17.026 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:35:18.089 INFO:tasks.workunit.client.0.smithi089.stdout:removed './157/17' 2023-10-30T12:35:18.089 INFO:tasks.workunit.client.0.smithi089.stdout:removed './157/35' 2023-10-30T12:35:18.089 INFO:tasks.workunit.client.0.smithi089.stdout:removed './157/37' 2023-10-30T12:35:18.090 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './157' 2023-10-30T12:35:18.090 INFO:tasks.workunit.client.0.smithi089.stdout:removed './158/19' 2023-10-30T12:35:18.090 INFO:tasks.workunit.client.0.smithi089.stdout:removed './158/15' 2023-10-30T12:35:18.090 INFO:tasks.workunit.client.0.smithi089.stdout:removed './158/31' 2023-10-30T12:35:18.091 INFO:tasks.workunit.client.0.smithi089.stdout:removed './158/36' 2023-10-30T12:35:18.091 INFO:tasks.workunit.client.0.smithi089.stdout:removed './158/20' 2023-10-30T12:35:18.091 INFO:tasks.workunit.client.0.smithi089.stdout:removed './158/12' 2023-10-30T12:35:18.091 INFO:tasks.workunit.client.0.smithi089.stdout:removed './158/39' 2023-10-30T12:35:18.091 INFO:tasks.workunit.client.0.smithi089.stdout:removed './158/10' 2023-10-30T12:35:18.091 INFO:tasks.workunit.client.0.smithi089.stdout:removed './158/34' 2023-10-30T12:35:18.092 INFO:tasks.workunit.client.0.smithi089.stdout:removed './158/38' 2023-10-30T12:35:18.092 INFO:tasks.workunit.client.0.smithi089.stdout:removed './158/11' 2023-10-30T12:35:18.092 INFO:tasks.workunit.client.0.smithi089.stdout:removed './158/16' 2023-10-30T12:35:18.092 INFO:tasks.workunit.client.0.smithi089.stdout:removed './158/32' 2023-10-30T12:35:18.092 INFO:tasks.workunit.client.0.smithi089.stdout:removed './158/30' 2023-10-30T12:35:18.093 INFO:tasks.workunit.client.0.smithi089.stdout:removed './158/14' 2023-10-30T12:35:18.093 INFO:tasks.workunit.client.0.smithi089.stdout:removed './158/18' 2023-10-30T12:35:18.093 INFO:tasks.workunit.client.0.smithi089.stdout:removed './158/13' 2023-10-30T12:35:18.093 INFO:tasks.workunit.client.0.smithi089.stdout:removed './158/33' 2023-10-30T12:35:18.093 INFO:tasks.workunit.client.0.smithi089.stdout:removed './158/17' 2023-10-30T12:35:18.094 INFO:tasks.workunit.client.0.smithi089.stdout:removed './158/35' 2023-10-30T12:35:18.094 INFO:tasks.workunit.client.0.smithi089.stdout:removed './158/37' 2023-10-30T12:35:18.094 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './158' 2023-10-30T12:35:18.094 INFO:tasks.workunit.client.0.smithi089.stdout:removed './159/19' 2023-10-30T12:35:18.095 INFO:tasks.workunit.client.0.smithi089.stdout:removed './159/15' 2023-10-30T12:35:18.095 INFO:tasks.workunit.client.0.smithi089.stdout:removed './159/31' 2023-10-30T12:35:18.095 INFO:tasks.workunit.client.0.smithi089.stdout:removed './159/36' 2023-10-30T12:35:18.095 INFO:tasks.workunit.client.0.smithi089.stdout:removed './159/20' 2023-10-30T12:35:18.095 INFO:tasks.workunit.client.0.smithi089.stdout:removed './159/12' 2023-10-30T12:35:18.096 INFO:tasks.workunit.client.0.smithi089.stdout:removed './159/39' 2023-10-30T12:35:18.096 INFO:tasks.workunit.client.0.smithi089.stdout:removed './159/10' 2023-10-30T12:35:18.096 INFO:tasks.workunit.client.0.smithi089.stdout:removed './159/34' 2023-10-30T12:35:18.096 INFO:tasks.workunit.client.0.smithi089.stdout:removed './159/38' 2023-10-30T12:35:18.096 INFO:tasks.workunit.client.0.smithi089.stdout:removed './159/11' 2023-10-30T12:35:18.097 INFO:tasks.workunit.client.0.smithi089.stdout:removed './159/16' 2023-10-30T12:35:18.097 INFO:tasks.workunit.client.0.smithi089.stdout:removed './159/32' 2023-10-30T12:35:18.097 INFO:tasks.workunit.client.0.smithi089.stdout:removed './159/30' 2023-10-30T12:35:18.097 INFO:tasks.workunit.client.0.smithi089.stdout:removed './159/14' 2023-10-30T12:35:18.097 INFO:tasks.workunit.client.0.smithi089.stdout:removed './159/18' 2023-10-30T12:35:18.098 INFO:tasks.workunit.client.0.smithi089.stdout:removed './159/13' 2023-10-30T12:35:18.098 INFO:tasks.workunit.client.0.smithi089.stdout:removed './159/33' 2023-10-30T12:35:18.098 INFO:tasks.workunit.client.0.smithi089.stdout:removed './159/17' 2023-10-30T12:35:18.098 INFO:tasks.workunit.client.0.smithi089.stdout:removed './159/35' 2023-10-30T12:35:18.098 INFO:tasks.workunit.client.0.smithi089.stdout:removed './159/37' 2023-10-30T12:35:18.099 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './159' 2023-10-30T12:35:18.099 INFO:tasks.workunit.client.0.smithi089.stdout:removed './160/19' 2023-10-30T12:35:18.099 INFO:tasks.workunit.client.0.smithi089.stdout:removed './160/15' 2023-10-30T12:35:18.099 INFO:tasks.workunit.client.0.smithi089.stdout:removed './160/31' 2023-10-30T12:35:18.099 INFO:tasks.workunit.client.0.smithi089.stdout:removed './160/36' 2023-10-30T12:35:18.100 INFO:tasks.workunit.client.0.smithi089.stdout:removed './160/20' 2023-10-30T12:35:18.100 INFO:tasks.workunit.client.0.smithi089.stdout:removed './160/12' 2023-10-30T12:35:18.100 INFO:tasks.workunit.client.0.smithi089.stdout:removed './160/39' 2023-10-30T12:35:18.100 INFO:tasks.workunit.client.0.smithi089.stdout:removed './160/10' 2023-10-30T12:35:18.100 INFO:tasks.workunit.client.0.smithi089.stdout:removed './160/34' 2023-10-30T12:35:18.101 INFO:tasks.workunit.client.0.smithi089.stdout:removed './160/38' 2023-10-30T12:35:18.101 INFO:tasks.workunit.client.0.smithi089.stdout:removed './160/11' 2023-10-30T12:35:18.101 INFO:tasks.workunit.client.0.smithi089.stdout:removed './160/16' 2023-10-30T12:35:18.101 INFO:tasks.workunit.client.0.smithi089.stdout:removed './160/32' 2023-10-30T12:35:18.101 INFO:tasks.workunit.client.0.smithi089.stdout:removed './160/30' 2023-10-30T12:35:18.101 INFO:tasks.workunit.client.0.smithi089.stdout:removed './160/14' 2023-10-30T12:35:18.102 INFO:tasks.workunit.client.0.smithi089.stdout:removed './160/18' 2023-10-30T12:35:18.102 INFO:tasks.workunit.client.0.smithi089.stdout:removed './160/13' 2023-10-30T12:35:18.102 INFO:tasks.workunit.client.0.smithi089.stdout:removed './160/33' 2023-10-30T12:35:18.102 INFO:tasks.workunit.client.0.smithi089.stdout:removed './160/17' 2023-10-30T12:35:18.102 INFO:tasks.workunit.client.0.smithi089.stdout:removed './160/35' 2023-10-30T12:35:18.103 INFO:tasks.workunit.client.0.smithi089.stdout:removed './160/37' 2023-10-30T12:35:18.103 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './160' 2023-10-30T12:35:18.103 INFO:tasks.workunit.client.0.smithi089.stdout:removed './161/19' 2023-10-30T12:35:18.103 INFO:tasks.workunit.client.0.smithi089.stdout:removed './161/15' 2023-10-30T12:35:18.103 INFO:tasks.workunit.client.0.smithi089.stdout:removed './161/31' 2023-10-30T12:35:18.104 INFO:tasks.workunit.client.0.smithi089.stdout:removed './161/36' 2023-10-30T12:35:18.104 INFO:tasks.workunit.client.0.smithi089.stdout:removed './161/20' 2023-10-30T12:35:18.104 INFO:tasks.workunit.client.0.smithi089.stdout:removed './161/12' 2023-10-30T12:35:18.104 INFO:tasks.workunit.client.0.smithi089.stdout:removed './161/39' 2023-10-30T12:35:18.104 INFO:tasks.workunit.client.0.smithi089.stdout:removed './161/10' 2023-10-30T12:35:18.105 INFO:tasks.workunit.client.0.smithi089.stdout:removed './161/34' 2023-10-30T12:35:18.105 INFO:tasks.workunit.client.0.smithi089.stdout:removed './161/38' 2023-10-30T12:35:18.105 INFO:tasks.workunit.client.0.smithi089.stdout:removed './161/11' 2023-10-30T12:35:18.105 INFO:tasks.workunit.client.0.smithi089.stdout:removed './161/16' 2023-10-30T12:35:18.105 INFO:tasks.workunit.client.0.smithi089.stdout:removed './161/32' 2023-10-30T12:35:18.106 INFO:tasks.workunit.client.0.smithi089.stdout:removed './161/30' 2023-10-30T12:35:18.106 INFO:tasks.workunit.client.0.smithi089.stdout:removed './161/14' 2023-10-30T12:35:18.106 INFO:tasks.workunit.client.0.smithi089.stdout:removed './161/18' 2023-10-30T12:35:18.106 INFO:tasks.workunit.client.0.smithi089.stdout:removed './161/13' 2023-10-30T12:35:18.106 INFO:tasks.workunit.client.0.smithi089.stdout:removed './161/33' 2023-10-30T12:35:18.106 INFO:tasks.workunit.client.0.smithi089.stdout:removed './161/17' 2023-10-30T12:35:18.107 INFO:tasks.workunit.client.0.smithi089.stdout:removed './161/35' 2023-10-30T12:35:18.107 INFO:tasks.workunit.client.0.smithi089.stdout:removed './161/37' 2023-10-30T12:35:18.107 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './161' 2023-10-30T12:35:18.107 INFO:tasks.workunit.client.0.smithi089.stdout:removed './162/19' 2023-10-30T12:35:18.107 INFO:tasks.workunit.client.0.smithi089.stdout:removed './162/15' 2023-10-30T12:35:18.108 INFO:tasks.workunit.client.0.smithi089.stdout:removed './162/31' 2023-10-30T12:35:18.108 INFO:tasks.workunit.client.0.smithi089.stdout:removed './162/36' 2023-10-30T12:35:18.108 INFO:tasks.workunit.client.0.smithi089.stdout:removed './162/20' 2023-10-30T12:35:18.108 INFO:tasks.workunit.client.0.smithi089.stdout:removed './162/12' 2023-10-30T12:35:18.108 INFO:tasks.workunit.client.0.smithi089.stdout:removed './162/39' 2023-10-30T12:35:18.109 INFO:tasks.workunit.client.0.smithi089.stdout:removed './162/10' 2023-10-30T12:35:18.109 INFO:tasks.workunit.client.0.smithi089.stdout:removed './162/34' 2023-10-30T12:35:18.109 INFO:tasks.workunit.client.0.smithi089.stdout:removed './162/38' 2023-10-30T12:35:18.109 INFO:tasks.workunit.client.0.smithi089.stdout:removed './162/11' 2023-10-30T12:35:18.109 INFO:tasks.workunit.client.0.smithi089.stdout:removed './162/16' 2023-10-30T12:35:18.109 INFO:tasks.workunit.client.0.smithi089.stdout:removed './162/32' 2023-10-30T12:35:18.110 INFO:tasks.workunit.client.0.smithi089.stdout:removed './162/30' 2023-10-30T12:35:18.110 INFO:tasks.workunit.client.0.smithi089.stdout:removed './162/14' 2023-10-30T12:35:18.110 INFO:tasks.workunit.client.0.smithi089.stdout:removed './162/18' 2023-10-30T12:35:18.110 INFO:tasks.workunit.client.0.smithi089.stdout:removed './162/13' 2023-10-30T12:35:18.110 INFO:tasks.workunit.client.0.smithi089.stdout:removed './162/33' 2023-10-30T12:35:18.111 INFO:tasks.workunit.client.0.smithi089.stdout:removed './162/17' 2023-10-30T12:35:18.111 INFO:tasks.workunit.client.0.smithi089.stdout:removed './162/35' 2023-10-30T12:35:18.111 INFO:tasks.workunit.client.0.smithi089.stdout:removed './162/37' 2023-10-30T12:35:18.111 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './162' 2023-10-30T12:35:18.111 INFO:tasks.workunit.client.0.smithi089.stdout:removed './163/19' 2023-10-30T12:35:18.112 INFO:tasks.workunit.client.0.smithi089.stdout:removed './163/15' 2023-10-30T12:35:18.112 INFO:tasks.workunit.client.0.smithi089.stdout:removed './163/31' 2023-10-30T12:35:18.112 INFO:tasks.workunit.client.0.smithi089.stdout:removed './163/36' 2023-10-30T12:35:18.112 INFO:tasks.workunit.client.0.smithi089.stdout:removed './163/20' 2023-10-30T12:35:18.112 INFO:tasks.workunit.client.0.smithi089.stdout:removed './163/12' 2023-10-30T12:35:18.112 INFO:tasks.workunit.client.0.smithi089.stdout:removed './163/39' 2023-10-30T12:35:18.113 INFO:tasks.workunit.client.0.smithi089.stdout:removed './163/10' 2023-10-30T12:35:18.113 INFO:tasks.workunit.client.0.smithi089.stdout:removed './163/34' 2023-10-30T12:35:18.113 INFO:tasks.workunit.client.0.smithi089.stdout:removed './163/38' 2023-10-30T12:35:18.113 INFO:tasks.workunit.client.0.smithi089.stdout:removed './163/11' 2023-10-30T12:35:18.113 INFO:tasks.workunit.client.0.smithi089.stdout:removed './163/16' 2023-10-30T12:35:18.114 INFO:tasks.workunit.client.0.smithi089.stdout:removed './163/32' 2023-10-30T12:35:18.114 INFO:tasks.workunit.client.0.smithi089.stdout:removed './163/30' 2023-10-30T12:35:18.114 INFO:tasks.workunit.client.0.smithi089.stdout:removed './163/14' 2023-10-30T12:35:18.114 INFO:tasks.workunit.client.0.smithi089.stdout:removed './163/18' 2023-10-30T12:35:18.114 INFO:tasks.workunit.client.0.smithi089.stdout:removed './163/13' 2023-10-30T12:35:18.115 INFO:tasks.workunit.client.0.smithi089.stdout:removed './163/33' 2023-10-30T12:35:18.115 INFO:tasks.workunit.client.0.smithi089.stdout:removed './163/17' 2023-10-30T12:35:18.115 INFO:tasks.workunit.client.0.smithi089.stdout:removed './163/35' 2023-10-30T12:35:18.115 INFO:tasks.workunit.client.0.smithi089.stdout:removed './163/37' 2023-10-30T12:35:18.115 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './163' 2023-10-30T12:35:18.116 INFO:tasks.workunit.client.0.smithi089.stdout:removed './164/19' 2023-10-30T12:35:18.116 INFO:tasks.workunit.client.0.smithi089.stdout:removed './164/15' 2023-10-30T12:35:18.116 INFO:tasks.workunit.client.0.smithi089.stdout:removed './164/31' 2023-10-30T12:35:18.116 INFO:tasks.workunit.client.0.smithi089.stdout:removed './164/36' 2023-10-30T12:35:18.116 INFO:tasks.workunit.client.0.smithi089.stdout:removed './164/20' 2023-10-30T12:35:18.117 INFO:tasks.workunit.client.0.smithi089.stdout:removed './164/12' 2023-10-30T12:35:18.117 INFO:tasks.workunit.client.0.smithi089.stdout:removed './164/39' 2023-10-30T12:35:18.117 INFO:tasks.workunit.client.0.smithi089.stdout:removed './164/10' 2023-10-30T12:35:18.117 INFO:tasks.workunit.client.0.smithi089.stdout:removed './164/34' 2023-10-30T12:35:18.117 INFO:tasks.workunit.client.0.smithi089.stdout:removed './164/38' 2023-10-30T12:35:18.118 INFO:tasks.workunit.client.0.smithi089.stdout:removed './164/11' 2023-10-30T12:35:18.118 INFO:tasks.workunit.client.0.smithi089.stdout:removed './164/16' 2023-10-30T12:35:18.118 INFO:tasks.workunit.client.0.smithi089.stdout:removed './164/32' 2023-10-30T12:35:18.118 INFO:tasks.workunit.client.0.smithi089.stdout:removed './164/30' 2023-10-30T12:35:18.118 INFO:tasks.workunit.client.0.smithi089.stdout:removed './164/14' 2023-10-30T12:35:18.119 INFO:tasks.workunit.client.0.smithi089.stdout:removed './164/18' 2023-10-30T12:35:18.119 INFO:tasks.workunit.client.0.smithi089.stdout:removed './164/13' 2023-10-30T12:35:18.119 INFO:tasks.workunit.client.0.smithi089.stdout:removed './164/33' 2023-10-30T12:35:18.119 INFO:tasks.workunit.client.0.smithi089.stdout:removed './164/17' 2023-10-30T12:35:18.119 INFO:tasks.workunit.client.0.smithi089.stdout:removed './164/35' 2023-10-30T12:35:18.120 INFO:tasks.workunit.client.0.smithi089.stdout:removed './164/37' 2023-10-30T12:35:18.120 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './164' 2023-10-30T12:35:18.120 INFO:tasks.workunit.client.0.smithi089.stdout:removed './165/19' 2023-10-30T12:35:18.120 INFO:tasks.workunit.client.0.smithi089.stdout:removed './165/15' 2023-10-30T12:35:18.120 INFO:tasks.workunit.client.0.smithi089.stdout:removed './165/31' 2023-10-30T12:35:18.120 INFO:tasks.workunit.client.0.smithi089.stdout:removed './165/36' 2023-10-30T12:35:18.121 INFO:tasks.workunit.client.0.smithi089.stdout:removed './165/20' 2023-10-30T12:35:18.121 INFO:tasks.workunit.client.0.smithi089.stdout:removed './165/12' 2023-10-30T12:35:18.121 INFO:tasks.workunit.client.0.smithi089.stdout:removed './165/39' 2023-10-30T12:35:18.121 INFO:tasks.workunit.client.0.smithi089.stdout:removed './165/10' 2023-10-30T12:35:18.121 INFO:tasks.workunit.client.0.smithi089.stdout:removed './165/34' 2023-10-30T12:35:18.122 INFO:tasks.workunit.client.0.smithi089.stdout:removed './165/38' 2023-10-30T12:35:18.122 INFO:tasks.workunit.client.0.smithi089.stdout:removed './165/11' 2023-10-30T12:35:18.122 INFO:tasks.workunit.client.0.smithi089.stdout:removed './165/16' 2023-10-30T12:35:18.122 INFO:tasks.workunit.client.0.smithi089.stdout:removed './165/32' 2023-10-30T12:35:18.122 INFO:tasks.workunit.client.0.smithi089.stdout:removed './165/30' 2023-10-30T12:35:18.123 INFO:tasks.workunit.client.0.smithi089.stdout:removed './165/14' 2023-10-30T12:35:18.123 INFO:tasks.workunit.client.0.smithi089.stdout:removed './165/18' 2023-10-30T12:35:18.123 INFO:tasks.workunit.client.0.smithi089.stdout:removed './165/13' 2023-10-30T12:35:18.123 INFO:tasks.workunit.client.0.smithi089.stdout:removed './165/33' 2023-10-30T12:35:18.123 INFO:tasks.workunit.client.0.smithi089.stdout:removed './165/17' 2023-10-30T12:35:18.123 INFO:tasks.workunit.client.0.smithi089.stdout:removed './165/35' 2023-10-30T12:35:18.124 INFO:tasks.workunit.client.0.smithi089.stdout:removed './165/37' 2023-10-30T12:35:18.124 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './165' 2023-10-30T12:35:18.124 INFO:tasks.workunit.client.0.smithi089.stdout:removed './166/19' 2023-10-30T12:35:18.124 INFO:tasks.workunit.client.0.smithi089.stdout:removed './166/15' 2023-10-30T12:35:18.124 INFO:tasks.workunit.client.0.smithi089.stdout:removed './166/31' 2023-10-30T12:35:18.125 INFO:tasks.workunit.client.0.smithi089.stdout:removed './166/36' 2023-10-30T12:35:18.125 INFO:tasks.workunit.client.0.smithi089.stdout:removed './166/20' 2023-10-30T12:35:18.125 INFO:tasks.workunit.client.0.smithi089.stdout:removed './166/12' 2023-10-30T12:35:18.125 INFO:tasks.workunit.client.0.smithi089.stdout:removed './166/39' 2023-10-30T12:35:18.125 INFO:tasks.workunit.client.0.smithi089.stdout:removed './166/10' 2023-10-30T12:35:18.126 INFO:tasks.workunit.client.0.smithi089.stdout:removed './166/34' 2023-10-30T12:35:18.126 INFO:tasks.workunit.client.0.smithi089.stdout:removed './166/38' 2023-10-30T12:35:18.126 INFO:tasks.workunit.client.0.smithi089.stdout:removed './166/11' 2023-10-30T12:35:18.126 INFO:tasks.workunit.client.0.smithi089.stdout:removed './166/16' 2023-10-30T12:35:18.126 INFO:tasks.workunit.client.0.smithi089.stdout:removed './166/32' 2023-10-30T12:35:18.127 INFO:tasks.workunit.client.0.smithi089.stdout:removed './166/30' 2023-10-30T12:35:18.127 INFO:tasks.workunit.client.0.smithi089.stdout:removed './166/14' 2023-10-30T12:35:18.127 INFO:tasks.workunit.client.0.smithi089.stdout:removed './166/18' 2023-10-30T12:35:18.127 INFO:tasks.workunit.client.0.smithi089.stdout:removed './166/13' 2023-10-30T12:35:18.127 INFO:tasks.workunit.client.0.smithi089.stdout:removed './166/33' 2023-10-30T12:35:18.127 INFO:tasks.workunit.client.0.smithi089.stdout:removed './166/17' 2023-10-30T12:35:18.128 INFO:tasks.workunit.client.0.smithi089.stdout:removed './166/35' 2023-10-30T12:35:18.128 INFO:tasks.workunit.client.0.smithi089.stdout:removed './166/37' 2023-10-30T12:35:18.128 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './166' 2023-10-30T12:35:18.128 INFO:tasks.workunit.client.0.smithi089.stdout:removed './167/19' 2023-10-30T12:35:18.128 INFO:tasks.workunit.client.0.smithi089.stdout:removed './167/15' 2023-10-30T12:35:18.129 INFO:tasks.workunit.client.0.smithi089.stdout:removed './167/31' 2023-10-30T12:35:18.129 INFO:tasks.workunit.client.0.smithi089.stdout:removed './167/36' 2023-10-30T12:35:18.129 INFO:tasks.workunit.client.0.smithi089.stdout:removed './167/20' 2023-10-30T12:35:18.129 INFO:tasks.workunit.client.0.smithi089.stdout:removed './167/12' 2023-10-30T12:35:18.129 INFO:tasks.workunit.client.0.smithi089.stdout:removed './167/39' 2023-10-30T12:35:18.129 INFO:tasks.workunit.client.0.smithi089.stdout:removed './167/10' 2023-10-30T12:35:18.130 INFO:tasks.workunit.client.0.smithi089.stdout:removed './167/34' 2023-10-30T12:35:18.130 INFO:tasks.workunit.client.0.smithi089.stdout:removed './167/38' 2023-10-30T12:35:19.026 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:19.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.172+0000 7f194b279640 1 Processor -- start 2023-10-30T12:35:19.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.172+0000 7f194b279640 1 -- start start 2023-10-30T12:35:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.176+0000 7f194b279640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f19441538a0 0x7f194415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.176+0000 7f194b279640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19440cf160 0x7f19440cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.176+0000 7f194b279640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19440cb800 con 0x7f19440ca0e0 2023-10-30T12:35:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.176+0000 7f194b279640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f194415dc30 con 0x7f19441538a0 2023-10-30T12:35:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.176+0000 7f194b279640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19440cb330 con 0x7f19440cf160 2023-10-30T12:35:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.176+0000 7f1949015640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f19441538a0 0x7f194415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.176+0000 7f1948814640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f19440ca0e0 0x7f194415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44380/0 (socket says 172.21.15.89:44380) 2023-10-30T12:35:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.176+0000 7f1948814640 1 -- 172.21.15.89:0/1730366334 learned_addr learned my addr 172.21.15.89:0/1730366334 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:35:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.176+0000 7f1949816640 1 --2- 172.21.15.89:0/1730366334 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19440cf160 0x7f19440cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.176+0000 7f1949015640 1 -- 172.21.15.89:0/1730366334 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19440cf160 msgr2=0x7f19440cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:19.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.176+0000 7f1949015640 1 --2- 172.21.15.89:0/1730366334 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19440cf160 0x7f19440cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:19.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.176+0000 7f1949015640 1 -- 172.21.15.89:0/1730366334 >> v1:172.21.15.89:6789/0 conn(0x7f19440ca0e0 legacy=0x7f194415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:19.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.176+0000 7f193bfff640 1 -- 172.21.15.89:0/1730366334 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708940643 0 0) 0x7f19440cb800 con 0x7f19440ca0e0 2023-10-30T12:35:19.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.176+0000 7f1949015640 1 -- 172.21.15.89:0/1730366334 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19440cad70 con 0x7f19441538a0 2023-10-30T12:35:19.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.180+0000 7f1949015640 1 --2- 172.21.15.89:0/1730366334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19441538a0 0x7f194415d6f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f19400097c0 tx=0x7f194001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd9bb1a4baec3c80 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:19.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.180+0000 7f193bfff640 1 -- 172.21.15.89:0/1730366334 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19400a8970 con 0x7f19441538a0 2023-10-30T12:35:19.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.180+0000 7f193bfff640 1 -- 172.21.15.89:0/1730366334 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19400a8ad0 con 0x7f19441538a0 2023-10-30T12:35:19.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.180+0000 7f193bfff640 1 -- 172.21.15.89:0/1730366334 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19400a8e60 con 0x7f19441538a0 2023-10-30T12:35:19.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.180+0000 7f194b279640 1 -- 172.21.15.89:0/1730366334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19441538a0 msgr2=0x7f194415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:19.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.180+0000 7f194b279640 1 --2- 172.21.15.89:0/1730366334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19441538a0 0x7f194415d6f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f19400097c0 tx=0x7f194001ec70 comp rx=0 tx=0).stop 2023-10-30T12:35:19.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.180+0000 7f194b279640 1 -- 172.21.15.89:0/1730366334 shutdown_connections 2023-10-30T12:35:19.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.180+0000 7f194b279640 1 --2- 172.21.15.89:0/1730366334 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19440cf160 0x7f19440cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:19.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.180+0000 7f194b279640 1 --2- 172.21.15.89:0/1730366334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19441538a0 0x7f194415d6f0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:19.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.180+0000 7f194b279640 1 -- 172.21.15.89:0/1730366334 >> 172.21.15.89:0/1730366334 conn(0x7f194406bfe0 msgr2=0x7f194406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:19.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.180+0000 7f194b279640 1 -- 172.21.15.89:0/1730366334 shutdown_connections 2023-10-30T12:35:19.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.180+0000 7f194b279640 1 -- 172.21.15.89:0/1730366334 wait complete. 2023-10-30T12:35:19.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.180+0000 7f194b279640 1 Processor -- start 2023-10-30T12:35:19.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.180+0000 7f194b279640 1 -- start start 2023-10-30T12:35:19.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.184+0000 7f194b279640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19440ca0e0 0x7f194415aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:19.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.184+0000 7f194b279640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19440cf160 0x7f19440ce930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:19.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.184+0000 7f194b279640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19441538a0 0x7f19441ef320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:19.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.184+0000 7f194b279640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19441f3160 con 0x7f19441538a0 2023-10-30T12:35:19.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.184+0000 7f194b279640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194415b3e0 con 0x7f19440cf160 2023-10-30T12:35:19.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.184+0000 7f194b279640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19441f2890 con 0x7f19440ca0e0 2023-10-30T12:35:19.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.184+0000 7f1949816640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19441538a0 0x7f19441ef320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:19.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.184+0000 7f1949816640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19441538a0 0x7f19441ef320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41590/0 (socket says 172.21.15.89:41590) 2023-10-30T12:35:19.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.184+0000 7f1949816640 1 -- 172.21.15.89:0/3734547011 learned_addr learned my addr 172.21.15.89:0/3734547011 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:35:19.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.184+0000 7f1948814640 1 --2- 172.21.15.89:0/3734547011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19440cf160 0x7f19440ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:19.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.184+0000 7f1949015640 1 --2- 172.21.15.89:0/3734547011 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19440ca0e0 0x7f194415aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:19.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.184+0000 7f1949816640 1 -- 172.21.15.89:0/3734547011 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19440ca0e0 msgr2=0x7f194415aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:19.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.184+0000 7f1949816640 1 --2- 172.21.15.89:0/3734547011 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19440ca0e0 0x7f194415aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:19.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.184+0000 7f1949816640 1 -- 172.21.15.89:0/3734547011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19440cf160 msgr2=0x7f19440ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:19.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.184+0000 7f1949816640 1 --2- 172.21.15.89:0/3734547011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19440cf160 0x7f19440ce930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:19.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.184+0000 7f1949816640 1 -- 172.21.15.89:0/3734547011 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1940009420 con 0x7f19441538a0 2023-10-30T12:35:19.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.184+0000 7f1949015640 1 --2- 172.21.15.89:0/3734547011 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19440ca0e0 0x7f194415aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:35:19.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.184+0000 7f1949816640 1 --2- 172.21.15.89:0/3734547011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19441538a0 0x7f19441ef320 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f193c007d90 tx=0x7f193c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:19.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.184+0000 7f1939ffb640 1 -- 172.21.15.89:0/3734547011 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193c014420 con 0x7f19441538a0 2023-10-30T12:35:19.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.184+0000 7f1939ffb640 1 -- 172.21.15.89:0/3734547011 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f193c0145a0 con 0x7f19441538a0 2023-10-30T12:35:19.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.184+0000 7f1939ffb640 1 -- 172.21.15.89:0/3734547011 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193c014950 con 0x7f19441538a0 2023-10-30T12:35:19.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.184+0000 7f194b279640 1 -- 172.21.15.89:0/3734547011 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19440b51f0 con 0x7f19441538a0 2023-10-30T12:35:19.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.184+0000 7f194b279640 1 -- 172.21.15.89:0/3734547011 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19440b37a0 con 0x7f19441538a0 2023-10-30T12:35:19.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.184+0000 7f194b279640 1 -- 172.21.15.89:0/3734547011 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19440b2a50 con 0x7f19441538a0 2023-10-30T12:35:19.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.184+0000 7f1939ffb640 1 -- 172.21.15.89:0/3734547011 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f193c014ab0 con 0x7f19441538a0 2023-10-30T12:35:19.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.188+0000 7f1939ffb640 1 --2- 172.21.15.89:0/3734547011 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f19280664a0 0x7f1928068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:19.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.188+0000 7f1939ffb640 1 -- 172.21.15.89:0/3734547011 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f193c084770 con 0x7f19441538a0 2023-10-30T12:35:19.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.188+0000 7f1949015640 1 --2- 172.21.15.89:0/3734547011 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f19280664a0 0x7f1928068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:19.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.188+0000 7f1949015640 1 --2- 172.21.15.89:0/3734547011 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f19280664a0 0x7f1928068960 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f19400b2000 tx=0x7f1940008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:19.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.192+0000 7f1939ffb640 1 -- 172.21.15.89:0/3734547011 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f193c0519f0 con 0x7f19441538a0 2023-10-30T12:35:19.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.376+0000 7f194b279640 1 -- 172.21.15.89:0/3734547011 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19441f32a0 con 0x7f19441538a0 2023-10-30T12:35:19.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.376+0000 7f1939ffb640 1 -- 172.21.15.89:0/3734547011 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+5745 (secure 0 0 0) 0x7f19441f32a0 con 0x7f19441538a0 2023-10-30T12:35:19.383 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:35:19.383 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:35:18.090610+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:clientreplay","state_seq":27,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:19.384 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 37 2023-10-30T12:35:19.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.380+0000 7f194b279640 1 -- 172.21.15.89:0/3734547011 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f19280664a0 msgr2=0x7f1928068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:19.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.380+0000 7f194b279640 1 --2- 172.21.15.89:0/3734547011 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f19280664a0 0x7f1928068960 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f19400b2000 tx=0x7f1940008040 comp rx=0 tx=0).stop 2023-10-30T12:35:19.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.380+0000 7f194b279640 1 -- 172.21.15.89:0/3734547011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19441538a0 msgr2=0x7f19441ef320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:19.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.380+0000 7f194b279640 1 --2- 172.21.15.89:0/3734547011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19441538a0 0x7f19441ef320 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f193c007d90 tx=0x7f193c00e4b0 comp rx=0 tx=0).stop 2023-10-30T12:35:19.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.380+0000 7f194b279640 1 -- 172.21.15.89:0/3734547011 shutdown_connections 2023-10-30T12:35:19.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.380+0000 7f194b279640 1 --2- 172.21.15.89:0/3734547011 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f19280664a0 0x7f1928068960 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:19.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.380+0000 7f194b279640 1 --2- 172.21.15.89:0/3734547011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19440cf160 0x7f19440ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:19.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.380+0000 7f194b279640 1 --2- 172.21.15.89:0/3734547011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19441538a0 0x7f19441ef320 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:19.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.380+0000 7f194b279640 1 --2- 172.21.15.89:0/3734547011 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19440ca0e0 0x7f194415aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:19.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.380+0000 7f194b279640 1 -- 172.21.15.89:0/3734547011 >> 172.21.15.89:0/3734547011 conn(0x7f194406bfe0 msgr2=0x7f194414db50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:19.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.380+0000 7f194b279640 1 -- 172.21.15.89:0/3734547011 shutdown_connections 2023-10-30T12:35:19.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:19.380+0000 7f194b279640 1 -- 172.21.15.89:0/3734547011 wait complete. 2023-10-30T12:35:19.404 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:35:21.406 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:21.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.548+0000 7f6925965640 1 Processor -- start 2023-10-30T12:35:21.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.548+0000 7f6925965640 1 -- start start 2023-10-30T12:35:21.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.552+0000 7f6925965640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f692015d6f0 0x7f692015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:21.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.552+0000 7f6925965640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69200cf160 0x7f69200cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:21.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.552+0000 7f6925965640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69200cb800 con 0x7f69201538a0 2023-10-30T12:35:21.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.552+0000 7f6925965640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69200cb330 con 0x7f692015d6f0 2023-10-30T12:35:21.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.552+0000 7f6925965640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69200cad70 con 0x7f69200cf160 2023-10-30T12:35:21.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.552+0000 7f691effd640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f69201538a0 0x7f69200ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56884/0 (socket says 172.21.15.89:56884) 2023-10-30T12:35:21.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.552+0000 7f691effd640 1 -- 172.21.15.89:0/3837813421 learned_addr learned my addr 172.21.15.89:0/3837813421 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:35:21.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.552+0000 7f69167fc640 1 --2- 172.21.15.89:0/3837813421 >> v2:172.21.15.89:3300/0 conn(0x7f692015d6f0 0x7f692015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:21.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.552+0000 7f691f7fe640 1 --2- 172.21.15.89:0/3837813421 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69200cf160 0x7f69200cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:21.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.552+0000 7f69167fc640 1 -- 172.21.15.89:0/3837813421 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69200cf160 msgr2=0x7f69200cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:21.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.552+0000 7f69167fc640 1 --2- 172.21.15.89:0/3837813421 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69200cf160 0x7f69200cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:21.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.552+0000 7f69167fc640 1 -- 172.21.15.89:0/3837813421 >> v1:172.21.15.89:6789/0 conn(0x7f69201538a0 legacy=0x7f69200ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:21.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.552+0000 7f69167fc640 1 -- 172.21.15.89:0/3837813421 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69200cb050 con 0x7f692015d6f0 2023-10-30T12:35:21.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.556+0000 7f69167fc640 1 --2- 172.21.15.89:0/3837813421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f692015d6f0 0x7f692015a090 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f69080097c0 tx=0x7f690801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f386195e03e6639 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:21.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.556+0000 7f691e7fc640 1 -- 172.21.15.89:0/3837813421 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69080a8970 con 0x7f692015d6f0 2023-10-30T12:35:21.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.556+0000 7f691e7fc640 1 -- 172.21.15.89:0/3837813421 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69080a8ad0 con 0x7f692015d6f0 2023-10-30T12:35:21.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.556+0000 7f691e7fc640 1 -- 172.21.15.89:0/3837813421 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69080a8e60 con 0x7f692015d6f0 2023-10-30T12:35:21.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.556+0000 7f6925965640 1 -- 172.21.15.89:0/3837813421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f692015d6f0 msgr2=0x7f692015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:21.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.556+0000 7f6925965640 1 --2- 172.21.15.89:0/3837813421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f692015d6f0 0x7f692015a090 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f69080097c0 tx=0x7f690801ec70 comp rx=0 tx=0).stop 2023-10-30T12:35:21.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.556+0000 7f6925965640 1 -- 172.21.15.89:0/3837813421 shutdown_connections 2023-10-30T12:35:21.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.556+0000 7f6925965640 1 --2- 172.21.15.89:0/3837813421 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69200cf160 0x7f69200cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:21.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.556+0000 7f6925965640 1 --2- 172.21.15.89:0/3837813421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f692015d6f0 0x7f692015a090 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:21.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.556+0000 7f6925965640 1 -- 172.21.15.89:0/3837813421 >> 172.21.15.89:0/3837813421 conn(0x7f692006bfe0 msgr2=0x7f692006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:21.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.556+0000 7f6925965640 1 -- 172.21.15.89:0/3837813421 shutdown_connections 2023-10-30T12:35:21.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.556+0000 7f6925965640 1 -- 172.21.15.89:0/3837813421 wait complete. 2023-10-30T12:35:21.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.556+0000 7f6925965640 1 Processor -- start 2023-10-30T12:35:21.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.556+0000 7f6925965640 1 -- start start 2023-10-30T12:35:21.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.560+0000 7f6925965640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69200cf160 0x7f6920159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:21.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.560+0000 7f6925965640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69201538a0 0x7f69200c52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:21.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.560+0000 7f6925965640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f692015d6f0 0x7f69201e2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:21.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.560+0000 7f6925965640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69201e3bf0 con 0x7f69200cf160 2023-10-30T12:35:21.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.560+0000 7f6925965640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69201580f0 con 0x7f692015d6f0 2023-10-30T12:35:21.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.560+0000 7f6925965640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69200c59a0 con 0x7f69201538a0 2023-10-30T12:35:21.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.560+0000 7f691effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69200cf160 0x7f6920159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:21.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.560+0000 7f691effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69200cf160 0x7f6920159300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60186/0 (socket says 172.21.15.89:60186) 2023-10-30T12:35:21.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.560+0000 7f691effd640 1 -- 172.21.15.89:0/450736301 learned_addr learned my addr 172.21.15.89:0/450736301 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:35:21.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.560+0000 7f69167fc640 1 --2- 172.21.15.89:0/450736301 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69201538a0 0x7f69200c52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:21.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.560+0000 7f691f7fe640 1 --2- 172.21.15.89:0/450736301 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f692015d6f0 0x7f69201e2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:21.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.560+0000 7f691effd640 1 -- 172.21.15.89:0/450736301 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69201538a0 msgr2=0x7f69200c52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:21.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.560+0000 7f691effd640 1 --2- 172.21.15.89:0/450736301 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69201538a0 0x7f69200c52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:21.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.560+0000 7f691effd640 1 -- 172.21.15.89:0/450736301 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f692015d6f0 msgr2=0x7f69201e2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:21.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.560+0000 7f691effd640 1 --2- 172.21.15.89:0/450736301 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f692015d6f0 0x7f69201e2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:21.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.560+0000 7f691effd640 1 -- 172.21.15.89:0/450736301 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6908009420 con 0x7f69200cf160 2023-10-30T12:35:21.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.560+0000 7f691effd640 1 --2- 172.21.15.89:0/450736301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69200cf160 0x7f6920159300 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f691000c500 tx=0x7f691000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:21.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.560+0000 7f691cff9640 1 -- 172.21.15.89:0/450736301 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6910005cf0 con 0x7f69200cf160 2023-10-30T12:35:21.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.560+0000 7f691cff9640 1 -- 172.21.15.89:0/450736301 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6910005e50 con 0x7f69200cf160 2023-10-30T12:35:21.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.560+0000 7f6925965640 1 -- 172.21.15.89:0/450736301 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69201584d0 con 0x7f69200cf160 2023-10-30T12:35:21.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.560+0000 7f691cff9640 1 -- 172.21.15.89:0/450736301 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69100072f0 con 0x7f69200cf160 2023-10-30T12:35:21.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.560+0000 7f6925965640 1 -- 172.21.15.89:0/450736301 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69200c4f60 con 0x7f69200cf160 2023-10-30T12:35:21.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.560+0000 7f6925965640 1 -- 172.21.15.89:0/450736301 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69200b2a50 con 0x7f69200cf160 2023-10-30T12:35:21.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.560+0000 7f691cff9640 1 -- 172.21.15.89:0/450736301 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6910007450 con 0x7f69200cf160 2023-10-30T12:35:21.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.564+0000 7f691cff9640 1 --2- 172.21.15.89:0/450736301 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f69000664d0 0x7f6900068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:21.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.564+0000 7f691cff9640 1 -- 172.21.15.89:0/450736301 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f69100834e0 con 0x7f69200cf160 2023-10-30T12:35:21.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.564+0000 7f69167fc640 1 --2- 172.21.15.89:0/450736301 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f69000664d0 0x7f6900068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:21.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.564+0000 7f69167fc640 1 --2- 172.21.15.89:0/450736301 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f69000664d0 0x7f6900068990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f69080b2000 tx=0x7f6908008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:21.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.568+0000 7f691cff9640 1 -- 172.21.15.89:0/450736301 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f69100508b0 con 0x7f69200cf160 2023-10-30T12:35:21.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.748+0000 7f6925965640 1 -- 172.21.15.89:0/450736301 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f692004e170 con 0x7f69200cf160 2023-10-30T12:35:21.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.748+0000 7f691cff9640 1 -- 172.21.15.89:0/450736301 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+5745 (secure 0 0 0) 0x7f6910053ed0 con 0x7f69200cf160 2023-10-30T12:35:21.753 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:35:21.753 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:35:18.090610+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:clientreplay","state_seq":27,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:21.753 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 37 2023-10-30T12:35:21.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.748+0000 7f6925965640 1 -- 172.21.15.89:0/450736301 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f69000664d0 msgr2=0x7f6900068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:21.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.748+0000 7f6925965640 1 --2- 172.21.15.89:0/450736301 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f69000664d0 0x7f6900068990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f69080b2000 tx=0x7f6908008040 comp rx=0 tx=0).stop 2023-10-30T12:35:21.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.748+0000 7f6925965640 1 -- 172.21.15.89:0/450736301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69200cf160 msgr2=0x7f6920159300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:21.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.748+0000 7f6925965640 1 --2- 172.21.15.89:0/450736301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69200cf160 0x7f6920159300 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f691000c500 tx=0x7f691000c9c0 comp rx=0 tx=0).stop 2023-10-30T12:35:21.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.748+0000 7f6925965640 1 -- 172.21.15.89:0/450736301 shutdown_connections 2023-10-30T12:35:21.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.748+0000 7f6925965640 1 --2- 172.21.15.89:0/450736301 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f69000664d0 0x7f6900068990 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:21.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.748+0000 7f6925965640 1 --2- 172.21.15.89:0/450736301 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f692015d6f0 0x7f69201e2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:21.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.748+0000 7f6925965640 1 --2- 172.21.15.89:0/450736301 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f69201538a0 0x7f69200c52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:21.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.748+0000 7f6925965640 1 --2- 172.21.15.89:0/450736301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69200cf160 0x7f6920159300 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:21.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.748+0000 7f6925965640 1 -- 172.21.15.89:0/450736301 >> 172.21.15.89:0/450736301 conn(0x7f692006bfe0 msgr2=0x7f692014ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:21.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.748+0000 7f6925965640 1 -- 172.21.15.89:0/450736301 shutdown_connections 2023-10-30T12:35:21.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:21.748+0000 7f6925965640 1 -- 172.21.15.89:0/450736301 wait complete. 2023-10-30T12:35:21.774 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 37, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:50.729245+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 37, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4318": { "addr": "172.21.15.192:6837/1002124471", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6836", "nonce": 1002124471, "type": "v2" }, { "addr": "172.21.15.192:6837", "nonce": 1002124471, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4318, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4781": { "addr": "172.21.15.192:6835/654256299", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6834", "nonce": 654256299, "type": "v2" }, { "addr": "172.21.15.192:6835", "nonce": 654256299, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4781, "incarnation": 24, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 27 }, "gid_5139": { "addr": "172.21.15.89:6837/951276252", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6836", "nonce": 951276252, "type": "v2" }, { "addr": "172.21.15.89:6837", "nonce": 951276252, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5139, "incarnation": 33, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:clientreplay", "state_seq": 27 } }, "last_failure": 0, "last_failure_osd_epoch": 44, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:35:18.090610+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4318, "mds_1": 5139, "mds_2": 4781 } } } ], "standbys": [ { "addr": "172.21.15.89:6834/2909790173", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6832", "nonce": 2909790173, "type": "v2" }, { "addr": "172.21.15.89:6834", "nonce": 2909790173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.89:6835/2387357615", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6833", "nonce": 2387357615, "type": "v2" }, { "addr": "172.21.15.89:6835", "nonce": 2387357615, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4239, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:35:23.775 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:23.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.920+0000 7f7dcd3c5640 1 Processor -- start 2023-10-30T12:35:23.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.920+0000 7f7dcd3c5640 1 -- start start 2023-10-30T12:35:23.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.920+0000 7f7dcd3c5640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7dc80bfec0 0x7f7dc80c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:23.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.920+0000 7f7dcd3c5640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7dc806a720 0x7f7dc806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:23.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.920+0000 7f7dcd3c5640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc8071290 con 0x7f7dc8159e60 2023-10-30T12:35:23.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.920+0000 7f7dcd3c5640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dc8071570 con 0x7f7dc80bfec0 2023-10-30T12:35:23.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.920+0000 7f7dcd3c5640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dc8071850 con 0x7f7dc806a720 2023-10-30T12:35:23.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.920+0000 7f7dbe7fc640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7dc80bfec0 0x7f7dc80c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:23.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.920+0000 7f7dbe7fc640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7dc80bfec0 0x7f7dc80c1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60200/0 (socket says 172.21.15.89:60200) 2023-10-30T12:35:23.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.920+0000 7f7dbe7fc640 1 -- 172.21.15.89:0/816463383 learned_addr learned my addr 172.21.15.89:0/816463383 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:35:23.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.920+0000 7f7dc77fe640 1 --2- 172.21.15.89:0/816463383 >> v2:172.21.15.192:3300/0 conn(0x7f7dc806a720 0x7f7dc806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:23.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.924+0000 7f7dbe7fc640 1 -- 172.21.15.89:0/816463383 >> v2:172.21.15.192:3300/0 conn(0x7f7dc806a720 msgr2=0x7f7dc806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:23.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.924+0000 7f7dbe7fc640 1 --2- 172.21.15.89:0/816463383 >> v2:172.21.15.192:3300/0 conn(0x7f7dc806a720 0x7f7dc806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:23.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.924+0000 7f7dbe7fc640 1 -- 172.21.15.89:0/816463383 >> v1:172.21.15.192:6789/0 conn(0x7f7dc8159e60 legacy=0x7f7dc806f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:23.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.924+0000 7f7dbe7fc640 1 -- 172.21.15.89:0/816463383 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc8071b30 con 0x7f7dc80bfec0 2023-10-30T12:35:23.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.924+0000 7f7dc77fe640 1 --2- 172.21.15.89:0/816463383 >> v2:172.21.15.192:3300/0 conn(0x7f7dc806a720 0x7f7dc806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:35:23.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.924+0000 7f7dbe7fc640 1 --2- 172.21.15.89:0/816463383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dc80bfec0 0x7f7dc80c1a40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f7dc8071e10 tx=0x7f7db80213c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21eb40c83027b7e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:23.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.924+0000 7f7dc67fc640 1 -- 172.21.15.89:0/816463383 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db80b1600 con 0x7f7dc80bfec0 2023-10-30T12:35:23.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.924+0000 7f7dc67fc640 1 -- 172.21.15.89:0/816463383 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db80ad050 con 0x7f7dc80bfec0 2023-10-30T12:35:23.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.924+0000 7f7dc67fc640 1 -- 172.21.15.89:0/816463383 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db80b1980 con 0x7f7dc80bfec0 2023-10-30T12:35:23.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.924+0000 7f7dcd3c5640 1 -- 172.21.15.89:0/816463383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dc80bfec0 msgr2=0x7f7dc80c1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:23.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.924+0000 7f7dcd3c5640 1 --2- 172.21.15.89:0/816463383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dc80bfec0 0x7f7dc80c1a40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f7dc8071e10 tx=0x7f7db80213c0 comp rx=0 tx=0).stop 2023-10-30T12:35:23.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.928+0000 7f7dcd3c5640 1 -- 172.21.15.89:0/816463383 shutdown_connections 2023-10-30T12:35:23.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.928+0000 7f7dcd3c5640 1 --2- 172.21.15.89:0/816463383 >> v2:172.21.15.192:3300/0 conn(0x7f7dc806a720 0x7f7dc806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:23.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.928+0000 7f7dcd3c5640 1 --2- 172.21.15.89:0/816463383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dc80bfec0 0x7f7dc80c1a40 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:23.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.928+0000 7f7dcd3c5640 1 -- 172.21.15.89:0/816463383 >> 172.21.15.89:0/816463383 conn(0x7f7dc80c7910 msgr2=0x7f7dc80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:23.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.928+0000 7f7dcd3c5640 1 -- 172.21.15.89:0/816463383 shutdown_connections 2023-10-30T12:35:23.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.928+0000 7f7dcd3c5640 1 -- 172.21.15.89:0/816463383 wait complete. 2023-10-30T12:35:23.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.928+0000 7f7dcd3c5640 1 Processor -- start 2023-10-30T12:35:23.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.928+0000 7f7dcd3c5640 1 -- start start 2023-10-30T12:35:23.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.928+0000 7f7dcd3c5640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dc806a720 0x7f7dc80c67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:23.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.928+0000 7f7dcd3c5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dc80bfec0 0x7f7dc814b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:23.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.928+0000 7f7dcd3c5640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7dc8159e60 0x7f7dc81461d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:23.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.928+0000 7f7dcd3c5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc814c8d0 con 0x7f7dc80bfec0 2023-10-30T12:35:23.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.928+0000 7f7dcd3c5640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc8152c30 con 0x7f7dc8159e60 2023-10-30T12:35:23.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.928+0000 7f7dcd3c5640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc80c6ce0 con 0x7f7dc806a720 2023-10-30T12:35:23.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.928+0000 7f7dbe7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dc80bfec0 0x7f7dc814b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:23.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.928+0000 7f7dbe7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dc80bfec0 0x7f7dc814b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60206/0 (socket says 172.21.15.89:60206) 2023-10-30T12:35:23.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.928+0000 7f7dbe7fc640 1 -- 172.21.15.89:0/34592940 learned_addr learned my addr 172.21.15.89:0/34592940 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:35:23.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.928+0000 7f7dc6ffd640 1 --2- 172.21.15.89:0/34592940 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dc806a720 0x7f7dc80c67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:23.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.928+0000 7f7dc77fe640 1 --2- 172.21.15.89:0/34592940 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7dc8159e60 0x7f7dc81461d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:23.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.928+0000 7f7dbe7fc640 1 -- 172.21.15.89:0/34592940 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dc806a720 msgr2=0x7f7dc80c67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:23.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.928+0000 7f7dbe7fc640 1 --2- 172.21.15.89:0/34592940 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dc806a720 0x7f7dc80c67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:23.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.928+0000 7f7dbe7fc640 1 -- 172.21.15.89:0/34592940 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7dc8159e60 msgr2=0x7f7dc81461d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:23.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.928+0000 7f7dbe7fc640 1 --2- 172.21.15.89:0/34592940 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7dc8159e60 0x7f7dc81461d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:23.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.928+0000 7f7dbe7fc640 1 -- 172.21.15.89:0/34592940 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db800b770 con 0x7f7dc80bfec0 2023-10-30T12:35:23.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.932+0000 7f7dbe7fc640 1 --2- 172.21.15.89:0/34592940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dc80bfec0 0x7f7dc814b2d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7db801dfd0 tx=0x7f7db801fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:23.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.932+0000 7f7dbffff640 1 -- 172.21.15.89:0/34592940 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db80b97c0 con 0x7f7dc80bfec0 2023-10-30T12:35:23.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.932+0000 7f7dbffff640 1 -- 172.21.15.89:0/34592940 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db80b9920 con 0x7f7dc80bfec0 2023-10-30T12:35:23.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.932+0000 7f7dcd3c5640 1 -- 172.21.15.89:0/34592940 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dc814c030 con 0x7f7dc80bfec0 2023-10-30T12:35:23.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.932+0000 7f7dbffff640 1 -- 172.21.15.89:0/34592940 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db80b9cb0 con 0x7f7dc80bfec0 2023-10-30T12:35:23.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.932+0000 7f7dcd3c5640 1 -- 172.21.15.89:0/34592940 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dc814c5f0 con 0x7f7dc80bfec0 2023-10-30T12:35:23.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.932+0000 7f7dcd3c5640 1 -- 172.21.15.89:0/34592940 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dc804e170 con 0x7f7dc80bfec0 2023-10-30T12:35:23.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.932+0000 7f7dbffff640 1 -- 172.21.15.89:0/34592940 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7db80c5460 con 0x7f7dc80bfec0 2023-10-30T12:35:23.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.932+0000 7f7dbffff640 1 --2- 172.21.15.89:0/34592940 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f7da80664a0 0x7f7da8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:23.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.932+0000 7f7dbffff640 1 -- 172.21.15.89:0/34592940 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f7db81229e0 con 0x7f7dc80bfec0 2023-10-30T12:35:23.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.932+0000 7f7dc6ffd640 1 --2- 172.21.15.89:0/34592940 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f7da80664a0 0x7f7da8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:23.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.936+0000 7f7dc6ffd640 1 --2- 172.21.15.89:0/34592940 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f7da80664a0 0x7f7da8068960 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7db0000f70 tx=0x7f7db0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:23.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:23.936+0000 7f7dbffff640 1 -- 172.21.15.89:0/34592940 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7db80efc60 con 0x7f7dc80bfec0 2023-10-30T12:35:24.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:24.132+0000 7f7dcd3c5640 1 -- 172.21.15.89:0/34592940 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7dc80c81a0 con 0x7f7dc80bfec0 2023-10-30T12:35:24.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:24.132+0000 7f7dbffff640 1 -- 172.21.15.89:0/34592940 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+5745 (secure 0 0 0) 0x7f7db80b6070 con 0x7f7dc80bfec0 2023-10-30T12:35:24.137 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:35:24.137 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:35:18.090610+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:clientreplay","state_seq":27,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:24.137 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 37 2023-10-30T12:35:24.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:24.132+0000 7f7dcd3c5640 1 -- 172.21.15.89:0/34592940 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f7da80664a0 msgr2=0x7f7da8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:24.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:24.132+0000 7f7dcd3c5640 1 --2- 172.21.15.89:0/34592940 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f7da80664a0 0x7f7da8068960 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7db0000f70 tx=0x7f7db0009040 comp rx=0 tx=0).stop 2023-10-30T12:35:24.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:24.132+0000 7f7dcd3c5640 1 -- 172.21.15.89:0/34592940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dc80bfec0 msgr2=0x7f7dc814b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:24.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:24.132+0000 7f7dcd3c5640 1 --2- 172.21.15.89:0/34592940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dc80bfec0 0x7f7dc814b2d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7db801dfd0 tx=0x7f7db801fc00 comp rx=0 tx=0).stop 2023-10-30T12:35:24.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:24.132+0000 7f7dcd3c5640 1 -- 172.21.15.89:0/34592940 shutdown_connections 2023-10-30T12:35:24.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:24.132+0000 7f7dcd3c5640 1 --2- 172.21.15.89:0/34592940 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f7da80664a0 0x7f7da8068960 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:24.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:24.132+0000 7f7dcd3c5640 1 --2- 172.21.15.89:0/34592940 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7dc8159e60 0x7f7dc81461d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:24.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:24.132+0000 7f7dcd3c5640 1 --2- 172.21.15.89:0/34592940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7dc80bfec0 0x7f7dc814b2d0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:24.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:24.132+0000 7f7dcd3c5640 1 --2- 172.21.15.89:0/34592940 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dc806a720 0x7f7dc80c67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:24.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:24.132+0000 7f7dcd3c5640 1 -- 172.21.15.89:0/34592940 >> 172.21.15.89:0/34592940 conn(0x7f7dc80c7910 msgr2=0x7f7dc80c8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:24.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:24.136+0000 7f7dcd3c5640 1 -- 172.21.15.89:0/34592940 shutdown_connections 2023-10-30T12:35:24.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:24.136+0000 7f7dcd3c5640 1 -- 172.21.15.89:0/34592940 wait complete. 2023-10-30T12:35:24.165 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:35:25.346 INFO:tasks.ceph.mds.f.smithi192.stderr:2023-10-30T12:35:25.341+0000 7f0036e29640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T12:35:26.166 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:26.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.308+0000 7fa9b3a46640 1 Processor -- start 2023-10-30T12:35:26.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.308+0000 7fa9b3a46640 1 -- start start 2023-10-30T12:35:26.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.312+0000 7fa9b3a46640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa9ac0c6b80 0x7fa9ac0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:26.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.312+0000 7fa9b3a46640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9ac13f4a0 0x7fa9ac0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:26.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.312+0000 7fa9b3a46640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9ac0c0a50 con 0x7fa9ac0c6f60 2023-10-30T12:35:26.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.312+0000 7fa9b3a46640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9ac06f6e0 con 0x7fa9ac0c6b80 2023-10-30T12:35:26.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.312+0000 7fa9b3a46640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac06f9c0 con 0x7fa9ac13f4a0 2023-10-30T12:35:26.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.312+0000 7fa9b1fe3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9ac13f4a0 0x7fa9ac0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:26.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.312+0000 7fa9b17e2640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa9ac0c6b80 0x7fa9ac0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:26.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.312+0000 7fa9b0fe1640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fa9ac0c6f60 0x7fa9ac13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:40320/0 (socket says 172.21.15.89:40320) 2023-10-30T12:35:26.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.312+0000 7fa9b1fe3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9ac13f4a0 0x7fa9ac0c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:42778/0 (socket says 172.21.15.89:42778) 2023-10-30T12:35:26.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.312+0000 7fa9b17e2640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa9ac0c6b80 0x7fa9ac0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:45412/0 (socket says 172.21.15.89:45412) 2023-10-30T12:35:26.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.312+0000 7fa9b0fe1640 1 -- 172.21.15.89:0/2880137322 learned_addr learned my addr 172.21.15.89:0/2880137322 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:35:26.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.312+0000 7fa9a3fff640 1 -- 172.21.15.89:0/2880137322 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3870315574 0 0) 0x7fa9ac0c0a50 con 0x7fa9ac0c6f60 2023-10-30T12:35:26.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.312+0000 7fa9a3fff640 1 -- 172.21.15.89:0/2880137322 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa998003530 con 0x7fa9ac0c6f60 2023-10-30T12:35:26.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.312+0000 7fa9b17e2640 1 -- 172.21.15.89:0/2880137322 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9ac13f4a0 msgr2=0x7fa9ac0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:26.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.312+0000 7fa9b17e2640 1 --2- 172.21.15.89:0/2880137322 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9ac13f4a0 0x7fa9ac0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:26.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.312+0000 7fa9b17e2640 1 -- 172.21.15.89:0/2880137322 >> v1:172.21.15.192:6789/0 conn(0x7fa9ac0c6f60 legacy=0x7fa9ac13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:26.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.312+0000 7fa9b17e2640 1 -- 172.21.15.89:0/2880137322 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ac06fca0 con 0x7fa9ac0c6b80 2023-10-30T12:35:26.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.312+0000 7fa9b1fe3640 1 --2- 172.21.15.89:0/2880137322 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9ac13f4a0 0x7fa9ac0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:35:26.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.316+0000 7fa9b17e2640 1 --2- 172.21.15.89:0/2880137322 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9ac0c6b80 0x7fa9ac0bfec0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa9a80097c0 tx=0x7fa9a801c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=490b1eb4cdee8f32 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:26.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.316+0000 7fa9a3fff640 1 -- 172.21.15.89:0/2880137322 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a80ad640 con 0x7fa9ac0c6b80 2023-10-30T12:35:26.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.316+0000 7fa9a3fff640 1 -- 172.21.15.89:0/2880137322 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a80ad7a0 con 0x7fa9ac0c6b80 2023-10-30T12:35:26.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.316+0000 7fa9a3fff640 1 -- 172.21.15.89:0/2880137322 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a80adb10 con 0x7fa9ac0c6b80 2023-10-30T12:35:26.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.316+0000 7fa9b3a46640 1 -- 172.21.15.89:0/2880137322 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9ac0c6b80 msgr2=0x7fa9ac0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:26.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.316+0000 7fa9b3a46640 1 --2- 172.21.15.89:0/2880137322 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9ac0c6b80 0x7fa9ac0bfec0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa9a80097c0 tx=0x7fa9a801c480 comp rx=0 tx=0).stop 2023-10-30T12:35:26.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.316+0000 7fa9b3a46640 1 -- 172.21.15.89:0/2880137322 shutdown_connections 2023-10-30T12:35:26.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.316+0000 7fa9b3a46640 1 --2- 172.21.15.89:0/2880137322 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9ac13f4a0 0x7fa9ac0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:26.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.316+0000 7fa9b3a46640 1 --2- 172.21.15.89:0/2880137322 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9ac0c6b80 0x7fa9ac0bfec0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:26.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.316+0000 7fa9b3a46640 1 -- 172.21.15.89:0/2880137322 >> 172.21.15.89:0/2880137322 conn(0x7fa9ac0c7910 msgr2=0x7fa9ac0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:26.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.316+0000 7fa9b3a46640 1 -- 172.21.15.89:0/2880137322 shutdown_connections 2023-10-30T12:35:26.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.316+0000 7fa9b3a46640 1 -- 172.21.15.89:0/2880137322 wait complete. 2023-10-30T12:35:26.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.316+0000 7fa9b3a46640 1 Processor -- start 2023-10-30T12:35:26.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.320+0000 7fa9b3a46640 1 -- start start 2023-10-30T12:35:26.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.320+0000 7fa9b3a46640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9ac0c6b80 0x7fa9ac155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:26.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.320+0000 7fa9b3a46640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9ac0c6f60 0x7fa9ac154670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:26.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.320+0000 7fa9b3a46640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9ac13f4a0 0x7fa9ac158e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:26.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.320+0000 7fa9b3a46640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac156500 con 0x7fa9ac0c6b80 2023-10-30T12:35:26.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.320+0000 7fa9b3a46640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac1369d0 con 0x7fa9ac13f4a0 2023-10-30T12:35:26.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.320+0000 7fa9b3a46640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac13af80 con 0x7fa9ac0c6f60 2023-10-30T12:35:26.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.320+0000 7fa9b17e2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9ac0c6b80 0x7fa9ac155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:26.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.320+0000 7fa9b17e2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9ac0c6b80 0x7fa9ac155fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60218/0 (socket says 172.21.15.89:60218) 2023-10-30T12:35:26.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.320+0000 7fa9b17e2640 1 -- 172.21.15.89:0/4008305944 learned_addr learned my addr 172.21.15.89:0/4008305944 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:35:26.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.320+0000 7fa9b0fe1640 1 --2- 172.21.15.89:0/4008305944 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9ac0c6f60 0x7fa9ac154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:26.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.320+0000 7fa9b1fe3640 1 --2- 172.21.15.89:0/4008305944 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9ac13f4a0 0x7fa9ac158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:26.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.320+0000 7fa9b17e2640 1 -- 172.21.15.89:0/4008305944 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9ac0c6f60 msgr2=0x7fa9ac154670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:26.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.320+0000 7fa9b17e2640 1 --2- 172.21.15.89:0/4008305944 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9ac0c6f60 0x7fa9ac154670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:26.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.320+0000 7fa9b17e2640 1 -- 172.21.15.89:0/4008305944 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9ac13f4a0 msgr2=0x7fa9ac158e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:26.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.320+0000 7fa9b17e2640 1 --2- 172.21.15.89:0/4008305944 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9ac13f4a0 0x7fa9ac158e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:26.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.320+0000 7fa9b17e2640 1 -- 172.21.15.89:0/4008305944 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a4009a70 con 0x7fa9ac0c6b80 2023-10-30T12:35:26.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.320+0000 7fa9b0fe1640 1 --2- 172.21.15.89:0/4008305944 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9ac0c6f60 0x7fa9ac154670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:35:26.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.320+0000 7fa9b17e2640 1 --2- 172.21.15.89:0/4008305944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9ac0c6b80 0x7fa9ac155fc0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fa9a801a0b0 tx=0x7fa9a8019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:26.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.320+0000 7fa9a1ffb640 1 -- 172.21.15.89:0/4008305944 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a80bc070 con 0x7fa9ac0c6b80 2023-10-30T12:35:26.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.320+0000 7fa9a1ffb640 1 -- 172.21.15.89:0/4008305944 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a80a6050 con 0x7fa9ac0c6b80 2023-10-30T12:35:26.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.320+0000 7fa9a1ffb640 1 -- 172.21.15.89:0/4008305944 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a80b6840 con 0x7fa9ac0c6b80 2023-10-30T12:35:26.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.320+0000 7fa9b3a46640 1 -- 172.21.15.89:0/4008305944 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9a8009420 con 0x7fa9ac0c6b80 2023-10-30T12:35:26.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.320+0000 7fa9b3a46640 1 -- 172.21.15.89:0/4008305944 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9ac13b500 con 0x7fa9ac0c6b80 2023-10-30T12:35:26.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.320+0000 7fa9b3a46640 1 -- 172.21.15.89:0/4008305944 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9ac04e170 con 0x7fa9ac0c6b80 2023-10-30T12:35:26.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.324+0000 7fa9a1ffb640 1 -- 172.21.15.89:0/4008305944 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa9a80b4040 con 0x7fa9ac0c6b80 2023-10-30T12:35:26.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.324+0000 7fa9a1ffb640 1 --2- 172.21.15.89:0/4008305944 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa9900664a0 0x7fa990068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:26.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.324+0000 7fa9a1ffb640 1 -- 172.21.15.89:0/4008305944 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa9a811dbe0 con 0x7fa9ac0c6b80 2023-10-30T12:35:26.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.324+0000 7fa9b0fe1640 1 --2- 172.21.15.89:0/4008305944 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa9900664a0 0x7fa990068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:26.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.324+0000 7fa9b0fe1640 1 --2- 172.21.15.89:0/4008305944 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa9900664a0 0x7fa990068960 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac0c56e0 tx=0x7fa99c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:26.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.328+0000 7fa9a1ffb640 1 -- 172.21.15.89:0/4008305944 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa9a80eae60 con 0x7fa9ac0c6b80 2023-10-30T12:35:26.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.508+0000 7fa9b3a46640 1 -- 172.21.15.89:0/4008305944 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9ac0b0cf0 con 0x7fa9ac0c6b80 2023-10-30T12:35:26.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.512+0000 7fa9a1ffb640 1 -- 172.21.15.89:0/4008305944 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5739 (secure 0 0 0) 0x7fa9a80ee480 con 0x7fa9ac0c6b80 2023-10-30T12:35:26.515 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:35:26.516 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:35:24.153774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:26.516 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 38 2023-10-30T12:35:26.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.512+0000 7fa9b3a46640 1 -- 172.21.15.89:0/4008305944 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa9900664a0 msgr2=0x7fa990068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:26.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.512+0000 7fa9b3a46640 1 --2- 172.21.15.89:0/4008305944 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa9900664a0 0x7fa990068960 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac0c56e0 tx=0x7fa99c009040 comp rx=0 tx=0).stop 2023-10-30T12:35:26.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.512+0000 7fa9b3a46640 1 -- 172.21.15.89:0/4008305944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9ac0c6b80 msgr2=0x7fa9ac155fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:26.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.512+0000 7fa9b3a46640 1 --2- 172.21.15.89:0/4008305944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9ac0c6b80 0x7fa9ac155fc0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fa9a801a0b0 tx=0x7fa9a8019220 comp rx=0 tx=0).stop 2023-10-30T12:35:26.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.512+0000 7fa9b3a46640 1 -- 172.21.15.89:0/4008305944 shutdown_connections 2023-10-30T12:35:26.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.512+0000 7fa9b3a46640 1 --2- 172.21.15.89:0/4008305944 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa9900664a0 0x7fa990068960 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:26.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.512+0000 7fa9b3a46640 1 --2- 172.21.15.89:0/4008305944 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9ac13f4a0 0x7fa9ac158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:26.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.512+0000 7fa9b3a46640 1 --2- 172.21.15.89:0/4008305944 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9ac0c6f60 0x7fa9ac154670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:26.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.512+0000 7fa9b3a46640 1 --2- 172.21.15.89:0/4008305944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9ac0c6b80 0x7fa9ac155fc0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:26.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.512+0000 7fa9b3a46640 1 -- 172.21.15.89:0/4008305944 >> 172.21.15.89:0/4008305944 conn(0x7fa9ac0c7910 msgr2=0x7fa9ac0c5810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:26.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.512+0000 7fa9b3a46640 1 -- 172.21.15.89:0/4008305944 shutdown_connections 2023-10-30T12:35:26.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:26.512+0000 7fa9b3a46640 1 -- 172.21.15.89:0/4008305944 wait complete. 2023-10-30T12:35:26.535 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=1, replacing gid=4262 2023-10-30T12:35:26.536 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.d 2023-10-30T12:35:28.537 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-10-30T12:35:28.537 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-30T12:35:28.538 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-30T12:35:28.541 INFO:tasks.ceph.mds.d:Started 2023-10-30T12:35:28.541 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:28.617 INFO:tasks.ceph.mds.d.smithi192.stdout:starting mds.d at 2023-10-30T12:35:28.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.684+0000 7f770f8bb640 1 Processor -- start 2023-10-30T12:35:28.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.684+0000 7f770f8bb640 1 -- start start 2023-10-30T12:35:28.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.684+0000 7f770f8bb640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f770806f7a0 0x7f770806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:28.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.684+0000 7f770f8bb640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f77080c6bd0 0x7f7708069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:28.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.684+0000 7f770f8bb640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f770806a3d0 con 0x7f7708071110 2023-10-30T12:35:28.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.684+0000 7f770f8bb640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77080700c0 con 0x7f770806f7a0 2023-10-30T12:35:28.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.684+0000 7f770f8bb640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7708066cf0 con 0x7f77080c6bd0 2023-10-30T12:35:28.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.684+0000 7f770d657640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f770806f7a0 0x7f770806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:28.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.684+0000 7f770d657640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f770806f7a0 0x7f770806fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60232/0 (socket says 172.21.15.89:60232) 2023-10-30T12:35:28.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.684+0000 7f770d657640 1 -- 172.21.15.89:0/4029785438 learned_addr learned my addr 172.21.15.89:0/4029785438 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:35:28.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.688+0000 7f770ce56640 1 --2- 172.21.15.89:0/4029785438 >> v2:172.21.15.192:3300/0 conn(0x7f77080c6bd0 0x7f7708069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:28.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.688+0000 7f770d657640 1 -- 172.21.15.89:0/4029785438 >> v2:172.21.15.192:3300/0 conn(0x7f77080c6bd0 msgr2=0x7f7708069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:28.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.688+0000 7f770d657640 1 --2- 172.21.15.89:0/4029785438 >> v2:172.21.15.192:3300/0 conn(0x7f77080c6bd0 0x7f7708069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:28.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.688+0000 7f770d657640 1 -- 172.21.15.89:0/4029785438 >> v1:172.21.15.192:6789/0 conn(0x7f7708071110 legacy=0x7f7708158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:28.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.688+0000 7f770d657640 1 -- 172.21.15.89:0/4029785438 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77080c70d0 con 0x7f770806f7a0 2023-10-30T12:35:28.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.688+0000 7f770ce56640 1 --2- 172.21.15.89:0/4029785438 >> v2:172.21.15.192:3300/0 conn(0x7f77080c6bd0 0x7f7708069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:35:28.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.688+0000 7f770d657640 1 --2- 172.21.15.89:0/4029785438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f770806f7a0 0x7f770806fb80 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f770806f1c0 tx=0x7f770401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0ead38ace251070 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:28.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.688+0000 7f76fffff640 1 -- 172.21.15.89:0/4029785438 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77040a89f0 con 0x7f770806f7a0 2023-10-30T12:35:28.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.688+0000 7f76fffff640 1 -- 172.21.15.89:0/4029785438 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77040a8b50 con 0x7f770806f7a0 2023-10-30T12:35:28.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.688+0000 7f76fffff640 1 -- 172.21.15.89:0/4029785438 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77040b0440 con 0x7f770806f7a0 2023-10-30T12:35:28.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.688+0000 7f770f8bb640 1 -- 172.21.15.89:0/4029785438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f770806f7a0 msgr2=0x7f770806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:28.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.688+0000 7f770f8bb640 1 --2- 172.21.15.89:0/4029785438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f770806f7a0 0x7f770806fb80 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f770806f1c0 tx=0x7f770401ecf0 comp rx=0 tx=0).stop 2023-10-30T12:35:28.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.692+0000 7f770f8bb640 1 -- 172.21.15.89:0/4029785438 shutdown_connections 2023-10-30T12:35:28.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.692+0000 7f770f8bb640 1 --2- 172.21.15.89:0/4029785438 >> v2:172.21.15.192:3300/0 conn(0x7f77080c6bd0 0x7f7708069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:28.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.692+0000 7f770f8bb640 1 --2- 172.21.15.89:0/4029785438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f770806f7a0 0x7f770806fb80 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:28.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.692+0000 7f770f8bb640 1 -- 172.21.15.89:0/4029785438 >> 172.21.15.89:0/4029785438 conn(0x7f77080c7910 msgr2=0x7f77080c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:28.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.692+0000 7f770f8bb640 1 -- 172.21.15.89:0/4029785438 shutdown_connections 2023-10-30T12:35:28.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.692+0000 7f770f8bb640 1 -- 172.21.15.89:0/4029785438 wait complete. 2023-10-30T12:35:28.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.692+0000 7f770f8bb640 1 Processor -- start 2023-10-30T12:35:28.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.692+0000 7f770f8bb640 1 -- start start 2023-10-30T12:35:28.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.692+0000 7f770f8bb640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f770806f7a0 0x7f77081f2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:28.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.692+0000 7f770f8bb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7708071110 0x7f77081e1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:28.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.692+0000 7f770f8bb640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77080c6bd0 0x7f77081e2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:28.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.692+0000 7f770f8bb640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77081579f0 con 0x7f7708071110 2023-10-30T12:35:28.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.692+0000 7f770f8bb640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7708150fe0 con 0x7f77080c6bd0 2023-10-30T12:35:28.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.692+0000 7f770f8bb640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77081f2cd0 con 0x7f770806f7a0 2023-10-30T12:35:28.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.692+0000 7f770ce56640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7708071110 0x7f77081e1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:28.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.692+0000 7f770ce56640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7708071110 0x7f77081e1bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60242/0 (socket says 172.21.15.89:60242) 2023-10-30T12:35:28.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.692+0000 7f770ce56640 1 -- 172.21.15.89:0/358968943 learned_addr learned my addr 172.21.15.89:0/358968943 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:35:28.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.692+0000 7f770d657640 1 --2- 172.21.15.89:0/358968943 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f770806f7a0 0x7f77081f2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:28.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.692+0000 7f770de58640 1 --2- 172.21.15.89:0/358968943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77080c6bd0 0x7f77081e2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:28.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.692+0000 7f770ce56640 1 -- 172.21.15.89:0/358968943 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f770806f7a0 msgr2=0x7f77081f2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:28.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.692+0000 7f770ce56640 1 --2- 172.21.15.89:0/358968943 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f770806f7a0 0x7f77081f2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:28.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.692+0000 7f770ce56640 1 -- 172.21.15.89:0/358968943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77080c6bd0 msgr2=0x7f77081e2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:28.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.692+0000 7f770ce56640 1 --2- 172.21.15.89:0/358968943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77080c6bd0 0x7f77081e2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:28.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.692+0000 7f770ce56640 1 -- 172.21.15.89:0/358968943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77040094d0 con 0x7f7708071110 2023-10-30T12:35:28.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.692+0000 7f770de58640 1 --2- 172.21.15.89:0/358968943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77080c6bd0 0x7f77081e2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:35:28.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.696+0000 7f770ce56640 1 --2- 172.21.15.89:0/358968943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7708071110 0x7f77081e1bd0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f76f800d7d0 tx=0x7f76f800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:28.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.696+0000 7f76fdffb640 1 -- 172.21.15.89:0/358968943 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f80096a0 con 0x7f7708071110 2023-10-30T12:35:28.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.696+0000 7f76fdffb640 1 -- 172.21.15.89:0/358968943 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76f8009800 con 0x7f7708071110 2023-10-30T12:35:28.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.696+0000 7f76fdffb640 1 -- 172.21.15.89:0/358968943 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f8009bb0 con 0x7f7708071110 2023-10-30T12:35:28.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.696+0000 7f770f8bb640 1 -- 172.21.15.89:0/358968943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7708157050 con 0x7f7708071110 2023-10-30T12:35:28.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.696+0000 7f770f8bb640 1 -- 172.21.15.89:0/358968943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7708157610 con 0x7f7708071110 2023-10-30T12:35:28.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.696+0000 7f770f8bb640 1 -- 172.21.15.89:0/358968943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7708157c60 con 0x7f7708071110 2023-10-30T12:35:28.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.696+0000 7f76fdffb640 1 -- 172.21.15.89:0/358968943 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f76f8009d10 con 0x7f7708071110 2023-10-30T12:35:28.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.700+0000 7f76fdffb640 1 --2- 172.21.15.89:0/358968943 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f76ec0664a0 0x7f76ec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:28.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.700+0000 7f76fdffb640 1 -- 172.21.15.89:0/358968943 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f76f8084740 con 0x7f7708071110 2023-10-30T12:35:28.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.700+0000 7f770d657640 1 --2- 172.21.15.89:0/358968943 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f76ec0664a0 0x7f76ec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:28.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.700+0000 7f770d657640 1 --2- 172.21.15.89:0/358968943 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f76ec0664a0 0x7f76ec068960 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f77040b2000 tx=0x7f7704009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:28.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.704+0000 7f76fdffb640 1 -- 172.21.15.89:0/358968943 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f76f80519c0 con 0x7f7708071110 2023-10-30T12:35:28.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.880+0000 7f770f8bb640 1 -- 172.21.15.89:0/358968943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f770804e170 con 0x7f7708071110 2023-10-30T12:35:28.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.880+0000 7f76fdffb640 1 -- 172.21.15.89:0/358968943 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f76f8054fe0 con 0x7f7708071110 2023-10-30T12:35:28.886 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:35:28.886 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:35:24.153774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:28.887 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 38 2023-10-30T12:35:28.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.884+0000 7f770f8bb640 1 -- 172.21.15.89:0/358968943 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f76ec0664a0 msgr2=0x7f76ec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:28.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.884+0000 7f770f8bb640 1 --2- 172.21.15.89:0/358968943 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f76ec0664a0 0x7f76ec068960 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f77040b2000 tx=0x7f7704009040 comp rx=0 tx=0).stop 2023-10-30T12:35:28.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.884+0000 7f770f8bb640 1 -- 172.21.15.89:0/358968943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7708071110 msgr2=0x7f77081e1bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:28.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.884+0000 7f770f8bb640 1 --2- 172.21.15.89:0/358968943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7708071110 0x7f77081e1bd0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f76f800d7d0 tx=0x7f76f800dc90 comp rx=0 tx=0).stop 2023-10-30T12:35:28.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.884+0000 7f770f8bb640 1 -- 172.21.15.89:0/358968943 shutdown_connections 2023-10-30T12:35:28.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.884+0000 7f770f8bb640 1 --2- 172.21.15.89:0/358968943 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f76ec0664a0 0x7f76ec068960 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:28.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.884+0000 7f770f8bb640 1 --2- 172.21.15.89:0/358968943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77080c6bd0 0x7f77081e2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:28.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.884+0000 7f770f8bb640 1 --2- 172.21.15.89:0/358968943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7708071110 0x7f77081e1bd0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:28.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.884+0000 7f770f8bb640 1 --2- 172.21.15.89:0/358968943 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f770806f7a0 0x7f77081f2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:28.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.884+0000 7f770f8bb640 1 -- 172.21.15.89:0/358968943 >> 172.21.15.89:0/358968943 conn(0x7f77080c7910 msgr2=0x7f77080c9450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:28.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.884+0000 7f770f8bb640 1 -- 172.21.15.89:0/358968943 shutdown_connections 2023-10-30T12:35:28.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:28.884+0000 7f770f8bb640 1 -- 172.21.15.89:0/358968943 wait complete. 2023-10-30T12:35:28.907 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-10-30T12:35:30.908 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:31.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.052+0000 7fdcf166b640 1 Processor -- start 2023-10-30T12:35:31.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.052+0000 7fdcf166b640 1 -- start start 2023-10-30T12:35:31.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.056+0000 7fdcf166b640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fdcec0c91b0 0x7fdcec15a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:31.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.056+0000 7fdcf166b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcec0c6550 0x7fdcec0c6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:31.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.056+0000 7fdcf166b640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcec06ac40 con 0x7fdcec071800 2023-10-30T12:35:31.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.056+0000 7fdcf166b640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcec06bce0 con 0x7fdcec0c91b0 2023-10-30T12:35:31.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.056+0000 7fdcf166b640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec06a030 con 0x7fdcec0c6550 2023-10-30T12:35:31.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.056+0000 7fdcea7fc640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fdcec0c91b0 0x7fdcec15a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:31.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.056+0000 7fdceb7fe640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcec0c6550 0x7fdcec0c6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:31.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.056+0000 7fdceaffd640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fdcec071800 0x7fdcec071be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:32942/0 (socket says 172.21.15.89:32942) 2023-10-30T12:35:31.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.056+0000 7fdceb7fe640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcec0c6550 0x7fdcec0c6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:51222/0 (socket says 172.21.15.89:51222) 2023-10-30T12:35:31.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.056+0000 7fdcea7fc640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fdcec0c91b0 0x7fdcec15a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:37748/0 (socket says 172.21.15.89:37748) 2023-10-30T12:35:31.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.056+0000 7fdceaffd640 1 -- 172.21.15.89:0/2994149442 learned_addr learned my addr 172.21.15.89:0/2994149442 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:35:31.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.056+0000 7fdcea7fc640 1 -- 172.21.15.89:0/2994149442 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcec0c6550 msgr2=0x7fdcec0c6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:31.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.056+0000 7fdcea7fc640 1 --2- 172.21.15.89:0/2994149442 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcec0c6550 0x7fdcec0c6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:31.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.056+0000 7fdcea7fc640 1 -- 172.21.15.89:0/2994149442 >> v1:172.21.15.192:6789/0 conn(0x7fdcec071800 legacy=0x7fdcec071be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:31.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.056+0000 7fdcea7fc640 1 -- 172.21.15.89:0/2994149442 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcec06a310 con 0x7fdcec0c91b0 2023-10-30T12:35:31.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.056+0000 7fdceb7fe640 1 --2- 172.21.15.89:0/2994149442 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcec0c6550 0x7fdcec0c6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:35:31.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.060+0000 7fdcea7fc640 1 --2- 172.21.15.89:0/2994149442 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcec0c91b0 0x7fdcec15a050 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdce00097c0 tx=0x7fdce001c310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f3ef447df342893 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:31.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.060+0000 7fdce9ffb640 1 -- 172.21.15.89:0/2994149442 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce00ab600 con 0x7fdcec0c91b0 2023-10-30T12:35:31.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.060+0000 7fdce9ffb640 1 -- 172.21.15.89:0/2994149442 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdce00a7050 con 0x7fdcec0c91b0 2023-10-30T12:35:31.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.060+0000 7fdce9ffb640 1 -- 172.21.15.89:0/2994149442 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce00ab980 con 0x7fdcec0c91b0 2023-10-30T12:35:31.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.060+0000 7fdcf166b640 1 -- 172.21.15.89:0/2994149442 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcec0c91b0 msgr2=0x7fdcec15a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:31.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.060+0000 7fdcf166b640 1 --2- 172.21.15.89:0/2994149442 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcec0c91b0 0x7fdcec15a050 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdce00097c0 tx=0x7fdce001c310 comp rx=0 tx=0).stop 2023-10-30T12:35:31.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.060+0000 7fdcf166b640 1 -- 172.21.15.89:0/2994149442 shutdown_connections 2023-10-30T12:35:31.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.060+0000 7fdcf166b640 1 --2- 172.21.15.89:0/2994149442 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcec0c6550 0x7fdcec0c6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:31.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.060+0000 7fdcf166b640 1 --2- 172.21.15.89:0/2994149442 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcec0c91b0 0x7fdcec15a050 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:31.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.060+0000 7fdcf166b640 1 -- 172.21.15.89:0/2994149442 >> 172.21.15.89:0/2994149442 conn(0x7fdcec0c8650 msgr2=0x7fdcec0c8a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:31.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.060+0000 7fdcf166b640 1 -- 172.21.15.89:0/2994149442 shutdown_connections 2023-10-30T12:35:31.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.060+0000 7fdcf166b640 1 -- 172.21.15.89:0/2994149442 wait complete. 2023-10-30T12:35:31.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.060+0000 7fdcf166b640 1 Processor -- start 2023-10-30T12:35:31.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.060+0000 7fdcf166b640 1 -- start start 2023-10-30T12:35:31.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.064+0000 7fdcf166b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcec071800 0x7fdcec0cb280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:31.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.064+0000 7fdcf166b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcec0c6550 0x7fdcec0ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:31.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.064+0000 7fdcf166b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcec0c91b0 0x7fdcec0ca790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:31.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.064+0000 7fdcf166b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec0c9dc0 con 0x7fdcec0c91b0 2023-10-30T12:35:31.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.064+0000 7fdcf166b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec1554d0 con 0x7fdcec071800 2023-10-30T12:35:31.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.064+0000 7fdcf166b640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec155770 con 0x7fdcec0c6550 2023-10-30T12:35:31.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.064+0000 7fdceb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcec0c91b0 0x7fdcec0ca790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:31.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.064+0000 7fdceb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcec0c91b0 0x7fdcec0ca790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53068/0 (socket says 172.21.15.89:53068) 2023-10-30T12:35:31.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.064+0000 7fdceb7fe640 1 -- 172.21.15.89:0/524017037 learned_addr learned my addr 172.21.15.89:0/524017037 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:35:31.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.064+0000 7fdceaffd640 1 --2- 172.21.15.89:0/524017037 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcec071800 0x7fdcec0cb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:31.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.064+0000 7fdcea7fc640 1 --2- 172.21.15.89:0/524017037 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcec0c6550 0x7fdcec0ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:31.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.064+0000 7fdceb7fe640 1 -- 172.21.15.89:0/524017037 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcec0c6550 msgr2=0x7fdcec0ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:31.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.064+0000 7fdceb7fe640 1 --2- 172.21.15.89:0/524017037 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcec0c6550 0x7fdcec0ca250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:31.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.064+0000 7fdceb7fe640 1 -- 172.21.15.89:0/524017037 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcec071800 msgr2=0x7fdcec0cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:31.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.064+0000 7fdceb7fe640 1 --2- 172.21.15.89:0/524017037 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcec071800 0x7fdcec0cb280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:31.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.064+0000 7fdceb7fe640 1 -- 172.21.15.89:0/524017037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce4009a70 con 0x7fdcec0c91b0 2023-10-30T12:35:31.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.064+0000 7fdceaffd640 1 --2- 172.21.15.89:0/524017037 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcec071800 0x7fdcec0cb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:35:31.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.064+0000 7fdceb7fe640 1 --2- 172.21.15.89:0/524017037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcec0c91b0 0x7fdcec0ca790 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fdce400efc0 tx=0x7fdce400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:31.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.064+0000 7fdcd37fe640 1 -- 172.21.15.89:0/524017037 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce4009cd0 con 0x7fdcec0c91b0 2023-10-30T12:35:31.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.064+0000 7fdcd37fe640 1 -- 172.21.15.89:0/524017037 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdce400f040 con 0x7fdcec0c91b0 2023-10-30T12:35:31.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.064+0000 7fdcd37fe640 1 -- 172.21.15.89:0/524017037 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce40039a0 con 0x7fdcec0c91b0 2023-10-30T12:35:31.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.064+0000 7fdcf166b640 1 -- 172.21.15.89:0/524017037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdce0009420 con 0x7fdcec0c91b0 2023-10-30T12:35:31.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.064+0000 7fdcf166b640 1 -- 172.21.15.89:0/524017037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcec0c9a50 con 0x7fdcec0c91b0 2023-10-30T12:35:31.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.064+0000 7fdcf166b640 1 -- 172.21.15.89:0/524017037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcec06ef10 con 0x7fdcec0c91b0 2023-10-30T12:35:31.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.064+0000 7fdcd37fe640 1 -- 172.21.15.89:0/524017037 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdce4014480 con 0x7fdcec0c91b0 2023-10-30T12:35:31.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.068+0000 7fdcd37fe640 1 --2- 172.21.15.89:0/524017037 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fdcd80664a0 0x7fdcd8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:31.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.068+0000 7fdcd37fe640 1 -- 172.21.15.89:0/524017037 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fdce4084c80 con 0x7fdcec0c91b0 2023-10-30T12:35:31.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.068+0000 7fdceaffd640 1 --2- 172.21.15.89:0/524017037 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fdcd80664a0 0x7fdcd8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:31.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.068+0000 7fdceaffd640 1 --2- 172.21.15.89:0/524017037 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fdcd80664a0 0x7fdcd8068960 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc000f70 tx=0x7fdcdc00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:31.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.072+0000 7fdcd37fe640 1 -- 172.21.15.89:0/524017037 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdce4051e70 con 0x7fdcec0c91b0 2023-10-30T12:35:31.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.256+0000 7fdcf166b640 1 -- 172.21.15.89:0/524017037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdcec072530 con 0x7fdcec0c91b0 2023-10-30T12:35:31.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.256+0000 7fdcd37fe640 1 -- 172.21.15.89:0/524017037 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+6527 (secure 0 0 0) 0x7fdce4056e60 con 0x7fdcec0c91b0 2023-10-30T12:35:31.260 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:35:31.260 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5018,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:35:24.153774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:31.261 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 39 2023-10-30T12:35:31.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.256+0000 7fdcf166b640 1 -- 172.21.15.89:0/524017037 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fdcd80664a0 msgr2=0x7fdcd8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:31.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.256+0000 7fdcf166b640 1 --2- 172.21.15.89:0/524017037 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fdcd80664a0 0x7fdcd8068960 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc000f70 tx=0x7fdcdc00b040 comp rx=0 tx=0).stop 2023-10-30T12:35:31.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.256+0000 7fdcf166b640 1 -- 172.21.15.89:0/524017037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcec0c91b0 msgr2=0x7fdcec0ca790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:31.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.256+0000 7fdcf166b640 1 --2- 172.21.15.89:0/524017037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcec0c91b0 0x7fdcec0ca790 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fdce400efc0 tx=0x7fdce400c490 comp rx=0 tx=0).stop 2023-10-30T12:35:31.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.256+0000 7fdcf166b640 1 -- 172.21.15.89:0/524017037 shutdown_connections 2023-10-30T12:35:31.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.256+0000 7fdcf166b640 1 --2- 172.21.15.89:0/524017037 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fdcd80664a0 0x7fdcd8068960 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:31.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.256+0000 7fdcf166b640 1 --2- 172.21.15.89:0/524017037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcec0c91b0 0x7fdcec0ca790 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:31.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.256+0000 7fdcf166b640 1 --2- 172.21.15.89:0/524017037 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcec0c6550 0x7fdcec0ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:31.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.256+0000 7fdcf166b640 1 --2- 172.21.15.89:0/524017037 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcec071800 0x7fdcec0cb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:31.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.256+0000 7fdcf166b640 1 -- 172.21.15.89:0/524017037 >> 172.21.15.89:0/524017037 conn(0x7fdcec0c8650 msgr2=0x7fdcec14e010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:31.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.256+0000 7fdcf166b640 1 -- 172.21.15.89:0/524017037 shutdown_connections 2023-10-30T12:35:31.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:35:31.256+0000 7fdcf166b640 1 -- 172.21.15.89:0/524017037 wait complete. 2023-10-30T12:35:31.280 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-10-30T12:35:31.281 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 107 secs before thrashing 2023-10-30T12:35:32.783 INFO:tasks.workunit.client.0.smithi089.stdout:removed './167/11' 2023-10-30T12:35:32.783 INFO:tasks.workunit.client.0.smithi089.stdout:removed './167/16' 2023-10-30T12:35:32.783 INFO:tasks.workunit.client.0.smithi089.stdout:removed './167/32' 2023-10-30T12:35:32.784 INFO:tasks.workunit.client.0.smithi089.stdout:removed './167/30' 2023-10-30T12:35:32.784 INFO:tasks.workunit.client.0.smithi089.stdout:removed './167/14' 2023-10-30T12:35:32.784 INFO:tasks.workunit.client.0.smithi089.stdout:removed './167/18' 2023-10-30T12:35:32.784 INFO:tasks.workunit.client.0.smithi089.stdout:removed './167/13' 2023-10-30T12:35:32.784 INFO:tasks.workunit.client.0.smithi089.stdout:removed './167/33' 2023-10-30T12:35:32.784 INFO:tasks.workunit.client.0.smithi089.stdout:removed './167/17' 2023-10-30T12:35:32.785 INFO:tasks.workunit.client.0.smithi089.stdout:removed './167/35' 2023-10-30T12:35:32.785 INFO:tasks.workunit.client.0.smithi089.stdout:removed './167/37' 2023-10-30T12:35:32.785 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './167' 2023-10-30T12:35:32.785 INFO:tasks.workunit.client.0.smithi089.stdout:removed './168/19' 2023-10-30T12:35:32.785 INFO:tasks.workunit.client.0.smithi089.stdout:removed './168/15' 2023-10-30T12:35:32.786 INFO:tasks.workunit.client.0.smithi089.stdout:removed './168/31' 2023-10-30T12:35:32.786 INFO:tasks.workunit.client.0.smithi089.stdout:removed './168/36' 2023-10-30T12:35:32.786 INFO:tasks.workunit.client.0.smithi089.stdout:removed './168/20' 2023-10-30T12:35:32.786 INFO:tasks.workunit.client.0.smithi089.stdout:removed './168/12' 2023-10-30T12:35:32.786 INFO:tasks.workunit.client.0.smithi089.stdout:removed './168/39' 2023-10-30T12:35:32.787 INFO:tasks.workunit.client.0.smithi089.stdout:removed './168/10' 2023-10-30T12:35:32.787 INFO:tasks.workunit.client.0.smithi089.stdout:removed './168/34' 2023-10-30T12:35:32.787 INFO:tasks.workunit.client.0.smithi089.stdout:removed './168/38' 2023-10-30T12:35:32.787 INFO:tasks.workunit.client.0.smithi089.stdout:removed './168/11' 2023-10-30T12:35:32.787 INFO:tasks.workunit.client.0.smithi089.stdout:removed './168/16' 2023-10-30T12:35:32.787 INFO:tasks.workunit.client.0.smithi089.stdout:removed './168/32' 2023-10-30T12:35:32.788 INFO:tasks.workunit.client.0.smithi089.stdout:removed './168/30' 2023-10-30T12:35:32.788 INFO:tasks.workunit.client.0.smithi089.stdout:removed './168/14' 2023-10-30T12:35:32.788 INFO:tasks.workunit.client.0.smithi089.stdout:removed './168/18' 2023-10-30T12:35:32.788 INFO:tasks.workunit.client.0.smithi089.stdout:removed './168/13' 2023-10-30T12:35:32.788 INFO:tasks.workunit.client.0.smithi089.stdout:removed './168/33' 2023-10-30T12:35:32.789 INFO:tasks.workunit.client.0.smithi089.stdout:removed './168/17' 2023-10-30T12:35:32.789 INFO:tasks.workunit.client.0.smithi089.stdout:removed './168/35' 2023-10-30T12:35:32.789 INFO:tasks.workunit.client.0.smithi089.stdout:removed './168/37' 2023-10-30T12:35:32.789 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './168' 2023-10-30T12:35:32.789 INFO:tasks.workunit.client.0.smithi089.stdout:removed './169/19' 2023-10-30T12:35:32.790 INFO:tasks.workunit.client.0.smithi089.stdout:removed './169/15' 2023-10-30T12:35:32.790 INFO:tasks.workunit.client.0.smithi089.stdout:removed './169/31' 2023-10-30T12:35:32.790 INFO:tasks.workunit.client.0.smithi089.stdout:removed './169/36' 2023-10-30T12:35:32.790 INFO:tasks.workunit.client.0.smithi089.stdout:removed './169/20' 2023-10-30T12:35:32.790 INFO:tasks.workunit.client.0.smithi089.stdout:removed './169/12' 2023-10-30T12:35:32.790 INFO:tasks.workunit.client.0.smithi089.stdout:removed './169/39' 2023-10-30T12:35:32.791 INFO:tasks.workunit.client.0.smithi089.stdout:removed './169/10' 2023-10-30T12:35:32.791 INFO:tasks.workunit.client.0.smithi089.stdout:removed './169/34' 2023-10-30T12:35:32.791 INFO:tasks.workunit.client.0.smithi089.stdout:removed './169/38' 2023-10-30T12:35:32.791 INFO:tasks.workunit.client.0.smithi089.stdout:removed './169/11' 2023-10-30T12:35:32.791 INFO:tasks.workunit.client.0.smithi089.stdout:removed './169/16' 2023-10-30T12:35:32.791 INFO:tasks.workunit.client.0.smithi089.stdout:removed './169/32' 2023-10-30T12:35:32.792 INFO:tasks.workunit.client.0.smithi089.stdout:removed './169/30' 2023-10-30T12:35:32.792 INFO:tasks.workunit.client.0.smithi089.stdout:removed './169/14' 2023-10-30T12:35:32.792 INFO:tasks.workunit.client.0.smithi089.stdout:removed './169/18' 2023-10-30T12:35:32.792 INFO:tasks.workunit.client.0.smithi089.stdout:removed './169/13' 2023-10-30T12:35:32.792 INFO:tasks.workunit.client.0.smithi089.stdout:removed './169/33' 2023-10-30T12:35:32.793 INFO:tasks.workunit.client.0.smithi089.stdout:removed './169/17' 2023-10-30T12:35:32.793 INFO:tasks.workunit.client.0.smithi089.stdout:removed './169/35' 2023-10-30T12:35:32.793 INFO:tasks.workunit.client.0.smithi089.stdout:removed './169/37' 2023-10-30T12:35:32.793 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './169' 2023-10-30T12:35:32.793 INFO:tasks.workunit.client.0.smithi089.stdout:removed './170/19' 2023-10-30T12:35:32.794 INFO:tasks.workunit.client.0.smithi089.stdout:removed './170/15' 2023-10-30T12:35:32.794 INFO:tasks.workunit.client.0.smithi089.stdout:removed './170/31' 2023-10-30T12:35:32.794 INFO:tasks.workunit.client.0.smithi089.stdout:removed './170/36' 2023-10-30T12:35:32.794 INFO:tasks.workunit.client.0.smithi089.stdout:removed './170/20' 2023-10-30T12:35:32.794 INFO:tasks.workunit.client.0.smithi089.stdout:removed './170/12' 2023-10-30T12:35:32.795 INFO:tasks.workunit.client.0.smithi089.stdout:removed './170/39' 2023-10-30T12:35:32.795 INFO:tasks.workunit.client.0.smithi089.stdout:removed './170/10' 2023-10-30T12:35:32.795 INFO:tasks.workunit.client.0.smithi089.stdout:removed './170/34' 2023-10-30T12:35:32.795 INFO:tasks.workunit.client.0.smithi089.stdout:removed './170/38' 2023-10-30T12:35:32.795 INFO:tasks.workunit.client.0.smithi089.stdout:removed './170/11' 2023-10-30T12:35:32.795 INFO:tasks.workunit.client.0.smithi089.stdout:removed './170/16' 2023-10-30T12:35:32.796 INFO:tasks.workunit.client.0.smithi089.stdout:removed './170/32' 2023-10-30T12:35:32.796 INFO:tasks.workunit.client.0.smithi089.stdout:removed './170/30' 2023-10-30T12:35:32.796 INFO:tasks.workunit.client.0.smithi089.stdout:removed './170/14' 2023-10-30T12:35:32.796 INFO:tasks.workunit.client.0.smithi089.stdout:removed './170/18' 2023-10-30T12:35:32.796 INFO:tasks.workunit.client.0.smithi089.stdout:removed './170/13' 2023-10-30T12:35:32.797 INFO:tasks.workunit.client.0.smithi089.stdout:removed './170/33' 2023-10-30T12:35:32.797 INFO:tasks.workunit.client.0.smithi089.stdout:removed './170/17' 2023-10-30T12:35:32.797 INFO:tasks.workunit.client.0.smithi089.stdout:removed './170/35' 2023-10-30T12:35:32.797 INFO:tasks.workunit.client.0.smithi089.stdout:removed './170/37' 2023-10-30T12:35:32.797 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './170' 2023-10-30T12:35:32.798 INFO:tasks.workunit.client.0.smithi089.stdout:removed './171/19' 2023-10-30T12:35:32.798 INFO:tasks.workunit.client.0.smithi089.stdout:removed './171/15' 2023-10-30T12:35:32.798 INFO:tasks.workunit.client.0.smithi089.stdout:removed './171/31' 2023-10-30T12:35:32.798 INFO:tasks.workunit.client.0.smithi089.stdout:removed './171/36' 2023-10-30T12:35:32.798 INFO:tasks.workunit.client.0.smithi089.stdout:removed './171/20' 2023-10-30T12:35:32.799 INFO:tasks.workunit.client.0.smithi089.stdout:removed './171/12' 2023-10-30T12:35:32.799 INFO:tasks.workunit.client.0.smithi089.stdout:removed './171/39' 2023-10-30T12:35:32.799 INFO:tasks.workunit.client.0.smithi089.stdout:removed './171/10' 2023-10-30T12:35:32.799 INFO:tasks.workunit.client.0.smithi089.stdout:removed './171/34' 2023-10-30T12:35:32.799 INFO:tasks.workunit.client.0.smithi089.stdout:removed './171/38' 2023-10-30T12:35:32.799 INFO:tasks.workunit.client.0.smithi089.stdout:removed './171/11' 2023-10-30T12:35:32.800 INFO:tasks.workunit.client.0.smithi089.stdout:removed './171/16' 2023-10-30T12:35:32.800 INFO:tasks.workunit.client.0.smithi089.stdout:removed './171/32' 2023-10-30T12:35:32.800 INFO:tasks.workunit.client.0.smithi089.stdout:removed './171/30' 2023-10-30T12:35:32.800 INFO:tasks.workunit.client.0.smithi089.stdout:removed './171/14' 2023-10-30T12:35:32.800 INFO:tasks.workunit.client.0.smithi089.stdout:removed './171/18' 2023-10-30T12:35:32.801 INFO:tasks.workunit.client.0.smithi089.stdout:removed './171/13' 2023-10-30T12:35:32.801 INFO:tasks.workunit.client.0.smithi089.stdout:removed './171/33' 2023-10-30T12:35:32.801 INFO:tasks.workunit.client.0.smithi089.stdout:removed './171/17' 2023-10-30T12:35:32.801 INFO:tasks.workunit.client.0.smithi089.stdout:removed './171/35' 2023-10-30T12:35:32.801 INFO:tasks.workunit.client.0.smithi089.stdout:removed './171/37' 2023-10-30T12:35:32.801 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './171' 2023-10-30T12:35:32.802 INFO:tasks.workunit.client.0.smithi089.stdout:removed './172/19' 2023-10-30T12:35:32.802 INFO:tasks.workunit.client.0.smithi089.stdout:removed './172/15' 2023-10-30T12:35:32.802 INFO:tasks.workunit.client.0.smithi089.stdout:removed './172/31' 2023-10-30T12:35:32.802 INFO:tasks.workunit.client.0.smithi089.stdout:removed './172/36' 2023-10-30T12:35:32.802 INFO:tasks.workunit.client.0.smithi089.stdout:removed './172/20' 2023-10-30T12:35:32.802 INFO:tasks.workunit.client.0.smithi089.stdout:removed './172/12' 2023-10-30T12:35:32.803 INFO:tasks.workunit.client.0.smithi089.stdout:removed './172/39' 2023-10-30T12:35:32.803 INFO:tasks.workunit.client.0.smithi089.stdout:removed './172/10' 2023-10-30T12:35:32.803 INFO:tasks.workunit.client.0.smithi089.stdout:removed './172/34' 2023-10-30T12:35:32.803 INFO:tasks.workunit.client.0.smithi089.stdout:removed './172/38' 2023-10-30T12:35:32.803 INFO:tasks.workunit.client.0.smithi089.stdout:removed './172/11' 2023-10-30T12:35:32.803 INFO:tasks.workunit.client.0.smithi089.stdout:removed './172/16' 2023-10-30T12:35:32.804 INFO:tasks.workunit.client.0.smithi089.stdout:removed './172/32' 2023-10-30T12:35:32.804 INFO:tasks.workunit.client.0.smithi089.stdout:removed './172/30' 2023-10-30T12:35:32.804 INFO:tasks.workunit.client.0.smithi089.stdout:removed './172/14' 2023-10-30T12:35:32.804 INFO:tasks.workunit.client.0.smithi089.stdout:removed './172/18' 2023-10-30T12:35:32.804 INFO:tasks.workunit.client.0.smithi089.stdout:removed './172/13' 2023-10-30T12:35:32.805 INFO:tasks.workunit.client.0.smithi089.stdout:removed './172/33' 2023-10-30T12:35:32.805 INFO:tasks.workunit.client.0.smithi089.stdout:removed './172/17' 2023-10-30T12:35:32.805 INFO:tasks.workunit.client.0.smithi089.stdout:removed './172/35' 2023-10-30T12:35:32.805 INFO:tasks.workunit.client.0.smithi089.stdout:removed './172/37' 2023-10-30T12:35:32.805 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './172' 2023-10-30T12:35:32.806 INFO:tasks.workunit.client.0.smithi089.stdout:removed './173/19' 2023-10-30T12:35:32.806 INFO:tasks.workunit.client.0.smithi089.stdout:removed './173/15' 2023-10-30T12:35:32.806 INFO:tasks.workunit.client.0.smithi089.stdout:removed './173/31' 2023-10-30T12:35:32.806 INFO:tasks.workunit.client.0.smithi089.stdout:removed './173/36' 2023-10-30T12:35:32.807 INFO:tasks.workunit.client.0.smithi089.stdout:removed './173/20' 2023-10-30T12:35:32.807 INFO:tasks.workunit.client.0.smithi089.stdout:removed './173/12' 2023-10-30T12:35:32.807 INFO:tasks.workunit.client.0.smithi089.stdout:removed './173/39' 2023-10-30T12:35:32.807 INFO:tasks.workunit.client.0.smithi089.stdout:removed './173/10' 2023-10-30T12:35:32.807 INFO:tasks.workunit.client.0.smithi089.stdout:removed './173/34' 2023-10-30T12:35:32.808 INFO:tasks.workunit.client.0.smithi089.stdout:removed './173/38' 2023-10-30T12:35:32.808 INFO:tasks.workunit.client.0.smithi089.stdout:removed './173/11' 2023-10-30T12:35:32.808 INFO:tasks.workunit.client.0.smithi089.stdout:removed './173/16' 2023-10-30T12:35:32.808 INFO:tasks.workunit.client.0.smithi089.stdout:removed './173/32' 2023-10-30T12:35:32.808 INFO:tasks.workunit.client.0.smithi089.stdout:removed './173/30' 2023-10-30T12:35:32.808 INFO:tasks.workunit.client.0.smithi089.stdout:removed './173/14' 2023-10-30T12:35:32.809 INFO:tasks.workunit.client.0.smithi089.stdout:removed './173/18' 2023-10-30T12:35:32.809 INFO:tasks.workunit.client.0.smithi089.stdout:removed './173/13' 2023-10-30T12:35:32.809 INFO:tasks.workunit.client.0.smithi089.stdout:removed './173/33' 2023-10-30T12:35:32.809 INFO:tasks.workunit.client.0.smithi089.stdout:removed './173/17' 2023-10-30T12:35:32.809 INFO:tasks.workunit.client.0.smithi089.stdout:removed './173/35' 2023-10-30T12:35:32.810 INFO:tasks.workunit.client.0.smithi089.stdout:removed './173/37' 2023-10-30T12:35:32.810 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './173' 2023-10-30T12:35:32.810 INFO:tasks.workunit.client.0.smithi089.stdout:removed './174/19' 2023-10-30T12:35:32.810 INFO:tasks.workunit.client.0.smithi089.stdout:removed './174/15' 2023-10-30T12:35:32.810 INFO:tasks.workunit.client.0.smithi089.stdout:removed './174/31' 2023-10-30T12:35:32.811 INFO:tasks.workunit.client.0.smithi089.stdout:removed './174/36' 2023-10-30T12:35:32.811 INFO:tasks.workunit.client.0.smithi089.stdout:removed './174/20' 2023-10-30T12:35:32.811 INFO:tasks.workunit.client.0.smithi089.stdout:removed './174/12' 2023-10-30T12:35:32.811 INFO:tasks.workunit.client.0.smithi089.stdout:removed './174/39' 2023-10-30T12:35:32.811 INFO:tasks.workunit.client.0.smithi089.stdout:removed './174/10' 2023-10-30T12:35:32.812 INFO:tasks.workunit.client.0.smithi089.stdout:removed './174/34' 2023-10-30T12:35:32.812 INFO:tasks.workunit.client.0.smithi089.stdout:removed './174/38' 2023-10-30T12:35:32.812 INFO:tasks.workunit.client.0.smithi089.stdout:removed './174/11' 2023-10-30T12:35:32.812 INFO:tasks.workunit.client.0.smithi089.stdout:removed './174/16' 2023-10-30T12:35:32.812 INFO:tasks.workunit.client.0.smithi089.stdout:removed './174/32' 2023-10-30T12:35:32.813 INFO:tasks.workunit.client.0.smithi089.stdout:removed './174/30' 2023-10-30T12:35:32.813 INFO:tasks.workunit.client.0.smithi089.stdout:removed './174/14' 2023-10-30T12:35:32.813 INFO:tasks.workunit.client.0.smithi089.stdout:removed './174/18' 2023-10-30T12:35:32.813 INFO:tasks.workunit.client.0.smithi089.stdout:removed './174/13' 2023-10-30T12:35:32.813 INFO:tasks.workunit.client.0.smithi089.stdout:removed './174/33' 2023-10-30T12:35:32.813 INFO:tasks.workunit.client.0.smithi089.stdout:removed './174/17' 2023-10-30T12:35:32.814 INFO:tasks.workunit.client.0.smithi089.stdout:removed './174/35' 2023-10-30T12:35:32.814 INFO:tasks.workunit.client.0.smithi089.stdout:removed './174/37' 2023-10-30T12:35:32.814 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './174' 2023-10-30T12:35:32.814 INFO:tasks.workunit.client.0.smithi089.stdout:removed './175/19' 2023-10-30T12:35:32.814 INFO:tasks.workunit.client.0.smithi089.stdout:removed './175/15' 2023-10-30T12:35:32.815 INFO:tasks.workunit.client.0.smithi089.stdout:removed './175/31' 2023-10-30T12:35:32.815 INFO:tasks.workunit.client.0.smithi089.stdout:removed './175/36' 2023-10-30T12:35:32.815 INFO:tasks.workunit.client.0.smithi089.stdout:removed './175/20' 2023-10-30T12:35:32.815 INFO:tasks.workunit.client.0.smithi089.stdout:removed './175/12' 2023-10-30T12:35:32.815 INFO:tasks.workunit.client.0.smithi089.stdout:removed './175/39' 2023-10-30T12:35:32.816 INFO:tasks.workunit.client.0.smithi089.stdout:removed './175/10' 2023-10-30T12:35:32.816 INFO:tasks.workunit.client.0.smithi089.stdout:removed './175/34' 2023-10-30T12:35:32.816 INFO:tasks.workunit.client.0.smithi089.stdout:removed './175/38' 2023-10-30T12:35:32.816 INFO:tasks.workunit.client.0.smithi089.stdout:removed './175/11' 2023-10-30T12:35:32.816 INFO:tasks.workunit.client.0.smithi089.stdout:removed './175/16' 2023-10-30T12:35:32.817 INFO:tasks.workunit.client.0.smithi089.stdout:removed './175/32' 2023-10-30T12:35:32.817 INFO:tasks.workunit.client.0.smithi089.stdout:removed './175/30' 2023-10-30T12:35:32.817 INFO:tasks.workunit.client.0.smithi089.stdout:removed './175/14' 2023-10-30T12:35:32.817 INFO:tasks.workunit.client.0.smithi089.stdout:removed './175/18' 2023-10-30T12:35:32.817 INFO:tasks.workunit.client.0.smithi089.stdout:removed './175/13' 2023-10-30T12:35:32.818 INFO:tasks.workunit.client.0.smithi089.stdout:removed './175/33' 2023-10-30T12:35:32.818 INFO:tasks.workunit.client.0.smithi089.stdout:removed './175/17' 2023-10-30T12:35:32.818 INFO:tasks.workunit.client.0.smithi089.stdout:removed './175/35' 2023-10-30T12:35:32.818 INFO:tasks.workunit.client.0.smithi089.stdout:removed './175/37' 2023-10-30T12:35:32.818 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './175' 2023-10-30T12:35:32.819 INFO:tasks.workunit.client.0.smithi089.stdout:removed './176/19' 2023-10-30T12:35:32.819 INFO:tasks.workunit.client.0.smithi089.stdout:removed './176/15' 2023-10-30T12:35:32.819 INFO:tasks.workunit.client.0.smithi089.stdout:removed './176/31' 2023-10-30T12:35:32.819 INFO:tasks.workunit.client.0.smithi089.stdout:removed './176/36' 2023-10-30T12:35:32.819 INFO:tasks.workunit.client.0.smithi089.stdout:removed './176/20' 2023-10-30T12:35:32.820 INFO:tasks.workunit.client.0.smithi089.stdout:removed './176/12' 2023-10-30T12:35:32.820 INFO:tasks.workunit.client.0.smithi089.stdout:removed './176/39' 2023-10-30T12:35:32.820 INFO:tasks.workunit.client.0.smithi089.stdout:removed './176/10' 2023-10-30T12:35:32.820 INFO:tasks.workunit.client.0.smithi089.stdout:removed './176/34' 2023-10-30T12:35:32.820 INFO:tasks.workunit.client.0.smithi089.stdout:removed './176/38' 2023-10-30T12:35:32.820 INFO:tasks.workunit.client.0.smithi089.stdout:removed './176/11' 2023-10-30T12:35:32.821 INFO:tasks.workunit.client.0.smithi089.stdout:removed './176/16' 2023-10-30T12:35:32.821 INFO:tasks.workunit.client.0.smithi089.stdout:removed './176/32' 2023-10-30T12:35:32.821 INFO:tasks.workunit.client.0.smithi089.stdout:removed './176/30' 2023-10-30T12:35:32.821 INFO:tasks.workunit.client.0.smithi089.stdout:removed './176/14' 2023-10-30T12:35:32.821 INFO:tasks.workunit.client.0.smithi089.stdout:removed './176/18' 2023-10-30T12:35:32.821 INFO:tasks.workunit.client.0.smithi089.stdout:removed './176/13' 2023-10-30T12:35:32.822 INFO:tasks.workunit.client.0.smithi089.stdout:removed './176/33' 2023-10-30T12:35:32.822 INFO:tasks.workunit.client.0.smithi089.stdout:removed './176/17' 2023-10-30T12:35:32.822 INFO:tasks.workunit.client.0.smithi089.stdout:removed './176/35' 2023-10-30T12:35:32.822 INFO:tasks.workunit.client.0.smithi089.stdout:removed './176/37' 2023-10-30T12:35:32.822 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './176' 2023-10-30T12:35:32.823 INFO:tasks.workunit.client.0.smithi089.stdout:removed './177/19' 2023-10-30T12:35:32.823 INFO:tasks.workunit.client.0.smithi089.stdout:removed './177/15' 2023-10-30T12:35:35.148 INFO:tasks.workunit.client.0.smithi089.stdout:removed './177/31' 2023-10-30T12:35:35.149 INFO:tasks.workunit.client.0.smithi089.stdout:removed './177/36' 2023-10-30T12:35:35.149 INFO:tasks.workunit.client.0.smithi089.stdout:removed './177/20' 2023-10-30T12:35:35.149 INFO:tasks.workunit.client.0.smithi089.stdout:removed './177/12' 2023-10-30T12:35:35.149 INFO:tasks.workunit.client.0.smithi089.stdout:removed './177/39' 2023-10-30T12:35:35.149 INFO:tasks.workunit.client.0.smithi089.stdout:removed './177/10' 2023-10-30T12:35:35.150 INFO:tasks.workunit.client.0.smithi089.stdout:removed './177/34' 2023-10-30T12:35:35.150 INFO:tasks.workunit.client.0.smithi089.stdout:removed './177/38' 2023-10-30T12:35:35.150 INFO:tasks.workunit.client.0.smithi089.stdout:removed './177/11' 2023-10-30T12:35:35.150 INFO:tasks.workunit.client.0.smithi089.stdout:removed './177/16' 2023-10-30T12:35:35.150 INFO:tasks.workunit.client.0.smithi089.stdout:removed './177/32' 2023-10-30T12:35:35.151 INFO:tasks.workunit.client.0.smithi089.stdout:removed './177/30' 2023-10-30T12:35:35.151 INFO:tasks.workunit.client.0.smithi089.stdout:removed './177/14' 2023-10-30T12:35:35.151 INFO:tasks.workunit.client.0.smithi089.stdout:removed './177/18' 2023-10-30T12:35:35.151 INFO:tasks.workunit.client.0.smithi089.stdout:removed './177/13' 2023-10-30T12:35:35.151 INFO:tasks.workunit.client.0.smithi089.stdout:removed './177/33' 2023-10-30T12:35:35.152 INFO:tasks.workunit.client.0.smithi089.stdout:removed './177/17' 2023-10-30T12:35:35.152 INFO:tasks.workunit.client.0.smithi089.stdout:removed './177/35' 2023-10-30T12:35:35.152 INFO:tasks.workunit.client.0.smithi089.stdout:removed './177/37' 2023-10-30T12:35:35.152 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './177' 2023-10-30T12:35:35.152 INFO:tasks.workunit.client.0.smithi089.stdout:removed './178/19' 2023-10-30T12:35:35.153 INFO:tasks.workunit.client.0.smithi089.stdout:removed './178/15' 2023-10-30T12:35:35.153 INFO:tasks.workunit.client.0.smithi089.stdout:removed './178/31' 2023-10-30T12:35:35.153 INFO:tasks.workunit.client.0.smithi089.stdout:removed './178/36' 2023-10-30T12:35:35.153 INFO:tasks.workunit.client.0.smithi089.stdout:removed './178/20' 2023-10-30T12:35:35.153 INFO:tasks.workunit.client.0.smithi089.stdout:removed './178/12' 2023-10-30T12:35:35.154 INFO:tasks.workunit.client.0.smithi089.stdout:removed './178/39' 2023-10-30T12:35:35.154 INFO:tasks.workunit.client.0.smithi089.stdout:removed './178/10' 2023-10-30T12:35:35.154 INFO:tasks.workunit.client.0.smithi089.stdout:removed './178/34' 2023-10-30T12:35:35.154 INFO:tasks.workunit.client.0.smithi089.stdout:removed './178/38' 2023-10-30T12:35:35.155 INFO:tasks.workunit.client.0.smithi089.stdout:removed './178/11' 2023-10-30T12:35:35.155 INFO:tasks.workunit.client.0.smithi089.stdout:removed './178/16' 2023-10-30T12:35:35.155 INFO:tasks.workunit.client.0.smithi089.stdout:removed './178/32' 2023-10-30T12:35:35.155 INFO:tasks.workunit.client.0.smithi089.stdout:removed './178/30' 2023-10-30T12:35:35.155 INFO:tasks.workunit.client.0.smithi089.stdout:removed './178/14' 2023-10-30T12:35:35.156 INFO:tasks.workunit.client.0.smithi089.stdout:removed './178/18' 2023-10-30T12:35:35.156 INFO:tasks.workunit.client.0.smithi089.stdout:removed './178/13' 2023-10-30T12:35:35.156 INFO:tasks.workunit.client.0.smithi089.stdout:removed './178/33' 2023-10-30T12:35:35.156 INFO:tasks.workunit.client.0.smithi089.stdout:removed './178/17' 2023-10-30T12:35:35.156 INFO:tasks.workunit.client.0.smithi089.stdout:removed './178/35' 2023-10-30T12:35:35.157 INFO:tasks.workunit.client.0.smithi089.stdout:removed './178/37' 2023-10-30T12:35:35.157 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './178' 2023-10-30T12:35:35.157 INFO:tasks.workunit.client.0.smithi089.stdout:removed './179/19' 2023-10-30T12:35:35.157 INFO:tasks.workunit.client.0.smithi089.stdout:removed './179/15' 2023-10-30T12:35:35.158 INFO:tasks.workunit.client.0.smithi089.stdout:removed './179/31' 2023-10-30T12:35:35.158 INFO:tasks.workunit.client.0.smithi089.stdout:removed './179/36' 2023-10-30T12:35:35.158 INFO:tasks.workunit.client.0.smithi089.stdout:removed './179/20' 2023-10-30T12:35:35.158 INFO:tasks.workunit.client.0.smithi089.stdout:removed './179/12' 2023-10-30T12:35:35.158 INFO:tasks.workunit.client.0.smithi089.stdout:removed './179/39' 2023-10-30T12:35:35.159 INFO:tasks.workunit.client.0.smithi089.stdout:removed './179/10' 2023-10-30T12:35:35.159 INFO:tasks.workunit.client.0.smithi089.stdout:removed './179/34' 2023-10-30T12:35:35.159 INFO:tasks.workunit.client.0.smithi089.stdout:removed './179/38' 2023-10-30T12:35:35.159 INFO:tasks.workunit.client.0.smithi089.stdout:removed './179/11' 2023-10-30T12:35:35.159 INFO:tasks.workunit.client.0.smithi089.stdout:removed './179/16' 2023-10-30T12:35:35.160 INFO:tasks.workunit.client.0.smithi089.stdout:removed './179/32' 2023-10-30T12:35:35.160 INFO:tasks.workunit.client.0.smithi089.stdout:removed './179/30' 2023-10-30T12:35:35.160 INFO:tasks.workunit.client.0.smithi089.stdout:removed './179/14' 2023-10-30T12:35:35.160 INFO:tasks.workunit.client.0.smithi089.stdout:removed './179/18' 2023-10-30T12:35:35.160 INFO:tasks.workunit.client.0.smithi089.stdout:removed './179/13' 2023-10-30T12:35:35.160 INFO:tasks.workunit.client.0.smithi089.stdout:removed './179/33' 2023-10-30T12:35:35.161 INFO:tasks.workunit.client.0.smithi089.stdout:removed './179/17' 2023-10-30T12:35:35.161 INFO:tasks.workunit.client.0.smithi089.stdout:removed './179/35' 2023-10-30T12:35:35.161 INFO:tasks.workunit.client.0.smithi089.stdout:removed './179/37' 2023-10-30T12:35:35.161 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './179' 2023-10-30T12:35:35.161 INFO:tasks.workunit.client.0.smithi089.stdout:removed './180/19' 2023-10-30T12:35:35.162 INFO:tasks.workunit.client.0.smithi089.stdout:removed './180/15' 2023-10-30T12:35:35.162 INFO:tasks.workunit.client.0.smithi089.stdout:removed './180/31' 2023-10-30T12:35:35.162 INFO:tasks.workunit.client.0.smithi089.stdout:removed './180/36' 2023-10-30T12:35:35.162 INFO:tasks.workunit.client.0.smithi089.stdout:removed './180/20' 2023-10-30T12:35:35.162 INFO:tasks.workunit.client.0.smithi089.stdout:removed './180/12' 2023-10-30T12:35:35.163 INFO:tasks.workunit.client.0.smithi089.stdout:removed './180/39' 2023-10-30T12:35:35.163 INFO:tasks.workunit.client.0.smithi089.stdout:removed './180/10' 2023-10-30T12:35:35.163 INFO:tasks.workunit.client.0.smithi089.stdout:removed './180/34' 2023-10-30T12:35:35.163 INFO:tasks.workunit.client.0.smithi089.stdout:removed './180/38' 2023-10-30T12:35:35.163 INFO:tasks.workunit.client.0.smithi089.stdout:removed './180/11' 2023-10-30T12:35:35.163 INFO:tasks.workunit.client.0.smithi089.stdout:removed './180/16' 2023-10-30T12:35:35.164 INFO:tasks.workunit.client.0.smithi089.stdout:removed './180/32' 2023-10-30T12:35:35.164 INFO:tasks.workunit.client.0.smithi089.stdout:removed './180/30' 2023-10-30T12:35:35.164 INFO:tasks.workunit.client.0.smithi089.stdout:removed './180/14' 2023-10-30T12:35:35.164 INFO:tasks.workunit.client.0.smithi089.stdout:removed './180/18' 2023-10-30T12:35:35.164 INFO:tasks.workunit.client.0.smithi089.stdout:removed './180/13' 2023-10-30T12:35:35.165 INFO:tasks.workunit.client.0.smithi089.stdout:removed './180/33' 2023-10-30T12:35:35.165 INFO:tasks.workunit.client.0.smithi089.stdout:removed './180/17' 2023-10-30T12:35:35.165 INFO:tasks.workunit.client.0.smithi089.stdout:removed './180/35' 2023-10-30T12:35:35.165 INFO:tasks.workunit.client.0.smithi089.stdout:removed './180/37' 2023-10-30T12:35:35.165 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './180' 2023-10-30T12:35:35.165 INFO:tasks.workunit.client.0.smithi089.stdout:removed './181/19' 2023-10-30T12:35:35.166 INFO:tasks.workunit.client.0.smithi089.stdout:removed './181/15' 2023-10-30T12:35:35.166 INFO:tasks.workunit.client.0.smithi089.stdout:removed './181/31' 2023-10-30T12:35:35.166 INFO:tasks.workunit.client.0.smithi089.stdout:removed './181/36' 2023-10-30T12:35:35.166 INFO:tasks.workunit.client.0.smithi089.stdout:removed './181/20' 2023-10-30T12:35:35.166 INFO:tasks.workunit.client.0.smithi089.stdout:removed './181/12' 2023-10-30T12:35:35.167 INFO:tasks.workunit.client.0.smithi089.stdout:removed './181/39' 2023-10-30T12:35:35.167 INFO:tasks.workunit.client.0.smithi089.stdout:removed './181/10' 2023-10-30T12:35:35.167 INFO:tasks.workunit.client.0.smithi089.stdout:removed './181/34' 2023-10-30T12:35:35.167 INFO:tasks.workunit.client.0.smithi089.stdout:removed './181/38' 2023-10-30T12:35:35.167 INFO:tasks.workunit.client.0.smithi089.stdout:removed './181/11' 2023-10-30T12:35:35.168 INFO:tasks.workunit.client.0.smithi089.stdout:removed './181/16' 2023-10-30T12:35:35.168 INFO:tasks.workunit.client.0.smithi089.stdout:removed './181/32' 2023-10-30T12:35:35.168 INFO:tasks.workunit.client.0.smithi089.stdout:removed './181/30' 2023-10-30T12:35:35.168 INFO:tasks.workunit.client.0.smithi089.stdout:removed './181/14' 2023-10-30T12:35:35.168 INFO:tasks.workunit.client.0.smithi089.stdout:removed './181/18' 2023-10-30T12:35:35.169 INFO:tasks.workunit.client.0.smithi089.stdout:removed './181/13' 2023-10-30T12:35:35.169 INFO:tasks.workunit.client.0.smithi089.stdout:removed './181/33' 2023-10-30T12:35:35.169 INFO:tasks.workunit.client.0.smithi089.stdout:removed './181/17' 2023-10-30T12:35:35.169 INFO:tasks.workunit.client.0.smithi089.stdout:removed './181/35' 2023-10-30T12:35:35.169 INFO:tasks.workunit.client.0.smithi089.stdout:removed './181/37' 2023-10-30T12:35:35.169 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './181' 2023-10-30T12:35:35.170 INFO:tasks.workunit.client.0.smithi089.stdout:removed './182/19' 2023-10-30T12:35:35.170 INFO:tasks.workunit.client.0.smithi089.stdout:removed './182/15' 2023-10-30T12:35:35.170 INFO:tasks.workunit.client.0.smithi089.stdout:removed './182/31' 2023-10-30T12:35:35.170 INFO:tasks.workunit.client.0.smithi089.stdout:removed './182/36' 2023-10-30T12:35:35.170 INFO:tasks.workunit.client.0.smithi089.stdout:removed './182/20' 2023-10-30T12:35:35.171 INFO:tasks.workunit.client.0.smithi089.stdout:removed './182/12' 2023-10-30T12:35:35.171 INFO:tasks.workunit.client.0.smithi089.stdout:removed './182/39' 2023-10-30T12:35:35.171 INFO:tasks.workunit.client.0.smithi089.stdout:removed './182/10' 2023-10-30T12:35:35.171 INFO:tasks.workunit.client.0.smithi089.stdout:removed './182/34' 2023-10-30T12:35:35.171 INFO:tasks.workunit.client.0.smithi089.stdout:removed './182/38' 2023-10-30T12:35:35.171 INFO:tasks.workunit.client.0.smithi089.stdout:removed './182/11' 2023-10-30T12:35:35.172 INFO:tasks.workunit.client.0.smithi089.stdout:removed './182/16' 2023-10-30T12:35:35.172 INFO:tasks.workunit.client.0.smithi089.stdout:removed './182/32' 2023-10-30T12:35:35.172 INFO:tasks.workunit.client.0.smithi089.stdout:removed './182/30' 2023-10-30T12:35:35.172 INFO:tasks.workunit.client.0.smithi089.stdout:removed './182/14' 2023-10-30T12:35:35.172 INFO:tasks.workunit.client.0.smithi089.stdout:removed './182/18' 2023-10-30T12:35:35.173 INFO:tasks.workunit.client.0.smithi089.stdout:removed './182/13' 2023-10-30T12:35:35.173 INFO:tasks.workunit.client.0.smithi089.stdout:removed './182/33' 2023-10-30T12:35:35.173 INFO:tasks.workunit.client.0.smithi089.stdout:removed './182/17' 2023-10-30T12:35:35.173 INFO:tasks.workunit.client.0.smithi089.stdout:removed './182/35' 2023-10-30T12:35:35.173 INFO:tasks.workunit.client.0.smithi089.stdout:removed './182/37' 2023-10-30T12:35:35.174 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './182' 2023-10-30T12:35:35.174 INFO:tasks.workunit.client.0.smithi089.stdout:removed './183/19' 2023-10-30T12:35:35.174 INFO:tasks.workunit.client.0.smithi089.stdout:removed './183/15' 2023-10-30T12:35:35.174 INFO:tasks.workunit.client.0.smithi089.stdout:removed './183/31' 2023-10-30T12:35:35.174 INFO:tasks.workunit.client.0.smithi089.stdout:removed './183/36' 2023-10-30T12:35:35.175 INFO:tasks.workunit.client.0.smithi089.stdout:removed './183/20' 2023-10-30T12:35:35.175 INFO:tasks.workunit.client.0.smithi089.stdout:removed './183/12' 2023-10-30T12:35:35.175 INFO:tasks.workunit.client.0.smithi089.stdout:removed './183/39' 2023-10-30T12:35:35.175 INFO:tasks.workunit.client.0.smithi089.stdout:removed './183/10' 2023-10-30T12:35:35.175 INFO:tasks.workunit.client.0.smithi089.stdout:removed './183/34' 2023-10-30T12:35:35.175 INFO:tasks.workunit.client.0.smithi089.stdout:removed './183/38' 2023-10-30T12:35:35.176 INFO:tasks.workunit.client.0.smithi089.stdout:removed './183/11' 2023-10-30T12:35:35.176 INFO:tasks.workunit.client.0.smithi089.stdout:removed './183/16' 2023-10-30T12:35:35.176 INFO:tasks.workunit.client.0.smithi089.stdout:removed './183/32' 2023-10-30T12:35:35.176 INFO:tasks.workunit.client.0.smithi089.stdout:removed './183/30' 2023-10-30T12:35:35.176 INFO:tasks.workunit.client.0.smithi089.stdout:removed './183/14' 2023-10-30T12:35:35.177 INFO:tasks.workunit.client.0.smithi089.stdout:removed './183/18' 2023-10-30T12:35:35.177 INFO:tasks.workunit.client.0.smithi089.stdout:removed './183/13' 2023-10-30T12:35:35.177 INFO:tasks.workunit.client.0.smithi089.stdout:removed './183/33' 2023-10-30T12:35:35.177 INFO:tasks.workunit.client.0.smithi089.stdout:removed './183/17' 2023-10-30T12:35:35.177 INFO:tasks.workunit.client.0.smithi089.stdout:removed './183/35' 2023-10-30T12:35:35.178 INFO:tasks.workunit.client.0.smithi089.stdout:removed './183/37' 2023-10-30T12:35:35.178 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './183' 2023-10-30T12:35:35.178 INFO:tasks.workunit.client.0.smithi089.stdout:removed './184/19' 2023-10-30T12:35:35.178 INFO:tasks.workunit.client.0.smithi089.stdout:removed './184/15' 2023-10-30T12:35:35.178 INFO:tasks.workunit.client.0.smithi089.stdout:removed './184/31' 2023-10-30T12:35:35.179 INFO:tasks.workunit.client.0.smithi089.stdout:removed './184/36' 2023-10-30T12:35:35.179 INFO:tasks.workunit.client.0.smithi089.stdout:removed './184/20' 2023-10-30T12:35:35.179 INFO:tasks.workunit.client.0.smithi089.stdout:removed './184/12' 2023-10-30T12:35:35.179 INFO:tasks.workunit.client.0.smithi089.stdout:removed './184/39' 2023-10-30T12:35:35.179 INFO:tasks.workunit.client.0.smithi089.stdout:removed './184/10' 2023-10-30T12:35:35.180 INFO:tasks.workunit.client.0.smithi089.stdout:removed './184/34' 2023-10-30T12:35:35.180 INFO:tasks.workunit.client.0.smithi089.stdout:removed './184/38' 2023-10-30T12:35:35.180 INFO:tasks.workunit.client.0.smithi089.stdout:removed './184/11' 2023-10-30T12:35:35.180 INFO:tasks.workunit.client.0.smithi089.stdout:removed './184/16' 2023-10-30T12:35:35.180 INFO:tasks.workunit.client.0.smithi089.stdout:removed './184/32' 2023-10-30T12:35:35.181 INFO:tasks.workunit.client.0.smithi089.stdout:removed './184/30' 2023-10-30T12:35:35.181 INFO:tasks.workunit.client.0.smithi089.stdout:removed './184/14' 2023-10-30T12:35:35.181 INFO:tasks.workunit.client.0.smithi089.stdout:removed './184/18' 2023-10-30T12:35:35.181 INFO:tasks.workunit.client.0.smithi089.stdout:removed './184/13' 2023-10-30T12:35:35.182 INFO:tasks.workunit.client.0.smithi089.stdout:removed './184/33' 2023-10-30T12:35:35.182 INFO:tasks.workunit.client.0.smithi089.stdout:removed './184/17' 2023-10-30T12:35:35.182 INFO:tasks.workunit.client.0.smithi089.stdout:removed './184/35' 2023-10-30T12:35:35.182 INFO:tasks.workunit.client.0.smithi089.stdout:removed './184/37' 2023-10-30T12:35:35.182 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './184' 2023-10-30T12:35:35.183 INFO:tasks.workunit.client.0.smithi089.stdout:removed './185/19' 2023-10-30T12:35:35.183 INFO:tasks.workunit.client.0.smithi089.stdout:removed './185/15' 2023-10-30T12:35:35.183 INFO:tasks.workunit.client.0.smithi089.stdout:removed './185/31' 2023-10-30T12:35:35.183 INFO:tasks.workunit.client.0.smithi089.stdout:removed './185/36' 2023-10-30T12:35:35.183 INFO:tasks.workunit.client.0.smithi089.stdout:removed './185/20' 2023-10-30T12:35:35.183 INFO:tasks.workunit.client.0.smithi089.stdout:removed './185/12' 2023-10-30T12:35:35.184 INFO:tasks.workunit.client.0.smithi089.stdout:removed './185/39' 2023-10-30T12:35:35.184 INFO:tasks.workunit.client.0.smithi089.stdout:removed './185/10' 2023-10-30T12:35:35.184 INFO:tasks.workunit.client.0.smithi089.stdout:removed './185/34' 2023-10-30T12:35:35.184 INFO:tasks.workunit.client.0.smithi089.stdout:removed './185/38' 2023-10-30T12:35:35.185 INFO:tasks.workunit.client.0.smithi089.stdout:removed './185/11' 2023-10-30T12:35:35.185 INFO:tasks.workunit.client.0.smithi089.stdout:removed './185/16' 2023-10-30T12:35:35.185 INFO:tasks.workunit.client.0.smithi089.stdout:removed './185/32' 2023-10-30T12:35:35.185 INFO:tasks.workunit.client.0.smithi089.stdout:removed './185/30' 2023-10-30T12:35:35.186 INFO:tasks.workunit.client.0.smithi089.stdout:removed './185/14' 2023-10-30T12:35:35.186 INFO:tasks.workunit.client.0.smithi089.stdout:removed './185/18' 2023-10-30T12:35:35.186 INFO:tasks.workunit.client.0.smithi089.stdout:removed './185/13' 2023-10-30T12:35:35.186 INFO:tasks.workunit.client.0.smithi089.stdout:removed './185/33' 2023-10-30T12:35:35.186 INFO:tasks.workunit.client.0.smithi089.stdout:removed './185/17' 2023-10-30T12:35:35.187 INFO:tasks.workunit.client.0.smithi089.stdout:removed './185/35' 2023-10-30T12:35:35.187 INFO:tasks.workunit.client.0.smithi089.stdout:removed './185/37' 2023-10-30T12:35:35.187 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './185' 2023-10-30T12:35:35.187 INFO:tasks.workunit.client.0.smithi089.stdout:removed './186/19' 2023-10-30T12:35:35.187 INFO:tasks.workunit.client.0.smithi089.stdout:removed './186/15' 2023-10-30T12:35:35.188 INFO:tasks.workunit.client.0.smithi089.stdout:removed './186/31' 2023-10-30T12:35:35.188 INFO:tasks.workunit.client.0.smithi089.stdout:removed './186/36' 2023-10-30T12:35:35.188 INFO:tasks.workunit.client.0.smithi089.stdout:removed './186/20' 2023-10-30T12:35:35.188 INFO:tasks.workunit.client.0.smithi089.stdout:removed './186/12' 2023-10-30T12:35:35.188 INFO:tasks.workunit.client.0.smithi089.stdout:removed './186/39' 2023-10-30T12:35:35.189 INFO:tasks.workunit.client.0.smithi089.stdout:removed './186/10' 2023-10-30T12:35:35.189 INFO:tasks.workunit.client.0.smithi089.stdout:removed './186/34' 2023-10-30T12:35:35.189 INFO:tasks.workunit.client.0.smithi089.stdout:removed './186/38' 2023-10-30T12:35:35.189 INFO:tasks.workunit.client.0.smithi089.stdout:removed './186/11' 2023-10-30T12:35:35.189 INFO:tasks.workunit.client.0.smithi089.stdout:removed './186/16' 2023-10-30T12:35:35.190 INFO:tasks.workunit.client.0.smithi089.stdout:removed './186/32' 2023-10-30T12:35:35.190 INFO:tasks.workunit.client.0.smithi089.stdout:removed './186/30' 2023-10-30T12:35:35.190 INFO:tasks.workunit.client.0.smithi089.stdout:removed './186/14' 2023-10-30T12:35:35.190 INFO:tasks.workunit.client.0.smithi089.stdout:removed './186/18' 2023-10-30T12:35:39.862 INFO:tasks.workunit.client.0.smithi089.stdout:removed './186/13' 2023-10-30T12:35:39.863 INFO:tasks.workunit.client.0.smithi089.stdout:removed './186/33' 2023-10-30T12:35:39.863 INFO:tasks.workunit.client.0.smithi089.stdout:removed './186/17' 2023-10-30T12:35:39.863 INFO:tasks.workunit.client.0.smithi089.stdout:removed './186/35' 2023-10-30T12:35:39.863 INFO:tasks.workunit.client.0.smithi089.stdout:removed './186/37' 2023-10-30T12:35:39.864 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './186' 2023-10-30T12:35:39.864 INFO:tasks.workunit.client.0.smithi089.stdout:removed './187/19' 2023-10-30T12:35:39.864 INFO:tasks.workunit.client.0.smithi089.stdout:removed './187/15' 2023-10-30T12:35:39.864 INFO:tasks.workunit.client.0.smithi089.stdout:removed './187/31' 2023-10-30T12:35:39.864 INFO:tasks.workunit.client.0.smithi089.stdout:removed './187/36' 2023-10-30T12:35:39.865 INFO:tasks.workunit.client.0.smithi089.stdout:removed './187/20' 2023-10-30T12:35:39.865 INFO:tasks.workunit.client.0.smithi089.stdout:removed './187/12' 2023-10-30T12:35:39.865 INFO:tasks.workunit.client.0.smithi089.stdout:removed './187/39' 2023-10-30T12:35:39.865 INFO:tasks.workunit.client.0.smithi089.stdout:removed './187/10' 2023-10-30T12:35:39.865 INFO:tasks.workunit.client.0.smithi089.stdout:removed './187/34' 2023-10-30T12:35:39.865 INFO:tasks.workunit.client.0.smithi089.stdout:removed './187/38' 2023-10-30T12:35:39.866 INFO:tasks.workunit.client.0.smithi089.stdout:removed './187/11' 2023-10-30T12:35:39.866 INFO:tasks.workunit.client.0.smithi089.stdout:removed './187/16' 2023-10-30T12:35:39.866 INFO:tasks.workunit.client.0.smithi089.stdout:removed './187/32' 2023-10-30T12:35:39.866 INFO:tasks.workunit.client.0.smithi089.stdout:removed './187/30' 2023-10-30T12:35:39.866 INFO:tasks.workunit.client.0.smithi089.stdout:removed './187/14' 2023-10-30T12:35:39.867 INFO:tasks.workunit.client.0.smithi089.stdout:removed './187/18' 2023-10-30T12:35:39.867 INFO:tasks.workunit.client.0.smithi089.stdout:removed './187/13' 2023-10-30T12:35:39.867 INFO:tasks.workunit.client.0.smithi089.stdout:removed './187/33' 2023-10-30T12:35:39.867 INFO:tasks.workunit.client.0.smithi089.stdout:removed './187/17' 2023-10-30T12:35:39.867 INFO:tasks.workunit.client.0.smithi089.stdout:removed './187/35' 2023-10-30T12:35:39.868 INFO:tasks.workunit.client.0.smithi089.stdout:removed './187/37' 2023-10-30T12:35:39.868 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './187' 2023-10-30T12:35:39.868 INFO:tasks.workunit.client.0.smithi089.stdout:removed './188/19' 2023-10-30T12:35:39.868 INFO:tasks.workunit.client.0.smithi089.stdout:removed './188/15' 2023-10-30T12:35:39.868 INFO:tasks.workunit.client.0.smithi089.stdout:removed './188/31' 2023-10-30T12:35:39.868 INFO:tasks.workunit.client.0.smithi089.stdout:removed './188/36' 2023-10-30T12:35:39.869 INFO:tasks.workunit.client.0.smithi089.stdout:removed './188/20' 2023-10-30T12:35:39.869 INFO:tasks.workunit.client.0.smithi089.stdout:removed './188/12' 2023-10-30T12:35:39.869 INFO:tasks.workunit.client.0.smithi089.stdout:removed './188/39' 2023-10-30T12:35:39.870 INFO:tasks.workunit.client.0.smithi089.stdout:removed './188/10' 2023-10-30T12:35:39.870 INFO:tasks.workunit.client.0.smithi089.stdout:removed './188/34' 2023-10-30T12:35:39.871 INFO:tasks.workunit.client.0.smithi089.stdout:removed './188/38' 2023-10-30T12:35:39.871 INFO:tasks.workunit.client.0.smithi089.stdout:removed './188/11' 2023-10-30T12:35:39.872 INFO:tasks.workunit.client.0.smithi089.stdout:removed './188/16' 2023-10-30T12:35:39.872 INFO:tasks.workunit.client.0.smithi089.stdout:removed './188/32' 2023-10-30T12:35:39.872 INFO:tasks.workunit.client.0.smithi089.stdout:removed './188/30' 2023-10-30T12:35:39.872 INFO:tasks.workunit.client.0.smithi089.stdout:removed './188/14' 2023-10-30T12:35:39.872 INFO:tasks.workunit.client.0.smithi089.stdout:removed './188/18' 2023-10-30T12:35:39.873 INFO:tasks.workunit.client.0.smithi089.stdout:removed './188/13' 2023-10-30T12:35:39.873 INFO:tasks.workunit.client.0.smithi089.stdout:removed './188/33' 2023-10-30T12:35:39.873 INFO:tasks.workunit.client.0.smithi089.stdout:removed './188/17' 2023-10-30T12:35:39.873 INFO:tasks.workunit.client.0.smithi089.stdout:removed './188/35' 2023-10-30T12:35:39.873 INFO:tasks.workunit.client.0.smithi089.stdout:removed './188/37' 2023-10-30T12:35:39.874 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './188' 2023-10-30T12:35:39.874 INFO:tasks.workunit.client.0.smithi089.stdout:removed './189/19' 2023-10-30T12:35:39.874 INFO:tasks.workunit.client.0.smithi089.stdout:removed './189/15' 2023-10-30T12:35:39.874 INFO:tasks.workunit.client.0.smithi089.stdout:removed './189/31' 2023-10-30T12:35:39.875 INFO:tasks.workunit.client.0.smithi089.stdout:removed './189/36' 2023-10-30T12:35:39.875 INFO:tasks.workunit.client.0.smithi089.stdout:removed './189/20' 2023-10-30T12:35:39.876 INFO:tasks.workunit.client.0.smithi089.stdout:removed './189/12' 2023-10-30T12:35:39.876 INFO:tasks.workunit.client.0.smithi089.stdout:removed './189/39' 2023-10-30T12:35:39.876 INFO:tasks.workunit.client.0.smithi089.stdout:removed './189/10' 2023-10-30T12:35:39.876 INFO:tasks.workunit.client.0.smithi089.stdout:removed './189/34' 2023-10-30T12:35:39.876 INFO:tasks.workunit.client.0.smithi089.stdout:removed './189/38' 2023-10-30T12:35:39.876 INFO:tasks.workunit.client.0.smithi089.stdout:removed './189/11' 2023-10-30T12:35:39.877 INFO:tasks.workunit.client.0.smithi089.stdout:removed './189/16' 2023-10-30T12:35:39.877 INFO:tasks.workunit.client.0.smithi089.stdout:removed './189/32' 2023-10-30T12:35:39.877 INFO:tasks.workunit.client.0.smithi089.stdout:removed './189/30' 2023-10-30T12:35:39.877 INFO:tasks.workunit.client.0.smithi089.stdout:removed './189/14' 2023-10-30T12:35:39.877 INFO:tasks.workunit.client.0.smithi089.stdout:removed './189/18' 2023-10-30T12:35:39.878 INFO:tasks.workunit.client.0.smithi089.stdout:removed './189/13' 2023-10-30T12:35:39.878 INFO:tasks.workunit.client.0.smithi089.stdout:removed './189/33' 2023-10-30T12:35:39.878 INFO:tasks.workunit.client.0.smithi089.stdout:removed './189/17' 2023-10-30T12:35:39.878 INFO:tasks.workunit.client.0.smithi089.stdout:removed './189/35' 2023-10-30T12:35:39.878 INFO:tasks.workunit.client.0.smithi089.stdout:removed './189/37' 2023-10-30T12:35:39.879 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './189' 2023-10-30T12:35:39.879 INFO:tasks.workunit.client.0.smithi089.stdout:removed './190/19' 2023-10-30T12:35:39.879 INFO:tasks.workunit.client.0.smithi089.stdout:removed './190/15' 2023-10-30T12:35:39.879 INFO:tasks.workunit.client.0.smithi089.stdout:removed './190/31' 2023-10-30T12:35:39.879 INFO:tasks.workunit.client.0.smithi089.stdout:removed './190/36' 2023-10-30T12:35:39.879 INFO:tasks.workunit.client.0.smithi089.stdout:removed './190/20' 2023-10-30T12:35:39.880 INFO:tasks.workunit.client.0.smithi089.stdout:removed './190/12' 2023-10-30T12:35:39.880 INFO:tasks.workunit.client.0.smithi089.stdout:removed './190/39' 2023-10-30T12:35:39.880 INFO:tasks.workunit.client.0.smithi089.stdout:removed './190/10' 2023-10-30T12:35:39.880 INFO:tasks.workunit.client.0.smithi089.stdout:removed './190/34' 2023-10-30T12:35:39.881 INFO:tasks.workunit.client.0.smithi089.stdout:removed './190/38' 2023-10-30T12:35:39.881 INFO:tasks.workunit.client.0.smithi089.stdout:removed './190/11' 2023-10-30T12:35:39.881 INFO:tasks.workunit.client.0.smithi089.stdout:removed './190/16' 2023-10-30T12:35:39.882 INFO:tasks.workunit.client.0.smithi089.stdout:removed './190/32' 2023-10-30T12:35:39.882 INFO:tasks.workunit.client.0.smithi089.stdout:removed './190/30' 2023-10-30T12:35:39.882 INFO:tasks.workunit.client.0.smithi089.stdout:removed './190/14' 2023-10-30T12:35:39.883 INFO:tasks.workunit.client.0.smithi089.stdout:removed './190/18' 2023-10-30T12:35:39.883 INFO:tasks.workunit.client.0.smithi089.stdout:removed './190/13' 2023-10-30T12:35:39.883 INFO:tasks.workunit.client.0.smithi089.stdout:removed './190/33' 2023-10-30T12:35:39.884 INFO:tasks.workunit.client.0.smithi089.stdout:removed './190/17' 2023-10-30T12:35:39.884 INFO:tasks.workunit.client.0.smithi089.stdout:removed './190/35' 2023-10-30T12:35:39.884 INFO:tasks.workunit.client.0.smithi089.stdout:removed './190/37' 2023-10-30T12:35:39.885 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './190' 2023-10-30T12:35:39.885 INFO:tasks.workunit.client.0.smithi089.stdout:removed './191/19' 2023-10-30T12:35:39.885 INFO:tasks.workunit.client.0.smithi089.stdout:removed './191/15' 2023-10-30T12:35:39.886 INFO:tasks.workunit.client.0.smithi089.stdout:removed './191/31' 2023-10-30T12:35:39.886 INFO:tasks.workunit.client.0.smithi089.stdout:removed './191/36' 2023-10-30T12:35:39.887 INFO:tasks.workunit.client.0.smithi089.stdout:removed './191/20' 2023-10-30T12:35:39.887 INFO:tasks.workunit.client.0.smithi089.stdout:removed './191/12' 2023-10-30T12:35:39.887 INFO:tasks.workunit.client.0.smithi089.stdout:removed './191/39' 2023-10-30T12:35:39.888 INFO:tasks.workunit.client.0.smithi089.stdout:removed './191/10' 2023-10-30T12:35:39.888 INFO:tasks.workunit.client.0.smithi089.stdout:removed './191/34' 2023-10-30T12:35:39.888 INFO:tasks.workunit.client.0.smithi089.stdout:removed './191/38' 2023-10-30T12:35:39.889 INFO:tasks.workunit.client.0.smithi089.stdout:removed './191/11' 2023-10-30T12:35:39.889 INFO:tasks.workunit.client.0.smithi089.stdout:removed './191/16' 2023-10-30T12:35:39.889 INFO:tasks.workunit.client.0.smithi089.stdout:removed './191/32' 2023-10-30T12:35:39.890 INFO:tasks.workunit.client.0.smithi089.stdout:removed './191/30' 2023-10-30T12:35:39.890 INFO:tasks.workunit.client.0.smithi089.stdout:removed './191/14' 2023-10-30T12:35:39.890 INFO:tasks.workunit.client.0.smithi089.stdout:removed './191/18' 2023-10-30T12:35:39.891 INFO:tasks.workunit.client.0.smithi089.stdout:removed './191/13' 2023-10-30T12:35:39.891 INFO:tasks.workunit.client.0.smithi089.stdout:removed './191/33' 2023-10-30T12:35:39.892 INFO:tasks.workunit.client.0.smithi089.stdout:removed './191/17' 2023-10-30T12:35:39.892 INFO:tasks.workunit.client.0.smithi089.stdout:removed './191/35' 2023-10-30T12:35:39.892 INFO:tasks.workunit.client.0.smithi089.stdout:removed './191/37' 2023-10-30T12:35:39.893 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './191' 2023-10-30T12:35:39.893 INFO:tasks.workunit.client.0.smithi089.stdout:removed './192/19' 2023-10-30T12:35:39.893 INFO:tasks.workunit.client.0.smithi089.stdout:removed './192/15' 2023-10-30T12:35:39.894 INFO:tasks.workunit.client.0.smithi089.stdout:removed './192/31' 2023-10-30T12:35:39.894 INFO:tasks.workunit.client.0.smithi089.stdout:removed './192/36' 2023-10-30T12:35:39.895 INFO:tasks.workunit.client.0.smithi089.stdout:removed './192/20' 2023-10-30T12:35:39.895 INFO:tasks.workunit.client.0.smithi089.stdout:removed './192/12' 2023-10-30T12:35:39.895 INFO:tasks.workunit.client.0.smithi089.stdout:removed './192/39' 2023-10-30T12:35:39.896 INFO:tasks.workunit.client.0.smithi089.stdout:removed './192/10' 2023-10-30T12:35:39.896 INFO:tasks.workunit.client.0.smithi089.stdout:removed './192/34' 2023-10-30T12:35:39.896 INFO:tasks.workunit.client.0.smithi089.stdout:removed './192/38' 2023-10-30T12:35:39.897 INFO:tasks.workunit.client.0.smithi089.stdout:removed './192/11' 2023-10-30T12:35:39.897 INFO:tasks.workunit.client.0.smithi089.stdout:removed './192/16' 2023-10-30T12:35:39.897 INFO:tasks.workunit.client.0.smithi089.stdout:removed './192/32' 2023-10-30T12:35:39.898 INFO:tasks.workunit.client.0.smithi089.stdout:removed './192/30' 2023-10-30T12:35:39.898 INFO:tasks.workunit.client.0.smithi089.stdout:removed './192/14' 2023-10-30T12:35:39.899 INFO:tasks.workunit.client.0.smithi089.stdout:removed './192/18' 2023-10-30T12:35:39.899 INFO:tasks.workunit.client.0.smithi089.stdout:removed './192/13' 2023-10-30T12:35:39.899 INFO:tasks.workunit.client.0.smithi089.stdout:removed './192/33' 2023-10-30T12:35:39.900 INFO:tasks.workunit.client.0.smithi089.stdout:removed './192/17' 2023-10-30T12:35:39.900 INFO:tasks.workunit.client.0.smithi089.stdout:removed './192/35' 2023-10-30T12:35:39.900 INFO:tasks.workunit.client.0.smithi089.stdout:removed './192/37' 2023-10-30T12:35:39.901 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './192' 2023-10-30T12:35:39.901 INFO:tasks.workunit.client.0.smithi089.stdout:removed './193/19' 2023-10-30T12:35:39.902 INFO:tasks.workunit.client.0.smithi089.stdout:removed './193/15' 2023-10-30T12:35:39.902 INFO:tasks.workunit.client.0.smithi089.stdout:removed './193/31' 2023-10-30T12:35:39.902 INFO:tasks.workunit.client.0.smithi089.stdout:removed './193/36' 2023-10-30T12:35:39.903 INFO:tasks.workunit.client.0.smithi089.stdout:removed './193/20' 2023-10-30T12:35:39.903 INFO:tasks.workunit.client.0.smithi089.stdout:removed './193/12' 2023-10-30T12:35:39.903 INFO:tasks.workunit.client.0.smithi089.stdout:removed './193/39' 2023-10-30T12:35:39.904 INFO:tasks.workunit.client.0.smithi089.stdout:removed './193/10' 2023-10-30T12:35:39.904 INFO:tasks.workunit.client.0.smithi089.stdout:removed './193/34' 2023-10-30T12:35:39.904 INFO:tasks.workunit.client.0.smithi089.stdout:removed './193/38' 2023-10-30T12:35:39.905 INFO:tasks.workunit.client.0.smithi089.stdout:removed './193/11' 2023-10-30T12:35:39.905 INFO:tasks.workunit.client.0.smithi089.stdout:removed './193/16' 2023-10-30T12:35:39.905 INFO:tasks.workunit.client.0.smithi089.stdout:removed './193/32' 2023-10-30T12:35:39.906 INFO:tasks.workunit.client.0.smithi089.stdout:removed './193/30' 2023-10-30T12:35:39.906 INFO:tasks.workunit.client.0.smithi089.stdout:removed './193/14' 2023-10-30T12:35:39.906 INFO:tasks.workunit.client.0.smithi089.stdout:removed './193/18' 2023-10-30T12:35:39.907 INFO:tasks.workunit.client.0.smithi089.stdout:removed './193/13' 2023-10-30T12:35:39.907 INFO:tasks.workunit.client.0.smithi089.stdout:removed './193/33' 2023-10-30T12:35:39.907 INFO:tasks.workunit.client.0.smithi089.stdout:removed './193/17' 2023-10-30T12:35:39.908 INFO:tasks.workunit.client.0.smithi089.stdout:removed './193/35' 2023-10-30T12:35:39.908 INFO:tasks.workunit.client.0.smithi089.stdout:removed './193/37' 2023-10-30T12:35:39.908 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './193' 2023-10-30T12:35:39.908 INFO:tasks.workunit.client.0.smithi089.stdout:removed './194/19' 2023-10-30T12:35:39.909 INFO:tasks.workunit.client.0.smithi089.stdout:removed './194/15' 2023-10-30T12:35:39.909 INFO:tasks.workunit.client.0.smithi089.stdout:removed './194/31' 2023-10-30T12:35:39.909 INFO:tasks.workunit.client.0.smithi089.stdout:removed './194/36' 2023-10-30T12:35:39.909 INFO:tasks.workunit.client.0.smithi089.stdout:removed './194/20' 2023-10-30T12:35:39.910 INFO:tasks.workunit.client.0.smithi089.stdout:removed './194/12' 2023-10-30T12:35:39.910 INFO:tasks.workunit.client.0.smithi089.stdout:removed './194/39' 2023-10-30T12:35:39.910 INFO:tasks.workunit.client.0.smithi089.stdout:removed './194/10' 2023-10-30T12:35:39.911 INFO:tasks.workunit.client.0.smithi089.stdout:removed './194/34' 2023-10-30T12:35:39.911 INFO:tasks.workunit.client.0.smithi089.stdout:removed './194/38' 2023-10-30T12:35:39.911 INFO:tasks.workunit.client.0.smithi089.stdout:removed './194/11' 2023-10-30T12:35:39.911 INFO:tasks.workunit.client.0.smithi089.stdout:removed './194/16' 2023-10-30T12:35:39.911 INFO:tasks.workunit.client.0.smithi089.stdout:removed './194/32' 2023-10-30T12:35:39.912 INFO:tasks.workunit.client.0.smithi089.stdout:removed './194/30' 2023-10-30T12:35:39.912 INFO:tasks.workunit.client.0.smithi089.stdout:removed './194/14' 2023-10-30T12:35:39.912 INFO:tasks.workunit.client.0.smithi089.stdout:removed './194/18' 2023-10-30T12:35:39.912 INFO:tasks.workunit.client.0.smithi089.stdout:removed './194/13' 2023-10-30T12:35:39.912 INFO:tasks.workunit.client.0.smithi089.stdout:removed './194/33' 2023-10-30T12:35:39.912 INFO:tasks.workunit.client.0.smithi089.stdout:removed './194/17' 2023-10-30T12:35:39.913 INFO:tasks.workunit.client.0.smithi089.stdout:removed './194/35' 2023-10-30T12:35:39.913 INFO:tasks.workunit.client.0.smithi089.stdout:removed './194/37' 2023-10-30T12:35:39.913 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './194' 2023-10-30T12:35:39.913 INFO:tasks.workunit.client.0.smithi089.stdout:removed './195/19' 2023-10-30T12:35:39.913 INFO:tasks.workunit.client.0.smithi089.stdout:removed './195/15' 2023-10-30T12:35:39.914 INFO:tasks.workunit.client.0.smithi089.stdout:removed './195/31' 2023-10-30T12:35:39.914 INFO:tasks.workunit.client.0.smithi089.stdout:removed './195/36' 2023-10-30T12:35:39.914 INFO:tasks.workunit.client.0.smithi089.stdout:removed './195/20' 2023-10-30T12:35:39.914 INFO:tasks.workunit.client.0.smithi089.stdout:removed './195/12' 2023-10-30T12:35:39.914 INFO:tasks.workunit.client.0.smithi089.stdout:removed './195/39' 2023-10-30T12:35:39.915 INFO:tasks.workunit.client.0.smithi089.stdout:removed './195/10' 2023-10-30T12:35:39.915 INFO:tasks.workunit.client.0.smithi089.stdout:removed './195/34' 2023-10-30T12:35:39.915 INFO:tasks.workunit.client.0.smithi089.stdout:removed './195/38' 2023-10-30T12:35:39.915 INFO:tasks.workunit.client.0.smithi089.stdout:removed './195/11' 2023-10-30T12:35:39.915 INFO:tasks.workunit.client.0.smithi089.stdout:removed './195/16' 2023-10-30T12:35:39.916 INFO:tasks.workunit.client.0.smithi089.stdout:removed './195/32' 2023-10-30T12:35:39.916 INFO:tasks.workunit.client.0.smithi089.stdout:removed './195/30' 2023-10-30T12:35:39.916 INFO:tasks.workunit.client.0.smithi089.stdout:removed './195/14' 2023-10-30T12:35:39.916 INFO:tasks.workunit.client.0.smithi089.stdout:removed './195/18' 2023-10-30T12:35:39.916 INFO:tasks.workunit.client.0.smithi089.stdout:removed './195/13' 2023-10-30T12:35:39.917 INFO:tasks.workunit.client.0.smithi089.stdout:removed './195/33' 2023-10-30T12:35:39.917 INFO:tasks.workunit.client.0.smithi089.stdout:removed './195/17' 2023-10-30T12:35:39.917 INFO:tasks.workunit.client.0.smithi089.stdout:removed './195/35' 2023-10-30T12:35:39.917 INFO:tasks.workunit.client.0.smithi089.stdout:removed './195/37' 2023-10-30T12:35:39.917 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './195' 2023-10-30T12:35:39.917 INFO:tasks.workunit.client.0.smithi089.stdout:removed './196/19' 2023-10-30T12:35:39.918 INFO:tasks.workunit.client.0.smithi089.stdout:removed './196/15' 2023-10-30T12:35:39.918 INFO:tasks.workunit.client.0.smithi089.stdout:removed './196/31' 2023-10-30T12:35:39.918 INFO:tasks.workunit.client.0.smithi089.stdout:removed './196/36' 2023-10-30T12:35:39.918 INFO:tasks.workunit.client.0.smithi089.stdout:removed './196/20' 2023-10-30T12:35:39.918 INFO:tasks.workunit.client.0.smithi089.stdout:removed './196/12' 2023-10-30T12:35:39.919 INFO:tasks.workunit.client.0.smithi089.stdout:removed './196/39' 2023-10-30T12:35:39.919 INFO:tasks.workunit.client.0.smithi089.stdout:removed './196/10' 2023-10-30T12:35:45.656 INFO:tasks.workunit.client.0.smithi089.stdout:removed './196/34' 2023-10-30T12:35:45.656 INFO:tasks.workunit.client.0.smithi089.stdout:removed './196/38' 2023-10-30T12:35:45.656 INFO:tasks.workunit.client.0.smithi089.stdout:removed './196/11' 2023-10-30T12:35:45.657 INFO:tasks.workunit.client.0.smithi089.stdout:removed './196/16' 2023-10-30T12:35:45.657 INFO:tasks.workunit.client.0.smithi089.stdout:removed './196/32' 2023-10-30T12:35:45.657 INFO:tasks.workunit.client.0.smithi089.stdout:removed './196/30' 2023-10-30T12:35:45.657 INFO:tasks.workunit.client.0.smithi089.stdout:removed './196/14' 2023-10-30T12:35:45.657 INFO:tasks.workunit.client.0.smithi089.stdout:removed './196/18' 2023-10-30T12:35:45.658 INFO:tasks.workunit.client.0.smithi089.stdout:removed './196/13' 2023-10-30T12:35:45.658 INFO:tasks.workunit.client.0.smithi089.stdout:removed './196/33' 2023-10-30T12:35:45.658 INFO:tasks.workunit.client.0.smithi089.stdout:removed './196/17' 2023-10-30T12:35:45.658 INFO:tasks.workunit.client.0.smithi089.stdout:removed './196/35' 2023-10-30T12:35:45.658 INFO:tasks.workunit.client.0.smithi089.stdout:removed './196/37' 2023-10-30T12:35:45.658 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './196' 2023-10-30T12:35:45.659 INFO:tasks.workunit.client.0.smithi089.stdout:removed './197/19' 2023-10-30T12:35:45.659 INFO:tasks.workunit.client.0.smithi089.stdout:removed './197/15' 2023-10-30T12:35:45.659 INFO:tasks.workunit.client.0.smithi089.stdout:removed './197/31' 2023-10-30T12:35:45.659 INFO:tasks.workunit.client.0.smithi089.stdout:removed './197/36' 2023-10-30T12:35:45.659 INFO:tasks.workunit.client.0.smithi089.stdout:removed './197/20' 2023-10-30T12:35:45.660 INFO:tasks.workunit.client.0.smithi089.stdout:removed './197/12' 2023-10-30T12:35:45.660 INFO:tasks.workunit.client.0.smithi089.stdout:removed './197/39' 2023-10-30T12:35:45.660 INFO:tasks.workunit.client.0.smithi089.stdout:removed './197/10' 2023-10-30T12:35:45.660 INFO:tasks.workunit.client.0.smithi089.stdout:removed './197/34' 2023-10-30T12:35:45.660 INFO:tasks.workunit.client.0.smithi089.stdout:removed './197/38' 2023-10-30T12:35:45.661 INFO:tasks.workunit.client.0.smithi089.stdout:removed './197/11' 2023-10-30T12:35:45.661 INFO:tasks.workunit.client.0.smithi089.stdout:removed './197/16' 2023-10-30T12:35:45.661 INFO:tasks.workunit.client.0.smithi089.stdout:removed './197/32' 2023-10-30T12:35:45.661 INFO:tasks.workunit.client.0.smithi089.stdout:removed './197/30' 2023-10-30T12:35:45.661 INFO:tasks.workunit.client.0.smithi089.stdout:removed './197/14' 2023-10-30T12:35:45.662 INFO:tasks.workunit.client.0.smithi089.stdout:removed './197/18' 2023-10-30T12:35:45.662 INFO:tasks.workunit.client.0.smithi089.stdout:removed './197/13' 2023-10-30T12:35:45.662 INFO:tasks.workunit.client.0.smithi089.stdout:removed './197/33' 2023-10-30T12:35:45.662 INFO:tasks.workunit.client.0.smithi089.stdout:removed './197/17' 2023-10-30T12:35:45.662 INFO:tasks.workunit.client.0.smithi089.stdout:removed './197/35' 2023-10-30T12:35:45.663 INFO:tasks.workunit.client.0.smithi089.stdout:removed './197/37' 2023-10-30T12:35:45.663 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './197' 2023-10-30T12:35:45.663 INFO:tasks.workunit.client.0.smithi089.stdout:removed './198/19' 2023-10-30T12:35:45.663 INFO:tasks.workunit.client.0.smithi089.stdout:removed './198/15' 2023-10-30T12:35:45.663 INFO:tasks.workunit.client.0.smithi089.stdout:removed './198/31' 2023-10-30T12:35:45.663 INFO:tasks.workunit.client.0.smithi089.stdout:removed './198/36' 2023-10-30T12:35:45.664 INFO:tasks.workunit.client.0.smithi089.stdout:removed './198/20' 2023-10-30T12:35:45.664 INFO:tasks.workunit.client.0.smithi089.stdout:removed './198/12' 2023-10-30T12:35:45.664 INFO:tasks.workunit.client.0.smithi089.stdout:removed './198/39' 2023-10-30T12:35:45.664 INFO:tasks.workunit.client.0.smithi089.stdout:removed './198/10' 2023-10-30T12:35:45.664 INFO:tasks.workunit.client.0.smithi089.stdout:removed './198/34' 2023-10-30T12:35:45.665 INFO:tasks.workunit.client.0.smithi089.stdout:removed './198/38' 2023-10-30T12:35:45.665 INFO:tasks.workunit.client.0.smithi089.stdout:removed './198/11' 2023-10-30T12:35:45.665 INFO:tasks.workunit.client.0.smithi089.stdout:removed './198/16' 2023-10-30T12:35:45.665 INFO:tasks.workunit.client.0.smithi089.stdout:removed './198/32' 2023-10-30T12:35:45.665 INFO:tasks.workunit.client.0.smithi089.stdout:removed './198/30' 2023-10-30T12:35:45.666 INFO:tasks.workunit.client.0.smithi089.stdout:removed './198/14' 2023-10-30T12:35:45.666 INFO:tasks.workunit.client.0.smithi089.stdout:removed './198/18' 2023-10-30T12:35:45.666 INFO:tasks.workunit.client.0.smithi089.stdout:removed './198/13' 2023-10-30T12:35:45.666 INFO:tasks.workunit.client.0.smithi089.stdout:removed './198/33' 2023-10-30T12:35:45.666 INFO:tasks.workunit.client.0.smithi089.stdout:removed './198/17' 2023-10-30T12:35:45.667 INFO:tasks.workunit.client.0.smithi089.stdout:removed './198/35' 2023-10-30T12:35:45.669 INFO:tasks.workunit.client.0.smithi089.stdout:removed './198/37' 2023-10-30T12:35:45.669 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './198' 2023-10-30T12:35:45.669 INFO:tasks.workunit.client.0.smithi089.stdout:removed './199/19' 2023-10-30T12:35:45.670 INFO:tasks.workunit.client.0.smithi089.stdout:removed './199/15' 2023-10-30T12:35:45.670 INFO:tasks.workunit.client.0.smithi089.stdout:removed './199/31' 2023-10-30T12:35:45.670 INFO:tasks.workunit.client.0.smithi089.stdout:removed './199/36' 2023-10-30T12:35:45.670 INFO:tasks.workunit.client.0.smithi089.stdout:removed './199/20' 2023-10-30T12:35:45.670 INFO:tasks.workunit.client.0.smithi089.stdout:removed './199/12' 2023-10-30T12:35:45.671 INFO:tasks.workunit.client.0.smithi089.stdout:removed './199/39' 2023-10-30T12:35:45.671 INFO:tasks.workunit.client.0.smithi089.stdout:removed './199/10' 2023-10-30T12:35:45.671 INFO:tasks.workunit.client.0.smithi089.stdout:removed './199/34' 2023-10-30T12:35:45.671 INFO:tasks.workunit.client.0.smithi089.stdout:removed './199/38' 2023-10-30T12:35:45.671 INFO:tasks.workunit.client.0.smithi089.stdout:removed './199/11' 2023-10-30T12:35:45.672 INFO:tasks.workunit.client.0.smithi089.stdout:removed './199/16' 2023-10-30T12:35:45.672 INFO:tasks.workunit.client.0.smithi089.stdout:removed './199/32' 2023-10-30T12:35:45.672 INFO:tasks.workunit.client.0.smithi089.stdout:removed './199/30' 2023-10-30T12:35:45.672 INFO:tasks.workunit.client.0.smithi089.stdout:removed './199/14' 2023-10-30T12:35:45.672 INFO:tasks.workunit.client.0.smithi089.stdout:removed './199/18' 2023-10-30T12:35:45.673 INFO:tasks.workunit.client.0.smithi089.stdout:removed './199/13' 2023-10-30T12:35:45.673 INFO:tasks.workunit.client.0.smithi089.stdout:removed './199/33' 2023-10-30T12:35:45.673 INFO:tasks.workunit.client.0.smithi089.stdout:removed './199/17' 2023-10-30T12:35:45.673 INFO:tasks.workunit.client.0.smithi089.stdout:removed './199/35' 2023-10-30T12:35:45.673 INFO:tasks.workunit.client.0.smithi089.stdout:removed './199/37' 2023-10-30T12:35:45.674 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './199' 2023-10-30T12:35:45.674 INFO:tasks.workunit.client.0.smithi089.stdout:removed './200/21' 2023-10-30T12:35:45.674 INFO:tasks.workunit.client.0.smithi089.stdout:removed './200/27' 2023-10-30T12:35:45.674 INFO:tasks.workunit.client.0.smithi089.stdout:removed './200/31' 2023-10-30T12:35:45.674 INFO:tasks.workunit.client.0.smithi089.stdout:removed './200/36' 2023-10-30T12:35:45.674 INFO:tasks.workunit.client.0.smithi089.stdout:removed './200/20' 2023-10-30T12:35:45.675 INFO:tasks.workunit.client.0.smithi089.stdout:removed './200/39' 2023-10-30T12:35:45.675 INFO:tasks.workunit.client.0.smithi089.stdout:removed './200/34' 2023-10-30T12:35:45.675 INFO:tasks.workunit.client.0.smithi089.stdout:removed './200/38' 2023-10-30T12:35:45.675 INFO:tasks.workunit.client.0.smithi089.stdout:removed './200/28' 2023-10-30T12:35:45.675 INFO:tasks.workunit.client.0.smithi089.stdout:removed './200/24' 2023-10-30T12:35:45.676 INFO:tasks.workunit.client.0.smithi089.stdout:removed './200/29' 2023-10-30T12:35:45.676 INFO:tasks.workunit.client.0.smithi089.stdout:removed './200/32' 2023-10-30T12:35:45.676 INFO:tasks.workunit.client.0.smithi089.stdout:removed './200/22' 2023-10-30T12:35:45.676 INFO:tasks.workunit.client.0.smithi089.stdout:removed './200/25' 2023-10-30T12:35:45.676 INFO:tasks.workunit.client.0.smithi089.stdout:removed './200/30' 2023-10-30T12:35:45.677 INFO:tasks.workunit.client.0.smithi089.stdout:removed './200/33' 2023-10-30T12:35:45.677 INFO:tasks.workunit.client.0.smithi089.stdout:removed './200/23' 2023-10-30T12:35:45.677 INFO:tasks.workunit.client.0.smithi089.stdout:removed './200/35' 2023-10-30T12:35:45.677 INFO:tasks.workunit.client.0.smithi089.stdout:removed './200/26' 2023-10-30T12:35:45.677 INFO:tasks.workunit.client.0.smithi089.stdout:removed './200/37' 2023-10-30T12:35:45.678 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './200' 2023-10-30T12:35:45.678 INFO:tasks.workunit.client.0.smithi089.stdout:removed './201/21' 2023-10-30T12:35:45.678 INFO:tasks.workunit.client.0.smithi089.stdout:removed './201/27' 2023-10-30T12:35:45.678 INFO:tasks.workunit.client.0.smithi089.stdout:removed './201/31' 2023-10-30T12:35:45.678 INFO:tasks.workunit.client.0.smithi089.stdout:removed './201/36' 2023-10-30T12:35:45.678 INFO:tasks.workunit.client.0.smithi089.stdout:removed './201/20' 2023-10-30T12:35:45.679 INFO:tasks.workunit.client.0.smithi089.stdout:removed './201/39' 2023-10-30T12:35:45.679 INFO:tasks.workunit.client.0.smithi089.stdout:removed './201/34' 2023-10-30T12:35:45.679 INFO:tasks.workunit.client.0.smithi089.stdout:removed './201/38' 2023-10-30T12:35:45.679 INFO:tasks.workunit.client.0.smithi089.stdout:removed './201/28' 2023-10-30T12:35:45.679 INFO:tasks.workunit.client.0.smithi089.stdout:removed './201/24' 2023-10-30T12:35:45.680 INFO:tasks.workunit.client.0.smithi089.stdout:removed './201/29' 2023-10-30T12:35:45.680 INFO:tasks.workunit.client.0.smithi089.stdout:removed './201/32' 2023-10-30T12:35:45.680 INFO:tasks.workunit.client.0.smithi089.stdout:removed './201/22' 2023-10-30T12:35:45.680 INFO:tasks.workunit.client.0.smithi089.stdout:removed './201/25' 2023-10-30T12:35:45.680 INFO:tasks.workunit.client.0.smithi089.stdout:removed './201/30' 2023-10-30T12:35:45.681 INFO:tasks.workunit.client.0.smithi089.stdout:removed './201/33' 2023-10-30T12:35:45.681 INFO:tasks.workunit.client.0.smithi089.stdout:removed './201/23' 2023-10-30T12:35:45.681 INFO:tasks.workunit.client.0.smithi089.stdout:removed './201/35' 2023-10-30T12:35:45.681 INFO:tasks.workunit.client.0.smithi089.stdout:removed './201/26' 2023-10-30T12:35:45.681 INFO:tasks.workunit.client.0.smithi089.stdout:removed './201/37' 2023-10-30T12:35:45.682 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './201' 2023-10-30T12:35:45.682 INFO:tasks.workunit.client.0.smithi089.stdout:removed './202/21' 2023-10-30T12:35:45.682 INFO:tasks.workunit.client.0.smithi089.stdout:removed './202/27' 2023-10-30T12:35:45.682 INFO:tasks.workunit.client.0.smithi089.stdout:removed './202/31' 2023-10-30T12:35:45.682 INFO:tasks.workunit.client.0.smithi089.stdout:removed './202/36' 2023-10-30T12:35:45.682 INFO:tasks.workunit.client.0.smithi089.stdout:removed './202/20' 2023-10-30T12:35:45.683 INFO:tasks.workunit.client.0.smithi089.stdout:removed './202/39' 2023-10-30T12:35:45.683 INFO:tasks.workunit.client.0.smithi089.stdout:removed './202/34' 2023-10-30T12:35:45.683 INFO:tasks.workunit.client.0.smithi089.stdout:removed './202/38' 2023-10-30T12:35:45.683 INFO:tasks.workunit.client.0.smithi089.stdout:removed './202/28' 2023-10-30T12:35:45.683 INFO:tasks.workunit.client.0.smithi089.stdout:removed './202/24' 2023-10-30T12:35:45.684 INFO:tasks.workunit.client.0.smithi089.stdout:removed './202/29' 2023-10-30T12:35:45.684 INFO:tasks.workunit.client.0.smithi089.stdout:removed './202/32' 2023-10-30T12:35:45.684 INFO:tasks.workunit.client.0.smithi089.stdout:removed './202/22' 2023-10-30T12:35:45.684 INFO:tasks.workunit.client.0.smithi089.stdout:removed './202/25' 2023-10-30T12:35:45.684 INFO:tasks.workunit.client.0.smithi089.stdout:removed './202/30' 2023-10-30T12:35:45.685 INFO:tasks.workunit.client.0.smithi089.stdout:removed './202/33' 2023-10-30T12:35:45.685 INFO:tasks.workunit.client.0.smithi089.stdout:removed './202/23' 2023-10-30T12:35:45.685 INFO:tasks.workunit.client.0.smithi089.stdout:removed './202/35' 2023-10-30T12:35:45.685 INFO:tasks.workunit.client.0.smithi089.stdout:removed './202/26' 2023-10-30T12:35:45.685 INFO:tasks.workunit.client.0.smithi089.stdout:removed './202/37' 2023-10-30T12:35:45.685 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './202' 2023-10-30T12:35:45.686 INFO:tasks.workunit.client.0.smithi089.stdout:removed './203/21' 2023-10-30T12:35:45.686 INFO:tasks.workunit.client.0.smithi089.stdout:removed './203/27' 2023-10-30T12:35:45.686 INFO:tasks.workunit.client.0.smithi089.stdout:removed './203/31' 2023-10-30T12:35:45.686 INFO:tasks.workunit.client.0.smithi089.stdout:removed './203/36' 2023-10-30T12:35:45.686 INFO:tasks.workunit.client.0.smithi089.stdout:removed './203/20' 2023-10-30T12:35:45.687 INFO:tasks.workunit.client.0.smithi089.stdout:removed './203/39' 2023-10-30T12:35:45.687 INFO:tasks.workunit.client.0.smithi089.stdout:removed './203/34' 2023-10-30T12:35:45.687 INFO:tasks.workunit.client.0.smithi089.stdout:removed './203/38' 2023-10-30T12:35:45.687 INFO:tasks.workunit.client.0.smithi089.stdout:removed './203/28' 2023-10-30T12:35:45.687 INFO:tasks.workunit.client.0.smithi089.stdout:removed './203/24' 2023-10-30T12:35:45.688 INFO:tasks.workunit.client.0.smithi089.stdout:removed './203/29' 2023-10-30T12:35:45.688 INFO:tasks.workunit.client.0.smithi089.stdout:removed './203/32' 2023-10-30T12:35:45.688 INFO:tasks.workunit.client.0.smithi089.stdout:removed './203/22' 2023-10-30T12:35:45.688 INFO:tasks.workunit.client.0.smithi089.stdout:removed './203/25' 2023-10-30T12:35:45.688 INFO:tasks.workunit.client.0.smithi089.stdout:removed './203/30' 2023-10-30T12:35:45.688 INFO:tasks.workunit.client.0.smithi089.stdout:removed './203/33' 2023-10-30T12:35:45.689 INFO:tasks.workunit.client.0.smithi089.stdout:removed './203/23' 2023-10-30T12:35:45.689 INFO:tasks.workunit.client.0.smithi089.stdout:removed './203/35' 2023-10-30T12:35:45.689 INFO:tasks.workunit.client.0.smithi089.stdout:removed './203/26' 2023-10-30T12:35:45.689 INFO:tasks.workunit.client.0.smithi089.stdout:removed './203/37' 2023-10-30T12:35:45.689 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './203' 2023-10-30T12:35:45.690 INFO:tasks.workunit.client.0.smithi089.stdout:removed './204/21' 2023-10-30T12:35:45.690 INFO:tasks.workunit.client.0.smithi089.stdout:removed './204/27' 2023-10-30T12:35:45.690 INFO:tasks.workunit.client.0.smithi089.stdout:removed './204/31' 2023-10-30T12:35:45.690 INFO:tasks.workunit.client.0.smithi089.stdout:removed './204/36' 2023-10-30T12:35:45.690 INFO:tasks.workunit.client.0.smithi089.stdout:removed './204/20' 2023-10-30T12:35:45.691 INFO:tasks.workunit.client.0.smithi089.stdout:removed './204/39' 2023-10-30T12:35:45.691 INFO:tasks.workunit.client.0.smithi089.stdout:removed './204/34' 2023-10-30T12:35:45.691 INFO:tasks.workunit.client.0.smithi089.stdout:removed './204/38' 2023-10-30T12:35:45.691 INFO:tasks.workunit.client.0.smithi089.stdout:removed './204/28' 2023-10-30T12:35:45.691 INFO:tasks.workunit.client.0.smithi089.stdout:removed './204/24' 2023-10-30T12:35:45.692 INFO:tasks.workunit.client.0.smithi089.stdout:removed './204/29' 2023-10-30T12:35:45.692 INFO:tasks.workunit.client.0.smithi089.stdout:removed './204/32' 2023-10-30T12:35:45.692 INFO:tasks.workunit.client.0.smithi089.stdout:removed './204/22' 2023-10-30T12:35:45.692 INFO:tasks.workunit.client.0.smithi089.stdout:removed './204/25' 2023-10-30T12:35:45.692 INFO:tasks.workunit.client.0.smithi089.stdout:removed './204/30' 2023-10-30T12:35:45.692 INFO:tasks.workunit.client.0.smithi089.stdout:removed './204/33' 2023-10-30T12:35:45.693 INFO:tasks.workunit.client.0.smithi089.stdout:removed './204/23' 2023-10-30T12:35:45.693 INFO:tasks.workunit.client.0.smithi089.stdout:removed './204/35' 2023-10-30T12:35:45.693 INFO:tasks.workunit.client.0.smithi089.stdout:removed './204/26' 2023-10-30T12:35:45.693 INFO:tasks.workunit.client.0.smithi089.stdout:removed './204/37' 2023-10-30T12:35:45.693 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './204' 2023-10-30T12:35:45.694 INFO:tasks.workunit.client.0.smithi089.stdout:removed './205/21' 2023-10-30T12:35:45.694 INFO:tasks.workunit.client.0.smithi089.stdout:removed './205/27' 2023-10-30T12:35:45.694 INFO:tasks.workunit.client.0.smithi089.stdout:removed './205/31' 2023-10-30T12:35:45.694 INFO:tasks.workunit.client.0.smithi089.stdout:removed './205/36' 2023-10-30T12:35:45.694 INFO:tasks.workunit.client.0.smithi089.stdout:removed './205/20' 2023-10-30T12:35:45.694 INFO:tasks.workunit.client.0.smithi089.stdout:removed './205/39' 2023-10-30T12:35:45.695 INFO:tasks.workunit.client.0.smithi089.stdout:removed './205/34' 2023-10-30T12:35:45.695 INFO:tasks.workunit.client.0.smithi089.stdout:removed './205/38' 2023-10-30T12:35:45.695 INFO:tasks.workunit.client.0.smithi089.stdout:removed './205/28' 2023-10-30T12:35:45.695 INFO:tasks.workunit.client.0.smithi089.stdout:removed './205/24' 2023-10-30T12:35:45.695 INFO:tasks.workunit.client.0.smithi089.stdout:removed './205/29' 2023-10-30T12:35:45.696 INFO:tasks.workunit.client.0.smithi089.stdout:removed './205/32' 2023-10-30T12:35:45.696 INFO:tasks.workunit.client.0.smithi089.stdout:removed './205/22' 2023-10-30T12:35:45.696 INFO:tasks.workunit.client.0.smithi089.stdout:removed './205/25' 2023-10-30T12:35:45.696 INFO:tasks.workunit.client.0.smithi089.stdout:removed './205/30' 2023-10-30T12:35:45.696 INFO:tasks.workunit.client.0.smithi089.stdout:removed './205/33' 2023-10-30T12:35:45.697 INFO:tasks.workunit.client.0.smithi089.stdout:removed './205/23' 2023-10-30T12:35:45.697 INFO:tasks.workunit.client.0.smithi089.stdout:removed './205/35' 2023-10-30T12:35:45.697 INFO:tasks.workunit.client.0.smithi089.stdout:removed './205/26' 2023-10-30T12:35:45.697 INFO:tasks.workunit.client.0.smithi089.stdout:removed './205/37' 2023-10-30T12:35:45.697 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './205' 2023-10-30T12:35:45.697 INFO:tasks.workunit.client.0.smithi089.stdout:removed './206/21' 2023-10-30T12:35:45.698 INFO:tasks.workunit.client.0.smithi089.stdout:removed './206/27' 2023-10-30T12:35:45.698 INFO:tasks.workunit.client.0.smithi089.stdout:removed './206/31' 2023-10-30T12:35:45.698 INFO:tasks.workunit.client.0.smithi089.stdout:removed './206/36' 2023-10-30T12:35:45.698 INFO:tasks.workunit.client.0.smithi089.stdout:removed './206/20' 2023-10-30T12:35:45.698 INFO:tasks.workunit.client.0.smithi089.stdout:removed './206/39' 2023-10-30T12:35:45.833 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:35:45.838 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:35:49.561 INFO:tasks.workunit.client.0.smithi089.stdout:removed './206/34' 2023-10-30T12:35:49.561 INFO:tasks.workunit.client.0.smithi089.stdout:removed './206/38' 2023-10-30T12:35:49.561 INFO:tasks.workunit.client.0.smithi089.stdout:removed './206/28' 2023-10-30T12:35:49.562 INFO:tasks.workunit.client.0.smithi089.stdout:removed './206/24' 2023-10-30T12:35:49.562 INFO:tasks.workunit.client.0.smithi089.stdout:removed './206/29' 2023-10-30T12:35:49.562 INFO:tasks.workunit.client.0.smithi089.stdout:removed './206/32' 2023-10-30T12:35:49.562 INFO:tasks.workunit.client.0.smithi089.stdout:removed './206/22' 2023-10-30T12:35:49.563 INFO:tasks.workunit.client.0.smithi089.stdout:removed './206/25' 2023-10-30T12:35:49.563 INFO:tasks.workunit.client.0.smithi089.stdout:removed './206/30' 2023-10-30T12:35:49.563 INFO:tasks.workunit.client.0.smithi089.stdout:removed './206/33' 2023-10-30T12:35:49.563 INFO:tasks.workunit.client.0.smithi089.stdout:removed './206/23' 2023-10-30T12:35:49.564 INFO:tasks.workunit.client.0.smithi089.stdout:removed './206/35' 2023-10-30T12:35:49.564 INFO:tasks.workunit.client.0.smithi089.stdout:removed './206/26' 2023-10-30T12:35:49.564 INFO:tasks.workunit.client.0.smithi089.stdout:removed './206/37' 2023-10-30T12:35:49.564 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './206' 2023-10-30T12:35:49.564 INFO:tasks.workunit.client.0.smithi089.stdout:removed './207/21' 2023-10-30T12:35:49.565 INFO:tasks.workunit.client.0.smithi089.stdout:removed './207/27' 2023-10-30T12:35:49.565 INFO:tasks.workunit.client.0.smithi089.stdout:removed './207/31' 2023-10-30T12:35:49.565 INFO:tasks.workunit.client.0.smithi089.stdout:removed './207/36' 2023-10-30T12:35:49.565 INFO:tasks.workunit.client.0.smithi089.stdout:removed './207/20' 2023-10-30T12:35:49.565 INFO:tasks.workunit.client.0.smithi089.stdout:removed './207/39' 2023-10-30T12:35:49.566 INFO:tasks.workunit.client.0.smithi089.stdout:removed './207/34' 2023-10-30T12:35:49.566 INFO:tasks.workunit.client.0.smithi089.stdout:removed './207/38' 2023-10-30T12:35:49.566 INFO:tasks.workunit.client.0.smithi089.stdout:removed './207/28' 2023-10-30T12:35:49.566 INFO:tasks.workunit.client.0.smithi089.stdout:removed './207/24' 2023-10-30T12:35:49.566 INFO:tasks.workunit.client.0.smithi089.stdout:removed './207/29' 2023-10-30T12:35:49.567 INFO:tasks.workunit.client.0.smithi089.stdout:removed './207/32' 2023-10-30T12:35:49.567 INFO:tasks.workunit.client.0.smithi089.stdout:removed './207/22' 2023-10-30T12:35:49.567 INFO:tasks.workunit.client.0.smithi089.stdout:removed './207/25' 2023-10-30T12:35:49.567 INFO:tasks.workunit.client.0.smithi089.stdout:removed './207/30' 2023-10-30T12:35:49.567 INFO:tasks.workunit.client.0.smithi089.stdout:removed './207/33' 2023-10-30T12:35:49.568 INFO:tasks.workunit.client.0.smithi089.stdout:removed './207/23' 2023-10-30T12:35:49.568 INFO:tasks.workunit.client.0.smithi089.stdout:removed './207/35' 2023-10-30T12:35:49.568 INFO:tasks.workunit.client.0.smithi089.stdout:removed './207/26' 2023-10-30T12:35:49.568 INFO:tasks.workunit.client.0.smithi089.stdout:removed './207/37' 2023-10-30T12:35:49.568 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './207' 2023-10-30T12:35:49.569 INFO:tasks.workunit.client.0.smithi089.stdout:removed './208/21' 2023-10-30T12:35:49.569 INFO:tasks.workunit.client.0.smithi089.stdout:removed './208/27' 2023-10-30T12:35:49.569 INFO:tasks.workunit.client.0.smithi089.stdout:removed './208/31' 2023-10-30T12:35:49.569 INFO:tasks.workunit.client.0.smithi089.stdout:removed './208/36' 2023-10-30T12:35:49.569 INFO:tasks.workunit.client.0.smithi089.stdout:removed './208/20' 2023-10-30T12:35:49.570 INFO:tasks.workunit.client.0.smithi089.stdout:removed './208/39' 2023-10-30T12:35:49.570 INFO:tasks.workunit.client.0.smithi089.stdout:removed './208/34' 2023-10-30T12:35:49.570 INFO:tasks.workunit.client.0.smithi089.stdout:removed './208/38' 2023-10-30T12:35:49.570 INFO:tasks.workunit.client.0.smithi089.stdout:removed './208/28' 2023-10-30T12:35:49.570 INFO:tasks.workunit.client.0.smithi089.stdout:removed './208/24' 2023-10-30T12:35:49.571 INFO:tasks.workunit.client.0.smithi089.stdout:removed './208/29' 2023-10-30T12:35:49.571 INFO:tasks.workunit.client.0.smithi089.stdout:removed './208/32' 2023-10-30T12:35:49.571 INFO:tasks.workunit.client.0.smithi089.stdout:removed './208/22' 2023-10-30T12:35:49.571 INFO:tasks.workunit.client.0.smithi089.stdout:removed './208/25' 2023-10-30T12:35:49.571 INFO:tasks.workunit.client.0.smithi089.stdout:removed './208/30' 2023-10-30T12:35:49.572 INFO:tasks.workunit.client.0.smithi089.stdout:removed './208/33' 2023-10-30T12:35:49.572 INFO:tasks.workunit.client.0.smithi089.stdout:removed './208/23' 2023-10-30T12:35:49.572 INFO:tasks.workunit.client.0.smithi089.stdout:removed './208/35' 2023-10-30T12:35:49.572 INFO:tasks.workunit.client.0.smithi089.stdout:removed './208/26' 2023-10-30T12:35:49.572 INFO:tasks.workunit.client.0.smithi089.stdout:removed './208/37' 2023-10-30T12:35:49.573 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './208' 2023-10-30T12:35:49.573 INFO:tasks.workunit.client.0.smithi089.stdout:removed './209/21' 2023-10-30T12:35:49.573 INFO:tasks.workunit.client.0.smithi089.stdout:removed './209/27' 2023-10-30T12:35:49.573 INFO:tasks.workunit.client.0.smithi089.stdout:removed './209/31' 2023-10-30T12:35:49.573 INFO:tasks.workunit.client.0.smithi089.stdout:removed './209/36' 2023-10-30T12:35:49.574 INFO:tasks.workunit.client.0.smithi089.stdout:removed './209/20' 2023-10-30T12:35:49.574 INFO:tasks.workunit.client.0.smithi089.stdout:removed './209/39' 2023-10-30T12:35:49.574 INFO:tasks.workunit.client.0.smithi089.stdout:removed './209/34' 2023-10-30T12:35:49.574 INFO:tasks.workunit.client.0.smithi089.stdout:removed './209/38' 2023-10-30T12:35:49.574 INFO:tasks.workunit.client.0.smithi089.stdout:removed './209/28' 2023-10-30T12:35:49.575 INFO:tasks.workunit.client.0.smithi089.stdout:removed './209/24' 2023-10-30T12:35:49.575 INFO:tasks.workunit.client.0.smithi089.stdout:removed './209/29' 2023-10-30T12:35:49.575 INFO:tasks.workunit.client.0.smithi089.stdout:removed './209/32' 2023-10-30T12:35:49.575 INFO:tasks.workunit.client.0.smithi089.stdout:removed './209/22' 2023-10-30T12:35:49.575 INFO:tasks.workunit.client.0.smithi089.stdout:removed './209/25' 2023-10-30T12:35:49.575 INFO:tasks.workunit.client.0.smithi089.stdout:removed './209/30' 2023-10-30T12:35:49.576 INFO:tasks.workunit.client.0.smithi089.stdout:removed './209/33' 2023-10-30T12:35:49.576 INFO:tasks.workunit.client.0.smithi089.stdout:removed './209/23' 2023-10-30T12:35:49.576 INFO:tasks.workunit.client.0.smithi089.stdout:removed './209/35' 2023-10-30T12:35:49.576 INFO:tasks.workunit.client.0.smithi089.stdout:removed './209/26' 2023-10-30T12:35:49.576 INFO:tasks.workunit.client.0.smithi089.stdout:removed './209/37' 2023-10-30T12:35:49.577 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './209' 2023-10-30T12:35:49.577 INFO:tasks.workunit.client.0.smithi089.stdout:removed './210/21' 2023-10-30T12:35:49.577 INFO:tasks.workunit.client.0.smithi089.stdout:removed './210/27' 2023-10-30T12:35:49.577 INFO:tasks.workunit.client.0.smithi089.stdout:removed './210/31' 2023-10-30T12:35:49.577 INFO:tasks.workunit.client.0.smithi089.stdout:removed './210/36' 2023-10-30T12:35:49.578 INFO:tasks.workunit.client.0.smithi089.stdout:removed './210/20' 2023-10-30T12:35:49.578 INFO:tasks.workunit.client.0.smithi089.stdout:removed './210/39' 2023-10-30T12:35:49.578 INFO:tasks.workunit.client.0.smithi089.stdout:removed './210/34' 2023-10-30T12:35:49.578 INFO:tasks.workunit.client.0.smithi089.stdout:removed './210/38' 2023-10-30T12:35:49.578 INFO:tasks.workunit.client.0.smithi089.stdout:removed './210/28' 2023-10-30T12:35:49.579 INFO:tasks.workunit.client.0.smithi089.stdout:removed './210/24' 2023-10-30T12:35:49.579 INFO:tasks.workunit.client.0.smithi089.stdout:removed './210/29' 2023-10-30T12:35:49.579 INFO:tasks.workunit.client.0.smithi089.stdout:removed './210/32' 2023-10-30T12:35:49.579 INFO:tasks.workunit.client.0.smithi089.stdout:removed './210/22' 2023-10-30T12:35:49.579 INFO:tasks.workunit.client.0.smithi089.stdout:removed './210/25' 2023-10-30T12:35:49.580 INFO:tasks.workunit.client.0.smithi089.stdout:removed './210/30' 2023-10-30T12:35:49.580 INFO:tasks.workunit.client.0.smithi089.stdout:removed './210/33' 2023-10-30T12:35:49.580 INFO:tasks.workunit.client.0.smithi089.stdout:removed './210/23' 2023-10-30T12:35:49.580 INFO:tasks.workunit.client.0.smithi089.stdout:removed './210/35' 2023-10-30T12:35:49.580 INFO:tasks.workunit.client.0.smithi089.stdout:removed './210/26' 2023-10-30T12:35:49.581 INFO:tasks.workunit.client.0.smithi089.stdout:removed './210/37' 2023-10-30T12:35:49.581 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './210' 2023-10-30T12:35:49.581 INFO:tasks.workunit.client.0.smithi089.stdout:removed './211/21' 2023-10-30T12:35:49.581 INFO:tasks.workunit.client.0.smithi089.stdout:removed './211/27' 2023-10-30T12:35:49.581 INFO:tasks.workunit.client.0.smithi089.stdout:removed './211/31' 2023-10-30T12:35:49.582 INFO:tasks.workunit.client.0.smithi089.stdout:removed './211/36' 2023-10-30T12:35:49.582 INFO:tasks.workunit.client.0.smithi089.stdout:removed './211/20' 2023-10-30T12:35:49.582 INFO:tasks.workunit.client.0.smithi089.stdout:removed './211/39' 2023-10-30T12:35:49.582 INFO:tasks.workunit.client.0.smithi089.stdout:removed './211/34' 2023-10-30T12:35:49.582 INFO:tasks.workunit.client.0.smithi089.stdout:removed './211/38' 2023-10-30T12:35:49.583 INFO:tasks.workunit.client.0.smithi089.stdout:removed './211/28' 2023-10-30T12:35:49.583 INFO:tasks.workunit.client.0.smithi089.stdout:removed './211/24' 2023-10-30T12:35:49.583 INFO:tasks.workunit.client.0.smithi089.stdout:removed './211/29' 2023-10-30T12:35:49.583 INFO:tasks.workunit.client.0.smithi089.stdout:removed './211/32' 2023-10-30T12:35:49.583 INFO:tasks.workunit.client.0.smithi089.stdout:removed './211/22' 2023-10-30T12:35:49.584 INFO:tasks.workunit.client.0.smithi089.stdout:removed './211/25' 2023-10-30T12:35:49.584 INFO:tasks.workunit.client.0.smithi089.stdout:removed './211/30' 2023-10-30T12:35:49.584 INFO:tasks.workunit.client.0.smithi089.stdout:removed './211/33' 2023-10-30T12:35:49.584 INFO:tasks.workunit.client.0.smithi089.stdout:removed './211/23' 2023-10-30T12:35:49.584 INFO:tasks.workunit.client.0.smithi089.stdout:removed './211/35' 2023-10-30T12:35:49.584 INFO:tasks.workunit.client.0.smithi089.stdout:removed './211/26' 2023-10-30T12:35:49.585 INFO:tasks.workunit.client.0.smithi089.stdout:removed './211/37' 2023-10-30T12:35:49.585 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './211' 2023-10-30T12:35:49.585 INFO:tasks.workunit.client.0.smithi089.stdout:removed './212/21' 2023-10-30T12:35:49.585 INFO:tasks.workunit.client.0.smithi089.stdout:removed './212/27' 2023-10-30T12:35:49.585 INFO:tasks.workunit.client.0.smithi089.stdout:removed './212/31' 2023-10-30T12:35:49.586 INFO:tasks.workunit.client.0.smithi089.stdout:removed './212/36' 2023-10-30T12:35:49.586 INFO:tasks.workunit.client.0.smithi089.stdout:removed './212/20' 2023-10-30T12:35:49.586 INFO:tasks.workunit.client.0.smithi089.stdout:removed './212/39' 2023-10-30T12:35:49.586 INFO:tasks.workunit.client.0.smithi089.stdout:removed './212/34' 2023-10-30T12:35:49.586 INFO:tasks.workunit.client.0.smithi089.stdout:removed './212/38' 2023-10-30T12:35:49.587 INFO:tasks.workunit.client.0.smithi089.stdout:removed './212/28' 2023-10-30T12:35:49.587 INFO:tasks.workunit.client.0.smithi089.stdout:removed './212/24' 2023-10-30T12:35:49.587 INFO:tasks.workunit.client.0.smithi089.stdout:removed './212/29' 2023-10-30T12:35:49.587 INFO:tasks.workunit.client.0.smithi089.stdout:removed './212/32' 2023-10-30T12:35:49.587 INFO:tasks.workunit.client.0.smithi089.stdout:removed './212/22' 2023-10-30T12:35:49.588 INFO:tasks.workunit.client.0.smithi089.stdout:removed './212/25' 2023-10-30T12:35:49.588 INFO:tasks.workunit.client.0.smithi089.stdout:removed './212/30' 2023-10-30T12:35:49.588 INFO:tasks.workunit.client.0.smithi089.stdout:removed './212/33' 2023-10-30T12:35:49.588 INFO:tasks.workunit.client.0.smithi089.stdout:removed './212/23' 2023-10-30T12:35:49.588 INFO:tasks.workunit.client.0.smithi089.stdout:removed './212/35' 2023-10-30T12:35:49.589 INFO:tasks.workunit.client.0.smithi089.stdout:removed './212/26' 2023-10-30T12:35:49.589 INFO:tasks.workunit.client.0.smithi089.stdout:removed './212/37' 2023-10-30T12:35:49.589 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './212' 2023-10-30T12:35:49.589 INFO:tasks.workunit.client.0.smithi089.stdout:removed './213/21' 2023-10-30T12:35:49.589 INFO:tasks.workunit.client.0.smithi089.stdout:removed './213/27' 2023-10-30T12:35:49.589 INFO:tasks.workunit.client.0.smithi089.stdout:removed './213/31' 2023-10-30T12:35:49.590 INFO:tasks.workunit.client.0.smithi089.stdout:removed './213/36' 2023-10-30T12:35:49.590 INFO:tasks.workunit.client.0.smithi089.stdout:removed './213/20' 2023-10-30T12:35:49.590 INFO:tasks.workunit.client.0.smithi089.stdout:removed './213/39' 2023-10-30T12:35:49.590 INFO:tasks.workunit.client.0.smithi089.stdout:removed './213/34' 2023-10-30T12:35:49.590 INFO:tasks.workunit.client.0.smithi089.stdout:removed './213/38' 2023-10-30T12:35:49.591 INFO:tasks.workunit.client.0.smithi089.stdout:removed './213/28' 2023-10-30T12:35:49.591 INFO:tasks.workunit.client.0.smithi089.stdout:removed './213/24' 2023-10-30T12:35:49.591 INFO:tasks.workunit.client.0.smithi089.stdout:removed './213/29' 2023-10-30T12:35:49.591 INFO:tasks.workunit.client.0.smithi089.stdout:removed './213/32' 2023-10-30T12:35:49.591 INFO:tasks.workunit.client.0.smithi089.stdout:removed './213/22' 2023-10-30T12:35:49.592 INFO:tasks.workunit.client.0.smithi089.stdout:removed './213/25' 2023-10-30T12:35:49.592 INFO:tasks.workunit.client.0.smithi089.stdout:removed './213/30' 2023-10-30T12:35:49.592 INFO:tasks.workunit.client.0.smithi089.stdout:removed './213/33' 2023-10-30T12:35:49.592 INFO:tasks.workunit.client.0.smithi089.stdout:removed './213/23' 2023-10-30T12:35:49.592 INFO:tasks.workunit.client.0.smithi089.stdout:removed './213/35' 2023-10-30T12:35:49.593 INFO:tasks.workunit.client.0.smithi089.stdout:removed './213/26' 2023-10-30T12:35:49.593 INFO:tasks.workunit.client.0.smithi089.stdout:removed './213/37' 2023-10-30T12:35:49.593 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './213' 2023-10-30T12:35:49.593 INFO:tasks.workunit.client.0.smithi089.stdout:removed './214/21' 2023-10-30T12:35:49.593 INFO:tasks.workunit.client.0.smithi089.stdout:removed './214/27' 2023-10-30T12:35:49.593 INFO:tasks.workunit.client.0.smithi089.stdout:removed './214/31' 2023-10-30T12:35:49.594 INFO:tasks.workunit.client.0.smithi089.stdout:removed './214/36' 2023-10-30T12:35:49.594 INFO:tasks.workunit.client.0.smithi089.stdout:removed './214/20' 2023-10-30T12:35:49.594 INFO:tasks.workunit.client.0.smithi089.stdout:removed './214/39' 2023-10-30T12:35:49.594 INFO:tasks.workunit.client.0.smithi089.stdout:removed './214/34' 2023-10-30T12:35:49.594 INFO:tasks.workunit.client.0.smithi089.stdout:removed './214/38' 2023-10-30T12:35:49.595 INFO:tasks.workunit.client.0.smithi089.stdout:removed './214/28' 2023-10-30T12:35:49.595 INFO:tasks.workunit.client.0.smithi089.stdout:removed './214/24' 2023-10-30T12:35:49.595 INFO:tasks.workunit.client.0.smithi089.stdout:removed './214/29' 2023-10-30T12:35:49.595 INFO:tasks.workunit.client.0.smithi089.stdout:removed './214/32' 2023-10-30T12:35:49.595 INFO:tasks.workunit.client.0.smithi089.stdout:removed './214/22' 2023-10-30T12:35:49.596 INFO:tasks.workunit.client.0.smithi089.stdout:removed './214/25' 2023-10-30T12:35:49.596 INFO:tasks.workunit.client.0.smithi089.stdout:removed './214/30' 2023-10-30T12:35:49.596 INFO:tasks.workunit.client.0.smithi089.stdout:removed './214/33' 2023-10-30T12:35:49.596 INFO:tasks.workunit.client.0.smithi089.stdout:removed './214/23' 2023-10-30T12:35:49.596 INFO:tasks.workunit.client.0.smithi089.stdout:removed './214/35' 2023-10-30T12:35:49.596 INFO:tasks.workunit.client.0.smithi089.stdout:removed './214/26' 2023-10-30T12:35:49.597 INFO:tasks.workunit.client.0.smithi089.stdout:removed './214/37' 2023-10-30T12:35:49.597 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './214' 2023-10-30T12:35:49.597 INFO:tasks.workunit.client.0.smithi089.stdout:removed './215/21' 2023-10-30T12:35:49.597 INFO:tasks.workunit.client.0.smithi089.stdout:removed './215/27' 2023-10-30T12:35:49.597 INFO:tasks.workunit.client.0.smithi089.stdout:removed './215/31' 2023-10-30T12:35:49.598 INFO:tasks.workunit.client.0.smithi089.stdout:removed './215/36' 2023-10-30T12:35:49.598 INFO:tasks.workunit.client.0.smithi089.stdout:removed './215/20' 2023-10-30T12:35:49.598 INFO:tasks.workunit.client.0.smithi089.stdout:removed './215/39' 2023-10-30T12:35:49.598 INFO:tasks.workunit.client.0.smithi089.stdout:removed './215/34' 2023-10-30T12:35:49.598 INFO:tasks.workunit.client.0.smithi089.stdout:removed './215/38' 2023-10-30T12:35:49.599 INFO:tasks.workunit.client.0.smithi089.stdout:removed './215/28' 2023-10-30T12:35:49.599 INFO:tasks.workunit.client.0.smithi089.stdout:removed './215/24' 2023-10-30T12:35:49.599 INFO:tasks.workunit.client.0.smithi089.stdout:removed './215/29' 2023-10-30T12:35:49.599 INFO:tasks.workunit.client.0.smithi089.stdout:removed './215/32' 2023-10-30T12:35:49.599 INFO:tasks.workunit.client.0.smithi089.stdout:removed './215/22' 2023-10-30T12:35:49.600 INFO:tasks.workunit.client.0.smithi089.stdout:removed './215/25' 2023-10-30T12:35:49.600 INFO:tasks.workunit.client.0.smithi089.stdout:removed './215/30' 2023-10-30T12:35:49.600 INFO:tasks.workunit.client.0.smithi089.stdout:removed './215/33' 2023-10-30T12:35:49.600 INFO:tasks.workunit.client.0.smithi089.stdout:removed './215/23' 2023-10-30T12:35:49.600 INFO:tasks.workunit.client.0.smithi089.stdout:removed './215/35' 2023-10-30T12:35:49.601 INFO:tasks.workunit.client.0.smithi089.stdout:removed './215/26' 2023-10-30T12:35:49.601 INFO:tasks.workunit.client.0.smithi089.stdout:removed './215/37' 2023-10-30T12:35:49.601 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './215' 2023-10-30T12:35:49.601 INFO:tasks.workunit.client.0.smithi089.stdout:removed './216/21' 2023-10-30T12:35:49.601 INFO:tasks.workunit.client.0.smithi089.stdout:removed './216/27' 2023-10-30T12:35:49.602 INFO:tasks.workunit.client.0.smithi089.stdout:removed './216/31' 2023-10-30T12:35:49.602 INFO:tasks.workunit.client.0.smithi089.stdout:removed './216/36' 2023-10-30T12:35:49.602 INFO:tasks.workunit.client.0.smithi089.stdout:removed './216/20' 2023-10-30T12:35:49.602 INFO:tasks.workunit.client.0.smithi089.stdout:removed './216/39' 2023-10-30T12:35:49.602 INFO:tasks.workunit.client.0.smithi089.stdout:removed './216/34' 2023-10-30T12:35:49.602 INFO:tasks.workunit.client.0.smithi089.stdout:removed './216/38' 2023-10-30T12:35:52.502 INFO:tasks.workunit.client.0.smithi089.stdout:removed './216/28' 2023-10-30T12:35:52.503 INFO:tasks.workunit.client.0.smithi089.stdout:removed './216/24' 2023-10-30T12:35:52.503 INFO:tasks.workunit.client.0.smithi089.stdout:removed './216/29' 2023-10-30T12:35:52.503 INFO:tasks.workunit.client.0.smithi089.stdout:removed './216/32' 2023-10-30T12:35:52.503 INFO:tasks.workunit.client.0.smithi089.stdout:removed './216/22' 2023-10-30T12:35:52.504 INFO:tasks.workunit.client.0.smithi089.stdout:removed './216/25' 2023-10-30T12:35:52.504 INFO:tasks.workunit.client.0.smithi089.stdout:removed './216/30' 2023-10-30T12:35:52.504 INFO:tasks.workunit.client.0.smithi089.stdout:removed './216/33' 2023-10-30T12:35:52.504 INFO:tasks.workunit.client.0.smithi089.stdout:removed './216/23' 2023-10-30T12:35:52.504 INFO:tasks.workunit.client.0.smithi089.stdout:removed './216/35' 2023-10-30T12:35:52.505 INFO:tasks.workunit.client.0.smithi089.stdout:removed './216/26' 2023-10-30T12:35:52.505 INFO:tasks.workunit.client.0.smithi089.stdout:removed './216/37' 2023-10-30T12:35:52.505 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './216' 2023-10-30T12:35:52.505 INFO:tasks.workunit.client.0.smithi089.stdout:removed './217/21' 2023-10-30T12:35:52.505 INFO:tasks.workunit.client.0.smithi089.stdout:removed './217/27' 2023-10-30T12:35:52.506 INFO:tasks.workunit.client.0.smithi089.stdout:removed './217/31' 2023-10-30T12:35:52.506 INFO:tasks.workunit.client.0.smithi089.stdout:removed './217/36' 2023-10-30T12:35:52.506 INFO:tasks.workunit.client.0.smithi089.stdout:removed './217/20' 2023-10-30T12:35:52.506 INFO:tasks.workunit.client.0.smithi089.stdout:removed './217/39' 2023-10-30T12:35:52.506 INFO:tasks.workunit.client.0.smithi089.stdout:removed './217/34' 2023-10-30T12:35:52.507 INFO:tasks.workunit.client.0.smithi089.stdout:removed './217/38' 2023-10-30T12:35:52.507 INFO:tasks.workunit.client.0.smithi089.stdout:removed './217/28' 2023-10-30T12:35:52.507 INFO:tasks.workunit.client.0.smithi089.stdout:removed './217/24' 2023-10-30T12:35:52.507 INFO:tasks.workunit.client.0.smithi089.stdout:removed './217/29' 2023-10-30T12:35:52.507 INFO:tasks.workunit.client.0.smithi089.stdout:removed './217/32' 2023-10-30T12:35:52.508 INFO:tasks.workunit.client.0.smithi089.stdout:removed './217/22' 2023-10-30T12:35:52.508 INFO:tasks.workunit.client.0.smithi089.stdout:removed './217/25' 2023-10-30T12:35:52.508 INFO:tasks.workunit.client.0.smithi089.stdout:removed './217/30' 2023-10-30T12:35:52.508 INFO:tasks.workunit.client.0.smithi089.stdout:removed './217/33' 2023-10-30T12:35:52.508 INFO:tasks.workunit.client.0.smithi089.stdout:removed './217/23' 2023-10-30T12:35:52.509 INFO:tasks.workunit.client.0.smithi089.stdout:removed './217/35' 2023-10-30T12:35:52.509 INFO:tasks.workunit.client.0.smithi089.stdout:removed './217/26' 2023-10-30T12:35:52.509 INFO:tasks.workunit.client.0.smithi089.stdout:removed './217/37' 2023-10-30T12:35:52.509 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './217' 2023-10-30T12:35:52.509 INFO:tasks.workunit.client.0.smithi089.stdout:removed './218/21' 2023-10-30T12:35:52.509 INFO:tasks.workunit.client.0.smithi089.stdout:removed './218/27' 2023-10-30T12:35:52.510 INFO:tasks.workunit.client.0.smithi089.stdout:removed './218/31' 2023-10-30T12:35:52.510 INFO:tasks.workunit.client.0.smithi089.stdout:removed './218/36' 2023-10-30T12:35:52.510 INFO:tasks.workunit.client.0.smithi089.stdout:removed './218/20' 2023-10-30T12:35:52.510 INFO:tasks.workunit.client.0.smithi089.stdout:removed './218/39' 2023-10-30T12:35:52.511 INFO:tasks.workunit.client.0.smithi089.stdout:removed './218/34' 2023-10-30T12:35:52.511 INFO:tasks.workunit.client.0.smithi089.stdout:removed './218/38' 2023-10-30T12:35:52.511 INFO:tasks.workunit.client.0.smithi089.stdout:removed './218/28' 2023-10-30T12:35:52.511 INFO:tasks.workunit.client.0.smithi089.stdout:removed './218/24' 2023-10-30T12:35:52.511 INFO:tasks.workunit.client.0.smithi089.stdout:removed './218/29' 2023-10-30T12:35:52.511 INFO:tasks.workunit.client.0.smithi089.stdout:removed './218/32' 2023-10-30T12:35:52.512 INFO:tasks.workunit.client.0.smithi089.stdout:removed './218/22' 2023-10-30T12:35:52.512 INFO:tasks.workunit.client.0.smithi089.stdout:removed './218/25' 2023-10-30T12:35:52.512 INFO:tasks.workunit.client.0.smithi089.stdout:removed './218/30' 2023-10-30T12:35:52.512 INFO:tasks.workunit.client.0.smithi089.stdout:removed './218/33' 2023-10-30T12:35:52.512 INFO:tasks.workunit.client.0.smithi089.stdout:removed './218/23' 2023-10-30T12:35:52.513 INFO:tasks.workunit.client.0.smithi089.stdout:removed './218/35' 2023-10-30T12:35:52.513 INFO:tasks.workunit.client.0.smithi089.stdout:removed './218/26' 2023-10-30T12:35:52.513 INFO:tasks.workunit.client.0.smithi089.stdout:removed './218/37' 2023-10-30T12:35:52.513 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './218' 2023-10-30T12:35:52.513 INFO:tasks.workunit.client.0.smithi089.stdout:removed './219/21' 2023-10-30T12:35:52.514 INFO:tasks.workunit.client.0.smithi089.stdout:removed './219/27' 2023-10-30T12:35:52.514 INFO:tasks.workunit.client.0.smithi089.stdout:removed './219/31' 2023-10-30T12:35:52.514 INFO:tasks.workunit.client.0.smithi089.stdout:removed './219/36' 2023-10-30T12:35:52.514 INFO:tasks.workunit.client.0.smithi089.stdout:removed './219/20' 2023-10-30T12:35:52.514 INFO:tasks.workunit.client.0.smithi089.stdout:removed './219/39' 2023-10-30T12:35:52.515 INFO:tasks.workunit.client.0.smithi089.stdout:removed './219/34' 2023-10-30T12:35:52.515 INFO:tasks.workunit.client.0.smithi089.stdout:removed './219/38' 2023-10-30T12:35:52.515 INFO:tasks.workunit.client.0.smithi089.stdout:removed './219/28' 2023-10-30T12:35:52.515 INFO:tasks.workunit.client.0.smithi089.stdout:removed './219/24' 2023-10-30T12:35:52.515 INFO:tasks.workunit.client.0.smithi089.stdout:removed './219/29' 2023-10-30T12:35:52.515 INFO:tasks.workunit.client.0.smithi089.stdout:removed './219/32' 2023-10-30T12:35:52.516 INFO:tasks.workunit.client.0.smithi089.stdout:removed './219/22' 2023-10-30T12:35:52.516 INFO:tasks.workunit.client.0.smithi089.stdout:removed './219/25' 2023-10-30T12:35:52.516 INFO:tasks.workunit.client.0.smithi089.stdout:removed './219/30' 2023-10-30T12:35:52.516 INFO:tasks.workunit.client.0.smithi089.stdout:removed './219/33' 2023-10-30T12:35:52.516 INFO:tasks.workunit.client.0.smithi089.stdout:removed './219/23' 2023-10-30T12:35:52.517 INFO:tasks.workunit.client.0.smithi089.stdout:removed './219/35' 2023-10-30T12:35:52.517 INFO:tasks.workunit.client.0.smithi089.stdout:removed './219/26' 2023-10-30T12:35:52.517 INFO:tasks.workunit.client.0.smithi089.stdout:removed './219/37' 2023-10-30T12:35:52.517 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './219' 2023-10-30T12:35:52.517 INFO:tasks.workunit.client.0.smithi089.stdout:removed './220/21' 2023-10-30T12:35:52.518 INFO:tasks.workunit.client.0.smithi089.stdout:removed './220/27' 2023-10-30T12:35:52.518 INFO:tasks.workunit.client.0.smithi089.stdout:removed './220/31' 2023-10-30T12:35:52.518 INFO:tasks.workunit.client.0.smithi089.stdout:removed './220/36' 2023-10-30T12:35:52.518 INFO:tasks.workunit.client.0.smithi089.stdout:removed './220/20' 2023-10-30T12:35:52.518 INFO:tasks.workunit.client.0.smithi089.stdout:removed './220/39' 2023-10-30T12:35:52.519 INFO:tasks.workunit.client.0.smithi089.stdout:removed './220/34' 2023-10-30T12:35:52.519 INFO:tasks.workunit.client.0.smithi089.stdout:removed './220/38' 2023-10-30T12:35:52.519 INFO:tasks.workunit.client.0.smithi089.stdout:removed './220/28' 2023-10-30T12:35:52.519 INFO:tasks.workunit.client.0.smithi089.stdout:removed './220/24' 2023-10-30T12:35:52.519 INFO:tasks.workunit.client.0.smithi089.stdout:removed './220/29' 2023-10-30T12:35:52.520 INFO:tasks.workunit.client.0.smithi089.stdout:removed './220/32' 2023-10-30T12:35:52.520 INFO:tasks.workunit.client.0.smithi089.stdout:removed './220/22' 2023-10-30T12:35:52.520 INFO:tasks.workunit.client.0.smithi089.stdout:removed './220/25' 2023-10-30T12:35:52.520 INFO:tasks.workunit.client.0.smithi089.stdout:removed './220/30' 2023-10-30T12:35:52.520 INFO:tasks.workunit.client.0.smithi089.stdout:removed './220/33' 2023-10-30T12:35:52.520 INFO:tasks.workunit.client.0.smithi089.stdout:removed './220/23' 2023-10-30T12:35:52.521 INFO:tasks.workunit.client.0.smithi089.stdout:removed './220/35' 2023-10-30T12:35:52.521 INFO:tasks.workunit.client.0.smithi089.stdout:removed './220/26' 2023-10-30T12:35:52.521 INFO:tasks.workunit.client.0.smithi089.stdout:removed './220/37' 2023-10-30T12:35:52.521 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './220' 2023-10-30T12:35:52.521 INFO:tasks.workunit.client.0.smithi089.stdout:removed './221/21' 2023-10-30T12:35:52.522 INFO:tasks.workunit.client.0.smithi089.stdout:removed './221/27' 2023-10-30T12:35:52.522 INFO:tasks.workunit.client.0.smithi089.stdout:removed './221/31' 2023-10-30T12:35:52.522 INFO:tasks.workunit.client.0.smithi089.stdout:removed './221/36' 2023-10-30T12:35:52.522 INFO:tasks.workunit.client.0.smithi089.stdout:removed './221/20' 2023-10-30T12:35:52.522 INFO:tasks.workunit.client.0.smithi089.stdout:removed './221/39' 2023-10-30T12:35:52.523 INFO:tasks.workunit.client.0.smithi089.stdout:removed './221/34' 2023-10-30T12:35:52.523 INFO:tasks.workunit.client.0.smithi089.stdout:removed './221/38' 2023-10-30T12:35:52.523 INFO:tasks.workunit.client.0.smithi089.stdout:removed './221/28' 2023-10-30T12:35:52.523 INFO:tasks.workunit.client.0.smithi089.stdout:removed './221/24' 2023-10-30T12:35:52.523 INFO:tasks.workunit.client.0.smithi089.stdout:removed './221/29' 2023-10-30T12:35:52.524 INFO:tasks.workunit.client.0.smithi089.stdout:removed './221/32' 2023-10-30T12:35:52.524 INFO:tasks.workunit.client.0.smithi089.stdout:removed './221/22' 2023-10-30T12:35:52.524 INFO:tasks.workunit.client.0.smithi089.stdout:removed './221/25' 2023-10-30T12:35:52.524 INFO:tasks.workunit.client.0.smithi089.stdout:removed './221/30' 2023-10-30T12:35:52.524 INFO:tasks.workunit.client.0.smithi089.stdout:removed './221/33' 2023-10-30T12:35:52.524 INFO:tasks.workunit.client.0.smithi089.stdout:removed './221/23' 2023-10-30T12:35:52.525 INFO:tasks.workunit.client.0.smithi089.stdout:removed './221/35' 2023-10-30T12:35:52.525 INFO:tasks.workunit.client.0.smithi089.stdout:removed './221/26' 2023-10-30T12:35:52.525 INFO:tasks.workunit.client.0.smithi089.stdout:removed './221/37' 2023-10-30T12:35:52.525 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './221' 2023-10-30T12:35:52.525 INFO:tasks.workunit.client.0.smithi089.stdout:removed './222/21' 2023-10-30T12:35:52.526 INFO:tasks.workunit.client.0.smithi089.stdout:removed './222/27' 2023-10-30T12:35:52.526 INFO:tasks.workunit.client.0.smithi089.stdout:removed './222/31' 2023-10-30T12:35:52.526 INFO:tasks.workunit.client.0.smithi089.stdout:removed './222/36' 2023-10-30T12:35:52.526 INFO:tasks.workunit.client.0.smithi089.stdout:removed './222/20' 2023-10-30T12:35:52.526 INFO:tasks.workunit.client.0.smithi089.stdout:removed './222/39' 2023-10-30T12:35:52.527 INFO:tasks.workunit.client.0.smithi089.stdout:removed './222/34' 2023-10-30T12:35:52.527 INFO:tasks.workunit.client.0.smithi089.stdout:removed './222/38' 2023-10-30T12:35:52.527 INFO:tasks.workunit.client.0.smithi089.stdout:removed './222/28' 2023-10-30T12:35:52.527 INFO:tasks.workunit.client.0.smithi089.stdout:removed './222/24' 2023-10-30T12:35:52.527 INFO:tasks.workunit.client.0.smithi089.stdout:removed './222/29' 2023-10-30T12:35:52.527 INFO:tasks.workunit.client.0.smithi089.stdout:removed './222/32' 2023-10-30T12:35:52.528 INFO:tasks.workunit.client.0.smithi089.stdout:removed './222/22' 2023-10-30T12:35:52.528 INFO:tasks.workunit.client.0.smithi089.stdout:removed './222/25' 2023-10-30T12:35:52.528 INFO:tasks.workunit.client.0.smithi089.stdout:removed './222/30' 2023-10-30T12:35:52.528 INFO:tasks.workunit.client.0.smithi089.stdout:removed './222/33' 2023-10-30T12:35:52.528 INFO:tasks.workunit.client.0.smithi089.stdout:removed './222/23' 2023-10-30T12:35:52.529 INFO:tasks.workunit.client.0.smithi089.stdout:removed './222/35' 2023-10-30T12:35:52.529 INFO:tasks.workunit.client.0.smithi089.stdout:removed './222/26' 2023-10-30T12:35:52.529 INFO:tasks.workunit.client.0.smithi089.stdout:removed './222/37' 2023-10-30T12:35:52.529 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './222' 2023-10-30T12:35:52.529 INFO:tasks.workunit.client.0.smithi089.stdout:removed './223/21' 2023-10-30T12:35:52.530 INFO:tasks.workunit.client.0.smithi089.stdout:removed './223/27' 2023-10-30T12:35:52.530 INFO:tasks.workunit.client.0.smithi089.stdout:removed './223/31' 2023-10-30T12:35:52.530 INFO:tasks.workunit.client.0.smithi089.stdout:removed './223/36' 2023-10-30T12:35:52.530 INFO:tasks.workunit.client.0.smithi089.stdout:removed './223/20' 2023-10-30T12:35:52.530 INFO:tasks.workunit.client.0.smithi089.stdout:removed './223/39' 2023-10-30T12:35:52.531 INFO:tasks.workunit.client.0.smithi089.stdout:removed './223/34' 2023-10-30T12:35:52.531 INFO:tasks.workunit.client.0.smithi089.stdout:removed './223/38' 2023-10-30T12:35:52.531 INFO:tasks.workunit.client.0.smithi089.stdout:removed './223/28' 2023-10-30T12:35:52.531 INFO:tasks.workunit.client.0.smithi089.stdout:removed './223/24' 2023-10-30T12:35:52.531 INFO:tasks.workunit.client.0.smithi089.stdout:removed './223/29' 2023-10-30T12:35:52.532 INFO:tasks.workunit.client.0.smithi089.stdout:removed './223/32' 2023-10-30T12:35:52.532 INFO:tasks.workunit.client.0.smithi089.stdout:removed './223/22' 2023-10-30T12:35:52.532 INFO:tasks.workunit.client.0.smithi089.stdout:removed './223/25' 2023-10-30T12:35:52.532 INFO:tasks.workunit.client.0.smithi089.stdout:removed './223/30' 2023-10-30T12:35:52.532 INFO:tasks.workunit.client.0.smithi089.stdout:removed './223/33' 2023-10-30T12:35:52.532 INFO:tasks.workunit.client.0.smithi089.stdout:removed './223/23' 2023-10-30T12:35:52.533 INFO:tasks.workunit.client.0.smithi089.stdout:removed './223/35' 2023-10-30T12:35:52.533 INFO:tasks.workunit.client.0.smithi089.stdout:removed './223/26' 2023-10-30T12:35:52.533 INFO:tasks.workunit.client.0.smithi089.stdout:removed './223/37' 2023-10-30T12:35:52.533 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './223' 2023-10-30T12:35:52.533 INFO:tasks.workunit.client.0.smithi089.stdout:removed './224/21' 2023-10-30T12:35:52.534 INFO:tasks.workunit.client.0.smithi089.stdout:removed './224/27' 2023-10-30T12:35:52.534 INFO:tasks.workunit.client.0.smithi089.stdout:removed './224/31' 2023-10-30T12:35:52.534 INFO:tasks.workunit.client.0.smithi089.stdout:removed './224/36' 2023-10-30T12:35:52.534 INFO:tasks.workunit.client.0.smithi089.stdout:removed './224/20' 2023-10-30T12:35:52.534 INFO:tasks.workunit.client.0.smithi089.stdout:removed './224/39' 2023-10-30T12:35:52.535 INFO:tasks.workunit.client.0.smithi089.stdout:removed './224/34' 2023-10-30T12:35:52.535 INFO:tasks.workunit.client.0.smithi089.stdout:removed './224/38' 2023-10-30T12:35:52.535 INFO:tasks.workunit.client.0.smithi089.stdout:removed './224/28' 2023-10-30T12:35:52.535 INFO:tasks.workunit.client.0.smithi089.stdout:removed './224/24' 2023-10-30T12:35:52.535 INFO:tasks.workunit.client.0.smithi089.stdout:removed './224/29' 2023-10-30T12:35:52.536 INFO:tasks.workunit.client.0.smithi089.stdout:removed './224/32' 2023-10-30T12:35:52.536 INFO:tasks.workunit.client.0.smithi089.stdout:removed './224/22' 2023-10-30T12:35:52.536 INFO:tasks.workunit.client.0.smithi089.stdout:removed './224/25' 2023-10-30T12:35:52.536 INFO:tasks.workunit.client.0.smithi089.stdout:removed './224/30' 2023-10-30T12:35:52.536 INFO:tasks.workunit.client.0.smithi089.stdout:removed './224/33' 2023-10-30T12:35:52.537 INFO:tasks.workunit.client.0.smithi089.stdout:removed './224/23' 2023-10-30T12:35:52.537 INFO:tasks.workunit.client.0.smithi089.stdout:removed './224/35' 2023-10-30T12:35:52.537 INFO:tasks.workunit.client.0.smithi089.stdout:removed './224/26' 2023-10-30T12:35:52.537 INFO:tasks.workunit.client.0.smithi089.stdout:removed './224/37' 2023-10-30T12:35:52.537 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './224' 2023-10-30T12:35:52.537 INFO:tasks.workunit.client.0.smithi089.stdout:removed './225/21' 2023-10-30T12:35:52.538 INFO:tasks.workunit.client.0.smithi089.stdout:removed './225/27' 2023-10-30T12:35:52.538 INFO:tasks.workunit.client.0.smithi089.stdout:removed './225/31' 2023-10-30T12:35:52.538 INFO:tasks.workunit.client.0.smithi089.stdout:removed './225/36' 2023-10-30T12:35:52.538 INFO:tasks.workunit.client.0.smithi089.stdout:removed './225/20' 2023-10-30T12:35:52.538 INFO:tasks.workunit.client.0.smithi089.stdout:removed './225/39' 2023-10-30T12:35:52.539 INFO:tasks.workunit.client.0.smithi089.stdout:removed './225/34' 2023-10-30T12:35:52.539 INFO:tasks.workunit.client.0.smithi089.stdout:removed './225/38' 2023-10-30T12:35:52.539 INFO:tasks.workunit.client.0.smithi089.stdout:removed './225/28' 2023-10-30T12:35:52.539 INFO:tasks.workunit.client.0.smithi089.stdout:removed './225/24' 2023-10-30T12:35:52.539 INFO:tasks.workunit.client.0.smithi089.stdout:removed './225/29' 2023-10-30T12:35:52.540 INFO:tasks.workunit.client.0.smithi089.stdout:removed './225/32' 2023-10-30T12:35:52.540 INFO:tasks.workunit.client.0.smithi089.stdout:removed './225/22' 2023-10-30T12:35:52.540 INFO:tasks.workunit.client.0.smithi089.stdout:removed './225/25' 2023-10-30T12:35:52.540 INFO:tasks.workunit.client.0.smithi089.stdout:removed './225/30' 2023-10-30T12:35:52.540 INFO:tasks.workunit.client.0.smithi089.stdout:removed './225/33' 2023-10-30T12:35:52.541 INFO:tasks.workunit.client.0.smithi089.stdout:removed './225/23' 2023-10-30T12:35:52.541 INFO:tasks.workunit.client.0.smithi089.stdout:removed './225/35' 2023-10-30T12:35:52.541 INFO:tasks.workunit.client.0.smithi089.stdout:removed './225/26' 2023-10-30T12:35:52.541 INFO:tasks.workunit.client.0.smithi089.stdout:removed './225/37' 2023-10-30T12:35:52.541 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './225' 2023-10-30T12:35:52.541 INFO:tasks.workunit.client.0.smithi089.stdout:removed './226/21' 2023-10-30T12:35:52.542 INFO:tasks.workunit.client.0.smithi089.stdout:removed './226/27' 2023-10-30T12:35:52.542 INFO:tasks.workunit.client.0.smithi089.stdout:removed './226/31' 2023-10-30T12:35:52.542 INFO:tasks.workunit.client.0.smithi089.stdout:removed './226/36' 2023-10-30T12:35:52.542 INFO:tasks.workunit.client.0.smithi089.stdout:removed './226/20' 2023-10-30T12:35:52.542 INFO:tasks.workunit.client.0.smithi089.stdout:removed './226/39' 2023-10-30T12:35:52.543 INFO:tasks.workunit.client.0.smithi089.stdout:removed './226/34' 2023-10-30T12:35:52.543 INFO:tasks.workunit.client.0.smithi089.stdout:removed './226/38' 2023-10-30T12:35:52.543 INFO:tasks.workunit.client.0.smithi089.stdout:removed './226/28' 2023-10-30T12:35:52.543 INFO:tasks.workunit.client.0.smithi089.stdout:removed './226/24' 2023-10-30T12:35:59.868 INFO:tasks.workunit.client.0.smithi089.stdout:removed './226/29' 2023-10-30T12:35:59.868 INFO:tasks.workunit.client.0.smithi089.stdout:removed './226/32' 2023-10-30T12:35:59.869 INFO:tasks.workunit.client.0.smithi089.stdout:removed './226/22' 2023-10-30T12:35:59.869 INFO:tasks.workunit.client.0.smithi089.stdout:removed './226/25' 2023-10-30T12:35:59.869 INFO:tasks.workunit.client.0.smithi089.stdout:removed './226/30' 2023-10-30T12:35:59.869 INFO:tasks.workunit.client.0.smithi089.stdout:removed './226/33' 2023-10-30T12:35:59.869 INFO:tasks.workunit.client.0.smithi089.stdout:removed './226/23' 2023-10-30T12:35:59.870 INFO:tasks.workunit.client.0.smithi089.stdout:removed './226/35' 2023-10-30T12:35:59.870 INFO:tasks.workunit.client.0.smithi089.stdout:removed './226/26' 2023-10-30T12:35:59.870 INFO:tasks.workunit.client.0.smithi089.stdout:removed './226/37' 2023-10-30T12:35:59.870 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './226' 2023-10-30T12:35:59.871 INFO:tasks.workunit.client.0.smithi089.stdout:removed './227/21' 2023-10-30T12:35:59.871 INFO:tasks.workunit.client.0.smithi089.stdout:removed './227/27' 2023-10-30T12:35:59.871 INFO:tasks.workunit.client.0.smithi089.stdout:removed './227/31' 2023-10-30T12:35:59.871 INFO:tasks.workunit.client.0.smithi089.stdout:removed './227/36' 2023-10-30T12:35:59.871 INFO:tasks.workunit.client.0.smithi089.stdout:removed './227/20' 2023-10-30T12:35:59.872 INFO:tasks.workunit.client.0.smithi089.stdout:removed './227/39' 2023-10-30T12:35:59.872 INFO:tasks.workunit.client.0.smithi089.stdout:removed './227/34' 2023-10-30T12:35:59.872 INFO:tasks.workunit.client.0.smithi089.stdout:removed './227/38' 2023-10-30T12:35:59.873 INFO:tasks.workunit.client.0.smithi089.stdout:removed './227/28' 2023-10-30T12:35:59.873 INFO:tasks.workunit.client.0.smithi089.stdout:removed './227/24' 2023-10-30T12:35:59.873 INFO:tasks.workunit.client.0.smithi089.stdout:removed './227/29' 2023-10-30T12:35:59.873 INFO:tasks.workunit.client.0.smithi089.stdout:removed './227/32' 2023-10-30T12:35:59.874 INFO:tasks.workunit.client.0.smithi089.stdout:removed './227/22' 2023-10-30T12:35:59.874 INFO:tasks.workunit.client.0.smithi089.stdout:removed './227/25' 2023-10-30T12:35:59.875 INFO:tasks.workunit.client.0.smithi089.stdout:removed './227/30' 2023-10-30T12:35:59.875 INFO:tasks.workunit.client.0.smithi089.stdout:removed './227/33' 2023-10-30T12:35:59.875 INFO:tasks.workunit.client.0.smithi089.stdout:removed './227/23' 2023-10-30T12:35:59.875 INFO:tasks.workunit.client.0.smithi089.stdout:removed './227/35' 2023-10-30T12:35:59.876 INFO:tasks.workunit.client.0.smithi089.stdout:removed './227/26' 2023-10-30T12:35:59.876 INFO:tasks.workunit.client.0.smithi089.stdout:removed './227/37' 2023-10-30T12:35:59.877 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './227' 2023-10-30T12:35:59.877 INFO:tasks.workunit.client.0.smithi089.stdout:removed './228/21' 2023-10-30T12:35:59.877 INFO:tasks.workunit.client.0.smithi089.stdout:removed './228/27' 2023-10-30T12:35:59.878 INFO:tasks.workunit.client.0.smithi089.stdout:removed './228/31' 2023-10-30T12:35:59.878 INFO:tasks.workunit.client.0.smithi089.stdout:removed './228/36' 2023-10-30T12:35:59.878 INFO:tasks.workunit.client.0.smithi089.stdout:removed './228/20' 2023-10-30T12:35:59.879 INFO:tasks.workunit.client.0.smithi089.stdout:removed './228/39' 2023-10-30T12:35:59.879 INFO:tasks.workunit.client.0.smithi089.stdout:removed './228/34' 2023-10-30T12:35:59.879 INFO:tasks.workunit.client.0.smithi089.stdout:removed './228/38' 2023-10-30T12:35:59.880 INFO:tasks.workunit.client.0.smithi089.stdout:removed './228/28' 2023-10-30T12:35:59.880 INFO:tasks.workunit.client.0.smithi089.stdout:removed './228/24' 2023-10-30T12:35:59.880 INFO:tasks.workunit.client.0.smithi089.stdout:removed './228/29' 2023-10-30T12:35:59.881 INFO:tasks.workunit.client.0.smithi089.stdout:removed './228/32' 2023-10-30T12:35:59.881 INFO:tasks.workunit.client.0.smithi089.stdout:removed './228/22' 2023-10-30T12:35:59.882 INFO:tasks.workunit.client.0.smithi089.stdout:removed './228/25' 2023-10-30T12:35:59.882 INFO:tasks.workunit.client.0.smithi089.stdout:removed './228/30' 2023-10-30T12:35:59.882 INFO:tasks.workunit.client.0.smithi089.stdout:removed './228/33' 2023-10-30T12:35:59.883 INFO:tasks.workunit.client.0.smithi089.stdout:removed './228/23' 2023-10-30T12:35:59.883 INFO:tasks.workunit.client.0.smithi089.stdout:removed './228/35' 2023-10-30T12:35:59.883 INFO:tasks.workunit.client.0.smithi089.stdout:removed './228/26' 2023-10-30T12:35:59.884 INFO:tasks.workunit.client.0.smithi089.stdout:removed './228/37' 2023-10-30T12:35:59.884 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './228' 2023-10-30T12:35:59.884 INFO:tasks.workunit.client.0.smithi089.stdout:removed './229/21' 2023-10-30T12:35:59.884 INFO:tasks.workunit.client.0.smithi089.stdout:removed './229/27' 2023-10-30T12:35:59.884 INFO:tasks.workunit.client.0.smithi089.stdout:removed './229/31' 2023-10-30T12:35:59.885 INFO:tasks.workunit.client.0.smithi089.stdout:removed './229/36' 2023-10-30T12:35:59.885 INFO:tasks.workunit.client.0.smithi089.stdout:removed './229/20' 2023-10-30T12:35:59.885 INFO:tasks.workunit.client.0.smithi089.stdout:removed './229/39' 2023-10-30T12:35:59.885 INFO:tasks.workunit.client.0.smithi089.stdout:removed './229/34' 2023-10-30T12:35:59.885 INFO:tasks.workunit.client.0.smithi089.stdout:removed './229/38' 2023-10-30T12:35:59.886 INFO:tasks.workunit.client.0.smithi089.stdout:removed './229/28' 2023-10-30T12:35:59.886 INFO:tasks.workunit.client.0.smithi089.stdout:removed './229/24' 2023-10-30T12:35:59.886 INFO:tasks.workunit.client.0.smithi089.stdout:removed './229/29' 2023-10-30T12:35:59.886 INFO:tasks.workunit.client.0.smithi089.stdout:removed './229/32' 2023-10-30T12:35:59.886 INFO:tasks.workunit.client.0.smithi089.stdout:removed './229/22' 2023-10-30T12:35:59.887 INFO:tasks.workunit.client.0.smithi089.stdout:removed './229/25' 2023-10-30T12:35:59.887 INFO:tasks.workunit.client.0.smithi089.stdout:removed './229/30' 2023-10-30T12:35:59.887 INFO:tasks.workunit.client.0.smithi089.stdout:removed './229/33' 2023-10-30T12:35:59.887 INFO:tasks.workunit.client.0.smithi089.stdout:removed './229/23' 2023-10-30T12:35:59.887 INFO:tasks.workunit.client.0.smithi089.stdout:removed './229/35' 2023-10-30T12:35:59.888 INFO:tasks.workunit.client.0.smithi089.stdout:removed './229/26' 2023-10-30T12:35:59.888 INFO:tasks.workunit.client.0.smithi089.stdout:removed './229/37' 2023-10-30T12:35:59.888 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './229' 2023-10-30T12:35:59.888 INFO:tasks.workunit.client.0.smithi089.stdout:removed './230/21' 2023-10-30T12:35:59.889 INFO:tasks.workunit.client.0.smithi089.stdout:removed './230/27' 2023-10-30T12:35:59.889 INFO:tasks.workunit.client.0.smithi089.stdout:removed './230/31' 2023-10-30T12:35:59.889 INFO:tasks.workunit.client.0.smithi089.stdout:removed './230/36' 2023-10-30T12:35:59.890 INFO:tasks.workunit.client.0.smithi089.stdout:removed './230/20' 2023-10-30T12:35:59.890 INFO:tasks.workunit.client.0.smithi089.stdout:removed './230/39' 2023-10-30T12:35:59.890 INFO:tasks.workunit.client.0.smithi089.stdout:removed './230/34' 2023-10-30T12:35:59.890 INFO:tasks.workunit.client.0.smithi089.stdout:removed './230/38' 2023-10-30T12:35:59.891 INFO:tasks.workunit.client.0.smithi089.stdout:removed './230/28' 2023-10-30T12:35:59.891 INFO:tasks.workunit.client.0.smithi089.stdout:removed './230/24' 2023-10-30T12:35:59.891 INFO:tasks.workunit.client.0.smithi089.stdout:removed './230/29' 2023-10-30T12:35:59.892 INFO:tasks.workunit.client.0.smithi089.stdout:removed './230/32' 2023-10-30T12:35:59.892 INFO:tasks.workunit.client.0.smithi089.stdout:removed './230/22' 2023-10-30T12:35:59.892 INFO:tasks.workunit.client.0.smithi089.stdout:removed './230/25' 2023-10-30T12:35:59.893 INFO:tasks.workunit.client.0.smithi089.stdout:removed './230/30' 2023-10-30T12:35:59.893 INFO:tasks.workunit.client.0.smithi089.stdout:removed './230/33' 2023-10-30T12:35:59.893 INFO:tasks.workunit.client.0.smithi089.stdout:removed './230/23' 2023-10-30T12:35:59.894 INFO:tasks.workunit.client.0.smithi089.stdout:removed './230/35' 2023-10-30T12:35:59.894 INFO:tasks.workunit.client.0.smithi089.stdout:removed './230/26' 2023-10-30T12:35:59.894 INFO:tasks.workunit.client.0.smithi089.stdout:removed './230/37' 2023-10-30T12:35:59.895 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './230' 2023-10-30T12:35:59.895 INFO:tasks.workunit.client.0.smithi089.stdout:removed './231/21' 2023-10-30T12:35:59.895 INFO:tasks.workunit.client.0.smithi089.stdout:removed './231/27' 2023-10-30T12:35:59.896 INFO:tasks.workunit.client.0.smithi089.stdout:removed './231/31' 2023-10-30T12:35:59.896 INFO:tasks.workunit.client.0.smithi089.stdout:removed './231/36' 2023-10-30T12:35:59.896 INFO:tasks.workunit.client.0.smithi089.stdout:removed './231/20' 2023-10-30T12:35:59.897 INFO:tasks.workunit.client.0.smithi089.stdout:removed './231/39' 2023-10-30T12:35:59.897 INFO:tasks.workunit.client.0.smithi089.stdout:removed './231/34' 2023-10-30T12:35:59.898 INFO:tasks.workunit.client.0.smithi089.stdout:removed './231/38' 2023-10-30T12:35:59.898 INFO:tasks.workunit.client.0.smithi089.stdout:removed './231/28' 2023-10-30T12:35:59.898 INFO:tasks.workunit.client.0.smithi089.stdout:removed './231/24' 2023-10-30T12:35:59.899 INFO:tasks.workunit.client.0.smithi089.stdout:removed './231/29' 2023-10-30T12:35:59.899 INFO:tasks.workunit.client.0.smithi089.stdout:removed './231/32' 2023-10-30T12:35:59.899 INFO:tasks.workunit.client.0.smithi089.stdout:removed './231/22' 2023-10-30T12:35:59.899 INFO:tasks.workunit.client.0.smithi089.stdout:removed './231/25' 2023-10-30T12:35:59.899 INFO:tasks.workunit.client.0.smithi089.stdout:removed './231/30' 2023-10-30T12:35:59.900 INFO:tasks.workunit.client.0.smithi089.stdout:removed './231/33' 2023-10-30T12:35:59.900 INFO:tasks.workunit.client.0.smithi089.stdout:removed './231/23' 2023-10-30T12:35:59.900 INFO:tasks.workunit.client.0.smithi089.stdout:removed './231/35' 2023-10-30T12:35:59.900 INFO:tasks.workunit.client.0.smithi089.stdout:removed './231/26' 2023-10-30T12:35:59.900 INFO:tasks.workunit.client.0.smithi089.stdout:removed './231/37' 2023-10-30T12:35:59.901 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './231' 2023-10-30T12:35:59.901 INFO:tasks.workunit.client.0.smithi089.stdout:removed './232/21' 2023-10-30T12:35:59.901 INFO:tasks.workunit.client.0.smithi089.stdout:removed './232/27' 2023-10-30T12:35:59.901 INFO:tasks.workunit.client.0.smithi089.stdout:removed './232/31' 2023-10-30T12:35:59.901 INFO:tasks.workunit.client.0.smithi089.stdout:removed './232/36' 2023-10-30T12:35:59.901 INFO:tasks.workunit.client.0.smithi089.stdout:removed './232/20' 2023-10-30T12:35:59.902 INFO:tasks.workunit.client.0.smithi089.stdout:removed './232/39' 2023-10-30T12:35:59.902 INFO:tasks.workunit.client.0.smithi089.stdout:removed './232/34' 2023-10-30T12:35:59.902 INFO:tasks.workunit.client.0.smithi089.stdout:removed './232/38' 2023-10-30T12:35:59.902 INFO:tasks.workunit.client.0.smithi089.stdout:removed './232/28' 2023-10-30T12:35:59.902 INFO:tasks.workunit.client.0.smithi089.stdout:removed './232/24' 2023-10-30T12:35:59.903 INFO:tasks.workunit.client.0.smithi089.stdout:removed './232/29' 2023-10-30T12:35:59.903 INFO:tasks.workunit.client.0.smithi089.stdout:removed './232/32' 2023-10-30T12:35:59.903 INFO:tasks.workunit.client.0.smithi089.stdout:removed './232/22' 2023-10-30T12:35:59.903 INFO:tasks.workunit.client.0.smithi089.stdout:removed './232/25' 2023-10-30T12:35:59.903 INFO:tasks.workunit.client.0.smithi089.stdout:removed './232/30' 2023-10-30T12:35:59.904 INFO:tasks.workunit.client.0.smithi089.stdout:removed './232/33' 2023-10-30T12:35:59.904 INFO:tasks.workunit.client.0.smithi089.stdout:removed './232/23' 2023-10-30T12:35:59.904 INFO:tasks.workunit.client.0.smithi089.stdout:removed './232/35' 2023-10-30T12:35:59.905 INFO:tasks.workunit.client.0.smithi089.stdout:removed './232/26' 2023-10-30T12:35:59.905 INFO:tasks.workunit.client.0.smithi089.stdout:removed './232/37' 2023-10-30T12:35:59.905 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './232' 2023-10-30T12:35:59.906 INFO:tasks.workunit.client.0.smithi089.stdout:removed './233/21' 2023-10-30T12:35:59.906 INFO:tasks.workunit.client.0.smithi089.stdout:removed './233/27' 2023-10-30T12:35:59.906 INFO:tasks.workunit.client.0.smithi089.stdout:removed './233/31' 2023-10-30T12:35:59.906 INFO:tasks.workunit.client.0.smithi089.stdout:removed './233/36' 2023-10-30T12:35:59.906 INFO:tasks.workunit.client.0.smithi089.stdout:removed './233/20' 2023-10-30T12:35:59.907 INFO:tasks.workunit.client.0.smithi089.stdout:removed './233/39' 2023-10-30T12:35:59.907 INFO:tasks.workunit.client.0.smithi089.stdout:removed './233/34' 2023-10-30T12:35:59.907 INFO:tasks.workunit.client.0.smithi089.stdout:removed './233/38' 2023-10-30T12:35:59.907 INFO:tasks.workunit.client.0.smithi089.stdout:removed './233/28' 2023-10-30T12:35:59.907 INFO:tasks.workunit.client.0.smithi089.stdout:removed './233/24' 2023-10-30T12:35:59.908 INFO:tasks.workunit.client.0.smithi089.stdout:removed './233/29' 2023-10-30T12:35:59.908 INFO:tasks.workunit.client.0.smithi089.stdout:removed './233/32' 2023-10-30T12:35:59.908 INFO:tasks.workunit.client.0.smithi089.stdout:removed './233/22' 2023-10-30T12:35:59.908 INFO:tasks.workunit.client.0.smithi089.stdout:removed './233/25' 2023-10-30T12:35:59.908 INFO:tasks.workunit.client.0.smithi089.stdout:removed './233/30' 2023-10-30T12:35:59.909 INFO:tasks.workunit.client.0.smithi089.stdout:removed './233/33' 2023-10-30T12:35:59.909 INFO:tasks.workunit.client.0.smithi089.stdout:removed './233/23' 2023-10-30T12:35:59.909 INFO:tasks.workunit.client.0.smithi089.stdout:removed './233/35' 2023-10-30T12:35:59.909 INFO:tasks.workunit.client.0.smithi089.stdout:removed './233/26' 2023-10-30T12:35:59.909 INFO:tasks.workunit.client.0.smithi089.stdout:removed './233/37' 2023-10-30T12:35:59.910 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './233' 2023-10-30T12:35:59.910 INFO:tasks.workunit.client.0.smithi089.stdout:removed './234/21' 2023-10-30T12:35:59.910 INFO:tasks.workunit.client.0.smithi089.stdout:removed './234/27' 2023-10-30T12:35:59.910 INFO:tasks.workunit.client.0.smithi089.stdout:removed './234/31' 2023-10-30T12:35:59.910 INFO:tasks.workunit.client.0.smithi089.stdout:removed './234/36' 2023-10-30T12:35:59.911 INFO:tasks.workunit.client.0.smithi089.stdout:removed './234/20' 2023-10-30T12:35:59.911 INFO:tasks.workunit.client.0.smithi089.stdout:removed './234/39' 2023-10-30T12:35:59.911 INFO:tasks.workunit.client.0.smithi089.stdout:removed './234/34' 2023-10-30T12:35:59.911 INFO:tasks.workunit.client.0.smithi089.stdout:removed './234/38' 2023-10-30T12:35:59.912 INFO:tasks.workunit.client.0.smithi089.stdout:removed './234/28' 2023-10-30T12:35:59.912 INFO:tasks.workunit.client.0.smithi089.stdout:removed './234/24' 2023-10-30T12:35:59.912 INFO:tasks.workunit.client.0.smithi089.stdout:removed './234/29' 2023-10-30T12:35:59.912 INFO:tasks.workunit.client.0.smithi089.stdout:removed './234/32' 2023-10-30T12:35:59.912 INFO:tasks.workunit.client.0.smithi089.stdout:removed './234/22' 2023-10-30T12:35:59.913 INFO:tasks.workunit.client.0.smithi089.stdout:removed './234/25' 2023-10-30T12:35:59.913 INFO:tasks.workunit.client.0.smithi089.stdout:removed './234/30' 2023-10-30T12:35:59.913 INFO:tasks.workunit.client.0.smithi089.stdout:removed './234/33' 2023-10-30T12:35:59.913 INFO:tasks.workunit.client.0.smithi089.stdout:removed './234/23' 2023-10-30T12:35:59.913 INFO:tasks.workunit.client.0.smithi089.stdout:removed './234/35' 2023-10-30T12:35:59.913 INFO:tasks.workunit.client.0.smithi089.stdout:removed './234/26' 2023-10-30T12:35:59.914 INFO:tasks.workunit.client.0.smithi089.stdout:removed './234/37' 2023-10-30T12:35:59.914 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './234' 2023-10-30T12:35:59.914 INFO:tasks.workunit.client.0.smithi089.stdout:removed './235/21' 2023-10-30T12:35:59.914 INFO:tasks.workunit.client.0.smithi089.stdout:removed './235/27' 2023-10-30T12:35:59.915 INFO:tasks.workunit.client.0.smithi089.stdout:removed './235/31' 2023-10-30T12:35:59.915 INFO:tasks.workunit.client.0.smithi089.stdout:removed './235/36' 2023-10-30T12:35:59.915 INFO:tasks.workunit.client.0.smithi089.stdout:removed './235/20' 2023-10-30T12:35:59.915 INFO:tasks.workunit.client.0.smithi089.stdout:removed './235/39' 2023-10-30T12:35:59.915 INFO:tasks.workunit.client.0.smithi089.stdout:removed './235/34' 2023-10-30T12:35:59.916 INFO:tasks.workunit.client.0.smithi089.stdout:removed './235/38' 2023-10-30T12:35:59.916 INFO:tasks.workunit.client.0.smithi089.stdout:removed './235/28' 2023-10-30T12:35:59.916 INFO:tasks.workunit.client.0.smithi089.stdout:removed './235/24' 2023-10-30T12:35:59.916 INFO:tasks.workunit.client.0.smithi089.stdout:removed './235/29' 2023-10-30T12:35:59.917 INFO:tasks.workunit.client.0.smithi089.stdout:removed './235/32' 2023-10-30T12:35:59.917 INFO:tasks.workunit.client.0.smithi089.stdout:removed './235/22' 2023-10-30T12:35:59.917 INFO:tasks.workunit.client.0.smithi089.stdout:removed './235/25' 2023-10-30T12:35:59.917 INFO:tasks.workunit.client.0.smithi089.stdout:removed './235/30' 2023-10-30T12:35:59.917 INFO:tasks.workunit.client.0.smithi089.stdout:removed './235/33' 2023-10-30T12:35:59.918 INFO:tasks.workunit.client.0.smithi089.stdout:removed './235/23' 2023-10-30T12:35:59.918 INFO:tasks.workunit.client.0.smithi089.stdout:removed './235/35' 2023-10-30T12:35:59.918 INFO:tasks.workunit.client.0.smithi089.stdout:removed './235/26' 2023-10-30T12:35:59.918 INFO:tasks.workunit.client.0.smithi089.stdout:removed './235/37' 2023-10-30T12:35:59.918 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './235' 2023-10-30T12:35:59.919 INFO:tasks.workunit.client.0.smithi089.stdout:removed './236/21' 2023-10-30T12:35:59.919 INFO:tasks.workunit.client.0.smithi089.stdout:removed './236/27' 2023-10-30T12:35:59.919 INFO:tasks.workunit.client.0.smithi089.stdout:removed './236/31' 2023-10-30T12:35:59.919 INFO:tasks.workunit.client.0.smithi089.stdout:removed './236/36' 2023-10-30T12:35:59.919 INFO:tasks.workunit.client.0.smithi089.stdout:removed './236/20' 2023-10-30T12:35:59.920 INFO:tasks.workunit.client.0.smithi089.stdout:removed './236/39' 2023-10-30T12:35:59.920 INFO:tasks.workunit.client.0.smithi089.stdout:removed './236/34' 2023-10-30T12:35:59.920 INFO:tasks.workunit.client.0.smithi089.stdout:removed './236/38' 2023-10-30T12:35:59.920 INFO:tasks.workunit.client.0.smithi089.stdout:removed './236/28' 2023-10-30T12:35:59.920 INFO:tasks.workunit.client.0.smithi089.stdout:removed './236/24' 2023-10-30T12:35:59.920 INFO:tasks.workunit.client.0.smithi089.stdout:removed './236/29' 2023-10-30T12:35:59.921 INFO:tasks.workunit.client.0.smithi089.stdout:removed './236/32' 2023-10-30T12:36:02.090 INFO:tasks.workunit.client.0.smithi089.stdout:removed './236/22' 2023-10-30T12:36:02.091 INFO:tasks.workunit.client.0.smithi089.stdout:removed './236/25' 2023-10-30T12:36:02.091 INFO:tasks.workunit.client.0.smithi089.stdout:removed './236/30' 2023-10-30T12:36:02.091 INFO:tasks.workunit.client.0.smithi089.stdout:removed './236/33' 2023-10-30T12:36:02.091 INFO:tasks.workunit.client.0.smithi089.stdout:removed './236/23' 2023-10-30T12:36:02.092 INFO:tasks.workunit.client.0.smithi089.stdout:removed './236/35' 2023-10-30T12:36:02.092 INFO:tasks.workunit.client.0.smithi089.stdout:removed './236/26' 2023-10-30T12:36:02.092 INFO:tasks.workunit.client.0.smithi089.stdout:removed './236/37' 2023-10-30T12:36:02.092 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './236' 2023-10-30T12:36:02.092 INFO:tasks.workunit.client.0.smithi089.stdout:removed './237/21' 2023-10-30T12:36:02.092 INFO:tasks.workunit.client.0.smithi089.stdout:removed './237/27' 2023-10-30T12:36:02.093 INFO:tasks.workunit.client.0.smithi089.stdout:removed './237/31' 2023-10-30T12:36:02.093 INFO:tasks.workunit.client.0.smithi089.stdout:removed './237/36' 2023-10-30T12:36:02.093 INFO:tasks.workunit.client.0.smithi089.stdout:removed './237/20' 2023-10-30T12:36:02.093 INFO:tasks.workunit.client.0.smithi089.stdout:removed './237/39' 2023-10-30T12:36:02.093 INFO:tasks.workunit.client.0.smithi089.stdout:removed './237/34' 2023-10-30T12:36:02.094 INFO:tasks.workunit.client.0.smithi089.stdout:removed './237/38' 2023-10-30T12:36:02.094 INFO:tasks.workunit.client.0.smithi089.stdout:removed './237/28' 2023-10-30T12:36:02.094 INFO:tasks.workunit.client.0.smithi089.stdout:removed './237/24' 2023-10-30T12:36:02.094 INFO:tasks.workunit.client.0.smithi089.stdout:removed './237/29' 2023-10-30T12:36:02.095 INFO:tasks.workunit.client.0.smithi089.stdout:removed './237/32' 2023-10-30T12:36:02.095 INFO:tasks.workunit.client.0.smithi089.stdout:removed './237/22' 2023-10-30T12:36:02.095 INFO:tasks.workunit.client.0.smithi089.stdout:removed './237/25' 2023-10-30T12:36:02.095 INFO:tasks.workunit.client.0.smithi089.stdout:removed './237/30' 2023-10-30T12:36:02.095 INFO:tasks.workunit.client.0.smithi089.stdout:removed './237/33' 2023-10-30T12:36:02.096 INFO:tasks.workunit.client.0.smithi089.stdout:removed './237/23' 2023-10-30T12:36:02.096 INFO:tasks.workunit.client.0.smithi089.stdout:removed './237/35' 2023-10-30T12:36:02.096 INFO:tasks.workunit.client.0.smithi089.stdout:removed './237/26' 2023-10-30T12:36:02.096 INFO:tasks.workunit.client.0.smithi089.stdout:removed './237/37' 2023-10-30T12:36:02.096 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './237' 2023-10-30T12:36:02.097 INFO:tasks.workunit.client.0.smithi089.stdout:removed './238/21' 2023-10-30T12:36:02.097 INFO:tasks.workunit.client.0.smithi089.stdout:removed './238/27' 2023-10-30T12:36:02.097 INFO:tasks.workunit.client.0.smithi089.stdout:removed './238/31' 2023-10-30T12:36:02.097 INFO:tasks.workunit.client.0.smithi089.stdout:removed './238/36' 2023-10-30T12:36:02.098 INFO:tasks.workunit.client.0.smithi089.stdout:removed './238/20' 2023-10-30T12:36:02.098 INFO:tasks.workunit.client.0.smithi089.stdout:removed './238/39' 2023-10-30T12:36:02.098 INFO:tasks.workunit.client.0.smithi089.stdout:removed './238/34' 2023-10-30T12:36:02.098 INFO:tasks.workunit.client.0.smithi089.stdout:removed './238/38' 2023-10-30T12:36:02.098 INFO:tasks.workunit.client.0.smithi089.stdout:removed './238/28' 2023-10-30T12:36:02.099 INFO:tasks.workunit.client.0.smithi089.stdout:removed './238/24' 2023-10-30T12:36:02.099 INFO:tasks.workunit.client.0.smithi089.stdout:removed './238/29' 2023-10-30T12:36:02.099 INFO:tasks.workunit.client.0.smithi089.stdout:removed './238/32' 2023-10-30T12:36:02.099 INFO:tasks.workunit.client.0.smithi089.stdout:removed './238/22' 2023-10-30T12:36:02.099 INFO:tasks.workunit.client.0.smithi089.stdout:removed './238/25' 2023-10-30T12:36:02.099 INFO:tasks.workunit.client.0.smithi089.stdout:removed './238/30' 2023-10-30T12:36:02.100 INFO:tasks.workunit.client.0.smithi089.stdout:removed './238/33' 2023-10-30T12:36:02.100 INFO:tasks.workunit.client.0.smithi089.stdout:removed './238/23' 2023-10-30T12:36:02.100 INFO:tasks.workunit.client.0.smithi089.stdout:removed './238/35' 2023-10-30T12:36:02.100 INFO:tasks.workunit.client.0.smithi089.stdout:removed './238/26' 2023-10-30T12:36:02.100 INFO:tasks.workunit.client.0.smithi089.stdout:removed './238/37' 2023-10-30T12:36:02.101 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './238' 2023-10-30T12:36:02.101 INFO:tasks.workunit.client.0.smithi089.stdout:removed './239/21' 2023-10-30T12:36:02.101 INFO:tasks.workunit.client.0.smithi089.stdout:removed './239/27' 2023-10-30T12:36:02.101 INFO:tasks.workunit.client.0.smithi089.stdout:removed './239/31' 2023-10-30T12:36:02.101 INFO:tasks.workunit.client.0.smithi089.stdout:removed './239/36' 2023-10-30T12:36:02.101 INFO:tasks.workunit.client.0.smithi089.stdout:removed './239/20' 2023-10-30T12:36:02.102 INFO:tasks.workunit.client.0.smithi089.stdout:removed './239/39' 2023-10-30T12:36:02.102 INFO:tasks.workunit.client.0.smithi089.stdout:removed './239/34' 2023-10-30T12:36:02.102 INFO:tasks.workunit.client.0.smithi089.stdout:removed './239/38' 2023-10-30T12:36:02.102 INFO:tasks.workunit.client.0.smithi089.stdout:removed './239/28' 2023-10-30T12:36:02.102 INFO:tasks.workunit.client.0.smithi089.stdout:removed './239/24' 2023-10-30T12:36:02.103 INFO:tasks.workunit.client.0.smithi089.stdout:removed './239/29' 2023-10-30T12:36:02.103 INFO:tasks.workunit.client.0.smithi089.stdout:removed './239/32' 2023-10-30T12:36:02.103 INFO:tasks.workunit.client.0.smithi089.stdout:removed './239/22' 2023-10-30T12:36:02.103 INFO:tasks.workunit.client.0.smithi089.stdout:removed './239/25' 2023-10-30T12:36:02.103 INFO:tasks.workunit.client.0.smithi089.stdout:removed './239/30' 2023-10-30T12:36:02.103 INFO:tasks.workunit.client.0.smithi089.stdout:removed './239/33' 2023-10-30T12:36:02.104 INFO:tasks.workunit.client.0.smithi089.stdout:removed './239/23' 2023-10-30T12:36:02.104 INFO:tasks.workunit.client.0.smithi089.stdout:removed './239/35' 2023-10-30T12:36:02.104 INFO:tasks.workunit.client.0.smithi089.stdout:removed './239/26' 2023-10-30T12:36:02.104 INFO:tasks.workunit.client.0.smithi089.stdout:removed './239/37' 2023-10-30T12:36:02.104 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './239' 2023-10-30T12:36:02.105 INFO:tasks.workunit.client.0.smithi089.stdout:removed './240/21' 2023-10-30T12:36:02.105 INFO:tasks.workunit.client.0.smithi089.stdout:removed './240/27' 2023-10-30T12:36:02.105 INFO:tasks.workunit.client.0.smithi089.stdout:removed './240/31' 2023-10-30T12:36:02.105 INFO:tasks.workunit.client.0.smithi089.stdout:removed './240/36' 2023-10-30T12:36:02.105 INFO:tasks.workunit.client.0.smithi089.stdout:removed './240/20' 2023-10-30T12:36:02.106 INFO:tasks.workunit.client.0.smithi089.stdout:removed './240/39' 2023-10-30T12:36:02.106 INFO:tasks.workunit.client.0.smithi089.stdout:removed './240/34' 2023-10-30T12:36:02.106 INFO:tasks.workunit.client.0.smithi089.stdout:removed './240/38' 2023-10-30T12:36:02.106 INFO:tasks.workunit.client.0.smithi089.stdout:removed './240/28' 2023-10-30T12:36:02.106 INFO:tasks.workunit.client.0.smithi089.stdout:removed './240/24' 2023-10-30T12:36:02.107 INFO:tasks.workunit.client.0.smithi089.stdout:removed './240/29' 2023-10-30T12:36:02.107 INFO:tasks.workunit.client.0.smithi089.stdout:removed './240/32' 2023-10-30T12:36:02.107 INFO:tasks.workunit.client.0.smithi089.stdout:removed './240/22' 2023-10-30T12:36:02.107 INFO:tasks.workunit.client.0.smithi089.stdout:removed './240/25' 2023-10-30T12:36:02.107 INFO:tasks.workunit.client.0.smithi089.stdout:removed './240/30' 2023-10-30T12:36:02.108 INFO:tasks.workunit.client.0.smithi089.stdout:removed './240/33' 2023-10-30T12:36:02.108 INFO:tasks.workunit.client.0.smithi089.stdout:removed './240/23' 2023-10-30T12:36:02.108 INFO:tasks.workunit.client.0.smithi089.stdout:removed './240/35' 2023-10-30T12:36:02.108 INFO:tasks.workunit.client.0.smithi089.stdout:removed './240/26' 2023-10-30T12:36:02.108 INFO:tasks.workunit.client.0.smithi089.stdout:removed './240/37' 2023-10-30T12:36:02.109 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './240' 2023-10-30T12:36:02.109 INFO:tasks.workunit.client.0.smithi089.stdout:removed './241/21' 2023-10-30T12:36:02.109 INFO:tasks.workunit.client.0.smithi089.stdout:removed './241/27' 2023-10-30T12:36:02.109 INFO:tasks.workunit.client.0.smithi089.stdout:removed './241/31' 2023-10-30T12:36:02.109 INFO:tasks.workunit.client.0.smithi089.stdout:removed './241/36' 2023-10-30T12:36:02.110 INFO:tasks.workunit.client.0.smithi089.stdout:removed './241/20' 2023-10-30T12:36:02.110 INFO:tasks.workunit.client.0.smithi089.stdout:removed './241/39' 2023-10-30T12:36:02.110 INFO:tasks.workunit.client.0.smithi089.stdout:removed './241/34' 2023-10-30T12:36:02.110 INFO:tasks.workunit.client.0.smithi089.stdout:removed './241/38' 2023-10-30T12:36:02.110 INFO:tasks.workunit.client.0.smithi089.stdout:removed './241/28' 2023-10-30T12:36:02.111 INFO:tasks.workunit.client.0.smithi089.stdout:removed './241/24' 2023-10-30T12:36:02.111 INFO:tasks.workunit.client.0.smithi089.stdout:removed './241/29' 2023-10-30T12:36:02.111 INFO:tasks.workunit.client.0.smithi089.stdout:removed './241/32' 2023-10-30T12:36:02.111 INFO:tasks.workunit.client.0.smithi089.stdout:removed './241/22' 2023-10-30T12:36:02.111 INFO:tasks.workunit.client.0.smithi089.stdout:removed './241/25' 2023-10-30T12:36:02.112 INFO:tasks.workunit.client.0.smithi089.stdout:removed './241/30' 2023-10-30T12:36:02.112 INFO:tasks.workunit.client.0.smithi089.stdout:removed './241/33' 2023-10-30T12:36:02.112 INFO:tasks.workunit.client.0.smithi089.stdout:removed './241/23' 2023-10-30T12:36:02.112 INFO:tasks.workunit.client.0.smithi089.stdout:removed './241/35' 2023-10-30T12:36:02.112 INFO:tasks.workunit.client.0.smithi089.stdout:removed './241/26' 2023-10-30T12:36:02.113 INFO:tasks.workunit.client.0.smithi089.stdout:removed './241/37' 2023-10-30T12:36:02.113 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './241' 2023-10-30T12:36:02.113 INFO:tasks.workunit.client.0.smithi089.stdout:removed './242/21' 2023-10-30T12:36:02.113 INFO:tasks.workunit.client.0.smithi089.stdout:removed './242/27' 2023-10-30T12:36:02.113 INFO:tasks.workunit.client.0.smithi089.stdout:removed './242/31' 2023-10-30T12:36:02.113 INFO:tasks.workunit.client.0.smithi089.stdout:removed './242/36' 2023-10-30T12:36:02.114 INFO:tasks.workunit.client.0.smithi089.stdout:removed './242/20' 2023-10-30T12:36:02.114 INFO:tasks.workunit.client.0.smithi089.stdout:removed './242/39' 2023-10-30T12:36:02.114 INFO:tasks.workunit.client.0.smithi089.stdout:removed './242/34' 2023-10-30T12:36:02.114 INFO:tasks.workunit.client.0.smithi089.stdout:removed './242/38' 2023-10-30T12:36:02.115 INFO:tasks.workunit.client.0.smithi089.stdout:removed './242/28' 2023-10-30T12:36:02.115 INFO:tasks.workunit.client.0.smithi089.stdout:removed './242/24' 2023-10-30T12:36:02.115 INFO:tasks.workunit.client.0.smithi089.stdout:removed './242/29' 2023-10-30T12:36:02.115 INFO:tasks.workunit.client.0.smithi089.stdout:removed './242/32' 2023-10-30T12:36:02.115 INFO:tasks.workunit.client.0.smithi089.stdout:removed './242/22' 2023-10-30T12:36:02.115 INFO:tasks.workunit.client.0.smithi089.stdout:removed './242/25' 2023-10-30T12:36:02.116 INFO:tasks.workunit.client.0.smithi089.stdout:removed './242/30' 2023-10-30T12:36:02.116 INFO:tasks.workunit.client.0.smithi089.stdout:removed './242/33' 2023-10-30T12:36:02.116 INFO:tasks.workunit.client.0.smithi089.stdout:removed './242/23' 2023-10-30T12:36:02.116 INFO:tasks.workunit.client.0.smithi089.stdout:removed './242/35' 2023-10-30T12:36:02.116 INFO:tasks.workunit.client.0.smithi089.stdout:removed './242/26' 2023-10-30T12:36:02.117 INFO:tasks.workunit.client.0.smithi089.stdout:removed './242/37' 2023-10-30T12:36:02.117 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './242' 2023-10-30T12:36:02.117 INFO:tasks.workunit.client.0.smithi089.stdout:removed './243/21' 2023-10-30T12:36:02.117 INFO:tasks.workunit.client.0.smithi089.stdout:removed './243/27' 2023-10-30T12:36:02.117 INFO:tasks.workunit.client.0.smithi089.stdout:removed './243/31' 2023-10-30T12:36:02.118 INFO:tasks.workunit.client.0.smithi089.stdout:removed './243/36' 2023-10-30T12:36:02.118 INFO:tasks.workunit.client.0.smithi089.stdout:removed './243/20' 2023-10-30T12:36:02.118 INFO:tasks.workunit.client.0.smithi089.stdout:removed './243/39' 2023-10-30T12:36:02.118 INFO:tasks.workunit.client.0.smithi089.stdout:removed './243/34' 2023-10-30T12:36:02.118 INFO:tasks.workunit.client.0.smithi089.stdout:removed './243/38' 2023-10-30T12:36:02.119 INFO:tasks.workunit.client.0.smithi089.stdout:removed './243/28' 2023-10-30T12:36:02.119 INFO:tasks.workunit.client.0.smithi089.stdout:removed './243/24' 2023-10-30T12:36:02.119 INFO:tasks.workunit.client.0.smithi089.stdout:removed './243/29' 2023-10-30T12:36:02.119 INFO:tasks.workunit.client.0.smithi089.stdout:removed './243/32' 2023-10-30T12:36:02.120 INFO:tasks.workunit.client.0.smithi089.stdout:removed './243/22' 2023-10-30T12:36:02.120 INFO:tasks.workunit.client.0.smithi089.stdout:removed './243/25' 2023-10-30T12:36:02.120 INFO:tasks.workunit.client.0.smithi089.stdout:removed './243/30' 2023-10-30T12:36:02.120 INFO:tasks.workunit.client.0.smithi089.stdout:removed './243/33' 2023-10-30T12:36:02.120 INFO:tasks.workunit.client.0.smithi089.stdout:removed './243/23' 2023-10-30T12:36:02.120 INFO:tasks.workunit.client.0.smithi089.stdout:removed './243/35' 2023-10-30T12:36:02.121 INFO:tasks.workunit.client.0.smithi089.stdout:removed './243/26' 2023-10-30T12:36:02.121 INFO:tasks.workunit.client.0.smithi089.stdout:removed './243/37' 2023-10-30T12:36:02.121 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './243' 2023-10-30T12:36:02.121 INFO:tasks.workunit.client.0.smithi089.stdout:removed './244/21' 2023-10-30T12:36:02.121 INFO:tasks.workunit.client.0.smithi089.stdout:removed './244/27' 2023-10-30T12:36:02.122 INFO:tasks.workunit.client.0.smithi089.stdout:removed './244/31' 2023-10-30T12:36:02.122 INFO:tasks.workunit.client.0.smithi089.stdout:removed './244/36' 2023-10-30T12:36:02.122 INFO:tasks.workunit.client.0.smithi089.stdout:removed './244/20' 2023-10-30T12:36:02.122 INFO:tasks.workunit.client.0.smithi089.stdout:removed './244/39' 2023-10-30T12:36:02.123 INFO:tasks.workunit.client.0.smithi089.stdout:removed './244/34' 2023-10-30T12:36:02.123 INFO:tasks.workunit.client.0.smithi089.stdout:removed './244/38' 2023-10-30T12:36:02.123 INFO:tasks.workunit.client.0.smithi089.stdout:removed './244/28' 2023-10-30T12:36:02.123 INFO:tasks.workunit.client.0.smithi089.stdout:removed './244/24' 2023-10-30T12:36:02.123 INFO:tasks.workunit.client.0.smithi089.stdout:removed './244/29' 2023-10-30T12:36:02.123 INFO:tasks.workunit.client.0.smithi089.stdout:removed './244/32' 2023-10-30T12:36:02.124 INFO:tasks.workunit.client.0.smithi089.stdout:removed './244/22' 2023-10-30T12:36:02.124 INFO:tasks.workunit.client.0.smithi089.stdout:removed './244/25' 2023-10-30T12:36:02.124 INFO:tasks.workunit.client.0.smithi089.stdout:removed './244/30' 2023-10-30T12:36:02.124 INFO:tasks.workunit.client.0.smithi089.stdout:removed './244/33' 2023-10-30T12:36:02.124 INFO:tasks.workunit.client.0.smithi089.stdout:removed './244/23' 2023-10-30T12:36:02.125 INFO:tasks.workunit.client.0.smithi089.stdout:removed './244/35' 2023-10-30T12:36:02.125 INFO:tasks.workunit.client.0.smithi089.stdout:removed './244/26' 2023-10-30T12:36:02.125 INFO:tasks.workunit.client.0.smithi089.stdout:removed './244/37' 2023-10-30T12:36:02.125 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './244' 2023-10-30T12:36:02.125 INFO:tasks.workunit.client.0.smithi089.stdout:removed './245/21' 2023-10-30T12:36:02.126 INFO:tasks.workunit.client.0.smithi089.stdout:removed './245/27' 2023-10-30T12:36:02.126 INFO:tasks.workunit.client.0.smithi089.stdout:removed './245/31' 2023-10-30T12:36:02.126 INFO:tasks.workunit.client.0.smithi089.stdout:removed './245/36' 2023-10-30T12:36:02.126 INFO:tasks.workunit.client.0.smithi089.stdout:removed './245/20' 2023-10-30T12:36:02.126 INFO:tasks.workunit.client.0.smithi089.stdout:removed './245/39' 2023-10-30T12:36:02.127 INFO:tasks.workunit.client.0.smithi089.stdout:removed './245/34' 2023-10-30T12:36:02.127 INFO:tasks.workunit.client.0.smithi089.stdout:removed './245/38' 2023-10-30T12:36:02.127 INFO:tasks.workunit.client.0.smithi089.stdout:removed './245/28' 2023-10-30T12:36:02.127 INFO:tasks.workunit.client.0.smithi089.stdout:removed './245/24' 2023-10-30T12:36:02.127 INFO:tasks.workunit.client.0.smithi089.stdout:removed './245/29' 2023-10-30T12:36:02.128 INFO:tasks.workunit.client.0.smithi089.stdout:removed './245/32' 2023-10-30T12:36:02.128 INFO:tasks.workunit.client.0.smithi089.stdout:removed './245/22' 2023-10-30T12:36:02.128 INFO:tasks.workunit.client.0.smithi089.stdout:removed './245/25' 2023-10-30T12:36:02.128 INFO:tasks.workunit.client.0.smithi089.stdout:removed './245/30' 2023-10-30T12:36:02.128 INFO:tasks.workunit.client.0.smithi089.stdout:removed './245/33' 2023-10-30T12:36:02.129 INFO:tasks.workunit.client.0.smithi089.stdout:removed './245/23' 2023-10-30T12:36:02.129 INFO:tasks.workunit.client.0.smithi089.stdout:removed './245/35' 2023-10-30T12:36:02.129 INFO:tasks.workunit.client.0.smithi089.stdout:removed './245/26' 2023-10-30T12:36:02.129 INFO:tasks.workunit.client.0.smithi089.stdout:removed './245/37' 2023-10-30T12:36:02.129 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './245' 2023-10-30T12:36:02.130 INFO:tasks.workunit.client.0.smithi089.stdout:removed './246/21' 2023-10-30T12:36:02.130 INFO:tasks.workunit.client.0.smithi089.stdout:removed './246/27' 2023-10-30T12:36:02.130 INFO:tasks.workunit.client.0.smithi089.stdout:removed './246/31' 2023-10-30T12:36:02.130 INFO:tasks.workunit.client.0.smithi089.stdout:removed './246/36' 2023-10-30T12:36:02.130 INFO:tasks.workunit.client.0.smithi089.stdout:removed './246/20' 2023-10-30T12:36:02.131 INFO:tasks.workunit.client.0.smithi089.stdout:removed './246/39' 2023-10-30T12:36:02.131 INFO:tasks.workunit.client.0.smithi089.stdout:removed './246/34' 2023-10-30T12:36:02.131 INFO:tasks.workunit.client.0.smithi089.stdout:removed './246/38' 2023-10-30T12:36:02.131 INFO:tasks.workunit.client.0.smithi089.stdout:removed './246/28' 2023-10-30T12:36:02.131 INFO:tasks.workunit.client.0.smithi089.stdout:removed './246/24' 2023-10-30T12:36:02.132 INFO:tasks.workunit.client.0.smithi089.stdout:removed './246/29' 2023-10-30T12:36:02.132 INFO:tasks.workunit.client.0.smithi089.stdout:removed './246/32' 2023-10-30T12:36:02.132 INFO:tasks.workunit.client.0.smithi089.stdout:removed './246/22' 2023-10-30T12:36:06.818 INFO:tasks.workunit.client.0.smithi089.stdout:removed './246/25' 2023-10-30T12:36:06.819 INFO:tasks.workunit.client.0.smithi089.stdout:removed './246/30' 2023-10-30T12:36:06.819 INFO:tasks.workunit.client.0.smithi089.stdout:removed './246/33' 2023-10-30T12:36:06.819 INFO:tasks.workunit.client.0.smithi089.stdout:removed './246/23' 2023-10-30T12:36:06.819 INFO:tasks.workunit.client.0.smithi089.stdout:removed './246/35' 2023-10-30T12:36:06.820 INFO:tasks.workunit.client.0.smithi089.stdout:removed './246/26' 2023-10-30T12:36:06.820 INFO:tasks.workunit.client.0.smithi089.stdout:removed './246/37' 2023-10-30T12:36:06.820 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './246' 2023-10-30T12:36:06.820 INFO:tasks.workunit.client.0.smithi089.stdout:removed './247/21' 2023-10-30T12:36:06.820 INFO:tasks.workunit.client.0.smithi089.stdout:removed './247/27' 2023-10-30T12:36:06.821 INFO:tasks.workunit.client.0.smithi089.stdout:removed './247/31' 2023-10-30T12:36:06.821 INFO:tasks.workunit.client.0.smithi089.stdout:removed './247/36' 2023-10-30T12:36:06.821 INFO:tasks.workunit.client.0.smithi089.stdout:removed './247/20' 2023-10-30T12:36:06.821 INFO:tasks.workunit.client.0.smithi089.stdout:removed './247/39' 2023-10-30T12:36:06.821 INFO:tasks.workunit.client.0.smithi089.stdout:removed './247/34' 2023-10-30T12:36:06.822 INFO:tasks.workunit.client.0.smithi089.stdout:removed './247/38' 2023-10-30T12:36:06.822 INFO:tasks.workunit.client.0.smithi089.stdout:removed './247/28' 2023-10-30T12:36:06.822 INFO:tasks.workunit.client.0.smithi089.stdout:removed './247/24' 2023-10-30T12:36:06.822 INFO:tasks.workunit.client.0.smithi089.stdout:removed './247/29' 2023-10-30T12:36:06.822 INFO:tasks.workunit.client.0.smithi089.stdout:removed './247/32' 2023-10-30T12:36:06.823 INFO:tasks.workunit.client.0.smithi089.stdout:removed './247/22' 2023-10-30T12:36:06.823 INFO:tasks.workunit.client.0.smithi089.stdout:removed './247/25' 2023-10-30T12:36:06.823 INFO:tasks.workunit.client.0.smithi089.stdout:removed './247/30' 2023-10-30T12:36:06.823 INFO:tasks.workunit.client.0.smithi089.stdout:removed './247/33' 2023-10-30T12:36:06.823 INFO:tasks.workunit.client.0.smithi089.stdout:removed './247/23' 2023-10-30T12:36:06.824 INFO:tasks.workunit.client.0.smithi089.stdout:removed './247/35' 2023-10-30T12:36:06.824 INFO:tasks.workunit.client.0.smithi089.stdout:removed './247/26' 2023-10-30T12:36:06.824 INFO:tasks.workunit.client.0.smithi089.stdout:removed './247/37' 2023-10-30T12:36:06.824 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './247' 2023-10-30T12:36:06.824 INFO:tasks.workunit.client.0.smithi089.stdout:removed './248/21' 2023-10-30T12:36:06.825 INFO:tasks.workunit.client.0.smithi089.stdout:removed './248/27' 2023-10-30T12:36:06.825 INFO:tasks.workunit.client.0.smithi089.stdout:removed './248/31' 2023-10-30T12:36:06.825 INFO:tasks.workunit.client.0.smithi089.stdout:removed './248/36' 2023-10-30T12:36:06.825 INFO:tasks.workunit.client.0.smithi089.stdout:removed './248/20' 2023-10-30T12:36:06.825 INFO:tasks.workunit.client.0.smithi089.stdout:removed './248/39' 2023-10-30T12:36:06.825 INFO:tasks.workunit.client.0.smithi089.stdout:removed './248/34' 2023-10-30T12:36:06.826 INFO:tasks.workunit.client.0.smithi089.stdout:removed './248/38' 2023-10-30T12:36:06.826 INFO:tasks.workunit.client.0.smithi089.stdout:removed './248/28' 2023-10-30T12:36:06.826 INFO:tasks.workunit.client.0.smithi089.stdout:removed './248/24' 2023-10-30T12:36:06.826 INFO:tasks.workunit.client.0.smithi089.stdout:removed './248/29' 2023-10-30T12:36:06.826 INFO:tasks.workunit.client.0.smithi089.stdout:removed './248/32' 2023-10-30T12:36:06.827 INFO:tasks.workunit.client.0.smithi089.stdout:removed './248/22' 2023-10-30T12:36:06.827 INFO:tasks.workunit.client.0.smithi089.stdout:removed './248/25' 2023-10-30T12:36:06.827 INFO:tasks.workunit.client.0.smithi089.stdout:removed './248/30' 2023-10-30T12:36:06.827 INFO:tasks.workunit.client.0.smithi089.stdout:removed './248/33' 2023-10-30T12:36:06.827 INFO:tasks.workunit.client.0.smithi089.stdout:removed './248/23' 2023-10-30T12:36:06.828 INFO:tasks.workunit.client.0.smithi089.stdout:removed './248/35' 2023-10-30T12:36:06.828 INFO:tasks.workunit.client.0.smithi089.stdout:removed './248/26' 2023-10-30T12:36:06.828 INFO:tasks.workunit.client.0.smithi089.stdout:removed './248/37' 2023-10-30T12:36:06.828 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './248' 2023-10-30T12:36:06.829 INFO:tasks.workunit.client.0.smithi089.stdout:removed './249/21' 2023-10-30T12:36:06.829 INFO:tasks.workunit.client.0.smithi089.stdout:removed './249/27' 2023-10-30T12:36:06.829 INFO:tasks.workunit.client.0.smithi089.stdout:removed './249/31' 2023-10-30T12:36:06.829 INFO:tasks.workunit.client.0.smithi089.stdout:removed './249/36' 2023-10-30T12:36:06.829 INFO:tasks.workunit.client.0.smithi089.stdout:removed './249/20' 2023-10-30T12:36:06.830 INFO:tasks.workunit.client.0.smithi089.stdout:removed './249/39' 2023-10-30T12:36:06.830 INFO:tasks.workunit.client.0.smithi089.stdout:removed './249/34' 2023-10-30T12:36:06.830 INFO:tasks.workunit.client.0.smithi089.stdout:removed './249/38' 2023-10-30T12:36:06.830 INFO:tasks.workunit.client.0.smithi089.stdout:removed './249/28' 2023-10-30T12:36:06.830 INFO:tasks.workunit.client.0.smithi089.stdout:removed './249/24' 2023-10-30T12:36:06.831 INFO:tasks.workunit.client.0.smithi089.stdout:removed './249/29' 2023-10-30T12:36:06.831 INFO:tasks.workunit.client.0.smithi089.stdout:removed './249/32' 2023-10-30T12:36:06.831 INFO:tasks.workunit.client.0.smithi089.stdout:removed './249/22' 2023-10-30T12:36:06.831 INFO:tasks.workunit.client.0.smithi089.stdout:removed './249/25' 2023-10-30T12:36:06.831 INFO:tasks.workunit.client.0.smithi089.stdout:removed './249/30' 2023-10-30T12:36:06.832 INFO:tasks.workunit.client.0.smithi089.stdout:removed './249/33' 2023-10-30T12:36:06.832 INFO:tasks.workunit.client.0.smithi089.stdout:removed './249/23' 2023-10-30T12:36:06.832 INFO:tasks.workunit.client.0.smithi089.stdout:removed './249/35' 2023-10-30T12:36:06.832 INFO:tasks.workunit.client.0.smithi089.stdout:removed './249/26' 2023-10-30T12:36:06.832 INFO:tasks.workunit.client.0.smithi089.stdout:removed './249/37' 2023-10-30T12:36:06.833 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './249' 2023-10-30T12:36:06.833 INFO:tasks.workunit.client.0.smithi089.stdout:removed './250/21' 2023-10-30T12:36:06.833 INFO:tasks.workunit.client.0.smithi089.stdout:removed './250/27' 2023-10-30T12:36:06.833 INFO:tasks.workunit.client.0.smithi089.stdout:removed './250/31' 2023-10-30T12:36:06.833 INFO:tasks.workunit.client.0.smithi089.stdout:removed './250/36' 2023-10-30T12:36:06.834 INFO:tasks.workunit.client.0.smithi089.stdout:removed './250/20' 2023-10-30T12:36:06.834 INFO:tasks.workunit.client.0.smithi089.stdout:removed './250/39' 2023-10-30T12:36:06.834 INFO:tasks.workunit.client.0.smithi089.stdout:removed './250/34' 2023-10-30T12:36:06.834 INFO:tasks.workunit.client.0.smithi089.stdout:removed './250/38' 2023-10-30T12:36:06.834 INFO:tasks.workunit.client.0.smithi089.stdout:removed './250/28' 2023-10-30T12:36:06.835 INFO:tasks.workunit.client.0.smithi089.stdout:removed './250/24' 2023-10-30T12:36:06.835 INFO:tasks.workunit.client.0.smithi089.stdout:removed './250/29' 2023-10-30T12:36:06.835 INFO:tasks.workunit.client.0.smithi089.stdout:removed './250/32' 2023-10-30T12:36:06.835 INFO:tasks.workunit.client.0.smithi089.stdout:removed './250/22' 2023-10-30T12:36:06.835 INFO:tasks.workunit.client.0.smithi089.stdout:removed './250/25' 2023-10-30T12:36:06.836 INFO:tasks.workunit.client.0.smithi089.stdout:removed './250/30' 2023-10-30T12:36:06.836 INFO:tasks.workunit.client.0.smithi089.stdout:removed './250/33' 2023-10-30T12:36:06.836 INFO:tasks.workunit.client.0.smithi089.stdout:removed './250/23' 2023-10-30T12:36:06.836 INFO:tasks.workunit.client.0.smithi089.stdout:removed './250/35' 2023-10-30T12:36:06.836 INFO:tasks.workunit.client.0.smithi089.stdout:removed './250/26' 2023-10-30T12:36:06.836 INFO:tasks.workunit.client.0.smithi089.stdout:removed './250/37' 2023-10-30T12:36:06.837 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './250' 2023-10-30T12:36:06.837 INFO:tasks.workunit.client.0.smithi089.stdout:removed './251/21' 2023-10-30T12:36:06.837 INFO:tasks.workunit.client.0.smithi089.stdout:removed './251/27' 2023-10-30T12:36:06.837 INFO:tasks.workunit.client.0.smithi089.stdout:removed './251/31' 2023-10-30T12:36:06.837 INFO:tasks.workunit.client.0.smithi089.stdout:removed './251/36' 2023-10-30T12:36:06.838 INFO:tasks.workunit.client.0.smithi089.stdout:removed './251/20' 2023-10-30T12:36:06.838 INFO:tasks.workunit.client.0.smithi089.stdout:removed './251/39' 2023-10-30T12:36:06.838 INFO:tasks.workunit.client.0.smithi089.stdout:removed './251/34' 2023-10-30T12:36:06.838 INFO:tasks.workunit.client.0.smithi089.stdout:removed './251/38' 2023-10-30T12:36:06.838 INFO:tasks.workunit.client.0.smithi089.stdout:removed './251/28' 2023-10-30T12:36:06.839 INFO:tasks.workunit.client.0.smithi089.stdout:removed './251/24' 2023-10-30T12:36:06.839 INFO:tasks.workunit.client.0.smithi089.stdout:removed './251/29' 2023-10-30T12:36:06.839 INFO:tasks.workunit.client.0.smithi089.stdout:removed './251/32' 2023-10-30T12:36:06.839 INFO:tasks.workunit.client.0.smithi089.stdout:removed './251/22' 2023-10-30T12:36:06.839 INFO:tasks.workunit.client.0.smithi089.stdout:removed './251/25' 2023-10-30T12:36:06.840 INFO:tasks.workunit.client.0.smithi089.stdout:removed './251/30' 2023-10-30T12:36:06.840 INFO:tasks.workunit.client.0.smithi089.stdout:removed './251/33' 2023-10-30T12:36:06.840 INFO:tasks.workunit.client.0.smithi089.stdout:removed './251/23' 2023-10-30T12:36:06.840 INFO:tasks.workunit.client.0.smithi089.stdout:removed './251/35' 2023-10-30T12:36:06.840 INFO:tasks.workunit.client.0.smithi089.stdout:removed './251/26' 2023-10-30T12:36:06.841 INFO:tasks.workunit.client.0.smithi089.stdout:removed './251/37' 2023-10-30T12:36:06.841 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './251' 2023-10-30T12:36:06.841 INFO:tasks.workunit.client.0.smithi089.stdout:removed './252/21' 2023-10-30T12:36:06.841 INFO:tasks.workunit.client.0.smithi089.stdout:removed './252/27' 2023-10-30T12:36:06.841 INFO:tasks.workunit.client.0.smithi089.stdout:removed './252/31' 2023-10-30T12:36:06.842 INFO:tasks.workunit.client.0.smithi089.stdout:removed './252/36' 2023-10-30T12:36:06.842 INFO:tasks.workunit.client.0.smithi089.stdout:removed './252/20' 2023-10-30T12:36:06.842 INFO:tasks.workunit.client.0.smithi089.stdout:removed './252/39' 2023-10-30T12:36:06.842 INFO:tasks.workunit.client.0.smithi089.stdout:removed './252/34' 2023-10-30T12:36:06.842 INFO:tasks.workunit.client.0.smithi089.stdout:removed './252/38' 2023-10-30T12:36:06.843 INFO:tasks.workunit.client.0.smithi089.stdout:removed './252/28' 2023-10-30T12:36:06.843 INFO:tasks.workunit.client.0.smithi089.stdout:removed './252/24' 2023-10-30T12:36:06.843 INFO:tasks.workunit.client.0.smithi089.stdout:removed './252/29' 2023-10-30T12:36:06.843 INFO:tasks.workunit.client.0.smithi089.stdout:removed './252/32' 2023-10-30T12:36:06.843 INFO:tasks.workunit.client.0.smithi089.stdout:removed './252/22' 2023-10-30T12:36:06.843 INFO:tasks.workunit.client.0.smithi089.stdout:removed './252/25' 2023-10-30T12:36:06.844 INFO:tasks.workunit.client.0.smithi089.stdout:removed './252/30' 2023-10-30T12:36:06.844 INFO:tasks.workunit.client.0.smithi089.stdout:removed './252/33' 2023-10-30T12:36:06.844 INFO:tasks.workunit.client.0.smithi089.stdout:removed './252/23' 2023-10-30T12:36:06.844 INFO:tasks.workunit.client.0.smithi089.stdout:removed './252/35' 2023-10-30T12:36:06.844 INFO:tasks.workunit.client.0.smithi089.stdout:removed './252/26' 2023-10-30T12:36:06.845 INFO:tasks.workunit.client.0.smithi089.stdout:removed './252/37' 2023-10-30T12:36:06.845 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './252' 2023-10-30T12:36:06.845 INFO:tasks.workunit.client.0.smithi089.stdout:removed './253/21' 2023-10-30T12:36:06.845 INFO:tasks.workunit.client.0.smithi089.stdout:removed './253/27' 2023-10-30T12:36:06.845 INFO:tasks.workunit.client.0.smithi089.stdout:removed './253/31' 2023-10-30T12:36:06.846 INFO:tasks.workunit.client.0.smithi089.stdout:removed './253/36' 2023-10-30T12:36:06.846 INFO:tasks.workunit.client.0.smithi089.stdout:removed './253/20' 2023-10-30T12:36:06.846 INFO:tasks.workunit.client.0.smithi089.stdout:removed './253/39' 2023-10-30T12:36:06.846 INFO:tasks.workunit.client.0.smithi089.stdout:removed './253/34' 2023-10-30T12:36:06.846 INFO:tasks.workunit.client.0.smithi089.stdout:removed './253/38' 2023-10-30T12:36:06.847 INFO:tasks.workunit.client.0.smithi089.stdout:removed './253/28' 2023-10-30T12:36:06.847 INFO:tasks.workunit.client.0.smithi089.stdout:removed './253/24' 2023-10-30T12:36:06.847 INFO:tasks.workunit.client.0.smithi089.stdout:removed './253/29' 2023-10-30T12:36:06.847 INFO:tasks.workunit.client.0.smithi089.stdout:removed './253/32' 2023-10-30T12:36:06.847 INFO:tasks.workunit.client.0.smithi089.stdout:removed './253/22' 2023-10-30T12:36:06.848 INFO:tasks.workunit.client.0.smithi089.stdout:removed './253/25' 2023-10-30T12:36:06.848 INFO:tasks.workunit.client.0.smithi089.stdout:removed './253/30' 2023-10-30T12:36:06.848 INFO:tasks.workunit.client.0.smithi089.stdout:removed './253/33' 2023-10-30T12:36:06.848 INFO:tasks.workunit.client.0.smithi089.stdout:removed './253/23' 2023-10-30T12:36:06.848 INFO:tasks.workunit.client.0.smithi089.stdout:removed './253/35' 2023-10-30T12:36:06.849 INFO:tasks.workunit.client.0.smithi089.stdout:removed './253/26' 2023-10-30T12:36:06.849 INFO:tasks.workunit.client.0.smithi089.stdout:removed './253/37' 2023-10-30T12:36:06.849 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './253' 2023-10-30T12:36:06.849 INFO:tasks.workunit.client.0.smithi089.stdout:removed './254/21' 2023-10-30T12:36:06.849 INFO:tasks.workunit.client.0.smithi089.stdout:removed './254/27' 2023-10-30T12:36:06.850 INFO:tasks.workunit.client.0.smithi089.stdout:removed './254/31' 2023-10-30T12:36:06.850 INFO:tasks.workunit.client.0.smithi089.stdout:removed './254/36' 2023-10-30T12:36:06.850 INFO:tasks.workunit.client.0.smithi089.stdout:removed './254/20' 2023-10-30T12:36:06.850 INFO:tasks.workunit.client.0.smithi089.stdout:removed './254/39' 2023-10-30T12:36:06.850 INFO:tasks.workunit.client.0.smithi089.stdout:removed './254/34' 2023-10-30T12:36:06.851 INFO:tasks.workunit.client.0.smithi089.stdout:removed './254/38' 2023-10-30T12:36:06.851 INFO:tasks.workunit.client.0.smithi089.stdout:removed './254/28' 2023-10-30T12:36:06.851 INFO:tasks.workunit.client.0.smithi089.stdout:removed './254/24' 2023-10-30T12:36:06.851 INFO:tasks.workunit.client.0.smithi089.stdout:removed './254/29' 2023-10-30T12:36:06.851 INFO:tasks.workunit.client.0.smithi089.stdout:removed './254/32' 2023-10-30T12:36:06.852 INFO:tasks.workunit.client.0.smithi089.stdout:removed './254/22' 2023-10-30T12:36:06.852 INFO:tasks.workunit.client.0.smithi089.stdout:removed './254/25' 2023-10-30T12:36:06.852 INFO:tasks.workunit.client.0.smithi089.stdout:removed './254/30' 2023-10-30T12:36:06.852 INFO:tasks.workunit.client.0.smithi089.stdout:removed './254/33' 2023-10-30T12:36:06.852 INFO:tasks.workunit.client.0.smithi089.stdout:removed './254/23' 2023-10-30T12:36:06.853 INFO:tasks.workunit.client.0.smithi089.stdout:removed './254/35' 2023-10-30T12:36:06.853 INFO:tasks.workunit.client.0.smithi089.stdout:removed './254/26' 2023-10-30T12:36:06.853 INFO:tasks.workunit.client.0.smithi089.stdout:removed './254/37' 2023-10-30T12:36:06.853 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './254' 2023-10-30T12:36:06.853 INFO:tasks.workunit.client.0.smithi089.stdout:removed './255/21' 2023-10-30T12:36:06.854 INFO:tasks.workunit.client.0.smithi089.stdout:removed './255/27' 2023-10-30T12:36:06.854 INFO:tasks.workunit.client.0.smithi089.stdout:removed './255/31' 2023-10-30T12:36:06.854 INFO:tasks.workunit.client.0.smithi089.stdout:removed './255/36' 2023-10-30T12:36:06.854 INFO:tasks.workunit.client.0.smithi089.stdout:removed './255/20' 2023-10-30T12:36:06.854 INFO:tasks.workunit.client.0.smithi089.stdout:removed './255/39' 2023-10-30T12:36:06.855 INFO:tasks.workunit.client.0.smithi089.stdout:removed './255/34' 2023-10-30T12:36:06.855 INFO:tasks.workunit.client.0.smithi089.stdout:removed './255/38' 2023-10-30T12:36:06.855 INFO:tasks.workunit.client.0.smithi089.stdout:removed './255/28' 2023-10-30T12:36:06.855 INFO:tasks.workunit.client.0.smithi089.stdout:removed './255/24' 2023-10-30T12:36:06.855 INFO:tasks.workunit.client.0.smithi089.stdout:removed './255/29' 2023-10-30T12:36:06.856 INFO:tasks.workunit.client.0.smithi089.stdout:removed './255/32' 2023-10-30T12:36:06.856 INFO:tasks.workunit.client.0.smithi089.stdout:removed './255/22' 2023-10-30T12:36:06.856 INFO:tasks.workunit.client.0.smithi089.stdout:removed './255/25' 2023-10-30T12:36:06.856 INFO:tasks.workunit.client.0.smithi089.stdout:removed './255/30' 2023-10-30T12:36:06.856 INFO:tasks.workunit.client.0.smithi089.stdout:removed './255/33' 2023-10-30T12:36:06.857 INFO:tasks.workunit.client.0.smithi089.stdout:removed './255/23' 2023-10-30T12:36:06.857 INFO:tasks.workunit.client.0.smithi089.stdout:removed './255/35' 2023-10-30T12:36:06.857 INFO:tasks.workunit.client.0.smithi089.stdout:removed './255/26' 2023-10-30T12:36:06.857 INFO:tasks.workunit.client.0.smithi089.stdout:removed './255/37' 2023-10-30T12:36:06.857 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './255' 2023-10-30T12:36:06.858 INFO:tasks.workunit.client.0.smithi089.stdout:removed './256/21' 2023-10-30T12:36:06.858 INFO:tasks.workunit.client.0.smithi089.stdout:removed './256/27' 2023-10-30T12:36:06.858 INFO:tasks.workunit.client.0.smithi089.stdout:removed './256/31' 2023-10-30T12:36:06.858 INFO:tasks.workunit.client.0.smithi089.stdout:removed './256/36' 2023-10-30T12:36:06.859 INFO:tasks.workunit.client.0.smithi089.stdout:removed './256/20' 2023-10-30T12:36:06.859 INFO:tasks.workunit.client.0.smithi089.stdout:removed './256/39' 2023-10-30T12:36:06.859 INFO:tasks.workunit.client.0.smithi089.stdout:removed './256/34' 2023-10-30T12:36:06.859 INFO:tasks.workunit.client.0.smithi089.stdout:removed './256/38' 2023-10-30T12:36:06.859 INFO:tasks.workunit.client.0.smithi089.stdout:removed './256/28' 2023-10-30T12:36:06.860 INFO:tasks.workunit.client.0.smithi089.stdout:removed './256/24' 2023-10-30T12:36:06.860 INFO:tasks.workunit.client.0.smithi089.stdout:removed './256/29' 2023-10-30T12:36:06.860 INFO:tasks.workunit.client.0.smithi089.stdout:removed './256/32' 2023-10-30T12:36:06.860 INFO:tasks.workunit.client.0.smithi089.stdout:removed './256/22' 2023-10-30T12:36:06.860 INFO:tasks.workunit.client.0.smithi089.stdout:removed './256/25' 2023-10-30T12:36:06.861 INFO:tasks.workunit.client.0.smithi089.stdout:removed './256/30' 2023-10-30T12:36:15.914 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:36:15.919 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:36:17.776 INFO:tasks.workunit.client.0.smithi089.stdout:removed './256/33' 2023-10-30T12:36:17.776 INFO:tasks.workunit.client.0.smithi089.stdout:removed './256/23' 2023-10-30T12:36:17.777 INFO:tasks.workunit.client.0.smithi089.stdout:removed './256/35' 2023-10-30T12:36:17.777 INFO:tasks.workunit.client.0.smithi089.stdout:removed './256/26' 2023-10-30T12:36:17.777 INFO:tasks.workunit.client.0.smithi089.stdout:removed './256/37' 2023-10-30T12:36:17.777 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './256' 2023-10-30T12:36:17.777 INFO:tasks.workunit.client.0.smithi089.stdout:removed './257/21' 2023-10-30T12:36:17.778 INFO:tasks.workunit.client.0.smithi089.stdout:removed './257/27' 2023-10-30T12:36:17.778 INFO:tasks.workunit.client.0.smithi089.stdout:removed './257/31' 2023-10-30T12:36:17.778 INFO:tasks.workunit.client.0.smithi089.stdout:removed './257/36' 2023-10-30T12:36:17.778 INFO:tasks.workunit.client.0.smithi089.stdout:removed './257/20' 2023-10-30T12:36:17.778 INFO:tasks.workunit.client.0.smithi089.stdout:removed './257/39' 2023-10-30T12:36:17.779 INFO:tasks.workunit.client.0.smithi089.stdout:removed './257/34' 2023-10-30T12:36:17.779 INFO:tasks.workunit.client.0.smithi089.stdout:removed './257/38' 2023-10-30T12:36:17.779 INFO:tasks.workunit.client.0.smithi089.stdout:removed './257/28' 2023-10-30T12:36:17.779 INFO:tasks.workunit.client.0.smithi089.stdout:removed './257/24' 2023-10-30T12:36:17.779 INFO:tasks.workunit.client.0.smithi089.stdout:removed './257/29' 2023-10-30T12:36:17.780 INFO:tasks.workunit.client.0.smithi089.stdout:removed './257/32' 2023-10-30T12:36:17.780 INFO:tasks.workunit.client.0.smithi089.stdout:removed './257/22' 2023-10-30T12:36:17.780 INFO:tasks.workunit.client.0.smithi089.stdout:removed './257/25' 2023-10-30T12:36:17.780 INFO:tasks.workunit.client.0.smithi089.stdout:removed './257/30' 2023-10-30T12:36:17.780 INFO:tasks.workunit.client.0.smithi089.stdout:removed './257/33' 2023-10-30T12:36:17.781 INFO:tasks.workunit.client.0.smithi089.stdout:removed './257/23' 2023-10-30T12:36:17.781 INFO:tasks.workunit.client.0.smithi089.stdout:removed './257/35' 2023-10-30T12:36:17.781 INFO:tasks.workunit.client.0.smithi089.stdout:removed './257/26' 2023-10-30T12:36:17.781 INFO:tasks.workunit.client.0.smithi089.stdout:removed './257/37' 2023-10-30T12:36:17.781 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './257' 2023-10-30T12:36:17.782 INFO:tasks.workunit.client.0.smithi089.stdout:removed './258/21' 2023-10-30T12:36:17.782 INFO:tasks.workunit.client.0.smithi089.stdout:removed './258/27' 2023-10-30T12:36:17.782 INFO:tasks.workunit.client.0.smithi089.stdout:removed './258/31' 2023-10-30T12:36:17.782 INFO:tasks.workunit.client.0.smithi089.stdout:removed './258/36' 2023-10-30T12:36:17.782 INFO:tasks.workunit.client.0.smithi089.stdout:removed './258/20' 2023-10-30T12:36:17.783 INFO:tasks.workunit.client.0.smithi089.stdout:removed './258/39' 2023-10-30T12:36:17.783 INFO:tasks.workunit.client.0.smithi089.stdout:removed './258/34' 2023-10-30T12:36:17.783 INFO:tasks.workunit.client.0.smithi089.stdout:removed './258/38' 2023-10-30T12:36:17.783 INFO:tasks.workunit.client.0.smithi089.stdout:removed './258/28' 2023-10-30T12:36:17.783 INFO:tasks.workunit.client.0.smithi089.stdout:removed './258/24' 2023-10-30T12:36:17.783 INFO:tasks.workunit.client.0.smithi089.stdout:removed './258/29' 2023-10-30T12:36:17.784 INFO:tasks.workunit.client.0.smithi089.stdout:removed './258/32' 2023-10-30T12:36:17.784 INFO:tasks.workunit.client.0.smithi089.stdout:removed './258/22' 2023-10-30T12:36:17.784 INFO:tasks.workunit.client.0.smithi089.stdout:removed './258/25' 2023-10-30T12:36:17.784 INFO:tasks.workunit.client.0.smithi089.stdout:removed './258/30' 2023-10-30T12:36:17.784 INFO:tasks.workunit.client.0.smithi089.stdout:removed './258/33' 2023-10-30T12:36:17.785 INFO:tasks.workunit.client.0.smithi089.stdout:removed './258/23' 2023-10-30T12:36:17.785 INFO:tasks.workunit.client.0.smithi089.stdout:removed './258/35' 2023-10-30T12:36:17.785 INFO:tasks.workunit.client.0.smithi089.stdout:removed './258/26' 2023-10-30T12:36:17.785 INFO:tasks.workunit.client.0.smithi089.stdout:removed './258/37' 2023-10-30T12:36:17.785 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './258' 2023-10-30T12:36:17.786 INFO:tasks.workunit.client.0.smithi089.stdout:removed './259/21' 2023-10-30T12:36:17.786 INFO:tasks.workunit.client.0.smithi089.stdout:removed './259/27' 2023-10-30T12:36:17.786 INFO:tasks.workunit.client.0.smithi089.stdout:removed './259/31' 2023-10-30T12:36:17.786 INFO:tasks.workunit.client.0.smithi089.stdout:removed './259/36' 2023-10-30T12:36:17.786 INFO:tasks.workunit.client.0.smithi089.stdout:removed './259/20' 2023-10-30T12:36:17.787 INFO:tasks.workunit.client.0.smithi089.stdout:removed './259/39' 2023-10-30T12:36:17.787 INFO:tasks.workunit.client.0.smithi089.stdout:removed './259/34' 2023-10-30T12:36:17.787 INFO:tasks.workunit.client.0.smithi089.stdout:removed './259/38' 2023-10-30T12:36:17.787 INFO:tasks.workunit.client.0.smithi089.stdout:removed './259/28' 2023-10-30T12:36:17.787 INFO:tasks.workunit.client.0.smithi089.stdout:removed './259/24' 2023-10-30T12:36:17.787 INFO:tasks.workunit.client.0.smithi089.stdout:removed './259/29' 2023-10-30T12:36:17.788 INFO:tasks.workunit.client.0.smithi089.stdout:removed './259/32' 2023-10-30T12:36:17.788 INFO:tasks.workunit.client.0.smithi089.stdout:removed './259/22' 2023-10-30T12:36:17.788 INFO:tasks.workunit.client.0.smithi089.stdout:removed './259/25' 2023-10-30T12:36:17.788 INFO:tasks.workunit.client.0.smithi089.stdout:removed './259/30' 2023-10-30T12:36:17.788 INFO:tasks.workunit.client.0.smithi089.stdout:removed './259/33' 2023-10-30T12:36:17.789 INFO:tasks.workunit.client.0.smithi089.stdout:removed './259/23' 2023-10-30T12:36:17.789 INFO:tasks.workunit.client.0.smithi089.stdout:removed './259/35' 2023-10-30T12:36:17.789 INFO:tasks.workunit.client.0.smithi089.stdout:removed './259/26' 2023-10-30T12:36:17.789 INFO:tasks.workunit.client.0.smithi089.stdout:removed './259/37' 2023-10-30T12:36:17.789 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './259' 2023-10-30T12:36:17.790 INFO:tasks.workunit.client.0.smithi089.stdout:removed './260/21' 2023-10-30T12:36:17.790 INFO:tasks.workunit.client.0.smithi089.stdout:removed './260/27' 2023-10-30T12:36:17.790 INFO:tasks.workunit.client.0.smithi089.stdout:removed './260/31' 2023-10-30T12:36:17.790 INFO:tasks.workunit.client.0.smithi089.stdout:removed './260/36' 2023-10-30T12:36:17.790 INFO:tasks.workunit.client.0.smithi089.stdout:removed './260/20' 2023-10-30T12:36:17.791 INFO:tasks.workunit.client.0.smithi089.stdout:removed './260/39' 2023-10-30T12:36:17.791 INFO:tasks.workunit.client.0.smithi089.stdout:removed './260/34' 2023-10-30T12:36:17.791 INFO:tasks.workunit.client.0.smithi089.stdout:removed './260/38' 2023-10-30T12:36:17.791 INFO:tasks.workunit.client.0.smithi089.stdout:removed './260/28' 2023-10-30T12:36:17.791 INFO:tasks.workunit.client.0.smithi089.stdout:removed './260/24' 2023-10-30T12:36:17.792 INFO:tasks.workunit.client.0.smithi089.stdout:removed './260/29' 2023-10-30T12:36:17.792 INFO:tasks.workunit.client.0.smithi089.stdout:removed './260/32' 2023-10-30T12:36:17.792 INFO:tasks.workunit.client.0.smithi089.stdout:removed './260/22' 2023-10-30T12:36:17.792 INFO:tasks.workunit.client.0.smithi089.stdout:removed './260/25' 2023-10-30T12:36:17.792 INFO:tasks.workunit.client.0.smithi089.stdout:removed './260/30' 2023-10-30T12:36:17.793 INFO:tasks.workunit.client.0.smithi089.stdout:removed './260/33' 2023-10-30T12:36:17.793 INFO:tasks.workunit.client.0.smithi089.stdout:removed './260/23' 2023-10-30T12:36:17.793 INFO:tasks.workunit.client.0.smithi089.stdout:removed './260/35' 2023-10-30T12:36:17.793 INFO:tasks.workunit.client.0.smithi089.stdout:removed './260/26' 2023-10-30T12:36:17.793 INFO:tasks.workunit.client.0.smithi089.stdout:removed './260/37' 2023-10-30T12:36:17.793 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './260' 2023-10-30T12:36:17.794 INFO:tasks.workunit.client.0.smithi089.stdout:removed './261/21' 2023-10-30T12:36:17.794 INFO:tasks.workunit.client.0.smithi089.stdout:removed './261/27' 2023-10-30T12:36:17.794 INFO:tasks.workunit.client.0.smithi089.stdout:removed './261/31' 2023-10-30T12:36:17.794 INFO:tasks.workunit.client.0.smithi089.stdout:removed './261/36' 2023-10-30T12:36:17.794 INFO:tasks.workunit.client.0.smithi089.stdout:removed './261/20' 2023-10-30T12:36:17.795 INFO:tasks.workunit.client.0.smithi089.stdout:removed './261/39' 2023-10-30T12:36:17.795 INFO:tasks.workunit.client.0.smithi089.stdout:removed './261/34' 2023-10-30T12:36:17.795 INFO:tasks.workunit.client.0.smithi089.stdout:removed './261/38' 2023-10-30T12:36:17.795 INFO:tasks.workunit.client.0.smithi089.stdout:removed './261/28' 2023-10-30T12:36:17.795 INFO:tasks.workunit.client.0.smithi089.stdout:removed './261/24' 2023-10-30T12:36:17.796 INFO:tasks.workunit.client.0.smithi089.stdout:removed './261/29' 2023-10-30T12:36:17.796 INFO:tasks.workunit.client.0.smithi089.stdout:removed './261/32' 2023-10-30T12:36:17.796 INFO:tasks.workunit.client.0.smithi089.stdout:removed './261/22' 2023-10-30T12:36:17.796 INFO:tasks.workunit.client.0.smithi089.stdout:removed './261/25' 2023-10-30T12:36:17.796 INFO:tasks.workunit.client.0.smithi089.stdout:removed './261/30' 2023-10-30T12:36:17.797 INFO:tasks.workunit.client.0.smithi089.stdout:removed './261/33' 2023-10-30T12:36:17.797 INFO:tasks.workunit.client.0.smithi089.stdout:removed './261/23' 2023-10-30T12:36:17.797 INFO:tasks.workunit.client.0.smithi089.stdout:removed './261/35' 2023-10-30T12:36:17.797 INFO:tasks.workunit.client.0.smithi089.stdout:removed './261/26' 2023-10-30T12:36:17.797 INFO:tasks.workunit.client.0.smithi089.stdout:removed './261/37' 2023-10-30T12:36:17.798 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './261' 2023-10-30T12:36:17.798 INFO:tasks.workunit.client.0.smithi089.stdout:removed './262/21' 2023-10-30T12:36:17.798 INFO:tasks.workunit.client.0.smithi089.stdout:removed './262/27' 2023-10-30T12:36:17.798 INFO:tasks.workunit.client.0.smithi089.stdout:removed './262/31' 2023-10-30T12:36:17.798 INFO:tasks.workunit.client.0.smithi089.stdout:removed './262/36' 2023-10-30T12:36:17.799 INFO:tasks.workunit.client.0.smithi089.stdout:removed './262/20' 2023-10-30T12:36:17.799 INFO:tasks.workunit.client.0.smithi089.stdout:removed './262/39' 2023-10-30T12:36:17.799 INFO:tasks.workunit.client.0.smithi089.stdout:removed './262/34' 2023-10-30T12:36:17.799 INFO:tasks.workunit.client.0.smithi089.stdout:removed './262/38' 2023-10-30T12:36:17.799 INFO:tasks.workunit.client.0.smithi089.stdout:removed './262/28' 2023-10-30T12:36:17.800 INFO:tasks.workunit.client.0.smithi089.stdout:removed './262/24' 2023-10-30T12:36:17.800 INFO:tasks.workunit.client.0.smithi089.stdout:removed './262/29' 2023-10-30T12:36:17.800 INFO:tasks.workunit.client.0.smithi089.stdout:removed './262/32' 2023-10-30T12:36:17.800 INFO:tasks.workunit.client.0.smithi089.stdout:removed './262/22' 2023-10-30T12:36:17.800 INFO:tasks.workunit.client.0.smithi089.stdout:removed './262/25' 2023-10-30T12:36:17.800 INFO:tasks.workunit.client.0.smithi089.stdout:removed './262/30' 2023-10-30T12:36:17.801 INFO:tasks.workunit.client.0.smithi089.stdout:removed './262/33' 2023-10-30T12:36:17.801 INFO:tasks.workunit.client.0.smithi089.stdout:removed './262/23' 2023-10-30T12:36:17.801 INFO:tasks.workunit.client.0.smithi089.stdout:removed './262/35' 2023-10-30T12:36:17.801 INFO:tasks.workunit.client.0.smithi089.stdout:removed './262/26' 2023-10-30T12:36:17.801 INFO:tasks.workunit.client.0.smithi089.stdout:removed './262/37' 2023-10-30T12:36:17.802 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './262' 2023-10-30T12:36:17.802 INFO:tasks.workunit.client.0.smithi089.stdout:removed './263/21' 2023-10-30T12:36:17.802 INFO:tasks.workunit.client.0.smithi089.stdout:removed './263/27' 2023-10-30T12:36:17.802 INFO:tasks.workunit.client.0.smithi089.stdout:removed './263/31' 2023-10-30T12:36:17.802 INFO:tasks.workunit.client.0.smithi089.stdout:removed './263/36' 2023-10-30T12:36:17.803 INFO:tasks.workunit.client.0.smithi089.stdout:removed './263/20' 2023-10-30T12:36:17.803 INFO:tasks.workunit.client.0.smithi089.stdout:removed './263/39' 2023-10-30T12:36:17.803 INFO:tasks.workunit.client.0.smithi089.stdout:removed './263/34' 2023-10-30T12:36:17.803 INFO:tasks.workunit.client.0.smithi089.stdout:removed './263/38' 2023-10-30T12:36:17.803 INFO:tasks.workunit.client.0.smithi089.stdout:removed './263/28' 2023-10-30T12:36:17.804 INFO:tasks.workunit.client.0.smithi089.stdout:removed './263/24' 2023-10-30T12:36:17.804 INFO:tasks.workunit.client.0.smithi089.stdout:removed './263/29' 2023-10-30T12:36:17.804 INFO:tasks.workunit.client.0.smithi089.stdout:removed './263/32' 2023-10-30T12:36:17.804 INFO:tasks.workunit.client.0.smithi089.stdout:removed './263/22' 2023-10-30T12:36:17.804 INFO:tasks.workunit.client.0.smithi089.stdout:removed './263/25' 2023-10-30T12:36:17.804 INFO:tasks.workunit.client.0.smithi089.stdout:removed './263/30' 2023-10-30T12:36:17.805 INFO:tasks.workunit.client.0.smithi089.stdout:removed './263/33' 2023-10-30T12:36:17.805 INFO:tasks.workunit.client.0.smithi089.stdout:removed './263/23' 2023-10-30T12:36:17.805 INFO:tasks.workunit.client.0.smithi089.stdout:removed './263/35' 2023-10-30T12:36:17.805 INFO:tasks.workunit.client.0.smithi089.stdout:removed './263/26' 2023-10-30T12:36:17.805 INFO:tasks.workunit.client.0.smithi089.stdout:removed './263/37' 2023-10-30T12:36:17.806 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './263' 2023-10-30T12:36:17.806 INFO:tasks.workunit.client.0.smithi089.stdout:removed './264/21' 2023-10-30T12:36:17.806 INFO:tasks.workunit.client.0.smithi089.stdout:removed './264/27' 2023-10-30T12:36:17.806 INFO:tasks.workunit.client.0.smithi089.stdout:removed './264/31' 2023-10-30T12:36:17.806 INFO:tasks.workunit.client.0.smithi089.stdout:removed './264/36' 2023-10-30T12:36:17.807 INFO:tasks.workunit.client.0.smithi089.stdout:removed './264/20' 2023-10-30T12:36:17.807 INFO:tasks.workunit.client.0.smithi089.stdout:removed './264/39' 2023-10-30T12:36:17.807 INFO:tasks.workunit.client.0.smithi089.stdout:removed './264/34' 2023-10-30T12:36:17.807 INFO:tasks.workunit.client.0.smithi089.stdout:removed './264/38' 2023-10-30T12:36:17.807 INFO:tasks.workunit.client.0.smithi089.stdout:removed './264/28' 2023-10-30T12:36:17.807 INFO:tasks.workunit.client.0.smithi089.stdout:removed './264/24' 2023-10-30T12:36:17.808 INFO:tasks.workunit.client.0.smithi089.stdout:removed './264/29' 2023-10-30T12:36:17.808 INFO:tasks.workunit.client.0.smithi089.stdout:removed './264/32' 2023-10-30T12:36:17.808 INFO:tasks.workunit.client.0.smithi089.stdout:removed './264/22' 2023-10-30T12:36:17.808 INFO:tasks.workunit.client.0.smithi089.stdout:removed './264/25' 2023-10-30T12:36:17.808 INFO:tasks.workunit.client.0.smithi089.stdout:removed './264/30' 2023-10-30T12:36:17.809 INFO:tasks.workunit.client.0.smithi089.stdout:removed './264/33' 2023-10-30T12:36:17.809 INFO:tasks.workunit.client.0.smithi089.stdout:removed './264/23' 2023-10-30T12:36:17.809 INFO:tasks.workunit.client.0.smithi089.stdout:removed './264/35' 2023-10-30T12:36:17.809 INFO:tasks.workunit.client.0.smithi089.stdout:removed './264/26' 2023-10-30T12:36:17.809 INFO:tasks.workunit.client.0.smithi089.stdout:removed './264/37' 2023-10-30T12:36:17.810 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './264' 2023-10-30T12:36:17.810 INFO:tasks.workunit.client.0.smithi089.stdout:removed './265/21' 2023-10-30T12:36:17.810 INFO:tasks.workunit.client.0.smithi089.stdout:removed './265/27' 2023-10-30T12:36:17.810 INFO:tasks.workunit.client.0.smithi089.stdout:removed './265/31' 2023-10-30T12:36:17.810 INFO:tasks.workunit.client.0.smithi089.stdout:removed './265/36' 2023-10-30T12:36:17.811 INFO:tasks.workunit.client.0.smithi089.stdout:removed './265/20' 2023-10-30T12:36:17.811 INFO:tasks.workunit.client.0.smithi089.stdout:removed './265/39' 2023-10-30T12:36:17.811 INFO:tasks.workunit.client.0.smithi089.stdout:removed './265/34' 2023-10-30T12:36:17.811 INFO:tasks.workunit.client.0.smithi089.stdout:removed './265/38' 2023-10-30T12:36:17.811 INFO:tasks.workunit.client.0.smithi089.stdout:removed './265/28' 2023-10-30T12:36:17.812 INFO:tasks.workunit.client.0.smithi089.stdout:removed './265/24' 2023-10-30T12:36:17.812 INFO:tasks.workunit.client.0.smithi089.stdout:removed './265/29' 2023-10-30T12:36:17.812 INFO:tasks.workunit.client.0.smithi089.stdout:removed './265/32' 2023-10-30T12:36:17.812 INFO:tasks.workunit.client.0.smithi089.stdout:removed './265/22' 2023-10-30T12:36:17.812 INFO:tasks.workunit.client.0.smithi089.stdout:removed './265/25' 2023-10-30T12:36:17.812 INFO:tasks.workunit.client.0.smithi089.stdout:removed './265/30' 2023-10-30T12:36:17.813 INFO:tasks.workunit.client.0.smithi089.stdout:removed './265/33' 2023-10-30T12:36:17.813 INFO:tasks.workunit.client.0.smithi089.stdout:removed './265/23' 2023-10-30T12:36:17.813 INFO:tasks.workunit.client.0.smithi089.stdout:removed './265/35' 2023-10-30T12:36:17.813 INFO:tasks.workunit.client.0.smithi089.stdout:removed './265/26' 2023-10-30T12:36:17.813 INFO:tasks.workunit.client.0.smithi089.stdout:removed './265/37' 2023-10-30T12:36:17.814 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './265' 2023-10-30T12:36:17.814 INFO:tasks.workunit.client.0.smithi089.stdout:removed './266/21' 2023-10-30T12:36:17.814 INFO:tasks.workunit.client.0.smithi089.stdout:removed './266/27' 2023-10-30T12:36:17.814 INFO:tasks.workunit.client.0.smithi089.stdout:removed './266/31' 2023-10-30T12:36:17.814 INFO:tasks.workunit.client.0.smithi089.stdout:removed './266/36' 2023-10-30T12:36:17.815 INFO:tasks.workunit.client.0.smithi089.stdout:removed './266/20' 2023-10-30T12:36:17.815 INFO:tasks.workunit.client.0.smithi089.stdout:removed './266/39' 2023-10-30T12:36:17.815 INFO:tasks.workunit.client.0.smithi089.stdout:removed './266/34' 2023-10-30T12:36:17.815 INFO:tasks.workunit.client.0.smithi089.stdout:removed './266/38' 2023-10-30T12:36:17.815 INFO:tasks.workunit.client.0.smithi089.stdout:removed './266/28' 2023-10-30T12:36:17.815 INFO:tasks.workunit.client.0.smithi089.stdout:removed './266/24' 2023-10-30T12:36:17.816 INFO:tasks.workunit.client.0.smithi089.stdout:removed './266/29' 2023-10-30T12:36:17.816 INFO:tasks.workunit.client.0.smithi089.stdout:removed './266/32' 2023-10-30T12:36:17.816 INFO:tasks.workunit.client.0.smithi089.stdout:removed './266/22' 2023-10-30T12:36:17.816 INFO:tasks.workunit.client.0.smithi089.stdout:removed './266/25' 2023-10-30T12:36:17.816 INFO:tasks.workunit.client.0.smithi089.stdout:removed './266/30' 2023-10-30T12:36:17.817 INFO:tasks.workunit.client.0.smithi089.stdout:removed './266/33' 2023-10-30T12:36:17.817 INFO:tasks.workunit.client.0.smithi089.stdout:removed './266/23' 2023-10-30T12:36:20.270 INFO:tasks.workunit.client.0.smithi089.stdout:removed './266/35' 2023-10-30T12:36:20.270 INFO:tasks.workunit.client.0.smithi089.stdout:removed './266/26' 2023-10-30T12:36:20.270 INFO:tasks.workunit.client.0.smithi089.stdout:removed './266/37' 2023-10-30T12:36:20.271 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './266' 2023-10-30T12:36:20.271 INFO:tasks.workunit.client.0.smithi089.stdout:removed './267/21' 2023-10-30T12:36:20.271 INFO:tasks.workunit.client.0.smithi089.stdout:removed './267/27' 2023-10-30T12:36:20.271 INFO:tasks.workunit.client.0.smithi089.stdout:removed './267/31' 2023-10-30T12:36:20.272 INFO:tasks.workunit.client.0.smithi089.stdout:removed './267/36' 2023-10-30T12:36:20.272 INFO:tasks.workunit.client.0.smithi089.stdout:removed './267/20' 2023-10-30T12:36:20.272 INFO:tasks.workunit.client.0.smithi089.stdout:removed './267/39' 2023-10-30T12:36:20.272 INFO:tasks.workunit.client.0.smithi089.stdout:removed './267/34' 2023-10-30T12:36:20.272 INFO:tasks.workunit.client.0.smithi089.stdout:removed './267/38' 2023-10-30T12:36:20.273 INFO:tasks.workunit.client.0.smithi089.stdout:removed './267/28' 2023-10-30T12:36:20.273 INFO:tasks.workunit.client.0.smithi089.stdout:removed './267/24' 2023-10-30T12:36:20.273 INFO:tasks.workunit.client.0.smithi089.stdout:removed './267/29' 2023-10-30T12:36:20.273 INFO:tasks.workunit.client.0.smithi089.stdout:removed './267/32' 2023-10-30T12:36:20.274 INFO:tasks.workunit.client.0.smithi089.stdout:removed './267/22' 2023-10-30T12:36:20.274 INFO:tasks.workunit.client.0.smithi089.stdout:removed './267/25' 2023-10-30T12:36:20.274 INFO:tasks.workunit.client.0.smithi089.stdout:removed './267/30' 2023-10-30T12:36:20.274 INFO:tasks.workunit.client.0.smithi089.stdout:removed './267/33' 2023-10-30T12:36:20.275 INFO:tasks.workunit.client.0.smithi089.stdout:removed './267/23' 2023-10-30T12:36:20.275 INFO:tasks.workunit.client.0.smithi089.stdout:removed './267/35' 2023-10-30T12:36:20.275 INFO:tasks.workunit.client.0.smithi089.stdout:removed './267/26' 2023-10-30T12:36:20.276 INFO:tasks.workunit.client.0.smithi089.stdout:removed './267/37' 2023-10-30T12:36:20.276 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './267' 2023-10-30T12:36:20.276 INFO:tasks.workunit.client.0.smithi089.stdout:removed './268/21' 2023-10-30T12:36:20.276 INFO:tasks.workunit.client.0.smithi089.stdout:removed './268/27' 2023-10-30T12:36:20.276 INFO:tasks.workunit.client.0.smithi089.stdout:removed './268/31' 2023-10-30T12:36:20.277 INFO:tasks.workunit.client.0.smithi089.stdout:removed './268/36' 2023-10-30T12:36:20.277 INFO:tasks.workunit.client.0.smithi089.stdout:removed './268/20' 2023-10-30T12:36:20.277 INFO:tasks.workunit.client.0.smithi089.stdout:removed './268/39' 2023-10-30T12:36:20.277 INFO:tasks.workunit.client.0.smithi089.stdout:removed './268/34' 2023-10-30T12:36:20.277 INFO:tasks.workunit.client.0.smithi089.stdout:removed './268/38' 2023-10-30T12:36:20.277 INFO:tasks.workunit.client.0.smithi089.stdout:removed './268/28' 2023-10-30T12:36:20.278 INFO:tasks.workunit.client.0.smithi089.stdout:removed './268/24' 2023-10-30T12:36:20.278 INFO:tasks.workunit.client.0.smithi089.stdout:removed './268/29' 2023-10-30T12:36:20.278 INFO:tasks.workunit.client.0.smithi089.stdout:removed './268/32' 2023-10-30T12:36:20.278 INFO:tasks.workunit.client.0.smithi089.stdout:removed './268/22' 2023-10-30T12:36:20.279 INFO:tasks.workunit.client.0.smithi089.stdout:removed './268/25' 2023-10-30T12:36:20.279 INFO:tasks.workunit.client.0.smithi089.stdout:removed './268/30' 2023-10-30T12:36:20.279 INFO:tasks.workunit.client.0.smithi089.stdout:removed './268/33' 2023-10-30T12:36:20.279 INFO:tasks.workunit.client.0.smithi089.stdout:removed './268/23' 2023-10-30T12:36:20.279 INFO:tasks.workunit.client.0.smithi089.stdout:removed './268/35' 2023-10-30T12:36:20.279 INFO:tasks.workunit.client.0.smithi089.stdout:removed './268/26' 2023-10-30T12:36:20.280 INFO:tasks.workunit.client.0.smithi089.stdout:removed './268/37' 2023-10-30T12:36:20.280 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './268' 2023-10-30T12:36:20.280 INFO:tasks.workunit.client.0.smithi089.stdout:removed './269/21' 2023-10-30T12:36:20.280 INFO:tasks.workunit.client.0.smithi089.stdout:removed './269/27' 2023-10-30T12:36:20.280 INFO:tasks.workunit.client.0.smithi089.stdout:removed './269/31' 2023-10-30T12:36:20.281 INFO:tasks.workunit.client.0.smithi089.stdout:removed './269/36' 2023-10-30T12:36:20.281 INFO:tasks.workunit.client.0.smithi089.stdout:removed './269/20' 2023-10-30T12:36:20.281 INFO:tasks.workunit.client.0.smithi089.stdout:removed './269/39' 2023-10-30T12:36:20.281 INFO:tasks.workunit.client.0.smithi089.stdout:removed './269/34' 2023-10-30T12:36:20.281 INFO:tasks.workunit.client.0.smithi089.stdout:removed './269/38' 2023-10-30T12:36:20.282 INFO:tasks.workunit.client.0.smithi089.stdout:removed './269/28' 2023-10-30T12:36:20.282 INFO:tasks.workunit.client.0.smithi089.stdout:removed './269/24' 2023-10-30T12:36:20.282 INFO:tasks.workunit.client.0.smithi089.stdout:removed './269/29' 2023-10-30T12:36:20.282 INFO:tasks.workunit.client.0.smithi089.stdout:removed './269/32' 2023-10-30T12:36:20.282 INFO:tasks.workunit.client.0.smithi089.stdout:removed './269/22' 2023-10-30T12:36:20.283 INFO:tasks.workunit.client.0.smithi089.stdout:removed './269/25' 2023-10-30T12:36:20.283 INFO:tasks.workunit.client.0.smithi089.stdout:removed './269/30' 2023-10-30T12:36:20.283 INFO:tasks.workunit.client.0.smithi089.stdout:removed './269/33' 2023-10-30T12:36:20.283 INFO:tasks.workunit.client.0.smithi089.stdout:removed './269/23' 2023-10-30T12:36:20.283 INFO:tasks.workunit.client.0.smithi089.stdout:removed './269/35' 2023-10-30T12:36:20.284 INFO:tasks.workunit.client.0.smithi089.stdout:removed './269/26' 2023-10-30T12:36:20.284 INFO:tasks.workunit.client.0.smithi089.stdout:removed './269/37' 2023-10-30T12:36:20.284 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './269' 2023-10-30T12:36:20.284 INFO:tasks.workunit.client.0.smithi089.stdout:removed './270/21' 2023-10-30T12:36:20.285 INFO:tasks.workunit.client.0.smithi089.stdout:removed './270/27' 2023-10-30T12:36:20.285 INFO:tasks.workunit.client.0.smithi089.stdout:removed './270/31' 2023-10-30T12:36:20.285 INFO:tasks.workunit.client.0.smithi089.stdout:removed './270/36' 2023-10-30T12:36:20.285 INFO:tasks.workunit.client.0.smithi089.stdout:removed './270/20' 2023-10-30T12:36:20.285 INFO:tasks.workunit.client.0.smithi089.stdout:removed './270/39' 2023-10-30T12:36:20.286 INFO:tasks.workunit.client.0.smithi089.stdout:removed './270/34' 2023-10-30T12:36:20.286 INFO:tasks.workunit.client.0.smithi089.stdout:removed './270/38' 2023-10-30T12:36:20.286 INFO:tasks.workunit.client.0.smithi089.stdout:removed './270/28' 2023-10-30T12:36:20.286 INFO:tasks.workunit.client.0.smithi089.stdout:removed './270/24' 2023-10-30T12:36:20.286 INFO:tasks.workunit.client.0.smithi089.stdout:removed './270/29' 2023-10-30T12:36:20.287 INFO:tasks.workunit.client.0.smithi089.stdout:removed './270/32' 2023-10-30T12:36:20.287 INFO:tasks.workunit.client.0.smithi089.stdout:removed './270/22' 2023-10-30T12:36:20.287 INFO:tasks.workunit.client.0.smithi089.stdout:removed './270/25' 2023-10-30T12:36:20.287 INFO:tasks.workunit.client.0.smithi089.stdout:removed './270/30' 2023-10-30T12:36:20.287 INFO:tasks.workunit.client.0.smithi089.stdout:removed './270/33' 2023-10-30T12:36:20.288 INFO:tasks.workunit.client.0.smithi089.stdout:removed './270/23' 2023-10-30T12:36:20.288 INFO:tasks.workunit.client.0.smithi089.stdout:removed './270/35' 2023-10-30T12:36:20.288 INFO:tasks.workunit.client.0.smithi089.stdout:removed './270/26' 2023-10-30T12:36:20.288 INFO:tasks.workunit.client.0.smithi089.stdout:removed './270/37' 2023-10-30T12:36:20.288 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './270' 2023-10-30T12:36:20.289 INFO:tasks.workunit.client.0.smithi089.stdout:removed './271/21' 2023-10-30T12:36:20.289 INFO:tasks.workunit.client.0.smithi089.stdout:removed './271/27' 2023-10-30T12:36:20.289 INFO:tasks.workunit.client.0.smithi089.stdout:removed './271/31' 2023-10-30T12:36:20.289 INFO:tasks.workunit.client.0.smithi089.stdout:removed './271/36' 2023-10-30T12:36:20.289 INFO:tasks.workunit.client.0.smithi089.stdout:removed './271/20' 2023-10-30T12:36:20.290 INFO:tasks.workunit.client.0.smithi089.stdout:removed './271/39' 2023-10-30T12:36:20.290 INFO:tasks.workunit.client.0.smithi089.stdout:removed './271/34' 2023-10-30T12:36:20.290 INFO:tasks.workunit.client.0.smithi089.stdout:removed './271/38' 2023-10-30T12:36:20.290 INFO:tasks.workunit.client.0.smithi089.stdout:removed './271/28' 2023-10-30T12:36:20.290 INFO:tasks.workunit.client.0.smithi089.stdout:removed './271/24' 2023-10-30T12:36:20.291 INFO:tasks.workunit.client.0.smithi089.stdout:removed './271/29' 2023-10-30T12:36:20.291 INFO:tasks.workunit.client.0.smithi089.stdout:removed './271/32' 2023-10-30T12:36:20.291 INFO:tasks.workunit.client.0.smithi089.stdout:removed './271/22' 2023-10-30T12:36:20.291 INFO:tasks.workunit.client.0.smithi089.stdout:removed './271/25' 2023-10-30T12:36:20.291 INFO:tasks.workunit.client.0.smithi089.stdout:removed './271/30' 2023-10-30T12:36:20.292 INFO:tasks.workunit.client.0.smithi089.stdout:removed './271/33' 2023-10-30T12:36:20.292 INFO:tasks.workunit.client.0.smithi089.stdout:removed './271/23' 2023-10-30T12:36:20.292 INFO:tasks.workunit.client.0.smithi089.stdout:removed './271/35' 2023-10-30T12:36:20.292 INFO:tasks.workunit.client.0.smithi089.stdout:removed './271/26' 2023-10-30T12:36:20.292 INFO:tasks.workunit.client.0.smithi089.stdout:removed './271/37' 2023-10-30T12:36:20.292 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './271' 2023-10-30T12:36:20.293 INFO:tasks.workunit.client.0.smithi089.stdout:removed './272/21' 2023-10-30T12:36:20.293 INFO:tasks.workunit.client.0.smithi089.stdout:removed './272/27' 2023-10-30T12:36:20.293 INFO:tasks.workunit.client.0.smithi089.stdout:removed './272/31' 2023-10-30T12:36:20.293 INFO:tasks.workunit.client.0.smithi089.stdout:removed './272/36' 2023-10-30T12:36:20.293 INFO:tasks.workunit.client.0.smithi089.stdout:removed './272/20' 2023-10-30T12:36:20.294 INFO:tasks.workunit.client.0.smithi089.stdout:removed './272/39' 2023-10-30T12:36:20.294 INFO:tasks.workunit.client.0.smithi089.stdout:removed './272/34' 2023-10-30T12:36:20.294 INFO:tasks.workunit.client.0.smithi089.stdout:removed './272/38' 2023-10-30T12:36:20.294 INFO:tasks.workunit.client.0.smithi089.stdout:removed './272/28' 2023-10-30T12:36:20.295 INFO:tasks.workunit.client.0.smithi089.stdout:removed './272/24' 2023-10-30T12:36:20.295 INFO:tasks.workunit.client.0.smithi089.stdout:removed './272/29' 2023-10-30T12:36:20.295 INFO:tasks.workunit.client.0.smithi089.stdout:removed './272/32' 2023-10-30T12:36:20.295 INFO:tasks.workunit.client.0.smithi089.stdout:removed './272/22' 2023-10-30T12:36:20.295 INFO:tasks.workunit.client.0.smithi089.stdout:removed './272/25' 2023-10-30T12:36:20.296 INFO:tasks.workunit.client.0.smithi089.stdout:removed './272/30' 2023-10-30T12:36:20.296 INFO:tasks.workunit.client.0.smithi089.stdout:removed './272/33' 2023-10-30T12:36:20.296 INFO:tasks.workunit.client.0.smithi089.stdout:removed './272/23' 2023-10-30T12:36:20.296 INFO:tasks.workunit.client.0.smithi089.stdout:removed './272/35' 2023-10-30T12:36:20.296 INFO:tasks.workunit.client.0.smithi089.stdout:removed './272/26' 2023-10-30T12:36:20.296 INFO:tasks.workunit.client.0.smithi089.stdout:removed './272/37' 2023-10-30T12:36:20.297 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './272' 2023-10-30T12:36:20.297 INFO:tasks.workunit.client.0.smithi089.stdout:removed './273/21' 2023-10-30T12:36:20.297 INFO:tasks.workunit.client.0.smithi089.stdout:removed './273/27' 2023-10-30T12:36:20.297 INFO:tasks.workunit.client.0.smithi089.stdout:removed './273/31' 2023-10-30T12:36:20.297 INFO:tasks.workunit.client.0.smithi089.stdout:removed './273/36' 2023-10-30T12:36:20.298 INFO:tasks.workunit.client.0.smithi089.stdout:removed './273/20' 2023-10-30T12:36:20.298 INFO:tasks.workunit.client.0.smithi089.stdout:removed './273/39' 2023-10-30T12:36:20.298 INFO:tasks.workunit.client.0.smithi089.stdout:removed './273/34' 2023-10-30T12:36:20.298 INFO:tasks.workunit.client.0.smithi089.stdout:removed './273/38' 2023-10-30T12:36:20.298 INFO:tasks.workunit.client.0.smithi089.stdout:removed './273/28' 2023-10-30T12:36:20.299 INFO:tasks.workunit.client.0.smithi089.stdout:removed './273/24' 2023-10-30T12:36:20.299 INFO:tasks.workunit.client.0.smithi089.stdout:removed './273/29' 2023-10-30T12:36:20.299 INFO:tasks.workunit.client.0.smithi089.stdout:removed './273/32' 2023-10-30T12:36:20.299 INFO:tasks.workunit.client.0.smithi089.stdout:removed './273/22' 2023-10-30T12:36:20.299 INFO:tasks.workunit.client.0.smithi089.stdout:removed './273/25' 2023-10-30T12:36:20.299 INFO:tasks.workunit.client.0.smithi089.stdout:removed './273/30' 2023-10-30T12:36:20.300 INFO:tasks.workunit.client.0.smithi089.stdout:removed './273/33' 2023-10-30T12:36:20.300 INFO:tasks.workunit.client.0.smithi089.stdout:removed './273/23' 2023-10-30T12:36:20.300 INFO:tasks.workunit.client.0.smithi089.stdout:removed './273/35' 2023-10-30T12:36:20.300 INFO:tasks.workunit.client.0.smithi089.stdout:removed './273/26' 2023-10-30T12:36:20.300 INFO:tasks.workunit.client.0.smithi089.stdout:removed './273/37' 2023-10-30T12:36:20.301 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './273' 2023-10-30T12:36:20.301 INFO:tasks.workunit.client.0.smithi089.stdout:removed './274/21' 2023-10-30T12:36:20.301 INFO:tasks.workunit.client.0.smithi089.stdout:removed './274/27' 2023-10-30T12:36:20.301 INFO:tasks.workunit.client.0.smithi089.stdout:removed './274/31' 2023-10-30T12:36:20.301 INFO:tasks.workunit.client.0.smithi089.stdout:removed './274/36' 2023-10-30T12:36:20.301 INFO:tasks.workunit.client.0.smithi089.stdout:removed './274/20' 2023-10-30T12:36:20.302 INFO:tasks.workunit.client.0.smithi089.stdout:removed './274/39' 2023-10-30T12:36:20.302 INFO:tasks.workunit.client.0.smithi089.stdout:removed './274/34' 2023-10-30T12:36:20.302 INFO:tasks.workunit.client.0.smithi089.stdout:removed './274/38' 2023-10-30T12:36:20.302 INFO:tasks.workunit.client.0.smithi089.stdout:removed './274/28' 2023-10-30T12:36:20.303 INFO:tasks.workunit.client.0.smithi089.stdout:removed './274/24' 2023-10-30T12:36:20.303 INFO:tasks.workunit.client.0.smithi089.stdout:removed './274/29' 2023-10-30T12:36:20.303 INFO:tasks.workunit.client.0.smithi089.stdout:removed './274/32' 2023-10-30T12:36:20.303 INFO:tasks.workunit.client.0.smithi089.stdout:removed './274/22' 2023-10-30T12:36:20.303 INFO:tasks.workunit.client.0.smithi089.stdout:removed './274/25' 2023-10-30T12:36:20.303 INFO:tasks.workunit.client.0.smithi089.stdout:removed './274/30' 2023-10-30T12:36:20.304 INFO:tasks.workunit.client.0.smithi089.stdout:removed './274/33' 2023-10-30T12:36:20.304 INFO:tasks.workunit.client.0.smithi089.stdout:removed './274/23' 2023-10-30T12:36:20.304 INFO:tasks.workunit.client.0.smithi089.stdout:removed './274/35' 2023-10-30T12:36:20.304 INFO:tasks.workunit.client.0.smithi089.stdout:removed './274/26' 2023-10-30T12:36:20.304 INFO:tasks.workunit.client.0.smithi089.stdout:removed './274/37' 2023-10-30T12:36:20.304 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './274' 2023-10-30T12:36:20.305 INFO:tasks.workunit.client.0.smithi089.stdout:removed './275/21' 2023-10-30T12:36:20.305 INFO:tasks.workunit.client.0.smithi089.stdout:removed './275/27' 2023-10-30T12:36:20.305 INFO:tasks.workunit.client.0.smithi089.stdout:removed './275/31' 2023-10-30T12:36:20.305 INFO:tasks.workunit.client.0.smithi089.stdout:removed './275/36' 2023-10-30T12:36:20.305 INFO:tasks.workunit.client.0.smithi089.stdout:removed './275/20' 2023-10-30T12:36:20.306 INFO:tasks.workunit.client.0.smithi089.stdout:removed './275/39' 2023-10-30T12:36:20.306 INFO:tasks.workunit.client.0.smithi089.stdout:removed './275/34' 2023-10-30T12:36:20.306 INFO:tasks.workunit.client.0.smithi089.stdout:removed './275/38' 2023-10-30T12:36:20.306 INFO:tasks.workunit.client.0.smithi089.stdout:removed './275/28' 2023-10-30T12:36:20.306 INFO:tasks.workunit.client.0.smithi089.stdout:removed './275/24' 2023-10-30T12:36:20.307 INFO:tasks.workunit.client.0.smithi089.stdout:removed './275/29' 2023-10-30T12:36:20.307 INFO:tasks.workunit.client.0.smithi089.stdout:removed './275/32' 2023-10-30T12:36:20.307 INFO:tasks.workunit.client.0.smithi089.stdout:removed './275/22' 2023-10-30T12:36:20.307 INFO:tasks.workunit.client.0.smithi089.stdout:removed './275/25' 2023-10-30T12:36:20.307 INFO:tasks.workunit.client.0.smithi089.stdout:removed './275/30' 2023-10-30T12:36:20.308 INFO:tasks.workunit.client.0.smithi089.stdout:removed './275/33' 2023-10-30T12:36:20.308 INFO:tasks.workunit.client.0.smithi089.stdout:removed './275/23' 2023-10-30T12:36:20.308 INFO:tasks.workunit.client.0.smithi089.stdout:removed './275/35' 2023-10-30T12:36:20.308 INFO:tasks.workunit.client.0.smithi089.stdout:removed './275/26' 2023-10-30T12:36:20.308 INFO:tasks.workunit.client.0.smithi089.stdout:removed './275/37' 2023-10-30T12:36:20.309 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './275' 2023-10-30T12:36:20.309 INFO:tasks.workunit.client.0.smithi089.stdout:removed './276/21' 2023-10-30T12:36:20.309 INFO:tasks.workunit.client.0.smithi089.stdout:removed './276/27' 2023-10-30T12:36:20.309 INFO:tasks.workunit.client.0.smithi089.stdout:removed './276/31' 2023-10-30T12:36:20.309 INFO:tasks.workunit.client.0.smithi089.stdout:removed './276/36' 2023-10-30T12:36:20.310 INFO:tasks.workunit.client.0.smithi089.stdout:removed './276/20' 2023-10-30T12:36:20.310 INFO:tasks.workunit.client.0.smithi089.stdout:removed './276/39' 2023-10-30T12:36:20.310 INFO:tasks.workunit.client.0.smithi089.stdout:removed './276/34' 2023-10-30T12:36:20.310 INFO:tasks.workunit.client.0.smithi089.stdout:removed './276/38' 2023-10-30T12:36:20.310 INFO:tasks.workunit.client.0.smithi089.stdout:removed './276/28' 2023-10-30T12:36:20.311 INFO:tasks.workunit.client.0.smithi089.stdout:removed './276/24' 2023-10-30T12:36:20.311 INFO:tasks.workunit.client.0.smithi089.stdout:removed './276/29' 2023-10-30T12:36:20.311 INFO:tasks.workunit.client.0.smithi089.stdout:removed './276/32' 2023-10-30T12:36:20.311 INFO:tasks.workunit.client.0.smithi089.stdout:removed './276/22' 2023-10-30T12:36:20.311 INFO:tasks.workunit.client.0.smithi089.stdout:removed './276/25' 2023-10-30T12:36:20.312 INFO:tasks.workunit.client.0.smithi089.stdout:removed './276/30' 2023-10-30T12:36:20.312 INFO:tasks.workunit.client.0.smithi089.stdout:removed './276/33' 2023-10-30T12:36:20.312 INFO:tasks.workunit.client.0.smithi089.stdout:removed './276/23' 2023-10-30T12:36:20.312 INFO:tasks.workunit.client.0.smithi089.stdout:removed './276/35' 2023-10-30T12:36:20.312 INFO:tasks.workunit.client.0.smithi089.stdout:removed './276/26' 2023-10-30T12:36:25.067 INFO:tasks.workunit.client.0.smithi089.stdout:removed './276/37' 2023-10-30T12:36:25.068 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './276' 2023-10-30T12:36:25.068 INFO:tasks.workunit.client.0.smithi089.stdout:removed './277/21' 2023-10-30T12:36:25.068 INFO:tasks.workunit.client.0.smithi089.stdout:removed './277/27' 2023-10-30T12:36:25.068 INFO:tasks.workunit.client.0.smithi089.stdout:removed './277/31' 2023-10-30T12:36:25.069 INFO:tasks.workunit.client.0.smithi089.stdout:removed './277/36' 2023-10-30T12:36:25.069 INFO:tasks.workunit.client.0.smithi089.stdout:removed './277/20' 2023-10-30T12:36:25.069 INFO:tasks.workunit.client.0.smithi089.stdout:removed './277/39' 2023-10-30T12:36:25.069 INFO:tasks.workunit.client.0.smithi089.stdout:removed './277/34' 2023-10-30T12:36:25.069 INFO:tasks.workunit.client.0.smithi089.stdout:removed './277/38' 2023-10-30T12:36:25.070 INFO:tasks.workunit.client.0.smithi089.stdout:removed './277/28' 2023-10-30T12:36:25.070 INFO:tasks.workunit.client.0.smithi089.stdout:removed './277/24' 2023-10-30T12:36:25.070 INFO:tasks.workunit.client.0.smithi089.stdout:removed './277/29' 2023-10-30T12:36:25.070 INFO:tasks.workunit.client.0.smithi089.stdout:removed './277/32' 2023-10-30T12:36:25.070 INFO:tasks.workunit.client.0.smithi089.stdout:removed './277/22' 2023-10-30T12:36:25.071 INFO:tasks.workunit.client.0.smithi089.stdout:removed './277/25' 2023-10-30T12:36:25.071 INFO:tasks.workunit.client.0.smithi089.stdout:removed './277/30' 2023-10-30T12:36:25.071 INFO:tasks.workunit.client.0.smithi089.stdout:removed './277/33' 2023-10-30T12:36:25.071 INFO:tasks.workunit.client.0.smithi089.stdout:removed './277/23' 2023-10-30T12:36:25.072 INFO:tasks.workunit.client.0.smithi089.stdout:removed './277/35' 2023-10-30T12:36:25.072 INFO:tasks.workunit.client.0.smithi089.stdout:removed './277/26' 2023-10-30T12:36:25.072 INFO:tasks.workunit.client.0.smithi089.stdout:removed './277/37' 2023-10-30T12:36:25.072 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './277' 2023-10-30T12:36:25.073 INFO:tasks.workunit.client.0.smithi089.stdout:removed './278/21' 2023-10-30T12:36:25.073 INFO:tasks.workunit.client.0.smithi089.stdout:removed './278/27' 2023-10-30T12:36:25.073 INFO:tasks.workunit.client.0.smithi089.stdout:removed './278/31' 2023-10-30T12:36:25.073 INFO:tasks.workunit.client.0.smithi089.stdout:removed './278/36' 2023-10-30T12:36:25.073 INFO:tasks.workunit.client.0.smithi089.stdout:removed './278/20' 2023-10-30T12:36:25.074 INFO:tasks.workunit.client.0.smithi089.stdout:removed './278/39' 2023-10-30T12:36:25.074 INFO:tasks.workunit.client.0.smithi089.stdout:removed './278/34' 2023-10-30T12:36:25.074 INFO:tasks.workunit.client.0.smithi089.stdout:removed './278/38' 2023-10-30T12:36:25.074 INFO:tasks.workunit.client.0.smithi089.stdout:removed './278/28' 2023-10-30T12:36:25.074 INFO:tasks.workunit.client.0.smithi089.stdout:removed './278/24' 2023-10-30T12:36:25.075 INFO:tasks.workunit.client.0.smithi089.stdout:removed './278/29' 2023-10-30T12:36:25.075 INFO:tasks.workunit.client.0.smithi089.stdout:removed './278/32' 2023-10-30T12:36:25.075 INFO:tasks.workunit.client.0.smithi089.stdout:removed './278/22' 2023-10-30T12:36:25.075 INFO:tasks.workunit.client.0.smithi089.stdout:removed './278/25' 2023-10-30T12:36:25.075 INFO:tasks.workunit.client.0.smithi089.stdout:removed './278/30' 2023-10-30T12:36:25.076 INFO:tasks.workunit.client.0.smithi089.stdout:removed './278/33' 2023-10-30T12:36:25.076 INFO:tasks.workunit.client.0.smithi089.stdout:removed './278/23' 2023-10-30T12:36:25.076 INFO:tasks.workunit.client.0.smithi089.stdout:removed './278/35' 2023-10-30T12:36:25.076 INFO:tasks.workunit.client.0.smithi089.stdout:removed './278/26' 2023-10-30T12:36:25.076 INFO:tasks.workunit.client.0.smithi089.stdout:removed './278/37' 2023-10-30T12:36:25.077 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './278' 2023-10-30T12:36:25.077 INFO:tasks.workunit.client.0.smithi089.stdout:removed './279/21' 2023-10-30T12:36:25.077 INFO:tasks.workunit.client.0.smithi089.stdout:removed './279/27' 2023-10-30T12:36:25.077 INFO:tasks.workunit.client.0.smithi089.stdout:removed './279/31' 2023-10-30T12:36:25.077 INFO:tasks.workunit.client.0.smithi089.stdout:removed './279/36' 2023-10-30T12:36:25.078 INFO:tasks.workunit.client.0.smithi089.stdout:removed './279/20' 2023-10-30T12:36:25.078 INFO:tasks.workunit.client.0.smithi089.stdout:removed './279/39' 2023-10-30T12:36:25.078 INFO:tasks.workunit.client.0.smithi089.stdout:removed './279/34' 2023-10-30T12:36:25.078 INFO:tasks.workunit.client.0.smithi089.stdout:removed './279/38' 2023-10-30T12:36:25.079 INFO:tasks.workunit.client.0.smithi089.stdout:removed './279/28' 2023-10-30T12:36:25.079 INFO:tasks.workunit.client.0.smithi089.stdout:removed './279/24' 2023-10-30T12:36:25.079 INFO:tasks.workunit.client.0.smithi089.stdout:removed './279/29' 2023-10-30T12:36:25.079 INFO:tasks.workunit.client.0.smithi089.stdout:removed './279/32' 2023-10-30T12:36:25.080 INFO:tasks.workunit.client.0.smithi089.stdout:removed './279/22' 2023-10-30T12:36:25.080 INFO:tasks.workunit.client.0.smithi089.stdout:removed './279/25' 2023-10-30T12:36:25.080 INFO:tasks.workunit.client.0.smithi089.stdout:removed './279/30' 2023-10-30T12:36:25.080 INFO:tasks.workunit.client.0.smithi089.stdout:removed './279/33' 2023-10-30T12:36:25.080 INFO:tasks.workunit.client.0.smithi089.stdout:removed './279/23' 2023-10-30T12:36:25.080 INFO:tasks.workunit.client.0.smithi089.stdout:removed './279/35' 2023-10-30T12:36:25.081 INFO:tasks.workunit.client.0.smithi089.stdout:removed './279/26' 2023-10-30T12:36:25.081 INFO:tasks.workunit.client.0.smithi089.stdout:removed './279/37' 2023-10-30T12:36:25.081 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './279' 2023-10-30T12:36:25.081 INFO:tasks.workunit.client.0.smithi089.stdout:removed './280/21' 2023-10-30T12:36:25.081 INFO:tasks.workunit.client.0.smithi089.stdout:removed './280/27' 2023-10-30T12:36:25.082 INFO:tasks.workunit.client.0.smithi089.stdout:removed './280/31' 2023-10-30T12:36:25.082 INFO:tasks.workunit.client.0.smithi089.stdout:removed './280/36' 2023-10-30T12:36:25.082 INFO:tasks.workunit.client.0.smithi089.stdout:removed './280/20' 2023-10-30T12:36:25.082 INFO:tasks.workunit.client.0.smithi089.stdout:removed './280/39' 2023-10-30T12:36:25.082 INFO:tasks.workunit.client.0.smithi089.stdout:removed './280/34' 2023-10-30T12:36:25.083 INFO:tasks.workunit.client.0.smithi089.stdout:removed './280/38' 2023-10-30T12:36:25.083 INFO:tasks.workunit.client.0.smithi089.stdout:removed './280/28' 2023-10-30T12:36:25.083 INFO:tasks.workunit.client.0.smithi089.stdout:removed './280/24' 2023-10-30T12:36:25.083 INFO:tasks.workunit.client.0.smithi089.stdout:removed './280/29' 2023-10-30T12:36:25.083 INFO:tasks.workunit.client.0.smithi089.stdout:removed './280/32' 2023-10-30T12:36:25.083 INFO:tasks.workunit.client.0.smithi089.stdout:removed './280/22' 2023-10-30T12:36:25.084 INFO:tasks.workunit.client.0.smithi089.stdout:removed './280/25' 2023-10-30T12:36:25.084 INFO:tasks.workunit.client.0.smithi089.stdout:removed './280/30' 2023-10-30T12:36:25.084 INFO:tasks.workunit.client.0.smithi089.stdout:removed './280/33' 2023-10-30T12:36:25.084 INFO:tasks.workunit.client.0.smithi089.stdout:removed './280/23' 2023-10-30T12:36:25.084 INFO:tasks.workunit.client.0.smithi089.stdout:removed './280/35' 2023-10-30T12:36:25.085 INFO:tasks.workunit.client.0.smithi089.stdout:removed './280/26' 2023-10-30T12:36:25.085 INFO:tasks.workunit.client.0.smithi089.stdout:removed './280/37' 2023-10-30T12:36:25.085 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './280' 2023-10-30T12:36:25.085 INFO:tasks.workunit.client.0.smithi089.stdout:removed './281/21' 2023-10-30T12:36:25.086 INFO:tasks.workunit.client.0.smithi089.stdout:removed './281/27' 2023-10-30T12:36:25.086 INFO:tasks.workunit.client.0.smithi089.stdout:removed './281/31' 2023-10-30T12:36:25.086 INFO:tasks.workunit.client.0.smithi089.stdout:removed './281/36' 2023-10-30T12:36:25.086 INFO:tasks.workunit.client.0.smithi089.stdout:removed './281/20' 2023-10-30T12:36:25.086 INFO:tasks.workunit.client.0.smithi089.stdout:removed './281/39' 2023-10-30T12:36:25.087 INFO:tasks.workunit.client.0.smithi089.stdout:removed './281/34' 2023-10-30T12:36:25.087 INFO:tasks.workunit.client.0.smithi089.stdout:removed './281/38' 2023-10-30T12:36:25.087 INFO:tasks.workunit.client.0.smithi089.stdout:removed './281/28' 2023-10-30T12:36:25.087 INFO:tasks.workunit.client.0.smithi089.stdout:removed './281/24' 2023-10-30T12:36:25.087 INFO:tasks.workunit.client.0.smithi089.stdout:removed './281/29' 2023-10-30T12:36:25.088 INFO:tasks.workunit.client.0.smithi089.stdout:removed './281/32' 2023-10-30T12:36:25.088 INFO:tasks.workunit.client.0.smithi089.stdout:removed './281/22' 2023-10-30T12:36:25.088 INFO:tasks.workunit.client.0.smithi089.stdout:removed './281/25' 2023-10-30T12:36:25.088 INFO:tasks.workunit.client.0.smithi089.stdout:removed './281/30' 2023-10-30T12:36:25.088 INFO:tasks.workunit.client.0.smithi089.stdout:removed './281/33' 2023-10-30T12:36:25.089 INFO:tasks.workunit.client.0.smithi089.stdout:removed './281/23' 2023-10-30T12:36:25.089 INFO:tasks.workunit.client.0.smithi089.stdout:removed './281/35' 2023-10-30T12:36:25.089 INFO:tasks.workunit.client.0.smithi089.stdout:removed './281/26' 2023-10-30T12:36:25.089 INFO:tasks.workunit.client.0.smithi089.stdout:removed './281/37' 2023-10-30T12:36:25.089 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './281' 2023-10-30T12:36:25.089 INFO:tasks.workunit.client.0.smithi089.stdout:removed './282/21' 2023-10-30T12:36:25.090 INFO:tasks.workunit.client.0.smithi089.stdout:removed './282/27' 2023-10-30T12:36:25.090 INFO:tasks.workunit.client.0.smithi089.stdout:removed './282/31' 2023-10-30T12:36:25.090 INFO:tasks.workunit.client.0.smithi089.stdout:removed './282/36' 2023-10-30T12:36:25.090 INFO:tasks.workunit.client.0.smithi089.stdout:removed './282/20' 2023-10-30T12:36:25.090 INFO:tasks.workunit.client.0.smithi089.stdout:removed './282/39' 2023-10-30T12:36:25.091 INFO:tasks.workunit.client.0.smithi089.stdout:removed './282/34' 2023-10-30T12:36:25.091 INFO:tasks.workunit.client.0.smithi089.stdout:removed './282/38' 2023-10-30T12:36:25.091 INFO:tasks.workunit.client.0.smithi089.stdout:removed './282/28' 2023-10-30T12:36:25.091 INFO:tasks.workunit.client.0.smithi089.stdout:removed './282/24' 2023-10-30T12:36:25.091 INFO:tasks.workunit.client.0.smithi089.stdout:removed './282/29' 2023-10-30T12:36:25.092 INFO:tasks.workunit.client.0.smithi089.stdout:removed './282/32' 2023-10-30T12:36:25.092 INFO:tasks.workunit.client.0.smithi089.stdout:removed './282/22' 2023-10-30T12:36:25.092 INFO:tasks.workunit.client.0.smithi089.stdout:removed './282/25' 2023-10-30T12:36:25.092 INFO:tasks.workunit.client.0.smithi089.stdout:removed './282/30' 2023-10-30T12:36:25.092 INFO:tasks.workunit.client.0.smithi089.stdout:removed './282/33' 2023-10-30T12:36:25.093 INFO:tasks.workunit.client.0.smithi089.stdout:removed './282/23' 2023-10-30T12:36:25.093 INFO:tasks.workunit.client.0.smithi089.stdout:removed './282/35' 2023-10-30T12:36:25.093 INFO:tasks.workunit.client.0.smithi089.stdout:removed './282/26' 2023-10-30T12:36:25.093 INFO:tasks.workunit.client.0.smithi089.stdout:removed './282/37' 2023-10-30T12:36:25.093 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './282' 2023-10-30T12:36:25.093 INFO:tasks.workunit.client.0.smithi089.stdout:removed './283/21' 2023-10-30T12:36:25.094 INFO:tasks.workunit.client.0.smithi089.stdout:removed './283/27' 2023-10-30T12:36:25.094 INFO:tasks.workunit.client.0.smithi089.stdout:removed './283/31' 2023-10-30T12:36:25.094 INFO:tasks.workunit.client.0.smithi089.stdout:removed './283/36' 2023-10-30T12:36:25.094 INFO:tasks.workunit.client.0.smithi089.stdout:removed './283/20' 2023-10-30T12:36:25.094 INFO:tasks.workunit.client.0.smithi089.stdout:removed './283/39' 2023-10-30T12:36:25.095 INFO:tasks.workunit.client.0.smithi089.stdout:removed './283/34' 2023-10-30T12:36:25.095 INFO:tasks.workunit.client.0.smithi089.stdout:removed './283/38' 2023-10-30T12:36:25.095 INFO:tasks.workunit.client.0.smithi089.stdout:removed './283/28' 2023-10-30T12:36:25.095 INFO:tasks.workunit.client.0.smithi089.stdout:removed './283/24' 2023-10-30T12:36:25.095 INFO:tasks.workunit.client.0.smithi089.stdout:removed './283/29' 2023-10-30T12:36:25.096 INFO:tasks.workunit.client.0.smithi089.stdout:removed './283/32' 2023-10-30T12:36:25.096 INFO:tasks.workunit.client.0.smithi089.stdout:removed './283/22' 2023-10-30T12:36:25.096 INFO:tasks.workunit.client.0.smithi089.stdout:removed './283/25' 2023-10-30T12:36:25.096 INFO:tasks.workunit.client.0.smithi089.stdout:removed './283/30' 2023-10-30T12:36:25.096 INFO:tasks.workunit.client.0.smithi089.stdout:removed './283/33' 2023-10-30T12:36:25.097 INFO:tasks.workunit.client.0.smithi089.stdout:removed './283/23' 2023-10-30T12:36:25.097 INFO:tasks.workunit.client.0.smithi089.stdout:removed './283/35' 2023-10-30T12:36:25.097 INFO:tasks.workunit.client.0.smithi089.stdout:removed './283/26' 2023-10-30T12:36:25.097 INFO:tasks.workunit.client.0.smithi089.stdout:removed './283/37' 2023-10-30T12:36:25.097 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './283' 2023-10-30T12:36:25.098 INFO:tasks.workunit.client.0.smithi089.stdout:removed './284/21' 2023-10-30T12:36:25.098 INFO:tasks.workunit.client.0.smithi089.stdout:removed './284/27' 2023-10-30T12:36:25.098 INFO:tasks.workunit.client.0.smithi089.stdout:removed './284/31' 2023-10-30T12:36:25.098 INFO:tasks.workunit.client.0.smithi089.stdout:removed './284/36' 2023-10-30T12:36:25.098 INFO:tasks.workunit.client.0.smithi089.stdout:removed './284/20' 2023-10-30T12:36:25.099 INFO:tasks.workunit.client.0.smithi089.stdout:removed './284/39' 2023-10-30T12:36:25.099 INFO:tasks.workunit.client.0.smithi089.stdout:removed './284/34' 2023-10-30T12:36:25.099 INFO:tasks.workunit.client.0.smithi089.stdout:removed './284/38' 2023-10-30T12:36:25.099 INFO:tasks.workunit.client.0.smithi089.stdout:removed './284/28' 2023-10-30T12:36:25.099 INFO:tasks.workunit.client.0.smithi089.stdout:removed './284/24' 2023-10-30T12:36:25.099 INFO:tasks.workunit.client.0.smithi089.stdout:removed './284/29' 2023-10-30T12:36:25.100 INFO:tasks.workunit.client.0.smithi089.stdout:removed './284/32' 2023-10-30T12:36:25.100 INFO:tasks.workunit.client.0.smithi089.stdout:removed './284/22' 2023-10-30T12:36:25.100 INFO:tasks.workunit.client.0.smithi089.stdout:removed './284/25' 2023-10-30T12:36:25.100 INFO:tasks.workunit.client.0.smithi089.stdout:removed './284/30' 2023-10-30T12:36:25.100 INFO:tasks.workunit.client.0.smithi089.stdout:removed './284/33' 2023-10-30T12:36:25.101 INFO:tasks.workunit.client.0.smithi089.stdout:removed './284/23' 2023-10-30T12:36:25.101 INFO:tasks.workunit.client.0.smithi089.stdout:removed './284/35' 2023-10-30T12:36:25.101 INFO:tasks.workunit.client.0.smithi089.stdout:removed './284/26' 2023-10-30T12:36:25.101 INFO:tasks.workunit.client.0.smithi089.stdout:removed './284/37' 2023-10-30T12:36:25.101 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './284' 2023-10-30T12:36:25.101 INFO:tasks.workunit.client.0.smithi089.stdout:removed './285/21' 2023-10-30T12:36:25.102 INFO:tasks.workunit.client.0.smithi089.stdout:removed './285/27' 2023-10-30T12:36:25.102 INFO:tasks.workunit.client.0.smithi089.stdout:removed './285/31' 2023-10-30T12:36:25.102 INFO:tasks.workunit.client.0.smithi089.stdout:removed './285/36' 2023-10-30T12:36:25.102 INFO:tasks.workunit.client.0.smithi089.stdout:removed './285/20' 2023-10-30T12:36:25.102 INFO:tasks.workunit.client.0.smithi089.stdout:removed './285/39' 2023-10-30T12:36:25.103 INFO:tasks.workunit.client.0.smithi089.stdout:removed './285/34' 2023-10-30T12:36:25.103 INFO:tasks.workunit.client.0.smithi089.stdout:removed './285/38' 2023-10-30T12:36:25.103 INFO:tasks.workunit.client.0.smithi089.stdout:removed './285/28' 2023-10-30T12:36:25.103 INFO:tasks.workunit.client.0.smithi089.stdout:removed './285/24' 2023-10-30T12:36:25.103 INFO:tasks.workunit.client.0.smithi089.stdout:removed './285/29' 2023-10-30T12:36:25.104 INFO:tasks.workunit.client.0.smithi089.stdout:removed './285/32' 2023-10-30T12:36:25.104 INFO:tasks.workunit.client.0.smithi089.stdout:removed './285/22' 2023-10-30T12:36:25.104 INFO:tasks.workunit.client.0.smithi089.stdout:removed './285/25' 2023-10-30T12:36:25.104 INFO:tasks.workunit.client.0.smithi089.stdout:removed './285/30' 2023-10-30T12:36:25.104 INFO:tasks.workunit.client.0.smithi089.stdout:removed './285/33' 2023-10-30T12:36:25.105 INFO:tasks.workunit.client.0.smithi089.stdout:removed './285/23' 2023-10-30T12:36:25.105 INFO:tasks.workunit.client.0.smithi089.stdout:removed './285/35' 2023-10-30T12:36:25.105 INFO:tasks.workunit.client.0.smithi089.stdout:removed './285/26' 2023-10-30T12:36:25.105 INFO:tasks.workunit.client.0.smithi089.stdout:removed './285/37' 2023-10-30T12:36:25.105 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './285' 2023-10-30T12:36:25.105 INFO:tasks.workunit.client.0.smithi089.stdout:removed './286/21' 2023-10-30T12:36:25.106 INFO:tasks.workunit.client.0.smithi089.stdout:removed './286/27' 2023-10-30T12:36:25.106 INFO:tasks.workunit.client.0.smithi089.stdout:removed './286/31' 2023-10-30T12:36:25.106 INFO:tasks.workunit.client.0.smithi089.stdout:removed './286/36' 2023-10-30T12:36:25.106 INFO:tasks.workunit.client.0.smithi089.stdout:removed './286/20' 2023-10-30T12:36:25.106 INFO:tasks.workunit.client.0.smithi089.stdout:removed './286/39' 2023-10-30T12:36:25.107 INFO:tasks.workunit.client.0.smithi089.stdout:removed './286/34' 2023-10-30T12:36:25.107 INFO:tasks.workunit.client.0.smithi089.stdout:removed './286/38' 2023-10-30T12:36:25.107 INFO:tasks.workunit.client.0.smithi089.stdout:removed './286/28' 2023-10-30T12:36:25.107 INFO:tasks.workunit.client.0.smithi089.stdout:removed './286/24' 2023-10-30T12:36:25.107 INFO:tasks.workunit.client.0.smithi089.stdout:removed './286/29' 2023-10-30T12:36:25.108 INFO:tasks.workunit.client.0.smithi089.stdout:removed './286/32' 2023-10-30T12:36:25.108 INFO:tasks.workunit.client.0.smithi089.stdout:removed './286/22' 2023-10-30T12:36:25.108 INFO:tasks.workunit.client.0.smithi089.stdout:removed './286/25' 2023-10-30T12:36:25.108 INFO:tasks.workunit.client.0.smithi089.stdout:removed './286/30' 2023-10-30T12:36:25.108 INFO:tasks.workunit.client.0.smithi089.stdout:removed './286/33' 2023-10-30T12:36:25.108 INFO:tasks.workunit.client.0.smithi089.stdout:removed './286/23' 2023-10-30T12:36:25.109 INFO:tasks.workunit.client.0.smithi089.stdout:removed './286/35' 2023-10-30T12:36:25.109 INFO:tasks.workunit.client.0.smithi089.stdout:removed './286/26' 2023-10-30T12:36:25.109 INFO:tasks.workunit.client.0.smithi089.stdout:removed './286/37' 2023-10-30T12:36:25.109 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './286' 2023-10-30T12:36:27.013 INFO:tasks.workunit.client.0.smithi089.stdout:removed './287/21' 2023-10-30T12:36:27.013 INFO:tasks.workunit.client.0.smithi089.stdout:removed './287/27' 2023-10-30T12:36:27.014 INFO:tasks.workunit.client.0.smithi089.stdout:removed './287/31' 2023-10-30T12:36:27.014 INFO:tasks.workunit.client.0.smithi089.stdout:removed './287/36' 2023-10-30T12:36:27.014 INFO:tasks.workunit.client.0.smithi089.stdout:removed './287/20' 2023-10-30T12:36:27.014 INFO:tasks.workunit.client.0.smithi089.stdout:removed './287/39' 2023-10-30T12:36:27.015 INFO:tasks.workunit.client.0.smithi089.stdout:removed './287/34' 2023-10-30T12:36:27.015 INFO:tasks.workunit.client.0.smithi089.stdout:removed './287/38' 2023-10-30T12:36:27.015 INFO:tasks.workunit.client.0.smithi089.stdout:removed './287/28' 2023-10-30T12:36:27.015 INFO:tasks.workunit.client.0.smithi089.stdout:removed './287/24' 2023-10-30T12:36:27.015 INFO:tasks.workunit.client.0.smithi089.stdout:removed './287/29' 2023-10-30T12:36:27.016 INFO:tasks.workunit.client.0.smithi089.stdout:removed './287/32' 2023-10-30T12:36:27.016 INFO:tasks.workunit.client.0.smithi089.stdout:removed './287/22' 2023-10-30T12:36:27.016 INFO:tasks.workunit.client.0.smithi089.stdout:removed './287/25' 2023-10-30T12:36:27.016 INFO:tasks.workunit.client.0.smithi089.stdout:removed './287/30' 2023-10-30T12:36:27.016 INFO:tasks.workunit.client.0.smithi089.stdout:removed './287/33' 2023-10-30T12:36:27.017 INFO:tasks.workunit.client.0.smithi089.stdout:removed './287/23' 2023-10-30T12:36:27.017 INFO:tasks.workunit.client.0.smithi089.stdout:removed './287/35' 2023-10-30T12:36:27.017 INFO:tasks.workunit.client.0.smithi089.stdout:removed './287/26' 2023-10-30T12:36:27.017 INFO:tasks.workunit.client.0.smithi089.stdout:removed './287/37' 2023-10-30T12:36:27.017 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './287' 2023-10-30T12:36:27.018 INFO:tasks.workunit.client.0.smithi089.stdout:removed './288/21' 2023-10-30T12:36:27.018 INFO:tasks.workunit.client.0.smithi089.stdout:removed './288/27' 2023-10-30T12:36:27.018 INFO:tasks.workunit.client.0.smithi089.stdout:removed './288/31' 2023-10-30T12:36:27.018 INFO:tasks.workunit.client.0.smithi089.stdout:removed './288/36' 2023-10-30T12:36:27.018 INFO:tasks.workunit.client.0.smithi089.stdout:removed './288/20' 2023-10-30T12:36:27.019 INFO:tasks.workunit.client.0.smithi089.stdout:removed './288/39' 2023-10-30T12:36:27.019 INFO:tasks.workunit.client.0.smithi089.stdout:removed './288/34' 2023-10-30T12:36:27.019 INFO:tasks.workunit.client.0.smithi089.stdout:removed './288/38' 2023-10-30T12:36:27.019 INFO:tasks.workunit.client.0.smithi089.stdout:removed './288/28' 2023-10-30T12:36:27.019 INFO:tasks.workunit.client.0.smithi089.stdout:removed './288/24' 2023-10-30T12:36:27.020 INFO:tasks.workunit.client.0.smithi089.stdout:removed './288/29' 2023-10-30T12:36:27.020 INFO:tasks.workunit.client.0.smithi089.stdout:removed './288/32' 2023-10-30T12:36:27.020 INFO:tasks.workunit.client.0.smithi089.stdout:removed './288/22' 2023-10-30T12:36:27.020 INFO:tasks.workunit.client.0.smithi089.stdout:removed './288/25' 2023-10-30T12:36:27.020 INFO:tasks.workunit.client.0.smithi089.stdout:removed './288/30' 2023-10-30T12:36:27.021 INFO:tasks.workunit.client.0.smithi089.stdout:removed './288/33' 2023-10-30T12:36:27.021 INFO:tasks.workunit.client.0.smithi089.stdout:removed './288/23' 2023-10-30T12:36:27.021 INFO:tasks.workunit.client.0.smithi089.stdout:removed './288/35' 2023-10-30T12:36:27.021 INFO:tasks.workunit.client.0.smithi089.stdout:removed './288/26' 2023-10-30T12:36:27.021 INFO:tasks.workunit.client.0.smithi089.stdout:removed './288/37' 2023-10-30T12:36:27.021 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './288' 2023-10-30T12:36:27.022 INFO:tasks.workunit.client.0.smithi089.stdout:removed './289/21' 2023-10-30T12:36:27.022 INFO:tasks.workunit.client.0.smithi089.stdout:removed './289/27' 2023-10-30T12:36:27.022 INFO:tasks.workunit.client.0.smithi089.stdout:removed './289/31' 2023-10-30T12:36:27.022 INFO:tasks.workunit.client.0.smithi089.stdout:removed './289/36' 2023-10-30T12:36:27.023 INFO:tasks.workunit.client.0.smithi089.stdout:removed './289/20' 2023-10-30T12:36:27.023 INFO:tasks.workunit.client.0.smithi089.stdout:removed './289/39' 2023-10-30T12:36:27.023 INFO:tasks.workunit.client.0.smithi089.stdout:removed './289/34' 2023-10-30T12:36:27.023 INFO:tasks.workunit.client.0.smithi089.stdout:removed './289/38' 2023-10-30T12:36:27.023 INFO:tasks.workunit.client.0.smithi089.stdout:removed './289/28' 2023-10-30T12:36:27.024 INFO:tasks.workunit.client.0.smithi089.stdout:removed './289/24' 2023-10-30T12:36:27.024 INFO:tasks.workunit.client.0.smithi089.stdout:removed './289/29' 2023-10-30T12:36:27.024 INFO:tasks.workunit.client.0.smithi089.stdout:removed './289/32' 2023-10-30T12:36:27.024 INFO:tasks.workunit.client.0.smithi089.stdout:removed './289/22' 2023-10-30T12:36:27.024 INFO:tasks.workunit.client.0.smithi089.stdout:removed './289/25' 2023-10-30T12:36:27.025 INFO:tasks.workunit.client.0.smithi089.stdout:removed './289/30' 2023-10-30T12:36:27.025 INFO:tasks.workunit.client.0.smithi089.stdout:removed './289/33' 2023-10-30T12:36:27.025 INFO:tasks.workunit.client.0.smithi089.stdout:removed './289/23' 2023-10-30T12:36:27.025 INFO:tasks.workunit.client.0.smithi089.stdout:removed './289/35' 2023-10-30T12:36:27.025 INFO:tasks.workunit.client.0.smithi089.stdout:removed './289/26' 2023-10-30T12:36:27.026 INFO:tasks.workunit.client.0.smithi089.stdout:removed './289/37' 2023-10-30T12:36:27.026 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './289' 2023-10-30T12:36:27.026 INFO:tasks.workunit.client.0.smithi089.stdout:removed './290/21' 2023-10-30T12:36:27.026 INFO:tasks.workunit.client.0.smithi089.stdout:removed './290/27' 2023-10-30T12:36:27.026 INFO:tasks.workunit.client.0.smithi089.stdout:removed './290/31' 2023-10-30T12:36:27.027 INFO:tasks.workunit.client.0.smithi089.stdout:removed './290/36' 2023-10-30T12:36:27.027 INFO:tasks.workunit.client.0.smithi089.stdout:removed './290/20' 2023-10-30T12:36:27.027 INFO:tasks.workunit.client.0.smithi089.stdout:removed './290/39' 2023-10-30T12:36:27.027 INFO:tasks.workunit.client.0.smithi089.stdout:removed './290/34' 2023-10-30T12:36:27.027 INFO:tasks.workunit.client.0.smithi089.stdout:removed './290/38' 2023-10-30T12:36:27.027 INFO:tasks.workunit.client.0.smithi089.stdout:removed './290/28' 2023-10-30T12:36:27.028 INFO:tasks.workunit.client.0.smithi089.stdout:removed './290/24' 2023-10-30T12:36:27.028 INFO:tasks.workunit.client.0.smithi089.stdout:removed './290/29' 2023-10-30T12:36:27.028 INFO:tasks.workunit.client.0.smithi089.stdout:removed './290/32' 2023-10-30T12:36:27.028 INFO:tasks.workunit.client.0.smithi089.stdout:removed './290/22' 2023-10-30T12:36:27.028 INFO:tasks.workunit.client.0.smithi089.stdout:removed './290/25' 2023-10-30T12:36:27.028 INFO:tasks.workunit.client.0.smithi089.stdout:removed './290/30' 2023-10-30T12:36:27.029 INFO:tasks.workunit.client.0.smithi089.stdout:removed './290/33' 2023-10-30T12:36:27.029 INFO:tasks.workunit.client.0.smithi089.stdout:removed './290/23' 2023-10-30T12:36:27.029 INFO:tasks.workunit.client.0.smithi089.stdout:removed './290/35' 2023-10-30T12:36:27.029 INFO:tasks.workunit.client.0.smithi089.stdout:removed './290/26' 2023-10-30T12:36:27.029 INFO:tasks.workunit.client.0.smithi089.stdout:removed './290/37' 2023-10-30T12:36:27.030 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './290' 2023-10-30T12:36:27.030 INFO:tasks.workunit.client.0.smithi089.stdout:removed './291/21' 2023-10-30T12:36:27.030 INFO:tasks.workunit.client.0.smithi089.stdout:removed './291/27' 2023-10-30T12:36:27.030 INFO:tasks.workunit.client.0.smithi089.stdout:removed './291/31' 2023-10-30T12:36:27.030 INFO:tasks.workunit.client.0.smithi089.stdout:removed './291/36' 2023-10-30T12:36:27.031 INFO:tasks.workunit.client.0.smithi089.stdout:removed './291/20' 2023-10-30T12:36:27.031 INFO:tasks.workunit.client.0.smithi089.stdout:removed './291/39' 2023-10-30T12:36:27.031 INFO:tasks.workunit.client.0.smithi089.stdout:removed './291/34' 2023-10-30T12:36:27.031 INFO:tasks.workunit.client.0.smithi089.stdout:removed './291/38' 2023-10-30T12:36:27.031 INFO:tasks.workunit.client.0.smithi089.stdout:removed './291/28' 2023-10-30T12:36:27.032 INFO:tasks.workunit.client.0.smithi089.stdout:removed './291/24' 2023-10-30T12:36:27.032 INFO:tasks.workunit.client.0.smithi089.stdout:removed './291/29' 2023-10-30T12:36:27.032 INFO:tasks.workunit.client.0.smithi089.stdout:removed './291/32' 2023-10-30T12:36:27.032 INFO:tasks.workunit.client.0.smithi089.stdout:removed './291/22' 2023-10-30T12:36:27.032 INFO:tasks.workunit.client.0.smithi089.stdout:removed './291/25' 2023-10-30T12:36:27.033 INFO:tasks.workunit.client.0.smithi089.stdout:removed './291/30' 2023-10-30T12:36:27.033 INFO:tasks.workunit.client.0.smithi089.stdout:removed './291/33' 2023-10-30T12:36:27.033 INFO:tasks.workunit.client.0.smithi089.stdout:removed './291/23' 2023-10-30T12:36:27.033 INFO:tasks.workunit.client.0.smithi089.stdout:removed './291/35' 2023-10-30T12:36:27.033 INFO:tasks.workunit.client.0.smithi089.stdout:removed './291/26' 2023-10-30T12:36:27.033 INFO:tasks.workunit.client.0.smithi089.stdout:removed './291/37' 2023-10-30T12:36:27.034 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './291' 2023-10-30T12:36:27.034 INFO:tasks.workunit.client.0.smithi089.stdout:removed './292/21' 2023-10-30T12:36:27.034 INFO:tasks.workunit.client.0.smithi089.stdout:removed './292/27' 2023-10-30T12:36:27.034 INFO:tasks.workunit.client.0.smithi089.stdout:removed './292/31' 2023-10-30T12:36:27.035 INFO:tasks.workunit.client.0.smithi089.stdout:removed './292/36' 2023-10-30T12:36:27.035 INFO:tasks.workunit.client.0.smithi089.stdout:removed './292/20' 2023-10-30T12:36:27.035 INFO:tasks.workunit.client.0.smithi089.stdout:removed './292/39' 2023-10-30T12:36:27.036 INFO:tasks.workunit.client.0.smithi089.stdout:removed './292/34' 2023-10-30T12:36:27.036 INFO:tasks.workunit.client.0.smithi089.stdout:removed './292/38' 2023-10-30T12:36:27.036 INFO:tasks.workunit.client.0.smithi089.stdout:removed './292/28' 2023-10-30T12:36:27.036 INFO:tasks.workunit.client.0.smithi089.stdout:removed './292/24' 2023-10-30T12:36:27.036 INFO:tasks.workunit.client.0.smithi089.stdout:removed './292/29' 2023-10-30T12:36:27.036 INFO:tasks.workunit.client.0.smithi089.stdout:removed './292/32' 2023-10-30T12:36:27.037 INFO:tasks.workunit.client.0.smithi089.stdout:removed './292/22' 2023-10-30T12:36:27.037 INFO:tasks.workunit.client.0.smithi089.stdout:removed './292/25' 2023-10-30T12:36:27.037 INFO:tasks.workunit.client.0.smithi089.stdout:removed './292/30' 2023-10-30T12:36:27.037 INFO:tasks.workunit.client.0.smithi089.stdout:removed './292/33' 2023-10-30T12:36:27.037 INFO:tasks.workunit.client.0.smithi089.stdout:removed './292/23' 2023-10-30T12:36:27.038 INFO:tasks.workunit.client.0.smithi089.stdout:removed './292/35' 2023-10-30T12:36:27.038 INFO:tasks.workunit.client.0.smithi089.stdout:removed './292/26' 2023-10-30T12:36:27.038 INFO:tasks.workunit.client.0.smithi089.stdout:removed './292/37' 2023-10-30T12:36:27.038 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './292' 2023-10-30T12:36:27.039 INFO:tasks.workunit.client.0.smithi089.stdout:removed './293/21' 2023-10-30T12:36:27.039 INFO:tasks.workunit.client.0.smithi089.stdout:removed './293/27' 2023-10-30T12:36:27.039 INFO:tasks.workunit.client.0.smithi089.stdout:removed './293/31' 2023-10-30T12:36:27.039 INFO:tasks.workunit.client.0.smithi089.stdout:removed './293/36' 2023-10-30T12:36:27.039 INFO:tasks.workunit.client.0.smithi089.stdout:removed './293/20' 2023-10-30T12:36:27.040 INFO:tasks.workunit.client.0.smithi089.stdout:removed './293/39' 2023-10-30T12:36:27.040 INFO:tasks.workunit.client.0.smithi089.stdout:removed './293/34' 2023-10-30T12:36:27.040 INFO:tasks.workunit.client.0.smithi089.stdout:removed './293/38' 2023-10-30T12:36:27.040 INFO:tasks.workunit.client.0.smithi089.stdout:removed './293/28' 2023-10-30T12:36:27.040 INFO:tasks.workunit.client.0.smithi089.stdout:removed './293/24' 2023-10-30T12:36:27.041 INFO:tasks.workunit.client.0.smithi089.stdout:removed './293/29' 2023-10-30T12:36:27.041 INFO:tasks.workunit.client.0.smithi089.stdout:removed './293/32' 2023-10-30T12:36:27.041 INFO:tasks.workunit.client.0.smithi089.stdout:removed './293/22' 2023-10-30T12:36:27.041 INFO:tasks.workunit.client.0.smithi089.stdout:removed './293/25' 2023-10-30T12:36:27.041 INFO:tasks.workunit.client.0.smithi089.stdout:removed './293/30' 2023-10-30T12:36:27.042 INFO:tasks.workunit.client.0.smithi089.stdout:removed './293/33' 2023-10-30T12:36:27.042 INFO:tasks.workunit.client.0.smithi089.stdout:removed './293/23' 2023-10-30T12:36:27.042 INFO:tasks.workunit.client.0.smithi089.stdout:removed './293/35' 2023-10-30T12:36:27.042 INFO:tasks.workunit.client.0.smithi089.stdout:removed './293/26' 2023-10-30T12:36:27.043 INFO:tasks.workunit.client.0.smithi089.stdout:removed './293/37' 2023-10-30T12:36:27.043 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './293' 2023-10-30T12:36:27.043 INFO:tasks.workunit.client.0.smithi089.stdout:removed './294/21' 2023-10-30T12:36:27.043 INFO:tasks.workunit.client.0.smithi089.stdout:removed './294/27' 2023-10-30T12:36:27.043 INFO:tasks.workunit.client.0.smithi089.stdout:removed './294/31' 2023-10-30T12:36:27.043 INFO:tasks.workunit.client.0.smithi089.stdout:removed './294/36' 2023-10-30T12:36:27.044 INFO:tasks.workunit.client.0.smithi089.stdout:removed './294/20' 2023-10-30T12:36:27.044 INFO:tasks.workunit.client.0.smithi089.stdout:removed './294/39' 2023-10-30T12:36:27.044 INFO:tasks.workunit.client.0.smithi089.stdout:removed './294/34' 2023-10-30T12:36:27.044 INFO:tasks.workunit.client.0.smithi089.stdout:removed './294/38' 2023-10-30T12:36:27.044 INFO:tasks.workunit.client.0.smithi089.stdout:removed './294/28' 2023-10-30T12:36:27.045 INFO:tasks.workunit.client.0.smithi089.stdout:removed './294/24' 2023-10-30T12:36:27.045 INFO:tasks.workunit.client.0.smithi089.stdout:removed './294/29' 2023-10-30T12:36:27.045 INFO:tasks.workunit.client.0.smithi089.stdout:removed './294/32' 2023-10-30T12:36:27.045 INFO:tasks.workunit.client.0.smithi089.stdout:removed './294/22' 2023-10-30T12:36:27.046 INFO:tasks.workunit.client.0.smithi089.stdout:removed './294/25' 2023-10-30T12:36:27.046 INFO:tasks.workunit.client.0.smithi089.stdout:removed './294/30' 2023-10-30T12:36:27.046 INFO:tasks.workunit.client.0.smithi089.stdout:removed './294/33' 2023-10-30T12:36:27.046 INFO:tasks.workunit.client.0.smithi089.stdout:removed './294/23' 2023-10-30T12:36:27.046 INFO:tasks.workunit.client.0.smithi089.stdout:removed './294/35' 2023-10-30T12:36:27.047 INFO:tasks.workunit.client.0.smithi089.stdout:removed './294/26' 2023-10-30T12:36:27.047 INFO:tasks.workunit.client.0.smithi089.stdout:removed './294/37' 2023-10-30T12:36:27.047 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './294' 2023-10-30T12:36:27.047 INFO:tasks.workunit.client.0.smithi089.stdout:removed './295/21' 2023-10-30T12:36:27.047 INFO:tasks.workunit.client.0.smithi089.stdout:removed './295/27' 2023-10-30T12:36:27.048 INFO:tasks.workunit.client.0.smithi089.stdout:removed './295/31' 2023-10-30T12:36:27.048 INFO:tasks.workunit.client.0.smithi089.stdout:removed './295/36' 2023-10-30T12:36:27.048 INFO:tasks.workunit.client.0.smithi089.stdout:removed './295/20' 2023-10-30T12:36:27.048 INFO:tasks.workunit.client.0.smithi089.stdout:removed './295/39' 2023-10-30T12:36:27.048 INFO:tasks.workunit.client.0.smithi089.stdout:removed './295/34' 2023-10-30T12:36:27.049 INFO:tasks.workunit.client.0.smithi089.stdout:removed './295/38' 2023-10-30T12:36:27.049 INFO:tasks.workunit.client.0.smithi089.stdout:removed './295/28' 2023-10-30T12:36:27.049 INFO:tasks.workunit.client.0.smithi089.stdout:removed './295/24' 2023-10-30T12:36:27.049 INFO:tasks.workunit.client.0.smithi089.stdout:removed './295/29' 2023-10-30T12:36:27.049 INFO:tasks.workunit.client.0.smithi089.stdout:removed './295/32' 2023-10-30T12:36:27.049 INFO:tasks.workunit.client.0.smithi089.stdout:removed './295/22' 2023-10-30T12:36:27.050 INFO:tasks.workunit.client.0.smithi089.stdout:removed './295/25' 2023-10-30T12:36:27.050 INFO:tasks.workunit.client.0.smithi089.stdout:removed './295/30' 2023-10-30T12:36:27.050 INFO:tasks.workunit.client.0.smithi089.stdout:removed './295/33' 2023-10-30T12:36:27.050 INFO:tasks.workunit.client.0.smithi089.stdout:removed './295/23' 2023-10-30T12:36:27.051 INFO:tasks.workunit.client.0.smithi089.stdout:removed './295/35' 2023-10-30T12:36:27.051 INFO:tasks.workunit.client.0.smithi089.stdout:removed './295/26' 2023-10-30T12:36:27.051 INFO:tasks.workunit.client.0.smithi089.stdout:removed './295/37' 2023-10-30T12:36:27.052 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './295' 2023-10-30T12:36:27.052 INFO:tasks.workunit.client.0.smithi089.stdout:removed './296/21' 2023-10-30T12:36:27.053 INFO:tasks.workunit.client.0.smithi089.stdout:removed './296/27' 2023-10-30T12:36:27.053 INFO:tasks.workunit.client.0.smithi089.stdout:removed './296/31' 2023-10-30T12:36:27.053 INFO:tasks.workunit.client.0.smithi089.stdout:removed './296/36' 2023-10-30T12:36:27.054 INFO:tasks.workunit.client.0.smithi089.stdout:removed './296/20' 2023-10-30T12:36:27.054 INFO:tasks.workunit.client.0.smithi089.stdout:removed './296/39' 2023-10-30T12:36:27.054 INFO:tasks.workunit.client.0.smithi089.stdout:removed './296/34' 2023-10-30T12:36:27.055 INFO:tasks.workunit.client.0.smithi089.stdout:removed './296/38' 2023-10-30T12:36:27.055 INFO:tasks.workunit.client.0.smithi089.stdout:removed './296/28' 2023-10-30T12:36:27.056 INFO:tasks.workunit.client.0.smithi089.stdout:removed './296/24' 2023-10-30T12:36:27.056 INFO:tasks.workunit.client.0.smithi089.stdout:removed './296/29' 2023-10-30T12:36:27.056 INFO:tasks.workunit.client.0.smithi089.stdout:removed './296/32' 2023-10-30T12:36:27.057 INFO:tasks.workunit.client.0.smithi089.stdout:removed './296/22' 2023-10-30T12:36:27.057 INFO:tasks.workunit.client.0.smithi089.stdout:removed './296/25' 2023-10-30T12:36:27.057 INFO:tasks.workunit.client.0.smithi089.stdout:removed './296/30' 2023-10-30T12:36:27.058 INFO:tasks.workunit.client.0.smithi089.stdout:removed './296/33' 2023-10-30T12:36:27.058 INFO:tasks.workunit.client.0.smithi089.stdout:removed './296/23' 2023-10-30T12:36:27.058 INFO:tasks.workunit.client.0.smithi089.stdout:removed './296/35' 2023-10-30T12:36:27.059 INFO:tasks.workunit.client.0.smithi089.stdout:removed './296/26' 2023-10-30T12:36:27.059 INFO:tasks.workunit.client.0.smithi089.stdout:removed './296/37' 2023-10-30T12:36:27.060 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './296' 2023-10-30T12:36:27.060 INFO:tasks.workunit.client.0.smithi089.stdout:removed './297/21' 2023-10-30T12:36:27.060 INFO:tasks.workunit.client.0.smithi089.stdout:removed './297/27' 2023-10-30T12:36:28.379 INFO:tasks.workunit.client.0.smithi089.stdout:removed './297/31' 2023-10-30T12:36:28.379 INFO:tasks.workunit.client.0.smithi089.stdout:removed './297/36' 2023-10-30T12:36:28.380 INFO:tasks.workunit.client.0.smithi089.stdout:removed './297/20' 2023-10-30T12:36:28.380 INFO:tasks.workunit.client.0.smithi089.stdout:removed './297/39' 2023-10-30T12:36:28.380 INFO:tasks.workunit.client.0.smithi089.stdout:removed './297/34' 2023-10-30T12:36:28.380 INFO:tasks.workunit.client.0.smithi089.stdout:removed './297/38' 2023-10-30T12:36:28.380 INFO:tasks.workunit.client.0.smithi089.stdout:removed './297/28' 2023-10-30T12:36:28.381 INFO:tasks.workunit.client.0.smithi089.stdout:removed './297/24' 2023-10-30T12:36:28.381 INFO:tasks.workunit.client.0.smithi089.stdout:removed './297/29' 2023-10-30T12:36:28.381 INFO:tasks.workunit.client.0.smithi089.stdout:removed './297/32' 2023-10-30T12:36:28.381 INFO:tasks.workunit.client.0.smithi089.stdout:removed './297/22' 2023-10-30T12:36:28.381 INFO:tasks.workunit.client.0.smithi089.stdout:removed './297/25' 2023-10-30T12:36:28.382 INFO:tasks.workunit.client.0.smithi089.stdout:removed './297/30' 2023-10-30T12:36:28.382 INFO:tasks.workunit.client.0.smithi089.stdout:removed './297/33' 2023-10-30T12:36:28.382 INFO:tasks.workunit.client.0.smithi089.stdout:removed './297/23' 2023-10-30T12:36:28.382 INFO:tasks.workunit.client.0.smithi089.stdout:removed './297/35' 2023-10-30T12:36:28.382 INFO:tasks.workunit.client.0.smithi089.stdout:removed './297/26' 2023-10-30T12:36:28.383 INFO:tasks.workunit.client.0.smithi089.stdout:removed './297/37' 2023-10-30T12:36:28.383 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './297' 2023-10-30T12:36:28.383 INFO:tasks.workunit.client.0.smithi089.stdout:removed './298/21' 2023-10-30T12:36:28.383 INFO:tasks.workunit.client.0.smithi089.stdout:removed './298/27' 2023-10-30T12:36:28.383 INFO:tasks.workunit.client.0.smithi089.stdout:removed './298/31' 2023-10-30T12:36:28.384 INFO:tasks.workunit.client.0.smithi089.stdout:removed './298/36' 2023-10-30T12:36:28.384 INFO:tasks.workunit.client.0.smithi089.stdout:removed './298/20' 2023-10-30T12:36:28.384 INFO:tasks.workunit.client.0.smithi089.stdout:removed './298/39' 2023-10-30T12:36:28.384 INFO:tasks.workunit.client.0.smithi089.stdout:removed './298/34' 2023-10-30T12:36:28.384 INFO:tasks.workunit.client.0.smithi089.stdout:removed './298/38' 2023-10-30T12:36:28.384 INFO:tasks.workunit.client.0.smithi089.stdout:removed './298/28' 2023-10-30T12:36:28.385 INFO:tasks.workunit.client.0.smithi089.stdout:removed './298/24' 2023-10-30T12:36:28.385 INFO:tasks.workunit.client.0.smithi089.stdout:removed './298/29' 2023-10-30T12:36:28.385 INFO:tasks.workunit.client.0.smithi089.stdout:removed './298/32' 2023-10-30T12:36:28.385 INFO:tasks.workunit.client.0.smithi089.stdout:removed './298/22' 2023-10-30T12:36:28.385 INFO:tasks.workunit.client.0.smithi089.stdout:removed './298/25' 2023-10-30T12:36:28.386 INFO:tasks.workunit.client.0.smithi089.stdout:removed './298/30' 2023-10-30T12:36:28.386 INFO:tasks.workunit.client.0.smithi089.stdout:removed './298/33' 2023-10-30T12:36:28.386 INFO:tasks.workunit.client.0.smithi089.stdout:removed './298/23' 2023-10-30T12:36:28.386 INFO:tasks.workunit.client.0.smithi089.stdout:removed './298/35' 2023-10-30T12:36:28.386 INFO:tasks.workunit.client.0.smithi089.stdout:removed './298/26' 2023-10-30T12:36:28.387 INFO:tasks.workunit.client.0.smithi089.stdout:removed './298/37' 2023-10-30T12:36:28.387 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './298' 2023-10-30T12:36:28.387 INFO:tasks.workunit.client.0.smithi089.stdout:removed './299/21' 2023-10-30T12:36:28.387 INFO:tasks.workunit.client.0.smithi089.stdout:removed './299/27' 2023-10-30T12:36:28.387 INFO:tasks.workunit.client.0.smithi089.stdout:removed './299/31' 2023-10-30T12:36:28.388 INFO:tasks.workunit.client.0.smithi089.stdout:removed './299/36' 2023-10-30T12:36:28.388 INFO:tasks.workunit.client.0.smithi089.stdout:removed './299/20' 2023-10-30T12:36:28.388 INFO:tasks.workunit.client.0.smithi089.stdout:removed './299/39' 2023-10-30T12:36:28.388 INFO:tasks.workunit.client.0.smithi089.stdout:removed './299/34' 2023-10-30T12:36:28.389 INFO:tasks.workunit.client.0.smithi089.stdout:removed './299/38' 2023-10-30T12:36:28.389 INFO:tasks.workunit.client.0.smithi089.stdout:removed './299/28' 2023-10-30T12:36:28.389 INFO:tasks.workunit.client.0.smithi089.stdout:removed './299/24' 2023-10-30T12:36:28.389 INFO:tasks.workunit.client.0.smithi089.stdout:removed './299/29' 2023-10-30T12:36:28.389 INFO:tasks.workunit.client.0.smithi089.stdout:removed './299/32' 2023-10-30T12:36:28.390 INFO:tasks.workunit.client.0.smithi089.stdout:removed './299/22' 2023-10-30T12:36:28.390 INFO:tasks.workunit.client.0.smithi089.stdout:removed './299/25' 2023-10-30T12:36:28.390 INFO:tasks.workunit.client.0.smithi089.stdout:removed './299/30' 2023-10-30T12:36:28.390 INFO:tasks.workunit.client.0.smithi089.stdout:removed './299/33' 2023-10-30T12:36:28.390 INFO:tasks.workunit.client.0.smithi089.stdout:removed './299/23' 2023-10-30T12:36:28.391 INFO:tasks.workunit.client.0.smithi089.stdout:removed './299/35' 2023-10-30T12:36:28.391 INFO:tasks.workunit.client.0.smithi089.stdout:removed './299/26' 2023-10-30T12:36:28.391 INFO:tasks.workunit.client.0.smithi089.stdout:removed './299/37' 2023-10-30T12:36:28.391 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './299' 2023-10-30T12:36:28.391 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/166/19' 2023-10-30T12:36:28.392 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/166/15' 2023-10-30T12:36:28.392 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/166/20' 2023-10-30T12:36:28.392 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/166/12' 2023-10-30T12:36:28.392 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/166/10' 2023-10-30T12:36:28.392 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/166/11' 2023-10-30T12:36:28.393 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/166/16' 2023-10-30T12:36:28.393 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/166/14' 2023-10-30T12:36:28.393 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/166/18' 2023-10-30T12:36:28.393 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/166/13' 2023-10-30T12:36:28.393 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/166/17' 2023-10-30T12:36:28.393 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/166' 2023-10-30T12:36:28.394 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/147/19' 2023-10-30T12:36:28.394 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/147/15' 2023-10-30T12:36:28.394 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/147/20' 2023-10-30T12:36:28.394 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/147/12' 2023-10-30T12:36:28.395 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/147/10' 2023-10-30T12:36:28.395 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/147/11' 2023-10-30T12:36:28.395 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/147/16' 2023-10-30T12:36:28.395 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/147/14' 2023-10-30T12:36:28.395 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/147/18' 2023-10-30T12:36:28.396 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/147/13' 2023-10-30T12:36:28.396 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/147/17' 2023-10-30T12:36:28.396 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/147' 2023-10-30T12:36:28.396 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/151/19' 2023-10-30T12:36:28.396 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/151/15' 2023-10-30T12:36:28.397 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/151/20' 2023-10-30T12:36:28.397 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/151/12' 2023-10-30T12:36:28.397 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/151/10' 2023-10-30T12:36:28.397 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/151/11' 2023-10-30T12:36:28.397 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/151/16' 2023-10-30T12:36:28.398 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/151/14' 2023-10-30T12:36:28.398 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/151/18' 2023-10-30T12:36:28.398 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/151/13' 2023-10-30T12:36:28.398 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/151/17' 2023-10-30T12:36:28.398 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/151' 2023-10-30T12:36:28.399 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/170/19' 2023-10-30T12:36:28.399 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/170/15' 2023-10-30T12:36:28.399 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/170/20' 2023-10-30T12:36:28.399 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/170/12' 2023-10-30T12:36:28.399 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/170/10' 2023-10-30T12:36:28.400 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/170/11' 2023-10-30T12:36:28.400 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/170/16' 2023-10-30T12:36:28.400 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/170/14' 2023-10-30T12:36:28.400 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/170/18' 2023-10-30T12:36:28.400 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/170/13' 2023-10-30T12:36:28.401 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/170/17' 2023-10-30T12:36:28.401 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/170' 2023-10-30T12:36:28.401 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/167/19' 2023-10-30T12:36:28.401 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/167/15' 2023-10-30T12:36:28.401 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/167/20' 2023-10-30T12:36:28.401 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/167/12' 2023-10-30T12:36:28.402 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/167/10' 2023-10-30T12:36:28.402 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/167/11' 2023-10-30T12:36:28.402 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/167/16' 2023-10-30T12:36:28.402 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/167/14' 2023-10-30T12:36:28.402 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/167/18' 2023-10-30T12:36:28.403 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/167/13' 2023-10-30T12:36:28.403 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/167/17' 2023-10-30T12:36:28.403 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/167' 2023-10-30T12:36:28.403 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/140/19' 2023-10-30T12:36:28.403 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/140/15' 2023-10-30T12:36:28.404 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/140/20' 2023-10-30T12:36:28.404 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/140/12' 2023-10-30T12:36:28.404 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/140/10' 2023-10-30T12:36:28.404 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/140/11' 2023-10-30T12:36:28.404 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/140/16' 2023-10-30T12:36:28.405 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/140/14' 2023-10-30T12:36:28.405 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/140/18' 2023-10-30T12:36:28.405 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/140/13' 2023-10-30T12:36:28.405 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/140/17' 2023-10-30T12:36:28.405 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/140' 2023-10-30T12:36:28.405 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/142/19' 2023-10-30T12:36:28.406 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/142/15' 2023-10-30T12:36:28.406 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/142/20' 2023-10-30T12:36:28.406 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/142/12' 2023-10-30T12:36:28.406 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/142/10' 2023-10-30T12:36:32.829 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/142/11' 2023-10-30T12:36:32.829 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/142/16' 2023-10-30T12:36:32.830 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/142/14' 2023-10-30T12:36:32.830 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/142/18' 2023-10-30T12:36:32.830 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/142/13' 2023-10-30T12:36:32.830 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/142/17' 2023-10-30T12:36:32.830 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/142' 2023-10-30T12:36:32.831 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/150/19' 2023-10-30T12:36:32.831 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/150/15' 2023-10-30T12:36:32.831 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/150/20' 2023-10-30T12:36:32.831 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/150/12' 2023-10-30T12:36:32.831 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/150/10' 2023-10-30T12:36:32.832 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/150/11' 2023-10-30T12:36:32.832 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/150/16' 2023-10-30T12:36:32.832 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/150/14' 2023-10-30T12:36:32.832 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/150/18' 2023-10-30T12:36:32.832 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/150/13' 2023-10-30T12:36:32.833 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/150/17' 2023-10-30T12:36:32.833 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/150' 2023-10-30T12:36:32.833 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/105/19' 2023-10-30T12:36:32.833 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/105/15' 2023-10-30T12:36:32.833 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/105/20' 2023-10-30T12:36:32.834 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/105/12' 2023-10-30T12:36:32.834 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/105/10' 2023-10-30T12:36:32.834 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/105/11' 2023-10-30T12:36:32.834 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/105/16' 2023-10-30T12:36:32.834 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/105/14' 2023-10-30T12:36:32.835 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/105/18' 2023-10-30T12:36:32.835 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/105/13' 2023-10-30T12:36:32.835 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/105/17' 2023-10-30T12:36:32.835 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/105' 2023-10-30T12:36:32.835 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/149/19' 2023-10-30T12:36:32.836 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/149/15' 2023-10-30T12:36:32.836 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/149/20' 2023-10-30T12:36:32.836 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/149/12' 2023-10-30T12:36:32.836 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/149/10' 2023-10-30T12:36:32.836 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/149/11' 2023-10-30T12:36:32.837 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/149/16' 2023-10-30T12:36:32.837 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/149/14' 2023-10-30T12:36:32.837 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/149/18' 2023-10-30T12:36:32.837 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/149/13' 2023-10-30T12:36:32.837 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/149/17' 2023-10-30T12:36:32.838 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/149' 2023-10-30T12:36:32.838 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/109/19' 2023-10-30T12:36:32.838 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/109/15' 2023-10-30T12:36:32.838 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/109/20' 2023-10-30T12:36:32.838 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/109/12' 2023-10-30T12:36:32.839 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/109/10' 2023-10-30T12:36:32.839 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/109/11' 2023-10-30T12:36:32.839 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/109/16' 2023-10-30T12:36:32.839 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/109/14' 2023-10-30T12:36:32.839 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/109/18' 2023-10-30T12:36:32.839 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/109/13' 2023-10-30T12:36:32.840 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/109/17' 2023-10-30T12:36:32.840 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/109' 2023-10-30T12:36:32.840 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/158/19' 2023-10-30T12:36:32.840 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/158/15' 2023-10-30T12:36:32.840 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/158/20' 2023-10-30T12:36:32.841 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/158/12' 2023-10-30T12:36:32.841 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/158/10' 2023-10-30T12:36:32.841 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/158/11' 2023-10-30T12:36:32.841 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/158/16' 2023-10-30T12:36:32.841 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/158/14' 2023-10-30T12:36:32.842 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/158/18' 2023-10-30T12:36:32.842 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/158/13' 2023-10-30T12:36:32.842 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/158/17' 2023-10-30T12:36:32.842 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/158' 2023-10-30T12:36:32.842 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/106/19' 2023-10-30T12:36:32.843 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/106/15' 2023-10-30T12:36:32.843 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/106/20' 2023-10-30T12:36:32.843 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/106/12' 2023-10-30T12:36:32.843 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/106/10' 2023-10-30T12:36:32.843 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/106/11' 2023-10-30T12:36:32.844 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/106/16' 2023-10-30T12:36:32.844 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/106/14' 2023-10-30T12:36:32.844 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/106/18' 2023-10-30T12:36:32.844 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/106/13' 2023-10-30T12:36:32.844 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/106/17' 2023-10-30T12:36:32.844 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/106' 2023-10-30T12:36:32.845 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/189/19' 2023-10-30T12:36:32.845 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/189/15' 2023-10-30T12:36:32.845 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/189/20' 2023-10-30T12:36:32.845 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/189/12' 2023-10-30T12:36:32.845 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/189/10' 2023-10-30T12:36:32.846 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/189/11' 2023-10-30T12:36:32.846 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/189/16' 2023-10-30T12:36:32.846 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/189/14' 2023-10-30T12:36:32.846 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/189/18' 2023-10-30T12:36:32.846 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/189/13' 2023-10-30T12:36:32.847 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/189/17' 2023-10-30T12:36:32.847 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/189' 2023-10-30T12:36:32.847 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/133/19' 2023-10-30T12:36:32.847 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/133/15' 2023-10-30T12:36:32.847 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/133/20' 2023-10-30T12:36:32.848 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/133/12' 2023-10-30T12:36:32.848 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/133/10' 2023-10-30T12:36:32.848 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/133/11' 2023-10-30T12:36:32.848 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/133/16' 2023-10-30T12:36:32.848 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/133/14' 2023-10-30T12:36:32.848 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/133/18' 2023-10-30T12:36:32.849 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/133/13' 2023-10-30T12:36:32.849 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/133/17' 2023-10-30T12:36:32.849 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/133' 2023-10-30T12:36:32.849 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/173/19' 2023-10-30T12:36:32.849 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/173/15' 2023-10-30T12:36:32.850 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/173/20' 2023-10-30T12:36:32.850 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/173/12' 2023-10-30T12:36:32.850 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/173/10' 2023-10-30T12:36:32.850 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/173/11' 2023-10-30T12:36:34.010 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/173/16' 2023-10-30T12:36:34.010 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/173/14' 2023-10-30T12:36:34.010 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/173/18' 2023-10-30T12:36:34.011 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/173/13' 2023-10-30T12:36:34.011 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/173/17' 2023-10-30T12:36:34.011 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/173' 2023-10-30T12:36:34.011 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/174/19' 2023-10-30T12:36:34.012 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/174/15' 2023-10-30T12:36:34.012 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/174/20' 2023-10-30T12:36:34.012 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/174/12' 2023-10-30T12:36:34.012 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/174/10' 2023-10-30T12:36:34.012 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/174/11' 2023-10-30T12:36:34.013 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/174/16' 2023-10-30T12:36:34.013 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/174/14' 2023-10-30T12:36:34.013 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/174/18' 2023-10-30T12:36:34.013 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/174/13' 2023-10-30T12:36:34.013 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/174/17' 2023-10-30T12:36:34.014 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/174' 2023-10-30T12:36:34.014 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/155/19' 2023-10-30T12:36:34.014 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/155/15' 2023-10-30T12:36:34.014 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/155/20' 2023-10-30T12:36:34.014 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/155/12' 2023-10-30T12:36:34.015 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/155/10' 2023-10-30T12:36:34.015 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/155/11' 2023-10-30T12:36:34.015 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/155/16' 2023-10-30T12:36:34.015 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/155/14' 2023-10-30T12:36:34.015 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/155/18' 2023-10-30T12:36:34.016 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/155/13' 2023-10-30T12:36:34.016 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/155/17' 2023-10-30T12:36:34.016 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/155' 2023-10-30T12:36:34.016 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/114/19' 2023-10-30T12:36:34.016 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/114/15' 2023-10-30T12:36:34.017 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/114/20' 2023-10-30T12:36:34.017 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/114/12' 2023-10-30T12:36:34.017 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/114/10' 2023-10-30T12:36:34.017 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/114/11' 2023-10-30T12:36:34.017 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/114/16' 2023-10-30T12:36:34.018 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/114/14' 2023-10-30T12:36:34.018 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/114/18' 2023-10-30T12:36:34.018 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/114/13' 2023-10-30T12:36:34.018 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/114/17' 2023-10-30T12:36:34.018 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/114' 2023-10-30T12:36:34.019 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/159/19' 2023-10-30T12:36:34.019 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/159/15' 2023-10-30T12:36:34.019 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/159/20' 2023-10-30T12:36:34.019 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/159/12' 2023-10-30T12:36:34.019 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/159/10' 2023-10-30T12:36:34.019 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/159/11' 2023-10-30T12:36:34.020 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/159/16' 2023-10-30T12:36:34.020 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/159/14' 2023-10-30T12:36:34.020 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/159/18' 2023-10-30T12:36:34.020 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/159/13' 2023-10-30T12:36:34.020 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/159/17' 2023-10-30T12:36:34.021 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/159' 2023-10-30T12:36:34.021 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/187/19' 2023-10-30T12:36:34.021 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/187/15' 2023-10-30T12:36:34.021 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/187/20' 2023-10-30T12:36:34.021 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/187/12' 2023-10-30T12:36:34.021 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/187/10' 2023-10-30T12:36:34.022 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/187/11' 2023-10-30T12:36:34.022 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/187/16' 2023-10-30T12:36:34.022 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/187/14' 2023-10-30T12:36:34.022 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/187/18' 2023-10-30T12:36:34.022 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/187/13' 2023-10-30T12:36:34.023 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/187/17' 2023-10-30T12:36:34.023 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/187' 2023-10-30T12:36:34.023 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/185/19' 2023-10-30T12:36:34.023 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/185/15' 2023-10-30T12:36:34.023 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/185/20' 2023-10-30T12:36:34.023 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/185/12' 2023-10-30T12:36:34.024 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/185/10' 2023-10-30T12:36:34.024 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/185/11' 2023-10-30T12:36:34.024 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/185/16' 2023-10-30T12:36:34.024 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/185/14' 2023-10-30T12:36:34.024 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/185/18' 2023-10-30T12:36:34.025 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/185/13' 2023-10-30T12:36:34.025 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/185/17' 2023-10-30T12:36:34.025 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/185' 2023-10-30T12:36:34.025 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/120/19' 2023-10-30T12:36:34.025 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/120/15' 2023-10-30T12:36:34.026 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/120/20' 2023-10-30T12:36:34.026 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/120/12' 2023-10-30T12:36:34.026 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/120/10' 2023-10-30T12:36:34.026 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/120/11' 2023-10-30T12:36:34.026 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/120/16' 2023-10-30T12:36:34.027 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/120/14' 2023-10-30T12:36:34.027 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/120/18' 2023-10-30T12:36:34.027 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/120/13' 2023-10-30T12:36:34.027 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/120/17' 2023-10-30T12:36:34.027 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/120' 2023-10-30T12:36:34.028 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/110/19' 2023-10-30T12:36:34.028 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/110/15' 2023-10-30T12:36:34.028 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/110/20' 2023-10-30T12:36:34.028 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/110/12' 2023-10-30T12:36:34.028 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/110/10' 2023-10-30T12:36:34.028 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/110/11' 2023-10-30T12:36:34.029 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/110/16' 2023-10-30T12:36:34.029 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/110/14' 2023-10-30T12:36:34.029 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/110/18' 2023-10-30T12:36:34.029 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/110/13' 2023-10-30T12:36:34.029 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/110/17' 2023-10-30T12:36:34.030 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/110' 2023-10-30T12:36:34.030 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/137/19' 2023-10-30T12:36:34.030 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/137/15' 2023-10-30T12:36:34.030 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/137/20' 2023-10-30T12:36:34.030 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/137/12' 2023-10-30T12:36:34.031 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/137/10' 2023-10-30T12:36:34.031 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/137/11' 2023-10-30T12:36:34.031 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/137/16' 2023-10-30T12:36:35.275 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/137/14' 2023-10-30T12:36:35.276 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/137/18' 2023-10-30T12:36:35.276 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/137/13' 2023-10-30T12:36:35.276 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/137/17' 2023-10-30T12:36:35.276 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/137' 2023-10-30T12:36:35.276 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/198/19' 2023-10-30T12:36:35.277 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/198/15' 2023-10-30T12:36:35.277 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/198/20' 2023-10-30T12:36:35.277 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/198/12' 2023-10-30T12:36:35.277 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/198/10' 2023-10-30T12:36:35.277 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/198/11' 2023-10-30T12:36:35.278 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/198/16' 2023-10-30T12:36:35.278 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/198/14' 2023-10-30T12:36:35.278 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/198/18' 2023-10-30T12:36:35.278 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/198/13' 2023-10-30T12:36:35.278 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/198/17' 2023-10-30T12:36:35.279 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/198' 2023-10-30T12:36:35.279 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/136/19' 2023-10-30T12:36:35.279 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/136/15' 2023-10-30T12:36:35.279 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/136/20' 2023-10-30T12:36:35.279 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/136/12' 2023-10-30T12:36:35.280 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/136/10' 2023-10-30T12:36:35.280 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/136/11' 2023-10-30T12:36:35.280 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/136/16' 2023-10-30T12:36:35.280 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/136/14' 2023-10-30T12:36:35.280 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/136/18' 2023-10-30T12:36:35.281 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/136/13' 2023-10-30T12:36:35.281 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/136/17' 2023-10-30T12:36:35.281 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/136' 2023-10-30T12:36:35.281 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/125/19' 2023-10-30T12:36:35.281 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/125/15' 2023-10-30T12:36:35.281 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/125/20' 2023-10-30T12:36:35.282 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/125/12' 2023-10-30T12:36:35.282 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/125/10' 2023-10-30T12:36:35.282 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/125/11' 2023-10-30T12:36:35.282 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/125/16' 2023-10-30T12:36:35.282 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/125/14' 2023-10-30T12:36:35.283 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/125/18' 2023-10-30T12:36:35.283 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/125/13' 2023-10-30T12:36:35.283 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/125/17' 2023-10-30T12:36:35.283 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/125' 2023-10-30T12:36:35.283 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/194/19' 2023-10-30T12:36:35.284 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/194/15' 2023-10-30T12:36:35.284 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/194/20' 2023-10-30T12:36:35.284 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/194/12' 2023-10-30T12:36:35.284 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/194/10' 2023-10-30T12:36:35.284 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/194/11' 2023-10-30T12:36:35.284 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/194/16' 2023-10-30T12:36:35.285 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/194/14' 2023-10-30T12:36:35.285 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/194/18' 2023-10-30T12:36:35.285 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/194/13' 2023-10-30T12:36:35.285 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/194/17' 2023-10-30T12:36:35.285 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/194' 2023-10-30T12:36:35.286 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/119/19' 2023-10-30T12:36:35.286 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/119/15' 2023-10-30T12:36:35.286 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/119/20' 2023-10-30T12:36:35.286 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/119/12' 2023-10-30T12:36:35.286 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/119/10' 2023-10-30T12:36:35.286 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/119/11' 2023-10-30T12:36:35.287 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/119/16' 2023-10-30T12:36:35.287 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/119/14' 2023-10-30T12:36:35.287 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/119/18' 2023-10-30T12:36:35.287 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/119/13' 2023-10-30T12:36:35.287 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/119/17' 2023-10-30T12:36:35.288 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/119' 2023-10-30T12:36:35.288 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/157/19' 2023-10-30T12:36:35.288 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/157/15' 2023-10-30T12:36:35.288 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/157/20' 2023-10-30T12:36:35.288 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/157/12' 2023-10-30T12:36:35.289 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/157/10' 2023-10-30T12:36:35.289 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/157/11' 2023-10-30T12:36:35.289 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/157/16' 2023-10-30T12:36:35.289 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/157/14' 2023-10-30T12:36:35.289 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/157/18' 2023-10-30T12:36:35.290 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/157/13' 2023-10-30T12:36:35.290 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/157/17' 2023-10-30T12:36:35.290 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/157' 2023-10-30T12:36:35.290 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/172/19' 2023-10-30T12:36:35.290 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/172/15' 2023-10-30T12:36:35.291 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/172/20' 2023-10-30T12:36:35.291 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/172/12' 2023-10-30T12:36:35.291 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/172/10' 2023-10-30T12:36:35.291 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/172/11' 2023-10-30T12:36:35.291 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/172/16' 2023-10-30T12:36:35.291 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/172/14' 2023-10-30T12:36:35.292 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/172/18' 2023-10-30T12:36:35.292 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/172/13' 2023-10-30T12:36:35.292 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/172/17' 2023-10-30T12:36:35.292 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/172' 2023-10-30T12:36:35.292 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/186/19' 2023-10-30T12:36:35.293 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/186/15' 2023-10-30T12:36:35.293 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/186/20' 2023-10-30T12:36:35.293 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/186/12' 2023-10-30T12:36:35.293 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/186/10' 2023-10-30T12:36:35.293 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/186/11' 2023-10-30T12:36:35.293 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/186/16' 2023-10-30T12:36:35.294 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/186/14' 2023-10-30T12:36:35.294 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/186/18' 2023-10-30T12:36:35.294 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/186/13' 2023-10-30T12:36:35.294 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/186/17' 2023-10-30T12:36:35.294 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/186' 2023-10-30T12:36:35.295 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/190/19' 2023-10-30T12:36:35.295 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/190/15' 2023-10-30T12:36:35.295 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/190/20' 2023-10-30T12:36:35.295 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/190/12' 2023-10-30T12:36:35.295 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/190/10' 2023-10-30T12:36:35.296 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/190/11' 2023-10-30T12:36:35.296 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/190/16' 2023-10-30T12:36:35.296 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/190/14' 2023-10-30T12:36:37.897 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/190/18' 2023-10-30T12:36:37.897 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/190/13' 2023-10-30T12:36:37.897 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/190/17' 2023-10-30T12:36:37.898 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/190' 2023-10-30T12:36:37.898 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/104/19' 2023-10-30T12:36:37.898 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/104/15' 2023-10-30T12:36:37.898 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/104/20' 2023-10-30T12:36:37.898 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/104/12' 2023-10-30T12:36:37.899 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/104/10' 2023-10-30T12:36:37.899 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/104/11' 2023-10-30T12:36:37.899 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/104/16' 2023-10-30T12:36:37.899 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/104/14' 2023-10-30T12:36:37.899 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/104/18' 2023-10-30T12:36:37.900 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/104/13' 2023-10-30T12:36:37.900 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/104/17' 2023-10-30T12:36:37.900 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/104' 2023-10-30T12:36:37.900 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/164/19' 2023-10-30T12:36:37.901 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/164/15' 2023-10-30T12:36:37.901 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/164/20' 2023-10-30T12:36:37.901 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/164/12' 2023-10-30T12:36:37.901 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/164/10' 2023-10-30T12:36:37.901 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/164/11' 2023-10-30T12:36:37.901 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/164/16' 2023-10-30T12:36:37.902 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/164/14' 2023-10-30T12:36:37.902 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/164/18' 2023-10-30T12:36:37.902 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/164/13' 2023-10-30T12:36:37.902 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/164/17' 2023-10-30T12:36:37.902 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/164' 2023-10-30T12:36:37.903 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/112/19' 2023-10-30T12:36:37.903 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/112/15' 2023-10-30T12:36:37.903 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/112/20' 2023-10-30T12:36:37.903 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/112/12' 2023-10-30T12:36:37.903 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/112/10' 2023-10-30T12:36:37.904 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/112/11' 2023-10-30T12:36:37.904 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/112/16' 2023-10-30T12:36:37.904 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/112/14' 2023-10-30T12:36:37.904 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/112/18' 2023-10-30T12:36:37.904 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/112/13' 2023-10-30T12:36:37.905 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/112/17' 2023-10-30T12:36:37.905 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/112' 2023-10-30T12:36:37.905 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/100/19' 2023-10-30T12:36:37.905 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/100/15' 2023-10-30T12:36:37.905 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/100/20' 2023-10-30T12:36:37.905 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/100/12' 2023-10-30T12:36:37.906 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/100/10' 2023-10-30T12:36:37.906 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/100/11' 2023-10-30T12:36:37.906 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/100/16' 2023-10-30T12:36:37.906 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/100/14' 2023-10-30T12:36:37.906 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/100/18' 2023-10-30T12:36:37.907 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/100/13' 2023-10-30T12:36:37.907 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/100/17' 2023-10-30T12:36:37.907 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/100' 2023-10-30T12:36:37.907 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/134/19' 2023-10-30T12:36:37.907 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/134/15' 2023-10-30T12:36:37.907 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/134/20' 2023-10-30T12:36:37.908 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/134/12' 2023-10-30T12:36:37.908 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/134/10' 2023-10-30T12:36:37.908 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/134/11' 2023-10-30T12:36:37.908 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/134/16' 2023-10-30T12:36:37.909 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/134/14' 2023-10-30T12:36:37.909 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/134/18' 2023-10-30T12:36:37.909 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/134/13' 2023-10-30T12:36:37.909 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/134/17' 2023-10-30T12:36:37.909 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/134' 2023-10-30T12:36:37.909 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/115/19' 2023-10-30T12:36:37.910 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/115/15' 2023-10-30T12:36:37.910 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/115/20' 2023-10-30T12:36:37.910 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/115/12' 2023-10-30T12:36:37.910 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/115/10' 2023-10-30T12:36:37.910 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/115/11' 2023-10-30T12:36:37.911 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/115/16' 2023-10-30T12:36:37.911 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/115/14' 2023-10-30T12:36:37.911 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/115/18' 2023-10-30T12:36:37.911 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/115/13' 2023-10-30T12:36:37.911 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/115/17' 2023-10-30T12:36:37.912 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/115' 2023-10-30T12:36:37.912 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/116/19' 2023-10-30T12:36:37.912 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/116/15' 2023-10-30T12:36:37.912 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/116/20' 2023-10-30T12:36:37.912 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/116/12' 2023-10-30T12:36:37.912 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/116/10' 2023-10-30T12:36:37.913 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/116/11' 2023-10-30T12:36:37.913 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/116/16' 2023-10-30T12:36:37.913 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/116/14' 2023-10-30T12:36:37.913 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/116/18' 2023-10-30T12:36:37.913 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/116/13' 2023-10-30T12:36:37.913 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/116/17' 2023-10-30T12:36:37.914 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/116' 2023-10-30T12:36:37.914 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/195/19' 2023-10-30T12:36:37.914 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/195/15' 2023-10-30T12:36:37.914 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/195/20' 2023-10-30T12:36:37.914 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/195/12' 2023-10-30T12:36:37.915 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/195/10' 2023-10-30T12:36:37.915 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/195/11' 2023-10-30T12:36:37.915 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/195/16' 2023-10-30T12:36:37.915 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/195/14' 2023-10-30T12:36:37.915 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/195/18' 2023-10-30T12:36:37.915 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/195/13' 2023-10-30T12:36:37.916 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/195/17' 2023-10-30T12:36:37.916 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/195' 2023-10-30T12:36:37.916 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/138/19' 2023-10-30T12:36:37.916 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/138/15' 2023-10-30T12:36:37.916 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/138/20' 2023-10-30T12:36:37.917 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/138/12' 2023-10-30T12:36:37.917 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/138/10' 2023-10-30T12:36:37.917 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/138/11' 2023-10-30T12:36:37.917 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/138/16' 2023-10-30T12:36:37.917 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/138/14' 2023-10-30T12:36:37.917 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/138/18' 2023-10-30T12:36:41.396 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/138/13' 2023-10-30T12:36:41.397 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/138/17' 2023-10-30T12:36:41.397 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/138' 2023-10-30T12:36:41.397 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/135/19' 2023-10-30T12:36:41.397 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/135/15' 2023-10-30T12:36:41.398 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/135/20' 2023-10-30T12:36:41.398 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/135/12' 2023-10-30T12:36:41.398 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/135/10' 2023-10-30T12:36:41.398 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/135/11' 2023-10-30T12:36:41.398 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/135/16' 2023-10-30T12:36:41.399 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/135/14' 2023-10-30T12:36:41.399 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/135/18' 2023-10-30T12:36:41.399 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/135/13' 2023-10-30T12:36:41.399 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/135/17' 2023-10-30T12:36:41.399 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/135' 2023-10-30T12:36:41.400 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/117/19' 2023-10-30T12:36:41.400 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/117/15' 2023-10-30T12:36:41.400 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/117/20' 2023-10-30T12:36:41.400 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/117/12' 2023-10-30T12:36:41.400 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/117/10' 2023-10-30T12:36:41.401 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/117/11' 2023-10-30T12:36:41.401 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/117/16' 2023-10-30T12:36:41.401 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/117/14' 2023-10-30T12:36:41.401 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/117/18' 2023-10-30T12:36:41.401 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/117/13' 2023-10-30T12:36:41.402 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/117/17' 2023-10-30T12:36:41.402 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/117' 2023-10-30T12:36:41.402 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/143/19' 2023-10-30T12:36:41.402 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/143/15' 2023-10-30T12:36:41.402 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/143/20' 2023-10-30T12:36:41.403 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/143/12' 2023-10-30T12:36:41.403 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/143/10' 2023-10-30T12:36:41.403 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/143/11' 2023-10-30T12:36:41.403 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/143/16' 2023-10-30T12:36:41.403 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/143/14' 2023-10-30T12:36:41.404 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/143/18' 2023-10-30T12:36:41.404 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/143/13' 2023-10-30T12:36:41.404 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/143/17' 2023-10-30T12:36:41.404 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/143' 2023-10-30T12:36:41.404 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/180/19' 2023-10-30T12:36:41.405 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/180/15' 2023-10-30T12:36:41.405 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/180/20' 2023-10-30T12:36:41.405 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/180/12' 2023-10-30T12:36:41.405 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/180/10' 2023-10-30T12:36:41.405 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/180/11' 2023-10-30T12:36:41.406 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/180/16' 2023-10-30T12:36:41.406 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/180/14' 2023-10-30T12:36:41.406 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/180/18' 2023-10-30T12:36:41.406 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/180/13' 2023-10-30T12:36:41.407 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/180/17' 2023-10-30T12:36:41.407 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/180' 2023-10-30T12:36:41.407 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/126/19' 2023-10-30T12:36:41.407 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/126/15' 2023-10-30T12:36:41.407 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/126/20' 2023-10-30T12:36:41.408 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/126/12' 2023-10-30T12:36:41.408 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/126/10' 2023-10-30T12:36:41.408 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/126/11' 2023-10-30T12:36:41.408 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/126/16' 2023-10-30T12:36:41.408 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/126/14' 2023-10-30T12:36:41.409 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/126/18' 2023-10-30T12:36:41.409 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/126/13' 2023-10-30T12:36:41.409 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/126/17' 2023-10-30T12:36:41.409 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/126' 2023-10-30T12:36:41.409 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/161/19' 2023-10-30T12:36:41.410 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/161/15' 2023-10-30T12:36:41.410 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/161/20' 2023-10-30T12:36:41.410 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/161/12' 2023-10-30T12:36:41.410 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/161/10' 2023-10-30T12:36:41.410 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/161/11' 2023-10-30T12:36:41.411 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/161/16' 2023-10-30T12:36:41.411 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/161/14' 2023-10-30T12:36:41.411 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/161/18' 2023-10-30T12:36:41.411 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/161/13' 2023-10-30T12:36:41.411 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/161/17' 2023-10-30T12:36:41.412 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/161' 2023-10-30T12:36:41.412 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/176/19' 2023-10-30T12:36:41.412 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/176/15' 2023-10-30T12:36:41.412 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/176/20' 2023-10-30T12:36:41.412 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/176/12' 2023-10-30T12:36:41.413 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/176/10' 2023-10-30T12:36:41.413 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/176/11' 2023-10-30T12:36:41.413 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/176/16' 2023-10-30T12:36:41.413 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/176/14' 2023-10-30T12:36:41.413 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/176/18' 2023-10-30T12:36:41.414 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/176/13' 2023-10-30T12:36:41.414 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/176/17' 2023-10-30T12:36:41.414 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/176' 2023-10-30T12:36:41.414 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/111/19' 2023-10-30T12:36:41.414 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/111/15' 2023-10-30T12:36:41.415 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/111/20' 2023-10-30T12:36:41.415 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/111/12' 2023-10-30T12:36:41.415 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/111/10' 2023-10-30T12:36:41.415 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/111/11' 2023-10-30T12:36:41.415 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/111/16' 2023-10-30T12:36:41.416 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/111/14' 2023-10-30T12:36:41.416 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/111/18' 2023-10-30T12:36:41.416 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/111/13' 2023-10-30T12:36:41.416 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/111/17' 2023-10-30T12:36:41.417 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/111' 2023-10-30T12:36:41.417 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/191/19' 2023-10-30T12:36:41.417 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/191/15' 2023-10-30T12:36:41.417 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/191/20' 2023-10-30T12:36:41.417 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/191/12' 2023-10-30T12:36:41.418 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/191/10' 2023-10-30T12:36:41.418 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/191/11' 2023-10-30T12:36:41.418 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/191/16' 2023-10-30T12:36:41.418 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/191/14' 2023-10-30T12:36:41.419 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/191/18' 2023-10-30T12:36:41.419 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/191/13' 2023-10-30T12:36:42.489 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/191/17' 2023-10-30T12:36:42.490 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/191' 2023-10-30T12:36:42.490 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/184/19' 2023-10-30T12:36:42.490 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/184/15' 2023-10-30T12:36:42.490 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/184/20' 2023-10-30T12:36:42.491 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/184/12' 2023-10-30T12:36:42.491 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/184/10' 2023-10-30T12:36:42.491 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/184/11' 2023-10-30T12:36:42.491 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/184/16' 2023-10-30T12:36:42.491 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/184/14' 2023-10-30T12:36:42.492 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/184/18' 2023-10-30T12:36:42.492 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/184/13' 2023-10-30T12:36:42.492 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/184/17' 2023-10-30T12:36:42.492 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/184' 2023-10-30T12:36:42.492 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/192/19' 2023-10-30T12:36:42.493 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/192/15' 2023-10-30T12:36:42.493 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/192/20' 2023-10-30T12:36:42.493 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/192/12' 2023-10-30T12:36:42.493 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/192/10' 2023-10-30T12:36:42.493 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/192/11' 2023-10-30T12:36:42.494 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/192/16' 2023-10-30T12:36:42.494 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/192/14' 2023-10-30T12:36:42.494 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/192/18' 2023-10-30T12:36:42.494 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/192/13' 2023-10-30T12:36:42.494 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/192/17' 2023-10-30T12:36:42.495 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/192' 2023-10-30T12:36:42.495 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/103/19' 2023-10-30T12:36:42.495 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/103/15' 2023-10-30T12:36:42.495 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/103/20' 2023-10-30T12:36:42.495 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/103/12' 2023-10-30T12:36:42.496 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/103/10' 2023-10-30T12:36:42.496 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/103/11' 2023-10-30T12:36:42.496 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/103/16' 2023-10-30T12:36:42.496 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/103/14' 2023-10-30T12:36:42.496 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/103/18' 2023-10-30T12:36:42.497 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/103/13' 2023-10-30T12:36:42.497 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/103/17' 2023-10-30T12:36:42.497 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/103' 2023-10-30T12:36:42.497 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/128/19' 2023-10-30T12:36:42.497 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/128/15' 2023-10-30T12:36:42.498 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/128/20' 2023-10-30T12:36:42.498 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/128/12' 2023-10-30T12:36:42.498 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/128/10' 2023-10-30T12:36:42.498 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/128/11' 2023-10-30T12:36:42.498 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/128/16' 2023-10-30T12:36:42.499 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/128/14' 2023-10-30T12:36:42.499 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/128/18' 2023-10-30T12:36:42.499 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/128/13' 2023-10-30T12:36:42.499 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/128/17' 2023-10-30T12:36:42.499 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/128' 2023-10-30T12:36:42.500 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/144/19' 2023-10-30T12:36:42.500 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/144/15' 2023-10-30T12:36:42.500 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/144/20' 2023-10-30T12:36:42.500 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/144/12' 2023-10-30T12:36:42.500 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/144/10' 2023-10-30T12:36:42.501 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/144/11' 2023-10-30T12:36:42.501 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/144/16' 2023-10-30T12:36:42.501 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/144/14' 2023-10-30T12:36:42.501 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/144/18' 2023-10-30T12:36:42.501 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/144/13' 2023-10-30T12:36:42.502 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/144/17' 2023-10-30T12:36:42.502 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/144' 2023-10-30T12:36:42.502 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/146/19' 2023-10-30T12:36:42.502 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/146/15' 2023-10-30T12:36:42.502 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/146/20' 2023-10-30T12:36:42.503 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/146/12' 2023-10-30T12:36:42.503 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/146/10' 2023-10-30T12:36:42.503 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/146/11' 2023-10-30T12:36:42.503 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/146/16' 2023-10-30T12:36:42.503 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/146/14' 2023-10-30T12:36:42.503 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/146/18' 2023-10-30T12:36:42.504 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/146/13' 2023-10-30T12:36:42.504 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/146/17' 2023-10-30T12:36:42.504 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/146' 2023-10-30T12:36:42.504 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/141/19' 2023-10-30T12:36:42.504 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/141/15' 2023-10-30T12:36:42.505 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/141/20' 2023-10-30T12:36:42.505 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/141/12' 2023-10-30T12:36:42.505 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/141/10' 2023-10-30T12:36:42.505 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/141/11' 2023-10-30T12:36:42.505 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/141/16' 2023-10-30T12:36:42.506 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/141/14' 2023-10-30T12:36:42.506 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/141/18' 2023-10-30T12:36:42.506 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/141/13' 2023-10-30T12:36:42.506 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/141/17' 2023-10-30T12:36:42.506 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/141' 2023-10-30T12:36:42.507 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/182/19' 2023-10-30T12:36:42.507 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/182/15' 2023-10-30T12:36:42.507 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/182/20' 2023-10-30T12:36:42.507 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/182/12' 2023-10-30T12:36:42.507 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/182/10' 2023-10-30T12:36:42.507 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/182/11' 2023-10-30T12:36:42.508 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/182/16' 2023-10-30T12:36:42.508 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/182/14' 2023-10-30T12:36:42.508 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/182/18' 2023-10-30T12:36:42.508 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/182/13' 2023-10-30T12:36:42.508 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/182/17' 2023-10-30T12:36:42.509 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/182' 2023-10-30T12:36:42.509 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/107/19' 2023-10-30T12:36:42.509 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/107/15' 2023-10-30T12:36:42.509 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/107/20' 2023-10-30T12:36:42.509 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/107/12' 2023-10-30T12:36:42.510 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/107/10' 2023-10-30T12:36:42.510 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/107/11' 2023-10-30T12:36:42.510 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/107/16' 2023-10-30T12:36:42.510 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/107/14' 2023-10-30T12:36:42.510 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/107/18' 2023-10-30T12:36:42.511 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/107/13' 2023-10-30T12:36:42.511 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/107/17' 2023-10-30T12:36:44.348 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/107' 2023-10-30T12:36:44.349 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/178/19' 2023-10-30T12:36:44.349 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/178/15' 2023-10-30T12:36:44.349 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/178/20' 2023-10-30T12:36:44.349 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/178/12' 2023-10-30T12:36:44.350 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/178/10' 2023-10-30T12:36:44.350 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/178/11' 2023-10-30T12:36:44.350 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/178/16' 2023-10-30T12:36:44.350 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/178/14' 2023-10-30T12:36:44.350 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/178/18' 2023-10-30T12:36:44.351 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/178/13' 2023-10-30T12:36:44.351 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/178/17' 2023-10-30T12:36:44.351 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/178' 2023-10-30T12:36:44.351 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/156/19' 2023-10-30T12:36:44.351 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/156/15' 2023-10-30T12:36:44.352 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/156/20' 2023-10-30T12:36:44.352 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/156/12' 2023-10-30T12:36:44.352 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/156/10' 2023-10-30T12:36:44.352 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/156/11' 2023-10-30T12:36:44.352 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/156/16' 2023-10-30T12:36:44.353 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/156/14' 2023-10-30T12:36:44.353 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/156/18' 2023-10-30T12:36:44.353 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/156/13' 2023-10-30T12:36:44.353 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/156/17' 2023-10-30T12:36:44.353 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/156' 2023-10-30T12:36:44.353 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/131/19' 2023-10-30T12:36:44.354 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/131/15' 2023-10-30T12:36:44.354 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/131/20' 2023-10-30T12:36:44.354 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/131/12' 2023-10-30T12:36:44.354 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/131/10' 2023-10-30T12:36:44.354 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/131/11' 2023-10-30T12:36:44.355 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/131/16' 2023-10-30T12:36:44.355 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/131/14' 2023-10-30T12:36:44.355 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/131/18' 2023-10-30T12:36:44.355 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/131/13' 2023-10-30T12:36:44.355 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/131/17' 2023-10-30T12:36:44.356 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/131' 2023-10-30T12:36:44.356 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/123/19' 2023-10-30T12:36:44.356 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/123/15' 2023-10-30T12:36:44.356 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/123/20' 2023-10-30T12:36:44.356 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/123/12' 2023-10-30T12:36:44.356 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/123/10' 2023-10-30T12:36:44.357 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/123/11' 2023-10-30T12:36:44.357 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/123/16' 2023-10-30T12:36:44.357 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/123/14' 2023-10-30T12:36:44.357 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/123/18' 2023-10-30T12:36:44.357 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/123/13' 2023-10-30T12:36:44.358 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/123/17' 2023-10-30T12:36:44.358 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/123' 2023-10-30T12:36:44.358 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/160/19' 2023-10-30T12:36:44.358 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/160/15' 2023-10-30T12:36:44.358 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/160/20' 2023-10-30T12:36:44.359 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/160/12' 2023-10-30T12:36:44.359 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/160/10' 2023-10-30T12:36:44.359 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/160/11' 2023-10-30T12:36:44.359 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/160/16' 2023-10-30T12:36:44.359 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/160/14' 2023-10-30T12:36:44.360 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/160/18' 2023-10-30T12:36:44.360 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/160/13' 2023-10-30T12:36:44.360 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/160/17' 2023-10-30T12:36:44.360 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/160' 2023-10-30T12:36:44.360 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/165/19' 2023-10-30T12:36:44.360 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/165/15' 2023-10-30T12:36:44.361 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/165/20' 2023-10-30T12:36:44.361 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/165/12' 2023-10-30T12:36:44.361 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/165/10' 2023-10-30T12:36:44.361 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/165/11' 2023-10-30T12:36:44.361 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/165/16' 2023-10-30T12:36:44.362 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/165/14' 2023-10-30T12:36:44.362 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/165/18' 2023-10-30T12:36:44.362 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/165/13' 2023-10-30T12:36:44.362 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/165/17' 2023-10-30T12:36:44.363 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/165' 2023-10-30T12:36:44.363 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/101/19' 2023-10-30T12:36:44.363 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/101/15' 2023-10-30T12:36:44.363 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/101/20' 2023-10-30T12:36:44.363 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/101/12' 2023-10-30T12:36:44.363 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/101/10' 2023-10-30T12:36:44.364 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/101/11' 2023-10-30T12:36:44.364 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/101/16' 2023-10-30T12:36:44.364 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/101/14' 2023-10-30T12:36:44.364 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/101/18' 2023-10-30T12:36:44.364 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/101/13' 2023-10-30T12:36:44.365 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/101/17' 2023-10-30T12:36:44.365 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/101' 2023-10-30T12:36:44.365 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/148/19' 2023-10-30T12:36:44.365 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/148/15' 2023-10-30T12:36:44.365 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/148/20' 2023-10-30T12:36:44.366 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/148/12' 2023-10-30T12:36:44.366 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/148/10' 2023-10-30T12:36:44.366 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/148/11' 2023-10-30T12:36:44.366 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/148/16' 2023-10-30T12:36:44.367 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/148/14' 2023-10-30T12:36:44.367 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/148/18' 2023-10-30T12:36:44.367 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/148/13' 2023-10-30T12:36:44.367 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/148/17' 2023-10-30T12:36:44.367 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/148' 2023-10-30T12:36:44.367 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/168/19' 2023-10-30T12:36:44.368 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/168/15' 2023-10-30T12:36:44.368 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/168/20' 2023-10-30T12:36:44.368 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/168/12' 2023-10-30T12:36:44.368 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/168/10' 2023-10-30T12:36:44.368 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/168/11' 2023-10-30T12:36:44.369 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/168/16' 2023-10-30T12:36:44.369 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/168/14' 2023-10-30T12:36:44.369 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/168/18' 2023-10-30T12:36:44.369 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/168/13' 2023-10-30T12:36:44.370 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/168/17' 2023-10-30T12:36:44.370 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/168' 2023-10-30T12:36:44.950 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/179/19' 2023-10-30T12:36:44.950 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/179/15' 2023-10-30T12:36:44.951 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/179/20' 2023-10-30T12:36:44.951 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/179/12' 2023-10-30T12:36:44.951 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/179/10' 2023-10-30T12:36:44.951 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/179/11' 2023-10-30T12:36:44.951 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/179/16' 2023-10-30T12:36:44.952 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/179/14' 2023-10-30T12:36:44.952 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/179/18' 2023-10-30T12:36:44.952 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/179/13' 2023-10-30T12:36:44.952 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/179/17' 2023-10-30T12:36:44.952 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/179' 2023-10-30T12:36:44.953 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/132/19' 2023-10-30T12:36:44.953 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/132/15' 2023-10-30T12:36:44.953 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/132/20' 2023-10-30T12:36:44.953 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/132/12' 2023-10-30T12:36:44.953 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/132/10' 2023-10-30T12:36:44.953 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/132/11' 2023-10-30T12:36:44.954 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/132/16' 2023-10-30T12:36:44.954 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/132/14' 2023-10-30T12:36:44.954 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/132/18' 2023-10-30T12:36:44.954 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/132/13' 2023-10-30T12:36:44.954 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/132/17' 2023-10-30T12:36:44.955 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/132' 2023-10-30T12:36:44.955 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/199/19' 2023-10-30T12:36:44.955 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/199/15' 2023-10-30T12:36:44.955 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/199/20' 2023-10-30T12:36:44.955 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/199/12' 2023-10-30T12:36:44.956 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/199/10' 2023-10-30T12:36:44.956 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/199/11' 2023-10-30T12:36:44.956 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/199/16' 2023-10-30T12:36:44.956 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/199/14' 2023-10-30T12:36:44.956 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/199/18' 2023-10-30T12:36:44.957 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/199/13' 2023-10-30T12:36:44.957 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/199/17' 2023-10-30T12:36:44.957 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/199' 2023-10-30T12:36:44.957 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/163/19' 2023-10-30T12:36:44.957 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/163/15' 2023-10-30T12:36:44.957 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/163/20' 2023-10-30T12:36:44.958 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/163/12' 2023-10-30T12:36:44.958 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/163/10' 2023-10-30T12:36:44.958 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/163/11' 2023-10-30T12:36:44.958 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/163/16' 2023-10-30T12:36:44.958 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/163/14' 2023-10-30T12:36:44.959 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/163/18' 2023-10-30T12:36:44.959 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/163/13' 2023-10-30T12:36:44.959 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/163/17' 2023-10-30T12:36:44.959 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/163' 2023-10-30T12:36:44.959 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/153/19' 2023-10-30T12:36:44.960 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/153/15' 2023-10-30T12:36:44.960 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/153/20' 2023-10-30T12:36:44.960 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/153/12' 2023-10-30T12:36:44.960 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/153/10' 2023-10-30T12:36:44.960 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/153/11' 2023-10-30T12:36:44.960 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/153/16' 2023-10-30T12:36:44.961 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/153/14' 2023-10-30T12:36:44.961 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/153/18' 2023-10-30T12:36:44.961 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/153/13' 2023-10-30T12:36:44.961 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/153/17' 2023-10-30T12:36:44.961 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/153' 2023-10-30T12:36:44.962 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/122/19' 2023-10-30T12:36:44.962 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/122/15' 2023-10-30T12:36:44.962 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/122/20' 2023-10-30T12:36:44.962 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/122/12' 2023-10-30T12:36:44.962 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/122/10' 2023-10-30T12:36:44.963 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/122/11' 2023-10-30T12:36:44.963 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/122/16' 2023-10-30T12:36:44.963 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/122/14' 2023-10-30T12:36:44.963 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/122/18' 2023-10-30T12:36:44.963 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/122/13' 2023-10-30T12:36:44.964 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/122/17' 2023-10-30T12:36:44.964 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/122' 2023-10-30T12:36:44.964 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/162/19' 2023-10-30T12:36:44.964 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/162/15' 2023-10-30T12:36:44.964 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/162/20' 2023-10-30T12:36:44.964 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/162/12' 2023-10-30T12:36:44.965 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/162/10' 2023-10-30T12:36:44.965 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/162/11' 2023-10-30T12:36:44.965 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/162/16' 2023-10-30T12:36:44.965 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/162/14' 2023-10-30T12:36:44.965 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/162/18' 2023-10-30T12:36:44.966 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/162/13' 2023-10-30T12:36:44.966 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/162/17' 2023-10-30T12:36:44.966 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/162' 2023-10-30T12:36:44.966 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/197/19' 2023-10-30T12:36:44.967 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/197/15' 2023-10-30T12:36:44.967 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/197/20' 2023-10-30T12:36:44.967 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/197/12' 2023-10-30T12:36:44.967 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/197/10' 2023-10-30T12:36:44.967 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/197/11' 2023-10-30T12:36:44.967 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/197/16' 2023-10-30T12:36:44.968 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/197/14' 2023-10-30T12:36:44.968 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/197/18' 2023-10-30T12:36:44.968 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/197/13' 2023-10-30T12:36:44.968 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/197/17' 2023-10-30T12:36:44.968 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/197' 2023-10-30T12:36:44.968 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/113/19' 2023-10-30T12:36:44.969 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/113/15' 2023-10-30T12:36:44.969 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/113/20' 2023-10-30T12:36:44.969 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/113/12' 2023-10-30T12:36:44.969 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/113/10' 2023-10-30T12:36:44.969 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/113/11' 2023-10-30T12:36:44.970 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/113/16' 2023-10-30T12:36:44.970 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/113/14' 2023-10-30T12:36:44.970 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/113/18' 2023-10-30T12:36:44.970 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/113/13' 2023-10-30T12:36:44.970 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/113/17' 2023-10-30T12:36:44.971 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/113' 2023-10-30T12:36:44.971 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/152/19' 2023-10-30T12:36:45.970 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:36:45.974 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:36:55.009 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/152/15' 2023-10-30T12:36:55.009 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/152/20' 2023-10-30T12:36:55.009 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/152/12' 2023-10-30T12:36:55.010 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/152/10' 2023-10-30T12:36:55.010 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/152/11' 2023-10-30T12:36:55.010 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/152/16' 2023-10-30T12:36:55.010 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/152/14' 2023-10-30T12:36:55.011 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/152/18' 2023-10-30T12:36:55.011 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/152/13' 2023-10-30T12:36:55.011 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/152/17' 2023-10-30T12:36:55.011 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/152' 2023-10-30T12:36:55.011 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/121/19' 2023-10-30T12:36:55.012 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/121/15' 2023-10-30T12:36:55.012 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/121/20' 2023-10-30T12:36:55.012 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/121/12' 2023-10-30T12:36:55.012 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/121/10' 2023-10-30T12:36:55.012 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/121/11' 2023-10-30T12:36:55.012 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/121/16' 2023-10-30T12:36:55.013 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/121/14' 2023-10-30T12:36:55.013 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/121/18' 2023-10-30T12:36:55.013 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/121/13' 2023-10-30T12:36:55.013 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/121/17' 2023-10-30T12:36:55.014 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/121' 2023-10-30T12:36:55.014 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/118/19' 2023-10-30T12:36:55.014 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/118/15' 2023-10-30T12:36:55.014 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/118/20' 2023-10-30T12:36:55.014 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/118/12' 2023-10-30T12:36:55.015 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/118/10' 2023-10-30T12:36:55.015 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/118/11' 2023-10-30T12:36:55.015 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/118/16' 2023-10-30T12:36:55.015 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/118/14' 2023-10-30T12:36:55.015 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/118/18' 2023-10-30T12:36:55.016 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/118/13' 2023-10-30T12:36:55.016 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/118/17' 2023-10-30T12:36:55.016 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/118' 2023-10-30T12:36:55.016 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/196/19' 2023-10-30T12:36:55.016 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/196/15' 2023-10-30T12:36:55.017 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/196/20' 2023-10-30T12:36:55.017 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/196/12' 2023-10-30T12:36:55.017 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/196/10' 2023-10-30T12:36:55.017 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/196/11' 2023-10-30T12:36:55.017 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/196/16' 2023-10-30T12:36:55.017 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/196/14' 2023-10-30T12:36:55.018 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/196/18' 2023-10-30T12:36:55.018 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/196/13' 2023-10-30T12:36:55.018 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/196/17' 2023-10-30T12:36:55.018 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/196' 2023-10-30T12:36:55.018 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/124/19' 2023-10-30T12:36:55.019 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/124/15' 2023-10-30T12:36:55.019 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/124/20' 2023-10-30T12:36:55.019 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/124/12' 2023-10-30T12:36:55.019 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/124/10' 2023-10-30T12:36:55.019 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/124/11' 2023-10-30T12:36:55.020 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/124/16' 2023-10-30T12:36:55.020 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/124/14' 2023-10-30T12:36:55.020 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/124/18' 2023-10-30T12:36:55.020 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/124/13' 2023-10-30T12:36:55.020 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/124/17' 2023-10-30T12:36:55.021 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/124' 2023-10-30T12:36:55.021 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/188/19' 2023-10-30T12:36:55.021 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/188/15' 2023-10-30T12:36:55.021 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/188/20' 2023-10-30T12:36:55.021 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/188/12' 2023-10-30T12:36:55.021 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/188/10' 2023-10-30T12:36:55.022 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/188/11' 2023-10-30T12:36:55.022 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/188/16' 2023-10-30T12:36:55.022 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/188/14' 2023-10-30T12:36:55.022 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/188/18' 2023-10-30T12:36:55.022 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/188/13' 2023-10-30T12:36:55.023 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/188/17' 2023-10-30T12:36:55.023 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/188' 2023-10-30T12:36:55.023 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/145/19' 2023-10-30T12:36:55.023 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/145/15' 2023-10-30T12:36:55.023 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/145/20' 2023-10-30T12:36:55.024 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/145/12' 2023-10-30T12:36:55.024 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/145/10' 2023-10-30T12:36:55.024 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/145/11' 2023-10-30T12:36:55.024 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/145/16' 2023-10-30T12:36:55.024 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/145/14' 2023-10-30T12:36:55.025 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/145/18' 2023-10-30T12:36:55.025 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/145/13' 2023-10-30T12:36:55.025 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/145/17' 2023-10-30T12:36:55.025 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/145' 2023-10-30T12:36:55.025 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/130/19' 2023-10-30T12:36:55.026 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/130/15' 2023-10-30T12:36:55.026 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/130/20' 2023-10-30T12:36:55.026 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/130/12' 2023-10-30T12:36:55.026 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/130/10' 2023-10-30T12:36:55.026 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/130/11' 2023-10-30T12:36:55.027 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/130/16' 2023-10-30T12:36:55.027 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/130/14' 2023-10-30T12:36:55.027 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/130/18' 2023-10-30T12:36:55.027 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/130/13' 2023-10-30T12:36:55.027 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/130/17' 2023-10-30T12:36:55.029 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/130' 2023-10-30T12:36:55.029 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/193/19' 2023-10-30T12:36:55.029 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/193/15' 2023-10-30T12:36:55.029 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/193/20' 2023-10-30T12:36:55.030 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/193/12' 2023-10-30T12:36:55.030 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/193/10' 2023-10-30T12:36:55.030 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/193/11' 2023-10-30T12:36:55.030 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/193/16' 2023-10-30T12:36:55.030 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/193/14' 2023-10-30T12:36:55.031 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/193/18' 2023-10-30T12:36:55.031 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/193/13' 2023-10-30T12:36:55.031 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/193/17' 2023-10-30T12:36:55.031 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/193' 2023-10-30T12:36:55.031 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/175/19' 2023-10-30T12:36:55.032 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/175/15' 2023-10-30T12:36:59.805 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/175/20' 2023-10-30T12:36:59.805 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/175/12' 2023-10-30T12:36:59.805 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/175/10' 2023-10-30T12:36:59.806 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/175/11' 2023-10-30T12:36:59.806 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/175/16' 2023-10-30T12:36:59.806 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/175/14' 2023-10-30T12:36:59.806 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/175/18' 2023-10-30T12:36:59.806 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/175/13' 2023-10-30T12:36:59.807 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/175/17' 2023-10-30T12:36:59.807 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/175' 2023-10-30T12:36:59.807 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/181/19' 2023-10-30T12:36:59.807 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/181/15' 2023-10-30T12:36:59.807 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/181/20' 2023-10-30T12:36:59.808 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/181/12' 2023-10-30T12:36:59.808 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/181/10' 2023-10-30T12:36:59.808 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/181/11' 2023-10-30T12:36:59.808 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/181/16' 2023-10-30T12:36:59.808 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/181/14' 2023-10-30T12:36:59.809 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/181/18' 2023-10-30T12:36:59.809 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/181/13' 2023-10-30T12:36:59.809 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/181/17' 2023-10-30T12:36:59.809 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/181' 2023-10-30T12:36:59.809 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/127/19' 2023-10-30T12:36:59.810 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/127/15' 2023-10-30T12:36:59.810 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/127/20' 2023-10-30T12:36:59.810 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/127/12' 2023-10-30T12:36:59.810 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/127/10' 2023-10-30T12:36:59.810 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/127/11' 2023-10-30T12:36:59.811 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/127/16' 2023-10-30T12:36:59.811 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/127/14' 2023-10-30T12:36:59.811 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/127/18' 2023-10-30T12:36:59.811 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/127/13' 2023-10-30T12:36:59.812 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/127/17' 2023-10-30T12:36:59.812 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/127' 2023-10-30T12:36:59.812 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/108/19' 2023-10-30T12:36:59.812 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/108/15' 2023-10-30T12:36:59.812 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/108/20' 2023-10-30T12:36:59.813 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/108/12' 2023-10-30T12:36:59.813 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/108/10' 2023-10-30T12:36:59.813 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/108/11' 2023-10-30T12:36:59.813 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/108/16' 2023-10-30T12:36:59.813 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/108/14' 2023-10-30T12:36:59.813 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/108/18' 2023-10-30T12:36:59.814 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/108/13' 2023-10-30T12:36:59.814 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/108/17' 2023-10-30T12:36:59.814 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/108' 2023-10-30T12:36:59.814 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/183/19' 2023-10-30T12:36:59.815 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/183/15' 2023-10-30T12:36:59.815 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/183/20' 2023-10-30T12:36:59.815 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/183/12' 2023-10-30T12:36:59.815 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/183/10' 2023-10-30T12:36:59.815 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/183/11' 2023-10-30T12:36:59.816 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/183/16' 2023-10-30T12:36:59.816 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/183/14' 2023-10-30T12:36:59.816 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/183/18' 2023-10-30T12:36:59.816 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/183/13' 2023-10-30T12:36:59.816 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/183/17' 2023-10-30T12:36:59.817 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/183' 2023-10-30T12:36:59.817 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/177/19' 2023-10-30T12:36:59.817 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/177/15' 2023-10-30T12:36:59.817 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/177/20' 2023-10-30T12:36:59.817 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/177/12' 2023-10-30T12:36:59.818 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/177/10' 2023-10-30T12:36:59.818 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/177/11' 2023-10-30T12:36:59.818 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/177/16' 2023-10-30T12:36:59.818 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/177/14' 2023-10-30T12:36:59.818 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/177/18' 2023-10-30T12:36:59.819 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/177/13' 2023-10-30T12:36:59.819 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/177/17' 2023-10-30T12:36:59.819 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/177' 2023-10-30T12:36:59.819 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/169/19' 2023-10-30T12:36:59.819 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/169/15' 2023-10-30T12:36:59.820 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/169/20' 2023-10-30T12:36:59.820 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/169/12' 2023-10-30T12:36:59.820 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/169/10' 2023-10-30T12:36:59.820 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/169/11' 2023-10-30T12:36:59.820 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/169/16' 2023-10-30T12:36:59.821 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/169/14' 2023-10-30T12:36:59.821 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/169/18' 2023-10-30T12:36:59.821 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/169/13' 2023-10-30T12:36:59.821 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/169/17' 2023-10-30T12:36:59.821 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/169' 2023-10-30T12:36:59.821 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/154/19' 2023-10-30T12:36:59.822 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/154/15' 2023-10-30T12:36:59.822 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/154/20' 2023-10-30T12:36:59.822 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/154/12' 2023-10-30T12:36:59.822 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/154/10' 2023-10-30T12:36:59.822 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/154/11' 2023-10-30T12:36:59.823 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/154/16' 2023-10-30T12:36:59.823 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/154/14' 2023-10-30T12:36:59.823 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/154/18' 2023-10-30T12:36:59.823 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/154/13' 2023-10-30T12:36:59.823 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/154/17' 2023-10-30T12:36:59.824 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/154' 2023-10-30T12:36:59.824 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/171/19' 2023-10-30T12:36:59.824 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/171/15' 2023-10-30T12:36:59.824 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/171/20' 2023-10-30T12:36:59.824 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/171/12' 2023-10-30T12:36:59.825 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/171/10' 2023-10-30T12:36:59.825 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/171/11' 2023-10-30T12:36:59.825 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/171/16' 2023-10-30T12:36:59.825 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/171/14' 2023-10-30T12:36:59.825 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/171/18' 2023-10-30T12:36:59.826 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/171/13' 2023-10-30T12:36:59.826 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/171/17' 2023-10-30T12:36:59.826 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/171' 2023-10-30T12:36:59.826 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/139/19' 2023-10-30T12:36:59.826 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/139/15' 2023-10-30T12:36:59.827 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/139/20' 2023-10-30T12:37:00.020 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/139/12' 2023-10-30T12:37:00.021 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/139/10' 2023-10-30T12:37:00.021 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/139/11' 2023-10-30T12:37:00.021 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/139/16' 2023-10-30T12:37:00.021 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/139/14' 2023-10-30T12:37:00.022 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/139/18' 2023-10-30T12:37:00.022 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/139/13' 2023-10-30T12:37:00.022 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/139/17' 2023-10-30T12:37:00.023 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/139' 2023-10-30T12:37:00.023 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/102/19' 2023-10-30T12:37:00.023 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/102/15' 2023-10-30T12:37:00.024 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/102/20' 2023-10-30T12:37:00.024 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/102/12' 2023-10-30T12:37:00.024 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/102/10' 2023-10-30T12:37:00.024 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/102/11' 2023-10-30T12:37:00.024 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/102/16' 2023-10-30T12:37:00.025 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/102/14' 2023-10-30T12:37:00.025 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/102/18' 2023-10-30T12:37:00.025 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/102/13' 2023-10-30T12:37:00.025 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/102/17' 2023-10-30T12:37:00.026 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/102' 2023-10-30T12:37:00.026 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/129/19' 2023-10-30T12:37:00.026 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/129/15' 2023-10-30T12:37:00.026 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/129/20' 2023-10-30T12:37:00.026 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/129/12' 2023-10-30T12:37:00.027 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/129/10' 2023-10-30T12:37:00.027 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/129/11' 2023-10-30T12:37:00.027 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/129/16' 2023-10-30T12:37:00.027 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/129/14' 2023-10-30T12:37:00.027 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/129/18' 2023-10-30T12:37:00.028 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/129/13' 2023-10-30T12:37:00.028 INFO:tasks.workunit.client.0.smithi089.stdout:removed './copyofsnap1/test1/129/17' 2023-10-30T12:37:00.028 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1/129' 2023-10-30T12:37:00.028 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1/test1' 2023-10-30T12:37:00.028 INFO:tasks.workunit.client.0.smithi089.stdout:removed directory './copyofsnap1' 2023-10-30T12:37:00.029 INFO:tasks.workunit.client.0.smithi089.stdout:OK 2023-10-30T12:37:00.030 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T12:37:00.031 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T12:37:00.088 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-authwb.sh... 2023-10-30T12:37:00.089 DEBUG:teuthology.orchestra.run.smithi089:workunit test fs/snaps/snaptest-authwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-authwb.sh 2023-10-30T12:37:00.109 INFO:tasks.workunit.client.0.smithi089.stderr:+ set -e 2023-10-30T12:37:00.109 INFO:tasks.workunit.client.0.smithi089.stderr:+ touch foo 2023-10-30T12:37:00.112 INFO:tasks.workunit.client.0.smithi089.stderr:+ chmod +x foo 2023-10-30T12:37:00.113 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir .snap/s 2023-10-30T12:37:00.135 INFO:tasks.workunit.client.0.smithi089.stderr:+ find .snap/s/foo -executable 2023-10-30T12:37:00.135 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep foo 2023-10-30T12:37:00.566 INFO:tasks.workunit.client.0.smithi089.stdout:.snap/s/foo 2023-10-30T12:37:00.566 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir .snap/s 2023-10-30T12:37:00.610 INFO:tasks.workunit.client.0.smithi089.stderr:+ rm foo 2023-10-30T12:37:00.645 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo OK 2023-10-30T12:37:00.646 INFO:tasks.workunit.client.0.smithi089.stdout:OK 2023-10-30T12:37:00.646 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T12:37:00.647 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T12:37:01.379 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-capwb.sh... 2023-10-30T12:37:01.380 DEBUG:teuthology.orchestra.run.smithi089:workunit test fs/snaps/snaptest-capwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-capwb.sh 2023-10-30T12:37:01.400 INFO:tasks.workunit.client.0.smithi089.stderr:+ set -e 2023-10-30T12:37:01.401 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir foo 2023-10-30T12:37:01.405 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo x 2023-10-30T12:37:01.405 INFO:tasks.workunit.client.0.smithi089.stderr:+ sync 2023-10-30T12:37:01.704 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir foo/.snap/ss 2023-10-30T12:37:01.720 INFO:tasks.workunit.client.0.smithi089.stderr:+ ln foo/x foo/xx 2023-10-30T12:37:01.724 INFO:tasks.workunit.client.0.smithi089.stderr:+ cat foo/.snap/ss/x 2023-10-30T12:37:01.735 INFO:tasks.workunit.client.0.smithi089.stdout:x 2023-10-30T12:37:01.736 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir foo/.snap/ss 2023-10-30T12:37:01.750 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo a 2023-10-30T12:37:01.750 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo b 2023-10-30T12:37:01.750 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir foo/.snap/s 2023-10-30T12:37:01.769 INFO:tasks.workunit.client.0.smithi089.stderr:+ cat foo/.snap/s/a 2023-10-30T12:37:01.783 INFO:tasks.workunit.client.0.smithi089.stderr:+ r=a 2023-10-30T12:37:01.783 INFO:tasks.workunit.client.0.smithi089.stderr:+ [ -z a ] 2023-10-30T12:37:01.783 INFO:tasks.workunit.client.0.smithi089.stderr:+ ln foo/b foo/b2 2023-10-30T12:37:01.786 INFO:tasks.workunit.client.0.smithi089.stderr:+ cat foo/.snap/s/b 2023-10-30T12:37:01.801 INFO:tasks.workunit.client.0.smithi089.stdout:b 2023-10-30T12:37:01.802 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo this used to hang: 2023-10-30T12:37:01.802 INFO:tasks.workunit.client.0.smithi089.stdout:this used to hang: 2023-10-30T12:37:01.803 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo more 2023-10-30T12:37:01.803 INFO:tasks.workunit.client.0.smithi089.stdout:oh, it didn't hang! good job. 2023-10-30T12:37:01.804 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo oh, it didn't hang! good job. 2023-10-30T12:37:01.804 INFO:tasks.workunit.client.0.smithi089.stderr:+ cat foo/b 2023-10-30T12:37:01.804 INFO:tasks.workunit.client.0.smithi089.stdout:b 2023-10-30T12:37:01.805 INFO:tasks.workunit.client.0.smithi089.stdout:more 2023-10-30T12:37:01.805 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir foo/.snap/s 2023-10-30T12:37:01.813 INFO:tasks.workunit.client.0.smithi089.stderr:+ rm -r foo 2023-10-30T12:37:01.848 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo OK 2023-10-30T12:37:01.849 INFO:tasks.workunit.client.0.smithi089.stdout:OK 2023-10-30T12:37:01.849 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T12:37:01.849 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T12:37:01.932 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-dir-rename.sh... 2023-10-30T12:37:01.933 DEBUG:teuthology.orchestra.run.smithi089:workunit test fs/snaps/snaptest-dir-rename.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-dir-rename.sh 2023-10-30T12:37:01.997 INFO:tasks.workunit.client.0.smithi089.stderr:+ set -e 2023-10-30T12:37:01.998 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a 2023-10-30T12:37:02.001 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir .snap/bar 2023-10-30T12:37:02.025 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/foo 2023-10-30T12:37:02.039 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a/.snap/foo 2023-10-30T12:37:02.779 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a 2023-10-30T12:37:02.787 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/bar/a 2023-10-30T12:37:07.402 INFO:tasks.workunit.client.0.smithi089.stdout: File: .snap/bar/a 2023-10-30T12:37:07.403 INFO:tasks.workunit.client.0.smithi089.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-10-30T12:37:07.403 INFO:tasks.workunit.client.0.smithi089.stdout:Device: fah/250d Inode: 1099511632752 Links: 2 2023-10-30T12:37:07.403 INFO:tasks.workunit.client.0.smithi089.stdout:Access: (0775/drwxrwxr-x) Uid: ( 1000/ ubuntu) Gid: ( 1271/ ubuntu) 2023-10-30T12:37:07.403 INFO:tasks.workunit.client.0.smithi089.stdout:Access: 2023-10-30 12:37:01.999401119 +0000 2023-10-30T12:37:07.404 INFO:tasks.workunit.client.0.smithi089.stdout:Modify: 2023-10-30 12:37:01.999401119 +0000 2023-10-30T12:37:07.404 INFO:tasks.workunit.client.0.smithi089.stdout:Change: 2023-10-30 12:37:01.999401119 +0000 2023-10-30T12:37:07.404 INFO:tasks.workunit.client.0.smithi089.stdout: Birth: 2023-10-30 12:37:01.999401119 +0000 2023-10-30T12:37:07.404 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir .snap/bar 2023-10-30T12:37:07.417 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo OK 2023-10-30T12:37:07.418 INFO:tasks.workunit.client.0.smithi089.stdout:OK 2023-10-30T12:37:07.419 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T12:37:07.419 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T12:37:07.494 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-double-null.sh... 2023-10-30T12:37:07.494 DEBUG:teuthology.orchestra.run.smithi089:workunit test fs/snaps/snaptest-double-null.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-double-null.sh 2023-10-30T12:37:07.561 INFO:tasks.workunit.client.0.smithi089.stderr:+ set -e 2023-10-30T12:37:07.561 INFO:tasks.workunit.client.0.smithi089.stderr:+ seq 1 20 2023-10-30T12:37:07.562 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a 2023-10-30T12:37:07.566 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:07.568 INFO:tasks.workunit.client.0.smithi089.stderr:+ cat 2023-10-30T12:37:07.590 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:07.601 INFO:tasks.workunit.client.0.smithi089.stderr:+ chmod 777 a/foo 2023-10-30T12:37:07.602 INFO:tasks.workunit.client.0.smithi089.stderr:+ sync 2023-10-30T12:37:07.828 INFO:tasks.workunit.client.0.smithi089.stderr:+ ps 2023-10-30T12:37:07.843 INFO:tasks.workunit.client.0.smithi089.stdout: PID TTY TIME CMD 2023-10-30T12:37:07.844 INFO:tasks.workunit.client.0.smithi089.stdout: 1293 ? 00:00:00 systemd 2023-10-30T12:37:07.844 INFO:tasks.workunit.client.0.smithi089.stdout: 1294 ? 00:00:00 (sd-pam) 2023-10-30T12:37:07.844 INFO:tasks.workunit.client.0.smithi089.stdout: 27974 ? 00:00:01 sshd 2023-10-30T12:37:07.844 INFO:tasks.workunit.client.0.smithi089.stdout: 49407 ? 00:00:00 timeout 2023-10-30T12:37:07.845 INFO:tasks.workunit.client.0.smithi089.stdout: 49410 ? 00:00:00 snaptest-double 2023-10-30T12:37:07.845 INFO:tasks.workunit.client.0.smithi089.stdout: 49418 ? 00:00:00 ps 2023-10-30T12:37:07.845 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:07.867 INFO:tasks.workunit.client.0.smithi089.stderr:+ rm a/foo 2023-10-30T12:37:07.872 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a 2023-10-30T12:37:07.886 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a 2023-10-30T12:37:07.890 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:07.892 INFO:tasks.workunit.client.0.smithi089.stderr:+ cat 2023-10-30T12:37:07.911 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:07.924 INFO:tasks.workunit.client.0.smithi089.stderr:+ chmod 777 a/foo 2023-10-30T12:37:07.925 INFO:tasks.workunit.client.0.smithi089.stderr:+ sync 2023-10-30T12:37:07.969 INFO:tasks.workunit.client.0.smithi089.stderr:+ ps 2023-10-30T12:37:07.984 INFO:tasks.workunit.client.0.smithi089.stdout: PID TTY TIME CMD 2023-10-30T12:37:07.984 INFO:tasks.workunit.client.0.smithi089.stdout: 1293 ? 00:00:00 systemd 2023-10-30T12:37:07.985 INFO:tasks.workunit.client.0.smithi089.stdout: 1294 ? 00:00:00 (sd-pam) 2023-10-30T12:37:07.985 INFO:tasks.workunit.client.0.smithi089.stdout: 27974 ? 00:00:01 sshd 2023-10-30T12:37:07.985 INFO:tasks.workunit.client.0.smithi089.stdout: 49407 ? 00:00:00 timeout 2023-10-30T12:37:07.985 INFO:tasks.workunit.client.0.smithi089.stdout: 49410 ? 00:00:00 snaptest-double 2023-10-30T12:37:07.985 INFO:tasks.workunit.client.0.smithi089.stdout: 49428 ? 00:00:00 ps 2023-10-30T12:37:07.986 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:08.008 INFO:tasks.workunit.client.0.smithi089.stderr:+ rm a/foo 2023-10-30T12:37:08.013 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a 2023-10-30T12:37:08.027 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a 2023-10-30T12:37:08.031 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:08.033 INFO:tasks.workunit.client.0.smithi089.stderr:+ cat 2023-10-30T12:37:08.053 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:08.066 INFO:tasks.workunit.client.0.smithi089.stderr:+ chmod 777 a/foo 2023-10-30T12:37:08.067 INFO:tasks.workunit.client.0.smithi089.stderr:+ sync 2023-10-30T12:37:08.101 INFO:tasks.workunit.client.0.smithi089.stderr:+ ps 2023-10-30T12:37:08.116 INFO:tasks.workunit.client.0.smithi089.stdout: PID TTY TIME CMD 2023-10-30T12:37:08.116 INFO:tasks.workunit.client.0.smithi089.stdout: 1293 ? 00:00:00 systemd 2023-10-30T12:37:08.116 INFO:tasks.workunit.client.0.smithi089.stdout: 1294 ? 00:00:00 (sd-pam) 2023-10-30T12:37:08.117 INFO:tasks.workunit.client.0.smithi089.stdout: 27974 ? 00:00:01 sshd 2023-10-30T12:37:08.117 INFO:tasks.workunit.client.0.smithi089.stdout: 49407 ? 00:00:00 timeout 2023-10-30T12:37:08.117 INFO:tasks.workunit.client.0.smithi089.stdout: 49410 ? 00:00:00 snaptest-double 2023-10-30T12:37:08.117 INFO:tasks.workunit.client.0.smithi089.stdout: 49438 ? 00:00:00 ps 2023-10-30T12:37:08.118 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:09.103 INFO:tasks.workunit.client.0.smithi089.stderr:+ rm a/foo 2023-10-30T12:37:09.112 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a 2023-10-30T12:37:09.128 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a 2023-10-30T12:37:09.132 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:09.135 INFO:tasks.workunit.client.0.smithi089.stderr:+ cat 2023-10-30T12:37:09.155 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:09.167 INFO:tasks.workunit.client.0.smithi089.stderr:+ chmod 777 a/foo 2023-10-30T12:37:09.168 INFO:tasks.workunit.client.0.smithi089.stderr:+ sync 2023-10-30T12:37:09.227 INFO:tasks.workunit.client.0.smithi089.stderr:+ ps 2023-10-30T12:37:09.242 INFO:tasks.workunit.client.0.smithi089.stdout: PID TTY TIME CMD 2023-10-30T12:37:09.243 INFO:tasks.workunit.client.0.smithi089.stdout: 1293 ? 00:00:00 systemd 2023-10-30T12:37:09.243 INFO:tasks.workunit.client.0.smithi089.stdout: 1294 ? 00:00:00 (sd-pam) 2023-10-30T12:37:09.243 INFO:tasks.workunit.client.0.smithi089.stdout: 27974 ? 00:00:01 sshd 2023-10-30T12:37:09.243 INFO:tasks.workunit.client.0.smithi089.stdout: 49407 ? 00:00:00 timeout 2023-10-30T12:37:09.244 INFO:tasks.workunit.client.0.smithi089.stdout: 49410 ? 00:00:00 snaptest-double 2023-10-30T12:37:09.244 INFO:tasks.workunit.client.0.smithi089.stdout: 49448 ? 00:00:00 ps 2023-10-30T12:37:09.244 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:09.266 INFO:tasks.workunit.client.0.smithi089.stderr:+ rm a/foo 2023-10-30T12:37:09.272 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a 2023-10-30T12:37:09.287 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a 2023-10-30T12:37:09.291 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:09.293 INFO:tasks.workunit.client.0.smithi089.stderr:+ cat 2023-10-30T12:37:09.312 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:09.324 INFO:tasks.workunit.client.0.smithi089.stderr:+ chmod 777 a/foo 2023-10-30T12:37:09.326 INFO:tasks.workunit.client.0.smithi089.stderr:+ sync 2023-10-30T12:37:09.367 INFO:tasks.workunit.client.0.smithi089.stderr:+ ps 2023-10-30T12:37:09.382 INFO:tasks.workunit.client.0.smithi089.stdout: PID TTY TIME CMD 2023-10-30T12:37:09.383 INFO:tasks.workunit.client.0.smithi089.stdout: 1293 ? 00:00:00 systemd 2023-10-30T12:37:09.383 INFO:tasks.workunit.client.0.smithi089.stdout: 1294 ? 00:00:00 (sd-pam) 2023-10-30T12:37:09.383 INFO:tasks.workunit.client.0.smithi089.stdout: 27974 ? 00:00:01 sshd 2023-10-30T12:37:09.383 INFO:tasks.workunit.client.0.smithi089.stdout: 49407 ? 00:00:00 timeout 2023-10-30T12:37:09.383 INFO:tasks.workunit.client.0.smithi089.stdout: 49410 ? 00:00:00 snaptest-double 2023-10-30T12:37:09.384 INFO:tasks.workunit.client.0.smithi089.stdout: 49458 ? 00:00:00 ps 2023-10-30T12:37:09.384 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:09.407 INFO:tasks.workunit.client.0.smithi089.stderr:+ rm a/foo 2023-10-30T12:37:09.413 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a 2023-10-30T12:37:09.428 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a 2023-10-30T12:37:09.432 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:09.434 INFO:tasks.workunit.client.0.smithi089.stderr:+ cat 2023-10-30T12:37:09.456 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:09.469 INFO:tasks.workunit.client.0.smithi089.stderr:+ chmod 777 a/foo 2023-10-30T12:37:09.470 INFO:tasks.workunit.client.0.smithi089.stderr:+ sync 2023-10-30T12:37:09.518 INFO:tasks.workunit.client.0.smithi089.stderr:+ ps 2023-10-30T12:37:09.533 INFO:tasks.workunit.client.0.smithi089.stdout: PID TTY TIME CMD 2023-10-30T12:37:09.534 INFO:tasks.workunit.client.0.smithi089.stdout: 1293 ? 00:00:00 systemd 2023-10-30T12:37:09.534 INFO:tasks.workunit.client.0.smithi089.stdout: 1294 ? 00:00:00 (sd-pam) 2023-10-30T12:37:09.534 INFO:tasks.workunit.client.0.smithi089.stdout: 27974 ? 00:00:01 sshd 2023-10-30T12:37:09.534 INFO:tasks.workunit.client.0.smithi089.stdout: 49407 ? 00:00:00 timeout 2023-10-30T12:37:09.534 INFO:tasks.workunit.client.0.smithi089.stdout: 49410 ? 00:00:00 snaptest-double 2023-10-30T12:37:09.535 INFO:tasks.workunit.client.0.smithi089.stdout: 49468 ? 00:00:00 ps 2023-10-30T12:37:09.535 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:09.559 INFO:tasks.workunit.client.0.smithi089.stderr:+ rm a/foo 2023-10-30T12:37:09.567 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a 2023-10-30T12:37:09.585 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a 2023-10-30T12:37:09.591 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:09.594 INFO:tasks.workunit.client.0.smithi089.stderr:+ cat 2023-10-30T12:37:09.615 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:09.629 INFO:tasks.workunit.client.0.smithi089.stderr:+ chmod 777 a/foo 2023-10-30T12:37:09.630 INFO:tasks.workunit.client.0.smithi089.stderr:+ sync 2023-10-30T12:37:09.677 INFO:tasks.workunit.client.0.smithi089.stderr:+ ps 2023-10-30T12:37:09.692 INFO:tasks.workunit.client.0.smithi089.stdout: PID TTY TIME CMD 2023-10-30T12:37:09.692 INFO:tasks.workunit.client.0.smithi089.stdout: 1293 ? 00:00:00 systemd 2023-10-30T12:37:09.693 INFO:tasks.workunit.client.0.smithi089.stdout: 1294 ? 00:00:00 (sd-pam) 2023-10-30T12:37:09.693 INFO:tasks.workunit.client.0.smithi089.stdout: 27974 ? 00:00:01 sshd 2023-10-30T12:37:09.693 INFO:tasks.workunit.client.0.smithi089.stdout: 49407 ? 00:00:00 timeout 2023-10-30T12:37:09.693 INFO:tasks.workunit.client.0.smithi089.stdout: 49410 ? 00:00:00 snaptest-double 2023-10-30T12:37:09.694 INFO:tasks.workunit.client.0.smithi089.stdout: 49478 ? 00:00:00 ps 2023-10-30T12:37:09.694 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:09.719 INFO:tasks.workunit.client.0.smithi089.stderr:+ rm a/foo 2023-10-30T12:37:09.727 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a 2023-10-30T12:37:09.742 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a 2023-10-30T12:37:09.746 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:09.748 INFO:tasks.workunit.client.0.smithi089.stderr:+ cat 2023-10-30T12:37:09.769 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:09.783 INFO:tasks.workunit.client.0.smithi089.stderr:+ chmod 777 a/foo 2023-10-30T12:37:09.784 INFO:tasks.workunit.client.0.smithi089.stderr:+ sync 2023-10-30T12:37:09.826 INFO:tasks.workunit.client.0.smithi089.stderr:+ ps 2023-10-30T12:37:09.841 INFO:tasks.workunit.client.0.smithi089.stdout: PID TTY TIME CMD 2023-10-30T12:37:09.841 INFO:tasks.workunit.client.0.smithi089.stdout: 1293 ? 00:00:00 systemd 2023-10-30T12:37:09.842 INFO:tasks.workunit.client.0.smithi089.stdout: 1294 ? 00:00:00 (sd-pam) 2023-10-30T12:37:09.842 INFO:tasks.workunit.client.0.smithi089.stdout: 27974 ? 00:00:01 sshd 2023-10-30T12:37:09.842 INFO:tasks.workunit.client.0.smithi089.stdout: 49407 ? 00:00:00 timeout 2023-10-30T12:37:09.842 INFO:tasks.workunit.client.0.smithi089.stdout: 49410 ? 00:00:00 snaptest-double 2023-10-30T12:37:09.842 INFO:tasks.workunit.client.0.smithi089.stdout: 49488 ? 00:00:00 ps 2023-10-30T12:37:09.843 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:09.867 INFO:tasks.workunit.client.0.smithi089.stderr:+ rm a/foo 2023-10-30T12:37:09.875 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a 2023-10-30T12:37:09.895 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a 2023-10-30T12:37:09.916 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:09.918 INFO:tasks.workunit.client.0.smithi089.stderr:+ cat 2023-10-30T12:37:09.953 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:09.966 INFO:tasks.workunit.client.0.smithi089.stderr:+ chmod 777 a/foo 2023-10-30T12:37:09.967 INFO:tasks.workunit.client.0.smithi089.stderr:+ sync 2023-10-30T12:37:10.011 INFO:tasks.workunit.client.0.smithi089.stderr:+ ps 2023-10-30T12:37:10.026 INFO:tasks.workunit.client.0.smithi089.stdout: PID TTY TIME CMD 2023-10-30T12:37:10.026 INFO:tasks.workunit.client.0.smithi089.stdout: 1293 ? 00:00:00 systemd 2023-10-30T12:37:10.026 INFO:tasks.workunit.client.0.smithi089.stdout: 1294 ? 00:00:00 (sd-pam) 2023-10-30T12:37:10.026 INFO:tasks.workunit.client.0.smithi089.stdout: 27974 ? 00:00:01 sshd 2023-10-30T12:37:10.027 INFO:tasks.workunit.client.0.smithi089.stdout: 49407 ? 00:00:00 timeout 2023-10-30T12:37:10.027 INFO:tasks.workunit.client.0.smithi089.stdout: 49410 ? 00:00:00 snaptest-double 2023-10-30T12:37:10.027 INFO:tasks.workunit.client.0.smithi089.stdout: 49498 ? 00:00:00 ps 2023-10-30T12:37:10.028 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:10.050 INFO:tasks.workunit.client.0.smithi089.stderr:+ rm a/foo 2023-10-30T12:37:10.056 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a 2023-10-30T12:37:10.072 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a 2023-10-30T12:37:10.077 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:10.080 INFO:tasks.workunit.client.0.smithi089.stderr:+ cat 2023-10-30T12:37:10.098 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:10.110 INFO:tasks.workunit.client.0.smithi089.stderr:+ chmod 777 a/foo 2023-10-30T12:37:10.111 INFO:tasks.workunit.client.0.smithi089.stderr:+ sync 2023-10-30T12:37:10.169 INFO:tasks.workunit.client.0.smithi089.stderr:+ ps 2023-10-30T12:37:10.184 INFO:tasks.workunit.client.0.smithi089.stdout: PID TTY TIME CMD 2023-10-30T12:37:10.184 INFO:tasks.workunit.client.0.smithi089.stdout: 1293 ? 00:00:00 systemd 2023-10-30T12:37:10.185 INFO:tasks.workunit.client.0.smithi089.stdout: 1294 ? 00:00:00 (sd-pam) 2023-10-30T12:37:10.185 INFO:tasks.workunit.client.0.smithi089.stdout: 27974 ? 00:00:01 sshd 2023-10-30T12:37:10.185 INFO:tasks.workunit.client.0.smithi089.stdout: 49407 ? 00:00:00 timeout 2023-10-30T12:37:10.185 INFO:tasks.workunit.client.0.smithi089.stdout: 49410 ? 00:00:00 snaptest-double 2023-10-30T12:37:10.185 INFO:tasks.workunit.client.0.smithi089.stdout: 49508 ? 00:00:00 ps 2023-10-30T12:37:10.186 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:10.214 INFO:tasks.workunit.client.0.smithi089.stderr:+ rm a/foo 2023-10-30T12:37:10.222 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a 2023-10-30T12:37:10.613 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a 2023-10-30T12:37:10.617 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:10.620 INFO:tasks.workunit.client.0.smithi089.stderr:+ cat 2023-10-30T12:37:10.641 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:10.655 INFO:tasks.workunit.client.0.smithi089.stderr:+ chmod 777 a/foo 2023-10-30T12:37:10.656 INFO:tasks.workunit.client.0.smithi089.stderr:+ sync 2023-10-30T12:37:10.713 INFO:tasks.workunit.client.0.smithi089.stderr:+ ps 2023-10-30T12:37:10.728 INFO:tasks.workunit.client.0.smithi089.stdout: PID TTY TIME CMD 2023-10-30T12:37:10.728 INFO:tasks.workunit.client.0.smithi089.stdout: 1293 ? 00:00:00 systemd 2023-10-30T12:37:10.728 INFO:tasks.workunit.client.0.smithi089.stdout: 1294 ? 00:00:00 (sd-pam) 2023-10-30T12:37:10.729 INFO:tasks.workunit.client.0.smithi089.stdout: 27974 ? 00:00:01 sshd 2023-10-30T12:37:10.729 INFO:tasks.workunit.client.0.smithi089.stdout: 49407 ? 00:00:00 timeout 2023-10-30T12:37:10.729 INFO:tasks.workunit.client.0.smithi089.stdout: 49410 ? 00:00:00 snaptest-double 2023-10-30T12:37:10.729 INFO:tasks.workunit.client.0.smithi089.stdout: 49518 ? 00:00:00 ps 2023-10-30T12:37:10.730 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:11.427 INFO:tasks.workunit.client.0.smithi089.stderr:+ rm a/foo 2023-10-30T12:37:11.707 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a 2023-10-30T12:37:11.723 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a 2023-10-30T12:37:11.727 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:11.729 INFO:tasks.workunit.client.0.smithi089.stderr:+ cat 2023-10-30T12:37:11.750 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:11.764 INFO:tasks.workunit.client.0.smithi089.stderr:+ chmod 777 a/foo 2023-10-30T12:37:11.765 INFO:tasks.workunit.client.0.smithi089.stderr:+ sync 2023-10-30T12:37:11.830 INFO:tasks.workunit.client.0.smithi089.stderr:+ ps 2023-10-30T12:37:11.845 INFO:tasks.workunit.client.0.smithi089.stdout: PID TTY TIME CMD 2023-10-30T12:37:11.846 INFO:tasks.workunit.client.0.smithi089.stdout: 1293 ? 00:00:00 systemd 2023-10-30T12:37:11.846 INFO:tasks.workunit.client.0.smithi089.stdout: 1294 ? 00:00:00 (sd-pam) 2023-10-30T12:37:11.847 INFO:tasks.workunit.client.0.smithi089.stdout: 27974 ? 00:00:01 sshd 2023-10-30T12:37:11.849 INFO:tasks.workunit.client.0.smithi089.stdout: 49407 ? 00:00:00 timeout 2023-10-30T12:37:11.849 INFO:tasks.workunit.client.0.smithi089.stdout: 49410 ? 00:00:00 snaptest-double 2023-10-30T12:37:11.850 INFO:tasks.workunit.client.0.smithi089.stdout: 49528 ? 00:00:00 ps 2023-10-30T12:37:11.850 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:12.123 INFO:tasks.workunit.client.0.smithi089.stderr:+ rm a/foo 2023-10-30T12:37:12.648 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a 2023-10-30T12:37:12.947 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a 2023-10-30T12:37:12.951 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:12.954 INFO:tasks.workunit.client.0.smithi089.stderr:+ cat 2023-10-30T12:37:12.973 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:12.984 INFO:tasks.workunit.client.0.smithi089.stderr:+ chmod 777 a/foo 2023-10-30T12:37:12.985 INFO:tasks.workunit.client.0.smithi089.stderr:+ sync 2023-10-30T12:37:13.454 INFO:tasks.workunit.client.0.smithi089.stderr:+ ps 2023-10-30T12:37:13.469 INFO:tasks.workunit.client.0.smithi089.stdout: PID TTY TIME CMD 2023-10-30T12:37:13.469 INFO:tasks.workunit.client.0.smithi089.stdout: 1293 ? 00:00:00 systemd 2023-10-30T12:37:13.469 INFO:tasks.workunit.client.0.smithi089.stdout: 1294 ? 00:00:00 (sd-pam) 2023-10-30T12:37:13.469 INFO:tasks.workunit.client.0.smithi089.stdout: 27974 ? 00:00:01 sshd 2023-10-30T12:37:13.470 INFO:tasks.workunit.client.0.smithi089.stdout: 49407 ? 00:00:00 timeout 2023-10-30T12:37:13.470 INFO:tasks.workunit.client.0.smithi089.stdout: 49410 ? 00:00:00 snaptest-double 2023-10-30T12:37:13.470 INFO:tasks.workunit.client.0.smithi089.stdout: 49538 ? 00:00:00 ps 2023-10-30T12:37:13.471 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:13.494 INFO:tasks.workunit.client.0.smithi089.stderr:+ rm a/foo 2023-10-30T12:37:13.501 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a 2023-10-30T12:37:13.520 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a 2023-10-30T12:37:13.526 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:13.529 INFO:tasks.workunit.client.0.smithi089.stderr:+ cat 2023-10-30T12:37:13.552 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:13.566 INFO:tasks.workunit.client.0.smithi089.stderr:+ chmod 777 a/foo 2023-10-30T12:37:13.567 INFO:tasks.workunit.client.0.smithi089.stderr:+ sync 2023-10-30T12:37:13.638 INFO:tasks.workunit.client.0.smithi089.stderr:+ ps 2023-10-30T12:37:13.651 INFO:tasks.workunit.client.0.smithi089.stdout: PID TTY TIME CMD 2023-10-30T12:37:13.651 INFO:tasks.workunit.client.0.smithi089.stdout: 1293 ? 00:00:00 systemd 2023-10-30T12:37:13.651 INFO:tasks.workunit.client.0.smithi089.stdout: 1294 ? 00:00:00 (sd-pam) 2023-10-30T12:37:13.652 INFO:tasks.workunit.client.0.smithi089.stdout: 27974 ? 00:00:01 sshd 2023-10-30T12:37:13.652 INFO:tasks.workunit.client.0.smithi089.stdout: 49407 ? 00:00:00 timeout 2023-10-30T12:37:13.652 INFO:tasks.workunit.client.0.smithi089.stdout: 49410 ? 00:00:00 snaptest-double 2023-10-30T12:37:13.652 INFO:tasks.workunit.client.0.smithi089.stdout: 49548 ? 00:00:00 ps 2023-10-30T12:37:13.653 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:13.676 INFO:tasks.workunit.client.0.smithi089.stderr:+ rm a/foo 2023-10-30T12:37:13.681 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a 2023-10-30T12:37:14.141 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a 2023-10-30T12:37:14.150 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:14.162 INFO:tasks.workunit.client.0.smithi089.stderr:+ cat 2023-10-30T12:37:14.702 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:14.714 INFO:tasks.workunit.client.0.smithi089.stderr:+ chmod 777 a/foo 2023-10-30T12:37:14.717 INFO:tasks.workunit.client.0.smithi089.stderr:+ sync 2023-10-30T12:37:14.797 INFO:tasks.workunit.client.0.smithi089.stderr:+ ps 2023-10-30T12:37:14.811 INFO:tasks.workunit.client.0.smithi089.stdout: PID TTY TIME CMD 2023-10-30T12:37:14.812 INFO:tasks.workunit.client.0.smithi089.stdout: 1293 ? 00:00:00 systemd 2023-10-30T12:37:14.812 INFO:tasks.workunit.client.0.smithi089.stdout: 1294 ? 00:00:00 (sd-pam) 2023-10-30T12:37:14.812 INFO:tasks.workunit.client.0.smithi089.stdout: 27974 ? 00:00:01 sshd 2023-10-30T12:37:14.812 INFO:tasks.workunit.client.0.smithi089.stdout: 49407 ? 00:00:00 timeout 2023-10-30T12:37:14.812 INFO:tasks.workunit.client.0.smithi089.stdout: 49410 ? 00:00:00 snaptest-double 2023-10-30T12:37:14.813 INFO:tasks.workunit.client.0.smithi089.stdout: 49558 ? 00:00:00 ps 2023-10-30T12:37:14.813 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:14.836 INFO:tasks.workunit.client.0.smithi089.stderr:+ rm a/foo 2023-10-30T12:37:14.841 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a 2023-10-30T12:37:14.857 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a 2023-10-30T12:37:14.862 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:14.864 INFO:tasks.workunit.client.0.smithi089.stderr:+ cat 2023-10-30T12:37:15.040 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:15.054 INFO:tasks.workunit.client.0.smithi089.stderr:+ chmod 777 a/foo 2023-10-30T12:37:15.055 INFO:tasks.workunit.client.0.smithi089.stderr:+ sync 2023-10-30T12:37:15.120 INFO:tasks.workunit.client.0.smithi089.stderr:+ ps 2023-10-30T12:37:15.135 INFO:tasks.workunit.client.0.smithi089.stdout: PID TTY TIME CMD 2023-10-30T12:37:15.135 INFO:tasks.workunit.client.0.smithi089.stdout: 1293 ? 00:00:00 systemd 2023-10-30T12:37:15.136 INFO:tasks.workunit.client.0.smithi089.stdout: 1294 ? 00:00:00 (sd-pam) 2023-10-30T12:37:15.136 INFO:tasks.workunit.client.0.smithi089.stdout: 27974 ? 00:00:01 sshd 2023-10-30T12:37:15.136 INFO:tasks.workunit.client.0.smithi089.stdout: 49407 ? 00:00:00 timeout 2023-10-30T12:37:15.136 INFO:tasks.workunit.client.0.smithi089.stdout: 49410 ? 00:00:00 snaptest-double 2023-10-30T12:37:15.137 INFO:tasks.workunit.client.0.smithi089.stdout: 49568 ? 00:00:00 ps 2023-10-30T12:37:15.137 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:15.196 INFO:tasks.workunit.client.0.smithi089.stderr:+ rm a/foo 2023-10-30T12:37:15.203 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a 2023-10-30T12:37:15.215 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a 2023-10-30T12:37:15.219 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:15.221 INFO:tasks.workunit.client.0.smithi089.stderr:+ cat 2023-10-30T12:37:15.237 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:15.248 INFO:tasks.workunit.client.0.smithi089.stderr:+ chmod 777 a/foo 2023-10-30T12:37:15.249 INFO:tasks.workunit.client.0.smithi089.stderr:+ sync 2023-10-30T12:37:15.313 INFO:tasks.workunit.client.0.smithi089.stderr:+ ps 2023-10-30T12:37:15.327 INFO:tasks.workunit.client.0.smithi089.stdout: PID TTY TIME CMD 2023-10-30T12:37:15.328 INFO:tasks.workunit.client.0.smithi089.stdout: 1293 ? 00:00:00 systemd 2023-10-30T12:37:15.328 INFO:tasks.workunit.client.0.smithi089.stdout: 1294 ? 00:00:00 (sd-pam) 2023-10-30T12:37:15.328 INFO:tasks.workunit.client.0.smithi089.stdout: 27974 ? 00:00:01 sshd 2023-10-30T12:37:15.328 INFO:tasks.workunit.client.0.smithi089.stdout: 49407 ? 00:00:00 timeout 2023-10-30T12:37:15.328 INFO:tasks.workunit.client.0.smithi089.stdout: 49410 ? 00:00:00 snaptest-double 2023-10-30T12:37:15.329 INFO:tasks.workunit.client.0.smithi089.stdout: 49578 ? 00:00:00 ps 2023-10-30T12:37:15.329 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:15.351 INFO:tasks.workunit.client.0.smithi089.stderr:+ rm a/foo 2023-10-30T12:37:15.658 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a 2023-10-30T12:37:16.043 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:37:16.047 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:37:16.199 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a 2023-10-30T12:37:16.203 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:16.205 INFO:tasks.workunit.client.0.smithi089.stderr:+ cat 2023-10-30T12:37:16.235 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:16.249 INFO:tasks.workunit.client.0.smithi089.stderr:+ chmod 777 a/foo 2023-10-30T12:37:16.250 INFO:tasks.workunit.client.0.smithi089.stderr:+ sync 2023-10-30T12:37:16.566 INFO:tasks.workunit.client.0.smithi089.stderr:+ ps 2023-10-30T12:37:16.580 INFO:tasks.workunit.client.0.smithi089.stdout: PID TTY TIME CMD 2023-10-30T12:37:16.580 INFO:tasks.workunit.client.0.smithi089.stdout: 1293 ? 00:00:00 systemd 2023-10-30T12:37:16.580 INFO:tasks.workunit.client.0.smithi089.stdout: 1294 ? 00:00:00 (sd-pam) 2023-10-30T12:37:16.580 INFO:tasks.workunit.client.0.smithi089.stdout: 27974 ? 00:00:01 sshd 2023-10-30T12:37:16.581 INFO:tasks.workunit.client.0.smithi089.stdout: 49407 ? 00:00:00 timeout 2023-10-30T12:37:16.581 INFO:tasks.workunit.client.0.smithi089.stdout: 49410 ? 00:00:00 snaptest-double 2023-10-30T12:37:16.581 INFO:tasks.workunit.client.0.smithi089.stdout: 49590 ? 00:00:00 ps 2023-10-30T12:37:16.581 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:16.606 INFO:tasks.workunit.client.0.smithi089.stderr:+ rm a/foo 2023-10-30T12:37:16.618 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a 2023-10-30T12:37:16.634 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a 2023-10-30T12:37:16.639 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:16.641 INFO:tasks.workunit.client.0.smithi089.stderr:+ cat 2023-10-30T12:37:16.664 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:17.064 INFO:tasks.workunit.client.0.smithi089.stderr:+ chmod 777 a/foo 2023-10-30T12:37:17.065 INFO:tasks.workunit.client.0.smithi089.stderr:+ sync 2023-10-30T12:37:17.834 INFO:tasks.workunit.client.0.smithi089.stderr:+ ps 2023-10-30T12:37:17.849 INFO:tasks.workunit.client.0.smithi089.stdout: PID TTY TIME CMD 2023-10-30T12:37:17.849 INFO:tasks.workunit.client.0.smithi089.stdout: 1293 ? 00:00:00 systemd 2023-10-30T12:37:17.849 INFO:tasks.workunit.client.0.smithi089.stdout: 1294 ? 00:00:00 (sd-pam) 2023-10-30T12:37:17.849 INFO:tasks.workunit.client.0.smithi089.stdout: 27974 ? 00:00:01 sshd 2023-10-30T12:37:17.849 INFO:tasks.workunit.client.0.smithi089.stdout: 49407 ? 00:00:00 timeout 2023-10-30T12:37:17.850 INFO:tasks.workunit.client.0.smithi089.stdout: 49410 ? 00:00:00 snaptest-double 2023-10-30T12:37:17.850 INFO:tasks.workunit.client.0.smithi089.stdout: 49600 ? 00:00:00 ps 2023-10-30T12:37:17.850 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:17.874 INFO:tasks.workunit.client.0.smithi089.stderr:+ rm a/foo 2023-10-30T12:37:17.878 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a 2023-10-30T12:37:17.890 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a 2023-10-30T12:37:17.894 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:17.896 INFO:tasks.workunit.client.0.smithi089.stderr:+ cat 2023-10-30T12:37:17.913 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:17.924 INFO:tasks.workunit.client.0.smithi089.stderr:+ chmod 777 a/foo 2023-10-30T12:37:17.925 INFO:tasks.workunit.client.0.smithi089.stderr:+ sync 2023-10-30T12:37:18.091 INFO:tasks.workunit.client.0.smithi089.stderr:+ ps 2023-10-30T12:37:18.105 INFO:tasks.workunit.client.0.smithi089.stdout: PID TTY TIME CMD 2023-10-30T12:37:18.105 INFO:tasks.workunit.client.0.smithi089.stdout: 1293 ? 00:00:00 systemd 2023-10-30T12:37:18.105 INFO:tasks.workunit.client.0.smithi089.stdout: 1294 ? 00:00:00 (sd-pam) 2023-10-30T12:37:18.106 INFO:tasks.workunit.client.0.smithi089.stdout: 27974 ? 00:00:01 sshd 2023-10-30T12:37:18.106 INFO:tasks.workunit.client.0.smithi089.stdout: 49407 ? 00:00:00 timeout 2023-10-30T12:37:18.106 INFO:tasks.workunit.client.0.smithi089.stdout: 49410 ? 00:00:00 snaptest-double 2023-10-30T12:37:18.106 INFO:tasks.workunit.client.0.smithi089.stdout: 49610 ? 00:00:00 ps 2023-10-30T12:37:18.107 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:18.282 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:37:18.326 INFO:tasks.workunit.client.0.smithi089.stderr:+ rm a/foo 2023-10-30T12:37:18.330 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir a 2023-10-30T12:37:18.342 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo OK 2023-10-30T12:37:18.343 INFO:tasks.workunit.client.0.smithi089.stdout:OK 2023-10-30T12:37:18.344 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T12:37:18.344 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T12:37:18.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.423+0000 7f94ed11e640 1 Processor -- start 2023-10-30T12:37:18.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.423+0000 7f94ed11e640 1 -- start start 2023-10-30T12:37:18.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.427+0000 7f94ed11e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f94e81538f0 0x7f94e8159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:18.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.427+0000 7f94ed11e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f94e815cd50 0x7f94e80c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:18.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.427+0000 7f94ed11e640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94e815d300 con 0x7f94e80cefa0 2023-10-30T12:37:18.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.427+0000 7f94ed11e640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94e815d5e0 con 0x7f94e815cd50 2023-10-30T12:37:18.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.427+0000 7f94ed11e640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94e815d8c0 con 0x7f94e81538f0 2023-10-30T12:37:18.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.427+0000 7f94e77fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f94e815cd50 0x7f94e80c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:18.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.427+0000 7f94ec91d640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f94e80cefa0 0x7f94e80cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39924/0 (socket says 172.21.15.89:39924) 2023-10-30T12:37:18.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.427+0000 7f94ec91d640 1 -- 172.21.15.89:0/4229852298 learned_addr learned my addr 172.21.15.89:0/4229852298 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:37:18.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.427+0000 7f94e7fff640 1 --2- 172.21.15.89:0/4229852298 >> v2:172.21.15.192:3300/0 conn(0x7f94e81538f0 0x7f94e8159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:18.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.427+0000 7f94e6ffd640 1 -- 172.21.15.89:0/4229852298 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 7375374 0 0) 0x7f94e815d300 con 0x7f94e80cefa0 2023-10-30T12:37:18.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.427+0000 7f94e6ffd640 1 -- 172.21.15.89:0/4229852298 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94d0003530 con 0x7f94e80cefa0 2023-10-30T12:37:18.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.427+0000 7f94e77fe640 1 -- 172.21.15.89:0/4229852298 >> v2:172.21.15.192:3300/0 conn(0x7f94e81538f0 msgr2=0x7f94e8159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:18.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.427+0000 7f94e77fe640 1 --2- 172.21.15.89:0/4229852298 >> v2:172.21.15.192:3300/0 conn(0x7f94e81538f0 0x7f94e8159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:18.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.427+0000 7f94e77fe640 1 -- 172.21.15.89:0/4229852298 >> v1:172.21.15.89:6789/0 conn(0x7f94e80cefa0 legacy=0x7f94e80cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:18.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.427+0000 7f94e77fe640 1 -- 172.21.15.89:0/4229852298 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e815dba0 con 0x7f94e815cd50 2023-10-30T12:37:18.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.431+0000 7f94e77fe640 1 --2- 172.21.15.89:0/4229852298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94e815cd50 0x7f94e80c9f90 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f94e80c9cb0 tx=0x7f94dc01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7c5c7e7d47a4ac2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:18.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.431+0000 7f94e6ffd640 1 -- 172.21.15.89:0/4229852298 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94dc0a8950 con 0x7f94e815cd50 2023-10-30T12:37:18.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.431+0000 7f94e6ffd640 1 -- 172.21.15.89:0/4229852298 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94dc0a8ab0 con 0x7f94e815cd50 2023-10-30T12:37:18.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.431+0000 7f94e6ffd640 1 -- 172.21.15.89:0/4229852298 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94dc0a8e40 con 0x7f94e815cd50 2023-10-30T12:37:18.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.431+0000 7f94ed11e640 1 -- 172.21.15.89:0/4229852298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94e815cd50 msgr2=0x7f94e80c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:18.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.431+0000 7f94ed11e640 1 --2- 172.21.15.89:0/4229852298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94e815cd50 0x7f94e80c9f90 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f94e80c9cb0 tx=0x7f94dc01ec50 comp rx=0 tx=0).stop 2023-10-30T12:37:18.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.431+0000 7f94ed11e640 1 -- 172.21.15.89:0/4229852298 shutdown_connections 2023-10-30T12:37:18.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.431+0000 7f94ed11e640 1 --2- 172.21.15.89:0/4229852298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94e815cd50 0x7f94e80c9f90 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:18.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.431+0000 7f94ed11e640 1 --2- 172.21.15.89:0/4229852298 >> v2:172.21.15.192:3300/0 conn(0x7f94e81538f0 0x7f94e8159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:18.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.431+0000 7f94ed11e640 1 -- 172.21.15.89:0/4229852298 >> 172.21.15.89:0/4229852298 conn(0x7f94e806a640 msgr2=0x7f94e8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:18.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.431+0000 7f94ed11e640 1 -- 172.21.15.89:0/4229852298 shutdown_connections 2023-10-30T12:37:18.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.431+0000 7f94ed11e640 1 -- 172.21.15.89:0/4229852298 wait complete. 2023-10-30T12:37:18.436 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.431+0000 7f94ed11e640 1 Processor -- start 2023-10-30T12:37:18.436 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.431+0000 7f94ed11e640 1 -- start start 2023-10-30T12:37:18.436 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.435+0000 7f94ed11e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94e80cefa0 0x7f94e8147b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:18.436 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.435+0000 7f94ed11e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94e81538f0 0x7f94e8145910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:18.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.435+0000 7f94ed11e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94e815cd50 0x7f94e8145e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:18.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.435+0000 7f94ed11e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e81573a0 con 0x7f94e815cd50 2023-10-30T12:37:18.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.435+0000 7f94ed11e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e815a670 con 0x7f94e80cefa0 2023-10-30T12:37:18.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.435+0000 7f94ed11e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94e815a950 con 0x7f94e81538f0 2023-10-30T12:37:18.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.435+0000 7f94ec91d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94e815cd50 0x7f94e8145e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:18.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.435+0000 7f94ec91d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94e815cd50 0x7f94e8145e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36540/0 (socket says 172.21.15.89:36540) 2023-10-30T12:37:18.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.435+0000 7f94ec91d640 1 -- 172.21.15.89:0/859912231 learned_addr learned my addr 172.21.15.89:0/859912231 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:37:18.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.435+0000 7f94e7fff640 1 --2- 172.21.15.89:0/859912231 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94e80cefa0 0x7f94e8147b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:18.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.435+0000 7f94e77fe640 1 --2- 172.21.15.89:0/859912231 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94e81538f0 0x7f94e8145910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:18.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.435+0000 7f94ec91d640 1 -- 172.21.15.89:0/859912231 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94e81538f0 msgr2=0x7f94e8145910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:18.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.435+0000 7f94ec91d640 1 --2- 172.21.15.89:0/859912231 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94e81538f0 0x7f94e8145910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:18.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.435+0000 7f94ec91d640 1 -- 172.21.15.89:0/859912231 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94e80cefa0 msgr2=0x7f94e8147b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:18.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.435+0000 7f94ec91d640 1 --2- 172.21.15.89:0/859912231 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94e80cefa0 0x7f94e8147b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:18.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.435+0000 7f94ec91d640 1 -- 172.21.15.89:0/859912231 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94dc009420 con 0x7f94e815cd50 2023-10-30T12:37:18.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.435+0000 7f94e77fe640 1 --2- 172.21.15.89:0/859912231 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94e81538f0 0x7f94e8145910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:37:18.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.435+0000 7f94ec91d640 1 --2- 172.21.15.89:0/859912231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94e815cd50 0x7f94e8145e50 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f94e000cb40 tx=0x7f94e000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:18.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.435+0000 7f94e4ff9640 1 -- 172.21.15.89:0/859912231 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e000aca0 con 0x7f94e815cd50 2023-10-30T12:37:18.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.435+0000 7f94e4ff9640 1 -- 172.21.15.89:0/859912231 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94e000ae00 con 0x7f94e815cd50 2023-10-30T12:37:18.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.435+0000 7f94e4ff9640 1 -- 172.21.15.89:0/859912231 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e0012620 con 0x7f94e815cd50 2023-10-30T12:37:18.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.435+0000 7f94ed11e640 1 -- 172.21.15.89:0/859912231 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94e815ac30 con 0x7f94e815cd50 2023-10-30T12:37:18.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.435+0000 7f94ed11e640 1 -- 172.21.15.89:0/859912231 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94e815b1f0 con 0x7f94e815cd50 2023-10-30T12:37:18.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.435+0000 7f94ed11e640 1 -- 172.21.15.89:0/859912231 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94e8159730 con 0x7f94e815cd50 2023-10-30T12:37:18.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.439+0000 7f94e4ff9640 1 -- 172.21.15.89:0/859912231 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f94e0007070 con 0x7f94e815cd50 2023-10-30T12:37:18.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.439+0000 7f94e4ff9640 1 --2- 172.21.15.89:0/859912231 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f94d40664a0 0x7f94d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:18.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.439+0000 7f94e4ff9640 1 -- 172.21.15.89:0/859912231 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6118+0+0 (secure 0 0 0) 0x7f94e0084090 con 0x7f94e815cd50 2023-10-30T12:37:18.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.439+0000 7f94e7fff640 1 --2- 172.21.15.89:0/859912231 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f94d40664a0 0x7f94d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:18.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.439+0000 7f94e7fff640 1 --2- 172.21.15.89:0/859912231 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f94d40664a0 0x7f94d4068960 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f94d8009b60 tx=0x7f94d8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:18.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.443+0000 7f94e4ff9640 1 -- 172.21.15.89:0/859912231 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f94e0051380 con 0x7f94e815cd50 2023-10-30T12:37:18.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.623+0000 7f94ed11e640 1 -- 172.21.15.89:0/859912231 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94e804e170 con 0x7f94e815cd50 2023-10-30T12:37:18.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.623+0000 7f94e4ff9640 1 -- 172.21.15.89:0/859912231 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6527 (secure 0 0 0) 0x7f94e00549a0 con 0x7f94e815cd50 2023-10-30T12:37:18.624 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:37:18.624 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5018,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:37:11.414400+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:37:18.624 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 43 2023-10-30T12:37:18.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.623+0000 7f94ed11e640 1 -- 172.21.15.89:0/859912231 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f94d40664a0 msgr2=0x7f94d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:18.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.623+0000 7f94ed11e640 1 --2- 172.21.15.89:0/859912231 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f94d40664a0 0x7f94d4068960 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f94d8009b60 tx=0x7f94d8009190 comp rx=0 tx=0).stop 2023-10-30T12:37:18.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.623+0000 7f94ed11e640 1 -- 172.21.15.89:0/859912231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94e815cd50 msgr2=0x7f94e8145e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:18.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.623+0000 7f94ed11e640 1 --2- 172.21.15.89:0/859912231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94e815cd50 0x7f94e8145e50 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f94e000cb40 tx=0x7f94e000a060 comp rx=0 tx=0).stop 2023-10-30T12:37:18.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.623+0000 7f94ed11e640 1 -- 172.21.15.89:0/859912231 shutdown_connections 2023-10-30T12:37:18.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.623+0000 7f94ed11e640 1 --2- 172.21.15.89:0/859912231 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f94d40664a0 0x7f94d4068960 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:18.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.623+0000 7f94ed11e640 1 --2- 172.21.15.89:0/859912231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94e815cd50 0x7f94e8145e50 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:18.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.623+0000 7f94ed11e640 1 --2- 172.21.15.89:0/859912231 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94e81538f0 0x7f94e8145910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:18.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.623+0000 7f94ed11e640 1 --2- 172.21.15.89:0/859912231 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94e80cefa0 0x7f94e8147b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:18.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.623+0000 7f94ed11e640 1 -- 172.21.15.89:0/859912231 >> 172.21.15.89:0/859912231 conn(0x7f94e806a640 msgr2=0x7f94e80cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:18.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.627+0000 7f94ed11e640 1 -- 172.21.15.89:0/859912231 shutdown_connections 2023-10-30T12:37:18.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.627+0000 7f94ed11e640 1 -- 172.21.15.89:0/859912231 wait complete. 2023-10-30T12:37:18.644 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=0) 2023-10-30T12:37:18.645 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-30T12:37:18.645 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:37:18.669 INFO:tasks.ceph.mds.f:Stopped 2023-10-30T12:37:18.669 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:37:18.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.819+0000 7f705470f640 1 Processor -- start 2023-10-30T12:37:18.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.819+0000 7f705470f640 1 -- start start 2023-10-30T12:37:18.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.823+0000 7f705470f640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f704c1539a0 0x7f704c15a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:18.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.823+0000 7f705470f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f704c069e90 0x7f704c06a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:18.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.823+0000 7f705470f640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f704c15a8c0 con 0x7f704c070a00 2023-10-30T12:37:18.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.823+0000 7f705470f640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f704c06fbb0 con 0x7f704c1539a0 2023-10-30T12:37:18.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.823+0000 7f705470f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f704c153d80 con 0x7f704c069e90 2023-10-30T12:37:18.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.823+0000 7f70524ab640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f704c1539a0 0x7f704c15a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:18.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.823+0000 7f70524ab640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f704c1539a0 0x7f704c15a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36550/0 (socket says 172.21.15.89:36550) 2023-10-30T12:37:18.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.823+0000 7f70524ab640 1 -- 172.21.15.89:0/2350834778 learned_addr learned my addr 172.21.15.89:0/2350834778 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:37:18.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.823+0000 7f7052cac640 1 --2- 172.21.15.89:0/2350834778 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f704c069e90 0x7f704c06a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:18.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.823+0000 7f70524ab640 1 -- 172.21.15.89:0/2350834778 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f704c069e90 msgr2=0x7f704c06a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:18.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.823+0000 7f70524ab640 1 --2- 172.21.15.89:0/2350834778 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f704c069e90 0x7f704c06a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:18.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.823+0000 7f70524ab640 1 -- 172.21.15.89:0/2350834778 >> v1:172.21.15.192:6789/0 conn(0x7f704c070a00 legacy=0x7f704c070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:18.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.823+0000 7f70524ab640 1 -- 172.21.15.89:0/2350834778 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f704c0c5c50 con 0x7f704c1539a0 2023-10-30T12:37:18.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.827+0000 7f70524ab640 1 --2- 172.21.15.89:0/2350834778 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f704c1539a0 0x7f704c15a380 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7048009870 tx=0x7f704801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5980d93ecea03f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:18.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.827+0000 7f70514a9640 1 -- 172.21.15.89:0/2350834778 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70480ac640 con 0x7f704c1539a0 2023-10-30T12:37:18.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.827+0000 7f70514a9640 1 -- 172.21.15.89:0/2350834778 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70480ac7a0 con 0x7f704c1539a0 2023-10-30T12:37:18.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.827+0000 7f70514a9640 1 -- 172.21.15.89:0/2350834778 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70480acb30 con 0x7f704c1539a0 2023-10-30T12:37:18.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.827+0000 7f705470f640 1 -- 172.21.15.89:0/2350834778 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f704c1539a0 msgr2=0x7f704c15a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:18.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.827+0000 7f705470f640 1 --2- 172.21.15.89:0/2350834778 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f704c1539a0 0x7f704c15a380 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7048009870 tx=0x7f704801c440 comp rx=0 tx=0).stop 2023-10-30T12:37:18.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.827+0000 7f705470f640 1 -- 172.21.15.89:0/2350834778 shutdown_connections 2023-10-30T12:37:18.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.827+0000 7f705470f640 1 --2- 172.21.15.89:0/2350834778 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f704c069e90 0x7f704c06a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:18.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.827+0000 7f705470f640 1 --2- 172.21.15.89:0/2350834778 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f704c1539a0 0x7f704c15a380 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:18.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.827+0000 7f705470f640 1 -- 172.21.15.89:0/2350834778 >> 172.21.15.89:0/2350834778 conn(0x7f704c0c7910 msgr2=0x7f704c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:18.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.827+0000 7f705470f640 1 -- 172.21.15.89:0/2350834778 shutdown_connections 2023-10-30T12:37:18.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.827+0000 7f705470f640 1 -- 172.21.15.89:0/2350834778 wait complete. 2023-10-30T12:37:18.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.827+0000 7f705470f640 1 Processor -- start 2023-10-30T12:37:18.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.827+0000 7f705470f640 1 -- start start 2023-10-30T12:37:18.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.831+0000 7f705470f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f704c069e90 0x7f704c0c4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:18.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.831+0000 7f705470f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f704c070a00 0x7f704c0c2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:18.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.831+0000 7f705470f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f704c1539a0 0x7f704c0c2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:18.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.831+0000 7f705470f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704c1f68e0 con 0x7f704c069e90 2023-10-30T12:37:18.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.831+0000 7f705470f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704c149600 con 0x7f704c1539a0 2023-10-30T12:37:18.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.831+0000 7f705470f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f704c1f6340 con 0x7f704c070a00 2023-10-30T12:37:18.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.831+0000 7f70524ab640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f704c069e90 0x7f704c0c4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:18.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.831+0000 7f70524ab640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f704c069e90 0x7f704c0c4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36566/0 (socket says 172.21.15.89:36566) 2023-10-30T12:37:18.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.831+0000 7f70524ab640 1 -- 172.21.15.89:0/1269513832 learned_addr learned my addr 172.21.15.89:0/1269513832 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:37:18.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.831+0000 7f7051caa640 1 --2- 172.21.15.89:0/1269513832 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f704c070a00 0x7f704c0c2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:18.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.831+0000 7f7052cac640 1 --2- 172.21.15.89:0/1269513832 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f704c1539a0 0x7f704c0c2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:18.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.831+0000 7f70524ab640 1 -- 172.21.15.89:0/1269513832 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f704c070a00 msgr2=0x7f704c0c2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:18.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.831+0000 7f70524ab640 1 --2- 172.21.15.89:0/1269513832 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f704c070a00 0x7f704c0c2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:18.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.831+0000 7f70524ab640 1 -- 172.21.15.89:0/1269513832 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f704c1539a0 msgr2=0x7f704c0c2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:18.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.831+0000 7f70524ab640 1 --2- 172.21.15.89:0/1269513832 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f704c1539a0 0x7f704c0c2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:18.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.831+0000 7f70524ab640 1 -- 172.21.15.89:0/1269513832 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70480094d0 con 0x7f704c069e90 2023-10-30T12:37:18.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.831+0000 7f70524ab640 1 --2- 172.21.15.89:0/1269513832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f704c069e90 0x7f704c0c4940 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f70480af040 tx=0x7f704801ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:18.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.831+0000 7f7042ffd640 1 -- 172.21.15.89:0/1269513832 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70480b7830 con 0x7f704c069e90 2023-10-30T12:37:18.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.831+0000 7f7042ffd640 1 -- 172.21.15.89:0/1269513832 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70480b7990 con 0x7f704c069e90 2023-10-30T12:37:18.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.831+0000 7f705470f640 1 -- 172.21.15.89:0/1269513832 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f704c1f6600 con 0x7f704c069e90 2023-10-30T12:37:18.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.831+0000 7f7042ffd640 1 -- 172.21.15.89:0/1269513832 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70480b7d20 con 0x7f704c069e90 2023-10-30T12:37:18.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.831+0000 7f705470f640 1 -- 172.21.15.89:0/1269513832 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f704c148fb0 con 0x7f704c069e90 2023-10-30T12:37:18.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.831+0000 7f705470f640 1 -- 172.21.15.89:0/1269513832 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f704c04e170 con 0x7f704c069e90 2023-10-30T12:37:18.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.831+0000 7f7042ffd640 1 -- 172.21.15.89:0/1269513832 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f70480b3040 con 0x7f704c069e90 2023-10-30T12:37:18.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.835+0000 7f7042ffd640 1 --2- 172.21.15.89:0/1269513832 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f70300664a0 0x7f7030068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:18.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.835+0000 7f7042ffd640 1 -- 172.21.15.89:0/1269513832 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6118+0+0 (secure 0 0 0) 0x7f704811e780 con 0x7f704c069e90 2023-10-30T12:37:18.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.835+0000 7f7051caa640 1 --2- 172.21.15.89:0/1269513832 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f70300664a0 0x7f7030068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:18.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.835+0000 7f7051caa640 1 --2- 172.21.15.89:0/1269513832 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f70300664a0 0x7f7030068960 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f704c14a450 tx=0x7f703c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:18.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:18.839+0000 7f7042ffd640 1 -- 172.21.15.89:0/1269513832 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f70480eb9c0 con 0x7f704c069e90 2023-10-30T12:37:19.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:19.019+0000 7f705470f640 1 -- 172.21.15.89:0/1269513832 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f704c071020 con 0x7f704c069e90 2023-10-30T12:37:19.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:19.019+0000 7f7042ffd640 1 -- 172.21.15.89:0/1269513832 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6527 (secure 0 0 0) 0x7f70480eefe0 con 0x7f704c069e90 2023-10-30T12:37:19.019 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:37:19.019 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5018,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:37:11.414400+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:37:19.020 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 43 2023-10-30T12:37:19.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:19.019+0000 7f705470f640 1 -- 172.21.15.89:0/1269513832 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f70300664a0 msgr2=0x7f7030068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:19.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:19.019+0000 7f705470f640 1 --2- 172.21.15.89:0/1269513832 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f70300664a0 0x7f7030068960 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f704c14a450 tx=0x7f703c009040 comp rx=0 tx=0).stop 2023-10-30T12:37:19.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:19.019+0000 7f705470f640 1 -- 172.21.15.89:0/1269513832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f704c069e90 msgr2=0x7f704c0c4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:19.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:19.019+0000 7f705470f640 1 --2- 172.21.15.89:0/1269513832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f704c069e90 0x7f704c0c4940 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f70480af040 tx=0x7f704801ac80 comp rx=0 tx=0).stop 2023-10-30T12:37:19.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:19.019+0000 7f705470f640 1 -- 172.21.15.89:0/1269513832 shutdown_connections 2023-10-30T12:37:19.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:19.019+0000 7f705470f640 1 --2- 172.21.15.89:0/1269513832 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f70300664a0 0x7f7030068960 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:19.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:19.019+0000 7f705470f640 1 --2- 172.21.15.89:0/1269513832 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f704c1539a0 0x7f704c0c2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:19.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:19.019+0000 7f705470f640 1 --2- 172.21.15.89:0/1269513832 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f704c070a00 0x7f704c0c2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:19.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:19.019+0000 7f705470f640 1 --2- 172.21.15.89:0/1269513832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f704c069e90 0x7f704c0c4940 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:19.023 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:19.019+0000 7f705470f640 1 -- 172.21.15.89:0/1269513832 >> 172.21.15.89:0/1269513832 conn(0x7f704c0c7910 msgr2=0x7f704c146670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:19.023 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:19.019+0000 7f705470f640 1 -- 172.21.15.89:0/1269513832 shutdown_connections 2023-10-30T12:37:19.023 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:19.019+0000 7f705470f640 1 -- 172.21.15.89:0/1269513832 wait complete. 2023-10-30T12:37:19.040 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:37:21.041 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:37:21.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.195+0000 7ff80ad65640 1 Processor -- start 2023-10-30T12:37:21.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.195+0000 7ff80ad65640 1 -- start start 2023-10-30T12:37:21.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.199+0000 7ff80ad65640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff804153bf0 0x7ff804071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:21.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.199+0000 7ff80ad65640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff8040c6fe0 0x7ff8040c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:21.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.199+0000 7ff80ad65640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8040700b0 con 0x7ff8040c79e0 2023-10-30T12:37:21.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.199+0000 7ff80ad65640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8040c6810 con 0x7ff8040c6fe0 2023-10-30T12:37:21.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.199+0000 7ff80ad65640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff804159900 con 0x7ff804153bf0 2023-10-30T12:37:21.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.199+0000 7ff7fbfff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff8040c6fe0 0x7ff8040c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:21.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.199+0000 7ff809302640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ff8040c79e0 0x7ff8040cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42754/0 (socket says 172.21.15.89:42754) 2023-10-30T12:37:21.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.199+0000 7ff809302640 1 -- 172.21.15.89:0/96469108 learned_addr learned my addr 172.21.15.89:0/96469108 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:37:21.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.199+0000 7ff808b01640 1 --2- 172.21.15.89:0/96469108 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff804153bf0 0x7ff804071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:21.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.199+0000 7ff7fbfff640 1 -- 172.21.15.89:0/96469108 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff804153bf0 msgr2=0x7ff804071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:21.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.199+0000 7ff7fbfff640 1 --2- 172.21.15.89:0/96469108 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff804153bf0 0x7ff804071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:21.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.199+0000 7ff7fbfff640 1 -- 172.21.15.89:0/96469108 >> v1:172.21.15.89:6789/0 conn(0x7ff8040c79e0 legacy=0x7ff8040cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:21.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.199+0000 7ff7fbfff640 1 -- 172.21.15.89:0/96469108 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8040c6530 con 0x7ff8040c6fe0 2023-10-30T12:37:21.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.199+0000 7ff7fb7fe640 1 -- 172.21.15.89:0/96469108 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2026547687 0 0) 0x7ff8040700b0 con 0x7ff8040c79e0 2023-10-30T12:37:21.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.199+0000 7ff7fbfff640 1 --2- 172.21.15.89:0/96469108 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8040c6fe0 0x7ff8040c74a0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff7f40097c0 tx=0x7ff7f401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=347f8aa9f0d3cc4a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:21.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.199+0000 7ff7fb7fe640 1 -- 172.21.15.89:0/96469108 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f40a8970 con 0x7ff8040c6fe0 2023-10-30T12:37:21.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.203+0000 7ff7fb7fe640 1 -- 172.21.15.89:0/96469108 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7f40a8ad0 con 0x7ff8040c6fe0 2023-10-30T12:37:21.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.203+0000 7ff7fb7fe640 1 -- 172.21.15.89:0/96469108 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f40a8e60 con 0x7ff8040c6fe0 2023-10-30T12:37:21.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.203+0000 7ff80ad65640 1 -- 172.21.15.89:0/96469108 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8040c6fe0 msgr2=0x7ff8040c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:21.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.203+0000 7ff80ad65640 1 --2- 172.21.15.89:0/96469108 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8040c6fe0 0x7ff8040c74a0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff7f40097c0 tx=0x7ff7f401ec70 comp rx=0 tx=0).stop 2023-10-30T12:37:21.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.203+0000 7ff80ad65640 1 -- 172.21.15.89:0/96469108 shutdown_connections 2023-10-30T12:37:21.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.203+0000 7ff80ad65640 1 --2- 172.21.15.89:0/96469108 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8040c6fe0 0x7ff8040c74a0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:21.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.203+0000 7ff80ad65640 1 --2- 172.21.15.89:0/96469108 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff804153bf0 0x7ff804071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:21.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.203+0000 7ff80ad65640 1 -- 172.21.15.89:0/96469108 >> 172.21.15.89:0/96469108 conn(0x7ff80406c6e0 msgr2=0x7ff804069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:21.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.203+0000 7ff80ad65640 1 -- 172.21.15.89:0/96469108 shutdown_connections 2023-10-30T12:37:21.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.203+0000 7ff80ad65640 1 -- 172.21.15.89:0/96469108 wait complete. 2023-10-30T12:37:21.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.203+0000 7ff80ad65640 1 Processor -- start 2023-10-30T12:37:21.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.203+0000 7ff80ad65640 1 -- start start 2023-10-30T12:37:21.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.207+0000 7ff80ad65640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8040c6fe0 0x7ff804151420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:21.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.207+0000 7ff80ad65640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8040c79e0 0x7ff8041f2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:21.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.207+0000 7ff80ad65640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff804153bf0 0x7ff8041492a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:21.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.207+0000 7ff80ad65640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80414bd60 con 0x7ff8040c79e0 2023-10-30T12:37:21.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.207+0000 7ff80ad65640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80414ba50 con 0x7ff804153bf0 2023-10-30T12:37:21.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.207+0000 7ff80ad65640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8041f3090 con 0x7ff8040c6fe0 2023-10-30T12:37:21.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.207+0000 7ff7fbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8040c79e0 0x7ff8041f2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:21.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.207+0000 7ff7fbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8040c79e0 0x7ff8041f2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36108/0 (socket says 172.21.15.89:36108) 2023-10-30T12:37:21.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.207+0000 7ff7fbfff640 1 -- 172.21.15.89:0/3456027623 learned_addr learned my addr 172.21.15.89:0/3456027623 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:37:21.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.207+0000 7ff808b01640 1 --2- 172.21.15.89:0/3456027623 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8040c6fe0 0x7ff804151420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:21.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.207+0000 7ff809302640 1 --2- 172.21.15.89:0/3456027623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff804153bf0 0x7ff8041492a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:21.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.207+0000 7ff7fbfff640 1 -- 172.21.15.89:0/3456027623 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8040c6fe0 msgr2=0x7ff804151420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:21.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.207+0000 7ff7fbfff640 1 --2- 172.21.15.89:0/3456027623 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8040c6fe0 0x7ff804151420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:21.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.207+0000 7ff7fbfff640 1 -- 172.21.15.89:0/3456027623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff804153bf0 msgr2=0x7ff8041492a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:21.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.207+0000 7ff7fbfff640 1 --2- 172.21.15.89:0/3456027623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff804153bf0 0x7ff8041492a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:21.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.207+0000 7ff7fbfff640 1 -- 172.21.15.89:0/3456027623 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f4009420 con 0x7ff8040c79e0 2023-10-30T12:37:21.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.207+0000 7ff808b01640 1 --2- 172.21.15.89:0/3456027623 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8040c6fe0 0x7ff804151420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:37:21.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.207+0000 7ff809302640 1 --2- 172.21.15.89:0/3456027623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff804153bf0 0x7ff8041492a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:37:21.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.207+0000 7ff7fbfff640 1 --2- 172.21.15.89:0/3456027623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8040c79e0 0x7ff8041f2450 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff7f40097c0 tx=0x7ff7f401d4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:21.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.207+0000 7ff7f97fa640 1 -- 172.21.15.89:0/3456027623 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f40a8970 con 0x7ff8040c79e0 2023-10-30T12:37:21.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.207+0000 7ff7f97fa640 1 -- 172.21.15.89:0/3456027623 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7f4006e20 con 0x7ff8040c79e0 2023-10-30T12:37:21.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.207+0000 7ff80ad65640 1 -- 172.21.15.89:0/3456027623 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8041f2db0 con 0x7ff8040c79e0 2023-10-30T12:37:21.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.207+0000 7ff7f97fa640 1 -- 172.21.15.89:0/3456027623 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4004230 con 0x7ff8040c79e0 2023-10-30T12:37:21.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.207+0000 7ff80ad65640 1 -- 172.21.15.89:0/3456027623 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff80414cf50 con 0x7ff8040c79e0 2023-10-30T12:37:21.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.207+0000 7ff80ad65640 1 -- 172.21.15.89:0/3456027623 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8040cc430 con 0x7ff8040c79e0 2023-10-30T12:37:21.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.207+0000 7ff7f97fa640 1 -- 172.21.15.89:0/3456027623 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff7f4004390 con 0x7ff8040c79e0 2023-10-30T12:37:21.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.211+0000 7ff7f97fa640 1 --2- 172.21.15.89:0/3456027623 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff7f00664a0 0x7ff7f0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:21.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.211+0000 7ff7f97fa640 1 -- 172.21.15.89:0/3456027623 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6090+0+0 (secure 0 0 0) 0x7ff7f4121d90 con 0x7ff8040c79e0 2023-10-30T12:37:21.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.211+0000 7ff808b01640 1 --2- 172.21.15.89:0/3456027623 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff7f00664a0 0x7ff7f0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:21.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.211+0000 7ff808b01640 1 --2- 172.21.15.89:0/3456027623 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff7f00664a0 0x7ff7f0068960 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff80000b9f0 tx=0x7ff800008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:21.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.215+0000 7ff7f97fa640 1 -- 172.21.15.89:0/3456027623 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff7f40eef60 con 0x7ff8040c79e0 2023-10-30T12:37:21.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.395+0000 7ff80ad65640 1 -- 172.21.15.89:0/3456027623 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff80404e170 con 0x7ff8040c79e0 2023-10-30T12:37:21.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.395+0000 7ff7f97fa640 1 -- 172.21.15.89:0/3456027623 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6527 (secure 0 0 0) 0x7ff7f40b5090 con 0x7ff8040c79e0 2023-10-30T12:37:21.398 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:37:21.398 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5018,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:37:11.414400+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:37:21.398 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 43 2023-10-30T12:37:21.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.399+0000 7ff80ad65640 1 -- 172.21.15.89:0/3456027623 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff7f00664a0 msgr2=0x7ff7f0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:21.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.399+0000 7ff80ad65640 1 --2- 172.21.15.89:0/3456027623 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff7f00664a0 0x7ff7f0068960 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff80000b9f0 tx=0x7ff800008040 comp rx=0 tx=0).stop 2023-10-30T12:37:21.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.399+0000 7ff80ad65640 1 -- 172.21.15.89:0/3456027623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8040c79e0 msgr2=0x7ff8041f2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:21.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.399+0000 7ff80ad65640 1 --2- 172.21.15.89:0/3456027623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8040c79e0 0x7ff8041f2450 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff7f40097c0 tx=0x7ff7f401d4b0 comp rx=0 tx=0).stop 2023-10-30T12:37:21.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.399+0000 7ff80ad65640 1 -- 172.21.15.89:0/3456027623 shutdown_connections 2023-10-30T12:37:21.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.399+0000 7ff80ad65640 1 --2- 172.21.15.89:0/3456027623 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff7f00664a0 0x7ff7f0068960 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:21.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.399+0000 7ff80ad65640 1 --2- 172.21.15.89:0/3456027623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff804153bf0 0x7ff8041492a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:21.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.399+0000 7ff80ad65640 1 --2- 172.21.15.89:0/3456027623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8040c79e0 0x7ff8041f2450 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:21.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.399+0000 7ff80ad65640 1 --2- 172.21.15.89:0/3456027623 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8040c6fe0 0x7ff804151420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:21.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.399+0000 7ff80ad65640 1 -- 172.21.15.89:0/3456027623 >> 172.21.15.89:0/3456027623 conn(0x7ff80406c6e0 msgr2=0x7ff8040c7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:21.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.399+0000 7ff80ad65640 1 -- 172.21.15.89:0/3456027623 shutdown_connections 2023-10-30T12:37:21.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:21.399+0000 7ff80ad65640 1 -- 172.21.15.89:0/3456027623 wait complete. 2023-10-30T12:37:21.418 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:37:23.419 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:37:23.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.567+0000 7fa430054640 1 Processor -- start 2023-10-30T12:37:23.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.567+0000 7fa430054640 1 -- start start 2023-10-30T12:37:23.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.567+0000 7fa430054640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4280c05d0 0x7fa42813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:23.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.567+0000 7fa430054640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4280c0a50 con 0x7fa4280c6b80 2023-10-30T12:37:23.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.567+0000 7fa430054640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa42806f6e0 con 0x7fa4280c6f60 2023-10-30T12:37:23.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.567+0000 7fa430054640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa42806f9c0 con 0x7fa4280c05d0 2023-10-30T12:37:23.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.567+0000 7fa42ddf0640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa4280c6b80 0x7fa4280bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42758/0 (socket says 172.21.15.89:42758) 2023-10-30T12:37:23.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.567+0000 7fa42ddf0640 1 -- 172.21.15.89:0/2487921644 learned_addr learned my addr 172.21.15.89:0/2487921644 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:37:23.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.567+0000 7fa42d5ef640 1 --2- 172.21.15.89:0/2487921644 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4280c05d0 0x7fa42813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:23.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.567+0000 7fa42cdee640 1 -- 172.21.15.89:0/2487921644 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2487883566 0 0) 0x7fa4280c0a50 con 0x7fa4280c6b80 2023-10-30T12:37:23.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.571+0000 7fa42cdee640 1 -- 172.21.15.89:0/2487921644 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa414003530 con 0x7fa4280c6b80 2023-10-30T12:37:23.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.571+0000 7fa42cdee640 1 -- 172.21.15.89:0/2487921644 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1867997315 0 0) 0x7fa414003530 con 0x7fa4280c6b80 2023-10-30T12:37:23.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.571+0000 7fa42cdee640 1 -- 172.21.15.89:0/2487921644 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4280c05d0 msgr2=0x7fa42813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:23.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.571+0000 7fa42cdee640 1 --2- 172.21.15.89:0/2487921644 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4280c05d0 0x7fa42813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:23.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.571+0000 7fa42cdee640 1 -- 172.21.15.89:0/2487921644 >> v1:172.21.15.192:6789/0 conn(0x7fa4280c6f60 legacy=0x7fa4281566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:23.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.571+0000 7fa42cdee640 1 -- 172.21.15.89:0/2487921644 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa42806fca0 con 0x7fa4280c6b80 2023-10-30T12:37:23.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.571+0000 7fa42cdee640 1 -- 172.21.15.89:0/2487921644 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7fa424001680 con 0x7fa4280c6b80 2023-10-30T12:37:23.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.571+0000 7fa42cdee640 1 -- 172.21.15.89:0/2487921644 >> v1:172.21.15.89:6789/0 conn(0x7fa4280c6b80 legacy=0x7fa4280bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:23.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.571+0000 7fa42cdee640 1 --2- 172.21.15.89:0/2487921644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4140039b0 0x7fa414003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:23.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.571+0000 7fa42cdee640 1 --2- 172.21.15.89:0/2487921644 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4280c05d0 0x7fa4140047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:23.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.571+0000 7fa42cdee640 1 --2- 172.21.15.89:0/2487921644 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa414004d10 0x7fa414005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:23.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.571+0000 7fa42cdee640 1 -- 172.21.15.89:0/2487921644 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42806f9c0 con 0x7fa4140039b0 2023-10-30T12:37:23.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.571+0000 7fa42cdee640 1 -- 172.21.15.89:0/2487921644 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41400b9c0 con 0x7fa414004d10 2023-10-30T12:37:23.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.571+0000 7fa42cdee640 1 -- 172.21.15.89:0/2487921644 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa41400dd90 con 0x7fa4280c05d0 2023-10-30T12:37:23.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.571+0000 7fa42ddf0640 1 --2- 172.21.15.89:0/2487921644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4140039b0 0x7fa414003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:23.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.571+0000 7fa42d5ef640 1 --2- 172.21.15.89:0/2487921644 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4280c05d0 0x7fa4140047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:23.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.571+0000 7fa42e5f1640 1 --2- 172.21.15.89:0/2487921644 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa414004d10 0x7fa414005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:23.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.571+0000 7fa42ddf0640 1 -- 172.21.15.89:0/2487921644 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4280c05d0 msgr2=0x7fa4140047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:23.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.571+0000 7fa42ddf0640 1 --2- 172.21.15.89:0/2487921644 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4280c05d0 0x7fa4140047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:23.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.571+0000 7fa42ddf0640 1 -- 172.21.15.89:0/2487921644 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa414004d10 msgr2=0x7fa414005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:23.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.571+0000 7fa42ddf0640 1 --2- 172.21.15.89:0/2487921644 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa414004d10 0x7fa414005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:23.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.571+0000 7fa42ddf0640 1 -- 172.21.15.89:0/2487921644 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa414003530 con 0x7fa4140039b0 2023-10-30T12:37:23.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.571+0000 7fa42d5ef640 1 --2- 172.21.15.89:0/2487921644 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa4280c05d0 0x7fa4140047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:37:23.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.571+0000 7fa42ddf0640 1 --2- 172.21.15.89:0/2487921644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4140039b0 0x7fa414003d90 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa42400c190 tx=0x7fa42401cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f90e7b7b6763b5ce server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:23.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.575+0000 7fa42cdee640 1 -- 172.21.15.89:0/2487921644 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa424005ae0 con 0x7fa4140039b0 2023-10-30T12:37:23.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.575+0000 7fa42cdee640 1 -- 172.21.15.89:0/2487921644 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa424005c60 con 0x7fa4140039b0 2023-10-30T12:37:23.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.575+0000 7fa42cdee640 1 -- 172.21.15.89:0/2487921644 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa424007070 con 0x7fa4140039b0 2023-10-30T12:37:23.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.575+0000 7fa430054640 1 -- 172.21.15.89:0/2487921644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4140039b0 msgr2=0x7fa414003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:23.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.575+0000 7fa430054640 1 --2- 172.21.15.89:0/2487921644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4140039b0 0x7fa414003d90 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa42400c190 tx=0x7fa42401cb20 comp rx=0 tx=0).stop 2023-10-30T12:37:23.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.575+0000 7fa42e5f1640 1 -- 172.21.15.89:0/2487921644 reap_dead start 2023-10-30T12:37:23.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.575+0000 7fa430054640 1 -- 172.21.15.89:0/2487921644 shutdown_connections 2023-10-30T12:37:23.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.575+0000 7fa430054640 1 -- 172.21.15.89:0/2487921644 >> 172.21.15.89:0/2487921644 conn(0x7fa4280c7910 msgr2=0x7fa4280c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:23.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.575+0000 7fa430054640 1 -- 172.21.15.89:0/2487921644 shutdown_connections 2023-10-30T12:37:23.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.575+0000 7fa430054640 1 -- 172.21.15.89:0/2487921644 wait complete. 2023-10-30T12:37:23.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.575+0000 7fa430054640 1 Processor -- start 2023-10-30T12:37:23.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.575+0000 7fa430054640 1 -- start start 2023-10-30T12:37:23.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.575+0000 7fa430054640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4140039b0 0x7fa4281f1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:23.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.575+0000 7fa430054640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa428154fc0 0x7fa4281e4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:23.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.575+0000 7fa430054640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4280c9820 0x7fa4280c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:23.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.575+0000 7fa430054640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4281595b0 con 0x7fa4140039b0 2023-10-30T12:37:23.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.575+0000 7fa430054640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4280c8e10 con 0x7fa4280c9820 2023-10-30T12:37:23.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.575+0000 7fa430054640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4281f20b0 con 0x7fa428154fc0 2023-10-30T12:37:23.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.575+0000 7fa42ddf0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4140039b0 0x7fa4281f1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:23.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.575+0000 7fa42ddf0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4140039b0 0x7fa4281f1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36134/0 (socket says 172.21.15.89:36134) 2023-10-30T12:37:23.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.575+0000 7fa42ddf0640 1 -- 172.21.15.89:0/2710480157 learned_addr learned my addr 172.21.15.89:0/2710480157 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:37:23.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.575+0000 7fa42e5f1640 1 --2- 172.21.15.89:0/2710480157 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4280c9820 0x7fa4280c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:23.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.575+0000 7fa42d5ef640 1 --2- 172.21.15.89:0/2710480157 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa428154fc0 0x7fa4281e4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:23.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.579+0000 7fa42ddf0640 1 -- 172.21.15.89:0/2710480157 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa428154fc0 msgr2=0x7fa4281e4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:23.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.579+0000 7fa42ddf0640 1 --2- 172.21.15.89:0/2710480157 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa428154fc0 0x7fa4281e4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:23.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.579+0000 7fa42ddf0640 1 -- 172.21.15.89:0/2710480157 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4280c9820 msgr2=0x7fa4280c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:23.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.579+0000 7fa42ddf0640 1 --2- 172.21.15.89:0/2710480157 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4280c9820 0x7fa4280c9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:23.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.579+0000 7fa42ddf0640 1 -- 172.21.15.89:0/2710480157 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4280c9390 con 0x7fa4140039b0 2023-10-30T12:37:23.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.579+0000 7fa42e5f1640 1 --2- 172.21.15.89:0/2710480157 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4280c9820 0x7fa4280c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:37:23.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.579+0000 7fa42d5ef640 1 --2- 172.21.15.89:0/2710480157 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa428154fc0 0x7fa4281e4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:37:23.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.579+0000 7fa42ddf0640 1 --2- 172.21.15.89:0/2710480157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4140039b0 0x7fa4281f1b70 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa42401c950 tx=0x7fa414005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:23.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.579+0000 7fa41e7fc640 1 -- 172.21.15.89:0/2710480157 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4240b4070 con 0x7fa4140039b0 2023-10-30T12:37:23.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.579+0000 7fa41e7fc640 1 -- 172.21.15.89:0/2710480157 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa424001500 con 0x7fa4140039b0 2023-10-30T12:37:23.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.579+0000 7fa41e7fc640 1 -- 172.21.15.89:0/2710480157 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa424007290 con 0x7fa4140039b0 2023-10-30T12:37:23.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.579+0000 7fa430054640 1 -- 172.21.15.89:0/2710480157 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4280ca1f0 con 0x7fa4140039b0 2023-10-30T12:37:23.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.579+0000 7fa430054640 1 -- 172.21.15.89:0/2710480157 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa428155730 con 0x7fa4140039b0 2023-10-30T12:37:23.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.579+0000 7fa430054640 1 -- 172.21.15.89:0/2710480157 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa42804e170 con 0x7fa4140039b0 2023-10-30T12:37:23.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.579+0000 7fa41e7fc640 1 -- 172.21.15.89:0/2710480157 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa4240073f0 con 0x7fa4140039b0 2023-10-30T12:37:23.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.579+0000 7fa41e7fc640 1 --2- 172.21.15.89:0/2710480157 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa40c0664a0 0x7fa40c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:23.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.579+0000 7fa41e7fc640 1 -- 172.21.15.89:0/2710480157 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6090+0+0 (secure 0 0 0) 0x7fa42411d670 con 0x7fa4140039b0 2023-10-30T12:37:23.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.583+0000 7fa42d5ef640 1 --2- 172.21.15.89:0/2710480157 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa40c0664a0 0x7fa40c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:23.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.583+0000 7fa42d5ef640 1 --2- 172.21.15.89:0/2710480157 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa40c0664a0 0x7fa40c068960 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa4281553a0 tx=0x7fa418008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:23.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.583+0000 7fa41e7fc640 1 -- 172.21.15.89:0/2710480157 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa4240ea840 con 0x7fa4140039b0 2023-10-30T12:37:23.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.763+0000 7fa430054640 1 -- 172.21.15.89:0/2710480157 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4281345e0 con 0x7fa4140039b0 2023-10-30T12:37:23.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.763+0000 7fa41e7fc640 1 -- 172.21.15.89:0/2710480157 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6527 (secure 0 0 0) 0x7fa4240be090 con 0x7fa4140039b0 2023-10-30T12:37:23.765 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:37:23.766 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5018,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:37:11.414400+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:37:23.766 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 43 2023-10-30T12:37:23.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.767+0000 7fa430054640 1 -- 172.21.15.89:0/2710480157 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa40c0664a0 msgr2=0x7fa40c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:23.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.767+0000 7fa430054640 1 --2- 172.21.15.89:0/2710480157 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa40c0664a0 0x7fa40c068960 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa4281553a0 tx=0x7fa418008040 comp rx=0 tx=0).stop 2023-10-30T12:37:23.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.767+0000 7fa430054640 1 -- 172.21.15.89:0/2710480157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4140039b0 msgr2=0x7fa4281f1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:23.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.767+0000 7fa430054640 1 --2- 172.21.15.89:0/2710480157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4140039b0 0x7fa4281f1b70 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa42401c950 tx=0x7fa414005800 comp rx=0 tx=0).stop 2023-10-30T12:37:23.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.767+0000 7fa430054640 1 -- 172.21.15.89:0/2710480157 shutdown_connections 2023-10-30T12:37:23.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.767+0000 7fa430054640 1 --2- 172.21.15.89:0/2710480157 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa40c0664a0 0x7fa40c068960 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:23.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.767+0000 7fa430054640 1 --2- 172.21.15.89:0/2710480157 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4280c9820 0x7fa4280c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:23.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.767+0000 7fa430054640 1 --2- 172.21.15.89:0/2710480157 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa428154fc0 0x7fa4281e4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:23.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.767+0000 7fa430054640 1 --2- 172.21.15.89:0/2710480157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4140039b0 0x7fa4281f1b70 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:23.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.767+0000 7fa430054640 1 -- 172.21.15.89:0/2710480157 >> 172.21.15.89:0/2710480157 conn(0x7fa4280c7910 msgr2=0x7fa4280bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:23.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.767+0000 7fa430054640 1 -- 172.21.15.89:0/2710480157 shutdown_connections 2023-10-30T12:37:23.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:23.767+0000 7fa430054640 1 -- 172.21.15.89:0/2710480157 wait complete. 2023-10-30T12:37:23.786 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:37:25.787 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:37:25.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.935+0000 7fa58c49a640 1 Processor -- start 2023-10-30T12:37:25.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.935+0000 7fa58c49a640 1 -- start start 2023-10-30T12:37:25.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.935+0000 7fa58c49a640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5841538a0 0x7fa58415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:25.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.935+0000 7fa58c49a640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa5840ca0e0 0x7fa58415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:25.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.935+0000 7fa58c49a640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5840cb800 con 0x7fa5840cf160 2023-10-30T12:37:25.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.935+0000 7fa58c49a640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa58415dc30 con 0x7fa5840ca0e0 2023-10-30T12:37:25.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.935+0000 7fa58c49a640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5840cb330 con 0x7fa5841538a0 2023-10-30T12:37:25.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.935+0000 7fa58aa37640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa5840cf160 0x7fa5840cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42768/0 (socket says 172.21.15.89:42768) 2023-10-30T12:37:25.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.935+0000 7fa58aa37640 1 -- 172.21.15.89:0/617202196 learned_addr learned my addr 172.21.15.89:0/617202196 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:37:25.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.935+0000 7fa589a35640 1 --2- 172.21.15.89:0/617202196 >> v2:172.21.15.192:3300/0 conn(0x7fa5840ca0e0 0x7fa58415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:25.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.935+0000 7fa58a236640 1 --2- 172.21.15.89:0/617202196 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5841538a0 0x7fa58415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:25.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.935+0000 7fa589234640 1 -- 172.21.15.89:0/617202196 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 972216224 0 0) 0x7fa5840cb800 con 0x7fa5840cf160 2023-10-30T12:37:25.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.935+0000 7fa589234640 1 -- 172.21.15.89:0/617202196 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa570003530 con 0x7fa5840cf160 2023-10-30T12:37:25.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.935+0000 7fa589234640 1 -- 172.21.15.89:0/617202196 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1762516372 0 0) 0x7fa570003530 con 0x7fa5840cf160 2023-10-30T12:37:25.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.935+0000 7fa589234640 1 -- 172.21.15.89:0/617202196 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5841538a0 msgr2=0x7fa58415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:25.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.935+0000 7fa589234640 1 --2- 172.21.15.89:0/617202196 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5841538a0 0x7fa58415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:25.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.935+0000 7fa589234640 1 -- 172.21.15.89:0/617202196 >> v2:172.21.15.192:3300/0 conn(0x7fa5840ca0e0 msgr2=0x7fa58415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:25.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.935+0000 7fa589234640 1 --2- 172.21.15.89:0/617202196 >> v2:172.21.15.192:3300/0 conn(0x7fa5840ca0e0 0x7fa58415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:25.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.935+0000 7fa589234640 1 -- 172.21.15.89:0/617202196 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5840cad70 con 0x7fa5840cf160 2023-10-30T12:37:25.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.935+0000 7fa589234640 1 -- 172.21.15.89:0/617202196 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7fa57c001a00 con 0x7fa5840cf160 2023-10-30T12:37:25.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.939+0000 7fa589234640 1 -- 172.21.15.89:0/617202196 >> v1:172.21.15.89:6789/0 conn(0x7fa5840cf160 legacy=0x7fa5840cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:25.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.939+0000 7fa589234640 1 --2- 172.21.15.89:0/617202196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa570003980 0x7fa570003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:25.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.939+0000 7fa589234640 1 --2- 172.21.15.89:0/617202196 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5841538a0 0x7fa5700047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:25.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.939+0000 7fa589234640 1 --2- 172.21.15.89:0/617202196 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa570004d10 0x7fa570005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:25.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.939+0000 7fa589234640 1 -- 172.21.15.89:0/617202196 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58415dc30 con 0x7fa570003980 2023-10-30T12:37:25.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.939+0000 7fa589234640 1 -- 172.21.15.89:0/617202196 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5840cb330 con 0x7fa570004d10 2023-10-30T12:37:25.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.939+0000 7fa589234640 1 -- 172.21.15.89:0/617202196 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa57000b990 con 0x7fa5841538a0 2023-10-30T12:37:25.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.939+0000 7fa58a236640 1 --2- 172.21.15.89:0/617202196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa570003980 0x7fa570003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:25.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.939+0000 7fa589a35640 1 --2- 172.21.15.89:0/617202196 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5841538a0 0x7fa5700047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:25.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.939+0000 7fa58aa37640 1 --2- 172.21.15.89:0/617202196 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa570004d10 0x7fa570005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:25.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.939+0000 7fa58a236640 1 -- 172.21.15.89:0/617202196 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5841538a0 msgr2=0x7fa5700047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:25.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.939+0000 7fa58a236640 1 --2- 172.21.15.89:0/617202196 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5841538a0 0x7fa5700047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:25.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.939+0000 7fa58a236640 1 -- 172.21.15.89:0/617202196 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa570004d10 msgr2=0x7fa570005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:25.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.939+0000 7fa58a236640 1 --2- 172.21.15.89:0/617202196 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa570004d10 0x7fa570005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:25.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.939+0000 7fa58a236640 1 -- 172.21.15.89:0/617202196 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa570003530 con 0x7fa570003980 2023-10-30T12:37:25.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.939+0000 7fa58a236640 1 --2- 172.21.15.89:0/617202196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa570003980 0x7fa570003da0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fa580009ae0 tx=0x7fa58001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=184b1d27f5a24022 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:25.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.939+0000 7fa589234640 1 -- 172.21.15.89:0/617202196 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5800a5530 con 0x7fa570003980 2023-10-30T12:37:25.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.939+0000 7fa589234640 1 -- 172.21.15.89:0/617202196 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5800a5690 con 0x7fa570003980 2023-10-30T12:37:25.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.939+0000 7fa589234640 1 -- 172.21.15.89:0/617202196 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5800a5a40 con 0x7fa570003980 2023-10-30T12:37:25.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.943+0000 7fa58c49a640 1 -- 172.21.15.89:0/617202196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa570003980 msgr2=0x7fa570003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:25.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.943+0000 7fa58c49a640 1 --2- 172.21.15.89:0/617202196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa570003980 0x7fa570003da0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fa580009ae0 tx=0x7fa58001ca50 comp rx=0 tx=0).stop 2023-10-30T12:37:25.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.943+0000 7fa58aa37640 1 -- 172.21.15.89:0/617202196 reap_dead start 2023-10-30T12:37:25.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.943+0000 7fa58c49a640 1 -- 172.21.15.89:0/617202196 shutdown_connections 2023-10-30T12:37:25.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.943+0000 7fa58c49a640 1 -- 172.21.15.89:0/617202196 >> 172.21.15.89:0/617202196 conn(0x7fa58406bfe0 msgr2=0x7fa58406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:25.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.943+0000 7fa58c49a640 1 -- 172.21.15.89:0/617202196 shutdown_connections 2023-10-30T12:37:25.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.943+0000 7fa58c49a640 1 -- 172.21.15.89:0/617202196 wait complete. 2023-10-30T12:37:25.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.943+0000 7fa58c49a640 1 Processor -- start 2023-10-30T12:37:25.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.943+0000 7fa58c49a640 1 -- start start 2023-10-30T12:37:25.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.943+0000 7fa58c49a640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa570003980 0x7fa5841f3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:25.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.943+0000 7fa58c49a640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5841efa10 0x7fa584145050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:25.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.943+0000 7fa58c49a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa58415d3e0 0x7fa584158db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:25.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.943+0000 7fa58c49a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5841efe50 con 0x7fa58415d3e0 2023-10-30T12:37:25.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.943+0000 7fa58c49a640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa584146230 con 0x7fa570003980 2023-10-30T12:37:25.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.943+0000 7fa58c49a640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa58415ceb0 con 0x7fa5841efa10 2023-10-30T12:37:25.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.943+0000 7fa58aa37640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa58415d3e0 0x7fa584158db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:25.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.943+0000 7fa58aa37640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa58415d3e0 0x7fa584158db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36148/0 (socket says 172.21.15.89:36148) 2023-10-30T12:37:25.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.943+0000 7fa58aa37640 1 -- 172.21.15.89:0/3558061301 learned_addr learned my addr 172.21.15.89:0/3558061301 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:37:25.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.943+0000 7fa589a35640 1 --2- 172.21.15.89:0/3558061301 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5841efa10 0x7fa584145050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:25.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.943+0000 7fa58a236640 1 --2- 172.21.15.89:0/3558061301 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa570003980 0x7fa5841f3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:25.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.943+0000 7fa58aa37640 1 -- 172.21.15.89:0/3558061301 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5841efa10 msgr2=0x7fa584145050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:25.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.943+0000 7fa58aa37640 1 --2- 172.21.15.89:0/3558061301 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5841efa10 0x7fa584145050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:25.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.943+0000 7fa58aa37640 1 -- 172.21.15.89:0/3558061301 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa570003980 msgr2=0x7fa5841f3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:25.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.943+0000 7fa58aa37640 1 --2- 172.21.15.89:0/3558061301 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa570003980 0x7fa5841f3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:25.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.947+0000 7fa58aa37640 1 -- 172.21.15.89:0/3558061301 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5841f2bf0 con 0x7fa58415d3e0 2023-10-30T12:37:25.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.947+0000 7fa58aa37640 1 --2- 172.21.15.89:0/3558061301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa58415d3e0 0x7fa584158db0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa5840cfd10 tx=0x7fa57c011c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:25.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.947+0000 7fa57affd640 1 -- 172.21.15.89:0/3558061301 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57c00f920 con 0x7fa58415d3e0 2023-10-30T12:37:25.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.947+0000 7fa57affd640 1 -- 172.21.15.89:0/3558061301 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa57c00faa0 con 0x7fa58415d3e0 2023-10-30T12:37:25.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.947+0000 7fa57affd640 1 -- 172.21.15.89:0/3558061301 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57c00fe50 con 0x7fa58415d3e0 2023-10-30T12:37:25.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.947+0000 7fa58c49a640 1 -- 172.21.15.89:0/3558061301 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa58415d100 con 0x7fa58415d3e0 2023-10-30T12:37:25.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.947+0000 7fa58c49a640 1 -- 172.21.15.89:0/3558061301 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5841f2880 con 0x7fa58415d3e0 2023-10-30T12:37:25.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.947+0000 7fa58c49a640 1 -- 172.21.15.89:0/3558061301 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5841f2a40 con 0x7fa58415d3e0 2023-10-30T12:37:25.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.947+0000 7fa57affd640 1 -- 172.21.15.89:0/3558061301 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa57c01f460 con 0x7fa58415d3e0 2023-10-30T12:37:25.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.947+0000 7fa57affd640 1 --2- 172.21.15.89:0/3558061301 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa5680664a0 0x7fa568068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:25.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.951+0000 7fa57affd640 1 -- 172.21.15.89:0/3558061301 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6090+0+0 (secure 0 0 0) 0x7fa57c086fe0 con 0x7fa58415d3e0 2023-10-30T12:37:25.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.951+0000 7fa58a236640 1 --2- 172.21.15.89:0/3558061301 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa5680664a0 0x7fa568068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:25.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.951+0000 7fa58a236640 1 --2- 172.21.15.89:0/3558061301 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa5680664a0 0x7fa568068960 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa5800af000 tx=0x7fa5800046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:25.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:25.951+0000 7fa57affd640 1 -- 172.21.15.89:0/3558061301 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa57c0542d0 con 0x7fa58415d3e0 2023-10-30T12:37:26.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:26.135+0000 7fa58c49a640 1 -- 172.21.15.89:0/3558061301 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa58404e170 con 0x7fa58415d3e0 2023-10-30T12:37:26.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:26.135+0000 7fa57affd640 1 -- 172.21.15.89:0/3558061301 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6527 (secure 0 0 0) 0x7fa57c0578f0 con 0x7fa58415d3e0 2023-10-30T12:37:26.135 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:37:26.135 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5018,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:37:11.414400+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:37:26.136 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 43 2023-10-30T12:37:26.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:26.135+0000 7fa58c49a640 1 -- 172.21.15.89:0/3558061301 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa5680664a0 msgr2=0x7fa568068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:26.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:26.135+0000 7fa58c49a640 1 --2- 172.21.15.89:0/3558061301 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa5680664a0 0x7fa568068960 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa5800af000 tx=0x7fa5800046d0 comp rx=0 tx=0).stop 2023-10-30T12:37:26.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:26.135+0000 7fa58c49a640 1 -- 172.21.15.89:0/3558061301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa58415d3e0 msgr2=0x7fa584158db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:26.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:26.135+0000 7fa58c49a640 1 --2- 172.21.15.89:0/3558061301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa58415d3e0 0x7fa584158db0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa5840cfd10 tx=0x7fa57c011c20 comp rx=0 tx=0).stop 2023-10-30T12:37:26.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:26.135+0000 7fa58c49a640 1 -- 172.21.15.89:0/3558061301 shutdown_connections 2023-10-30T12:37:26.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:26.135+0000 7fa58c49a640 1 --2- 172.21.15.89:0/3558061301 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa5680664a0 0x7fa568068960 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:26.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:26.135+0000 7fa58c49a640 1 --2- 172.21.15.89:0/3558061301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa58415d3e0 0x7fa584158db0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:26.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:26.135+0000 7fa58c49a640 1 --2- 172.21.15.89:0/3558061301 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5841efa10 0x7fa584145050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:26.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:26.135+0000 7fa58c49a640 1 --2- 172.21.15.89:0/3558061301 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa570003980 0x7fa5841f3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:26.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:26.135+0000 7fa58c49a640 1 -- 172.21.15.89:0/3558061301 >> 172.21.15.89:0/3558061301 conn(0x7fa58406bfe0 msgr2=0x7fa584065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:26.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:26.135+0000 7fa58c49a640 1 -- 172.21.15.89:0/3558061301 shutdown_connections 2023-10-30T12:37:26.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:26.135+0000 7fa58c49a640 1 -- 172.21.15.89:0/3558061301 wait complete. 2023-10-30T12:37:26.156 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:37:28.157 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:37:28.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.307+0000 7fbf07c91640 1 Processor -- start 2023-10-30T12:37:28.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.307+0000 7fbf07c91640 1 -- start start 2023-10-30T12:37:28.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.307+0000 7fbf07c91640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf0015d6f0 0x7fbf0015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:28.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.307+0000 7fbf07c91640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf000cb800 con 0x7fbf000cf160 2023-10-30T12:37:28.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.307+0000 7fbf07c91640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf000cb330 con 0x7fbf001538a0 2023-10-30T12:37:28.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.307+0000 7fbf07c91640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf000cad70 con 0x7fbf0015d6f0 2023-10-30T12:37:28.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.307+0000 7fbf0622e640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fbf000cf160 0x7fbf00164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:42780/0 (socket says 172.21.15.89:42780) 2023-10-30T12:37:28.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.307+0000 7fbf0622e640 1 -- 172.21.15.89:0/2308206207 learned_addr learned my addr 172.21.15.89:0/2308206207 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:37:28.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.307+0000 7fbf0522c640 1 --2- 172.21.15.89:0/2308206207 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf0015d6f0 0x7fbf0015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:28.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.307+0000 7fbf04a2b640 1 -- 172.21.15.89:0/2308206207 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3824834695 0 0) 0x7fbf000cb800 con 0x7fbf000cf160 2023-10-30T12:37:28.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.307+0000 7fbf04a2b640 1 -- 172.21.15.89:0/2308206207 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbeec003530 con 0x7fbf000cf160 2023-10-30T12:37:28.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.307+0000 7fbf04a2b640 1 -- 172.21.15.89:0/2308206207 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746272201 0 0) 0x7fbf000cb330 con 0x7fbf001538a0 2023-10-30T12:37:28.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.307+0000 7fbf04a2b640 1 -- 172.21.15.89:0/2308206207 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf000cb800 con 0x7fbf001538a0 2023-10-30T12:37:28.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.307+0000 7fbf04a2b640 1 -- 172.21.15.89:0/2308206207 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2370382056 0 0) 0x7fbeec003530 con 0x7fbf000cf160 2023-10-30T12:37:28.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.307+0000 7fbf04a2b640 1 -- 172.21.15.89:0/2308206207 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf0015d6f0 msgr2=0x7fbf0015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:28.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.307+0000 7fbf04a2b640 1 --2- 172.21.15.89:0/2308206207 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf0015d6f0 0x7fbf0015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:28.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.311+0000 7fbf04a2b640 1 -- 172.21.15.89:0/2308206207 >> v1:172.21.15.192:6789/0 conn(0x7fbf001538a0 legacy=0x7fbf000ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:28.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.311+0000 7fbf04a2b640 1 -- 172.21.15.89:0/2308206207 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf000cb050 con 0x7fbf000cf160 2023-10-30T12:37:28.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.311+0000 7fbf04a2b640 1 -- 172.21.15.89:0/2308206207 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7fbef8001a00 con 0x7fbf000cf160 2023-10-30T12:37:28.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.311+0000 7fbf04a2b640 1 -- 172.21.15.89:0/2308206207 >> v1:172.21.15.89:6789/0 conn(0x7fbf000cf160 legacy=0x7fbf00164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:28.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.311+0000 7fbf04a2b640 1 --2- 172.21.15.89:0/2308206207 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf0015d6f0 0x7fbeec003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:28.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.311+0000 7fbf04a2b640 1 --2- 172.21.15.89:0/2308206207 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbeec0040a0 0x7fbeec0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:28.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.311+0000 7fbf04a2b640 1 --2- 172.21.15.89:0/2308206207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbeec004d10 0x7fbeec005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:28.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.311+0000 7fbf04a2b640 1 -- 172.21.15.89:0/2308206207 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf000cad70 con 0x7fbeec004d10 2023-10-30T12:37:28.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.311+0000 7fbf04a2b640 1 -- 172.21.15.89:0/2308206207 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec00ba70 con 0x7fbeec0040a0 2023-10-30T12:37:28.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.311+0000 7fbf04a2b640 1 -- 172.21.15.89:0/2308206207 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec00bc30 con 0x7fbf0015d6f0 2023-10-30T12:37:28.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.311+0000 7fbf0622e640 1 --2- 172.21.15.89:0/2308206207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbeec004d10 0x7fbeec005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:28.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.311+0000 7fbf0522c640 1 --2- 172.21.15.89:0/2308206207 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbeec0040a0 0x7fbeec0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:28.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.311+0000 7fbf05a2d640 1 --2- 172.21.15.89:0/2308206207 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf0015d6f0 0x7fbeec003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:28.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.311+0000 7fbf0622e640 1 -- 172.21.15.89:0/2308206207 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf0015d6f0 msgr2=0x7fbeec003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:28.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.311+0000 7fbf0622e640 1 --2- 172.21.15.89:0/2308206207 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf0015d6f0 0x7fbeec003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:28.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.311+0000 7fbf0622e640 1 -- 172.21.15.89:0/2308206207 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbeec0040a0 msgr2=0x7fbeec0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:28.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.311+0000 7fbf0622e640 1 --2- 172.21.15.89:0/2308206207 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbeec0040a0 0x7fbeec0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:28.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.311+0000 7fbf0622e640 1 -- 172.21.15.89:0/2308206207 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeec003530 con 0x7fbeec004d10 2023-10-30T12:37:28.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.311+0000 7fbf0622e640 1 --2- 172.21.15.89:0/2308206207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbeec004d10 0x7fbeec005190 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fbef800a7a0 tx=0x7fbef801d2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=734881f76d78f68a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:28.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.315+0000 7fbf04a2b640 1 -- 172.21.15.89:0/2308206207 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef80ad630 con 0x7fbeec004d10 2023-10-30T12:37:28.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.315+0000 7fbf04a2b640 1 -- 172.21.15.89:0/2308206207 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef80a8070 con 0x7fbeec004d10 2023-10-30T12:37:28.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.315+0000 7fbf04a2b640 1 -- 172.21.15.89:0/2308206207 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef80ada30 con 0x7fbeec004d10 2023-10-30T12:37:28.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.315+0000 7fbf07c91640 1 -- 172.21.15.89:0/2308206207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbeec004d10 msgr2=0x7fbeec005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:28.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.315+0000 7fbf07c91640 1 --2- 172.21.15.89:0/2308206207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbeec004d10 0x7fbeec005190 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fbef800a7a0 tx=0x7fbef801d2a0 comp rx=0 tx=0).stop 2023-10-30T12:37:28.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.315+0000 7fbf0622e640 1 -- 172.21.15.89:0/2308206207 reap_dead start 2023-10-30T12:37:28.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.315+0000 7fbf07c91640 1 -- 172.21.15.89:0/2308206207 shutdown_connections 2023-10-30T12:37:28.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.315+0000 7fbf07c91640 1 -- 172.21.15.89:0/2308206207 >> 172.21.15.89:0/2308206207 conn(0x7fbf0006bfe0 msgr2=0x7fbf0006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:28.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.315+0000 7fbf07c91640 1 -- 172.21.15.89:0/2308206207 shutdown_connections 2023-10-30T12:37:28.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.315+0000 7fbf07c91640 1 -- 172.21.15.89:0/2308206207 wait complete. 2023-10-30T12:37:28.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.315+0000 7fbf07c91640 1 Processor -- start 2023-10-30T12:37:28.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.315+0000 7fbf07c91640 1 -- start start 2023-10-30T12:37:28.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.315+0000 7fbf07c91640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbeec004d10 0x7fbf001ef280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:28.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.315+0000 7fbf07c91640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf00158f10 0x7fbf0014cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:28.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.315+0000 7fbf07c91640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf00159350 0x7fbf0014d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:28.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.315+0000 7fbf07c91640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf000b3da0 con 0x7fbf00159350 2023-10-30T12:37:28.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.315+0000 7fbf07c91640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf00158680 con 0x7fbf00158f10 2023-10-30T12:37:28.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.315+0000 7fbf07c91640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf00158100 con 0x7fbeec004d10 2023-10-30T12:37:28.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.315+0000 7fbf0622e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf00159350 0x7fbf0014d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:28.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.315+0000 7fbf0622e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf00159350 0x7fbf0014d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36170/0 (socket says 172.21.15.89:36170) 2023-10-30T12:37:28.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.315+0000 7fbf0622e640 1 -- 172.21.15.89:0/904417069 learned_addr learned my addr 172.21.15.89:0/904417069 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:37:28.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.315+0000 7fbf0522c640 1 --2- 172.21.15.89:0/904417069 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf00158f10 0x7fbf0014cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:28.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.315+0000 7fbf05a2d640 1 --2- 172.21.15.89:0/904417069 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbeec004d10 0x7fbf001ef280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:28.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.319+0000 7fbf0622e640 1 -- 172.21.15.89:0/904417069 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbeec004d10 msgr2=0x7fbf001ef280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:28.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.319+0000 7fbf0622e640 1 --2- 172.21.15.89:0/904417069 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbeec004d10 0x7fbf001ef280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:28.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.319+0000 7fbf0622e640 1 -- 172.21.15.89:0/904417069 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf00158f10 msgr2=0x7fbf0014cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:28.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.319+0000 7fbf0622e640 1 --2- 172.21.15.89:0/904417069 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf00158f10 0x7fbf0014cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:28.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.319+0000 7fbf0622e640 1 -- 172.21.15.89:0/904417069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf001583a0 con 0x7fbf00159350 2023-10-30T12:37:28.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.319+0000 7fbf0622e640 1 --2- 172.21.15.89:0/904417069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf00159350 0x7fbf0014d240 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fbef800a7a0 tx=0x7fbeec005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:28.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.319+0000 7fbef67fc640 1 -- 172.21.15.89:0/904417069 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef8001f60 con 0x7fbf00159350 2023-10-30T12:37:28.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.319+0000 7fbef67fc640 1 -- 172.21.15.89:0/904417069 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef8001800 con 0x7fbf00159350 2023-10-30T12:37:28.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.319+0000 7fbef67fc640 1 -- 172.21.15.89:0/904417069 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef8004b70 con 0x7fbf00159350 2023-10-30T12:37:28.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.319+0000 7fbf07c91640 1 -- 172.21.15.89:0/904417069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf000b3470 con 0x7fbf00159350 2023-10-30T12:37:28.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.319+0000 7fbf07c91640 1 -- 172.21.15.89:0/904417069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf000b3a30 con 0x7fbf00159350 2023-10-30T12:37:28.326 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.319+0000 7fbf07c91640 1 -- 172.21.15.89:0/904417069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf001ef7c0 con 0x7fbf00159350 2023-10-30T12:37:28.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.319+0000 7fbef67fc640 1 -- 172.21.15.89:0/904417069 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbef80aa030 con 0x7fbf00159350 2023-10-30T12:37:28.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.319+0000 7fbef67fc640 1 --2- 172.21.15.89:0/904417069 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbee40664a0 0x7fbee4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:28.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.319+0000 7fbef67fc640 1 -- 172.21.15.89:0/904417069 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6090+0+0 (secure 0 0 0) 0x7fbef811d8c0 con 0x7fbf00159350 2023-10-30T12:37:28.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.323+0000 7fbf05a2d640 1 --2- 172.21.15.89:0/904417069 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbee40664a0 0x7fbee4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:28.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.323+0000 7fbf05a2d640 1 --2- 172.21.15.89:0/904417069 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbee40664a0 0x7fbee4068960 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbefc007fd0 tx=0x7fbefc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:28.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.323+0000 7fbef67fc640 1 -- 172.21.15.89:0/904417069 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbef80eabe0 con 0x7fbf00159350 2023-10-30T12:37:28.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.507+0000 7fbf07c91640 1 -- 172.21.15.89:0/904417069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf0004e170 con 0x7fbf00159350 2023-10-30T12:37:28.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.507+0000 7fbef67fc640 1 -- 172.21.15.89:0/904417069 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6527 (secure 0 0 0) 0x7fbef80ee200 con 0x7fbf00159350 2023-10-30T12:37:28.510 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:37:28.510 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5018,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:37:11.414400+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:37:28.511 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 43 2023-10-30T12:37:28.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.511+0000 7fbf07c91640 1 -- 172.21.15.89:0/904417069 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbee40664a0 msgr2=0x7fbee4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:28.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.511+0000 7fbf07c91640 1 --2- 172.21.15.89:0/904417069 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbee40664a0 0x7fbee4068960 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbefc007fd0 tx=0x7fbefc009040 comp rx=0 tx=0).stop 2023-10-30T12:37:28.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.511+0000 7fbf07c91640 1 -- 172.21.15.89:0/904417069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf00159350 msgr2=0x7fbf0014d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:28.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.511+0000 7fbf07c91640 1 --2- 172.21.15.89:0/904417069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf00159350 0x7fbf0014d240 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fbef800a7a0 tx=0x7fbeec005800 comp rx=0 tx=0).stop 2023-10-30T12:37:28.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.511+0000 7fbf07c91640 1 -- 172.21.15.89:0/904417069 shutdown_connections 2023-10-30T12:37:28.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.511+0000 7fbf07c91640 1 --2- 172.21.15.89:0/904417069 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbee40664a0 0x7fbee4068960 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:28.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.511+0000 7fbf07c91640 1 --2- 172.21.15.89:0/904417069 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf00158f10 0x7fbf0014cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:28.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.511+0000 7fbf07c91640 1 --2- 172.21.15.89:0/904417069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf00159350 0x7fbf0014d240 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:28.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.511+0000 7fbf07c91640 1 --2- 172.21.15.89:0/904417069 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbeec004d10 0x7fbf001ef280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:28.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.511+0000 7fbf07c91640 1 -- 172.21.15.89:0/904417069 >> 172.21.15.89:0/904417069 conn(0x7fbf0006bfe0 msgr2=0x7fbf000cf540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:28.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.511+0000 7fbf07c91640 1 -- 172.21.15.89:0/904417069 shutdown_connections 2023-10-30T12:37:28.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:28.511+0000 7fbf07c91640 1 -- 172.21.15.89:0/904417069 wait complete. 2023-10-30T12:37:28.529 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:37:30.531 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:37:30.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.679+0000 7f78fe10f640 1 Processor -- start 2023-10-30T12:37:30.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.679+0000 7f78fe10f640 1 -- start start 2023-10-30T12:37:30.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.683+0000 7f78fe10f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78f8070c90 0x7f78f8165130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:30.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.683+0000 7f78fe10f640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f80c2f20 con 0x7f78f8070650 2023-10-30T12:37:30.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.683+0000 7f78fe10f640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f80c85c0 con 0x7f78f8156f80 2023-10-30T12:37:30.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.683+0000 7f78fe10f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f8157360 con 0x7f78f8070c90 2023-10-30T12:37:30.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.683+0000 7f78f6ffd640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f78f8070650 0x7f78f8071bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40150/0 (socket says 172.21.15.89:40150) 2023-10-30T12:37:30.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.683+0000 7f78f6ffd640 1 -- 172.21.15.89:0/3817045464 learned_addr learned my addr 172.21.15.89:0/3817045464 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:37:30.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.683+0000 7f78f7fff640 1 --2- 172.21.15.89:0/3817045464 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78f8070c90 0x7f78f8165130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:30.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.683+0000 7f78f67fc640 1 -- 172.21.15.89:0/3817045464 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3354844681 0 0) 0x7f78f80c2f20 con 0x7f78f8070650 2023-10-30T12:37:30.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.683+0000 7f78f67fc640 1 -- 172.21.15.89:0/3817045464 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78e0003530 con 0x7f78f8070650 2023-10-30T12:37:30.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.683+0000 7f78f67fc640 1 -- 172.21.15.89:0/3817045464 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2030381605 0 0) 0x7f78e0003530 con 0x7f78f8070650 2023-10-30T12:37:30.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.683+0000 7f78f67fc640 1 -- 172.21.15.89:0/3817045464 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78f8070c90 msgr2=0x7f78f8165130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:30.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.683+0000 7f78f67fc640 1 --2- 172.21.15.89:0/3817045464 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78f8070c90 0x7f78f8165130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:30.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.683+0000 7f78f67fc640 1 -- 172.21.15.89:0/3817045464 >> v1:172.21.15.192:6789/0 conn(0x7f78f8156f80 legacy=0x7f78f8159880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:30.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.683+0000 7f78f67fc640 1 -- 172.21.15.89:0/3817045464 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f8071230 con 0x7f78f8070650 2023-10-30T12:37:30.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.683+0000 7f78f67fc640 1 -- 172.21.15.89:0/3817045464 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f78ec001680 con 0x7f78f8070650 2023-10-30T12:37:30.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.683+0000 7f78f67fc640 1 -- 172.21.15.89:0/3817045464 >> v1:172.21.15.89:6789/0 conn(0x7f78f8070650 legacy=0x7f78f8071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:30.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.683+0000 7f78f67fc640 1 --2- 172.21.15.89:0/3817045464 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78f8070c90 0x7f78e00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:30.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.683+0000 7f78f67fc640 1 --2- 172.21.15.89:0/3817045464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78e0003f30 0x7f78e00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:30.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.683+0000 7f78f67fc640 1 --2- 172.21.15.89:0/3817045464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78e0004d10 0x7f78e0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:30.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.683+0000 7f78f67fc640 1 -- 172.21.15.89:0/3817045464 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f8157360 con 0x7f78e0004d10 2023-10-30T12:37:30.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.683+0000 7f78f67fc640 1 -- 172.21.15.89:0/3817045464 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e000ba70 con 0x7f78e0003f30 2023-10-30T12:37:30.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.683+0000 7f78f67fc640 1 -- 172.21.15.89:0/3817045464 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78e000bc30 con 0x7f78f8070c90 2023-10-30T12:37:30.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.683+0000 7f78f7fff640 1 --2- 172.21.15.89:0/3817045464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78e0004d10 0x7f78e0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:30.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.683+0000 7f78f77fe640 1 --2- 172.21.15.89:0/3817045464 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78f8070c90 0x7f78e00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:30.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.683+0000 7f78f6ffd640 1 --2- 172.21.15.89:0/3817045464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78e0003f30 0x7f78e00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:30.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.683+0000 7f78f7fff640 1 -- 172.21.15.89:0/3817045464 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78f8070c90 msgr2=0x7f78e00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:30.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.683+0000 7f78f7fff640 1 --2- 172.21.15.89:0/3817045464 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78f8070c90 0x7f78e00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:30.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.683+0000 7f78f7fff640 1 -- 172.21.15.89:0/3817045464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78e0003f30 msgr2=0x7f78e00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:30.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.683+0000 7f78f7fff640 1 --2- 172.21.15.89:0/3817045464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78e0003f30 0x7f78e00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:30.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.683+0000 7f78f7fff640 1 -- 172.21.15.89:0/3817045464 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78e0003530 con 0x7f78e0004d10 2023-10-30T12:37:30.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.687+0000 7f78f7fff640 1 --2- 172.21.15.89:0/3817045464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78e0004d10 0x7f78e0005190 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f78f0009fc0 tx=0x7f78f001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=906fe3bb01e3a9c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:30.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.687+0000 7f78f67fc640 1 -- 172.21.15.89:0/3817045464 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f00a5c70 con 0x7f78e0004d10 2023-10-30T12:37:30.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.687+0000 7f78f67fc640 1 -- 172.21.15.89:0/3817045464 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f000f040 con 0x7f78e0004d10 2023-10-30T12:37:30.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.687+0000 7f78f67fc640 1 -- 172.21.15.89:0/3817045464 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f00ad440 con 0x7f78e0004d10 2023-10-30T12:37:30.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.687+0000 7f78fe10f640 1 -- 172.21.15.89:0/3817045464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78e0004d10 msgr2=0x7f78e0005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:30.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.687+0000 7f78fe10f640 1 --2- 172.21.15.89:0/3817045464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78e0004d10 0x7f78e0005190 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f78f0009fc0 tx=0x7f78f001cc60 comp rx=0 tx=0).stop 2023-10-30T12:37:30.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.687+0000 7f78f7fff640 1 -- 172.21.15.89:0/3817045464 reap_dead start 2023-10-30T12:37:30.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.687+0000 7f78fe10f640 1 -- 172.21.15.89:0/3817045464 shutdown_connections 2023-10-30T12:37:30.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.687+0000 7f78fe10f640 1 -- 172.21.15.89:0/3817045464 >> 172.21.15.89:0/3817045464 conn(0x7f78f80cbeb0 msgr2=0x7f78f80cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:30.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.687+0000 7f78fe10f640 1 -- 172.21.15.89:0/3817045464 shutdown_connections 2023-10-30T12:37:30.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.687+0000 7f78fe10f640 1 -- 172.21.15.89:0/3817045464 wait complete. 2023-10-30T12:37:30.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.687+0000 7f78fe10f640 1 Processor -- start 2023-10-30T12:37:30.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.687+0000 7f78fe10f640 1 -- start start 2023-10-30T12:37:30.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.691+0000 7f78fe10f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78e0004d10 0x7f78f81ee6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:30.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.691+0000 7f78fe10f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78f81f2670 0x7f78f814bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:30.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.691+0000 7f78fe10f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78f814c3e0 0x7f78f80c7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:30.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.691+0000 7f78fe10f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f8071800 con 0x7f78e0004d10 2023-10-30T12:37:30.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.691+0000 7f78fe10f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f81f1c20 con 0x7f78f814c3e0 2023-10-30T12:37:30.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.691+0000 7f78fe10f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f81f2ad0 con 0x7f78f81f2670 2023-10-30T12:37:30.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.691+0000 7f78f77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78e0004d10 0x7f78f81ee6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:30.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.691+0000 7f78f77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78e0004d10 0x7f78f81ee6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35866/0 (socket says 172.21.15.89:35866) 2023-10-30T12:37:30.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.691+0000 7f78f77fe640 1 -- 172.21.15.89:0/1366634586 learned_addr learned my addr 172.21.15.89:0/1366634586 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:37:30.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.691+0000 7f78f6ffd640 1 --2- 172.21.15.89:0/1366634586 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78f81f2670 0x7f78f814bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:30.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.691+0000 7f78f7fff640 1 --2- 172.21.15.89:0/1366634586 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78f814c3e0 0x7f78f80c7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:30.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.691+0000 7f78f77fe640 1 -- 172.21.15.89:0/1366634586 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78f81f2670 msgr2=0x7f78f814bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:30.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.691+0000 7f78f77fe640 1 --2- 172.21.15.89:0/1366634586 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78f81f2670 0x7f78f814bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:30.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.691+0000 7f78f77fe640 1 -- 172.21.15.89:0/1366634586 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78f814c3e0 msgr2=0x7f78f80c7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:30.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.691+0000 7f78f77fe640 1 --2- 172.21.15.89:0/1366634586 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78f814c3e0 0x7f78f80c7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:30.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.691+0000 7f78f77fe640 1 -- 172.21.15.89:0/1366634586 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f80c5980 con 0x7f78e0004d10 2023-10-30T12:37:30.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.691+0000 7f78f77fe640 1 --2- 172.21.15.89:0/1366634586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78e0004d10 0x7f78f81ee6b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f78e800f7f0 tx=0x7f78e800fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:30.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.691+0000 7f78dffff640 1 -- 172.21.15.89:0/1366634586 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e80050d0 con 0x7f78e0004d10 2023-10-30T12:37:30.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.691+0000 7f78dffff640 1 -- 172.21.15.89:0/1366634586 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e8005230 con 0x7f78e0004d10 2023-10-30T12:37:30.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.691+0000 7f78dffff640 1 -- 172.21.15.89:0/1366634586 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e800d730 con 0x7f78e0004d10 2023-10-30T12:37:30.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.691+0000 7f78fe10f640 1 -- 172.21.15.89:0/1366634586 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78f80c5c60 con 0x7f78e0004d10 2023-10-30T12:37:30.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.691+0000 7f78fe10f640 1 -- 172.21.15.89:0/1366634586 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78f81f1e70 con 0x7f78e0004d10 2023-10-30T12:37:30.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.691+0000 7f78fe10f640 1 -- 172.21.15.89:0/1366634586 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78f804e170 con 0x7f78e0004d10 2023-10-30T12:37:30.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.695+0000 7f78dffff640 1 -- 172.21.15.89:0/1366634586 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f78e800d890 con 0x7f78e0004d10 2023-10-30T12:37:30.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.695+0000 7f78dffff640 1 --2- 172.21.15.89:0/1366634586 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f78e40664d0 0x7f78e4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:30.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.695+0000 7f78dffff640 1 -- 172.21.15.89:0/1366634586 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6090+0+0 (secure 0 0 0) 0x7f78e8083fb0 con 0x7f78e0004d10 2023-10-30T12:37:30.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.695+0000 7f78f6ffd640 1 --2- 172.21.15.89:0/1366634586 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f78e40664d0 0x7f78e4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:30.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.695+0000 7f78f6ffd640 1 --2- 172.21.15.89:0/1366634586 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f78e40664d0 0x7f78e4068990 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f78ec001060 tx=0x7f78ec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:30.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.699+0000 7f78dffff640 1 -- 172.21.15.89:0/1366634586 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f78e80b5090 con 0x7f78e0004d10 2023-10-30T12:37:30.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.887+0000 7f78fe10f640 1 -- 172.21.15.89:0/1366634586 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78f80c8220 con 0x7f78e0004d10 2023-10-30T12:37:30.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.887+0000 7f78dffff640 1 -- 172.21.15.89:0/1366634586 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6527 (secure 0 0 0) 0x7f78e80512b0 con 0x7f78e0004d10 2023-10-30T12:37:30.887 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:37:30.888 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5018,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:37:11.414400+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:37:30.888 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 43 2023-10-30T12:37:30.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.887+0000 7f78fe10f640 1 -- 172.21.15.89:0/1366634586 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f78e40664d0 msgr2=0x7f78e4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:30.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.887+0000 7f78fe10f640 1 --2- 172.21.15.89:0/1366634586 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f78e40664d0 0x7f78e4068990 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f78ec001060 tx=0x7f78ec009040 comp rx=0 tx=0).stop 2023-10-30T12:37:30.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.887+0000 7f78fe10f640 1 -- 172.21.15.89:0/1366634586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78e0004d10 msgr2=0x7f78f81ee6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:30.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.887+0000 7f78fe10f640 1 --2- 172.21.15.89:0/1366634586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78e0004d10 0x7f78f81ee6b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f78e800f7f0 tx=0x7f78e800fcb0 comp rx=0 tx=0).stop 2023-10-30T12:37:30.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.887+0000 7f78fe10f640 1 -- 172.21.15.89:0/1366634586 shutdown_connections 2023-10-30T12:37:30.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.887+0000 7f78fe10f640 1 --2- 172.21.15.89:0/1366634586 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f78e40664d0 0x7f78e4068990 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:30.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.887+0000 7f78fe10f640 1 --2- 172.21.15.89:0/1366634586 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78f814c3e0 0x7f78f80c7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:30.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.887+0000 7f78fe10f640 1 --2- 172.21.15.89:0/1366634586 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78f81f2670 0x7f78f814bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:30.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.887+0000 7f78fe10f640 1 --2- 172.21.15.89:0/1366634586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78e0004d10 0x7f78f81ee6b0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:30.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.887+0000 7f78fe10f640 1 -- 172.21.15.89:0/1366634586 >> 172.21.15.89:0/1366634586 conn(0x7f78f80cbeb0 msgr2=0x7f78f81547b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:30.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.887+0000 7f78fe10f640 1 -- 172.21.15.89:0/1366634586 shutdown_connections 2023-10-30T12:37:30.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:30.887+0000 7f78fe10f640 1 -- 172.21.15.89:0/1366634586 wait complete. 2023-10-30T12:37:30.908 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:37:32.910 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:37:33.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.059+0000 7f67b82d6640 1 Processor -- start 2023-10-30T12:37:33.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.059+0000 7f67b82d6640 1 -- start start 2023-10-30T12:37:33.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.059+0000 7f67b82d6640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f67b015d6f0 0x7f67b015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:33.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.059+0000 7f67b82d6640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67b00cb800 con 0x7f67b00cf160 2023-10-30T12:37:33.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.059+0000 7f67b82d6640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67b00cb330 con 0x7f67b01538a0 2023-10-30T12:37:33.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.059+0000 7f67b82d6640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67b00cad70 con 0x7f67b015d6f0 2023-10-30T12:37:33.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.059+0000 7f67b6873640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f67b00cf160 0x7f67b0164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40166/0 (socket says 172.21.15.89:40166) 2023-10-30T12:37:33.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.059+0000 7f67b6873640 1 -- 172.21.15.89:0/1946708638 learned_addr learned my addr 172.21.15.89:0/1946708638 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:37:33.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.059+0000 7f67b5871640 1 --2- 172.21.15.89:0/1946708638 >> v2:172.21.15.192:3300/0 conn(0x7f67b015d6f0 0x7f67b015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:33.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.059+0000 7f67b5070640 1 -- 172.21.15.89:0/1946708638 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1794616888 0 0) 0x7f67b00cb800 con 0x7f67b00cf160 2023-10-30T12:37:33.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.059+0000 7f67b5070640 1 -- 172.21.15.89:0/1946708638 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f679c003530 con 0x7f67b00cf160 2023-10-30T12:37:33.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.059+0000 7f67b5070640 1 -- 172.21.15.89:0/1946708638 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3954955729 0 0) 0x7f679c003530 con 0x7f67b00cf160 2023-10-30T12:37:33.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.059+0000 7f67b5070640 1 -- 172.21.15.89:0/1946708638 >> v2:172.21.15.192:3300/0 conn(0x7f67b015d6f0 msgr2=0x7f67b015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:33.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.059+0000 7f67b5070640 1 --2- 172.21.15.89:0/1946708638 >> v2:172.21.15.192:3300/0 conn(0x7f67b015d6f0 0x7f67b015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:33.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.059+0000 7f67b5070640 1 -- 172.21.15.89:0/1946708638 >> v1:172.21.15.192:6789/0 conn(0x7f67b01538a0 legacy=0x7f67b00ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:33.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.059+0000 7f67b5070640 1 -- 172.21.15.89:0/1946708638 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b00cb050 con 0x7f67b00cf160 2023-10-30T12:37:33.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.059+0000 7f67b5070640 1 -- 172.21.15.89:0/1946708638 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f67a8001a00 con 0x7f67b00cf160 2023-10-30T12:37:33.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.059+0000 7f67b5070640 1 -- 172.21.15.89:0/1946708638 >> v1:172.21.15.89:6789/0 conn(0x7f67b00cf160 legacy=0x7f67b0164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:33.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.059+0000 7f67b5070640 1 --2- 172.21.15.89:0/1946708638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f679c003980 0x7f679c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:33.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.059+0000 7f67b5070640 1 --2- 172.21.15.89:0/1946708638 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f679c0047d0 0x7f679c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:33.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.059+0000 7f67b5070640 1 --2- 172.21.15.89:0/1946708638 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f679c005190 0x7f679c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:33.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.059+0000 7f67b5070640 1 -- 172.21.15.89:0/1946708638 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b00cad70 con 0x7f679c003980 2023-10-30T12:37:33.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.059+0000 7f67b5070640 1 -- 172.21.15.89:0/1946708638 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c00c040 con 0x7f679c005190 2023-10-30T12:37:33.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.059+0000 7f67b5070640 1 -- 172.21.15.89:0/1946708638 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f679c00c200 con 0x7f679c0047d0 2023-10-30T12:37:33.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.063+0000 7f67b6072640 1 --2- 172.21.15.89:0/1946708638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f679c003980 0x7f679c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:33.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.063+0000 7f67b5871640 1 --2- 172.21.15.89:0/1946708638 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f679c0047d0 0x7f679c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:33.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.063+0000 7f67b6873640 1 --2- 172.21.15.89:0/1946708638 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f679c005190 0x7f679c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:33.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.063+0000 7f67b6072640 1 -- 172.21.15.89:0/1946708638 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f679c0047d0 msgr2=0x7f679c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:33.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.063+0000 7f67b6072640 1 --2- 172.21.15.89:0/1946708638 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f679c0047d0 0x7f679c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:33.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.063+0000 7f67b6072640 1 -- 172.21.15.89:0/1946708638 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f679c005190 msgr2=0x7f679c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:33.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.063+0000 7f67b6072640 1 --2- 172.21.15.89:0/1946708638 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f679c005190 0x7f679c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:33.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.063+0000 7f67b6072640 1 -- 172.21.15.89:0/1946708638 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f679c003530 con 0x7f679c003980 2023-10-30T12:37:33.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.063+0000 7f67b6873640 1 --2- 172.21.15.89:0/1946708638 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f679c005190 0x7f679c00b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:37:33.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.063+0000 7f67b6873640 1 -- 172.21.15.89:0/1946708638 reap_dead start 2023-10-30T12:37:33.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.063+0000 7f67b6072640 1 --2- 172.21.15.89:0/1946708638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f679c003980 0x7f679c003da0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f67ac001780 tx=0x7f67ac01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c24152f2145727b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:33.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.063+0000 7f67b5070640 1 -- 172.21.15.89:0/1946708638 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ac0052d0 con 0x7f679c003980 2023-10-30T12:37:33.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.063+0000 7f67b5070640 1 -- 172.21.15.89:0/1946708638 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67ac0a7050 con 0x7f679c003980 2023-10-30T12:37:33.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.063+0000 7f67b5070640 1 -- 172.21.15.89:0/1946708638 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ac003600 con 0x7f679c003980 2023-10-30T12:37:33.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.063+0000 7f67b82d6640 1 -- 172.21.15.89:0/1946708638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f679c003980 msgr2=0x7f679c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:33.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.063+0000 7f67b82d6640 1 --2- 172.21.15.89:0/1946708638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f679c003980 0x7f679c003da0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f67ac001780 tx=0x7f67ac01c3e0 comp rx=0 tx=0).stop 2023-10-30T12:37:33.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.067+0000 7f67b82d6640 1 -- 172.21.15.89:0/1946708638 shutdown_connections 2023-10-30T12:37:33.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.067+0000 7f67b82d6640 1 --2- 172.21.15.89:0/1946708638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f679c003980 0x7f679c003da0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:33.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.067+0000 7f67b82d6640 1 -- 172.21.15.89:0/1946708638 >> 172.21.15.89:0/1946708638 conn(0x7f67b006bfe0 msgr2=0x7f67b006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:33.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.067+0000 7f67b82d6640 1 -- 172.21.15.89:0/1946708638 shutdown_connections 2023-10-30T12:37:33.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.067+0000 7f67b82d6640 1 -- 172.21.15.89:0/1946708638 wait complete. 2023-10-30T12:37:33.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.067+0000 7f67b82d6640 1 Processor -- start 2023-10-30T12:37:33.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.067+0000 7f67b82d6640 1 -- start start 2023-10-30T12:37:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.067+0000 7f67b82d6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f679c003980 0x7f67b00b4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.067+0000 7f67b82d6640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f67b01e2aa0 0x7f67b00b3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.067+0000 7f67b82d6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67b0146bd0 0x7f67b0145460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.067+0000 7f67b82d6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b00b34e0 con 0x7f67b0146bd0 2023-10-30T12:37:33.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.067+0000 7f67b82d6640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b00b50e0 con 0x7f679c003980 2023-10-30T12:37:33.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.067+0000 7f67b82d6640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67b015a920 con 0x7f67b01e2aa0 2023-10-30T12:37:33.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.067+0000 7f67b6873640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67b0146bd0 0x7f67b0145460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:33.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.067+0000 7f67b6873640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67b0146bd0 0x7f67b0145460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35896/0 (socket says 172.21.15.89:35896) 2023-10-30T12:37:33.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.067+0000 7f67b6873640 1 -- 172.21.15.89:0/2843667267 learned_addr learned my addr 172.21.15.89:0/2843667267 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:37:33.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.067+0000 7f67b6072640 1 --2- 172.21.15.89:0/2843667267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f679c003980 0x7f67b00b4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:33.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.067+0000 7f67b5871640 1 --2- 172.21.15.89:0/2843667267 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f67b01e2aa0 0x7f67b00b3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:33.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.067+0000 7f67b6873640 1 -- 172.21.15.89:0/2843667267 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f67b01e2aa0 msgr2=0x7f67b00b3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:33.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.067+0000 7f67b6873640 1 --2- 172.21.15.89:0/2843667267 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f67b01e2aa0 0x7f67b00b3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:33.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.067+0000 7f67b6873640 1 -- 172.21.15.89:0/2843667267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f679c003980 msgr2=0x7f67b00b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:33.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.067+0000 7f67b6873640 1 --2- 172.21.15.89:0/2843667267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f679c003980 0x7f67b00b4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:33.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.067+0000 7f67b6873640 1 -- 172.21.15.89:0/2843667267 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b01576e0 con 0x7f67b0146bd0 2023-10-30T12:37:33.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.067+0000 7f67b6072640 1 --2- 172.21.15.89:0/2843667267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f679c003980 0x7f67b00b4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:37:33.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.067+0000 7f67b5871640 1 --2- 172.21.15.89:0/2843667267 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f67b01e2aa0 0x7f67b00b3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:37:33.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.071+0000 7f67b6873640 1 --2- 172.21.15.89:0/2843667267 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67b0146bd0 0x7f67b0145460 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f67b006cb00 tx=0x7f67a8010bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:33.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.071+0000 7f67a6ffd640 1 -- 172.21.15.89:0/2843667267 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a800e830 con 0x7f67b0146bd0 2023-10-30T12:37:33.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.071+0000 7f67a6ffd640 1 -- 172.21.15.89:0/2843667267 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67a800e990 con 0x7f67b0146bd0 2023-10-30T12:37:33.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.071+0000 7f67a6ffd640 1 -- 172.21.15.89:0/2843667267 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a800ed20 con 0x7f67b0146bd0 2023-10-30T12:37:33.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.071+0000 7f67b82d6640 1 -- 172.21.15.89:0/2843667267 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67b01579c0 con 0x7f67b0146bd0 2023-10-30T12:37:33.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.071+0000 7f67b82d6640 1 -- 172.21.15.89:0/2843667267 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67b00b3170 con 0x7f67b0146bd0 2023-10-30T12:37:33.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.071+0000 7f67b82d6640 1 -- 172.21.15.89:0/2843667267 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67b004e170 con 0x7f67b0146bd0 2023-10-30T12:37:33.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.071+0000 7f67a6ffd640 1 -- 172.21.15.89:0/2843667267 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f67a801e3b0 con 0x7f67b0146bd0 2023-10-30T12:37:33.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.071+0000 7f67a6ffd640 1 --2- 172.21.15.89:0/2843667267 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f67940664a0 0x7f6794068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:33.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.071+0000 7f67a6ffd640 1 -- 172.21.15.89:0/2843667267 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6090+0+0 (secure 0 0 0) 0x7f67a8086ac0 con 0x7f67b0146bd0 2023-10-30T12:37:33.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.071+0000 7f67b6072640 1 --2- 172.21.15.89:0/2843667267 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f67940664a0 0x7f6794068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:33.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.075+0000 7f67b6072640 1 --2- 172.21.15.89:0/2843667267 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f67940664a0 0x7f6794068960 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f67ac001750 tx=0x7f67ac0011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:33.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.075+0000 7f67a6ffd640 1 -- 172.21.15.89:0/2843667267 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f67a8053dc0 con 0x7f67b0146bd0 2023-10-30T12:37:33.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.259+0000 7f67b82d6640 1 -- 172.21.15.89:0/2843667267 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67b00b3620 con 0x7f67b0146bd0 2023-10-30T12:37:33.259 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:37:33.260 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5018,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:37:11.414400+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:37:33.260 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 43 2023-10-30T12:37:33.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.259+0000 7f67a6ffd640 1 -- 172.21.15.89:0/2843667267 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6527 (secure 0 0 0) 0x7f67a8086e60 con 0x7f67b0146bd0 2023-10-30T12:37:33.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.263+0000 7f67b82d6640 1 -- 172.21.15.89:0/2843667267 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f67940664a0 msgr2=0x7f6794068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:33.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.263+0000 7f67b82d6640 1 --2- 172.21.15.89:0/2843667267 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f67940664a0 0x7f6794068960 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f67ac001750 tx=0x7f67ac0011a0 comp rx=0 tx=0).stop 2023-10-30T12:37:33.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.263+0000 7f67b82d6640 1 -- 172.21.15.89:0/2843667267 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67b0146bd0 msgr2=0x7f67b0145460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:33.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.263+0000 7f67b82d6640 1 --2- 172.21.15.89:0/2843667267 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67b0146bd0 0x7f67b0145460 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f67b006cb00 tx=0x7f67a8010bb0 comp rx=0 tx=0).stop 2023-10-30T12:37:33.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.263+0000 7f67b82d6640 1 -- 172.21.15.89:0/2843667267 shutdown_connections 2023-10-30T12:37:33.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.263+0000 7f67b82d6640 1 --2- 172.21.15.89:0/2843667267 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f67940664a0 0x7f6794068960 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:33.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.263+0000 7f67b82d6640 1 --2- 172.21.15.89:0/2843667267 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67b0146bd0 0x7f67b0145460 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:33.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.263+0000 7f67b82d6640 1 --2- 172.21.15.89:0/2843667267 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f67b01e2aa0 0x7f67b00b3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:33.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.263+0000 7f67b82d6640 1 --2- 172.21.15.89:0/2843667267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f679c003980 0x7f67b00b4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:33.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.263+0000 7f67b82d6640 1 -- 172.21.15.89:0/2843667267 >> 172.21.15.89:0/2843667267 conn(0x7f67b006bfe0 msgr2=0x7f67b01520f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:33.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.263+0000 7f67b82d6640 1 -- 172.21.15.89:0/2843667267 shutdown_connections 2023-10-30T12:37:33.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:33.263+0000 7f67b82d6640 1 -- 172.21.15.89:0/2843667267 wait complete. 2023-10-30T12:37:33.307 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:37:35.307 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:37:35.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.455+0000 7f962d009640 1 Processor -- start 2023-10-30T12:37:35.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.455+0000 7f962d009640 1 -- start start 2023-10-30T12:37:35.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.459+0000 7f962d009640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96280c6b80 0x7f96280bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:35.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.459+0000 7f962d009640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96280c0a50 con 0x7f962813f4a0 2023-10-30T12:37:35.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.459+0000 7f962d009640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f962806f6e0 con 0x7f96280c6f60 2023-10-30T12:37:35.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.459+0000 7f962d009640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f962806f9c0 con 0x7f96280c6b80 2023-10-30T12:37:35.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.459+0000 7f962759e640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f962813f4a0 0x7f96281566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40178/0 (socket says 172.21.15.89:40178) 2023-10-30T12:37:35.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.459+0000 7f962759e640 1 -- 172.21.15.89:0/2903725413 learned_addr learned my addr 172.21.15.89:0/2903725413 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:37:35.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.459+0000 7f9626d9d640 1 --2- 172.21.15.89:0/2903725413 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96280c6b80 0x7f96280bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:35.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.459+0000 7f962659c640 1 -- 172.21.15.89:0/2903725413 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 525974263 0 0) 0x7f96280c0a50 con 0x7f962813f4a0 2023-10-30T12:37:35.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.459+0000 7f962659c640 1 -- 172.21.15.89:0/2903725413 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9614005740 con 0x7f962813f4a0 2023-10-30T12:37:35.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.459+0000 7f962659c640 1 -- 172.21.15.89:0/2903725413 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 931347397 0 0) 0x7f9614005740 con 0x7f962813f4a0 2023-10-30T12:37:35.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.459+0000 7f962659c640 1 -- 172.21.15.89:0/2903725413 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96280c6b80 msgr2=0x7f96280bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:35.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.459+0000 7f962659c640 1 --2- 172.21.15.89:0/2903725413 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96280c6b80 0x7f96280bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:35.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.459+0000 7f962659c640 1 -- 172.21.15.89:0/2903725413 >> v1:172.21.15.192:6789/0 conn(0x7f96280c6f60 legacy=0x7f962813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:35.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.459+0000 7f962659c640 1 -- 172.21.15.89:0/2903725413 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f962806fca0 con 0x7f962813f4a0 2023-10-30T12:37:35.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.459+0000 7f962659c640 1 -- 172.21.15.89:0/2903725413 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f9620001a00 con 0x7f962813f4a0 2023-10-30T12:37:35.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.459+0000 7f962659c640 1 -- 172.21.15.89:0/2903725413 >> v1:172.21.15.89:6789/0 conn(0x7f962813f4a0 legacy=0x7f96281566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:35.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.459+0000 7f962659c640 1 --2- 172.21.15.89:0/2903725413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9614005bc0 0x7f9614005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:35.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.459+0000 7f962659c640 1 --2- 172.21.15.89:0/2903725413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9614006570 0x7f9614006a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:35.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.459+0000 7f962659c640 1 --2- 172.21.15.89:0/2903725413 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96280c6b80 0x7f96140072a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:35.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.459+0000 7f962659c640 1 -- 172.21.15.89:0/2903725413 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962806f9c0 con 0x7f9614005bc0 2023-10-30T12:37:35.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.459+0000 7f962659c640 1 -- 172.21.15.89:0/2903725413 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961400dba0 con 0x7f9614006570 2023-10-30T12:37:35.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.459+0000 7f962659c640 1 -- 172.21.15.89:0/2903725413 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f961400dd60 con 0x7f96280c6b80 2023-10-30T12:37:35.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.459+0000 7f9626d9d640 1 --2- 172.21.15.89:0/2903725413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9614005bc0 0x7f9614005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:35.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.459+0000 7f961e59c640 1 --2- 172.21.15.89:0/2903725413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9614006570 0x7f9614006a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:35.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.459+0000 7f962759e640 1 --2- 172.21.15.89:0/2903725413 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96280c6b80 0x7f96140072a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:35.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.459+0000 7f9626d9d640 1 -- 172.21.15.89:0/2903725413 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96280c6b80 msgr2=0x7f96140072a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:35.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.459+0000 7f9626d9d640 1 --2- 172.21.15.89:0/2903725413 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f96280c6b80 0x7f96140072a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:35.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.459+0000 7f9626d9d640 1 -- 172.21.15.89:0/2903725413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9614006570 msgr2=0x7f9614006a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:35.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.459+0000 7f9626d9d640 1 --2- 172.21.15.89:0/2903725413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9614006570 0x7f9614006a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:35.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.459+0000 7f9626d9d640 1 -- 172.21.15.89:0/2903725413 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9614005740 con 0x7f9614005bc0 2023-10-30T12:37:35.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.463+0000 7f9626d9d640 1 --2- 172.21.15.89:0/2903725413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9614005bc0 0x7f9614005fa0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9610009ae0 tx=0x7f961001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b3fd4974a3dba5e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:35.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.463+0000 7f962659c640 1 -- 172.21.15.89:0/2903725413 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96100a5640 con 0x7f9614005bc0 2023-10-30T12:37:35.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.463+0000 7f962659c640 1 -- 172.21.15.89:0/2903725413 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96100a57a0 con 0x7f9614005bc0 2023-10-30T12:37:35.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.463+0000 7f962659c640 1 -- 172.21.15.89:0/2903725413 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96100a5b50 con 0x7f9614005bc0 2023-10-30T12:37:35.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.463+0000 7f962d009640 1 -- 172.21.15.89:0/2903725413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9614005bc0 msgr2=0x7f9614005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:35.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.463+0000 7f962d009640 1 --2- 172.21.15.89:0/2903725413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9614005bc0 0x7f9614005fa0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9610009ae0 tx=0x7f961001ca50 comp rx=0 tx=0).stop 2023-10-30T12:37:35.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.463+0000 7f962759e640 1 -- 172.21.15.89:0/2903725413 reap_dead start 2023-10-30T12:37:35.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.463+0000 7f962d009640 1 -- 172.21.15.89:0/2903725413 shutdown_connections 2023-10-30T12:37:35.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.463+0000 7f962d009640 1 -- 172.21.15.89:0/2903725413 >> 172.21.15.89:0/2903725413 conn(0x7f96280c7910 msgr2=0x7f96280c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:35.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.463+0000 7f962d009640 1 -- 172.21.15.89:0/2903725413 shutdown_connections 2023-10-30T12:37:35.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.463+0000 7f962d009640 1 -- 172.21.15.89:0/2903725413 wait complete. 2023-10-30T12:37:35.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.463+0000 7f962d009640 1 Processor -- start 2023-10-30T12:37:35.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.463+0000 7f962d009640 1 -- start start 2023-10-30T12:37:35.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.467+0000 7f962d009640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9614005bc0 0x7f962813c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:35.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.467+0000 7f962d009640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9628155830 0x7f962813e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:35.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.467+0000 7f962d009640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9628154eb0 0x7f9628152f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:35.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.467+0000 7f962d009640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962813dd40 con 0x7f9628155830 2023-10-30T12:37:35.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.467+0000 7f962d009640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962813a3a0 con 0x7f9628154eb0 2023-10-30T12:37:35.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.467+0000 7f962d009640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f962813a8d0 con 0x7f9614005bc0 2023-10-30T12:37:35.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.467+0000 7f961e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9628155830 0x7f962813e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:35.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.467+0000 7f961e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9628155830 0x7f962813e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35922/0 (socket says 172.21.15.89:35922) 2023-10-30T12:37:35.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.467+0000 7f961e59c640 1 -- 172.21.15.89:0/2123764602 learned_addr learned my addr 172.21.15.89:0/2123764602 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:37:35.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.467+0000 7f9626d9d640 1 --2- 172.21.15.89:0/2123764602 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9614005bc0 0x7f962813c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:35.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.467+0000 7f962759e640 1 --2- 172.21.15.89:0/2123764602 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9628154eb0 0x7f9628152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:35.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.467+0000 7f961e59c640 1 -- 172.21.15.89:0/2123764602 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9614005bc0 msgr2=0x7f962813c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:35.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.467+0000 7f961e59c640 1 --2- 172.21.15.89:0/2123764602 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9614005bc0 0x7f962813c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:35.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.467+0000 7f961e59c640 1 -- 172.21.15.89:0/2123764602 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9628154eb0 msgr2=0x7f9628152f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:35.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.467+0000 7f961e59c640 1 --2- 172.21.15.89:0/2123764602 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9628154eb0 0x7f9628152f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:35.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.467+0000 7f961e59c640 1 -- 172.21.15.89:0/2123764602 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9628156500 con 0x7f9628155830 2023-10-30T12:37:35.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.467+0000 7f9626d9d640 1 --2- 172.21.15.89:0/2123764602 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9614005bc0 0x7f962813c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:37:35.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.467+0000 7f962759e640 1 --2- 172.21.15.89:0/2123764602 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9628154eb0 0x7f9628152f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:37:35.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.467+0000 7f961e59c640 1 --2- 172.21.15.89:0/2123764602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9628155830 0x7f962813e310 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f9618010f10 tx=0x7f961800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:35.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.467+0000 7f961ffff640 1 -- 172.21.15.89:0/2123764602 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9618005270 con 0x7f9628155830 2023-10-30T12:37:35.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.467+0000 7f961ffff640 1 -- 172.21.15.89:0/2123764602 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f961800d030 con 0x7f9628155830 2023-10-30T12:37:35.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.467+0000 7f962d009640 1 -- 172.21.15.89:0/2123764602 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96280b0400 con 0x7f9628155830 2023-10-30T12:37:35.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.467+0000 7f961ffff640 1 -- 172.21.15.89:0/2123764602 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961800ee50 con 0x7f9628155830 2023-10-30T12:37:35.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.467+0000 7f962d009640 1 -- 172.21.15.89:0/2123764602 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96280c8430 con 0x7f9628155830 2023-10-30T12:37:35.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.467+0000 7f962d009640 1 -- 172.21.15.89:0/2123764602 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f962804e170 con 0x7f9628155830 2023-10-30T12:37:35.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.471+0000 7f961ffff640 1 -- 172.21.15.89:0/2123764602 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9618003600 con 0x7f9628155830 2023-10-30T12:37:35.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.471+0000 7f961ffff640 1 --2- 172.21.15.89:0/2123764602 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f96080664a0 0x7f9608068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:35.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.471+0000 7f961ffff640 1 -- 172.21.15.89:0/2123764602 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6090+0+0 (secure 0 0 0) 0x7f9618087890 con 0x7f9628155830 2023-10-30T12:37:35.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.471+0000 7f9626d9d640 1 --2- 172.21.15.89:0/2123764602 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f96080664a0 0x7f9608068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:35.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.471+0000 7f9626d9d640 1 --2- 172.21.15.89:0/2123764602 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f96080664a0 0x7f9608068960 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9610009ae0 tx=0x7f96100046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:35.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.475+0000 7f961ffff640 1 -- 172.21.15.89:0/2123764602 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9618058610 con 0x7f9628155830 2023-10-30T12:37:35.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.651+0000 7f962d009640 1 -- 172.21.15.89:0/2123764602 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9628138040 con 0x7f9628155830 2023-10-30T12:37:35.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.651+0000 7f961ffff640 1 -- 172.21.15.89:0/2123764602 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6527 (secure 0 0 0) 0x7f9618018090 con 0x7f9628155830 2023-10-30T12:37:35.653 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:37:35.654 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5018,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:37:11.414400+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4318,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4318":{"gid":4318,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.192:6837/1002124471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1002124471},{"type":"v1","addr":"172.21.15.192:6837","nonce":1002124471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:37:35.654 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 43 2023-10-30T12:37:35.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.655+0000 7f962d009640 1 -- 172.21.15.89:0/2123764602 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f96080664a0 msgr2=0x7f9608068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:35.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.655+0000 7f962d009640 1 --2- 172.21.15.89:0/2123764602 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f96080664a0 0x7f9608068960 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9610009ae0 tx=0x7f96100046d0 comp rx=0 tx=0).stop 2023-10-30T12:37:35.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.655+0000 7f962d009640 1 -- 172.21.15.89:0/2123764602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9628155830 msgr2=0x7f962813e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:35.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.655+0000 7f962d009640 1 --2- 172.21.15.89:0/2123764602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9628155830 0x7f962813e310 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f9618010f10 tx=0x7f961800e530 comp rx=0 tx=0).stop 2023-10-30T12:37:35.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.655+0000 7f962d009640 1 -- 172.21.15.89:0/2123764602 shutdown_connections 2023-10-30T12:37:35.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.655+0000 7f962d009640 1 --2- 172.21.15.89:0/2123764602 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f96080664a0 0x7f9608068960 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:35.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.655+0000 7f962d009640 1 --2- 172.21.15.89:0/2123764602 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9628154eb0 0x7f9628152f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:35.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.655+0000 7f962d009640 1 --2- 172.21.15.89:0/2123764602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9628155830 0x7f962813e310 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:35.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.655+0000 7f962d009640 1 --2- 172.21.15.89:0/2123764602 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9614005bc0 0x7f962813c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:35.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.655+0000 7f962d009640 1 -- 172.21.15.89:0/2123764602 >> 172.21.15.89:0/2123764602 conn(0x7f96280c7910 msgr2=0x7f96280c10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:35.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.655+0000 7f962d009640 1 -- 172.21.15.89:0/2123764602 shutdown_connections 2023-10-30T12:37:35.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:35.655+0000 7f962d009640 1 -- 172.21.15.89:0/2123764602 wait complete. 2023-10-30T12:37:35.674 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:37:37.676 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:37:37.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.819+0000 7f512989c640 1 Processor -- start 2023-10-30T12:37:37.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.819+0000 7f512989c640 1 -- start start 2023-10-30T12:37:37.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.819+0000 7f512989c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f51241538a0 0x7f512415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:37.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.819+0000 7f512989c640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51240cf160 0x7f51240cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:37.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.819+0000 7f512989c640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51240cb800 con 0x7f51240ca0e0 2023-10-30T12:37:37.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.819+0000 7f512989c640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f512415dc30 con 0x7f51241538a0 2023-10-30T12:37:37.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.819+0000 7f512989c640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51240cb330 con 0x7f51240cf160 2023-10-30T12:37:37.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.819+0000 7f5122ffd640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f51241538a0 0x7f512415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:37.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.819+0000 7f5122ffd640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f51241538a0 0x7f512415d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35930/0 (socket says 172.21.15.89:35930) 2023-10-30T12:37:37.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.819+0000 7f5122ffd640 1 -- 172.21.15.89:0/1077924653 learned_addr learned my addr 172.21.15.89:0/1077924653 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:37:37.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.819+0000 7f51237fe640 1 --2- 172.21.15.89:0/1077924653 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51240cf160 0x7f51240cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:37.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.819+0000 7f5122ffd640 1 -- 172.21.15.89:0/1077924653 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51240cf160 msgr2=0x7f51240cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:37.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.819+0000 7f5122ffd640 1 --2- 172.21.15.89:0/1077924653 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51240cf160 0x7f51240cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:37.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.819+0000 7f5122ffd640 1 -- 172.21.15.89:0/1077924653 >> v1:172.21.15.192:6789/0 conn(0x7f51240ca0e0 legacy=0x7f512415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:37.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.819+0000 7f5122ffd640 1 -- 172.21.15.89:0/1077924653 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51240cad70 con 0x7f51241538a0 2023-10-30T12:37:37.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.823+0000 7f5122ffd640 1 --2- 172.21.15.89:0/1077924653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51241538a0 0x7f512415d6f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f5114009870 tx=0x7f511401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79e3bf10469c718b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:37.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.823+0000 7f51227fc640 1 -- 172.21.15.89:0/1077924653 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51140a89b0 con 0x7f51241538a0 2023-10-30T12:37:37.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.823+0000 7f51227fc640 1 -- 172.21.15.89:0/1077924653 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51140a8b10 con 0x7f51241538a0 2023-10-30T12:37:37.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.823+0000 7f51227fc640 1 -- 172.21.15.89:0/1077924653 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51140a8ea0 con 0x7f51241538a0 2023-10-30T12:37:37.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.823+0000 7f512989c640 1 -- 172.21.15.89:0/1077924653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51241538a0 msgr2=0x7f512415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:37.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.823+0000 7f512989c640 1 --2- 172.21.15.89:0/1077924653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51241538a0 0x7f512415d6f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f5114009870 tx=0x7f511401ecf0 comp rx=0 tx=0).stop 2023-10-30T12:37:37.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.823+0000 7f512989c640 1 -- 172.21.15.89:0/1077924653 shutdown_connections 2023-10-30T12:37:37.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.823+0000 7f512989c640 1 --2- 172.21.15.89:0/1077924653 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51240cf160 0x7f51240cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:37.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.823+0000 7f512989c640 1 --2- 172.21.15.89:0/1077924653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51241538a0 0x7f512415d6f0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:37.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.823+0000 7f512989c640 1 -- 172.21.15.89:0/1077924653 >> 172.21.15.89:0/1077924653 conn(0x7f512406bfe0 msgr2=0x7f512406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:37.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.823+0000 7f512989c640 1 -- 172.21.15.89:0/1077924653 shutdown_connections 2023-10-30T12:37:37.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.823+0000 7f512989c640 1 -- 172.21.15.89:0/1077924653 wait complete. 2023-10-30T12:37:37.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.827+0000 7f512989c640 1 Processor -- start 2023-10-30T12:37:37.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.827+0000 7f512989c640 1 -- start start 2023-10-30T12:37:37.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.827+0000 7f512989c640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51240ca0e0 0x7f512415ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:37.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.827+0000 7f512989c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51240cf160 0x7f512415b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:37.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.827+0000 7f512989c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51241538a0 0x7f51241e3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:37.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.827+0000 7f512989c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5124158dd0 con 0x7f51241538a0 2023-10-30T12:37:37.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.827+0000 7f512989c640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5124158250 con 0x7f51240cf160 2023-10-30T12:37:37.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.827+0000 7f512989c640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5124158530 con 0x7f51240ca0e0 2023-10-30T12:37:37.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.827+0000 7f51237fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51241538a0 0x7f51241e3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:37.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.827+0000 7f51237fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51241538a0 0x7f51241e3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35944/0 (socket says 172.21.15.89:35944) 2023-10-30T12:37:37.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.827+0000 7f51237fe640 1 -- 172.21.15.89:0/4244349106 learned_addr learned my addr 172.21.15.89:0/4244349106 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:37:37.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.827+0000 7f5122ffd640 1 --2- 172.21.15.89:0/4244349106 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51240ca0e0 0x7f512415ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:37.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.827+0000 7f511a7fc640 1 --2- 172.21.15.89:0/4244349106 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51240cf160 0x7f512415b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:37.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.827+0000 7f51237fe640 1 -- 172.21.15.89:0/4244349106 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51240ca0e0 msgr2=0x7f512415ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:37.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.827+0000 7f51237fe640 1 --2- 172.21.15.89:0/4244349106 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51240ca0e0 0x7f512415ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:37.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.827+0000 7f51237fe640 1 -- 172.21.15.89:0/4244349106 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51240cf160 msgr2=0x7f512415b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:37.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.827+0000 7f51237fe640 1 --2- 172.21.15.89:0/4244349106 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51240cf160 0x7f512415b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:37.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.827+0000 7f51237fe640 1 -- 172.21.15.89:0/4244349106 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51140094d0 con 0x7f51241538a0 2023-10-30T12:37:37.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.827+0000 7f5122ffd640 1 --2- 172.21.15.89:0/4244349106 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51240ca0e0 0x7f512415ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:37:37.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.827+0000 7f51237fe640 1 --2- 172.21.15.89:0/4244349106 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51241538a0 0x7f51241e3030 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f511c00d580 tx=0x7f511c00da40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:37.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.827+0000 7f5120ff9640 1 -- 172.21.15.89:0/4244349106 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f511c0125f0 con 0x7f51241538a0 2023-10-30T12:37:37.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.827+0000 7f5120ff9640 1 -- 172.21.15.89:0/4244349106 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f511c012750 con 0x7f51241538a0 2023-10-30T12:37:37.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.827+0000 7f512989c640 1 -- 172.21.15.89:0/4244349106 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5124158810 con 0x7f51241538a0 2023-10-30T12:37:37.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.827+0000 7f5120ff9640 1 -- 172.21.15.89:0/4244349106 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f511c012b00 con 0x7f51241538a0 2023-10-30T12:37:37.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.827+0000 7f512989c640 1 -- 172.21.15.89:0/4244349106 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f512415c300 con 0x7f51241538a0 2023-10-30T12:37:37.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.831+0000 7f512989c640 1 -- 172.21.15.89:0/4244349106 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51240b2a50 con 0x7f51241538a0 2023-10-30T12:37:37.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.831+0000 7f5120ff9640 1 -- 172.21.15.89:0/4244349106 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f511c012c60 con 0x7f51241538a0 2023-10-30T12:37:37.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.831+0000 7f5120ff9640 1 --2- 172.21.15.89:0/4244349106 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f51040664d0 0x7f5104068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:37.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.831+0000 7f5120ff9640 1 -- 172.21.15.89:0/4244349106 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f511c084580 con 0x7f51241538a0 2023-10-30T12:37:37.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.831+0000 7f5122ffd640 1 --2- 172.21.15.89:0/4244349106 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f51040664d0 0x7f5104068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:37.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.831+0000 7f5122ffd640 1 --2- 172.21.15.89:0/4244349106 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f51040664d0 0x7f5104068990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f51140b2000 tx=0x7f5114008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:37.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:37.835+0000 7f5120ff9640 1 -- 172.21.15.89:0/4244349106 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f511c0517b0 con 0x7f51241538a0 2023-10-30T12:37:38.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.019+0000 7f512989c640 1 -- 172.21.15.89:0/4244349106 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5124159870 con 0x7f51241538a0 2023-10-30T12:37:38.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.019+0000 7f5120ff9640 1 -- 172.21.15.89:0/4244349106 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+5734 (secure 0 0 0) 0x7f5124159870 con 0x7f51241538a0 2023-10-30T12:37:38.019 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:37:38.019 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:37:36.081246+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:37:38.021 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 44 2023-10-30T12:37:38.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.019+0000 7f512989c640 1 -- 172.21.15.89:0/4244349106 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f51040664d0 msgr2=0x7f5104068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:38.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.019+0000 7f512989c640 1 --2- 172.21.15.89:0/4244349106 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f51040664d0 0x7f5104068990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f51140b2000 tx=0x7f5114008040 comp rx=0 tx=0).stop 2023-10-30T12:37:38.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.019+0000 7f512989c640 1 -- 172.21.15.89:0/4244349106 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51241538a0 msgr2=0x7f51241e3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:38.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.019+0000 7f512989c640 1 --2- 172.21.15.89:0/4244349106 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51241538a0 0x7f51241e3030 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f511c00d580 tx=0x7f511c00da40 comp rx=0 tx=0).stop 2023-10-30T12:37:38.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.019+0000 7f512989c640 1 -- 172.21.15.89:0/4244349106 shutdown_connections 2023-10-30T12:37:38.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.019+0000 7f512989c640 1 --2- 172.21.15.89:0/4244349106 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f51040664d0 0x7f5104068990 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:38.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.019+0000 7f512989c640 1 --2- 172.21.15.89:0/4244349106 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51240cf160 0x7f512415b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:38.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.019+0000 7f512989c640 1 --2- 172.21.15.89:0/4244349106 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51241538a0 0x7f51241e3030 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:38.023 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.019+0000 7f512989c640 1 --2- 172.21.15.89:0/4244349106 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51240ca0e0 0x7f512415ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:38.023 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.019+0000 7f512989c640 1 -- 172.21.15.89:0/4244349106 >> 172.21.15.89:0/4244349106 conn(0x7f512406bfe0 msgr2=0x7f51240c4f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:38.023 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.019+0000 7f512989c640 1 -- 172.21.15.89:0/4244349106 shutdown_connections 2023-10-30T12:37:38.023 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.019+0000 7f512989c640 1 -- 172.21.15.89:0/4244349106 wait complete. 2023-10-30T12:37:38.039 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-10-30T12:37:38.040 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:37:38.040 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:37:38.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.175+0000 7f52c7ff5640 1 Processor -- start 2023-10-30T12:37:38.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.179+0000 7f52c7ff5640 1 -- start start 2023-10-30T12:37:38.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.179+0000 7f52c7ff5640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f52c0153d70 0x7f52c006ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:38.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.179+0000 7f52c7ff5640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52c00c8490 0x7f52c00c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:38.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.179+0000 7f52c7ff5640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c006afa0 con 0x7f52c006e470 2023-10-30T12:37:38.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.179+0000 7f52c7ff5640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c006d780 con 0x7f52c0153d70 2023-10-30T12:37:38.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.179+0000 7f52c7ff5640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c0154150 con 0x7f52c00c8490 2023-10-30T12:37:38.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.179+0000 7f52c6592640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52c00c8490 0x7f52c00c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:38.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.179+0000 7f52c5d91640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f52c0153d70 0x7f52c006ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:38.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.179+0000 7f52c6592640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52c00c8490 0x7f52c00c8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:56504/0 (socket says 172.21.15.89:56504) 2023-10-30T12:37:38.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.179+0000 7f52c6592640 1 -- 172.21.15.89:0/3866789748 learned_addr learned my addr 172.21.15.89:0/3866789748 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:37:38.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.179+0000 7f52c5590640 1 --1- 172.21.15.89:0/3866789748 >> v1:172.21.15.192:6789/0 conn(0x7f52c006e470 0x7f52c00cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:51084/0 (socket says 172.21.15.89:51084) 2023-10-30T12:37:38.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.179+0000 7f52c6592640 1 -- 172.21.15.89:0/3866789748 >> v2:172.21.15.192:3300/0 conn(0x7f52c0153d70 msgr2=0x7f52c006ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:38.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.179+0000 7f52c6592640 1 --2- 172.21.15.89:0/3866789748 >> v2:172.21.15.192:3300/0 conn(0x7f52c0153d70 0x7f52c006ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:38.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.179+0000 7f52c4d8f640 1 -- 172.21.15.89:0/3866789748 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3573127989 0 0) 0x7f52c006afa0 con 0x7f52c006e470 2023-10-30T12:37:38.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.179+0000 7f52c6592640 1 -- 172.21.15.89:0/3866789748 >> v1:172.21.15.192:6789/0 conn(0x7f52c006e470 legacy=0x7f52c00cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:38.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.179+0000 7f52c6592640 1 -- 172.21.15.89:0/3866789748 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c006e030 con 0x7f52c00c8490 2023-10-30T12:37:38.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.183+0000 7f52c6592640 1 --2- 172.21.15.89:0/3866789748 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52c00c8490 0x7f52c00c8cc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f52b8009e50 tx=0x7f52b801c9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e75d75b43442e17d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:38.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.183+0000 7f52c4d8f640 1 -- 172.21.15.89:0/3866789748 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b80a58a0 con 0x7f52c00c8490 2023-10-30T12:37:38.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.183+0000 7f52c4d8f640 1 -- 172.21.15.89:0/3866789748 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b80a5a00 con 0x7f52c00c8490 2023-10-30T12:37:38.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.183+0000 7f52c4d8f640 1 -- 172.21.15.89:0/3866789748 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b80a5d90 con 0x7f52c00c8490 2023-10-30T12:37:38.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.183+0000 7f52c7ff5640 1 -- 172.21.15.89:0/3866789748 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52c00c8490 msgr2=0x7f52c00c8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:38.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.183+0000 7f52c7ff5640 1 --2- 172.21.15.89:0/3866789748 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52c00c8490 0x7f52c00c8cc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f52b8009e50 tx=0x7f52b801c9e0 comp rx=0 tx=0).stop 2023-10-30T12:37:38.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.183+0000 7f52c7ff5640 1 -- 172.21.15.89:0/3866789748 shutdown_connections 2023-10-30T12:37:38.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.183+0000 7f52c7ff5640 1 --2- 172.21.15.89:0/3866789748 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52c00c8490 0x7f52c00c8cc0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:38.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.183+0000 7f52c7ff5640 1 --2- 172.21.15.89:0/3866789748 >> v2:172.21.15.192:3300/0 conn(0x7f52c0153d70 0x7f52c006ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:38.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.183+0000 7f52c7ff5640 1 -- 172.21.15.89:0/3866789748 >> 172.21.15.89:0/3866789748 conn(0x7f52c0071ea0 msgr2=0x7f52c0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:38.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.183+0000 7f52c7ff5640 1 -- 172.21.15.89:0/3866789748 shutdown_connections 2023-10-30T12:37:38.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.183+0000 7f52c7ff5640 1 -- 172.21.15.89:0/3866789748 wait complete. 2023-10-30T12:37:38.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.187+0000 7f52c7ff5640 1 Processor -- start 2023-10-30T12:37:38.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.187+0000 7f52c7ff5640 1 -- start start 2023-10-30T12:37:38.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.187+0000 7f52c7ff5640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52c006e470 0x7f52c01f29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:38.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.187+0000 7f52c7ff5640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52c00c8490 0x7f52c0152510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:38.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.187+0000 7f52c7ff5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c0153d70 0x7f52c01ebfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:38.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.187+0000 7f52c7ff5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c01546f0 con 0x7f52c0153d70 2023-10-30T12:37:38.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.187+0000 7f52c7ff5640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c0147260 con 0x7f52c006e470 2023-10-30T12:37:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.187+0000 7f52c7ff5640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c01ebb30 con 0x7f52c00c8490 2023-10-30T12:37:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.187+0000 7f52c6592640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c0153d70 0x7f52c01ebfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.187+0000 7f52c6592640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c0153d70 0x7f52c01ebfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35946/0 (socket says 172.21.15.89:35946) 2023-10-30T12:37:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.187+0000 7f52c6592640 1 -- 172.21.15.89:0/185398272 learned_addr learned my addr 172.21.15.89:0/185398272 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:37:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.187+0000 7f52c5590640 1 --2- 172.21.15.89:0/185398272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52c00c8490 0x7f52c0152510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:38.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.187+0000 7f52c5d91640 1 --2- 172.21.15.89:0/185398272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52c006e470 0x7f52c01f29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:38.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.187+0000 7f52c6592640 1 -- 172.21.15.89:0/185398272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52c00c8490 msgr2=0x7f52c0152510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:38.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.187+0000 7f52c6592640 1 --2- 172.21.15.89:0/185398272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52c00c8490 0x7f52c0152510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:38.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.187+0000 7f52c6592640 1 -- 172.21.15.89:0/185398272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52c006e470 msgr2=0x7f52c01f29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:38.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.187+0000 7f52c6592640 1 --2- 172.21.15.89:0/185398272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52c006e470 0x7f52c01f29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:38.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.187+0000 7f52c6592640 1 -- 172.21.15.89:0/185398272 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b8009b30 con 0x7f52c0153d70 2023-10-30T12:37:38.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.187+0000 7f52c5d91640 1 --2- 172.21.15.89:0/185398272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52c006e470 0x7f52c01f29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:37:38.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.187+0000 7f52c6592640 1 --2- 172.21.15.89:0/185398272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c0153d70 0x7f52c01ebfc0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f52b80af040 tx=0x7f52b8019780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:38.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.187+0000 7f52b67fc640 1 -- 172.21.15.89:0/185398272 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b80b8670 con 0x7f52c0153d70 2023-10-30T12:37:38.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.187+0000 7f52b67fc640 1 -- 172.21.15.89:0/185398272 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b80b87d0 con 0x7f52c0153d70 2023-10-30T12:37:38.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.187+0000 7f52b67fc640 1 -- 172.21.15.89:0/185398272 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b80b8b80 con 0x7f52c0153d70 2023-10-30T12:37:38.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.187+0000 7f52c7ff5640 1 -- 172.21.15.89:0/185398272 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c01f2ef0 con 0x7f52c0153d70 2023-10-30T12:37:38.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.187+0000 7f52c7ff5640 1 -- 172.21.15.89:0/185398272 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c0147490 con 0x7f52c0153d70 2023-10-30T12:37:38.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.187+0000 7f52c7ff5640 1 -- 172.21.15.89:0/185398272 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c004e170 con 0x7f52c0153d70 2023-10-30T12:37:38.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.191+0000 7f52b67fc640 1 -- 172.21.15.89:0/185398272 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f52b80b8ce0 con 0x7f52c0153d70 2023-10-30T12:37:38.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.191+0000 7f52b67fc640 1 --2- 172.21.15.89:0/185398272 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f52a40664a0 0x7f52a4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:38.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.191+0000 7f52b67fc640 1 -- 172.21.15.89:0/185398272 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f52b811eb60 con 0x7f52c0153d70 2023-10-30T12:37:38.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.191+0000 7f52c5d91640 1 --2- 172.21.15.89:0/185398272 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f52a40664a0 0x7f52a4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:38.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.191+0000 7f52c5d91640 1 --2- 172.21.15.89:0/185398272 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f52a40664a0 0x7f52a4068960 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f52bc009bb0 tx=0x7f52bc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:38.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.195+0000 7f52b67fc640 1 -- 172.21.15.89:0/185398272 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f52b80ebda0 con 0x7f52c0153d70 2023-10-30T12:37:38.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.379+0000 7f52c7ff5640 1 -- 172.21.15.89:0/185398272 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52c01ebc70 con 0x7f52c0153d70 2023-10-30T12:37:38.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.379+0000 7f52b67fc640 1 -- 172.21.15.89:0/185398272 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+5734 (secure 0 0 0) 0x7f52b80ef3c0 con 0x7f52c0153d70 2023-10-30T12:37:38.380 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:37:38.380 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:37:36.081246+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:37:38.381 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 44 2023-10-30T12:37:38.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.379+0000 7f52c7ff5640 1 -- 172.21.15.89:0/185398272 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f52a40664a0 msgr2=0x7f52a4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:38.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.379+0000 7f52c7ff5640 1 --2- 172.21.15.89:0/185398272 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f52a40664a0 0x7f52a4068960 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f52bc009bb0 tx=0x7f52bc008040 comp rx=0 tx=0).stop 2023-10-30T12:37:38.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.379+0000 7f52c7ff5640 1 -- 172.21.15.89:0/185398272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c0153d70 msgr2=0x7f52c01ebfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:38.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.379+0000 7f52c7ff5640 1 --2- 172.21.15.89:0/185398272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c0153d70 0x7f52c01ebfc0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f52b80af040 tx=0x7f52b8019780 comp rx=0 tx=0).stop 2023-10-30T12:37:38.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.379+0000 7f52c7ff5640 1 -- 172.21.15.89:0/185398272 shutdown_connections 2023-10-30T12:37:38.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.379+0000 7f52c7ff5640 1 --2- 172.21.15.89:0/185398272 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f52a40664a0 0x7f52a4068960 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:38.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.379+0000 7f52c7ff5640 1 --2- 172.21.15.89:0/185398272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c0153d70 0x7f52c01ebfc0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:38.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.379+0000 7f52c7ff5640 1 --2- 172.21.15.89:0/185398272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52c00c8490 0x7f52c0152510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:38.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.379+0000 7f52c7ff5640 1 --2- 172.21.15.89:0/185398272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52c006e470 0x7f52c01f29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:38.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.379+0000 7f52c7ff5640 1 -- 172.21.15.89:0/185398272 >> 172.21.15.89:0/185398272 conn(0x7f52c0071ea0 msgr2=0x7f52c01501d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:38.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.379+0000 7f52c7ff5640 1 -- 172.21.15.89:0/185398272 shutdown_connections 2023-10-30T12:37:38.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:38.379+0000 7f52c7ff5640 1 -- 172.21.15.89:0/185398272 wait complete. 2023-10-30T12:37:38.401 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 44, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:50.729245+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 44, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4781": { "addr": "172.21.15.192:6835/654256299", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6834", "nonce": 654256299, "type": "v2" }, { "addr": "172.21.15.192:6835", "nonce": 654256299, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4781, "incarnation": 24, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 27 }, "gid_5018": { "addr": "172.21.15.192:6839/93914587", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6838", "nonce": 93914587, "type": "v2" }, { "addr": "172.21.15.192:6839", "nonce": 93914587, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5018, "incarnation": 44, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 }, "gid_5139": { "addr": "172.21.15.89:6837/951276252", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6836", "nonce": 951276252, "type": "v2" }, { "addr": "172.21.15.89:6837", "nonce": 951276252, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5139, "incarnation": 33, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 29 } }, "last_failure": 0, "last_failure_osd_epoch": 55, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:37:36.081246+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5018, "mds_1": 5139, "mds_2": 4781 } } } ], "standbys": [ { "addr": "172.21.15.89:6834/2909790173", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6832", "nonce": 2909790173, "type": "v2" }, { "addr": "172.21.15.89:6834", "nonce": 2909790173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.89:6835/2387357615", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6833", "nonce": 2387357615, "type": "v2" }, { "addr": "172.21.15.89:6835", "nonce": 2387357615, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4239, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:37:40.402 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:37:40.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.551+0000 7fb7dcf1d640 1 Processor -- start 2023-10-30T12:37:40.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.551+0000 7fb7dcf1d640 1 -- start start 2023-10-30T12:37:40.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.551+0000 7fb7dcf1d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7d806f7a0 0x7fb7d806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:40.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.551+0000 7fb7dcf1d640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d8066cf0 con 0x7fb7d8071110 2023-10-30T12:37:40.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.551+0000 7fb7dcf1d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d80c70d0 con 0x7fb7d80c6bd0 2023-10-30T12:37:40.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.551+0000 7fb7dcf1d640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d80700c0 con 0x7fb7d806f7a0 2023-10-30T12:37:40.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.551+0000 7fb7d6d9d640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb7d8071110 0x7fb7d8158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58260/0 (socket says 172.21.15.89:58260) 2023-10-30T12:37:40.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.551+0000 7fb7d6d9d640 1 -- 172.21.15.89:0/3593109051 learned_addr learned my addr 172.21.15.89:0/3593109051 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:37:40.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.551+0000 7fb7d659c640 1 --2- 172.21.15.89:0/3593109051 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7d806f7a0 0x7fb7d806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:40.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.551+0000 7fb7d5d9b640 1 -- 172.21.15.89:0/3593109051 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085452181 0 0) 0x7fb7d8066cf0 con 0x7fb7d8071110 2023-10-30T12:37:40.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.551+0000 7fb7d5d9b640 1 -- 172.21.15.89:0/3593109051 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7c4003530 con 0x7fb7d8071110 2023-10-30T12:37:40.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.555+0000 7fb7d5d9b640 1 -- 172.21.15.89:0/3593109051 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2250268310 0 0) 0x7fb7d80c70d0 con 0x7fb7d80c6bd0 2023-10-30T12:37:40.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.555+0000 7fb7d5d9b640 1 -- 172.21.15.89:0/3593109051 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7d8066cf0 con 0x7fb7d80c6bd0 2023-10-30T12:37:40.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.555+0000 7fb7d5d9b640 1 -- 172.21.15.89:0/3593109051 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3705443954 0 0) 0x7fb7c4003530 con 0x7fb7d8071110 2023-10-30T12:37:40.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.555+0000 7fb7d5d9b640 1 -- 172.21.15.89:0/3593109051 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7d806f7a0 msgr2=0x7fb7d806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:40.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.555+0000 7fb7d5d9b640 1 --2- 172.21.15.89:0/3593109051 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7d806f7a0 0x7fb7d806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:40.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.555+0000 7fb7d5d9b640 1 -- 172.21.15.89:0/3593109051 >> v1:172.21.15.192:6789/0 conn(0x7fb7d80c6bd0 legacy=0x7fb7d8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:40.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.555+0000 7fb7d5d9b640 1 -- 172.21.15.89:0/3593109051 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d80c7390 con 0x7fb7d8071110 2023-10-30T12:37:40.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.555+0000 7fb7d5d9b640 1 -- 172.21.15.89:0/3593109051 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7fb7d0001a00 con 0x7fb7d8071110 2023-10-30T12:37:40.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.555+0000 7fb7d5d9b640 1 -- 172.21.15.89:0/3593109051 >> v1:172.21.15.89:6789/0 conn(0x7fb7d8071110 legacy=0x7fb7d8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:40.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.555+0000 7fb7d5d9b640 1 --2- 172.21.15.89:0/3593109051 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7d806f7a0 0x7fb7c4003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:40.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.555+0000 7fb7d5d9b640 1 --2- 172.21.15.89:0/3593109051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c40040f0 0x7fb7c40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:40.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.555+0000 7fb7d5d9b640 1 --2- 172.21.15.89:0/3593109051 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7c4004d10 0x7fb7c4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:40.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.555+0000 7fb7d5d9b640 1 -- 172.21.15.89:0/3593109051 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d80700c0 con 0x7fb7c40040f0 2023-10-30T12:37:40.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.555+0000 7fb7d5d9b640 1 -- 172.21.15.89:0/3593109051 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c400bad0 con 0x7fb7c4004d10 2023-10-30T12:37:40.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.555+0000 7fb7d5d9b640 1 -- 172.21.15.89:0/3593109051 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c400bc90 con 0x7fb7d806f7a0 2023-10-30T12:37:40.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.555+0000 7fb7cdd9b640 1 --2- 172.21.15.89:0/3593109051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c40040f0 0x7fb7c40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:40.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.555+0000 7fb7d659c640 1 --2- 172.21.15.89:0/3593109051 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7d806f7a0 0x7fb7c4003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:40.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.555+0000 7fb7d6d9d640 1 --2- 172.21.15.89:0/3593109051 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7c4004d10 0x7fb7c4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:40.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.555+0000 7fb7cdd9b640 1 -- 172.21.15.89:0/3593109051 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7d806f7a0 msgr2=0x7fb7c4003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:40.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.555+0000 7fb7cdd9b640 1 --2- 172.21.15.89:0/3593109051 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7d806f7a0 0x7fb7c4003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:40.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.555+0000 7fb7cdd9b640 1 -- 172.21.15.89:0/3593109051 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7c4004d10 msgr2=0x7fb7c4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:40.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.555+0000 7fb7cdd9b640 1 --2- 172.21.15.89:0/3593109051 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7c4004d10 0x7fb7c4005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:40.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.555+0000 7fb7cdd9b640 1 -- 172.21.15.89:0/3593109051 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c4003530 con 0x7fb7c40040f0 2023-10-30T12:37:40.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.555+0000 7fb7cdd9b640 1 --2- 172.21.15.89:0/3593109051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c40040f0 0x7fb7c40047d0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb7c80018d0 tx=0x7fb7c801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a7f3643818bffe8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:40.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.555+0000 7fb7d5d9b640 1 -- 172.21.15.89:0/3593109051 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c80052d0 con 0x7fb7c40040f0 2023-10-30T12:37:40.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.559+0000 7fb7d5d9b640 1 -- 172.21.15.89:0/3593109051 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c80a7050 con 0x7fb7c40040f0 2023-10-30T12:37:40.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.559+0000 7fb7d5d9b640 1 -- 172.21.15.89:0/3593109051 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c8003600 con 0x7fb7c40040f0 2023-10-30T12:37:40.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.559+0000 7fb7dcf1d640 1 -- 172.21.15.89:0/3593109051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c40040f0 msgr2=0x7fb7c40047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:40.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.559+0000 7fb7dcf1d640 1 --2- 172.21.15.89:0/3593109051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c40040f0 0x7fb7c40047d0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb7c80018d0 tx=0x7fb7c801c3e0 comp rx=0 tx=0).stop 2023-10-30T12:37:40.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.559+0000 7fb7d6d9d640 1 -- 172.21.15.89:0/3593109051 reap_dead start 2023-10-30T12:37:40.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.559+0000 7fb7dcf1d640 1 -- 172.21.15.89:0/3593109051 shutdown_connections 2023-10-30T12:37:40.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.559+0000 7fb7dcf1d640 1 -- 172.21.15.89:0/3593109051 >> 172.21.15.89:0/3593109051 conn(0x7fb7d80c7910 msgr2=0x7fb7d80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:40.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.559+0000 7fb7dcf1d640 1 -- 172.21.15.89:0/3593109051 shutdown_connections 2023-10-30T12:37:40.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.559+0000 7fb7dcf1d640 1 -- 172.21.15.89:0/3593109051 wait complete. 2023-10-30T12:37:40.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.559+0000 7fb7dcf1d640 1 Processor -- start 2023-10-30T12:37:40.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.559+0000 7fb7dcf1d640 1 -- start start 2023-10-30T12:37:40.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.559+0000 7fb7dcf1d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c40040f0 0x7fb7d81f2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:40.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.559+0000 7fb7dcf1d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7d81f2760 0x7fb7d81efec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:40.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.559+0000 7fb7dcf1d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7d8154d90 0x7fb7d81f0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:40.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.559+0000 7fb7dcf1d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d81f2ed0 con 0x7fb7c40040f0 2023-10-30T12:37:40.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.559+0000 7fb7dcf1d640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d81570c0 con 0x7fb7d8154d90 2023-10-30T12:37:40.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.559+0000 7fb7dcf1d640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d80c1910 con 0x7fb7d81f2760 2023-10-30T12:37:40.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.563+0000 7fb7d659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c40040f0 0x7fb7d81f2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:40.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.563+0000 7fb7d659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c40040f0 0x7fb7d81f2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59978/0 (socket says 172.21.15.89:59978) 2023-10-30T12:37:40.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.563+0000 7fb7d659c640 1 -- 172.21.15.89:0/205044338 learned_addr learned my addr 172.21.15.89:0/205044338 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:37:40.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.563+0000 7fb7cdd9b640 1 --2- 172.21.15.89:0/205044338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7d81f2760 0x7fb7d81efec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:40.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.563+0000 7fb7d6d9d640 1 --2- 172.21.15.89:0/205044338 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7d8154d90 0x7fb7d81f0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:40.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.563+0000 7fb7d659c640 1 -- 172.21.15.89:0/205044338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7d81f2760 msgr2=0x7fb7d81efec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:40.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.563+0000 7fb7d659c640 1 --2- 172.21.15.89:0/205044338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7d81f2760 0x7fb7d81efec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:40.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.563+0000 7fb7d659c640 1 -- 172.21.15.89:0/205044338 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7d8154d90 msgr2=0x7fb7d81f0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:40.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.563+0000 7fb7d659c640 1 --2- 172.21.15.89:0/205044338 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7d8154d90 0x7fb7d81f0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:40.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.563+0000 7fb7d659c640 1 -- 172.21.15.89:0/205044338 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d81f2bf0 con 0x7fb7c40040f0 2023-10-30T12:37:40.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.563+0000 7fb7d659c640 1 --2- 172.21.15.89:0/205044338 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c40040f0 0x7fb7d81f2220 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0011a00 tx=0x7fb7c0011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:40.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.563+0000 7fb7cf7fe640 1 -- 172.21.15.89:0/205044338 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c000faf0 con 0x7fb7c40040f0 2023-10-30T12:37:40.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.563+0000 7fb7cf7fe640 1 -- 172.21.15.89:0/205044338 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c000fc50 con 0x7fb7c40040f0 2023-10-30T12:37:40.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.563+0000 7fb7dcf1d640 1 -- 172.21.15.89:0/205044338 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d8156790 con 0x7fb7c40040f0 2023-10-30T12:37:40.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.563+0000 7fb7cf7fe640 1 -- 172.21.15.89:0/205044338 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c0014440 con 0x7fb7c40040f0 2023-10-30T12:37:40.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.563+0000 7fb7dcf1d640 1 -- 172.21.15.89:0/205044338 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d8156d50 con 0x7fb7c40040f0 2023-10-30T12:37:40.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.563+0000 7fb7dcf1d640 1 -- 172.21.15.89:0/205044338 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d804e170 con 0x7fb7c40040f0 2023-10-30T12:37:40.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.563+0000 7fb7cf7fe640 1 -- 172.21.15.89:0/205044338 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb7c00145a0 con 0x7fb7c40040f0 2023-10-30T12:37:40.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.563+0000 7fb7cf7fe640 1 --2- 172.21.15.89:0/205044338 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb7b80664a0 0x7fb7b8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:40.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.567+0000 7fb7cf7fe640 1 -- 172.21.15.89:0/205044338 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fb7c0084900 con 0x7fb7c40040f0 2023-10-30T12:37:40.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.567+0000 7fb7cdd9b640 1 --2- 172.21.15.89:0/205044338 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb7b80664a0 0x7fb7b8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:40.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.567+0000 7fb7cdd9b640 1 --2- 172.21.15.89:0/205044338 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb7b80664a0 0x7fb7b8068960 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb7c80ac000 tx=0x7fb7c80a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:40.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.567+0000 7fb7cf7fe640 1 -- 172.21.15.89:0/205044338 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb7c0051be0 con 0x7fb7c40040f0 2023-10-30T12:37:40.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.746+0000 7fb7dcf1d640 1 -- 172.21.15.89:0/205044338 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7d80c4a70 con 0x7fb7c40040f0 2023-10-30T12:37:40.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.746+0000 7fb7cf7fe640 1 -- 172.21.15.89:0/205044338 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+5734 (secure 0 0 0) 0x7fb7c0055200 con 0x7fb7c40040f0 2023-10-30T12:37:40.748 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:37:40.748 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:37:36.081246+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:37:40.748 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 44 2023-10-30T12:37:40.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.746+0000 7fb7dcf1d640 1 -- 172.21.15.89:0/205044338 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb7b80664a0 msgr2=0x7fb7b8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:40.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.746+0000 7fb7dcf1d640 1 --2- 172.21.15.89:0/205044338 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb7b80664a0 0x7fb7b8068960 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb7c80ac000 tx=0x7fb7c80a6000 comp rx=0 tx=0).stop 2023-10-30T12:37:40.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.746+0000 7fb7dcf1d640 1 -- 172.21.15.89:0/205044338 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c40040f0 msgr2=0x7fb7d81f2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:40.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.746+0000 7fb7dcf1d640 1 --2- 172.21.15.89:0/205044338 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c40040f0 0x7fb7d81f2220 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0011a00 tx=0x7fb7c0011ec0 comp rx=0 tx=0).stop 2023-10-30T12:37:40.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.746+0000 7fb7dcf1d640 1 -- 172.21.15.89:0/205044338 shutdown_connections 2023-10-30T12:37:40.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.746+0000 7fb7dcf1d640 1 --2- 172.21.15.89:0/205044338 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb7b80664a0 0x7fb7b8068960 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:40.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.746+0000 7fb7dcf1d640 1 --2- 172.21.15.89:0/205044338 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7d8154d90 0x7fb7d81f0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:40.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.746+0000 7fb7dcf1d640 1 --2- 172.21.15.89:0/205044338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb7d81f2760 0x7fb7d81efec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:40.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.746+0000 7fb7dcf1d640 1 --2- 172.21.15.89:0/205044338 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7c40040f0 0x7fb7d81f2220 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:40.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.746+0000 7fb7dcf1d640 1 -- 172.21.15.89:0/205044338 >> 172.21.15.89:0/205044338 conn(0x7fb7d80c7910 msgr2=0x7fb7d80c0900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:40.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.750+0000 7fb7dcf1d640 1 -- 172.21.15.89:0/205044338 shutdown_connections 2023-10-30T12:37:40.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:40.750+0000 7fb7dcf1d640 1 -- 172.21.15.89:0/205044338 wait complete. 2023-10-30T12:37:40.768 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:37:42.769 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:37:42.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.914+0000 7f250a70f640 1 Processor -- start 2023-10-30T12:37:42.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f250a70f640 1 -- start start 2023-10-30T12:37:42.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f250a70f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25040c6b80 0x7f25040bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:42.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f250a70f640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25040c0a50 con 0x7f250413f4a0 2023-10-30T12:37:42.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f250a70f640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f250406f6e0 con 0x7f25040c6f60 2023-10-30T12:37:42.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f250a70f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f250406f9c0 con 0x7f25040c6b80 2023-10-30T12:37:42.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f2508cac640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f250413f4a0 0x7f25041566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58266/0 (socket says 172.21.15.89:58266) 2023-10-30T12:37:42.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f2508cac640 1 -- 172.21.15.89:0/1314857797 learned_addr learned my addr 172.21.15.89:0/1314857797 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:37:42.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f2503fff640 1 --2- 172.21.15.89:0/1314857797 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25040c6b80 0x7f25040bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:42.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f2502ffd640 1 -- 172.21.15.89:0/1314857797 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3926544270 0 0) 0x7f25040c0a50 con 0x7f250413f4a0 2023-10-30T12:37:42.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f2502ffd640 1 -- 172.21.15.89:0/1314857797 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24ec005740 con 0x7f250413f4a0 2023-10-30T12:37:42.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f2502ffd640 1 -- 172.21.15.89:0/1314857797 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2259743870 0 0) 0x7f24ec005740 con 0x7f250413f4a0 2023-10-30T12:37:42.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f2502ffd640 1 -- 172.21.15.89:0/1314857797 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25040c6b80 msgr2=0x7f25040bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:42.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f2502ffd640 1 --2- 172.21.15.89:0/1314857797 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25040c6b80 0x7f25040bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:42.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f2502ffd640 1 -- 172.21.15.89:0/1314857797 >> v1:172.21.15.192:6789/0 conn(0x7f25040c6f60 legacy=0x7f250413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:42.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f2502ffd640 1 -- 172.21.15.89:0/1314857797 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f250406fca0 con 0x7f250413f4a0 2023-10-30T12:37:42.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f2502ffd640 1 -- 172.21.15.89:0/1314857797 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f24fc001a00 con 0x7f250413f4a0 2023-10-30T12:37:42.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f2502ffd640 1 -- 172.21.15.89:0/1314857797 >> v1:172.21.15.89:6789/0 conn(0x7f250413f4a0 legacy=0x7f25041566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:42.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f2502ffd640 1 --2- 172.21.15.89:0/1314857797 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24ec005bc0 0x7f24ec005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:42.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f2502ffd640 1 --2- 172.21.15.89:0/1314857797 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25040c6b80 0x7f24ec0069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:42.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f2502ffd640 1 --2- 172.21.15.89:0/1314857797 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24ec006f20 0x7f24ec0073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:42.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f2502ffd640 1 -- 172.21.15.89:0/1314857797 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250406f9c0 con 0x7f24ec006f20 2023-10-30T12:37:42.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f2502ffd640 1 -- 172.21.15.89:0/1314857797 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec00dba0 con 0x7f24ec005bc0 2023-10-30T12:37:42.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f2502ffd640 1 -- 172.21.15.89:0/1314857797 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec00dd60 con 0x7f25040c6b80 2023-10-30T12:37:42.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f2508cac640 1 --2- 172.21.15.89:0/1314857797 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24ec006f20 0x7f24ec0073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:42.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f2503fff640 1 --2- 172.21.15.89:0/1314857797 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24ec005bc0 0x7f24ec005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:42.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f25037fe640 1 --2- 172.21.15.89:0/1314857797 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25040c6b80 0x7f24ec0069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:42.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f2508cac640 1 -- 172.21.15.89:0/1314857797 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25040c6b80 msgr2=0x7f24ec0069e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:42.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f2508cac640 1 --2- 172.21.15.89:0/1314857797 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25040c6b80 0x7f24ec0069e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:42.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f2508cac640 1 -- 172.21.15.89:0/1314857797 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24ec005bc0 msgr2=0x7f24ec005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:42.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f2508cac640 1 --2- 172.21.15.89:0/1314857797 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24ec005bc0 0x7f24ec005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:42.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.918+0000 7f2508cac640 1 -- 172.21.15.89:0/1314857797 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ec005740 con 0x7f24ec006f20 2023-10-30T12:37:42.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.922+0000 7f2508cac640 1 --2- 172.21.15.89:0/1314857797 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24ec006f20 0x7f24ec0073a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f250406f4f0 tx=0x7f24fc01f9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd2857867276834f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:42.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.922+0000 7f2502ffd640 1 -- 172.21.15.89:0/1314857797 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24fc0ac670 con 0x7f24ec006f20 2023-10-30T12:37:42.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.922+0000 7f2502ffd640 1 -- 172.21.15.89:0/1314857797 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24fc0ac7f0 con 0x7f24ec006f20 2023-10-30T12:37:42.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.922+0000 7f2502ffd640 1 -- 172.21.15.89:0/1314857797 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24fc0acba0 con 0x7f24ec006f20 2023-10-30T12:37:42.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.922+0000 7f250a70f640 1 -- 172.21.15.89:0/1314857797 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24ec006f20 msgr2=0x7f24ec0073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:42.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.922+0000 7f250a70f640 1 --2- 172.21.15.89:0/1314857797 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24ec006f20 0x7f24ec0073a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f250406f4f0 tx=0x7f24fc01f9c0 comp rx=0 tx=0).stop 2023-10-30T12:37:42.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.922+0000 7f2508cac640 1 -- 172.21.15.89:0/1314857797 reap_dead start 2023-10-30T12:37:42.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.922+0000 7f250a70f640 1 -- 172.21.15.89:0/1314857797 shutdown_connections 2023-10-30T12:37:42.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.922+0000 7f250a70f640 1 -- 172.21.15.89:0/1314857797 >> 172.21.15.89:0/1314857797 conn(0x7f25040c7910 msgr2=0x7f25040c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:42.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.922+0000 7f250a70f640 1 -- 172.21.15.89:0/1314857797 shutdown_connections 2023-10-30T12:37:42.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.922+0000 7f250a70f640 1 -- 172.21.15.89:0/1314857797 wait complete. 2023-10-30T12:37:42.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.926+0000 7f250a70f640 1 Processor -- start 2023-10-30T12:37:42.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.926+0000 7f250a70f640 1 -- start start 2023-10-30T12:37:42.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.926+0000 7f250a70f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24ec006f20 0x7f2504153410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:42.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.926+0000 7f250a70f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f250413edd0 0x7f2504155bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:42.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.926+0000 7f250a70f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2504159140 0x7f25041edcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:42.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.926+0000 7f250a70f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2504158cb0 con 0x7f250413edd0 2023-10-30T12:37:42.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.926+0000 7f250a70f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250413bd50 con 0x7f24ec006f20 2023-10-30T12:37:42.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.926+0000 7f250a70f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2504153950 con 0x7f2504159140 2023-10-30T12:37:42.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.926+0000 7f25037fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f250413edd0 0x7f2504155bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:42.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.926+0000 7f25037fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f250413edd0 0x7f2504155bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60000/0 (socket says 172.21.15.89:60000) 2023-10-30T12:37:42.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.926+0000 7f25037fe640 1 -- 172.21.15.89:0/1737753777 learned_addr learned my addr 172.21.15.89:0/1737753777 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:37:42.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.926+0000 7f2503fff640 1 --2- 172.21.15.89:0/1737753777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24ec006f20 0x7f2504153410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:42.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.926+0000 7f2508cac640 1 --2- 172.21.15.89:0/1737753777 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2504159140 0x7f25041edcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:42.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.926+0000 7f25037fe640 1 -- 172.21.15.89:0/1737753777 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2504159140 msgr2=0x7f25041edcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:42.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.926+0000 7f25037fe640 1 --2- 172.21.15.89:0/1737753777 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2504159140 0x7f25041edcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:42.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.926+0000 7f25037fe640 1 -- 172.21.15.89:0/1737753777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24ec006f20 msgr2=0x7f2504153410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:42.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.926+0000 7f25037fe640 1 --2- 172.21.15.89:0/1737753777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24ec006f20 0x7f2504153410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:42.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.926+0000 7f25037fe640 1 -- 172.21.15.89:0/1737753777 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25041560f0 con 0x7f250413edd0 2023-10-30T12:37:42.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.926+0000 7f2508cac640 1 --2- 172.21.15.89:0/1737753777 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2504159140 0x7f25041edcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:37:42.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.926+0000 7f25037fe640 1 --2- 172.21.15.89:0/1737753777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f250413edd0 0x7f2504155bb0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f24f800e5a0 tx=0x7f24f800ea60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:42.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.926+0000 7f2500ff9640 1 -- 172.21.15.89:0/1737753777 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f80052e0 con 0x7f250413edd0 2023-10-30T12:37:42.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.926+0000 7f2500ff9640 1 -- 172.21.15.89:0/1737753777 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f8005440 con 0x7f250413edd0 2023-10-30T12:37:42.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.926+0000 7f2500ff9640 1 -- 172.21.15.89:0/1737753777 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f8006a20 con 0x7f250413edd0 2023-10-30T12:37:42.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.926+0000 7f250a70f640 1 -- 172.21.15.89:0/1737753777 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25041555f0 con 0x7f250413edd0 2023-10-30T12:37:42.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.926+0000 7f250a70f640 1 -- 172.21.15.89:0/1737753777 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2504158940 con 0x7f250413edd0 2023-10-30T12:37:42.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.930+0000 7f250a70f640 1 -- 172.21.15.89:0/1737753777 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2504158e60 con 0x7f250413edd0 2023-10-30T12:37:42.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.930+0000 7f2500ff9640 1 -- 172.21.15.89:0/1737753777 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f24f8006b80 con 0x7f250413edd0 2023-10-30T12:37:42.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.930+0000 7f2500ff9640 1 --2- 172.21.15.89:0/1737753777 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f24f00664d0 0x7f24f0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:42.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.930+0000 7f2500ff9640 1 -- 172.21.15.89:0/1737753777 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f24f8083990 con 0x7f250413edd0 2023-10-30T12:37:42.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.930+0000 7f2503fff640 1 --2- 172.21.15.89:0/1737753777 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f24f00664d0 0x7f24f0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:42.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.930+0000 7f2503fff640 1 --2- 172.21.15.89:0/1737753777 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f24f00664d0 0x7f24f0068990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f24f4009b20 tx=0x7f24f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:42.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:42.934+0000 7f2500ff9640 1 -- 172.21.15.89:0/1737753777 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2504158e60 con 0x7f250413edd0 2023-10-30T12:37:43.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:43.114+0000 7f250a70f640 1 -- 172.21.15.89:0/1737753777 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2504158e60 con 0x7f250413edd0 2023-10-30T12:37:43.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:43.118+0000 7f2500ff9640 1 -- 172.21.15.89:0/1737753777 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+5734 (secure 0 0 0) 0x7f2504158e60 con 0x7f250413edd0 2023-10-30T12:37:43.118 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:37:43.118 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:37:36.081246+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:37:43.119 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 44 2023-10-30T12:37:43.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:43.118+0000 7f250a70f640 1 -- 172.21.15.89:0/1737753777 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f24f00664d0 msgr2=0x7f24f0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:43.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:43.118+0000 7f250a70f640 1 --2- 172.21.15.89:0/1737753777 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f24f00664d0 0x7f24f0068990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f24f4009b20 tx=0x7f24f4008040 comp rx=0 tx=0).stop 2023-10-30T12:37:43.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:43.118+0000 7f250a70f640 1 -- 172.21.15.89:0/1737753777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f250413edd0 msgr2=0x7f2504155bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:43.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:43.118+0000 7f250a70f640 1 --2- 172.21.15.89:0/1737753777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f250413edd0 0x7f2504155bb0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f24f800e5a0 tx=0x7f24f800ea60 comp rx=0 tx=0).stop 2023-10-30T12:37:43.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:43.118+0000 7f250a70f640 1 -- 172.21.15.89:0/1737753777 shutdown_connections 2023-10-30T12:37:43.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:43.118+0000 7f250a70f640 1 --2- 172.21.15.89:0/1737753777 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f24f00664d0 0x7f24f0068990 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:43.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:43.118+0000 7f250a70f640 1 --2- 172.21.15.89:0/1737753777 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2504159140 0x7f25041edcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:43.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:43.118+0000 7f250a70f640 1 --2- 172.21.15.89:0/1737753777 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f250413edd0 0x7f2504155bb0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:43.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:43.118+0000 7f250a70f640 1 --2- 172.21.15.89:0/1737753777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f24ec006f20 0x7f2504153410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:43.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:43.118+0000 7f250a70f640 1 -- 172.21.15.89:0/1737753777 >> 172.21.15.89:0/1737753777 conn(0x7f25040c7910 msgr2=0x7f25040b08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:43.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:43.118+0000 7f250a70f640 1 -- 172.21.15.89:0/1737753777 shutdown_connections 2023-10-30T12:37:43.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:43.118+0000 7f250a70f640 1 -- 172.21.15.89:0/1737753777 wait complete. 2023-10-30T12:37:43.138 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:37:45.140 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:37:45.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.290+0000 7f90237d3640 1 Processor -- start 2023-10-30T12:37:45.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.290+0000 7f90237d3640 1 -- start start 2023-10-30T12:37:45.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.290+0000 7f90237d3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f901c1538a0 0x7f901c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:45.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.290+0000 7f90237d3640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f901c0cf160 0x7f901c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:45.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.290+0000 7f90237d3640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f901c0cb800 con 0x7f901c0ca0e0 2023-10-30T12:37:45.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.290+0000 7f90237d3640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f901c15dc30 con 0x7f901c0cf160 2023-10-30T12:37:45.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.290+0000 7f90237d3640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f901c0cb330 con 0x7f901c1538a0 2023-10-30T12:37:45.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.290+0000 7f9020d6e640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f901c0ca0e0 0x7f901c15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58270/0 (socket says 172.21.15.89:58270) 2023-10-30T12:37:45.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.290+0000 7f9020d6e640 1 -- 172.21.15.89:0/3294690635 learned_addr learned my addr 172.21.15.89:0/3294690635 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:37:45.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.290+0000 7f9021d70640 1 --2- 172.21.15.89:0/3294690635 >> v2:172.21.15.192:3300/0 conn(0x7f901c0cf160 0x7f901c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:45.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.290+0000 7f902156f640 1 --2- 172.21.15.89:0/3294690635 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f901c1538a0 0x7f901c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:45.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.290+0000 7f9013fff640 1 -- 172.21.15.89:0/3294690635 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030745522 0 0) 0x7f901c0cb800 con 0x7f901c0ca0e0 2023-10-30T12:37:45.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.290+0000 7f9013fff640 1 -- 172.21.15.89:0/3294690635 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9008003530 con 0x7f901c0ca0e0 2023-10-30T12:37:45.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.290+0000 7f9013fff640 1 -- 172.21.15.89:0/3294690635 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1796523118 0 0) 0x7f9008003530 con 0x7f901c0ca0e0 2023-10-30T12:37:45.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.290+0000 7f9013fff640 1 -- 172.21.15.89:0/3294690635 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f901c1538a0 msgr2=0x7f901c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:45.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.290+0000 7f9013fff640 1 --2- 172.21.15.89:0/3294690635 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f901c1538a0 0x7f901c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:45.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.290+0000 7f9013fff640 1 -- 172.21.15.89:0/3294690635 >> v2:172.21.15.192:3300/0 conn(0x7f901c0cf160 msgr2=0x7f901c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:45.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.290+0000 7f9013fff640 1 --2- 172.21.15.89:0/3294690635 >> v2:172.21.15.192:3300/0 conn(0x7f901c0cf160 0x7f901c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:45.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.290+0000 7f9013fff640 1 -- 172.21.15.89:0/3294690635 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f901c0cad70 con 0x7f901c0ca0e0 2023-10-30T12:37:45.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.290+0000 7f9013fff640 1 -- 172.21.15.89:0/3294690635 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f900c001680 con 0x7f901c0ca0e0 2023-10-30T12:37:45.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.290+0000 7f9013fff640 1 -- 172.21.15.89:0/3294690635 >> v1:172.21.15.89:6789/0 conn(0x7f901c0ca0e0 legacy=0x7f901c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:45.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.290+0000 7f9013fff640 1 --2- 172.21.15.89:0/3294690635 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f901c1538a0 0x7f9008003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:45.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.290+0000 7f9013fff640 1 --2- 172.21.15.89:0/3294690635 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9008003f00 0x7f90080047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:45.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.290+0000 7f9013fff640 1 --2- 172.21.15.89:0/3294690635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9008004d10 0x7f9008005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:45.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.290+0000 7f9013fff640 1 -- 172.21.15.89:0/3294690635 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901c15dc30 con 0x7f9008004d10 2023-10-30T12:37:45.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.290+0000 7f9013fff640 1 -- 172.21.15.89:0/3294690635 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901c0cb330 con 0x7f9008003f00 2023-10-30T12:37:45.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.290+0000 7f9013fff640 1 -- 172.21.15.89:0/3294690635 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f900800b9c0 con 0x7f901c1538a0 2023-10-30T12:37:45.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.290+0000 7f9021d70640 1 --2- 172.21.15.89:0/3294690635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9008004d10 0x7f9008005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:45.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.290+0000 7f9020d6e640 1 --2- 172.21.15.89:0/3294690635 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9008003f00 0x7f90080047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:45.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.290+0000 7f902156f640 1 --2- 172.21.15.89:0/3294690635 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f901c1538a0 0x7f9008003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:45.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.294+0000 7f9021d70640 1 -- 172.21.15.89:0/3294690635 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f901c1538a0 msgr2=0x7f9008003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:45.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.294+0000 7f9021d70640 1 --2- 172.21.15.89:0/3294690635 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f901c1538a0 0x7f9008003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:45.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.294+0000 7f9021d70640 1 -- 172.21.15.89:0/3294690635 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9008003f00 msgr2=0x7f90080047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:45.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.294+0000 7f9021d70640 1 --2- 172.21.15.89:0/3294690635 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9008003f00 0x7f90080047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:45.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.294+0000 7f9021d70640 1 -- 172.21.15.89:0/3294690635 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9008003530 con 0x7f9008004d10 2023-10-30T12:37:45.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.294+0000 7f9021d70640 1 --2- 172.21.15.89:0/3294690635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9008004d10 0x7f9008005190 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9014009fc0 tx=0x7f901401cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff7fd9f44780f664 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:45.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.294+0000 7f9013fff640 1 -- 172.21.15.89:0/3294690635 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90140a5cb0 con 0x7f9008004d10 2023-10-30T12:37:45.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.294+0000 7f9013fff640 1 -- 172.21.15.89:0/3294690635 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f901400f040 con 0x7f9008004d10 2023-10-30T12:37:45.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.294+0000 7f9013fff640 1 -- 172.21.15.89:0/3294690635 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90140ad680 con 0x7f9008004d10 2023-10-30T12:37:45.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.294+0000 7f90237d3640 1 -- 172.21.15.89:0/3294690635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9008004d10 msgr2=0x7f9008005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:45.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.294+0000 7f90237d3640 1 --2- 172.21.15.89:0/3294690635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9008004d10 0x7f9008005190 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9014009fc0 tx=0x7f901401cca0 comp rx=0 tx=0).stop 2023-10-30T12:37:45.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.294+0000 7f9021d70640 1 -- 172.21.15.89:0/3294690635 reap_dead start 2023-10-30T12:37:45.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.294+0000 7f90237d3640 1 -- 172.21.15.89:0/3294690635 shutdown_connections 2023-10-30T12:37:45.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.294+0000 7f90237d3640 1 -- 172.21.15.89:0/3294690635 >> 172.21.15.89:0/3294690635 conn(0x7f901c06bfe0 msgr2=0x7f901c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:45.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.298+0000 7f90237d3640 1 -- 172.21.15.89:0/3294690635 shutdown_connections 2023-10-30T12:37:45.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.298+0000 7f90237d3640 1 -- 172.21.15.89:0/3294690635 wait complete. 2023-10-30T12:37:45.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.298+0000 7f90237d3640 1 Processor -- start 2023-10-30T12:37:45.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.298+0000 7f90237d3640 1 -- start start 2023-10-30T12:37:45.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.298+0000 7f90237d3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9008004d10 0x7f901c0c6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:45.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.298+0000 7f90237d3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f901c0ce770 0x7f901c15be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:45.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.298+0000 7f90237d3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f901c0cebb0 0x7f901c15c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:45.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.298+0000 7f90237d3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901c15ba70 con 0x7f901c0cebb0 2023-10-30T12:37:45.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.298+0000 7f90237d3640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901c15a8b0 con 0x7f9008004d10 2023-10-30T12:37:45.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.298+0000 7f90237d3640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f901c0c7420 con 0x7f901c0ce770 2023-10-30T12:37:45.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.298+0000 7f9021d70640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f901c0cebb0 0x7f901c15c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:45.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.298+0000 7f9021d70640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f901c0cebb0 0x7f901c15c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60020/0 (socket says 172.21.15.89:60020) 2023-10-30T12:37:45.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.298+0000 7f9021d70640 1 -- 172.21.15.89:0/1079961726 learned_addr learned my addr 172.21.15.89:0/1079961726 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:37:45.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.298+0000 7f902156f640 1 --2- 172.21.15.89:0/1079961726 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9008004d10 0x7f901c0c6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:45.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.298+0000 7f9020d6e640 1 --2- 172.21.15.89:0/1079961726 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f901c0ce770 0x7f901c15be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:45.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.298+0000 7f9021d70640 1 -- 172.21.15.89:0/1079961726 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f901c0ce770 msgr2=0x7f901c15be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:45.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.298+0000 7f9021d70640 1 --2- 172.21.15.89:0/1079961726 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f901c0ce770 0x7f901c15be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:45.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.298+0000 7f9021d70640 1 -- 172.21.15.89:0/1079961726 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9008004d10 msgr2=0x7f901c0c6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:45.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.298+0000 7f9021d70640 1 --2- 172.21.15.89:0/1079961726 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9008004d10 0x7f901c0c6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:45.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.298+0000 7f9021d70640 1 -- 172.21.15.89:0/1079961726 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f901c15b440 con 0x7f901c0cebb0 2023-10-30T12:37:45.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.298+0000 7f9021d70640 1 --2- 172.21.15.89:0/1079961726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f901c0cebb0 0x7f901c15c3d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9014002240 tx=0x7f9014009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:45.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.298+0000 7f9011ffb640 1 -- 172.21.15.89:0/1079961726 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90140039a0 con 0x7f901c0cebb0 2023-10-30T12:37:45.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.298+0000 7f9011ffb640 1 -- 172.21.15.89:0/1079961726 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9014003b00 con 0x7f901c0cebb0 2023-10-30T12:37:45.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.298+0000 7f90237d3640 1 -- 172.21.15.89:0/1079961726 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f901c15aae0 con 0x7f901c0cebb0 2023-10-30T12:37:45.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.302+0000 7f9011ffb640 1 -- 172.21.15.89:0/1079961726 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9014003e90 con 0x7f901c0cebb0 2023-10-30T12:37:45.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.302+0000 7f90237d3640 1 -- 172.21.15.89:0/1079961726 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f901c15b0a0 con 0x7f901c0cebb0 2023-10-30T12:37:45.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.302+0000 7f90237d3640 1 -- 172.21.15.89:0/1079961726 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f901c15bbb0 con 0x7f901c0cebb0 2023-10-30T12:37:45.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.302+0000 7f9011ffb640 1 -- 172.21.15.89:0/1079961726 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f90140b4040 con 0x7f901c0cebb0 2023-10-30T12:37:45.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.302+0000 7f9011ffb640 1 --2- 172.21.15.89:0/1079961726 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f90000664a0 0x7f9000068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:45.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.302+0000 7f9011ffb640 1 -- 172.21.15.89:0/1079961726 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f901411ded0 con 0x7f901c0cebb0 2023-10-30T12:37:45.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.302+0000 7f902156f640 1 --2- 172.21.15.89:0/1079961726 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f90000664a0 0x7f9000068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:45.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.306+0000 7f902156f640 1 --2- 172.21.15.89:0/1079961726 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f90000664a0 0x7f9000068960 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9018009a30 tx=0x7f9018008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:45.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.306+0000 7f9011ffb640 1 -- 172.21.15.89:0/1079961726 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f901c15bbb0 con 0x7f901c0cebb0 2023-10-30T12:37:45.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.490+0000 7f90237d3640 1 -- 172.21.15.89:0/1079961726 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f901c15bbb0 con 0x7f901c0cebb0 2023-10-30T12:37:45.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.490+0000 7f9011ffb640 1 -- 172.21.15.89:0/1079961726 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+5736 (secure 0 0 0) 0x7f901c15bbb0 con 0x7f901c0cebb0 2023-10-30T12:37:45.492 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:37:45.492 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:37:42.240386+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:resolve","state_seq":35,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:37:45.492 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 45 2023-10-30T12:37:45.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.490+0000 7f90237d3640 1 -- 172.21.15.89:0/1079961726 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f90000664a0 msgr2=0x7f9000068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:45.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.490+0000 7f90237d3640 1 --2- 172.21.15.89:0/1079961726 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f90000664a0 0x7f9000068960 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9018009a30 tx=0x7f9018008040 comp rx=0 tx=0).stop 2023-10-30T12:37:45.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.490+0000 7f90237d3640 1 -- 172.21.15.89:0/1079961726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f901c0cebb0 msgr2=0x7f901c15c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:45.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.490+0000 7f90237d3640 1 --2- 172.21.15.89:0/1079961726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f901c0cebb0 0x7f901c15c3d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9014002240 tx=0x7f9014009750 comp rx=0 tx=0).stop 2023-10-30T12:37:45.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.494+0000 7f90237d3640 1 -- 172.21.15.89:0/1079961726 shutdown_connections 2023-10-30T12:37:45.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.494+0000 7f90237d3640 1 --2- 172.21.15.89:0/1079961726 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f90000664a0 0x7f9000068960 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:45.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.494+0000 7f90237d3640 1 --2- 172.21.15.89:0/1079961726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f901c0cebb0 0x7f901c15c3d0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:45.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.494+0000 7f90237d3640 1 --2- 172.21.15.89:0/1079961726 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f901c0ce770 0x7f901c15be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:45.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.494+0000 7f90237d3640 1 --2- 172.21.15.89:0/1079961726 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9008004d10 0x7f901c0c6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:45.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.494+0000 7f90237d3640 1 -- 172.21.15.89:0/1079961726 >> 172.21.15.89:0/1079961726 conn(0x7f901c06bfe0 msgr2=0x7f901c0663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:45.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.494+0000 7f90237d3640 1 -- 172.21.15.89:0/1079961726 shutdown_connections 2023-10-30T12:37:45.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:45.494+0000 7f90237d3640 1 -- 172.21.15.89:0/1079961726 wait complete. 2023-10-30T12:37:45.512 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:37:46.123 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:37:46.127 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:37:47.513 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:37:47.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.658+0000 7f8f55446640 1 Processor -- start 2023-10-30T12:37:47.661 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.658+0000 7f8f55446640 1 -- start start 2023-10-30T12:37:47.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f55446640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f500c6b80 0x7f8f500bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:47.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f55446640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f500c0a50 con 0x7f8f500c6f60 2023-10-30T12:37:47.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f55446640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f5006f6e0 con 0x7f8f5013f4a0 2023-10-30T12:37:47.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f55446640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5006f9c0 con 0x7f8f500c6b80 2023-10-30T12:37:47.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f467fc640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f8f500c6f60 0x7f8f5013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:58276/0 (socket says 172.21.15.89:58276) 2023-10-30T12:37:47.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f467fc640 1 -- 172.21.15.89:0/3089432222 learned_addr learned my addr 172.21.15.89:0/3089432222 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:37:47.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f4effd640 1 --2- 172.21.15.89:0/3089432222 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f500c6b80 0x7f8f500bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:47.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f4e7fc640 1 -- 172.21.15.89:0/3089432222 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2816636233 0 0) 0x7f8f500c0a50 con 0x7f8f500c6f60 2023-10-30T12:37:47.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f4e7fc640 1 -- 172.21.15.89:0/3089432222 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f3c005740 con 0x7f8f500c6f60 2023-10-30T12:37:47.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f4e7fc640 1 -- 172.21.15.89:0/3089432222 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 681202837 0 0) 0x7f8f3c005740 con 0x7f8f500c6f60 2023-10-30T12:37:47.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f4e7fc640 1 -- 172.21.15.89:0/3089432222 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f500c6b80 msgr2=0x7f8f500bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:47.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f4e7fc640 1 --2- 172.21.15.89:0/3089432222 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f500c6b80 0x7f8f500bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:47.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f4e7fc640 1 -- 172.21.15.89:0/3089432222 >> v1:172.21.15.192:6789/0 conn(0x7f8f5013f4a0 legacy=0x7f8f501566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:47.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f4e7fc640 1 -- 172.21.15.89:0/3089432222 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f5006fca0 con 0x7f8f500c6f60 2023-10-30T12:37:47.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f4e7fc640 1 -- 172.21.15.89:0/3089432222 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f8f38001a00 con 0x7f8f500c6f60 2023-10-30T12:37:47.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f4e7fc640 1 -- 172.21.15.89:0/3089432222 >> v1:172.21.15.89:6789/0 conn(0x7f8f500c6f60 legacy=0x7f8f5013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:47.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f4e7fc640 1 --2- 172.21.15.89:0/3089432222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f3c005bc0 0x7f8f3c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:47.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f4e7fc640 1 --2- 172.21.15.89:0/3089432222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f3c006570 0x7f8f3c006a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:47.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f4e7fc640 1 --2- 172.21.15.89:0/3089432222 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f500c6b80 0x7f8f3c0072a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:47.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f4e7fc640 1 -- 172.21.15.89:0/3089432222 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5006f9c0 con 0x7f8f3c005bc0 2023-10-30T12:37:47.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f4e7fc640 1 -- 172.21.15.89:0/3089432222 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3c00dba0 con 0x7f8f3c006570 2023-10-30T12:37:47.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f4e7fc640 1 -- 172.21.15.89:0/3089432222 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3c00dd60 con 0x7f8f500c6b80 2023-10-30T12:37:47.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f4effd640 1 --2- 172.21.15.89:0/3089432222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f3c005bc0 0x7f8f3c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:47.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f467fc640 1 --2- 172.21.15.89:0/3089432222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f3c006570 0x7f8f3c006a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:47.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f4f7fe640 1 --2- 172.21.15.89:0/3089432222 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f500c6b80 0x7f8f3c0072a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:47.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f4effd640 1 -- 172.21.15.89:0/3089432222 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f500c6b80 msgr2=0x7f8f3c0072a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:47.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f4effd640 1 --2- 172.21.15.89:0/3089432222 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f500c6b80 0x7f8f3c0072a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:47.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f4effd640 1 -- 172.21.15.89:0/3089432222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f3c006570 msgr2=0x7f8f3c006a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:47.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f4effd640 1 --2- 172.21.15.89:0/3089432222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f3c006570 0x7f8f3c006a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:47.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f4effd640 1 -- 172.21.15.89:0/3089432222 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3c005740 con 0x7f8f3c005bc0 2023-10-30T12:37:47.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f467fc640 1 --2- 172.21.15.89:0/3089432222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f3c006570 0x7f8f3c006a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:37:47.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.662+0000 7f8f4f7fe640 1 --2- 172.21.15.89:0/3089432222 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f500c6b80 0x7f8f3c0072a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:37:47.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.666+0000 7f8f4effd640 1 --2- 172.21.15.89:0/3089432222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f3c005bc0 0x7f8f3c005fa0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8f40009ae0 tx=0x7f8f4001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b67ebb419a8e409 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:47.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.666+0000 7f8f4e7fc640 1 -- 172.21.15.89:0/3089432222 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f400a5640 con 0x7f8f3c005bc0 2023-10-30T12:37:47.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.666+0000 7f8f4e7fc640 1 -- 172.21.15.89:0/3089432222 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f400a57a0 con 0x7f8f3c005bc0 2023-10-30T12:37:47.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.666+0000 7f8f4e7fc640 1 -- 172.21.15.89:0/3089432222 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f400a5b50 con 0x7f8f3c005bc0 2023-10-30T12:37:47.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.666+0000 7f8f55446640 1 -- 172.21.15.89:0/3089432222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f3c005bc0 msgr2=0x7f8f3c005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:47.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.666+0000 7f8f55446640 1 --2- 172.21.15.89:0/3089432222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f3c005bc0 0x7f8f3c005fa0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8f40009ae0 tx=0x7f8f4001ca50 comp rx=0 tx=0).stop 2023-10-30T12:37:47.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.666+0000 7f8f4f7fe640 1 -- 172.21.15.89:0/3089432222 reap_dead start 2023-10-30T12:37:47.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.666+0000 7f8f55446640 1 -- 172.21.15.89:0/3089432222 shutdown_connections 2023-10-30T12:37:47.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.666+0000 7f8f55446640 1 -- 172.21.15.89:0/3089432222 >> 172.21.15.89:0/3089432222 conn(0x7f8f500c7910 msgr2=0x7f8f500c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:47.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.666+0000 7f8f55446640 1 -- 172.21.15.89:0/3089432222 shutdown_connections 2023-10-30T12:37:47.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.666+0000 7f8f55446640 1 -- 172.21.15.89:0/3089432222 wait complete. 2023-10-30T12:37:47.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.670+0000 7f8f55446640 1 Processor -- start 2023-10-30T12:37:47.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.670+0000 7f8f55446640 1 -- start start 2023-10-30T12:37:47.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.670+0000 7f8f55446640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f3c005bc0 0x7f8f500c18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:47.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.670+0000 7f8f55446640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f501544a0 0x7f8f500c44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:47.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.670+0000 7f8f55446640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f500c2c20 0x7f8f50138390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:47.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.670+0000 7f8f55446640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f501ea550 con 0x7f8f3c005bc0 2023-10-30T12:37:47.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.670+0000 7f8f55446640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f501eb520 con 0x7f8f500c2c20 2023-10-30T12:37:47.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.670+0000 7f8f55446640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f50154970 con 0x7f8f501544a0 2023-10-30T12:37:47.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.670+0000 7f8f4effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f3c005bc0 0x7f8f500c18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:47.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.670+0000 7f8f4effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f3c005bc0 0x7f8f500c18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60032/0 (socket says 172.21.15.89:60032) 2023-10-30T12:37:47.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.670+0000 7f8f4effd640 1 -- 172.21.15.89:0/509407728 learned_addr learned my addr 172.21.15.89:0/509407728 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:37:47.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.670+0000 7f8f467fc640 1 --2- 172.21.15.89:0/509407728 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f501544a0 0x7f8f500c44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:47.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.670+0000 7f8f4f7fe640 1 --2- 172.21.15.89:0/509407728 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f500c2c20 0x7f8f50138390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:47.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.670+0000 7f8f4effd640 1 -- 172.21.15.89:0/509407728 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f501544a0 msgr2=0x7f8f500c44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:47.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.670+0000 7f8f4effd640 1 --2- 172.21.15.89:0/509407728 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f501544a0 0x7f8f500c44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:47.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.670+0000 7f8f4effd640 1 -- 172.21.15.89:0/509407728 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f500c2c20 msgr2=0x7f8f50138390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:47.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.670+0000 7f8f4effd640 1 --2- 172.21.15.89:0/509407728 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f500c2c20 0x7f8f50138390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:47.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.670+0000 7f8f4effd640 1 -- 172.21.15.89:0/509407728 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f501ea760 con 0x7f8f3c005bc0 2023-10-30T12:37:47.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.670+0000 7f8f4effd640 1 --2- 172.21.15.89:0/509407728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f3c005bc0 0x7f8f500c18c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8f4001c880 tx=0x7f8f3c007a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:47.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.670+0000 7f8f47fff640 1 -- 172.21.15.89:0/509407728 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f400bc070 con 0x7f8f3c005bc0 2023-10-30T12:37:47.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.670+0000 7f8f47fff640 1 -- 172.21.15.89:0/509407728 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f40004020 con 0x7f8f3c005bc0 2023-10-30T12:37:47.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.670+0000 7f8f47fff640 1 -- 172.21.15.89:0/509407728 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f400043d0 con 0x7f8f3c005bc0 2023-10-30T12:37:47.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.670+0000 7f8f55446640 1 -- 172.21.15.89:0/509407728 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f501eaa40 con 0x7f8f3c005bc0 2023-10-30T12:37:47.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.670+0000 7f8f55446640 1 -- 172.21.15.89:0/509407728 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f501eb000 con 0x7f8f3c005bc0 2023-10-30T12:37:47.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.670+0000 7f8f55446640 1 -- 172.21.15.89:0/509407728 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f5004e170 con 0x7f8f3c005bc0 2023-10-30T12:37:47.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.674+0000 7f8f47fff640 1 -- 172.21.15.89:0/509407728 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8f400b4040 con 0x7f8f3c005bc0 2023-10-30T12:37:47.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.674+0000 7f8f47fff640 1 --2- 172.21.15.89:0/509407728 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8f300664a0 0x7f8f30068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:47.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.674+0000 7f8f47fff640 1 -- 172.21.15.89:0/509407728 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f8f4011de20 con 0x7f8f3c005bc0 2023-10-30T12:37:47.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.674+0000 7f8f467fc640 1 --2- 172.21.15.89:0/509407728 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8f300664a0 0x7f8f30068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:47.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.674+0000 7f8f467fc640 1 --2- 172.21.15.89:0/509407728 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8f300664a0 0x7f8f30068960 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8f501548a0 tx=0x7f8f38009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:47.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.678+0000 7f8f47fff640 1 -- 172.21.15.89:0/509407728 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8f400eb040 con 0x7f8f3c005bc0 2023-10-30T12:37:47.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.858+0000 7f8f55446640 1 -- 172.21.15.89:0/509407728 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f500b0650 con 0x7f8f3c005bc0 2023-10-30T12:37:47.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.858+0000 7f8f47fff640 1 -- 172.21.15.89:0/509407728 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+5736 (secure 0 0 0) 0x7f8f400ee660 con 0x7f8f3c005bc0 2023-10-30T12:37:47.859 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:37:47.859 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:37:42.240386+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:resolve","state_seq":35,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:37:47.859 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 45 2023-10-30T12:37:47.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.858+0000 7f8f55446640 1 -- 172.21.15.89:0/509407728 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8f300664a0 msgr2=0x7f8f30068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:47.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.858+0000 7f8f55446640 1 --2- 172.21.15.89:0/509407728 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8f300664a0 0x7f8f30068960 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8f501548a0 tx=0x7f8f38009040 comp rx=0 tx=0).stop 2023-10-30T12:37:47.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.858+0000 7f8f55446640 1 -- 172.21.15.89:0/509407728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f3c005bc0 msgr2=0x7f8f500c18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:47.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.858+0000 7f8f55446640 1 --2- 172.21.15.89:0/509407728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f3c005bc0 0x7f8f500c18c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8f4001c880 tx=0x7f8f3c007a10 comp rx=0 tx=0).stop 2023-10-30T12:37:47.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.858+0000 7f8f55446640 1 -- 172.21.15.89:0/509407728 shutdown_connections 2023-10-30T12:37:47.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.858+0000 7f8f55446640 1 --2- 172.21.15.89:0/509407728 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8f300664a0 0x7f8f30068960 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:47.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.858+0000 7f8f55446640 1 --2- 172.21.15.89:0/509407728 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f500c2c20 0x7f8f50138390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:47.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.858+0000 7f8f55446640 1 --2- 172.21.15.89:0/509407728 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f501544a0 0x7f8f500c44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:47.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.858+0000 7f8f55446640 1 --2- 172.21.15.89:0/509407728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f3c005bc0 0x7f8f500c18c0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:47.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.858+0000 7f8f55446640 1 -- 172.21.15.89:0/509407728 >> 172.21.15.89:0/509407728 conn(0x7f8f500c7910 msgr2=0x7f8f501392c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:47.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.858+0000 7f8f55446640 1 -- 172.21.15.89:0/509407728 shutdown_connections 2023-10-30T12:37:47.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:47.858+0000 7f8f55446640 1 -- 172.21.15.89:0/509407728 wait complete. 2023-10-30T12:37:47.880 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:37:49.881 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:37:50.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.030+0000 7fdce2a46640 1 Processor -- start 2023-10-30T12:37:50.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.030+0000 7fdce2a46640 1 -- start start 2023-10-30T12:37:50.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.030+0000 7fdce2a46640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fdcdc15d6f0 0x7fdcdc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:50.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.030+0000 7fdce2a46640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdcdc0cf160 0x7fdcdc0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:50.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.030+0000 7fdce2a46640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcdc0cb800 con 0x7fdcdc1538a0 2023-10-30T12:37:50.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.030+0000 7fdce2a46640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcdc0cb330 con 0x7fdcdc0cf160 2023-10-30T12:37:50.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.030+0000 7fdce2a46640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcdc0cad70 con 0x7fdcdc15d6f0 2023-10-30T12:37:50.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.030+0000 7fdce0fe3640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fdcdc0cf160 0x7fdcdc0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:50.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.030+0000 7fdcdbfff640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fdcdc1538a0 0x7fdcdc0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33922/0 (socket says 172.21.15.89:33922) 2023-10-30T12:37:50.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.030+0000 7fdcdbfff640 1 -- 172.21.15.89:0/636183992 learned_addr learned my addr 172.21.15.89:0/636183992 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:37:50.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.030+0000 7fdcd3fff640 1 --2- 172.21.15.89:0/636183992 >> v2:172.21.15.192:3300/0 conn(0x7fdcdc15d6f0 0x7fdcdc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:50.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.030+0000 7fdce0fe3640 1 -- 172.21.15.89:0/636183992 >> v2:172.21.15.192:3300/0 conn(0x7fdcdc15d6f0 msgr2=0x7fdcdc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:50.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.030+0000 7fdce0fe3640 1 --2- 172.21.15.89:0/636183992 >> v2:172.21.15.192:3300/0 conn(0x7fdcdc15d6f0 0x7fdcdc15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:50.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.030+0000 7fdce0fe3640 1 -- 172.21.15.89:0/636183992 >> v1:172.21.15.89:6789/0 conn(0x7fdcdc1538a0 legacy=0x7fdcdc0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:50.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.030+0000 7fdce0fe3640 1 -- 172.21.15.89:0/636183992 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcdc0cb050 con 0x7fdcdc0cf160 2023-10-30T12:37:50.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.030+0000 7fdcdb7fe640 1 -- 172.21.15.89:0/636183992 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611545334 0 0) 0x7fdcdc0cb800 con 0x7fdcdc1538a0 2023-10-30T12:37:50.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.034+0000 7fdce0fe3640 1 --2- 172.21.15.89:0/636183992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcdc0cf160 0x7fdcdc0cf590 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc0b30a0 tx=0x7fdcd401f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=223b255ba937455c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:50.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.034+0000 7fdcdb7fe640 1 -- 172.21.15.89:0/636183992 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd40b0440 con 0x7fdcdc0cf160 2023-10-30T12:37:50.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.034+0000 7fdcdb7fe640 1 -- 172.21.15.89:0/636183992 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcd40b05a0 con 0x7fdcdc0cf160 2023-10-30T12:37:50.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.034+0000 7fdcdb7fe640 1 -- 172.21.15.89:0/636183992 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd40b0930 con 0x7fdcdc0cf160 2023-10-30T12:37:50.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.034+0000 7fdce2a46640 1 -- 172.21.15.89:0/636183992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcdc0cf160 msgr2=0x7fdcdc0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:50.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.034+0000 7fdce2a46640 1 --2- 172.21.15.89:0/636183992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcdc0cf160 0x7fdcdc0cf590 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc0b30a0 tx=0x7fdcd401f0f0 comp rx=0 tx=0).stop 2023-10-30T12:37:50.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.034+0000 7fdce2a46640 1 -- 172.21.15.89:0/636183992 shutdown_connections 2023-10-30T12:37:50.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.034+0000 7fdce2a46640 1 --2- 172.21.15.89:0/636183992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcdc0cf160 0x7fdcdc0cf590 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:50.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.034+0000 7fdce2a46640 1 --2- 172.21.15.89:0/636183992 >> v2:172.21.15.192:3300/0 conn(0x7fdcdc15d6f0 0x7fdcdc15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:50.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.034+0000 7fdce2a46640 1 -- 172.21.15.89:0/636183992 >> 172.21.15.89:0/636183992 conn(0x7fdcdc06bfe0 msgr2=0x7fdcdc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:50.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.034+0000 7fdce2a46640 1 -- 172.21.15.89:0/636183992 shutdown_connections 2023-10-30T12:37:50.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.034+0000 7fdce2a46640 1 -- 172.21.15.89:0/636183992 wait complete. 2023-10-30T12:37:50.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.038+0000 7fdce2a46640 1 Processor -- start 2023-10-30T12:37:50.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.038+0000 7fdce2a46640 1 -- start start 2023-10-30T12:37:50.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.038+0000 7fdce2a46640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcdc0cf160 0x7fdcdc1504e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:50.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.038+0000 7fdce2a46640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcdc1538a0 0x7fdcdc155df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:50.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.038+0000 7fdce2a46640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcdc15d6f0 0x7fdcdc156330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:50.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.038+0000 7fdce2a46640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcdc14d700 con 0x7fdcdc15d6f0 2023-10-30T12:37:50.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.038+0000 7fdce2a46640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcdc1544f0 con 0x7fdcdc0cf160 2023-10-30T12:37:50.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.038+0000 7fdce2a46640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcdc153f90 con 0x7fdcdc1538a0 2023-10-30T12:37:50.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.038+0000 7fdce0fe3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcdc15d6f0 0x7fdcdc156330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:50.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.038+0000 7fdce0fe3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcdc15d6f0 0x7fdcdc156330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38254/0 (socket says 172.21.15.89:38254) 2023-10-30T12:37:50.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.038+0000 7fdce0fe3640 1 -- 172.21.15.89:0/2016108977 learned_addr learned my addr 172.21.15.89:0/2016108977 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:37:50.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.038+0000 7fdcdbfff640 1 --2- 172.21.15.89:0/2016108977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcdc0cf160 0x7fdcdc1504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:50.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.038+0000 7fdcd3fff640 1 --2- 172.21.15.89:0/2016108977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcdc1538a0 0x7fdcdc155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:50.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.038+0000 7fdce0fe3640 1 -- 172.21.15.89:0/2016108977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcdc1538a0 msgr2=0x7fdcdc155df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:50.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.038+0000 7fdce0fe3640 1 --2- 172.21.15.89:0/2016108977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcdc1538a0 0x7fdcdc155df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:50.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.038+0000 7fdce0fe3640 1 -- 172.21.15.89:0/2016108977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcdc0cf160 msgr2=0x7fdcdc1504e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:50.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.038+0000 7fdce0fe3640 1 --2- 172.21.15.89:0/2016108977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcdc0cf160 0x7fdcdc1504e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:50.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.038+0000 7fdce0fe3640 1 -- 172.21.15.89:0/2016108977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd4009a70 con 0x7fdcdc15d6f0 2023-10-30T12:37:50.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.038+0000 7fdcd3fff640 1 --2- 172.21.15.89:0/2016108977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcdc1538a0 0x7fdcdc155df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:37:50.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.038+0000 7fdce0fe3640 1 --2- 172.21.15.89:0/2016108977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcdc15d6f0 0x7fdcdc156330 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fdcd401bd00 tx=0x7fdcd401be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:50.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.038+0000 7fdcd97fa640 1 -- 172.21.15.89:0/2016108977 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd40b0440 con 0x7fdcdc15d6f0 2023-10-30T12:37:50.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.038+0000 7fdcd97fa640 1 -- 172.21.15.89:0/2016108977 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcd40bc860 con 0x7fdcdc15d6f0 2023-10-30T12:37:50.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.038+0000 7fdce2a46640 1 -- 172.21.15.89:0/2016108977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcdc154230 con 0x7fdcdc15d6f0 2023-10-30T12:37:50.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.038+0000 7fdcd97fa640 1 -- 172.21.15.89:0/2016108977 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd40bcc10 con 0x7fdcdc15d6f0 2023-10-30T12:37:50.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.038+0000 7fdce2a46640 1 -- 172.21.15.89:0/2016108977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcdc1590b0 con 0x7fdcdc15d6f0 2023-10-30T12:37:50.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.042+0000 7fdce2a46640 1 -- 172.21.15.89:0/2016108977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcdc0b2a50 con 0x7fdcdc15d6f0 2023-10-30T12:37:50.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.042+0000 7fdcd97fa640 1 -- 172.21.15.89:0/2016108977 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdcd40c7460 con 0x7fdcdc15d6f0 2023-10-30T12:37:50.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.042+0000 7fdcd97fa640 1 --2- 172.21.15.89:0/2016108977 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fdcbc0664a0 0x7fdcbc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:50.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.042+0000 7fdcd97fa640 1 -- 172.21.15.89:0/2016108977 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fdcd4122e60 con 0x7fdcdc15d6f0 2023-10-30T12:37:50.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.042+0000 7fdcdbfff640 1 --2- 172.21.15.89:0/2016108977 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fdcbc0664a0 0x7fdcbc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:50.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.042+0000 7fdcdbfff640 1 --2- 172.21.15.89:0/2016108977 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fdcbc0664a0 0x7fdcbc068960 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdccc000f70 tx=0x7fdccc007b60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:50.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.046+0000 7fdcd97fa640 1 -- 172.21.15.89:0/2016108977 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdcd40efff0 con 0x7fdcdc15d6f0 2023-10-30T12:37:50.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.226+0000 7fdce2a46640 1 -- 172.21.15.89:0/2016108977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdcdc155af0 con 0x7fdcdc15d6f0 2023-10-30T12:37:50.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.226+0000 7fdcd97fa640 1 -- 172.21.15.89:0/2016108977 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+5738 (secure 0 0 0) 0x7fdcd40b5020 con 0x7fdcdc15d6f0 2023-10-30T12:37:50.228 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:37:50.228 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:37:48.259529+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":37,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:37:50.229 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 46 2023-10-30T12:37:50.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.230+0000 7fdce2a46640 1 -- 172.21.15.89:0/2016108977 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fdcbc0664a0 msgr2=0x7fdcbc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:50.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.230+0000 7fdce2a46640 1 --2- 172.21.15.89:0/2016108977 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fdcbc0664a0 0x7fdcbc068960 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdccc000f70 tx=0x7fdccc007b60 comp rx=0 tx=0).stop 2023-10-30T12:37:50.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.230+0000 7fdce2a46640 1 -- 172.21.15.89:0/2016108977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcdc15d6f0 msgr2=0x7fdcdc156330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:50.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.230+0000 7fdce2a46640 1 --2- 172.21.15.89:0/2016108977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcdc15d6f0 0x7fdcdc156330 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fdcd401bd00 tx=0x7fdcd401be90 comp rx=0 tx=0).stop 2023-10-30T12:37:50.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.230+0000 7fdce2a46640 1 -- 172.21.15.89:0/2016108977 shutdown_connections 2023-10-30T12:37:50.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.230+0000 7fdce2a46640 1 --2- 172.21.15.89:0/2016108977 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fdcbc0664a0 0x7fdcbc068960 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:50.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.230+0000 7fdce2a46640 1 --2- 172.21.15.89:0/2016108977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcdc15d6f0 0x7fdcdc156330 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:50.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.230+0000 7fdce2a46640 1 --2- 172.21.15.89:0/2016108977 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcdc1538a0 0x7fdcdc155df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:50.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.230+0000 7fdce2a46640 1 --2- 172.21.15.89:0/2016108977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcdc0cf160 0x7fdcdc1504e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:50.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.230+0000 7fdce2a46640 1 -- 172.21.15.89:0/2016108977 >> 172.21.15.89:0/2016108977 conn(0x7fdcdc06bfe0 msgr2=0x7fdcdc0c49f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:50.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.230+0000 7fdce2a46640 1 -- 172.21.15.89:0/2016108977 shutdown_connections 2023-10-30T12:37:50.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:50.230+0000 7fdce2a46640 1 -- 172.21.15.89:0/2016108977 wait complete. 2023-10-30T12:37:50.249 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 46, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:50.729245+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 46, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4781": { "addr": "172.21.15.192:6835/654256299", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6834", "nonce": 654256299, "type": "v2" }, { "addr": "172.21.15.192:6835", "nonce": 654256299, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4781, "incarnation": 24, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 27 }, "gid_5018": { "addr": "172.21.15.192:6839/93914587", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6838", "nonce": 93914587, "type": "v2" }, { "addr": "172.21.15.192:6839", "nonce": 93914587, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5018, "incarnation": 44, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 37 }, "gid_5139": { "addr": "172.21.15.89:6837/951276252", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6836", "nonce": 951276252, "type": "v2" }, { "addr": "172.21.15.89:6837", "nonce": 951276252, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5139, "incarnation": 33, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 29 } }, "last_failure": 0, "last_failure_osd_epoch": 55, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:37:48.259529+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5018, "mds_1": 5139, "mds_2": 4781 } } } ], "standbys": [ { "addr": "172.21.15.89:6834/2909790173", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6832", "nonce": 2909790173, "type": "v2" }, { "addr": "172.21.15.89:6834", "nonce": 2909790173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.89:6835/2387357615", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6833", "nonce": 2387357615, "type": "v2" }, { "addr": "172.21.15.89:6835", "nonce": 2387357615, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4239, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:37:52.251 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:37:52.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.398+0000 7f0fcc6c9640 1 Processor -- start 2023-10-30T12:37:52.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.398+0000 7f0fcc6c9640 1 -- start start 2023-10-30T12:37:52.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.402+0000 7f0fcc6c9640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0fc40c6b80 0x7f0fc40bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:52.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.402+0000 7f0fcc6c9640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0fc40c6f60 0x7f0fc40c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:52.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.402+0000 7f0fcc6c9640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fc40c0a50 con 0x7f0fc40c8430 2023-10-30T12:37:52.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.402+0000 7f0fcc6c9640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fc406f6e0 con 0x7f0fc40c6b80 2023-10-30T12:37:52.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.402+0000 7f0fcc6c9640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fc406f9c0 con 0x7f0fc40c6f60 2023-10-30T12:37:52.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.402+0000 7f0fca465640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0fc40c6b80 0x7f0fc40bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:52.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.402+0000 7f0fca465640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0fc40c6b80 0x7f0fc40bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38266/0 (socket says 172.21.15.89:38266) 2023-10-30T12:37:52.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.402+0000 7f0fca465640 1 -- 172.21.15.89:0/1498238418 learned_addr learned my addr 172.21.15.89:0/1498238418 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:37:52.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.402+0000 7f0fc9c64640 1 --2- 172.21.15.89:0/1498238418 >> v2:172.21.15.192:3300/0 conn(0x7f0fc40c6f60 0x7f0fc40c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:52.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.402+0000 7f0fca465640 1 -- 172.21.15.89:0/1498238418 >> v2:172.21.15.192:3300/0 conn(0x7f0fc40c6f60 msgr2=0x7f0fc40c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:52.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.402+0000 7f0fca465640 1 --2- 172.21.15.89:0/1498238418 >> v2:172.21.15.192:3300/0 conn(0x7f0fc40c6f60 0x7f0fc40c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:52.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.402+0000 7f0fca465640 1 -- 172.21.15.89:0/1498238418 >> v1:172.21.15.89:6789/0 conn(0x7f0fc40c8430 legacy=0x7f0fc413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:52.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.402+0000 7f0fca465640 1 -- 172.21.15.89:0/1498238418 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc406fca0 con 0x7f0fc40c6b80 2023-10-30T12:37:52.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.402+0000 7f0fc9463640 1 -- 172.21.15.89:0/1498238418 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2459508448 0 0) 0x7f0fc40c0a50 con 0x7f0fc40c8430 2023-10-30T12:37:52.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.406+0000 7f0fca465640 1 --2- 172.21.15.89:0/1498238418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fc40c6b80 0x7f0fc40bfec0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4159b90 tx=0x7f0fc001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=558112de43f8ae1f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:52.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.406+0000 7f0fc9463640 1 -- 172.21.15.89:0/1498238418 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc00ab610 con 0x7f0fc40c6b80 2023-10-30T12:37:52.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.406+0000 7f0fc9463640 1 -- 172.21.15.89:0/1498238418 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc00ab770 con 0x7f0fc40c6b80 2023-10-30T12:37:52.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.406+0000 7f0fc9463640 1 -- 172.21.15.89:0/1498238418 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc00abb00 con 0x7f0fc40c6b80 2023-10-30T12:37:52.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.406+0000 7f0fcc6c9640 1 -- 172.21.15.89:0/1498238418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fc40c6b80 msgr2=0x7f0fc40bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:52.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.406+0000 7f0fcc6c9640 1 --2- 172.21.15.89:0/1498238418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fc40c6b80 0x7f0fc40bfec0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4159b90 tx=0x7f0fc001c500 comp rx=0 tx=0).stop 2023-10-30T12:37:52.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.406+0000 7f0fcc6c9640 1 -- 172.21.15.89:0/1498238418 shutdown_connections 2023-10-30T12:37:52.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.406+0000 7f0fcc6c9640 1 --2- 172.21.15.89:0/1498238418 >> v2:172.21.15.192:3300/0 conn(0x7f0fc40c6f60 0x7f0fc40c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:52.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.406+0000 7f0fcc6c9640 1 --2- 172.21.15.89:0/1498238418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fc40c6b80 0x7f0fc40bfec0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:52.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.406+0000 7f0fcc6c9640 1 -- 172.21.15.89:0/1498238418 >> 172.21.15.89:0/1498238418 conn(0x7f0fc40c7910 msgr2=0x7f0fc40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:52.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.406+0000 7f0fcc6c9640 1 -- 172.21.15.89:0/1498238418 shutdown_connections 2023-10-30T12:37:52.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.406+0000 7f0fcc6c9640 1 -- 172.21.15.89:0/1498238418 wait complete. 2023-10-30T12:37:52.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.406+0000 7f0fcc6c9640 1 Processor -- start 2023-10-30T12:37:52.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.406+0000 7f0fcc6c9640 1 -- start start 2023-10-30T12:37:52.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.410+0000 7f0fcc6c9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0fc40c6b80 0x7f0fc40c9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:52.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.410+0000 7f0fcc6c9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fc40c6f60 0x7f0fc4154eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:52.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.410+0000 7f0fcc6c9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fc40c8430 0x7f0fc41f4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:52.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.410+0000 7f0fcc6c9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc40c92d0 con 0x7f0fc40c8430 2023-10-30T12:37:52.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.410+0000 7f0fcc6c9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc41553f0 con 0x7f0fc40c6f60 2023-10-30T12:37:52.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.410+0000 7f0fcc6c9640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc41f5bd0 con 0x7f0fc40c6b80 2023-10-30T12:37:52.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.410+0000 7f0fcac66640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fc40c8430 0x7f0fc41f4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:52.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.410+0000 7f0fcac66640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fc40c8430 0x7f0fc41f4f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38282/0 (socket says 172.21.15.89:38282) 2023-10-30T12:37:52.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.410+0000 7f0fcac66640 1 -- 172.21.15.89:0/1626544862 learned_addr learned my addr 172.21.15.89:0/1626544862 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:37:52.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.410+0000 7f0fca465640 1 --2- 172.21.15.89:0/1626544862 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0fc40c6b80 0x7f0fc40c9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:52.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.410+0000 7f0fc9c64640 1 --2- 172.21.15.89:0/1626544862 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fc40c6f60 0x7f0fc4154eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:52.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.410+0000 7f0fcac66640 1 -- 172.21.15.89:0/1626544862 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0fc40c6b80 msgr2=0x7f0fc40c9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:52.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.410+0000 7f0fcac66640 1 --2- 172.21.15.89:0/1626544862 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0fc40c6b80 0x7f0fc40c9dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:52.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.410+0000 7f0fcac66640 1 -- 172.21.15.89:0/1626544862 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fc40c6f60 msgr2=0x7f0fc4154eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:52.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.410+0000 7f0fcac66640 1 --2- 172.21.15.89:0/1626544862 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fc40c6f60 0x7f0fc4154eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:52.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.410+0000 7f0fcac66640 1 -- 172.21.15.89:0/1626544862 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc00094d0 con 0x7f0fc40c8430 2023-10-30T12:37:52.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.410+0000 7f0fcac66640 1 --2- 172.21.15.89:0/1626544862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fc40c8430 0x7f0fc41f4f90 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc00c780 tx=0x7f0fbc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:52.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.410+0000 7f0fbaffd640 1 -- 172.21.15.89:0/1626544862 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc003db0 con 0x7f0fc40c8430 2023-10-30T12:37:52.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.410+0000 7f0fbaffd640 1 -- 172.21.15.89:0/1626544862 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fbc00ce30 con 0x7f0fc40c8430 2023-10-30T12:37:52.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.410+0000 7f0fbaffd640 1 -- 172.21.15.89:0/1626544862 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc0052b0 con 0x7f0fc40c8430 2023-10-30T12:37:52.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.410+0000 7f0fcc6c9640 1 -- 172.21.15.89:0/1626544862 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fc40c8ff0 con 0x7f0fc40c8430 2023-10-30T12:37:52.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.410+0000 7f0fcc6c9640 1 -- 172.21.15.89:0/1626544862 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fc40c9780 con 0x7f0fc40c8430 2023-10-30T12:37:52.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.410+0000 7f0fcc6c9640 1 -- 172.21.15.89:0/1626544862 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fc404e170 con 0x7f0fc40c8430 2023-10-30T12:37:52.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.410+0000 7f0fbaffd640 1 -- 172.21.15.89:0/1626544862 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0fbc005410 con 0x7f0fc40c8430 2023-10-30T12:37:52.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.414+0000 7f0fbaffd640 1 --2- 172.21.15.89:0/1626544862 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0fa80664a0 0x7f0fa8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:52.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.414+0000 7f0fbaffd640 1 -- 172.21.15.89:0/1626544862 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f0fbc083ef0 con 0x7f0fc40c8430 2023-10-30T12:37:52.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.414+0000 7f0fca465640 1 --2- 172.21.15.89:0/1626544862 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0fa80664a0 0x7f0fa8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:52.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.414+0000 7f0fca465640 1 --2- 172.21.15.89:0/1626544862 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0fa80664a0 0x7f0fa8068960 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0fc00af000 tx=0x7f0fc00b6040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:52.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.418+0000 7f0fbaffd640 1 -- 172.21.15.89:0/1626544862 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0fbc051080 con 0x7f0fc40c8430 2023-10-30T12:37:52.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.598+0000 7f0fcc6c9640 1 -- 172.21.15.89:0/1626544862 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0fc406adb0 con 0x7f0fc40c8430 2023-10-30T12:37:52.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.598+0000 7f0fbaffd640 1 -- 172.21.15.89:0/1626544862 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f0fbc0546a0 con 0x7f0fc40c8430 2023-10-30T12:37:52.599 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:37:52.599 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:37:48.259529+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":37,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:37:52.600 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 46 2023-10-30T12:37:52.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.598+0000 7f0fcc6c9640 1 -- 172.21.15.89:0/1626544862 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0fa80664a0 msgr2=0x7f0fa8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:52.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.598+0000 7f0fcc6c9640 1 --2- 172.21.15.89:0/1626544862 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0fa80664a0 0x7f0fa8068960 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0fc00af000 tx=0x7f0fc00b6040 comp rx=0 tx=0).stop 2023-10-30T12:37:52.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.598+0000 7f0fcc6c9640 1 -- 172.21.15.89:0/1626544862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fc40c8430 msgr2=0x7f0fc41f4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:52.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.598+0000 7f0fcc6c9640 1 --2- 172.21.15.89:0/1626544862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fc40c8430 0x7f0fc41f4f90 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc00c780 tx=0x7f0fbc00cc40 comp rx=0 tx=0).stop 2023-10-30T12:37:52.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.598+0000 7f0fcc6c9640 1 -- 172.21.15.89:0/1626544862 shutdown_connections 2023-10-30T12:37:52.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.598+0000 7f0fcc6c9640 1 --2- 172.21.15.89:0/1626544862 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0fa80664a0 0x7f0fa8068960 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:52.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.598+0000 7f0fcc6c9640 1 --2- 172.21.15.89:0/1626544862 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fc40c6f60 0x7f0fc4154eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:52.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.598+0000 7f0fcc6c9640 1 --2- 172.21.15.89:0/1626544862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fc40c8430 0x7f0fc41f4f90 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:52.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.598+0000 7f0fcc6c9640 1 --2- 172.21.15.89:0/1626544862 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0fc40c6b80 0x7f0fc40c9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:52.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.598+0000 7f0fcc6c9640 1 -- 172.21.15.89:0/1626544862 >> 172.21.15.89:0/1626544862 conn(0x7f0fc40c7910 msgr2=0x7f0fc40c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:52.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.598+0000 7f0fcc6c9640 1 -- 172.21.15.89:0/1626544862 shutdown_connections 2023-10-30T12:37:52.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:52.598+0000 7f0fcc6c9640 1 -- 172.21.15.89:0/1626544862 wait complete. 2023-10-30T12:37:52.619 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:37:54.620 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:37:54.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.766+0000 7f176e965640 1 Processor -- start 2023-10-30T12:37:54.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.766+0000 7f176e965640 1 -- start start 2023-10-30T12:37:54.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f176e965640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17680c6b80 0x7f17680bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:54.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f176e965640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17680c0a50 con 0x7f17680c6f60 2023-10-30T12:37:54.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f176e965640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f176806f6e0 con 0x7f176813f4a0 2023-10-30T12:37:54.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f176e965640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f176806f9c0 con 0x7f17680c6b80 2023-10-30T12:37:54.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f17677fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f17680c6f60 0x7f176813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33936/0 (socket says 172.21.15.89:33936) 2023-10-30T12:37:54.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f17677fe640 1 -- 172.21.15.89:0/2828052144 learned_addr learned my addr 172.21.15.89:0/2828052144 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:37:54.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f1767fff640 1 --2- 172.21.15.89:0/2828052144 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17680c6b80 0x7f17680bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:54.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f1766ffd640 1 -- 172.21.15.89:0/2828052144 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1017761666 0 0) 0x7f17680c0a50 con 0x7f17680c6f60 2023-10-30T12:37:54.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f1766ffd640 1 -- 172.21.15.89:0/2828052144 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1750003530 con 0x7f17680c6f60 2023-10-30T12:37:54.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f1766ffd640 1 -- 172.21.15.89:0/2828052144 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3019193517 0 0) 0x7f1750003530 con 0x7f17680c6f60 2023-10-30T12:37:54.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f1766ffd640 1 -- 172.21.15.89:0/2828052144 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17680c6b80 msgr2=0x7f17680bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:54.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f1766ffd640 1 --2- 172.21.15.89:0/2828052144 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17680c6b80 0x7f17680bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:54.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f1766ffd640 1 -- 172.21.15.89:0/2828052144 >> v1:172.21.15.192:6789/0 conn(0x7f176813f4a0 legacy=0x7f17681566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:54.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f1766ffd640 1 -- 172.21.15.89:0/2828052144 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f176806fca0 con 0x7f17680c6f60 2023-10-30T12:37:54.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f1766ffd640 1 -- 172.21.15.89:0/2828052144 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f175c001680 con 0x7f17680c6f60 2023-10-30T12:37:54.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f1766ffd640 1 -- 172.21.15.89:0/2828052144 >> v1:172.21.15.89:6789/0 conn(0x7f17680c6f60 legacy=0x7f176813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:54.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f1766ffd640 1 --2- 172.21.15.89:0/2828052144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17500039b0 0x7f1750003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:54.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f1766ffd640 1 --2- 172.21.15.89:0/2828052144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1750004360 0x7f1750004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:54.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f1766ffd640 1 --2- 172.21.15.89:0/2828052144 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17680c6b80 0x7f1750005090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:54.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f1766ffd640 1 -- 172.21.15.89:0/2828052144 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176806f9c0 con 0x7f17500039b0 2023-10-30T12:37:54.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f1766ffd640 1 -- 172.21.15.89:0/2828052144 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f175000b990 con 0x7f1750004360 2023-10-30T12:37:54.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f1766ffd640 1 -- 172.21.15.89:0/2828052144 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f175000dd60 con 0x7f17680c6b80 2023-10-30T12:37:54.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f1767fff640 1 --2- 172.21.15.89:0/2828052144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17500039b0 0x7f1750003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:54.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f176cf02640 1 --2- 172.21.15.89:0/2828052144 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17680c6b80 0x7f1750005090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:54.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f17677fe640 1 --2- 172.21.15.89:0/2828052144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1750004360 0x7f1750004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:54.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f1767fff640 1 -- 172.21.15.89:0/2828052144 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17680c6b80 msgr2=0x7f1750005090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:54.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f1767fff640 1 --2- 172.21.15.89:0/2828052144 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17680c6b80 0x7f1750005090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:54.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f1767fff640 1 -- 172.21.15.89:0/2828052144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1750004360 msgr2=0x7f1750004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:54.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f1767fff640 1 --2- 172.21.15.89:0/2828052144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1750004360 0x7f1750004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:54.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f1767fff640 1 -- 172.21.15.89:0/2828052144 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1750003530 con 0x7f17500039b0 2023-10-30T12:37:54.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.770+0000 7f17677fe640 1 --2- 172.21.15.89:0/2828052144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1750004360 0x7f1750004830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:37:54.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.774+0000 7f1767fff640 1 --2- 172.21.15.89:0/2828052144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17500039b0 0x7f1750003d90 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f1758009ae0 tx=0x7f175801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fc8a9c1bee06ede server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:54.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.774+0000 7f1766ffd640 1 -- 172.21.15.89:0/2828052144 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17580a5640 con 0x7f17500039b0 2023-10-30T12:37:54.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.774+0000 7f1766ffd640 1 -- 172.21.15.89:0/2828052144 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17580a57a0 con 0x7f17500039b0 2023-10-30T12:37:54.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.774+0000 7f1766ffd640 1 -- 172.21.15.89:0/2828052144 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17580a5b50 con 0x7f17500039b0 2023-10-30T12:37:54.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.774+0000 7f176e965640 1 -- 172.21.15.89:0/2828052144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17500039b0 msgr2=0x7f1750003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:54.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.774+0000 7f176e965640 1 --2- 172.21.15.89:0/2828052144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17500039b0 0x7f1750003d90 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f1758009ae0 tx=0x7f175801ca50 comp rx=0 tx=0).stop 2023-10-30T12:37:54.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.774+0000 7f176cf02640 1 -- 172.21.15.89:0/2828052144 reap_dead start 2023-10-30T12:37:54.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.774+0000 7f176e965640 1 -- 172.21.15.89:0/2828052144 shutdown_connections 2023-10-30T12:37:54.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.774+0000 7f176e965640 1 -- 172.21.15.89:0/2828052144 >> 172.21.15.89:0/2828052144 conn(0x7f17680c7910 msgr2=0x7f17680c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:54.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.774+0000 7f176e965640 1 -- 172.21.15.89:0/2828052144 shutdown_connections 2023-10-30T12:37:54.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.774+0000 7f176e965640 1 -- 172.21.15.89:0/2828052144 wait complete. 2023-10-30T12:37:54.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.774+0000 7f176e965640 1 Processor -- start 2023-10-30T12:37:54.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.778+0000 7f176e965640 1 -- start start 2023-10-30T12:37:54.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.778+0000 7f176e965640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17500039b0 0x7f17680c0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:54.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.778+0000 7f176e965640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17681e38d0 0x7f1768157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:54.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.778+0000 7f176e965640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1768153c50 0x7f1768157cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:54.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.778+0000 7f176e965640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1768158ce0 con 0x7f1768153c50 2023-10-30T12:37:54.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.778+0000 7f176e965640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17680c98c0 con 0x7f17500039b0 2023-10-30T12:37:54.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.778+0000 7f176e965640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17680c8cb0 con 0x7f17681e38d0 2023-10-30T12:37:54.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.778+0000 7f176cf02640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1768153c50 0x7f1768157cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:54.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.778+0000 7f176cf02640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1768153c50 0x7f1768157cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38300/0 (socket says 172.21.15.89:38300) 2023-10-30T12:37:54.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.778+0000 7f176cf02640 1 -- 172.21.15.89:0/2447736145 learned_addr learned my addr 172.21.15.89:0/2447736145 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:37:54.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.778+0000 7f17677fe640 1 --2- 172.21.15.89:0/2447736145 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17681e38d0 0x7f1768157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:54.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.778+0000 7f1767fff640 1 --2- 172.21.15.89:0/2447736145 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17500039b0 0x7f17680c0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:54.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.778+0000 7f176cf02640 1 -- 172.21.15.89:0/2447736145 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17681e38d0 msgr2=0x7f1768157790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:54.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.778+0000 7f176cf02640 1 --2- 172.21.15.89:0/2447736145 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17681e38d0 0x7f1768157790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:54.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.778+0000 7f176cf02640 1 -- 172.21.15.89:0/2447736145 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17500039b0 msgr2=0x7f17680c0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:54.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.778+0000 7f176cf02640 1 --2- 172.21.15.89:0/2447736145 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17500039b0 0x7f17680c0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:54.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.778+0000 7f176cf02640 1 -- 172.21.15.89:0/2447736145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17680c8f90 con 0x7f1768153c50 2023-10-30T12:37:54.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.778+0000 7f176cf02640 1 --2- 172.21.15.89:0/2447736145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1768153c50 0x7f1768157cd0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f176813f8d0 tx=0x7f1760010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:54.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.778+0000 7f1764ff9640 1 -- 172.21.15.89:0/2447736145 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1760003d90 con 0x7f1768153c50 2023-10-30T12:37:54.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.778+0000 7f1764ff9640 1 -- 172.21.15.89:0/2447736145 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f176000d070 con 0x7f1768153c50 2023-10-30T12:37:54.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.778+0000 7f1764ff9640 1 -- 172.21.15.89:0/2447736145 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f176000e9b0 con 0x7f1768153c50 2023-10-30T12:37:54.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.778+0000 7f176e965640 1 -- 172.21.15.89:0/2447736145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17680c9270 con 0x7f1768153c50 2023-10-30T12:37:54.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.778+0000 7f176e965640 1 -- 172.21.15.89:0/2447736145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17680caa10 con 0x7f1768153c50 2023-10-30T12:37:54.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.778+0000 7f176e965640 1 -- 172.21.15.89:0/2447736145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f176804e170 con 0x7f1768153c50 2023-10-30T12:37:54.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.782+0000 7f1764ff9640 1 -- 172.21.15.89:0/2447736145 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f176000eb10 con 0x7f1768153c50 2023-10-30T12:37:54.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.782+0000 7f1764ff9640 1 --2- 172.21.15.89:0/2447736145 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f17540664a0 0x7f1754068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:54.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.782+0000 7f1764ff9640 1 -- 172.21.15.89:0/2447736145 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f1760087180 con 0x7f1768153c50 2023-10-30T12:37:54.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.782+0000 7f1767fff640 1 --2- 172.21.15.89:0/2447736145 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f17540664a0 0x7f1754068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:54.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.782+0000 7f1767fff640 1 --2- 172.21.15.89:0/2447736145 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f17540664a0 0x7f1754068960 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1758009ae0 tx=0x7f17580046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:54.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.786+0000 7f1764ff9640 1 -- 172.21.15.89:0/2447736145 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1760054310 con 0x7f1768153c50 2023-10-30T12:37:54.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.966+0000 7f176e965640 1 -- 172.21.15.89:0/2447736145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17680c9a00 con 0x7f1768153c50 2023-10-30T12:37:54.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.966+0000 7f1764ff9640 1 -- 172.21.15.89:0/2447736145 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f1760057930 con 0x7f1768153c50 2023-10-30T12:37:54.967 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:37:54.967 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:37:48.259529+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":37,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:37:54.967 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 46 2023-10-30T12:37:54.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.966+0000 7f176e965640 1 -- 172.21.15.89:0/2447736145 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f17540664a0 msgr2=0x7f1754068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:54.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.966+0000 7f176e965640 1 --2- 172.21.15.89:0/2447736145 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f17540664a0 0x7f1754068960 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1758009ae0 tx=0x7f17580046d0 comp rx=0 tx=0).stop 2023-10-30T12:37:54.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.966+0000 7f176e965640 1 -- 172.21.15.89:0/2447736145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1768153c50 msgr2=0x7f1768157cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:54.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.966+0000 7f176e965640 1 --2- 172.21.15.89:0/2447736145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1768153c50 0x7f1768157cd0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f176813f8d0 tx=0x7f1760010e90 comp rx=0 tx=0).stop 2023-10-30T12:37:54.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.966+0000 7f176e965640 1 -- 172.21.15.89:0/2447736145 shutdown_connections 2023-10-30T12:37:54.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.966+0000 7f176e965640 1 --2- 172.21.15.89:0/2447736145 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f17540664a0 0x7f1754068960 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:54.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.966+0000 7f176e965640 1 --2- 172.21.15.89:0/2447736145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1768153c50 0x7f1768157cd0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:54.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.966+0000 7f176e965640 1 --2- 172.21.15.89:0/2447736145 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17681e38d0 0x7f1768157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:54.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.966+0000 7f176e965640 1 --2- 172.21.15.89:0/2447736145 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17500039b0 0x7f17680c0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:54.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.966+0000 7f176e965640 1 -- 172.21.15.89:0/2447736145 >> 172.21.15.89:0/2447736145 conn(0x7f17680c7910 msgr2=0x7f176813a630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:54.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.966+0000 7f176e965640 1 -- 172.21.15.89:0/2447736145 shutdown_connections 2023-10-30T12:37:54.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:54.966+0000 7f176e965640 1 -- 172.21.15.89:0/2447736145 wait complete. 2023-10-30T12:37:54.989 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:37:56.990 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:37:57.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.138+0000 7fa389e91640 1 Processor -- start 2023-10-30T12:37:57.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.138+0000 7fa389e91640 1 -- start start 2023-10-30T12:37:57.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.138+0000 7fa389e91640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa3841538a0 0x7fa38415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:57.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.138+0000 7fa389e91640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3840cb800 con 0x7fa3840ca0e0 2023-10-30T12:37:57.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.138+0000 7fa389e91640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa38415dc30 con 0x7fa3840cf160 2023-10-30T12:37:57.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.138+0000 7fa389e91640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3840cb330 con 0x7fa3841538a0 2023-10-30T12:37:57.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.138+0000 7fa382ffd640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa3840ca0e0 0x7fa38415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33946/0 (socket says 172.21.15.89:33946) 2023-10-30T12:37:57.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.138+0000 7fa382ffd640 1 -- 172.21.15.89:0/2120199254 learned_addr learned my addr 172.21.15.89:0/2120199254 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:37:57.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.138+0000 7fa3837fe640 1 --2- 172.21.15.89:0/2120199254 >> v2:172.21.15.192:3300/0 conn(0x7fa3841538a0 0x7fa38415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:57.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.138+0000 7fa3827fc640 1 -- 172.21.15.89:0/2120199254 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 813634616 0 0) 0x7fa3840cb800 con 0x7fa3840ca0e0 2023-10-30T12:37:57.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.138+0000 7fa3827fc640 1 -- 172.21.15.89:0/2120199254 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa36c003530 con 0x7fa3840ca0e0 2023-10-30T12:37:57.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.138+0000 7fa3827fc640 1 -- 172.21.15.89:0/2120199254 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 910137947 0 0) 0x7fa36c003530 con 0x7fa3840ca0e0 2023-10-30T12:37:57.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.138+0000 7fa3827fc640 1 -- 172.21.15.89:0/2120199254 >> v2:172.21.15.192:3300/0 conn(0x7fa3841538a0 msgr2=0x7fa38415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:57.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.138+0000 7fa3827fc640 1 --2- 172.21.15.89:0/2120199254 >> v2:172.21.15.192:3300/0 conn(0x7fa3841538a0 0x7fa38415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:57.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.138+0000 7fa3827fc640 1 -- 172.21.15.89:0/2120199254 >> v1:172.21.15.192:6789/0 conn(0x7fa3840cf160 legacy=0x7fa384164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:57.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.138+0000 7fa3827fc640 1 -- 172.21.15.89:0/2120199254 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3840cad70 con 0x7fa3840ca0e0 2023-10-30T12:37:57.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.138+0000 7fa3827fc640 1 -- 172.21.15.89:0/2120199254 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7fa378001680 con 0x7fa3840ca0e0 2023-10-30T12:37:57.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.138+0000 7fa3827fc640 1 -- 172.21.15.89:0/2120199254 >> v1:172.21.15.89:6789/0 conn(0x7fa3840ca0e0 legacy=0x7fa38415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:57.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.138+0000 7fa3827fc640 1 --2- 172.21.15.89:0/2120199254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa36c003980 0x7fa36c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:57.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.138+0000 7fa3827fc640 1 --2- 172.21.15.89:0/2120199254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa36c0047d0 0x7fa36c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:57.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.138+0000 7fa3827fc640 1 --2- 172.21.15.89:0/2120199254 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa36c005190 0x7fa36c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:57.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.138+0000 7fa3827fc640 1 -- 172.21.15.89:0/2120199254 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3840cb330 con 0x7fa36c003980 2023-10-30T12:37:57.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.138+0000 7fa3827fc640 1 -- 172.21.15.89:0/2120199254 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c00c040 con 0x7fa36c0047d0 2023-10-30T12:37:57.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.138+0000 7fa3827fc640 1 -- 172.21.15.89:0/2120199254 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c00c200 con 0x7fa36c005190 2023-10-30T12:37:57.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.138+0000 7fa3837fe640 1 --2- 172.21.15.89:0/2120199254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa36c003980 0x7fa36c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:57.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.138+0000 7fa382ffd640 1 --2- 172.21.15.89:0/2120199254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa36c0047d0 0x7fa36c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:57.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.142+0000 7fa383fff640 1 --2- 172.21.15.89:0/2120199254 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa36c005190 0x7fa36c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:57.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.142+0000 7fa3837fe640 1 -- 172.21.15.89:0/2120199254 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa36c005190 msgr2=0x7fa36c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:57.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.142+0000 7fa3837fe640 1 --2- 172.21.15.89:0/2120199254 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa36c005190 0x7fa36c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:57.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.142+0000 7fa3837fe640 1 -- 172.21.15.89:0/2120199254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa36c0047d0 msgr2=0x7fa36c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:57.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.142+0000 7fa3837fe640 1 --2- 172.21.15.89:0/2120199254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa36c0047d0 0x7fa36c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:57.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.142+0000 7fa3837fe640 1 -- 172.21.15.89:0/2120199254 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa36c003530 con 0x7fa36c003980 2023-10-30T12:37:57.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.142+0000 7fa383fff640 1 -- 172.21.15.89:0/2120199254 reap_dead start 2023-10-30T12:37:57.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.142+0000 7fa3837fe640 1 --2- 172.21.15.89:0/2120199254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa36c003980 0x7fa36c003da0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fa374009ae0 tx=0x7fa37401ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8bf1e4b93c21e9b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:57.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.142+0000 7fa3827fc640 1 -- 172.21.15.89:0/2120199254 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3740a5550 con 0x7fa36c003980 2023-10-30T12:37:57.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.142+0000 7fa3827fc640 1 -- 172.21.15.89:0/2120199254 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3740a56b0 con 0x7fa36c003980 2023-10-30T12:37:57.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.142+0000 7fa3827fc640 1 -- 172.21.15.89:0/2120199254 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3740a5a60 con 0x7fa36c003980 2023-10-30T12:37:57.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.142+0000 7fa389e91640 1 -- 172.21.15.89:0/2120199254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa36c003980 msgr2=0x7fa36c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:57.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.142+0000 7fa389e91640 1 --2- 172.21.15.89:0/2120199254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa36c003980 0x7fa36c003da0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fa374009ae0 tx=0x7fa37401ca70 comp rx=0 tx=0).stop 2023-10-30T12:37:57.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.142+0000 7fa389e91640 1 -- 172.21.15.89:0/2120199254 shutdown_connections 2023-10-30T12:37:57.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.142+0000 7fa389e91640 1 --2- 172.21.15.89:0/2120199254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa36c003980 0x7fa36c003da0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:57.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.142+0000 7fa389e91640 1 -- 172.21.15.89:0/2120199254 >> 172.21.15.89:0/2120199254 conn(0x7fa38406bfe0 msgr2=0x7fa38406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:57.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.142+0000 7fa389e91640 1 -- 172.21.15.89:0/2120199254 shutdown_connections 2023-10-30T12:37:57.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.142+0000 7fa389e91640 1 -- 172.21.15.89:0/2120199254 wait complete. 2023-10-30T12:37:57.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.146+0000 7fa389e91640 1 Processor -- start 2023-10-30T12:37:57.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.146+0000 7fa389e91640 1 -- start start 2023-10-30T12:37:57.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.146+0000 7fa389e91640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa36c003980 0x7fa3840ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:57.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.146+0000 7fa389e91640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3840cebe0 0x7fa3840b4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:57.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.146+0000 7fa389e91640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3841f3130 0x7fa3841409f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:57.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.146+0000 7fa389e91640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3840b4830 con 0x7fa3840cebe0 2023-10-30T12:37:57.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.146+0000 7fa389e91640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3841f3ae0 con 0x7fa36c003980 2023-10-30T12:37:57.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.146+0000 7fa389e91640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3841f3da0 con 0x7fa3841f3130 2023-10-30T12:37:57.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.146+0000 7fa382ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3840cebe0 0x7fa3840b4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:57.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.146+0000 7fa3837fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa36c003980 0x7fa3840ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:57.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.146+0000 7fa382ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3840cebe0 0x7fa3840b4cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38322/0 (socket says 172.21.15.89:38322) 2023-10-30T12:37:57.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.146+0000 7fa3837fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa36c003980 0x7fa3840ccfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:34188/0 (socket says 172.21.15.89:34188) 2023-10-30T12:37:57.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.146+0000 7fa382ffd640 1 -- 172.21.15.89:0/3409240703 learned_addr learned my addr 172.21.15.89:0/3409240703 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:37:57.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.146+0000 7fa383fff640 1 --2- 172.21.15.89:0/3409240703 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3841f3130 0x7fa3841409f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:57.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.146+0000 7fa382ffd640 1 -- 172.21.15.89:0/3409240703 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3841f3130 msgr2=0x7fa3841409f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:57.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.146+0000 7fa382ffd640 1 --2- 172.21.15.89:0/3409240703 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3841f3130 0x7fa3841409f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:57.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.146+0000 7fa382ffd640 1 -- 172.21.15.89:0/3409240703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa36c003980 msgr2=0x7fa3840ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:57.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.146+0000 7fa382ffd640 1 --2- 172.21.15.89:0/3409240703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa36c003980 0x7fa3840ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:57.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.146+0000 7fa382ffd640 1 -- 172.21.15.89:0/3409240703 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3840b32b0 con 0x7fa3840cebe0 2023-10-30T12:37:57.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.146+0000 7fa382ffd640 1 --2- 172.21.15.89:0/3409240703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3840cebe0 0x7fa3840b4cc0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fa3780117f0 tx=0x7fa378011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:57.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.150+0000 7fa36bfff640 1 -- 172.21.15.89:0/3409240703 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37800fb60 con 0x7fa3840cebe0 2023-10-30T12:37:57.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.150+0000 7fa36bfff640 1 -- 172.21.15.89:0/3409240703 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa37800fcc0 con 0x7fa3840cebe0 2023-10-30T12:37:57.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.150+0000 7fa36bfff640 1 -- 172.21.15.89:0/3409240703 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa378015630 con 0x7fa3840cebe0 2023-10-30T12:37:57.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.150+0000 7fa389e91640 1 -- 172.21.15.89:0/3409240703 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3840b44c0 con 0x7fa3840cebe0 2023-10-30T12:37:57.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.150+0000 7fa389e91640 1 -- 172.21.15.89:0/3409240703 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa384150bb0 con 0x7fa3840cebe0 2023-10-30T12:37:57.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.150+0000 7fa389e91640 1 -- 172.21.15.89:0/3409240703 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3840b4680 con 0x7fa3840cebe0 2023-10-30T12:37:57.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.150+0000 7fa36bfff640 1 -- 172.21.15.89:0/3409240703 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa378015790 con 0x7fa3840cebe0 2023-10-30T12:37:57.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.150+0000 7fa36bfff640 1 --2- 172.21.15.89:0/3409240703 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa3700664d0 0x7fa370068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:57.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.150+0000 7fa36bfff640 1 -- 172.21.15.89:0/3409240703 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6148+0+0 (secure 0 0 0) 0x7fa3780871e0 con 0x7fa3840cebe0 2023-10-30T12:37:57.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.150+0000 7fa3837fe640 1 --2- 172.21.15.89:0/3409240703 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa3700664d0 0x7fa370068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:57.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.154+0000 7fa3837fe640 1 --2- 172.21.15.89:0/3409240703 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa3700664d0 0x7fa370068990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa3740af000 tx=0x7fa3740046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:57.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.154+0000 7fa36bfff640 1 -- 172.21.15.89:0/3409240703 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa378054490 con 0x7fa3840cebe0 2023-10-30T12:37:57.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.338+0000 7fa389e91640 1 -- 172.21.15.89:0/3409240703 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa38404e170 con 0x7fa3840cebe0 2023-10-30T12:37:57.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.338+0000 7fa36bfff640 1 -- 172.21.15.89:0/3409240703 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+5735 (secure 0 0 0) 0x7fa378057ab0 con 0x7fa3840cebe0 2023-10-30T12:37:57.338 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:37:57.339 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:37:54.262375+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:37:57.339 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 47 2023-10-30T12:37:57.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.338+0000 7fa389e91640 1 -- 172.21.15.89:0/3409240703 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa3700664d0 msgr2=0x7fa370068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:57.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.338+0000 7fa389e91640 1 --2- 172.21.15.89:0/3409240703 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa3700664d0 0x7fa370068990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa3740af000 tx=0x7fa3740046d0 comp rx=0 tx=0).stop 2023-10-30T12:37:57.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.338+0000 7fa389e91640 1 -- 172.21.15.89:0/3409240703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3840cebe0 msgr2=0x7fa3840b4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:57.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.338+0000 7fa389e91640 1 --2- 172.21.15.89:0/3409240703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3840cebe0 0x7fa3840b4cc0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fa3780117f0 tx=0x7fa378011cb0 comp rx=0 tx=0).stop 2023-10-30T12:37:57.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.338+0000 7fa389e91640 1 -- 172.21.15.89:0/3409240703 shutdown_connections 2023-10-30T12:37:57.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.338+0000 7fa389e91640 1 --2- 172.21.15.89:0/3409240703 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa3700664d0 0x7fa370068990 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:57.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.338+0000 7fa389e91640 1 --2- 172.21.15.89:0/3409240703 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa3841f3130 0x7fa3841409f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:57.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.338+0000 7fa389e91640 1 --2- 172.21.15.89:0/3409240703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa3840cebe0 0x7fa3840b4cc0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:57.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.338+0000 7fa389e91640 1 --2- 172.21.15.89:0/3409240703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa36c003980 0x7fa3840ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:57.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.338+0000 7fa389e91640 1 -- 172.21.15.89:0/3409240703 >> 172.21.15.89:0/3409240703 conn(0x7fa38406bfe0 msgr2=0x7fa384142d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:57.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.338+0000 7fa389e91640 1 -- 172.21.15.89:0/3409240703 shutdown_connections 2023-10-30T12:37:57.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:57.338+0000 7fa389e91640 1 -- 172.21.15.89:0/3409240703 wait complete. 2023-10-30T12:37:57.359 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:37:59.360 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:37:59.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.506+0000 7ffa18b9d640 1 Processor -- start 2023-10-30T12:37:59.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.506+0000 7ffa18b9d640 1 -- start start 2023-10-30T12:37:59.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa18b9d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffa14153bf0 0x7ffa14071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:59.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa18b9d640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ffa140c6fe0 0x7ffa140c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:59.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa18b9d640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa140700b0 con 0x7ffa140c79e0 2023-10-30T12:37:59.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa18b9d640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa140c6810 con 0x7ffa140c6fe0 2023-10-30T12:37:59.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa18b9d640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa14159900 con 0x7ffa14153bf0 2023-10-30T12:37:59.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa12d9d640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ffa140c79e0 0x7ffa140cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33950/0 (socket says 172.21.15.89:33950) 2023-10-30T12:37:59.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa12d9d640 1 -- 172.21.15.89:0/2036056618 learned_addr learned my addr 172.21.15.89:0/2036056618 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:37:59.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa1259c640 1 --2- 172.21.15.89:0/2036056618 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffa14153bf0 0x7ffa14071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:59.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa09d9b640 1 --2- 172.21.15.89:0/2036056618 >> v2:172.21.15.192:3300/0 conn(0x7ffa140c6fe0 0x7ffa140c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:59.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa11d9b640 1 -- 172.21.15.89:0/2036056618 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3778619715 0 0) 0x7ffa140700b0 con 0x7ffa140c79e0 2023-10-30T12:37:59.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa11d9b640 1 -- 172.21.15.89:0/2036056618 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa00003530 con 0x7ffa140c79e0 2023-10-30T12:37:59.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa11d9b640 1 -- 172.21.15.89:0/2036056618 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 950235551 0 0) 0x7ffa00003530 con 0x7ffa140c79e0 2023-10-30T12:37:59.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa11d9b640 1 -- 172.21.15.89:0/2036056618 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffa14153bf0 msgr2=0x7ffa14071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:59.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa11d9b640 1 --2- 172.21.15.89:0/2036056618 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffa14153bf0 0x7ffa14071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:59.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa11d9b640 1 -- 172.21.15.89:0/2036056618 >> v2:172.21.15.192:3300/0 conn(0x7ffa140c6fe0 msgr2=0x7ffa140c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:59.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa11d9b640 1 --2- 172.21.15.89:0/2036056618 >> v2:172.21.15.192:3300/0 conn(0x7ffa140c6fe0 0x7ffa140c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:59.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa11d9b640 1 -- 172.21.15.89:0/2036056618 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa140c6530 con 0x7ffa140c79e0 2023-10-30T12:37:59.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa11d9b640 1 -- 172.21.15.89:0/2036056618 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7ffa0c001a00 con 0x7ffa140c79e0 2023-10-30T12:37:59.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa11d9b640 1 -- 172.21.15.89:0/2036056618 >> v1:172.21.15.89:6789/0 conn(0x7ffa140c79e0 legacy=0x7ffa140cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:59.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa11d9b640 1 --2- 172.21.15.89:0/2036056618 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffa14153bf0 0x7ffa00003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:59.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa11d9b640 1 --2- 172.21.15.89:0/2036056618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa00003f00 0x7ffa000047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:59.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa11d9b640 1 --2- 172.21.15.89:0/2036056618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa00004d10 0x7ffa00005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:59.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa11d9b640 1 -- 172.21.15.89:0/2036056618 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa140c6810 con 0x7ffa00003f00 2023-10-30T12:37:59.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa11d9b640 1 -- 172.21.15.89:0/2036056618 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa14159900 con 0x7ffa00004d10 2023-10-30T12:37:59.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa11d9b640 1 -- 172.21.15.89:0/2036056618 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0000b9c0 con 0x7ffa14153bf0 2023-10-30T12:37:59.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa09d9b640 1 --2- 172.21.15.89:0/2036056618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa00003f00 0x7ffa000047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:59.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa1259c640 1 --2- 172.21.15.89:0/2036056618 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffa14153bf0 0x7ffa00003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:59.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa12d9d640 1 --2- 172.21.15.89:0/2036056618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa00004d10 0x7ffa00005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:59.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa09d9b640 1 -- 172.21.15.89:0/2036056618 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffa14153bf0 msgr2=0x7ffa00003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:59.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa09d9b640 1 --2- 172.21.15.89:0/2036056618 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffa14153bf0 0x7ffa00003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:59.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa09d9b640 1 -- 172.21.15.89:0/2036056618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa00004d10 msgr2=0x7ffa00005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:59.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa09d9b640 1 --2- 172.21.15.89:0/2036056618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa00004d10 0x7ffa00005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:59.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.510+0000 7ffa09d9b640 1 -- 172.21.15.89:0/2036056618 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa00003530 con 0x7ffa00003f00 2023-10-30T12:37:59.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.514+0000 7ffa09d9b640 1 --2- 172.21.15.89:0/2036056618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa00003f00 0x7ffa000047d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ffa04009ae0 tx=0x7ffa0401ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3523df0eb7affcb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:59.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.514+0000 7ffa11d9b640 1 -- 172.21.15.89:0/2036056618 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa040a5680 con 0x7ffa00003f00 2023-10-30T12:37:59.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.514+0000 7ffa11d9b640 1 -- 172.21.15.89:0/2036056618 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa040a57e0 con 0x7ffa00003f00 2023-10-30T12:37:59.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.514+0000 7ffa11d9b640 1 -- 172.21.15.89:0/2036056618 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa040a5b90 con 0x7ffa00003f00 2023-10-30T12:37:59.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.514+0000 7ffa18b9d640 1 -- 172.21.15.89:0/2036056618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa00003f00 msgr2=0x7ffa000047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:59.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.514+0000 7ffa18b9d640 1 --2- 172.21.15.89:0/2036056618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa00003f00 0x7ffa000047d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ffa04009ae0 tx=0x7ffa0401ca90 comp rx=0 tx=0).stop 2023-10-30T12:37:59.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.514+0000 7ffa12d9d640 1 -- 172.21.15.89:0/2036056618 reap_dead start 2023-10-30T12:37:59.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.514+0000 7ffa18b9d640 1 -- 172.21.15.89:0/2036056618 shutdown_connections 2023-10-30T12:37:59.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.514+0000 7ffa18b9d640 1 -- 172.21.15.89:0/2036056618 >> 172.21.15.89:0/2036056618 conn(0x7ffa1406c6e0 msgr2=0x7ffa14069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:59.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.514+0000 7ffa18b9d640 1 -- 172.21.15.89:0/2036056618 shutdown_connections 2023-10-30T12:37:59.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.514+0000 7ffa18b9d640 1 -- 172.21.15.89:0/2036056618 wait complete. 2023-10-30T12:37:59.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.518+0000 7ffa18b9d640 1 Processor -- start 2023-10-30T12:37:59.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.518+0000 7ffa18b9d640 1 -- start start 2023-10-30T12:37:59.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.518+0000 7ffa18b9d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffa00003f00 0x7ffa140c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:59.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.518+0000 7ffa18b9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa140cadf0 0x7ffa141563b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:59.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.518+0000 7ffa18b9d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa141568f0 0x7ffa141f07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:59.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.518+0000 7ffa18b9d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa141f31c0 con 0x7ffa140cadf0 2023-10-30T12:37:59.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.518+0000 7ffa18b9d640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa141f03a0 con 0x7ffa141568f0 2023-10-30T12:37:59.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.518+0000 7ffa18b9d640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa140ca5d0 con 0x7ffa00003f00 2023-10-30T12:37:59.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.518+0000 7ffa09d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa140cadf0 0x7ffa141563b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:59.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.518+0000 7ffa09d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa140cadf0 0x7ffa141563b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38354/0 (socket says 172.21.15.89:38354) 2023-10-30T12:37:59.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.518+0000 7ffa09d9b640 1 -- 172.21.15.89:0/3258033034 learned_addr learned my addr 172.21.15.89:0/3258033034 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:37:59.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.518+0000 7ffa12d9d640 1 --2- 172.21.15.89:0/3258033034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa141568f0 0x7ffa141f07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:59.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.518+0000 7ffa1259c640 1 --2- 172.21.15.89:0/3258033034 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffa00003f00 0x7ffa140c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:59.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.518+0000 7ffa09d9b640 1 -- 172.21.15.89:0/3258033034 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffa00003f00 msgr2=0x7ffa140c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:59.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.518+0000 7ffa09d9b640 1 --2- 172.21.15.89:0/3258033034 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffa00003f00 0x7ffa140c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:59.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.518+0000 7ffa09d9b640 1 -- 172.21.15.89:0/3258033034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa141568f0 msgr2=0x7ffa141f07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:59.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.518+0000 7ffa09d9b640 1 --2- 172.21.15.89:0/3258033034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa141568f0 0x7ffa141f07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:59.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.518+0000 7ffa09d9b640 1 -- 172.21.15.89:0/3258033034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa14156d70 con 0x7ffa140cadf0 2023-10-30T12:37:59.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.518+0000 7ffa12d9d640 1 --2- 172.21.15.89:0/3258033034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa141568f0 0x7ffa141f07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:37:59.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.518+0000 7ffa09d9b640 1 --2- 172.21.15.89:0/3258033034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa140cadf0 0x7ffa141563b0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ffa04009ae0 tx=0x7ffa00005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:59.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.518+0000 7ffa0b7fe640 1 -- 172.21.15.89:0/3258033034 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa040be070 con 0x7ffa140cadf0 2023-10-30T12:37:59.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.518+0000 7ffa0b7fe640 1 -- 172.21.15.89:0/3258033034 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa04004020 con 0x7ffa140cadf0 2023-10-30T12:37:59.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.518+0000 7ffa18b9d640 1 -- 172.21.15.89:0/3258033034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa141f2640 con 0x7ffa140cadf0 2023-10-30T12:37:59.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.518+0000 7ffa0b7fe640 1 -- 172.21.15.89:0/3258033034 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa04004390 con 0x7ffa140cadf0 2023-10-30T12:37:59.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.518+0000 7ffa18b9d640 1 -- 172.21.15.89:0/3258033034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa141f2c00 con 0x7ffa140cadf0 2023-10-30T12:37:59.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.522+0000 7ffa18b9d640 1 -- 172.21.15.89:0/3258033034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa141f04e0 con 0x7ffa140cadf0 2023-10-30T12:37:59.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.522+0000 7ffa0b7fe640 1 -- 172.21.15.89:0/3258033034 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffa040b4040 con 0x7ffa140cadf0 2023-10-30T12:37:59.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.522+0000 7ffa0b7fe640 1 --2- 172.21.15.89:0/3258033034 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff9f40664d0 0x7ff9f4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:59.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.522+0000 7ffa0b7fe640 1 -- 172.21.15.89:0/3258033034 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6148+0+0 (secure 0 0 0) 0x7ffa0411e2e0 con 0x7ffa140cadf0 2023-10-30T12:37:59.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.522+0000 7ffa1259c640 1 --2- 172.21.15.89:0/3258033034 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff9f40664d0 0x7ff9f4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:59.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.522+0000 7ffa1259c640 1 --2- 172.21.15.89:0/3258033034 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff9f40664d0 0x7ff9f4068990 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc009950 tx=0x7ff9fc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:59.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.526+0000 7ffa0b7fe640 1 -- 172.21.15.89:0/3258033034 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffa141f04e0 con 0x7ffa140cadf0 2023-10-30T12:37:59.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.706+0000 7ffa18b9d640 1 -- 172.21.15.89:0/3258033034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa141f04e0 con 0x7ffa140cadf0 2023-10-30T12:37:59.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.706+0000 7ffa0b7fe640 1 -- 172.21.15.89:0/3258033034 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+5735 (secure 0 0 0) 0x7ffa141f04e0 con 0x7ffa140cadf0 2023-10-30T12:37:59.707 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:37:59.707 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:37:54.262375+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:37:59.707 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 47 2023-10-30T12:37:59.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.706+0000 7ffa18b9d640 1 -- 172.21.15.89:0/3258033034 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff9f40664d0 msgr2=0x7ff9f4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:59.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.706+0000 7ffa18b9d640 1 --2- 172.21.15.89:0/3258033034 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff9f40664d0 0x7ff9f4068990 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc009950 tx=0x7ff9fc008040 comp rx=0 tx=0).stop 2023-10-30T12:37:59.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.706+0000 7ffa18b9d640 1 -- 172.21.15.89:0/3258033034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa140cadf0 msgr2=0x7ffa141563b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:59.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.706+0000 7ffa18b9d640 1 --2- 172.21.15.89:0/3258033034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa140cadf0 0x7ffa141563b0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ffa04009ae0 tx=0x7ffa00005800 comp rx=0 tx=0).stop 2023-10-30T12:37:59.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.706+0000 7ffa18b9d640 1 -- 172.21.15.89:0/3258033034 shutdown_connections 2023-10-30T12:37:59.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.706+0000 7ffa18b9d640 1 --2- 172.21.15.89:0/3258033034 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff9f40664d0 0x7ff9f4068990 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:59.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.706+0000 7ffa18b9d640 1 --2- 172.21.15.89:0/3258033034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa141568f0 0x7ffa141f07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:59.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.706+0000 7ffa18b9d640 1 --2- 172.21.15.89:0/3258033034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa140cadf0 0x7ffa141563b0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:59.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.706+0000 7ffa18b9d640 1 --2- 172.21.15.89:0/3258033034 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffa00003f00 0x7ffa140c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:59.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.706+0000 7ffa18b9d640 1 -- 172.21.15.89:0/3258033034 >> 172.21.15.89:0/3258033034 conn(0x7ffa1406c6e0 msgr2=0x7ffa14158580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:59.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.706+0000 7ffa18b9d640 1 -- 172.21.15.89:0/3258033034 shutdown_connections 2023-10-30T12:37:59.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:37:59.706+0000 7ffa18b9d640 1 -- 172.21.15.89:0/3258033034 wait complete. 2023-10-30T12:37:59.727 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:38:01.728 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:38:01.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.874+0000 7f8c999c4640 1 Processor -- start 2023-10-30T12:38:01.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.874+0000 7f8c999c4640 1 -- start start 2023-10-30T12:38:01.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.878+0000 7f8c999c4640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8c94071110 0x7f8c9406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:01.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.878+0000 7f8c999c4640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c94066cf0 con 0x7f8c9406f7a0 2023-10-30T12:38:01.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.878+0000 7f8c999c4640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c940c70d0 con 0x7f8c940c6bd0 2023-10-30T12:38:01.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.878+0000 7f8c999c4640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c940700c0 con 0x7f8c94071110 2023-10-30T12:38:01.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.878+0000 7f8c92ffd640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f8c9406f7a0 0x7f8c94069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33372/0 (socket says 172.21.15.89:33372) 2023-10-30T12:38:01.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.878+0000 7f8c92ffd640 1 -- 172.21.15.89:0/1863141924 learned_addr learned my addr 172.21.15.89:0/1863141924 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:38:01.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.878+0000 7f8c937fe640 1 --2- 172.21.15.89:0/1863141924 >> v2:172.21.15.89:3300/0 conn(0x7f8c94071110 0x7f8c9406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:01.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.878+0000 7f8c927fc640 1 -- 172.21.15.89:0/1863141924 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524347211 0 0) 0x7f8c94066cf0 con 0x7f8c9406f7a0 2023-10-30T12:38:01.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.878+0000 7f8c927fc640 1 -- 172.21.15.89:0/1863141924 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c80003530 con 0x7f8c9406f7a0 2023-10-30T12:38:01.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.878+0000 7f8c937fe640 1 -- 172.21.15.89:0/1863141924 >> v1:172.21.15.192:6789/0 conn(0x7f8c940c6bd0 legacy=0x7f8c94158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:01.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.878+0000 7f8c937fe640 1 -- 172.21.15.89:0/1863141924 >> v1:172.21.15.89:6789/0 conn(0x7f8c9406f7a0 legacy=0x7f8c94069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:01.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.878+0000 7f8c937fe640 1 -- 172.21.15.89:0/1863141924 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c940c7390 con 0x7f8c94071110 2023-10-30T12:38:01.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.882+0000 7f8c937fe640 1 --2- 172.21.15.89:0/1863141924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c94071110 0x7f8c9406fb80 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c009d90 tx=0x7f8c8c01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a87f54fb5d4dbb94 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:01.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.882+0000 7f8c927fc640 1 -- 172.21.15.89:0/1863141924 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8c0af600 con 0x7f8c94071110 2023-10-30T12:38:01.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.882+0000 7f8c927fc640 1 -- 172.21.15.89:0/1863141924 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c8c0ab050 con 0x7f8c94071110 2023-10-30T12:38:01.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.882+0000 7f8c927fc640 1 -- 172.21.15.89:0/1863141924 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8c0af980 con 0x7f8c94071110 2023-10-30T12:38:01.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.882+0000 7f8c999c4640 1 -- 172.21.15.89:0/1863141924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c94071110 msgr2=0x7f8c9406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:01.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.882+0000 7f8c999c4640 1 --2- 172.21.15.89:0/1863141924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c94071110 0x7f8c9406fb80 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c009d90 tx=0x7f8c8c01f130 comp rx=0 tx=0).stop 2023-10-30T12:38:01.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.882+0000 7f8c999c4640 1 -- 172.21.15.89:0/1863141924 shutdown_connections 2023-10-30T12:38:01.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.882+0000 7f8c999c4640 1 --2- 172.21.15.89:0/1863141924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c94071110 0x7f8c9406fb80 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:01.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.882+0000 7f8c999c4640 1 -- 172.21.15.89:0/1863141924 >> 172.21.15.89:0/1863141924 conn(0x7f8c940c7910 msgr2=0x7f8c940c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:01.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.882+0000 7f8c999c4640 1 -- 172.21.15.89:0/1863141924 shutdown_connections 2023-10-30T12:38:01.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.882+0000 7f8c999c4640 1 -- 172.21.15.89:0/1863141924 wait complete. 2023-10-30T12:38:01.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.882+0000 7f8c999c4640 1 Processor -- start 2023-10-30T12:38:01.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.882+0000 7f8c999c4640 1 -- start start 2023-10-30T12:38:01.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.886+0000 7f8c999c4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c9406f7a0 0x7f8c94154b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:01.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.886+0000 7f8c999c4640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c94071110 0x7f8c94152a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:01.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.886+0000 7f8c999c4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c940c6bd0 0x7f8c940c5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:01.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.886+0000 7f8c999c4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c941f1f00 con 0x7f8c9406f7a0 2023-10-30T12:38:01.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.886+0000 7f8c999c4640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c94155090 con 0x7f8c940c6bd0 2023-10-30T12:38:01.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.886+0000 7f8c999c4640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c941f2760 con 0x7f8c94071110 2023-10-30T12:38:01.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.886+0000 7f8c92ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c9406f7a0 0x7f8c94154b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:01.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.886+0000 7f8c92ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c9406f7a0 0x7f8c94154b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53630/0 (socket says 172.21.15.89:53630) 2023-10-30T12:38:01.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.886+0000 7f8c92ffd640 1 -- 172.21.15.89:0/2578310958 learned_addr learned my addr 172.21.15.89:0/2578310958 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:38:01.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.886+0000 7f8c8bfff640 1 --2- 172.21.15.89:0/2578310958 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c94071110 0x7f8c94152a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:01.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.886+0000 7f8c937fe640 1 --2- 172.21.15.89:0/2578310958 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c940c6bd0 0x7f8c940c5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:01.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.886+0000 7f8c92ffd640 1 -- 172.21.15.89:0/2578310958 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c94071110 msgr2=0x7f8c94152a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:01.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.886+0000 7f8c92ffd640 1 --2- 172.21.15.89:0/2578310958 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c94071110 0x7f8c94152a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:01.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.886+0000 7f8c92ffd640 1 -- 172.21.15.89:0/2578310958 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c940c6bd0 msgr2=0x7f8c940c5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:01.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.886+0000 7f8c92ffd640 1 --2- 172.21.15.89:0/2578310958 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c940c6bd0 0x7f8c940c5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:01.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.886+0000 7f8c92ffd640 1 -- 172.21.15.89:0/2578310958 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c8c009a70 con 0x7f8c9406f7a0 2023-10-30T12:38:01.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.886+0000 7f8c8bfff640 1 --2- 172.21.15.89:0/2578310958 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c94071110 0x7f8c94152a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:38:01.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.886+0000 7f8c937fe640 1 --2- 172.21.15.89:0/2578310958 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c940c6bd0 0x7f8c940c5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:38:01.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.886+0000 7f8c92ffd640 1 --2- 172.21.15.89:0/2578310958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c9406f7a0 0x7f8c94154b50 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f8c8400c420 tx=0x7f8c8400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:01.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.886+0000 7f8c90ff9640 1 -- 172.21.15.89:0/2578310958 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c84005cf0 con 0x7f8c9406f7a0 2023-10-30T12:38:01.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.886+0000 7f8c90ff9640 1 -- 172.21.15.89:0/2578310958 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c84005e50 con 0x7f8c9406f7a0 2023-10-30T12:38:01.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.886+0000 7f8c90ff9640 1 -- 172.21.15.89:0/2578310958 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c840072d0 con 0x7f8c9406f7a0 2023-10-30T12:38:01.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.886+0000 7f8c999c4640 1 -- 172.21.15.89:0/2578310958 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c94151f20 con 0x7f8c9406f7a0 2023-10-30T12:38:01.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.886+0000 7f8c999c4640 1 -- 172.21.15.89:0/2578310958 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c941f23f0 con 0x7f8c9406f7a0 2023-10-30T12:38:01.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.886+0000 7f8c999c4640 1 -- 172.21.15.89:0/2578310958 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c9404e170 con 0x7f8c9406f7a0 2023-10-30T12:38:01.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.886+0000 7f8c90ff9640 1 -- 172.21.15.89:0/2578310958 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8c84007430 con 0x7f8c9406f7a0 2023-10-30T12:38:01.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.890+0000 7f8c90ff9640 1 --2- 172.21.15.89:0/2578310958 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8c740664a0 0x7f8c74068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:01.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.890+0000 7f8c90ff9640 1 -- 172.21.15.89:0/2578310958 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f8c84083560 con 0x7f8c9406f7a0 2023-10-30T12:38:01.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.890+0000 7f8c8bfff640 1 --2- 172.21.15.89:0/2578310958 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8c740664a0 0x7f8c74068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:01.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.890+0000 7f8c8bfff640 1 --2- 172.21.15.89:0/2578310958 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8c740664a0 0x7f8c74068960 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8c940c8610 tx=0x7f8c7c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:01.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:01.894+0000 7f8c90ff9640 1 -- 172.21.15.89:0/2578310958 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8c840508c0 con 0x7f8c9406f7a0 2023-10-30T12:38:02.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:02.070+0000 7f8c999c4640 1 -- 172.21.15.89:0/2578310958 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c940c3650 con 0x7f8c9406f7a0 2023-10-30T12:38:02.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:02.074+0000 7f8c90ff9640 1 -- 172.21.15.89:0/2578310958 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+5735 (secure 0 0 0) 0x7f8c84053ee0 con 0x7f8c9406f7a0 2023-10-30T12:38:02.074 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:38:02.074 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:38:00.149923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:38:02.075 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 48 2023-10-30T12:38:02.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:02.074+0000 7f8c999c4640 1 -- 172.21.15.89:0/2578310958 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8c740664a0 msgr2=0x7f8c74068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:02.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:02.074+0000 7f8c999c4640 1 --2- 172.21.15.89:0/2578310958 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8c740664a0 0x7f8c74068960 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8c940c8610 tx=0x7f8c7c009040 comp rx=0 tx=0).stop 2023-10-30T12:38:02.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:02.074+0000 7f8c999c4640 1 -- 172.21.15.89:0/2578310958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c9406f7a0 msgr2=0x7f8c94154b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:02.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:02.074+0000 7f8c999c4640 1 --2- 172.21.15.89:0/2578310958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c9406f7a0 0x7f8c94154b50 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f8c8400c420 tx=0x7f8c8400c8e0 comp rx=0 tx=0).stop 2023-10-30T12:38:02.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:02.074+0000 7f8c999c4640 1 -- 172.21.15.89:0/2578310958 shutdown_connections 2023-10-30T12:38:02.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:02.074+0000 7f8c999c4640 1 --2- 172.21.15.89:0/2578310958 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8c740664a0 0x7f8c74068960 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:02.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:02.074+0000 7f8c999c4640 1 --2- 172.21.15.89:0/2578310958 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c940c6bd0 0x7f8c940c5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:02.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:02.074+0000 7f8c999c4640 1 --2- 172.21.15.89:0/2578310958 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c94071110 0x7f8c94152a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:02.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:02.074+0000 7f8c999c4640 1 --2- 172.21.15.89:0/2578310958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c9406f7a0 0x7f8c94154b50 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:02.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:02.074+0000 7f8c999c4640 1 -- 172.21.15.89:0/2578310958 >> 172.21.15.89:0/2578310958 conn(0x7f8c940c7910 msgr2=0x7f8c940c9680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:02.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:02.074+0000 7f8c999c4640 1 -- 172.21.15.89:0/2578310958 shutdown_connections 2023-10-30T12:38:02.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:02.074+0000 7f8c999c4640 1 -- 172.21.15.89:0/2578310958 wait complete. 2023-10-30T12:38:02.094 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=0, replacing gid=4318 2023-10-30T12:38:02.095 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.f 2023-10-30T12:38:06.095 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-10-30T12:38:06.096 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-30T12:38:06.096 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-30T12:38:06.099 INFO:tasks.ceph.mds.f:Started 2023-10-30T12:38:06.099 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:38:06.186 INFO:tasks.ceph.mds.f.smithi192.stdout:starting mds.f at 2023-10-30T12:38:06.203 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-estale.sh... 2023-10-30T12:38:06.203 DEBUG:teuthology.orchestra.run.smithi089:workunit test fs/snaps/snaptest-estale.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-estale.sh 2023-10-30T12:38:06.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.254+0000 7f42eca46640 1 Processor -- start 2023-10-30T12:38:06.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.254+0000 7f42eca46640 1 -- start start 2023-10-30T12:38:06.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.258+0000 7f42eca46640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f42e8159e60 0x7f42e806f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:06.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.258+0000 7f42eca46640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42e806a720 0x7f42e806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:06.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.258+0000 7f42eca46640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e806fa80 con 0x7f42e80bfec0 2023-10-30T12:38:06.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.258+0000 7f42eca46640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42e8071290 con 0x7f42e8159e60 2023-10-30T12:38:06.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.258+0000 7f42eca46640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42e8071570 con 0x7f42e806a720 2023-10-30T12:38:06.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.258+0000 7f42e659c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f42e8159e60 0x7f42e806f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:06.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.258+0000 7f42ddd9b640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f42e80bfec0 0x7f42e80c1a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33382/0 (socket says 172.21.15.89:33382) 2023-10-30T12:38:06.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.258+0000 7f42ddd9b640 1 -- 172.21.15.89:0/2852913436 learned_addr learned my addr 172.21.15.89:0/2852913436 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:38:06.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.258+0000 7f42e6d9d640 1 --2- 172.21.15.89:0/2852913436 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42e806a720 0x7f42e806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:06.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.258+0000 7f42e659c640 1 -- 172.21.15.89:0/2852913436 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42e806a720 msgr2=0x7f42e806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:06.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.258+0000 7f42e659c640 1 --2- 172.21.15.89:0/2852913436 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42e806a720 0x7f42e806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:06.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.258+0000 7f42e659c640 1 -- 172.21.15.89:0/2852913436 >> v1:172.21.15.89:6789/0 conn(0x7f42e80bfec0 legacy=0x7f42e80c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:06.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.258+0000 7f42e659c640 1 -- 172.21.15.89:0/2852913436 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e8071850 con 0x7f42e8159e60 2023-10-30T12:38:06.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.258+0000 7f42e5d9b640 1 -- 172.21.15.89:0/2852913436 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3396057333 0 0) 0x7f42e806fa80 con 0x7f42e80bfec0 2023-10-30T12:38:06.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.262+0000 7f42e659c640 1 --2- 172.21.15.89:0/2852913436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42e8159e60 0x7f42e806f540 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f42d00097c0 tx=0x7f42d001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b65e4d8978f28e80 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:06.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.262+0000 7f42e5d9b640 1 -- 172.21.15.89:0/2852913436 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d00a8970 con 0x7f42e8159e60 2023-10-30T12:38:06.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.262+0000 7f42e5d9b640 1 -- 172.21.15.89:0/2852913436 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42d00a8ad0 con 0x7f42e8159e60 2023-10-30T12:38:06.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.262+0000 7f42e5d9b640 1 -- 172.21.15.89:0/2852913436 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d00a8e60 con 0x7f42e8159e60 2023-10-30T12:38:06.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.262+0000 7f42eca46640 1 -- 172.21.15.89:0/2852913436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42e8159e60 msgr2=0x7f42e806f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:06.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.262+0000 7f42eca46640 1 --2- 172.21.15.89:0/2852913436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42e8159e60 0x7f42e806f540 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f42d00097c0 tx=0x7f42d001ec70 comp rx=0 tx=0).stop 2023-10-30T12:38:06.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.262+0000 7f42eca46640 1 -- 172.21.15.89:0/2852913436 shutdown_connections 2023-10-30T12:38:06.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.262+0000 7f42eca46640 1 --2- 172.21.15.89:0/2852913436 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42e806a720 0x7f42e806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:06.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.262+0000 7f42eca46640 1 --2- 172.21.15.89:0/2852913436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42e8159e60 0x7f42e806f540 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:06.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.262+0000 7f42eca46640 1 -- 172.21.15.89:0/2852913436 >> 172.21.15.89:0/2852913436 conn(0x7f42e80c7910 msgr2=0x7f42e80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:06.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.262+0000 7f42eca46640 1 -- 172.21.15.89:0/2852913436 shutdown_connections 2023-10-30T12:38:06.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.262+0000 7f42eca46640 1 -- 172.21.15.89:0/2852913436 wait complete. 2023-10-30T12:38:06.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.262+0000 7f42eca46640 1 Processor -- start 2023-10-30T12:38:06.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.262+0000 7f42eca46640 1 -- start start 2023-10-30T12:38:06.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.266+0000 7f42eca46640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42e806a720 0x7f42e8164290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:06.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.266+0000 7f42eca46640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42e80bfec0 0x7f42e8142200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:06.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.266+0000 7f42eca46640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42e8159e60 0x7f42e81456d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:06.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.266+0000 7f42eca46640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e81452b0 con 0x7f42e80bfec0 2023-10-30T12:38:06.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.266+0000 7f42eca46640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e8143cb0 con 0x7f42e806a720 2023-10-30T12:38:06.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.266+0000 7f42eca46640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42e8144b50 con 0x7f42e8159e60 2023-10-30T12:38:06.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.266+0000 7f42ddd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42e80bfec0 0x7f42e8142200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:06.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.266+0000 7f42ddd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42e80bfec0 0x7f42e8142200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53640/0 (socket says 172.21.15.89:53640) 2023-10-30T12:38:06.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.266+0000 7f42ddd9b640 1 -- 172.21.15.89:0/3204278958 learned_addr learned my addr 172.21.15.89:0/3204278958 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:38:06.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.266+0000 7f42e659c640 1 --2- 172.21.15.89:0/3204278958 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42e806a720 0x7f42e8164290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:06.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.266+0000 7f42e6d9d640 1 --2- 172.21.15.89:0/3204278958 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42e8159e60 0x7f42e81456d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:06.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.266+0000 7f42ddd9b640 1 -- 172.21.15.89:0/3204278958 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42e8159e60 msgr2=0x7f42e81456d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:06.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.266+0000 7f42ddd9b640 1 --2- 172.21.15.89:0/3204278958 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42e8159e60 0x7f42e81456d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:06.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.266+0000 7f42ddd9b640 1 -- 172.21.15.89:0/3204278958 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42e806a720 msgr2=0x7f42e8164290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:06.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.266+0000 7f42ddd9b640 1 --2- 172.21.15.89:0/3204278958 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42e806a720 0x7f42e8164290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:06.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.266+0000 7f42ddd9b640 1 -- 172.21.15.89:0/3204278958 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d0009420 con 0x7f42e80bfec0 2023-10-30T12:38:06.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.266+0000 7f42ddd9b640 1 --2- 172.21.15.89:0/3204278958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42e80bfec0 0x7f42e8142200 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f42d80011f0 tx=0x7f42d800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:06.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.266+0000 7f42df7fe640 1 -- 172.21.15.89:0/3204278958 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d80074e0 con 0x7f42e80bfec0 2023-10-30T12:38:06.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.266+0000 7f42df7fe640 1 -- 172.21.15.89:0/3204278958 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42d8007640 con 0x7f42e80bfec0 2023-10-30T12:38:06.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.266+0000 7f42df7fe640 1 -- 172.21.15.89:0/3204278958 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d8004a10 con 0x7f42e80bfec0 2023-10-30T12:38:06.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.266+0000 7f42eca46640 1 -- 172.21.15.89:0/3204278958 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e80ca490 con 0x7f42e80bfec0 2023-10-30T12:38:06.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.266+0000 7f42eca46640 1 -- 172.21.15.89:0/3204278958 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e8142e90 con 0x7f42e80bfec0 2023-10-30T12:38:06.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.266+0000 7f42eca46640 1 -- 172.21.15.89:0/3204278958 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42e8071b10 con 0x7f42e80bfec0 2023-10-30T12:38:06.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.266+0000 7f42df7fe640 1 -- 172.21.15.89:0/3204278958 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f42d8004b70 con 0x7f42e80bfec0 2023-10-30T12:38:06.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.270+0000 7f42df7fe640 1 --2- 172.21.15.89:0/3204278958 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f42c80664a0 0x7f42c8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:06.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.270+0000 7f42df7fe640 1 -- 172.21.15.89:0/3204278958 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6148+0+0 (secure 0 0 0) 0x7f42d80832a0 con 0x7f42e80bfec0 2023-10-30T12:38:06.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.270+0000 7f42e659c640 1 --2- 172.21.15.89:0/3204278958 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f42c80664a0 0x7f42c8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:06.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.270+0000 7f42e659c640 1 --2- 172.21.15.89:0/3204278958 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f42c80664a0 0x7f42c8068960 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f42d00b2000 tx=0x7f42d0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:06.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.274+0000 7f42df7fe640 1 -- 172.21.15.89:0/3204278958 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f42e8071b10 con 0x7f42e80bfec0 2023-10-30T12:38:06.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.454+0000 7f42eca46640 1 -- 172.21.15.89:0/3204278958 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42e81453f0 con 0x7f42e80bfec0 2023-10-30T12:38:06.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.454+0000 7f42df7fe640 1 -- 172.21.15.89:0/3204278958 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+5735 (secure 0 0 0) 0x7f42e81453f0 con 0x7f42e80bfec0 2023-10-30T12:38:06.455 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:38:06.455 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:38:00.149923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:38:06.456 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 48 2023-10-30T12:38:06.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.454+0000 7f42eca46640 1 -- 172.21.15.89:0/3204278958 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f42c80664a0 msgr2=0x7f42c8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:06.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.454+0000 7f42eca46640 1 --2- 172.21.15.89:0/3204278958 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f42c80664a0 0x7f42c8068960 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f42d00b2000 tx=0x7f42d0008040 comp rx=0 tx=0).stop 2023-10-30T12:38:06.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.454+0000 7f42eca46640 1 -- 172.21.15.89:0/3204278958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42e80bfec0 msgr2=0x7f42e8142200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:06.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.454+0000 7f42eca46640 1 --2- 172.21.15.89:0/3204278958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42e80bfec0 0x7f42e8142200 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f42d80011f0 tx=0x7f42d800c9b0 comp rx=0 tx=0).stop 2023-10-30T12:38:06.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.454+0000 7f42eca46640 1 -- 172.21.15.89:0/3204278958 shutdown_connections 2023-10-30T12:38:06.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.454+0000 7f42eca46640 1 --2- 172.21.15.89:0/3204278958 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f42c80664a0 0x7f42c8068960 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:06.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.454+0000 7f42eca46640 1 --2- 172.21.15.89:0/3204278958 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42e8159e60 0x7f42e81456d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:06.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.454+0000 7f42eca46640 1 --2- 172.21.15.89:0/3204278958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42e80bfec0 0x7f42e8142200 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:06.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.454+0000 7f42eca46640 1 --2- 172.21.15.89:0/3204278958 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42e806a720 0x7f42e8164290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:06.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.454+0000 7f42eca46640 1 -- 172.21.15.89:0/3204278958 >> 172.21.15.89:0/3204278958 conn(0x7f42e80c7910 msgr2=0x7f42e80c8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:06.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.454+0000 7f42eca46640 1 -- 172.21.15.89:0/3204278958 shutdown_connections 2023-10-30T12:38:06.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:06.454+0000 7f42eca46640 1 -- 172.21.15.89:0/3204278958 wait complete. 2023-10-30T12:38:06.475 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2023-10-30T12:38:08.476 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:38:08.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.622+0000 7f9bf04ea640 1 Processor -- start 2023-10-30T12:38:08.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.622+0000 7f9bf04ea640 1 -- start start 2023-10-30T12:38:08.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.626+0000 7f9bf04ea640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9be80c6f60 0x7f9be81566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:08.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.626+0000 7f9bf04ea640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9be80c0a50 con 0x7f9be80c6b80 2023-10-30T12:38:08.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.626+0000 7f9bf04ea640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9be806f6e0 con 0x7f9be80c05d0 2023-10-30T12:38:08.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.626+0000 7f9bf04ea640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9be806f9c0 con 0x7f9be80c6f60 2023-10-30T12:38:08.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.626+0000 7f9bee286640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9be80c6b80 0x7f9be80bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:33394/0 (socket says 172.21.15.89:33394) 2023-10-30T12:38:08.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.626+0000 7f9bee286640 1 -- 172.21.15.89:0/1378474073 learned_addr learned my addr 172.21.15.89:0/1378474073 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:38:08.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.626+0000 7f9beea87640 1 --2- 172.21.15.89:0/1378474073 >> v2:172.21.15.89:3300/0 conn(0x7f9be80c6f60 0x7f9be81566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:08.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.626+0000 7f9bed284640 1 -- 172.21.15.89:0/1378474073 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1117325338 0 0) 0x7f9be80c0a50 con 0x7f9be80c6b80 2023-10-30T12:38:08.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.626+0000 7f9bed284640 1 -- 172.21.15.89:0/1378474073 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bd4003530 con 0x7f9be80c6b80 2023-10-30T12:38:08.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.626+0000 7f9beea87640 1 -- 172.21.15.89:0/1378474073 >> v1:172.21.15.192:6789/0 conn(0x7f9be80c05d0 legacy=0x7f9be813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:08.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.626+0000 7f9beea87640 1 -- 172.21.15.89:0/1378474073 >> v1:172.21.15.89:6789/0 conn(0x7f9be80c6b80 legacy=0x7f9be80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:08.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.626+0000 7f9beea87640 1 -- 172.21.15.89:0/1378474073 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be806fca0 con 0x7f9be80c6f60 2023-10-30T12:38:08.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.626+0000 7f9beea87640 1 --2- 172.21.15.89:0/1378474073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9be80c6f60 0x7f9be81566a0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f9be0009d90 tx=0x7f9be001c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c4bfc0dcff2c732 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:08.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.626+0000 7f9bed284640 1 -- 172.21.15.89:0/1378474073 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be00a5920 con 0x7f9be80c6f60 2023-10-30T12:38:08.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.630+0000 7f9bed284640 1 -- 172.21.15.89:0/1378474073 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be00a5a80 con 0x7f9be80c6f60 2023-10-30T12:38:08.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.630+0000 7f9bed284640 1 -- 172.21.15.89:0/1378474073 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be00a5e10 con 0x7f9be80c6f60 2023-10-30T12:38:08.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.630+0000 7f9bf04ea640 1 -- 172.21.15.89:0/1378474073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9be80c6f60 msgr2=0x7f9be81566a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:08.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.630+0000 7f9bf04ea640 1 --2- 172.21.15.89:0/1378474073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9be80c6f60 0x7f9be81566a0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f9be0009d90 tx=0x7f9be001c950 comp rx=0 tx=0).stop 2023-10-30T12:38:08.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.630+0000 7f9bf04ea640 1 -- 172.21.15.89:0/1378474073 shutdown_connections 2023-10-30T12:38:08.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.630+0000 7f9bf04ea640 1 --2- 172.21.15.89:0/1378474073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9be80c6f60 0x7f9be81566a0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:08.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.630+0000 7f9bf04ea640 1 -- 172.21.15.89:0/1378474073 >> 172.21.15.89:0/1378474073 conn(0x7f9be80c7910 msgr2=0x7f9be80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:08.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.630+0000 7f9bf04ea640 1 -- 172.21.15.89:0/1378474073 shutdown_connections 2023-10-30T12:38:08.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.630+0000 7f9bf04ea640 1 -- 172.21.15.89:0/1378474073 wait complete. 2023-10-30T12:38:08.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.630+0000 7f9bf04ea640 1 Processor -- start 2023-10-30T12:38:08.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.630+0000 7f9bf04ea640 1 -- start start 2023-10-30T12:38:08.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.630+0000 7f9bf04ea640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9be80c05d0 0x7f9be81e9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:08.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.630+0000 7f9bf04ea640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9be80c6b80 0x7f9be80ca310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:08.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.630+0000 7f9bf04ea640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9be80c6f60 0x7f9be81ea3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:08.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.630+0000 7f9bf04ea640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be813ab20 con 0x7f9be80c05d0 2023-10-30T12:38:08.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.630+0000 7f9bf04ea640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be8150dd0 con 0x7f9be80c6f60 2023-10-30T12:38:08.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.630+0000 7f9bf04ea640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9be81ea900 con 0x7f9be80c6b80 2023-10-30T12:38:08.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.630+0000 7f9bee286640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9be80c05d0 0x7f9be81e9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:08.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.630+0000 7f9bee286640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9be80c05d0 0x7f9be81e9790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53664/0 (socket says 172.21.15.89:53664) 2023-10-30T12:38:08.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.630+0000 7f9bee286640 1 -- 172.21.15.89:0/3244013568 learned_addr learned my addr 172.21.15.89:0/3244013568 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:38:08.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.630+0000 7f9beea87640 1 --2- 172.21.15.89:0/3244013568 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9be80c6f60 0x7f9be81ea3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:08.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.630+0000 7f9beda85640 1 --2- 172.21.15.89:0/3244013568 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9be80c6b80 0x7f9be80ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:08.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.634+0000 7f9bee286640 1 -- 172.21.15.89:0/3244013568 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9be80c6b80 msgr2=0x7f9be80ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:08.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.634+0000 7f9bee286640 1 --2- 172.21.15.89:0/3244013568 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9be80c6b80 0x7f9be80ca310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:08.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.634+0000 7f9bee286640 1 -- 172.21.15.89:0/3244013568 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9be80c6f60 msgr2=0x7f9be81ea3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:08.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.634+0000 7f9bee286640 1 --2- 172.21.15.89:0/3244013568 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9be80c6f60 0x7f9be81ea3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:08.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.634+0000 7f9bee286640 1 -- 172.21.15.89:0/3244013568 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be0009a70 con 0x7f9be80c05d0 2023-10-30T12:38:08.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.634+0000 7f9bee286640 1 --2- 172.21.15.89:0/3244013568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9be80c05d0 0x7f9be81e9790 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f9be400c500 tx=0x7f9be400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:08.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.634+0000 7f9bdeffd640 1 -- 172.21.15.89:0/3244013568 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be4005cf0 con 0x7f9be80c05d0 2023-10-30T12:38:08.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.634+0000 7f9bdeffd640 1 -- 172.21.15.89:0/3244013568 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be4005e50 con 0x7f9be80c05d0 2023-10-30T12:38:08.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.634+0000 7f9bf04ea640 1 -- 172.21.15.89:0/3244013568 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9be80c9e80 con 0x7f9be80c05d0 2023-10-30T12:38:08.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.634+0000 7f9bdeffd640 1 -- 172.21.15.89:0/3244013568 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be40072f0 con 0x7f9be80c05d0 2023-10-30T12:38:08.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.634+0000 7f9bf04ea640 1 -- 172.21.15.89:0/3244013568 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9be81f4050 con 0x7f9be80c05d0 2023-10-30T12:38:08.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.634+0000 7f9bf04ea640 1 -- 172.21.15.89:0/3244013568 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9be804e170 con 0x7f9be80c05d0 2023-10-30T12:38:08.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.634+0000 7f9bdeffd640 1 -- 172.21.15.89:0/3244013568 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9be4007450 con 0x7f9be80c05d0 2023-10-30T12:38:08.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.634+0000 7f9bdeffd640 1 --2- 172.21.15.89:0/3244013568 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9bcc0664a0 0x7f9bcc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:08.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.634+0000 7f9bdeffd640 1 -- 172.21.15.89:0/3244013568 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6204+0+0 (secure 0 0 0) 0x7f9be40835d0 con 0x7f9be80c05d0 2023-10-30T12:38:08.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.634+0000 7f9beda85640 1 --2- 172.21.15.89:0/3244013568 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9bcc0664a0 0x7f9bcc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:08.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.638+0000 7f9beda85640 1 --2- 172.21.15.89:0/3244013568 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9bcc0664a0 0x7f9bcc068960 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9be81f1250 tx=0x7f9bd8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:08.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.638+0000 7f9bdeffd640 1 -- 172.21.15.89:0/3244013568 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9be40509a0 con 0x7f9be80c05d0 2023-10-30T12:38:08.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.818+0000 7f9bf04ea640 1 -- 172.21.15.89:0/3244013568 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9be81f42b0 con 0x7f9be80c05d0 2023-10-30T12:38:08.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.818+0000 7f9bdeffd640 1 -- 172.21.15.89:0/3244013568 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+6529 (secure 0 0 0) 0x7f9be4053fc0 con 0x7f9be80c05d0 2023-10-30T12:38:08.820 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:38:08.820 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5129,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:38:00.149923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:38:08.820 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 49 2023-10-30T12:38:08.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.822+0000 7f9bf04ea640 1 -- 172.21.15.89:0/3244013568 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9bcc0664a0 msgr2=0x7f9bcc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:08.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.822+0000 7f9bf04ea640 1 --2- 172.21.15.89:0/3244013568 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9bcc0664a0 0x7f9bcc068960 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9be81f1250 tx=0x7f9bd8009040 comp rx=0 tx=0).stop 2023-10-30T12:38:08.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.822+0000 7f9bf04ea640 1 -- 172.21.15.89:0/3244013568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9be80c05d0 msgr2=0x7f9be81e9790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:08.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.822+0000 7f9bf04ea640 1 --2- 172.21.15.89:0/3244013568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9be80c05d0 0x7f9be81e9790 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f9be400c500 tx=0x7f9be400c9c0 comp rx=0 tx=0).stop 2023-10-30T12:38:08.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.822+0000 7f9bf04ea640 1 -- 172.21.15.89:0/3244013568 shutdown_connections 2023-10-30T12:38:08.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.822+0000 7f9bf04ea640 1 --2- 172.21.15.89:0/3244013568 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9bcc0664a0 0x7f9bcc068960 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:08.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.822+0000 7f9bf04ea640 1 --2- 172.21.15.89:0/3244013568 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9be80c6f60 0x7f9be81ea3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:08.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.822+0000 7f9bf04ea640 1 --2- 172.21.15.89:0/3244013568 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9be80c6b80 0x7f9be80ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:08.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.822+0000 7f9bf04ea640 1 --2- 172.21.15.89:0/3244013568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9be80c05d0 0x7f9be81e9790 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:08.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.822+0000 7f9bf04ea640 1 -- 172.21.15.89:0/3244013568 >> 172.21.15.89:0/3244013568 conn(0x7f9be80c7910 msgr2=0x7f9be81532b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:08.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.822+0000 7f9bf04ea640 1 -- 172.21.15.89:0/3244013568 shutdown_connections 2023-10-30T12:38:08.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:38:08.822+0000 7f9bf04ea640 1 -- 172.21.15.89:0/3244013568 wait complete. 2023-10-30T12:38:08.840 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-10-30T12:38:08.841 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 104 secs before thrashing 2023-10-30T12:38:09.170 INFO:tasks.ceph.mds.d.smithi192.stderr:2023-10-30T12:38:09.167+0000 7f24c55b4640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T12:38:09.170 INFO:tasks.ceph.mds.d.smithi192.stderr:2023-10-30T12:38:09.167+0000 7f24c55b4640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T12:38:09.170 INFO:tasks.ceph.mds.d.smithi192.stderr:2023-10-30T12:38:09.167+0000 7f24c55b4640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T12:38:11.868 INFO:tasks.workunit.client.0.smithi089.stderr:+ mkdir .snap/foo 2023-10-30T12:38:11.885 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo We want ENOENT, not ESTALE, here. 2023-10-30T12:38:11.885 INFO:tasks.workunit.client.0.smithi089.stdout:We want ENOENT, not ESTALE, here. 2023-10-30T12:38:11.886 INFO:tasks.workunit.client.0.smithi089.stderr:+ seq 1 100 2023-10-30T12:38:11.887 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/1 2023-10-30T12:38:11.887 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.892 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/1': No such file or directory 2023-10-30T12:38:11.893 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/2 2023-10-30T12:38:11.893 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.895 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/2': No such file or directory 2023-10-30T12:38:11.896 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/3 2023-10-30T12:38:11.896 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.898 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/3': No such file or directory 2023-10-30T12:38:11.898 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/4 2023-10-30T12:38:11.899 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.900 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/4': No such file or directory 2023-10-30T12:38:11.901 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/5 2023-10-30T12:38:11.902 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.903 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/5': No such file or directory 2023-10-30T12:38:11.904 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/6 2023-10-30T12:38:11.904 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.906 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/6': No such file or directory 2023-10-30T12:38:11.907 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/7 2023-10-30T12:38:11.907 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.909 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/7': No such file or directory 2023-10-30T12:38:11.910 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/8 2023-10-30T12:38:11.910 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.911 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/8': No such file or directory 2023-10-30T12:38:11.912 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/9 2023-10-30T12:38:11.913 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.914 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/9': No such file or directory 2023-10-30T12:38:11.915 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/10 2023-10-30T12:38:11.915 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.917 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/10': No such file or directory 2023-10-30T12:38:11.918 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/11 2023-10-30T12:38:11.918 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.920 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/11': No such file or directory 2023-10-30T12:38:11.921 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/12 2023-10-30T12:38:11.921 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.922 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/12': No such file or directory 2023-10-30T12:38:11.923 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/13 2023-10-30T12:38:11.924 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.925 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/13': No such file or directory 2023-10-30T12:38:11.926 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/14 2023-10-30T12:38:11.927 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.928 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/14': No such file or directory 2023-10-30T12:38:11.929 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/15 2023-10-30T12:38:11.929 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.931 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/15': No such file or directory 2023-10-30T12:38:11.932 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/16 2023-10-30T12:38:11.932 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.933 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/16': No such file or directory 2023-10-30T12:38:11.934 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/17 2023-10-30T12:38:11.935 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.936 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/17': No such file or directory 2023-10-30T12:38:11.937 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/18 2023-10-30T12:38:11.937 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.939 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/18': No such file or directory 2023-10-30T12:38:11.940 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/19 2023-10-30T12:38:11.940 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.942 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/19': No such file or directory 2023-10-30T12:38:11.942 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/20 2023-10-30T12:38:11.943 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.944 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/20': No such file or directory 2023-10-30T12:38:11.945 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/21 2023-10-30T12:38:11.945 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.947 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/21': No such file or directory 2023-10-30T12:38:11.948 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/22 2023-10-30T12:38:11.948 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.950 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/22': No such file or directory 2023-10-30T12:38:11.951 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/23 2023-10-30T12:38:11.951 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.952 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/23': No such file or directory 2023-10-30T12:38:11.953 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/24 2023-10-30T12:38:11.954 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.955 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/24': No such file or directory 2023-10-30T12:38:11.956 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/25 2023-10-30T12:38:11.956 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.958 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/25': No such file or directory 2023-10-30T12:38:11.959 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/26 2023-10-30T12:38:11.959 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.960 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/26': No such file or directory 2023-10-30T12:38:11.961 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/27 2023-10-30T12:38:11.962 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.963 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/27': No such file or directory 2023-10-30T12:38:11.964 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/28 2023-10-30T12:38:11.964 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.966 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/28': No such file or directory 2023-10-30T12:38:11.966 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/29 2023-10-30T12:38:11.967 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.968 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/29': No such file or directory 2023-10-30T12:38:11.969 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/30 2023-10-30T12:38:11.969 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.971 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/30': No such file or directory 2023-10-30T12:38:11.972 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/31 2023-10-30T12:38:11.972 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.973 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/31': No such file or directory 2023-10-30T12:38:11.974 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/32 2023-10-30T12:38:11.975 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.976 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/32': No such file or directory 2023-10-30T12:38:11.977 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/33 2023-10-30T12:38:11.977 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.979 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/33': No such file or directory 2023-10-30T12:38:11.979 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/34 2023-10-30T12:38:11.980 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.981 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/34': No such file or directory 2023-10-30T12:38:11.982 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/35 2023-10-30T12:38:11.982 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.984 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/35': No such file or directory 2023-10-30T12:38:11.985 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/36 2023-10-30T12:38:11.985 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.986 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/36': No such file or directory 2023-10-30T12:38:11.987 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/37 2023-10-30T12:38:11.988 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.989 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/37': No such file or directory 2023-10-30T12:38:11.990 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/38 2023-10-30T12:38:11.990 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.992 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/38': No such file or directory 2023-10-30T12:38:11.993 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/39 2023-10-30T12:38:11.993 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.994 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/39': No such file or directory 2023-10-30T12:38:11.995 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/40 2023-10-30T12:38:11.995 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.997 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/40': No such file or directory 2023-10-30T12:38:11.998 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/41 2023-10-30T12:38:11.998 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:11.999 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/41': No such file or directory 2023-10-30T12:38:12.000 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/42 2023-10-30T12:38:12.000 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.002 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/42': No such file or directory 2023-10-30T12:38:12.003 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/43 2023-10-30T12:38:12.003 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.005 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/43': No such file or directory 2023-10-30T12:38:12.005 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/44 2023-10-30T12:38:12.006 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.007 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/44': No such file or directory 2023-10-30T12:38:12.008 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/45 2023-10-30T12:38:12.008 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.010 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/45': No such file or directory 2023-10-30T12:38:12.011 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/46 2023-10-30T12:38:12.011 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.013 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/46': No such file or directory 2023-10-30T12:38:12.014 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/47 2023-10-30T12:38:12.014 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.015 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/47': No such file or directory 2023-10-30T12:38:12.016 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/48 2023-10-30T12:38:12.016 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.018 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/48': No such file or directory 2023-10-30T12:38:12.019 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/49 2023-10-30T12:38:12.019 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.022 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/49': No such file or directory 2023-10-30T12:38:12.023 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/50 2023-10-30T12:38:12.023 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.023 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/50': No such file or directory 2023-10-30T12:38:12.024 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/51 2023-10-30T12:38:12.024 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.026 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/51': No such file or directory 2023-10-30T12:38:12.027 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/52 2023-10-30T12:38:12.027 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.028 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/52': No such file or directory 2023-10-30T12:38:12.029 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/53 2023-10-30T12:38:12.030 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.031 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/53': No such file or directory 2023-10-30T12:38:12.032 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/54 2023-10-30T12:38:12.032 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.034 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/54': No such file or directory 2023-10-30T12:38:12.035 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/55 2023-10-30T12:38:12.035 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.037 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/55': No such file or directory 2023-10-30T12:38:12.037 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/56 2023-10-30T12:38:12.038 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.039 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/56': No such file or directory 2023-10-30T12:38:12.040 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/57 2023-10-30T12:38:12.040 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.042 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/57': No such file or directory 2023-10-30T12:38:12.043 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/58 2023-10-30T12:38:12.043 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.044 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/58': No such file or directory 2023-10-30T12:38:12.045 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/59 2023-10-30T12:38:12.045 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.047 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/59': No such file or directory 2023-10-30T12:38:12.048 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/60 2023-10-30T12:38:12.048 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.049 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/60': No such file or directory 2023-10-30T12:38:12.050 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/61 2023-10-30T12:38:12.051 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.052 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/61': No such file or directory 2023-10-30T12:38:12.053 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/62 2023-10-30T12:38:12.053 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.055 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/62': No such file or directory 2023-10-30T12:38:12.056 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/63 2023-10-30T12:38:12.056 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.057 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/63': No such file or directory 2023-10-30T12:38:12.058 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/64 2023-10-30T12:38:12.058 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.314 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/64': No such file or directory 2023-10-30T12:38:12.315 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/65 2023-10-30T12:38:12.316 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.317 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/65': No such file or directory 2023-10-30T12:38:12.318 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/66 2023-10-30T12:38:12.318 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.320 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/66': No such file or directory 2023-10-30T12:38:12.321 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/67 2023-10-30T12:38:12.321 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.323 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/67': No such file or directory 2023-10-30T12:38:12.324 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/68 2023-10-30T12:38:12.324 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.325 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/68': No such file or directory 2023-10-30T12:38:12.326 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/69 2023-10-30T12:38:12.326 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.328 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/69': No such file or directory 2023-10-30T12:38:12.329 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/70 2023-10-30T12:38:12.329 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.330 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/70': No such file or directory 2023-10-30T12:38:12.331 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/71 2023-10-30T12:38:12.331 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.333 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/71': No such file or directory 2023-10-30T12:38:12.334 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/72 2023-10-30T12:38:12.334 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.336 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/72': No such file or directory 2023-10-30T12:38:12.337 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/73 2023-10-30T12:38:12.337 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.338 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/73': No such file or directory 2023-10-30T12:38:12.339 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/74 2023-10-30T12:38:12.339 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.341 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/74': No such file or directory 2023-10-30T12:38:12.342 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/75 2023-10-30T12:38:12.342 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.344 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/75': No such file or directory 2023-10-30T12:38:12.345 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/76 2023-10-30T12:38:12.345 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.346 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/76': No such file or directory 2023-10-30T12:38:12.347 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/77 2023-10-30T12:38:12.347 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.349 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/77': No such file or directory 2023-10-30T12:38:12.350 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/78 2023-10-30T12:38:12.350 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.351 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/78': No such file or directory 2023-10-30T12:38:12.352 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/79 2023-10-30T12:38:12.352 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.354 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/79': No such file or directory 2023-10-30T12:38:12.355 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/80 2023-10-30T12:38:12.355 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.356 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/80': No such file or directory 2023-10-30T12:38:12.357 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/81 2023-10-30T12:38:12.358 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.359 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/81': No such file or directory 2023-10-30T12:38:12.360 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/82 2023-10-30T12:38:12.360 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.362 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/82': No such file or directory 2023-10-30T12:38:12.362 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/83 2023-10-30T12:38:12.363 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.364 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/83': No such file or directory 2023-10-30T12:38:12.365 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/84 2023-10-30T12:38:12.365 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.367 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/84': No such file or directory 2023-10-30T12:38:12.368 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/85 2023-10-30T12:38:12.368 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.370 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/85': No such file or directory 2023-10-30T12:38:12.371 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/86 2023-10-30T12:38:12.371 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.372 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/86': No such file or directory 2023-10-30T12:38:12.373 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/87 2023-10-30T12:38:12.373 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.375 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/87': No such file or directory 2023-10-30T12:38:12.376 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/88 2023-10-30T12:38:12.376 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.377 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/88': No such file or directory 2023-10-30T12:38:12.378 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/89 2023-10-30T12:38:12.379 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.380 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/89': No such file or directory 2023-10-30T12:38:12.381 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/90 2023-10-30T12:38:12.381 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.383 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/90': No such file or directory 2023-10-30T12:38:12.383 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/91 2023-10-30T12:38:12.384 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.385 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/91': No such file or directory 2023-10-30T12:38:12.386 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/92 2023-10-30T12:38:12.386 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.388 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/92': No such file or directory 2023-10-30T12:38:12.389 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/93 2023-10-30T12:38:12.389 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.391 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/93': No such file or directory 2023-10-30T12:38:12.392 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/94 2023-10-30T12:38:12.392 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.393 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/94': No such file or directory 2023-10-30T12:38:12.394 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/95 2023-10-30T12:38:12.394 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.396 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/95': No such file or directory 2023-10-30T12:38:12.397 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/96 2023-10-30T12:38:12.397 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.398 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/96': No such file or directory 2023-10-30T12:38:12.399 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/97 2023-10-30T12:38:12.400 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.401 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/97': No such file or directory 2023-10-30T12:38:12.402 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/98 2023-10-30T12:38:12.402 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.403 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/98': No such file or directory 2023-10-30T12:38:12.404 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/99 2023-10-30T12:38:12.404 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.406 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/99': No such file or directory 2023-10-30T12:38:12.407 INFO:tasks.workunit.client.0.smithi089.stderr:+ stat .snap/foo/100 2023-10-30T12:38:12.407 INFO:tasks.workunit.client.0.smithi089.stderr:+ grep No such file 2023-10-30T12:38:12.408 INFO:tasks.workunit.client.0.smithi089.stdout:stat: cannot statx '.snap/foo/100': No such file or directory 2023-10-30T12:38:12.409 INFO:tasks.workunit.client.0.smithi089.stderr:+ rmdir .snap/foo 2023-10-30T12:38:12.420 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo OK 2023-10-30T12:38:12.421 INFO:tasks.workunit.client.0.smithi089.stdout:OK 2023-10-30T12:38:12.422 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T12:38:12.422 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T12:38:12.453 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-git-ceph.sh... 2023-10-30T12:38:12.453 DEBUG:teuthology.orchestra.run.smithi089:workunit test fs/snaps/snaptest-git-ceph.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh 2023-10-30T12:38:12.519 INFO:tasks.workunit.client.0.smithi089.stderr:+ set -e 2023-10-30T12:38:12.519 INFO:tasks.workunit.client.0.smithi089.stderr:+ sudo git config --global http.sslVerify false 2023-10-30T12:38:12.537 INFO:tasks.workunit.client.0.smithi089.stderr:+ sudo git config --global http.postBuffer 1048576000 2023-10-30T12:38:12.551 INFO:tasks.workunit.client.0.smithi089.stderr:+ retried=false 2023-10-30T12:38:12.551 INFO:tasks.workunit.client.0.smithi089.stderr:+ trap -- retry EXIT 2023-10-30T12:38:12.551 INFO:tasks.workunit.client.0.smithi089.stderr:+ rm -rf ceph 2023-10-30T12:38:12.552 INFO:tasks.workunit.client.0.smithi089.stderr:+ timeout 1800 git clone https://git.ceph.com/ceph.git 2023-10-30T12:38:12.557 INFO:tasks.workunit.client.0.smithi089.stderr:Cloning into 'ceph'... 2023-10-30T12:38:16.327 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:38:16.334 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:38:46.463 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:38:46.469 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:39:16.519 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:39:16.523 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:39:46.574 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:39:46.579 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:39:52.842 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:39:52.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.981+0000 7f3b6fcc9640 1 Processor -- start 2023-10-30T12:39:52.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.981+0000 7f3b6fcc9640 1 -- start start 2023-10-30T12:39:52.985 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.985+0000 7f3b6fcc9640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3b680c6bd0 0x7f3b6806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:52.985 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.985+0000 7f3b6fcc9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b68071110 0x7f3b68158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:52.985 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.985+0000 7f3b6fcc9640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b68066cf0 con 0x7f3b6806f7a0 2023-10-30T12:39:52.985 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.985+0000 7f3b6fcc9640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b680700c0 con 0x7f3b680c6bd0 2023-10-30T12:39:52.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.985+0000 7f3b6fcc9640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b680c70d0 con 0x7f3b68071110 2023-10-30T12:39:52.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.985+0000 7f3b6d264640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3b680c6bd0 0x7f3b6806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:52.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.985+0000 7f3b6d264640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3b680c6bd0 0x7f3b6806fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33284/0 (socket says 172.21.15.89:33284) 2023-10-30T12:39:52.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.985+0000 7f3b6d264640 1 -- 172.21.15.89:0/1975447509 learned_addr learned my addr 172.21.15.89:0/1975447509 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:39:52.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.985+0000 7f3b6da65640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f3b6806f7a0 0x7f3b68069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47020/0 (socket says 172.21.15.89:47020) 2023-10-30T12:39:52.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.985+0000 7f3b6e266640 1 --2- 172.21.15.89:0/1975447509 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b68071110 0x7f3b68158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:52.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.985+0000 7f3b6d264640 1 -- 172.21.15.89:0/1975447509 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b68071110 msgr2=0x7f3b68158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:52.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.985+0000 7f3b6d264640 1 --2- 172.21.15.89:0/1975447509 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b68071110 0x7f3b68158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:52.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.985+0000 7f3b6d264640 1 -- 172.21.15.89:0/1975447509 >> v1:172.21.15.89:6789/0 conn(0x7f3b6806f7a0 legacy=0x7f3b68069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:52.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.985+0000 7f3b6d264640 1 -- 172.21.15.89:0/1975447509 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b680c7390 con 0x7f3b680c6bd0 2023-10-30T12:39:52.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.985+0000 7f3b6ca63640 1 -- 172.21.15.89:0/1975447509 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 650069629 0 0) 0x7f3b68066cf0 con 0x7f3b6806f7a0 2023-10-30T12:39:52.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.989+0000 7f3b6d264640 1 --2- 172.21.15.89:0/1975447509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b680c6bd0 0x7f3b6806fb80 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f3b58009870 tx=0x7f3b5801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=631870bba6c23697 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:52.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.989+0000 7f3b6ca63640 1 -- 172.21.15.89:0/1975447509 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b580a8a30 con 0x7f3b680c6bd0 2023-10-30T12:39:52.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.989+0000 7f3b6ca63640 1 -- 172.21.15.89:0/1975447509 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b580a8b90 con 0x7f3b680c6bd0 2023-10-30T12:39:52.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.989+0000 7f3b6ca63640 1 -- 172.21.15.89:0/1975447509 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b580b0440 con 0x7f3b680c6bd0 2023-10-30T12:39:52.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.989+0000 7f3b6fcc9640 1 -- 172.21.15.89:0/1975447509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b680c6bd0 msgr2=0x7f3b6806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:52.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.989+0000 7f3b6fcc9640 1 --2- 172.21.15.89:0/1975447509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b680c6bd0 0x7f3b6806fb80 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f3b58009870 tx=0x7f3b5801ed30 comp rx=0 tx=0).stop 2023-10-30T12:39:52.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.989+0000 7f3b6fcc9640 1 -- 172.21.15.89:0/1975447509 shutdown_connections 2023-10-30T12:39:52.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.989+0000 7f3b6fcc9640 1 --2- 172.21.15.89:0/1975447509 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b68071110 0x7f3b68158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:52.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.989+0000 7f3b6fcc9640 1 --2- 172.21.15.89:0/1975447509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b680c6bd0 0x7f3b6806fb80 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:52.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.989+0000 7f3b6fcc9640 1 -- 172.21.15.89:0/1975447509 >> 172.21.15.89:0/1975447509 conn(0x7f3b680c7910 msgr2=0x7f3b680c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:52.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.989+0000 7f3b6fcc9640 1 -- 172.21.15.89:0/1975447509 shutdown_connections 2023-10-30T12:39:52.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.989+0000 7f3b6fcc9640 1 -- 172.21.15.89:0/1975447509 wait complete. 2023-10-30T12:39:52.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.989+0000 7f3b6fcc9640 1 Processor -- start 2023-10-30T12:39:52.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.989+0000 7f3b6fcc9640 1 -- start start 2023-10-30T12:39:52.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.993+0000 7f3b6fcc9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b6806f7a0 0x7f3b68157930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:52.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.993+0000 7f3b6fcc9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b68071110 0x7f3b680c3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:52.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.993+0000 7f3b6fcc9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b680c6bd0 0x7f3b68151940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:52.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.993+0000 7f3b6fcc9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b68154ae0 con 0x7f3b68071110 2023-10-30T12:39:52.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.993+0000 7f3b6fcc9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b68155c30 con 0x7f3b6806f7a0 2023-10-30T12:39:52.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.993+0000 7f3b6fcc9640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b680c9780 con 0x7f3b680c6bd0 2023-10-30T12:39:52.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.993+0000 7f3b6d264640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b68071110 0x7f3b680c3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:52.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.993+0000 7f3b6d264640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b68071110 0x7f3b680c3600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33298/0 (socket says 172.21.15.89:33298) 2023-10-30T12:39:52.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.993+0000 7f3b6d264640 1 -- 172.21.15.89:0/2016157062 learned_addr learned my addr 172.21.15.89:0/2016157062 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:39:52.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.993+0000 7f3b6da65640 1 --2- 172.21.15.89:0/2016157062 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b6806f7a0 0x7f3b68157930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:52.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.993+0000 7f3b6e266640 1 --2- 172.21.15.89:0/2016157062 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b680c6bd0 0x7f3b68151940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:52.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.993+0000 7f3b6d264640 1 -- 172.21.15.89:0/2016157062 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b680c6bd0 msgr2=0x7f3b68151940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:52.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.993+0000 7f3b6d264640 1 --2- 172.21.15.89:0/2016157062 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b680c6bd0 0x7f3b68151940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:52.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.993+0000 7f3b6d264640 1 -- 172.21.15.89:0/2016157062 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b6806f7a0 msgr2=0x7f3b68157930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:52.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.993+0000 7f3b6d264640 1 --2- 172.21.15.89:0/2016157062 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b6806f7a0 0x7f3b68157930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:52.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.993+0000 7f3b6d264640 1 -- 172.21.15.89:0/2016157062 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b580094d0 con 0x7f3b68071110 2023-10-30T12:39:52.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.993+0000 7f3b6da65640 1 --2- 172.21.15.89:0/2016157062 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b6806f7a0 0x7f3b68157930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:39:52.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.993+0000 7f3b6d264640 1 --2- 172.21.15.89:0/2016157062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b68071110 0x7f3b680c3600 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3b580b3040 tx=0x7f3b5801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:52.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.993+0000 7f3b5e7fc640 1 -- 172.21.15.89:0/2016157062 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b580b9620 con 0x7f3b68071110 2023-10-30T12:39:52.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.993+0000 7f3b5e7fc640 1 -- 172.21.15.89:0/2016157062 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b580b97c0 con 0x7f3b68071110 2023-10-30T12:39:52.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.993+0000 7f3b5e7fc640 1 -- 172.21.15.89:0/2016157062 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b580b9b70 con 0x7f3b68071110 2023-10-30T12:39:52.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.993+0000 7f3b6fcc9640 1 -- 172.21.15.89:0/2016157062 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b68157f50 con 0x7f3b68071110 2023-10-30T12:39:52.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.993+0000 7f3b6fcc9640 1 -- 172.21.15.89:0/2016157062 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b68155300 con 0x7f3b68071110 2023-10-30T12:39:53.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.993+0000 7f3b6fcc9640 1 -- 172.21.15.89:0/2016157062 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b68070330 con 0x7f3b68071110 2023-10-30T12:39:53.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.993+0000 7f3b5e7fc640 1 -- 172.21.15.89:0/2016157062 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3b580b7070 con 0x7f3b68071110 2023-10-30T12:39:53.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.997+0000 7f3b5e7fc640 1 --2- 172.21.15.89:0/2016157062 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3b4c0664a0 0x7f3b4c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:53.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.997+0000 7f3b5e7fc640 1 -- 172.21.15.89:0/2016157062 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6176+0+0 (secure 0 0 0) 0x7f3b58121fc0 con 0x7f3b68071110 2023-10-30T12:39:53.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.997+0000 7f3b6da65640 1 --2- 172.21.15.89:0/2016157062 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3b4c0664a0 0x7f3b4c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:53.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:52.997+0000 7f3b6da65640 1 --2- 172.21.15.89:0/2016157062 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3b4c0664a0 0x7f3b4c068960 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3b64000f70 tx=0x7f3b64007cd0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:53.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.001+0000 7f3b5e7fc640 1 -- 172.21.15.89:0/2016157062 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3b68070330 con 0x7f3b68071110 2023-10-30T12:39:53.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.185+0000 7f3b6fcc9640 1 -- 172.21.15.89:0/2016157062 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b68070330 con 0x7f3b68071110 2023-10-30T12:39:53.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.185+0000 7f3b5e7fc640 1 -- 172.21.15.89:0/2016157062 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+6526 (secure 0 0 0) 0x7f3b68070330 con 0x7f3b68071110 2023-10-30T12:39:53.185 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:39:53.185 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5129,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:39:31.166072+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:39:53.186 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 54 2023-10-30T12:39:53.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.185+0000 7f3b6fcc9640 1 -- 172.21.15.89:0/2016157062 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3b4c0664a0 msgr2=0x7f3b4c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:53.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.185+0000 7f3b6fcc9640 1 --2- 172.21.15.89:0/2016157062 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3b4c0664a0 0x7f3b4c068960 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3b64000f70 tx=0x7f3b64007cd0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.185+0000 7f3b6fcc9640 1 -- 172.21.15.89:0/2016157062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b68071110 msgr2=0x7f3b680c3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:53.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.185+0000 7f3b6fcc9640 1 --2- 172.21.15.89:0/2016157062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b68071110 0x7f3b680c3600 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3b580b3040 tx=0x7f3b5801bad0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.185+0000 7f3b6fcc9640 1 -- 172.21.15.89:0/2016157062 shutdown_connections 2023-10-30T12:39:53.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.185+0000 7f3b6fcc9640 1 --2- 172.21.15.89:0/2016157062 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3b4c0664a0 0x7f3b4c068960 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.185+0000 7f3b6fcc9640 1 --2- 172.21.15.89:0/2016157062 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b680c6bd0 0x7f3b68151940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.185+0000 7f3b6fcc9640 1 --2- 172.21.15.89:0/2016157062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b68071110 0x7f3b680c3600 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.185+0000 7f3b6fcc9640 1 --2- 172.21.15.89:0/2016157062 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b6806f7a0 0x7f3b68157930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.185+0000 7f3b6fcc9640 1 -- 172.21.15.89:0/2016157062 >> 172.21.15.89:0/2016157062 conn(0x7f3b680c7910 msgr2=0x7f3b680c8f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:53.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.185+0000 7f3b6fcc9640 1 -- 172.21.15.89:0/2016157062 shutdown_connections 2023-10-30T12:39:53.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.185+0000 7f3b6fcc9640 1 -- 172.21.15.89:0/2016157062 wait complete. 2023-10-30T12:39:53.205 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=2) 2023-10-30T12:39:53.206 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-30T12:39:53.206 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:39:53.230 INFO:tasks.ceph.mds.b:Stopped 2023-10-30T12:39:53.231 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:39:53.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.369+0000 7fc71150f640 1 Processor -- start 2023-10-30T12:39:53.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.369+0000 7fc71150f640 1 -- start start 2023-10-30T12:39:53.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.369+0000 7fc71150f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc70c1538a0 0x7fc70c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:53.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.369+0000 7fc71150f640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc70c0ca0e0 0x7fc70c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:53.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.369+0000 7fc71150f640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc70c0cf160 0x7fc70c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:53.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.369+0000 7fc71150f640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc70c0cb800 con 0x7fc70c0cf160 2023-10-30T12:39:53.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.369+0000 7fc71150f640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc70c15dc30 con 0x7fc70c0ca0e0 2023-10-30T12:39:53.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.369+0000 7fc71150f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc70c0cb330 con 0x7fc70c1538a0 2023-10-30T12:39:53.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.369+0000 7fc70b7fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc70c0cf160 0x7fc70c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:53.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.369+0000 7fc70b7fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc70c0cf160 0x7fc70c0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33304/0 (socket says 172.21.15.89:33304) 2023-10-30T12:39:53.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.369+0000 7fc70b7fe640 1 -- 172.21.15.89:0/1478458814 learned_addr learned my addr 172.21.15.89:0/1478458814 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:39:53.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.369+0000 7fc7027fc640 1 --2- 172.21.15.89:0/1478458814 >> v2:172.21.15.192:3300/0 conn(0x7fc70c0ca0e0 0x7fc70c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:53.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.369+0000 7fc70affd640 1 --2- 172.21.15.89:0/1478458814 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc70c1538a0 0x7fc70c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:53.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.369+0000 7fc70b7fe640 1 -- 172.21.15.89:0/1478458814 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc70c1538a0 msgr2=0x7fc70c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:53.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.369+0000 7fc70b7fe640 1 --2- 172.21.15.89:0/1478458814 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc70c1538a0 0x7fc70c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.369+0000 7fc70b7fe640 1 -- 172.21.15.89:0/1478458814 >> v2:172.21.15.192:3300/0 conn(0x7fc70c0ca0e0 msgr2=0x7fc70c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:53.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.369+0000 7fc70b7fe640 1 --2- 172.21.15.89:0/1478458814 >> v2:172.21.15.192:3300/0 conn(0x7fc70c0ca0e0 0x7fc70c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.369+0000 7fc70b7fe640 1 -- 172.21.15.89:0/1478458814 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc70c0cad70 con 0x7fc70c0cf160 2023-10-30T12:39:53.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.373+0000 7fc70b7fe640 1 --2- 172.21.15.89:0/1478458814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc70c0cf160 0x7fc70c0cf590 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fc70c0ccd10 tx=0x7fc70401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93e4bf894e251d7e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:53.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.373+0000 7fc70a7fc640 1 -- 172.21.15.89:0/1478458814 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7040af400 con 0x7fc70c0cf160 2023-10-30T12:39:53.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.373+0000 7fc70a7fc640 1 -- 172.21.15.89:0/1478458814 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7040ab050 con 0x7fc70c0cf160 2023-10-30T12:39:53.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.373+0000 7fc70a7fc640 1 -- 172.21.15.89:0/1478458814 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7040af780 con 0x7fc70c0cf160 2023-10-30T12:39:53.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.373+0000 7fc71150f640 1 -- 172.21.15.89:0/1478458814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc70c0cf160 msgr2=0x7fc70c0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:53.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.373+0000 7fc71150f640 1 --2- 172.21.15.89:0/1478458814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc70c0cf160 0x7fc70c0cf590 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fc70c0ccd10 tx=0x7fc70401f1c0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.373+0000 7fc71150f640 1 -- 172.21.15.89:0/1478458814 shutdown_connections 2023-10-30T12:39:53.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.373+0000 7fc71150f640 1 --2- 172.21.15.89:0/1478458814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc70c0cf160 0x7fc70c0cf590 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.373+0000 7fc71150f640 1 --2- 172.21.15.89:0/1478458814 >> v2:172.21.15.192:3300/0 conn(0x7fc70c0ca0e0 0x7fc70c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.373+0000 7fc71150f640 1 --2- 172.21.15.89:0/1478458814 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc70c1538a0 0x7fc70c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.373+0000 7fc71150f640 1 -- 172.21.15.89:0/1478458814 >> 172.21.15.89:0/1478458814 conn(0x7fc70c06bfe0 msgr2=0x7fc70c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:53.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.373+0000 7fc71150f640 1 -- 172.21.15.89:0/1478458814 shutdown_connections 2023-10-30T12:39:53.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.373+0000 7fc71150f640 1 -- 172.21.15.89:0/1478458814 wait complete. 2023-10-30T12:39:53.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.377+0000 7fc71150f640 1 Processor -- start 2023-10-30T12:39:53.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.377+0000 7fc71150f640 1 -- start start 2023-10-30T12:39:53.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.377+0000 7fc71150f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc70c0ca0e0 0x7fc70c15d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:53.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.377+0000 7fc71150f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc70c0cf160 0x7fc70c0ce290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:53.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.377+0000 7fc71150f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc70c1538a0 0x7fc70c15a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:53.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.377+0000 7fc71150f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70c159280 con 0x7fc70c0ca0e0 2023-10-30T12:39:53.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.377+0000 7fc71150f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70c159560 con 0x7fc70c0cf160 2023-10-30T12:39:53.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.377+0000 7fc71150f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc70c1f2430 con 0x7fc70c1538a0 2023-10-30T12:39:53.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.377+0000 7fc70affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc70c0ca0e0 0x7fc70c15d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:53.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.377+0000 7fc70affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc70c0ca0e0 0x7fc70c15d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33308/0 (socket says 172.21.15.89:33308) 2023-10-30T12:39:53.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.377+0000 7fc70affd640 1 -- 172.21.15.89:0/3611396450 learned_addr learned my addr 172.21.15.89:0/3611396450 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:39:53.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.377+0000 7fc70b7fe640 1 --2- 172.21.15.89:0/3611396450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc70c1538a0 0x7fc70c15a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:53.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.377+0000 7fc7027fc640 1 --2- 172.21.15.89:0/3611396450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc70c0cf160 0x7fc70c0ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:53.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.377+0000 7fc70affd640 1 -- 172.21.15.89:0/3611396450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc70c1538a0 msgr2=0x7fc70c15a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:53.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.377+0000 7fc70affd640 1 --2- 172.21.15.89:0/3611396450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc70c1538a0 0x7fc70c15a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.377+0000 7fc70affd640 1 -- 172.21.15.89:0/3611396450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc70c0cf160 msgr2=0x7fc70c0ce290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:53.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.377+0000 7fc70affd640 1 --2- 172.21.15.89:0/3611396450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc70c0cf160 0x7fc70c0ce290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.377+0000 7fc70affd640 1 -- 172.21.15.89:0/3611396450 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc704009b30 con 0x7fc70c0ca0e0 2023-10-30T12:39:53.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.377+0000 7fc70affd640 1 --2- 172.21.15.89:0/3611396450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc70c0ca0e0 0x7fc70c15d0b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc00d5f0 tx=0x7fc6fc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:53.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.377+0000 7fc703fff640 1 -- 172.21.15.89:0/3611396450 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc0125f0 con 0x7fc70c0ca0e0 2023-10-30T12:39:53.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.377+0000 7fc703fff640 1 -- 172.21.15.89:0/3611396450 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6fc012750 con 0x7fc70c0ca0e0 2023-10-30T12:39:53.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.377+0000 7fc703fff640 1 -- 172.21.15.89:0/3611396450 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc012ae0 con 0x7fc70c0ca0e0 2023-10-30T12:39:53.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.377+0000 7fc71150f640 1 -- 172.21.15.89:0/3611396450 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc70c1f2710 con 0x7fc70c0ca0e0 2023-10-30T12:39:53.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.381+0000 7fc71150f640 1 -- 172.21.15.89:0/3611396450 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc70c0cee60 con 0x7fc70c0ca0e0 2023-10-30T12:39:53.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.381+0000 7fc71150f640 1 -- 172.21.15.89:0/3611396450 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc70c0b2a50 con 0x7fc70c0ca0e0 2023-10-30T12:39:53.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.381+0000 7fc703fff640 1 -- 172.21.15.89:0/3611396450 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc6fc012c40 con 0x7fc70c0ca0e0 2023-10-30T12:39:53.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.381+0000 7fc703fff640 1 --2- 172.21.15.89:0/3611396450 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc6ec0664a0 0x7fc6ec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:53.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.381+0000 7fc703fff640 1 -- 172.21.15.89:0/3611396450 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6176+0+0 (secure 0 0 0) 0x7fc6fc084460 con 0x7fc70c0ca0e0 2023-10-30T12:39:53.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.381+0000 7fc7027fc640 1 --2- 172.21.15.89:0/3611396450 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc6ec0664a0 0x7fc6ec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:53.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.385+0000 7fc7027fc640 1 --2- 172.21.15.89:0/3611396450 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc6ec0664a0 0x7fc6ec068960 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc70c158a50 tx=0x7fc6f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:53.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.385+0000 7fc703fff640 1 -- 172.21.15.89:0/3611396450 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc6fc051740 con 0x7fc70c0ca0e0 2023-10-30T12:39:53.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.565+0000 7fc71150f640 1 -- 172.21.15.89:0/3611396450 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc70c158ff0 con 0x7fc70c0ca0e0 2023-10-30T12:39:53.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.569+0000 7fc703fff640 1 -- 172.21.15.89:0/3611396450 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+6526 (secure 0 0 0) 0x7fc6fc007380 con 0x7fc70c0ca0e0 2023-10-30T12:39:53.568 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:39:53.568 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5129,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:39:31.166072+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:39:53.569 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 54 2023-10-30T12:39:53.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.569+0000 7fc71150f640 1 -- 172.21.15.89:0/3611396450 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc6ec0664a0 msgr2=0x7fc6ec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:53.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.569+0000 7fc71150f640 1 --2- 172.21.15.89:0/3611396450 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc6ec0664a0 0x7fc6ec068960 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc70c158a50 tx=0x7fc6f4008040 comp rx=0 tx=0).stop 2023-10-30T12:39:53.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.569+0000 7fc71150f640 1 -- 172.21.15.89:0/3611396450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc70c0ca0e0 msgr2=0x7fc70c15d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:53.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.569+0000 7fc71150f640 1 --2- 172.21.15.89:0/3611396450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc70c0ca0e0 0x7fc70c15d0b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc00d5f0 tx=0x7fc6fc00dab0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.569+0000 7fc71150f640 1 -- 172.21.15.89:0/3611396450 shutdown_connections 2023-10-30T12:39:53.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.569+0000 7fc71150f640 1 --2- 172.21.15.89:0/3611396450 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc6ec0664a0 0x7fc6ec068960 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.569+0000 7fc71150f640 1 --2- 172.21.15.89:0/3611396450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc70c0cf160 0x7fc70c0ce290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.569+0000 7fc71150f640 1 --2- 172.21.15.89:0/3611396450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc70c1538a0 0x7fc70c15a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.569+0000 7fc71150f640 1 --2- 172.21.15.89:0/3611396450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc70c0ca0e0 0x7fc70c15d0b0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:53.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.569+0000 7fc71150f640 1 -- 172.21.15.89:0/3611396450 >> 172.21.15.89:0/3611396450 conn(0x7fc70c06bfe0 msgr2=0x7fc70c150ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:53.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.569+0000 7fc71150f640 1 -- 172.21.15.89:0/3611396450 shutdown_connections 2023-10-30T12:39:53.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:53.569+0000 7fc71150f640 1 -- 172.21.15.89:0/3611396450 wait complete. 2023-10-30T12:39:53.589 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T12:39:55.589 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:39:55.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.725+0000 7f996f6d6640 1 Processor -- start 2023-10-30T12:39:55.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.729+0000 7f996f6d6640 1 -- start start 2023-10-30T12:39:55.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.729+0000 7f996f6d6640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9968069e90 0x7f996815a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:55.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.729+0000 7f996f6d6640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f996806a270 0x7f99680c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:55.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.729+0000 7f996f6d6640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99680c6c10 con 0x7f996806cb80 2023-10-30T12:39:55.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.729+0000 7f996f6d6640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99680c8570 con 0x7f996806a270 2023-10-30T12:39:55.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.729+0000 7f996f6d6640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99680c6ef0 con 0x7f9968069e90 2023-10-30T12:39:55.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.729+0000 7f996cc71640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f996806a270 0x7f99680c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:55.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.729+0000 7f996cc71640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f996806a270 0x7f99680c9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33316/0 (socket says 172.21.15.89:33316) 2023-10-30T12:39:55.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.729+0000 7f996cc71640 1 -- 172.21.15.89:0/882300892 learned_addr learned my addr 172.21.15.89:0/882300892 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:39:55.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.729+0000 7f996d472640 1 --2- 172.21.15.89:0/882300892 >> v2:172.21.15.192:3300/0 conn(0x7f9968069e90 0x7f996815a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:55.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.729+0000 7f996cc71640 1 -- 172.21.15.89:0/882300892 >> v2:172.21.15.192:3300/0 conn(0x7f9968069e90 msgr2=0x7f996815a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:55.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.729+0000 7f996cc71640 1 --2- 172.21.15.89:0/882300892 >> v2:172.21.15.192:3300/0 conn(0x7f9968069e90 0x7f996815a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:55.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.729+0000 7f996cc71640 1 -- 172.21.15.89:0/882300892 >> v1:172.21.15.192:6789/0 conn(0x7f996806cb80 legacy=0x7f996806cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:55.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.729+0000 7f996cc71640 1 -- 172.21.15.89:0/882300892 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99680c71d0 con 0x7f996806a270 2023-10-30T12:39:55.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.733+0000 7f996cc71640 1 --2- 172.21.15.89:0/882300892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f996806a270 0x7f99680c9cb0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f996806ed20 tx=0x7f995801c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8adb8067f62ab00a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:55.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.733+0000 7f995ffff640 1 -- 172.21.15.89:0/882300892 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99580ac410 con 0x7f996806a270 2023-10-30T12:39:55.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.733+0000 7f995ffff640 1 -- 172.21.15.89:0/882300892 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99580ac570 con 0x7f996806a270 2023-10-30T12:39:55.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.733+0000 7f995ffff640 1 -- 172.21.15.89:0/882300892 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99580ac900 con 0x7f996806a270 2023-10-30T12:39:55.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.733+0000 7f996f6d6640 1 -- 172.21.15.89:0/882300892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f996806a270 msgr2=0x7f99680c9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:55.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.733+0000 7f996f6d6640 1 --2- 172.21.15.89:0/882300892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f996806a270 0x7f99680c9cb0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f996806ed20 tx=0x7f995801c4c0 comp rx=0 tx=0).stop 2023-10-30T12:39:55.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.733+0000 7f996f6d6640 1 -- 172.21.15.89:0/882300892 shutdown_connections 2023-10-30T12:39:55.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.733+0000 7f996f6d6640 1 --2- 172.21.15.89:0/882300892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f996806a270 0x7f99680c9cb0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:55.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.733+0000 7f996f6d6640 1 --2- 172.21.15.89:0/882300892 >> v2:172.21.15.192:3300/0 conn(0x7f9968069e90 0x7f996815a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:55.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.733+0000 7f996f6d6640 1 -- 172.21.15.89:0/882300892 >> 172.21.15.89:0/882300892 conn(0x7f996806dfd0 msgr2=0x7f996806e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:55.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.733+0000 7f996f6d6640 1 -- 172.21.15.89:0/882300892 shutdown_connections 2023-10-30T12:39:55.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.733+0000 7f996f6d6640 1 -- 172.21.15.89:0/882300892 wait complete. 2023-10-30T12:39:55.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.733+0000 7f996f6d6640 1 Processor -- start 2023-10-30T12:39:55.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.733+0000 7f996f6d6640 1 -- start start 2023-10-30T12:39:55.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.737+0000 7f996f6d6640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9968069e90 0x7f99681f2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:55.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.737+0000 7f996f6d6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f996806a270 0x7f99681492b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:55.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.737+0000 7f996f6d6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f996806cb80 0x7f9968071690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:55.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.737+0000 7f996f6d6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996815b250 con 0x7f996806cb80 2023-10-30T12:39:55.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.737+0000 7f996f6d6640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9968152520 con 0x7f996806a270 2023-10-30T12:39:55.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.737+0000 7f996f6d6640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9968151220 con 0x7f9968069e90 2023-10-30T12:39:55.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.737+0000 7f996dc73640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f996806cb80 0x7f9968071690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:55.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.737+0000 7f996dc73640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f996806cb80 0x7f9968071690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33332/0 (socket says 172.21.15.89:33332) 2023-10-30T12:39:55.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.737+0000 7f996dc73640 1 -- 172.21.15.89:0/4018282474 learned_addr learned my addr 172.21.15.89:0/4018282474 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:39:55.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.737+0000 7f996d472640 1 --2- 172.21.15.89:0/4018282474 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9968069e90 0x7f99681f2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:55.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.737+0000 7f996cc71640 1 --2- 172.21.15.89:0/4018282474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f996806a270 0x7f99681492b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:55.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.737+0000 7f996dc73640 1 -- 172.21.15.89:0/4018282474 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9968069e90 msgr2=0x7f99681f2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:55.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.737+0000 7f996dc73640 1 --2- 172.21.15.89:0/4018282474 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9968069e90 0x7f99681f2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:55.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.737+0000 7f996dc73640 1 -- 172.21.15.89:0/4018282474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f996806a270 msgr2=0x7f99681492b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:55.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.737+0000 7f996dc73640 1 --2- 172.21.15.89:0/4018282474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f996806a270 0x7f99681492b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:55.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.737+0000 7f996dc73640 1 -- 172.21.15.89:0/4018282474 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99580094d0 con 0x7f996806cb80 2023-10-30T12:39:55.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.737+0000 7f996cc71640 1 --2- 172.21.15.89:0/4018282474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f996806a270 0x7f99681492b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:39:55.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.737+0000 7f996dc73640 1 --2- 172.21.15.89:0/4018282474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f996806cb80 0x7f9968071690 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f996000b970 tx=0x7f996000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:55.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.737+0000 7f995dffb640 1 -- 172.21.15.89:0/4018282474 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996000cb40 con 0x7f996806cb80 2023-10-30T12:39:55.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.737+0000 7f995dffb640 1 -- 172.21.15.89:0/4018282474 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f996000cca0 con 0x7f996806cb80 2023-10-30T12:39:55.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.737+0000 7f995dffb640 1 -- 172.21.15.89:0/4018282474 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9960014680 con 0x7f996806cb80 2023-10-30T12:39:55.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.737+0000 7f996f6d6640 1 -- 172.21.15.89:0/4018282474 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9968152770 con 0x7f996806cb80 2023-10-30T12:39:55.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.737+0000 7f996f6d6640 1 -- 172.21.15.89:0/4018282474 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9968150eb0 con 0x7f996806cb80 2023-10-30T12:39:55.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.737+0000 7f996f6d6640 1 -- 172.21.15.89:0/4018282474 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9968152c90 con 0x7f996806cb80 2023-10-30T12:39:55.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.737+0000 7f995dffb640 1 -- 172.21.15.89:0/4018282474 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f996001d3b0 con 0x7f996806cb80 2023-10-30T12:39:55.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.741+0000 7f995dffb640 1 --2- 172.21.15.89:0/4018282474 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f994c0664a0 0x7f994c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:55.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.741+0000 7f995dffb640 1 -- 172.21.15.89:0/4018282474 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6176+0+0 (secure 0 0 0) 0x7f9960083d60 con 0x7f996806cb80 2023-10-30T12:39:55.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.741+0000 7f996d472640 1 --2- 172.21.15.89:0/4018282474 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f994c0664a0 0x7f994c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:55.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.741+0000 7f996d472640 1 --2- 172.21.15.89:0/4018282474 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f994c0664a0 0x7f994c068960 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9964007a60 tx=0x7f9964008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:55.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.745+0000 7f995dffb640 1 -- 172.21.15.89:0/4018282474 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9960051150 con 0x7f996806cb80 2023-10-30T12:39:55.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.925+0000 7f996f6d6640 1 -- 172.21.15.89:0/4018282474 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9968151390 con 0x7f996806cb80 2023-10-30T12:39:55.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.929+0000 7f995dffb640 1 -- 172.21.15.89:0/4018282474 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+6527 (secure 0 0 0) 0x7f9960054770 con 0x7f996806cb80 2023-10-30T12:39:55.928 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:39:55.928 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5129,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:39:55.163412+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:39:55.928 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 55 2023-10-30T12:39:55.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.929+0000 7f996f6d6640 1 -- 172.21.15.89:0/4018282474 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f994c0664a0 msgr2=0x7f994c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:55.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.929+0000 7f996f6d6640 1 --2- 172.21.15.89:0/4018282474 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f994c0664a0 0x7f994c068960 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9964007a60 tx=0x7f9964008040 comp rx=0 tx=0).stop 2023-10-30T12:39:55.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.929+0000 7f996f6d6640 1 -- 172.21.15.89:0/4018282474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f996806cb80 msgr2=0x7f9968071690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:55.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.929+0000 7f996f6d6640 1 --2- 172.21.15.89:0/4018282474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f996806cb80 0x7f9968071690 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f996000b970 tx=0x7f996000be30 comp rx=0 tx=0).stop 2023-10-30T12:39:55.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.929+0000 7f996f6d6640 1 -- 172.21.15.89:0/4018282474 shutdown_connections 2023-10-30T12:39:55.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.929+0000 7f996f6d6640 1 --2- 172.21.15.89:0/4018282474 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f994c0664a0 0x7f994c068960 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:55.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.929+0000 7f996f6d6640 1 --2- 172.21.15.89:0/4018282474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f996806a270 0x7f99681492b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:55.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.929+0000 7f996f6d6640 1 --2- 172.21.15.89:0/4018282474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f996806cb80 0x7f9968071690 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:55.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.929+0000 7f996f6d6640 1 --2- 172.21.15.89:0/4018282474 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9968069e90 0x7f99681f2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:55.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.929+0000 7f996f6d6640 1 -- 172.21.15.89:0/4018282474 >> 172.21.15.89:0/4018282474 conn(0x7f996806dfd0 msgr2=0x7f996814f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:55.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.929+0000 7f996f6d6640 1 -- 172.21.15.89:0/4018282474 shutdown_connections 2023-10-30T12:39:55.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:55.929+0000 7f996f6d6640 1 -- 172.21.15.89:0/4018282474 wait complete. 2023-10-30T12:39:55.951 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T12:39:57.952 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:39:58.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.089+0000 7f21997d3640 1 Processor -- start 2023-10-30T12:39:58.091 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.089+0000 7f21997d3640 1 -- start start 2023-10-30T12:39:58.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.093+0000 7f21997d3640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2194159e60 0x7f219406f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:58.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.093+0000 7f21997d3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f21940bfec0 0x7f21940c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:58.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.093+0000 7f21997d3640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f219406a720 0x7f219406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:58.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.093+0000 7f21997d3640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f219406fa80 con 0x7f219406a720 2023-10-30T12:39:58.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.093+0000 7f21997d3640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2194071290 con 0x7f2194159e60 2023-10-30T12:39:58.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.093+0000 7f21997d3640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2194071570 con 0x7f21940bfec0 2023-10-30T12:39:58.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.093+0000 7f21937fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f219406a720 0x7f219406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:58.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.093+0000 7f21937fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f219406a720 0x7f219406abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33334/0 (socket says 172.21.15.89:33334) 2023-10-30T12:39:58.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.093+0000 7f21937fe640 1 -- 172.21.15.89:0/79526458 learned_addr learned my addr 172.21.15.89:0/79526458 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:39:58.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.093+0000 7f218a7fc640 1 --2- 172.21.15.89:0/79526458 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f21940bfec0 0x7f21940c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:58.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.093+0000 7f2192ffd640 1 --2- 172.21.15.89:0/79526458 >> v2:172.21.15.192:3300/0 conn(0x7f2194159e60 0x7f219406f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:58.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.093+0000 7f21937fe640 1 -- 172.21.15.89:0/79526458 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f21940bfec0 msgr2=0x7f21940c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:58.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.093+0000 7f21937fe640 1 --2- 172.21.15.89:0/79526458 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f21940bfec0 0x7f21940c1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:58.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.093+0000 7f21937fe640 1 -- 172.21.15.89:0/79526458 >> v2:172.21.15.192:3300/0 conn(0x7f2194159e60 msgr2=0x7f219406f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:39:58.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.093+0000 7f21937fe640 1 --2- 172.21.15.89:0/79526458 >> v2:172.21.15.192:3300/0 conn(0x7f2194159e60 0x7f219406f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:58.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.093+0000 7f21937fe640 1 -- 172.21.15.89:0/79526458 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2194071850 con 0x7f219406a720 2023-10-30T12:39:58.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.093+0000 7f21937fe640 1 --2- 172.21.15.89:0/79526458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f219406a720 0x7f219406abb0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f21940c8950 tx=0x7f218c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ac8cfbbe05b9a3f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:58.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.097+0000 7f21927fc640 1 -- 172.21.15.89:0/79526458 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f218c0a89b0 con 0x7f219406a720 2023-10-30T12:39:58.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.097+0000 7f21927fc640 1 -- 172.21.15.89:0/79526458 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f218c0a8b10 con 0x7f219406a720 2023-10-30T12:39:58.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.097+0000 7f21927fc640 1 -- 172.21.15.89:0/79526458 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f218c0a8ea0 con 0x7f219406a720 2023-10-30T12:39:58.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.097+0000 7f21997d3640 1 -- 172.21.15.89:0/79526458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f219406a720 msgr2=0x7f219406abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:58.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.097+0000 7f21997d3640 1 --2- 172.21.15.89:0/79526458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f219406a720 0x7f219406abb0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f21940c8950 tx=0x7f218c01ecf0 comp rx=0 tx=0).stop 2023-10-30T12:39:58.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.097+0000 7f21997d3640 1 -- 172.21.15.89:0/79526458 shutdown_connections 2023-10-30T12:39:58.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.097+0000 7f21997d3640 1 --2- 172.21.15.89:0/79526458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f219406a720 0x7f219406abb0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:58.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.097+0000 7f21997d3640 1 --2- 172.21.15.89:0/79526458 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f21940bfec0 0x7f21940c1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:58.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.097+0000 7f21997d3640 1 --2- 172.21.15.89:0/79526458 >> v2:172.21.15.192:3300/0 conn(0x7f2194159e60 0x7f219406f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:58.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.097+0000 7f21997d3640 1 -- 172.21.15.89:0/79526458 >> 172.21.15.89:0/79526458 conn(0x7f21940c7910 msgr2=0x7f21940c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:58.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.097+0000 7f21997d3640 1 -- 172.21.15.89:0/79526458 shutdown_connections 2023-10-30T12:39:58.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.097+0000 7f21997d3640 1 -- 172.21.15.89:0/79526458 wait complete. 2023-10-30T12:39:58.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.097+0000 7f21997d3640 1 Processor -- start 2023-10-30T12:39:58.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.097+0000 7f21997d3640 1 -- start start 2023-10-30T12:39:58.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.097+0000 7f21997d3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f219406a720 0x7f219414dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:58.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.097+0000 7f21997d3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21940bfec0 0x7f21940adc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:58.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.097+0000 7f21997d3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2194159e60 0x7f21940c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:58.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.097+0000 7f21997d3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21940af1b0 con 0x7f21940bfec0 2023-10-30T12:39:58.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.097+0000 7f21997d3640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21940b0390 con 0x7f2194159e60 2023-10-30T12:39:58.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.097+0000 7f21997d3640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21941574a0 con 0x7f219406a720 2023-10-30T12:39:58.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.097+0000 7f218a7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21940bfec0 0x7f21940adc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:58.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.097+0000 7f218a7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21940bfec0 0x7f21940adc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33338/0 (socket says 172.21.15.89:33338) 2023-10-30T12:39:58.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.097+0000 7f218a7fc640 1 -- 172.21.15.89:0/988453165 learned_addr learned my addr 172.21.15.89:0/988453165 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:39:58.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.101+0000 7f2192ffd640 1 --2- 172.21.15.89:0/988453165 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f219406a720 0x7f219414dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:58.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.101+0000 7f21937fe640 1 --2- 172.21.15.89:0/988453165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2194159e60 0x7f21940c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:58.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.101+0000 7f218a7fc640 1 -- 172.21.15.89:0/988453165 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f219406a720 msgr2=0x7f219414dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:58.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.101+0000 7f218a7fc640 1 --2- 172.21.15.89:0/988453165 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f219406a720 0x7f219414dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:58.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.101+0000 7f218a7fc640 1 -- 172.21.15.89:0/988453165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2194159e60 msgr2=0x7f21940c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:58.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.101+0000 7f218a7fc640 1 --2- 172.21.15.89:0/988453165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2194159e60 0x7f21940c8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:58.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.101+0000 7f218a7fc640 1 -- 172.21.15.89:0/988453165 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f218c0094d0 con 0x7f21940bfec0 2023-10-30T12:39:58.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.101+0000 7f218a7fc640 1 --2- 172.21.15.89:0/988453165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21940bfec0 0x7f21940adc90 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f217c009e30 tx=0x7f217c0075c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:58.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.101+0000 7f218bfff640 1 -- 172.21.15.89:0/988453165 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217c011070 con 0x7f21940bfec0 2023-10-30T12:39:58.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.101+0000 7f218bfff640 1 -- 172.21.15.89:0/988453165 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f217c016470 con 0x7f21940bfec0 2023-10-30T12:39:58.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.101+0000 7f21997d3640 1 -- 172.21.15.89:0/988453165 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2194157760 con 0x7f21940bfec0 2023-10-30T12:39:58.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.101+0000 7f218bfff640 1 -- 172.21.15.89:0/988453165 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217c00add0 con 0x7f21940bfec0 2023-10-30T12:39:58.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.101+0000 7f21997d3640 1 -- 172.21.15.89:0/988453165 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2194157d80 con 0x7f21940bfec0 2023-10-30T12:39:58.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.101+0000 7f21997d3640 1 -- 172.21.15.89:0/988453165 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f219414e150 con 0x7f21940bfec0 2023-10-30T12:39:58.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.101+0000 7f218bfff640 1 -- 172.21.15.89:0/988453165 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f217c0167e0 con 0x7f21940bfec0 2023-10-30T12:39:58.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.101+0000 7f218bfff640 1 --2- 172.21.15.89:0/988453165 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f21740664d0 0x7f2174068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:39:58.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.101+0000 7f218bfff640 1 -- 172.21.15.89:0/988453165 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6176+0+0 (secure 0 0 0) 0x7f217c0842a0 con 0x7f21940bfec0 2023-10-30T12:39:58.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.105+0000 7f2192ffd640 1 --2- 172.21.15.89:0/988453165 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f21740664d0 0x7f2174068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:39:58.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.105+0000 7f2192ffd640 1 --2- 172.21.15.89:0/988453165 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f21740664d0 0x7f2174068990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f218400b930 tx=0x7f2184008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:39:58.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.105+0000 7f218bfff640 1 -- 172.21.15.89:0/988453165 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f217c051550 con 0x7f21940bfec0 2023-10-30T12:39:58.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.297+0000 7f21997d3640 1 -- 172.21.15.89:0/988453165 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f219404e170 con 0x7f21940bfec0 2023-10-30T12:39:58.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.297+0000 7f218bfff640 1 -- 172.21.15.89:0/988453165 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+6527 (secure 0 0 0) 0x7f217c01b090 con 0x7f21940bfec0 2023-10-30T12:39:58.300 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:39:58.300 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5129,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:39:55.163412+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:39:58.301 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 55 2023-10-30T12:39:58.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.301+0000 7f21997d3640 1 -- 172.21.15.89:0/988453165 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f21740664d0 msgr2=0x7f2174068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:58.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.301+0000 7f21997d3640 1 --2- 172.21.15.89:0/988453165 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f21740664d0 0x7f2174068990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f218400b930 tx=0x7f2184008040 comp rx=0 tx=0).stop 2023-10-30T12:39:58.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.301+0000 7f21997d3640 1 -- 172.21.15.89:0/988453165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21940bfec0 msgr2=0x7f21940adc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:39:58.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.301+0000 7f21997d3640 1 --2- 172.21.15.89:0/988453165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21940bfec0 0x7f21940adc90 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f217c009e30 tx=0x7f217c0075c0 comp rx=0 tx=0).stop 2023-10-30T12:39:58.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.301+0000 7f21997d3640 1 -- 172.21.15.89:0/988453165 shutdown_connections 2023-10-30T12:39:58.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.301+0000 7f21997d3640 1 --2- 172.21.15.89:0/988453165 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f21740664d0 0x7f2174068990 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:58.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.301+0000 7f21997d3640 1 --2- 172.21.15.89:0/988453165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2194159e60 0x7f21940c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:58.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.301+0000 7f21997d3640 1 --2- 172.21.15.89:0/988453165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21940bfec0 0x7f21940adc90 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:58.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.301+0000 7f21997d3640 1 --2- 172.21.15.89:0/988453165 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f219406a720 0x7f219414dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:39:58.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.301+0000 7f21997d3640 1 -- 172.21.15.89:0/988453165 >> 172.21.15.89:0/988453165 conn(0x7f21940c7910 msgr2=0x7f2194155f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:39:58.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.301+0000 7f21997d3640 1 -- 172.21.15.89:0/988453165 shutdown_connections 2023-10-30T12:39:58.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:39:58.301+0000 7f21997d3640 1 -- 172.21.15.89:0/988453165 wait complete. 2023-10-30T12:39:58.319 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T12:40:00.320 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:00.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.461+0000 7f6ba65d3640 1 Processor -- start 2023-10-30T12:40:00.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.461+0000 7f6ba65d3640 1 -- start start 2023-10-30T12:40:00.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.461+0000 7f6ba65d3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba00cf160 0x7f6ba00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:00.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.461+0000 7f6ba65d3640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba00cb800 con 0x7f6ba015d6f0 2023-10-30T12:40:00.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.461+0000 7f6ba65d3640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba00cb330 con 0x7f6ba01538a0 2023-10-30T12:40:00.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.461+0000 7f6ba65d3640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba00cad70 con 0x7f6ba00cf160 2023-10-30T12:40:00.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.461+0000 7f6b9f7fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6ba015d6f0 0x7f6ba015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:60946/0 (socket says 172.21.15.89:60946) 2023-10-30T12:40:00.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.461+0000 7f6b9f7fe640 1 -- 172.21.15.89:0/1029429025 learned_addr learned my addr 172.21.15.89:0/1029429025 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:40:00.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.461+0000 7f6ba4b70640 1 --2- 172.21.15.89:0/1029429025 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba00cf160 0x7f6ba00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:00.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.461+0000 7f6b9effd640 1 -- 172.21.15.89:0/1029429025 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3445597641 0 0) 0x7f6ba00cb800 con 0x7f6ba015d6f0 2023-10-30T12:40:00.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.465+0000 7f6b9effd640 1 -- 172.21.15.89:0/1029429025 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b88003530 con 0x7f6ba015d6f0 2023-10-30T12:40:00.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.465+0000 7f6b9effd640 1 -- 172.21.15.89:0/1029429025 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1091182659 0 0) 0x7f6ba00cb330 con 0x7f6ba01538a0 2023-10-30T12:40:00.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.465+0000 7f6b9effd640 1 -- 172.21.15.89:0/1029429025 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ba00cb800 con 0x7f6ba01538a0 2023-10-30T12:40:00.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.465+0000 7f6b9effd640 1 -- 172.21.15.89:0/1029429025 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1666121861 0 0) 0x7f6b88003530 con 0x7f6ba015d6f0 2023-10-30T12:40:00.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.465+0000 7f6ba4b70640 1 -- 172.21.15.89:0/1029429025 >> v1:172.21.15.192:6789/0 conn(0x7f6ba01538a0 legacy=0x7f6ba00ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:00.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.465+0000 7f6ba4b70640 1 -- 172.21.15.89:0/1029429025 >> v1:172.21.15.89:6789/0 conn(0x7f6ba015d6f0 legacy=0x7f6ba015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:00.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.465+0000 7f6ba4b70640 1 -- 172.21.15.89:0/1029429025 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba00cb050 con 0x7f6ba00cf160 2023-10-30T12:40:00.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.465+0000 7f6ba4b70640 1 --2- 172.21.15.89:0/1029429025 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba00cf160 0x7f6ba00cf590 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6b98009dc0 tx=0x7f6b9801f080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=218c6b3c8d779ab3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:00.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.465+0000 7f6b9effd640 1 -- 172.21.15.89:0/1029429025 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b94001680 con 0x7f6ba00cf160 2023-10-30T12:40:00.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.465+0000 7f6b9effd640 1 -- 172.21.15.89:0/1029429025 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b980a8c10 con 0x7f6ba00cf160 2023-10-30T12:40:00.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.465+0000 7f6b9effd640 1 -- 172.21.15.89:0/1029429025 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b980b0440 con 0x7f6ba00cf160 2023-10-30T12:40:00.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.469+0000 7f6ba65d3640 1 -- 172.21.15.89:0/1029429025 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba00cf160 msgr2=0x7f6ba00cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:00.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.469+0000 7f6ba65d3640 1 --2- 172.21.15.89:0/1029429025 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba00cf160 0x7f6ba00cf590 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6b98009dc0 tx=0x7f6b9801f080 comp rx=0 tx=0).stop 2023-10-30T12:40:00.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.469+0000 7f6ba65d3640 1 -- 172.21.15.89:0/1029429025 shutdown_connections 2023-10-30T12:40:00.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.469+0000 7f6ba65d3640 1 --2- 172.21.15.89:0/1029429025 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba00cf160 0x7f6ba00cf590 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:00.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.469+0000 7f6ba65d3640 1 -- 172.21.15.89:0/1029429025 >> 172.21.15.89:0/1029429025 conn(0x7f6ba006bfe0 msgr2=0x7f6ba006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:00.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.469+0000 7f6ba65d3640 1 -- 172.21.15.89:0/1029429025 shutdown_connections 2023-10-30T12:40:00.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.469+0000 7f6ba65d3640 1 -- 172.21.15.89:0/1029429025 wait complete. 2023-10-30T12:40:00.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.469+0000 7f6ba65d3640 1 Processor -- start 2023-10-30T12:40:00.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.469+0000 7f6ba65d3640 1 -- start start 2023-10-30T12:40:00.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.469+0000 7f6ba65d3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba00cf160 0x7f6ba00cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:00.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.469+0000 7f6ba65d3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ba01538a0 0x7f6ba00ce230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:00.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.469+0000 7f6ba65d3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ba015d6f0 0x7f6ba014cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:00.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.469+0000 7f6ba65d3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba00cd8d0 con 0x7f6ba015d6f0 2023-10-30T12:40:00.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.469+0000 7f6ba65d3640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba0159270 con 0x7f6ba01538a0 2023-10-30T12:40:00.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.469+0000 7f6ba65d3640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba0158530 con 0x7f6ba00cf160 2023-10-30T12:40:00.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.469+0000 7f6ba4b70640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ba015d6f0 0x7f6ba014cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:00.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.469+0000 7f6ba4b70640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ba015d6f0 0x7f6ba014cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33900/0 (socket says 172.21.15.89:33900) 2023-10-30T12:40:00.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.469+0000 7f6ba4b70640 1 -- 172.21.15.89:0/1670665155 learned_addr learned my addr 172.21.15.89:0/1670665155 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:40:00.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.469+0000 7f6b9ffff640 1 --2- 172.21.15.89:0/1670665155 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba00cf160 0x7f6ba00cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:00.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.469+0000 7f6b9f7fe640 1 --2- 172.21.15.89:0/1670665155 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ba01538a0 0x7f6ba00ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:00.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.473+0000 7f6ba4b70640 1 -- 172.21.15.89:0/1670665155 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba00cf160 msgr2=0x7f6ba00cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:00.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.473+0000 7f6ba4b70640 1 --2- 172.21.15.89:0/1670665155 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba00cf160 0x7f6ba00cdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:00.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.473+0000 7f6ba4b70640 1 -- 172.21.15.89:0/1670665155 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ba01538a0 msgr2=0x7f6ba00ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:00.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.473+0000 7f6ba4b70640 1 --2- 172.21.15.89:0/1670665155 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ba01538a0 0x7f6ba00ce230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:00.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.473+0000 7f6ba4b70640 1 -- 172.21.15.89:0/1670665155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b98009a70 con 0x7f6ba015d6f0 2023-10-30T12:40:00.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.473+0000 7f6b9f7fe640 1 --2- 172.21.15.89:0/1670665155 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ba01538a0 0x7f6ba00ce230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:40:00.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.473+0000 7f6ba4b70640 1 --2- 172.21.15.89:0/1670665155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ba015d6f0 0x7f6ba014cc70 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f6b9801d730 tx=0x7f6b9801be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:00.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.473+0000 7f6b9cff9640 1 -- 172.21.15.89:0/1670665155 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b980c0070 con 0x7f6ba015d6f0 2023-10-30T12:40:00.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.473+0000 7f6b9cff9640 1 -- 172.21.15.89:0/1670665155 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b98009450 con 0x7f6ba015d6f0 2023-10-30T12:40:00.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.473+0000 7f6b9cff9640 1 -- 172.21.15.89:0/1670665155 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b98004450 con 0x7f6ba015d6f0 2023-10-30T12:40:00.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.473+0000 7f6ba65d3640 1 -- 172.21.15.89:0/1670665155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ba00cd050 con 0x7f6ba015d6f0 2023-10-30T12:40:00.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.473+0000 7f6ba65d3640 1 -- 172.21.15.89:0/1670665155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ba0158250 con 0x7f6ba015d6f0 2023-10-30T12:40:00.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.473+0000 7f6ba65d3640 1 -- 172.21.15.89:0/1670665155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ba015db00 con 0x7f6ba015d6f0 2023-10-30T12:40:00.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.473+0000 7f6b9cff9640 1 -- 172.21.15.89:0/1670665155 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6b980b7030 con 0x7f6ba015d6f0 2023-10-30T12:40:00.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.473+0000 7f6b9cff9640 1 --2- 172.21.15.89:0/1670665155 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6b8c0664d0 0x7f6b8c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:00.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.473+0000 7f6b9cff9640 1 -- 172.21.15.89:0/1670665155 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6176+0+0 (secure 0 0 0) 0x7f6b98121cd0 con 0x7f6ba015d6f0 2023-10-30T12:40:00.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.477+0000 7f6b9ffff640 1 --2- 172.21.15.89:0/1670665155 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6b8c0664d0 0x7f6b8c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:00.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.477+0000 7f6b9ffff640 1 --2- 172.21.15.89:0/1670665155 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6b8c0664d0 0x7f6b8c068990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6b900011f0 tx=0x7f6b90009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:00.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.477+0000 7f6b9cff9640 1 -- 172.21.15.89:0/1670665155 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6ba015db00 con 0x7f6ba015d6f0 2023-10-30T12:40:00.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.657+0000 7f6ba65d3640 1 -- 172.21.15.89:0/1670665155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ba00cda10 con 0x7f6ba015d6f0 2023-10-30T12:40:00.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.657+0000 7f6b9cff9640 1 -- 172.21.15.89:0/1670665155 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6529 (secure 0 0 0) 0x7f6ba00cda10 con 0x7f6ba015d6f0 2023-10-30T12:40:00.658 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:40:00.658 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5129,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:40:00.157769+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:00.659 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 56 2023-10-30T12:40:00.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.661+0000 7f6ba65d3640 1 -- 172.21.15.89:0/1670665155 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6b8c0664d0 msgr2=0x7f6b8c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:00.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.661+0000 7f6ba65d3640 1 --2- 172.21.15.89:0/1670665155 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6b8c0664d0 0x7f6b8c068990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6b900011f0 tx=0x7f6b90009040 comp rx=0 tx=0).stop 2023-10-30T12:40:00.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.661+0000 7f6ba65d3640 1 -- 172.21.15.89:0/1670665155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ba015d6f0 msgr2=0x7f6ba014cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:00.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.661+0000 7f6ba65d3640 1 --2- 172.21.15.89:0/1670665155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ba015d6f0 0x7f6ba014cc70 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f6b9801d730 tx=0x7f6b9801be20 comp rx=0 tx=0).stop 2023-10-30T12:40:00.661 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.661+0000 7f6ba65d3640 1 -- 172.21.15.89:0/1670665155 shutdown_connections 2023-10-30T12:40:00.661 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.661+0000 7f6ba65d3640 1 --2- 172.21.15.89:0/1670665155 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6b8c0664d0 0x7f6b8c068990 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:00.661 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.661+0000 7f6ba65d3640 1 --2- 172.21.15.89:0/1670665155 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ba01538a0 0x7f6ba00ce230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:00.661 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.661+0000 7f6ba65d3640 1 --2- 172.21.15.89:0/1670665155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ba015d6f0 0x7f6ba014cc70 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:00.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.661+0000 7f6ba65d3640 1 --2- 172.21.15.89:0/1670665155 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba00cf160 0x7f6ba00cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:00.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.661+0000 7f6ba65d3640 1 -- 172.21.15.89:0/1670665155 >> 172.21.15.89:0/1670665155 conn(0x7f6ba006bfe0 msgr2=0x7f6ba015aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:00.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.661+0000 7f6ba65d3640 1 -- 172.21.15.89:0/1670665155 shutdown_connections 2023-10-30T12:40:00.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:00.661+0000 7f6ba65d3640 1 -- 172.21.15.89:0/1670665155 wait complete. 2023-10-30T12:40:00.679 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T12:40:02.680 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:02.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.833+0000 7f590fdd9640 1 Processor -- start 2023-10-30T12:40:02.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.833+0000 7f590fdd9640 1 -- start start 2023-10-30T12:40:02.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.833+0000 7f590fdd9640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f59080c6b80 0x7f59080bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:02.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.833+0000 7f590fdd9640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59080c0a50 con 0x7f59080c6f60 2023-10-30T12:40:02.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.833+0000 7f590fdd9640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f590806f6e0 con 0x7f590813f4a0 2023-10-30T12:40:02.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.833+0000 7f590fdd9640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f590806f9c0 con 0x7f59080c6b80 2023-10-30T12:40:02.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.833+0000 7f590d374640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f59080c6f60 0x7f590813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:60958/0 (socket says 172.21.15.89:60958) 2023-10-30T12:40:02.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.833+0000 7f590d374640 1 -- 172.21.15.89:0/944057767 learned_addr learned my addr 172.21.15.89:0/944057767 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:40:02.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.833+0000 7f590db75640 1 --2- 172.21.15.89:0/944057767 >> v2:172.21.15.192:3300/0 conn(0x7f59080c6b80 0x7f59080bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:02.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.833+0000 7f590cb73640 1 -- 172.21.15.89:0/944057767 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2062725232 0 0) 0x7f59080c0a50 con 0x7f59080c6f60 2023-10-30T12:40:02.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.833+0000 7f590cb73640 1 -- 172.21.15.89:0/944057767 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58f4005740 con 0x7f59080c6f60 2023-10-30T12:40:02.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.833+0000 7f590cb73640 1 -- 172.21.15.89:0/944057767 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1347433429 0 0) 0x7f58f4005740 con 0x7f59080c6f60 2023-10-30T12:40:02.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.833+0000 7f590cb73640 1 -- 172.21.15.89:0/944057767 >> v2:172.21.15.192:3300/0 conn(0x7f59080c6b80 msgr2=0x7f59080bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:02.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.833+0000 7f590cb73640 1 --2- 172.21.15.89:0/944057767 >> v2:172.21.15.192:3300/0 conn(0x7f59080c6b80 0x7f59080bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:02.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.833+0000 7f590cb73640 1 -- 172.21.15.89:0/944057767 >> v1:172.21.15.192:6789/0 conn(0x7f590813f4a0 legacy=0x7f59081566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:02.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.833+0000 7f590cb73640 1 -- 172.21.15.89:0/944057767 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f590806fca0 con 0x7f59080c6f60 2023-10-30T12:40:02.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.833+0000 7f590cb73640 1 -- 172.21.15.89:0/944057767 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f58f8001680 con 0x7f59080c6f60 2023-10-30T12:40:02.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.833+0000 7f590cb73640 1 -- 172.21.15.89:0/944057767 >> v1:172.21.15.89:6789/0 conn(0x7f59080c6f60 legacy=0x7f590813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:02.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.833+0000 7f590cb73640 1 --2- 172.21.15.89:0/944057767 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58f4005b90 0x7f58f4005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:02.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.833+0000 7f590cb73640 1 --2- 172.21.15.89:0/944057767 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58f40069e0 0x7f58f4006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:02.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.833+0000 7f590cb73640 1 --2- 172.21.15.89:0/944057767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58f40073a0 0x7f58f400dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:02.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.833+0000 7f590cb73640 1 -- 172.21.15.89:0/944057767 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590806f9c0 con 0x7f58f40073a0 2023-10-30T12:40:02.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.833+0000 7f590cb73640 1 -- 172.21.15.89:0/944057767 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f400e240 con 0x7f58f4005b90 2023-10-30T12:40:02.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.833+0000 7f590cb73640 1 -- 172.21.15.89:0/944057767 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58f400e400 con 0x7f58f40069e0 2023-10-30T12:40:02.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.833+0000 7f590e376640 1 --2- 172.21.15.89:0/944057767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58f40073a0 0x7f58f400dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:02.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.833+0000 7f590db75640 1 --2- 172.21.15.89:0/944057767 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58f4005b90 0x7f58f4005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:02.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.833+0000 7f590d374640 1 --2- 172.21.15.89:0/944057767 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58f40069e0 0x7f58f4006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:02.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.837+0000 7f590e376640 1 -- 172.21.15.89:0/944057767 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58f40069e0 msgr2=0x7f58f4006e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:02.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.837+0000 7f590e376640 1 --2- 172.21.15.89:0/944057767 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58f40069e0 0x7f58f4006e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:02.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.837+0000 7f590e376640 1 -- 172.21.15.89:0/944057767 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58f4005b90 msgr2=0x7f58f4005fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:02.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.837+0000 7f590e376640 1 --2- 172.21.15.89:0/944057767 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58f4005b90 0x7f58f4005fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:02.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.837+0000 7f590e376640 1 -- 172.21.15.89:0/944057767 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f4005740 con 0x7f58f40073a0 2023-10-30T12:40:02.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.837+0000 7f590e376640 1 -- 172.21.15.89:0/944057767 reap_dead start 2023-10-30T12:40:02.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.837+0000 7f590e376640 1 --2- 172.21.15.89:0/944057767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58f40073a0 0x7f58f400dba0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f5900001e70 tx=0x7f590001c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f9b1d5162d98ce8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:02.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.837+0000 7f590cb73640 1 -- 172.21.15.89:0/944057767 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5900003b80 con 0x7f58f40073a0 2023-10-30T12:40:02.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.837+0000 7f590cb73640 1 -- 172.21.15.89:0/944057767 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5900003d00 con 0x7f58f40073a0 2023-10-30T12:40:02.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.837+0000 7f590cb73640 1 -- 172.21.15.89:0/944057767 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5900005260 con 0x7f58f40073a0 2023-10-30T12:40:02.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.837+0000 7f590fdd9640 1 -- 172.21.15.89:0/944057767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58f40073a0 msgr2=0x7f58f400dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:02.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.837+0000 7f590fdd9640 1 --2- 172.21.15.89:0/944057767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58f40073a0 0x7f58f400dba0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f5900001e70 tx=0x7f590001c760 comp rx=0 tx=0).stop 2023-10-30T12:40:02.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.837+0000 7f590fdd9640 1 -- 172.21.15.89:0/944057767 shutdown_connections 2023-10-30T12:40:02.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.837+0000 7f590fdd9640 1 --2- 172.21.15.89:0/944057767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58f40073a0 0x7f58f400dba0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:02.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.837+0000 7f590fdd9640 1 -- 172.21.15.89:0/944057767 >> 172.21.15.89:0/944057767 conn(0x7f59080c7910 msgr2=0x7f59080c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:02.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.841+0000 7f590fdd9640 1 -- 172.21.15.89:0/944057767 shutdown_connections 2023-10-30T12:40:02.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.841+0000 7f590fdd9640 1 -- 172.21.15.89:0/944057767 wait complete. 2023-10-30T12:40:02.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.841+0000 7f590fdd9640 1 Processor -- start 2023-10-30T12:40:02.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.841+0000 7f590fdd9640 1 -- start start 2023-10-30T12:40:02.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.841+0000 7f590fdd9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58f40073a0 0x7f59081ef2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:02.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.841+0000 7f590fdd9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59080ca7e0 0x7f590813a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:02.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.841+0000 7f590fdd9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f590813a8a0 0x7f59081e0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:02.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.841+0000 7f590fdd9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59081dffe0 con 0x7f58f40073a0 2023-10-30T12:40:02.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.841+0000 7f590fdd9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59081e0880 con 0x7f59080ca7e0 2023-10-30T12:40:02.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.841+0000 7f590fdd9640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59080c8d90 con 0x7f590813a8a0 2023-10-30T12:40:02.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.841+0000 7f590db75640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58f40073a0 0x7f59081ef2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:02.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.841+0000 7f590db75640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58f40073a0 0x7f59081ef2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33918/0 (socket says 172.21.15.89:33918) 2023-10-30T12:40:02.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.841+0000 7f590db75640 1 -- 172.21.15.89:0/4065401430 learned_addr learned my addr 172.21.15.89:0/4065401430 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:40:02.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.841+0000 7f590e376640 1 --2- 172.21.15.89:0/4065401430 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f590813a8a0 0x7f59081e0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:02.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.841+0000 7f590d374640 1 --2- 172.21.15.89:0/4065401430 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59080ca7e0 0x7f590813a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:02.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.841+0000 7f590db75640 1 -- 172.21.15.89:0/4065401430 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f590813a8a0 msgr2=0x7f59081e0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:02.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.841+0000 7f590db75640 1 --2- 172.21.15.89:0/4065401430 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f590813a8a0 0x7f59081e0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:02.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.841+0000 7f590db75640 1 -- 172.21.15.89:0/4065401430 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59080ca7e0 msgr2=0x7f590813a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:02.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.841+0000 7f590db75640 1 --2- 172.21.15.89:0/4065401430 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59080ca7e0 0x7f590813a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:02.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.841+0000 7f590db75640 1 -- 172.21.15.89:0/4065401430 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59080c9070 con 0x7f58f40073a0 2023-10-30T12:40:02.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.841+0000 7f590db75640 1 --2- 172.21.15.89:0/4065401430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58f40073a0 0x7f59081ef2c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f5904011990 tx=0x7f5904011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:02.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.845+0000 7f58fe7fc640 1 -- 172.21.15.89:0/4065401430 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590400fb30 con 0x7f58f40073a0 2023-10-30T12:40:02.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.845+0000 7f58fe7fc640 1 -- 172.21.15.89:0/4065401430 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f590400fc90 con 0x7f58f40073a0 2023-10-30T12:40:02.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.845+0000 7f58fe7fc640 1 -- 172.21.15.89:0/4065401430 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5904014440 con 0x7f58f40073a0 2023-10-30T12:40:02.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.845+0000 7f590fdd9640 1 -- 172.21.15.89:0/4065401430 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59081e0230 con 0x7f58f40073a0 2023-10-30T12:40:02.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.845+0000 7f590fdd9640 1 -- 172.21.15.89:0/4065401430 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f590806ada0 con 0x7f58f40073a0 2023-10-30T12:40:02.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.845+0000 7f590fdd9640 1 -- 172.21.15.89:0/4065401430 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f590813ac80 con 0x7f58f40073a0 2023-10-30T12:40:02.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.845+0000 7f58fe7fc640 1 -- 172.21.15.89:0/4065401430 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f59040145a0 con 0x7f58f40073a0 2023-10-30T12:40:02.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.845+0000 7f58fe7fc640 1 --2- 172.21.15.89:0/4065401430 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f58ec0664a0 0x7f58ec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:02.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.845+0000 7f58fe7fc640 1 -- 172.21.15.89:0/4065401430 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6176+0+0 (secure 0 0 0) 0x7f5904084790 con 0x7f58f40073a0 2023-10-30T12:40:02.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.845+0000 7f590d374640 1 --2- 172.21.15.89:0/4065401430 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f58ec0664a0 0x7f58ec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:02.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.849+0000 7f590d374640 1 --2- 172.21.15.89:0/4065401430 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f58ec0664a0 0x7f58ec068960 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f58f8001060 tx=0x7f58f8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:02.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:02.849+0000 7f58fe7fc640 1 -- 172.21.15.89:0/4065401430 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5904051b00 con 0x7f58f40073a0 2023-10-30T12:40:03.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:03.065+0000 7f590fdd9640 1 -- 172.21.15.89:0/4065401430 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f590804e170 con 0x7f58f40073a0 2023-10-30T12:40:03.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:03.065+0000 7f58fe7fc640 1 -- 172.21.15.89:0/4065401430 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6529 (secure 0 0 0) 0x7f5904055120 con 0x7f58f40073a0 2023-10-30T12:40:03.066 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:40:03.066 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5129,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:40:00.157769+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:03.066 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 56 2023-10-30T12:40:03.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:03.065+0000 7f590fdd9640 1 -- 172.21.15.89:0/4065401430 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f58ec0664a0 msgr2=0x7f58ec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:03.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:03.065+0000 7f590fdd9640 1 --2- 172.21.15.89:0/4065401430 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f58ec0664a0 0x7f58ec068960 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f58f8001060 tx=0x7f58f8009040 comp rx=0 tx=0).stop 2023-10-30T12:40:03.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:03.065+0000 7f590fdd9640 1 -- 172.21.15.89:0/4065401430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58f40073a0 msgr2=0x7f59081ef2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:03.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:03.065+0000 7f590fdd9640 1 --2- 172.21.15.89:0/4065401430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58f40073a0 0x7f59081ef2c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f5904011990 tx=0x7f5904011e50 comp rx=0 tx=0).stop 2023-10-30T12:40:03.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:03.069+0000 7f590fdd9640 1 -- 172.21.15.89:0/4065401430 shutdown_connections 2023-10-30T12:40:03.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:03.069+0000 7f590fdd9640 1 --2- 172.21.15.89:0/4065401430 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f58ec0664a0 0x7f58ec068960 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:03.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:03.069+0000 7f590fdd9640 1 --2- 172.21.15.89:0/4065401430 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59080ca7e0 0x7f590813a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:03.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:03.069+0000 7f590fdd9640 1 --2- 172.21.15.89:0/4065401430 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f590813a8a0 0x7f59081e0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:03.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:03.069+0000 7f590fdd9640 1 --2- 172.21.15.89:0/4065401430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58f40073a0 0x7f59081ef2c0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:03.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:03.069+0000 7f590fdd9640 1 -- 172.21.15.89:0/4065401430 >> 172.21.15.89:0/4065401430 conn(0x7f59080c7910 msgr2=0x7f590813b160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:03.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:03.069+0000 7f590fdd9640 1 -- 172.21.15.89:0/4065401430 shutdown_connections 2023-10-30T12:40:03.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:03.069+0000 7f590fdd9640 1 -- 172.21.15.89:0/4065401430 wait complete. 2023-10-30T12:40:03.086 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T12:40:05.087 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:05.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.249+0000 7fdcd60f7640 1 Processor -- start 2023-10-30T12:40:05.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.249+0000 7fdcd60f7640 1 -- start start 2023-10-30T12:40:05.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.249+0000 7fdcd60f7640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fdcd00c6f60 0x7fdcd01566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:05.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.249+0000 7fdcd60f7640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd00c0a50 con 0x7fdcd00c6b80 2023-10-30T12:40:05.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.249+0000 7fdcd60f7640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd006f6e0 con 0x7fdcd00c05d0 2023-10-30T12:40:05.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.249+0000 7fdcd60f7640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcd006f9c0 con 0x7fdcd00c6f60 2023-10-30T12:40:05.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.249+0000 7fdccf7fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fdcd00c6b80 0x7fdcd00bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:60970/0 (socket says 172.21.15.89:60970) 2023-10-30T12:40:05.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.249+0000 7fdccf7fe640 1 -- 172.21.15.89:0/3583071341 learned_addr learned my addr 172.21.15.89:0/3583071341 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:40:05.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.249+0000 7fdccffff640 1 --2- 172.21.15.89:0/3583071341 >> v2:172.21.15.192:3300/0 conn(0x7fdcd00c6f60 0x7fdcd01566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:05.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.253+0000 7fdcce7fc640 1 -- 172.21.15.89:0/3583071341 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2576129688 0 0) 0x7fdcd00c0a50 con 0x7fdcd00c6b80 2023-10-30T12:40:05.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.253+0000 7fdcce7fc640 1 -- 172.21.15.89:0/3583071341 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcb8003530 con 0x7fdcd00c6b80 2023-10-30T12:40:05.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.253+0000 7fdcce7fc640 1 -- 172.21.15.89:0/3583071341 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 681068168 0 0) 0x7fdcb8003530 con 0x7fdcd00c6b80 2023-10-30T12:40:05.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.253+0000 7fdcce7fc640 1 -- 172.21.15.89:0/3583071341 >> v2:172.21.15.192:3300/0 conn(0x7fdcd00c6f60 msgr2=0x7fdcd01566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:05.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.253+0000 7fdcce7fc640 1 --2- 172.21.15.89:0/3583071341 >> v2:172.21.15.192:3300/0 conn(0x7fdcd00c6f60 0x7fdcd01566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:05.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.253+0000 7fdcce7fc640 1 -- 172.21.15.89:0/3583071341 >> v1:172.21.15.192:6789/0 conn(0x7fdcd00c05d0 legacy=0x7fdcd013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:05.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.253+0000 7fdcce7fc640 1 -- 172.21.15.89:0/3583071341 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd006fca0 con 0x7fdcd00c6b80 2023-10-30T12:40:05.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.253+0000 7fdcce7fc640 1 -- 172.21.15.89:0/3583071341 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7fdcc0001680 con 0x7fdcd00c6b80 2023-10-30T12:40:05.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.253+0000 7fdcce7fc640 1 -- 172.21.15.89:0/3583071341 >> v1:172.21.15.89:6789/0 conn(0x7fdcd00c6b80 legacy=0x7fdcd00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:05.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.253+0000 7fdcce7fc640 1 --2- 172.21.15.89:0/3583071341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcb8003980 0x7fdcb8003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:05.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.253+0000 7fdcce7fc640 1 --2- 172.21.15.89:0/3583071341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcb8004380 0x7fdcb8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:05.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.253+0000 7fdcce7fc640 1 --2- 172.21.15.89:0/3583071341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcb8005050 0x7fdcb800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:05.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.253+0000 7fdcce7fc640 1 -- 172.21.15.89:0/3583071341 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd006f9c0 con 0x7fdcb8004380 2023-10-30T12:40:05.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.253+0000 7fdcce7fc640 1 -- 172.21.15.89:0/3583071341 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb800bfb0 con 0x7fdcb8003980 2023-10-30T12:40:05.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.253+0000 7fdcce7fc640 1 -- 172.21.15.89:0/3583071341 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb800c170 con 0x7fdcb8005050 2023-10-30T12:40:05.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.253+0000 7fdcceffd640 1 --2- 172.21.15.89:0/3583071341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcb8004380 0x7fdcb8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:05.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.253+0000 7fdccf7fe640 1 --2- 172.21.15.89:0/3583071341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcb8003980 0x7fdcb8003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:05.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.253+0000 7fdccffff640 1 --2- 172.21.15.89:0/3583071341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcb8005050 0x7fdcb800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:05.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.253+0000 7fdcceffd640 1 -- 172.21.15.89:0/3583071341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcb8005050 msgr2=0x7fdcb800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:05.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.253+0000 7fdcceffd640 1 --2- 172.21.15.89:0/3583071341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcb8005050 0x7fdcb800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:05.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.253+0000 7fdcceffd640 1 -- 172.21.15.89:0/3583071341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcb8003980 msgr2=0x7fdcb8003d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:05.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.253+0000 7fdcceffd640 1 --2- 172.21.15.89:0/3583071341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcb8003980 0x7fdcb8003d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:05.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.253+0000 7fdcceffd640 1 -- 172.21.15.89:0/3583071341 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb8003530 con 0x7fdcb8004380 2023-10-30T12:40:05.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.253+0000 7fdccffff640 1 -- 172.21.15.89:0/3583071341 reap_dead start 2023-10-30T12:40:05.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.253+0000 7fdccf7fe640 1 --2- 172.21.15.89:0/3583071341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcb8003980 0x7fdcb8003d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:40:05.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.257+0000 7fdcceffd640 1 --2- 172.21.15.89:0/3583071341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcb8004380 0x7fdcb8004850 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fdcd006c9a0 tx=0x7fdcc401ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=664839bc066d9581 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:05.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.257+0000 7fdcce7fc640 1 -- 172.21.15.89:0/3583071341 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc40a8810 con 0x7fdcb8004380 2023-10-30T12:40:05.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.257+0000 7fdcce7fc640 1 -- 172.21.15.89:0/3583071341 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc40a8970 con 0x7fdcb8004380 2023-10-30T12:40:05.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.257+0000 7fdcce7fc640 1 -- 172.21.15.89:0/3583071341 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc40a8d60 con 0x7fdcb8004380 2023-10-30T12:40:05.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.257+0000 7fdcd60f7640 1 -- 172.21.15.89:0/3583071341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcb8004380 msgr2=0x7fdcb8004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:05.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.257+0000 7fdcd60f7640 1 --2- 172.21.15.89:0/3583071341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcb8004380 0x7fdcb8004850 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fdcd006c9a0 tx=0x7fdcc401ebf0 comp rx=0 tx=0).stop 2023-10-30T12:40:05.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.257+0000 7fdcd60f7640 1 -- 172.21.15.89:0/3583071341 shutdown_connections 2023-10-30T12:40:05.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.257+0000 7fdcd60f7640 1 --2- 172.21.15.89:0/3583071341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcb8004380 0x7fdcb8004850 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:05.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.257+0000 7fdcd60f7640 1 -- 172.21.15.89:0/3583071341 >> 172.21.15.89:0/3583071341 conn(0x7fdcd00c7910 msgr2=0x7fdcd00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:05.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.257+0000 7fdcd60f7640 1 -- 172.21.15.89:0/3583071341 shutdown_connections 2023-10-30T12:40:05.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.257+0000 7fdcd60f7640 1 -- 172.21.15.89:0/3583071341 wait complete. 2023-10-30T12:40:05.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.257+0000 7fdcd60f7640 1 Processor -- start 2023-10-30T12:40:05.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.257+0000 7fdcd60f7640 1 -- start start 2023-10-30T12:40:05.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.257+0000 7fdcd60f7640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcb8004380 0x7fdcd00c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:05.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.257+0000 7fdcd60f7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcd00c9830 0x7fdcd01eec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:05.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.257+0000 7fdcd60f7640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcd01ef1a0 0x7fdcd0153060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:05.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.261+0000 7fdcd60f7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd00add50 con 0x7fdcd00c9830 2023-10-30T12:40:05.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.261+0000 7fdcd60f7640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd01f2d60 con 0x7fdcd01ef1a0 2023-10-30T12:40:05.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.261+0000 7fdcd60f7640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd01ef620 con 0x7fdcb8004380 2023-10-30T12:40:05.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.261+0000 7fdcceffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcd00c9830 0x7fdcd01eec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:05.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.261+0000 7fdcceffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcd00c9830 0x7fdcd01eec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33938/0 (socket says 172.21.15.89:33938) 2023-10-30T12:40:05.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.261+0000 7fdcceffd640 1 -- 172.21.15.89:0/2188827427 learned_addr learned my addr 172.21.15.89:0/2188827427 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:40:05.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.261+0000 7fdccf7fe640 1 --2- 172.21.15.89:0/2188827427 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcb8004380 0x7fdcd00c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:05.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.261+0000 7fdccffff640 1 --2- 172.21.15.89:0/2188827427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcd01ef1a0 0x7fdcd0153060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:05.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.261+0000 7fdcceffd640 1 -- 172.21.15.89:0/2188827427 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcb8004380 msgr2=0x7fdcd00c8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:05.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.261+0000 7fdcceffd640 1 --2- 172.21.15.89:0/2188827427 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcb8004380 0x7fdcd00c8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:05.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.261+0000 7fdcceffd640 1 -- 172.21.15.89:0/2188827427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcd01ef1a0 msgr2=0x7fdcd0153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:05.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.261+0000 7fdcceffd640 1 --2- 172.21.15.89:0/2188827427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcd01ef1a0 0x7fdcd0153060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:05.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.261+0000 7fdcceffd640 1 -- 172.21.15.89:0/2188827427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd01f1e90 con 0x7fdcd00c9830 2023-10-30T12:40:05.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.261+0000 7fdccffff640 1 --2- 172.21.15.89:0/2188827427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcd01ef1a0 0x7fdcd0153060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:40:05.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.261+0000 7fdcceffd640 1 --2- 172.21.15.89:0/2188827427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcd00c9830 0x7fdcd01eec60 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fdcc40b3040 tx=0x7fdcb8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:05.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.261+0000 7fdcb7fff640 1 -- 172.21.15.89:0/2188827427 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc40069e0 con 0x7fdcd00c9830 2023-10-30T12:40:05.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.261+0000 7fdcb7fff640 1 -- 172.21.15.89:0/2188827427 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc4006b40 con 0x7fdcd00c9830 2023-10-30T12:40:05.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.261+0000 7fdcd60f7640 1 -- 172.21.15.89:0/2188827427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd00c1f10 con 0x7fdcd00c9830 2023-10-30T12:40:05.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.261+0000 7fdcb7fff640 1 -- 172.21.15.89:0/2188827427 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc4007040 con 0x7fdcd00c9830 2023-10-30T12:40:05.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.261+0000 7fdcd60f7640 1 -- 172.21.15.89:0/2188827427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd01f2430 con 0x7fdcd00c9830 2023-10-30T12:40:05.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.261+0000 7fdcd60f7640 1 -- 172.21.15.89:0/2188827427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd004e170 con 0x7fdcd00c9830 2023-10-30T12:40:05.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.261+0000 7fdcb7fff640 1 -- 172.21.15.89:0/2188827427 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdcc40b7040 con 0x7fdcd00c9830 2023-10-30T12:40:05.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.261+0000 7fdcb7fff640 1 --2- 172.21.15.89:0/2188827427 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fdcbc0664a0 0x7fdcbc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:05.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.265+0000 7fdcb7fff640 1 -- 172.21.15.89:0/2188827427 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6176+0+0 (secure 0 0 0) 0x7fdcc4121430 con 0x7fdcd00c9830 2023-10-30T12:40:05.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.265+0000 7fdccf7fe640 1 --2- 172.21.15.89:0/2188827427 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fdcbc0664a0 0x7fdcbc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:05.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.265+0000 7fdccf7fe640 1 --2- 172.21.15.89:0/2188827427 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fdcbc0664a0 0x7fdcbc068960 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0001cc0 tx=0x7fdcc0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:05.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.269+0000 7fdcb7fff640 1 -- 172.21.15.89:0/2188827427 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdcc40ee780 con 0x7fdcd00c9830 2023-10-30T12:40:05.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.449+0000 7fdcd60f7640 1 -- 172.21.15.89:0/2188827427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdcd0071420 con 0x7fdcd00c9830 2023-10-30T12:40:05.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.453+0000 7fdcb7fff640 1 -- 172.21.15.89:0/2188827427 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6529 (secure 0 0 0) 0x7fdcc40b5090 con 0x7fdcd00c9830 2023-10-30T12:40:05.452 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:40:05.452 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5129,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:40:00.157769+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:05.453 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 56 2023-10-30T12:40:05.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.453+0000 7fdcd60f7640 1 -- 172.21.15.89:0/2188827427 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fdcbc0664a0 msgr2=0x7fdcbc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:05.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.453+0000 7fdcd60f7640 1 --2- 172.21.15.89:0/2188827427 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fdcbc0664a0 0x7fdcbc068960 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0001cc0 tx=0x7fdcc0009040 comp rx=0 tx=0).stop 2023-10-30T12:40:05.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.453+0000 7fdcd60f7640 1 -- 172.21.15.89:0/2188827427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcd00c9830 msgr2=0x7fdcd01eec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:05.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.453+0000 7fdcd60f7640 1 --2- 172.21.15.89:0/2188827427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcd00c9830 0x7fdcd01eec60 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fdcc40b3040 tx=0x7fdcb8005800 comp rx=0 tx=0).stop 2023-10-30T12:40:05.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.453+0000 7fdcd60f7640 1 -- 172.21.15.89:0/2188827427 shutdown_connections 2023-10-30T12:40:05.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.453+0000 7fdcd60f7640 1 --2- 172.21.15.89:0/2188827427 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fdcbc0664a0 0x7fdcbc068960 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:05.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.453+0000 7fdcd60f7640 1 --2- 172.21.15.89:0/2188827427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcd01ef1a0 0x7fdcd0153060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:05.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.453+0000 7fdcd60f7640 1 --2- 172.21.15.89:0/2188827427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcd00c9830 0x7fdcd01eec60 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:05.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.453+0000 7fdcd60f7640 1 --2- 172.21.15.89:0/2188827427 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcb8004380 0x7fdcd00c8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:05.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.453+0000 7fdcd60f7640 1 -- 172.21.15.89:0/2188827427 >> 172.21.15.89:0/2188827427 conn(0x7fdcd00c7910 msgr2=0x7fdcd013a610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:05.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.453+0000 7fdcd60f7640 1 -- 172.21.15.89:0/2188827427 shutdown_connections 2023-10-30T12:40:05.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:05.453+0000 7fdcd60f7640 1 -- 172.21.15.89:0/2188827427 wait complete. 2023-10-30T12:40:05.472 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T12:40:07.474 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:07.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.621+0000 7f20d0446640 1 Processor -- start 2023-10-30T12:40:07.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.621+0000 7f20d0446640 1 -- start start 2023-10-30T12:40:07.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.621+0000 7f20d0446640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f20c81538a0 0x7f20c815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:07.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.621+0000 7f20d0446640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20c80cf160 0x7f20c80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:07.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.625+0000 7f20d0446640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20c80cb800 con 0x7f20c80ca0e0 2023-10-30T12:40:07.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.625+0000 7f20d0446640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20c815dc30 con 0x7f20c81538a0 2023-10-30T12:40:07.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.625+0000 7f20d0446640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20c80cb330 con 0x7f20c80cf160 2023-10-30T12:40:07.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.625+0000 7f20ce1e2640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f20c81538a0 0x7f20c815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:07.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.625+0000 7f20cd9e1640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f20c80ca0e0 0x7f20c815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:60982/0 (socket says 172.21.15.89:60982) 2023-10-30T12:40:07.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.625+0000 7f20cd9e1640 1 -- 172.21.15.89:0/1083812993 learned_addr learned my addr 172.21.15.89:0/1083812993 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:40:07.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.625+0000 7f20ce9e3640 1 --2- 172.21.15.89:0/1083812993 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20c80cf160 0x7f20c80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:07.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.625+0000 7f20ce1e2640 1 -- 172.21.15.89:0/1083812993 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20c80cf160 msgr2=0x7f20c80cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:07.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.625+0000 7f20ce1e2640 1 --2- 172.21.15.89:0/1083812993 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20c80cf160 0x7f20c80cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:07.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.625+0000 7f20ce1e2640 1 -- 172.21.15.89:0/1083812993 >> v1:172.21.15.89:6789/0 conn(0x7f20c80ca0e0 legacy=0x7f20c815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:07.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.625+0000 7f20ce1e2640 1 -- 172.21.15.89:0/1083812993 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c80cad70 con 0x7f20c81538a0 2023-10-30T12:40:07.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.625+0000 7f20ce1e2640 1 --2- 172.21.15.89:0/1083812993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20c81538a0 0x7f20c815d6f0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f20c40097c0 tx=0x7f20c401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94cd317a83a46116 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:07.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.625+0000 7f20cd1e0640 1 -- 172.21.15.89:0/1083812993 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c40a8970 con 0x7f20c81538a0 2023-10-30T12:40:07.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.625+0000 7f20cd1e0640 1 -- 172.21.15.89:0/1083812993 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c40a8ad0 con 0x7f20c81538a0 2023-10-30T12:40:07.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.625+0000 7f20cd1e0640 1 -- 172.21.15.89:0/1083812993 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c40a8e60 con 0x7f20c81538a0 2023-10-30T12:40:07.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.629+0000 7f20d0446640 1 -- 172.21.15.89:0/1083812993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20c81538a0 msgr2=0x7f20c815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:07.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.629+0000 7f20d0446640 1 --2- 172.21.15.89:0/1083812993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20c81538a0 0x7f20c815d6f0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f20c40097c0 tx=0x7f20c401ec70 comp rx=0 tx=0).stop 2023-10-30T12:40:07.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.629+0000 7f20d0446640 1 -- 172.21.15.89:0/1083812993 shutdown_connections 2023-10-30T12:40:07.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.629+0000 7f20d0446640 1 --2- 172.21.15.89:0/1083812993 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20c80cf160 0x7f20c80cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:07.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.629+0000 7f20d0446640 1 --2- 172.21.15.89:0/1083812993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20c81538a0 0x7f20c815d6f0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:07.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.629+0000 7f20d0446640 1 -- 172.21.15.89:0/1083812993 >> 172.21.15.89:0/1083812993 conn(0x7f20c806bfe0 msgr2=0x7f20c806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:07.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.629+0000 7f20d0446640 1 -- 172.21.15.89:0/1083812993 shutdown_connections 2023-10-30T12:40:07.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.629+0000 7f20d0446640 1 -- 172.21.15.89:0/1083812993 wait complete. 2023-10-30T12:40:07.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.629+0000 7f20d0446640 1 Processor -- start 2023-10-30T12:40:07.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.629+0000 7f20d0446640 1 -- start start 2023-10-30T12:40:07.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.629+0000 7f20d0446640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20c80ca0e0 0x7f20c81e4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:07.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.629+0000 7f20d0446640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20c80cf160 0x7f20c815b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:07.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.629+0000 7f20d0446640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20c81538a0 0x7f20c815bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:07.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.629+0000 7f20d0446640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c814d4f0 con 0x7f20c81538a0 2023-10-30T12:40:07.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.629+0000 7f20d0446640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c81e4d40 con 0x7f20c80cf160 2023-10-30T12:40:07.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.629+0000 7f20d0446640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20c815b2a0 con 0x7f20c80ca0e0 2023-10-30T12:40:07.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.629+0000 7f20ce9e3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20c81538a0 0x7f20c815bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:07.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.629+0000 7f20ce9e3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20c81538a0 0x7f20c815bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33956/0 (socket says 172.21.15.89:33956) 2023-10-30T12:40:07.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.629+0000 7f20ce9e3640 1 -- 172.21.15.89:0/3636298357 learned_addr learned my addr 172.21.15.89:0/3636298357 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:40:07.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.629+0000 7f20ce1e2640 1 --2- 172.21.15.89:0/3636298357 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20c80ca0e0 0x7f20c81e4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:07.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.633+0000 7f20cd9e1640 1 --2- 172.21.15.89:0/3636298357 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20c80cf160 0x7f20c815b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:07.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.633+0000 7f20ce9e3640 1 -- 172.21.15.89:0/3636298357 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20c80ca0e0 msgr2=0x7f20c81e4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:07.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.633+0000 7f20ce9e3640 1 --2- 172.21.15.89:0/3636298357 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20c80ca0e0 0x7f20c81e4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:07.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.633+0000 7f20ce9e3640 1 -- 172.21.15.89:0/3636298357 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20c80cf160 msgr2=0x7f20c815b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:07.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.633+0000 7f20ce9e3640 1 --2- 172.21.15.89:0/3636298357 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20c80cf160 0x7f20c815b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:07.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.633+0000 7f20ce9e3640 1 -- 172.21.15.89:0/3636298357 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c4009420 con 0x7f20c81538a0 2023-10-30T12:40:07.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.633+0000 7f20ce9e3640 1 --2- 172.21.15.89:0/3636298357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20c81538a0 0x7f20c815bb70 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f20c0007d90 tx=0x7f20c000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:07.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.633+0000 7f20beffd640 1 -- 172.21.15.89:0/3636298357 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c0014420 con 0x7f20c81538a0 2023-10-30T12:40:07.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.633+0000 7f20beffd640 1 -- 172.21.15.89:0/3636298357 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c00145a0 con 0x7f20c81538a0 2023-10-30T12:40:07.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.633+0000 7f20beffd640 1 -- 172.21.15.89:0/3636298357 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c0014950 con 0x7f20c81538a0 2023-10-30T12:40:07.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.633+0000 7f20d0446640 1 -- 172.21.15.89:0/3636298357 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20c80cef10 con 0x7f20c81538a0 2023-10-30T12:40:07.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.633+0000 7f20d0446640 1 -- 172.21.15.89:0/3636298357 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20c80c50b0 con 0x7f20c81538a0 2023-10-30T12:40:07.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.633+0000 7f20d0446640 1 -- 172.21.15.89:0/3636298357 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20c80b2a50 con 0x7f20c81538a0 2023-10-30T12:40:07.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.633+0000 7f20beffd640 1 -- 172.21.15.89:0/3636298357 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f20c0014ab0 con 0x7f20c81538a0 2023-10-30T12:40:07.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.633+0000 7f20beffd640 1 --2- 172.21.15.89:0/3636298357 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f20ac0664a0 0x7f20ac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:07.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.633+0000 7f20beffd640 1 -- 172.21.15.89:0/3636298357 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6176+0+0 (secure 0 0 0) 0x7f20c0084970 con 0x7f20c81538a0 2023-10-30T12:40:07.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.633+0000 7f20ce1e2640 1 --2- 172.21.15.89:0/3636298357 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f20ac0664a0 0x7f20ac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:07.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.637+0000 7f20ce1e2640 1 --2- 172.21.15.89:0/3636298357 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f20ac0664a0 0x7f20ac068960 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f20c40b2000 tx=0x7f20c4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:07.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.637+0000 7f20beffd640 1 -- 172.21.15.89:0/3636298357 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f20c0051ba0 con 0x7f20c81538a0 2023-10-30T12:40:07.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.817+0000 7f20d0446640 1 -- 172.21.15.89:0/3636298357 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20c81e4eb0 con 0x7f20c81538a0 2023-10-30T12:40:07.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.817+0000 7f20beffd640 1 -- 172.21.15.89:0/3636298357 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6529 (secure 0 0 0) 0x7f20c00551c0 con 0x7f20c81538a0 2023-10-30T12:40:07.819 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:40:07.819 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5129,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:40:00.157769+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:07.819 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 56 2023-10-30T12:40:07.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.821+0000 7f20d0446640 1 -- 172.21.15.89:0/3636298357 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f20ac0664a0 msgr2=0x7f20ac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:07.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.821+0000 7f20d0446640 1 --2- 172.21.15.89:0/3636298357 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f20ac0664a0 0x7f20ac068960 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f20c40b2000 tx=0x7f20c4008040 comp rx=0 tx=0).stop 2023-10-30T12:40:07.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.821+0000 7f20d0446640 1 -- 172.21.15.89:0/3636298357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20c81538a0 msgr2=0x7f20c815bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:07.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.821+0000 7f20d0446640 1 --2- 172.21.15.89:0/3636298357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20c81538a0 0x7f20c815bb70 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f20c0007d90 tx=0x7f20c000e4b0 comp rx=0 tx=0).stop 2023-10-30T12:40:07.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.821+0000 7f20d0446640 1 -- 172.21.15.89:0/3636298357 shutdown_connections 2023-10-30T12:40:07.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.821+0000 7f20d0446640 1 --2- 172.21.15.89:0/3636298357 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f20ac0664a0 0x7f20ac068960 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:07.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.821+0000 7f20d0446640 1 --2- 172.21.15.89:0/3636298357 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20c80cf160 0x7f20c815b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:07.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.821+0000 7f20d0446640 1 --2- 172.21.15.89:0/3636298357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20c81538a0 0x7f20c815bb70 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:07.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.821+0000 7f20d0446640 1 --2- 172.21.15.89:0/3636298357 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20c80ca0e0 0x7f20c81e4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:07.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.821+0000 7f20d0446640 1 -- 172.21.15.89:0/3636298357 >> 172.21.15.89:0/3636298357 conn(0x7f20c806bfe0 msgr2=0x7f20c8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:07.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.821+0000 7f20d0446640 1 -- 172.21.15.89:0/3636298357 shutdown_connections 2023-10-30T12:40:07.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:07.821+0000 7f20d0446640 1 -- 172.21.15.89:0/3636298357 wait complete. 2023-10-30T12:40:07.839 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T12:40:09.840 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:09.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.989+0000 7f8fd66d6640 1 Processor -- start 2023-10-30T12:40:09.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.989+0000 7f8fd66d6640 1 -- start start 2023-10-30T12:40:09.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.989+0000 7f8fd66d6640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8fd01538a0 0x7f8fd015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:09.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.989+0000 7f8fd66d6640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8fd00ca0e0 0x7f8fd015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:09.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.989+0000 7f8fd66d6640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd00cb800 con 0x7f8fd00cf160 2023-10-30T12:40:09.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.989+0000 7f8fd66d6640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fd015dc30 con 0x7f8fd01538a0 2023-10-30T12:40:09.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.989+0000 7f8fd66d6640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd00cb330 con 0x7f8fd00ca0e0 2023-10-30T12:40:09.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.989+0000 7f8fcf7fe640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8fd00ca0e0 0x7f8fd015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:09.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.989+0000 7f8fcffff640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8fd01538a0 0x7f8fd015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:09.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.989+0000 7f8fd4c73640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f8fd00cf160 0x7f8fd00cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:45438/0 (socket says 172.21.15.89:45438) 2023-10-30T12:40:09.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.989+0000 7f8fd4c73640 1 -- 172.21.15.89:0/2363376282 learned_addr learned my addr 172.21.15.89:0/2363376282 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:40:09.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.989+0000 7f8fcffff640 1 -- 172.21.15.89:0/2363376282 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8fd00ca0e0 msgr2=0x7f8fd015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:09.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.989+0000 7f8fcffff640 1 --2- 172.21.15.89:0/2363376282 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8fd00ca0e0 0x7f8fd015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:09.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.989+0000 7f8fcffff640 1 -- 172.21.15.89:0/2363376282 >> v1:172.21.15.192:6789/0 conn(0x7f8fd00cf160 legacy=0x7f8fd00cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:09.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.989+0000 7f8fcffff640 1 -- 172.21.15.89:0/2363376282 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd00cad70 con 0x7f8fd01538a0 2023-10-30T12:40:09.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.989+0000 7f8fcf7fe640 1 --2- 172.21.15.89:0/2363376282 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8fd00ca0e0 0x7f8fd015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:40:09.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.993+0000 7f8fcffff640 1 --2- 172.21.15.89:0/2363376282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fd01538a0 0x7f8fd015d6f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8fc00097c0 tx=0x7f8fc001ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35122ab7a341a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:09.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.993+0000 7f8fceffd640 1 -- 172.21.15.89:0/2363376282 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc00a8970 con 0x7f8fd01538a0 2023-10-30T12:40:09.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.993+0000 7f8fceffd640 1 -- 172.21.15.89:0/2363376282 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc00a8ad0 con 0x7f8fd01538a0 2023-10-30T12:40:09.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.993+0000 7f8fceffd640 1 -- 172.21.15.89:0/2363376282 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc00a8e60 con 0x7f8fd01538a0 2023-10-30T12:40:09.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.993+0000 7f8fd66d6640 1 -- 172.21.15.89:0/2363376282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fd01538a0 msgr2=0x7f8fd015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:09.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.993+0000 7f8fd66d6640 1 --2- 172.21.15.89:0/2363376282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fd01538a0 0x7f8fd015d6f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8fc00097c0 tx=0x7f8fc001ec70 comp rx=0 tx=0).stop 2023-10-30T12:40:09.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.993+0000 7f8fd66d6640 1 -- 172.21.15.89:0/2363376282 shutdown_connections 2023-10-30T12:40:09.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.993+0000 7f8fd66d6640 1 --2- 172.21.15.89:0/2363376282 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8fd00ca0e0 0x7f8fd015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:09.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.993+0000 7f8fd66d6640 1 --2- 172.21.15.89:0/2363376282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fd01538a0 0x7f8fd015d6f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:09.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.993+0000 7f8fd66d6640 1 -- 172.21.15.89:0/2363376282 >> 172.21.15.89:0/2363376282 conn(0x7f8fd006bfe0 msgr2=0x7f8fd006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:09.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.993+0000 7f8fd66d6640 1 -- 172.21.15.89:0/2363376282 shutdown_connections 2023-10-30T12:40:09.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.993+0000 7f8fd66d6640 1 -- 172.21.15.89:0/2363376282 wait complete. 2023-10-30T12:40:09.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.997+0000 7f8fd66d6640 1 Processor -- start 2023-10-30T12:40:09.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.997+0000 7f8fd66d6640 1 -- start start 2023-10-30T12:40:10.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.997+0000 7f8fd66d6640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8fd00ca0e0 0x7f8fd00ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:10.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.997+0000 7f8fd66d6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fd00cf160 0x7f8fd0158250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:10.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.997+0000 7f8fd66d6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fd01538a0 0x7f8fd014cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:10.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.997+0000 7f8fd66d6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd01ef540 con 0x7f8fd00cf160 2023-10-30T12:40:10.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.997+0000 7f8fd66d6640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd01f0510 con 0x7f8fd01538a0 2023-10-30T12:40:10.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.997+0000 7f8fd66d6640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd00666b0 con 0x7f8fd00ca0e0 2023-10-30T12:40:10.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.997+0000 7f8fcf7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fd00cf160 0x7f8fd0158250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:10.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.997+0000 7f8fcf7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fd00cf160 0x7f8fd0158250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50052/0 (socket says 172.21.15.89:50052) 2023-10-30T12:40:10.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.997+0000 7f8fcf7fe640 1 -- 172.21.15.89:0/3687339305 learned_addr learned my addr 172.21.15.89:0/3687339305 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:40:10.002 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.997+0000 7f8fcffff640 1 --2- 172.21.15.89:0/3687339305 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8fd00ca0e0 0x7f8fd00ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:10.002 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.997+0000 7f8fd4c73640 1 --2- 172.21.15.89:0/3687339305 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fd01538a0 0x7f8fd014cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:10.002 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.997+0000 7f8fcf7fe640 1 -- 172.21.15.89:0/3687339305 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8fd00ca0e0 msgr2=0x7f8fd00ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:10.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.997+0000 7f8fcf7fe640 1 --2- 172.21.15.89:0/3687339305 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8fd00ca0e0 0x7f8fd00ce390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:10.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.997+0000 7f8fcf7fe640 1 -- 172.21.15.89:0/3687339305 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fd01538a0 msgr2=0x7f8fd014cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:10.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.997+0000 7f8fcf7fe640 1 --2- 172.21.15.89:0/3687339305 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fd01538a0 0x7f8fd014cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:10.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.997+0000 7f8fcf7fe640 1 -- 172.21.15.89:0/3687339305 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc4009420 con 0x7f8fd00cf160 2023-10-30T12:40:10.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.997+0000 7f8fcf7fe640 1 --2- 172.21.15.89:0/3687339305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fd00cf160 0x7f8fd0158250 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400ebe0 tx=0x7f8fc400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:10.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.997+0000 7f8fccff9640 1 -- 172.21.15.89:0/3687339305 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc400eee0 con 0x7f8fd00cf160 2023-10-30T12:40:10.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.997+0000 7f8fccff9640 1 -- 172.21.15.89:0/3687339305 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc400cc30 con 0x7f8fd00cf160 2023-10-30T12:40:10.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.997+0000 7f8fd66d6640 1 -- 172.21.15.89:0/3687339305 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fc0009420 con 0x7f8fd00cf160 2023-10-30T12:40:10.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.997+0000 7f8fccff9640 1 -- 172.21.15.89:0/3687339305 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4013470 con 0x7f8fd00cf160 2023-10-30T12:40:10.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:09.997+0000 7f8fd66d6640 1 -- 172.21.15.89:0/3687339305 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd01efa30 con 0x7f8fd00cf160 2023-10-30T12:40:10.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:10.001+0000 7f8fd66d6640 1 -- 172.21.15.89:0/3687339305 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd00b2a50 con 0x7f8fd00cf160 2023-10-30T12:40:10.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:10.001+0000 7f8fccff9640 1 -- 172.21.15.89:0/3687339305 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8fc40135d0 con 0x7f8fd00cf160 2023-10-30T12:40:10.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:10.001+0000 7f8fccff9640 1 --2- 172.21.15.89:0/3687339305 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8fbc0664a0 0x7f8fbc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:10.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:10.001+0000 7f8fccff9640 1 -- 172.21.15.89:0/3687339305 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6176+0+0 (secure 0 0 0) 0x7f8fc4084320 con 0x7f8fd00cf160 2023-10-30T12:40:10.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:10.001+0000 7f8fcffff640 1 --2- 172.21.15.89:0/3687339305 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8fbc0664a0 0x7f8fbc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:10.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:10.001+0000 7f8fcffff640 1 --2- 172.21.15.89:0/3687339305 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8fbc0664a0 0x7f8fbc068960 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8fc00b2000 tx=0x7f8fc0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:10.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:10.005+0000 7f8fccff9640 1 -- 172.21.15.89:0/3687339305 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8fc4051600 con 0x7f8fd00cf160 2023-10-30T12:40:10.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:10.189+0000 7f8fd66d6640 1 -- 172.21.15.89:0/3687339305 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fd004e170 con 0x7f8fd00cf160 2023-10-30T12:40:10.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:10.189+0000 7f8fccff9640 1 -- 172.21.15.89:0/3687339305 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6529 (secure 0 0 0) 0x7f8fc4054c20 con 0x7f8fd00cf160 2023-10-30T12:40:10.192 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:40:10.192 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5129,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:40:00.157769+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":4781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4781":{"gid":4781,"name":"b","rank":2,"incarnation":24,"state":"up:active","state_seq":27,"addr":"172.21.15.192:6835/654256299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":654256299},{"type":"v1","addr":"172.21.15.192:6835","nonce":654256299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:10.192 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 56 2023-10-30T12:40:10.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:10.193+0000 7f8fd66d6640 1 -- 172.21.15.89:0/3687339305 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8fbc0664a0 msgr2=0x7f8fbc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:10.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:10.193+0000 7f8fd66d6640 1 --2- 172.21.15.89:0/3687339305 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8fbc0664a0 0x7f8fbc068960 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8fc00b2000 tx=0x7f8fc0008040 comp rx=0 tx=0).stop 2023-10-30T12:40:10.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:10.193+0000 7f8fd66d6640 1 -- 172.21.15.89:0/3687339305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fd00cf160 msgr2=0x7f8fd0158250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:10.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:10.193+0000 7f8fd66d6640 1 --2- 172.21.15.89:0/3687339305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fd00cf160 0x7f8fd0158250 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400ebe0 tx=0x7f8fc400c520 comp rx=0 tx=0).stop 2023-10-30T12:40:10.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:10.193+0000 7f8fd66d6640 1 -- 172.21.15.89:0/3687339305 shutdown_connections 2023-10-30T12:40:10.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:10.193+0000 7f8fd66d6640 1 --2- 172.21.15.89:0/3687339305 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8fbc0664a0 0x7f8fbc068960 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:10.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:10.193+0000 7f8fd66d6640 1 --2- 172.21.15.89:0/3687339305 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fd01538a0 0x7f8fd014cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:10.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:10.193+0000 7f8fd66d6640 1 --2- 172.21.15.89:0/3687339305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fd00cf160 0x7f8fd0158250 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:10.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:10.193+0000 7f8fd66d6640 1 --2- 172.21.15.89:0/3687339305 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8fd00ca0e0 0x7f8fd00ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:10.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:10.193+0000 7f8fd66d6640 1 -- 172.21.15.89:0/3687339305 >> 172.21.15.89:0/3687339305 conn(0x7f8fd006bfe0 msgr2=0x7f8fd014db50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:10.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:10.193+0000 7f8fd66d6640 1 -- 172.21.15.89:0/3687339305 shutdown_connections 2023-10-30T12:40:10.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:10.193+0000 7f8fd66d6640 1 -- 172.21.15.89:0/3687339305 wait complete. 2023-10-30T12:40:10.212 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T12:40:12.214 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:12.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.365+0000 7fbd8973e640 1 Processor -- start 2023-10-30T12:40:12.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.365+0000 7fbd8973e640 1 -- start start 2023-10-30T12:40:12.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.365+0000 7fbd8973e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbd8415d6f0 0x7fbd8415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:12.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.365+0000 7fbd8973e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd840cf160 0x7fbd840cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:12.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.365+0000 7fbd8973e640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd840cb800 con 0x7fbd841538a0 2023-10-30T12:40:12.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.365+0000 7fbd8973e640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd840cb330 con 0x7fbd8415d6f0 2023-10-30T12:40:12.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.365+0000 7fbd8973e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd840cad70 con 0x7fbd840cf160 2023-10-30T12:40:12.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.365+0000 7fbd7bfff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbd8415d6f0 0x7fbd8415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:12.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.365+0000 7fbd82ffd640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fbd841538a0 0x7fbd840ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44844/0 (socket says 172.21.15.89:44844) 2023-10-30T12:40:12.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.365+0000 7fbd82ffd640 1 -- 172.21.15.89:0/2372157804 learned_addr learned my addr 172.21.15.89:0/2372157804 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:40:12.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.365+0000 7fbd837fe640 1 --2- 172.21.15.89:0/2372157804 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd840cf160 0x7fbd840cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:12.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.365+0000 7fbd7bfff640 1 -- 172.21.15.89:0/2372157804 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd840cf160 msgr2=0x7fbd840cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:12.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.365+0000 7fbd7bfff640 1 --2- 172.21.15.89:0/2372157804 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd840cf160 0x7fbd840cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:12.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.365+0000 7fbd7bfff640 1 -- 172.21.15.89:0/2372157804 >> v1:172.21.15.89:6789/0 conn(0x7fbd841538a0 legacy=0x7fbd840ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:12.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.365+0000 7fbd7bfff640 1 -- 172.21.15.89:0/2372157804 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd840cb050 con 0x7fbd8415d6f0 2023-10-30T12:40:12.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.369+0000 7fbd7bfff640 1 --2- 172.21.15.89:0/2372157804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd8415d6f0 0x7fbd8415a090 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c0097c0 tx=0x7fbd6c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2e675338499efac server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:12.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.369+0000 7fbd827fc640 1 -- 172.21.15.89:0/2372157804 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c0a8970 con 0x7fbd8415d6f0 2023-10-30T12:40:12.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.369+0000 7fbd827fc640 1 -- 172.21.15.89:0/2372157804 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd6c0a8ad0 con 0x7fbd8415d6f0 2023-10-30T12:40:12.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.369+0000 7fbd827fc640 1 -- 172.21.15.89:0/2372157804 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c0a8e60 con 0x7fbd8415d6f0 2023-10-30T12:40:12.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.369+0000 7fbd8973e640 1 -- 172.21.15.89:0/2372157804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd8415d6f0 msgr2=0x7fbd8415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:12.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.369+0000 7fbd8973e640 1 --2- 172.21.15.89:0/2372157804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd8415d6f0 0x7fbd8415a090 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c0097c0 tx=0x7fbd6c01ec70 comp rx=0 tx=0).stop 2023-10-30T12:40:12.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.369+0000 7fbd8973e640 1 -- 172.21.15.89:0/2372157804 shutdown_connections 2023-10-30T12:40:12.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.369+0000 7fbd8973e640 1 --2- 172.21.15.89:0/2372157804 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd840cf160 0x7fbd840cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:12.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.369+0000 7fbd8973e640 1 --2- 172.21.15.89:0/2372157804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd8415d6f0 0x7fbd8415a090 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:12.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.369+0000 7fbd8973e640 1 -- 172.21.15.89:0/2372157804 >> 172.21.15.89:0/2372157804 conn(0x7fbd8406bfe0 msgr2=0x7fbd8406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:12.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.369+0000 7fbd8973e640 1 -- 172.21.15.89:0/2372157804 shutdown_connections 2023-10-30T12:40:12.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.369+0000 7fbd8973e640 1 -- 172.21.15.89:0/2372157804 wait complete. 2023-10-30T12:40:12.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.373+0000 7fbd8973e640 1 Processor -- start 2023-10-30T12:40:12.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.373+0000 7fbd8973e640 1 -- start start 2023-10-30T12:40:12.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.373+0000 7fbd8973e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd840cf160 0x7fbd84159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:12.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.373+0000 7fbd8973e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd841538a0 0x7fbd840c52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:12.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.373+0000 7fbd8973e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd8415d6f0 0x7fbd841e2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:12.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.373+0000 7fbd8973e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd841e3bf0 con 0x7fbd840cf160 2023-10-30T12:40:12.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.373+0000 7fbd8973e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd841580f0 con 0x7fbd841538a0 2023-10-30T12:40:12.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.373+0000 7fbd8973e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd840c59a0 con 0x7fbd8415d6f0 2023-10-30T12:40:12.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.373+0000 7fbd82ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd840cf160 0x7fbd84159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:12.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.373+0000 7fbd82ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd840cf160 0x7fbd84159300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50080/0 (socket says 172.21.15.89:50080) 2023-10-30T12:40:12.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.373+0000 7fbd82ffd640 1 -- 172.21.15.89:0/2743779381 learned_addr learned my addr 172.21.15.89:0/2743779381 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:40:12.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.373+0000 7fbd837fe640 1 --2- 172.21.15.89:0/2743779381 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd8415d6f0 0x7fbd841e2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:12.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.373+0000 7fbd7bfff640 1 --2- 172.21.15.89:0/2743779381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd841538a0 0x7fbd840c52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:12.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.373+0000 7fbd82ffd640 1 -- 172.21.15.89:0/2743779381 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd8415d6f0 msgr2=0x7fbd841e2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:12.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.373+0000 7fbd82ffd640 1 --2- 172.21.15.89:0/2743779381 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd8415d6f0 0x7fbd841e2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:12.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.373+0000 7fbd82ffd640 1 -- 172.21.15.89:0/2743779381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd841538a0 msgr2=0x7fbd840c52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:12.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.373+0000 7fbd82ffd640 1 --2- 172.21.15.89:0/2743779381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd841538a0 0x7fbd840c52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:12.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.373+0000 7fbd82ffd640 1 -- 172.21.15.89:0/2743779381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd6c009420 con 0x7fbd840cf160 2023-10-30T12:40:12.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.373+0000 7fbd82ffd640 1 --2- 172.21.15.89:0/2743779381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd840cf160 0x7fbd84159300 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fbd7400c500 tx=0x7fbd7400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:12.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.373+0000 7fbd7b7fe640 1 -- 172.21.15.89:0/2743779381 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd74005cf0 con 0x7fbd840cf160 2023-10-30T12:40:12.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.373+0000 7fbd7b7fe640 1 -- 172.21.15.89:0/2743779381 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd74005e50 con 0x7fbd840cf160 2023-10-30T12:40:12.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.373+0000 7fbd7b7fe640 1 -- 172.21.15.89:0/2743779381 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd740072f0 con 0x7fbd840cf160 2023-10-30T12:40:12.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.373+0000 7fbd8973e640 1 -- 172.21.15.89:0/2743779381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd841584d0 con 0x7fbd840cf160 2023-10-30T12:40:12.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.373+0000 7fbd8973e640 1 -- 172.21.15.89:0/2743779381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd840c4f60 con 0x7fbd840cf160 2023-10-30T12:40:12.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.377+0000 7fbd8973e640 1 -- 172.21.15.89:0/2743779381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd840b2a50 con 0x7fbd840cf160 2023-10-30T12:40:12.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.377+0000 7fbd7b7fe640 1 -- 172.21.15.89:0/2743779381 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbd74007450 con 0x7fbd840cf160 2023-10-30T12:40:12.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.377+0000 7fbd7b7fe640 1 --2- 172.21.15.89:0/2743779381 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbd640664d0 0x7fbd64068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:12.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.377+0000 7fbd7b7fe640 1 -- 172.21.15.89:0/2743779381 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6234+0+0 (secure 0 0 0) 0x7fbd74083630 con 0x7fbd840cf160 2023-10-30T12:40:12.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.377+0000 7fbd7bfff640 1 --2- 172.21.15.89:0/2743779381 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbd640664d0 0x7fbd64068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:12.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.377+0000 7fbd7bfff640 1 --2- 172.21.15.89:0/2743779381 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbd640664d0 0x7fbd64068990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c0b2000 tx=0x7fbd6c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:12.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.381+0000 7fbd7b7fe640 1 -- 172.21.15.89:0/2743779381 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbd740508b0 con 0x7fbd840cf160 2023-10-30T12:40:12.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.561+0000 7fbd8973e640 1 -- 172.21.15.89:0/2743779381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd8404e170 con 0x7fbd840cf160 2023-10-30T12:40:12.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.561+0000 7fbd7b7fe640 1 -- 172.21.15.89:0/2743779381 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+5737 (secure 0 0 0) 0x7fbd74053ed0 con 0x7fbd840cf160 2023-10-30T12:40:12.561 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:40:12.562 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:40:11.121142+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:12.562 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 57 2023-10-30T12:40:12.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.561+0000 7fbd8973e640 1 -- 172.21.15.89:0/2743779381 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbd640664d0 msgr2=0x7fbd64068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:12.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.561+0000 7fbd8973e640 1 --2- 172.21.15.89:0/2743779381 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbd640664d0 0x7fbd64068990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c0b2000 tx=0x7fbd6c008040 comp rx=0 tx=0).stop 2023-10-30T12:40:12.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.561+0000 7fbd8973e640 1 -- 172.21.15.89:0/2743779381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd840cf160 msgr2=0x7fbd84159300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:12.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.561+0000 7fbd8973e640 1 --2- 172.21.15.89:0/2743779381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd840cf160 0x7fbd84159300 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fbd7400c500 tx=0x7fbd7400c9c0 comp rx=0 tx=0).stop 2023-10-30T12:40:12.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.561+0000 7fbd8973e640 1 -- 172.21.15.89:0/2743779381 shutdown_connections 2023-10-30T12:40:12.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.561+0000 7fbd8973e640 1 --2- 172.21.15.89:0/2743779381 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbd640664d0 0x7fbd64068990 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:12.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.561+0000 7fbd8973e640 1 --2- 172.21.15.89:0/2743779381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd841538a0 0x7fbd840c52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:12.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.561+0000 7fbd8973e640 1 --2- 172.21.15.89:0/2743779381 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbd8415d6f0 0x7fbd841e2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:12.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.561+0000 7fbd8973e640 1 --2- 172.21.15.89:0/2743779381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd840cf160 0x7fbd84159300 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:12.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.561+0000 7fbd8973e640 1 -- 172.21.15.89:0/2743779381 >> 172.21.15.89:0/2743779381 conn(0x7fbd8406bfe0 msgr2=0x7fbd8414ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:12.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.561+0000 7fbd8973e640 1 -- 172.21.15.89:0/2743779381 shutdown_connections 2023-10-30T12:40:12.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.561+0000 7fbd8973e640 1 -- 172.21.15.89:0/2743779381 wait complete. 2023-10-30T12:40:12.582 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2023-10-30T12:40:12.582 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:40:12.582 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:12.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.717+0000 7fdcc5cea640 1 Processor -- start 2023-10-30T12:40:12.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.721+0000 7fdcc5cea640 1 -- start start 2023-10-30T12:40:12.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.721+0000 7fdcc5cea640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcc0069e90 0x7fdcc015a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:12.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.721+0000 7fdcc5cea640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc00c6c10 con 0x7fdcc006cb80 2023-10-30T12:40:12.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.721+0000 7fdcc5cea640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc00c6ef0 con 0x7fdcc006a270 2023-10-30T12:40:12.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.721+0000 7fdcc5cea640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc00c8570 con 0x7fdcc0069e90 2023-10-30T12:40:12.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.721+0000 7fdcbffff640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fdcc006cb80 0x7fdcc006d010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44856/0 (socket says 172.21.15.89:44856) 2023-10-30T12:40:12.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.721+0000 7fdcbffff640 1 -- 172.21.15.89:0/4190918419 learned_addr learned my addr 172.21.15.89:0/4190918419 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:40:12.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.721+0000 7fdcbf7fe640 1 --2- 172.21.15.89:0/4190918419 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcc0069e90 0x7fdcc015a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:12.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.721+0000 7fdcbe7fc640 1 -- 172.21.15.89:0/4190918419 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3119225078 0 0) 0x7fdcc00c6c10 con 0x7fdcc006cb80 2023-10-30T12:40:12.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.721+0000 7fdcbe7fc640 1 -- 172.21.15.89:0/4190918419 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdca8003530 con 0x7fdcc006cb80 2023-10-30T12:40:12.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.721+0000 7fdcbe7fc640 1 -- 172.21.15.89:0/4190918419 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2953894558 0 0) 0x7fdca8003530 con 0x7fdcc006cb80 2023-10-30T12:40:12.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.721+0000 7fdcbf7fe640 1 -- 172.21.15.89:0/4190918419 >> v1:172.21.15.192:6789/0 conn(0x7fdcc006a270 legacy=0x7fdcc00c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:12.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.721+0000 7fdcbf7fe640 1 -- 172.21.15.89:0/4190918419 >> v1:172.21.15.89:6789/0 conn(0x7fdcc006cb80 legacy=0x7fdcc006d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:12.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.721+0000 7fdcbf7fe640 1 -- 172.21.15.89:0/4190918419 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc00c71d0 con 0x7fdcc0069e90 2023-10-30T12:40:12.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.725+0000 7fdcbf7fe640 1 --2- 172.21.15.89:0/4190918419 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcc0069e90 0x7fdcc015a410 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdcb00097f0 tx=0x7fdcb001c290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=caad3e393cee87dc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:12.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.725+0000 7fdcbe7fc640 1 -- 172.21.15.89:0/4190918419 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb8001a00 con 0x7fdcc0069e90 2023-10-30T12:40:12.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.725+0000 7fdcbe7fc640 1 -- 172.21.15.89:0/4190918419 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb00a7050 con 0x7fdcc0069e90 2023-10-30T12:40:12.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.725+0000 7fdcbe7fc640 1 -- 172.21.15.89:0/4190918419 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb00ab600 con 0x7fdcc0069e90 2023-10-30T12:40:12.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.725+0000 7fdcc5cea640 1 -- 172.21.15.89:0/4190918419 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcc0069e90 msgr2=0x7fdcc015a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.725+0000 7fdcc5cea640 1 --2- 172.21.15.89:0/4190918419 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcc0069e90 0x7fdcc015a410 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdcb00097f0 tx=0x7fdcb001c290 comp rx=0 tx=0).stop 2023-10-30T12:40:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.725+0000 7fdcc5cea640 1 -- 172.21.15.89:0/4190918419 shutdown_connections 2023-10-30T12:40:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.725+0000 7fdcc5cea640 1 --2- 172.21.15.89:0/4190918419 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcc0069e90 0x7fdcc015a410 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.725+0000 7fdcc5cea640 1 -- 172.21.15.89:0/4190918419 >> 172.21.15.89:0/4190918419 conn(0x7fdcc006dfd0 msgr2=0x7fdcc006e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:12.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.725+0000 7fdcc5cea640 1 -- 172.21.15.89:0/4190918419 shutdown_connections 2023-10-30T12:40:12.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.725+0000 7fdcc5cea640 1 -- 172.21.15.89:0/4190918419 wait complete. 2023-10-30T12:40:12.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.729+0000 7fdcc5cea640 1 Processor -- start 2023-10-30T12:40:12.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.729+0000 7fdcc5cea640 1 -- start start 2023-10-30T12:40:12.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.729+0000 7fdcc5cea640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcc0069e90 0x7fdcc014de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:12.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.729+0000 7fdcc5cea640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcc006a270 0x7fdcc015bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:12.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.729+0000 7fdcc5cea640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcc006cb80 0x7fdcc015ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:12.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.729+0000 7fdcc5cea640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc01478d0 con 0x7fdcc006a270 2023-10-30T12:40:12.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.729+0000 7fdcc5cea640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc015d310 con 0x7fdcc0069e90 2023-10-30T12:40:12.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.729+0000 7fdcc5cea640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc015c210 con 0x7fdcc006cb80 2023-10-30T12:40:12.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.729+0000 7fdcbeffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcc006a270 0x7fdcc015bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:12.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.729+0000 7fdcbeffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcc006a270 0x7fdcc015bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50084/0 (socket says 172.21.15.89:50084) 2023-10-30T12:40:12.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.729+0000 7fdcbeffd640 1 -- 172.21.15.89:0/1247763634 learned_addr learned my addr 172.21.15.89:0/1247763634 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:40:12.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.729+0000 7fdcbf7fe640 1 --2- 172.21.15.89:0/1247763634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcc0069e90 0x7fdcc014de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:12.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.729+0000 7fdcbffff640 1 --2- 172.21.15.89:0/1247763634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcc006cb80 0x7fdcc015ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:12.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.729+0000 7fdcbeffd640 1 -- 172.21.15.89:0/1247763634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcc006cb80 msgr2=0x7fdcc015ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:12.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.729+0000 7fdcbeffd640 1 --2- 172.21.15.89:0/1247763634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcc006cb80 0x7fdcc015ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:12.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.729+0000 7fdcbeffd640 1 -- 172.21.15.89:0/1247763634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcc0069e90 msgr2=0x7fdcc014de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:12.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.729+0000 7fdcbeffd640 1 --2- 172.21.15.89:0/1247763634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcc0069e90 0x7fdcc014de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:12.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.729+0000 7fdcbeffd640 1 -- 172.21.15.89:0/1247763634 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb0009420 con 0x7fdcc006a270 2023-10-30T12:40:12.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.729+0000 7fdcbf7fe640 1 --2- 172.21.15.89:0/1247763634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcc0069e90 0x7fdcc014de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:40:12.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.729+0000 7fdcbeffd640 1 --2- 172.21.15.89:0/1247763634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcc006a270 0x7fdcc015bcd0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fdcb400c7b0 tx=0x7fdcb400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:12.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.729+0000 7fdca7fff640 1 -- 172.21.15.89:0/1247763634 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb4005270 con 0x7fdcc006a270 2023-10-30T12:40:12.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.729+0000 7fdca7fff640 1 -- 172.21.15.89:0/1247763634 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb40053d0 con 0x7fdcc006a270 2023-10-30T12:40:12.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.729+0000 7fdcc5cea640 1 -- 172.21.15.89:0/1247763634 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcc014d9e0 con 0x7fdcc006a270 2023-10-30T12:40:12.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.729+0000 7fdca7fff640 1 -- 172.21.15.89:0/1247763634 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb4003840 con 0x7fdcc006a270 2023-10-30T12:40:12.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.729+0000 7fdcc5cea640 1 -- 172.21.15.89:0/1247763634 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcc015d030 con 0x7fdcc006a270 2023-10-30T12:40:12.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.729+0000 7fdcc5cea640 1 -- 172.21.15.89:0/1247763634 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcc014db70 con 0x7fdcc006a270 2023-10-30T12:40:12.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.733+0000 7fdca7fff640 1 -- 172.21.15.89:0/1247763634 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdcb40039a0 con 0x7fdcc006a270 2023-10-30T12:40:12.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.733+0000 7fdca7fff640 1 --2- 172.21.15.89:0/1247763634 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fdcac0664a0 0x7fdcac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:12.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.733+0000 7fdca7fff640 1 -- 172.21.15.89:0/1247763634 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6234+0+0 (secure 0 0 0) 0x7fdcb40837f0 con 0x7fdcc006a270 2023-10-30T12:40:12.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.733+0000 7fdcbf7fe640 1 --2- 172.21.15.89:0/1247763634 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fdcac0664a0 0x7fdcac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:12.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.733+0000 7fdcbf7fe640 1 --2- 172.21.15.89:0/1247763634 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fdcac0664a0 0x7fdcac068960 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fdcb00ad000 tx=0x7fdcb00a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:12.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.737+0000 7fdca7fff640 1 -- 172.21.15.89:0/1247763634 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdcb40509f0 con 0x7fdcc006a270 2023-10-30T12:40:12.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.917+0000 7fdcc5cea640 1 -- 172.21.15.89:0/1247763634 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdcc004e170 con 0x7fdcc006a270 2023-10-30T12:40:12.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.917+0000 7fdca7fff640 1 -- 172.21.15.89:0/1247763634 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+5737 (secure 0 0 0) 0x7fdcb4017020 con 0x7fdcc006a270 2023-10-30T12:40:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:40:12.919 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:40:11.121142+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:12.919 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 57 2023-10-30T12:40:12.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.921+0000 7fdcc5cea640 1 -- 172.21.15.89:0/1247763634 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fdcac0664a0 msgr2=0x7fdcac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:12.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.921+0000 7fdcc5cea640 1 --2- 172.21.15.89:0/1247763634 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fdcac0664a0 0x7fdcac068960 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fdcb00ad000 tx=0x7fdcb00a6000 comp rx=0 tx=0).stop 2023-10-30T12:40:12.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.921+0000 7fdcc5cea640 1 -- 172.21.15.89:0/1247763634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcc006a270 msgr2=0x7fdcc015bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:12.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.921+0000 7fdcc5cea640 1 --2- 172.21.15.89:0/1247763634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcc006a270 0x7fdcc015bcd0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fdcb400c7b0 tx=0x7fdcb400cc70 comp rx=0 tx=0).stop 2023-10-30T12:40:12.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.921+0000 7fdcc5cea640 1 -- 172.21.15.89:0/1247763634 shutdown_connections 2023-10-30T12:40:12.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.921+0000 7fdcc5cea640 1 --2- 172.21.15.89:0/1247763634 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fdcac0664a0 0x7fdcac068960 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:12.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.921+0000 7fdcc5cea640 1 --2- 172.21.15.89:0/1247763634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcc006cb80 0x7fdcc015ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:12.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.921+0000 7fdcc5cea640 1 --2- 172.21.15.89:0/1247763634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdcc006a270 0x7fdcc015bcd0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:12.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.921+0000 7fdcc5cea640 1 --2- 172.21.15.89:0/1247763634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcc0069e90 0x7fdcc014de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:12.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.921+0000 7fdcc5cea640 1 -- 172.21.15.89:0/1247763634 >> 172.21.15.89:0/1247763634 conn(0x7fdcc006dfd0 msgr2=0x7fdcc0071e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:12.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.921+0000 7fdcc5cea640 1 -- 172.21.15.89:0/1247763634 shutdown_connections 2023-10-30T12:40:12.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:12.921+0000 7fdcc5cea640 1 -- 172.21.15.89:0/1247763634 wait complete. 2023-10-30T12:40:12.940 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 57, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:50.729245+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 57, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5018": { "addr": "172.21.15.192:6839/93914587", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6838", "nonce": 93914587, "type": "v2" }, { "addr": "172.21.15.192:6839", "nonce": 93914587, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5018, "incarnation": 44, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 41 }, "gid_5129": { "addr": "172.21.15.192:6837/1413360446", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6836", "nonce": 1413360446, "type": "v2" }, { "addr": "172.21.15.192:6837", "nonce": 1413360446, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5129, "incarnation": 57, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:replay", "state_seq": 1 }, "gid_5139": { "addr": "172.21.15.89:6837/951276252", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6836", "nonce": 951276252, "type": "v2" }, { "addr": "172.21.15.89:6837", "nonce": 951276252, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5139, "incarnation": 33, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 29 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:40:11.121142+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5018, "mds_1": 5139, "mds_2": 5129 } } } ], "standbys": [ { "addr": "172.21.15.89:6834/2909790173", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6832", "nonce": 2909790173, "type": "v2" }, { "addr": "172.21.15.89:6834", "nonce": 2909790173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.89:6835/2387357615", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6833", "nonce": 2387357615, "type": "v2" }, { "addr": "172.21.15.89:6835", "nonce": 2387357615, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4239, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:40:14.941 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:15.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56c4f9d640 1 Processor -- start 2023-10-30T12:40:15.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56c4f9d640 1 -- start start 2023-10-30T12:40:15.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56c4f9d640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f56c01538a0 0x7f56c015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:15.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56c4f9d640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56c00cb800 con 0x7f56c00ca0e0 2023-10-30T12:40:15.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56c4f9d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56c015dc30 con 0x7f56c00cf160 2023-10-30T12:40:15.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56c4f9d640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56c00cb330 con 0x7f56c01538a0 2023-10-30T12:40:15.091 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56bdd9b640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f56c00ca0e0 0x7f56c015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:44860/0 (socket says 172.21.15.89:44860) 2023-10-30T12:40:15.091 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56bdd9b640 1 -- 172.21.15.89:0/196082059 learned_addr learned my addr 172.21.15.89:0/196082059 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:40:15.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56be59c640 1 --2- 172.21.15.89:0/196082059 >> v2:172.21.15.192:3300/0 conn(0x7f56c01538a0 0x7f56c015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:15.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56bd59a640 1 -- 172.21.15.89:0/196082059 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1590712056 0 0) 0x7f56c00cb800 con 0x7f56c00ca0e0 2023-10-30T12:40:15.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56bd59a640 1 -- 172.21.15.89:0/196082059 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56a8003530 con 0x7f56c00ca0e0 2023-10-30T12:40:15.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56bd59a640 1 -- 172.21.15.89:0/196082059 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3467781390 0 0) 0x7f56a8003530 con 0x7f56c00ca0e0 2023-10-30T12:40:15.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56bd59a640 1 -- 172.21.15.89:0/196082059 >> v2:172.21.15.192:3300/0 conn(0x7f56c01538a0 msgr2=0x7f56c015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:15.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56bd59a640 1 --2- 172.21.15.89:0/196082059 >> v2:172.21.15.192:3300/0 conn(0x7f56c01538a0 0x7f56c015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:15.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56bd59a640 1 -- 172.21.15.89:0/196082059 >> v1:172.21.15.192:6789/0 conn(0x7f56c00cf160 legacy=0x7f56c0164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:15.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56bd59a640 1 -- 172.21.15.89:0/196082059 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c00cad70 con 0x7f56c00ca0e0 2023-10-30T12:40:15.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56bd59a640 1 -- 172.21.15.89:0/196082059 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f56b4001680 con 0x7f56c00ca0e0 2023-10-30T12:40:15.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56be59c640 1 --2- 172.21.15.89:0/196082059 >> v2:172.21.15.192:3300/0 conn(0x7f56c01538a0 0x7f56c015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:40:15.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56bd59a640 1 -- 172.21.15.89:0/196082059 >> v1:172.21.15.89:6789/0 conn(0x7f56c00ca0e0 legacy=0x7f56c015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:15.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56bd59a640 1 --2- 172.21.15.89:0/196082059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a8003980 0x7f56a8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:15.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56bd59a640 1 --2- 172.21.15.89:0/196082059 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56a80047d0 0x7f56a8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:15.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56bd59a640 1 --2- 172.21.15.89:0/196082059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56a8005190 0x7f56a800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:15.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56bd59a640 1 -- 172.21.15.89:0/196082059 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c00cb330 con 0x7f56a8003980 2023-10-30T12:40:15.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56bd59a640 1 -- 172.21.15.89:0/196082059 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a800c040 con 0x7f56a8005190 2023-10-30T12:40:15.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56bd59a640 1 -- 172.21.15.89:0/196082059 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56a800c200 con 0x7f56a80047d0 2023-10-30T12:40:15.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56be59c640 1 --2- 172.21.15.89:0/196082059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a8003980 0x7f56a8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:15.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56bed9d640 1 --2- 172.21.15.89:0/196082059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56a8005190 0x7f56a800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:15.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56bdd9b640 1 --2- 172.21.15.89:0/196082059 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56a80047d0 0x7f56a8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:15.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56be59c640 1 -- 172.21.15.89:0/196082059 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56a80047d0 msgr2=0x7f56a8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:15.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56be59c640 1 --2- 172.21.15.89:0/196082059 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56a80047d0 0x7f56a8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:15.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56be59c640 1 -- 172.21.15.89:0/196082059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56a8005190 msgr2=0x7f56a800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:15.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56be59c640 1 --2- 172.21.15.89:0/196082059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56a8005190 0x7f56a800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:15.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56be59c640 1 -- 172.21.15.89:0/196082059 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a8003530 con 0x7f56a8003980 2023-10-30T12:40:15.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.089+0000 7f56bed9d640 1 -- 172.21.15.89:0/196082059 reap_dead start 2023-10-30T12:40:15.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.093+0000 7f56be59c640 1 --2- 172.21.15.89:0/196082059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a8003980 0x7f56a8003da0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f56b0009cb0 tx=0x7f56b001cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78e226512afc8c4d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:15.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.093+0000 7f56bd59a640 1 -- 172.21.15.89:0/196082059 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b00a57a0 con 0x7f56a8003980 2023-10-30T12:40:15.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.093+0000 7f56bd59a640 1 -- 172.21.15.89:0/196082059 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b00a5900 con 0x7f56a8003980 2023-10-30T12:40:15.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.093+0000 7f56bd59a640 1 -- 172.21.15.89:0/196082059 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b00a5cf0 con 0x7f56a8003980 2023-10-30T12:40:15.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.093+0000 7f56c4f9d640 1 -- 172.21.15.89:0/196082059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a8003980 msgr2=0x7f56a8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:15.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.093+0000 7f56c4f9d640 1 --2- 172.21.15.89:0/196082059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a8003980 0x7f56a8003da0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f56b0009cb0 tx=0x7f56b001cbf0 comp rx=0 tx=0).stop 2023-10-30T12:40:15.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.093+0000 7f56c4f9d640 1 -- 172.21.15.89:0/196082059 shutdown_connections 2023-10-30T12:40:15.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.093+0000 7f56c4f9d640 1 --2- 172.21.15.89:0/196082059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56a8003980 0x7f56a8003da0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:15.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.093+0000 7f56c4f9d640 1 -- 172.21.15.89:0/196082059 >> 172.21.15.89:0/196082059 conn(0x7f56c006bfe0 msgr2=0x7f56c006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:15.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.093+0000 7f56c4f9d640 1 -- 172.21.15.89:0/196082059 shutdown_connections 2023-10-30T12:40:15.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.093+0000 7f56c4f9d640 1 -- 172.21.15.89:0/196082059 wait complete. 2023-10-30T12:40:15.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.097+0000 7f56c4f9d640 1 Processor -- start 2023-10-30T12:40:15.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.097+0000 7f56c4f9d640 1 -- start start 2023-10-30T12:40:15.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.097+0000 7f56c4f9d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56a8003980 0x7f56c01e4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:15.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.097+0000 7f56c4f9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56c01e5330 0x7f56c0146ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:15.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.097+0000 7f56c4f9d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56c01e41a0 0x7f56c01509f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:15.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.097+0000 7f56c4f9d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c01f2740 con 0x7f56c01e5330 2023-10-30T12:40:15.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.097+0000 7f56c4f9d640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c01f35a0 con 0x7f56a8003980 2023-10-30T12:40:15.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.097+0000 7f56c4f9d640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c01f2990 con 0x7f56c01e41a0 2023-10-30T12:40:15.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.097+0000 7f56bdd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56c01e5330 0x7f56c0146ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:15.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.097+0000 7f56bdd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56c01e5330 0x7f56c0146ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50108/0 (socket says 172.21.15.89:50108) 2023-10-30T12:40:15.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.097+0000 7f56bdd9b640 1 -- 172.21.15.89:0/3413068574 learned_addr learned my addr 172.21.15.89:0/3413068574 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:40:15.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.097+0000 7f56be59c640 1 --2- 172.21.15.89:0/3413068574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56a8003980 0x7f56c01e4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:15.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.097+0000 7f56bed9d640 1 --2- 172.21.15.89:0/3413068574 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56c01e41a0 0x7f56c01509f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:15.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.097+0000 7f56bdd9b640 1 -- 172.21.15.89:0/3413068574 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56c01e41a0 msgr2=0x7f56c01509f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:15.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.097+0000 7f56bdd9b640 1 --2- 172.21.15.89:0/3413068574 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56c01e41a0 0x7f56c01509f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:15.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.097+0000 7f56bdd9b640 1 -- 172.21.15.89:0/3413068574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56a8003980 msgr2=0x7f56c01e4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:15.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.097+0000 7f56bdd9b640 1 --2- 172.21.15.89:0/3413068574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56a8003980 0x7f56c01e4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:15.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.097+0000 7f56bdd9b640 1 -- 172.21.15.89:0/3413068574 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c01f2c70 con 0x7f56c01e5330 2023-10-30T12:40:15.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.097+0000 7f56be59c640 1 --2- 172.21.15.89:0/3413068574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56a8003980 0x7f56c01e4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:40:15.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.097+0000 7f56bdd9b640 1 --2- 172.21.15.89:0/3413068574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56c01e5330 0x7f56c0146ee0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f56b4011980 tx=0x7f56b4011e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:15.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.097+0000 7f56aeffd640 1 -- 172.21.15.89:0/3413068574 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b400fc20 con 0x7f56c01e5330 2023-10-30T12:40:15.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.097+0000 7f56aeffd640 1 -- 172.21.15.89:0/3413068574 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b400fda0 con 0x7f56c01e5330 2023-10-30T12:40:15.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.097+0000 7f56aeffd640 1 -- 172.21.15.89:0/3413068574 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b4015650 con 0x7f56c01e5330 2023-10-30T12:40:15.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.097+0000 7f56c4f9d640 1 -- 172.21.15.89:0/3413068574 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56c01f2f50 con 0x7f56c01e5330 2023-10-30T12:40:15.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.097+0000 7f56c4f9d640 1 -- 172.21.15.89:0/3413068574 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56c0147420 con 0x7f56c01e5330 2023-10-30T12:40:15.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.101+0000 7f56c4f9d640 1 -- 172.21.15.89:0/3413068574 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56c01ef1d0 con 0x7f56c01e5330 2023-10-30T12:40:15.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.101+0000 7f56aeffd640 1 -- 172.21.15.89:0/3413068574 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f56b40157b0 con 0x7f56c01e5330 2023-10-30T12:40:15.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.101+0000 7f56aeffd640 1 --2- 172.21.15.89:0/3413068574 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f56a00664a0 0x7f56a0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:15.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.101+0000 7f56aeffd640 1 -- 172.21.15.89:0/3413068574 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6234+0+0 (secure 0 0 0) 0x7f56b4087520 con 0x7f56c01e5330 2023-10-30T12:40:15.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.101+0000 7f56be59c640 1 --2- 172.21.15.89:0/3413068574 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f56a00664a0 0x7f56a0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:15.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.101+0000 7f56be59c640 1 --2- 172.21.15.89:0/3413068574 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f56a00664a0 0x7f56a0068960 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f56b00af000 tx=0x7f56b0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:15.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.105+0000 7f56aeffd640 1 -- 172.21.15.89:0/3413068574 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f56b40546f0 con 0x7f56c01e5330 2023-10-30T12:40:15.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.297+0000 7f56c4f9d640 1 -- 172.21.15.89:0/3413068574 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56c004e170 con 0x7f56c01e5330 2023-10-30T12:40:15.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.297+0000 7f56aeffd640 1 -- 172.21.15.89:0/3413068574 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+5737 (secure 0 0 0) 0x7f56b4057d10 con 0x7f56c01e5330 2023-10-30T12:40:15.296 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:40:15.296 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:40:11.121142+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:15.297 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 57 2023-10-30T12:40:15.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.297+0000 7f56c4f9d640 1 -- 172.21.15.89:0/3413068574 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f56a00664a0 msgr2=0x7f56a0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:15.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.297+0000 7f56c4f9d640 1 --2- 172.21.15.89:0/3413068574 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f56a00664a0 0x7f56a0068960 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f56b00af000 tx=0x7f56b0008040 comp rx=0 tx=0).stop 2023-10-30T12:40:15.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.297+0000 7f56c4f9d640 1 -- 172.21.15.89:0/3413068574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56c01e5330 msgr2=0x7f56c0146ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:15.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.297+0000 7f56c4f9d640 1 --2- 172.21.15.89:0/3413068574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56c01e5330 0x7f56c0146ee0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f56b4011980 tx=0x7f56b4011e40 comp rx=0 tx=0).stop 2023-10-30T12:40:15.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.297+0000 7f56c4f9d640 1 -- 172.21.15.89:0/3413068574 shutdown_connections 2023-10-30T12:40:15.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.297+0000 7f56c4f9d640 1 --2- 172.21.15.89:0/3413068574 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f56a00664a0 0x7f56a0068960 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:15.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.297+0000 7f56c4f9d640 1 --2- 172.21.15.89:0/3413068574 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f56c01e41a0 0x7f56c01509f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:15.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.297+0000 7f56c4f9d640 1 --2- 172.21.15.89:0/3413068574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56c01e5330 0x7f56c0146ee0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:15.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.297+0000 7f56c4f9d640 1 --2- 172.21.15.89:0/3413068574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56a8003980 0x7f56c01e4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:15.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.297+0000 7f56c4f9d640 1 -- 172.21.15.89:0/3413068574 >> 172.21.15.89:0/3413068574 conn(0x7f56c006bfe0 msgr2=0x7f56c0073480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:15.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.297+0000 7f56c4f9d640 1 -- 172.21.15.89:0/3413068574 shutdown_connections 2023-10-30T12:40:15.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:15.297+0000 7f56c4f9d640 1 -- 172.21.15.89:0/3413068574 wait complete. 2023-10-30T12:40:15.317 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:40:16.756 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:40:16.760 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:40:17.317 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:17.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.469+0000 7fa9861d3640 1 Processor -- start 2023-10-30T12:40:17.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.469+0000 7fa9861d3640 1 -- start start 2023-10-30T12:40:17.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.469+0000 7fa9861d3640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa98006f7a0 0x7fa98006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:17.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.469+0000 7fa9861d3640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa980071110 0x7fa980158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:17.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.469+0000 7fa9861d3640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa980066cf0 con 0x7fa9800c6bd0 2023-10-30T12:40:17.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.469+0000 7fa9861d3640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9800700c0 con 0x7fa980071110 2023-10-30T12:40:17.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.469+0000 7fa9861d3640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9800c70d0 con 0x7fa98006f7a0 2023-10-30T12:40:17.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.469+0000 7fa97ffff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa980071110 0x7fa980158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:17.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.469+0000 7fa97ffff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa980071110 0x7fa980158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50114/0 (socket says 172.21.15.89:50114) 2023-10-30T12:40:17.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.469+0000 7fa97ffff640 1 -- 172.21.15.89:0/832056416 learned_addr learned my addr 172.21.15.89:0/832056416 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:40:17.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.469+0000 7fa97f7fe640 1 --2- 172.21.15.89:0/832056416 >> v2:172.21.15.192:3300/0 conn(0x7fa98006f7a0 0x7fa98006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:17.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.469+0000 7fa97ffff640 1 -- 172.21.15.89:0/832056416 >> v2:172.21.15.192:3300/0 conn(0x7fa98006f7a0 msgr2=0x7fa98006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:17.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.469+0000 7fa97ffff640 1 --2- 172.21.15.89:0/832056416 >> v2:172.21.15.192:3300/0 conn(0x7fa98006f7a0 0x7fa98006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:17.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.469+0000 7fa97ffff640 1 -- 172.21.15.89:0/832056416 >> v1:172.21.15.192:6789/0 conn(0x7fa9800c6bd0 legacy=0x7fa980069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:17.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.469+0000 7fa97ffff640 1 -- 172.21.15.89:0/832056416 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9800c7390 con 0x7fa980071110 2023-10-30T12:40:17.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.473+0000 7fa97ffff640 1 --2- 172.21.15.89:0/832056416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa980071110 0x7fa980158110 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fa98006c830 tx=0x7fa97801f180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=793984e8f9a56af1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:17.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.473+0000 7fa97e7fc640 1 -- 172.21.15.89:0/832056416 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9780af400 con 0x7fa980071110 2023-10-30T12:40:17.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.473+0000 7fa97e7fc640 1 -- 172.21.15.89:0/832056416 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9780ab050 con 0x7fa980071110 2023-10-30T12:40:17.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.473+0000 7fa97e7fc640 1 -- 172.21.15.89:0/832056416 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9780af780 con 0x7fa980071110 2023-10-30T12:40:17.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.473+0000 7fa9861d3640 1 -- 172.21.15.89:0/832056416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa980071110 msgr2=0x7fa980158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:17.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.473+0000 7fa9861d3640 1 --2- 172.21.15.89:0/832056416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa980071110 0x7fa980158110 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fa98006c830 tx=0x7fa97801f180 comp rx=0 tx=0).stop 2023-10-30T12:40:17.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.473+0000 7fa9861d3640 1 -- 172.21.15.89:0/832056416 shutdown_connections 2023-10-30T12:40:17.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.473+0000 7fa9861d3640 1 --2- 172.21.15.89:0/832056416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa980071110 0x7fa980158110 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:17.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.473+0000 7fa9861d3640 1 --2- 172.21.15.89:0/832056416 >> v2:172.21.15.192:3300/0 conn(0x7fa98006f7a0 0x7fa98006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:17.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.473+0000 7fa9861d3640 1 -- 172.21.15.89:0/832056416 >> 172.21.15.89:0/832056416 conn(0x7fa9800c7910 msgr2=0x7fa9800c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:17.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.473+0000 7fa9861d3640 1 -- 172.21.15.89:0/832056416 shutdown_connections 2023-10-30T12:40:17.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.473+0000 7fa9861d3640 1 -- 172.21.15.89:0/832056416 wait complete. 2023-10-30T12:40:17.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.477+0000 7fa9861d3640 1 Processor -- start 2023-10-30T12:40:17.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.477+0000 7fa9861d3640 1 -- start start 2023-10-30T12:40:17.480 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.477+0000 7fa9861d3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa98006f7a0 0x7fa9801de950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:17.480 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.477+0000 7fa9861d3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa980071110 0x7fa98014d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:17.480 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.477+0000 7fa9861d3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9800c6bd0 0x7fa980151f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:17.480 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.477+0000 7fa9861d3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9801f07f0 con 0x7fa98006f7a0 2023-10-30T12:40:17.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.477+0000 7fa9861d3640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9801eff00 con 0x7fa980071110 2023-10-30T12:40:17.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.477+0000 7fa9861d3640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9801e3fc0 con 0x7fa9800c6bd0 2023-10-30T12:40:17.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.477+0000 7fa97f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa98006f7a0 0x7fa9801de950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:17.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.477+0000 7fa97f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa98006f7a0 0x7fa9801de950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50130/0 (socket says 172.21.15.89:50130) 2023-10-30T12:40:17.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.477+0000 7fa97f7fe640 1 -- 172.21.15.89:0/705417240 learned_addr learned my addr 172.21.15.89:0/705417240 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:40:17.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.477+0000 7fa97ffff640 1 --2- 172.21.15.89:0/705417240 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9800c6bd0 0x7fa980151f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:17.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.477+0000 7fa97effd640 1 --2- 172.21.15.89:0/705417240 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa980071110 0x7fa98014d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:17.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.477+0000 7fa97f7fe640 1 -- 172.21.15.89:0/705417240 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9800c6bd0 msgr2=0x7fa980151f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:17.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.477+0000 7fa97f7fe640 1 --2- 172.21.15.89:0/705417240 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9800c6bd0 0x7fa980151f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:17.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.477+0000 7fa97f7fe640 1 -- 172.21.15.89:0/705417240 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa980071110 msgr2=0x7fa98014d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:17.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.477+0000 7fa97f7fe640 1 --2- 172.21.15.89:0/705417240 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa980071110 0x7fa98014d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:17.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.477+0000 7fa97f7fe640 1 -- 172.21.15.89:0/705417240 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa978009b30 con 0x7fa98006f7a0 2023-10-30T12:40:17.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.477+0000 7fa97ffff640 1 --2- 172.21.15.89:0/705417240 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9800c6bd0 0x7fa980151f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:40:17.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.477+0000 7fa97effd640 1 --2- 172.21.15.89:0/705417240 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa980071110 0x7fa98014d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:40:17.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.477+0000 7fa97f7fe640 1 --2- 172.21.15.89:0/705417240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa98006f7a0 0x7fa9801de950 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa97000d5f0 tx=0x7fa97000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:17.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.477+0000 7fa967fff640 1 -- 172.21.15.89:0/705417240 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9700125f0 con 0x7fa98006f7a0 2023-10-30T12:40:17.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.477+0000 7fa967fff640 1 -- 172.21.15.89:0/705417240 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa970012750 con 0x7fa98006f7a0 2023-10-30T12:40:17.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.477+0000 7fa967fff640 1 -- 172.21.15.89:0/705417240 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa970012ae0 con 0x7fa98006f7a0 2023-10-30T12:40:17.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.477+0000 7fa9861d3640 1 -- 172.21.15.89:0/705417240 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9801e42a0 con 0x7fa98006f7a0 2023-10-30T12:40:17.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.477+0000 7fa9861d3640 1 -- 172.21.15.89:0/705417240 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9801f0480 con 0x7fa98006f7a0 2023-10-30T12:40:17.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.481+0000 7fa9861d3640 1 -- 172.21.15.89:0/705417240 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa98004e170 con 0x7fa98006f7a0 2023-10-30T12:40:17.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.481+0000 7fa967fff640 1 -- 172.21.15.89:0/705417240 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa970012c40 con 0x7fa98006f7a0 2023-10-30T12:40:17.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.481+0000 7fa967fff640 1 --2- 172.21.15.89:0/705417240 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa96c0664d0 0x7fa96c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:17.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.481+0000 7fa967fff640 1 -- 172.21.15.89:0/705417240 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6234+0+0 (secure 0 0 0) 0x7fa970084500 con 0x7fa98006f7a0 2023-10-30T12:40:17.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.481+0000 7fa97effd640 1 --2- 172.21.15.89:0/705417240 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa96c0664d0 0x7fa96c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:17.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.481+0000 7fa97effd640 1 --2- 172.21.15.89:0/705417240 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa96c0664d0 0x7fa96c068990 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa980158410 tx=0x7fa974009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:17.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.485+0000 7fa967fff640 1 -- 172.21.15.89:0/705417240 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa970051760 con 0x7fa98006f7a0 2023-10-30T12:40:17.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.665+0000 7fa9861d3640 1 -- 172.21.15.89:0/705417240 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9801e4610 con 0x7fa98006f7a0 2023-10-30T12:40:17.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.665+0000 7fa967fff640 1 -- 172.21.15.89:0/705417240 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+5739 (secure 0 0 0) 0x7fa970007380 con 0x7fa98006f7a0 2023-10-30T12:40:17.666 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:40:17.666 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:40:16.958000+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:resolve","state_seq":34,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:17.667 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 58 2023-10-30T12:40:17.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.665+0000 7fa9861d3640 1 -- 172.21.15.89:0/705417240 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa96c0664d0 msgr2=0x7fa96c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:17.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.665+0000 7fa9861d3640 1 --2- 172.21.15.89:0/705417240 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa96c0664d0 0x7fa96c068990 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa980158410 tx=0x7fa974009040 comp rx=0 tx=0).stop 2023-10-30T12:40:17.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.665+0000 7fa9861d3640 1 -- 172.21.15.89:0/705417240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa98006f7a0 msgr2=0x7fa9801de950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:17.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.665+0000 7fa9861d3640 1 --2- 172.21.15.89:0/705417240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa98006f7a0 0x7fa9801de950 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa97000d5f0 tx=0x7fa97000dab0 comp rx=0 tx=0).stop 2023-10-30T12:40:17.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.665+0000 7fa9861d3640 1 -- 172.21.15.89:0/705417240 shutdown_connections 2023-10-30T12:40:17.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.665+0000 7fa9861d3640 1 --2- 172.21.15.89:0/705417240 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa96c0664d0 0x7fa96c068990 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:17.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.665+0000 7fa9861d3640 1 --2- 172.21.15.89:0/705417240 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa980071110 0x7fa98014d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:17.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.665+0000 7fa9861d3640 1 --2- 172.21.15.89:0/705417240 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa9800c6bd0 0x7fa980151f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:17.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.665+0000 7fa9861d3640 1 --2- 172.21.15.89:0/705417240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa98006f7a0 0x7fa9801de950 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:17.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.665+0000 7fa9861d3640 1 -- 172.21.15.89:0/705417240 >> 172.21.15.89:0/705417240 conn(0x7fa9800c7910 msgr2=0x7fa9800c9270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:17.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.669+0000 7fa9861d3640 1 -- 172.21.15.89:0/705417240 shutdown_connections 2023-10-30T12:40:17.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:17.669+0000 7fa9861d3640 1 -- 172.21.15.89:0/705417240 wait complete. 2023-10-30T12:40:17.686 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:40:19.687 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:19.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.837+0000 7fac6333b640 1 Processor -- start 2023-10-30T12:40:19.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.837+0000 7fac6333b640 1 -- start start 2023-10-30T12:40:19.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.837+0000 7fac6333b640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fac5c1538f0 0x7fac5c159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:19.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.837+0000 7fac6333b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac5c15cd50 0x7fac5c0c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:19.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.837+0000 7fac6333b640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac5c15d300 con 0x7fac5c0cefa0 2023-10-30T12:40:19.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.837+0000 7fac6333b640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac5c15d5e0 con 0x7fac5c1538f0 2023-10-30T12:40:19.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.837+0000 7fac6333b640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c15d8c0 con 0x7fac5c15cd50 2023-10-30T12:40:19.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.837+0000 7fac62339640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fac5c1538f0 0x7fac5c159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:19.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.837+0000 7fac62b3a640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fac5c0cefa0 0x7fac5c0cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:45876/0 (socket says 172.21.15.89:45876) 2023-10-30T12:40:19.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.837+0000 7fac62b3a640 1 -- 172.21.15.89:0/2653842294 learned_addr learned my addr 172.21.15.89:0/2653842294 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:40:19.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.837+0000 7fac61b38640 1 --2- 172.21.15.89:0/2653842294 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac5c15cd50 0x7fac5c0c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:19.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.837+0000 7fac62339640 1 -- 172.21.15.89:0/2653842294 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac5c15cd50 msgr2=0x7fac5c0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:19.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.837+0000 7fac62339640 1 --2- 172.21.15.89:0/2653842294 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac5c15cd50 0x7fac5c0c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:19.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.837+0000 7fac62339640 1 -- 172.21.15.89:0/2653842294 >> v1:172.21.15.89:6789/0 conn(0x7fac5c0cefa0 legacy=0x7fac5c0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:19.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.837+0000 7fac62339640 1 -- 172.21.15.89:0/2653842294 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5c15dba0 con 0x7fac5c1538f0 2023-10-30T12:40:19.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.841+0000 7fac62339640 1 --2- 172.21.15.89:0/2653842294 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac5c1538f0 0x7fac5c159fb0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fac580097c0 tx=0x7fac5801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9cb6f6aced4c1b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:19.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.841+0000 7fac61337640 1 -- 172.21.15.89:0/2653842294 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac580a8970 con 0x7fac5c1538f0 2023-10-30T12:40:19.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.841+0000 7fac61337640 1 -- 172.21.15.89:0/2653842294 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac580a8ad0 con 0x7fac5c1538f0 2023-10-30T12:40:19.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.841+0000 7fac61337640 1 -- 172.21.15.89:0/2653842294 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac580a8e60 con 0x7fac5c1538f0 2023-10-30T12:40:19.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.841+0000 7fac6333b640 1 -- 172.21.15.89:0/2653842294 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac5c1538f0 msgr2=0x7fac5c159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:19.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.841+0000 7fac6333b640 1 --2- 172.21.15.89:0/2653842294 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac5c1538f0 0x7fac5c159fb0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fac580097c0 tx=0x7fac5801ec70 comp rx=0 tx=0).stop 2023-10-30T12:40:19.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.841+0000 7fac6333b640 1 -- 172.21.15.89:0/2653842294 shutdown_connections 2023-10-30T12:40:19.842 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.841+0000 7fac6333b640 1 --2- 172.21.15.89:0/2653842294 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac5c15cd50 0x7fac5c0c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:19.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.841+0000 7fac6333b640 1 --2- 172.21.15.89:0/2653842294 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac5c1538f0 0x7fac5c159fb0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:19.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.841+0000 7fac6333b640 1 -- 172.21.15.89:0/2653842294 >> 172.21.15.89:0/2653842294 conn(0x7fac5c06a640 msgr2=0x7fac5c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:19.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.841+0000 7fac6333b640 1 -- 172.21.15.89:0/2653842294 shutdown_connections 2023-10-30T12:40:19.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.841+0000 7fac6333b640 1 -- 172.21.15.89:0/2653842294 wait complete. 2023-10-30T12:40:19.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.841+0000 7fac6333b640 1 Processor -- start 2023-10-30T12:40:19.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.841+0000 7fac6333b640 1 -- start start 2023-10-30T12:40:19.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.845+0000 7fac6333b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac5c0cefa0 0x7fac5c0b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:19.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.845+0000 7fac6333b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac5c1538f0 0x7fac5c1e3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:19.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.845+0000 7fac6333b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac5c15cd50 0x7fac5c1f2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:19.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.845+0000 7fac6333b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c0b48f0 con 0x7fac5c1538f0 2023-10-30T12:40:19.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.845+0000 7fac6333b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c148180 con 0x7fac5c0cefa0 2023-10-30T12:40:19.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.845+0000 7fac6333b640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c1f2780 con 0x7fac5c15cd50 2023-10-30T12:40:19.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.845+0000 7fac61b38640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac5c1538f0 0x7fac5c1e3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:19.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.845+0000 7fac61b38640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac5c1538f0 0x7fac5c1e3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57924/0 (socket says 172.21.15.89:57924) 2023-10-30T12:40:19.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.845+0000 7fac61b38640 1 -- 172.21.15.89:0/742728979 learned_addr learned my addr 172.21.15.89:0/742728979 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:40:19.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.845+0000 7fac62339640 1 --2- 172.21.15.89:0/742728979 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac5c0cefa0 0x7fac5c0b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:19.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.845+0000 7fac62b3a640 1 --2- 172.21.15.89:0/742728979 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac5c15cd50 0x7fac5c1f2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:19.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.845+0000 7fac61b38640 1 -- 172.21.15.89:0/742728979 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac5c15cd50 msgr2=0x7fac5c1f2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:19.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.845+0000 7fac61b38640 1 --2- 172.21.15.89:0/742728979 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac5c15cd50 0x7fac5c1f2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:19.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.845+0000 7fac61b38640 1 -- 172.21.15.89:0/742728979 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac5c0cefa0 msgr2=0x7fac5c0b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:19.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.845+0000 7fac61b38640 1 --2- 172.21.15.89:0/742728979 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac5c0cefa0 0x7fac5c0b43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:19.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.845+0000 7fac61b38640 1 -- 172.21.15.89:0/742728979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac58009420 con 0x7fac5c1538f0 2023-10-30T12:40:19.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.845+0000 7fac61b38640 1 --2- 172.21.15.89:0/742728979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac5c1538f0 0x7fac5c1e3300 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fac4c00d620 tx=0x7fac4c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:19.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.845+0000 7fac52ffd640 1 -- 172.21.15.89:0/742728979 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4c00b620 con 0x7fac5c1538f0 2023-10-30T12:40:19.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.845+0000 7fac52ffd640 1 -- 172.21.15.89:0/742728979 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac4c00b780 con 0x7fac5c1538f0 2023-10-30T12:40:19.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.845+0000 7fac52ffd640 1 -- 172.21.15.89:0/742728979 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4c00bb10 con 0x7fac5c1538f0 2023-10-30T12:40:19.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.845+0000 7fac6333b640 1 -- 172.21.15.89:0/742728979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac5c1483b0 con 0x7fac5c1538f0 2023-10-30T12:40:19.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.845+0000 7fac6333b640 1 -- 172.21.15.89:0/742728979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac5c1e5510 con 0x7fac5c1538f0 2023-10-30T12:40:19.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.845+0000 7fac6333b640 1 -- 172.21.15.89:0/742728979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac5c1f2930 con 0x7fac5c1538f0 2023-10-30T12:40:19.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.849+0000 7fac52ffd640 1 -- 172.21.15.89:0/742728979 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fac4c00bc70 con 0x7fac5c1538f0 2023-10-30T12:40:19.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.849+0000 7fac52ffd640 1 --2- 172.21.15.89:0/742728979 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fac400664d0 0x7fac40068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:19.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.849+0000 7fac52ffd640 1 -- 172.21.15.89:0/742728979 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6234+0+0 (secure 0 0 0) 0x7fac4c084590 con 0x7fac5c1538f0 2023-10-30T12:40:19.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.849+0000 7fac62339640 1 --2- 172.21.15.89:0/742728979 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fac400664d0 0x7fac40068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:19.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.849+0000 7fac62339640 1 --2- 172.21.15.89:0/742728979 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fac400664d0 0x7fac40068990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fac580b2000 tx=0x7fac58008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:19.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:19.853+0000 7fac52ffd640 1 -- 172.21.15.89:0/742728979 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fac5c1f2930 con 0x7fac5c1538f0 2023-10-30T12:40:20.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:20.033+0000 7fac6333b640 1 -- 172.21.15.89:0/742728979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac5c1f28f0 con 0x7fac5c1538f0 2023-10-30T12:40:20.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:20.033+0000 7fac52ffd640 1 -- 172.21.15.89:0/742728979 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+5739 (secure 0 0 0) 0x7fac4c051800 con 0x7fac5c1538f0 2023-10-30T12:40:20.033 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:40:20.034 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:40:16.958000+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:resolve","state_seq":34,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:20.034 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 58 2023-10-30T12:40:20.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:20.033+0000 7fac6333b640 1 -- 172.21.15.89:0/742728979 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fac400664d0 msgr2=0x7fac40068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:20.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:20.033+0000 7fac6333b640 1 --2- 172.21.15.89:0/742728979 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fac400664d0 0x7fac40068990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fac580b2000 tx=0x7fac58008040 comp rx=0 tx=0).stop 2023-10-30T12:40:20.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:20.033+0000 7fac6333b640 1 -- 172.21.15.89:0/742728979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac5c1538f0 msgr2=0x7fac5c1e3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:20.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:20.033+0000 7fac6333b640 1 --2- 172.21.15.89:0/742728979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac5c1538f0 0x7fac5c1e3300 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fac4c00d620 tx=0x7fac4c00dae0 comp rx=0 tx=0).stop 2023-10-30T12:40:20.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:20.033+0000 7fac6333b640 1 -- 172.21.15.89:0/742728979 shutdown_connections 2023-10-30T12:40:20.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:20.033+0000 7fac6333b640 1 --2- 172.21.15.89:0/742728979 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fac400664d0 0x7fac40068990 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:20.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:20.033+0000 7fac6333b640 1 --2- 172.21.15.89:0/742728979 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fac5c15cd50 0x7fac5c1f2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:20.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:20.033+0000 7fac6333b640 1 --2- 172.21.15.89:0/742728979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac5c1538f0 0x7fac5c1e3300 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:20.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:20.033+0000 7fac6333b640 1 --2- 172.21.15.89:0/742728979 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac5c0cefa0 0x7fac5c0b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:20.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:20.033+0000 7fac6333b640 1 -- 172.21.15.89:0/742728979 >> 172.21.15.89:0/742728979 conn(0x7fac5c06a640 msgr2=0x7fac5c1489b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:20.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:20.037+0000 7fac6333b640 1 -- 172.21.15.89:0/742728979 shutdown_connections 2023-10-30T12:40:20.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:20.037+0000 7fac6333b640 1 -- 172.21.15.89:0/742728979 wait complete. 2023-10-30T12:40:20.054 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:40:22.055 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:22.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.205+0000 7f019ce46640 1 Processor -- start 2023-10-30T12:40:22.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.205+0000 7f019ce46640 1 -- start start 2023-10-30T12:40:22.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.205+0000 7f019ce46640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f01980c05d0 0x7f019813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:22.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.205+0000 7f019ce46640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f01980c6f60 0x7f019813f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:22.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.205+0000 7f019ce46640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01980c0a50 con 0x7f01980c6b80 2023-10-30T12:40:22.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.205+0000 7f019ce46640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f019806f6e0 con 0x7f01980c6f60 2023-10-30T12:40:22.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.205+0000 7f019ce46640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f019806f9c0 con 0x7f01980c05d0 2023-10-30T12:40:22.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.205+0000 7f019659c640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f01980c6b80 0x7f01980bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:45890/0 (socket says 172.21.15.89:45890) 2023-10-30T12:40:22.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.205+0000 7f019659c640 1 -- 172.21.15.89:0/176369287 learned_addr learned my addr 172.21.15.89:0/176369287 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:40:22.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.205+0000 7f0196d9d640 1 --2- 172.21.15.89:0/176369287 >> v2:172.21.15.89:3300/0 conn(0x7f01980c6f60 0x7f019813f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:22.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.205+0000 7f018dd9b640 1 --2- 172.21.15.89:0/176369287 >> v2:172.21.15.192:3300/0 conn(0x7f01980c05d0 0x7f019813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:22.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.205+0000 7f0195d9b640 1 -- 172.21.15.89:0/176369287 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3944084624 0 0) 0x7f01980c0a50 con 0x7f01980c6b80 2023-10-30T12:40:22.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.205+0000 7f0195d9b640 1 -- 172.21.15.89:0/176369287 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0184003530 con 0x7f01980c6b80 2023-10-30T12:40:22.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.205+0000 7f0196d9d640 1 -- 172.21.15.89:0/176369287 >> v2:172.21.15.192:3300/0 conn(0x7f01980c05d0 msgr2=0x7f019813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:22.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.205+0000 7f0196d9d640 1 --2- 172.21.15.89:0/176369287 >> v2:172.21.15.192:3300/0 conn(0x7f01980c05d0 0x7f019813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:22.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.205+0000 7f0196d9d640 1 -- 172.21.15.89:0/176369287 >> v1:172.21.15.89:6789/0 conn(0x7f01980c6b80 legacy=0x7f01980bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:22.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.205+0000 7f0196d9d640 1 -- 172.21.15.89:0/176369287 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019806fca0 con 0x7f01980c6f60 2023-10-30T12:40:22.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.205+0000 7f018dd9b640 1 --2- 172.21.15.89:0/176369287 >> v2:172.21.15.192:3300/0 conn(0x7f01980c05d0 0x7f019813ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:40:22.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.209+0000 7f0196d9d640 1 --2- 172.21.15.89:0/176369287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01980c6f60 0x7f019813f2d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f019806c7e0 tx=0x7f019001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5431ad97c23252e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:22.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.209+0000 7f0195d9b640 1 -- 172.21.15.89:0/176369287 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01900ab600 con 0x7f01980c6f60 2023-10-30T12:40:22.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.209+0000 7f0195d9b640 1 -- 172.21.15.89:0/176369287 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01900a7050 con 0x7f01980c6f60 2023-10-30T12:40:22.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.209+0000 7f0195d9b640 1 -- 172.21.15.89:0/176369287 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01900ab980 con 0x7f01980c6f60 2023-10-30T12:40:22.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.209+0000 7f019ce46640 1 -- 172.21.15.89:0/176369287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01980c6f60 msgr2=0x7f019813f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:22.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.209+0000 7f019ce46640 1 --2- 172.21.15.89:0/176369287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01980c6f60 0x7f019813f2d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f019806c7e0 tx=0x7f019001c460 comp rx=0 tx=0).stop 2023-10-30T12:40:22.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.209+0000 7f019ce46640 1 -- 172.21.15.89:0/176369287 shutdown_connections 2023-10-30T12:40:22.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.209+0000 7f019ce46640 1 --2- 172.21.15.89:0/176369287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01980c6f60 0x7f019813f2d0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:22.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.209+0000 7f019ce46640 1 --2- 172.21.15.89:0/176369287 >> v2:172.21.15.192:3300/0 conn(0x7f01980c05d0 0x7f019813ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:22.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.209+0000 7f019ce46640 1 -- 172.21.15.89:0/176369287 >> 172.21.15.89:0/176369287 conn(0x7f01980c7910 msgr2=0x7f01980c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:22.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.209+0000 7f019ce46640 1 -- 172.21.15.89:0/176369287 shutdown_connections 2023-10-30T12:40:22.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.209+0000 7f019ce46640 1 -- 172.21.15.89:0/176369287 wait complete. 2023-10-30T12:40:22.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.213+0000 7f019ce46640 1 Processor -- start 2023-10-30T12:40:22.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.213+0000 7f019ce46640 1 -- start start 2023-10-30T12:40:22.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.213+0000 7f019ce46640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01980c05d0 0x7f0198150e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:22.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.213+0000 7f019ce46640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01980c6b80 0x7f0198158af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:22.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.213+0000 7f019ce46640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01980c6f60 0x7f0198154480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:22.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.213+0000 7f019ce46640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019814eb30 con 0x7f01980c6f60 2023-10-30T12:40:22.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.213+0000 7f019ce46640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019814efc0 con 0x7f01980c6b80 2023-10-30T12:40:22.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.213+0000 7f019ce46640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01981595e0 con 0x7f01980c05d0 2023-10-30T12:40:22.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.213+0000 7f0196d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01980c6f60 0x7f0198154480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:22.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.213+0000 7f0196d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01980c6f60 0x7f0198154480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57946/0 (socket says 172.21.15.89:57946) 2023-10-30T12:40:22.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.213+0000 7f0196d9d640 1 -- 172.21.15.89:0/752625963 learned_addr learned my addr 172.21.15.89:0/752625963 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:40:22.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.213+0000 7f018dd9b640 1 --2- 172.21.15.89:0/752625963 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01980c6b80 0x7f0198158af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:22.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.213+0000 7f019659c640 1 --2- 172.21.15.89:0/752625963 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01980c05d0 0x7f0198150e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:22.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.213+0000 7f0196d9d640 1 -- 172.21.15.89:0/752625963 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01980c05d0 msgr2=0x7f0198150e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:22.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.213+0000 7f0196d9d640 1 --2- 172.21.15.89:0/752625963 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01980c05d0 0x7f0198150e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:22.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.213+0000 7f0196d9d640 1 -- 172.21.15.89:0/752625963 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01980c6b80 msgr2=0x7f0198158af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:22.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.213+0000 7f0196d9d640 1 --2- 172.21.15.89:0/752625963 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01980c6b80 0x7f0198158af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:22.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.213+0000 7f0196d9d640 1 -- 172.21.15.89:0/752625963 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0190009420 con 0x7f01980c6f60 2023-10-30T12:40:22.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.213+0000 7f0196d9d640 1 --2- 172.21.15.89:0/752625963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01980c6f60 0x7f0198154480 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f01900ae040 tx=0x7f0190019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:22.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.213+0000 7f018f7fe640 1 -- 172.21.15.89:0/752625963 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01900b4840 con 0x7f01980c6f60 2023-10-30T12:40:22.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.213+0000 7f018f7fe640 1 -- 172.21.15.89:0/752625963 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01900b49a0 con 0x7f01980c6f60 2023-10-30T12:40:22.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.213+0000 7f019ce46640 1 -- 172.21.15.89:0/752625963 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f019814e4e0 con 0x7f01980c6f60 2023-10-30T12:40:22.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.213+0000 7f018f7fe640 1 -- 172.21.15.89:0/752625963 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01900b4d30 con 0x7f01980c6f60 2023-10-30T12:40:22.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.213+0000 7f019ce46640 1 -- 172.21.15.89:0/752625963 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01981598c0 con 0x7f01980c6f60 2023-10-30T12:40:22.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.217+0000 7f019ce46640 1 -- 172.21.15.89:0/752625963 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0198071de0 con 0x7f01980c6f60 2023-10-30T12:40:22.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.217+0000 7f018f7fe640 1 -- 172.21.15.89:0/752625963 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f01900b2040 con 0x7f01980c6f60 2023-10-30T12:40:22.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.217+0000 7f018f7fe640 1 --2- 172.21.15.89:0/752625963 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f01780664a0 0x7f0178068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:22.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.217+0000 7f018f7fe640 1 -- 172.21.15.89:0/752625963 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6234+0+0 (secure 0 0 0) 0x7f019011e400 con 0x7f01980c6f60 2023-10-30T12:40:22.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.217+0000 7f019659c640 1 --2- 172.21.15.89:0/752625963 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f01780664a0 0x7f0178068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:22.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.217+0000 7f019659c640 1 --2- 172.21.15.89:0/752625963 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f01780664a0 0x7f0178068960 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f01880011f0 tx=0x7f0188009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:22.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.221+0000 7f018f7fe640 1 -- 172.21.15.89:0/752625963 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f01900eb5d0 con 0x7f01980c6f60 2023-10-30T12:40:22.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.405+0000 7f019ce46640 1 -- 172.21.15.89:0/752625963 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f019804e170 con 0x7f01980c6f60 2023-10-30T12:40:22.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.405+0000 7f018f7fe640 1 -- 172.21.15.89:0/752625963 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f01900b0090 con 0x7f01980c6f60 2023-10-30T12:40:22.406 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:40:22.407 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:40:16.958000+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:resolve","state_seq":34,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:22.407 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 58 2023-10-30T12:40:22.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.409+0000 7f019ce46640 1 -- 172.21.15.89:0/752625963 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f01780664a0 msgr2=0x7f0178068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:22.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.409+0000 7f019ce46640 1 --2- 172.21.15.89:0/752625963 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f01780664a0 0x7f0178068960 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f01880011f0 tx=0x7f0188009040 comp rx=0 tx=0).stop 2023-10-30T12:40:22.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.409+0000 7f019ce46640 1 -- 172.21.15.89:0/752625963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01980c6f60 msgr2=0x7f0198154480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:22.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.409+0000 7f019ce46640 1 --2- 172.21.15.89:0/752625963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01980c6f60 0x7f0198154480 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f01900ae040 tx=0x7f0190019200 comp rx=0 tx=0).stop 2023-10-30T12:40:22.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.409+0000 7f019ce46640 1 -- 172.21.15.89:0/752625963 shutdown_connections 2023-10-30T12:40:22.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.409+0000 7f019ce46640 1 --2- 172.21.15.89:0/752625963 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f01780664a0 0x7f0178068960 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:22.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.409+0000 7f019ce46640 1 --2- 172.21.15.89:0/752625963 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01980c6b80 0x7f0198158af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:22.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.409+0000 7f019ce46640 1 --2- 172.21.15.89:0/752625963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01980c6f60 0x7f0198154480 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:22.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.409+0000 7f019ce46640 1 --2- 172.21.15.89:0/752625963 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01980c05d0 0x7f0198150e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:22.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.409+0000 7f019ce46640 1 -- 172.21.15.89:0/752625963 >> 172.21.15.89:0/752625963 conn(0x7f01980c7910 msgr2=0x7f01980bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:22.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.409+0000 7f019ce46640 1 -- 172.21.15.89:0/752625963 shutdown_connections 2023-10-30T12:40:22.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:22.409+0000 7f019ce46640 1 -- 172.21.15.89:0/752625963 wait complete. 2023-10-30T12:40:22.427 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:40:24.428 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:24.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.577+0000 7f4652609640 1 Processor -- start 2023-10-30T12:40:24.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.577+0000 7f4652609640 1 -- start start 2023-10-30T12:40:24.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.577+0000 7f4652609640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f464c1538a0 0x7f464c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:24.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.577+0000 7f4652609640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f464c0cf160 0x7f464c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:24.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.577+0000 7f4652609640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f464c0cb800 con 0x7f464c0ca0e0 2023-10-30T12:40:24.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.577+0000 7f4652609640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f464c15dc30 con 0x7f464c1538a0 2023-10-30T12:40:24.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.577+0000 7f4652609640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f464c0cb330 con 0x7f464c0cf160 2023-10-30T12:40:24.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.577+0000 7f464bfff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f464c1538a0 0x7f464c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:24.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.577+0000 7f464bfff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f464c1538a0 0x7f464c15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57950/0 (socket says 172.21.15.89:57950) 2023-10-30T12:40:24.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.577+0000 7f464bfff640 1 -- 172.21.15.89:0/4039306648 learned_addr learned my addr 172.21.15.89:0/4039306648 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:40:24.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.577+0000 7f4650ba6640 1 --2- 172.21.15.89:0/4039306648 >> v2:172.21.15.192:3300/0 conn(0x7f464c0cf160 0x7f464c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:24.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.577+0000 7f464bfff640 1 -- 172.21.15.89:0/4039306648 >> v2:172.21.15.192:3300/0 conn(0x7f464c0cf160 msgr2=0x7f464c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:24.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.577+0000 7f464bfff640 1 --2- 172.21.15.89:0/4039306648 >> v2:172.21.15.192:3300/0 conn(0x7f464c0cf160 0x7f464c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:24.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.577+0000 7f464bfff640 1 -- 172.21.15.89:0/4039306648 >> v1:172.21.15.192:6789/0 conn(0x7f464c0ca0e0 legacy=0x7f464c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:24.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.577+0000 7f464bfff640 1 -- 172.21.15.89:0/4039306648 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464c0cad70 con 0x7f464c1538a0 2023-10-30T12:40:24.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.581+0000 7f464bfff640 1 --2- 172.21.15.89:0/4039306648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f464c1538a0 0x7f464c15d6f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f464c0b5330 tx=0x7f463c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15f4a41a88b15755 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:24.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.581+0000 7f464affd640 1 -- 172.21.15.89:0/4039306648 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c0a89f0 con 0x7f464c1538a0 2023-10-30T12:40:24.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.581+0000 7f464affd640 1 -- 172.21.15.89:0/4039306648 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f463c0a8b50 con 0x7f464c1538a0 2023-10-30T12:40:24.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.581+0000 7f464affd640 1 -- 172.21.15.89:0/4039306648 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c0b0440 con 0x7f464c1538a0 2023-10-30T12:40:24.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.581+0000 7f4652609640 1 -- 172.21.15.89:0/4039306648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f464c1538a0 msgr2=0x7f464c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:24.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.581+0000 7f4652609640 1 --2- 172.21.15.89:0/4039306648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f464c1538a0 0x7f464c15d6f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f464c0b5330 tx=0x7f463c01ecf0 comp rx=0 tx=0).stop 2023-10-30T12:40:24.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.581+0000 7f4652609640 1 -- 172.21.15.89:0/4039306648 shutdown_connections 2023-10-30T12:40:24.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.581+0000 7f4652609640 1 --2- 172.21.15.89:0/4039306648 >> v2:172.21.15.192:3300/0 conn(0x7f464c0cf160 0x7f464c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:24.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.581+0000 7f4652609640 1 --2- 172.21.15.89:0/4039306648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f464c1538a0 0x7f464c15d6f0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:24.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.581+0000 7f4652609640 1 -- 172.21.15.89:0/4039306648 >> 172.21.15.89:0/4039306648 conn(0x7f464c06bfe0 msgr2=0x7f464c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:24.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.581+0000 7f4652609640 1 -- 172.21.15.89:0/4039306648 shutdown_connections 2023-10-30T12:40:24.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.581+0000 7f4652609640 1 -- 172.21.15.89:0/4039306648 wait complete. 2023-10-30T12:40:24.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.585+0000 7f4652609640 1 Processor -- start 2023-10-30T12:40:24.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.585+0000 7f4652609640 1 -- start start 2023-10-30T12:40:24.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.585+0000 7f4652609640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f464c0ca0e0 0x7f464c158840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:24.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.585+0000 7f4652609640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f464c0cf160 0x7f464c0ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:24.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.585+0000 7f4652609640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f464c1538a0 0x7f464c154ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:24.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.585+0000 7f4652609640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464c1e3100 con 0x7f464c0ca0e0 2023-10-30T12:40:24.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.585+0000 7f4652609640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464c1596d0 con 0x7f464c1538a0 2023-10-30T12:40:24.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.585+0000 7f4652609640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f464c1e33e0 con 0x7f464c0cf160 2023-10-30T12:40:24.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.585+0000 7f464bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f464c0ca0e0 0x7f464c158840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:24.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.585+0000 7f464bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f464c0ca0e0 0x7f464c158840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57962/0 (socket says 172.21.15.89:57962) 2023-10-30T12:40:24.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.585+0000 7f464bfff640 1 -- 172.21.15.89:0/3243326572 learned_addr learned my addr 172.21.15.89:0/3243326572 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:40:24.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.585+0000 7f464b7fe640 1 --2- 172.21.15.89:0/3243326572 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f464c0cf160 0x7f464c0ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.585+0000 7f4650ba6640 1 --2- 172.21.15.89:0/3243326572 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f464c1538a0 0x7f464c154ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.585+0000 7f464bfff640 1 -- 172.21.15.89:0/3243326572 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f464c0cf160 msgr2=0x7f464c0ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.585+0000 7f464bfff640 1 --2- 172.21.15.89:0/3243326572 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f464c0cf160 0x7f464c0ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.585+0000 7f464bfff640 1 -- 172.21.15.89:0/3243326572 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f464c1538a0 msgr2=0x7f464c154ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.585+0000 7f464bfff640 1 --2- 172.21.15.89:0/3243326572 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f464c1538a0 0x7f464c154ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.585+0000 7f464bfff640 1 -- 172.21.15.89:0/3243326572 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f463c0094d0 con 0x7f464c0ca0e0 2023-10-30T12:40:24.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.585+0000 7f464bfff640 1 --2- 172.21.15.89:0/3243326572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f464c0ca0e0 0x7f464c158840 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f463c01b900 tx=0x7f463c01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:24.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.585+0000 7f4648ff9640 1 -- 172.21.15.89:0/3243326572 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c0ba600 con 0x7f464c0ca0e0 2023-10-30T12:40:24.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.585+0000 7f4648ff9640 1 -- 172.21.15.89:0/3243326572 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f463c0ba760 con 0x7f464c0ca0e0 2023-10-30T12:40:24.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.585+0000 7f4648ff9640 1 -- 172.21.15.89:0/3243326572 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c0bab10 con 0x7f464c0ca0e0 2023-10-30T12:40:24.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.585+0000 7f4652609640 1 -- 172.21.15.89:0/3243326572 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f464c1e51b0 con 0x7f464c0ca0e0 2023-10-30T12:40:24.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.589+0000 7f4652609640 1 -- 172.21.15.89:0/3243326572 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f464c1e2d90 con 0x7f464c0ca0e0 2023-10-30T12:40:24.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.589+0000 7f4652609640 1 -- 172.21.15.89:0/3243326572 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f464c0b2a50 con 0x7f464c0ca0e0 2023-10-30T12:40:24.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.589+0000 7f4648ff9640 1 -- 172.21.15.89:0/3243326572 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f463c0bac70 con 0x7f464c0ca0e0 2023-10-30T12:40:24.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.589+0000 7f4648ff9640 1 --2- 172.21.15.89:0/3243326572 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f46380664a0 0x7f4638068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:24.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.589+0000 7f4648ff9640 1 -- 172.21.15.89:0/3243326572 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6234+0+0 (secure 0 0 0) 0x7f463c1226d0 con 0x7f464c0ca0e0 2023-10-30T12:40:24.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.589+0000 7f464b7fe640 1 --2- 172.21.15.89:0/3243326572 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f46380664a0 0x7f4638068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:24.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.593+0000 7f464b7fe640 1 --2- 172.21.15.89:0/3243326572 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f46380664a0 0x7f4638068960 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f464c0c97b0 tx=0x7f4640009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:24.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.593+0000 7f4648ff9640 1 -- 172.21.15.89:0/3243326572 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f463c0ef810 con 0x7f464c0ca0e0 2023-10-30T12:40:24.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.777+0000 7f4652609640 1 -- 172.21.15.89:0/3243326572 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f464c04e170 con 0x7f464c0ca0e0 2023-10-30T12:40:24.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.777+0000 7f4648ff9640 1 -- 172.21.15.89:0/3243326572 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f463c0f2e30 con 0x7f464c0ca0e0 2023-10-30T12:40:24.777 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:40:24.777 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:40:22.393022+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:24.778 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 59 2023-10-30T12:40:24.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.777+0000 7f4652609640 1 -- 172.21.15.89:0/3243326572 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f46380664a0 msgr2=0x7f4638068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:24.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.777+0000 7f4652609640 1 --2- 172.21.15.89:0/3243326572 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f46380664a0 0x7f4638068960 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f464c0c97b0 tx=0x7f4640009040 comp rx=0 tx=0).stop 2023-10-30T12:40:24.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.777+0000 7f4652609640 1 -- 172.21.15.89:0/3243326572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f464c0ca0e0 msgr2=0x7f464c158840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:24.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.777+0000 7f4652609640 1 --2- 172.21.15.89:0/3243326572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f464c0ca0e0 0x7f464c158840 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f463c01b900 tx=0x7f463c01ba90 comp rx=0 tx=0).stop 2023-10-30T12:40:24.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.777+0000 7f4652609640 1 -- 172.21.15.89:0/3243326572 shutdown_connections 2023-10-30T12:40:24.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.777+0000 7f4652609640 1 --2- 172.21.15.89:0/3243326572 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f46380664a0 0x7f4638068960 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:24.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.777+0000 7f4652609640 1 --2- 172.21.15.89:0/3243326572 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f464c1538a0 0x7f464c154ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:24.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.777+0000 7f4652609640 1 --2- 172.21.15.89:0/3243326572 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f464c0cf160 0x7f464c0ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:24.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.777+0000 7f4652609640 1 --2- 172.21.15.89:0/3243326572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f464c0ca0e0 0x7f464c158840 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:24.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.777+0000 7f4652609640 1 -- 172.21.15.89:0/3243326572 >> 172.21.15.89:0/3243326572 conn(0x7f464c06bfe0 msgr2=0x7f464c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:24.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.781+0000 7f4652609640 1 -- 172.21.15.89:0/3243326572 shutdown_connections 2023-10-30T12:40:24.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:24.781+0000 7f4652609640 1 -- 172.21.15.89:0/3243326572 wait complete. 2023-10-30T12:40:24.799 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 59, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:50.729245+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 59, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5018": { "addr": "172.21.15.192:6839/93914587", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6838", "nonce": 93914587, "type": "v2" }, { "addr": "172.21.15.192:6839", "nonce": 93914587, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5018, "incarnation": 44, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 41 }, "gid_5129": { "addr": "172.21.15.192:6837/1413360446", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6836", "nonce": 1413360446, "type": "v2" }, { "addr": "172.21.15.192:6837", "nonce": 1413360446, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5129, "incarnation": 57, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:reconnect", "state_seq": 36 }, "gid_5139": { "addr": "172.21.15.89:6837/951276252", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6836", "nonce": 951276252, "type": "v2" }, { "addr": "172.21.15.89:6837", "nonce": 951276252, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5139, "incarnation": 33, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 29 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:40:22.393022+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5018, "mds_1": 5139, "mds_2": 5129 } } } ], "standbys": [ { "addr": "172.21.15.89:6834/2909790173", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6832", "nonce": 2909790173, "type": "v2" }, { "addr": "172.21.15.89:6834", "nonce": 2909790173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.89:6835/2387357615", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6833", "nonce": 2387357615, "type": "v2" }, { "addr": "172.21.15.89:6835", "nonce": 2387357615, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4239, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:40:26.800 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:26.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.949+0000 7f97adeea640 1 Processor -- start 2023-10-30T12:40:26.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.949+0000 7f97adeea640 1 -- start start 2023-10-30T12:40:26.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.949+0000 7f97adeea640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f97a80c05d0 0x7f97a813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:26.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.949+0000 7f97adeea640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f97a80c6f60 0x7f97a813f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:26.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.949+0000 7f97adeea640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97a80c0a50 con 0x7f97a80c6b80 2023-10-30T12:40:26.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.949+0000 7f97adeea640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97a806f6e0 con 0x7f97a80c05d0 2023-10-30T12:40:26.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.949+0000 7f97adeea640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97a806f9c0 con 0x7f97a80c6f60 2023-10-30T12:40:26.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.953+0000 7f97a6ffd640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f97a80c05d0 0x7f97a813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:26.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.953+0000 7f97a6ffd640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f97a80c05d0 0x7f97a813ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57976/0 (socket says 172.21.15.89:57976) 2023-10-30T12:40:26.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.953+0000 7f97a6ffd640 1 -- 172.21.15.89:0/3746325382 learned_addr learned my addr 172.21.15.89:0/3746325382 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:40:26.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.953+0000 7f97a7fff640 1 --2- 172.21.15.89:0/3746325382 >> v2:172.21.15.192:3300/0 conn(0x7f97a80c6f60 0x7f97a813f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:26.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.953+0000 7f97a6ffd640 1 -- 172.21.15.89:0/3746325382 >> v2:172.21.15.192:3300/0 conn(0x7f97a80c6f60 msgr2=0x7f97a813f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:26.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.953+0000 7f97a6ffd640 1 --2- 172.21.15.89:0/3746325382 >> v2:172.21.15.192:3300/0 conn(0x7f97a80c6f60 0x7f97a813f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:26.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.953+0000 7f97a67fc640 1 -- 172.21.15.89:0/3746325382 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2232430741 0 0) 0x7f97a80c0a50 con 0x7f97a80c6b80 2023-10-30T12:40:26.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.953+0000 7f97a6ffd640 1 -- 172.21.15.89:0/3746325382 >> v1:172.21.15.89:6789/0 conn(0x7f97a80c6b80 legacy=0x7f97a80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:26.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.953+0000 7f97a6ffd640 1 -- 172.21.15.89:0/3746325382 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a806fca0 con 0x7f97a80c05d0 2023-10-30T12:40:26.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.953+0000 7f97a6ffd640 1 --2- 172.21.15.89:0/3746325382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97a80c05d0 0x7f97a813ed90 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f97a80c89b0 tx=0x7f979c01c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72c72f581acf756a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:26.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.957+0000 7f97a67fc640 1 -- 172.21.15.89:0/3746325382 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979c0a5990 con 0x7f97a80c05d0 2023-10-30T12:40:26.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.957+0000 7f97a67fc640 1 -- 172.21.15.89:0/3746325382 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f979c0a5af0 con 0x7f97a80c05d0 2023-10-30T12:40:26.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.957+0000 7f97a67fc640 1 -- 172.21.15.89:0/3746325382 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979c0a5e80 con 0x7f97a80c05d0 2023-10-30T12:40:26.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.957+0000 7f97adeea640 1 -- 172.21.15.89:0/3746325382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97a80c05d0 msgr2=0x7f97a813ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:26.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.957+0000 7f97adeea640 1 --2- 172.21.15.89:0/3746325382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97a80c05d0 0x7f97a813ed90 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f97a80c89b0 tx=0x7f979c01c9c0 comp rx=0 tx=0).stop 2023-10-30T12:40:26.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.957+0000 7f97adeea640 1 -- 172.21.15.89:0/3746325382 shutdown_connections 2023-10-30T12:40:26.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.957+0000 7f97adeea640 1 --2- 172.21.15.89:0/3746325382 >> v2:172.21.15.192:3300/0 conn(0x7f97a80c6f60 0x7f97a813f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:26.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.957+0000 7f97adeea640 1 --2- 172.21.15.89:0/3746325382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97a80c05d0 0x7f97a813ed90 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:26.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.957+0000 7f97adeea640 1 -- 172.21.15.89:0/3746325382 >> 172.21.15.89:0/3746325382 conn(0x7f97a80c7910 msgr2=0x7f97a80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:26.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.957+0000 7f97adeea640 1 -- 172.21.15.89:0/3746325382 shutdown_connections 2023-10-30T12:40:26.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.957+0000 7f97adeea640 1 -- 172.21.15.89:0/3746325382 wait complete. 2023-10-30T12:40:26.960 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.957+0000 7f97adeea640 1 Processor -- start 2023-10-30T12:40:26.960 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.957+0000 7f97adeea640 1 -- start start 2023-10-30T12:40:26.960 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.957+0000 7f97adeea640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97a80c05d0 0x7f97a8153910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:26.960 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.957+0000 7f97adeea640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97a80c6b80 0x7f97a81544a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:26.960 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.961+0000 7f97adeea640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97a80c6f60 0x7f97a80c9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:26.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.961+0000 7f97adeea640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a8134380 con 0x7f97a80c05d0 2023-10-30T12:40:26.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.961+0000 7f97adeea640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a80c8c10 con 0x7f97a80c6b80 2023-10-30T12:40:26.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.961+0000 7f97adeea640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97a8156230 con 0x7f97a80c6f60 2023-10-30T12:40:26.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.961+0000 7f97a77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97a80c05d0 0x7f97a8153910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:26.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.961+0000 7f97a77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97a80c05d0 0x7f97a8153910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57980/0 (socket says 172.21.15.89:57980) 2023-10-30T12:40:26.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.961+0000 7f97a77fe640 1 -- 172.21.15.89:0/2247824760 learned_addr learned my addr 172.21.15.89:0/2247824760 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:40:26.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.961+0000 7f97a6ffd640 1 --2- 172.21.15.89:0/2247824760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97a80c6b80 0x7f97a81544a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:26.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.961+0000 7f97a7fff640 1 --2- 172.21.15.89:0/2247824760 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97a80c6f60 0x7f97a80c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:26.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.961+0000 7f97a77fe640 1 -- 172.21.15.89:0/2247824760 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97a80c6f60 msgr2=0x7f97a80c9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:26.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.961+0000 7f97a77fe640 1 --2- 172.21.15.89:0/2247824760 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97a80c6f60 0x7f97a80c9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:26.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.961+0000 7f97a77fe640 1 -- 172.21.15.89:0/2247824760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97a80c6b80 msgr2=0x7f97a81544a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:26.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.961+0000 7f97a77fe640 1 --2- 172.21.15.89:0/2247824760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97a80c6b80 0x7f97a81544a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:26.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.961+0000 7f97a77fe640 1 -- 172.21.15.89:0/2247824760 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f979c00b580 con 0x7f97a80c05d0 2023-10-30T12:40:26.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.961+0000 7f97a77fe640 1 --2- 172.21.15.89:0/2247824760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97a80c05d0 0x7f97a8153910 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f979800c4d0 tx=0x7f979800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:26.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.961+0000 7f978ffff640 1 -- 172.21.15.89:0/2247824760 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9798005480 con 0x7f97a80c05d0 2023-10-30T12:40:26.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.961+0000 7f978ffff640 1 -- 172.21.15.89:0/2247824760 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9798003600 con 0x7f97a80c05d0 2023-10-30T12:40:26.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.961+0000 7f97adeea640 1 -- 172.21.15.89:0/2247824760 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97a81597f0 con 0x7f97a80c05d0 2023-10-30T12:40:26.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.961+0000 7f978ffff640 1 -- 172.21.15.89:0/2247824760 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97980039b0 con 0x7f97a80c05d0 2023-10-30T12:40:26.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.961+0000 7f97adeea640 1 -- 172.21.15.89:0/2247824760 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97a8134570 con 0x7f97a80c05d0 2023-10-30T12:40:26.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.961+0000 7f97adeea640 1 -- 172.21.15.89:0/2247824760 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97a806c4d0 con 0x7f97a80c05d0 2023-10-30T12:40:26.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.961+0000 7f978ffff640 1 -- 172.21.15.89:0/2247824760 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9798003b10 con 0x7f97a80c05d0 2023-10-30T12:40:26.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.965+0000 7f978ffff640 1 --2- 172.21.15.89:0/2247824760 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f97940664a0 0x7f9794068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:26.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.965+0000 7f978ffff640 1 -- 172.21.15.89:0/2247824760 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6234+0+0 (secure 0 0 0) 0x7f97980836c0 con 0x7f97a80c05d0 2023-10-30T12:40:26.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.965+0000 7f97a6ffd640 1 --2- 172.21.15.89:0/2247824760 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f97940664a0 0x7f9794068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:26.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.965+0000 7f97a6ffd640 1 --2- 172.21.15.89:0/2247824760 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f97940664a0 0x7f9794068960 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f979c0af000 tx=0x7f979c0a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:26.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:26.969+0000 7f978ffff640 1 -- 172.21.15.89:0/2247824760 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f97980509f0 con 0x7f97a80c05d0 2023-10-30T12:40:27.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:27.149+0000 7f97adeea640 1 -- 172.21.15.89:0/2247824760 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97a81563a0 con 0x7f97a80c05d0 2023-10-30T12:40:27.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:27.149+0000 7f978ffff640 1 -- 172.21.15.89:0/2247824760 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9798017090 con 0x7f97a80c05d0 2023-10-30T12:40:27.149 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:40:27.149 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:40:22.393022+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:27.149 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 59 2023-10-30T12:40:27.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:27.149+0000 7f97adeea640 1 -- 172.21.15.89:0/2247824760 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f97940664a0 msgr2=0x7f9794068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:27.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:27.149+0000 7f97adeea640 1 --2- 172.21.15.89:0/2247824760 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f97940664a0 0x7f9794068960 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f979c0af000 tx=0x7f979c0a7070 comp rx=0 tx=0).stop 2023-10-30T12:40:27.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:27.149+0000 7f97adeea640 1 -- 172.21.15.89:0/2247824760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97a80c05d0 msgr2=0x7f97a8153910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:27.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:27.149+0000 7f97adeea640 1 --2- 172.21.15.89:0/2247824760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97a80c05d0 0x7f97a8153910 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f979800c4d0 tx=0x7f979800c990 comp rx=0 tx=0).stop 2023-10-30T12:40:27.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:27.149+0000 7f97adeea640 1 -- 172.21.15.89:0/2247824760 shutdown_connections 2023-10-30T12:40:27.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:27.149+0000 7f97adeea640 1 --2- 172.21.15.89:0/2247824760 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f97940664a0 0x7f9794068960 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:27.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:27.149+0000 7f97adeea640 1 --2- 172.21.15.89:0/2247824760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97a80c6b80 0x7f97a81544a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:27.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:27.149+0000 7f97adeea640 1 --2- 172.21.15.89:0/2247824760 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97a80c6f60 0x7f97a80c9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:27.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:27.149+0000 7f97adeea640 1 --2- 172.21.15.89:0/2247824760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97a80c05d0 0x7f97a8153910 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:27.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:27.149+0000 7f97adeea640 1 -- 172.21.15.89:0/2247824760 >> 172.21.15.89:0/2247824760 conn(0x7f97a80c7910 msgr2=0x7f97a80bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:27.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:27.149+0000 7f97adeea640 1 -- 172.21.15.89:0/2247824760 shutdown_connections 2023-10-30T12:40:27.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:27.149+0000 7f97adeea640 1 -- 172.21.15.89:0/2247824760 wait complete. 2023-10-30T12:40:27.169 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:40:29.169 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:29.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.309+0000 7f3f1d0c9640 1 Processor -- start 2023-10-30T12:40:29.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.309+0000 7f3f1d0c9640 1 -- start start 2023-10-30T12:40:29.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.309+0000 7f3f1d0c9640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3f180c6b80 0x7f3f180bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:29.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.309+0000 7f3f1d0c9640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3f180c6f60 0x7f3f180c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:29.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.309+0000 7f3f1d0c9640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f180c0a50 con 0x7f3f180c8430 2023-10-30T12:40:29.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.309+0000 7f3f1d0c9640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f1806f6e0 con 0x7f3f180c6f60 2023-10-30T12:40:29.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.309+0000 7f3f1d0c9640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f1806f9c0 con 0x7f3f180c6b80 2023-10-30T12:40:29.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.313+0000 7f3f0ffff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3f180c6f60 0x7f3f180c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:29.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.313+0000 7f3f0ffff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3f180c6f60 0x7f3f180c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57988/0 (socket says 172.21.15.89:57988) 2023-10-30T12:40:29.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.313+0000 7f3f0ffff640 1 -- 172.21.15.89:0/1866316987 learned_addr learned my addr 172.21.15.89:0/1866316987 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:40:29.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.313+0000 7f3f1759e640 1 --1- 172.21.15.89:0/1866316987 >> v1:172.21.15.89:6789/0 conn(0x7f3f180c8430 0x7f3f1813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:45918/0 (socket says 172.21.15.89:45918) 2023-10-30T12:40:29.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.313+0000 7f3f16d9d640 1 --2- 172.21.15.89:0/1866316987 >> v2:172.21.15.192:3300/0 conn(0x7f3f180c6b80 0x7f3f180bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:29.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.313+0000 7f3f0ffff640 1 -- 172.21.15.89:0/1866316987 >> v2:172.21.15.192:3300/0 conn(0x7f3f180c6b80 msgr2=0x7f3f180bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:29.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.313+0000 7f3f0ffff640 1 --2- 172.21.15.89:0/1866316987 >> v2:172.21.15.192:3300/0 conn(0x7f3f180c6b80 0x7f3f180bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:29.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.313+0000 7f3f0ffff640 1 -- 172.21.15.89:0/1866316987 >> v1:172.21.15.89:6789/0 conn(0x7f3f180c8430 legacy=0x7f3f1813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:29.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.313+0000 7f3f1659c640 1 -- 172.21.15.89:0/1866316987 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3227853390 0 0) 0x7f3f180c0a50 con 0x7f3f180c8430 2023-10-30T12:40:29.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.313+0000 7f3f0ffff640 1 -- 172.21.15.89:0/1866316987 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f1806fca0 con 0x7f3f180c6f60 2023-10-30T12:40:29.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.313+0000 7f3f16d9d640 1 --2- 172.21.15.89:0/1866316987 >> v2:172.21.15.192:3300/0 conn(0x7f3f180c6b80 0x7f3f180bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:40:29.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.313+0000 7f3f0ffff640 1 --2- 172.21.15.89:0/1866316987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f180c6f60 0x7f3f180c0400 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3f1806f1b0 tx=0x7f3f0001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dea7cb27397964cc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:29.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.313+0000 7f3f1659c640 1 -- 172.21.15.89:0/1866316987 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f000ab610 con 0x7f3f180c6f60 2023-10-30T12:40:29.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.313+0000 7f3f1659c640 1 -- 172.21.15.89:0/1866316987 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f000ab770 con 0x7f3f180c6f60 2023-10-30T12:40:29.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.313+0000 7f3f1659c640 1 -- 172.21.15.89:0/1866316987 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f000abb00 con 0x7f3f180c6f60 2023-10-30T12:40:29.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.317+0000 7f3f1d0c9640 1 -- 172.21.15.89:0/1866316987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f180c6f60 msgr2=0x7f3f180c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:29.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.317+0000 7f3f1d0c9640 1 --2- 172.21.15.89:0/1866316987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f180c6f60 0x7f3f180c0400 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3f1806f1b0 tx=0x7f3f0001c500 comp rx=0 tx=0).stop 2023-10-30T12:40:29.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.317+0000 7f3f1d0c9640 1 -- 172.21.15.89:0/1866316987 shutdown_connections 2023-10-30T12:40:29.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.317+0000 7f3f1d0c9640 1 --2- 172.21.15.89:0/1866316987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f180c6f60 0x7f3f180c0400 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:29.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.317+0000 7f3f1d0c9640 1 --2- 172.21.15.89:0/1866316987 >> v2:172.21.15.192:3300/0 conn(0x7f3f180c6b80 0x7f3f180bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:29.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.317+0000 7f3f1d0c9640 1 -- 172.21.15.89:0/1866316987 >> 172.21.15.89:0/1866316987 conn(0x7f3f180c7910 msgr2=0x7f3f180c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:29.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.317+0000 7f3f1d0c9640 1 -- 172.21.15.89:0/1866316987 shutdown_connections 2023-10-30T12:40:29.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.317+0000 7f3f1d0c9640 1 -- 172.21.15.89:0/1866316987 wait complete. 2023-10-30T12:40:29.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.317+0000 7f3f1d0c9640 1 Processor -- start 2023-10-30T12:40:29.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.317+0000 7f3f1d0c9640 1 -- start start 2023-10-30T12:40:29.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.317+0000 7f3f1d0c9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f180c6b80 0x7f3f18155a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:29.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.317+0000 7f3f1d0c9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f180c6f60 0x7f3f1815eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:29.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.317+0000 7f3f1d0c9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f180c8430 0x7f3f1815f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:29.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.317+0000 7f3f1d0c9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f18139980 con 0x7f3f180c6f60 2023-10-30T12:40:29.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.317+0000 7f3f1d0c9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f18138e00 con 0x7f3f180c6b80 2023-10-30T12:40:29.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.317+0000 7f3f1d0c9640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f181390e0 con 0x7f3f180c8430 2023-10-30T12:40:29.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.317+0000 7f3f0ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f180c6f60 0x7f3f1815eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:29.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.317+0000 7f3f0ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f180c6f60 0x7f3f1815eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57998/0 (socket says 172.21.15.89:57998) 2023-10-30T12:40:29.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.317+0000 7f3f0ffff640 1 -- 172.21.15.89:0/1506941093 learned_addr learned my addr 172.21.15.89:0/1506941093 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:40:29.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.317+0000 7f3f16d9d640 1 --2- 172.21.15.89:0/1506941093 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f180c6b80 0x7f3f18155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:29.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.317+0000 7f3f1759e640 1 --2- 172.21.15.89:0/1506941093 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f180c8430 0x7f3f1815f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:29.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.317+0000 7f3f0ffff640 1 -- 172.21.15.89:0/1506941093 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f180c8430 msgr2=0x7f3f1815f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:29.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.317+0000 7f3f0ffff640 1 --2- 172.21.15.89:0/1506941093 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f180c8430 0x7f3f1815f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:29.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.317+0000 7f3f0ffff640 1 -- 172.21.15.89:0/1506941093 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f180c6b80 msgr2=0x7f3f18155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:29.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.317+0000 7f3f0ffff640 1 --2- 172.21.15.89:0/1506941093 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f180c6b80 0x7f3f18155a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:29.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.317+0000 7f3f0ffff640 1 -- 172.21.15.89:0/1506941093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f000094d0 con 0x7f3f180c6f60 2023-10-30T12:40:29.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.321+0000 7f3f0ffff640 1 --2- 172.21.15.89:0/1506941093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f180c6f60 0x7f3f1815eae0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f3f00019110 tx=0x7f3f0001ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:29.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.321+0000 7f3f0f7fe640 1 -- 172.21.15.89:0/1506941093 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f000b4800 con 0x7f3f180c6f60 2023-10-30T12:40:29.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.321+0000 7f3f0f7fe640 1 -- 172.21.15.89:0/1506941093 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f000b4960 con 0x7f3f180c6f60 2023-10-30T12:40:29.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.321+0000 7f3f1d0c9640 1 -- 172.21.15.89:0/1506941093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f181393c0 con 0x7f3f180c6f60 2023-10-30T12:40:29.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.321+0000 7f3f0f7fe640 1 -- 172.21.15.89:0/1506941093 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f000b4d10 con 0x7f3f180c6f60 2023-10-30T12:40:29.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.321+0000 7f3f1d0c9640 1 -- 172.21.15.89:0/1506941093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f1815e5c0 con 0x7f3f180c6f60 2023-10-30T12:40:29.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.321+0000 7f3f1d0c9640 1 -- 172.21.15.89:0/1506941093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f1815e7b0 con 0x7f3f180c6f60 2023-10-30T12:40:29.326 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.321+0000 7f3f0f7fe640 1 -- 172.21.15.89:0/1506941093 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3f000b2410 con 0x7f3f180c6f60 2023-10-30T12:40:29.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.321+0000 7f3f0f7fe640 1 --2- 172.21.15.89:0/1506941093 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3ef80664a0 0x7f3ef8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:29.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.321+0000 7f3f0f7fe640 1 -- 172.21.15.89:0/1506941093 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6234+0+0 (secure 0 0 0) 0x7f3f0011e8e0 con 0x7f3f180c6f60 2023-10-30T12:40:29.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.321+0000 7f3f16d9d640 1 --2- 172.21.15.89:0/1506941093 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3ef80664a0 0x7f3ef8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:29.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.325+0000 7f3f16d9d640 1 --2- 172.21.15.89:0/1506941093 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3ef80664a0 0x7f3ef8068960 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3f0800c900 tx=0x7f3f08008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:29.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.325+0000 7f3f0f7fe640 1 -- 172.21.15.89:0/1506941093 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3f000ebb50 con 0x7f3f180c6f60 2023-10-30T12:40:29.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.505+0000 7f3f1d0c9640 1 -- 172.21.15.89:0/1506941093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f1804e170 con 0x7f3f180c6f60 2023-10-30T12:40:29.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.509+0000 7f3f0f7fe640 1 -- 172.21.15.89:0/1506941093 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f3f000c4090 con 0x7f3f180c6f60 2023-10-30T12:40:29.508 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:40:29.508 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:40:28.147139+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:29.509 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 60 2023-10-30T12:40:29.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.509+0000 7f3f1d0c9640 1 -- 172.21.15.89:0/1506941093 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3ef80664a0 msgr2=0x7f3ef8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:29.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.509+0000 7f3f1d0c9640 1 --2- 172.21.15.89:0/1506941093 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3ef80664a0 0x7f3ef8068960 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3f0800c900 tx=0x7f3f08008040 comp rx=0 tx=0).stop 2023-10-30T12:40:29.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.509+0000 7f3f1d0c9640 1 -- 172.21.15.89:0/1506941093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f180c6f60 msgr2=0x7f3f1815eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:29.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.509+0000 7f3f1d0c9640 1 --2- 172.21.15.89:0/1506941093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f180c6f60 0x7f3f1815eae0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f3f00019110 tx=0x7f3f0001ad40 comp rx=0 tx=0).stop 2023-10-30T12:40:29.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.509+0000 7f3f1d0c9640 1 -- 172.21.15.89:0/1506941093 shutdown_connections 2023-10-30T12:40:29.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.509+0000 7f3f1d0c9640 1 --2- 172.21.15.89:0/1506941093 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3ef80664a0 0x7f3ef8068960 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:29.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.509+0000 7f3f1d0c9640 1 --2- 172.21.15.89:0/1506941093 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3f180c8430 0x7f3f1815f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:29.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.509+0000 7f3f1d0c9640 1 --2- 172.21.15.89:0/1506941093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f180c6f60 0x7f3f1815eae0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:29.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.509+0000 7f3f1d0c9640 1 --2- 172.21.15.89:0/1506941093 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3f180c6b80 0x7f3f18155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:29.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.509+0000 7f3f1d0c9640 1 -- 172.21.15.89:0/1506941093 >> 172.21.15.89:0/1506941093 conn(0x7f3f180c7910 msgr2=0x7f3f180c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:29.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.509+0000 7f3f1d0c9640 1 -- 172.21.15.89:0/1506941093 shutdown_connections 2023-10-30T12:40:29.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:29.509+0000 7f3f1d0c9640 1 -- 172.21.15.89:0/1506941093 wait complete. 2023-10-30T12:40:29.528 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:40:31.528 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:31.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.677+0000 7f0c97c46640 1 Processor -- start 2023-10-30T12:40:31.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.677+0000 7f0c97c46640 1 -- start start 2023-10-30T12:40:31.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.681+0000 7f0c97c46640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0c90071110 0x7f0c9006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:31.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.681+0000 7f0c97c46640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c90066cf0 con 0x7f0c900c6bd0 2023-10-30T12:40:31.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.681+0000 7f0c97c46640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c900c70d0 con 0x7f0c9006f7a0 2023-10-30T12:40:31.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.681+0000 7f0c97c46640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c900700c0 con 0x7f0c90071110 2023-10-30T12:40:31.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.681+0000 7f0c961e3640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0c90071110 0x7f0c9006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:31.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.681+0000 7f0c951e1640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f0c900c6bd0 0x7f0c90158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:49312/0 (socket says 172.21.15.89:49312) 2023-10-30T12:40:31.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.681+0000 7f0c951e1640 1 -- 172.21.15.89:0/3089088082 learned_addr learned my addr 172.21.15.89:0/3089088082 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:40:31.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.681+0000 7f0c961e3640 1 -- 172.21.15.89:0/3089088082 >> v1:172.21.15.192:6789/0 conn(0x7f0c9006f7a0 legacy=0x7f0c90069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:31.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.681+0000 7f0c961e3640 1 -- 172.21.15.89:0/3089088082 >> v1:172.21.15.89:6789/0 conn(0x7f0c900c6bd0 legacy=0x7f0c90158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:31.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.681+0000 7f0c961e3640 1 -- 172.21.15.89:0/3089088082 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c900c7390 con 0x7f0c90071110 2023-10-30T12:40:31.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.681+0000 7f0c949e0640 1 -- 172.21.15.89:0/3089088082 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3718706117 0 0) 0x7f0c90066cf0 con 0x7f0c900c6bd0 2023-10-30T12:40:31.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.685+0000 7f0c961e3640 1 --2- 172.21.15.89:0/3089088082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c90071110 0x7f0c9006fb80 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f0c88009d90 tx=0x7f0c8801f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f31908ae14d5804 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:31.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.685+0000 7f0c949e0640 1 -- 172.21.15.89:0/3089088082 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c880af600 con 0x7f0c90071110 2023-10-30T12:40:31.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.685+0000 7f0c949e0640 1 -- 172.21.15.89:0/3089088082 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c880ab050 con 0x7f0c90071110 2023-10-30T12:40:31.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.685+0000 7f0c949e0640 1 -- 172.21.15.89:0/3089088082 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c880af980 con 0x7f0c90071110 2023-10-30T12:40:31.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.685+0000 7f0c97c46640 1 -- 172.21.15.89:0/3089088082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c90071110 msgr2=0x7f0c9006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:31.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.685+0000 7f0c97c46640 1 --2- 172.21.15.89:0/3089088082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c90071110 0x7f0c9006fb80 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f0c88009d90 tx=0x7f0c8801f130 comp rx=0 tx=0).stop 2023-10-30T12:40:31.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.685+0000 7f0c97c46640 1 -- 172.21.15.89:0/3089088082 shutdown_connections 2023-10-30T12:40:31.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.685+0000 7f0c97c46640 1 --2- 172.21.15.89:0/3089088082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c90071110 0x7f0c9006fb80 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:31.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.685+0000 7f0c97c46640 1 -- 172.21.15.89:0/3089088082 >> 172.21.15.89:0/3089088082 conn(0x7f0c900c7910 msgr2=0x7f0c900c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:31.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.685+0000 7f0c97c46640 1 -- 172.21.15.89:0/3089088082 shutdown_connections 2023-10-30T12:40:31.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.685+0000 7f0c97c46640 1 -- 172.21.15.89:0/3089088082 wait complete. 2023-10-30T12:40:31.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.685+0000 7f0c97c46640 1 Processor -- start 2023-10-30T12:40:31.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.685+0000 7f0c97c46640 1 -- start start 2023-10-30T12:40:31.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.689+0000 7f0c97c46640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c9006f7a0 0x7f0c900c0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:31.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.689+0000 7f0c97c46640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c90071110 0x7f0c9014c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:31.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.689+0000 7f0c97c46640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c900c6bd0 0x7f0c9014cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:31.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.689+0000 7f0c97c46640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c901e38f0 con 0x7f0c90071110 2023-10-30T12:40:31.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.689+0000 7f0c97c46640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c901e22f0 con 0x7f0c900c6bd0 2023-10-30T12:40:31.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.689+0000 7f0c97c46640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c901e2560 con 0x7f0c9006f7a0 2023-10-30T12:40:31.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.689+0000 7f0c951e1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c90071110 0x7f0c9014c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:31.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.689+0000 7f0c951e1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c90071110 0x7f0c9014c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34044/0 (socket says 172.21.15.89:34044) 2023-10-30T12:40:31.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.689+0000 7f0c951e1640 1 -- 172.21.15.89:0/3090073308 learned_addr learned my addr 172.21.15.89:0/3090073308 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:40:31.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.689+0000 7f0c959e2640 1 --2- 172.21.15.89:0/3090073308 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c9006f7a0 0x7f0c900c0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:31.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.689+0000 7f0c961e3640 1 --2- 172.21.15.89:0/3090073308 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c900c6bd0 0x7f0c9014cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:31.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.689+0000 7f0c951e1640 1 -- 172.21.15.89:0/3090073308 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c9006f7a0 msgr2=0x7f0c900c0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:31.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.689+0000 7f0c951e1640 1 --2- 172.21.15.89:0/3090073308 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c9006f7a0 0x7f0c900c0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:31.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.689+0000 7f0c951e1640 1 -- 172.21.15.89:0/3090073308 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c900c6bd0 msgr2=0x7f0c9014cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:31.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.689+0000 7f0c951e1640 1 --2- 172.21.15.89:0/3090073308 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c900c6bd0 0x7f0c9014cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:31.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.689+0000 7f0c951e1640 1 -- 172.21.15.89:0/3090073308 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c88009a70 con 0x7f0c90071110 2023-10-30T12:40:31.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.689+0000 7f0c951e1640 1 --2- 172.21.15.89:0/3090073308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c90071110 0x7f0c9014c760 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f0c8000c4b0 tx=0x7f0c8000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:31.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.689+0000 7f0c867fc640 1 -- 172.21.15.89:0/3090073308 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c800074e0 con 0x7f0c90071110 2023-10-30T12:40:31.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.689+0000 7f0c867fc640 1 -- 172.21.15.89:0/3090073308 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c80007640 con 0x7f0c90071110 2023-10-30T12:40:31.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.689+0000 7f0c867fc640 1 -- 172.21.15.89:0/3090073308 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c80004a30 con 0x7f0c90071110 2023-10-30T12:40:31.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.689+0000 7f0c97c46640 1 -- 172.21.15.89:0/3090073308 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c901e2840 con 0x7f0c90071110 2023-10-30T12:40:31.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.689+0000 7f0c97c46640 1 -- 172.21.15.89:0/3090073308 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c9006fb80 con 0x7f0c90071110 2023-10-30T12:40:31.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.689+0000 7f0c97c46640 1 -- 172.21.15.89:0/3090073308 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c9004e170 con 0x7f0c90071110 2023-10-30T12:40:31.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.689+0000 7f0c867fc640 1 -- 172.21.15.89:0/3090073308 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0c80004b90 con 0x7f0c90071110 2023-10-30T12:40:31.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.693+0000 7f0c867fc640 1 --2- 172.21.15.89:0/3090073308 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0c740664a0 0x7f0c74068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:31.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.693+0000 7f0c867fc640 1 -- 172.21.15.89:0/3090073308 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6234+0+0 (secure 0 0 0) 0x7f0c80083340 con 0x7f0c90071110 2023-10-30T12:40:31.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.693+0000 7f0c959e2640 1 --2- 172.21.15.89:0/3090073308 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0c740664a0 0x7f0c74068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:31.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.693+0000 7f0c959e2640 1 --2- 172.21.15.89:0/3090073308 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0c740664a0 0x7f0c74068960 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c000f70 tx=0x7f0c8c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:31.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.697+0000 7f0c867fc640 1 -- 172.21.15.89:0/3090073308 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0c800505a0 con 0x7f0c90071110 2023-10-30T12:40:31.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.877+0000 7f0c97c46640 1 -- 172.21.15.89:0/3090073308 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c90151400 con 0x7f0c90071110 2023-10-30T12:40:31.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.877+0000 7f0c867fc640 1 -- 172.21.15.89:0/3090073308 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f0c80018090 con 0x7f0c90071110 2023-10-30T12:40:31.876 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:40:31.877 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:40:28.147139+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:31.877 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 60 2023-10-30T12:40:31.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.877+0000 7f0c97c46640 1 -- 172.21.15.89:0/3090073308 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0c740664a0 msgr2=0x7f0c74068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:31.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.877+0000 7f0c97c46640 1 --2- 172.21.15.89:0/3090073308 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0c740664a0 0x7f0c74068960 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c000f70 tx=0x7f0c8c009040 comp rx=0 tx=0).stop 2023-10-30T12:40:31.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.877+0000 7f0c97c46640 1 -- 172.21.15.89:0/3090073308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c90071110 msgr2=0x7f0c9014c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:31.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.877+0000 7f0c97c46640 1 --2- 172.21.15.89:0/3090073308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c90071110 0x7f0c9014c760 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f0c8000c4b0 tx=0x7f0c8000c970 comp rx=0 tx=0).stop 2023-10-30T12:40:31.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.877+0000 7f0c97c46640 1 -- 172.21.15.89:0/3090073308 shutdown_connections 2023-10-30T12:40:31.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.877+0000 7f0c97c46640 1 --2- 172.21.15.89:0/3090073308 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0c740664a0 0x7f0c74068960 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:31.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.877+0000 7f0c97c46640 1 --2- 172.21.15.89:0/3090073308 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c900c6bd0 0x7f0c9014cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:31.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.877+0000 7f0c97c46640 1 --2- 172.21.15.89:0/3090073308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c90071110 0x7f0c9014c760 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:31.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.877+0000 7f0c97c46640 1 --2- 172.21.15.89:0/3090073308 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c9006f7a0 0x7f0c900c0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:31.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.877+0000 7f0c97c46640 1 -- 172.21.15.89:0/3090073308 >> 172.21.15.89:0/3090073308 conn(0x7f0c900c7910 msgr2=0x7f0c900c3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:31.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.877+0000 7f0c97c46640 1 -- 172.21.15.89:0/3090073308 shutdown_connections 2023-10-30T12:40:31.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:31.877+0000 7f0c97c46640 1 -- 172.21.15.89:0/3090073308 wait complete. 2023-10-30T12:40:31.897 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:40:33.898 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:34.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.037+0000 7f8e30aea640 1 Processor -- start 2023-10-30T12:40:34.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.037+0000 7f8e30aea640 1 -- start start 2023-10-30T12:40:34.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.041+0000 7f8e30aea640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e2c1538a0 0x7f8e2c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:34.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.041+0000 7f8e30aea640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8e2c0ca0e0 0x7f8e2c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:34.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.041+0000 7f8e30aea640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e2c0cb800 con 0x7f8e2c0cf160 2023-10-30T12:40:34.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.041+0000 7f8e30aea640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e2c15dc30 con 0x7f8e2c0ca0e0 2023-10-30T12:40:34.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.041+0000 7f8e30aea640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2c0cb330 con 0x7f8e2c1538a0 2023-10-30T12:40:34.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.041+0000 7f8e2a59c640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e2c1538a0 0x7f8e2c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:34.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.041+0000 7f8e29d9b640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8e2c0ca0e0 0x7f8e2c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:34.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.041+0000 7f8e2a59c640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e2c1538a0 0x7f8e2c15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:52102/0 (socket says 172.21.15.89:52102) 2023-10-30T12:40:34.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.041+0000 7f8e2a59c640 1 -- 172.21.15.89:0/2578808246 learned_addr learned my addr 172.21.15.89:0/2578808246 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:40:34.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.041+0000 7f8e2a59c640 1 -- 172.21.15.89:0/2578808246 >> v2:172.21.15.192:3300/0 conn(0x7f8e2c0ca0e0 msgr2=0x7f8e2c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:34.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.041+0000 7f8e2a59c640 1 --2- 172.21.15.89:0/2578808246 >> v2:172.21.15.192:3300/0 conn(0x7f8e2c0ca0e0 0x7f8e2c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:34.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.041+0000 7f8e2a59c640 1 -- 172.21.15.89:0/2578808246 >> v1:172.21.15.192:6789/0 conn(0x7f8e2c0cf160 legacy=0x7f8e2c0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:34.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.041+0000 7f8e2a59c640 1 -- 172.21.15.89:0/2578808246 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e2c0cad70 con 0x7f8e2c1538a0 2023-10-30T12:40:34.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.041+0000 7f8e29d9b640 1 --2- 172.21.15.89:0/2578808246 >> v2:172.21.15.192:3300/0 conn(0x7f8e2c0ca0e0 0x7f8e2c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:40:34.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.045+0000 7f8e2a59c640 1 --2- 172.21.15.89:0/2578808246 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e2c1538a0 0x7f8e2c15d6f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c009870 tx=0x7f8e1c01ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee067a499e76bc09 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:34.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.045+0000 7f8e2959a640 1 -- 172.21.15.89:0/2578808246 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c0a8a30 con 0x7f8e2c1538a0 2023-10-30T12:40:34.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.045+0000 7f8e2959a640 1 -- 172.21.15.89:0/2578808246 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e1c0a8b90 con 0x7f8e2c1538a0 2023-10-30T12:40:34.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.045+0000 7f8e2959a640 1 -- 172.21.15.89:0/2578808246 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c0b0440 con 0x7f8e2c1538a0 2023-10-30T12:40:34.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.045+0000 7f8e30aea640 1 -- 172.21.15.89:0/2578808246 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e2c1538a0 msgr2=0x7f8e2c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:34.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.045+0000 7f8e30aea640 1 --2- 172.21.15.89:0/2578808246 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e2c1538a0 0x7f8e2c15d6f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c009870 tx=0x7f8e1c01ed30 comp rx=0 tx=0).stop 2023-10-30T12:40:34.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.045+0000 7f8e30aea640 1 -- 172.21.15.89:0/2578808246 shutdown_connections 2023-10-30T12:40:34.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.045+0000 7f8e30aea640 1 --2- 172.21.15.89:0/2578808246 >> v2:172.21.15.192:3300/0 conn(0x7f8e2c0ca0e0 0x7f8e2c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:34.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.045+0000 7f8e30aea640 1 --2- 172.21.15.89:0/2578808246 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e2c1538a0 0x7f8e2c15d6f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:34.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.045+0000 7f8e30aea640 1 -- 172.21.15.89:0/2578808246 >> 172.21.15.89:0/2578808246 conn(0x7f8e2c06bfe0 msgr2=0x7f8e2c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:34.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.045+0000 7f8e30aea640 1 -- 172.21.15.89:0/2578808246 shutdown_connections 2023-10-30T12:40:34.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.045+0000 7f8e30aea640 1 -- 172.21.15.89:0/2578808246 wait complete. 2023-10-30T12:40:34.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.045+0000 7f8e30aea640 1 Processor -- start 2023-10-30T12:40:34.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.045+0000 7f8e30aea640 1 -- start start 2023-10-30T12:40:34.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.049+0000 7f8e30aea640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e2c0ca0e0 0x7f8e2c0ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:34.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.049+0000 7f8e30aea640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e2c0cf160 0x7f8e2c0b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:34.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.049+0000 7f8e30aea640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e2c1538a0 0x7f8e2c0b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:34.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.049+0000 7f8e30aea640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2c1efcc0 con 0x7f8e2c0cf160 2023-10-30T12:40:34.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.049+0000 7f8e30aea640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2c1f01f0 con 0x7f8e2c1538a0 2023-10-30T12:40:34.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.049+0000 7f8e30aea640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2c1eff10 con 0x7f8e2c0ca0e0 2023-10-30T12:40:34.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.049+0000 7f8e29d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e2c0cf160 0x7f8e2c0b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:34.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.049+0000 7f8e29d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e2c0cf160 0x7f8e2c0b3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34052/0 (socket says 172.21.15.89:34052) 2023-10-30T12:40:34.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.049+0000 7f8e29d9b640 1 -- 172.21.15.89:0/97996392 learned_addr learned my addr 172.21.15.89:0/97996392 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:40:34.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.049+0000 7f8e2a59c640 1 --2- 172.21.15.89:0/97996392 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e2c0ca0e0 0x7f8e2c0ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:34.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.049+0000 7f8e2ad9d640 1 --2- 172.21.15.89:0/97996392 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e2c1538a0 0x7f8e2c0b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:34.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.049+0000 7f8e29d9b640 1 -- 172.21.15.89:0/97996392 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e2c0ca0e0 msgr2=0x7f8e2c0ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:34.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.049+0000 7f8e29d9b640 1 --2- 172.21.15.89:0/97996392 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e2c0ca0e0 0x7f8e2c0ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:34.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.049+0000 7f8e29d9b640 1 -- 172.21.15.89:0/97996392 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e2c1538a0 msgr2=0x7f8e2c0b42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:34.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.049+0000 7f8e29d9b640 1 --2- 172.21.15.89:0/97996392 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e2c1538a0 0x7f8e2c0b42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:34.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.049+0000 7f8e29d9b640 1 -- 172.21.15.89:0/97996392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e20009420 con 0x7f8e2c0cf160 2023-10-30T12:40:34.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.049+0000 7f8e29d9b640 1 --2- 172.21.15.89:0/97996392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e2c0cf160 0x7f8e2c0b3db0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f8e2000ebe0 tx=0x7f8e2000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:34.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.049+0000 7f8e1affd640 1 -- 172.21.15.89:0/97996392 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2000eee0 con 0x7f8e2c0cf160 2023-10-30T12:40:34.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.049+0000 7f8e1affd640 1 -- 172.21.15.89:0/97996392 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e2000cc30 con 0x7f8e2c0cf160 2023-10-30T12:40:34.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.049+0000 7f8e30aea640 1 -- 172.21.15.89:0/97996392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e1c0094d0 con 0x7f8e2c0cf160 2023-10-30T12:40:34.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.049+0000 7f8e1affd640 1 -- 172.21.15.89:0/97996392 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e20013470 con 0x7f8e2c0cf160 2023-10-30T12:40:34.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.049+0000 7f8e30aea640 1 -- 172.21.15.89:0/97996392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e2c0b35b0 con 0x7f8e2c0cf160 2023-10-30T12:40:34.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.049+0000 7f8e30aea640 1 -- 172.21.15.89:0/97996392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e2c0b2a50 con 0x7f8e2c0cf160 2023-10-30T12:40:34.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.049+0000 7f8e1affd640 1 -- 172.21.15.89:0/97996392 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8e200135d0 con 0x7f8e2c0cf160 2023-10-30T12:40:34.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.053+0000 7f8e1affd640 1 --2- 172.21.15.89:0/97996392 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8e0c0664d0 0x7f8e0c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:34.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.053+0000 7f8e1affd640 1 -- 172.21.15.89:0/97996392 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6234+0+0 (secure 0 0 0) 0x7f8e20084630 con 0x7f8e2c0cf160 2023-10-30T12:40:34.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.053+0000 7f8e2a59c640 1 --2- 172.21.15.89:0/97996392 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8e0c0664d0 0x7f8e0c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:34.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.053+0000 7f8e2a59c640 1 --2- 172.21.15.89:0/97996392 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8e0c0664d0 0x7f8e0c068990 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c0b2000 tx=0x7f8e1c0aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:34.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.057+0000 7f8e1affd640 1 -- 172.21.15.89:0/97996392 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8e20051940 con 0x7f8e2c0cf160 2023-10-30T12:40:34.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.237+0000 7f8e30aea640 1 -- 172.21.15.89:0/97996392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e2c04e170 con 0x7f8e2c0cf160 2023-10-30T12:40:34.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.237+0000 7f8e1affd640 1 -- 172.21.15.89:0/97996392 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f8e20054f60 con 0x7f8e2c0cf160 2023-10-30T12:40:34.238 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:40:34.238 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:40:28.147139+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:34.239 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 60 2023-10-30T12:40:34.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.241+0000 7f8e30aea640 1 -- 172.21.15.89:0/97996392 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8e0c0664d0 msgr2=0x7f8e0c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:34.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.241+0000 7f8e30aea640 1 --2- 172.21.15.89:0/97996392 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8e0c0664d0 0x7f8e0c068990 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c0b2000 tx=0x7f8e1c0aa070 comp rx=0 tx=0).stop 2023-10-30T12:40:34.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.241+0000 7f8e30aea640 1 -- 172.21.15.89:0/97996392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e2c0cf160 msgr2=0x7f8e2c0b3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:34.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.241+0000 7f8e30aea640 1 --2- 172.21.15.89:0/97996392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e2c0cf160 0x7f8e2c0b3db0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f8e2000ebe0 tx=0x7f8e2000c520 comp rx=0 tx=0).stop 2023-10-30T12:40:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.241+0000 7f8e30aea640 1 -- 172.21.15.89:0/97996392 shutdown_connections 2023-10-30T12:40:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.241+0000 7f8e30aea640 1 --2- 172.21.15.89:0/97996392 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8e0c0664d0 0x7f8e0c068990 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.241+0000 7f8e30aea640 1 --2- 172.21.15.89:0/97996392 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e2c1538a0 0x7f8e2c0b42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.241+0000 7f8e30aea640 1 --2- 172.21.15.89:0/97996392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e2c0cf160 0x7f8e2c0b3db0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:34.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.241+0000 7f8e30aea640 1 --2- 172.21.15.89:0/97996392 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e2c0ca0e0 0x7f8e2c0ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:34.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.241+0000 7f8e30aea640 1 -- 172.21.15.89:0/97996392 >> 172.21.15.89:0/97996392 conn(0x7f8e2c06bfe0 msgr2=0x7f8e2c152870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:34.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.241+0000 7f8e30aea640 1 -- 172.21.15.89:0/97996392 shutdown_connections 2023-10-30T12:40:34.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:34.241+0000 7f8e30aea640 1 -- 172.21.15.89:0/97996392 wait complete. 2023-10-30T12:40:34.260 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:40:36.260 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:36.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.409+0000 7f68f1646640 1 Processor -- start 2023-10-30T12:40:36.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.409+0000 7f68f1646640 1 -- start start 2023-10-30T12:40:36.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.409+0000 7f68f1646640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68ec1538a0 0x7f68ec15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:36.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.409+0000 7f68f1646640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f68ec0cf160 0x7f68ec0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:36.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.413+0000 7f68f1646640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68ec0cb800 con 0x7f68ec0ca0e0 2023-10-30T12:40:36.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.413+0000 7f68f1646640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68ec15dc30 con 0x7f68ec0cf160 2023-10-30T12:40:36.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.413+0000 7f68f1646640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec0cb330 con 0x7f68ec1538a0 2023-10-30T12:40:36.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.413+0000 7f68eb7fe640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f68ec0cf160 0x7f68ec0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:36.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.413+0000 7f68eaffd640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68ec1538a0 0x7f68ec15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:36.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.413+0000 7f68e3fff640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f68ec0ca0e0 0x7f68ec15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:55034/0 (socket says 172.21.15.89:55034) 2023-10-30T12:40:36.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.413+0000 7f68e3fff640 1 -- 172.21.15.89:0/1050910424 learned_addr learned my addr 172.21.15.89:0/1050910424 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:40:36.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.413+0000 7f68eaffd640 1 -- 172.21.15.89:0/1050910424 >> v2:172.21.15.192:3300/0 conn(0x7f68ec0cf160 msgr2=0x7f68ec0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:36.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.413+0000 7f68eaffd640 1 --2- 172.21.15.89:0/1050910424 >> v2:172.21.15.192:3300/0 conn(0x7f68ec0cf160 0x7f68ec0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.413+0000 7f68eaffd640 1 -- 172.21.15.89:0/1050910424 >> v1:172.21.15.192:6789/0 conn(0x7f68ec0ca0e0 legacy=0x7f68ec15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:36.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.413+0000 7f68eaffd640 1 -- 172.21.15.89:0/1050910424 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ec0cad70 con 0x7f68ec1538a0 2023-10-30T12:40:36.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.413+0000 7f68eb7fe640 1 --2- 172.21.15.89:0/1050910424 >> v2:172.21.15.192:3300/0 conn(0x7f68ec0cf160 0x7f68ec0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:40:36.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.417+0000 7f68eaffd640 1 --2- 172.21.15.89:0/1050910424 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68ec1538a0 0x7f68ec15d6f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f68dc0097c0 tx=0x7f68dc01ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c10aa3123e9ebd8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:36.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.417+0000 7f68ea7fc640 1 -- 172.21.15.89:0/1050910424 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68dc0a8970 con 0x7f68ec1538a0 2023-10-30T12:40:36.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.417+0000 7f68ea7fc640 1 -- 172.21.15.89:0/1050910424 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68dc0a8ad0 con 0x7f68ec1538a0 2023-10-30T12:40:36.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.417+0000 7f68ea7fc640 1 -- 172.21.15.89:0/1050910424 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68dc0a8e60 con 0x7f68ec1538a0 2023-10-30T12:40:36.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.417+0000 7f68f1646640 1 -- 172.21.15.89:0/1050910424 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68ec1538a0 msgr2=0x7f68ec15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:36.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.417+0000 7f68f1646640 1 --2- 172.21.15.89:0/1050910424 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68ec1538a0 0x7f68ec15d6f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f68dc0097c0 tx=0x7f68dc01ec70 comp rx=0 tx=0).stop 2023-10-30T12:40:36.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.417+0000 7f68f1646640 1 -- 172.21.15.89:0/1050910424 shutdown_connections 2023-10-30T12:40:36.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.417+0000 7f68f1646640 1 --2- 172.21.15.89:0/1050910424 >> v2:172.21.15.192:3300/0 conn(0x7f68ec0cf160 0x7f68ec0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.417+0000 7f68f1646640 1 --2- 172.21.15.89:0/1050910424 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68ec1538a0 0x7f68ec15d6f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.417+0000 7f68f1646640 1 -- 172.21.15.89:0/1050910424 >> 172.21.15.89:0/1050910424 conn(0x7f68ec06bfe0 msgr2=0x7f68ec06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:36.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.417+0000 7f68f1646640 1 -- 172.21.15.89:0/1050910424 shutdown_connections 2023-10-30T12:40:36.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.417+0000 7f68f1646640 1 -- 172.21.15.89:0/1050910424 wait complete. 2023-10-30T12:40:36.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.417+0000 7f68f1646640 1 Processor -- start 2023-10-30T12:40:36.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.417+0000 7f68f1646640 1 -- start start 2023-10-30T12:40:36.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.421+0000 7f68f1646640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68ec0ca0e0 0x7f68ec158910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:36.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.421+0000 7f68f1646640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68ec0cf160 0x7f68ec0ce590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:36.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.421+0000 7f68f1646640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68ec1538a0 0x7f68ec0cead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:36.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.421+0000 7f68f1646640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec1f36b0 con 0x7f68ec1538a0 2023-10-30T12:40:36.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.421+0000 7f68f1646640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec157a90 con 0x7f68ec0ca0e0 2023-10-30T12:40:36.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.421+0000 7f68f1646640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec14d790 con 0x7f68ec0cf160 2023-10-30T12:40:36.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.421+0000 7f68eb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68ec1538a0 0x7f68ec0cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:36.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.421+0000 7f68eb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68ec1538a0 0x7f68ec0cead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34068/0 (socket says 172.21.15.89:34068) 2023-10-30T12:40:36.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.421+0000 7f68eb7fe640 1 -- 172.21.15.89:0/2176320792 learned_addr learned my addr 172.21.15.89:0/2176320792 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:40:36.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.421+0000 7f68eaffd640 1 --2- 172.21.15.89:0/2176320792 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68ec0ca0e0 0x7f68ec158910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:36.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.421+0000 7f68e3fff640 1 --2- 172.21.15.89:0/2176320792 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68ec0cf160 0x7f68ec0ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:36.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.421+0000 7f68eb7fe640 1 -- 172.21.15.89:0/2176320792 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68ec0cf160 msgr2=0x7f68ec0ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:36.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.421+0000 7f68eb7fe640 1 --2- 172.21.15.89:0/2176320792 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68ec0cf160 0x7f68ec0ce590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.421+0000 7f68eb7fe640 1 -- 172.21.15.89:0/2176320792 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68ec0ca0e0 msgr2=0x7f68ec158910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:36.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.421+0000 7f68eb7fe640 1 --2- 172.21.15.89:0/2176320792 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68ec0ca0e0 0x7f68ec158910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.421+0000 7f68eb7fe640 1 -- 172.21.15.89:0/2176320792 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e4009a70 con 0x7f68ec1538a0 2023-10-30T12:40:36.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.421+0000 7f68e3fff640 1 --2- 172.21.15.89:0/2176320792 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68ec0cf160 0x7f68ec0ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:40:36.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.421+0000 7f68eb7fe640 1 --2- 172.21.15.89:0/2176320792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68ec1538a0 0x7f68ec0cead0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f68e400efc0 tx=0x7f68e400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:36.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.421+0000 7f68e37fe640 1 -- 172.21.15.89:0/2176320792 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e4009cc0 con 0x7f68ec1538a0 2023-10-30T12:40:36.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.421+0000 7f68e37fe640 1 -- 172.21.15.89:0/2176320792 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e400f040 con 0x7f68ec1538a0 2023-10-30T12:40:36.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.421+0000 7f68e37fe640 1 -- 172.21.15.89:0/2176320792 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e40039a0 con 0x7f68ec1538a0 2023-10-30T12:40:36.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.421+0000 7f68f1646640 1 -- 172.21.15.89:0/2176320792 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68dc009420 con 0x7f68ec1538a0 2023-10-30T12:40:36.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.421+0000 7f68f1646640 1 -- 172.21.15.89:0/2176320792 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68ec1f2d10 con 0x7f68ec1538a0 2023-10-30T12:40:36.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.421+0000 7f68f1646640 1 -- 172.21.15.89:0/2176320792 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68ec0b2a50 con 0x7f68ec1538a0 2023-10-30T12:40:36.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.421+0000 7f68e37fe640 1 -- 172.21.15.89:0/2176320792 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f68e4014480 con 0x7f68ec1538a0 2023-10-30T12:40:36.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.425+0000 7f68e37fe640 1 --2- 172.21.15.89:0/2176320792 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f68cc0664a0 0x7f68cc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:36.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.425+0000 7f68e37fe640 1 -- 172.21.15.89:0/2176320792 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6234+0+0 (secure 0 0 0) 0x7f68e4084c30 con 0x7f68ec1538a0 2023-10-30T12:40:36.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.425+0000 7f68eaffd640 1 --2- 172.21.15.89:0/2176320792 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f68cc0664a0 0x7f68cc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:36.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.425+0000 7f68eaffd640 1 --2- 172.21.15.89:0/2176320792 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f68cc0664a0 0x7f68cc068960 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f68dc0b2000 tx=0x7f68dc0aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:36.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.429+0000 7f68e37fe640 1 -- 172.21.15.89:0/2176320792 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f68e4051e90 con 0x7f68ec1538a0 2023-10-30T12:40:36.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.609+0000 7f68f1646640 1 -- 172.21.15.89:0/2176320792 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68ec1f37f0 con 0x7f68ec1538a0 2023-10-30T12:40:36.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.609+0000 7f68e37fe640 1 -- 172.21.15.89:0/2176320792 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f68ec1f37f0 con 0x7f68ec1538a0 2023-10-30T12:40:36.611 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:40:36.611 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:40:33.262671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:36.611 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 61 2023-10-30T12:40:36.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.613+0000 7f68f1646640 1 -- 172.21.15.89:0/2176320792 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f68cc0664a0 msgr2=0x7f68cc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:36.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.613+0000 7f68f1646640 1 --2- 172.21.15.89:0/2176320792 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f68cc0664a0 0x7f68cc068960 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f68dc0b2000 tx=0x7f68dc0aa070 comp rx=0 tx=0).stop 2023-10-30T12:40:36.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.613+0000 7f68f1646640 1 -- 172.21.15.89:0/2176320792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68ec1538a0 msgr2=0x7f68ec0cead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:36.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.613+0000 7f68f1646640 1 --2- 172.21.15.89:0/2176320792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68ec1538a0 0x7f68ec0cead0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f68e400efc0 tx=0x7f68e400c490 comp rx=0 tx=0).stop 2023-10-30T12:40:36.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.613+0000 7f68f1646640 1 -- 172.21.15.89:0/2176320792 shutdown_connections 2023-10-30T12:40:36.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.613+0000 7f68f1646640 1 --2- 172.21.15.89:0/2176320792 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f68cc0664a0 0x7f68cc068960 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.613+0000 7f68f1646640 1 --2- 172.21.15.89:0/2176320792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68ec1538a0 0x7f68ec0cead0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.613+0000 7f68f1646640 1 --2- 172.21.15.89:0/2176320792 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68ec0cf160 0x7f68ec0ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.613+0000 7f68f1646640 1 --2- 172.21.15.89:0/2176320792 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68ec0ca0e0 0x7f68ec158910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.613+0000 7f68f1646640 1 -- 172.21.15.89:0/2176320792 >> 172.21.15.89:0/2176320792 conn(0x7f68ec06bfe0 msgr2=0x7f68ec1510d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:36.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.613+0000 7f68f1646640 1 -- 172.21.15.89:0/2176320792 shutdown_connections 2023-10-30T12:40:36.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.613+0000 7f68f1646640 1 -- 172.21.15.89:0/2176320792 wait complete. 2023-10-30T12:40:36.631 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=2, replacing gid=4781 2023-10-30T12:40:36.631 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.b 2023-10-30T12:40:36.631 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2023-10-30T12:40:36.632 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-30T12:40:36.632 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-30T12:40:36.635 INFO:tasks.ceph.mds.b:Started 2023-10-30T12:40:36.635 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:36.710 INFO:tasks.ceph.mds.b.smithi192.stdout:starting mds.b at 2023-10-30T12:40:36.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.773+0000 7f657e09c640 1 Processor -- start 2023-10-30T12:40:36.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.773+0000 7f657e09c640 1 -- start start 2023-10-30T12:40:36.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.773+0000 7f657e09c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f65780c6bd0 0x7f657806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:36.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.773+0000 7f657e09c640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6578066cf0 con 0x7f6578071110 2023-10-30T12:40:36.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.773+0000 7f657e09c640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65780c70d0 con 0x7f657806f7a0 2023-10-30T12:40:36.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.773+0000 7f657e09c640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65780700c0 con 0x7f65780c6bd0 2023-10-30T12:40:36.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.773+0000 7f6576ffd640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f65780c6bd0 0x7f657806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:36.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.773+0000 7f6577fff640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6578071110 0x7f6578158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:49314/0 (socket says 172.21.15.89:49314) 2023-10-30T12:40:36.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.773+0000 7f6577fff640 1 -- 172.21.15.89:0/3263090394 learned_addr learned my addr 172.21.15.89:0/3263090394 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:40:36.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.777+0000 7f6576ffd640 1 -- 172.21.15.89:0/3263090394 >> v1:172.21.15.192:6789/0 conn(0x7f657806f7a0 legacy=0x7f6578069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:36.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.777+0000 7f6576ffd640 1 -- 172.21.15.89:0/3263090394 >> v1:172.21.15.89:6789/0 conn(0x7f6578071110 legacy=0x7f6578158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:36.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.777+0000 7f6576ffd640 1 -- 172.21.15.89:0/3263090394 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65780c7390 con 0x7f65780c6bd0 2023-10-30T12:40:36.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.777+0000 7f65767fc640 1 -- 172.21.15.89:0/3263090394 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2305145252 0 0) 0x7f6578066cf0 con 0x7f6578071110 2023-10-30T12:40:36.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.777+0000 7f6576ffd640 1 --2- 172.21.15.89:0/3263090394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65780c6bd0 0x7f657806fb80 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f656c0097c0 tx=0x7f656c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cada99d07fe98856 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:36.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.777+0000 7f65767fc640 1 -- 172.21.15.89:0/3263090394 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c0a8970 con 0x7f65780c6bd0 2023-10-30T12:40:36.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.777+0000 7f65767fc640 1 -- 172.21.15.89:0/3263090394 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656c0a8ad0 con 0x7f65780c6bd0 2023-10-30T12:40:36.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.777+0000 7f65767fc640 1 -- 172.21.15.89:0/3263090394 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c0a8e60 con 0x7f65780c6bd0 2023-10-30T12:40:36.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.777+0000 7f657e09c640 1 -- 172.21.15.89:0/3263090394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65780c6bd0 msgr2=0x7f657806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:36.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.777+0000 7f657e09c640 1 --2- 172.21.15.89:0/3263090394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65780c6bd0 0x7f657806fb80 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f656c0097c0 tx=0x7f656c01ec70 comp rx=0 tx=0).stop 2023-10-30T12:40:36.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.777+0000 7f657e09c640 1 -- 172.21.15.89:0/3263090394 shutdown_connections 2023-10-30T12:40:36.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.777+0000 7f657e09c640 1 --2- 172.21.15.89:0/3263090394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65780c6bd0 0x7f657806fb80 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.777+0000 7f657e09c640 1 -- 172.21.15.89:0/3263090394 >> 172.21.15.89:0/3263090394 conn(0x7f65780c7910 msgr2=0x7f65780c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:36.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.781+0000 7f657e09c640 1 -- 172.21.15.89:0/3263090394 shutdown_connections 2023-10-30T12:40:36.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.781+0000 7f657e09c640 1 -- 172.21.15.89:0/3263090394 wait complete. 2023-10-30T12:40:36.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.781+0000 7f657e09c640 1 Processor -- start 2023-10-30T12:40:36.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.781+0000 7f657e09c640 1 -- start start 2023-10-30T12:40:36.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.781+0000 7f657e09c640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f657806f7a0 0x7f6578154ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:36.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.781+0000 7f657e09c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6578071110 0x7f65780c2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:36.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.781+0000 7f657e09c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65780c6bd0 0x7f65780c2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:36.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.781+0000 7f657e09c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65781584c0 con 0x7f6578071110 2023-10-30T12:40:36.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.781+0000 7f657e09c640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65780ad960 con 0x7f65780c6bd0 2023-10-30T12:40:36.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.781+0000 7f657e09c640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6578157c60 con 0x7f657806f7a0 2023-10-30T12:40:36.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.781+0000 7f6576ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6578071110 0x7f65780c2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:36.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.781+0000 7f6576ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6578071110 0x7f65780c2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34082/0 (socket says 172.21.15.89:34082) 2023-10-30T12:40:36.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.781+0000 7f6576ffd640 1 -- 172.21.15.89:0/425694341 learned_addr learned my addr 172.21.15.89:0/425694341 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:40:36.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.781+0000 7f65777fe640 1 --2- 172.21.15.89:0/425694341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f657806f7a0 0x7f6578154ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:36.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.781+0000 7f6577fff640 1 --2- 172.21.15.89:0/425694341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65780c6bd0 0x7f65780c2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:36.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.781+0000 7f6576ffd640 1 -- 172.21.15.89:0/425694341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f657806f7a0 msgr2=0x7f6578154ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:36.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.781+0000 7f6576ffd640 1 --2- 172.21.15.89:0/425694341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f657806f7a0 0x7f6578154ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.781+0000 7f6576ffd640 1 -- 172.21.15.89:0/425694341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65780c6bd0 msgr2=0x7f65780c2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:36.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.781+0000 7f6576ffd640 1 --2- 172.21.15.89:0/425694341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65780c6bd0 0x7f65780c2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.781+0000 7f6576ffd640 1 -- 172.21.15.89:0/425694341 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f656c009420 con 0x7f6578071110 2023-10-30T12:40:36.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.781+0000 7f6576ffd640 1 --2- 172.21.15.89:0/425694341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6578071110 0x7f65780c2640 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f656c01b880 tx=0x7f656c01ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:36.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.785+0000 7f655ffff640 1 -- 172.21.15.89:0/425694341 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c0b0ea0 con 0x7f6578071110 2023-10-30T12:40:36.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.785+0000 7f655ffff640 1 -- 172.21.15.89:0/425694341 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656c0bb420 con 0x7f6578071110 2023-10-30T12:40:36.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.785+0000 7f655ffff640 1 -- 172.21.15.89:0/425694341 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c0bb7f0 con 0x7f6578071110 2023-10-30T12:40:36.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.785+0000 7f657e09c640 1 -- 172.21.15.89:0/425694341 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65781efd20 con 0x7f6578071110 2023-10-30T12:40:36.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.785+0000 7f657e09c640 1 -- 172.21.15.89:0/425694341 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65781581e0 con 0x7f6578071110 2023-10-30T12:40:36.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.785+0000 7f657e09c640 1 -- 172.21.15.89:0/425694341 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f657804e170 con 0x7f6578071110 2023-10-30T12:40:36.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.785+0000 7f655ffff640 1 -- 172.21.15.89:0/425694341 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f656c0b7040 con 0x7f6578071110 2023-10-30T12:40:36.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.785+0000 7f655ffff640 1 --2- 172.21.15.89:0/425694341 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f65640664a0 0x7f6564068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:36.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.785+0000 7f655ffff640 1 -- 172.21.15.89:0/425694341 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6234+0+0 (secure 0 0 0) 0x7f656c1222f0 con 0x7f6578071110 2023-10-30T12:40:36.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.785+0000 7f65777fe640 1 --2- 172.21.15.89:0/425694341 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f65640664a0 0x7f6564068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:36.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.789+0000 7f65777fe640 1 --2- 172.21.15.89:0/425694341 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f65640664a0 0x7f6564068960 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6568000f70 tx=0x7f6568009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:36.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.789+0000 7f655ffff640 1 -- 172.21.15.89:0/425694341 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f656c0ef430 con 0x7f6578071110 2023-10-30T12:40:36.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.969+0000 7f657e09c640 1 -- 172.21.15.89:0/425694341 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65780c1c80 con 0x7f6578071110 2023-10-30T12:40:36.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.969+0000 7f655ffff640 1 -- 172.21.15.89:0/425694341 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f656c0b5090 con 0x7f6578071110 2023-10-30T12:40:36.971 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:40:36.971 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:40:33.262671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:36.972 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 61 2023-10-30T12:40:36.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.973+0000 7f657e09c640 1 -- 172.21.15.89:0/425694341 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f65640664a0 msgr2=0x7f6564068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:36.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.973+0000 7f657e09c640 1 --2- 172.21.15.89:0/425694341 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f65640664a0 0x7f6564068960 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6568000f70 tx=0x7f6568009040 comp rx=0 tx=0).stop 2023-10-30T12:40:36.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.973+0000 7f657e09c640 1 -- 172.21.15.89:0/425694341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6578071110 msgr2=0x7f65780c2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:36.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.973+0000 7f657e09c640 1 --2- 172.21.15.89:0/425694341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6578071110 0x7f65780c2640 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f656c01b880 tx=0x7f656c01ba10 comp rx=0 tx=0).stop 2023-10-30T12:40:36.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.973+0000 7f657e09c640 1 -- 172.21.15.89:0/425694341 shutdown_connections 2023-10-30T12:40:36.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.973+0000 7f657e09c640 1 --2- 172.21.15.89:0/425694341 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f65640664a0 0x7f6564068960 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.973+0000 7f657e09c640 1 --2- 172.21.15.89:0/425694341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65780c6bd0 0x7f65780c2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.973+0000 7f657e09c640 1 --2- 172.21.15.89:0/425694341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6578071110 0x7f65780c2640 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.973+0000 7f657e09c640 1 --2- 172.21.15.89:0/425694341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f657806f7a0 0x7f6578154ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.973+0000 7f657e09c640 1 -- 172.21.15.89:0/425694341 >> 172.21.15.89:0/425694341 conn(0x7f65780c7910 msgr2=0x7f65780c3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:36.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.973+0000 7f657e09c640 1 -- 172.21.15.89:0/425694341 shutdown_connections 2023-10-30T12:40:36.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:36.973+0000 7f657e09c640 1 -- 172.21.15.89:0/425694341 wait complete. 2023-10-30T12:40:36.991 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2023-10-30T12:40:38.992 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:39.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.145+0000 7fae60f0f640 1 Processor -- start 2023-10-30T12:40:39.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.145+0000 7fae60f0f640 1 -- start start 2023-10-30T12:40:39.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.149+0000 7fae60f0f640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fae5c1539a0 0x7fae5c15a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:39.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.149+0000 7fae60f0f640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fae5c070a00 0x7fae5c070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:39.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.149+0000 7fae60f0f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae5c069e90 0x7fae5c06a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:39.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.149+0000 7fae60f0f640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae5c06fbb0 con 0x7fae5c1539a0 2023-10-30T12:40:39.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.149+0000 7fae60f0f640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae5c15a8c0 con 0x7fae5c070a00 2023-10-30T12:40:39.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.149+0000 7fae60f0f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae5c0713b0 con 0x7fae5c069e90 2023-10-30T12:40:39.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.149+0000 7fae5a59c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fae5c1539a0 0x7fae5c15a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:39.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.149+0000 7fae5a59c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fae5c1539a0 0x7fae5c15a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34086/0 (socket says 172.21.15.89:34086) 2023-10-30T12:40:39.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.149+0000 7fae5a59c640 1 -- 172.21.15.89:0/3546095871 learned_addr learned my addr 172.21.15.89:0/3546095871 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:40:39.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.149+0000 7fae53fff640 1 --2- 172.21.15.89:0/3546095871 >> v2:172.21.15.192:3300/0 conn(0x7fae5c070a00 0x7fae5c070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:39.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.149+0000 7fae5ad9d640 1 --2- 172.21.15.89:0/3546095871 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae5c069e90 0x7fae5c06a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:39.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.149+0000 7fae5a59c640 1 -- 172.21.15.89:0/3546095871 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae5c069e90 msgr2=0x7fae5c06a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:39.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.149+0000 7fae5a59c640 1 --2- 172.21.15.89:0/3546095871 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae5c069e90 0x7fae5c06a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:39.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.149+0000 7fae5a59c640 1 -- 172.21.15.89:0/3546095871 >> v2:172.21.15.192:3300/0 conn(0x7fae5c070a00 msgr2=0x7fae5c070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:39.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.149+0000 7fae5a59c640 1 --2- 172.21.15.89:0/3546095871 >> v2:172.21.15.192:3300/0 conn(0x7fae5c070a00 0x7fae5c070e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:39.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.149+0000 7fae5a59c640 1 -- 172.21.15.89:0/3546095871 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae5c153d80 con 0x7fae5c1539a0 2023-10-30T12:40:39.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.149+0000 7fae5ad9d640 1 --2- 172.21.15.89:0/3546095871 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae5c069e90 0x7fae5c06a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:40:39.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.149+0000 7fae53fff640 1 --2- 172.21.15.89:0/3546095871 >> v2:172.21.15.192:3300/0 conn(0x7fae5c070a00 0x7fae5c070e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:40:39.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.153+0000 7fae5a59c640 1 --2- 172.21.15.89:0/3546095871 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae5c1539a0 0x7fae5c15a380 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fae5c072640 tx=0x7fae4c01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23da2ca901ec0bed server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:39.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.153+0000 7fae59d9b640 1 -- 172.21.15.89:0/3546095871 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4c0ac640 con 0x7fae5c1539a0 2023-10-30T12:40:39.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.153+0000 7fae59d9b640 1 -- 172.21.15.89:0/3546095871 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae4c0ac7a0 con 0x7fae5c1539a0 2023-10-30T12:40:39.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.153+0000 7fae59d9b640 1 -- 172.21.15.89:0/3546095871 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4c0acb30 con 0x7fae5c1539a0 2023-10-30T12:40:39.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.153+0000 7fae60f0f640 1 -- 172.21.15.89:0/3546095871 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae5c1539a0 msgr2=0x7fae5c15a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:39.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.153+0000 7fae60f0f640 1 --2- 172.21.15.89:0/3546095871 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae5c1539a0 0x7fae5c15a380 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fae5c072640 tx=0x7fae4c01c440 comp rx=0 tx=0).stop 2023-10-30T12:40:39.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.153+0000 7fae60f0f640 1 -- 172.21.15.89:0/3546095871 shutdown_connections 2023-10-30T12:40:39.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.153+0000 7fae60f0f640 1 --2- 172.21.15.89:0/3546095871 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae5c069e90 0x7fae5c06a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:39.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.153+0000 7fae60f0f640 1 --2- 172.21.15.89:0/3546095871 >> v2:172.21.15.192:3300/0 conn(0x7fae5c070a00 0x7fae5c070e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:39.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.153+0000 7fae60f0f640 1 --2- 172.21.15.89:0/3546095871 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae5c1539a0 0x7fae5c15a380 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:39.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.153+0000 7fae60f0f640 1 -- 172.21.15.89:0/3546095871 >> 172.21.15.89:0/3546095871 conn(0x7fae5c0c7910 msgr2=0x7fae5c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:39.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.153+0000 7fae60f0f640 1 -- 172.21.15.89:0/3546095871 shutdown_connections 2023-10-30T12:40:39.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.153+0000 7fae60f0f640 1 -- 172.21.15.89:0/3546095871 wait complete. 2023-10-30T12:40:39.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.153+0000 7fae60f0f640 1 Processor -- start 2023-10-30T12:40:39.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.153+0000 7fae60f0f640 1 -- start start 2023-10-30T12:40:39.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.157+0000 7fae60f0f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae5c069e90 0x7fae5c150860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:39.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.157+0000 7fae60f0f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae5c070a00 0x7fae5c145b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:39.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.157+0000 7fae60f0f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae5c1539a0 0x7fae5c0c1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:39.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.157+0000 7fae60f0f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5c154640 con 0x7fae5c069e90 2023-10-30T12:40:39.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.157+0000 7fae60f0f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5c1f2180 con 0x7fae5c070a00 2023-10-30T12:40:39.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.157+0000 7fae60f0f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae5c150da0 con 0x7fae5c1539a0 2023-10-30T12:40:39.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.157+0000 7fae5a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae5c069e90 0x7fae5c150860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:39.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.157+0000 7fae5a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae5c069e90 0x7fae5c150860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34094/0 (socket says 172.21.15.89:34094) 2023-10-30T12:40:39.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.157+0000 7fae5a59c640 1 -- 172.21.15.89:0/3356425628 learned_addr learned my addr 172.21.15.89:0/3356425628 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:40:39.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.157+0000 7fae53fff640 1 --2- 172.21.15.89:0/3356425628 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae5c070a00 0x7fae5c145b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:39.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.157+0000 7fae5ad9d640 1 --2- 172.21.15.89:0/3356425628 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae5c1539a0 0x7fae5c0c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:39.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.157+0000 7fae5a59c640 1 -- 172.21.15.89:0/3356425628 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae5c1539a0 msgr2=0x7fae5c0c1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:39.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.157+0000 7fae5a59c640 1 --2- 172.21.15.89:0/3356425628 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae5c1539a0 0x7fae5c0c1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:39.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.157+0000 7fae5a59c640 1 -- 172.21.15.89:0/3356425628 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae5c070a00 msgr2=0x7fae5c145b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:39.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.157+0000 7fae5a59c640 1 --2- 172.21.15.89:0/3356425628 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae5c070a00 0x7fae5c145b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:39.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.157+0000 7fae5a59c640 1 -- 172.21.15.89:0/3356425628 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae4c0094d0 con 0x7fae5c069e90 2023-10-30T12:40:39.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.157+0000 7fae53fff640 1 --2- 172.21.15.89:0/3356425628 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae5c070a00 0x7fae5c145b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:40:39.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.157+0000 7fae5a59c640 1 --2- 172.21.15.89:0/3356425628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae5c069e90 0x7fae5c150860 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fae4c0af040 tx=0x7fae4c01ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:39.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.157+0000 7fae52ffd640 1 -- 172.21.15.89:0/3356425628 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4c0b7830 con 0x7fae5c069e90 2023-10-30T12:40:39.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.157+0000 7fae52ffd640 1 -- 172.21.15.89:0/3356425628 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae4c0b7990 con 0x7fae5c069e90 2023-10-30T12:40:39.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.157+0000 7fae52ffd640 1 -- 172.21.15.89:0/3356425628 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4c0b7d20 con 0x7fae5c069e90 2023-10-30T12:40:39.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.157+0000 7fae60f0f640 1 -- 172.21.15.89:0/3356425628 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae5c1f2390 con 0x7fae5c069e90 2023-10-30T12:40:39.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.157+0000 7fae60f0f640 1 -- 172.21.15.89:0/3356425628 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae5c1f2950 con 0x7fae5c069e90 2023-10-30T12:40:39.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.157+0000 7fae60f0f640 1 -- 172.21.15.89:0/3356425628 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae5c04e170 con 0x7fae5c069e90 2023-10-30T12:40:39.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.157+0000 7fae52ffd640 1 -- 172.21.15.89:0/3356425628 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fae4c0b3040 con 0x7fae5c069e90 2023-10-30T12:40:39.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.161+0000 7fae52ffd640 1 --2- 172.21.15.89:0/3356425628 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fae3c0664a0 0x7fae3c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:39.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.161+0000 7fae52ffd640 1 -- 172.21.15.89:0/3356425628 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6234+0+0 (secure 0 0 0) 0x7fae4c11e730 con 0x7fae5c069e90 2023-10-30T12:40:39.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.161+0000 7fae53fff640 1 --2- 172.21.15.89:0/3356425628 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fae3c0664a0 0x7fae3c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:39.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.161+0000 7fae53fff640 1 --2- 172.21.15.89:0/3356425628 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fae3c0664a0 0x7fae3c068960 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fae5c158d60 tx=0x7fae44008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:39.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.165+0000 7fae52ffd640 1 -- 172.21.15.89:0/3356425628 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fae4c0eb870 con 0x7fae5c069e90 2023-10-30T12:40:39.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.349+0000 7fae60f0f640 1 -- 172.21.15.89:0/3356425628 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae5c06a590 con 0x7fae5c069e90 2023-10-30T12:40:39.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.349+0000 7fae52ffd640 1 -- 172.21.15.89:0/3356425628 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+6529 (secure 0 0 0) 0x7fae4c0eee90 con 0x7fae5c069e90 2023-10-30T12:40:39.351 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:40:39.351 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5575,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:40:33.262671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:39.352 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 62 2023-10-30T12:40:39.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.353+0000 7fae60f0f640 1 -- 172.21.15.89:0/3356425628 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fae3c0664a0 msgr2=0x7fae3c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:39.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.353+0000 7fae60f0f640 1 --2- 172.21.15.89:0/3356425628 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fae3c0664a0 0x7fae3c068960 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fae5c158d60 tx=0x7fae44008040 comp rx=0 tx=0).stop 2023-10-30T12:40:39.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.353+0000 7fae60f0f640 1 -- 172.21.15.89:0/3356425628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae5c069e90 msgr2=0x7fae5c150860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:39.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.353+0000 7fae60f0f640 1 --2- 172.21.15.89:0/3356425628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae5c069e90 0x7fae5c150860 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fae4c0af040 tx=0x7fae4c01ac80 comp rx=0 tx=0).stop 2023-10-30T12:40:39.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.353+0000 7fae60f0f640 1 -- 172.21.15.89:0/3356425628 shutdown_connections 2023-10-30T12:40:39.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.353+0000 7fae60f0f640 1 --2- 172.21.15.89:0/3356425628 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fae3c0664a0 0x7fae3c068960 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:39.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.353+0000 7fae60f0f640 1 --2- 172.21.15.89:0/3356425628 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae5c070a00 0x7fae5c145b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:39.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.353+0000 7fae60f0f640 1 --2- 172.21.15.89:0/3356425628 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fae5c1539a0 0x7fae5c0c1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:39.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.353+0000 7fae60f0f640 1 --2- 172.21.15.89:0/3356425628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae5c069e90 0x7fae5c150860 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:39.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.353+0000 7fae60f0f640 1 -- 172.21.15.89:0/3356425628 >> 172.21.15.89:0/3356425628 conn(0x7fae5c0c7910 msgr2=0x7fae5c158040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:39.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.353+0000 7fae60f0f640 1 -- 172.21.15.89:0/3356425628 shutdown_connections 2023-10-30T12:40:39.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:40:39.353+0000 7fae60f0f640 1 -- 172.21.15.89:0/3356425628 wait complete. 2023-10-30T12:40:39.371 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2023-10-30T12:40:39.371 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 89 secs before thrashing 2023-10-30T12:40:46.958 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:40:46.963 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:41:17.120 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:41:17.125 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:41:47.170 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:41:47.174 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:42:08.373 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:08.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.520+0000 7fee7893e640 1 Processor -- start 2023-10-30T12:42:08.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.520+0000 7fee7893e640 1 -- start start 2023-10-30T12:42:08.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.524+0000 7fee7893e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fee74159e60 0x7fee7406f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:08.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.524+0000 7fee7893e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fee740bfec0 0x7fee740c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:08.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.524+0000 7fee7893e640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee7406fa80 con 0x7fee7406a720 2023-10-30T12:42:08.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.524+0000 7fee7893e640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee74071290 con 0x7fee74159e60 2023-10-30T12:42:08.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.524+0000 7fee7893e640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee74071570 con 0x7fee740bfec0 2023-10-30T12:42:08.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.524+0000 7fee7259c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fee74159e60 0x7fee7406f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:08.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.524+0000 7fee72d9d640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fee7406a720 0x7fee74155d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:49748/0 (socket says 172.21.15.89:49748) 2023-10-30T12:42:08.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.524+0000 7fee72d9d640 1 -- 172.21.15.89:0/890227382 learned_addr learned my addr 172.21.15.89:0/890227382 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:42:08.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.524+0000 7fee6bfff640 1 --2- 172.21.15.89:0/890227382 >> v2:172.21.15.192:3300/0 conn(0x7fee740bfec0 0x7fee740c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:08.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.524+0000 7fee7259c640 1 -- 172.21.15.89:0/890227382 >> v2:172.21.15.192:3300/0 conn(0x7fee740bfec0 msgr2=0x7fee740c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:08.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.524+0000 7fee7259c640 1 --2- 172.21.15.89:0/890227382 >> v2:172.21.15.192:3300/0 conn(0x7fee740bfec0 0x7fee740c1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:08.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.524+0000 7fee7259c640 1 -- 172.21.15.89:0/890227382 >> v1:172.21.15.89:6789/0 conn(0x7fee7406a720 legacy=0x7fee74155d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:08.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.524+0000 7fee7259c640 1 -- 172.21.15.89:0/890227382 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee74071850 con 0x7fee74159e60 2023-10-30T12:42:08.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.524+0000 7fee71d9b640 1 -- 172.21.15.89:0/890227382 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 472890544 0 0) 0x7fee7406fa80 con 0x7fee7406a720 2023-10-30T12:42:08.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.528+0000 7fee7259c640 1 --2- 172.21.15.89:0/890227382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee74159e60 0x7fee7406f540 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fee7406f200 tx=0x7fee6401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb9e2d2b2cd91eb7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:08.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.528+0000 7fee71d9b640 1 -- 172.21.15.89:0/890227382 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee640a8950 con 0x7fee74159e60 2023-10-30T12:42:08.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.528+0000 7fee71d9b640 1 -- 172.21.15.89:0/890227382 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee640a8ab0 con 0x7fee74159e60 2023-10-30T12:42:08.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.528+0000 7fee71d9b640 1 -- 172.21.15.89:0/890227382 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee640a8e40 con 0x7fee74159e60 2023-10-30T12:42:08.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.528+0000 7fee7893e640 1 -- 172.21.15.89:0/890227382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee74159e60 msgr2=0x7fee7406f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:08.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.528+0000 7fee7893e640 1 --2- 172.21.15.89:0/890227382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee74159e60 0x7fee7406f540 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fee7406f200 tx=0x7fee6401ec50 comp rx=0 tx=0).stop 2023-10-30T12:42:08.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.528+0000 7fee7893e640 1 -- 172.21.15.89:0/890227382 shutdown_connections 2023-10-30T12:42:08.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.528+0000 7fee7893e640 1 --2- 172.21.15.89:0/890227382 >> v2:172.21.15.192:3300/0 conn(0x7fee740bfec0 0x7fee740c1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:08.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.528+0000 7fee7893e640 1 --2- 172.21.15.89:0/890227382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee74159e60 0x7fee7406f540 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:08.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.528+0000 7fee7893e640 1 -- 172.21.15.89:0/890227382 >> 172.21.15.89:0/890227382 conn(0x7fee740c7910 msgr2=0x7fee740c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:08.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.528+0000 7fee7893e640 1 -- 172.21.15.89:0/890227382 shutdown_connections 2023-10-30T12:42:08.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.528+0000 7fee7893e640 1 -- 172.21.15.89:0/890227382 wait complete. 2023-10-30T12:42:08.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.528+0000 7fee7893e640 1 Processor -- start 2023-10-30T12:42:08.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.528+0000 7fee7893e640 1 -- start start 2023-10-30T12:42:08.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.528+0000 7fee7893e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee7406a720 0x7fee740c6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:08.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.528+0000 7fee7893e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee740bfec0 0x7fee74140860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:08.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.528+0000 7fee7893e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee74159e60 0x7fee74140da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:08.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.528+0000 7fee7893e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee741459f0 con 0x7fee7406a720 2023-10-30T12:42:08.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.528+0000 7fee7893e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee74145410 con 0x7fee74159e60 2023-10-30T12:42:08.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.528+0000 7fee7893e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee741565d0 con 0x7fee740bfec0 2023-10-30T12:42:08.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.532+0000 7fee7259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee7406a720 0x7fee740c6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:08.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.532+0000 7fee7259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee7406a720 0x7fee740c6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47094/0 (socket says 172.21.15.89:47094) 2023-10-30T12:42:08.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.532+0000 7fee7259c640 1 -- 172.21.15.89:0/5375204 learned_addr learned my addr 172.21.15.89:0/5375204 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:08.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.532+0000 7fee72d9d640 1 --2- 172.21.15.89:0/5375204 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee74159e60 0x7fee74140da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:08.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.532+0000 7fee6bfff640 1 --2- 172.21.15.89:0/5375204 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee740bfec0 0x7fee74140860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:08.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.532+0000 7fee7259c640 1 -- 172.21.15.89:0/5375204 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee740bfec0 msgr2=0x7fee74140860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:08.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.532+0000 7fee7259c640 1 --2- 172.21.15.89:0/5375204 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee740bfec0 0x7fee74140860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:08.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.532+0000 7fee7259c640 1 -- 172.21.15.89:0/5375204 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee74159e60 msgr2=0x7fee74140da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:08.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.532+0000 7fee7259c640 1 --2- 172.21.15.89:0/5375204 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee74159e60 0x7fee74140da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:08.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.532+0000 7fee7259c640 1 -- 172.21.15.89:0/5375204 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee64009420 con 0x7fee7406a720 2023-10-30T12:42:08.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.532+0000 7fee72d9d640 1 --2- 172.21.15.89:0/5375204 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee74159e60 0x7fee74140da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:42:08.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.532+0000 7fee7259c640 1 --2- 172.21.15.89:0/5375204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee7406a720 0x7fee740c6860 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fee6401c880 tx=0x7fee6401b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:08.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.532+0000 7fee6affd640 1 -- 172.21.15.89:0/5375204 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee640b1d40 con 0x7fee7406a720 2023-10-30T12:42:08.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.532+0000 7fee6affd640 1 -- 172.21.15.89:0/5375204 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee640b9410 con 0x7fee7406a720 2023-10-30T12:42:08.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.532+0000 7fee7893e640 1 -- 172.21.15.89:0/5375204 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee741568b0 con 0x7fee7406a720 2023-10-30T12:42:08.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.532+0000 7fee6affd640 1 -- 172.21.15.89:0/5375204 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee640b97c0 con 0x7fee7406a720 2023-10-30T12:42:08.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.532+0000 7fee7893e640 1 -- 172.21.15.89:0/5375204 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee741f0950 con 0x7fee7406a720 2023-10-30T12:42:08.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.532+0000 7fee7893e640 1 -- 172.21.15.89:0/5375204 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee7404e170 con 0x7fee7406a720 2023-10-30T12:42:08.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.532+0000 7fee6affd640 1 -- 172.21.15.89:0/5375204 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fee640b7040 con 0x7fee7406a720 2023-10-30T12:42:08.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.532+0000 7fee6affd640 1 --2- 172.21.15.89:0/5375204 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fee540664a0 0x7fee54068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:08.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.536+0000 7fee6affd640 1 -- 172.21.15.89:0/5375204 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6234+0+0 (secure 0 0 0) 0x7fee641219d0 con 0x7fee7406a720 2023-10-30T12:42:08.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.536+0000 7fee6bfff640 1 --2- 172.21.15.89:0/5375204 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fee540664a0 0x7fee54068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:08.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.536+0000 7fee6bfff640 1 --2- 172.21.15.89:0/5375204 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fee540664a0 0x7fee54068960 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fee7406fa00 tx=0x7fee5c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:08.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.540+0000 7fee6affd640 1 -- 172.21.15.89:0/5375204 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fee640eec30 con 0x7fee7406a720 2023-10-30T12:42:08.718 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.720+0000 7fee7893e640 1 -- 172.21.15.89:0/5375204 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee7406f750 con 0x7fee7406a720 2023-10-30T12:42:08.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.720+0000 7fee6affd640 1 -- 172.21.15.89:0/5375204 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+6529 (secure 0 0 0) 0x7fee640b5020 con 0x7fee7406a720 2023-10-30T12:42:08.719 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:42:08.719 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5575,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:40:33.262671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:08.720 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 62 2023-10-30T12:42:08.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.720+0000 7fee7893e640 1 -- 172.21.15.89:0/5375204 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fee540664a0 msgr2=0x7fee54068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:08.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.720+0000 7fee7893e640 1 --2- 172.21.15.89:0/5375204 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fee540664a0 0x7fee54068960 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fee7406fa00 tx=0x7fee5c008040 comp rx=0 tx=0).stop 2023-10-30T12:42:08.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.720+0000 7fee7893e640 1 -- 172.21.15.89:0/5375204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee7406a720 msgr2=0x7fee740c6860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:08.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.720+0000 7fee7893e640 1 --2- 172.21.15.89:0/5375204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee7406a720 0x7fee740c6860 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fee6401c880 tx=0x7fee6401b9f0 comp rx=0 tx=0).stop 2023-10-30T12:42:08.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.720+0000 7fee7893e640 1 -- 172.21.15.89:0/5375204 shutdown_connections 2023-10-30T12:42:08.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.720+0000 7fee7893e640 1 --2- 172.21.15.89:0/5375204 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fee540664a0 0x7fee54068960 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:08.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.720+0000 7fee7893e640 1 --2- 172.21.15.89:0/5375204 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee74159e60 0x7fee74140da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:08.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.720+0000 7fee7893e640 1 --2- 172.21.15.89:0/5375204 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fee740bfec0 0x7fee74140860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:08.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.720+0000 7fee7893e640 1 --2- 172.21.15.89:0/5375204 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee7406a720 0x7fee740c6860 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:08.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.720+0000 7fee7893e640 1 -- 172.21.15.89:0/5375204 >> 172.21.15.89:0/5375204 conn(0x7fee740c7910 msgr2=0x7fee740c9ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:08.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.720+0000 7fee7893e640 1 -- 172.21.15.89:0/5375204 shutdown_connections 2023-10-30T12:42:08.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.720+0000 7fee7893e640 1 -- 172.21.15.89:0/5375204 wait complete. 2023-10-30T12:42:08.739 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2023-10-30T12:42:08.740 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-30T12:42:08.740 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:42:08.761 INFO:tasks.ceph.mds.d:Stopped 2023-10-30T12:42:08.761 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:08.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.896+0000 7ffa64965640 1 Processor -- start 2023-10-30T12:42:08.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.896+0000 7ffa64965640 1 -- start start 2023-10-30T12:42:08.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.900+0000 7ffa64965640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffa600c05d0 0x7ffa6013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:08.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.900+0000 7ffa64965640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ffa600c6f60 0x7ffa6013f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:08.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.900+0000 7ffa64965640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa600c0a50 con 0x7ffa600c6b80 2023-10-30T12:42:08.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.900+0000 7ffa64965640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa6006f6e0 con 0x7ffa600c6f60 2023-10-30T12:42:08.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.900+0000 7ffa64965640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6006f9c0 con 0x7ffa600c05d0 2023-10-30T12:42:08.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.900+0000 7ffa55d9b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffa600c05d0 0x7ffa6013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:08.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.900+0000 7ffa5ed9d640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ffa600c6f60 0x7ffa6013f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:08.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.900+0000 7ffa55d9b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffa600c05d0 0x7ffa6013ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:56244/0 (socket says 172.21.15.89:56244) 2023-10-30T12:42:08.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.900+0000 7ffa55d9b640 1 -- 172.21.15.89:0/3487454428 learned_addr learned my addr 172.21.15.89:0/3487454428 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:08.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.900+0000 7ffa55d9b640 1 -- 172.21.15.89:0/3487454428 >> v2:172.21.15.192:3300/0 conn(0x7ffa600c6f60 msgr2=0x7ffa6013f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:08.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.900+0000 7ffa55d9b640 1 --2- 172.21.15.89:0/3487454428 >> v2:172.21.15.192:3300/0 conn(0x7ffa600c6f60 0x7ffa6013f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:08.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.900+0000 7ffa55d9b640 1 -- 172.21.15.89:0/3487454428 >> v1:172.21.15.192:6789/0 conn(0x7ffa600c6b80 legacy=0x7ffa600bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:08.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.900+0000 7ffa55d9b640 1 -- 172.21.15.89:0/3487454428 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa6006fca0 con 0x7ffa600c05d0 2023-10-30T12:42:08.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.904+0000 7ffa55d9b640 1 --2- 172.21.15.89:0/3487454428 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffa600c05d0 0x7ffa6013ed90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ffa48009e50 tx=0x7ffa4801ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45675c177401f315 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:08.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.904+0000 7ffa5dd9b640 1 -- 172.21.15.89:0/3487454428 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa480a57f0 con 0x7ffa600c05d0 2023-10-30T12:42:08.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.904+0000 7ffa5dd9b640 1 -- 172.21.15.89:0/3487454428 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa480a5950 con 0x7ffa600c05d0 2023-10-30T12:42:08.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.904+0000 7ffa5dd9b640 1 -- 172.21.15.89:0/3487454428 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa480a5ce0 con 0x7ffa600c05d0 2023-10-30T12:42:08.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.904+0000 7ffa64965640 1 -- 172.21.15.89:0/3487454428 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffa600c05d0 msgr2=0x7ffa6013ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:08.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.904+0000 7ffa64965640 1 --2- 172.21.15.89:0/3487454428 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffa600c05d0 0x7ffa6013ed90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ffa48009e50 tx=0x7ffa4801ca10 comp rx=0 tx=0).stop 2023-10-30T12:42:08.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.904+0000 7ffa64965640 1 -- 172.21.15.89:0/3487454428 shutdown_connections 2023-10-30T12:42:08.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.904+0000 7ffa64965640 1 --2- 172.21.15.89:0/3487454428 >> v2:172.21.15.192:3300/0 conn(0x7ffa600c6f60 0x7ffa6013f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:08.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.904+0000 7ffa64965640 1 --2- 172.21.15.89:0/3487454428 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffa600c05d0 0x7ffa6013ed90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:08.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.904+0000 7ffa64965640 1 -- 172.21.15.89:0/3487454428 >> 172.21.15.89:0/3487454428 conn(0x7ffa600c7910 msgr2=0x7ffa600c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:08.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.904+0000 7ffa64965640 1 -- 172.21.15.89:0/3487454428 shutdown_connections 2023-10-30T12:42:08.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.904+0000 7ffa64965640 1 -- 172.21.15.89:0/3487454428 wait complete. 2023-10-30T12:42:08.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.908+0000 7ffa64965640 1 Processor -- start 2023-10-30T12:42:08.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.908+0000 7ffa64965640 1 -- start start 2023-10-30T12:42:08.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.908+0000 7ffa64965640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffa600c05d0 0x7ffa600c9130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:08.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.908+0000 7ffa64965640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa600c6b80 0x7ffa60131670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:08.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.908+0000 7ffa64965640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa600c6f60 0x7ffa60131bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:08.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.908+0000 7ffa64965640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa600c97e0 con 0x7ffa600c6f60 2023-10-30T12:42:08.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.908+0000 7ffa64965640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa60158620 con 0x7ffa600c6b80 2023-10-30T12:42:08.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.908+0000 7ffa64965640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa60158aa0 con 0x7ffa600c05d0 2023-10-30T12:42:08.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.908+0000 7ffa5ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa600c6f60 0x7ffa60131bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:08.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.908+0000 7ffa5ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa600c6f60 0x7ffa60131bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47102/0 (socket says 172.21.15.89:47102) 2023-10-30T12:42:08.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.908+0000 7ffa5ed9d640 1 -- 172.21.15.89:0/3001702745 learned_addr learned my addr 172.21.15.89:0/3001702745 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:08.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.908+0000 7ffa5e59c640 1 --2- 172.21.15.89:0/3001702745 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffa600c05d0 0x7ffa600c9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:08.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.908+0000 7ffa55d9b640 1 --2- 172.21.15.89:0/3001702745 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa600c6b80 0x7ffa60131670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:08.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.908+0000 7ffa5ed9d640 1 -- 172.21.15.89:0/3001702745 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffa600c05d0 msgr2=0x7ffa600c9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:08.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.908+0000 7ffa5ed9d640 1 --2- 172.21.15.89:0/3001702745 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffa600c05d0 0x7ffa600c9130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:08.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.908+0000 7ffa5ed9d640 1 -- 172.21.15.89:0/3001702745 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa600c6b80 msgr2=0x7ffa60131670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:08.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.908+0000 7ffa5ed9d640 1 --2- 172.21.15.89:0/3001702745 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa600c6b80 0x7ffa60131670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:08.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.908+0000 7ffa5ed9d640 1 -- 172.21.15.89:0/3001702745 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa48009b30 con 0x7ffa600c6f60 2023-10-30T12:42:08.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.908+0000 7ffa5ed9d640 1 --2- 172.21.15.89:0/3001702745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa600c6f60 0x7ffa60131bb0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ffa5800e7f0 tx=0x7ffa5800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:08.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.908+0000 7ffa577fe640 1 -- 172.21.15.89:0/3001702745 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa58009530 con 0x7ffa600c6f60 2023-10-30T12:42:08.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.908+0000 7ffa577fe640 1 -- 172.21.15.89:0/3001702745 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa5800eea0 con 0x7ffa600c6f60 2023-10-30T12:42:08.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.908+0000 7ffa64965640 1 -- 172.21.15.89:0/3001702745 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa60158d80 con 0x7ffa600c6f60 2023-10-30T12:42:08.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.908+0000 7ffa577fe640 1 -- 172.21.15.89:0/3001702745 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5800cb00 con 0x7ffa600c6f60 2023-10-30T12:42:08.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.908+0000 7ffa64965640 1 -- 172.21.15.89:0/3001702745 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa60159340 con 0x7ffa600c6f60 2023-10-30T12:42:08.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.908+0000 7ffa64965640 1 -- 172.21.15.89:0/3001702745 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa6004e170 con 0x7ffa600c6f60 2023-10-30T12:42:08.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.912+0000 7ffa577fe640 1 -- 172.21.15.89:0/3001702745 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffa5800cc60 con 0x7ffa600c6f60 2023-10-30T12:42:08.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.912+0000 7ffa577fe640 1 --2- 172.21.15.89:0/3001702745 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ffa400664a0 0x7ffa40068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:08.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.912+0000 7ffa577fe640 1 -- 172.21.15.89:0/3001702745 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6234+0+0 (secure 0 0 0) 0x7ffa58084130 con 0x7ffa600c6f60 2023-10-30T12:42:08.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.912+0000 7ffa5e59c640 1 --2- 172.21.15.89:0/3001702745 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ffa400664a0 0x7ffa40068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:08.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.912+0000 7ffa5e59c640 1 --2- 172.21.15.89:0/3001702745 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ffa400664a0 0x7ffa40068960 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ffa50007fd0 tx=0x7ffa50009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:08.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:08.916+0000 7ffa577fe640 1 -- 172.21.15.89:0/3001702745 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffa58051320 con 0x7ffa600c6f60 2023-10-30T12:42:09.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:09.104+0000 7ffa64965640 1 -- 172.21.15.89:0/3001702745 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa60157970 con 0x7ffa600c6f60 2023-10-30T12:42:09.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:09.104+0000 7ffa577fe640 1 -- 172.21.15.89:0/3001702745 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+6529 (secure 0 0 0) 0x7ffa58054940 con 0x7ffa600c6f60 2023-10-30T12:42:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:42:09.106 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5575,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:40:33.262671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:09.107 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 62 2023-10-30T12:42:09.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:09.108+0000 7ffa64965640 1 -- 172.21.15.89:0/3001702745 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ffa400664a0 msgr2=0x7ffa40068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:09.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:09.108+0000 7ffa64965640 1 --2- 172.21.15.89:0/3001702745 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ffa400664a0 0x7ffa40068960 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ffa50007fd0 tx=0x7ffa50009040 comp rx=0 tx=0).stop 2023-10-30T12:42:09.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:09.108+0000 7ffa64965640 1 -- 172.21.15.89:0/3001702745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa600c6f60 msgr2=0x7ffa60131bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:09.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:09.108+0000 7ffa64965640 1 --2- 172.21.15.89:0/3001702745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa600c6f60 0x7ffa60131bb0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ffa5800e7f0 tx=0x7ffa5800ecb0 comp rx=0 tx=0).stop 2023-10-30T12:42:09.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:09.108+0000 7ffa64965640 1 -- 172.21.15.89:0/3001702745 shutdown_connections 2023-10-30T12:42:09.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:09.108+0000 7ffa64965640 1 --2- 172.21.15.89:0/3001702745 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ffa400664a0 0x7ffa40068960 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:09.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:09.108+0000 7ffa64965640 1 --2- 172.21.15.89:0/3001702745 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa600c6b80 0x7ffa60131670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:09.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:09.108+0000 7ffa64965640 1 --2- 172.21.15.89:0/3001702745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa600c6f60 0x7ffa60131bb0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:09.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:09.108+0000 7ffa64965640 1 --2- 172.21.15.89:0/3001702745 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffa600c05d0 0x7ffa600c9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:09.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:09.108+0000 7ffa64965640 1 -- 172.21.15.89:0/3001702745 >> 172.21.15.89:0/3001702745 conn(0x7ffa600c7910 msgr2=0x7ffa600bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:09.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:09.108+0000 7ffa64965640 1 -- 172.21.15.89:0/3001702745 shutdown_connections 2023-10-30T12:42:09.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:09.108+0000 7ffa64965640 1 -- 172.21.15.89:0/3001702745 wait complete. 2023-10-30T12:42:09.125 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T12:42:11.126 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:11.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.276+0000 7f23aac91640 1 Processor -- start 2023-10-30T12:42:11.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.276+0000 7f23aac91640 1 -- start start 2023-10-30T12:42:11.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.276+0000 7f23aac91640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f23a41538a0 0x7f23a415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:11.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.276+0000 7f23aac91640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a40cb800 con 0x7f23a40ca0e0 2023-10-30T12:42:11.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.276+0000 7f23aac91640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a415dc30 con 0x7f23a40cf160 2023-10-30T12:42:11.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.276+0000 7f23aac91640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23a40cb330 con 0x7f23a41538a0 2023-10-30T12:42:11.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.276+0000 7f239bfff640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f23a40ca0e0 0x7f23a415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57910/0 (socket says 172.21.15.89:57910) 2023-10-30T12:42:11.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.276+0000 7f239bfff640 1 -- 172.21.15.89:0/1027281011 learned_addr learned my addr 172.21.15.89:0/1027281011 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:42:11.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.276+0000 7f23a8a2d640 1 --2- 172.21.15.89:0/1027281011 >> v2:172.21.15.192:3300/0 conn(0x7f23a41538a0 0x7f23a415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:11.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.276+0000 7f239b7fe640 1 -- 172.21.15.89:0/1027281011 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2141296897 0 0) 0x7f23a40cb800 con 0x7f23a40ca0e0 2023-10-30T12:42:11.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.276+0000 7f239b7fe640 1 -- 172.21.15.89:0/1027281011 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f238c003530 con 0x7f23a40ca0e0 2023-10-30T12:42:11.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.276+0000 7f239b7fe640 1 -- 172.21.15.89:0/1027281011 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3080835272 0 0) 0x7f239c0014a0 con 0x7f23a40cf160 2023-10-30T12:42:11.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.276+0000 7f239b7fe640 1 -- 172.21.15.89:0/1027281011 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23a40cb800 con 0x7f23a40cf160 2023-10-30T12:42:11.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.276+0000 7f239b7fe640 1 -- 172.21.15.89:0/1027281011 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2166860390 0 0) 0x7f238c003530 con 0x7f23a40ca0e0 2023-10-30T12:42:11.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.276+0000 7f239b7fe640 1 -- 172.21.15.89:0/1027281011 >> v2:172.21.15.192:3300/0 conn(0x7f23a41538a0 msgr2=0x7f23a415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:11.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.276+0000 7f239b7fe640 1 --2- 172.21.15.89:0/1027281011 >> v2:172.21.15.192:3300/0 conn(0x7f23a41538a0 0x7f23a415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:11.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.276+0000 7f239b7fe640 1 -- 172.21.15.89:0/1027281011 >> v1:172.21.15.192:6789/0 conn(0x7f23a40cf160 legacy=0x7f23a4164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:11.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.276+0000 7f239b7fe640 1 -- 172.21.15.89:0/1027281011 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a40cad70 con 0x7f23a40ca0e0 2023-10-30T12:42:11.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.276+0000 7f239b7fe640 1 -- 172.21.15.89:0/1027281011 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f2394001680 con 0x7f23a40ca0e0 2023-10-30T12:42:11.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.276+0000 7f239b7fe640 1 -- 172.21.15.89:0/1027281011 >> v1:172.21.15.89:6789/0 conn(0x7f23a40ca0e0 legacy=0x7f23a415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:11.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.276+0000 7f239b7fe640 1 --2- 172.21.15.89:0/1027281011 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f238c003b60 0x7f238c003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:11.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.276+0000 7f239b7fe640 1 --2- 172.21.15.89:0/1027281011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f238c0047d0 0x7f238c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:11.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.276+0000 7f239b7fe640 1 --2- 172.21.15.89:0/1027281011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f238c005190 0x7f238c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:11.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.276+0000 7f239b7fe640 1 -- 172.21.15.89:0/1027281011 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a40cb330 con 0x7f238c0047d0 2023-10-30T12:42:11.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.276+0000 7f239b7fe640 1 -- 172.21.15.89:0/1027281011 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238c00bfb0 con 0x7f238c005190 2023-10-30T12:42:11.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.276+0000 7f239b7fe640 1 -- 172.21.15.89:0/1027281011 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f238c00c170 con 0x7f238c003b60 2023-10-30T12:42:11.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.276+0000 7f239bfff640 1 --2- 172.21.15.89:0/1027281011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f238c0047d0 0x7f238c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:11.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.280+0000 7f23a8a2d640 1 --2- 172.21.15.89:0/1027281011 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f238c003b60 0x7f238c003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:11.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.280+0000 7f23a922e640 1 --2- 172.21.15.89:0/1027281011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f238c005190 0x7f238c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:11.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.280+0000 7f239bfff640 1 -- 172.21.15.89:0/1027281011 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f238c003b60 msgr2=0x7f238c003f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:11.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.280+0000 7f239bfff640 1 --2- 172.21.15.89:0/1027281011 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f238c003b60 0x7f238c003f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:11.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.280+0000 7f239bfff640 1 -- 172.21.15.89:0/1027281011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f238c005190 msgr2=0x7f238c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:11.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.280+0000 7f239bfff640 1 --2- 172.21.15.89:0/1027281011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f238c005190 0x7f238c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:11.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.280+0000 7f239bfff640 1 -- 172.21.15.89:0/1027281011 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f238c003530 con 0x7f238c0047d0 2023-10-30T12:42:11.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.280+0000 7f23a922e640 1 --2- 172.21.15.89:0/1027281011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f238c005190 0x7f238c00b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:42:11.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.280+0000 7f23a922e640 1 -- 172.21.15.89:0/1027281011 reap_dead start 2023-10-30T12:42:11.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.280+0000 7f239bfff640 1 --2- 172.21.15.89:0/1027281011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f238c0047d0 0x7f238c004c50 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f2394007fb0 tx=0x7f239401f230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71eee38a2d7f2100 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:11.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.280+0000 7f239b7fe640 1 -- 172.21.15.89:0/1027281011 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23940a8e50 con 0x7f238c0047d0 2023-10-30T12:42:11.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.280+0000 7f239b7fe640 1 -- 172.21.15.89:0/1027281011 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23940058e0 con 0x7f238c0047d0 2023-10-30T12:42:11.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.280+0000 7f239b7fe640 1 -- 172.21.15.89:0/1027281011 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2394005cd0 con 0x7f238c0047d0 2023-10-30T12:42:11.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.280+0000 7f23aac91640 1 -- 172.21.15.89:0/1027281011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f238c0047d0 msgr2=0x7f238c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:11.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.284+0000 7f23aac91640 1 --2- 172.21.15.89:0/1027281011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f238c0047d0 0x7f238c004c50 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f2394007fb0 tx=0x7f239401f230 comp rx=0 tx=0).stop 2023-10-30T12:42:11.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.284+0000 7f23aac91640 1 -- 172.21.15.89:0/1027281011 shutdown_connections 2023-10-30T12:42:11.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.284+0000 7f23aac91640 1 --2- 172.21.15.89:0/1027281011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f238c0047d0 0x7f238c004c50 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:11.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.284+0000 7f23aac91640 1 -- 172.21.15.89:0/1027281011 >> 172.21.15.89:0/1027281011 conn(0x7f23a406bfe0 msgr2=0x7f23a406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:11.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.284+0000 7f23aac91640 1 -- 172.21.15.89:0/1027281011 shutdown_connections 2023-10-30T12:42:11.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.284+0000 7f23aac91640 1 -- 172.21.15.89:0/1027281011 wait complete. 2023-10-30T12:42:11.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.284+0000 7f23aac91640 1 Processor -- start 2023-10-30T12:42:11.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.284+0000 7f23aac91640 1 -- start start 2023-10-30T12:42:11.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.284+0000 7f23aac91640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f238c0047d0 0x7f23a41f29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:11.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.284+0000 7f23aac91640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23a40b4ce0 0x7f23a4158000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:11.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.284+0000 7f23aac91640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23a414d690 0x7f23a4158540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:11.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.284+0000 7f23aac91640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a40b41c0 con 0x7f23a414d690 2023-10-30T12:42:11.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.284+0000 7f23aac91640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a40b3560 con 0x7f238c0047d0 2023-10-30T12:42:11.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.284+0000 7f23aac91640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a40b3840 con 0x7f23a40b4ce0 2023-10-30T12:42:11.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.284+0000 7f23a922e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23a414d690 0x7f23a4158540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:11.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.284+0000 7f23a922e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23a414d690 0x7f23a4158540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35090/0 (socket says 172.21.15.89:35090) 2023-10-30T12:42:11.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.284+0000 7f23a922e640 1 -- 172.21.15.89:0/3796455909 learned_addr learned my addr 172.21.15.89:0/3796455909 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:11.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.284+0000 7f239bfff640 1 --2- 172.21.15.89:0/3796455909 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23a40b4ce0 0x7f23a4158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:11.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.284+0000 7f23a8a2d640 1 --2- 172.21.15.89:0/3796455909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f238c0047d0 0x7f23a41f29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:11.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.284+0000 7f23a922e640 1 -- 172.21.15.89:0/3796455909 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23a40b4ce0 msgr2=0x7f23a4158000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:11.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.284+0000 7f23a922e640 1 --2- 172.21.15.89:0/3796455909 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23a40b4ce0 0x7f23a4158000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:11.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.284+0000 7f23a922e640 1 -- 172.21.15.89:0/3796455909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f238c0047d0 msgr2=0x7f23a41f29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:11.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.284+0000 7f23a922e640 1 --2- 172.21.15.89:0/3796455909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f238c0047d0 0x7f23a41f29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:11.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.284+0000 7f23a922e640 1 -- 172.21.15.89:0/3796455909 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a40b3de0 con 0x7f23a414d690 2023-10-30T12:42:11.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.284+0000 7f23a8a2d640 1 --2- 172.21.15.89:0/3796455909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f238c0047d0 0x7f23a41f29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:42:11.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.284+0000 7f23a922e640 1 --2- 172.21.15.89:0/3796455909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23a414d690 0x7f23a4158540 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f23a4072ce0 tx=0x7f239c00eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:11.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.288+0000 7f23997fa640 1 -- 172.21.15.89:0/3796455909 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239c006b60 con 0x7f23a414d690 2023-10-30T12:42:11.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.288+0000 7f23997fa640 1 -- 172.21.15.89:0/3796455909 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239c006d00 con 0x7f23a414d690 2023-10-30T12:42:11.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.288+0000 7f23997fa640 1 -- 172.21.15.89:0/3796455909 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239c003bc0 con 0x7f23a414d690 2023-10-30T12:42:11.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.288+0000 7f23aac91640 1 -- 172.21.15.89:0/3796455909 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23a41f3460 con 0x7f23a414d690 2023-10-30T12:42:11.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.288+0000 7f23aac91640 1 -- 172.21.15.89:0/3796455909 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a4066080 con 0x7f23a414d690 2023-10-30T12:42:11.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.288+0000 7f23aac91640 1 -- 172.21.15.89:0/3796455909 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a40b3c40 con 0x7f23a414d690 2023-10-30T12:42:11.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.288+0000 7f23997fa640 1 -- 172.21.15.89:0/3796455909 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f239c003d20 con 0x7f23a414d690 2023-10-30T12:42:11.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.288+0000 7f23997fa640 1 --2- 172.21.15.89:0/3796455909 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23900664a0 0x7f2390068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:11.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.288+0000 7f23997fa640 1 -- 172.21.15.89:0/3796455909 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6234+0+0 (secure 0 0 0) 0x7f239c083900 con 0x7f23a414d690 2023-10-30T12:42:11.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.288+0000 7f23a8a2d640 1 --2- 172.21.15.89:0/3796455909 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23900664a0 0x7f2390068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:11.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.292+0000 7f23a8a2d640 1 --2- 172.21.15.89:0/3796455909 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23900664a0 0x7f2390068960 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f23a0009b80 tx=0x7f23a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:11.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.292+0000 7f23997fa640 1 -- 172.21.15.89:0/3796455909 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f23a40b3c40 con 0x7f23a414d690 2023-10-30T12:42:11.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.476+0000 7f23aac91640 1 -- 172.21.15.89:0/3796455909 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23a415dc30 con 0x7f23a414d690 2023-10-30T12:42:11.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.480+0000 7f23997fa640 1 -- 172.21.15.89:0/3796455909 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+6529 (secure 0 0 0) 0x7f23a415dc30 con 0x7f23a414d690 2023-10-30T12:42:11.478 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:42:11.479 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5575,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:40:33.262671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:11.480 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 62 2023-10-30T12:42:11.480 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.480+0000 7f23aac91640 1 -- 172.21.15.89:0/3796455909 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23900664a0 msgr2=0x7f2390068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:11.480 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.480+0000 7f23aac91640 1 --2- 172.21.15.89:0/3796455909 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23900664a0 0x7f2390068960 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f23a0009b80 tx=0x7f23a0008040 comp rx=0 tx=0).stop 2023-10-30T12:42:11.480 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.480+0000 7f23aac91640 1 -- 172.21.15.89:0/3796455909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23a414d690 msgr2=0x7f23a4158540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:11.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.480+0000 7f23aac91640 1 --2- 172.21.15.89:0/3796455909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23a414d690 0x7f23a4158540 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f23a4072ce0 tx=0x7f239c00eb90 comp rx=0 tx=0).stop 2023-10-30T12:42:11.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.480+0000 7f23aac91640 1 -- 172.21.15.89:0/3796455909 shutdown_connections 2023-10-30T12:42:11.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.480+0000 7f23aac91640 1 --2- 172.21.15.89:0/3796455909 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23900664a0 0x7f2390068960 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:11.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.480+0000 7f23aac91640 1 --2- 172.21.15.89:0/3796455909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23a414d690 0x7f23a4158540 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:11.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.480+0000 7f23aac91640 1 --2- 172.21.15.89:0/3796455909 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23a40b4ce0 0x7f23a4158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:11.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.480+0000 7f23aac91640 1 --2- 172.21.15.89:0/3796455909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f238c0047d0 0x7f23a41f29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:11.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.480+0000 7f23aac91640 1 -- 172.21.15.89:0/3796455909 >> 172.21.15.89:0/3796455909 conn(0x7f23a406bfe0 msgr2=0x7f23a406c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:11.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.480+0000 7f23aac91640 1 -- 172.21.15.89:0/3796455909 shutdown_connections 2023-10-30T12:42:11.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:11.480+0000 7f23aac91640 1 -- 172.21.15.89:0/3796455909 wait complete. 2023-10-30T12:42:11.499 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T12:42:13.500 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:13.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.640+0000 7f217c13e640 1 Processor -- start 2023-10-30T12:42:13.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.640+0000 7f217c13e640 1 -- start start 2023-10-30T12:42:13.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.644+0000 7f217c13e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f21740c6b80 0x7f21740bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:13.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.644+0000 7f217c13e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f217413f4a0 0x7f21740c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:13.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.644+0000 7f217c13e640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21740c0a50 con 0x7f21740c6f60 2023-10-30T12:42:13.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.644+0000 7f217c13e640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f217406f6e0 con 0x7f217413f4a0 2023-10-30T12:42:13.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.644+0000 7f217c13e640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f217406f9c0 con 0x7f21740c6b80 2023-10-30T12:42:13.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.644+0000 7f217a6db640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f217413f4a0 0x7f21740c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:13.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.644+0000 7f21796d9640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f21740c6f60 0x7f217413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57912/0 (socket says 172.21.15.89:57912) 2023-10-30T12:42:13.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.644+0000 7f21796d9640 1 -- 172.21.15.89:0/3904257257 learned_addr learned my addr 172.21.15.89:0/3904257257 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:42:13.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.644+0000 7f2179eda640 1 --2- 172.21.15.89:0/3904257257 >> v2:172.21.15.192:3300/0 conn(0x7f21740c6b80 0x7f21740bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:13.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.644+0000 7f217a6db640 1 -- 172.21.15.89:0/3904257257 >> v2:172.21.15.192:3300/0 conn(0x7f21740c6b80 msgr2=0x7f21740bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:13.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.644+0000 7f217a6db640 1 --2- 172.21.15.89:0/3904257257 >> v2:172.21.15.192:3300/0 conn(0x7f21740c6b80 0x7f21740bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:13.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.644+0000 7f217a6db640 1 -- 172.21.15.89:0/3904257257 >> v1:172.21.15.89:6789/0 conn(0x7f21740c6f60 legacy=0x7f217413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:13.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.644+0000 7f217a6db640 1 -- 172.21.15.89:0/3904257257 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f217406fca0 con 0x7f217413f4a0 2023-10-30T12:42:13.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.648+0000 7f217a6db640 1 --2- 172.21.15.89:0/3904257257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f217413f4a0 0x7f21740c0400 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f217406f1b0 tx=0x7f216c01eb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53b034d95bfbefd0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:13.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.648+0000 7f2178ed8640 1 -- 172.21.15.89:0/3904257257 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216c00b2d0 con 0x7f217413f4a0 2023-10-30T12:42:13.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.648+0000 7f2178ed8640 1 -- 172.21.15.89:0/3904257257 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f216c0a78f0 con 0x7f217413f4a0 2023-10-30T12:42:13.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.648+0000 7f2178ed8640 1 -- 172.21.15.89:0/3904257257 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216c0a7c80 con 0x7f217413f4a0 2023-10-30T12:42:13.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.648+0000 7f217c13e640 1 -- 172.21.15.89:0/3904257257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f217413f4a0 msgr2=0x7f21740c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:13.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.648+0000 7f217c13e640 1 --2- 172.21.15.89:0/3904257257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f217413f4a0 0x7f21740c0400 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f217406f1b0 tx=0x7f216c01eb10 comp rx=0 tx=0).stop 2023-10-30T12:42:13.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.648+0000 7f217c13e640 1 -- 172.21.15.89:0/3904257257 shutdown_connections 2023-10-30T12:42:13.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.648+0000 7f217c13e640 1 --2- 172.21.15.89:0/3904257257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f217413f4a0 0x7f21740c0400 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:13.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.648+0000 7f217c13e640 1 --2- 172.21.15.89:0/3904257257 >> v2:172.21.15.192:3300/0 conn(0x7f21740c6b80 0x7f21740bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:13.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.648+0000 7f217c13e640 1 -- 172.21.15.89:0/3904257257 >> 172.21.15.89:0/3904257257 conn(0x7f21740c7910 msgr2=0x7f21740c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:13.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.648+0000 7f217c13e640 1 -- 172.21.15.89:0/3904257257 shutdown_connections 2023-10-30T12:42:13.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.648+0000 7f217c13e640 1 -- 172.21.15.89:0/3904257257 wait complete. 2023-10-30T12:42:13.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.648+0000 7f217c13e640 1 Processor -- start 2023-10-30T12:42:13.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.648+0000 7f217c13e640 1 -- start start 2023-10-30T12:42:13.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.648+0000 7f217c13e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f21740c6b80 0x7f2174132dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:13.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.648+0000 7f217c13e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21740c6f60 0x7f2174133300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:13.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.648+0000 7f217c13e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f217413f4a0 0x7f217414df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:13.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.648+0000 7f217c13e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2174153690 con 0x7f217413f4a0 2023-10-30T12:42:13.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.648+0000 7f217c13e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21741538e0 con 0x7f21740c6f60 2023-10-30T12:42:13.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.648+0000 7f217c13e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2174153bc0 con 0x7f21740c6b80 2023-10-30T12:42:13.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.648+0000 7f217a6db640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f217413f4a0 0x7f217414df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:13.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.648+0000 7f217a6db640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f217413f4a0 0x7f217414df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35112/0 (socket says 172.21.15.89:35112) 2023-10-30T12:42:13.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.648+0000 7f217a6db640 1 -- 172.21.15.89:0/2783403171 learned_addr learned my addr 172.21.15.89:0/2783403171 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:13.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.652+0000 7f21796d9640 1 --2- 172.21.15.89:0/2783403171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21740c6f60 0x7f2174133300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:13.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.652+0000 7f2179eda640 1 --2- 172.21.15.89:0/2783403171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f21740c6b80 0x7f2174132dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:13.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.652+0000 7f217a6db640 1 -- 172.21.15.89:0/2783403171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f21740c6b80 msgr2=0x7f2174132dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:13.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.652+0000 7f217a6db640 1 --2- 172.21.15.89:0/2783403171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f21740c6b80 0x7f2174132dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:13.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.652+0000 7f217a6db640 1 -- 172.21.15.89:0/2783403171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21740c6f60 msgr2=0x7f2174133300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:13.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.652+0000 7f217a6db640 1 --2- 172.21.15.89:0/2783403171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21740c6f60 0x7f2174133300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:13.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.652+0000 7f217a6db640 1 -- 172.21.15.89:0/2783403171 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f216c00bc80 con 0x7f217413f4a0 2023-10-30T12:42:13.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.652+0000 7f21796d9640 1 --2- 172.21.15.89:0/2783403171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21740c6f60 0x7f2174133300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:42:13.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.652+0000 7f217a6db640 1 --2- 172.21.15.89:0/2783403171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f217413f4a0 0x7f217414df10 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f216c01b720 tx=0x7f216c01b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:13.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.652+0000 7f216a7fc640 1 -- 172.21.15.89:0/2783403171 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216c009950 con 0x7f217413f4a0 2023-10-30T12:42:13.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.652+0000 7f216a7fc640 1 -- 172.21.15.89:0/2783403171 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f216c009ad0 con 0x7f217413f4a0 2023-10-30T12:42:13.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.652+0000 7f216a7fc640 1 -- 172.21.15.89:0/2783403171 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216c009e80 con 0x7f217413f4a0 2023-10-30T12:42:13.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.652+0000 7f217c13e640 1 -- 172.21.15.89:0/2783403171 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2174153ea0 con 0x7f217413f4a0 2023-10-30T12:42:13.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.652+0000 7f217c13e640 1 -- 172.21.15.89:0/2783403171 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21741328a0 con 0x7f217413f4a0 2023-10-30T12:42:13.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.652+0000 7f217c13e640 1 -- 172.21.15.89:0/2783403171 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21741540c0 con 0x7f217413f4a0 2023-10-30T12:42:13.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.652+0000 7f216a7fc640 1 -- 172.21.15.89:0/2783403171 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f216c0ba470 con 0x7f217413f4a0 2023-10-30T12:42:13.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.652+0000 7f216a7fc640 1 --2- 172.21.15.89:0/2783403171 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f21580664a0 0x7f2158068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:13.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.652+0000 7f216a7fc640 1 -- 172.21.15.89:0/2783403171 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6234+0+0 (secure 0 0 0) 0x7f216c121b50 con 0x7f217413f4a0 2023-10-30T12:42:13.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.656+0000 7f2179eda640 1 --2- 172.21.15.89:0/2783403171 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f21580664a0 0x7f2158068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:13.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.656+0000 7f2179eda640 1 --2- 172.21.15.89:0/2783403171 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f21580664a0 0x7f2158068960 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f217000b9d0 tx=0x7f2170008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:13.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.656+0000 7f216a7fc640 1 -- 172.21.15.89:0/2783403171 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f216c0eee40 con 0x7f217413f4a0 2023-10-30T12:42:13.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.836+0000 7f217c13e640 1 -- 172.21.15.89:0/2783403171 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2174132a90 con 0x7f217413f4a0 2023-10-30T12:42:13.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.836+0000 7f216a7fc640 1 -- 172.21.15.89:0/2783403171 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+6529 (secure 0 0 0) 0x7f216c0f2460 con 0x7f217413f4a0 2023-10-30T12:42:13.837 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:42:13.837 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5575,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:40:33.262671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:13.838 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 62 2023-10-30T12:42:13.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.840+0000 7f217c13e640 1 -- 172.21.15.89:0/2783403171 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f21580664a0 msgr2=0x7f2158068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:13.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.840+0000 7f217c13e640 1 --2- 172.21.15.89:0/2783403171 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f21580664a0 0x7f2158068960 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f217000b9d0 tx=0x7f2170008040 comp rx=0 tx=0).stop 2023-10-30T12:42:13.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.840+0000 7f217c13e640 1 -- 172.21.15.89:0/2783403171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f217413f4a0 msgr2=0x7f217414df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:13.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.840+0000 7f217c13e640 1 --2- 172.21.15.89:0/2783403171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f217413f4a0 0x7f217414df10 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f216c01b720 tx=0x7f216c01b8b0 comp rx=0 tx=0).stop 2023-10-30T12:42:13.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.840+0000 7f217c13e640 1 -- 172.21.15.89:0/2783403171 shutdown_connections 2023-10-30T12:42:13.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.840+0000 7f217c13e640 1 --2- 172.21.15.89:0/2783403171 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f21580664a0 0x7f2158068960 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:13.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.840+0000 7f217c13e640 1 --2- 172.21.15.89:0/2783403171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21740c6f60 0x7f2174133300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:13.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.840+0000 7f217c13e640 1 --2- 172.21.15.89:0/2783403171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f217413f4a0 0x7f217414df10 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:13.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.840+0000 7f217c13e640 1 --2- 172.21.15.89:0/2783403171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f21740c6b80 0x7f2174132dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:13.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.840+0000 7f217c13e640 1 -- 172.21.15.89:0/2783403171 >> 172.21.15.89:0/2783403171 conn(0x7f21740c7910 msgr2=0x7f21740c1340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:13.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.840+0000 7f217c13e640 1 -- 172.21.15.89:0/2783403171 shutdown_connections 2023-10-30T12:42:13.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:13.840+0000 7f217c13e640 1 -- 172.21.15.89:0/2783403171 wait complete. 2023-10-30T12:42:13.857 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T12:42:15.858 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:16.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42b0bd0640 1 Processor -- start 2023-10-30T12:42:16.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42b0bd0640 1 -- start start 2023-10-30T12:42:16.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42b0bd0640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f42ac1538f0 0x7f42ac159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:16.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42b0bd0640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42ac15cd50 0x7f42ac0c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:16.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42b0bd0640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42ac15d300 con 0x7f42ac0cefa0 2023-10-30T12:42:16.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42b0bd0640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42ac15d5e0 con 0x7f42ac1538f0 2023-10-30T12:42:16.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42b0bd0640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42ac15d8c0 con 0x7f42ac15cd50 2023-10-30T12:42:16.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42abfff640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f42ac0cefa0 0x7f42ac0cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57924/0 (socket says 172.21.15.89:57924) 2023-10-30T12:42:16.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42abfff640 1 -- 172.21.15.89:0/1700329783 learned_addr learned my addr 172.21.15.89:0/1700329783 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:42:16.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42ab7fe640 1 --2- 172.21.15.89:0/1700329783 >> v2:172.21.15.192:3300/0 conn(0x7f42ac1538f0 0x7f42ac159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:16.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42aaffd640 1 --2- 172.21.15.89:0/1700329783 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42ac15cd50 0x7f42ac0c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:16.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42aa7fc640 1 -- 172.21.15.89:0/1700329783 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2449890202 0 0) 0x7f42ac15d300 con 0x7f42ac0cefa0 2023-10-30T12:42:16.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42aa7fc640 1 -- 172.21.15.89:0/1700329783 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4294003530 con 0x7f42ac0cefa0 2023-10-30T12:42:16.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42aa7fc640 1 -- 172.21.15.89:0/1700329783 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 178742830 0 0) 0x7f4294003530 con 0x7f42ac0cefa0 2023-10-30T12:42:16.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42aa7fc640 1 -- 172.21.15.89:0/1700329783 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42ac15cd50 msgr2=0x7f42ac0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:16.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42aa7fc640 1 --2- 172.21.15.89:0/1700329783 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42ac15cd50 0x7f42ac0c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:16.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42aa7fc640 1 -- 172.21.15.89:0/1700329783 >> v2:172.21.15.192:3300/0 conn(0x7f42ac1538f0 msgr2=0x7f42ac159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:16.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42aa7fc640 1 --2- 172.21.15.89:0/1700329783 >> v2:172.21.15.192:3300/0 conn(0x7f42ac1538f0 0x7f42ac159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:16.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42aa7fc640 1 -- 172.21.15.89:0/1700329783 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ac15dba0 con 0x7f42ac0cefa0 2023-10-30T12:42:16.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42aa7fc640 1 -- 172.21.15.89:0/1700329783 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f42a4001a00 con 0x7f42ac0cefa0 2023-10-30T12:42:16.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42aa7fc640 1 -- 172.21.15.89:0/1700329783 >> v1:172.21.15.89:6789/0 conn(0x7f42ac0cefa0 legacy=0x7f42ac0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:16.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42aa7fc640 1 --2- 172.21.15.89:0/1700329783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4294003980 0x7f4294003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:16.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42aa7fc640 1 --2- 172.21.15.89:0/1700329783 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42940047d0 0x7f4294004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:16.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42aa7fc640 1 --2- 172.21.15.89:0/1700329783 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42ac15cd50 0x7f4294005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:16.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42aa7fc640 1 -- 172.21.15.89:0/1700329783 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ac15d5e0 con 0x7f4294003980 2023-10-30T12:42:16.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42aa7fc640 1 -- 172.21.15.89:0/1700329783 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ac15d8c0 con 0x7f42940047d0 2023-10-30T12:42:16.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42aa7fc640 1 -- 172.21.15.89:0/1700329783 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f429400b9c0 con 0x7f42ac15cd50 2023-10-30T12:42:16.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42ab7fe640 1 --2- 172.21.15.89:0/1700329783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4294003980 0x7f4294003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:16.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42aaffd640 1 --2- 172.21.15.89:0/1700329783 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42940047d0 0x7f4294004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:16.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42abfff640 1 --2- 172.21.15.89:0/1700329783 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42ac15cd50 0x7f4294005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:16.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42ab7fe640 1 -- 172.21.15.89:0/1700329783 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42ac15cd50 msgr2=0x7f4294005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:16.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42ab7fe640 1 --2- 172.21.15.89:0/1700329783 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42ac15cd50 0x7f4294005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:16.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42ab7fe640 1 -- 172.21.15.89:0/1700329783 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42940047d0 msgr2=0x7f4294004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:16.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42ab7fe640 1 --2- 172.21.15.89:0/1700329783 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42940047d0 0x7f4294004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:16.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42ab7fe640 1 -- 172.21.15.89:0/1700329783 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4294003530 con 0x7f4294003980 2023-10-30T12:42:16.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.008+0000 7f42aaffd640 1 --2- 172.21.15.89:0/1700329783 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42940047d0 0x7f4294004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:42:16.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.012+0000 7f42ab7fe640 1 --2- 172.21.15.89:0/1700329783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4294003980 0x7f4294003da0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f429c009ae0 tx=0x7f429c01ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2401f7936a7be30 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:16.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.012+0000 7f42aa7fc640 1 -- 172.21.15.89:0/1700329783 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429c0a64a0 con 0x7f4294003980 2023-10-30T12:42:16.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.012+0000 7f42aa7fc640 1 -- 172.21.15.89:0/1700329783 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f429c0a6600 con 0x7f4294003980 2023-10-30T12:42:16.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.012+0000 7f42aa7fc640 1 -- 172.21.15.89:0/1700329783 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429c0a69b0 con 0x7f4294003980 2023-10-30T12:42:16.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.012+0000 7f42b0bd0640 1 -- 172.21.15.89:0/1700329783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4294003980 msgr2=0x7f4294003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:16.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.012+0000 7f42b0bd0640 1 --2- 172.21.15.89:0/1700329783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4294003980 0x7f4294003da0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f429c009ae0 tx=0x7f429c01ca90 comp rx=0 tx=0).stop 2023-10-30T12:42:16.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.012+0000 7f42abfff640 1 -- 172.21.15.89:0/1700329783 reap_dead start 2023-10-30T12:42:16.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.012+0000 7f42b0bd0640 1 -- 172.21.15.89:0/1700329783 shutdown_connections 2023-10-30T12:42:16.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.012+0000 7f42b0bd0640 1 -- 172.21.15.89:0/1700329783 >> 172.21.15.89:0/1700329783 conn(0x7f42ac06a640 msgr2=0x7f42ac065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:16.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.012+0000 7f42b0bd0640 1 -- 172.21.15.89:0/1700329783 shutdown_connections 2023-10-30T12:42:16.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.012+0000 7f42b0bd0640 1 -- 172.21.15.89:0/1700329783 wait complete. 2023-10-30T12:42:16.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.016+0000 7f42b0bd0640 1 Processor -- start 2023-10-30T12:42:16.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.016+0000 7f42b0bd0640 1 -- start start 2023-10-30T12:42:16.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.016+0000 7f42b0bd0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4294003980 0x7f42ac15b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:16.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.016+0000 7f42b0bd0640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42ac15bdf0 0x7f42ac1ef5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:16.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.016+0000 7f42b0bd0640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42ac14d8e0 0x7f42ac0ce500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:16.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.016+0000 7f42b0bd0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ac1eee70 con 0x7f4294003980 2023-10-30T12:42:16.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.016+0000 7f42b0bd0640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ac146720 con 0x7f42ac14d8e0 2023-10-30T12:42:16.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.016+0000 7f42b0bd0640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42ac0b2e10 con 0x7f42ac15bdf0 2023-10-30T12:42:16.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.016+0000 7f42ab7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4294003980 0x7f42ac15b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:16.023 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.016+0000 7f42ab7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4294003980 0x7f42ac15b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35132/0 (socket says 172.21.15.89:35132) 2023-10-30T12:42:16.023 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.016+0000 7f42ab7fe640 1 -- 172.21.15.89:0/3647837583 learned_addr learned my addr 172.21.15.89:0/3647837583 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:16.023 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.016+0000 7f42abfff640 1 --2- 172.21.15.89:0/3647837583 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42ac14d8e0 0x7f42ac0ce500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:16.023 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.016+0000 7f42aaffd640 1 --2- 172.21.15.89:0/3647837583 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42ac15bdf0 0x7f42ac1ef5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:16.023 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.016+0000 7f42ab7fe640 1 -- 172.21.15.89:0/3647837583 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42ac15bdf0 msgr2=0x7f42ac1ef5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:16.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.016+0000 7f42ab7fe640 1 --2- 172.21.15.89:0/3647837583 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42ac15bdf0 0x7f42ac1ef5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:16.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.016+0000 7f42ab7fe640 1 -- 172.21.15.89:0/3647837583 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42ac14d8e0 msgr2=0x7f42ac0ce500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:16.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.016+0000 7f42ab7fe640 1 --2- 172.21.15.89:0/3647837583 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42ac14d8e0 0x7f42ac0ce500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:16.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.016+0000 7f42ab7fe640 1 -- 172.21.15.89:0/3647837583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ac0b3bc0 con 0x7f4294003980 2023-10-30T12:42:16.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.016+0000 7f42ab7fe640 1 --2- 172.21.15.89:0/3647837583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4294003980 0x7f42ac15b8b0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f429c01c8c0 tx=0x7f4294005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:16.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.016+0000 7f4293fff640 1 -- 172.21.15.89:0/3647837583 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429c0a7480 con 0x7f4294003980 2023-10-30T12:42:16.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.016+0000 7f4293fff640 1 -- 172.21.15.89:0/3647837583 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f429c0a75e0 con 0x7f4294003980 2023-10-30T12:42:16.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.016+0000 7f42b0bd0640 1 -- 172.21.15.89:0/3647837583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42ac0b3040 con 0x7f4294003980 2023-10-30T12:42:16.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.016+0000 7f4293fff640 1 -- 172.21.15.89:0/3647837583 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429c0a7970 con 0x7f4294003980 2023-10-30T12:42:16.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.016+0000 7f42b0bd0640 1 -- 172.21.15.89:0/3647837583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42ac0b3600 con 0x7f4294003980 2023-10-30T12:42:16.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.020+0000 7f42b0bd0640 1 -- 172.21.15.89:0/3647837583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42ac04e170 con 0x7f4294003980 2023-10-30T12:42:16.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.020+0000 7f4293fff640 1 -- 172.21.15.89:0/3647837583 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f429c0a7ad0 con 0x7f4294003980 2023-10-30T12:42:16.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.020+0000 7f4293fff640 1 --2- 172.21.15.89:0/3647837583 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f42980664a0 0x7f4298068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:16.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.020+0000 7f4293fff640 1 -- 172.21.15.89:0/3647837583 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6234+0+0 (secure 0 0 0) 0x7f429c11e260 con 0x7f4294003980 2023-10-30T12:42:16.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.020+0000 7f42aaffd640 1 --2- 172.21.15.89:0/3647837583 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f42980664a0 0x7f4298068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:16.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.020+0000 7f42aaffd640 1 --2- 172.21.15.89:0/3647837583 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f42980664a0 0x7f4298068960 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f42ac157130 tx=0x7f42a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:16.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.024+0000 7f4293fff640 1 -- 172.21.15.89:0/3647837583 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f429c0eb3a0 con 0x7f4294003980 2023-10-30T12:42:16.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.204+0000 7f42b0bd0640 1 -- 172.21.15.89:0/3647837583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42ac0c4ee0 con 0x7f4294003980 2023-10-30T12:42:16.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.204+0000 7f4293fff640 1 -- 172.21.15.89:0/3647837583 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+6529 (secure 0 0 0) 0x7f429c0ee9c0 con 0x7f4294003980 2023-10-30T12:42:16.205 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:42:16.205 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5575,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:40:33.262671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:16.205 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 62 2023-10-30T12:42:16.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.208+0000 7f42b0bd0640 1 -- 172.21.15.89:0/3647837583 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f42980664a0 msgr2=0x7f4298068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:16.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.208+0000 7f42b0bd0640 1 --2- 172.21.15.89:0/3647837583 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f42980664a0 0x7f4298068960 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f42ac157130 tx=0x7f42a0008040 comp rx=0 tx=0).stop 2023-10-30T12:42:16.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.208+0000 7f42b0bd0640 1 -- 172.21.15.89:0/3647837583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4294003980 msgr2=0x7f42ac15b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:16.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.208+0000 7f42b0bd0640 1 --2- 172.21.15.89:0/3647837583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4294003980 0x7f42ac15b8b0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f429c01c8c0 tx=0x7f4294005800 comp rx=0 tx=0).stop 2023-10-30T12:42:16.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.208+0000 7f42b0bd0640 1 -- 172.21.15.89:0/3647837583 shutdown_connections 2023-10-30T12:42:16.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.208+0000 7f42b0bd0640 1 --2- 172.21.15.89:0/3647837583 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f42980664a0 0x7f4298068960 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:16.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.208+0000 7f42b0bd0640 1 --2- 172.21.15.89:0/3647837583 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42ac14d8e0 0x7f42ac0ce500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:16.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.208+0000 7f42b0bd0640 1 --2- 172.21.15.89:0/3647837583 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42ac15bdf0 0x7f42ac1ef5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:16.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.208+0000 7f42b0bd0640 1 --2- 172.21.15.89:0/3647837583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4294003980 0x7f42ac15b8b0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:16.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.208+0000 7f42b0bd0640 1 -- 172.21.15.89:0/3647837583 >> 172.21.15.89:0/3647837583 conn(0x7f42ac06a640 msgr2=0x7f42ac0c9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:16.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.208+0000 7f42b0bd0640 1 -- 172.21.15.89:0/3647837583 shutdown_connections 2023-10-30T12:42:16.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:16.208+0000 7f42b0bd0640 1 -- 172.21.15.89:0/3647837583 wait complete. 2023-10-30T12:42:16.225 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T12:42:17.242 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:42:17.246 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:42:18.226 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:18.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.376+0000 7f8dfe4d6640 1 Processor -- start 2023-10-30T12:42:18.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.376+0000 7f8dfe4d6640 1 -- start start 2023-10-30T12:42:18.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.380+0000 7f8dfe4d6640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8df806f7a0 0x7f8df806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:18.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.380+0000 7f8dfe4d6640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8df80c6bd0 0x7f8df8069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:18.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.380+0000 7f8dfe4d6640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8df806a3d0 con 0x7f8df8071110 2023-10-30T12:42:18.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.380+0000 7f8dfe4d6640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8df80700c0 con 0x7f8df806f7a0 2023-10-30T12:42:18.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.380+0000 7f8dfe4d6640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8df8066cf0 con 0x7f8df80c6bd0 2023-10-30T12:42:18.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.380+0000 7f8df7fff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f8df806f7a0 0x7f8df806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:18.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.380+0000 7f8dfca73640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f8df8071110 0x7f8df8158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57934/0 (socket says 172.21.15.89:57934) 2023-10-30T12:42:18.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.380+0000 7f8dfca73640 1 -- 172.21.15.89:0/2711696766 learned_addr learned my addr 172.21.15.89:0/2711696766 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:42:18.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.380+0000 7f8df77fe640 1 --2- 172.21.15.89:0/2711696766 >> v2:172.21.15.192:3300/0 conn(0x7f8df80c6bd0 0x7f8df8069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:18.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.380+0000 7f8df7fff640 1 -- 172.21.15.89:0/2711696766 >> v2:172.21.15.192:3300/0 conn(0x7f8df80c6bd0 msgr2=0x7f8df8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:18.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.380+0000 7f8df7fff640 1 --2- 172.21.15.89:0/2711696766 >> v2:172.21.15.192:3300/0 conn(0x7f8df80c6bd0 0x7f8df8069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:18.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.380+0000 7f8df7fff640 1 -- 172.21.15.89:0/2711696766 >> v1:172.21.15.89:6789/0 conn(0x7f8df8071110 legacy=0x7f8df8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:18.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.380+0000 7f8df7fff640 1 -- 172.21.15.89:0/2711696766 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df80c70d0 con 0x7f8df806f7a0 2023-10-30T12:42:18.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.380+0000 7f8df77fe640 1 --2- 172.21.15.89:0/2711696766 >> v2:172.21.15.192:3300/0 conn(0x7f8df80c6bd0 0x7f8df8069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:42:18.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.380+0000 7f8df6ffd640 1 -- 172.21.15.89:0/2711696766 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2090321363 0 0) 0x7f8df806a3d0 con 0x7f8df8071110 2023-10-30T12:42:18.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.384+0000 7f8df7fff640 1 --2- 172.21.15.89:0/2711696766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8df806f7a0 0x7f8df806fb80 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8df806f1c0 tx=0x7f8de801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77fb9f2bf1314a09 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:18.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.384+0000 7f8df6ffd640 1 -- 172.21.15.89:0/2711696766 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de80a8950 con 0x7f8df806f7a0 2023-10-30T12:42:18.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.384+0000 7f8df6ffd640 1 -- 172.21.15.89:0/2711696766 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de80a8ab0 con 0x7f8df806f7a0 2023-10-30T12:42:18.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.384+0000 7f8df6ffd640 1 -- 172.21.15.89:0/2711696766 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de80a8e40 con 0x7f8df806f7a0 2023-10-30T12:42:18.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.384+0000 7f8dfe4d6640 1 -- 172.21.15.89:0/2711696766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8df806f7a0 msgr2=0x7f8df806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:18.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.384+0000 7f8dfe4d6640 1 --2- 172.21.15.89:0/2711696766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8df806f7a0 0x7f8df806fb80 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8df806f1c0 tx=0x7f8de801ec50 comp rx=0 tx=0).stop 2023-10-30T12:42:18.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.384+0000 7f8dfe4d6640 1 -- 172.21.15.89:0/2711696766 shutdown_connections 2023-10-30T12:42:18.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.384+0000 7f8dfe4d6640 1 --2- 172.21.15.89:0/2711696766 >> v2:172.21.15.192:3300/0 conn(0x7f8df80c6bd0 0x7f8df8069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:18.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.384+0000 7f8dfe4d6640 1 --2- 172.21.15.89:0/2711696766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8df806f7a0 0x7f8df806fb80 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:18.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.384+0000 7f8dfe4d6640 1 -- 172.21.15.89:0/2711696766 >> 172.21.15.89:0/2711696766 conn(0x7f8df80c7910 msgr2=0x7f8df80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:18.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.384+0000 7f8dfe4d6640 1 -- 172.21.15.89:0/2711696766 shutdown_connections 2023-10-30T12:42:18.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.384+0000 7f8dfe4d6640 1 -- 172.21.15.89:0/2711696766 wait complete. 2023-10-30T12:42:18.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.388+0000 7f8dfe4d6640 1 Processor -- start 2023-10-30T12:42:18.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.388+0000 7f8dfe4d6640 1 -- start start 2023-10-30T12:42:18.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.388+0000 7f8dfe4d6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8df806f7a0 0x7f8df80af090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:18.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.388+0000 7f8dfe4d6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8df8071110 0x7f8df81eed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:18.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.388+0000 7f8dfe4d6640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8df80c6bd0 0x7f8df81ef2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:18.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.388+0000 7f8dfe4d6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df8158540 con 0x7f8df806f7a0 2023-10-30T12:42:18.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.388+0000 7f8dfe4d6640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df81534b0 con 0x7f8df8071110 2023-10-30T12:42:18.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.388+0000 7f8dfe4d6640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df8158260 con 0x7f8df80c6bd0 2023-10-30T12:42:18.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.388+0000 7f8df7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8df806f7a0 0x7f8df80af090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:18.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.388+0000 7f8df7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8df806f7a0 0x7f8df80af090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35154/0 (socket says 172.21.15.89:35154) 2023-10-30T12:42:18.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.388+0000 7f8df7fff640 1 -- 172.21.15.89:0/2795687967 learned_addr learned my addr 172.21.15.89:0/2795687967 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:18.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.388+0000 7f8df77fe640 1 --2- 172.21.15.89:0/2795687967 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8df8071110 0x7f8df81eed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:18.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.388+0000 7f8dfca73640 1 --2- 172.21.15.89:0/2795687967 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8df80c6bd0 0x7f8df81ef2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:18.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.388+0000 7f8df7fff640 1 -- 172.21.15.89:0/2795687967 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8df80c6bd0 msgr2=0x7f8df81ef2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:18.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.388+0000 7f8df7fff640 1 --2- 172.21.15.89:0/2795687967 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8df80c6bd0 0x7f8df81ef2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:18.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.388+0000 7f8df7fff640 1 -- 172.21.15.89:0/2795687967 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8df8071110 msgr2=0x7f8df81eed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:18.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.388+0000 7f8df7fff640 1 --2- 172.21.15.89:0/2795687967 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8df8071110 0x7f8df81eed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:18.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.388+0000 7f8df7fff640 1 -- 172.21.15.89:0/2795687967 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de8009420 con 0x7f8df806f7a0 2023-10-30T12:42:18.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.388+0000 7f8dfca73640 1 --2- 172.21.15.89:0/2795687967 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8df80c6bd0 0x7f8df81ef2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:42:18.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.388+0000 7f8df7fff640 1 --2- 172.21.15.89:0/2795687967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8df806f7a0 0x7f8df80af090 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f8de80b3040 tx=0x7f8de801b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:18.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.388+0000 7f8df4ff9640 1 -- 172.21.15.89:0/2795687967 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de80b9630 con 0x7f8df806f7a0 2023-10-30T12:42:18.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.388+0000 7f8df4ff9640 1 -- 172.21.15.89:0/2795687967 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de80b97b0 con 0x7f8df806f7a0 2023-10-30T12:42:18.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.388+0000 7f8df4ff9640 1 -- 172.21.15.89:0/2795687967 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de80b9b60 con 0x7f8df806f7a0 2023-10-30T12:42:18.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.388+0000 7f8dfe4d6640 1 -- 172.21.15.89:0/2795687967 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8df8157930 con 0x7f8df806f7a0 2023-10-30T12:42:18.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.388+0000 7f8dfe4d6640 1 -- 172.21.15.89:0/2795687967 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8df8157ef0 con 0x7f8df806f7a0 2023-10-30T12:42:18.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.388+0000 7f8dfe4d6640 1 -- 172.21.15.89:0/2795687967 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8df804e170 con 0x7f8df806f7a0 2023-10-30T12:42:18.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.392+0000 7f8df4ff9640 1 -- 172.21.15.89:0/2795687967 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8de80b7040 con 0x7f8df806f7a0 2023-10-30T12:42:18.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.392+0000 7f8df4ff9640 1 --2- 172.21.15.89:0/2795687967 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8de40664a0 0x7f8de4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:18.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.392+0000 7f8df4ff9640 1 -- 172.21.15.89:0/2795687967 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6234+0+0 (secure 0 0 0) 0x7f8de8122090 con 0x7f8df806f7a0 2023-10-30T12:42:18.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.392+0000 7f8df77fe640 1 --2- 172.21.15.89:0/2795687967 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8de40664a0 0x7f8de4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:18.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.392+0000 7f8df77fe640 1 --2- 172.21.15.89:0/2795687967 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8de40664a0 0x7f8de4068960 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8df80b02d0 tx=0x7f8dec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:18.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.396+0000 7f8df4ff9640 1 -- 172.21.15.89:0/2795687967 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8de80ef260 con 0x7f8df806f7a0 2023-10-30T12:42:18.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.576+0000 7f8dfe4d6640 1 -- 172.21.15.89:0/2795687967 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8df80b06c0 con 0x7f8df806f7a0 2023-10-30T12:42:18.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.580+0000 7f8df4ff9640 1 -- 172.21.15.89:0/2795687967 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+6529 (secure 0 0 0) 0x7f8de80f2880 con 0x7f8df806f7a0 2023-10-30T12:42:18.579 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:42:18.579 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5575,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:40:33.262671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:18.579 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 62 2023-10-30T12:42:18.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.580+0000 7f8dfe4d6640 1 -- 172.21.15.89:0/2795687967 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8de40664a0 msgr2=0x7f8de4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:18.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.580+0000 7f8dfe4d6640 1 --2- 172.21.15.89:0/2795687967 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8de40664a0 0x7f8de4068960 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8df80b02d0 tx=0x7f8dec008040 comp rx=0 tx=0).stop 2023-10-30T12:42:18.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.580+0000 7f8dfe4d6640 1 -- 172.21.15.89:0/2795687967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8df806f7a0 msgr2=0x7f8df80af090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:18.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.580+0000 7f8dfe4d6640 1 --2- 172.21.15.89:0/2795687967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8df806f7a0 0x7f8df80af090 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f8de80b3040 tx=0x7f8de801b9f0 comp rx=0 tx=0).stop 2023-10-30T12:42:18.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.580+0000 7f8dfe4d6640 1 -- 172.21.15.89:0/2795687967 shutdown_connections 2023-10-30T12:42:18.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.580+0000 7f8dfe4d6640 1 --2- 172.21.15.89:0/2795687967 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8de40664a0 0x7f8de4068960 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:18.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.580+0000 7f8dfe4d6640 1 --2- 172.21.15.89:0/2795687967 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8df8071110 0x7f8df81eed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:18.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.580+0000 7f8dfe4d6640 1 --2- 172.21.15.89:0/2795687967 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8df80c6bd0 0x7f8df81ef2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:18.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.580+0000 7f8dfe4d6640 1 --2- 172.21.15.89:0/2795687967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8df806f7a0 0x7f8df80af090 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:18.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.580+0000 7f8dfe4d6640 1 -- 172.21.15.89:0/2795687967 >> 172.21.15.89:0/2795687967 conn(0x7f8df80c7910 msgr2=0x7f8df80c8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:18.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.580+0000 7f8dfe4d6640 1 -- 172.21.15.89:0/2795687967 shutdown_connections 2023-10-30T12:42:18.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:18.580+0000 7f8dfe4d6640 1 -- 172.21.15.89:0/2795687967 wait complete. 2023-10-30T12:42:18.599 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T12:42:20.600 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:20.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.748+0000 7f0f2c86b640 1 Processor -- start 2023-10-30T12:42:20.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.748+0000 7f0f2c86b640 1 -- start start 2023-10-30T12:42:20.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.752+0000 7f0f2c86b640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0f2806f7a0 0x7f0f2806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:20.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.752+0000 7f0f2c86b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f280c6bd0 0x7f0f28069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:20.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.752+0000 7f0f2c86b640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0f28071110 0x7f0f28158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:20.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.752+0000 7f0f2c86b640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f280700c0 con 0x7f0f2806f7a0 2023-10-30T12:42:20.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.752+0000 7f0f2c86b640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f2806a3d0 con 0x7f0f28071110 2023-10-30T12:42:20.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.752+0000 7f0f2c86b640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f28066cf0 con 0x7f0f280c6bd0 2023-10-30T12:42:20.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.752+0000 7f0f2659c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0f2806f7a0 0x7f0f2806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:20.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.752+0000 7f0f2659c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0f2806f7a0 0x7f0f2806fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58228/0 (socket says 172.21.15.89:58228) 2023-10-30T12:42:20.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.752+0000 7f0f2659c640 1 -- 172.21.15.89:0/2402651276 learned_addr learned my addr 172.21.15.89:0/2402651276 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:20.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.752+0000 7f0f26d9d640 1 --2- 172.21.15.89:0/2402651276 >> v2:172.21.15.192:3300/0 conn(0x7f0f28071110 0x7f0f28158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:20.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.752+0000 7f0f1dd9b640 1 --2- 172.21.15.89:0/2402651276 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f280c6bd0 0x7f0f28069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:20.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.752+0000 7f0f2659c640 1 -- 172.21.15.89:0/2402651276 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f280c6bd0 msgr2=0x7f0f28069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:20.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.752+0000 7f0f2659c640 1 --2- 172.21.15.89:0/2402651276 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f280c6bd0 0x7f0f28069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:20.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.752+0000 7f0f2659c640 1 -- 172.21.15.89:0/2402651276 >> v2:172.21.15.192:3300/0 conn(0x7f0f28071110 msgr2=0x7f0f28158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:20.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.752+0000 7f0f2659c640 1 --2- 172.21.15.89:0/2402651276 >> v2:172.21.15.192:3300/0 conn(0x7f0f28071110 0x7f0f28158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:20.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.752+0000 7f0f2659c640 1 -- 172.21.15.89:0/2402651276 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f280c70d0 con 0x7f0f2806f7a0 2023-10-30T12:42:20.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.752+0000 7f0f26d9d640 1 --2- 172.21.15.89:0/2402651276 >> v2:172.21.15.192:3300/0 conn(0x7f0f28071110 0x7f0f28158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:42:20.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.756+0000 7f0f2659c640 1 --2- 172.21.15.89:0/2402651276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f2806f7a0 0x7f0f2806fb80 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f0f2806c7d0 tx=0x7f0f1801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=117272939bb6ecaf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:20.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.756+0000 7f0f25d9b640 1 -- 172.21.15.89:0/2402651276 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f180a89b0 con 0x7f0f2806f7a0 2023-10-30T12:42:20.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.756+0000 7f0f25d9b640 1 -- 172.21.15.89:0/2402651276 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f180a8b10 con 0x7f0f2806f7a0 2023-10-30T12:42:20.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.756+0000 7f0f25d9b640 1 -- 172.21.15.89:0/2402651276 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f180a8ea0 con 0x7f0f2806f7a0 2023-10-30T12:42:20.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.756+0000 7f0f2c86b640 1 -- 172.21.15.89:0/2402651276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f2806f7a0 msgr2=0x7f0f2806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:20.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.756+0000 7f0f2c86b640 1 --2- 172.21.15.89:0/2402651276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f2806f7a0 0x7f0f2806fb80 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f0f2806c7d0 tx=0x7f0f1801ecf0 comp rx=0 tx=0).stop 2023-10-30T12:42:20.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.756+0000 7f0f2c86b640 1 -- 172.21.15.89:0/2402651276 shutdown_connections 2023-10-30T12:42:20.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.756+0000 7f0f2c86b640 1 --2- 172.21.15.89:0/2402651276 >> v2:172.21.15.192:3300/0 conn(0x7f0f28071110 0x7f0f28158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:20.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.756+0000 7f0f2c86b640 1 --2- 172.21.15.89:0/2402651276 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f280c6bd0 0x7f0f28069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:20.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.756+0000 7f0f2c86b640 1 --2- 172.21.15.89:0/2402651276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f2806f7a0 0x7f0f2806fb80 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:20.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.756+0000 7f0f2c86b640 1 -- 172.21.15.89:0/2402651276 >> 172.21.15.89:0/2402651276 conn(0x7f0f280c7910 msgr2=0x7f0f280c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:20.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.756+0000 7f0f2c86b640 1 -- 172.21.15.89:0/2402651276 shutdown_connections 2023-10-30T12:42:20.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.756+0000 7f0f2c86b640 1 -- 172.21.15.89:0/2402651276 wait complete. 2023-10-30T12:42:20.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.756+0000 7f0f2c86b640 1 Processor -- start 2023-10-30T12:42:20.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.756+0000 7f0f2c86b640 1 -- start start 2023-10-30T12:42:20.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.760+0000 7f0f2c86b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f2806f7a0 0x7f0f281ef4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:20.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.760+0000 7f0f2c86b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f28071110 0x7f0f280ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:20.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.760+0000 7f0f2c86b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f280c6bd0 0x7f0f28150350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:20.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.760+0000 7f0f2c86b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f281f2d90 con 0x7f0f280c6bd0 2023-10-30T12:42:20.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.760+0000 7f0f2c86b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f281efa30 con 0x7f0f28071110 2023-10-30T12:42:20.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.760+0000 7f0f2c86b640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f280ae0a0 con 0x7f0f2806f7a0 2023-10-30T12:42:20.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.760+0000 7f0f26d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f280c6bd0 0x7f0f28150350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:20.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.760+0000 7f0f26d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f280c6bd0 0x7f0f28150350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58234/0 (socket says 172.21.15.89:58234) 2023-10-30T12:42:20.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.760+0000 7f0f26d9d640 1 -- 172.21.15.89:0/1251424389 learned_addr learned my addr 172.21.15.89:0/1251424389 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:20.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.760+0000 7f0f1dd9b640 1 --2- 172.21.15.89:0/1251424389 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f28071110 0x7f0f280ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:20.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.760+0000 7f0f2659c640 1 --2- 172.21.15.89:0/1251424389 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f2806f7a0 0x7f0f281ef4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:20.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.760+0000 7f0f26d9d640 1 -- 172.21.15.89:0/1251424389 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f2806f7a0 msgr2=0x7f0f281ef4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:20.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.760+0000 7f0f26d9d640 1 --2- 172.21.15.89:0/1251424389 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f2806f7a0 0x7f0f281ef4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:20.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.760+0000 7f0f26d9d640 1 -- 172.21.15.89:0/1251424389 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f28071110 msgr2=0x7f0f280ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:20.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.760+0000 7f0f26d9d640 1 --2- 172.21.15.89:0/1251424389 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f28071110 0x7f0f280ad8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:20.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.760+0000 7f0f26d9d640 1 -- 172.21.15.89:0/1251424389 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f180094d0 con 0x7f0f280c6bd0 2023-10-30T12:42:20.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.760+0000 7f0f26d9d640 1 --2- 172.21.15.89:0/1251424389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f280c6bd0 0x7f0f28150350 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f0f20007de0 tx=0x7f0f2000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:20.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.760+0000 7f0f1f7fe640 1 -- 172.21.15.89:0/1251424389 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f200039a0 con 0x7f0f280c6bd0 2023-10-30T12:42:20.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.760+0000 7f0f1f7fe640 1 -- 172.21.15.89:0/1251424389 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f2000f040 con 0x7f0f280c6bd0 2023-10-30T12:42:20.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.760+0000 7f0f1f7fe640 1 -- 172.21.15.89:0/1251424389 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f20003d60 con 0x7f0f280c6bd0 2023-10-30T12:42:20.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.760+0000 7f0f2c86b640 1 -- 172.21.15.89:0/1251424389 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f280ade10 con 0x7f0f280c6bd0 2023-10-30T12:42:20.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.760+0000 7f0f2c86b640 1 -- 172.21.15.89:0/1251424389 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f281f23f0 con 0x7f0f280c6bd0 2023-10-30T12:42:20.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.760+0000 7f0f2c86b640 1 -- 172.21.15.89:0/1251424389 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f2804e170 con 0x7f0f280c6bd0 2023-10-30T12:42:20.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.760+0000 7f0f1f7fe640 1 -- 172.21.15.89:0/1251424389 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0f20004030 con 0x7f0f280c6bd0 2023-10-30T12:42:20.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.764+0000 7f0f1f7fe640 1 --2- 172.21.15.89:0/1251424389 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0f080664d0 0x7f0f08068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:20.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.764+0000 7f0f1f7fe640 1 -- 172.21.15.89:0/1251424389 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6234+0+0 (secure 0 0 0) 0x7f0f200843b0 con 0x7f0f280c6bd0 2023-10-30T12:42:20.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.764+0000 7f0f2659c640 1 --2- 172.21.15.89:0/1251424389 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0f080664d0 0x7f0f08068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:20.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.764+0000 7f0f2659c640 1 --2- 172.21.15.89:0/1251424389 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0f080664d0 0x7f0f08068990 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0f180b2000 tx=0x7f0f18008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:20.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.768+0000 7f0f1f7fe640 1 -- 172.21.15.89:0/1251424389 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0f20051610 con 0x7f0f280c6bd0 2023-10-30T12:42:20.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.948+0000 7f0f2c86b640 1 -- 172.21.15.89:0/1251424389 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f281f2ed0 con 0x7f0f280c6bd0 2023-10-30T12:42:20.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.948+0000 7f0f1f7fe640 1 -- 172.21.15.89:0/1251424389 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+6529 (secure 0 0 0) 0x7f0f20054c30 con 0x7f0f280c6bd0 2023-10-30T12:42:20.949 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:42:20.949 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5575,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:40:33.262671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:20.950 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 62 2023-10-30T12:42:20.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.952+0000 7f0f2c86b640 1 -- 172.21.15.89:0/1251424389 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0f080664d0 msgr2=0x7f0f08068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:20.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.952+0000 7f0f2c86b640 1 --2- 172.21.15.89:0/1251424389 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0f080664d0 0x7f0f08068990 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0f180b2000 tx=0x7f0f18008040 comp rx=0 tx=0).stop 2023-10-30T12:42:20.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.952+0000 7f0f2c86b640 1 -- 172.21.15.89:0/1251424389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f280c6bd0 msgr2=0x7f0f28150350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:20.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.952+0000 7f0f2c86b640 1 --2- 172.21.15.89:0/1251424389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f280c6bd0 0x7f0f28150350 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f0f20007de0 tx=0x7f0f2000d4a0 comp rx=0 tx=0).stop 2023-10-30T12:42:20.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.952+0000 7f0f2c86b640 1 -- 172.21.15.89:0/1251424389 shutdown_connections 2023-10-30T12:42:20.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.952+0000 7f0f2c86b640 1 --2- 172.21.15.89:0/1251424389 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0f080664d0 0x7f0f08068990 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:20.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.952+0000 7f0f2c86b640 1 --2- 172.21.15.89:0/1251424389 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f28071110 0x7f0f280ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:20.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.952+0000 7f0f2c86b640 1 --2- 172.21.15.89:0/1251424389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f280c6bd0 0x7f0f28150350 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:20.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.952+0000 7f0f2c86b640 1 --2- 172.21.15.89:0/1251424389 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0f2806f7a0 0x7f0f281ef4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:20.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.952+0000 7f0f2c86b640 1 -- 172.21.15.89:0/1251424389 >> 172.21.15.89:0/1251424389 conn(0x7f0f280c7910 msgr2=0x7f0f2806c050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:20.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.952+0000 7f0f2c86b640 1 -- 172.21.15.89:0/1251424389 shutdown_connections 2023-10-30T12:42:20.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:20.952+0000 7f0f2c86b640 1 -- 172.21.15.89:0/1251424389 wait complete. 2023-10-30T12:42:20.969 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T12:42:22.970 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:23.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.116+0000 7fc0dfdd3640 1 Processor -- start 2023-10-30T12:42:23.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.116+0000 7fc0dfdd3640 1 -- start start 2023-10-30T12:42:23.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0dfdd3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0d80c6540 0x7fc0d815e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:23.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0dfdd3640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d80c69d0 con 0x7fc0d81580e0 2023-10-30T12:42:23.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0dfdd3640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d80c5810 con 0x7fc0d8070740 2023-10-30T12:42:23.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0dfdd3640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d806acb0 con 0x7fc0d80c6540 2023-10-30T12:42:23.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0ddb6f640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc0d81580e0 0x7fc0d8069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41562/0 (socket says 172.21.15.89:41562) 2023-10-30T12:42:23.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0ddb6f640 1 -- 172.21.15.89:0/1870297053 learned_addr learned my addr 172.21.15.89:0/1870297053 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:42:23.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0de370640 1 --2- 172.21.15.89:0/1870297053 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0d80c6540 0x7fc0d815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:23.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0dcb6d640 1 -- 172.21.15.89:0/1870297053 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 931807335 0 0) 0x7fc0d80c69d0 con 0x7fc0d81580e0 2023-10-30T12:42:23.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0dcb6d640 1 -- 172.21.15.89:0/1870297053 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0c4003530 con 0x7fc0d81580e0 2023-10-30T12:42:23.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0dcb6d640 1 -- 172.21.15.89:0/1870297053 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738152851 0 0) 0x7fc0d80c5810 con 0x7fc0d8070740 2023-10-30T12:42:23.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0dcb6d640 1 -- 172.21.15.89:0/1870297053 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0d80c69d0 con 0x7fc0d8070740 2023-10-30T12:42:23.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0dcb6d640 1 -- 172.21.15.89:0/1870297053 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 637256228 0 0) 0x7fc0c4003530 con 0x7fc0d81580e0 2023-10-30T12:42:23.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0dcb6d640 1 -- 172.21.15.89:0/1870297053 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0d80c6540 msgr2=0x7fc0d815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:23.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0dcb6d640 1 --2- 172.21.15.89:0/1870297053 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0d80c6540 0x7fc0d815e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:23.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0dcb6d640 1 -- 172.21.15.89:0/1870297053 >> v1:172.21.15.192:6789/0 conn(0x7fc0d8070740 legacy=0x7fc0d806a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:23.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0dcb6d640 1 -- 172.21.15.89:0/1870297053 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d80c6bb0 con 0x7fc0d81580e0 2023-10-30T12:42:23.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0dcb6d640 1 -- 172.21.15.89:0/1870297053 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7fc0d4001680 con 0x7fc0d81580e0 2023-10-30T12:42:23.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0dcb6d640 1 -- 172.21.15.89:0/1870297053 >> v1:172.21.15.89:6789/0 conn(0x7fc0d81580e0 legacy=0x7fc0d8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:23.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0dcb6d640 1 --2- 172.21.15.89:0/1870297053 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0c4003b40 0x7fc0c4003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:23.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0dcb6d640 1 --2- 172.21.15.89:0/1870297053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0c40047d0 0x7fc0c4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:23.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0dcb6d640 1 --2- 172.21.15.89:0/1870297053 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0d80c6540 0x7fc0c4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:23.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0dcb6d640 1 -- 172.21.15.89:0/1870297053 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c400ba40 con 0x7fc0c40047d0 2023-10-30T12:42:23.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0dcb6d640 1 -- 172.21.15.89:0/1870297053 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c400bb80 con 0x7fc0c4003b40 2023-10-30T12:42:23.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0dcb6d640 1 -- 172.21.15.89:0/1870297053 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c400bd10 con 0x7fc0d80c6540 2023-10-30T12:42:23.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0dd36e640 1 --2- 172.21.15.89:0/1870297053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0c40047d0 0x7fc0c4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:23.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0ddb6f640 1 --2- 172.21.15.89:0/1870297053 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0c4003b40 0x7fc0c4003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:23.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0de370640 1 --2- 172.21.15.89:0/1870297053 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0d80c6540 0x7fc0c4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:23.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0dd36e640 1 -- 172.21.15.89:0/1870297053 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0d80c6540 msgr2=0x7fc0c4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:23.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0dd36e640 1 --2- 172.21.15.89:0/1870297053 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0d80c6540 0x7fc0c4005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:23.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0dd36e640 1 -- 172.21.15.89:0/1870297053 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0c4003b40 msgr2=0x7fc0c4003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:23.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0dd36e640 1 --2- 172.21.15.89:0/1870297053 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0c4003b40 0x7fc0c4003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:23.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.120+0000 7fc0dd36e640 1 -- 172.21.15.89:0/1870297053 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c4003530 con 0x7fc0c40047d0 2023-10-30T12:42:23.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.124+0000 7fc0dd36e640 1 --2- 172.21.15.89:0/1870297053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0c40047d0 0x7fc0c4004c50 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fc0c80018d0 tx=0x7fc0c801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d47b2f5674ee2a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:23.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.124+0000 7fc0dcb6d640 1 -- 172.21.15.89:0/1870297053 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c80052d0 con 0x7fc0c40047d0 2023-10-30T12:42:23.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.124+0000 7fc0dcb6d640 1 -- 172.21.15.89:0/1870297053 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c8005450 con 0x7fc0c40047d0 2023-10-30T12:42:23.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.124+0000 7fc0dcb6d640 1 -- 172.21.15.89:0/1870297053 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c80a88d0 con 0x7fc0c40047d0 2023-10-30T12:42:23.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.124+0000 7fc0dfdd3640 1 -- 172.21.15.89:0/1870297053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0c40047d0 msgr2=0x7fc0c4004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:23.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.124+0000 7fc0dfdd3640 1 --2- 172.21.15.89:0/1870297053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0c40047d0 0x7fc0c4004c50 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fc0c80018d0 tx=0x7fc0c801ec70 comp rx=0 tx=0).stop 2023-10-30T12:42:23.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.124+0000 7fc0de370640 1 -- 172.21.15.89:0/1870297053 reap_dead start 2023-10-30T12:42:23.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.124+0000 7fc0dfdd3640 1 -- 172.21.15.89:0/1870297053 shutdown_connections 2023-10-30T12:42:23.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.124+0000 7fc0dfdd3640 1 -- 172.21.15.89:0/1870297053 >> 172.21.15.89:0/1870297053 conn(0x7fc0d80c7910 msgr2=0x7fc0d80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:23.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.124+0000 7fc0dfdd3640 1 -- 172.21.15.89:0/1870297053 shutdown_connections 2023-10-30T12:42:23.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.124+0000 7fc0dfdd3640 1 -- 172.21.15.89:0/1870297053 wait complete. 2023-10-30T12:42:23.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.124+0000 7fc0dfdd3640 1 Processor -- start 2023-10-30T12:42:23.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.124+0000 7fc0dfdd3640 1 -- start start 2023-10-30T12:42:23.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.128+0000 7fc0dfdd3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0c40047d0 0x7fc0d80c4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:23.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.128+0000 7fc0dfdd3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0d814c320 0x7fc0d814ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:23.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.128+0000 7fc0dfdd3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0d81f3a30 0x7fc0d81f3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:23.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.128+0000 7fc0dfdd3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d81e9fd0 con 0x7fc0c40047d0 2023-10-30T12:42:23.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.128+0000 7fc0dfdd3640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d81f43f0 con 0x7fc0d814c320 2023-10-30T12:42:23.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.128+0000 7fc0dfdd3640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d81f3530 con 0x7fc0d81f3a30 2023-10-30T12:42:23.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.128+0000 7fc0ddb6f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0c40047d0 0x7fc0d80c4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:23.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.128+0000 7fc0ddb6f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0c40047d0 0x7fc0d80c4df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58262/0 (socket says 172.21.15.89:58262) 2023-10-30T12:42:23.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.128+0000 7fc0ddb6f640 1 -- 172.21.15.89:0/3863009986 learned_addr learned my addr 172.21.15.89:0/3863009986 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:23.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.128+0000 7fc0de370640 1 --2- 172.21.15.89:0/3863009986 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0d81f3a30 0x7fc0d81f3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:23.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.128+0000 7fc0dd36e640 1 --2- 172.21.15.89:0/3863009986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0d814c320 0x7fc0d814ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:23.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.128+0000 7fc0ddb6f640 1 -- 172.21.15.89:0/3863009986 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0d81f3a30 msgr2=0x7fc0d81f3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:23.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.128+0000 7fc0ddb6f640 1 --2- 172.21.15.89:0/3863009986 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0d81f3a30 0x7fc0d81f3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:23.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.128+0000 7fc0ddb6f640 1 -- 172.21.15.89:0/3863009986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0d814c320 msgr2=0x7fc0d814ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:23.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.128+0000 7fc0ddb6f640 1 --2- 172.21.15.89:0/3863009986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0d814c320 0x7fc0d814ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:23.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.128+0000 7fc0ddb6f640 1 -- 172.21.15.89:0/3863009986 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d80c24f0 con 0x7fc0c40047d0 2023-10-30T12:42:23.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.128+0000 7fc0ddb6f640 1 --2- 172.21.15.89:0/3863009986 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0c40047d0 0x7fc0d80c4df0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4004f50 tx=0x7fc0d40074f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:23.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.128+0000 7fc0ce7fc640 1 -- 172.21.15.89:0/3863009986 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d4007740 con 0x7fc0c40047d0 2023-10-30T12:42:23.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.128+0000 7fc0ce7fc640 1 -- 172.21.15.89:0/3863009986 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0d400f940 con 0x7fc0c40047d0 2023-10-30T12:42:23.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.128+0000 7fc0dfdd3640 1 -- 172.21.15.89:0/3863009986 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0d81ea220 con 0x7fc0c40047d0 2023-10-30T12:42:23.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.128+0000 7fc0ce7fc640 1 -- 172.21.15.89:0/3863009986 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d400fcd0 con 0x7fc0c40047d0 2023-10-30T12:42:23.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.128+0000 7fc0dfdd3640 1 -- 172.21.15.89:0/3863009986 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0d81ea7e0 con 0x7fc0c40047d0 2023-10-30T12:42:23.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.128+0000 7fc0dfdd3640 1 -- 172.21.15.89:0/3863009986 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0d804e170 con 0x7fc0c40047d0 2023-10-30T12:42:23.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.128+0000 7fc0ce7fc640 1 -- 172.21.15.89:0/3863009986 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc0d401d460 con 0x7fc0c40047d0 2023-10-30T12:42:23.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.132+0000 7fc0ce7fc640 1 --2- 172.21.15.89:0/3863009986 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc0bc0664a0 0x7fc0bc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:23.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.132+0000 7fc0ce7fc640 1 -- 172.21.15.89:0/3863009986 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6234+0+0 (secure 0 0 0) 0x7fc0d4084200 con 0x7fc0c40047d0 2023-10-30T12:42:23.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.132+0000 7fc0dd36e640 1 --2- 172.21.15.89:0/3863009986 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc0bc0664a0 0x7fc0bc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:23.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.132+0000 7fc0dd36e640 1 --2- 172.21.15.89:0/3863009986 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc0bc0664a0 0x7fc0bc068960 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc0c80b0000 tx=0x7fc0c80aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:23.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.136+0000 7fc0ce7fc640 1 -- 172.21.15.89:0/3863009986 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc0d4051470 con 0x7fc0c40047d0 2023-10-30T12:42:23.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.320+0000 7fc0dfdd3640 1 -- 172.21.15.89:0/3863009986 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0d8152170 con 0x7fc0c40047d0 2023-10-30T12:42:23.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.320+0000 7fc0ce7fc640 1 -- 172.21.15.89:0/3863009986 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+6529 (secure 0 0 0) 0x7fc0d4054a90 con 0x7fc0c40047d0 2023-10-30T12:42:23.319 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:42:23.319 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5575,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:40:33.262671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:23.319 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 62 2023-10-30T12:42:23.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.320+0000 7fc0dfdd3640 1 -- 172.21.15.89:0/3863009986 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc0bc0664a0 msgr2=0x7fc0bc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:23.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.320+0000 7fc0dfdd3640 1 --2- 172.21.15.89:0/3863009986 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc0bc0664a0 0x7fc0bc068960 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc0c80b0000 tx=0x7fc0c80aa070 comp rx=0 tx=0).stop 2023-10-30T12:42:23.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.320+0000 7fc0dfdd3640 1 -- 172.21.15.89:0/3863009986 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0c40047d0 msgr2=0x7fc0d80c4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:23.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.320+0000 7fc0dfdd3640 1 --2- 172.21.15.89:0/3863009986 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0c40047d0 0x7fc0d80c4df0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4004f50 tx=0x7fc0d40074f0 comp rx=0 tx=0).stop 2023-10-30T12:42:23.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.320+0000 7fc0dfdd3640 1 -- 172.21.15.89:0/3863009986 shutdown_connections 2023-10-30T12:42:23.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.320+0000 7fc0dfdd3640 1 --2- 172.21.15.89:0/3863009986 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc0bc0664a0 0x7fc0bc068960 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:23.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.320+0000 7fc0dfdd3640 1 --2- 172.21.15.89:0/3863009986 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0d814c320 0x7fc0d814ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:23.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.320+0000 7fc0dfdd3640 1 --2- 172.21.15.89:0/3863009986 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc0d81f3a30 0x7fc0d81f3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:23.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.320+0000 7fc0dfdd3640 1 --2- 172.21.15.89:0/3863009986 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0c40047d0 0x7fc0d80c4df0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:23.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.320+0000 7fc0dfdd3640 1 -- 172.21.15.89:0/3863009986 >> 172.21.15.89:0/3863009986 conn(0x7fc0d80c7910 msgr2=0x7fc0d80b08a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:23.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.320+0000 7fc0dfdd3640 1 -- 172.21.15.89:0/3863009986 shutdown_connections 2023-10-30T12:42:23.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:23.320+0000 7fc0dfdd3640 1 -- 172.21.15.89:0/3863009986 wait complete. 2023-10-30T12:42:23.339 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T12:42:25.340 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:25.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.484+0000 7f7abcd1d640 1 Processor -- start 2023-10-30T12:42:25.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.484+0000 7f7abcd1d640 1 -- start start 2023-10-30T12:42:25.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.488+0000 7f7abcd1d640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7ab8069e90 0x7f7ab815a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:25.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.488+0000 7f7abcd1d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7ab806cb80 0x7f7ab806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:25.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.488+0000 7f7abcd1d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ab806d530 con 0x7f7ab806a270 2023-10-30T12:42:25.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.488+0000 7f7abcd1d640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ab80c8570 con 0x7f7ab806cb80 2023-10-30T12:42:25.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.488+0000 7f7abcd1d640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ab80c6c10 con 0x7f7ab8069e90 2023-10-30T12:42:25.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.488+0000 7f7ab6d9d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7ab806cb80 0x7f7ab806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:25.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.488+0000 7f7ab6d9d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7ab806cb80 0x7f7ab806cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58278/0 (socket says 172.21.15.89:58278) 2023-10-30T12:42:25.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.488+0000 7f7ab6d9d640 1 -- 172.21.15.89:0/1699668414 learned_addr learned my addr 172.21.15.89:0/1699668414 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:25.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.488+0000 7f7ab659c640 1 --2- 172.21.15.89:0/1699668414 >> v2:172.21.15.192:3300/0 conn(0x7f7ab8069e90 0x7f7ab815a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:25.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.488+0000 7f7ab6d9d640 1 -- 172.21.15.89:0/1699668414 >> v2:172.21.15.192:3300/0 conn(0x7f7ab8069e90 msgr2=0x7f7ab815a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:25.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.488+0000 7f7ab6d9d640 1 --2- 172.21.15.89:0/1699668414 >> v2:172.21.15.192:3300/0 conn(0x7f7ab8069e90 0x7f7ab815a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:25.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.488+0000 7f7ab6d9d640 1 -- 172.21.15.89:0/1699668414 >> v1:172.21.15.192:6789/0 conn(0x7f7ab806a270 legacy=0x7f7ab80c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:25.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.488+0000 7f7ab6d9d640 1 -- 172.21.15.89:0/1699668414 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab80c6ef0 con 0x7f7ab806cb80 2023-10-30T12:42:25.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.492+0000 7f7ab6d9d640 1 --2- 172.21.15.89:0/1699668414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab806cb80 0x7f7ab806cff0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f7ab806ed20 tx=0x7f7ab001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae1eb48f8b485fc6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:25.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.492+0000 7f7ab5d9b640 1 -- 172.21.15.89:0/1699668414 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab00ab5f0 con 0x7f7ab806cb80 2023-10-30T12:42:25.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.492+0000 7f7ab5d9b640 1 -- 172.21.15.89:0/1699668414 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab00ab750 con 0x7f7ab806cb80 2023-10-30T12:42:25.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.492+0000 7f7ab5d9b640 1 -- 172.21.15.89:0/1699668414 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab00abae0 con 0x7f7ab806cb80 2023-10-30T12:42:25.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.492+0000 7f7abcd1d640 1 -- 172.21.15.89:0/1699668414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab806cb80 msgr2=0x7f7ab806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:25.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.492+0000 7f7abcd1d640 1 --2- 172.21.15.89:0/1699668414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab806cb80 0x7f7ab806cff0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f7ab806ed20 tx=0x7f7ab001c500 comp rx=0 tx=0).stop 2023-10-30T12:42:25.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.492+0000 7f7abcd1d640 1 -- 172.21.15.89:0/1699668414 shutdown_connections 2023-10-30T12:42:25.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.492+0000 7f7abcd1d640 1 --2- 172.21.15.89:0/1699668414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab806cb80 0x7f7ab806cff0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:25.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.492+0000 7f7abcd1d640 1 --2- 172.21.15.89:0/1699668414 >> v2:172.21.15.192:3300/0 conn(0x7f7ab8069e90 0x7f7ab815a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:25.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.492+0000 7f7abcd1d640 1 -- 172.21.15.89:0/1699668414 >> 172.21.15.89:0/1699668414 conn(0x7f7ab806dfd0 msgr2=0x7f7ab806e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:25.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.492+0000 7f7abcd1d640 1 -- 172.21.15.89:0/1699668414 shutdown_connections 2023-10-30T12:42:25.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.492+0000 7f7abcd1d640 1 -- 172.21.15.89:0/1699668414 wait complete. 2023-10-30T12:42:25.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.492+0000 7f7abcd1d640 1 Processor -- start 2023-10-30T12:42:25.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.492+0000 7f7abcd1d640 1 -- start start 2023-10-30T12:42:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.492+0000 7f7abcd1d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7ab8069e90 0x7f7ab8140b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.492+0000 7f7abcd1d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab806a270 0x7f7ab8147210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.492+0000 7f7abcd1d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ab806cb80 0x7f7ab8147750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.492+0000 7f7abcd1d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab814c610 con 0x7f7ab806a270 2023-10-30T12:42:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.492+0000 7f7abcd1d640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab814cb90 con 0x7f7ab806cb80 2023-10-30T12:42:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.492+0000 7f7abcd1d640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab8159050 con 0x7f7ab8069e90 2023-10-30T12:42:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.496+0000 7f7aadd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab806a270 0x7f7ab8147210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.496+0000 7f7aadd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab806a270 0x7f7ab8147210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58284/0 (socket says 172.21.15.89:58284) 2023-10-30T12:42:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.496+0000 7f7aadd9b640 1 -- 172.21.15.89:0/3399603007 learned_addr learned my addr 172.21.15.89:0/3399603007 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.496+0000 7f7ab6d9d640 1 --2- 172.21.15.89:0/3399603007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ab806cb80 0x7f7ab8147750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:25.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.496+0000 7f7aadd9b640 1 -- 172.21.15.89:0/3399603007 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7ab8069e90 msgr2=0x7f7ab8140b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:25.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.496+0000 7f7aadd9b640 1 --2- 172.21.15.89:0/3399603007 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7ab8069e90 0x7f7ab8140b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:25.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.496+0000 7f7aadd9b640 1 -- 172.21.15.89:0/3399603007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ab806cb80 msgr2=0x7f7ab8147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:25.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.496+0000 7f7aadd9b640 1 --2- 172.21.15.89:0/3399603007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ab806cb80 0x7f7ab8147750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:25.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.496+0000 7f7aadd9b640 1 -- 172.21.15.89:0/3399603007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab00094d0 con 0x7f7ab806a270 2023-10-30T12:42:25.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.496+0000 7f7aadd9b640 1 --2- 172.21.15.89:0/3399603007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab806a270 0x7f7ab8147210 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f7aa000c7f0 tx=0x7f7aa000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.496+0000 7f7aaf7fe640 1 -- 172.21.15.89:0/3399603007 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0003b80 con 0x7f7ab806a270 2023-10-30T12:42:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.496+0000 7f7aaf7fe640 1 -- 172.21.15.89:0/3399603007 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa0003ce0 con 0x7f7ab806a270 2023-10-30T12:42:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.496+0000 7f7aaf7fe640 1 -- 172.21.15.89:0/3399603007 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa000aa90 con 0x7f7ab806a270 2023-10-30T12:42:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.496+0000 7f7abcd1d640 1 -- 172.21.15.89:0/3399603007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ab814c8b0 con 0x7f7ab806a270 2023-10-30T12:42:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.496+0000 7f7abcd1d640 1 -- 172.21.15.89:0/3399603007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ab81410b0 con 0x7f7ab806a270 2023-10-30T12:42:25.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.496+0000 7f7abcd1d640 1 -- 172.21.15.89:0/3399603007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ab804e170 con 0x7f7ab806a270 2023-10-30T12:42:25.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.496+0000 7f7aaf7fe640 1 -- 172.21.15.89:0/3399603007 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7aa000abf0 con 0x7f7ab806a270 2023-10-30T12:42:25.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.500+0000 7f7aaf7fe640 1 --2- 172.21.15.89:0/3399603007 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f7a980664a0 0x7f7a98068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:25.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.500+0000 7f7aaf7fe640 1 -- 172.21.15.89:0/3399603007 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6234+0+0 (secure 0 0 0) 0x7f7aa0083fe0 con 0x7f7ab806a270 2023-10-30T12:42:25.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.500+0000 7f7ab659c640 1 --2- 172.21.15.89:0/3399603007 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f7a980664a0 0x7f7a98068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:25.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.500+0000 7f7ab659c640 1 --2- 172.21.15.89:0/3399603007 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f7a980664a0 0x7f7a98068960 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8006aa0 tx=0x7f7aa8006a30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:25.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.504+0000 7f7aaf7fe640 1 -- 172.21.15.89:0/3399603007 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7aa0051140 con 0x7f7ab806a270 2023-10-30T12:42:25.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.684+0000 7f7abcd1d640 1 -- 172.21.15.89:0/3399603007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ab814cdc0 con 0x7f7ab806a270 2023-10-30T12:42:25.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.684+0000 7f7aaf7fe640 1 -- 172.21.15.89:0/3399603007 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+6529 (secure 0 0 0) 0x7f7aa0054760 con 0x7f7ab806a270 2023-10-30T12:42:25.684 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:42:25.684 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5575,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:40:33.262671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5018,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5018":{"gid":5018,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":41,"addr":"172.21.15.192:6839/93914587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":93914587},{"type":"v1","addr":"172.21.15.192:6839","nonce":93914587}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:25.685 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 62 2023-10-30T12:42:25.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.688+0000 7f7abcd1d640 1 -- 172.21.15.89:0/3399603007 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f7a980664a0 msgr2=0x7f7a98068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:25.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.688+0000 7f7abcd1d640 1 --2- 172.21.15.89:0/3399603007 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f7a980664a0 0x7f7a98068960 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8006aa0 tx=0x7f7aa8006a30 comp rx=0 tx=0).stop 2023-10-30T12:42:25.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.688+0000 7f7abcd1d640 1 -- 172.21.15.89:0/3399603007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab806a270 msgr2=0x7f7ab8147210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:25.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.688+0000 7f7abcd1d640 1 --2- 172.21.15.89:0/3399603007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab806a270 0x7f7ab8147210 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f7aa000c7f0 tx=0x7f7aa000ccb0 comp rx=0 tx=0).stop 2023-10-30T12:42:25.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.688+0000 7f7abcd1d640 1 -- 172.21.15.89:0/3399603007 shutdown_connections 2023-10-30T12:42:25.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.688+0000 7f7abcd1d640 1 --2- 172.21.15.89:0/3399603007 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f7a980664a0 0x7f7a98068960 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:25.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.688+0000 7f7abcd1d640 1 --2- 172.21.15.89:0/3399603007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7ab806cb80 0x7f7ab8147750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:25.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.688+0000 7f7abcd1d640 1 --2- 172.21.15.89:0/3399603007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab806a270 0x7f7ab8147210 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:25.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.688+0000 7f7abcd1d640 1 --2- 172.21.15.89:0/3399603007 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7ab8069e90 0x7f7ab8140b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:25.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.688+0000 7f7abcd1d640 1 -- 172.21.15.89:0/3399603007 >> 172.21.15.89:0/3399603007 conn(0x7f7ab806dfd0 msgr2=0x7f7ab814ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:25.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.688+0000 7f7abcd1d640 1 -- 172.21.15.89:0/3399603007 shutdown_connections 2023-10-30T12:42:25.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:25.688+0000 7f7abcd1d640 1 -- 172.21.15.89:0/3399603007 wait complete. 2023-10-30T12:42:25.705 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T12:42:27.706 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:27.855 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.856+0000 7f81eae9e640 1 Processor -- start 2023-10-30T12:42:27.855 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.856+0000 7f81eae9e640 1 -- start start 2023-10-30T12:42:27.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.856+0000 7f81eae9e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81e415d6f0 0x7f81e415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:27.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.856+0000 7f81eae9e640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e40cb800 con 0x7f81e41538a0 2023-10-30T12:42:27.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.856+0000 7f81eae9e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e40cb330 con 0x7f81e40cf160 2023-10-30T12:42:27.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.856+0000 7f81eae9e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81e40cad70 con 0x7f81e415d6f0 2023-10-30T12:42:27.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.856+0000 7f81e8c3a640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f81e41538a0 0x7f81e40ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41576/0 (socket says 172.21.15.89:41576) 2023-10-30T12:42:27.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.856+0000 7f81e8c3a640 1 -- 172.21.15.89:0/232707354 learned_addr learned my addr 172.21.15.89:0/232707354 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:42:27.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.856+0000 7f81dbfff640 1 --2- 172.21.15.89:0/232707354 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81e415d6f0 0x7f81e415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:27.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.856+0000 7f81db7fe640 1 -- 172.21.15.89:0/232707354 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 947178040 0 0) 0x7f81e40cb800 con 0x7f81e41538a0 2023-10-30T12:42:27.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.856+0000 7f81db7fe640 1 -- 172.21.15.89:0/232707354 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81cc003530 con 0x7f81e41538a0 2023-10-30T12:42:27.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.856+0000 7f81db7fe640 1 -- 172.21.15.89:0/232707354 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373716262 0 0) 0x7f81e40cb330 con 0x7f81e40cf160 2023-10-30T12:42:27.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.856+0000 7f81db7fe640 1 -- 172.21.15.89:0/232707354 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81e40cb800 con 0x7f81e40cf160 2023-10-30T12:42:27.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.856+0000 7f81db7fe640 1 -- 172.21.15.89:0/232707354 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 840941280 0 0) 0x7f81cc003530 con 0x7f81e41538a0 2023-10-30T12:42:27.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.856+0000 7f81db7fe640 1 -- 172.21.15.89:0/232707354 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81e415d6f0 msgr2=0x7f81e415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:27.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.856+0000 7f81db7fe640 1 --2- 172.21.15.89:0/232707354 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81e415d6f0 0x7f81e415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:27.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.856+0000 7f81db7fe640 1 -- 172.21.15.89:0/232707354 >> v1:172.21.15.192:6789/0 conn(0x7f81e40cf160 legacy=0x7f81e4164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:27.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.856+0000 7f81db7fe640 1 -- 172.21.15.89:0/232707354 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e40cb050 con 0x7f81e41538a0 2023-10-30T12:42:27.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.856+0000 7f81db7fe640 1 -- 172.21.15.89:0/232707354 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f81e0001680 con 0x7f81e41538a0 2023-10-30T12:42:27.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.856+0000 7f81db7fe640 1 -- 172.21.15.89:0/232707354 >> v1:172.21.15.89:6789/0 conn(0x7f81e41538a0 legacy=0x7f81e40ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:27.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.860+0000 7f81db7fe640 1 --2- 172.21.15.89:0/232707354 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81cc003b40 0x7f81cc003f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:27.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.860+0000 7f81db7fe640 1 --2- 172.21.15.89:0/232707354 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81e415d6f0 0x7f81cc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:27.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.860+0000 7f81db7fe640 1 --2- 172.21.15.89:0/232707354 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81cc004d10 0x7f81cc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:27.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.860+0000 7f81db7fe640 1 -- 172.21.15.89:0/232707354 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e40cad70 con 0x7f81cc003b40 2023-10-30T12:42:27.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.860+0000 7f81db7fe640 1 -- 172.21.15.89:0/232707354 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81cc00b9c0 con 0x7f81cc004d10 2023-10-30T12:42:27.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.860+0000 7f81db7fe640 1 -- 172.21.15.89:0/232707354 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81cc00bb80 con 0x7f81e415d6f0 2023-10-30T12:42:27.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.860+0000 7f81e8c3a640 1 --2- 172.21.15.89:0/232707354 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81cc003b40 0x7f81cc003f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:27.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.860+0000 7f81e943b640 1 --2- 172.21.15.89:0/232707354 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81cc004d10 0x7f81cc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:27.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.860+0000 7f81dbfff640 1 --2- 172.21.15.89:0/232707354 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81e415d6f0 0x7f81cc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:27.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.860+0000 7f81e8c3a640 1 -- 172.21.15.89:0/232707354 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81e415d6f0 msgr2=0x7f81cc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:27.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.860+0000 7f81e8c3a640 1 --2- 172.21.15.89:0/232707354 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81e415d6f0 0x7f81cc0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:27.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.860+0000 7f81e8c3a640 1 -- 172.21.15.89:0/232707354 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81cc004d10 msgr2=0x7f81cc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:27.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.860+0000 7f81e8c3a640 1 --2- 172.21.15.89:0/232707354 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81cc004d10 0x7f81cc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:27.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.860+0000 7f81e8c3a640 1 -- 172.21.15.89:0/232707354 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81cc003530 con 0x7f81cc003b40 2023-10-30T12:42:27.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.860+0000 7f81e8c3a640 1 --2- 172.21.15.89:0/232707354 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81cc003b40 0x7f81cc003f20 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f81e0005000 tx=0x7f81e001d100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fad7dfe079fa6bd5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:27.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.860+0000 7f81db7fe640 1 -- 172.21.15.89:0/232707354 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e0005ae0 con 0x7f81cc003b40 2023-10-30T12:42:27.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.860+0000 7f81db7fe640 1 -- 172.21.15.89:0/232707354 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81e00a8030 con 0x7f81cc003b40 2023-10-30T12:42:27.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.860+0000 7f81db7fe640 1 -- 172.21.15.89:0/232707354 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e0005ea0 con 0x7f81cc003b40 2023-10-30T12:42:27.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.864+0000 7f81eae9e640 1 -- 172.21.15.89:0/232707354 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81cc003b40 msgr2=0x7f81cc003f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:27.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.864+0000 7f81eae9e640 1 --2- 172.21.15.89:0/232707354 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81cc003b40 0x7f81cc003f20 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f81e0005000 tx=0x7f81e001d100 comp rx=0 tx=0).stop 2023-10-30T12:42:27.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.864+0000 7f81e943b640 1 -- 172.21.15.89:0/232707354 reap_dead start 2023-10-30T12:42:27.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.864+0000 7f81eae9e640 1 -- 172.21.15.89:0/232707354 shutdown_connections 2023-10-30T12:42:27.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.864+0000 7f81eae9e640 1 -- 172.21.15.89:0/232707354 >> 172.21.15.89:0/232707354 conn(0x7f81e406bfe0 msgr2=0x7f81e406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:27.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.864+0000 7f81eae9e640 1 -- 172.21.15.89:0/232707354 shutdown_connections 2023-10-30T12:42:27.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.864+0000 7f81eae9e640 1 -- 172.21.15.89:0/232707354 wait complete. 2023-10-30T12:42:27.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.864+0000 7f81eae9e640 1 Processor -- start 2023-10-30T12:42:27.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.864+0000 7f81eae9e640 1 -- start start 2023-10-30T12:42:27.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.864+0000 7f81eae9e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81cc003b40 0x7f81e41e5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:27.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.864+0000 7f81eae9e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81e40b4ef0 0x7f81e414c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:27.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.864+0000 7f81eae9e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81e41e5650 0x7f81e414cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:27.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.864+0000 7f81eae9e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e414dd00 con 0x7f81e40b4ef0 2023-10-30T12:42:27.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.864+0000 7f81eae9e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e41f2c60 con 0x7f81e41e5650 2023-10-30T12:42:27.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.864+0000 7f81eae9e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81e41f3190 con 0x7f81cc003b40 2023-10-30T12:42:27.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.864+0000 7f81dbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81e40b4ef0 0x7f81e414c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:27.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.864+0000 7f81dbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81e40b4ef0 0x7f81e414c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58304/0 (socket says 172.21.15.89:58304) 2023-10-30T12:42:27.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.864+0000 7f81dbfff640 1 -- 172.21.15.89:0/3570603293 learned_addr learned my addr 172.21.15.89:0/3570603293 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:27.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.864+0000 7f81e8c3a640 1 --2- 172.21.15.89:0/3570603293 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81cc003b40 0x7f81e41e5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:27.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.864+0000 7f81e943b640 1 --2- 172.21.15.89:0/3570603293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81e41e5650 0x7f81e414cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:27.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.868+0000 7f81dbfff640 1 -- 172.21.15.89:0/3570603293 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81cc003b40 msgr2=0x7f81e41e5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:27.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.868+0000 7f81dbfff640 1 --2- 172.21.15.89:0/3570603293 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81cc003b40 0x7f81e41e5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:27.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.868+0000 7f81dbfff640 1 -- 172.21.15.89:0/3570603293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81e41e5650 msgr2=0x7f81e414cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:27.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.868+0000 7f81dbfff640 1 --2- 172.21.15.89:0/3570603293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81e41e5650 0x7f81e414cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:27.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.868+0000 7f81dbfff640 1 -- 172.21.15.89:0/3570603293 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e41f05f0 con 0x7f81e40b4ef0 2023-10-30T12:42:27.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.868+0000 7f81dbfff640 1 --2- 172.21.15.89:0/3570603293 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81e40b4ef0 0x7f81e414c5f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f81d4011610 tx=0x7f81d4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:27.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.868+0000 7f81d97fa640 1 -- 172.21.15.89:0/3570603293 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d400f8f0 con 0x7f81e40b4ef0 2023-10-30T12:42:27.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.868+0000 7f81d97fa640 1 -- 172.21.15.89:0/3570603293 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d400fa50 con 0x7f81e40b4ef0 2023-10-30T12:42:27.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.868+0000 7f81eae9e640 1 -- 172.21.15.89:0/3570603293 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81e40c52e0 con 0x7f81e40b4ef0 2023-10-30T12:42:27.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.868+0000 7f81d97fa640 1 -- 172.21.15.89:0/3570603293 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d400fdc0 con 0x7f81e40b4ef0 2023-10-30T12:42:27.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.868+0000 7f81eae9e640 1 -- 172.21.15.89:0/3570603293 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81e41f0280 con 0x7f81e40b4ef0 2023-10-30T12:42:27.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.868+0000 7f81eae9e640 1 -- 172.21.15.89:0/3570603293 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81e41f0780 con 0x7f81e40b4ef0 2023-10-30T12:42:27.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.868+0000 7f81d97fa640 1 -- 172.21.15.89:0/3570603293 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f81d4015460 con 0x7f81e40b4ef0 2023-10-30T12:42:27.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.868+0000 7f81d97fa640 1 --2- 172.21.15.89:0/3570603293 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f81d00664d0 0x7f81d0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:27.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.868+0000 7f81d97fa640 1 -- 172.21.15.89:0/3570603293 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6320+0+0 (secure 0 0 0) 0x7f81d40848c0 con 0x7f81e40b4ef0 2023-10-30T12:42:27.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.872+0000 7f81e8c3a640 1 --2- 172.21.15.89:0/3570603293 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f81d00664d0 0x7f81d0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:27.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.872+0000 7f81e8c3a640 1 --2- 172.21.15.89:0/3570603293 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f81d00664d0 0x7f81d0068990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f81e00ab000 tx=0x7f81e00b0040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:27.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:27.872+0000 7f81d97fa640 1 -- 172.21.15.89:0/3570603293 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f81d4051af0 con 0x7f81e40b4ef0 2023-10-30T12:42:28.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.052+0000 7f81eae9e640 1 -- 172.21.15.89:0/3570603293 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81e404e170 con 0x7f81e40b4ef0 2023-10-30T12:42:28.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.052+0000 7f81d97fa640 1 -- 172.21.15.89:0/3570603293 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+5737 (secure 0 0 0) 0x7f81d4055110 con 0x7f81e40b4ef0 2023-10-30T12:42:28.053 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:42:28.054 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:42:26.158905+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:28.054 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 63 2023-10-30T12:42:28.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.056+0000 7f81eae9e640 1 -- 172.21.15.89:0/3570603293 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f81d00664d0 msgr2=0x7f81d0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:28.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.056+0000 7f81eae9e640 1 --2- 172.21.15.89:0/3570603293 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f81d00664d0 0x7f81d0068990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f81e00ab000 tx=0x7f81e00b0040 comp rx=0 tx=0).stop 2023-10-30T12:42:28.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.056+0000 7f81eae9e640 1 -- 172.21.15.89:0/3570603293 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81e40b4ef0 msgr2=0x7f81e414c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:28.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.056+0000 7f81eae9e640 1 --2- 172.21.15.89:0/3570603293 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81e40b4ef0 0x7f81e414c5f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f81d4011610 tx=0x7f81d4011ad0 comp rx=0 tx=0).stop 2023-10-30T12:42:28.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.056+0000 7f81eae9e640 1 -- 172.21.15.89:0/3570603293 shutdown_connections 2023-10-30T12:42:28.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.056+0000 7f81eae9e640 1 --2- 172.21.15.89:0/3570603293 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f81d00664d0 0x7f81d0068990 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:28.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.056+0000 7f81eae9e640 1 --2- 172.21.15.89:0/3570603293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81e41e5650 0x7f81e414cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:28.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.056+0000 7f81eae9e640 1 --2- 172.21.15.89:0/3570603293 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81e40b4ef0 0x7f81e414c5f0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:28.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.056+0000 7f81eae9e640 1 --2- 172.21.15.89:0/3570603293 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f81cc003b40 0x7f81e41e5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:28.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.056+0000 7f81eae9e640 1 -- 172.21.15.89:0/3570603293 >> 172.21.15.89:0/3570603293 conn(0x7f81e406bfe0 msgr2=0x7f81e41510d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:28.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.056+0000 7f81eae9e640 1 -- 172.21.15.89:0/3570603293 shutdown_connections 2023-10-30T12:42:28.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.056+0000 7f81eae9e640 1 -- 172.21.15.89:0/3570603293 wait complete. 2023-10-30T12:42:28.074 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-10-30T12:42:28.075 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:42:28.075 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:28.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.212+0000 7fbdb91f8640 1 Processor -- start 2023-10-30T12:42:28.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.212+0000 7fbdb91f8640 1 -- start start 2023-10-30T12:42:28.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.216+0000 7fbdb91f8640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbdb4153bf0 0x7fbdb4071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:28.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.216+0000 7fbdb91f8640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbdb40c6fe0 0x7fbdb40c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:28.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.216+0000 7fbdb91f8640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdb40700b0 con 0x7fbdb40c79e0 2023-10-30T12:42:28.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.216+0000 7fbdb91f8640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdb40c6810 con 0x7fbdb4153bf0 2023-10-30T12:42:28.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.216+0000 7fbdb91f8640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb4159900 con 0x7fbdb40c6fe0 2023-10-30T12:42:28.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.216+0000 7fbdabfff640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbdb40c6fe0 0x7fbdb40c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:28.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.216+0000 7fbdb2d9d640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbdb4153bf0 0x7fbdb4071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:28.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.216+0000 7fbdb359e640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fbdb40c79e0 0x7fbdb40cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:33748/0 (socket says 172.21.15.89:33748) 2023-10-30T12:42:28.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.216+0000 7fbdb359e640 1 -- 172.21.15.89:0/882348153 learned_addr learned my addr 172.21.15.89:0/882348153 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:42:28.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.216+0000 7fbdb2d9d640 1 --2- 172.21.15.89:0/882348153 >> v2:172.21.15.192:3300/0 conn(0x7fbdb4153bf0 0x7fbdb4071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:54138/0 (socket says 172.21.15.89:54138) 2023-10-30T12:42:28.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.216+0000 7fbdabfff640 1 -- 172.21.15.89:0/882348153 >> v2:172.21.15.192:3300/0 conn(0x7fbdb4153bf0 msgr2=0x7fbdb4071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:28.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.216+0000 7fbdabfff640 1 --2- 172.21.15.89:0/882348153 >> v2:172.21.15.192:3300/0 conn(0x7fbdb4153bf0 0x7fbdb4071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:28.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.216+0000 7fbdabfff640 1 -- 172.21.15.89:0/882348153 >> v1:172.21.15.192:6789/0 conn(0x7fbdb40c79e0 legacy=0x7fbdb40cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:28.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.216+0000 7fbdabfff640 1 -- 172.21.15.89:0/882348153 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb40c6530 con 0x7fbdb40c6fe0 2023-10-30T12:42:28.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.216+0000 7fbdb259c640 1 -- 172.21.15.89:0/882348153 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1281798116 0 0) 0x7fbdb40700b0 con 0x7fbdb40c79e0 2023-10-30T12:42:28.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.220+0000 7fbdabfff640 1 --2- 172.21.15.89:0/882348153 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbdb40c6fe0 0x7fbdb40c74a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c0097c0 tx=0x7fbd9c01ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85c20279715b385e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:28.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.220+0000 7fbdb259c640 1 -- 172.21.15.89:0/882348153 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c0a8970 con 0x7fbdb40c6fe0 2023-10-30T12:42:28.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.220+0000 7fbdb259c640 1 -- 172.21.15.89:0/882348153 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd9c0a8ad0 con 0x7fbdb40c6fe0 2023-10-30T12:42:28.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.220+0000 7fbdb259c640 1 -- 172.21.15.89:0/882348153 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c0a8e60 con 0x7fbdb40c6fe0 2023-10-30T12:42:28.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.220+0000 7fbdb91f8640 1 -- 172.21.15.89:0/882348153 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbdb40c6fe0 msgr2=0x7fbdb40c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:28.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.220+0000 7fbdb91f8640 1 --2- 172.21.15.89:0/882348153 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbdb40c6fe0 0x7fbdb40c74a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c0097c0 tx=0x7fbd9c01ec70 comp rx=0 tx=0).stop 2023-10-30T12:42:28.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.220+0000 7fbdb91f8640 1 -- 172.21.15.89:0/882348153 shutdown_connections 2023-10-30T12:42:28.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.220+0000 7fbdb91f8640 1 --2- 172.21.15.89:0/882348153 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbdb40c6fe0 0x7fbdb40c74a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:28.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.220+0000 7fbdb91f8640 1 --2- 172.21.15.89:0/882348153 >> v2:172.21.15.192:3300/0 conn(0x7fbdb4153bf0 0x7fbdb4071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:28.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.220+0000 7fbdb91f8640 1 -- 172.21.15.89:0/882348153 >> 172.21.15.89:0/882348153 conn(0x7fbdb406c6e0 msgr2=0x7fbdb4069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:28.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.220+0000 7fbdb91f8640 1 -- 172.21.15.89:0/882348153 shutdown_connections 2023-10-30T12:42:28.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.220+0000 7fbdb91f8640 1 -- 172.21.15.89:0/882348153 wait complete. 2023-10-30T12:42:28.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.224+0000 7fbdb91f8640 1 Processor -- start 2023-10-30T12:42:28.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.224+0000 7fbdb91f8640 1 -- start start 2023-10-30T12:42:28.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.224+0000 7fbdb91f8640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbdb40c6fe0 0x7fbdb40c32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:28.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.224+0000 7fbdb91f8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdb40c79e0 0x7fbdb414af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:28.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.224+0000 7fbdb91f8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbdb4153bf0 0x7fbdb414eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:28.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.224+0000 7fbdb91f8640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb40711f0 con 0x7fbdb40c79e0 2023-10-30T12:42:28.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.224+0000 7fbdb91f8640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb41f2180 con 0x7fbdb4153bf0 2023-10-30T12:42:28.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.224+0000 7fbdb91f8640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb4158bc0 con 0x7fbdb40c6fe0 2023-10-30T12:42:28.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.224+0000 7fbdabfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdb40c79e0 0x7fbdb414af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:28.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.224+0000 7fbdabfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdb40c79e0 0x7fbdb414af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58312/0 (socket says 172.21.15.89:58312) 2023-10-30T12:42:28.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.224+0000 7fbdabfff640 1 -- 172.21.15.89:0/1217238955 learned_addr learned my addr 172.21.15.89:0/1217238955 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:28.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.224+0000 7fbdb359e640 1 --2- 172.21.15.89:0/1217238955 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbdb4153bf0 0x7fbdb414eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:28.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.224+0000 7fbdb2d9d640 1 --2- 172.21.15.89:0/1217238955 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbdb40c6fe0 0x7fbdb40c32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:28.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.224+0000 7fbdabfff640 1 -- 172.21.15.89:0/1217238955 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbdb40c6fe0 msgr2=0x7fbdb40c32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:28.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.224+0000 7fbdabfff640 1 --2- 172.21.15.89:0/1217238955 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbdb40c6fe0 0x7fbdb40c32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:28.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.224+0000 7fbdabfff640 1 -- 172.21.15.89:0/1217238955 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbdb4153bf0 msgr2=0x7fbdb414eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:28.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.224+0000 7fbdabfff640 1 --2- 172.21.15.89:0/1217238955 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbdb4153bf0 0x7fbdb414eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:28.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.224+0000 7fbdabfff640 1 -- 172.21.15.89:0/1217238955 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9c009420 con 0x7fbdb40c79e0 2023-10-30T12:42:28.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.224+0000 7fbdb359e640 1 --2- 172.21.15.89:0/1217238955 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbdb4153bf0 0x7fbdb414eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:42:28.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.224+0000 7fbdb2d9d640 1 --2- 172.21.15.89:0/1217238955 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbdb40c6fe0 0x7fbdb40c32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:42:28.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.224+0000 7fbdabfff640 1 --2- 172.21.15.89:0/1217238955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdb40c79e0 0x7fbdb414af00 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c01c8a0 tx=0x7fbd9c01d4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:28.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.224+0000 7fbdab7fe640 1 -- 172.21.15.89:0/1217238955 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c0b1e00 con 0x7fbdb40c79e0 2023-10-30T12:42:28.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.224+0000 7fbdab7fe640 1 -- 172.21.15.89:0/1217238955 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd9c006e20 con 0x7fbdb40c79e0 2023-10-30T12:42:28.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.224+0000 7fbdb91f8640 1 -- 172.21.15.89:0/1217238955 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdb4071400 con 0x7fbdb40c79e0 2023-10-30T12:42:28.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.224+0000 7fbdab7fe640 1 -- 172.21.15.89:0/1217238955 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c004230 con 0x7fbdb40c79e0 2023-10-30T12:42:28.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.224+0000 7fbdb91f8640 1 -- 172.21.15.89:0/1217238955 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdb4156ee0 con 0x7fbdb40c79e0 2023-10-30T12:42:28.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.224+0000 7fbdb91f8640 1 -- 172.21.15.89:0/1217238955 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdb4156d40 con 0x7fbdb40c79e0 2023-10-30T12:42:28.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.228+0000 7fbdab7fe640 1 -- 172.21.15.89:0/1217238955 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbd9c004390 con 0x7fbdb40c79e0 2023-10-30T12:42:28.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.228+0000 7fbdab7fe640 1 --2- 172.21.15.89:0/1217238955 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbd940664a0 0x7fbd94068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:28.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.228+0000 7fbdab7fe640 1 -- 172.21.15.89:0/1217238955 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6320+0+0 (secure 0 0 0) 0x7fbd9c1219e0 con 0x7fbdb40c79e0 2023-10-30T12:42:28.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.228+0000 7fbdb2d9d640 1 --2- 172.21.15.89:0/1217238955 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbd940664a0 0x7fbd94068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:28.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.228+0000 7fbdb2d9d640 1 --2- 172.21.15.89:0/1217238955 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbd940664a0 0x7fbd94068960 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbda4009b10 tx=0x7fbda4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:28.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.232+0000 7fbdab7fe640 1 -- 172.21.15.89:0/1217238955 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbdb4156d40 con 0x7fbdb40c79e0 2023-10-30T12:42:28.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.412+0000 7fbdb91f8640 1 -- 172.21.15.89:0/1217238955 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbdb4156d40 con 0x7fbdb40c79e0 2023-10-30T12:42:28.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.412+0000 7fbdab7fe640 1 -- 172.21.15.89:0/1217238955 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+5737 (secure 0 0 0) 0x7fbdb4156d40 con 0x7fbdb40c79e0 2023-10-30T12:42:28.413 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:42:28.413 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:42:26.158905+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:28.414 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 63 2023-10-30T12:42:28.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.416+0000 7fbdb91f8640 1 -- 172.21.15.89:0/1217238955 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbd940664a0 msgr2=0x7fbd94068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:28.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.416+0000 7fbdb91f8640 1 --2- 172.21.15.89:0/1217238955 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbd940664a0 0x7fbd94068960 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbda4009b10 tx=0x7fbda4008040 comp rx=0 tx=0).stop 2023-10-30T12:42:28.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.416+0000 7fbdb91f8640 1 -- 172.21.15.89:0/1217238955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdb40c79e0 msgr2=0x7fbdb414af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:28.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.416+0000 7fbdb91f8640 1 --2- 172.21.15.89:0/1217238955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdb40c79e0 0x7fbdb414af00 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c01c8a0 tx=0x7fbd9c01d4b0 comp rx=0 tx=0).stop 2023-10-30T12:42:28.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.416+0000 7fbdb91f8640 1 -- 172.21.15.89:0/1217238955 shutdown_connections 2023-10-30T12:42:28.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.416+0000 7fbdb91f8640 1 --2- 172.21.15.89:0/1217238955 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbd940664a0 0x7fbd94068960 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:28.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.416+0000 7fbdb91f8640 1 --2- 172.21.15.89:0/1217238955 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbdb4153bf0 0x7fbdb414eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:28.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.416+0000 7fbdb91f8640 1 --2- 172.21.15.89:0/1217238955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdb40c79e0 0x7fbdb414af00 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:28.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.416+0000 7fbdb91f8640 1 --2- 172.21.15.89:0/1217238955 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbdb40c6fe0 0x7fbdb40c32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:28.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.416+0000 7fbdb91f8640 1 -- 172.21.15.89:0/1217238955 >> 172.21.15.89:0/1217238955 conn(0x7fbdb406c6e0 msgr2=0x7fbdb40c7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:28.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.416+0000 7fbdb91f8640 1 -- 172.21.15.89:0/1217238955 shutdown_connections 2023-10-30T12:42:28.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:28.416+0000 7fbdb91f8640 1 -- 172.21.15.89:0/1217238955 wait complete. 2023-10-30T12:42:28.434 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 63, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:50.729245+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 63, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5129": { "addr": "172.21.15.192:6837/1413360446", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6836", "nonce": 1413360446, "type": "v2" }, { "addr": "172.21.15.192:6837", "nonce": 1413360446, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5129, "incarnation": 57, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:active", "state_seq": 40 }, "gid_5139": { "addr": "172.21.15.89:6837/951276252", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6836", "nonce": 951276252, "type": "v2" }, { "addr": "172.21.15.89:6837", "nonce": 951276252, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5139, "incarnation": 33, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 29 }, "gid_5575": { "addr": "172.21.15.192:6835/501291700", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6834", "nonce": 501291700, "type": "v2" }, { "addr": "172.21.15.192:6835", "nonce": 501291700, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5575, "incarnation": 63, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 61, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:42:26.158905+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5575, "mds_1": 5139, "mds_2": 5129 } } } ], "standbys": [ { "addr": "172.21.15.89:6834/2909790173", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6832", "nonce": 2909790173, "type": "v2" }, { "addr": "172.21.15.89:6834", "nonce": 2909790173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.89:6835/2387357615", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6833", "nonce": 2387357615, "type": "v2" }, { "addr": "172.21.15.89:6835", "nonce": 2387357615, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4239, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:42:30.435 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:30.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.584+0000 7f1ad110f640 1 Processor -- start 2023-10-30T12:42:30.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.584+0000 7f1ad110f640 1 -- start start 2023-10-30T12:42:30.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.584+0000 7f1ad110f640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1acc06f7a0 0x7f1acc06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:30.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.584+0000 7f1ad110f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1acc0c6bd0 0x7f1acc069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:30.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.584+0000 7f1ad110f640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1acc06a3d0 con 0x7f1acc071110 2023-10-30T12:42:30.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.584+0000 7f1ad110f640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1acc0700c0 con 0x7f1acc06f7a0 2023-10-30T12:42:30.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.584+0000 7f1ad110f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc066cf0 con 0x7f1acc0c6bd0 2023-10-30T12:42:30.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.584+0000 7f1acb59e640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f1acc071110 0x7f1acc158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:35154/0 (socket says 172.21.15.89:35154) 2023-10-30T12:42:30.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.584+0000 7f1acb59e640 1 -- 172.21.15.89:0/1260572054 learned_addr learned my addr 172.21.15.89:0/1260572054 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:42:30.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.584+0000 7f1ac259c640 1 --2- 172.21.15.89:0/1260572054 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1acc0c6bd0 0x7f1acc069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:30.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.584+0000 7f1acad9d640 1 --2- 172.21.15.89:0/1260572054 >> v2:172.21.15.192:3300/0 conn(0x7f1acc06f7a0 0x7f1acc06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:30.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.584+0000 7f1aca59c640 1 -- 172.21.15.89:0/1260572054 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2251240230 0 0) 0x7f1acc06a3d0 con 0x7f1acc071110 2023-10-30T12:42:30.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.584+0000 7f1aca59c640 1 -- 172.21.15.89:0/1260572054 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ab8003530 con 0x7f1acc071110 2023-10-30T12:42:30.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.584+0000 7f1aca59c640 1 -- 172.21.15.89:0/1260572054 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1241327193 0 0) 0x7f1ab8003530 con 0x7f1acc071110 2023-10-30T12:42:30.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.584+0000 7f1aca59c640 1 -- 172.21.15.89:0/1260572054 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1acc0c6bd0 msgr2=0x7f1acc069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:30.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.584+0000 7f1aca59c640 1 --2- 172.21.15.89:0/1260572054 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1acc0c6bd0 0x7f1acc069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:30.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.584+0000 7f1aca59c640 1 -- 172.21.15.89:0/1260572054 >> v2:172.21.15.192:3300/0 conn(0x7f1acc06f7a0 msgr2=0x7f1acc06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:30.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.584+0000 7f1aca59c640 1 --2- 172.21.15.89:0/1260572054 >> v2:172.21.15.192:3300/0 conn(0x7f1acc06f7a0 0x7f1acc06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:30.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.584+0000 7f1aca59c640 1 -- 172.21.15.89:0/1260572054 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1acc0c70d0 con 0x7f1acc071110 2023-10-30T12:42:30.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.584+0000 7f1aca59c640 1 -- 172.21.15.89:0/1260572054 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f1ac4001a00 con 0x7f1acc071110 2023-10-30T12:42:30.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.584+0000 7f1aca59c640 1 -- 172.21.15.89:0/1260572054 >> v1:172.21.15.89:6789/0 conn(0x7f1acc071110 legacy=0x7f1acc158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:30.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.584+0000 7f1aca59c640 1 --2- 172.21.15.89:0/1260572054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ab8003980 0x7f1ab8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:30.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.584+0000 7f1aca59c640 1 --2- 172.21.15.89:0/1260572054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ab80047d0 0x7f1ab8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:30.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.584+0000 7f1aca59c640 1 --2- 172.21.15.89:0/1260572054 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1acc0c6bd0 0x7f1ab8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:30.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.584+0000 7f1aca59c640 1 -- 172.21.15.89:0/1260572054 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc0700c0 con 0x7f1ab8003980 2023-10-30T12:42:30.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.584+0000 7f1aca59c640 1 -- 172.21.15.89:0/1260572054 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc066cf0 con 0x7f1ab80047d0 2023-10-30T12:42:30.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.584+0000 7f1aca59c640 1 -- 172.21.15.89:0/1260572054 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab800b990 con 0x7f1acc0c6bd0 2023-10-30T12:42:30.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.584+0000 7f1acad9d640 1 --2- 172.21.15.89:0/1260572054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ab8003980 0x7f1ab8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:30.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.588+0000 7f1ac259c640 1 --2- 172.21.15.89:0/1260572054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ab80047d0 0x7f1ab8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:30.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.588+0000 7f1acb59e640 1 --2- 172.21.15.89:0/1260572054 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1acc0c6bd0 0x7f1ab8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:30.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.588+0000 7f1acad9d640 1 -- 172.21.15.89:0/1260572054 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1acc0c6bd0 msgr2=0x7f1ab8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:30.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.588+0000 7f1acad9d640 1 --2- 172.21.15.89:0/1260572054 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1acc0c6bd0 0x7f1ab8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:30.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.588+0000 7f1acad9d640 1 -- 172.21.15.89:0/1260572054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ab80047d0 msgr2=0x7f1ab8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:30.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.588+0000 7f1acad9d640 1 --2- 172.21.15.89:0/1260572054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ab80047d0 0x7f1ab8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:30.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.588+0000 7f1acad9d640 1 -- 172.21.15.89:0/1260572054 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab8003530 con 0x7f1ab8003980 2023-10-30T12:42:30.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.588+0000 7f1acad9d640 1 --2- 172.21.15.89:0/1260572054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ab8003980 0x7f1ab8003da0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4009ae0 tx=0x7f1ab401ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f3ff5a84334ab1a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:30.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.588+0000 7f1aca59c640 1 -- 172.21.15.89:0/1260572054 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab40a64a0 con 0x7f1ab8003980 2023-10-30T12:42:30.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.588+0000 7f1aca59c640 1 -- 172.21.15.89:0/1260572054 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab40a6600 con 0x7f1ab8003980 2023-10-30T12:42:30.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.588+0000 7f1aca59c640 1 -- 172.21.15.89:0/1260572054 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab40a69b0 con 0x7f1ab8003980 2023-10-30T12:42:30.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.588+0000 7f1ad110f640 1 -- 172.21.15.89:0/1260572054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ab8003980 msgr2=0x7f1ab8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:30.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.588+0000 7f1ad110f640 1 --2- 172.21.15.89:0/1260572054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ab8003980 0x7f1ab8003da0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4009ae0 tx=0x7f1ab401ca90 comp rx=0 tx=0).stop 2023-10-30T12:42:30.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.588+0000 7f1acb59e640 1 -- 172.21.15.89:0/1260572054 reap_dead start 2023-10-30T12:42:30.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.588+0000 7f1ad110f640 1 -- 172.21.15.89:0/1260572054 shutdown_connections 2023-10-30T12:42:30.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.588+0000 7f1ad110f640 1 -- 172.21.15.89:0/1260572054 >> 172.21.15.89:0/1260572054 conn(0x7f1acc0c7910 msgr2=0x7f1acc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:30.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.592+0000 7f1ad110f640 1 -- 172.21.15.89:0/1260572054 shutdown_connections 2023-10-30T12:42:30.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.592+0000 7f1ad110f640 1 -- 172.21.15.89:0/1260572054 wait complete. 2023-10-30T12:42:30.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.592+0000 7f1ad110f640 1 Processor -- start 2023-10-30T12:42:30.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.592+0000 7f1ad110f640 1 -- start start 2023-10-30T12:42:30.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.592+0000 7f1ad110f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ab8003980 0x7f1acc0c1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:30.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.592+0000 7f1ad110f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1acc157d40 0x7f1acc155550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:30.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.592+0000 7f1ad110f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1acc154c70 0x7f1acc0c3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:30.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.592+0000 7f1ad110f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc0c0b70 con 0x7f1ab8003980 2023-10-30T12:42:30.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.592+0000 7f1ad110f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc0c0280 con 0x7f1acc154c70 2023-10-30T12:42:30.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.592+0000 7f1ad110f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc1f6a70 con 0x7f1acc157d40 2023-10-30T12:42:30.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.592+0000 7f1acad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ab8003980 0x7f1acc0c1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:30.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.592+0000 7f1acad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ab8003980 0x7f1acc0c1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38160/0 (socket says 172.21.15.89:38160) 2023-10-30T12:42:30.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.592+0000 7f1acad9d640 1 -- 172.21.15.89:0/932545452 learned_addr learned my addr 172.21.15.89:0/932545452 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:30.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.592+0000 7f1ac259c640 1 --2- 172.21.15.89:0/932545452 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1acc157d40 0x7f1acc155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:30.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.592+0000 7f1acb59e640 1 --2- 172.21.15.89:0/932545452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1acc154c70 0x7f1acc0c3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:30.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.592+0000 7f1acad9d640 1 -- 172.21.15.89:0/932545452 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1acc157d40 msgr2=0x7f1acc155550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:30.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.592+0000 7f1acad9d640 1 --2- 172.21.15.89:0/932545452 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1acc157d40 0x7f1acc155550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:30.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.592+0000 7f1acad9d640 1 -- 172.21.15.89:0/932545452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1acc154c70 msgr2=0x7f1acc0c3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:30.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.592+0000 7f1acad9d640 1 --2- 172.21.15.89:0/932545452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1acc154c70 0x7f1acc0c3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:30.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.592+0000 7f1acad9d640 1 -- 172.21.15.89:0/932545452 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1acc0c0520 con 0x7f1ab8003980 2023-10-30T12:42:30.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.592+0000 7f1acb59e640 1 --2- 172.21.15.89:0/932545452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1acc154c70 0x7f1acc0c3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:42:30.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.596+0000 7f1acad9d640 1 --2- 172.21.15.89:0/932545452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ab8003980 0x7f1acc0c1000 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4009ae0 tx=0x7f1ab8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:30.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.596+0000 7f1ac3fff640 1 -- 172.21.15.89:0/932545452 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab40b5070 con 0x7f1ab8003980 2023-10-30T12:42:30.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.596+0000 7f1ac3fff640 1 -- 172.21.15.89:0/932545452 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab400f000 con 0x7f1ab8003980 2023-10-30T12:42:30.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.596+0000 7f1ad110f640 1 -- 172.21.15.89:0/932545452 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1acc0c0800 con 0x7f1ab8003980 2023-10-30T12:42:30.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.596+0000 7f1ac3fff640 1 -- 172.21.15.89:0/932545452 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab40a64a0 con 0x7f1ab8003980 2023-10-30T12:42:30.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.596+0000 7f1ad110f640 1 -- 172.21.15.89:0/932545452 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1acc0b0980 con 0x7f1ab8003980 2023-10-30T12:42:30.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.596+0000 7f1ad110f640 1 -- 172.21.15.89:0/932545452 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1acc04e170 con 0x7f1ab8003980 2023-10-30T12:42:30.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.596+0000 7f1ac3fff640 1 -- 172.21.15.89:0/932545452 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1ab40b9070 con 0x7f1ab8003980 2023-10-30T12:42:30.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.596+0000 7f1ac3fff640 1 --2- 172.21.15.89:0/932545452 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1aac0664d0 0x7f1aac068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:30.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.596+0000 7f1ac3fff640 1 -- 172.21.15.89:0/932545452 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6320+0+0 (secure 0 0 0) 0x7f1ab411dc00 con 0x7f1ab8003980 2023-10-30T12:42:30.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.596+0000 7f1ac259c640 1 --2- 172.21.15.89:0/932545452 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1aac0664d0 0x7f1aac068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:30.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.600+0000 7f1ac259c640 1 --2- 172.21.15.89:0/932545452 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1aac0664d0 0x7f1aac068990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1acc0c17b0 tx=0x7f1abc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:30.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.600+0000 7f1ac3fff640 1 -- 172.21.15.89:0/932545452 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1ab40eae30 con 0x7f1ab8003980 2023-10-30T12:42:30.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.780+0000 7f1ad110f640 1 -- 172.21.15.89:0/932545452 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1acc0c0990 con 0x7f1ab8003980 2023-10-30T12:42:30.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.780+0000 7f1ac3fff640 1 -- 172.21.15.89:0/932545452 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+5737 (secure 0 0 0) 0x7f1ab40c0070 con 0x7f1ab8003980 2023-10-30T12:42:30.781 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:42:30.781 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:42:26.158905+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:30.782 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 63 2023-10-30T12:42:30.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.784+0000 7f1ad110f640 1 -- 172.21.15.89:0/932545452 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1aac0664d0 msgr2=0x7f1aac068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:30.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.784+0000 7f1ad110f640 1 --2- 172.21.15.89:0/932545452 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1aac0664d0 0x7f1aac068990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1acc0c17b0 tx=0x7f1abc009190 comp rx=0 tx=0).stop 2023-10-30T12:42:30.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.784+0000 7f1ad110f640 1 -- 172.21.15.89:0/932545452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ab8003980 msgr2=0x7f1acc0c1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:30.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.784+0000 7f1ad110f640 1 --2- 172.21.15.89:0/932545452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ab8003980 0x7f1acc0c1000 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4009ae0 tx=0x7f1ab8005800 comp rx=0 tx=0).stop 2023-10-30T12:42:30.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.784+0000 7f1ad110f640 1 -- 172.21.15.89:0/932545452 shutdown_connections 2023-10-30T12:42:30.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.784+0000 7f1ad110f640 1 --2- 172.21.15.89:0/932545452 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1aac0664d0 0x7f1aac068990 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:30.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.784+0000 7f1ad110f640 1 --2- 172.21.15.89:0/932545452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1acc154c70 0x7f1acc0c3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:30.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.784+0000 7f1ad110f640 1 --2- 172.21.15.89:0/932545452 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1acc157d40 0x7f1acc155550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:30.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.784+0000 7f1ad110f640 1 --2- 172.21.15.89:0/932545452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ab8003980 0x7f1acc0c1000 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:30.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.784+0000 7f1ad110f640 1 -- 172.21.15.89:0/932545452 >> 172.21.15.89:0/932545452 conn(0x7f1acc0c7910 msgr2=0x7f1acc0c4530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:30.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.784+0000 7f1ad110f640 1 -- 172.21.15.89:0/932545452 shutdown_connections 2023-10-30T12:42:30.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:30.784+0000 7f1ad110f640 1 -- 172.21.15.89:0/932545452 wait complete. 2023-10-30T12:42:30.802 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:42:32.803 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:32.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.952+0000 7faf00268640 1 Processor -- start 2023-10-30T12:42:32.950 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.952+0000 7faf00268640 1 -- start start 2023-10-30T12:42:32.951 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.952+0000 7faf00268640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7faef81538a0 0x7faef815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:32.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.952+0000 7faf00268640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7faef80ca0e0 0x7faef815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:32.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.952+0000 7faf00268640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef80cb800 con 0x7faef80cf160 2023-10-30T12:42:32.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.952+0000 7faf00268640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faef815dc30 con 0x7faef80ca0e0 2023-10-30T12:42:32.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.952+0000 7faf00268640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faef80cb330 con 0x7faef81538a0 2023-10-30T12:42:32.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.952+0000 7faefe805640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7faef80cf160 0x7faef80cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:35162/0 (socket says 172.21.15.89:35162) 2023-10-30T12:42:32.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.952+0000 7faefe805640 1 -- 172.21.15.89:0/1161738382 learned_addr learned my addr 172.21.15.89:0/1161738382 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:42:32.953 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.952+0000 7faefd803640 1 --2- 172.21.15.89:0/1161738382 >> v2:172.21.15.89:3300/0 conn(0x7faef80ca0e0 0x7faef815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:32.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.952+0000 7faefe004640 1 --2- 172.21.15.89:0/1161738382 >> v2:172.21.15.192:3300/0 conn(0x7faef81538a0 0x7faef815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:32.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.952+0000 7faefd002640 1 -- 172.21.15.89:0/1161738382 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1485703022 0 0) 0x7faef80cb800 con 0x7faef80cf160 2023-10-30T12:42:32.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.952+0000 7faefd002640 1 -- 172.21.15.89:0/1161738382 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faee4003530 con 0x7faef80cf160 2023-10-30T12:42:32.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.952+0000 7faefd803640 1 -- 172.21.15.89:0/1161738382 >> v2:172.21.15.192:3300/0 conn(0x7faef81538a0 msgr2=0x7faef815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:32.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.952+0000 7faefd803640 1 --2- 172.21.15.89:0/1161738382 >> v2:172.21.15.192:3300/0 conn(0x7faef81538a0 0x7faef815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:32.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.952+0000 7faefd803640 1 -- 172.21.15.89:0/1161738382 >> v1:172.21.15.89:6789/0 conn(0x7faef80cf160 legacy=0x7faef80cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:32.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.952+0000 7faefd803640 1 -- 172.21.15.89:0/1161738382 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef80cad70 con 0x7faef80ca0e0 2023-10-30T12:42:32.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.956+0000 7faefd803640 1 --2- 172.21.15.89:0/1161738382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faef80ca0e0 0x7faef815a090 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7faef80cfd10 tx=0x7faee801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42c2e470073ca374 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:32.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.956+0000 7faefd002640 1 -- 172.21.15.89:0/1161738382 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee80a8950 con 0x7faef80ca0e0 2023-10-30T12:42:32.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.956+0000 7faefd002640 1 -- 172.21.15.89:0/1161738382 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faee80a8ab0 con 0x7faef80ca0e0 2023-10-30T12:42:32.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.956+0000 7faefd002640 1 -- 172.21.15.89:0/1161738382 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee80a8e40 con 0x7faef80ca0e0 2023-10-30T12:42:32.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.956+0000 7faf00268640 1 -- 172.21.15.89:0/1161738382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faef80ca0e0 msgr2=0x7faef815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:32.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.956+0000 7faf00268640 1 --2- 172.21.15.89:0/1161738382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faef80ca0e0 0x7faef815a090 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7faef80cfd10 tx=0x7faee801ec50 comp rx=0 tx=0).stop 2023-10-30T12:42:32.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.956+0000 7faf00268640 1 -- 172.21.15.89:0/1161738382 shutdown_connections 2023-10-30T12:42:32.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.956+0000 7faf00268640 1 --2- 172.21.15.89:0/1161738382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faef80ca0e0 0x7faef815a090 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:32.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.956+0000 7faf00268640 1 --2- 172.21.15.89:0/1161738382 >> v2:172.21.15.192:3300/0 conn(0x7faef81538a0 0x7faef815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:32.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.956+0000 7faf00268640 1 -- 172.21.15.89:0/1161738382 >> 172.21.15.89:0/1161738382 conn(0x7faef806bfe0 msgr2=0x7faef806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:32.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.956+0000 7faf00268640 1 -- 172.21.15.89:0/1161738382 shutdown_connections 2023-10-30T12:42:32.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.956+0000 7faf00268640 1 -- 172.21.15.89:0/1161738382 wait complete. 2023-10-30T12:42:32.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.960+0000 7faf00268640 1 Processor -- start 2023-10-30T12:42:32.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.960+0000 7faf00268640 1 -- start start 2023-10-30T12:42:32.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.960+0000 7faf00268640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faef80ca0e0 0x7faef80ce000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:32.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.960+0000 7faf00268640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faef80cf160 0x7faef80ce540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:32.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.960+0000 7faf00268640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faef81538a0 0x7faef8157860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:32.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.960+0000 7faf00268640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef81e4af0 con 0x7faef81538a0 2023-10-30T12:42:32.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.960+0000 7faf00268640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef815da80 con 0x7faef80ca0e0 2023-10-30T12:42:32.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.960+0000 7faf00268640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faef81e5910 con 0x7faef80cf160 2023-10-30T12:42:32.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.960+0000 7faefe805640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faef81538a0 0x7faef8157860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:32.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.960+0000 7faefe805640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faef81538a0 0x7faef8157860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38184/0 (socket says 172.21.15.89:38184) 2023-10-30T12:42:32.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.960+0000 7faefe805640 1 -- 172.21.15.89:0/1844628500 learned_addr learned my addr 172.21.15.89:0/1844628500 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:32.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.960+0000 7faefe004640 1 --2- 172.21.15.89:0/1844628500 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faef80ca0e0 0x7faef80ce000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:32.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.960+0000 7faefd803640 1 --2- 172.21.15.89:0/1844628500 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faef80cf160 0x7faef80ce540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:32.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.960+0000 7faefe805640 1 -- 172.21.15.89:0/1844628500 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faef80cf160 msgr2=0x7faef80ce540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:32.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.960+0000 7faefe805640 1 --2- 172.21.15.89:0/1844628500 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faef80cf160 0x7faef80ce540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:32.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.960+0000 7faefe805640 1 -- 172.21.15.89:0/1844628500 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faef80ca0e0 msgr2=0x7faef80ce000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:32.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.960+0000 7faefe805640 1 --2- 172.21.15.89:0/1844628500 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faef80ca0e0 0x7faef80ce000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:32.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.960+0000 7faefe805640 1 -- 172.21.15.89:0/1844628500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee8009420 con 0x7faef81538a0 2023-10-30T12:42:32.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.960+0000 7faefe805640 1 --2- 172.21.15.89:0/1844628500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faef81538a0 0x7faef8157860 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7faef000caa0 tx=0x7faef000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:32.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.960+0000 7faeeeffd640 1 -- 172.21.15.89:0/1844628500 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef000ac00 con 0x7faef81538a0 2023-10-30T12:42:32.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.960+0000 7faeeeffd640 1 -- 172.21.15.89:0/1844628500 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faef000ad60 con 0x7faef81538a0 2023-10-30T12:42:32.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.960+0000 7faeeeffd640 1 -- 172.21.15.89:0/1844628500 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef0012620 con 0x7faef81538a0 2023-10-30T12:42:32.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.960+0000 7faf00268640 1 -- 172.21.15.89:0/1844628500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faef81e4550 con 0x7faef81538a0 2023-10-30T12:42:32.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.960+0000 7faf00268640 1 -- 172.21.15.89:0/1844628500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faef81e2c80 con 0x7faef81538a0 2023-10-30T12:42:32.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.960+0000 7faf00268640 1 -- 172.21.15.89:0/1844628500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faef80b2a50 con 0x7faef81538a0 2023-10-30T12:42:32.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.964+0000 7faeeeffd640 1 -- 172.21.15.89:0/1844628500 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faef0007070 con 0x7faef81538a0 2023-10-30T12:42:32.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.964+0000 7faeeeffd640 1 --2- 172.21.15.89:0/1844628500 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faedc0664d0 0x7faedc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:32.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.964+0000 7faeeeffd640 1 -- 172.21.15.89:0/1844628500 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6320+0+0 (secure 0 0 0) 0x7faef0083ff0 con 0x7faef81538a0 2023-10-30T12:42:32.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.964+0000 7faefe004640 1 --2- 172.21.15.89:0/1844628500 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faedc0664d0 0x7faedc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:32.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.964+0000 7faefe004640 1 --2- 172.21.15.89:0/1844628500 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faedc0664d0 0x7faedc068990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7faef4009ac0 tx=0x7faef4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:32.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:32.968+0000 7faeeeffd640 1 -- 172.21.15.89:0/1844628500 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7faef0051240 con 0x7faef81538a0 2023-10-30T12:42:33.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:33.148+0000 7faf00268640 1 -- 172.21.15.89:0/1844628500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faef81e2ae0 con 0x7faef81538a0 2023-10-30T12:42:33.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:33.148+0000 7faeeeffd640 1 -- 172.21.15.89:0/1844628500 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+5737 (secure 0 0 0) 0x7faef81e2ae0 con 0x7faef81538a0 2023-10-30T12:42:33.149 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:42:33.149 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:42:26.158905+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:33.149 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 63 2023-10-30T12:42:33.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:33.152+0000 7faf00268640 1 -- 172.21.15.89:0/1844628500 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faedc0664d0 msgr2=0x7faedc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:33.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:33.152+0000 7faf00268640 1 --2- 172.21.15.89:0/1844628500 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faedc0664d0 0x7faedc068990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7faef4009ac0 tx=0x7faef4009190 comp rx=0 tx=0).stop 2023-10-30T12:42:33.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:33.152+0000 7faf00268640 1 -- 172.21.15.89:0/1844628500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faef81538a0 msgr2=0x7faef8157860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:33.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:33.152+0000 7faf00268640 1 --2- 172.21.15.89:0/1844628500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faef81538a0 0x7faef8157860 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7faef000caa0 tx=0x7faef000cf60 comp rx=0 tx=0).stop 2023-10-30T12:42:33.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:33.152+0000 7faf00268640 1 -- 172.21.15.89:0/1844628500 shutdown_connections 2023-10-30T12:42:33.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:33.152+0000 7faf00268640 1 --2- 172.21.15.89:0/1844628500 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faedc0664d0 0x7faedc068990 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:33.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:33.152+0000 7faf00268640 1 --2- 172.21.15.89:0/1844628500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faef81538a0 0x7faef8157860 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:33.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:33.152+0000 7faf00268640 1 --2- 172.21.15.89:0/1844628500 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faef80cf160 0x7faef80ce540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:33.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:33.152+0000 7faf00268640 1 --2- 172.21.15.89:0/1844628500 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faef80ca0e0 0x7faef80ce000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:33.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:33.152+0000 7faf00268640 1 -- 172.21.15.89:0/1844628500 >> 172.21.15.89:0/1844628500 conn(0x7faef806bfe0 msgr2=0x7faef8148c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:33.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:33.152+0000 7faf00268640 1 -- 172.21.15.89:0/1844628500 shutdown_connections 2023-10-30T12:42:33.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:33.152+0000 7faf00268640 1 -- 172.21.15.89:0/1844628500 wait complete. 2023-10-30T12:42:33.169 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:42:35.170 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:35.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.320+0000 7f5a74a6b640 1 Processor -- start 2023-10-30T12:42:35.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.320+0000 7f5a74a6b640 1 -- start start 2023-10-30T12:42:35.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.320+0000 7f5a74a6b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a701538a0 0x7f5a7015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:35.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.320+0000 7f5a74a6b640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5a700cf160 0x7f5a700cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:35.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.320+0000 7f5a74a6b640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a700cb800 con 0x7f5a700ca0e0 2023-10-30T12:42:35.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.320+0000 7f5a74a6b640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a7015dc30 con 0x7f5a700cf160 2023-10-30T12:42:35.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.320+0000 7f5a74a6b640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a700cb330 con 0x7f5a701538a0 2023-10-30T12:42:35.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.320+0000 7f5a6dd9b640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f5a700ca0e0 0x7f5a7015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:35176/0 (socket says 172.21.15.89:35176) 2023-10-30T12:42:35.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.320+0000 7f5a6dd9b640 1 -- 172.21.15.89:0/3198245982 learned_addr learned my addr 172.21.15.89:0/3198245982 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:42:35.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.320+0000 7f5a6e59c640 1 --2- 172.21.15.89:0/3198245982 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a701538a0 0x7f5a7015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:35.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.320+0000 7f5a6ed9d640 1 --2- 172.21.15.89:0/3198245982 >> v2:172.21.15.192:3300/0 conn(0x7f5a700cf160 0x7f5a700cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:35.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.320+0000 7f5a6d59a640 1 -- 172.21.15.89:0/3198245982 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 902992138 0 0) 0x7f5a700cb800 con 0x7f5a700ca0e0 2023-10-30T12:42:35.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.320+0000 7f5a6d59a640 1 -- 172.21.15.89:0/3198245982 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a58003530 con 0x7f5a700ca0e0 2023-10-30T12:42:35.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.320+0000 7f5a6d59a640 1 -- 172.21.15.89:0/3198245982 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4224225369 0 0) 0x7f5a58003530 con 0x7f5a700ca0e0 2023-10-30T12:42:35.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.320+0000 7f5a6d59a640 1 -- 172.21.15.89:0/3198245982 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a701538a0 msgr2=0x7f5a7015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:35.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.320+0000 7f5a6d59a640 1 --2- 172.21.15.89:0/3198245982 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a701538a0 0x7f5a7015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:35.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.320+0000 7f5a6d59a640 1 -- 172.21.15.89:0/3198245982 >> v2:172.21.15.192:3300/0 conn(0x7f5a700cf160 msgr2=0x7f5a700cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:35.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.320+0000 7f5a6d59a640 1 --2- 172.21.15.89:0/3198245982 >> v2:172.21.15.192:3300/0 conn(0x7f5a700cf160 0x7f5a700cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:35.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.320+0000 7f5a6d59a640 1 -- 172.21.15.89:0/3198245982 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a700cad70 con 0x7f5a700ca0e0 2023-10-30T12:42:35.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.320+0000 7f5a6d59a640 1 -- 172.21.15.89:0/3198245982 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f5a64001680 con 0x7f5a700ca0e0 2023-10-30T12:42:35.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.320+0000 7f5a6d59a640 1 -- 172.21.15.89:0/3198245982 >> v1:172.21.15.89:6789/0 conn(0x7f5a700ca0e0 legacy=0x7f5a7015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:35.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.320+0000 7f5a6d59a640 1 --2- 172.21.15.89:0/3198245982 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a701538a0 0x7f5a58003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:35.326 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.320+0000 7f5a6d59a640 1 --2- 172.21.15.89:0/3198245982 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a58003f00 0x7f5a580047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:35.326 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.320+0000 7f5a6d59a640 1 --2- 172.21.15.89:0/3198245982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a58004d10 0x7f5a58005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:35.326 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.320+0000 7f5a6d59a640 1 -- 172.21.15.89:0/3198245982 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7015dc30 con 0x7f5a58004d10 2023-10-30T12:42:35.326 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.320+0000 7f5a6d59a640 1 -- 172.21.15.89:0/3198245982 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a700cb330 con 0x7f5a58003f00 2023-10-30T12:42:35.326 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.320+0000 7f5a6d59a640 1 -- 172.21.15.89:0/3198245982 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5800b9c0 con 0x7f5a701538a0 2023-10-30T12:42:35.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.324+0000 7f5a6ed9d640 1 --2- 172.21.15.89:0/3198245982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a58004d10 0x7f5a58005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:35.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.324+0000 7f5a6e59c640 1 --2- 172.21.15.89:0/3198245982 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a701538a0 0x7f5a58003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:35.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.324+0000 7f5a6dd9b640 1 --2- 172.21.15.89:0/3198245982 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a58003f00 0x7f5a580047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:35.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.324+0000 7f5a6ed9d640 1 -- 172.21.15.89:0/3198245982 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a701538a0 msgr2=0x7f5a58003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:35.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.324+0000 7f5a6ed9d640 1 --2- 172.21.15.89:0/3198245982 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a701538a0 0x7f5a58003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:35.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.324+0000 7f5a6ed9d640 1 -- 172.21.15.89:0/3198245982 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a58003f00 msgr2=0x7f5a580047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:35.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.324+0000 7f5a6ed9d640 1 --2- 172.21.15.89:0/3198245982 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a58003f00 0x7f5a580047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:35.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.324+0000 7f5a6ed9d640 1 -- 172.21.15.89:0/3198245982 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a58003530 con 0x7f5a58004d10 2023-10-30T12:42:35.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.324+0000 7f5a6ed9d640 1 --2- 172.21.15.89:0/3198245982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a58004d10 0x7f5a58005190 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f5a68009fc0 tx=0x7f5a6801cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1074e50b92ff6a5b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:35.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.324+0000 7f5a6d59a640 1 -- 172.21.15.89:0/3198245982 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a680a5cb0 con 0x7f5a58004d10 2023-10-30T12:42:35.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.324+0000 7f5a6d59a640 1 -- 172.21.15.89:0/3198245982 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a6800f040 con 0x7f5a58004d10 2023-10-30T12:42:35.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.324+0000 7f5a6d59a640 1 -- 172.21.15.89:0/3198245982 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a680ad680 con 0x7f5a58004d10 2023-10-30T12:42:35.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.328+0000 7f5a74a6b640 1 -- 172.21.15.89:0/3198245982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a58004d10 msgr2=0x7f5a58005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:35.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.328+0000 7f5a74a6b640 1 --2- 172.21.15.89:0/3198245982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a58004d10 0x7f5a58005190 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f5a68009fc0 tx=0x7f5a6801cca0 comp rx=0 tx=0).stop 2023-10-30T12:42:35.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.328+0000 7f5a6ed9d640 1 -- 172.21.15.89:0/3198245982 reap_dead start 2023-10-30T12:42:35.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.328+0000 7f5a74a6b640 1 -- 172.21.15.89:0/3198245982 shutdown_connections 2023-10-30T12:42:35.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.328+0000 7f5a74a6b640 1 -- 172.21.15.89:0/3198245982 >> 172.21.15.89:0/3198245982 conn(0x7f5a7006bfe0 msgr2=0x7f5a7006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:35.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.328+0000 7f5a74a6b640 1 -- 172.21.15.89:0/3198245982 shutdown_connections 2023-10-30T12:42:35.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.328+0000 7f5a74a6b640 1 -- 172.21.15.89:0/3198245982 wait complete. 2023-10-30T12:42:35.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.328+0000 7f5a74a6b640 1 Processor -- start 2023-10-30T12:42:35.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.328+0000 7f5a74a6b640 1 -- start start 2023-10-30T12:42:35.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.328+0000 7f5a74a6b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a58004d10 0x7f5a700c6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:35.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.328+0000 7f5a74a6b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a700ce770 0x7f5a7015be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:35.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.328+0000 7f5a74a6b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a700cebb0 0x7f5a7015c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:35.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.328+0000 7f5a74a6b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7015ba70 con 0x7f5a58004d10 2023-10-30T12:42:35.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.328+0000 7f5a74a6b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7015a8b0 con 0x7f5a700ce770 2023-10-30T12:42:35.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.328+0000 7f5a74a6b640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a700c7420 con 0x7f5a700cebb0 2023-10-30T12:42:35.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.328+0000 7f5a6e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a58004d10 0x7f5a700c6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:35.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.328+0000 7f5a6e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a58004d10 0x7f5a700c6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38192/0 (socket says 172.21.15.89:38192) 2023-10-30T12:42:35.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.328+0000 7f5a6e59c640 1 -- 172.21.15.89:0/1429492153 learned_addr learned my addr 172.21.15.89:0/1429492153 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:35.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.328+0000 7f5a6dd9b640 1 --2- 172.21.15.89:0/1429492153 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a700ce770 0x7f5a7015be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:35.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.328+0000 7f5a6ed9d640 1 --2- 172.21.15.89:0/1429492153 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a700cebb0 0x7f5a7015c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:35.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.332+0000 7f5a6e59c640 1 -- 172.21.15.89:0/1429492153 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a700cebb0 msgr2=0x7f5a7015c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:35.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.332+0000 7f5a6e59c640 1 --2- 172.21.15.89:0/1429492153 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a700cebb0 0x7f5a7015c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:35.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.332+0000 7f5a6e59c640 1 -- 172.21.15.89:0/1429492153 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a700ce770 msgr2=0x7f5a7015be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:35.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.332+0000 7f5a6e59c640 1 --2- 172.21.15.89:0/1429492153 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a700ce770 0x7f5a7015be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:35.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.332+0000 7f5a6e59c640 1 -- 172.21.15.89:0/1429492153 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a7015b440 con 0x7f5a58004d10 2023-10-30T12:42:35.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.332+0000 7f5a6e59c640 1 --2- 172.21.15.89:0/1429492153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a58004d10 0x7f5a700c6ee0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f5a600115e0 tx=0x7f5a60011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:35.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.332+0000 7f5a5effd640 1 -- 172.21.15.89:0/1429492153 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6000f8f0 con 0x7f5a58004d10 2023-10-30T12:42:35.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.332+0000 7f5a5effd640 1 -- 172.21.15.89:0/1429492153 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a6000fa50 con 0x7f5a58004d10 2023-10-30T12:42:35.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.332+0000 7f5a5effd640 1 -- 172.21.15.89:0/1429492153 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6000fdc0 con 0x7f5a58004d10 2023-10-30T12:42:35.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.332+0000 7f5a74a6b640 1 -- 172.21.15.89:0/1429492153 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a7015aae0 con 0x7f5a58004d10 2023-10-30T12:42:35.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.332+0000 7f5a74a6b640 1 -- 172.21.15.89:0/1429492153 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a7015b0a0 con 0x7f5a58004d10 2023-10-30T12:42:35.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.332+0000 7f5a74a6b640 1 -- 172.21.15.89:0/1429492153 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a7015bce0 con 0x7f5a58004d10 2023-10-30T12:42:35.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.332+0000 7f5a5effd640 1 -- 172.21.15.89:0/1429492153 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5a60015460 con 0x7f5a58004d10 2023-10-30T12:42:35.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.332+0000 7f5a5effd640 1 --2- 172.21.15.89:0/1429492153 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5a500664a0 0x7f5a50068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:35.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.336+0000 7f5a5effd640 1 -- 172.21.15.89:0/1429492153 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6320+0+0 (secure 0 0 0) 0x7f5a60084580 con 0x7f5a58004d10 2023-10-30T12:42:35.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.336+0000 7f5a6dd9b640 1 --2- 172.21.15.89:0/1429492153 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5a500664a0 0x7f5a50068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:35.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.336+0000 7f5a6dd9b640 1 --2- 172.21.15.89:0/1429492153 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5a500664a0 0x7f5a50068960 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5a700c42e0 tx=0x7f5a64007040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:35.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.340+0000 7f5a5effd640 1 -- 172.21.15.89:0/1429492153 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5a60051740 con 0x7f5a58004d10 2023-10-30T12:42:35.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.516+0000 7f5a74a6b640 1 -- 172.21.15.89:0/1429492153 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a7004e170 con 0x7f5a58004d10 2023-10-30T12:42:35.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.520+0000 7f5a5effd640 1 -- 172.21.15.89:0/1429492153 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f5a60054d60 con 0x7f5a58004d10 2023-10-30T12:42:35.518 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:42:35.519 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:42:33.197674+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:resolve","state_seq":31,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:35.519 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 64 2023-10-30T12:42:35.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.520+0000 7f5a74a6b640 1 -- 172.21.15.89:0/1429492153 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5a500664a0 msgr2=0x7f5a50068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:35.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.520+0000 7f5a74a6b640 1 --2- 172.21.15.89:0/1429492153 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5a500664a0 0x7f5a50068960 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5a700c42e0 tx=0x7f5a64007040 comp rx=0 tx=0).stop 2023-10-30T12:42:35.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.520+0000 7f5a74a6b640 1 -- 172.21.15.89:0/1429492153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a58004d10 msgr2=0x7f5a700c6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:35.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.520+0000 7f5a74a6b640 1 --2- 172.21.15.89:0/1429492153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a58004d10 0x7f5a700c6ee0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f5a600115e0 tx=0x7f5a60011aa0 comp rx=0 tx=0).stop 2023-10-30T12:42:35.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.520+0000 7f5a74a6b640 1 -- 172.21.15.89:0/1429492153 shutdown_connections 2023-10-30T12:42:35.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.520+0000 7f5a74a6b640 1 --2- 172.21.15.89:0/1429492153 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5a500664a0 0x7f5a50068960 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:35.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.520+0000 7f5a74a6b640 1 --2- 172.21.15.89:0/1429492153 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5a700ce770 0x7f5a7015be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:35.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.520+0000 7f5a74a6b640 1 --2- 172.21.15.89:0/1429492153 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5a700cebb0 0x7f5a7015c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:35.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.520+0000 7f5a74a6b640 1 --2- 172.21.15.89:0/1429492153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a58004d10 0x7f5a700c6ee0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:35.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.520+0000 7f5a74a6b640 1 -- 172.21.15.89:0/1429492153 >> 172.21.15.89:0/1429492153 conn(0x7f5a7006bfe0 msgr2=0x7f5a700663b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:35.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.520+0000 7f5a74a6b640 1 -- 172.21.15.89:0/1429492153 shutdown_connections 2023-10-30T12:42:35.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:35.520+0000 7f5a74a6b640 1 -- 172.21.15.89:0/1429492153 wait complete. 2023-10-30T12:42:35.539 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:42:37.540 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:37.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.687+0000 7fb65fcea640 1 Processor -- start 2023-10-30T12:42:37.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.687+0000 7fb65fcea640 1 -- start start 2023-10-30T12:42:37.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.687+0000 7fb65fcea640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb658151920 0x7fb658065da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:37.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.687+0000 7fb65fcea640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6580c5f60 0x7fb6580cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:37.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.687+0000 7fb65fcea640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb658151d00 con 0x7fb6580c6b00 2023-10-30T12:42:37.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.687+0000 7fb65fcea640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6580c4830 con 0x7fb658151920 2023-10-30T12:42:37.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.687+0000 7fb65fcea640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6580c6790 con 0x7fb6580c5f60 2023-10-30T12:42:37.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.687+0000 7fb65da86640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb658151920 0x7fb658065da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:37.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.687+0000 7fb65da86640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb658151920 0x7fb658065da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38208/0 (socket says 172.21.15.89:38208) 2023-10-30T12:42:37.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.687+0000 7fb65da86640 1 -- 172.21.15.89:0/2693564355 learned_addr learned my addr 172.21.15.89:0/2693564355 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:37.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.687+0000 7fb65e287640 1 --2- 172.21.15.89:0/2693564355 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6580c5f60 0x7fb6580cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:37.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.687+0000 7fb65da86640 1 -- 172.21.15.89:0/2693564355 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6580c5f60 msgr2=0x7fb6580cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:37.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.687+0000 7fb65da86640 1 --2- 172.21.15.89:0/2693564355 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6580c5f60 0x7fb6580cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:37.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.687+0000 7fb65da86640 1 -- 172.21.15.89:0/2693564355 >> v1:172.21.15.89:6789/0 conn(0x7fb6580c6b00 legacy=0x7fb658157e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:37.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.687+0000 7fb65da86640 1 -- 172.21.15.89:0/2693564355 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6580c71e0 con 0x7fb658151920 2023-10-30T12:42:37.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.687+0000 7fb65ca84640 1 -- 172.21.15.89:0/2693564355 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3961139575 0 0) 0x7fb658151d00 con 0x7fb6580c6b00 2023-10-30T12:42:37.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.691+0000 7fb65da86640 1 --2- 172.21.15.89:0/2693564355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb658151920 0x7fb658065da0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fb654009870 tx=0x7fb65401c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=516a87210bf24ca6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:37.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.691+0000 7fb65ca84640 1 -- 172.21.15.89:0/2693564355 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6540ab600 con 0x7fb658151920 2023-10-30T12:42:37.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.691+0000 7fb65ca84640 1 -- 172.21.15.89:0/2693564355 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6540a7050 con 0x7fb658151920 2023-10-30T12:42:37.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.691+0000 7fb65ca84640 1 -- 172.21.15.89:0/2693564355 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6540ab980 con 0x7fb658151920 2023-10-30T12:42:37.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.691+0000 7fb65fcea640 1 -- 172.21.15.89:0/2693564355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb658151920 msgr2=0x7fb658065da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:37.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.691+0000 7fb65fcea640 1 --2- 172.21.15.89:0/2693564355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb658151920 0x7fb658065da0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fb654009870 tx=0x7fb65401c540 comp rx=0 tx=0).stop 2023-10-30T12:42:37.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.691+0000 7fb65fcea640 1 -- 172.21.15.89:0/2693564355 shutdown_connections 2023-10-30T12:42:37.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.691+0000 7fb65fcea640 1 --2- 172.21.15.89:0/2693564355 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6580c5f60 0x7fb6580cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:37.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.691+0000 7fb65fcea640 1 --2- 172.21.15.89:0/2693564355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb658151920 0x7fb658065da0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:37.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.691+0000 7fb65fcea640 1 -- 172.21.15.89:0/2693564355 >> 172.21.15.89:0/2693564355 conn(0x7fb65806c6e0 msgr2=0x7fb658069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:37.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.691+0000 7fb65fcea640 1 -- 172.21.15.89:0/2693564355 shutdown_connections 2023-10-30T12:42:37.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.691+0000 7fb65fcea640 1 -- 172.21.15.89:0/2693564355 wait complete. 2023-10-30T12:42:37.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.695+0000 7fb65fcea640 1 Processor -- start 2023-10-30T12:42:37.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.695+0000 7fb65fcea640 1 -- start start 2023-10-30T12:42:37.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.695+0000 7fb65fcea640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6580c5f60 0x7fb658150c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:37.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.695+0000 7fb65fcea640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6580c6b00 0x7fb658146f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:37.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.695+0000 7fb65fcea640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb658151920 0x7fb658147470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:37.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.695+0000 7fb65fcea640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb658150960 con 0x7fb658151920 2023-10-30T12:42:37.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.695+0000 7fb65fcea640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65815b4c0 con 0x7fb6580c6b00 2023-10-30T12:42:37.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.695+0000 7fb65fcea640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb65815ba40 con 0x7fb6580c5f60 2023-10-30T12:42:37.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.695+0000 7fb65e287640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb658151920 0x7fb658147470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:37.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.695+0000 7fb65e287640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb658151920 0x7fb658147470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38220/0 (socket says 172.21.15.89:38220) 2023-10-30T12:42:37.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.695+0000 7fb65e287640 1 -- 172.21.15.89:0/1673319108 learned_addr learned my addr 172.21.15.89:0/1673319108 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:37.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.695+0000 7fb65da86640 1 --2- 172.21.15.89:0/1673319108 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6580c5f60 0x7fb658150c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:37.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.695+0000 7fb65d285640 1 --2- 172.21.15.89:0/1673319108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6580c6b00 0x7fb658146f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:37.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.695+0000 7fb65e287640 1 -- 172.21.15.89:0/1673319108 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6580c5f60 msgr2=0x7fb658150c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:37.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.695+0000 7fb65e287640 1 --2- 172.21.15.89:0/1673319108 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6580c5f60 0x7fb658150c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:37.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.695+0000 7fb65e287640 1 -- 172.21.15.89:0/1673319108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6580c6b00 msgr2=0x7fb658146f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:37.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.695+0000 7fb65e287640 1 --2- 172.21.15.89:0/1673319108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6580c6b00 0x7fb658146f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:37.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.695+0000 7fb65e287640 1 -- 172.21.15.89:0/1673319108 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6540094d0 con 0x7fb658151920 2023-10-30T12:42:37.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.695+0000 7fb65e287640 1 --2- 172.21.15.89:0/1673319108 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb658151920 0x7fb658147470 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fb650007e60 tx=0x7fb65000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:37.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.695+0000 7fb64e7fc640 1 -- 172.21.15.89:0/1673319108 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb650014420 con 0x7fb658151920 2023-10-30T12:42:37.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.695+0000 7fb64e7fc640 1 -- 172.21.15.89:0/1673319108 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6500145a0 con 0x7fb658151920 2023-10-30T12:42:37.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.695+0000 7fb65fcea640 1 -- 172.21.15.89:0/1673319108 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6581f2860 con 0x7fb658151920 2023-10-30T12:42:37.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.695+0000 7fb64e7fc640 1 -- 172.21.15.89:0/1673319108 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb650014950 con 0x7fb658151920 2023-10-30T12:42:37.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.695+0000 7fb65fcea640 1 -- 172.21.15.89:0/1673319108 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb65814f030 con 0x7fb658151920 2023-10-30T12:42:37.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.699+0000 7fb65fcea640 1 -- 172.21.15.89:0/1673319108 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb65814f220 con 0x7fb658151920 2023-10-30T12:42:37.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.699+0000 7fb64e7fc640 1 -- 172.21.15.89:0/1673319108 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb650014ab0 con 0x7fb658151920 2023-10-30T12:42:37.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.699+0000 7fb64e7fc640 1 --2- 172.21.15.89:0/1673319108 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb63c0664a0 0x7fb63c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:37.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.699+0000 7fb64e7fc640 1 -- 172.21.15.89:0/1673319108 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6320+0+0 (secure 0 0 0) 0x7fb650084a90 con 0x7fb658151920 2023-10-30T12:42:37.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.699+0000 7fb65da86640 1 --2- 172.21.15.89:0/1673319108 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb63c0664a0 0x7fb63c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:37.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.703+0000 7fb65da86640 1 --2- 172.21.15.89:0/1673319108 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb63c0664a0 0x7fb63c068960 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb6580c5c00 tx=0x7fb6540a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:37.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.703+0000 7fb64e7fc640 1 -- 172.21.15.89:0/1673319108 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb650051c30 con 0x7fb658151920 2023-10-30T12:42:37.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.883+0000 7fb65fcea640 1 -- 172.21.15.89:0/1673319108 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb65804e170 con 0x7fb658151920 2023-10-30T12:42:37.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.887+0000 7fb64e7fc640 1 -- 172.21.15.89:0/1673319108 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+5739 (secure 0 0 0) 0x7fb650055250 con 0x7fb658151920 2023-10-30T12:42:37.886 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:42:37.886 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:42:33.197674+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:resolve","state_seq":31,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:37.886 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 64 2023-10-30T12:42:37.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.887+0000 7fb65fcea640 1 -- 172.21.15.89:0/1673319108 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb63c0664a0 msgr2=0x7fb63c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:37.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.887+0000 7fb65fcea640 1 --2- 172.21.15.89:0/1673319108 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb63c0664a0 0x7fb63c068960 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb6580c5c00 tx=0x7fb6540a6000 comp rx=0 tx=0).stop 2023-10-30T12:42:37.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.887+0000 7fb65fcea640 1 -- 172.21.15.89:0/1673319108 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb658151920 msgr2=0x7fb658147470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:37.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.887+0000 7fb65fcea640 1 --2- 172.21.15.89:0/1673319108 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb658151920 0x7fb658147470 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fb650007e60 tx=0x7fb65000e4b0 comp rx=0 tx=0).stop 2023-10-30T12:42:37.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.887+0000 7fb65fcea640 1 -- 172.21.15.89:0/1673319108 shutdown_connections 2023-10-30T12:42:37.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.887+0000 7fb65fcea640 1 --2- 172.21.15.89:0/1673319108 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb63c0664a0 0x7fb63c068960 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:37.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.887+0000 7fb65fcea640 1 --2- 172.21.15.89:0/1673319108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6580c6b00 0x7fb658146f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:37.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.887+0000 7fb65fcea640 1 --2- 172.21.15.89:0/1673319108 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb658151920 0x7fb658147470 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:37.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.887+0000 7fb65fcea640 1 --2- 172.21.15.89:0/1673319108 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6580c5f60 0x7fb658150c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:37.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.887+0000 7fb65fcea640 1 -- 172.21.15.89:0/1673319108 >> 172.21.15.89:0/1673319108 conn(0x7fb65806c6e0 msgr2=0x7fb65806d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:37.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.887+0000 7fb65fcea640 1 -- 172.21.15.89:0/1673319108 shutdown_connections 2023-10-30T12:42:37.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:37.887+0000 7fb65fcea640 1 -- 172.21.15.89:0/1673319108 wait complete. 2023-10-30T12:42:37.906 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:42:39.908 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:40.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.055+0000 7f88e4c9c640 1 Processor -- start 2023-10-30T12:42:40.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.055+0000 7f88e4c9c640 1 -- start start 2023-10-30T12:42:40.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.055+0000 7f88e4c9c640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88e00cf160 0x7f88e00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:40.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.055+0000 7f88e4c9c640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88e00cb800 con 0x7f88e01538a0 2023-10-30T12:42:40.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.055+0000 7f88e4c9c640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88e00cb330 con 0x7f88e015d6f0 2023-10-30T12:42:40.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.055+0000 7f88e4c9c640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88e00cad70 con 0x7f88e00cf160 2023-10-30T12:42:40.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.055+0000 7f88de59c640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f88e01538a0 0x7f88e00ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:60440/0 (socket says 172.21.15.89:60440) 2023-10-30T12:42:40.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.055+0000 7f88de59c640 1 -- 172.21.15.89:0/3306005881 learned_addr learned my addr 172.21.15.89:0/3306005881 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:42:40.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.055+0000 7f88ded9d640 1 --2- 172.21.15.89:0/3306005881 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88e00cf160 0x7f88e00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:40.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.055+0000 7f88ddd9b640 1 -- 172.21.15.89:0/3306005881 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3588811811 0 0) 0x7f88e00cb800 con 0x7f88e01538a0 2023-10-30T12:42:40.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.055+0000 7f88ddd9b640 1 -- 172.21.15.89:0/3306005881 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88cc003530 con 0x7f88e01538a0 2023-10-30T12:42:40.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.055+0000 7f88ddd9b640 1 -- 172.21.15.89:0/3306005881 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2055787717 0 0) 0x7f88cc003530 con 0x7f88e01538a0 2023-10-30T12:42:40.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.055+0000 7f88ddd9b640 1 -- 172.21.15.89:0/3306005881 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88e00cf160 msgr2=0x7f88e00cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:40.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.055+0000 7f88ddd9b640 1 --2- 172.21.15.89:0/3306005881 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88e00cf160 0x7f88e00cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:40.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.055+0000 7f88ddd9b640 1 -- 172.21.15.89:0/3306005881 >> v1:172.21.15.192:6789/0 conn(0x7f88e015d6f0 legacy=0x7f88e015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:40.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.055+0000 7f88ddd9b640 1 -- 172.21.15.89:0/3306005881 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e00cb050 con 0x7f88e01538a0 2023-10-30T12:42:40.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.055+0000 7f88ddd9b640 1 -- 172.21.15.89:0/3306005881 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f88d0001a00 con 0x7f88e01538a0 2023-10-30T12:42:40.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.055+0000 7f88ddd9b640 1 -- 172.21.15.89:0/3306005881 >> v1:172.21.15.89:6789/0 conn(0x7f88e01538a0 legacy=0x7f88e00ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:40.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.055+0000 7f88ddd9b640 1 --2- 172.21.15.89:0/3306005881 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88cc0039b0 0x7f88cc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:40.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.055+0000 7f88ddd9b640 1 --2- 172.21.15.89:0/3306005881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88cc004360 0x7f88cc004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:40.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.055+0000 7f88ddd9b640 1 --2- 172.21.15.89:0/3306005881 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88e00cf160 0x7f88cc005090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:40.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.055+0000 7f88ddd9b640 1 -- 172.21.15.89:0/3306005881 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e00cad70 con 0x7f88cc004360 2023-10-30T12:42:40.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.055+0000 7f88ddd9b640 1 -- 172.21.15.89:0/3306005881 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88cc00b990 con 0x7f88cc0039b0 2023-10-30T12:42:40.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.055+0000 7f88ddd9b640 1 -- 172.21.15.89:0/3306005881 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88cc00bb50 con 0x7f88e00cf160 2023-10-30T12:42:40.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.055+0000 7f88d5d9b640 1 --2- 172.21.15.89:0/3306005881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88cc004360 0x7f88cc004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:40.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.059+0000 7f88de59c640 1 --2- 172.21.15.89:0/3306005881 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88cc0039b0 0x7f88cc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:40.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.059+0000 7f88ded9d640 1 --2- 172.21.15.89:0/3306005881 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88e00cf160 0x7f88cc005090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:40.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.059+0000 7f88d5d9b640 1 -- 172.21.15.89:0/3306005881 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88e00cf160 msgr2=0x7f88cc005090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:40.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.059+0000 7f88d5d9b640 1 --2- 172.21.15.89:0/3306005881 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88e00cf160 0x7f88cc005090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:40.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.059+0000 7f88d5d9b640 1 -- 172.21.15.89:0/3306005881 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88cc0039b0 msgr2=0x7f88cc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:40.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.059+0000 7f88d5d9b640 1 --2- 172.21.15.89:0/3306005881 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88cc0039b0 0x7f88cc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:40.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.059+0000 7f88d5d9b640 1 -- 172.21.15.89:0/3306005881 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88cc003530 con 0x7f88cc004360 2023-10-30T12:42:40.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.059+0000 7f88de59c640 1 --2- 172.21.15.89:0/3306005881 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88cc0039b0 0x7f88cc003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:42:40.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.059+0000 7f88d5d9b640 1 --2- 172.21.15.89:0/3306005881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88cc004360 0x7f88cc004830 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f88c8001860 tx=0x7f88c801c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f34b5775e19f2e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:40.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.059+0000 7f88ddd9b640 1 -- 172.21.15.89:0/3306005881 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c80ab620 con 0x7f88cc004360 2023-10-30T12:42:40.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.059+0000 7f88ddd9b640 1 -- 172.21.15.89:0/3306005881 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c80ab780 con 0x7f88cc004360 2023-10-30T12:42:40.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.059+0000 7f88ddd9b640 1 -- 172.21.15.89:0/3306005881 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c80abb30 con 0x7f88cc004360 2023-10-30T12:42:40.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.059+0000 7f88e4c9c640 1 -- 172.21.15.89:0/3306005881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88cc004360 msgr2=0x7f88cc004830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:40.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.059+0000 7f88e4c9c640 1 --2- 172.21.15.89:0/3306005881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88cc004360 0x7f88cc004830 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f88c8001860 tx=0x7f88c801c4e0 comp rx=0 tx=0).stop 2023-10-30T12:42:40.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.059+0000 7f88ded9d640 1 -- 172.21.15.89:0/3306005881 reap_dead start 2023-10-30T12:42:40.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.059+0000 7f88e4c9c640 1 -- 172.21.15.89:0/3306005881 shutdown_connections 2023-10-30T12:42:40.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.059+0000 7f88e4c9c640 1 -- 172.21.15.89:0/3306005881 >> 172.21.15.89:0/3306005881 conn(0x7f88e006bfe0 msgr2=0x7f88e006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:40.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.063+0000 7f88e4c9c640 1 -- 172.21.15.89:0/3306005881 shutdown_connections 2023-10-30T12:42:40.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.063+0000 7f88e4c9c640 1 -- 172.21.15.89:0/3306005881 wait complete. 2023-10-30T12:42:40.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.063+0000 7f88e4c9c640 1 Processor -- start 2023-10-30T12:42:40.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.063+0000 7f88e4c9c640 1 -- start start 2023-10-30T12:42:40.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.063+0000 7f88e4c9c640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88cc004360 0x7f88e01506c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:40.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.063+0000 7f88e4c9c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88e01582a0 0x7f88e00ce300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:40.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.063+0000 7f88e4c9c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88e0158720 0x7f88e00b4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:40.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.063+0000 7f88e4c9c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e00b3e10 con 0x7f88e0158720 2023-10-30T12:42:40.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.063+0000 7f88e4c9c640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e00c6d50 con 0x7f88e01582a0 2023-10-30T12:42:40.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.063+0000 7f88e4c9c640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88e00c6a90 con 0x7f88cc004360 2023-10-30T12:42:40.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.063+0000 7f88ded9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88e0158720 0x7f88e00b4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:40.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.063+0000 7f88ded9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88e0158720 0x7f88e00b4230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35142/0 (socket says 172.21.15.89:35142) 2023-10-30T12:42:40.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.063+0000 7f88ded9d640 1 -- 172.21.15.89:0/1341884616 learned_addr learned my addr 172.21.15.89:0/1341884616 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:40.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.063+0000 7f88de59c640 1 --2- 172.21.15.89:0/1341884616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88cc004360 0x7f88e01506c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:40.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.063+0000 7f88d5d9b640 1 --2- 172.21.15.89:0/1341884616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88e01582a0 0x7f88e00ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:40.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.063+0000 7f88ded9d640 1 -- 172.21.15.89:0/1341884616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88cc004360 msgr2=0x7f88e01506c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:40.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.063+0000 7f88ded9d640 1 --2- 172.21.15.89:0/1341884616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88cc004360 0x7f88e01506c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:40.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.063+0000 7f88ded9d640 1 -- 172.21.15.89:0/1341884616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88e01582a0 msgr2=0x7f88e00ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:40.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.063+0000 7f88ded9d640 1 --2- 172.21.15.89:0/1341884616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88e01582a0 0x7f88e00ce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:40.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.063+0000 7f88ded9d640 1 -- 172.21.15.89:0/1341884616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e00b3190 con 0x7f88e0158720 2023-10-30T12:42:40.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.063+0000 7f88d5d9b640 1 --2- 172.21.15.89:0/1341884616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88e01582a0 0x7f88e00ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:42:40.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.063+0000 7f88ded9d640 1 --2- 172.21.15.89:0/1341884616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88e0158720 0x7f88e00b4230 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f88e0072ce0 tx=0x7f88d8011c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:40.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.067+0000 7f88d77fe640 1 -- 172.21.15.89:0/1341884616 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d8004030 con 0x7f88e0158720 2023-10-30T12:42:40.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.067+0000 7f88d77fe640 1 -- 172.21.15.89:0/1341884616 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88d80041b0 con 0x7f88e0158720 2023-10-30T12:42:40.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.067+0000 7f88d77fe640 1 -- 172.21.15.89:0/1341884616 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d800f720 con 0x7f88e0158720 2023-10-30T12:42:40.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.067+0000 7f88e4c9c640 1 -- 172.21.15.89:0/1341884616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88e00b3470 con 0x7f88e0158720 2023-10-30T12:42:40.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.067+0000 7f88e4c9c640 1 -- 172.21.15.89:0/1341884616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88e00b3a30 con 0x7f88e0158720 2023-10-30T12:42:40.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.067+0000 7f88e4c9c640 1 -- 172.21.15.89:0/1341884616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88e004e170 con 0x7f88e0158720 2023-10-30T12:42:40.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.067+0000 7f88d77fe640 1 -- 172.21.15.89:0/1341884616 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f88d800fa90 con 0x7f88e0158720 2023-10-30T12:42:40.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.067+0000 7f88d77fe640 1 --2- 172.21.15.89:0/1341884616 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f88c00664a0 0x7f88c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:40.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.067+0000 7f88d77fe640 1 -- 172.21.15.89:0/1341884616 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6320+0+0 (secure 0 0 0) 0x7f88d8084730 con 0x7f88e0158720 2023-10-30T12:42:40.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.067+0000 7f88de59c640 1 --2- 172.21.15.89:0/1341884616 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f88c00664a0 0x7f88c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:40.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.071+0000 7f88de59c640 1 --2- 172.21.15.89:0/1341884616 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f88c00664a0 0x7f88c0068960 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f88d0000f90 tx=0x7f88d00017e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:40.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.071+0000 7f88d77fe640 1 -- 172.21.15.89:0/1341884616 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f88d80518e0 con 0x7f88e0158720 2023-10-30T12:42:40.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.271+0000 7f88e4c9c640 1 -- 172.21.15.89:0/1341884616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88e00b1dd0 con 0x7f88e0158720 2023-10-30T12:42:40.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.271+0000 7f88d77fe640 1 -- 172.21.15.89:0/1341884616 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f88d8054f00 con 0x7f88e0158720 2023-10-30T12:42:40.269 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:42:40.269 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:42:38.393178+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:reconnect","state_seq":33,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:40.269 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 65 2023-10-30T12:42:40.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.275+0000 7f88e4c9c640 1 -- 172.21.15.89:0/1341884616 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f88c00664a0 msgr2=0x7f88c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:40.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.275+0000 7f88e4c9c640 1 --2- 172.21.15.89:0/1341884616 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f88c00664a0 0x7f88c0068960 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f88d0000f90 tx=0x7f88d00017e0 comp rx=0 tx=0).stop 2023-10-30T12:42:40.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.275+0000 7f88e4c9c640 1 -- 172.21.15.89:0/1341884616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88e0158720 msgr2=0x7f88e00b4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:40.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.275+0000 7f88e4c9c640 1 --2- 172.21.15.89:0/1341884616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88e0158720 0x7f88e00b4230 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f88e0072ce0 tx=0x7f88d8011c30 comp rx=0 tx=0).stop 2023-10-30T12:42:40.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.275+0000 7f88e4c9c640 1 -- 172.21.15.89:0/1341884616 shutdown_connections 2023-10-30T12:42:40.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.275+0000 7f88e4c9c640 1 --2- 172.21.15.89:0/1341884616 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f88c00664a0 0x7f88c0068960 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:40.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.275+0000 7f88e4c9c640 1 --2- 172.21.15.89:0/1341884616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88e01582a0 0x7f88e00ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:40.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.275+0000 7f88e4c9c640 1 --2- 172.21.15.89:0/1341884616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f88e0158720 0x7f88e00b4230 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:40.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.275+0000 7f88e4c9c640 1 --2- 172.21.15.89:0/1341884616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88cc004360 0x7f88e01506c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:40.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.275+0000 7f88e4c9c640 1 -- 172.21.15.89:0/1341884616 >> 172.21.15.89:0/1341884616 conn(0x7f88e006bfe0 msgr2=0x7f88e01517c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:40.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.275+0000 7f88e4c9c640 1 -- 172.21.15.89:0/1341884616 shutdown_connections 2023-10-30T12:42:40.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:40.275+0000 7f88e4c9c640 1 -- 172.21.15.89:0/1341884616 wait complete. 2023-10-30T12:42:40.291 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 65, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:50.729245+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 65, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5129": { "addr": "172.21.15.192:6837/1413360446", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6836", "nonce": 1413360446, "type": "v2" }, { "addr": "172.21.15.192:6837", "nonce": 1413360446, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5129, "incarnation": 57, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:active", "state_seq": 40 }, "gid_5139": { "addr": "172.21.15.89:6837/951276252", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6836", "nonce": 951276252, "type": "v2" }, { "addr": "172.21.15.89:6837", "nonce": 951276252, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5139, "incarnation": 33, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 29 }, "gid_5575": { "addr": "172.21.15.192:6835/501291700", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6834", "nonce": 501291700, "type": "v2" }, { "addr": "172.21.15.192:6835", "nonce": 501291700, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5575, "incarnation": 63, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 33 } }, "last_failure": 0, "last_failure_osd_epoch": 61, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:42:38.393178+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5575, "mds_1": 5139, "mds_2": 5129 } } } ], "standbys": [ { "addr": "172.21.15.89:6834/2909790173", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6832", "nonce": 2909790173, "type": "v2" }, { "addr": "172.21.15.89:6834", "nonce": 2909790173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.89:6835/2387357615", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6833", "nonce": 2387357615, "type": "v2" }, { "addr": "172.21.15.89:6835", "nonce": 2387357615, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4239, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:42:42.293 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:42.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.447+0000 7f3e5c93e640 1 Processor -- start 2023-10-30T12:42:42.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.447+0000 7f3e5c93e640 1 -- start start 2023-10-30T12:42:42.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.451+0000 7f3e5c93e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3e581538a0 0x7f3e5815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:42.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.451+0000 7f3e5c93e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3e580ca0e0 0x7f3e5815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:42.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.451+0000 7f3e5c93e640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e580cb800 con 0x7f3e580cf160 2023-10-30T12:42:42.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.451+0000 7f3e5c93e640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e5815dc30 con 0x7f3e581538a0 2023-10-30T12:42:42.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.451+0000 7f3e5c93e640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e580cb330 con 0x7f3e580ca0e0 2023-10-30T12:42:42.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.451+0000 7f3e5659c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3e581538a0 0x7f3e5815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:42.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.451+0000 7f3e5659c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3e581538a0 0x7f3e5815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35150/0 (socket says 172.21.15.89:35150) 2023-10-30T12:42:42.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.451+0000 7f3e5659c640 1 -- 172.21.15.89:0/28556681 learned_addr learned my addr 172.21.15.89:0/28556681 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:42.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.451+0000 7f3e56d9d640 1 --1- 172.21.15.89:0/28556681 >> v1:172.21.15.89:6789/0 conn(0x7f3e580cf160 0x7f3e580cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:60456/0 (socket says 172.21.15.89:60456) 2023-10-30T12:42:42.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.451+0000 7f3e4ffff640 1 --2- 172.21.15.89:0/28556681 >> v2:172.21.15.192:3300/0 conn(0x7f3e580ca0e0 0x7f3e5815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:42.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.451+0000 7f3e5659c640 1 -- 172.21.15.89:0/28556681 >> v2:172.21.15.192:3300/0 conn(0x7f3e580ca0e0 msgr2=0x7f3e5815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:42.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.451+0000 7f3e5659c640 1 --2- 172.21.15.89:0/28556681 >> v2:172.21.15.192:3300/0 conn(0x7f3e580ca0e0 0x7f3e5815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:42.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.451+0000 7f3e5659c640 1 -- 172.21.15.89:0/28556681 >> v1:172.21.15.89:6789/0 conn(0x7f3e580cf160 legacy=0x7f3e580cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:42.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.451+0000 7f3e5659c640 1 -- 172.21.15.89:0/28556681 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e580cad70 con 0x7f3e581538a0 2023-10-30T12:42:42.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.451+0000 7f3e55d9b640 1 -- 172.21.15.89:0/28556681 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3694290857 0 0) 0x7f3e580cb800 con 0x7f3e580cf160 2023-10-30T12:42:42.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.451+0000 7f3e4ffff640 1 --2- 172.21.15.89:0/28556681 >> v2:172.21.15.192:3300/0 conn(0x7f3e580ca0e0 0x7f3e5815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:42:42.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.455+0000 7f3e5659c640 1 --2- 172.21.15.89:0/28556681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e581538a0 0x7f3e5815d6f0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f3e580b30a0 tx=0x7f3e4801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8477d9af9f3fdbb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:42.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.455+0000 7f3e55d9b640 1 -- 172.21.15.89:0/28556681 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e480a89f0 con 0x7f3e581538a0 2023-10-30T12:42:42.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.455+0000 7f3e55d9b640 1 -- 172.21.15.89:0/28556681 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e480a8b50 con 0x7f3e581538a0 2023-10-30T12:42:42.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.455+0000 7f3e55d9b640 1 -- 172.21.15.89:0/28556681 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e480b0440 con 0x7f3e581538a0 2023-10-30T12:42:42.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.455+0000 7f3e5c93e640 1 -- 172.21.15.89:0/28556681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e581538a0 msgr2=0x7f3e5815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:42.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.455+0000 7f3e5c93e640 1 --2- 172.21.15.89:0/28556681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e581538a0 0x7f3e5815d6f0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f3e580b30a0 tx=0x7f3e4801ecf0 comp rx=0 tx=0).stop 2023-10-30T12:42:42.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.455+0000 7f3e5c93e640 1 -- 172.21.15.89:0/28556681 shutdown_connections 2023-10-30T12:42:42.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.455+0000 7f3e5c93e640 1 --2- 172.21.15.89:0/28556681 >> v2:172.21.15.192:3300/0 conn(0x7f3e580ca0e0 0x7f3e5815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:42.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.455+0000 7f3e5c93e640 1 --2- 172.21.15.89:0/28556681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e581538a0 0x7f3e5815d6f0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:42.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.455+0000 7f3e5c93e640 1 -- 172.21.15.89:0/28556681 >> 172.21.15.89:0/28556681 conn(0x7f3e5806bfe0 msgr2=0x7f3e5806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:42.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.455+0000 7f3e5c93e640 1 -- 172.21.15.89:0/28556681 shutdown_connections 2023-10-30T12:42:42.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.455+0000 7f3e5c93e640 1 -- 172.21.15.89:0/28556681 wait complete. 2023-10-30T12:42:42.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.455+0000 7f3e5c93e640 1 Processor -- start 2023-10-30T12:42:42.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.455+0000 7f3e5c93e640 1 -- start start 2023-10-30T12:42:42.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.459+0000 7f3e5c93e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e580ca0e0 0x7f3e581f3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:42.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.459+0000 7f3e5c93e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e580cf160 0x7f3e5815d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:42.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.459+0000 7f3e5c93e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e581538a0 0x7f3e58158de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:42.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.459+0000 7f3e5c93e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e581f37b0 con 0x7f3e581538a0 2023-10-30T12:42:42.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.459+0000 7f3e5c93e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e581eff70 con 0x7f3e580cf160 2023-10-30T12:42:42.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.459+0000 7f3e5c93e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5815d970 con 0x7f3e580ca0e0 2023-10-30T12:42:42.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.459+0000 7f3e56d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e581538a0 0x7f3e58158de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:42.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.459+0000 7f3e56d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e581538a0 0x7f3e58158de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35164/0 (socket says 172.21.15.89:35164) 2023-10-30T12:42:42.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.459+0000 7f3e56d9d640 1 -- 172.21.15.89:0/4253083007 learned_addr learned my addr 172.21.15.89:0/4253083007 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:42.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.459+0000 7f3e5659c640 1 --2- 172.21.15.89:0/4253083007 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e580ca0e0 0x7f3e581f3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:42.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.459+0000 7f3e4ffff640 1 --2- 172.21.15.89:0/4253083007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e580cf160 0x7f3e5815d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:42.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.459+0000 7f3e56d9d640 1 -- 172.21.15.89:0/4253083007 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e580ca0e0 msgr2=0x7f3e581f3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:42.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.459+0000 7f3e56d9d640 1 --2- 172.21.15.89:0/4253083007 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e580ca0e0 0x7f3e581f3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:42.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.459+0000 7f3e56d9d640 1 -- 172.21.15.89:0/4253083007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e580cf160 msgr2=0x7f3e5815d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:42.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.459+0000 7f3e56d9d640 1 --2- 172.21.15.89:0/4253083007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e580cf160 0x7f3e5815d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:42.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.459+0000 7f3e56d9d640 1 -- 172.21.15.89:0/4253083007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e480094d0 con 0x7f3e581538a0 2023-10-30T12:42:42.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.459+0000 7f3e4ffff640 1 --2- 172.21.15.89:0/4253083007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e580cf160 0x7f3e5815d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:42:42.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.459+0000 7f3e56d9d640 1 --2- 172.21.15.89:0/4253083007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e581538a0 0x7f3e58158de0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f3e50001570 tx=0x7f3e5000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:42.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.459+0000 7f3e4effd640 1 -- 172.21.15.89:0/4253083007 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e5000ce40 con 0x7f3e581538a0 2023-10-30T12:42:42.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.459+0000 7f3e4effd640 1 -- 172.21.15.89:0/4253083007 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e5000a5e0 con 0x7f3e581538a0 2023-10-30T12:42:42.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.459+0000 7f3e4effd640 1 -- 172.21.15.89:0/4253083007 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e5000a990 con 0x7f3e581538a0 2023-10-30T12:42:42.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.459+0000 7f3e5c93e640 1 -- 172.21.15.89:0/4253083007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e580ced20 con 0x7f3e581538a0 2023-10-30T12:42:42.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.459+0000 7f3e5c93e640 1 -- 172.21.15.89:0/4253083007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e581595b0 con 0x7f3e581538a0 2023-10-30T12:42:42.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.459+0000 7f3e5c93e640 1 -- 172.21.15.89:0/4253083007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e580b2a50 con 0x7f3e581538a0 2023-10-30T12:42:42.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.459+0000 7f3e4effd640 1 -- 172.21.15.89:0/4253083007 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3e5000aaf0 con 0x7f3e581538a0 2023-10-30T12:42:42.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.463+0000 7f3e4effd640 1 --2- 172.21.15.89:0/4253083007 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3e380664d0 0x7f3e38068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:42.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.463+0000 7f3e4effd640 1 -- 172.21.15.89:0/4253083007 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6320+0+0 (secure 0 0 0) 0x7f3e500834c0 con 0x7f3e581538a0 2023-10-30T12:42:42.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.463+0000 7f3e5659c640 1 --2- 172.21.15.89:0/4253083007 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3e380664d0 0x7f3e38068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:42.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.463+0000 7f3e5659c640 1 --2- 172.21.15.89:0/4253083007 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3e380664d0 0x7f3e38068990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3e480b2000 tx=0x7f3e48009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:42.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.467+0000 7f3e4effd640 1 -- 172.21.15.89:0/4253083007 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3e50050680 con 0x7f3e581538a0 2023-10-30T12:42:42.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.643+0000 7f3e5c93e640 1 -- 172.21.15.89:0/4253083007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e581f0120 con 0x7f3e581538a0 2023-10-30T12:42:42.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.647+0000 7f3e4effd640 1 -- 172.21.15.89:0/4253083007 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3e581f0120 con 0x7f3e581538a0 2023-10-30T12:42:42.643 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:42:42.643 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:42:38.393178+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:reconnect","state_seq":33,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:42.644 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 65 2023-10-30T12:42:42.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.647+0000 7f3e5c93e640 1 -- 172.21.15.89:0/4253083007 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3e380664d0 msgr2=0x7f3e38068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:42.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.647+0000 7f3e5c93e640 1 --2- 172.21.15.89:0/4253083007 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3e380664d0 0x7f3e38068990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3e480b2000 tx=0x7f3e48009040 comp rx=0 tx=0).stop 2023-10-30T12:42:42.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.647+0000 7f3e5c93e640 1 -- 172.21.15.89:0/4253083007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e581538a0 msgr2=0x7f3e58158de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:42.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.647+0000 7f3e5c93e640 1 --2- 172.21.15.89:0/4253083007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e581538a0 0x7f3e58158de0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f3e50001570 tx=0x7f3e5000cc50 comp rx=0 tx=0).stop 2023-10-30T12:42:42.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.647+0000 7f3e5c93e640 1 -- 172.21.15.89:0/4253083007 shutdown_connections 2023-10-30T12:42:42.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.647+0000 7f3e5c93e640 1 --2- 172.21.15.89:0/4253083007 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3e380664d0 0x7f3e38068990 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:42.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.647+0000 7f3e5c93e640 1 --2- 172.21.15.89:0/4253083007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e580cf160 0x7f3e5815d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:42.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.647+0000 7f3e5c93e640 1 --2- 172.21.15.89:0/4253083007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e581538a0 0x7f3e58158de0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:42.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.647+0000 7f3e5c93e640 1 --2- 172.21.15.89:0/4253083007 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e580ca0e0 0x7f3e581f3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:42.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.647+0000 7f3e5c93e640 1 -- 172.21.15.89:0/4253083007 >> 172.21.15.89:0/4253083007 conn(0x7f3e5806bfe0 msgr2=0x7f3e581519a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:42.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.647+0000 7f3e5c93e640 1 -- 172.21.15.89:0/4253083007 shutdown_connections 2023-10-30T12:42:42.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:42.647+0000 7f3e5c93e640 1 -- 172.21.15.89:0/4253083007 wait complete. 2023-10-30T12:42:42.663 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:42:44.664 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:44.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.815+0000 7f209f846640 1 Processor -- start 2023-10-30T12:42:44.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.815+0000 7f209f846640 1 -- start start 2023-10-30T12:42:44.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.819+0000 7f209f846640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20980c05d0 0x7f209813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:44.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.819+0000 7f209f846640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f20980c6f60 0x7f209813f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:44.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.819+0000 7f209f846640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20980c0a50 con 0x7f20980c6b80 2023-10-30T12:42:44.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.819+0000 7f209f846640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f209806f6e0 con 0x7f20980c6f60 2023-10-30T12:42:44.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.819+0000 7f209f846640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f209806f9c0 con 0x7f20980c05d0 2023-10-30T12:42:44.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.819+0000 7f209dde3640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f20980c6f60 0x7f209813f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:44.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.819+0000 7f209d5e2640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f20980c6b80 0x7f20980bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:60458/0 (socket says 172.21.15.89:60458) 2023-10-30T12:42:44.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.819+0000 7f209d5e2640 1 -- 172.21.15.89:0/3360048035 learned_addr learned my addr 172.21.15.89:0/3360048035 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:42:44.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.819+0000 7f209cde1640 1 --2- 172.21.15.89:0/3360048035 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20980c05d0 0x7f209813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:44.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.819+0000 7f209dde3640 1 -- 172.21.15.89:0/3360048035 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20980c05d0 msgr2=0x7f209813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:44.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.819+0000 7f209dde3640 1 --2- 172.21.15.89:0/3360048035 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20980c05d0 0x7f209813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:44.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.819+0000 7f209dde3640 1 -- 172.21.15.89:0/3360048035 >> v1:172.21.15.89:6789/0 conn(0x7f20980c6b80 legacy=0x7f20980bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:44.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.819+0000 7f209dde3640 1 -- 172.21.15.89:0/3360048035 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f209806fca0 con 0x7f20980c6f60 2023-10-30T12:42:44.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.819+0000 7f208ffff640 1 -- 172.21.15.89:0/3360048035 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3885480726 0 0) 0x7f20980c0a50 con 0x7f20980c6b80 2023-10-30T12:42:44.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.819+0000 7f209cde1640 1 --2- 172.21.15.89:0/3360048035 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20980c05d0 0x7f209813ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:42:44.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.823+0000 7f209dde3640 1 --2- 172.21.15.89:0/3360048035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20980c6f60 0x7f209813f2d0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f2090009d90 tx=0x7f209001c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0b67c1616427686 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:44.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.823+0000 7f208ffff640 1 -- 172.21.15.89:0/3360048035 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20900a5920 con 0x7f20980c6f60 2023-10-30T12:42:44.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.823+0000 7f208ffff640 1 -- 172.21.15.89:0/3360048035 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20900a5a80 con 0x7f20980c6f60 2023-10-30T12:42:44.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.823+0000 7f208ffff640 1 -- 172.21.15.89:0/3360048035 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20900a5e10 con 0x7f20980c6f60 2023-10-30T12:42:44.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.823+0000 7f209f846640 1 -- 172.21.15.89:0/3360048035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20980c6f60 msgr2=0x7f209813f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:44.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.823+0000 7f209f846640 1 --2- 172.21.15.89:0/3360048035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20980c6f60 0x7f209813f2d0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f2090009d90 tx=0x7f209001c950 comp rx=0 tx=0).stop 2023-10-30T12:42:44.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.823+0000 7f209f846640 1 -- 172.21.15.89:0/3360048035 shutdown_connections 2023-10-30T12:42:44.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.823+0000 7f209f846640 1 --2- 172.21.15.89:0/3360048035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20980c6f60 0x7f209813f2d0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:44.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.823+0000 7f209f846640 1 --2- 172.21.15.89:0/3360048035 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20980c05d0 0x7f209813ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:44.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.823+0000 7f209f846640 1 -- 172.21.15.89:0/3360048035 >> 172.21.15.89:0/3360048035 conn(0x7f20980c7910 msgr2=0x7f20980c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:44.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.823+0000 7f209f846640 1 -- 172.21.15.89:0/3360048035 shutdown_connections 2023-10-30T12:42:44.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.823+0000 7f209f846640 1 -- 172.21.15.89:0/3360048035 wait complete. 2023-10-30T12:42:44.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.823+0000 7f209f846640 1 Processor -- start 2023-10-30T12:42:44.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.823+0000 7f209f846640 1 -- start start 2023-10-30T12:42:44.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.827+0000 7f209f846640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20980c05d0 0x7f2098153510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:44.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.827+0000 7f209f846640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20980c6b80 0x7f20981ef9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:44.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.827+0000 7f209f846640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20980c6f60 0x7f209814ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:44.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.827+0000 7f209f846640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2098156440 con 0x7f20980c05d0 2023-10-30T12:42:44.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.827+0000 7f209f846640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20981f3e10 con 0x7f20980c6b80 2023-10-30T12:42:44.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.827+0000 7f209f846640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2098156160 con 0x7f20980c6f60 2023-10-30T12:42:44.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.827+0000 7f209d5e2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20980c05d0 0x7f2098153510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:44.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.827+0000 7f209d5e2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20980c05d0 0x7f2098153510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35176/0 (socket says 172.21.15.89:35176) 2023-10-30T12:42:44.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.827+0000 7f209d5e2640 1 -- 172.21.15.89:0/1070496490 learned_addr learned my addr 172.21.15.89:0/1070496490 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:44.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.827+0000 7f209cde1640 1 --2- 172.21.15.89:0/1070496490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20980c6b80 0x7f20981ef9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:44.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.827+0000 7f209dde3640 1 --2- 172.21.15.89:0/1070496490 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20980c6f60 0x7f209814ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:44.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.827+0000 7f209d5e2640 1 -- 172.21.15.89:0/1070496490 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20980c6f60 msgr2=0x7f209814ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:44.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.827+0000 7f209d5e2640 1 --2- 172.21.15.89:0/1070496490 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20980c6f60 0x7f209814ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:44.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.827+0000 7f209d5e2640 1 -- 172.21.15.89:0/1070496490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20980c6b80 msgr2=0x7f20981ef9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:44.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.827+0000 7f209d5e2640 1 --2- 172.21.15.89:0/1070496490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20980c6b80 0x7f20981ef9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:44.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.827+0000 7f209d5e2640 1 -- 172.21.15.89:0/1070496490 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2090009a70 con 0x7f20980c05d0 2023-10-30T12:42:44.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.827+0000 7f209d5e2640 1 --2- 172.21.15.89:0/1070496490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20980c05d0 0x7f2098153510 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f20940011f0 tx=0x7f209400c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:44.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.827+0000 7f208dffb640 1 -- 172.21.15.89:0/1070496490 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20940074e0 con 0x7f20980c05d0 2023-10-30T12:42:44.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.827+0000 7f208dffb640 1 -- 172.21.15.89:0/1070496490 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2094007640 con 0x7f20980c05d0 2023-10-30T12:42:44.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.827+0000 7f209f846640 1 -- 172.21.15.89:0/1070496490 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20981f3050 con 0x7f20980c05d0 2023-10-30T12:42:44.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.827+0000 7f208dffb640 1 -- 172.21.15.89:0/1070496490 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2094004a10 con 0x7f20980c05d0 2023-10-30T12:42:44.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.827+0000 7f209f846640 1 -- 172.21.15.89:0/1070496490 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20981f3610 con 0x7f20980c05d0 2023-10-30T12:42:44.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.827+0000 7f209f846640 1 -- 172.21.15.89:0/1070496490 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f209804e170 con 0x7f20980c05d0 2023-10-30T12:42:44.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.827+0000 7f208dffb640 1 -- 172.21.15.89:0/1070496490 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2094004b70 con 0x7f20980c05d0 2023-10-30T12:42:44.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.831+0000 7f208dffb640 1 --2- 172.21.15.89:0/1070496490 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f207c0664a0 0x7f207c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:44.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.831+0000 7f208dffb640 1 -- 172.21.15.89:0/1070496490 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6320+0+0 (secure 0 0 0) 0x7f20940836b0 con 0x7f20980c05d0 2023-10-30T12:42:44.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.831+0000 7f209cde1640 1 --2- 172.21.15.89:0/1070496490 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f207c0664a0 0x7f207c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:44.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.831+0000 7f209cde1640 1 --2- 172.21.15.89:0/1070496490 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f207c0664a0 0x7f207c068960 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f20981f0ae0 tx=0x7f2088008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:44.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:44.835+0000 7f208dffb640 1 -- 172.21.15.89:0/1070496490 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f20940508e0 con 0x7f20980c05d0 2023-10-30T12:42:45.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:45.015+0000 7f209f846640 1 -- 172.21.15.89:0/1070496490 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2098136f20 con 0x7f20980c05d0 2023-10-30T12:42:45.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:45.015+0000 7f208dffb640 1 -- 172.21.15.89:0/1070496490 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2094053f00 con 0x7f20980c05d0 2023-10-30T12:42:45.013 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:42:45.013 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:42:38.393178+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:reconnect","state_seq":33,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:45.014 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 65 2023-10-30T12:42:45.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:45.019+0000 7f209f846640 1 -- 172.21.15.89:0/1070496490 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f207c0664a0 msgr2=0x7f207c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:45.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:45.019+0000 7f209f846640 1 --2- 172.21.15.89:0/1070496490 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f207c0664a0 0x7f207c068960 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f20981f0ae0 tx=0x7f2088008040 comp rx=0 tx=0).stop 2023-10-30T12:42:45.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:45.019+0000 7f209f846640 1 -- 172.21.15.89:0/1070496490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20980c05d0 msgr2=0x7f2098153510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:45.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:45.019+0000 7f209f846640 1 --2- 172.21.15.89:0/1070496490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20980c05d0 0x7f2098153510 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f20940011f0 tx=0x7f209400c9b0 comp rx=0 tx=0).stop 2023-10-30T12:42:45.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:45.019+0000 7f209f846640 1 -- 172.21.15.89:0/1070496490 shutdown_connections 2023-10-30T12:42:45.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:45.019+0000 7f209f846640 1 --2- 172.21.15.89:0/1070496490 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f207c0664a0 0x7f207c068960 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:45.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:45.019+0000 7f209f846640 1 --2- 172.21.15.89:0/1070496490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20980c6b80 0x7f20981ef9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:45.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:45.019+0000 7f209f846640 1 --2- 172.21.15.89:0/1070496490 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20980c6f60 0x7f209814ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:45.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:45.019+0000 7f209f846640 1 --2- 172.21.15.89:0/1070496490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20980c05d0 0x7f2098153510 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:45.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:45.019+0000 7f209f846640 1 -- 172.21.15.89:0/1070496490 >> 172.21.15.89:0/1070496490 conn(0x7f20980c7910 msgr2=0x7f20980c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:45.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:45.019+0000 7f209f846640 1 -- 172.21.15.89:0/1070496490 shutdown_connections 2023-10-30T12:42:45.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:45.019+0000 7f209f846640 1 -- 172.21.15.89:0/1070496490 wait complete. 2023-10-30T12:42:45.034 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:42:47.037 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:47.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.191+0000 7facc2ac9640 1 Processor -- start 2023-10-30T12:42:47.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.191+0000 7facc2ac9640 1 -- start start 2023-10-30T12:42:47.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.191+0000 7facc2ac9640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7facbc1538a0 0x7facbc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:47.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.191+0000 7facc2ac9640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7facbc0ca0e0 0x7facbc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:47.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.191+0000 7facc2ac9640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facbc0cb800 con 0x7facbc0cf160 2023-10-30T12:42:47.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.191+0000 7facc2ac9640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facbc15dc30 con 0x7facbc1538a0 2023-10-30T12:42:47.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.191+0000 7facc2ac9640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facbc0cb330 con 0x7facbc0ca0e0 2023-10-30T12:42:47.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.191+0000 7facc0865640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7facbc1538a0 0x7facbc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:47.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.191+0000 7facc0865640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7facbc1538a0 0x7facbc15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35192/0 (socket says 172.21.15.89:35192) 2023-10-30T12:42:47.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.191+0000 7facc0865640 1 -- 172.21.15.89:0/9929899 learned_addr learned my addr 172.21.15.89:0/9929899 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:47.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.191+0000 7facbbfff640 1 --2- 172.21.15.89:0/9929899 >> v2:172.21.15.192:3300/0 conn(0x7facbc0ca0e0 0x7facbc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:47.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.191+0000 7facc0865640 1 -- 172.21.15.89:0/9929899 >> v2:172.21.15.192:3300/0 conn(0x7facbc0ca0e0 msgr2=0x7facbc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:47.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.191+0000 7facc0865640 1 --2- 172.21.15.89:0/9929899 >> v2:172.21.15.192:3300/0 conn(0x7facbc0ca0e0 0x7facbc15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:47.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.191+0000 7facc0865640 1 -- 172.21.15.89:0/9929899 >> v1:172.21.15.192:6789/0 conn(0x7facbc0cf160 legacy=0x7facbc0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:47.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.191+0000 7facc0865640 1 -- 172.21.15.89:0/9929899 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facbc0cad70 con 0x7facbc1538a0 2023-10-30T12:42:47.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.195+0000 7facc0865640 1 --2- 172.21.15.89:0/9929899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facbc1538a0 0x7facbc15d6f0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7facbc0b30a0 tx=0x7facb001ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d9263beb55d5b43 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:47.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.195+0000 7facbb7fe640 1 -- 172.21.15.89:0/9929899 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb00a89b0 con 0x7facbc1538a0 2023-10-30T12:42:47.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.195+0000 7facbb7fe640 1 -- 172.21.15.89:0/9929899 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb00a8b10 con 0x7facbc1538a0 2023-10-30T12:42:47.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.195+0000 7facbb7fe640 1 -- 172.21.15.89:0/9929899 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb00a8ea0 con 0x7facbc1538a0 2023-10-30T12:42:47.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.195+0000 7facc2ac9640 1 -- 172.21.15.89:0/9929899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facbc1538a0 msgr2=0x7facbc15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:47.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.195+0000 7facc2ac9640 1 --2- 172.21.15.89:0/9929899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facbc1538a0 0x7facbc15d6f0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7facbc0b30a0 tx=0x7facb001ecb0 comp rx=0 tx=0).stop 2023-10-30T12:42:47.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.195+0000 7facc2ac9640 1 -- 172.21.15.89:0/9929899 shutdown_connections 2023-10-30T12:42:47.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.195+0000 7facc2ac9640 1 --2- 172.21.15.89:0/9929899 >> v2:172.21.15.192:3300/0 conn(0x7facbc0ca0e0 0x7facbc15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:47.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.195+0000 7facc2ac9640 1 --2- 172.21.15.89:0/9929899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facbc1538a0 0x7facbc15d6f0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:47.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.195+0000 7facc2ac9640 1 -- 172.21.15.89:0/9929899 >> 172.21.15.89:0/9929899 conn(0x7facbc06bfe0 msgr2=0x7facbc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:47.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.195+0000 7facc2ac9640 1 -- 172.21.15.89:0/9929899 shutdown_connections 2023-10-30T12:42:47.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.195+0000 7facc2ac9640 1 -- 172.21.15.89:0/9929899 wait complete. 2023-10-30T12:42:47.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.199+0000 7facc2ac9640 1 Processor -- start 2023-10-30T12:42:47.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.199+0000 7facc2ac9640 1 -- start start 2023-10-30T12:42:47.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.199+0000 7facc2ac9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facbc0ca0e0 0x7facbc0b4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:47.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.199+0000 7facc2ac9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facbc0cf160 0x7facbc1466e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:47.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.199+0000 7facc2ac9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7facbc1538a0 0x7facbc1498e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:47.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.199+0000 7facc2ac9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facbc0cda00 con 0x7facbc0ca0e0 2023-10-30T12:42:47.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.199+0000 7facc2ac9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facbc1e2ae0 con 0x7facbc1538a0 2023-10-30T12:42:47.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.199+0000 7facc2ac9640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facbc0b3570 con 0x7facbc0cf160 2023-10-30T12:42:47.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.199+0000 7facc0865640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facbc0ca0e0 0x7facbc0b4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:47.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.199+0000 7facc0865640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facbc0ca0e0 0x7facbc0b4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35198/0 (socket says 172.21.15.89:35198) 2023-10-30T12:42:47.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.199+0000 7facc0865640 1 -- 172.21.15.89:0/516381758 learned_addr learned my addr 172.21.15.89:0/516381758 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:47.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.199+0000 7facbbfff640 1 --2- 172.21.15.89:0/516381758 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facbc0cf160 0x7facbc1466e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:47.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.199+0000 7facc1066640 1 --2- 172.21.15.89:0/516381758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7facbc1538a0 0x7facbc1498e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:47.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.199+0000 7facc0865640 1 -- 172.21.15.89:0/516381758 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facbc0cf160 msgr2=0x7facbc1466e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:47.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.199+0000 7facc0865640 1 --2- 172.21.15.89:0/516381758 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facbc0cf160 0x7facbc1466e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:47.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.199+0000 7facc0865640 1 -- 172.21.15.89:0/516381758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7facbc1538a0 msgr2=0x7facbc1498e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:47.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.199+0000 7facc0865640 1 --2- 172.21.15.89:0/516381758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7facbc1538a0 0x7facbc1498e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:47.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.199+0000 7facc0865640 1 -- 172.21.15.89:0/516381758 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb00094d0 con 0x7facbc0ca0e0 2023-10-30T12:42:47.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.199+0000 7facc0865640 1 --2- 172.21.15.89:0/516381758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facbc0ca0e0 0x7facbc0b4d60 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7facb00b3040 tx=0x7facb001ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:47.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.199+0000 7facb97fa640 1 -- 172.21.15.89:0/516381758 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb00b0e80 con 0x7facbc0ca0e0 2023-10-30T12:42:47.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.199+0000 7facb97fa640 1 -- 172.21.15.89:0/516381758 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb00bb480 con 0x7facbc0ca0e0 2023-10-30T12:42:47.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.199+0000 7facb97fa640 1 -- 172.21.15.89:0/516381758 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb00bb810 con 0x7facbc0ca0e0 2023-10-30T12:42:47.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.199+0000 7facc2ac9640 1 -- 172.21.15.89:0/516381758 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facbc145ff0 con 0x7facbc0ca0e0 2023-10-30T12:42:47.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.199+0000 7facc2ac9640 1 -- 172.21.15.89:0/516381758 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facbc0ccfb0 con 0x7facbc0ca0e0 2023-10-30T12:42:47.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.203+0000 7facc2ac9640 1 -- 172.21.15.89:0/516381758 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facbc0b2a50 con 0x7facbc0ca0e0 2023-10-30T12:42:47.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.203+0000 7facb97fa640 1 -- 172.21.15.89:0/516381758 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7facb00bb970 con 0x7facbc0ca0e0 2023-10-30T12:42:47.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.203+0000 7facb97fa640 1 --2- 172.21.15.89:0/516381758 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faca00664a0 0x7faca0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:47.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.203+0000 7facb97fa640 1 -- 172.21.15.89:0/516381758 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6320+0+0 (secure 0 0 0) 0x7facb0122680 con 0x7facbc0ca0e0 2023-10-30T12:42:47.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.203+0000 7facbbfff640 1 --2- 172.21.15.89:0/516381758 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faca00664a0 0x7faca0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:47.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.203+0000 7facbbfff640 1 --2- 172.21.15.89:0/516381758 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faca00664a0 0x7faca0068960 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7facbc1e5710 tx=0x7faca8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:47.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.207+0000 7facb97fa640 1 -- 172.21.15.89:0/516381758 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7facb00ef8b0 con 0x7facbc0ca0e0 2023-10-30T12:42:47.320 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:42:47.323 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:42:47.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.391+0000 7facc2ac9640 1 -- 172.21.15.89:0/516381758 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7facbc0b2a50 con 0x7facbc0ca0e0 2023-10-30T12:42:47.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.391+0000 7facb97fa640 1 -- 172.21.15.89:0/516381758 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+5738 (secure 0 0 0) 0x7facb00f2ed0 con 0x7facbc0ca0e0 2023-10-30T12:42:47.389 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:42:47.390 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:42:44.375246+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:rejoin","state_seq":35,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:47.390 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 66 2023-10-30T12:42:47.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.395+0000 7facc2ac9640 1 -- 172.21.15.89:0/516381758 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faca00664a0 msgr2=0x7faca0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:47.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.395+0000 7facc2ac9640 1 --2- 172.21.15.89:0/516381758 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faca00664a0 0x7faca0068960 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7facbc1e5710 tx=0x7faca8008040 comp rx=0 tx=0).stop 2023-10-30T12:42:47.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.395+0000 7facc2ac9640 1 -- 172.21.15.89:0/516381758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facbc0ca0e0 msgr2=0x7facbc0b4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:47.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.395+0000 7facc2ac9640 1 --2- 172.21.15.89:0/516381758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facbc0ca0e0 0x7facbc0b4d60 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7facb00b3040 tx=0x7facb001ba50 comp rx=0 tx=0).stop 2023-10-30T12:42:47.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.395+0000 7facc2ac9640 1 -- 172.21.15.89:0/516381758 shutdown_connections 2023-10-30T12:42:47.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.395+0000 7facc2ac9640 1 --2- 172.21.15.89:0/516381758 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faca00664a0 0x7faca0068960 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:47.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.395+0000 7facc2ac9640 1 --2- 172.21.15.89:0/516381758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7facbc1538a0 0x7facbc1498e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:47.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.395+0000 7facc2ac9640 1 --2- 172.21.15.89:0/516381758 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facbc0cf160 0x7facbc1466e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:47.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.395+0000 7facc2ac9640 1 --2- 172.21.15.89:0/516381758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facbc0ca0e0 0x7facbc0b4d60 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:47.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.395+0000 7facc2ac9640 1 -- 172.21.15.89:0/516381758 >> 172.21.15.89:0/516381758 conn(0x7facbc06bfe0 msgr2=0x7facbc0c5460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:47.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.395+0000 7facc2ac9640 1 -- 172.21.15.89:0/516381758 shutdown_connections 2023-10-30T12:42:47.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:47.395+0000 7facc2ac9640 1 -- 172.21.15.89:0/516381758 wait complete. 2023-10-30T12:42:47.410 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:42:49.411 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:49.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.559+0000 7f89451d9640 1 Processor -- start 2023-10-30T12:42:49.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.559+0000 7f89451d9640 1 -- start start 2023-10-30T12:42:49.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.559+0000 7f89451d9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89401538a0 0x7f894015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:49.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.559+0000 7f89451d9640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f89400cf160 0x7f89400cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:49.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.559+0000 7f89451d9640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89400cb800 con 0x7f89400ca0e0 2023-10-30T12:42:49.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.559+0000 7f89451d9640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f894015dc30 con 0x7f89400cf160 2023-10-30T12:42:49.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.559+0000 7f89451d9640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89400cb330 con 0x7f89401538a0 2023-10-30T12:42:49.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.559+0000 7f893f59e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f89400cf160 0x7f89400cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:49.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.559+0000 7f893ed9d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89401538a0 0x7f894015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:49.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.559+0000 7f893659c640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f89400ca0e0 0x7f894015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:55204/0 (socket says 172.21.15.89:55204) 2023-10-30T12:42:49.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.559+0000 7f893ed9d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89401538a0 0x7f894015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:37844/0 (socket says 172.21.15.89:37844) 2023-10-30T12:42:49.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.559+0000 7f893ed9d640 1 -- 172.21.15.89:0/615480684 learned_addr learned my addr 172.21.15.89:0/615480684 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:49.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.559+0000 7f893f59e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f89400cf160 0x7f89400cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:40064/0 (socket says 172.21.15.89:40064) 2023-10-30T12:42:49.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.559+0000 7f893f59e640 1 -- 172.21.15.89:0/615480684 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89401538a0 msgr2=0x7f894015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:49.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.559+0000 7f893f59e640 1 --2- 172.21.15.89:0/615480684 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89401538a0 0x7f894015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:49.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.559+0000 7f893f59e640 1 -- 172.21.15.89:0/615480684 >> v1:172.21.15.192:6789/0 conn(0x7f89400ca0e0 legacy=0x7f894015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:49.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.559+0000 7f893f59e640 1 -- 172.21.15.89:0/615480684 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89400cad70 con 0x7f89400cf160 2023-10-30T12:42:49.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.563+0000 7f893f59e640 1 --2- 172.21.15.89:0/615480684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89400cf160 0x7f89400cf590 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8938009d90 tx=0x7f893801f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10b674abb271cede server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:49.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.563+0000 7f893e59c640 1 -- 172.21.15.89:0/615480684 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89380af600 con 0x7f89400cf160 2023-10-30T12:42:49.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.563+0000 7f893e59c640 1 -- 172.21.15.89:0/615480684 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89380ab050 con 0x7f89400cf160 2023-10-30T12:42:49.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.563+0000 7f893e59c640 1 -- 172.21.15.89:0/615480684 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89380af980 con 0x7f89400cf160 2023-10-30T12:42:49.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.563+0000 7f89451d9640 1 -- 172.21.15.89:0/615480684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89400cf160 msgr2=0x7f89400cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:49.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.563+0000 7f89451d9640 1 --2- 172.21.15.89:0/615480684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89400cf160 0x7f89400cf590 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8938009d90 tx=0x7f893801f130 comp rx=0 tx=0).stop 2023-10-30T12:42:49.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.567+0000 7f89451d9640 1 -- 172.21.15.89:0/615480684 shutdown_connections 2023-10-30T12:42:49.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.567+0000 7f89451d9640 1 --2- 172.21.15.89:0/615480684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89400cf160 0x7f89400cf590 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:49.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.567+0000 7f89451d9640 1 --2- 172.21.15.89:0/615480684 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89401538a0 0x7f894015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:49.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.567+0000 7f89451d9640 1 -- 172.21.15.89:0/615480684 >> 172.21.15.89:0/615480684 conn(0x7f894006bfe0 msgr2=0x7f894006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:49.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.567+0000 7f89451d9640 1 -- 172.21.15.89:0/615480684 shutdown_connections 2023-10-30T12:42:49.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.567+0000 7f89451d9640 1 -- 172.21.15.89:0/615480684 wait complete. 2023-10-30T12:42:49.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.567+0000 7f89451d9640 1 Processor -- start 2023-10-30T12:42:49.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.567+0000 7f89451d9640 1 -- start start 2023-10-30T12:42:49.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.567+0000 7f89451d9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89400ca0e0 0x7f894014a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:49.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.567+0000 7f89451d9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89400cf160 0x7f89401e5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:49.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.567+0000 7f89451d9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89401538a0 0x7f89401ef360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:49.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.567+0000 7f89451d9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894015b620 con 0x7f89400cf160 2023-10-30T12:42:49.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.567+0000 7f89451d9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894015b340 con 0x7f89401538a0 2023-10-30T12:42:49.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.567+0000 7f89451d9640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f894015bfd0 con 0x7f89400ca0e0 2023-10-30T12:42:49.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.567+0000 7f893659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89400cf160 0x7f89401e5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:49.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.567+0000 7f893659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89400cf160 0x7f89401e5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35208/0 (socket says 172.21.15.89:35208) 2023-10-30T12:42:49.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.567+0000 7f893659c640 1 -- 172.21.15.89:0/3499749441 learned_addr learned my addr 172.21.15.89:0/3499749441 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:49.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.567+0000 7f893f59e640 1 --2- 172.21.15.89:0/3499749441 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89401538a0 0x7f89401ef360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:49.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.567+0000 7f893ed9d640 1 --2- 172.21.15.89:0/3499749441 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89400ca0e0 0x7f894014a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:49.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.567+0000 7f893659c640 1 -- 172.21.15.89:0/3499749441 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89400ca0e0 msgr2=0x7f894014a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:49.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.567+0000 7f893659c640 1 --2- 172.21.15.89:0/3499749441 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89400ca0e0 0x7f894014a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:49.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.567+0000 7f893659c640 1 -- 172.21.15.89:0/3499749441 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89401538a0 msgr2=0x7f89401ef360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:49.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.567+0000 7f893659c640 1 --2- 172.21.15.89:0/3499749441 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89401538a0 0x7f89401ef360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:49.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.567+0000 7f893659c640 1 -- 172.21.15.89:0/3499749441 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8938009a70 con 0x7f89400cf160 2023-10-30T12:42:49.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.571+0000 7f893659c640 1 --2- 172.21.15.89:0/3499749441 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89400cf160 0x7f89401e5de0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f893000c420 tx=0x7f893000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:49.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.571+0000 7f8937fff640 1 -- 172.21.15.89:0/3499749441 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89300074e0 con 0x7f89400cf160 2023-10-30T12:42:49.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.571+0000 7f8937fff640 1 -- 172.21.15.89:0/3499749441 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8930007640 con 0x7f89400cf160 2023-10-30T12:42:49.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.571+0000 7f8937fff640 1 -- 172.21.15.89:0/3499749441 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8930004a30 con 0x7f89400cf160 2023-10-30T12:42:49.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.571+0000 7f89451d9640 1 -- 172.21.15.89:0/3499749441 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f894015c2b0 con 0x7f89400cf160 2023-10-30T12:42:49.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.571+0000 7f89451d9640 1 -- 172.21.15.89:0/3499749441 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f894015ca90 con 0x7f89400cf160 2023-10-30T12:42:49.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.571+0000 7f89451d9640 1 -- 172.21.15.89:0/3499749441 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89400b2a50 con 0x7f89400cf160 2023-10-30T12:42:49.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.571+0000 7f8937fff640 1 -- 172.21.15.89:0/3499749441 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8930004b90 con 0x7f89400cf160 2023-10-30T12:42:49.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.571+0000 7f8937fff640 1 --2- 172.21.15.89:0/3499749441 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f89200664a0 0x7f8920068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:49.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.571+0000 7f8937fff640 1 -- 172.21.15.89:0/3499749441 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6320+0+0 (secure 0 0 0) 0x7f8930083a80 con 0x7f89400cf160 2023-10-30T12:42:49.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.571+0000 7f893ed9d640 1 --2- 172.21.15.89:0/3499749441 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f89200664a0 0x7f8920068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:49.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.575+0000 7f893ed9d640 1 --2- 172.21.15.89:0/3499749441 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f89200664a0 0x7f8920068960 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8928009ab0 tx=0x7f8928009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:49.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.575+0000 7f8937fff640 1 -- 172.21.15.89:0/3499749441 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8930050cd0 con 0x7f89400cf160 2023-10-30T12:42:49.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.755+0000 7f89451d9640 1 -- 172.21.15.89:0/3499749441 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f894004e170 con 0x7f89400cf160 2023-10-30T12:42:49.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.755+0000 7f8937fff640 1 -- 172.21.15.89:0/3499749441 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f89300542f0 con 0x7f89400cf160 2023-10-30T12:42:49.752 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:42:49.753 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:42:44.375246+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:rejoin","state_seq":35,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:49.754 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 66 2023-10-30T12:42:49.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.759+0000 7f89451d9640 1 -- 172.21.15.89:0/3499749441 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f89200664a0 msgr2=0x7f8920068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:49.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.759+0000 7f89451d9640 1 --2- 172.21.15.89:0/3499749441 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f89200664a0 0x7f8920068960 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8928009ab0 tx=0x7f8928009240 comp rx=0 tx=0).stop 2023-10-30T12:42:49.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.759+0000 7f89451d9640 1 -- 172.21.15.89:0/3499749441 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89400cf160 msgr2=0x7f89401e5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:49.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.759+0000 7f89451d9640 1 --2- 172.21.15.89:0/3499749441 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89400cf160 0x7f89401e5de0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f893000c420 tx=0x7f893000c8e0 comp rx=0 tx=0).stop 2023-10-30T12:42:49.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.759+0000 7f89451d9640 1 -- 172.21.15.89:0/3499749441 shutdown_connections 2023-10-30T12:42:49.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.759+0000 7f89451d9640 1 --2- 172.21.15.89:0/3499749441 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f89200664a0 0x7f8920068960 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:49.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.759+0000 7f89451d9640 1 --2- 172.21.15.89:0/3499749441 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89401538a0 0x7f89401ef360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:49.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.759+0000 7f89451d9640 1 --2- 172.21.15.89:0/3499749441 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89400cf160 0x7f89401e5de0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:49.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.759+0000 7f89451d9640 1 --2- 172.21.15.89:0/3499749441 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89400ca0e0 0x7f894014a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:49.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.759+0000 7f89451d9640 1 -- 172.21.15.89:0/3499749441 >> 172.21.15.89:0/3499749441 conn(0x7f894006bfe0 msgr2=0x7f89400c4100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:49.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.759+0000 7f89451d9640 1 -- 172.21.15.89:0/3499749441 shutdown_connections 2023-10-30T12:42:49.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:49.759+0000 7f89451d9640 1 -- 172.21.15.89:0/3499749441 wait complete. 2023-10-30T12:42:49.773 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:42:51.774 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:51.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.919+0000 7f358550f640 1 Processor -- start 2023-10-30T12:42:51.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.919+0000 7f358550f640 1 -- start start 2023-10-30T12:42:51.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.919+0000 7f358550f640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f35800c8cf0 0x7f35800c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:51.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.919+0000 7f358550f640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f35800ca340 0x7f358015d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:51.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.919+0000 7f358550f640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f358015ce30 con 0x7f35800c90d0 2023-10-30T12:42:51.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.919+0000 7f358550f640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3580159880 con 0x7f35800c8cf0 2023-10-30T12:42:51.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.919+0000 7f358550f640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3580159ae0 con 0x7f35800ca340 2023-10-30T12:42:51.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.919+0000 7f357effd640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f35800c8cf0 0x7f35800c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:51.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.919+0000 7f35767fc640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f35800c90d0 0x7f35800cef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:34746/0 (socket says 172.21.15.89:34746) 2023-10-30T12:42:51.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.919+0000 7f35767fc640 1 -- 172.21.15.89:0/1647352382 learned_addr learned my addr 172.21.15.89:0/1647352382 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:42:51.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.919+0000 7f357f7fe640 1 --2- 172.21.15.89:0/1647352382 >> v2:172.21.15.192:3300/0 conn(0x7f35800ca340 0x7f358015d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:51.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.919+0000 7f357effd640 1 -- 172.21.15.89:0/1647352382 >> v2:172.21.15.192:3300/0 conn(0x7f35800ca340 msgr2=0x7f358015d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:51.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.919+0000 7f357effd640 1 --2- 172.21.15.89:0/1647352382 >> v2:172.21.15.192:3300/0 conn(0x7f35800ca340 0x7f358015d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:51.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.919+0000 7f357effd640 1 -- 172.21.15.89:0/1647352382 >> v1:172.21.15.89:6789/0 conn(0x7f35800c90d0 legacy=0x7f35800cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:51.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.919+0000 7f357effd640 1 -- 172.21.15.89:0/1647352382 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3580159dc0 con 0x7f35800c8cf0 2023-10-30T12:42:51.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.919+0000 7f357e7fc640 1 -- 172.21.15.89:0/1647352382 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1704676178 0 0) 0x7f358015ce30 con 0x7f35800c90d0 2023-10-30T12:42:51.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.923+0000 7f357effd640 1 --2- 172.21.15.89:0/1647352382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35800c8cf0 0x7f35800c9e00 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f35800c60c0 tx=0x7f356801c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5acdaa977deb90a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:51.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.923+0000 7f357e7fc640 1 -- 172.21.15.89:0/1647352382 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35680ab600 con 0x7f35800c8cf0 2023-10-30T12:42:51.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.923+0000 7f357e7fc640 1 -- 172.21.15.89:0/1647352382 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35680a7050 con 0x7f35800c8cf0 2023-10-30T12:42:51.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.923+0000 7f357e7fc640 1 -- 172.21.15.89:0/1647352382 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35680ab980 con 0x7f35800c8cf0 2023-10-30T12:42:51.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.923+0000 7f358550f640 1 -- 172.21.15.89:0/1647352382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35800c8cf0 msgr2=0x7f35800c9e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:51.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.923+0000 7f358550f640 1 --2- 172.21.15.89:0/1647352382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35800c8cf0 0x7f35800c9e00 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f35800c60c0 tx=0x7f356801c430 comp rx=0 tx=0).stop 2023-10-30T12:42:51.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.923+0000 7f358550f640 1 -- 172.21.15.89:0/1647352382 shutdown_connections 2023-10-30T12:42:51.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.923+0000 7f358550f640 1 --2- 172.21.15.89:0/1647352382 >> v2:172.21.15.192:3300/0 conn(0x7f35800ca340 0x7f358015d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:51.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.923+0000 7f358550f640 1 --2- 172.21.15.89:0/1647352382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35800c8cf0 0x7f35800c9e00 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:51.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.923+0000 7f358550f640 1 -- 172.21.15.89:0/1647352382 >> 172.21.15.89:0/1647352382 conn(0x7f358006be40 msgr2=0x7f358006c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:51.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.923+0000 7f358550f640 1 -- 172.21.15.89:0/1647352382 shutdown_connections 2023-10-30T12:42:51.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.923+0000 7f358550f640 1 -- 172.21.15.89:0/1647352382 wait complete. 2023-10-30T12:42:51.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.927+0000 7f358550f640 1 Processor -- start 2023-10-30T12:42:51.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.927+0000 7f358550f640 1 -- start start 2023-10-30T12:42:51.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.927+0000 7f358550f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35800c8cf0 0x7f3580153140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:51.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.927+0000 7f358550f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35800c90d0 0x7f35800c39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:51.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.927+0000 7f358550f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35800ca340 0x7f35800c3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:51.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.927+0000 7f358550f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35801f2910 con 0x7f35800c8cf0 2023-10-30T12:42:51.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.927+0000 7f358550f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35801f5d80 con 0x7f35800c90d0 2023-10-30T12:42:51.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.927+0000 7f358550f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35801f1d00 con 0x7f35800ca340 2023-10-30T12:42:51.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.927+0000 7f357effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35800c8cf0 0x7f3580153140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:51.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.927+0000 7f357effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35800c8cf0 0x7f3580153140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41918/0 (socket says 172.21.15.89:41918) 2023-10-30T12:42:51.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.927+0000 7f357effd640 1 -- 172.21.15.89:0/3466597254 learned_addr learned my addr 172.21.15.89:0/3466597254 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:51.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.927+0000 7f35767fc640 1 --2- 172.21.15.89:0/3466597254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35800c90d0 0x7f35800c39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:51.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.927+0000 7f357f7fe640 1 --2- 172.21.15.89:0/3466597254 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35800ca340 0x7f35800c3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:51.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.927+0000 7f357effd640 1 -- 172.21.15.89:0/3466597254 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35800ca340 msgr2=0x7f35800c3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:51.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.927+0000 7f357effd640 1 --2- 172.21.15.89:0/3466597254 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35800ca340 0x7f35800c3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:51.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.927+0000 7f357effd640 1 -- 172.21.15.89:0/3466597254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35800c90d0 msgr2=0x7f35800c39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:51.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.927+0000 7f357effd640 1 --2- 172.21.15.89:0/3466597254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35800c90d0 0x7f35800c39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:51.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.927+0000 7f357effd640 1 -- 172.21.15.89:0/3466597254 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3568009420 con 0x7f35800c8cf0 2023-10-30T12:42:51.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.927+0000 7f35767fc640 1 --2- 172.21.15.89:0/3466597254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35800c90d0 0x7f35800c39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:42:51.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.927+0000 7f357effd640 1 --2- 172.21.15.89:0/3466597254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35800c8cf0 0x7f3580153140 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f356801a060 tx=0x7f356801ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:51.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.927+0000 7f3577fff640 1 -- 172.21.15.89:0/3466597254 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35680b4870 con 0x7f35800c8cf0 2023-10-30T12:42:51.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.927+0000 7f3577fff640 1 -- 172.21.15.89:0/3466597254 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35680b49f0 con 0x7f35800c8cf0 2023-10-30T12:42:51.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.927+0000 7f358550f640 1 -- 172.21.15.89:0/3466597254 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35801f1fe0 con 0x7f35800c8cf0 2023-10-30T12:42:51.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.927+0000 7f3577fff640 1 -- 172.21.15.89:0/3466597254 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35680b4da0 con 0x7f35800c8cf0 2023-10-30T12:42:51.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.927+0000 7f358550f640 1 -- 172.21.15.89:0/3466597254 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35801f25a0 con 0x7f35800c8cf0 2023-10-30T12:42:51.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.931+0000 7f358550f640 1 -- 172.21.15.89:0/3466597254 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f358004e170 con 0x7f35800c8cf0 2023-10-30T12:42:51.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.931+0000 7f3577fff640 1 -- 172.21.15.89:0/3466597254 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f35680b2070 con 0x7f35800c8cf0 2023-10-30T12:42:51.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.931+0000 7f3577fff640 1 --2- 172.21.15.89:0/3466597254 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f35600664d0 0x7f3560068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:51.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.931+0000 7f3577fff640 1 -- 172.21.15.89:0/3466597254 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6320+0+0 (secure 0 0 0) 0x7f356811e440 con 0x7f35800c8cf0 2023-10-30T12:42:51.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.931+0000 7f35767fc640 1 --2- 172.21.15.89:0/3466597254 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f35600664d0 0x7f3560068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:51.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.935+0000 7f35767fc640 1 --2- 172.21.15.89:0/3466597254 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f35600664d0 0x7f3560068990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f35800cc460 tx=0x7f3570007c40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:51.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:51.935+0000 7f3577fff640 1 -- 172.21.15.89:0/3466597254 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f35680eb550 con 0x7f35800c8cf0 2023-10-30T12:42:52.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:52.115+0000 7f358550f640 1 -- 172.21.15.89:0/3466597254 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35800ca9d0 con 0x7f35800c8cf0 2023-10-30T12:42:52.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:52.115+0000 7f3577fff640 1 -- 172.21.15.89:0/3466597254 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f35680b0090 con 0x7f35800c8cf0 2023-10-30T12:42:52.111 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:42:52.112 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:42:50.247004+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:52.112 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 67 2023-10-30T12:42:52.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:52.115+0000 7f358550f640 1 -- 172.21.15.89:0/3466597254 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f35600664d0 msgr2=0x7f3560068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:52.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:52.115+0000 7f358550f640 1 --2- 172.21.15.89:0/3466597254 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f35600664d0 0x7f3560068990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f35800cc460 tx=0x7f3570007c40 comp rx=0 tx=0).stop 2023-10-30T12:42:52.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:52.115+0000 7f358550f640 1 -- 172.21.15.89:0/3466597254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35800c8cf0 msgr2=0x7f3580153140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:52.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:52.115+0000 7f358550f640 1 --2- 172.21.15.89:0/3466597254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35800c8cf0 0x7f3580153140 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f356801a060 tx=0x7f356801ac70 comp rx=0 tx=0).stop 2023-10-30T12:42:52.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:52.115+0000 7f358550f640 1 -- 172.21.15.89:0/3466597254 shutdown_connections 2023-10-30T12:42:52.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:52.115+0000 7f358550f640 1 --2- 172.21.15.89:0/3466597254 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f35600664d0 0x7f3560068990 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:52.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:52.115+0000 7f358550f640 1 --2- 172.21.15.89:0/3466597254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35800c90d0 0x7f35800c39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:52.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:52.115+0000 7f358550f640 1 --2- 172.21.15.89:0/3466597254 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35800ca340 0x7f35800c3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:52.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:52.115+0000 7f358550f640 1 --2- 172.21.15.89:0/3466597254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35800c8cf0 0x7f3580153140 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:52.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:52.115+0000 7f358550f640 1 -- 172.21.15.89:0/3466597254 >> 172.21.15.89:0/3466597254 conn(0x7f358006be40 msgr2=0x7f35800b3470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:52.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:52.115+0000 7f358550f640 1 -- 172.21.15.89:0/3466597254 shutdown_connections 2023-10-30T12:42:52.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:52.115+0000 7f358550f640 1 -- 172.21.15.89:0/3466597254 wait complete. 2023-10-30T12:42:52.132 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=5018 2023-10-30T12:42:52.132 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.d 2023-10-30T12:42:56.133 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-10-30T12:42:56.134 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-30T12:42:56.134 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-30T12:42:56.137 INFO:tasks.ceph.mds.d:Started 2023-10-30T12:42:56.138 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:56.214 INFO:tasks.ceph.mds.d.smithi192.stdout:starting mds.d at 2023-10-30T12:42:56.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.287+0000 7f73c88d6640 1 Processor -- start 2023-10-30T12:42:56.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.287+0000 7f73c88d6640 1 -- start start 2023-10-30T12:42:56.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.291+0000 7f73c88d6640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f73c41538a0 0x7f73c415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:56.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.291+0000 7f73c88d6640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f73c40cf160 0x7f73c40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:56.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.291+0000 7f73c88d6640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c40cb800 con 0x7f73c40ca0e0 2023-10-30T12:42:56.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.291+0000 7f73c88d6640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73c415dc30 con 0x7f73c41538a0 2023-10-30T12:42:56.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.291+0000 7f73c88d6640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73c40cb330 con 0x7f73c40cf160 2023-10-30T12:42:56.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.291+0000 7f73c2d9d640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f73c40cf160 0x7f73c40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:56.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.291+0000 7f73c259c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f73c41538a0 0x7f73c415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:56.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.291+0000 7f73c2d9d640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f73c40cf160 0x7f73c40cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:52960/0 (socket says 172.21.15.89:52960) 2023-10-30T12:42:56.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.291+0000 7f73c2d9d640 1 -- 172.21.15.89:0/4196380702 learned_addr learned my addr 172.21.15.89:0/4196380702 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:56.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.291+0000 7f73c1d9b640 1 -- 172.21.15.89:0/4196380702 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 254802174 0 0) 0x7f73c40cb800 con 0x7f73c40ca0e0 2023-10-30T12:42:56.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.291+0000 7f73c1d9b640 1 -- 172.21.15.89:0/4196380702 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73b0003530 con 0x7f73c40ca0e0 2023-10-30T12:42:56.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.291+0000 7f73c259c640 1 -- 172.21.15.89:0/4196380702 >> v2:172.21.15.192:3300/0 conn(0x7f73c40cf160 msgr2=0x7f73c40cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:56.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.291+0000 7f73c259c640 1 --2- 172.21.15.89:0/4196380702 >> v2:172.21.15.192:3300/0 conn(0x7f73c40cf160 0x7f73c40cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:56.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.291+0000 7f73c259c640 1 -- 172.21.15.89:0/4196380702 >> v1:172.21.15.89:6789/0 conn(0x7f73c40ca0e0 legacy=0x7f73c415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:56.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.291+0000 7f73c259c640 1 -- 172.21.15.89:0/4196380702 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c40cad70 con 0x7f73c41538a0 2023-10-30T12:42:56.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.295+0000 7f73c259c640 1 --2- 172.21.15.89:0/4196380702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73c41538a0 0x7f73c415d6f0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f73c40b5330 tx=0x7f73b401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4138037283f5e018 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:56.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.295+0000 7f73c1d9b640 1 -- 172.21.15.89:0/4196380702 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b40a8950 con 0x7f73c41538a0 2023-10-30T12:42:56.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.295+0000 7f73c1d9b640 1 -- 172.21.15.89:0/4196380702 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b40a8ab0 con 0x7f73c41538a0 2023-10-30T12:42:56.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.295+0000 7f73c1d9b640 1 -- 172.21.15.89:0/4196380702 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b40a8e40 con 0x7f73c41538a0 2023-10-30T12:42:56.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.295+0000 7f73c88d6640 1 -- 172.21.15.89:0/4196380702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73c41538a0 msgr2=0x7f73c415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:56.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.295+0000 7f73c88d6640 1 --2- 172.21.15.89:0/4196380702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73c41538a0 0x7f73c415d6f0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f73c40b5330 tx=0x7f73b401ec50 comp rx=0 tx=0).stop 2023-10-30T12:42:56.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.295+0000 7f73c88d6640 1 -- 172.21.15.89:0/4196380702 shutdown_connections 2023-10-30T12:42:56.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.295+0000 7f73c88d6640 1 --2- 172.21.15.89:0/4196380702 >> v2:172.21.15.192:3300/0 conn(0x7f73c40cf160 0x7f73c40cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:56.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.295+0000 7f73c88d6640 1 --2- 172.21.15.89:0/4196380702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73c41538a0 0x7f73c415d6f0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:56.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.295+0000 7f73c88d6640 1 -- 172.21.15.89:0/4196380702 >> 172.21.15.89:0/4196380702 conn(0x7f73c406bfe0 msgr2=0x7f73c406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:56.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.295+0000 7f73c88d6640 1 -- 172.21.15.89:0/4196380702 shutdown_connections 2023-10-30T12:42:56.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.295+0000 7f73c88d6640 1 -- 172.21.15.89:0/4196380702 wait complete. 2023-10-30T12:42:56.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.299+0000 7f73c88d6640 1 Processor -- start 2023-10-30T12:42:56.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.299+0000 7f73c88d6640 1 -- start start 2023-10-30T12:42:56.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.299+0000 7f73c88d6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73c40ca0e0 0x7f73c40cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:56.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.299+0000 7f73c88d6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73c40cf160 0x7f73c40ce620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:56.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.299+0000 7f73c88d6640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73c41538a0 0x7f73c40ceb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:56.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.299+0000 7f73c88d6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c4158c30 con 0x7f73c40ca0e0 2023-10-30T12:42:56.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.299+0000 7f73c88d6640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c4158320 con 0x7f73c40cf160 2023-10-30T12:42:56.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.299+0000 7f73c88d6640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73c41e6d20 con 0x7f73c41538a0 2023-10-30T12:42:56.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.299+0000 7f73c259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73c40ca0e0 0x7f73c40cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:56.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.299+0000 7f73c259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73c40ca0e0 0x7f73c40cd2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41938/0 (socket says 172.21.15.89:41938) 2023-10-30T12:42:56.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.299+0000 7f73c259c640 1 -- 172.21.15.89:0/1295384448 learned_addr learned my addr 172.21.15.89:0/1295384448 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:56.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.299+0000 7f73c2d9d640 1 --2- 172.21.15.89:0/1295384448 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73c41538a0 0x7f73c40ceb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:56.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.299+0000 7f73bbfff640 1 --2- 172.21.15.89:0/1295384448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73c40cf160 0x7f73c40ce620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:56.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.299+0000 7f73c259c640 1 -- 172.21.15.89:0/1295384448 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73c41538a0 msgr2=0x7f73c40ceb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:56.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.299+0000 7f73c259c640 1 --2- 172.21.15.89:0/1295384448 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73c41538a0 0x7f73c40ceb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:56.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.299+0000 7f73c259c640 1 -- 172.21.15.89:0/1295384448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73c40cf160 msgr2=0x7f73c40ce620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:56.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.299+0000 7f73c259c640 1 --2- 172.21.15.89:0/1295384448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73c40cf160 0x7f73c40ce620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:56.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.299+0000 7f73c259c640 1 -- 172.21.15.89:0/1295384448 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b4009420 con 0x7f73c40ca0e0 2023-10-30T12:42:56.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.299+0000 7f73c2d9d640 1 --2- 172.21.15.89:0/1295384448 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73c41538a0 0x7f73c40ceb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:42:56.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.299+0000 7f73c259c640 1 --2- 172.21.15.89:0/1295384448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73c40ca0e0 0x7f73c40cd2f0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f73b40b3040 tx=0x7f73b401b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:56.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.299+0000 7f73baffd640 1 -- 172.21.15.89:0/1295384448 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b40b0d20 con 0x7f73c40ca0e0 2023-10-30T12:42:56.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.299+0000 7f73baffd640 1 -- 172.21.15.89:0/1295384448 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b40b9430 con 0x7f73c40ca0e0 2023-10-30T12:42:56.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.299+0000 7f73baffd640 1 -- 172.21.15.89:0/1295384448 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b40b0ea0 con 0x7f73c40ca0e0 2023-10-30T12:42:56.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.299+0000 7f73c88d6640 1 -- 172.21.15.89:0/1295384448 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73c41e6760 con 0x7f73c40ca0e0 2023-10-30T12:42:56.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.299+0000 7f73c88d6640 1 -- 172.21.15.89:0/1295384448 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73c40cde20 con 0x7f73c40ca0e0 2023-10-30T12:42:56.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.299+0000 7f73c88d6640 1 -- 172.21.15.89:0/1295384448 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73c40b2a50 con 0x7f73c40ca0e0 2023-10-30T12:42:56.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.303+0000 7f73baffd640 1 -- 172.21.15.89:0/1295384448 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f73b40b7040 con 0x7f73c40ca0e0 2023-10-30T12:42:56.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.303+0000 7f73baffd640 1 --2- 172.21.15.89:0/1295384448 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f73a40664a0 0x7f73a4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:56.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.303+0000 7f73baffd640 1 -- 172.21.15.89:0/1295384448 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6320+0+0 (secure 0 0 0) 0x7f73b4121ab0 con 0x7f73c40ca0e0 2023-10-30T12:42:56.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.303+0000 7f73bbfff640 1 --2- 172.21.15.89:0/1295384448 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f73a40664a0 0x7f73a4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:56.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.303+0000 7f73bbfff640 1 --2- 172.21.15.89:0/1295384448 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f73a40664a0 0x7f73a4068960 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f73c4150f10 tx=0x7f73ac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:56.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.307+0000 7f73baffd640 1 -- 172.21.15.89:0/1295384448 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f73b40eed00 con 0x7f73c40ca0e0 2023-10-30T12:42:56.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.491+0000 7f73c88d6640 1 -- 172.21.15.89:0/1295384448 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73c40cdc80 con 0x7f73c40ca0e0 2023-10-30T12:42:56.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.491+0000 7f73baffd640 1 -- 172.21.15.89:0/1295384448 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+5736 (secure 0 0 0) 0x7f73c40cdc80 con 0x7f73c40ca0e0 2023-10-30T12:42:56.487 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:42:56.487 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:42:55.156710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:56.488 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 68 2023-10-30T12:42:56.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.491+0000 7f73c88d6640 1 -- 172.21.15.89:0/1295384448 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f73a40664a0 msgr2=0x7f73a4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:56.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.491+0000 7f73c88d6640 1 --2- 172.21.15.89:0/1295384448 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f73a40664a0 0x7f73a4068960 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f73c4150f10 tx=0x7f73ac009040 comp rx=0 tx=0).stop 2023-10-30T12:42:56.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.491+0000 7f73c88d6640 1 -- 172.21.15.89:0/1295384448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73c40ca0e0 msgr2=0x7f73c40cd2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:56.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.491+0000 7f73c88d6640 1 --2- 172.21.15.89:0/1295384448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73c40ca0e0 0x7f73c40cd2f0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f73b40b3040 tx=0x7f73b401b9f0 comp rx=0 tx=0).stop 2023-10-30T12:42:56.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.491+0000 7f73c88d6640 1 -- 172.21.15.89:0/1295384448 shutdown_connections 2023-10-30T12:42:56.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.491+0000 7f73c88d6640 1 --2- 172.21.15.89:0/1295384448 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f73a40664a0 0x7f73a4068960 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:56.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.491+0000 7f73c88d6640 1 --2- 172.21.15.89:0/1295384448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73c40cf160 0x7f73c40ce620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:56.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.491+0000 7f73c88d6640 1 --2- 172.21.15.89:0/1295384448 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73c41538a0 0x7f73c40ceb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:56.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.491+0000 7f73c88d6640 1 --2- 172.21.15.89:0/1295384448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73c40ca0e0 0x7f73c40cd2f0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:56.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.491+0000 7f73c88d6640 1 -- 172.21.15.89:0/1295384448 >> 172.21.15.89:0/1295384448 conn(0x7f73c406bfe0 msgr2=0x7f73c41526f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:56.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.491+0000 7f73c88d6640 1 -- 172.21.15.89:0/1295384448 shutdown_connections 2023-10-30T12:42:56.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:56.491+0000 7f73c88d6640 1 -- 172.21.15.89:0/1295384448 wait complete. 2023-10-30T12:42:56.508 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-10-30T12:42:57.277 INFO:tasks.ceph.mds.b.smithi192.stderr:2023-10-30T12:42:57.279+0000 7f5231cd7640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T12:42:57.277 INFO:tasks.ceph.mds.b.smithi192.stderr:2023-10-30T12:42:57.279+0000 7f5231cd7640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T12:42:57.277 INFO:tasks.ceph.mds.b.smithi192.stderr:2023-10-30T12:42:57.279+0000 7f5231cd7640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T12:42:58.509 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:58.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc40531d640 1 Processor -- start 2023-10-30T12:42:58.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc40531d640 1 -- start start 2023-10-30T12:42:58.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc40531d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc40015a410 0x7fc40006a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:58.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc40531d640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4000c6c10 con 0x7fc40006cb80 2023-10-30T12:42:58.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc40531d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4000c6ef0 con 0x7fc400069e90 2023-10-30T12:42:58.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc40531d640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4000c8570 con 0x7fc40015a410 2023-10-30T12:42:58.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc3ff7fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc40006cb80 0x7fc40006d010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:34772/0 (socket says 172.21.15.89:34772) 2023-10-30T12:42:58.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc3ff7fe640 1 -- 172.21.15.89:0/2141820724 learned_addr learned my addr 172.21.15.89:0/2141820724 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:42:58.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc3fe7fc640 1 --2- 172.21.15.89:0/2141820724 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc40015a410 0x7fc40006a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:58.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc3fdffb640 1 -- 172.21.15.89:0/2141820724 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1949262163 0 0) 0x7fc4000c6c10 con 0x7fc40006cb80 2023-10-30T12:42:58.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc3fdffb640 1 -- 172.21.15.89:0/2141820724 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3e8003530 con 0x7fc40006cb80 2023-10-30T12:42:58.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc3fdffb640 1 -- 172.21.15.89:0/2141820724 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2517017344 0 0) 0x7fc3e8003530 con 0x7fc40006cb80 2023-10-30T12:42:58.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc3fdffb640 1 -- 172.21.15.89:0/2141820724 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc40015a410 msgr2=0x7fc40006a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:58.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc3fdffb640 1 --2- 172.21.15.89:0/2141820724 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc40015a410 0x7fc40006a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:58.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc3fdffb640 1 -- 172.21.15.89:0/2141820724 >> v1:172.21.15.192:6789/0 conn(0x7fc400069e90 legacy=0x7fc4000c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:58.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc3fdffb640 1 -- 172.21.15.89:0/2141820724 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4000c71d0 con 0x7fc40006cb80 2023-10-30T12:42:58.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc3fdffb640 1 -- 172.21.15.89:0/2141820724 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7fc3f8001a00 con 0x7fc40006cb80 2023-10-30T12:42:58.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc3fdffb640 1 -- 172.21.15.89:0/2141820724 >> v1:172.21.15.89:6789/0 conn(0x7fc40006cb80 legacy=0x7fc40006d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:58.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc3fdffb640 1 --2- 172.21.15.89:0/2141820724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3e80039b0 0x7fc3e8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:58.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc3fdffb640 1 --2- 172.21.15.89:0/2141820724 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc40015a410 0x7fc3e80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:58.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc3fdffb640 1 --2- 172.21.15.89:0/2141820724 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3e8004d10 0x7fc3e8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:58.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc3fdffb640 1 -- 172.21.15.89:0/2141820724 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4000c8570 con 0x7fc3e8004d10 2023-10-30T12:42:58.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc3fdffb640 1 -- 172.21.15.89:0/2141820724 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e800b990 con 0x7fc3e80039b0 2023-10-30T12:42:58.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc3fdffb640 1 -- 172.21.15.89:0/2141820724 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e800bb50 con 0x7fc40015a410 2023-10-30T12:42:58.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc3ff7fe640 1 --2- 172.21.15.89:0/2141820724 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3e8004d10 0x7fc3e8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:58.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc3fe7fc640 1 --2- 172.21.15.89:0/2141820724 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc40015a410 0x7fc3e80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:58.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc3feffd640 1 --2- 172.21.15.89:0/2141820724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3e80039b0 0x7fc3e8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:58.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc3ff7fe640 1 -- 172.21.15.89:0/2141820724 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc40015a410 msgr2=0x7fc3e80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:58.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc3ff7fe640 1 --2- 172.21.15.89:0/2141820724 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc40015a410 0x7fc3e80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:58.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc3ff7fe640 1 -- 172.21.15.89:0/2141820724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3e80039b0 msgr2=0x7fc3e8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:58.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc3ff7fe640 1 --2- 172.21.15.89:0/2141820724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3e80039b0 0x7fc3e8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:58.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc3ff7fe640 1 -- 172.21.15.89:0/2141820724 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e8003530 con 0x7fc3e8004d10 2023-10-30T12:42:58.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.663+0000 7fc3feffd640 1 --2- 172.21.15.89:0/2141820724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3e80039b0 0x7fc3e8003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:42:58.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.667+0000 7fc3ff7fe640 1 --2- 172.21.15.89:0/2141820724 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3e8004d10 0x7fc3e8005190 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fc3f800a7d0 tx=0x7fc3f801fa20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89db69eea0a40364 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:58.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.667+0000 7fc3fdffb640 1 -- 172.21.15.89:0/2141820724 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f80aa650 con 0x7fc3e8004d10 2023-10-30T12:42:58.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.667+0000 7fc3fdffb640 1 -- 172.21.15.89:0/2141820724 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f80aa7d0 con 0x7fc3e8004d10 2023-10-30T12:42:58.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.667+0000 7fc3fdffb640 1 -- 172.21.15.89:0/2141820724 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f80aab80 con 0x7fc3e8004d10 2023-10-30T12:42:58.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.667+0000 7fc40531d640 1 -- 172.21.15.89:0/2141820724 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3e8004d10 msgr2=0x7fc3e8005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:58.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.667+0000 7fc40531d640 1 --2- 172.21.15.89:0/2141820724 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3e8004d10 0x7fc3e8005190 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fc3f800a7d0 tx=0x7fc3f801fa20 comp rx=0 tx=0).stop 2023-10-30T12:42:58.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.667+0000 7fc3ff7fe640 1 -- 172.21.15.89:0/2141820724 reap_dead start 2023-10-30T12:42:58.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.667+0000 7fc40531d640 1 -- 172.21.15.89:0/2141820724 shutdown_connections 2023-10-30T12:42:58.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.667+0000 7fc40531d640 1 -- 172.21.15.89:0/2141820724 >> 172.21.15.89:0/2141820724 conn(0x7fc40006dfd0 msgr2=0x7fc40006e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:58.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.667+0000 7fc40531d640 1 -- 172.21.15.89:0/2141820724 shutdown_connections 2023-10-30T12:42:58.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.667+0000 7fc40531d640 1 -- 172.21.15.89:0/2141820724 wait complete. 2023-10-30T12:42:58.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.671+0000 7fc40531d640 1 Processor -- start 2023-10-30T12:42:58.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.671+0000 7fc40531d640 1 -- start start 2023-10-30T12:42:58.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.671+0000 7fc40531d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3e8004d10 0x7fc40015d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:58.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.671+0000 7fc40531d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4001ea720 0x7fc400158920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:58.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.671+0000 7fc40531d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc400152010 0x7fc400152490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:58.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.671+0000 7fc40531d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4000c4c70 con 0x7fc3e8004d10 2023-10-30T12:42:58.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.671+0000 7fc40531d640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4001f6cd0 con 0x7fc4001ea720 2023-10-30T12:42:58.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.671+0000 7fc40531d640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4001529d0 con 0x7fc400152010 2023-10-30T12:42:58.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.671+0000 7fc3feffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3e8004d10 0x7fc40015d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:58.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.671+0000 7fc3feffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3e8004d10 0x7fc40015d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41944/0 (socket says 172.21.15.89:41944) 2023-10-30T12:42:58.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.671+0000 7fc3feffd640 1 -- 172.21.15.89:0/1473101022 learned_addr learned my addr 172.21.15.89:0/1473101022 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:42:58.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.671+0000 7fc3fe7fc640 1 --2- 172.21.15.89:0/1473101022 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4001ea720 0x7fc400158920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:58.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.671+0000 7fc3ff7fe640 1 --2- 172.21.15.89:0/1473101022 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc400152010 0x7fc400152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:58.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.671+0000 7fc3feffd640 1 -- 172.21.15.89:0/1473101022 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc400152010 msgr2=0x7fc400152490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:58.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.671+0000 7fc3feffd640 1 --2- 172.21.15.89:0/1473101022 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc400152010 0x7fc400152490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:58.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.671+0000 7fc3feffd640 1 -- 172.21.15.89:0/1473101022 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4001ea720 msgr2=0x7fc400158920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:58.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.671+0000 7fc3feffd640 1 --2- 172.21.15.89:0/1473101022 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4001ea720 0x7fc400158920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:58.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.671+0000 7fc3feffd640 1 -- 172.21.15.89:0/1473101022 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc40006b7f0 con 0x7fc3e8004d10 2023-10-30T12:42:58.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.671+0000 7fc3feffd640 1 --2- 172.21.15.89:0/1473101022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3e8004d10 0x7fc40015d510 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fc3f000e520 tx=0x7fc3f000e920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:58.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.671+0000 7fc3e77fe640 1 -- 172.21.15.89:0/1473101022 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f00052e0 con 0x7fc3e8004d10 2023-10-30T12:42:58.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.671+0000 7fc3e77fe640 1 -- 172.21.15.89:0/1473101022 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f0005460 con 0x7fc3e8004d10 2023-10-30T12:42:58.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.671+0000 7fc40531d640 1 -- 172.21.15.89:0/1473101022 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4000c57e0 con 0x7fc3e8004d10 2023-10-30T12:42:58.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.671+0000 7fc3e77fe640 1 -- 172.21.15.89:0/1473101022 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f0003840 con 0x7fc3e8004d10 2023-10-30T12:42:58.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.671+0000 7fc40531d640 1 -- 172.21.15.89:0/1473101022 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4001f6430 con 0x7fc3e8004d10 2023-10-30T12:42:58.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.675+0000 7fc40531d640 1 -- 172.21.15.89:0/1473101022 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc40006c640 con 0x7fc3e8004d10 2023-10-30T12:42:58.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.675+0000 7fc3e77fe640 1 -- 172.21.15.89:0/1473101022 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc3f00039a0 con 0x7fc3e8004d10 2023-10-30T12:42:58.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.675+0000 7fc3e77fe640 1 --2- 172.21.15.89:0/1473101022 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc3ec0664d0 0x7fc3ec068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:58.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.675+0000 7fc3e77fe640 1 -- 172.21.15.89:0/1473101022 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6320+0+0 (secure 0 0 0) 0x7fc3f0083f60 con 0x7fc3e8004d10 2023-10-30T12:42:58.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.675+0000 7fc3fe7fc640 1 --2- 172.21.15.89:0/1473101022 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc3ec0664d0 0x7fc3ec068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:58.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.675+0000 7fc3fe7fc640 1 --2- 172.21.15.89:0/1473101022 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc3ec0664d0 0x7fc3ec068990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc4001ea6d0 tx=0x7fc3f4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:58.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.679+0000 7fc3e77fe640 1 -- 172.21.15.89:0/1473101022 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc40006c640 con 0x7fc3e8004d10 2023-10-30T12:42:58.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.859+0000 7fc40531d640 1 -- 172.21.15.89:0/1473101022 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc40006c640 con 0x7fc3e8004d10 2023-10-30T12:42:58.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.863+0000 7fc3e77fe640 1 -- 172.21.15.89:0/1473101022 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+6530 (secure 0 0 0) 0x7fc40006c640 con 0x7fc3e8004d10 2023-10-30T12:42:58.858 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:42:58.859 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5728,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:42:55.156710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:58.860 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 69 2023-10-30T12:42:58.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.863+0000 7fc40531d640 1 -- 172.21.15.89:0/1473101022 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc3ec0664d0 msgr2=0x7fc3ec068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:58.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.863+0000 7fc40531d640 1 --2- 172.21.15.89:0/1473101022 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc3ec0664d0 0x7fc3ec068990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc4001ea6d0 tx=0x7fc3f4009190 comp rx=0 tx=0).stop 2023-10-30T12:42:58.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.863+0000 7fc40531d640 1 -- 172.21.15.89:0/1473101022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3e8004d10 msgr2=0x7fc40015d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:58.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.863+0000 7fc40531d640 1 --2- 172.21.15.89:0/1473101022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3e8004d10 0x7fc40015d510 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fc3f000e520 tx=0x7fc3f000e920 comp rx=0 tx=0).stop 2023-10-30T12:42:58.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.863+0000 7fc40531d640 1 -- 172.21.15.89:0/1473101022 shutdown_connections 2023-10-30T12:42:58.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.863+0000 7fc40531d640 1 --2- 172.21.15.89:0/1473101022 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc3ec0664d0 0x7fc3ec068990 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:58.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.863+0000 7fc40531d640 1 --2- 172.21.15.89:0/1473101022 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc4001ea720 0x7fc400158920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:58.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.863+0000 7fc40531d640 1 --2- 172.21.15.89:0/1473101022 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc400152010 0x7fc400152490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:58.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.863+0000 7fc40531d640 1 --2- 172.21.15.89:0/1473101022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3e8004d10 0x7fc40015d510 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:58.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.863+0000 7fc40531d640 1 -- 172.21.15.89:0/1473101022 >> 172.21.15.89:0/1473101022 conn(0x7fc40006dfd0 msgr2=0x7fc40006e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:58.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.863+0000 7fc40531d640 1 -- 172.21.15.89:0/1473101022 shutdown_connections 2023-10-30T12:42:58.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:42:58.863+0000 7fc40531d640 1 -- 172.21.15.89:0/1473101022 wait complete. 2023-10-30T12:42:58.879 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-10-30T12:42:58.879 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 19 secs before thrashing 2023-10-30T12:43:17.372 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:43:17.376 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:43:17.881 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:43:18.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.047+0000 7f67a1846640 1 Processor -- start 2023-10-30T12:43:18.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.047+0000 7f67a1846640 1 -- start start 2023-10-30T12:43:18.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.051+0000 7f67a1846640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f679c0c6bd0 0x7f679c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:18.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.051+0000 7f67a1846640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f679c066cf0 con 0x7f679c06f7a0 2023-10-30T12:43:18.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.051+0000 7f67a1846640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f679c0c70d0 con 0x7f679c071110 2023-10-30T12:43:18.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.051+0000 7f67a1846640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f679c0700c0 con 0x7f679c0c6bd0 2023-10-30T12:43:18.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.051+0000 7f679affd640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f679c06f7a0 0x7f679c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57436/0 (socket says 172.21.15.89:57436) 2023-10-30T12:43:18.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.051+0000 7f679affd640 1 -- 172.21.15.89:0/360324058 learned_addr learned my addr 172.21.15.89:0/360324058 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:43:18.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.051+0000 7f67927fc640 1 --2- 172.21.15.89:0/360324058 >> v2:172.21.15.192:3300/0 conn(0x7f679c0c6bd0 0x7f679c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:18.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.051+0000 7f679a7fc640 1 -- 172.21.15.89:0/360324058 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 147696924 0 0) 0x7f679c066cf0 con 0x7f679c06f7a0 2023-10-30T12:43:18.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.051+0000 7f679a7fc640 1 -- 172.21.15.89:0/360324058 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6788003530 con 0x7f679c06f7a0 2023-10-30T12:43:18.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.051+0000 7f679a7fc640 1 -- 172.21.15.89:0/360324058 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 266734173 0 0) 0x7f6788003530 con 0x7f679c06f7a0 2023-10-30T12:43:18.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.051+0000 7f679a7fc640 1 -- 172.21.15.89:0/360324058 >> v2:172.21.15.192:3300/0 conn(0x7f679c0c6bd0 msgr2=0x7f679c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:18.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.051+0000 7f679a7fc640 1 --2- 172.21.15.89:0/360324058 >> v2:172.21.15.192:3300/0 conn(0x7f679c0c6bd0 0x7f679c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:18.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.051+0000 7f679a7fc640 1 -- 172.21.15.89:0/360324058 >> v1:172.21.15.192:6789/0 conn(0x7f679c071110 legacy=0x7f679c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:18.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.051+0000 7f679a7fc640 1 -- 172.21.15.89:0/360324058 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f679c0c7390 con 0x7f679c06f7a0 2023-10-30T12:43:18.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.051+0000 7f679a7fc640 1 -- 172.21.15.89:0/360324058 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f678c001a00 con 0x7f679c06f7a0 2023-10-30T12:43:18.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.051+0000 7f679a7fc640 1 -- 172.21.15.89:0/360324058 >> v1:172.21.15.89:6789/0 conn(0x7f679c06f7a0 legacy=0x7f679c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:18.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.051+0000 7f679a7fc640 1 --2- 172.21.15.89:0/360324058 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6788003980 0x7f6788003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:18.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.051+0000 7f679a7fc640 1 --2- 172.21.15.89:0/360324058 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f67880047d0 0x7f6788004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:18.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.051+0000 7f679a7fc640 1 --2- 172.21.15.89:0/360324058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6788005190 0x7f678800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:18.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.051+0000 7f679a7fc640 1 -- 172.21.15.89:0/360324058 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c0700c0 con 0x7f6788005190 2023-10-30T12:43:18.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.051+0000 7f679a7fc640 1 -- 172.21.15.89:0/360324058 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678800c040 con 0x7f6788003980 2023-10-30T12:43:18.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.051+0000 7f679a7fc640 1 -- 172.21.15.89:0/360324058 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f678800c200 con 0x7f67880047d0 2023-10-30T12:43:18.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.051+0000 7f679b7fe640 1 --2- 172.21.15.89:0/360324058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6788005190 0x7f678800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:18.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.051+0000 7f679affd640 1 --2- 172.21.15.89:0/360324058 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6788003980 0x7f6788003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:18.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.051+0000 7f67927fc640 1 --2- 172.21.15.89:0/360324058 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f67880047d0 0x7f6788004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:18.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.051+0000 7f679b7fe640 1 -- 172.21.15.89:0/360324058 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f67880047d0 msgr2=0x7f6788004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:18.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.051+0000 7f679b7fe640 1 --2- 172.21.15.89:0/360324058 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f67880047d0 0x7f6788004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:18.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.051+0000 7f679b7fe640 1 -- 172.21.15.89:0/360324058 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6788003980 msgr2=0x7f6788003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:18.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.051+0000 7f679b7fe640 1 --2- 172.21.15.89:0/360324058 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6788003980 0x7f6788003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:18.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.051+0000 7f679b7fe640 1 -- 172.21.15.89:0/360324058 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6788003530 con 0x7f6788005190 2023-10-30T12:43:18.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.055+0000 7f679b7fe640 1 -- 172.21.15.89:0/360324058 reap_dead start 2023-10-30T12:43:18.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.055+0000 7f679b7fe640 1 --2- 172.21.15.89:0/360324058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6788005190 0x7f678800b990 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f6794001780 tx=0x7f679401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e8cc954bfbacdb0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:18.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.055+0000 7f679a7fc640 1 -- 172.21.15.89:0/360324058 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67940052d0 con 0x7f6788005190 2023-10-30T12:43:18.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.055+0000 7f679a7fc640 1 -- 172.21.15.89:0/360324058 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67940a7050 con 0x7f6788005190 2023-10-30T12:43:18.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.055+0000 7f679a7fc640 1 -- 172.21.15.89:0/360324058 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6794003840 con 0x7f6788005190 2023-10-30T12:43:18.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.055+0000 7f67a1846640 1 -- 172.21.15.89:0/360324058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6788005190 msgr2=0x7f678800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:18.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.055+0000 7f67a1846640 1 --2- 172.21.15.89:0/360324058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6788005190 0x7f678800b990 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f6794001780 tx=0x7f679401c400 comp rx=0 tx=0).stop 2023-10-30T12:43:18.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.055+0000 7f67a1846640 1 -- 172.21.15.89:0/360324058 shutdown_connections 2023-10-30T12:43:18.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.055+0000 7f67a1846640 1 --2- 172.21.15.89:0/360324058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6788005190 0x7f678800b990 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:18.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.055+0000 7f67a1846640 1 -- 172.21.15.89:0/360324058 >> 172.21.15.89:0/360324058 conn(0x7f679c0c7910 msgr2=0x7f679c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:18.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.055+0000 7f67a1846640 1 -- 172.21.15.89:0/360324058 shutdown_connections 2023-10-30T12:43:18.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.055+0000 7f67a1846640 1 -- 172.21.15.89:0/360324058 wait complete. 2023-10-30T12:43:18.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.059+0000 7f67a1846640 1 Processor -- start 2023-10-30T12:43:18.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.059+0000 7f67a1846640 1 -- start start 2023-10-30T12:43:18.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.059+0000 7f67a1846640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6788005190 0x7f679c1eeae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:18.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.059+0000 7f67a1846640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f679c14fc50 0x7f679c1ef020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:18.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.059+0000 7f67a1846640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f679c1588a0 0x7f679c158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:18.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.059+0000 7f67a1846640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c1ef6f0 con 0x7f6788005190 2023-10-30T12:43:18.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.059+0000 7f67a1846640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c159260 con 0x7f679c1588a0 2023-10-30T12:43:18.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.059+0000 7f67a1846640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f679c15a2d0 con 0x7f679c14fc50 2023-10-30T12:43:18.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.059+0000 7f679affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6788005190 0x7f679c1eeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:18.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.059+0000 7f679affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6788005190 0x7f679c1eeae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54030/0 (socket says 172.21.15.89:54030) 2023-10-30T12:43:18.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.059+0000 7f679affd640 1 -- 172.21.15.89:0/2971384270 learned_addr learned my addr 172.21.15.89:0/2971384270 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:43:18.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.059+0000 7f679b7fe640 1 --2- 172.21.15.89:0/2971384270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f679c1588a0 0x7f679c158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:18.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.059+0000 7f67927fc640 1 --2- 172.21.15.89:0/2971384270 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f679c14fc50 0x7f679c1ef020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:18.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.059+0000 7f679affd640 1 -- 172.21.15.89:0/2971384270 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f679c14fc50 msgr2=0x7f679c1ef020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:18.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.059+0000 7f679affd640 1 --2- 172.21.15.89:0/2971384270 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f679c14fc50 0x7f679c1ef020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:18.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.059+0000 7f679affd640 1 -- 172.21.15.89:0/2971384270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f679c1588a0 msgr2=0x7f679c158d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:18.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.059+0000 7f679affd640 1 --2- 172.21.15.89:0/2971384270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f679c1588a0 0x7f679c158d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:18.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.059+0000 7f679affd640 1 -- 172.21.15.89:0/2971384270 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f679c15a570 con 0x7f6788005190 2023-10-30T12:43:18.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.059+0000 7f679affd640 1 --2- 172.21.15.89:0/2971384270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6788005190 0x7f679c1eeae0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f678c007a70 tx=0x7f678c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:18.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.059+0000 7f6798ff9640 1 -- 172.21.15.89:0/2971384270 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678c0019d0 con 0x7f6788005190 2023-10-30T12:43:18.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.059+0000 7f6798ff9640 1 -- 172.21.15.89:0/2971384270 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f678c00ee50 con 0x7f6788005190 2023-10-30T12:43:18.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.059+0000 7f6798ff9640 1 -- 172.21.15.89:0/2971384270 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678c013660 con 0x7f6788005190 2023-10-30T12:43:18.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.059+0000 7f67a1846640 1 -- 172.21.15.89:0/2971384270 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f679c15a850 con 0x7f6788005190 2023-10-30T12:43:18.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.059+0000 7f67a1846640 1 -- 172.21.15.89:0/2971384270 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f679c15ae10 con 0x7f6788005190 2023-10-30T12:43:18.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.059+0000 7f67a1846640 1 -- 172.21.15.89:0/2971384270 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f679c04e170 con 0x7f6788005190 2023-10-30T12:43:18.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.063+0000 7f6798ff9640 1 -- 172.21.15.89:0/2971384270 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f678c0137c0 con 0x7f6788005190 2023-10-30T12:43:18.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.063+0000 7f6798ff9640 1 --2- 172.21.15.89:0/2971384270 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f677c0664a0 0x7f677c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:18.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.063+0000 7f6798ff9640 1 -- 172.21.15.89:0/2971384270 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6320+0+0 (secure 0 0 0) 0x7f678c084970 con 0x7f6788005190 2023-10-30T12:43:18.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.063+0000 7f67927fc640 1 --2- 172.21.15.89:0/2971384270 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f677c0664a0 0x7f677c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:18.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.063+0000 7f67927fc640 1 --2- 172.21.15.89:0/2971384270 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f677c0664a0 0x7f677c068960 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f679c159bb0 tx=0x7f6784008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:18.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.067+0000 7f6798ff9640 1 -- 172.21.15.89:0/2971384270 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f678c051a80 con 0x7f6788005190 2023-10-30T12:43:18.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.251+0000 7f67a1846640 1 -- 172.21.15.89:0/2971384270 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f679c0720a0 con 0x7f6788005190 2023-10-30T12:43:18.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.251+0000 7f6798ff9640 1 -- 172.21.15.89:0/2971384270 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f678c0550a0 con 0x7f6788005190 2023-10-30T12:43:18.249 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:43:18.249 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5728,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:43:10.164217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:43:18.249 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 72 2023-10-30T12:43:18.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.255+0000 7f67a1846640 1 -- 172.21.15.89:0/2971384270 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f677c0664a0 msgr2=0x7f677c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:18.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.255+0000 7f67a1846640 1 --2- 172.21.15.89:0/2971384270 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f677c0664a0 0x7f677c068960 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f679c159bb0 tx=0x7f6784008040 comp rx=0 tx=0).stop 2023-10-30T12:43:18.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.255+0000 7f67a1846640 1 -- 172.21.15.89:0/2971384270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6788005190 msgr2=0x7f679c1eeae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:18.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.255+0000 7f67a1846640 1 --2- 172.21.15.89:0/2971384270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6788005190 0x7f679c1eeae0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f678c007a70 tx=0x7f678c00e530 comp rx=0 tx=0).stop 2023-10-30T12:43:18.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.255+0000 7f67a1846640 1 -- 172.21.15.89:0/2971384270 shutdown_connections 2023-10-30T12:43:18.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.255+0000 7f67a1846640 1 --2- 172.21.15.89:0/2971384270 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f677c0664a0 0x7f677c068960 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:18.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.255+0000 7f67a1846640 1 --2- 172.21.15.89:0/2971384270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f679c1588a0 0x7f679c158d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:18.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.255+0000 7f67a1846640 1 --2- 172.21.15.89:0/2971384270 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f679c14fc50 0x7f679c1ef020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:18.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.255+0000 7f67a1846640 1 --2- 172.21.15.89:0/2971384270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6788005190 0x7f679c1eeae0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:18.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.255+0000 7f67a1846640 1 -- 172.21.15.89:0/2971384270 >> 172.21.15.89:0/2971384270 conn(0x7f679c0c7910 msgr2=0x7f679c0c8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:18.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.255+0000 7f67a1846640 1 -- 172.21.15.89:0/2971384270 shutdown_connections 2023-10-30T12:43:18.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.255+0000 7f67a1846640 1 -- 172.21.15.89:0/2971384270 wait complete. 2023-10-30T12:43:18.269 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=2) 2023-10-30T12:43:18.270 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-30T12:43:18.270 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:43:18.286 INFO:tasks.ceph.mds.f:Stopped 2023-10-30T12:43:18.286 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:43:18.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d92ff8640 1 Processor -- start 2023-10-30T12:43:18.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d92ff8640 1 -- start start 2023-10-30T12:43:18.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d92ff8640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2d8c0cf160 0x7f2d8c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:18.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d92ff8640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8c0cb800 con 0x7f2d8c1538a0 2023-10-30T12:43:18.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d92ff8640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8c0cb330 con 0x7f2d8c15d6f0 2023-10-30T12:43:18.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d92ff8640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c0cad70 con 0x7f2d8c0cf160 2023-10-30T12:43:18.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d90d94640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2d8c1538a0 0x7f2d8c0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57440/0 (socket says 172.21.15.89:57440) 2023-10-30T12:43:18.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d90d94640 1 -- 172.21.15.89:0/3148507025 learned_addr learned my addr 172.21.15.89:0/3148507025 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:43:18.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d91595640 1 --2- 172.21.15.89:0/3148507025 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2d8c0cf160 0x7f2d8c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:18.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d8b7fe640 1 -- 172.21.15.89:0/3148507025 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706586960 0 0) 0x7f2d8c0cb800 con 0x7f2d8c1538a0 2023-10-30T12:43:18.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d8b7fe640 1 -- 172.21.15.89:0/3148507025 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d7c003530 con 0x7f2d8c1538a0 2023-10-30T12:43:18.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d8b7fe640 1 -- 172.21.15.89:0/3148507025 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 961252543 0 0) 0x7f2d7c003530 con 0x7f2d8c1538a0 2023-10-30T12:43:18.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d8b7fe640 1 -- 172.21.15.89:0/3148507025 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2d8c0cf160 msgr2=0x7f2d8c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:18.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d8b7fe640 1 --2- 172.21.15.89:0/3148507025 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2d8c0cf160 0x7f2d8c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:18.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d8b7fe640 1 -- 172.21.15.89:0/3148507025 >> v1:172.21.15.192:6789/0 conn(0x7f2d8c15d6f0 legacy=0x7f2d8c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:18.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d8b7fe640 1 -- 172.21.15.89:0/3148507025 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d8c0cb050 con 0x7f2d8c1538a0 2023-10-30T12:43:18.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d8b7fe640 1 -- 172.21.15.89:0/3148507025 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f2d80001680 con 0x7f2d8c1538a0 2023-10-30T12:43:18.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d8b7fe640 1 -- 172.21.15.89:0/3148507025 >> v1:172.21.15.89:6789/0 conn(0x7f2d8c1538a0 legacy=0x7f2d8c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:18.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d8b7fe640 1 --2- 172.21.15.89:0/3148507025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d7c0039b0 0x7f2d7c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:18.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d8b7fe640 1 --2- 172.21.15.89:0/3148507025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2d7c0047d0 0x7f2d7c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:18.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d8b7fe640 1 --2- 172.21.15.89:0/3148507025 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2d8c0cf160 0x7f2d7c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:18.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d8b7fe640 1 -- 172.21.15.89:0/3148507025 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c0cad70 con 0x7f2d7c0039b0 2023-10-30T12:43:18.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d8b7fe640 1 -- 172.21.15.89:0/3148507025 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c00bad0 con 0x7f2d7c0047d0 2023-10-30T12:43:18.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d8b7fe640 1 -- 172.21.15.89:0/3148507025 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c00bc90 con 0x7f2d8c0cf160 2023-10-30T12:43:18.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d90d94640 1 --2- 172.21.15.89:0/3148507025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d7c0039b0 0x7f2d7c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:18.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d8bfff640 1 --2- 172.21.15.89:0/3148507025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2d7c0047d0 0x7f2d7c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:18.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d91595640 1 --2- 172.21.15.89:0/3148507025 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2d8c0cf160 0x7f2d7c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:18.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d90d94640 1 -- 172.21.15.89:0/3148507025 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2d8c0cf160 msgr2=0x7f2d7c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:18.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d90d94640 1 --2- 172.21.15.89:0/3148507025 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2d8c0cf160 0x7f2d7c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:18.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d90d94640 1 -- 172.21.15.89:0/3148507025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2d7c0047d0 msgr2=0x7f2d7c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:18.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d90d94640 1 --2- 172.21.15.89:0/3148507025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2d7c0047d0 0x7f2d7c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:18.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.455+0000 7f2d90d94640 1 -- 172.21.15.89:0/3148507025 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7c003530 con 0x7f2d7c0039b0 2023-10-30T12:43:18.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.459+0000 7f2d90d94640 1 --2- 172.21.15.89:0/3148507025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d7c0039b0 0x7f2d7c003dd0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f2d8000c210 tx=0x7f2d8001f350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bffc1ddb8246b9c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:18.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.459+0000 7f2d8b7fe640 1 -- 172.21.15.89:0/3148507025 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d80005ae0 con 0x7f2d7c0039b0 2023-10-30T12:43:18.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.459+0000 7f2d8b7fe640 1 -- 172.21.15.89:0/3148507025 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d800ab070 con 0x7f2d7c0039b0 2023-10-30T12:43:18.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.459+0000 7f2d8b7fe640 1 -- 172.21.15.89:0/3148507025 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d80007070 con 0x7f2d7c0039b0 2023-10-30T12:43:18.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.459+0000 7f2d92ff8640 1 -- 172.21.15.89:0/3148507025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d7c0039b0 msgr2=0x7f2d7c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:18.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.459+0000 7f2d92ff8640 1 --2- 172.21.15.89:0/3148507025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d7c0039b0 0x7f2d7c003dd0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f2d8000c210 tx=0x7f2d8001f350 comp rx=0 tx=0).stop 2023-10-30T12:43:18.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.459+0000 7f2d91595640 1 -- 172.21.15.89:0/3148507025 reap_dead start 2023-10-30T12:43:18.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.459+0000 7f2d92ff8640 1 -- 172.21.15.89:0/3148507025 shutdown_connections 2023-10-30T12:43:18.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.459+0000 7f2d92ff8640 1 -- 172.21.15.89:0/3148507025 >> 172.21.15.89:0/3148507025 conn(0x7f2d8c06bfe0 msgr2=0x7f2d8c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:18.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.459+0000 7f2d92ff8640 1 -- 172.21.15.89:0/3148507025 shutdown_connections 2023-10-30T12:43:18.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.459+0000 7f2d92ff8640 1 -- 172.21.15.89:0/3148507025 wait complete. 2023-10-30T12:43:18.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.463+0000 7f2d92ff8640 1 Processor -- start 2023-10-30T12:43:18.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.463+0000 7f2d92ff8640 1 -- start start 2023-10-30T12:43:18.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.463+0000 7f2d92ff8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2d8c0b4a90 0x7f2d8c154d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:18.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.463+0000 7f2d92ff8640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2d8c0b4e70 0x7f2d8c155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:18.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.463+0000 7f2d92ff8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d8c0665c0 0x7f2d8c1560d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:18.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.463+0000 7f2d92ff8640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c154180 con 0x7f2d8c0665c0 2023-10-30T12:43:18.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.463+0000 7f2d92ff8640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c155250 con 0x7f2d8c0b4a90 2023-10-30T12:43:18.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.463+0000 7f2d92ff8640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c1f2ae0 con 0x7f2d8c0b4e70 2023-10-30T12:43:18.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.463+0000 7f2d91595640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d8c0665c0 0x7f2d8c1560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:18.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.463+0000 7f2d91595640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d8c0665c0 0x7f2d8c1560d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54042/0 (socket says 172.21.15.89:54042) 2023-10-30T12:43:18.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.463+0000 7f2d91595640 1 -- 172.21.15.89:0/3847505955 learned_addr learned my addr 172.21.15.89:0/3847505955 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:43:18.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.463+0000 7f2d8bfff640 1 --2- 172.21.15.89:0/3847505955 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2d8c0b4e70 0x7f2d8c155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:18.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.463+0000 7f2d90d94640 1 --2- 172.21.15.89:0/3847505955 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2d8c0b4a90 0x7f2d8c154d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:18.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.463+0000 7f2d91595640 1 -- 172.21.15.89:0/3847505955 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2d8c0b4e70 msgr2=0x7f2d8c155b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:18.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.463+0000 7f2d91595640 1 --2- 172.21.15.89:0/3847505955 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2d8c0b4e70 0x7f2d8c155b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:18.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.463+0000 7f2d91595640 1 -- 172.21.15.89:0/3847505955 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2d8c0b4a90 msgr2=0x7f2d8c154d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:18.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.463+0000 7f2d91595640 1 --2- 172.21.15.89:0/3847505955 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2d8c0b4a90 0x7f2d8c154d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:18.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.463+0000 7f2d91595640 1 -- 172.21.15.89:0/3847505955 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d8c0c6570 con 0x7f2d8c0665c0 2023-10-30T12:43:18.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.463+0000 7f2d91595640 1 --2- 172.21.15.89:0/3847505955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d8c0665c0 0x7f2d8c1560d0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c06cb00 tx=0x7f2d8400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:18.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.463+0000 7f2d897fa640 1 -- 172.21.15.89:0/3847505955 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8400e040 con 0x7f2d8c0665c0 2023-10-30T12:43:18.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.463+0000 7f2d897fa640 1 -- 172.21.15.89:0/3847505955 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d8400f040 con 0x7f2d8c0665c0 2023-10-30T12:43:18.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.463+0000 7f2d92ff8640 1 -- 172.21.15.89:0/3847505955 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d8c0c6850 con 0x7f2d8c0665c0 2023-10-30T12:43:18.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.463+0000 7f2d897fa640 1 -- 172.21.15.89:0/3847505955 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d84015430 con 0x7f2d8c0665c0 2023-10-30T12:43:18.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.463+0000 7f2d92ff8640 1 -- 172.21.15.89:0/3847505955 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d8c0c9820 con 0x7f2d8c0665c0 2023-10-30T12:43:18.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.467+0000 7f2d92ff8640 1 -- 172.21.15.89:0/3847505955 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d8c04e170 con 0x7f2d8c0665c0 2023-10-30T12:43:18.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.467+0000 7f2d897fa640 1 -- 172.21.15.89:0/3847505955 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2d840157a0 con 0x7f2d8c0665c0 2023-10-30T12:43:18.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.467+0000 7f2d897fa640 1 --2- 172.21.15.89:0/3847505955 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2d700664d0 0x7f2d70068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:18.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.467+0000 7f2d897fa640 1 -- 172.21.15.89:0/3847505955 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6320+0+0 (secure 0 0 0) 0x7f2d84084c70 con 0x7f2d8c0665c0 2023-10-30T12:43:18.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.467+0000 7f2d90d94640 1 --2- 172.21.15.89:0/3847505955 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2d700664d0 0x7f2d70068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:18.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.467+0000 7f2d90d94640 1 --2- 172.21.15.89:0/3847505955 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2d700664d0 0x7f2d70068990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2d800af000 tx=0x7f2d800aa000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:18.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.471+0000 7f2d897fa640 1 -- 172.21.15.89:0/3847505955 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2d84051ea0 con 0x7f2d8c0665c0 2023-10-30T12:43:18.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.655+0000 7f2d92ff8640 1 -- 172.21.15.89:0/3847505955 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d8c155840 con 0x7f2d8c0665c0 2023-10-30T12:43:18.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.655+0000 7f2d897fa640 1 -- 172.21.15.89:0/3847505955 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f2d840554c0 con 0x7f2d8c0665c0 2023-10-30T12:43:18.651 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:43:18.652 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5728,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:43:10.164217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:43:18.652 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 72 2023-10-30T12:43:18.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.655+0000 7f2d92ff8640 1 -- 172.21.15.89:0/3847505955 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2d700664d0 msgr2=0x7f2d70068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:18.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.655+0000 7f2d92ff8640 1 --2- 172.21.15.89:0/3847505955 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2d700664d0 0x7f2d70068990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2d800af000 tx=0x7f2d800aa000 comp rx=0 tx=0).stop 2023-10-30T12:43:18.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.655+0000 7f2d92ff8640 1 -- 172.21.15.89:0/3847505955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d8c0665c0 msgr2=0x7f2d8c1560d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:18.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.655+0000 7f2d92ff8640 1 --2- 172.21.15.89:0/3847505955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d8c0665c0 0x7f2d8c1560d0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c06cb00 tx=0x7f2d8400f580 comp rx=0 tx=0).stop 2023-10-30T12:43:18.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.655+0000 7f2d92ff8640 1 -- 172.21.15.89:0/3847505955 shutdown_connections 2023-10-30T12:43:18.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.655+0000 7f2d92ff8640 1 --2- 172.21.15.89:0/3847505955 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2d700664d0 0x7f2d70068990 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:18.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.655+0000 7f2d92ff8640 1 --2- 172.21.15.89:0/3847505955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d8c0665c0 0x7f2d8c1560d0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:18.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.655+0000 7f2d92ff8640 1 --2- 172.21.15.89:0/3847505955 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2d8c0b4e70 0x7f2d8c155b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:18.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.655+0000 7f2d92ff8640 1 --2- 172.21.15.89:0/3847505955 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2d8c0b4a90 0x7f2d8c154d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:18.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.655+0000 7f2d92ff8640 1 -- 172.21.15.89:0/3847505955 >> 172.21.15.89:0/3847505955 conn(0x7f2d8c06bfe0 msgr2=0x7f2d8c0c53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:18.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.655+0000 7f2d92ff8640 1 -- 172.21.15.89:0/3847505955 shutdown_connections 2023-10-30T12:43:18.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:18.659+0000 7f2d92ff8640 1 -- 172.21.15.89:0/3847505955 wait complete. 2023-10-30T12:43:18.673 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:43:20.673 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:43:20.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.815+0000 7f9e82c17640 1 Processor -- start 2023-10-30T12:43:20.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.815+0000 7f9e82c17640 1 -- start start 2023-10-30T12:43:20.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.815+0000 7f9e82c17640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9e7c0c6bd0 0x7f9e7c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:20.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.815+0000 7f9e82c17640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e7c066cf0 con 0x7f9e7c06f7a0 2023-10-30T12:43:20.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.815+0000 7f9e82c17640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e7c0c70d0 con 0x7f9e7c071110 2023-10-30T12:43:20.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.815+0000 7f9e82c17640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e7c0700c0 con 0x7f9e7c0c6bd0 2023-10-30T12:43:20.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.819+0000 7f9e809b3640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9e7c06f7a0 0x7f9e7c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39678/0 (socket says 172.21.15.89:39678) 2023-10-30T12:43:20.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.819+0000 7f9e809b3640 1 -- 172.21.15.89:0/1126207709 learned_addr learned my addr 172.21.15.89:0/1126207709 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:43:20.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.819+0000 7f9e73fff640 1 --2- 172.21.15.89:0/1126207709 >> v2:172.21.15.192:3300/0 conn(0x7f9e7c0c6bd0 0x7f9e7c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:20.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.819+0000 7f9e737fe640 1 -- 172.21.15.89:0/1126207709 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2638879972 0 0) 0x7f9e7c066cf0 con 0x7f9e7c06f7a0 2023-10-30T12:43:20.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.819+0000 7f9e737fe640 1 -- 172.21.15.89:0/1126207709 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e64003530 con 0x7f9e7c06f7a0 2023-10-30T12:43:20.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.819+0000 7f9e737fe640 1 -- 172.21.15.89:0/1126207709 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 231871816 0 0) 0x7f9e64003530 con 0x7f9e7c06f7a0 2023-10-30T12:43:20.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.819+0000 7f9e737fe640 1 -- 172.21.15.89:0/1126207709 >> v2:172.21.15.192:3300/0 conn(0x7f9e7c0c6bd0 msgr2=0x7f9e7c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:20.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.819+0000 7f9e737fe640 1 --2- 172.21.15.89:0/1126207709 >> v2:172.21.15.192:3300/0 conn(0x7f9e7c0c6bd0 0x7f9e7c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:20.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.819+0000 7f9e737fe640 1 -- 172.21.15.89:0/1126207709 >> v1:172.21.15.192:6789/0 conn(0x7f9e7c071110 legacy=0x7f9e7c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:20.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.819+0000 7f9e737fe640 1 -- 172.21.15.89:0/1126207709 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e7c0c7390 con 0x7f9e7c06f7a0 2023-10-30T12:43:20.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.819+0000 7f9e737fe640 1 -- 172.21.15.89:0/1126207709 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f9e78001680 con 0x7f9e7c06f7a0 2023-10-30T12:43:20.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.819+0000 7f9e737fe640 1 -- 172.21.15.89:0/1126207709 >> v1:172.21.15.89:6789/0 conn(0x7f9e7c06f7a0 legacy=0x7f9e7c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:20.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.819+0000 7f9e737fe640 1 --2- 172.21.15.89:0/1126207709 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e64003980 0x7f9e64003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:20.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.819+0000 7f9e737fe640 1 --2- 172.21.15.89:0/1126207709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e640047d0 0x7f9e64004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:20.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.819+0000 7f9e737fe640 1 --2- 172.21.15.89:0/1126207709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e64005190 0x7f9e6400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:20.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.819+0000 7f9e737fe640 1 -- 172.21.15.89:0/1126207709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7c0700c0 con 0x7f9e64005190 2023-10-30T12:43:20.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.819+0000 7f9e737fe640 1 -- 172.21.15.89:0/1126207709 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6400c040 con 0x7f9e640047d0 2023-10-30T12:43:20.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.819+0000 7f9e737fe640 1 -- 172.21.15.89:0/1126207709 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6400c200 con 0x7f9e64003980 2023-10-30T12:43:20.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.819+0000 7f9e811b4640 1 --2- 172.21.15.89:0/1126207709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e64005190 0x7f9e6400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:20.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.819+0000 7f9e73fff640 1 --2- 172.21.15.89:0/1126207709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e640047d0 0x7f9e64004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:20.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.819+0000 7f9e809b3640 1 --2- 172.21.15.89:0/1126207709 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e64003980 0x7f9e64003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:20.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.819+0000 7f9e811b4640 1 -- 172.21.15.89:0/1126207709 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e64003980 msgr2=0x7f9e64003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:20.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.819+0000 7f9e811b4640 1 --2- 172.21.15.89:0/1126207709 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e64003980 0x7f9e64003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:20.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.819+0000 7f9e811b4640 1 -- 172.21.15.89:0/1126207709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e640047d0 msgr2=0x7f9e64004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:20.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.819+0000 7f9e811b4640 1 --2- 172.21.15.89:0/1126207709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e640047d0 0x7f9e64004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:20.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.819+0000 7f9e811b4640 1 -- 172.21.15.89:0/1126207709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e64003530 con 0x7f9e64005190 2023-10-30T12:43:20.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.823+0000 7f9e811b4640 1 -- 172.21.15.89:0/1126207709 reap_dead start 2023-10-30T12:43:20.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.823+0000 7f9e811b4640 1 --2- 172.21.15.89:0/1126207709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e64005190 0x7f9e6400b990 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f9e74001e70 tx=0x7f9e7401c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=831a64f3e74c1ae8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:20.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.823+0000 7f9e737fe640 1 -- 172.21.15.89:0/1126207709 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e74003b80 con 0x7f9e64005190 2023-10-30T12:43:20.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.823+0000 7f9e737fe640 1 -- 172.21.15.89:0/1126207709 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e74003d20 con 0x7f9e64005190 2023-10-30T12:43:20.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.823+0000 7f9e737fe640 1 -- 172.21.15.89:0/1126207709 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e74005260 con 0x7f9e64005190 2023-10-30T12:43:20.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.823+0000 7f9e82c17640 1 -- 172.21.15.89:0/1126207709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e64005190 msgr2=0x7f9e6400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:20.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.823+0000 7f9e82c17640 1 --2- 172.21.15.89:0/1126207709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e64005190 0x7f9e6400b990 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f9e74001e70 tx=0x7f9e7401c760 comp rx=0 tx=0).stop 2023-10-30T12:43:20.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.823+0000 7f9e82c17640 1 -- 172.21.15.89:0/1126207709 shutdown_connections 2023-10-30T12:43:20.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.823+0000 7f9e82c17640 1 --2- 172.21.15.89:0/1126207709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e64005190 0x7f9e6400b990 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:20.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.823+0000 7f9e82c17640 1 -- 172.21.15.89:0/1126207709 >> 172.21.15.89:0/1126207709 conn(0x7f9e7c0c7910 msgr2=0x7f9e7c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:20.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.823+0000 7f9e82c17640 1 -- 172.21.15.89:0/1126207709 shutdown_connections 2023-10-30T12:43:20.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.823+0000 7f9e82c17640 1 -- 172.21.15.89:0/1126207709 wait complete. 2023-10-30T12:43:20.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.823+0000 7f9e82c17640 1 Processor -- start 2023-10-30T12:43:20.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.823+0000 7f9e82c17640 1 -- start start 2023-10-30T12:43:20.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.827+0000 7f9e82c17640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e64005190 0x7f9e7c0afed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:20.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.827+0000 7f9e82c17640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e7c0ad8d0 0x7f9e7c0c17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:20.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.827+0000 7f9e82c17640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e7c0bfec0 0x7f9e7c143660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:20.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.827+0000 7f9e82c17640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7c1496b0 con 0x7f9e7c0ad8d0 2023-10-30T12:43:20.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.827+0000 7f9e82c17640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7c0aea50 con 0x7f9e64005190 2023-10-30T12:43:20.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.827+0000 7f9e82c17640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7c0aed30 con 0x7f9e7c0bfec0 2023-10-30T12:43:20.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.827+0000 7f9e73fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e7c0ad8d0 0x7f9e7c0c17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:20.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.827+0000 7f9e73fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e7c0ad8d0 0x7f9e7c0c17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49332/0 (socket says 172.21.15.89:49332) 2023-10-30T12:43:20.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.827+0000 7f9e73fff640 1 -- 172.21.15.89:0/2708883128 learned_addr learned my addr 172.21.15.89:0/2708883128 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:43:20.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.827+0000 7f9e809b3640 1 --2- 172.21.15.89:0/2708883128 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e64005190 0x7f9e7c0afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:20.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.827+0000 7f9e811b4640 1 --2- 172.21.15.89:0/2708883128 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e7c0bfec0 0x7f9e7c143660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:20.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.827+0000 7f9e73fff640 1 -- 172.21.15.89:0/2708883128 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e7c0bfec0 msgr2=0x7f9e7c143660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:20.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.827+0000 7f9e73fff640 1 --2- 172.21.15.89:0/2708883128 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e7c0bfec0 0x7f9e7c143660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:20.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.827+0000 7f9e73fff640 1 -- 172.21.15.89:0/2708883128 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e64005190 msgr2=0x7f9e7c0afed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:20.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.827+0000 7f9e73fff640 1 --2- 172.21.15.89:0/2708883128 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e64005190 0x7f9e7c0afed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:20.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.827+0000 7f9e73fff640 1 -- 172.21.15.89:0/2708883128 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e7c0af010 con 0x7f9e7c0ad8d0 2023-10-30T12:43:20.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.827+0000 7f9e73fff640 1 --2- 172.21.15.89:0/2708883128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e7c0ad8d0 0x7f9e7c0c17d0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c0118b0 tx=0x7f9e6c011d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:20.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.827+0000 7f9e717fa640 1 -- 172.21.15.89:0/2708883128 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c00fa30 con 0x7f9e7c0ad8d0 2023-10-30T12:43:20.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.827+0000 7f9e717fa640 1 -- 172.21.15.89:0/2708883128 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6c00fb90 con 0x7f9e7c0ad8d0 2023-10-30T12:43:20.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.827+0000 7f9e717fa640 1 -- 172.21.15.89:0/2708883128 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c014440 con 0x7f9e7c0ad8d0 2023-10-30T12:43:20.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.827+0000 7f9e82c17640 1 -- 172.21.15.89:0/2708883128 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e7c0af2f0 con 0x7f9e7c0ad8d0 2023-10-30T12:43:20.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.827+0000 7f9e82c17640 1 -- 172.21.15.89:0/2708883128 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e7c0ca1d0 con 0x7f9e7c0ad8d0 2023-10-30T12:43:20.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.827+0000 7f9e82c17640 1 -- 172.21.15.89:0/2708883128 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e7c04e170 con 0x7f9e7c0ad8d0 2023-10-30T12:43:20.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.827+0000 7f9e717fa640 1 -- 172.21.15.89:0/2708883128 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9e6c0145a0 con 0x7f9e7c0ad8d0 2023-10-30T12:43:20.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.831+0000 7f9e717fa640 1 --2- 172.21.15.89:0/2708883128 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9e680664a0 0x7f9e68068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:20.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.831+0000 7f9e717fa640 1 -- 172.21.15.89:0/2708883128 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6320+0+0 (secure 0 0 0) 0x7f9e6c084960 con 0x7f9e7c0ad8d0 2023-10-30T12:43:20.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.831+0000 7f9e809b3640 1 --2- 172.21.15.89:0/2708883128 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9e680664a0 0x7f9e68068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:20.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.831+0000 7f9e809b3640 1 --2- 172.21.15.89:0/2708883128 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9e680664a0 0x7f9e68068960 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f9e78001060 tx=0x7f9e78009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:20.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:20.835+0000 7f9e717fa640 1 -- 172.21.15.89:0/2708883128 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9e6c051b00 con 0x7f9e7c0ad8d0 2023-10-30T12:43:21.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:21.015+0000 7f9e82c17640 1 -- 172.21.15.89:0/2708883128 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e7c0721c0 con 0x7f9e7c0ad8d0 2023-10-30T12:43:21.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:21.015+0000 7f9e717fa640 1 -- 172.21.15.89:0/2708883128 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f9e6c022090 con 0x7f9e7c0ad8d0 2023-10-30T12:43:21.010 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:43:21.011 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5728,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:43:10.164217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:43:21.011 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 72 2023-10-30T12:43:21.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:21.015+0000 7f9e82c17640 1 -- 172.21.15.89:0/2708883128 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9e680664a0 msgr2=0x7f9e68068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:21.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:21.015+0000 7f9e82c17640 1 --2- 172.21.15.89:0/2708883128 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9e680664a0 0x7f9e68068960 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f9e78001060 tx=0x7f9e78009040 comp rx=0 tx=0).stop 2023-10-30T12:43:21.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:21.015+0000 7f9e82c17640 1 -- 172.21.15.89:0/2708883128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e7c0ad8d0 msgr2=0x7f9e7c0c17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:21.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:21.015+0000 7f9e82c17640 1 --2- 172.21.15.89:0/2708883128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e7c0ad8d0 0x7f9e7c0c17d0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c0118b0 tx=0x7f9e6c011d70 comp rx=0 tx=0).stop 2023-10-30T12:43:21.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:21.015+0000 7f9e82c17640 1 -- 172.21.15.89:0/2708883128 shutdown_connections 2023-10-30T12:43:21.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:21.015+0000 7f9e82c17640 1 --2- 172.21.15.89:0/2708883128 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9e680664a0 0x7f9e68068960 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:21.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:21.015+0000 7f9e82c17640 1 --2- 172.21.15.89:0/2708883128 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e7c0bfec0 0x7f9e7c143660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:21.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:21.015+0000 7f9e82c17640 1 --2- 172.21.15.89:0/2708883128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e7c0ad8d0 0x7f9e7c0c17d0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:21.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:21.015+0000 7f9e82c17640 1 --2- 172.21.15.89:0/2708883128 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e64005190 0x7f9e7c0afed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:21.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:21.015+0000 7f9e82c17640 1 -- 172.21.15.89:0/2708883128 >> 172.21.15.89:0/2708883128 conn(0x7f9e7c0c7910 msgr2=0x7f9e7c0ca5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:21.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:21.015+0000 7f9e82c17640 1 -- 172.21.15.89:0/2708883128 shutdown_connections 2023-10-30T12:43:21.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:21.015+0000 7f9e82c17640 1 -- 172.21.15.89:0/2708883128 wait complete. 2023-10-30T12:43:21.031 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:43:23.032 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:43:23.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.183+0000 7f0ca9a2f640 1 Processor -- start 2023-10-30T12:43:23.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.183+0000 7f0ca9a2f640 1 -- start start 2023-10-30T12:43:23.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.183+0000 7f0ca9a2f640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0ca41538f0 0x7f0ca4159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:23.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.183+0000 7f0ca9a2f640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0ca415cd50 0x7f0ca40c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:23.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.183+0000 7f0ca9a2f640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca415d300 con 0x7f0ca40cefa0 2023-10-30T12:43:23.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.183+0000 7f0ca9a2f640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ca415d5e0 con 0x7f0ca415cd50 2023-10-30T12:43:23.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.183+0000 7f0ca9a2f640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ca415d8c0 con 0x7f0ca41538f0 2023-10-30T12:43:23.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.183+0000 7f0c9bfff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0ca415cd50 0x7f0ca40c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:23.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.183+0000 7f0c9bfff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0ca415cd50 0x7f0ca40c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49348/0 (socket says 172.21.15.89:49348) 2023-10-30T12:43:23.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.183+0000 7f0c9bfff640 1 -- 172.21.15.89:0/2294311939 learned_addr learned my addr 172.21.15.89:0/2294311939 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:43:23.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.187+0000 7f0ca8a2d640 1 --2- 172.21.15.89:0/2294311939 >> v2:172.21.15.192:3300/0 conn(0x7f0ca41538f0 0x7f0ca4159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:23.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.187+0000 7f0c9bfff640 1 -- 172.21.15.89:0/2294311939 >> v2:172.21.15.192:3300/0 conn(0x7f0ca41538f0 msgr2=0x7f0ca4159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:23.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.187+0000 7f0c9bfff640 1 --2- 172.21.15.89:0/2294311939 >> v2:172.21.15.192:3300/0 conn(0x7f0ca41538f0 0x7f0ca4159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:23.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.187+0000 7f0c9bfff640 1 -- 172.21.15.89:0/2294311939 >> v1:172.21.15.89:6789/0 conn(0x7f0ca40cefa0 legacy=0x7f0ca40cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:23.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.187+0000 7f0c9bfff640 1 -- 172.21.15.89:0/2294311939 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca415dba0 con 0x7f0ca415cd50 2023-10-30T12:43:23.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.187+0000 7f0c9b7fe640 1 -- 172.21.15.89:0/2294311939 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 526391349 0 0) 0x7f0ca415d300 con 0x7f0ca40cefa0 2023-10-30T12:43:23.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.187+0000 7f0c9bfff640 1 --2- 172.21.15.89:0/2294311939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ca415cd50 0x7f0ca40c9f90 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f0ca40c9cb0 tx=0x7f0c9401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e408de7c7471dbf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:23.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.187+0000 7f0c9b7fe640 1 -- 172.21.15.89:0/2294311939 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c940a89f0 con 0x7f0ca415cd50 2023-10-30T12:43:23.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.187+0000 7f0c9b7fe640 1 -- 172.21.15.89:0/2294311939 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c940a8b50 con 0x7f0ca415cd50 2023-10-30T12:43:23.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.187+0000 7f0c9b7fe640 1 -- 172.21.15.89:0/2294311939 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c940b0440 con 0x7f0ca415cd50 2023-10-30T12:43:23.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.187+0000 7f0ca9a2f640 1 -- 172.21.15.89:0/2294311939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ca415cd50 msgr2=0x7f0ca40c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:23.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.187+0000 7f0ca9a2f640 1 --2- 172.21.15.89:0/2294311939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ca415cd50 0x7f0ca40c9f90 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f0ca40c9cb0 tx=0x7f0c9401ecf0 comp rx=0 tx=0).stop 2023-10-30T12:43:23.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.191+0000 7f0ca9a2f640 1 -- 172.21.15.89:0/2294311939 shutdown_connections 2023-10-30T12:43:23.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.191+0000 7f0ca9a2f640 1 --2- 172.21.15.89:0/2294311939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ca415cd50 0x7f0ca40c9f90 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:23.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.191+0000 7f0ca9a2f640 1 --2- 172.21.15.89:0/2294311939 >> v2:172.21.15.192:3300/0 conn(0x7f0ca41538f0 0x7f0ca4159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:23.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.191+0000 7f0ca9a2f640 1 -- 172.21.15.89:0/2294311939 >> 172.21.15.89:0/2294311939 conn(0x7f0ca406a640 msgr2=0x7f0ca4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:23.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.191+0000 7f0ca9a2f640 1 -- 172.21.15.89:0/2294311939 shutdown_connections 2023-10-30T12:43:23.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.191+0000 7f0ca9a2f640 1 -- 172.21.15.89:0/2294311939 wait complete. 2023-10-30T12:43:23.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.191+0000 7f0ca9a2f640 1 Processor -- start 2023-10-30T12:43:23.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.191+0000 7f0ca9a2f640 1 -- start start 2023-10-30T12:43:23.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.191+0000 7f0ca9a2f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ca40cefa0 0x7f0ca415b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:23.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.191+0000 7f0ca9a2f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ca41538f0 0x7f0ca40b4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:23.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.191+0000 7f0ca9a2f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ca415cd50 0x7f0ca4144810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:23.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.191+0000 7f0ca9a2f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca415bdd0 con 0x7f0ca415cd50 2023-10-30T12:43:23.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.191+0000 7f0ca9a2f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca41f2cd0 con 0x7f0ca41538f0 2023-10-30T12:43:23.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.191+0000 7f0ca9a2f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca41f20c0 con 0x7f0ca40cefa0 2023-10-30T12:43:23.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.191+0000 7f0ca922e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ca415cd50 0x7f0ca4144810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:23.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.191+0000 7f0ca922e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ca415cd50 0x7f0ca4144810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49362/0 (socket says 172.21.15.89:49362) 2023-10-30T12:43:23.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.191+0000 7f0ca922e640 1 -- 172.21.15.89:0/2899907129 learned_addr learned my addr 172.21.15.89:0/2899907129 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:43:23.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.191+0000 7f0c9bfff640 1 --2- 172.21.15.89:0/2899907129 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ca41538f0 0x7f0ca40b4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:23.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.191+0000 7f0ca8a2d640 1 --2- 172.21.15.89:0/2899907129 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ca40cefa0 0x7f0ca415b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:23.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.191+0000 7f0ca922e640 1 -- 172.21.15.89:0/2899907129 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ca40cefa0 msgr2=0x7f0ca415b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:23.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.191+0000 7f0ca922e640 1 --2- 172.21.15.89:0/2899907129 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ca40cefa0 0x7f0ca415b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:23.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.191+0000 7f0ca922e640 1 -- 172.21.15.89:0/2899907129 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ca41538f0 msgr2=0x7f0ca40b4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:23.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.191+0000 7f0ca922e640 1 --2- 172.21.15.89:0/2899907129 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ca41538f0 0x7f0ca40b4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:23.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.191+0000 7f0ca922e640 1 -- 172.21.15.89:0/2899907129 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c940094d0 con 0x7f0ca415cd50 2023-10-30T12:43:23.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.191+0000 7f0ca922e640 1 --2- 172.21.15.89:0/2899907129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ca415cd50 0x7f0ca4144810 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c00c890 tx=0x7f0c9c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:23.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.195+0000 7f0c997fa640 1 -- 172.21.15.89:0/2899907129 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c003db0 con 0x7f0ca415cd50 2023-10-30T12:43:23.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.195+0000 7f0ca9a2f640 1 -- 172.21.15.89:0/2899907129 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ca41f23a0 con 0x7f0ca415cd50 2023-10-30T12:43:23.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.195+0000 7f0c997fa640 1 -- 172.21.15.89:0/2899907129 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c9c005070 con 0x7f0ca415cd50 2023-10-30T12:43:23.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.195+0000 7f0c997fa640 1 -- 172.21.15.89:0/2899907129 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c005420 con 0x7f0ca415cd50 2023-10-30T12:43:23.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.195+0000 7f0ca9a2f640 1 -- 172.21.15.89:0/2899907129 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ca41f2960 con 0x7f0ca415cd50 2023-10-30T12:43:23.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.195+0000 7f0ca9a2f640 1 -- 172.21.15.89:0/2899907129 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ca41f2e10 con 0x7f0ca415cd50 2023-10-30T12:43:23.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.195+0000 7f0c997fa640 1 -- 172.21.15.89:0/2899907129 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0c9c005580 con 0x7f0ca415cd50 2023-10-30T12:43:23.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.195+0000 7f0c997fa640 1 --2- 172.21.15.89:0/2899907129 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0c900664d0 0x7f0c90068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:23.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.195+0000 7f0c997fa640 1 -- 172.21.15.89:0/2899907129 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6320+0+0 (secure 0 0 0) 0x7f0c9c0839f0 con 0x7f0ca415cd50 2023-10-30T12:43:23.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.195+0000 7f0ca8a2d640 1 --2- 172.21.15.89:0/2899907129 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0c900664d0 0x7f0c90068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:23.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.199+0000 7f0ca8a2d640 1 --2- 172.21.15.89:0/2899907129 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0c900664d0 0x7f0c90068990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0009af0 tx=0x7f0ca0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:23.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.199+0000 7f0c997fa640 1 -- 172.21.15.89:0/2899907129 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0ca41f2e10 con 0x7f0ca415cd50 2023-10-30T12:43:23.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.379+0000 7f0ca9a2f640 1 -- 172.21.15.89:0/2899907129 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ca41f2e10 con 0x7f0ca415cd50 2023-10-30T12:43:23.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.383+0000 7f0c997fa640 1 -- 172.21.15.89:0/2899907129 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f0ca41f2e10 con 0x7f0ca415cd50 2023-10-30T12:43:23.378 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:43:23.378 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5728,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:43:10.164217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:43:23.378 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 72 2023-10-30T12:43:23.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.383+0000 7f0ca9a2f640 1 -- 172.21.15.89:0/2899907129 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0c900664d0 msgr2=0x7f0c90068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:23.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.383+0000 7f0ca9a2f640 1 --2- 172.21.15.89:0/2899907129 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0c900664d0 0x7f0c90068990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0009af0 tx=0x7f0ca0008040 comp rx=0 tx=0).stop 2023-10-30T12:43:23.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.383+0000 7f0ca9a2f640 1 -- 172.21.15.89:0/2899907129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ca415cd50 msgr2=0x7f0ca4144810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:23.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.383+0000 7f0ca9a2f640 1 --2- 172.21.15.89:0/2899907129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ca415cd50 0x7f0ca4144810 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c00c890 tx=0x7f0c9c00cd50 comp rx=0 tx=0).stop 2023-10-30T12:43:23.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.383+0000 7f0ca9a2f640 1 -- 172.21.15.89:0/2899907129 shutdown_connections 2023-10-30T12:43:23.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.383+0000 7f0ca9a2f640 1 --2- 172.21.15.89:0/2899907129 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0c900664d0 0x7f0c90068990 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:23.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.383+0000 7f0ca9a2f640 1 --2- 172.21.15.89:0/2899907129 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ca41538f0 0x7f0ca40b4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:23.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.383+0000 7f0ca9a2f640 1 --2- 172.21.15.89:0/2899907129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ca415cd50 0x7f0ca4144810 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:23.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.383+0000 7f0ca9a2f640 1 --2- 172.21.15.89:0/2899907129 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ca40cefa0 0x7f0ca415b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:23.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.383+0000 7f0ca9a2f640 1 -- 172.21.15.89:0/2899907129 >> 172.21.15.89:0/2899907129 conn(0x7f0ca406a640 msgr2=0x7f0ca40c9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:23.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.383+0000 7f0ca9a2f640 1 -- 172.21.15.89:0/2899907129 shutdown_connections 2023-10-30T12:43:23.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:23.383+0000 7f0ca9a2f640 1 -- 172.21.15.89:0/2899907129 wait complete. 2023-10-30T12:43:23.398 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:43:25.400 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:43:25.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.551+0000 7fd3edfee640 1 Processor -- start 2023-10-30T12:43:25.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.551+0000 7fd3edfee640 1 -- start start 2023-10-30T12:43:25.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.551+0000 7fd3edfee640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3e8069e90 0x7fd3e815a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:25.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.551+0000 7fd3edfee640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3e80c6c10 con 0x7fd3e806cb80 2023-10-30T12:43:25.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.551+0000 7fd3edfee640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3e80c6ef0 con 0x7fd3e806a270 2023-10-30T12:43:25.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.551+0000 7fd3edfee640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e80c8570 con 0x7fd3e8069e90 2023-10-30T12:43:25.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.551+0000 7fd3e7fff640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd3e806cb80 0x7fd3e806d010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39700/0 (socket says 172.21.15.89:39700) 2023-10-30T12:43:25.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.551+0000 7fd3e7fff640 1 -- 172.21.15.89:0/3838027261 learned_addr learned my addr 172.21.15.89:0/3838027261 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:43:25.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.551+0000 7fd3e77fe640 1 --2- 172.21.15.89:0/3838027261 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3e8069e90 0x7fd3e815a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:25.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.551+0000 7fd3e67fc640 1 -- 172.21.15.89:0/3838027261 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1449432001 0 0) 0x7fd3e80c6c10 con 0x7fd3e806cb80 2023-10-30T12:43:25.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.551+0000 7fd3e67fc640 1 -- 172.21.15.89:0/3838027261 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3d0003530 con 0x7fd3e806cb80 2023-10-30T12:43:25.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.555+0000 7fd3e67fc640 1 -- 172.21.15.89:0/3838027261 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3800042923 0 0) 0x7fd3d0003530 con 0x7fd3e806cb80 2023-10-30T12:43:25.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.555+0000 7fd3e67fc640 1 -- 172.21.15.89:0/3838027261 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3e8069e90 msgr2=0x7fd3e815a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:25.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.555+0000 7fd3e67fc640 1 --2- 172.21.15.89:0/3838027261 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3e8069e90 0x7fd3e815a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:25.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.555+0000 7fd3e67fc640 1 -- 172.21.15.89:0/3838027261 >> v1:172.21.15.192:6789/0 conn(0x7fd3e806a270 legacy=0x7fd3e80c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:25.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.555+0000 7fd3e67fc640 1 -- 172.21.15.89:0/3838027261 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e80c71d0 con 0x7fd3e806cb80 2023-10-30T12:43:25.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.555+0000 7fd3e67fc640 1 -- 172.21.15.89:0/3838027261 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7fd3e0001a00 con 0x7fd3e806cb80 2023-10-30T12:43:25.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.555+0000 7fd3e67fc640 1 -- 172.21.15.89:0/3838027261 >> v1:172.21.15.89:6789/0 conn(0x7fd3e806cb80 legacy=0x7fd3e806d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:25.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.555+0000 7fd3e67fc640 1 --2- 172.21.15.89:0/3838027261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3d00039b0 0x7fd3d0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:25.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.555+0000 7fd3e67fc640 1 --2- 172.21.15.89:0/3838027261 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3e8069e90 0x7fd3d00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:25.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.555+0000 7fd3e67fc640 1 --2- 172.21.15.89:0/3838027261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3d0004d10 0x7fd3d0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:25.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.555+0000 7fd3e67fc640 1 -- 172.21.15.89:0/3838027261 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e80c8570 con 0x7fd3d0004d10 2023-10-30T12:43:25.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.555+0000 7fd3e67fc640 1 -- 172.21.15.89:0/3838027261 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d000b990 con 0x7fd3d00039b0 2023-10-30T12:43:25.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.555+0000 7fd3e67fc640 1 -- 172.21.15.89:0/3838027261 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d000bb50 con 0x7fd3e8069e90 2023-10-30T12:43:25.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.555+0000 7fd3e7fff640 1 --2- 172.21.15.89:0/3838027261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3d0004d10 0x7fd3d0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:25.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.555+0000 7fd3e77fe640 1 --2- 172.21.15.89:0/3838027261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3d00039b0 0x7fd3d0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:25.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.555+0000 7fd3e6ffd640 1 --2- 172.21.15.89:0/3838027261 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3e8069e90 0x7fd3d00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:25.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.555+0000 7fd3e7fff640 1 -- 172.21.15.89:0/3838027261 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3e8069e90 msgr2=0x7fd3d00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:25.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.555+0000 7fd3e7fff640 1 --2- 172.21.15.89:0/3838027261 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3e8069e90 0x7fd3d00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:25.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.555+0000 7fd3e7fff640 1 -- 172.21.15.89:0/3838027261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3d00039b0 msgr2=0x7fd3d0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:25.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.555+0000 7fd3e7fff640 1 --2- 172.21.15.89:0/3838027261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3d00039b0 0x7fd3d0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:25.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.555+0000 7fd3e7fff640 1 -- 172.21.15.89:0/3838027261 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d0003530 con 0x7fd3d0004d10 2023-10-30T12:43:25.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.555+0000 7fd3e7fff640 1 --2- 172.21.15.89:0/3838027261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3d0004d10 0x7fd3d0005190 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fd3e000a830 tx=0x7fd3e001faa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d4e776e9fcdb965 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:25.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.559+0000 7fd3e67fc640 1 -- 172.21.15.89:0/3838027261 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e00aa6d0 con 0x7fd3d0004d10 2023-10-30T12:43:25.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.559+0000 7fd3e67fc640 1 -- 172.21.15.89:0/3838027261 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e00aa850 con 0x7fd3d0004d10 2023-10-30T12:43:25.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.559+0000 7fd3e67fc640 1 -- 172.21.15.89:0/3838027261 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e00aac00 con 0x7fd3d0004d10 2023-10-30T12:43:25.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.559+0000 7fd3edfee640 1 -- 172.21.15.89:0/3838027261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3d0004d10 msgr2=0x7fd3d0005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:25.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.559+0000 7fd3edfee640 1 --2- 172.21.15.89:0/3838027261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3d0004d10 0x7fd3d0005190 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fd3e000a830 tx=0x7fd3e001faa0 comp rx=0 tx=0).stop 2023-10-30T12:43:25.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.559+0000 7fd3e7fff640 1 -- 172.21.15.89:0/3838027261 reap_dead start 2023-10-30T12:43:25.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.559+0000 7fd3edfee640 1 -- 172.21.15.89:0/3838027261 shutdown_connections 2023-10-30T12:43:25.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.559+0000 7fd3edfee640 1 -- 172.21.15.89:0/3838027261 >> 172.21.15.89:0/3838027261 conn(0x7fd3e806dfd0 msgr2=0x7fd3e806e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:25.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.559+0000 7fd3edfee640 1 -- 172.21.15.89:0/3838027261 shutdown_connections 2023-10-30T12:43:25.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.559+0000 7fd3edfee640 1 -- 172.21.15.89:0/3838027261 wait complete. 2023-10-30T12:43:25.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.559+0000 7fd3edfee640 1 Processor -- start 2023-10-30T12:43:25.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.559+0000 7fd3edfee640 1 -- start start 2023-10-30T12:43:25.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.559+0000 7fd3edfee640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3d0004d10 0x7fd3e81ec970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:25.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.559+0000 7fd3edfee640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3e81eceb0 0x7fd3e814f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:25.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.559+0000 7fd3edfee640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3e814fc30 0x7fd3e81f21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:25.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.559+0000 7fd3edfee640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e81f1d80 con 0x7fd3d0004d10 2023-10-30T12:43:25.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.559+0000 7fd3edfee640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e81f18f0 con 0x7fd3e81eceb0 2023-10-30T12:43:25.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.559+0000 7fd3edfee640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e81f0a00 con 0x7fd3e814fc30 2023-10-30T12:43:25.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.559+0000 7fd3e77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3d0004d10 0x7fd3e81ec970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:25.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.559+0000 7fd3e77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3d0004d10 0x7fd3e81ec970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49372/0 (socket says 172.21.15.89:49372) 2023-10-30T12:43:25.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.559+0000 7fd3e77fe640 1 -- 172.21.15.89:0/3880816601 learned_addr learned my addr 172.21.15.89:0/3880816601 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:43:25.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.563+0000 7fd3e7fff640 1 --2- 172.21.15.89:0/3880816601 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3e814fc30 0x7fd3e81f21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:25.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.563+0000 7fd3e6ffd640 1 --2- 172.21.15.89:0/3880816601 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3e81eceb0 0x7fd3e814f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:25.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.563+0000 7fd3e77fe640 1 -- 172.21.15.89:0/3880816601 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3e814fc30 msgr2=0x7fd3e81f21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:25.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.563+0000 7fd3e77fe640 1 --2- 172.21.15.89:0/3880816601 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3e814fc30 0x7fd3e81f21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:25.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.563+0000 7fd3e77fe640 1 -- 172.21.15.89:0/3880816601 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3e81eceb0 msgr2=0x7fd3e814f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:25.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.563+0000 7fd3e77fe640 1 --2- 172.21.15.89:0/3880816601 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3e81eceb0 0x7fd3e814f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:25.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.563+0000 7fd3e77fe640 1 -- 172.21.15.89:0/3880816601 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e81f0ce0 con 0x7fd3d0004d10 2023-10-30T12:43:25.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.563+0000 7fd3e6ffd640 1 --2- 172.21.15.89:0/3880816601 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3e81eceb0 0x7fd3e814f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:43:25.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.563+0000 7fd3e77fe640 1 --2- 172.21.15.89:0/3880816601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3d0004d10 0x7fd3e81ec970 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8011a00 tx=0x7fd3d8011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:25.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.563+0000 7fd3cffff640 1 -- 172.21.15.89:0/3880816601 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d800faf0 con 0x7fd3d0004d10 2023-10-30T12:43:25.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.563+0000 7fd3cffff640 1 -- 172.21.15.89:0/3880816601 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d800fc50 con 0x7fd3d0004d10 2023-10-30T12:43:25.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.563+0000 7fd3edfee640 1 -- 172.21.15.89:0/3880816601 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3e81f0fc0 con 0x7fd3d0004d10 2023-10-30T12:43:25.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.563+0000 7fd3cffff640 1 -- 172.21.15.89:0/3880816601 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d8014440 con 0x7fd3d0004d10 2023-10-30T12:43:25.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.563+0000 7fd3edfee640 1 -- 172.21.15.89:0/3880816601 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3e81f1580 con 0x7fd3d0004d10 2023-10-30T12:43:25.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.563+0000 7fd3edfee640 1 -- 172.21.15.89:0/3880816601 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3e804e170 con 0x7fd3d0004d10 2023-10-30T12:43:25.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.563+0000 7fd3cffff640 1 -- 172.21.15.89:0/3880816601 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd3d80145a0 con 0x7fd3d0004d10 2023-10-30T12:43:25.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.563+0000 7fd3cffff640 1 --2- 172.21.15.89:0/3880816601 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd3d40664d0 0x7fd3d4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:25.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.563+0000 7fd3cffff640 1 -- 172.21.15.89:0/3880816601 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6320+0+0 (secure 0 0 0) 0x7fd3d80849d0 con 0x7fd3d0004d10 2023-10-30T12:43:25.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.567+0000 7fd3e6ffd640 1 --2- 172.21.15.89:0/3880816601 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd3d40664d0 0x7fd3d4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:25.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.567+0000 7fd3e6ffd640 1 --2- 172.21.15.89:0/3880816601 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd3d40664d0 0x7fd3d4068990 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd3e806b500 tx=0x7fd3dc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:25.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.567+0000 7fd3cffff640 1 -- 172.21.15.89:0/3880816601 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd3d8051c50 con 0x7fd3d0004d10 2023-10-30T12:43:25.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.747+0000 7fd3edfee640 1 -- 172.21.15.89:0/3880816601 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3e80c9e40 con 0x7fd3d0004d10 2023-10-30T12:43:25.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.747+0000 7fd3cffff640 1 -- 172.21.15.89:0/3880816601 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+6532 (secure 0 0 0) 0x7fd3d8022070 con 0x7fd3d0004d10 2023-10-30T12:43:25.745 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:43:25.745 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5728,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:43:10.164217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:43:25.745 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 72 2023-10-30T12:43:25.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.751+0000 7fd3edfee640 1 -- 172.21.15.89:0/3880816601 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd3d40664d0 msgr2=0x7fd3d4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:25.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.751+0000 7fd3edfee640 1 --2- 172.21.15.89:0/3880816601 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd3d40664d0 0x7fd3d4068990 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd3e806b500 tx=0x7fd3dc009040 comp rx=0 tx=0).stop 2023-10-30T12:43:25.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.751+0000 7fd3edfee640 1 -- 172.21.15.89:0/3880816601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3d0004d10 msgr2=0x7fd3e81ec970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:25.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.751+0000 7fd3edfee640 1 --2- 172.21.15.89:0/3880816601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3d0004d10 0x7fd3e81ec970 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8011a00 tx=0x7fd3d8011ec0 comp rx=0 tx=0).stop 2023-10-30T12:43:25.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.751+0000 7fd3edfee640 1 -- 172.21.15.89:0/3880816601 shutdown_connections 2023-10-30T12:43:25.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.751+0000 7fd3edfee640 1 --2- 172.21.15.89:0/3880816601 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd3d40664d0 0x7fd3d4068990 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:25.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.751+0000 7fd3edfee640 1 --2- 172.21.15.89:0/3880816601 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3e81eceb0 0x7fd3e814f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:25.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.751+0000 7fd3edfee640 1 --2- 172.21.15.89:0/3880816601 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd3e814fc30 0x7fd3e81f21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:25.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.751+0000 7fd3edfee640 1 --2- 172.21.15.89:0/3880816601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3d0004d10 0x7fd3e81ec970 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:25.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.751+0000 7fd3edfee640 1 -- 172.21.15.89:0/3880816601 >> 172.21.15.89:0/3880816601 conn(0x7fd3e806dfd0 msgr2=0x7fd3e806ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:25.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.751+0000 7fd3edfee640 1 -- 172.21.15.89:0/3880816601 shutdown_connections 2023-10-30T12:43:25.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:25.751+0000 7fd3edfee640 1 -- 172.21.15.89:0/3880816601 wait complete. 2023-10-30T12:43:25.765 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:43:27.767 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:43:27.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.919+0000 7f284413e640 1 Processor -- start 2023-10-30T12:43:27.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.919+0000 7f284413e640 1 -- start start 2023-10-30T12:43:27.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.919+0000 7f284413e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f283c0c6b80 0x7f283c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:27.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.919+0000 7f284413e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f283c0c6f60 0x7f283c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:27.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.919+0000 7f284413e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f283c0c8430 0x7f283c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:27.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.919+0000 7f284413e640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f283c0c0a50 con 0x7f283c0c8430 2023-10-30T12:43:27.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.919+0000 7f284413e640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f283c06f6e0 con 0x7f283c0c6f60 2023-10-30T12:43:27.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.919+0000 7f284413e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f283c06f9c0 con 0x7f283c0c6b80 2023-10-30T12:43:27.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.919+0000 7f28426db640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f283c0c8430 0x7f283c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:27.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.919+0000 7f28426db640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f283c0c8430 0x7f283c13ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49376/0 (socket says 172.21.15.89:49376) 2023-10-30T12:43:27.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.919+0000 7f28426db640 1 -- 172.21.15.89:0/539412227 learned_addr learned my addr 172.21.15.89:0/539412227 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:43:27.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.919+0000 7f2841eda640 1 --2- 172.21.15.89:0/539412227 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f283c0c6b80 0x7f283c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:27.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.919+0000 7f28416d9640 1 --2- 172.21.15.89:0/539412227 >> v2:172.21.15.192:3300/0 conn(0x7f283c0c6f60 0x7f283c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:27.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.919+0000 7f28426db640 1 -- 172.21.15.89:0/539412227 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f283c0c6b80 msgr2=0x7f283c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:27.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.919+0000 7f28426db640 1 --2- 172.21.15.89:0/539412227 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f283c0c6b80 0x7f283c0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:27.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.919+0000 7f28426db640 1 -- 172.21.15.89:0/539412227 >> v2:172.21.15.192:3300/0 conn(0x7f283c0c6f60 msgr2=0x7f283c0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:27.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.919+0000 7f28426db640 1 --2- 172.21.15.89:0/539412227 >> v2:172.21.15.192:3300/0 conn(0x7f283c0c6f60 0x7f283c0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:27.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.919+0000 7f28426db640 1 -- 172.21.15.89:0/539412227 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f283c06fca0 con 0x7f283c0c8430 2023-10-30T12:43:27.920 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.919+0000 7f2841eda640 1 --2- 172.21.15.89:0/539412227 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f283c0c6b80 0x7f283c0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:43:27.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.923+0000 7f28426db640 1 --2- 172.21.15.89:0/539412227 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f283c0c8430 0x7f283c13ed90 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f283c06c830 tx=0x7f283401c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f84a0695a6eb9c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:27.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.923+0000 7f2840ed8640 1 -- 172.21.15.89:0/539412227 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28340a5760 con 0x7f283c0c8430 2023-10-30T12:43:27.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.923+0000 7f2840ed8640 1 -- 172.21.15.89:0/539412227 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28340a58c0 con 0x7f283c0c8430 2023-10-30T12:43:27.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.923+0000 7f2840ed8640 1 -- 172.21.15.89:0/539412227 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28340a5c50 con 0x7f283c0c8430 2023-10-30T12:43:27.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.923+0000 7f284413e640 1 -- 172.21.15.89:0/539412227 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f283c0c8430 msgr2=0x7f283c13ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:27.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.923+0000 7f284413e640 1 --2- 172.21.15.89:0/539412227 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f283c0c8430 0x7f283c13ed90 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f283c06c830 tx=0x7f283401c9d0 comp rx=0 tx=0).stop 2023-10-30T12:43:27.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.923+0000 7f284413e640 1 -- 172.21.15.89:0/539412227 shutdown_connections 2023-10-30T12:43:27.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.923+0000 7f284413e640 1 --2- 172.21.15.89:0/539412227 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f283c0c8430 0x7f283c13ed90 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:27.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.923+0000 7f284413e640 1 --2- 172.21.15.89:0/539412227 >> v2:172.21.15.192:3300/0 conn(0x7f283c0c6f60 0x7f283c0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:27.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.923+0000 7f284413e640 1 --2- 172.21.15.89:0/539412227 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f283c0c6b80 0x7f283c0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:27.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.923+0000 7f284413e640 1 -- 172.21.15.89:0/539412227 >> 172.21.15.89:0/539412227 conn(0x7f283c0c7910 msgr2=0x7f283c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:27.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.923+0000 7f284413e640 1 -- 172.21.15.89:0/539412227 shutdown_connections 2023-10-30T12:43:27.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.923+0000 7f284413e640 1 -- 172.21.15.89:0/539412227 wait complete. 2023-10-30T12:43:27.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.927+0000 7f284413e640 1 Processor -- start 2023-10-30T12:43:27.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.927+0000 7f284413e640 1 -- start start 2023-10-30T12:43:27.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.927+0000 7f284413e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f283c0c6b80 0x7f283c13e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:27.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.927+0000 7f284413e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f283c0c6f60 0x7f283c0c92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:27.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.927+0000 7f284413e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f283c0c8430 0x7f283c0c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:27.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.927+0000 7f284413e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283c0c8e20 con 0x7f283c0c6f60 2023-10-30T12:43:27.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.927+0000 7f284413e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283c1efdb0 con 0x7f283c0c8430 2023-10-30T12:43:27.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.927+0000 7f284413e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f283c1f0090 con 0x7f283c0c6b80 2023-10-30T12:43:27.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.927+0000 7f28416d9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f283c0c6f60 0x7f283c0c92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:27.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.927+0000 7f28416d9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f283c0c6f60 0x7f283c0c92b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49388/0 (socket says 172.21.15.89:49388) 2023-10-30T12:43:27.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.927+0000 7f28416d9640 1 -- 172.21.15.89:0/592201384 learned_addr learned my addr 172.21.15.89:0/592201384 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:43:27.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.927+0000 7f2841eda640 1 --2- 172.21.15.89:0/592201384 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f283c0c6b80 0x7f283c13e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:27.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.927+0000 7f28426db640 1 --2- 172.21.15.89:0/592201384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f283c0c8430 0x7f283c0c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:27.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.927+0000 7f28416d9640 1 -- 172.21.15.89:0/592201384 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f283c0c6b80 msgr2=0x7f283c13e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:27.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.927+0000 7f28416d9640 1 --2- 172.21.15.89:0/592201384 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f283c0c6b80 0x7f283c13e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:27.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.927+0000 7f28416d9640 1 -- 172.21.15.89:0/592201384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f283c0c8430 msgr2=0x7f283c0c97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:27.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.927+0000 7f28416d9640 1 --2- 172.21.15.89:0/592201384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f283c0c8430 0x7f283c0c97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:27.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.927+0000 7f28416d9640 1 -- 172.21.15.89:0/592201384 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2834009b30 con 0x7f283c0c6f60 2023-10-30T12:43:27.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.927+0000 7f28426db640 1 --2- 172.21.15.89:0/592201384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f283c0c8430 0x7f283c0c97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:43:27.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.927+0000 7f28416d9640 1 --2- 172.21.15.89:0/592201384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f283c0c6f60 0x7f283c0c92b0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f282c00e900 tx=0x7f282c00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:27.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.927+0000 7f28327fc640 1 -- 172.21.15.89:0/592201384 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282c009690 con 0x7f283c0c6f60 2023-10-30T12:43:27.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.927+0000 7f28327fc640 1 -- 172.21.15.89:0/592201384 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f282c00ca00 con 0x7f283c0c6f60 2023-10-30T12:43:27.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.927+0000 7f284413e640 1 -- 172.21.15.89:0/592201384 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f283c1f0370 con 0x7f283c0c6f60 2023-10-30T12:43:27.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.927+0000 7f28327fc640 1 -- 172.21.15.89:0/592201384 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282c00cdb0 con 0x7f283c0c6f60 2023-10-30T12:43:27.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.927+0000 7f284413e640 1 -- 172.21.15.89:0/592201384 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f283c13e6b0 con 0x7f283c0c6f60 2023-10-30T12:43:27.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.931+0000 7f284413e640 1 -- 172.21.15.89:0/592201384 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f283c0c8fd0 con 0x7f283c0c6f60 2023-10-30T12:43:27.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.931+0000 7f28327fc640 1 -- 172.21.15.89:0/592201384 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f282c013470 con 0x7f283c0c6f60 2023-10-30T12:43:27.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.931+0000 7f28327fc640 1 --2- 172.21.15.89:0/592201384 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f28200664a0 0x7f2820068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:27.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.931+0000 7f28327fc640 1 -- 172.21.15.89:0/592201384 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6320+0+0 (secure 0 0 0) 0x7f282c0843f0 con 0x7f283c0c6f60 2023-10-30T12:43:27.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.931+0000 7f2841eda640 1 --2- 172.21.15.89:0/592201384 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f28200664a0 0x7f2820068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:27.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.931+0000 7f2841eda640 1 --2- 172.21.15.89:0/592201384 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f28200664a0 0x7f2820068960 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f283800c880 tx=0x7f2838008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:27.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:27.935+0000 7f28327fc640 1 -- 172.21.15.89:0/592201384 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f283c0c8fd0 con 0x7f283c0c6f60 2023-10-30T12:43:28.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:28.115+0000 7f284413e640 1 -- 172.21.15.89:0/592201384 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f283c0c8fd0 con 0x7f283c0c6f60 2023-10-30T12:43:28.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:28.115+0000 7f28327fc640 1 -- 172.21.15.89:0/592201384 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f283c0c8fd0 con 0x7f283c0c6f60 2023-10-30T12:43:28.111 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:43:28.111 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5728,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:43:10.164217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:43:28.112 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 72 2023-10-30T12:43:28.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:28.115+0000 7f284413e640 1 -- 172.21.15.89:0/592201384 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f28200664a0 msgr2=0x7f2820068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:28.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:28.115+0000 7f284413e640 1 --2- 172.21.15.89:0/592201384 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f28200664a0 0x7f2820068960 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f283800c880 tx=0x7f2838008040 comp rx=0 tx=0).stop 2023-10-30T12:43:28.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:28.115+0000 7f284413e640 1 -- 172.21.15.89:0/592201384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f283c0c6f60 msgr2=0x7f283c0c92b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:28.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:28.115+0000 7f284413e640 1 --2- 172.21.15.89:0/592201384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f283c0c6f60 0x7f283c0c92b0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f282c00e900 tx=0x7f282c00edc0 comp rx=0 tx=0).stop 2023-10-30T12:43:28.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:28.115+0000 7f284413e640 1 -- 172.21.15.89:0/592201384 shutdown_connections 2023-10-30T12:43:28.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:28.115+0000 7f284413e640 1 --2- 172.21.15.89:0/592201384 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f28200664a0 0x7f2820068960 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:28.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:28.115+0000 7f284413e640 1 --2- 172.21.15.89:0/592201384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f283c0c8430 0x7f283c0c97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:28.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:28.115+0000 7f284413e640 1 --2- 172.21.15.89:0/592201384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f283c0c6f60 0x7f283c0c92b0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:28.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:28.115+0000 7f284413e640 1 --2- 172.21.15.89:0/592201384 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f283c0c6b80 0x7f283c13e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:28.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:28.115+0000 7f284413e640 1 -- 172.21.15.89:0/592201384 >> 172.21.15.89:0/592201384 conn(0x7f283c0c7910 msgr2=0x7f283c0b0890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:28.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:28.115+0000 7f284413e640 1 -- 172.21.15.89:0/592201384 shutdown_connections 2023-10-30T12:43:28.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:28.115+0000 7f284413e640 1 -- 172.21.15.89:0/592201384 wait complete. 2023-10-30T12:43:28.132 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:43:30.132 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:43:30.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.283+0000 7fe6fd417640 1 Processor -- start 2023-10-30T12:43:30.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6fd417640 1 -- start start 2023-10-30T12:43:30.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6fd417640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6f815d2c0 0x7fe6f815d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:30.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6fd417640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f815ce30 con 0x7fe6f80c90d0 2023-10-30T12:43:30.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6fd417640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f815dcb0 con 0x7fe6f80c8cf0 2023-10-30T12:43:30.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6fd417640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f8159880 con 0x7fe6f815d2c0 2023-10-30T12:43:30.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6f67fc640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe6f80c90d0 0x7fe6f80cef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:50912/0 (socket says 172.21.15.89:50912) 2023-10-30T12:43:30.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6f67fc640 1 -- 172.21.15.89:0/4032315394 learned_addr learned my addr 172.21.15.89:0/4032315394 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:43:30.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6f77fe640 1 --2- 172.21.15.89:0/4032315394 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6f815d2c0 0x7fe6f815d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:30.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6f5ffb640 1 -- 172.21.15.89:0/4032315394 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2865226844 0 0) 0x7fe6f815ce30 con 0x7fe6f80c90d0 2023-10-30T12:43:30.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6f5ffb640 1 -- 172.21.15.89:0/4032315394 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6e0003530 con 0x7fe6f80c90d0 2023-10-30T12:43:30.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6f5ffb640 1 -- 172.21.15.89:0/4032315394 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2714440834 0 0) 0x7fe6e0003530 con 0x7fe6f80c90d0 2023-10-30T12:43:30.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6f5ffb640 1 -- 172.21.15.89:0/4032315394 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6f815d2c0 msgr2=0x7fe6f815d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:30.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6f5ffb640 1 --2- 172.21.15.89:0/4032315394 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6f815d2c0 0x7fe6f815d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:30.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6f5ffb640 1 -- 172.21.15.89:0/4032315394 >> v1:172.21.15.192:6789/0 conn(0x7fe6f80c8cf0 legacy=0x7fe6f80c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:30.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6f5ffb640 1 -- 172.21.15.89:0/4032315394 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f8159ae0 con 0x7fe6f80c90d0 2023-10-30T12:43:30.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6f5ffb640 1 -- 172.21.15.89:0/4032315394 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7fe6ec001680 con 0x7fe6f80c90d0 2023-10-30T12:43:30.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6f5ffb640 1 -- 172.21.15.89:0/4032315394 >> v1:172.21.15.89:6789/0 conn(0x7fe6f80c90d0 legacy=0x7fe6f80cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:30.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6f5ffb640 1 --2- 172.21.15.89:0/4032315394 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6e00039b0 0x7fe6e0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:30.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6f5ffb640 1 --2- 172.21.15.89:0/4032315394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e00047d0 0x7fe6e0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:30.287 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6f5ffb640 1 --2- 172.21.15.89:0/4032315394 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6f815d2c0 0x7fe6e0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:30.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6f5ffb640 1 -- 172.21.15.89:0/4032315394 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f8159880 con 0x7fe6e00047d0 2023-10-30T12:43:30.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6f5ffb640 1 -- 172.21.15.89:0/4032315394 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e000ba40 con 0x7fe6e00039b0 2023-10-30T12:43:30.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6f5ffb640 1 -- 172.21.15.89:0/4032315394 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e000bc00 con 0x7fe6f815d2c0 2023-10-30T12:43:30.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6f67fc640 1 --2- 172.21.15.89:0/4032315394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e00047d0 0x7fe6e0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:30.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6f6ffd640 1 --2- 172.21.15.89:0/4032315394 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6e00039b0 0x7fe6e0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:30.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6f77fe640 1 --2- 172.21.15.89:0/4032315394 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6f815d2c0 0x7fe6e0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:30.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6f67fc640 1 -- 172.21.15.89:0/4032315394 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6f815d2c0 msgr2=0x7fe6e0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:30.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6f67fc640 1 --2- 172.21.15.89:0/4032315394 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6f815d2c0 0x7fe6e0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:30.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6f67fc640 1 -- 172.21.15.89:0/4032315394 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6e00039b0 msgr2=0x7fe6e0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:30.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6f67fc640 1 --2- 172.21.15.89:0/4032315394 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6e00039b0 0x7fe6e0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:30.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.287+0000 7fe6f67fc640 1 -- 172.21.15.89:0/4032315394 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e0003530 con 0x7fe6e00047d0 2023-10-30T12:43:30.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.291+0000 7fe6f67fc640 1 --2- 172.21.15.89:0/4032315394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e00047d0 0x7fe6e0004c50 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec00c210 tx=0x7fe6ec01f350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2a58cb0badbb08b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:30.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.291+0000 7fe6f5ffb640 1 -- 172.21.15.89:0/4032315394 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec005ae0 con 0x7fe6e00047d0 2023-10-30T12:43:30.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.291+0000 7fe6f5ffb640 1 -- 172.21.15.89:0/4032315394 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6ec0ab070 con 0x7fe6e00047d0 2023-10-30T12:43:30.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.291+0000 7fe6f5ffb640 1 -- 172.21.15.89:0/4032315394 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec007070 con 0x7fe6e00047d0 2023-10-30T12:43:30.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.291+0000 7fe6fd417640 1 -- 172.21.15.89:0/4032315394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e00047d0 msgr2=0x7fe6e0004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:30.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.291+0000 7fe6fd417640 1 --2- 172.21.15.89:0/4032315394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6e00047d0 0x7fe6e0004c50 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec00c210 tx=0x7fe6ec01f350 comp rx=0 tx=0).stop 2023-10-30T12:43:30.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.291+0000 7fe6f77fe640 1 -- 172.21.15.89:0/4032315394 reap_dead start 2023-10-30T12:43:30.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.291+0000 7fe6fd417640 1 -- 172.21.15.89:0/4032315394 shutdown_connections 2023-10-30T12:43:30.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.291+0000 7fe6fd417640 1 -- 172.21.15.89:0/4032315394 >> 172.21.15.89:0/4032315394 conn(0x7fe6f806be40 msgr2=0x7fe6f806c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:30.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.291+0000 7fe6fd417640 1 -- 172.21.15.89:0/4032315394 shutdown_connections 2023-10-30T12:43:30.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.291+0000 7fe6fd417640 1 -- 172.21.15.89:0/4032315394 wait complete. 2023-10-30T12:43:30.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.295+0000 7fe6fd417640 1 Processor -- start 2023-10-30T12:43:30.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.295+0000 7fe6fd417640 1 -- start start 2023-10-30T12:43:30.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.295+0000 7fe6fd417640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6e00047d0 0x7fe6f81506a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:30.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.295+0000 7fe6fd417640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6f80c7780 0x7fe6f815c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:30.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.295+0000 7fe6fd417640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6f8156860 0x7fe6f815c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:30.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.295+0000 7fe6fd417640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f81ef420 con 0x7fe6f8156860 2023-10-30T12:43:30.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.295+0000 7fe6fd417640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f8155d30 con 0x7fe6e00047d0 2023-10-30T12:43:30.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.295+0000 7fe6fd417640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f8155730 con 0x7fe6f80c7780 2023-10-30T12:43:30.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.295+0000 7fe6f77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6f8156860 0x7fe6f815c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:30.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.295+0000 7fe6f6ffd640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6e00047d0 0x7fe6f81506a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:30.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.295+0000 7fe6f77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6f8156860 0x7fe6f815c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51898/0 (socket says 172.21.15.89:51898) 2023-10-30T12:43:30.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.295+0000 7fe6f77fe640 1 -- 172.21.15.89:0/2975719136 learned_addr learned my addr 172.21.15.89:0/2975719136 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:43:30.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.295+0000 7fe6f67fc640 1 --2- 172.21.15.89:0/2975719136 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6f80c7780 0x7fe6f815c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:30.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.295+0000 7fe6f77fe640 1 -- 172.21.15.89:0/2975719136 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6f80c7780 msgr2=0x7fe6f815c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:30.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.295+0000 7fe6f77fe640 1 --2- 172.21.15.89:0/2975719136 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6f80c7780 0x7fe6f815c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:30.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.295+0000 7fe6f77fe640 1 -- 172.21.15.89:0/2975719136 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6e00047d0 msgr2=0x7fe6f81506a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:30.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.295+0000 7fe6f77fe640 1 --2- 172.21.15.89:0/2975719136 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6e00047d0 0x7fe6f81506a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:30.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.295+0000 7fe6f77fe640 1 -- 172.21.15.89:0/2975719136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f81ef140 con 0x7fe6f8156860 2023-10-30T12:43:30.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.295+0000 7fe6f67fc640 1 --2- 172.21.15.89:0/2975719136 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6f80c7780 0x7fe6f815c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:43:30.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.295+0000 7fe6f77fe640 1 --2- 172.21.15.89:0/2975719136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6f8156860 0x7fe6f815c7f0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8072eb0 tx=0x7fe6f00115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:30.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.295+0000 7fe6df7fe640 1 -- 172.21.15.89:0/2975719136 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f000e070 con 0x7fe6f8156860 2023-10-30T12:43:30.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.295+0000 7fe6df7fe640 1 -- 172.21.15.89:0/2975719136 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f0011ed0 con 0x7fe6f8156860 2023-10-30T12:43:30.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.295+0000 7fe6df7fe640 1 -- 172.21.15.89:0/2975719136 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f00039a0 con 0x7fe6f8156860 2023-10-30T12:43:30.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.295+0000 7fe6fd417640 1 -- 172.21.15.89:0/2975719136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6f8157050 con 0x7fe6f8156860 2023-10-30T12:43:30.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.295+0000 7fe6fd417640 1 -- 172.21.15.89:0/2975719136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6f815bd90 con 0x7fe6f8156860 2023-10-30T12:43:30.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.295+0000 7fe6fd417640 1 -- 172.21.15.89:0/2975719136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6f8156e90 con 0x7fe6f8156860 2023-10-30T12:43:30.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.299+0000 7fe6df7fe640 1 -- 172.21.15.89:0/2975719136 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe6f0003b20 con 0x7fe6f8156860 2023-10-30T12:43:30.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.299+0000 7fe6df7fe640 1 --2- 172.21.15.89:0/2975719136 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe6e40664d0 0x7fe6e4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:30.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.299+0000 7fe6df7fe640 1 -- 172.21.15.89:0/2975719136 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6320+0+0 (secure 0 0 0) 0x7fe6f00847c0 con 0x7fe6f8156860 2023-10-30T12:43:30.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.299+0000 7fe6f6ffd640 1 --2- 172.21.15.89:0/2975719136 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe6e40664d0 0x7fe6e4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:30.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.299+0000 7fe6f6ffd640 1 --2- 172.21.15.89:0/2975719136 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe6e40664d0 0x7fe6e4068990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8001660 tx=0x7fe6e8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:30.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.303+0000 7fe6df7fe640 1 -- 172.21.15.89:0/2975719136 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe6f0051960 con 0x7fe6f8156860 2023-10-30T12:43:30.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.491+0000 7fe6fd417640 1 -- 172.21.15.89:0/2975719136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6f815bf80 con 0x7fe6f8156860 2023-10-30T12:43:30.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.491+0000 7fe6df7fe640 1 -- 172.21.15.89:0/2975719136 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+6532 (secure 0 0 0) 0x7fe6f0054f80 con 0x7fe6f8156860 2023-10-30T12:43:30.486 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:43:30.487 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5728,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:43:10.164217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:43:30.487 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 72 2023-10-30T12:43:30.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.491+0000 7fe6fd417640 1 -- 172.21.15.89:0/2975719136 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe6e40664d0 msgr2=0x7fe6e4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:30.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.491+0000 7fe6fd417640 1 --2- 172.21.15.89:0/2975719136 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe6e40664d0 0x7fe6e4068990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8001660 tx=0x7fe6e8009040 comp rx=0 tx=0).stop 2023-10-30T12:43:30.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.491+0000 7fe6fd417640 1 -- 172.21.15.89:0/2975719136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6f8156860 msgr2=0x7fe6f815c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:30.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.491+0000 7fe6fd417640 1 --2- 172.21.15.89:0/2975719136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6f8156860 0x7fe6f815c7f0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8072eb0 tx=0x7fe6f00115a0 comp rx=0 tx=0).stop 2023-10-30T12:43:30.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.491+0000 7fe6fd417640 1 -- 172.21.15.89:0/2975719136 shutdown_connections 2023-10-30T12:43:30.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.491+0000 7fe6fd417640 1 --2- 172.21.15.89:0/2975719136 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe6e40664d0 0x7fe6e4068990 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:30.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.491+0000 7fe6fd417640 1 --2- 172.21.15.89:0/2975719136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6f8156860 0x7fe6f815c7f0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:30.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.491+0000 7fe6fd417640 1 --2- 172.21.15.89:0/2975719136 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe6f80c7780 0x7fe6f815c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:30.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.491+0000 7fe6fd417640 1 --2- 172.21.15.89:0/2975719136 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe6e00047d0 0x7fe6f81506a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:30.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.491+0000 7fe6fd417640 1 -- 172.21.15.89:0/2975719136 >> 172.21.15.89:0/2975719136 conn(0x7fe6f806be40 msgr2=0x7fe6f80b22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:30.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.491+0000 7fe6fd417640 1 -- 172.21.15.89:0/2975719136 shutdown_connections 2023-10-30T12:43:30.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:30.491+0000 7fe6fd417640 1 -- 172.21.15.89:0/2975719136 wait complete. 2023-10-30T12:43:30.507 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:43:32.508 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:43:32.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.658+0000 7f526c66b640 1 Processor -- start 2023-10-30T12:43:32.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.658+0000 7f526c66b640 1 -- start start 2023-10-30T12:43:32.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f526c66b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52640c6b80 0x7f52640bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:32.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f526c66b640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f526413f4a0 0x7f52640c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:32.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f526c66b640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52640c0a50 con 0x7f52640c6f60 2023-10-30T12:43:32.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f526c66b640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f526406f6e0 con 0x7f526413f4a0 2023-10-30T12:43:32.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f526c66b640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f526406f9c0 con 0x7f52640c6b80 2023-10-30T12:43:32.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f5269c06640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f52640c6f60 0x7f526413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:50914/0 (socket says 172.21.15.89:50914) 2023-10-30T12:43:32.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f5269c06640 1 -- 172.21.15.89:0/512761219 learned_addr learned my addr 172.21.15.89:0/512761219 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:43:32.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f526ac08640 1 --2- 172.21.15.89:0/512761219 >> v2:172.21.15.192:3300/0 conn(0x7f526413f4a0 0x7f52640c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:32.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f526a407640 1 --2- 172.21.15.89:0/512761219 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52640c6b80 0x7f52640bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:32.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f5269405640 1 -- 172.21.15.89:0/512761219 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3247400247 0 0) 0x7f52640c0a50 con 0x7f52640c6f60 2023-10-30T12:43:32.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f5269405640 1 -- 172.21.15.89:0/512761219 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5250005740 con 0x7f52640c6f60 2023-10-30T12:43:32.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f5269405640 1 -- 172.21.15.89:0/512761219 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 995742172 0 0) 0x7f5250005740 con 0x7f52640c6f60 2023-10-30T12:43:32.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f5269405640 1 -- 172.21.15.89:0/512761219 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52640c6b80 msgr2=0x7f52640bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:32.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f5269405640 1 --2- 172.21.15.89:0/512761219 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52640c6b80 0x7f52640bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:32.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f5269405640 1 -- 172.21.15.89:0/512761219 >> v2:172.21.15.192:3300/0 conn(0x7f526413f4a0 msgr2=0x7f52640c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:32.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f5269405640 1 --2- 172.21.15.89:0/512761219 >> v2:172.21.15.192:3300/0 conn(0x7f526413f4a0 0x7f52640c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:32.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f5269405640 1 -- 172.21.15.89:0/512761219 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f526406fca0 con 0x7f52640c6f60 2023-10-30T12:43:32.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f5269405640 1 -- 172.21.15.89:0/512761219 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f5254001680 con 0x7f52640c6f60 2023-10-30T12:43:32.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f5269405640 1 -- 172.21.15.89:0/512761219 >> v1:172.21.15.89:6789/0 conn(0x7f52640c6f60 legacy=0x7f526413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:32.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f5269405640 1 --2- 172.21.15.89:0/512761219 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5250005b90 0x7f5250005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:32.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f5269405640 1 --2- 172.21.15.89:0/512761219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52500069e0 0x7f5250006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:32.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f5269405640 1 --2- 172.21.15.89:0/512761219 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52640c6b80 0x7f52500073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:32.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f5269405640 1 -- 172.21.15.89:0/512761219 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f526406f6e0 con 0x7f52500069e0 2023-10-30T12:43:32.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f5269405640 1 -- 172.21.15.89:0/512761219 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f526406f9c0 con 0x7f5250005b90 2023-10-30T12:43:32.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f5269405640 1 -- 172.21.15.89:0/512761219 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f525000dba0 con 0x7f52640c6b80 2023-10-30T12:43:32.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f5269c06640 1 --2- 172.21.15.89:0/512761219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52500069e0 0x7f5250006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:32.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f526ac08640 1 --2- 172.21.15.89:0/512761219 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52640c6b80 0x7f52500073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:32.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f526a407640 1 --2- 172.21.15.89:0/512761219 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5250005b90 0x7f5250005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:32.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f5269c06640 1 -- 172.21.15.89:0/512761219 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52640c6b80 msgr2=0x7f52500073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:32.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f5269c06640 1 --2- 172.21.15.89:0/512761219 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52640c6b80 0x7f52500073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:32.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f5269c06640 1 -- 172.21.15.89:0/512761219 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5250005b90 msgr2=0x7f5250005fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:32.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f5269c06640 1 --2- 172.21.15.89:0/512761219 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5250005b90 0x7f5250005fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:32.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.662+0000 7f5269c06640 1 -- 172.21.15.89:0/512761219 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5250005740 con 0x7f52500069e0 2023-10-30T12:43:32.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.666+0000 7f5269c06640 1 --2- 172.21.15.89:0/512761219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52500069e0 0x7f5250006e60 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f525400c210 tx=0x7f525401f390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d834327db4f080af server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:32.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.666+0000 7f5269405640 1 -- 172.21.15.89:0/512761219 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52540a8cb0 con 0x7f52500069e0 2023-10-30T12:43:32.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.666+0000 7f5269405640 1 -- 172.21.15.89:0/512761219 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52540a8e10 con 0x7f52500069e0 2023-10-30T12:43:32.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.666+0000 7f5269405640 1 -- 172.21.15.89:0/512761219 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5254005b00 con 0x7f52500069e0 2023-10-30T12:43:32.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.666+0000 7f526c66b640 1 -- 172.21.15.89:0/512761219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52500069e0 msgr2=0x7f5250006e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:32.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.666+0000 7f526c66b640 1 --2- 172.21.15.89:0/512761219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52500069e0 0x7f5250006e60 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f525400c210 tx=0x7f525401f390 comp rx=0 tx=0).stop 2023-10-30T12:43:32.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.666+0000 7f526ac08640 1 -- 172.21.15.89:0/512761219 reap_dead start 2023-10-30T12:43:32.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.666+0000 7f526c66b640 1 -- 172.21.15.89:0/512761219 shutdown_connections 2023-10-30T12:43:32.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.666+0000 7f526c66b640 1 -- 172.21.15.89:0/512761219 >> 172.21.15.89:0/512761219 conn(0x7f52640c7910 msgr2=0x7f52640c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:32.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.666+0000 7f526c66b640 1 -- 172.21.15.89:0/512761219 shutdown_connections 2023-10-30T12:43:32.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.666+0000 7f526c66b640 1 -- 172.21.15.89:0/512761219 wait complete. 2023-10-30T12:43:32.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.670+0000 7f526c66b640 1 Processor -- start 2023-10-30T12:43:32.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.670+0000 7f526c66b640 1 -- start start 2023-10-30T12:43:32.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.670+0000 7f526c66b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52500069e0 0x7f526413b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:32.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.670+0000 7f526c66b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f526413dad0 0x7f52640c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:32.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.670+0000 7f526c66b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f526413def0 0x7f52640af830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:32.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.670+0000 7f526c66b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f526413bca0 con 0x7f526413dad0 2023-10-30T12:43:32.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.670+0000 7f526c66b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52640ca570 con 0x7f526413def0 2023-10-30T12:43:32.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.670+0000 7f526c66b640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52641f1ef0 con 0x7f52500069e0 2023-10-30T12:43:32.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.670+0000 7f5269c06640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f526413dad0 0x7f52640c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:32.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.670+0000 7f5269c06640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f526413dad0 0x7f52640c8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51914/0 (socket says 172.21.15.89:51914) 2023-10-30T12:43:32.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.670+0000 7f5269c06640 1 -- 172.21.15.89:0/673421641 learned_addr learned my addr 172.21.15.89:0/673421641 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:43:32.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.670+0000 7f526ac08640 1 --2- 172.21.15.89:0/673421641 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f526413def0 0x7f52640af830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:32.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.670+0000 7f526a407640 1 --2- 172.21.15.89:0/673421641 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52500069e0 0x7f526413b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:32.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.670+0000 7f5269c06640 1 -- 172.21.15.89:0/673421641 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52500069e0 msgr2=0x7f526413b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:32.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.670+0000 7f5269c06640 1 --2- 172.21.15.89:0/673421641 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52500069e0 0x7f526413b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:32.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.670+0000 7f5269c06640 1 -- 172.21.15.89:0/673421641 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f526413def0 msgr2=0x7f52640af830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:32.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.670+0000 7f5269c06640 1 --2- 172.21.15.89:0/673421641 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f526413def0 0x7f52640af830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:32.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.670+0000 7f5269c06640 1 -- 172.21.15.89:0/673421641 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52640ca830 con 0x7f526413dad0 2023-10-30T12:43:32.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.670+0000 7f526a407640 1 --2- 172.21.15.89:0/673421641 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52500069e0 0x7f526413b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:43:32.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.670+0000 7f5269c06640 1 --2- 172.21.15.89:0/673421641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f526413dad0 0x7f52640c8c10 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f52540ae040 tx=0x7f5250007a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:32.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.670+0000 7f525affd640 1 -- 172.21.15.89:0/673421641 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52540049f0 con 0x7f526413dad0 2023-10-30T12:43:32.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.670+0000 7f525affd640 1 -- 172.21.15.89:0/673421641 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52540b6070 con 0x7f526413dad0 2023-10-30T12:43:32.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.670+0000 7f525affd640 1 -- 172.21.15.89:0/673421641 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5254004d70 con 0x7f526413dad0 2023-10-30T12:43:32.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.670+0000 7f526c66b640 1 -- 172.21.15.89:0/673421641 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52640cab10 con 0x7f526413dad0 2023-10-30T12:43:32.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.670+0000 7f526c66b640 1 -- 172.21.15.89:0/673421641 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52641f2490 con 0x7f526413dad0 2023-10-30T12:43:32.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.670+0000 7f526c66b640 1 -- 172.21.15.89:0/673421641 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f526404e170 con 0x7f526413dad0 2023-10-30T12:43:32.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.674+0000 7f525affd640 1 -- 172.21.15.89:0/673421641 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f52540070d0 con 0x7f526413dad0 2023-10-30T12:43:32.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.674+0000 7f525affd640 1 --2- 172.21.15.89:0/673421641 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f52480664a0 0x7f5248068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:32.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.674+0000 7f525affd640 1 -- 172.21.15.89:0/673421641 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6320+0+0 (secure 0 0 0) 0x7f52540b8070 con 0x7f526413dad0 2023-10-30T12:43:32.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.674+0000 7f526a407640 1 --2- 172.21.15.89:0/673421641 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f52480664a0 0x7f5248068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:32.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.674+0000 7f526a407640 1 --2- 172.21.15.89:0/673421641 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f52480664a0 0x7f5248068960 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f5260009990 tx=0x7f5260008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:32.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.678+0000 7f525affd640 1 -- 172.21.15.89:0/673421641 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f52540ee200 con 0x7f526413dad0 2023-10-30T12:43:32.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.858+0000 7f526c66b640 1 -- 172.21.15.89:0/673421641 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52640ad8d0 con 0x7f526413dad0 2023-10-30T12:43:32.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.858+0000 7f525affd640 1 -- 172.21.15.89:0/673421641 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f52540f1820 con 0x7f526413dad0 2023-10-30T12:43:32.854 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:43:32.855 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5728,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:43:10.164217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:43:32.855 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 72 2023-10-30T12:43:32.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.858+0000 7f526c66b640 1 -- 172.21.15.89:0/673421641 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f52480664a0 msgr2=0x7f5248068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:32.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.858+0000 7f526c66b640 1 --2- 172.21.15.89:0/673421641 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f52480664a0 0x7f5248068960 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f5260009990 tx=0x7f5260008040 comp rx=0 tx=0).stop 2023-10-30T12:43:32.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.858+0000 7f526c66b640 1 -- 172.21.15.89:0/673421641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f526413dad0 msgr2=0x7f52640c8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:32.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.858+0000 7f526c66b640 1 --2- 172.21.15.89:0/673421641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f526413dad0 0x7f52640c8c10 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f52540ae040 tx=0x7f5250007a10 comp rx=0 tx=0).stop 2023-10-30T12:43:32.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.858+0000 7f526c66b640 1 -- 172.21.15.89:0/673421641 shutdown_connections 2023-10-30T12:43:32.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.858+0000 7f526c66b640 1 --2- 172.21.15.89:0/673421641 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f52480664a0 0x7f5248068960 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:32.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.858+0000 7f526c66b640 1 --2- 172.21.15.89:0/673421641 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f526413def0 0x7f52640af830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:32.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.858+0000 7f526c66b640 1 --2- 172.21.15.89:0/673421641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f526413dad0 0x7f52640c8c10 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:32.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.858+0000 7f526c66b640 1 --2- 172.21.15.89:0/673421641 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52500069e0 0x7f526413b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:32.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.858+0000 7f526c66b640 1 -- 172.21.15.89:0/673421641 >> 172.21.15.89:0/673421641 conn(0x7f52640c7910 msgr2=0x7f52640c0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:32.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.858+0000 7f526c66b640 1 -- 172.21.15.89:0/673421641 shutdown_connections 2023-10-30T12:43:32.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:32.858+0000 7f526c66b640 1 -- 172.21.15.89:0/673421641 wait complete. 2023-10-30T12:43:32.875 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:43:34.876 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:43:35.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.034+0000 7f58a726b640 1 Processor -- start 2023-10-30T12:43:35.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.034+0000 7f58a726b640 1 -- start start 2023-10-30T12:43:35.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.038+0000 7f58a726b640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f58a01580e0 0x7f58a0070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:35.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.038+0000 7f58a726b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58a00c6540 0x7f58a015e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:35.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.038+0000 7f58a726b640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f58a0069e90 0x7f58a006a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:35.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.038+0000 7f58a726b640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58a00c5810 con 0x7f58a0069e90 2023-10-30T12:43:35.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.038+0000 7f58a726b640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58a00c6960 con 0x7f58a01580e0 2023-10-30T12:43:35.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.038+0000 7f58a726b640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58a00c6bb0 con 0x7f58a00c6540 2023-10-30T12:43:35.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.038+0000 7f58a5808640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f58a0069e90 0x7f58a006a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:35.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.038+0000 7f58a5808640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f58a0069e90 0x7f58a006a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51918/0 (socket says 172.21.15.89:51918) 2023-10-30T12:43:35.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.038+0000 7f58a5808640 1 -- 172.21.15.89:0/2133363586 learned_addr learned my addr 172.21.15.89:0/2133363586 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:43:35.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.038+0000 7f58a5007640 1 --2- 172.21.15.89:0/2133363586 >> v2:172.21.15.192:3300/0 conn(0x7f58a01580e0 0x7f58a0070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:35.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.038+0000 7f58a4806640 1 --2- 172.21.15.89:0/2133363586 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58a00c6540 0x7f58a015e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:35.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.038+0000 7f58a5808640 1 -- 172.21.15.89:0/2133363586 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58a00c6540 msgr2=0x7f58a015e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:35.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.038+0000 7f58a5808640 1 --2- 172.21.15.89:0/2133363586 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58a00c6540 0x7f58a015e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:35.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.038+0000 7f58a5808640 1 -- 172.21.15.89:0/2133363586 >> v2:172.21.15.192:3300/0 conn(0x7f58a01580e0 msgr2=0x7f58a0070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:35.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.038+0000 7f58a5808640 1 --2- 172.21.15.89:0/2133363586 >> v2:172.21.15.192:3300/0 conn(0x7f58a01580e0 0x7f58a0070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:35.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.038+0000 7f58a5808640 1 -- 172.21.15.89:0/2133363586 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a00c6e70 con 0x7f58a0069e90 2023-10-30T12:43:35.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.042+0000 7f58a5808640 1 --2- 172.21.15.89:0/2133363586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58a0069e90 0x7f58a006a320 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f58a006eea0 tx=0x7f589801c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fee198971e4f79e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:35.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.042+0000 7f5897fff640 1 -- 172.21.15.89:0/2133363586 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58980a5760 con 0x7f58a0069e90 2023-10-30T12:43:35.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.042+0000 7f5897fff640 1 -- 172.21.15.89:0/2133363586 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58980a58c0 con 0x7f58a0069e90 2023-10-30T12:43:35.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.042+0000 7f5897fff640 1 -- 172.21.15.89:0/2133363586 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58980a5c50 con 0x7f58a0069e90 2023-10-30T12:43:35.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.042+0000 7f58a726b640 1 -- 172.21.15.89:0/2133363586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58a0069e90 msgr2=0x7f58a006a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:35.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.042+0000 7f58a726b640 1 --2- 172.21.15.89:0/2133363586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58a0069e90 0x7f58a006a320 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f58a006eea0 tx=0x7f589801c9d0 comp rx=0 tx=0).stop 2023-10-30T12:43:35.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.042+0000 7f58a726b640 1 -- 172.21.15.89:0/2133363586 shutdown_connections 2023-10-30T12:43:35.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.042+0000 7f58a726b640 1 --2- 172.21.15.89:0/2133363586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58a0069e90 0x7f58a006a320 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:35.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.042+0000 7f58a726b640 1 --2- 172.21.15.89:0/2133363586 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58a00c6540 0x7f58a015e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:35.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.042+0000 7f58a726b640 1 --2- 172.21.15.89:0/2133363586 >> v2:172.21.15.192:3300/0 conn(0x7f58a01580e0 0x7f58a0070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:35.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.042+0000 7f58a726b640 1 -- 172.21.15.89:0/2133363586 >> 172.21.15.89:0/2133363586 conn(0x7f58a00c7910 msgr2=0x7f58a00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:35.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.042+0000 7f58a726b640 1 -- 172.21.15.89:0/2133363586 shutdown_connections 2023-10-30T12:43:35.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.042+0000 7f58a726b640 1 -- 172.21.15.89:0/2133363586 wait complete. 2023-10-30T12:43:35.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.042+0000 7f58a726b640 1 Processor -- start 2023-10-30T12:43:35.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.042+0000 7f58a726b640 1 -- start start 2023-10-30T12:43:35.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.046+0000 7f58a726b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58a0069e90 0x7f58a01f6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:35.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.046+0000 7f58a726b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58a00c6540 0x7f58a00c42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:35.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.046+0000 7f58a726b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58a01580e0 0x7f58a0160930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:35.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.046+0000 7f58a726b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a00c3ec0 con 0x7f58a00c6540 2023-10-30T12:43:35.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.046+0000 7f58a726b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a01f74c0 con 0x7f58a0069e90 2023-10-30T12:43:35.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.046+0000 7f58a726b640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58a00c9c60 con 0x7f58a01580e0 2023-10-30T12:43:35.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.046+0000 7f58a4806640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58a00c6540 0x7f58a00c42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:35.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.046+0000 7f58a4806640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58a00c6540 0x7f58a00c42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51928/0 (socket says 172.21.15.89:51928) 2023-10-30T12:43:35.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.046+0000 7f58a4806640 1 -- 172.21.15.89:0/3107458092 learned_addr learned my addr 172.21.15.89:0/3107458092 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:43:35.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.046+0000 7f58a5007640 1 --2- 172.21.15.89:0/3107458092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58a0069e90 0x7f58a01f6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:35.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.046+0000 7f58a5808640 1 --2- 172.21.15.89:0/3107458092 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58a01580e0 0x7f58a0160930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:35.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.046+0000 7f58a4806640 1 -- 172.21.15.89:0/3107458092 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58a01580e0 msgr2=0x7f58a0160930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:35.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.046+0000 7f58a4806640 1 --2- 172.21.15.89:0/3107458092 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58a01580e0 0x7f58a0160930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:35.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.046+0000 7f58a4806640 1 -- 172.21.15.89:0/3107458092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58a0069e90 msgr2=0x7f58a01f6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:35.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.046+0000 7f58a4806640 1 --2- 172.21.15.89:0/3107458092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58a0069e90 0x7f58a01f6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:35.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.046+0000 7f58a4806640 1 -- 172.21.15.89:0/3107458092 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5898009b30 con 0x7f58a00c6540 2023-10-30T12:43:35.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.046+0000 7f58a5808640 1 --2- 172.21.15.89:0/3107458092 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58a01580e0 0x7f58a0160930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:43:35.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.046+0000 7f58a4806640 1 --2- 172.21.15.89:0/3107458092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58a00c6540 0x7f58a00c42e0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f589000d800 tx=0x7f589000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:35.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.046+0000 7f5895ffb640 1 -- 172.21.15.89:0/3107458092 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5890009640 con 0x7f58a00c6540 2023-10-30T12:43:35.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.046+0000 7f5895ffb640 1 -- 172.21.15.89:0/3107458092 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58900097a0 con 0x7f58a00c6540 2023-10-30T12:43:35.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.046+0000 7f58a726b640 1 -- 172.21.15.89:0/3107458092 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58a00c9f00 con 0x7f58a00c6540 2023-10-30T12:43:35.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.046+0000 7f5895ffb640 1 -- 172.21.15.89:0/3107458092 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5890009b10 con 0x7f58a00c6540 2023-10-30T12:43:35.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.046+0000 7f58a726b640 1 -- 172.21.15.89:0/3107458092 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58a00ca4a0 con 0x7f58a00c6540 2023-10-30T12:43:35.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.046+0000 7f58a726b640 1 -- 172.21.15.89:0/3107458092 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a00c4000 con 0x7f58a00c6540 2023-10-30T12:43:35.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.046+0000 7f5895ffb640 1 -- 172.21.15.89:0/3107458092 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5890009c70 con 0x7f58a00c6540 2023-10-30T12:43:35.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.050+0000 7f5895ffb640 1 --2- 172.21.15.89:0/3107458092 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f58840664a0 0x7f5884068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:35.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.050+0000 7f5895ffb640 1 -- 172.21.15.89:0/3107458092 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6320+0+0 (secure 0 0 0) 0x7f5890084910 con 0x7f58a00c6540 2023-10-30T12:43:35.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.050+0000 7f58a5007640 1 --2- 172.21.15.89:0/3107458092 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f58840664a0 0x7f5884068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:35.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.050+0000 7f58a5007640 1 --2- 172.21.15.89:0/3107458092 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f58840664a0 0x7f5884068960 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f589c00b930 tx=0x7f589c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:35.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.054+0000 7f5895ffb640 1 -- 172.21.15.89:0/3107458092 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f58a00c4000 con 0x7f58a00c6540 2023-10-30T12:43:35.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.234+0000 7f58a726b640 1 -- 172.21.15.89:0/3107458092 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58a00c4000 con 0x7f58a00c6540 2023-10-30T12:43:35.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.234+0000 7f5895ffb640 1 -- 172.21.15.89:0/3107458092 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f58a00c4000 con 0x7f58a00c6540 2023-10-30T12:43:35.232 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:43:35.232 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5728,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:43:10.164217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5129},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5129":{"gid":5129,"name":"f","rank":2,"incarnation":57,"state":"up:active","state_seq":40,"addr":"172.21.15.192:6837/1413360446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":1413360446},{"type":"v1","addr":"172.21.15.192:6837","nonce":1413360446}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:43:35.232 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 72 2023-10-30T12:43:35.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.234+0000 7f58a726b640 1 -- 172.21.15.89:0/3107458092 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f58840664a0 msgr2=0x7f5884068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:35.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.238+0000 7f58a726b640 1 --2- 172.21.15.89:0/3107458092 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f58840664a0 0x7f5884068960 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f589c00b930 tx=0x7f589c008040 comp rx=0 tx=0).stop 2023-10-30T12:43:35.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.238+0000 7f58a726b640 1 -- 172.21.15.89:0/3107458092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58a00c6540 msgr2=0x7f58a00c42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:35.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.238+0000 7f58a726b640 1 --2- 172.21.15.89:0/3107458092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58a00c6540 0x7f58a00c42e0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f589000d800 tx=0x7f589000dcc0 comp rx=0 tx=0).stop 2023-10-30T12:43:35.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.238+0000 7f58a726b640 1 -- 172.21.15.89:0/3107458092 shutdown_connections 2023-10-30T12:43:35.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.238+0000 7f58a726b640 1 --2- 172.21.15.89:0/3107458092 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f58840664a0 0x7f5884068960 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:35.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.238+0000 7f58a726b640 1 --2- 172.21.15.89:0/3107458092 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58a01580e0 0x7f58a0160930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:35.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.238+0000 7f58a726b640 1 --2- 172.21.15.89:0/3107458092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58a00c6540 0x7f58a00c42e0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:35.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.238+0000 7f58a726b640 1 --2- 172.21.15.89:0/3107458092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58a0069e90 0x7f58a01f6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:35.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.238+0000 7f58a726b640 1 -- 172.21.15.89:0/3107458092 >> 172.21.15.89:0/3107458092 conn(0x7f58a00c7910 msgr2=0x7f58a00c0720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:35.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.238+0000 7f58a726b640 1 -- 172.21.15.89:0/3107458092 shutdown_connections 2023-10-30T12:43:35.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:35.238+0000 7f58a726b640 1 -- 172.21.15.89:0/3107458092 wait complete. 2023-10-30T12:43:35.252 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:43:37.253 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:43:37.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.394+0000 7f6431846640 1 Processor -- start 2023-10-30T12:43:37.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.394+0000 7f6431846640 1 -- start start 2023-10-30T12:43:37.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.394+0000 7f6431846640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f642c0c6bd0 0x7f642c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:37.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.394+0000 7f6431846640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f642c071110 0x7f642c158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:37.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.394+0000 7f6431846640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f642c066cf0 con 0x7f642c06f7a0 2023-10-30T12:43:37.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.394+0000 7f6431846640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f642c0700c0 con 0x7f642c071110 2023-10-30T12:43:37.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.394+0000 7f6431846640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f642c0c70d0 con 0x7f642c0c6bd0 2023-10-30T12:43:37.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.398+0000 7f642b7fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f642c071110 0x7f642c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:37.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.398+0000 7f642b7fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f642c071110 0x7f642c158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51944/0 (socket says 172.21.15.89:51944) 2023-10-30T12:43:37.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.398+0000 7f642b7fe640 1 -- 172.21.15.89:0/323904459 learned_addr learned my addr 172.21.15.89:0/323904459 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:43:37.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.398+0000 7f64227fc640 1 --2- 172.21.15.89:0/323904459 >> v2:172.21.15.192:3300/0 conn(0x7f642c0c6bd0 0x7f642c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:37.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.398+0000 7f642b7fe640 1 -- 172.21.15.89:0/323904459 >> v2:172.21.15.192:3300/0 conn(0x7f642c0c6bd0 msgr2=0x7f642c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:37.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.398+0000 7f642b7fe640 1 --2- 172.21.15.89:0/323904459 >> v2:172.21.15.192:3300/0 conn(0x7f642c0c6bd0 0x7f642c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:37.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.398+0000 7f642b7fe640 1 -- 172.21.15.89:0/323904459 >> v1:172.21.15.192:6789/0 conn(0x7f642c06f7a0 legacy=0x7f642c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:37.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.398+0000 7f642b7fe640 1 -- 172.21.15.89:0/323904459 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f642c0c7390 con 0x7f642c071110 2023-10-30T12:43:37.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.398+0000 7f642b7fe640 1 --2- 172.21.15.89:0/323904459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f642c071110 0x7f642c158110 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f642c06f1c0 tx=0x7f642401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0088da4c2f72274 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:37.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.398+0000 7f642a7fc640 1 -- 172.21.15.89:0/323904459 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64240a89f0 con 0x7f642c071110 2023-10-30T12:43:37.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.398+0000 7f642a7fc640 1 -- 172.21.15.89:0/323904459 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64240a8b50 con 0x7f642c071110 2023-10-30T12:43:37.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.398+0000 7f642a7fc640 1 -- 172.21.15.89:0/323904459 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64240b0440 con 0x7f642c071110 2023-10-30T12:43:37.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.402+0000 7f6431846640 1 -- 172.21.15.89:0/323904459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f642c071110 msgr2=0x7f642c158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:37.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.402+0000 7f6431846640 1 --2- 172.21.15.89:0/323904459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f642c071110 0x7f642c158110 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f642c06f1c0 tx=0x7f642401ecf0 comp rx=0 tx=0).stop 2023-10-30T12:43:37.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.402+0000 7f6431846640 1 -- 172.21.15.89:0/323904459 shutdown_connections 2023-10-30T12:43:37.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.402+0000 7f6431846640 1 --2- 172.21.15.89:0/323904459 >> v2:172.21.15.192:3300/0 conn(0x7f642c0c6bd0 0x7f642c06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:37.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.402+0000 7f6431846640 1 --2- 172.21.15.89:0/323904459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f642c071110 0x7f642c158110 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:37.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.402+0000 7f6431846640 1 -- 172.21.15.89:0/323904459 >> 172.21.15.89:0/323904459 conn(0x7f642c0c7910 msgr2=0x7f642c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:37.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.402+0000 7f6431846640 1 -- 172.21.15.89:0/323904459 shutdown_connections 2023-10-30T12:43:37.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.402+0000 7f6431846640 1 -- 172.21.15.89:0/323904459 wait complete. 2023-10-30T12:43:37.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.402+0000 7f6431846640 1 Processor -- start 2023-10-30T12:43:37.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.402+0000 7f6431846640 1 -- start start 2023-10-30T12:43:37.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.402+0000 7f6431846640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f642c06f7a0 0x7f642c1df420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:37.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.402+0000 7f6431846640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f642c071110 0x7f642c1efd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:37.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.402+0000 7f6431846640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f642c0c6bd0 0x7f642c1570e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:37.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.402+0000 7f6431846640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c1e3f90 con 0x7f642c0c6bd0 2023-10-30T12:43:37.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.402+0000 7f6431846640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c152fe0 con 0x7f642c06f7a0 2023-10-30T12:43:37.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.402+0000 7f6431846640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f642c1535a0 con 0x7f642c071110 2023-10-30T12:43:37.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.402+0000 7f642b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f642c0c6bd0 0x7f642c1570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:37.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.402+0000 7f642b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f642c0c6bd0 0x7f642c1570e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51956/0 (socket says 172.21.15.89:51956) 2023-10-30T12:43:37.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.402+0000 7f642b7fe640 1 -- 172.21.15.89:0/3274114601 learned_addr learned my addr 172.21.15.89:0/3274114601 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:43:37.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.402+0000 7f642affd640 1 --2- 172.21.15.89:0/3274114601 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f642c06f7a0 0x7f642c1df420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:37.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.402+0000 7f64227fc640 1 --2- 172.21.15.89:0/3274114601 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f642c071110 0x7f642c1efd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:37.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.402+0000 7f642b7fe640 1 -- 172.21.15.89:0/3274114601 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f642c071110 msgr2=0x7f642c1efd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:37.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.402+0000 7f642b7fe640 1 --2- 172.21.15.89:0/3274114601 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f642c071110 0x7f642c1efd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:37.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.402+0000 7f642b7fe640 1 -- 172.21.15.89:0/3274114601 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f642c06f7a0 msgr2=0x7f642c1df420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:37.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.402+0000 7f642b7fe640 1 --2- 172.21.15.89:0/3274114601 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f642c06f7a0 0x7f642c1df420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:37.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.402+0000 7f642b7fe640 1 -- 172.21.15.89:0/3274114601 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64240094d0 con 0x7f642c0c6bd0 2023-10-30T12:43:37.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.402+0000 7f64227fc640 1 --2- 172.21.15.89:0/3274114601 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f642c071110 0x7f642c1efd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:43:37.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.406+0000 7f642b7fe640 1 --2- 172.21.15.89:0/3274114601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f642c0c6bd0 0x7f642c1570e0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f6424009930 tx=0x7f642401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:37.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.406+0000 7f6428ff9640 1 -- 172.21.15.89:0/3274114601 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64240b0de0 con 0x7f642c0c6bd0 2023-10-30T12:43:37.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.406+0000 7f6428ff9640 1 -- 172.21.15.89:0/3274114601 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64240bb460 con 0x7f642c0c6bd0 2023-10-30T12:43:37.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.406+0000 7f6428ff9640 1 -- 172.21.15.89:0/3274114601 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64240bb810 con 0x7f642c0c6bd0 2023-10-30T12:43:37.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.406+0000 7f6431846640 1 -- 172.21.15.89:0/3274114601 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f642c1f02b0 con 0x7f642c0c6bd0 2023-10-30T12:43:37.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.406+0000 7f6431846640 1 -- 172.21.15.89:0/3274114601 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f642c1e41e0 con 0x7f642c0c6bd0 2023-10-30T12:43:37.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.406+0000 7f6431846640 1 -- 172.21.15.89:0/3274114601 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f642c153730 con 0x7f642c0c6bd0 2023-10-30T12:43:37.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.406+0000 7f6428ff9640 1 -- 172.21.15.89:0/3274114601 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f64240b83f0 con 0x7f642c0c6bd0 2023-10-30T12:43:37.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.406+0000 7f6428ff9640 1 --2- 172.21.15.89:0/3274114601 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f640c0664a0 0x7f640c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:37.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.406+0000 7f6428ff9640 1 -- 172.21.15.89:0/3274114601 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6406+0+0 (secure 0 0 0) 0x7f6424121c30 con 0x7f642c0c6bd0 2023-10-30T12:43:37.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.406+0000 7f642affd640 1 --2- 172.21.15.89:0/3274114601 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f640c0664a0 0x7f640c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:37.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.410+0000 7f642affd640 1 --2- 172.21.15.89:0/3274114601 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f640c0664a0 0x7f640c068960 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f641c007fd0 tx=0x7f641c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:37.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.410+0000 7f6428ff9640 1 -- 172.21.15.89:0/3274114601 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f64240eee90 con 0x7f642c0c6bd0 2023-10-30T12:43:37.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.594+0000 7f6431846640 1 -- 172.21.15.89:0/3274114601 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f642c04e170 con 0x7f642c0c6bd0 2023-10-30T12:43:37.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.594+0000 7f6428ff9640 1 -- 172.21.15.89:0/3274114601 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5734 (secure 0 0 0) 0x7f64240f24b0 con 0x7f642c0c6bd0 2023-10-30T12:43:37.591 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:43:37.591 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:43:36.177060+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:43:37.592 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 73 2023-10-30T12:43:37.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.594+0000 7f6431846640 1 -- 172.21.15.89:0/3274114601 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f640c0664a0 msgr2=0x7f640c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:37.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.594+0000 7f6431846640 1 --2- 172.21.15.89:0/3274114601 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f640c0664a0 0x7f640c068960 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f641c007fd0 tx=0x7f641c009040 comp rx=0 tx=0).stop 2023-10-30T12:43:37.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.594+0000 7f6431846640 1 -- 172.21.15.89:0/3274114601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f642c0c6bd0 msgr2=0x7f642c1570e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:37.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.594+0000 7f6431846640 1 --2- 172.21.15.89:0/3274114601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f642c0c6bd0 0x7f642c1570e0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f6424009930 tx=0x7f642401ba90 comp rx=0 tx=0).stop 2023-10-30T12:43:37.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.594+0000 7f6431846640 1 -- 172.21.15.89:0/3274114601 shutdown_connections 2023-10-30T12:43:37.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.594+0000 7f6431846640 1 --2- 172.21.15.89:0/3274114601 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f640c0664a0 0x7f640c068960 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:37.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.594+0000 7f6431846640 1 --2- 172.21.15.89:0/3274114601 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f642c0c6bd0 0x7f642c1570e0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:37.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.594+0000 7f6431846640 1 --2- 172.21.15.89:0/3274114601 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f642c071110 0x7f642c1efd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:37.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.594+0000 7f6431846640 1 --2- 172.21.15.89:0/3274114601 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f642c06f7a0 0x7f642c1df420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:37.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.594+0000 7f6431846640 1 -- 172.21.15.89:0/3274114601 >> 172.21.15.89:0/3274114601 conn(0x7f642c0c7910 msgr2=0x7f642c0c92d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:37.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.594+0000 7f6431846640 1 -- 172.21.15.89:0/3274114601 shutdown_connections 2023-10-30T12:43:37.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.594+0000 7f6431846640 1 -- 172.21.15.89:0/3274114601 wait complete. 2023-10-30T12:43:37.611 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-10-30T12:43:37.611 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:43:37.611 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:43:37.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.750+0000 7fecc8d65640 1 Processor -- start 2023-10-30T12:43:37.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.750+0000 7fecc8d65640 1 -- start start 2023-10-30T12:43:37.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.750+0000 7fecc8d65640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fecc40c6bd0 0x7fecc406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:37.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.750+0000 7fecc8d65640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc4066cf0 con 0x7fecc406f7a0 2023-10-30T12:43:37.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.750+0000 7fecc8d65640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc40c70d0 con 0x7fecc4071110 2023-10-30T12:43:37.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.750+0000 7fecc8d65640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecc40700c0 con 0x7fecc40c6bd0 2023-10-30T12:43:37.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.750+0000 7fecc259c640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fecc406f7a0 0x7fecc4069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:50916/0 (socket says 172.21.15.89:50916) 2023-10-30T12:43:37.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.750+0000 7fecc259c640 1 -- 172.21.15.89:0/3277234891 learned_addr learned my addr 172.21.15.89:0/3277234891 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:43:37.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.750+0000 7fecb9d9b640 1 --2- 172.21.15.89:0/3277234891 >> v2:172.21.15.89:3300/0 conn(0x7fecc40c6bd0 0x7fecc406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:37.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.754+0000 7fecc1d9b640 1 -- 172.21.15.89:0/3277234891 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 532401778 0 0) 0x7fecc4066cf0 con 0x7fecc406f7a0 2023-10-30T12:43:37.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.754+0000 7fecc1d9b640 1 -- 172.21.15.89:0/3277234891 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecb0003530 con 0x7fecc406f7a0 2023-10-30T12:43:37.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.754+0000 7fecb9d9b640 1 -- 172.21.15.89:0/3277234891 >> v1:172.21.15.192:6789/0 conn(0x7fecc4071110 legacy=0x7fecc4158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:37.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.754+0000 7fecb9d9b640 1 -- 172.21.15.89:0/3277234891 >> v1:172.21.15.89:6789/0 conn(0x7fecc406f7a0 legacy=0x7fecc4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:37.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.754+0000 7fecb9d9b640 1 -- 172.21.15.89:0/3277234891 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc40c7390 con 0x7fecc40c6bd0 2023-10-30T12:43:37.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.754+0000 7fecb9d9b640 1 --2- 172.21.15.89:0/3277234891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecc40c6bd0 0x7fecc406fb80 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fecac009d90 tx=0x7fecac01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c50e91d16aee052 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:37.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.754+0000 7fecc1d9b640 1 -- 172.21.15.89:0/3277234891 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecac0af600 con 0x7fecc40c6bd0 2023-10-30T12:43:37.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.754+0000 7fecc1d9b640 1 -- 172.21.15.89:0/3277234891 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecac0ab050 con 0x7fecc40c6bd0 2023-10-30T12:43:37.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.754+0000 7fecc1d9b640 1 -- 172.21.15.89:0/3277234891 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecac0af980 con 0x7fecc40c6bd0 2023-10-30T12:43:37.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.754+0000 7fecc8d65640 1 -- 172.21.15.89:0/3277234891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecc40c6bd0 msgr2=0x7fecc406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:37.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.754+0000 7fecc8d65640 1 --2- 172.21.15.89:0/3277234891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecc40c6bd0 0x7fecc406fb80 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fecac009d90 tx=0x7fecac01f130 comp rx=0 tx=0).stop 2023-10-30T12:43:37.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.754+0000 7fecc8d65640 1 -- 172.21.15.89:0/3277234891 shutdown_connections 2023-10-30T12:43:37.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.754+0000 7fecc8d65640 1 --2- 172.21.15.89:0/3277234891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecc40c6bd0 0x7fecc406fb80 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:37.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.754+0000 7fecc8d65640 1 -- 172.21.15.89:0/3277234891 >> 172.21.15.89:0/3277234891 conn(0x7fecc40c7910 msgr2=0x7fecc40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:37.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.754+0000 7fecc8d65640 1 -- 172.21.15.89:0/3277234891 shutdown_connections 2023-10-30T12:43:37.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.754+0000 7fecc8d65640 1 -- 172.21.15.89:0/3277234891 wait complete. 2023-10-30T12:43:37.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.758+0000 7fecc8d65640 1 Processor -- start 2023-10-30T12:43:37.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.758+0000 7fecc8d65640 1 -- start start 2023-10-30T12:43:37.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.758+0000 7fecc8d65640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fecc406f7a0 0x7fecc414edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:37.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.758+0000 7fecc8d65640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecc4071110 0x7fecc4154590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:37.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.758+0000 7fecc8d65640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecc40c6bd0 0x7fecc4154ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:37.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.758+0000 7fecc8d65640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc41e6890 con 0x7fecc40c6bd0 2023-10-30T12:43:37.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.758+0000 7fecc8d65640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc414f300 con 0x7fecc406f7a0 2023-10-30T12:43:37.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.758+0000 7fecc8d65640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecc41e61a0 con 0x7fecc4071110 2023-10-30T12:43:37.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.758+0000 7fecc2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecc40c6bd0 0x7fecc4154ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:37.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.758+0000 7fecc2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecc40c6bd0 0x7fecc4154ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51970/0 (socket says 172.21.15.89:51970) 2023-10-30T12:43:37.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.758+0000 7fecc2d9d640 1 -- 172.21.15.89:0/2420821559 learned_addr learned my addr 172.21.15.89:0/2420821559 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:43:37.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.758+0000 7fecc259c640 1 --2- 172.21.15.89:0/2420821559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fecc406f7a0 0x7fecc414edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:37.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.758+0000 7fecb9d9b640 1 --2- 172.21.15.89:0/2420821559 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecc4071110 0x7fecc4154590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:37.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.758+0000 7fecc2d9d640 1 -- 172.21.15.89:0/2420821559 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecc4071110 msgr2=0x7fecc4154590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:37.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.758+0000 7fecc2d9d640 1 --2- 172.21.15.89:0/2420821559 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecc4071110 0x7fecc4154590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:37.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.758+0000 7fecc2d9d640 1 -- 172.21.15.89:0/2420821559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fecc406f7a0 msgr2=0x7fecc414edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:37.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.758+0000 7fecc2d9d640 1 --2- 172.21.15.89:0/2420821559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fecc406f7a0 0x7fecc414edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:37.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.758+0000 7fecc2d9d640 1 -- 172.21.15.89:0/2420821559 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecac009a70 con 0x7fecc40c6bd0 2023-10-30T12:43:37.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.758+0000 7fecc2d9d640 1 --2- 172.21.15.89:0/2420821559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecc40c6bd0 0x7fecc4154ad0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fecbc00c370 tx=0x7fecbc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:37.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.762+0000 7fecbb7fe640 1 -- 172.21.15.89:0/2420821559 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecbc005220 con 0x7fecc40c6bd0 2023-10-30T12:43:37.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.762+0000 7fecbb7fe640 1 -- 172.21.15.89:0/2420821559 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecbc005380 con 0x7fecc40c6bd0 2023-10-30T12:43:37.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.762+0000 7fecc8d65640 1 -- 172.21.15.89:0/2420821559 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecc40c7cf0 con 0x7fecc40c6bd0 2023-10-30T12:43:37.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.762+0000 7fecbb7fe640 1 -- 172.21.15.89:0/2420821559 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecbc007270 con 0x7fecc40c6bd0 2023-10-30T12:43:37.762 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.762+0000 7fecc8d65640 1 -- 172.21.15.89:0/2420821559 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecc41ef380 con 0x7fecc40c6bd0 2023-10-30T12:43:37.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.762+0000 7fecc8d65640 1 -- 172.21.15.89:0/2420821559 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecc404e170 con 0x7fecc40c6bd0 2023-10-30T12:43:37.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.762+0000 7fecbb7fe640 1 -- 172.21.15.89:0/2420821559 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fecbc0073d0 con 0x7fecc40c6bd0 2023-10-30T12:43:37.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.762+0000 7fecbb7fe640 1 --2- 172.21.15.89:0/2420821559 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7feca40664d0 0x7feca4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:37.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.762+0000 7fecbb7fe640 1 -- 172.21.15.89:0/2420821559 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6406+0+0 (secure 0 0 0) 0x7fecbc083490 con 0x7fecc40c6bd0 2023-10-30T12:43:37.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.762+0000 7fecc259c640 1 --2- 172.21.15.89:0/2420821559 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7feca40664d0 0x7feca4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:37.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.766+0000 7fecc259c640 1 --2- 172.21.15.89:0/2420821559 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7feca40664d0 0x7feca4068990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fecb40011f0 tx=0x7fecb4007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:37.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.766+0000 7fecbb7fe640 1 -- 172.21.15.89:0/2420821559 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fecbc050660 con 0x7fecc40c6bd0 2023-10-30T12:43:37.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.946+0000 7fecc8d65640 1 -- 172.21.15.89:0/2420821559 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fecc406fea0 con 0x7fecc40c6bd0 2023-10-30T12:43:37.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.946+0000 7fecbb7fe640 1 -- 172.21.15.89:0/2420821559 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5734 (secure 0 0 0) 0x7fecbc053c80 con 0x7fecc40c6bd0 2023-10-30T12:43:37.945 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:43:37.946 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:43:36.177060+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:43:37.946 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 73 2023-10-30T12:43:37.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.950+0000 7fecc8d65640 1 -- 172.21.15.89:0/2420821559 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7feca40664d0 msgr2=0x7feca4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:37.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.950+0000 7fecc8d65640 1 --2- 172.21.15.89:0/2420821559 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7feca40664d0 0x7feca4068990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fecb40011f0 tx=0x7fecb4007d40 comp rx=0 tx=0).stop 2023-10-30T12:43:37.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.950+0000 7fecc8d65640 1 -- 172.21.15.89:0/2420821559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecc40c6bd0 msgr2=0x7fecc4154ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:37.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.950+0000 7fecc8d65640 1 --2- 172.21.15.89:0/2420821559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecc40c6bd0 0x7fecc4154ad0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fecbc00c370 tx=0x7fecbc00c830 comp rx=0 tx=0).stop 2023-10-30T12:43:37.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.950+0000 7fecc8d65640 1 -- 172.21.15.89:0/2420821559 shutdown_connections 2023-10-30T12:43:37.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.950+0000 7fecc8d65640 1 --2- 172.21.15.89:0/2420821559 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7feca40664d0 0x7feca4068990 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:37.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.950+0000 7fecc8d65640 1 --2- 172.21.15.89:0/2420821559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecc40c6bd0 0x7fecc4154ad0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:37.948 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.950+0000 7fecc8d65640 1 --2- 172.21.15.89:0/2420821559 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fecc4071110 0x7fecc4154590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:37.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.950+0000 7fecc8d65640 1 --2- 172.21.15.89:0/2420821559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fecc406f7a0 0x7fecc414edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:37.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.950+0000 7fecc8d65640 1 -- 172.21.15.89:0/2420821559 >> 172.21.15.89:0/2420821559 conn(0x7fecc40c7910 msgr2=0x7fecc40c2640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:37.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.950+0000 7fecc8d65640 1 -- 172.21.15.89:0/2420821559 shutdown_connections 2023-10-30T12:43:37.949 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:37.950+0000 7fecc8d65640 1 -- 172.21.15.89:0/2420821559 wait complete. 2023-10-30T12:43:37.966 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 73, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:50.729245+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 73, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5139": { "addr": "172.21.15.89:6837/951276252", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6836", "nonce": 951276252, "type": "v2" }, { "addr": "172.21.15.89:6837", "nonce": 951276252, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5139, "incarnation": 33, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 29 }, "gid_5575": { "addr": "172.21.15.192:6835/501291700", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6834", "nonce": 501291700, "type": "v2" }, { "addr": "172.21.15.192:6835", "nonce": 501291700, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5575, "incarnation": 63, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 37 }, "gid_5728": { "addr": "172.21.15.192:6839/3991891597", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6838", "nonce": 3991891597, "type": "v2" }, { "addr": "172.21.15.192:6839", "nonce": 3991891597, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5728, "incarnation": 73, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 62, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:43:36.177060+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5575, "mds_1": 5139, "mds_2": 5728 } } } ], "standbys": [ { "addr": "172.21.15.89:6834/2909790173", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6832", "nonce": 2909790173, "type": "v2" }, { "addr": "172.21.15.89:6834", "nonce": 2909790173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.89:6835/2387357615", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6833", "nonce": 2387357615, "type": "v2" }, { "addr": "172.21.15.89:6835", "nonce": 2387357615, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4239, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:43:39.967 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:43:40.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.118+0000 7efe982d6640 1 Processor -- start 2023-10-30T12:43:40.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.118+0000 7efe982d6640 1 -- start start 2023-10-30T12:43:40.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.118+0000 7efe982d6640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efe900bfec0 0x7efe900c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:40.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.118+0000 7efe982d6640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7efe9006a720 0x7efe9006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:40.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.118+0000 7efe982d6640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe90071290 con 0x7efe90159e60 2023-10-30T12:43:40.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.118+0000 7efe982d6640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe90071570 con 0x7efe900bfec0 2023-10-30T12:43:40.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.118+0000 7efe982d6640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe90071850 con 0x7efe9006a720 2023-10-30T12:43:40.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.118+0000 7efe95871640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efe900bfec0 0x7efe900c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:40.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.118+0000 7efe95871640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7efe900bfec0 0x7efe900c1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50424/0 (socket says 172.21.15.89:50424) 2023-10-30T12:43:40.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.118+0000 7efe95871640 1 -- 172.21.15.89:0/3645844261 learned_addr learned my addr 172.21.15.89:0/3645844261 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:43:40.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.118+0000 7efe96072640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7efe90159e60 0x7efe9006f540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48728/0 (socket says 172.21.15.89:48728) 2023-10-30T12:43:40.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.118+0000 7efe96873640 1 --2- 172.21.15.89:0/3645844261 >> v2:172.21.15.192:3300/0 conn(0x7efe9006a720 0x7efe9006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:40.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.122+0000 7efe95871640 1 -- 172.21.15.89:0/3645844261 >> v2:172.21.15.192:3300/0 conn(0x7efe9006a720 msgr2=0x7efe9006abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:40.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.122+0000 7efe95871640 1 --2- 172.21.15.89:0/3645844261 >> v2:172.21.15.192:3300/0 conn(0x7efe9006a720 0x7efe9006abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:40.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.122+0000 7efe95871640 1 -- 172.21.15.89:0/3645844261 >> v1:172.21.15.89:6789/0 conn(0x7efe90159e60 legacy=0x7efe9006f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:40.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.122+0000 7efe95871640 1 -- 172.21.15.89:0/3645844261 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe90071b30 con 0x7efe900bfec0 2023-10-30T12:43:40.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.122+0000 7efe95070640 1 -- 172.21.15.89:0/3645844261 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130880567 0 0) 0x7efe90071290 con 0x7efe90159e60 2023-10-30T12:43:40.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.122+0000 7efe95871640 1 --2- 172.21.15.89:0/3645844261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe900bfec0 0x7efe900c1a40 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7efe90071e10 tx=0x7efe800213c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30d440013a1bb8ac server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:40.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.122+0000 7efe95070640 1 -- 172.21.15.89:0/3645844261 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe800b1600 con 0x7efe900bfec0 2023-10-30T12:43:40.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.122+0000 7efe95070640 1 -- 172.21.15.89:0/3645844261 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe800ad050 con 0x7efe900bfec0 2023-10-30T12:43:40.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.122+0000 7efe95070640 1 -- 172.21.15.89:0/3645844261 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe800b1980 con 0x7efe900bfec0 2023-10-30T12:43:40.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.122+0000 7efe982d6640 1 -- 172.21.15.89:0/3645844261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe900bfec0 msgr2=0x7efe900c1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:40.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.122+0000 7efe982d6640 1 --2- 172.21.15.89:0/3645844261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe900bfec0 0x7efe900c1a40 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7efe90071e10 tx=0x7efe800213c0 comp rx=0 tx=0).stop 2023-10-30T12:43:40.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.122+0000 7efe982d6640 1 -- 172.21.15.89:0/3645844261 shutdown_connections 2023-10-30T12:43:40.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.122+0000 7efe982d6640 1 --2- 172.21.15.89:0/3645844261 >> v2:172.21.15.192:3300/0 conn(0x7efe9006a720 0x7efe9006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:40.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.122+0000 7efe982d6640 1 --2- 172.21.15.89:0/3645844261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe900bfec0 0x7efe900c1a40 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:40.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.122+0000 7efe982d6640 1 -- 172.21.15.89:0/3645844261 >> 172.21.15.89:0/3645844261 conn(0x7efe900c7910 msgr2=0x7efe900c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:40.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.126+0000 7efe982d6640 1 -- 172.21.15.89:0/3645844261 shutdown_connections 2023-10-30T12:43:40.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.126+0000 7efe982d6640 1 -- 172.21.15.89:0/3645844261 wait complete. 2023-10-30T12:43:40.126 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.126+0000 7efe982d6640 1 Processor -- start 2023-10-30T12:43:40.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.126+0000 7efe982d6640 1 -- start start 2023-10-30T12:43:40.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.126+0000 7efe982d6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe9006a720 0x7efe900c61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:40.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.126+0000 7efe982d6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efe900bfec0 0x7efe9014eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:40.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.126+0000 7efe982d6640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efe90159e60 0x7efe9014d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:40.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.126+0000 7efe982d6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe90156990 con 0x7efe9006a720 2023-10-30T12:43:40.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.126+0000 7efe982d6640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe901eb470 con 0x7efe900bfec0 2023-10-30T12:43:40.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.126+0000 7efe982d6640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe901eb980 con 0x7efe90159e60 2023-10-30T12:43:40.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.126+0000 7efe96072640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe9006a720 0x7efe900c61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:40.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.126+0000 7efe96072640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe9006a720 0x7efe900c61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50430/0 (socket says 172.21.15.89:50430) 2023-10-30T12:43:40.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.126+0000 7efe96072640 1 -- 172.21.15.89:0/368804512 learned_addr learned my addr 172.21.15.89:0/368804512 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:43:40.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.126+0000 7efe95871640 1 --2- 172.21.15.89:0/368804512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efe900bfec0 0x7efe9014eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:40.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.126+0000 7efe96873640 1 --2- 172.21.15.89:0/368804512 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efe90159e60 0x7efe9014d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:40.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.126+0000 7efe96072640 1 -- 172.21.15.89:0/368804512 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efe90159e60 msgr2=0x7efe9014d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:40.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.126+0000 7efe96072640 1 --2- 172.21.15.89:0/368804512 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efe90159e60 0x7efe9014d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:40.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.126+0000 7efe96072640 1 -- 172.21.15.89:0/368804512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efe900bfec0 msgr2=0x7efe9014eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:40.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.126+0000 7efe96072640 1 --2- 172.21.15.89:0/368804512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efe900bfec0 0x7efe9014eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:40.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.126+0000 7efe96072640 1 -- 172.21.15.89:0/368804512 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe8000b770 con 0x7efe9006a720 2023-10-30T12:43:40.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.126+0000 7efe96072640 1 --2- 172.21.15.89:0/368804512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe9006a720 0x7efe900c61e0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7efe8c00c4b0 tx=0x7efe8c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:40.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.126+0000 7efe86ffd640 1 -- 172.21.15.89:0/368804512 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8c0074e0 con 0x7efe9006a720 2023-10-30T12:43:40.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.126+0000 7efe86ffd640 1 -- 172.21.15.89:0/368804512 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe8c007660 con 0x7efe9006a720 2023-10-30T12:43:40.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.126+0000 7efe982d6640 1 -- 172.21.15.89:0/368804512 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe900c6720 con 0x7efe9006a720 2023-10-30T12:43:40.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.126+0000 7efe86ffd640 1 -- 172.21.15.89:0/368804512 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8c004a30 con 0x7efe9006a720 2023-10-30T12:43:40.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.126+0000 7efe982d6640 1 -- 172.21.15.89:0/368804512 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe900c5420 con 0x7efe9006a720 2023-10-30T12:43:40.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.130+0000 7efe982d6640 1 -- 172.21.15.89:0/368804512 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe9006cc30 con 0x7efe9006a720 2023-10-30T12:43:40.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.130+0000 7efe86ffd640 1 -- 172.21.15.89:0/368804512 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efe8c004b90 con 0x7efe9006a720 2023-10-30T12:43:40.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.130+0000 7efe86ffd640 1 --2- 172.21.15.89:0/368804512 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7efe740664d0 0x7efe74068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:40.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.130+0000 7efe86ffd640 1 -- 172.21.15.89:0/368804512 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6406+0+0 (secure 0 0 0) 0x7efe8c083670 con 0x7efe9006a720 2023-10-30T12:43:40.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.130+0000 7efe95871640 1 --2- 172.21.15.89:0/368804512 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7efe740664d0 0x7efe74068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:40.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.130+0000 7efe95871640 1 --2- 172.21.15.89:0/368804512 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7efe740664d0 0x7efe74068990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7efe800b3000 tx=0x7efe800ac000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:40.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.134+0000 7efe86ffd640 1 -- 172.21.15.89:0/368804512 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efe8c050790 con 0x7efe9006a720 2023-10-30T12:43:40.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.318+0000 7efe982d6640 1 -- 172.21.15.89:0/368804512 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe9004e170 con 0x7efe9006a720 2023-10-30T12:43:40.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.318+0000 7efe86ffd640 1 -- 172.21.15.89:0/368804512 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5734 (secure 0 0 0) 0x7efe8c053db0 con 0x7efe9006a720 2023-10-30T12:43:40.316 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:43:40.316 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:43:36.177060+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:43:40.316 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 73 2023-10-30T12:43:40.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.318+0000 7efe982d6640 1 -- 172.21.15.89:0/368804512 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7efe740664d0 msgr2=0x7efe74068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:40.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.318+0000 7efe982d6640 1 --2- 172.21.15.89:0/368804512 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7efe740664d0 0x7efe74068990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7efe800b3000 tx=0x7efe800ac000 comp rx=0 tx=0).stop 2023-10-30T12:43:40.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.318+0000 7efe982d6640 1 -- 172.21.15.89:0/368804512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe9006a720 msgr2=0x7efe900c61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:40.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.318+0000 7efe982d6640 1 --2- 172.21.15.89:0/368804512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe9006a720 0x7efe900c61e0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7efe8c00c4b0 tx=0x7efe8c00c970 comp rx=0 tx=0).stop 2023-10-30T12:43:40.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.318+0000 7efe982d6640 1 -- 172.21.15.89:0/368804512 shutdown_connections 2023-10-30T12:43:40.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.318+0000 7efe982d6640 1 --2- 172.21.15.89:0/368804512 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7efe740664d0 0x7efe74068990 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:40.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.318+0000 7efe982d6640 1 --2- 172.21.15.89:0/368804512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efe900bfec0 0x7efe9014eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:40.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.318+0000 7efe982d6640 1 --2- 172.21.15.89:0/368804512 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efe90159e60 0x7efe9014d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:40.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.318+0000 7efe982d6640 1 --2- 172.21.15.89:0/368804512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe9006a720 0x7efe900c61e0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:40.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.318+0000 7efe982d6640 1 -- 172.21.15.89:0/368804512 >> 172.21.15.89:0/368804512 conn(0x7efe900c7910 msgr2=0x7efe9014f770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:40.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.318+0000 7efe982d6640 1 -- 172.21.15.89:0/368804512 shutdown_connections 2023-10-30T12:43:40.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:40.318+0000 7efe982d6640 1 -- 172.21.15.89:0/368804512 wait complete. 2023-10-30T12:43:40.336 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:43:42.337 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:43:42.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.490+0000 7f0865a6b640 1 Processor -- start 2023-10-30T12:43:42.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.490+0000 7f0865a6b640 1 -- start start 2023-10-30T12:43:42.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.490+0000 7f0865a6b640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0860159e60 0x7f086006f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:42.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.490+0000 7f0865a6b640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f086006a720 0x7f086006ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:42.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.490+0000 7f0865a6b640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f086006fa80 con 0x7f08600bfec0 2023-10-30T12:43:42.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.490+0000 7f0865a6b640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0860071290 con 0x7f0860159e60 2023-10-30T12:43:42.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.490+0000 7f0865a6b640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0860071570 con 0x7f086006a720 2023-10-30T12:43:42.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.490+0000 7f085f7fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0860159e60 0x7f086006f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:42.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.494+0000 7f085f7fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0860159e60 0x7f086006f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50432/0 (socket says 172.21.15.89:50432) 2023-10-30T12:43:42.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.494+0000 7f085f7fe640 1 -- 172.21.15.89:0/2695755313 learned_addr learned my addr 172.21.15.89:0/2695755313 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:43:42.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.494+0000 7f085ffff640 1 --2- 172.21.15.89:0/2695755313 >> v2:172.21.15.192:3300/0 conn(0x7f086006a720 0x7f086006ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:42.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.494+0000 7f085f7fe640 1 -- 172.21.15.89:0/2695755313 >> v2:172.21.15.192:3300/0 conn(0x7f086006a720 msgr2=0x7f086006ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:42.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.494+0000 7f085f7fe640 1 --2- 172.21.15.89:0/2695755313 >> v2:172.21.15.192:3300/0 conn(0x7f086006a720 0x7f086006ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:42.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.494+0000 7f085f7fe640 1 -- 172.21.15.89:0/2695755313 >> v1:172.21.15.89:6789/0 conn(0x7f08600bfec0 legacy=0x7f08600c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:42.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.494+0000 7f085f7fe640 1 -- 172.21.15.89:0/2695755313 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0860071850 con 0x7f0860159e60 2023-10-30T12:43:42.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.494+0000 7f085f7fe640 1 --2- 172.21.15.89:0/2695755313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0860159e60 0x7f086006f540 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f086006f4d0 tx=0x7f085001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30f1e4ba6df842ea server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:42.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.494+0000 7f085e7fc640 1 -- 172.21.15.89:0/2695755313 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08500a89f0 con 0x7f0860159e60 2023-10-30T12:43:42.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.494+0000 7f085e7fc640 1 -- 172.21.15.89:0/2695755313 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08500a8b50 con 0x7f0860159e60 2023-10-30T12:43:42.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.494+0000 7f085e7fc640 1 -- 172.21.15.89:0/2695755313 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08500b0440 con 0x7f0860159e60 2023-10-30T12:43:42.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.498+0000 7f0865a6b640 1 -- 172.21.15.89:0/2695755313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0860159e60 msgr2=0x7f086006f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:42.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.498+0000 7f0865a6b640 1 --2- 172.21.15.89:0/2695755313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0860159e60 0x7f086006f540 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f086006f4d0 tx=0x7f085001ecf0 comp rx=0 tx=0).stop 2023-10-30T12:43:42.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.498+0000 7f0865a6b640 1 -- 172.21.15.89:0/2695755313 shutdown_connections 2023-10-30T12:43:42.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.498+0000 7f0865a6b640 1 --2- 172.21.15.89:0/2695755313 >> v2:172.21.15.192:3300/0 conn(0x7f086006a720 0x7f086006ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:42.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.498+0000 7f0865a6b640 1 --2- 172.21.15.89:0/2695755313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0860159e60 0x7f086006f540 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:42.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.498+0000 7f0865a6b640 1 -- 172.21.15.89:0/2695755313 >> 172.21.15.89:0/2695755313 conn(0x7f08600c7910 msgr2=0x7f08600c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:42.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.498+0000 7f0865a6b640 1 -- 172.21.15.89:0/2695755313 shutdown_connections 2023-10-30T12:43:42.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.498+0000 7f0865a6b640 1 -- 172.21.15.89:0/2695755313 wait complete. 2023-10-30T12:43:42.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.498+0000 7f0865a6b640 1 Processor -- start 2023-10-30T12:43:42.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.498+0000 7f0865a6b640 1 -- start start 2023-10-30T12:43:42.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.498+0000 7f0865a6b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f086006a720 0x7f08601ef3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:42.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.498+0000 7f0865a6b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08600bfec0 0x7f086014c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:42.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.498+0000 7f0865a6b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0860159e60 0x7f08601f27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:42.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.498+0000 7f0865a6b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0860150af0 con 0x7f086006a720 2023-10-30T12:43:42.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.498+0000 7f0865a6b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0860156b00 con 0x7f08600bfec0 2023-10-30T12:43:42.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.498+0000 7f0865a6b640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08601ef8f0 con 0x7f0860159e60 2023-10-30T12:43:42.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.498+0000 7f085f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f086006a720 0x7f08601ef3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:42.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.498+0000 7f085f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f086006a720 0x7f08601ef3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50444/0 (socket says 172.21.15.89:50444) 2023-10-30T12:43:42.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.498+0000 7f085f7fe640 1 -- 172.21.15.89:0/832034439 learned_addr learned my addr 172.21.15.89:0/832034439 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:43:42.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.498+0000 7f085effd640 1 --2- 172.21.15.89:0/832034439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08600bfec0 0x7f086014c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:42.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.498+0000 7f085ffff640 1 --2- 172.21.15.89:0/832034439 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0860159e60 0x7f08601f27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:42.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.498+0000 7f085f7fe640 1 -- 172.21.15.89:0/832034439 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0860159e60 msgr2=0x7f08601f27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:42.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.498+0000 7f085f7fe640 1 --2- 172.21.15.89:0/832034439 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0860159e60 0x7f08601f27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:42.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.498+0000 7f085f7fe640 1 -- 172.21.15.89:0/832034439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08600bfec0 msgr2=0x7f086014c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:42.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.498+0000 7f085f7fe640 1 --2- 172.21.15.89:0/832034439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08600bfec0 0x7f086014c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:42.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.498+0000 7f085f7fe640 1 -- 172.21.15.89:0/832034439 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08500094d0 con 0x7f086006a720 2023-10-30T12:43:42.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.502+0000 7f085f7fe640 1 --2- 172.21.15.89:0/832034439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f086006a720 0x7f08601ef3b0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f085001c920 tx=0x7f085001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:42.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.502+0000 7f0847fff640 1 -- 172.21.15.89:0/832034439 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08500b0dd0 con 0x7f086006a720 2023-10-30T12:43:42.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.502+0000 7f0847fff640 1 -- 172.21.15.89:0/832034439 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08500bb460 con 0x7f086006a720 2023-10-30T12:43:42.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.502+0000 7f0865a6b640 1 -- 172.21.15.89:0/832034439 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08600ae570 con 0x7f086006a720 2023-10-30T12:43:42.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.502+0000 7f0847fff640 1 -- 172.21.15.89:0/832034439 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08500bb810 con 0x7f086006a720 2023-10-30T12:43:42.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.502+0000 7f0865a6b640 1 -- 172.21.15.89:0/832034439 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08600adfb0 con 0x7f086006a720 2023-10-30T12:43:42.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.502+0000 7f0865a6b640 1 -- 172.21.15.89:0/832034439 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f086004e170 con 0x7f086006a720 2023-10-30T12:43:42.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.502+0000 7f0847fff640 1 -- 172.21.15.89:0/832034439 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f08500b83f0 con 0x7f086006a720 2023-10-30T12:43:42.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.502+0000 7f0847fff640 1 --2- 172.21.15.89:0/832034439 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f084c0664a0 0x7f084c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:42.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.502+0000 7f0847fff640 1 -- 172.21.15.89:0/832034439 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6406+0+0 (secure 0 0 0) 0x7f0850121ac0 con 0x7f086006a720 2023-10-30T12:43:42.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.502+0000 7f085effd640 1 --2- 172.21.15.89:0/832034439 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f084c0664a0 0x7f084c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:42.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.506+0000 7f085effd640 1 --2- 172.21.15.89:0/832034439 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f084c0664a0 0x7f084c068960 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f08600b1160 tx=0x7f0854009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:42.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.506+0000 7f0847fff640 1 -- 172.21.15.89:0/832034439 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f08500eeb60 con 0x7f086006a720 2023-10-30T12:43:42.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.690+0000 7f0865a6b640 1 -- 172.21.15.89:0/832034439 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f086006adc0 con 0x7f086006a720 2023-10-30T12:43:42.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.690+0000 7f0847fff640 1 -- 172.21.15.89:0/832034439 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5734 (secure 0 0 0) 0x7f08500f2180 con 0x7f086006a720 2023-10-30T12:43:42.687 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:43:42.687 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:43:36.177060+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:43:42.688 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 73 2023-10-30T12:43:42.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.690+0000 7f0865a6b640 1 -- 172.21.15.89:0/832034439 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f084c0664a0 msgr2=0x7f084c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:42.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.690+0000 7f0865a6b640 1 --2- 172.21.15.89:0/832034439 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f084c0664a0 0x7f084c068960 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f08600b1160 tx=0x7f0854009040 comp rx=0 tx=0).stop 2023-10-30T12:43:42.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.690+0000 7f0865a6b640 1 -- 172.21.15.89:0/832034439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f086006a720 msgr2=0x7f08601ef3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:42.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.690+0000 7f0865a6b640 1 --2- 172.21.15.89:0/832034439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f086006a720 0x7f08601ef3b0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f085001c920 tx=0x7f085001ba90 comp rx=0 tx=0).stop 2023-10-30T12:43:42.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.690+0000 7f0865a6b640 1 -- 172.21.15.89:0/832034439 shutdown_connections 2023-10-30T12:43:42.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.690+0000 7f0865a6b640 1 --2- 172.21.15.89:0/832034439 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f084c0664a0 0x7f084c068960 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:42.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.690+0000 7f0865a6b640 1 --2- 172.21.15.89:0/832034439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08600bfec0 0x7f086014c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:42.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.690+0000 7f0865a6b640 1 --2- 172.21.15.89:0/832034439 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0860159e60 0x7f08601f27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:42.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.690+0000 7f0865a6b640 1 --2- 172.21.15.89:0/832034439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f086006a720 0x7f08601ef3b0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:42.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.690+0000 7f0865a6b640 1 -- 172.21.15.89:0/832034439 >> 172.21.15.89:0/832034439 conn(0x7f08600c7910 msgr2=0x7f08600ca120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:42.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.690+0000 7f0865a6b640 1 -- 172.21.15.89:0/832034439 shutdown_connections 2023-10-30T12:43:42.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:42.690+0000 7f0865a6b640 1 -- 172.21.15.89:0/832034439 wait complete. 2023-10-30T12:43:42.707 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:43:44.707 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:43:44.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.858+0000 7fcb48f65640 1 Processor -- start 2023-10-30T12:43:44.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.858+0000 7fcb48f65640 1 -- start start 2023-10-30T12:43:44.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.858+0000 7fcb48f65640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fcb440c6b80 0x7fcb440bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:44.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb48f65640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb4413f4a0 0x7fcb440c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:44.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb48f65640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb440c0a50 con 0x7fcb440c6f60 2023-10-30T12:43:44.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb48f65640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb4406f6e0 con 0x7fcb440c6b80 2023-10-30T12:43:44.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb48f65640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4406f9c0 con 0x7fcb4413f4a0 2023-10-30T12:43:44.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb41d9b640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fcb440c6f60 0x7fcb4413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:48754/0 (socket says 172.21.15.89:48754) 2023-10-30T12:43:44.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb41d9b640 1 -- 172.21.15.89:0/2791007225 learned_addr learned my addr 172.21.15.89:0/2791007225 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:43:44.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb42d9d640 1 --2- 172.21.15.89:0/2791007225 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb4413f4a0 0x7fcb440c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:44.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb4259c640 1 --2- 172.21.15.89:0/2791007225 >> v2:172.21.15.192:3300/0 conn(0x7fcb440c6b80 0x7fcb440bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:44.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb4159a640 1 -- 172.21.15.89:0/2791007225 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3552061793 0 0) 0x7fcb440c0a50 con 0x7fcb440c6f60 2023-10-30T12:43:44.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb4159a640 1 -- 172.21.15.89:0/2791007225 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb2c003530 con 0x7fcb440c6f60 2023-10-30T12:43:44.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb4159a640 1 -- 172.21.15.89:0/2791007225 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 294824484 0 0) 0x7fcb2c003530 con 0x7fcb440c6f60 2023-10-30T12:43:44.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb4159a640 1 -- 172.21.15.89:0/2791007225 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb4413f4a0 msgr2=0x7fcb440c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:44.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb4159a640 1 --2- 172.21.15.89:0/2791007225 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb4413f4a0 0x7fcb440c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:44.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb4159a640 1 -- 172.21.15.89:0/2791007225 >> v2:172.21.15.192:3300/0 conn(0x7fcb440c6b80 msgr2=0x7fcb440bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:44.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb4159a640 1 --2- 172.21.15.89:0/2791007225 >> v2:172.21.15.192:3300/0 conn(0x7fcb440c6b80 0x7fcb440bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:44.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb4159a640 1 -- 172.21.15.89:0/2791007225 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4406fca0 con 0x7fcb440c6f60 2023-10-30T12:43:44.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb4159a640 1 -- 172.21.15.89:0/2791007225 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7fcb38001680 con 0x7fcb440c6f60 2023-10-30T12:43:44.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb4159a640 1 -- 172.21.15.89:0/2791007225 >> v1:172.21.15.89:6789/0 conn(0x7fcb440c6f60 legacy=0x7fcb4413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:44.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb4159a640 1 --2- 172.21.15.89:0/2791007225 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb2c003980 0x7fcb2c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:44.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb4159a640 1 --2- 172.21.15.89:0/2791007225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb2c0047d0 0x7fcb2c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:44.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb4159a640 1 --2- 172.21.15.89:0/2791007225 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb4413f4a0 0x7fcb2c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:44.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb4159a640 1 -- 172.21.15.89:0/2791007225 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4406f6e0 con 0x7fcb2c0047d0 2023-10-30T12:43:44.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb4159a640 1 -- 172.21.15.89:0/2791007225 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4406f9c0 con 0x7fcb2c003980 2023-10-30T12:43:44.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb4159a640 1 -- 172.21.15.89:0/2791007225 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2c00b990 con 0x7fcb4413f4a0 2023-10-30T12:43:44.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb41d9b640 1 --2- 172.21.15.89:0/2791007225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb2c0047d0 0x7fcb2c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:44.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb4259c640 1 --2- 172.21.15.89:0/2791007225 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb2c003980 0x7fcb2c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:44.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb42d9d640 1 --2- 172.21.15.89:0/2791007225 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb4413f4a0 0x7fcb2c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:44.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb41d9b640 1 -- 172.21.15.89:0/2791007225 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb4413f4a0 msgr2=0x7fcb2c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:44.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb41d9b640 1 --2- 172.21.15.89:0/2791007225 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb4413f4a0 0x7fcb2c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:44.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb41d9b640 1 -- 172.21.15.89:0/2791007225 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb2c003980 msgr2=0x7fcb2c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:44.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb41d9b640 1 --2- 172.21.15.89:0/2791007225 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb2c003980 0x7fcb2c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:44.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.862+0000 7fcb41d9b640 1 -- 172.21.15.89:0/2791007225 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb2c003530 con 0x7fcb2c0047d0 2023-10-30T12:43:44.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.866+0000 7fcb41d9b640 1 --2- 172.21.15.89:0/2791007225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb2c0047d0 0x7fcb2c004c50 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fcb3800c290 tx=0x7fcb3801f430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19946857564b4baf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:44.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.866+0000 7fcb4159a640 1 -- 172.21.15.89:0/2791007225 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb380a8cd0 con 0x7fcb2c0047d0 2023-10-30T12:43:44.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.866+0000 7fcb4159a640 1 -- 172.21.15.89:0/2791007225 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb380a8e30 con 0x7fcb2c0047d0 2023-10-30T12:43:44.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.866+0000 7fcb4159a640 1 -- 172.21.15.89:0/2791007225 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb38005b20 con 0x7fcb2c0047d0 2023-10-30T12:43:44.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.866+0000 7fcb48f65640 1 -- 172.21.15.89:0/2791007225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb2c0047d0 msgr2=0x7fcb2c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:44.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.866+0000 7fcb48f65640 1 --2- 172.21.15.89:0/2791007225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb2c0047d0 0x7fcb2c004c50 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fcb3800c290 tx=0x7fcb3801f430 comp rx=0 tx=0).stop 2023-10-30T12:43:44.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.866+0000 7fcb42d9d640 1 -- 172.21.15.89:0/2791007225 reap_dead start 2023-10-30T12:43:44.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.866+0000 7fcb48f65640 1 -- 172.21.15.89:0/2791007225 shutdown_connections 2023-10-30T12:43:44.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.866+0000 7fcb48f65640 1 -- 172.21.15.89:0/2791007225 >> 172.21.15.89:0/2791007225 conn(0x7fcb440c7910 msgr2=0x7fcb440c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:44.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.866+0000 7fcb48f65640 1 -- 172.21.15.89:0/2791007225 shutdown_connections 2023-10-30T12:43:44.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.866+0000 7fcb48f65640 1 -- 172.21.15.89:0/2791007225 wait complete. 2023-10-30T12:43:44.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.866+0000 7fcb48f65640 1 Processor -- start 2023-10-30T12:43:44.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.866+0000 7fcb48f65640 1 -- start start 2023-10-30T12:43:44.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.870+0000 7fcb48f65640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb2c0047d0 0x7fcb4413bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:44.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.870+0000 7fcb48f65640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb441ef220 0x7fcb44155a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:44.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.870+0000 7fcb48f65640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb44155f90 0x7fcb44137510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:44.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.870+0000 7fcb48f65640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb44155190 con 0x7fcb2c0047d0 2023-10-30T12:43:44.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.870+0000 7fcb48f65640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4413d740 con 0x7fcb441ef220 2023-10-30T12:43:44.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.870+0000 7fcb48f65640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4413d2b0 con 0x7fcb44155f90 2023-10-30T12:43:44.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.870+0000 7fcb4259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb2c0047d0 0x7fcb4413bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:44.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.870+0000 7fcb4259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb2c0047d0 0x7fcb4413bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50460/0 (socket says 172.21.15.89:50460) 2023-10-30T12:43:44.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.870+0000 7fcb4259c640 1 -- 172.21.15.89:0/461680 learned_addr learned my addr 172.21.15.89:0/461680 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:43:44.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.870+0000 7fcb41d9b640 1 --2- 172.21.15.89:0/461680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb441ef220 0x7fcb44155a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:44.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.870+0000 7fcb42d9d640 1 --2- 172.21.15.89:0/461680 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb44155f90 0x7fcb44137510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:44.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.870+0000 7fcb4259c640 1 -- 172.21.15.89:0/461680 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb44155f90 msgr2=0x7fcb44137510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:44.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.870+0000 7fcb4259c640 1 --2- 172.21.15.89:0/461680 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb44155f90 0x7fcb44137510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:44.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.870+0000 7fcb4259c640 1 -- 172.21.15.89:0/461680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb441ef220 msgr2=0x7fcb44155a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:44.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.870+0000 7fcb4259c640 1 --2- 172.21.15.89:0/461680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb441ef220 0x7fcb44155a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:44.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.870+0000 7fcb4259c640 1 -- 172.21.15.89:0/461680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb441ef710 con 0x7fcb2c0047d0 2023-10-30T12:43:44.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.870+0000 7fcb42d9d640 1 --2- 172.21.15.89:0/461680 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb44155f90 0x7fcb44137510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:43:44.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.870+0000 7fcb4259c640 1 --2- 172.21.15.89:0/461680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb2c0047d0 0x7fcb4413bb20 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fcb34011590 tx=0x7fcb34011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:44.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.870+0000 7fcb32ffd640 1 -- 172.21.15.89:0/461680 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb34015930 con 0x7fcb2c0047d0 2023-10-30T12:43:44.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.870+0000 7fcb32ffd640 1 -- 172.21.15.89:0/461680 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb34015a90 con 0x7fcb2c0047d0 2023-10-30T12:43:44.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.870+0000 7fcb32ffd640 1 -- 172.21.15.89:0/461680 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb34015e20 con 0x7fcb2c0047d0 2023-10-30T12:43:44.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.870+0000 7fcb48f65640 1 -- 172.21.15.89:0/461680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb4413c6e0 con 0x7fcb2c0047d0 2023-10-30T12:43:44.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.870+0000 7fcb48f65640 1 -- 172.21.15.89:0/461680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb4413cc60 con 0x7fcb2c0047d0 2023-10-30T12:43:44.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.870+0000 7fcb48f65640 1 -- 172.21.15.89:0/461680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb4406ff60 con 0x7fcb2c0047d0 2023-10-30T12:43:44.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.870+0000 7fcb32ffd640 1 -- 172.21.15.89:0/461680 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcb3400f460 con 0x7fcb2c0047d0 2023-10-30T12:43:44.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.874+0000 7fcb32ffd640 1 --2- 172.21.15.89:0/461680 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fcb240664f0 0x7fcb240689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:44.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.874+0000 7fcb32ffd640 1 -- 172.21.15.89:0/461680 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6406+0+0 (secure 0 0 0) 0x7fcb340849a0 con 0x7fcb2c0047d0 2023-10-30T12:43:44.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.874+0000 7fcb41d9b640 1 --2- 172.21.15.89:0/461680 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fcb240664f0 0x7fcb240689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:44.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.874+0000 7fcb41d9b640 1 --2- 172.21.15.89:0/461680 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fcb240664f0 0x7fcb240689b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fcb380ad000 tx=0x7fcb38001a10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:44.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:44.878+0000 7fcb32ffd640 1 -- 172.21.15.89:0/461680 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcb4406ff60 con 0x7fcb2c0047d0 2023-10-30T12:43:45.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:45.054+0000 7fcb48f65640 1 -- 172.21.15.89:0/461680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb4406ff60 con 0x7fcb2c0047d0 2023-10-30T12:43:45.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:45.058+0000 7fcb32ffd640 1 -- 172.21.15.89:0/461680 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+5736 (secure 0 0 0) 0x7fcb4406ff60 con 0x7fcb2c0047d0 2023-10-30T12:43:45.054 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:43:45.054 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:43:42.437815+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:resolve","state_seq":13,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:43:45.055 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 74 2023-10-30T12:43:45.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:45.058+0000 7fcb48f65640 1 -- 172.21.15.89:0/461680 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fcb240664f0 msgr2=0x7fcb240689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:45.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:45.058+0000 7fcb48f65640 1 --2- 172.21.15.89:0/461680 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fcb240664f0 0x7fcb240689b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fcb380ad000 tx=0x7fcb38001a10 comp rx=0 tx=0).stop 2023-10-30T12:43:45.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:45.058+0000 7fcb48f65640 1 -- 172.21.15.89:0/461680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb2c0047d0 msgr2=0x7fcb4413bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:45.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:45.058+0000 7fcb48f65640 1 --2- 172.21.15.89:0/461680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb2c0047d0 0x7fcb4413bb20 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fcb34011590 tx=0x7fcb34011a50 comp rx=0 tx=0).stop 2023-10-30T12:43:45.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:45.058+0000 7fcb48f65640 1 -- 172.21.15.89:0/461680 shutdown_connections 2023-10-30T12:43:45.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:45.058+0000 7fcb48f65640 1 --2- 172.21.15.89:0/461680 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fcb240664f0 0x7fcb240689b0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:45.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:45.058+0000 7fcb48f65640 1 --2- 172.21.15.89:0/461680 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb441ef220 0x7fcb44155a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:45.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:45.058+0000 7fcb48f65640 1 --2- 172.21.15.89:0/461680 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb44155f90 0x7fcb44137510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:45.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:45.058+0000 7fcb48f65640 1 --2- 172.21.15.89:0/461680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb2c0047d0 0x7fcb4413bb20 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:45.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:45.058+0000 7fcb48f65640 1 -- 172.21.15.89:0/461680 >> 172.21.15.89:0/461680 conn(0x7fcb440c7910 msgr2=0x7fcb440c5810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:45.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:45.058+0000 7fcb48f65640 1 -- 172.21.15.89:0/461680 shutdown_connections 2023-10-30T12:43:45.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:45.058+0000 7fcb48f65640 1 -- 172.21.15.89:0/461680 wait complete. 2023-10-30T12:43:45.074 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:43:47.076 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:43:47.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.226+0000 7f6107617640 1 Processor -- start 2023-10-30T12:43:47.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.226+0000 7f6107617640 1 -- start start 2023-10-30T12:43:47.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.226+0000 7f6107617640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f610006f7a0 0x7f610006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:47.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.226+0000 7f6107617640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f61000c6bd0 0x7f6100069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:47.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.230+0000 7f6107617640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6100071110 0x7f6100158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:47.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.230+0000 7f6107617640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61000700c0 con 0x7f610006f7a0 2023-10-30T12:43:47.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.230+0000 7f6107617640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f610006a3d0 con 0x7f61000c6bd0 2023-10-30T12:43:47.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.230+0000 7f6107617640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6100066cf0 con 0x7f6100071110 2023-10-30T12:43:47.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.230+0000 7f61053b3640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f610006f7a0 0x7f610006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:47.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.230+0000 7f61053b3640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f610006f7a0 0x7f610006fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50462/0 (socket says 172.21.15.89:50462) 2023-10-30T12:43:47.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.230+0000 7f61053b3640 1 -- 172.21.15.89:0/2923707859 learned_addr learned my addr 172.21.15.89:0/2923707859 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:43:47.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.230+0000 7f6104bb2640 1 --2- 172.21.15.89:0/2923707859 >> v2:172.21.15.192:3300/0 conn(0x7f61000c6bd0 0x7f6100069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:47.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.230+0000 7f6105bb4640 1 --2- 172.21.15.89:0/2923707859 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6100071110 0x7f6100158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:47.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.230+0000 7f61053b3640 1 -- 172.21.15.89:0/2923707859 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6100071110 msgr2=0x7f6100158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:47.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.230+0000 7f61053b3640 1 --2- 172.21.15.89:0/2923707859 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6100071110 0x7f6100158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:47.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.230+0000 7f61053b3640 1 -- 172.21.15.89:0/2923707859 >> v2:172.21.15.192:3300/0 conn(0x7f61000c6bd0 msgr2=0x7f6100069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:47.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.230+0000 7f61053b3640 1 --2- 172.21.15.89:0/2923707859 >> v2:172.21.15.192:3300/0 conn(0x7f61000c6bd0 0x7f6100069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:47.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.230+0000 7f61053b3640 1 -- 172.21.15.89:0/2923707859 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61000c70d0 con 0x7f610006f7a0 2023-10-30T12:43:47.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.230+0000 7f6104bb2640 1 --2- 172.21.15.89:0/2923707859 >> v2:172.21.15.192:3300/0 conn(0x7f61000c6bd0 0x7f6100069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:43:47.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.230+0000 7f61053b3640 1 --2- 172.21.15.89:0/2923707859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f610006f7a0 0x7f610006fb80 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f610006f1c0 tx=0x7f60fc01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d61d525ae5a7f30 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:47.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.230+0000 7f60f7fff640 1 -- 172.21.15.89:0/2923707859 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60fc0a89b0 con 0x7f610006f7a0 2023-10-30T12:43:47.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.230+0000 7f60f7fff640 1 -- 172.21.15.89:0/2923707859 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60fc0a8b10 con 0x7f610006f7a0 2023-10-30T12:43:47.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.230+0000 7f60f7fff640 1 -- 172.21.15.89:0/2923707859 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60fc0a8ea0 con 0x7f610006f7a0 2023-10-30T12:43:47.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.234+0000 7f6107617640 1 -- 172.21.15.89:0/2923707859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f610006f7a0 msgr2=0x7f610006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:47.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.234+0000 7f6107617640 1 --2- 172.21.15.89:0/2923707859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f610006f7a0 0x7f610006fb80 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f610006f1c0 tx=0x7f60fc01ecf0 comp rx=0 tx=0).stop 2023-10-30T12:43:47.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.234+0000 7f6107617640 1 -- 172.21.15.89:0/2923707859 shutdown_connections 2023-10-30T12:43:47.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.234+0000 7f6107617640 1 --2- 172.21.15.89:0/2923707859 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6100071110 0x7f6100158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:47.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.234+0000 7f6107617640 1 --2- 172.21.15.89:0/2923707859 >> v2:172.21.15.192:3300/0 conn(0x7f61000c6bd0 0x7f6100069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:47.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.234+0000 7f6107617640 1 --2- 172.21.15.89:0/2923707859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f610006f7a0 0x7f610006fb80 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:47.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.234+0000 7f6107617640 1 -- 172.21.15.89:0/2923707859 >> 172.21.15.89:0/2923707859 conn(0x7f61000c7910 msgr2=0x7f61000c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:47.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.234+0000 7f6107617640 1 -- 172.21.15.89:0/2923707859 shutdown_connections 2023-10-30T12:43:47.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.234+0000 7f6107617640 1 -- 172.21.15.89:0/2923707859 wait complete. 2023-10-30T12:43:47.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.234+0000 7f6107617640 1 Processor -- start 2023-10-30T12:43:47.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.234+0000 7f6107617640 1 -- start start 2023-10-30T12:43:47.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.234+0000 7f6107617640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f610006f7a0 0x7f61001f32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:47.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.234+0000 7f6107617640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6100071110 0x7f6100153610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:47.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.234+0000 7f6107617640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61000c6bd0 0x7f61001ef1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:47.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.234+0000 7f6107617640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6100153b50 con 0x7f6100071110 2023-10-30T12:43:47.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.234+0000 7f6107617640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61001ef700 con 0x7f61000c6bd0 2023-10-30T12:43:47.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.234+0000 7f6107617640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61000af2e0 con 0x7f610006f7a0 2023-10-30T12:43:47.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.234+0000 7f6104bb2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6100071110 0x7f6100153610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:47.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.234+0000 7f6104bb2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6100071110 0x7f6100153610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50466/0 (socket says 172.21.15.89:50466) 2023-10-30T12:43:47.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.234+0000 7f6104bb2640 1 -- 172.21.15.89:0/2532395111 learned_addr learned my addr 172.21.15.89:0/2532395111 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:43:47.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.238+0000 7f6105bb4640 1 --2- 172.21.15.89:0/2532395111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61000c6bd0 0x7f61001ef1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:47.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.238+0000 7f61053b3640 1 --2- 172.21.15.89:0/2532395111 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f610006f7a0 0x7f61001f32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:47.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.238+0000 7f6104bb2640 1 -- 172.21.15.89:0/2532395111 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f610006f7a0 msgr2=0x7f61001f32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:47.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.238+0000 7f6104bb2640 1 --2- 172.21.15.89:0/2532395111 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f610006f7a0 0x7f61001f32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:47.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.238+0000 7f6104bb2640 1 -- 172.21.15.89:0/2532395111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61000c6bd0 msgr2=0x7f61001ef1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:47.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.238+0000 7f6104bb2640 1 --2- 172.21.15.89:0/2532395111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61000c6bd0 0x7f61001ef1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:47.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.238+0000 7f6104bb2640 1 -- 172.21.15.89:0/2532395111 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60fc0094d0 con 0x7f6100071110 2023-10-30T12:43:47.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.238+0000 7f6104bb2640 1 --2- 172.21.15.89:0/2532395111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6100071110 0x7f6100153610 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f60f000c820 tx=0x7f60f000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:47.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.238+0000 7f60f5ffb640 1 -- 172.21.15.89:0/2532395111 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f0007a30 con 0x7f6100071110 2023-10-30T12:43:47.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.238+0000 7f60f5ffb640 1 -- 172.21.15.89:0/2532395111 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f0007b90 con 0x7f6100071110 2023-10-30T12:43:47.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.238+0000 7f60f5ffb640 1 -- 172.21.15.89:0/2532395111 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f000ce10 con 0x7f6100071110 2023-10-30T12:43:47.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.238+0000 7f6107617640 1 -- 172.21.15.89:0/2532395111 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61000af580 con 0x7f6100071110 2023-10-30T12:43:47.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.238+0000 7f6107617640 1 -- 172.21.15.89:0/2532395111 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61001f27d0 con 0x7f6100071110 2023-10-30T12:43:47.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.238+0000 7f6107617640 1 -- 172.21.15.89:0/2532395111 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f610004e170 con 0x7f6100071110 2023-10-30T12:43:47.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.238+0000 7f60f5ffb640 1 -- 172.21.15.89:0/2532395111 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f60f00113f0 con 0x7f6100071110 2023-10-30T12:43:47.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.238+0000 7f60f5ffb640 1 --2- 172.21.15.89:0/2532395111 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f60e40664a0 0x7f60e4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:47.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.238+0000 7f60f5ffb640 1 -- 172.21.15.89:0/2532395111 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6406+0+0 (secure 0 0 0) 0x7f60f00841e0 con 0x7f6100071110 2023-10-30T12:43:47.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.238+0000 7f61053b3640 1 --2- 172.21.15.89:0/2532395111 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f60e40664a0 0x7f60e4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:47.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.242+0000 7f61053b3640 1 --2- 172.21.15.89:0/2532395111 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f60e40664a0 0x7f60e4068960 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f60fc0b2000 tx=0x7f60fc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:47.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.242+0000 7f60f5ffb640 1 -- 172.21.15.89:0/2532395111 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f60f0051390 con 0x7f6100071110 2023-10-30T12:43:47.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.426+0000 7f6107617640 1 -- 172.21.15.89:0/2532395111 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6100070620 con 0x7f6100071110 2023-10-30T12:43:47.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.426+0000 7f60f5ffb640 1 -- 172.21.15.89:0/2532395111 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+5736 (secure 0 0 0) 0x7f60f00549b0 con 0x7f6100071110 2023-10-30T12:43:47.425 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:43:47.425 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:43:42.437815+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:resolve","state_seq":13,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:43:47.425 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 74 2023-10-30T12:43:47.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.430+0000 7f6107617640 1 -- 172.21.15.89:0/2532395111 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f60e40664a0 msgr2=0x7f60e4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:47.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.430+0000 7f6107617640 1 --2- 172.21.15.89:0/2532395111 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f60e40664a0 0x7f60e4068960 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f60fc0b2000 tx=0x7f60fc008040 comp rx=0 tx=0).stop 2023-10-30T12:43:47.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.430+0000 7f6107617640 1 -- 172.21.15.89:0/2532395111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6100071110 msgr2=0x7f6100153610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:47.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.430+0000 7f6107617640 1 --2- 172.21.15.89:0/2532395111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6100071110 0x7f6100153610 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f60f000c820 tx=0x7f60f000cce0 comp rx=0 tx=0).stop 2023-10-30T12:43:47.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.430+0000 7f6107617640 1 -- 172.21.15.89:0/2532395111 shutdown_connections 2023-10-30T12:43:47.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.430+0000 7f6107617640 1 --2- 172.21.15.89:0/2532395111 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f60e40664a0 0x7f60e4068960 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:47.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.430+0000 7f6107617640 1 --2- 172.21.15.89:0/2532395111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61000c6bd0 0x7f61001ef1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:47.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.430+0000 7f6107617640 1 --2- 172.21.15.89:0/2532395111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6100071110 0x7f6100153610 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:47.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.430+0000 7f6107617640 1 --2- 172.21.15.89:0/2532395111 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f610006f7a0 0x7f61001f32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:47.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.430+0000 7f6107617640 1 -- 172.21.15.89:0/2532395111 >> 172.21.15.89:0/2532395111 conn(0x7f61000c7910 msgr2=0x7f6100156bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:47.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.430+0000 7f6107617640 1 -- 172.21.15.89:0/2532395111 shutdown_connections 2023-10-30T12:43:47.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:47.430+0000 7f6107617640 1 -- 172.21.15.89:0/2532395111 wait complete. 2023-10-30T12:43:47.429 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:43:47.433 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:43:47.446 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:43:49.447 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:43:49.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.598+0000 7f9400194640 1 Processor -- start 2023-10-30T12:43:49.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.598+0000 7f9400194640 1 -- start start 2023-10-30T12:43:49.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.598+0000 7f9400194640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f93f8153bf0 0x7f93f8071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:49.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.598+0000 7f9400194640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93f80c5f60 0x7f93f80cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:49.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.598+0000 7f9400194640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93f80700b0 con 0x7f93f80c6fe0 2023-10-30T12:43:49.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.598+0000 7f9400194640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93f80c6810 con 0x7f93f8153bf0 2023-10-30T12:43:49.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.598+0000 7f9400194640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93f8159900 con 0x7f93f80c5f60 2023-10-30T12:43:49.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.602+0000 7f93fdf30640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f93f8153bf0 0x7f93f8071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:49.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.602+0000 7f93fe731640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93f80c5f60 0x7f93f80cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:49.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.602+0000 7f93fdf30640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f93f8153bf0 0x7f93f8071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:43794/0 (socket says 172.21.15.89:43794) 2023-10-30T12:43:49.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.602+0000 7f93fdf30640 1 -- 172.21.15.89:0/622269562 learned_addr learned my addr 172.21.15.89:0/622269562 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:43:49.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.602+0000 7f93fdf30640 1 -- 172.21.15.89:0/622269562 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93f80c5f60 msgr2=0x7f93f80cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:49.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.602+0000 7f93fdf30640 1 --2- 172.21.15.89:0/622269562 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93f80c5f60 0x7f93f80cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:49.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.602+0000 7f93fdf30640 1 -- 172.21.15.89:0/622269562 >> v1:172.21.15.192:6789/0 conn(0x7f93f80c6fe0 legacy=0x7f93f80c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:49.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.602+0000 7f93fdf30640 1 -- 172.21.15.89:0/622269562 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f80c6530 con 0x7f93f8153bf0 2023-10-30T12:43:49.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.606+0000 7f93fdf30640 1 --2- 172.21.15.89:0/622269562 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93f8153bf0 0x7f93f8071260 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f93f4009870 tx=0x7f93f401ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=524835d33c6c24a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:49.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.606+0000 7f93fcf2e640 1 -- 172.21.15.89:0/622269562 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f40a8a30 con 0x7f93f8153bf0 2023-10-30T12:43:49.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.606+0000 7f93fcf2e640 1 -- 172.21.15.89:0/622269562 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f40a8b90 con 0x7f93f8153bf0 2023-10-30T12:43:49.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.606+0000 7f93fcf2e640 1 -- 172.21.15.89:0/622269562 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f40b0440 con 0x7f93f8153bf0 2023-10-30T12:43:49.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.606+0000 7f9400194640 1 -- 172.21.15.89:0/622269562 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93f8153bf0 msgr2=0x7f93f8071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:49.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.606+0000 7f9400194640 1 --2- 172.21.15.89:0/622269562 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93f8153bf0 0x7f93f8071260 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f93f4009870 tx=0x7f93f401ed30 comp rx=0 tx=0).stop 2023-10-30T12:43:49.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.606+0000 7f9400194640 1 -- 172.21.15.89:0/622269562 shutdown_connections 2023-10-30T12:43:49.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.606+0000 7f9400194640 1 --2- 172.21.15.89:0/622269562 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93f80c5f60 0x7f93f80cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:49.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.606+0000 7f9400194640 1 --2- 172.21.15.89:0/622269562 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93f8153bf0 0x7f93f8071260 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:49.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.606+0000 7f9400194640 1 -- 172.21.15.89:0/622269562 >> 172.21.15.89:0/622269562 conn(0x7f93f806c6e0 msgr2=0x7f93f8069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:49.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.606+0000 7f9400194640 1 -- 172.21.15.89:0/622269562 shutdown_connections 2023-10-30T12:43:49.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.606+0000 7f9400194640 1 -- 172.21.15.89:0/622269562 wait complete. 2023-10-30T12:43:49.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.606+0000 7f9400194640 1 Processor -- start 2023-10-30T12:43:49.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.606+0000 7f9400194640 1 -- start start 2023-10-30T12:43:49.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.606+0000 7f9400194640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93f80c5f60 0x7f93f814f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:49.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.606+0000 7f9400194640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93f80c6fe0 0x7f93f814ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:49.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.610+0000 7f9400194640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93f8153bf0 0x7f93f8152d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:49.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.610+0000 7f9400194640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f814f240 con 0x7f93f8153bf0 2023-10-30T12:43:49.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.610+0000 7f9400194640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f81efbf0 con 0x7f93f80c6fe0 2023-10-30T12:43:49.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.610+0000 7f9400194640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93f814df40 con 0x7f93f80c5f60 2023-10-30T12:43:49.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.610+0000 7f93fe731640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93f8153bf0 0x7f93f8152d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:49.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.610+0000 7f93fe731640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93f8153bf0 0x7f93f8152d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50472/0 (socket says 172.21.15.89:50472) 2023-10-30T12:43:49.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.610+0000 7f93fe731640 1 -- 172.21.15.89:0/2906705609 learned_addr learned my addr 172.21.15.89:0/2906705609 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:43:49.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.610+0000 7f93fdf30640 1 --2- 172.21.15.89:0/2906705609 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93f80c5f60 0x7f93f814f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:49.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.610+0000 7f93fd72f640 1 --2- 172.21.15.89:0/2906705609 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93f80c6fe0 0x7f93f814ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:49.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.610+0000 7f93fe731640 1 -- 172.21.15.89:0/2906705609 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93f80c5f60 msgr2=0x7f93f814f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:49.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.610+0000 7f93fe731640 1 --2- 172.21.15.89:0/2906705609 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93f80c5f60 0x7f93f814f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:49.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.610+0000 7f93fe731640 1 -- 172.21.15.89:0/2906705609 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93f80c6fe0 msgr2=0x7f93f814ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:49.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.610+0000 7f93fe731640 1 --2- 172.21.15.89:0/2906705609 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93f80c6fe0 0x7f93f814ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:49.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.610+0000 7f93fe731640 1 -- 172.21.15.89:0/2906705609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f40094d0 con 0x7f93f8153bf0 2023-10-30T12:43:49.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.610+0000 7f93fdf30640 1 --2- 172.21.15.89:0/2906705609 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93f80c5f60 0x7f93f814f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:43:49.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.610+0000 7f93fe731640 1 --2- 172.21.15.89:0/2906705609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93f8153bf0 0x7f93f8152d30 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f93f000efc0 tx=0x7f93f000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:49.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.610+0000 7f93ee7fc640 1 -- 172.21.15.89:0/2906705609 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f0009ba0 con 0x7f93f8153bf0 2023-10-30T12:43:49.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.610+0000 7f93ee7fc640 1 -- 172.21.15.89:0/2906705609 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f000f040 con 0x7f93f8153bf0 2023-10-30T12:43:49.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.610+0000 7f93ee7fc640 1 -- 172.21.15.89:0/2906705609 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f00146c0 con 0x7f93f8153bf0 2023-10-30T12:43:49.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.610+0000 7f9400194640 1 -- 172.21.15.89:0/2906705609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93f814e220 con 0x7f93f8153bf0 2023-10-30T12:43:49.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.610+0000 7f9400194640 1 -- 172.21.15.89:0/2906705609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93f814e7e0 con 0x7f93f8153bf0 2023-10-30T12:43:49.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.610+0000 7f9400194640 1 -- 172.21.15.89:0/2906705609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93f814ea00 con 0x7f93f8153bf0 2023-10-30T12:43:49.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.610+0000 7f93ee7fc640 1 -- 172.21.15.89:0/2906705609 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f93f0014820 con 0x7f93f8153bf0 2023-10-30T12:43:49.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.614+0000 7f93ee7fc640 1 --2- 172.21.15.89:0/2906705609 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f93dc0664a0 0x7f93dc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:49.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.614+0000 7f93ee7fc640 1 -- 172.21.15.89:0/2906705609 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6406+0+0 (secure 0 0 0) 0x7f93f0084f60 con 0x7f93f8153bf0 2023-10-30T12:43:49.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.614+0000 7f93fdf30640 1 --2- 172.21.15.89:0/2906705609 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f93dc0664a0 0x7f93dc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:49.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.614+0000 7f93fdf30640 1 --2- 172.21.15.89:0/2906705609 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f93dc0664a0 0x7f93dc068960 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f93f40b2000 tx=0x7f93f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:49.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.618+0000 7f93ee7fc640 1 -- 172.21.15.89:0/2906705609 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f93f0052080 con 0x7f93f8153bf0 2023-10-30T12:43:49.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.798+0000 7f9400194640 1 -- 172.21.15.89:0/2906705609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93f804e170 con 0x7f93f8153bf0 2023-10-30T12:43:49.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.798+0000 7f93ee7fc640 1 -- 172.21.15.89:0/2906705609 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f93f0018090 con 0x7f93f8153bf0 2023-10-30T12:43:49.796 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:43:49.796 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:43:48.249195+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:43:49.796 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 75 2023-10-30T12:43:49.797 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.798+0000 7f9400194640 1 -- 172.21.15.89:0/2906705609 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f93dc0664a0 msgr2=0x7f93dc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:49.798 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.798+0000 7f9400194640 1 --2- 172.21.15.89:0/2906705609 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f93dc0664a0 0x7f93dc068960 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f93f40b2000 tx=0x7f93f4008040 comp rx=0 tx=0).stop 2023-10-30T12:43:49.798 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.798+0000 7f9400194640 1 -- 172.21.15.89:0/2906705609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93f8153bf0 msgr2=0x7f93f8152d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:49.798 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.798+0000 7f9400194640 1 --2- 172.21.15.89:0/2906705609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93f8153bf0 0x7f93f8152d30 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f93f000efc0 tx=0x7f93f000c490 comp rx=0 tx=0).stop 2023-10-30T12:43:49.798 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.798+0000 7f9400194640 1 -- 172.21.15.89:0/2906705609 shutdown_connections 2023-10-30T12:43:49.798 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.798+0000 7f9400194640 1 --2- 172.21.15.89:0/2906705609 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f93dc0664a0 0x7f93dc068960 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:49.799 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.798+0000 7f9400194640 1 --2- 172.21.15.89:0/2906705609 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93f80c6fe0 0x7f93f814ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:49.799 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.798+0000 7f9400194640 1 --2- 172.21.15.89:0/2906705609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93f8153bf0 0x7f93f8152d30 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:49.799 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.798+0000 7f9400194640 1 --2- 172.21.15.89:0/2906705609 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93f80c5f60 0x7f93f814f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:49.799 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.798+0000 7f9400194640 1 -- 172.21.15.89:0/2906705609 >> 172.21.15.89:0/2906705609 conn(0x7f93f806c6e0 msgr2=0x7f93f8158e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:49.800 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.802+0000 7f9400194640 1 -- 172.21.15.89:0/2906705609 shutdown_connections 2023-10-30T12:43:49.800 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:49.802+0000 7f9400194640 1 -- 172.21.15.89:0/2906705609 wait complete. 2023-10-30T12:43:49.817 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 75, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:50.729245+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 75, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5139": { "addr": "172.21.15.89:6837/951276252", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6836", "nonce": 951276252, "type": "v2" }, { "addr": "172.21.15.89:6837", "nonce": 951276252, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5139, "incarnation": 33, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 29 }, "gid_5575": { "addr": "172.21.15.192:6835/501291700", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6834", "nonce": 501291700, "type": "v2" }, { "addr": "172.21.15.192:6835", "nonce": 501291700, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5575, "incarnation": 63, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 37 }, "gid_5728": { "addr": "172.21.15.192:6839/3991891597", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6838", "nonce": 3991891597, "type": "v2" }, { "addr": "172.21.15.192:6839", "nonce": 3991891597, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5728, "incarnation": 73, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:reconnect", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 62, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:43:48.249195+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5575, "mds_1": 5139, "mds_2": 5728 } } } ], "standbys": [ { "addr": "172.21.15.89:6834/2909790173", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6832", "nonce": 2909790173, "type": "v2" }, { "addr": "172.21.15.89:6834", "nonce": 2909790173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.89:6835/2387357615", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6833", "nonce": 2387357615, "type": "v2" }, { "addr": "172.21.15.89:6835", "nonce": 2387357615, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4239, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:43:51.818 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:43:51.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.970+0000 7f563426b640 1 Processor -- start 2023-10-30T12:43:51.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.970+0000 7f563426b640 1 -- start start 2023-10-30T12:43:51.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.970+0000 7f563426b640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f562c1538a0 0x7f562c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:51.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.970+0000 7f563426b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f562c0ca0e0 0x7f562c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:51.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.970+0000 7f563426b640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f562c0cb800 con 0x7f562c0cf160 2023-10-30T12:43:51.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.970+0000 7f563426b640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f562c15dc30 con 0x7f562c1538a0 2023-10-30T12:43:51.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.970+0000 7f563426b640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f562c0cb330 con 0x7f562c0ca0e0 2023-10-30T12:43:51.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.970+0000 7f5632007640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f562c1538a0 0x7f562c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:51.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.970+0000 7f5632808640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f562c0cf160 0x7f562c0cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46900/0 (socket says 172.21.15.89:46900) 2023-10-30T12:43:51.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.970+0000 7f5632808640 1 -- 172.21.15.89:0/1178697235 learned_addr learned my addr 172.21.15.89:0/1178697235 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:43:51.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.970+0000 7f5631806640 1 --2- 172.21.15.89:0/1178697235 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f562c0ca0e0 0x7f562c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:51.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.970+0000 7f5632007640 1 -- 172.21.15.89:0/1178697235 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f562c0ca0e0 msgr2=0x7f562c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:51.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.970+0000 7f5632007640 1 --2- 172.21.15.89:0/1178697235 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f562c0ca0e0 0x7f562c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:51.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.970+0000 7f5632007640 1 -- 172.21.15.89:0/1178697235 >> v1:172.21.15.89:6789/0 conn(0x7f562c0cf160 legacy=0x7f562c0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:51.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.970+0000 7f5632007640 1 -- 172.21.15.89:0/1178697235 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f562c0cad70 con 0x7f562c1538a0 2023-10-30T12:43:51.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.970+0000 7f5631005640 1 -- 172.21.15.89:0/1178697235 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3166596791 0 0) 0x7f562c0cb800 con 0x7f562c0cf160 2023-10-30T12:43:51.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.974+0000 7f5632007640 1 --2- 172.21.15.89:0/1178697235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c1538a0 0x7f562c15d6f0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f56280097c0 tx=0x7f562801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11be7b771a3de940 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:51.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.974+0000 7f5631005640 1 -- 172.21.15.89:0/1178697235 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56280a8970 con 0x7f562c1538a0 2023-10-30T12:43:51.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.974+0000 7f5631005640 1 -- 172.21.15.89:0/1178697235 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56280a8ad0 con 0x7f562c1538a0 2023-10-30T12:43:51.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.974+0000 7f5631005640 1 -- 172.21.15.89:0/1178697235 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56280a8e60 con 0x7f562c1538a0 2023-10-30T12:43:51.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.974+0000 7f563426b640 1 -- 172.21.15.89:0/1178697235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c1538a0 msgr2=0x7f562c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:51.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.974+0000 7f563426b640 1 --2- 172.21.15.89:0/1178697235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c1538a0 0x7f562c15d6f0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f56280097c0 tx=0x7f562801ec70 comp rx=0 tx=0).stop 2023-10-30T12:43:51.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.974+0000 7f563426b640 1 -- 172.21.15.89:0/1178697235 shutdown_connections 2023-10-30T12:43:51.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.974+0000 7f563426b640 1 --2- 172.21.15.89:0/1178697235 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f562c0ca0e0 0x7f562c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:51.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.974+0000 7f563426b640 1 --2- 172.21.15.89:0/1178697235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c1538a0 0x7f562c15d6f0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:51.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.974+0000 7f563426b640 1 -- 172.21.15.89:0/1178697235 >> 172.21.15.89:0/1178697235 conn(0x7f562c06bfe0 msgr2=0x7f562c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:51.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.974+0000 7f563426b640 1 -- 172.21.15.89:0/1178697235 shutdown_connections 2023-10-30T12:43:51.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.974+0000 7f563426b640 1 -- 172.21.15.89:0/1178697235 wait complete. 2023-10-30T12:43:51.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.978+0000 7f563426b640 1 Processor -- start 2023-10-30T12:43:51.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.978+0000 7f563426b640 1 -- start start 2023-10-30T12:43:51.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.978+0000 7f563426b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c0ca0e0 0x7f562c140530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:51.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.978+0000 7f563426b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f562c0cf160 0x7f562c146bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:51.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.978+0000 7f563426b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f562c1538a0 0x7f562c147110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:51.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.978+0000 7f563426b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562c0ce8c0 con 0x7f562c0ca0e0 2023-10-30T12:43:51.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.978+0000 7f563426b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562c14eed0 con 0x7f562c0cf160 2023-10-30T12:43:51.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.978+0000 7f563426b640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f562c1f3290 con 0x7f562c1538a0 2023-10-30T12:43:51.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.978+0000 7f5632007640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c0ca0e0 0x7f562c140530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:51.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.978+0000 7f5632007640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c0ca0e0 0x7f562c140530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33004/0 (socket says 172.21.15.89:33004) 2023-10-30T12:43:51.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.978+0000 7f5632007640 1 -- 172.21.15.89:0/2576337438 learned_addr learned my addr 172.21.15.89:0/2576337438 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:43:51.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.978+0000 7f5631806640 1 --2- 172.21.15.89:0/2576337438 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f562c0cf160 0x7f562c146bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:51.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.978+0000 7f5632808640 1 --2- 172.21.15.89:0/2576337438 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f562c1538a0 0x7f562c147110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:51.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.978+0000 7f5632007640 1 -- 172.21.15.89:0/2576337438 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f562c1538a0 msgr2=0x7f562c147110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:51.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.978+0000 7f5632007640 1 --2- 172.21.15.89:0/2576337438 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f562c1538a0 0x7f562c147110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:51.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.978+0000 7f5632007640 1 -- 172.21.15.89:0/2576337438 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f562c0cf160 msgr2=0x7f562c146bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:51.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.978+0000 7f5632007640 1 --2- 172.21.15.89:0/2576337438 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f562c0cf160 0x7f562c146bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:51.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.978+0000 7f5632007640 1 -- 172.21.15.89:0/2576337438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5628009420 con 0x7f562c0ca0e0 2023-10-30T12:43:51.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.978+0000 7f5632007640 1 --2- 172.21.15.89:0/2576337438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c0ca0e0 0x7f562c140530 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f56280b3040 tx=0x7f562801ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:51.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.978+0000 7f5622ffd640 1 -- 172.21.15.89:0/2576337438 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56280b9630 con 0x7f562c0ca0e0 2023-10-30T12:43:51.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.978+0000 7f5622ffd640 1 -- 172.21.15.89:0/2576337438 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56280b97b0 con 0x7f562c0ca0e0 2023-10-30T12:43:51.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.978+0000 7f563426b640 1 -- 172.21.15.89:0/2576337438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f562c1f2430 con 0x7f562c0ca0e0 2023-10-30T12:43:51.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.978+0000 7f5622ffd640 1 -- 172.21.15.89:0/2576337438 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56280b9b60 con 0x7f562c0ca0e0 2023-10-30T12:43:51.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.978+0000 7f563426b640 1 -- 172.21.15.89:0/2576337438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f562c1f29f0 con 0x7f562c0ca0e0 2023-10-30T12:43:51.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.978+0000 7f563426b640 1 -- 172.21.15.89:0/2576337438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f562c0b2a50 con 0x7f562c0ca0e0 2023-10-30T12:43:51.982 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.982+0000 7f5622ffd640 1 -- 172.21.15.89:0/2576337438 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f56280b7040 con 0x7f562c0ca0e0 2023-10-30T12:43:51.982 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.982+0000 7f5622ffd640 1 --2- 172.21.15.89:0/2576337438 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f56100664a0 0x7f5610068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:51.982 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.982+0000 7f5622ffd640 1 -- 172.21.15.89:0/2576337438 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6406+0+0 (secure 0 0 0) 0x7f5628122270 con 0x7f562c0ca0e0 2023-10-30T12:43:51.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.982+0000 7f5631806640 1 --2- 172.21.15.89:0/2576337438 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f56100664a0 0x7f5610068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:51.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.982+0000 7f5631806640 1 --2- 172.21.15.89:0/2576337438 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f56100664a0 0x7f5610068960 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f562c0c4500 tx=0x7f561c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:51.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:51.986+0000 7f5622ffd640 1 -- 172.21.15.89:0/2576337438 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f56280ef420 con 0x7f562c0ca0e0 2023-10-30T12:43:52.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:52.166+0000 7f563426b640 1 -- 172.21.15.89:0/2576337438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f562c04e170 con 0x7f562c0ca0e0 2023-10-30T12:43:52.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:52.166+0000 7f5622ffd640 1 -- 172.21.15.89:0/2576337438 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f56280b5020 con 0x7f562c0ca0e0 2023-10-30T12:43:52.163 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:43:52.163 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:43:48.249195+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:43:52.164 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 75 2023-10-30T12:43:52.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:52.166+0000 7f563426b640 1 -- 172.21.15.89:0/2576337438 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f56100664a0 msgr2=0x7f5610068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:52.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:52.166+0000 7f563426b640 1 --2- 172.21.15.89:0/2576337438 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f56100664a0 0x7f5610068960 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f562c0c4500 tx=0x7f561c008040 comp rx=0 tx=0).stop 2023-10-30T12:43:52.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:52.166+0000 7f563426b640 1 -- 172.21.15.89:0/2576337438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c0ca0e0 msgr2=0x7f562c140530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:52.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:52.166+0000 7f563426b640 1 --2- 172.21.15.89:0/2576337438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c0ca0e0 0x7f562c140530 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f56280b3040 tx=0x7f562801ba10 comp rx=0 tx=0).stop 2023-10-30T12:43:52.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:52.166+0000 7f563426b640 1 -- 172.21.15.89:0/2576337438 shutdown_connections 2023-10-30T12:43:52.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:52.166+0000 7f563426b640 1 --2- 172.21.15.89:0/2576337438 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f56100664a0 0x7f5610068960 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:52.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:52.166+0000 7f563426b640 1 --2- 172.21.15.89:0/2576337438 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f562c0cf160 0x7f562c146bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:52.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:52.166+0000 7f563426b640 1 --2- 172.21.15.89:0/2576337438 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f562c1538a0 0x7f562c147110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:52.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:52.166+0000 7f563426b640 1 --2- 172.21.15.89:0/2576337438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c0ca0e0 0x7f562c140530 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:52.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:52.166+0000 7f563426b640 1 -- 172.21.15.89:0/2576337438 >> 172.21.15.89:0/2576337438 conn(0x7f562c06bfe0 msgr2=0x7f562c1527a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:52.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:52.166+0000 7f563426b640 1 -- 172.21.15.89:0/2576337438 shutdown_connections 2023-10-30T12:43:52.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:52.166+0000 7f563426b640 1 -- 172.21.15.89:0/2576337438 wait complete. 2023-10-30T12:43:52.184 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:43:54.185 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:43:54.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.338+0000 7fe5e45c4640 1 Processor -- start 2023-10-30T12:43:54.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.338+0000 7fe5e45c4640 1 -- start start 2023-10-30T12:43:54.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.338+0000 7fe5e45c4640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe5dc1538a0 0x7fe5dc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:54.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.338+0000 7fe5e45c4640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5dc0cf160 0x7fe5dc0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:54.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.338+0000 7fe5e45c4640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5dc0cb800 con 0x7fe5dc0ca0e0 2023-10-30T12:43:54.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.338+0000 7fe5e45c4640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5dc15dc30 con 0x7fe5dc1538a0 2023-10-30T12:43:54.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.338+0000 7fe5e45c4640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5dc0cb330 con 0x7fe5dc0cf160 2023-10-30T12:43:54.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.338+0000 7fe5e1b5f640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe5dc0ca0e0 0x7fe5dc15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46916/0 (socket says 172.21.15.89:46916) 2023-10-30T12:43:54.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.338+0000 7fe5e1b5f640 1 -- 172.21.15.89:0/2565187611 learned_addr learned my addr 172.21.15.89:0/2565187611 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:43:54.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.338+0000 7fe5e2360640 1 --2- 172.21.15.89:0/2565187611 >> v2:172.21.15.192:3300/0 conn(0x7fe5dc1538a0 0x7fe5dc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:54.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.338+0000 7fe5e2b61640 1 --2- 172.21.15.89:0/2565187611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5dc0cf160 0x7fe5dc0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:54.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.338+0000 7fe5e135e640 1 -- 172.21.15.89:0/2565187611 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2781857998 0 0) 0x7fe5dc0cb800 con 0x7fe5dc0ca0e0 2023-10-30T12:43:54.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.338+0000 7fe5e135e640 1 -- 172.21.15.89:0/2565187611 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5c8003530 con 0x7fe5dc0ca0e0 2023-10-30T12:43:54.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.338+0000 7fe5e135e640 1 -- 172.21.15.89:0/2565187611 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 673406164 0 0) 0x7fe5c8003530 con 0x7fe5dc0ca0e0 2023-10-30T12:43:54.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.338+0000 7fe5e135e640 1 -- 172.21.15.89:0/2565187611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5dc0cf160 msgr2=0x7fe5dc0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:54.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.338+0000 7fe5e135e640 1 --2- 172.21.15.89:0/2565187611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5dc0cf160 0x7fe5dc0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:54.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.338+0000 7fe5e135e640 1 -- 172.21.15.89:0/2565187611 >> v2:172.21.15.192:3300/0 conn(0x7fe5dc1538a0 msgr2=0x7fe5dc15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:54.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.338+0000 7fe5e135e640 1 --2- 172.21.15.89:0/2565187611 >> v2:172.21.15.192:3300/0 conn(0x7fe5dc1538a0 0x7fe5dc15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:54.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.338+0000 7fe5e135e640 1 -- 172.21.15.89:0/2565187611 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5dc0cad70 con 0x7fe5dc0ca0e0 2023-10-30T12:43:54.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.338+0000 7fe5e135e640 1 -- 172.21.15.89:0/2565187611 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7fe5cc001680 con 0x7fe5dc0ca0e0 2023-10-30T12:43:54.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.338+0000 7fe5e135e640 1 -- 172.21.15.89:0/2565187611 >> v1:172.21.15.89:6789/0 conn(0x7fe5dc0ca0e0 legacy=0x7fe5dc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:54.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.338+0000 7fe5e135e640 1 --2- 172.21.15.89:0/2565187611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5c8003980 0x7fe5c8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:54.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.338+0000 7fe5e135e640 1 --2- 172.21.15.89:0/2565187611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5c80047d0 0x7fe5c8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:54.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.338+0000 7fe5e135e640 1 --2- 172.21.15.89:0/2565187611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5dc0cf160 0x7fe5c8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:54.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.338+0000 7fe5e135e640 1 -- 172.21.15.89:0/2565187611 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5dc15dc30 con 0x7fe5c80047d0 2023-10-30T12:43:54.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.338+0000 7fe5e135e640 1 -- 172.21.15.89:0/2565187611 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5dc0cb330 con 0x7fe5c8003980 2023-10-30T12:43:54.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.338+0000 7fe5e135e640 1 -- 172.21.15.89:0/2565187611 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c800b990 con 0x7fe5dc0cf160 2023-10-30T12:43:54.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.338+0000 7fe5e1b5f640 1 --2- 172.21.15.89:0/2565187611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5c80047d0 0x7fe5c8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:54.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.338+0000 7fe5e2360640 1 --2- 172.21.15.89:0/2565187611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5c8003980 0x7fe5c8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:54.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.342+0000 7fe5e2b61640 1 --2- 172.21.15.89:0/2565187611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5dc0cf160 0x7fe5c8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:54.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.342+0000 7fe5e1b5f640 1 -- 172.21.15.89:0/2565187611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5dc0cf160 msgr2=0x7fe5c8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:54.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.342+0000 7fe5e1b5f640 1 --2- 172.21.15.89:0/2565187611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5dc0cf160 0x7fe5c8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:54.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.342+0000 7fe5e1b5f640 1 -- 172.21.15.89:0/2565187611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5c8003980 msgr2=0x7fe5c8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:54.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.342+0000 7fe5e1b5f640 1 --2- 172.21.15.89:0/2565187611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5c8003980 0x7fe5c8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:54.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.342+0000 7fe5e1b5f640 1 -- 172.21.15.89:0/2565187611 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c8003530 con 0x7fe5c80047d0 2023-10-30T12:43:54.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.342+0000 7fe5e1b5f640 1 --2- 172.21.15.89:0/2565187611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5c80047d0 0x7fe5c8004c50 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc00c240 tx=0x7fe5cc01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55e4974ccd8181f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:54.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.342+0000 7fe5e135e640 1 -- 172.21.15.89:0/2565187611 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc0058e0 con 0x7fe5c80047d0 2023-10-30T12:43:54.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.342+0000 7fe5e135e640 1 -- 172.21.15.89:0/2565187611 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5cc005a40 con 0x7fe5c80047d0 2023-10-30T12:43:54.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.342+0000 7fe5e135e640 1 -- 172.21.15.89:0/2565187611 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc005e10 con 0x7fe5c80047d0 2023-10-30T12:43:54.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.342+0000 7fe5e45c4640 1 -- 172.21.15.89:0/2565187611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5c80047d0 msgr2=0x7fe5c8004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:54.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.342+0000 7fe5e45c4640 1 --2- 172.21.15.89:0/2565187611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5c80047d0 0x7fe5c8004c50 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc00c240 tx=0x7fe5cc01cc60 comp rx=0 tx=0).stop 2023-10-30T12:43:54.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.342+0000 7fe5e2b61640 1 -- 172.21.15.89:0/2565187611 reap_dead start 2023-10-30T12:43:54.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.346+0000 7fe5e45c4640 1 -- 172.21.15.89:0/2565187611 shutdown_connections 2023-10-30T12:43:54.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.346+0000 7fe5e45c4640 1 -- 172.21.15.89:0/2565187611 >> 172.21.15.89:0/2565187611 conn(0x7fe5dc06bfe0 msgr2=0x7fe5dc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:54.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.346+0000 7fe5e45c4640 1 -- 172.21.15.89:0/2565187611 shutdown_connections 2023-10-30T12:43:54.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.346+0000 7fe5e45c4640 1 -- 172.21.15.89:0/2565187611 wait complete. 2023-10-30T12:43:54.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.346+0000 7fe5e45c4640 1 Processor -- start 2023-10-30T12:43:54.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.346+0000 7fe5e45c4640 1 -- start start 2023-10-30T12:43:54.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.346+0000 7fe5e45c4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5c80047d0 0x7fe5dc158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:54.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.346+0000 7fe5e45c4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5dc15d7f0 0x7fe5dc0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:54.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.346+0000 7fe5e45c4640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5dc157a20 0x7fe5dc0ce8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:54.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.346+0000 7fe5e45c4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5dc158e90 con 0x7fe5c80047d0 2023-10-30T12:43:54.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.346+0000 7fe5e45c4640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5dc1e5800 con 0x7fe5dc15d7f0 2023-10-30T12:43:54.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.346+0000 7fe5e45c4640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5dc1e34f0 con 0x7fe5dc157a20 2023-10-30T12:43:54.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.346+0000 7fe5e2360640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5c80047d0 0x7fe5dc158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:54.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.346+0000 7fe5e2360640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5c80047d0 0x7fe5dc158450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33012/0 (socket says 172.21.15.89:33012) 2023-10-30T12:43:54.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.346+0000 7fe5e2360640 1 -- 172.21.15.89:0/2599909939 learned_addr learned my addr 172.21.15.89:0/2599909939 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:43:54.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.346+0000 7fe5e1b5f640 1 --2- 172.21.15.89:0/2599909939 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5dc15d7f0 0x7fe5dc0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:54.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.346+0000 7fe5e2b61640 1 --2- 172.21.15.89:0/2599909939 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5dc157a20 0x7fe5dc0ce8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:54.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.346+0000 7fe5e2360640 1 -- 172.21.15.89:0/2599909939 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5dc157a20 msgr2=0x7fe5dc0ce8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:54.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.346+0000 7fe5e2360640 1 --2- 172.21.15.89:0/2599909939 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5dc157a20 0x7fe5dc0ce8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:54.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.346+0000 7fe5e2360640 1 -- 172.21.15.89:0/2599909939 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5dc15d7f0 msgr2=0x7fe5dc0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:54.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.346+0000 7fe5e2360640 1 --2- 172.21.15.89:0/2599909939 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5dc15d7f0 0x7fe5dc0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:54.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.346+0000 7fe5e2360640 1 -- 172.21.15.89:0/2599909939 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5dc1e3740 con 0x7fe5c80047d0 2023-10-30T12:43:54.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.346+0000 7fe5e2360640 1 --2- 172.21.15.89:0/2599909939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5c80047d0 0x7fe5dc158450 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fe5d80119d0 tx=0x7fe5d8011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:54.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.350+0000 7fe5d2ffd640 1 -- 172.21.15.89:0/2599909939 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d800fb50 con 0x7fe5c80047d0 2023-10-30T12:43:54.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.350+0000 7fe5d2ffd640 1 -- 172.21.15.89:0/2599909939 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d800fcb0 con 0x7fe5c80047d0 2023-10-30T12:43:54.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.350+0000 7fe5d2ffd640 1 -- 172.21.15.89:0/2599909939 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8014440 con 0x7fe5c80047d0 2023-10-30T12:43:54.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.350+0000 7fe5e45c4640 1 -- 172.21.15.89:0/2599909939 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5dc1e3a20 con 0x7fe5c80047d0 2023-10-30T12:43:54.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.350+0000 7fe5e45c4640 1 -- 172.21.15.89:0/2599909939 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5dc1e3fe0 con 0x7fe5c80047d0 2023-10-30T12:43:54.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.350+0000 7fe5e45c4640 1 -- 172.21.15.89:0/2599909939 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5dc0ccfb0 con 0x7fe5c80047d0 2023-10-30T12:43:54.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.350+0000 7fe5d2ffd640 1 -- 172.21.15.89:0/2599909939 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe5d80145a0 con 0x7fe5c80047d0 2023-10-30T12:43:54.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.350+0000 7fe5d2ffd640 1 --2- 172.21.15.89:0/2599909939 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe5c00664a0 0x7fe5c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:54.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.350+0000 7fe5d2ffd640 1 -- 172.21.15.89:0/2599909939 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6406+0+0 (secure 0 0 0) 0x7fe5d8084ab0 con 0x7fe5c80047d0 2023-10-30T12:43:54.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.350+0000 7fe5e1b5f640 1 --2- 172.21.15.89:0/2599909939 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe5c00664a0 0x7fe5c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:54.357 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.354+0000 7fe5e1b5f640 1 --2- 172.21.15.89:0/2599909939 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe5c00664a0 0x7fe5c0068960 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc0aa000 tx=0x7fe5cc001a80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:54.357 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.354+0000 7fe5d2ffd640 1 -- 172.21.15.89:0/2599909939 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe5d8051c90 con 0x7fe5c80047d0 2023-10-30T12:43:54.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.538+0000 7fe5e45c4640 1 -- 172.21.15.89:0/2599909939 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5dc04e170 con 0x7fe5c80047d0 2023-10-30T12:43:54.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.538+0000 7fe5d2ffd640 1 -- 172.21.15.89:0/2599909939 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+5735 (secure 0 0 0) 0x7fe5d8022020 con 0x7fe5c80047d0 2023-10-30T12:43:54.537 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:43:54.538 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:43:53.317932+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:43:54.538 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 76 2023-10-30T12:43:54.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.542+0000 7fe5e45c4640 1 -- 172.21.15.89:0/2599909939 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe5c00664a0 msgr2=0x7fe5c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:54.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.542+0000 7fe5e45c4640 1 --2- 172.21.15.89:0/2599909939 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe5c00664a0 0x7fe5c0068960 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc0aa000 tx=0x7fe5cc001a80 comp rx=0 tx=0).stop 2023-10-30T12:43:54.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.542+0000 7fe5e45c4640 1 -- 172.21.15.89:0/2599909939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5c80047d0 msgr2=0x7fe5dc158450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:54.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.542+0000 7fe5e45c4640 1 --2- 172.21.15.89:0/2599909939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5c80047d0 0x7fe5dc158450 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fe5d80119d0 tx=0x7fe5d8011e90 comp rx=0 tx=0).stop 2023-10-30T12:43:54.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.542+0000 7fe5e45c4640 1 -- 172.21.15.89:0/2599909939 shutdown_connections 2023-10-30T12:43:54.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.542+0000 7fe5e45c4640 1 --2- 172.21.15.89:0/2599909939 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe5c00664a0 0x7fe5c0068960 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:54.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.542+0000 7fe5e45c4640 1 --2- 172.21.15.89:0/2599909939 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5dc15d7f0 0x7fe5dc0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:54.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.542+0000 7fe5e45c4640 1 --2- 172.21.15.89:0/2599909939 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe5dc157a20 0x7fe5dc0ce8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:54.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.542+0000 7fe5e45c4640 1 --2- 172.21.15.89:0/2599909939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe5c80047d0 0x7fe5dc158450 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:54.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.542+0000 7fe5e45c4640 1 -- 172.21.15.89:0/2599909939 >> 172.21.15.89:0/2599909939 conn(0x7fe5dc06bfe0 msgr2=0x7fe5dc1510d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:54.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.542+0000 7fe5e45c4640 1 -- 172.21.15.89:0/2599909939 shutdown_connections 2023-10-30T12:43:54.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:54.542+0000 7fe5e45c4640 1 -- 172.21.15.89:0/2599909939 wait complete. 2023-10-30T12:43:54.558 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:43:56.560 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:43:56.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.710+0000 7ff3fe89c640 1 Processor -- start 2023-10-30T12:43:56.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.710+0000 7ff3fe89c640 1 -- start start 2023-10-30T12:43:56.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.714+0000 7ff3fe89c640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3f8159e60 0x7ff3f806f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:56.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.714+0000 7ff3fe89c640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff3f806a720 0x7ff3f806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:56.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.714+0000 7ff3fe89c640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f806fa80 con 0x7ff3f80bfec0 2023-10-30T12:43:56.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.714+0000 7ff3fe89c640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3f8071290 con 0x7ff3f806a720 2023-10-30T12:43:56.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.714+0000 7ff3fe89c640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f8071570 con 0x7ff3f8159e60 2023-10-30T12:43:56.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.714+0000 7ff3f7fff640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3f8159e60 0x7ff3f806f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:56.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.714+0000 7ff3fce39640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff3f806a720 0x7ff3f806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:56.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.714+0000 7ff3f77fe640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7ff3f80bfec0 0x7ff3f80c1a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:57242/0 (socket says 172.21.15.89:57242) 2023-10-30T12:43:56.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.714+0000 7ff3f77fe640 1 -- 172.21.15.89:0/1350629265 learned_addr learned my addr 172.21.15.89:0/1350629265 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:43:56.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.714+0000 7ff3f6ffd640 1 -- 172.21.15.89:0/1350629265 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2321936023 0 0) 0x7ff3f806fa80 con 0x7ff3f80bfec0 2023-10-30T12:43:56.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.714+0000 7ff3f6ffd640 1 -- 172.21.15.89:0/1350629265 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3e0005740 con 0x7ff3f80bfec0 2023-10-30T12:43:56.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.714+0000 7ff3f7fff640 1 -- 172.21.15.89:0/1350629265 >> v2:172.21.15.192:3300/0 conn(0x7ff3f806a720 msgr2=0x7ff3f806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:56.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.714+0000 7ff3f7fff640 1 --2- 172.21.15.89:0/1350629265 >> v2:172.21.15.192:3300/0 conn(0x7ff3f806a720 0x7ff3f806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:56.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.714+0000 7ff3f7fff640 1 -- 172.21.15.89:0/1350629265 >> v1:172.21.15.192:6789/0 conn(0x7ff3f80bfec0 legacy=0x7ff3f80c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:56.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.714+0000 7ff3f7fff640 1 -- 172.21.15.89:0/1350629265 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f8071850 con 0x7ff3f8159e60 2023-10-30T12:43:56.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.714+0000 7ff3fce39640 1 --2- 172.21.15.89:0/1350629265 >> v2:172.21.15.192:3300/0 conn(0x7ff3f806a720 0x7ff3f806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:43:56.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.718+0000 7ff3f7fff640 1 --2- 172.21.15.89:0/1350629265 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3f8159e60 0x7ff3f806f540 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff3e800a960 tx=0x7ff3e801ef30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc072fa12776894f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:56.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.718+0000 7ff3f6ffd640 1 -- 172.21.15.89:0/1350629265 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e80a8d40 con 0x7ff3f8159e60 2023-10-30T12:43:56.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.718+0000 7ff3f6ffd640 1 -- 172.21.15.89:0/1350629265 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e80b0440 con 0x7ff3f8159e60 2023-10-30T12:43:56.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.718+0000 7ff3f6ffd640 1 -- 172.21.15.89:0/1350629265 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e80a8ea0 con 0x7ff3f8159e60 2023-10-30T12:43:56.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.718+0000 7ff3fe89c640 1 -- 172.21.15.89:0/1350629265 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3f8159e60 msgr2=0x7ff3f806f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:56.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.718+0000 7ff3fe89c640 1 --2- 172.21.15.89:0/1350629265 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3f8159e60 0x7ff3f806f540 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff3e800a960 tx=0x7ff3e801ef30 comp rx=0 tx=0).stop 2023-10-30T12:43:56.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.718+0000 7ff3fe89c640 1 -- 172.21.15.89:0/1350629265 shutdown_connections 2023-10-30T12:43:56.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.718+0000 7ff3fe89c640 1 --2- 172.21.15.89:0/1350629265 >> v2:172.21.15.192:3300/0 conn(0x7ff3f806a720 0x7ff3f806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:56.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.718+0000 7ff3fe89c640 1 --2- 172.21.15.89:0/1350629265 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3f8159e60 0x7ff3f806f540 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:56.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.718+0000 7ff3fe89c640 1 -- 172.21.15.89:0/1350629265 >> 172.21.15.89:0/1350629265 conn(0x7ff3f80c7910 msgr2=0x7ff3f80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:56.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.718+0000 7ff3fe89c640 1 -- 172.21.15.89:0/1350629265 shutdown_connections 2023-10-30T12:43:56.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.718+0000 7ff3fe89c640 1 -- 172.21.15.89:0/1350629265 wait complete. 2023-10-30T12:43:56.718 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.718+0000 7ff3fe89c640 1 Processor -- start 2023-10-30T12:43:56.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.718+0000 7ff3fe89c640 1 -- start start 2023-10-30T12:43:56.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.722+0000 7ff3fe89c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3f806a720 0x7ff3f8151b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:56.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.722+0000 7ff3fe89c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3f80bfec0 0x7ff3f8152810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:56.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.722+0000 7ff3fe89c640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3f8159e60 0x7ff3f8152d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:56.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.722+0000 7ff3fe89c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f81597f0 con 0x7ff3f806a720 2023-10-30T12:43:56.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.722+0000 7ff3fe89c640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f814d680 con 0x7ff3f80bfec0 2023-10-30T12:43:56.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.722+0000 7ff3fe89c640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f814d920 con 0x7ff3f8159e60 2023-10-30T12:43:56.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.722+0000 7ff3f7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3f806a720 0x7ff3f8151b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:56.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.722+0000 7ff3f7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3f806a720 0x7ff3f8151b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33016/0 (socket says 172.21.15.89:33016) 2023-10-30T12:43:56.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.722+0000 7ff3f7fff640 1 -- 172.21.15.89:0/465025016 learned_addr learned my addr 172.21.15.89:0/465025016 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:43:56.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.722+0000 7ff3fce39640 1 --2- 172.21.15.89:0/465025016 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3f8159e60 0x7ff3f8152d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:56.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.722+0000 7ff3f77fe640 1 --2- 172.21.15.89:0/465025016 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3f80bfec0 0x7ff3f8152810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:56.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.722+0000 7ff3f7fff640 1 -- 172.21.15.89:0/465025016 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3f8159e60 msgr2=0x7ff3f8152d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:56.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.722+0000 7ff3f7fff640 1 --2- 172.21.15.89:0/465025016 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3f8159e60 0x7ff3f8152d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:56.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.722+0000 7ff3f7fff640 1 -- 172.21.15.89:0/465025016 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3f80bfec0 msgr2=0x7ff3f8152810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:56.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.722+0000 7ff3f7fff640 1 --2- 172.21.15.89:0/465025016 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3f80bfec0 0x7ff3f8152810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:56.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.722+0000 7ff3f7fff640 1 -- 172.21.15.89:0/465025016 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f0009a70 con 0x7ff3f806a720 2023-10-30T12:43:56.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.722+0000 7ff3f7fff640 1 --2- 172.21.15.89:0/465025016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3f806a720 0x7ff3f8151b50 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7ff3e80b3040 tx=0x7ff3e801bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:56.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.722+0000 7ff3f4ff9640 1 -- 172.21.15.89:0/465025016 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e80b0b80 con 0x7ff3f806a720 2023-10-30T12:43:56.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.722+0000 7ff3f4ff9640 1 -- 172.21.15.89:0/465025016 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e80b0d00 con 0x7ff3f806a720 2023-10-30T12:43:56.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.722+0000 7ff3f4ff9640 1 -- 172.21.15.89:0/465025016 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e80ba680 con 0x7ff3f806a720 2023-10-30T12:43:56.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.722+0000 7ff3fe89c640 1 -- 172.21.15.89:0/465025016 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3e800a5c0 con 0x7ff3f806a720 2023-10-30T12:43:56.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.722+0000 7ff3fe89c640 1 -- 172.21.15.89:0/465025016 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f8150a30 con 0x7ff3f806a720 2023-10-30T12:43:56.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.722+0000 7ff3fe89c640 1 -- 172.21.15.89:0/465025016 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3f804e170 con 0x7ff3f806a720 2023-10-30T12:43:56.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.726+0000 7ff3f4ff9640 1 -- 172.21.15.89:0/465025016 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff3e80ba7e0 con 0x7ff3f806a720 2023-10-30T12:43:56.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.726+0000 7ff3f4ff9640 1 --2- 172.21.15.89:0/465025016 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff3e40664d0 0x7ff3e4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:56.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.726+0000 7ff3f4ff9640 1 -- 172.21.15.89:0/465025016 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6406+0+0 (secure 0 0 0) 0x7ff3e81220c0 con 0x7ff3f806a720 2023-10-30T12:43:56.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.726+0000 7ff3f77fe640 1 --2- 172.21.15.89:0/465025016 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff3e40664d0 0x7ff3e4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:56.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.726+0000 7ff3f77fe640 1 --2- 172.21.15.89:0/465025016 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff3e40664d0 0x7ff3e4068990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff3f80c14f0 tx=0x7ff3ec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:56.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.730+0000 7ff3f4ff9640 1 -- 172.21.15.89:0/465025016 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff3e80ef390 con 0x7ff3f806a720 2023-10-30T12:43:56.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.906+0000 7ff3fe89c640 1 -- 172.21.15.89:0/465025016 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3f80c8db0 con 0x7ff3f806a720 2023-10-30T12:43:56.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.906+0000 7ff3f4ff9640 1 -- 172.21.15.89:0/465025016 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+5735 (secure 0 0 0) 0x7ff3e80f29b0 con 0x7ff3f806a720 2023-10-30T12:43:56.906 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:43:56.906 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:43:53.317932+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:43:56.906 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 76 2023-10-30T12:43:56.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.910+0000 7ff3fe89c640 1 -- 172.21.15.89:0/465025016 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff3e40664d0 msgr2=0x7ff3e4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:56.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.910+0000 7ff3fe89c640 1 --2- 172.21.15.89:0/465025016 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff3e40664d0 0x7ff3e4068990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff3f80c14f0 tx=0x7ff3ec009040 comp rx=0 tx=0).stop 2023-10-30T12:43:56.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.910+0000 7ff3fe89c640 1 -- 172.21.15.89:0/465025016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3f806a720 msgr2=0x7ff3f8151b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:56.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.910+0000 7ff3fe89c640 1 --2- 172.21.15.89:0/465025016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3f806a720 0x7ff3f8151b50 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7ff3e80b3040 tx=0x7ff3e801bcd0 comp rx=0 tx=0).stop 2023-10-30T12:43:56.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.910+0000 7ff3fe89c640 1 -- 172.21.15.89:0/465025016 shutdown_connections 2023-10-30T12:43:56.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.910+0000 7ff3fe89c640 1 --2- 172.21.15.89:0/465025016 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff3e40664d0 0x7ff3e4068990 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:56.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.910+0000 7ff3fe89c640 1 --2- 172.21.15.89:0/465025016 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff3f80bfec0 0x7ff3f8152810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:56.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.910+0000 7ff3fe89c640 1 --2- 172.21.15.89:0/465025016 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3f8159e60 0x7ff3f8152d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:56.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.910+0000 7ff3fe89c640 1 --2- 172.21.15.89:0/465025016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3f806a720 0x7ff3f8151b50 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:56.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.910+0000 7ff3fe89c640 1 -- 172.21.15.89:0/465025016 >> 172.21.15.89:0/465025016 conn(0x7ff3f80c7910 msgr2=0x7ff3f8073480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:56.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.910+0000 7ff3fe89c640 1 -- 172.21.15.89:0/465025016 shutdown_connections 2023-10-30T12:43:56.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:56.910+0000 7ff3fe89c640 1 -- 172.21.15.89:0/465025016 wait complete. 2023-10-30T12:43:56.927 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:43:58.928 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:43:59.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.078+0000 7f10e169a640 1 Processor -- start 2023-10-30T12:43:59.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.078+0000 7f10e169a640 1 -- start start 2023-10-30T12:43:59.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.082+0000 7f10e169a640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f10dc0c6b80 0x7f10dc0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:59.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.082+0000 7f10e169a640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10dc0c6f60 0x7f10dc0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:59.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.082+0000 7f10e169a640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10dc0c0a50 con 0x7f10dc0c8430 2023-10-30T12:43:59.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.082+0000 7f10e169a640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10dc06f6e0 con 0x7f10dc0c6b80 2023-10-30T12:43:59.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.082+0000 7f10e169a640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc06f9c0 con 0x7f10dc0c6f60 2023-10-30T12:43:59.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.082+0000 7f10daffd640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f10dc0c6b80 0x7f10dc0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:59.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.082+0000 7f10db7fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f10dc0c8430 0x7f10dc13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46918/0 (socket says 172.21.15.89:46918) 2023-10-30T12:43:59.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.082+0000 7f10daffd640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f10dc0c6b80 0x7f10dc0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33018/0 (socket says 172.21.15.89:33018) 2023-10-30T12:43:59.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.082+0000 7f10db7fe640 1 -- 172.21.15.89:0/1963405349 learned_addr learned my addr 172.21.15.89:0/1963405349 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:43:59.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.082+0000 7f10d27fc640 1 --2- 172.21.15.89:0/1963405349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10dc0c6f60 0x7f10dc0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:59.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.082+0000 7f10daffd640 1 -- 172.21.15.89:0/1963405349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10dc0c6f60 msgr2=0x7f10dc0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:59.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.082+0000 7f10daffd640 1 --2- 172.21.15.89:0/1963405349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10dc0c6f60 0x7f10dc0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:59.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.082+0000 7f10daffd640 1 -- 172.21.15.89:0/1963405349 >> v1:172.21.15.89:6789/0 conn(0x7f10dc0c8430 legacy=0x7f10dc13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:59.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.082+0000 7f10daffd640 1 -- 172.21.15.89:0/1963405349 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10dc06fca0 con 0x7f10dc0c6b80 2023-10-30T12:43:59.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.082+0000 7f10da7fc640 1 -- 172.21.15.89:0/1963405349 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1419641676 0 0) 0x7f10dc0c0a50 con 0x7f10dc0c8430 2023-10-30T12:43:59.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.082+0000 7f10daffd640 1 --2- 172.21.15.89:0/1963405349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10dc0c6b80 0x7f10dc0bfec0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f10cc0097c0 tx=0x7f10cc01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6192811f2c6ddc4a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:59.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.086+0000 7f10da7fc640 1 -- 172.21.15.89:0/1963405349 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc0ab600 con 0x7f10dc0c6b80 2023-10-30T12:43:59.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.086+0000 7f10da7fc640 1 -- 172.21.15.89:0/1963405349 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10cc0a7050 con 0x7f10dc0c6b80 2023-10-30T12:43:59.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.086+0000 7f10da7fc640 1 -- 172.21.15.89:0/1963405349 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc0ab980 con 0x7f10dc0c6b80 2023-10-30T12:43:59.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.086+0000 7f10e169a640 1 -- 172.21.15.89:0/1963405349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10dc0c6b80 msgr2=0x7f10dc0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:59.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.086+0000 7f10e169a640 1 --2- 172.21.15.89:0/1963405349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10dc0c6b80 0x7f10dc0bfec0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f10cc0097c0 tx=0x7f10cc01c3f0 comp rx=0 tx=0).stop 2023-10-30T12:43:59.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.086+0000 7f10e169a640 1 -- 172.21.15.89:0/1963405349 shutdown_connections 2023-10-30T12:43:59.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.086+0000 7f10e169a640 1 --2- 172.21.15.89:0/1963405349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10dc0c6f60 0x7f10dc0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:59.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.086+0000 7f10e169a640 1 --2- 172.21.15.89:0/1963405349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10dc0c6b80 0x7f10dc0bfec0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:59.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.086+0000 7f10e169a640 1 -- 172.21.15.89:0/1963405349 >> 172.21.15.89:0/1963405349 conn(0x7f10dc0c7910 msgr2=0x7f10dc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:59.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.086+0000 7f10e169a640 1 -- 172.21.15.89:0/1963405349 shutdown_connections 2023-10-30T12:43:59.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.086+0000 7f10e169a640 1 -- 172.21.15.89:0/1963405349 wait complete. 2023-10-30T12:43:59.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.086+0000 7f10e169a640 1 Processor -- start 2023-10-30T12:43:59.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.086+0000 7f10e169a640 1 -- start start 2023-10-30T12:43:59.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.086+0000 7f10e169a640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10dc0c6b80 0x7f10dc13ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:59.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.086+0000 7f10e169a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10dc0c6f60 0x7f10dc150160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:59.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.086+0000 7f10e169a640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10dc0c8430 0x7f10dc1506a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:59.091 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.086+0000 7f10e169a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc0c1350 con 0x7f10dc0c6f60 2023-10-30T12:43:59.091 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.086+0000 7f10e169a640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc06ab60 con 0x7f10dc0c8430 2023-10-30T12:43:59.091 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.086+0000 7f10e169a640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc158730 con 0x7f10dc0c6b80 2023-10-30T12:43:59.091 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.086+0000 7f10d27fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10dc0c6f60 0x7f10dc150160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:59.091 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.086+0000 7f10d27fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10dc0c6f60 0x7f10dc150160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33022/0 (socket says 172.21.15.89:33022) 2023-10-30T12:43:59.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.090+0000 7f10d27fc640 1 -- 172.21.15.89:0/3858952367 learned_addr learned my addr 172.21.15.89:0/3858952367 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:43:59.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.090+0000 7f10db7fe640 1 --2- 172.21.15.89:0/3858952367 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10dc0c8430 0x7f10dc1506a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:59.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.090+0000 7f10daffd640 1 --2- 172.21.15.89:0/3858952367 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10dc0c6b80 0x7f10dc13ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:59.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.090+0000 7f10d27fc640 1 -- 172.21.15.89:0/3858952367 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10dc0c6b80 msgr2=0x7f10dc13ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:59.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.090+0000 7f10d27fc640 1 --2- 172.21.15.89:0/3858952367 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10dc0c6b80 0x7f10dc13ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:59.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.090+0000 7f10d27fc640 1 -- 172.21.15.89:0/3858952367 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10dc0c8430 msgr2=0x7f10dc1506a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:59.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.090+0000 7f10d27fc640 1 --2- 172.21.15.89:0/3858952367 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10dc0c8430 0x7f10dc1506a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:59.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.090+0000 7f10d27fc640 1 -- 172.21.15.89:0/3858952367 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10cc009420 con 0x7f10dc0c6f60 2023-10-30T12:43:59.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.090+0000 7f10d27fc640 1 --2- 172.21.15.89:0/3858952367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10dc0c6f60 0x7f10dc150160 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f10c400d800 tx=0x7f10c400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:59.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.090+0000 7f10d3fff640 1 -- 172.21.15.89:0/3858952367 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c400b6c0 con 0x7f10dc0c6f60 2023-10-30T12:43:59.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.090+0000 7f10d3fff640 1 -- 172.21.15.89:0/3858952367 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c400b820 con 0x7f10dc0c6f60 2023-10-30T12:43:59.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.090+0000 7f10e169a640 1 -- 172.21.15.89:0/3858952367 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10dc06ae00 con 0x7f10dc0c6f60 2023-10-30T12:43:59.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.090+0000 7f10d3fff640 1 -- 172.21.15.89:0/3858952367 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c400bb90 con 0x7f10dc0c6f60 2023-10-30T12:43:59.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.090+0000 7f10e169a640 1 -- 172.21.15.89:0/3858952367 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10dc159750 con 0x7f10dc0c6f60 2023-10-30T12:43:59.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.090+0000 7f10e169a640 1 -- 172.21.15.89:0/3858952367 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10dc159940 con 0x7f10dc0c6f60 2023-10-30T12:43:59.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.090+0000 7f10d3fff640 1 -- 172.21.15.89:0/3858952367 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f10c400bcf0 con 0x7f10dc0c6f60 2023-10-30T12:43:59.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.094+0000 7f10d3fff640 1 --2- 172.21.15.89:0/3858952367 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f10bc0664a0 0x7f10bc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:59.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.094+0000 7f10d3fff640 1 -- 172.21.15.89:0/3858952367 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6406+0+0 (secure 0 0 0) 0x7f10c4084800 con 0x7f10dc0c6f60 2023-10-30T12:43:59.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.094+0000 7f10daffd640 1 --2- 172.21.15.89:0/3858952367 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f10bc0664a0 0x7f10bc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:59.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.094+0000 7f10daffd640 1 --2- 172.21.15.89:0/3858952367 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f10bc0664a0 0x7f10bc068960 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f10cc0ad000 tx=0x7f10cc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:59.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.098+0000 7f10d3fff640 1 -- 172.21.15.89:0/3858952367 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f10c4051890 con 0x7f10dc0c6f60 2023-10-30T12:43:59.276 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.278+0000 7f10e169a640 1 -- 172.21.15.89:0/3858952367 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10dc04e170 con 0x7f10dc0c6f60 2023-10-30T12:43:59.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.278+0000 7f10d3fff640 1 -- 172.21.15.89:0/3858952367 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+5735 (secure 0 0 0) 0x7f10c4011340 con 0x7f10dc0c6f60 2023-10-30T12:43:59.277 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:43:59.277 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:43:53.317932+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:43:59.278 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 76 2023-10-30T12:43:59.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.282+0000 7f10e169a640 1 -- 172.21.15.89:0/3858952367 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f10bc0664a0 msgr2=0x7f10bc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:59.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.282+0000 7f10e169a640 1 --2- 172.21.15.89:0/3858952367 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f10bc0664a0 0x7f10bc068960 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f10cc0ad000 tx=0x7f10cc008040 comp rx=0 tx=0).stop 2023-10-30T12:43:59.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.282+0000 7f10e169a640 1 -- 172.21.15.89:0/3858952367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10dc0c6f60 msgr2=0x7f10dc150160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:59.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.282+0000 7f10e169a640 1 --2- 172.21.15.89:0/3858952367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10dc0c6f60 0x7f10dc150160 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f10c400d800 tx=0x7f10c400dcc0 comp rx=0 tx=0).stop 2023-10-30T12:43:59.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.282+0000 7f10e169a640 1 -- 172.21.15.89:0/3858952367 shutdown_connections 2023-10-30T12:43:59.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.282+0000 7f10e169a640 1 --2- 172.21.15.89:0/3858952367 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f10bc0664a0 0x7f10bc068960 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:59.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.282+0000 7f10e169a640 1 --2- 172.21.15.89:0/3858952367 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10dc0c8430 0x7f10dc1506a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:59.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.282+0000 7f10e169a640 1 --2- 172.21.15.89:0/3858952367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10dc0c6f60 0x7f10dc150160 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:59.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.282+0000 7f10e169a640 1 --2- 172.21.15.89:0/3858952367 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10dc0c6b80 0x7f10dc13ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:59.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.282+0000 7f10e169a640 1 -- 172.21.15.89:0/3858952367 >> 172.21.15.89:0/3858952367 conn(0x7f10dc0c7910 msgr2=0x7f10dc0bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:59.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.282+0000 7f10e169a640 1 -- 172.21.15.89:0/3858952367 shutdown_connections 2023-10-30T12:43:59.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:43:59.282+0000 7f10e169a640 1 -- 172.21.15.89:0/3858952367 wait complete. 2023-10-30T12:43:59.298 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:44:01.299 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:44:01.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.450+0000 7f42048a9640 1 Processor -- start 2023-10-30T12:44:01.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.450+0000 7f42048a9640 1 -- start start 2023-10-30T12:44:01.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.450+0000 7f42048a9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42000c6bd0 0x7f420006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:01.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.450+0000 7f42048a9640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4200066cf0 con 0x7f420006f7a0 2023-10-30T12:44:01.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.450+0000 7f42048a9640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42000c70d0 con 0x7f4200071110 2023-10-30T12:44:01.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.450+0000 7f42048a9640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42000700c0 con 0x7f42000c6bd0 2023-10-30T12:44:01.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.450+0000 7f41fe59c640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f420006f7a0 0x7f4200069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46766/0 (socket says 172.21.15.89:46766) 2023-10-30T12:44:01.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.450+0000 7f41fe59c640 1 -- 172.21.15.89:0/1986122201 learned_addr learned my addr 172.21.15.89:0/1986122201 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:44:01.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.450+0000 7f41fdd9b640 1 --2- 172.21.15.89:0/1986122201 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42000c6bd0 0x7f420006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:01.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.450+0000 7f41fd59a640 1 -- 172.21.15.89:0/1986122201 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3279525103 0 0) 0x7f4200066cf0 con 0x7f420006f7a0 2023-10-30T12:44:01.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.450+0000 7f41fd59a640 1 -- 172.21.15.89:0/1986122201 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41e8003530 con 0x7f420006f7a0 2023-10-30T12:44:01.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.454+0000 7f41fd59a640 1 -- 172.21.15.89:0/1986122201 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 888454739 0 0) 0x7f41e8003530 con 0x7f420006f7a0 2023-10-30T12:44:01.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.454+0000 7f41fd59a640 1 -- 172.21.15.89:0/1986122201 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42000c6bd0 msgr2=0x7f420006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:44:01.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.454+0000 7f41fd59a640 1 --2- 172.21.15.89:0/1986122201 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42000c6bd0 0x7f420006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:01.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.454+0000 7f41fd59a640 1 -- 172.21.15.89:0/1986122201 >> v1:172.21.15.192:6789/0 conn(0x7f4200071110 legacy=0x7f4200158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:01.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.454+0000 7f41fd59a640 1 -- 172.21.15.89:0/1986122201 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42000c7390 con 0x7f420006f7a0 2023-10-30T12:44:01.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.454+0000 7f41fd59a640 1 -- 172.21.15.89:0/1986122201 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f41f0001680 con 0x7f420006f7a0 2023-10-30T12:44:01.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.454+0000 7f41fd59a640 1 -- 172.21.15.89:0/1986122201 >> v1:172.21.15.89:6789/0 conn(0x7f420006f7a0 legacy=0x7f4200069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:01.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.454+0000 7f41fd59a640 1 --2- 172.21.15.89:0/1986122201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41e80039b0 0x7f41e8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:01.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.454+0000 7f41fd59a640 1 --2- 172.21.15.89:0/1986122201 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41e80047d0 0x7f41e8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:01.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.454+0000 7f41fd59a640 1 --2- 172.21.15.89:0/1986122201 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42000c6bd0 0x7f41e8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:01.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.454+0000 7f41fd59a640 1 -- 172.21.15.89:0/1986122201 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42000700c0 con 0x7f41e80039b0 2023-10-30T12:44:01.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.454+0000 7f41fd59a640 1 -- 172.21.15.89:0/1986122201 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e800bad0 con 0x7f41e80047d0 2023-10-30T12:44:01.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.454+0000 7f41fd59a640 1 -- 172.21.15.89:0/1986122201 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41e800bc90 con 0x7f42000c6bd0 2023-10-30T12:44:01.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.454+0000 7f41fe59c640 1 --2- 172.21.15.89:0/1986122201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41e80039b0 0x7f41e8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:01.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.454+0000 7f41fdd9b640 1 --2- 172.21.15.89:0/1986122201 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41e80047d0 0x7f41e8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:01.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.454+0000 7f41fed9d640 1 --2- 172.21.15.89:0/1986122201 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42000c6bd0 0x7f41e8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:01.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.454+0000 7f41fe59c640 1 -- 172.21.15.89:0/1986122201 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42000c6bd0 msgr2=0x7f41e8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:44:01.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.454+0000 7f41fe59c640 1 --2- 172.21.15.89:0/1986122201 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42000c6bd0 0x7f41e8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:01.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.454+0000 7f41fe59c640 1 -- 172.21.15.89:0/1986122201 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41e80047d0 msgr2=0x7f41e8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:44:01.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.454+0000 7f41fe59c640 1 --2- 172.21.15.89:0/1986122201 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41e80047d0 0x7f41e8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:01.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.454+0000 7f41fe59c640 1 -- 172.21.15.89:0/1986122201 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e8003530 con 0x7f41e80039b0 2023-10-30T12:44:01.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.454+0000 7f41fe59c640 1 --2- 172.21.15.89:0/1986122201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41e80039b0 0x7f41e8003dd0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f41f000c210 tx=0x7f41f001f350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3f30ddea45e47b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:01.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.458+0000 7f41fd59a640 1 -- 172.21.15.89:0/1986122201 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f0005ae0 con 0x7f41e80039b0 2023-10-30T12:44:01.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.458+0000 7f41fd59a640 1 -- 172.21.15.89:0/1986122201 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f00ab070 con 0x7f41e80039b0 2023-10-30T12:44:01.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.458+0000 7f41fd59a640 1 -- 172.21.15.89:0/1986122201 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f0007070 con 0x7f41e80039b0 2023-10-30T12:44:01.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.458+0000 7f42048a9640 1 -- 172.21.15.89:0/1986122201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41e80039b0 msgr2=0x7f41e8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:01.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.458+0000 7f42048a9640 1 --2- 172.21.15.89:0/1986122201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41e80039b0 0x7f41e8003dd0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f41f000c210 tx=0x7f41f001f350 comp rx=0 tx=0).stop 2023-10-30T12:44:01.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.458+0000 7f41fed9d640 1 -- 172.21.15.89:0/1986122201 reap_dead start 2023-10-30T12:44:01.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.458+0000 7f42048a9640 1 -- 172.21.15.89:0/1986122201 shutdown_connections 2023-10-30T12:44:01.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.458+0000 7f42048a9640 1 -- 172.21.15.89:0/1986122201 >> 172.21.15.89:0/1986122201 conn(0x7f42000c7910 msgr2=0x7f42000c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:01.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.458+0000 7f42048a9640 1 -- 172.21.15.89:0/1986122201 shutdown_connections 2023-10-30T12:44:01.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.458+0000 7f42048a9640 1 -- 172.21.15.89:0/1986122201 wait complete. 2023-10-30T12:44:01.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.458+0000 7f42048a9640 1 Processor -- start 2023-10-30T12:44:01.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.458+0000 7f42048a9640 1 -- start start 2023-10-30T12:44:01.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.458+0000 7f42048a9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41e80039b0 0x7f42001555a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:01.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.458+0000 7f42048a9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42001588a0 0x7f42000c37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:01.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.458+0000 7f42048a9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42001468c0 0x7f4200146d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:01.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.458+0000 7f42048a9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4200157690 con 0x7f41e80039b0 2023-10-30T12:44:01.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.458+0000 7f42048a9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4200155bd0 con 0x7f42001588a0 2023-10-30T12:44:01.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.458+0000 7f42048a9640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42001578a0 con 0x7f42001468c0 2023-10-30T12:44:01.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.458+0000 7f41fe59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41e80039b0 0x7f42001555a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:01.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.462+0000 7f41fe59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41e80039b0 0x7f42001555a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51244/0 (socket says 172.21.15.89:51244) 2023-10-30T12:44:01.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.462+0000 7f41fe59c640 1 -- 172.21.15.89:0/2205120044 learned_addr learned my addr 172.21.15.89:0/2205120044 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:44:01.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.462+0000 7f41fed9d640 1 --2- 172.21.15.89:0/2205120044 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42001468c0 0x7f4200146d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:01.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.462+0000 7f41fdd9b640 1 --2- 172.21.15.89:0/2205120044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42001588a0 0x7f42000c37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:01.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.462+0000 7f41fe59c640 1 -- 172.21.15.89:0/2205120044 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42001468c0 msgr2=0x7f4200146d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:01.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.462+0000 7f41fe59c640 1 --2- 172.21.15.89:0/2205120044 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42001468c0 0x7f4200146d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:01.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.462+0000 7f41fe59c640 1 -- 172.21.15.89:0/2205120044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42001588a0 msgr2=0x7f42000c37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:01.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.462+0000 7f41fe59c640 1 --2- 172.21.15.89:0/2205120044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42001588a0 0x7f42000c37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:01.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.462+0000 7f41fe59c640 1 -- 172.21.15.89:0/2205120044 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4200157b80 con 0x7f41e80039b0 2023-10-30T12:44:01.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.462+0000 7f41fed9d640 1 --2- 172.21.15.89:0/2205120044 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42001468c0 0x7f4200146d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:44:01.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.462+0000 7f41fe59c640 1 --2- 172.21.15.89:0/2205120044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41e80039b0 0x7f42001555a0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f41f001f180 tx=0x7f41e8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:01.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.462+0000 7f41eeffd640 1 -- 172.21.15.89:0/2205120044 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f0001640 con 0x7f41e80039b0 2023-10-30T12:44:01.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.462+0000 7f41eeffd640 1 -- 172.21.15.89:0/2205120044 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f0007070 con 0x7f41e80039b0 2023-10-30T12:44:01.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.462+0000 7f42048a9640 1 -- 172.21.15.89:0/2205120044 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4200157e60 con 0x7f41e80039b0 2023-10-30T12:44:01.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.462+0000 7f41eeffd640 1 -- 172.21.15.89:0/2205120044 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f0007420 con 0x7f41e80039b0 2023-10-30T12:44:01.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.462+0000 7f42048a9640 1 -- 172.21.15.89:0/2205120044 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4200157260 con 0x7f41e80039b0 2023-10-30T12:44:01.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.462+0000 7f42048a9640 1 -- 172.21.15.89:0/2205120044 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f420004e170 con 0x7f41e80039b0 2023-10-30T12:44:01.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.462+0000 7f41eeffd640 1 -- 172.21.15.89:0/2205120044 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f41f0007580 con 0x7f41e80039b0 2023-10-30T12:44:01.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.462+0000 7f41eeffd640 1 --2- 172.21.15.89:0/2205120044 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f41e00664d0 0x7f41e0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:01.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.462+0000 7f41eeffd640 1 -- 172.21.15.89:0/2205120044 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6406+0+0 (secure 0 0 0) 0x7f41f00b7040 con 0x7f41e80039b0 2023-10-30T12:44:01.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.466+0000 7f41fdd9b640 1 --2- 172.21.15.89:0/2205120044 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f41e00664d0 0x7f41e0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:01.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.466+0000 7f41fdd9b640 1 --2- 172.21.15.89:0/2205120044 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f41e00664d0 0x7f41e0068990 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f42000c8350 tx=0x7f41f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:01.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.470+0000 7f41eeffd640 1 -- 172.21.15.89:0/2205120044 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f41f00ee820 con 0x7f41e80039b0 2023-10-30T12:44:01.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.650+0000 7f42048a9640 1 -- 172.21.15.89:0/2205120044 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42000c7dc0 con 0x7f41e80039b0 2023-10-30T12:44:01.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.650+0000 7f41eeffd640 1 -- 172.21.15.89:0/2205120044 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+5735 (secure 0 0 0) 0x7f41f00f1e40 con 0x7f41e80039b0 2023-10-30T12:44:01.647 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:44:01.647 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:43:59.373052+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:44:01.648 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 77 2023-10-30T12:44:01.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.650+0000 7f42048a9640 1 -- 172.21.15.89:0/2205120044 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f41e00664d0 msgr2=0x7f41e0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:01.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.650+0000 7f42048a9640 1 --2- 172.21.15.89:0/2205120044 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f41e00664d0 0x7f41e0068990 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f42000c8350 tx=0x7f41f4008040 comp rx=0 tx=0).stop 2023-10-30T12:44:01.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.650+0000 7f42048a9640 1 -- 172.21.15.89:0/2205120044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41e80039b0 msgr2=0x7f42001555a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:01.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.650+0000 7f42048a9640 1 --2- 172.21.15.89:0/2205120044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41e80039b0 0x7f42001555a0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f41f001f180 tx=0x7f41e8005800 comp rx=0 tx=0).stop 2023-10-30T12:44:01.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.650+0000 7f42048a9640 1 -- 172.21.15.89:0/2205120044 shutdown_connections 2023-10-30T12:44:01.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.650+0000 7f42048a9640 1 --2- 172.21.15.89:0/2205120044 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f41e00664d0 0x7f41e0068990 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:01.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.650+0000 7f42048a9640 1 --2- 172.21.15.89:0/2205120044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f42001588a0 0x7f42000c37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:01.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.650+0000 7f42048a9640 1 --2- 172.21.15.89:0/2205120044 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f42001468c0 0x7f4200146d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:01.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.650+0000 7f42048a9640 1 --2- 172.21.15.89:0/2205120044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41e80039b0 0x7f42001555a0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:01.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.650+0000 7f42048a9640 1 -- 172.21.15.89:0/2205120044 >> 172.21.15.89:0/2205120044 conn(0x7f42000c7910 msgr2=0x7f42000c2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:01.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.650+0000 7f42048a9640 1 -- 172.21.15.89:0/2205120044 shutdown_connections 2023-10-30T12:44:01.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:01.650+0000 7f42048a9640 1 -- 172.21.15.89:0/2205120044 wait complete. 2023-10-30T12:44:01.668 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=2, replacing gid=5129 2023-10-30T12:44:01.669 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.f 2023-10-30T12:44:02.282 INFO:tasks.ceph.mds.b.smithi192.stderr:2023-10-30T12:44:02.282+0000 7f5231cd7640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T12:44:04.670 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-10-30T12:44:04.670 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-30T12:44:04.671 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-30T12:44:04.674 INFO:tasks.ceph.mds.f:Started 2023-10-30T12:44:04.674 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:44:04.749 INFO:tasks.ceph.mds.f.smithi192.stdout:starting mds.f at 2023-10-30T12:44:04.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.821+0000 7f10f8017640 1 Processor -- start 2023-10-30T12:44:04.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.821+0000 7f10f8017640 1 -- start start 2023-10-30T12:44:04.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.821+0000 7f10f8017640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10f01538a0 0x7f10f015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:04.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.821+0000 7f10f8017640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f10f00ca0e0 0x7f10f015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:04.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.821+0000 7f10f8017640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f00cb800 con 0x7f10f00cf160 2023-10-30T12:44:04.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.821+0000 7f10f8017640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10f015dc30 con 0x7f10f00ca0e0 2023-10-30T12:44:04.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.821+0000 7f10f8017640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10f00cb330 con 0x7f10f01538a0 2023-10-30T12:44:04.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.821+0000 7f10f65b4640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f10f00cf160 0x7f10f00cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46768/0 (socket says 172.21.15.89:46768) 2023-10-30T12:44:04.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.821+0000 7f10f65b4640 1 -- 172.21.15.89:0/191261896 learned_addr learned my addr 172.21.15.89:0/191261896 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:44:04.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.821+0000 7f10f55b2640 1 --2- 172.21.15.89:0/191261896 >> v2:172.21.15.192:3300/0 conn(0x7f10f00ca0e0 0x7f10f015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:04.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.821+0000 7f10f5db3640 1 --2- 172.21.15.89:0/191261896 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10f01538a0 0x7f10f015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:04.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.821+0000 7f10f4db1640 1 -- 172.21.15.89:0/191261896 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017205333 0 0) 0x7f10f00cb800 con 0x7f10f00cf160 2023-10-30T12:44:04.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.821+0000 7f10f4db1640 1 -- 172.21.15.89:0/191261896 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10dc003530 con 0x7f10f00cf160 2023-10-30T12:44:04.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.825+0000 7f10f4db1640 1 -- 172.21.15.89:0/191261896 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2898699893 0 0) 0x7f10dc003530 con 0x7f10f00cf160 2023-10-30T12:44:04.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.825+0000 7f10f4db1640 1 -- 172.21.15.89:0/191261896 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10f01538a0 msgr2=0x7f10f015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:44:04.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.825+0000 7f10f4db1640 1 --2- 172.21.15.89:0/191261896 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10f01538a0 0x7f10f015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:04.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.825+0000 7f10f4db1640 1 -- 172.21.15.89:0/191261896 >> v2:172.21.15.192:3300/0 conn(0x7f10f00ca0e0 msgr2=0x7f10f015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:44:04.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.825+0000 7f10f4db1640 1 --2- 172.21.15.89:0/191261896 >> v2:172.21.15.192:3300/0 conn(0x7f10f00ca0e0 0x7f10f015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:04.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.825+0000 7f10f4db1640 1 -- 172.21.15.89:0/191261896 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f00cad70 con 0x7f10f00cf160 2023-10-30T12:44:04.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.825+0000 7f10f4db1640 1 -- 172.21.15.89:0/191261896 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f10e8001a00 con 0x7f10f00cf160 2023-10-30T12:44:04.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.825+0000 7f10f4db1640 1 -- 172.21.15.89:0/191261896 >> v1:172.21.15.89:6789/0 conn(0x7f10f00cf160 legacy=0x7f10f00cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:04.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.825+0000 7f10f4db1640 1 --2- 172.21.15.89:0/191261896 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10dc003980 0x7f10dc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:04.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.825+0000 7f10f4db1640 1 --2- 172.21.15.89:0/191261896 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10f01538a0 0x7f10dc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:04.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.825+0000 7f10f4db1640 1 --2- 172.21.15.89:0/191261896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10dc004d10 0x7f10dc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:04.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.825+0000 7f10f4db1640 1 -- 172.21.15.89:0/191261896 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f015dc30 con 0x7f10dc003980 2023-10-30T12:44:04.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.825+0000 7f10f4db1640 1 -- 172.21.15.89:0/191261896 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f00cb330 con 0x7f10dc004d10 2023-10-30T12:44:04.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.825+0000 7f10f4db1640 1 -- 172.21.15.89:0/191261896 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc00b990 con 0x7f10f01538a0 2023-10-30T12:44:04.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.825+0000 7f10f5db3640 1 --2- 172.21.15.89:0/191261896 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10dc003980 0x7f10dc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:04.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.825+0000 7f10f55b2640 1 --2- 172.21.15.89:0/191261896 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10f01538a0 0x7f10dc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:04.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.825+0000 7f10f65b4640 1 --2- 172.21.15.89:0/191261896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10dc004d10 0x7f10dc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:04.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.825+0000 7f10f5db3640 1 -- 172.21.15.89:0/191261896 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10f01538a0 msgr2=0x7f10dc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:44:04.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.825+0000 7f10f5db3640 1 --2- 172.21.15.89:0/191261896 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10f01538a0 0x7f10dc0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:04.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.825+0000 7f10f5db3640 1 -- 172.21.15.89:0/191261896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10dc004d10 msgr2=0x7f10dc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:44:04.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.825+0000 7f10f5db3640 1 --2- 172.21.15.89:0/191261896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10dc004d10 0x7f10dc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:04.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.825+0000 7f10f5db3640 1 -- 172.21.15.89:0/191261896 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10dc003530 con 0x7f10dc003980 2023-10-30T12:44:04.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.825+0000 7f10f65b4640 1 --2- 172.21.15.89:0/191261896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10dc004d10 0x7f10dc005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:44:04.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.825+0000 7f10f5db3640 1 --2- 172.21.15.89:0/191261896 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10dc003980 0x7f10dc003da0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f10ec009ae0 tx=0x7f10ec01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce4b75783eefc0c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:04.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.829+0000 7f10f4db1640 1 -- 172.21.15.89:0/191261896 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ec0a5640 con 0x7f10dc003980 2023-10-30T12:44:04.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.829+0000 7f10f4db1640 1 -- 172.21.15.89:0/191261896 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10ec0a57a0 con 0x7f10dc003980 2023-10-30T12:44:04.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.829+0000 7f10f4db1640 1 -- 172.21.15.89:0/191261896 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ec0a5b50 con 0x7f10dc003980 2023-10-30T12:44:04.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.829+0000 7f10f8017640 1 -- 172.21.15.89:0/191261896 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10dc003980 msgr2=0x7f10dc003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:04.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.829+0000 7f10f8017640 1 --2- 172.21.15.89:0/191261896 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10dc003980 0x7f10dc003da0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f10ec009ae0 tx=0x7f10ec01ca50 comp rx=0 tx=0).stop 2023-10-30T12:44:04.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.829+0000 7f10f65b4640 1 -- 172.21.15.89:0/191261896 reap_dead start 2023-10-30T12:44:04.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.829+0000 7f10f8017640 1 -- 172.21.15.89:0/191261896 shutdown_connections 2023-10-30T12:44:04.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.829+0000 7f10f8017640 1 -- 172.21.15.89:0/191261896 >> 172.21.15.89:0/191261896 conn(0x7f10f006bfe0 msgr2=0x7f10f006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:04.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.829+0000 7f10f8017640 1 -- 172.21.15.89:0/191261896 shutdown_connections 2023-10-30T12:44:04.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.829+0000 7f10f8017640 1 -- 172.21.15.89:0/191261896 wait complete. 2023-10-30T12:44:04.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.829+0000 7f10f8017640 1 Processor -- start 2023-10-30T12:44:04.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.829+0000 7f10f8017640 1 -- start start 2023-10-30T12:44:04.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.829+0000 7f10f8017640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10dc003980 0x7f10f0157bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:04.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.829+0000 7f10f8017640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10f01591f0 0x7f10f01574c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:04.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.829+0000 7f10f8017640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10f01efde0 0x7f10f0158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:04.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.829+0000 7f10f8017640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f0158c80 con 0x7f10f01591f0 2023-10-30T12:44:04.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.829+0000 7f10f8017640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f01f0260 con 0x7f10f01efde0 2023-10-30T12:44:04.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.829+0000 7f10f8017640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10f01f3340 con 0x7f10dc003980 2023-10-30T12:44:04.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.829+0000 7f10f55b2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10f01591f0 0x7f10f01574c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:04.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.829+0000 7f10f55b2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10f01591f0 0x7f10f01574c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51258/0 (socket says 172.21.15.89:51258) 2023-10-30T12:44:04.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.829+0000 7f10f55b2640 1 -- 172.21.15.89:0/2770892640 learned_addr learned my addr 172.21.15.89:0/2770892640 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:44:04.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.829+0000 7f10f5db3640 1 --2- 172.21.15.89:0/2770892640 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10dc003980 0x7f10f0157bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:04.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.829+0000 7f10f65b4640 1 --2- 172.21.15.89:0/2770892640 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10f01efde0 0x7f10f0158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:04.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.829+0000 7f10f55b2640 1 -- 172.21.15.89:0/2770892640 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10dc003980 msgr2=0x7f10f0157bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:04.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.833+0000 7f10f55b2640 1 --2- 172.21.15.89:0/2770892640 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10dc003980 0x7f10f0157bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:04.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.833+0000 7f10f55b2640 1 -- 172.21.15.89:0/2770892640 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10f01efde0 msgr2=0x7f10f0158740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:04.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.833+0000 7f10f55b2640 1 --2- 172.21.15.89:0/2770892640 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10f01efde0 0x7f10f0158740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:04.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.833+0000 7f10f55b2640 1 -- 172.21.15.89:0/2770892640 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f01ef320 con 0x7f10f01591f0 2023-10-30T12:44:04.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.833+0000 7f10f65b4640 1 --2- 172.21.15.89:0/2770892640 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10f01efde0 0x7f10f0158740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:44:04.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.833+0000 7f10f55b2640 1 --2- 172.21.15.89:0/2770892640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10f01591f0 0x7f10f01574c0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f10e0011aa0 tx=0x7f10e0011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:04.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.833+0000 7f10e67fc640 1 -- 172.21.15.89:0/2770892640 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e000fb80 con 0x7f10f01591f0 2023-10-30T12:44:04.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.833+0000 7f10e67fc640 1 -- 172.21.15.89:0/2770892640 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10e000fce0 con 0x7f10f01591f0 2023-10-30T12:44:04.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.833+0000 7f10f8017640 1 -- 172.21.15.89:0/2770892640 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10f01f35e0 con 0x7f10f01591f0 2023-10-30T12:44:04.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.833+0000 7f10e67fc640 1 -- 172.21.15.89:0/2770892640 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e0014680 con 0x7f10f01591f0 2023-10-30T12:44:04.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.833+0000 7f10f8017640 1 -- 172.21.15.89:0/2770892640 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10f01ef8c0 con 0x7f10f01591f0 2023-10-30T12:44:04.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.833+0000 7f10f8017640 1 -- 172.21.15.89:0/2770892640 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10f01efa80 con 0x7f10f01591f0 2023-10-30T12:44:04.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.833+0000 7f10e67fc640 1 -- 172.21.15.89:0/2770892640 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f10e00147e0 con 0x7f10f01591f0 2023-10-30T12:44:04.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.833+0000 7f10e67fc640 1 --2- 172.21.15.89:0/2770892640 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f10d40664d0 0x7f10d4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:04.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.833+0000 7f10e67fc640 1 -- 172.21.15.89:0/2770892640 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6406+0+0 (secure 0 0 0) 0x7f10e0084970 con 0x7f10f01591f0 2023-10-30T12:44:04.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.833+0000 7f10f5db3640 1 --2- 172.21.15.89:0/2770892640 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f10d40664d0 0x7f10d4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:04.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.837+0000 7f10f5db3640 1 --2- 172.21.15.89:0/2770892640 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f10d40664d0 0x7f10d4068990 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f10ec0af000 tx=0x7f10ec0046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:04.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:04.837+0000 7f10e67fc640 1 -- 172.21.15.89:0/2770892640 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f10e0051a90 con 0x7f10f01591f0 2023-10-30T12:44:05.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:05.017+0000 7f10f8017640 1 -- 172.21.15.89:0/2770892640 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10f004e170 con 0x7f10f01591f0 2023-10-30T12:44:05.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:05.017+0000 7f10e67fc640 1 -- 172.21.15.89:0/2770892640 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+5735 (secure 0 0 0) 0x7f10e00550b0 con 0x7f10f01591f0 2023-10-30T12:44:05.017 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:44:05.017 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:43:59.373052+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:44:05.018 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 77 2023-10-30T12:44:05.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:05.021+0000 7f10f8017640 1 -- 172.21.15.89:0/2770892640 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f10d40664d0 msgr2=0x7f10d4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:05.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:05.021+0000 7f10f8017640 1 --2- 172.21.15.89:0/2770892640 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f10d40664d0 0x7f10d4068990 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f10ec0af000 tx=0x7f10ec0046d0 comp rx=0 tx=0).stop 2023-10-30T12:44:05.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:05.021+0000 7f10f8017640 1 -- 172.21.15.89:0/2770892640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10f01591f0 msgr2=0x7f10f01574c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:05.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:05.021+0000 7f10f8017640 1 --2- 172.21.15.89:0/2770892640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10f01591f0 0x7f10f01574c0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f10e0011aa0 tx=0x7f10e0011f60 comp rx=0 tx=0).stop 2023-10-30T12:44:05.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:05.021+0000 7f10f8017640 1 -- 172.21.15.89:0/2770892640 shutdown_connections 2023-10-30T12:44:05.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:05.021+0000 7f10f8017640 1 --2- 172.21.15.89:0/2770892640 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f10d40664d0 0x7f10d4068990 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:05.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:05.021+0000 7f10f8017640 1 --2- 172.21.15.89:0/2770892640 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10f01efde0 0x7f10f0158740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:05.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:05.021+0000 7f10f8017640 1 --2- 172.21.15.89:0/2770892640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10f01591f0 0x7f10f01574c0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:05.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:05.021+0000 7f10f8017640 1 --2- 172.21.15.89:0/2770892640 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10dc003980 0x7f10f0157bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:05.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:05.021+0000 7f10f8017640 1 -- 172.21.15.89:0/2770892640 >> 172.21.15.89:0/2770892640 conn(0x7f10f006bfe0 msgr2=0x7f10f01e2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:05.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:05.021+0000 7f10f8017640 1 -- 172.21.15.89:0/2770892640 shutdown_connections 2023-10-30T12:44:05.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:05.021+0000 7f10f8017640 1 -- 172.21.15.89:0/2770892640 wait complete. 2023-10-30T12:44:05.038 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2023-10-30T12:44:05.481 INFO:tasks.workunit.client.0.smithi089.stderr:Updating files: 0% (49/12071) Updating files: 1% (121/12071) Updating files: 1% (237/12071) Updating files: 2% (242/12071) Updating files: 3% (363/12071) Updating files: 4% (483/12071) Updating files: 4% (497/12071) Updating files: 4% (573/12071) Updating files: 5% (604/12071) Updating files: 6% (725/12071) Updating files: 6% (753/12071) Updating files: 7% (845/12071) Updating files: 8% (966/12071) Updating files: 8% (1014/12071) Updating files: 8% (1075/12071) Updating files: 9% (1087/12071) Updating files: 10% (1208/12071) Updating files: 10% (1293/12071) Updating files: 11% (1328/12071) Updating files: 11% (1378/12071) Updating files: 11% (1379/12071) Updating files: 12% (1449/12071) Updating files: 12% (1525/12071) Updating files: 13% (1570/12071) Updating files: 13% (1639/12071) Updating files: 13% (1660/12071) Updating files: 13% (1684/12071) Updating files: 14% (1690/12071) Updating files: 14% (1734/12071) Updating files: 15% (1811/12071) Updating files: 15% (1864/12071) Updating files: 15% (1900/12071) Updating files: 15% (1918/12071) Updating files: 16% (1932/12071) Updating files: 17% (2053/12071) Updating files: 18% (2173/12071) Updating files: 18% (2232/12071) Updating files: 18% (2240/12071) Updating files: 19% (2294/12071) Updating files: 19% (2303/12071) Updating files: 19% (2385/12071) Updating files: 20% (2415/12071) Updating files: 21% (2535/12071) Updating files: 21% (2536/12071) Updating files: 22% (2656/12071) Updating files: 23% (2777/12071) Updating files: 23% (2895/12071) Updating files: 24% (2898/12071) Updating files: 24% (2999/12071) Updating files: 25% (3018/12071) Updating files: 25% (3043/12071) Updating files: 26% (3139/12071) Updating files: 26% (3205/12071) Updating files: 27% (3260/12071) Updating files: 27% (3359/12071) Updating files: 28% (3380/12071) Updating files: 29% (3501/12071) Updating files: 29% (3545/12071) Updating files: 29% (3575/12071) Updating files: 29% (3596/12071) Updating files: 30% (3622/12071) Updating files: 30% (3660/12071) Updating files: 31% (3743/12071) Updating files: 32% (3863/12071) Updating files: 32% (3942/12071) Updating files: 32% (3957/12071) Updating files: 32% (3963/12071) Updating files: 33% (3984/12071) Updating files: 34% (4105/12071) Updating files: 34% (4185/12071) Updating files: 35% (4225/12071) Updating files: 35% (4244/12071) Updating files: 35% (4247/12071) Updating files: 35% (4305/12071) Updating files: 35% (4309/12071) Updating files: 36% (4346/12071) Updating files: 36% (4348/12071) Updating files: 36% (4418/12071) Updating files: 37% (4467/12071) Updating files: 38% (4587/12071) Updating files: 39% (4708/12071) Updating files: 39% (4735/12071) Updating files: 40% (4829/12071) Updating files: 40% (4939/12071) Updating files: 41% (4950/12071) Updating files: 41% (4986/12071) Updating files: 41% (5014/12071) Updating files: 42% (5070/12071) Updating files: 42% (5165/12071) Updating files: 43% (5191/12071) Updating files: 43% (5225/12071) Updating files: 43% (5245/12071) Updating files: 44% (5312/12071) Updating files: 44% (5395/12071) Updating files: 45% (5432/12071) Updating files: 45% (5435/12071) Updating files: 46% (5553/12071) Updating files: 46% (5594/12071) Updating files: 47% (5674/12071) Updating files: 47% (5678/12071) Updating files: 47% (5743/12071) Updating files: 48% (5795/12071) Updating files: 49% (5915/12071) Updating files: 49% (6006/12071) Updating files: 50% (6036/12071) Updating files: 50% (6046/12071) Updating files: 50% (6110/12071) Updating files: 51% (6157/12071) Updating files: 51% (6203/12071) Updating files: 52% (6277/12071) Updating files: 52% (6308/12071) Updating files: 52% (6334/12071) Updating files: 53% (6398/12071) Updating files: 53% (6415/12071) Updating files: 53% (6465/12071) Updating files: 53% (6488/12071) Updating files: 54% (6519/12071) Updating files: 54% (6544/12071) Updating files: 55% (6640/12071) Updating files: 55% (6653/12071) Updating files: 56% (6760/12071) Updating files: 56% (6762/12071) Updating files: 57% (6881/12071) Updating files: 57% (6982/12071) Updating files: 58% (7002/12071) Updating files: 58% (7006/12071) Updating files: 59% (7122/12071) Updating files: 59% (7198/12071) Updating files: 60% (7243/12071) Updating files: 61% (7364/12071) Updating files: 61% (7403/12071) Updating files: 61% (7470/12071) Updating files: 62% (7485/12071) Updating files: 62% (7532/12071) Updating files: 62% (7582/12071) Updating files: 63% (7605/12071) Updating files: 63% (7653/12071) Updating files: 64% (7726/12071) Updating files: 65% (7847/12071) Updating files: 66% (7967/12071) Updating files: 66% (8004/12071) Updating files: 67% (8088/12071) Updating files: 67% (8191/12071) Updating files: 68% (8209/12071) Updating files: 68% (8268/12071) Updating files: 68% (8293/12071) Updating files: 69% (8329/12071) Updating files: 69% (8344/12071) Updating files: 69% (8349/12071) Updating files: 69% (8403/12071) Updating files: 70% (8450/12071) Updating files: 70% (8550/12071) Updating files: 71% (8571/12071) Updating files: 72% (8692/12071) Updating files: 72% (8696/12071) Updating files: 72% (8800/12071) Updating files: 73% (8812/12071) Updating files: 74% (8933/12071) Updating files: 74% (8962/12071) Updating files: 75% (9054/12071) Updating files: 75% (9077/12071) Updating files: 76% (9174/12071) Updating files: 76% (9242/12071) Updating files: 77% (9295/12071) Updating files: 77% (9314/12071) Updating files: 78% (9416/12071) Updating files: 78% (9532/12071) Updating files: 79% (9537/12071) Updating files: 79% (9611/12071) Updating files: 79% (9614/12071) Updating files: 79% (9619/12071) Updating files: 79% (9634/12071) Updating files: 80% (9657/12071) Updating files: 81% (9778/12071) Updating files: 81% (9830/12071) Updating files: 82% (9899/12071) Updating files: 83% (10019/12071) Updating files: 83% (10047/12071) Updating files: 84% (10140/12071) Updating files: 85% (10261/12071) Updating files: 85% (10347/12071) Updating files: 86% (10382/12071) Updating files: 86% (10410/12071) Updating files: 86% (10436/12071) Updating files: 87% (10502/12071) Updating files: 88% (10623/12071) Updating files: 88% (10647/12071) Updating files: 89% (10744/12071) Updating files: 90% (10864/12071) Updating files: 90% (10870/12071) Updating files: 90% (10885/12071) Updating files: 91% (10985/12071) Updating files: 92% (11106/12071) Updating files: 92% (11115/12071) Updating files: 92% (11144/12071) Updating files: 92% (11203/12071) Updating files: 92% (11224/12071) Updating files: 93% (11227/12071) Updating files: 94% (11347/12071) Updating files: 94% (11426/12071) Updating files: 95% (11468/12071) Updating files: 95% (11554/12071) Updating files: 96% (11589/12071) Updating files: 96% (11603/12071) Updating files: 97% (11709/12071) Updating files: 98% (11830/12071) Updating files: 98% (11891/12071) Updating files: 98% (11900/12071) Updating files: 99% (11951/12071) Updating files: 99% (11979/12071) Updating files: 99% (12040/12071) Updating files: 100% (12071/12071) Updating files: 100% (12071/12071), done. 2023-10-30T12:44:05.561 INFO:tasks.workunit.client.0.smithi089.stderr:+ trap - EXIT 2023-10-30T12:44:05.562 INFO:tasks.workunit.client.0.smithi089.stderr:+ cd ceph 2023-10-30T12:44:05.562 INFO:tasks.workunit.client.0.smithi089.stderr:+ seq 1 90 2023-10-30T12:44:05.563 INFO:tasks.workunit.client.0.smithi089.stderr:+ versions=1 2023-10-30T12:44:05.563 INFO:tasks.workunit.client.0.smithi089.stderr:2 2023-10-30T12:44:05.564 INFO:tasks.workunit.client.0.smithi089.stderr:3 2023-10-30T12:44:05.564 INFO:tasks.workunit.client.0.smithi089.stderr:4 2023-10-30T12:44:05.564 INFO:tasks.workunit.client.0.smithi089.stderr:5 2023-10-30T12:44:05.564 INFO:tasks.workunit.client.0.smithi089.stderr:6 2023-10-30T12:44:05.564 INFO:tasks.workunit.client.0.smithi089.stderr:7 2023-10-30T12:44:05.564 INFO:tasks.workunit.client.0.smithi089.stderr:8 2023-10-30T12:44:05.565 INFO:tasks.workunit.client.0.smithi089.stderr:9 2023-10-30T12:44:05.565 INFO:tasks.workunit.client.0.smithi089.stderr:10 2023-10-30T12:44:05.565 INFO:tasks.workunit.client.0.smithi089.stderr:11 2023-10-30T12:44:05.565 INFO:tasks.workunit.client.0.smithi089.stderr:12 2023-10-30T12:44:05.565 INFO:tasks.workunit.client.0.smithi089.stderr:13 2023-10-30T12:44:05.566 INFO:tasks.workunit.client.0.smithi089.stderr:14 2023-10-30T12:44:05.566 INFO:tasks.workunit.client.0.smithi089.stderr:15 2023-10-30T12:44:05.566 INFO:tasks.workunit.client.0.smithi089.stderr:16 2023-10-30T12:44:05.566 INFO:tasks.workunit.client.0.smithi089.stderr:17 2023-10-30T12:44:05.566 INFO:tasks.workunit.client.0.smithi089.stderr:18 2023-10-30T12:44:05.567 INFO:tasks.workunit.client.0.smithi089.stderr:19 2023-10-30T12:44:05.567 INFO:tasks.workunit.client.0.smithi089.stderr:20 2023-10-30T12:44:05.567 INFO:tasks.workunit.client.0.smithi089.stderr:21 2023-10-30T12:44:05.567 INFO:tasks.workunit.client.0.smithi089.stderr:22 2023-10-30T12:44:05.567 INFO:tasks.workunit.client.0.smithi089.stderr:23 2023-10-30T12:44:05.567 INFO:tasks.workunit.client.0.smithi089.stderr:24 2023-10-30T12:44:05.568 INFO:tasks.workunit.client.0.smithi089.stderr:25 2023-10-30T12:44:05.568 INFO:tasks.workunit.client.0.smithi089.stderr:26 2023-10-30T12:44:05.568 INFO:tasks.workunit.client.0.smithi089.stderr:27 2023-10-30T12:44:05.568 INFO:tasks.workunit.client.0.smithi089.stderr:28 2023-10-30T12:44:05.568 INFO:tasks.workunit.client.0.smithi089.stderr:29 2023-10-30T12:44:05.569 INFO:tasks.workunit.client.0.smithi089.stderr:30 2023-10-30T12:44:05.569 INFO:tasks.workunit.client.0.smithi089.stderr:31 2023-10-30T12:44:05.569 INFO:tasks.workunit.client.0.smithi089.stderr:32 2023-10-30T12:44:05.569 INFO:tasks.workunit.client.0.smithi089.stderr:33 2023-10-30T12:44:05.569 INFO:tasks.workunit.client.0.smithi089.stderr:34 2023-10-30T12:44:05.570 INFO:tasks.workunit.client.0.smithi089.stderr:35 2023-10-30T12:44:05.570 INFO:tasks.workunit.client.0.smithi089.stderr:36 2023-10-30T12:44:05.570 INFO:tasks.workunit.client.0.smithi089.stderr:37 2023-10-30T12:44:05.570 INFO:tasks.workunit.client.0.smithi089.stderr:38 2023-10-30T12:44:05.570 INFO:tasks.workunit.client.0.smithi089.stderr:39 2023-10-30T12:44:05.571 INFO:tasks.workunit.client.0.smithi089.stderr:40 2023-10-30T12:44:05.571 INFO:tasks.workunit.client.0.smithi089.stderr:41 2023-10-30T12:44:05.571 INFO:tasks.workunit.client.0.smithi089.stderr:42 2023-10-30T12:44:05.571 INFO:tasks.workunit.client.0.smithi089.stderr:43 2023-10-30T12:44:05.571 INFO:tasks.workunit.client.0.smithi089.stderr:44 2023-10-30T12:44:05.572 INFO:tasks.workunit.client.0.smithi089.stderr:45 2023-10-30T12:44:05.572 INFO:tasks.workunit.client.0.smithi089.stderr:46 2023-10-30T12:44:05.572 INFO:tasks.workunit.client.0.smithi089.stderr:47 2023-10-30T12:44:05.572 INFO:tasks.workunit.client.0.smithi089.stderr:48 2023-10-30T12:44:05.572 INFO:tasks.workunit.client.0.smithi089.stderr:49 2023-10-30T12:44:05.572 INFO:tasks.workunit.client.0.smithi089.stderr:50 2023-10-30T12:44:05.573 INFO:tasks.workunit.client.0.smithi089.stderr:51 2023-10-30T12:44:05.573 INFO:tasks.workunit.client.0.smithi089.stderr:52 2023-10-30T12:44:05.573 INFO:tasks.workunit.client.0.smithi089.stderr:53 2023-10-30T12:44:05.573 INFO:tasks.workunit.client.0.smithi089.stderr:54 2023-10-30T12:44:05.573 INFO:tasks.workunit.client.0.smithi089.stderr:55 2023-10-30T12:44:05.574 INFO:tasks.workunit.client.0.smithi089.stderr:56 2023-10-30T12:44:05.574 INFO:tasks.workunit.client.0.smithi089.stderr:57 2023-10-30T12:44:05.574 INFO:tasks.workunit.client.0.smithi089.stderr:58 2023-10-30T12:44:05.574 INFO:tasks.workunit.client.0.smithi089.stderr:59 2023-10-30T12:44:05.574 INFO:tasks.workunit.client.0.smithi089.stderr:60 2023-10-30T12:44:05.575 INFO:tasks.workunit.client.0.smithi089.stderr:61 2023-10-30T12:44:05.575 INFO:tasks.workunit.client.0.smithi089.stderr:62 2023-10-30T12:44:05.575 INFO:tasks.workunit.client.0.smithi089.stderr:63 2023-10-30T12:44:05.575 INFO:tasks.workunit.client.0.smithi089.stderr:64 2023-10-30T12:44:05.575 INFO:tasks.workunit.client.0.smithi089.stderr:65 2023-10-30T12:44:05.576 INFO:tasks.workunit.client.0.smithi089.stderr:66 2023-10-30T12:44:05.576 INFO:tasks.workunit.client.0.smithi089.stderr:67 2023-10-30T12:44:05.576 INFO:tasks.workunit.client.0.smithi089.stderr:68 2023-10-30T12:44:05.576 INFO:tasks.workunit.client.0.smithi089.stderr:69 2023-10-30T12:44:05.576 INFO:tasks.workunit.client.0.smithi089.stderr:70 2023-10-30T12:44:05.577 INFO:tasks.workunit.client.0.smithi089.stderr:71 2023-10-30T12:44:05.577 INFO:tasks.workunit.client.0.smithi089.stderr:72 2023-10-30T12:44:05.577 INFO:tasks.workunit.client.0.smithi089.stderr:73 2023-10-30T12:44:05.577 INFO:tasks.workunit.client.0.smithi089.stderr:74 2023-10-30T12:44:05.577 INFO:tasks.workunit.client.0.smithi089.stderr:75 2023-10-30T12:44:05.578 INFO:tasks.workunit.client.0.smithi089.stderr:76 2023-10-30T12:44:05.578 INFO:tasks.workunit.client.0.smithi089.stderr:77 2023-10-30T12:44:05.578 INFO:tasks.workunit.client.0.smithi089.stderr:78 2023-10-30T12:44:05.578 INFO:tasks.workunit.client.0.smithi089.stderr:79 2023-10-30T12:44:05.578 INFO:tasks.workunit.client.0.smithi089.stderr:80 2023-10-30T12:44:05.579 INFO:tasks.workunit.client.0.smithi089.stderr:81 2023-10-30T12:44:05.579 INFO:tasks.workunit.client.0.smithi089.stderr:82 2023-10-30T12:44:05.579 INFO:tasks.workunit.client.0.smithi089.stderr:83 2023-10-30T12:44:05.579 INFO:tasks.workunit.client.0.smithi089.stderr:84 2023-10-30T12:44:05.579 INFO:tasks.workunit.client.0.smithi089.stderr:85 2023-10-30T12:44:05.580 INFO:tasks.workunit.client.0.smithi089.stderr:86 2023-10-30T12:44:05.580 INFO:tasks.workunit.client.0.smithi089.stderr:87 2023-10-30T12:44:05.580 INFO:tasks.workunit.client.0.smithi089.stderr:88 2023-10-30T12:44:05.580 INFO:tasks.workunit.client.0.smithi089.stderr:89 2023-10-30T12:44:05.580 INFO:tasks.workunit.client.0.smithi089.stderr:90 2023-10-30T12:44:05.580 INFO:tasks.workunit.client.0.smithi089.stderr:+ [ 1 -eq 48 ] 2023-10-30T12:44:05.581 INFO:tasks.workunit.client.0.smithi089.stderr:+ ver=v0.1 2023-10-30T12:44:05.581 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo v0.1 2023-10-30T12:44:05.581 INFO:tasks.workunit.client.0.smithi089.stdout:v0.1 2023-10-30T12:44:05.581 INFO:tasks.workunit.client.0.smithi089.stderr:+ git reset --hard v0.1 2023-10-30T12:44:07.038 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:44:07.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.185+0000 7f1ffa73e640 1 Processor -- start 2023-10-30T12:44:07.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.185+0000 7f1ffa73e640 1 -- start start 2023-10-30T12:44:07.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.185+0000 7f1ffa73e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ff415d6f0 0x7f1ff415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:07.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.185+0000 7f1ffa73e640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff40cb800 con 0x7f1ff40cf160 2023-10-30T12:44:07.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.185+0000 7f1ffa73e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff40cb330 con 0x7f1ff41538a0 2023-10-30T12:44:07.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.185+0000 7f1ffa73e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff40cad70 con 0x7f1ff415d6f0 2023-10-30T12:44:07.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.185+0000 7f1ff8cdb640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f1ff40cf160 0x7f1ff4164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46774/0 (socket says 172.21.15.89:46774) 2023-10-30T12:44:07.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.185+0000 7f1ff8cdb640 1 -- 172.21.15.89:0/3431412774 learned_addr learned my addr 172.21.15.89:0/3431412774 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:44:07.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.185+0000 7f1ff37fe640 1 --2- 172.21.15.89:0/3431412774 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ff415d6f0 0x7f1ff415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:07.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.185+0000 7f1ff2ffd640 1 -- 172.21.15.89:0/3431412774 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1899034796 0 0) 0x7f1ff40cb800 con 0x7f1ff40cf160 2023-10-30T12:44:07.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.185+0000 7f1ff2ffd640 1 -- 172.21.15.89:0/3431412774 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fdc003530 con 0x7f1ff40cf160 2023-10-30T12:44:07.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.185+0000 7f1ff2ffd640 1 -- 172.21.15.89:0/3431412774 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3228620065 0 0) 0x7f1fdc003530 con 0x7f1ff40cf160 2023-10-30T12:44:07.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.185+0000 7f1ff37fe640 1 -- 172.21.15.89:0/3431412774 >> v1:172.21.15.192:6789/0 conn(0x7f1ff41538a0 legacy=0x7f1ff40ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:07.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.185+0000 7f1ff37fe640 1 -- 172.21.15.89:0/3431412774 >> v1:172.21.15.89:6789/0 conn(0x7f1ff40cf160 legacy=0x7f1ff4164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:07.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.185+0000 7f1ff37fe640 1 -- 172.21.15.89:0/3431412774 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff40cb050 con 0x7f1ff415d6f0 2023-10-30T12:44:07.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.189+0000 7f1ff37fe640 1 --2- 172.21.15.89:0/3431412774 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ff415d6f0 0x7f1ff415a090 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1fe80097f0 tx=0x7f1fe801ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f3beef512a0aa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:07.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.189+0000 7f1ff2ffd640 1 -- 172.21.15.89:0/3431412774 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec001a00 con 0x7f1ff415d6f0 2023-10-30T12:44:07.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.189+0000 7f1ff2ffd640 1 -- 172.21.15.89:0/3431412774 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe80a8650 con 0x7f1ff415d6f0 2023-10-30T12:44:07.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.189+0000 7f1ff2ffd640 1 -- 172.21.15.89:0/3431412774 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe80a89c0 con 0x7f1ff415d6f0 2023-10-30T12:44:07.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.189+0000 7f1ffa73e640 1 -- 172.21.15.89:0/3431412774 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ff415d6f0 msgr2=0x7f1ff415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:07.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.189+0000 7f1ffa73e640 1 --2- 172.21.15.89:0/3431412774 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ff415d6f0 0x7f1ff415a090 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1fe80097f0 tx=0x7f1fe801ebd0 comp rx=0 tx=0).stop 2023-10-30T12:44:07.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.189+0000 7f1ffa73e640 1 -- 172.21.15.89:0/3431412774 shutdown_connections 2023-10-30T12:44:07.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.189+0000 7f1ffa73e640 1 --2- 172.21.15.89:0/3431412774 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ff415d6f0 0x7f1ff415a090 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:07.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.189+0000 7f1ffa73e640 1 -- 172.21.15.89:0/3431412774 >> 172.21.15.89:0/3431412774 conn(0x7f1ff406bfe0 msgr2=0x7f1ff406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:07.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.193+0000 7f1ffa73e640 1 -- 172.21.15.89:0/3431412774 shutdown_connections 2023-10-30T12:44:07.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.193+0000 7f1ffa73e640 1 -- 172.21.15.89:0/3431412774 wait complete. 2023-10-30T12:44:07.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.193+0000 7f1ffa73e640 1 Processor -- start 2023-10-30T12:44:07.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.193+0000 7f1ffa73e640 1 -- start start 2023-10-30T12:44:07.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.193+0000 7f1ffa73e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ff40cf160 0x7f1ff41449c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:07.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.193+0000 7f1ffa73e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ff41538a0 0x7f1ff40ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:07.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.193+0000 7f1ffa73e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ff415d6f0 0x7f1ff40b3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:07.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.193+0000 7f1ffa73e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff414efc0 con 0x7f1ff415d6f0 2023-10-30T12:44:07.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.193+0000 7f1ffa73e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff40b30c0 con 0x7f1ff41538a0 2023-10-30T12:44:07.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.193+0000 7f1ffa73e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff414f1d0 con 0x7f1ff40cf160 2023-10-30T12:44:07.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.193+0000 7f1ff8cdb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ff415d6f0 0x7f1ff40b3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:07.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.193+0000 7f1ff8cdb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ff415d6f0 0x7f1ff40b3e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51262/0 (socket says 172.21.15.89:51262) 2023-10-30T12:44:07.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.193+0000 7f1ff8cdb640 1 -- 172.21.15.89:0/844359390 learned_addr learned my addr 172.21.15.89:0/844359390 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:44:07.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.193+0000 7f1ff3fff640 1 --2- 172.21.15.89:0/844359390 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ff40cf160 0x7f1ff41449c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:07.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.193+0000 7f1ff37fe640 1 --2- 172.21.15.89:0/844359390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ff41538a0 0x7f1ff40ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:07.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.193+0000 7f1ff8cdb640 1 -- 172.21.15.89:0/844359390 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ff40cf160 msgr2=0x7f1ff41449c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:07.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.193+0000 7f1ff8cdb640 1 --2- 172.21.15.89:0/844359390 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ff40cf160 0x7f1ff41449c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:07.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.193+0000 7f1ff8cdb640 1 -- 172.21.15.89:0/844359390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ff41538a0 msgr2=0x7f1ff40ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:07.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.193+0000 7f1ff8cdb640 1 --2- 172.21.15.89:0/844359390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ff41538a0 0x7f1ff40ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:07.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.193+0000 7f1ff8cdb640 1 -- 172.21.15.89:0/844359390 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe8009420 con 0x7f1ff415d6f0 2023-10-30T12:44:07.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.193+0000 7f1ff3fff640 1 --2- 172.21.15.89:0/844359390 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ff40cf160 0x7f1ff41449c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:44:07.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.193+0000 7f1ff37fe640 1 --2- 172.21.15.89:0/844359390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ff41538a0 0x7f1ff40ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:44:07.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.193+0000 7f1ff8cdb640 1 --2- 172.21.15.89:0/844359390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ff415d6f0 0x7f1ff40b3e60 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00cbf0 tx=0x7f1fec00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:07.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.197+0000 7f1ff0ff9640 1 -- 172.21.15.89:0/844359390 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec00ad50 con 0x7f1ff415d6f0 2023-10-30T12:44:07.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.197+0000 7f1ff0ff9640 1 -- 172.21.15.89:0/844359390 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fec012400 con 0x7f1ff415d6f0 2023-10-30T12:44:07.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.197+0000 7f1ffa73e640 1 -- 172.21.15.89:0/844359390 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ff40b3380 con 0x7f1ff415d6f0 2023-10-30T12:44:07.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.197+0000 7f1ff0ff9640 1 -- 172.21.15.89:0/844359390 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec0127f0 con 0x7f1ff415d6f0 2023-10-30T12:44:07.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.197+0000 7f1ffa73e640 1 -- 172.21.15.89:0/844359390 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff40b3940 con 0x7f1ff415d6f0 2023-10-30T12:44:07.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.197+0000 7f1ffa73e640 1 -- 172.21.15.89:0/844359390 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff40b3b30 con 0x7f1ff415d6f0 2023-10-30T12:44:07.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.197+0000 7f1ff0ff9640 1 -- 172.21.15.89:0/844359390 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1fec012950 con 0x7f1ff415d6f0 2023-10-30T12:44:07.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.197+0000 7f1ff0ff9640 1 --2- 172.21.15.89:0/844359390 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1fe00664d0 0x7f1fe0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:07.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.197+0000 7f1ff0ff9640 1 -- 172.21.15.89:0/844359390 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6406+0+0 (secure 0 0 0) 0x7f1fec084800 con 0x7f1ff415d6f0 2023-10-30T12:44:07.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.197+0000 7f1ff3fff640 1 --2- 172.21.15.89:0/844359390 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1fe00664d0 0x7f1fe0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:07.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.201+0000 7f1ff3fff640 1 --2- 172.21.15.89:0/844359390 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1fe00664d0 0x7f1fe0068990 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4000f70 tx=0x7f1fe4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:07.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.201+0000 7f1ff0ff9640 1 -- 172.21.15.89:0/844359390 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1fec0519c0 con 0x7f1ff415d6f0 2023-10-30T12:44:07.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.389+0000 7f1ffa73e640 1 -- 172.21.15.89:0/844359390 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ff404e170 con 0x7f1ff415d6f0 2023-10-30T12:44:07.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.389+0000 7f1ff0ff9640 1 -- 172.21.15.89:0/844359390 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+6526 (secure 0 0 0) 0x7f1fec054fe0 con 0x7f1ff415d6f0 2023-10-30T12:44:07.389 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:44:07.389 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5450,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:43:59.373052+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:44:07.389 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 78 2023-10-30T12:44:07.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.393+0000 7f1ffa73e640 1 -- 172.21.15.89:0/844359390 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1fe00664d0 msgr2=0x7f1fe0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:07.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.393+0000 7f1ffa73e640 1 --2- 172.21.15.89:0/844359390 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1fe00664d0 0x7f1fe0068990 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4000f70 tx=0x7f1fe4009040 comp rx=0 tx=0).stop 2023-10-30T12:44:07.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.393+0000 7f1ffa73e640 1 -- 172.21.15.89:0/844359390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ff415d6f0 msgr2=0x7f1ff40b3e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:07.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.393+0000 7f1ffa73e640 1 --2- 172.21.15.89:0/844359390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ff415d6f0 0x7f1ff40b3e60 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00cbf0 tx=0x7f1fec00a430 comp rx=0 tx=0).stop 2023-10-30T12:44:07.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.393+0000 7f1ffa73e640 1 -- 172.21.15.89:0/844359390 shutdown_connections 2023-10-30T12:44:07.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.393+0000 7f1ffa73e640 1 --2- 172.21.15.89:0/844359390 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1fe00664d0 0x7f1fe0068990 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:07.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.393+0000 7f1ffa73e640 1 --2- 172.21.15.89:0/844359390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ff41538a0 0x7f1ff40ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:07.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.393+0000 7f1ffa73e640 1 --2- 172.21.15.89:0/844359390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ff415d6f0 0x7f1ff40b3e60 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:07.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.393+0000 7f1ffa73e640 1 --2- 172.21.15.89:0/844359390 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ff40cf160 0x7f1ff41449c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:07.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.393+0000 7f1ffa73e640 1 -- 172.21.15.89:0/844359390 >> 172.21.15.89:0/844359390 conn(0x7f1ff406bfe0 msgr2=0x7f1ff4151cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:07.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.393+0000 7f1ffa73e640 1 -- 172.21.15.89:0/844359390 shutdown_connections 2023-10-30T12:44:07.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:44:07.393+0000 7f1ffa73e640 1 -- 172.21.15.89:0/844359390 wait complete. 2023-10-30T12:44:07.411 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-10-30T12:44:07.411 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 87 secs before thrashing 2023-10-30T12:44:17.515 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:44:17.520 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:44:47.587 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:44:47.593 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:45:17.644 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:45:17.648 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:45:34.412 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:45:34.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.555+0000 7f8eece17640 1 Processor -- start 2023-10-30T12:45:34.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.555+0000 7f8eece17640 1 -- start start 2023-10-30T12:45:34.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.559+0000 7f8eece17640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8ee80c6b80 0x7f8ee80bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:34.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.559+0000 7f8eece17640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee80c0a50 con 0x7f8ee813f4a0 2023-10-30T12:45:34.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.559+0000 7f8eece17640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee806f6e0 con 0x7f8ee80c6f60 2023-10-30T12:45:34.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.559+0000 7f8eece17640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ee806f9c0 con 0x7f8ee80c6b80 2023-10-30T12:45:34.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.559+0000 7f8ee6d9d640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f8ee813f4a0 0x7f8ee81566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:49412/0 (socket says 172.21.15.89:49412) 2023-10-30T12:45:34.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.559+0000 7f8ee6d9d640 1 -- 172.21.15.89:0/2725820825 learned_addr learned my addr 172.21.15.89:0/2725820825 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:45:34.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.559+0000 7f8ee659c640 1 --2- 172.21.15.89:0/2725820825 >> v2:172.21.15.192:3300/0 conn(0x7f8ee80c6b80 0x7f8ee80bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:34.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.559+0000 7f8ee5d9b640 1 -- 172.21.15.89:0/2725820825 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2241484739 0 0) 0x7f8ee80c0a50 con 0x7f8ee813f4a0 2023-10-30T12:45:34.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.559+0000 7f8ee5d9b640 1 -- 172.21.15.89:0/2725820825 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ed4003530 con 0x7f8ee813f4a0 2023-10-30T12:45:34.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.559+0000 7f8ee659c640 1 -- 172.21.15.89:0/2725820825 >> v1:172.21.15.192:6789/0 conn(0x7f8ee80c6f60 legacy=0x7f8ee813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:34.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.559+0000 7f8ee659c640 1 -- 172.21.15.89:0/2725820825 >> v1:172.21.15.89:6789/0 conn(0x7f8ee813f4a0 legacy=0x7f8ee81566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:34.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.559+0000 7f8ee659c640 1 -- 172.21.15.89:0/2725820825 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee806fca0 con 0x7f8ee80c6b80 2023-10-30T12:45:34.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.563+0000 7f8ee659c640 1 --2- 172.21.15.89:0/2725820825 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ee80c6b80 0x7f8ee80bfec0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8ed80097c0 tx=0x7f8ed801c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf5db9ede73fedfe server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:34.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.563+0000 7f8ee5d9b640 1 -- 172.21.15.89:0/2725820825 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed80a8050 con 0x7f8ee80c6b80 2023-10-30T12:45:34.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.563+0000 7f8ee5d9b640 1 -- 172.21.15.89:0/2725820825 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ed80ac5f0 con 0x7f8ee80c6b80 2023-10-30T12:45:34.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.563+0000 7f8ee5d9b640 1 -- 172.21.15.89:0/2725820825 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed80ac960 con 0x7f8ee80c6b80 2023-10-30T12:45:34.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.563+0000 7f8eece17640 1 -- 172.21.15.89:0/2725820825 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ee80c6b80 msgr2=0x7f8ee80bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:34.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.563+0000 7f8eece17640 1 --2- 172.21.15.89:0/2725820825 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ee80c6b80 0x7f8ee80bfec0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8ed80097c0 tx=0x7f8ed801c480 comp rx=0 tx=0).stop 2023-10-30T12:45:34.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.563+0000 7f8eece17640 1 -- 172.21.15.89:0/2725820825 shutdown_connections 2023-10-30T12:45:34.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.563+0000 7f8eece17640 1 --2- 172.21.15.89:0/2725820825 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ee80c6b80 0x7f8ee80bfec0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:34.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.563+0000 7f8eece17640 1 -- 172.21.15.89:0/2725820825 >> 172.21.15.89:0/2725820825 conn(0x7f8ee80c7910 msgr2=0x7f8ee80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:45:34.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.563+0000 7f8eece17640 1 -- 172.21.15.89:0/2725820825 shutdown_connections 2023-10-30T12:45:34.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.563+0000 7f8eece17640 1 -- 172.21.15.89:0/2725820825 wait complete. 2023-10-30T12:45:34.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.563+0000 7f8eece17640 1 Processor -- start 2023-10-30T12:45:34.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.563+0000 7f8eece17640 1 -- start start 2023-10-30T12:45:34.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.567+0000 7f8eece17640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ee80c6b80 0x7f8ee80ab400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:34.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.567+0000 7f8eece17640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ee80c6f60 0x7f8ee81f4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:34.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.567+0000 7f8eece17640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ee813f4a0 0x7f8ee813ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:34.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.567+0000 7f8eece17640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee81f4a70 con 0x7f8ee813f4a0 2023-10-30T12:45:34.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.567+0000 7f8eece17640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee80c2b60 con 0x7f8ee80c6b80 2023-10-30T12:45:34.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.567+0000 7f8eece17640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee80a9860 con 0x7f8ee80c6f60 2023-10-30T12:45:34.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.567+0000 7f8ee6d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ee813f4a0 0x7f8ee813ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:34.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.567+0000 7f8ee6d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ee813f4a0 0x7f8ee813ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59134/0 (socket says 172.21.15.89:59134) 2023-10-30T12:45:34.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.567+0000 7f8ee6d9d640 1 -- 172.21.15.89:0/828878119 learned_addr learned my addr 172.21.15.89:0/828878119 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:45:34.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.567+0000 7f8ee659c640 1 --2- 172.21.15.89:0/828878119 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ee80c6b80 0x7f8ee80ab400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:34.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.567+0000 7f8eddd9b640 1 --2- 172.21.15.89:0/828878119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ee80c6f60 0x7f8ee81f4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:34.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.567+0000 7f8ee6d9d640 1 -- 172.21.15.89:0/828878119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ee80c6f60 msgr2=0x7f8ee81f4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:34.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.567+0000 7f8ee6d9d640 1 --2- 172.21.15.89:0/828878119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ee80c6f60 0x7f8ee81f4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:34.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.567+0000 7f8ee6d9d640 1 -- 172.21.15.89:0/828878119 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ee80c6b80 msgr2=0x7f8ee80ab400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:34.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.567+0000 7f8ee6d9d640 1 --2- 172.21.15.89:0/828878119 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ee80c6b80 0x7f8ee80ab400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:34.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.567+0000 7f8ee6d9d640 1 -- 172.21.15.89:0/828878119 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed8009420 con 0x7f8ee813f4a0 2023-10-30T12:45:34.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.567+0000 7f8ee6d9d640 1 --2- 172.21.15.89:0/828878119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ee813f4a0 0x7f8ee813ab70 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f8ee000c3f0 tx=0x7f8ee000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:34.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.567+0000 7f8edf7fe640 1 -- 172.21.15.89:0/828878119 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee0005cf0 con 0x7f8ee813f4a0 2023-10-30T12:45:34.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.567+0000 7f8edf7fe640 1 -- 172.21.15.89:0/828878119 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ee0005e50 con 0x7f8ee813f4a0 2023-10-30T12:45:34.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.567+0000 7f8eece17640 1 -- 172.21.15.89:0/828878119 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ee80a9560 con 0x7f8ee813f4a0 2023-10-30T12:45:34.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.567+0000 7f8edf7fe640 1 -- 172.21.15.89:0/828878119 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee0007310 con 0x7f8ee813f4a0 2023-10-30T12:45:34.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.567+0000 7f8eece17640 1 -- 172.21.15.89:0/828878119 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ee80c3090 con 0x7f8ee813f4a0 2023-10-30T12:45:34.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.567+0000 7f8eece17640 1 -- 172.21.15.89:0/828878119 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ee804e170 con 0x7f8ee813f4a0 2023-10-30T12:45:34.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.567+0000 7f8edf7fe640 1 -- 172.21.15.89:0/828878119 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8ee0007470 con 0x7f8ee813f4a0 2023-10-30T12:45:34.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.571+0000 7f8edf7fe640 1 --2- 172.21.15.89:0/828878119 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8ec80664a0 0x7f8ec8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:34.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.571+0000 7f8edf7fe640 1 -- 172.21.15.89:0/828878119 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6406+0+0 (secure 0 0 0) 0x7f8ee00832b0 con 0x7f8ee813f4a0 2023-10-30T12:45:34.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.571+0000 7f8ee659c640 1 --2- 172.21.15.89:0/828878119 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8ec80664a0 0x7f8ec8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:34.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.571+0000 7f8ee659c640 1 --2- 172.21.15.89:0/828878119 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8ec80664a0 0x7f8ec8068960 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8ed80ae000 tx=0x7f8ed80a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:34.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.575+0000 7f8edf7fe640 1 -- 172.21.15.89:0/828878119 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8ee0050520 con 0x7f8ee813f4a0 2023-10-30T12:45:34.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.755+0000 7f8eece17640 1 -- 172.21.15.89:0/828878119 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ee80a99a0 con 0x7f8ee813f4a0 2023-10-30T12:45:34.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.755+0000 7f8edf7fe640 1 -- 172.21.15.89:0/828878119 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f8ee0053b40 con 0x7f8ee813f4a0 2023-10-30T12:45:34.752 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:45:34.752 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5450,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:44:45.324321+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:45:34.752 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 82 2023-10-30T12:45:34.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.755+0000 7f8eece17640 1 -- 172.21.15.89:0/828878119 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8ec80664a0 msgr2=0x7f8ec8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:34.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.755+0000 7f8eece17640 1 --2- 172.21.15.89:0/828878119 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8ec80664a0 0x7f8ec8068960 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8ed80ae000 tx=0x7f8ed80a7000 comp rx=0 tx=0).stop 2023-10-30T12:45:34.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.755+0000 7f8eece17640 1 -- 172.21.15.89:0/828878119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ee813f4a0 msgr2=0x7f8ee813ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:34.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.755+0000 7f8eece17640 1 --2- 172.21.15.89:0/828878119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ee813f4a0 0x7f8ee813ab70 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f8ee000c3f0 tx=0x7f8ee000c8b0 comp rx=0 tx=0).stop 2023-10-30T12:45:34.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.755+0000 7f8eece17640 1 -- 172.21.15.89:0/828878119 shutdown_connections 2023-10-30T12:45:34.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.755+0000 7f8eece17640 1 --2- 172.21.15.89:0/828878119 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8ec80664a0 0x7f8ec8068960 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:34.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.755+0000 7f8eece17640 1 --2- 172.21.15.89:0/828878119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8ee813f4a0 0x7f8ee813ab70 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:34.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.755+0000 7f8eece17640 1 --2- 172.21.15.89:0/828878119 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8ee80c6f60 0x7f8ee81f4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:34.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.755+0000 7f8eece17640 1 --2- 172.21.15.89:0/828878119 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ee80c6b80 0x7f8ee80ab400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:34.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.755+0000 7f8eece17640 1 -- 172.21.15.89:0/828878119 >> 172.21.15.89:0/828878119 conn(0x7f8ee80c7910 msgr2=0x7f8ee80c18f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:45:34.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.755+0000 7f8eece17640 1 -- 172.21.15.89:0/828878119 shutdown_connections 2023-10-30T12:45:34.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.755+0000 7f8eece17640 1 -- 172.21.15.89:0/828878119 wait complete. 2023-10-30T12:45:34.772 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=1) 2023-10-30T12:45:34.773 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T12:45:34.773 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:45:34.791 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T12:45:34.792 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:45:34.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.979+0000 7f6aeafd9640 1 Processor -- start 2023-10-30T12:45:34.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.979+0000 7f6aeafd9640 1 -- start start 2023-10-30T12:45:34.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.979+0000 7f6aeafd9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ae41538a0 0x7f6ae415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:34.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.979+0000 7f6aeafd9640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6ae40ca0e0 0x7f6ae415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:34.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.979+0000 7f6aeafd9640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae40cb800 con 0x7f6ae40cf160 2023-10-30T12:45:34.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.979+0000 7f6aeafd9640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ae415dc30 con 0x7f6ae40ca0e0 2023-10-30T12:45:34.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.979+0000 7f6aeafd9640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae40cb330 con 0x7f6ae41538a0 2023-10-30T12:45:34.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.979+0000 7f6ae8d75640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ae41538a0 0x7f6ae415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:34.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.979+0000 7f6ae3fff640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6ae40ca0e0 0x7f6ae415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:34.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.979+0000 7f6ae8d75640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ae41538a0 0x7f6ae415d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:60480/0 (socket says 172.21.15.89:60480) 2023-10-30T12:45:34.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.979+0000 7f6ae8d75640 1 -- 172.21.15.89:0/3189394663 learned_addr learned my addr 172.21.15.89:0/3189394663 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:45:34.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.979+0000 7f6ae8d75640 1 -- 172.21.15.89:0/3189394663 >> v2:172.21.15.192:3300/0 conn(0x7f6ae40ca0e0 msgr2=0x7f6ae415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:45:34.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.979+0000 7f6ae8d75640 1 --2- 172.21.15.89:0/3189394663 >> v2:172.21.15.192:3300/0 conn(0x7f6ae40ca0e0 0x7f6ae415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:34.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.979+0000 7f6ae8d75640 1 -- 172.21.15.89:0/3189394663 >> v1:172.21.15.192:6789/0 conn(0x7f6ae40cf160 legacy=0x7f6ae40cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:34.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.979+0000 7f6ae8d75640 1 -- 172.21.15.89:0/3189394663 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae40cad70 con 0x7f6ae41538a0 2023-10-30T12:45:34.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.979+0000 7f6ae3fff640 1 --2- 172.21.15.89:0/3189394663 >> v2:172.21.15.192:3300/0 conn(0x7f6ae40ca0e0 0x7f6ae415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:45:34.982 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.983+0000 7f6ae8d75640 1 --2- 172.21.15.89:0/3189394663 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ae41538a0 0x7f6ae415d6f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8009870 tx=0x7f6ad801ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7fbd86d2e8fb421 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:34.982 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.983+0000 7f6ae37fe640 1 -- 172.21.15.89:0/3189394663 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad80a8a30 con 0x7f6ae41538a0 2023-10-30T12:45:34.982 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.983+0000 7f6ae37fe640 1 -- 172.21.15.89:0/3189394663 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad80a8b90 con 0x7f6ae41538a0 2023-10-30T12:45:34.982 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.983+0000 7f6ae37fe640 1 -- 172.21.15.89:0/3189394663 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad80b0440 con 0x7f6ae41538a0 2023-10-30T12:45:34.982 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.983+0000 7f6aeafd9640 1 -- 172.21.15.89:0/3189394663 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ae41538a0 msgr2=0x7f6ae415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:34.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.983+0000 7f6aeafd9640 1 --2- 172.21.15.89:0/3189394663 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ae41538a0 0x7f6ae415d6f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8009870 tx=0x7f6ad801ed30 comp rx=0 tx=0).stop 2023-10-30T12:45:34.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.983+0000 7f6aeafd9640 1 -- 172.21.15.89:0/3189394663 shutdown_connections 2023-10-30T12:45:34.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.983+0000 7f6aeafd9640 1 --2- 172.21.15.89:0/3189394663 >> v2:172.21.15.192:3300/0 conn(0x7f6ae40ca0e0 0x7f6ae415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:34.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.983+0000 7f6aeafd9640 1 --2- 172.21.15.89:0/3189394663 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ae41538a0 0x7f6ae415d6f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:34.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.983+0000 7f6aeafd9640 1 -- 172.21.15.89:0/3189394663 >> 172.21.15.89:0/3189394663 conn(0x7f6ae406bfe0 msgr2=0x7f6ae406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:45:34.984 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.983+0000 7f6aeafd9640 1 -- 172.21.15.89:0/3189394663 shutdown_connections 2023-10-30T12:45:34.984 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.983+0000 7f6aeafd9640 1 -- 172.21.15.89:0/3189394663 wait complete. 2023-10-30T12:45:34.985 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.987+0000 7f6aeafd9640 1 Processor -- start 2023-10-30T12:45:34.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.987+0000 7f6aeafd9640 1 -- start start 2023-10-30T12:45:34.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.987+0000 7f6aeafd9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ae40ca0e0 0x7f6ae40ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:34.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.987+0000 7f6aeafd9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ae40cf160 0x7f6ae40b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:34.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.987+0000 7f6aeafd9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ae41538a0 0x7f6ae40b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:34.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.987+0000 7f6aeafd9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae41efcc0 con 0x7f6ae40ca0e0 2023-10-30T12:45:34.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.987+0000 7f6aeafd9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae41f01f0 con 0x7f6ae40cf160 2023-10-30T12:45:34.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.987+0000 7f6aeafd9640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae41eff10 con 0x7f6ae41538a0 2023-10-30T12:45:34.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.987+0000 7f6ae8d75640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ae40ca0e0 0x7f6ae40ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:34.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.987+0000 7f6ae8d75640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ae40ca0e0 0x7f6ae40ce260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59136/0 (socket says 172.21.15.89:59136) 2023-10-30T12:45:34.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.987+0000 7f6ae8d75640 1 -- 172.21.15.89:0/3980466743 learned_addr learned my addr 172.21.15.89:0/3980466743 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:45:34.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.987+0000 7f6ae9576640 1 --2- 172.21.15.89:0/3980466743 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ae41538a0 0x7f6ae40b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:34.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.987+0000 7f6ae3fff640 1 --2- 172.21.15.89:0/3980466743 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ae40cf160 0x7f6ae40b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:34.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.987+0000 7f6ae8d75640 1 -- 172.21.15.89:0/3980466743 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ae41538a0 msgr2=0x7f6ae40b42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:34.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.987+0000 7f6ae8d75640 1 --2- 172.21.15.89:0/3980466743 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ae41538a0 0x7f6ae40b42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:34.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.987+0000 7f6ae8d75640 1 -- 172.21.15.89:0/3980466743 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ae40cf160 msgr2=0x7f6ae40b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:34.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.987+0000 7f6ae8d75640 1 --2- 172.21.15.89:0/3980466743 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ae40cf160 0x7f6ae40b3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:34.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.987+0000 7f6ae8d75640 1 -- 172.21.15.89:0/3980466743 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad0009420 con 0x7f6ae40ca0e0 2023-10-30T12:45:34.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.987+0000 7f6ae8d75640 1 --2- 172.21.15.89:0/3980466743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ae40ca0e0 0x7f6ae40ce260 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f6ad80b3040 tx=0x7f6ad801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:34.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.987+0000 7f6ae17fa640 1 -- 172.21.15.89:0/3980466743 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad80b9620 con 0x7f6ae40ca0e0 2023-10-30T12:45:34.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.987+0000 7f6ae17fa640 1 -- 172.21.15.89:0/3980466743 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad80b97a0 con 0x7f6ae40ca0e0 2023-10-30T12:45:34.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.987+0000 7f6aeafd9640 1 -- 172.21.15.89:0/3980466743 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ad80094d0 con 0x7f6ae40ca0e0 2023-10-30T12:45:34.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.987+0000 7f6ae17fa640 1 -- 172.21.15.89:0/3980466743 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad80b9b50 con 0x7f6ae40ca0e0 2023-10-30T12:45:34.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.987+0000 7f6aeafd9640 1 -- 172.21.15.89:0/3980466743 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ae40b35b0 con 0x7f6ae40ca0e0 2023-10-30T12:45:34.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.991+0000 7f6aeafd9640 1 -- 172.21.15.89:0/3980466743 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ae40b2a50 con 0x7f6ae40ca0e0 2023-10-30T12:45:34.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.991+0000 7f6ae17fa640 1 -- 172.21.15.89:0/3980466743 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6ad80b7070 con 0x7f6ae40ca0e0 2023-10-30T12:45:34.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.991+0000 7f6ae17fa640 1 --2- 172.21.15.89:0/3980466743 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6ac80664d0 0x7f6ac8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:34.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.991+0000 7f6ae17fa640 1 -- 172.21.15.89:0/3980466743 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6406+0+0 (secure 0 0 0) 0x7f6ad8121f80 con 0x7f6ae40ca0e0 2023-10-30T12:45:34.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.991+0000 7f6ae3fff640 1 --2- 172.21.15.89:0/3980466743 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6ac80664d0 0x7f6ac8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:34.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.991+0000 7f6ae3fff640 1 --2- 172.21.15.89:0/3980466743 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6ac80664d0 0x7f6ac8068990 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6ae41f3870 tx=0x7f6ad0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:34.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:34.995+0000 7f6ae17fa640 1 -- 172.21.15.89:0/3980466743 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6ad80ef130 con 0x7f6ae40ca0e0 2023-10-30T12:45:35.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:35.175+0000 7f6aeafd9640 1 -- 172.21.15.89:0/3980466743 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ae404e170 con 0x7f6ae40ca0e0 2023-10-30T12:45:35.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:35.175+0000 7f6ae17fa640 1 -- 172.21.15.89:0/3980466743 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f6ad80f2750 con 0x7f6ae40ca0e0 2023-10-30T12:45:35.174 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:45:35.174 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5450,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:44:45.324321+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:45:35.175 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 82 2023-10-30T12:45:35.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:35.179+0000 7f6aeafd9640 1 -- 172.21.15.89:0/3980466743 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6ac80664d0 msgr2=0x7f6ac8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:35.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:35.179+0000 7f6aeafd9640 1 --2- 172.21.15.89:0/3980466743 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6ac80664d0 0x7f6ac8068990 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6ae41f3870 tx=0x7f6ad0008040 comp rx=0 tx=0).stop 2023-10-30T12:45:35.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:35.179+0000 7f6aeafd9640 1 -- 172.21.15.89:0/3980466743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ae40ca0e0 msgr2=0x7f6ae40ce260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:35.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:35.179+0000 7f6aeafd9640 1 --2- 172.21.15.89:0/3980466743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ae40ca0e0 0x7f6ae40ce260 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f6ad80b3040 tx=0x7f6ad801bad0 comp rx=0 tx=0).stop 2023-10-30T12:45:35.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:35.179+0000 7f6aeafd9640 1 -- 172.21.15.89:0/3980466743 shutdown_connections 2023-10-30T12:45:35.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:35.179+0000 7f6aeafd9640 1 --2- 172.21.15.89:0/3980466743 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6ac80664d0 0x7f6ac8068990 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:35.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:35.179+0000 7f6aeafd9640 1 --2- 172.21.15.89:0/3980466743 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ae40cf160 0x7f6ae40b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:35.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:35.179+0000 7f6aeafd9640 1 --2- 172.21.15.89:0/3980466743 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ae41538a0 0x7f6ae40b42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:35.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:35.179+0000 7f6aeafd9640 1 --2- 172.21.15.89:0/3980466743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ae40ca0e0 0x7f6ae40ce260 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:35.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:35.179+0000 7f6aeafd9640 1 -- 172.21.15.89:0/3980466743 >> 172.21.15.89:0/3980466743 conn(0x7f6ae406bfe0 msgr2=0x7f6ae4152870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:45:35.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:35.179+0000 7f6aeafd9640 1 -- 172.21.15.89:0/3980466743 shutdown_connections 2023-10-30T12:45:35.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:35.179+0000 7f6aeafd9640 1 -- 172.21.15.89:0/3980466743 wait complete. 2023-10-30T12:45:35.195 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:45:37.197 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:45:37.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.347+0000 7f1567180640 1 Processor -- start 2023-10-30T12:45:37.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.347+0000 7f1567180640 1 -- start start 2023-10-30T12:45:37.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.347+0000 7f1567180640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f156006f7a0 0x7f156006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:37.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.347+0000 7f1567180640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f15600c6bd0 0x7f1560069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:37.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.347+0000 7f1567180640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1560071110 0x7f1560158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:37.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.347+0000 7f1567180640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15600700c0 con 0x7f156006f7a0 2023-10-30T12:45:37.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.347+0000 7f1567180640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f156006a3d0 con 0x7f15600c6bd0 2023-10-30T12:45:37.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.347+0000 7f1567180640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1560066cf0 con 0x7f1560071110 2023-10-30T12:45:37.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.347+0000 7f1564f1c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f156006f7a0 0x7f156006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:37.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.347+0000 7f1564f1c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f156006f7a0 0x7f156006fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59146/0 (socket says 172.21.15.89:59146) 2023-10-30T12:45:37.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.347+0000 7f1564f1c640 1 -- 172.21.15.89:0/3226335012 learned_addr learned my addr 172.21.15.89:0/3226335012 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:45:37.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.351+0000 7f156571d640 1 --2- 172.21.15.89:0/3226335012 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1560071110 0x7f1560158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:37.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.351+0000 7f155ffff640 1 --2- 172.21.15.89:0/3226335012 >> v2:172.21.15.192:3300/0 conn(0x7f15600c6bd0 0x7f1560069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:37.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.351+0000 7f1564f1c640 1 -- 172.21.15.89:0/3226335012 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1560071110 msgr2=0x7f1560158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:45:37.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.351+0000 7f1564f1c640 1 --2- 172.21.15.89:0/3226335012 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1560071110 0x7f1560158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:37.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.351+0000 7f1564f1c640 1 -- 172.21.15.89:0/3226335012 >> v2:172.21.15.192:3300/0 conn(0x7f15600c6bd0 msgr2=0x7f1560069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:45:37.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.351+0000 7f1564f1c640 1 --2- 172.21.15.89:0/3226335012 >> v2:172.21.15.192:3300/0 conn(0x7f15600c6bd0 0x7f1560069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:37.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.351+0000 7f1564f1c640 1 -- 172.21.15.89:0/3226335012 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15600c70d0 con 0x7f156006f7a0 2023-10-30T12:45:37.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.351+0000 7f1564f1c640 1 --2- 172.21.15.89:0/3226335012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f156006f7a0 0x7f156006fb80 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f156006f1c0 tx=0x7f155401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5de0d91db21a8391 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:37.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.351+0000 7f155f7fe640 1 -- 172.21.15.89:0/3226335012 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15540a89b0 con 0x7f156006f7a0 2023-10-30T12:45:37.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.351+0000 7f155f7fe640 1 -- 172.21.15.89:0/3226335012 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15540a8b10 con 0x7f156006f7a0 2023-10-30T12:45:37.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.351+0000 7f155f7fe640 1 -- 172.21.15.89:0/3226335012 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15540a8ea0 con 0x7f156006f7a0 2023-10-30T12:45:37.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.355+0000 7f1567180640 1 -- 172.21.15.89:0/3226335012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f156006f7a0 msgr2=0x7f156006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:37.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.355+0000 7f1567180640 1 --2- 172.21.15.89:0/3226335012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f156006f7a0 0x7f156006fb80 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f156006f1c0 tx=0x7f155401ecf0 comp rx=0 tx=0).stop 2023-10-30T12:45:37.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.355+0000 7f1567180640 1 -- 172.21.15.89:0/3226335012 shutdown_connections 2023-10-30T12:45:37.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.355+0000 7f1567180640 1 --2- 172.21.15.89:0/3226335012 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1560071110 0x7f1560158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:37.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.355+0000 7f1567180640 1 --2- 172.21.15.89:0/3226335012 >> v2:172.21.15.192:3300/0 conn(0x7f15600c6bd0 0x7f1560069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:37.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.355+0000 7f1567180640 1 --2- 172.21.15.89:0/3226335012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f156006f7a0 0x7f156006fb80 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:37.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.355+0000 7f1567180640 1 -- 172.21.15.89:0/3226335012 >> 172.21.15.89:0/3226335012 conn(0x7f15600c7910 msgr2=0x7f15600c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:45:37.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.355+0000 7f1567180640 1 -- 172.21.15.89:0/3226335012 shutdown_connections 2023-10-30T12:45:37.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.355+0000 7f1567180640 1 -- 172.21.15.89:0/3226335012 wait complete. 2023-10-30T12:45:37.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.355+0000 7f1567180640 1 Processor -- start 2023-10-30T12:45:37.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.355+0000 7f1567180640 1 -- start start 2023-10-30T12:45:37.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.355+0000 7f1567180640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f156006f7a0 0x7f1560147c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:37.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.355+0000 7f1567180640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1560071110 0x7f1560150760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:37.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.355+0000 7f1567180640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15600c6bd0 0x7f1560150ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:37.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.355+0000 7f1567180640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15601f2d40 con 0x7f156006f7a0 2023-10-30T12:45:37.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.355+0000 7f1567180640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1560148170 con 0x7f15600c6bd0 2023-10-30T12:45:37.357 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.355+0000 7f1567180640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15601f2f70 con 0x7f1560071110 2023-10-30T12:45:37.357 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.355+0000 7f1564f1c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f156006f7a0 0x7f1560147c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:37.357 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.355+0000 7f1564f1c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f156006f7a0 0x7f1560147c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59158/0 (socket says 172.21.15.89:59158) 2023-10-30T12:45:37.357 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.355+0000 7f1564f1c640 1 -- 172.21.15.89:0/3640678795 learned_addr learned my addr 172.21.15.89:0/3640678795 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:45:37.357 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.355+0000 7f155ffff640 1 --2- 172.21.15.89:0/3640678795 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1560071110 0x7f1560150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:37.357 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.355+0000 7f156571d640 1 --2- 172.21.15.89:0/3640678795 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15600c6bd0 0x7f1560150ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:37.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.355+0000 7f1564f1c640 1 -- 172.21.15.89:0/3640678795 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1560071110 msgr2=0x7f1560150760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:37.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.355+0000 7f1564f1c640 1 --2- 172.21.15.89:0/3640678795 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1560071110 0x7f1560150760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:37.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.355+0000 7f1564f1c640 1 -- 172.21.15.89:0/3640678795 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15600c6bd0 msgr2=0x7f1560150ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:37.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.355+0000 7f1564f1c640 1 --2- 172.21.15.89:0/3640678795 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15600c6bd0 0x7f1560150ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:37.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.355+0000 7f1564f1c640 1 -- 172.21.15.89:0/3640678795 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15540094d0 con 0x7f156006f7a0 2023-10-30T12:45:37.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.355+0000 7f155ffff640 1 --2- 172.21.15.89:0/3640678795 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1560071110 0x7f1560150760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:45:37.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.359+0000 7f1564f1c640 1 --2- 172.21.15.89:0/3640678795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f156006f7a0 0x7f1560147c30 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f155401d3a0 tx=0x7f155401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:37.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.359+0000 7f155d7fa640 1 -- 172.21.15.89:0/3640678795 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15540ba600 con 0x7f156006f7a0 2023-10-30T12:45:37.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.359+0000 7f155d7fa640 1 -- 172.21.15.89:0/3640678795 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15540ba760 con 0x7f156006f7a0 2023-10-30T12:45:37.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.359+0000 7f155d7fa640 1 -- 172.21.15.89:0/3640678795 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15540bab10 con 0x7f156006f7a0 2023-10-30T12:45:37.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.359+0000 7f1567180640 1 -- 172.21.15.89:0/3640678795 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15601f3230 con 0x7f156006f7a0 2023-10-30T12:45:37.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.359+0000 7f1567180640 1 -- 172.21.15.89:0/3640678795 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15600c99a0 con 0x7f156006f7a0 2023-10-30T12:45:37.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.359+0000 7f1567180640 1 -- 172.21.15.89:0/3640678795 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f156004e170 con 0x7f156006f7a0 2023-10-30T12:45:37.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.359+0000 7f155d7fa640 1 -- 172.21.15.89:0/3640678795 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f15540bac70 con 0x7f156006f7a0 2023-10-30T12:45:37.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.359+0000 7f155d7fa640 1 --2- 172.21.15.89:0/3640678795 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f15440664a0 0x7f1544068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:37.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.359+0000 7f155d7fa640 1 -- 172.21.15.89:0/3640678795 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6406+0+0 (secure 0 0 0) 0x7f1554122700 con 0x7f156006f7a0 2023-10-30T12:45:37.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.359+0000 7f155ffff640 1 --2- 172.21.15.89:0/3640678795 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f15440664a0 0x7f1544068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:37.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.363+0000 7f155ffff640 1 --2- 172.21.15.89:0/3640678795 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f15440664a0 0x7f1544068960 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f156014eea0 tx=0x7f154c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:37.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.363+0000 7f155d7fa640 1 -- 172.21.15.89:0/3640678795 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f15540ef8a0 con 0x7f156006f7a0 2023-10-30T12:45:37.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.547+0000 7f1567180640 1 -- 172.21.15.89:0/3640678795 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1560155090 con 0x7f156006f7a0 2023-10-30T12:45:37.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.547+0000 7f155d7fa640 1 -- 172.21.15.89:0/3640678795 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f15540f2ec0 con 0x7f156006f7a0 2023-10-30T12:45:37.544 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:45:37.544 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5450,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:44:45.324321+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:45:37.544 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 82 2023-10-30T12:45:37.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.547+0000 7f1567180640 1 -- 172.21.15.89:0/3640678795 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f15440664a0 msgr2=0x7f1544068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:37.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.547+0000 7f1567180640 1 --2- 172.21.15.89:0/3640678795 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f15440664a0 0x7f1544068960 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f156014eea0 tx=0x7f154c008040 comp rx=0 tx=0).stop 2023-10-30T12:45:37.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.547+0000 7f1567180640 1 -- 172.21.15.89:0/3640678795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f156006f7a0 msgr2=0x7f1560147c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:37.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.547+0000 7f1567180640 1 --2- 172.21.15.89:0/3640678795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f156006f7a0 0x7f1560147c30 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f155401d3a0 tx=0x7f155401ba90 comp rx=0 tx=0).stop 2023-10-30T12:45:37.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.547+0000 7f1567180640 1 -- 172.21.15.89:0/3640678795 shutdown_connections 2023-10-30T12:45:37.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.547+0000 7f1567180640 1 --2- 172.21.15.89:0/3640678795 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f15440664a0 0x7f1544068960 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:37.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.547+0000 7f1567180640 1 --2- 172.21.15.89:0/3640678795 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15600c6bd0 0x7f1560150ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:37.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.547+0000 7f1567180640 1 --2- 172.21.15.89:0/3640678795 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1560071110 0x7f1560150760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:37.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.547+0000 7f1567180640 1 --2- 172.21.15.89:0/3640678795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f156006f7a0 0x7f1560147c30 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:37.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.547+0000 7f1567180640 1 -- 172.21.15.89:0/3640678795 >> 172.21.15.89:0/3640678795 conn(0x7f15600c7910 msgr2=0x7f156006be40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:45:37.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.547+0000 7f1567180640 1 -- 172.21.15.89:0/3640678795 shutdown_connections 2023-10-30T12:45:37.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:37.547+0000 7f1567180640 1 -- 172.21.15.89:0/3640678795 wait complete. 2023-10-30T12:45:37.565 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:45:39.566 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:45:39.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db91a7640 1 Processor -- start 2023-10-30T12:45:39.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db91a7640 1 -- start start 2023-10-30T12:45:39.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db91a7640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0db40c6840 0x7f0db415a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:39.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db91a7640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db40c3b30 con 0x7f0db40c3640 2023-10-30T12:45:39.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db91a7640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db40c3df0 con 0x7f0db40c6c20 2023-10-30T12:45:39.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db91a7640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db40c2aa0 con 0x7f0db40c6840 2023-10-30T12:45:39.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db37fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f0db40c3640 0x7f0db4153730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43334/0 (socket says 172.21.15.89:43334) 2023-10-30T12:45:39.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db37fe640 1 -- 172.21.15.89:0/1523230114 learned_addr learned my addr 172.21.15.89:0/1523230114 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:45:39.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db3fff640 1 --2- 172.21.15.89:0/1523230114 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0db40c6840 0x7f0db415a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:39.718 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db2ffd640 1 -- 172.21.15.89:0/1523230114 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 758784044 0 0) 0x7f0db40c3b30 con 0x7f0db40c3640 2023-10-30T12:45:39.718 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db2ffd640 1 -- 172.21.15.89:0/1523230114 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d9c005740 con 0x7f0db40c3640 2023-10-30T12:45:39.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db2ffd640 1 -- 172.21.15.89:0/1523230114 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048371559 0 0) 0x7f0db40c3df0 con 0x7f0db40c6c20 2023-10-30T12:45:39.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db2ffd640 1 -- 172.21.15.89:0/1523230114 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db40c3b30 con 0x7f0db40c6c20 2023-10-30T12:45:39.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db2ffd640 1 -- 172.21.15.89:0/1523230114 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 42586340 0 0) 0x7f0d9c005740 con 0x7f0db40c3640 2023-10-30T12:45:39.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db2ffd640 1 -- 172.21.15.89:0/1523230114 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0db40c6840 msgr2=0x7f0db415a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:45:39.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db2ffd640 1 --2- 172.21.15.89:0/1523230114 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0db40c6840 0x7f0db415a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:39.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db2ffd640 1 -- 172.21.15.89:0/1523230114 >> v1:172.21.15.192:6789/0 conn(0x7f0db40c6c20 legacy=0x7f0db4160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:39.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db2ffd640 1 -- 172.21.15.89:0/1523230114 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db4159ba0 con 0x7f0db40c3640 2023-10-30T12:45:39.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db2ffd640 1 -- 172.21.15.89:0/1523230114 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f0da8001680 con 0x7f0db40c3640 2023-10-30T12:45:39.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db2ffd640 1 -- 172.21.15.89:0/1523230114 >> v1:172.21.15.89:6789/0 conn(0x7f0db40c3640 legacy=0x7f0db4153730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:39.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db2ffd640 1 --2- 172.21.15.89:0/1523230114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d9c005d50 0x7f0d9c006150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:39.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db2ffd640 1 --2- 172.21.15.89:0/1523230114 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0db40c6840 0x7f0d9c0069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:39.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db2ffd640 1 --2- 172.21.15.89:0/1523230114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d9c006f20 0x7f0d9c0073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:39.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db2ffd640 1 -- 172.21.15.89:0/1523230114 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9c00dc50 con 0x7f0d9c006f20 2023-10-30T12:45:39.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db2ffd640 1 -- 172.21.15.89:0/1523230114 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9c00dd90 con 0x7f0d9c005d50 2023-10-30T12:45:39.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db2ffd640 1 -- 172.21.15.89:0/1523230114 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9c00df50 con 0x7f0db40c6840 2023-10-30T12:45:39.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db89a6640 1 --2- 172.21.15.89:0/1523230114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d9c006f20 0x7f0d9c0073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:39.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db37fe640 1 --2- 172.21.15.89:0/1523230114 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0db40c6840 0x7f0d9c0069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:39.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db3fff640 1 --2- 172.21.15.89:0/1523230114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d9c005d50 0x7f0d9c006150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:39.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db89a6640 1 -- 172.21.15.89:0/1523230114 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0db40c6840 msgr2=0x7f0d9c0069e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:45:39.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db89a6640 1 --2- 172.21.15.89:0/1523230114 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0db40c6840 0x7f0d9c0069e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:39.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db89a6640 1 -- 172.21.15.89:0/1523230114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d9c005d50 msgr2=0x7f0d9c006150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:45:39.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db89a6640 1 --2- 172.21.15.89:0/1523230114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d9c005d50 0x7f0d9c006150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:39.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.719+0000 7f0db89a6640 1 -- 172.21.15.89:0/1523230114 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d9c005740 con 0x7f0d9c006f20 2023-10-30T12:45:39.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.723+0000 7f0db89a6640 1 --2- 172.21.15.89:0/1523230114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d9c006f20 0x7f0d9c0073a0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f0db40c7040 tx=0x7f0dac01c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24e9ac27bea4a314 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:39.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.723+0000 7f0db2ffd640 1 -- 172.21.15.89:0/1523230114 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dac003b80 con 0x7f0d9c006f20 2023-10-30T12:45:39.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.723+0000 7f0db2ffd640 1 -- 172.21.15.89:0/1523230114 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dac003d00 con 0x7f0d9c006f20 2023-10-30T12:45:39.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.723+0000 7f0db2ffd640 1 -- 172.21.15.89:0/1523230114 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dac005310 con 0x7f0d9c006f20 2023-10-30T12:45:39.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.723+0000 7f0db91a7640 1 -- 172.21.15.89:0/1523230114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d9c006f20 msgr2=0x7f0d9c0073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:39.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.723+0000 7f0db91a7640 1 --2- 172.21.15.89:0/1523230114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d9c006f20 0x7f0d9c0073a0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f0db40c7040 tx=0x7f0dac01c8c0 comp rx=0 tx=0).stop 2023-10-30T12:45:39.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.723+0000 7f0db89a6640 1 -- 172.21.15.89:0/1523230114 reap_dead start 2023-10-30T12:45:39.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.723+0000 7f0db91a7640 1 -- 172.21.15.89:0/1523230114 shutdown_connections 2023-10-30T12:45:39.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.723+0000 7f0db91a7640 1 -- 172.21.15.89:0/1523230114 >> 172.21.15.89:0/1523230114 conn(0x7f0db40c75d0 msgr2=0x7f0db40c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:45:39.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.723+0000 7f0db91a7640 1 -- 172.21.15.89:0/1523230114 shutdown_connections 2023-10-30T12:45:39.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.723+0000 7f0db91a7640 1 -- 172.21.15.89:0/1523230114 wait complete. 2023-10-30T12:45:39.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.727+0000 7f0db91a7640 1 Processor -- start 2023-10-30T12:45:39.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.727+0000 7f0db91a7640 1 -- start start 2023-10-30T12:45:39.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.727+0000 7f0db91a7640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d9c006f20 0x7f0db40c5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:39.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.727+0000 7f0db91a7640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0db40ca170 0x7f0db40c8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:39.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.727+0000 7f0db91a7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0db40c8e70 0x7f0db4146f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:39.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.727+0000 7f0db91a7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db406fc20 con 0x7f0db40c8e70 2023-10-30T12:45:39.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.727+0000 7f0db91a7640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db4070910 con 0x7f0d9c006f20 2023-10-30T12:45:39.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.727+0000 7f0db91a7640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db414d6b0 con 0x7f0db40ca170 2023-10-30T12:45:39.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.727+0000 7f0db89a6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0db40c8e70 0x7f0db4146f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:39.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.727+0000 7f0db89a6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0db40c8e70 0x7f0db4146f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52892/0 (socket says 172.21.15.89:52892) 2023-10-30T12:45:39.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.727+0000 7f0db89a6640 1 -- 172.21.15.89:0/1668401395 learned_addr learned my addr 172.21.15.89:0/1668401395 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:45:39.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.727+0000 7f0db3fff640 1 --2- 172.21.15.89:0/1668401395 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d9c006f20 0x7f0db40c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:39.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.727+0000 7f0db37fe640 1 --2- 172.21.15.89:0/1668401395 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0db40ca170 0x7f0db40c8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:39.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.727+0000 7f0db89a6640 1 -- 172.21.15.89:0/1668401395 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0db40ca170 msgr2=0x7f0db40c8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:39.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.727+0000 7f0db89a6640 1 --2- 172.21.15.89:0/1668401395 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0db40ca170 0x7f0db40c8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:39.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.727+0000 7f0db89a6640 1 -- 172.21.15.89:0/1668401395 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d9c006f20 msgr2=0x7f0db40c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:39.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.727+0000 7f0db89a6640 1 --2- 172.21.15.89:0/1668401395 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d9c006f20 0x7f0db40c5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:39.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.727+0000 7f0db89a6640 1 -- 172.21.15.89:0/1668401395 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db414d990 con 0x7f0db40c8e70 2023-10-30T12:45:39.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.727+0000 7f0db89a6640 1 --2- 172.21.15.89:0/1668401395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0db40c8e70 0x7f0db4146f60 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f0dac005040 tx=0x7f0d9c007a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:39.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.727+0000 7f0db0ff9640 1 -- 172.21.15.89:0/1668401395 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dac003b90 con 0x7f0db40c8e70 2023-10-30T12:45:39.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.727+0000 7f0db0ff9640 1 -- 172.21.15.89:0/1668401395 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dac003e70 con 0x7f0db40c8e70 2023-10-30T12:45:39.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.727+0000 7f0db0ff9640 1 -- 172.21.15.89:0/1668401395 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dac006d60 con 0x7f0db40c8e70 2023-10-30T12:45:39.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.727+0000 7f0db91a7640 1 -- 172.21.15.89:0/1668401395 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db406fe30 con 0x7f0db40c8e70 2023-10-30T12:45:39.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.727+0000 7f0db91a7640 1 -- 172.21.15.89:0/1668401395 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db40703f0 con 0x7f0db40c8e70 2023-10-30T12:45:39.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.731+0000 7f0db91a7640 1 -- 172.21.15.89:0/1668401395 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db4070610 con 0x7f0db40c8e70 2023-10-30T12:45:39.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.731+0000 7f0db0ff9640 1 -- 172.21.15.89:0/1668401395 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0dac005540 con 0x7f0db40c8e70 2023-10-30T12:45:39.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.731+0000 7f0db0ff9640 1 --2- 172.21.15.89:0/1668401395 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0da00664a0 0x7f0da0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:39.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.731+0000 7f0db0ff9640 1 -- 172.21.15.89:0/1668401395 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6406+0+0 (secure 0 0 0) 0x7f0dac11d6f0 con 0x7f0db40c8e70 2023-10-30T12:45:39.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.731+0000 7f0db3fff640 1 --2- 172.21.15.89:0/1668401395 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0da00664a0 0x7f0da0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:39.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.731+0000 7f0db3fff640 1 --2- 172.21.15.89:0/1668401395 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0da00664a0 0x7f0da0068960 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0da4009b00 tx=0x7f0da4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:39.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.735+0000 7f0db0ff9640 1 -- 172.21.15.89:0/1668401395 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0dac0ea780 con 0x7f0db40c8e70 2023-10-30T12:45:39.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.915+0000 7f0db91a7640 1 -- 172.21.15.89:0/1668401395 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0db4070610 con 0x7f0db40c8e70 2023-10-30T12:45:39.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.915+0000 7f0db0ff9640 1 -- 172.21.15.89:0/1668401395 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f0dac0bc090 con 0x7f0db40c8e70 2023-10-30T12:45:39.914 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:45:39.914 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5450,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:44:45.324321+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:45:39.914 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 82 2023-10-30T12:45:39.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.919+0000 7f0db91a7640 1 -- 172.21.15.89:0/1668401395 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0da00664a0 msgr2=0x7f0da0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:39.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.919+0000 7f0db91a7640 1 --2- 172.21.15.89:0/1668401395 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0da00664a0 0x7f0da0068960 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0da4009b00 tx=0x7f0da4008040 comp rx=0 tx=0).stop 2023-10-30T12:45:39.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.919+0000 7f0db91a7640 1 -- 172.21.15.89:0/1668401395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0db40c8e70 msgr2=0x7f0db4146f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:39.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.919+0000 7f0db91a7640 1 --2- 172.21.15.89:0/1668401395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0db40c8e70 0x7f0db4146f60 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f0dac005040 tx=0x7f0d9c007a10 comp rx=0 tx=0).stop 2023-10-30T12:45:39.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.919+0000 7f0db91a7640 1 -- 172.21.15.89:0/1668401395 shutdown_connections 2023-10-30T12:45:39.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.919+0000 7f0db91a7640 1 --2- 172.21.15.89:0/1668401395 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0da00664a0 0x7f0da0068960 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:39.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.919+0000 7f0db91a7640 1 --2- 172.21.15.89:0/1668401395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0db40c8e70 0x7f0db4146f60 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:39.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.919+0000 7f0db91a7640 1 --2- 172.21.15.89:0/1668401395 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0db40ca170 0x7f0db40c8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:39.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.919+0000 7f0db91a7640 1 --2- 172.21.15.89:0/1668401395 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d9c006f20 0x7f0db40c5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:39.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.919+0000 7f0db91a7640 1 -- 172.21.15.89:0/1668401395 >> 172.21.15.89:0/1668401395 conn(0x7f0db40c75d0 msgr2=0x7f0db4151190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:45:39.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.919+0000 7f0db91a7640 1 -- 172.21.15.89:0/1668401395 shutdown_connections 2023-10-30T12:45:39.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:39.919+0000 7f0db91a7640 1 -- 172.21.15.89:0/1668401395 wait complete. 2023-10-30T12:45:39.934 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:45:41.935 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:45:42.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.087+0000 7f31ef7f5640 1 Processor -- start 2023-10-30T12:45:42.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.087+0000 7f31ef7f5640 1 -- start start 2023-10-30T12:45:42.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.087+0000 7f31ef7f5640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31e806f7a0 0x7f31e806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:42.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.087+0000 7f31ef7f5640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f31e80c6bd0 0x7f31e8069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:42.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.087+0000 7f31ef7f5640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f31e8071110 0x7f31e8158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:42.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.087+0000 7f31ef7f5640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31e80700c0 con 0x7f31e8071110 2023-10-30T12:45:42.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.087+0000 7f31ef7f5640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31e806a3d0 con 0x7f31e80c6bd0 2023-10-30T12:45:42.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.087+0000 7f31ef7f5640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31e8066cf0 con 0x7f31e806f7a0 2023-10-30T12:45:42.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.087+0000 7f31edd92640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f31e8071110 0x7f31e8158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:42.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.087+0000 7f31edd92640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f31e8071110 0x7f31e8158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52900/0 (socket says 172.21.15.89:52900) 2023-10-30T12:45:42.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.087+0000 7f31ed591640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31e806f7a0 0x7f31e806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:42.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.087+0000 7f31edd92640 1 -- 172.21.15.89:0/1381849800 learned_addr learned my addr 172.21.15.89:0/1381849800 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:45:42.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.087+0000 7f31ecd90640 1 --2- 172.21.15.89:0/1381849800 >> v2:172.21.15.192:3300/0 conn(0x7f31e80c6bd0 0x7f31e8069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:42.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.091+0000 7f31edd92640 1 -- 172.21.15.89:0/1381849800 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31e806f7a0 msgr2=0x7f31e806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:45:42.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.091+0000 7f31edd92640 1 --2- 172.21.15.89:0/1381849800 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31e806f7a0 0x7f31e806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:42.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.091+0000 7f31edd92640 1 -- 172.21.15.89:0/1381849800 >> v2:172.21.15.192:3300/0 conn(0x7f31e80c6bd0 msgr2=0x7f31e8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:45:42.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.091+0000 7f31edd92640 1 --2- 172.21.15.89:0/1381849800 >> v2:172.21.15.192:3300/0 conn(0x7f31e80c6bd0 0x7f31e8069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:42.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.091+0000 7f31edd92640 1 -- 172.21.15.89:0/1381849800 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e80c70d0 con 0x7f31e8071110 2023-10-30T12:45:42.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.091+0000 7f31edd92640 1 --2- 172.21.15.89:0/1381849800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31e8071110 0x7f31e8158110 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f31e80b1210 tx=0x7f31e001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da8dc51c38dc0069 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:42.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.091+0000 7f31dffff640 1 -- 172.21.15.89:0/1381849800 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e00af400 con 0x7f31e8071110 2023-10-30T12:45:42.091 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.091+0000 7f31dffff640 1 -- 172.21.15.89:0/1381849800 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31e00ab050 con 0x7f31e8071110 2023-10-30T12:45:42.091 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.091+0000 7f31dffff640 1 -- 172.21.15.89:0/1381849800 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e00af780 con 0x7f31e8071110 2023-10-30T12:45:42.091 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.091+0000 7f31ef7f5640 1 -- 172.21.15.89:0/1381849800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31e8071110 msgr2=0x7f31e8158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:42.091 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.091+0000 7f31ef7f5640 1 --2- 172.21.15.89:0/1381849800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31e8071110 0x7f31e8158110 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f31e80b1210 tx=0x7f31e001f1c0 comp rx=0 tx=0).stop 2023-10-30T12:45:42.091 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.091+0000 7f31ef7f5640 1 -- 172.21.15.89:0/1381849800 shutdown_connections 2023-10-30T12:45:42.091 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.091+0000 7f31ef7f5640 1 --2- 172.21.15.89:0/1381849800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31e8071110 0x7f31e8158110 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:42.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.091+0000 7f31ef7f5640 1 --2- 172.21.15.89:0/1381849800 >> v2:172.21.15.192:3300/0 conn(0x7f31e80c6bd0 0x7f31e8069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:42.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.091+0000 7f31ef7f5640 1 --2- 172.21.15.89:0/1381849800 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31e806f7a0 0x7f31e806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:42.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.095+0000 7f31ef7f5640 1 -- 172.21.15.89:0/1381849800 >> 172.21.15.89:0/1381849800 conn(0x7f31e80c7910 msgr2=0x7f31e80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:45:42.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.095+0000 7f31ef7f5640 1 -- 172.21.15.89:0/1381849800 shutdown_connections 2023-10-30T12:45:42.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.095+0000 7f31ef7f5640 1 -- 172.21.15.89:0/1381849800 wait complete. 2023-10-30T12:45:42.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.095+0000 7f31ef7f5640 1 Processor -- start 2023-10-30T12:45:42.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.095+0000 7f31ef7f5640 1 -- start start 2023-10-30T12:45:42.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.095+0000 7f31ef7f5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31e806f7a0 0x7f31e814d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:42.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.095+0000 7f31ef7f5640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31e8071110 0x7f31e80c1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:42.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.095+0000 7f31ef7f5640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31e80c6bd0 0x7f31e80aea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:42.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.095+0000 7f31ef7f5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e80ae5f0 con 0x7f31e806f7a0 2023-10-30T12:45:42.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.095+0000 7f31ef7f5640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e814d550 con 0x7f31e8071110 2023-10-30T12:45:42.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.095+0000 7f31ef7f5640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31e81ef360 con 0x7f31e80c6bd0 2023-10-30T12:45:42.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.095+0000 7f31ed591640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31e806f7a0 0x7f31e814d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:42.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.095+0000 7f31ed591640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31e806f7a0 0x7f31e814d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52914/0 (socket says 172.21.15.89:52914) 2023-10-30T12:45:42.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.095+0000 7f31ed591640 1 -- 172.21.15.89:0/575429270 learned_addr learned my addr 172.21.15.89:0/575429270 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:45:42.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.095+0000 7f31edd92640 1 --2- 172.21.15.89:0/575429270 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31e80c6bd0 0x7f31e80aea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:42.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.095+0000 7f31ecd90640 1 --2- 172.21.15.89:0/575429270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31e8071110 0x7f31e80c1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:42.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.095+0000 7f31ed591640 1 -- 172.21.15.89:0/575429270 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31e80c6bd0 msgr2=0x7f31e80aea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:42.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.095+0000 7f31ed591640 1 --2- 172.21.15.89:0/575429270 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31e80c6bd0 0x7f31e80aea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:42.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.095+0000 7f31ed591640 1 -- 172.21.15.89:0/575429270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31e8071110 msgr2=0x7f31e80c1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:42.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.095+0000 7f31ed591640 1 --2- 172.21.15.89:0/575429270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31e8071110 0x7f31e80c1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:42.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.095+0000 7f31ed591640 1 -- 172.21.15.89:0/575429270 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e0009b30 con 0x7f31e806f7a0 2023-10-30T12:45:42.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.095+0000 7f31ecd90640 1 --2- 172.21.15.89:0/575429270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31e8071110 0x7f31e80c1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:45:42.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.095+0000 7f31ed591640 1 --2- 172.21.15.89:0/575429270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31e806f7a0 0x7f31e814d010 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f31e400e8a0 tx=0x7f31e400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:42.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.095+0000 7f31ddffb640 1 -- 172.21.15.89:0/575429270 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e400ee90 con 0x7f31e806f7a0 2023-10-30T12:45:42.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.095+0000 7f31ddffb640 1 -- 172.21.15.89:0/575429270 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31e400c940 con 0x7f31e806f7a0 2023-10-30T12:45:42.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.095+0000 7f31ddffb640 1 -- 172.21.15.89:0/575429270 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e400ccf0 con 0x7f31e806f7a0 2023-10-30T12:45:42.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.099+0000 7f31ef7f5640 1 -- 172.21.15.89:0/575429270 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31e814dbb0 con 0x7f31e806f7a0 2023-10-30T12:45:42.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.099+0000 7f31ef7f5640 1 -- 172.21.15.89:0/575429270 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31e80adc50 con 0x7f31e806f7a0 2023-10-30T12:45:42.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.099+0000 7f31ef7f5640 1 -- 172.21.15.89:0/575429270 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31e804e170 con 0x7f31e806f7a0 2023-10-30T12:45:42.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.099+0000 7f31ddffb640 1 -- 172.21.15.89:0/575429270 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f31e4013470 con 0x7f31e806f7a0 2023-10-30T12:45:42.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.099+0000 7f31ddffb640 1 --2- 172.21.15.89:0/575429270 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f31cc0664d0 0x7f31cc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:42.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.099+0000 7f31ddffb640 1 -- 172.21.15.89:0/575429270 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6406+0+0 (secure 0 0 0) 0x7f31e40841c0 con 0x7f31e806f7a0 2023-10-30T12:45:42.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.099+0000 7f31ecd90640 1 --2- 172.21.15.89:0/575429270 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f31cc0664d0 0x7f31cc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:42.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.103+0000 7f31ecd90640 1 --2- 172.21.15.89:0/575429270 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f31cc0664d0 0x7f31cc068990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f31e8158550 tx=0x7f31d8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:42.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.103+0000 7f31ddffb640 1 -- 172.21.15.89:0/575429270 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f31e4051390 con 0x7f31e806f7a0 2023-10-30T12:45:42.288 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.291+0000 7f31ef7f5640 1 -- 172.21.15.89:0/575429270 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31e80c4990 con 0x7f31e806f7a0 2023-10-30T12:45:42.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.291+0000 7f31ddffb640 1 -- 172.21.15.89:0/575429270 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f31e40549b0 con 0x7f31e806f7a0 2023-10-30T12:45:42.289 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:45:42.289 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5450,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:44:45.324321+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:45:42.290 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 82 2023-10-30T12:45:42.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.295+0000 7f31ef7f5640 1 -- 172.21.15.89:0/575429270 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f31cc0664d0 msgr2=0x7f31cc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:42.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.295+0000 7f31ef7f5640 1 --2- 172.21.15.89:0/575429270 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f31cc0664d0 0x7f31cc068990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f31e8158550 tx=0x7f31d8008040 comp rx=0 tx=0).stop 2023-10-30T12:45:42.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.295+0000 7f31ef7f5640 1 -- 172.21.15.89:0/575429270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31e806f7a0 msgr2=0x7f31e814d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:42.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.295+0000 7f31ef7f5640 1 --2- 172.21.15.89:0/575429270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31e806f7a0 0x7f31e814d010 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f31e400e8a0 tx=0x7f31e400ed60 comp rx=0 tx=0).stop 2023-10-30T12:45:42.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.295+0000 7f31ef7f5640 1 -- 172.21.15.89:0/575429270 shutdown_connections 2023-10-30T12:45:42.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.295+0000 7f31ef7f5640 1 --2- 172.21.15.89:0/575429270 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f31cc0664d0 0x7f31cc068990 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:42.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.295+0000 7f31ef7f5640 1 --2- 172.21.15.89:0/575429270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31e8071110 0x7f31e80c1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:42.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.295+0000 7f31ef7f5640 1 --2- 172.21.15.89:0/575429270 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f31e80c6bd0 0x7f31e80aea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:42.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.295+0000 7f31ef7f5640 1 --2- 172.21.15.89:0/575429270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31e806f7a0 0x7f31e814d010 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:42.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.295+0000 7f31ef7f5640 1 -- 172.21.15.89:0/575429270 >> 172.21.15.89:0/575429270 conn(0x7f31e80c7910 msgr2=0x7f31e80c5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:45:42.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.295+0000 7f31ef7f5640 1 -- 172.21.15.89:0/575429270 shutdown_connections 2023-10-30T12:45:42.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:42.295+0000 7f31ef7f5640 1 -- 172.21.15.89:0/575429270 wait complete. 2023-10-30T12:45:42.310 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:45:44.312 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:45:44.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.459+0000 7fc6731c5640 1 Processor -- start 2023-10-30T12:45:44.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.459+0000 7fc6731c5640 1 -- start start 2023-10-30T12:45:44.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.463+0000 7fc6731c5640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc66c1580e0 0x7fc66c070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:44.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.463+0000 7fc6731c5640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc66c15e700 0x7fc66c06a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:44.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.463+0000 7fc6731c5640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66c0c5810 con 0x7fc66c0c6540 2023-10-30T12:45:44.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.463+0000 7fc6731c5640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc66c15eb90 con 0x7fc66c15e700 2023-10-30T12:45:44.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.463+0000 7fc6731c5640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc66c0c6960 con 0x7fc66c1580e0 2023-10-30T12:45:44.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.463+0000 7fc671762640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc66c15e700 0x7fc66c06a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:44.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.463+0000 7fc671762640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc66c15e700 0x7fc66c06a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52930/0 (socket says 172.21.15.89:52930) 2023-10-30T12:45:44.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.463+0000 7fc671762640 1 -- 172.21.15.89:0/2876403258 learned_addr learned my addr 172.21.15.89:0/2876403258 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:45:44.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.463+0000 7fc670f61640 1 --2- 172.21.15.89:0/2876403258 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc66c1580e0 0x7fc66c070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:44.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.463+0000 7fc671762640 1 -- 172.21.15.89:0/2876403258 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc66c1580e0 msgr2=0x7fc66c070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:45:44.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.463+0000 7fc671762640 1 --2- 172.21.15.89:0/2876403258 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc66c1580e0 0x7fc66c070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:44.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.463+0000 7fc671762640 1 -- 172.21.15.89:0/2876403258 >> v1:172.21.15.89:6789/0 conn(0x7fc66c0c6540 legacy=0x7fc66c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:44.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.463+0000 7fc671762640 1 -- 172.21.15.89:0/2876403258 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc66c0c6bb0 con 0x7fc66c15e700 2023-10-30T12:45:44.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.467+0000 7fc671762640 1 --2- 172.21.15.89:0/2876403258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc66c15e700 0x7fc66c06a5a0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fc664009e50 tx=0x7fc66401c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2fe5d1a472ef27d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:44.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.467+0000 7fc66b7fe640 1 -- 172.21.15.89:0/2876403258 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6640a58a0 con 0x7fc66c15e700 2023-10-30T12:45:44.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.467+0000 7fc66b7fe640 1 -- 172.21.15.89:0/2876403258 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6640a5a00 con 0x7fc66c15e700 2023-10-30T12:45:44.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.467+0000 7fc66b7fe640 1 -- 172.21.15.89:0/2876403258 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6640a5d90 con 0x7fc66c15e700 2023-10-30T12:45:44.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.467+0000 7fc6731c5640 1 -- 172.21.15.89:0/2876403258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc66c15e700 msgr2=0x7fc66c06a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:44.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.467+0000 7fc6731c5640 1 --2- 172.21.15.89:0/2876403258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc66c15e700 0x7fc66c06a5a0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fc664009e50 tx=0x7fc66401c9e0 comp rx=0 tx=0).stop 2023-10-30T12:45:44.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.467+0000 7fc6731c5640 1 -- 172.21.15.89:0/2876403258 shutdown_connections 2023-10-30T12:45:44.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.467+0000 7fc6731c5640 1 --2- 172.21.15.89:0/2876403258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc66c15e700 0x7fc66c06a5a0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:44.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.467+0000 7fc6731c5640 1 --2- 172.21.15.89:0/2876403258 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc66c1580e0 0x7fc66c070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:44.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.467+0000 7fc6731c5640 1 -- 172.21.15.89:0/2876403258 >> 172.21.15.89:0/2876403258 conn(0x7fc66c0c7910 msgr2=0x7fc66c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:45:44.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.467+0000 7fc6731c5640 1 -- 172.21.15.89:0/2876403258 shutdown_connections 2023-10-30T12:45:44.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.467+0000 7fc6731c5640 1 -- 172.21.15.89:0/2876403258 wait complete. 2023-10-30T12:45:44.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.467+0000 7fc6731c5640 1 Processor -- start 2023-10-30T12:45:44.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.467+0000 7fc6731c5640 1 -- start start 2023-10-30T12:45:44.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.467+0000 7fc6731c5640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc66c0c6540 0x7fc66c0c49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:44.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.467+0000 7fc6731c5640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc66c1580e0 0x7fc66c1491d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:44.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.467+0000 7fc6731c5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc66c15e700 0x7fc66c15f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:44.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.467+0000 7fc6731c5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc66c0c50a0 con 0x7fc66c15e700 2023-10-30T12:45:44.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.467+0000 7fc6731c5640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc66c15ef20 con 0x7fc66c0c6540 2023-10-30T12:45:44.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.467+0000 7fc6731c5640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc66c0b0cf0 con 0x7fc66c1580e0 2023-10-30T12:45:44.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.467+0000 7fc671762640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc66c15e700 0x7fc66c15f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:44.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.471+0000 7fc671762640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc66c15e700 0x7fc66c15f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52944/0 (socket says 172.21.15.89:52944) 2023-10-30T12:45:44.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.471+0000 7fc671762640 1 -- 172.21.15.89:0/4225990264 learned_addr learned my addr 172.21.15.89:0/4225990264 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:45:44.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.471+0000 7fc670f61640 1 --2- 172.21.15.89:0/4225990264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc66c0c6540 0x7fc66c0c49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:44.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.471+0000 7fc66bfff640 1 --2- 172.21.15.89:0/4225990264 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc66c1580e0 0x7fc66c1491d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:44.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.471+0000 7fc671762640 1 -- 172.21.15.89:0/4225990264 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc66c1580e0 msgr2=0x7fc66c1491d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:44.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.471+0000 7fc671762640 1 --2- 172.21.15.89:0/4225990264 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc66c1580e0 0x7fc66c1491d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:44.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.471+0000 7fc671762640 1 -- 172.21.15.89:0/4225990264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc66c0c6540 msgr2=0x7fc66c0c49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:44.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.471+0000 7fc671762640 1 --2- 172.21.15.89:0/4225990264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc66c0c6540 0x7fc66c0c49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:44.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.471+0000 7fc671762640 1 -- 172.21.15.89:0/4225990264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc664009b30 con 0x7fc66c15e700 2023-10-30T12:45:44.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.471+0000 7fc671762640 1 --2- 172.21.15.89:0/4225990264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc66c15e700 0x7fc66c15f340 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fc66401a610 tx=0x7fc664019780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:44.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.471+0000 7fc6697fa640 1 -- 172.21.15.89:0/4225990264 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6640b8640 con 0x7fc66c15e700 2023-10-30T12:45:44.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.471+0000 7fc6697fa640 1 -- 172.21.15.89:0/4225990264 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6640b87c0 con 0x7fc66c15e700 2023-10-30T12:45:44.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.471+0000 7fc6731c5640 1 -- 172.21.15.89:0/4225990264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc66c071020 con 0x7fc66c15e700 2023-10-30T12:45:44.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.471+0000 7fc6697fa640 1 -- 172.21.15.89:0/4225990264 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6640b8b70 con 0x7fc66c15e700 2023-10-30T12:45:44.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.471+0000 7fc6731c5640 1 -- 172.21.15.89:0/4225990264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc66c0715e0 con 0x7fc66c15e700 2023-10-30T12:45:44.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.471+0000 7fc6731c5640 1 -- 172.21.15.89:0/4225990264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc66c15f060 con 0x7fc66c15e700 2023-10-30T12:45:44.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.471+0000 7fc6697fa640 1 -- 172.21.15.89:0/4225990264 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc6640b8cd0 con 0x7fc66c15e700 2023-10-30T12:45:44.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.471+0000 7fc6697fa640 1 --2- 172.21.15.89:0/4225990264 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc6500664a0 0x7fc650068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:44.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.471+0000 7fc6697fa640 1 -- 172.21.15.89:0/4225990264 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6406+0+0 (secure 0 0 0) 0x7fc66411f6d0 con 0x7fc66c15e700 2023-10-30T12:45:44.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.471+0000 7fc670f61640 1 --2- 172.21.15.89:0/4225990264 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc6500664a0 0x7fc650068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:44.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.475+0000 7fc670f61640 1 --2- 172.21.15.89:0/4225990264 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc6500664a0 0x7fc650068960 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc660009af0 tx=0x7fc660008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:44.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.475+0000 7fc6697fa640 1 -- 172.21.15.89:0/4225990264 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc66c15f060 con 0x7fc66c15e700 2023-10-30T12:45:44.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.655+0000 7fc6731c5640 1 -- 172.21.15.89:0/4225990264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc66c15f060 con 0x7fc66c15e700 2023-10-30T12:45:44.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.655+0000 7fc6697fa640 1 -- 172.21.15.89:0/4225990264 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+6532 (secure 0 0 0) 0x7fc66c15f060 con 0x7fc66c15e700 2023-10-30T12:45:44.654 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:45:44.655 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5450,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:44:45.324321+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:45:44.656 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 82 2023-10-30T12:45:44.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.659+0000 7fc6731c5640 1 -- 172.21.15.89:0/4225990264 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc6500664a0 msgr2=0x7fc650068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:44.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.659+0000 7fc6731c5640 1 --2- 172.21.15.89:0/4225990264 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc6500664a0 0x7fc650068960 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc660009af0 tx=0x7fc660008040 comp rx=0 tx=0).stop 2023-10-30T12:45:44.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.659+0000 7fc6731c5640 1 -- 172.21.15.89:0/4225990264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc66c15e700 msgr2=0x7fc66c15f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:44.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.659+0000 7fc6731c5640 1 --2- 172.21.15.89:0/4225990264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc66c15e700 0x7fc66c15f340 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fc66401a610 tx=0x7fc664019780 comp rx=0 tx=0).stop 2023-10-30T12:45:44.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.659+0000 7fc6731c5640 1 -- 172.21.15.89:0/4225990264 shutdown_connections 2023-10-30T12:45:44.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.659+0000 7fc6731c5640 1 --2- 172.21.15.89:0/4225990264 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc6500664a0 0x7fc650068960 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:44.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.659+0000 7fc6731c5640 1 --2- 172.21.15.89:0/4225990264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc66c15e700 0x7fc66c15f340 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:44.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.659+0000 7fc6731c5640 1 --2- 172.21.15.89:0/4225990264 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc66c1580e0 0x7fc66c1491d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:44.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.659+0000 7fc6731c5640 1 --2- 172.21.15.89:0/4225990264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc66c0c6540 0x7fc66c0c49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:44.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.659+0000 7fc6731c5640 1 -- 172.21.15.89:0/4225990264 >> 172.21.15.89:0/4225990264 conn(0x7fc66c0c7910 msgr2=0x7fc66c1567f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:45:44.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.659+0000 7fc6731c5640 1 -- 172.21.15.89:0/4225990264 shutdown_connections 2023-10-30T12:45:44.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:44.659+0000 7fc6731c5640 1 -- 172.21.15.89:0/4225990264 wait complete. 2023-10-30T12:45:44.675 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:45:46.676 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:45:46.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.826+0000 7f576a3c5640 1 Processor -- start 2023-10-30T12:45:46.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.826+0000 7f576a3c5640 1 -- start start 2023-10-30T12:45:46.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.826+0000 7f576a3c5640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f57640bfec0 0x7f57640c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:46.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.826+0000 7f576a3c5640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5764071290 con 0x7f5764159e60 2023-10-30T12:45:46.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.830+0000 7f576a3c5640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5764071570 con 0x7f576406a720 2023-10-30T12:45:46.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.830+0000 7f576a3c5640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5764071850 con 0x7f57640bfec0 2023-10-30T12:45:46.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.830+0000 7f57637fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f57640bfec0 0x7f57640c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:46.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.830+0000 7f57637fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f57640bfec0 0x7f57640c1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52954/0 (socket says 172.21.15.89:52954) 2023-10-30T12:45:46.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.830+0000 7f57637fe640 1 -- 172.21.15.89:0/3845223982 learned_addr learned my addr 172.21.15.89:0/3845223982 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:45:46.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.830+0000 7f57637fe640 1 -- 172.21.15.89:0/3845223982 >> v1:172.21.15.192:6789/0 conn(0x7f576406a720 legacy=0x7f5764160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:46.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.830+0000 7f57637fe640 1 -- 172.21.15.89:0/3845223982 >> v1:172.21.15.89:6789/0 conn(0x7f5764159e60 legacy=0x7f576406f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:46.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.830+0000 7f57637fe640 1 -- 172.21.15.89:0/3845223982 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5764071b30 con 0x7f57640bfec0 2023-10-30T12:45:46.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.830+0000 7f57637fe640 1 --2- 172.21.15.89:0/3845223982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57640bfec0 0x7f57640c1a40 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f575800bb10 tx=0x7f5758021400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5102eda0ea736d3c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:46.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.830+0000 7f5762ffd640 1 -- 172.21.15.89:0/3845223982 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57580b1600 con 0x7f57640bfec0 2023-10-30T12:45:46.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.830+0000 7f5762ffd640 1 -- 172.21.15.89:0/3845223982 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57580ad050 con 0x7f57640bfec0 2023-10-30T12:45:46.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.830+0000 7f5762ffd640 1 -- 172.21.15.89:0/3845223982 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57580b1980 con 0x7f57640bfec0 2023-10-30T12:45:46.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.834+0000 7f576a3c5640 1 -- 172.21.15.89:0/3845223982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57640bfec0 msgr2=0x7f57640c1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:46.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.834+0000 7f576a3c5640 1 --2- 172.21.15.89:0/3845223982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57640bfec0 0x7f57640c1a40 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f575800bb10 tx=0x7f5758021400 comp rx=0 tx=0).stop 2023-10-30T12:45:46.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.834+0000 7f576a3c5640 1 -- 172.21.15.89:0/3845223982 shutdown_connections 2023-10-30T12:45:46.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.834+0000 7f576a3c5640 1 --2- 172.21.15.89:0/3845223982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57640bfec0 0x7f57640c1a40 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:46.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.834+0000 7f576a3c5640 1 -- 172.21.15.89:0/3845223982 >> 172.21.15.89:0/3845223982 conn(0x7f57640c7910 msgr2=0x7f57640c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:45:46.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.834+0000 7f576a3c5640 1 -- 172.21.15.89:0/3845223982 shutdown_connections 2023-10-30T12:45:46.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.834+0000 7f576a3c5640 1 -- 172.21.15.89:0/3845223982 wait complete. 2023-10-30T12:45:46.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.834+0000 7f576a3c5640 1 Processor -- start 2023-10-30T12:45:46.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.834+0000 7f576a3c5640 1 -- start start 2023-10-30T12:45:46.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.834+0000 7f576a3c5640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f576406a720 0x7f57640c1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:46.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.834+0000 7f576a3c5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57640bfec0 0x7f57640c2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:46.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.834+0000 7f576a3c5640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5764159e60 0x7f57640c3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:46.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.834+0000 7f576a3c5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57641eb880 con 0x7f57640bfec0 2023-10-30T12:45:46.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.834+0000 7f576a3c5640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57640c3530 con 0x7f576406a720 2023-10-30T12:45:46.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.834+0000 7f576a3c5640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57641ebb20 con 0x7f5764159e60 2023-10-30T12:45:46.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.834+0000 7f57637fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57640bfec0 0x7f57640c2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:46.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.834+0000 7f57637fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57640bfec0 0x7f57640c2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52956/0 (socket says 172.21.15.89:52956) 2023-10-30T12:45:46.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.834+0000 7f57637fe640 1 -- 172.21.15.89:0/3359443275 learned_addr learned my addr 172.21.15.89:0/3359443275 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:45:46.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.834+0000 7f5763fff640 1 --2- 172.21.15.89:0/3359443275 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f576406a720 0x7f57640c1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:46.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.834+0000 7f5768962640 1 --2- 172.21.15.89:0/3359443275 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5764159e60 0x7f57640c3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:46.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.834+0000 7f57637fe640 1 -- 172.21.15.89:0/3359443275 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5764159e60 msgr2=0x7f57640c3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:46.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.834+0000 7f57637fe640 1 --2- 172.21.15.89:0/3359443275 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5764159e60 0x7f57640c3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:46.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.834+0000 7f57637fe640 1 -- 172.21.15.89:0/3359443275 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f576406a720 msgr2=0x7f57640c1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:46.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.834+0000 7f57637fe640 1 --2- 172.21.15.89:0/3359443275 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f576406a720 0x7f57640c1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:46.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.838+0000 7f57637fe640 1 -- 172.21.15.89:0/3359443275 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575800b770 con 0x7f57640bfec0 2023-10-30T12:45:46.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.838+0000 7f57637fe640 1 --2- 172.21.15.89:0/3359443275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57640bfec0 0x7f57640c2330 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f575800b2c0 tx=0x7f575801e1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:46.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.838+0000 7f5760ff9640 1 -- 172.21.15.89:0/3359443275 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57580c2070 con 0x7f57640bfec0 2023-10-30T12:45:46.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.838+0000 7f5760ff9640 1 -- 172.21.15.89:0/3359443275 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57580ba610 con 0x7f57640bfec0 2023-10-30T12:45:46.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.838+0000 7f5760ff9640 1 -- 172.21.15.89:0/3359443275 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57580ba9e0 con 0x7f57640bfec0 2023-10-30T12:45:46.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.838+0000 7f576a3c5640 1 -- 172.21.15.89:0/3359443275 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57640c6bd0 con 0x7f57640bfec0 2023-10-30T12:45:46.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.838+0000 7f576a3c5640 1 -- 172.21.15.89:0/3359443275 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57640c2a90 con 0x7f57640bfec0 2023-10-30T12:45:46.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.838+0000 7f576a3c5640 1 -- 172.21.15.89:0/3359443275 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57640c6a30 con 0x7f57640bfec0 2023-10-30T12:45:46.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.838+0000 7f5760ff9640 1 -- 172.21.15.89:0/3359443275 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f57580b8040 con 0x7f57640bfec0 2023-10-30T12:45:46.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.838+0000 7f5760ff9640 1 --2- 172.21.15.89:0/3359443275 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f57500664d0 0x7f5750068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:46.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.838+0000 7f5760ff9640 1 -- 172.21.15.89:0/3359443275 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6406+0+0 (secure 0 0 0) 0x7f5758123f50 con 0x7f57640bfec0 2023-10-30T12:45:46.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.838+0000 7f5763fff640 1 --2- 172.21.15.89:0/3359443275 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f57500664d0 0x7f5750068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:46.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.842+0000 7f5763fff640 1 --2- 172.21.15.89:0/3359443275 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f57500664d0 0x7f5750068990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f5754000f70 tx=0x7f5754009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:46.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:46.842+0000 7f5760ff9640 1 -- 172.21.15.89:0/3359443275 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f57640c6a30 con 0x7f57640bfec0 2023-10-30T12:45:47.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:47.026+0000 7f576a3c5640 1 -- 172.21.15.89:0/3359443275 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57640c6a30 con 0x7f57640bfec0 2023-10-30T12:45:47.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:47.026+0000 7f5760ff9640 1 -- 172.21.15.89:0/3359443275 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f57640c6a30 con 0x7f57640bfec0 2023-10-30T12:45:47.025 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:45:47.025 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5450,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:44:45.324321+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:45:47.026 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 82 2023-10-30T12:45:47.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:47.030+0000 7f576a3c5640 1 -- 172.21.15.89:0/3359443275 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f57500664d0 msgr2=0x7f5750068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:47.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:47.030+0000 7f576a3c5640 1 --2- 172.21.15.89:0/3359443275 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f57500664d0 0x7f5750068990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f5754000f70 tx=0x7f5754009040 comp rx=0 tx=0).stop 2023-10-30T12:45:47.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:47.030+0000 7f576a3c5640 1 -- 172.21.15.89:0/3359443275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57640bfec0 msgr2=0x7f57640c2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:47.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:47.030+0000 7f576a3c5640 1 --2- 172.21.15.89:0/3359443275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57640bfec0 0x7f57640c2330 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f575800b2c0 tx=0x7f575801e1a0 comp rx=0 tx=0).stop 2023-10-30T12:45:47.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:47.030+0000 7f576a3c5640 1 -- 172.21.15.89:0/3359443275 shutdown_connections 2023-10-30T12:45:47.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:47.030+0000 7f576a3c5640 1 --2- 172.21.15.89:0/3359443275 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f57500664d0 0x7f5750068990 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:47.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:47.030+0000 7f576a3c5640 1 --2- 172.21.15.89:0/3359443275 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5764159e60 0x7f57640c3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:47.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:47.030+0000 7f576a3c5640 1 --2- 172.21.15.89:0/3359443275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57640bfec0 0x7f57640c2330 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:47.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:47.030+0000 7f576a3c5640 1 --2- 172.21.15.89:0/3359443275 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f576406a720 0x7f57640c1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:47.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:47.030+0000 7f576a3c5640 1 -- 172.21.15.89:0/3359443275 >> 172.21.15.89:0/3359443275 conn(0x7f57640c7910 msgr2=0x7f57640ca4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:45:47.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:47.030+0000 7f576a3c5640 1 -- 172.21.15.89:0/3359443275 shutdown_connections 2023-10-30T12:45:47.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:47.030+0000 7f576a3c5640 1 -- 172.21.15.89:0/3359443275 wait complete. 2023-10-30T12:45:47.046 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:45:47.734 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:45:47.739 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:45:49.047 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:45:49.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.198+0000 7f641d691640 1 Processor -- start 2023-10-30T12:45:49.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.198+0000 7f641d691640 1 -- start start 2023-10-30T12:45:49.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.198+0000 7f641d691640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f64181538a0 0x7f641815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:49.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.198+0000 7f641d691640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64180cb800 con 0x7f64180cf160 2023-10-30T12:45:49.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.198+0000 7f641d691640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f641815dc30 con 0x7f64180ca0e0 2023-10-30T12:45:49.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.198+0000 7f641d691640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64180cb330 con 0x7f64181538a0 2023-10-30T12:45:49.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.198+0000 7f64177fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f64180cf160 0x7f6418164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43364/0 (socket says 172.21.15.89:43364) 2023-10-30T12:45:49.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.198+0000 7f64177fe640 1 -- 172.21.15.89:0/4217461678 learned_addr learned my addr 172.21.15.89:0/4217461678 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:45:49.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.198+0000 7f6416ffd640 1 --2- 172.21.15.89:0/4217461678 >> v2:172.21.15.192:3300/0 conn(0x7f64181538a0 0x7f641815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:49.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.198+0000 7f64167fc640 1 -- 172.21.15.89:0/4217461678 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2258012153 0 0) 0x7f64180cb800 con 0x7f64180cf160 2023-10-30T12:45:49.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.198+0000 7f64167fc640 1 -- 172.21.15.89:0/4217461678 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6404003530 con 0x7f64180cf160 2023-10-30T12:45:49.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.198+0000 7f64167fc640 1 -- 172.21.15.89:0/4217461678 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3826583133 0 0) 0x7f6404003530 con 0x7f64180cf160 2023-10-30T12:45:49.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.198+0000 7f64167fc640 1 -- 172.21.15.89:0/4217461678 >> v2:172.21.15.192:3300/0 conn(0x7f64181538a0 msgr2=0x7f641815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:45:49.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.198+0000 7f64167fc640 1 --2- 172.21.15.89:0/4217461678 >> v2:172.21.15.192:3300/0 conn(0x7f64181538a0 0x7f641815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:49.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.198+0000 7f64167fc640 1 -- 172.21.15.89:0/4217461678 >> v1:172.21.15.192:6789/0 conn(0x7f64180ca0e0 legacy=0x7f641815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:49.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.198+0000 7f64167fc640 1 -- 172.21.15.89:0/4217461678 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64180cad70 con 0x7f64180cf160 2023-10-30T12:45:49.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.198+0000 7f64167fc640 1 -- 172.21.15.89:0/4217461678 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f6410001a00 con 0x7f64180cf160 2023-10-30T12:45:49.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.198+0000 7f64167fc640 1 -- 172.21.15.89:0/4217461678 >> v1:172.21.15.89:6789/0 conn(0x7f64180cf160 legacy=0x7f6418164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:49.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.198+0000 7f64167fc640 1 --2- 172.21.15.89:0/4217461678 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6404003980 0x7f6404003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:49.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.198+0000 7f64167fc640 1 --2- 172.21.15.89:0/4217461678 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64040047d0 0x7f6404004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:49.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.198+0000 7f64167fc640 1 --2- 172.21.15.89:0/4217461678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6404005190 0x7f640400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:49.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.198+0000 7f64167fc640 1 -- 172.21.15.89:0/4217461678 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64180cb330 con 0x7f6404003980 2023-10-30T12:45:49.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.198+0000 7f64167fc640 1 -- 172.21.15.89:0/4217461678 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640400c040 con 0x7f6404005190 2023-10-30T12:45:49.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.198+0000 7f64167fc640 1 -- 172.21.15.89:0/4217461678 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f640400c200 con 0x7f64040047d0 2023-10-30T12:45:49.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.202+0000 7f6416ffd640 1 --2- 172.21.15.89:0/4217461678 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6404003980 0x7f6404003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:49.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.202+0000 7f64177fe640 1 --2- 172.21.15.89:0/4217461678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6404005190 0x7f640400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:49.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.202+0000 7f640e7fc640 1 --2- 172.21.15.89:0/4217461678 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64040047d0 0x7f6404004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:49.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.202+0000 7f6416ffd640 1 -- 172.21.15.89:0/4217461678 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64040047d0 msgr2=0x7f6404004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:45:49.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.202+0000 7f6416ffd640 1 --2- 172.21.15.89:0/4217461678 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64040047d0 0x7f6404004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:49.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.202+0000 7f6416ffd640 1 -- 172.21.15.89:0/4217461678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6404005190 msgr2=0x7f640400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:45:49.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.202+0000 7f6416ffd640 1 --2- 172.21.15.89:0/4217461678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6404005190 0x7f640400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:49.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.202+0000 7f64177fe640 1 -- 172.21.15.89:0/4217461678 reap_dead start 2023-10-30T12:45:49.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.202+0000 7f6416ffd640 1 -- 172.21.15.89:0/4217461678 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6404003530 con 0x7f6404003980 2023-10-30T12:45:49.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.202+0000 7f6416ffd640 1 --2- 172.21.15.89:0/4217461678 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6404003980 0x7f6404003da0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f6400009ae0 tx=0x7f640001ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1152614d299adaad server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:49.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.202+0000 7f64167fc640 1 -- 172.21.15.89:0/4217461678 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64000a55c0 con 0x7f6404003980 2023-10-30T12:45:49.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.202+0000 7f64167fc640 1 -- 172.21.15.89:0/4217461678 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64000a5720 con 0x7f6404003980 2023-10-30T12:45:49.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.202+0000 7f64167fc640 1 -- 172.21.15.89:0/4217461678 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64000a5ad0 con 0x7f6404003980 2023-10-30T12:45:49.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.202+0000 7f641d691640 1 -- 172.21.15.89:0/4217461678 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6404003980 msgr2=0x7f6404003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:49.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.202+0000 7f641d691640 1 --2- 172.21.15.89:0/4217461678 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6404003980 0x7f6404003da0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f6400009ae0 tx=0x7f640001ca10 comp rx=0 tx=0).stop 2023-10-30T12:45:49.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.206+0000 7f641d691640 1 -- 172.21.15.89:0/4217461678 shutdown_connections 2023-10-30T12:45:49.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.206+0000 7f641d691640 1 --2- 172.21.15.89:0/4217461678 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6404003980 0x7f6404003da0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:49.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.206+0000 7f641d691640 1 -- 172.21.15.89:0/4217461678 >> 172.21.15.89:0/4217461678 conn(0x7f641806bfe0 msgr2=0x7f641806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:45:49.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.206+0000 7f641d691640 1 -- 172.21.15.89:0/4217461678 shutdown_connections 2023-10-30T12:45:49.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.206+0000 7f641d691640 1 -- 172.21.15.89:0/4217461678 wait complete. 2023-10-30T12:45:49.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.206+0000 7f641d691640 1 Processor -- start 2023-10-30T12:45:49.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.206+0000 7f641d691640 1 -- start start 2023-10-30T12:45:49.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.206+0000 7f641d691640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6404003980 0x7f6418158810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:49.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.206+0000 7f641d691640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6418148a60 0x7f64181e4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:49.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.206+0000 7f641d691640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6418148eb0 0x7f64181e5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:49.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.206+0000 7f641d691640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64181e3660 con 0x7f6404003980 2023-10-30T12:45:49.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.206+0000 7f641d691640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64181e4840 con 0x7f6418148eb0 2023-10-30T12:45:49.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.206+0000 7f641d691640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64181e31d0 con 0x7f6418148a60 2023-10-30T12:45:49.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.206+0000 7f6416ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6404003980 0x7f6418158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:49.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.206+0000 7f6416ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6404003980 0x7f6418158810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52978/0 (socket says 172.21.15.89:52978) 2023-10-30T12:45:49.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.206+0000 7f6416ffd640 1 -- 172.21.15.89:0/2111259065 learned_addr learned my addr 172.21.15.89:0/2111259065 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:45:49.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.206+0000 7f640e7fc640 1 --2- 172.21.15.89:0/2111259065 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6418148a60 0x7f64181e4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:49.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.206+0000 7f64177fe640 1 --2- 172.21.15.89:0/2111259065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6418148eb0 0x7f64181e5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:49.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.206+0000 7f6416ffd640 1 -- 172.21.15.89:0/2111259065 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6418148a60 msgr2=0x7f64181e4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:49.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.206+0000 7f6416ffd640 1 --2- 172.21.15.89:0/2111259065 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6418148a60 0x7f64181e4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:49.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.206+0000 7f6416ffd640 1 -- 172.21.15.89:0/2111259065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6418148eb0 msgr2=0x7f64181e5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:49.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.206+0000 7f6416ffd640 1 --2- 172.21.15.89:0/2111259065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6418148eb0 0x7f64181e5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:49.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.206+0000 7f6416ffd640 1 -- 172.21.15.89:0/2111259065 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6418150e50 con 0x7f6404003980 2023-10-30T12:45:49.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.206+0000 7f640e7fc640 1 --2- 172.21.15.89:0/2111259065 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6418148a60 0x7f64181e4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:45:49.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.206+0000 7f6416ffd640 1 --2- 172.21.15.89:0/2111259065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6404003980 0x7f6418158810 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f640001c840 tx=0x7f64000197b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:49.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.210+0000 7f640ffff640 1 -- 172.21.15.89:0/2111259065 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64000bd070 con 0x7f6404003980 2023-10-30T12:45:49.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.210+0000 7f640ffff640 1 -- 172.21.15.89:0/2111259065 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6400004020 con 0x7f6404003980 2023-10-30T12:45:49.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.210+0000 7f640ffff640 1 -- 172.21.15.89:0/2111259065 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64000043d0 con 0x7f6404003980 2023-10-30T12:45:49.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.210+0000 7f641d691640 1 -- 172.21.15.89:0/2111259065 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64181497d0 con 0x7f6404003980 2023-10-30T12:45:49.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.210+0000 7f641d691640 1 -- 172.21.15.89:0/2111259065 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64181e2b80 con 0x7f6404003980 2023-10-30T12:45:49.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.210+0000 7f641d691640 1 -- 172.21.15.89:0/2111259065 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f641804e170 con 0x7f6404003980 2023-10-30T12:45:49.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.210+0000 7f640ffff640 1 -- 172.21.15.89:0/2111259065 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f64000b4070 con 0x7f6404003980 2023-10-30T12:45:49.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.210+0000 7f640ffff640 1 --2- 172.21.15.89:0/2111259065 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f63f80664d0 0x7f63f8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:49.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.210+0000 7f640ffff640 1 -- 172.21.15.89:0/2111259065 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6406+0+0 (secure 0 0 0) 0x7f640011e5a0 con 0x7f6404003980 2023-10-30T12:45:49.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.210+0000 7f640e7fc640 1 --2- 172.21.15.89:0/2111259065 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f63f80664d0 0x7f63f8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:49.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.214+0000 7f640e7fc640 1 --2- 172.21.15.89:0/2111259065 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f63f80664d0 0x7f63f8068990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f64180c58a0 tx=0x7f6408009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:49.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.214+0000 7f640ffff640 1 -- 172.21.15.89:0/2111259065 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f64000eb6c0 con 0x7f6404003980 2023-10-30T12:45:49.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.398+0000 7f641d691640 1 -- 172.21.15.89:0/2111259065 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6418149d40 con 0x7f6404003980 2023-10-30T12:45:49.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.398+0000 7f640ffff640 1 -- 172.21.15.89:0/2111259065 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f64000b2090 con 0x7f6404003980 2023-10-30T12:45:49.398 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:45:49.398 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5450,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:44:45.324321+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5139,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5139":{"gid":5139,"name":"c","rank":1,"incarnation":33,"state":"up:active","state_seq":29,"addr":"172.21.15.89:6837/951276252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":951276252},{"type":"v1","addr":"172.21.15.89:6837","nonce":951276252}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:45:49.398 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 82 2023-10-30T12:45:49.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.402+0000 7f641d691640 1 -- 172.21.15.89:0/2111259065 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f63f80664d0 msgr2=0x7f63f8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:49.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.402+0000 7f641d691640 1 --2- 172.21.15.89:0/2111259065 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f63f80664d0 0x7f63f8068990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f64180c58a0 tx=0x7f6408009040 comp rx=0 tx=0).stop 2023-10-30T12:45:49.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.402+0000 7f641d691640 1 -- 172.21.15.89:0/2111259065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6404003980 msgr2=0x7f6418158810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:49.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.402+0000 7f641d691640 1 --2- 172.21.15.89:0/2111259065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6404003980 0x7f6418158810 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f640001c840 tx=0x7f64000197b0 comp rx=0 tx=0).stop 2023-10-30T12:45:49.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.402+0000 7f641d691640 1 -- 172.21.15.89:0/2111259065 shutdown_connections 2023-10-30T12:45:49.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.402+0000 7f641d691640 1 --2- 172.21.15.89:0/2111259065 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f63f80664d0 0x7f63f8068990 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:49.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.402+0000 7f641d691640 1 --2- 172.21.15.89:0/2111259065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6418148eb0 0x7f64181e5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:49.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.402+0000 7f641d691640 1 --2- 172.21.15.89:0/2111259065 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6418148a60 0x7f64181e4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:49.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.402+0000 7f641d691640 1 --2- 172.21.15.89:0/2111259065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6404003980 0x7f6418158810 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:49.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.402+0000 7f641d691640 1 -- 172.21.15.89:0/2111259065 >> 172.21.15.89:0/2111259065 conn(0x7f641806bfe0 msgr2=0x7f6418145e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:45:49.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.402+0000 7f641d691640 1 -- 172.21.15.89:0/2111259065 shutdown_connections 2023-10-30T12:45:49.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:49.402+0000 7f641d691640 1 -- 172.21.15.89:0/2111259065 wait complete. 2023-10-30T12:45:49.419 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:45:51.420 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:45:51.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.566+0000 7f6c21c91640 1 Processor -- start 2023-10-30T12:45:51.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.566+0000 7f6c21c91640 1 -- start start 2023-10-30T12:45:51.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.570+0000 7f6c21c91640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6c1c153bf0 0x7f6c1c071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:51.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.570+0000 7f6c21c91640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6c1c0c6fe0 0x7f6c1c0c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:51.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.570+0000 7f6c21c91640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c1c0700b0 con 0x7f6c1c0c79e0 2023-10-30T12:45:51.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.570+0000 7f6c21c91640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c1c0c6810 con 0x7f6c1c0c6fe0 2023-10-30T12:45:51.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.570+0000 7f6c21c91640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c1c159900 con 0x7f6c1c153bf0 2023-10-30T12:45:51.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.570+0000 7f6c1affd640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6c1c0c6fe0 0x7f6c1c0c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:51.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.570+0000 7f6c1affd640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6c1c0c6fe0 0x7f6c1c0c74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44922/0 (socket says 172.21.15.89:44922) 2023-10-30T12:45:51.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.570+0000 7f6c1affd640 1 -- 172.21.15.89:0/1220055129 learned_addr learned my addr 172.21.15.89:0/1220055129 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:45:51.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.570+0000 7f6c1b7fe640 1 --2- 172.21.15.89:0/1220055129 >> v2:172.21.15.192:3300/0 conn(0x7f6c1c153bf0 0x7f6c1c071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:51.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.570+0000 7f6c1affd640 1 -- 172.21.15.89:0/1220055129 >> v2:172.21.15.192:3300/0 conn(0x7f6c1c153bf0 msgr2=0x7f6c1c071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:45:51.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.570+0000 7f6c1affd640 1 --2- 172.21.15.89:0/1220055129 >> v2:172.21.15.192:3300/0 conn(0x7f6c1c153bf0 0x7f6c1c071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:51.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.570+0000 7f6c1affd640 1 -- 172.21.15.89:0/1220055129 >> v1:172.21.15.192:6789/0 conn(0x7f6c1c0c79e0 legacy=0x7f6c1c0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:45:51.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.570+0000 7f6c1affd640 1 -- 172.21.15.89:0/1220055129 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c1c0c6530 con 0x7f6c1c0c6fe0 2023-10-30T12:45:51.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.570+0000 7f6c1affd640 1 --2- 172.21.15.89:0/1220055129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c1c0c6fe0 0x7f6c1c0c74a0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c0c9150 tx=0x7f6c1001ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e3efc3639458e67 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:51.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.574+0000 7f6c1a7fc640 1 -- 172.21.15.89:0/1220055129 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c100a89b0 con 0x7f6c1c0c6fe0 2023-10-30T12:45:51.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.574+0000 7f6c1a7fc640 1 -- 172.21.15.89:0/1220055129 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c100a8b10 con 0x7f6c1c0c6fe0 2023-10-30T12:45:51.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.574+0000 7f6c1a7fc640 1 -- 172.21.15.89:0/1220055129 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c100a8ea0 con 0x7f6c1c0c6fe0 2023-10-30T12:45:51.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.574+0000 7f6c21c91640 1 -- 172.21.15.89:0/1220055129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c1c0c6fe0 msgr2=0x7f6c1c0c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:51.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.574+0000 7f6c21c91640 1 --2- 172.21.15.89:0/1220055129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c1c0c6fe0 0x7f6c1c0c74a0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c0c9150 tx=0x7f6c1001ecb0 comp rx=0 tx=0).stop 2023-10-30T12:45:51.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.574+0000 7f6c21c91640 1 -- 172.21.15.89:0/1220055129 shutdown_connections 2023-10-30T12:45:51.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.574+0000 7f6c21c91640 1 --2- 172.21.15.89:0/1220055129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c1c0c6fe0 0x7f6c1c0c74a0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:51.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.574+0000 7f6c21c91640 1 --2- 172.21.15.89:0/1220055129 >> v2:172.21.15.192:3300/0 conn(0x7f6c1c153bf0 0x7f6c1c071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:51.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.574+0000 7f6c21c91640 1 -- 172.21.15.89:0/1220055129 >> 172.21.15.89:0/1220055129 conn(0x7f6c1c06c6e0 msgr2=0x7f6c1c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:45:51.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.574+0000 7f6c21c91640 1 -- 172.21.15.89:0/1220055129 shutdown_connections 2023-10-30T12:45:51.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.574+0000 7f6c21c91640 1 -- 172.21.15.89:0/1220055129 wait complete. 2023-10-30T12:45:51.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.574+0000 7f6c21c91640 1 Processor -- start 2023-10-30T12:45:51.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.574+0000 7f6c21c91640 1 -- start start 2023-10-30T12:45:51.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.574+0000 7f6c21c91640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c1c0c6fe0 0x7f6c1c14c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:51.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.574+0000 7f6c21c91640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6c1c0c79e0 0x7f6c1c14e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:51.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.574+0000 7f6c21c91640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c1c153bf0 0x7f6c1c1498f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:51.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.574+0000 7f6c21c91640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c1eec80 con 0x7f6c1c0c6fe0 2023-10-30T12:45:51.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.574+0000 7f6c21c91640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c14ff30 con 0x7f6c1c153bf0 2023-10-30T12:45:51.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.574+0000 7f6c21c91640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c14dc70 con 0x7f6c1c0c79e0 2023-10-30T12:45:51.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.574+0000 7f6c1b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c1c0c6fe0 0x7f6c1c14c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:51.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.574+0000 7f6c1b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c1c0c6fe0 0x7f6c1c14c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44924/0 (socket says 172.21.15.89:44924) 2023-10-30T12:45:51.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.574+0000 7f6c1b7fe640 1 -- 172.21.15.89:0/141135243 learned_addr learned my addr 172.21.15.89:0/141135243 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:45:51.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.578+0000 7f6c1affd640 1 --2- 172.21.15.89:0/141135243 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6c1c0c79e0 0x7f6c1c14e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:51.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.578+0000 7f6c1bfff640 1 --2- 172.21.15.89:0/141135243 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c1c153bf0 0x7f6c1c1498f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:51.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.578+0000 7f6c1b7fe640 1 -- 172.21.15.89:0/141135243 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6c1c0c79e0 msgr2=0x7f6c1c14e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:51.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.578+0000 7f6c1b7fe640 1 --2- 172.21.15.89:0/141135243 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6c1c0c79e0 0x7f6c1c14e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:51.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.578+0000 7f6c1b7fe640 1 -- 172.21.15.89:0/141135243 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c1c153bf0 msgr2=0x7f6c1c1498f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:51.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.578+0000 7f6c1b7fe640 1 --2- 172.21.15.89:0/141135243 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c1c153bf0 0x7f6c1c1498f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:51.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.578+0000 7f6c1b7fe640 1 -- 172.21.15.89:0/141135243 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c100094d0 con 0x7f6c1c0c6fe0 2023-10-30T12:45:51.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.578+0000 7f6c1b7fe640 1 --2- 172.21.15.89:0/141135243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c1c0c6fe0 0x7f6c1c14c000 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c00d620 tx=0x7f6c0c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:51.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.578+0000 7f6c03fff640 1 -- 172.21.15.89:0/141135243 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c00b620 con 0x7f6c1c0c6fe0 2023-10-30T12:45:51.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.578+0000 7f6c03fff640 1 -- 172.21.15.89:0/141135243 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c0c00b780 con 0x7f6c1c0c6fe0 2023-10-30T12:45:51.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.578+0000 7f6c03fff640 1 -- 172.21.15.89:0/141135243 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c00bb10 con 0x7f6c1c0c6fe0 2023-10-30T12:45:51.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.578+0000 7f6c21c91640 1 -- 172.21.15.89:0/141135243 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c1c155010 con 0x7f6c1c0c6fe0 2023-10-30T12:45:51.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.578+0000 7f6c21c91640 1 -- 172.21.15.89:0/141135243 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c1c14dec0 con 0x7f6c1c0c6fe0 2023-10-30T12:45:51.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.578+0000 7f6c21c91640 1 -- 172.21.15.89:0/141135243 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c1c04e170 con 0x7f6c1c0c6fe0 2023-10-30T12:45:51.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.578+0000 7f6c03fff640 1 -- 172.21.15.89:0/141135243 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6c0c00bc70 con 0x7f6c1c0c6fe0 2023-10-30T12:45:51.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.578+0000 7f6c03fff640 1 --2- 172.21.15.89:0/141135243 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6c080664a0 0x7f6c08068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:51.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.578+0000 7f6c03fff640 1 -- 172.21.15.89:0/141135243 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6492+0+0 (secure 0 0 0) 0x7f6c0c084670 con 0x7f6c1c0c6fe0 2023-10-30T12:45:51.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.582+0000 7f6c1affd640 1 --2- 172.21.15.89:0/141135243 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6c080664a0 0x7f6c08068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:51.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.582+0000 7f6c1affd640 1 --2- 172.21.15.89:0/141135243 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6c080664a0 0x7f6c08068960 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6c100b2000 tx=0x7f6c10009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:51.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.582+0000 7f6c03fff640 1 -- 172.21.15.89:0/141135243 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6c0c051740 con 0x7f6c1c0c6fe0 2023-10-30T12:45:51.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.762+0000 7f6c21c91640 1 -- 172.21.15.89:0/141135243 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c1c06fe90 con 0x7f6c1c0c6fe0 2023-10-30T12:45:51.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.762+0000 7f6c03fff640 1 -- 172.21.15.89:0/141135243 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+5740 (secure 0 0 0) 0x7f6c0c054d60 con 0x7f6c1c0c6fe0 2023-10-30T12:45:51.762 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:45:51.762 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:45:51.226565+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5450,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":1,"incarnation":83,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:45:51.762 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 83 2023-10-30T12:45:51.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.766+0000 7f6c21c91640 1 -- 172.21.15.89:0/141135243 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6c080664a0 msgr2=0x7f6c08068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:51.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.766+0000 7f6c21c91640 1 --2- 172.21.15.89:0/141135243 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6c080664a0 0x7f6c08068960 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6c100b2000 tx=0x7f6c10009040 comp rx=0 tx=0).stop 2023-10-30T12:45:51.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.766+0000 7f6c21c91640 1 -- 172.21.15.89:0/141135243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c1c0c6fe0 msgr2=0x7f6c1c14c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:51.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.766+0000 7f6c21c91640 1 --2- 172.21.15.89:0/141135243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c1c0c6fe0 0x7f6c1c14c000 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c00d620 tx=0x7f6c0c00dae0 comp rx=0 tx=0).stop 2023-10-30T12:45:51.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.766+0000 7f6c21c91640 1 -- 172.21.15.89:0/141135243 shutdown_connections 2023-10-30T12:45:51.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.766+0000 7f6c21c91640 1 --2- 172.21.15.89:0/141135243 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6c080664a0 0x7f6c08068960 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:51.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.766+0000 7f6c21c91640 1 --2- 172.21.15.89:0/141135243 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c1c153bf0 0x7f6c1c1498f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:51.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.766+0000 7f6c21c91640 1 --2- 172.21.15.89:0/141135243 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6c1c0c79e0 0x7f6c1c14e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:51.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.766+0000 7f6c21c91640 1 --2- 172.21.15.89:0/141135243 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c1c0c6fe0 0x7f6c1c14c000 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:51.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.766+0000 7f6c21c91640 1 -- 172.21.15.89:0/141135243 >> 172.21.15.89:0/141135243 conn(0x7f6c1c06c6e0 msgr2=0x7f6c1c158e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:45:51.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.766+0000 7f6c21c91640 1 -- 172.21.15.89:0/141135243 shutdown_connections 2023-10-30T12:45:51.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.766+0000 7f6c21c91640 1 -- 172.21.15.89:0/141135243 wait complete. 2023-10-30T12:45:51.783 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T12:45:51.783 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:45:51.784 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:45:51.921 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.922+0000 7f3066c46640 1 Processor -- start 2023-10-30T12:45:51.922 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.922+0000 7f3066c46640 1 -- start start 2023-10-30T12:45:51.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.926+0000 7f3066c46640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f30600c6fe0 0x7f3060071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:51.923 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.926+0000 7f3066c46640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f30600c5f60 0x7f30600cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:51.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.926+0000 7f3066c46640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30600700b0 con 0x7f3060153bf0 2023-10-30T12:45:51.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.926+0000 7f3066c46640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30600c6810 con 0x7f30600c5f60 2023-10-30T12:45:51.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.926+0000 7f3066c46640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3060159900 con 0x7f30600c6fe0 2023-10-30T12:45:51.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.926+0000 7f30651e3640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f30600c5f60 0x7f30600cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:51.924 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.926+0000 7f30651e3640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f30600c5f60 0x7f30600cb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44930/0 (socket says 172.21.15.89:44930) 2023-10-30T12:45:51.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.926+0000 7f30651e3640 1 -- 172.21.15.89:0/350918563 learned_addr learned my addr 172.21.15.89:0/350918563 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:45:51.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.926+0000 7f3057fff640 1 --2- 172.21.15.89:0/350918563 >> v2:172.21.15.192:3300/0 conn(0x7f30600c6fe0 0x7f3060071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:51.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.926+0000 7f30651e3640 1 -- 172.21.15.89:0/350918563 >> v2:172.21.15.192:3300/0 conn(0x7f30600c6fe0 msgr2=0x7f3060071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:45:51.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.926+0000 7f30651e3640 1 --2- 172.21.15.89:0/350918563 >> v2:172.21.15.192:3300/0 conn(0x7f30600c6fe0 0x7f3060071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:51.925 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.926+0000 7f30651e3640 1 -- 172.21.15.89:0/350918563 >> v1:172.21.15.192:6789/0 conn(0x7f3060153bf0 legacy=0x7f30600c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:51.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.926+0000 7f30651e3640 1 -- 172.21.15.89:0/350918563 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30600c6530 con 0x7f30600c5f60 2023-10-30T12:45:51.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.930+0000 7f30651e3640 1 --2- 172.21.15.89:0/350918563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30600c5f60 0x7f30600cb570 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f30600723d0 tx=0x7f305801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfc88e2590b95ae6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:51.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.930+0000 7f30577fe640 1 -- 172.21.15.89:0/350918563 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30580af400 con 0x7f30600c5f60 2023-10-30T12:45:51.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.930+0000 7f30577fe640 1 -- 172.21.15.89:0/350918563 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30580ab050 con 0x7f30600c5f60 2023-10-30T12:45:51.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.930+0000 7f30577fe640 1 -- 172.21.15.89:0/350918563 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30580af780 con 0x7f30600c5f60 2023-10-30T12:45:51.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.930+0000 7f3066c46640 1 -- 172.21.15.89:0/350918563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30600c5f60 msgr2=0x7f30600cb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:51.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.930+0000 7f3066c46640 1 --2- 172.21.15.89:0/350918563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30600c5f60 0x7f30600cb570 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f30600723d0 tx=0x7f305801f1c0 comp rx=0 tx=0).stop 2023-10-30T12:45:51.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.930+0000 7f3066c46640 1 -- 172.21.15.89:0/350918563 shutdown_connections 2023-10-30T12:45:51.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.930+0000 7f3066c46640 1 --2- 172.21.15.89:0/350918563 >> v2:172.21.15.192:3300/0 conn(0x7f30600c6fe0 0x7f3060071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:51.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.930+0000 7f3066c46640 1 --2- 172.21.15.89:0/350918563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30600c5f60 0x7f30600cb570 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:51.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.930+0000 7f3066c46640 1 -- 172.21.15.89:0/350918563 >> 172.21.15.89:0/350918563 conn(0x7f306006c6e0 msgr2=0x7f3060069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:45:51.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.930+0000 7f3066c46640 1 -- 172.21.15.89:0/350918563 shutdown_connections 2023-10-30T12:45:51.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.930+0000 7f3066c46640 1 -- 172.21.15.89:0/350918563 wait complete. 2023-10-30T12:45:51.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.930+0000 7f3066c46640 1 Processor -- start 2023-10-30T12:45:51.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.930+0000 7f3066c46640 1 -- start start 2023-10-30T12:45:51.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.930+0000 7f3066c46640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30600c5f60 0x7f30600c5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:51.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.930+0000 7f3066c46640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f30600c6fe0 0x7f3060155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:51.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.934+0000 7f3066c46640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3060153bf0 0x7f30601f2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:51.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.934+0000 7f3066c46640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30600c46b0 con 0x7f30600c5f60 2023-10-30T12:45:51.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.934+0000 7f3066c46640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306014f9a0 con 0x7f3060153bf0 2023-10-30T12:45:51.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.934+0000 7f3066c46640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f306014e8b0 con 0x7f30600c6fe0 2023-10-30T12:45:51.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.934+0000 7f30649e2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30600c5f60 0x7f30600c5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:51.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.934+0000 7f30649e2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30600c5f60 0x7f30600c5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44938/0 (socket says 172.21.15.89:44938) 2023-10-30T12:45:51.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.934+0000 7f30649e2640 1 -- 172.21.15.89:0/652362261 learned_addr learned my addr 172.21.15.89:0/652362261 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:45:51.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.934+0000 7f30651e3640 1 --2- 172.21.15.89:0/652362261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3060153bf0 0x7f30601f2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:51.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.934+0000 7f3057fff640 1 --2- 172.21.15.89:0/652362261 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f30600c6fe0 0x7f3060155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:51.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.934+0000 7f30649e2640 1 -- 172.21.15.89:0/652362261 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f30600c6fe0 msgr2=0x7f3060155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:51.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.934+0000 7f30649e2640 1 --2- 172.21.15.89:0/652362261 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f30600c6fe0 0x7f3060155ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:51.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.934+0000 7f30649e2640 1 -- 172.21.15.89:0/652362261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3060153bf0 msgr2=0x7f30601f2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:51.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.934+0000 7f30649e2640 1 --2- 172.21.15.89:0/652362261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3060153bf0 0x7f30601f2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:51.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.934+0000 7f30649e2640 1 -- 172.21.15.89:0/652362261 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3058009b30 con 0x7f30600c5f60 2023-10-30T12:45:51.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.934+0000 7f30649e2640 1 --2- 172.21.15.89:0/652362261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30600c5f60 0x7f30600c5650 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f305c00c3c0 tx=0x7f305c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:51.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.934+0000 7f30557fa640 1 -- 172.21.15.89:0/652362261 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305c0052d0 con 0x7f30600c5f60 2023-10-30T12:45:51.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.934+0000 7f30557fa640 1 -- 172.21.15.89:0/652362261 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f305c005430 con 0x7f30600c5f60 2023-10-30T12:45:51.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.934+0000 7f3066c46640 1 -- 172.21.15.89:0/652362261 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30600c51c0 con 0x7f30600c5f60 2023-10-30T12:45:51.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.934+0000 7f30557fa640 1 -- 172.21.15.89:0/652362261 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305c003840 con 0x7f30600c5f60 2023-10-30T12:45:51.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.934+0000 7f3066c46640 1 -- 172.21.15.89:0/652362261 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30600c4890 con 0x7f30600c5f60 2023-10-30T12:45:51.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.934+0000 7f3066c46640 1 -- 172.21.15.89:0/652362261 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f306004e170 con 0x7f30600c5f60 2023-10-30T12:45:51.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.934+0000 7f30557fa640 1 -- 172.21.15.89:0/652362261 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f305c0039a0 con 0x7f30600c5f60 2023-10-30T12:45:51.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.938+0000 7f30557fa640 1 --2- 172.21.15.89:0/652362261 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f304c0664a0 0x7f304c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:51.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.938+0000 7f30557fa640 1 -- 172.21.15.89:0/652362261 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6492+0+0 (secure 0 0 0) 0x7f305c083680 con 0x7f30600c5f60 2023-10-30T12:45:51.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.938+0000 7f3057fff640 1 --2- 172.21.15.89:0/652362261 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f304c0664a0 0x7f304c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:51.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.938+0000 7f3057fff640 1 --2- 172.21.15.89:0/652362261 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f304c0664a0 0x7f304c068960 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f306014d950 tx=0x7f3050008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:51.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:51.942+0000 7f30557fa640 1 -- 172.21.15.89:0/652362261 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f305c050880 con 0x7f30600c5f60 2023-10-30T12:45:52.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:52.118+0000 7f3066c46640 1 -- 172.21.15.89:0/652362261 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f306006fe90 con 0x7f30600c5f60 2023-10-30T12:45:52.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:52.118+0000 7f30557fa640 1 -- 172.21.15.89:0/652362261 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+5740 (secure 0 0 0) 0x7f305c053ea0 con 0x7f30600c5f60 2023-10-30T12:45:52.118 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:45:52.118 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:45:51.226565+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5450,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":1,"incarnation":83,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:45:52.119 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 83 2023-10-30T12:45:52.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:52.122+0000 7f3066c46640 1 -- 172.21.15.89:0/652362261 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f304c0664a0 msgr2=0x7f304c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:52.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:52.122+0000 7f3066c46640 1 --2- 172.21.15.89:0/652362261 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f304c0664a0 0x7f304c068960 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f306014d950 tx=0x7f3050008040 comp rx=0 tx=0).stop 2023-10-30T12:45:52.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:52.122+0000 7f3066c46640 1 -- 172.21.15.89:0/652362261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30600c5f60 msgr2=0x7f30600c5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:52.120 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:52.122+0000 7f3066c46640 1 --2- 172.21.15.89:0/652362261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30600c5f60 0x7f30600c5650 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f305c00c3c0 tx=0x7f305c00c880 comp rx=0 tx=0).stop 2023-10-30T12:45:52.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:52.122+0000 7f3066c46640 1 -- 172.21.15.89:0/652362261 shutdown_connections 2023-10-30T12:45:52.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:52.122+0000 7f3066c46640 1 --2- 172.21.15.89:0/652362261 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f304c0664a0 0x7f304c068960 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:52.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:52.122+0000 7f3066c46640 1 --2- 172.21.15.89:0/652362261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3060153bf0 0x7f30601f2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:52.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:52.122+0000 7f3066c46640 1 --2- 172.21.15.89:0/652362261 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f30600c6fe0 0x7f3060155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:52.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:52.122+0000 7f3066c46640 1 --2- 172.21.15.89:0/652362261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30600c5f60 0x7f30600c5650 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:52.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:52.122+0000 7f3066c46640 1 -- 172.21.15.89:0/652362261 >> 172.21.15.89:0/652362261 conn(0x7f306006c6e0 msgr2=0x7f3060069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:45:52.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:52.122+0000 7f3066c46640 1 -- 172.21.15.89:0/652362261 shutdown_connections 2023-10-30T12:45:52.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:52.122+0000 7f3066c46640 1 -- 172.21.15.89:0/652362261 wait complete. 2023-10-30T12:45:52.140 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 83, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:50.729245+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 83, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5450": { "addr": "172.21.15.192:6837/604357386", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6836", "nonce": 604357386, "type": "v2" }, { "addr": "172.21.15.192:6837", "nonce": 604357386, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5450, "incarnation": 83, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:replay", "state_seq": 1 }, "gid_5575": { "addr": "172.21.15.192:6835/501291700", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6834", "nonce": 501291700, "type": "v2" }, { "addr": "172.21.15.192:6835", "nonce": 501291700, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5575, "incarnation": 63, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 37 }, "gid_5728": { "addr": "172.21.15.192:6839/3991891597", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6838", "nonce": 3991891597, "type": "v2" }, { "addr": "172.21.15.192:6839", "nonce": 3991891597, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5728, "incarnation": 73, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 19 } }, "last_failure": 0, "last_failure_osd_epoch": 63, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:45:51.226565+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5575, "mds_1": 5450, "mds_2": 5728 } } } ], "standbys": [ { "addr": "172.21.15.89:6834/2909790173", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6832", "nonce": 2909790173, "type": "v2" }, { "addr": "172.21.15.89:6834", "nonce": 2909790173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.89:6835/2387357615", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6833", "nonce": 2387357615, "type": "v2" }, { "addr": "172.21.15.89:6835", "nonce": 2387357615, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4239, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:45:54.141 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:45:54.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.290+0000 7f139dc9c640 1 Processor -- start 2023-10-30T12:45:54.289 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.290+0000 7f139dc9c640 1 -- start start 2023-10-30T12:45:54.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.290+0000 7f139dc9c640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13980c6b80 0x7f13980bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:54.290 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.290+0000 7f139dc9c640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f13980c6f60 0x7f13980c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:54.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.290+0000 7f139dc9c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f13980c8430 0x7f139813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:54.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.290+0000 7f139dc9c640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13980c0a50 con 0x7f13980c8430 2023-10-30T12:45:54.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.290+0000 7f139dc9c640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f139806f6e0 con 0x7f13980c6f60 2023-10-30T12:45:54.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.290+0000 7f139dc9c640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f139806f9c0 con 0x7f13980c6b80 2023-10-30T12:45:54.291 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.294+0000 7f1397fff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f13980c8430 0x7f139813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:54.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.294+0000 7f13977fe640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13980c6b80 0x7f13980bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:54.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.294+0000 7f1397fff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f13980c8430 0x7f139813ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44940/0 (socket says 172.21.15.89:44940) 2023-10-30T12:45:54.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.294+0000 7f1397fff640 1 -- 172.21.15.89:0/4007419682 learned_addr learned my addr 172.21.15.89:0/4007419682 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:45:54.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.294+0000 7f1396ffd640 1 --2- 172.21.15.89:0/4007419682 >> v2:172.21.15.192:3300/0 conn(0x7f13980c6f60 0x7f13980c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:54.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.294+0000 7f1397fff640 1 -- 172.21.15.89:0/4007419682 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13980c6b80 msgr2=0x7f13980bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:45:54.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.294+0000 7f1397fff640 1 --2- 172.21.15.89:0/4007419682 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13980c6b80 0x7f13980bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:54.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.294+0000 7f1397fff640 1 -- 172.21.15.89:0/4007419682 >> v2:172.21.15.192:3300/0 conn(0x7f13980c6f60 msgr2=0x7f13980c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:45:54.293 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.294+0000 7f1397fff640 1 --2- 172.21.15.89:0/4007419682 >> v2:172.21.15.192:3300/0 conn(0x7f13980c6f60 0x7f13980c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:54.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.294+0000 7f1397fff640 1 -- 172.21.15.89:0/4007419682 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f139806fca0 con 0x7f13980c8430 2023-10-30T12:45:54.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.294+0000 7f1397fff640 1 --2- 172.21.15.89:0/4007419682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13980c8430 0x7f139813ed90 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f139806c830 tx=0x7f139001c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbd07b7f19791143 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:54.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.294+0000 7f13967fc640 1 -- 172.21.15.89:0/4007419682 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13900a5760 con 0x7f13980c8430 2023-10-30T12:45:54.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.294+0000 7f13967fc640 1 -- 172.21.15.89:0/4007419682 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13900a58c0 con 0x7f13980c8430 2023-10-30T12:45:54.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.294+0000 7f13967fc640 1 -- 172.21.15.89:0/4007419682 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13900a5c50 con 0x7f13980c8430 2023-10-30T12:45:54.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.298+0000 7f139dc9c640 1 -- 172.21.15.89:0/4007419682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13980c8430 msgr2=0x7f139813ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:54.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.298+0000 7f139dc9c640 1 --2- 172.21.15.89:0/4007419682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13980c8430 0x7f139813ed90 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f139806c830 tx=0x7f139001c9d0 comp rx=0 tx=0).stop 2023-10-30T12:45:54.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.298+0000 7f139dc9c640 1 -- 172.21.15.89:0/4007419682 shutdown_connections 2023-10-30T12:45:54.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.298+0000 7f139dc9c640 1 --2- 172.21.15.89:0/4007419682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13980c8430 0x7f139813ed90 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:54.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.298+0000 7f139dc9c640 1 --2- 172.21.15.89:0/4007419682 >> v2:172.21.15.192:3300/0 conn(0x7f13980c6f60 0x7f13980c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:54.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.298+0000 7f139dc9c640 1 --2- 172.21.15.89:0/4007419682 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13980c6b80 0x7f13980bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:54.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.298+0000 7f139dc9c640 1 -- 172.21.15.89:0/4007419682 >> 172.21.15.89:0/4007419682 conn(0x7f13980c7910 msgr2=0x7f13980c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:45:54.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.298+0000 7f139dc9c640 1 -- 172.21.15.89:0/4007419682 shutdown_connections 2023-10-30T12:45:54.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.298+0000 7f139dc9c640 1 -- 172.21.15.89:0/4007419682 wait complete. 2023-10-30T12:45:54.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.298+0000 7f139dc9c640 1 Processor -- start 2023-10-30T12:45:54.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.298+0000 7f139dc9c640 1 -- start start 2023-10-30T12:45:54.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.298+0000 7f139dc9c640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13980c6b80 0x7f13981388b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:54.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.298+0000 7f139dc9c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13980c6f60 0x7f13981f2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:54.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.298+0000 7f139dc9c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13980c8430 0x7f139813d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:54.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.298+0000 7f139dc9c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1398155370 con 0x7f13980c8430 2023-10-30T12:45:54.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.298+0000 7f139dc9c640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1398155ef0 con 0x7f13980c6f60 2023-10-30T12:45:54.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.298+0000 7f139dc9c640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f139813d6a0 con 0x7f13980c6b80 2023-10-30T12:45:54.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.298+0000 7f1397fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13980c8430 0x7f139813d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:54.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.298+0000 7f1397fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13980c8430 0x7f139813d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44942/0 (socket says 172.21.15.89:44942) 2023-10-30T12:45:54.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.298+0000 7f1397fff640 1 -- 172.21.15.89:0/1453135737 learned_addr learned my addr 172.21.15.89:0/1453135737 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:45:54.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.298+0000 7f13977fe640 1 --2- 172.21.15.89:0/1453135737 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13980c6b80 0x7f13981388b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:54.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.298+0000 7f1396ffd640 1 --2- 172.21.15.89:0/1453135737 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13980c6f60 0x7f13981f2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:54.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.302+0000 7f1397fff640 1 -- 172.21.15.89:0/1453135737 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13980c6b80 msgr2=0x7f13981388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:54.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.302+0000 7f1397fff640 1 --2- 172.21.15.89:0/1453135737 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13980c6b80 0x7f13981388b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:54.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.302+0000 7f1397fff640 1 -- 172.21.15.89:0/1453135737 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13980c6f60 msgr2=0x7f13981f2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:54.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.302+0000 7f1397fff640 1 --2- 172.21.15.89:0/1453135737 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13980c6f60 0x7f13981f2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:54.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.302+0000 7f1397fff640 1 -- 172.21.15.89:0/1453135737 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1390009b30 con 0x7f13980c8430 2023-10-30T12:45:54.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.302+0000 7f13977fe640 1 --2- 172.21.15.89:0/1453135737 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13980c6b80 0x7f13981388b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:45:54.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.302+0000 7f1397fff640 1 --2- 172.21.15.89:0/1453135737 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13980c8430 0x7f139813d9f0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f13900af040 tx=0x7f1390019770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:54.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.302+0000 7f137ffff640 1 -- 172.21.15.89:0/1453135737 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13900a5760 con 0x7f13980c8430 2023-10-30T12:45:54.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.302+0000 7f137ffff640 1 -- 172.21.15.89:0/1453135737 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13900b88a0 con 0x7f13980c8430 2023-10-30T12:45:54.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.302+0000 7f139dc9c640 1 -- 172.21.15.89:0/1453135737 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13981545b0 con 0x7f13980c8430 2023-10-30T12:45:54.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.302+0000 7f137ffff640 1 -- 172.21.15.89:0/1453135737 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13900b8c50 con 0x7f13980c8430 2023-10-30T12:45:54.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.302+0000 7f139dc9c640 1 -- 172.21.15.89:0/1453135737 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1398155830 con 0x7f13980c8430 2023-10-30T12:45:54.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.302+0000 7f139dc9c640 1 -- 172.21.15.89:0/1453135737 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1398156030 con 0x7f13980c8430 2023-10-30T12:45:54.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.302+0000 7f137ffff640 1 -- 172.21.15.89:0/1453135737 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f13900b6440 con 0x7f13980c8430 2023-10-30T12:45:54.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.302+0000 7f137ffff640 1 --2- 172.21.15.89:0/1453135737 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f13840664d0 0x7f1384068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:54.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.302+0000 7f137ffff640 1 -- 172.21.15.89:0/1453135737 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6492+0+0 (secure 0 0 0) 0x7f139011f740 con 0x7f13980c8430 2023-10-30T12:45:54.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.306+0000 7f13977fe640 1 --2- 172.21.15.89:0/1453135737 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f13840664d0 0x7f1384068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:54.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.306+0000 7f13977fe640 1 --2- 172.21.15.89:0/1453135737 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f13840664d0 0x7f1384068990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f138800b240 tx=0x7f138800b3a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:54.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.306+0000 7f137ffff640 1 -- 172.21.15.89:0/1453135737 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1398156030 con 0x7f13980c8430 2023-10-30T12:45:54.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.494+0000 7f139dc9c640 1 -- 172.21.15.89:0/1453135737 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1398156030 con 0x7f13980c8430 2023-10-30T12:45:54.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.494+0000 7f137ffff640 1 -- 172.21.15.89:0/1453135737 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+5740 (secure 0 0 0) 0x7f1398156030 con 0x7f13980c8430 2023-10-30T12:45:54.494 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:45:54.494 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:45:51.226565+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5450,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":1,"incarnation":83,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:45:54.494 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 83 2023-10-30T12:45:54.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.498+0000 7f139dc9c640 1 -- 172.21.15.89:0/1453135737 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f13840664d0 msgr2=0x7f1384068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:54.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.498+0000 7f139dc9c640 1 --2- 172.21.15.89:0/1453135737 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f13840664d0 0x7f1384068990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f138800b240 tx=0x7f138800b3a0 comp rx=0 tx=0).stop 2023-10-30T12:45:54.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.498+0000 7f139dc9c640 1 -- 172.21.15.89:0/1453135737 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13980c8430 msgr2=0x7f139813d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:54.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.498+0000 7f139dc9c640 1 --2- 172.21.15.89:0/1453135737 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13980c8430 0x7f139813d9f0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f13900af040 tx=0x7f1390019770 comp rx=0 tx=0).stop 2023-10-30T12:45:54.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.498+0000 7f139dc9c640 1 -- 172.21.15.89:0/1453135737 shutdown_connections 2023-10-30T12:45:54.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.498+0000 7f139dc9c640 1 --2- 172.21.15.89:0/1453135737 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f13840664d0 0x7f1384068990 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:54.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.498+0000 7f139dc9c640 1 --2- 172.21.15.89:0/1453135737 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13980c6f60 0x7f13981f2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:54.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.498+0000 7f139dc9c640 1 --2- 172.21.15.89:0/1453135737 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f13980c8430 0x7f139813d9f0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:54.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.498+0000 7f139dc9c640 1 --2- 172.21.15.89:0/1453135737 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f13980c6b80 0x7f13981388b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:54.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.498+0000 7f139dc9c640 1 -- 172.21.15.89:0/1453135737 >> 172.21.15.89:0/1453135737 conn(0x7f13980c7910 msgr2=0x7f139806a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:45:54.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.498+0000 7f139dc9c640 1 -- 172.21.15.89:0/1453135737 shutdown_connections 2023-10-30T12:45:54.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:54.498+0000 7f139dc9c640 1 -- 172.21.15.89:0/1453135737 wait complete. 2023-10-30T12:45:54.515 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:45:56.516 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:45:56.661 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.662+0000 7f25e0a09640 1 Processor -- start 2023-10-30T12:45:56.661 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.662+0000 7f25e0a09640 1 -- start start 2023-10-30T12:45:56.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25e0a09640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25dc0c6b80 0x7f25dc0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:56.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25e0a09640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f25dc0c6f60 0x7f25dc0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:56.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25e0a09640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25dc0c0a50 con 0x7f25dc0c8430 2023-10-30T12:45:56.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25e0a09640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25dc06f6e0 con 0x7f25dc0c6f60 2023-10-30T12:45:56.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25e0a09640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc06f9c0 con 0x7f25dc0c6b80 2023-10-30T12:45:56.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25dad9d640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f25dc0c8430 0x7f25dc13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36908/0 (socket says 172.21.15.89:36908) 2023-10-30T12:45:56.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25dad9d640 1 -- 172.21.15.89:0/628734904 learned_addr learned my addr 172.21.15.89:0/628734904 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:45:56.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25da59c640 1 --2- 172.21.15.89:0/628734904 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25dc0c6b80 0x7f25dc0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:56.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25d9d9b640 1 --2- 172.21.15.89:0/628734904 >> v2:172.21.15.192:3300/0 conn(0x7f25dc0c6f60 0x7f25dc0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:56.666 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25d959a640 1 -- 172.21.15.89:0/628734904 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461773969 0 0) 0x7f25dc0c0a50 con 0x7f25dc0c8430 2023-10-30T12:45:56.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25d959a640 1 -- 172.21.15.89:0/628734904 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25c4005740 con 0x7f25dc0c8430 2023-10-30T12:45:56.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25d959a640 1 -- 172.21.15.89:0/628734904 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 434540541 0 0) 0x7f25c4005740 con 0x7f25dc0c8430 2023-10-30T12:45:56.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25d959a640 1 -- 172.21.15.89:0/628734904 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25dc0c6b80 msgr2=0x7f25dc0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:45:56.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25d959a640 1 --2- 172.21.15.89:0/628734904 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25dc0c6b80 0x7f25dc0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:56.667 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25d959a640 1 -- 172.21.15.89:0/628734904 >> v2:172.21.15.192:3300/0 conn(0x7f25dc0c6f60 msgr2=0x7f25dc0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:45:56.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25d959a640 1 --2- 172.21.15.89:0/628734904 >> v2:172.21.15.192:3300/0 conn(0x7f25dc0c6f60 0x7f25dc0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:56.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25d959a640 1 -- 172.21.15.89:0/628734904 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25dc06fca0 con 0x7f25dc0c8430 2023-10-30T12:45:56.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25d959a640 1 -- 172.21.15.89:0/628734904 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f25d4001a00 con 0x7f25dc0c8430 2023-10-30T12:45:56.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25d959a640 1 -- 172.21.15.89:0/628734904 >> v1:172.21.15.89:6789/0 conn(0x7f25dc0c8430 legacy=0x7f25dc13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:56.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25d959a640 1 --2- 172.21.15.89:0/628734904 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25dc0c6b80 0x7f25c4005b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:56.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25d959a640 1 --2- 172.21.15.89:0/628734904 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25c4006110 0x7f25c40069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:56.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25d959a640 1 --2- 172.21.15.89:0/628734904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25c4006f20 0x7f25c40073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:56.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25d959a640 1 -- 172.21.15.89:0/628734904 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc06f6e0 con 0x7f25c4006f20 2023-10-30T12:45:56.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25d959a640 1 -- 172.21.15.89:0/628734904 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc06f9c0 con 0x7f25c4006110 2023-10-30T12:45:56.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25d959a640 1 -- 172.21.15.89:0/628734904 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c400dba0 con 0x7f25dc0c6b80 2023-10-30T12:45:56.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25dad9d640 1 --2- 172.21.15.89:0/628734904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25c4006f20 0x7f25c40073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:56.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25da59c640 1 --2- 172.21.15.89:0/628734904 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25dc0c6b80 0x7f25c4005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:56.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25d9d9b640 1 --2- 172.21.15.89:0/628734904 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25c4006110 0x7f25c40069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:56.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25dad9d640 1 -- 172.21.15.89:0/628734904 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25dc0c6b80 msgr2=0x7f25c4005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:45:56.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25dad9d640 1 --2- 172.21.15.89:0/628734904 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25dc0c6b80 0x7f25c4005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:56.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25dad9d640 1 -- 172.21.15.89:0/628734904 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25c4006110 msgr2=0x7f25c40069e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:45:56.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25dad9d640 1 --2- 172.21.15.89:0/628734904 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25c4006110 0x7f25c40069e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:56.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.666+0000 7f25dad9d640 1 -- 172.21.15.89:0/628734904 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c4005740 con 0x7f25c4006f20 2023-10-30T12:45:56.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.670+0000 7f25dad9d640 1 --2- 172.21.15.89:0/628734904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25c4006f20 0x7f25c40073a0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f25d400a860 tx=0x7f25d401d300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d26690274d8fe8d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:56.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.670+0000 7f25d959a640 1 -- 172.21.15.89:0/628734904 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d40ad630 con 0x7f25c4006f20 2023-10-30T12:45:56.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.670+0000 7f25d959a640 1 -- 172.21.15.89:0/628734904 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d40a8070 con 0x7f25c4006f20 2023-10-30T12:45:56.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.670+0000 7f25d959a640 1 -- 172.21.15.89:0/628734904 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d40ada30 con 0x7f25c4006f20 2023-10-30T12:45:56.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.670+0000 7f25e0a09640 1 -- 172.21.15.89:0/628734904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25c4006f20 msgr2=0x7f25c40073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:56.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.670+0000 7f25e0a09640 1 --2- 172.21.15.89:0/628734904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25c4006f20 0x7f25c40073a0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f25d400a860 tx=0x7f25d401d300 comp rx=0 tx=0).stop 2023-10-30T12:45:56.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.670+0000 7f25dad9d640 1 -- 172.21.15.89:0/628734904 reap_dead start 2023-10-30T12:45:56.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.670+0000 7f25e0a09640 1 -- 172.21.15.89:0/628734904 shutdown_connections 2023-10-30T12:45:56.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.670+0000 7f25e0a09640 1 -- 172.21.15.89:0/628734904 >> 172.21.15.89:0/628734904 conn(0x7f25dc0c7910 msgr2=0x7f25dc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:45:56.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.670+0000 7f25e0a09640 1 -- 172.21.15.89:0/628734904 shutdown_connections 2023-10-30T12:45:56.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.670+0000 7f25e0a09640 1 -- 172.21.15.89:0/628734904 wait complete. 2023-10-30T12:45:56.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.670+0000 7f25e0a09640 1 Processor -- start 2023-10-30T12:45:56.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.674+0000 7f25e0a09640 1 -- start start 2023-10-30T12:45:56.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.674+0000 7f25e0a09640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25c4006f20 0x7f25dc13d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:56.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.674+0000 7f25e0a09640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25dc13e440 0x7f25dc0ca590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:56.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.674+0000 7f25e0a09640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25dc0af5b0 0x7f25dc0afa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:56.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.674+0000 7f25e0a09640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc13d840 con 0x7f25c4006f20 2023-10-30T12:45:56.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.674+0000 7f25e0a09640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc0af120 con 0x7f25dc0af5b0 2023-10-30T12:45:56.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.674+0000 7f25e0a09640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc0aff70 con 0x7f25dc13e440 2023-10-30T12:45:56.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.674+0000 7f25da59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25c4006f20 0x7f25dc13d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:56.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.674+0000 7f25da59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25c4006f20 0x7f25dc13d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44960/0 (socket says 172.21.15.89:44960) 2023-10-30T12:45:56.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.674+0000 7f25da59c640 1 -- 172.21.15.89:0/2579069118 learned_addr learned my addr 172.21.15.89:0/2579069118 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:45:56.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.674+0000 7f25dad9d640 1 --2- 172.21.15.89:0/2579069118 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25dc0af5b0 0x7f25dc0afa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:56.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.674+0000 7f25d9d9b640 1 --2- 172.21.15.89:0/2579069118 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25dc13e440 0x7f25dc0ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:56.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.674+0000 7f25da59c640 1 -- 172.21.15.89:0/2579069118 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25dc13e440 msgr2=0x7f25dc0ca590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:56.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.674+0000 7f25da59c640 1 --2- 172.21.15.89:0/2579069118 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25dc13e440 0x7f25dc0ca590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:56.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.674+0000 7f25da59c640 1 -- 172.21.15.89:0/2579069118 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25dc0af5b0 msgr2=0x7f25dc0afa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:56.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.674+0000 7f25da59c640 1 --2- 172.21.15.89:0/2579069118 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25dc0af5b0 0x7f25dc0afa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:56.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.674+0000 7f25da59c640 1 -- 172.21.15.89:0/2579069118 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25dc13db70 con 0x7f25c4006f20 2023-10-30T12:45:56.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.674+0000 7f25da59c640 1 --2- 172.21.15.89:0/2579069118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25c4006f20 0x7f25dc13d300 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f25cc011a00 tx=0x7f25cc011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:56.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.674+0000 7f25caffd640 1 -- 172.21.15.89:0/2579069118 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25cc00faf0 con 0x7f25c4006f20 2023-10-30T12:45:56.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.674+0000 7f25caffd640 1 -- 172.21.15.89:0/2579069118 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25cc00fc50 con 0x7f25c4006f20 2023-10-30T12:45:56.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.674+0000 7f25e0a09640 1 -- 172.21.15.89:0/2579069118 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25dc13de50 con 0x7f25c4006f20 2023-10-30T12:45:56.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.674+0000 7f25caffd640 1 -- 172.21.15.89:0/2579069118 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25cc014440 con 0x7f25c4006f20 2023-10-30T12:45:56.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.674+0000 7f25e0a09640 1 -- 172.21.15.89:0/2579069118 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25dc1f2f20 con 0x7f25c4006f20 2023-10-30T12:45:56.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.674+0000 7f25e0a09640 1 -- 172.21.15.89:0/2579069118 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25dc0aec10 con 0x7f25c4006f20 2023-10-30T12:45:56.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.678+0000 7f25caffd640 1 -- 172.21.15.89:0/2579069118 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f25cc0145a0 con 0x7f25c4006f20 2023-10-30T12:45:56.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.678+0000 7f25caffd640 1 --2- 172.21.15.89:0/2579069118 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f25bc0664a0 0x7f25bc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:56.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.678+0000 7f25caffd640 1 -- 172.21.15.89:0/2579069118 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6492+0+0 (secure 0 0 0) 0x7f25cc084a00 con 0x7f25c4006f20 2023-10-30T12:45:56.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.678+0000 7f25d9d9b640 1 --2- 172.21.15.89:0/2579069118 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f25bc0664a0 0x7f25bc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:56.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.678+0000 7f25d9d9b640 1 --2- 172.21.15.89:0/2579069118 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f25bc0664a0 0x7f25bc068960 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f25dc0b0a70 tx=0x7f25d0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:56.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.682+0000 7f25caffd640 1 -- 172.21.15.89:0/2579069118 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f25dc0aec10 con 0x7f25c4006f20 2023-10-30T12:45:56.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.862+0000 7f25e0a09640 1 -- 172.21.15.89:0/2579069118 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25dc0aec10 con 0x7f25c4006f20 2023-10-30T12:45:56.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.862+0000 7f25caffd640 1 -- 172.21.15.89:0/2579069118 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+5740 (secure 0 0 0) 0x7f25dc0aec10 con 0x7f25c4006f20 2023-10-30T12:45:56.862 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:45:56.862 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:45:51.226565+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5450,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":1,"incarnation":83,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:45:56.862 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 83 2023-10-30T12:45:56.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.866+0000 7f25e0a09640 1 -- 172.21.15.89:0/2579069118 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f25bc0664a0 msgr2=0x7f25bc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:56.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.866+0000 7f25e0a09640 1 --2- 172.21.15.89:0/2579069118 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f25bc0664a0 0x7f25bc068960 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f25dc0b0a70 tx=0x7f25d0009190 comp rx=0 tx=0).stop 2023-10-30T12:45:56.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.866+0000 7f25e0a09640 1 -- 172.21.15.89:0/2579069118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25c4006f20 msgr2=0x7f25dc13d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:56.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.866+0000 7f25e0a09640 1 --2- 172.21.15.89:0/2579069118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25c4006f20 0x7f25dc13d300 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f25cc011a00 tx=0x7f25cc011e00 comp rx=0 tx=0).stop 2023-10-30T12:45:56.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.866+0000 7f25e0a09640 1 -- 172.21.15.89:0/2579069118 shutdown_connections 2023-10-30T12:45:56.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.866+0000 7f25e0a09640 1 --2- 172.21.15.89:0/2579069118 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f25bc0664a0 0x7f25bc068960 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:56.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.866+0000 7f25e0a09640 1 --2- 172.21.15.89:0/2579069118 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25dc0af5b0 0x7f25dc0afa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:56.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.866+0000 7f25e0a09640 1 --2- 172.21.15.89:0/2579069118 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f25dc13e440 0x7f25dc0ca590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:56.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.866+0000 7f25e0a09640 1 --2- 172.21.15.89:0/2579069118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25c4006f20 0x7f25dc13d300 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:56.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.866+0000 7f25e0a09640 1 -- 172.21.15.89:0/2579069118 >> 172.21.15.89:0/2579069118 conn(0x7f25dc0c7910 msgr2=0x7f25dc0c0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:45:56.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.866+0000 7f25e0a09640 1 -- 172.21.15.89:0/2579069118 shutdown_connections 2023-10-30T12:45:56.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:56.866+0000 7f25e0a09640 1 -- 172.21.15.89:0/2579069118 wait complete. 2023-10-30T12:45:56.882 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:45:58.883 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:45:59.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.034+0000 7f5816cc9640 1 Processor -- start 2023-10-30T12:45:59.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.034+0000 7f5816cc9640 1 -- start start 2023-10-30T12:45:59.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.034+0000 7f5816cc9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58100c6b80 0x7f58100bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:59.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.034+0000 7f5816cc9640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f581013f4a0 0x7f58100c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:59.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.034+0000 7f5816cc9640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58100c0a50 con 0x7f58100c6f60 2023-10-30T12:45:59.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.034+0000 7f5816cc9640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f581006f6e0 con 0x7f581013f4a0 2023-10-30T12:45:59.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.034+0000 7f5816cc9640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f581006f9c0 con 0x7f58100c6b80 2023-10-30T12:45:59.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.034+0000 7f5815266640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f581013f4a0 0x7f58100c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:59.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.034+0000 7f5815266640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f581013f4a0 0x7f58100c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44966/0 (socket says 172.21.15.89:44966) 2023-10-30T12:45:59.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.034+0000 7f5815266640 1 -- 172.21.15.89:0/2235322371 learned_addr learned my addr 172.21.15.89:0/2235322371 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:45:59.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.034+0000 7f5814a65640 1 --2- 172.21.15.89:0/2235322371 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58100c6b80 0x7f58100bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:59.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.034+0000 7f5815266640 1 -- 172.21.15.89:0/2235322371 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58100c6b80 msgr2=0x7f58100bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:45:59.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.034+0000 7f5815266640 1 --2- 172.21.15.89:0/2235322371 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58100c6b80 0x7f58100bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:59.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.034+0000 7f5815266640 1 -- 172.21.15.89:0/2235322371 >> v1:172.21.15.192:6789/0 conn(0x7f58100c6f60 legacy=0x7f581013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:59.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.034+0000 7f5815266640 1 -- 172.21.15.89:0/2235322371 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f581006fca0 con 0x7f581013f4a0 2023-10-30T12:45:59.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.038+0000 7f5815266640 1 --2- 172.21.15.89:0/2235322371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f581013f4a0 0x7f58100c0400 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f580800d150 tx=0x7f580801ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39cf6c045a3c4566 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:59.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.038+0000 7f58077fe640 1 -- 172.21.15.89:0/2235322371 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580800b2d0 con 0x7f581013f4a0 2023-10-30T12:45:59.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.038+0000 7f58077fe640 1 -- 172.21.15.89:0/2235322371 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f580800b430 con 0x7f581013f4a0 2023-10-30T12:45:59.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.038+0000 7f58077fe640 1 -- 172.21.15.89:0/2235322371 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58080a7c10 con 0x7f581013f4a0 2023-10-30T12:45:59.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.038+0000 7f5816cc9640 1 -- 172.21.15.89:0/2235322371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f581013f4a0 msgr2=0x7f58100c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:59.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.038+0000 7f5816cc9640 1 --2- 172.21.15.89:0/2235322371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f581013f4a0 0x7f58100c0400 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f580800d150 tx=0x7f580801ec10 comp rx=0 tx=0).stop 2023-10-30T12:45:59.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.038+0000 7f5816cc9640 1 -- 172.21.15.89:0/2235322371 shutdown_connections 2023-10-30T12:45:59.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.038+0000 7f5816cc9640 1 --2- 172.21.15.89:0/2235322371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f581013f4a0 0x7f58100c0400 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:59.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.038+0000 7f5816cc9640 1 --2- 172.21.15.89:0/2235322371 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f58100c6b80 0x7f58100bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:59.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.038+0000 7f5816cc9640 1 -- 172.21.15.89:0/2235322371 >> 172.21.15.89:0/2235322371 conn(0x7f58100c7910 msgr2=0x7f58100c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:45:59.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.038+0000 7f5816cc9640 1 -- 172.21.15.89:0/2235322371 shutdown_connections 2023-10-30T12:45:59.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.038+0000 7f5816cc9640 1 -- 172.21.15.89:0/2235322371 wait complete. 2023-10-30T12:45:59.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.042+0000 7f5816cc9640 1 Processor -- start 2023-10-30T12:45:59.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.042+0000 7f5816cc9640 1 -- start start 2023-10-30T12:45:59.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.042+0000 7f5816cc9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58100c6b80 0x7f5810137980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:59.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.042+0000 7f5816cc9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58100c6f60 0x7f58101f5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:59.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.042+0000 7f5816cc9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f581013f4a0 0x7f581006b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:59.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.042+0000 7f5816cc9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f581015ec30 con 0x7f58100c6f60 2023-10-30T12:45:59.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.042+0000 7f5816cc9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58101f2e40 con 0x7f58100c6b80 2023-10-30T12:45:59.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.042+0000 7f5816cc9640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58101f3400 con 0x7f581013f4a0 2023-10-30T12:45:59.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.042+0000 7f5807fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58100c6f60 0x7f58101f5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:59.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.042+0000 7f5807fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58100c6f60 0x7f58101f5900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44968/0 (socket says 172.21.15.89:44968) 2023-10-30T12:45:59.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.042+0000 7f5807fff640 1 -- 172.21.15.89:0/3802648277 learned_addr learned my addr 172.21.15.89:0/3802648277 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:45:59.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.042+0000 7f5814a65640 1 --2- 172.21.15.89:0/3802648277 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58100c6b80 0x7f5810137980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:59.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.042+0000 7f5815266640 1 --2- 172.21.15.89:0/3802648277 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f581013f4a0 0x7f581006b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:59.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.042+0000 7f5807fff640 1 -- 172.21.15.89:0/3802648277 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f581013f4a0 msgr2=0x7f581006b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:59.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.042+0000 7f5807fff640 1 --2- 172.21.15.89:0/3802648277 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f581013f4a0 0x7f581006b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:59.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.042+0000 7f5807fff640 1 -- 172.21.15.89:0/3802648277 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58100c6b80 msgr2=0x7f5810137980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:59.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.042+0000 7f5807fff640 1 --2- 172.21.15.89:0/3802648277 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58100c6b80 0x7f5810137980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:59.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.042+0000 7f5807fff640 1 -- 172.21.15.89:0/3802648277 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f580800bd40 con 0x7f58100c6f60 2023-10-30T12:45:59.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.042+0000 7f5807fff640 1 --2- 172.21.15.89:0/3802648277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58100c6f60 0x7f58101f5900 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f580000c500 tx=0x7f580000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:59.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.042+0000 7f58057fa640 1 -- 172.21.15.89:0/3802648277 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58000052d0 con 0x7f58100c6f60 2023-10-30T12:45:59.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.042+0000 7f58057fa640 1 -- 172.21.15.89:0/3802648277 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5800005430 con 0x7f58100c6f60 2023-10-30T12:45:59.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.042+0000 7f58057fa640 1 -- 172.21.15.89:0/3802648277 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5800012830 con 0x7f58100c6f60 2023-10-30T12:45:59.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.042+0000 7f5816cc9640 1 -- 172.21.15.89:0/3802648277 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5810137ec0 con 0x7f58100c6f60 2023-10-30T12:45:59.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.042+0000 7f5816cc9640 1 -- 172.21.15.89:0/3802648277 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f581015fbc0 con 0x7f58100c6f60 2023-10-30T12:45:59.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.046+0000 7f5816cc9640 1 -- 172.21.15.89:0/3802648277 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f581004e170 con 0x7f58100c6f60 2023-10-30T12:45:59.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.046+0000 7f58057fa640 1 -- 172.21.15.89:0/3802648277 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5800012990 con 0x7f58100c6f60 2023-10-30T12:45:59.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.046+0000 7f58057fa640 1 --2- 172.21.15.89:0/3802648277 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f57fc0664a0 0x7f57fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:45:59.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.046+0000 7f58057fa640 1 -- 172.21.15.89:0/3802648277 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6492+0+0 (secure 0 0 0) 0x7f5800083be0 con 0x7f58100c6f60 2023-10-30T12:45:59.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.046+0000 7f5814a65640 1 --2- 172.21.15.89:0/3802648277 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f57fc0664a0 0x7f57fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:45:59.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.046+0000 7f5814a65640 1 --2- 172.21.15.89:0/3802648277 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f57fc0664a0 0x7f57fc068960 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f580c007a60 tx=0x7f580c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:45:59.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.050+0000 7f58057fa640 1 -- 172.21.15.89:0/3802648277 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5800050d70 con 0x7f58100c6f60 2023-10-30T12:45:59.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.230+0000 7f5816cc9640 1 -- 172.21.15.89:0/3802648277 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58100b0b40 con 0x7f58100c6f60 2023-10-30T12:45:59.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.230+0000 7f58057fa640 1 -- 172.21.15.89:0/3802648277 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+5742 (secure 0 0 0) 0x7f5800054390 con 0x7f58100c6f60 2023-10-30T12:45:59.230 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:45:59.230 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:45:58.323422+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5450,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":1,"incarnation":83,"state":"up:resolve","state_seq":30,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:45:59.230 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 84 2023-10-30T12:45:59.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.234+0000 7f5816cc9640 1 -- 172.21.15.89:0/3802648277 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f57fc0664a0 msgr2=0x7f57fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:59.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.234+0000 7f5816cc9640 1 --2- 172.21.15.89:0/3802648277 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f57fc0664a0 0x7f57fc068960 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f580c007a60 tx=0x7f580c008040 comp rx=0 tx=0).stop 2023-10-30T12:45:59.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.234+0000 7f5816cc9640 1 -- 172.21.15.89:0/3802648277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58100c6f60 msgr2=0x7f58101f5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:45:59.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.234+0000 7f5816cc9640 1 --2- 172.21.15.89:0/3802648277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58100c6f60 0x7f58101f5900 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f580000c500 tx=0x7f580000c9c0 comp rx=0 tx=0).stop 2023-10-30T12:45:59.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.234+0000 7f5816cc9640 1 -- 172.21.15.89:0/3802648277 shutdown_connections 2023-10-30T12:45:59.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.234+0000 7f5816cc9640 1 --2- 172.21.15.89:0/3802648277 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f57fc0664a0 0x7f57fc068960 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:59.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.234+0000 7f5816cc9640 1 --2- 172.21.15.89:0/3802648277 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f581013f4a0 0x7f581006b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:59.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.234+0000 7f5816cc9640 1 --2- 172.21.15.89:0/3802648277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58100c6f60 0x7f58101f5900 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:59.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.234+0000 7f5816cc9640 1 --2- 172.21.15.89:0/3802648277 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58100c6b80 0x7f5810137980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:45:59.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.234+0000 7f5816cc9640 1 -- 172.21.15.89:0/3802648277 >> 172.21.15.89:0/3802648277 conn(0x7f58100c7910 msgr2=0x7f58100c5f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:45:59.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.234+0000 7f5816cc9640 1 -- 172.21.15.89:0/3802648277 shutdown_connections 2023-10-30T12:45:59.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:45:59.234+0000 7f5816cc9640 1 -- 172.21.15.89:0/3802648277 wait complete. 2023-10-30T12:45:59.251 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:46:01.252 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:46:01.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.402+0000 7fa29a9d3640 1 Processor -- start 2023-10-30T12:46:01.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.402+0000 7fa29a9d3640 1 -- start start 2023-10-30T12:46:01.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.402+0000 7fa29a9d3640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa2941538a0 0x7fa29415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:01.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.402+0000 7fa29a9d3640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa2940ca0e0 0x7fa29415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:01.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.402+0000 7fa29a9d3640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2940cb800 con 0x7fa2940cf160 2023-10-30T12:46:01.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.406+0000 7fa29a9d3640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa29415dc30 con 0x7fa2941538a0 2023-10-30T12:46:01.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.406+0000 7fa29a9d3640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2940cb330 con 0x7fa2940ca0e0 2023-10-30T12:46:01.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.406+0000 7fa293fff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa2941538a0 0x7fa29415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:01.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.406+0000 7fa293fff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fa2941538a0 0x7fa29415d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43272/0 (socket says 172.21.15.89:43272) 2023-10-30T12:46:01.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.406+0000 7fa293fff640 1 -- 172.21.15.89:0/2551082819 learned_addr learned my addr 172.21.15.89:0/2551082819 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:46:01.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.406+0000 7fa2937fe640 1 --2- 172.21.15.89:0/2551082819 >> v2:172.21.15.192:3300/0 conn(0x7fa2940ca0e0 0x7fa29415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:01.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.406+0000 7fa293fff640 1 -- 172.21.15.89:0/2551082819 >> v2:172.21.15.192:3300/0 conn(0x7fa2940ca0e0 msgr2=0x7fa29415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:01.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.406+0000 7fa293fff640 1 --2- 172.21.15.89:0/2551082819 >> v2:172.21.15.192:3300/0 conn(0x7fa2940ca0e0 0x7fa29415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:01.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.406+0000 7fa293fff640 1 -- 172.21.15.89:0/2551082819 >> v1:172.21.15.89:6789/0 conn(0x7fa2940cf160 legacy=0x7fa2940cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:01.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.406+0000 7fa293fff640 1 -- 172.21.15.89:0/2551082819 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2940cad70 con 0x7fa2941538a0 2023-10-30T12:46:01.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.406+0000 7fa293fff640 1 --2- 172.21.15.89:0/2551082819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2941538a0 0x7fa29415d6f0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fa2940b30a0 tx=0x7fa28401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f30a8d2ea74f6660 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:01.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.406+0000 7fa292ffd640 1 -- 172.21.15.89:0/2551082819 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2840a89f0 con 0x7fa2941538a0 2023-10-30T12:46:01.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.410+0000 7fa292ffd640 1 -- 172.21.15.89:0/2551082819 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2840a8b50 con 0x7fa2941538a0 2023-10-30T12:46:01.407 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.410+0000 7fa292ffd640 1 -- 172.21.15.89:0/2551082819 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2840b0440 con 0x7fa2941538a0 2023-10-30T12:46:01.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.410+0000 7fa29a9d3640 1 -- 172.21.15.89:0/2551082819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2941538a0 msgr2=0x7fa29415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:01.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.410+0000 7fa29a9d3640 1 --2- 172.21.15.89:0/2551082819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2941538a0 0x7fa29415d6f0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fa2940b30a0 tx=0x7fa28401ecf0 comp rx=0 tx=0).stop 2023-10-30T12:46:01.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.410+0000 7fa29a9d3640 1 -- 172.21.15.89:0/2551082819 shutdown_connections 2023-10-30T12:46:01.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.410+0000 7fa29a9d3640 1 --2- 172.21.15.89:0/2551082819 >> v2:172.21.15.192:3300/0 conn(0x7fa2940ca0e0 0x7fa29415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:01.408 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.410+0000 7fa29a9d3640 1 --2- 172.21.15.89:0/2551082819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2941538a0 0x7fa29415d6f0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:01.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.410+0000 7fa29a9d3640 1 -- 172.21.15.89:0/2551082819 >> 172.21.15.89:0/2551082819 conn(0x7fa29406bfe0 msgr2=0x7fa29406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:01.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.410+0000 7fa29a9d3640 1 -- 172.21.15.89:0/2551082819 shutdown_connections 2023-10-30T12:46:01.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.410+0000 7fa29a9d3640 1 -- 172.21.15.89:0/2551082819 wait complete. 2023-10-30T12:46:01.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.410+0000 7fa29a9d3640 1 Processor -- start 2023-10-30T12:46:01.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.410+0000 7fa29a9d3640 1 -- start start 2023-10-30T12:46:01.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.410+0000 7fa29a9d3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2940ca0e0 0x7fa2940c8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:01.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.410+0000 7fa29a9d3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2940cf160 0x7fa29414ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:01.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.410+0000 7fa29a9d3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2941538a0 0x7fa2941ef210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:01.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.410+0000 7fa29a9d3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa294151a80 con 0x7fa2940cf160 2023-10-30T12:46:01.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.410+0000 7fa29a9d3640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2941f3330 con 0x7fa2941538a0 2023-10-30T12:46:01.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.410+0000 7fa29a9d3640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2940c9430 con 0x7fa2940ca0e0 2023-10-30T12:46:01.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.410+0000 7fa2937fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2940cf160 0x7fa29414ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:01.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.410+0000 7fa2937fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2940cf160 0x7fa29414ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43286/0 (socket says 172.21.15.89:43286) 2023-10-30T12:46:01.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.410+0000 7fa2937fe640 1 -- 172.21.15.89:0/1344627616 learned_addr learned my addr 172.21.15.89:0/1344627616 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:46:01.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.414+0000 7fa293fff640 1 --2- 172.21.15.89:0/1344627616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2940ca0e0 0x7fa2940c8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:01.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.414+0000 7fa298f70640 1 --2- 172.21.15.89:0/1344627616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2941538a0 0x7fa2941ef210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:01.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.414+0000 7fa2937fe640 1 -- 172.21.15.89:0/1344627616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2940ca0e0 msgr2=0x7fa2940c8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:01.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.414+0000 7fa2937fe640 1 --2- 172.21.15.89:0/1344627616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2940ca0e0 0x7fa2940c8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:01.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.414+0000 7fa2937fe640 1 -- 172.21.15.89:0/1344627616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2941538a0 msgr2=0x7fa2941ef210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:01.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.414+0000 7fa2937fe640 1 --2- 172.21.15.89:0/1344627616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2941538a0 0x7fa2941ef210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:01.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.414+0000 7fa2937fe640 1 -- 172.21.15.89:0/1344627616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2840094d0 con 0x7fa2940cf160 2023-10-30T12:46:01.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.414+0000 7fa2937fe640 1 --2- 172.21.15.89:0/1344627616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2940cf160 0x7fa29414ddb0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fa28800d860 tx=0x7fa28800dd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:01.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.414+0000 7fa290ff9640 1 -- 172.21.15.89:0/1344627616 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28800de50 con 0x7fa2940cf160 2023-10-30T12:46:01.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.414+0000 7fa290ff9640 1 -- 172.21.15.89:0/1344627616 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa28800b8d0 con 0x7fa2940cf160 2023-10-30T12:46:01.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.414+0000 7fa29a9d3640 1 -- 172.21.15.89:0/1344627616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2941f2720 con 0x7fa2940cf160 2023-10-30T12:46:01.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.414+0000 7fa290ff9640 1 -- 172.21.15.89:0/1344627616 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28800bc40 con 0x7fa2940cf160 2023-10-30T12:46:01.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.414+0000 7fa29a9d3640 1 -- 172.21.15.89:0/1344627616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2941f2ce0 con 0x7fa2940cf160 2023-10-30T12:46:01.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.414+0000 7fa29a9d3640 1 -- 172.21.15.89:0/1344627616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2940b2a50 con 0x7fa2940cf160 2023-10-30T12:46:01.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.414+0000 7fa290ff9640 1 -- 172.21.15.89:0/1344627616 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa28801e460 con 0x7fa2940cf160 2023-10-30T12:46:01.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.418+0000 7fa290ff9640 1 --2- 172.21.15.89:0/1344627616 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa2800664a0 0x7fa280068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:01.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.418+0000 7fa290ff9640 1 -- 172.21.15.89:0/1344627616 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6492+0+0 (secure 0 0 0) 0x7fa288084790 con 0x7fa2940cf160 2023-10-30T12:46:01.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.418+0000 7fa293fff640 1 --2- 172.21.15.89:0/1344627616 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa2800664a0 0x7fa280068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:01.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.418+0000 7fa293fff640 1 --2- 172.21.15.89:0/1344627616 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa2800664a0 0x7fa280068960 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fa2840b2000 tx=0x7fa284008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:01.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.422+0000 7fa290ff9640 1 -- 172.21.15.89:0/1344627616 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa288051870 con 0x7fa2940cf160 2023-10-30T12:46:01.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.602+0000 7fa29a9d3640 1 -- 172.21.15.89:0/1344627616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa29404e170 con 0x7fa2940cf160 2023-10-30T12:46:01.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.602+0000 7fa290ff9640 1 -- 172.21.15.89:0/1344627616 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+5742 (secure 0 0 0) 0x7fa288054e90 con 0x7fa2940cf160 2023-10-30T12:46:01.599 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:46:01.600 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:45:58.323422+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5450,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":1,"incarnation":83,"state":"up:resolve","state_seq":30,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:46:01.600 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 84 2023-10-30T12:46:01.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.602+0000 7fa29a9d3640 1 -- 172.21.15.89:0/1344627616 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa2800664a0 msgr2=0x7fa280068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:01.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.602+0000 7fa29a9d3640 1 --2- 172.21.15.89:0/1344627616 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa2800664a0 0x7fa280068960 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fa2840b2000 tx=0x7fa284008040 comp rx=0 tx=0).stop 2023-10-30T12:46:01.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.602+0000 7fa29a9d3640 1 -- 172.21.15.89:0/1344627616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2940cf160 msgr2=0x7fa29414ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:01.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.602+0000 7fa29a9d3640 1 --2- 172.21.15.89:0/1344627616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2940cf160 0x7fa29414ddb0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fa28800d860 tx=0x7fa28800dd20 comp rx=0 tx=0).stop 2023-10-30T12:46:01.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.602+0000 7fa29a9d3640 1 -- 172.21.15.89:0/1344627616 shutdown_connections 2023-10-30T12:46:01.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.602+0000 7fa29a9d3640 1 --2- 172.21.15.89:0/1344627616 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa2800664a0 0x7fa280068960 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:01.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.602+0000 7fa29a9d3640 1 --2- 172.21.15.89:0/1344627616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2941538a0 0x7fa2941ef210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:01.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.602+0000 7fa29a9d3640 1 --2- 172.21.15.89:0/1344627616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2940cf160 0x7fa29414ddb0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:01.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.602+0000 7fa29a9d3640 1 --2- 172.21.15.89:0/1344627616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2940ca0e0 0x7fa2940c8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:01.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.602+0000 7fa29a9d3640 1 -- 172.21.15.89:0/1344627616 >> 172.21.15.89:0/1344627616 conn(0x7fa29406bfe0 msgr2=0x7fa2940c4500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:01.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.602+0000 7fa29a9d3640 1 -- 172.21.15.89:0/1344627616 shutdown_connections 2023-10-30T12:46:01.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:01.602+0000 7fa29a9d3640 1 -- 172.21.15.89:0/1344627616 wait complete. 2023-10-30T12:46:01.620 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:46:03.621 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:46:03.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.770+0000 7fe584046640 1 Processor -- start 2023-10-30T12:46:03.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.770+0000 7fe584046640 1 -- start start 2023-10-30T12:46:03.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.774+0000 7fe584046640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe57c153d70 0x7fe57c06ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:03.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.774+0000 7fe584046640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe57c06afa0 con 0x7fe57c0c8490 2023-10-30T12:46:03.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.774+0000 7fe584046640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe57c154150 con 0x7fe57c06e470 2023-10-30T12:46:03.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.774+0000 7fe584046640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe57c06d780 con 0x7fe57c153d70 2023-10-30T12:46:03.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.774+0000 7fe581de2640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe57c153d70 0x7fe57c06ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:03.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.774+0000 7fe581de2640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe57c153d70 0x7fe57c06ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43296/0 (socket says 172.21.15.89:43296) 2023-10-30T12:46:03.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.774+0000 7fe581de2640 1 -- 172.21.15.89:0/1929413466 learned_addr learned my addr 172.21.15.89:0/1929413466 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:46:03.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.774+0000 7fe581de2640 1 -- 172.21.15.89:0/1929413466 >> v1:172.21.15.192:6789/0 conn(0x7fe57c06e470 legacy=0x7fe57c0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:03.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.774+0000 7fe581de2640 1 -- 172.21.15.89:0/1929413466 >> v1:172.21.15.89:6789/0 conn(0x7fe57c0c8490 legacy=0x7fe57c0c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:03.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.774+0000 7fe581de2640 1 -- 172.21.15.89:0/1929413466 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe57c06e030 con 0x7fe57c153d70 2023-10-30T12:46:03.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.778+0000 7fe581de2640 1 --2- 172.21.15.89:0/1929413466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe57c153d70 0x7fe57c06ca80 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fe578009870 tx=0x7fe57801c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9e3e5be9a749634 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:03.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.778+0000 7fe580de0640 1 -- 172.21.15.89:0/1929413466 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5780aa670 con 0x7fe57c153d70 2023-10-30T12:46:03.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.778+0000 7fe580de0640 1 -- 172.21.15.89:0/1929413466 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5780aa7d0 con 0x7fe57c153d70 2023-10-30T12:46:03.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.778+0000 7fe580de0640 1 -- 172.21.15.89:0/1929413466 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5780aab60 con 0x7fe57c153d70 2023-10-30T12:46:03.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.778+0000 7fe584046640 1 -- 172.21.15.89:0/1929413466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe57c153d70 msgr2=0x7fe57c06ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:03.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.778+0000 7fe584046640 1 --2- 172.21.15.89:0/1929413466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe57c153d70 0x7fe57c06ca80 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fe578009870 tx=0x7fe57801c510 comp rx=0 tx=0).stop 2023-10-30T12:46:03.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.778+0000 7fe584046640 1 -- 172.21.15.89:0/1929413466 shutdown_connections 2023-10-30T12:46:03.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.778+0000 7fe584046640 1 --2- 172.21.15.89:0/1929413466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe57c153d70 0x7fe57c06ca80 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:03.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.778+0000 7fe584046640 1 -- 172.21.15.89:0/1929413466 >> 172.21.15.89:0/1929413466 conn(0x7fe57c071ea0 msgr2=0x7fe57c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:03.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.778+0000 7fe584046640 1 -- 172.21.15.89:0/1929413466 shutdown_connections 2023-10-30T12:46:03.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.778+0000 7fe584046640 1 -- 172.21.15.89:0/1929413466 wait complete. 2023-10-30T12:46:03.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.778+0000 7fe584046640 1 Processor -- start 2023-10-30T12:46:03.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.778+0000 7fe584046640 1 -- start start 2023-10-30T12:46:03.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.782+0000 7fe584046640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe57c06e470 0x7fe57c142d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:03.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.782+0000 7fe584046640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe57c0c8490 0x7fe57c1f2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:03.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.782+0000 7fe584046640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe57c153d70 0x7fe57c149c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:03.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.782+0000 7fe584046640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57c14af60 con 0x7fe57c0c8490 2023-10-30T12:46:03.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.782+0000 7fe584046640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57c0cbf80 con 0x7fe57c153d70 2023-10-30T12:46:03.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.782+0000 7fe584046640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe57c15a850 con 0x7fe57c06e470 2023-10-30T12:46:03.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.782+0000 7fe5815e1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe57c0c8490 0x7fe57c1f2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:03.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.782+0000 7fe5815e1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe57c0c8490 0x7fe57c1f2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43310/0 (socket says 172.21.15.89:43310) 2023-10-30T12:46:03.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.782+0000 7fe5815e1640 1 -- 172.21.15.89:0/242998828 learned_addr learned my addr 172.21.15.89:0/242998828 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:46:03.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.782+0000 7fe581de2640 1 --2- 172.21.15.89:0/242998828 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe57c06e470 0x7fe57c142d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:03.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.782+0000 7fe5825e3640 1 --2- 172.21.15.89:0/242998828 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe57c153d70 0x7fe57c149c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:03.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.782+0000 7fe5815e1640 1 -- 172.21.15.89:0/242998828 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe57c06e470 msgr2=0x7fe57c142d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:03.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.782+0000 7fe5815e1640 1 --2- 172.21.15.89:0/242998828 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe57c06e470 0x7fe57c142d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:03.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.782+0000 7fe5815e1640 1 -- 172.21.15.89:0/242998828 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe57c153d70 msgr2=0x7fe57c149c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:03.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.782+0000 7fe5815e1640 1 --2- 172.21.15.89:0/242998828 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe57c153d70 0x7fe57c149c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:03.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.782+0000 7fe5815e1640 1 -- 172.21.15.89:0/242998828 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5780094d0 con 0x7fe57c0c8490 2023-10-30T12:46:03.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.782+0000 7fe5825e3640 1 --2- 172.21.15.89:0/242998828 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe57c153d70 0x7fe57c149c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:46:03.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.782+0000 7fe5815e1640 1 --2- 172.21.15.89:0/242998828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe57c0c8490 0x7fe57c1f2fe0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fe56c00c3c0 tx=0x7fe56c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:03.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.782+0000 7fe5727fc640 1 -- 172.21.15.89:0/242998828 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56c0052d0 con 0x7fe57c0c8490 2023-10-30T12:46:03.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.782+0000 7fe5727fc640 1 -- 172.21.15.89:0/242998828 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe56c005430 con 0x7fe57c0c8490 2023-10-30T12:46:03.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.782+0000 7fe5727fc640 1 -- 172.21.15.89:0/242998828 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56c003840 con 0x7fe57c0c8490 2023-10-30T12:46:03.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.782+0000 7fe584046640 1 -- 172.21.15.89:0/242998828 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe57c149570 con 0x7fe57c0c8490 2023-10-30T12:46:03.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.782+0000 7fe584046640 1 -- 172.21.15.89:0/242998828 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe57c141390 con 0x7fe57c0c8490 2023-10-30T12:46:03.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.782+0000 7fe584046640 1 -- 172.21.15.89:0/242998828 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe57c04e170 con 0x7fe57c0c8490 2023-10-30T12:46:03.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.782+0000 7fe5727fc640 1 -- 172.21.15.89:0/242998828 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe56c0039a0 con 0x7fe57c0c8490 2023-10-30T12:46:03.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.786+0000 7fe5727fc640 1 --2- 172.21.15.89:0/242998828 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe5600664d0 0x7fe560068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:03.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.786+0000 7fe5727fc640 1 -- 172.21.15.89:0/242998828 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6492+0+0 (secure 0 0 0) 0x7fe56c0836d0 con 0x7fe57c0c8490 2023-10-30T12:46:03.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.786+0000 7fe581de2640 1 --2- 172.21.15.89:0/242998828 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe5600664d0 0x7fe560068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:03.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.786+0000 7fe581de2640 1 --2- 172.21.15.89:0/242998828 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe5600664d0 0x7fe560068990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe5780ae000 tx=0x7fe5780b5040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:03.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.790+0000 7fe5727fc640 1 -- 172.21.15.89:0/242998828 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe56c050850 con 0x7fe57c0c8490 2023-10-30T12:46:03.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.970+0000 7fe584046640 1 -- 172.21.15.89:0/242998828 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe57c06c2a0 con 0x7fe57c0c8490 2023-10-30T12:46:03.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.970+0000 7fe5727fc640 1 -- 172.21.15.89:0/242998828 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+5742 (secure 0 0 0) 0x7fe56c053e70 con 0x7fe57c0c8490 2023-10-30T12:46:03.970 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:46:03.970 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:45:58.323422+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5450,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":1,"incarnation":83,"state":"up:resolve","state_seq":30,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:46:03.971 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 84 2023-10-30T12:46:03.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.974+0000 7fe584046640 1 -- 172.21.15.89:0/242998828 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe5600664d0 msgr2=0x7fe560068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:03.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.974+0000 7fe584046640 1 --2- 172.21.15.89:0/242998828 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe5600664d0 0x7fe560068990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe5780ae000 tx=0x7fe5780b5040 comp rx=0 tx=0).stop 2023-10-30T12:46:03.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.974+0000 7fe584046640 1 -- 172.21.15.89:0/242998828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe57c0c8490 msgr2=0x7fe57c1f2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:03.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.974+0000 7fe584046640 1 --2- 172.21.15.89:0/242998828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe57c0c8490 0x7fe57c1f2fe0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fe56c00c3c0 tx=0x7fe56c00c880 comp rx=0 tx=0).stop 2023-10-30T12:46:03.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.974+0000 7fe584046640 1 -- 172.21.15.89:0/242998828 shutdown_connections 2023-10-30T12:46:03.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.974+0000 7fe584046640 1 --2- 172.21.15.89:0/242998828 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe5600664d0 0x7fe560068990 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:03.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.974+0000 7fe584046640 1 --2- 172.21.15.89:0/242998828 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe57c153d70 0x7fe57c149c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:03.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.974+0000 7fe584046640 1 --2- 172.21.15.89:0/242998828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe57c0c8490 0x7fe57c1f2fe0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:03.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.974+0000 7fe584046640 1 --2- 172.21.15.89:0/242998828 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe57c06e470 0x7fe57c142d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:03.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.974+0000 7fe584046640 1 -- 172.21.15.89:0/242998828 >> 172.21.15.89:0/242998828 conn(0x7fe57c071ea0 msgr2=0x7fe57c0c7520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:03.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.974+0000 7fe584046640 1 -- 172.21.15.89:0/242998828 shutdown_connections 2023-10-30T12:46:03.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:03.974+0000 7fe584046640 1 -- 172.21.15.89:0/242998828 wait complete. 2023-10-30T12:46:03.990 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 84, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:50.729245+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 84, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5450": { "addr": "172.21.15.192:6837/604357386", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6836", "nonce": 604357386, "type": "v2" }, { "addr": "172.21.15.192:6837", "nonce": 604357386, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5450, "incarnation": 83, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:resolve", "state_seq": 30 }, "gid_5575": { "addr": "172.21.15.192:6835/501291700", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6834", "nonce": 501291700, "type": "v2" }, { "addr": "172.21.15.192:6835", "nonce": 501291700, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5575, "incarnation": 63, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 37 }, "gid_5728": { "addr": "172.21.15.192:6839/3991891597", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6838", "nonce": 3991891597, "type": "v2" }, { "addr": "172.21.15.192:6839", "nonce": 3991891597, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5728, "incarnation": 73, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 19 } }, "last_failure": 0, "last_failure_osd_epoch": 63, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:45:58.323422+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5575, "mds_1": 5450, "mds_2": 5728 } } } ], "standbys": [ { "addr": "172.21.15.89:6834/2909790173", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6832", "nonce": 2909790173, "type": "v2" }, { "addr": "172.21.15.89:6834", "nonce": 2909790173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.89:6835/2387357615", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6833", "nonce": 2387357615, "type": "v2" }, { "addr": "172.21.15.89:6835", "nonce": 2387357615, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4239, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:46:05.991 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:46:06.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.142+0000 7f8a5a372640 1 Processor -- start 2023-10-30T12:46:06.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.142+0000 7f8a5a372640 1 -- start start 2023-10-30T12:46:06.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.142+0000 7f8a5a372640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a541538a0 0x7f8a5415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:06.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.142+0000 7f8a5a372640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a540cb800 con 0x7f8a540cf160 2023-10-30T12:46:06.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.142+0000 7f8a5a372640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a5415dc30 con 0x7f8a540ca0e0 2023-10-30T12:46:06.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.142+0000 7f8a5a372640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a540cb330 con 0x7f8a541538a0 2023-10-30T12:46:06.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.142+0000 7f8a5890f640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f8a540cf160 0x7f8a54164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43722/0 (socket says 172.21.15.89:43722) 2023-10-30T12:46:06.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.142+0000 7f8a5890f640 1 -- 172.21.15.89:0/872453286 learned_addr learned my addr 172.21.15.89:0/872453286 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:46:06.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.142+0000 7f8a53fff640 1 --2- 172.21.15.89:0/872453286 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a541538a0 0x7f8a5415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:06.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.142+0000 7f8a52ffd640 1 -- 172.21.15.89:0/872453286 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 841264617 0 0) 0x7f8a540cb800 con 0x7f8a540cf160 2023-10-30T12:46:06.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.142+0000 7f8a52ffd640 1 -- 172.21.15.89:0/872453286 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a3c003530 con 0x7f8a540cf160 2023-10-30T12:46:06.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.142+0000 7f8a52ffd640 1 -- 172.21.15.89:0/872453286 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2308094349 0 0) 0x7f8a3c003530 con 0x7f8a540cf160 2023-10-30T12:46:06.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.142+0000 7f8a52ffd640 1 -- 172.21.15.89:0/872453286 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a541538a0 msgr2=0x7f8a5415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:06.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.142+0000 7f8a52ffd640 1 --2- 172.21.15.89:0/872453286 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a541538a0 0x7f8a5415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:06.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.142+0000 7f8a52ffd640 1 -- 172.21.15.89:0/872453286 >> v1:172.21.15.192:6789/0 conn(0x7f8a540ca0e0 legacy=0x7f8a5415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:06.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.142+0000 7f8a52ffd640 1 -- 172.21.15.89:0/872453286 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a540cad70 con 0x7f8a540cf160 2023-10-30T12:46:06.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.142+0000 7f8a52ffd640 1 -- 172.21.15.89:0/872453286 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f8a4c001a00 con 0x7f8a540cf160 2023-10-30T12:46:06.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.142+0000 7f8a52ffd640 1 -- 172.21.15.89:0/872453286 >> v1:172.21.15.89:6789/0 conn(0x7f8a540cf160 legacy=0x7f8a54164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:06.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.142+0000 7f8a52ffd640 1 --2- 172.21.15.89:0/872453286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a3c0039b0 0x7f8a3c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:06.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.142+0000 7f8a52ffd640 1 --2- 172.21.15.89:0/872453286 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a541538a0 0x7f8a3c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:06.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.142+0000 7f8a52ffd640 1 --2- 172.21.15.89:0/872453286 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a3c004d10 0x7f8a3c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:06.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.142+0000 7f8a52ffd640 1 -- 172.21.15.89:0/872453286 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a540cb330 con 0x7f8a3c0039b0 2023-10-30T12:46:06.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.142+0000 7f8a52ffd640 1 -- 172.21.15.89:0/872453286 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3c00bad0 con 0x7f8a3c004d10 2023-10-30T12:46:06.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.142+0000 7f8a52ffd640 1 -- 172.21.15.89:0/872453286 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3c00bc90 con 0x7f8a541538a0 2023-10-30T12:46:06.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.142+0000 7f8a53fff640 1 --2- 172.21.15.89:0/872453286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a3c0039b0 0x7f8a3c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:06.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.146+0000 7f8a5890f640 1 --2- 172.21.15.89:0/872453286 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a3c004d10 0x7f8a3c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:06.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.146+0000 7f8a537fe640 1 --2- 172.21.15.89:0/872453286 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a541538a0 0x7f8a3c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:06.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.146+0000 7f8a53fff640 1 -- 172.21.15.89:0/872453286 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a541538a0 msgr2=0x7f8a3c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:06.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.146+0000 7f8a53fff640 1 --2- 172.21.15.89:0/872453286 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a541538a0 0x7f8a3c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:06.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.146+0000 7f8a53fff640 1 -- 172.21.15.89:0/872453286 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a3c004d10 msgr2=0x7f8a3c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:06.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.146+0000 7f8a53fff640 1 --2- 172.21.15.89:0/872453286 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a3c004d10 0x7f8a3c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:06.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.146+0000 7f8a53fff640 1 -- 172.21.15.89:0/872453286 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a3c003530 con 0x7f8a3c0039b0 2023-10-30T12:46:06.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.146+0000 7f8a5890f640 1 --2- 172.21.15.89:0/872453286 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a3c004d10 0x7f8a3c005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:46:06.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.146+0000 7f8a53fff640 1 --2- 172.21.15.89:0/872453286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a3c0039b0 0x7f8a3c003dd0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f8a44009ab0 tx=0x7f8a4401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b3559b36a53886b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:06.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.146+0000 7f8a52ffd640 1 -- 172.21.15.89:0/872453286 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a440a5510 con 0x7f8a3c0039b0 2023-10-30T12:46:06.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.146+0000 7f8a52ffd640 1 -- 172.21.15.89:0/872453286 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a440a5670 con 0x7f8a3c0039b0 2023-10-30T12:46:06.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.146+0000 7f8a52ffd640 1 -- 172.21.15.89:0/872453286 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a440a5a20 con 0x7f8a3c0039b0 2023-10-30T12:46:06.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.150+0000 7f8a5a372640 1 -- 172.21.15.89:0/872453286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a3c0039b0 msgr2=0x7f8a3c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:06.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.150+0000 7f8a5a372640 1 --2- 172.21.15.89:0/872453286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a3c0039b0 0x7f8a3c003dd0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f8a44009ab0 tx=0x7f8a4401ca30 comp rx=0 tx=0).stop 2023-10-30T12:46:06.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.150+0000 7f8a5890f640 1 -- 172.21.15.89:0/872453286 reap_dead start 2023-10-30T12:46:06.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.150+0000 7f8a5a372640 1 -- 172.21.15.89:0/872453286 shutdown_connections 2023-10-30T12:46:06.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.150+0000 7f8a5a372640 1 -- 172.21.15.89:0/872453286 >> 172.21.15.89:0/872453286 conn(0x7f8a5406bfe0 msgr2=0x7f8a5406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:06.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.150+0000 7f8a5a372640 1 -- 172.21.15.89:0/872453286 shutdown_connections 2023-10-30T12:46:06.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.150+0000 7f8a5a372640 1 -- 172.21.15.89:0/872453286 wait complete. 2023-10-30T12:46:06.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.150+0000 7f8a5a372640 1 Processor -- start 2023-10-30T12:46:06.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.150+0000 7f8a5a372640 1 -- start start 2023-10-30T12:46:06.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.150+0000 7f8a5a372640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a3c0039b0 0x7f8a541e5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:06.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.150+0000 7f8a5a372640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a54150c80 0x7f8a54142c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:06.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.150+0000 7f8a5a372640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a540b4d70 0x7f8a54143180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:06.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.150+0000 7f8a5a372640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a540b31b0 con 0x7f8a3c0039b0 2023-10-30T12:46:06.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.150+0000 7f8a5a372640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a540b3490 con 0x7f8a540b4d70 2023-10-30T12:46:06.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.150+0000 7f8a5a372640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a540b3770 con 0x7f8a54150c80 2023-10-30T12:46:06.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.150+0000 7f8a53fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a3c0039b0 0x7f8a541e5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:06.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.150+0000 7f8a53fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a3c0039b0 0x7f8a541e5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43326/0 (socket says 172.21.15.89:43326) 2023-10-30T12:46:06.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.150+0000 7f8a53fff640 1 -- 172.21.15.89:0/2054040050 learned_addr learned my addr 172.21.15.89:0/2054040050 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:46:06.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.150+0000 7f8a537fe640 1 --2- 172.21.15.89:0/2054040050 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a54150c80 0x7f8a54142c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:06.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.150+0000 7f8a5890f640 1 --2- 172.21.15.89:0/2054040050 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a540b4d70 0x7f8a54143180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:06.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.150+0000 7f8a53fff640 1 -- 172.21.15.89:0/2054040050 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a54150c80 msgr2=0x7f8a54142c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:06.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.150+0000 7f8a53fff640 1 --2- 172.21.15.89:0/2054040050 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a54150c80 0x7f8a54142c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:06.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.150+0000 7f8a53fff640 1 -- 172.21.15.89:0/2054040050 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a540b4d70 msgr2=0x7f8a54143180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:06.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.150+0000 7f8a53fff640 1 --2- 172.21.15.89:0/2054040050 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a540b4d70 0x7f8a54143180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:06.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.150+0000 7f8a53fff640 1 -- 172.21.15.89:0/2054040050 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a540b3a50 con 0x7f8a3c0039b0 2023-10-30T12:46:06.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.150+0000 7f8a5890f640 1 --2- 172.21.15.89:0/2054040050 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a540b4d70 0x7f8a54143180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:46:06.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.154+0000 7f8a53fff640 1 --2- 172.21.15.89:0/2054040050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a3c0039b0 0x7f8a541e5e70 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f8a440b0040 tx=0x7f8a3c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:06.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.154+0000 7f8a50ff9640 1 -- 172.21.15.89:0/2054040050 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a44004220 con 0x7f8a3c0039b0 2023-10-30T12:46:06.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.154+0000 7f8a50ff9640 1 -- 172.21.15.89:0/2054040050 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a44004380 con 0x7f8a3c0039b0 2023-10-30T12:46:06.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.154+0000 7f8a5a372640 1 -- 172.21.15.89:0/2054040050 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a540b3d30 con 0x7f8a3c0039b0 2023-10-30T12:46:06.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.154+0000 7f8a50ff9640 1 -- 172.21.15.89:0/2054040050 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a44003860 con 0x7f8a3c0039b0 2023-10-30T12:46:06.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.154+0000 7f8a5a372640 1 -- 172.21.15.89:0/2054040050 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a541e6a10 con 0x7f8a3c0039b0 2023-10-30T12:46:06.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.154+0000 7f8a5a372640 1 -- 172.21.15.89:0/2054040050 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a541f2d70 con 0x7f8a3c0039b0 2023-10-30T12:46:06.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.154+0000 7f8a50ff9640 1 -- 172.21.15.89:0/2054040050 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8a440b4040 con 0x7f8a3c0039b0 2023-10-30T12:46:06.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.154+0000 7f8a50ff9640 1 --2- 172.21.15.89:0/2054040050 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8a400664d0 0x7f8a40068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:06.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.154+0000 7f8a50ff9640 1 -- 172.21.15.89:0/2054040050 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6492+0+0 (secure 0 0 0) 0x7f8a4411da70 con 0x7f8a3c0039b0 2023-10-30T12:46:06.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.154+0000 7f8a537fe640 1 --2- 172.21.15.89:0/2054040050 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8a400664d0 0x7f8a40068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:06.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.158+0000 7f8a537fe640 1 --2- 172.21.15.89:0/2054040050 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8a400664d0 0x7f8a40068990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8a54148b90 tx=0x7f8a48009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:06.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.158+0000 7f8a50ff9640 1 -- 172.21.15.89:0/2054040050 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8a440ead20 con 0x7f8a3c0039b0 2023-10-30T12:46:06.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.338+0000 7f8a5a372640 1 -- 172.21.15.89:0/2054040050 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a540b3ec0 con 0x7f8a3c0039b0 2023-10-30T12:46:06.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.342+0000 7f8a50ff9640 1 -- 172.21.15.89:0/2054040050 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5744 (secure 0 0 0) 0x7f8a440ee340 con 0x7f8a3c0039b0 2023-10-30T12:46:06.339 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:46:06.340 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:46:03.477376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5450,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":1,"incarnation":83,"state":"up:reconnect","state_seq":32,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:46:06.340 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 85 2023-10-30T12:46:06.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.342+0000 7f8a5a372640 1 -- 172.21.15.89:0/2054040050 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8a400664d0 msgr2=0x7f8a40068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:06.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.342+0000 7f8a5a372640 1 --2- 172.21.15.89:0/2054040050 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8a400664d0 0x7f8a40068990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8a54148b90 tx=0x7f8a48009040 comp rx=0 tx=0).stop 2023-10-30T12:46:06.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.342+0000 7f8a5a372640 1 -- 172.21.15.89:0/2054040050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a3c0039b0 msgr2=0x7f8a541e5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:06.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.342+0000 7f8a5a372640 1 --2- 172.21.15.89:0/2054040050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a3c0039b0 0x7f8a541e5e70 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f8a440b0040 tx=0x7f8a3c005800 comp rx=0 tx=0).stop 2023-10-30T12:46:06.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.342+0000 7f8a5a372640 1 -- 172.21.15.89:0/2054040050 shutdown_connections 2023-10-30T12:46:06.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.342+0000 7f8a5a372640 1 --2- 172.21.15.89:0/2054040050 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8a400664d0 0x7f8a40068990 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:06.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.342+0000 7f8a5a372640 1 --2- 172.21.15.89:0/2054040050 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a540b4d70 0x7f8a54143180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:06.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.342+0000 7f8a5a372640 1 --2- 172.21.15.89:0/2054040050 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a54150c80 0x7f8a54142c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:06.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.342+0000 7f8a5a372640 1 --2- 172.21.15.89:0/2054040050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a3c0039b0 0x7f8a541e5e70 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:06.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.342+0000 7f8a5a372640 1 -- 172.21.15.89:0/2054040050 >> 172.21.15.89:0/2054040050 conn(0x7f8a5406bfe0 msgr2=0x7f8a54151800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:06.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.342+0000 7f8a5a372640 1 -- 172.21.15.89:0/2054040050 shutdown_connections 2023-10-30T12:46:06.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:06.342+0000 7f8a5a372640 1 -- 172.21.15.89:0/2054040050 wait complete. 2023-10-30T12:46:06.361 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:46:08.361 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:46:08.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.510+0000 7f5821f3e640 1 Processor -- start 2023-10-30T12:46:08.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.510+0000 7f5821f3e640 1 -- start start 2023-10-30T12:46:08.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.514+0000 7f5821f3e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f581c0c6b80 0x7f581c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:08.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.514+0000 7f5821f3e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f581c0c6f60 0x7f581c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:08.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.514+0000 7f5821f3e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f581c0c0a50 con 0x7f581c0c8430 2023-10-30T12:46:08.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.514+0000 7f5821f3e640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f581c06f6e0 con 0x7f581c0c6f60 2023-10-30T12:46:08.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.514+0000 7f5821f3e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f581c06f9c0 con 0x7f581c0c6b80 2023-10-30T12:46:08.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.514+0000 7f581affd640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f581c0c6f60 0x7f581c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:08.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.514+0000 7f581b7fe640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f581c0c6b80 0x7f581c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:08.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.514+0000 7f581affd640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f581c0c6f60 0x7f581c0c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:41598/0 (socket says 172.21.15.89:41598) 2023-10-30T12:46:08.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.514+0000 7f581affd640 1 -- 172.21.15.89:0/3538505954 learned_addr learned my addr 172.21.15.89:0/3538505954 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:46:08.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.514+0000 7f581b7fe640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f581c0c6b80 0x7f581c0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:40468/0 (socket says 172.21.15.89:40468) 2023-10-30T12:46:08.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.514+0000 7f581b7fe640 1 -- 172.21.15.89:0/3538505954 >> v2:172.21.15.192:3300/0 conn(0x7f581c0c6f60 msgr2=0x7f581c0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:08.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.514+0000 7f581b7fe640 1 --2- 172.21.15.89:0/3538505954 >> v2:172.21.15.192:3300/0 conn(0x7f581c0c6f60 0x7f581c0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:08.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.514+0000 7f581b7fe640 1 -- 172.21.15.89:0/3538505954 >> v1:172.21.15.192:6789/0 conn(0x7f581c0c8430 legacy=0x7f581c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:08.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.514+0000 7f581b7fe640 1 -- 172.21.15.89:0/3538505954 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f581c06fca0 con 0x7f581c0c6b80 2023-10-30T12:46:08.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.514+0000 7f581affd640 1 --2- 172.21.15.89:0/3538505954 >> v2:172.21.15.192:3300/0 conn(0x7f581c0c6f60 0x7f581c0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:46:08.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.518+0000 7f581b7fe640 1 --2- 172.21.15.89:0/3538505954 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f581c0c6b80 0x7f581c0bfec0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f580c0097c0 tx=0x7f580c01c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5d4955b57e4cd3f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:08.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.518+0000 7f581a7fc640 1 -- 172.21.15.89:0/3538505954 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580c0ad640 con 0x7f581c0c6b80 2023-10-30T12:46:08.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.518+0000 7f581a7fc640 1 -- 172.21.15.89:0/3538505954 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f580c0ad7a0 con 0x7f581c0c6b80 2023-10-30T12:46:08.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.518+0000 7f581a7fc640 1 -- 172.21.15.89:0/3538505954 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580c0adb30 con 0x7f581c0c6b80 2023-10-30T12:46:08.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.518+0000 7f5821f3e640 1 -- 172.21.15.89:0/3538505954 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f581c0c6b80 msgr2=0x7f581c0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:08.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.518+0000 7f5821f3e640 1 --2- 172.21.15.89:0/3538505954 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f581c0c6b80 0x7f581c0bfec0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f580c0097c0 tx=0x7f580c01c480 comp rx=0 tx=0).stop 2023-10-30T12:46:08.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.518+0000 7f5821f3e640 1 -- 172.21.15.89:0/3538505954 shutdown_connections 2023-10-30T12:46:08.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.518+0000 7f5821f3e640 1 --2- 172.21.15.89:0/3538505954 >> v2:172.21.15.192:3300/0 conn(0x7f581c0c6f60 0x7f581c0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:08.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.518+0000 7f5821f3e640 1 --2- 172.21.15.89:0/3538505954 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f581c0c6b80 0x7f581c0bfec0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:08.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.518+0000 7f5821f3e640 1 -- 172.21.15.89:0/3538505954 >> 172.21.15.89:0/3538505954 conn(0x7f581c0c7910 msgr2=0x7f581c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:08.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.518+0000 7f5821f3e640 1 -- 172.21.15.89:0/3538505954 shutdown_connections 2023-10-30T12:46:08.551 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.518+0000 7f5821f3e640 1 -- 172.21.15.89:0/3538505954 wait complete. 2023-10-30T12:46:08.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.522+0000 7f5821f3e640 1 Processor -- start 2023-10-30T12:46:08.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.522+0000 7f5821f3e640 1 -- start start 2023-10-30T12:46:08.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.522+0000 7f5821f3e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f581c0c6b80 0x7f581c0c1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:08.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.522+0000 7f5821f3e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f581c0c6f60 0x7f581c158d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:08.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.522+0000 7f5821f3e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f581c0c8430 0x7f581c1f2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:08.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.522+0000 7f5821f3e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f581c1f22a0 con 0x7f581c0c6b80 2023-10-30T12:46:08.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.522+0000 7f5821f3e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f581c155ed0 con 0x7f581c0c8430 2023-10-30T12:46:08.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.522+0000 7f5821f3e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f581c156450 con 0x7f581c0c6f60 2023-10-30T12:46:08.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.522+0000 7f581b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f581c0c6b80 0x7f581c0c1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:08.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.522+0000 7f581b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f581c0c6b80 0x7f581c0c1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43342/0 (socket says 172.21.15.89:43342) 2023-10-30T12:46:08.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.522+0000 7f581b7fe640 1 -- 172.21.15.89:0/797850883 learned_addr learned my addr 172.21.15.89:0/797850883 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:46:08.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.522+0000 7f581affd640 1 --2- 172.21.15.89:0/797850883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f581c0c6f60 0x7f581c158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:08.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.522+0000 7f581bfff640 1 --2- 172.21.15.89:0/797850883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f581c0c8430 0x7f581c1f2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:08.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.522+0000 7f581b7fe640 1 -- 172.21.15.89:0/797850883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f581c0c6f60 msgr2=0x7f581c158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:08.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.522+0000 7f581b7fe640 1 --2- 172.21.15.89:0/797850883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f581c0c6f60 0x7f581c158d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:08.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.522+0000 7f581b7fe640 1 -- 172.21.15.89:0/797850883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f581c0c8430 msgr2=0x7f581c1f2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:08.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.522+0000 7f581b7fe640 1 --2- 172.21.15.89:0/797850883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f581c0c8430 0x7f581c1f2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:08.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.522+0000 7f581b7fe640 1 -- 172.21.15.89:0/797850883 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58100094d0 con 0x7f581c0c6b80 2023-10-30T12:46:08.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.522+0000 7f581bfff640 1 --2- 172.21.15.89:0/797850883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f581c0c8430 0x7f581c1f2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:46:08.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.522+0000 7f581b7fe640 1 --2- 172.21.15.89:0/797850883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f581c0c6b80 0x7f581c0c1570 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f580c01a0b0 tx=0x7f580c019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:08.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.522+0000 7f5803fff640 1 -- 172.21.15.89:0/797850883 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580c0b6820 con 0x7f581c0c6b80 2023-10-30T12:46:08.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.522+0000 7f5803fff640 1 -- 172.21.15.89:0/797850883 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f580c0a6040 con 0x7f581c0c6b80 2023-10-30T12:46:08.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.522+0000 7f5803fff640 1 -- 172.21.15.89:0/797850883 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580c0b6be0 con 0x7f581c0c6b80 2023-10-30T12:46:08.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.522+0000 7f5821f3e640 1 -- 172.21.15.89:0/797850883 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f580c009420 con 0x7f581c0c6b80 2023-10-30T12:46:08.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.522+0000 7f5821f3e640 1 -- 172.21.15.89:0/797850883 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f581c156170 con 0x7f581c0c6b80 2023-10-30T12:46:08.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.522+0000 7f5821f3e640 1 -- 172.21.15.89:0/797850883 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f581c1f2100 con 0x7f581c0c6b80 2023-10-30T12:46:08.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.526+0000 7f5803fff640 1 -- 172.21.15.89:0/797850883 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f580c0b4040 con 0x7f581c0c6b80 2023-10-30T12:46:08.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.526+0000 7f5803fff640 1 --2- 172.21.15.89:0/797850883 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f58080664a0 0x7f5808068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:08.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.526+0000 7f5803fff640 1 -- 172.21.15.89:0/797850883 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6492+0+0 (secure 0 0 0) 0x7f580c11e5d0 con 0x7f581c0c6b80 2023-10-30T12:46:08.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.526+0000 7f581affd640 1 --2- 172.21.15.89:0/797850883 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f58080664a0 0x7f5808068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:08.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.526+0000 7f581affd640 1 --2- 172.21.15.89:0/797850883 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f58080664a0 0x7f5808068960 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f581c0c0920 tx=0x7f5810008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:08.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.530+0000 7f5803fff640 1 -- 172.21.15.89:0/797850883 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f581c1f2100 con 0x7f581c0c6b80 2023-10-30T12:46:08.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.710+0000 7f5821f3e640 1 -- 172.21.15.89:0/797850883 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f581c1f2100 con 0x7f581c0c6b80 2023-10-30T12:46:08.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.710+0000 7f5803fff640 1 -- 172.21.15.89:0/797850883 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5744 (secure 0 0 0) 0x7f581c1f2100 con 0x7f581c0c6b80 2023-10-30T12:46:08.708 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:46:08.708 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:46:03.477376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5450,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":1,"incarnation":83,"state":"up:reconnect","state_seq":32,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:46:08.709 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 85 2023-10-30T12:46:08.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.710+0000 7f5821f3e640 1 -- 172.21.15.89:0/797850883 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f58080664a0 msgr2=0x7f5808068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:08.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.710+0000 7f5821f3e640 1 --2- 172.21.15.89:0/797850883 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f58080664a0 0x7f5808068960 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f581c0c0920 tx=0x7f5810008040 comp rx=0 tx=0).stop 2023-10-30T12:46:08.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.710+0000 7f5821f3e640 1 -- 172.21.15.89:0/797850883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f581c0c6b80 msgr2=0x7f581c0c1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:08.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.710+0000 7f5821f3e640 1 --2- 172.21.15.89:0/797850883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f581c0c6b80 0x7f581c0c1570 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f580c01a0b0 tx=0x7f580c019220 comp rx=0 tx=0).stop 2023-10-30T12:46:08.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.710+0000 7f5821f3e640 1 -- 172.21.15.89:0/797850883 shutdown_connections 2023-10-30T12:46:08.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.710+0000 7f5821f3e640 1 --2- 172.21.15.89:0/797850883 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f58080664a0 0x7f5808068960 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:08.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.710+0000 7f5821f3e640 1 --2- 172.21.15.89:0/797850883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f581c0c8430 0x7f581c1f2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:08.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.710+0000 7f5821f3e640 1 --2- 172.21.15.89:0/797850883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f581c0c6f60 0x7f581c158d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:08.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.710+0000 7f5821f3e640 1 --2- 172.21.15.89:0/797850883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f581c0c6b80 0x7f581c0c1570 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:08.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.710+0000 7f5821f3e640 1 -- 172.21.15.89:0/797850883 >> 172.21.15.89:0/797850883 conn(0x7f581c0c7910 msgr2=0x7f581c0bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:08.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.710+0000 7f5821f3e640 1 -- 172.21.15.89:0/797850883 shutdown_connections 2023-10-30T12:46:08.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:08.710+0000 7f5821f3e640 1 -- 172.21.15.89:0/797850883 wait complete. 2023-10-30T12:46:08.728 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:46:10.730 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:46:10.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.882+0000 7f78554c9640 1 Processor -- start 2023-10-30T12:46:10.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.882+0000 7f78554c9640 1 -- start start 2023-10-30T12:46:10.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.882+0000 7f78554c9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78501538a0 0x7f785015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:10.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.882+0000 7f78554c9640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f78500ca0e0 0x7f785015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:10.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.882+0000 7f78554c9640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f78500cf160 0x7f78500cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:10.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.882+0000 7f78554c9640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78500cb800 con 0x7f78500ca0e0 2023-10-30T12:46:10.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.882+0000 7f78554c9640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f785015dc30 con 0x7f78500cf160 2023-10-30T12:46:10.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.882+0000 7f78554c9640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78500cb330 con 0x7f78501538a0 2023-10-30T12:46:10.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.882+0000 7f78467fc640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f78500ca0e0 0x7f785015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:10.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.882+0000 7f78467fc640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f78500ca0e0 0x7f785015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57262/0 (socket says 172.21.15.89:57262) 2023-10-30T12:46:10.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.882+0000 7f78467fc640 1 -- 172.21.15.89:0/3038105182 learned_addr learned my addr 172.21.15.89:0/3038105182 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:46:10.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.882+0000 7f784effd640 1 --2- 172.21.15.89:0/3038105182 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78501538a0 0x7f785015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:10.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.882+0000 7f784f7fe640 1 --2- 172.21.15.89:0/3038105182 >> v2:172.21.15.192:3300/0 conn(0x7f78500cf160 0x7f78500cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:10.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.882+0000 7f78467fc640 1 -- 172.21.15.89:0/3038105182 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78501538a0 msgr2=0x7f785015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:10.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.882+0000 7f78467fc640 1 --2- 172.21.15.89:0/3038105182 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78501538a0 0x7f785015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:10.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.882+0000 7f78467fc640 1 -- 172.21.15.89:0/3038105182 >> v2:172.21.15.192:3300/0 conn(0x7f78500cf160 msgr2=0x7f78500cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:10.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.882+0000 7f78467fc640 1 --2- 172.21.15.89:0/3038105182 >> v2:172.21.15.192:3300/0 conn(0x7f78500cf160 0x7f78500cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:10.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.882+0000 7f78467fc640 1 -- 172.21.15.89:0/3038105182 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78500cad70 con 0x7f78500ca0e0 2023-10-30T12:46:10.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.886+0000 7f78467fc640 1 --2- 172.21.15.89:0/3038105182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78500ca0e0 0x7f785015a090 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f785006cb00 tx=0x7f783801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c10de243e8cf4886 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:10.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.886+0000 7f784e7fc640 1 -- 172.21.15.89:0/3038105182 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78380af400 con 0x7f78500ca0e0 2023-10-30T12:46:10.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.886+0000 7f784e7fc640 1 -- 172.21.15.89:0/3038105182 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78380ab050 con 0x7f78500ca0e0 2023-10-30T12:46:10.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.886+0000 7f784e7fc640 1 -- 172.21.15.89:0/3038105182 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78380af780 con 0x7f78500ca0e0 2023-10-30T12:46:10.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.886+0000 7f78554c9640 1 -- 172.21.15.89:0/3038105182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78500ca0e0 msgr2=0x7f785015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:10.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.886+0000 7f78554c9640 1 --2- 172.21.15.89:0/3038105182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78500ca0e0 0x7f785015a090 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f785006cb00 tx=0x7f783801f1c0 comp rx=0 tx=0).stop 2023-10-30T12:46:10.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.886+0000 7f78554c9640 1 -- 172.21.15.89:0/3038105182 shutdown_connections 2023-10-30T12:46:10.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.886+0000 7f78554c9640 1 --2- 172.21.15.89:0/3038105182 >> v2:172.21.15.192:3300/0 conn(0x7f78500cf160 0x7f78500cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:10.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.886+0000 7f78554c9640 1 --2- 172.21.15.89:0/3038105182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78500ca0e0 0x7f785015a090 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:10.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.886+0000 7f78554c9640 1 --2- 172.21.15.89:0/3038105182 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78501538a0 0x7f785015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:10.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.886+0000 7f78554c9640 1 -- 172.21.15.89:0/3038105182 >> 172.21.15.89:0/3038105182 conn(0x7f785006bfe0 msgr2=0x7f785006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:10.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.886+0000 7f78554c9640 1 -- 172.21.15.89:0/3038105182 shutdown_connections 2023-10-30T12:46:10.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.886+0000 7f78554c9640 1 -- 172.21.15.89:0/3038105182 wait complete. 2023-10-30T12:46:10.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.886+0000 7f78554c9640 1 Processor -- start 2023-10-30T12:46:10.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.886+0000 7f78554c9640 1 -- start start 2023-10-30T12:46:10.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.890+0000 7f78554c9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78500ca0e0 0x7f785015d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:10.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.890+0000 7f78554c9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78500cf160 0x7f785014a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:10.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.890+0000 7f78554c9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78501538a0 0x7f78501f33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:10.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.890+0000 7f78554c9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78501f2fd0 con 0x7f78500cf160 2023-10-30T12:46:10.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.890+0000 7f78554c9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785015ad70 con 0x7f78501538a0 2023-10-30T12:46:10.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.890+0000 7f78554c9640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f785014bab0 con 0x7f78500ca0e0 2023-10-30T12:46:10.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.890+0000 7f78467fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78500cf160 0x7f785014a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:10.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.890+0000 7f78467fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78500cf160 0x7f785014a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57274/0 (socket says 172.21.15.89:57274) 2023-10-30T12:46:10.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.890+0000 7f78467fc640 1 -- 172.21.15.89:0/1279581398 learned_addr learned my addr 172.21.15.89:0/1279581398 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:46:10.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.890+0000 7f784effd640 1 --2- 172.21.15.89:0/1279581398 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78500ca0e0 0x7f785015d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:10.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.890+0000 7f784f7fe640 1 --2- 172.21.15.89:0/1279581398 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78501538a0 0x7f78501f33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:10.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.890+0000 7f78467fc640 1 -- 172.21.15.89:0/1279581398 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78500ca0e0 msgr2=0x7f785015d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:10.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.890+0000 7f78467fc640 1 --2- 172.21.15.89:0/1279581398 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78500ca0e0 0x7f785015d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:10.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.890+0000 7f78467fc640 1 -- 172.21.15.89:0/1279581398 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78501538a0 msgr2=0x7f78501f33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:10.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.890+0000 7f78467fc640 1 --2- 172.21.15.89:0/1279581398 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78501538a0 0x7f78501f33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:10.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.890+0000 7f78467fc640 1 -- 172.21.15.89:0/1279581398 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7838009b30 con 0x7f78500cf160 2023-10-30T12:46:10.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.890+0000 7f78467fc640 1 --2- 172.21.15.89:0/1279581398 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78500cf160 0x7f785014a6f0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f78380b2040 tx=0x7f783801bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:10.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.890+0000 7f7847fff640 1 -- 172.21.15.89:0/1279581398 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78380afe80 con 0x7f78500cf160 2023-10-30T12:46:10.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.890+0000 7f7847fff640 1 -- 172.21.15.89:0/1279581398 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7838009510 con 0x7f78500cf160 2023-10-30T12:46:10.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.890+0000 7f7847fff640 1 -- 172.21.15.89:0/1279581398 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78380ba680 con 0x7f78500cf160 2023-10-30T12:46:10.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.890+0000 7f78554c9640 1 -- 172.21.15.89:0/1279581398 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78500ce130 con 0x7f78500cf160 2023-10-30T12:46:10.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.890+0000 7f78554c9640 1 -- 172.21.15.89:0/1279581398 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78500ce380 con 0x7f78500cf160 2023-10-30T12:46:10.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.890+0000 7f78554c9640 1 -- 172.21.15.89:0/1279581398 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78500b2a50 con 0x7f78500cf160 2023-10-30T12:46:10.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.894+0000 7f7847fff640 1 -- 172.21.15.89:0/1279581398 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f78380b6040 con 0x7f78500cf160 2023-10-30T12:46:10.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.894+0000 7f7847fff640 1 --2- 172.21.15.89:0/1279581398 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f78300664d0 0x7f7830068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:10.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.894+0000 7f7847fff640 1 -- 172.21.15.89:0/1279581398 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6492+0+0 (secure 0 0 0) 0x7f7838122410 con 0x7f78500cf160 2023-10-30T12:46:10.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.894+0000 7f784effd640 1 --2- 172.21.15.89:0/1279581398 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f78300664d0 0x7f7830068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:10.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.894+0000 7f784effd640 1 --2- 172.21.15.89:0/1279581398 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f78300664d0 0x7f7830068990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f78400079c0 tx=0x7f7840008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:10.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:10.898+0000 7f7847fff640 1 -- 172.21.15.89:0/1279581398 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f78380ef5c0 con 0x7f78500cf160 2023-10-30T12:46:11.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:11.082+0000 7f78554c9640 1 -- 172.21.15.89:0/1279581398 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78501f3110 con 0x7f78500cf160 2023-10-30T12:46:11.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:11.082+0000 7f7847fff640 1 -- 172.21.15.89:0/1279581398 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f78501f3110 con 0x7f78500cf160 2023-10-30T12:46:11.081 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:46:11.081 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:46:09.375531+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5450,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":1,"incarnation":83,"state":"up:rejoin","state_seq":34,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:46:11.082 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 86 2023-10-30T12:46:11.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:11.086+0000 7f78554c9640 1 -- 172.21.15.89:0/1279581398 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f78300664d0 msgr2=0x7f7830068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:11.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:11.086+0000 7f78554c9640 1 --2- 172.21.15.89:0/1279581398 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f78300664d0 0x7f7830068990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f78400079c0 tx=0x7f7840008040 comp rx=0 tx=0).stop 2023-10-30T12:46:11.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:11.086+0000 7f78554c9640 1 -- 172.21.15.89:0/1279581398 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78500cf160 msgr2=0x7f785014a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:11.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:11.086+0000 7f78554c9640 1 --2- 172.21.15.89:0/1279581398 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78500cf160 0x7f785014a6f0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f78380b2040 tx=0x7f783801bf60 comp rx=0 tx=0).stop 2023-10-30T12:46:11.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:11.086+0000 7f78554c9640 1 -- 172.21.15.89:0/1279581398 shutdown_connections 2023-10-30T12:46:11.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:11.086+0000 7f78554c9640 1 --2- 172.21.15.89:0/1279581398 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f78300664d0 0x7f7830068990 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:11.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:11.086+0000 7f78554c9640 1 --2- 172.21.15.89:0/1279581398 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f78501538a0 0x7f78501f33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:11.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:11.086+0000 7f78554c9640 1 --2- 172.21.15.89:0/1279581398 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78500cf160 0x7f785014a6f0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:11.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:11.086+0000 7f78554c9640 1 --2- 172.21.15.89:0/1279581398 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f78500ca0e0 0x7f785015d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:11.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:11.086+0000 7f78554c9640 1 -- 172.21.15.89:0/1279581398 >> 172.21.15.89:0/1279581398 conn(0x7f785006bfe0 msgr2=0x7f78501483a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:11.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:11.086+0000 7f78554c9640 1 -- 172.21.15.89:0/1279581398 shutdown_connections 2023-10-30T12:46:11.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:11.086+0000 7f78554c9640 1 -- 172.21.15.89:0/1279581398 wait complete. 2023-10-30T12:46:11.102 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:46:13.103 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:46:13.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.254+0000 7f6bf76bb640 1 Processor -- start 2023-10-30T12:46:13.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.254+0000 7f6bf76bb640 1 -- start start 2023-10-30T12:46:13.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.254+0000 7f6bf76bb640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6bf0153b10 0x7f6bf015ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:13.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.254+0000 7f6bf76bb640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bf00c5e70 0x7f6bf01641f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:13.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.254+0000 7f6bf76bb640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf00c74f0 con 0x7f6bf00b0dc0 2023-10-30T12:46:13.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.254+0000 7f6bf76bb640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bf00c1f20 con 0x7f6bf0153b10 2023-10-30T12:46:13.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.254+0000 7f6bf76bb640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf00c77b0 con 0x7f6bf00c5e70 2023-10-30T12:46:13.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.254+0000 7f6bf5c58640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bf00c5e70 0x7f6bf01641f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:13.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.254+0000 7f6bf5457640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6bf0153b10 0x7f6bf015ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:13.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.254+0000 7f6bf5c58640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bf00c5e70 0x7f6bf01641f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:37772/0 (socket says 172.21.15.89:37772) 2023-10-30T12:46:13.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.254+0000 7f6bf5c58640 1 -- 172.21.15.89:0/901354004 learned_addr learned my addr 172.21.15.89:0/901354004 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:46:13.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.254+0000 7f6bf5c58640 1 -- 172.21.15.89:0/901354004 >> v2:172.21.15.192:3300/0 conn(0x7f6bf0153b10 msgr2=0x7f6bf015ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:13.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.254+0000 7f6bf5c58640 1 --2- 172.21.15.89:0/901354004 >> v2:172.21.15.192:3300/0 conn(0x7f6bf0153b10 0x7f6bf015ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:13.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.254+0000 7f6bf5c58640 1 -- 172.21.15.89:0/901354004 >> v1:172.21.15.192:6789/0 conn(0x7f6bf00b0dc0 legacy=0x7f6bf00c7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:13.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.254+0000 7f6bf5c58640 1 -- 172.21.15.89:0/901354004 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf0159b80 con 0x7f6bf00c5e70 2023-10-30T12:46:13.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.258+0000 7f6bf5c58640 1 --2- 172.21.15.89:0/901354004 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bf00c5e70 0x7f6bf01641f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6be8009e50 tx=0x7f6be801f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a266e3399806cee5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:13.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.258+0000 7f6be7fff640 1 -- 172.21.15.89:0/901354004 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be80af400 con 0x7f6bf00c5e70 2023-10-30T12:46:13.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.258+0000 7f6be7fff640 1 -- 172.21.15.89:0/901354004 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6be80ab050 con 0x7f6bf00c5e70 2023-10-30T12:46:13.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.258+0000 7f6be7fff640 1 -- 172.21.15.89:0/901354004 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be80af780 con 0x7f6bf00c5e70 2023-10-30T12:46:13.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.258+0000 7f6bf76bb640 1 -- 172.21.15.89:0/901354004 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bf00c5e70 msgr2=0x7f6bf01641f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:13.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.258+0000 7f6bf76bb640 1 --2- 172.21.15.89:0/901354004 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bf00c5e70 0x7f6bf01641f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6be8009e50 tx=0x7f6be801f1f0 comp rx=0 tx=0).stop 2023-10-30T12:46:13.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.258+0000 7f6bf76bb640 1 -- 172.21.15.89:0/901354004 shutdown_connections 2023-10-30T12:46:13.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.258+0000 7f6bf76bb640 1 --2- 172.21.15.89:0/901354004 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bf00c5e70 0x7f6bf01641f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:13.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.258+0000 7f6bf76bb640 1 --2- 172.21.15.89:0/901354004 >> v2:172.21.15.192:3300/0 conn(0x7f6bf0153b10 0x7f6bf015ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:13.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.258+0000 7f6bf76bb640 1 -- 172.21.15.89:0/901354004 >> 172.21.15.89:0/901354004 conn(0x7f6bf006d770 msgr2=0x7f6bf006db80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:13.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.258+0000 7f6bf76bb640 1 -- 172.21.15.89:0/901354004 shutdown_connections 2023-10-30T12:46:13.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.258+0000 7f6bf76bb640 1 -- 172.21.15.89:0/901354004 wait complete. 2023-10-30T12:46:13.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.262+0000 7f6bf76bb640 1 Processor -- start 2023-10-30T12:46:13.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.262+0000 7f6bf76bb640 1 -- start start 2023-10-30T12:46:13.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.262+0000 7f6bf76bb640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bf00b0dc0 0x7f6bf014f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:13.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.262+0000 7f6bf76bb640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bf00c5e70 0x7f6bf015a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:13.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.262+0000 7f6bf76bb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bf0153b10 0x7f6bf015ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:13.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.262+0000 7f6bf76bb640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf0141d40 con 0x7f6bf0153b10 2023-10-30T12:46:13.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.262+0000 7f6bf76bb640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf01477c0 con 0x7f6bf00c5e70 2023-10-30T12:46:13.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.262+0000 7f6bf76bb640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf015d290 con 0x7f6bf00b0dc0 2023-10-30T12:46:13.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.262+0000 7f6bf5c58640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bf0153b10 0x7f6bf015ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:13.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.262+0000 7f6bf5c58640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bf0153b10 0x7f6bf015ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57282/0 (socket says 172.21.15.89:57282) 2023-10-30T12:46:13.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.262+0000 7f6bf5c58640 1 -- 172.21.15.89:0/55252095 learned_addr learned my addr 172.21.15.89:0/55252095 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:46:13.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.262+0000 7f6bf5457640 1 --2- 172.21.15.89:0/55252095 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bf00b0dc0 0x7f6bf014f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:13.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.262+0000 7f6bf4c56640 1 --2- 172.21.15.89:0/55252095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bf00c5e70 0x7f6bf015a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:13.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.262+0000 7f6bf5c58640 1 -- 172.21.15.89:0/55252095 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bf00b0dc0 msgr2=0x7f6bf014f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:13.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.262+0000 7f6bf5c58640 1 --2- 172.21.15.89:0/55252095 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bf00b0dc0 0x7f6bf014f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:13.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.262+0000 7f6bf5c58640 1 -- 172.21.15.89:0/55252095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bf00c5e70 msgr2=0x7f6bf015a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:13.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.262+0000 7f6bf5c58640 1 --2- 172.21.15.89:0/55252095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bf00c5e70 0x7f6bf015a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:13.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.262+0000 7f6bf5c58640 1 -- 172.21.15.89:0/55252095 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be8009b30 con 0x7f6bf0153b10 2023-10-30T12:46:13.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.262+0000 7f6bf5c58640 1 --2- 172.21.15.89:0/55252095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bf0153b10 0x7f6bf015ae40 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f6be801ce20 tx=0x7f6be801bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:13.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.262+0000 7f6be5ffb640 1 -- 172.21.15.89:0/55252095 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be80afd30 con 0x7f6bf0153b10 2023-10-30T12:46:13.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.262+0000 7f6be5ffb640 1 -- 172.21.15.89:0/55252095 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6be80afe90 con 0x7f6bf0153b10 2023-10-30T12:46:13.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.262+0000 7f6be5ffb640 1 -- 172.21.15.89:0/55252095 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be80ba660 con 0x7f6bf0153b10 2023-10-30T12:46:13.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.262+0000 7f6bf76bb640 1 -- 172.21.15.89:0/55252095 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bf015c680 con 0x7f6bf0153b10 2023-10-30T12:46:13.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.262+0000 7f6bf76bb640 1 -- 172.21.15.89:0/55252095 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bf015cc40 con 0x7f6bf0153b10 2023-10-30T12:46:13.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.266+0000 7f6bf76bb640 1 -- 172.21.15.89:0/55252095 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bf0074250 con 0x7f6bf0153b10 2023-10-30T12:46:13.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.266+0000 7f6be5ffb640 1 -- 172.21.15.89:0/55252095 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6be80b6040 con 0x7f6bf0153b10 2023-10-30T12:46:13.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.266+0000 7f6be5ffb640 1 --2- 172.21.15.89:0/55252095 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6bd40664d0 0x7f6bd4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:13.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.266+0000 7f6be5ffb640 1 -- 172.21.15.89:0/55252095 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6492+0+0 (secure 0 0 0) 0x7f6be8122220 con 0x7f6bf0153b10 2023-10-30T12:46:13.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.266+0000 7f6bf5457640 1 --2- 172.21.15.89:0/55252095 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6bd40664d0 0x7f6bd4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:13.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.270+0000 7f6bf5457640 1 --2- 172.21.15.89:0/55252095 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6bd40664d0 0x7f6bd4068990 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f6bec009ac0 tx=0x7f6bec009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:13.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.270+0000 7f6be5ffb640 1 -- 172.21.15.89:0/55252095 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6be80ef580 con 0x7f6bf0153b10 2023-10-30T12:46:13.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.458+0000 7f6bf76bb640 1 -- 172.21.15.89:0/55252095 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6bf004e170 con 0x7f6bf0153b10 2023-10-30T12:46:13.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.458+0000 7f6be5ffb640 1 -- 172.21.15.89:0/55252095 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6be8009510 con 0x7f6bf0153b10 2023-10-30T12:46:13.456 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:46:13.456 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:46:09.375531+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5450,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":1,"incarnation":83,"state":"up:rejoin","state_seq":34,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:46:13.457 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 86 2023-10-30T12:46:13.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.458+0000 7f6bf76bb640 1 -- 172.21.15.89:0/55252095 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6bd40664d0 msgr2=0x7f6bd4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:13.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.458+0000 7f6bf76bb640 1 --2- 172.21.15.89:0/55252095 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6bd40664d0 0x7f6bd4068990 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f6bec009ac0 tx=0x7f6bec009190 comp rx=0 tx=0).stop 2023-10-30T12:46:13.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.458+0000 7f6bf76bb640 1 -- 172.21.15.89:0/55252095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bf0153b10 msgr2=0x7f6bf015ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:13.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.458+0000 7f6bf76bb640 1 --2- 172.21.15.89:0/55252095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bf0153b10 0x7f6bf015ae40 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f6be801ce20 tx=0x7f6be801bf90 comp rx=0 tx=0).stop 2023-10-30T12:46:13.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.458+0000 7f6bf76bb640 1 -- 172.21.15.89:0/55252095 shutdown_connections 2023-10-30T12:46:13.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.458+0000 7f6bf76bb640 1 --2- 172.21.15.89:0/55252095 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6bd40664d0 0x7f6bd4068990 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:13.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.458+0000 7f6bf76bb640 1 --2- 172.21.15.89:0/55252095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bf00c5e70 0x7f6bf015a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:13.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.458+0000 7f6bf76bb640 1 --2- 172.21.15.89:0/55252095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6bf0153b10 0x7f6bf015ae40 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:13.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.458+0000 7f6bf76bb640 1 --2- 172.21.15.89:0/55252095 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6bf00b0dc0 0x7f6bf014f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:13.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.458+0000 7f6bf76bb640 1 -- 172.21.15.89:0/55252095 >> 172.21.15.89:0/55252095 conn(0x7f6bf006d770 msgr2=0x7f6bf0069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:13.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.458+0000 7f6bf76bb640 1 -- 172.21.15.89:0/55252095 shutdown_connections 2023-10-30T12:46:13.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:13.458+0000 7f6bf76bb640 1 -- 172.21.15.89:0/55252095 wait complete. 2023-10-30T12:46:13.477 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:46:15.478 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:46:15.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.630+0000 7f93e3846640 1 Processor -- start 2023-10-30T12:46:15.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.630+0000 7f93e3846640 1 -- start start 2023-10-30T12:46:15.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.630+0000 7f93e3846640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f93dc0c2570 0x7f93dc0cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:15.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.630+0000 7f93e3846640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f93dc0c29d0 0x7f93dc0cb610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:15.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.630+0000 7f93e3846640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93dc0c68b0 con 0x7f93dc150920 2023-10-30T12:46:15.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.630+0000 7f93e3846640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93dc150d00 con 0x7f93dc0c29d0 2023-10-30T12:46:15.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.630+0000 7f93e3846640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93dc0c2e90 con 0x7f93dc0c2570 2023-10-30T12:46:15.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.630+0000 7f93e1de3640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f93dc0c29d0 0x7f93dc0cb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:15.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.630+0000 7f93e1de3640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f93dc0c29d0 0x7f93dc0cb610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57292/0 (socket says 172.21.15.89:57292) 2023-10-30T12:46:15.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.630+0000 7f93e1de3640 1 -- 172.21.15.89:0/3386584993 learned_addr learned my addr 172.21.15.89:0/3386584993 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:46:15.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.630+0000 7f93e0de1640 1 --2- 172.21.15.89:0/3386584993 >> v2:172.21.15.192:3300/0 conn(0x7f93dc0c2570 0x7f93dc0cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:15.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.630+0000 7f93e1de3640 1 -- 172.21.15.89:0/3386584993 >> v2:172.21.15.192:3300/0 conn(0x7f93dc0c2570 msgr2=0x7f93dc0cb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:15.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.630+0000 7f93e1de3640 1 --2- 172.21.15.89:0/3386584993 >> v2:172.21.15.192:3300/0 conn(0x7f93dc0c2570 0x7f93dc0cb0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:15.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.630+0000 7f93e1de3640 1 -- 172.21.15.89:0/3386584993 >> v1:172.21.15.192:6789/0 conn(0x7f93dc150920 legacy=0x7f93dc06d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:15.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.630+0000 7f93e1de3640 1 -- 172.21.15.89:0/3386584993 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93dc0c6b00 con 0x7f93dc0c29d0 2023-10-30T12:46:15.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.634+0000 7f93e1de3640 1 --2- 172.21.15.89:0/3386584993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93dc0c29d0 0x7f93dc0cb610 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f93dc06dfa0 tx=0x7f93d401c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53c46bad39e2452d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:15.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.634+0000 7f93d3fff640 1 -- 172.21.15.89:0/3386584993 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d40a56b0 con 0x7f93dc0c29d0 2023-10-30T12:46:15.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.634+0000 7f93d3fff640 1 -- 172.21.15.89:0/3386584993 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93d40a5810 con 0x7f93dc0c29d0 2023-10-30T12:46:15.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.634+0000 7f93d3fff640 1 -- 172.21.15.89:0/3386584993 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d40a5ba0 con 0x7f93dc0c29d0 2023-10-30T12:46:15.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.634+0000 7f93e3846640 1 -- 172.21.15.89:0/3386584993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93dc0c29d0 msgr2=0x7f93dc0cb610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:15.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.634+0000 7f93e3846640 1 --2- 172.21.15.89:0/3386584993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93dc0c29d0 0x7f93dc0cb610 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f93dc06dfa0 tx=0x7f93d401c900 comp rx=0 tx=0).stop 2023-10-30T12:46:15.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.634+0000 7f93e3846640 1 -- 172.21.15.89:0/3386584993 shutdown_connections 2023-10-30T12:46:15.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.634+0000 7f93e3846640 1 --2- 172.21.15.89:0/3386584993 >> v2:172.21.15.192:3300/0 conn(0x7f93dc0c2570 0x7f93dc0cb0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:15.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.634+0000 7f93e3846640 1 --2- 172.21.15.89:0/3386584993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93dc0c29d0 0x7f93dc0cb610 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:15.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.634+0000 7f93e3846640 1 -- 172.21.15.89:0/3386584993 >> 172.21.15.89:0/3386584993 conn(0x7f93dc06dfd0 msgr2=0x7f93dc06e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:15.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.634+0000 7f93e3846640 1 -- 172.21.15.89:0/3386584993 shutdown_connections 2023-10-30T12:46:15.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.634+0000 7f93e3846640 1 -- 172.21.15.89:0/3386584993 wait complete. 2023-10-30T12:46:15.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.638+0000 7f93e3846640 1 Processor -- start 2023-10-30T12:46:15.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.638+0000 7f93e3846640 1 -- start start 2023-10-30T12:46:15.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.638+0000 7f93e3846640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93dc0c2570 0x7f93dc14f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:15.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.638+0000 7f93e3846640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93dc0c29d0 0x7f93dc1588f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:15.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.638+0000 7f93e3846640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93dc150920 0x7f93dc155280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:15.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.638+0000 7f93e3846640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc14fc60 con 0x7f93dc0c29d0 2023-10-30T12:46:15.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.638+0000 7f93e3846640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc14f290 con 0x7f93dc150920 2023-10-30T12:46:15.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.638+0000 7f93e3846640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc155bc0 con 0x7f93dc0c2570 2023-10-30T12:46:15.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.638+0000 7f93e0de1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93dc0c29d0 0x7f93dc1588f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:15.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.638+0000 7f93e0de1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93dc0c29d0 0x7f93dc1588f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57308/0 (socket says 172.21.15.89:57308) 2023-10-30T12:46:15.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.638+0000 7f93e0de1640 1 -- 172.21.15.89:0/3097851493 learned_addr learned my addr 172.21.15.89:0/3097851493 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:46:15.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.638+0000 7f93e1de3640 1 --2- 172.21.15.89:0/3097851493 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93dc150920 0x7f93dc155280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:15.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.638+0000 7f93e15e2640 1 --2- 172.21.15.89:0/3097851493 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93dc0c2570 0x7f93dc14f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:15.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.638+0000 7f93e0de1640 1 -- 172.21.15.89:0/3097851493 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93dc0c2570 msgr2=0x7f93dc14f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:15.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.638+0000 7f93e0de1640 1 --2- 172.21.15.89:0/3097851493 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93dc0c2570 0x7f93dc14f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:15.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.638+0000 7f93e0de1640 1 -- 172.21.15.89:0/3097851493 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93dc150920 msgr2=0x7f93dc155280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:15.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.638+0000 7f93e0de1640 1 --2- 172.21.15.89:0/3097851493 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93dc150920 0x7f93dc155280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:15.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.638+0000 7f93e0de1640 1 -- 172.21.15.89:0/3097851493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d4009b30 con 0x7f93dc0c29d0 2023-10-30T12:46:15.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.638+0000 7f93e15e2640 1 --2- 172.21.15.89:0/3097851493 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93dc0c2570 0x7f93dc14f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:46:15.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.638+0000 7f93e0de1640 1 --2- 172.21.15.89:0/3097851493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93dc0c29d0 0x7f93dc1588f0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f93cc00d800 tx=0x7f93cc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:15.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.638+0000 7f93d1ffb640 1 -- 172.21.15.89:0/3097851493 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93cc00b6c0 con 0x7f93dc0c29d0 2023-10-30T12:46:15.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.638+0000 7f93d1ffb640 1 -- 172.21.15.89:0/3097851493 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93cc00b820 con 0x7f93dc0c29d0 2023-10-30T12:46:15.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.638+0000 7f93e3846640 1 -- 172.21.15.89:0/3097851493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93dc155ea0 con 0x7f93dc0c29d0 2023-10-30T12:46:15.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.638+0000 7f93d1ffb640 1 -- 172.21.15.89:0/3097851493 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93cc00bb90 con 0x7f93dc0c29d0 2023-10-30T12:46:15.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.638+0000 7f93e3846640 1 -- 172.21.15.89:0/3097851493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93dc14ef20 con 0x7f93dc0c29d0 2023-10-30T12:46:15.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.638+0000 7f93e3846640 1 -- 172.21.15.89:0/3097851493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93dc04e170 con 0x7f93dc0c29d0 2023-10-30T12:46:15.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.642+0000 7f93d1ffb640 1 -- 172.21.15.89:0/3097851493 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f93cc00bcf0 con 0x7f93dc0c29d0 2023-10-30T12:46:15.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.642+0000 7f93d1ffb640 1 --2- 172.21.15.89:0/3097851493 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f93c00664d0 0x7f93c0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:15.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.642+0000 7f93d1ffb640 1 -- 172.21.15.89:0/3097851493 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6492+0+0 (secure 0 0 0) 0x7f93cc084790 con 0x7f93dc0c29d0 2023-10-30T12:46:15.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.642+0000 7f93e15e2640 1 --2- 172.21.15.89:0/3097851493 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f93c00664d0 0x7f93c0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:15.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.642+0000 7f93e15e2640 1 --2- 172.21.15.89:0/3097851493 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f93c00664d0 0x7f93c0068990 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f93d80098b0 tx=0x7f93d8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:15.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.646+0000 7f93d1ffb640 1 -- 172.21.15.89:0/3097851493 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f93cc0517d0 con 0x7f93dc0c29d0 2023-10-30T12:46:15.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.822+0000 7f93e3846640 1 -- 172.21.15.89:0/3097851493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93dc06f010 con 0x7f93dc0c29d0 2023-10-30T12:46:15.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.826+0000 7f93d1ffb640 1 -- 172.21.15.89:0/3097851493 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f93cc054df0 con 0x7f93dc0c29d0 2023-10-30T12:46:15.823 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:46:15.823 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:46:14.472132+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5450,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":1,"incarnation":83,"state":"up:active","state_seq":36,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:46:15.824 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 87 2023-10-30T12:46:15.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.826+0000 7f93e3846640 1 -- 172.21.15.89:0/3097851493 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f93c00664d0 msgr2=0x7f93c0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:15.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.826+0000 7f93e3846640 1 --2- 172.21.15.89:0/3097851493 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f93c00664d0 0x7f93c0068990 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f93d80098b0 tx=0x7f93d8008040 comp rx=0 tx=0).stop 2023-10-30T12:46:15.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.826+0000 7f93e3846640 1 -- 172.21.15.89:0/3097851493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93dc0c29d0 msgr2=0x7f93dc1588f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:15.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.826+0000 7f93e3846640 1 --2- 172.21.15.89:0/3097851493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93dc0c29d0 0x7f93dc1588f0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f93cc00d800 tx=0x7f93cc00dcc0 comp rx=0 tx=0).stop 2023-10-30T12:46:15.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.826+0000 7f93e3846640 1 -- 172.21.15.89:0/3097851493 shutdown_connections 2023-10-30T12:46:15.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.826+0000 7f93e3846640 1 --2- 172.21.15.89:0/3097851493 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f93c00664d0 0x7f93c0068990 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:15.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.826+0000 7f93e3846640 1 --2- 172.21.15.89:0/3097851493 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93dc150920 0x7f93dc155280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:15.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.826+0000 7f93e3846640 1 --2- 172.21.15.89:0/3097851493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93dc0c29d0 0x7f93dc1588f0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:15.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.826+0000 7f93e3846640 1 --2- 172.21.15.89:0/3097851493 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f93dc0c2570 0x7f93dc14f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:15.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.826+0000 7f93e3846640 1 -- 172.21.15.89:0/3097851493 >> 172.21.15.89:0/3097851493 conn(0x7f93dc06dfd0 msgr2=0x7f93dc0c14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:15.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.826+0000 7f93e3846640 1 -- 172.21.15.89:0/3097851493 shutdown_connections 2023-10-30T12:46:15.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.826+0000 7f93e3846640 1 -- 172.21.15.89:0/3097851493 wait complete. 2023-10-30T12:46:15.844 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=1, replacing gid=5139 2023-10-30T12:46:15.844 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.c 2023-10-30T12:46:15.844 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T12:46:15.845 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T12:46:15.845 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T12:46:15.849 INFO:tasks.ceph.mds.c:Started 2023-10-30T12:46:15.849 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:46:15.933 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2023-10-30T12:46:15.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.990+0000 7f4a9afd3640 1 Processor -- start 2023-10-30T12:46:15.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.990+0000 7f4a9afd3640 1 -- start start 2023-10-30T12:46:15.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a9afd3640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4a940c6b80 0x7f4a940bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:15.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a9afd3640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a940c0a50 con 0x7f4a940c6f60 2023-10-30T12:46:15.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a9afd3640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a9406f6e0 con 0x7f4a9413f4a0 2023-10-30T12:46:15.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a9afd3640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a9406f9c0 con 0x7f4a940c6b80 2023-10-30T12:46:15.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a8bfff640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f4a940c6f60 0x7f4a9413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38790/0 (socket says 172.21.15.89:38790) 2023-10-30T12:46:15.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a8bfff640 1 -- 172.21.15.89:0/2439034912 learned_addr learned my addr 172.21.15.89:0/2439034912 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:46:15.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a98d6f640 1 --2- 172.21.15.89:0/2439034912 >> v2:172.21.15.192:3300/0 conn(0x7f4a940c6b80 0x7f4a940bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:15.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a8b7fe640 1 -- 172.21.15.89:0/2439034912 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2238792946 0 0) 0x7f4a940c0a50 con 0x7f4a940c6f60 2023-10-30T12:46:15.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a8b7fe640 1 -- 172.21.15.89:0/2439034912 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a7c003530 con 0x7f4a940c6f60 2023-10-30T12:46:15.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a8b7fe640 1 -- 172.21.15.89:0/2439034912 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3942043650 0 0) 0x7f4a7c003530 con 0x7f4a940c6f60 2023-10-30T12:46:15.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a8b7fe640 1 -- 172.21.15.89:0/2439034912 >> v2:172.21.15.192:3300/0 conn(0x7f4a940c6b80 msgr2=0x7f4a940bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:15.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a8b7fe640 1 --2- 172.21.15.89:0/2439034912 >> v2:172.21.15.192:3300/0 conn(0x7f4a940c6b80 0x7f4a940bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:15.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a8b7fe640 1 -- 172.21.15.89:0/2439034912 >> v1:172.21.15.192:6789/0 conn(0x7f4a9413f4a0 legacy=0x7f4a941566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:15.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a8b7fe640 1 -- 172.21.15.89:0/2439034912 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a9406fca0 con 0x7f4a940c6f60 2023-10-30T12:46:15.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a8b7fe640 1 -- 172.21.15.89:0/2439034912 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f4a84001680 con 0x7f4a940c6f60 2023-10-30T12:46:15.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a8b7fe640 1 -- 172.21.15.89:0/2439034912 >> v1:172.21.15.89:6789/0 conn(0x7f4a940c6f60 legacy=0x7f4a9413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:15.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a8b7fe640 1 --2- 172.21.15.89:0/2439034912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a7c003980 0x7f4a7c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:15.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a8b7fe640 1 --2- 172.21.15.89:0/2439034912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a7c0047d0 0x7f4a7c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:15.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a8b7fe640 1 --2- 172.21.15.89:0/2439034912 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a7c005190 0x7f4a7c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:15.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a8b7fe640 1 -- 172.21.15.89:0/2439034912 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9406f9c0 con 0x7f4a7c003980 2023-10-30T12:46:15.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a8b7fe640 1 -- 172.21.15.89:0/2439034912 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7c00c030 con 0x7f4a7c0047d0 2023-10-30T12:46:15.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a8b7fe640 1 -- 172.21.15.89:0/2439034912 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7c00c1f0 con 0x7f4a7c005190 2023-10-30T12:46:15.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a98d6f640 1 --2- 172.21.15.89:0/2439034912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a7c003980 0x7f4a7c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:15.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a8bfff640 1 --2- 172.21.15.89:0/2439034912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a7c0047d0 0x7f4a7c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:15.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a99570640 1 --2- 172.21.15.89:0/2439034912 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a7c005190 0x7f4a7c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:15.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a98d6f640 1 -- 172.21.15.89:0/2439034912 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a7c005190 msgr2=0x7f4a7c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:15.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a98d6f640 1 --2- 172.21.15.89:0/2439034912 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a7c005190 0x7f4a7c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:15.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a98d6f640 1 -- 172.21.15.89:0/2439034912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a7c0047d0 msgr2=0x7f4a7c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:15.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a98d6f640 1 --2- 172.21.15.89:0/2439034912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a7c0047d0 0x7f4a7c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:15.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a98d6f640 1 -- 172.21.15.89:0/2439034912 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a7c003530 con 0x7f4a7c003980 2023-10-30T12:46:15.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.994+0000 7f4a99570640 1 -- 172.21.15.89:0/2439034912 reap_dead start 2023-10-30T12:46:16.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.998+0000 7f4a98d6f640 1 --2- 172.21.15.89:0/2439034912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a7c003980 0x7f4a7c003da0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f4a90009ae0 tx=0x7f4a9001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48054298029a5869 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:16.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.998+0000 7f4a8b7fe640 1 -- 172.21.15.89:0/2439034912 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a900a5530 con 0x7f4a7c003980 2023-10-30T12:46:16.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.998+0000 7f4a8b7fe640 1 -- 172.21.15.89:0/2439034912 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a900a5690 con 0x7f4a7c003980 2023-10-30T12:46:16.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.998+0000 7f4a8b7fe640 1 -- 172.21.15.89:0/2439034912 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a900a5a40 con 0x7f4a7c003980 2023-10-30T12:46:16.001 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.998+0000 7f4a9afd3640 1 -- 172.21.15.89:0/2439034912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a7c003980 msgr2=0x7f4a7c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:16.002 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.998+0000 7f4a9afd3640 1 --2- 172.21.15.89:0/2439034912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a7c003980 0x7f4a7c003da0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f4a90009ae0 tx=0x7f4a9001ca50 comp rx=0 tx=0).stop 2023-10-30T12:46:16.002 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.998+0000 7f4a9afd3640 1 -- 172.21.15.89:0/2439034912 shutdown_connections 2023-10-30T12:46:16.002 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.998+0000 7f4a9afd3640 1 --2- 172.21.15.89:0/2439034912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a7c003980 0x7f4a7c003da0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:16.002 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.998+0000 7f4a9afd3640 1 -- 172.21.15.89:0/2439034912 >> 172.21.15.89:0/2439034912 conn(0x7f4a940c7910 msgr2=0x7f4a940c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:16.002 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.998+0000 7f4a9afd3640 1 -- 172.21.15.89:0/2439034912 shutdown_connections 2023-10-30T12:46:16.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.998+0000 7f4a9afd3640 1 -- 172.21.15.89:0/2439034912 wait complete. 2023-10-30T12:46:16.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.998+0000 7f4a9afd3640 1 Processor -- start 2023-10-30T12:46:16.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:15.998+0000 7f4a9afd3640 1 -- start start 2023-10-30T12:46:16.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.002+0000 7f4a9afd3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a7c003980 0x7f4a9406c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:16.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.002+0000 7f4a9afd3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a9413aa80 0x7f4a941f32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:16.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.002+0000 7f4a9afd3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a941f3800 0x7f4a941f53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:16.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.002+0000 7f4a9afd3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a94155f80 con 0x7f4a7c003980 2023-10-30T12:46:16.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.002+0000 7f4a9afd3640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9406c5c0 con 0x7f4a9413aa80 2023-10-30T12:46:16.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.002+0000 7f4a9afd3640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a940c9a60 con 0x7f4a941f3800 2023-10-30T12:46:16.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.002+0000 7f4a98d6f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a7c003980 0x7f4a9406c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:16.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.002+0000 7f4a98d6f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a7c003980 0x7f4a9406c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57336/0 (socket says 172.21.15.89:57336) 2023-10-30T12:46:16.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.002+0000 7f4a98d6f640 1 -- 172.21.15.89:0/3770842489 learned_addr learned my addr 172.21.15.89:0/3770842489 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:46:16.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.002+0000 7f4a99570640 1 --2- 172.21.15.89:0/3770842489 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a941f3800 0x7f4a941f53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:16.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.002+0000 7f4a8bfff640 1 --2- 172.21.15.89:0/3770842489 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a9413aa80 0x7f4a941f32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:16.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.002+0000 7f4a98d6f640 1 -- 172.21.15.89:0/3770842489 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a941f3800 msgr2=0x7f4a941f53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:16.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.002+0000 7f4a98d6f640 1 --2- 172.21.15.89:0/3770842489 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a941f3800 0x7f4a941f53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:16.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.002+0000 7f4a98d6f640 1 -- 172.21.15.89:0/3770842489 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a9413aa80 msgr2=0x7f4a941f32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:16.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.002+0000 7f4a98d6f640 1 --2- 172.21.15.89:0/3770842489 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a9413aa80 0x7f4a941f32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:16.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.002+0000 7f4a98d6f640 1 -- 172.21.15.89:0/3770842489 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a940c9d40 con 0x7f4a7c003980 2023-10-30T12:46:16.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.002+0000 7f4a8bfff640 1 --2- 172.21.15.89:0/3770842489 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a9413aa80 0x7f4a941f32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:46:16.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.002+0000 7f4a98d6f640 1 --2- 172.21.15.89:0/3770842489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a7c003980 0x7f4a9406c080 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f4a9001c880 tx=0x7f4a900197f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:16.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.002+0000 7f4a897fa640 1 -- 172.21.15.89:0/3770842489 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a900bd070 con 0x7f4a7c003980 2023-10-30T12:46:16.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.002+0000 7f4a897fa640 1 -- 172.21.15.89:0/3770842489 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a90004020 con 0x7f4a7c003980 2023-10-30T12:46:16.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.002+0000 7f4a897fa640 1 -- 172.21.15.89:0/3770842489 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a900a5530 con 0x7f4a7c003980 2023-10-30T12:46:16.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.002+0000 7f4a9afd3640 1 -- 172.21.15.89:0/3770842489 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a940ca020 con 0x7f4a7c003980 2023-10-30T12:46:16.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.002+0000 7f4a9afd3640 1 -- 172.21.15.89:0/3770842489 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a94155910 con 0x7f4a7c003980 2023-10-30T12:46:16.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.002+0000 7f4a9afd3640 1 -- 172.21.15.89:0/3770842489 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a9404e170 con 0x7f4a7c003980 2023-10-30T12:46:16.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.006+0000 7f4a897fa640 1 -- 172.21.15.89:0/3770842489 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4a900b4070 con 0x7f4a7c003980 2023-10-30T12:46:16.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.006+0000 7f4a897fa640 1 --2- 172.21.15.89:0/3770842489 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4a800664a0 0x7f4a80068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:16.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.006+0000 7f4a897fa640 1 -- 172.21.15.89:0/3770842489 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6492+0+0 (secure 0 0 0) 0x7f4a9011dff0 con 0x7f4a7c003980 2023-10-30T12:46:16.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.006+0000 7f4a8bfff640 1 --2- 172.21.15.89:0/3770842489 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4a800664a0 0x7f4a80068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:16.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.006+0000 7f4a8bfff640 1 --2- 172.21.15.89:0/3770842489 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4a800664a0 0x7f4a80068960 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4a9413b3f0 tx=0x7f4a84009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:16.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.010+0000 7f4a897fa640 1 -- 172.21.15.89:0/3770842489 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4a900eb0e0 con 0x7f4a7c003980 2023-10-30T12:46:16.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.186+0000 7f4a9afd3640 1 -- 172.21.15.89:0/3770842489 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a940c8070 con 0x7f4a7c003980 2023-10-30T12:46:16.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.186+0000 7f4a897fa640 1 -- 172.21.15.89:0/3770842489 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4a900ee700 con 0x7f4a7c003980 2023-10-30T12:46:16.186 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:46:16.187 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:46:14.472132+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5450,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":1,"incarnation":83,"state":"up:active","state_seq":36,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:46:16.187 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 87 2023-10-30T12:46:16.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.190+0000 7f4a9afd3640 1 -- 172.21.15.89:0/3770842489 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4a800664a0 msgr2=0x7f4a80068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:16.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.190+0000 7f4a9afd3640 1 --2- 172.21.15.89:0/3770842489 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4a800664a0 0x7f4a80068960 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4a9413b3f0 tx=0x7f4a84009040 comp rx=0 tx=0).stop 2023-10-30T12:46:16.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.190+0000 7f4a9afd3640 1 -- 172.21.15.89:0/3770842489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a7c003980 msgr2=0x7f4a9406c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:16.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.190+0000 7f4a9afd3640 1 --2- 172.21.15.89:0/3770842489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a7c003980 0x7f4a9406c080 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f4a9001c880 tx=0x7f4a900197f0 comp rx=0 tx=0).stop 2023-10-30T12:46:16.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.190+0000 7f4a9afd3640 1 -- 172.21.15.89:0/3770842489 shutdown_connections 2023-10-30T12:46:16.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.190+0000 7f4a9afd3640 1 --2- 172.21.15.89:0/3770842489 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4a800664a0 0x7f4a80068960 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:16.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.190+0000 7f4a9afd3640 1 --2- 172.21.15.89:0/3770842489 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a9413aa80 0x7f4a941f32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:16.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.190+0000 7f4a9afd3640 1 --2- 172.21.15.89:0/3770842489 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a941f3800 0x7f4a941f53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:16.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.190+0000 7f4a9afd3640 1 --2- 172.21.15.89:0/3770842489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a7c003980 0x7f4a9406c080 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:16.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.190+0000 7f4a9afd3640 1 -- 172.21.15.89:0/3770842489 >> 172.21.15.89:0/3770842489 conn(0x7f4a940c7910 msgr2=0x7f4a94069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:16.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.190+0000 7f4a9afd3640 1 -- 172.21.15.89:0/3770842489 shutdown_connections 2023-10-30T12:46:16.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:16.190+0000 7f4a9afd3640 1 -- 172.21.15.89:0/3770842489 wait complete. 2023-10-30T12:46:16.207 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-30T12:46:17.294 INFO:tasks.ceph.mds.b.smithi192.stderr:2023-10-30T12:46:17.294+0000 7f5231cd7640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T12:46:17.790 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:46:17.794 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:46:18.208 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:46:18.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.358+0000 7f044910e640 1 Processor -- start 2023-10-30T12:46:18.357 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.358+0000 7f044910e640 1 -- start start 2023-10-30T12:46:18.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.358+0000 7f044910e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04440c6b80 0x7f04440bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:18.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.358+0000 7f044910e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f044413f4a0 0x7f04440c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:18.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.358+0000 7f044910e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04440c0a50 con 0x7f04440c6f60 2023-10-30T12:46:18.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.358+0000 7f044910e640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f044406f6e0 con 0x7f044413f4a0 2023-10-30T12:46:18.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.358+0000 7f044910e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f044406f9c0 con 0x7f04440c6b80 2023-10-30T12:46:18.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.358+0000 7f044359e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f044413f4a0 0x7f04440c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:18.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.362+0000 7f044359e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f044413f4a0 0x7f04440c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57342/0 (socket says 172.21.15.89:57342) 2023-10-30T12:46:18.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.362+0000 7f044359e640 1 -- 172.21.15.89:0/864769645 learned_addr learned my addr 172.21.15.89:0/864769645 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:46:18.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.362+0000 7f0442d9d640 1 --2- 172.21.15.89:0/864769645 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04440c6b80 0x7f04440bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:18.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.362+0000 7f044359e640 1 -- 172.21.15.89:0/864769645 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04440c6b80 msgr2=0x7f04440bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:18.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.362+0000 7f044359e640 1 --2- 172.21.15.89:0/864769645 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04440c6b80 0x7f04440bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:18.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.362+0000 7f044359e640 1 -- 172.21.15.89:0/864769645 >> v1:172.21.15.192:6789/0 conn(0x7f04440c6f60 legacy=0x7f044413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:18.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.362+0000 7f044359e640 1 -- 172.21.15.89:0/864769645 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f044406fca0 con 0x7f044413f4a0 2023-10-30T12:46:18.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.362+0000 7f044359e640 1 --2- 172.21.15.89:0/864769645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f044413f4a0 0x7f04440c0400 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f043c009870 tx=0x7f043c01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d12726e37e3acdb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:18.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.362+0000 7f0441d9b640 1 -- 172.21.15.89:0/864769645 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043c0ab650 con 0x7f044413f4a0 2023-10-30T12:46:18.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.362+0000 7f0441d9b640 1 -- 172.21.15.89:0/864769645 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043c0ab7b0 con 0x7f044413f4a0 2023-10-30T12:46:18.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.362+0000 7f0441d9b640 1 -- 172.21.15.89:0/864769645 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043c0abb40 con 0x7f044413f4a0 2023-10-30T12:46:18.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.366+0000 7f044910e640 1 -- 172.21.15.89:0/864769645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f044413f4a0 msgr2=0x7f04440c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:18.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.366+0000 7f044910e640 1 --2- 172.21.15.89:0/864769645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f044413f4a0 0x7f04440c0400 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f043c009870 tx=0x7f043c01c540 comp rx=0 tx=0).stop 2023-10-30T12:46:18.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.366+0000 7f044910e640 1 -- 172.21.15.89:0/864769645 shutdown_connections 2023-10-30T12:46:18.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.366+0000 7f044910e640 1 --2- 172.21.15.89:0/864769645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f044413f4a0 0x7f04440c0400 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:18.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.366+0000 7f044910e640 1 --2- 172.21.15.89:0/864769645 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04440c6b80 0x7f04440bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:18.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.366+0000 7f044910e640 1 -- 172.21.15.89:0/864769645 >> 172.21.15.89:0/864769645 conn(0x7f04440c7910 msgr2=0x7f04440c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:18.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.366+0000 7f044910e640 1 -- 172.21.15.89:0/864769645 shutdown_connections 2023-10-30T12:46:18.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.366+0000 7f044910e640 1 -- 172.21.15.89:0/864769645 wait complete. 2023-10-30T12:46:18.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.366+0000 7f044910e640 1 Processor -- start 2023-10-30T12:46:18.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.366+0000 7f044910e640 1 -- start start 2023-10-30T12:46:18.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.366+0000 7f044910e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04440c6b80 0x7f04440c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:18.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.366+0000 7f044910e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04440c6f60 0x7f044413e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:18.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.366+0000 7f044910e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f044413f4a0 0x7f04440c6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:18.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.366+0000 7f044910e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04441580e0 con 0x7f04440c6f60 2023-10-30T12:46:18.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.366+0000 7f044910e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0444153fd0 con 0x7f04440c6b80 2023-10-30T12:46:18.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.366+0000 7f044910e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f044413de70 con 0x7f044413f4a0 2023-10-30T12:46:18.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.366+0000 7f044259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04440c6f60 0x7f044413e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:18.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.366+0000 7f044259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04440c6f60 0x7f044413e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57348/0 (socket says 172.21.15.89:57348) 2023-10-30T12:46:18.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.366+0000 7f044259c640 1 -- 172.21.15.89:0/1272418968 learned_addr learned my addr 172.21.15.89:0/1272418968 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:46:18.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.366+0000 7f0442d9d640 1 --2- 172.21.15.89:0/1272418968 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04440c6b80 0x7f04440c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:18.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.366+0000 7f044359e640 1 --2- 172.21.15.89:0/1272418968 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f044413f4a0 0x7f04440c6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:18.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.366+0000 7f044259c640 1 -- 172.21.15.89:0/1272418968 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f044413f4a0 msgr2=0x7f04440c6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:18.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.366+0000 7f044259c640 1 --2- 172.21.15.89:0/1272418968 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f044413f4a0 0x7f04440c6010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:18.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.366+0000 7f044259c640 1 -- 172.21.15.89:0/1272418968 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04440c6b80 msgr2=0x7f04440c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:18.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.366+0000 7f044259c640 1 --2- 172.21.15.89:0/1272418968 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04440c6b80 0x7f04440c8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:18.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.366+0000 7f044259c640 1 -- 172.21.15.89:0/1272418968 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f043c0094d0 con 0x7f04440c6f60 2023-10-30T12:46:18.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.370+0000 7f044259c640 1 --2- 172.21.15.89:0/1272418968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04440c6f60 0x7f044413e300 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f043800c500 tx=0x7f043800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:18.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.370+0000 7f042b7fe640 1 -- 172.21.15.89:0/1272418968 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04380052d0 con 0x7f04440c6f60 2023-10-30T12:46:18.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.370+0000 7f042b7fe640 1 -- 172.21.15.89:0/1272418968 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0438005430 con 0x7f04440c6f60 2023-10-30T12:46:18.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.370+0000 7f044910e640 1 -- 172.21.15.89:0/1272418968 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0444159240 con 0x7f04440c6f60 2023-10-30T12:46:18.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.370+0000 7f042b7fe640 1 -- 172.21.15.89:0/1272418968 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0438012830 con 0x7f04440c6f60 2023-10-30T12:46:18.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.370+0000 7f044910e640 1 -- 172.21.15.89:0/1272418968 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04441594b0 con 0x7f04440c6f60 2023-10-30T12:46:18.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.370+0000 7f044910e640 1 -- 172.21.15.89:0/1272418968 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f044413dfe0 con 0x7f04440c6f60 2023-10-30T12:46:18.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.370+0000 7f042b7fe640 1 -- 172.21.15.89:0/1272418968 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0438012990 con 0x7f04440c6f60 2023-10-30T12:46:18.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.370+0000 7f042b7fe640 1 --2- 172.21.15.89:0/1272418968 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f04300664d0 0x7f0430068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:18.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.370+0000 7f042b7fe640 1 -- 172.21.15.89:0/1272418968 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6492+0+0 (secure 0 0 0) 0x7f0438083d80 con 0x7f04440c6f60 2023-10-30T12:46:18.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.370+0000 7f0442d9d640 1 --2- 172.21.15.89:0/1272418968 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f04300664d0 0x7f0430068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:18.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.374+0000 7f0442d9d640 1 --2- 172.21.15.89:0/1272418968 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f04300664d0 0x7f0430068990 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f0434003fd0 tx=0x7f04340039a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:18.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.374+0000 7f042b7fe640 1 -- 172.21.15.89:0/1272418968 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0438050e60 con 0x7f04440c6f60 2023-10-30T12:46:18.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.562+0000 7f044910e640 1 -- 172.21.15.89:0/1272418968 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f044413dc60 con 0x7f04440c6f60 2023-10-30T12:46:18.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.562+0000 7f042b7fe640 1 -- 172.21.15.89:0/1272418968 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f0438054480 con 0x7f04440c6f60 2023-10-30T12:46:18.560 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:46:18.561 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6075,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:46:14.472132+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5450,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":1,"incarnation":83,"state":"up:active","state_seq":36,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:46:18.561 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 88 2023-10-30T12:46:18.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.562+0000 7f044910e640 1 -- 172.21.15.89:0/1272418968 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f04300664d0 msgr2=0x7f0430068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:18.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.562+0000 7f044910e640 1 --2- 172.21.15.89:0/1272418968 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f04300664d0 0x7f0430068990 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f0434003fd0 tx=0x7f04340039a0 comp rx=0 tx=0).stop 2023-10-30T12:46:18.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.562+0000 7f044910e640 1 -- 172.21.15.89:0/1272418968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04440c6f60 msgr2=0x7f044413e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:18.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.562+0000 7f044910e640 1 --2- 172.21.15.89:0/1272418968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04440c6f60 0x7f044413e300 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f043800c500 tx=0x7f043800c9c0 comp rx=0 tx=0).stop 2023-10-30T12:46:18.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.562+0000 7f044910e640 1 -- 172.21.15.89:0/1272418968 shutdown_connections 2023-10-30T12:46:18.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.562+0000 7f044910e640 1 --2- 172.21.15.89:0/1272418968 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f04300664d0 0x7f0430068990 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:18.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.562+0000 7f044910e640 1 --2- 172.21.15.89:0/1272418968 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f044413f4a0 0x7f04440c6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:18.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.562+0000 7f044910e640 1 --2- 172.21.15.89:0/1272418968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04440c6f60 0x7f044413e300 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:18.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.562+0000 7f044910e640 1 --2- 172.21.15.89:0/1272418968 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04440c6b80 0x7f04440c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:18.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.562+0000 7f044910e640 1 -- 172.21.15.89:0/1272418968 >> 172.21.15.89:0/1272418968 conn(0x7f04440c7910 msgr2=0x7f0444160800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:18.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.562+0000 7f044910e640 1 -- 172.21.15.89:0/1272418968 shutdown_connections 2023-10-30T12:46:18.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:46:18.562+0000 7f044910e640 1 -- 172.21.15.89:0/1272418968 wait complete. 2023-10-30T12:46:18.581 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T12:46:18.581 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 96 secs before thrashing 2023-10-30T12:46:47.852 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:46:47.857 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:47:17.964 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:47:17.968 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:47:48.011 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:47:48.015 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:47:54.583 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:47:54.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.732+0000 7f57bdc05640 1 Processor -- start 2023-10-30T12:47:54.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.732+0000 7f57bdc05640 1 -- start start 2023-10-30T12:47:54.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.732+0000 7f57bdc05640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f57b81538a0 0x7f57b815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:54.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.732+0000 7f57bdc05640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b80cb800 con 0x7f57b80cf160 2023-10-30T12:47:54.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.732+0000 7f57bdc05640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b815dc30 con 0x7f57b80ca0e0 2023-10-30T12:47:54.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.732+0000 7f57bdc05640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57b80cb330 con 0x7f57b81538a0 2023-10-30T12:47:54.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.732+0000 7f57b7fff640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f57b80cf160 0x7f57b8164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:50906/0 (socket says 172.21.15.89:50906) 2023-10-30T12:47:54.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.732+0000 7f57b7fff640 1 -- 172.21.15.89:0/1172483926 learned_addr learned my addr 172.21.15.89:0/1172483926 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:47:54.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.732+0000 7f57b77fe640 1 --2- 172.21.15.89:0/1172483926 >> v2:172.21.15.192:3300/0 conn(0x7f57b81538a0 0x7f57b815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:54.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.732+0000 7f57b67fc640 1 -- 172.21.15.89:0/1172483926 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1194535350 0 0) 0x7f57b80cb800 con 0x7f57b80cf160 2023-10-30T12:47:54.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.732+0000 7f57b67fc640 1 -- 172.21.15.89:0/1172483926 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57a0003530 con 0x7f57b80cf160 2023-10-30T12:47:54.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.732+0000 7f57b67fc640 1 -- 172.21.15.89:0/1172483926 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 620046872 0 0) 0x7f57a0003530 con 0x7f57b80cf160 2023-10-30T12:47:54.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.732+0000 7f57b67fc640 1 -- 172.21.15.89:0/1172483926 >> v2:172.21.15.192:3300/0 conn(0x7f57b81538a0 msgr2=0x7f57b815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:54.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.732+0000 7f57b67fc640 1 --2- 172.21.15.89:0/1172483926 >> v2:172.21.15.192:3300/0 conn(0x7f57b81538a0 0x7f57b815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:54.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.732+0000 7f57b67fc640 1 -- 172.21.15.89:0/1172483926 >> v1:172.21.15.192:6789/0 conn(0x7f57b80ca0e0 legacy=0x7f57b815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:54.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.732+0000 7f57b67fc640 1 -- 172.21.15.89:0/1172483926 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b80cad70 con 0x7f57b80cf160 2023-10-30T12:47:54.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.732+0000 7f57b67fc640 1 -- 172.21.15.89:0/1172483926 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f57b0001a00 con 0x7f57b80cf160 2023-10-30T12:47:54.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.732+0000 7f57b67fc640 1 -- 172.21.15.89:0/1172483926 >> v1:172.21.15.89:6789/0 conn(0x7f57b80cf160 legacy=0x7f57b8164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:54.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.732+0000 7f57b67fc640 1 --2- 172.21.15.89:0/1172483926 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57a0003980 0x7f57a0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:54.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.732+0000 7f57b67fc640 1 --2- 172.21.15.89:0/1172483926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57a00047d0 0x7f57a0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:54.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.732+0000 7f57b67fc640 1 --2- 172.21.15.89:0/1172483926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57a0005190 0x7f57a000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:54.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.732+0000 7f57b67fc640 1 -- 172.21.15.89:0/1172483926 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b80cb330 con 0x7f57a0005190 2023-10-30T12:47:54.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.732+0000 7f57b67fc640 1 -- 172.21.15.89:0/1172483926 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a000c040 con 0x7f57a00047d0 2023-10-30T12:47:54.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.732+0000 7f57b67fc640 1 -- 172.21.15.89:0/1172483926 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a000c200 con 0x7f57a0003980 2023-10-30T12:47:54.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.732+0000 7f57b77fe640 1 --2- 172.21.15.89:0/1172483926 >> v2:172.21.15.192:3300/0 conn(0x7f57b81538a0 0x7f57b815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:47:54.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.736+0000 7f57b7fff640 1 --2- 172.21.15.89:0/1172483926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57a0005190 0x7f57a000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:54.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.736+0000 7f57b6ffd640 1 --2- 172.21.15.89:0/1172483926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57a00047d0 0x7f57a0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:54.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.736+0000 7f57b77fe640 1 --2- 172.21.15.89:0/1172483926 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57a0003980 0x7f57a0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:54.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.736+0000 7f57b7fff640 1 -- 172.21.15.89:0/1172483926 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57a0003980 msgr2=0x7f57a0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:54.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.736+0000 7f57b7fff640 1 --2- 172.21.15.89:0/1172483926 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57a0003980 0x7f57a0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:54.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.736+0000 7f57b7fff640 1 -- 172.21.15.89:0/1172483926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57a00047d0 msgr2=0x7f57a0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:54.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.736+0000 7f57b7fff640 1 --2- 172.21.15.89:0/1172483926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57a00047d0 0x7f57a0004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:54.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.736+0000 7f57b7fff640 1 -- 172.21.15.89:0/1172483926 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a0003530 con 0x7f57a0005190 2023-10-30T12:47:54.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.736+0000 7f57b7fff640 1 -- 172.21.15.89:0/1172483926 reap_dead start 2023-10-30T12:47:54.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.736+0000 7f57b7fff640 1 --2- 172.21.15.89:0/1172483926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57a0005190 0x7f57a000b990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f57b000a820 tx=0x7f57b001d320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f63829ef69b57c6d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:54.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.736+0000 7f57b67fc640 1 -- 172.21.15.89:0/1172483926 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b00072d0 con 0x7f57a0005190 2023-10-30T12:47:54.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.736+0000 7f57b67fc640 1 -- 172.21.15.89:0/1172483926 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b00a8070 con 0x7f57a0005190 2023-10-30T12:47:54.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.736+0000 7f57b67fc640 1 -- 172.21.15.89:0/1172483926 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b00076b0 con 0x7f57a0005190 2023-10-30T12:47:54.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.740+0000 7f57bdc05640 1 -- 172.21.15.89:0/1172483926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57a0005190 msgr2=0x7f57a000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:54.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.740+0000 7f57bdc05640 1 --2- 172.21.15.89:0/1172483926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57a0005190 0x7f57a000b990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f57b000a820 tx=0x7f57b001d320 comp rx=0 tx=0).stop 2023-10-30T12:47:54.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.740+0000 7f57bdc05640 1 -- 172.21.15.89:0/1172483926 shutdown_connections 2023-10-30T12:47:54.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.740+0000 7f57bdc05640 1 --2- 172.21.15.89:0/1172483926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57a0005190 0x7f57a000b990 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:54.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.740+0000 7f57bdc05640 1 -- 172.21.15.89:0/1172483926 >> 172.21.15.89:0/1172483926 conn(0x7f57b806bfe0 msgr2=0x7f57b806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:54.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.740+0000 7f57bdc05640 1 -- 172.21.15.89:0/1172483926 shutdown_connections 2023-10-30T12:47:54.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.740+0000 7f57bdc05640 1 -- 172.21.15.89:0/1172483926 wait complete. 2023-10-30T12:47:54.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.740+0000 7f57bdc05640 1 Processor -- start 2023-10-30T12:47:54.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.740+0000 7f57bdc05640 1 -- start start 2023-10-30T12:47:54.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.740+0000 7f57bdc05640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57a0005190 0x7f57b80c65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:54.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.740+0000 7f57bdc05640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57b8145990 0x7f57b8147a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:54.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.740+0000 7f57bdc05640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57b80cdca0 0x7f57b80ce130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:54.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.740+0000 7f57bdc05640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b815b4a0 con 0x7f57a0005190 2023-10-30T12:47:54.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.740+0000 7f57bdc05640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b80ce670 con 0x7f57b8145990 2023-10-30T12:47:54.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.740+0000 7f57bdc05640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57b8152f50 con 0x7f57b80cdca0 2023-10-30T12:47:54.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.740+0000 7f57b77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57a0005190 0x7f57b80c65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:54.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.740+0000 7f57b77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57a0005190 0x7f57b80c65e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39216/0 (socket says 172.21.15.89:39216) 2023-10-30T12:47:54.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.740+0000 7f57b77fe640 1 -- 172.21.15.89:0/349002290 learned_addr learned my addr 172.21.15.89:0/349002290 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:47:54.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.740+0000 7f57b7fff640 1 --2- 172.21.15.89:0/349002290 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57b80cdca0 0x7f57b80ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:54.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.740+0000 7f57b6ffd640 1 --2- 172.21.15.89:0/349002290 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57b8145990 0x7f57b8147a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:54.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.740+0000 7f57b77fe640 1 -- 172.21.15.89:0/349002290 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57b80cdca0 msgr2=0x7f57b80ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:54.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.740+0000 7f57b77fe640 1 --2- 172.21.15.89:0/349002290 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57b80cdca0 0x7f57b80ce130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:54.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.740+0000 7f57b77fe640 1 -- 172.21.15.89:0/349002290 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57b8145990 msgr2=0x7f57b8147a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:54.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.740+0000 7f57b77fe640 1 --2- 172.21.15.89:0/349002290 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57b8145990 0x7f57b8147a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:54.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.740+0000 7f57b77fe640 1 -- 172.21.15.89:0/349002290 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b815ae70 con 0x7f57a0005190 2023-10-30T12:47:54.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.740+0000 7f57b77fe640 1 --2- 172.21.15.89:0/349002290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57a0005190 0x7f57b80c65e0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f57a8011900 tx=0x7f57a8011dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:54.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.744+0000 7f579ffff640 1 -- 172.21.15.89:0/349002290 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a8009650 con 0x7f57a0005190 2023-10-30T12:47:54.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.744+0000 7f579ffff640 1 -- 172.21.15.89:0/349002290 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57a800fbd0 con 0x7f57a0005190 2023-10-30T12:47:54.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.744+0000 7f57bdc05640 1 -- 172.21.15.89:0/349002290 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b815aa50 con 0x7f57a0005190 2023-10-30T12:47:54.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.744+0000 7f579ffff640 1 -- 172.21.15.89:0/349002290 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a8015460 con 0x7f57a0005190 2023-10-30T12:47:54.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.744+0000 7f57bdc05640 1 -- 172.21.15.89:0/349002290 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57b8147f90 con 0x7f57a0005190 2023-10-30T12:47:54.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.744+0000 7f57bdc05640 1 -- 172.21.15.89:0/349002290 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57b804e170 con 0x7f57a0005190 2023-10-30T12:47:54.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.744+0000 7f579ffff640 1 -- 172.21.15.89:0/349002290 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f57a80155c0 con 0x7f57a0005190 2023-10-30T12:47:54.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.744+0000 7f579ffff640 1 --2- 172.21.15.89:0/349002290 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f57a40664a0 0x7f57a4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:54.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.744+0000 7f579ffff640 1 -- 172.21.15.89:0/349002290 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6492+0+0 (secure 0 0 0) 0x7f57a8084810 con 0x7f57a0005190 2023-10-30T12:47:54.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.744+0000 7f57b6ffd640 1 --2- 172.21.15.89:0/349002290 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f57a40664a0 0x7f57a4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:54.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.748+0000 7f57b6ffd640 1 --2- 172.21.15.89:0/349002290 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f57a40664a0 0x7f57a4068960 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f57b815bc00 tx=0x7f57ac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:54.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.748+0000 7f579ffff640 1 -- 172.21.15.89:0/349002290 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f57a8051970 con 0x7f57a0005190 2023-10-30T12:47:54.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.936+0000 7f57bdc05640 1 -- 172.21.15.89:0/349002290 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57b80b1dd0 con 0x7f57a0005190 2023-10-30T12:47:54.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.936+0000 7f579ffff640 1 -- 172.21.15.89:0/349002290 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f57a8054f90 con 0x7f57a0005190 2023-10-30T12:47:54.937 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:47:54.938 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6075,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:47:10.393441+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5450,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":1,"incarnation":83,"state":"up:active","state_seq":36,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:47:54.938 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 91 2023-10-30T12:47:54.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.940+0000 7f57bdc05640 1 -- 172.21.15.89:0/349002290 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f57a40664a0 msgr2=0x7f57a4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:54.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.940+0000 7f57bdc05640 1 --2- 172.21.15.89:0/349002290 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f57a40664a0 0x7f57a4068960 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f57b815bc00 tx=0x7f57ac009040 comp rx=0 tx=0).stop 2023-10-30T12:47:54.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.940+0000 7f57bdc05640 1 -- 172.21.15.89:0/349002290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57a0005190 msgr2=0x7f57b80c65e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:54.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.940+0000 7f57bdc05640 1 --2- 172.21.15.89:0/349002290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57a0005190 0x7f57b80c65e0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f57a8011900 tx=0x7f57a8011dc0 comp rx=0 tx=0).stop 2023-10-30T12:47:54.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.940+0000 7f57bdc05640 1 -- 172.21.15.89:0/349002290 shutdown_connections 2023-10-30T12:47:54.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.940+0000 7f57bdc05640 1 --2- 172.21.15.89:0/349002290 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f57a40664a0 0x7f57a4068960 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:54.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.940+0000 7f57bdc05640 1 --2- 172.21.15.89:0/349002290 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57b8145990 0x7f57b8147a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:54.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.940+0000 7f57bdc05640 1 --2- 172.21.15.89:0/349002290 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57b80cdca0 0x7f57b80ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:54.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.940+0000 7f57bdc05640 1 --2- 172.21.15.89:0/349002290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57a0005190 0x7f57b80c65e0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:54.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.940+0000 7f57bdc05640 1 -- 172.21.15.89:0/349002290 >> 172.21.15.89:0/349002290 conn(0x7f57b806bfe0 msgr2=0x7f57b8158a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:54.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.940+0000 7f57bdc05640 1 -- 172.21.15.89:0/349002290 shutdown_connections 2023-10-30T12:47:54.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:54.940+0000 7f57bdc05640 1 -- 172.21.15.89:0/349002290 wait complete. 2023-10-30T12:47:54.959 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=1) 2023-10-30T12:47:54.959 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-30T12:47:54.959 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:47:54.983 INFO:tasks.ceph.mds.f:Stopped 2023-10-30T12:47:54.983 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:47:55.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.128+0000 7f385c33e640 1 Processor -- start 2023-10-30T12:47:55.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.128+0000 7f385c33e640 1 -- start start 2023-10-30T12:47:55.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.128+0000 7f385c33e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f38540c8460 0x7f38540715c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:55.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.128+0000 7f385c33e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3854070650 0x7f3854159810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:55.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.128+0000 7f385c33e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3854159d50 con 0x7f3854157010 2023-10-30T12:47:55.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.128+0000 7f385c33e640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38540c21b0 con 0x7f3854070650 2023-10-30T12:47:55.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.128+0000 7f385c33e640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38540c1530 con 0x7f38540c8460 2023-10-30T12:47:55.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.132+0000 7f385a8db640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3854070650 0x7f3854159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:55.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.132+0000 7f385a8db640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3854070650 0x7f3854159810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39220/0 (socket says 172.21.15.89:39220) 2023-10-30T12:47:55.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.132+0000 7f385a8db640 1 -- 172.21.15.89:0/3980401135 learned_addr learned my addr 172.21.15.89:0/3980401135 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:47:55.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.132+0000 7f38598d9640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f38540c8460 0x7f38540715c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:55.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.132+0000 7f385a8db640 1 -- 172.21.15.89:0/3980401135 >> v2:172.21.15.192:3300/0 conn(0x7f38540c8460 msgr2=0x7f38540715c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:55.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.132+0000 7f385a8db640 1 --2- 172.21.15.89:0/3980401135 >> v2:172.21.15.192:3300/0 conn(0x7f38540c8460 0x7f38540715c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:55.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.132+0000 7f385a8db640 1 -- 172.21.15.89:0/3980401135 >> v1:172.21.15.192:6789/0 conn(0x7f3854157010 legacy=0x7f3854071bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:55.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.132+0000 7f385a8db640 1 -- 172.21.15.89:0/3980401135 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38540c0f70 con 0x7f3854070650 2023-10-30T12:47:55.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.132+0000 7f385a8db640 1 --2- 172.21.15.89:0/3980401135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3854070650 0x7f3854159810 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f385406c3f0 tx=0x7f384c01c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca4ae1d2cb375e56 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:55.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.132+0000 7f38590d8640 1 -- 172.21.15.89:0/3980401135 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384c0a5780 con 0x7f3854070650 2023-10-30T12:47:55.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.132+0000 7f38590d8640 1 -- 172.21.15.89:0/3980401135 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f384c0a58e0 con 0x7f3854070650 2023-10-30T12:47:55.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.132+0000 7f38590d8640 1 -- 172.21.15.89:0/3980401135 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384c0a5c70 con 0x7f3854070650 2023-10-30T12:47:55.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.136+0000 7f385c33e640 1 -- 172.21.15.89:0/3980401135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3854070650 msgr2=0x7f3854159810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:55.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.136+0000 7f385c33e640 1 --2- 172.21.15.89:0/3980401135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3854070650 0x7f3854159810 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f385406c3f0 tx=0x7f384c01c9d0 comp rx=0 tx=0).stop 2023-10-30T12:47:55.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.136+0000 7f385c33e640 1 -- 172.21.15.89:0/3980401135 shutdown_connections 2023-10-30T12:47:55.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.136+0000 7f385c33e640 1 --2- 172.21.15.89:0/3980401135 >> v2:172.21.15.192:3300/0 conn(0x7f38540c8460 0x7f38540715c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:55.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.136+0000 7f385c33e640 1 --2- 172.21.15.89:0/3980401135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3854070650 0x7f3854159810 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:55.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.136+0000 7f385c33e640 1 -- 172.21.15.89:0/3980401135 >> 172.21.15.89:0/3980401135 conn(0x7f38540cbeb0 msgr2=0x7f38540cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:55.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.136+0000 7f385c33e640 1 -- 172.21.15.89:0/3980401135 shutdown_connections 2023-10-30T12:47:55.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.136+0000 7f385c33e640 1 -- 172.21.15.89:0/3980401135 wait complete. 2023-10-30T12:47:55.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.136+0000 7f385c33e640 1 Processor -- start 2023-10-30T12:47:55.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.136+0000 7f385c33e640 1 -- start start 2023-10-30T12:47:55.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.136+0000 7f385c33e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3854070650 0x7f38541f59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:55.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.136+0000 7f385c33e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f38540c8460 0x7f38541f1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:55.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.136+0000 7f385c33e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3854157010 0x7f3854150b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:55.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.136+0000 7f385c33e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38541f2950 con 0x7f3854070650 2023-10-30T12:47:55.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.136+0000 7f385c33e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385414c250 con 0x7f38540c8460 2023-10-30T12:47:55.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.136+0000 7f385c33e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f385414c530 con 0x7f3854157010 2023-10-30T12:47:55.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.136+0000 7f385a0da640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3854070650 0x7f38541f59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:55.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.136+0000 7f385a0da640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3854070650 0x7f38541f59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39236/0 (socket says 172.21.15.89:39236) 2023-10-30T12:47:55.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.136+0000 7f385a0da640 1 -- 172.21.15.89:0/1643882427 learned_addr learned my addr 172.21.15.89:0/1643882427 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:47:55.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.136+0000 7f38598d9640 1 --2- 172.21.15.89:0/1643882427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f38540c8460 0x7f38541f1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:55.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.136+0000 7f385a8db640 1 --2- 172.21.15.89:0/1643882427 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3854157010 0x7f3854150b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:55.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.140+0000 7f385a0da640 1 -- 172.21.15.89:0/1643882427 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3854157010 msgr2=0x7f3854150b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:55.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.140+0000 7f385a0da640 1 --2- 172.21.15.89:0/1643882427 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3854157010 0x7f3854150b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:55.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.140+0000 7f385a0da640 1 -- 172.21.15.89:0/1643882427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f38540c8460 msgr2=0x7f38541f1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:55.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.140+0000 7f385a0da640 1 --2- 172.21.15.89:0/1643882427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f38540c8460 0x7f38541f1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:55.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.140+0000 7f385a0da640 1 -- 172.21.15.89:0/1643882427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f384c009b30 con 0x7f3854070650 2023-10-30T12:47:55.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.140+0000 7f385a0da640 1 --2- 172.21.15.89:0/1643882427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3854070650 0x7f38541f59a0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f385000c500 tx=0x7f385000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:55.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.140+0000 7f384affd640 1 -- 172.21.15.89:0/1643882427 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3850005220 con 0x7f3854070650 2023-10-30T12:47:55.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.140+0000 7f384affd640 1 -- 172.21.15.89:0/1643882427 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3850005380 con 0x7f3854070650 2023-10-30T12:47:55.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.140+0000 7f385c33e640 1 -- 172.21.15.89:0/1643882427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f385414c810 con 0x7f3854070650 2023-10-30T12:47:55.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.140+0000 7f384affd640 1 -- 172.21.15.89:0/1643882427 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3850007270 con 0x7f3854070650 2023-10-30T12:47:55.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.140+0000 7f385c33e640 1 -- 172.21.15.89:0/1643882427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38541f51a0 con 0x7f3854070650 2023-10-30T12:47:55.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.140+0000 7f385c33e640 1 -- 172.21.15.89:0/1643882427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f385404e170 con 0x7f3854070650 2023-10-30T12:47:55.144 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.140+0000 7f384affd640 1 -- 172.21.15.89:0/1643882427 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f38500073d0 con 0x7f3854070650 2023-10-30T12:47:55.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.140+0000 7f384affd640 1 --2- 172.21.15.89:0/1643882427 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3838066470 0x7f3838068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:55.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.140+0000 7f384affd640 1 -- 172.21.15.89:0/1643882427 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6492+0+0 (secure 0 0 0) 0x7f3850083b90 con 0x7f3854070650 2023-10-30T12:47:55.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.140+0000 7f38598d9640 1 --2- 172.21.15.89:0/1643882427 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3838066470 0x7f3838068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:55.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.144+0000 7f38598d9640 1 --2- 172.21.15.89:0/1643882427 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3838066470 0x7f3838068930 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f385414dea0 tx=0x7f3844008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:55.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.144+0000 7f384affd640 1 -- 172.21.15.89:0/1643882427 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3850050c70 con 0x7f3854070650 2023-10-30T12:47:55.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.332+0000 7f385c33e640 1 -- 172.21.15.89:0/1643882427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38540c6640 con 0x7f3854070650 2023-10-30T12:47:55.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.332+0000 7f384affd640 1 -- 172.21.15.89:0/1643882427 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f3850054290 con 0x7f3854070650 2023-10-30T12:47:55.331 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:47:55.332 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6075,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:47:10.393441+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5450,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":1,"incarnation":83,"state":"up:active","state_seq":36,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:47:55.332 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 91 2023-10-30T12:47:55.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.332+0000 7f385c33e640 1 -- 172.21.15.89:0/1643882427 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3838066470 msgr2=0x7f3838068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:55.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.332+0000 7f385c33e640 1 --2- 172.21.15.89:0/1643882427 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3838066470 0x7f3838068930 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f385414dea0 tx=0x7f3844008040 comp rx=0 tx=0).stop 2023-10-30T12:47:55.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.332+0000 7f385c33e640 1 -- 172.21.15.89:0/1643882427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3854070650 msgr2=0x7f38541f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:55.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.332+0000 7f385c33e640 1 --2- 172.21.15.89:0/1643882427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3854070650 0x7f38541f59a0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f385000c500 tx=0x7f385000c9c0 comp rx=0 tx=0).stop 2023-10-30T12:47:55.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.332+0000 7f385c33e640 1 -- 172.21.15.89:0/1643882427 shutdown_connections 2023-10-30T12:47:55.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.332+0000 7f385c33e640 1 --2- 172.21.15.89:0/1643882427 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3838066470 0x7f3838068930 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:55.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.332+0000 7f385c33e640 1 --2- 172.21.15.89:0/1643882427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f38540c8460 0x7f38541f1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:55.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.332+0000 7f385c33e640 1 --2- 172.21.15.89:0/1643882427 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3854157010 0x7f3854150b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:55.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.332+0000 7f385c33e640 1 --2- 172.21.15.89:0/1643882427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3854070650 0x7f38541f59a0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:55.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.332+0000 7f385c33e640 1 -- 172.21.15.89:0/1643882427 >> 172.21.15.89:0/1643882427 conn(0x7f38540cbeb0 msgr2=0x7f38540cc290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:55.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.332+0000 7f385c33e640 1 -- 172.21.15.89:0/1643882427 shutdown_connections 2023-10-30T12:47:55.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:55.332+0000 7f385c33e640 1 -- 172.21.15.89:0/1643882427 wait complete. 2023-10-30T12:47:55.352 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:47:57.353 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:47:57.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.504+0000 7fd2fa11d640 1 Processor -- start 2023-10-30T12:47:57.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.504+0000 7fd2fa11d640 1 -- start start 2023-10-30T12:47:57.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.504+0000 7fd2fa11d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd2f40c6fe0 0x7fd2f4071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:57.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.504+0000 7fd2fa11d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2f40c5f60 0x7fd2f40cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:57.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.504+0000 7fd2fa11d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2f40700b0 con 0x7fd2f4153bf0 2023-10-30T12:47:57.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.504+0000 7fd2fa11d640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2f40c6810 con 0x7fd2f40c6fe0 2023-10-30T12:47:57.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.504+0000 7fd2fa11d640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f4159900 con 0x7fd2f40c5f60 2023-10-30T12:47:57.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.504+0000 7fd2f2ffd640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd2f40c6fe0 0x7fd2f4071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:57.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.504+0000 7fd2f2ffd640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fd2f40c6fe0 0x7fd2f4071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39248/0 (socket says 172.21.15.89:39248) 2023-10-30T12:47:57.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.504+0000 7fd2f2ffd640 1 -- 172.21.15.89:0/49230304 learned_addr learned my addr 172.21.15.89:0/49230304 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:47:57.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.504+0000 7fd2f3fff640 1 --2- 172.21.15.89:0/49230304 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2f40c5f60 0x7fd2f40cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:57.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.504+0000 7fd2f2ffd640 1 -- 172.21.15.89:0/49230304 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2f40c5f60 msgr2=0x7fd2f40cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:57.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.504+0000 7fd2f2ffd640 1 --2- 172.21.15.89:0/49230304 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2f40c5f60 0x7fd2f40cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:57.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.504+0000 7fd2f2ffd640 1 -- 172.21.15.89:0/49230304 >> v1:172.21.15.192:6789/0 conn(0x7fd2f4153bf0 legacy=0x7fd2f40c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:57.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.504+0000 7fd2f2ffd640 1 -- 172.21.15.89:0/49230304 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f40c6530 con 0x7fd2f40c6fe0 2023-10-30T12:47:57.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.508+0000 7fd2f2ffd640 1 --2- 172.21.15.89:0/49230304 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2f40c6fe0 0x7fd2f4071260 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8009870 tx=0x7fd2e801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebd4bfd089aae7cf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:57.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.508+0000 7fd2f27fc640 1 -- 172.21.15.89:0/49230304 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e80a8a30 con 0x7fd2f40c6fe0 2023-10-30T12:47:57.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.508+0000 7fd2f27fc640 1 -- 172.21.15.89:0/49230304 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e80a8b90 con 0x7fd2f40c6fe0 2023-10-30T12:47:57.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.508+0000 7fd2f27fc640 1 -- 172.21.15.89:0/49230304 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e80b0440 con 0x7fd2f40c6fe0 2023-10-30T12:47:57.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.508+0000 7fd2fa11d640 1 -- 172.21.15.89:0/49230304 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2f40c6fe0 msgr2=0x7fd2f4071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:57.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.508+0000 7fd2fa11d640 1 --2- 172.21.15.89:0/49230304 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2f40c6fe0 0x7fd2f4071260 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8009870 tx=0x7fd2e801ed30 comp rx=0 tx=0).stop 2023-10-30T12:47:57.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.508+0000 7fd2fa11d640 1 -- 172.21.15.89:0/49230304 shutdown_connections 2023-10-30T12:47:57.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.508+0000 7fd2fa11d640 1 --2- 172.21.15.89:0/49230304 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2f40c5f60 0x7fd2f40cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:57.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.508+0000 7fd2fa11d640 1 --2- 172.21.15.89:0/49230304 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2f40c6fe0 0x7fd2f4071260 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:57.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.508+0000 7fd2fa11d640 1 -- 172.21.15.89:0/49230304 >> 172.21.15.89:0/49230304 conn(0x7fd2f406c6e0 msgr2=0x7fd2f4069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:57.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.508+0000 7fd2fa11d640 1 -- 172.21.15.89:0/49230304 shutdown_connections 2023-10-30T12:47:57.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.508+0000 7fd2fa11d640 1 -- 172.21.15.89:0/49230304 wait complete. 2023-10-30T12:47:57.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.512+0000 7fd2fa11d640 1 Processor -- start 2023-10-30T12:47:57.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.512+0000 7fd2fa11d640 1 -- start start 2023-10-30T12:47:57.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.512+0000 7fd2fa11d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2f40c5f60 0x7fd2f4149a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:57.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.512+0000 7fd2fa11d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2f40c6fe0 0x7fd2f4149f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:57.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.512+0000 7fd2fa11d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2f4153bf0 0x7fd2f414dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:57.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.512+0000 7fd2fa11d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f414f570 con 0x7fd2f40c5f60 2023-10-30T12:47:57.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.512+0000 7fd2fa11d640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f414c930 con 0x7fd2f40c6fe0 2023-10-30T12:47:57.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.512+0000 7fd2fa11d640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f414bc00 con 0x7fd2f4153bf0 2023-10-30T12:47:57.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.512+0000 7fd2f37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2f40c5f60 0x7fd2f4149a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:57.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.512+0000 7fd2f37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2f40c5f60 0x7fd2f4149a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39258/0 (socket says 172.21.15.89:39258) 2023-10-30T12:47:57.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.512+0000 7fd2f37fe640 1 -- 172.21.15.89:0/2865604194 learned_addr learned my addr 172.21.15.89:0/2865604194 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:47:57.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.512+0000 7fd2f3fff640 1 --2- 172.21.15.89:0/2865604194 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2f4153bf0 0x7fd2f414dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:57.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.512+0000 7fd2f2ffd640 1 --2- 172.21.15.89:0/2865604194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2f40c6fe0 0x7fd2f4149f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:57.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.512+0000 7fd2f37fe640 1 -- 172.21.15.89:0/2865604194 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2f4153bf0 msgr2=0x7fd2f414dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:57.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.512+0000 7fd2f37fe640 1 --2- 172.21.15.89:0/2865604194 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2f4153bf0 0x7fd2f414dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:57.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.512+0000 7fd2f37fe640 1 -- 172.21.15.89:0/2865604194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2f40c6fe0 msgr2=0x7fd2f4149f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:57.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.512+0000 7fd2f37fe640 1 --2- 172.21.15.89:0/2865604194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2f40c6fe0 0x7fd2f4149f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:57.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.512+0000 7fd2f37fe640 1 -- 172.21.15.89:0/2865604194 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e80094d0 con 0x7fd2f40c5f60 2023-10-30T12:47:57.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.512+0000 7fd2f37fe640 1 --2- 172.21.15.89:0/2865604194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2f40c5f60 0x7fd2f4149a10 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fd2e400c530 tx=0x7fd2e400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:57.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.512+0000 7fd2dbfff640 1 -- 172.21.15.89:0/2865604194 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4005220 con 0x7fd2f40c5f60 2023-10-30T12:47:57.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.512+0000 7fd2dbfff640 1 -- 172.21.15.89:0/2865604194 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e4005380 con 0x7fd2f40c5f60 2023-10-30T12:47:57.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.512+0000 7fd2dbfff640 1 -- 172.21.15.89:0/2865604194 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4007270 con 0x7fd2f40c5f60 2023-10-30T12:47:57.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.512+0000 7fd2fa11d640 1 -- 172.21.15.89:0/2865604194 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2f41e6600 con 0x7fd2f40c5f60 2023-10-30T12:47:57.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.512+0000 7fd2fa11d640 1 -- 172.21.15.89:0/2865604194 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f414be50 con 0x7fd2f40c5f60 2023-10-30T12:47:57.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.516+0000 7fd2fa11d640 1 -- 172.21.15.89:0/2865604194 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2f404e170 con 0x7fd2f40c5f60 2023-10-30T12:47:57.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.516+0000 7fd2dbfff640 1 -- 172.21.15.89:0/2865604194 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd2e40073d0 con 0x7fd2f40c5f60 2023-10-30T12:47:57.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.516+0000 7fd2dbfff640 1 --2- 172.21.15.89:0/2865604194 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd2e00664a0 0x7fd2e0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:57.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.516+0000 7fd2dbfff640 1 -- 172.21.15.89:0/2865604194 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6492+0+0 (secure 0 0 0) 0x7fd2e4083960 con 0x7fd2f40c5f60 2023-10-30T12:47:57.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.516+0000 7fd2f2ffd640 1 --2- 172.21.15.89:0/2865604194 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd2e00664a0 0x7fd2e0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:57.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.516+0000 7fd2f2ffd640 1 --2- 172.21.15.89:0/2865604194 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd2e00664a0 0x7fd2e0068960 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd2e80b2000 tx=0x7fd2e8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:57.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.520+0000 7fd2dbfff640 1 -- 172.21.15.89:0/2865604194 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd2e4050b80 con 0x7fd2f40c5f60 2023-10-30T12:47:57.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.704+0000 7fd2fa11d640 1 -- 172.21.15.89:0/2865604194 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2f414f6b0 con 0x7fd2f40c5f60 2023-10-30T12:47:57.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.704+0000 7fd2dbfff640 1 -- 172.21.15.89:0/2865604194 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6538 (secure 0 0 0) 0x7fd2e40541a0 con 0x7fd2f40c5f60 2023-10-30T12:47:57.703 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:47:57.704 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6075,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:47:10.393441+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5450,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":1,"incarnation":83,"state":"up:active","state_seq":36,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:47:57.704 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 91 2023-10-30T12:47:57.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.704+0000 7fd2fa11d640 1 -- 172.21.15.89:0/2865604194 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd2e00664a0 msgr2=0x7fd2e0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:57.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.704+0000 7fd2fa11d640 1 --2- 172.21.15.89:0/2865604194 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd2e00664a0 0x7fd2e0068960 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd2e80b2000 tx=0x7fd2e8009040 comp rx=0 tx=0).stop 2023-10-30T12:47:57.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.704+0000 7fd2fa11d640 1 -- 172.21.15.89:0/2865604194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2f40c5f60 msgr2=0x7fd2f4149a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:57.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.704+0000 7fd2fa11d640 1 --2- 172.21.15.89:0/2865604194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2f40c5f60 0x7fd2f4149a10 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fd2e400c530 tx=0x7fd2e400c9f0 comp rx=0 tx=0).stop 2023-10-30T12:47:57.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.704+0000 7fd2fa11d640 1 -- 172.21.15.89:0/2865604194 shutdown_connections 2023-10-30T12:47:57.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.704+0000 7fd2fa11d640 1 --2- 172.21.15.89:0/2865604194 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd2e00664a0 0x7fd2e0068960 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:57.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.704+0000 7fd2fa11d640 1 --2- 172.21.15.89:0/2865604194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd2f40c6fe0 0x7fd2f4149f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:57.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.704+0000 7fd2fa11d640 1 --2- 172.21.15.89:0/2865604194 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd2f4153bf0 0x7fd2f414dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:57.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.704+0000 7fd2fa11d640 1 --2- 172.21.15.89:0/2865604194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2f40c5f60 0x7fd2f4149a10 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:57.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.704+0000 7fd2fa11d640 1 -- 172.21.15.89:0/2865604194 >> 172.21.15.89:0/2865604194 conn(0x7fd2f406c6e0 msgr2=0x7fd2f40c7430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:57.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.704+0000 7fd2fa11d640 1 -- 172.21.15.89:0/2865604194 shutdown_connections 2023-10-30T12:47:57.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:57.704+0000 7fd2fa11d640 1 -- 172.21.15.89:0/2865604194 wait complete. 2023-10-30T12:47:57.724 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:47:59.725 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:47:59.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.864+0000 7f35aaf0f640 1 Processor -- start 2023-10-30T12:47:59.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.864+0000 7f35aaf0f640 1 -- start start 2023-10-30T12:47:59.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.868+0000 7f35aaf0f640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f35a406f7a0 0x7f35a406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:59.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.868+0000 7f35aaf0f640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a4066cf0 con 0x7f35a4071110 2023-10-30T12:47:59.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.868+0000 7f35aaf0f640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a40c70d0 con 0x7f35a40c6bd0 2023-10-30T12:47:59.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.868+0000 7f35aaf0f640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a40700c0 con 0x7f35a406f7a0 2023-10-30T12:47:59.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.868+0000 7f35a8cab640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f35a406f7a0 0x7f35a406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:59.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.868+0000 7f35a94ac640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f35a4071110 0x7f35a4158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47796/0 (socket says 172.21.15.89:47796) 2023-10-30T12:47:59.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.868+0000 7f35a94ac640 1 -- 172.21.15.89:0/4066140740 learned_addr learned my addr 172.21.15.89:0/4066140740 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:47:59.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.868+0000 7f35a8cab640 1 -- 172.21.15.89:0/4066140740 >> v1:172.21.15.192:6789/0 conn(0x7f35a40c6bd0 legacy=0x7f35a4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:59.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.868+0000 7f35a8cab640 1 -- 172.21.15.89:0/4066140740 >> v1:172.21.15.89:6789/0 conn(0x7f35a4071110 legacy=0x7f35a4158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:59.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.868+0000 7f35a8cab640 1 -- 172.21.15.89:0/4066140740 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a40c7390 con 0x7f35a406f7a0 2023-10-30T12:47:59.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.868+0000 7f359b7fe640 1 -- 172.21.15.89:0/4066140740 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3495201262 0 0) 0x7f35a4066cf0 con 0x7f35a4071110 2023-10-30T12:47:59.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.872+0000 7f35a8cab640 1 --2- 172.21.15.89:0/4066140740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a406f7a0 0x7f35a406fb80 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f35a00097c0 tx=0x7f35a001ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acaea1e14753ac5b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:59.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.872+0000 7f359b7fe640 1 -- 172.21.15.89:0/4066140740 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a00a8930 con 0x7f35a406f7a0 2023-10-30T12:47:59.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.872+0000 7f359b7fe640 1 -- 172.21.15.89:0/4066140740 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a00a8a90 con 0x7f35a406f7a0 2023-10-30T12:47:59.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.872+0000 7f359b7fe640 1 -- 172.21.15.89:0/4066140740 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a00a8e20 con 0x7f35a406f7a0 2023-10-30T12:47:59.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.872+0000 7f35aaf0f640 1 -- 172.21.15.89:0/4066140740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a406f7a0 msgr2=0x7f35a406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:59.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.872+0000 7f35aaf0f640 1 --2- 172.21.15.89:0/4066140740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a406f7a0 0x7f35a406fb80 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f35a00097c0 tx=0x7f35a001ec30 comp rx=0 tx=0).stop 2023-10-30T12:47:59.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.872+0000 7f35aaf0f640 1 -- 172.21.15.89:0/4066140740 shutdown_connections 2023-10-30T12:47:59.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.872+0000 7f35aaf0f640 1 --2- 172.21.15.89:0/4066140740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a406f7a0 0x7f35a406fb80 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:59.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.872+0000 7f35aaf0f640 1 -- 172.21.15.89:0/4066140740 >> 172.21.15.89:0/4066140740 conn(0x7f35a40c7910 msgr2=0x7f35a40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:59.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.872+0000 7f35aaf0f640 1 -- 172.21.15.89:0/4066140740 shutdown_connections 2023-10-30T12:47:59.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.872+0000 7f35aaf0f640 1 -- 172.21.15.89:0/4066140740 wait complete. 2023-10-30T12:47:59.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.872+0000 7f35aaf0f640 1 Processor -- start 2023-10-30T12:47:59.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.872+0000 7f35aaf0f640 1 -- start start 2023-10-30T12:47:59.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.876+0000 7f35aaf0f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a406f7a0 0x7f35a4152cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:59.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.876+0000 7f35aaf0f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35a4071110 0x7f35a41f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:59.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.876+0000 7f35aaf0f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35a40c6bd0 0x7f35a4157270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:59.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.876+0000 7f35aaf0f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a4155f00 con 0x7f35a406f7a0 2023-10-30T12:47:59.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.876+0000 7f35aaf0f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a40af210 con 0x7f35a40c6bd0 2023-10-30T12:47:59.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.876+0000 7f35aaf0f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a40af4f0 con 0x7f35a4071110 2023-10-30T12:47:59.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.876+0000 7f35a8cab640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a406f7a0 0x7f35a4152cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:59.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.876+0000 7f35a8cab640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a406f7a0 0x7f35a4152cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54158/0 (socket says 172.21.15.89:54158) 2023-10-30T12:47:59.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.876+0000 7f35a8cab640 1 -- 172.21.15.89:0/1345167193 learned_addr learned my addr 172.21.15.89:0/1345167193 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:47:59.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.876+0000 7f359bfff640 1 --2- 172.21.15.89:0/1345167193 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35a4071110 0x7f35a41f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:59.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.876+0000 7f35a94ac640 1 --2- 172.21.15.89:0/1345167193 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35a40c6bd0 0x7f35a4157270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:59.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.876+0000 7f35a8cab640 1 -- 172.21.15.89:0/1345167193 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35a4071110 msgr2=0x7f35a41f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:59.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.876+0000 7f35a8cab640 1 --2- 172.21.15.89:0/1345167193 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35a4071110 0x7f35a41f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:59.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.876+0000 7f35a8cab640 1 -- 172.21.15.89:0/1345167193 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35a40c6bd0 msgr2=0x7f35a4157270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:59.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.876+0000 7f35a8cab640 1 --2- 172.21.15.89:0/1345167193 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35a40c6bd0 0x7f35a4157270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:59.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.876+0000 7f35a8cab640 1 -- 172.21.15.89:0/1345167193 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a0009420 con 0x7f35a406f7a0 2023-10-30T12:47:59.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.876+0000 7f359bfff640 1 --2- 172.21.15.89:0/1345167193 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35a4071110 0x7f35a41f20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:47:59.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.876+0000 7f35a8cab640 1 --2- 172.21.15.89:0/1345167193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a406f7a0 0x7f35a4152cf0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f35a00b3040 tx=0x7f35a001b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:59.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.876+0000 7f35997fa640 1 -- 172.21.15.89:0/1345167193 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a00b0e40 con 0x7f35a406f7a0 2023-10-30T12:47:59.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.876+0000 7f35997fa640 1 -- 172.21.15.89:0/1345167193 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a00b9410 con 0x7f35a406f7a0 2023-10-30T12:47:59.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.876+0000 7f35997fa640 1 -- 172.21.15.89:0/1345167193 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a00b97c0 con 0x7f35a406f7a0 2023-10-30T12:47:59.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.876+0000 7f35aaf0f640 1 -- 172.21.15.89:0/1345167193 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a41f2a50 con 0x7f35a406f7a0 2023-10-30T12:47:59.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.876+0000 7f35aaf0f640 1 -- 172.21.15.89:0/1345167193 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a41563f0 con 0x7f35a406f7a0 2023-10-30T12:47:59.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.876+0000 7f35aaf0f640 1 -- 172.21.15.89:0/1345167193 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a404e170 con 0x7f35a406f7a0 2023-10-30T12:47:59.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.876+0000 7f35997fa640 1 -- 172.21.15.89:0/1345167193 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f35a00b7040 con 0x7f35a406f7a0 2023-10-30T12:47:59.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.880+0000 7f35997fa640 1 --2- 172.21.15.89:0/1345167193 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f35900664d0 0x7f3590068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:59.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.880+0000 7f35997fa640 1 -- 172.21.15.89:0/1345167193 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6492+0+0 (secure 0 0 0) 0x7f35a0121fe0 con 0x7f35a406f7a0 2023-10-30T12:47:59.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.880+0000 7f359bfff640 1 --2- 172.21.15.89:0/1345167193 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f35900664d0 0x7f3590068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:59.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.880+0000 7f359bfff640 1 --2- 172.21.15.89:0/1345167193 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f35900664d0 0x7f3590068990 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f35a40cac00 tx=0x7f3594009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:59.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:47:59.884+0000 7f35997fa640 1 -- 172.21.15.89:0/1345167193 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f35a00ef140 con 0x7f35a406f7a0 2023-10-30T12:48:00.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:00.064+0000 7f35aaf0f640 1 -- 172.21.15.89:0/1345167193 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35a4150230 con 0x7f35a406f7a0 2023-10-30T12:48:00.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:00.064+0000 7f35997fa640 1 -- 172.21.15.89:0/1345167193 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f35a00c7090 con 0x7f35a406f7a0 2023-10-30T12:48:00.065 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:48:00.066 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6075,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:47:10.393441+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5450,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":1,"incarnation":83,"state":"up:active","state_seq":36,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:00.066 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 91 2023-10-30T12:48:00.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:00.068+0000 7f35aaf0f640 1 -- 172.21.15.89:0/1345167193 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f35900664d0 msgr2=0x7f3590068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:00.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:00.068+0000 7f35aaf0f640 1 --2- 172.21.15.89:0/1345167193 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f35900664d0 0x7f3590068990 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f35a40cac00 tx=0x7f3594009040 comp rx=0 tx=0).stop 2023-10-30T12:48:00.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:00.068+0000 7f35aaf0f640 1 -- 172.21.15.89:0/1345167193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a406f7a0 msgr2=0x7f35a4152cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:00.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:00.068+0000 7f35aaf0f640 1 --2- 172.21.15.89:0/1345167193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a406f7a0 0x7f35a4152cf0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f35a00b3040 tx=0x7f35a001b9d0 comp rx=0 tx=0).stop 2023-10-30T12:48:00.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:00.068+0000 7f35aaf0f640 1 -- 172.21.15.89:0/1345167193 shutdown_connections 2023-10-30T12:48:00.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:00.068+0000 7f35aaf0f640 1 --2- 172.21.15.89:0/1345167193 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f35900664d0 0x7f3590068990 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:00.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:00.068+0000 7f35aaf0f640 1 --2- 172.21.15.89:0/1345167193 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35a40c6bd0 0x7f35a4157270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:00.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:00.068+0000 7f35aaf0f640 1 --2- 172.21.15.89:0/1345167193 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35a4071110 0x7f35a41f20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:00.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:00.068+0000 7f35aaf0f640 1 --2- 172.21.15.89:0/1345167193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a406f7a0 0x7f35a4152cf0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:00.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:00.068+0000 7f35aaf0f640 1 -- 172.21.15.89:0/1345167193 >> 172.21.15.89:0/1345167193 conn(0x7f35a40c7910 msgr2=0x7f35a40c54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:00.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:00.068+0000 7f35aaf0f640 1 -- 172.21.15.89:0/1345167193 shutdown_connections 2023-10-30T12:48:00.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:00.068+0000 7f35aaf0f640 1 -- 172.21.15.89:0/1345167193 wait complete. 2023-10-30T12:48:00.087 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:48:02.089 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:02.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.227+0000 7f3fc20c9640 1 Processor -- start 2023-10-30T12:48:02.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.227+0000 7f3fc20c9640 1 -- start start 2023-10-30T12:48:02.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.231+0000 7f3fc20c9640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3fbc0bfec0 0x7f3fbc0c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:02.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.231+0000 7f3fc20c9640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3fbc06a720 0x7f3fbc06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:02.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.231+0000 7f3fc20c9640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fbc071290 con 0x7f3fbc159e60 2023-10-30T12:48:02.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.231+0000 7f3fc20c9640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fbc071570 con 0x7f3fbc0bfec0 2023-10-30T12:48:02.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.231+0000 7f3fc20c9640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fbc071850 con 0x7f3fbc06a720 2023-10-30T12:48:02.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.231+0000 7f3fbb7fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f3fbc159e60 0x7f3fbc06f540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47808/0 (socket says 172.21.15.89:47808) 2023-10-30T12:48:02.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.231+0000 7f3fbb7fe640 1 -- 172.21.15.89:0/283444671 learned_addr learned my addr 172.21.15.89:0/283444671 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:48:02.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.231+0000 7f3fbaffd640 1 --2- 172.21.15.89:0/283444671 >> v2:172.21.15.89:3300/0 conn(0x7f3fbc0bfec0 0x7f3fbc0c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:02.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.231+0000 7f3fbbfff640 1 --2- 172.21.15.89:0/283444671 >> v2:172.21.15.192:3300/0 conn(0x7f3fbc06a720 0x7f3fbc06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:02.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.231+0000 7f3fba7fc640 1 -- 172.21.15.89:0/283444671 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2935170509 0 0) 0x7f3fbc071290 con 0x7f3fbc159e60 2023-10-30T12:48:02.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.231+0000 7f3fba7fc640 1 -- 172.21.15.89:0/283444671 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fa4003450 con 0x7f3fbc159e60 2023-10-30T12:48:02.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.231+0000 7f3fbaffd640 1 -- 172.21.15.89:0/283444671 >> v2:172.21.15.192:3300/0 conn(0x7f3fbc06a720 msgr2=0x7f3fbc06abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:02.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.231+0000 7f3fbaffd640 1 --2- 172.21.15.89:0/283444671 >> v2:172.21.15.192:3300/0 conn(0x7f3fbc06a720 0x7f3fbc06abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:02.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.231+0000 7f3fbaffd640 1 -- 172.21.15.89:0/283444671 >> v1:172.21.15.89:6789/0 conn(0x7f3fbc159e60 legacy=0x7f3fbc06f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:02.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.231+0000 7f3fbaffd640 1 -- 172.21.15.89:0/283444671 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fbc071b30 con 0x7f3fbc0bfec0 2023-10-30T12:48:02.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.231+0000 7f3fbbfff640 1 --2- 172.21.15.89:0/283444671 >> v2:172.21.15.192:3300/0 conn(0x7f3fbc06a720 0x7f3fbc06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:02.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.235+0000 7f3fbaffd640 1 --2- 172.21.15.89:0/283444671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fbc0bfec0 0x7f3fbc0c1a40 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc071e10 tx=0x7f3fb001ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6251ee8858fa6ea server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:02.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.235+0000 7f3fba7fc640 1 -- 172.21.15.89:0/283444671 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb00a8950 con 0x7f3fbc0bfec0 2023-10-30T12:48:02.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.235+0000 7f3fba7fc640 1 -- 172.21.15.89:0/283444671 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb00a8ab0 con 0x7f3fbc0bfec0 2023-10-30T12:48:02.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.235+0000 7f3fba7fc640 1 -- 172.21.15.89:0/283444671 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb00a8e20 con 0x7f3fbc0bfec0 2023-10-30T12:48:02.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.235+0000 7f3fc20c9640 1 -- 172.21.15.89:0/283444671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fbc0bfec0 msgr2=0x7f3fbc0c1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:02.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.235+0000 7f3fc20c9640 1 --2- 172.21.15.89:0/283444671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fbc0bfec0 0x7f3fbc0c1a40 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc071e10 tx=0x7f3fb001ec50 comp rx=0 tx=0).stop 2023-10-30T12:48:02.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.235+0000 7f3fc20c9640 1 -- 172.21.15.89:0/283444671 shutdown_connections 2023-10-30T12:48:02.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.235+0000 7f3fc20c9640 1 --2- 172.21.15.89:0/283444671 >> v2:172.21.15.192:3300/0 conn(0x7f3fbc06a720 0x7f3fbc06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:02.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.235+0000 7f3fc20c9640 1 --2- 172.21.15.89:0/283444671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fbc0bfec0 0x7f3fbc0c1a40 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:02.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.235+0000 7f3fc20c9640 1 -- 172.21.15.89:0/283444671 >> 172.21.15.89:0/283444671 conn(0x7f3fbc0c7910 msgr2=0x7f3fbc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:02.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.235+0000 7f3fc20c9640 1 -- 172.21.15.89:0/283444671 shutdown_connections 2023-10-30T12:48:02.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.235+0000 7f3fc20c9640 1 -- 172.21.15.89:0/283444671 wait complete. 2023-10-30T12:48:02.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.235+0000 7f3fc20c9640 1 Processor -- start 2023-10-30T12:48:02.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.235+0000 7f3fc20c9640 1 -- start start 2023-10-30T12:48:02.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.239+0000 7f3fc20c9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3fbc06a720 0x7f3fbc0c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:02.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.239+0000 7f3fc20c9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3fbc0bfec0 0x7f3fbc0c2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:02.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.239+0000 7f3fc20c9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fbc159e60 0x7f3fbc0c2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:02.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.239+0000 7f3fc20c9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc14c710 con 0x7f3fbc159e60 2023-10-30T12:48:02.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.239+0000 7f3fc20c9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc0c6fe0 con 0x7f3fbc0bfec0 2023-10-30T12:48:02.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.239+0000 7f3fc20c9640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc0c20c0 con 0x7f3fbc06a720 2023-10-30T12:48:02.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.239+0000 7f3fbbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fbc159e60 0x7f3fbc0c2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:02.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.239+0000 7f3fbbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fbc159e60 0x7f3fbc0c2a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54172/0 (socket says 172.21.15.89:54172) 2023-10-30T12:48:02.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.239+0000 7f3fbbfff640 1 -- 172.21.15.89:0/2528433474 learned_addr learned my addr 172.21.15.89:0/2528433474 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:02.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.239+0000 7f3fbaffd640 1 --2- 172.21.15.89:0/2528433474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3fbc0bfec0 0x7f3fbc0c2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:02.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.239+0000 7f3fbb7fe640 1 --2- 172.21.15.89:0/2528433474 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3fbc06a720 0x7f3fbc0c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:02.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.239+0000 7f3fbbfff640 1 -- 172.21.15.89:0/2528433474 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3fbc06a720 msgr2=0x7f3fbc0c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:02.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.239+0000 7f3fbbfff640 1 --2- 172.21.15.89:0/2528433474 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3fbc06a720 0x7f3fbc0c6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:02.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.239+0000 7f3fbbfff640 1 -- 172.21.15.89:0/2528433474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3fbc0bfec0 msgr2=0x7f3fbc0c2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:02.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.239+0000 7f3fbbfff640 1 --2- 172.21.15.89:0/2528433474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3fbc0bfec0 0x7f3fbc0c2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:02.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.239+0000 7f3fbbfff640 1 -- 172.21.15.89:0/2528433474 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb0009420 con 0x7f3fbc159e60 2023-10-30T12:48:02.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.239+0000 7f3fbbfff640 1 --2- 172.21.15.89:0/2528433474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fbc159e60 0x7f3fbc0c2a90 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f3fb4007f50 tx=0x7f3fb400d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:02.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.239+0000 7f3fa3fff640 1 -- 172.21.15.89:0/2528433474 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb4003ba0 con 0x7f3fbc159e60 2023-10-30T12:48:02.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.239+0000 7f3fa3fff640 1 -- 172.21.15.89:0/2528433474 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb400f040 con 0x7f3fbc159e60 2023-10-30T12:48:02.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.239+0000 7f3fc20c9640 1 -- 172.21.15.89:0/2528433474 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fbc0c4080 con 0x7f3fbc159e60 2023-10-30T12:48:02.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.239+0000 7f3fa3fff640 1 -- 172.21.15.89:0/2528433474 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb400dec0 con 0x7f3fbc159e60 2023-10-30T12:48:02.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.239+0000 7f3fc20c9640 1 -- 172.21.15.89:0/2528433474 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fbc0c1a70 con 0x7f3fbc159e60 2023-10-30T12:48:02.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.239+0000 7f3fc20c9640 1 -- 172.21.15.89:0/2528433474 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fbc04e170 con 0x7f3fbc159e60 2023-10-30T12:48:02.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.239+0000 7f3fa3fff640 1 -- 172.21.15.89:0/2528433474 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3fb4004030 con 0x7f3fbc159e60 2023-10-30T12:48:02.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.243+0000 7f3fa3fff640 1 --2- 172.21.15.89:0/2528433474 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3fa80664d0 0x7f3fa8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:02.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.243+0000 7f3fa3fff640 1 -- 172.21.15.89:0/2528433474 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6492+0+0 (secure 0 0 0) 0x7f3fb40844c0 con 0x7f3fbc159e60 2023-10-30T12:48:02.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.243+0000 7f3fbb7fe640 1 --2- 172.21.15.89:0/2528433474 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3fa80664d0 0x7f3fa8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:02.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.243+0000 7f3fbb7fe640 1 --2- 172.21.15.89:0/2528433474 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3fa80664d0 0x7f3fa8068990 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3fac005890 tx=0x7f3fac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:02.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.247+0000 7f3fa3fff640 1 -- 172.21.15.89:0/2528433474 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3fb4051640 con 0x7f3fbc159e60 2023-10-30T12:48:02.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.435+0000 7f3fc20c9640 1 -- 172.21.15.89:0/2528433474 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fbc0c2200 con 0x7f3fbc159e60 2023-10-30T12:48:02.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.439+0000 7f3fa3fff640 1 -- 172.21.15.89:0/2528433474 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f3fb4054c60 con 0x7f3fbc159e60 2023-10-30T12:48:02.439 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:48:02.440 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6075,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:47:10.393441+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5450,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":1,"incarnation":83,"state":"up:active","state_seq":36,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:02.440 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 91 2023-10-30T12:48:02.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.439+0000 7f3fc20c9640 1 -- 172.21.15.89:0/2528433474 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3fa80664d0 msgr2=0x7f3fa8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:02.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.439+0000 7f3fc20c9640 1 --2- 172.21.15.89:0/2528433474 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3fa80664d0 0x7f3fa8068990 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3fac005890 tx=0x7f3fac009040 comp rx=0 tx=0).stop 2023-10-30T12:48:02.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.439+0000 7f3fc20c9640 1 -- 172.21.15.89:0/2528433474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fbc159e60 msgr2=0x7f3fbc0c2a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:02.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.439+0000 7f3fc20c9640 1 --2- 172.21.15.89:0/2528433474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fbc159e60 0x7f3fbc0c2a90 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f3fb4007f50 tx=0x7f3fb400d560 comp rx=0 tx=0).stop 2023-10-30T12:48:02.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.439+0000 7f3fc20c9640 1 -- 172.21.15.89:0/2528433474 shutdown_connections 2023-10-30T12:48:02.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.439+0000 7f3fc20c9640 1 --2- 172.21.15.89:0/2528433474 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3fa80664d0 0x7f3fa8068990 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:02.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.439+0000 7f3fc20c9640 1 --2- 172.21.15.89:0/2528433474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3fbc0bfec0 0x7f3fbc0c2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:02.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.439+0000 7f3fc20c9640 1 --2- 172.21.15.89:0/2528433474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fbc159e60 0x7f3fbc0c2a90 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:02.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.439+0000 7f3fc20c9640 1 --2- 172.21.15.89:0/2528433474 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3fbc06a720 0x7f3fbc0c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:02.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.439+0000 7f3fc20c9640 1 -- 172.21.15.89:0/2528433474 >> 172.21.15.89:0/2528433474 conn(0x7f3fbc0c7910 msgr2=0x7f3fbc0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:02.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.439+0000 7f3fc20c9640 1 -- 172.21.15.89:0/2528433474 shutdown_connections 2023-10-30T12:48:02.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:02.439+0000 7f3fc20c9640 1 -- 172.21.15.89:0/2528433474 wait complete. 2023-10-30T12:48:02.460 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:48:04.461 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:04.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.607+0000 7f647ed8d640 1 Processor -- start 2023-10-30T12:48:04.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.607+0000 7f647ed8d640 1 -- start start 2023-10-30T12:48:04.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.607+0000 7f647ed8d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f64780c6b80 0x7f64780bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:04.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.607+0000 7f647ed8d640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64780c0a50 con 0x7f64780c6f60 2023-10-30T12:48:04.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.607+0000 7f647ed8d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f647806f6e0 con 0x7f647813f4a0 2023-10-30T12:48:04.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.607+0000 7f647ed8d640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f647806f9c0 con 0x7f64780c6b80 2023-10-30T12:48:04.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.607+0000 7f647cb29640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f64780c6b80 0x7f64780bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:04.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.607+0000 7f646ffff640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f64780c6f60 0x7f647813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:47820/0 (socket says 172.21.15.89:47820) 2023-10-30T12:48:04.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.607+0000 7f646ffff640 1 -- 172.21.15.89:0/1401596465 learned_addr learned my addr 172.21.15.89:0/1401596465 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:48:04.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.607+0000 7f647cb29640 1 -- 172.21.15.89:0/1401596465 >> v1:172.21.15.192:6789/0 conn(0x7f647813f4a0 legacy=0x7f64781566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:04.611 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.607+0000 7f647cb29640 1 -- 172.21.15.89:0/1401596465 >> v1:172.21.15.89:6789/0 conn(0x7f64780c6f60 legacy=0x7f647813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:04.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.607+0000 7f647cb29640 1 -- 172.21.15.89:0/1401596465 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647806fca0 con 0x7f64780c6b80 2023-10-30T12:48:04.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.607+0000 7f646f7fe640 1 -- 172.21.15.89:0/1401596465 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 175713482 0 0) 0x7f64780c0a50 con 0x7f64780c6f60 2023-10-30T12:48:04.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.611+0000 7f647cb29640 1 --2- 172.21.15.89:0/1401596465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64780c6b80 0x7f64780bfec0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f64740097c0 tx=0x7f647401c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de309d771aa51f72 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:04.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.611+0000 7f646f7fe640 1 -- 172.21.15.89:0/1401596465 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6474004230 con 0x7f64780c6b80 2023-10-30T12:48:04.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.611+0000 7f646f7fe640 1 -- 172.21.15.89:0/1401596465 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64740a7050 con 0x7f64780c6b80 2023-10-30T12:48:04.613 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.611+0000 7f646f7fe640 1 -- 172.21.15.89:0/1401596465 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6474003620 con 0x7f64780c6b80 2023-10-30T12:48:04.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.611+0000 7f647ed8d640 1 -- 172.21.15.89:0/1401596465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64780c6b80 msgr2=0x7f64780bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:04.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.611+0000 7f647ed8d640 1 --2- 172.21.15.89:0/1401596465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64780c6b80 0x7f64780bfec0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f64740097c0 tx=0x7f647401c440 comp rx=0 tx=0).stop 2023-10-30T12:48:04.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.611+0000 7f647ed8d640 1 -- 172.21.15.89:0/1401596465 shutdown_connections 2023-10-30T12:48:04.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.611+0000 7f647ed8d640 1 --2- 172.21.15.89:0/1401596465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64780c6b80 0x7f64780bfec0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:04.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.611+0000 7f647ed8d640 1 -- 172.21.15.89:0/1401596465 >> 172.21.15.89:0/1401596465 conn(0x7f64780c7910 msgr2=0x7f64780c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:04.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.611+0000 7f647ed8d640 1 -- 172.21.15.89:0/1401596465 shutdown_connections 2023-10-30T12:48:04.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.611+0000 7f647ed8d640 1 -- 172.21.15.89:0/1401596465 wait complete. 2023-10-30T12:48:04.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.615+0000 7f647ed8d640 1 Processor -- start 2023-10-30T12:48:04.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.615+0000 7f647ed8d640 1 -- start start 2023-10-30T12:48:04.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.615+0000 7f647ed8d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64780c6b80 0x7f6478134b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:04.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.615+0000 7f647ed8d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64780c6f60 0x7f64781f5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:04.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.615+0000 7f647ed8d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f647813f4a0 0x7f64781f5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:04.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.615+0000 7f647ed8d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6478137810 con 0x7f64780c6f60 2023-10-30T12:48:04.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.615+0000 7f647ed8d640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64780ca670 con 0x7f647813f4a0 2023-10-30T12:48:04.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.615+0000 7f647ed8d640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6478138a30 con 0x7f64780c6b80 2023-10-30T12:48:04.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.615+0000 7f646ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64780c6f60 0x7f64781f5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:04.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.615+0000 7f646ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64780c6f60 0x7f64781f5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54182/0 (socket says 172.21.15.89:54182) 2023-10-30T12:48:04.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.615+0000 7f646ffff640 1 -- 172.21.15.89:0/1922731171 learned_addr learned my addr 172.21.15.89:0/1922731171 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:04.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.615+0000 7f647cb29640 1 --2- 172.21.15.89:0/1922731171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64780c6b80 0x7f6478134b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:04.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.615+0000 7f647d32a640 1 --2- 172.21.15.89:0/1922731171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f647813f4a0 0x7f64781f5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:04.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.615+0000 7f646ffff640 1 -- 172.21.15.89:0/1922731171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64780c6b80 msgr2=0x7f6478134b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:04.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.615+0000 7f646ffff640 1 --2- 172.21.15.89:0/1922731171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64780c6b80 0x7f6478134b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:04.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.615+0000 7f646ffff640 1 -- 172.21.15.89:0/1922731171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f647813f4a0 msgr2=0x7f64781f5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:04.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.615+0000 7f646ffff640 1 --2- 172.21.15.89:0/1922731171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f647813f4a0 0x7f64781f5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:04.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.615+0000 7f646ffff640 1 -- 172.21.15.89:0/1922731171 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6474009420 con 0x7f64780c6f60 2023-10-30T12:48:04.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.615+0000 7f647cb29640 1 --2- 172.21.15.89:0/1922731171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64780c6b80 0x7f6478134b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:04.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.615+0000 7f646ffff640 1 --2- 172.21.15.89:0/1922731171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64780c6f60 0x7f64781f5a90 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f64680011f0 tx=0x7f646800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:04.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.615+0000 7f646d7fa640 1 -- 172.21.15.89:0/1922731171 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64680074e0 con 0x7f64780c6f60 2023-10-30T12:48:04.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.615+0000 7f646d7fa640 1 -- 172.21.15.89:0/1922731171 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6468007640 con 0x7f64780c6f60 2023-10-30T12:48:04.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.615+0000 7f647ed8d640 1 -- 172.21.15.89:0/1922731171 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6478138ca0 con 0x7f64780c6f60 2023-10-30T12:48:04.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.615+0000 7f646d7fa640 1 -- 172.21.15.89:0/1922731171 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6468004a10 con 0x7f64780c6f60 2023-10-30T12:48:04.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.615+0000 7f647ed8d640 1 -- 172.21.15.89:0/1922731171 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64780caba0 con 0x7f64780c6f60 2023-10-30T12:48:04.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.619+0000 7f647ed8d640 1 -- 172.21.15.89:0/1922731171 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6478072560 con 0x7f64780c6f60 2023-10-30T12:48:04.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.619+0000 7f646d7fa640 1 -- 172.21.15.89:0/1922731171 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6468004b70 con 0x7f64780c6f60 2023-10-30T12:48:04.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.619+0000 7f646d7fa640 1 --2- 172.21.15.89:0/1922731171 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f64640664d0 0x7f6464068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:04.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.619+0000 7f646d7fa640 1 -- 172.21.15.89:0/1922731171 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6492+0+0 (secure 0 0 0) 0x7f6468083530 con 0x7f64780c6f60 2023-10-30T12:48:04.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.619+0000 7f647cb29640 1 --2- 172.21.15.89:0/1922731171 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f64640664d0 0x7f6464068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:04.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.619+0000 7f647cb29640 1 --2- 172.21.15.89:0/1922731171 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f64640664d0 0x7f6464068990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f64740ad000 tx=0x7f6474008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:04.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.623+0000 7f646d7fa640 1 -- 172.21.15.89:0/1922731171 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f64680507e0 con 0x7f64780c6f60 2023-10-30T12:48:04.804 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.803+0000 7f647ed8d640 1 -- 172.21.15.89:0/1922731171 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f647804e170 con 0x7f64780c6f60 2023-10-30T12:48:04.805 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.803+0000 7f646d7fa640 1 -- 172.21.15.89:0/1922731171 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f6468053e00 con 0x7f64780c6f60 2023-10-30T12:48:04.805 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:48:04.805 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6075,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:47:10.393441+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5450,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":1,"incarnation":83,"state":"up:active","state_seq":36,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:04.806 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 91 2023-10-30T12:48:04.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.807+0000 7f647ed8d640 1 -- 172.21.15.89:0/1922731171 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f64640664d0 msgr2=0x7f6464068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:04.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.807+0000 7f647ed8d640 1 --2- 172.21.15.89:0/1922731171 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f64640664d0 0x7f6464068990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f64740ad000 tx=0x7f6474008040 comp rx=0 tx=0).stop 2023-10-30T12:48:04.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.807+0000 7f647ed8d640 1 -- 172.21.15.89:0/1922731171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64780c6f60 msgr2=0x7f64781f5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:04.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.807+0000 7f647ed8d640 1 --2- 172.21.15.89:0/1922731171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64780c6f60 0x7f64781f5a90 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f64680011f0 tx=0x7f646800c9b0 comp rx=0 tx=0).stop 2023-10-30T12:48:04.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.807+0000 7f647ed8d640 1 -- 172.21.15.89:0/1922731171 shutdown_connections 2023-10-30T12:48:04.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.807+0000 7f647ed8d640 1 --2- 172.21.15.89:0/1922731171 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f64640664d0 0x7f6464068990 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:04.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.807+0000 7f647ed8d640 1 --2- 172.21.15.89:0/1922731171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f647813f4a0 0x7f64781f5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:04.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.807+0000 7f647ed8d640 1 --2- 172.21.15.89:0/1922731171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64780c6f60 0x7f64781f5a90 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:04.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.807+0000 7f647ed8d640 1 --2- 172.21.15.89:0/1922731171 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64780c6b80 0x7f6478134b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:04.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.807+0000 7f647ed8d640 1 -- 172.21.15.89:0/1922731171 >> 172.21.15.89:0/1922731171 conn(0x7f64780c7910 msgr2=0x7f64780c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:04.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.807+0000 7f647ed8d640 1 -- 172.21.15.89:0/1922731171 shutdown_connections 2023-10-30T12:48:04.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:04.807+0000 7f647ed8d640 1 -- 172.21.15.89:0/1922731171 wait complete. 2023-10-30T12:48:04.826 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:48:06.827 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:06.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:06.995+0000 7f745b172640 1 Processor -- start 2023-10-30T12:48:06.992 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:06.995+0000 7f745b172640 1 -- start start 2023-10-30T12:48:06.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:06.995+0000 7f745b172640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f74541538a0 0x7f745415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:06.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:06.995+0000 7f745b172640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f74540cf160 0x7f74540cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:06.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:06.995+0000 7f745b172640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74540cb800 con 0x7f74540ca0e0 2023-10-30T12:48:06.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:06.995+0000 7f745b172640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f745415dc30 con 0x7f74541538a0 2023-10-30T12:48:06.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:06.995+0000 7f745b172640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74540cb330 con 0x7f74540cf160 2023-10-30T12:48:06.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:06.995+0000 7f745970f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f74540cf160 0x7f74540cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:06.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:06.995+0000 7f7458f0e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f74541538a0 0x7f745415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:06.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:06.995+0000 7f745970f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f74540cf160 0x7f74540cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:35476/0 (socket says 172.21.15.89:35476) 2023-10-30T12:48:06.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:06.995+0000 7f745970f640 1 -- 172.21.15.89:0/2703163653 learned_addr learned my addr 172.21.15.89:0/2703163653 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:06.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:06.995+0000 7f7453fff640 1 --1- 172.21.15.89:0/2703163653 >> v1:172.21.15.192:6789/0 conn(0x7f74540ca0e0 0x7f745415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:55878/0 (socket says 172.21.15.89:55878) 2023-10-30T12:48:06.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:06.995+0000 7f745970f640 1 -- 172.21.15.89:0/2703163653 >> v2:172.21.15.192:3300/0 conn(0x7f74541538a0 msgr2=0x7f745415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:06.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:06.995+0000 7f745970f640 1 --2- 172.21.15.89:0/2703163653 >> v2:172.21.15.192:3300/0 conn(0x7f74541538a0 0x7f745415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:06.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:06.995+0000 7f745970f640 1 -- 172.21.15.89:0/2703163653 >> v1:172.21.15.192:6789/0 conn(0x7f74540ca0e0 legacy=0x7f745415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:06.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:06.995+0000 7f745970f640 1 -- 172.21.15.89:0/2703163653 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74540cad70 con 0x7f74540cf160 2023-10-30T12:48:06.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:06.995+0000 7f7458f0e640 1 --2- 172.21.15.89:0/2703163653 >> v2:172.21.15.192:3300/0 conn(0x7f74541538a0 0x7f745415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:48:06.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:06.995+0000 7f74537fe640 1 -- 172.21.15.89:0/2703163653 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3246225681 0 0) 0x7f74540cb800 con 0x7f74540ca0e0 2023-10-30T12:48:06.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:06.999+0000 7f745970f640 1 --2- 172.21.15.89:0/2703163653 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f74540cf160 0x7f74540cf590 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f744c009e50 tx=0x7f744c01f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74ac93d3098973ea server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:06.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:06.999+0000 7f74537fe640 1 -- 172.21.15.89:0/2703163653 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744c0af400 con 0x7f74540cf160 2023-10-30T12:48:06.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:06.999+0000 7f74537fe640 1 -- 172.21.15.89:0/2703163653 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f744c0ab050 con 0x7f74540cf160 2023-10-30T12:48:06.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:06.999+0000 7f74537fe640 1 -- 172.21.15.89:0/2703163653 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744c0af780 con 0x7f74540cf160 2023-10-30T12:48:06.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:06.999+0000 7f745b172640 1 -- 172.21.15.89:0/2703163653 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f74540cf160 msgr2=0x7f74540cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:06.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:06.999+0000 7f745b172640 1 --2- 172.21.15.89:0/2703163653 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f74540cf160 0x7f74540cf590 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f744c009e50 tx=0x7f744c01f1f0 comp rx=0 tx=0).stop 2023-10-30T12:48:06.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:06.999+0000 7f745b172640 1 -- 172.21.15.89:0/2703163653 shutdown_connections 2023-10-30T12:48:06.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:06.999+0000 7f745b172640 1 --2- 172.21.15.89:0/2703163653 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f74540cf160 0x7f74540cf590 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:07.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:06.999+0000 7f745b172640 1 --2- 172.21.15.89:0/2703163653 >> v2:172.21.15.192:3300/0 conn(0x7f74541538a0 0x7f745415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:07.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:06.999+0000 7f745b172640 1 -- 172.21.15.89:0/2703163653 >> 172.21.15.89:0/2703163653 conn(0x7f745406bfe0 msgr2=0x7f745406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:07.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:06.999+0000 7f745b172640 1 -- 172.21.15.89:0/2703163653 shutdown_connections 2023-10-30T12:48:07.000 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:06.999+0000 7f745b172640 1 -- 172.21.15.89:0/2703163653 wait complete. 2023-10-30T12:48:07.002 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.003+0000 7f745b172640 1 Processor -- start 2023-10-30T12:48:07.002 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.003+0000 7f745b172640 1 -- start start 2023-10-30T12:48:07.002 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.003+0000 7f745b172640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f74540ca0e0 0x7f74540ce3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:07.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.003+0000 7f745b172640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74540cf160 0x7f745415c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:07.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.003+0000 7f745b172640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74541538a0 0x7f74541ef640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:07.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.003+0000 7f745b172640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745415cf10 con 0x7f74541538a0 2023-10-30T12:48:07.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.003+0000 7f745b172640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745415da50 con 0x7f74540cf160 2023-10-30T12:48:07.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.003+0000 7f745b172640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f745415cb60 con 0x7f74540ca0e0 2023-10-30T12:48:07.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.003+0000 7f745970f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74541538a0 0x7f74541ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:07.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.003+0000 7f7458f0e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f74540ca0e0 0x7f74540ce3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:07.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.003+0000 7f7458f0e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f74540ca0e0 0x7f74540ce3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:35488/0 (socket says 172.21.15.89:35488) 2023-10-30T12:48:07.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.003+0000 7f7458f0e640 1 -- 172.21.15.89:0/3515190579 learned_addr learned my addr 172.21.15.89:0/3515190579 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:07.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.003+0000 7f7453fff640 1 --2- 172.21.15.89:0/3515190579 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74540cf160 0x7f745415c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:07.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.003+0000 7f745970f640 1 -- 172.21.15.89:0/3515190579 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f74540ca0e0 msgr2=0x7f74540ce3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:07.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.003+0000 7f745970f640 1 --2- 172.21.15.89:0/3515190579 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f74540ca0e0 0x7f74540ce3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:07.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.003+0000 7f745970f640 1 -- 172.21.15.89:0/3515190579 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74540cf160 msgr2=0x7f745415c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:07.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.003+0000 7f745970f640 1 --2- 172.21.15.89:0/3515190579 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74540cf160 0x7f745415c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:07.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.003+0000 7f745970f640 1 -- 172.21.15.89:0/3515190579 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7448009420 con 0x7f74541538a0 2023-10-30T12:48:07.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.003+0000 7f745970f640 1 --2- 172.21.15.89:0/3515190579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74541538a0 0x7f74541ef640 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f744c0ae000 tx=0x7f744c01bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:07.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.003+0000 7f74517fa640 1 -- 172.21.15.89:0/3515190579 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744c0afcc0 con 0x7f74541538a0 2023-10-30T12:48:07.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.003+0000 7f74517fa640 1 -- 172.21.15.89:0/3515190579 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f744c0afe20 con 0x7f74541538a0 2023-10-30T12:48:07.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.003+0000 7f74517fa640 1 -- 172.21.15.89:0/3515190579 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744c003ba0 con 0x7f74541538a0 2023-10-30T12:48:07.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.003+0000 7f745b172640 1 -- 172.21.15.89:0/3515190579 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f744c009b30 con 0x7f74541538a0 2023-10-30T12:48:07.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.003+0000 7f745b172640 1 -- 172.21.15.89:0/3515190579 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f745415d400 con 0x7f74541538a0 2023-10-30T12:48:07.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.007+0000 7f745b172640 1 -- 172.21.15.89:0/3515190579 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74540b2a50 con 0x7f74541538a0 2023-10-30T12:48:07.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.007+0000 7f74517fa640 1 -- 172.21.15.89:0/3515190579 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f744c0b6070 con 0x7f74541538a0 2023-10-30T12:48:07.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.007+0000 7f74517fa640 1 --2- 172.21.15.89:0/3515190579 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f74380664a0 0x7f7438068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:07.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.007+0000 7f74517fa640 1 -- 172.21.15.89:0/3515190579 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6492+0+0 (secure 0 0 0) 0x7f744c1215e0 con 0x7f74541538a0 2023-10-30T12:48:07.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.007+0000 7f7458f0e640 1 --2- 172.21.15.89:0/3515190579 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f74380664a0 0x7f7438068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:07.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.007+0000 7f7458f0e640 1 --2- 172.21.15.89:0/3515190579 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f74380664a0 0x7f7438068960 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7448009d50 tx=0x7f7448009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:07.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.011+0000 7f74517fa640 1 -- 172.21.15.89:0/3515190579 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f744c0ee6d0 con 0x7f74541538a0 2023-10-30T12:48:07.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.191+0000 7f745b172640 1 -- 172.21.15.89:0/3515190579 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f745404e170 con 0x7f74541538a0 2023-10-30T12:48:07.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.191+0000 7f74517fa640 1 -- 172.21.15.89:0/3515190579 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f744c009510 con 0x7f74541538a0 2023-10-30T12:48:07.189 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:48:07.189 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6075,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:47:10.393441+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5450,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":1,"incarnation":83,"state":"up:active","state_seq":36,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:07.190 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 91 2023-10-30T12:48:07.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.191+0000 7f745b172640 1 -- 172.21.15.89:0/3515190579 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f74380664a0 msgr2=0x7f7438068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:07.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.191+0000 7f745b172640 1 --2- 172.21.15.89:0/3515190579 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f74380664a0 0x7f7438068960 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7448009d50 tx=0x7f7448009190 comp rx=0 tx=0).stop 2023-10-30T12:48:07.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.191+0000 7f745b172640 1 -- 172.21.15.89:0/3515190579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74541538a0 msgr2=0x7f74541ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:07.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.191+0000 7f745b172640 1 --2- 172.21.15.89:0/3515190579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74541538a0 0x7f74541ef640 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f744c0ae000 tx=0x7f744c01bf90 comp rx=0 tx=0).stop 2023-10-30T12:48:07.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.195+0000 7f745b172640 1 -- 172.21.15.89:0/3515190579 shutdown_connections 2023-10-30T12:48:07.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.195+0000 7f745b172640 1 --2- 172.21.15.89:0/3515190579 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f74380664a0 0x7f7438068960 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:07.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.195+0000 7f745b172640 1 --2- 172.21.15.89:0/3515190579 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74540cf160 0x7f745415c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:07.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.195+0000 7f745b172640 1 --2- 172.21.15.89:0/3515190579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f74541538a0 0x7f74541ef640 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:07.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.195+0000 7f745b172640 1 --2- 172.21.15.89:0/3515190579 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f74540ca0e0 0x7f74540ce3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:07.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.195+0000 7f745b172640 1 -- 172.21.15.89:0/3515190579 >> 172.21.15.89:0/3515190579 conn(0x7f745406bfe0 msgr2=0x7f7454152410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:07.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.195+0000 7f745b172640 1 -- 172.21.15.89:0/3515190579 shutdown_connections 2023-10-30T12:48:07.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:07.195+0000 7f745b172640 1 -- 172.21.15.89:0/3515190579 wait complete. 2023-10-30T12:48:07.210 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:48:09.211 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:09.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.363+0000 7f152ab1d640 1 Processor -- start 2023-10-30T12:48:09.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.363+0000 7f152ab1d640 1 -- start start 2023-10-30T12:48:09.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.363+0000 7f152ab1d640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f15240c05d0 0x7f152413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:09.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.363+0000 7f152ab1d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15240c6f60 0x7f152413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:09.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.363+0000 7f152ab1d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15240c0a50 con 0x7f15240c6b80 2023-10-30T12:48:09.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.363+0000 7f152ab1d640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f152406f6e0 con 0x7f15240c05d0 2023-10-30T12:48:09.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.363+0000 7f152ab1d640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f152406f9c0 con 0x7f15240c6f60 2023-10-30T12:48:09.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.367+0000 7f15290ba640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15240c6f60 0x7f152413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:09.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.367+0000 7f1523fff640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f15240c05d0 0x7f152413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:09.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.367+0000 7f15290ba640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15240c6f60 0x7f152413f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:35498/0 (socket says 172.21.15.89:35498) 2023-10-30T12:48:09.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.367+0000 7f15290ba640 1 -- 172.21.15.89:0/3960459905 learned_addr learned my addr 172.21.15.89:0/3960459905 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:09.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.367+0000 7f15290ba640 1 -- 172.21.15.89:0/3960459905 >> v2:172.21.15.192:3300/0 conn(0x7f15240c05d0 msgr2=0x7f152413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:09.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.367+0000 7f15290ba640 1 --2- 172.21.15.89:0/3960459905 >> v2:172.21.15.192:3300/0 conn(0x7f15240c05d0 0x7f152413ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:09.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.367+0000 7f15290ba640 1 -- 172.21.15.89:0/3960459905 >> v1:172.21.15.192:6789/0 conn(0x7f15240c6b80 legacy=0x7f15240bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:09.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.367+0000 7f15290ba640 1 -- 172.21.15.89:0/3960459905 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f152406fca0 con 0x7f15240c6f60 2023-10-30T12:48:09.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.367+0000 7f1523fff640 1 --2- 172.21.15.89:0/3960459905 >> v2:172.21.15.192:3300/0 conn(0x7f15240c05d0 0x7f152413ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:48:09.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.367+0000 7f15290ba640 1 --2- 172.21.15.89:0/3960459905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15240c6f60 0x7f152413f2d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f151c009e50 tx=0x7f151c01ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=211f6e28a875367b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:09.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.371+0000 7f15237fe640 1 -- 172.21.15.89:0/3960459905 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f151c0a57f0 con 0x7f15240c6f60 2023-10-30T12:48:09.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.371+0000 7f15237fe640 1 -- 172.21.15.89:0/3960459905 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f151c0a5950 con 0x7f15240c6f60 2023-10-30T12:48:09.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.371+0000 7f15237fe640 1 -- 172.21.15.89:0/3960459905 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f151c0a5ce0 con 0x7f15240c6f60 2023-10-30T12:48:09.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.371+0000 7f152ab1d640 1 -- 172.21.15.89:0/3960459905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15240c6f60 msgr2=0x7f152413f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:09.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.371+0000 7f152ab1d640 1 --2- 172.21.15.89:0/3960459905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15240c6f60 0x7f152413f2d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f151c009e50 tx=0x7f151c01ca10 comp rx=0 tx=0).stop 2023-10-30T12:48:09.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.371+0000 7f152ab1d640 1 -- 172.21.15.89:0/3960459905 shutdown_connections 2023-10-30T12:48:09.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.371+0000 7f152ab1d640 1 --2- 172.21.15.89:0/3960459905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15240c6f60 0x7f152413f2d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:09.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.371+0000 7f152ab1d640 1 --2- 172.21.15.89:0/3960459905 >> v2:172.21.15.192:3300/0 conn(0x7f15240c05d0 0x7f152413ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:09.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.371+0000 7f152ab1d640 1 -- 172.21.15.89:0/3960459905 >> 172.21.15.89:0/3960459905 conn(0x7f15240c7910 msgr2=0x7f15240c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:09.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.371+0000 7f152ab1d640 1 -- 172.21.15.89:0/3960459905 shutdown_connections 2023-10-30T12:48:09.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.371+0000 7f152ab1d640 1 -- 172.21.15.89:0/3960459905 wait complete. 2023-10-30T12:48:09.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.371+0000 7f152ab1d640 1 Processor -- start 2023-10-30T12:48:09.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.371+0000 7f152ab1d640 1 -- start start 2023-10-30T12:48:09.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.371+0000 7f152ab1d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15240c05d0 0x7f1524151540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:09.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.371+0000 7f152ab1d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15240c6b80 0x7f1524154860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:09.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.371+0000 7f152ab1d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15240c6f60 0x7f1524154da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:09.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.371+0000 7f152ab1d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f152413e860 con 0x7f15240c05d0 2023-10-30T12:48:09.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.371+0000 7f152ab1d640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1524150960 con 0x7f15240c6b80 2023-10-30T12:48:09.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.371+0000 7f152ab1d640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f152414f7e0 con 0x7f15240c6f60 2023-10-30T12:48:09.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.371+0000 7f15288b9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15240c05d0 0x7f1524151540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:09.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.375+0000 7f15288b9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15240c05d0 0x7f1524151540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54202/0 (socket says 172.21.15.89:54202) 2023-10-30T12:48:09.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.375+0000 7f15288b9640 1 -- 172.21.15.89:0/577240249 learned_addr learned my addr 172.21.15.89:0/577240249 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:09.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.375+0000 7f1523fff640 1 --2- 172.21.15.89:0/577240249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15240c6b80 0x7f1524154860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:09.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.375+0000 7f15290ba640 1 --2- 172.21.15.89:0/577240249 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15240c6f60 0x7f1524154da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:09.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.375+0000 7f15288b9640 1 -- 172.21.15.89:0/577240249 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15240c6f60 msgr2=0x7f1524154da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:09.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.375+0000 7f15288b9640 1 --2- 172.21.15.89:0/577240249 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15240c6f60 0x7f1524154da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:09.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.375+0000 7f15288b9640 1 -- 172.21.15.89:0/577240249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15240c6b80 msgr2=0x7f1524154860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:09.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.375+0000 7f15288b9640 1 --2- 172.21.15.89:0/577240249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15240c6b80 0x7f1524154860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:09.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.375+0000 7f15288b9640 1 -- 172.21.15.89:0/577240249 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1510009420 con 0x7f15240c05d0 2023-10-30T12:48:09.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.375+0000 7f15290ba640 1 --2- 172.21.15.89:0/577240249 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15240c6f60 0x7f1524154da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:09.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.375+0000 7f15288b9640 1 --2- 172.21.15.89:0/577240249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15240c05d0 0x7f1524151540 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f151800c500 tx=0x7f151800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:09.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.375+0000 7f15217fa640 1 -- 172.21.15.89:0/577240249 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1518005220 con 0x7f15240c05d0 2023-10-30T12:48:09.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.375+0000 7f15217fa640 1 -- 172.21.15.89:0/577240249 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1518005380 con 0x7f15240c05d0 2023-10-30T12:48:09.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.375+0000 7f15217fa640 1 -- 172.21.15.89:0/577240249 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1518007270 con 0x7f15240c05d0 2023-10-30T12:48:09.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.375+0000 7f152ab1d640 1 -- 172.21.15.89:0/577240249 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f151c009b30 con 0x7f15240c05d0 2023-10-30T12:48:09.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.375+0000 7f152ab1d640 1 -- 172.21.15.89:0/577240249 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f152414fda0 con 0x7f15240c05d0 2023-10-30T12:48:09.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.375+0000 7f152ab1d640 1 -- 172.21.15.89:0/577240249 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f152404e170 con 0x7f15240c05d0 2023-10-30T12:48:09.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.375+0000 7f15217fa640 1 -- 172.21.15.89:0/577240249 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f15180073d0 con 0x7f15240c05d0 2023-10-30T12:48:09.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.379+0000 7f15217fa640 1 --2- 172.21.15.89:0/577240249 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f15080664a0 0x7f1508068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:09.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.379+0000 7f15217fa640 1 -- 172.21.15.89:0/577240249 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6492+0+0 (secure 0 0 0) 0x7f1518083a10 con 0x7f15240c05d0 2023-10-30T12:48:09.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.379+0000 7f1523fff640 1 --2- 172.21.15.89:0/577240249 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f15080664a0 0x7f1508068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:09.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.379+0000 7f1523fff640 1 --2- 172.21.15.89:0/577240249 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f15080664a0 0x7f1508068960 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f152413ea10 tx=0x7f1510009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:09.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.383+0000 7f15217fa640 1 -- 172.21.15.89:0/577240249 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1518050b70 con 0x7f15240c05d0 2023-10-30T12:48:09.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.563+0000 7f152ab1d640 1 -- 172.21.15.89:0/577240249 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1524154480 con 0x7f15240c05d0 2023-10-30T12:48:09.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.563+0000 7f15217fa640 1 -- 172.21.15.89:0/577240249 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f1518054190 con 0x7f15240c05d0 2023-10-30T12:48:09.562 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:48:09.562 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6075,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":88}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:47:10.393441+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":5450,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5450":{"gid":5450,"name":"f","rank":1,"incarnation":83,"state":"up:active","state_seq":36,"addr":"172.21.15.192:6837/604357386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":604357386},{"type":"v1","addr":"172.21.15.192:6837","nonce":604357386}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:09.563 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 91 2023-10-30T12:48:09.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.567+0000 7f152ab1d640 1 -- 172.21.15.89:0/577240249 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f15080664a0 msgr2=0x7f1508068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:09.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.567+0000 7f152ab1d640 1 --2- 172.21.15.89:0/577240249 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f15080664a0 0x7f1508068960 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f152413ea10 tx=0x7f1510009190 comp rx=0 tx=0).stop 2023-10-30T12:48:09.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.567+0000 7f152ab1d640 1 -- 172.21.15.89:0/577240249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15240c05d0 msgr2=0x7f1524151540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:09.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.567+0000 7f152ab1d640 1 --2- 172.21.15.89:0/577240249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15240c05d0 0x7f1524151540 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f151800c500 tx=0x7f151800c9c0 comp rx=0 tx=0).stop 2023-10-30T12:48:09.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.567+0000 7f152ab1d640 1 -- 172.21.15.89:0/577240249 shutdown_connections 2023-10-30T12:48:09.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.567+0000 7f152ab1d640 1 --2- 172.21.15.89:0/577240249 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f15080664a0 0x7f1508068960 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:09.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.567+0000 7f152ab1d640 1 --2- 172.21.15.89:0/577240249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15240c6b80 0x7f1524154860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:09.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.567+0000 7f152ab1d640 1 --2- 172.21.15.89:0/577240249 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15240c6f60 0x7f1524154da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:09.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.567+0000 7f152ab1d640 1 --2- 172.21.15.89:0/577240249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15240c05d0 0x7f1524151540 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:09.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.567+0000 7f152ab1d640 1 -- 172.21.15.89:0/577240249 >> 172.21.15.89:0/577240249 conn(0x7f15240c7910 msgr2=0x7f1524134770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:09.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.567+0000 7f152ab1d640 1 -- 172.21.15.89:0/577240249 shutdown_connections 2023-10-30T12:48:09.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:09.567+0000 7f152ab1d640 1 -- 172.21.15.89:0/577240249 wait complete. 2023-10-30T12:48:09.583 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:48:11.584 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:11.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.739+0000 7fc6e1e17640 1 Processor -- start 2023-10-30T12:48:11.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.739+0000 7fc6e1e17640 1 -- start start 2023-10-30T12:48:11.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.739+0000 7fc6e1e17640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc6dc153bf0 0x7fc6dc071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:11.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.739+0000 7fc6e1e17640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc6dc0c5f60 0x7fc6dc0cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:11.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.739+0000 7fc6e1e17640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6dc0700b0 con 0x7fc6dc0c6fe0 2023-10-30T12:48:11.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.739+0000 7fc6e1e17640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6dc0c6810 con 0x7fc6dc153bf0 2023-10-30T12:48:11.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.739+0000 7fc6e1e17640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6dc159900 con 0x7fc6dc0c5f60 2023-10-30T12:48:11.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.739+0000 7fc6daffd640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc6dc0c6fe0 0x7fc6dc0c74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57516/0 (socket says 172.21.15.89:57516) 2023-10-30T12:48:11.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.739+0000 7fc6daffd640 1 -- 172.21.15.89:0/2689598846 learned_addr learned my addr 172.21.15.89:0/2689598846 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:48:11.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.739+0000 7fc6db7fe640 1 --2- 172.21.15.89:0/2689598846 >> v2:172.21.15.89:3300/0 conn(0x7fc6dc153bf0 0x7fc6dc071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:11.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.743+0000 7fc6dbfff640 1 --2- 172.21.15.89:0/2689598846 >> v2:172.21.15.192:3300/0 conn(0x7fc6dc0c5f60 0x7fc6dc0cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:11.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.743+0000 7fc6da7fc640 1 -- 172.21.15.89:0/2689598846 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3251325044 0 0) 0x7fc6dc0700b0 con 0x7fc6dc0c6fe0 2023-10-30T12:48:11.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.743+0000 7fc6da7fc640 1 -- 172.21.15.89:0/2689598846 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6c4003530 con 0x7fc6dc0c6fe0 2023-10-30T12:48:11.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.743+0000 7fc6db7fe640 1 -- 172.21.15.89:0/2689598846 >> v2:172.21.15.192:3300/0 conn(0x7fc6dc0c5f60 msgr2=0x7fc6dc0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:11.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.743+0000 7fc6db7fe640 1 --2- 172.21.15.89:0/2689598846 >> v2:172.21.15.192:3300/0 conn(0x7fc6dc0c5f60 0x7fc6dc0cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:11.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.743+0000 7fc6db7fe640 1 -- 172.21.15.89:0/2689598846 >> v1:172.21.15.89:6789/0 conn(0x7fc6dc0c6fe0 legacy=0x7fc6dc0c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:11.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.743+0000 7fc6db7fe640 1 -- 172.21.15.89:0/2689598846 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6dc0c6530 con 0x7fc6dc153bf0 2023-10-30T12:48:11.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.743+0000 7fc6dbfff640 1 --2- 172.21.15.89:0/2689598846 >> v2:172.21.15.192:3300/0 conn(0x7fc6dc0c5f60 0x7fc6dc0cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:11.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.743+0000 7fc6db7fe640 1 --2- 172.21.15.89:0/2689598846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6dc153bf0 0x7fc6dc071260 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc06ce90 tx=0x7fc6cc01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eea0f088d59f6983 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:11.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.743+0000 7fc6da7fc640 1 -- 172.21.15.89:0/2689598846 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc0a8950 con 0x7fc6dc153bf0 2023-10-30T12:48:11.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.743+0000 7fc6da7fc640 1 -- 172.21.15.89:0/2689598846 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6cc0a8ab0 con 0x7fc6dc153bf0 2023-10-30T12:48:11.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.743+0000 7fc6da7fc640 1 -- 172.21.15.89:0/2689598846 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc0a8e20 con 0x7fc6dc153bf0 2023-10-30T12:48:11.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.747+0000 7fc6e1e17640 1 -- 172.21.15.89:0/2689598846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6dc153bf0 msgr2=0x7fc6dc071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:11.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.747+0000 7fc6e1e17640 1 --2- 172.21.15.89:0/2689598846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6dc153bf0 0x7fc6dc071260 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc06ce90 tx=0x7fc6cc01ec50 comp rx=0 tx=0).stop 2023-10-30T12:48:11.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.747+0000 7fc6e1e17640 1 -- 172.21.15.89:0/2689598846 shutdown_connections 2023-10-30T12:48:11.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.747+0000 7fc6e1e17640 1 --2- 172.21.15.89:0/2689598846 >> v2:172.21.15.192:3300/0 conn(0x7fc6dc0c5f60 0x7fc6dc0cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:11.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.747+0000 7fc6e1e17640 1 --2- 172.21.15.89:0/2689598846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6dc153bf0 0x7fc6dc071260 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:11.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.747+0000 7fc6e1e17640 1 -- 172.21.15.89:0/2689598846 >> 172.21.15.89:0/2689598846 conn(0x7fc6dc06c6e0 msgr2=0x7fc6dc069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:11.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.747+0000 7fc6e1e17640 1 -- 172.21.15.89:0/2689598846 shutdown_connections 2023-10-30T12:48:11.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.747+0000 7fc6e1e17640 1 -- 172.21.15.89:0/2689598846 wait complete. 2023-10-30T12:48:11.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.747+0000 7fc6e1e17640 1 Processor -- start 2023-10-30T12:48:11.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.747+0000 7fc6e1e17640 1 -- start start 2023-10-30T12:48:11.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.747+0000 7fc6e1e17640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6dc0c5f60 0x7fc6dc14e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:11.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.747+0000 7fc6e1e17640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6dc0c6fe0 0x7fc6dc1f2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:11.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.747+0000 7fc6e1e17640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6dc153bf0 0x7fc6dc14f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:11.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.747+0000 7fc6e1e17640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc0c4810 con 0x7fc6dc153bf0 2023-10-30T12:48:11.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.747+0000 7fc6e1e17640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc0c41e0 con 0x7fc6dc0c5f60 2023-10-30T12:48:11.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.747+0000 7fc6e1e17640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc1f2c90 con 0x7fc6dc0c6fe0 2023-10-30T12:48:11.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.747+0000 7fc6dbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6dc153bf0 0x7fc6dc14f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:11.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.747+0000 7fc6dbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6dc153bf0 0x7fc6dc14f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37418/0 (socket says 172.21.15.89:37418) 2023-10-30T12:48:11.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.747+0000 7fc6dbfff640 1 -- 172.21.15.89:0/634712439 learned_addr learned my addr 172.21.15.89:0/634712439 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:11.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.747+0000 7fc6db7fe640 1 --2- 172.21.15.89:0/634712439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6dc0c5f60 0x7fc6dc14e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:11.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.747+0000 7fc6daffd640 1 --2- 172.21.15.89:0/634712439 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6dc0c6fe0 0x7fc6dc1f2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:11.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.747+0000 7fc6dbfff640 1 -- 172.21.15.89:0/634712439 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6dc0c6fe0 msgr2=0x7fc6dc1f2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:11.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.747+0000 7fc6dbfff640 1 --2- 172.21.15.89:0/634712439 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6dc0c6fe0 0x7fc6dc1f2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:11.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.747+0000 7fc6dbfff640 1 -- 172.21.15.89:0/634712439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6dc0c5f60 msgr2=0x7fc6dc14e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:11.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.747+0000 7fc6dbfff640 1 --2- 172.21.15.89:0/634712439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6dc0c5f60 0x7fc6dc14e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:11.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.747+0000 7fc6dbfff640 1 -- 172.21.15.89:0/634712439 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6cc009420 con 0x7fc6dc153bf0 2023-10-30T12:48:11.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.747+0000 7fc6db7fe640 1 --2- 172.21.15.89:0/634712439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6dc0c5f60 0x7fc6dc14e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:11.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.751+0000 7fc6dbfff640 1 --2- 172.21.15.89:0/634712439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6dc153bf0 0x7fc6dc14f4e0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4007ef0 tx=0x7fc6d400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:11.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.751+0000 7fc6c3fff640 1 -- 172.21.15.89:0/634712439 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d4003ba0 con 0x7fc6dc153bf0 2023-10-30T12:48:11.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.751+0000 7fc6c3fff640 1 -- 172.21.15.89:0/634712439 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d400f040 con 0x7fc6dc153bf0 2023-10-30T12:48:11.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.751+0000 7fc6e1e17640 1 -- 172.21.15.89:0/634712439 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6dc0c44a0 con 0x7fc6dc153bf0 2023-10-30T12:48:11.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.751+0000 7fc6c3fff640 1 -- 172.21.15.89:0/634712439 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d400de80 con 0x7fc6dc153bf0 2023-10-30T12:48:11.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.751+0000 7fc6e1e17640 1 -- 172.21.15.89:0/634712439 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6dc14efe0 con 0x7fc6dc153bf0 2023-10-30T12:48:11.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.751+0000 7fc6e1e17640 1 -- 172.21.15.89:0/634712439 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6dc0c49c0 con 0x7fc6dc153bf0 2023-10-30T12:48:11.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.751+0000 7fc6c3fff640 1 -- 172.21.15.89:0/634712439 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc6d4004030 con 0x7fc6dc153bf0 2023-10-30T12:48:11.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.751+0000 7fc6c3fff640 1 --2- 172.21.15.89:0/634712439 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc6c80664a0 0x7fc6c8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:11.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.751+0000 7fc6c3fff640 1 -- 172.21.15.89:0/634712439 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6578+0+0 (secure 0 0 0) 0x7fc6d4084400 con 0x7fc6dc153bf0 2023-10-30T12:48:11.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.751+0000 7fc6db7fe640 1 --2- 172.21.15.89:0/634712439 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc6c80664a0 0x7fc6c8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:11.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.755+0000 7fc6db7fe640 1 --2- 172.21.15.89:0/634712439 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc6c80664a0 0x7fc6c8068960 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc0b2000 tx=0x7fc6cc0aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:11.754 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.755+0000 7fc6c3fff640 1 -- 172.21.15.89:0/634712439 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc6dc0c49c0 con 0x7fc6dc153bf0 2023-10-30T12:48:11.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.943+0000 7fc6e1e17640 1 -- 172.21.15.89:0/634712439 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6dc04e170 con 0x7fc6dc153bf0 2023-10-30T12:48:11.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.943+0000 7fc6c3fff640 1 -- 172.21.15.89:0/634712439 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+5743 (secure 0 0 0) 0x7fc6d40515b0 con 0x7fc6dc153bf0 2023-10-30T12:48:11.941 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:48:11.942 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:48:11.263481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:replay","state_seq":1,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:11.942 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 92 2023-10-30T12:48:11.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.947+0000 7fc6e1e17640 1 -- 172.21.15.89:0/634712439 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc6c80664a0 msgr2=0x7fc6c8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:11.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.947+0000 7fc6e1e17640 1 --2- 172.21.15.89:0/634712439 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc6c80664a0 0x7fc6c8068960 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc0b2000 tx=0x7fc6cc0aa070 comp rx=0 tx=0).stop 2023-10-30T12:48:11.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.947+0000 7fc6e1e17640 1 -- 172.21.15.89:0/634712439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6dc153bf0 msgr2=0x7fc6dc14f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:11.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.947+0000 7fc6e1e17640 1 --2- 172.21.15.89:0/634712439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6dc153bf0 0x7fc6dc14f4e0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4007ef0 tx=0x7fc6d400d4a0 comp rx=0 tx=0).stop 2023-10-30T12:48:11.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.947+0000 7fc6e1e17640 1 -- 172.21.15.89:0/634712439 shutdown_connections 2023-10-30T12:48:11.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.947+0000 7fc6e1e17640 1 --2- 172.21.15.89:0/634712439 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc6c80664a0 0x7fc6c8068960 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:11.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.947+0000 7fc6e1e17640 1 --2- 172.21.15.89:0/634712439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6dc153bf0 0x7fc6dc14f4e0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:11.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.947+0000 7fc6e1e17640 1 --2- 172.21.15.89:0/634712439 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6dc0c6fe0 0x7fc6dc1f2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:11.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.947+0000 7fc6e1e17640 1 --2- 172.21.15.89:0/634712439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6dc0c5f60 0x7fc6dc14e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:11.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.947+0000 7fc6e1e17640 1 -- 172.21.15.89:0/634712439 >> 172.21.15.89:0/634712439 conn(0x7fc6dc06c6e0 msgr2=0x7fc6dc158fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:11.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.947+0000 7fc6e1e17640 1 -- 172.21.15.89:0/634712439 shutdown_connections 2023-10-30T12:48:11.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:11.947+0000 7fc6e1e17640 1 -- 172.21.15.89:0/634712439 wait complete. 2023-10-30T12:48:11.962 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-10-30T12:48:11.962 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:48:11.962 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:12.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f631e73b640 1 Processor -- start 2023-10-30T12:48:12.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f631e73b640 1 -- start start 2023-10-30T12:48:12.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f631e73b640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f63181538f0 0x7f6318159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:12.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f631e73b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f63180cefa0 0x7f63180cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:12.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f631e73b640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63180ca6a0 con 0x7f631815cd50 2023-10-30T12:48:12.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f631e73b640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f631815d300 con 0x7f63181538f0 2023-10-30T12:48:12.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f631e73b640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f631815d5e0 con 0x7f63180cefa0 2023-10-30T12:48:12.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f631cf38640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f631815cd50 0x7f63180c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57518/0 (socket says 172.21.15.89:57518) 2023-10-30T12:48:12.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f631cf38640 1 -- 172.21.15.89:0/1518904703 learned_addr learned my addr 172.21.15.89:0/1518904703 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:48:12.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f631df3a640 1 --2- 172.21.15.89:0/1518904703 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f63180cefa0 0x7f63180cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:12.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f631d739640 1 --2- 172.21.15.89:0/1518904703 >> v2:172.21.15.192:3300/0 conn(0x7f63181538f0 0x7f6318159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:12.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f630ffff640 1 -- 172.21.15.89:0/1518904703 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1142512449 0 0) 0x7f63180ca6a0 con 0x7f631815cd50 2023-10-30T12:48:12.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f630ffff640 1 -- 172.21.15.89:0/1518904703 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6304003530 con 0x7f631815cd50 2023-10-30T12:48:12.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f630ffff640 1 -- 172.21.15.89:0/1518904703 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2744275184 0 0) 0x7f6304003530 con 0x7f631815cd50 2023-10-30T12:48:12.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f630ffff640 1 -- 172.21.15.89:0/1518904703 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f63180cefa0 msgr2=0x7f63180cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:12.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f630ffff640 1 --2- 172.21.15.89:0/1518904703 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f63180cefa0 0x7f63180cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:12.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f630ffff640 1 -- 172.21.15.89:0/1518904703 >> v2:172.21.15.192:3300/0 conn(0x7f63181538f0 msgr2=0x7f6318159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:12.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f630ffff640 1 --2- 172.21.15.89:0/1518904703 >> v2:172.21.15.192:3300/0 conn(0x7f63181538f0 0x7f6318159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:12.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f630ffff640 1 -- 172.21.15.89:0/1518904703 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f631815d8c0 con 0x7f631815cd50 2023-10-30T12:48:12.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f630ffff640 1 -- 172.21.15.89:0/1518904703 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f6308001680 con 0x7f631815cd50 2023-10-30T12:48:12.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f630ffff640 1 -- 172.21.15.89:0/1518904703 >> v1:172.21.15.89:6789/0 conn(0x7f631815cd50 legacy=0x7f63180c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:12.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f630ffff640 1 --2- 172.21.15.89:0/1518904703 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f63180cefa0 0x7f6304003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:12.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f630ffff640 1 --2- 172.21.15.89:0/1518904703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6304003f00 0x7f63040047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:12.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f630ffff640 1 --2- 172.21.15.89:0/1518904703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6304004d10 0x7f6304005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:12.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f630ffff640 1 -- 172.21.15.89:0/1518904703 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631815d300 con 0x7f6304004d10 2023-10-30T12:48:12.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f630ffff640 1 -- 172.21.15.89:0/1518904703 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631815d5e0 con 0x7f6304003f00 2023-10-30T12:48:12.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f630ffff640 1 -- 172.21.15.89:0/1518904703 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f630400b9c0 con 0x7f63180cefa0 2023-10-30T12:48:12.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f631df3a640 1 --2- 172.21.15.89:0/1518904703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6304004d10 0x7f6304005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:12.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f631d739640 1 --2- 172.21.15.89:0/1518904703 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f63180cefa0 0x7f6304003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:12.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f631cf38640 1 --2- 172.21.15.89:0/1518904703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6304003f00 0x7f63040047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:12.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f631df3a640 1 -- 172.21.15.89:0/1518904703 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f63180cefa0 msgr2=0x7f6304003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:12.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f631df3a640 1 --2- 172.21.15.89:0/1518904703 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f63180cefa0 0x7f6304003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:12.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f631df3a640 1 -- 172.21.15.89:0/1518904703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6304003f00 msgr2=0x7f63040047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:12.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f631df3a640 1 --2- 172.21.15.89:0/1518904703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6304003f00 0x7f63040047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:12.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.103+0000 7f631df3a640 1 -- 172.21.15.89:0/1518904703 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6304003530 con 0x7f6304004d10 2023-10-30T12:48:12.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.107+0000 7f631df3a640 1 --2- 172.21.15.89:0/1518904703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6304004d10 0x7f6304005190 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f6310009fc0 tx=0x7f631001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a9aaedfb83544d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:12.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.107+0000 7f630ffff640 1 -- 172.21.15.89:0/1518904703 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63100a5c70 con 0x7f6304004d10 2023-10-30T12:48:12.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.107+0000 7f630ffff640 1 -- 172.21.15.89:0/1518904703 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f631000f040 con 0x7f6304004d10 2023-10-30T12:48:12.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.107+0000 7f630ffff640 1 -- 172.21.15.89:0/1518904703 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63100ad440 con 0x7f6304004d10 2023-10-30T12:48:12.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.107+0000 7f631e73b640 1 -- 172.21.15.89:0/1518904703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6304004d10 msgr2=0x7f6304005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:12.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.107+0000 7f631e73b640 1 --2- 172.21.15.89:0/1518904703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6304004d10 0x7f6304005190 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f6310009fc0 tx=0x7f631001cc60 comp rx=0 tx=0).stop 2023-10-30T12:48:12.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.107+0000 7f631df3a640 1 -- 172.21.15.89:0/1518904703 reap_dead start 2023-10-30T12:48:12.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.107+0000 7f631e73b640 1 -- 172.21.15.89:0/1518904703 shutdown_connections 2023-10-30T12:48:12.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.107+0000 7f631e73b640 1 -- 172.21.15.89:0/1518904703 >> 172.21.15.89:0/1518904703 conn(0x7f631806a640 msgr2=0x7f6318065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:12.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.107+0000 7f631e73b640 1 -- 172.21.15.89:0/1518904703 shutdown_connections 2023-10-30T12:48:12.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.107+0000 7f631e73b640 1 -- 172.21.15.89:0/1518904703 wait complete. 2023-10-30T12:48:12.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.111+0000 7f631e73b640 1 Processor -- start 2023-10-30T12:48:12.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.111+0000 7f631e73b640 1 -- start start 2023-10-30T12:48:12.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.111+0000 7f631e73b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6304004d10 0x7f631814d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:12.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.111+0000 7f631e73b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6318157db0 0x7f631815c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:12.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.111+0000 7f631e73b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6318150e70 0x7f6318151320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:12.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.111+0000 7f631e73b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6318150760 con 0x7f6304004d10 2023-10-30T12:48:12.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.111+0000 7f631e73b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6318151860 con 0x7f6318157db0 2023-10-30T12:48:12.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.111+0000 7f631e73b640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f631814d950 con 0x7f6318150e70 2023-10-30T12:48:12.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.111+0000 7f631d739640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6304004d10 0x7f631814d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:12.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.111+0000 7f631d739640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6304004d10 0x7f631814d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37446/0 (socket says 172.21.15.89:37446) 2023-10-30T12:48:12.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.111+0000 7f631d739640 1 -- 172.21.15.89:0/2013938157 learned_addr learned my addr 172.21.15.89:0/2013938157 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:12.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.111+0000 7f631cf38640 1 --2- 172.21.15.89:0/2013938157 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6318157db0 0x7f631815c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:12.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.111+0000 7f631df3a640 1 --2- 172.21.15.89:0/2013938157 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6318150e70 0x7f6318151320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:12.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.111+0000 7f631d739640 1 -- 172.21.15.89:0/2013938157 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6318150e70 msgr2=0x7f6318151320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:12.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.111+0000 7f631d739640 1 --2- 172.21.15.89:0/2013938157 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6318150e70 0x7f6318151320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:12.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.111+0000 7f631d739640 1 -- 172.21.15.89:0/2013938157 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6318157db0 msgr2=0x7f631815c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:12.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.111+0000 7f631d739640 1 --2- 172.21.15.89:0/2013938157 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6318157db0 0x7f631815c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:12.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.111+0000 7f631d739640 1 -- 172.21.15.89:0/2013938157 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6318158380 con 0x7f6304004d10 2023-10-30T12:48:12.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.111+0000 7f631d739640 1 --2- 172.21.15.89:0/2013938157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6304004d10 0x7f631814d410 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f6314011690 tx=0x7f6314011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:12.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.111+0000 7f630dffb640 1 -- 172.21.15.89:0/2013938157 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631400f9c0 con 0x7f6304004d10 2023-10-30T12:48:12.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.111+0000 7f630dffb640 1 -- 172.21.15.89:0/2013938157 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f631400fb20 con 0x7f6304004d10 2023-10-30T12:48:12.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.111+0000 7f630dffb640 1 -- 172.21.15.89:0/2013938157 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631400fe90 con 0x7f6304004d10 2023-10-30T12:48:12.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.111+0000 7f631e73b640 1 -- 172.21.15.89:0/2013938157 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f631814df60 con 0x7f6304004d10 2023-10-30T12:48:12.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.111+0000 7f631e73b640 1 -- 172.21.15.89:0/2013938157 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6318150970 con 0x7f6304004d10 2023-10-30T12:48:12.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.115+0000 7f631e73b640 1 -- 172.21.15.89:0/2013938157 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f631804e170 con 0x7f6304004d10 2023-10-30T12:48:12.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.115+0000 7f630dffb640 1 -- 172.21.15.89:0/2013938157 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6314015460 con 0x7f6304004d10 2023-10-30T12:48:12.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.115+0000 7f630dffb640 1 --2- 172.21.15.89:0/2013938157 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f62fc0664d0 0x7f62fc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:12.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.115+0000 7f630dffb640 1 -- 172.21.15.89:0/2013938157 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6578+0+0 (secure 0 0 0) 0x7f6314084820 con 0x7f6304004d10 2023-10-30T12:48:12.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.115+0000 7f631cf38640 1 --2- 172.21.15.89:0/2013938157 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f62fc0664d0 0x7f62fc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:12.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.115+0000 7f631cf38640 1 --2- 172.21.15.89:0/2013938157 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f62fc0664d0 0x7f62fc068990 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f6308001060 tx=0x7f6308009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:12.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.119+0000 7f630dffb640 1 -- 172.21.15.89:0/2013938157 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f63140518a0 con 0x7f6304004d10 2023-10-30T12:48:12.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.303+0000 7f631e73b640 1 -- 172.21.15.89:0/2013938157 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63180c48a0 con 0x7f6304004d10 2023-10-30T12:48:12.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.303+0000 7f630dffb640 1 -- 172.21.15.89:0/2013938157 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+5743 (secure 0 0 0) 0x7f6314022090 con 0x7f6304004d10 2023-10-30T12:48:12.300 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:48:12.300 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:48:11.263481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:replay","state_seq":1,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:12.301 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 92 2023-10-30T12:48:12.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.303+0000 7f631e73b640 1 -- 172.21.15.89:0/2013938157 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f62fc0664d0 msgr2=0x7f62fc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:12.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.303+0000 7f631e73b640 1 --2- 172.21.15.89:0/2013938157 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f62fc0664d0 0x7f62fc068990 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f6308001060 tx=0x7f6308009040 comp rx=0 tx=0).stop 2023-10-30T12:48:12.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.303+0000 7f631e73b640 1 -- 172.21.15.89:0/2013938157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6304004d10 msgr2=0x7f631814d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:12.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.303+0000 7f631e73b640 1 --2- 172.21.15.89:0/2013938157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6304004d10 0x7f631814d410 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f6314011690 tx=0x7f6314011b50 comp rx=0 tx=0).stop 2023-10-30T12:48:12.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.303+0000 7f631e73b640 1 -- 172.21.15.89:0/2013938157 shutdown_connections 2023-10-30T12:48:12.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.303+0000 7f631e73b640 1 --2- 172.21.15.89:0/2013938157 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f62fc0664d0 0x7f62fc068990 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:12.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.303+0000 7f631e73b640 1 --2- 172.21.15.89:0/2013938157 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6318157db0 0x7f631815c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:12.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.303+0000 7f631e73b640 1 --2- 172.21.15.89:0/2013938157 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6318150e70 0x7f6318151320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:12.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.303+0000 7f631e73b640 1 --2- 172.21.15.89:0/2013938157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6304004d10 0x7f631814d410 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:12.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.303+0000 7f631e73b640 1 -- 172.21.15.89:0/2013938157 >> 172.21.15.89:0/2013938157 conn(0x7f631806a640 msgr2=0x7f63180cf3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:12.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.303+0000 7f631e73b640 1 -- 172.21.15.89:0/2013938157 shutdown_connections 2023-10-30T12:48:12.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:12.303+0000 7f631e73b640 1 -- 172.21.15.89:0/2013938157 wait complete. 2023-10-30T12:48:12.321 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 92, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:50.729245+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 92, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5575": { "addr": "172.21.15.192:6835/501291700", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6834", "nonce": 501291700, "type": "v2" }, { "addr": "172.21.15.192:6835", "nonce": 501291700, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5575, "incarnation": 63, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 37 }, "gid_5728": { "addr": "172.21.15.192:6839/3991891597", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6838", "nonce": 3991891597, "type": "v2" }, { "addr": "172.21.15.192:6839", "nonce": 3991891597, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 5728, "incarnation": 73, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 19 }, "gid_6075": { "addr": "172.21.15.89:6837/2412618858", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6836", "nonce": 2412618858, "type": "v2" }, { "addr": "172.21.15.89:6837", "nonce": 2412618858, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6075, "incarnation": 92, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:48:11.263481+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5575, "mds_1": 6075, "mds_2": 5728 } } } ], "standbys": [ { "addr": "172.21.15.89:6834/2909790173", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6832", "nonce": 2909790173, "type": "v2" }, { "addr": "172.21.15.89:6834", "nonce": 2909790173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.89:6835/2387357615", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6833", "nonce": 2387357615, "type": "v2" }, { "addr": "172.21.15.89:6835", "nonce": 2387357615, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4239, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:48:14.323 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:14.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.475+0000 7fddf8ba1640 1 Processor -- start 2023-10-30T12:48:14.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.475+0000 7fddf8ba1640 1 -- start start 2023-10-30T12:48:14.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.475+0000 7fddf8ba1640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fddf40c6bd0 0x7fddf406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:14.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.475+0000 7fddf8ba1640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fddf4071110 0x7fddf4158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:14.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.475+0000 7fddf8ba1640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf4066cf0 con 0x7fddf406f7a0 2023-10-30T12:48:14.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.475+0000 7fddf8ba1640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddf40700c0 con 0x7fddf40c6bd0 2023-10-30T12:48:14.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.475+0000 7fddf8ba1640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddf40c70d0 con 0x7fddf4071110 2023-10-30T12:48:14.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.475+0000 7fdde9d9b640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fddf40c6bd0 0x7fddf406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:14.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.475+0000 7fdde9d9b640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fddf40c6bd0 0x7fddf406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37460/0 (socket says 172.21.15.89:37460) 2023-10-30T12:48:14.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.475+0000 7fddf259c640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fddf406f7a0 0x7fddf4069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57526/0 (socket says 172.21.15.89:57526) 2023-10-30T12:48:14.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.475+0000 7fdde9d9b640 1 -- 172.21.15.89:0/3913725241 learned_addr learned my addr 172.21.15.89:0/3913725241 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:14.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.479+0000 7fddf2d9d640 1 --2- 172.21.15.89:0/3913725241 >> v2:172.21.15.192:3300/0 conn(0x7fddf4071110 0x7fddf4158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:14.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.479+0000 7fdde9d9b640 1 -- 172.21.15.89:0/3913725241 >> v2:172.21.15.192:3300/0 conn(0x7fddf4071110 msgr2=0x7fddf4158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:14.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.479+0000 7fdde9d9b640 1 --2- 172.21.15.89:0/3913725241 >> v2:172.21.15.192:3300/0 conn(0x7fddf4071110 0x7fddf4158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:14.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.479+0000 7fdde9d9b640 1 -- 172.21.15.89:0/3913725241 >> v1:172.21.15.89:6789/0 conn(0x7fddf406f7a0 legacy=0x7fddf4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:14.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.479+0000 7fdde9d9b640 1 -- 172.21.15.89:0/3913725241 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf40c7390 con 0x7fddf40c6bd0 2023-10-30T12:48:14.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.479+0000 7fddf1d9b640 1 -- 172.21.15.89:0/3913725241 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113742033 0 0) 0x7fddf4066cf0 con 0x7fddf406f7a0 2023-10-30T12:48:14.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.479+0000 7fdde9d9b640 1 --2- 172.21.15.89:0/3913725241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddf40c6bd0 0x7fddf406fb80 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fddf406cdd0 tx=0x7fdddc01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8914f23d58368fb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:14.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.479+0000 7fddf1d9b640 1 -- 172.21.15.89:0/3913725241 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc0a89f0 con 0x7fddf40c6bd0 2023-10-30T12:48:14.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.479+0000 7fddf1d9b640 1 -- 172.21.15.89:0/3913725241 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdddc0a8b50 con 0x7fddf40c6bd0 2023-10-30T12:48:14.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.479+0000 7fddf1d9b640 1 -- 172.21.15.89:0/3913725241 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc0b0440 con 0x7fddf40c6bd0 2023-10-30T12:48:14.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.479+0000 7fddf8ba1640 1 -- 172.21.15.89:0/3913725241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddf40c6bd0 msgr2=0x7fddf406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:14.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.479+0000 7fddf8ba1640 1 --2- 172.21.15.89:0/3913725241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddf40c6bd0 0x7fddf406fb80 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fddf406cdd0 tx=0x7fdddc01ecf0 comp rx=0 tx=0).stop 2023-10-30T12:48:14.480 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.483+0000 7fddf8ba1640 1 -- 172.21.15.89:0/3913725241 shutdown_connections 2023-10-30T12:48:14.480 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.483+0000 7fddf8ba1640 1 --2- 172.21.15.89:0/3913725241 >> v2:172.21.15.192:3300/0 conn(0x7fddf4071110 0x7fddf4158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:14.480 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.483+0000 7fddf8ba1640 1 --2- 172.21.15.89:0/3913725241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddf40c6bd0 0x7fddf406fb80 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:14.480 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.483+0000 7fddf8ba1640 1 -- 172.21.15.89:0/3913725241 >> 172.21.15.89:0/3913725241 conn(0x7fddf40c7910 msgr2=0x7fddf40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:14.480 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.483+0000 7fddf8ba1640 1 -- 172.21.15.89:0/3913725241 shutdown_connections 2023-10-30T12:48:14.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.483+0000 7fddf8ba1640 1 -- 172.21.15.89:0/3913725241 wait complete. 2023-10-30T12:48:14.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.483+0000 7fddf8ba1640 1 Processor -- start 2023-10-30T12:48:14.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.483+0000 7fddf8ba1640 1 -- start start 2023-10-30T12:48:14.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.483+0000 7fddf8ba1640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fddf406f7a0 0x7fddf41f2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:14.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.483+0000 7fddf8ba1640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fddf4071110 0x7fddf4157d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:14.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.483+0000 7fddf8ba1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddf40c6bd0 0x7fddf414c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:14.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.483+0000 7fddf8ba1640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf40c1cd0 con 0x7fddf40c6bd0 2023-10-30T12:48:14.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.483+0000 7fddf8ba1640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf4154d10 con 0x7fddf406f7a0 2023-10-30T12:48:14.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.483+0000 7fddf8ba1640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf41f2ae0 con 0x7fddf4071110 2023-10-30T12:48:14.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.483+0000 7fddf2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddf40c6bd0 0x7fddf414c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:14.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.483+0000 7fddf2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddf40c6bd0 0x7fddf414c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37464/0 (socket says 172.21.15.89:37464) 2023-10-30T12:48:14.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.483+0000 7fddf2d9d640 1 -- 172.21.15.89:0/1950936371 learned_addr learned my addr 172.21.15.89:0/1950936371 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:14.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.483+0000 7fddf259c640 1 --2- 172.21.15.89:0/1950936371 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fddf406f7a0 0x7fddf41f2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:14.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.483+0000 7fdde9d9b640 1 --2- 172.21.15.89:0/1950936371 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fddf4071110 0x7fddf4157d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:14.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.483+0000 7fddf2d9d640 1 -- 172.21.15.89:0/1950936371 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fddf4071110 msgr2=0x7fddf4157d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:14.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.483+0000 7fddf2d9d640 1 --2- 172.21.15.89:0/1950936371 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fddf4071110 0x7fddf4157d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:14.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.483+0000 7fddf2d9d640 1 -- 172.21.15.89:0/1950936371 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fddf406f7a0 msgr2=0x7fddf41f2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:14.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.483+0000 7fddf2d9d640 1 --2- 172.21.15.89:0/1950936371 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fddf406f7a0 0x7fddf41f2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:14.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.483+0000 7fddf2d9d640 1 -- 172.21.15.89:0/1950936371 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdddc0094d0 con 0x7fddf40c6bd0 2023-10-30T12:48:14.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.483+0000 7fdde9d9b640 1 --2- 172.21.15.89:0/1950936371 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fddf4071110 0x7fddf4157d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:14.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.487+0000 7fddf2d9d640 1 --2- 172.21.15.89:0/1950936371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddf40c6bd0 0x7fddf414c430 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fddec00d560 tx=0x7fddec00da20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:14.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.487+0000 7fddeb7fe640 1 -- 172.21.15.89:0/1950936371 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddec0125f0 con 0x7fddf40c6bd0 2023-10-30T12:48:14.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.487+0000 7fddeb7fe640 1 -- 172.21.15.89:0/1950936371 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddec012770 con 0x7fddf40c6bd0 2023-10-30T12:48:14.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.487+0000 7fddf8ba1640 1 -- 172.21.15.89:0/1950936371 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf40c19a0 con 0x7fddf40c6bd0 2023-10-30T12:48:14.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.487+0000 7fddeb7fe640 1 -- 172.21.15.89:0/1950936371 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddec012b20 con 0x7fddf40c6bd0 2023-10-30T12:48:14.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.487+0000 7fddf8ba1640 1 -- 172.21.15.89:0/1950936371 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf4154f80 con 0x7fddf40c6bd0 2023-10-30T12:48:14.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.487+0000 7fddf8ba1640 1 -- 172.21.15.89:0/1950936371 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf40c1e40 con 0x7fddf40c6bd0 2023-10-30T12:48:14.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.487+0000 7fddeb7fe640 1 -- 172.21.15.89:0/1950936371 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fddec012c80 con 0x7fddf40c6bd0 2023-10-30T12:48:14.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.487+0000 7fddeb7fe640 1 --2- 172.21.15.89:0/1950936371 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fddd40664a0 0x7fddd4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:14.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.487+0000 7fddeb7fe640 1 -- 172.21.15.89:0/1950936371 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6578+0+0 (secure 0 0 0) 0x7fddec084780 con 0x7fddf40c6bd0 2023-10-30T12:48:14.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.487+0000 7fddf259c640 1 --2- 172.21.15.89:0/1950936371 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fddd40664a0 0x7fddd4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:14.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.491+0000 7fddf259c640 1 --2- 172.21.15.89:0/1950936371 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fddd40664a0 0x7fddd4068960 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fdde40012f0 tx=0x7fdde4001690 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:14.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.491+0000 7fddeb7fe640 1 -- 172.21.15.89:0/1950936371 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fddec051760 con 0x7fddf40c6bd0 2023-10-30T12:48:14.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.671+0000 7fddf8ba1640 1 -- 172.21.15.89:0/1950936371 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fddf404e170 con 0x7fddf40c6bd0 2023-10-30T12:48:14.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.671+0000 7fddeb7fe640 1 -- 172.21.15.89:0/1950936371 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+5743 (secure 0 0 0) 0x7fddec054d80 con 0x7fddf40c6bd0 2023-10-30T12:48:14.670 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:48:14.670 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:48:11.263481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:replay","state_seq":1,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:14.670 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 92 2023-10-30T12:48:14.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.675+0000 7fddf8ba1640 1 -- 172.21.15.89:0/1950936371 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fddd40664a0 msgr2=0x7fddd4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:14.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.675+0000 7fddf8ba1640 1 --2- 172.21.15.89:0/1950936371 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fddd40664a0 0x7fddd4068960 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fdde40012f0 tx=0x7fdde4001690 comp rx=0 tx=0).stop 2023-10-30T12:48:14.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.675+0000 7fddf8ba1640 1 -- 172.21.15.89:0/1950936371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddf40c6bd0 msgr2=0x7fddf414c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:14.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.675+0000 7fddf8ba1640 1 --2- 172.21.15.89:0/1950936371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddf40c6bd0 0x7fddf414c430 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fddec00d560 tx=0x7fddec00da20 comp rx=0 tx=0).stop 2023-10-30T12:48:14.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.675+0000 7fddf8ba1640 1 -- 172.21.15.89:0/1950936371 shutdown_connections 2023-10-30T12:48:14.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.675+0000 7fddf8ba1640 1 --2- 172.21.15.89:0/1950936371 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fddd40664a0 0x7fddd4068960 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:14.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.675+0000 7fddf8ba1640 1 --2- 172.21.15.89:0/1950936371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddf40c6bd0 0x7fddf414c430 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:14.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.675+0000 7fddf8ba1640 1 --2- 172.21.15.89:0/1950936371 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fddf4071110 0x7fddf4157d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:14.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.675+0000 7fddf8ba1640 1 --2- 172.21.15.89:0/1950936371 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fddf406f7a0 0x7fddf41f2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:14.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.675+0000 7fddf8ba1640 1 -- 172.21.15.89:0/1950936371 >> 172.21.15.89:0/1950936371 conn(0x7fddf40c7910 msgr2=0x7fddf40c54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:14.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.675+0000 7fddf8ba1640 1 -- 172.21.15.89:0/1950936371 shutdown_connections 2023-10-30T12:48:14.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:14.675+0000 7fddf8ba1640 1 -- 172.21.15.89:0/1950936371 wait complete. 2023-10-30T12:48:14.690 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:48:16.691 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:16.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.859+0000 7f9907817640 1 Processor -- start 2023-10-30T12:48:16.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.859+0000 7f9907817640 1 -- start start 2023-10-30T12:48:16.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.863+0000 7f9907817640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f990006e470 0x7f990006ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:16.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.863+0000 7f9907817640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99000c8490 0x7f99000c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:16.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.863+0000 7f9907817640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f990006afa0 con 0x7f9900153d70 2023-10-30T12:48:16.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.863+0000 7f9907817640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f990006d780 con 0x7f990006e470 2023-10-30T12:48:16.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.863+0000 7f9907817640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9900154150 con 0x7f99000c8490 2023-10-30T12:48:16.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.863+0000 7f99055b3640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9900153d70 0x7f99000cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57530/0 (socket says 172.21.15.89:57530) 2023-10-30T12:48:16.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.863+0000 7f99055b3640 1 -- 172.21.15.89:0/3532920947 learned_addr learned my addr 172.21.15.89:0/3532920947 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:48:16.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.863+0000 7f9904db2640 1 --2- 172.21.15.89:0/3532920947 >> v2:172.21.15.192:3300/0 conn(0x7f990006e470 0x7f990006ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:16.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.863+0000 7f9905db4640 1 --2- 172.21.15.89:0/3532920947 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99000c8490 0x7f99000c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:16.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.863+0000 7f98f7fff640 1 -- 172.21.15.89:0/3532920947 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2579216042 0 0) 0x7f990006afa0 con 0x7f9900153d70 2023-10-30T12:48:16.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.863+0000 7f98f7fff640 1 -- 172.21.15.89:0/3532920947 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98ec003530 con 0x7f9900153d70 2023-10-30T12:48:16.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.863+0000 7f9904db2640 1 -- 172.21.15.89:0/3532920947 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99000c8490 msgr2=0x7f99000c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:16.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.863+0000 7f9904db2640 1 --2- 172.21.15.89:0/3532920947 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99000c8490 0x7f99000c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:16.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.863+0000 7f9904db2640 1 -- 172.21.15.89:0/3532920947 >> v1:172.21.15.89:6789/0 conn(0x7f9900153d70 legacy=0x7f99000cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:16.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.863+0000 7f9904db2640 1 -- 172.21.15.89:0/3532920947 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f990006e030 con 0x7f990006e470 2023-10-30T12:48:16.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.863+0000 7f9905db4640 1 --2- 172.21.15.89:0/3532920947 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99000c8490 0x7f99000c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:48:16.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.867+0000 7f9904db2640 1 --2- 172.21.15.89:0/3532920947 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f990006e470 0x7f990006ca80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f98f00097c0 tx=0x7f98f001c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b87b6ac1fb50ec80 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:16.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.867+0000 7f98f7fff640 1 -- 172.21.15.89:0/3532920947 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f00ab650 con 0x7f990006e470 2023-10-30T12:48:16.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.867+0000 7f98f7fff640 1 -- 172.21.15.89:0/3532920947 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98f00ab7b0 con 0x7f990006e470 2023-10-30T12:48:16.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.867+0000 7f98f7fff640 1 -- 172.21.15.89:0/3532920947 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f00abb40 con 0x7f990006e470 2023-10-30T12:48:16.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.867+0000 7f9907817640 1 -- 172.21.15.89:0/3532920947 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f990006e470 msgr2=0x7f990006ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:16.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.867+0000 7f9907817640 1 --2- 172.21.15.89:0/3532920947 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f990006e470 0x7f990006ca80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f98f00097c0 tx=0x7f98f001c450 comp rx=0 tx=0).stop 2023-10-30T12:48:16.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.867+0000 7f9907817640 1 -- 172.21.15.89:0/3532920947 shutdown_connections 2023-10-30T12:48:16.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.867+0000 7f9907817640 1 --2- 172.21.15.89:0/3532920947 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99000c8490 0x7f99000c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:16.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.867+0000 7f9907817640 1 --2- 172.21.15.89:0/3532920947 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f990006e470 0x7f990006ca80 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:16.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.867+0000 7f9907817640 1 -- 172.21.15.89:0/3532920947 >> 172.21.15.89:0/3532920947 conn(0x7f9900071ea0 msgr2=0x7f9900065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:16.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.867+0000 7f9907817640 1 -- 172.21.15.89:0/3532920947 shutdown_connections 2023-10-30T12:48:16.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.867+0000 7f9907817640 1 -- 172.21.15.89:0/3532920947 wait complete. 2023-10-30T12:48:16.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.867+0000 7f9907817640 1 Processor -- start 2023-10-30T12:48:16.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.871+0000 7f9907817640 1 -- start start 2023-10-30T12:48:16.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.871+0000 7f9907817640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f990006e470 0x7f9900152000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:16.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.871+0000 7f9907817640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99000c8490 0x7f9900142e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:16.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.871+0000 7f9907817640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9900153d70 0x7f9900143380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:16.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.871+0000 7f9907817640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990014b1c0 con 0x7f9900153d70 2023-10-30T12:48:16.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.871+0000 7f9907817640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9900142a20 con 0x7f990006e470 2023-10-30T12:48:16.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.871+0000 7f9907817640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f990014ce10 con 0x7f99000c8490 2023-10-30T12:48:16.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.871+0000 7f9905db4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9900153d70 0x7f9900143380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:16.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.871+0000 7f9905db4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9900153d70 0x7f9900143380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37478/0 (socket says 172.21.15.89:37478) 2023-10-30T12:48:16.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.871+0000 7f9905db4640 1 -- 172.21.15.89:0/3016138452 learned_addr learned my addr 172.21.15.89:0/3016138452 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:16.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.871+0000 7f9904db2640 1 --2- 172.21.15.89:0/3016138452 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99000c8490 0x7f9900142e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:16.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.871+0000 7f99055b3640 1 --2- 172.21.15.89:0/3016138452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f990006e470 0x7f9900152000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:16.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.871+0000 7f9905db4640 1 -- 172.21.15.89:0/3016138452 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99000c8490 msgr2=0x7f9900142e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:16.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.871+0000 7f9905db4640 1 --2- 172.21.15.89:0/3016138452 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99000c8490 0x7f9900142e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:16.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.871+0000 7f9905db4640 1 -- 172.21.15.89:0/3016138452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f990006e470 msgr2=0x7f9900152000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:16.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.871+0000 7f9905db4640 1 --2- 172.21.15.89:0/3016138452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f990006e470 0x7f9900152000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:16.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.871+0000 7f9905db4640 1 -- 172.21.15.89:0/3016138452 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f8009a70 con 0x7f9900153d70 2023-10-30T12:48:16.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.871+0000 7f9905db4640 1 --2- 172.21.15.89:0/3016138452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9900153d70 0x7f9900143380 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f98f800efc0 tx=0x7f98f8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:16.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.871+0000 7f98f5ffb640 1 -- 172.21.15.89:0/3016138452 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f8009d70 con 0x7f9900153d70 2023-10-30T12:48:16.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.871+0000 7f98f5ffb640 1 -- 172.21.15.89:0/3016138452 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98f800f040 con 0x7f9900153d70 2023-10-30T12:48:16.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.871+0000 7f9907817640 1 -- 172.21.15.89:0/3016138452 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98f0009420 con 0x7f9900153d70 2023-10-30T12:48:16.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.871+0000 7f98f5ffb640 1 -- 172.21.15.89:0/3016138452 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f80039a0 con 0x7f9900153d70 2023-10-30T12:48:16.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.871+0000 7f9907817640 1 -- 172.21.15.89:0/3016138452 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f990014f7c0 con 0x7f9900153d70 2023-10-30T12:48:16.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.871+0000 7f9907817640 1 -- 172.21.15.89:0/3016138452 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9900142b60 con 0x7f9900153d70 2023-10-30T12:48:16.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.875+0000 7f98f5ffb640 1 -- 172.21.15.89:0/3016138452 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f98f8014480 con 0x7f9900153d70 2023-10-30T12:48:16.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.875+0000 7f98f5ffb640 1 --2- 172.21.15.89:0/3016138452 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f98e40664a0 0x7f98e4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:16.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.875+0000 7f98f5ffb640 1 -- 172.21.15.89:0/3016138452 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6578+0+0 (secure 0 0 0) 0x7f98f8084da0 con 0x7f9900153d70 2023-10-30T12:48:16.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.875+0000 7f99055b3640 1 --2- 172.21.15.89:0/3016138452 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f98e40664a0 0x7f98e4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:16.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.875+0000 7f99055b3640 1 --2- 172.21.15.89:0/3016138452 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f98e40664a0 0x7f98e4068960 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f98fc0011f0 tx=0x7f98fc007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:16.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:16.879+0000 7f98f5ffb640 1 -- 172.21.15.89:0/3016138452 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f98f8051ec0 con 0x7f9900153d70 2023-10-30T12:48:17.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:17.171+0000 7f9907817640 1 -- 172.21.15.89:0/3016138452 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f990006dd20 con 0x7f9900153d70 2023-10-30T12:48:17.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:17.171+0000 7f98f5ffb640 1 -- 172.21.15.89:0/3016138452 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+5743 (secure 0 0 0) 0x7f98f8018090 con 0x7f9900153d70 2023-10-30T12:48:17.170 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:48:17.171 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:48:11.263481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:replay","state_seq":1,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:17.171 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 92 2023-10-30T12:48:17.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:17.175+0000 7f9907817640 1 -- 172.21.15.89:0/3016138452 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f98e40664a0 msgr2=0x7f98e4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:17.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:17.175+0000 7f9907817640 1 --2- 172.21.15.89:0/3016138452 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f98e40664a0 0x7f98e4068960 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f98fc0011f0 tx=0x7f98fc007d40 comp rx=0 tx=0).stop 2023-10-30T12:48:17.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:17.175+0000 7f9907817640 1 -- 172.21.15.89:0/3016138452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9900153d70 msgr2=0x7f9900143380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:17.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:17.175+0000 7f9907817640 1 --2- 172.21.15.89:0/3016138452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9900153d70 0x7f9900143380 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f98f800efc0 tx=0x7f98f8009f90 comp rx=0 tx=0).stop 2023-10-30T12:48:17.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:17.175+0000 7f9907817640 1 -- 172.21.15.89:0/3016138452 shutdown_connections 2023-10-30T12:48:17.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:17.175+0000 7f9907817640 1 --2- 172.21.15.89:0/3016138452 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f98e40664a0 0x7f98e4068960 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:17.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:17.175+0000 7f9907817640 1 --2- 172.21.15.89:0/3016138452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9900153d70 0x7f9900143380 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:17.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:17.175+0000 7f9907817640 1 --2- 172.21.15.89:0/3016138452 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99000c8490 0x7f9900142e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:17.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:17.175+0000 7f9907817640 1 --2- 172.21.15.89:0/3016138452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f990006e470 0x7f9900152000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:17.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:17.175+0000 7f9907817640 1 -- 172.21.15.89:0/3016138452 >> 172.21.15.89:0/3016138452 conn(0x7f9900071ea0 msgr2=0x7f99000c7520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:17.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:17.175+0000 7f9907817640 1 -- 172.21.15.89:0/3016138452 shutdown_connections 2023-10-30T12:48:17.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:17.175+0000 7f9907817640 1 -- 172.21.15.89:0/3016138452 wait complete. 2023-10-30T12:48:17.192 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:48:18.159 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:48:18.163 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:48:19.193 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:19.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.339+0000 7f494a1a7640 1 Processor -- start 2023-10-30T12:48:19.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.339+0000 7f494a1a7640 1 -- start start 2023-10-30T12:48:19.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.339+0000 7f494a1a7640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f49440cefa0 0x7f49440cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:19.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.339+0000 7f494a1a7640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f494415a6c0 con 0x7f49441538f0 2023-10-30T12:48:19.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.339+0000 7f494a1a7640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49440ca6a0 con 0x7f494415cd50 2023-10-30T12:48:19.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.339+0000 7f494a1a7640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49440cf8f0 con 0x7f49440cefa0 2023-10-30T12:48:19.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.339+0000 7f49491a5640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f49441538f0 0x7f4944159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57540/0 (socket says 172.21.15.89:57540) 2023-10-30T12:48:19.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.339+0000 7f49491a5640 1 -- 172.21.15.89:0/582297989 learned_addr learned my addr 172.21.15.89:0/582297989 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:48:19.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.339+0000 7f49499a6640 1 --2- 172.21.15.89:0/582297989 >> v2:172.21.15.89:3300/0 conn(0x7f49440cefa0 0x7f49440cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:19.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.339+0000 7f493bfff640 1 -- 172.21.15.89:0/582297989 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1727606110 0 0) 0x7f494415a6c0 con 0x7f49441538f0 2023-10-30T12:48:19.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.339+0000 7f493bfff640 1 -- 172.21.15.89:0/582297989 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4930003530 con 0x7f49441538f0 2023-10-30T12:48:19.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.339+0000 7f49499a6640 1 -- 172.21.15.89:0/582297989 >> v1:172.21.15.192:6789/0 conn(0x7f494415cd50 legacy=0x7f49440c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:19.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.339+0000 7f49499a6640 1 -- 172.21.15.89:0/582297989 >> v1:172.21.15.89:6789/0 conn(0x7f49441538f0 legacy=0x7f4944159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:19.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.339+0000 7f49499a6640 1 -- 172.21.15.89:0/582297989 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f494415d300 con 0x7f49440cefa0 2023-10-30T12:48:19.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.343+0000 7f49499a6640 1 --2- 172.21.15.89:0/582297989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49440cefa0 0x7f49440cf3b0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f493c009d90 tx=0x7f493c01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11d3268733c08bf5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:19.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.343+0000 7f493bfff640 1 -- 172.21.15.89:0/582297989 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493c0af600 con 0x7f49440cefa0 2023-10-30T12:48:19.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.343+0000 7f493bfff640 1 -- 172.21.15.89:0/582297989 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f493c0ab050 con 0x7f49440cefa0 2023-10-30T12:48:19.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.343+0000 7f493bfff640 1 -- 172.21.15.89:0/582297989 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493c0af980 con 0x7f49440cefa0 2023-10-30T12:48:19.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.343+0000 7f494a1a7640 1 -- 172.21.15.89:0/582297989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49440cefa0 msgr2=0x7f49440cf3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:19.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.343+0000 7f494a1a7640 1 --2- 172.21.15.89:0/582297989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49440cefa0 0x7f49440cf3b0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f493c009d90 tx=0x7f493c01f130 comp rx=0 tx=0).stop 2023-10-30T12:48:19.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.343+0000 7f494a1a7640 1 -- 172.21.15.89:0/582297989 shutdown_connections 2023-10-30T12:48:19.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.343+0000 7f494a1a7640 1 --2- 172.21.15.89:0/582297989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49440cefa0 0x7f49440cf3b0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:19.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.343+0000 7f494a1a7640 1 -- 172.21.15.89:0/582297989 >> 172.21.15.89:0/582297989 conn(0x7f494406a640 msgr2=0x7f4944065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:19.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.343+0000 7f494a1a7640 1 -- 172.21.15.89:0/582297989 shutdown_connections 2023-10-30T12:48:19.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.343+0000 7f494a1a7640 1 -- 172.21.15.89:0/582297989 wait complete. 2023-10-30T12:48:19.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.347+0000 7f494a1a7640 1 Processor -- start 2023-10-30T12:48:19.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.347+0000 7f494a1a7640 1 -- start start 2023-10-30T12:48:19.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.347+0000 7f494a1a7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49440cefa0 0x7f49440b4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:19.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.347+0000 7f494a1a7640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49441538f0 0x7f49440cd830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:19.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.347+0000 7f494a1a7640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f494415cd50 0x7f4944142770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:19.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.347+0000 7f494a1a7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49440b46d0 con 0x7f49440cefa0 2023-10-30T12:48:19.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.347+0000 7f494a1a7640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49440b30f0 con 0x7f494415cd50 2023-10-30T12:48:19.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.347+0000 7f494a1a7640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49440b33d0 con 0x7f49441538f0 2023-10-30T12:48:19.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.347+0000 7f49491a5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49440cefa0 0x7f49440b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:19.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.347+0000 7f49491a5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49440cefa0 0x7f49440b4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37488/0 (socket says 172.21.15.89:37488) 2023-10-30T12:48:19.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.347+0000 7f49491a5640 1 -- 172.21.15.89:0/1651405751 learned_addr learned my addr 172.21.15.89:0/1651405751 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:19.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.347+0000 7f49499a6640 1 --2- 172.21.15.89:0/1651405751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f494415cd50 0x7f4944142770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:19.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.347+0000 7f49489a4640 1 --2- 172.21.15.89:0/1651405751 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49441538f0 0x7f49440cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:19.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.347+0000 7f49491a5640 1 -- 172.21.15.89:0/1651405751 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49441538f0 msgr2=0x7f49440cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:19.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.347+0000 7f49491a5640 1 --2- 172.21.15.89:0/1651405751 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49441538f0 0x7f49440cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:19.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.347+0000 7f49491a5640 1 -- 172.21.15.89:0/1651405751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f494415cd50 msgr2=0x7f4944142770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:19.348 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.347+0000 7f49491a5640 1 --2- 172.21.15.89:0/1651405751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f494415cd50 0x7f4944142770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:19.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.347+0000 7f49491a5640 1 -- 172.21.15.89:0/1651405751 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f493c009a70 con 0x7f49440cefa0 2023-10-30T12:48:19.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.347+0000 7f49491a5640 1 --2- 172.21.15.89:0/1651405751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49440cefa0 0x7f49440b4190 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f494000c4d0 tx=0x7f494000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:19.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.347+0000 7f4939ffb640 1 -- 172.21.15.89:0/1651405751 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4940005cf0 con 0x7f49440cefa0 2023-10-30T12:48:19.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.347+0000 7f4939ffb640 1 -- 172.21.15.89:0/1651405751 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4940005e50 con 0x7f49440cefa0 2023-10-30T12:48:19.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.347+0000 7f494a1a7640 1 -- 172.21.15.89:0/1651405751 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49440b36b0 con 0x7f49440cefa0 2023-10-30T12:48:19.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.347+0000 7f4939ffb640 1 -- 172.21.15.89:0/1651405751 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4940007270 con 0x7f49440cefa0 2023-10-30T12:48:19.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.347+0000 7f494a1a7640 1 -- 172.21.15.89:0/1651405751 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49440b3c70 con 0x7f49440cefa0 2023-10-30T12:48:19.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.351+0000 7f494a1a7640 1 -- 172.21.15.89:0/1651405751 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49440b3e60 con 0x7f49440cefa0 2023-10-30T12:48:19.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.351+0000 7f4939ffb640 1 -- 172.21.15.89:0/1651405751 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f49400073d0 con 0x7f49440cefa0 2023-10-30T12:48:19.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.351+0000 7f4939ffb640 1 --2- 172.21.15.89:0/1651405751 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f49280664d0 0x7f4928068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:19.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.351+0000 7f4939ffb640 1 -- 172.21.15.89:0/1651405751 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6578+0+0 (secure 0 0 0) 0x7f49400837b0 con 0x7f49440cefa0 2023-10-30T12:48:19.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.351+0000 7f49489a4640 1 --2- 172.21.15.89:0/1651405751 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f49280664d0 0x7f4928068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:19.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.351+0000 7f49489a4640 1 --2- 172.21.15.89:0/1651405751 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f49280664d0 0x7f4928068990 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f49440c5850 tx=0x7f4934009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:19.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.355+0000 7f4939ffb640 1 -- 172.21.15.89:0/1651405751 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f49400508d0 con 0x7f49440cefa0 2023-10-30T12:48:19.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.543+0000 7f494a1a7640 1 -- 172.21.15.89:0/1651405751 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f494404e170 con 0x7f49440cefa0 2023-10-30T12:48:19.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.543+0000 7f4939ffb640 1 -- 172.21.15.89:0/1651405751 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+5745 (secure 0 0 0) 0x7f4940053ef0 con 0x7f49440cefa0 2023-10-30T12:48:19.541 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:48:19.541 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:48:18.480022+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:resolve","state_seq":32,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:19.541 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 93 2023-10-30T12:48:19.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.547+0000 7f494a1a7640 1 -- 172.21.15.89:0/1651405751 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f49280664d0 msgr2=0x7f4928068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:19.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.547+0000 7f494a1a7640 1 --2- 172.21.15.89:0/1651405751 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f49280664d0 0x7f4928068990 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f49440c5850 tx=0x7f4934009040 comp rx=0 tx=0).stop 2023-10-30T12:48:19.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.547+0000 7f494a1a7640 1 -- 172.21.15.89:0/1651405751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49440cefa0 msgr2=0x7f49440b4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:19.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.547+0000 7f494a1a7640 1 --2- 172.21.15.89:0/1651405751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49440cefa0 0x7f49440b4190 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f494000c4d0 tx=0x7f494000c990 comp rx=0 tx=0).stop 2023-10-30T12:48:19.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.547+0000 7f494a1a7640 1 -- 172.21.15.89:0/1651405751 shutdown_connections 2023-10-30T12:48:19.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.547+0000 7f494a1a7640 1 --2- 172.21.15.89:0/1651405751 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f49280664d0 0x7f4928068990 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:19.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.547+0000 7f494a1a7640 1 --2- 172.21.15.89:0/1651405751 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f494415cd50 0x7f4944142770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:19.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.547+0000 7f494a1a7640 1 --2- 172.21.15.89:0/1651405751 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49441538f0 0x7f49440cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:19.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.547+0000 7f494a1a7640 1 --2- 172.21.15.89:0/1651405751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49440cefa0 0x7f49440b4190 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:19.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.547+0000 7f494a1a7640 1 -- 172.21.15.89:0/1651405751 >> 172.21.15.89:0/1651405751 conn(0x7f494406a640 msgr2=0x7f49440c48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:19.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.547+0000 7f494a1a7640 1 -- 172.21.15.89:0/1651405751 shutdown_connections 2023-10-30T12:48:19.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:19.547+0000 7f494a1a7640 1 -- 172.21.15.89:0/1651405751 wait complete. 2023-10-30T12:48:19.561 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:48:21.562 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:21.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.707+0000 7f6549046640 1 Processor -- start 2023-10-30T12:48:21.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.707+0000 7f6549046640 1 -- start start 2023-10-30T12:48:21.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.707+0000 7f6549046640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65440c6b80 0x7f65440bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:21.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.707+0000 7f6549046640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f65440c6f60 0x7f65440c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:21.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.707+0000 7f6549046640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65440c0a50 con 0x7f65440c8430 2023-10-30T12:48:21.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.707+0000 7f6549046640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f654406f6e0 con 0x7f65440c6f60 2023-10-30T12:48:21.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.707+0000 7f6549046640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f654406f9c0 con 0x7f65440c6b80 2023-10-30T12:48:21.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.707+0000 7f6542d9d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65440c6b80 0x7f65440bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:21.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.707+0000 7f654259c640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f65440c6f60 0x7f65440c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:21.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.707+0000 7f6542d9d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65440c6b80 0x7f65440bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:56456/0 (socket says 172.21.15.89:56456) 2023-10-30T12:48:21.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.707+0000 7f654259c640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f65440c6f60 0x7f65440c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:44818/0 (socket says 172.21.15.89:44818) 2023-10-30T12:48:21.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.707+0000 7f6542d9d640 1 -- 172.21.15.89:0/3108715253 learned_addr learned my addr 172.21.15.89:0/3108715253 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:21.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.707+0000 7f6542d9d640 1 -- 172.21.15.89:0/3108715253 >> v2:172.21.15.192:3300/0 conn(0x7f65440c6f60 msgr2=0x7f65440c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:21.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.707+0000 7f6542d9d640 1 --2- 172.21.15.89:0/3108715253 >> v2:172.21.15.192:3300/0 conn(0x7f65440c6f60 0x7f65440c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:21.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.707+0000 7f6542d9d640 1 -- 172.21.15.89:0/3108715253 >> v1:172.21.15.192:6789/0 conn(0x7f65440c8430 legacy=0x7f654413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:21.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.707+0000 7f6542d9d640 1 -- 172.21.15.89:0/3108715253 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f654406fca0 con 0x7f65440c6b80 2023-10-30T12:48:21.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.711+0000 7f6542d9d640 1 --2- 172.21.15.89:0/3108715253 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65440c6b80 0x7f65440bfec0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6534009870 tx=0x7f653401c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cbaad1594890bf4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:21.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.711+0000 7f6541d9b640 1 -- 172.21.15.89:0/3108715253 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65340ab650 con 0x7f65440c6b80 2023-10-30T12:48:21.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.711+0000 7f6541d9b640 1 -- 172.21.15.89:0/3108715253 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65340ab7b0 con 0x7f65440c6b80 2023-10-30T12:48:21.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.711+0000 7f6541d9b640 1 -- 172.21.15.89:0/3108715253 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65340abb40 con 0x7f65440c6b80 2023-10-30T12:48:21.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.715+0000 7f6549046640 1 -- 172.21.15.89:0/3108715253 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65440c6b80 msgr2=0x7f65440bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:21.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.715+0000 7f6549046640 1 --2- 172.21.15.89:0/3108715253 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65440c6b80 0x7f65440bfec0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6534009870 tx=0x7f653401c540 comp rx=0 tx=0).stop 2023-10-30T12:48:21.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.715+0000 7f6549046640 1 -- 172.21.15.89:0/3108715253 shutdown_connections 2023-10-30T12:48:21.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.715+0000 7f6549046640 1 --2- 172.21.15.89:0/3108715253 >> v2:172.21.15.192:3300/0 conn(0x7f65440c6f60 0x7f65440c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:21.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.715+0000 7f6549046640 1 --2- 172.21.15.89:0/3108715253 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65440c6b80 0x7f65440bfec0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:21.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.715+0000 7f6549046640 1 -- 172.21.15.89:0/3108715253 >> 172.21.15.89:0/3108715253 conn(0x7f65440c7910 msgr2=0x7f65440c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:21.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.715+0000 7f6549046640 1 -- 172.21.15.89:0/3108715253 shutdown_connections 2023-10-30T12:48:21.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.715+0000 7f6549046640 1 -- 172.21.15.89:0/3108715253 wait complete. 2023-10-30T12:48:21.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.715+0000 7f6549046640 1 Processor -- start 2023-10-30T12:48:21.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.715+0000 7f6549046640 1 -- start start 2023-10-30T12:48:21.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.715+0000 7f6549046640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65440c6b80 0x7f65441e96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:21.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.715+0000 7f6549046640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65440c6f60 0x7f654415e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:21.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.715+0000 7f6549046640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65440c8430 0x7f65440c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:21.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.715+0000 7f6549046640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6544155990 con 0x7f65440c6f60 2023-10-30T12:48:21.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.715+0000 7f6549046640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65440ca700 con 0x7f65440c6b80 2023-10-30T12:48:21.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.715+0000 7f6549046640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f654415e450 con 0x7f65440c8430 2023-10-30T12:48:21.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.715+0000 7f654259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65440c6f60 0x7f654415e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:21.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.715+0000 7f654259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65440c6f60 0x7f654415e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34936/0 (socket says 172.21.15.89:34936) 2023-10-30T12:48:21.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.715+0000 7f654259c640 1 -- 172.21.15.89:0/94533713 learned_addr learned my addr 172.21.15.89:0/94533713 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:21.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.715+0000 7f6542d9d640 1 --2- 172.21.15.89:0/94533713 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65440c6b80 0x7f65441e96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:21.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.715+0000 7f654359e640 1 --2- 172.21.15.89:0/94533713 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65440c8430 0x7f65440c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:21.718 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.719+0000 7f654259c640 1 -- 172.21.15.89:0/94533713 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65440c8430 msgr2=0x7f65440c8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:21.718 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.719+0000 7f654259c640 1 --2- 172.21.15.89:0/94533713 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65440c8430 0x7f65440c8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:21.718 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.719+0000 7f654259c640 1 -- 172.21.15.89:0/94533713 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65440c6b80 msgr2=0x7f65441e96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:21.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.719+0000 7f654259c640 1 --2- 172.21.15.89:0/94533713 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65440c6b80 0x7f65441e96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:21.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.719+0000 7f654259c640 1 -- 172.21.15.89:0/94533713 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65340094d0 con 0x7f65440c6f60 2023-10-30T12:48:21.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.719+0000 7f654259c640 1 --2- 172.21.15.89:0/94533713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65440c6f60 0x7f654415e740 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f653800e900 tx=0x7f653800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:21.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.719+0000 7f652b7fe640 1 -- 172.21.15.89:0/94533713 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6538009690 con 0x7f65440c6f60 2023-10-30T12:48:21.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.719+0000 7f652b7fe640 1 -- 172.21.15.89:0/94533713 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f653800ca00 con 0x7f65440c6f60 2023-10-30T12:48:21.719 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.719+0000 7f652b7fe640 1 -- 172.21.15.89:0/94533713 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653800cdb0 con 0x7f65440c6f60 2023-10-30T12:48:21.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.719+0000 7f6549046640 1 -- 172.21.15.89:0/94533713 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65440c04d0 con 0x7f65440c6f60 2023-10-30T12:48:21.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.719+0000 7f6549046640 1 -- 172.21.15.89:0/94533713 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65440ca910 con 0x7f65440c6f60 2023-10-30T12:48:21.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.719+0000 7f6549046640 1 -- 172.21.15.89:0/94533713 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65441575c0 con 0x7f65440c6f60 2023-10-30T12:48:21.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.719+0000 7f652b7fe640 1 -- 172.21.15.89:0/94533713 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6538013470 con 0x7f65440c6f60 2023-10-30T12:48:21.720 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.719+0000 7f652b7fe640 1 --2- 172.21.15.89:0/94533713 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f65300664a0 0x7f6530068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:21.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.719+0000 7f652b7fe640 1 -- 172.21.15.89:0/94533713 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6578+0+0 (secure 0 0 0) 0x7f6538084710 con 0x7f65440c6f60 2023-10-30T12:48:21.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.723+0000 7f6542d9d640 1 --2- 172.21.15.89:0/94533713 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f65300664a0 0x7f6530068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:21.721 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.723+0000 7f6542d9d640 1 --2- 172.21.15.89:0/94533713 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f65300664a0 0x7f6530068960 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f65340af000 tx=0x7f65340b6040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:21.722 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.723+0000 7f652b7fe640 1 -- 172.21.15.89:0/94533713 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f65380518d0 con 0x7f65440c6f60 2023-10-30T12:48:21.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.907+0000 7f6549046640 1 -- 172.21.15.89:0/94533713 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f654404e170 con 0x7f65440c6f60 2023-10-30T12:48:21.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.907+0000 7f652b7fe640 1 -- 172.21.15.89:0/94533713 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+5745 (secure 0 0 0) 0x7f6538054ef0 con 0x7f65440c6f60 2023-10-30T12:48:21.905 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:48:21.905 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:48:18.480022+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:resolve","state_seq":32,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:21.905 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 93 2023-10-30T12:48:21.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.911+0000 7f6549046640 1 -- 172.21.15.89:0/94533713 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f65300664a0 msgr2=0x7f6530068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:21.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.911+0000 7f6549046640 1 --2- 172.21.15.89:0/94533713 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f65300664a0 0x7f6530068960 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f65340af000 tx=0x7f65340b6040 comp rx=0 tx=0).stop 2023-10-30T12:48:21.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.911+0000 7f6549046640 1 -- 172.21.15.89:0/94533713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65440c6f60 msgr2=0x7f654415e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:21.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.911+0000 7f6549046640 1 --2- 172.21.15.89:0/94533713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65440c6f60 0x7f654415e740 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f653800e900 tx=0x7f653800edc0 comp rx=0 tx=0).stop 2023-10-30T12:48:21.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.911+0000 7f6549046640 1 -- 172.21.15.89:0/94533713 shutdown_connections 2023-10-30T12:48:21.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.911+0000 7f6549046640 1 --2- 172.21.15.89:0/94533713 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f65300664a0 0x7f6530068960 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:21.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.911+0000 7f6549046640 1 --2- 172.21.15.89:0/94533713 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f65440c8430 0x7f65440c8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:21.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.911+0000 7f6549046640 1 --2- 172.21.15.89:0/94533713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65440c6f60 0x7f654415e740 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:21.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.911+0000 7f6549046640 1 --2- 172.21.15.89:0/94533713 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65440c6b80 0x7f65441e96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:21.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.911+0000 7f6549046640 1 -- 172.21.15.89:0/94533713 >> 172.21.15.89:0/94533713 conn(0x7f65440c7910 msgr2=0x7f6544139f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:21.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.911+0000 7f6549046640 1 -- 172.21.15.89:0/94533713 shutdown_connections 2023-10-30T12:48:21.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:21.911+0000 7f6549046640 1 -- 172.21.15.89:0/94533713 wait complete. 2023-10-30T12:48:21.926 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:48:23.927 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:24.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.079+0000 7f71f4ad6640 1 Processor -- start 2023-10-30T12:48:24.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.079+0000 7f71f4ad6640 1 -- start start 2023-10-30T12:48:24.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.083+0000 7f71f4ad6640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71f00c05d0 0x7f71f013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:24.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.083+0000 7f71f4ad6640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71f00c0a50 con 0x7f71f00c6f60 2023-10-30T12:48:24.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.083+0000 7f71f4ad6640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71f006f6e0 con 0x7f71f00c6b80 2023-10-30T12:48:24.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.083+0000 7f71f4ad6640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f006f9c0 con 0x7f71f00c05d0 2023-10-30T12:48:24.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.083+0000 7f71eed9d640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f71f00c6f60 0x7f71f01566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:45974/0 (socket says 172.21.15.89:45974) 2023-10-30T12:48:24.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.083+0000 7f71eed9d640 1 -- 172.21.15.89:0/1003610583 learned_addr learned my addr 172.21.15.89:0/1003610583 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:48:24.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.083+0000 7f71e5d9b640 1 --2- 172.21.15.89:0/1003610583 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71f00c05d0 0x7f71f013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:24.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.083+0000 7f71edd9b640 1 -- 172.21.15.89:0/1003610583 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595294405 0 0) 0x7f71f00c0a50 con 0x7f71f00c6f60 2023-10-30T12:48:24.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.083+0000 7f71edd9b640 1 -- 172.21.15.89:0/1003610583 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71dc003530 con 0x7f71f00c6f60 2023-10-30T12:48:24.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.083+0000 7f71edd9b640 1 -- 172.21.15.89:0/1003610583 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3834982899 0 0) 0x7f71dc003530 con 0x7f71f00c6f60 2023-10-30T12:48:24.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.083+0000 7f71edd9b640 1 -- 172.21.15.89:0/1003610583 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71f00c05d0 msgr2=0x7f71f013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:24.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.083+0000 7f71edd9b640 1 --2- 172.21.15.89:0/1003610583 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71f00c05d0 0x7f71f013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:24.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.083+0000 7f71edd9b640 1 -- 172.21.15.89:0/1003610583 >> v1:172.21.15.192:6789/0 conn(0x7f71f00c6b80 legacy=0x7f71f00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:24.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.083+0000 7f71edd9b640 1 -- 172.21.15.89:0/1003610583 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f006fca0 con 0x7f71f00c6f60 2023-10-30T12:48:24.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.083+0000 7f71edd9b640 1 -- 172.21.15.89:0/1003610583 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f71e8001a00 con 0x7f71f00c6f60 2023-10-30T12:48:24.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.083+0000 7f71edd9b640 1 -- 172.21.15.89:0/1003610583 >> v1:172.21.15.89:6789/0 conn(0x7f71f00c6f60 legacy=0x7f71f01566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:24.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.083+0000 7f71edd9b640 1 --2- 172.21.15.89:0/1003610583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71dc0039b0 0x7f71dc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:24.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.083+0000 7f71edd9b640 1 --2- 172.21.15.89:0/1003610583 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71dc0047d0 0x7f71dc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:24.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.083+0000 7f71edd9b640 1 --2- 172.21.15.89:0/1003610583 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71f00c05d0 0x7f71dc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:24.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.083+0000 7f71edd9b640 1 -- 172.21.15.89:0/1003610583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f006f9c0 con 0x7f71dc0039b0 2023-10-30T12:48:24.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.083+0000 7f71edd9b640 1 -- 172.21.15.89:0/1003610583 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71dc00ba70 con 0x7f71dc0047d0 2023-10-30T12:48:24.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.083+0000 7f71edd9b640 1 -- 172.21.15.89:0/1003610583 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71dc00de40 con 0x7f71f00c05d0 2023-10-30T12:48:24.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.083+0000 7f71ee59c640 1 --2- 172.21.15.89:0/1003610583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71dc0039b0 0x7f71dc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:24.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.083+0000 7f71eed9d640 1 --2- 172.21.15.89:0/1003610583 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71f00c05d0 0x7f71dc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:24.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.083+0000 7f71e5d9b640 1 --2- 172.21.15.89:0/1003610583 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71dc0047d0 0x7f71dc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:24.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.083+0000 7f71ee59c640 1 -- 172.21.15.89:0/1003610583 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71f00c05d0 msgr2=0x7f71dc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:24.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.083+0000 7f71ee59c640 1 --2- 172.21.15.89:0/1003610583 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71f00c05d0 0x7f71dc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:24.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.083+0000 7f71ee59c640 1 -- 172.21.15.89:0/1003610583 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71dc0047d0 msgr2=0x7f71dc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:24.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.083+0000 7f71ee59c640 1 --2- 172.21.15.89:0/1003610583 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71dc0047d0 0x7f71dc004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:24.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.083+0000 7f71ee59c640 1 -- 172.21.15.89:0/1003610583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71dc003530 con 0x7f71dc0039b0 2023-10-30T12:48:24.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.087+0000 7f71ee59c640 1 --2- 172.21.15.89:0/1003610583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71dc0039b0 0x7f71dc003dd0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f71e0001750 tx=0x7f71e001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d73a85d729745dc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:24.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.087+0000 7f71edd9b640 1 -- 172.21.15.89:0/1003610583 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e00052d0 con 0x7f71dc0039b0 2023-10-30T12:48:24.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.087+0000 7f71edd9b640 1 -- 172.21.15.89:0/1003610583 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e00a7050 con 0x7f71dc0039b0 2023-10-30T12:48:24.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.087+0000 7f71edd9b640 1 -- 172.21.15.89:0/1003610583 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e0003600 con 0x7f71dc0039b0 2023-10-30T12:48:24.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.087+0000 7f71f4ad6640 1 -- 172.21.15.89:0/1003610583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71dc0039b0 msgr2=0x7f71dc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:24.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.087+0000 7f71f4ad6640 1 --2- 172.21.15.89:0/1003610583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71dc0039b0 0x7f71dc003dd0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f71e0001750 tx=0x7f71e001c3a0 comp rx=0 tx=0).stop 2023-10-30T12:48:24.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.087+0000 7f71eed9d640 1 -- 172.21.15.89:0/1003610583 reap_dead start 2023-10-30T12:48:24.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.087+0000 7f71f4ad6640 1 -- 172.21.15.89:0/1003610583 shutdown_connections 2023-10-30T12:48:24.091 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.087+0000 7f71f4ad6640 1 -- 172.21.15.89:0/1003610583 >> 172.21.15.89:0/1003610583 conn(0x7f71f00c7910 msgr2=0x7f71f00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:24.091 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.087+0000 7f71f4ad6640 1 -- 172.21.15.89:0/1003610583 shutdown_connections 2023-10-30T12:48:24.091 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.087+0000 7f71f4ad6640 1 -- 172.21.15.89:0/1003610583 wait complete. 2023-10-30T12:48:24.091 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.087+0000 7f71f4ad6640 1 Processor -- start 2023-10-30T12:48:24.091 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.091+0000 7f71f4ad6640 1 -- start start 2023-10-30T12:48:24.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.091+0000 7f71f4ad6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71dc0039b0 0x7f71f01f1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:24.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.091+0000 7f71f4ad6640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71f01549c0 0x7f71f00c4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:24.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.091+0000 7f71f4ad6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71f00c1b60 0x7f71f0154e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:24.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.091+0000 7f71f4ad6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f0155a80 con 0x7f71f00c1b60 2023-10-30T12:48:24.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.091+0000 7f71f4ad6640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f013a580 con 0x7f71dc0039b0 2023-10-30T12:48:24.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.091+0000 7f71f4ad6640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f0154530 con 0x7f71f01549c0 2023-10-30T12:48:24.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.091+0000 7f71eed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71f00c1b60 0x7f71f0154e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:24.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.091+0000 7f71eed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71f00c1b60 0x7f71f0154e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34962/0 (socket says 172.21.15.89:34962) 2023-10-30T12:48:24.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.091+0000 7f71eed9d640 1 -- 172.21.15.89:0/2990216812 learned_addr learned my addr 172.21.15.89:0/2990216812 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:24.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.091+0000 7f71ee59c640 1 --2- 172.21.15.89:0/2990216812 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71dc0039b0 0x7f71f01f1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:24.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.091+0000 7f71e5d9b640 1 --2- 172.21.15.89:0/2990216812 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71f01549c0 0x7f71f00c4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:24.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.091+0000 7f71eed9d640 1 -- 172.21.15.89:0/2990216812 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71f01549c0 msgr2=0x7f71f00c4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:24.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.091+0000 7f71eed9d640 1 --2- 172.21.15.89:0/2990216812 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71f01549c0 0x7f71f00c4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:24.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.091+0000 7f71eed9d640 1 -- 172.21.15.89:0/2990216812 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71dc0039b0 msgr2=0x7f71f01f1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:24.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.091+0000 7f71eed9d640 1 --2- 172.21.15.89:0/2990216812 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71dc0039b0 0x7f71f01f1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:24.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.091+0000 7f71eed9d640 1 -- 172.21.15.89:0/2990216812 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f01f5ec0 con 0x7f71f00c1b60 2023-10-30T12:48:24.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.091+0000 7f71ee59c640 1 --2- 172.21.15.89:0/2990216812 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71dc0039b0 0x7f71f01f1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:24.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.091+0000 7f71eed9d640 1 --2- 172.21.15.89:0/2990216812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71f00c1b60 0x7f71f0154e40 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f71f00b1190 tx=0x7f71e800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:24.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.091+0000 7f71e77fe640 1 -- 172.21.15.89:0/2990216812 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e800ebe0 con 0x7f71f00c1b60 2023-10-30T12:48:24.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.091+0000 7f71e77fe640 1 -- 172.21.15.89:0/2990216812 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e800ed40 con 0x7f71f00c1b60 2023-10-30T12:48:24.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.091+0000 7f71f4ad6640 1 -- 172.21.15.89:0/2990216812 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f013e440 con 0x7f71f00c1b60 2023-10-30T12:48:24.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.091+0000 7f71e77fe640 1 -- 172.21.15.89:0/2990216812 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e8014670 con 0x7f71f00c1b60 2023-10-30T12:48:24.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.091+0000 7f71f4ad6640 1 -- 172.21.15.89:0/2990216812 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f013a7d0 con 0x7f71f00c1b60 2023-10-30T12:48:24.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.091+0000 7f71f4ad6640 1 -- 172.21.15.89:0/2990216812 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71f004e170 con 0x7f71f00c1b60 2023-10-30T12:48:24.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.095+0000 7f71e77fe640 1 -- 172.21.15.89:0/2990216812 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f71e80147d0 con 0x7f71f00c1b60 2023-10-30T12:48:24.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.095+0000 7f71e77fe640 1 --2- 172.21.15.89:0/2990216812 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f71d00664d0 0x7f71d0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:24.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.095+0000 7f71e77fe640 1 -- 172.21.15.89:0/2990216812 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6578+0+0 (secure 0 0 0) 0x7f71e8083d60 con 0x7f71f00c1b60 2023-10-30T12:48:24.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.095+0000 7f71ee59c640 1 --2- 172.21.15.89:0/2990216812 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f71d00664d0 0x7f71d0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:24.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.095+0000 7f71ee59c640 1 --2- 172.21.15.89:0/2990216812 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f71d00664d0 0x7f71d0068990 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f71dc0057b0 tx=0x7f71e00011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:24.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.099+0000 7f71e77fe640 1 -- 172.21.15.89:0/2990216812 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f71e8051050 con 0x7f71f00c1b60 2023-10-30T12:48:24.274 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.275+0000 7f71f4ad6640 1 -- 172.21.15.89:0/2990216812 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71f00bff60 con 0x7f71f00c1b60 2023-10-30T12:48:24.275 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.279+0000 7f71e77fe640 1 -- 172.21.15.89:0/2990216812 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+5745 (secure 0 0 0) 0x7f71e8054670 con 0x7f71f00c1b60 2023-10-30T12:48:24.275 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:48:24.275 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:48:18.480022+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:resolve","state_seq":32,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:24.277 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 93 2023-10-30T12:48:24.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.279+0000 7f71f4ad6640 1 -- 172.21.15.89:0/2990216812 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f71d00664d0 msgr2=0x7f71d0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:24.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.279+0000 7f71f4ad6640 1 --2- 172.21.15.89:0/2990216812 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f71d00664d0 0x7f71d0068990 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f71dc0057b0 tx=0x7f71e00011a0 comp rx=0 tx=0).stop 2023-10-30T12:48:24.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.279+0000 7f71f4ad6640 1 -- 172.21.15.89:0/2990216812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71f00c1b60 msgr2=0x7f71f0154e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:24.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.279+0000 7f71f4ad6640 1 --2- 172.21.15.89:0/2990216812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71f00c1b60 0x7f71f0154e40 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f71f00b1190 tx=0x7f71e800e530 comp rx=0 tx=0).stop 2023-10-30T12:48:24.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.279+0000 7f71f4ad6640 1 -- 172.21.15.89:0/2990216812 shutdown_connections 2023-10-30T12:48:24.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.279+0000 7f71f4ad6640 1 --2- 172.21.15.89:0/2990216812 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f71d00664d0 0x7f71d0068990 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:24.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.279+0000 7f71f4ad6640 1 --2- 172.21.15.89:0/2990216812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71f00c1b60 0x7f71f0154e40 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:24.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.279+0000 7f71f4ad6640 1 --2- 172.21.15.89:0/2990216812 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f71f01549c0 0x7f71f00c4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:24.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.279+0000 7f71f4ad6640 1 --2- 172.21.15.89:0/2990216812 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71dc0039b0 0x7f71f01f1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:24.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.279+0000 7f71f4ad6640 1 -- 172.21.15.89:0/2990216812 >> 172.21.15.89:0/2990216812 conn(0x7f71f00c7910 msgr2=0x7f71f013bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:24.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.279+0000 7f71f4ad6640 1 -- 172.21.15.89:0/2990216812 shutdown_connections 2023-10-30T12:48:24.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:24.279+0000 7f71f4ad6640 1 -- 172.21.15.89:0/2990216812 wait complete. 2023-10-30T12:48:24.296 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 93, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:50.729245+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 93, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5575": { "addr": "172.21.15.192:6835/501291700", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6834", "nonce": 501291700, "type": "v2" }, { "addr": "172.21.15.192:6835", "nonce": 501291700, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5575, "incarnation": 63, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 37 }, "gid_5728": { "addr": "172.21.15.192:6839/3991891597", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6838", "nonce": 3991891597, "type": "v2" }, { "addr": "172.21.15.192:6839", "nonce": 3991891597, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 5728, "incarnation": 73, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 19 }, "gid_6075": { "addr": "172.21.15.89:6837/2412618858", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6836", "nonce": 2412618858, "type": "v2" }, { "addr": "172.21.15.89:6837", "nonce": 2412618858, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6075, "incarnation": 92, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:resolve", "state_seq": 32 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:48:18.480022+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5575, "mds_1": 6075, "mds_2": 5728 } } } ], "standbys": [ { "addr": "172.21.15.89:6834/2909790173", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6832", "nonce": 2909790173, "type": "v2" }, { "addr": "172.21.15.89:6834", "nonce": 2909790173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.89:6835/2387357615", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6833", "nonce": 2387357615, "type": "v2" }, { "addr": "172.21.15.89:6835", "nonce": 2387357615, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4239, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:48:26.297 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:26.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c6cf48640 1 Processor -- start 2023-10-30T12:48:26.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c6cf48640 1 -- start start 2023-10-30T12:48:26.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c6cf48640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3c680cefa0 0x7f3c680cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:26.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c6cf48640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c6815a6c0 con 0x7f3c6815cd50 2023-10-30T12:48:26.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c6cf48640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c680ca6a0 con 0x7f3c681538f0 2023-10-30T12:48:26.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c6cf48640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6815d300 con 0x7f3c680cefa0 2023-10-30T12:48:26.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c5ffff640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f3c6815cd50 0x7f3c680c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:45982/0 (socket says 172.21.15.89:45982) 2023-10-30T12:48:26.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c5ffff640 1 -- 172.21.15.89:0/787219096 learned_addr learned my addr 172.21.15.89:0/787219096 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:48:26.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c67fff640 1 --2- 172.21.15.89:0/787219096 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3c680cefa0 0x7f3c680cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:26.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c66ffd640 1 -- 172.21.15.89:0/787219096 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564583912 0 0) 0x7f3c6815a6c0 con 0x7f3c6815cd50 2023-10-30T12:48:26.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c66ffd640 1 -- 172.21.15.89:0/787219096 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c54003530 con 0x7f3c6815cd50 2023-10-30T12:48:26.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c66ffd640 1 -- 172.21.15.89:0/787219096 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3408522918 0 0) 0x7f3c54003530 con 0x7f3c6815cd50 2023-10-30T12:48:26.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c66ffd640 1 -- 172.21.15.89:0/787219096 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3c680cefa0 msgr2=0x7f3c680cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:26.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c66ffd640 1 --2- 172.21.15.89:0/787219096 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3c680cefa0 0x7f3c680cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:26.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c66ffd640 1 -- 172.21.15.89:0/787219096 >> v1:172.21.15.192:6789/0 conn(0x7f3c681538f0 legacy=0x7f3c68159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:26.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c66ffd640 1 -- 172.21.15.89:0/787219096 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c6815d5e0 con 0x7f3c6815cd50 2023-10-30T12:48:26.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c66ffd640 1 -- 172.21.15.89:0/787219096 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f3c50001680 con 0x7f3c6815cd50 2023-10-30T12:48:26.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c66ffd640 1 -- 172.21.15.89:0/787219096 >> v1:172.21.15.89:6789/0 conn(0x7f3c6815cd50 legacy=0x7f3c680c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:26.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c66ffd640 1 --2- 172.21.15.89:0/787219096 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c540039b0 0x7f3c54003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:26.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c66ffd640 1 --2- 172.21.15.89:0/787219096 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3c680cefa0 0x7f3c540047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:26.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c66ffd640 1 --2- 172.21.15.89:0/787219096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c54004d10 0x7f3c54005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:26.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c66ffd640 1 -- 172.21.15.89:0/787219096 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6815d300 con 0x7f3c54004d10 2023-10-30T12:48:26.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c66ffd640 1 -- 172.21.15.89:0/787219096 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5400bad0 con 0x7f3c540039b0 2023-10-30T12:48:26.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c66ffd640 1 -- 172.21.15.89:0/787219096 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5400bc90 con 0x7f3c680cefa0 2023-10-30T12:48:26.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c67fff640 1 --2- 172.21.15.89:0/787219096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c54004d10 0x7f3c54005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:26.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c677fe640 1 --2- 172.21.15.89:0/787219096 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c540039b0 0x7f3c54003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:26.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c5ffff640 1 --2- 172.21.15.89:0/787219096 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3c680cefa0 0x7f3c540047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:26.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c67fff640 1 -- 172.21.15.89:0/787219096 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3c680cefa0 msgr2=0x7f3c540047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:26.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c67fff640 1 --2- 172.21.15.89:0/787219096 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3c680cefa0 0x7f3c540047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:26.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c67fff640 1 -- 172.21.15.89:0/787219096 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c540039b0 msgr2=0x7f3c54003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:26.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c67fff640 1 --2- 172.21.15.89:0/787219096 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c540039b0 0x7f3c54003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:26.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.451+0000 7f3c67fff640 1 -- 172.21.15.89:0/787219096 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c54003530 con 0x7f3c54004d10 2023-10-30T12:48:26.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.455+0000 7f3c67fff640 1 --2- 172.21.15.89:0/787219096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c54004d10 0x7f3c54005190 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f3c60009f90 tx=0x7f3c6001cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=544641c1cff6a51f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:26.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.455+0000 7f3c66ffd640 1 -- 172.21.15.89:0/787219096 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c600a5c30 con 0x7f3c54004d10 2023-10-30T12:48:26.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.455+0000 7f3c66ffd640 1 -- 172.21.15.89:0/787219096 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c6000f040 con 0x7f3c54004d10 2023-10-30T12:48:26.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.455+0000 7f3c66ffd640 1 -- 172.21.15.89:0/787219096 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c600ad440 con 0x7f3c54004d10 2023-10-30T12:48:26.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.455+0000 7f3c6cf48640 1 -- 172.21.15.89:0/787219096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c54004d10 msgr2=0x7f3c54005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:26.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.455+0000 7f3c6cf48640 1 --2- 172.21.15.89:0/787219096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c54004d10 0x7f3c54005190 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f3c60009f90 tx=0x7f3c6001cc20 comp rx=0 tx=0).stop 2023-10-30T12:48:26.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.455+0000 7f3c67fff640 1 -- 172.21.15.89:0/787219096 reap_dead start 2023-10-30T12:48:26.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.455+0000 7f3c6cf48640 1 -- 172.21.15.89:0/787219096 shutdown_connections 2023-10-30T12:48:26.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.455+0000 7f3c6cf48640 1 -- 172.21.15.89:0/787219096 >> 172.21.15.89:0/787219096 conn(0x7f3c6806a640 msgr2=0x7f3c68065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:26.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.455+0000 7f3c6cf48640 1 -- 172.21.15.89:0/787219096 shutdown_connections 2023-10-30T12:48:26.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.455+0000 7f3c6cf48640 1 -- 172.21.15.89:0/787219096 wait complete. 2023-10-30T12:48:26.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.459+0000 7f3c6cf48640 1 Processor -- start 2023-10-30T12:48:26.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.459+0000 7f3c6cf48640 1 -- start start 2023-10-30T12:48:26.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.459+0000 7f3c6cf48640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c54004d10 0x7f3c68147890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:26.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.459+0000 7f3c6cf48640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c6815c230 0x7f3c681efa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:26.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.459+0000 7f3c6cf48640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3c680b33e0 0x7f3c680b3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:26.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.459+0000 7f3c6cf48640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c680b4400 con 0x7f3c54004d10 2023-10-30T12:48:26.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.459+0000 7f3c6cf48640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6814d1d0 con 0x7f3c6815c230 2023-10-30T12:48:26.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.459+0000 7f3c6cf48640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c681ef2a0 con 0x7f3c680b33e0 2023-10-30T12:48:26.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.459+0000 7f3c677fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c54004d10 0x7f3c68147890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:26.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.459+0000 7f3c677fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c54004d10 0x7f3c68147890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34984/0 (socket says 172.21.15.89:34984) 2023-10-30T12:48:26.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.459+0000 7f3c677fe640 1 -- 172.21.15.89:0/2394679578 learned_addr learned my addr 172.21.15.89:0/2394679578 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:26.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.459+0000 7f3c5ffff640 1 --2- 172.21.15.89:0/2394679578 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c6815c230 0x7f3c681efa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:26.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.459+0000 7f3c67fff640 1 --2- 172.21.15.89:0/2394679578 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3c680b33e0 0x7f3c680b3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:26.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.459+0000 7f3c677fe640 1 -- 172.21.15.89:0/2394679578 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3c680b33e0 msgr2=0x7f3c680b3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:26.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.459+0000 7f3c677fe640 1 --2- 172.21.15.89:0/2394679578 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3c680b33e0 0x7f3c680b3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:26.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.459+0000 7f3c677fe640 1 -- 172.21.15.89:0/2394679578 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c6815c230 msgr2=0x7f3c681efa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:26.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.459+0000 7f3c677fe640 1 --2- 172.21.15.89:0/2394679578 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c6815c230 0x7f3c681efa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:26.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.459+0000 7f3c677fe640 1 -- 172.21.15.89:0/2394679578 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c681f3470 con 0x7f3c54004d10 2023-10-30T12:48:26.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.459+0000 7f3c5ffff640 1 --2- 172.21.15.89:0/2394679578 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c6815c230 0x7f3c681efa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:26.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.459+0000 7f3c677fe640 1 --2- 172.21.15.89:0/2394679578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c54004d10 0x7f3c68147890 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f3c5800e580 tx=0x7f3c5800ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:26.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.459+0000 7f3c64ff9640 1 -- 172.21.15.89:0/2394679578 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c580052e0 con 0x7f3c54004d10 2023-10-30T12:48:26.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.459+0000 7f3c64ff9640 1 -- 172.21.15.89:0/2394679578 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c58005440 con 0x7f3c54004d10 2023-10-30T12:48:26.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.459+0000 7f3c64ff9640 1 -- 172.21.15.89:0/2394679578 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c58003820 con 0x7f3c54004d10 2023-10-30T12:48:26.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.459+0000 7f3c6cf48640 1 -- 172.21.15.89:0/2394679578 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c6814d420 con 0x7f3c54004d10 2023-10-30T12:48:26.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.459+0000 7f3c6cf48640 1 -- 172.21.15.89:0/2394679578 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c6814d700 con 0x7f3c54004d10 2023-10-30T12:48:26.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.463+0000 7f3c6cf48640 1 -- 172.21.15.89:0/2394679578 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c6804e170 con 0x7f3c54004d10 2023-10-30T12:48:26.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.463+0000 7f3c64ff9640 1 -- 172.21.15.89:0/2394679578 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3c58003980 con 0x7f3c54004d10 2023-10-30T12:48:26.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.463+0000 7f3c64ff9640 1 --2- 172.21.15.89:0/2394679578 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3c480664a0 0x7f3c48068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:26.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.463+0000 7f3c64ff9640 1 -- 172.21.15.89:0/2394679578 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6578+0+0 (secure 0 0 0) 0x7f3c58083ff0 con 0x7f3c54004d10 2023-10-30T12:48:26.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.463+0000 7f3c5ffff640 1 --2- 172.21.15.89:0/2394679578 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3c480664a0 0x7f3c48068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:26.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.463+0000 7f3c5ffff640 1 --2- 172.21.15.89:0/2394679578 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3c480664a0 0x7f3c48068960 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3c6815c980 tx=0x7f3c50009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:26.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.467+0000 7f3c64ff9640 1 -- 172.21.15.89:0/2394679578 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3c580b5090 con 0x7f3c54004d10 2023-10-30T12:48:26.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.647+0000 7f3c6cf48640 1 -- 172.21.15.89:0/2394679578 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c680b2670 con 0x7f3c54004d10 2023-10-30T12:48:26.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.647+0000 7f3c64ff9640 1 -- 172.21.15.89:0/2394679578 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+5747 (secure 0 0 0) 0x7f3c580510b0 con 0x7f3c54004d10 2023-10-30T12:48:26.646 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:48:26.646 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:48:24.490193+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:reconnect","state_seq":34,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:26.647 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 94 2023-10-30T12:48:26.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.647+0000 7f3c6cf48640 1 -- 172.21.15.89:0/2394679578 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3c480664a0 msgr2=0x7f3c48068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.647+0000 7f3c6cf48640 1 --2- 172.21.15.89:0/2394679578 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3c480664a0 0x7f3c48068960 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3c6815c980 tx=0x7f3c50009040 comp rx=0 tx=0).stop 2023-10-30T12:48:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.647+0000 7f3c6cf48640 1 -- 172.21.15.89:0/2394679578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c54004d10 msgr2=0x7f3c68147890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.647+0000 7f3c6cf48640 1 --2- 172.21.15.89:0/2394679578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c54004d10 0x7f3c68147890 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f3c5800e580 tx=0x7f3c5800ea40 comp rx=0 tx=0).stop 2023-10-30T12:48:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.651+0000 7f3c6cf48640 1 -- 172.21.15.89:0/2394679578 shutdown_connections 2023-10-30T12:48:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.651+0000 7f3c6cf48640 1 --2- 172.21.15.89:0/2394679578 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3c480664a0 0x7f3c48068960 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:26.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.651+0000 7f3c6cf48640 1 --2- 172.21.15.89:0/2394679578 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3c6815c230 0x7f3c681efa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:26.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.651+0000 7f3c6cf48640 1 --2- 172.21.15.89:0/2394679578 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3c680b33e0 0x7f3c680b3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:26.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.651+0000 7f3c6cf48640 1 --2- 172.21.15.89:0/2394679578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c54004d10 0x7f3c68147890 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:26.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.651+0000 7f3c6cf48640 1 -- 172.21.15.89:0/2394679578 >> 172.21.15.89:0/2394679578 conn(0x7f3c6806a640 msgr2=0x7f3c68149060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:26.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.651+0000 7f3c6cf48640 1 -- 172.21.15.89:0/2394679578 shutdown_connections 2023-10-30T12:48:26.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:26.651+0000 7f3c6cf48640 1 -- 172.21.15.89:0/2394679578 wait complete. 2023-10-30T12:48:26.665 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:48:28.667 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:28.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d86c17640 1 Processor -- start 2023-10-30T12:48:28.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d86c17640 1 -- start start 2023-10-30T12:48:28.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d86c17640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6d8015d6f0 0x7f6d8015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:28.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d86c17640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d800cb800 con 0x7f6d800cf160 2023-10-30T12:48:28.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d86c17640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d800cb330 con 0x7f6d801538a0 2023-10-30T12:48:28.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d86c17640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d800cad70 con 0x7f6d8015d6f0 2023-10-30T12:48:28.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d851b4640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6d800cf160 0x7f6d80164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:45998/0 (socket says 172.21.15.89:45998) 2023-10-30T12:48:28.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d851b4640 1 -- 172.21.15.89:0/2393569090 learned_addr learned my addr 172.21.15.89:0/2393569090 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:48:28.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d7ffff640 1 --2- 172.21.15.89:0/2393569090 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6d8015d6f0 0x7f6d8015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:28.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d7f7fe640 1 -- 172.21.15.89:0/2393569090 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 543330782 0 0) 0x7f6d800cb800 con 0x7f6d800cf160 2023-10-30T12:48:28.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d7f7fe640 1 -- 172.21.15.89:0/2393569090 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d70003530 con 0x7f6d800cf160 2023-10-30T12:48:28.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d7f7fe640 1 -- 172.21.15.89:0/2393569090 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4178530756 0 0) 0x7f6d70003530 con 0x7f6d800cf160 2023-10-30T12:48:28.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d7f7fe640 1 -- 172.21.15.89:0/2393569090 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6d8015d6f0 msgr2=0x7f6d8015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:28.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d7f7fe640 1 --2- 172.21.15.89:0/2393569090 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6d8015d6f0 0x7f6d8015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:28.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d7f7fe640 1 -- 172.21.15.89:0/2393569090 >> v1:172.21.15.192:6789/0 conn(0x7f6d801538a0 legacy=0x7f6d800ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:28.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d7f7fe640 1 -- 172.21.15.89:0/2393569090 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d800cb050 con 0x7f6d800cf160 2023-10-30T12:48:28.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d7f7fe640 1 -- 172.21.15.89:0/2393569090 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f6d78001a00 con 0x7f6d800cf160 2023-10-30T12:48:28.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d7ffff640 1 --2- 172.21.15.89:0/2393569090 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6d8015d6f0 0x7f6d8015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:48:28.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d7f7fe640 1 -- 172.21.15.89:0/2393569090 >> v1:172.21.15.89:6789/0 conn(0x7f6d800cf160 legacy=0x7f6d80164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:28.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d7f7fe640 1 --2- 172.21.15.89:0/2393569090 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6d8015d6f0 0x7f6d700039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:28.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d7f7fe640 1 --2- 172.21.15.89:0/2393569090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d70003ef0 0x7f6d700047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:28.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d7f7fe640 1 --2- 172.21.15.89:0/2393569090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d70004d10 0x7f6d70005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:28.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d7f7fe640 1 -- 172.21.15.89:0/2393569090 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d800cad70 con 0x7f6d70004d10 2023-10-30T12:48:28.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d7f7fe640 1 -- 172.21.15.89:0/2393569090 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7000b9f0 con 0x7f6d70003ef0 2023-10-30T12:48:28.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d7f7fe640 1 -- 172.21.15.89:0/2393569090 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7000bbb0 con 0x7f6d8015d6f0 2023-10-30T12:48:28.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d851b4640 1 --2- 172.21.15.89:0/2393569090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d70004d10 0x7f6d70005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:28.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d849b3640 1 --2- 172.21.15.89:0/2393569090 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6d8015d6f0 0x7f6d700039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:28.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d7ffff640 1 --2- 172.21.15.89:0/2393569090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d70003ef0 0x7f6d700047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:28.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d851b4640 1 -- 172.21.15.89:0/2393569090 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6d8015d6f0 msgr2=0x7f6d700039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:28.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d851b4640 1 --2- 172.21.15.89:0/2393569090 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6d8015d6f0 0x7f6d700039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:28.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d851b4640 1 -- 172.21.15.89:0/2393569090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d70003ef0 msgr2=0x7f6d700047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:28.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d851b4640 1 --2- 172.21.15.89:0/2393569090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d70003ef0 0x7f6d700047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:28.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.819+0000 7f6d851b4640 1 -- 172.21.15.89:0/2393569090 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d6c009420 con 0x7f6d70004d10 2023-10-30T12:48:28.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.823+0000 7f6d851b4640 1 --2- 172.21.15.89:0/2393569090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d70004d10 0x7f6d70005190 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f6d7800a830 tx=0x7f6d7801d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f8e4f7651cefa8d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:28.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.823+0000 7f6d7f7fe640 1 -- 172.21.15.89:0/2393569090 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d780ad630 con 0x7f6d70004d10 2023-10-30T12:48:28.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.823+0000 7f6d7f7fe640 1 -- 172.21.15.89:0/2393569090 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d780a8070 con 0x7f6d70004d10 2023-10-30T12:48:28.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.823+0000 7f6d7f7fe640 1 -- 172.21.15.89:0/2393569090 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d780ada30 con 0x7f6d70004d10 2023-10-30T12:48:28.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.823+0000 7f6d86c17640 1 -- 172.21.15.89:0/2393569090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d70004d10 msgr2=0x7f6d70005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:28.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.823+0000 7f6d86c17640 1 --2- 172.21.15.89:0/2393569090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d70004d10 0x7f6d70005190 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f6d7800a830 tx=0x7f6d7801d340 comp rx=0 tx=0).stop 2023-10-30T12:48:28.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.823+0000 7f6d851b4640 1 -- 172.21.15.89:0/2393569090 reap_dead start 2023-10-30T12:48:28.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.823+0000 7f6d86c17640 1 -- 172.21.15.89:0/2393569090 shutdown_connections 2023-10-30T12:48:28.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.823+0000 7f6d86c17640 1 -- 172.21.15.89:0/2393569090 >> 172.21.15.89:0/2393569090 conn(0x7f6d8006bfe0 msgr2=0x7f6d8006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:28.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.823+0000 7f6d86c17640 1 -- 172.21.15.89:0/2393569090 shutdown_connections 2023-10-30T12:48:28.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.823+0000 7f6d86c17640 1 -- 172.21.15.89:0/2393569090 wait complete. 2023-10-30T12:48:28.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.827+0000 7f6d86c17640 1 Processor -- start 2023-10-30T12:48:28.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.827+0000 7f6d86c17640 1 -- start start 2023-10-30T12:48:28.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.827+0000 7f6d86c17640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d70004d10 0x7f6d80159080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.827+0000 7f6d86c17640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6d800b4640 0x7f6d80143dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.827+0000 7f6d86c17640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d8014edf0 0x7f6d80148e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.827+0000 7f6d86c17640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d800b51c0 con 0x7f6d8014edf0 2023-10-30T12:48:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.827+0000 7f6d86c17640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d801595c0 con 0x7f6d70004d10 2023-10-30T12:48:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.827+0000 7f6d86c17640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d80150700 con 0x7f6d800b4640 2023-10-30T12:48:28.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.827+0000 7f6d851b4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d8014edf0 0x7f6d80148e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:28.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.827+0000 7f6d851b4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d8014edf0 0x7f6d80148e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34994/0 (socket says 172.21.15.89:34994) 2023-10-30T12:48:28.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.827+0000 7f6d851b4640 1 -- 172.21.15.89:0/239321272 learned_addr learned my addr 172.21.15.89:0/239321272 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:28.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.827+0000 7f6d849b3640 1 --2- 172.21.15.89:0/239321272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d70004d10 0x7f6d80159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:28.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.827+0000 7f6d7ffff640 1 --2- 172.21.15.89:0/239321272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6d800b4640 0x7f6d80143dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:28.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.827+0000 7f6d851b4640 1 -- 172.21.15.89:0/239321272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6d800b4640 msgr2=0x7f6d80143dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:28.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.827+0000 7f6d851b4640 1 --2- 172.21.15.89:0/239321272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6d800b4640 0x7f6d80143dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:28.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.827+0000 7f6d851b4640 1 -- 172.21.15.89:0/239321272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d70004d10 msgr2=0x7f6d80159080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:28.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.827+0000 7f6d851b4640 1 --2- 172.21.15.89:0/239321272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d70004d10 0x7f6d80159080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:28.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.827+0000 7f6d851b4640 1 -- 172.21.15.89:0/239321272 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d801509a0 con 0x7f6d8014edf0 2023-10-30T12:48:28.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.827+0000 7f6d7ffff640 1 --2- 172.21.15.89:0/239321272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6d800b4640 0x7f6d80143dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:28.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.827+0000 7f6d849b3640 1 --2- 172.21.15.89:0/239321272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d70004d10 0x7f6d80159080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:28.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.827+0000 7f6d851b4640 1 --2- 172.21.15.89:0/239321272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d8014edf0 0x7f6d80148e70 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f6d7800a830 tx=0x7f6d70005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:28.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.827+0000 7f6d7d7fa640 1 -- 172.21.15.89:0/239321272 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d78001a00 con 0x7f6d8014edf0 2023-10-30T12:48:28.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.827+0000 7f6d7d7fa640 1 -- 172.21.15.89:0/239321272 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d78004b70 con 0x7f6d8014edf0 2023-10-30T12:48:28.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.827+0000 7f6d7d7fa640 1 -- 172.21.15.89:0/239321272 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d780ad430 con 0x7f6d8014edf0 2023-10-30T12:48:28.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.827+0000 7f6d86c17640 1 -- 172.21.15.89:0/239321272 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d8014dcf0 con 0x7f6d8014edf0 2023-10-30T12:48:28.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.827+0000 7f6d86c17640 1 -- 172.21.15.89:0/239321272 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d800cedd0 con 0x7f6d8014edf0 2023-10-30T12:48:28.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.831+0000 7f6d86c17640 1 -- 172.21.15.89:0/239321272 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d8014db50 con 0x7f6d8014edf0 2023-10-30T12:48:28.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.831+0000 7f6d7d7fa640 1 -- 172.21.15.89:0/239321272 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6d780aa030 con 0x7f6d8014edf0 2023-10-30T12:48:28.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.831+0000 7f6d7d7fa640 1 --2- 172.21.15.89:0/239321272 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6d640664a0 0x7f6d64068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:28.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.831+0000 7f6d7d7fa640 1 -- 172.21.15.89:0/239321272 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6578+0+0 (secure 0 0 0) 0x7f6d7811e640 con 0x7f6d8014edf0 2023-10-30T12:48:28.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.831+0000 7f6d849b3640 1 --2- 172.21.15.89:0/239321272 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6d640664a0 0x7f6d64068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:28.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.831+0000 7f6d849b3640 1 --2- 172.21.15.89:0/239321272 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6d640664a0 0x7f6d64068960 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f6d74007fd0 tx=0x7f6d74009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:28.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:28.835+0000 7f6d7d7fa640 1 -- 172.21.15.89:0/239321272 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6d8014db50 con 0x7f6d8014edf0 2023-10-30T12:48:29.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:29.015+0000 7f6d86c17640 1 -- 172.21.15.89:0/239321272 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d8014db50 con 0x7f6d8014edf0 2023-10-30T12:48:29.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:29.015+0000 7f6d7d7fa640 1 -- 172.21.15.89:0/239321272 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+5747 (secure 0 0 0) 0x7f6d8014db50 con 0x7f6d8014edf0 2023-10-30T12:48:29.015 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:48:29.015 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:48:24.490193+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:reconnect","state_seq":34,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:29.016 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 94 2023-10-30T12:48:29.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:29.019+0000 7f6d86c17640 1 -- 172.21.15.89:0/239321272 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6d640664a0 msgr2=0x7f6d64068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:29.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:29.019+0000 7f6d86c17640 1 --2- 172.21.15.89:0/239321272 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6d640664a0 0x7f6d64068960 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f6d74007fd0 tx=0x7f6d74009040 comp rx=0 tx=0).stop 2023-10-30T12:48:29.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:29.019+0000 7f6d86c17640 1 -- 172.21.15.89:0/239321272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d8014edf0 msgr2=0x7f6d80148e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:29.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:29.019+0000 7f6d86c17640 1 --2- 172.21.15.89:0/239321272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d8014edf0 0x7f6d80148e70 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f6d7800a830 tx=0x7f6d70005800 comp rx=0 tx=0).stop 2023-10-30T12:48:29.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:29.019+0000 7f6d86c17640 1 -- 172.21.15.89:0/239321272 shutdown_connections 2023-10-30T12:48:29.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:29.019+0000 7f6d86c17640 1 --2- 172.21.15.89:0/239321272 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f6d640664a0 0x7f6d64068960 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:29.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:29.019+0000 7f6d86c17640 1 --2- 172.21.15.89:0/239321272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d8014edf0 0x7f6d80148e70 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:29.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:29.019+0000 7f6d86c17640 1 --2- 172.21.15.89:0/239321272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6d800b4640 0x7f6d80143dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:29.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:29.019+0000 7f6d86c17640 1 --2- 172.21.15.89:0/239321272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d70004d10 0x7f6d80159080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:29.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:29.019+0000 7f6d86c17640 1 -- 172.21.15.89:0/239321272 >> 172.21.15.89:0/239321272 conn(0x7f6d8006bfe0 msgr2=0x7f6d80151c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:29.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:29.019+0000 7f6d86c17640 1 -- 172.21.15.89:0/239321272 shutdown_connections 2023-10-30T12:48:29.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:29.019+0000 7f6d86c17640 1 -- 172.21.15.89:0/239321272 wait complete. 2023-10-30T12:48:29.036 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:48:31.036 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:31.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.187+0000 7f2f6313e640 1 Processor -- start 2023-10-30T12:48:31.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.187+0000 7f2f6313e640 1 -- start start 2023-10-30T12:48:31.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.191+0000 7f2f6313e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f5c06f7a0 0x7f2f5c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:31.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.191+0000 7f2f6313e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2f5c0c6bd0 0x7f2f5c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:31.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.191+0000 7f2f6313e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5c06a3d0 con 0x7f2f5c071110 2023-10-30T12:48:31.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.191+0000 7f2f6313e640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f5c0700c0 con 0x7f2f5c0c6bd0 2023-10-30T12:48:31.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.191+0000 7f2f6313e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5c066cf0 con 0x7f2f5c06f7a0 2023-10-30T12:48:31.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.191+0000 7f2f60eda640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f5c06f7a0 0x7f2f5c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:31.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.191+0000 7f2f5bfff640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2f5c0c6bd0 0x7f2f5c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:31.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.191+0000 7f2f5bfff640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2f5c0c6bd0 0x7f2f5c069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:37746/0 (socket says 172.21.15.89:37746) 2023-10-30T12:48:31.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.191+0000 7f2f5bfff640 1 -- 172.21.15.89:0/1095775842 learned_addr learned my addr 172.21.15.89:0/1095775842 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:31.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.191+0000 7f2f5bfff640 1 -- 172.21.15.89:0/1095775842 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f5c06f7a0 msgr2=0x7f2f5c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:31.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.191+0000 7f2f5bfff640 1 --2- 172.21.15.89:0/1095775842 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f5c06f7a0 0x7f2f5c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:31.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.191+0000 7f2f5bfff640 1 -- 172.21.15.89:0/1095775842 >> v1:172.21.15.192:6789/0 conn(0x7f2f5c071110 legacy=0x7f2f5c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:31.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.191+0000 7f2f5bfff640 1 -- 172.21.15.89:0/1095775842 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5c0c70d0 con 0x7f2f5c0c6bd0 2023-10-30T12:48:31.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.191+0000 7f2f60eda640 1 --2- 172.21.15.89:0/1095775842 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f5c06f7a0 0x7f2f5c06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:48:31.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.195+0000 7f2f5bfff640 1 --2- 172.21.15.89:0/1095775842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f5c0c6bd0 0x7f2f5c069e90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2f48009870 tx=0x7f2f4801ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dac2926dcd300eae server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:31.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.195+0000 7f2f5b7fe640 1 -- 172.21.15.89:0/1095775842 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f480a8a30 con 0x7f2f5c0c6bd0 2023-10-30T12:48:31.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.195+0000 7f2f5b7fe640 1 -- 172.21.15.89:0/1095775842 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f480a8b90 con 0x7f2f5c0c6bd0 2023-10-30T12:48:31.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.195+0000 7f2f5b7fe640 1 -- 172.21.15.89:0/1095775842 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f480b0440 con 0x7f2f5c0c6bd0 2023-10-30T12:48:31.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.195+0000 7f2f6313e640 1 -- 172.21.15.89:0/1095775842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f5c0c6bd0 msgr2=0x7f2f5c069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:31.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.195+0000 7f2f6313e640 1 --2- 172.21.15.89:0/1095775842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f5c0c6bd0 0x7f2f5c069e90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2f48009870 tx=0x7f2f4801ed30 comp rx=0 tx=0).stop 2023-10-30T12:48:31.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.195+0000 7f2f6313e640 1 -- 172.21.15.89:0/1095775842 shutdown_connections 2023-10-30T12:48:31.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.195+0000 7f2f6313e640 1 --2- 172.21.15.89:0/1095775842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f5c0c6bd0 0x7f2f5c069e90 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:31.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.195+0000 7f2f6313e640 1 --2- 172.21.15.89:0/1095775842 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f5c06f7a0 0x7f2f5c06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:31.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.195+0000 7f2f6313e640 1 -- 172.21.15.89:0/1095775842 >> 172.21.15.89:0/1095775842 conn(0x7f2f5c0c7910 msgr2=0x7f2f5c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:31.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.195+0000 7f2f6313e640 1 -- 172.21.15.89:0/1095775842 shutdown_connections 2023-10-30T12:48:31.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.195+0000 7f2f6313e640 1 -- 172.21.15.89:0/1095775842 wait complete. 2023-10-30T12:48:31.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.195+0000 7f2f6313e640 1 Processor -- start 2023-10-30T12:48:31.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.195+0000 7f2f6313e640 1 -- start start 2023-10-30T12:48:31.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.199+0000 7f2f6313e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f5c06f7a0 0x7f2f5c0c2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:31.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.199+0000 7f2f6313e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f5c071110 0x7f2f5c0ca660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:31.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.199+0000 7f2f6313e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f5c0c6bd0 0x7f2f5c145900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:31.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.199+0000 7f2f6313e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5c146470 con 0x7f2f5c06f7a0 2023-10-30T12:48:31.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.199+0000 7f2f6313e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5c06a5f0 con 0x7f2f5c0c6bd0 2023-10-30T12:48:31.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.199+0000 7f2f6313e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5c0c41d0 con 0x7f2f5c071110 2023-10-30T12:48:31.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.199+0000 7f2f60eda640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f5c06f7a0 0x7f2f5c0c2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:31.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.199+0000 7f2f60eda640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f5c06f7a0 0x7f2f5c0c2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44556/0 (socket says 172.21.15.89:44556) 2023-10-30T12:48:31.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.199+0000 7f2f60eda640 1 -- 172.21.15.89:0/3418559036 learned_addr learned my addr 172.21.15.89:0/3418559036 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:31.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.199+0000 7f2f5bfff640 1 --2- 172.21.15.89:0/3418559036 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f5c071110 0x7f2f5c0ca660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:31.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.199+0000 7f2f616db640 1 --2- 172.21.15.89:0/3418559036 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f5c0c6bd0 0x7f2f5c145900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:31.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.199+0000 7f2f60eda640 1 -- 172.21.15.89:0/3418559036 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f5c071110 msgr2=0x7f2f5c0ca660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:31.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.199+0000 7f2f60eda640 1 --2- 172.21.15.89:0/3418559036 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f5c071110 0x7f2f5c0ca660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:31.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.199+0000 7f2f60eda640 1 -- 172.21.15.89:0/3418559036 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f5c0c6bd0 msgr2=0x7f2f5c145900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:31.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.199+0000 7f2f60eda640 1 --2- 172.21.15.89:0/3418559036 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f5c0c6bd0 0x7f2f5c145900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:31.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.199+0000 7f2f60eda640 1 -- 172.21.15.89:0/3418559036 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f50009420 con 0x7f2f5c06f7a0 2023-10-30T12:48:31.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.199+0000 7f2f616db640 1 --2- 172.21.15.89:0/3418559036 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f5c0c6bd0 0x7f2f5c145900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:31.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.199+0000 7f2f60eda640 1 --2- 172.21.15.89:0/3418559036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f5c06f7a0 0x7f2f5c0c2640 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f2f5000e980 tx=0x7f2f5000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:31.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.199+0000 7f2f597fa640 1 -- 172.21.15.89:0/3418559036 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5000ca00 con 0x7f2f5c06f7a0 2023-10-30T12:48:31.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.199+0000 7f2f597fa640 1 -- 172.21.15.89:0/3418559036 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f5000cb60 con 0x7f2f5c06f7a0 2023-10-30T12:48:31.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.199+0000 7f2f597fa640 1 -- 172.21.15.89:0/3418559036 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f50013470 con 0x7f2f5c06f7a0 2023-10-30T12:48:31.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.199+0000 7f2f6313e640 1 -- 172.21.15.89:0/3418559036 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f480094d0 con 0x7f2f5c06f7a0 2023-10-30T12:48:31.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.199+0000 7f2f6313e640 1 -- 172.21.15.89:0/3418559036 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f5c1edae0 con 0x7f2f5c06f7a0 2023-10-30T12:48:31.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.199+0000 7f2f6313e640 1 -- 172.21.15.89:0/3418559036 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f5c04e170 con 0x7f2f5c06f7a0 2023-10-30T12:48:31.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.203+0000 7f2f597fa640 1 -- 172.21.15.89:0/3418559036 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2f500135d0 con 0x7f2f5c06f7a0 2023-10-30T12:48:31.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.203+0000 7f2f597fa640 1 --2- 172.21.15.89:0/3418559036 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2f400664a0 0x7f2f40068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:31.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.203+0000 7f2f597fa640 1 -- 172.21.15.89:0/3418559036 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6578+0+0 (secure 0 0 0) 0x7f2f50084450 con 0x7f2f5c06f7a0 2023-10-30T12:48:31.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.203+0000 7f2f5bfff640 1 --2- 172.21.15.89:0/3418559036 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2f400664a0 0x7f2f40068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:31.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.203+0000 7f2f5bfff640 1 --2- 172.21.15.89:0/3418559036 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2f400664a0 0x7f2f40068960 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2f480b2000 tx=0x7f2f48009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:31.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.207+0000 7f2f597fa640 1 -- 172.21.15.89:0/3418559036 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2f50051570 con 0x7f2f5c06f7a0 2023-10-30T12:48:31.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.391+0000 7f2f6313e640 1 -- 172.21.15.89:0/3418559036 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f5c0c8400 con 0x7f2f5c06f7a0 2023-10-30T12:48:31.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.391+0000 7f2f597fa640 1 -- 172.21.15.89:0/3418559036 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5744 (secure 0 0 0) 0x7f2f50054b90 con 0x7f2f5c06f7a0 2023-10-30T12:48:31.388 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:48:31.388 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:48:30.428940+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:rejoin","state_seq":36,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:31.389 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 95 2023-10-30T12:48:31.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.391+0000 7f2f6313e640 1 -- 172.21.15.89:0/3418559036 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2f400664a0 msgr2=0x7f2f40068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:31.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.391+0000 7f2f6313e640 1 --2- 172.21.15.89:0/3418559036 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2f400664a0 0x7f2f40068960 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2f480b2000 tx=0x7f2f48009040 comp rx=0 tx=0).stop 2023-10-30T12:48:31.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.391+0000 7f2f6313e640 1 -- 172.21.15.89:0/3418559036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f5c06f7a0 msgr2=0x7f2f5c0c2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:31.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.391+0000 7f2f6313e640 1 --2- 172.21.15.89:0/3418559036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f5c06f7a0 0x7f2f5c0c2640 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f2f5000e980 tx=0x7f2f5000ee40 comp rx=0 tx=0).stop 2023-10-30T12:48:31.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.391+0000 7f2f6313e640 1 -- 172.21.15.89:0/3418559036 shutdown_connections 2023-10-30T12:48:31.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.391+0000 7f2f6313e640 1 --2- 172.21.15.89:0/3418559036 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2f400664a0 0x7f2f40068960 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:31.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.391+0000 7f2f6313e640 1 --2- 172.21.15.89:0/3418559036 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f5c0c6bd0 0x7f2f5c145900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:31.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.391+0000 7f2f6313e640 1 --2- 172.21.15.89:0/3418559036 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2f5c071110 0x7f2f5c0ca660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:31.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.391+0000 7f2f6313e640 1 --2- 172.21.15.89:0/3418559036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f5c06f7a0 0x7f2f5c0c2640 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:31.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.391+0000 7f2f6313e640 1 -- 172.21.15.89:0/3418559036 >> 172.21.15.89:0/3418559036 conn(0x7f2f5c0c7910 msgr2=0x7f2f5c0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:31.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.391+0000 7f2f6313e640 1 -- 172.21.15.89:0/3418559036 shutdown_connections 2023-10-30T12:48:31.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:31.391+0000 7f2f6313e640 1 -- 172.21.15.89:0/3418559036 wait complete. 2023-10-30T12:48:31.409 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:48:33.410 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:33.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.559+0000 7fe804b3e640 1 Processor -- start 2023-10-30T12:48:33.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.559+0000 7fe804b3e640 1 -- start start 2023-10-30T12:48:33.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.563+0000 7fe804b3e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe80006f7a0 0x7fe80006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:33.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.563+0000 7fe804b3e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe800071110 0x7fe800158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:33.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.563+0000 7fe804b3e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe800066cf0 con 0x7fe8000c6bd0 2023-10-30T12:48:33.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.563+0000 7fe804b3e640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8000700c0 con 0x7fe800071110 2023-10-30T12:48:33.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.563+0000 7fe804b3e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8000c70d0 con 0x7fe80006f7a0 2023-10-30T12:48:33.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.563+0000 7fe7fed9d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe800071110 0x7fe800158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:33.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.563+0000 7fe7fed9d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe800071110 0x7fe800158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44572/0 (socket says 172.21.15.89:44572) 2023-10-30T12:48:33.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.563+0000 7fe7fed9d640 1 -- 172.21.15.89:0/666352537 learned_addr learned my addr 172.21.15.89:0/666352537 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:33.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.563+0000 7fe7fe59c640 1 --2- 172.21.15.89:0/666352537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe80006f7a0 0x7fe80006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:33.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.563+0000 7fe7fed9d640 1 -- 172.21.15.89:0/666352537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe80006f7a0 msgr2=0x7fe80006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:33.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.563+0000 7fe7fed9d640 1 --2- 172.21.15.89:0/666352537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe80006f7a0 0x7fe80006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:33.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.563+0000 7fe7fed9d640 1 -- 172.21.15.89:0/666352537 >> v1:172.21.15.192:6789/0 conn(0x7fe8000c6bd0 legacy=0x7fe800069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:33.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.563+0000 7fe7fed9d640 1 -- 172.21.15.89:0/666352537 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8000c7390 con 0x7fe800071110 2023-10-30T12:48:33.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.567+0000 7fe7fed9d640 1 --2- 172.21.15.89:0/666352537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe800071110 0x7fe800158110 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fe7f8009e50 tx=0x7fe7f801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cbd1870522b639f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:33.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.567+0000 7fe7fdd9b640 1 -- 172.21.15.89:0/666352537 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f80af400 con 0x7fe800071110 2023-10-30T12:48:33.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.567+0000 7fe7fdd9b640 1 -- 172.21.15.89:0/666352537 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f80ab050 con 0x7fe800071110 2023-10-30T12:48:33.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.567+0000 7fe7fdd9b640 1 -- 172.21.15.89:0/666352537 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f80af780 con 0x7fe800071110 2023-10-30T12:48:33.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.567+0000 7fe804b3e640 1 -- 172.21.15.89:0/666352537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe800071110 msgr2=0x7fe800158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:33.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.567+0000 7fe804b3e640 1 --2- 172.21.15.89:0/666352537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe800071110 0x7fe800158110 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fe7f8009e50 tx=0x7fe7f801f1f0 comp rx=0 tx=0).stop 2023-10-30T12:48:33.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.567+0000 7fe804b3e640 1 -- 172.21.15.89:0/666352537 shutdown_connections 2023-10-30T12:48:33.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.567+0000 7fe804b3e640 1 --2- 172.21.15.89:0/666352537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe800071110 0x7fe800158110 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:33.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.567+0000 7fe804b3e640 1 --2- 172.21.15.89:0/666352537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe80006f7a0 0x7fe80006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:33.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.567+0000 7fe804b3e640 1 -- 172.21.15.89:0/666352537 >> 172.21.15.89:0/666352537 conn(0x7fe8000c7910 msgr2=0x7fe8000c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:33.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.567+0000 7fe804b3e640 1 -- 172.21.15.89:0/666352537 shutdown_connections 2023-10-30T12:48:33.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.567+0000 7fe804b3e640 1 -- 172.21.15.89:0/666352537 wait complete. 2023-10-30T12:48:33.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.567+0000 7fe804b3e640 1 Processor -- start 2023-10-30T12:48:33.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.567+0000 7fe804b3e640 1 -- start start 2023-10-30T12:48:33.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.567+0000 7fe804b3e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe80006f7a0 0x7fe8001f1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:33.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.567+0000 7fe804b3e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe800071110 0x7fe800157700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:33.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.567+0000 7fe804b3e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8000c6bd0 0x7fe800157c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:33.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.567+0000 7fe804b3e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8001588a0 con 0x7fe80006f7a0 2023-10-30T12:48:33.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.567+0000 7fe804b3e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80006c680 con 0x7fe800071110 2023-10-30T12:48:33.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.567+0000 7fe804b3e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe80014fd40 con 0x7fe8000c6bd0 2023-10-30T12:48:33.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.571+0000 7fe7fe59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe80006f7a0 0x7fe8001f1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:33.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.571+0000 7fe7fe59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe80006f7a0 0x7fe8001f1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44582/0 (socket says 172.21.15.89:44582) 2023-10-30T12:48:33.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.571+0000 7fe7fe59c640 1 -- 172.21.15.89:0/2418336193 learned_addr learned my addr 172.21.15.89:0/2418336193 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:33.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.571+0000 7fe7f7fff640 1 --2- 172.21.15.89:0/2418336193 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe800071110 0x7fe800157700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:33.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.571+0000 7fe7fed9d640 1 --2- 172.21.15.89:0/2418336193 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8000c6bd0 0x7fe800157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:33.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.571+0000 7fe7fe59c640 1 -- 172.21.15.89:0/2418336193 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8000c6bd0 msgr2=0x7fe800157c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:33.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.571+0000 7fe7fe59c640 1 --2- 172.21.15.89:0/2418336193 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8000c6bd0 0x7fe800157c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:33.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.571+0000 7fe7fe59c640 1 -- 172.21.15.89:0/2418336193 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe800071110 msgr2=0x7fe800157700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:33.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.571+0000 7fe7fe59c640 1 --2- 172.21.15.89:0/2418336193 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe800071110 0x7fe800157700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:33.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.571+0000 7fe7fe59c640 1 -- 172.21.15.89:0/2418336193 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f8009b30 con 0x7fe80006f7a0 2023-10-30T12:48:33.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.571+0000 7fe7f7fff640 1 --2- 172.21.15.89:0/2418336193 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe800071110 0x7fe800157700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:33.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.571+0000 7fe7fed9d640 1 --2- 172.21.15.89:0/2418336193 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8000c6bd0 0x7fe800157c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:33.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.571+0000 7fe7fe59c640 1 --2- 172.21.15.89:0/2418336193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe80006f7a0 0x7fe8001f1a20 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fe7f000d5f0 tx=0x7fe7f000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:33.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.571+0000 7fe7f6ffd640 1 -- 172.21.15.89:0/2418336193 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f00125f0 con 0x7fe80006f7a0 2023-10-30T12:48:33.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.571+0000 7fe7f6ffd640 1 -- 172.21.15.89:0/2418336193 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f0012750 con 0x7fe80006f7a0 2023-10-30T12:48:33.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.571+0000 7fe804b3e640 1 -- 172.21.15.89:0/2418336193 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe800150020 con 0x7fe80006f7a0 2023-10-30T12:48:33.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.571+0000 7fe7f6ffd640 1 -- 172.21.15.89:0/2418336193 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f0012ae0 con 0x7fe80006f7a0 2023-10-30T12:48:33.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.571+0000 7fe804b3e640 1 -- 172.21.15.89:0/2418336193 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe800158af0 con 0x7fe80006f7a0 2023-10-30T12:48:33.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.571+0000 7fe804b3e640 1 -- 172.21.15.89:0/2418336193 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe80004e170 con 0x7fe80006f7a0 2023-10-30T12:48:33.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.571+0000 7fe7f6ffd640 1 -- 172.21.15.89:0/2418336193 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe7f0012c40 con 0x7fe80006f7a0 2023-10-30T12:48:33.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.575+0000 7fe7f6ffd640 1 --2- 172.21.15.89:0/2418336193 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe7e00664a0 0x7fe7e0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:33.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.575+0000 7fe7f6ffd640 1 -- 172.21.15.89:0/2418336193 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6578+0+0 (secure 0 0 0) 0x7fe7f0084740 con 0x7fe80006f7a0 2023-10-30T12:48:33.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.575+0000 7fe7f7fff640 1 --2- 172.21.15.89:0/2418336193 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe7e00664a0 0x7fe7e0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:33.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.575+0000 7fe7f7fff640 1 --2- 172.21.15.89:0/2418336193 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe7e00664a0 0x7fe7e0068960 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe8000c4560 tx=0x7fe7e8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:33.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.579+0000 7fe7f6ffd640 1 -- 172.21.15.89:0/2418336193 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe7f0051840 con 0x7fe80006f7a0 2023-10-30T12:48:33.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.759+0000 7fe804b3e640 1 -- 172.21.15.89:0/2418336193 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8001510d0 con 0x7fe80006f7a0 2023-10-30T12:48:33.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.759+0000 7fe7f6ffd640 1 -- 172.21.15.89:0/2418336193 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5744 (secure 0 0 0) 0x7fe7f0054e60 con 0x7fe80006f7a0 2023-10-30T12:48:33.757 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:48:33.757 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:48:30.428940+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:rejoin","state_seq":36,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:33.757 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 95 2023-10-30T12:48:33.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.759+0000 7fe804b3e640 1 -- 172.21.15.89:0/2418336193 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe7e00664a0 msgr2=0x7fe7e0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.759+0000 7fe804b3e640 1 --2- 172.21.15.89:0/2418336193 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe7e00664a0 0x7fe7e0068960 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe8000c4560 tx=0x7fe7e8009040 comp rx=0 tx=0).stop 2023-10-30T12:48:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.763+0000 7fe804b3e640 1 -- 172.21.15.89:0/2418336193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe80006f7a0 msgr2=0x7fe8001f1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.763+0000 7fe804b3e640 1 --2- 172.21.15.89:0/2418336193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe80006f7a0 0x7fe8001f1a20 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fe7f000d5f0 tx=0x7fe7f000dab0 comp rx=0 tx=0).stop 2023-10-30T12:48:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.763+0000 7fe804b3e640 1 -- 172.21.15.89:0/2418336193 shutdown_connections 2023-10-30T12:48:33.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.763+0000 7fe804b3e640 1 --2- 172.21.15.89:0/2418336193 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe7e00664a0 0x7fe7e0068960 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:33.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.763+0000 7fe804b3e640 1 --2- 172.21.15.89:0/2418336193 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe800071110 0x7fe800157700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:33.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.763+0000 7fe804b3e640 1 --2- 172.21.15.89:0/2418336193 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8000c6bd0 0x7fe800157c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:33.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.763+0000 7fe804b3e640 1 --2- 172.21.15.89:0/2418336193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe80006f7a0 0x7fe8001f1a20 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:33.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.763+0000 7fe804b3e640 1 -- 172.21.15.89:0/2418336193 >> 172.21.15.89:0/2418336193 conn(0x7fe8000c7910 msgr2=0x7fe8000ca670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:33.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.763+0000 7fe804b3e640 1 -- 172.21.15.89:0/2418336193 shutdown_connections 2023-10-30T12:48:33.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:33.763+0000 7fe804b3e640 1 -- 172.21.15.89:0/2418336193 wait complete. 2023-10-30T12:48:33.777 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:48:35.778 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:35.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.931+0000 7f9602691640 1 Processor -- start 2023-10-30T12:48:35.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.931+0000 7f9602691640 1 -- start start 2023-10-30T12:48:35.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.931+0000 7f9602691640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f95fc15d6f0 0x7f95fc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:35.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.931+0000 7f9602691640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f95fc0cf160 0x7f95fc0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:35.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.931+0000 7f9602691640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95fc0cb800 con 0x7f95fc1538a0 2023-10-30T12:48:35.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.931+0000 7f9602691640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95fc0cb330 con 0x7f95fc15d6f0 2023-10-30T12:48:35.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.931+0000 7f9602691640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc0cad70 con 0x7f95fc0cf160 2023-10-30T12:48:35.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.931+0000 7f95fb7fe640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f95fc15d6f0 0x7f95fc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:35.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.931+0000 7f95fb7fe640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f95fc15d6f0 0x7f95fc15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:37764/0 (socket says 172.21.15.89:37764) 2023-10-30T12:48:35.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.931+0000 7f95fb7fe640 1 -- 172.21.15.89:0/1094339004 learned_addr learned my addr 172.21.15.89:0/1094339004 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:35.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.931+0000 7f9600c2e640 1 --2- 172.21.15.89:0/1094339004 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f95fc0cf160 0x7f95fc0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:35.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.931+0000 7f95fb7fe640 1 -- 172.21.15.89:0/1094339004 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f95fc0cf160 msgr2=0x7f95fc0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:35.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.931+0000 7f95fb7fe640 1 --2- 172.21.15.89:0/1094339004 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f95fc0cf160 0x7f95fc0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:35.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.931+0000 7f95fb7fe640 1 -- 172.21.15.89:0/1094339004 >> v1:172.21.15.192:6789/0 conn(0x7f95fc1538a0 legacy=0x7f95fc0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:35.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.931+0000 7f95fb7fe640 1 -- 172.21.15.89:0/1094339004 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95fc0cb050 con 0x7f95fc15d6f0 2023-10-30T12:48:35.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.935+0000 7f95fb7fe640 1 --2- 172.21.15.89:0/1094339004 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95fc15d6f0 0x7f95fc15a090 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f95f0009870 tx=0x7f95f001ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d27e8cb39a97ad3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:35.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.935+0000 7f95faffd640 1 -- 172.21.15.89:0/1094339004 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f00a8a30 con 0x7f95fc15d6f0 2023-10-30T12:48:35.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.935+0000 7f95faffd640 1 -- 172.21.15.89:0/1094339004 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95f00a8b90 con 0x7f95fc15d6f0 2023-10-30T12:48:35.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.935+0000 7f95faffd640 1 -- 172.21.15.89:0/1094339004 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f00b0440 con 0x7f95fc15d6f0 2023-10-30T12:48:35.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.935+0000 7f9602691640 1 -- 172.21.15.89:0/1094339004 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95fc15d6f0 msgr2=0x7f95fc15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:35.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.935+0000 7f9602691640 1 --2- 172.21.15.89:0/1094339004 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95fc15d6f0 0x7f95fc15a090 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f95f0009870 tx=0x7f95f001ed30 comp rx=0 tx=0).stop 2023-10-30T12:48:35.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.935+0000 7f9602691640 1 -- 172.21.15.89:0/1094339004 shutdown_connections 2023-10-30T12:48:35.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.935+0000 7f9602691640 1 --2- 172.21.15.89:0/1094339004 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f95fc0cf160 0x7f95fc0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:35.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.935+0000 7f9602691640 1 --2- 172.21.15.89:0/1094339004 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95fc15d6f0 0x7f95fc15a090 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:35.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.935+0000 7f9602691640 1 -- 172.21.15.89:0/1094339004 >> 172.21.15.89:0/1094339004 conn(0x7f95fc06bfe0 msgr2=0x7f95fc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:35.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.935+0000 7f9602691640 1 -- 172.21.15.89:0/1094339004 shutdown_connections 2023-10-30T12:48:35.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.935+0000 7f9602691640 1 -- 172.21.15.89:0/1094339004 wait complete. 2023-10-30T12:48:35.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.939+0000 7f9602691640 1 Processor -- start 2023-10-30T12:48:35.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.939+0000 7f9602691640 1 -- start start 2023-10-30T12:48:35.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.939+0000 7f9602691640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95fc0cf160 0x7f95fc1e34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:35.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.939+0000 7f9602691640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f95fc1538a0 0x7f95fc14ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:35.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.939+0000 7f9602691640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95fc15d6f0 0x7f95fc1ef870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:35.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.939+0000 7f9602691640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc1e3a10 con 0x7f95fc0cf160 2023-10-30T12:48:35.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.939+0000 7f9602691640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc1596f0 con 0x7f95fc15d6f0 2023-10-30T12:48:35.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.939+0000 7f9602691640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc157a90 con 0x7f95fc1538a0 2023-10-30T12:48:35.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.939+0000 7f95fbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95fc0cf160 0x7f95fc1e34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:35.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.939+0000 7f95fbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95fc0cf160 0x7f95fc1e34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44590/0 (socket says 172.21.15.89:44590) 2023-10-30T12:48:35.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.939+0000 7f95fbfff640 1 -- 172.21.15.89:0/2570110817 learned_addr learned my addr 172.21.15.89:0/2570110817 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:35.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.939+0000 7f9600c2e640 1 --2- 172.21.15.89:0/2570110817 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95fc15d6f0 0x7f95fc1ef870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:35.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.939+0000 7f95fb7fe640 1 --2- 172.21.15.89:0/2570110817 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f95fc1538a0 0x7f95fc14ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:35.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.939+0000 7f95fbfff640 1 -- 172.21.15.89:0/2570110817 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f95fc1538a0 msgr2=0x7f95fc14ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:35.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.939+0000 7f95fbfff640 1 --2- 172.21.15.89:0/2570110817 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f95fc1538a0 0x7f95fc14ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:35.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.939+0000 7f95fbfff640 1 -- 172.21.15.89:0/2570110817 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95fc15d6f0 msgr2=0x7f95fc1ef870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:35.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.939+0000 7f95fbfff640 1 --2- 172.21.15.89:0/2570110817 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95fc15d6f0 0x7f95fc1ef870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:35.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.939+0000 7f95fbfff640 1 -- 172.21.15.89:0/2570110817 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f00094d0 con 0x7f95fc0cf160 2023-10-30T12:48:35.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.939+0000 7f9600c2e640 1 --2- 172.21.15.89:0/2570110817 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95fc15d6f0 0x7f95fc1ef870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:35.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.939+0000 7f95fb7fe640 1 --2- 172.21.15.89:0/2570110817 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f95fc1538a0 0x7f95fc14ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:35.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.939+0000 7f95fbfff640 1 --2- 172.21.15.89:0/2570110817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95fc0cf160 0x7f95fc1e34d0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f95ec00c450 tx=0x7f95ec00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:35.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.939+0000 7f95f8ff9640 1 -- 172.21.15.89:0/2570110817 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ec0052d0 con 0x7f95fc0cf160 2023-10-30T12:48:35.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.939+0000 7f95f8ff9640 1 -- 172.21.15.89:0/2570110817 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95ec005430 con 0x7f95fc0cf160 2023-10-30T12:48:35.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.939+0000 7f9602691640 1 -- 172.21.15.89:0/2570110817 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95fc1ef090 con 0x7f95fc0cf160 2023-10-30T12:48:35.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.939+0000 7f95f8ff9640 1 -- 172.21.15.89:0/2570110817 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ec007260 con 0x7f95fc0cf160 2023-10-30T12:48:35.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.943+0000 7f9602691640 1 -- 172.21.15.89:0/2570110817 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95fc157d30 con 0x7f95fc0cf160 2023-10-30T12:48:35.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.943+0000 7f9602691640 1 -- 172.21.15.89:0/2570110817 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95fc0b2a50 con 0x7f95fc0cf160 2023-10-30T12:48:35.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.943+0000 7f95f8ff9640 1 -- 172.21.15.89:0/2570110817 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f95ec0073c0 con 0x7f95fc0cf160 2023-10-30T12:48:35.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.943+0000 7f95f8ff9640 1 --2- 172.21.15.89:0/2570110817 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f95e80664a0 0x7f95e8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:35.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.943+0000 7f95f8ff9640 1 -- 172.21.15.89:0/2570110817 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6578+0+0 (secure 0 0 0) 0x7f95ec0836c0 con 0x7f95fc0cf160 2023-10-30T12:48:35.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.943+0000 7f95fb7fe640 1 --2- 172.21.15.89:0/2570110817 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f95e80664a0 0x7f95e8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:35.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.947+0000 7f95fb7fe640 1 --2- 172.21.15.89:0/2570110817 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f95e80664a0 0x7f95e8068960 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f95f00b2000 tx=0x7f95f00aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:35.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:35.947+0000 7f95f8ff9640 1 -- 172.21.15.89:0/2570110817 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f95ec0506b0 con 0x7f95fc0cf160 2023-10-30T12:48:36.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:36.127+0000 7f9602691640 1 -- 172.21.15.89:0/2570110817 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95fc159830 con 0x7f95fc0cf160 2023-10-30T12:48:36.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:36.127+0000 7f95f8ff9640 1 -- 172.21.15.89:0/2570110817 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5744 (secure 0 0 0) 0x7f95ec053cd0 con 0x7f95fc0cf160 2023-10-30T12:48:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:48:36.125 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:48:30.428940+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:rejoin","state_seq":36,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:36.125 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 95 2023-10-30T12:48:36.126 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:36.127+0000 7f9602691640 1 -- 172.21.15.89:0/2570110817 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f95e80664a0 msgr2=0x7f95e8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:36.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:36.127+0000 7f9602691640 1 --2- 172.21.15.89:0/2570110817 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f95e80664a0 0x7f95e8068960 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f95f00b2000 tx=0x7f95f00aa070 comp rx=0 tx=0).stop 2023-10-30T12:48:36.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:36.131+0000 7f9602691640 1 -- 172.21.15.89:0/2570110817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95fc0cf160 msgr2=0x7f95fc1e34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:36.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:36.131+0000 7f9602691640 1 --2- 172.21.15.89:0/2570110817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95fc0cf160 0x7f95fc1e34d0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f95ec00c450 tx=0x7f95ec00c910 comp rx=0 tx=0).stop 2023-10-30T12:48:36.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:36.131+0000 7f9602691640 1 -- 172.21.15.89:0/2570110817 shutdown_connections 2023-10-30T12:48:36.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:36.131+0000 7f9602691640 1 --2- 172.21.15.89:0/2570110817 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f95e80664a0 0x7f95e8068960 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:36.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:36.131+0000 7f9602691640 1 --2- 172.21.15.89:0/2570110817 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95fc15d6f0 0x7f95fc1ef870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:36.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:36.131+0000 7f9602691640 1 --2- 172.21.15.89:0/2570110817 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f95fc1538a0 0x7f95fc14ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:36.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:36.131+0000 7f9602691640 1 --2- 172.21.15.89:0/2570110817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95fc0cf160 0x7f95fc1e34d0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:36.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:36.131+0000 7f9602691640 1 -- 172.21.15.89:0/2570110817 >> 172.21.15.89:0/2570110817 conn(0x7f95fc06bfe0 msgr2=0x7f95fc0cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:36.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:36.131+0000 7f9602691640 1 -- 172.21.15.89:0/2570110817 shutdown_connections 2023-10-30T12:48:36.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:36.131+0000 7f9602691640 1 -- 172.21.15.89:0/2570110817 wait complete. 2023-10-30T12:48:36.146 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 95, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:50.729245+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 95, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5575": { "addr": "172.21.15.192:6835/501291700", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6834", "nonce": 501291700, "type": "v2" }, { "addr": "172.21.15.192:6835", "nonce": 501291700, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5575, "incarnation": 63, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 37 }, "gid_5728": { "addr": "172.21.15.192:6839/3991891597", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6838", "nonce": 3991891597, "type": "v2" }, { "addr": "172.21.15.192:6839", "nonce": 3991891597, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 5728, "incarnation": 73, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 19 }, "gid_6075": { "addr": "172.21.15.89:6837/2412618858", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6836", "nonce": 2412618858, "type": "v2" }, { "addr": "172.21.15.89:6837", "nonce": 2412618858, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6075, "incarnation": 92, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:rejoin", "state_seq": 36 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:48:30.428940+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5575, "mds_1": 6075, "mds_2": 5728 } } } ], "standbys": [ { "addr": "172.21.15.89:6834/2909790173", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6832", "nonce": 2909790173, "type": "v2" }, { "addr": "172.21.15.89:6834", "nonce": 2909790173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.89:6835/2387357615", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6833", "nonce": 2387357615, "type": "v2" }, { "addr": "172.21.15.89:6835", "nonce": 2387357615, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4239, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:48:37.305 INFO:tasks.ceph.mds.b.smithi192.stderr:2023-10-30T12:48:37.307+0000 7f5231cd7640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T12:48:38.147 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:38.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.299+0000 7f610dd0f640 1 Processor -- start 2023-10-30T12:48:38.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.299+0000 7f610dd0f640 1 -- start start 2023-10-30T12:48:38.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.299+0000 7f610dd0f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61081538a0 0x7f610815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:38.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.299+0000 7f610dd0f640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61080cb800 con 0x7f61080cf160 2023-10-30T12:48:38.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.299+0000 7f610dd0f640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f610815dc30 con 0x7f61080ca0e0 2023-10-30T12:48:38.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.299+0000 7f610dd0f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61080cb330 con 0x7f61081538a0 2023-10-30T12:48:38.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.299+0000 7f6107fff640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f61080cf160 0x7f6108164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:59126/0 (socket says 172.21.15.89:59126) 2023-10-30T12:48:38.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.299+0000 7f6107fff640 1 -- 172.21.15.89:0/1244874444 learned_addr learned my addr 172.21.15.89:0/1244874444 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:48:38.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.299+0000 7f61077fe640 1 --2- 172.21.15.89:0/1244874444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61081538a0 0x7f610815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:38.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.299+0000 7f6106ffd640 1 -- 172.21.15.89:0/1244874444 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1313047885 0 0) 0x7f61080cb800 con 0x7f61080cf160 2023-10-30T12:48:38.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.299+0000 7f6106ffd640 1 -- 172.21.15.89:0/1244874444 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60f4003530 con 0x7f61080cf160 2023-10-30T12:48:38.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.299+0000 7f6106ffd640 1 -- 172.21.15.89:0/1244874444 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4225479387 0 0) 0x7f60f4003530 con 0x7f61080cf160 2023-10-30T12:48:38.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.299+0000 7f6106ffd640 1 -- 172.21.15.89:0/1244874444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61081538a0 msgr2=0x7f610815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:38.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.299+0000 7f6106ffd640 1 --2- 172.21.15.89:0/1244874444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61081538a0 0x7f610815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:38.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.299+0000 7f6106ffd640 1 -- 172.21.15.89:0/1244874444 >> v1:172.21.15.192:6789/0 conn(0x7f61080ca0e0 legacy=0x7f610815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:38.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.299+0000 7f6106ffd640 1 -- 172.21.15.89:0/1244874444 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61080cad70 con 0x7f61080cf160 2023-10-30T12:48:38.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.299+0000 7f6106ffd640 1 -- 172.21.15.89:0/1244874444 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f6100001a00 con 0x7f61080cf160 2023-10-30T12:48:38.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.299+0000 7f6106ffd640 1 -- 172.21.15.89:0/1244874444 >> v1:172.21.15.89:6789/0 conn(0x7f61080cf160 legacy=0x7f6108164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:38.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.303+0000 7f6106ffd640 1 --2- 172.21.15.89:0/1244874444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60f40039b0 0x7f60f4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:38.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.303+0000 7f6106ffd640 1 --2- 172.21.15.89:0/1244874444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61081538a0 0x7f60f40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:38.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.303+0000 7f6106ffd640 1 --2- 172.21.15.89:0/1244874444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60f4004d10 0x7f60f4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:38.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.303+0000 7f6106ffd640 1 -- 172.21.15.89:0/1244874444 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61080cb330 con 0x7f60f40039b0 2023-10-30T12:48:38.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.303+0000 7f6106ffd640 1 -- 172.21.15.89:0/1244874444 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f400bad0 con 0x7f60f4004d10 2023-10-30T12:48:38.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.303+0000 7f6106ffd640 1 -- 172.21.15.89:0/1244874444 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60f400bc90 con 0x7f61081538a0 2023-10-30T12:48:38.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.303+0000 7f61077fe640 1 --2- 172.21.15.89:0/1244874444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60f40039b0 0x7f60f4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:38.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.303+0000 7f6107fff640 1 --2- 172.21.15.89:0/1244874444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60f4004d10 0x7f60f4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:38.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.303+0000 7f60fffff640 1 --2- 172.21.15.89:0/1244874444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61081538a0 0x7f60f40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:38.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.303+0000 7f61077fe640 1 -- 172.21.15.89:0/1244874444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61081538a0 msgr2=0x7f60f40047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:38.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.303+0000 7f61077fe640 1 --2- 172.21.15.89:0/1244874444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61081538a0 0x7f60f40047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:38.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.303+0000 7f61077fe640 1 -- 172.21.15.89:0/1244874444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60f4004d10 msgr2=0x7f60f4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:38.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.303+0000 7f61077fe640 1 --2- 172.21.15.89:0/1244874444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60f4004d10 0x7f60f4005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:38.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.303+0000 7f61077fe640 1 -- 172.21.15.89:0/1244874444 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f4003530 con 0x7f60f40039b0 2023-10-30T12:48:38.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.303+0000 7f61077fe640 1 --2- 172.21.15.89:0/1244874444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60f40039b0 0x7f60f4003dd0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f60f8009ab0 tx=0x7f60f801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c9ff520e0ae869e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:38.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.303+0000 7f6106ffd640 1 -- 172.21.15.89:0/1244874444 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f80a5510 con 0x7f60f40039b0 2023-10-30T12:48:38.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.303+0000 7f6106ffd640 1 -- 172.21.15.89:0/1244874444 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f80a5670 con 0x7f60f40039b0 2023-10-30T12:48:38.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.303+0000 7f6106ffd640 1 -- 172.21.15.89:0/1244874444 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f80a5a20 con 0x7f60f40039b0 2023-10-30T12:48:38.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.307+0000 7f610dd0f640 1 -- 172.21.15.89:0/1244874444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60f40039b0 msgr2=0x7f60f4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:38.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.307+0000 7f610dd0f640 1 --2- 172.21.15.89:0/1244874444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60f40039b0 0x7f60f4003dd0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f60f8009ab0 tx=0x7f60f801ca30 comp rx=0 tx=0).stop 2023-10-30T12:48:38.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.307+0000 7f6107fff640 1 -- 172.21.15.89:0/1244874444 reap_dead start 2023-10-30T12:48:38.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.307+0000 7f610dd0f640 1 -- 172.21.15.89:0/1244874444 shutdown_connections 2023-10-30T12:48:38.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.307+0000 7f610dd0f640 1 -- 172.21.15.89:0/1244874444 >> 172.21.15.89:0/1244874444 conn(0x7f610806bfe0 msgr2=0x7f610806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:38.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.307+0000 7f610dd0f640 1 -- 172.21.15.89:0/1244874444 shutdown_connections 2023-10-30T12:48:38.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.307+0000 7f610dd0f640 1 -- 172.21.15.89:0/1244874444 wait complete. 2023-10-30T12:48:38.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.307+0000 7f610dd0f640 1 Processor -- start 2023-10-30T12:48:38.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.307+0000 7f610dd0f640 1 -- start start 2023-10-30T12:48:38.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.307+0000 7f610dd0f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60f40039b0 0x7f61081eff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:38.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.307+0000 7f610dd0f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6108150bf0 0x7f61081e5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:38.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.307+0000 7f610dd0f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61080ccfb0 0x7f61080b3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:38.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.307+0000 7f610dd0f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61080cd430 con 0x7f60f40039b0 2023-10-30T12:48:38.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.307+0000 7f610dd0f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61081efac0 con 0x7f6108150bf0 2023-10-30T12:48:38.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.307+0000 7f610dd0f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61080cdc20 con 0x7f61080ccfb0 2023-10-30T12:48:38.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.307+0000 7f61077fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60f40039b0 0x7f61081eff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:38.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.307+0000 7f61077fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60f40039b0 0x7f61081eff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44600/0 (socket says 172.21.15.89:44600) 2023-10-30T12:48:38.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.307+0000 7f61077fe640 1 -- 172.21.15.89:0/3329748453 learned_addr learned my addr 172.21.15.89:0/3329748453 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:38.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.307+0000 7f60fffff640 1 --2- 172.21.15.89:0/3329748453 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6108150bf0 0x7f61081e5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:38.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.307+0000 7f6107fff640 1 --2- 172.21.15.89:0/3329748453 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61080ccfb0 0x7f61080b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:38.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.307+0000 7f61077fe640 1 -- 172.21.15.89:0/3329748453 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61080ccfb0 msgr2=0x7f61080b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:38.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.307+0000 7f61077fe640 1 --2- 172.21.15.89:0/3329748453 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61080ccfb0 0x7f61080b3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:38.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.311+0000 7f61077fe640 1 -- 172.21.15.89:0/3329748453 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6108150bf0 msgr2=0x7f61081e5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:38.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.311+0000 7f61077fe640 1 --2- 172.21.15.89:0/3329748453 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6108150bf0 0x7f61081e5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:38.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.311+0000 7f61077fe640 1 -- 172.21.15.89:0/3329748453 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61080cd660 con 0x7f60f40039b0 2023-10-30T12:48:38.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.311+0000 7f60fffff640 1 --2- 172.21.15.89:0/3329748453 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6108150bf0 0x7f61081e5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:38.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.311+0000 7f61077fe640 1 --2- 172.21.15.89:0/3329748453 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60f40039b0 0x7f61081eff50 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f60f8009ab0 tx=0x7f60f4005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:38.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.311+0000 7f6104ff9640 1 -- 172.21.15.89:0/3329748453 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f80be070 con 0x7f60f40039b0 2023-10-30T12:48:38.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.311+0000 7f6104ff9640 1 -- 172.21.15.89:0/3329748453 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f8004020 con 0x7f60f40039b0 2023-10-30T12:48:38.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.311+0000 7f610dd0f640 1 -- 172.21.15.89:0/3329748453 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61080cd940 con 0x7f60f40039b0 2023-10-30T12:48:38.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.311+0000 7f6104ff9640 1 -- 172.21.15.89:0/3329748453 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f8004390 con 0x7f60f40039b0 2023-10-30T12:48:38.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.311+0000 7f610dd0f640 1 -- 172.21.15.89:0/3329748453 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61081ef750 con 0x7f60f40039b0 2023-10-30T12:48:38.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.311+0000 7f610dd0f640 1 -- 172.21.15.89:0/3329748453 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f610804e170 con 0x7f60f40039b0 2023-10-30T12:48:38.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.311+0000 7f6104ff9640 1 -- 172.21.15.89:0/3329748453 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f60f80b4040 con 0x7f60f40039b0 2023-10-30T12:48:38.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.311+0000 7f6104ff9640 1 --2- 172.21.15.89:0/3329748453 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f60e80664a0 0x7f60e8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:38.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.311+0000 7f6104ff9640 1 -- 172.21.15.89:0/3329748453 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6578+0+0 (secure 0 0 0) 0x7f60f811dfd0 con 0x7f60f40039b0 2023-10-30T12:48:38.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.315+0000 7f60fffff640 1 --2- 172.21.15.89:0/3329748453 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f60e80664a0 0x7f60e8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:38.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.315+0000 7f60fffff640 1 --2- 172.21.15.89:0/3329748453 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f60e80664a0 0x7f60e8068960 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f61081e5920 tx=0x7f60f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:38.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.315+0000 7f6104ff9640 1 -- 172.21.15.89:0/3329748453 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f60f80eb170 con 0x7f60f40039b0 2023-10-30T12:48:38.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.495+0000 7f610dd0f640 1 -- 172.21.15.89:0/3329748453 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61080b1dd0 con 0x7f60f40039b0 2023-10-30T12:48:38.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.499+0000 7f6104ff9640 1 -- 172.21.15.89:0/3329748453 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+5744 (secure 0 0 0) 0x7f60f80044f0 con 0x7f60f40039b0 2023-10-30T12:48:38.495 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:48:38.495 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:48:35.514221+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:38.495 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 96 2023-10-30T12:48:38.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.499+0000 7f610dd0f640 1 -- 172.21.15.89:0/3329748453 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f60e80664a0 msgr2=0x7f60e8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:38.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.499+0000 7f610dd0f640 1 --2- 172.21.15.89:0/3329748453 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f60e80664a0 0x7f60e8068960 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f61081e5920 tx=0x7f60f0009040 comp rx=0 tx=0).stop 2023-10-30T12:48:38.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.499+0000 7f610dd0f640 1 -- 172.21.15.89:0/3329748453 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60f40039b0 msgr2=0x7f61081eff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:38.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.499+0000 7f610dd0f640 1 --2- 172.21.15.89:0/3329748453 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60f40039b0 0x7f61081eff50 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f60f8009ab0 tx=0x7f60f4005800 comp rx=0 tx=0).stop 2023-10-30T12:48:38.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.499+0000 7f610dd0f640 1 -- 172.21.15.89:0/3329748453 shutdown_connections 2023-10-30T12:48:38.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.499+0000 7f610dd0f640 1 --2- 172.21.15.89:0/3329748453 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f60e80664a0 0x7f60e8068960 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:38.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.499+0000 7f610dd0f640 1 --2- 172.21.15.89:0/3329748453 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6108150bf0 0x7f61081e5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:38.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.499+0000 7f610dd0f640 1 --2- 172.21.15.89:0/3329748453 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f61080ccfb0 0x7f61080b3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:38.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.499+0000 7f610dd0f640 1 --2- 172.21.15.89:0/3329748453 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60f40039b0 0x7f61081eff50 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:38.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.499+0000 7f610dd0f640 1 -- 172.21.15.89:0/3329748453 >> 172.21.15.89:0/3329748453 conn(0x7f610806bfe0 msgr2=0x7f6108152710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:38.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.499+0000 7f610dd0f640 1 -- 172.21.15.89:0/3329748453 shutdown_connections 2023-10-30T12:48:38.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:38.499+0000 7f610dd0f640 1 -- 172.21.15.89:0/3329748453 wait complete. 2023-10-30T12:48:38.518 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=1, replacing gid=5450 2023-10-30T12:48:38.519 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.f 2023-10-30T12:48:43.520 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-10-30T12:48:43.520 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-30T12:48:43.521 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-30T12:48:43.524 INFO:tasks.ceph.mds.f:Started 2023-10-30T12:48:43.524 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:43.600 INFO:tasks.ceph.mds.f.smithi192.stdout:starting mds.f at 2023-10-30T12:48:43.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.679+0000 7f85b5f3e640 1 Processor -- start 2023-10-30T12:48:43.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.679+0000 7f85b5f3e640 1 -- start start 2023-10-30T12:48:43.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.679+0000 7f85b5f3e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f85b01538a0 0x7f85b015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:43.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.679+0000 7f85b5f3e640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85b00cb800 con 0x7f85b00cf160 2023-10-30T12:48:43.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.679+0000 7f85b5f3e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85b015dc30 con 0x7f85b00ca0e0 2023-10-30T12:48:43.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.679+0000 7f85b5f3e640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85b00cb330 con 0x7f85b01538a0 2023-10-30T12:48:43.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.683+0000 7f85af7fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f85b01538a0 0x7f85b015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:43.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.683+0000 7f85affff640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f85b00cf160 0x7f85b0164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:53610/0 (socket says 172.21.15.89:53610) 2023-10-30T12:48:43.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.683+0000 7f85affff640 1 -- 172.21.15.89:0/3989094862 learned_addr learned my addr 172.21.15.89:0/3989094862 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:48:43.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.683+0000 7f85af7fe640 1 -- 172.21.15.89:0/3989094862 >> v1:172.21.15.192:6789/0 conn(0x7f85b00ca0e0 legacy=0x7f85b015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:43.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.683+0000 7f85ae7fc640 1 -- 172.21.15.89:0/3989094862 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3445219682 0 0) 0x7f85b00cb800 con 0x7f85b00cf160 2023-10-30T12:48:43.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.683+0000 7f85af7fe640 1 -- 172.21.15.89:0/3989094862 >> v1:172.21.15.89:6789/0 conn(0x7f85b00cf160 legacy=0x7f85b0164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:43.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.683+0000 7f85af7fe640 1 -- 172.21.15.89:0/3989094862 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b00cad70 con 0x7f85b01538a0 2023-10-30T12:48:43.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.683+0000 7f85af7fe640 1 --2- 172.21.15.89:0/3989094862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85b01538a0 0x7f85b015d6f0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f85a00097c0 tx=0x7f85a001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45396e1965f1ebd4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:43.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.687+0000 7f85ae7fc640 1 -- 172.21.15.89:0/3989094862 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a00a8970 con 0x7f85b01538a0 2023-10-30T12:48:43.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.687+0000 7f85ae7fc640 1 -- 172.21.15.89:0/3989094862 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a00a8ad0 con 0x7f85b01538a0 2023-10-30T12:48:43.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.687+0000 7f85ae7fc640 1 -- 172.21.15.89:0/3989094862 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a00a8e60 con 0x7f85b01538a0 2023-10-30T12:48:43.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.687+0000 7f85b5f3e640 1 -- 172.21.15.89:0/3989094862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85b01538a0 msgr2=0x7f85b015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:43.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.687+0000 7f85b5f3e640 1 --2- 172.21.15.89:0/3989094862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85b01538a0 0x7f85b015d6f0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f85a00097c0 tx=0x7f85a001ec70 comp rx=0 tx=0).stop 2023-10-30T12:48:43.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.687+0000 7f85b5f3e640 1 -- 172.21.15.89:0/3989094862 shutdown_connections 2023-10-30T12:48:43.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.687+0000 7f85b5f3e640 1 --2- 172.21.15.89:0/3989094862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85b01538a0 0x7f85b015d6f0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:43.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.687+0000 7f85b5f3e640 1 -- 172.21.15.89:0/3989094862 >> 172.21.15.89:0/3989094862 conn(0x7f85b006bfe0 msgr2=0x7f85b006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:43.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.687+0000 7f85b5f3e640 1 -- 172.21.15.89:0/3989094862 shutdown_connections 2023-10-30T12:48:43.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.687+0000 7f85b5f3e640 1 -- 172.21.15.89:0/3989094862 wait complete. 2023-10-30T12:48:43.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.687+0000 7f85b5f3e640 1 Processor -- start 2023-10-30T12:48:43.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.687+0000 7f85b5f3e640 1 -- start start 2023-10-30T12:48:43.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.687+0000 7f85b5f3e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85b00ca0e0 0x7f85b00c4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:43.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.687+0000 7f85b5f3e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85b00cf160 0x7f85b01507d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:43.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.687+0000 7f85b5f3e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85b01538a0 0x7f85b00b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:43.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.687+0000 7f85b5f3e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b0157830 con 0x7f85b01538a0 2023-10-30T12:48:43.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.687+0000 7f85b5f3e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b01575c0 con 0x7f85b00ca0e0 2023-10-30T12:48:43.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.687+0000 7f85b5f3e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b00b3d90 con 0x7f85b00cf160 2023-10-30T12:48:43.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.687+0000 7f85affff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85b01538a0 0x7f85b00b42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:43.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.687+0000 7f85affff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85b01538a0 0x7f85b00b42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58170/0 (socket says 172.21.15.89:58170) 2023-10-30T12:48:43.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.687+0000 7f85affff640 1 -- 172.21.15.89:0/3056339607 learned_addr learned my addr 172.21.15.89:0/3056339607 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:43.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.687+0000 7f85af7fe640 1 --2- 172.21.15.89:0/3056339607 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85b00ca0e0 0x7f85b00c4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:43.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.687+0000 7f85aeffd640 1 --2- 172.21.15.89:0/3056339607 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85b00cf160 0x7f85b01507d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:43.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.691+0000 7f85affff640 1 -- 172.21.15.89:0/3056339607 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85b00cf160 msgr2=0x7f85b01507d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:43.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.691+0000 7f85affff640 1 --2- 172.21.15.89:0/3056339607 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85b00cf160 0x7f85b01507d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:43.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.691+0000 7f85affff640 1 -- 172.21.15.89:0/3056339607 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85b00ca0e0 msgr2=0x7f85b00c4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:43.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.691+0000 7f85affff640 1 --2- 172.21.15.89:0/3056339607 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85b00ca0e0 0x7f85b00c4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:43.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.691+0000 7f85affff640 1 -- 172.21.15.89:0/3056339607 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a0009420 con 0x7f85b01538a0 2023-10-30T12:48:43.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.691+0000 7f85affff640 1 --2- 172.21.15.89:0/3056339607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85b01538a0 0x7f85b00b42e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f85a8001570 tx=0x7f85a800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:43.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.691+0000 7f8597fff640 1 -- 172.21.15.89:0/3056339607 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a800cea0 con 0x7f85b01538a0 2023-10-30T12:48:43.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.691+0000 7f8597fff640 1 -- 172.21.15.89:0/3056339607 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a800a5e0 con 0x7f85b01538a0 2023-10-30T12:48:43.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.691+0000 7f8597fff640 1 -- 172.21.15.89:0/3056339607 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a800a990 con 0x7f85b01538a0 2023-10-30T12:48:43.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.691+0000 7f85b5f3e640 1 -- 172.21.15.89:0/3056339607 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85b0159040 con 0x7f85b01538a0 2023-10-30T12:48:43.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.691+0000 7f85b5f3e640 1 -- 172.21.15.89:0/3056339607 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85b00b4010 con 0x7f85b01538a0 2023-10-30T12:48:43.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.691+0000 7f85b5f3e640 1 -- 172.21.15.89:0/3056339607 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85b004e170 con 0x7f85b01538a0 2023-10-30T12:48:43.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.691+0000 7f8597fff640 1 -- 172.21.15.89:0/3056339607 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f85a800aaf0 con 0x7f85b01538a0 2023-10-30T12:48:43.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.691+0000 7f8597fff640 1 --2- 172.21.15.89:0/3056339607 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f859c0664a0 0x7f859c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:43.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.691+0000 7f8597fff640 1 -- 172.21.15.89:0/3056339607 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6578+0+0 (secure 0 0 0) 0x7f85a8083810 con 0x7f85b01538a0 2023-10-30T12:48:43.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.691+0000 7f85af7fe640 1 --2- 172.21.15.89:0/3056339607 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f859c0664a0 0x7f859c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:43.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.695+0000 7f85af7fe640 1 --2- 172.21.15.89:0/3056339607 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f859c0664a0 0x7f859c068960 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f85a00b2000 tx=0x7f85a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:43.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.695+0000 7f8597fff640 1 -- 172.21.15.89:0/3056339607 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f85a8050800 con 0x7f85b01538a0 2023-10-30T12:48:43.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.899+0000 7f85b5f3e640 1 -- 172.21.15.89:0/3056339607 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85b0159b40 con 0x7f85b01538a0 2023-10-30T12:48:43.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.899+0000 7f8597fff640 1 -- 172.21.15.89:0/3056339607 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+5744 (secure 0 0 0) 0x7f85a8053e20 con 0x7f85b01538a0 2023-10-30T12:48:43.898 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:48:43.898 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:48:35.514221+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:43.899 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 96 2023-10-30T12:48:43.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.903+0000 7f85b5f3e640 1 -- 172.21.15.89:0/3056339607 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f859c0664a0 msgr2=0x7f859c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:43.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.903+0000 7f85b5f3e640 1 --2- 172.21.15.89:0/3056339607 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f859c0664a0 0x7f859c068960 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f85a00b2000 tx=0x7f85a0008040 comp rx=0 tx=0).stop 2023-10-30T12:48:43.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.903+0000 7f85b5f3e640 1 -- 172.21.15.89:0/3056339607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85b01538a0 msgr2=0x7f85b00b42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:43.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.903+0000 7f85b5f3e640 1 --2- 172.21.15.89:0/3056339607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85b01538a0 0x7f85b00b42e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f85a8001570 tx=0x7f85a800cc50 comp rx=0 tx=0).stop 2023-10-30T12:48:43.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.903+0000 7f85b5f3e640 1 -- 172.21.15.89:0/3056339607 shutdown_connections 2023-10-30T12:48:43.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.903+0000 7f85b5f3e640 1 --2- 172.21.15.89:0/3056339607 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f859c0664a0 0x7f859c068960 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:43.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.903+0000 7f85b5f3e640 1 --2- 172.21.15.89:0/3056339607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85b01538a0 0x7f85b00b42e0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:43.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.903+0000 7f85b5f3e640 1 --2- 172.21.15.89:0/3056339607 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f85b00cf160 0x7f85b01507d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:43.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.903+0000 7f85b5f3e640 1 --2- 172.21.15.89:0/3056339607 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85b00ca0e0 0x7f85b00c4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:43.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.903+0000 7f85b5f3e640 1 -- 172.21.15.89:0/3056339607 >> 172.21.15.89:0/3056339607 conn(0x7f85b006bfe0 msgr2=0x7f85b0152310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:43.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.903+0000 7f85b5f3e640 1 -- 172.21.15.89:0/3056339607 shutdown_connections 2023-10-30T12:48:43.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:43.903+0000 7f85b5f3e640 1 -- 172.21.15.89:0/3056339607 wait complete. 2023-10-30T12:48:43.920 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2023-10-30T12:48:45.921 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:46.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.091+0000 7f3dba31d640 1 Processor -- start 2023-10-30T12:48:46.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.095+0000 7f3dba31d640 1 -- start start 2023-10-30T12:48:46.091 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.095+0000 7f3dba31d640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3db415d6f0 0x7f3db415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:46.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.095+0000 7f3dba31d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3db40cf160 0x7f3db40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:46.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.095+0000 7f3dba31d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db40cb800 con 0x7f3db41538a0 2023-10-30T12:48:46.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.095+0000 7f3dba31d640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3db40cb330 con 0x7f3db40cf160 2023-10-30T12:48:46.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.095+0000 7f3dba31d640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3db40cad70 con 0x7f3db415d6f0 2023-10-30T12:48:46.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.095+0000 7f3db88ba640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3db40cf160 0x7f3db40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:46.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.095+0000 7f3db88ba640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3db40cf160 0x7f3db40cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58186/0 (socket says 172.21.15.89:58186) 2023-10-30T12:48:46.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.095+0000 7f3db88ba640 1 -- 172.21.15.89:0/4228694672 learned_addr learned my addr 172.21.15.89:0/4228694672 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:46.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.095+0000 7f3db37fe640 1 --2- 172.21.15.89:0/4228694672 >> v2:172.21.15.192:3300/0 conn(0x7f3db415d6f0 0x7f3db415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:46.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.095+0000 7f3db88ba640 1 -- 172.21.15.89:0/4228694672 >> v2:172.21.15.192:3300/0 conn(0x7f3db415d6f0 msgr2=0x7f3db415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:46.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.095+0000 7f3db88ba640 1 --2- 172.21.15.89:0/4228694672 >> v2:172.21.15.192:3300/0 conn(0x7f3db415d6f0 0x7f3db415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:46.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.095+0000 7f3db88ba640 1 -- 172.21.15.89:0/4228694672 >> v1:172.21.15.192:6789/0 conn(0x7f3db41538a0 legacy=0x7f3db40ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:46.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.095+0000 7f3db88ba640 1 -- 172.21.15.89:0/4228694672 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db40cb050 con 0x7f3db40cf160 2023-10-30T12:48:46.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.099+0000 7f3db88ba640 1 --2- 172.21.15.89:0/4228694672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3db40cf160 0x7f3db40cf590 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f3db40b30a0 tx=0x7f3dac01f180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aee4c7cfa063b111 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:46.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.099+0000 7f3db2ffd640 1 -- 172.21.15.89:0/4228694672 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dac0af400 con 0x7f3db40cf160 2023-10-30T12:48:46.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.099+0000 7f3db2ffd640 1 -- 172.21.15.89:0/4228694672 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dac0ab050 con 0x7f3db40cf160 2023-10-30T12:48:46.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.099+0000 7f3db2ffd640 1 -- 172.21.15.89:0/4228694672 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dac0af780 con 0x7f3db40cf160 2023-10-30T12:48:46.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.099+0000 7f3dba31d640 1 -- 172.21.15.89:0/4228694672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3db40cf160 msgr2=0x7f3db40cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:46.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.099+0000 7f3dba31d640 1 --2- 172.21.15.89:0/4228694672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3db40cf160 0x7f3db40cf590 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f3db40b30a0 tx=0x7f3dac01f180 comp rx=0 tx=0).stop 2023-10-30T12:48:46.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.099+0000 7f3dba31d640 1 -- 172.21.15.89:0/4228694672 shutdown_connections 2023-10-30T12:48:46.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.099+0000 7f3dba31d640 1 --2- 172.21.15.89:0/4228694672 >> v2:172.21.15.192:3300/0 conn(0x7f3db415d6f0 0x7f3db415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:46.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.099+0000 7f3dba31d640 1 --2- 172.21.15.89:0/4228694672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3db40cf160 0x7f3db40cf590 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:46.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.099+0000 7f3dba31d640 1 -- 172.21.15.89:0/4228694672 >> 172.21.15.89:0/4228694672 conn(0x7f3db406bfe0 msgr2=0x7f3db406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:46.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.099+0000 7f3dba31d640 1 -- 172.21.15.89:0/4228694672 shutdown_connections 2023-10-30T12:48:46.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.099+0000 7f3dba31d640 1 -- 172.21.15.89:0/4228694672 wait complete. 2023-10-30T12:48:46.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.099+0000 7f3dba31d640 1 Processor -- start 2023-10-30T12:48:46.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.099+0000 7f3dba31d640 1 -- start start 2023-10-30T12:48:46.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.103+0000 7f3dba31d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3db40cf160 0x7f3db41ef890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:46.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.103+0000 7f3dba31d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3db41538a0 0x7f3db40b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:46.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.103+0000 7f3dba31d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3db415d6f0 0x7f3db41e3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:46.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.103+0000 7f3dba31d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db41e4e10 con 0x7f3db41538a0 2023-10-30T12:48:46.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.103+0000 7f3dba31d640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db414d680 con 0x7f3db40cf160 2023-10-30T12:48:46.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.103+0000 7f3dba31d640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3db41e2a40 con 0x7f3db415d6f0 2023-10-30T12:48:46.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.103+0000 7f3db37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3db41538a0 0x7f3db40b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:46.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.103+0000 7f3db37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3db41538a0 0x7f3db40b4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58190/0 (socket says 172.21.15.89:58190) 2023-10-30T12:48:46.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.103+0000 7f3db37fe640 1 -- 172.21.15.89:0/3938838032 learned_addr learned my addr 172.21.15.89:0/3938838032 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:46.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.103+0000 7f3db3fff640 1 --2- 172.21.15.89:0/3938838032 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3db40cf160 0x7f3db41ef890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:46.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.103+0000 7f3db88ba640 1 --2- 172.21.15.89:0/3938838032 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3db415d6f0 0x7f3db41e3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:46.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.103+0000 7f3db37fe640 1 -- 172.21.15.89:0/3938838032 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3db415d6f0 msgr2=0x7f3db41e3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:46.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.103+0000 7f3db37fe640 1 --2- 172.21.15.89:0/3938838032 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3db415d6f0 0x7f3db41e3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:46.106 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.103+0000 7f3db37fe640 1 -- 172.21.15.89:0/3938838032 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3db40cf160 msgr2=0x7f3db41ef890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:46.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.103+0000 7f3db37fe640 1 --2- 172.21.15.89:0/3938838032 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3db40cf160 0x7f3db41ef890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:46.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.103+0000 7f3db37fe640 1 -- 172.21.15.89:0/3938838032 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dac009b30 con 0x7f3db41538a0 2023-10-30T12:48:46.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.103+0000 7f3db37fe640 1 --2- 172.21.15.89:0/3938838032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3db41538a0 0x7f3db40b4580 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f3da800d620 tx=0x7f3da800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:46.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.103+0000 7f3db0ff9640 1 -- 172.21.15.89:0/3938838032 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da800b620 con 0x7f3db41538a0 2023-10-30T12:48:46.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.103+0000 7f3db0ff9640 1 -- 172.21.15.89:0/3938838032 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3da800b780 con 0x7f3db41538a0 2023-10-30T12:48:46.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.103+0000 7f3dba31d640 1 -- 172.21.15.89:0/3938838032 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3db4158f70 con 0x7f3db41538a0 2023-10-30T12:48:46.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.103+0000 7f3db0ff9640 1 -- 172.21.15.89:0/3938838032 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da800bb10 con 0x7f3db41538a0 2023-10-30T12:48:46.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.103+0000 7f3dba31d640 1 -- 172.21.15.89:0/3938838032 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3db41594f0 con 0x7f3db41538a0 2023-10-30T12:48:46.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.103+0000 7f3dba31d640 1 -- 172.21.15.89:0/3938838032 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3db40b2a50 con 0x7f3db41538a0 2023-10-30T12:48:46.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.103+0000 7f3db0ff9640 1 -- 172.21.15.89:0/3938838032 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3da800bc70 con 0x7f3db41538a0 2023-10-30T12:48:46.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.107+0000 7f3db0ff9640 1 --2- 172.21.15.89:0/3938838032 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3da00664a0 0x7f3da0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:46.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.107+0000 7f3db0ff9640 1 -- 172.21.15.89:0/3938838032 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6578+0+0 (secure 0 0 0) 0x7f3da8084610 con 0x7f3db41538a0 2023-10-30T12:48:46.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.107+0000 7f3db3fff640 1 --2- 172.21.15.89:0/3938838032 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3da00664a0 0x7f3da0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:46.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.107+0000 7f3db3fff640 1 --2- 172.21.15.89:0/3938838032 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3da00664a0 0x7f3da0068960 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3da4007fd0 tx=0x7f3da4007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:46.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.111+0000 7f3db0ff9640 1 -- 172.21.15.89:0/3938838032 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3da8051730 con 0x7f3db41538a0 2023-10-30T12:48:46.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.295+0000 7f3dba31d640 1 -- 172.21.15.89:0/3938838032 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3db406c3c0 con 0x7f3db41538a0 2023-10-30T12:48:46.292 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.295+0000 7f3db0ff9640 1 -- 172.21.15.89:0/3938838032 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f3da8007380 con 0x7f3db41538a0 2023-10-30T12:48:46.293 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:48:46.293 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5678,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:48:35.514221+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:46.293 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 97 2023-10-30T12:48:46.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.299+0000 7f3dba31d640 1 -- 172.21.15.89:0/3938838032 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3da00664a0 msgr2=0x7f3da0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:46.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.299+0000 7f3dba31d640 1 --2- 172.21.15.89:0/3938838032 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3da00664a0 0x7f3da0068960 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3da4007fd0 tx=0x7f3da4007e40 comp rx=0 tx=0).stop 2023-10-30T12:48:46.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.299+0000 7f3dba31d640 1 -- 172.21.15.89:0/3938838032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3db41538a0 msgr2=0x7f3db40b4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:46.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.299+0000 7f3dba31d640 1 --2- 172.21.15.89:0/3938838032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3db41538a0 0x7f3db40b4580 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f3da800d620 tx=0x7f3da800dae0 comp rx=0 tx=0).stop 2023-10-30T12:48:46.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.299+0000 7f3dba31d640 1 -- 172.21.15.89:0/3938838032 shutdown_connections 2023-10-30T12:48:46.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.299+0000 7f3dba31d640 1 --2- 172.21.15.89:0/3938838032 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3da00664a0 0x7f3da0068960 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:46.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.299+0000 7f3dba31d640 1 --2- 172.21.15.89:0/3938838032 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3db415d6f0 0x7f3db41e3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:46.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.299+0000 7f3dba31d640 1 --2- 172.21.15.89:0/3938838032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3db41538a0 0x7f3db40b4580 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:46.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.299+0000 7f3dba31d640 1 --2- 172.21.15.89:0/3938838032 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3db40cf160 0x7f3db41ef890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:46.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.299+0000 7f3dba31d640 1 -- 172.21.15.89:0/3938838032 >> 172.21.15.89:0/3938838032 conn(0x7f3db406bfe0 msgr2=0x7f3db40c4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:46.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.299+0000 7f3dba31d640 1 -- 172.21.15.89:0/3938838032 shutdown_connections 2023-10-30T12:48:46.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:46.299+0000 7f3dba31d640 1 -- 172.21.15.89:0/3938838032 wait complete. 2023-10-30T12:48:46.313 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-10-30T12:48:46.313 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before thrashing 2023-10-30T12:48:48.216 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:48:48.220 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:48:49.314 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:49.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.455+0000 7fbde3571640 1 Processor -- start 2023-10-30T12:48:49.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.455+0000 7fbde3571640 1 -- start start 2023-10-30T12:48:49.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde3571640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbddc0cefa0 0x7fbddc0cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:49.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde3571640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbddc15a6c0 con 0x7fbddc15cd50 2023-10-30T12:48:49.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde3571640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbddc0ca6a0 con 0x7fbddc1538f0 2023-10-30T12:48:49.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde3571640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbddc0cf8f0 con 0x7fbddc0cefa0 2023-10-30T12:48:49.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde1d6e640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fbddc15cd50 0x7fbddc0c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:53614/0 (socket says 172.21.15.89:53614) 2023-10-30T12:48:49.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde1d6e640 1 -- 172.21.15.89:0/256936931 learned_addr learned my addr 172.21.15.89:0/256936931 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:48:49.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde2d70640 1 --2- 172.21.15.89:0/256936931 >> v2:172.21.15.192:3300/0 conn(0x7fbddc0cefa0 0x7fbddc0cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:49.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde156d640 1 -- 172.21.15.89:0/256936931 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2556822146 0 0) 0x7fbddc15a6c0 con 0x7fbddc15cd50 2023-10-30T12:48:49.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde156d640 1 -- 172.21.15.89:0/256936931 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdc8003530 con 0x7fbddc15cd50 2023-10-30T12:48:49.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde156d640 1 -- 172.21.15.89:0/256936931 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4075548453 0 0) 0x7fbdc8003530 con 0x7fbddc15cd50 2023-10-30T12:48:49.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde156d640 1 -- 172.21.15.89:0/256936931 >> v2:172.21.15.192:3300/0 conn(0x7fbddc0cefa0 msgr2=0x7fbddc0cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:49.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde156d640 1 --2- 172.21.15.89:0/256936931 >> v2:172.21.15.192:3300/0 conn(0x7fbddc0cefa0 0x7fbddc0cf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:49.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde156d640 1 -- 172.21.15.89:0/256936931 >> v1:172.21.15.192:6789/0 conn(0x7fbddc1538f0 legacy=0x7fbddc159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:49.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde156d640 1 -- 172.21.15.89:0/256936931 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbddc15d300 con 0x7fbddc15cd50 2023-10-30T12:48:49.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde156d640 1 -- 172.21.15.89:0/256936931 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7fbdcc001680 con 0x7fbddc15cd50 2023-10-30T12:48:49.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde156d640 1 -- 172.21.15.89:0/256936931 >> v1:172.21.15.89:6789/0 conn(0x7fbddc15cd50 legacy=0x7fbddc0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:49.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde156d640 1 --2- 172.21.15.89:0/256936931 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbdc8003980 0x7fbdc8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:49.460 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde156d640 1 --2- 172.21.15.89:0/256936931 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbdc80047d0 0x7fbdc8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:49.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde156d640 1 --2- 172.21.15.89:0/256936931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdc8005190 0x7fbdc800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:49.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde156d640 1 -- 172.21.15.89:0/256936931 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbddc0cf8f0 con 0x7fbdc8005190 2023-10-30T12:48:49.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde156d640 1 -- 172.21.15.89:0/256936931 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc800c040 con 0x7fbdc8003980 2023-10-30T12:48:49.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde156d640 1 -- 172.21.15.89:0/256936931 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc800c200 con 0x7fbdc80047d0 2023-10-30T12:48:49.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde2d70640 1 --2- 172.21.15.89:0/256936931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdc8005190 0x7fbdc800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:49.461 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde256f640 1 --2- 172.21.15.89:0/256936931 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbdc8003980 0x7fbdc8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:49.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde1d6e640 1 --2- 172.21.15.89:0/256936931 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbdc80047d0 0x7fbdc8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:49.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde2d70640 1 -- 172.21.15.89:0/256936931 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbdc80047d0 msgr2=0x7fbdc8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:49.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde2d70640 1 --2- 172.21.15.89:0/256936931 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbdc80047d0 0x7fbdc8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:49.462 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde2d70640 1 -- 172.21.15.89:0/256936931 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbdc8003980 msgr2=0x7fbdc8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:49.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde2d70640 1 --2- 172.21.15.89:0/256936931 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbdc8003980 0x7fbdc8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:49.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde2d70640 1 -- 172.21.15.89:0/256936931 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc8003530 con 0x7fbdc8005190 2023-10-30T12:48:49.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.459+0000 7fbde1d6e640 1 --2- 172.21.15.89:0/256936931 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbdc80047d0 0x7fbdc8004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:49.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.463+0000 7fbde2d70640 1 -- 172.21.15.89:0/256936931 reap_dead start 2023-10-30T12:48:49.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.463+0000 7fbde2d70640 1 --2- 172.21.15.89:0/256936931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdc8005190 0x7fbdc800b990 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fbdd4009fc0 tx=0x7fbdd401cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dae144668b2b272 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:49.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.463+0000 7fbde156d640 1 -- 172.21.15.89:0/256936931 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd40a5cb0 con 0x7fbdc8005190 2023-10-30T12:48:49.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.463+0000 7fbde156d640 1 -- 172.21.15.89:0/256936931 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdd400f040 con 0x7fbdc8005190 2023-10-30T12:48:49.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.463+0000 7fbde156d640 1 -- 172.21.15.89:0/256936931 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd4003be0 con 0x7fbdc8005190 2023-10-30T12:48:49.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.463+0000 7fbde3571640 1 -- 172.21.15.89:0/256936931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdc8005190 msgr2=0x7fbdc800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:49.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.463+0000 7fbde3571640 1 --2- 172.21.15.89:0/256936931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdc8005190 0x7fbdc800b990 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fbdd4009fc0 tx=0x7fbdd401cca0 comp rx=0 tx=0).stop 2023-10-30T12:48:49.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.463+0000 7fbde3571640 1 -- 172.21.15.89:0/256936931 shutdown_connections 2023-10-30T12:48:49.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.463+0000 7fbde3571640 1 --2- 172.21.15.89:0/256936931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbdc8005190 0x7fbdc800b990 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:49.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.463+0000 7fbde3571640 1 -- 172.21.15.89:0/256936931 >> 172.21.15.89:0/256936931 conn(0x7fbddc06a640 msgr2=0x7fbddc065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:49.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.463+0000 7fbde3571640 1 -- 172.21.15.89:0/256936931 shutdown_connections 2023-10-30T12:48:49.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.463+0000 7fbde3571640 1 -- 172.21.15.89:0/256936931 wait complete. 2023-10-30T12:48:49.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.467+0000 7fbde3571640 1 Processor -- start 2023-10-30T12:48:49.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.467+0000 7fbde3571640 1 -- start start 2023-10-30T12:48:49.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.467+0000 7fbde3571640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbdc8005190 0x7fbddc1573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:49.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.467+0000 7fbde3571640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbddc142ef0 0x7fbddc1452b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:49.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.467+0000 7fbde3571640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbddc157b20 0x7fbddc15bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:49.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.467+0000 7fbde3571640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbddc143330 con 0x7fbddc142ef0 2023-10-30T12:48:49.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.467+0000 7fbde3571640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbddc157fa0 con 0x7fbdc8005190 2023-10-30T12:48:49.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.467+0000 7fbde3571640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbddc1437f0 con 0x7fbddc157b20 2023-10-30T12:48:49.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.467+0000 7fbde1d6e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbddc142ef0 0x7fbddc1452b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:49.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.467+0000 7fbde1d6e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbddc142ef0 0x7fbddc1452b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58206/0 (socket says 172.21.15.89:58206) 2023-10-30T12:48:49.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.467+0000 7fbde1d6e640 1 -- 172.21.15.89:0/4291587358 learned_addr learned my addr 172.21.15.89:0/4291587358 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:49.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.467+0000 7fbde2d70640 1 --2- 172.21.15.89:0/4291587358 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbddc157b20 0x7fbddc15bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:49.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.467+0000 7fbde256f640 1 --2- 172.21.15.89:0/4291587358 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbdc8005190 0x7fbddc1573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:49.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.467+0000 7fbde1d6e640 1 -- 172.21.15.89:0/4291587358 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbddc157b20 msgr2=0x7fbddc15bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:49.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.467+0000 7fbde1d6e640 1 --2- 172.21.15.89:0/4291587358 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbddc157b20 0x7fbddc15bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:49.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.467+0000 7fbde1d6e640 1 -- 172.21.15.89:0/4291587358 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbdc8005190 msgr2=0x7fbddc1573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:49.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.467+0000 7fbde1d6e640 1 --2- 172.21.15.89:0/4291587358 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbdc8005190 0x7fbddc1573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:49.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.467+0000 7fbde1d6e640 1 -- 172.21.15.89:0/4291587358 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbddc0b4a00 con 0x7fbddc142ef0 2023-10-30T12:48:49.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.467+0000 7fbde1d6e640 1 --2- 172.21.15.89:0/4291587358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbddc142ef0 0x7fbddc1452b0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc0017b0 tx=0x7fbdcc00ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:49.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.467+0000 7fbdd2ffd640 1 -- 172.21.15.89:0/4291587358 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdcc00c660 con 0x7fbddc142ef0 2023-10-30T12:48:49.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.467+0000 7fbdd2ffd640 1 -- 172.21.15.89:0/4291587358 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdcc00c7c0 con 0x7fbddc142ef0 2023-10-30T12:48:49.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.467+0000 7fbdd2ffd640 1 -- 172.21.15.89:0/4291587358 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdcc00cb30 con 0x7fbddc142ef0 2023-10-30T12:48:49.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.467+0000 7fbde3571640 1 -- 172.21.15.89:0/4291587358 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbddc144e20 con 0x7fbddc142ef0 2023-10-30T12:48:49.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.467+0000 7fbde3571640 1 -- 172.21.15.89:0/4291587358 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbddc0b5210 con 0x7fbddc142ef0 2023-10-30T12:48:49.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.467+0000 7fbde3571640 1 -- 172.21.15.89:0/4291587358 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbddc144fb0 con 0x7fbddc142ef0 2023-10-30T12:48:49.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.471+0000 7fbdd2ffd640 1 -- 172.21.15.89:0/4291587358 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbdcc00cc90 con 0x7fbddc142ef0 2023-10-30T12:48:49.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.471+0000 7fbdd2ffd640 1 --2- 172.21.15.89:0/4291587358 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbdc00664a0 0x7fbdc0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:49.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.471+0000 7fbdd2ffd640 1 -- 172.21.15.89:0/4291587358 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6578+0+0 (secure 0 0 0) 0x7fbdcc084080 con 0x7fbddc142ef0 2023-10-30T12:48:49.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.471+0000 7fbde256f640 1 --2- 172.21.15.89:0/4291587358 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbdc00664a0 0x7fbdc0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:49.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.471+0000 7fbde256f640 1 --2- 172.21.15.89:0/4291587358 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbdc00664a0 0x7fbdc0068960 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8007fd0 tx=0x7fbdd8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:49.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.475+0000 7fbdd2ffd640 1 -- 172.21.15.89:0/4291587358 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbdcc0b5090 con 0x7fbddc142ef0 2023-10-30T12:48:49.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.663+0000 7fbde3571640 1 -- 172.21.15.89:0/4291587358 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbddc04e170 con 0x7fbddc142ef0 2023-10-30T12:48:49.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.663+0000 7fbdd2ffd640 1 -- 172.21.15.89:0/4291587358 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+6538 (secure 0 0 0) 0x7fbdcc0511a0 con 0x7fbddc142ef0 2023-10-30T12:48:49.660 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:48:49.660 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5678,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:48:35.514221+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:49.661 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 97 2023-10-30T12:48:49.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.663+0000 7fbde3571640 1 -- 172.21.15.89:0/4291587358 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbdc00664a0 msgr2=0x7fbdc0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:49.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.663+0000 7fbde3571640 1 --2- 172.21.15.89:0/4291587358 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbdc00664a0 0x7fbdc0068960 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8007fd0 tx=0x7fbdd8009040 comp rx=0 tx=0).stop 2023-10-30T12:48:49.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.663+0000 7fbde3571640 1 -- 172.21.15.89:0/4291587358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbddc142ef0 msgr2=0x7fbddc1452b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:49.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.663+0000 7fbde3571640 1 --2- 172.21.15.89:0/4291587358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbddc142ef0 0x7fbddc1452b0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc0017b0 tx=0x7fbdcc00ec00 comp rx=0 tx=0).stop 2023-10-30T12:48:49.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.663+0000 7fbde3571640 1 -- 172.21.15.89:0/4291587358 shutdown_connections 2023-10-30T12:48:49.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.663+0000 7fbde3571640 1 --2- 172.21.15.89:0/4291587358 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbdc00664a0 0x7fbdc0068960 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:49.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.663+0000 7fbde3571640 1 --2- 172.21.15.89:0/4291587358 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbddc157b20 0x7fbddc15bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:49.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.663+0000 7fbde3571640 1 --2- 172.21.15.89:0/4291587358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbddc142ef0 0x7fbddc1452b0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:49.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.663+0000 7fbde3571640 1 --2- 172.21.15.89:0/4291587358 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbdc8005190 0x7fbddc1573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:49.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.663+0000 7fbde3571640 1 -- 172.21.15.89:0/4291587358 >> 172.21.15.89:0/4291587358 conn(0x7fbddc06a640 msgr2=0x7fbddc0c51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:49.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.663+0000 7fbde3571640 1 -- 172.21.15.89:0/4291587358 shutdown_connections 2023-10-30T12:48:49.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.663+0000 7fbde3571640 1 -- 172.21.15.89:0/4291587358 wait complete. 2023-10-30T12:48:49.680 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2023-10-30T12:48:49.681 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-30T12:48:49.681 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:48:49.700 INFO:tasks.ceph.mds.b:Stopped 2023-10-30T12:48:49.700 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:49.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.839+0000 7fe8f86ea640 1 Processor -- start 2023-10-30T12:48:49.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.839+0000 7fe8f86ea640 1 -- start start 2023-10-30T12:48:49.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.843+0000 7fe8f86ea640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe8f006f7a0 0x7fe8f006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:49.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.843+0000 7fe8f86ea640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe8f0071110 0x7fe8f0158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:49.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.843+0000 7fe8f86ea640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8f0066cf0 con 0x7fe8f00c6bd0 2023-10-30T12:48:49.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.843+0000 7fe8f86ea640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8f00700c0 con 0x7fe8f006f7a0 2023-10-30T12:48:49.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.843+0000 7fe8f86ea640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8f00c70d0 con 0x7fe8f0071110 2023-10-30T12:48:49.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.843+0000 7fe8f6486640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe8f006f7a0 0x7fe8f006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:49.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.843+0000 7fe8f5c85640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe8f00c6bd0 0x7fe8f0069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39854/0 (socket says 172.21.15.89:39854) 2023-10-30T12:48:49.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.843+0000 7fe8f5c85640 1 -- 172.21.15.89:0/526199640 learned_addr learned my addr 172.21.15.89:0/526199640 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:48:49.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.843+0000 7fe8f6c87640 1 --2- 172.21.15.89:0/526199640 >> v2:172.21.15.192:3300/0 conn(0x7fe8f0071110 0x7fe8f0158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:49.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.843+0000 7fe8f6486640 1 -- 172.21.15.89:0/526199640 >> v2:172.21.15.192:3300/0 conn(0x7fe8f0071110 msgr2=0x7fe8f0158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:49.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.843+0000 7fe8f6486640 1 --2- 172.21.15.89:0/526199640 >> v2:172.21.15.192:3300/0 conn(0x7fe8f0071110 0x7fe8f0158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:49.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.843+0000 7fe8f6486640 1 -- 172.21.15.89:0/526199640 >> v1:172.21.15.89:6789/0 conn(0x7fe8f00c6bd0 legacy=0x7fe8f0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:49.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.843+0000 7fe8f6486640 1 -- 172.21.15.89:0/526199640 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f00c7390 con 0x7fe8f006f7a0 2023-10-30T12:48:49.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.843+0000 7fe8f6c87640 1 --2- 172.21.15.89:0/526199640 >> v2:172.21.15.192:3300/0 conn(0x7fe8f0071110 0x7fe8f0158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:48:49.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.843+0000 7fe8f5484640 1 -- 172.21.15.89:0/526199640 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2598024874 0 0) 0x7fe8f0066cf0 con 0x7fe8f00c6bd0 2023-10-30T12:48:49.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.843+0000 7fe8f6486640 1 --2- 172.21.15.89:0/526199640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8f006f7a0 0x7fe8f006fb80 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fe8f006cdd0 tx=0x7fe8ec01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8621848decb1a03 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:49.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.843+0000 7fe8f5484640 1 -- 172.21.15.89:0/526199640 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec0a8950 con 0x7fe8f006f7a0 2023-10-30T12:48:49.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.847+0000 7fe8f5484640 1 -- 172.21.15.89:0/526199640 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8ec0a8ab0 con 0x7fe8f006f7a0 2023-10-30T12:48:49.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.847+0000 7fe8f5484640 1 -- 172.21.15.89:0/526199640 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec0a8e40 con 0x7fe8f006f7a0 2023-10-30T12:48:49.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.847+0000 7fe8f86ea640 1 -- 172.21.15.89:0/526199640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8f006f7a0 msgr2=0x7fe8f006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:49.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.847+0000 7fe8f86ea640 1 --2- 172.21.15.89:0/526199640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8f006f7a0 0x7fe8f006fb80 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fe8f006cdd0 tx=0x7fe8ec01ec50 comp rx=0 tx=0).stop 2023-10-30T12:48:49.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.847+0000 7fe8f86ea640 1 -- 172.21.15.89:0/526199640 shutdown_connections 2023-10-30T12:48:49.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.847+0000 7fe8f86ea640 1 --2- 172.21.15.89:0/526199640 >> v2:172.21.15.192:3300/0 conn(0x7fe8f0071110 0x7fe8f0158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:49.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.847+0000 7fe8f86ea640 1 --2- 172.21.15.89:0/526199640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8f006f7a0 0x7fe8f006fb80 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:49.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.847+0000 7fe8f86ea640 1 -- 172.21.15.89:0/526199640 >> 172.21.15.89:0/526199640 conn(0x7fe8f00c7910 msgr2=0x7fe8f00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:49.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.847+0000 7fe8f86ea640 1 -- 172.21.15.89:0/526199640 shutdown_connections 2023-10-30T12:48:49.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.847+0000 7fe8f86ea640 1 -- 172.21.15.89:0/526199640 wait complete. 2023-10-30T12:48:49.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.847+0000 7fe8f86ea640 1 Processor -- start 2023-10-30T12:48:49.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.847+0000 7fe8f86ea640 1 -- start start 2023-10-30T12:48:49.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.847+0000 7fe8f86ea640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8f006f7a0 0x7fe8f00aa0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:49.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.847+0000 7fe8f86ea640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8f0071110 0x7fe8f01566b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:49.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.847+0000 7fe8f86ea640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8f00c6bd0 0x7fe8f014d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:49.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.847+0000 7fe8f86ea640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f01ed0d0 con 0x7fe8f00c6bd0 2023-10-30T12:48:49.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.847+0000 7fe8f86ea640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f00ab560 con 0x7fe8f0071110 2023-10-30T12:48:49.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.847+0000 7fe8f86ea640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f01ed600 con 0x7fe8f006f7a0 2023-10-30T12:48:49.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.847+0000 7fe8f6c87640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8f00c6bd0 0x7fe8f014d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:49.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.847+0000 7fe8f6c87640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8f00c6bd0 0x7fe8f014d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49392/0 (socket says 172.21.15.89:49392) 2023-10-30T12:48:49.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.847+0000 7fe8f6c87640 1 -- 172.21.15.89:0/3695094794 learned_addr learned my addr 172.21.15.89:0/3695094794 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:49.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.847+0000 7fe8f6486640 1 --2- 172.21.15.89:0/3695094794 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8f006f7a0 0x7fe8f00aa0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:49.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.847+0000 7fe8f5c85640 1 --2- 172.21.15.89:0/3695094794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8f0071110 0x7fe8f01566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:49.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.851+0000 7fe8f6c87640 1 -- 172.21.15.89:0/3695094794 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8f006f7a0 msgr2=0x7fe8f00aa0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:49.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.851+0000 7fe8f6c87640 1 --2- 172.21.15.89:0/3695094794 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8f006f7a0 0x7fe8f00aa0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:49.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.851+0000 7fe8f6c87640 1 -- 172.21.15.89:0/3695094794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8f0071110 msgr2=0x7fe8f01566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:49.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.851+0000 7fe8f6c87640 1 --2- 172.21.15.89:0/3695094794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8f0071110 0x7fe8f01566b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:49.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.851+0000 7fe8f6c87640 1 -- 172.21.15.89:0/3695094794 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8ec009420 con 0x7fe8f00c6bd0 2023-10-30T12:48:49.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.851+0000 7fe8f6486640 1 --2- 172.21.15.89:0/3695094794 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8f006f7a0 0x7fe8f00aa0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:49.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.851+0000 7fe8f6c87640 1 --2- 172.21.15.89:0/3695094794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8f00c6bd0 0x7fe8f014d570 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fe8e800cd50 tx=0x7fe8e800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:49.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.851+0000 7fe8e6ffd640 1 -- 172.21.15.89:0/3695094794 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e800f680 con 0x7fe8f00c6bd0 2023-10-30T12:48:49.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.851+0000 7fe8e6ffd640 1 -- 172.21.15.89:0/3695094794 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8e800f800 con 0x7fe8f00c6bd0 2023-10-30T12:48:49.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.851+0000 7fe8e6ffd640 1 -- 172.21.15.89:0/3695094794 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e800fbb0 con 0x7fe8f00c6bd0 2023-10-30T12:48:49.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.851+0000 7fe8f86ea640 1 -- 172.21.15.89:0/3695094794 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8f01ed320 con 0x7fe8f00c6bd0 2023-10-30T12:48:49.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.851+0000 7fe8f86ea640 1 -- 172.21.15.89:0/3695094794 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f00a98d0 con 0x7fe8f00c6bd0 2023-10-30T12:48:49.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.851+0000 7fe8f86ea640 1 -- 172.21.15.89:0/3695094794 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f004e170 con 0x7fe8f00c6bd0 2023-10-30T12:48:49.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.851+0000 7fe8e6ffd640 1 -- 172.21.15.89:0/3695094794 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe8e800b040 con 0x7fe8f00c6bd0 2023-10-30T12:48:49.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.851+0000 7fe8e6ffd640 1 --2- 172.21.15.89:0/3695094794 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe8d40664a0 0x7fe8d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:49.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.851+0000 7fe8e6ffd640 1 -- 172.21.15.89:0/3695094794 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6578+0+0 (secure 0 0 0) 0x7fe8e8084c10 con 0x7fe8f00c6bd0 2023-10-30T12:48:49.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.851+0000 7fe8f6486640 1 --2- 172.21.15.89:0/3695094794 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe8d40664a0 0x7fe8d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:49.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.855+0000 7fe8f6486640 1 --2- 172.21.15.89:0/3695094794 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe8d40664a0 0x7fe8d4068960 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec0b2000 tx=0x7fe8ec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:49.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:49.855+0000 7fe8e6ffd640 1 -- 172.21.15.89:0/3695094794 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe8e8051d10 con 0x7fe8f00c6bd0 2023-10-30T12:48:50.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:50.039+0000 7fe8f86ea640 1 -- 172.21.15.89:0/3695094794 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8f0070b50 con 0x7fe8f00c6bd0 2023-10-30T12:48:50.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:50.039+0000 7fe8e6ffd640 1 -- 172.21.15.89:0/3695094794 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+6538 (secure 0 0 0) 0x7fe8e801d020 con 0x7fe8f00c6bd0 2023-10-30T12:48:50.036 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:48:50.037 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5678,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:48:35.514221+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:50.037 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 97 2023-10-30T12:48:50.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:50.039+0000 7fe8f86ea640 1 -- 172.21.15.89:0/3695094794 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe8d40664a0 msgr2=0x7fe8d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:50.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:50.039+0000 7fe8f86ea640 1 --2- 172.21.15.89:0/3695094794 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe8d40664a0 0x7fe8d4068960 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec0b2000 tx=0x7fe8ec008040 comp rx=0 tx=0).stop 2023-10-30T12:48:50.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:50.039+0000 7fe8f86ea640 1 -- 172.21.15.89:0/3695094794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8f00c6bd0 msgr2=0x7fe8f014d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:50.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:50.039+0000 7fe8f86ea640 1 --2- 172.21.15.89:0/3695094794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8f00c6bd0 0x7fe8f014d570 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fe8e800cd50 tx=0x7fe8e800a650 comp rx=0 tx=0).stop 2023-10-30T12:48:50.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:50.039+0000 7fe8f86ea640 1 -- 172.21.15.89:0/3695094794 shutdown_connections 2023-10-30T12:48:50.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:50.039+0000 7fe8f86ea640 1 --2- 172.21.15.89:0/3695094794 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe8d40664a0 0x7fe8d4068960 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:50.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:50.039+0000 7fe8f86ea640 1 --2- 172.21.15.89:0/3695094794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8f0071110 0x7fe8f01566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:50.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:50.039+0000 7fe8f86ea640 1 --2- 172.21.15.89:0/3695094794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8f00c6bd0 0x7fe8f014d570 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:50.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:50.039+0000 7fe8f86ea640 1 --2- 172.21.15.89:0/3695094794 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8f006f7a0 0x7fe8f00aa0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:50.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:50.039+0000 7fe8f86ea640 1 -- 172.21.15.89:0/3695094794 >> 172.21.15.89:0/3695094794 conn(0x7fe8f00c7910 msgr2=0x7fe8f006a820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:50.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:50.043+0000 7fe8f86ea640 1 -- 172.21.15.89:0/3695094794 shutdown_connections 2023-10-30T12:48:50.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:50.043+0000 7fe8f86ea640 1 -- 172.21.15.89:0/3695094794 wait complete. 2023-10-30T12:48:50.057 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T12:48:52.058 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:52.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.211+0000 7f974ea6b640 1 Processor -- start 2023-10-30T12:48:52.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.211+0000 7f974ea6b640 1 -- start start 2023-10-30T12:48:52.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.211+0000 7f974ea6b640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f974815d6f0 0x7f974815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:52.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.211+0000 7f974ea6b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97480cf160 0x7f97480cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:52.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.211+0000 7f974ea6b640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97480cb800 con 0x7f97481538a0 2023-10-30T12:48:52.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.211+0000 7f974ea6b640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97480cb330 con 0x7f974815d6f0 2023-10-30T12:48:52.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.211+0000 7f974ea6b640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97480cad70 con 0x7f97480cf160 2023-10-30T12:48:52.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.211+0000 7f974c807640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f97481538a0 0x7f97480ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39860/0 (socket says 172.21.15.89:39860) 2023-10-30T12:48:52.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.211+0000 7f974c807640 1 -- 172.21.15.89:0/1061282545 learned_addr learned my addr 172.21.15.89:0/1061282545 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:48:52.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.211+0000 7f9747fff640 1 --2- 172.21.15.89:0/1061282545 >> v2:172.21.15.192:3300/0 conn(0x7f974815d6f0 0x7f974815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:52.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.211+0000 7f974d008640 1 --2- 172.21.15.89:0/1061282545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97480cf160 0x7f97480cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:52.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.211+0000 7f97477fe640 1 -- 172.21.15.89:0/1061282545 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618947514 0 0) 0x7f97480cb800 con 0x7f97481538a0 2023-10-30T12:48:52.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.211+0000 7f97477fe640 1 -- 172.21.15.89:0/1061282545 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9738003530 con 0x7f97481538a0 2023-10-30T12:48:52.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.211+0000 7f97477fe640 1 -- 172.21.15.89:0/1061282545 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1077774445 0 0) 0x7f9738003530 con 0x7f97481538a0 2023-10-30T12:48:52.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.211+0000 7f97477fe640 1 -- 172.21.15.89:0/1061282545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97480cf160 msgr2=0x7f97480cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:52.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.211+0000 7f97477fe640 1 --2- 172.21.15.89:0/1061282545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97480cf160 0x7f97480cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:52.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.211+0000 7f97477fe640 1 -- 172.21.15.89:0/1061282545 >> v2:172.21.15.192:3300/0 conn(0x7f974815d6f0 msgr2=0x7f974815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:52.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.211+0000 7f97477fe640 1 --2- 172.21.15.89:0/1061282545 >> v2:172.21.15.192:3300/0 conn(0x7f974815d6f0 0x7f974815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:52.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.211+0000 7f97477fe640 1 -- 172.21.15.89:0/1061282545 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97480cb050 con 0x7f97481538a0 2023-10-30T12:48:52.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.211+0000 7f97477fe640 1 -- 172.21.15.89:0/1061282545 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f973c001680 con 0x7f97481538a0 2023-10-30T12:48:52.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.211+0000 7f97477fe640 1 -- 172.21.15.89:0/1061282545 >> v1:172.21.15.89:6789/0 conn(0x7f97481538a0 legacy=0x7f97480ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:52.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.211+0000 7f97477fe640 1 --2- 172.21.15.89:0/1061282545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9738003980 0x7f9738003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:52.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.211+0000 7f97477fe640 1 --2- 172.21.15.89:0/1061282545 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97380047d0 0x7f9738004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:52.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.211+0000 7f97477fe640 1 --2- 172.21.15.89:0/1061282545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97480cf160 0x7f9738005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:52.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.211+0000 7f97477fe640 1 -- 172.21.15.89:0/1061282545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97480cb330 con 0x7f9738003980 2023-10-30T12:48:52.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.211+0000 7f97477fe640 1 -- 172.21.15.89:0/1061282545 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97480cad70 con 0x7f97380047d0 2023-10-30T12:48:52.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.211+0000 7f97477fe640 1 -- 172.21.15.89:0/1061282545 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f973800b990 con 0x7f97480cf160 2023-10-30T12:48:52.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.211+0000 7f974c807640 1 --2- 172.21.15.89:0/1061282545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9738003980 0x7f9738003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:52.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.211+0000 7f9747fff640 1 --2- 172.21.15.89:0/1061282545 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97380047d0 0x7f9738004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:52.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.211+0000 7f974d008640 1 --2- 172.21.15.89:0/1061282545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97480cf160 0x7f9738005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:52.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.215+0000 7f974c807640 1 -- 172.21.15.89:0/1061282545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97480cf160 msgr2=0x7f9738005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:52.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.215+0000 7f974c807640 1 --2- 172.21.15.89:0/1061282545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97480cf160 0x7f9738005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:52.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.215+0000 7f974c807640 1 -- 172.21.15.89:0/1061282545 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97380047d0 msgr2=0x7f9738004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:52.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.215+0000 7f974c807640 1 --2- 172.21.15.89:0/1061282545 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97380047d0 0x7f9738004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:52.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.215+0000 7f974c807640 1 -- 172.21.15.89:0/1061282545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9738003530 con 0x7f9738003980 2023-10-30T12:48:52.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.215+0000 7f974c807640 1 --2- 172.21.15.89:0/1061282545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9738003980 0x7f9738003da0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f973c00c210 tx=0x7f973c01f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa20c66273007bac server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:52.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.215+0000 7f97477fe640 1 -- 172.21.15.89:0/1061282545 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973c0a8ce0 con 0x7f9738003980 2023-10-30T12:48:52.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.215+0000 7f97477fe640 1 -- 172.21.15.89:0/1061282545 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973c0a8e40 con 0x7f9738003980 2023-10-30T12:48:52.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.215+0000 7f97477fe640 1 -- 172.21.15.89:0/1061282545 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973c005b00 con 0x7f9738003980 2023-10-30T12:48:52.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.215+0000 7f974ea6b640 1 -- 172.21.15.89:0/1061282545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9738003980 msgr2=0x7f9738003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:52.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.215+0000 7f974ea6b640 1 --2- 172.21.15.89:0/1061282545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9738003980 0x7f9738003da0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f973c00c210 tx=0x7f973c01f3b0 comp rx=0 tx=0).stop 2023-10-30T12:48:52.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.215+0000 7f974d008640 1 -- 172.21.15.89:0/1061282545 reap_dead start 2023-10-30T12:48:52.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.219+0000 7f974ea6b640 1 -- 172.21.15.89:0/1061282545 shutdown_connections 2023-10-30T12:48:52.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.219+0000 7f974ea6b640 1 -- 172.21.15.89:0/1061282545 >> 172.21.15.89:0/1061282545 conn(0x7f974806bfe0 msgr2=0x7f974806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:52.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.219+0000 7f974ea6b640 1 -- 172.21.15.89:0/1061282545 shutdown_connections 2023-10-30T12:48:52.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.219+0000 7f974ea6b640 1 -- 172.21.15.89:0/1061282545 wait complete. 2023-10-30T12:48:52.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.219+0000 7f974ea6b640 1 Processor -- start 2023-10-30T12:48:52.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.219+0000 7f974ea6b640 1 -- start start 2023-10-30T12:48:52.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.219+0000 7f974ea6b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9738003980 0x7f97481e3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:52.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.219+0000 7f974ea6b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97481e59a0 0x7f974815c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:52.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.219+0000 7f974ea6b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97481f28c0 0x7f9748158d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:52.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.219+0000 7f974ea6b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97480b48b0 con 0x7f97481e59a0 2023-10-30T12:48:52.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.219+0000 7f974ea6b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9748148420 con 0x7f97481f28c0 2023-10-30T12:48:52.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.219+0000 7f974ea6b640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97481e4e20 con 0x7f9738003980 2023-10-30T12:48:52.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.219+0000 7f9747fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97481e59a0 0x7f974815c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:52.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.219+0000 7f9747fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97481e59a0 0x7f974815c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49408/0 (socket says 172.21.15.89:49408) 2023-10-30T12:48:52.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.219+0000 7f9747fff640 1 -- 172.21.15.89:0/1715059113 learned_addr learned my addr 172.21.15.89:0/1715059113 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:52.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.219+0000 7f974c807640 1 --2- 172.21.15.89:0/1715059113 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9738003980 0x7f97481e3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:52.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.219+0000 7f974d008640 1 --2- 172.21.15.89:0/1715059113 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97481f28c0 0x7f9748158d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:52.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.219+0000 7f9747fff640 1 -- 172.21.15.89:0/1715059113 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9738003980 msgr2=0x7f97481e3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:52.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.219+0000 7f9747fff640 1 --2- 172.21.15.89:0/1715059113 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9738003980 0x7f97481e3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:52.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.219+0000 7f9747fff640 1 -- 172.21.15.89:0/1715059113 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97481f28c0 msgr2=0x7f9748158d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:52.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.219+0000 7f9747fff640 1 --2- 172.21.15.89:0/1715059113 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97481f28c0 0x7f9748158d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:52.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.219+0000 7f9747fff640 1 -- 172.21.15.89:0/1715059113 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9748157820 con 0x7f97481e59a0 2023-10-30T12:48:52.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.219+0000 7f974c807640 1 --2- 172.21.15.89:0/1715059113 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9738003980 0x7f97481e3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:52.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.219+0000 7f9747fff640 1 --2- 172.21.15.89:0/1715059113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97481e59a0 0x7f974815c150 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f9734011a00 tx=0x7f9734011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:52.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.223+0000 7f97457fa640 1 -- 172.21.15.89:0/1715059113 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973400fa90 con 0x7f97481e59a0 2023-10-30T12:48:52.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.223+0000 7f97457fa640 1 -- 172.21.15.89:0/1715059113 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973400fbf0 con 0x7f97481e59a0 2023-10-30T12:48:52.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.223+0000 7f97457fa640 1 -- 172.21.15.89:0/1715059113 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9734014440 con 0x7f97481e59a0 2023-10-30T12:48:52.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.223+0000 7f974ea6b640 1 -- 172.21.15.89:0/1715059113 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97480b32b0 con 0x7f97481e59a0 2023-10-30T12:48:52.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.223+0000 7f974ea6b640 1 -- 172.21.15.89:0/1715059113 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97481e4580 con 0x7f97481e59a0 2023-10-30T12:48:52.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.223+0000 7f974ea6b640 1 -- 172.21.15.89:0/1715059113 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97480b3110 con 0x7f97481e59a0 2023-10-30T12:48:52.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.223+0000 7f97457fa640 1 -- 172.21.15.89:0/1715059113 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f97340145a0 con 0x7f97481e59a0 2023-10-30T12:48:52.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.223+0000 7f97457fa640 1 --2- 172.21.15.89:0/1715059113 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f972c0664a0 0x7f972c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:52.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.223+0000 7f97457fa640 1 -- 172.21.15.89:0/1715059113 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6578+0+0 (secure 0 0 0) 0x7f9734084810 con 0x7f97481e59a0 2023-10-30T12:48:52.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.223+0000 7f974c807640 1 --2- 172.21.15.89:0/1715059113 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f972c0664a0 0x7f972c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:52.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.227+0000 7f974c807640 1 --2- 172.21.15.89:0/1715059113 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f972c0664a0 0x7f972c068960 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f973c0ad000 tx=0x7f973c001a10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:52.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.227+0000 7f97457fa640 1 -- 172.21.15.89:0/1715059113 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f97480b3110 con 0x7f97481e59a0 2023-10-30T12:48:52.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.415+0000 7f974ea6b640 1 -- 172.21.15.89:0/1715059113 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9748157820 con 0x7f97481e59a0 2023-10-30T12:48:52.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.415+0000 7f97457fa640 1 -- 172.21.15.89:0/1715059113 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f9748157820 con 0x7f97481e59a0 2023-10-30T12:48:52.413 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:48:52.413 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5678,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:48:35.514221+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:52.415 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 97 2023-10-30T12:48:52.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.415+0000 7f974ea6b640 1 -- 172.21.15.89:0/1715059113 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f972c0664a0 msgr2=0x7f972c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:52.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.415+0000 7f974ea6b640 1 --2- 172.21.15.89:0/1715059113 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f972c0664a0 0x7f972c068960 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f973c0ad000 tx=0x7f973c001a10 comp rx=0 tx=0).stop 2023-10-30T12:48:52.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.415+0000 7f974ea6b640 1 -- 172.21.15.89:0/1715059113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97481e59a0 msgr2=0x7f974815c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:52.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.415+0000 7f974ea6b640 1 --2- 172.21.15.89:0/1715059113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97481e59a0 0x7f974815c150 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f9734011a00 tx=0x7f9734011ec0 comp rx=0 tx=0).stop 2023-10-30T12:48:52.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.419+0000 7f974ea6b640 1 -- 172.21.15.89:0/1715059113 shutdown_connections 2023-10-30T12:48:52.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.419+0000 7f974ea6b640 1 --2- 172.21.15.89:0/1715059113 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f972c0664a0 0x7f972c068960 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:52.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.419+0000 7f974ea6b640 1 --2- 172.21.15.89:0/1715059113 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97481f28c0 0x7f9748158d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:52.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.419+0000 7f974ea6b640 1 --2- 172.21.15.89:0/1715059113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97481e59a0 0x7f974815c150 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:52.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.419+0000 7f974ea6b640 1 --2- 172.21.15.89:0/1715059113 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9738003980 0x7f97481e3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:52.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.419+0000 7f974ea6b640 1 -- 172.21.15.89:0/1715059113 >> 172.21.15.89:0/1715059113 conn(0x7f974806bfe0 msgr2=0x7f9748065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:52.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.419+0000 7f974ea6b640 1 -- 172.21.15.89:0/1715059113 shutdown_connections 2023-10-30T12:48:52.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:52.419+0000 7f974ea6b640 1 -- 172.21.15.89:0/1715059113 wait complete. 2023-10-30T12:48:52.434 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T12:48:54.435 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:54.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.587+0000 7f0548d72640 1 Processor -- start 2023-10-30T12:48:54.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.587+0000 7f0548d72640 1 -- start start 2023-10-30T12:48:54.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.587+0000 7f0548d72640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f05440c6b80 0x7f05440bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:54.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.587+0000 7f0548d72640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05440c0a50 con 0x7f054413f4a0 2023-10-30T12:48:54.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.587+0000 7f0548d72640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f054406f6e0 con 0x7f05440c6f60 2023-10-30T12:48:54.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.587+0000 7f0548d72640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f054406f9c0 con 0x7f05440c6b80 2023-10-30T12:48:54.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.591+0000 7f054259c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f05440c6b80 0x7f05440bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:54.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.591+0000 7f0542d9d640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f054413f4a0 0x7f05441566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39862/0 (socket says 172.21.15.89:39862) 2023-10-30T12:48:54.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.591+0000 7f0542d9d640 1 -- 172.21.15.89:0/1159959432 learned_addr learned my addr 172.21.15.89:0/1159959432 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:48:54.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.591+0000 7f054259c640 1 -- 172.21.15.89:0/1159959432 >> v1:172.21.15.192:6789/0 conn(0x7f05440c6f60 legacy=0x7f054413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:54.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.591+0000 7f054259c640 1 -- 172.21.15.89:0/1159959432 >> v1:172.21.15.89:6789/0 conn(0x7f054413f4a0 legacy=0x7f05441566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:54.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.591+0000 7f0541d9b640 1 -- 172.21.15.89:0/1159959432 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3897274661 0 0) 0x7f05440c0a50 con 0x7f054413f4a0 2023-10-30T12:48:54.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.591+0000 7f054259c640 1 -- 172.21.15.89:0/1159959432 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054406fca0 con 0x7f05440c6b80 2023-10-30T12:48:54.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.591+0000 7f054259c640 1 --2- 172.21.15.89:0/1159959432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05440c6b80 0x7f05440bfec0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f052c0097c0 tx=0x7f052c01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=176b7c044588807a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:54.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.591+0000 7f0541d9b640 1 -- 172.21.15.89:0/1159959432 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c004230 con 0x7f05440c6b80 2023-10-30T12:48:54.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.591+0000 7f0541d9b640 1 -- 172.21.15.89:0/1159959432 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f052c0a7050 con 0x7f05440c6b80 2023-10-30T12:48:54.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.591+0000 7f0541d9b640 1 -- 172.21.15.89:0/1159959432 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c003620 con 0x7f05440c6b80 2023-10-30T12:48:54.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.595+0000 7f0548d72640 1 -- 172.21.15.89:0/1159959432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05440c6b80 msgr2=0x7f05440bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:54.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.595+0000 7f0548d72640 1 --2- 172.21.15.89:0/1159959432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05440c6b80 0x7f05440bfec0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f052c0097c0 tx=0x7f052c01c440 comp rx=0 tx=0).stop 2023-10-30T12:48:54.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.595+0000 7f0548d72640 1 -- 172.21.15.89:0/1159959432 shutdown_connections 2023-10-30T12:48:54.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.595+0000 7f0548d72640 1 --2- 172.21.15.89:0/1159959432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05440c6b80 0x7f05440bfec0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:54.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.595+0000 7f0548d72640 1 -- 172.21.15.89:0/1159959432 >> 172.21.15.89:0/1159959432 conn(0x7f05440c7910 msgr2=0x7f05440c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:54.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.595+0000 7f0548d72640 1 -- 172.21.15.89:0/1159959432 shutdown_connections 2023-10-30T12:48:54.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.595+0000 7f0548d72640 1 -- 172.21.15.89:0/1159959432 wait complete. 2023-10-30T12:48:54.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.595+0000 7f0548d72640 1 Processor -- start 2023-10-30T12:48:54.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.595+0000 7f0548d72640 1 -- start start 2023-10-30T12:48:54.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.595+0000 7f0548d72640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05440c6b80 0x7f05441f0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:54.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.595+0000 7f0548d72640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05440c6f60 0x7f0544134690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:54.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.595+0000 7f0548d72640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f054413f4a0 0x7f0544154a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:54.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.595+0000 7f0548d72640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054413c700 con 0x7f05440c6b80 2023-10-30T12:48:54.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.595+0000 7f0548d72640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054413e8b0 con 0x7f05440c6f60 2023-10-30T12:48:54.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.595+0000 7f0548d72640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f054413d570 con 0x7f054413f4a0 2023-10-30T12:48:54.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.595+0000 7f054259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05440c6b80 0x7f05441f0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:54.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.595+0000 7f054259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05440c6b80 0x7f05441f0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49432/0 (socket says 172.21.15.89:49432) 2023-10-30T12:48:54.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.595+0000 7f054259c640 1 -- 172.21.15.89:0/177622176 learned_addr learned my addr 172.21.15.89:0/177622176 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:54.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.595+0000 7f0542d9d640 1 --2- 172.21.15.89:0/177622176 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f054413f4a0 0x7f0544154a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:54.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.595+0000 7f0539d9b640 1 --2- 172.21.15.89:0/177622176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05440c6f60 0x7f0544134690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:54.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.599+0000 7f054259c640 1 -- 172.21.15.89:0/177622176 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f054413f4a0 msgr2=0x7f0544154a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:54.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.599+0000 7f054259c640 1 --2- 172.21.15.89:0/177622176 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f054413f4a0 0x7f0544154a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:54.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.599+0000 7f054259c640 1 -- 172.21.15.89:0/177622176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05440c6f60 msgr2=0x7f0544134690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:54.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.599+0000 7f054259c640 1 --2- 172.21.15.89:0/177622176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05440c6f60 0x7f0544134690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:54.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.599+0000 7f054259c640 1 -- 172.21.15.89:0/177622176 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f052c009420 con 0x7f05440c6b80 2023-10-30T12:48:54.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.599+0000 7f0542d9d640 1 --2- 172.21.15.89:0/177622176 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f054413f4a0 0x7f0544154a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:54.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.599+0000 7f054259c640 1 --2- 172.21.15.89:0/177622176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05440c6b80 0x7f05441f0740 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f052c0ae040 tx=0x7f052c0191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:54.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.599+0000 7f053b7fe640 1 -- 172.21.15.89:0/177622176 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c003cf0 con 0x7f05440c6b80 2023-10-30T12:48:54.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.599+0000 7f053b7fe640 1 -- 172.21.15.89:0/177622176 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f052c003e50 con 0x7f05440c6b80 2023-10-30T12:48:54.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.599+0000 7f053b7fe640 1 -- 172.21.15.89:0/177622176 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c0ab660 con 0x7f05440c6b80 2023-10-30T12:48:54.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.599+0000 7f0548d72640 1 -- 172.21.15.89:0/177622176 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f054413d830 con 0x7f05440c6b80 2023-10-30T12:48:54.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.599+0000 7f0548d72640 1 -- 172.21.15.89:0/177622176 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f054413ddd0 con 0x7f05440c6b80 2023-10-30T12:48:54.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.599+0000 7f0548d72640 1 -- 172.21.15.89:0/177622176 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f054404e170 con 0x7f05440c6b80 2023-10-30T12:48:54.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.599+0000 7f053b7fe640 1 -- 172.21.15.89:0/177622176 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f052c0b2040 con 0x7f05440c6b80 2023-10-30T12:48:54.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.599+0000 7f053b7fe640 1 --2- 172.21.15.89:0/177622176 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f05240664a0 0x7f0524068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:54.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.599+0000 7f053b7fe640 1 -- 172.21.15.89:0/177622176 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6578+0+0 (secure 0 0 0) 0x7f052c11e320 con 0x7f05440c6b80 2023-10-30T12:48:54.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.603+0000 7f0539d9b640 1 --2- 172.21.15.89:0/177622176 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f05240664a0 0x7f0524068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:54.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.603+0000 7f0539d9b640 1 --2- 172.21.15.89:0/177622176 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f05240664a0 0x7f0524068960 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f05441f42f0 tx=0x7f0534009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:54.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.603+0000 7f053b7fe640 1 -- 172.21.15.89:0/177622176 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f052c0eb420 con 0x7f05440c6b80 2023-10-30T12:48:54.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.787+0000 7f0548d72640 1 -- 172.21.15.89:0/177622176 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05440705c0 con 0x7f05440c6b80 2023-10-30T12:48:54.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.787+0000 7f053b7fe640 1 -- 172.21.15.89:0/177622176 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f052c0eea40 con 0x7f05440c6b80 2023-10-30T12:48:54.783 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:48:54.784 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5678,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:48:35.514221+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:54.784 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 97 2023-10-30T12:48:54.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.787+0000 7f0548d72640 1 -- 172.21.15.89:0/177622176 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f05240664a0 msgr2=0x7f0524068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:54.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.787+0000 7f0548d72640 1 --2- 172.21.15.89:0/177622176 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f05240664a0 0x7f0524068960 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f05441f42f0 tx=0x7f0534009040 comp rx=0 tx=0).stop 2023-10-30T12:48:54.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.787+0000 7f0548d72640 1 -- 172.21.15.89:0/177622176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05440c6b80 msgr2=0x7f05441f0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:54.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.787+0000 7f0548d72640 1 --2- 172.21.15.89:0/177622176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05440c6b80 0x7f05441f0740 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f052c0ae040 tx=0x7f052c0191e0 comp rx=0 tx=0).stop 2023-10-30T12:48:54.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.787+0000 7f0548d72640 1 -- 172.21.15.89:0/177622176 shutdown_connections 2023-10-30T12:48:54.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.787+0000 7f0548d72640 1 --2- 172.21.15.89:0/177622176 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f05240664a0 0x7f0524068960 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:54.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.787+0000 7f0548d72640 1 --2- 172.21.15.89:0/177622176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05440c6f60 0x7f0544134690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:54.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.787+0000 7f0548d72640 1 --2- 172.21.15.89:0/177622176 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f054413f4a0 0x7f0544154a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:54.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.787+0000 7f0548d72640 1 --2- 172.21.15.89:0/177622176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05440c6b80 0x7f05441f0740 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:54.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.787+0000 7f0548d72640 1 -- 172.21.15.89:0/177622176 >> 172.21.15.89:0/177622176 conn(0x7f05440c7910 msgr2=0x7f05440c1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:54.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.787+0000 7f0548d72640 1 -- 172.21.15.89:0/177622176 shutdown_connections 2023-10-30T12:48:54.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:54.787+0000 7f0548d72640 1 -- 172.21.15.89:0/177622176 wait complete. 2023-10-30T12:48:54.804 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T12:48:56.805 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:56.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.954+0000 7f57f9291640 1 Processor -- start 2023-10-30T12:48:56.954 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.958+0000 7f57f9291640 1 -- start start 2023-10-30T12:48:56.955 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.958+0000 7f57f9291640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f57f406f7a0 0x7f57f406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:56.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.958+0000 7f57f9291640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57f4066cf0 con 0x7f57f4071110 2023-10-30T12:48:56.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.958+0000 7f57f9291640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57f40c70d0 con 0x7f57f40c6bd0 2023-10-30T12:48:56.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.958+0000 7f57f9291640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57f40700c0 con 0x7f57f406f7a0 2023-10-30T12:48:56.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.958+0000 7f57f37fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f57f4071110 0x7f57f4158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39878/0 (socket says 172.21.15.89:39878) 2023-10-30T12:48:56.956 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.958+0000 7f57f37fe640 1 -- 172.21.15.89:0/1097736838 learned_addr learned my addr 172.21.15.89:0/1097736838 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:48:56.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.958+0000 7f57f2ffd640 1 --2- 172.21.15.89:0/1097736838 >> v2:172.21.15.192:3300/0 conn(0x7f57f406f7a0 0x7f57f406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:56.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.958+0000 7f57f27fc640 1 -- 172.21.15.89:0/1097736838 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3390886985 0 0) 0x7f57f4066cf0 con 0x7f57f4071110 2023-10-30T12:48:56.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.958+0000 7f57f27fc640 1 -- 172.21.15.89:0/1097736838 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57e0003530 con 0x7f57f4071110 2023-10-30T12:48:56.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.958+0000 7f57f27fc640 1 -- 172.21.15.89:0/1097736838 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2869243670 0 0) 0x7f57e0003530 con 0x7f57f4071110 2023-10-30T12:48:56.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.958+0000 7f57f2ffd640 1 -- 172.21.15.89:0/1097736838 >> v1:172.21.15.192:6789/0 conn(0x7f57f40c6bd0 legacy=0x7f57f4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:56.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.958+0000 7f57f2ffd640 1 -- 172.21.15.89:0/1097736838 >> v1:172.21.15.89:6789/0 conn(0x7f57f4071110 legacy=0x7f57f4158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:56.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.958+0000 7f57f2ffd640 1 -- 172.21.15.89:0/1097736838 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f40c7390 con 0x7f57f406f7a0 2023-10-30T12:48:56.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.962+0000 7f57f2ffd640 1 --2- 172.21.15.89:0/1097736838 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57f406f7a0 0x7f57f406fb80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f57e40097c0 tx=0x7f57e401eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e1e5c2bfdf2f254 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:56.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.962+0000 7f57f27fc640 1 -- 172.21.15.89:0/1097736838 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ec001a00 con 0x7f57f406f7a0 2023-10-30T12:48:56.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.962+0000 7f57f27fc640 1 -- 172.21.15.89:0/1097736838 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57e40a8610 con 0x7f57f406f7a0 2023-10-30T12:48:56.962 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.962+0000 7f57f27fc640 1 -- 172.21.15.89:0/1097736838 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e40a89a0 con 0x7f57f406f7a0 2023-10-30T12:48:56.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.962+0000 7f57f9291640 1 -- 172.21.15.89:0/1097736838 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57f406f7a0 msgr2=0x7f57f406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:56.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.962+0000 7f57f9291640 1 --2- 172.21.15.89:0/1097736838 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57f406f7a0 0x7f57f406fb80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f57e40097c0 tx=0x7f57e401eb90 comp rx=0 tx=0).stop 2023-10-30T12:48:56.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.962+0000 7f57f9291640 1 -- 172.21.15.89:0/1097736838 shutdown_connections 2023-10-30T12:48:56.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.962+0000 7f57f9291640 1 --2- 172.21.15.89:0/1097736838 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57f406f7a0 0x7f57f406fb80 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:56.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.962+0000 7f57f9291640 1 -- 172.21.15.89:0/1097736838 >> 172.21.15.89:0/1097736838 conn(0x7f57f40c7910 msgr2=0x7f57f40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:56.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.962+0000 7f57f9291640 1 -- 172.21.15.89:0/1097736838 shutdown_connections 2023-10-30T12:48:56.964 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.962+0000 7f57f9291640 1 -- 172.21.15.89:0/1097736838 wait complete. 2023-10-30T12:48:56.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.966+0000 7f57f9291640 1 Processor -- start 2023-10-30T12:48:56.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.966+0000 7f57f9291640 1 -- start start 2023-10-30T12:48:56.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.966+0000 7f57f9291640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57f406f7a0 0x7f57f40c0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:56.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.966+0000 7f57f9291640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57f4071110 0x7f57f414d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:56.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.966+0000 7f57f9291640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57f40c6bd0 0x7f57f40c54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:56.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.966+0000 7f57f9291640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f40c1b20 con 0x7f57f4071110 2023-10-30T12:48:56.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.966+0000 7f57f9291640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f40c14b0 con 0x7f57f40c6bd0 2023-10-30T12:48:56.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.966+0000 7f57f9291640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57f41efdb0 con 0x7f57f406f7a0 2023-10-30T12:48:56.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.966+0000 7f57ebfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57f4071110 0x7f57f414d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:56.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.966+0000 7f57ebfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57f4071110 0x7f57f414d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49442/0 (socket says 172.21.15.89:49442) 2023-10-30T12:48:56.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.966+0000 7f57ebfff640 1 -- 172.21.15.89:0/1302005963 learned_addr learned my addr 172.21.15.89:0/1302005963 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:56.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.966+0000 7f57f2ffd640 1 --2- 172.21.15.89:0/1302005963 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57f406f7a0 0x7f57f40c0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:56.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.966+0000 7f57f37fe640 1 --2- 172.21.15.89:0/1302005963 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57f40c6bd0 0x7f57f40c54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:56.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.966+0000 7f57ebfff640 1 -- 172.21.15.89:0/1302005963 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57f406f7a0 msgr2=0x7f57f40c0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:56.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.966+0000 7f57ebfff640 1 --2- 172.21.15.89:0/1302005963 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57f406f7a0 0x7f57f40c0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:56.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.966+0000 7f57ebfff640 1 -- 172.21.15.89:0/1302005963 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57f40c6bd0 msgr2=0x7f57f40c54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:56.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.966+0000 7f57ebfff640 1 --2- 172.21.15.89:0/1302005963 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57f40c6bd0 0x7f57f40c54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:56.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.966+0000 7f57ebfff640 1 -- 172.21.15.89:0/1302005963 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e4009420 con 0x7f57f4071110 2023-10-30T12:48:56.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.966+0000 7f57f2ffd640 1 --2- 172.21.15.89:0/1302005963 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57f406f7a0 0x7f57f40c0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:56.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.966+0000 7f57ebfff640 1 --2- 172.21.15.89:0/1302005963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57f4071110 0x7f57f414d1c0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00c4d0 tx=0x7f57dc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:56.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.966+0000 7f57eb7fe640 1 -- 172.21.15.89:0/1302005963 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc0052d0 con 0x7f57f4071110 2023-10-30T12:48:56.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.966+0000 7f57eb7fe640 1 -- 172.21.15.89:0/1302005963 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57dc005430 con 0x7f57f4071110 2023-10-30T12:48:56.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.966+0000 7f57eb7fe640 1 -- 172.21.15.89:0/1302005963 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc012830 con 0x7f57f4071110 2023-10-30T12:48:56.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.966+0000 7f57f9291640 1 -- 172.21.15.89:0/1302005963 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57f414aac0 con 0x7f57f4071110 2023-10-30T12:48:56.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.966+0000 7f57f9291640 1 -- 172.21.15.89:0/1302005963 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57f40c0ef0 con 0x7f57f4071110 2023-10-30T12:48:56.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.966+0000 7f57f9291640 1 -- 172.21.15.89:0/1302005963 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57f406fb80 con 0x7f57f4071110 2023-10-30T12:48:56.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.970+0000 7f57eb7fe640 1 -- 172.21.15.89:0/1302005963 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f57dc012990 con 0x7f57f4071110 2023-10-30T12:48:56.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.970+0000 7f57eb7fe640 1 --2- 172.21.15.89:0/1302005963 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f57d40664a0 0x7f57d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:56.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.970+0000 7f57eb7fe640 1 -- 172.21.15.89:0/1302005963 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6578+0+0 (secure 0 0 0) 0x7f57dc083de0 con 0x7f57f4071110 2023-10-30T12:48:56.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.970+0000 7f57f2ffd640 1 --2- 172.21.15.89:0/1302005963 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f57d40664a0 0x7f57d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:56.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.970+0000 7f57f2ffd640 1 --2- 172.21.15.89:0/1302005963 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f57d40664a0 0x7f57d4068960 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f57e40b2000 tx=0x7f57e4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:56.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:56.974+0000 7f57eb7fe640 1 -- 172.21.15.89:0/1302005963 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f57dc050dc0 con 0x7f57f4071110 2023-10-30T12:48:57.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:57.154+0000 7f57f9291640 1 -- 172.21.15.89:0/1302005963 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57f4070600 con 0x7f57f4071110 2023-10-30T12:48:57.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:57.154+0000 7f57eb7fe640 1 -- 172.21.15.89:0/1302005963 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f57dc0543e0 con 0x7f57f4071110 2023-10-30T12:48:57.152 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:48:57.152 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5678,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:48:35.514221+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:57.152 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 97 2023-10-30T12:48:57.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:57.154+0000 7f57f9291640 1 -- 172.21.15.89:0/1302005963 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f57d40664a0 msgr2=0x7f57d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:57.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:57.154+0000 7f57f9291640 1 --2- 172.21.15.89:0/1302005963 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f57d40664a0 0x7f57d4068960 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f57e40b2000 tx=0x7f57e4008040 comp rx=0 tx=0).stop 2023-10-30T12:48:57.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:57.154+0000 7f57f9291640 1 -- 172.21.15.89:0/1302005963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57f4071110 msgr2=0x7f57f414d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:57.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:57.154+0000 7f57f9291640 1 --2- 172.21.15.89:0/1302005963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57f4071110 0x7f57f414d1c0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00c4d0 tx=0x7f57dc00c990 comp rx=0 tx=0).stop 2023-10-30T12:48:57.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:57.154+0000 7f57f9291640 1 -- 172.21.15.89:0/1302005963 shutdown_connections 2023-10-30T12:48:57.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:57.154+0000 7f57f9291640 1 --2- 172.21.15.89:0/1302005963 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f57d40664a0 0x7f57d4068960 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:57.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:57.154+0000 7f57f9291640 1 --2- 172.21.15.89:0/1302005963 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57f40c6bd0 0x7f57f40c54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:57.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:57.154+0000 7f57f9291640 1 --2- 172.21.15.89:0/1302005963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57f4071110 0x7f57f414d1c0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:57.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:57.154+0000 7f57f9291640 1 --2- 172.21.15.89:0/1302005963 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57f406f7a0 0x7f57f40c0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:57.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:57.154+0000 7f57f9291640 1 -- 172.21.15.89:0/1302005963 >> 172.21.15.89:0/1302005963 conn(0x7f57f40c7910 msgr2=0x7f57f40c3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:57.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:57.154+0000 7f57f9291640 1 -- 172.21.15.89:0/1302005963 shutdown_connections 2023-10-30T12:48:57.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:57.154+0000 7f57f9291640 1 -- 172.21.15.89:0/1302005963 wait complete. 2023-10-30T12:48:57.172 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T12:48:59.173 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:59.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.318+0000 7fb552a6b640 1 Processor -- start 2023-10-30T12:48:59.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.318+0000 7fb552a6b640 1 -- start start 2023-10-30T12:48:59.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.322+0000 7fb552a6b640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb54c069e90 0x7fb54c06a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:59.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.322+0000 7fb552a6b640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb54c153d80 con 0x7fb54c1539a0 2023-10-30T12:48:59.319 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.322+0000 7fb552a6b640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb54c0c5c50 con 0x7fb54c070a00 2023-10-30T12:48:59.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.322+0000 7fb552a6b640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb54c06fbb0 con 0x7fb54c069e90 2023-10-30T12:48:59.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.322+0000 7fb550807640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb54c1539a0 0x7fb54c15a380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39892/0 (socket says 172.21.15.89:39892) 2023-10-30T12:48:59.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.322+0000 7fb550807640 1 -- 172.21.15.89:0/3408191459 learned_addr learned my addr 172.21.15.89:0/3408191459 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:48:59.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.322+0000 7fb551008640 1 --2- 172.21.15.89:0/3408191459 >> v2:172.21.15.89:3300/0 conn(0x7fb54c069e90 0x7fb54c06a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:59.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.322+0000 7fb5437fe640 1 -- 172.21.15.89:0/3408191459 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2870401692 0 0) 0x7fb54c153d80 con 0x7fb54c1539a0 2023-10-30T12:48:59.321 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.322+0000 7fb5437fe640 1 -- 172.21.15.89:0/3408191459 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb534003530 con 0x7fb54c1539a0 2023-10-30T12:48:59.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.322+0000 7fb551008640 1 -- 172.21.15.89:0/3408191459 >> v1:172.21.15.192:6789/0 conn(0x7fb54c070a00 legacy=0x7fb54c070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:59.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.322+0000 7fb551008640 1 -- 172.21.15.89:0/3408191459 >> v1:172.21.15.89:6789/0 conn(0x7fb54c1539a0 legacy=0x7fb54c15a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:59.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.322+0000 7fb551008640 1 -- 172.21.15.89:0/3408191459 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb54c0c5f30 con 0x7fb54c069e90 2023-10-30T12:48:59.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.326+0000 7fb551008640 1 --2- 172.21.15.89:0/3408191459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb54c069e90 0x7fb54c06a300 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fb544009d90 tx=0x7fb54401c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8756fc43223d8abc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:59.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.326+0000 7fb5437fe640 1 -- 172.21.15.89:0/3408191459 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5440a5860 con 0x7fb54c069e90 2023-10-30T12:48:59.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.326+0000 7fb5437fe640 1 -- 172.21.15.89:0/3408191459 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5440a59c0 con 0x7fb54c069e90 2023-10-30T12:48:59.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.326+0000 7fb5437fe640 1 -- 172.21.15.89:0/3408191459 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5440a5d50 con 0x7fb54c069e90 2023-10-30T12:48:59.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.326+0000 7fb552a6b640 1 -- 172.21.15.89:0/3408191459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb54c069e90 msgr2=0x7fb54c06a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:59.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.326+0000 7fb552a6b640 1 --2- 172.21.15.89:0/3408191459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb54c069e90 0x7fb54c06a300 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fb544009d90 tx=0x7fb54401c890 comp rx=0 tx=0).stop 2023-10-30T12:48:59.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.326+0000 7fb552a6b640 1 -- 172.21.15.89:0/3408191459 shutdown_connections 2023-10-30T12:48:59.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.326+0000 7fb552a6b640 1 --2- 172.21.15.89:0/3408191459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb54c069e90 0x7fb54c06a300 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:59.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.326+0000 7fb552a6b640 1 -- 172.21.15.89:0/3408191459 >> 172.21.15.89:0/3408191459 conn(0x7fb54c0c7910 msgr2=0x7fb54c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:59.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.326+0000 7fb552a6b640 1 -- 172.21.15.89:0/3408191459 shutdown_connections 2023-10-30T12:48:59.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.326+0000 7fb552a6b640 1 -- 172.21.15.89:0/3408191459 wait complete. 2023-10-30T12:48:59.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.326+0000 7fb552a6b640 1 Processor -- start 2023-10-30T12:48:59.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.326+0000 7fb552a6b640 1 -- start start 2023-10-30T12:48:59.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.330+0000 7fb552a6b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb54c069e90 0x7fb54c144880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:59.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.330+0000 7fb552a6b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb54c070a00 0x7fb54c146fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:59.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.330+0000 7fb552a6b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb54c1539a0 0x7fb54c1474f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:59.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.330+0000 7fb552a6b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c1464d0 con 0x7fb54c070a00 2023-10-30T12:48:59.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.330+0000 7fb552a6b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c0c1bf0 con 0x7fb54c1539a0 2023-10-30T12:48:59.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.330+0000 7fb552a6b640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c0c1690 con 0x7fb54c069e90 2023-10-30T12:48:59.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.330+0000 7fb543fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb54c070a00 0x7fb54c146fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:59.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.330+0000 7fb543fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb54c070a00 0x7fb54c146fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49450/0 (socket says 172.21.15.89:49450) 2023-10-30T12:48:59.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.330+0000 7fb543fff640 1 -- 172.21.15.89:0/2509113103 learned_addr learned my addr 172.21.15.89:0/2509113103 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:48:59.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.330+0000 7fb551008640 1 --2- 172.21.15.89:0/2509113103 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb54c1539a0 0x7fb54c1474f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:59.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.330+0000 7fb550807640 1 --2- 172.21.15.89:0/2509113103 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb54c069e90 0x7fb54c144880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:59.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.330+0000 7fb543fff640 1 -- 172.21.15.89:0/2509113103 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb54c069e90 msgr2=0x7fb54c144880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:59.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.330+0000 7fb543fff640 1 --2- 172.21.15.89:0/2509113103 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb54c069e90 0x7fb54c144880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:59.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.330+0000 7fb543fff640 1 -- 172.21.15.89:0/2509113103 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb54c1539a0 msgr2=0x7fb54c1474f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:59.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.330+0000 7fb543fff640 1 --2- 172.21.15.89:0/2509113103 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb54c1539a0 0x7fb54c1474f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:59.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.330+0000 7fb543fff640 1 -- 172.21.15.89:0/2509113103 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb544009a70 con 0x7fb54c070a00 2023-10-30T12:48:59.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.330+0000 7fb550807640 1 --2- 172.21.15.89:0/2509113103 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb54c069e90 0x7fb54c144880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:59.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.330+0000 7fb551008640 1 --2- 172.21.15.89:0/2509113103 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb54c1539a0 0x7fb54c1474f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:59.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.330+0000 7fb543fff640 1 --2- 172.21.15.89:0/2509113103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb54c070a00 0x7fb54c146fb0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fb53c00c500 tx=0x7fb53c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:59.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.330+0000 7fb5417fa640 1 -- 172.21.15.89:0/2509113103 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53c005220 con 0x7fb54c070a00 2023-10-30T12:48:59.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.330+0000 7fb5417fa640 1 -- 172.21.15.89:0/2509113103 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb53c005380 con 0x7fb54c070a00 2023-10-30T12:48:59.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.330+0000 7fb5417fa640 1 -- 172.21.15.89:0/2509113103 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53c007270 con 0x7fb54c070a00 2023-10-30T12:48:59.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.330+0000 7fb552a6b640 1 -- 172.21.15.89:0/2509113103 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb54c1450f0 con 0x7fb54c070a00 2023-10-30T12:48:59.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.330+0000 7fb552a6b640 1 -- 172.21.15.89:0/2509113103 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb54c0c1930 con 0x7fb54c070a00 2023-10-30T12:48:59.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.330+0000 7fb552a6b640 1 -- 172.21.15.89:0/2509113103 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb54c146640 con 0x7fb54c070a00 2023-10-30T12:48:59.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.330+0000 7fb5417fa640 1 -- 172.21.15.89:0/2509113103 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb53c0073d0 con 0x7fb54c070a00 2023-10-30T12:48:59.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.334+0000 7fb5417fa640 1 --2- 172.21.15.89:0/2509113103 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb5380664a0 0x7fb538068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:59.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.334+0000 7fb5417fa640 1 -- 172.21.15.89:0/2509113103 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6578+0+0 (secure 0 0 0) 0x7fb53c083bf0 con 0x7fb54c070a00 2023-10-30T12:48:59.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.334+0000 7fb550807640 1 --2- 172.21.15.89:0/2509113103 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb5380664a0 0x7fb538068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:59.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.334+0000 7fb550807640 1 --2- 172.21.15.89:0/2509113103 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb5380664a0 0x7fb538068960 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb5480011f0 tx=0x7fb548009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:59.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.338+0000 7fb5417fa640 1 -- 172.21.15.89:0/2509113103 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb53c050c70 con 0x7fb54c070a00 2023-10-30T12:48:59.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.518+0000 7fb552a6b640 1 -- 172.21.15.89:0/2509113103 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb54c0c67a0 con 0x7fb54c070a00 2023-10-30T12:48:59.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.522+0000 7fb5417fa640 1 -- 172.21.15.89:0/2509113103 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+6538 (secure 0 0 0) 0x7fb53c054290 con 0x7fb54c070a00 2023-10-30T12:48:59.519 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:48:59.519 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5678,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:48:35.514221+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5575,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5575":{"gid":5575,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":37,"addr":"172.21.15.192:6835/501291700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":501291700},{"type":"v1","addr":"172.21.15.192:6835","nonce":501291700}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:59.519 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 97 2023-10-30T12:48:59.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.522+0000 7fb552a6b640 1 -- 172.21.15.89:0/2509113103 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb5380664a0 msgr2=0x7fb538068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:59.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.522+0000 7fb552a6b640 1 --2- 172.21.15.89:0/2509113103 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb5380664a0 0x7fb538068960 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb5480011f0 tx=0x7fb548009040 comp rx=0 tx=0).stop 2023-10-30T12:48:59.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.522+0000 7fb552a6b640 1 -- 172.21.15.89:0/2509113103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb54c070a00 msgr2=0x7fb54c146fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:59.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.522+0000 7fb552a6b640 1 --2- 172.21.15.89:0/2509113103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb54c070a00 0x7fb54c146fb0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fb53c00c500 tx=0x7fb53c00c9c0 comp rx=0 tx=0).stop 2023-10-30T12:48:59.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.522+0000 7fb552a6b640 1 -- 172.21.15.89:0/2509113103 shutdown_connections 2023-10-30T12:48:59.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.522+0000 7fb552a6b640 1 --2- 172.21.15.89:0/2509113103 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb5380664a0 0x7fb538068960 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:59.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.522+0000 7fb552a6b640 1 --2- 172.21.15.89:0/2509113103 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb54c1539a0 0x7fb54c1474f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:59.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.522+0000 7fb552a6b640 1 --2- 172.21.15.89:0/2509113103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb54c070a00 0x7fb54c146fb0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:59.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.522+0000 7fb552a6b640 1 --2- 172.21.15.89:0/2509113103 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb54c069e90 0x7fb54c144880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:59.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.522+0000 7fb552a6b640 1 -- 172.21.15.89:0/2509113103 >> 172.21.15.89:0/2509113103 conn(0x7fb54c0c7910 msgr2=0x7fb54c14e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:59.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.522+0000 7fb552a6b640 1 -- 172.21.15.89:0/2509113103 shutdown_connections 2023-10-30T12:48:59.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:48:59.522+0000 7fb552a6b640 1 -- 172.21.15.89:0/2509113103 wait complete. 2023-10-30T12:48:59.539 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-30T12:49:01.541 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:01.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.690+0000 7fc50cf72640 1 Processor -- start 2023-10-30T12:49:01.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.690+0000 7fc50cf72640 1 -- start start 2023-10-30T12:49:01.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.690+0000 7fc50cf72640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5080c6b00 0x7fc508065da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:01.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.690+0000 7fc50cf72640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc5080c5f60 0x7fc5080cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:01.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.690+0000 7fc50cf72640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc508151d00 con 0x7fc508151920 2023-10-30T12:49:01.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.690+0000 7fc50cf72640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5080c4830 con 0x7fc5080c5f60 2023-10-30T12:49:01.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.690+0000 7fc50cf72640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5080c6790 con 0x7fc5080c6b00 2023-10-30T12:49:01.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.694+0000 7fc506d9d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc5080c5f60 0x7fc5080cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:01.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.694+0000 7fc50659c640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc508151920 0x7fc508157e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51046/0 (socket says 172.21.15.89:51046) 2023-10-30T12:49:01.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.694+0000 7fc50659c640 1 -- 172.21.15.89:0/2446053941 learned_addr learned my addr 172.21.15.89:0/2446053941 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:49:01.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.694+0000 7fc4fdd9b640 1 --2- 172.21.15.89:0/2446053941 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5080c6b00 0x7fc508065da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:01.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.694+0000 7fc506d9d640 1 -- 172.21.15.89:0/2446053941 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5080c6b00 msgr2=0x7fc508065da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:01.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.694+0000 7fc506d9d640 1 --2- 172.21.15.89:0/2446053941 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5080c6b00 0x7fc508065da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:01.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.694+0000 7fc506d9d640 1 -- 172.21.15.89:0/2446053941 >> v1:172.21.15.89:6789/0 conn(0x7fc508151920 legacy=0x7fc508157e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:01.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.694+0000 7fc505d9b640 1 -- 172.21.15.89:0/2446053941 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3010599247 0 0) 0x7fc508151d00 con 0x7fc508151920 2023-10-30T12:49:01.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.694+0000 7fc506d9d640 1 -- 172.21.15.89:0/2446053941 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5080c71e0 con 0x7fc5080c5f60 2023-10-30T12:49:01.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.694+0000 7fc506d9d640 1 --2- 172.21.15.89:0/2446053941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5080c5f60 0x7fc5080cb570 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fc500009d90 tx=0x7fc50001c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cba8549a022c7f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:01.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.694+0000 7fc505d9b640 1 -- 172.21.15.89:0/2446053941 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5000a5810 con 0x7fc5080c5f60 2023-10-30T12:49:01.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.694+0000 7fc505d9b640 1 -- 172.21.15.89:0/2446053941 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5000a5970 con 0x7fc5080c5f60 2023-10-30T12:49:01.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.694+0000 7fc505d9b640 1 -- 172.21.15.89:0/2446053941 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5000a5d00 con 0x7fc5080c5f60 2023-10-30T12:49:01.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.698+0000 7fc50cf72640 1 -- 172.21.15.89:0/2446053941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5080c5f60 msgr2=0x7fc5080cb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:01.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.698+0000 7fc50cf72640 1 --2- 172.21.15.89:0/2446053941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5080c5f60 0x7fc5080cb570 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fc500009d90 tx=0x7fc50001c950 comp rx=0 tx=0).stop 2023-10-30T12:49:01.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.698+0000 7fc50cf72640 1 -- 172.21.15.89:0/2446053941 shutdown_connections 2023-10-30T12:49:01.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.698+0000 7fc50cf72640 1 --2- 172.21.15.89:0/2446053941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5080c5f60 0x7fc5080cb570 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:01.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.698+0000 7fc50cf72640 1 --2- 172.21.15.89:0/2446053941 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5080c6b00 0x7fc508065da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:01.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.698+0000 7fc50cf72640 1 -- 172.21.15.89:0/2446053941 >> 172.21.15.89:0/2446053941 conn(0x7fc50806c6e0 msgr2=0x7fc508069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:01.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.698+0000 7fc50cf72640 1 -- 172.21.15.89:0/2446053941 shutdown_connections 2023-10-30T12:49:01.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.698+0000 7fc50cf72640 1 -- 172.21.15.89:0/2446053941 wait complete. 2023-10-30T12:49:01.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.698+0000 7fc50cf72640 1 Processor -- start 2023-10-30T12:49:01.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.698+0000 7fc50cf72640 1 -- start start 2023-10-30T12:49:01.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.698+0000 7fc50cf72640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5080c5f60 0x7fc508155350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:01.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.698+0000 7fc50cf72640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5080c6b00 0x7fc508156c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:01.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.698+0000 7fc50cf72640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc508151920 0x7fc50813e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:01.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.698+0000 7fc50cf72640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc508155d20 con 0x7fc508151920 2023-10-30T12:49:01.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.698+0000 7fc50cf72640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc50813cca0 con 0x7fc5080c6b00 2023-10-30T12:49:01.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.698+0000 7fc50cf72640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc508142110 con 0x7fc5080c5f60 2023-10-30T12:49:01.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.698+0000 7fc506d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc508151920 0x7fc50813e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:01.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.698+0000 7fc506d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc508151920 0x7fc50813e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56076/0 (socket says 172.21.15.89:56076) 2023-10-30T12:49:01.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.698+0000 7fc506d9d640 1 -- 172.21.15.89:0/1325558531 learned_addr learned my addr 172.21.15.89:0/1325558531 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:49:01.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.698+0000 7fc50659c640 1 --2- 172.21.15.89:0/1325558531 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5080c5f60 0x7fc508155350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:01.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.698+0000 7fc4fdd9b640 1 --2- 172.21.15.89:0/1325558531 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5080c6b00 0x7fc508156c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:01.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.702+0000 7fc506d9d640 1 -- 172.21.15.89:0/1325558531 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5080c5f60 msgr2=0x7fc508155350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:01.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.702+0000 7fc506d9d640 1 --2- 172.21.15.89:0/1325558531 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5080c5f60 0x7fc508155350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:01.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.702+0000 7fc506d9d640 1 -- 172.21.15.89:0/1325558531 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5080c6b00 msgr2=0x7fc508156c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:01.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.702+0000 7fc506d9d640 1 --2- 172.21.15.89:0/1325558531 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5080c6b00 0x7fc508156c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:01.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.702+0000 7fc506d9d640 1 -- 172.21.15.89:0/1325558531 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc500009a70 con 0x7fc508151920 2023-10-30T12:49:01.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.702+0000 7fc4fdd9b640 1 --2- 172.21.15.89:0/1325558531 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5080c6b00 0x7fc508156c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:49:01.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.702+0000 7fc50659c640 1 --2- 172.21.15.89:0/1325558531 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5080c5f60 0x7fc508155350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:49:01.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.702+0000 7fc506d9d640 1 --2- 172.21.15.89:0/1325558531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc508151920 0x7fc50813e2e0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fc50001b000 tx=0x7fc5000196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:01.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.702+0000 7fc4ff7fe640 1 -- 172.21.15.89:0/1325558531 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5000ad8d0 con 0x7fc508151920 2023-10-30T12:49:01.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.702+0000 7fc4ff7fe640 1 -- 172.21.15.89:0/1325558531 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5000ada30 con 0x7fc508151920 2023-10-30T12:49:01.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.702+0000 7fc4ff7fe640 1 -- 172.21.15.89:0/1325558531 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5000adda0 con 0x7fc508151920 2023-10-30T12:49:01.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.702+0000 7fc50cf72640 1 -- 172.21.15.89:0/1325558531 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc508142c50 con 0x7fc508151920 2023-10-30T12:49:01.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.702+0000 7fc50cf72640 1 -- 172.21.15.89:0/1325558531 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc508142320 con 0x7fc508151920 2023-10-30T12:49:01.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.702+0000 7fc50cf72640 1 -- 172.21.15.89:0/1325558531 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc508142de0 con 0x7fc508151920 2023-10-30T12:49:01.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.702+0000 7fc4ff7fe640 1 -- 172.21.15.89:0/1325558531 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc5000b4050 con 0x7fc508151920 2023-10-30T12:49:01.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.702+0000 7fc4ff7fe640 1 --2- 172.21.15.89:0/1325558531 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc4e80664a0 0x7fc4e8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:01.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.702+0000 7fc4ff7fe640 1 -- 172.21.15.89:0/1325558531 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6664+0+0 (secure 0 0 0) 0x7fc50011f2b0 con 0x7fc508151920 2023-10-30T12:49:01.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.706+0000 7fc50659c640 1 --2- 172.21.15.89:0/1325558531 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc4e80664a0 0x7fc4e8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:01.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.706+0000 7fc50659c640 1 --2- 172.21.15.89:0/1325558531 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc4e80664a0 0x7fc4e8068960 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0000f70 tx=0x7fc4f0007cb0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:01.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.706+0000 7fc4ff7fe640 1 -- 172.21.15.89:0/1325558531 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc5000ec400 con 0x7fc508151920 2023-10-30T12:49:01.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.886+0000 7fc50cf72640 1 -- 172.21.15.89:0/1325558531 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc50804e170 con 0x7fc508151920 2023-10-30T12:49:01.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.890+0000 7fc4ff7fe640 1 -- 172.21.15.89:0/1325558531 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+5743 (secure 0 0 0) 0x7fc5000efa20 con 0x7fc508151920 2023-10-30T12:49:01.887 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:49:01.887 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:49:01.291912+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5678,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5678":{"gid":5678,"name":"f","rank":0,"incarnation":98,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:01.888 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 98 2023-10-30T12:49:01.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.890+0000 7fc50cf72640 1 -- 172.21.15.89:0/1325558531 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc4e80664a0 msgr2=0x7fc4e8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:01.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.890+0000 7fc50cf72640 1 --2- 172.21.15.89:0/1325558531 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc4e80664a0 0x7fc4e8068960 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0000f70 tx=0x7fc4f0007cb0 comp rx=0 tx=0).stop 2023-10-30T12:49:01.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.890+0000 7fc50cf72640 1 -- 172.21.15.89:0/1325558531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc508151920 msgr2=0x7fc50813e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:01.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.890+0000 7fc50cf72640 1 --2- 172.21.15.89:0/1325558531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc508151920 0x7fc50813e2e0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fc50001b000 tx=0x7fc5000196f0 comp rx=0 tx=0).stop 2023-10-30T12:49:01.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.890+0000 7fc50cf72640 1 -- 172.21.15.89:0/1325558531 shutdown_connections 2023-10-30T12:49:01.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.890+0000 7fc50cf72640 1 --2- 172.21.15.89:0/1325558531 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc4e80664a0 0x7fc4e8068960 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:01.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.890+0000 7fc50cf72640 1 --2- 172.21.15.89:0/1325558531 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5080c6b00 0x7fc508156c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:01.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.890+0000 7fc50cf72640 1 --2- 172.21.15.89:0/1325558531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc508151920 0x7fc50813e2e0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:01.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.890+0000 7fc50cf72640 1 --2- 172.21.15.89:0/1325558531 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5080c5f60 0x7fc508155350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:01.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.890+0000 7fc50cf72640 1 -- 172.21.15.89:0/1325558531 >> 172.21.15.89:0/1325558531 conn(0x7fc50806c6e0 msgr2=0x7fc5081448b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:01.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.890+0000 7fc50cf72640 1 -- 172.21.15.89:0/1325558531 shutdown_connections 2023-10-30T12:49:01.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:01.890+0000 7fc50cf72640 1 -- 172.21.15.89:0/1325558531 wait complete. 2023-10-30T12:49:01.908 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2023-10-30T12:49:01.908 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:49:01.909 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:02.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.050+0000 7fe767dd3640 1 Processor -- start 2023-10-30T12:49:02.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.050+0000 7fe767dd3640 1 -- start start 2023-10-30T12:49:02.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe767dd3640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe760070a00 0x7fe760070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:02.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe767dd3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe760069e90 0x7fe76006a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:02.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe767dd3640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7600713b0 con 0x7fe7601539a0 2023-10-30T12:49:02.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe767dd3640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe76006fbb0 con 0x7fe760070a00 2023-10-30T12:49:02.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe767dd3640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe760153d80 con 0x7fe760069e90 2023-10-30T12:49:02.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe765b6f640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe7601539a0 0x7fe76015a380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51048/0 (socket says 172.21.15.89:51048) 2023-10-30T12:49:02.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe765b6f640 1 -- 172.21.15.89:0/1473475987 learned_addr learned my addr 172.21.15.89:0/1473475987 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:49:02.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe766370640 1 --2- 172.21.15.89:0/1473475987 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe760069e90 0x7fe76006a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:02.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe76536e640 1 --2- 172.21.15.89:0/1473475987 >> v2:172.21.15.192:3300/0 conn(0x7fe760070a00 0x7fe760070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:02.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe764b6d640 1 -- 172.21.15.89:0/1473475987 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1103632493 0 0) 0x7fe7600713b0 con 0x7fe7601539a0 2023-10-30T12:49:02.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe764b6d640 1 -- 172.21.15.89:0/1473475987 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe74c003530 con 0x7fe7601539a0 2023-10-30T12:49:02.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe764b6d640 1 -- 172.21.15.89:0/1473475987 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3916748971 0 0) 0x7fe74c003530 con 0x7fe7601539a0 2023-10-30T12:49:02.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe764b6d640 1 -- 172.21.15.89:0/1473475987 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe760069e90 msgr2=0x7fe76006a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:02.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe764b6d640 1 --2- 172.21.15.89:0/1473475987 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe760069e90 0x7fe76006a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:02.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe764b6d640 1 -- 172.21.15.89:0/1473475987 >> v2:172.21.15.192:3300/0 conn(0x7fe760070a00 msgr2=0x7fe760070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:02.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe764b6d640 1 --2- 172.21.15.89:0/1473475987 >> v2:172.21.15.192:3300/0 conn(0x7fe760070a00 0x7fe760070e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:02.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe764b6d640 1 -- 172.21.15.89:0/1473475987 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7600c5c50 con 0x7fe7601539a0 2023-10-30T12:49:02.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe764b6d640 1 -- 172.21.15.89:0/1473475987 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7fe75c001680 con 0x7fe7601539a0 2023-10-30T12:49:02.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe764b6d640 1 -- 172.21.15.89:0/1473475987 >> v1:172.21.15.89:6789/0 conn(0x7fe7601539a0 legacy=0x7fe76015a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:02.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe764b6d640 1 --2- 172.21.15.89:0/1473475987 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe760069e90 0x7fe74c003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:02.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe764b6d640 1 --2- 172.21.15.89:0/1473475987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe74c003f00 0x7fe74c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:02.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe764b6d640 1 --2- 172.21.15.89:0/1473475987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe74c004d10 0x7fe74c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:02.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe764b6d640 1 -- 172.21.15.89:0/1473475987 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76006fbb0 con 0x7fe74c003f00 2023-10-30T12:49:02.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe764b6d640 1 -- 172.21.15.89:0/1473475987 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe760153d80 con 0x7fe74c004d10 2023-10-30T12:49:02.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe764b6d640 1 -- 172.21.15.89:0/1473475987 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe74c00b9e0 con 0x7fe760069e90 2023-10-30T12:49:02.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe76536e640 1 --2- 172.21.15.89:0/1473475987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe74c003f00 0x7fe74c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:02.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe765b6f640 1 --2- 172.21.15.89:0/1473475987 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe760069e90 0x7fe74c003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:02.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe766370640 1 --2- 172.21.15.89:0/1473475987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe74c004d10 0x7fe74c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:02.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe76536e640 1 -- 172.21.15.89:0/1473475987 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe760069e90 msgr2=0x7fe74c003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:02.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe76536e640 1 --2- 172.21.15.89:0/1473475987 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe760069e90 0x7fe74c003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:02.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe76536e640 1 -- 172.21.15.89:0/1473475987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe74c004d10 msgr2=0x7fe74c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:02.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe76536e640 1 --2- 172.21.15.89:0/1473475987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe74c004d10 0x7fe74c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:02.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.054+0000 7fe76536e640 1 -- 172.21.15.89:0/1473475987 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe74c003530 con 0x7fe74c003f00 2023-10-30T12:49:02.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.058+0000 7fe76536e640 1 --2- 172.21.15.89:0/1473475987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe74c003f00 0x7fe74c0047d0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fe750009b10 tx=0x7fe75001ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b02cd9456028207e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:02.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.058+0000 7fe764b6d640 1 -- 172.21.15.89:0/1473475987 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7500a5630 con 0x7fe74c003f00 2023-10-30T12:49:02.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.058+0000 7fe764b6d640 1 -- 172.21.15.89:0/1473475987 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7500a5790 con 0x7fe74c003f00 2023-10-30T12:49:02.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.058+0000 7fe764b6d640 1 -- 172.21.15.89:0/1473475987 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7500a5b40 con 0x7fe74c003f00 2023-10-30T12:49:02.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.058+0000 7fe767dd3640 1 -- 172.21.15.89:0/1473475987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe74c003f00 msgr2=0x7fe74c0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:02.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.058+0000 7fe767dd3640 1 --2- 172.21.15.89:0/1473475987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe74c003f00 0x7fe74c0047d0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fe750009b10 tx=0x7fe75001ca60 comp rx=0 tx=0).stop 2023-10-30T12:49:02.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.058+0000 7fe766370640 1 -- 172.21.15.89:0/1473475987 reap_dead start 2023-10-30T12:49:02.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.058+0000 7fe767dd3640 1 -- 172.21.15.89:0/1473475987 shutdown_connections 2023-10-30T12:49:02.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.058+0000 7fe767dd3640 1 -- 172.21.15.89:0/1473475987 >> 172.21.15.89:0/1473475987 conn(0x7fe7600c7910 msgr2=0x7fe7600c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:02.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.058+0000 7fe767dd3640 1 -- 172.21.15.89:0/1473475987 shutdown_connections 2023-10-30T12:49:02.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.058+0000 7fe767dd3640 1 -- 172.21.15.89:0/1473475987 wait complete. 2023-10-30T12:49:02.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.062+0000 7fe767dd3640 1 Processor -- start 2023-10-30T12:49:02.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.062+0000 7fe767dd3640 1 -- start start 2023-10-30T12:49:02.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.062+0000 7fe767dd3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe74c003f00 0x7fe760158af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:02.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.062+0000 7fe767dd3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7600ca240 0x7fe76014ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:02.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.062+0000 7fe767dd3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe760159640 0x7fe760143990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:02.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.062+0000 7fe767dd3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7601f2e20 con 0x7fe7600ca240 2023-10-30T12:49:02.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.062+0000 7fe767dd3640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7600c9c70 con 0x7fe760159640 2023-10-30T12:49:02.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.062+0000 7fe767dd3640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7600c9f10 con 0x7fe74c003f00 2023-10-30T12:49:02.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.062+0000 7fe76536e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7600ca240 0x7fe76014ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:02.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.062+0000 7fe76536e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7600ca240 0x7fe76014ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56096/0 (socket says 172.21.15.89:56096) 2023-10-30T12:49:02.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.062+0000 7fe76536e640 1 -- 172.21.15.89:0/2894220340 learned_addr learned my addr 172.21.15.89:0/2894220340 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:49:02.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.062+0000 7fe765b6f640 1 --2- 172.21.15.89:0/2894220340 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe74c003f00 0x7fe760158af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:02.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.062+0000 7fe766370640 1 --2- 172.21.15.89:0/2894220340 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe760159640 0x7fe760143990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:02.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.062+0000 7fe76536e640 1 -- 172.21.15.89:0/2894220340 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe74c003f00 msgr2=0x7fe760158af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:02.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.062+0000 7fe76536e640 1 --2- 172.21.15.89:0/2894220340 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe74c003f00 0x7fe760158af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:02.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.062+0000 7fe76536e640 1 -- 172.21.15.89:0/2894220340 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe760159640 msgr2=0x7fe760143990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:02.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.062+0000 7fe76536e640 1 --2- 172.21.15.89:0/2894220340 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe760159640 0x7fe760143990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:02.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.062+0000 7fe76536e640 1 -- 172.21.15.89:0/2894220340 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7600ca790 con 0x7fe7600ca240 2023-10-30T12:49:02.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.062+0000 7fe76536e640 1 --2- 172.21.15.89:0/2894220340 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7600ca240 0x7fe76014ec40 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fe75001c890 tx=0x7fe74c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:02.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.062+0000 7fe7567fc640 1 -- 172.21.15.89:0/2894220340 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7500be070 con 0x7fe7600ca240 2023-10-30T12:49:02.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.062+0000 7fe7567fc640 1 -- 172.21.15.89:0/2894220340 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe75000f000 con 0x7fe7600ca240 2023-10-30T12:49:02.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.062+0000 7fe767dd3640 1 -- 172.21.15.89:0/2894220340 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7600caa70 con 0x7fe7600ca240 2023-10-30T12:49:02.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.062+0000 7fe7567fc640 1 -- 172.21.15.89:0/2894220340 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7500a5630 con 0x7fe7600ca240 2023-10-30T12:49:02.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.062+0000 7fe767dd3640 1 -- 172.21.15.89:0/2894220340 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7600c97e0 con 0x7fe7600ca240 2023-10-30T12:49:02.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.062+0000 7fe767dd3640 1 -- 172.21.15.89:0/2894220340 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7600cac00 con 0x7fe7600ca240 2023-10-30T12:49:02.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.066+0000 7fe7567fc640 1 -- 172.21.15.89:0/2894220340 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe7500b4040 con 0x7fe7600ca240 2023-10-30T12:49:02.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.066+0000 7fe7567fc640 1 --2- 172.21.15.89:0/2894220340 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe7440664a0 0x7fe744068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:02.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.066+0000 7fe7567fc640 1 -- 172.21.15.89:0/2894220340 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6664+0+0 (secure 0 0 0) 0x7fe75011dc30 con 0x7fe7600ca240 2023-10-30T12:49:02.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.066+0000 7fe765b6f640 1 --2- 172.21.15.89:0/2894220340 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe7440664a0 0x7fe744068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:02.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.066+0000 7fe765b6f640 1 --2- 172.21.15.89:0/2894220340 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe7440664a0 0x7fe744068960 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fe75c001830 tx=0x7fe75c007040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:02.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.070+0000 7fe7567fc640 1 -- 172.21.15.89:0/2894220340 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe7500ead00 con 0x7fe7600ca240 2023-10-30T12:49:02.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.250+0000 7fe767dd3640 1 -- 172.21.15.89:0/2894220340 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7600c6db0 con 0x7fe7600ca240 2023-10-30T12:49:02.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.250+0000 7fe7567fc640 1 -- 172.21.15.89:0/2894220340 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+5743 (secure 0 0 0) 0x7fe7500ee320 con 0x7fe7600ca240 2023-10-30T12:49:02.248 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:49:02.248 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:49:01.291912+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5678,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5678":{"gid":5678,"name":"f","rank":0,"incarnation":98,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:02.249 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 98 2023-10-30T12:49:02.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.250+0000 7fe767dd3640 1 -- 172.21.15.89:0/2894220340 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe7440664a0 msgr2=0x7fe744068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:02.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.250+0000 7fe767dd3640 1 --2- 172.21.15.89:0/2894220340 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe7440664a0 0x7fe744068960 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fe75c001830 tx=0x7fe75c007040 comp rx=0 tx=0).stop 2023-10-30T12:49:02.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.250+0000 7fe767dd3640 1 -- 172.21.15.89:0/2894220340 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7600ca240 msgr2=0x7fe76014ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:02.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.250+0000 7fe767dd3640 1 --2- 172.21.15.89:0/2894220340 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7600ca240 0x7fe76014ec40 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fe75001c890 tx=0x7fe74c005800 comp rx=0 tx=0).stop 2023-10-30T12:49:02.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.250+0000 7fe767dd3640 1 -- 172.21.15.89:0/2894220340 shutdown_connections 2023-10-30T12:49:02.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.250+0000 7fe767dd3640 1 --2- 172.21.15.89:0/2894220340 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe7440664a0 0x7fe744068960 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:02.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.250+0000 7fe767dd3640 1 --2- 172.21.15.89:0/2894220340 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe760159640 0x7fe760143990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:02.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.250+0000 7fe767dd3640 1 --2- 172.21.15.89:0/2894220340 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7600ca240 0x7fe76014ec40 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:02.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.250+0000 7fe767dd3640 1 --2- 172.21.15.89:0/2894220340 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe74c003f00 0x7fe760158af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:02.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.250+0000 7fe767dd3640 1 -- 172.21.15.89:0/2894220340 >> 172.21.15.89:0/2894220340 conn(0x7fe7600c7910 msgr2=0x7fe76006b560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:02.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.250+0000 7fe767dd3640 1 -- 172.21.15.89:0/2894220340 shutdown_connections 2023-10-30T12:49:02.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:02.250+0000 7fe767dd3640 1 -- 172.21.15.89:0/2894220340 wait complete. 2023-10-30T12:49:02.269 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 98, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:50.729245+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 98, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5678": { "addr": "172.21.15.192:6837/2130807115", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6836", "nonce": 2130807115, "type": "v2" }, { "addr": "172.21.15.192:6837", "nonce": 2130807115, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5678, "incarnation": 98, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:replay", "state_seq": 1 }, "gid_5728": { "addr": "172.21.15.192:6839/3991891597", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6838", "nonce": 3991891597, "type": "v2" }, { "addr": "172.21.15.192:6839", "nonce": 3991891597, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 5728, "incarnation": 73, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 19 }, "gid_6075": { "addr": "172.21.15.89:6837/2412618858", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6836", "nonce": 2412618858, "type": "v2" }, { "addr": "172.21.15.89:6837", "nonce": 2412618858, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6075, "incarnation": 92, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 38 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:49:01.291912+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5678, "mds_1": 6075, "mds_2": 5728 } } } ], "standbys": [ { "addr": "172.21.15.89:6834/2909790173", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6832", "nonce": 2909790173, "type": "v2" }, { "addr": "172.21.15.89:6834", "nonce": 2909790173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.89:6835/2387357615", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6833", "nonce": 2387357615, "type": "v2" }, { "addr": "172.21.15.89:6835", "nonce": 2387357615, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4239, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:49:04.270 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:04.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.422+0000 7f015ce9e640 1 Processor -- start 2023-10-30T12:49:04.419 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.422+0000 7f015ce9e640 1 -- start start 2023-10-30T12:49:04.420 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.422+0000 7f015ce9e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f01580c6b80 0x7f01580bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:04.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.422+0000 7f015ce9e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01580c6f60 0x7f01580c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:04.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.422+0000 7f015ce9e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01580c0a50 con 0x7f01580c8430 2023-10-30T12:49:04.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.422+0000 7f015ce9e640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f015806f6e0 con 0x7f01580c6b80 2023-10-30T12:49:04.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.422+0000 7f015ce9e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015806f9c0 con 0x7f01580c6f60 2023-10-30T12:49:04.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.422+0000 7f015659c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f01580c6b80 0x7f01580bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:04.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.422+0000 7f015659c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f01580c6b80 0x7f01580bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56100/0 (socket says 172.21.15.89:56100) 2023-10-30T12:49:04.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.422+0000 7f015659c640 1 -- 172.21.15.89:0/371473062 learned_addr learned my addr 172.21.15.89:0/371473062 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:49:04.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.422+0000 7f0155d9b640 1 --2- 172.21.15.89:0/371473062 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01580c6f60 0x7f01580c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:04.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.422+0000 7f015659c640 1 -- 172.21.15.89:0/371473062 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01580c6f60 msgr2=0x7f01580c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:04.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.422+0000 7f015659c640 1 --2- 172.21.15.89:0/371473062 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01580c6f60 0x7f01580c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:04.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.422+0000 7f015659c640 1 -- 172.21.15.89:0/371473062 >> v1:172.21.15.192:6789/0 conn(0x7f01580c8430 legacy=0x7f015813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:04.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.422+0000 7f015659c640 1 -- 172.21.15.89:0/371473062 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f015806fca0 con 0x7f01580c6b80 2023-10-30T12:49:04.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.426+0000 7f015659c640 1 --2- 172.21.15.89:0/371473062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01580c6b80 0x7f01580bfec0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f0148009870 tx=0x7f014801e680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d9c74855edbc82e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:04.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.426+0000 7f015559a640 1 -- 172.21.15.89:0/371473062 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01480ab620 con 0x7f01580c6b80 2023-10-30T12:49:04.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.426+0000 7f015559a640 1 -- 172.21.15.89:0/371473062 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01480ab780 con 0x7f01580c6b80 2023-10-30T12:49:04.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.426+0000 7f015559a640 1 -- 172.21.15.89:0/371473062 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01480abb10 con 0x7f01580c6b80 2023-10-30T12:49:04.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.426+0000 7f015ce9e640 1 -- 172.21.15.89:0/371473062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01580c6b80 msgr2=0x7f01580bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:04.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.426+0000 7f015ce9e640 1 --2- 172.21.15.89:0/371473062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01580c6b80 0x7f01580bfec0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f0148009870 tx=0x7f014801e680 comp rx=0 tx=0).stop 2023-10-30T12:49:04.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.426+0000 7f015ce9e640 1 -- 172.21.15.89:0/371473062 shutdown_connections 2023-10-30T12:49:04.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.426+0000 7f015ce9e640 1 --2- 172.21.15.89:0/371473062 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01580c6f60 0x7f01580c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:04.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.426+0000 7f015ce9e640 1 --2- 172.21.15.89:0/371473062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01580c6b80 0x7f01580bfec0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:04.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.426+0000 7f015ce9e640 1 -- 172.21.15.89:0/371473062 >> 172.21.15.89:0/371473062 conn(0x7f01580c7910 msgr2=0x7f01580c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:04.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.426+0000 7f015ce9e640 1 -- 172.21.15.89:0/371473062 shutdown_connections 2023-10-30T12:49:04.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.426+0000 7f015ce9e640 1 -- 172.21.15.89:0/371473062 wait complete. 2023-10-30T12:49:04.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.430+0000 7f015ce9e640 1 Processor -- start 2023-10-30T12:49:04.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.430+0000 7f015ce9e640 1 -- start start 2023-10-30T12:49:04.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.430+0000 7f015ce9e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01580c6b80 0x7f015813c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:04.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.430+0000 7f015ce9e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01580c6f60 0x7f0158158210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:04.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.430+0000 7f015ce9e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01580c8430 0x7f0158134a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:04.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.430+0000 7f015ce9e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01580ca320 con 0x7f01580c8430 2023-10-30T12:49:04.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.430+0000 7f015ce9e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01580c9a60 con 0x7f01580c6b80 2023-10-30T12:49:04.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.430+0000 7f015ce9e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0158158750 con 0x7f01580c6f60 2023-10-30T12:49:04.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.430+0000 7f0156d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01580c8430 0x7f0158134a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:04.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.430+0000 7f0156d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01580c8430 0x7f0158134a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56106/0 (socket says 172.21.15.89:56106) 2023-10-30T12:49:04.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.430+0000 7f0156d9d640 1 -- 172.21.15.89:0/1882089828 learned_addr learned my addr 172.21.15.89:0/1882089828 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:49:04.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.430+0000 7f0155d9b640 1 --2- 172.21.15.89:0/1882089828 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01580c6f60 0x7f0158158210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:04.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.430+0000 7f015659c640 1 --2- 172.21.15.89:0/1882089828 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01580c6b80 0x7f015813c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:04.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.430+0000 7f0156d9d640 1 -- 172.21.15.89:0/1882089828 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01580c6f60 msgr2=0x7f0158158210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:04.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.430+0000 7f0156d9d640 1 --2- 172.21.15.89:0/1882089828 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01580c6f60 0x7f0158158210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:04.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.430+0000 7f0156d9d640 1 -- 172.21.15.89:0/1882089828 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01580c6b80 msgr2=0x7f015813c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:04.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.430+0000 7f0156d9d640 1 --2- 172.21.15.89:0/1882089828 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01580c6b80 0x7f015813c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:04.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.430+0000 7f0156d9d640 1 -- 172.21.15.89:0/1882089828 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01480094d0 con 0x7f01580c8430 2023-10-30T12:49:04.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.430+0000 7f0156d9d640 1 --2- 172.21.15.89:0/1882089828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01580c8430 0x7f0158134a40 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f015000b9a0 tx=0x7f015000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:04.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.430+0000 7f0146ffd640 1 -- 172.21.15.89:0/1882089828 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015000cba0 con 0x7f01580c8430 2023-10-30T12:49:04.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.430+0000 7f0146ffd640 1 -- 172.21.15.89:0/1882089828 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f015000cd00 con 0x7f01580c8430 2023-10-30T12:49:04.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.430+0000 7f0146ffd640 1 -- 172.21.15.89:0/1882089828 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0150014680 con 0x7f01580c8430 2023-10-30T12:49:04.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.430+0000 7f015ce9e640 1 -- 172.21.15.89:0/1882089828 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01580c90e0 con 0x7f01580c8430 2023-10-30T12:49:04.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.430+0000 7f015ce9e640 1 -- 172.21.15.89:0/1882089828 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01580c9fb0 con 0x7f01580c8430 2023-10-30T12:49:04.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.430+0000 7f015ce9e640 1 -- 172.21.15.89:0/1882089828 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01580ca4d0 con 0x7f01580c8430 2023-10-30T12:49:04.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.434+0000 7f0146ffd640 1 -- 172.21.15.89:0/1882089828 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f015001d3b0 con 0x7f01580c8430 2023-10-30T12:49:04.435 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.434+0000 7f0146ffd640 1 --2- 172.21.15.89:0/1882089828 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f01380664a0 0x7f0138068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:04.435 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.434+0000 7f0146ffd640 1 -- 172.21.15.89:0/1882089828 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6664+0+0 (secure 0 0 0) 0x7f0150083fe0 con 0x7f01580c8430 2023-10-30T12:49:04.435 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.434+0000 7f015659c640 1 --2- 172.21.15.89:0/1882089828 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f01380664a0 0x7f0138068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:04.435 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.434+0000 7f015659c640 1 --2- 172.21.15.89:0/1882089828 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f01380664a0 0x7f0138068960 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f01480b0000 tx=0x7f01480b5040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:04.436 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.438+0000 7f0146ffd640 1 -- 172.21.15.89:0/1882089828 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f01580ca4d0 con 0x7f01580c8430 2023-10-30T12:49:04.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.618+0000 7f015ce9e640 1 -- 172.21.15.89:0/1882089828 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01580ca4d0 con 0x7f01580c8430 2023-10-30T12:49:04.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.622+0000 7f0146ffd640 1 -- 172.21.15.89:0/1882089828 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+5743 (secure 0 0 0) 0x7f01580ca4d0 con 0x7f01580c8430 2023-10-30T12:49:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:49:04.619 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:49:01.291912+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5678,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5678":{"gid":5678,"name":"f","rank":0,"incarnation":98,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:04.619 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 98 2023-10-30T12:49:04.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.622+0000 7f015ce9e640 1 -- 172.21.15.89:0/1882089828 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f01380664a0 msgr2=0x7f0138068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:04.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.622+0000 7f015ce9e640 1 --2- 172.21.15.89:0/1882089828 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f01380664a0 0x7f0138068960 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f01480b0000 tx=0x7f01480b5040 comp rx=0 tx=0).stop 2023-10-30T12:49:04.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.622+0000 7f015ce9e640 1 -- 172.21.15.89:0/1882089828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01580c8430 msgr2=0x7f0158134a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:04.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.622+0000 7f015ce9e640 1 --2- 172.21.15.89:0/1882089828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01580c8430 0x7f0158134a40 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f015000b9a0 tx=0x7f015000be60 comp rx=0 tx=0).stop 2023-10-30T12:49:04.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.622+0000 7f015ce9e640 1 -- 172.21.15.89:0/1882089828 shutdown_connections 2023-10-30T12:49:04.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.622+0000 7f015ce9e640 1 --2- 172.21.15.89:0/1882089828 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f01380664a0 0x7f0138068960 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:04.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.622+0000 7f015ce9e640 1 --2- 172.21.15.89:0/1882089828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f01580c8430 0x7f0158134a40 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:04.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.622+0000 7f015ce9e640 1 --2- 172.21.15.89:0/1882089828 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f01580c6f60 0x7f0158158210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:04.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.622+0000 7f015ce9e640 1 --2- 172.21.15.89:0/1882089828 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01580c6b80 0x7f015813c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:04.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.622+0000 7f015ce9e640 1 -- 172.21.15.89:0/1882089828 >> 172.21.15.89:0/1882089828 conn(0x7f01580c7910 msgr2=0x7f01580b08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:04.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.622+0000 7f015ce9e640 1 -- 172.21.15.89:0/1882089828 shutdown_connections 2023-10-30T12:49:04.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:04.622+0000 7f015ce9e640 1 -- 172.21.15.89:0/1882089828 wait complete. 2023-10-30T12:49:04.639 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:49:06.640 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:06.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.782+0000 7f4e9f1c4640 1 Processor -- start 2023-10-30T12:49:06.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.782+0000 7f4e9f1c4640 1 -- start start 2023-10-30T12:49:06.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.782+0000 7f4e9f1c4640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e980c6b80 0x7f4e980bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:06.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.782+0000 7f4e9f1c4640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4e9813f4a0 0x7f4e980c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:06.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.782+0000 7f4e9f1c4640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e980c0a50 con 0x7f4e980c6f60 2023-10-30T12:49:06.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.782+0000 7f4e9f1c4640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e9806f6e0 con 0x7f4e9813f4a0 2023-10-30T12:49:06.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.782+0000 7f4e9f1c4640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9806f9c0 con 0x7f4e980c6b80 2023-10-30T12:49:06.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.782+0000 7f4e8ffff640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f4e980c6f60 0x7f4e9813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:51058/0 (socket says 172.21.15.89:51058) 2023-10-30T12:49:06.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.782+0000 7f4e8ffff640 1 -- 172.21.15.89:0/368334596 learned_addr learned my addr 172.21.15.89:0/368334596 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:49:06.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.782+0000 7f4e9d761640 1 --2- 172.21.15.89:0/368334596 >> v2:172.21.15.89:3300/0 conn(0x7f4e9813f4a0 0x7f4e980c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:06.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.782+0000 7f4e9cf60640 1 --2- 172.21.15.89:0/368334596 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e980c6b80 0x7f4e980bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:06.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.782+0000 7f4e9d761640 1 -- 172.21.15.89:0/368334596 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e980c6b80 msgr2=0x7f4e980bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:06.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.782+0000 7f4e9d761640 1 --2- 172.21.15.89:0/368334596 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e980c6b80 0x7f4e980bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:06.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.782+0000 7f4e9d761640 1 -- 172.21.15.89:0/368334596 >> v1:172.21.15.89:6789/0 conn(0x7f4e980c6f60 legacy=0x7f4e9813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:06.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.782+0000 7f4e9d761640 1 -- 172.21.15.89:0/368334596 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9806fca0 con 0x7f4e9813f4a0 2023-10-30T12:49:06.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.782+0000 7f4e8f7fe640 1 -- 172.21.15.89:0/368334596 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3363181589 0 0) 0x7f4e980c0a50 con 0x7f4e980c6f60 2023-10-30T12:49:06.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.786+0000 7f4e9d761640 1 --2- 172.21.15.89:0/368334596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e9813f4a0 0x7f4e980c0400 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f4e90009d90 tx=0x7f4e9001c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3307f1e9d141f4f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:06.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.786+0000 7f4e8f7fe640 1 -- 172.21.15.89:0/368334596 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e900a5920 con 0x7f4e9813f4a0 2023-10-30T12:49:06.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.786+0000 7f4e8f7fe640 1 -- 172.21.15.89:0/368334596 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e900a5a80 con 0x7f4e9813f4a0 2023-10-30T12:49:06.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.786+0000 7f4e8f7fe640 1 -- 172.21.15.89:0/368334596 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e900a5e10 con 0x7f4e9813f4a0 2023-10-30T12:49:06.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.786+0000 7f4e9f1c4640 1 -- 172.21.15.89:0/368334596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e9813f4a0 msgr2=0x7f4e980c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:06.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.786+0000 7f4e9f1c4640 1 --2- 172.21.15.89:0/368334596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e9813f4a0 0x7f4e980c0400 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f4e90009d90 tx=0x7f4e9001c950 comp rx=0 tx=0).stop 2023-10-30T12:49:06.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.786+0000 7f4e9f1c4640 1 -- 172.21.15.89:0/368334596 shutdown_connections 2023-10-30T12:49:06.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.786+0000 7f4e9f1c4640 1 --2- 172.21.15.89:0/368334596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e9813f4a0 0x7f4e980c0400 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:06.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.786+0000 7f4e9f1c4640 1 --2- 172.21.15.89:0/368334596 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e980c6b80 0x7f4e980bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:06.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.786+0000 7f4e9f1c4640 1 -- 172.21.15.89:0/368334596 >> 172.21.15.89:0/368334596 conn(0x7f4e980c7910 msgr2=0x7f4e980c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:06.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.786+0000 7f4e9f1c4640 1 -- 172.21.15.89:0/368334596 shutdown_connections 2023-10-30T12:49:06.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.786+0000 7f4e9f1c4640 1 -- 172.21.15.89:0/368334596 wait complete. 2023-10-30T12:49:06.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.790+0000 7f4e9f1c4640 1 Processor -- start 2023-10-30T12:49:06.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.790+0000 7f4e9f1c4640 1 -- start start 2023-10-30T12:49:06.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.790+0000 7f4e9f1c4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e980c6b80 0x7f4e98154580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:06.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.790+0000 7f4e9f1c4640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e980c6f60 0x7f4e981538a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:06.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.790+0000 7f4e9f1c4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e9813f4a0 0x7f4e980c93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:06.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.790+0000 7f4e9f1c4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e981f53a0 con 0x7f4e9813f4a0 2023-10-30T12:49:06.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.790+0000 7f4e9f1c4640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e981f55d0 con 0x7f4e980c6b80 2023-10-30T12:49:06.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.790+0000 7f4e9f1c4640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e98152fa0 con 0x7f4e980c6f60 2023-10-30T12:49:06.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.790+0000 7f4e9d761640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e9813f4a0 0x7f4e980c93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:06.796 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.790+0000 7f4e9d761640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e9813f4a0 0x7f4e980c93c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56114/0 (socket says 172.21.15.89:56114) 2023-10-30T12:49:06.796 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.790+0000 7f4e9d761640 1 -- 172.21.15.89:0/661705829 learned_addr learned my addr 172.21.15.89:0/661705829 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:49:06.796 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.790+0000 7f4e8ffff640 1 --2- 172.21.15.89:0/661705829 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e980c6f60 0x7f4e981538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:06.796 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.790+0000 7f4e9cf60640 1 --2- 172.21.15.89:0/661705829 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e980c6b80 0x7f4e98154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:06.797 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.790+0000 7f4e9d761640 1 -- 172.21.15.89:0/661705829 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e980c6f60 msgr2=0x7f4e981538a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:06.797 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.790+0000 7f4e9d761640 1 --2- 172.21.15.89:0/661705829 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e980c6f60 0x7f4e981538a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:06.797 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.790+0000 7f4e9d761640 1 -- 172.21.15.89:0/661705829 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e980c6b80 msgr2=0x7f4e98154580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:06.797 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.790+0000 7f4e9d761640 1 --2- 172.21.15.89:0/661705829 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e980c6b80 0x7f4e98154580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:06.798 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.790+0000 7f4e9d761640 1 -- 172.21.15.89:0/661705829 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e90009a70 con 0x7f4e9813f4a0 2023-10-30T12:49:06.798 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.790+0000 7f4e9d761640 1 --2- 172.21.15.89:0/661705829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e9813f4a0 0x7f4e980c93c0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f4e900b0040 tx=0x7f4e900196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:06.798 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.790+0000 7f4e8d7fa640 1 -- 172.21.15.89:0/661705829 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e900ad9d0 con 0x7f4e9813f4a0 2023-10-30T12:49:06.798 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.790+0000 7f4e8d7fa640 1 -- 172.21.15.89:0/661705829 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e900adb30 con 0x7f4e9813f4a0 2023-10-30T12:49:06.799 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.790+0000 7f4e9f1c4640 1 -- 172.21.15.89:0/661705829 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e981f5870 con 0x7f4e9813f4a0 2023-10-30T12:49:06.799 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.790+0000 7f4e8d7fa640 1 -- 172.21.15.89:0/661705829 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e900b7410 con 0x7f4e9813f4a0 2023-10-30T12:49:06.799 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.790+0000 7f4e9f1c4640 1 -- 172.21.15.89:0/661705829 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e981f5e30 con 0x7f4e9813f4a0 2023-10-30T12:49:06.799 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.794+0000 7f4e9f1c4640 1 -- 172.21.15.89:0/661705829 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e980c90e0 con 0x7f4e9813f4a0 2023-10-30T12:49:06.799 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.794+0000 7f4e8d7fa640 1 -- 172.21.15.89:0/661705829 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4e900b4050 con 0x7f4e9813f4a0 2023-10-30T12:49:06.801 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.794+0000 7f4e8d7fa640 1 --2- 172.21.15.89:0/661705829 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4e840664a0 0x7f4e84068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:06.801 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.794+0000 7f4e8d7fa640 1 -- 172.21.15.89:0/661705829 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6664+0+0 (secure 0 0 0) 0x7f4e9011f590 con 0x7f4e9813f4a0 2023-10-30T12:49:06.801 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.794+0000 7f4e9cf60640 1 --2- 172.21.15.89:0/661705829 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4e840664a0 0x7f4e84068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:06.801 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.794+0000 7f4e9cf60640 1 --2- 172.21.15.89:0/661705829 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4e840664a0 0x7f4e84068960 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f4e94009af0 tx=0x7f4e94008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:06.801 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.798+0000 7f4e8d7fa640 1 -- 172.21.15.89:0/661705829 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4e980c90e0 con 0x7f4e9813f4a0 2023-10-30T12:49:06.985 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.986+0000 7f4e9f1c4640 1 -- 172.21.15.89:0/661705829 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e980c90e0 con 0x7f4e9813f4a0 2023-10-30T12:49:06.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.986+0000 7f4e8d7fa640 1 -- 172.21.15.89:0/661705829 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+5743 (secure 0 0 0) 0x7f4e980c90e0 con 0x7f4e9813f4a0 2023-10-30T12:49:06.986 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:49:06.987 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:49:01.291912+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5678,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5678":{"gid":5678,"name":"f","rank":0,"incarnation":98,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:06.987 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 98 2023-10-30T12:49:06.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.990+0000 7f4e9f1c4640 1 -- 172.21.15.89:0/661705829 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4e840664a0 msgr2=0x7f4e84068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:06.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.990+0000 7f4e9f1c4640 1 --2- 172.21.15.89:0/661705829 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4e840664a0 0x7f4e84068960 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f4e94009af0 tx=0x7f4e94008040 comp rx=0 tx=0).stop 2023-10-30T12:49:06.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.990+0000 7f4e9f1c4640 1 -- 172.21.15.89:0/661705829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e9813f4a0 msgr2=0x7f4e980c93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:06.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.990+0000 7f4e9f1c4640 1 --2- 172.21.15.89:0/661705829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e9813f4a0 0x7f4e980c93c0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f4e900b0040 tx=0x7f4e900196f0 comp rx=0 tx=0).stop 2023-10-30T12:49:06.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.990+0000 7f4e9f1c4640 1 -- 172.21.15.89:0/661705829 shutdown_connections 2023-10-30T12:49:06.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.990+0000 7f4e9f1c4640 1 --2- 172.21.15.89:0/661705829 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4e840664a0 0x7f4e84068960 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:06.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.990+0000 7f4e9f1c4640 1 --2- 172.21.15.89:0/661705829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4e9813f4a0 0x7f4e980c93c0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:06.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.990+0000 7f4e9f1c4640 1 --2- 172.21.15.89:0/661705829 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e980c6f60 0x7f4e981538a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:06.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.990+0000 7f4e9f1c4640 1 --2- 172.21.15.89:0/661705829 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e980c6b80 0x7f4e98154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:06.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.990+0000 7f4e9f1c4640 1 -- 172.21.15.89:0/661705829 >> 172.21.15.89:0/661705829 conn(0x7f4e980c7910 msgr2=0x7f4e98069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:06.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.990+0000 7f4e9f1c4640 1 -- 172.21.15.89:0/661705829 shutdown_connections 2023-10-30T12:49:06.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:06.990+0000 7f4e9f1c4640 1 -- 172.21.15.89:0/661705829 wait complete. 2023-10-30T12:49:07.008 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:49:09.009 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:09.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.154+0000 7f240913e640 1 Processor -- start 2023-10-30T12:49:09.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.154+0000 7f240913e640 1 -- start start 2023-10-30T12:49:09.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.158+0000 7f240913e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f24040c6bd0 0x7f240406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:09.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.158+0000 7f240913e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2404071110 0x7f2404158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:09.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.158+0000 7f240913e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2404066cf0 con 0x7f240406f7a0 2023-10-30T12:49:09.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.158+0000 7f240913e640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24040700c0 con 0x7f2404071110 2023-10-30T12:49:09.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.158+0000 7f240913e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24040c70d0 con 0x7f24040c6bd0 2023-10-30T12:49:09.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.158+0000 7f2402d9d640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f240406f7a0 0x7f2404069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:51558/0 (socket says 172.21.15.89:51558) 2023-10-30T12:49:09.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.158+0000 7f240259c640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f24040c6bd0 0x7f240406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:09.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.158+0000 7f2402d9d640 1 -- 172.21.15.89:0/4266650817 learned_addr learned my addr 172.21.15.89:0/4266650817 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:49:09.156 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.158+0000 7f240359e640 1 --2- 172.21.15.89:0/4266650817 >> v2:172.21.15.192:3300/0 conn(0x7f2404071110 0x7f2404158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:09.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.158+0000 7f240359e640 1 -- 172.21.15.89:0/4266650817 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f24040c6bd0 msgr2=0x7f240406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:09.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.158+0000 7f240359e640 1 --2- 172.21.15.89:0/4266650817 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f24040c6bd0 0x7f240406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:09.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.158+0000 7f240359e640 1 -- 172.21.15.89:0/4266650817 >> v1:172.21.15.192:6789/0 conn(0x7f240406f7a0 legacy=0x7f2404069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:09.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.158+0000 7f240359e640 1 -- 172.21.15.89:0/4266650817 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24040c7390 con 0x7f2404071110 2023-10-30T12:49:09.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.158+0000 7f240259c640 1 --2- 172.21.15.89:0/4266650817 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f24040c6bd0 0x7f240406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:49:09.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.162+0000 7f240359e640 1 --2- 172.21.15.89:0/4266650817 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2404071110 0x7f2404158110 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f23fc009d90 tx=0x7f23fc01f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42eb7598d53e52d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:09.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.162+0000 7f2401d9b640 1 -- 172.21.15.89:0/4266650817 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc0af600 con 0x7f2404071110 2023-10-30T12:49:09.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.162+0000 7f2401d9b640 1 -- 172.21.15.89:0/4266650817 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23fc0ab050 con 0x7f2404071110 2023-10-30T12:49:09.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.162+0000 7f2401d9b640 1 -- 172.21.15.89:0/4266650817 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc0af980 con 0x7f2404071110 2023-10-30T12:49:09.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.162+0000 7f240913e640 1 -- 172.21.15.89:0/4266650817 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2404071110 msgr2=0x7f2404158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:09.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.162+0000 7f240913e640 1 --2- 172.21.15.89:0/4266650817 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2404071110 0x7f2404158110 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f23fc009d90 tx=0x7f23fc01f130 comp rx=0 tx=0).stop 2023-10-30T12:49:09.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.162+0000 7f240913e640 1 -- 172.21.15.89:0/4266650817 shutdown_connections 2023-10-30T12:49:09.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.162+0000 7f240913e640 1 --2- 172.21.15.89:0/4266650817 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2404071110 0x7f2404158110 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:09.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.162+0000 7f240913e640 1 --2- 172.21.15.89:0/4266650817 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f24040c6bd0 0x7f240406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:09.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.162+0000 7f240913e640 1 -- 172.21.15.89:0/4266650817 >> 172.21.15.89:0/4266650817 conn(0x7f24040c7910 msgr2=0x7f24040c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:09.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.162+0000 7f240913e640 1 -- 172.21.15.89:0/4266650817 shutdown_connections 2023-10-30T12:49:09.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.162+0000 7f240913e640 1 -- 172.21.15.89:0/4266650817 wait complete. 2023-10-30T12:49:09.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.162+0000 7f240913e640 1 Processor -- start 2023-10-30T12:49:09.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.162+0000 7f240913e640 1 -- start start 2023-10-30T12:49:09.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.162+0000 7f240913e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f240406f7a0 0x7f24041f0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:09.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.162+0000 7f240913e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2404071110 0x7f24041545c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:09.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.162+0000 7f240913e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24040c6bd0 0x7f2404156d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:09.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.162+0000 7f240913e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24041f08d0 con 0x7f24040c6bd0 2023-10-30T12:49:09.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.162+0000 7f240913e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24040c83e0 con 0x7f240406f7a0 2023-10-30T12:49:09.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.166+0000 7f240913e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2404154b00 con 0x7f2404071110 2023-10-30T12:49:09.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.166+0000 7f240359e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24040c6bd0 0x7f2404156d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:09.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.166+0000 7f240359e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24040c6bd0 0x7f2404156d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56120/0 (socket says 172.21.15.89:56120) 2023-10-30T12:49:09.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.166+0000 7f240359e640 1 -- 172.21.15.89:0/4163770905 learned_addr learned my addr 172.21.15.89:0/4163770905 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:49:09.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.166+0000 7f2402d9d640 1 --2- 172.21.15.89:0/4163770905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f240406f7a0 0x7f24041f0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:09.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.166+0000 7f240259c640 1 --2- 172.21.15.89:0/4163770905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2404071110 0x7f24041545c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:09.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.166+0000 7f240359e640 1 -- 172.21.15.89:0/4163770905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2404071110 msgr2=0x7f24041545c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:09.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.166+0000 7f240359e640 1 --2- 172.21.15.89:0/4163770905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2404071110 0x7f24041545c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:09.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.166+0000 7f240359e640 1 -- 172.21.15.89:0/4163770905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f240406f7a0 msgr2=0x7f24041f0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:09.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.166+0000 7f240359e640 1 --2- 172.21.15.89:0/4163770905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f240406f7a0 0x7f24041f0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:09.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.166+0000 7f240359e640 1 -- 172.21.15.89:0/4163770905 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f8009420 con 0x7f24040c6bd0 2023-10-30T12:49:09.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.166+0000 7f240359e640 1 --2- 172.21.15.89:0/4163770905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24040c6bd0 0x7f2404156d40 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f23fc0b2040 tx=0x7f23fc01bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:09.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.166+0000 7f23eb7fe640 1 -- 172.21.15.89:0/4163770905 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc009450 con 0x7f24040c6bd0 2023-10-30T12:49:09.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.166+0000 7f23eb7fe640 1 -- 172.21.15.89:0/4163770905 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23fc0b9410 con 0x7f24040c6bd0 2023-10-30T12:49:09.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.166+0000 7f23eb7fe640 1 -- 172.21.15.89:0/4163770905 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc0b97c0 con 0x7f24040c6bd0 2023-10-30T12:49:09.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.166+0000 7f240913e640 1 -- 172.21.15.89:0/4163770905 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23fc009a70 con 0x7f24040c6bd0 2023-10-30T12:49:09.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.166+0000 7f240913e640 1 -- 172.21.15.89:0/4163770905 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24041ed2e0 con 0x7f24040c6bd0 2023-10-30T12:49:09.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.166+0000 7f240913e640 1 -- 172.21.15.89:0/4163770905 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f240404e170 con 0x7f24040c6bd0 2023-10-30T12:49:09.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.166+0000 7f23eb7fe640 1 -- 172.21.15.89:0/4163770905 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f23fc0b6050 con 0x7f24040c6bd0 2023-10-30T12:49:09.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.170+0000 7f23eb7fe640 1 --2- 172.21.15.89:0/4163770905 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23f00664a0 0x7f23f0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:09.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.170+0000 7f23eb7fe640 1 -- 172.21.15.89:0/4163770905 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6664+0+0 (secure 0 0 0) 0x7f23fc122620 con 0x7f24040c6bd0 2023-10-30T12:49:09.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.170+0000 7f2402d9d640 1 --2- 172.21.15.89:0/4163770905 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23f00664a0 0x7f23f0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:09.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.170+0000 7f2402d9d640 1 --2- 172.21.15.89:0/4163770905 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23f00664a0 0x7f23f0068960 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f23f4005890 tx=0x7f23f4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:09.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.174+0000 7f23eb7fe640 1 -- 172.21.15.89:0/4163770905 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f23fc0ef6f0 con 0x7f24040c6bd0 2023-10-30T12:49:09.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.354+0000 7f240913e640 1 -- 172.21.15.89:0/4163770905 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2404154db0 con 0x7f24040c6bd0 2023-10-30T12:49:09.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.354+0000 7f23eb7fe640 1 -- 172.21.15.89:0/4163770905 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+5743 (secure 0 0 0) 0x7f23fc0f2d10 con 0x7f24040c6bd0 2023-10-30T12:49:09.353 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:49:09.353 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:49:01.291912+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5678,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5678":{"gid":5678,"name":"f","rank":0,"incarnation":98,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:09.354 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 98 2023-10-30T12:49:09.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.358+0000 7f240913e640 1 -- 172.21.15.89:0/4163770905 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23f00664a0 msgr2=0x7f23f0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:09.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.358+0000 7f240913e640 1 --2- 172.21.15.89:0/4163770905 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23f00664a0 0x7f23f0068960 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f23f4005890 tx=0x7f23f4009040 comp rx=0 tx=0).stop 2023-10-30T12:49:09.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.358+0000 7f240913e640 1 -- 172.21.15.89:0/4163770905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24040c6bd0 msgr2=0x7f2404156d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:09.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.358+0000 7f240913e640 1 --2- 172.21.15.89:0/4163770905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24040c6bd0 0x7f2404156d40 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f23fc0b2040 tx=0x7f23fc01bed0 comp rx=0 tx=0).stop 2023-10-30T12:49:09.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.358+0000 7f240913e640 1 -- 172.21.15.89:0/4163770905 shutdown_connections 2023-10-30T12:49:09.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.358+0000 7f240913e640 1 --2- 172.21.15.89:0/4163770905 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23f00664a0 0x7f23f0068960 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:09.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.358+0000 7f240913e640 1 --2- 172.21.15.89:0/4163770905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24040c6bd0 0x7f2404156d40 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:09.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.358+0000 7f240913e640 1 --2- 172.21.15.89:0/4163770905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2404071110 0x7f24041545c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:09.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.358+0000 7f240913e640 1 --2- 172.21.15.89:0/4163770905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f240406f7a0 0x7f24041f0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:09.357 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.358+0000 7f240913e640 1 -- 172.21.15.89:0/4163770905 >> 172.21.15.89:0/4163770905 conn(0x7f24040c7910 msgr2=0x7f24040ca5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:09.357 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.358+0000 7f240913e640 1 -- 172.21.15.89:0/4163770905 shutdown_connections 2023-10-30T12:49:09.357 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:09.358+0000 7f240913e640 1 -- 172.21.15.89:0/4163770905 wait complete. 2023-10-30T12:49:09.374 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:49:11.375 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:11.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.526+0000 7f41f8e79640 1 Processor -- start 2023-10-30T12:49:11.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.526+0000 7f41f8e79640 1 -- start start 2023-10-30T12:49:11.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.526+0000 7f41f8e79640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f41f41538a0 0x7f41f415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:11.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.526+0000 7f41f8e79640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41f40cf160 0x7f41f40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:11.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.526+0000 7f41f8e79640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f40cb800 con 0x7f41f40ca0e0 2023-10-30T12:49:11.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.526+0000 7f41f8e79640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41f415dc30 con 0x7f41f41538a0 2023-10-30T12:49:11.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.526+0000 7f41f8e79640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41f40cb330 con 0x7f41f40cf160 2023-10-30T12:49:11.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.526+0000 7f41f1d9b640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f41f40ca0e0 0x7f41f415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36474/0 (socket says 172.21.15.89:36474) 2023-10-30T12:49:11.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.526+0000 7f41f1d9b640 1 -- 172.21.15.89:0/875080396 learned_addr learned my addr 172.21.15.89:0/875080396 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:49:11.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.526+0000 7f41f2d9d640 1 --2- 172.21.15.89:0/875080396 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41f40cf160 0x7f41f40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:11.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.526+0000 7f41f259c640 1 --2- 172.21.15.89:0/875080396 >> v2:172.21.15.192:3300/0 conn(0x7f41f41538a0 0x7f41f415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:11.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.526+0000 7f41f159a640 1 -- 172.21.15.89:0/875080396 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3513814312 0 0) 0x7f41f40cb800 con 0x7f41f40ca0e0 2023-10-30T12:49:11.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.526+0000 7f41f159a640 1 -- 172.21.15.89:0/875080396 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41dc003530 con 0x7f41f40ca0e0 2023-10-30T12:49:11.529 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.526+0000 7f41f159a640 1 -- 172.21.15.89:0/875080396 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3873003164 0 0) 0x7f41dc003530 con 0x7f41f40ca0e0 2023-10-30T12:49:11.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.526+0000 7f41f159a640 1 -- 172.21.15.89:0/875080396 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41f40cf160 msgr2=0x7f41f40cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:11.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.526+0000 7f41f159a640 1 --2- 172.21.15.89:0/875080396 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41f40cf160 0x7f41f40cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:11.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.526+0000 7f41f159a640 1 -- 172.21.15.89:0/875080396 >> v2:172.21.15.192:3300/0 conn(0x7f41f41538a0 msgr2=0x7f41f415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:11.530 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.526+0000 7f41f159a640 1 --2- 172.21.15.89:0/875080396 >> v2:172.21.15.192:3300/0 conn(0x7f41f41538a0 0x7f41f415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:11.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.526+0000 7f41f159a640 1 -- 172.21.15.89:0/875080396 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f40cad70 con 0x7f41f40ca0e0 2023-10-30T12:49:11.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.526+0000 7f41f159a640 1 -- 172.21.15.89:0/875080396 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f41e8001680 con 0x7f41f40ca0e0 2023-10-30T12:49:11.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.530+0000 7f41f159a640 1 -- 172.21.15.89:0/875080396 >> v1:172.21.15.89:6789/0 conn(0x7f41f40ca0e0 legacy=0x7f41f415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:11.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.530+0000 7f41f159a640 1 --2- 172.21.15.89:0/875080396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41dc003980 0x7f41dc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:11.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.530+0000 7f41f159a640 1 --2- 172.21.15.89:0/875080396 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41f40cf160 0x7f41dc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:11.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.530+0000 7f41f159a640 1 --2- 172.21.15.89:0/875080396 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41dc004d10 0x7f41dc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:11.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.530+0000 7f41f159a640 1 -- 172.21.15.89:0/875080396 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f415dc30 con 0x7f41dc003980 2023-10-30T12:49:11.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.530+0000 7f41f159a640 1 -- 172.21.15.89:0/875080396 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f40cb330 con 0x7f41dc004d10 2023-10-30T12:49:11.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.530+0000 7f41f159a640 1 -- 172.21.15.89:0/875080396 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc00b990 con 0x7f41f40cf160 2023-10-30T12:49:11.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.530+0000 7f41f259c640 1 --2- 172.21.15.89:0/875080396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41dc003980 0x7f41dc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:11.532 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.530+0000 7f41f1d9b640 1 --2- 172.21.15.89:0/875080396 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41f40cf160 0x7f41dc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:11.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.530+0000 7f41f2d9d640 1 --2- 172.21.15.89:0/875080396 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41dc004d10 0x7f41dc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:11.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.530+0000 7f41f259c640 1 -- 172.21.15.89:0/875080396 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41f40cf160 msgr2=0x7f41dc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:11.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.530+0000 7f41f259c640 1 --2- 172.21.15.89:0/875080396 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41f40cf160 0x7f41dc0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:11.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.530+0000 7f41f259c640 1 -- 172.21.15.89:0/875080396 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41dc004d10 msgr2=0x7f41dc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:11.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.530+0000 7f41f259c640 1 --2- 172.21.15.89:0/875080396 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41dc004d10 0x7f41dc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:11.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.530+0000 7f41f259c640 1 -- 172.21.15.89:0/875080396 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41dc003530 con 0x7f41dc003980 2023-10-30T12:49:11.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.530+0000 7f41f259c640 1 --2- 172.21.15.89:0/875080396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41dc003980 0x7f41dc003da0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f41e4009ae0 tx=0x7f41e401ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ec8bc565e677834 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:11.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.530+0000 7f41f159a640 1 -- 172.21.15.89:0/875080396 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e40a5570 con 0x7f41dc003980 2023-10-30T12:49:11.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.530+0000 7f41f159a640 1 -- 172.21.15.89:0/875080396 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e40a56d0 con 0x7f41dc003980 2023-10-30T12:49:11.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.530+0000 7f41f159a640 1 -- 172.21.15.89:0/875080396 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e40a5a80 con 0x7f41dc003980 2023-10-30T12:49:11.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.534+0000 7f41f8e79640 1 -- 172.21.15.89:0/875080396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41dc003980 msgr2=0x7f41dc003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:11.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.534+0000 7f41f8e79640 1 --2- 172.21.15.89:0/875080396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41dc003980 0x7f41dc003da0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f41e4009ae0 tx=0x7f41e401ca90 comp rx=0 tx=0).stop 2023-10-30T12:49:11.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.534+0000 7f41f2d9d640 1 -- 172.21.15.89:0/875080396 reap_dead start 2023-10-30T12:49:11.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.534+0000 7f41f8e79640 1 -- 172.21.15.89:0/875080396 shutdown_connections 2023-10-30T12:49:11.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.534+0000 7f41f8e79640 1 -- 172.21.15.89:0/875080396 >> 172.21.15.89:0/875080396 conn(0x7f41f406bfe0 msgr2=0x7f41f406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:11.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.534+0000 7f41f8e79640 1 -- 172.21.15.89:0/875080396 shutdown_connections 2023-10-30T12:49:11.538 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.534+0000 7f41f8e79640 1 -- 172.21.15.89:0/875080396 wait complete. 2023-10-30T12:49:11.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.534+0000 7f41f8e79640 1 Processor -- start 2023-10-30T12:49:11.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.534+0000 7f41f8e79640 1 -- start start 2023-10-30T12:49:11.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.534+0000 7f41f8e79640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41dc003980 0x7f41f4158b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:11.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.534+0000 7f41f8e79640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41f415bcc0 0x7f41f41507c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:11.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.534+0000 7f41f8e79640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41f40ce260 0x7f41f41efac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:11.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.534+0000 7f41f8e79640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f406c990 con 0x7f41f415bcc0 2023-10-30T12:49:11.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.534+0000 7f41f8e79640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f415b4e0 con 0x7f41f40ce260 2023-10-30T12:49:11.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.534+0000 7f41f8e79640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41f41ef630 con 0x7f41dc003980 2023-10-30T12:49:11.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.534+0000 7f41f1d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41f415bcc0 0x7f41f41507c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:11.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.538+0000 7f41f1d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41f415bcc0 0x7f41f41507c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56016/0 (socket says 172.21.15.89:56016) 2023-10-30T12:49:11.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.538+0000 7f41f1d9b640 1 -- 172.21.15.89:0/2645823203 learned_addr learned my addr 172.21.15.89:0/2645823203 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:49:11.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.538+0000 7f41f2d9d640 1 --2- 172.21.15.89:0/2645823203 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41f40ce260 0x7f41f41efac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:11.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.538+0000 7f41f259c640 1 --2- 172.21.15.89:0/2645823203 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41dc003980 0x7f41f4158b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:11.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.538+0000 7f41f1d9b640 1 -- 172.21.15.89:0/2645823203 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41dc003980 msgr2=0x7f41f4158b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:11.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.538+0000 7f41f1d9b640 1 --2- 172.21.15.89:0/2645823203 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41dc003980 0x7f41f4158b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:11.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.538+0000 7f41f1d9b640 1 -- 172.21.15.89:0/2645823203 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41f40ce260 msgr2=0x7f41f41efac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:11.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.538+0000 7f41f1d9b640 1 --2- 172.21.15.89:0/2645823203 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41f40ce260 0x7f41f41efac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:11.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.538+0000 7f41f1d9b640 1 -- 172.21.15.89:0/2645823203 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f4159100 con 0x7f41f415bcc0 2023-10-30T12:49:11.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.538+0000 7f41f259c640 1 --2- 172.21.15.89:0/2645823203 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41dc003980 0x7f41f4158b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:49:11.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.538+0000 7f41f1d9b640 1 --2- 172.21.15.89:0/2645823203 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41f415bcc0 0x7f41f41507c0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f41e80119b0 tx=0x7f41e8011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:11.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.538+0000 7f41e2ffd640 1 -- 172.21.15.89:0/2645823203 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e800fbc0 con 0x7f41f415bcc0 2023-10-30T12:49:11.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.538+0000 7f41e2ffd640 1 -- 172.21.15.89:0/2645823203 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e800fd20 con 0x7f41f415bcc0 2023-10-30T12:49:11.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.538+0000 7f41f8e79640 1 -- 172.21.15.89:0/2645823203 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41f41593a0 con 0x7f41f415bcc0 2023-10-30T12:49:11.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.538+0000 7f41e2ffd640 1 -- 172.21.15.89:0/2645823203 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e8015650 con 0x7f41f415bcc0 2023-10-30T12:49:11.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.538+0000 7f41f8e79640 1 -- 172.21.15.89:0/2645823203 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41f415b9d0 con 0x7f41f415bcc0 2023-10-30T12:49:11.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.538+0000 7f41f8e79640 1 -- 172.21.15.89:0/2645823203 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41f404e170 con 0x7f41f415bcc0 2023-10-30T12:49:11.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.538+0000 7f41e2ffd640 1 -- 172.21.15.89:0/2645823203 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f41e80157b0 con 0x7f41f415bcc0 2023-10-30T12:49:11.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.538+0000 7f41e2ffd640 1 --2- 172.21.15.89:0/2645823203 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f41d40664a0 0x7f41d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:11.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.542+0000 7f41e2ffd640 1 -- 172.21.15.89:0/2645823203 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6664+0+0 (secure 0 0 0) 0x7f41e8087620 con 0x7f41f415bcc0 2023-10-30T12:49:11.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.542+0000 7f41f259c640 1 --2- 172.21.15.89:0/2645823203 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f41d40664a0 0x7f41d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:11.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.542+0000 7f41f259c640 1 --2- 172.21.15.89:0/2645823203 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f41d40664a0 0x7f41d4068960 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f41e40af000 tx=0x7f41e40046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:11.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.542+0000 7f41e2ffd640 1 -- 172.21.15.89:0/2645823203 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f41e80545b0 con 0x7f41f415bcc0 2023-10-30T12:49:11.723 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.726+0000 7f41f8e79640 1 -- 172.21.15.89:0/2645823203 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41f40b1dd0 con 0x7f41f415bcc0 2023-10-30T12:49:11.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.726+0000 7f41e2ffd640 1 -- 172.21.15.89:0/2645823203 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+5744 (secure 0 0 0) 0x7f41e8057bd0 con 0x7f41f415bcc0 2023-10-30T12:49:11.724 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:49:11.724 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:49:09.411390+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5678,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5678":{"gid":5678,"name":"f","rank":0,"incarnation":98,"state":"up:resolve","state_seq":8,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:11.725 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 99 2023-10-30T12:49:11.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.726+0000 7f41f8e79640 1 -- 172.21.15.89:0/2645823203 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f41d40664a0 msgr2=0x7f41d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:11.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.726+0000 7f41f8e79640 1 --2- 172.21.15.89:0/2645823203 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f41d40664a0 0x7f41d4068960 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f41e40af000 tx=0x7f41e40046d0 comp rx=0 tx=0).stop 2023-10-30T12:49:11.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.726+0000 7f41f8e79640 1 -- 172.21.15.89:0/2645823203 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41f415bcc0 msgr2=0x7f41f41507c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:11.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.726+0000 7f41f8e79640 1 --2- 172.21.15.89:0/2645823203 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41f415bcc0 0x7f41f41507c0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f41e80119b0 tx=0x7f41e8011e70 comp rx=0 tx=0).stop 2023-10-30T12:49:11.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.726+0000 7f41f8e79640 1 -- 172.21.15.89:0/2645823203 shutdown_connections 2023-10-30T12:49:11.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.726+0000 7f41f8e79640 1 --2- 172.21.15.89:0/2645823203 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f41d40664a0 0x7f41d4068960 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:11.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.726+0000 7f41f8e79640 1 --2- 172.21.15.89:0/2645823203 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f41f40ce260 0x7f41f41efac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:11.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.726+0000 7f41f8e79640 1 --2- 172.21.15.89:0/2645823203 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41f415bcc0 0x7f41f41507c0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:11.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.726+0000 7f41f8e79640 1 --2- 172.21.15.89:0/2645823203 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f41dc003980 0x7f41f4158b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:11.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.726+0000 7f41f8e79640 1 -- 172.21.15.89:0/2645823203 >> 172.21.15.89:0/2645823203 conn(0x7f41f406bfe0 msgr2=0x7f41f4152660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:11.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.726+0000 7f41f8e79640 1 -- 172.21.15.89:0/2645823203 shutdown_connections 2023-10-30T12:49:11.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:11.726+0000 7f41f8e79640 1 -- 172.21.15.89:0/2645823203 wait complete. 2023-10-30T12:49:11.744 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:49:13.745 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:13.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.898+0000 7f08b17f8640 1 Processor -- start 2023-10-30T12:49:13.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.898+0000 7f08b17f8640 1 -- start start 2023-10-30T12:49:13.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.898+0000 7f08b17f8640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f08ac0bfec0 0x7f08ac0c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:13.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.898+0000 7f08b17f8640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f08ac06a720 0x7f08ac06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:13.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.898+0000 7f08b17f8640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08ac071290 con 0x7f08ac159e60 2023-10-30T12:49:13.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.898+0000 7f08b17f8640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08ac071570 con 0x7f08ac06a720 2023-10-30T12:49:13.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.898+0000 7f08b17f8640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08ac071850 con 0x7f08ac0bfec0 2023-10-30T12:49:13.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.898+0000 7f08aaffd640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f08ac159e60 0x7f08ac06f540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36486/0 (socket says 172.21.15.89:36486) 2023-10-30T12:49:13.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.898+0000 7f08aaffd640 1 -- 172.21.15.89:0/4005576123 learned_addr learned my addr 172.21.15.89:0/4005576123 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:49:13.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.898+0000 7f08ab7fe640 1 --2- 172.21.15.89:0/4005576123 >> v2:172.21.15.89:3300/0 conn(0x7f08ac06a720 0x7f08ac06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:13.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.898+0000 7f08a27fc640 1 --2- 172.21.15.89:0/4005576123 >> v2:172.21.15.192:3300/0 conn(0x7f08ac0bfec0 0x7f08ac0c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:13.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.898+0000 7f08ab7fe640 1 -- 172.21.15.89:0/4005576123 >> v2:172.21.15.192:3300/0 conn(0x7f08ac0bfec0 msgr2=0x7f08ac0c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:13.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.898+0000 7f08ab7fe640 1 --2- 172.21.15.89:0/4005576123 >> v2:172.21.15.192:3300/0 conn(0x7f08ac0bfec0 0x7f08ac0c1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:13.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.898+0000 7f08ab7fe640 1 -- 172.21.15.89:0/4005576123 >> v1:172.21.15.89:6789/0 conn(0x7f08ac159e60 legacy=0x7f08ac06f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:13.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.898+0000 7f08ab7fe640 1 -- 172.21.15.89:0/4005576123 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ac071b30 con 0x7f08ac06a720 2023-10-30T12:49:13.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.898+0000 7f08aa7fc640 1 -- 172.21.15.89:0/4005576123 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2676290801 0 0) 0x7f08ac071290 con 0x7f08ac159e60 2023-10-30T12:49:13.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.902+0000 7f08ab7fe640 1 --2- 172.21.15.89:0/4005576123 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08ac06a720 0x7f08ac06abb0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f08ac06f200 tx=0x7f08a401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=768a1cb0709c5e5b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:13.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.902+0000 7f08aa7fc640 1 -- 172.21.15.89:0/4005576123 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a40a8950 con 0x7f08ac06a720 2023-10-30T12:49:13.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.902+0000 7f08aa7fc640 1 -- 172.21.15.89:0/4005576123 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a40a8ab0 con 0x7f08ac06a720 2023-10-30T12:49:13.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.902+0000 7f08aa7fc640 1 -- 172.21.15.89:0/4005576123 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a40a8e40 con 0x7f08ac06a720 2023-10-30T12:49:13.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.902+0000 7f08b17f8640 1 -- 172.21.15.89:0/4005576123 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08ac06a720 msgr2=0x7f08ac06abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:13.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.902+0000 7f08b17f8640 1 --2- 172.21.15.89:0/4005576123 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08ac06a720 0x7f08ac06abb0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f08ac06f200 tx=0x7f08a401ec50 comp rx=0 tx=0).stop 2023-10-30T12:49:13.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.902+0000 7f08b17f8640 1 -- 172.21.15.89:0/4005576123 shutdown_connections 2023-10-30T12:49:13.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.902+0000 7f08b17f8640 1 --2- 172.21.15.89:0/4005576123 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08ac06a720 0x7f08ac06abb0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:13.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.902+0000 7f08b17f8640 1 --2- 172.21.15.89:0/4005576123 >> v2:172.21.15.192:3300/0 conn(0x7f08ac0bfec0 0x7f08ac0c1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:13.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.902+0000 7f08b17f8640 1 -- 172.21.15.89:0/4005576123 >> 172.21.15.89:0/4005576123 conn(0x7f08ac0c7910 msgr2=0x7f08ac0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:13.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.902+0000 7f08b17f8640 1 -- 172.21.15.89:0/4005576123 shutdown_connections 2023-10-30T12:49:13.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.902+0000 7f08b17f8640 1 -- 172.21.15.89:0/4005576123 wait complete. 2023-10-30T12:49:13.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.906+0000 7f08b17f8640 1 Processor -- start 2023-10-30T12:49:13.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.906+0000 7f08b17f8640 1 -- start start 2023-10-30T12:49:13.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.906+0000 7f08b17f8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08ac06a720 0x7f08ac1ecf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:13.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.906+0000 7f08b17f8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08ac0bfec0 0x7f08ac0c3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:13.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.906+0000 7f08b17f8640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f08ac159e60 0x7f08ac0c4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:13.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.906+0000 7f08b17f8640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ac1ecaf0 con 0x7f08ac0bfec0 2023-10-30T12:49:13.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.906+0000 7f08b17f8640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ac154770 con 0x7f08ac06a720 2023-10-30T12:49:13.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.906+0000 7f08b17f8640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08ac162560 con 0x7f08ac159e60 2023-10-30T12:49:13.908 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.906+0000 7f08a27fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08ac0bfec0 0x7f08ac0c3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:13.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.906+0000 7f08a27fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08ac0bfec0 0x7f08ac0c3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56024/0 (socket says 172.21.15.89:56024) 2023-10-30T12:49:13.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.906+0000 7f08a27fc640 1 -- 172.21.15.89:0/1421778512 learned_addr learned my addr 172.21.15.89:0/1421778512 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:49:13.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.906+0000 7f08aaffd640 1 --2- 172.21.15.89:0/1421778512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08ac06a720 0x7f08ac1ecf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:13.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.906+0000 7f08ab7fe640 1 --2- 172.21.15.89:0/1421778512 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f08ac159e60 0x7f08ac0c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:13.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.906+0000 7f08a27fc640 1 -- 172.21.15.89:0/1421778512 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f08ac159e60 msgr2=0x7f08ac0c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:13.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.906+0000 7f08a27fc640 1 --2- 172.21.15.89:0/1421778512 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f08ac159e60 0x7f08ac0c4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:13.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.906+0000 7f08a27fc640 1 -- 172.21.15.89:0/1421778512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08ac06a720 msgr2=0x7f08ac1ecf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:13.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.906+0000 7f08a27fc640 1 --2- 172.21.15.89:0/1421778512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08ac06a720 0x7f08ac1ecf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:13.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.906+0000 7f08a27fc640 1 -- 172.21.15.89:0/1421778512 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a4009420 con 0x7f08ac0bfec0 2023-10-30T12:49:13.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.906+0000 7f08a27fc640 1 --2- 172.21.15.89:0/1421778512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08ac0bfec0 0x7f08ac0c3ed0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f0894007f20 tx=0x7f089400e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:13.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.906+0000 7f08a3fff640 1 -- 172.21.15.89:0/1421778512 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0894014620 con 0x7f08ac0bfec0 2023-10-30T12:49:13.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.906+0000 7f08a3fff640 1 -- 172.21.15.89:0/1421778512 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0894014780 con 0x7f08ac0bfec0 2023-10-30T12:49:13.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.906+0000 7f08b17f8640 1 -- 172.21.15.89:0/1421778512 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08ac162280 con 0x7f08ac0bfec0 2023-10-30T12:49:13.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.906+0000 7f08a3fff640 1 -- 172.21.15.89:0/1421778512 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0894014b30 con 0x7f08ac0bfec0 2023-10-30T12:49:13.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.910+0000 7f08b17f8640 1 -- 172.21.15.89:0/1421778512 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08ac1f0a10 con 0x7f08ac0bfec0 2023-10-30T12:49:13.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.910+0000 7f08b17f8640 1 -- 172.21.15.89:0/1421778512 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08ac1ecd60 con 0x7f08ac0bfec0 2023-10-30T12:49:13.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.910+0000 7f08a3fff640 1 -- 172.21.15.89:0/1421778512 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0894014c90 con 0x7f08ac0bfec0 2023-10-30T12:49:13.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.910+0000 7f08a3fff640 1 --2- 172.21.15.89:0/1421778512 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f088c0664a0 0x7f088c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:13.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.910+0000 7f08a3fff640 1 -- 172.21.15.89:0/1421778512 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6664+0+0 (secure 0 0 0) 0x7f0894084c30 con 0x7f08ac0bfec0 2023-10-30T12:49:13.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.910+0000 7f08aaffd640 1 --2- 172.21.15.89:0/1421778512 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f088c0664a0 0x7f088c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:13.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.914+0000 7f08aaffd640 1 --2- 172.21.15.89:0/1421778512 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f088c0664a0 0x7f088c068960 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f089c000f70 tx=0x7f089c007c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:13.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:13.914+0000 7f08a3fff640 1 -- 172.21.15.89:0/1421778512 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0894051ce0 con 0x7f08ac0bfec0 2023-10-30T12:49:14.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:14.094+0000 7f08b17f8640 1 -- 172.21.15.89:0/1421778512 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08ac04e170 con 0x7f08ac0bfec0 2023-10-30T12:49:14.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:14.094+0000 7f08a3fff640 1 -- 172.21.15.89:0/1421778512 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+5744 (secure 0 0 0) 0x7f0894055300 con 0x7f08ac0bfec0 2023-10-30T12:49:14.094 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:49:14.094 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:49:09.411390+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5678,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5678":{"gid":5678,"name":"f","rank":0,"incarnation":98,"state":"up:resolve","state_seq":8,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:14.094 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 99 2023-10-30T12:49:14.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:14.098+0000 7f08b17f8640 1 -- 172.21.15.89:0/1421778512 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f088c0664a0 msgr2=0x7f088c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:14.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:14.098+0000 7f08b17f8640 1 --2- 172.21.15.89:0/1421778512 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f088c0664a0 0x7f088c068960 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f089c000f70 tx=0x7f089c007c10 comp rx=0 tx=0).stop 2023-10-30T12:49:14.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:14.098+0000 7f08b17f8640 1 -- 172.21.15.89:0/1421778512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08ac0bfec0 msgr2=0x7f08ac0c3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:14.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:14.098+0000 7f08b17f8640 1 --2- 172.21.15.89:0/1421778512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08ac0bfec0 0x7f08ac0c3ed0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f0894007f20 tx=0x7f089400e570 comp rx=0 tx=0).stop 2023-10-30T12:49:14.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:14.098+0000 7f08b17f8640 1 -- 172.21.15.89:0/1421778512 shutdown_connections 2023-10-30T12:49:14.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:14.098+0000 7f08b17f8640 1 --2- 172.21.15.89:0/1421778512 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f088c0664a0 0x7f088c068960 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:14.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:14.098+0000 7f08b17f8640 1 --2- 172.21.15.89:0/1421778512 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f08ac159e60 0x7f08ac0c4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:14.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:14.098+0000 7f08b17f8640 1 --2- 172.21.15.89:0/1421778512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08ac0bfec0 0x7f08ac0c3ed0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:14.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:14.098+0000 7f08b17f8640 1 --2- 172.21.15.89:0/1421778512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f08ac06a720 0x7f08ac1ecf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:14.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:14.098+0000 7f08b17f8640 1 -- 172.21.15.89:0/1421778512 >> 172.21.15.89:0/1421778512 conn(0x7f08ac0c7910 msgr2=0x7f08ac0c9fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:14.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:14.098+0000 7f08b17f8640 1 -- 172.21.15.89:0/1421778512 shutdown_connections 2023-10-30T12:49:14.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:14.098+0000 7f08b17f8640 1 -- 172.21.15.89:0/1421778512 wait complete. 2023-10-30T12:49:14.115 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 99, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:50.729245+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 99, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5678": { "addr": "172.21.15.192:6837/2130807115", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6836", "nonce": 2130807115, "type": "v2" }, { "addr": "172.21.15.192:6837", "nonce": 2130807115, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5678, "incarnation": 98, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:resolve", "state_seq": 8 }, "gid_5728": { "addr": "172.21.15.192:6839/3991891597", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6838", "nonce": 3991891597, "type": "v2" }, { "addr": "172.21.15.192:6839", "nonce": 3991891597, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 5728, "incarnation": 73, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 19 }, "gid_6075": { "addr": "172.21.15.89:6837/2412618858", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6836", "nonce": 2412618858, "type": "v2" }, { "addr": "172.21.15.89:6837", "nonce": 2412618858, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6075, "incarnation": 92, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 38 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:49:09.411390+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5678, "mds_1": 6075, "mds_2": 5728 } } } ], "standbys": [ { "addr": "172.21.15.89:6834/2909790173", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6832", "nonce": 2909790173, "type": "v2" }, { "addr": "172.21.15.89:6834", "nonce": 2909790173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.89:6835/2387357615", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6833", "nonce": 2387357615, "type": "v2" }, { "addr": "172.21.15.89:6835", "nonce": 2387357615, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4239, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:49:16.115 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:16.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cdcb0e640 1 Processor -- start 2023-10-30T12:49:16.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cdcb0e640 1 -- start start 2023-10-30T12:49:16.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cdcb0e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cd80c05d0 0x7f2cd813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:16.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cdcb0e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2cd80c6f60 0x7f2cd813f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:16.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cdcb0e640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd80c0a50 con 0x7f2cd80c6b80 2023-10-30T12:49:16.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cdcb0e640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cd806f6e0 con 0x7f2cd80c6f60 2023-10-30T12:49:16.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cdcb0e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd806f9c0 con 0x7f2cd80c05d0 2023-10-30T12:49:16.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cd659c640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2cd80c6b80 0x7f2cd80bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36498/0 (socket says 172.21.15.89:36498) 2023-10-30T12:49:16.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cd659c640 1 -- 172.21.15.89:0/724990554 learned_addr learned my addr 172.21.15.89:0/724990554 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:49:16.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cd5d9b640 1 --2- 172.21.15.89:0/724990554 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cd80c05d0 0x7f2cd813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:16.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cd6d9d640 1 --2- 172.21.15.89:0/724990554 >> v2:172.21.15.192:3300/0 conn(0x7f2cd80c6f60 0x7f2cd813f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:16.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cd559a640 1 -- 172.21.15.89:0/724990554 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2207620484 0 0) 0x7f2cd80c0a50 con 0x7f2cd80c6b80 2023-10-30T12:49:16.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cd559a640 1 -- 172.21.15.89:0/724990554 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cc0005740 con 0x7f2cd80c6b80 2023-10-30T12:49:16.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cd559a640 1 -- 172.21.15.89:0/724990554 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1992202784 0 0) 0x7f2cc0005740 con 0x7f2cd80c6b80 2023-10-30T12:49:16.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cd559a640 1 -- 172.21.15.89:0/724990554 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cd80c05d0 msgr2=0x7f2cd813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:16.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cd559a640 1 --2- 172.21.15.89:0/724990554 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cd80c05d0 0x7f2cd813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:16.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cd559a640 1 -- 172.21.15.89:0/724990554 >> v2:172.21.15.192:3300/0 conn(0x7f2cd80c6f60 msgr2=0x7f2cd813f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:16.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cd559a640 1 --2- 172.21.15.89:0/724990554 >> v2:172.21.15.192:3300/0 conn(0x7f2cd80c6f60 0x7f2cd813f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:16.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cd559a640 1 -- 172.21.15.89:0/724990554 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd806fca0 con 0x7f2cd80c6b80 2023-10-30T12:49:16.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cd559a640 1 -- 172.21.15.89:0/724990554 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f2cc8001680 con 0x7f2cd80c6b80 2023-10-30T12:49:16.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cd559a640 1 -- 172.21.15.89:0/724990554 >> v1:172.21.15.89:6789/0 conn(0x7f2cd80c6b80 legacy=0x7f2cd80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:16.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cd559a640 1 --2- 172.21.15.89:0/724990554 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cc0005b90 0x7f2cc0005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:16.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cd559a640 1 --2- 172.21.15.89:0/724990554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cc00069e0 0x7f2cc0006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:16.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cd559a640 1 --2- 172.21.15.89:0/724990554 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cd80c05d0 0x7f2cc00073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:16.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cd559a640 1 -- 172.21.15.89:0/724990554 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd806f6e0 con 0x7f2cc00069e0 2023-10-30T12:49:16.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cd559a640 1 -- 172.21.15.89:0/724990554 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd806f9c0 con 0x7f2cc0005b90 2023-10-30T12:49:16.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cd559a640 1 -- 172.21.15.89:0/724990554 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc000dba0 con 0x7f2cd80c05d0 2023-10-30T12:49:16.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cd5d9b640 1 --2- 172.21.15.89:0/724990554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cc00069e0 0x7f2cc0006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:16.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cd6d9d640 1 --2- 172.21.15.89:0/724990554 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cd80c05d0 0x7f2cc00073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:16.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cd659c640 1 --2- 172.21.15.89:0/724990554 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cc0005b90 0x7f2cc0005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:16.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cd5d9b640 1 -- 172.21.15.89:0/724990554 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cd80c05d0 msgr2=0x7f2cc00073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:16.272 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cd5d9b640 1 --2- 172.21.15.89:0/724990554 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cd80c05d0 0x7f2cc00073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:16.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cd5d9b640 1 -- 172.21.15.89:0/724990554 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cc0005b90 msgr2=0x7f2cc0005fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:16.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cd5d9b640 1 --2- 172.21.15.89:0/724990554 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cc0005b90 0x7f2cc0005fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:16.273 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.266+0000 7f2cd5d9b640 1 -- 172.21.15.89:0/724990554 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc0005740 con 0x7f2cc00069e0 2023-10-30T12:49:16.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.270+0000 7f2cd5d9b640 1 --2- 172.21.15.89:0/724990554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cc00069e0 0x7f2cc0006e60 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc009ae0 tx=0x7f2ccc01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb0690a80bef294e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:16.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.270+0000 7f2cd559a640 1 -- 172.21.15.89:0/724990554 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc0a6460 con 0x7f2cc00069e0 2023-10-30T12:49:16.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.270+0000 7f2cd559a640 1 -- 172.21.15.89:0/724990554 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ccc0a65c0 con 0x7f2cc00069e0 2023-10-30T12:49:16.277 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.270+0000 7f2cd559a640 1 -- 172.21.15.89:0/724990554 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc0a6970 con 0x7f2cc00069e0 2023-10-30T12:49:16.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.270+0000 7f2cdcb0e640 1 -- 172.21.15.89:0/724990554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cc00069e0 msgr2=0x7f2cc0006e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:16.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.270+0000 7f2cdcb0e640 1 --2- 172.21.15.89:0/724990554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cc00069e0 0x7f2cc0006e60 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc009ae0 tx=0x7f2ccc01ca50 comp rx=0 tx=0).stop 2023-10-30T12:49:16.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.270+0000 7f2cd6d9d640 1 -- 172.21.15.89:0/724990554 reap_dead start 2023-10-30T12:49:16.278 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.270+0000 7f2cdcb0e640 1 -- 172.21.15.89:0/724990554 shutdown_connections 2023-10-30T12:49:16.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.270+0000 7f2cdcb0e640 1 -- 172.21.15.89:0/724990554 >> 172.21.15.89:0/724990554 conn(0x7f2cd80c7910 msgr2=0x7f2cd80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:16.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.270+0000 7f2cdcb0e640 1 -- 172.21.15.89:0/724990554 shutdown_connections 2023-10-30T12:49:16.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.270+0000 7f2cdcb0e640 1 -- 172.21.15.89:0/724990554 wait complete. 2023-10-30T12:49:16.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.274+0000 7f2cdcb0e640 1 Processor -- start 2023-10-30T12:49:16.279 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.274+0000 7f2cdcb0e640 1 -- start start 2023-10-30T12:49:16.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.274+0000 7f2cdcb0e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cc00069e0 0x7f2cd81eb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:16.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.274+0000 7f2cdcb0e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cd8153d20 0x7f2cd8135e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:16.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.274+0000 7f2cdcb0e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cd81dda20 0x7f2cd80c9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:16.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.274+0000 7f2cdcb0e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd8136380 con 0x7f2cc00069e0 2023-10-30T12:49:16.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.274+0000 7f2cdcb0e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd81353f0 con 0x7f2cd8153d20 2023-10-30T12:49:16.280 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.274+0000 7f2cdcb0e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd81359b0 con 0x7f2cd81dda20 2023-10-30T12:49:16.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.274+0000 7f2cd659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cc00069e0 0x7f2cd81eb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:16.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.274+0000 7f2cd659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cc00069e0 0x7f2cd81eb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56038/0 (socket says 172.21.15.89:56038) 2023-10-30T12:49:16.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.274+0000 7f2cd659c640 1 -- 172.21.15.89:0/3409071055 learned_addr learned my addr 172.21.15.89:0/3409071055 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:49:16.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.274+0000 7f2cd5d9b640 1 --2- 172.21.15.89:0/3409071055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cd8153d20 0x7f2cd8135e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:16.281 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.274+0000 7f2cd6d9d640 1 --2- 172.21.15.89:0/3409071055 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cd81dda20 0x7f2cd80c9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:16.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.274+0000 7f2cd659c640 1 -- 172.21.15.89:0/3409071055 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cd81dda20 msgr2=0x7f2cd80c9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:16.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.274+0000 7f2cd659c640 1 --2- 172.21.15.89:0/3409071055 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cd81dda20 0x7f2cd80c9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:16.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.274+0000 7f2cd659c640 1 -- 172.21.15.89:0/3409071055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cd8153d20 msgr2=0x7f2cd8135e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:16.282 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.274+0000 7f2cd659c640 1 --2- 172.21.15.89:0/3409071055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cd8153d20 0x7f2cd8135e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:16.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.274+0000 7f2cd659c640 1 -- 172.21.15.89:0/3409071055 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd81eb850 con 0x7f2cc00069e0 2023-10-30T12:49:16.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.274+0000 7f2cd659c640 1 --2- 172.21.15.89:0/3409071055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cc00069e0 0x7f2cd81eb260 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8011880 tx=0x7f2cc8011d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:16.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.274+0000 7f2cc6ffd640 1 -- 172.21.15.89:0/3409071055 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc800fb00 con 0x7f2cc00069e0 2023-10-30T12:49:16.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.274+0000 7f2cc6ffd640 1 -- 172.21.15.89:0/3409071055 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cc800fc80 con 0x7f2cc00069e0 2023-10-30T12:49:16.283 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.274+0000 7f2cdcb0e640 1 -- 172.21.15.89:0/3409071055 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cd81dd1a0 con 0x7f2cc00069e0 2023-10-30T12:49:16.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.274+0000 7f2cc6ffd640 1 -- 172.21.15.89:0/3409071055 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc8015410 con 0x7f2cc00069e0 2023-10-30T12:49:16.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.274+0000 7f2cdcb0e640 1 -- 172.21.15.89:0/3409071055 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cd8135640 con 0x7f2cc00069e0 2023-10-30T12:49:16.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.278+0000 7f2cdcb0e640 1 -- 172.21.15.89:0/3409071055 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cd804e170 con 0x7f2cc00069e0 2023-10-30T12:49:16.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.278+0000 7f2cc6ffd640 1 -- 172.21.15.89:0/3409071055 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2cc8015570 con 0x7f2cc00069e0 2023-10-30T12:49:16.284 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.278+0000 7f2cc6ffd640 1 --2- 172.21.15.89:0/3409071055 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2cb80664d0 0x7f2cb8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:16.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.278+0000 7f2cc6ffd640 1 -- 172.21.15.89:0/3409071055 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6664+0+0 (secure 0 0 0) 0x7f2cc8087340 con 0x7f2cc00069e0 2023-10-30T12:49:16.285 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.278+0000 7f2cd5d9b640 1 --2- 172.21.15.89:0/3409071055 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2cb80664d0 0x7f2cb8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:16.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.278+0000 7f2cd5d9b640 1 --2- 172.21.15.89:0/3409071055 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2cb80664d0 0x7f2cb8068990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc0ad000 tx=0x7f2ccc0b3040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:16.286 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.282+0000 7f2cc6ffd640 1 -- 172.21.15.89:0/3409071055 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2cc8054540 con 0x7f2cc00069e0 2023-10-30T12:49:16.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.466+0000 7f2cdcb0e640 1 -- 172.21.15.89:0/3409071055 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2cd80c1670 con 0x7f2cc00069e0 2023-10-30T12:49:16.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.466+0000 7f2cc6ffd640 1 -- 172.21.15.89:0/3409071055 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+5749 (secure 0 0 0) 0x7f2cc8057b60 con 0x7f2cc00069e0 2023-10-30T12:49:16.466 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:49:16.466 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:49:14.524095+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5678,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5678":{"gid":5678,"name":"f","rank":0,"incarnation":98,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:16.467 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 100 2023-10-30T12:49:16.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.470+0000 7f2cdcb0e640 1 -- 172.21.15.89:0/3409071055 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2cb80664d0 msgr2=0x7f2cb8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:16.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.470+0000 7f2cdcb0e640 1 --2- 172.21.15.89:0/3409071055 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2cb80664d0 0x7f2cb8068990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc0ad000 tx=0x7f2ccc0b3040 comp rx=0 tx=0).stop 2023-10-30T12:49:16.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.470+0000 7f2cdcb0e640 1 -- 172.21.15.89:0/3409071055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cc00069e0 msgr2=0x7f2cd81eb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:16.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.470+0000 7f2cdcb0e640 1 --2- 172.21.15.89:0/3409071055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cc00069e0 0x7f2cd81eb260 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8011880 tx=0x7f2cc8011d40 comp rx=0 tx=0).stop 2023-10-30T12:49:16.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.470+0000 7f2cdcb0e640 1 -- 172.21.15.89:0/3409071055 shutdown_connections 2023-10-30T12:49:16.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.470+0000 7f2cdcb0e640 1 --2- 172.21.15.89:0/3409071055 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2cb80664d0 0x7f2cb8068990 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:16.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.470+0000 7f2cdcb0e640 1 --2- 172.21.15.89:0/3409071055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cd8153d20 0x7f2cd8135e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:16.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.470+0000 7f2cdcb0e640 1 --2- 172.21.15.89:0/3409071055 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cd81dda20 0x7f2cd80c9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:16.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.470+0000 7f2cdcb0e640 1 --2- 172.21.15.89:0/3409071055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cc00069e0 0x7f2cd81eb260 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:16.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.470+0000 7f2cdcb0e640 1 -- 172.21.15.89:0/3409071055 >> 172.21.15.89:0/3409071055 conn(0x7f2cd80c7910 msgr2=0x7f2cd8136930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:16.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.470+0000 7f2cdcb0e640 1 -- 172.21.15.89:0/3409071055 shutdown_connections 2023-10-30T12:49:16.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:16.470+0000 7f2cdcb0e640 1 -- 172.21.15.89:0/3409071055 wait complete. 2023-10-30T12:49:16.487 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:49:18.291 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:49:18.296 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:49:18.488 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:18.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.638+0000 7f8806bf5640 1 Processor -- start 2023-10-30T12:49:18.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.638+0000 7f8806bf5640 1 -- start start 2023-10-30T12:49:18.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.638+0000 7f8806bf5640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f88000c05d0 0x7f880013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:18.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.638+0000 7f8806bf5640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88000c0a50 con 0x7f88000c6f60 2023-10-30T12:49:18.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.638+0000 7f8806bf5640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f880006f6e0 con 0x7f88000c6b80 2023-10-30T12:49:18.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.638+0000 7f8806bf5640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f880006f9c0 con 0x7f88000c05d0 2023-10-30T12:49:18.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.638+0000 7f8805192640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f88000c6f60 0x7f88001566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36512/0 (socket says 172.21.15.89:36512) 2023-10-30T12:49:18.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.638+0000 7f8805192640 1 -- 172.21.15.89:0/4276814342 learned_addr learned my addr 172.21.15.89:0/4276814342 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:49:18.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.638+0000 7f87fffff640 1 --2- 172.21.15.89:0/4276814342 >> v2:172.21.15.192:3300/0 conn(0x7f88000c05d0 0x7f880013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:18.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.638+0000 7f87ff7fe640 1 -- 172.21.15.89:0/4276814342 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1183590986 0 0) 0x7f88000c0a50 con 0x7f88000c6f60 2023-10-30T12:49:18.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.638+0000 7f87ff7fe640 1 -- 172.21.15.89:0/4276814342 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87f0005740 con 0x7f88000c6f60 2023-10-30T12:49:18.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.642+0000 7f87ff7fe640 1 -- 172.21.15.89:0/4276814342 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 546034477 0 0) 0x7f87f0005740 con 0x7f88000c6f60 2023-10-30T12:49:18.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.642+0000 7f87ff7fe640 1 -- 172.21.15.89:0/4276814342 >> v2:172.21.15.192:3300/0 conn(0x7f88000c05d0 msgr2=0x7f880013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:18.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.642+0000 7f87ff7fe640 1 --2- 172.21.15.89:0/4276814342 >> v2:172.21.15.192:3300/0 conn(0x7f88000c05d0 0x7f880013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:18.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.642+0000 7f87ff7fe640 1 -- 172.21.15.89:0/4276814342 >> v1:172.21.15.192:6789/0 conn(0x7f88000c6b80 legacy=0x7f88000bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:18.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.642+0000 7f87ff7fe640 1 -- 172.21.15.89:0/4276814342 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f880006fca0 con 0x7f88000c6f60 2023-10-30T12:49:18.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.642+0000 7f87ff7fe640 1 -- 172.21.15.89:0/4276814342 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f87f8001a00 con 0x7f88000c6f60 2023-10-30T12:49:18.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.642+0000 7f87ff7fe640 1 -- 172.21.15.89:0/4276814342 >> v1:172.21.15.89:6789/0 conn(0x7f88000c6f60 legacy=0x7f88001566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:18.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.642+0000 7f87ff7fe640 1 --2- 172.21.15.89:0/4276814342 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87f0005b90 0x7f87f0005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:18.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.642+0000 7f87ff7fe640 1 --2- 172.21.15.89:0/4276814342 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87f00069e0 0x7f87f0006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:18.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.642+0000 7f87ff7fe640 1 --2- 172.21.15.89:0/4276814342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87f00073a0 0x7f87f000dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:18.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.642+0000 7f87ff7fe640 1 -- 172.21.15.89:0/4276814342 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f880006f9c0 con 0x7f87f00073a0 2023-10-30T12:49:18.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.642+0000 7f87ff7fe640 1 -- 172.21.15.89:0/4276814342 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f000e240 con 0x7f87f0005b90 2023-10-30T12:49:18.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.642+0000 7f87ff7fe640 1 -- 172.21.15.89:0/4276814342 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f000e400 con 0x7f87f00069e0 2023-10-30T12:49:18.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.642+0000 7f8805192640 1 --2- 172.21.15.89:0/4276814342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87f00073a0 0x7f87f000dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:18.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.642+0000 7f8804991640 1 --2- 172.21.15.89:0/4276814342 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87f0005b90 0x7f87f0005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:18.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.642+0000 7f87fffff640 1 --2- 172.21.15.89:0/4276814342 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87f00069e0 0x7f87f0006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:18.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.642+0000 7f8805192640 1 -- 172.21.15.89:0/4276814342 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87f00069e0 msgr2=0x7f87f0006e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:18.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.642+0000 7f8805192640 1 --2- 172.21.15.89:0/4276814342 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f87f00069e0 0x7f87f0006e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:18.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.642+0000 7f8805192640 1 -- 172.21.15.89:0/4276814342 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87f0005b90 msgr2=0x7f87f0005fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:18.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.642+0000 7f8805192640 1 --2- 172.21.15.89:0/4276814342 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87f0005b90 0x7f87f0005fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:18.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.642+0000 7f8805192640 1 -- 172.21.15.89:0/4276814342 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f0005740 con 0x7f87f00073a0 2023-10-30T12:49:18.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.642+0000 7f8805192640 1 -- 172.21.15.89:0/4276814342 reap_dead start 2023-10-30T12:49:18.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.642+0000 7f8805192640 1 --2- 172.21.15.89:0/4276814342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87f00073a0 0x7f87f000dba0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f87f800a830 tx=0x7f87f801d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=726447fee1d60a4b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:18.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.646+0000 7f87ff7fe640 1 -- 172.21.15.89:0/4276814342 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f80072d0 con 0x7f87f00073a0 2023-10-30T12:49:18.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.646+0000 7f87ff7fe640 1 -- 172.21.15.89:0/4276814342 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f80a8070 con 0x7f87f00073a0 2023-10-30T12:49:18.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.646+0000 7f87ff7fe640 1 -- 172.21.15.89:0/4276814342 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f80076b0 con 0x7f87f00073a0 2023-10-30T12:49:18.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.646+0000 7f8806bf5640 1 -- 172.21.15.89:0/4276814342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87f00073a0 msgr2=0x7f87f000dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:18.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.646+0000 7f8806bf5640 1 --2- 172.21.15.89:0/4276814342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87f00073a0 0x7f87f000dba0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f87f800a830 tx=0x7f87f801d340 comp rx=0 tx=0).stop 2023-10-30T12:49:18.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.646+0000 7f8806bf5640 1 -- 172.21.15.89:0/4276814342 shutdown_connections 2023-10-30T12:49:18.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.646+0000 7f8806bf5640 1 --2- 172.21.15.89:0/4276814342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87f00073a0 0x7f87f000dba0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:18.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.646+0000 7f8806bf5640 1 -- 172.21.15.89:0/4276814342 >> 172.21.15.89:0/4276814342 conn(0x7f88000c7910 msgr2=0x7f88000c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:18.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.646+0000 7f8806bf5640 1 -- 172.21.15.89:0/4276814342 shutdown_connections 2023-10-30T12:49:18.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.646+0000 7f8806bf5640 1 -- 172.21.15.89:0/4276814342 wait complete. 2023-10-30T12:49:18.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.646+0000 7f8806bf5640 1 Processor -- start 2023-10-30T12:49:18.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.646+0000 7f8806bf5640 1 -- start start 2023-10-30T12:49:18.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.646+0000 7f8806bf5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87f00073a0 0x7f88001f24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:18.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.646+0000 7f8806bf5640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88000c0fc0 0x7f88001f0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:18.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.646+0000 7f8806bf5640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88001f1480 0x7f880013db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:18.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.646+0000 7f8806bf5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88001f1900 con 0x7f87f00073a0 2023-10-30T12:49:18.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.646+0000 7f8806bf5640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88001f2a30 con 0x7f88001f1480 2023-10-30T12:49:18.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.646+0000 7f8806bf5640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88000c9960 con 0x7f88000c0fc0 2023-10-30T12:49:18.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.646+0000 7f8804991640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87f00073a0 0x7f88001f24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:18.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.646+0000 7f8804991640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87f00073a0 0x7f88001f24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56056/0 (socket says 172.21.15.89:56056) 2023-10-30T12:49:18.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.646+0000 7f8804991640 1 -- 172.21.15.89:0/1114280826 learned_addr learned my addr 172.21.15.89:0/1114280826 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:49:18.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.650+0000 7f87fffff640 1 --2- 172.21.15.89:0/1114280826 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88000c0fc0 0x7f88001f0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:18.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.650+0000 7f8805192640 1 --2- 172.21.15.89:0/1114280826 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88001f1480 0x7f880013db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:18.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.650+0000 7f8804991640 1 -- 172.21.15.89:0/1114280826 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88000c0fc0 msgr2=0x7f88001f0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:18.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.650+0000 7f8804991640 1 --2- 172.21.15.89:0/1114280826 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88000c0fc0 0x7f88001f0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:18.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.650+0000 7f8804991640 1 -- 172.21.15.89:0/1114280826 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88001f1480 msgr2=0x7f880013db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:18.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.650+0000 7f8804991640 1 --2- 172.21.15.89:0/1114280826 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88001f1480 0x7f880013db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:18.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.650+0000 7f8804991640 1 -- 172.21.15.89:0/1114280826 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88000c9c40 con 0x7f87f00073a0 2023-10-30T12:49:18.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.650+0000 7f8804991640 1 --2- 172.21.15.89:0/1114280826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87f00073a0 0x7f88001f24f0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f87f400f5f0 tx=0x7f87f400fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:18.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.650+0000 7f87fd7fa640 1 -- 172.21.15.89:0/1114280826 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f40052e0 con 0x7f87f00073a0 2023-10-30T12:49:18.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.650+0000 7f87fd7fa640 1 -- 172.21.15.89:0/1114280826 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f4005440 con 0x7f87f00073a0 2023-10-30T12:49:18.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.650+0000 7f8806bf5640 1 -- 172.21.15.89:0/1114280826 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88000c9f20 con 0x7f87f00073a0 2023-10-30T12:49:18.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.650+0000 7f87fd7fa640 1 -- 172.21.15.89:0/1114280826 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f4003820 con 0x7f87f00073a0 2023-10-30T12:49:18.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.650+0000 7f8806bf5640 1 -- 172.21.15.89:0/1114280826 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f880013d5f0 con 0x7f87f00073a0 2023-10-30T12:49:18.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.650+0000 7f8806bf5640 1 -- 172.21.15.89:0/1114280826 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f880004e170 con 0x7f87f00073a0 2023-10-30T12:49:18.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.650+0000 7f87fd7fa640 1 -- 172.21.15.89:0/1114280826 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f87f4003980 con 0x7f87f00073a0 2023-10-30T12:49:18.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.650+0000 7f87fd7fa640 1 --2- 172.21.15.89:0/1114280826 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f87e40664a0 0x7f87e4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:18.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.650+0000 7f87fd7fa640 1 -- 172.21.15.89:0/1114280826 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6664+0+0 (secure 0 0 0) 0x7f87f4084190 con 0x7f87f00073a0 2023-10-30T12:49:18.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.650+0000 7f87fffff640 1 --2- 172.21.15.89:0/1114280826 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f87e40664a0 0x7f87e4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:18.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.654+0000 7f87fffff640 1 --2- 172.21.15.89:0/1114280826 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f87e40664a0 0x7f87e4068960 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8800072470 tx=0x7f87ec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:18.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.654+0000 7f87fd7fa640 1 -- 172.21.15.89:0/1114280826 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f87f4051240 con 0x7f87f00073a0 2023-10-30T12:49:18.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.834+0000 7f8806bf5640 1 -- 172.21.15.89:0/1114280826 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88000c2050 con 0x7f87f00073a0 2023-10-30T12:49:18.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.838+0000 7f87fd7fa640 1 -- 172.21.15.89:0/1114280826 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+5749 (secure 0 0 0) 0x7f87f4054860 con 0x7f87f00073a0 2023-10-30T12:49:18.835 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:49:18.835 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:49:14.524095+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5678,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5678":{"gid":5678,"name":"f","rank":0,"incarnation":98,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:18.836 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 100 2023-10-30T12:49:18.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.838+0000 7f8806bf5640 1 -- 172.21.15.89:0/1114280826 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f87e40664a0 msgr2=0x7f87e4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:18.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.838+0000 7f8806bf5640 1 --2- 172.21.15.89:0/1114280826 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f87e40664a0 0x7f87e4068960 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8800072470 tx=0x7f87ec008040 comp rx=0 tx=0).stop 2023-10-30T12:49:18.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.838+0000 7f8806bf5640 1 -- 172.21.15.89:0/1114280826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87f00073a0 msgr2=0x7f88001f24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:18.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.838+0000 7f8806bf5640 1 --2- 172.21.15.89:0/1114280826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87f00073a0 0x7f88001f24f0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f87f400f5f0 tx=0x7f87f400fab0 comp rx=0 tx=0).stop 2023-10-30T12:49:18.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.838+0000 7f8806bf5640 1 -- 172.21.15.89:0/1114280826 shutdown_connections 2023-10-30T12:49:18.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.838+0000 7f8806bf5640 1 --2- 172.21.15.89:0/1114280826 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f87e40664a0 0x7f87e4068960 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:18.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.838+0000 7f8806bf5640 1 --2- 172.21.15.89:0/1114280826 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f88001f1480 0x7f880013db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:18.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.838+0000 7f8806bf5640 1 --2- 172.21.15.89:0/1114280826 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f88000c0fc0 0x7f88001f0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:18.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.838+0000 7f8806bf5640 1 --2- 172.21.15.89:0/1114280826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87f00073a0 0x7f88001f24f0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:18.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.838+0000 7f8806bf5640 1 -- 172.21.15.89:0/1114280826 >> 172.21.15.89:0/1114280826 conn(0x7f88000c7910 msgr2=0x7f88000bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:18.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.838+0000 7f8806bf5640 1 -- 172.21.15.89:0/1114280826 shutdown_connections 2023-10-30T12:49:18.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:18.838+0000 7f8806bf5640 1 -- 172.21.15.89:0/1114280826 wait complete. 2023-10-30T12:49:18.855 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:49:20.856 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:21.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.006+0000 7f4a9cf0f640 1 Processor -- start 2023-10-30T12:49:21.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.006+0000 7f4a9cf0f640 1 -- start start 2023-10-30T12:49:21.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.006+0000 7f4a9cf0f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a980bfec0 0x7f4a980c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:21.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.006+0000 7f4a9cf0f640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4a9806a720 0x7f4a9806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:21.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.006+0000 7f4a9cf0f640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a98071290 con 0x7f4a98159e60 2023-10-30T12:49:21.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.006+0000 7f4a9cf0f640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a98071570 con 0x7f4a9806a720 2023-10-30T12:49:21.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.006+0000 7f4a9cf0f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a98071850 con 0x7f4a980bfec0 2023-10-30T12:49:21.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.010+0000 7f4a96d9d640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4a9806a720 0x7f4a9806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:21.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.010+0000 7f4a96d9d640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4a9806a720 0x7f4a9806abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:56628/0 (socket says 172.21.15.89:56628) 2023-10-30T12:49:21.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.010+0000 7f4a96d9d640 1 -- 172.21.15.89:0/2857730317 learned_addr learned my addr 172.21.15.89:0/2857730317 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:49:21.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.010+0000 7f4a8dd9b640 1 --2- 172.21.15.89:0/2857730317 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a980bfec0 0x7f4a980c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:21.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.010+0000 7f4a96d9d640 1 -- 172.21.15.89:0/2857730317 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a980bfec0 msgr2=0x7f4a980c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:21.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.010+0000 7f4a96d9d640 1 --2- 172.21.15.89:0/2857730317 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a980bfec0 0x7f4a980c1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:21.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.010+0000 7f4a96d9d640 1 -- 172.21.15.89:0/2857730317 >> v1:172.21.15.192:6789/0 conn(0x7f4a98159e60 legacy=0x7f4a9806f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:21.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.010+0000 7f4a96d9d640 1 -- 172.21.15.89:0/2857730317 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a98071b30 con 0x7f4a9806a720 2023-10-30T12:49:21.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.010+0000 7f4a96d9d640 1 --2- 172.21.15.89:0/2857730317 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a9806a720 0x7f4a9806abb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4a9000af60 tx=0x7f4a90020310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f3d889bc4b21543 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:21.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.014+0000 7f4a95d9b640 1 -- 172.21.15.89:0/2857730317 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9000a2d0 con 0x7f4a9806a720 2023-10-30T12:49:21.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.014+0000 7f4a95d9b640 1 -- 172.21.15.89:0/2857730317 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a900ac050 con 0x7f4a9806a720 2023-10-30T12:49:21.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.014+0000 7f4a95d9b640 1 -- 172.21.15.89:0/2857730317 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a90008720 con 0x7f4a9806a720 2023-10-30T12:49:21.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.014+0000 7f4a9cf0f640 1 -- 172.21.15.89:0/2857730317 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a9806a720 msgr2=0x7f4a9806abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:21.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.014+0000 7f4a9cf0f640 1 --2- 172.21.15.89:0/2857730317 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a9806a720 0x7f4a9806abb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4a9000af60 tx=0x7f4a90020310 comp rx=0 tx=0).stop 2023-10-30T12:49:21.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.014+0000 7f4a9cf0f640 1 -- 172.21.15.89:0/2857730317 shutdown_connections 2023-10-30T12:49:21.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.014+0000 7f4a9cf0f640 1 --2- 172.21.15.89:0/2857730317 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a9806a720 0x7f4a9806abb0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:21.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.014+0000 7f4a9cf0f640 1 --2- 172.21.15.89:0/2857730317 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a980bfec0 0x7f4a980c1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:21.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.014+0000 7f4a9cf0f640 1 -- 172.21.15.89:0/2857730317 >> 172.21.15.89:0/2857730317 conn(0x7f4a980c7910 msgr2=0x7f4a980c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:21.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.014+0000 7f4a9cf0f640 1 -- 172.21.15.89:0/2857730317 shutdown_connections 2023-10-30T12:49:21.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.014+0000 7f4a9cf0f640 1 -- 172.21.15.89:0/2857730317 wait complete. 2023-10-30T12:49:21.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.014+0000 7f4a9cf0f640 1 Processor -- start 2023-10-30T12:49:21.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.014+0000 7f4a9cf0f640 1 -- start start 2023-10-30T12:49:21.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.014+0000 7f4a9cf0f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a9806a720 0x7f4a9814d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:21.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.014+0000 7f4a9cf0f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a980bfec0 0x7f4a98151400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:21.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.014+0000 7f4a9cf0f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a98159e60 0x7f4a98152de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:21.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.014+0000 7f4a9cf0f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a98155ee0 con 0x7f4a9806a720 2023-10-30T12:49:21.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.014+0000 7f4a9cf0f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a981ed320 con 0x7f4a980bfec0 2023-10-30T12:49:21.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.014+0000 7f4a9cf0f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a98156110 con 0x7f4a98159e60 2023-10-30T12:49:21.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.014+0000 7f4a9659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a9806a720 0x7f4a9814d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:21.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.018+0000 7f4a9659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a9806a720 0x7f4a9814d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51562/0 (socket says 172.21.15.89:51562) 2023-10-30T12:49:21.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.018+0000 7f4a9659c640 1 -- 172.21.15.89:0/3412823125 learned_addr learned my addr 172.21.15.89:0/3412823125 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:49:21.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.018+0000 7f4a96d9d640 1 --2- 172.21.15.89:0/3412823125 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a98159e60 0x7f4a98152de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:21.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.018+0000 7f4a8dd9b640 1 --2- 172.21.15.89:0/3412823125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a980bfec0 0x7f4a98151400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:21.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.018+0000 7f4a9659c640 1 -- 172.21.15.89:0/3412823125 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a98159e60 msgr2=0x7f4a98152de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:21.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.018+0000 7f4a9659c640 1 --2- 172.21.15.89:0/3412823125 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a98159e60 0x7f4a98152de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:21.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.018+0000 7f4a9659c640 1 -- 172.21.15.89:0/3412823125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a980bfec0 msgr2=0x7f4a98151400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:21.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.018+0000 7f4a9659c640 1 --2- 172.21.15.89:0/3412823125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a980bfec0 0x7f4a98151400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:21.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.018+0000 7f4a9659c640 1 -- 172.21.15.89:0/3412823125 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a9000ac40 con 0x7f4a9806a720 2023-10-30T12:49:21.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.018+0000 7f4a9659c640 1 --2- 172.21.15.89:0/3412823125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a9806a720 0x7f4a9814d480 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f4a8800c370 tx=0x7f4a8800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:21.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.018+0000 7f4a8f7fe640 1 -- 172.21.15.89:0/3412823125 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a880052d0 con 0x7f4a9806a720 2023-10-30T12:49:21.019 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.018+0000 7f4a8f7fe640 1 -- 172.21.15.89:0/3412823125 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a88005430 con 0x7f4a9806a720 2023-10-30T12:49:21.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.018+0000 7f4a8f7fe640 1 -- 172.21.15.89:0/3412823125 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a88007260 con 0x7f4a9806a720 2023-10-30T12:49:21.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.018+0000 7f4a9cf0f640 1 -- 172.21.15.89:0/3412823125 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a981ed7b0 con 0x7f4a9806a720 2023-10-30T12:49:21.020 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.018+0000 7f4a9cf0f640 1 -- 172.21.15.89:0/3412823125 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a981df7f0 con 0x7f4a9806a720 2023-10-30T12:49:21.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.018+0000 7f4a9cf0f640 1 -- 172.21.15.89:0/3412823125 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a9804e170 con 0x7f4a9806a720 2023-10-30T12:49:21.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.018+0000 7f4a8f7fe640 1 -- 172.21.15.89:0/3412823125 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4a880073c0 con 0x7f4a9806a720 2023-10-30T12:49:21.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.018+0000 7f4a8f7fe640 1 --2- 172.21.15.89:0/3412823125 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4a780664a0 0x7f4a78068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:21.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.018+0000 7f4a8f7fe640 1 -- 172.21.15.89:0/3412823125 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6664+0+0 (secure 0 0 0) 0x7f4a880835b0 con 0x7f4a9806a720 2023-10-30T12:49:21.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.022+0000 7f4a8dd9b640 1 --2- 172.21.15.89:0/3412823125 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4a780664a0 0x7f4a78068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:21.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.022+0000 7f4a8dd9b640 1 --2- 172.21.15.89:0/3412823125 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4a780664a0 0x7f4a78068960 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f4a980c9c50 tx=0x7f4a80008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:21.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.022+0000 7f4a8f7fe640 1 -- 172.21.15.89:0/3412823125 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4a88050680 con 0x7f4a9806a720 2023-10-30T12:49:21.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.202+0000 7f4a9cf0f640 1 -- 172.21.15.89:0/3412823125 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a980c99a0 con 0x7f4a9806a720 2023-10-30T12:49:21.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.202+0000 7f4a8f7fe640 1 -- 172.21.15.89:0/3412823125 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+5746 (secure 0 0 0) 0x7f4a88053ca0 con 0x7f4a9806a720 2023-10-30T12:49:21.201 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:49:21.201 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:49:20.452599+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5678,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5678":{"gid":5678,"name":"f","rank":0,"incarnation":98,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:21.201 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 101 2023-10-30T12:49:21.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.202+0000 7f4a9cf0f640 1 -- 172.21.15.89:0/3412823125 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4a780664a0 msgr2=0x7f4a78068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:21.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.202+0000 7f4a9cf0f640 1 --2- 172.21.15.89:0/3412823125 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4a780664a0 0x7f4a78068960 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f4a980c9c50 tx=0x7f4a80008040 comp rx=0 tx=0).stop 2023-10-30T12:49:21.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.202+0000 7f4a9cf0f640 1 -- 172.21.15.89:0/3412823125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a9806a720 msgr2=0x7f4a9814d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:21.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.202+0000 7f4a9cf0f640 1 --2- 172.21.15.89:0/3412823125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a9806a720 0x7f4a9814d480 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f4a8800c370 tx=0x7f4a8800c830 comp rx=0 tx=0).stop 2023-10-30T12:49:21.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.202+0000 7f4a9cf0f640 1 -- 172.21.15.89:0/3412823125 shutdown_connections 2023-10-30T12:49:21.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.202+0000 7f4a9cf0f640 1 --2- 172.21.15.89:0/3412823125 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4a780664a0 0x7f4a78068960 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:21.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.202+0000 7f4a9cf0f640 1 --2- 172.21.15.89:0/3412823125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a980bfec0 0x7f4a98151400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:21.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.202+0000 7f4a9cf0f640 1 --2- 172.21.15.89:0/3412823125 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a98159e60 0x7f4a98152de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:21.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.202+0000 7f4a9cf0f640 1 --2- 172.21.15.89:0/3412823125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a9806a720 0x7f4a9814d480 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:21.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.202+0000 7f4a9cf0f640 1 -- 172.21.15.89:0/3412823125 >> 172.21.15.89:0/3412823125 conn(0x7f4a980c7910 msgr2=0x7f4a980c8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:21.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.202+0000 7f4a9cf0f640 1 -- 172.21.15.89:0/3412823125 shutdown_connections 2023-10-30T12:49:21.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:21.206+0000 7f4a9cf0f640 1 -- 172.21.15.89:0/3412823125 wait complete. 2023-10-30T12:49:21.221 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:49:23.222 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:23.370 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.370+0000 7fbf05eea640 1 Processor -- start 2023-10-30T12:49:23.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.374+0000 7fbf05eea640 1 -- start start 2023-10-30T12:49:23.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.374+0000 7fbf05eea640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbf00069e90 0x7fbf0015a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:23.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.374+0000 7fbf05eea640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbf0006cb80 0x7fbf0006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:23.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.374+0000 7fbf05eea640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf0006d530 con 0x7fbf0006a270 2023-10-30T12:49:23.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.374+0000 7fbf05eea640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf000c8570 con 0x7fbf0006cb80 2023-10-30T12:49:23.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.374+0000 7fbf05eea640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf000c6c10 con 0x7fbf00069e90 2023-10-30T12:49:23.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.374+0000 7fbefffff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbf0006cb80 0x7fbf0006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:23.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.374+0000 7fbefeffd640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fbf0006a270 0x7fbf000c9cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43530/0 (socket says 172.21.15.89:43530) 2023-10-30T12:49:23.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.374+0000 7fbefffff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fbf0006cb80 0x7fbf0006cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51578/0 (socket says 172.21.15.89:51578) 2023-10-30T12:49:23.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.374+0000 7fbefeffd640 1 -- 172.21.15.89:0/357067310 learned_addr learned my addr 172.21.15.89:0/357067310 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:49:23.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.374+0000 7fbeff7fe640 1 --2- 172.21.15.89:0/357067310 >> v2:172.21.15.192:3300/0 conn(0x7fbf00069e90 0x7fbf0015a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:23.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.374+0000 7fbefffff640 1 -- 172.21.15.89:0/357067310 >> v2:172.21.15.192:3300/0 conn(0x7fbf00069e90 msgr2=0x7fbf0015a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:23.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.374+0000 7fbefffff640 1 --2- 172.21.15.89:0/357067310 >> v2:172.21.15.192:3300/0 conn(0x7fbf00069e90 0x7fbf0015a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:23.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.374+0000 7fbefffff640 1 -- 172.21.15.89:0/357067310 >> v1:172.21.15.89:6789/0 conn(0x7fbf0006a270 legacy=0x7fbf000c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:23.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.374+0000 7fbefffff640 1 -- 172.21.15.89:0/357067310 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf000c6ef0 con 0x7fbf0006cb80 2023-10-30T12:49:23.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.378+0000 7fbefffff640 1 --2- 172.21.15.89:0/357067310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf0006cb80 0x7fbf0006cff0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fbf0006ed20 tx=0x7fbef801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bfce7650183f08e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:23.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.378+0000 7fbefe7fc640 1 -- 172.21.15.89:0/357067310 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef80a58c0 con 0x7fbf0006cb80 2023-10-30T12:49:23.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.378+0000 7fbefe7fc640 1 -- 172.21.15.89:0/357067310 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef80a5a20 con 0x7fbf0006cb80 2023-10-30T12:49:23.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.378+0000 7fbefe7fc640 1 -- 172.21.15.89:0/357067310 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef80a5db0 con 0x7fbf0006cb80 2023-10-30T12:49:23.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.378+0000 7fbf05eea640 1 -- 172.21.15.89:0/357067310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf0006cb80 msgr2=0x7fbf0006cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:23.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.378+0000 7fbf05eea640 1 --2- 172.21.15.89:0/357067310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf0006cb80 0x7fbf0006cff0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fbf0006ed20 tx=0x7fbef801c910 comp rx=0 tx=0).stop 2023-10-30T12:49:23.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.378+0000 7fbf05eea640 1 -- 172.21.15.89:0/357067310 shutdown_connections 2023-10-30T12:49:23.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.378+0000 7fbf05eea640 1 --2- 172.21.15.89:0/357067310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf0006cb80 0x7fbf0006cff0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:23.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.378+0000 7fbf05eea640 1 --2- 172.21.15.89:0/357067310 >> v2:172.21.15.192:3300/0 conn(0x7fbf00069e90 0x7fbf0015a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:23.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.378+0000 7fbf05eea640 1 -- 172.21.15.89:0/357067310 >> 172.21.15.89:0/357067310 conn(0x7fbf0006dfd0 msgr2=0x7fbf0006e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:23.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.378+0000 7fbf05eea640 1 -- 172.21.15.89:0/357067310 shutdown_connections 2023-10-30T12:49:23.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.378+0000 7fbf05eea640 1 -- 172.21.15.89:0/357067310 wait complete. 2023-10-30T12:49:23.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.378+0000 7fbf05eea640 1 Processor -- start 2023-10-30T12:49:23.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.378+0000 7fbf05eea640 1 -- start start 2023-10-30T12:49:23.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.382+0000 7fbf05eea640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf00069e90 0x7fbf00152560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:23.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.382+0000 7fbf05eea640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf0006a270 0x7fbf00151420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:23.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.382+0000 7fbf05eea640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf0006cb80 0x7fbf001f4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:23.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.382+0000 7fbf05eea640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf00150f90 con 0x7fbf0006a270 2023-10-30T12:49:23.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.382+0000 7fbf05eea640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf00152aa0 con 0x7fbf00069e90 2023-10-30T12:49:23.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.382+0000 7fbf05eea640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0014db70 con 0x7fbf0006cb80 2023-10-30T12:49:23.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.382+0000 7fbefeffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf0006a270 0x7fbf00151420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:23.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.382+0000 7fbefeffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf0006a270 0x7fbf00151420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51594/0 (socket says 172.21.15.89:51594) 2023-10-30T12:49:23.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.382+0000 7fbefeffd640 1 -- 172.21.15.89:0/3272742518 learned_addr learned my addr 172.21.15.89:0/3272742518 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:49:23.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.382+0000 7fbefffff640 1 --2- 172.21.15.89:0/3272742518 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf0006cb80 0x7fbf001f4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:23.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.382+0000 7fbeff7fe640 1 --2- 172.21.15.89:0/3272742518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf00069e90 0x7fbf00152560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:23.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.382+0000 7fbefeffd640 1 -- 172.21.15.89:0/3272742518 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf0006cb80 msgr2=0x7fbf001f4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:23.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.382+0000 7fbefeffd640 1 --2- 172.21.15.89:0/3272742518 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf0006cb80 0x7fbf001f4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:23.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.382+0000 7fbefeffd640 1 -- 172.21.15.89:0/3272742518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf00069e90 msgr2=0x7fbf00152560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:23.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.382+0000 7fbefeffd640 1 --2- 172.21.15.89:0/3272742518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf00069e90 0x7fbf00152560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:23.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.382+0000 7fbefeffd640 1 -- 172.21.15.89:0/3272742518 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef8009a70 con 0x7fbf0006a270 2023-10-30T12:49:23.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.382+0000 7fbeff7fe640 1 --2- 172.21.15.89:0/3272742518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf00069e90 0x7fbf00152560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:49:23.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.382+0000 7fbefeffd640 1 --2- 172.21.15.89:0/3272742518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf0006a270 0x7fbf00151420 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fbef400c530 tx=0x7fbef400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:23.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.382+0000 7fbee7fff640 1 -- 172.21.15.89:0/3272742518 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef4005cf0 con 0x7fbf0006a270 2023-10-30T12:49:23.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.382+0000 7fbee7fff640 1 -- 172.21.15.89:0/3272742518 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef4005e50 con 0x7fbf0006a270 2023-10-30T12:49:23.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.382+0000 7fbf05eea640 1 -- 172.21.15.89:0/3272742518 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf000cb820 con 0x7fbf0006a270 2023-10-30T12:49:23.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.382+0000 7fbee7fff640 1 -- 172.21.15.89:0/3272742518 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef4007270 con 0x7fbf0006a270 2023-10-30T12:49:23.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.382+0000 7fbf05eea640 1 -- 172.21.15.89:0/3272742518 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf0014d5b0 con 0x7fbf0006a270 2023-10-30T12:49:23.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.382+0000 7fbf05eea640 1 -- 172.21.15.89:0/3272742518 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf00151140 con 0x7fbf0006a270 2023-10-30T12:49:23.387 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.382+0000 7fbee7fff640 1 -- 172.21.15.89:0/3272742518 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbef40073d0 con 0x7fbf0006a270 2023-10-30T12:49:23.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.386+0000 7fbee7fff640 1 --2- 172.21.15.89:0/3272742518 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbeec0664a0 0x7fbeec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:23.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.386+0000 7fbee7fff640 1 -- 172.21.15.89:0/3272742518 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6664+0+0 (secure 0 0 0) 0x7fbef4083710 con 0x7fbf0006a270 2023-10-30T12:49:23.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.386+0000 7fbeff7fe640 1 --2- 172.21.15.89:0/3272742518 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbeec0664a0 0x7fbeec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:23.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.386+0000 7fbeff7fe640 1 --2- 172.21.15.89:0/3272742518 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbeec0664a0 0x7fbeec068960 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fbef0007a70 tx=0x7fbef0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:23.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.390+0000 7fbee7fff640 1 -- 172.21.15.89:0/3272742518 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbef40507f0 con 0x7fbf0006a270 2023-10-30T12:49:23.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.570+0000 7fbf05eea640 1 -- 172.21.15.89:0/3272742518 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf00153d60 con 0x7fbf0006a270 2023-10-30T12:49:23.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.570+0000 7fbee7fff640 1 -- 172.21.15.89:0/3272742518 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+5746 (secure 0 0 0) 0x7fbef4017090 con 0x7fbf0006a270 2023-10-30T12:49:23.571 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:49:23.571 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:49:20.452599+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5678,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5678":{"gid":5678,"name":"f","rank":0,"incarnation":98,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:23.572 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 101 2023-10-30T12:49:23.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.574+0000 7fbf05eea640 1 -- 172.21.15.89:0/3272742518 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbeec0664a0 msgr2=0x7fbeec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:23.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.574+0000 7fbf05eea640 1 --2- 172.21.15.89:0/3272742518 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbeec0664a0 0x7fbeec068960 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fbef0007a70 tx=0x7fbef0008040 comp rx=0 tx=0).stop 2023-10-30T12:49:23.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.574+0000 7fbf05eea640 1 -- 172.21.15.89:0/3272742518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf0006a270 msgr2=0x7fbf00151420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:23.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.574+0000 7fbf05eea640 1 --2- 172.21.15.89:0/3272742518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf0006a270 0x7fbf00151420 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fbef400c530 tx=0x7fbef400c9f0 comp rx=0 tx=0).stop 2023-10-30T12:49:23.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.574+0000 7fbf05eea640 1 -- 172.21.15.89:0/3272742518 shutdown_connections 2023-10-30T12:49:23.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.574+0000 7fbf05eea640 1 --2- 172.21.15.89:0/3272742518 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fbeec0664a0 0x7fbeec068960 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:23.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.574+0000 7fbf05eea640 1 --2- 172.21.15.89:0/3272742518 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf0006cb80 0x7fbf001f4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:23.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.574+0000 7fbf05eea640 1 --2- 172.21.15.89:0/3272742518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf0006a270 0x7fbf00151420 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:23.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.574+0000 7fbf05eea640 1 --2- 172.21.15.89:0/3272742518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf00069e90 0x7fbf00152560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:23.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.574+0000 7fbf05eea640 1 -- 172.21.15.89:0/3272742518 >> 172.21.15.89:0/3272742518 conn(0x7fbf0006dfd0 msgr2=0x7fbf00150060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:23.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.574+0000 7fbf05eea640 1 -- 172.21.15.89:0/3272742518 shutdown_connections 2023-10-30T12:49:23.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:23.574+0000 7fbf05eea640 1 -- 172.21.15.89:0/3272742518 wait complete. 2023-10-30T12:49:23.593 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:49:25.594 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:25.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.741+0000 7fb56d046640 1 Processor -- start 2023-10-30T12:49:25.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.741+0000 7fb56d046640 1 -- start start 2023-10-30T12:49:25.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.741+0000 7fb56d046640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb56815a410 0x7fb56806a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:25.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.741+0000 7fb56d046640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5680c6c10 con 0x7fb568069e90 2023-10-30T12:49:25.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.741+0000 7fb56d046640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5680c6ef0 con 0x7fb56806cb80 2023-10-30T12:49:25.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.741+0000 7fb56d046640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5680c8570 con 0x7fb56815a410 2023-10-30T12:49:25.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.741+0000 7fb566d9d640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb568069e90 0x7fb5680c9cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:43538/0 (socket says 172.21.15.89:43538) 2023-10-30T12:49:25.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.741+0000 7fb566d9d640 1 -- 172.21.15.89:0/1271132485 learned_addr learned my addr 172.21.15.89:0/1271132485 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:49:25.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.741+0000 7fb55ffff640 1 --2- 172.21.15.89:0/1271132485 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb56815a410 0x7fb56806a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:25.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.741+0000 7fb56659c640 1 -- 172.21.15.89:0/1271132485 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683844586 0 0) 0x7fb5680c6c10 con 0x7fb568069e90 2023-10-30T12:49:25.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.741+0000 7fb56659c640 1 -- 172.21.15.89:0/1271132485 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb554003530 con 0x7fb568069e90 2023-10-30T12:49:25.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.741+0000 7fb55ffff640 1 -- 172.21.15.89:0/1271132485 >> v1:172.21.15.192:6789/0 conn(0x7fb56806cb80 legacy=0x7fb56806d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:25.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.741+0000 7fb55ffff640 1 -- 172.21.15.89:0/1271132485 >> v1:172.21.15.89:6789/0 conn(0x7fb568069e90 legacy=0x7fb5680c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:25.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.741+0000 7fb56659c640 1 -- 172.21.15.89:0/1271132485 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 494838118 0 0) 0x7fb554003530 con 0x7fb568069e90 2023-10-30T12:49:25.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.741+0000 7fb55ffff640 1 -- 172.21.15.89:0/1271132485 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5680c71d0 con 0x7fb56815a410 2023-10-30T12:49:25.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.745+0000 7fb55ffff640 1 --2- 172.21.15.89:0/1271132485 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb56815a410 0x7fb56806a270 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb5500097f0 tx=0x7fb55001c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e44bde76d0cbc1e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:25.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.745+0000 7fb56659c640 1 -- 172.21.15.89:0/1271132485 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb558001680 con 0x7fb56815a410 2023-10-30T12:49:25.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.745+0000 7fb56659c640 1 -- 172.21.15.89:0/1271132485 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5500a7050 con 0x7fb56815a410 2023-10-30T12:49:25.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.745+0000 7fb56659c640 1 -- 172.21.15.89:0/1271132485 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5500ab620 con 0x7fb56815a410 2023-10-30T12:49:25.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.745+0000 7fb56d046640 1 -- 172.21.15.89:0/1271132485 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb56815a410 msgr2=0x7fb56806a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:25.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.745+0000 7fb56d046640 1 --2- 172.21.15.89:0/1271132485 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb56815a410 0x7fb56806a270 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb5500097f0 tx=0x7fb55001c3e0 comp rx=0 tx=0).stop 2023-10-30T12:49:25.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.745+0000 7fb56d046640 1 -- 172.21.15.89:0/1271132485 shutdown_connections 2023-10-30T12:49:25.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.745+0000 7fb56d046640 1 --2- 172.21.15.89:0/1271132485 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb56815a410 0x7fb56806a270 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:25.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.745+0000 7fb56d046640 1 -- 172.21.15.89:0/1271132485 >> 172.21.15.89:0/1271132485 conn(0x7fb56806dfd0 msgr2=0x7fb56806e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:25.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.745+0000 7fb56d046640 1 -- 172.21.15.89:0/1271132485 shutdown_connections 2023-10-30T12:49:25.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.745+0000 7fb56d046640 1 -- 172.21.15.89:0/1271132485 wait complete. 2023-10-30T12:49:25.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.749+0000 7fb56d046640 1 Processor -- start 2023-10-30T12:49:25.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.749+0000 7fb56d046640 1 -- start start 2023-10-30T12:49:25.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.749+0000 7fb56d046640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb568069e90 0x7fb5681418b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:25.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.749+0000 7fb56d046640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb56806cb80 0x7fb56814c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:25.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.749+0000 7fb56d046640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb56815a410 0x7fb5681f5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:25.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.749+0000 7fb56d046640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5681597d0 con 0x7fb568069e90 2023-10-30T12:49:25.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.749+0000 7fb56d046640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb568145230 con 0x7fb56815a410 2023-10-30T12:49:25.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.749+0000 7fb56d046640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb568159210 con 0x7fb56806cb80 2023-10-30T12:49:25.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.749+0000 7fb566d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb568069e90 0x7fb5681418b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:25.751 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.749+0000 7fb566d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb568069e90 0x7fb5681418b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51608/0 (socket says 172.21.15.89:51608) 2023-10-30T12:49:25.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.749+0000 7fb566d9d640 1 -- 172.21.15.89:0/1270797041 learned_addr learned my addr 172.21.15.89:0/1270797041 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:49:25.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.749+0000 7fb55ffff640 1 --2- 172.21.15.89:0/1270797041 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb56806cb80 0x7fb56814c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:25.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.749+0000 7fb56759e640 1 --2- 172.21.15.89:0/1270797041 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb56815a410 0x7fb5681f5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:25.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.749+0000 7fb566d9d640 1 -- 172.21.15.89:0/1270797041 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb56806cb80 msgr2=0x7fb56814c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:25.752 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.749+0000 7fb566d9d640 1 --2- 172.21.15.89:0/1270797041 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb56806cb80 0x7fb56814c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:25.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.749+0000 7fb566d9d640 1 -- 172.21.15.89:0/1270797041 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb56815a410 msgr2=0x7fb5681f5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:25.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.749+0000 7fb566d9d640 1 --2- 172.21.15.89:0/1270797041 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb56815a410 0x7fb5681f5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:25.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.749+0000 7fb566d9d640 1 -- 172.21.15.89:0/1270797041 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb550009420 con 0x7fb568069e90 2023-10-30T12:49:25.753 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.749+0000 7fb566d9d640 1 --2- 172.21.15.89:0/1270797041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb568069e90 0x7fb5681418b0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fb5580011f0 tx=0x7fb55800cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:25.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.749+0000 7fb55f7fe640 1 -- 172.21.15.89:0/1270797041 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb558005ae0 con 0x7fb568069e90 2023-10-30T12:49:25.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.749+0000 7fb55f7fe640 1 -- 172.21.15.89:0/1270797041 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb558005c60 con 0x7fb568069e90 2023-10-30T12:49:25.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.749+0000 7fb55f7fe640 1 -- 172.21.15.89:0/1270797041 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5580072b0 con 0x7fb568069e90 2023-10-30T12:49:25.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.749+0000 7fb56d046640 1 -- 172.21.15.89:0/1270797041 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5681594f0 con 0x7fb568069e90 2023-10-30T12:49:25.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.749+0000 7fb56d046640 1 -- 172.21.15.89:0/1270797041 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb568140730 con 0x7fb568069e90 2023-10-30T12:49:25.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.753+0000 7fb56d046640 1 -- 172.21.15.89:0/1270797041 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb568144d20 con 0x7fb568069e90 2023-10-30T12:49:25.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.753+0000 7fb55f7fe640 1 -- 172.21.15.89:0/1270797041 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb558007410 con 0x7fb568069e90 2023-10-30T12:49:25.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.753+0000 7fb55f7fe640 1 --2- 172.21.15.89:0/1270797041 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb5480664a0 0x7fb548068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:25.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.753+0000 7fb55f7fe640 1 -- 172.21.15.89:0/1270797041 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6664+0+0 (secure 0 0 0) 0x7fb558083a70 con 0x7fb568069e90 2023-10-30T12:49:25.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.753+0000 7fb55ffff640 1 --2- 172.21.15.89:0/1270797041 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb5480664a0 0x7fb548068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:25.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.757+0000 7fb55ffff640 1 --2- 172.21.15.89:0/1270797041 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb5480664a0 0x7fb548068960 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb5500ad000 tx=0x7fb550008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:25.757 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.757+0000 7fb55f7fe640 1 -- 172.21.15.89:0/1270797041 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb568144d20 con 0x7fb568069e90 2023-10-30T12:49:25.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.941+0000 7fb56d046640 1 -- 172.21.15.89:0/1270797041 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb568144d20 con 0x7fb568069e90 2023-10-30T12:49:25.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.941+0000 7fb55f7fe640 1 -- 172.21.15.89:0/1270797041 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+5746 (secure 0 0 0) 0x7fb568144d20 con 0x7fb568069e90 2023-10-30T12:49:25.940 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:49:25.940 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:49:20.452599+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5678,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5678":{"gid":5678,"name":"f","rank":0,"incarnation":98,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:25.940 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 101 2023-10-30T12:49:25.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.941+0000 7fb56d046640 1 -- 172.21.15.89:0/1270797041 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb5480664a0 msgr2=0x7fb548068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:25.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.941+0000 7fb56d046640 1 --2- 172.21.15.89:0/1270797041 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb5480664a0 0x7fb548068960 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb5500ad000 tx=0x7fb550008040 comp rx=0 tx=0).stop 2023-10-30T12:49:25.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.941+0000 7fb56d046640 1 -- 172.21.15.89:0/1270797041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb568069e90 msgr2=0x7fb5681418b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:25.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.941+0000 7fb56d046640 1 --2- 172.21.15.89:0/1270797041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb568069e90 0x7fb5681418b0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fb5580011f0 tx=0x7fb55800cbe0 comp rx=0 tx=0).stop 2023-10-30T12:49:25.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.941+0000 7fb56d046640 1 -- 172.21.15.89:0/1270797041 shutdown_connections 2023-10-30T12:49:25.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.941+0000 7fb56d046640 1 --2- 172.21.15.89:0/1270797041 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb5480664a0 0x7fb548068960 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:25.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.941+0000 7fb56d046640 1 --2- 172.21.15.89:0/1270797041 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb56815a410 0x7fb5681f5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:25.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.941+0000 7fb56d046640 1 --2- 172.21.15.89:0/1270797041 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb56806cb80 0x7fb56814c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:25.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.941+0000 7fb56d046640 1 --2- 172.21.15.89:0/1270797041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb568069e90 0x7fb5681418b0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:25.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.941+0000 7fb56d046640 1 -- 172.21.15.89:0/1270797041 >> 172.21.15.89:0/1270797041 conn(0x7fb56806dfd0 msgr2=0x7fb56814bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:25.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.941+0000 7fb56d046640 1 -- 172.21.15.89:0/1270797041 shutdown_connections 2023-10-30T12:49:25.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:25.941+0000 7fb56d046640 1 -- 172.21.15.89:0/1270797041 wait complete. 2023-10-30T12:49:25.961 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 101, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:50.729245+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 101, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5678": { "addr": "172.21.15.192:6837/2130807115", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6836", "nonce": 2130807115, "type": "v2" }, { "addr": "172.21.15.192:6837", "nonce": 2130807115, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5678, "incarnation": 98, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:rejoin", "state_seq": 12 }, "gid_5728": { "addr": "172.21.15.192:6839/3991891597", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6838", "nonce": 3991891597, "type": "v2" }, { "addr": "172.21.15.192:6839", "nonce": 3991891597, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 5728, "incarnation": 73, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 19 }, "gid_6075": { "addr": "172.21.15.89:6837/2412618858", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6836", "nonce": 2412618858, "type": "v2" }, { "addr": "172.21.15.89:6837", "nonce": 2412618858, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6075, "incarnation": 92, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 38 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:49:20.452599+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5678, "mds_1": 6075, "mds_2": 5728 } } } ], "standbys": [ { "addr": "172.21.15.89:6834/2909790173", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6832", "nonce": 2909790173, "type": "v2" }, { "addr": "172.21.15.89:6834", "nonce": 2909790173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.89:6835/2387357615", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6833", "nonce": 2387357615, "type": "v2" }, { "addr": "172.21.15.89:6835", "nonce": 2387357615, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4239, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:49:27.962 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:28.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.113+0000 7f5f9a7aa640 1 Processor -- start 2023-10-30T12:49:28.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.113+0000 7f5f9a7aa640 1 -- start start 2023-10-30T12:49:28.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.113+0000 7f5f9a7aa640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5f941538a0 0x7f5f9415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:28.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.113+0000 7f5f9a7aa640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f940cf160 0x7f5f940cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:28.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.113+0000 7f5f9a7aa640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f940cb800 con 0x7f5f940ca0e0 2023-10-30T12:49:28.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.113+0000 7f5f9a7aa640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f9415dc30 con 0x7f5f941538a0 2023-10-30T12:49:28.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.113+0000 7f5f9a7aa640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f940cb330 con 0x7f5f940cf160 2023-10-30T12:49:28.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.113+0000 7f5f93fff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5f941538a0 0x7f5f9415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:28.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.113+0000 7f5f93fff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f5f941538a0 0x7f5f9415d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51610/0 (socket says 172.21.15.89:51610) 2023-10-30T12:49:28.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.113+0000 7f5f93fff640 1 -- 172.21.15.89:0/2482775727 learned_addr learned my addr 172.21.15.89:0/2482775727 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:49:28.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.113+0000 7f5f98d47640 1 --2- 172.21.15.89:0/2482775727 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f940cf160 0x7f5f940cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:28.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.113+0000 7f5f93fff640 1 -- 172.21.15.89:0/2482775727 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f940cf160 msgr2=0x7f5f940cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:28.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.113+0000 7f5f93fff640 1 --2- 172.21.15.89:0/2482775727 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f940cf160 0x7f5f940cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:28.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.113+0000 7f5f93fff640 1 -- 172.21.15.89:0/2482775727 >> v1:172.21.15.192:6789/0 conn(0x7f5f940ca0e0 legacy=0x7f5f9415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:28.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.113+0000 7f5f93fff640 1 -- 172.21.15.89:0/2482775727 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f940cad70 con 0x7f5f941538a0 2023-10-30T12:49:28.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.117+0000 7f5f93fff640 1 --2- 172.21.15.89:0/2482775727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f941538a0 0x7f5f9415d6f0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f5f84009870 tx=0x7f5f8401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7f7b484cefcb548 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:28.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.117+0000 7f5f92ffd640 1 -- 172.21.15.89:0/2482775727 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f840a8a30 con 0x7f5f941538a0 2023-10-30T12:49:28.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.117+0000 7f5f92ffd640 1 -- 172.21.15.89:0/2482775727 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f840a8b90 con 0x7f5f941538a0 2023-10-30T12:49:28.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.117+0000 7f5f92ffd640 1 -- 172.21.15.89:0/2482775727 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f840b0440 con 0x7f5f941538a0 2023-10-30T12:49:28.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.117+0000 7f5f9a7aa640 1 -- 172.21.15.89:0/2482775727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f941538a0 msgr2=0x7f5f9415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:28.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.117+0000 7f5f9a7aa640 1 --2- 172.21.15.89:0/2482775727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f941538a0 0x7f5f9415d6f0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f5f84009870 tx=0x7f5f8401ed30 comp rx=0 tx=0).stop 2023-10-30T12:49:28.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.117+0000 7f5f9a7aa640 1 -- 172.21.15.89:0/2482775727 shutdown_connections 2023-10-30T12:49:28.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.117+0000 7f5f9a7aa640 1 --2- 172.21.15.89:0/2482775727 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f940cf160 0x7f5f940cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:28.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.117+0000 7f5f9a7aa640 1 --2- 172.21.15.89:0/2482775727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f941538a0 0x7f5f9415d6f0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:28.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.117+0000 7f5f9a7aa640 1 -- 172.21.15.89:0/2482775727 >> 172.21.15.89:0/2482775727 conn(0x7f5f9406bfe0 msgr2=0x7f5f9406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:28.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.117+0000 7f5f9a7aa640 1 -- 172.21.15.89:0/2482775727 shutdown_connections 2023-10-30T12:49:28.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.117+0000 7f5f9a7aa640 1 -- 172.21.15.89:0/2482775727 wait complete. 2023-10-30T12:49:28.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.121+0000 7f5f9a7aa640 1 Processor -- start 2023-10-30T12:49:28.121 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.121+0000 7f5f9a7aa640 1 -- start start 2023-10-30T12:49:28.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.121+0000 7f5f9a7aa640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f940ca0e0 0x7f5f941e4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:28.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.121+0000 7f5f9a7aa640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f940cf160 0x7f5f9415b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:28.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.121+0000 7f5f9a7aa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f941538a0 0x7f5f9415bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:28.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.121+0000 7f5f9a7aa640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9414d4f0 con 0x7f5f941538a0 2023-10-30T12:49:28.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.121+0000 7f5f9a7aa640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f941e4d40 con 0x7f5f940cf160 2023-10-30T12:49:28.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.121+0000 7f5f9a7aa640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9415b2a0 con 0x7f5f940ca0e0 2023-10-30T12:49:28.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.121+0000 7f5f98d47640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f941538a0 0x7f5f9415bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:28.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.121+0000 7f5f98d47640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f941538a0 0x7f5f9415bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51622/0 (socket says 172.21.15.89:51622) 2023-10-30T12:49:28.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.121+0000 7f5f98d47640 1 -- 172.21.15.89:0/3994550795 learned_addr learned my addr 172.21.15.89:0/3994550795 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:49:28.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.121+0000 7f5f93fff640 1 --2- 172.21.15.89:0/3994550795 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f940ca0e0 0x7f5f941e4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:28.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.121+0000 7f5f937fe640 1 --2- 172.21.15.89:0/3994550795 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f940cf160 0x7f5f9415b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:28.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.121+0000 7f5f98d47640 1 -- 172.21.15.89:0/3994550795 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f940ca0e0 msgr2=0x7f5f941e4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:28.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.121+0000 7f5f98d47640 1 --2- 172.21.15.89:0/3994550795 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f940ca0e0 0x7f5f941e4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:28.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.121+0000 7f5f98d47640 1 -- 172.21.15.89:0/3994550795 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f940cf160 msgr2=0x7f5f9415b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:28.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.121+0000 7f5f98d47640 1 --2- 172.21.15.89:0/3994550795 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f940cf160 0x7f5f9415b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:28.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.121+0000 7f5f98d47640 1 -- 172.21.15.89:0/3994550795 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f840094d0 con 0x7f5f941538a0 2023-10-30T12:49:28.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.121+0000 7f5f98d47640 1 --2- 172.21.15.89:0/3994550795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f941538a0 0x7f5f9415bb70 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c007d90 tx=0x7f5f8c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:28.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.121+0000 7f5f90ff9640 1 -- 172.21.15.89:0/3994550795 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f8c014420 con 0x7f5f941538a0 2023-10-30T12:49:28.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.121+0000 7f5f90ff9640 1 -- 172.21.15.89:0/3994550795 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f8c0145a0 con 0x7f5f941538a0 2023-10-30T12:49:28.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.121+0000 7f5f9a7aa640 1 -- 172.21.15.89:0/3994550795 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f940cef10 con 0x7f5f941538a0 2023-10-30T12:49:28.126 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.121+0000 7f5f90ff9640 1 -- 172.21.15.89:0/3994550795 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f8c014950 con 0x7f5f941538a0 2023-10-30T12:49:28.126 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.121+0000 7f5f9a7aa640 1 -- 172.21.15.89:0/3994550795 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f940c50b0 con 0x7f5f941538a0 2023-10-30T12:49:28.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.125+0000 7f5f9a7aa640 1 -- 172.21.15.89:0/3994550795 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f940b2a50 con 0x7f5f941538a0 2023-10-30T12:49:28.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.125+0000 7f5f90ff9640 1 -- 172.21.15.89:0/3994550795 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5f8c014ab0 con 0x7f5f941538a0 2023-10-30T12:49:28.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.125+0000 7f5f90ff9640 1 --2- 172.21.15.89:0/3994550795 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5f800664a0 0x7f5f80068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:28.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.125+0000 7f5f90ff9640 1 -- 172.21.15.89:0/3994550795 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6664+0+0 (secure 0 0 0) 0x7f5f8c084b80 con 0x7f5f941538a0 2023-10-30T12:49:28.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.125+0000 7f5f93fff640 1 --2- 172.21.15.89:0/3994550795 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5f800664a0 0x7f5f80068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:28.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.125+0000 7f5f93fff640 1 --2- 172.21.15.89:0/3994550795 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5f800664a0 0x7f5f80068960 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f5f840b2000 tx=0x7f5f84009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:28.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.129+0000 7f5f90ff9640 1 -- 172.21.15.89:0/3994550795 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5f8c051ba0 con 0x7f5f941538a0 2023-10-30T12:49:28.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.309+0000 7f5f9a7aa640 1 -- 172.21.15.89:0/3994550795 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f941e4eb0 con 0x7f5f941538a0 2023-10-30T12:49:28.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.309+0000 7f5f90ff9640 1 -- 172.21.15.89:0/3994550795 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+5746 (secure 0 0 0) 0x7f5f8c084e60 con 0x7f5f941538a0 2023-10-30T12:49:28.308 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:49:28.309 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:49:25.550669+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5678,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5678":{"gid":5678,"name":"f","rank":0,"incarnation":98,"state":"up:active","state_seq":14,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:28.309 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 102 2023-10-30T12:49:28.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.309+0000 7f5f9a7aa640 1 -- 172.21.15.89:0/3994550795 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5f800664a0 msgr2=0x7f5f80068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:28.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.309+0000 7f5f9a7aa640 1 --2- 172.21.15.89:0/3994550795 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5f800664a0 0x7f5f80068960 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f5f840b2000 tx=0x7f5f84009040 comp rx=0 tx=0).stop 2023-10-30T12:49:28.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.309+0000 7f5f9a7aa640 1 -- 172.21.15.89:0/3994550795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f941538a0 msgr2=0x7f5f9415bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:28.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.309+0000 7f5f9a7aa640 1 --2- 172.21.15.89:0/3994550795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f941538a0 0x7f5f9415bb70 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c007d90 tx=0x7f5f8c00e4b0 comp rx=0 tx=0).stop 2023-10-30T12:49:28.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.309+0000 7f5f9a7aa640 1 -- 172.21.15.89:0/3994550795 shutdown_connections 2023-10-30T12:49:28.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.309+0000 7f5f9a7aa640 1 --2- 172.21.15.89:0/3994550795 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5f800664a0 0x7f5f80068960 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:28.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.309+0000 7f5f9a7aa640 1 --2- 172.21.15.89:0/3994550795 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f940cf160 0x7f5f9415b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:28.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.309+0000 7f5f9a7aa640 1 --2- 172.21.15.89:0/3994550795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5f941538a0 0x7f5f9415bb70 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:28.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.309+0000 7f5f9a7aa640 1 --2- 172.21.15.89:0/3994550795 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5f940ca0e0 0x7f5f941e4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:28.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.309+0000 7f5f9a7aa640 1 -- 172.21.15.89:0/3994550795 >> 172.21.15.89:0/3994550795 conn(0x7f5f9406bfe0 msgr2=0x7f5f94065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:28.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.309+0000 7f5f9a7aa640 1 -- 172.21.15.89:0/3994550795 shutdown_connections 2023-10-30T12:49:28.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:28.309+0000 7f5f9a7aa640 1 -- 172.21.15.89:0/3994550795 wait complete. 2023-10-30T12:49:28.329 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=0, replacing gid=5575 2023-10-30T12:49:28.330 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.b 2023-10-30T12:49:32.331 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2023-10-30T12:49:32.331 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-30T12:49:32.332 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-30T12:49:32.335 INFO:tasks.ceph.mds.b:Started 2023-10-30T12:49:32.335 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:32.412 INFO:tasks.ceph.mds.b.smithi192.stdout:starting mds.b at 2023-10-30T12:49:32.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.489+0000 7f5c5cc09640 1 Processor -- start 2023-10-30T12:49:32.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.489+0000 7f5c5cc09640 1 -- start start 2023-10-30T12:49:32.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c5cc09640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c580c05d0 0x7f5c5813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:32.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c5cc09640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c580c0a50 con 0x7f5c580c6f60 2023-10-30T12:49:32.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c5cc09640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c5806f6e0 con 0x7f5c580c6b80 2023-10-30T12:49:32.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c5cc09640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5806f9c0 con 0x7f5c580c05d0 2023-10-30T12:49:32.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c56d9d640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f5c580c6f60 0x7f5c581566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:45942/0 (socket says 172.21.15.89:45942) 2023-10-30T12:49:32.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c56d9d640 1 -- 172.21.15.89:0/171758316 learned_addr learned my addr 172.21.15.89:0/171758316 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:49:32.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c4ffff640 1 --2- 172.21.15.89:0/171758316 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c580c05d0 0x7f5c5813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:32.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c55d9b640 1 -- 172.21.15.89:0/171758316 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 367007550 0 0) 0x7f5c580c0a50 con 0x7f5c580c6f60 2023-10-30T12:49:32.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c55d9b640 1 -- 172.21.15.89:0/171758316 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c44003530 con 0x7f5c580c6f60 2023-10-30T12:49:32.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c55d9b640 1 -- 172.21.15.89:0/171758316 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1990582346 0 0) 0x7f5c5806f6e0 con 0x7f5c580c6b80 2023-10-30T12:49:32.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c55d9b640 1 -- 172.21.15.89:0/171758316 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c580c0a50 con 0x7f5c580c6b80 2023-10-30T12:49:32.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c55d9b640 1 -- 172.21.15.89:0/171758316 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 43941963 0 0) 0x7f5c44003530 con 0x7f5c580c6f60 2023-10-30T12:49:32.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c55d9b640 1 -- 172.21.15.89:0/171758316 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c580c05d0 msgr2=0x7f5c5813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:32.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c55d9b640 1 --2- 172.21.15.89:0/171758316 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c580c05d0 0x7f5c5813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:32.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c55d9b640 1 -- 172.21.15.89:0/171758316 >> v1:172.21.15.192:6789/0 conn(0x7f5c580c6b80 legacy=0x7f5c580bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:32.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c55d9b640 1 -- 172.21.15.89:0/171758316 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5806fca0 con 0x7f5c580c6f60 2023-10-30T12:49:32.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c55d9b640 1 -- 172.21.15.89:0/171758316 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f5c50001a00 con 0x7f5c580c6f60 2023-10-30T12:49:32.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c55d9b640 1 -- 172.21.15.89:0/171758316 >> v1:172.21.15.89:6789/0 conn(0x7f5c580c6f60 legacy=0x7f5c581566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:32.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c55d9b640 1 --2- 172.21.15.89:0/171758316 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c44003b40 0x7f5c44003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:32.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c55d9b640 1 --2- 172.21.15.89:0/171758316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c440047d0 0x7f5c44004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:32.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c55d9b640 1 --2- 172.21.15.89:0/171758316 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c580c05d0 0x7f5c44005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:32.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c55d9b640 1 -- 172.21.15.89:0/171758316 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5806f9c0 con 0x7f5c440047d0 2023-10-30T12:49:32.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c55d9b640 1 -- 172.21.15.89:0/171758316 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4400dc50 con 0x7f5c44003b40 2023-10-30T12:49:32.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c55d9b640 1 -- 172.21.15.89:0/171758316 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4400de10 con 0x7f5c580c05d0 2023-10-30T12:49:32.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c4ffff640 1 --2- 172.21.15.89:0/171758316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c440047d0 0x7f5c44004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:32.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c5659c640 1 --2- 172.21.15.89:0/171758316 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c44003b40 0x7f5c44003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:32.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c56d9d640 1 --2- 172.21.15.89:0/171758316 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c580c05d0 0x7f5c44005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:32.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c4ffff640 1 -- 172.21.15.89:0/171758316 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c580c05d0 msgr2=0x7f5c44005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:32.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c4ffff640 1 --2- 172.21.15.89:0/171758316 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c580c05d0 0x7f5c44005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:32.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c4ffff640 1 -- 172.21.15.89:0/171758316 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c44003b40 msgr2=0x7f5c44003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:32.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c4ffff640 1 --2- 172.21.15.89:0/171758316 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c44003b40 0x7f5c44003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:32.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c4ffff640 1 -- 172.21.15.89:0/171758316 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c44003530 con 0x7f5c440047d0 2023-10-30T12:49:32.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.493+0000 7f5c5659c640 1 --2- 172.21.15.89:0/171758316 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c44003b40 0x7f5c44003f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:49:32.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.497+0000 7f5c4ffff640 1 --2- 172.21.15.89:0/171758316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c440047d0 0x7f5c44004c50 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f5c40009ab0 tx=0x7f5c4001ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=543bc159ed384f04 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:32.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.497+0000 7f5c55d9b640 1 -- 172.21.15.89:0/171758316 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c400a5600 con 0x7f5c440047d0 2023-10-30T12:49:32.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.497+0000 7f5c55d9b640 1 -- 172.21.15.89:0/171758316 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c400a5760 con 0x7f5c440047d0 2023-10-30T12:49:32.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.497+0000 7f5c55d9b640 1 -- 172.21.15.89:0/171758316 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c400a5b10 con 0x7f5c440047d0 2023-10-30T12:49:32.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.497+0000 7f5c5cc09640 1 -- 172.21.15.89:0/171758316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c440047d0 msgr2=0x7f5c44004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:32.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.497+0000 7f5c5cc09640 1 --2- 172.21.15.89:0/171758316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c440047d0 0x7f5c44004c50 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f5c40009ab0 tx=0x7f5c4001ca10 comp rx=0 tx=0).stop 2023-10-30T12:49:32.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.497+0000 7f5c56d9d640 1 -- 172.21.15.89:0/171758316 reap_dead start 2023-10-30T12:49:32.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.497+0000 7f5c5cc09640 1 -- 172.21.15.89:0/171758316 shutdown_connections 2023-10-30T12:49:32.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.497+0000 7f5c5cc09640 1 -- 172.21.15.89:0/171758316 >> 172.21.15.89:0/171758316 conn(0x7f5c580c7910 msgr2=0x7f5c580c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:32.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.497+0000 7f5c5cc09640 1 -- 172.21.15.89:0/171758316 shutdown_connections 2023-10-30T12:49:32.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.497+0000 7f5c5cc09640 1 -- 172.21.15.89:0/171758316 wait complete. 2023-10-30T12:49:32.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.501+0000 7f5c5cc09640 1 Processor -- start 2023-10-30T12:49:32.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.501+0000 7f5c5cc09640 1 -- start start 2023-10-30T12:49:32.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.501+0000 7f5c5cc09640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c440047d0 0x7f5c581f0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:32.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.501+0000 7f5c5cc09640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c58157790 0x7f5c581f2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:32.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.501+0000 7f5c5cc09640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c581389b0 0x7f5c581eeec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:32.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.501+0000 7f5c5cc09640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c58153e60 con 0x7f5c58157790 2023-10-30T12:49:32.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.501+0000 7f5c5cc09640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c581ef730 con 0x7f5c440047d0 2023-10-30T12:49:32.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.501+0000 7f5c5cc09640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c581efa10 con 0x7f5c581389b0 2023-10-30T12:49:32.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.501+0000 7f5c4ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c58157790 0x7f5c581f2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:32.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.501+0000 7f5c4ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c58157790 0x7f5c581f2f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50006/0 (socket says 172.21.15.89:50006) 2023-10-30T12:49:32.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.501+0000 7f5c4ffff640 1 -- 172.21.15.89:0/2029911089 learned_addr learned my addr 172.21.15.89:0/2029911089 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:49:32.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.501+0000 7f5c56d9d640 1 --2- 172.21.15.89:0/2029911089 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c581389b0 0x7f5c581eeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:32.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.501+0000 7f5c5659c640 1 --2- 172.21.15.89:0/2029911089 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c440047d0 0x7f5c581f0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:32.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.501+0000 7f5c4ffff640 1 -- 172.21.15.89:0/2029911089 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c581389b0 msgr2=0x7f5c581eeec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:32.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.501+0000 7f5c4ffff640 1 --2- 172.21.15.89:0/2029911089 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c581389b0 0x7f5c581eeec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:32.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.501+0000 7f5c4ffff640 1 -- 172.21.15.89:0/2029911089 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c440047d0 msgr2=0x7f5c581f0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:32.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.501+0000 7f5c4ffff640 1 --2- 172.21.15.89:0/2029911089 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c440047d0 0x7f5c581f0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:32.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.501+0000 7f5c4ffff640 1 -- 172.21.15.89:0/2029911089 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c581efcf0 con 0x7f5c58157790 2023-10-30T12:49:32.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.501+0000 7f5c4ffff640 1 --2- 172.21.15.89:0/2029911089 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c58157790 0x7f5c581f2f10 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f5c400b0040 tx=0x7f5c44005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:32.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.501+0000 7f5c4effd640 1 -- 172.21.15.89:0/2029911089 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c400a5600 con 0x7f5c58157790 2023-10-30T12:49:32.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.501+0000 7f5c4effd640 1 -- 172.21.15.89:0/2029911089 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c40004260 con 0x7f5c58157790 2023-10-30T12:49:32.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.501+0000 7f5c5cc09640 1 -- 172.21.15.89:0/2029911089 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c581f2430 con 0x7f5c58157790 2023-10-30T12:49:32.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.501+0000 7f5c4effd640 1 -- 172.21.15.89:0/2029911089 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c40003620 con 0x7f5c58157790 2023-10-30T12:49:32.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.501+0000 7f5c5cc09640 1 -- 172.21.15.89:0/2029911089 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c581f29f0 con 0x7f5c58157790 2023-10-30T12:49:32.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.501+0000 7f5c5cc09640 1 -- 172.21.15.89:0/2029911089 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c5804e170 con 0x7f5c58157790 2023-10-30T12:49:32.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.505+0000 7f5c4effd640 1 -- 172.21.15.89:0/2029911089 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5c400b4040 con 0x7f5c58157790 2023-10-30T12:49:32.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.505+0000 7f5c4effd640 1 --2- 172.21.15.89:0/2029911089 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5c380664d0 0x7f5c38068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:32.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.505+0000 7f5c4effd640 1 -- 172.21.15.89:0/2029911089 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6664+0+0 (secure 0 0 0) 0x7f5c4011e3e0 con 0x7f5c58157790 2023-10-30T12:49:32.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.505+0000 7f5c5659c640 1 --2- 172.21.15.89:0/2029911089 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5c380664d0 0x7f5c38068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:32.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.505+0000 7f5c5659c640 1 --2- 172.21.15.89:0/2029911089 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5c380664d0 0x7f5c38068990 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f5c48005630 tx=0x7f5c48009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:32.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.509+0000 7f5c4effd640 1 -- 172.21.15.89:0/2029911089 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5c400eb700 con 0x7f5c58157790 2023-10-30T12:49:32.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.689+0000 7f5c5cc09640 1 -- 172.21.15.89:0/2029911089 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c5813b1d0 con 0x7f5c58157790 2023-10-30T12:49:32.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.689+0000 7f5c4effd640 1 -- 172.21.15.89:0/2029911089 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+5746 (secure 0 0 0) 0x7f5c400eed20 con 0x7f5c58157790 2023-10-30T12:49:32.690 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:49:32.690 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:49:25.550669+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5678,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5678":{"gid":5678,"name":"f","rank":0,"incarnation":98,"state":"up:active","state_seq":14,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:32.690 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 102 2023-10-30T12:49:32.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.693+0000 7f5c5cc09640 1 -- 172.21.15.89:0/2029911089 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5c380664d0 msgr2=0x7f5c38068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:32.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.693+0000 7f5c5cc09640 1 --2- 172.21.15.89:0/2029911089 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5c380664d0 0x7f5c38068990 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f5c48005630 tx=0x7f5c48009040 comp rx=0 tx=0).stop 2023-10-30T12:49:32.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.693+0000 7f5c5cc09640 1 -- 172.21.15.89:0/2029911089 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c58157790 msgr2=0x7f5c581f2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:32.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.693+0000 7f5c5cc09640 1 --2- 172.21.15.89:0/2029911089 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c58157790 0x7f5c581f2f10 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f5c400b0040 tx=0x7f5c44005800 comp rx=0 tx=0).stop 2023-10-30T12:49:32.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.693+0000 7f5c5cc09640 1 -- 172.21.15.89:0/2029911089 shutdown_connections 2023-10-30T12:49:32.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.693+0000 7f5c5cc09640 1 --2- 172.21.15.89:0/2029911089 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f5c380664d0 0x7f5c38068990 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:32.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.693+0000 7f5c5cc09640 1 --2- 172.21.15.89:0/2029911089 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c581389b0 0x7f5c581eeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:32.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.693+0000 7f5c5cc09640 1 --2- 172.21.15.89:0/2029911089 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c58157790 0x7f5c581f2f10 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:32.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.693+0000 7f5c5cc09640 1 --2- 172.21.15.89:0/2029911089 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c440047d0 0x7f5c581f0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:32.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.693+0000 7f5c5cc09640 1 -- 172.21.15.89:0/2029911089 >> 172.21.15.89:0/2029911089 conn(0x7f5c580c7910 msgr2=0x7f5c580bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:32.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.693+0000 7f5c5cc09640 1 -- 172.21.15.89:0/2029911089 shutdown_connections 2023-10-30T12:49:32.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:32.693+0000 7f5c5cc09640 1 -- 172.21.15.89:0/2029911089 wait complete. 2023-10-30T12:49:32.710 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2023-10-30T12:49:34.446 INFO:tasks.ceph.mds.f.smithi192.stderr:2023-10-30T12:49:34.446+0000 7f04f7102640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T12:49:34.447 INFO:tasks.ceph.mds.f.smithi192.stderr:2023-10-30T12:49:34.446+0000 7f04f7102640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T12:49:34.447 INFO:tasks.ceph.mds.f.smithi192.stderr:2023-10-30T12:49:34.446+0000 7f04f7102640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T12:49:34.711 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:34.859 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.861+0000 7f920b5f5640 1 Processor -- start 2023-10-30T12:49:34.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.861+0000 7f920b5f5640 1 -- start start 2023-10-30T12:49:34.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.861+0000 7f920b5f5640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f92040cf160 0x7f92040cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:34.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.861+0000 7f920b5f5640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92040cb800 con 0x7f92041538a0 2023-10-30T12:49:34.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.861+0000 7f920b5f5640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92040cb330 con 0x7f920415d6f0 2023-10-30T12:49:34.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.861+0000 7f920b5f5640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92040cad70 con 0x7f92040cf160 2023-10-30T12:49:34.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.861+0000 7f9209391640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f92041538a0 0x7f92040ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:45958/0 (socket says 172.21.15.89:45958) 2023-10-30T12:49:34.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.861+0000 7f9209391640 1 -- 172.21.15.89:0/195938496 learned_addr learned my addr 172.21.15.89:0/195938496 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:49:34.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.861+0000 7f9209b92640 1 --2- 172.21.15.89:0/195938496 >> v2:172.21.15.192:3300/0 conn(0x7f92040cf160 0x7f92040cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:34.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.861+0000 7f91fbfff640 1 -- 172.21.15.89:0/195938496 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1477532799 0 0) 0x7f92040cb800 con 0x7f92041538a0 2023-10-30T12:49:34.864 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.861+0000 7f91fbfff640 1 -- 172.21.15.89:0/195938496 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91f0003530 con 0x7f92041538a0 2023-10-30T12:49:34.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.861+0000 7f91fbfff640 1 -- 172.21.15.89:0/195938496 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1234032926 0 0) 0x7f91f0003530 con 0x7f92041538a0 2023-10-30T12:49:34.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.861+0000 7f91fbfff640 1 -- 172.21.15.89:0/195938496 >> v2:172.21.15.192:3300/0 conn(0x7f92040cf160 msgr2=0x7f92040cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:34.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.861+0000 7f91fbfff640 1 --2- 172.21.15.89:0/195938496 >> v2:172.21.15.192:3300/0 conn(0x7f92040cf160 0x7f92040cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:34.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.861+0000 7f91fbfff640 1 -- 172.21.15.89:0/195938496 >> v1:172.21.15.192:6789/0 conn(0x7f920415d6f0 legacy=0x7f920415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:34.865 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.861+0000 7f91fbfff640 1 -- 172.21.15.89:0/195938496 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92040cb050 con 0x7f92041538a0 2023-10-30T12:49:34.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.861+0000 7f91fbfff640 1 -- 172.21.15.89:0/195938496 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f9200001680 con 0x7f92041538a0 2023-10-30T12:49:34.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.861+0000 7f91fbfff640 1 -- 172.21.15.89:0/195938496 >> v1:172.21.15.89:6789/0 conn(0x7f92041538a0 legacy=0x7f92040ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:34.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.861+0000 7f91fbfff640 1 --2- 172.21.15.89:0/195938496 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91f0003980 0x7f91f0003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:34.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.861+0000 7f91fbfff640 1 --2- 172.21.15.89:0/195938496 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91f0004380 0x7f91f0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:34.866 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.861+0000 7f91fbfff640 1 --2- 172.21.15.89:0/195938496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91f0005050 0x7f91f000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:34.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.861+0000 7f91fbfff640 1 -- 172.21.15.89:0/195938496 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92040cad70 con 0x7f91f0005050 2023-10-30T12:49:34.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.861+0000 7f91fbfff640 1 -- 172.21.15.89:0/195938496 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f000bf00 con 0x7f91f0004380 2023-10-30T12:49:34.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.861+0000 7f91fbfff640 1 -- 172.21.15.89:0/195938496 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91f000c0c0 con 0x7f91f0003980 2023-10-30T12:49:34.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.865+0000 7f9209b92640 1 --2- 172.21.15.89:0/195938496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91f0005050 0x7f91f000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:34.867 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.865+0000 7f9209391640 1 --2- 172.21.15.89:0/195938496 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91f0003980 0x7f91f0003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:34.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.865+0000 7f9208b90640 1 --2- 172.21.15.89:0/195938496 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91f0004380 0x7f91f0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:34.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.865+0000 7f9209b92640 1 -- 172.21.15.89:0/195938496 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91f0003980 msgr2=0x7f91f0003d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:34.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.865+0000 7f9209b92640 1 --2- 172.21.15.89:0/195938496 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91f0003980 0x7f91f0003d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:34.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.865+0000 7f9209b92640 1 -- 172.21.15.89:0/195938496 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91f0004380 msgr2=0x7f91f0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:34.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.865+0000 7f9209b92640 1 --2- 172.21.15.89:0/195938496 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91f0004380 0x7f91f0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:34.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.865+0000 7f9209b92640 1 -- 172.21.15.89:0/195938496 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f0003530 con 0x7f91f0005050 2023-10-30T12:49:34.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.865+0000 7f9208b90640 1 --2- 172.21.15.89:0/195938496 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91f0004380 0x7f91f0004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:49:34.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.865+0000 7f9209b92640 1 -- 172.21.15.89:0/195938496 reap_dead start 2023-10-30T12:49:34.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.865+0000 7f9209b92640 1 --2- 172.21.15.89:0/195938496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91f0005050 0x7f91f000b990 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f91fc009fc0 tx=0x7f91fc01cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c32949e48ac2a23a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:34.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.865+0000 7f91fbfff640 1 -- 172.21.15.89:0/195938496 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91fc0a5cb0 con 0x7f91f0005050 2023-10-30T12:49:34.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.865+0000 7f91fbfff640 1 -- 172.21.15.89:0/195938496 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91fc00f040 con 0x7f91f0005050 2023-10-30T12:49:34.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.865+0000 7f91fbfff640 1 -- 172.21.15.89:0/195938496 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91fc003be0 con 0x7f91f0005050 2023-10-30T12:49:34.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.869+0000 7f920b5f5640 1 -- 172.21.15.89:0/195938496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91f0005050 msgr2=0x7f91f000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:34.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.869+0000 7f920b5f5640 1 --2- 172.21.15.89:0/195938496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91f0005050 0x7f91f000b990 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f91fc009fc0 tx=0x7f91fc01cca0 comp rx=0 tx=0).stop 2023-10-30T12:49:34.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.869+0000 7f920b5f5640 1 -- 172.21.15.89:0/195938496 shutdown_connections 2023-10-30T12:49:34.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.869+0000 7f920b5f5640 1 --2- 172.21.15.89:0/195938496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91f0005050 0x7f91f000b990 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:34.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.869+0000 7f920b5f5640 1 -- 172.21.15.89:0/195938496 >> 172.21.15.89:0/195938496 conn(0x7f920406bfe0 msgr2=0x7f920406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:34.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.869+0000 7f920b5f5640 1 -- 172.21.15.89:0/195938496 shutdown_connections 2023-10-30T12:49:34.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.869+0000 7f920b5f5640 1 -- 172.21.15.89:0/195938496 wait complete. 2023-10-30T12:49:34.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.869+0000 7f920b5f5640 1 Processor -- start 2023-10-30T12:49:34.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.869+0000 7f920b5f5640 1 -- start start 2023-10-30T12:49:34.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.869+0000 7f920b5f5640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91f0005050 0x7f92041f33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:34.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.869+0000 7f920b5f5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f920415b820 0x7f92040ce300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:34.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.869+0000 7f920b5f5640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f920414ed70 0x7f92041520d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:34.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.869+0000 7f920b5f5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92041f3910 con 0x7f920415b820 2023-10-30T12:49:34.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.869+0000 7f920b5f5640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92041f0320 con 0x7f920414ed70 2023-10-30T12:49:34.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.869+0000 7f920b5f5640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f920415bcb0 con 0x7f91f0005050 2023-10-30T12:49:34.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.869+0000 7f9208b90640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f920415b820 0x7f92040ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:34.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.869+0000 7f9208b90640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f920415b820 0x7f92040ce300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50024/0 (socket says 172.21.15.89:50024) 2023-10-30T12:49:34.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.869+0000 7f9208b90640 1 -- 172.21.15.89:0/3960898314 learned_addr learned my addr 172.21.15.89:0/3960898314 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:49:34.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.869+0000 7f9209391640 1 --2- 172.21.15.89:0/3960898314 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91f0005050 0x7f92041f33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:34.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.869+0000 7f9209b92640 1 --2- 172.21.15.89:0/3960898314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f920414ed70 0x7f92041520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:34.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.869+0000 7f9208b90640 1 -- 172.21.15.89:0/3960898314 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91f0005050 msgr2=0x7f92041f33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:34.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.869+0000 7f9208b90640 1 --2- 172.21.15.89:0/3960898314 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91f0005050 0x7f92041f33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:34.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.869+0000 7f9208b90640 1 -- 172.21.15.89:0/3960898314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f920414ed70 msgr2=0x7f92041520d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:34.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.869+0000 7f9208b90640 1 --2- 172.21.15.89:0/3960898314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f920414ed70 0x7f92041520d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:34.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.869+0000 7f9208b90640 1 -- 172.21.15.89:0/3960898314 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92041ef560 con 0x7f920415b820 2023-10-30T12:49:34.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.873+0000 7f9208b90640 1 --2- 172.21.15.89:0/3960898314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f920415b820 0x7f92040ce300 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f91f400e5d0 tx=0x7f91f400ea90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:34.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.873+0000 7f91f9ffb640 1 -- 172.21.15.89:0/3960898314 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f4017070 con 0x7f920415b820 2023-10-30T12:49:34.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.873+0000 7f91f9ffb640 1 -- 172.21.15.89:0/3960898314 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91f40052e0 con 0x7f920415b820 2023-10-30T12:49:34.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.873+0000 7f920b5f5640 1 -- 172.21.15.89:0/3960898314 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92041ef840 con 0x7f920415b820 2023-10-30T12:49:34.878 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.873+0000 7f91f9ffb640 1 -- 172.21.15.89:0/3960898314 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f4005450 con 0x7f920415b820 2023-10-30T12:49:34.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.873+0000 7f920b5f5640 1 -- 172.21.15.89:0/3960898314 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92041efe00 con 0x7f920415b820 2023-10-30T12:49:34.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.873+0000 7f920b5f5640 1 -- 172.21.15.89:0/3960898314 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f920404e170 con 0x7f920415b820 2023-10-30T12:49:34.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.873+0000 7f91f9ffb640 1 -- 172.21.15.89:0/3960898314 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f91f4007020 con 0x7f920415b820 2023-10-30T12:49:34.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.873+0000 7f91f9ffb640 1 --2- 172.21.15.89:0/3960898314 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f91e80664a0 0x7f91e8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:34.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.873+0000 7f91f9ffb640 1 -- 172.21.15.89:0/3960898314 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6664+0+0 (secure 0 0 0) 0x7f91f4083b00 con 0x7f920415b820 2023-10-30T12:49:34.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.873+0000 7f9209391640 1 --2- 172.21.15.89:0/3960898314 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f91e80664a0 0x7f91e8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:34.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.877+0000 7f9209391640 1 --2- 172.21.15.89:0/3960898314 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f91e80664a0 0x7f91e8068960 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9200000c10 tx=0x7f9200009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:34.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:34.877+0000 7f91f9ffb640 1 -- 172.21.15.89:0/3960898314 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f91f4050b30 con 0x7f920415b820 2023-10-30T12:49:35.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:35.057+0000 7f920b5f5640 1 -- 172.21.15.89:0/3960898314 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92040c4630 con 0x7f920415b820 2023-10-30T12:49:35.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:35.057+0000 7f91f9ffb640 1 -- 172.21.15.89:0/3960898314 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+6538 (secure 0 0 0) 0x7f91f4054150 con 0x7f920415b820 2023-10-30T12:49:35.058 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:49:35.058 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6271,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/659375964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":659375964},{"type":"v1","addr":"172.21.15.192:6835","nonce":659375964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:49:25.550669+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5678,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5678":{"gid":5678,"name":"f","rank":0,"incarnation":98,"state":"up:active","state_seq":14,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:35.058 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 103 2023-10-30T12:49:35.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:35.061+0000 7f920b5f5640 1 -- 172.21.15.89:0/3960898314 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f91e80664a0 msgr2=0x7f91e8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:35.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:35.061+0000 7f920b5f5640 1 --2- 172.21.15.89:0/3960898314 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f91e80664a0 0x7f91e8068960 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9200000c10 tx=0x7f9200009040 comp rx=0 tx=0).stop 2023-10-30T12:49:35.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:35.061+0000 7f920b5f5640 1 -- 172.21.15.89:0/3960898314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f920415b820 msgr2=0x7f92040ce300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:35.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:35.061+0000 7f920b5f5640 1 --2- 172.21.15.89:0/3960898314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f920415b820 0x7f92040ce300 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f91f400e5d0 tx=0x7f91f400ea90 comp rx=0 tx=0).stop 2023-10-30T12:49:35.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:35.061+0000 7f920b5f5640 1 -- 172.21.15.89:0/3960898314 shutdown_connections 2023-10-30T12:49:35.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:35.061+0000 7f920b5f5640 1 --2- 172.21.15.89:0/3960898314 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f91e80664a0 0x7f91e8068960 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:35.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:35.061+0000 7f920b5f5640 1 --2- 172.21.15.89:0/3960898314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f920414ed70 0x7f92041520d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:35.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:35.061+0000 7f920b5f5640 1 --2- 172.21.15.89:0/3960898314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f920415b820 0x7f92040ce300 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:35.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:35.061+0000 7f920b5f5640 1 --2- 172.21.15.89:0/3960898314 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91f0005050 0x7f92041f33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:35.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:35.061+0000 7f920b5f5640 1 -- 172.21.15.89:0/3960898314 >> 172.21.15.89:0/3960898314 conn(0x7f920406bfe0 msgr2=0x7f9204065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:35.061 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:35.061+0000 7f920b5f5640 1 -- 172.21.15.89:0/3960898314 shutdown_connections 2023-10-30T12:49:35.062 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:49:35.061+0000 7f920b5f5640 1 -- 172.21.15.89:0/3960898314 wait complete. 2023-10-30T12:49:35.078 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2023-10-30T12:49:35.079 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 47 secs before thrashing 2023-10-30T12:49:48.355 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:49:48.360 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:50:18.413 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:50:18.418 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:50:22.080 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:50:22.228 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.228+0000 7fb76b52a640 1 Processor -- start 2023-10-30T12:50:22.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.228+0000 7fb76b52a640 1 -- start start 2023-10-30T12:50:22.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.228+0000 7fb76b52a640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb76406f7a0 0x7fb76406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:22.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.228+0000 7fb76b52a640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb764071110 0x7fb764158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:22.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.228+0000 7fb76b52a640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb764066cf0 con 0x7fb7640c6bd0 2023-10-30T12:50:22.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.228+0000 7fb76b52a640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7640700c0 con 0x7fb76406f7a0 2023-10-30T12:50:22.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.228+0000 7fb76b52a640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7640c70d0 con 0x7fb764071110 2023-10-30T12:50:22.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.228+0000 7fb7692c6640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb76406f7a0 0x7fb76406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:22.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.228+0000 7fb7692c6640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb76406f7a0 0x7fb76406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59578/0 (socket says 172.21.15.89:59578) 2023-10-30T12:50:22.232 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.228+0000 7fb7692c6640 1 -- 172.21.15.89:0/676872960 learned_addr learned my addr 172.21.15.89:0/676872960 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:50:22.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.228+0000 7fb769ac7640 1 --2- 172.21.15.89:0/676872960 >> v2:172.21.15.192:3300/0 conn(0x7fb764071110 0x7fb764158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:22.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.228+0000 7fb7692c6640 1 -- 172.21.15.89:0/676872960 >> v2:172.21.15.192:3300/0 conn(0x7fb764071110 msgr2=0x7fb764158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:50:22.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.228+0000 7fb7692c6640 1 --2- 172.21.15.89:0/676872960 >> v2:172.21.15.192:3300/0 conn(0x7fb764071110 0x7fb764158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:22.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.232+0000 7fb7692c6640 1 -- 172.21.15.89:0/676872960 >> v1:172.21.15.192:6789/0 conn(0x7fb7640c6bd0 legacy=0x7fb764069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:50:22.233 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.232+0000 7fb7692c6640 1 -- 172.21.15.89:0/676872960 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7640c7390 con 0x7fb76406f7a0 2023-10-30T12:50:22.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.232+0000 7fb7692c6640 1 --2- 172.21.15.89:0/676872960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb76406f7a0 0x7fb76406fb80 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fb76406cdd0 tx=0x7fb76001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a203ca18c5887a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:22.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.232+0000 7fb75bfff640 1 -- 172.21.15.89:0/676872960 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7600a89f0 con 0x7fb76406f7a0 2023-10-30T12:50:22.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.232+0000 7fb75bfff640 1 -- 172.21.15.89:0/676872960 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7600a8b50 con 0x7fb76406f7a0 2023-10-30T12:50:22.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.232+0000 7fb75bfff640 1 -- 172.21.15.89:0/676872960 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7600b0440 con 0x7fb76406f7a0 2023-10-30T12:50:22.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.232+0000 7fb76b52a640 1 -- 172.21.15.89:0/676872960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb76406f7a0 msgr2=0x7fb76406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:22.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.232+0000 7fb76b52a640 1 --2- 172.21.15.89:0/676872960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb76406f7a0 0x7fb76406fb80 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fb76406cdd0 tx=0x7fb76001ecf0 comp rx=0 tx=0).stop 2023-10-30T12:50:22.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.232+0000 7fb76b52a640 1 -- 172.21.15.89:0/676872960 shutdown_connections 2023-10-30T12:50:22.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.232+0000 7fb76b52a640 1 --2- 172.21.15.89:0/676872960 >> v2:172.21.15.192:3300/0 conn(0x7fb764071110 0x7fb764158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:22.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.232+0000 7fb76b52a640 1 --2- 172.21.15.89:0/676872960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb76406f7a0 0x7fb76406fb80 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:22.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.232+0000 7fb76b52a640 1 -- 172.21.15.89:0/676872960 >> 172.21.15.89:0/676872960 conn(0x7fb7640c7910 msgr2=0x7fb7640c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:22.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.236+0000 7fb76b52a640 1 -- 172.21.15.89:0/676872960 shutdown_connections 2023-10-30T12:50:22.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.236+0000 7fb76b52a640 1 -- 172.21.15.89:0/676872960 wait complete. 2023-10-30T12:50:22.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.236+0000 7fb76b52a640 1 Processor -- start 2023-10-30T12:50:22.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.236+0000 7fb76b52a640 1 -- start start 2023-10-30T12:50:22.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.236+0000 7fb76b52a640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb76406f7a0 0x7fb764157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:22.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.236+0000 7fb76b52a640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb764071110 0x7fb7640c3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:22.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.236+0000 7fb76b52a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7640c6bd0 0x7fb76414f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:22.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.236+0000 7fb76b52a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb764156c50 con 0x7fb7640c6bd0 2023-10-30T12:50:22.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.236+0000 7fb76b52a640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb764157070 con 0x7fb764071110 2023-10-30T12:50:22.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.236+0000 7fb76b52a640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb764155cf0 con 0x7fb76406f7a0 2023-10-30T12:50:22.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.236+0000 7fb769ac7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7640c6bd0 0x7fb76414f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:22.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.236+0000 7fb769ac7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7640c6bd0 0x7fb76414f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59590/0 (socket says 172.21.15.89:59590) 2023-10-30T12:50:22.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.236+0000 7fb769ac7640 1 -- 172.21.15.89:0/1485685046 learned_addr learned my addr 172.21.15.89:0/1485685046 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:50:22.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.236+0000 7fb7692c6640 1 --2- 172.21.15.89:0/1485685046 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb76406f7a0 0x7fb764157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:22.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.236+0000 7fb768ac5640 1 --2- 172.21.15.89:0/1485685046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb764071110 0x7fb7640c3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:22.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.236+0000 7fb769ac7640 1 -- 172.21.15.89:0/1485685046 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb76406f7a0 msgr2=0x7fb764157d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:22.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.236+0000 7fb769ac7640 1 --2- 172.21.15.89:0/1485685046 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb76406f7a0 0x7fb764157d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:22.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.236+0000 7fb769ac7640 1 -- 172.21.15.89:0/1485685046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb764071110 msgr2=0x7fb7640c3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:22.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.236+0000 7fb769ac7640 1 --2- 172.21.15.89:0/1485685046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb764071110 0x7fb7640c3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:22.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.236+0000 7fb769ac7640 1 -- 172.21.15.89:0/1485685046 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7600094d0 con 0x7fb7640c6bd0 2023-10-30T12:50:22.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.236+0000 7fb769ac7640 1 --2- 172.21.15.89:0/1485685046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7640c6bd0 0x7fb76414f190 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fb75c007e60 tx=0x7fb75c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:22.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.236+0000 7fb759ffb640 1 -- 172.21.15.89:0/1485685046 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75c014420 con 0x7fb7640c6bd0 2023-10-30T12:50:22.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.236+0000 7fb759ffb640 1 -- 172.21.15.89:0/1485685046 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb75c0145a0 con 0x7fb7640c6bd0 2023-10-30T12:50:22.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.236+0000 7fb759ffb640 1 -- 172.21.15.89:0/1485685046 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75c014950 con 0x7fb7640c6bd0 2023-10-30T12:50:22.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.240+0000 7fb76b52a640 1 -- 172.21.15.89:0/1485685046 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb764155fd0 con 0x7fb7640c6bd0 2023-10-30T12:50:22.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.240+0000 7fb76b52a640 1 -- 172.21.15.89:0/1485685046 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb764156590 con 0x7fb7640c6bd0 2023-10-30T12:50:22.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.240+0000 7fb76b52a640 1 -- 172.21.15.89:0/1485685046 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb764156d90 con 0x7fb7640c6bd0 2023-10-30T12:50:22.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.240+0000 7fb759ffb640 1 -- 172.21.15.89:0/1485685046 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb75c014ab0 con 0x7fb7640c6bd0 2023-10-30T12:50:22.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.240+0000 7fb759ffb640 1 --2- 172.21.15.89:0/1485685046 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb7480664a0 0x7fb748068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:22.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.240+0000 7fb759ffb640 1 -- 172.21.15.89:0/1485685046 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6664+0+0 (secure 0 0 0) 0x7fb75c084b60 con 0x7fb7640c6bd0 2023-10-30T12:50:22.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.240+0000 7fb7692c6640 1 --2- 172.21.15.89:0/1485685046 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb7480664a0 0x7fb748068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:22.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.244+0000 7fb7692c6640 1 --2- 172.21.15.89:0/1485685046 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb7480664a0 0x7fb748068960 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb7600b2000 tx=0x7fb760008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:22.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.244+0000 7fb759ffb640 1 -- 172.21.15.89:0/1485685046 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb764156d90 con 0x7fb7640c6bd0 2023-10-30T12:50:22.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.428+0000 7fb76b52a640 1 -- 172.21.15.89:0/1485685046 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb764156d90 con 0x7fb7640c6bd0 2023-10-30T12:50:22.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.428+0000 7fb759ffb640 1 -- 172.21.15.89:0/1485685046 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+6544 (secure 0 0 0) 0x7fb764156d90 con 0x7fb7640c6bd0 2023-10-30T12:50:22.430 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:50:22.430 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":109,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6271,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/659375964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":659375964},{"type":"v1","addr":"172.21.15.192:6835","nonce":659375964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:50:20.403164+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5678,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5678":{"gid":5678,"name":"f","rank":0,"incarnation":98,"state":"up:active","state_seq":14,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:50:22.431 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 109 2023-10-30T12:50:22.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.432+0000 7fb76b52a640 1 -- 172.21.15.89:0/1485685046 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb7480664a0 msgr2=0x7fb748068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:22.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.432+0000 7fb76b52a640 1 --2- 172.21.15.89:0/1485685046 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb7480664a0 0x7fb748068960 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb7600b2000 tx=0x7fb760008040 comp rx=0 tx=0).stop 2023-10-30T12:50:22.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.432+0000 7fb76b52a640 1 -- 172.21.15.89:0/1485685046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7640c6bd0 msgr2=0x7fb76414f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:22.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.432+0000 7fb76b52a640 1 --2- 172.21.15.89:0/1485685046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7640c6bd0 0x7fb76414f190 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fb75c007e60 tx=0x7fb75c00e4b0 comp rx=0 tx=0).stop 2023-10-30T12:50:22.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.432+0000 7fb76b52a640 1 -- 172.21.15.89:0/1485685046 shutdown_connections 2023-10-30T12:50:22.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.432+0000 7fb76b52a640 1 --2- 172.21.15.89:0/1485685046 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb7480664a0 0x7fb748068960 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:22.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.432+0000 7fb76b52a640 1 --2- 172.21.15.89:0/1485685046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb764071110 0x7fb7640c3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:22.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.432+0000 7fb76b52a640 1 --2- 172.21.15.89:0/1485685046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7640c6bd0 0x7fb76414f190 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:22.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.432+0000 7fb76b52a640 1 --2- 172.21.15.89:0/1485685046 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb76406f7a0 0x7fb764157d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:22.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.432+0000 7fb76b52a640 1 -- 172.21.15.89:0/1485685046 >> 172.21.15.89:0/1485685046 conn(0x7fb7640c7910 msgr2=0x7fb7640c9550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:22.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.432+0000 7fb76b52a640 1 -- 172.21.15.89:0/1485685046 shutdown_connections 2023-10-30T12:50:22.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.432+0000 7fb76b52a640 1 -- 172.21.15.89:0/1485685046 wait complete. 2023-10-30T12:50:22.451 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=0) 2023-10-30T12:50:22.451 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-30T12:50:22.451 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:50:22.469 INFO:tasks.ceph.mds.f:Stopped 2023-10-30T12:50:22.470 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:50:22.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.608+0000 7f0cf01a1640 1 Processor -- start 2023-10-30T12:50:22.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.608+0000 7f0cf01a1640 1 -- start start 2023-10-30T12:50:22.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.608+0000 7f0cf01a1640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0ce81580e0 0x7f0ce8070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:22.614 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.608+0000 7f0cf01a1640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0ce815e700 0x7f0ce806a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:22.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.608+0000 7f0cf01a1640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce80c5810 con 0x7f0ce80c6540 2023-10-30T12:50:22.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.608+0000 7f0cf01a1640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ce80c6990 con 0x7f0ce81580e0 2023-10-30T12:50:22.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.608+0000 7f0cf01a1640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ce815eb90 con 0x7f0ce815e700 2023-10-30T12:50:22.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.608+0000 7f0cedf3d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0ce81580e0 0x7f0ce8070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:22.615 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.608+0000 7f0cedf3d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0ce81580e0 0x7f0ce8070740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59594/0 (socket says 172.21.15.89:59594) 2023-10-30T12:50:22.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.608+0000 7f0cedf3d640 1 -- 172.21.15.89:0/3965058290 learned_addr learned my addr 172.21.15.89:0/3965058290 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:50:22.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.608+0000 7f0ced73c640 1 --1- 172.21.15.89:0/3965058290 >> v1:172.21.15.89:6789/0 conn(0x7f0ce80c6540 0x7f0ce8069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41112/0 (socket says 172.21.15.89:41112) 2023-10-30T12:50:22.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.612+0000 7f0cee73e640 1 --2- 172.21.15.89:0/3965058290 >> v2:172.21.15.192:3300/0 conn(0x7f0ce815e700 0x7f0ce806a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:22.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.612+0000 7f0cedf3d640 1 -- 172.21.15.89:0/3965058290 >> v2:172.21.15.192:3300/0 conn(0x7f0ce815e700 msgr2=0x7f0ce806a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:50:22.616 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.612+0000 7f0cedf3d640 1 --2- 172.21.15.89:0/3965058290 >> v2:172.21.15.192:3300/0 conn(0x7f0ce815e700 0x7f0ce806a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:22.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.612+0000 7f0cedf3d640 1 -- 172.21.15.89:0/3965058290 >> v1:172.21.15.89:6789/0 conn(0x7f0ce80c6540 legacy=0x7f0ce8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:22.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.612+0000 7f0cedf3d640 1 -- 172.21.15.89:0/3965058290 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce80c6bb0 con 0x7f0ce81580e0 2023-10-30T12:50:22.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.612+0000 7f0cecf3b640 1 -- 172.21.15.89:0/3965058290 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 26281194 0 0) 0x7f0ce80c5810 con 0x7f0ce80c6540 2023-10-30T12:50:22.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.612+0000 7f0cee73e640 1 --2- 172.21.15.89:0/3965058290 >> v2:172.21.15.192:3300/0 conn(0x7f0ce815e700 0x7f0ce806a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:50:22.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.612+0000 7f0cedf3d640 1 --2- 172.21.15.89:0/3965058290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ce81580e0 0x7f0ce8070740 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8066ec0 tx=0x7f0ce401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9810d9dc945d36c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:22.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.612+0000 7f0cecf3b640 1 -- 172.21.15.89:0/3965058290 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce40aa430 con 0x7f0ce81580e0 2023-10-30T12:50:22.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.612+0000 7f0cecf3b640 1 -- 172.21.15.89:0/3965058290 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce40aa590 con 0x7f0ce81580e0 2023-10-30T12:50:22.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.612+0000 7f0cecf3b640 1 -- 172.21.15.89:0/3965058290 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce40aa920 con 0x7f0ce81580e0 2023-10-30T12:50:22.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.616+0000 7f0cf01a1640 1 -- 172.21.15.89:0/3965058290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ce81580e0 msgr2=0x7f0ce8070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:22.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.616+0000 7f0cf01a1640 1 --2- 172.21.15.89:0/3965058290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ce81580e0 0x7f0ce8070740 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8066ec0 tx=0x7f0ce401c4d0 comp rx=0 tx=0).stop 2023-10-30T12:50:22.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.616+0000 7f0cf01a1640 1 -- 172.21.15.89:0/3965058290 shutdown_connections 2023-10-30T12:50:22.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.616+0000 7f0cf01a1640 1 --2- 172.21.15.89:0/3965058290 >> v2:172.21.15.192:3300/0 conn(0x7f0ce815e700 0x7f0ce806a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:22.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.616+0000 7f0cf01a1640 1 --2- 172.21.15.89:0/3965058290 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ce81580e0 0x7f0ce8070740 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:22.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.616+0000 7f0cf01a1640 1 -- 172.21.15.89:0/3965058290 >> 172.21.15.89:0/3965058290 conn(0x7f0ce80c7910 msgr2=0x7f0ce80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:22.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.616+0000 7f0cf01a1640 1 -- 172.21.15.89:0/3965058290 shutdown_connections 2023-10-30T12:50:22.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.616+0000 7f0cf01a1640 1 -- 172.21.15.89:0/3965058290 wait complete. 2023-10-30T12:50:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.616+0000 7f0cf01a1640 1 Processor -- start 2023-10-30T12:50:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.616+0000 7f0cf01a1640 1 -- start start 2023-10-30T12:50:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.616+0000 7f0cf01a1640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ce80c6540 0x7f0ce8155ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.616+0000 7f0cf01a1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ce81580e0 0x7f0ce8161160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.616+0000 7f0cf01a1640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ce815e700 0x7f0ce81616a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.616+0000 7f0cf01a1640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce81f91c0 con 0x7f0ce81580e0 2023-10-30T12:50:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.616+0000 7f0cf01a1640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce81f94a0 con 0x7f0ce815e700 2023-10-30T12:50:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.616+0000 7f0cf01a1640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce81f9780 con 0x7f0ce80c6540 2023-10-30T12:50:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.616+0000 7f0ced73c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ce81580e0 0x7f0ce8161160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.616+0000 7f0ced73c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ce81580e0 0x7f0ce8161160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59602/0 (socket says 172.21.15.89:59602) 2023-10-30T12:50:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.616+0000 7f0ced73c640 1 -- 172.21.15.89:0/1391252311 learned_addr learned my addr 172.21.15.89:0/1391252311 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:50:22.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.616+0000 7f0cedf3d640 1 --2- 172.21.15.89:0/1391252311 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ce80c6540 0x7f0ce8155ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:22.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.616+0000 7f0cee73e640 1 --2- 172.21.15.89:0/1391252311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ce815e700 0x7f0ce81616a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:22.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.616+0000 7f0ced73c640 1 -- 172.21.15.89:0/1391252311 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ce80c6540 msgr2=0x7f0ce8155ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:22.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.616+0000 7f0ced73c640 1 --2- 172.21.15.89:0/1391252311 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ce80c6540 0x7f0ce8155ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:22.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.616+0000 7f0ced73c640 1 -- 172.21.15.89:0/1391252311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ce815e700 msgr2=0x7f0ce81616a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:22.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.616+0000 7f0ced73c640 1 --2- 172.21.15.89:0/1391252311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ce815e700 0x7f0ce81616a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:22.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.616+0000 7f0ced73c640 1 -- 172.21.15.89:0/1391252311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce40094d0 con 0x7f0ce81580e0 2023-10-30T12:50:22.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.616+0000 7f0ced73c640 1 --2- 172.21.15.89:0/1391252311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ce81580e0 0x7f0ce8161160 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f0cd80011f0 tx=0x7f0cd800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:22.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.616+0000 7f0cde7fc640 1 -- 172.21.15.89:0/1391252311 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd80074e0 con 0x7f0ce81580e0 2023-10-30T12:50:22.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.620+0000 7f0cde7fc640 1 -- 172.21.15.89:0/1391252311 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd8007640 con 0x7f0ce81580e0 2023-10-30T12:50:22.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.620+0000 7f0cde7fc640 1 -- 172.21.15.89:0/1391252311 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd8004a10 con 0x7f0ce81580e0 2023-10-30T12:50:22.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.620+0000 7f0cf01a1640 1 -- 172.21.15.89:0/1391252311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ce81f9a60 con 0x7f0ce81580e0 2023-10-30T12:50:22.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.620+0000 7f0cf01a1640 1 -- 172.21.15.89:0/1391252311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce8156f30 con 0x7f0ce81580e0 2023-10-30T12:50:22.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.620+0000 7f0cf01a1640 1 -- 172.21.15.89:0/1391252311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ce804e170 con 0x7f0ce81580e0 2023-10-30T12:50:22.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.620+0000 7f0cde7fc640 1 -- 172.21.15.89:0/1391252311 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0cd8004b70 con 0x7f0ce81580e0 2023-10-30T12:50:22.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.620+0000 7f0cde7fc640 1 --2- 172.21.15.89:0/1391252311 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0ccc0664a0 0x7f0ccc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:22.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.620+0000 7f0cde7fc640 1 -- 172.21.15.89:0/1391252311 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6664+0+0 (secure 0 0 0) 0x7f0cd8083490 con 0x7f0ce81580e0 2023-10-30T12:50:22.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.620+0000 7f0cedf3d640 1 --2- 172.21.15.89:0/1391252311 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0ccc0664a0 0x7f0ccc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:22.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.624+0000 7f0cedf3d640 1 --2- 172.21.15.89:0/1391252311 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0ccc0664a0 0x7f0ccc068960 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f0ce40ae000 tx=0x7f0ce40a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:22.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.624+0000 7f0cde7fc640 1 -- 172.21.15.89:0/1391252311 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0cd80505e0 con 0x7f0ce81580e0 2023-10-30T12:50:22.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.892+0000 7f0cf01a1640 1 -- 172.21.15.89:0/1391252311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ce806aed0 con 0x7f0ce81580e0 2023-10-30T12:50:22.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.892+0000 7f0cde7fc640 1 -- 172.21.15.89:0/1391252311 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+6544 (secure 0 0 0) 0x7f0cd8018090 con 0x7f0ce81580e0 2023-10-30T12:50:22.894 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:50:22.894 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":109,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6271,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/659375964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":659375964},{"type":"v1","addr":"172.21.15.192:6835","nonce":659375964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:50:20.403164+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5678,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5678":{"gid":5678,"name":"f","rank":0,"incarnation":98,"state":"up:active","state_seq":14,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:50:22.895 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 109 2023-10-30T12:50:22.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.896+0000 7f0cf01a1640 1 -- 172.21.15.89:0/1391252311 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0ccc0664a0 msgr2=0x7f0ccc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:22.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.896+0000 7f0cf01a1640 1 --2- 172.21.15.89:0/1391252311 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0ccc0664a0 0x7f0ccc068960 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f0ce40ae000 tx=0x7f0ce40a7000 comp rx=0 tx=0).stop 2023-10-30T12:50:22.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.896+0000 7f0cf01a1640 1 -- 172.21.15.89:0/1391252311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ce81580e0 msgr2=0x7f0ce8161160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:22.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.896+0000 7f0cf01a1640 1 --2- 172.21.15.89:0/1391252311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ce81580e0 0x7f0ce8161160 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f0cd80011f0 tx=0x7f0cd800c9b0 comp rx=0 tx=0).stop 2023-10-30T12:50:22.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.896+0000 7f0cf01a1640 1 -- 172.21.15.89:0/1391252311 shutdown_connections 2023-10-30T12:50:22.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.896+0000 7f0cf01a1640 1 --2- 172.21.15.89:0/1391252311 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0ccc0664a0 0x7f0ccc068960 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:22.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.896+0000 7f0cf01a1640 1 --2- 172.21.15.89:0/1391252311 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ce815e700 0x7f0ce81616a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:22.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.896+0000 7f0cf01a1640 1 --2- 172.21.15.89:0/1391252311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ce81580e0 0x7f0ce8161160 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:22.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.896+0000 7f0cf01a1640 1 --2- 172.21.15.89:0/1391252311 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ce80c6540 0x7f0ce8155ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:22.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.896+0000 7f0cf01a1640 1 -- 172.21.15.89:0/1391252311 >> 172.21.15.89:0/1391252311 conn(0x7f0ce80c7910 msgr2=0x7f0ce80c7e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:22.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.896+0000 7f0cf01a1640 1 -- 172.21.15.89:0/1391252311 shutdown_connections 2023-10-30T12:50:22.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:22.896+0000 7f0cf01a1640 1 -- 172.21.15.89:0/1391252311 wait complete. 2023-10-30T12:50:22.914 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:50:24.915 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:50:25.063 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.063+0000 7fb91f691640 1 Processor -- start 2023-10-30T12:50:25.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.063+0000 7fb91f691640 1 -- start start 2023-10-30T12:50:25.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.063+0000 7fb91f691640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9181538a0 0x7fb91815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:25.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.063+0000 7fb91f691640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb9180cf160 0x7fb9180cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:25.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.063+0000 7fb91f691640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9180cb800 con 0x7fb9180ca0e0 2023-10-30T12:50:25.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.063+0000 7fb91f691640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb91815dc30 con 0x7fb9180cf160 2023-10-30T12:50:25.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.063+0000 7fb91f691640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9180cb330 con 0x7fb9181538a0 2023-10-30T12:50:25.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.063+0000 7fb91dc2e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb9180cf160 0x7fb9180cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:25.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.063+0000 7fb91dc2e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb9180cf160 0x7fb9180cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59606/0 (socket says 172.21.15.89:59606) 2023-10-30T12:50:25.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.063+0000 7fb91dc2e640 1 -- 172.21.15.89:0/1021067151 learned_addr learned my addr 172.21.15.89:0/1021067151 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:50:25.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.063+0000 7fb91cc2c640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb9180ca0e0 0x7fb91815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41114/0 (socket says 172.21.15.89:41114) 2023-10-30T12:50:25.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.063+0000 7fb91d42d640 1 --2- 172.21.15.89:0/1021067151 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9181538a0 0x7fb91815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:25.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.063+0000 7fb91dc2e640 1 -- 172.21.15.89:0/1021067151 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9181538a0 msgr2=0x7fb91815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:50:25.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.063+0000 7fb91dc2e640 1 --2- 172.21.15.89:0/1021067151 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9181538a0 0x7fb91815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:25.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.063+0000 7fb91dc2e640 1 -- 172.21.15.89:0/1021067151 >> v1:172.21.15.89:6789/0 conn(0x7fb9180ca0e0 legacy=0x7fb91815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:25.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.063+0000 7fb91dc2e640 1 -- 172.21.15.89:0/1021067151 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9180cad70 con 0x7fb9180cf160 2023-10-30T12:50:25.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.067+0000 7fb91dc2e640 1 --2- 172.21.15.89:0/1021067151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9180cf160 0x7fb9180cf590 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fb910009e50 tx=0x7fb91001f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25d6e04ae4e39f25 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:25.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.067+0000 7fb90ffff640 1 -- 172.21.15.89:0/1021067151 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9100af400 con 0x7fb9180cf160 2023-10-30T12:50:25.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.067+0000 7fb90ffff640 1 -- 172.21.15.89:0/1021067151 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9100ab050 con 0x7fb9180cf160 2023-10-30T12:50:25.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.067+0000 7fb90ffff640 1 -- 172.21.15.89:0/1021067151 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9100af780 con 0x7fb9180cf160 2023-10-30T12:50:25.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.067+0000 7fb91f691640 1 -- 172.21.15.89:0/1021067151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9180cf160 msgr2=0x7fb9180cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:25.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.067+0000 7fb91f691640 1 --2- 172.21.15.89:0/1021067151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9180cf160 0x7fb9180cf590 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fb910009e50 tx=0x7fb91001f1f0 comp rx=0 tx=0).stop 2023-10-30T12:50:25.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.067+0000 7fb91f691640 1 -- 172.21.15.89:0/1021067151 shutdown_connections 2023-10-30T12:50:25.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.067+0000 7fb91f691640 1 --2- 172.21.15.89:0/1021067151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9180cf160 0x7fb9180cf590 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:25.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.067+0000 7fb91f691640 1 --2- 172.21.15.89:0/1021067151 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9181538a0 0x7fb91815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:25.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.067+0000 7fb91f691640 1 -- 172.21.15.89:0/1021067151 >> 172.21.15.89:0/1021067151 conn(0x7fb91806bfe0 msgr2=0x7fb91806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:25.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.067+0000 7fb91f691640 1 -- 172.21.15.89:0/1021067151 shutdown_connections 2023-10-30T12:50:25.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.067+0000 7fb91f691640 1 -- 172.21.15.89:0/1021067151 wait complete. 2023-10-30T12:50:25.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.071+0000 7fb91f691640 1 Processor -- start 2023-10-30T12:50:25.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.071+0000 7fb91f691640 1 -- start start 2023-10-30T12:50:25.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.071+0000 7fb91f691640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9180ca0e0 0x7fb91815a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:25.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.071+0000 7fb91f691640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9180cf160 0x7fb91815b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:25.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.071+0000 7fb91f691640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9181538a0 0x7fb91815b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:25.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.071+0000 7fb91f691640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9181e3960 con 0x7fb9180ca0e0 2023-10-30T12:50:25.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.071+0000 7fb91f691640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9181e4750 con 0x7fb9181538a0 2023-10-30T12:50:25.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.071+0000 7fb91f691640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9181e3ba0 con 0x7fb9180cf160 2023-10-30T12:50:25.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.071+0000 7fb91d42d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9180ca0e0 0x7fb91815a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:25.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.071+0000 7fb91d42d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9180ca0e0 0x7fb91815a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59616/0 (socket says 172.21.15.89:59616) 2023-10-30T12:50:25.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.071+0000 7fb91d42d640 1 -- 172.21.15.89:0/172846044 learned_addr learned my addr 172.21.15.89:0/172846044 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:50:25.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.071+0000 7fb91cc2c640 1 --2- 172.21.15.89:0/172846044 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9180cf160 0x7fb91815b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:25.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.071+0000 7fb91dc2e640 1 --2- 172.21.15.89:0/172846044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9181538a0 0x7fb91815b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:25.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.071+0000 7fb91d42d640 1 -- 172.21.15.89:0/172846044 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9180cf160 msgr2=0x7fb91815b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:25.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.071+0000 7fb91d42d640 1 --2- 172.21.15.89:0/172846044 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9180cf160 0x7fb91815b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:25.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.071+0000 7fb91d42d640 1 -- 172.21.15.89:0/172846044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9181538a0 msgr2=0x7fb91815b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:25.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.071+0000 7fb91d42d640 1 --2- 172.21.15.89:0/172846044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9181538a0 0x7fb91815b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:25.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.071+0000 7fb91d42d640 1 -- 172.21.15.89:0/172846044 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb910009b30 con 0x7fb9180ca0e0 2023-10-30T12:50:25.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.071+0000 7fb91dc2e640 1 --2- 172.21.15.89:0/172846044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9181538a0 0x7fb91815b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:50:25.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.071+0000 7fb91cc2c640 1 --2- 172.21.15.89:0/172846044 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9180cf160 0x7fb91815b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:50:25.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.071+0000 7fb91d42d640 1 --2- 172.21.15.89:0/172846044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9180ca0e0 0x7fb91815a8b0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fb91400d5f0 tx=0x7fb91400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:25.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.071+0000 7fb90dffb640 1 -- 172.21.15.89:0/172846044 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9140125f0 con 0x7fb9180ca0e0 2023-10-30T12:50:25.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.071+0000 7fb90dffb640 1 -- 172.21.15.89:0/172846044 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb914012750 con 0x7fb9180ca0e0 2023-10-30T12:50:25.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.071+0000 7fb90dffb640 1 -- 172.21.15.89:0/172846044 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb914012ae0 con 0x7fb9180ca0e0 2023-10-30T12:50:25.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.071+0000 7fb91f691640 1 -- 172.21.15.89:0/172846044 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9181e3430 con 0x7fb9180ca0e0 2023-10-30T12:50:25.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.071+0000 7fb91f691640 1 -- 172.21.15.89:0/172846044 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9181e3db0 con 0x7fb9180ca0e0 2023-10-30T12:50:25.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.071+0000 7fb91f691640 1 -- 172.21.15.89:0/172846044 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9181e49c0 con 0x7fb9180ca0e0 2023-10-30T12:50:25.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.075+0000 7fb90dffb640 1 -- 172.21.15.89:0/172846044 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb914012c40 con 0x7fb9180ca0e0 2023-10-30T12:50:25.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.075+0000 7fb90dffb640 1 --2- 172.21.15.89:0/172846044 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb8fc0664a0 0x7fb8fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:25.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.075+0000 7fb90dffb640 1 -- 172.21.15.89:0/172846044 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6664+0+0 (secure 0 0 0) 0x7fb914084780 con 0x7fb9180ca0e0 2023-10-30T12:50:25.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.075+0000 7fb91cc2c640 1 --2- 172.21.15.89:0/172846044 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb8fc0664a0 0x7fb8fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:25.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.075+0000 7fb91cc2c640 1 --2- 172.21.15.89:0/172846044 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb8fc0664a0 0x7fb8fc068960 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb91815a130 tx=0x7fb908007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:25.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.079+0000 7fb90dffb640 1 -- 172.21.15.89:0/172846044 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb9140517a0 con 0x7fb9180ca0e0 2023-10-30T12:50:25.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.263+0000 7fb91f691640 1 -- 172.21.15.89:0/172846044 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9180b2a50 con 0x7fb9180ca0e0 2023-10-30T12:50:25.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.263+0000 7fb90dffb640 1 -- 172.21.15.89:0/172846044 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+6544 (secure 0 0 0) 0x7fb914007380 con 0x7fb9180ca0e0 2023-10-30T12:50:25.264 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:50:25.265 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":109,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6271,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/659375964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":659375964},{"type":"v1","addr":"172.21.15.192:6835","nonce":659375964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:50:20.403164+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5678,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5678":{"gid":5678,"name":"f","rank":0,"incarnation":98,"state":"up:active","state_seq":14,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:50:25.265 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 109 2023-10-30T12:50:25.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.263+0000 7fb91f691640 1 -- 172.21.15.89:0/172846044 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb8fc0664a0 msgr2=0x7fb8fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:25.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.263+0000 7fb91f691640 1 --2- 172.21.15.89:0/172846044 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb8fc0664a0 0x7fb8fc068960 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb91815a130 tx=0x7fb908007d50 comp rx=0 tx=0).stop 2023-10-30T12:50:25.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.263+0000 7fb91f691640 1 -- 172.21.15.89:0/172846044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9180ca0e0 msgr2=0x7fb91815a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:25.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.263+0000 7fb91f691640 1 --2- 172.21.15.89:0/172846044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9180ca0e0 0x7fb91815a8b0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fb91400d5f0 tx=0x7fb91400dab0 comp rx=0 tx=0).stop 2023-10-30T12:50:25.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.263+0000 7fb91f691640 1 -- 172.21.15.89:0/172846044 shutdown_connections 2023-10-30T12:50:25.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.263+0000 7fb91f691640 1 --2- 172.21.15.89:0/172846044 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb8fc0664a0 0x7fb8fc068960 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:25.267 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.263+0000 7fb91f691640 1 --2- 172.21.15.89:0/172846044 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9181538a0 0x7fb91815b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:25.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.263+0000 7fb91f691640 1 --2- 172.21.15.89:0/172846044 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9180cf160 0x7fb91815b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:25.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.263+0000 7fb91f691640 1 --2- 172.21.15.89:0/172846044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb9180ca0e0 0x7fb91815a8b0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:25.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.263+0000 7fb91f691640 1 -- 172.21.15.89:0/172846044 >> 172.21.15.89:0/172846044 conn(0x7fb91806bfe0 msgr2=0x7fb9180c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:25.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.263+0000 7fb91f691640 1 -- 172.21.15.89:0/172846044 shutdown_connections 2023-10-30T12:50:25.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:25.263+0000 7fb91f691640 1 -- 172.21.15.89:0/172846044 wait complete. 2023-10-30T12:50:25.285 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:50:27.286 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:50:27.435 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.435+0000 7fc3b74c9640 1 Processor -- start 2023-10-30T12:50:27.435 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3b74c9640 1 -- start start 2023-10-30T12:50:27.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3b74c9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc3b00c6bd0 0x7fc3b006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:27.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3b74c9640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3b0066cf0 con 0x7fc3b006f7a0 2023-10-30T12:50:27.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3b74c9640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3b00c70d0 con 0x7fc3b0071110 2023-10-30T12:50:27.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3b74c9640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b00700c0 con 0x7fc3b00c6bd0 2023-10-30T12:50:27.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3b5265640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc3b006f7a0 0x7fc3b0069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:41128/0 (socket says 172.21.15.89:41128) 2023-10-30T12:50:27.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3b5265640 1 -- 172.21.15.89:0/149602302 learned_addr learned my addr 172.21.15.89:0/149602302 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:50:27.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3b4a64640 1 --2- 172.21.15.89:0/149602302 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc3b00c6bd0 0x7fc3b006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:27.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3a7fff640 1 -- 172.21.15.89:0/149602302 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4031715770 0 0) 0x7fc3b0066cf0 con 0x7fc3b006f7a0 2023-10-30T12:50:27.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3a7fff640 1 -- 172.21.15.89:0/149602302 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc39c003530 con 0x7fc3b006f7a0 2023-10-30T12:50:27.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3a7fff640 1 -- 172.21.15.89:0/149602302 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3483183970 0 0) 0x7fc39c003530 con 0x7fc3b006f7a0 2023-10-30T12:50:27.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3a7fff640 1 -- 172.21.15.89:0/149602302 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc3b00c6bd0 msgr2=0x7fc3b006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:50:27.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3a7fff640 1 --2- 172.21.15.89:0/149602302 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc3b00c6bd0 0x7fc3b006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:27.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3a7fff640 1 -- 172.21.15.89:0/149602302 >> v1:172.21.15.192:6789/0 conn(0x7fc3b0071110 legacy=0x7fc3b0158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:27.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3a7fff640 1 -- 172.21.15.89:0/149602302 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b00c7390 con 0x7fc3b006f7a0 2023-10-30T12:50:27.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3a7fff640 1 -- 172.21.15.89:0/149602302 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7fc3ac001680 con 0x7fc3b006f7a0 2023-10-30T12:50:27.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3a7fff640 1 -- 172.21.15.89:0/149602302 >> v1:172.21.15.89:6789/0 conn(0x7fc3b006f7a0 legacy=0x7fc3b0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:27.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3a7fff640 1 --2- 172.21.15.89:0/149602302 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc3b00c6bd0 0x7fc39c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:27.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3a7fff640 1 --2- 172.21.15.89:0/149602302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc39c003f30 0x7fc39c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:27.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3a7fff640 1 --2- 172.21.15.89:0/149602302 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc39c004d10 0x7fc39c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:27.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3a7fff640 1 -- 172.21.15.89:0/149602302 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b00700c0 con 0x7fc39c003f30 2023-10-30T12:50:27.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3a7fff640 1 -- 172.21.15.89:0/149602302 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc39c00bb00 con 0x7fc39c004d10 2023-10-30T12:50:27.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3a7fff640 1 -- 172.21.15.89:0/149602302 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc39c00bcc0 con 0x7fc3b00c6bd0 2023-10-30T12:50:27.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3b4a64640 1 --2- 172.21.15.89:0/149602302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc39c003f30 0x7fc39c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:27.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3b5265640 1 --2- 172.21.15.89:0/149602302 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc3b00c6bd0 0x7fc39c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:27.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3b5a66640 1 --2- 172.21.15.89:0/149602302 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc39c004d10 0x7fc39c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:27.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3b4a64640 1 -- 172.21.15.89:0/149602302 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc3b00c6bd0 msgr2=0x7fc39c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:50:27.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3b4a64640 1 --2- 172.21.15.89:0/149602302 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc3b00c6bd0 0x7fc39c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:27.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3b4a64640 1 -- 172.21.15.89:0/149602302 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc39c004d10 msgr2=0x7fc39c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:50:27.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3b4a64640 1 --2- 172.21.15.89:0/149602302 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc39c004d10 0x7fc39c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:27.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.439+0000 7fc3b4a64640 1 -- 172.21.15.89:0/149602302 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc39c003530 con 0x7fc39c003f30 2023-10-30T12:50:27.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.443+0000 7fc3b4a64640 1 --2- 172.21.15.89:0/149602302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc39c003f30 0x7fc39c0047d0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0009ae0 tx=0x7fc3a001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d41a0742a146f40 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:27.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.443+0000 7fc3a7fff640 1 -- 172.21.15.89:0/149602302 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a00a5640 con 0x7fc39c003f30 2023-10-30T12:50:27.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.443+0000 7fc3a7fff640 1 -- 172.21.15.89:0/149602302 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3a00a57a0 con 0x7fc39c003f30 2023-10-30T12:50:27.447 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.443+0000 7fc3a7fff640 1 -- 172.21.15.89:0/149602302 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a00a5b50 con 0x7fc39c003f30 2023-10-30T12:50:27.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.443+0000 7fc3b74c9640 1 -- 172.21.15.89:0/149602302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc39c003f30 msgr2=0x7fc39c0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:27.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.443+0000 7fc3b74c9640 1 --2- 172.21.15.89:0/149602302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc39c003f30 0x7fc39c0047d0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0009ae0 tx=0x7fc3a001ca50 comp rx=0 tx=0).stop 2023-10-30T12:50:27.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.443+0000 7fc3b5a66640 1 -- 172.21.15.89:0/149602302 reap_dead start 2023-10-30T12:50:27.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.443+0000 7fc3b74c9640 1 -- 172.21.15.89:0/149602302 shutdown_connections 2023-10-30T12:50:27.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.443+0000 7fc3b74c9640 1 -- 172.21.15.89:0/149602302 >> 172.21.15.89:0/149602302 conn(0x7fc3b00c7910 msgr2=0x7fc3b00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:27.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.443+0000 7fc3b74c9640 1 -- 172.21.15.89:0/149602302 shutdown_connections 2023-10-30T12:50:27.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.443+0000 7fc3b74c9640 1 -- 172.21.15.89:0/149602302 wait complete. 2023-10-30T12:50:27.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.447+0000 7fc3b74c9640 1 Processor -- start 2023-10-30T12:50:27.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.447+0000 7fc3b74c9640 1 -- start start 2023-10-30T12:50:27.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.447+0000 7fc3b74c9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc39c003f30 0x7fc3b00c1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:27.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.447+0000 7fc3b74c9640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3b00ad8d0 0x7fc3b00c5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:27.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.447+0000 7fc3b74c9640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc3b0149730 0x7fc3b0149bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:27.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.447+0000 7fc3b74c9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b00afff0 con 0x7fc39c003f30 2023-10-30T12:50:27.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.447+0000 7fc3b74c9640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b014a0f0 con 0x7fc3b00ad8d0 2023-10-30T12:50:27.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.447+0000 7fc3b74c9640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b01ef610 con 0x7fc3b0149730 2023-10-30T12:50:27.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.447+0000 7fc3b5265640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc39c003f30 0x7fc3b00c1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:27.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.447+0000 7fc3b5265640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc39c003f30 0x7fc3b00c1690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59638/0 (socket says 172.21.15.89:59638) 2023-10-30T12:50:27.454 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.447+0000 7fc3b5265640 1 -- 172.21.15.89:0/1319142949 learned_addr learned my addr 172.21.15.89:0/1319142949 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:50:27.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.447+0000 7fc3b4a64640 1 --2- 172.21.15.89:0/1319142949 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3b00ad8d0 0x7fc3b00c5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:27.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.447+0000 7fc3b5a66640 1 --2- 172.21.15.89:0/1319142949 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc3b0149730 0x7fc3b0149bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:27.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.447+0000 7fc3b5265640 1 -- 172.21.15.89:0/1319142949 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc3b0149730 msgr2=0x7fc3b0149bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:27.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.447+0000 7fc3b5265640 1 --2- 172.21.15.89:0/1319142949 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc3b0149730 0x7fc3b0149bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:27.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.447+0000 7fc3b5265640 1 -- 172.21.15.89:0/1319142949 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3b00ad8d0 msgr2=0x7fc3b00c5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:27.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.447+0000 7fc3b5265640 1 --2- 172.21.15.89:0/1319142949 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3b00ad8d0 0x7fc3b00c5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:27.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.447+0000 7fc3b5265640 1 -- 172.21.15.89:0/1319142949 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b00c2950 con 0x7fc39c003f30 2023-10-30T12:50:27.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.447+0000 7fc3b5265640 1 --2- 172.21.15.89:0/1319142949 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc39c003f30 0x7fc3b00c1690 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac007320 tx=0x7fc3ac007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:27.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.447+0000 7fc3a5ffb640 1 -- 172.21.15.89:0/1319142949 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac00fa00 con 0x7fc39c003f30 2023-10-30T12:50:27.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.447+0000 7fc3a5ffb640 1 -- 172.21.15.89:0/1319142949 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3ac00fb80 con 0x7fc39c003f30 2023-10-30T12:50:27.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.447+0000 7fc3b74c9640 1 -- 172.21.15.89:0/1319142949 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3b01f2ce0 con 0x7fc39c003f30 2023-10-30T12:50:27.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.447+0000 7fc3a5ffb640 1 -- 172.21.15.89:0/1319142949 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac0133f0 con 0x7fc39c003f30 2023-10-30T12:50:27.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.447+0000 7fc3b74c9640 1 -- 172.21.15.89:0/1319142949 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3b01eefc0 con 0x7fc39c003f30 2023-10-30T12:50:27.457 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.451+0000 7fc3b74c9640 1 -- 172.21.15.89:0/1319142949 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3b004e170 con 0x7fc39c003f30 2023-10-30T12:50:27.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.451+0000 7fc3a5ffb640 1 -- 172.21.15.89:0/1319142949 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc3ac013550 con 0x7fc39c003f30 2023-10-30T12:50:27.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.451+0000 7fc3a5ffb640 1 --2- 172.21.15.89:0/1319142949 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc3940664a0 0x7fc394068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:27.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.451+0000 7fc3a5ffb640 1 -- 172.21.15.89:0/1319142949 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6664+0+0 (secure 0 0 0) 0x7fc3ac084600 con 0x7fc39c003f30 2023-10-30T12:50:27.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.451+0000 7fc3b4a64640 1 --2- 172.21.15.89:0/1319142949 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc3940664a0 0x7fc394068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:27.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.451+0000 7fc3b4a64640 1 --2- 172.21.15.89:0/1319142949 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc3940664a0 0x7fc394068960 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fc3a00af000 tx=0x7fc3a00046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:27.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.455+0000 7fc3a5ffb640 1 -- 172.21.15.89:0/1319142949 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc3ac051590 con 0x7fc39c003f30 2023-10-30T12:50:27.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.639+0000 7fc3b74c9640 1 -- 172.21.15.89:0/1319142949 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3b00ca7b0 con 0x7fc39c003f30 2023-10-30T12:50:27.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.639+0000 7fc3a5ffb640 1 -- 172.21.15.89:0/1319142949 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+6544 (secure 0 0 0) 0x7fc3ac054bb0 con 0x7fc39c003f30 2023-10-30T12:50:27.637 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:50:27.638 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":109,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6271,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/659375964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":659375964},{"type":"v1","addr":"172.21.15.192:6835","nonce":659375964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:50:20.403164+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5678,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5678":{"gid":5678,"name":"f","rank":0,"incarnation":98,"state":"up:active","state_seq":14,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:50:27.638 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 109 2023-10-30T12:50:27.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.639+0000 7fc3b74c9640 1 -- 172.21.15.89:0/1319142949 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc3940664a0 msgr2=0x7fc394068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:27.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.639+0000 7fc3b74c9640 1 --2- 172.21.15.89:0/1319142949 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc3940664a0 0x7fc394068960 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fc3a00af000 tx=0x7fc3a00046d0 comp rx=0 tx=0).stop 2023-10-30T12:50:27.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.639+0000 7fc3b74c9640 1 -- 172.21.15.89:0/1319142949 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc39c003f30 msgr2=0x7fc3b00c1690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:27.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.639+0000 7fc3b74c9640 1 --2- 172.21.15.89:0/1319142949 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc39c003f30 0x7fc3b00c1690 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac007320 tx=0x7fc3ac007720 comp rx=0 tx=0).stop 2023-10-30T12:50:27.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.643+0000 7fc3b74c9640 1 -- 172.21.15.89:0/1319142949 shutdown_connections 2023-10-30T12:50:27.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.643+0000 7fc3b74c9640 1 --2- 172.21.15.89:0/1319142949 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc3940664a0 0x7fc394068960 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:27.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.643+0000 7fc3b74c9640 1 --2- 172.21.15.89:0/1319142949 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3b00ad8d0 0x7fc3b00c5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:27.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.643+0000 7fc3b74c9640 1 --2- 172.21.15.89:0/1319142949 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc3b0149730 0x7fc3b0149bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:27.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.643+0000 7fc3b74c9640 1 --2- 172.21.15.89:0/1319142949 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc39c003f30 0x7fc3b00c1690 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:27.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.643+0000 7fc3b74c9640 1 -- 172.21.15.89:0/1319142949 >> 172.21.15.89:0/1319142949 conn(0x7fc3b00c7910 msgr2=0x7fc3b006a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:27.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.643+0000 7fc3b74c9640 1 -- 172.21.15.89:0/1319142949 shutdown_connections 2023-10-30T12:50:27.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:27.643+0000 7fc3b74c9640 1 -- 172.21.15.89:0/1319142949 wait complete. 2023-10-30T12:50:27.658 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:50:29.659 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:50:29.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.811+0000 7fc992fc5640 1 Processor -- start 2023-10-30T12:50:29.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.811+0000 7fc992fc5640 1 -- start start 2023-10-30T12:50:29.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.811+0000 7fc992fc5640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc98c1538a0 0x7fc98c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:29.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.811+0000 7fc992fc5640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc98c0cf160 0x7fc98c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:29.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.811+0000 7fc992fc5640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc98c0cb800 con 0x7fc98c0ca0e0 2023-10-30T12:50:29.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.811+0000 7fc992fc5640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc98c15dc30 con 0x7fc98c1538a0 2023-10-30T12:50:29.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.811+0000 7fc992fc5640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c0cb330 con 0x7fc98c0cf160 2023-10-30T12:50:29.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.811+0000 7fc990d61640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc98c1538a0 0x7fc98c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:29.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.811+0000 7fc991562640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc98c0cf160 0x7fc98c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:29.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.811+0000 7fc990d61640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc98c1538a0 0x7fc98c15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:48718/0 (socket says 172.21.15.89:48718) 2023-10-30T12:50:29.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.811+0000 7fc98bfff640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fc98c0ca0e0 0x7fc98c15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:36872/0 (socket says 172.21.15.89:36872) 2023-10-30T12:50:29.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.811+0000 7fc991562640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc98c0cf160 0x7fc98c0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:51238/0 (socket says 172.21.15.89:51238) 2023-10-30T12:50:29.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.811+0000 7fc990d61640 1 -- 172.21.15.89:0/3913385986 learned_addr learned my addr 172.21.15.89:0/3913385986 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:50:29.812 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.811+0000 7fc991562640 1 -- 172.21.15.89:0/3913385986 >> v2:172.21.15.192:3300/0 conn(0x7fc98c1538a0 msgr2=0x7fc98c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:50:29.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.811+0000 7fc991562640 1 --2- 172.21.15.89:0/3913385986 >> v2:172.21.15.192:3300/0 conn(0x7fc98c1538a0 0x7fc98c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:29.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.811+0000 7fc991562640 1 -- 172.21.15.89:0/3913385986 >> v1:172.21.15.192:6789/0 conn(0x7fc98c0ca0e0 legacy=0x7fc98c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:29.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.811+0000 7fc991562640 1 -- 172.21.15.89:0/3913385986 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc98c0cad70 con 0x7fc98c0cf160 2023-10-30T12:50:29.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.811+0000 7fc98b7fe640 1 -- 172.21.15.89:0/3913385986 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 995431944 0 0) 0x7fc98c0cb800 con 0x7fc98c0ca0e0 2023-10-30T12:50:29.813 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.811+0000 7fc990d61640 1 --2- 172.21.15.89:0/3913385986 >> v2:172.21.15.192:3300/0 conn(0x7fc98c1538a0 0x7fc98c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:50:29.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.815+0000 7fc991562640 1 --2- 172.21.15.89:0/3913385986 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc98c0cf160 0x7fc98c0cf590 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc984009d90 tx=0x7fc98401f130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52e9cd8e1bc670b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:29.814 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.815+0000 7fc98b7fe640 1 -- 172.21.15.89:0/3913385986 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9840af600 con 0x7fc98c0cf160 2023-10-30T12:50:29.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.815+0000 7fc98b7fe640 1 -- 172.21.15.89:0/3913385986 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9840ab050 con 0x7fc98c0cf160 2023-10-30T12:50:29.815 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.815+0000 7fc98b7fe640 1 -- 172.21.15.89:0/3913385986 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9840af980 con 0x7fc98c0cf160 2023-10-30T12:50:29.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.815+0000 7fc992fc5640 1 -- 172.21.15.89:0/3913385986 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc98c0cf160 msgr2=0x7fc98c0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:29.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.815+0000 7fc992fc5640 1 --2- 172.21.15.89:0/3913385986 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc98c0cf160 0x7fc98c0cf590 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc984009d90 tx=0x7fc98401f130 comp rx=0 tx=0).stop 2023-10-30T12:50:29.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.815+0000 7fc992fc5640 1 -- 172.21.15.89:0/3913385986 shutdown_connections 2023-10-30T12:50:29.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.815+0000 7fc992fc5640 1 --2- 172.21.15.89:0/3913385986 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc98c0cf160 0x7fc98c0cf590 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:29.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.815+0000 7fc992fc5640 1 --2- 172.21.15.89:0/3913385986 >> v2:172.21.15.192:3300/0 conn(0x7fc98c1538a0 0x7fc98c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:29.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.815+0000 7fc992fc5640 1 -- 172.21.15.89:0/3913385986 >> 172.21.15.89:0/3913385986 conn(0x7fc98c06bfe0 msgr2=0x7fc98c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:29.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.815+0000 7fc992fc5640 1 -- 172.21.15.89:0/3913385986 shutdown_connections 2023-10-30T12:50:29.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.819+0000 7fc992fc5640 1 -- 172.21.15.89:0/3913385986 wait complete. 2023-10-30T12:50:29.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.819+0000 7fc992fc5640 1 Processor -- start 2023-10-30T12:50:29.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.819+0000 7fc992fc5640 1 -- start start 2023-10-30T12:50:29.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.819+0000 7fc992fc5640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc98c0ca0e0 0x7fc98c1441f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:29.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.819+0000 7fc992fc5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc98c0cf160 0x7fc98c146a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:29.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.819+0000 7fc992fc5640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc98c1538a0 0x7fc98c0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:29.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.819+0000 7fc992fc5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c1485f0 con 0x7fc98c0cf160 2023-10-30T12:50:29.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.819+0000 7fc992fc5640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c1efa80 con 0x7fc98c0ca0e0 2023-10-30T12:50:29.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.819+0000 7fc992fc5640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c1effb0 con 0x7fc98c1538a0 2023-10-30T12:50:29.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.819+0000 7fc98bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc98c0cf160 0x7fc98c146a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:29.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.819+0000 7fc98bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc98c0cf160 0x7fc98c146a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56078/0 (socket says 172.21.15.89:56078) 2023-10-30T12:50:29.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.819+0000 7fc98bfff640 1 -- 172.21.15.89:0/2372645446 learned_addr learned my addr 172.21.15.89:0/2372645446 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:50:29.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.819+0000 7fc991562640 1 --2- 172.21.15.89:0/2372645446 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc98c1538a0 0x7fc98c0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:29.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.819+0000 7fc990d61640 1 --2- 172.21.15.89:0/2372645446 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc98c0ca0e0 0x7fc98c1441f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:29.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.819+0000 7fc98bfff640 1 -- 172.21.15.89:0/2372645446 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc98c1538a0 msgr2=0x7fc98c0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:29.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.819+0000 7fc98bfff640 1 --2- 172.21.15.89:0/2372645446 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc98c1538a0 0x7fc98c0ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:29.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.819+0000 7fc98bfff640 1 -- 172.21.15.89:0/2372645446 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc98c0ca0e0 msgr2=0x7fc98c1441f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:29.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.819+0000 7fc98bfff640 1 --2- 172.21.15.89:0/2372645446 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc98c0ca0e0 0x7fc98c1441f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:29.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.819+0000 7fc98bfff640 1 -- 172.21.15.89:0/2372645446 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9800094d0 con 0x7fc98c0cf160 2023-10-30T12:50:29.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.819+0000 7fc990d61640 1 --2- 172.21.15.89:0/2372645446 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc98c0ca0e0 0x7fc98c1441f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:50:29.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.819+0000 7fc991562640 1 --2- 172.21.15.89:0/2372645446 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc98c1538a0 0x7fc98c0ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:50:29.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.819+0000 7fc98bfff640 1 --2- 172.21.15.89:0/2372645446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc98c0cf160 0x7fc98c146a50 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fc97800c520 tx=0x7fc97800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:29.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.823+0000 7fc9897fa640 1 -- 172.21.15.89:0/2372645446 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9780074e0 con 0x7fc98c0cf160 2023-10-30T12:50:29.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.823+0000 7fc9897fa640 1 -- 172.21.15.89:0/2372645446 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc978007640 con 0x7fc98c0cf160 2023-10-30T12:50:29.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.823+0000 7fc992fc5640 1 -- 172.21.15.89:0/2372645446 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc984009a70 con 0x7fc98c0cf160 2023-10-30T12:50:29.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.823+0000 7fc9897fa640 1 -- 172.21.15.89:0/2372645446 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9780049f0 con 0x7fc98c0cf160 2023-10-30T12:50:29.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.823+0000 7fc992fc5640 1 -- 172.21.15.89:0/2372645446 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc98c144bf0 con 0x7fc98c0cf160 2023-10-30T12:50:29.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.823+0000 7fc992fc5640 1 -- 172.21.15.89:0/2372645446 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc98c0b2a50 con 0x7fc98c0cf160 2023-10-30T12:50:29.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.823+0000 7fc9897fa640 1 -- 172.21.15.89:0/2372645446 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc978004b50 con 0x7fc98c0cf160 2023-10-30T12:50:29.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.823+0000 7fc9897fa640 1 --2- 172.21.15.89:0/2372645446 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc9700664d0 0x7fc970068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:29.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.823+0000 7fc9897fa640 1 -- 172.21.15.89:0/2372645446 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6664+0+0 (secure 0 0 0) 0x7fc978083810 con 0x7fc98c0cf160 2023-10-30T12:50:29.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.823+0000 7fc990d61640 1 --2- 172.21.15.89:0/2372645446 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc9700664d0 0x7fc970068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:29.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.827+0000 7fc990d61640 1 --2- 172.21.15.89:0/2372645446 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc9700664d0 0x7fc970068990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fc980009c90 tx=0x7fc980008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:29.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:29.827+0000 7fc9897fa640 1 -- 172.21.15.89:0/2372645446 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc9780508c0 con 0x7fc98c0cf160 2023-10-30T12:50:30.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:30.011+0000 7fc992fc5640 1 -- 172.21.15.89:0/2372645446 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc98c144d80 con 0x7fc98c0cf160 2023-10-30T12:50:30.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:30.011+0000 7fc9897fa640 1 -- 172.21.15.89:0/2372645446 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+6544 (secure 0 0 0) 0x7fc978053ee0 con 0x7fc98c0cf160 2023-10-30T12:50:30.010 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:50:30.010 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":109,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6271,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/659375964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":659375964},{"type":"v1","addr":"172.21.15.192:6835","nonce":659375964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:50:20.403164+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5678,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5678":{"gid":5678,"name":"f","rank":0,"incarnation":98,"state":"up:active","state_seq":14,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:50:30.011 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 109 2023-10-30T12:50:30.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:30.011+0000 7fc992fc5640 1 -- 172.21.15.89:0/2372645446 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc9700664d0 msgr2=0x7fc970068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:30.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:30.011+0000 7fc992fc5640 1 --2- 172.21.15.89:0/2372645446 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc9700664d0 0x7fc970068990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fc980009c90 tx=0x7fc980008040 comp rx=0 tx=0).stop 2023-10-30T12:50:30.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:30.011+0000 7fc992fc5640 1 -- 172.21.15.89:0/2372645446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc98c0cf160 msgr2=0x7fc98c146a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:30.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:30.011+0000 7fc992fc5640 1 --2- 172.21.15.89:0/2372645446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc98c0cf160 0x7fc98c146a50 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fc97800c520 tx=0x7fc97800c9e0 comp rx=0 tx=0).stop 2023-10-30T12:50:30.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:30.011+0000 7fc992fc5640 1 -- 172.21.15.89:0/2372645446 shutdown_connections 2023-10-30T12:50:30.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:30.011+0000 7fc992fc5640 1 --2- 172.21.15.89:0/2372645446 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc9700664d0 0x7fc970068990 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:30.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:30.011+0000 7fc992fc5640 1 --2- 172.21.15.89:0/2372645446 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc98c1538a0 0x7fc98c0ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:30.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:30.011+0000 7fc992fc5640 1 --2- 172.21.15.89:0/2372645446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc98c0cf160 0x7fc98c146a50 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:30.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:30.011+0000 7fc992fc5640 1 --2- 172.21.15.89:0/2372645446 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc98c0ca0e0 0x7fc98c1441f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:30.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:30.011+0000 7fc992fc5640 1 -- 172.21.15.89:0/2372645446 >> 172.21.15.89:0/2372645446 conn(0x7fc98c06bfe0 msgr2=0x7fc98c06c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:30.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:30.011+0000 7fc992fc5640 1 -- 172.21.15.89:0/2372645446 shutdown_connections 2023-10-30T12:50:30.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:30.011+0000 7fc992fc5640 1 -- 172.21.15.89:0/2372645446 wait complete. 2023-10-30T12:50:30.030 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:50:32.031 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:50:32.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.183+0000 7ffabd165640 1 Processor -- start 2023-10-30T12:50:32.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.183+0000 7ffabd165640 1 -- start start 2023-10-30T12:50:32.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.183+0000 7ffabd165640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffab81538a0 0x7ffab815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:32.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.183+0000 7ffabd165640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ffab80cf160 0x7ffab80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:32.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.183+0000 7ffabd165640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab80cb800 con 0x7ffab80ca0e0 2023-10-30T12:50:32.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.183+0000 7ffabd165640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffab815dc30 con 0x7ffab81538a0 2023-10-30T12:50:32.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.183+0000 7ffabd165640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffab80cb330 con 0x7ffab80cf160 2023-10-30T12:50:32.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.183+0000 7ffab6d9d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffab81538a0 0x7ffab815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:32.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.183+0000 7ffab6d9d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ffab81538a0 0x7ffab815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56094/0 (socket says 172.21.15.89:56094) 2023-10-30T12:50:32.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.183+0000 7ffab6d9d640 1 -- 172.21.15.89:0/860734791 learned_addr learned my addr 172.21.15.89:0/860734791 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:50:32.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.183+0000 7ffab759e640 1 --2- 172.21.15.89:0/860734791 >> v2:172.21.15.192:3300/0 conn(0x7ffab80cf160 0x7ffab80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:32.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.183+0000 7ffab6d9d640 1 -- 172.21.15.89:0/860734791 >> v2:172.21.15.192:3300/0 conn(0x7ffab80cf160 msgr2=0x7ffab80cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:50:32.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.183+0000 7ffab6d9d640 1 --2- 172.21.15.89:0/860734791 >> v2:172.21.15.192:3300/0 conn(0x7ffab80cf160 0x7ffab80cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:32.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.183+0000 7ffab6d9d640 1 -- 172.21.15.89:0/860734791 >> v1:172.21.15.192:6789/0 conn(0x7ffab80ca0e0 legacy=0x7ffab815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:50:32.184 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.183+0000 7ffab6d9d640 1 -- 172.21.15.89:0/860734791 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab80cad70 con 0x7ffab81538a0 2023-10-30T12:50:32.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.187+0000 7ffab6d9d640 1 --2- 172.21.15.89:0/860734791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffab81538a0 0x7ffab815d6f0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7ffab80b5330 tx=0x7ffaa801ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92e6234cba98fd7c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:32.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.187+0000 7ffab5d9b640 1 -- 172.21.15.89:0/860734791 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa80a89b0 con 0x7ffab81538a0 2023-10-30T12:50:32.185 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.187+0000 7ffab5d9b640 1 -- 172.21.15.89:0/860734791 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaa80a8b10 con 0x7ffab81538a0 2023-10-30T12:50:32.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.187+0000 7ffab5d9b640 1 -- 172.21.15.89:0/860734791 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa80a8ea0 con 0x7ffab81538a0 2023-10-30T12:50:32.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.187+0000 7ffabd165640 1 -- 172.21.15.89:0/860734791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffab81538a0 msgr2=0x7ffab815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:32.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.187+0000 7ffabd165640 1 --2- 172.21.15.89:0/860734791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffab81538a0 0x7ffab815d6f0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7ffab80b5330 tx=0x7ffaa801ecb0 comp rx=0 tx=0).stop 2023-10-30T12:50:32.186 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.187+0000 7ffabd165640 1 -- 172.21.15.89:0/860734791 shutdown_connections 2023-10-30T12:50:32.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.187+0000 7ffabd165640 1 --2- 172.21.15.89:0/860734791 >> v2:172.21.15.192:3300/0 conn(0x7ffab80cf160 0x7ffab80cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:32.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.187+0000 7ffabd165640 1 --2- 172.21.15.89:0/860734791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffab81538a0 0x7ffab815d6f0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:32.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.187+0000 7ffabd165640 1 -- 172.21.15.89:0/860734791 >> 172.21.15.89:0/860734791 conn(0x7ffab806bfe0 msgr2=0x7ffab806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:32.187 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.187+0000 7ffabd165640 1 -- 172.21.15.89:0/860734791 shutdown_connections 2023-10-30T12:50:32.188 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.187+0000 7ffabd165640 1 -- 172.21.15.89:0/860734791 wait complete. 2023-10-30T12:50:32.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.191+0000 7ffabd165640 1 Processor -- start 2023-10-30T12:50:32.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.191+0000 7ffabd165640 1 -- start start 2023-10-30T12:50:32.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.191+0000 7ffabd165640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffab80ca0e0 0x7ffab80c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:32.190 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.191+0000 7ffabd165640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffab80cf160 0x7ffab8157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:32.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.191+0000 7ffabd165640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffab81538a0 0x7ffab81e5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:32.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.191+0000 7ffabd165640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab81ef4c0 con 0x7ffab80cf160 2023-10-30T12:50:32.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.191+0000 7ffabd165640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab80c9180 con 0x7ffab81538a0 2023-10-30T12:50:32.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.191+0000 7ffabd165640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffab81ef730 con 0x7ffab80ca0e0 2023-10-30T12:50:32.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.191+0000 7ffab659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffab80cf160 0x7ffab8157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:32.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.191+0000 7ffab659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffab80cf160 0x7ffab8157560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56100/0 (socket says 172.21.15.89:56100) 2023-10-30T12:50:32.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.191+0000 7ffab659c640 1 -- 172.21.15.89:0/1597619944 learned_addr learned my addr 172.21.15.89:0/1597619944 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:50:32.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.191+0000 7ffab759e640 1 --2- 172.21.15.89:0/1597619944 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffab81538a0 0x7ffab81e5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:32.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.191+0000 7ffab6d9d640 1 --2- 172.21.15.89:0/1597619944 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffab80ca0e0 0x7ffab80c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:32.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.191+0000 7ffab659c640 1 -- 172.21.15.89:0/1597619944 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffab80ca0e0 msgr2=0x7ffab80c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:32.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.191+0000 7ffab659c640 1 --2- 172.21.15.89:0/1597619944 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffab80ca0e0 0x7ffab80c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:32.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.191+0000 7ffab659c640 1 -- 172.21.15.89:0/1597619944 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffab81538a0 msgr2=0x7ffab81e5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:32.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.191+0000 7ffab659c640 1 --2- 172.21.15.89:0/1597619944 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffab81538a0 0x7ffab81e5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:32.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.191+0000 7ffab659c640 1 -- 172.21.15.89:0/1597619944 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa80094d0 con 0x7ffab80cf160 2023-10-30T12:50:32.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.191+0000 7ffab659c640 1 --2- 172.21.15.89:0/1597619944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffab80cf160 0x7ffab8157560 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7ffaac00b410 tx=0x7ffaac00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:32.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.191+0000 7ffa9f7fe640 1 -- 172.21.15.89:0/1597619944 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaac004020 con 0x7ffab80cf160 2023-10-30T12:50:32.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.191+0000 7ffa9f7fe640 1 -- 172.21.15.89:0/1597619944 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaac004180 con 0x7ffab80cf160 2023-10-30T12:50:32.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.191+0000 7ffabd165640 1 -- 172.21.15.89:0/1597619944 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffab81efa10 con 0x7ffab80cf160 2023-10-30T12:50:32.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.191+0000 7ffa9f7fe640 1 -- 172.21.15.89:0/1597619944 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaac011630 con 0x7ffab80cf160 2023-10-30T12:50:32.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.191+0000 7ffabd165640 1 -- 172.21.15.89:0/1597619944 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffab80c9460 con 0x7ffab80cf160 2023-10-30T12:50:32.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.191+0000 7ffabd165640 1 -- 172.21.15.89:0/1597619944 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffab80b2a50 con 0x7ffab80cf160 2023-10-30T12:50:32.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.195+0000 7ffa9f7fe640 1 -- 172.21.15.89:0/1597619944 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffaac011790 con 0x7ffab80cf160 2023-10-30T12:50:32.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.195+0000 7ffa9f7fe640 1 --2- 172.21.15.89:0/1597619944 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ffaa40664a0 0x7ffaa4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:32.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.195+0000 7ffa9f7fe640 1 -- 172.21.15.89:0/1597619944 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6664+0+0 (secure 0 0 0) 0x7ffaac083f20 con 0x7ffab80cf160 2023-10-30T12:50:32.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.195+0000 7ffab6d9d640 1 --2- 172.21.15.89:0/1597619944 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ffaa40664a0 0x7ffaa4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:32.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.195+0000 7ffab6d9d640 1 --2- 172.21.15.89:0/1597619944 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ffaa40664a0 0x7ffaa4068960 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ffaa80b2000 tx=0x7ffaa8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:32.196 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.199+0000 7ffa9f7fe640 1 -- 172.21.15.89:0/1597619944 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffaac051000 con 0x7ffab80cf160 2023-10-30T12:50:32.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.383+0000 7ffabd165640 1 -- 172.21.15.89:0/1597619944 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffab81eff30 con 0x7ffab80cf160 2023-10-30T12:50:32.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.383+0000 7ffa9f7fe640 1 -- 172.21.15.89:0/1597619944 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+6544 (secure 0 0 0) 0x7ffab81eff30 con 0x7ffab80cf160 2023-10-30T12:50:32.380 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:50:32.381 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":109,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6271,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/659375964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":659375964},{"type":"v1","addr":"172.21.15.192:6835","nonce":659375964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:50:20.403164+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5678,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5678":{"gid":5678,"name":"f","rank":0,"incarnation":98,"state":"up:active","state_seq":14,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:50:32.381 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 109 2023-10-30T12:50:32.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.383+0000 7ffabd165640 1 -- 172.21.15.89:0/1597619944 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ffaa40664a0 msgr2=0x7ffaa4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:32.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.383+0000 7ffabd165640 1 --2- 172.21.15.89:0/1597619944 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ffaa40664a0 0x7ffaa4068960 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ffaa80b2000 tx=0x7ffaa8009040 comp rx=0 tx=0).stop 2023-10-30T12:50:32.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.383+0000 7ffabd165640 1 -- 172.21.15.89:0/1597619944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffab80cf160 msgr2=0x7ffab8157560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:32.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.383+0000 7ffabd165640 1 --2- 172.21.15.89:0/1597619944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffab80cf160 0x7ffab8157560 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7ffaac00b410 tx=0x7ffaac00b8d0 comp rx=0 tx=0).stop 2023-10-30T12:50:32.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.383+0000 7ffabd165640 1 -- 172.21.15.89:0/1597619944 shutdown_connections 2023-10-30T12:50:32.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.383+0000 7ffabd165640 1 --2- 172.21.15.89:0/1597619944 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ffaa40664a0 0x7ffaa4068960 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:32.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.383+0000 7ffabd165640 1 --2- 172.21.15.89:0/1597619944 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffab81538a0 0x7ffab81e5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:32.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.383+0000 7ffabd165640 1 --2- 172.21.15.89:0/1597619944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffab80cf160 0x7ffab8157560 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:32.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.383+0000 7ffabd165640 1 --2- 172.21.15.89:0/1597619944 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffab80ca0e0 0x7ffab80c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:32.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.383+0000 7ffabd165640 1 -- 172.21.15.89:0/1597619944 >> 172.21.15.89:0/1597619944 conn(0x7ffab806bfe0 msgr2=0x7ffab81519e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:32.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.383+0000 7ffabd165640 1 -- 172.21.15.89:0/1597619944 shutdown_connections 2023-10-30T12:50:32.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:32.383+0000 7ffabd165640 1 -- 172.21.15.89:0/1597619944 wait complete. 2023-10-30T12:50:32.402 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:50:34.403 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:50:34.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.559+0000 7f3b085a1640 1 Processor -- start 2023-10-30T12:50:34.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.559+0000 7f3b085a1640 1 -- start start 2023-10-30T12:50:34.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.559+0000 7f3b085a1640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3b0006f7a0 0x7f3b0006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:34.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.559+0000 7f3b085a1640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b000c6bd0 0x7f3b00069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:34.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.559+0000 7f3b085a1640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3b00071110 0x7f3b00158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:34.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.559+0000 7f3b085a1640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b000700c0 con 0x7f3b0006f7a0 2023-10-30T12:50:34.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.559+0000 7f3b085a1640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b0006a3d0 con 0x7f3b00071110 2023-10-30T12:50:34.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.559+0000 7f3b085a1640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b00066cf0 con 0x7f3b000c6bd0 2023-10-30T12:50:34.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.559+0000 7f3b0633d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3b0006f7a0 0x7f3b0006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:34.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.559+0000 7f3b0633d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3b0006f7a0 0x7f3b0006fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56112/0 (socket says 172.21.15.89:56112) 2023-10-30T12:50:34.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.559+0000 7f3b0633d640 1 -- 172.21.15.89:0/532108375 learned_addr learned my addr 172.21.15.89:0/532108375 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:50:34.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.559+0000 7f3b05b3c640 1 --2- 172.21.15.89:0/532108375 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b000c6bd0 0x7f3b00069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:34.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.559+0000 7f3b06b3e640 1 --2- 172.21.15.89:0/532108375 >> v2:172.21.15.192:3300/0 conn(0x7f3b00071110 0x7f3b00158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:34.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.559+0000 7f3b0633d640 1 -- 172.21.15.89:0/532108375 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b000c6bd0 msgr2=0x7f3b00069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:50:34.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.559+0000 7f3b0633d640 1 --2- 172.21.15.89:0/532108375 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b000c6bd0 0x7f3b00069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:34.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.559+0000 7f3b0633d640 1 -- 172.21.15.89:0/532108375 >> v2:172.21.15.192:3300/0 conn(0x7f3b00071110 msgr2=0x7f3b00158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:50:34.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.559+0000 7f3b0633d640 1 --2- 172.21.15.89:0/532108375 >> v2:172.21.15.192:3300/0 conn(0x7f3b00071110 0x7f3b00158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:34.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.559+0000 7f3b0633d640 1 -- 172.21.15.89:0/532108375 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b000c70d0 con 0x7f3b0006f7a0 2023-10-30T12:50:34.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.563+0000 7f3b0633d640 1 --2- 172.21.15.89:0/532108375 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b0006f7a0 0x7f3b0006fb80 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f3b0006c7d0 tx=0x7f3afc01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1347a1bb3a780821 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:34.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.563+0000 7f3b0533b640 1 -- 172.21.15.89:0/532108375 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc0a89b0 con 0x7f3b0006f7a0 2023-10-30T12:50:34.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.563+0000 7f3b0533b640 1 -- 172.21.15.89:0/532108375 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3afc0a8b10 con 0x7f3b0006f7a0 2023-10-30T12:50:34.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.563+0000 7f3b0533b640 1 -- 172.21.15.89:0/532108375 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc0a8ea0 con 0x7f3b0006f7a0 2023-10-30T12:50:34.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.563+0000 7f3b085a1640 1 -- 172.21.15.89:0/532108375 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b0006f7a0 msgr2=0x7f3b0006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:34.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.563+0000 7f3b085a1640 1 --2- 172.21.15.89:0/532108375 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b0006f7a0 0x7f3b0006fb80 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f3b0006c7d0 tx=0x7f3afc01ecf0 comp rx=0 tx=0).stop 2023-10-30T12:50:34.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.563+0000 7f3b085a1640 1 -- 172.21.15.89:0/532108375 shutdown_connections 2023-10-30T12:50:34.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.563+0000 7f3b085a1640 1 --2- 172.21.15.89:0/532108375 >> v2:172.21.15.192:3300/0 conn(0x7f3b00071110 0x7f3b00158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:34.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.563+0000 7f3b085a1640 1 --2- 172.21.15.89:0/532108375 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b000c6bd0 0x7f3b00069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:34.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.563+0000 7f3b085a1640 1 --2- 172.21.15.89:0/532108375 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b0006f7a0 0x7f3b0006fb80 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:34.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.563+0000 7f3b085a1640 1 -- 172.21.15.89:0/532108375 >> 172.21.15.89:0/532108375 conn(0x7f3b000c7910 msgr2=0x7f3b000c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:34.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.563+0000 7f3b085a1640 1 -- 172.21.15.89:0/532108375 shutdown_connections 2023-10-30T12:50:34.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.563+0000 7f3b085a1640 1 -- 172.21.15.89:0/532108375 wait complete. 2023-10-30T12:50:34.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.567+0000 7f3b085a1640 1 Processor -- start 2023-10-30T12:50:34.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.567+0000 7f3b085a1640 1 -- start start 2023-10-30T12:50:34.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.567+0000 7f3b085a1640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b0006f7a0 0x7f3b00149be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:34.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.567+0000 7f3b085a1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b00071110 0x7f3b000c1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:34.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.567+0000 7f3b085a1640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b000c6bd0 0x7f3b000c18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:34.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.567+0000 7f3b085a1640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b000c0f50 con 0x7f3b00071110 2023-10-30T12:50:34.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.567+0000 7f3b085a1640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0014a120 con 0x7f3b000c6bd0 2023-10-30T12:50:34.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.567+0000 7f3b085a1640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b000af3c0 con 0x7f3b0006f7a0 2023-10-30T12:50:34.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.567+0000 7f3b05b3c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b00071110 0x7f3b000c1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:34.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.567+0000 7f3b05b3c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b00071110 0x7f3b000c1370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56118/0 (socket says 172.21.15.89:56118) 2023-10-30T12:50:34.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.567+0000 7f3b05b3c640 1 -- 172.21.15.89:0/2913113007 learned_addr learned my addr 172.21.15.89:0/2913113007 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:50:34.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.567+0000 7f3b0633d640 1 --2- 172.21.15.89:0/2913113007 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b0006f7a0 0x7f3b00149be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:34.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.567+0000 7f3b06b3e640 1 --2- 172.21.15.89:0/2913113007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b000c6bd0 0x7f3b000c18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:34.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.567+0000 7f3b05b3c640 1 -- 172.21.15.89:0/2913113007 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b0006f7a0 msgr2=0x7f3b00149be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:34.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.567+0000 7f3b05b3c640 1 --2- 172.21.15.89:0/2913113007 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b0006f7a0 0x7f3b00149be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:34.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.567+0000 7f3b05b3c640 1 -- 172.21.15.89:0/2913113007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b000c6bd0 msgr2=0x7f3b000c18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:34.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.567+0000 7f3b05b3c640 1 --2- 172.21.15.89:0/2913113007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b000c6bd0 0x7f3b000c18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:34.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.567+0000 7f3b05b3c640 1 -- 172.21.15.89:0/2913113007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3afc0094d0 con 0x7f3b00071110 2023-10-30T12:50:34.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.567+0000 7f3b05b3c640 1 --2- 172.21.15.89:0/2913113007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b00071110 0x7f3b000c1370 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f3af000d620 tx=0x7f3af000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:34.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.567+0000 7f3af6ffd640 1 -- 172.21.15.89:0/2913113007 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af000b620 con 0x7f3b00071110 2023-10-30T12:50:34.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.567+0000 7f3af6ffd640 1 -- 172.21.15.89:0/2913113007 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af000b780 con 0x7f3b00071110 2023-10-30T12:50:34.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.567+0000 7f3b085a1640 1 -- 172.21.15.89:0/2913113007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b000aee00 con 0x7f3b00071110 2023-10-30T12:50:34.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.567+0000 7f3af6ffd640 1 -- 172.21.15.89:0/2913113007 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af000bb10 con 0x7f3b00071110 2023-10-30T12:50:34.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.567+0000 7f3b085a1640 1 -- 172.21.15.89:0/2913113007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b000c0890 con 0x7f3b00071110 2023-10-30T12:50:34.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.571+0000 7f3b085a1640 1 -- 172.21.15.89:0/2913113007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b000c06f0 con 0x7f3b00071110 2023-10-30T12:50:34.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.571+0000 7f3af6ffd640 1 -- 172.21.15.89:0/2913113007 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3af000bc70 con 0x7f3b00071110 2023-10-30T12:50:34.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.571+0000 7f3af6ffd640 1 --2- 172.21.15.89:0/2913113007 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3ae40664a0 0x7f3ae4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:34.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.571+0000 7f3af6ffd640 1 -- 172.21.15.89:0/2913113007 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6664+0+0 (secure 0 0 0) 0x7f3af0084680 con 0x7f3b00071110 2023-10-30T12:50:34.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.571+0000 7f3b0633d640 1 --2- 172.21.15.89:0/2913113007 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3ae40664a0 0x7f3ae4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:34.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.571+0000 7f3b0633d640 1 --2- 172.21.15.89:0/2913113007 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3ae40664a0 0x7f3ae4068960 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f3afc0b2000 tx=0x7f3afc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:34.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.575+0000 7f3af6ffd640 1 -- 172.21.15.89:0/2913113007 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3b000c06f0 con 0x7f3b00071110 2023-10-30T12:50:34.755 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.759+0000 7f3b085a1640 1 -- 172.21.15.89:0/2913113007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b000c1090 con 0x7f3b00071110 2023-10-30T12:50:34.756 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.759+0000 7f3af6ffd640 1 -- 172.21.15.89:0/2913113007 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+6544 (secure 0 0 0) 0x7f3b000c1090 con 0x7f3b00071110 2023-10-30T12:50:34.756 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:50:34.756 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":109,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6271,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/659375964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":659375964},{"type":"v1","addr":"172.21.15.192:6835","nonce":659375964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:50:20.403164+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5678,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5678":{"gid":5678,"name":"f","rank":0,"incarnation":98,"state":"up:active","state_seq":14,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:50:34.757 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 109 2023-10-30T12:50:34.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.759+0000 7f3b085a1640 1 -- 172.21.15.89:0/2913113007 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3ae40664a0 msgr2=0x7f3ae4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:34.758 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.759+0000 7f3b085a1640 1 --2- 172.21.15.89:0/2913113007 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3ae40664a0 0x7f3ae4068960 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f3afc0b2000 tx=0x7f3afc008040 comp rx=0 tx=0).stop 2023-10-30T12:50:34.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.759+0000 7f3b085a1640 1 -- 172.21.15.89:0/2913113007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b00071110 msgr2=0x7f3b000c1370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:34.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.759+0000 7f3b085a1640 1 --2- 172.21.15.89:0/2913113007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b00071110 0x7f3b000c1370 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f3af000d620 tx=0x7f3af000dae0 comp rx=0 tx=0).stop 2023-10-30T12:50:34.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.759+0000 7f3b085a1640 1 -- 172.21.15.89:0/2913113007 shutdown_connections 2023-10-30T12:50:34.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.759+0000 7f3b085a1640 1 --2- 172.21.15.89:0/2913113007 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3ae40664a0 0x7f3ae4068960 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:34.759 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.759+0000 7f3b085a1640 1 --2- 172.21.15.89:0/2913113007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3b000c6bd0 0x7f3b000c18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:34.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.759+0000 7f3b085a1640 1 --2- 172.21.15.89:0/2913113007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b00071110 0x7f3b000c1370 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:34.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.759+0000 7f3b085a1640 1 --2- 172.21.15.89:0/2913113007 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3b0006f7a0 0x7f3b00149be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:34.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.759+0000 7f3b085a1640 1 -- 172.21.15.89:0/2913113007 >> 172.21.15.89:0/2913113007 conn(0x7f3b000c7910 msgr2=0x7f3b00150180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:34.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.759+0000 7f3b085a1640 1 -- 172.21.15.89:0/2913113007 shutdown_connections 2023-10-30T12:50:34.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:34.759+0000 7f3b085a1640 1 -- 172.21.15.89:0/2913113007 wait complete. 2023-10-30T12:50:34.777 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:50:36.778 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:50:36.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.931+0000 7ff73c891640 1 Processor -- start 2023-10-30T12:50:36.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.931+0000 7ff73c891640 1 -- start start 2023-10-30T12:50:36.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.931+0000 7ff73c891640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7381538a0 0x7ff73815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:36.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.931+0000 7ff73c891640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff7380ca0e0 0x7ff73815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:36.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.931+0000 7ff73c891640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7380cb800 con 0x7ff7380cf160 2023-10-30T12:50:36.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.931+0000 7ff73c891640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff73815dc30 con 0x7ff7380ca0e0 2023-10-30T12:50:36.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.931+0000 7ff73c891640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7380cb330 con 0x7ff7381538a0 2023-10-30T12:50:36.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.931+0000 7ff73659c640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7381538a0 0x7ff73815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:36.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.931+0000 7ff735d9b640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff7380ca0e0 0x7ff73815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:36.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.931+0000 7ff736d9d640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7ff7380cf160 0x7ff7380cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:36884/0 (socket says 172.21.15.89:36884) 2023-10-30T12:50:36.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.931+0000 7ff736d9d640 1 -- 172.21.15.89:0/2314613666 learned_addr learned my addr 172.21.15.89:0/2314613666 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:50:36.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.931+0000 7ff73659c640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7381538a0 0x7ff73815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:51280/0 (socket says 172.21.15.89:51280) 2023-10-30T12:50:36.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.931+0000 7ff735d9b640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff7380ca0e0 0x7ff73815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:48780/0 (socket says 172.21.15.89:48780) 2023-10-30T12:50:36.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.931+0000 7ff73659c640 1 -- 172.21.15.89:0/2314613666 >> v2:172.21.15.192:3300/0 conn(0x7ff7380ca0e0 msgr2=0x7ff73815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:50:36.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.931+0000 7ff73659c640 1 --2- 172.21.15.89:0/2314613666 >> v2:172.21.15.192:3300/0 conn(0x7ff7380ca0e0 0x7ff73815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:36.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.931+0000 7ff73659c640 1 -- 172.21.15.89:0/2314613666 >> v1:172.21.15.192:6789/0 conn(0x7ff7380cf160 legacy=0x7ff7380cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:36.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.931+0000 7ff73659c640 1 -- 172.21.15.89:0/2314613666 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7380cad70 con 0x7ff7381538a0 2023-10-30T12:50:36.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.935+0000 7ff73659c640 1 --2- 172.21.15.89:0/2314613666 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7381538a0 0x7ff73815d6f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff7280097c0 tx=0x7ff72801ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=254825beb22c4fd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:36.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.935+0000 7ff73559a640 1 -- 172.21.15.89:0/2314613666 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7280a8970 con 0x7ff7381538a0 2023-10-30T12:50:36.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.935+0000 7ff73559a640 1 -- 172.21.15.89:0/2314613666 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7280a8ad0 con 0x7ff7381538a0 2023-10-30T12:50:36.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.935+0000 7ff73559a640 1 -- 172.21.15.89:0/2314613666 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7280a8e60 con 0x7ff7381538a0 2023-10-30T12:50:36.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.935+0000 7ff73c891640 1 -- 172.21.15.89:0/2314613666 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7381538a0 msgr2=0x7ff73815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:36.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.935+0000 7ff73c891640 1 --2- 172.21.15.89:0/2314613666 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7381538a0 0x7ff73815d6f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff7280097c0 tx=0x7ff72801ec70 comp rx=0 tx=0).stop 2023-10-30T12:50:36.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.935+0000 7ff73c891640 1 -- 172.21.15.89:0/2314613666 shutdown_connections 2023-10-30T12:50:36.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.935+0000 7ff73c891640 1 --2- 172.21.15.89:0/2314613666 >> v2:172.21.15.192:3300/0 conn(0x7ff7380ca0e0 0x7ff73815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:36.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.935+0000 7ff73c891640 1 --2- 172.21.15.89:0/2314613666 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7381538a0 0x7ff73815d6f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:36.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.935+0000 7ff73c891640 1 -- 172.21.15.89:0/2314613666 >> 172.21.15.89:0/2314613666 conn(0x7ff73806bfe0 msgr2=0x7ff73806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:36.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.935+0000 7ff73c891640 1 -- 172.21.15.89:0/2314613666 shutdown_connections 2023-10-30T12:50:36.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.935+0000 7ff73c891640 1 -- 172.21.15.89:0/2314613666 wait complete. 2023-10-30T12:50:36.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.939+0000 7ff73c891640 1 Processor -- start 2023-10-30T12:50:36.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.939+0000 7ff73c891640 1 -- start start 2023-10-30T12:50:36.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.939+0000 7ff73c891640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff7380ca0e0 0x7ff7380ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:36.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.939+0000 7ff73c891640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7380cf160 0x7ff7380b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:36.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.939+0000 7ff73c891640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7381538a0 0x7ff7380b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:36.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.939+0000 7ff73c891640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7381efcc0 con 0x7ff7381538a0 2023-10-30T12:50:36.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.939+0000 7ff73c891640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7381f01f0 con 0x7ff7380ca0e0 2023-10-30T12:50:36.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.939+0000 7ff73c891640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7381eff10 con 0x7ff7380cf160 2023-10-30T12:50:36.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.939+0000 7ff736d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7381538a0 0x7ff7380b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:36.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.939+0000 7ff736d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7381538a0 0x7ff7380b42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56124/0 (socket says 172.21.15.89:56124) 2023-10-30T12:50:36.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.939+0000 7ff736d9d640 1 -- 172.21.15.89:0/675748527 learned_addr learned my addr 172.21.15.89:0/675748527 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:50:36.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.939+0000 7ff73659c640 1 --2- 172.21.15.89:0/675748527 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff7380ca0e0 0x7ff7380ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:36.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.939+0000 7ff735d9b640 1 --2- 172.21.15.89:0/675748527 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7380cf160 0x7ff7380b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:36.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.939+0000 7ff736d9d640 1 -- 172.21.15.89:0/675748527 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7380cf160 msgr2=0x7ff7380b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:36.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.939+0000 7ff736d9d640 1 --2- 172.21.15.89:0/675748527 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7380cf160 0x7ff7380b3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:36.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.939+0000 7ff736d9d640 1 -- 172.21.15.89:0/675748527 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff7380ca0e0 msgr2=0x7ff7380ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:36.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.939+0000 7ff736d9d640 1 --2- 172.21.15.89:0/675748527 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff7380ca0e0 0x7ff7380ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:36.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.939+0000 7ff736d9d640 1 -- 172.21.15.89:0/675748527 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff728009420 con 0x7ff7381538a0 2023-10-30T12:50:36.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.939+0000 7ff736d9d640 1 --2- 172.21.15.89:0/675748527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7381538a0 0x7ff7380b42f0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7ff73000ca70 tx=0x7ff73000cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:36.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.939+0000 7ff726ffd640 1 -- 172.21.15.89:0/675748527 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff73000abd0 con 0x7ff7381538a0 2023-10-30T12:50:36.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.939+0000 7ff726ffd640 1 -- 172.21.15.89:0/675748527 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff73000ad30 con 0x7ff7381538a0 2023-10-30T12:50:36.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.939+0000 7ff73c891640 1 -- 172.21.15.89:0/675748527 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7380b32d0 con 0x7ff7381538a0 2023-10-30T12:50:36.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.939+0000 7ff726ffd640 1 -- 172.21.15.89:0/675748527 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff730012600 con 0x7ff7381538a0 2023-10-30T12:50:36.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.939+0000 7ff73c891640 1 -- 172.21.15.89:0/675748527 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7380b3890 con 0x7ff7381538a0 2023-10-30T12:50:36.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.943+0000 7ff73c891640 1 -- 172.21.15.89:0/675748527 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7380b2a50 con 0x7ff7381538a0 2023-10-30T12:50:36.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.943+0000 7ff726ffd640 1 -- 172.21.15.89:0/675748527 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff730007070 con 0x7ff7381538a0 2023-10-30T12:50:36.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.943+0000 7ff726ffd640 1 --2- 172.21.15.89:0/675748527 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff7180664d0 0x7ff718068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:36.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.943+0000 7ff726ffd640 1 -- 172.21.15.89:0/675748527 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6664+0+0 (secure 0 0 0) 0x7ff730083ff0 con 0x7ff7381538a0 2023-10-30T12:50:36.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.943+0000 7ff73659c640 1 --2- 172.21.15.89:0/675748527 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff7180664d0 0x7ff718068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:36.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.947+0000 7ff73659c640 1 --2- 172.21.15.89:0/675748527 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff7180664d0 0x7ff718068990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7ff7280b2000 tx=0x7ff7280aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:36.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:36.947+0000 7ff726ffd640 1 -- 172.21.15.89:0/675748527 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff730051150 con 0x7ff7381538a0 2023-10-30T12:50:37.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:37.131+0000 7ff73c891640 1 -- 172.21.15.89:0/675748527 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7380b3ab0 con 0x7ff7381538a0 2023-10-30T12:50:37.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:37.131+0000 7ff726ffd640 1 -- 172.21.15.89:0/675748527 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+6544 (secure 0 0 0) 0x7ff730017090 con 0x7ff7381538a0 2023-10-30T12:50:37.130 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:50:37.130 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":109,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6271,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/659375964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":659375964},{"type":"v1","addr":"172.21.15.192:6835","nonce":659375964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:50:20.403164+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5678,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5678":{"gid":5678,"name":"f","rank":0,"incarnation":98,"state":"up:active","state_seq":14,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:50:37.131 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 109 2023-10-30T12:50:37.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:37.135+0000 7ff73c891640 1 -- 172.21.15.89:0/675748527 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff7180664d0 msgr2=0x7ff718068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:37.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:37.135+0000 7ff73c891640 1 --2- 172.21.15.89:0/675748527 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff7180664d0 0x7ff718068990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7ff7280b2000 tx=0x7ff7280aa070 comp rx=0 tx=0).stop 2023-10-30T12:50:37.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:37.135+0000 7ff73c891640 1 -- 172.21.15.89:0/675748527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7381538a0 msgr2=0x7ff7380b42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:37.132 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:37.135+0000 7ff73c891640 1 --2- 172.21.15.89:0/675748527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7381538a0 0x7ff7380b42f0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7ff73000ca70 tx=0x7ff73000cf30 comp rx=0 tx=0).stop 2023-10-30T12:50:37.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:37.135+0000 7ff73c891640 1 -- 172.21.15.89:0/675748527 shutdown_connections 2023-10-30T12:50:37.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:37.135+0000 7ff73c891640 1 --2- 172.21.15.89:0/675748527 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff7180664d0 0x7ff718068990 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:37.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:37.135+0000 7ff73c891640 1 --2- 172.21.15.89:0/675748527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7381538a0 0x7ff7380b42f0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:37.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:37.135+0000 7ff73c891640 1 --2- 172.21.15.89:0/675748527 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7380cf160 0x7ff7380b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:37.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:37.135+0000 7ff73c891640 1 --2- 172.21.15.89:0/675748527 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff7380ca0e0 0x7ff7380ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:37.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:37.135+0000 7ff73c891640 1 -- 172.21.15.89:0/675748527 >> 172.21.15.89:0/675748527 conn(0x7ff73806bfe0 msgr2=0x7ff738152870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:37.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:37.135+0000 7ff73c891640 1 -- 172.21.15.89:0/675748527 shutdown_connections 2023-10-30T12:50:37.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:37.135+0000 7ff73c891640 1 -- 172.21.15.89:0/675748527 wait complete. 2023-10-30T12:50:37.151 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:50:39.152 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:50:39.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.303+0000 7f2d80254640 1 Processor -- start 2023-10-30T12:50:39.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.303+0000 7f2d80254640 1 -- start start 2023-10-30T12:50:39.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.303+0000 7f2d80254640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2d781538a0 0x7f2d7815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:39.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.303+0000 7f2d80254640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2d780cf160 0x7f2d780cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:39.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.303+0000 7f2d80254640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d780cb800 con 0x7f2d780ca0e0 2023-10-30T12:50:39.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.303+0000 7f2d80254640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d7815dc30 con 0x7f2d781538a0 2023-10-30T12:50:39.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.303+0000 7f2d80254640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d780cb330 con 0x7f2d780cf160 2023-10-30T12:50:39.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.303+0000 7f2d7dff0640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2d781538a0 0x7f2d7815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:39.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.303+0000 7f2d7dff0640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2d781538a0 0x7f2d7815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56132/0 (socket says 172.21.15.89:56132) 2023-10-30T12:50:39.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.303+0000 7f2d7dff0640 1 -- 172.21.15.89:0/966182403 learned_addr learned my addr 172.21.15.89:0/966182403 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:50:39.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.303+0000 7f2d7e7f1640 1 --2- 172.21.15.89:0/966182403 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2d780cf160 0x7f2d780cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:39.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.303+0000 7f2d7dff0640 1 -- 172.21.15.89:0/966182403 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2d780cf160 msgr2=0x7f2d780cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:50:39.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.303+0000 7f2d7dff0640 1 --2- 172.21.15.89:0/966182403 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2d780cf160 0x7f2d780cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:39.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.303+0000 7f2d7dff0640 1 -- 172.21.15.89:0/966182403 >> v1:172.21.15.89:6789/0 conn(0x7f2d780ca0e0 legacy=0x7f2d7815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:39.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.303+0000 7f2d7cfee640 1 -- 172.21.15.89:0/966182403 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 956065410 0 0) 0x7f2d780cb800 con 0x7f2d780ca0e0 2023-10-30T12:50:39.305 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.303+0000 7f2d7dff0640 1 -- 172.21.15.89:0/966182403 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d780cad70 con 0x7f2d781538a0 2023-10-30T12:50:39.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.307+0000 7f2d7dff0640 1 --2- 172.21.15.89:0/966182403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d781538a0 0x7f2d7815d6f0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f2d74009870 tx=0x7f2d7401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28b9b8a94f2e88ec server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:39.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.307+0000 7f2d7cfee640 1 -- 172.21.15.89:0/966182403 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d740a8a30 con 0x7f2d781538a0 2023-10-30T12:50:39.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.307+0000 7f2d7cfee640 1 -- 172.21.15.89:0/966182403 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d740a8b90 con 0x7f2d781538a0 2023-10-30T12:50:39.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.307+0000 7f2d7cfee640 1 -- 172.21.15.89:0/966182403 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d740b0440 con 0x7f2d781538a0 2023-10-30T12:50:39.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.307+0000 7f2d80254640 1 -- 172.21.15.89:0/966182403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d781538a0 msgr2=0x7f2d7815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:39.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.307+0000 7f2d80254640 1 --2- 172.21.15.89:0/966182403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d781538a0 0x7f2d7815d6f0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f2d74009870 tx=0x7f2d7401ed30 comp rx=0 tx=0).stop 2023-10-30T12:50:39.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.307+0000 7f2d80254640 1 -- 172.21.15.89:0/966182403 shutdown_connections 2023-10-30T12:50:39.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.307+0000 7f2d80254640 1 --2- 172.21.15.89:0/966182403 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2d780cf160 0x7f2d780cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:39.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.307+0000 7f2d80254640 1 --2- 172.21.15.89:0/966182403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d781538a0 0x7f2d7815d6f0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:39.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.307+0000 7f2d80254640 1 -- 172.21.15.89:0/966182403 >> 172.21.15.89:0/966182403 conn(0x7f2d7806bfe0 msgr2=0x7f2d7806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:39.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.307+0000 7f2d80254640 1 -- 172.21.15.89:0/966182403 shutdown_connections 2023-10-30T12:50:39.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.307+0000 7f2d80254640 1 -- 172.21.15.89:0/966182403 wait complete. 2023-10-30T12:50:39.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.311+0000 7f2d80254640 1 Processor -- start 2023-10-30T12:50:39.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.311+0000 7f2d80254640 1 -- start start 2023-10-30T12:50:39.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.311+0000 7f2d80254640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d780ca0e0 0x7f2d7815aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:39.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.311+0000 7f2d80254640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2d780cf160 0x7f2d780ce930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:39.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.311+0000 7f2d80254640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2d781538a0 0x7f2d781ef320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:39.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.311+0000 7f2d80254640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d781f3160 con 0x7f2d780ca0e0 2023-10-30T12:50:39.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.311+0000 7f2d80254640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7815b3e0 con 0x7f2d781538a0 2023-10-30T12:50:39.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.311+0000 7f2d80254640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d781f2890 con 0x7f2d780cf160 2023-10-30T12:50:39.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.311+0000 7f2d7dff0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d780ca0e0 0x7f2d7815aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:39.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.311+0000 7f2d7dff0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d780ca0e0 0x7f2d7815aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56138/0 (socket says 172.21.15.89:56138) 2023-10-30T12:50:39.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.311+0000 7f2d7dff0640 1 -- 172.21.15.89:0/729096326 learned_addr learned my addr 172.21.15.89:0/729096326 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:50:39.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.311+0000 7f2d7e7f1640 1 --2- 172.21.15.89:0/729096326 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2d781538a0 0x7f2d781ef320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:39.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.311+0000 7f2d7d7ef640 1 --2- 172.21.15.89:0/729096326 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2d780cf160 0x7f2d780ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:39.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.311+0000 7f2d7dff0640 1 -- 172.21.15.89:0/729096326 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2d780cf160 msgr2=0x7f2d780ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:39.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.311+0000 7f2d7dff0640 1 --2- 172.21.15.89:0/729096326 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2d780cf160 0x7f2d780ce930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:39.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.311+0000 7f2d7dff0640 1 -- 172.21.15.89:0/729096326 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2d781538a0 msgr2=0x7f2d781ef320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:39.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.311+0000 7f2d7dff0640 1 --2- 172.21.15.89:0/729096326 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2d781538a0 0x7f2d781ef320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:39.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.311+0000 7f2d7dff0640 1 -- 172.21.15.89:0/729096326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d740094d0 con 0x7f2d780ca0e0 2023-10-30T12:50:39.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.311+0000 7f2d7d7ef640 1 --2- 172.21.15.89:0/729096326 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2d780cf160 0x7f2d780ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:50:39.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.311+0000 7f2d7dff0640 1 --2- 172.21.15.89:0/729096326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d780ca0e0 0x7f2d7815aea0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f2d740b3040 tx=0x7f2d7401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:39.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.311+0000 7f2d6e7fc640 1 -- 172.21.15.89:0/729096326 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d740b0dd0 con 0x7f2d780ca0e0 2023-10-30T12:50:39.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.311+0000 7f2d6e7fc640 1 -- 172.21.15.89:0/729096326 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d740bb460 con 0x7f2d780ca0e0 2023-10-30T12:50:39.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.311+0000 7f2d6e7fc640 1 -- 172.21.15.89:0/729096326 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d740bb810 con 0x7f2d780ca0e0 2023-10-30T12:50:39.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.311+0000 7f2d80254640 1 -- 172.21.15.89:0/729096326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d780b51f0 con 0x7f2d780ca0e0 2023-10-30T12:50:39.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.311+0000 7f2d80254640 1 -- 172.21.15.89:0/729096326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d780b37a0 con 0x7f2d780ca0e0 2023-10-30T12:50:39.316 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.311+0000 7f2d80254640 1 -- 172.21.15.89:0/729096326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d780b2a50 con 0x7f2d780ca0e0 2023-10-30T12:50:39.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.315+0000 7f2d6e7fc640 1 -- 172.21.15.89:0/729096326 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2d740b83f0 con 0x7f2d780ca0e0 2023-10-30T12:50:39.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.315+0000 7f2d6e7fc640 1 --2- 172.21.15.89:0/729096326 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2d5c0664a0 0x7f2d5c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:39.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.315+0000 7f2d6e7fc640 1 -- 172.21.15.89:0/729096326 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6664+0+0 (secure 0 0 0) 0x7f2d74121920 con 0x7f2d780ca0e0 2023-10-30T12:50:39.317 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.315+0000 7f2d7d7ef640 1 --2- 172.21.15.89:0/729096326 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2d5c0664a0 0x7f2d5c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:39.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.315+0000 7f2d7d7ef640 1 --2- 172.21.15.89:0/729096326 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2d5c0664a0 0x7f2d5c068960 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2d780c4a80 tx=0x7f2d68009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:39.318 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.319+0000 7f2d6e7fc640 1 -- 172.21.15.89:0/729096326 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2d740ee940 con 0x7f2d780ca0e0 2023-10-30T12:50:39.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.499+0000 7f2d80254640 1 -- 172.21.15.89:0/729096326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d780b37a0 con 0x7f2d780ca0e0 2023-10-30T12:50:39.499 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.499+0000 7f2d6e7fc640 1 -- 172.21.15.89:0/729096326 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+6544 (secure 0 0 0) 0x7f2d780b37a0 con 0x7f2d780ca0e0 2023-10-30T12:50:39.500 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:50:39.500 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":109,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":6271,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/659375964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":659375964},{"type":"v1","addr":"172.21.15.192:6835","nonce":659375964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:50:20.403164+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5678,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5678":{"gid":5678,"name":"f","rank":0,"incarnation":98,"state":"up:active","state_seq":14,"addr":"172.21.15.192:6837/2130807115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":2130807115},{"type":"v1","addr":"172.21.15.192:6837","nonce":2130807115}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:50:39.501 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 109 2023-10-30T12:50:39.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.503+0000 7f2d80254640 1 -- 172.21.15.89:0/729096326 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2d5c0664a0 msgr2=0x7f2d5c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:39.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.503+0000 7f2d80254640 1 --2- 172.21.15.89:0/729096326 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2d5c0664a0 0x7f2d5c068960 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2d780c4a80 tx=0x7f2d68009040 comp rx=0 tx=0).stop 2023-10-30T12:50:39.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.503+0000 7f2d80254640 1 -- 172.21.15.89:0/729096326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d780ca0e0 msgr2=0x7f2d7815aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:39.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.503+0000 7f2d80254640 1 --2- 172.21.15.89:0/729096326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d780ca0e0 0x7f2d7815aea0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f2d740b3040 tx=0x7f2d7401bad0 comp rx=0 tx=0).stop 2023-10-30T12:50:39.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.503+0000 7f2d80254640 1 -- 172.21.15.89:0/729096326 shutdown_connections 2023-10-30T12:50:39.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.503+0000 7f2d80254640 1 --2- 172.21.15.89:0/729096326 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2d5c0664a0 0x7f2d5c068960 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:39.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.503+0000 7f2d80254640 1 --2- 172.21.15.89:0/729096326 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2d781538a0 0x7f2d781ef320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:39.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.503+0000 7f2d80254640 1 --2- 172.21.15.89:0/729096326 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2d780cf160 0x7f2d780ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:39.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.503+0000 7f2d80254640 1 --2- 172.21.15.89:0/729096326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d780ca0e0 0x7f2d7815aea0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:39.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.503+0000 7f2d80254640 1 -- 172.21.15.89:0/729096326 >> 172.21.15.89:0/729096326 conn(0x7f2d7806bfe0 msgr2=0x7f2d7814db50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:39.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.503+0000 7f2d80254640 1 -- 172.21.15.89:0/729096326 shutdown_connections 2023-10-30T12:50:39.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:39.503+0000 7f2d80254640 1 -- 172.21.15.89:0/729096326 wait complete. 2023-10-30T12:50:39.519 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:50:41.521 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:50:41.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.671+0000 7f124c279640 1 Processor -- start 2023-10-30T12:50:41.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.671+0000 7f124c279640 1 -- start start 2023-10-30T12:50:41.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.671+0000 7f124c279640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12440c6b80 0x7f12440bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:41.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.671+0000 7f124c279640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12440c0a50 con 0x7f12440c6f60 2023-10-30T12:50:41.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.671+0000 7f124c279640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f124406f6e0 con 0x7f124413f4a0 2023-10-30T12:50:41.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.671+0000 7f124c279640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f124406f9c0 con 0x7f12440c6b80 2023-10-30T12:50:41.671 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.671+0000 7f1249814640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f12440c6f60 0x7f124413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46334/0 (socket says 172.21.15.89:46334) 2023-10-30T12:50:41.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.671+0000 7f1249814640 1 -- 172.21.15.89:0/1386285115 learned_addr learned my addr 172.21.15.89:0/1386285115 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:50:41.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.671+0000 7f124a015640 1 --2- 172.21.15.89:0/1386285115 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12440c6b80 0x7f12440bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:41.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.671+0000 7f1249013640 1 -- 172.21.15.89:0/1386285115 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 877253012 0 0) 0x7f12440c0a50 con 0x7f12440c6f60 2023-10-30T12:50:41.672 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.671+0000 7f1249013640 1 -- 172.21.15.89:0/1386285115 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1230003530 con 0x7f12440c6f60 2023-10-30T12:50:41.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.671+0000 7f124a015640 1 -- 172.21.15.89:0/1386285115 >> v1:172.21.15.192:6789/0 conn(0x7f124413f4a0 legacy=0x7f12441566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:41.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.671+0000 7f1249013640 1 -- 172.21.15.89:0/1386285115 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3057250941 0 0) 0x7f1230003530 con 0x7f12440c6f60 2023-10-30T12:50:41.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.671+0000 7f124a015640 1 -- 172.21.15.89:0/1386285115 >> v1:172.21.15.89:6789/0 conn(0x7f12440c6f60 legacy=0x7f124413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:41.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.675+0000 7f124a015640 1 -- 172.21.15.89:0/1386285115 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f124406fca0 con 0x7f12440c6b80 2023-10-30T12:50:41.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.675+0000 7f124a015640 1 --2- 172.21.15.89:0/1386285115 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12440c6b80 0x7f12440bfec0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f12400097f0 tx=0x7f124001c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c71dd4aa5275d3df server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:41.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.675+0000 7f1249013640 1 -- 172.21.15.89:0/1386285115 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1234001680 con 0x7f12440c6b80 2023-10-30T12:50:41.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.675+0000 7f1249013640 1 -- 172.21.15.89:0/1386285115 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12400a7050 con 0x7f12440c6b80 2023-10-30T12:50:41.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.675+0000 7f1249013640 1 -- 172.21.15.89:0/1386285115 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12400ab600 con 0x7f12440c6b80 2023-10-30T12:50:41.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.679+0000 7f124c279640 1 -- 172.21.15.89:0/1386285115 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12440c6b80 msgr2=0x7f12440bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:41.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.679+0000 7f124c279640 1 --2- 172.21.15.89:0/1386285115 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12440c6b80 0x7f12440bfec0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f12400097f0 tx=0x7f124001c3e0 comp rx=0 tx=0).stop 2023-10-30T12:50:41.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.679+0000 7f124c279640 1 -- 172.21.15.89:0/1386285115 shutdown_connections 2023-10-30T12:50:41.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.679+0000 7f124c279640 1 --2- 172.21.15.89:0/1386285115 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12440c6b80 0x7f12440bfec0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:41.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.679+0000 7f124c279640 1 -- 172.21.15.89:0/1386285115 >> 172.21.15.89:0/1386285115 conn(0x7f12440c7910 msgr2=0x7f12440c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:41.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.679+0000 7f124c279640 1 -- 172.21.15.89:0/1386285115 shutdown_connections 2023-10-30T12:50:41.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.679+0000 7f124c279640 1 -- 172.21.15.89:0/1386285115 wait complete. 2023-10-30T12:50:41.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.679+0000 7f124c279640 1 Processor -- start 2023-10-30T12:50:41.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.679+0000 7f124c279640 1 -- start start 2023-10-30T12:50:41.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.679+0000 7f124c279640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12440c6b80 0x7f12441f2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:41.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.679+0000 7f124c279640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12440c6f60 0x7f124413d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:41.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.679+0000 7f124c279640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f124413f4a0 0x7f1244156050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:41.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.679+0000 7f124c279640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12441544a0 con 0x7f124413f4a0 2023-10-30T12:50:41.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.679+0000 7f124c279640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12441f32b0 con 0x7f12440c6b80 2023-10-30T12:50:41.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.679+0000 7f124c279640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12440c54b0 con 0x7f12440c6f60 2023-10-30T12:50:41.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.679+0000 7f124a816640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f124413f4a0 0x7f1244156050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:41.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.679+0000 7f124a816640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f124413f4a0 0x7f1244156050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39740/0 (socket says 172.21.15.89:39740) 2023-10-30T12:50:41.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.679+0000 7f124a816640 1 -- 172.21.15.89:0/3838751343 learned_addr learned my addr 172.21.15.89:0/3838751343 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:50:41.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.679+0000 7f1249814640 1 --2- 172.21.15.89:0/3838751343 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12440c6f60 0x7f124413d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:41.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.679+0000 7f124a015640 1 --2- 172.21.15.89:0/3838751343 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12440c6b80 0x7f12441f2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:41.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.683+0000 7f124a816640 1 -- 172.21.15.89:0/3838751343 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12440c6f60 msgr2=0x7f124413d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:41.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.683+0000 7f124a816640 1 --2- 172.21.15.89:0/3838751343 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12440c6f60 0x7f124413d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:41.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.683+0000 7f124a816640 1 -- 172.21.15.89:0/3838751343 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12440c6b80 msgr2=0x7f12441f2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:41.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.683+0000 7f124a816640 1 --2- 172.21.15.89:0/3838751343 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12440c6b80 0x7f12441f2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:41.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.683+0000 7f124a816640 1 -- 172.21.15.89:0/3838751343 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1240009420 con 0x7f124413f4a0 2023-10-30T12:50:41.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.683+0000 7f124a816640 1 --2- 172.21.15.89:0/3838751343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f124413f4a0 0x7f1244156050 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f123c00c760 tx=0x7f123c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:41.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.683+0000 7f123affd640 1 -- 172.21.15.89:0/3838751343 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123c003b80 con 0x7f124413f4a0 2023-10-30T12:50:41.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.683+0000 7f123affd640 1 -- 172.21.15.89:0/3838751343 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f123c003ce0 con 0x7f124413f4a0 2023-10-30T12:50:41.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.683+0000 7f123affd640 1 -- 172.21.15.89:0/3838751343 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123c00aa90 con 0x7f124413f4a0 2023-10-30T12:50:41.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.683+0000 7f124c279640 1 -- 172.21.15.89:0/3838751343 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12440c2550 con 0x7f124413f4a0 2023-10-30T12:50:41.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.683+0000 7f124c279640 1 -- 172.21.15.89:0/3838751343 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12440c2b10 con 0x7f124413f4a0 2023-10-30T12:50:41.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.683+0000 7f124c279640 1 -- 172.21.15.89:0/3838751343 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f124404e170 con 0x7f124413f4a0 2023-10-30T12:50:41.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.683+0000 7f123affd640 1 -- 172.21.15.89:0/3838751343 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f123c00abf0 con 0x7f124413f4a0 2023-10-30T12:50:41.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.683+0000 7f123affd640 1 --2- 172.21.15.89:0/3838751343 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f12280664d0 0x7f1228068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:41.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.683+0000 7f123affd640 1 -- 172.21.15.89:0/3838751343 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f123c083ff0 con 0x7f124413f4a0 2023-10-30T12:50:41.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.687+0000 7f124a015640 1 --2- 172.21.15.89:0/3838751343 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f12280664d0 0x7f1228068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:41.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.687+0000 7f124a015640 1 --2- 172.21.15.89:0/3838751343 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f12280664d0 0x7f1228068990 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f12400ad000 tx=0x7f12400a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:41.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.687+0000 7f123affd640 1 -- 172.21.15.89:0/3838751343 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f123c051040 con 0x7f124413f4a0 2023-10-30T12:50:41.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.871+0000 7f124c279640 1 -- 172.21.15.89:0/3838751343 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12440c0dd0 con 0x7f124413f4a0 2023-10-30T12:50:41.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.871+0000 7f123affd640 1 -- 172.21.15.89:0/3838751343 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+5746 (secure 0 0 0) 0x7f123c054660 con 0x7f124413f4a0 2023-10-30T12:50:41.870 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:50:41.870 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:50:41.318621+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6271,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6271":{"gid":6271,"name":"b","rank":0,"incarnation":110,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6835/659375964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":659375964},{"type":"v1","addr":"172.21.15.192:6835","nonce":659375964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:50:41.870 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 110 2023-10-30T12:50:41.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.875+0000 7f124c279640 1 -- 172.21.15.89:0/3838751343 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f12280664d0 msgr2=0x7f1228068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:41.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.875+0000 7f124c279640 1 --2- 172.21.15.89:0/3838751343 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f12280664d0 0x7f1228068990 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f12400ad000 tx=0x7f12400a6000 comp rx=0 tx=0).stop 2023-10-30T12:50:41.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.875+0000 7f124c279640 1 -- 172.21.15.89:0/3838751343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f124413f4a0 msgr2=0x7f1244156050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:41.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.875+0000 7f124c279640 1 --2- 172.21.15.89:0/3838751343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f124413f4a0 0x7f1244156050 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f123c00c760 tx=0x7f123c00cc20 comp rx=0 tx=0).stop 2023-10-30T12:50:41.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.875+0000 7f124c279640 1 -- 172.21.15.89:0/3838751343 shutdown_connections 2023-10-30T12:50:41.872 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.875+0000 7f124c279640 1 --2- 172.21.15.89:0/3838751343 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f12280664d0 0x7f1228068990 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:41.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.875+0000 7f124c279640 1 --2- 172.21.15.89:0/3838751343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f124413f4a0 0x7f1244156050 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:41.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.875+0000 7f124c279640 1 --2- 172.21.15.89:0/3838751343 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12440c6f60 0x7f124413d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:41.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.875+0000 7f124c279640 1 --2- 172.21.15.89:0/3838751343 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12440c6b80 0x7f12441f2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:41.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.875+0000 7f124c279640 1 -- 172.21.15.89:0/3838751343 >> 172.21.15.89:0/3838751343 conn(0x7f12440c7910 msgr2=0x7f12440c5e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:41.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.875+0000 7f124c279640 1 -- 172.21.15.89:0/3838751343 shutdown_connections 2023-10-30T12:50:41.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:41.875+0000 7f124c279640 1 -- 172.21.15.89:0/3838751343 wait complete. 2023-10-30T12:50:41.890 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-10-30T12:50:41.890 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:50:41.891 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:50:42.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.027+0000 7f779643c640 1 Processor -- start 2023-10-30T12:50:42.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f779643c640 1 -- start start 2023-10-30T12:50:42.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f779643c640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77900c6840 0x7f779015a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:42.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f779643c640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f77900c3640 0x7f7790156590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:42.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f779643c640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77900c3b30 con 0x7f77900c6c20 2023-10-30T12:50:42.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f779643c640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77900c2aa0 con 0x7f77900c3640 2023-10-30T12:50:42.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f779643c640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77900c3df0 con 0x7f77900c6840 2023-10-30T12:50:42.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f7795c3b640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f77900c6c20 0x7f7790153730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46348/0 (socket says 172.21.15.89:46348) 2023-10-30T12:50:42.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f7795c3b640 1 -- 172.21.15.89:0/628674504 learned_addr learned my addr 172.21.15.89:0/628674504 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:50:42.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f779543a640 1 --2- 172.21.15.89:0/628674504 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77900c6840 0x7f779015a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:42.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f7794c39640 1 --2- 172.21.15.89:0/628674504 >> v2:172.21.15.192:3300/0 conn(0x7f77900c3640 0x7f7790156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:42.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f7787fff640 1 -- 172.21.15.89:0/628674504 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169844301 0 0) 0x7f77900c3b30 con 0x7f77900c6c20 2023-10-30T12:50:42.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f7787fff640 1 -- 172.21.15.89:0/628674504 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f777c003530 con 0x7f77900c6c20 2023-10-30T12:50:42.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f7787fff640 1 -- 172.21.15.89:0/628674504 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 237426394 0 0) 0x7f777c003530 con 0x7f77900c6c20 2023-10-30T12:50:42.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f7787fff640 1 -- 172.21.15.89:0/628674504 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77900c6840 msgr2=0x7f779015a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:50:42.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f7787fff640 1 --2- 172.21.15.89:0/628674504 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77900c6840 0x7f779015a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:42.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f7787fff640 1 -- 172.21.15.89:0/628674504 >> v2:172.21.15.192:3300/0 conn(0x7f77900c3640 msgr2=0x7f7790156590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:50:42.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f7787fff640 1 --2- 172.21.15.89:0/628674504 >> v2:172.21.15.192:3300/0 conn(0x7f77900c3640 0x7f7790156590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:42.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f7787fff640 1 -- 172.21.15.89:0/628674504 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7790159ba0 con 0x7f77900c6c20 2023-10-30T12:50:42.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f7787fff640 1 -- 172.21.15.89:0/628674504 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f7788001a00 con 0x7f77900c6c20 2023-10-30T12:50:42.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f7787fff640 1 -- 172.21.15.89:0/628674504 >> v1:172.21.15.89:6789/0 conn(0x7f77900c6c20 legacy=0x7f7790153730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:42.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f7787fff640 1 --2- 172.21.15.89:0/628674504 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77900c6840 0x7f777c003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:42.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f7787fff640 1 --2- 172.21.15.89:0/628674504 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f777c003f00 0x7f777c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:42.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f7787fff640 1 --2- 172.21.15.89:0/628674504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f777c004d10 0x7f777c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:42.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f7787fff640 1 -- 172.21.15.89:0/628674504 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77900c3df0 con 0x7f777c004d10 2023-10-30T12:50:42.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f7787fff640 1 -- 172.21.15.89:0/628674504 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777c00b990 con 0x7f777c003f00 2023-10-30T12:50:42.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f7787fff640 1 -- 172.21.15.89:0/628674504 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f777c00bad0 con 0x7f77900c6840 2023-10-30T12:50:42.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f7795c3b640 1 --2- 172.21.15.89:0/628674504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f777c004d10 0x7f777c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:42.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f779543a640 1 --2- 172.21.15.89:0/628674504 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77900c6840 0x7f777c003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:42.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f7794c39640 1 --2- 172.21.15.89:0/628674504 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f777c003f00 0x7f777c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:42.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f7795c3b640 1 -- 172.21.15.89:0/628674504 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77900c6840 msgr2=0x7f777c003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:50:42.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f7795c3b640 1 --2- 172.21.15.89:0/628674504 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f77900c6840 0x7f777c003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:42.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f7795c3b640 1 -- 172.21.15.89:0/628674504 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f777c003f00 msgr2=0x7f777c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:50:42.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f7795c3b640 1 --2- 172.21.15.89:0/628674504 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f777c003f00 0x7f777c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:42.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.031+0000 7f7795c3b640 1 -- 172.21.15.89:0/628674504 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777c003530 con 0x7f777c004d10 2023-10-30T12:50:42.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.035+0000 7f7795c3b640 1 --2- 172.21.15.89:0/628674504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f777c004d10 0x7f777c005190 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f77900c7040 tx=0x7f778801f4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24b5ebb8b43635c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:42.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.035+0000 7f7787fff640 1 -- 172.21.15.89:0/628674504 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77880ae650 con 0x7f777c004d10 2023-10-30T12:50:42.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.035+0000 7f7787fff640 1 -- 172.21.15.89:0/628674504 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77880ae7b0 con 0x7f777c004d10 2023-10-30T12:50:42.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.035+0000 7f7787fff640 1 -- 172.21.15.89:0/628674504 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77880aebc0 con 0x7f777c004d10 2023-10-30T12:50:42.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.035+0000 7f779643c640 1 -- 172.21.15.89:0/628674504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f777c004d10 msgr2=0x7f777c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:42.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.035+0000 7f779643c640 1 --2- 172.21.15.89:0/628674504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f777c004d10 0x7f777c005190 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f77900c7040 tx=0x7f778801f4d0 comp rx=0 tx=0).stop 2023-10-30T12:50:42.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.035+0000 7f7795c3b640 1 -- 172.21.15.89:0/628674504 reap_dead start 2023-10-30T12:50:42.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.035+0000 7f779643c640 1 -- 172.21.15.89:0/628674504 shutdown_connections 2023-10-30T12:50:42.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.035+0000 7f779643c640 1 -- 172.21.15.89:0/628674504 >> 172.21.15.89:0/628674504 conn(0x7f77900c75d0 msgr2=0x7f77900c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:42.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.035+0000 7f779643c640 1 -- 172.21.15.89:0/628674504 shutdown_connections 2023-10-30T12:50:42.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.035+0000 7f779643c640 1 -- 172.21.15.89:0/628674504 wait complete. 2023-10-30T12:50:42.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.039+0000 7f779643c640 1 Processor -- start 2023-10-30T12:50:42.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.039+0000 7f779643c640 1 -- start start 2023-10-30T12:50:42.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.039+0000 7f779643c640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f777c004d10 0x7f7790155f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:42.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.039+0000 7f779643c640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77901556e0 0x7f7790158780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:42.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.039+0000 7f779643c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7790154400 0x7f7790158cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:42.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.039+0000 7f779643c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77901582f0 con 0x7f7790154400 2023-10-30T12:50:42.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.039+0000 7f779643c640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7790156ff0 con 0x7f77901556e0 2023-10-30T12:50:42.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.039+0000 7f779643c640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77901572d0 con 0x7f777c004d10 2023-10-30T12:50:42.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.039+0000 7f7795c3b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7790154400 0x7f7790158cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:42.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.039+0000 7f7795c3b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7790154400 0x7f7790158cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39758/0 (socket says 172.21.15.89:39758) 2023-10-30T12:50:42.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.039+0000 7f7795c3b640 1 -- 172.21.15.89:0/1373341696 learned_addr learned my addr 172.21.15.89:0/1373341696 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:50:42.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.039+0000 7f779543a640 1 --2- 172.21.15.89:0/1373341696 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f777c004d10 0x7f7790155f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:42.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.039+0000 7f7794c39640 1 --2- 172.21.15.89:0/1373341696 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77901556e0 0x7f7790158780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:42.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.039+0000 7f7795c3b640 1 -- 172.21.15.89:0/1373341696 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f777c004d10 msgr2=0x7f7790155f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:42.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.039+0000 7f7795c3b640 1 --2- 172.21.15.89:0/1373341696 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f777c004d10 0x7f7790155f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:42.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.039+0000 7f7795c3b640 1 -- 172.21.15.89:0/1373341696 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77901556e0 msgr2=0x7f7790158780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:42.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.039+0000 7f7795c3b640 1 --2- 172.21.15.89:0/1373341696 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77901556e0 0x7f7790158780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:42.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.039+0000 7f7795c3b640 1 -- 172.21.15.89:0/1373341696 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77901575b0 con 0x7f7790154400 2023-10-30T12:50:42.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.039+0000 7f779543a640 1 --2- 172.21.15.89:0/1373341696 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f777c004d10 0x7f7790155f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:50:42.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.039+0000 7f7794c39640 1 --2- 172.21.15.89:0/1373341696 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77901556e0 0x7f7790158780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:50:42.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.039+0000 7f7795c3b640 1 --2- 172.21.15.89:0/1373341696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7790154400 0x7f7790158cc0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f778801fd70 tx=0x7f777c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:42.046 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.039+0000 7f7785ffb640 1 -- 172.21.15.89:0/1373341696 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77880aeb20 con 0x7f7790154400 2023-10-30T12:50:42.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.039+0000 7f7785ffb640 1 -- 172.21.15.89:0/1373341696 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77880aec80 con 0x7f7790154400 2023-10-30T12:50:42.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.039+0000 7f779643c640 1 -- 172.21.15.89:0/1373341696 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7790157890 con 0x7f7790154400 2023-10-30T12:50:42.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.039+0000 7f7785ffb640 1 -- 172.21.15.89:0/1373341696 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778800f1d0 con 0x7f7790154400 2023-10-30T12:50:42.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.039+0000 7f779643c640 1 -- 172.21.15.89:0/1373341696 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7790071150 con 0x7f7790154400 2023-10-30T12:50:42.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.043+0000 7f779643c640 1 -- 172.21.15.89:0/1373341696 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7790158460 con 0x7f7790154400 2023-10-30T12:50:42.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.043+0000 7f7785ffb640 1 -- 172.21.15.89:0/1373341696 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f77880b8440 con 0x7f7790154400 2023-10-30T12:50:42.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.043+0000 7f7785ffb640 1 --2- 172.21.15.89:0/1373341696 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f77740664a0 0x7f7774068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:42.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.043+0000 7f7785ffb640 1 -- 172.21.15.89:0/1373341696 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f778811ecb0 con 0x7f7790154400 2023-10-30T12:50:42.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.043+0000 7f779543a640 1 --2- 172.21.15.89:0/1373341696 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f77740664a0 0x7f7774068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:42.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.043+0000 7f779543a640 1 --2- 172.21.15.89:0/1373341696 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f77740664a0 0x7f7774068960 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f778c009bc0 tx=0x7f778c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:42.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.047+0000 7f7785ffb640 1 -- 172.21.15.89:0/1373341696 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f77880ebcb0 con 0x7f7790154400 2023-10-30T12:50:42.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.227+0000 7f779643c640 1 -- 172.21.15.89:0/1373341696 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f779004e170 con 0x7f7790154400 2023-10-30T12:50:42.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.227+0000 7f7785ffb640 1 -- 172.21.15.89:0/1373341696 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+5746 (secure 0 0 0) 0x7f77880b3090 con 0x7f7790154400 2023-10-30T12:50:42.227 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:50:42.227 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:50:41.318621+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6271,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6271":{"gid":6271,"name":"b","rank":0,"incarnation":110,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6835/659375964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":659375964},{"type":"v1","addr":"172.21.15.192:6835","nonce":659375964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:50:42.227 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 110 2023-10-30T12:50:42.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.231+0000 7f779643c640 1 -- 172.21.15.89:0/1373341696 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f77740664a0 msgr2=0x7f7774068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:42.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.231+0000 7f779643c640 1 --2- 172.21.15.89:0/1373341696 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f77740664a0 0x7f7774068960 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f778c009bc0 tx=0x7f778c008040 comp rx=0 tx=0).stop 2023-10-30T12:50:42.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.231+0000 7f779643c640 1 -- 172.21.15.89:0/1373341696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7790154400 msgr2=0x7f7790158cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:42.229 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.231+0000 7f779643c640 1 --2- 172.21.15.89:0/1373341696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7790154400 0x7f7790158cc0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f778801fd70 tx=0x7f777c005800 comp rx=0 tx=0).stop 2023-10-30T12:50:42.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.231+0000 7f779643c640 1 -- 172.21.15.89:0/1373341696 shutdown_connections 2023-10-30T12:50:42.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.231+0000 7f779643c640 1 --2- 172.21.15.89:0/1373341696 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f77740664a0 0x7f7774068960 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:42.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.231+0000 7f779643c640 1 --2- 172.21.15.89:0/1373341696 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77901556e0 0x7f7790158780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:42.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.231+0000 7f779643c640 1 --2- 172.21.15.89:0/1373341696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7790154400 0x7f7790158cc0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:42.230 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.231+0000 7f779643c640 1 --2- 172.21.15.89:0/1373341696 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f777c004d10 0x7f7790155f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:42.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.231+0000 7f779643c640 1 -- 172.21.15.89:0/1373341696 >> 172.21.15.89:0/1373341696 conn(0x7f77900c75d0 msgr2=0x7f77900c4390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:42.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.231+0000 7f779643c640 1 -- 172.21.15.89:0/1373341696 shutdown_connections 2023-10-30T12:50:42.231 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:42.231+0000 7f779643c640 1 -- 172.21.15.89:0/1373341696 wait complete. 2023-10-30T12:50:42.248 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 110, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:50.729245+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 110, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5728": { "addr": "172.21.15.192:6839/3991891597", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6838", "nonce": 3991891597, "type": "v2" }, { "addr": "172.21.15.192:6839", "nonce": 3991891597, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 5728, "incarnation": 73, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 19 }, "gid_6075": { "addr": "172.21.15.89:6837/2412618858", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6836", "nonce": 2412618858, "type": "v2" }, { "addr": "172.21.15.89:6837", "nonce": 2412618858, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 6075, "incarnation": 92, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 38 }, "gid_6271": { "addr": "172.21.15.192:6835/659375964", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6834", "nonce": 659375964, "type": "v2" }, { "addr": "172.21.15.192:6835", "nonce": 659375964, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6271, "incarnation": 110, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:50:41.318621+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6271, "mds_1": 6075, "mds_2": 5728 } } } ], "standbys": [ { "addr": "172.21.15.89:6834/2909790173", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6832", "nonce": 2909790173, "type": "v2" }, { "addr": "172.21.15.89:6834", "nonce": 2909790173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.89:6835/2387357615", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6833", "nonce": 2387357615, "type": "v2" }, { "addr": "172.21.15.89:6835", "nonce": 2387357615, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4239, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:50:44.249 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:50:44.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.399+0000 7f2009b2a640 1 Processor -- start 2023-10-30T12:50:44.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.399+0000 7f2009b2a640 1 -- start start 2023-10-30T12:50:44.398 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.399+0000 7f2009b2a640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20040cf160 0x7f20040cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:44.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.399+0000 7f2009b2a640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20040cb800 con 0x7f20041538a0 2023-10-30T12:50:44.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.399+0000 7f2009b2a640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20040cb330 con 0x7f200415d6f0 2023-10-30T12:50:44.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.399+0000 7f2009b2a640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20040cad70 con 0x7f20040cf160 2023-10-30T12:50:44.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.399+0000 7f20037fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f20041538a0 0x7f20040ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46354/0 (socket says 172.21.15.89:46354) 2023-10-30T12:50:44.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.399+0000 7f20037fe640 1 -- 172.21.15.89:0/4159985979 learned_addr learned my addr 172.21.15.89:0/4159985979 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:50:44.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.399+0000 7f2003fff640 1 --2- 172.21.15.89:0/4159985979 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20040cf160 0x7f20040cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:44.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.399+0000 7f2002ffd640 1 -- 172.21.15.89:0/4159985979 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409182428 0 0) 0x7f20040cb800 con 0x7f20041538a0 2023-10-30T12:50:44.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.399+0000 7f2002ffd640 1 -- 172.21.15.89:0/4159985979 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ff0003530 con 0x7f20041538a0 2023-10-30T12:50:44.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.399+0000 7f2002ffd640 1 -- 172.21.15.89:0/4159985979 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2503389157 0 0) 0x7f1ff0003530 con 0x7f20041538a0 2023-10-30T12:50:44.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.399+0000 7f2002ffd640 1 -- 172.21.15.89:0/4159985979 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20040cf160 msgr2=0x7f20040cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:50:44.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.399+0000 7f2002ffd640 1 --2- 172.21.15.89:0/4159985979 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20040cf160 0x7f20040cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:44.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.399+0000 7f2002ffd640 1 -- 172.21.15.89:0/4159985979 >> v1:172.21.15.192:6789/0 conn(0x7f200415d6f0 legacy=0x7f200415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:44.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.403+0000 7f2002ffd640 1 -- 172.21.15.89:0/4159985979 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20040cb050 con 0x7f20041538a0 2023-10-30T12:50:44.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.403+0000 7f2002ffd640 1 -- 172.21.15.89:0/4159985979 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f1ff4001680 con 0x7f20041538a0 2023-10-30T12:50:44.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.403+0000 7f2002ffd640 1 -- 172.21.15.89:0/4159985979 >> v1:172.21.15.89:6789/0 conn(0x7f20041538a0 legacy=0x7f20040ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:44.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.403+0000 7f2002ffd640 1 --2- 172.21.15.89:0/4159985979 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20040cf160 0x7f1ff00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:44.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.403+0000 7f2002ffd640 1 --2- 172.21.15.89:0/4159985979 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ff0003ef0 0x7f1ff00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:44.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.403+0000 7f2002ffd640 1 --2- 172.21.15.89:0/4159985979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ff0004d10 0x7f1ff0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:44.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.403+0000 7f2002ffd640 1 -- 172.21.15.89:0/4159985979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20040cad70 con 0x7f1ff0004d10 2023-10-30T12:50:44.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.403+0000 7f2002ffd640 1 -- 172.21.15.89:0/4159985979 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff000b9f0 con 0x7f1ff0003ef0 2023-10-30T12:50:44.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.403+0000 7f2002ffd640 1 -- 172.21.15.89:0/4159985979 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff000bbb0 con 0x7f20040cf160 2023-10-30T12:50:44.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.403+0000 7f2003fff640 1 --2- 172.21.15.89:0/4159985979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ff0004d10 0x7f1ff0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:44.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.403+0000 7f1ffbfff640 1 --2- 172.21.15.89:0/4159985979 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ff0003ef0 0x7f1ff00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:44.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.403+0000 7f20037fe640 1 --2- 172.21.15.89:0/4159985979 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20040cf160 0x7f1ff00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:44.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.403+0000 7f2003fff640 1 -- 172.21.15.89:0/4159985979 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20040cf160 msgr2=0x7f1ff00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:50:44.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.403+0000 7f2003fff640 1 --2- 172.21.15.89:0/4159985979 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20040cf160 0x7f1ff00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:44.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.403+0000 7f2003fff640 1 -- 172.21.15.89:0/4159985979 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ff0003ef0 msgr2=0x7f1ff00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:50:44.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.403+0000 7f2003fff640 1 --2- 172.21.15.89:0/4159985979 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ff0003ef0 0x7f1ff00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:44.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.403+0000 7f2003fff640 1 -- 172.21.15.89:0/4159985979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff0003530 con 0x7f1ff0004d10 2023-10-30T12:50:44.406 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.403+0000 7f20037fe640 1 --2- 172.21.15.89:0/4159985979 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20040cf160 0x7f1ff00039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:50:44.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.403+0000 7f2003fff640 1 --2- 172.21.15.89:0/4159985979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ff0004d10 0x7f1ff0005190 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc009fc0 tx=0x7f1ffc01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f03d0226a9f052 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:44.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.403+0000 7f2002ffd640 1 -- 172.21.15.89:0/4159985979 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc0a5c70 con 0x7f1ff0004d10 2023-10-30T12:50:44.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.403+0000 7f2002ffd640 1 -- 172.21.15.89:0/4159985979 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ffc00f040 con 0x7f1ff0004d10 2023-10-30T12:50:44.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.403+0000 7f2002ffd640 1 -- 172.21.15.89:0/4159985979 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc0ad440 con 0x7f1ff0004d10 2023-10-30T12:50:44.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.407+0000 7f2009b2a640 1 -- 172.21.15.89:0/4159985979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ff0004d10 msgr2=0x7f1ff0005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:44.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.407+0000 7f2009b2a640 1 --2- 172.21.15.89:0/4159985979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ff0004d10 0x7f1ff0005190 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc009fc0 tx=0x7f1ffc01cc60 comp rx=0 tx=0).stop 2023-10-30T12:50:44.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.407+0000 7f2003fff640 1 -- 172.21.15.89:0/4159985979 reap_dead start 2023-10-30T12:50:44.410 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.407+0000 7f2009b2a640 1 -- 172.21.15.89:0/4159985979 shutdown_connections 2023-10-30T12:50:44.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.407+0000 7f2009b2a640 1 -- 172.21.15.89:0/4159985979 >> 172.21.15.89:0/4159985979 conn(0x7f200406bfe0 msgr2=0x7f200406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:44.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.407+0000 7f2009b2a640 1 -- 172.21.15.89:0/4159985979 shutdown_connections 2023-10-30T12:50:44.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.407+0000 7f2009b2a640 1 -- 172.21.15.89:0/4159985979 wait complete. 2023-10-30T12:50:44.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.407+0000 7f2009b2a640 1 Processor -- start 2023-10-30T12:50:44.411 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.407+0000 7f2009b2a640 1 -- start start 2023-10-30T12:50:44.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.407+0000 7f2009b2a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ff0004d10 0x7f200414f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:44.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.407+0000 7f2009b2a640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20040c7860 0x7f20041eea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:44.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.407+0000 7f2009b2a640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20040ce390 0x7f2004158d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:44.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.407+0000 7f2009b2a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f200414ffd0 con 0x7f1ff0004d10 2023-10-30T12:50:44.412 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.407+0000 7f2009b2a640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20041eef40 con 0x7f20040ce390 2023-10-30T12:50:44.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.407+0000 7f2009b2a640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2004150bd0 con 0x7f20040c7860 2023-10-30T12:50:44.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.407+0000 7f20037fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ff0004d10 0x7f200414f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:44.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.407+0000 7f20037fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ff0004d10 0x7f200414f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39764/0 (socket says 172.21.15.89:39764) 2023-10-30T12:50:44.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.407+0000 7f20037fe640 1 -- 172.21.15.89:0/3680099007 learned_addr learned my addr 172.21.15.89:0/3680099007 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:50:44.413 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.407+0000 7f2003fff640 1 --2- 172.21.15.89:0/3680099007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20040ce390 0x7f2004158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:44.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.407+0000 7f1ffbfff640 1 --2- 172.21.15.89:0/3680099007 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20040c7860 0x7f20041eea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:44.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.411+0000 7f20037fe640 1 -- 172.21.15.89:0/3680099007 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20040c7860 msgr2=0x7f20041eea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:44.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.411+0000 7f20037fe640 1 --2- 172.21.15.89:0/3680099007 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20040c7860 0x7f20041eea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:44.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.411+0000 7f20037fe640 1 -- 172.21.15.89:0/3680099007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20040ce390 msgr2=0x7f2004158d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:44.414 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.411+0000 7f20037fe640 1 --2- 172.21.15.89:0/3680099007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20040ce390 0x7f2004158d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:44.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.411+0000 7f20037fe640 1 -- 172.21.15.89:0/3680099007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20041508f0 con 0x7f1ff0004d10 2023-10-30T12:50:44.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.411+0000 7f1ffbfff640 1 --2- 172.21.15.89:0/3680099007 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20040c7860 0x7f20041eea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:50:44.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.411+0000 7f20037fe640 1 --2- 172.21.15.89:0/3680099007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ff0004d10 0x7f200414f720 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f1ff400e8f0 tx=0x7f1ff400edb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:44.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.411+0000 7f2000ff9640 1 -- 172.21.15.89:0/3680099007 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4017070 con 0x7f1ff0004d10 2023-10-30T12:50:44.415 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.411+0000 7f2000ff9640 1 -- 172.21.15.89:0/3680099007 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff400c990 con 0x7f1ff0004d10 2023-10-30T12:50:44.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.411+0000 7f2000ff9640 1 -- 172.21.15.89:0/3680099007 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff400cb00 con 0x7f1ff0004d10 2023-10-30T12:50:44.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.411+0000 7f2009b2a640 1 -- 172.21.15.89:0/3680099007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f200414ec40 con 0x7f1ff0004d10 2023-10-30T12:50:44.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.411+0000 7f2009b2a640 1 -- 172.21.15.89:0/3680099007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f200414f200 con 0x7f1ff0004d10 2023-10-30T12:50:44.416 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.411+0000 7f2009b2a640 1 -- 172.21.15.89:0/3680099007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20040c7c40 con 0x7f1ff0004d10 2023-10-30T12:50:44.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.411+0000 7f2000ff9640 1 -- 172.21.15.89:0/3680099007 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1ff400cc60 con 0x7f1ff0004d10 2023-10-30T12:50:44.417 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.411+0000 7f2000ff9640 1 --2- 172.21.15.89:0/3680099007 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1fe40664d0 0x7f1fe4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:44.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.411+0000 7f2000ff9640 1 -- 172.21.15.89:0/3680099007 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f1ff4083d90 con 0x7f1ff0004d10 2023-10-30T12:50:44.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.415+0000 7f1ffbfff640 1 --2- 172.21.15.89:0/3680099007 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1fe40664d0 0x7f1fe4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:44.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.415+0000 7f1ffbfff640 1 --2- 172.21.15.89:0/3680099007 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1fe40664d0 0x7f1fe4068990 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f2004153080 tx=0x7f1fec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:44.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.415+0000 7f2000ff9640 1 -- 172.21.15.89:0/3680099007 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1ff4050d70 con 0x7f1ff0004d10 2023-10-30T12:50:44.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.599+0000 7f2009b2a640 1 -- 172.21.15.89:0/3680099007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f200404e170 con 0x7f1ff0004d10 2023-10-30T12:50:44.599 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.599+0000 7f2000ff9640 1 -- 172.21.15.89:0/3680099007 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+5746 (secure 0 0 0) 0x7f1ff4054390 con 0x7f1ff0004d10 2023-10-30T12:50:44.599 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:50:44.600 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:50:41.318621+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6271,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6271":{"gid":6271,"name":"b","rank":0,"incarnation":110,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6835/659375964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":659375964},{"type":"v1","addr":"172.21.15.192:6835","nonce":659375964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:50:44.600 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 110 2023-10-30T12:50:44.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.603+0000 7f2009b2a640 1 -- 172.21.15.89:0/3680099007 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1fe40664d0 msgr2=0x7f1fe4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:44.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.603+0000 7f2009b2a640 1 --2- 172.21.15.89:0/3680099007 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1fe40664d0 0x7f1fe4068990 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f2004153080 tx=0x7f1fec009040 comp rx=0 tx=0).stop 2023-10-30T12:50:44.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.603+0000 7f2009b2a640 1 -- 172.21.15.89:0/3680099007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ff0004d10 msgr2=0x7f200414f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:44.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.603+0000 7f2009b2a640 1 --2- 172.21.15.89:0/3680099007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ff0004d10 0x7f200414f720 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f1ff400e8f0 tx=0x7f1ff400edb0 comp rx=0 tx=0).stop 2023-10-30T12:50:44.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.603+0000 7f2009b2a640 1 -- 172.21.15.89:0/3680099007 shutdown_connections 2023-10-30T12:50:44.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.603+0000 7f2009b2a640 1 --2- 172.21.15.89:0/3680099007 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1fe40664d0 0x7f1fe4068990 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:44.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.603+0000 7f2009b2a640 1 --2- 172.21.15.89:0/3680099007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f20040ce390 0x7f2004158d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:44.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.603+0000 7f2009b2a640 1 --2- 172.21.15.89:0/3680099007 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20040c7860 0x7f20041eea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:44.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.603+0000 7f2009b2a640 1 --2- 172.21.15.89:0/3680099007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ff0004d10 0x7f200414f720 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:44.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.603+0000 7f2009b2a640 1 -- 172.21.15.89:0/3680099007 >> 172.21.15.89:0/3680099007 conn(0x7f200406bfe0 msgr2=0x7f20040c5580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:44.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.603+0000 7f2009b2a640 1 -- 172.21.15.89:0/3680099007 shutdown_connections 2023-10-30T12:50:44.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:44.603+0000 7f2009b2a640 1 -- 172.21.15.89:0/3680099007 wait complete. 2023-10-30T12:50:44.619 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:50:46.621 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:50:46.760 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.763+0000 7f3ebf846640 1 Processor -- start 2023-10-30T12:50:46.761 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.763+0000 7f3ebf846640 1 -- start start 2023-10-30T12:50:46.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.763+0000 7f3ebf846640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3eb81580e0 0x7f3eb8070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:46.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.763+0000 7f3ebf846640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3eb815e700 0x7f3eb806a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:46.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.763+0000 7f3ebf846640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb80c5810 con 0x7f3eb80c6540 2023-10-30T12:50:46.763 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.763+0000 7f3ebf846640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb80c6990 con 0x7f3eb81580e0 2023-10-30T12:50:46.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.763+0000 7f3ebf846640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb815eb90 con 0x7f3eb815e700 2023-10-30T12:50:46.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.763+0000 7f3ebcde1640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f3eb80c6540 0x7f3eb8069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:46368/0 (socket says 172.21.15.89:46368) 2023-10-30T12:50:46.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.763+0000 7f3ebcde1640 1 -- 172.21.15.89:0/2248529797 learned_addr learned my addr 172.21.15.89:0/2248529797 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:50:46.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.763+0000 7f3ebd5e2640 1 --2- 172.21.15.89:0/2248529797 >> v2:172.21.15.192:3300/0 conn(0x7f3eb81580e0 0x7f3eb8070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:46.764 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.763+0000 7f3ebdde3640 1 --2- 172.21.15.89:0/2248529797 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3eb815e700 0x7f3eb806a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:46.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.763+0000 7f3eaffff640 1 -- 172.21.15.89:0/2248529797 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 988420913 0 0) 0x7f3eb80c5810 con 0x7f3eb80c6540 2023-10-30T12:50:46.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.763+0000 7f3eaffff640 1 -- 172.21.15.89:0/2248529797 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ea4003530 con 0x7f3eb80c6540 2023-10-30T12:50:46.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.763+0000 7f3ebdde3640 1 -- 172.21.15.89:0/2248529797 >> v2:172.21.15.192:3300/0 conn(0x7f3eb81580e0 msgr2=0x7f3eb8070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:50:46.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.763+0000 7f3ebdde3640 1 --2- 172.21.15.89:0/2248529797 >> v2:172.21.15.192:3300/0 conn(0x7f3eb81580e0 0x7f3eb8070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:46.765 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.763+0000 7f3eaffff640 1 -- 172.21.15.89:0/2248529797 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 816728795 0 0) 0x7f3ea4003530 con 0x7f3eb80c6540 2023-10-30T12:50:46.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.763+0000 7f3ebdde3640 1 -- 172.21.15.89:0/2248529797 >> v1:172.21.15.89:6789/0 conn(0x7f3eb80c6540 legacy=0x7f3eb8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:46.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.767+0000 7f3ebdde3640 1 -- 172.21.15.89:0/2248529797 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb80c6bb0 con 0x7f3eb815e700 2023-10-30T12:50:46.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.767+0000 7f3ebdde3640 1 --2- 172.21.15.89:0/2248529797 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3eb815e700 0x7f3eb806a5a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3eb0009d90 tx=0x7f3eb001c840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ec69c00873f031b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:46.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.767+0000 7f3eaffff640 1 -- 172.21.15.89:0/2248529797 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8001680 con 0x7f3eb815e700 2023-10-30T12:50:46.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.767+0000 7f3eaffff640 1 -- 172.21.15.89:0/2248529797 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb00a5640 con 0x7f3eb815e700 2023-10-30T12:50:46.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.767+0000 7f3eaffff640 1 -- 172.21.15.89:0/2248529797 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb00a59b0 con 0x7f3eb815e700 2023-10-30T12:50:46.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.767+0000 7f3ebf846640 1 -- 172.21.15.89:0/2248529797 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3eb815e700 msgr2=0x7f3eb806a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:46.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.767+0000 7f3ebf846640 1 --2- 172.21.15.89:0/2248529797 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3eb815e700 0x7f3eb806a5a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3eb0009d90 tx=0x7f3eb001c840 comp rx=0 tx=0).stop 2023-10-30T12:50:46.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.767+0000 7f3ebf846640 1 -- 172.21.15.89:0/2248529797 shutdown_connections 2023-10-30T12:50:46.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.767+0000 7f3ebf846640 1 --2- 172.21.15.89:0/2248529797 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3eb815e700 0x7f3eb806a5a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:46.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.767+0000 7f3ebf846640 1 --2- 172.21.15.89:0/2248529797 >> v2:172.21.15.192:3300/0 conn(0x7f3eb81580e0 0x7f3eb8070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:46.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.771+0000 7f3ebf846640 1 -- 172.21.15.89:0/2248529797 >> 172.21.15.89:0/2248529797 conn(0x7f3eb80c7910 msgr2=0x7f3eb80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:46.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.771+0000 7f3ebf846640 1 -- 172.21.15.89:0/2248529797 shutdown_connections 2023-10-30T12:50:46.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.771+0000 7f3ebf846640 1 -- 172.21.15.89:0/2248529797 wait complete. 2023-10-30T12:50:46.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.771+0000 7f3ebf846640 1 Processor -- start 2023-10-30T12:50:46.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.771+0000 7f3ebf846640 1 -- start start 2023-10-30T12:50:46.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.771+0000 7f3ebf846640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eb80c6540 0x7f3eb8155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:46.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.771+0000 7f3ebf846640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3eb81580e0 0x7f3eb8156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:46.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.771+0000 7f3ebf846640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3eb815e700 0x7f3eb81f6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:46.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.771+0000 7f3ebf846640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb8141cb0 con 0x7f3eb80c6540 2023-10-30T12:50:46.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.771+0000 7f3ebf846640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb81f5b90 con 0x7f3eb81580e0 2023-10-30T12:50:46.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.771+0000 7f3ebf846640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb8148300 con 0x7f3eb815e700 2023-10-30T12:50:46.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.771+0000 7f3ebd5e2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eb80c6540 0x7f3eb8155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:46.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.771+0000 7f3ebd5e2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eb80c6540 0x7f3eb8155bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39778/0 (socket says 172.21.15.89:39778) 2023-10-30T12:50:46.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.771+0000 7f3ebd5e2640 1 -- 172.21.15.89:0/2991955458 learned_addr learned my addr 172.21.15.89:0/2991955458 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:50:46.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.771+0000 7f3ebdde3640 1 --2- 172.21.15.89:0/2991955458 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3eb815e700 0x7f3eb81f6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:46.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.771+0000 7f3ebcde1640 1 --2- 172.21.15.89:0/2991955458 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3eb81580e0 0x7f3eb8156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:46.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.771+0000 7f3ebd5e2640 1 -- 172.21.15.89:0/2991955458 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3eb815e700 msgr2=0x7f3eb81f6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:46.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.771+0000 7f3ebd5e2640 1 --2- 172.21.15.89:0/2991955458 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3eb815e700 0x7f3eb81f6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:46.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.771+0000 7f3ebd5e2640 1 -- 172.21.15.89:0/2991955458 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3eb81580e0 msgr2=0x7f3eb8156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:46.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.771+0000 7f3ebd5e2640 1 --2- 172.21.15.89:0/2991955458 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3eb81580e0 0x7f3eb8156110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:46.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.771+0000 7f3ebd5e2640 1 -- 172.21.15.89:0/2991955458 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb0009a70 con 0x7f3eb80c6540 2023-10-30T12:50:46.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.771+0000 7f3ebcde1640 1 --2- 172.21.15.89:0/2991955458 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3eb81580e0 0x7f3eb8156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:50:46.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.771+0000 7f3ebd5e2640 1 --2- 172.21.15.89:0/2991955458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eb80c6540 0x7f3eb8155bd0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f3eb400e9b0 tx=0x7f3eb400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:46.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.775+0000 7f3eadffb640 1 -- 172.21.15.89:0/2991955458 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb40096b0 con 0x7f3eb80c6540 2023-10-30T12:50:46.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.775+0000 7f3eadffb640 1 -- 172.21.15.89:0/2991955458 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb400c9a0 con 0x7f3eb80c6540 2023-10-30T12:50:46.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.775+0000 7f3eadffb640 1 -- 172.21.15.89:0/2991955458 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb400cd30 con 0x7f3eb80c6540 2023-10-30T12:50:46.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.775+0000 7f3ebf846640 1 -- 172.21.15.89:0/2991955458 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb81485e0 con 0x7f3eb80c6540 2023-10-30T12:50:46.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.775+0000 7f3ebf846640 1 -- 172.21.15.89:0/2991955458 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb8148ba0 con 0x7f3eb80c6540 2023-10-30T12:50:46.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.775+0000 7f3ebf846640 1 -- 172.21.15.89:0/2991955458 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb804e170 con 0x7f3eb80c6540 2023-10-30T12:50:46.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.775+0000 7f3eadffb640 1 -- 172.21.15.89:0/2991955458 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3eb4013470 con 0x7f3eb80c6540 2023-10-30T12:50:46.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.775+0000 7f3eadffb640 1 --2- 172.21.15.89:0/2991955458 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3e9c0664d0 0x7f3e9c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:46.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.775+0000 7f3eadffb640 1 -- 172.21.15.89:0/2991955458 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f3eb4084510 con 0x7f3eb80c6540 2023-10-30T12:50:46.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.775+0000 7f3ebcde1640 1 --2- 172.21.15.89:0/2991955458 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3e9c0664d0 0x7f3e9c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:46.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.779+0000 7f3ebcde1640 1 --2- 172.21.15.89:0/2991955458 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3e9c0664d0 0x7f3e9c068990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8146e60 tx=0x7f3ea8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:46.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.779+0000 7f3eadffb640 1 -- 172.21.15.89:0/2991955458 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3eb4051590 con 0x7f3eb80c6540 2023-10-30T12:50:46.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.967+0000 7f3ebf846640 1 -- 172.21.15.89:0/2991955458 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3eb806aed0 con 0x7f3eb80c6540 2023-10-30T12:50:46.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.967+0000 7f3eadffb640 1 -- 172.21.15.89:0/2991955458 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+5746 (secure 0 0 0) 0x7f3eb4054bb0 con 0x7f3eb80c6540 2023-10-30T12:50:46.967 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:50:46.967 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:50:41.318621+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6271,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6271":{"gid":6271,"name":"b","rank":0,"incarnation":110,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6835/659375964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":659375964},{"type":"v1","addr":"172.21.15.192:6835","nonce":659375964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:50:46.967 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 110 2023-10-30T12:50:46.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.971+0000 7f3ebf846640 1 -- 172.21.15.89:0/2991955458 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3e9c0664d0 msgr2=0x7f3e9c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:46.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.971+0000 7f3ebf846640 1 --2- 172.21.15.89:0/2991955458 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3e9c0664d0 0x7f3e9c068990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8146e60 tx=0x7f3ea8009040 comp rx=0 tx=0).stop 2023-10-30T12:50:46.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.971+0000 7f3ebf846640 1 -- 172.21.15.89:0/2991955458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eb80c6540 msgr2=0x7f3eb8155bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:46.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.971+0000 7f3ebf846640 1 --2- 172.21.15.89:0/2991955458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eb80c6540 0x7f3eb8155bd0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f3eb400e9b0 tx=0x7f3eb400ee70 comp rx=0 tx=0).stop 2023-10-30T12:50:46.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.971+0000 7f3ebf846640 1 -- 172.21.15.89:0/2991955458 shutdown_connections 2023-10-30T12:50:46.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.971+0000 7f3ebf846640 1 --2- 172.21.15.89:0/2991955458 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3e9c0664d0 0x7f3e9c068990 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:46.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.971+0000 7f3ebf846640 1 --2- 172.21.15.89:0/2991955458 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3eb81580e0 0x7f3eb8156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:46.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.971+0000 7f3ebf846640 1 --2- 172.21.15.89:0/2991955458 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3eb815e700 0x7f3eb81f6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:46.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.971+0000 7f3ebf846640 1 --2- 172.21.15.89:0/2991955458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3eb80c6540 0x7f3eb8155bd0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:46.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.971+0000 7f3ebf846640 1 -- 172.21.15.89:0/2991955458 >> 172.21.15.89:0/2991955458 conn(0x7f3eb80c7910 msgr2=0x7f3eb814a480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:46.970 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.971+0000 7f3ebf846640 1 -- 172.21.15.89:0/2991955458 shutdown_connections 2023-10-30T12:50:46.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:46.971+0000 7f3ebf846640 1 -- 172.21.15.89:0/2991955458 wait complete. 2023-10-30T12:50:46.987 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:50:48.508 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:50:48.512 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:50:48.988 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:50:49.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.135+0000 7f20027c5640 1 Processor -- start 2023-10-30T12:50:49.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.139+0000 7f20027c5640 1 -- start start 2023-10-30T12:50:49.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.139+0000 7f20027c5640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1ffc0c6bd0 0x7f1ffc06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:49.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.139+0000 7f20027c5640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1ffc071110 0x7f1ffc158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:49.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.139+0000 7f20027c5640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ffc066cf0 con 0x7f1ffc06f7a0 2023-10-30T12:50:49.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.139+0000 7f20027c5640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ffc0700c0 con 0x7f1ffc0c6bd0 2023-10-30T12:50:49.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.139+0000 7f20027c5640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ffc0c70d0 con 0x7f1ffc071110 2023-10-30T12:50:49.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.139+0000 7f1ffb7fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1ffc0c6bd0 0x7f1ffc06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:49.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.139+0000 7f1ffb7fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1ffc0c6bd0 0x7f1ffc06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39790/0 (socket says 172.21.15.89:39790) 2023-10-30T12:50:49.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.139+0000 7f1ffb7fe640 1 -- 172.21.15.89:0/374964808 learned_addr learned my addr 172.21.15.89:0/374964808 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:50:49.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.139+0000 7f2000d62640 1 --2- 172.21.15.89:0/374964808 >> v2:172.21.15.192:3300/0 conn(0x7f1ffc071110 0x7f1ffc158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:49.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.139+0000 7f1ffb7fe640 1 -- 172.21.15.89:0/374964808 >> v2:172.21.15.192:3300/0 conn(0x7f1ffc071110 msgr2=0x7f1ffc158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:50:49.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.139+0000 7f1ffb7fe640 1 --2- 172.21.15.89:0/374964808 >> v2:172.21.15.192:3300/0 conn(0x7f1ffc071110 0x7f1ffc158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:49.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.139+0000 7f1ffb7fe640 1 -- 172.21.15.89:0/374964808 >> v1:172.21.15.192:6789/0 conn(0x7f1ffc06f7a0 legacy=0x7f1ffc069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:49.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.139+0000 7f1ffb7fe640 1 -- 172.21.15.89:0/374964808 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ffc0c7390 con 0x7f1ffc0c6bd0 2023-10-30T12:50:49.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.143+0000 7f1ffb7fe640 1 --2- 172.21.15.89:0/374964808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ffc0c6bd0 0x7f1ffc06fb80 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc06cdd0 tx=0x7f1ff001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20eb1877517a9c76 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:49.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.143+0000 7f1ffaffd640 1 -- 172.21.15.89:0/374964808 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff00a89f0 con 0x7f1ffc0c6bd0 2023-10-30T12:50:49.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.143+0000 7f1ffaffd640 1 -- 172.21.15.89:0/374964808 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff00a8b50 con 0x7f1ffc0c6bd0 2023-10-30T12:50:49.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.143+0000 7f1ffaffd640 1 -- 172.21.15.89:0/374964808 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff00b0440 con 0x7f1ffc0c6bd0 2023-10-30T12:50:49.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.143+0000 7f20027c5640 1 -- 172.21.15.89:0/374964808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ffc0c6bd0 msgr2=0x7f1ffc06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:49.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.143+0000 7f20027c5640 1 --2- 172.21.15.89:0/374964808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ffc0c6bd0 0x7f1ffc06fb80 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc06cdd0 tx=0x7f1ff001ecf0 comp rx=0 tx=0).stop 2023-10-30T12:50:49.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.143+0000 7f20027c5640 1 -- 172.21.15.89:0/374964808 shutdown_connections 2023-10-30T12:50:49.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.143+0000 7f20027c5640 1 --2- 172.21.15.89:0/374964808 >> v2:172.21.15.192:3300/0 conn(0x7f1ffc071110 0x7f1ffc158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:49.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.143+0000 7f20027c5640 1 --2- 172.21.15.89:0/374964808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ffc0c6bd0 0x7f1ffc06fb80 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:49.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.143+0000 7f20027c5640 1 -- 172.21.15.89:0/374964808 >> 172.21.15.89:0/374964808 conn(0x7f1ffc0c7910 msgr2=0x7f1ffc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:49.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.143+0000 7f20027c5640 1 -- 172.21.15.89:0/374964808 shutdown_connections 2023-10-30T12:50:49.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.143+0000 7f20027c5640 1 -- 172.21.15.89:0/374964808 wait complete. 2023-10-30T12:50:49.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.143+0000 7f20027c5640 1 Processor -- start 2023-10-30T12:50:49.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.143+0000 7f20027c5640 1 -- start start 2023-10-30T12:50:49.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.147+0000 7f20027c5640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ffc06f7a0 0x7f1ffc0c1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:49.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.147+0000 7f20027c5640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ffc071110 0x7f1ffc156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:49.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.147+0000 7f20027c5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ffc0c6bd0 0x7f1ffc154180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:49.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.147+0000 7f20027c5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc14f600 con 0x7f1ffc0c6bd0 2023-10-30T12:50:49.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.147+0000 7f20027c5640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc1ece80 con 0x7f1ffc06f7a0 2023-10-30T12:50:49.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.147+0000 7f20027c5640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc1562e0 con 0x7f1ffc071110 2023-10-30T12:50:49.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.147+0000 7f1ffbfff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ffc06f7a0 0x7f1ffc0c1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:49.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.147+0000 7f1ffbfff640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ffc06f7a0 0x7f1ffc0c1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:47832/0 (socket says 172.21.15.89:47832) 2023-10-30T12:50:49.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.147+0000 7f2000d62640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ffc0c6bd0 0x7f1ffc154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:49.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.147+0000 7f1ffbfff640 1 -- 172.21.15.89:0/1478201095 learned_addr learned my addr 172.21.15.89:0/1478201095 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:50:49.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.147+0000 7f1ffb7fe640 1 --2- 172.21.15.89:0/1478201095 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ffc071110 0x7f1ffc156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:49.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.147+0000 7f1ffbfff640 1 -- 172.21.15.89:0/1478201095 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ffc071110 msgr2=0x7f1ffc156770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:49.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.147+0000 7f1ffbfff640 1 --2- 172.21.15.89:0/1478201095 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ffc071110 0x7f1ffc156770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:49.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.147+0000 7f1ffbfff640 1 -- 172.21.15.89:0/1478201095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ffc0c6bd0 msgr2=0x7f1ffc154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:49.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.147+0000 7f1ffbfff640 1 --2- 172.21.15.89:0/1478201095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ffc0c6bd0 0x7f1ffc154180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:49.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.147+0000 7f1ffbfff640 1 -- 172.21.15.89:0/1478201095 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff00094d0 con 0x7f1ffc06f7a0 2023-10-30T12:50:49.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.147+0000 7f1ffbfff640 1 --2- 172.21.15.89:0/1478201095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ffc06f7a0 0x7f1ffc0c1170 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00c3f0 tx=0x7f1fec00c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:49.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.147+0000 7f1ff8ff9640 1 -- 172.21.15.89:0/1478201095 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec0052d0 con 0x7f1ffc06f7a0 2023-10-30T12:50:49.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.147+0000 7f20027c5640 1 -- 172.21.15.89:0/1478201095 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ffc1efa50 con 0x7f1ffc06f7a0 2023-10-30T12:50:49.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.147+0000 7f1ff8ff9640 1 -- 172.21.15.89:0/1478201095 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fec005430 con 0x7f1ffc06f7a0 2023-10-30T12:50:49.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.147+0000 7f1ff8ff9640 1 -- 172.21.15.89:0/1478201095 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec003840 con 0x7f1ffc06f7a0 2023-10-30T12:50:49.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.147+0000 7f20027c5640 1 -- 172.21.15.89:0/1478201095 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ffc1559b0 con 0x7f1ffc06f7a0 2023-10-30T12:50:49.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.147+0000 7f20027c5640 1 -- 172.21.15.89:0/1478201095 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ffc04e170 con 0x7f1ffc06f7a0 2023-10-30T12:50:49.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.151+0000 7f1ff8ff9640 1 -- 172.21.15.89:0/1478201095 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1fec0039a0 con 0x7f1ffc06f7a0 2023-10-30T12:50:49.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.151+0000 7f1ff8ff9640 1 --2- 172.21.15.89:0/1478201095 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1fe80664a0 0x7f1fe8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:49.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.151+0000 7f1ff8ff9640 1 -- 172.21.15.89:0/1478201095 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f1fec083a60 con 0x7f1ffc06f7a0 2023-10-30T12:50:49.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.151+0000 7f1ffb7fe640 1 --2- 172.21.15.89:0/1478201095 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1fe80664a0 0x7f1fe8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:49.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.151+0000 7f1ffb7fe640 1 --2- 172.21.15.89:0/1478201095 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1fe80664a0 0x7f1fe8068960 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1ff00b2000 tx=0x7f1ff00aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:49.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.163+0000 7f1ff8ff9640 1 -- 172.21.15.89:0/1478201095 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1fec050ae0 con 0x7f1ffc06f7a0 2023-10-30T12:50:49.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.343+0000 7f20027c5640 1 -- 172.21.15.89:0/1478201095 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ffc0c4990 con 0x7f1ffc06f7a0 2023-10-30T12:50:49.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.343+0000 7f1ff8ff9640 1 -- 172.21.15.89:0/1478201095 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+5746 (secure 0 0 0) 0x7f1fec054100 con 0x7f1ffc06f7a0 2023-10-30T12:50:49.343 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:50:49.343 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:50:41.318621+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6271,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6271":{"gid":6271,"name":"b","rank":0,"incarnation":110,"state":"up:replay","state_seq":1,"addr":"172.21.15.192:6835/659375964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":659375964},{"type":"v1","addr":"172.21.15.192:6835","nonce":659375964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:50:49.343 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 110 2023-10-30T12:50:49.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.347+0000 7f20027c5640 1 -- 172.21.15.89:0/1478201095 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1fe80664a0 msgr2=0x7f1fe8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:49.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.347+0000 7f20027c5640 1 --2- 172.21.15.89:0/1478201095 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1fe80664a0 0x7f1fe8068960 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1ff00b2000 tx=0x7f1ff00aa070 comp rx=0 tx=0).stop 2023-10-30T12:50:49.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.347+0000 7f20027c5640 1 -- 172.21.15.89:0/1478201095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ffc06f7a0 msgr2=0x7f1ffc0c1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:49.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.347+0000 7f20027c5640 1 --2- 172.21.15.89:0/1478201095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ffc06f7a0 0x7f1ffc0c1170 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00c3f0 tx=0x7f1fec00c8b0 comp rx=0 tx=0).stop 2023-10-30T12:50:49.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.347+0000 7f20027c5640 1 -- 172.21.15.89:0/1478201095 shutdown_connections 2023-10-30T12:50:49.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.347+0000 7f20027c5640 1 --2- 172.21.15.89:0/1478201095 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1fe80664a0 0x7f1fe8068960 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:49.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.347+0000 7f20027c5640 1 --2- 172.21.15.89:0/1478201095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ffc0c6bd0 0x7f1ffc154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:49.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.347+0000 7f20027c5640 1 --2- 172.21.15.89:0/1478201095 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1ffc071110 0x7f1ffc156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:49.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.347+0000 7f20027c5640 1 --2- 172.21.15.89:0/1478201095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ffc06f7a0 0x7f1ffc0c1170 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:49.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.347+0000 7f20027c5640 1 -- 172.21.15.89:0/1478201095 >> 172.21.15.89:0/1478201095 conn(0x7f1ffc0c7910 msgr2=0x7f1ffc0c2640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:49.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.347+0000 7f20027c5640 1 -- 172.21.15.89:0/1478201095 shutdown_connections 2023-10-30T12:50:49.347 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:49.347+0000 7f20027c5640 1 -- 172.21.15.89:0/1478201095 wait complete. 2023-10-30T12:50:49.363 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:50:51.364 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:50:51.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.511+0000 7fcb0bcea640 1 Processor -- start 2023-10-30T12:50:51.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.511+0000 7fcb0bcea640 1 -- start start 2023-10-30T12:50:51.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.511+0000 7fcb0bcea640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb040cb550 0x7fcb040cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:51.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.511+0000 7fcb0bcea640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcb04071250 0x7fcb040716a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:51.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.511+0000 7fcb0bcea640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb04071be0 con 0x7fcb0414fd50 2023-10-30T12:50:51.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.511+0000 7fcb0bcea640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb041560a0 con 0x7fcb04071250 2023-10-30T12:50:51.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.511+0000 7fcb0bcea640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb04156380 con 0x7fcb040cb550 2023-10-30T12:50:51.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.515+0000 7fcb0a287640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcb04071250 0x7fcb040716a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:51.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.515+0000 7fcb0a287640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fcb04071250 0x7fcb040716a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46602/0 (socket says 172.21.15.89:46602) 2023-10-30T12:50:51.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.515+0000 7fcb0a287640 1 -- 172.21.15.89:0/3342636529 learned_addr learned my addr 172.21.15.89:0/3342636529 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:50:51.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.515+0000 7fcb09285640 1 --2- 172.21.15.89:0/3342636529 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb040cb550 0x7fcb040cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:51.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.515+0000 7fcb0a287640 1 -- 172.21.15.89:0/3342636529 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb040cb550 msgr2=0x7fcb040cb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:50:51.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.515+0000 7fcb0a287640 1 --2- 172.21.15.89:0/3342636529 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb040cb550 0x7fcb040cb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:51.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.515+0000 7fcb0a287640 1 -- 172.21.15.89:0/3342636529 >> v1:172.21.15.192:6789/0 conn(0x7fcb0414fd50 legacy=0x7fcb040c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:51.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.515+0000 7fcb0a287640 1 -- 172.21.15.89:0/3342636529 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb04156660 con 0x7fcb04071250 2023-10-30T12:50:51.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.515+0000 7fcb0a287640 1 --2- 172.21.15.89:0/3342636529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb04071250 0x7fcb040716a0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fcafc009e50 tx=0x7fcafc01c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2b3cf95ab399bb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:51.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.515+0000 7fcb08a84640 1 -- 172.21.15.89:0/3342636529 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcafc0a5870 con 0x7fcb04071250 2023-10-30T12:50:51.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.515+0000 7fcb08a84640 1 -- 172.21.15.89:0/3342636529 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcafc0a59d0 con 0x7fcb04071250 2023-10-30T12:50:51.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.515+0000 7fcb08a84640 1 -- 172.21.15.89:0/3342636529 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcafc0a5d60 con 0x7fcb04071250 2023-10-30T12:50:51.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.519+0000 7fcb0bcea640 1 -- 172.21.15.89:0/3342636529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb04071250 msgr2=0x7fcb040716a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:51.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.519+0000 7fcb0bcea640 1 --2- 172.21.15.89:0/3342636529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb04071250 0x7fcb040716a0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fcafc009e50 tx=0x7fcafc01c9e0 comp rx=0 tx=0).stop 2023-10-30T12:50:51.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.519+0000 7fcb0bcea640 1 -- 172.21.15.89:0/3342636529 shutdown_connections 2023-10-30T12:50:51.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.519+0000 7fcb0bcea640 1 --2- 172.21.15.89:0/3342636529 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb040cb550 0x7fcb040cb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:51.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.519+0000 7fcb0bcea640 1 --2- 172.21.15.89:0/3342636529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb04071250 0x7fcb040716a0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:51.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.519+0000 7fcb0bcea640 1 -- 172.21.15.89:0/3342636529 >> 172.21.15.89:0/3342636529 conn(0x7fcb0406c6e0 msgr2=0x7fcb04069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:51.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.519+0000 7fcb0bcea640 1 -- 172.21.15.89:0/3342636529 shutdown_connections 2023-10-30T12:50:51.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.519+0000 7fcb0bcea640 1 -- 172.21.15.89:0/3342636529 wait complete. 2023-10-30T12:50:51.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.519+0000 7fcb0bcea640 1 Processor -- start 2023-10-30T12:50:51.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.519+0000 7fcb0bcea640 1 -- start start 2023-10-30T12:50:51.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.519+0000 7fcb0bcea640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb04071250 0x7fcb040c3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:51.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.519+0000 7fcb0bcea640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb040cb550 0x7fcb04162f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:51.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.519+0000 7fcb0bcea640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb0414fd50 0x7fcb04163470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:51.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.519+0000 7fcb0bcea640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb041f55d0 con 0x7fcb0414fd50 2023-10-30T12:50:51.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.519+0000 7fcb0bcea640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb041f5870 con 0x7fcb040cb550 2023-10-30T12:50:51.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.519+0000 7fcb0bcea640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb041f5b50 con 0x7fcb04071250 2023-10-30T12:50:51.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.519+0000 7fcb0a287640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb0414fd50 0x7fcb04163470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:51.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.519+0000 7fcb0a287640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb0414fd50 0x7fcb04163470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46610/0 (socket says 172.21.15.89:46610) 2023-10-30T12:50:51.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.519+0000 7fcb09285640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb040cb550 0x7fcb04162f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:51.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.519+0000 7fcb0a287640 1 -- 172.21.15.89:0/2452110005 learned_addr learned my addr 172.21.15.89:0/2452110005 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:50:51.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.519+0000 7fcb09a86640 1 --2- 172.21.15.89:0/2452110005 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb04071250 0x7fcb040c3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:51.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.523+0000 7fcb0a287640 1 -- 172.21.15.89:0/2452110005 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb04071250 msgr2=0x7fcb040c3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:51.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.523+0000 7fcb0a287640 1 --2- 172.21.15.89:0/2452110005 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb04071250 0x7fcb040c3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:51.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.523+0000 7fcb0a287640 1 -- 172.21.15.89:0/2452110005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb040cb550 msgr2=0x7fcb04162f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:51.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.523+0000 7fcb0a287640 1 --2- 172.21.15.89:0/2452110005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb040cb550 0x7fcb04162f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:51.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.523+0000 7fcb0a287640 1 -- 172.21.15.89:0/2452110005 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcafc009b30 con 0x7fcb0414fd50 2023-10-30T12:50:51.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.523+0000 7fcb0a287640 1 --2- 172.21.15.89:0/2452110005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb0414fd50 0x7fcb04163470 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fcafc01a610 tx=0x7fcafc019780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:51.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.523+0000 7fcafa7fc640 1 -- 172.21.15.89:0/2452110005 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcafc0b8670 con 0x7fcb0414fd50 2023-10-30T12:50:51.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.523+0000 7fcafa7fc640 1 -- 172.21.15.89:0/2452110005 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcafc0b87d0 con 0x7fcb0414fd50 2023-10-30T12:50:51.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.523+0000 7fcafa7fc640 1 -- 172.21.15.89:0/2452110005 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcafc0b8b80 con 0x7fcb0414fd50 2023-10-30T12:50:51.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.523+0000 7fcb0bcea640 1 -- 172.21.15.89:0/2452110005 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb041f5e30 con 0x7fcb0414fd50 2023-10-30T12:50:51.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.523+0000 7fcb0bcea640 1 -- 172.21.15.89:0/2452110005 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb040c40f0 con 0x7fcb0414fd50 2023-10-30T12:50:51.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.523+0000 7fcb0bcea640 1 -- 172.21.15.89:0/2452110005 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb041f6350 con 0x7fcb0414fd50 2023-10-30T12:50:51.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.527+0000 7fcafa7fc640 1 -- 172.21.15.89:0/2452110005 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcafc0b8ce0 con 0x7fcb0414fd50 2023-10-30T12:50:51.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.527+0000 7fcafa7fc640 1 --2- 172.21.15.89:0/2452110005 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fcae80664a0 0x7fcae8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:51.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.527+0000 7fcafa7fc640 1 -- 172.21.15.89:0/2452110005 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7fcafc11f850 con 0x7fcb0414fd50 2023-10-30T12:50:51.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.527+0000 7fcb09a86640 1 --2- 172.21.15.89:0/2452110005 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fcae80664a0 0x7fcae8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:51.527 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.527+0000 7fcb09a86640 1 --2- 172.21.15.89:0/2452110005 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fcae80664a0 0x7fcae8068960 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fcb00007fd0 tx=0x7fcb00009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:51.528 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.531+0000 7fcafa7fc640 1 -- 172.21.15.89:0/2452110005 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcb041f6350 con 0x7fcb0414fd50 2023-10-30T12:50:51.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.711+0000 7fcb0bcea640 1 -- 172.21.15.89:0/2452110005 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb041f6350 con 0x7fcb0414fd50 2023-10-30T12:50:51.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.711+0000 7fcafa7fc640 1 -- 172.21.15.89:0/2452110005 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+5748 (secure 0 0 0) 0x7fcafc0f03f0 con 0x7fcb0414fd50 2023-10-30T12:50:51.711 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:50:51.711 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:50:49.582176+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6271,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6271":{"gid":6271,"name":"b","rank":0,"incarnation":110,"state":"up:resolve","state_seq":21,"addr":"172.21.15.192:6835/659375964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":659375964},{"type":"v1","addr":"172.21.15.192:6835","nonce":659375964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:50:51.712 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 111 2023-10-30T12:50:51.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.715+0000 7fcb0bcea640 1 -- 172.21.15.89:0/2452110005 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fcae80664a0 msgr2=0x7fcae8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:51.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.715+0000 7fcb0bcea640 1 --2- 172.21.15.89:0/2452110005 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fcae80664a0 0x7fcae8068960 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fcb00007fd0 tx=0x7fcb00009040 comp rx=0 tx=0).stop 2023-10-30T12:50:51.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.715+0000 7fcb0bcea640 1 -- 172.21.15.89:0/2452110005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb0414fd50 msgr2=0x7fcb04163470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:51.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.715+0000 7fcb0bcea640 1 --2- 172.21.15.89:0/2452110005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb0414fd50 0x7fcb04163470 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fcafc01a610 tx=0x7fcafc019780 comp rx=0 tx=0).stop 2023-10-30T12:50:51.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.715+0000 7fcb0bcea640 1 -- 172.21.15.89:0/2452110005 shutdown_connections 2023-10-30T12:50:51.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.715+0000 7fcb0bcea640 1 --2- 172.21.15.89:0/2452110005 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fcae80664a0 0x7fcae8068960 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:51.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.715+0000 7fcb0bcea640 1 --2- 172.21.15.89:0/2452110005 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcb040cb550 0x7fcb04162f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:51.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.715+0000 7fcb0bcea640 1 --2- 172.21.15.89:0/2452110005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb0414fd50 0x7fcb04163470 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:51.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.715+0000 7fcb0bcea640 1 --2- 172.21.15.89:0/2452110005 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcb04071250 0x7fcb040c3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:51.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.715+0000 7fcb0bcea640 1 -- 172.21.15.89:0/2452110005 >> 172.21.15.89:0/2452110005 conn(0x7fcb0406c6e0 msgr2=0x7fcb0415f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:51.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.715+0000 7fcb0bcea640 1 -- 172.21.15.89:0/2452110005 shutdown_connections 2023-10-30T12:50:51.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:51.715+0000 7fcb0bcea640 1 -- 172.21.15.89:0/2452110005 wait complete. 2023-10-30T12:50:51.731 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:50:53.732 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:50:53.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.883+0000 7faaf2e79640 1 Processor -- start 2023-10-30T12:50:53.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.883+0000 7faaf2e79640 1 -- start start 2023-10-30T12:50:53.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.883+0000 7faaf2e79640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7faaec1538a0 0x7faaec15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:53.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.883+0000 7faaf2e79640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7faaec0cf160 0x7faaec0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:53.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.883+0000 7faaf2e79640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaec0cb800 con 0x7faaec0ca0e0 2023-10-30T12:50:53.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.883+0000 7faaf2e79640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaec15dc30 con 0x7faaec1538a0 2023-10-30T12:50:53.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.883+0000 7faaf2e79640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaec0cb330 con 0x7faaec0cf160 2023-10-30T12:50:53.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.883+0000 7faaf0c15640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7faaec1538a0 0x7faaec15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:53.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.883+0000 7faaf0c15640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7faaec1538a0 0x7faaec15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46626/0 (socket says 172.21.15.89:46626) 2023-10-30T12:50:53.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.883+0000 7faaf0c15640 1 -- 172.21.15.89:0/365135097 learned_addr learned my addr 172.21.15.89:0/365135097 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:50:53.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.883+0000 7faaebfff640 1 --1- 172.21.15.89:0/365135097 >> v1:172.21.15.89:6789/0 conn(0x7faaec0ca0e0 0x7faaec15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:34708/0 (socket says 172.21.15.89:34708) 2023-10-30T12:50:53.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.883+0000 7faaf1416640 1 --2- 172.21.15.89:0/365135097 >> v2:172.21.15.192:3300/0 conn(0x7faaec0cf160 0x7faaec0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:53.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.883+0000 7faaf0c15640 1 -- 172.21.15.89:0/365135097 >> v2:172.21.15.192:3300/0 conn(0x7faaec0cf160 msgr2=0x7faaec0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:50:53.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.883+0000 7faaf0c15640 1 --2- 172.21.15.89:0/365135097 >> v2:172.21.15.192:3300/0 conn(0x7faaec0cf160 0x7faaec0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:53.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.883+0000 7faaf0c15640 1 -- 172.21.15.89:0/365135097 >> v1:172.21.15.89:6789/0 conn(0x7faaec0ca0e0 legacy=0x7faaec15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:53.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.883+0000 7faaf0c15640 1 -- 172.21.15.89:0/365135097 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaec0cad70 con 0x7faaec1538a0 2023-10-30T12:50:53.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.883+0000 7faaeb7fe640 1 -- 172.21.15.89:0/365135097 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2248046704 0 0) 0x7faaec0cb800 con 0x7faaec0ca0e0 2023-10-30T12:50:53.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.887+0000 7faaf0c15640 1 --2- 172.21.15.89:0/365135097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaec1538a0 0x7faaec15d6f0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7faaec0b5330 tx=0x7faae001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3de039c1a05fbad9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:53.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.887+0000 7faaeb7fe640 1 -- 172.21.15.89:0/365135097 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae00a89f0 con 0x7faaec1538a0 2023-10-30T12:50:53.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.887+0000 7faaeb7fe640 1 -- 172.21.15.89:0/365135097 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae00a8b50 con 0x7faaec1538a0 2023-10-30T12:50:53.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.887+0000 7faaeb7fe640 1 -- 172.21.15.89:0/365135097 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae00b0440 con 0x7faaec1538a0 2023-10-30T12:50:53.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.887+0000 7faaf2e79640 1 -- 172.21.15.89:0/365135097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaec1538a0 msgr2=0x7faaec15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:53.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.887+0000 7faaf2e79640 1 --2- 172.21.15.89:0/365135097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaec1538a0 0x7faaec15d6f0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7faaec0b5330 tx=0x7faae001ecf0 comp rx=0 tx=0).stop 2023-10-30T12:50:53.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.887+0000 7faaf2e79640 1 -- 172.21.15.89:0/365135097 shutdown_connections 2023-10-30T12:50:53.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.887+0000 7faaf2e79640 1 --2- 172.21.15.89:0/365135097 >> v2:172.21.15.192:3300/0 conn(0x7faaec0cf160 0x7faaec0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:53.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.887+0000 7faaf2e79640 1 --2- 172.21.15.89:0/365135097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaec1538a0 0x7faaec15d6f0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:53.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.887+0000 7faaf2e79640 1 -- 172.21.15.89:0/365135097 >> 172.21.15.89:0/365135097 conn(0x7faaec06bfe0 msgr2=0x7faaec06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:53.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.887+0000 7faaf2e79640 1 -- 172.21.15.89:0/365135097 shutdown_connections 2023-10-30T12:50:53.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.887+0000 7faaf2e79640 1 -- 172.21.15.89:0/365135097 wait complete. 2023-10-30T12:50:53.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.891+0000 7faaf2e79640 1 Processor -- start 2023-10-30T12:50:53.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.891+0000 7faaf2e79640 1 -- start start 2023-10-30T12:50:53.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.891+0000 7faaf2e79640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaec0ca0e0 0x7faaec15b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:53.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.891+0000 7faaf2e79640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faaec0cf160 0x7faaec0b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:53.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.891+0000 7faaf2e79640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faaec1538a0 0x7faaec1ef940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:53.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.891+0000 7faaf2e79640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaec14ecd0 con 0x7faaec0ca0e0 2023-10-30T12:50:53.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.891+0000 7faaf2e79640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaec14e750 con 0x7faaec1538a0 2023-10-30T12:50:53.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.891+0000 7faaf2e79640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaec1f3380 con 0x7faaec0cf160 2023-10-30T12:50:53.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.891+0000 7faaf0c15640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaec0ca0e0 0x7faaec15b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:53.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.891+0000 7faaf0c15640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaec0ca0e0 0x7faaec15b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46634/0 (socket says 172.21.15.89:46634) 2023-10-30T12:50:53.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.891+0000 7faaf0c15640 1 -- 172.21.15.89:0/1745800725 learned_addr learned my addr 172.21.15.89:0/1745800725 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:50:53.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.891+0000 7faaf1416640 1 --2- 172.21.15.89:0/1745800725 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faaec1538a0 0x7faaec1ef940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:53.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.891+0000 7faaebfff640 1 --2- 172.21.15.89:0/1745800725 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faaec0cf160 0x7faaec0b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:53.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.891+0000 7faaf0c15640 1 -- 172.21.15.89:0/1745800725 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faaec0cf160 msgr2=0x7faaec0b4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:53.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.891+0000 7faaf0c15640 1 --2- 172.21.15.89:0/1745800725 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faaec0cf160 0x7faaec0b4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:53.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.891+0000 7faaf0c15640 1 -- 172.21.15.89:0/1745800725 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faaec1538a0 msgr2=0x7faaec1ef940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:53.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.891+0000 7faaf0c15640 1 --2- 172.21.15.89:0/1745800725 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faaec1538a0 0x7faaec1ef940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:53.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.891+0000 7faaf0c15640 1 -- 172.21.15.89:0/1745800725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae00094d0 con 0x7faaec0ca0e0 2023-10-30T12:50:53.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.891+0000 7faaebfff640 1 --2- 172.21.15.89:0/1745800725 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faaec0cf160 0x7faaec0b4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:50:53.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.891+0000 7faaf0c15640 1 --2- 172.21.15.89:0/1745800725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaec0ca0e0 0x7faaec15b060 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7faae001b900 tx=0x7faae001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:53.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.891+0000 7faae97fa640 1 -- 172.21.15.89:0/1745800725 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae00ba600 con 0x7faaec0ca0e0 2023-10-30T12:50:53.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.891+0000 7faae97fa640 1 -- 172.21.15.89:0/1745800725 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae00ba760 con 0x7faaec0ca0e0 2023-10-30T12:50:53.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.891+0000 7faae97fa640 1 -- 172.21.15.89:0/1745800725 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae00bab10 con 0x7faaec0ca0e0 2023-10-30T12:50:53.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.895+0000 7faaf2e79640 1 -- 172.21.15.89:0/1745800725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaec14e9f0 con 0x7faaec0ca0e0 2023-10-30T12:50:53.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.895+0000 7faaf2e79640 1 -- 172.21.15.89:0/1745800725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaec0b1b60 con 0x7faaec0ca0e0 2023-10-30T12:50:53.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.895+0000 7faaf2e79640 1 -- 172.21.15.89:0/1745800725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaec0b2a50 con 0x7faaec0ca0e0 2023-10-30T12:50:53.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.895+0000 7faae97fa640 1 -- 172.21.15.89:0/1745800725 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faae00bac70 con 0x7faaec0ca0e0 2023-10-30T12:50:53.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.895+0000 7faae97fa640 1 --2- 172.21.15.89:0/1745800725 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faad00664a0 0x7faad0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:53.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.895+0000 7faae97fa640 1 -- 172.21.15.89:0/1745800725 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7faae0122920 con 0x7faaec0ca0e0 2023-10-30T12:50:53.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.895+0000 7faaebfff640 1 --2- 172.21.15.89:0/1745800725 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faad00664a0 0x7faad0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:53.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.899+0000 7faaebfff640 1 --2- 172.21.15.89:0/1745800725 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faad00664a0 0x7faad0068960 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7faad8007fd0 tx=0x7faad8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:53.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:53.899+0000 7faae97fa640 1 -- 172.21.15.89:0/1745800725 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7faae00ef8f0 con 0x7faaec0ca0e0 2023-10-30T12:50:54.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:54.079+0000 7faaf2e79640 1 -- 172.21.15.89:0/1745800725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faaec1ef280 con 0x7faaec0ca0e0 2023-10-30T12:50:54.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:54.079+0000 7faae97fa640 1 -- 172.21.15.89:0/1745800725 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+5748 (secure 0 0 0) 0x7faaec1ef280 con 0x7faaec0ca0e0 2023-10-30T12:50:54.079 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:50:54.079 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:50:49.582176+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6271,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6271":{"gid":6271,"name":"b","rank":0,"incarnation":110,"state":"up:resolve","state_seq":21,"addr":"172.21.15.192:6835/659375964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":659375964},{"type":"v1","addr":"172.21.15.192:6835","nonce":659375964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:50:54.080 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 111 2023-10-30T12:50:54.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:54.083+0000 7faaf2e79640 1 -- 172.21.15.89:0/1745800725 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faad00664a0 msgr2=0x7faad0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:54.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:54.083+0000 7faaf2e79640 1 --2- 172.21.15.89:0/1745800725 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faad00664a0 0x7faad0068960 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7faad8007fd0 tx=0x7faad8009040 comp rx=0 tx=0).stop 2023-10-30T12:50:54.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:54.083+0000 7faaf2e79640 1 -- 172.21.15.89:0/1745800725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaec0ca0e0 msgr2=0x7faaec15b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:54.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:54.083+0000 7faaf2e79640 1 --2- 172.21.15.89:0/1745800725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaec0ca0e0 0x7faaec15b060 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7faae001b900 tx=0x7faae001ba90 comp rx=0 tx=0).stop 2023-10-30T12:50:54.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:54.083+0000 7faaf2e79640 1 -- 172.21.15.89:0/1745800725 shutdown_connections 2023-10-30T12:50:54.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:54.083+0000 7faaf2e79640 1 --2- 172.21.15.89:0/1745800725 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faad00664a0 0x7faad0068960 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:54.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:54.083+0000 7faaf2e79640 1 --2- 172.21.15.89:0/1745800725 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faaec1538a0 0x7faaec1ef940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:54.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:54.083+0000 7faaf2e79640 1 --2- 172.21.15.89:0/1745800725 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faaec0cf160 0x7faaec0b4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:54.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:54.083+0000 7faaf2e79640 1 --2- 172.21.15.89:0/1745800725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaec0ca0e0 0x7faaec15b060 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:54.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:54.083+0000 7faaf2e79640 1 -- 172.21.15.89:0/1745800725 >> 172.21.15.89:0/1745800725 conn(0x7faaec06bfe0 msgr2=0x7faaec065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:54.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:54.083+0000 7faaf2e79640 1 -- 172.21.15.89:0/1745800725 shutdown_connections 2023-10-30T12:50:54.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:54.083+0000 7faaf2e79640 1 -- 172.21.15.89:0/1745800725 wait complete. 2023-10-30T12:50:54.100 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 111, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:50.729245+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 111, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5728": { "addr": "172.21.15.192:6839/3991891597", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6838", "nonce": 3991891597, "type": "v2" }, { "addr": "172.21.15.192:6839", "nonce": 3991891597, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 5728, "incarnation": 73, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 19 }, "gid_6075": { "addr": "172.21.15.89:6837/2412618858", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6836", "nonce": 2412618858, "type": "v2" }, { "addr": "172.21.15.89:6837", "nonce": 2412618858, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 6075, "incarnation": 92, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 38 }, "gid_6271": { "addr": "172.21.15.192:6835/659375964", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6834", "nonce": 659375964, "type": "v2" }, { "addr": "172.21.15.192:6835", "nonce": 659375964, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6271, "incarnation": 110, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:resolve", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:50:49.582176+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6271, "mds_1": 6075, "mds_2": 5728 } } } ], "standbys": [ { "addr": "172.21.15.89:6834/2909790173", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6832", "nonce": 2909790173, "type": "v2" }, { "addr": "172.21.15.89:6834", "nonce": 2909790173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.89:6835/2387357615", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6833", "nonce": 2387357615, "type": "v2" }, { "addr": "172.21.15.89:6835", "nonce": 2387357615, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4239, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:50:56.102 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:50:56.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.251+0000 7f683f59e640 1 Processor -- start 2023-10-30T12:50:56.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.251+0000 7f683f59e640 1 -- start start 2023-10-30T12:50:56.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.251+0000 7f683f59e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6840070450 0x7f68400c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:56.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.251+0000 7f683f59e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f684006ded0 0x7f684006e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:56.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.251+0000 7f683f59e640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6840071fe0 con 0x7f68401539a0 2023-10-30T12:50:56.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.251+0000 7f683f59e640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68400722c0 con 0x7f6840070450 2023-10-30T12:50:56.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.251+0000 7f683f59e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f684015cb00 con 0x7f684006ded0 2023-10-30T12:50:56.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.251+0000 7f6835d9b640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6840070450 0x7f68400c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:56.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.251+0000 7f6835d9b640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f6840070450 0x7f68400c65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46638/0 (socket says 172.21.15.89:46638) 2023-10-30T12:50:56.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.251+0000 7f6835d9b640 1 -- 172.21.15.89:0/3906187153 learned_addr learned my addr 172.21.15.89:0/3906187153 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:50:56.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.251+0000 7f683ed9d640 1 --2- 172.21.15.89:0/3906187153 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f684006ded0 0x7f684006e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:56.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.251+0000 7f6835d9b640 1 -- 172.21.15.89:0/3906187153 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f684006ded0 msgr2=0x7f684006e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:50:56.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.251+0000 7f6835d9b640 1 --2- 172.21.15.89:0/3906187153 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f684006ded0 0x7f684006e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:56.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.251+0000 7f6835d9b640 1 -- 172.21.15.89:0/3906187153 >> v1:172.21.15.89:6789/0 conn(0x7f68401539a0 legacy=0x7f68400717c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:56.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.251+0000 7f6835d9b640 1 -- 172.21.15.89:0/3906187153 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684015cde0 con 0x7f6840070450 2023-10-30T12:50:56.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.251+0000 7f683dd9b640 1 -- 172.21.15.89:0/3906187153 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804942420 0 0) 0x7f6840071fe0 con 0x7f68401539a0 2023-10-30T12:50:56.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.255+0000 7f6835d9b640 1 --2- 172.21.15.89:0/3906187153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6840070450 0x7f68400c65f0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f6828009870 tx=0x7f682801c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1293338fb1efaaaf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:56.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.255+0000 7f683dd9b640 1 -- 172.21.15.89:0/3906187153 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68280ac620 con 0x7f6840070450 2023-10-30T12:50:56.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.255+0000 7f683dd9b640 1 -- 172.21.15.89:0/3906187153 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68280ac780 con 0x7f6840070450 2023-10-30T12:50:56.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.255+0000 7f683dd9b640 1 -- 172.21.15.89:0/3906187153 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68280acb10 con 0x7f6840070450 2023-10-30T12:50:56.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.255+0000 7f683f59e640 1 -- 172.21.15.89:0/3906187153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6840070450 msgr2=0x7f68400c65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:56.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.255+0000 7f683f59e640 1 --2- 172.21.15.89:0/3906187153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6840070450 0x7f68400c65f0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f6828009870 tx=0x7f682801c4b0 comp rx=0 tx=0).stop 2023-10-30T12:50:56.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.255+0000 7f683f59e640 1 -- 172.21.15.89:0/3906187153 shutdown_connections 2023-10-30T12:50:56.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.255+0000 7f683f59e640 1 --2- 172.21.15.89:0/3906187153 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f684006ded0 0x7f684006e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:56.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.255+0000 7f683f59e640 1 --2- 172.21.15.89:0/3906187153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6840070450 0x7f68400c65f0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:56.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.255+0000 7f683f59e640 1 -- 172.21.15.89:0/3906187153 >> 172.21.15.89:0/3906187153 conn(0x7f68400cbe40 msgr2=0x7f68400cc250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:56.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.255+0000 7f683f59e640 1 -- 172.21.15.89:0/3906187153 shutdown_connections 2023-10-30T12:50:56.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.255+0000 7f683f59e640 1 -- 172.21.15.89:0/3906187153 wait complete. 2023-10-30T12:50:56.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.259+0000 7f683f59e640 1 Processor -- start 2023-10-30T12:50:56.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.259+0000 7f683f59e640 1 -- start start 2023-10-30T12:50:56.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.259+0000 7f683f59e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f684006ded0 0x7f68400c3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:56.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.259+0000 7f683f59e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6840070450 0x7f6840150580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:56.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.259+0000 7f683f59e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68401539a0 0x7f684014af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:56.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.259+0000 7f683f59e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68401512e0 con 0x7f6840070450 2023-10-30T12:50:56.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.259+0000 7f683f59e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68400cafb0 con 0x7f684006ded0 2023-10-30T12:50:56.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.259+0000 7f683f59e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6840152150 con 0x7f68401539a0 2023-10-30T12:50:56.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.259+0000 7f6835d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6840070450 0x7f6840150580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:56.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.259+0000 7f6835d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6840070450 0x7f6840150580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46642/0 (socket says 172.21.15.89:46642) 2023-10-30T12:50:56.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.259+0000 7f6835d9b640 1 -- 172.21.15.89:0/2688267235 learned_addr learned my addr 172.21.15.89:0/2688267235 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:50:56.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.259+0000 7f683e59c640 1 --2- 172.21.15.89:0/2688267235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f684006ded0 0x7f68400c3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:56.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.259+0000 7f683ed9d640 1 --2- 172.21.15.89:0/2688267235 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68401539a0 0x7f684014af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:56.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.259+0000 7f6835d9b640 1 -- 172.21.15.89:0/2688267235 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68401539a0 msgr2=0x7f684014af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:56.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.259+0000 7f6835d9b640 1 --2- 172.21.15.89:0/2688267235 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68401539a0 0x7f684014af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:56.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.259+0000 7f6835d9b640 1 -- 172.21.15.89:0/2688267235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f684006ded0 msgr2=0x7f68400c3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:56.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.259+0000 7f6835d9b640 1 --2- 172.21.15.89:0/2688267235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f684006ded0 0x7f68400c3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:56.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.259+0000 7f6835d9b640 1 -- 172.21.15.89:0/2688267235 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68280094d0 con 0x7f6840070450 2023-10-30T12:50:56.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.259+0000 7f683e59c640 1 --2- 172.21.15.89:0/2688267235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f684006ded0 0x7f68400c3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:50:56.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.259+0000 7f6835d9b640 1 --2- 172.21.15.89:0/2688267235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6840070450 0x7f6840150580 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f68280af040 tx=0x7f6828019250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:56.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.259+0000 7f68377fe640 1 -- 172.21.15.89:0/2688267235 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68280b7470 con 0x7f6840070450 2023-10-30T12:50:56.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.259+0000 7f68377fe640 1 -- 172.21.15.89:0/2688267235 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68280b75d0 con 0x7f6840070450 2023-10-30T12:50:56.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.259+0000 7f68377fe640 1 -- 172.21.15.89:0/2688267235 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68280b7940 con 0x7f6840070450 2023-10-30T12:50:56.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.259+0000 7f683f59e640 1 -- 172.21.15.89:0/2688267235 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68401523a0 con 0x7f6840070450 2023-10-30T12:50:56.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.259+0000 7f683f59e640 1 -- 172.21.15.89:0/2688267235 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6840148c30 con 0x7f6840070450 2023-10-30T12:50:56.264 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.263+0000 7f683f59e640 1 -- 172.21.15.89:0/2688267235 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68400cb160 con 0x7f6840070450 2023-10-30T12:50:56.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.263+0000 7f68377fe640 1 -- 172.21.15.89:0/2688267235 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f68280b3070 con 0x7f6840070450 2023-10-30T12:50:56.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.263+0000 7f68377fe640 1 --2- 172.21.15.89:0/2688267235 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f68200664a0 0x7f6820068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:56.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.263+0000 7f68377fe640 1 -- 172.21.15.89:0/2688267235 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f682811e980 con 0x7f6840070450 2023-10-30T12:50:56.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.263+0000 7f683e59c640 1 --2- 172.21.15.89:0/2688267235 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f68200664a0 0x7f6820068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:56.265 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.263+0000 7f683e59c640 1 --2- 172.21.15.89:0/2688267235 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f68200664a0 0x7f6820068960 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f68300012f0 tx=0x7f6830009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:56.266 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.267+0000 7f68377fe640 1 -- 172.21.15.89:0/2688267235 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f68400cb160 con 0x7f6840070450 2023-10-30T12:50:56.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.451+0000 7f683f59e640 1 -- 172.21.15.89:0/2688267235 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68400cb160 con 0x7f6840070450 2023-10-30T12:50:56.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.451+0000 7f68377fe640 1 -- 172.21.15.89:0/2688267235 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+5748 (secure 0 0 0) 0x7f68400cb160 con 0x7f6840070450 2023-10-30T12:50:56.449 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:50:56.449 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:50:49.582176+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6271,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6271":{"gid":6271,"name":"b","rank":0,"incarnation":110,"state":"up:resolve","state_seq":21,"addr":"172.21.15.192:6835/659375964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":659375964},{"type":"v1","addr":"172.21.15.192:6835","nonce":659375964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:50:56.450 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 111 2023-10-30T12:50:56.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.451+0000 7f683f59e640 1 -- 172.21.15.89:0/2688267235 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f68200664a0 msgr2=0x7f6820068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:56.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.451+0000 7f683f59e640 1 --2- 172.21.15.89:0/2688267235 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f68200664a0 0x7f6820068960 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f68300012f0 tx=0x7f6830009040 comp rx=0 tx=0).stop 2023-10-30T12:50:56.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.451+0000 7f683f59e640 1 -- 172.21.15.89:0/2688267235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6840070450 msgr2=0x7f6840150580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:56.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.451+0000 7f683f59e640 1 --2- 172.21.15.89:0/2688267235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6840070450 0x7f6840150580 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f68280af040 tx=0x7f6828019250 comp rx=0 tx=0).stop 2023-10-30T12:50:56.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.451+0000 7f683f59e640 1 -- 172.21.15.89:0/2688267235 shutdown_connections 2023-10-30T12:50:56.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.451+0000 7f683f59e640 1 --2- 172.21.15.89:0/2688267235 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f68200664a0 0x7f6820068960 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:56.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.451+0000 7f683f59e640 1 --2- 172.21.15.89:0/2688267235 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f68401539a0 0x7f684014af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:56.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.451+0000 7f683f59e640 1 --2- 172.21.15.89:0/2688267235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6840070450 0x7f6840150580 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:56.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.451+0000 7f683f59e640 1 --2- 172.21.15.89:0/2688267235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f684006ded0 0x7f68400c3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:56.452 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.451+0000 7f683f59e640 1 -- 172.21.15.89:0/2688267235 >> 172.21.15.89:0/2688267235 conn(0x7f68400cbe40 msgr2=0x7f68400cc220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:56.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.451+0000 7f683f59e640 1 -- 172.21.15.89:0/2688267235 shutdown_connections 2023-10-30T12:50:56.453 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:56.451+0000 7f683f59e640 1 -- 172.21.15.89:0/2688267235 wait complete. 2023-10-30T12:50:56.469 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:50:58.471 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:50:58.619 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.618+0000 7ff555f3e640 1 Processor -- start 2023-10-30T12:50:58.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.618+0000 7ff555f3e640 1 -- start start 2023-10-30T12:50:58.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.622+0000 7ff555f3e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff55006f7a0 0x7ff55006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:58.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.622+0000 7ff555f3e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff5500c6bd0 0x7ff550069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:58.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.622+0000 7ff555f3e640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff55006a3d0 con 0x7ff550071110 2023-10-30T12:50:58.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.622+0000 7ff555f3e640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5500700c0 con 0x7ff5500c6bd0 2023-10-30T12:50:58.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.622+0000 7ff555f3e640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff550066cf0 con 0x7ff55006f7a0 2023-10-30T12:50:58.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.622+0000 7ff54effd640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff5500c6bd0 0x7ff550069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:58.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.622+0000 7ff54ffff640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ff550071110 0x7ff550158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:34726/0 (socket says 172.21.15.89:34726) 2023-10-30T12:50:58.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.622+0000 7ff54ffff640 1 -- 172.21.15.89:0/1611645439 learned_addr learned my addr 172.21.15.89:0/1611645439 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:50:58.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.622+0000 7ff54f7fe640 1 --2- 172.21.15.89:0/1611645439 >> v2:172.21.15.192:3300/0 conn(0x7ff55006f7a0 0x7ff55006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:58.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.622+0000 7ff54effd640 1 -- 172.21.15.89:0/1611645439 >> v2:172.21.15.192:3300/0 conn(0x7ff55006f7a0 msgr2=0x7ff55006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:50:58.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.622+0000 7ff54effd640 1 --2- 172.21.15.89:0/1611645439 >> v2:172.21.15.192:3300/0 conn(0x7ff55006f7a0 0x7ff55006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:58.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.622+0000 7ff54effd640 1 -- 172.21.15.89:0/1611645439 >> v1:172.21.15.89:6789/0 conn(0x7ff550071110 legacy=0x7ff550158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:58.623 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.622+0000 7ff54effd640 1 -- 172.21.15.89:0/1611645439 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5500c70d0 con 0x7ff5500c6bd0 2023-10-30T12:50:58.624 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.622+0000 7ff54e7fc640 1 -- 172.21.15.89:0/1611645439 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1483632939 0 0) 0x7ff55006a3d0 con 0x7ff550071110 2023-10-30T12:50:58.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.626+0000 7ff54effd640 1 --2- 172.21.15.89:0/1611645439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5500c6bd0 0x7ff550069e90 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7ff55006c830 tx=0x7ff54401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a106f3401a7ec2dd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:58.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.626+0000 7ff54e7fc640 1 -- 172.21.15.89:0/1611645439 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5440a8950 con 0x7ff5500c6bd0 2023-10-30T12:50:58.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.626+0000 7ff54e7fc640 1 -- 172.21.15.89:0/1611645439 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5440a8ab0 con 0x7ff5500c6bd0 2023-10-30T12:50:58.625 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.626+0000 7ff54e7fc640 1 -- 172.21.15.89:0/1611645439 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5440a8e40 con 0x7ff5500c6bd0 2023-10-30T12:50:58.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.626+0000 7ff555f3e640 1 -- 172.21.15.89:0/1611645439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5500c6bd0 msgr2=0x7ff550069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:58.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.626+0000 7ff555f3e640 1 --2- 172.21.15.89:0/1611645439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5500c6bd0 0x7ff550069e90 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7ff55006c830 tx=0x7ff54401ec50 comp rx=0 tx=0).stop 2023-10-30T12:50:58.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.626+0000 7ff555f3e640 1 -- 172.21.15.89:0/1611645439 shutdown_connections 2023-10-30T12:50:58.626 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.626+0000 7ff555f3e640 1 --2- 172.21.15.89:0/1611645439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5500c6bd0 0x7ff550069e90 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:58.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.626+0000 7ff555f3e640 1 --2- 172.21.15.89:0/1611645439 >> v2:172.21.15.192:3300/0 conn(0x7ff55006f7a0 0x7ff55006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:58.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.626+0000 7ff555f3e640 1 -- 172.21.15.89:0/1611645439 >> 172.21.15.89:0/1611645439 conn(0x7ff5500c7910 msgr2=0x7ff5500c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:58.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.626+0000 7ff555f3e640 1 -- 172.21.15.89:0/1611645439 shutdown_connections 2023-10-30T12:50:58.627 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.626+0000 7ff555f3e640 1 -- 172.21.15.89:0/1611645439 wait complete. 2023-10-30T12:50:58.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.626+0000 7ff555f3e640 1 Processor -- start 2023-10-30T12:50:58.629 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.626+0000 7ff555f3e640 1 -- start start 2023-10-30T12:50:58.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.630+0000 7ff555f3e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff55006f7a0 0x7ff550150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:58.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.630+0000 7ff555f3e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff550071110 0x7ff5501572e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:58.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.630+0000 7ff555f3e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5500c6bd0 0x7ff55014bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:58.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.630+0000 7ff555f3e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55014d3c0 con 0x7ff55006f7a0 2023-10-30T12:50:58.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.630+0000 7ff555f3e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55014d6a0 con 0x7ff550071110 2023-10-30T12:50:58.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.630+0000 7ff555f3e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff55014a420 con 0x7ff5500c6bd0 2023-10-30T12:50:58.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.630+0000 7ff54f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff55006f7a0 0x7ff550150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:58.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.630+0000 7ff54f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff55006f7a0 0x7ff550150060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46662/0 (socket says 172.21.15.89:46662) 2023-10-30T12:50:58.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.630+0000 7ff54f7fe640 1 -- 172.21.15.89:0/3790762544 learned_addr learned my addr 172.21.15.89:0/3790762544 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:50:58.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.630+0000 7ff54ffff640 1 --2- 172.21.15.89:0/3790762544 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5500c6bd0 0x7ff55014bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:58.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.630+0000 7ff54effd640 1 --2- 172.21.15.89:0/3790762544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff550071110 0x7ff5501572e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:58.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.630+0000 7ff54f7fe640 1 -- 172.21.15.89:0/3790762544 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5500c6bd0 msgr2=0x7ff55014bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:58.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.630+0000 7ff54f7fe640 1 --2- 172.21.15.89:0/3790762544 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5500c6bd0 0x7ff55014bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:58.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.630+0000 7ff54f7fe640 1 -- 172.21.15.89:0/3790762544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff550071110 msgr2=0x7ff5501572e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:58.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.630+0000 7ff54f7fe640 1 --2- 172.21.15.89:0/3790762544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff550071110 0x7ff5501572e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:58.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.630+0000 7ff54f7fe640 1 -- 172.21.15.89:0/3790762544 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff544009420 con 0x7ff55006f7a0 2023-10-30T12:50:58.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.630+0000 7ff54effd640 1 --2- 172.21.15.89:0/3790762544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff550071110 0x7ff5501572e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:50:58.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.630+0000 7ff54f7fe640 1 --2- 172.21.15.89:0/3790762544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff55006f7a0 0x7ff550150060 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7ff54000d5f0 tx=0x7ff54000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:58.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.630+0000 7ff537fff640 1 -- 172.21.15.89:0/3790762544 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5400125f0 con 0x7ff55006f7a0 2023-10-30T12:50:58.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.630+0000 7ff537fff640 1 -- 172.21.15.89:0/3790762544 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff540012750 con 0x7ff55006f7a0 2023-10-30T12:50:58.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.630+0000 7ff555f3e640 1 -- 172.21.15.89:0/3790762544 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff55014a700 con 0x7ff55006f7a0 2023-10-30T12:50:58.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.630+0000 7ff537fff640 1 -- 172.21.15.89:0/3790762544 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff540012ae0 con 0x7ff55006f7a0 2023-10-30T12:50:58.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.630+0000 7ff555f3e640 1 -- 172.21.15.89:0/3790762544 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff55014acc0 con 0x7ff55006f7a0 2023-10-30T12:50:58.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.630+0000 7ff555f3e640 1 -- 172.21.15.89:0/3790762544 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff55004e170 con 0x7ff55006f7a0 2023-10-30T12:50:58.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.634+0000 7ff537fff640 1 -- 172.21.15.89:0/3790762544 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff540012c40 con 0x7ff55006f7a0 2023-10-30T12:50:58.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.634+0000 7ff537fff640 1 --2- 172.21.15.89:0/3790762544 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff53c0664a0 0x7ff53c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:58.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.634+0000 7ff537fff640 1 -- 172.21.15.89:0/3790762544 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7ff540084700 con 0x7ff55006f7a0 2023-10-30T12:50:58.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.634+0000 7ff54effd640 1 --2- 172.21.15.89:0/3790762544 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff53c0664a0 0x7ff53c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:58.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.634+0000 7ff54effd640 1 --2- 172.21.15.89:0/3790762544 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff53c0664a0 0x7ff53c068960 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ff5440b2000 tx=0x7ff5440aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:58.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.638+0000 7ff537fff640 1 -- 172.21.15.89:0/3790762544 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff540051800 con 0x7ff55006f7a0 2023-10-30T12:50:58.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.818+0000 7ff555f3e640 1 -- 172.21.15.89:0/3790762544 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff550069fc0 con 0x7ff55006f7a0 2023-10-30T12:50:58.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.818+0000 7ff537fff640 1 -- 172.21.15.89:0/3790762544 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+5750 (secure 0 0 0) 0x7ff540054e20 con 0x7ff55006f7a0 2023-10-30T12:50:58.819 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:50:58.819 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:50:55.571210+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6271,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6271":{"gid":6271,"name":"b","rank":0,"incarnation":110,"state":"up:reconnect","state_seq":23,"addr":"172.21.15.192:6835/659375964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":659375964},{"type":"v1","addr":"172.21.15.192:6835","nonce":659375964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:50:58.820 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 112 2023-10-30T12:50:58.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.822+0000 7ff555f3e640 1 -- 172.21.15.89:0/3790762544 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff53c0664a0 msgr2=0x7ff53c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:58.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.822+0000 7ff555f3e640 1 --2- 172.21.15.89:0/3790762544 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff53c0664a0 0x7ff53c068960 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ff5440b2000 tx=0x7ff5440aa070 comp rx=0 tx=0).stop 2023-10-30T12:50:58.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.822+0000 7ff555f3e640 1 -- 172.21.15.89:0/3790762544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff55006f7a0 msgr2=0x7ff550150060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:58.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.822+0000 7ff555f3e640 1 --2- 172.21.15.89:0/3790762544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff55006f7a0 0x7ff550150060 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7ff54000d5f0 tx=0x7ff54000dab0 comp rx=0 tx=0).stop 2023-10-30T12:50:58.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.822+0000 7ff555f3e640 1 -- 172.21.15.89:0/3790762544 shutdown_connections 2023-10-30T12:50:58.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.822+0000 7ff555f3e640 1 --2- 172.21.15.89:0/3790762544 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff53c0664a0 0x7ff53c068960 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:58.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.822+0000 7ff555f3e640 1 --2- 172.21.15.89:0/3790762544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff550071110 0x7ff5501572e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:58.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.822+0000 7ff555f3e640 1 --2- 172.21.15.89:0/3790762544 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5500c6bd0 0x7ff55014bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:58.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.822+0000 7ff555f3e640 1 --2- 172.21.15.89:0/3790762544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff55006f7a0 0x7ff550150060 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:58.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.822+0000 7ff555f3e640 1 -- 172.21.15.89:0/3790762544 >> 172.21.15.89:0/3790762544 conn(0x7ff5500c7910 msgr2=0x7ff55006bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:58.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.822+0000 7ff555f3e640 1 -- 172.21.15.89:0/3790762544 shutdown_connections 2023-10-30T12:50:58.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:50:58.822+0000 7ff555f3e640 1 -- 172.21.15.89:0/3790762544 wait complete. 2023-10-30T12:50:58.839 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:51:00.840 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:00.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:00.990+0000 7fc63fd3e640 1 Processor -- start 2023-10-30T12:51:00.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:00.994+0000 7fc63fd3e640 1 -- start start 2023-10-30T12:51:00.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:00.994+0000 7fc63fd3e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc638071110 0x7fc63806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:00.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:00.994+0000 7fc63fd3e640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc638066cf0 con 0x7fc6380c6bd0 2023-10-30T12:51:00.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:00.994+0000 7fc63fd3e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6380c70d0 con 0x7fc63806f7a0 2023-10-30T12:51:00.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:00.994+0000 7fc63fd3e640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6380700c0 con 0x7fc638071110 2023-10-30T12:51:00.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:00.994+0000 7fc63d2d9640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc6380c6bd0 0x7fc638158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56980/0 (socket says 172.21.15.89:56980) 2023-10-30T12:51:00.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:00.994+0000 7fc63d2d9640 1 -- 172.21.15.89:0/412460662 learned_addr learned my addr 172.21.15.89:0/412460662 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:51:00.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:00.994+0000 7fc63e2db640 1 --2- 172.21.15.89:0/412460662 >> v2:172.21.15.89:3300/0 conn(0x7fc638071110 0x7fc63806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:00.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:00.994+0000 7fc63cad8640 1 -- 172.21.15.89:0/412460662 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1083498933 0 0) 0x7fc638066cf0 con 0x7fc6380c6bd0 2023-10-30T12:51:00.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:00.994+0000 7fc63cad8640 1 -- 172.21.15.89:0/412460662 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc624003530 con 0x7fc6380c6bd0 2023-10-30T12:51:00.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:00.994+0000 7fc63e2db640 1 -- 172.21.15.89:0/412460662 >> v1:172.21.15.192:6789/0 conn(0x7fc63806f7a0 legacy=0x7fc638069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:00.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:00.994+0000 7fc63e2db640 1 -- 172.21.15.89:0/412460662 >> v1:172.21.15.89:6789/0 conn(0x7fc6380c6bd0 legacy=0x7fc638158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:00.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:00.994+0000 7fc63e2db640 1 -- 172.21.15.89:0/412460662 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6380c7390 con 0x7fc638071110 2023-10-30T12:51:00.996 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:00.998+0000 7fc63e2db640 1 --2- 172.21.15.89:0/412460662 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc638071110 0x7fc63806fb80 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fc630009d90 tx=0x7fc63001f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9fd3e85ba36253d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:00.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:00.998+0000 7fc63cad8640 1 -- 172.21.15.89:0/412460662 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6300af600 con 0x7fc638071110 2023-10-30T12:51:00.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:00.998+0000 7fc63cad8640 1 -- 172.21.15.89:0/412460662 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6300ab050 con 0x7fc638071110 2023-10-30T12:51:00.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:00.998+0000 7fc63cad8640 1 -- 172.21.15.89:0/412460662 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6300af980 con 0x7fc638071110 2023-10-30T12:51:00.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:00.998+0000 7fc63fd3e640 1 -- 172.21.15.89:0/412460662 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc638071110 msgr2=0x7fc63806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:00.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:00.998+0000 7fc63fd3e640 1 --2- 172.21.15.89:0/412460662 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc638071110 0x7fc63806fb80 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fc630009d90 tx=0x7fc63001f130 comp rx=0 tx=0).stop 2023-10-30T12:51:00.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:00.998+0000 7fc63fd3e640 1 -- 172.21.15.89:0/412460662 shutdown_connections 2023-10-30T12:51:00.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:00.998+0000 7fc63fd3e640 1 --2- 172.21.15.89:0/412460662 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc638071110 0x7fc63806fb80 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:00.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:00.998+0000 7fc63fd3e640 1 -- 172.21.15.89:0/412460662 >> 172.21.15.89:0/412460662 conn(0x7fc6380c7910 msgr2=0x7fc6380c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:00.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:00.998+0000 7fc63fd3e640 1 -- 172.21.15.89:0/412460662 shutdown_connections 2023-10-30T12:51:00.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:00.998+0000 7fc63fd3e640 1 -- 172.21.15.89:0/412460662 wait complete. 2023-10-30T12:51:01.002 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:00.998+0000 7fc63fd3e640 1 Processor -- start 2023-10-30T12:51:01.002 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:00.998+0000 7fc63fd3e640 1 -- start start 2023-10-30T12:51:01.002 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.002+0000 7fc63fd3e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc63806f7a0 0x7fc6380c1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:01.002 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.002+0000 7fc63fd3e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc638071110 0x7fc6381f0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:01.002 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.002+0000 7fc63fd3e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6380c6bd0 0x7fc6381f0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:01.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.002+0000 7fc63fd3e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6381f01a0 con 0x7fc6380c6bd0 2023-10-30T12:51:01.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.002+0000 7fc63fd3e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6380ae380 con 0x7fc63806f7a0 2023-10-30T12:51:01.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.002+0000 7fc63fd3e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6380ae660 con 0x7fc638071110 2023-10-30T12:51:01.003 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.002+0000 7fc63e2db640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6380c6bd0 0x7fc6381f0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:01.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.002+0000 7fc63e2db640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6380c6bd0 0x7fc6381f0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55128/0 (socket says 172.21.15.89:55128) 2023-10-30T12:51:01.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.002+0000 7fc63e2db640 1 -- 172.21.15.89:0/1294586472 learned_addr learned my addr 172.21.15.89:0/1294586472 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:01.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.002+0000 7fc63d2d9640 1 --2- 172.21.15.89:0/1294586472 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc638071110 0x7fc6381f0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:01.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.002+0000 7fc63dada640 1 --2- 172.21.15.89:0/1294586472 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc63806f7a0 0x7fc6380c1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:01.004 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.002+0000 7fc63e2db640 1 -- 172.21.15.89:0/1294586472 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc638071110 msgr2=0x7fc6381f0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:01.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.002+0000 7fc63e2db640 1 --2- 172.21.15.89:0/1294586472 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc638071110 0x7fc6381f0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:01.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.002+0000 7fc63e2db640 1 -- 172.21.15.89:0/1294586472 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc63806f7a0 msgr2=0x7fc6380c1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:01.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.002+0000 7fc63e2db640 1 --2- 172.21.15.89:0/1294586472 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc63806f7a0 0x7fc6380c1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:01.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.002+0000 7fc63e2db640 1 -- 172.21.15.89:0/1294586472 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc630009a70 con 0x7fc6380c6bd0 2023-10-30T12:51:01.005 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.002+0000 7fc63d2d9640 1 --2- 172.21.15.89:0/1294586472 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc638071110 0x7fc6381f0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:01.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.002+0000 7fc63dada640 1 --2- 172.21.15.89:0/1294586472 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc63806f7a0 0x7fc6380c1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:01.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.002+0000 7fc63e2db640 1 --2- 172.21.15.89:0/1294586472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6380c6bd0 0x7fc6381f0b70 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fc63001cd60 tx=0x7fc63001bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:01.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.002+0000 7fc62e7fc640 1 -- 172.21.15.89:0/1294586472 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc630009450 con 0x7fc6380c6bd0 2023-10-30T12:51:01.006 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.002+0000 7fc62e7fc640 1 -- 172.21.15.89:0/1294586472 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6300039a0 con 0x7fc6380c6bd0 2023-10-30T12:51:01.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.002+0000 7fc63fd3e640 1 -- 172.21.15.89:0/1294586472 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6380ae940 con 0x7fc6380c6bd0 2023-10-30T12:51:01.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.002+0000 7fc62e7fc640 1 -- 172.21.15.89:0/1294586472 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc630003d10 con 0x7fc6380c6bd0 2023-10-30T12:51:01.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.002+0000 7fc63fd3e640 1 -- 172.21.15.89:0/1294586472 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6380aef00 con 0x7fc6380c6bd0 2023-10-30T12:51:01.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.002+0000 7fc63fd3e640 1 -- 172.21.15.89:0/1294586472 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6381f0310 con 0x7fc6380c6bd0 2023-10-30T12:51:01.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.006+0000 7fc62e7fc640 1 -- 172.21.15.89:0/1294586472 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc630004030 con 0x7fc6380c6bd0 2023-10-30T12:51:01.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.006+0000 7fc62e7fc640 1 --2- 172.21.15.89:0/1294586472 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc61c0664a0 0x7fc61c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:01.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.006+0000 7fc62e7fc640 1 -- 172.21.15.89:0/1294586472 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7fc630122ac0 con 0x7fc6380c6bd0 2023-10-30T12:51:01.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.006+0000 7fc63dada640 1 --2- 172.21.15.89:0/1294586472 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc61c0664a0 0x7fc61c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:01.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.006+0000 7fc63dada640 1 --2- 172.21.15.89:0/1294586472 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc61c0664a0 0x7fc61c068960 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fc634001670 tx=0x7fc634009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:01.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.010+0000 7fc62e7fc640 1 -- 172.21.15.89:0/1294586472 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc6300efc40 con 0x7fc6380c6bd0 2023-10-30T12:51:01.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.190+0000 7fc63fd3e640 1 -- 172.21.15.89:0/1294586472 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc638070600 con 0x7fc6380c6bd0 2023-10-30T12:51:01.189 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.190+0000 7fc62e7fc640 1 -- 172.21.15.89:0/1294586472 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+5748 (secure 0 0 0) 0x7fc6300f3260 con 0x7fc6380c6bd0 2023-10-30T12:51:01.190 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:51:01.190 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:51:00.403791+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6271,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6271":{"gid":6271,"name":"b","rank":0,"incarnation":110,"state":"up:reconnect","state_seq":23,"addr":"172.21.15.192:6835/659375964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":659375964},{"type":"v1","addr":"172.21.15.192:6835","nonce":659375964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:01.192 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 113 2023-10-30T12:51:01.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.190+0000 7fc63fd3e640 1 -- 172.21.15.89:0/1294586472 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc61c0664a0 msgr2=0x7fc61c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:01.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.190+0000 7fc63fd3e640 1 --2- 172.21.15.89:0/1294586472 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc61c0664a0 0x7fc61c068960 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fc634001670 tx=0x7fc634009040 comp rx=0 tx=0).stop 2023-10-30T12:51:01.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.190+0000 7fc63fd3e640 1 -- 172.21.15.89:0/1294586472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6380c6bd0 msgr2=0x7fc6381f0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:01.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.190+0000 7fc63fd3e640 1 --2- 172.21.15.89:0/1294586472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6380c6bd0 0x7fc6381f0b70 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fc63001cd60 tx=0x7fc63001bed0 comp rx=0 tx=0).stop 2023-10-30T12:51:01.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.194+0000 7fc63fd3e640 1 -- 172.21.15.89:0/1294586472 shutdown_connections 2023-10-30T12:51:01.193 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.194+0000 7fc63fd3e640 1 --2- 172.21.15.89:0/1294586472 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc61c0664a0 0x7fc61c068960 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:01.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.194+0000 7fc63fd3e640 1 --2- 172.21.15.89:0/1294586472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6380c6bd0 0x7fc6381f0b70 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:01.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.194+0000 7fc63fd3e640 1 --2- 172.21.15.89:0/1294586472 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc638071110 0x7fc6381f0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:01.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.194+0000 7fc63fd3e640 1 --2- 172.21.15.89:0/1294586472 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc63806f7a0 0x7fc6380c1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:01.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.194+0000 7fc63fd3e640 1 -- 172.21.15.89:0/1294586472 >> 172.21.15.89:0/1294586472 conn(0x7fc6380c7910 msgr2=0x7fc6380c39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:01.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.194+0000 7fc63fd3e640 1 -- 172.21.15.89:0/1294586472 shutdown_connections 2023-10-30T12:51:01.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:01.194+0000 7fc63fd3e640 1 -- 172.21.15.89:0/1294586472 wait complete. 2023-10-30T12:51:01.210 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:51:03.212 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:03.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.362+0000 7f05237d3640 1 Processor -- start 2023-10-30T12:51:03.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.362+0000 7f05237d3640 1 -- start start 2023-10-30T12:51:03.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.362+0000 7f05237d3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f051c1538a0 0x7f051c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:03.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.362+0000 7f05237d3640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f051c0ca0e0 0x7f051c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:03.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.362+0000 7f05237d3640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f051c0cb800 con 0x7f051c0cf160 2023-10-30T12:51:03.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.362+0000 7f05237d3640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f051c15dc30 con 0x7f051c0ca0e0 2023-10-30T12:51:03.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.362+0000 7f05237d3640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f051c0cb330 con 0x7f051c1538a0 2023-10-30T12:51:03.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.362+0000 7f0520d6e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f051c0ca0e0 0x7f051c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:03.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.362+0000 7f0521d70640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f051c0cf160 0x7f051c0cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.89:45424/0 (socket says 172.21.15.89:45424) 2023-10-30T12:51:03.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.362+0000 7f0521d70640 1 -- 172.21.15.89:0/3224093956 learned_addr learned my addr 172.21.15.89:0/3224093956 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:51:03.363 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.362+0000 7f052156f640 1 --2- 172.21.15.89:0/3224093956 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f051c1538a0 0x7f051c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:03.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.362+0000 7f052156f640 1 -- 172.21.15.89:0/3224093956 >> v2:172.21.15.192:3300/0 conn(0x7f051c0ca0e0 msgr2=0x7f051c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:03.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.362+0000 7f052156f640 1 --2- 172.21.15.89:0/3224093956 >> v2:172.21.15.192:3300/0 conn(0x7f051c0ca0e0 0x7f051c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:03.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.362+0000 7f052156f640 1 -- 172.21.15.89:0/3224093956 >> v1:172.21.15.192:6789/0 conn(0x7f051c0cf160 legacy=0x7f051c0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:03.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.362+0000 7f052156f640 1 -- 172.21.15.89:0/3224093956 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051c0cad70 con 0x7f051c1538a0 2023-10-30T12:51:03.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.362+0000 7f0520d6e640 1 --2- 172.21.15.89:0/3224093956 >> v2:172.21.15.192:3300/0 conn(0x7f051c0ca0e0 0x7f051c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:51:03.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.366+0000 7f052156f640 1 --2- 172.21.15.89:0/3224093956 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f051c1538a0 0x7f051c15d6f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f05180097c0 tx=0x7f051801ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cffca2ba6670f3f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:03.366 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.366+0000 7f0513fff640 1 -- 172.21.15.89:0/3224093956 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05180a8970 con 0x7f051c1538a0 2023-10-30T12:51:03.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.366+0000 7f0513fff640 1 -- 172.21.15.89:0/3224093956 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05180a8ad0 con 0x7f051c1538a0 2023-10-30T12:51:03.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.366+0000 7f0513fff640 1 -- 172.21.15.89:0/3224093956 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05180a8e60 con 0x7f051c1538a0 2023-10-30T12:51:03.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.366+0000 7f05237d3640 1 -- 172.21.15.89:0/3224093956 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f051c1538a0 msgr2=0x7f051c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:03.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.366+0000 7f05237d3640 1 --2- 172.21.15.89:0/3224093956 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f051c1538a0 0x7f051c15d6f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f05180097c0 tx=0x7f051801ec70 comp rx=0 tx=0).stop 2023-10-30T12:51:03.367 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.366+0000 7f05237d3640 1 -- 172.21.15.89:0/3224093956 shutdown_connections 2023-10-30T12:51:03.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.366+0000 7f05237d3640 1 --2- 172.21.15.89:0/3224093956 >> v2:172.21.15.192:3300/0 conn(0x7f051c0ca0e0 0x7f051c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:03.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.366+0000 7f05237d3640 1 --2- 172.21.15.89:0/3224093956 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f051c1538a0 0x7f051c15d6f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:03.368 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.366+0000 7f05237d3640 1 -- 172.21.15.89:0/3224093956 >> 172.21.15.89:0/3224093956 conn(0x7f051c06bfe0 msgr2=0x7f051c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:03.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.366+0000 7f05237d3640 1 -- 172.21.15.89:0/3224093956 shutdown_connections 2023-10-30T12:51:03.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.366+0000 7f05237d3640 1 -- 172.21.15.89:0/3224093956 wait complete. 2023-10-30T12:51:03.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.370+0000 7f05237d3640 1 Processor -- start 2023-10-30T12:51:03.369 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.370+0000 7f05237d3640 1 -- start start 2023-10-30T12:51:03.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.370+0000 7f05237d3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f051c0ca0e0 0x7f051c0ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:03.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.370+0000 7f05237d3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f051c0cf160 0x7f051c0b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:03.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.370+0000 7f05237d3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f051c1538a0 0x7f051c0b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:03.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.370+0000 7f05237d3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051c1efcc0 con 0x7f051c1538a0 2023-10-30T12:51:03.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.370+0000 7f05237d3640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051c1f01f0 con 0x7f051c0ca0e0 2023-10-30T12:51:03.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.370+0000 7f05237d3640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f051c1eff10 con 0x7f051c0cf160 2023-10-30T12:51:03.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.370+0000 7f0521d70640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f051c1538a0 0x7f051c0b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:03.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.370+0000 7f0521d70640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f051c1538a0 0x7f051c0b42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55136/0 (socket says 172.21.15.89:55136) 2023-10-30T12:51:03.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.370+0000 7f0521d70640 1 -- 172.21.15.89:0/1367196632 learned_addr learned my addr 172.21.15.89:0/1367196632 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:03.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.370+0000 7f052156f640 1 --2- 172.21.15.89:0/1367196632 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f051c0ca0e0 0x7f051c0ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:03.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.370+0000 7f0520d6e640 1 --2- 172.21.15.89:0/1367196632 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f051c0cf160 0x7f051c0b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:03.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.370+0000 7f0521d70640 1 -- 172.21.15.89:0/1367196632 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f051c0cf160 msgr2=0x7f051c0b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:03.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.370+0000 7f0521d70640 1 --2- 172.21.15.89:0/1367196632 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f051c0cf160 0x7f051c0b3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:03.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.370+0000 7f0521d70640 1 -- 172.21.15.89:0/1367196632 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f051c0ca0e0 msgr2=0x7f051c0ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:03.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.370+0000 7f0521d70640 1 --2- 172.21.15.89:0/1367196632 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f051c0ca0e0 0x7f051c0ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:03.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.370+0000 7f0521d70640 1 -- 172.21.15.89:0/1367196632 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050c009420 con 0x7f051c1538a0 2023-10-30T12:51:03.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.370+0000 7f0521d70640 1 --2- 172.21.15.89:0/1367196632 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f051c1538a0 0x7f051c0b42f0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f051400ca70 tx=0x7f051400cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:03.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.370+0000 7f0511ffb640 1 -- 172.21.15.89:0/1367196632 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051400abd0 con 0x7f051c1538a0 2023-10-30T12:51:03.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.370+0000 7f0511ffb640 1 -- 172.21.15.89:0/1367196632 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f051400ad30 con 0x7f051c1538a0 2023-10-30T12:51:03.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.370+0000 7f0511ffb640 1 -- 172.21.15.89:0/1367196632 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0514012600 con 0x7f051c1538a0 2023-10-30T12:51:03.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.370+0000 7f05237d3640 1 -- 172.21.15.89:0/1367196632 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0518009420 con 0x7f051c1538a0 2023-10-30T12:51:03.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.374+0000 7f05237d3640 1 -- 172.21.15.89:0/1367196632 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f051c0b35b0 con 0x7f051c1538a0 2023-10-30T12:51:03.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.374+0000 7f05237d3640 1 -- 172.21.15.89:0/1367196632 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f051c0b2a50 con 0x7f051c1538a0 2023-10-30T12:51:03.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.374+0000 7f0511ffb640 1 -- 172.21.15.89:0/1367196632 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0514007070 con 0x7f051c1538a0 2023-10-30T12:51:03.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.374+0000 7f0511ffb640 1 --2- 172.21.15.89:0/1367196632 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f05000664d0 0x7f0500068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:03.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.374+0000 7f0511ffb640 1 -- 172.21.15.89:0/1367196632 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f0514083fa0 con 0x7f051c1538a0 2023-10-30T12:51:03.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.374+0000 7f052156f640 1 --2- 172.21.15.89:0/1367196632 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f05000664d0 0x7f0500068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:03.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.374+0000 7f052156f640 1 --2- 172.21.15.89:0/1367196632 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f05000664d0 0x7f0500068990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f05180b2000 tx=0x7f05180aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:03.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.378+0000 7f0511ffb640 1 -- 172.21.15.89:0/1367196632 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f05140b5090 con 0x7f051c1538a0 2023-10-30T12:51:03.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.558+0000 7f05237d3640 1 -- 172.21.15.89:0/1367196632 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f051c04e170 con 0x7f051c1538a0 2023-10-30T12:51:03.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.558+0000 7f0511ffb640 1 -- 172.21.15.89:0/1367196632 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+5745 (secure 0 0 0) 0x7f0514017090 con 0x7f051c1538a0 2023-10-30T12:51:03.558 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:51:03.558 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:51:01.553267+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6271,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6271":{"gid":6271,"name":"b","rank":0,"incarnation":110,"state":"up:rejoin","state_seq":25,"addr":"172.21.15.192:6835/659375964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":659375964},{"type":"v1","addr":"172.21.15.192:6835","nonce":659375964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:03.559 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 114 2023-10-30T12:51:03.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.562+0000 7f05237d3640 1 -- 172.21.15.89:0/1367196632 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f05000664d0 msgr2=0x7f0500068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:03.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.562+0000 7f05237d3640 1 --2- 172.21.15.89:0/1367196632 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f05000664d0 0x7f0500068990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f05180b2000 tx=0x7f05180aa070 comp rx=0 tx=0).stop 2023-10-30T12:51:03.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.562+0000 7f05237d3640 1 -- 172.21.15.89:0/1367196632 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f051c1538a0 msgr2=0x7f051c0b42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:03.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.562+0000 7f05237d3640 1 --2- 172.21.15.89:0/1367196632 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f051c1538a0 0x7f051c0b42f0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f051400ca70 tx=0x7f051400cf30 comp rx=0 tx=0).stop 2023-10-30T12:51:03.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.562+0000 7f05237d3640 1 -- 172.21.15.89:0/1367196632 shutdown_connections 2023-10-30T12:51:03.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.562+0000 7f05237d3640 1 --2- 172.21.15.89:0/1367196632 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f05000664d0 0x7f0500068990 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:03.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.562+0000 7f05237d3640 1 --2- 172.21.15.89:0/1367196632 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f051c1538a0 0x7f051c0b42f0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:03.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.562+0000 7f05237d3640 1 --2- 172.21.15.89:0/1367196632 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f051c0cf160 0x7f051c0b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:03.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.562+0000 7f05237d3640 1 --2- 172.21.15.89:0/1367196632 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f051c0ca0e0 0x7f051c0ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:03.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.562+0000 7f05237d3640 1 -- 172.21.15.89:0/1367196632 >> 172.21.15.89:0/1367196632 conn(0x7f051c06bfe0 msgr2=0x7f051c152870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:03.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.562+0000 7f05237d3640 1 -- 172.21.15.89:0/1367196632 shutdown_connections 2023-10-30T12:51:03.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:03.562+0000 7f05237d3640 1 -- 172.21.15.89:0/1367196632 wait complete. 2023-10-30T12:51:03.579 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:51:05.580 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:05.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.730+0000 7fc5b7246640 1 Processor -- start 2023-10-30T12:51:05.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.730+0000 7fc5b7246640 1 -- start start 2023-10-30T12:51:05.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.730+0000 7fc5b7246640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5b00c6b80 0x7fc5b00bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:05.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.730+0000 7fc5b7246640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b00c0a50 con 0x7fc5b013f4a0 2023-10-30T12:51:05.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.730+0000 7fc5b7246640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b006f6e0 con 0x7fc5b00c6f60 2023-10-30T12:51:05.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.730+0000 7fc5b7246640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b006f9c0 con 0x7fc5b00c6b80 2023-10-30T12:51:05.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.730+0000 7fc5b57e3640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc5b013f4a0 0x7fc5b01566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56990/0 (socket says 172.21.15.89:56990) 2023-10-30T12:51:05.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.734+0000 7fc5b57e3640 1 -- 172.21.15.89:0/3701489411 learned_addr learned my addr 172.21.15.89:0/3701489411 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:51:05.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.734+0000 7fc5b4fe2640 1 --2- 172.21.15.89:0/3701489411 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5b00c6b80 0x7fc5b00bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:05.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.734+0000 7fc5a77fe640 1 -- 172.21.15.89:0/3701489411 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 832359014 0 0) 0x7fc5b00c0a50 con 0x7fc5b013f4a0 2023-10-30T12:51:05.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.734+0000 7fc5a77fe640 1 -- 172.21.15.89:0/3701489411 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc598005740 con 0x7fc5b013f4a0 2023-10-30T12:51:05.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.734+0000 7fc5a77fe640 1 -- 172.21.15.89:0/3701489411 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 41361580 0 0) 0x7fc598005740 con 0x7fc5b013f4a0 2023-10-30T12:51:05.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.734+0000 7fc5a77fe640 1 -- 172.21.15.89:0/3701489411 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5b00c6b80 msgr2=0x7fc5b00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:05.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.734+0000 7fc5a77fe640 1 --2- 172.21.15.89:0/3701489411 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5b00c6b80 0x7fc5b00bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:05.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.734+0000 7fc5a77fe640 1 -- 172.21.15.89:0/3701489411 >> v1:172.21.15.192:6789/0 conn(0x7fc5b00c6f60 legacy=0x7fc5b013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:05.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.734+0000 7fc5a77fe640 1 -- 172.21.15.89:0/3701489411 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b006fca0 con 0x7fc5b013f4a0 2023-10-30T12:51:05.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.734+0000 7fc5a77fe640 1 -- 172.21.15.89:0/3701489411 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7fc5a8001a00 con 0x7fc5b013f4a0 2023-10-30T12:51:05.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.734+0000 7fc5a77fe640 1 -- 172.21.15.89:0/3701489411 >> v1:172.21.15.89:6789/0 conn(0x7fc5b013f4a0 legacy=0x7fc5b01566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:05.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.734+0000 7fc5a77fe640 1 --2- 172.21.15.89:0/3701489411 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5b00c6b80 0x7fc598005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:05.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.734+0000 7fc5a77fe640 1 --2- 172.21.15.89:0/3701489411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc598006140 0x7fc5980069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:05.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.734+0000 7fc5a77fe640 1 --2- 172.21.15.89:0/3701489411 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc598006f20 0x7fc5980073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:05.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.734+0000 7fc5a77fe640 1 -- 172.21.15.89:0/3701489411 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b006f9c0 con 0x7fc598006140 2023-10-30T12:51:05.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.734+0000 7fc5a77fe640 1 -- 172.21.15.89:0/3701489411 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59800dc80 con 0x7fc598006f20 2023-10-30T12:51:05.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.734+0000 7fc5a77fe640 1 -- 172.21.15.89:0/3701489411 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc59800de40 con 0x7fc5b00c6b80 2023-10-30T12:51:05.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.734+0000 7fc5a7fff640 1 --2- 172.21.15.89:0/3701489411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc598006140 0x7fc5980069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:05.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.734+0000 7fc5b4fe2640 1 --2- 172.21.15.89:0/3701489411 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5b00c6b80 0x7fc598005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:05.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.734+0000 7fc5b57e3640 1 --2- 172.21.15.89:0/3701489411 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc598006f20 0x7fc5980073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:05.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.734+0000 7fc5a7fff640 1 -- 172.21.15.89:0/3701489411 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5b00c6b80 msgr2=0x7fc598005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:05.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.734+0000 7fc5a7fff640 1 --2- 172.21.15.89:0/3701489411 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5b00c6b80 0x7fc598005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:05.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.734+0000 7fc5a7fff640 1 -- 172.21.15.89:0/3701489411 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc598006f20 msgr2=0x7fc5980073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:05.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.734+0000 7fc5a7fff640 1 --2- 172.21.15.89:0/3701489411 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc598006f20 0x7fc5980073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:05.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.734+0000 7fc5a7fff640 1 -- 172.21.15.89:0/3701489411 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc598005740 con 0x7fc598006140 2023-10-30T12:51:05.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.734+0000 7fc5b57e3640 1 --2- 172.21.15.89:0/3701489411 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc598006f20 0x7fc5980073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:05.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.738+0000 7fc5a7fff640 1 --2- 172.21.15.89:0/3701489411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc598006140 0x7fc5980069e0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fc5a0001750 tx=0x7fc5a001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dc4a7a09ca0ec43 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:05.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.738+0000 7fc5a77fe640 1 -- 172.21.15.89:0/3701489411 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a00052d0 con 0x7fc598006140 2023-10-30T12:51:05.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.738+0000 7fc5a77fe640 1 -- 172.21.15.89:0/3701489411 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5a00a7050 con 0x7fc598006140 2023-10-30T12:51:05.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.738+0000 7fc5a77fe640 1 -- 172.21.15.89:0/3701489411 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a0003600 con 0x7fc598006140 2023-10-30T12:51:05.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.738+0000 7fc5b7246640 1 -- 172.21.15.89:0/3701489411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc598006140 msgr2=0x7fc5980069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:05.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.738+0000 7fc5b7246640 1 --2- 172.21.15.89:0/3701489411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc598006140 0x7fc5980069e0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fc5a0001750 tx=0x7fc5a001c3c0 comp rx=0 tx=0).stop 2023-10-30T12:51:05.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.738+0000 7fc5b57e3640 1 -- 172.21.15.89:0/3701489411 reap_dead start 2023-10-30T12:51:05.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.738+0000 7fc5b7246640 1 -- 172.21.15.89:0/3701489411 shutdown_connections 2023-10-30T12:51:05.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.738+0000 7fc5b7246640 1 -- 172.21.15.89:0/3701489411 >> 172.21.15.89:0/3701489411 conn(0x7fc5b00c7910 msgr2=0x7fc5b00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:05.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.738+0000 7fc5b7246640 1 -- 172.21.15.89:0/3701489411 shutdown_connections 2023-10-30T12:51:05.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.738+0000 7fc5b7246640 1 -- 172.21.15.89:0/3701489411 wait complete. 2023-10-30T12:51:05.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.738+0000 7fc5b7246640 1 Processor -- start 2023-10-30T12:51:05.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.738+0000 7fc5b7246640 1 -- start start 2023-10-30T12:51:05.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.742+0000 7fc5b7246640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc598006140 0x7fc5b0138560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:05.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.742+0000 7fc5b7246640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5b00c2740 0x7fc5b0155670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:05.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.742+0000 7fc5b7246640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5b0155bb0 0x7fc5b00c2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:05.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.742+0000 7fc5b7246640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b00c3ce0 con 0x7fc5b0155bb0 2023-10-30T12:51:05.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.742+0000 7fc5b7246640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b0154f00 con 0x7fc598006140 2023-10-30T12:51:05.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.742+0000 7fc5b7246640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b00c4e40 con 0x7fc5b00c2740 2023-10-30T12:51:05.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.742+0000 7fc5b57e3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5b0155bb0 0x7fc5b00c2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:05.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.742+0000 7fc5b57e3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5b0155bb0 0x7fc5b00c2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55150/0 (socket says 172.21.15.89:55150) 2023-10-30T12:51:05.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.742+0000 7fc5b57e3640 1 -- 172.21.15.89:0/3674175971 learned_addr learned my addr 172.21.15.89:0/3674175971 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:05.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.742+0000 7fc5a7fff640 1 --2- 172.21.15.89:0/3674175971 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5b00c2740 0x7fc5b0155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:05.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.742+0000 7fc5b4fe2640 1 --2- 172.21.15.89:0/3674175971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc598006140 0x7fc5b0138560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:05.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.742+0000 7fc5b57e3640 1 -- 172.21.15.89:0/3674175971 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5b00c2740 msgr2=0x7fc5b0155670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:05.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.742+0000 7fc5b57e3640 1 --2- 172.21.15.89:0/3674175971 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5b00c2740 0x7fc5b0155670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:05.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.742+0000 7fc5b57e3640 1 -- 172.21.15.89:0/3674175971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc598006140 msgr2=0x7fc5b0138560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:05.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.742+0000 7fc5b57e3640 1 --2- 172.21.15.89:0/3674175971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc598006140 0x7fc5b0138560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:05.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.742+0000 7fc5b57e3640 1 -- 172.21.15.89:0/3674175971 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b0156030 con 0x7fc5b0155bb0 2023-10-30T12:51:05.747 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.742+0000 7fc5b57e3640 1 --2- 172.21.15.89:0/3674175971 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5b0155bb0 0x7fc5b00c2c00 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fc5b013f8d0 tx=0x7fc5a8011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:05.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.742+0000 7fc5a57fa640 1 -- 172.21.15.89:0/3674175971 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a800f8a0 con 0x7fc5b0155bb0 2023-10-30T12:51:05.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.742+0000 7fc5a57fa640 1 -- 172.21.15.89:0/3674175971 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5a800fa20 con 0x7fc5b0155bb0 2023-10-30T12:51:05.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.742+0000 7fc5a57fa640 1 -- 172.21.15.89:0/3674175971 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a800fdd0 con 0x7fc5b0155bb0 2023-10-30T12:51:05.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.742+0000 7fc5b7246640 1 -- 172.21.15.89:0/3674175971 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b01ebc60 con 0x7fc5b0155bb0 2023-10-30T12:51:05.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.742+0000 7fc5b7246640 1 -- 172.21.15.89:0/3674175971 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b00c5390 con 0x7fc5b0155bb0 2023-10-30T12:51:05.748 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.742+0000 7fc5b7246640 1 -- 172.21.15.89:0/3674175971 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b004e170 con 0x7fc5b0155bb0 2023-10-30T12:51:05.749 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.742+0000 7fc5a57fa640 1 -- 172.21.15.89:0/3674175971 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc5a801f460 con 0x7fc5b0155bb0 2023-10-30T12:51:05.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.746+0000 7fc5a57fa640 1 --2- 172.21.15.89:0/3674175971 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc59c0664d0 0x7fc59c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:05.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.746+0000 7fc5a57fa640 1 -- 172.21.15.89:0/3674175971 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7fc5a80873b0 con 0x7fc5b0155bb0 2023-10-30T12:51:05.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.746+0000 7fc5b4fe2640 1 --2- 172.21.15.89:0/3674175971 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc59c0664d0 0x7fc59c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:05.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.746+0000 7fc5b4fe2640 1 --2- 172.21.15.89:0/3674175971 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc59c0664d0 0x7fc59c068990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac009b20 tx=0x7fc5ac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:05.750 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.750+0000 7fc5a57fa640 1 -- 172.21.15.89:0/3674175971 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc5a8057e00 con 0x7fc5b0155bb0 2023-10-30T12:51:05.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.926+0000 7fc5b7246640 1 -- 172.21.15.89:0/3674175971 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5b0071420 con 0x7fc5b0155bb0 2023-10-30T12:51:05.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.930+0000 7fc5a57fa640 1 -- 172.21.15.89:0/3674175971 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+5745 (secure 0 0 0) 0x7fc5a8057990 con 0x7fc5b0155bb0 2023-10-30T12:51:05.928 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:51:05.928 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:51:01.553267+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6271,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6271":{"gid":6271,"name":"b","rank":0,"incarnation":110,"state":"up:rejoin","state_seq":25,"addr":"172.21.15.192:6835/659375964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":659375964},{"type":"v1","addr":"172.21.15.192:6835","nonce":659375964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:05.929 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 114 2023-10-30T12:51:05.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.930+0000 7fc5b7246640 1 -- 172.21.15.89:0/3674175971 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc59c0664d0 msgr2=0x7fc59c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:05.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.930+0000 7fc5b7246640 1 --2- 172.21.15.89:0/3674175971 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc59c0664d0 0x7fc59c068990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac009b20 tx=0x7fc5ac008040 comp rx=0 tx=0).stop 2023-10-30T12:51:05.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.930+0000 7fc5b7246640 1 -- 172.21.15.89:0/3674175971 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5b0155bb0 msgr2=0x7fc5b00c2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:05.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.930+0000 7fc5b7246640 1 --2- 172.21.15.89:0/3674175971 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5b0155bb0 0x7fc5b00c2c00 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fc5b013f8d0 tx=0x7fc5a8011b70 comp rx=0 tx=0).stop 2023-10-30T12:51:05.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.930+0000 7fc5b7246640 1 -- 172.21.15.89:0/3674175971 shutdown_connections 2023-10-30T12:51:05.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.930+0000 7fc5b7246640 1 --2- 172.21.15.89:0/3674175971 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc59c0664d0 0x7fc59c068990 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:05.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.930+0000 7fc5b7246640 1 --2- 172.21.15.89:0/3674175971 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5b0155bb0 0x7fc5b00c2c00 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:05.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.930+0000 7fc5b7246640 1 --2- 172.21.15.89:0/3674175971 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5b00c2740 0x7fc5b0155670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:05.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.930+0000 7fc5b7246640 1 --2- 172.21.15.89:0/3674175971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc598006140 0x7fc5b0138560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:05.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.930+0000 7fc5b7246640 1 -- 172.21.15.89:0/3674175971 >> 172.21.15.89:0/3674175971 conn(0x7fc5b00c7910 msgr2=0x7fc5b00bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:05.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.930+0000 7fc5b7246640 1 -- 172.21.15.89:0/3674175971 shutdown_connections 2023-10-30T12:51:05.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:05.930+0000 7fc5b7246640 1 -- 172.21.15.89:0/3674175971 wait complete. 2023-10-30T12:51:05.949 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 114, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:50.729245+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 114, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5728": { "addr": "172.21.15.192:6839/3991891597", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6838", "nonce": 3991891597, "type": "v2" }, { "addr": "172.21.15.192:6839", "nonce": 3991891597, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 5728, "incarnation": 73, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 19 }, "gid_6075": { "addr": "172.21.15.89:6837/2412618858", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6836", "nonce": 2412618858, "type": "v2" }, { "addr": "172.21.15.89:6837", "nonce": 2412618858, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 6075, "incarnation": 92, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 38 }, "gid_6271": { "addr": "172.21.15.192:6835/659375964", "addrs": { "addrvec": [ { "addr": "172.21.15.192:6834", "nonce": 659375964, "type": "v2" }, { "addr": "172.21.15.192:6835", "nonce": 659375964, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6271, "incarnation": 110, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:rejoin", "state_seq": 25 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:51:01.553267+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6271, "mds_1": 6075, "mds_2": 5728 } } } ], "standbys": [ { "addr": "172.21.15.89:6834/2909790173", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6832", "nonce": 2909790173, "type": "v2" }, { "addr": "172.21.15.89:6834", "nonce": 2909790173, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4236, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.89:6835/2387357615", "addrs": { "addrvec": [ { "addr": "172.21.15.89:6833", "nonce": 2387357615, "type": "v2" }, { "addr": "172.21.15.89:6835", "nonce": 2387357615, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4239, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:51:07.951 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:08.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.098+0000 7ff12cc68640 1 Processor -- start 2023-10-30T12:51:08.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.098+0000 7ff12cc68640 1 -- start start 2023-10-30T12:51:08.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.098+0000 7ff12cc68640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff1280bfec0 0x7ff12806f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:08.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.098+0000 7ff12cc68640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff128159e60 0x7ff1280c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:08.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.098+0000 7ff12cc68640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff12806fa80 con 0x7ff128155d20 2023-10-30T12:51:08.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.098+0000 7ff12cc68640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff128071290 con 0x7ff128159e60 2023-10-30T12:51:08.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.098+0000 7ff12cc68640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff128071570 con 0x7ff1280bfec0 2023-10-30T12:51:08.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.098+0000 7ff11dd9b640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7ff128159e60 0x7ff1280c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:08.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.098+0000 7ff126d9d640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7ff128155d20 0x7ff128156190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57002/0 (socket says 172.21.15.89:57002) 2023-10-30T12:51:08.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.098+0000 7ff126d9d640 1 -- 172.21.15.89:0/1711323968 learned_addr learned my addr 172.21.15.89:0/1711323968 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:51:08.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.098+0000 7ff12659c640 1 --2- 172.21.15.89:0/1711323968 >> v2:172.21.15.192:3300/0 conn(0x7ff1280bfec0 0x7ff12806f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:08.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.102+0000 7ff11dd9b640 1 -- 172.21.15.89:0/1711323968 >> v2:172.21.15.192:3300/0 conn(0x7ff1280bfec0 msgr2=0x7ff12806f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:08.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.102+0000 7ff11dd9b640 1 --2- 172.21.15.89:0/1711323968 >> v2:172.21.15.192:3300/0 conn(0x7ff1280bfec0 0x7ff12806f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:08.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.102+0000 7ff11dd9b640 1 -- 172.21.15.89:0/1711323968 >> v1:172.21.15.89:6789/0 conn(0x7ff128155d20 legacy=0x7ff128156190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:08.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.102+0000 7ff11dd9b640 1 -- 172.21.15.89:0/1711323968 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff128071850 con 0x7ff128159e60 2023-10-30T12:51:08.102 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.102+0000 7ff125d9b640 1 -- 172.21.15.89:0/1711323968 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1954717784 0 0) 0x7ff12806fa80 con 0x7ff128155d20 2023-10-30T12:51:08.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.102+0000 7ff11dd9b640 1 --2- 172.21.15.89:0/1711323968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff128159e60 0x7ff1280c1a40 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7ff1280c8950 tx=0x7ff11801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a37ac51b14632a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:08.103 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.102+0000 7ff125d9b640 1 -- 172.21.15.89:0/1711323968 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1180a8950 con 0x7ff128159e60 2023-10-30T12:51:08.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.102+0000 7ff125d9b640 1 -- 172.21.15.89:0/1711323968 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1180a8ab0 con 0x7ff128159e60 2023-10-30T12:51:08.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.102+0000 7ff125d9b640 1 -- 172.21.15.89:0/1711323968 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1180a8e40 con 0x7ff128159e60 2023-10-30T12:51:08.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.102+0000 7ff12cc68640 1 -- 172.21.15.89:0/1711323968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff128159e60 msgr2=0x7ff1280c1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:08.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.102+0000 7ff12cc68640 1 --2- 172.21.15.89:0/1711323968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff128159e60 0x7ff1280c1a40 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7ff1280c8950 tx=0x7ff11801ec50 comp rx=0 tx=0).stop 2023-10-30T12:51:08.104 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.102+0000 7ff12cc68640 1 -- 172.21.15.89:0/1711323968 shutdown_connections 2023-10-30T12:51:08.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.102+0000 7ff12cc68640 1 --2- 172.21.15.89:0/1711323968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff128159e60 0x7ff1280c1a40 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:08.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.102+0000 7ff12cc68640 1 --2- 172.21.15.89:0/1711323968 >> v2:172.21.15.192:3300/0 conn(0x7ff1280bfec0 0x7ff12806f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:08.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.102+0000 7ff12cc68640 1 -- 172.21.15.89:0/1711323968 >> 172.21.15.89:0/1711323968 conn(0x7ff1280c7910 msgr2=0x7ff1280c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:08.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.102+0000 7ff12cc68640 1 -- 172.21.15.89:0/1711323968 shutdown_connections 2023-10-30T12:51:08.105 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.102+0000 7ff12cc68640 1 -- 172.21.15.89:0/1711323968 wait complete. 2023-10-30T12:51:08.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.106+0000 7ff12cc68640 1 Processor -- start 2023-10-30T12:51:08.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.106+0000 7ff12cc68640 1 -- start start 2023-10-30T12:51:08.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.106+0000 7ff12cc68640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff1280bfec0 0x7ff12806a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:08.108 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.106+0000 7ff12cc68640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff128155d20 0x7ff128157780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:08.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.106+0000 7ff12cc68640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff128159e60 0x7ff1281e2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:08.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.106+0000 7ff12cc68640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1281e3bf0 con 0x7ff128159e60 2023-10-30T12:51:08.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.106+0000 7ff12cc68640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1281572f0 con 0x7ff128155d20 2023-10-30T12:51:08.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.106+0000 7ff12cc68640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff12806aec0 con 0x7ff1280bfec0 2023-10-30T12:51:08.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.106+0000 7ff126d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff128159e60 0x7ff1281e2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:08.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.106+0000 7ff126d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff128159e60 0x7ff1281e2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55180/0 (socket says 172.21.15.89:55180) 2023-10-30T12:51:08.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.106+0000 7ff126d9d640 1 -- 172.21.15.89:0/3001867633 learned_addr learned my addr 172.21.15.89:0/3001867633 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:08.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.106+0000 7ff12659c640 1 --2- 172.21.15.89:0/3001867633 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff1280bfec0 0x7ff12806a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:08.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.106+0000 7ff11dd9b640 1 --2- 172.21.15.89:0/3001867633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff128155d20 0x7ff128157780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:08.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.106+0000 7ff126d9d640 1 -- 172.21.15.89:0/3001867633 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff1280bfec0 msgr2=0x7ff12806a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:08.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.106+0000 7ff126d9d640 1 --2- 172.21.15.89:0/3001867633 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff1280bfec0 0x7ff12806a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:08.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.106+0000 7ff126d9d640 1 -- 172.21.15.89:0/3001867633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff128155d20 msgr2=0x7ff128157780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:08.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.106+0000 7ff126d9d640 1 --2- 172.21.15.89:0/3001867633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff128155d20 0x7ff128157780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:08.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.106+0000 7ff126d9d640 1 -- 172.21.15.89:0/3001867633 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff118009420 con 0x7ff128159e60 2023-10-30T12:51:08.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.106+0000 7ff11dd9b640 1 --2- 172.21.15.89:0/3001867633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff128155d20 0x7ff128157780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:08.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.110+0000 7ff126d9d640 1 --2- 172.21.15.89:0/3001867633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff128159e60 0x7ff1281e2430 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7ff12000c810 tx=0x7ff12000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:08.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.110+0000 7ff11f7fe640 1 -- 172.21.15.89:0/3001867633 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12000a640 con 0x7ff128159e60 2023-10-30T12:51:08.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.110+0000 7ff11f7fe640 1 -- 172.21.15.89:0/3001867633 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff12000a7c0 con 0x7ff128159e60 2023-10-30T12:51:08.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.110+0000 7ff11f7fe640 1 -- 172.21.15.89:0/3001867633 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12000ab70 con 0x7ff128159e60 2023-10-30T12:51:08.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.110+0000 7ff12cc68640 1 -- 172.21.15.89:0/3001867633 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1281e3670 con 0x7ff128159e60 2023-10-30T12:51:08.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.110+0000 7ff12cc68640 1 -- 172.21.15.89:0/3001867633 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff128156f80 con 0x7ff128159e60 2023-10-30T12:51:08.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.110+0000 7ff12cc68640 1 -- 172.21.15.89:0/3001867633 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff12804e170 con 0x7ff128159e60 2023-10-30T12:51:08.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.110+0000 7ff11f7fe640 1 -- 172.21.15.89:0/3001867633 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff12000acd0 con 0x7ff128159e60 2023-10-30T12:51:08.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.110+0000 7ff11f7fe640 1 --2- 172.21.15.89:0/3001867633 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff1080664a0 0x7ff108068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:08.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.110+0000 7ff11f7fe640 1 -- 172.21.15.89:0/3001867633 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7ff1200836c0 con 0x7ff128159e60 2023-10-30T12:51:08.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.110+0000 7ff12659c640 1 --2- 172.21.15.89:0/3001867633 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff1080664a0 0x7ff108068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:08.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.114+0000 7ff12659c640 1 --2- 172.21.15.89:0/3001867633 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff1080664a0 0x7ff108068960 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff110007a60 tx=0x7ff110008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:08.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.114+0000 7ff11f7fe640 1 -- 172.21.15.89:0/3001867633 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff1200507d0 con 0x7ff128159e60 2023-10-30T12:51:08.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.294+0000 7ff12cc68640 1 -- 172.21.15.89:0/3001867633 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff128157430 con 0x7ff128159e60 2023-10-30T12:51:08.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.294+0000 7ff11f7fe640 1 -- 172.21.15.89:0/3001867633 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+5751 (secure 0 0 0) 0x7ff120053df0 con 0x7ff128159e60 2023-10-30T12:51:08.295 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:51:08.296 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:51:07.431790+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6271,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6271":{"gid":6271,"name":"b","rank":0,"incarnation":110,"state":"up:clientreplay","state_seq":27,"addr":"172.21.15.192:6835/659375964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":659375964},{"type":"v1","addr":"172.21.15.192:6835","nonce":659375964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:08.296 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 115 2023-10-30T12:51:08.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.298+0000 7ff12cc68640 1 -- 172.21.15.89:0/3001867633 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff1080664a0 msgr2=0x7ff108068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:08.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.298+0000 7ff12cc68640 1 --2- 172.21.15.89:0/3001867633 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff1080664a0 0x7ff108068960 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff110007a60 tx=0x7ff110008040 comp rx=0 tx=0).stop 2023-10-30T12:51:08.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.298+0000 7ff12cc68640 1 -- 172.21.15.89:0/3001867633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff128159e60 msgr2=0x7ff1281e2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:08.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.298+0000 7ff12cc68640 1 --2- 172.21.15.89:0/3001867633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff128159e60 0x7ff1281e2430 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7ff12000c810 tx=0x7ff12000ccd0 comp rx=0 tx=0).stop 2023-10-30T12:51:08.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.298+0000 7ff12cc68640 1 -- 172.21.15.89:0/3001867633 shutdown_connections 2023-10-30T12:51:08.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.298+0000 7ff12cc68640 1 --2- 172.21.15.89:0/3001867633 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff1080664a0 0x7ff108068960 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:08.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.298+0000 7ff12cc68640 1 --2- 172.21.15.89:0/3001867633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff128155d20 0x7ff128157780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:08.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.298+0000 7ff12cc68640 1 --2- 172.21.15.89:0/3001867633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff128159e60 0x7ff1281e2430 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:08.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.298+0000 7ff12cc68640 1 --2- 172.21.15.89:0/3001867633 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff1280bfec0 0x7ff12806a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:08.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.298+0000 7ff12cc68640 1 -- 172.21.15.89:0/3001867633 >> 172.21.15.89:0/3001867633 conn(0x7ff1280c7910 msgr2=0x7ff1280c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:08.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.298+0000 7ff12cc68640 1 -- 172.21.15.89:0/3001867633 shutdown_connections 2023-10-30T12:51:08.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:08.298+0000 7ff12cc68640 1 -- 172.21.15.89:0/3001867633 wait complete. 2023-10-30T12:51:08.316 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:51:10.317 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:10.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.466+0000 7f0ea726b640 1 Processor -- start 2023-10-30T12:51:10.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.466+0000 7f0ea726b640 1 -- start start 2023-10-30T12:51:10.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.466+0000 7f0ea726b640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0ea015d6f0 0x7f0ea015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:10.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.466+0000 7f0ea726b640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ea00cb800 con 0x7f0ea01538a0 2023-10-30T12:51:10.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.466+0000 7f0ea726b640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ea00cb330 con 0x7f0ea00cf160 2023-10-30T12:51:10.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.466+0000 7f0ea726b640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ea00cad70 con 0x7f0ea015d6f0 2023-10-30T12:51:10.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.466+0000 7f0ea5007640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f0ea01538a0 0x7f0ea00ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:60938/0 (socket says 172.21.15.89:60938) 2023-10-30T12:51:10.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.466+0000 7f0ea5007640 1 -- 172.21.15.89:0/3528915156 learned_addr learned my addr 172.21.15.89:0/3528915156 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:51:10.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.466+0000 7f0ea4806640 1 --2- 172.21.15.89:0/3528915156 >> v2:172.21.15.192:3300/0 conn(0x7f0ea015d6f0 0x7f0ea015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:10.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.466+0000 7f0e97fff640 1 -- 172.21.15.89:0/3528915156 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1294902787 0 0) 0x7f0ea00cb800 con 0x7f0ea01538a0 2023-10-30T12:51:10.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.466+0000 7f0e97fff640 1 -- 172.21.15.89:0/3528915156 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e8c003530 con 0x7f0ea01538a0 2023-10-30T12:51:10.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.466+0000 7f0e97fff640 1 -- 172.21.15.89:0/3528915156 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3751587927 0 0) 0x7f0e8c003530 con 0x7f0ea01538a0 2023-10-30T12:51:10.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.466+0000 7f0e97fff640 1 -- 172.21.15.89:0/3528915156 >> v2:172.21.15.192:3300/0 conn(0x7f0ea015d6f0 msgr2=0x7f0ea015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:10.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.466+0000 7f0e97fff640 1 --2- 172.21.15.89:0/3528915156 >> v2:172.21.15.192:3300/0 conn(0x7f0ea015d6f0 0x7f0ea015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:10.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.466+0000 7f0e97fff640 1 -- 172.21.15.89:0/3528915156 >> v1:172.21.15.192:6789/0 conn(0x7f0ea00cf160 legacy=0x7f0ea0164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:10.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.466+0000 7f0e97fff640 1 -- 172.21.15.89:0/3528915156 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea00cb050 con 0x7f0ea01538a0 2023-10-30T12:51:10.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.466+0000 7f0e97fff640 1 -- 172.21.15.89:0/3528915156 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f0e9c001680 con 0x7f0ea01538a0 2023-10-30T12:51:10.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.466+0000 7f0e97fff640 1 -- 172.21.15.89:0/3528915156 >> v1:172.21.15.89:6789/0 conn(0x7f0ea01538a0 legacy=0x7f0ea00ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:10.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.466+0000 7f0e97fff640 1 --2- 172.21.15.89:0/3528915156 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e8c003980 0x7f0e8c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:10.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.466+0000 7f0e97fff640 1 --2- 172.21.15.89:0/3528915156 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e8c0047d0 0x7f0e8c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:10.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.466+0000 7f0e97fff640 1 --2- 172.21.15.89:0/3528915156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e8c005190 0x7f0e8c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:10.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.466+0000 7f0e97fff640 1 -- 172.21.15.89:0/3528915156 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea00cad70 con 0x7f0e8c005190 2023-10-30T12:51:10.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.466+0000 7f0e97fff640 1 -- 172.21.15.89:0/3528915156 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8c00c040 con 0x7f0e8c003980 2023-10-30T12:51:10.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.466+0000 7f0e97fff640 1 -- 172.21.15.89:0/3528915156 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8c00c200 con 0x7f0e8c0047d0 2023-10-30T12:51:10.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.470+0000 7f0ea5808640 1 --2- 172.21.15.89:0/3528915156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e8c005190 0x7f0e8c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:10.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.470+0000 7f0ea5007640 1 --2- 172.21.15.89:0/3528915156 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e8c003980 0x7f0e8c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:10.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.470+0000 7f0ea4806640 1 --2- 172.21.15.89:0/3528915156 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e8c0047d0 0x7f0e8c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:10.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.470+0000 7f0ea5808640 1 -- 172.21.15.89:0/3528915156 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e8c0047d0 msgr2=0x7f0e8c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:10.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.470+0000 7f0ea5808640 1 --2- 172.21.15.89:0/3528915156 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e8c0047d0 0x7f0e8c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:10.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.470+0000 7f0ea5808640 1 -- 172.21.15.89:0/3528915156 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e8c003980 msgr2=0x7f0e8c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:10.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.470+0000 7f0ea5808640 1 --2- 172.21.15.89:0/3528915156 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e8c003980 0x7f0e8c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:10.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.470+0000 7f0ea5808640 1 -- 172.21.15.89:0/3528915156 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e8c003530 con 0x7f0e8c005190 2023-10-30T12:51:10.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.470+0000 7f0ea4806640 1 --2- 172.21.15.89:0/3528915156 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0e8c0047d0 0x7f0e8c004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:10.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.470+0000 7f0ea5808640 1 -- 172.21.15.89:0/3528915156 reap_dead start 2023-10-30T12:51:10.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.470+0000 7f0ea5808640 1 --2- 172.21.15.89:0/3528915156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e8c005190 0x7f0e8c00b990 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f0e98001e70 tx=0x7f0e9801c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=621e3572521675d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:10.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.470+0000 7f0e97fff640 1 -- 172.21.15.89:0/3528915156 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e98003b80 con 0x7f0e8c005190 2023-10-30T12:51:10.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.470+0000 7f0e97fff640 1 -- 172.21.15.89:0/3528915156 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e98003d20 con 0x7f0e8c005190 2023-10-30T12:51:10.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.470+0000 7f0e97fff640 1 -- 172.21.15.89:0/3528915156 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e98005260 con 0x7f0e8c005190 2023-10-30T12:51:10.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.474+0000 7f0ea726b640 1 -- 172.21.15.89:0/3528915156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e8c005190 msgr2=0x7f0e8c00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:10.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.474+0000 7f0ea726b640 1 --2- 172.21.15.89:0/3528915156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e8c005190 0x7f0e8c00b990 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f0e98001e70 tx=0x7f0e9801c760 comp rx=0 tx=0).stop 2023-10-30T12:51:10.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.474+0000 7f0ea726b640 1 -- 172.21.15.89:0/3528915156 shutdown_connections 2023-10-30T12:51:10.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.474+0000 7f0ea726b640 1 --2- 172.21.15.89:0/3528915156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e8c005190 0x7f0e8c00b990 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:10.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.474+0000 7f0ea726b640 1 -- 172.21.15.89:0/3528915156 >> 172.21.15.89:0/3528915156 conn(0x7f0ea006bfe0 msgr2=0x7f0ea006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:10.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.474+0000 7f0ea726b640 1 -- 172.21.15.89:0/3528915156 shutdown_connections 2023-10-30T12:51:10.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.474+0000 7f0ea726b640 1 -- 172.21.15.89:0/3528915156 wait complete. 2023-10-30T12:51:10.478 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.474+0000 7f0ea726b640 1 Processor -- start 2023-10-30T12:51:10.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.474+0000 7f0ea726b640 1 -- start start 2023-10-30T12:51:10.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.474+0000 7f0ea726b640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e8c005190 0x7f0ea00ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:10.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.474+0000 7f0ea726b640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ea00c5490 0x7f0ea014c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:10.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.474+0000 7f0ea726b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ea014c780 0x7f0ea01f01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:10.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.474+0000 7f0ea726b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea01e4f80 con 0x7f0ea014c780 2023-10-30T12:51:10.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.474+0000 7f0ea726b640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea014cc00 con 0x7f0e8c005190 2023-10-30T12:51:10.480 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.474+0000 7f0ea726b640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea01e5740 con 0x7f0ea00c5490 2023-10-30T12:51:10.480 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.474+0000 7f0ea5808640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ea014c780 0x7f0ea01f01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:10.480 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.474+0000 7f0ea5808640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ea014c780 0x7f0ea01f01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34558/0 (socket says 172.21.15.89:34558) 2023-10-30T12:51:10.480 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.474+0000 7f0ea5808640 1 -- 172.21.15.89:0/866452466 learned_addr learned my addr 172.21.15.89:0/866452466 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:10.480 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.474+0000 7f0ea4806640 1 --2- 172.21.15.89:0/866452466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ea00c5490 0x7f0ea014c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:10.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.474+0000 7f0ea5007640 1 --2- 172.21.15.89:0/866452466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e8c005190 0x7f0ea00ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:10.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.474+0000 7f0ea5808640 1 -- 172.21.15.89:0/866452466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ea00c5490 msgr2=0x7f0ea014c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:10.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.474+0000 7f0ea5808640 1 --2- 172.21.15.89:0/866452466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ea00c5490 0x7f0ea014c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:10.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.474+0000 7f0ea5808640 1 -- 172.21.15.89:0/866452466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e8c005190 msgr2=0x7f0ea00ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:10.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.474+0000 7f0ea5808640 1 --2- 172.21.15.89:0/866452466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e8c005190 0x7f0ea00ce390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:10.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.474+0000 7f0ea5808640 1 -- 172.21.15.89:0/866452466 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea00b3ae0 con 0x7f0ea014c780 2023-10-30T12:51:10.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.478+0000 7f0ea5808640 1 --2- 172.21.15.89:0/866452466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ea014c780 0x7f0ea01f01e0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f0e9801cc40 tx=0x7f0e8c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:10.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.478+0000 7f0e95ffb640 1 -- 172.21.15.89:0/866452466 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e980ac070 con 0x7f0ea014c780 2023-10-30T12:51:10.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.478+0000 7f0e95ffb640 1 -- 172.21.15.89:0/866452466 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e98007dc0 con 0x7f0ea014c780 2023-10-30T12:51:10.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.478+0000 7f0ea726b640 1 -- 172.21.15.89:0/866452466 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ea00b45d0 con 0x7f0ea014c780 2023-10-30T12:51:10.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.478+0000 7f0e95ffb640 1 -- 172.21.15.89:0/866452466 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e98006d60 con 0x7f0ea014c780 2023-10-30T12:51:10.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.478+0000 7f0ea726b640 1 -- 172.21.15.89:0/866452466 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ea00b3d30 con 0x7f0ea014c780 2023-10-30T12:51:10.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.478+0000 7f0ea726b640 1 -- 172.21.15.89:0/866452466 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ea004e170 con 0x7f0ea014c780 2023-10-30T12:51:10.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.478+0000 7f0e95ffb640 1 -- 172.21.15.89:0/866452466 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0e98007030 con 0x7f0ea014c780 2023-10-30T12:51:10.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.478+0000 7f0e95ffb640 1 --2- 172.21.15.89:0/866452466 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0e840664a0 0x7f0e84068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:10.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.478+0000 7f0e95ffb640 1 -- 172.21.15.89:0/866452466 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f0e9811d280 con 0x7f0ea014c780 2023-10-30T12:51:10.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.482+0000 7f0ea5007640 1 --2- 172.21.15.89:0/866452466 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0e840664a0 0x7f0e84068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:10.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.482+0000 7f0ea5007640 1 --2- 172.21.15.89:0/866452466 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0e840664a0 0x7f0e84068960 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c007d30 tx=0x7f0e9c000ee0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:10.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.482+0000 7f0e95ffb640 1 -- 172.21.15.89:0/866452466 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0e980ea3a0 con 0x7f0ea014c780 2023-10-30T12:51:10.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.666+0000 7f0ea726b640 1 -- 172.21.15.89:0/866452466 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ea01530d0 con 0x7f0ea014c780 2023-10-30T12:51:10.665 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.666+0000 7f0e95ffb640 1 -- 172.21.15.89:0/866452466 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+5751 (secure 0 0 0) 0x7f0e980ed9c0 con 0x7f0ea014c780 2023-10-30T12:51:10.666 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:51:10.666 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:51:07.431790+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6271,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6271":{"gid":6271,"name":"b","rank":0,"incarnation":110,"state":"up:clientreplay","state_seq":27,"addr":"172.21.15.192:6835/659375964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":659375964},{"type":"v1","addr":"172.21.15.192:6835","nonce":659375964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:10.666 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 115 2023-10-30T12:51:10.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.670+0000 7f0ea726b640 1 -- 172.21.15.89:0/866452466 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0e840664a0 msgr2=0x7f0e84068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:10.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.670+0000 7f0ea726b640 1 --2- 172.21.15.89:0/866452466 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0e840664a0 0x7f0e84068960 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c007d30 tx=0x7f0e9c000ee0 comp rx=0 tx=0).stop 2023-10-30T12:51:10.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.670+0000 7f0ea726b640 1 -- 172.21.15.89:0/866452466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ea014c780 msgr2=0x7f0ea01f01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:10.668 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.670+0000 7f0ea726b640 1 --2- 172.21.15.89:0/866452466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ea014c780 0x7f0ea01f01e0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f0e9801cc40 tx=0x7f0e8c005800 comp rx=0 tx=0).stop 2023-10-30T12:51:10.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.670+0000 7f0ea726b640 1 -- 172.21.15.89:0/866452466 shutdown_connections 2023-10-30T12:51:10.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.670+0000 7f0ea726b640 1 --2- 172.21.15.89:0/866452466 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0e840664a0 0x7f0e84068960 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:10.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.670+0000 7f0ea726b640 1 --2- 172.21.15.89:0/866452466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ea014c780 0x7f0ea01f01e0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:10.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.670+0000 7f0ea726b640 1 --2- 172.21.15.89:0/866452466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ea00c5490 0x7f0ea014c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:10.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.670+0000 7f0ea726b640 1 --2- 172.21.15.89:0/866452466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0e8c005190 0x7f0ea00ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:10.669 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.670+0000 7f0ea726b640 1 -- 172.21.15.89:0/866452466 >> 172.21.15.89:0/866452466 conn(0x7f0ea006bfe0 msgr2=0x7f0ea0157740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:10.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.670+0000 7f0ea726b640 1 -- 172.21.15.89:0/866452466 shutdown_connections 2023-10-30T12:51:10.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:10.670+0000 7f0ea726b640 1 -- 172.21.15.89:0/866452466 wait complete. 2023-10-30T12:51:10.686 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:51:12.688 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:12.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.830+0000 7f51faa91640 1 Processor -- start 2023-10-30T12:51:12.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.830+0000 7f51faa91640 1 -- start start 2023-10-30T12:51:12.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.834+0000 7f51faa91640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51f40c6b80 0x7f51f40bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:12.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.834+0000 7f51faa91640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f51f413f4a0 0x7f51f40c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:12.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.834+0000 7f51faa91640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f40c0a50 con 0x7f51f40c6f60 2023-10-30T12:51:12.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.834+0000 7f51faa91640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51f406f6e0 con 0x7f51f413f4a0 2023-10-30T12:51:12.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.834+0000 7f51faa91640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51f406f9c0 con 0x7f51f40c6b80 2023-10-30T12:51:12.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.834+0000 7f51f902e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f51f413f4a0 0x7f51f40c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:12.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.834+0000 7f51f882d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51f40c6b80 0x7f51f40bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:12.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.834+0000 7f51f902e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f51f413f4a0 0x7f51f40c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:38116/0 (socket says 172.21.15.89:38116) 2023-10-30T12:51:12.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.834+0000 7f51f882d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51f40c6b80 0x7f51f40bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:60608/0 (socket says 172.21.15.89:60608) 2023-10-30T12:51:12.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.834+0000 7f51f902e640 1 -- 172.21.15.89:0/483702021 learned_addr learned my addr 172.21.15.89:0/483702021 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:12.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.834+0000 7f51f882d640 1 -- 172.21.15.89:0/483702021 >> v2:172.21.15.192:3300/0 conn(0x7f51f413f4a0 msgr2=0x7f51f40c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:12.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.834+0000 7f51f882d640 1 --2- 172.21.15.89:0/483702021 >> v2:172.21.15.192:3300/0 conn(0x7f51f413f4a0 0x7f51f40c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:12.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.834+0000 7f51f882d640 1 -- 172.21.15.89:0/483702021 >> v1:172.21.15.192:6789/0 conn(0x7f51f40c6f60 legacy=0x7f51f413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:12.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.834+0000 7f51f882d640 1 -- 172.21.15.89:0/483702021 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f406fca0 con 0x7f51f40c6b80 2023-10-30T12:51:12.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.834+0000 7f51f902e640 1 --2- 172.21.15.89:0/483702021 >> v2:172.21.15.192:3300/0 conn(0x7f51f413f4a0 0x7f51f40c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:51:12.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.838+0000 7f51f882d640 1 --2- 172.21.15.89:0/483702021 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51f40c6b80 0x7f51f40bfec0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f51e80097c0 tx=0x7f51e801c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b28385a9676d23cc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:12.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.838+0000 7f51f37fe640 1 -- 172.21.15.89:0/483702021 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e80ad640 con 0x7f51f40c6b80 2023-10-30T12:51:12.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.838+0000 7f51f37fe640 1 -- 172.21.15.89:0/483702021 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51e80ad7a0 con 0x7f51f40c6b80 2023-10-30T12:51:12.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.838+0000 7f51f37fe640 1 -- 172.21.15.89:0/483702021 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e80adb30 con 0x7f51f40c6b80 2023-10-30T12:51:12.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.838+0000 7f51faa91640 1 -- 172.21.15.89:0/483702021 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51f40c6b80 msgr2=0x7f51f40bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:12.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.838+0000 7f51faa91640 1 --2- 172.21.15.89:0/483702021 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51f40c6b80 0x7f51f40bfec0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f51e80097c0 tx=0x7f51e801c480 comp rx=0 tx=0).stop 2023-10-30T12:51:12.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.838+0000 7f51faa91640 1 -- 172.21.15.89:0/483702021 shutdown_connections 2023-10-30T12:51:12.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.838+0000 7f51faa91640 1 --2- 172.21.15.89:0/483702021 >> v2:172.21.15.192:3300/0 conn(0x7f51f413f4a0 0x7f51f40c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:12.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.838+0000 7f51faa91640 1 --2- 172.21.15.89:0/483702021 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51f40c6b80 0x7f51f40bfec0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:12.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.838+0000 7f51faa91640 1 -- 172.21.15.89:0/483702021 >> 172.21.15.89:0/483702021 conn(0x7f51f40c7910 msgr2=0x7f51f40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:12.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.838+0000 7f51faa91640 1 -- 172.21.15.89:0/483702021 shutdown_connections 2023-10-30T12:51:12.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.838+0000 7f51faa91640 1 -- 172.21.15.89:0/483702021 wait complete. 2023-10-30T12:51:12.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.842+0000 7f51faa91640 1 Processor -- start 2023-10-30T12:51:12.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.842+0000 7f51faa91640 1 -- start start 2023-10-30T12:51:12.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.842+0000 7f51faa91640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51f40c6b80 0x7f51f41f5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:12.843 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.842+0000 7f51faa91640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51f40c6f60 0x7f51f41f60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:12.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.842+0000 7f51faa91640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51f413f4a0 0x7f51f40c9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:12.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.842+0000 7f51faa91640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f41564f0 con 0x7f51f413f4a0 2023-10-30T12:51:12.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.842+0000 7f51faa91640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f4155fa0 con 0x7f51f40c6f60 2023-10-30T12:51:12.844 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.842+0000 7f51faa91640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51f40c8db0 con 0x7f51f40c6b80 2023-10-30T12:51:12.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.842+0000 7f51f902e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51f413f4a0 0x7f51f40c9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:12.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.842+0000 7f51f902e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51f413f4a0 0x7f51f40c9570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34564/0 (socket says 172.21.15.89:34564) 2023-10-30T12:51:12.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.842+0000 7f51f902e640 1 -- 172.21.15.89:0/3583051579 learned_addr learned my addr 172.21.15.89:0/3583051579 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:12.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.842+0000 7f51f882d640 1 --2- 172.21.15.89:0/3583051579 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51f40c6b80 0x7f51f41f5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:12.845 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.842+0000 7f51f3fff640 1 --2- 172.21.15.89:0/3583051579 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51f40c6f60 0x7f51f41f60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:12.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.842+0000 7f51f902e640 1 -- 172.21.15.89:0/3583051579 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51f40c6b80 msgr2=0x7f51f41f5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:12.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.842+0000 7f51f902e640 1 --2- 172.21.15.89:0/3583051579 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51f40c6b80 0x7f51f41f5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:12.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.842+0000 7f51f902e640 1 -- 172.21.15.89:0/3583051579 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51f40c6f60 msgr2=0x7f51f41f60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:12.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.842+0000 7f51f902e640 1 --2- 172.21.15.89:0/3583051579 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51f40c6f60 0x7f51f41f60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:12.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.842+0000 7f51f902e640 1 -- 172.21.15.89:0/3583051579 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51ec009b30 con 0x7f51f413f4a0 2023-10-30T12:51:12.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.842+0000 7f51f3fff640 1 --2- 172.21.15.89:0/3583051579 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51f40c6f60 0x7f51f41f60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:12.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.842+0000 7f51f902e640 1 --2- 172.21.15.89:0/3583051579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51f413f4a0 0x7f51f40c9570 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f51ec00ef90 tx=0x7f51ec00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:12.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.842+0000 7f51f17fa640 1 -- 172.21.15.89:0/3583051579 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51ec00cd20 con 0x7f51f413f4a0 2023-10-30T12:51:12.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.842+0000 7f51f17fa640 1 -- 172.21.15.89:0/3583051579 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51ec00f040 con 0x7f51f413f4a0 2023-10-30T12:51:12.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.842+0000 7f51f17fa640 1 -- 172.21.15.89:0/3583051579 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51ec00cea0 con 0x7f51f413f4a0 2023-10-30T12:51:12.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.842+0000 7f51faa91640 1 -- 172.21.15.89:0/3583051579 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51e8009420 con 0x7f51f413f4a0 2023-10-30T12:51:12.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.842+0000 7f51faa91640 1 -- 172.21.15.89:0/3583051579 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51f40c9050 con 0x7f51f413f4a0 2023-10-30T12:51:12.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.842+0000 7f51faa91640 1 -- 172.21.15.89:0/3583051579 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51f40c8c10 con 0x7f51f413f4a0 2023-10-30T12:51:12.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.846+0000 7f51f17fa640 1 -- 172.21.15.89:0/3583051579 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f51ec003be0 con 0x7f51f413f4a0 2023-10-30T12:51:12.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.846+0000 7f51f17fa640 1 --2- 172.21.15.89:0/3583051579 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f51d80664a0 0x7f51d8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:12.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.846+0000 7f51f17fa640 1 -- 172.21.15.89:0/3583051579 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f51ec084880 con 0x7f51f413f4a0 2023-10-30T12:51:12.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.846+0000 7f51f882d640 1 --2- 172.21.15.89:0/3583051579 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f51d80664a0 0x7f51d8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:12.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.846+0000 7f51f882d640 1 --2- 172.21.15.89:0/3583051579 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f51d80664a0 0x7f51d8068960 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f51e80af000 tx=0x7f51e8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:12.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:12.850+0000 7f51f17fa640 1 -- 172.21.15.89:0/3583051579 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f51f40c8c10 con 0x7f51f413f4a0 2023-10-30T12:51:13.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:13.030+0000 7f51faa91640 1 -- 172.21.15.89:0/3583051579 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51f40c8c10 con 0x7f51f413f4a0 2023-10-30T12:51:13.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:13.030+0000 7f51f17fa640 1 -- 172.21.15.89:0/3583051579 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+5751 (secure 0 0 0) 0x7f51f40c8c10 con 0x7f51f413f4a0 2023-10-30T12:51:13.029 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:51:13.029 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:51:07.431790+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6271,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6271":{"gid":6271,"name":"b","rank":0,"incarnation":110,"state":"up:clientreplay","state_seq":27,"addr":"172.21.15.192:6835/659375964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":659375964},{"type":"v1","addr":"172.21.15.192:6835","nonce":659375964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:13.030 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 115 2023-10-30T12:51:13.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:13.030+0000 7f51faa91640 1 -- 172.21.15.89:0/3583051579 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f51d80664a0 msgr2=0x7f51d8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:13.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:13.030+0000 7f51faa91640 1 --2- 172.21.15.89:0/3583051579 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f51d80664a0 0x7f51d8068960 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f51e80af000 tx=0x7f51e8008040 comp rx=0 tx=0).stop 2023-10-30T12:51:13.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:13.030+0000 7f51faa91640 1 -- 172.21.15.89:0/3583051579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51f413f4a0 msgr2=0x7f51f40c9570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:13.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:13.030+0000 7f51faa91640 1 --2- 172.21.15.89:0/3583051579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51f413f4a0 0x7f51f40c9570 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f51ec00ef90 tx=0x7f51ec00c490 comp rx=0 tx=0).stop 2023-10-30T12:51:13.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:13.030+0000 7f51faa91640 1 -- 172.21.15.89:0/3583051579 shutdown_connections 2023-10-30T12:51:13.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:13.030+0000 7f51faa91640 1 --2- 172.21.15.89:0/3583051579 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f51d80664a0 0x7f51d8068960 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:13.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:13.030+0000 7f51faa91640 1 --2- 172.21.15.89:0/3583051579 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51f40c6f60 0x7f51f41f60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:13.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:13.030+0000 7f51faa91640 1 --2- 172.21.15.89:0/3583051579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51f413f4a0 0x7f51f40c9570 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:13.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:13.030+0000 7f51faa91640 1 --2- 172.21.15.89:0/3583051579 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f51f40c6b80 0x7f51f41f5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:13.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:13.030+0000 7f51faa91640 1 -- 172.21.15.89:0/3583051579 >> 172.21.15.89:0/3583051579 conn(0x7f51f40c7910 msgr2=0x7f51f4069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:13.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:13.030+0000 7f51faa91640 1 -- 172.21.15.89:0/3583051579 shutdown_connections 2023-10-30T12:51:13.033 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:13.030+0000 7f51faa91640 1 -- 172.21.15.89:0/3583051579 wait complete. 2023-10-30T12:51:13.049 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:51:15.050 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:15.197 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.198+0000 7f1a31fc4640 1 Processor -- start 2023-10-30T12:51:15.198 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.198+0000 7f1a31fc4640 1 -- start start 2023-10-30T12:51:15.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.198+0000 7f1a31fc4640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1a2c06f7a0 0x7f1a2c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:15.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.198+0000 7f1a31fc4640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2c066cf0 con 0x7f1a2c0c6bd0 2023-10-30T12:51:15.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.198+0000 7f1a31fc4640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2c0c70d0 con 0x7f1a2c071110 2023-10-30T12:51:15.199 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.198+0000 7f1a31fc4640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a2c0700c0 con 0x7f1a2c06f7a0 2023-10-30T12:51:15.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.198+0000 7f1a2b7fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1a2c06f7a0 0x7f1a2c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:15.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.198+0000 7f1a2b7fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f1a2c06f7a0 0x7f1a2c06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34572/0 (socket says 172.21.15.89:34572) 2023-10-30T12:51:15.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.198+0000 7f1a2b7fe640 1 -- 172.21.15.89:0/2998642575 learned_addr learned my addr 172.21.15.89:0/2998642575 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:15.200 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.198+0000 7f1a2affd640 1 --1- 172.21.15.89:0/2998642575 >> v1:172.21.15.89:6789/0 conn(0x7f1a2c0c6bd0 0x7f1a2c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:60950/0 (socket says 172.21.15.89:60950) 2023-10-30T12:51:15.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.202+0000 7f1a2b7fe640 1 -- 172.21.15.89:0/2998642575 >> v1:172.21.15.192:6789/0 conn(0x7f1a2c071110 legacy=0x7f1a2c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:15.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.202+0000 7f1a2b7fe640 1 -- 172.21.15.89:0/2998642575 >> v1:172.21.15.89:6789/0 conn(0x7f1a2c0c6bd0 legacy=0x7f1a2c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:15.201 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.202+0000 7f1a2a7fc640 1 -- 172.21.15.89:0/2998642575 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 902729927 0 0) 0x7f1a2c066cf0 con 0x7f1a2c0c6bd0 2023-10-30T12:51:15.202 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.202+0000 7f1a2b7fe640 1 -- 172.21.15.89:0/2998642575 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2c0c7390 con 0x7f1a2c06f7a0 2023-10-30T12:51:15.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.202+0000 7f1a2b7fe640 1 --2- 172.21.15.89:0/2998642575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a2c06f7a0 0x7f1a2c06fb80 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c009870 tx=0x7f1a1c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba2c237b66ed405b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:15.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.202+0000 7f1a2a7fc640 1 -- 172.21.15.89:0/2998642575 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c0a8a30 con 0x7f1a2c06f7a0 2023-10-30T12:51:15.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.202+0000 7f1a2a7fc640 1 -- 172.21.15.89:0/2998642575 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a1c0a8b90 con 0x7f1a2c06f7a0 2023-10-30T12:51:15.203 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.202+0000 7f1a2a7fc640 1 -- 172.21.15.89:0/2998642575 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c0b0440 con 0x7f1a2c06f7a0 2023-10-30T12:51:15.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.202+0000 7f1a31fc4640 1 -- 172.21.15.89:0/2998642575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a2c06f7a0 msgr2=0x7f1a2c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:15.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.202+0000 7f1a31fc4640 1 --2- 172.21.15.89:0/2998642575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a2c06f7a0 0x7f1a2c06fb80 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c009870 tx=0x7f1a1c01ed30 comp rx=0 tx=0).stop 2023-10-30T12:51:15.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.206+0000 7f1a31fc4640 1 -- 172.21.15.89:0/2998642575 shutdown_connections 2023-10-30T12:51:15.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.206+0000 7f1a31fc4640 1 --2- 172.21.15.89:0/2998642575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a2c06f7a0 0x7f1a2c06fb80 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:15.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.206+0000 7f1a31fc4640 1 -- 172.21.15.89:0/2998642575 >> 172.21.15.89:0/2998642575 conn(0x7f1a2c0c7910 msgr2=0x7f1a2c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:15.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.206+0000 7f1a31fc4640 1 -- 172.21.15.89:0/2998642575 shutdown_connections 2023-10-30T12:51:15.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.206+0000 7f1a31fc4640 1 -- 172.21.15.89:0/2998642575 wait complete. 2023-10-30T12:51:15.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.206+0000 7f1a31fc4640 1 Processor -- start 2023-10-30T12:51:15.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.206+0000 7f1a31fc4640 1 -- start start 2023-10-30T12:51:15.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.206+0000 7f1a31fc4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a2c06f7a0 0x7f1a2c1ee730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:15.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.206+0000 7f1a31fc4640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a2c071110 0x7f1a2c1500e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:15.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.206+0000 7f1a31fc4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a2c0c6bd0 0x7f1a2c156540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:15.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.206+0000 7f1a31fc4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c156120 con 0x7f1a2c0c6bd0 2023-10-30T12:51:15.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.206+0000 7f1a31fc4640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c150620 con 0x7f1a2c06f7a0 2023-10-30T12:51:15.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.206+0000 7f1a31fc4640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c155dd0 con 0x7f1a2c071110 2023-10-30T12:51:15.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.206+0000 7f1a2bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a2c0c6bd0 0x7f1a2c156540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.206+0000 7f1a2b7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a2c06f7a0 0x7f1a2c1ee730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.206+0000 7f1a2affd640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a2c071110 0x7f1a2c1500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.206+0000 7f1a2b7fe640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a2c06f7a0 0x7f1a2c1ee730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:38148/0 (socket says 172.21.15.89:38148) 2023-10-30T12:51:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.206+0000 7f1a2b7fe640 1 -- 172.21.15.89:0/3643290310 learned_addr learned my addr 172.21.15.89:0/3643290310 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.206+0000 7f1a2bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a2c0c6bd0 0x7f1a2c156540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34576/0 (socket says 172.21.15.89:34576) 2023-10-30T12:51:15.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.206+0000 7f1a2bfff640 1 -- 172.21.15.89:0/3643290310 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a2c071110 msgr2=0x7f1a2c1500e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:15.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.206+0000 7f1a2bfff640 1 --2- 172.21.15.89:0/3643290310 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a2c071110 0x7f1a2c1500e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:15.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.206+0000 7f1a2bfff640 1 -- 172.21.15.89:0/3643290310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a2c06f7a0 msgr2=0x7f1a2c1ee730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:15.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.206+0000 7f1a2bfff640 1 --2- 172.21.15.89:0/3643290310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a2c06f7a0 0x7f1a2c1ee730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:15.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.210+0000 7f1a2bfff640 1 -- 172.21.15.89:0/3643290310 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a1c0094d0 con 0x7f1a2c0c6bd0 2023-10-30T12:51:15.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.210+0000 7f1a2bfff640 1 --2- 172.21.15.89:0/3643290310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a2c0c6bd0 0x7f1a2c156540 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f1a2400c730 tx=0x7f1a2400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:15.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.210+0000 7f1a13fff640 1 -- 172.21.15.89:0/3643290310 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a24003b80 con 0x7f1a2c0c6bd0 2023-10-30T12:51:15.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.210+0000 7f1a13fff640 1 -- 172.21.15.89:0/3643290310 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a24003ce0 con 0x7f1a2c0c6bd0 2023-10-30T12:51:15.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.210+0000 7f1a13fff640 1 -- 172.21.15.89:0/3643290310 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a24005310 con 0x7f1a2c0c6bd0 2023-10-30T12:51:15.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.210+0000 7f1a31fc4640 1 -- 172.21.15.89:0/3643290310 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a2c156a80 con 0x7f1a2c0c6bd0 2023-10-30T12:51:15.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.210+0000 7f1a31fc4640 1 -- 172.21.15.89:0/3643290310 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a2c0adcf0 con 0x7f1a2c0c6bd0 2023-10-30T12:51:15.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.210+0000 7f1a31fc4640 1 -- 172.21.15.89:0/3643290310 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a2c156260 con 0x7f1a2c0c6bd0 2023-10-30T12:51:15.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.210+0000 7f1a13fff640 1 -- 172.21.15.89:0/3643290310 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1a24005680 con 0x7f1a2c0c6bd0 2023-10-30T12:51:15.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.210+0000 7f1a13fff640 1 --2- 172.21.15.89:0/3643290310 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1a180664a0 0x7f1a18068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:15.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.210+0000 7f1a13fff640 1 -- 172.21.15.89:0/3643290310 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f1a24084110 con 0x7f1a2c0c6bd0 2023-10-30T12:51:15.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.214+0000 7f1a2b7fe640 1 --2- 172.21.15.89:0/3643290310 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1a180664a0 0x7f1a18068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:15.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.214+0000 7f1a2b7fe640 1 --2- 172.21.15.89:0/3643290310 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1a180664a0 0x7f1a18068960 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c009870 tx=0x7f1a1c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:15.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.214+0000 7f1a13fff640 1 -- 172.21.15.89:0/3643290310 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1a2c156260 con 0x7f1a2c0c6bd0 2023-10-30T12:51:15.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.398+0000 7f1a31fc4640 1 -- 172.21.15.89:0/3643290310 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a2c156260 con 0x7f1a2c0c6bd0 2023-10-30T12:51:15.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.398+0000 7f1a13fff640 1 -- 172.21.15.89:0/3643290310 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+5745 (secure 0 0 0) 0x7f1a2c156260 con 0x7f1a2c0c6bd0 2023-10-30T12:51:15.397 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:51:15.398 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":116,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:51:12.676362+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6271,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6271":{"gid":6271,"name":"b","rank":0,"incarnation":110,"state":"up:active","state_seq":29,"addr":"172.21.15.192:6835/659375964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":659375964},{"type":"v1","addr":"172.21.15.192:6835","nonce":659375964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:15.399 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 116 2023-10-30T12:51:15.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.398+0000 7f1a31fc4640 1 -- 172.21.15.89:0/3643290310 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1a180664a0 msgr2=0x7f1a18068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:15.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.398+0000 7f1a31fc4640 1 --2- 172.21.15.89:0/3643290310 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1a180664a0 0x7f1a18068960 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c009870 tx=0x7f1a1c009040 comp rx=0 tx=0).stop 2023-10-30T12:51:15.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.398+0000 7f1a31fc4640 1 -- 172.21.15.89:0/3643290310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a2c0c6bd0 msgr2=0x7f1a2c156540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:15.399 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.398+0000 7f1a31fc4640 1 --2- 172.21.15.89:0/3643290310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a2c0c6bd0 0x7f1a2c156540 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f1a2400c730 tx=0x7f1a2400cbf0 comp rx=0 tx=0).stop 2023-10-30T12:51:15.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.398+0000 7f1a31fc4640 1 -- 172.21.15.89:0/3643290310 shutdown_connections 2023-10-30T12:51:15.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.398+0000 7f1a31fc4640 1 --2- 172.21.15.89:0/3643290310 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f1a180664a0 0x7f1a18068960 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:15.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.398+0000 7f1a31fc4640 1 --2- 172.21.15.89:0/3643290310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a2c0c6bd0 0x7f1a2c156540 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:15.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.398+0000 7f1a31fc4640 1 --2- 172.21.15.89:0/3643290310 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a2c071110 0x7f1a2c1500e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:15.400 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.398+0000 7f1a31fc4640 1 --2- 172.21.15.89:0/3643290310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a2c06f7a0 0x7f1a2c1ee730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:15.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.398+0000 7f1a31fc4640 1 -- 172.21.15.89:0/3643290310 >> 172.21.15.89:0/3643290310 conn(0x7f1a2c0c7910 msgr2=0x7f1a2c0c61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:15.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.398+0000 7f1a31fc4640 1 -- 172.21.15.89:0/3643290310 shutdown_connections 2023-10-30T12:51:15.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:15.398+0000 7f1a31fc4640 1 -- 172.21.15.89:0/3643290310 wait complete. 2023-10-30T12:51:15.418 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=5678 2023-10-30T12:51:15.419 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.f 2023-10-30T12:51:18.563 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:51:18.567 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:51:18.667 INFO:tasks.workunit.client.0.smithi089.stderr:Updating files: 1% (139/12861) Updating files: 1% (214/12861) Updating files: 2% (258/12861) Updating files: 2% (296/12861) Updating files: 3% (386/12861) Updating files: 3% (417/12861) Updating files: 3% (490/12861) Updating files: 4% (515/12861) Updating files: 4% (585/12861) Updating files: 5% (644/12861) Updating files: 5% (649/12861) Updating files: 5% (681/12861) Updating files: 5% (724/12861) Updating files: 6% (772/12861) Updating files: 6% (775/12861) Updating files: 6% (779/12861) Updating files: 6% (826/12861) Updating files: 6% (871/12861) Updating files: 7% (901/12861) Updating files: 7% (945/12861) Updating files: 7% (1013/12861) Updating files: 8% (1029/12861) Updating files: 8% (1085/12861) Updating files: 8% (1116/12861) Updating files: 8% (1152/12861) Updating files: 9% (1158/12861) Updating files: 9% (1202/12861) Updating files: 9% (1252/12861) Updating files: 9% (1262/12861) Updating files: 10% (1287/12861) Updating files: 10% (1355/12861) Updating files: 11% (1415/12861) Updating files: 11% (1453/12861) Updating files: 11% (1530/12861) Updating files: 11% (1543/12861) Updating files: 12% (1544/12861) Updating files: 12% (1558/12861) Updating files: 12% (1571/12861) Updating files: 12% (1648/12861) Updating files: 13% (1672/12861) Updating files: 13% (1707/12861) Updating files: 13% (1725/12861) Updating files: 13% (1751/12861) Updating files: 13% (1797/12861) Updating files: 14% (1801/12861) Updating files: 14% (1846/12861) Updating files: 14% (1878/12861) Updating files: 14% (1888/12861) Updating files: 14% (1913/12861) Updating files: 15% (1930/12861) Updating files: 15% (2022/12861) Updating files: 15% (2045/12861) Updating files: 16% (2058/12861) Updating files: 16% (2068/12861) Updating files: 16% (2080/12861) Updating files: 16% (2099/12861) Updating files: 16% (2178/12861) Updating files: 17% (2187/12861) Updating files: 17% (2275/12861) Updating files: 17% (2295/12861) Updating files: 18% (2315/12861) Updating files: 18% (2340/12861) Updating files: 18% (2376/12861) Updating files: 18% (2400/12861) Updating files: 18% (2430/12861) Updating files: 19% (2444/12861) Updating files: 19% (2471/12861) Updating files: 19% (2485/12861) Updating files: 20% (2573/12861) Updating files: 20% (2610/12861) Updating files: 20% (2640/12861) Updating files: 20% (2643/12861) Updating files: 20% (2653/12861) Updating files: 20% (2666/12861) Updating files: 20% (2685/12861) Updating files: 21% (2701/12861) Updating files: 21% (2707/12861) Updating files: 21% (2769/12861) Updating files: 21% (2804/12861) Updating files: 22% (2830/12861) Updating files: 22% (2874/12861) Updating files: 23% (2959/12861) Updating files: 23% (2960/12861) Updating files: 23% (3004/12861) Updating files: 23% (3018/12861) Updating files: 24% (3087/12861) Updating files: 24% (3119/12861) Updating files: 24% (3120/12861) Updating files: 24% (3185/12861) Updating files: 25% (3216/12861) Updating files: 25% (3287/12861) Updating files: 26% (3344/12861) Updating files: 26% (3418/12861) Updating files: 26% (3447/12861) Updating files: 27% (3473/12861) Updating files: 27% (3498/12861) Updating files: 28% (3602/12861) Updating files: 28% (3614/12861) Updating files: 28% (3649/12861) Updating files: 28% (3718/12861) Updating files: 29% (3730/12861) Updating files: 29% (3785/12861) Updating files: 29% (3830/12861) Updating files: 30% (3859/12861) Updating files: 30% (3889/12861) Updating files: 30% (3903/12861) Updating files: 30% (3922/12861) Updating files: 30% (3979/12861) Updating files: 31% (3987/12861) Updating files: 31% (4013/12861) Updating files: 31% (4049/12861) Updating files: 32% (4116/12861) Updating files: 32% (4130/12861) Updating files: 32% (4176/12861) Updating files: 32% (4177/12861) Updating files: 32% (4191/12861) Updating files: 32% (4213/12861) Updating files: 32% (4224/12861) Updating files: 32% (4235/12861) Updating files: 32% (4236/12861) Updating files: 33% (4245/12861) Updating files: 33% (4327/12861) Updating files: 34% (4373/12861) Updating files: 34% (4391/12861) Updating files: 34% (4411/12861) Updating files: 34% (4451/12861) Updating files: 35% (4502/12861) Updating files: 35% (4588/12861) Updating files: 35% (4621/12861) Updating files: 36% (4630/12861) Updating files: 36% (4654/12861) Updating files: 36% (4701/12861) Updating files: 37% (4759/12861) Updating files: 37% (4774/12861) Updating files: 38% (4888/12861) Updating files: 38% (4892/12861) Updating files: 38% (4946/12861) Updating files: 38% (4955/12861) Updating files: 39% (5016/12861) Updating files: 39% (5019/12861) Updating files: 39% (5036/12861) Updating files: 39% (5096/12861) Updating files: 39% (5128/12861) Updating files: 40% (5145/12861) Updating files: 40% (5195/12861) Updating files: 40% (5264/12861) Updating files: 41% (5274/12861) Updating files: 41% (5312/12861) Updating files: 41% (5363/12861) Updating files: 42% (5402/12861) Updating files: 42% (5461/12861) Updating files: 43% (5531/12861) Updating files: 43% (5631/12861) Updating files: 44% (5659/12861) Updating files: 44% (5660/12861) Updating files: 44% (5691/12861) Updating files: 44% (5716/12861) Updating files: 45% (5788/12861) Updating files: 45% (5816/12861) Updating files: 46% (5917/12861) Updating files: 46% (5928/12861) Updating files: 47% (6045/12861) Updating files: 47% (6050/12861) Updating files: 47% (6120/12861) Updating files: 48% (6174/12861) Updating files: 48% (6203/12861) Updating files: 48% (6235/12861) Updating files: 48% (6261/12861) Updating files: 48% (6271/12861) Updating files: 49% (6302/12861) Updating files: 49% (6303/12861) Updating files: 49% (6371/12861) Updating files: 49% (6401/12861) Updating files: 49% (6426/12861) Updating files: 49% (6428/12861) Updating files: 50% (6431/12861) Updating files: 50% (6522/12861) Updating files: 50% (6552/12861) Updating files: 51% (6560/12861) Updating files: 51% (6616/12861) Updating files: 52% (6688/12861) Updating files: 52% (6692/12861) Updating files: 53% (6817/12861) Updating files: 53% (6853/12861) Updating files: 53% (6900/12861) Updating files: 54% (6945/12861) Updating files: 54% (7042/12861) Updating files: 55% (7074/12861) Updating files: 55% (7183/12861) Updating files: 56% (7203/12861) Updating files: 57% (7331/12861) Updating files: 57% (7361/12861) Updating files: 57% (7447/12861) Updating files: 57% (7452/12861) Updating files: 58% (7460/12861) Updating files: 58% (7475/12861) Updating files: 58% (7540/12861) Updating files: 58% (7555/12861) Updating files: 59% (7588/12861) Updating files: 59% (7630/12861) Updating files: 59% (7661/12861) Updating files: 59% (7700/12861) Updating files: 60% (7717/12861) Updating files: 60% (7739/12861) Updating files: 60% (7831/12861) Updating files: 61% (7846/12861) Updating files: 61% (7856/12861) Updating files: 61% (7865/12861) Updating files: 61% (7937/12861) Updating files: 62% (7974/12861) Updating files: 62% (8030/12861) Updating files: 62% (8043/12861) Updating files: 62% (8082/12861) Updating files: 62% (8093/12861) Updating files: 63% (8103/12861) Updating files: 63% (8112/12861) Updating files: 63% (8200/12861) Updating files: 63% (8220/12861) Updating files: 64% (8232/12861) Updating files: 64% (8277/12861) Updating files: 64% (8317/12861) Updating files: 65% (8360/12861) Updating files: 65% (8377/12861) Updating files: 65% (8440/12861) Updating files: 66% (8489/12861) Updating files: 66% (8495/12861) Updating files: 66% (8527/12861) Updating files: 66% (8566/12861) Updating files: 66% (8580/12861) Updating files: 67% (8617/12861) Updating files: 67% (8654/12861) Updating files: 67% (8697/12861) Updating files: 68% (8746/12861) Updating files: 68% (8803/12861) Updating files: 68% (8851/12861) Updating files: 68% (8852/12861) Updating files: 68% (8860/12861) Updating files: 68% (8861/12861) Updating files: 68% (8866/12861) Updating files: 69% (8875/12861) Updating files: 69% (8885/12861) Updating files: 69% (8972/12861) Updating files: 69% (8989/12861) Updating files: 70% (9003/12861) Updating files: 70% (9030/12861) Updating files: 70% (9047/12861) Updating files: 70% (9081/12861) Updating files: 70% (9120/12861) Updating files: 71% (9132/12861) Updating files: 71% (9209/12861) Updating files: 71% (9253/12861) Updating files: 71% (9256/12861) Updating files: 72% (9260/12861) Updating files: 72% (9328/12861) Updating files: 73% (9389/12861) Updating files: 73% (9420/12861) Updating files: 73% (9479/12861) Updating files: 73% (9485/12861) Updating files: 73% (9495/12861) Updating files: 73% (9514/12861) Updating files: 74% (9518/12861) Updating files: 74% (9541/12861) Updating files: 74% (9556/12861) Updating files: 74% (9609/12861) Updating files: 74% (9624/12861) Updating files: 75% (9646/12861) Updating files: 75% (9700/12861) Updating files: 76% (9775/12861) Updating files: 76% (9847/12861) Updating files: 76% (9890/12861) Updating files: 77% (9903/12861) Updating files: 78% (10032/12861) Updating files: 78% (10134/12861) Updating files: 78% (10157/12861) Updating files: 79% (10161/12861) Updating files: 79% (10207/12861) Updating files: 79% (10288/12861) Updating files: 80% (10289/12861) Updating files: 80% (10293/12861) Updating files: 80% (10386/12861) Updating files: 81% (10418/12861) Updating files: 81% (10496/12861) Updating files: 82% (10547/12861) Updating files: 82% (10642/12861) Updating files: 83% (10675/12861) Updating files: 83% (10715/12861) Updating files: 84% (10804/12861) Updating files: 84% (10812/12861) Updating files: 85% (10932/12861) Updating files: 85% (10999/12861) Updating files: 85% (11007/12861) Updating files: 86% (11061/12861) Updating files: 86% (11183/12861) Updating files: 87% (11190/12861) Updating files: 87% (11218/12861) Updating files: 87% (11250/12861) Updating files: 88% (11318/12861) Updating files: 88% (11327/12861) Updating files: 88% (11422/12861) Updating files: 89% (11447/12861) Updating files: 89% (11449/12861) Updating files: 89% (11451/12861) Updating files: 89% (11487/12861) Updating files: 89% (11513/12861) Updating files: 89% (11570/12861) Updating files: 90% (11575/12861) Updating files: 90% (11604/12861) Updating files: 90% (11636/12861) Updating files: 91% (11704/12861) Updating files: 91% (11755/12861) Updating files: 91% (11762/12861) Updating files: 92% (11833/12861) Updating files: 92% (11888/12861) Updating files: 93% (11961/12861) Updating files: 94% (12090/12861) Updating files: 94% (12214/12861) Updating files: 95% (12218/12861) Updating files: 95% (12263/12861) Updating files: 95% (12294/12861) Updating files: 96% (12347/12861) error: unable to create file src/crush/mapper.h: File exists 2023-10-30T12:51:22.419 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-10-30T12:51:22.420 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-30T12:51:22.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-30T12:51:22.423 INFO:tasks.ceph.mds.f:Started 2023-10-30T12:51:22.423 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:22.493 INFO:tasks.ceph.mds.f.smithi192.stdout:starting mds.f at 2023-10-30T12:51:22.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.566+0000 7fb3fabd3640 1 Processor -- start 2023-10-30T12:51:22.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.566+0000 7fb3fabd3640 1 -- start start 2023-10-30T12:51:22.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.570+0000 7fb3fabd3640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb3f40c6b80 0x7fb3f40bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:22.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.570+0000 7fb3fabd3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb3f40c6f60 0x7fb3f40c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:22.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.570+0000 7fb3fabd3640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb3f40c8430 0x7fb3f413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:22.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.570+0000 7fb3fabd3640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3f40c0a50 con 0x7fb3f40c6b80 2023-10-30T12:51:22.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.570+0000 7fb3fabd3640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3f406f6e0 con 0x7fb3f40c8430 2023-10-30T12:51:22.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.570+0000 7fb3fabd3640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f406f9c0 con 0x7fb3f40c6f60 2023-10-30T12:51:22.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.570+0000 7fb3f896f640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb3f40c6b80 0x7fb3f40bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:22.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.570+0000 7fb3f896f640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fb3f40c6b80 0x7fb3f40bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40388/0 (socket says 172.21.15.89:40388) 2023-10-30T12:51:22.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.570+0000 7fb3f896f640 1 -- 172.21.15.89:0/1611180839 learned_addr learned my addr 172.21.15.89:0/1611180839 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:22.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.570+0000 7fb3ebfff640 1 --2- 172.21.15.89:0/1611180839 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb3f40c6f60 0x7fb3f40c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:22.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.570+0000 7fb3f9170640 1 --2- 172.21.15.89:0/1611180839 >> v2:172.21.15.192:3300/0 conn(0x7fb3f40c8430 0x7fb3f413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:22.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.570+0000 7fb3f896f640 1 -- 172.21.15.89:0/1611180839 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb3f40c6f60 msgr2=0x7fb3f40c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:22.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.570+0000 7fb3f896f640 1 --2- 172.21.15.89:0/1611180839 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb3f40c6f60 0x7fb3f40c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:22.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.570+0000 7fb3f896f640 1 -- 172.21.15.89:0/1611180839 >> v2:172.21.15.192:3300/0 conn(0x7fb3f40c8430 msgr2=0x7fb3f413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:22.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.570+0000 7fb3f896f640 1 --2- 172.21.15.89:0/1611180839 >> v2:172.21.15.192:3300/0 conn(0x7fb3f40c8430 0x7fb3f413ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:22.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.570+0000 7fb3f896f640 1 -- 172.21.15.89:0/1611180839 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f406fca0 con 0x7fb3f40c6b80 2023-10-30T12:51:22.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.574+0000 7fb3f896f640 1 --2- 172.21.15.89:0/1611180839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3f40c6b80 0x7fb3f40bfec0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fb3f40b0860 tx=0x7fb3f001eb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=731499064e444dab server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:22.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.574+0000 7fb3eb7fe640 1 -- 172.21.15.89:0/1611180839 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f00a7ae0 con 0x7fb3f40c6b80 2023-10-30T12:51:22.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.574+0000 7fb3eb7fe640 1 -- 172.21.15.89:0/1611180839 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f00a7c40 con 0x7fb3f40c6b80 2023-10-30T12:51:22.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.574+0000 7fb3eb7fe640 1 -- 172.21.15.89:0/1611180839 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f00af440 con 0x7fb3f40c6b80 2023-10-30T12:51:22.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.574+0000 7fb3fabd3640 1 -- 172.21.15.89:0/1611180839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3f40c6b80 msgr2=0x7fb3f40bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:22.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.574+0000 7fb3fabd3640 1 --2- 172.21.15.89:0/1611180839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3f40c6b80 0x7fb3f40bfec0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fb3f40b0860 tx=0x7fb3f001eb40 comp rx=0 tx=0).stop 2023-10-30T12:51:22.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.574+0000 7fb3fabd3640 1 -- 172.21.15.89:0/1611180839 shutdown_connections 2023-10-30T12:51:22.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.574+0000 7fb3fabd3640 1 --2- 172.21.15.89:0/1611180839 >> v2:172.21.15.192:3300/0 conn(0x7fb3f40c8430 0x7fb3f413ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:22.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.574+0000 7fb3fabd3640 1 --2- 172.21.15.89:0/1611180839 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb3f40c6f60 0x7fb3f40c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:22.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.574+0000 7fb3fabd3640 1 --2- 172.21.15.89:0/1611180839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3f40c6b80 0x7fb3f40bfec0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:22.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.574+0000 7fb3fabd3640 1 -- 172.21.15.89:0/1611180839 >> 172.21.15.89:0/1611180839 conn(0x7fb3f40c7910 msgr2=0x7fb3f40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:22.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.574+0000 7fb3fabd3640 1 -- 172.21.15.89:0/1611180839 shutdown_connections 2023-10-30T12:51:22.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.574+0000 7fb3fabd3640 1 -- 172.21.15.89:0/1611180839 wait complete. 2023-10-30T12:51:22.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.574+0000 7fb3fabd3640 1 Processor -- start 2023-10-30T12:51:22.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.574+0000 7fb3fabd3640 1 -- start start 2023-10-30T12:51:22.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.578+0000 7fb3fabd3640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb3f40c6b80 0x7fb3f41341d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:22.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.578+0000 7fb3fabd3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3f40c6f60 0x7fb3f4153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:22.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.578+0000 7fb3fabd3640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3f40c8430 0x7fb3f4153c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:22.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.578+0000 7fb3fabd3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f413e280 con 0x7fb3f40c6f60 2023-10-30T12:51:22.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.578+0000 7fb3fabd3640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f4134710 con 0x7fb3f40c8430 2023-10-30T12:51:22.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.578+0000 7fb3fabd3640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f4132900 con 0x7fb3f40c6b80 2023-10-30T12:51:22.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.578+0000 7fb3ebfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3f40c6f60 0x7fb3f4153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:22.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.578+0000 7fb3ebfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3f40c6f60 0x7fb3f4153700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40398/0 (socket says 172.21.15.89:40398) 2023-10-30T12:51:22.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.578+0000 7fb3ebfff640 1 -- 172.21.15.89:0/3506795976 learned_addr learned my addr 172.21.15.89:0/3506795976 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:22.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.578+0000 7fb3f896f640 1 --2- 172.21.15.89:0/3506795976 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb3f40c6b80 0x7fb3f41341d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:22.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.578+0000 7fb3f9170640 1 --2- 172.21.15.89:0/3506795976 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3f40c8430 0x7fb3f4153c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:22.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.578+0000 7fb3ebfff640 1 -- 172.21.15.89:0/3506795976 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb3f40c6b80 msgr2=0x7fb3f41341d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:22.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.578+0000 7fb3ebfff640 1 --2- 172.21.15.89:0/3506795976 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb3f40c6b80 0x7fb3f41341d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:22.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.578+0000 7fb3ebfff640 1 -- 172.21.15.89:0/3506795976 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3f40c8430 msgr2=0x7fb3f4153c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:22.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.578+0000 7fb3ebfff640 1 --2- 172.21.15.89:0/3506795976 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3f40c8430 0x7fb3f4153c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:22.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.578+0000 7fb3ebfff640 1 -- 172.21.15.89:0/3506795976 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f000b770 con 0x7fb3f40c6f60 2023-10-30T12:51:22.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.578+0000 7fb3f896f640 1 --2- 172.21.15.89:0/3506795976 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb3f40c6b80 0x7fb3f41341d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:22.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.578+0000 7fb3ebfff640 1 --2- 172.21.15.89:0/3506795976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3f40c6f60 0x7fb3f4153700 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fb3e400d800 tx=0x7fb3e400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:22.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.578+0000 7fb3e97fa640 1 -- 172.21.15.89:0/3506795976 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e40096a0 con 0x7fb3f40c6f60 2023-10-30T12:51:22.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.578+0000 7fb3e97fa640 1 -- 172.21.15.89:0/3506795976 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3e4009800 con 0x7fb3f40c6f60 2023-10-30T12:51:22.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.578+0000 7fb3e97fa640 1 -- 172.21.15.89:0/3506795976 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e4009b70 con 0x7fb3f40c6f60 2023-10-30T12:51:22.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.578+0000 7fb3fabd3640 1 -- 172.21.15.89:0/3506795976 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3f413e7b0 con 0x7fb3f40c6f60 2023-10-30T12:51:22.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.578+0000 7fb3fabd3640 1 -- 172.21.15.89:0/3506795976 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3f41521b0 con 0x7fb3f40c6f60 2023-10-30T12:51:22.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.578+0000 7fb3fabd3640 1 -- 172.21.15.89:0/3506795976 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3f404e170 con 0x7fb3f40c6f60 2023-10-30T12:51:22.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.578+0000 7fb3e97fa640 1 -- 172.21.15.89:0/3506795976 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb3e4009cd0 con 0x7fb3f40c6f60 2023-10-30T12:51:22.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.578+0000 7fb3e97fa640 1 --2- 172.21.15.89:0/3506795976 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb3e00664a0 0x7fb3e0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:22.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.582+0000 7fb3e97fa640 1 -- 172.21.15.89:0/3506795976 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7fb3e4084a00 con 0x7fb3f40c6f60 2023-10-30T12:51:22.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.582+0000 7fb3f896f640 1 --2- 172.21.15.89:0/3506795976 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb3e00664a0 0x7fb3e0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:22.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.582+0000 7fb3f896f640 1 --2- 172.21.15.89:0/3506795976 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb3e00664a0 0x7fb3e0068960 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb3f00b1000 tx=0x7fb3f00a9070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:22.584 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.586+0000 7fb3e97fa640 1 -- 172.21.15.89:0/3506795976 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb3e4051a60 con 0x7fb3f40c6f60 2023-10-30T12:51:22.766 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.766+0000 7fb3fabd3640 1 -- 172.21.15.89:0/3506795976 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3f40c8150 con 0x7fb3f40c6f60 2023-10-30T12:51:22.767 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.766+0000 7fb3e97fa640 1 -- 172.21.15.89:0/3506795976 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+5745 (secure 0 0 0) 0x7fb3e4055080 con 0x7fb3f40c6f60 2023-10-30T12:51:22.767 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:51:22.767 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":116,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:51:12.676362+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6271,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6271":{"gid":6271,"name":"b","rank":0,"incarnation":110,"state":"up:active","state_seq":29,"addr":"172.21.15.192:6835/659375964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":659375964},{"type":"v1","addr":"172.21.15.192:6835","nonce":659375964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:22.768 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 116 2023-10-30T12:51:22.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.770+0000 7fb3fabd3640 1 -- 172.21.15.89:0/3506795976 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb3e00664a0 msgr2=0x7fb3e0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:22.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.770+0000 7fb3fabd3640 1 --2- 172.21.15.89:0/3506795976 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb3e00664a0 0x7fb3e0068960 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb3f00b1000 tx=0x7fb3f00a9070 comp rx=0 tx=0).stop 2023-10-30T12:51:22.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.770+0000 7fb3fabd3640 1 -- 172.21.15.89:0/3506795976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3f40c6f60 msgr2=0x7fb3f4153700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:22.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.770+0000 7fb3fabd3640 1 --2- 172.21.15.89:0/3506795976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3f40c6f60 0x7fb3f4153700 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fb3e400d800 tx=0x7fb3e400dcc0 comp rx=0 tx=0).stop 2023-10-30T12:51:22.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.770+0000 7fb3fabd3640 1 -- 172.21.15.89:0/3506795976 shutdown_connections 2023-10-30T12:51:22.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.770+0000 7fb3fabd3640 1 --2- 172.21.15.89:0/3506795976 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fb3e00664a0 0x7fb3e0068960 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:22.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.770+0000 7fb3fabd3640 1 --2- 172.21.15.89:0/3506795976 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3f40c8430 0x7fb3f4153c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:22.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.770+0000 7fb3fabd3640 1 --2- 172.21.15.89:0/3506795976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3f40c6f60 0x7fb3f4153700 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:22.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.770+0000 7fb3fabd3640 1 --2- 172.21.15.89:0/3506795976 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb3f40c6b80 0x7fb3f41341d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:22.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.770+0000 7fb3fabd3640 1 -- 172.21.15.89:0/3506795976 >> 172.21.15.89:0/3506795976 conn(0x7fb3f40c7910 msgr2=0x7fb3f40c18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:22.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.770+0000 7fb3fabd3640 1 -- 172.21.15.89:0/3506795976 shutdown_connections 2023-10-30T12:51:22.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:22.770+0000 7fb3fabd3640 1 -- 172.21.15.89:0/3506795976 wait complete. 2023-10-30T12:51:22.787 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2023-10-30T12:51:23.345 INFO:tasks.ceph.mds.b.smithi192.stderr:2023-10-30T12:51:23.344+0000 7f239ddf6640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T12:51:23.345 INFO:tasks.ceph.mds.b.smithi192.stderr:2023-10-30T12:51:23.344+0000 7f239ddf6640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T12:51:23.345 INFO:tasks.ceph.mds.b.smithi192.stderr:2023-10-30T12:51:23.344+0000 7f239ddf6640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T12:51:24.788 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:24.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.930+0000 7f2a6fc17640 1 Processor -- start 2023-10-30T12:51:24.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.930+0000 7f2a6fc17640 1 -- start start 2023-10-30T12:51:24.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.934+0000 7f2a6fc17640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2a68153bf0 0x7f2a68071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:24.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.934+0000 7f2a6fc17640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2a680c5f60 0x7f2a680cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:24.932 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.934+0000 7f2a6fc17640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a680700b0 con 0x7f2a680c6fe0 2023-10-30T12:51:24.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.934+0000 7f2a6fc17640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a680c6810 con 0x7f2a68153bf0 2023-10-30T12:51:24.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.934+0000 7f2a6fc17640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a68159900 con 0x7f2a680c5f60 2023-10-30T12:51:24.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.934+0000 7f2a6d9b3640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2a68153bf0 0x7f2a68071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:24.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.934+0000 7f2a6d9b3640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2a68153bf0 0x7f2a68071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40404/0 (socket says 172.21.15.89:40404) 2023-10-30T12:51:24.933 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.934+0000 7f2a6d9b3640 1 -- 172.21.15.89:0/3318693188 learned_addr learned my addr 172.21.15.89:0/3318693188 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:24.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.934+0000 7f2a6e1b4640 1 --2- 172.21.15.89:0/3318693188 >> v2:172.21.15.192:3300/0 conn(0x7f2a680c5f60 0x7f2a680cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:24.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.934+0000 7f2a6d9b3640 1 -- 172.21.15.89:0/3318693188 >> v2:172.21.15.192:3300/0 conn(0x7f2a680c5f60 msgr2=0x7f2a680cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:24.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.934+0000 7f2a6d9b3640 1 --2- 172.21.15.89:0/3318693188 >> v2:172.21.15.192:3300/0 conn(0x7f2a680c5f60 0x7f2a680cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:24.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.934+0000 7f2a6d9b3640 1 -- 172.21.15.89:0/3318693188 >> v1:172.21.15.192:6789/0 conn(0x7f2a680c6fe0 legacy=0x7f2a680c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:24.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.934+0000 7f2a6d9b3640 1 -- 172.21.15.89:0/3318693188 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a680c6530 con 0x7f2a68153bf0 2023-10-30T12:51:24.935 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.934+0000 7f2a6d9b3640 1 --2- 172.21.15.89:0/3318693188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a68153bf0 0x7f2a68071260 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f2a6806ce90 tx=0x7f2a6401ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22a8d23ae8615b42 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:24.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.938+0000 7f2a6c9b1640 1 -- 172.21.15.89:0/3318693188 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a640a89b0 con 0x7f2a68153bf0 2023-10-30T12:51:24.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.938+0000 7f2a6c9b1640 1 -- 172.21.15.89:0/3318693188 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a640a8b10 con 0x7f2a68153bf0 2023-10-30T12:51:24.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.938+0000 7f2a6c9b1640 1 -- 172.21.15.89:0/3318693188 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a640a8ea0 con 0x7f2a68153bf0 2023-10-30T12:51:24.936 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.938+0000 7f2a6fc17640 1 -- 172.21.15.89:0/3318693188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a68153bf0 msgr2=0x7f2a68071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:24.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.938+0000 7f2a6fc17640 1 --2- 172.21.15.89:0/3318693188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a68153bf0 0x7f2a68071260 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f2a6806ce90 tx=0x7f2a6401ecb0 comp rx=0 tx=0).stop 2023-10-30T12:51:24.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.938+0000 7f2a6fc17640 1 -- 172.21.15.89:0/3318693188 shutdown_connections 2023-10-30T12:51:24.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.938+0000 7f2a6fc17640 1 --2- 172.21.15.89:0/3318693188 >> v2:172.21.15.192:3300/0 conn(0x7f2a680c5f60 0x7f2a680cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:24.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.938+0000 7f2a6fc17640 1 --2- 172.21.15.89:0/3318693188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a68153bf0 0x7f2a68071260 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:24.937 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.938+0000 7f2a6fc17640 1 -- 172.21.15.89:0/3318693188 >> 172.21.15.89:0/3318693188 conn(0x7f2a6806c6e0 msgr2=0x7f2a68069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:24.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.938+0000 7f2a6fc17640 1 -- 172.21.15.89:0/3318693188 shutdown_connections 2023-10-30T12:51:24.938 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.938+0000 7f2a6fc17640 1 -- 172.21.15.89:0/3318693188 wait complete. 2023-10-30T12:51:24.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.938+0000 7f2a6fc17640 1 Processor -- start 2023-10-30T12:51:24.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.938+0000 7f2a6fc17640 1 -- start start 2023-10-30T12:51:24.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.938+0000 7f2a6fc17640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a680c5f60 0x7f2a680c2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:24.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.938+0000 7f2a6fc17640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a680c6fe0 0x7f2a6814e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:24.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.938+0000 7f2a6fc17640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a68153bf0 0x7f2a68155b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:24.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.938+0000 7f2a6fc17640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a681efb40 con 0x7f2a680c5f60 2023-10-30T12:51:24.941 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.938+0000 7f2a6fc17640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a68157130 con 0x7f2a68153bf0 2023-10-30T12:51:24.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.938+0000 7f2a6fc17640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a68157410 con 0x7f2a680c6fe0 2023-10-30T12:51:24.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.942+0000 7f2a6d9b3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a680c5f60 0x7f2a680c2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:24.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.942+0000 7f2a6d9b3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a680c5f60 0x7f2a680c2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40418/0 (socket says 172.21.15.89:40418) 2023-10-30T12:51:24.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.942+0000 7f2a6d9b3640 1 -- 172.21.15.89:0/1565311852 learned_addr learned my addr 172.21.15.89:0/1565311852 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:24.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.942+0000 7f2a6d1b2640 1 --2- 172.21.15.89:0/1565311852 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a680c6fe0 0x7f2a6814e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:24.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.942+0000 7f2a6e1b4640 1 --2- 172.21.15.89:0/1565311852 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a68153bf0 0x7f2a68155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:24.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.942+0000 7f2a6d9b3640 1 -- 172.21.15.89:0/1565311852 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a680c6fe0 msgr2=0x7f2a6814e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:24.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.942+0000 7f2a6d9b3640 1 --2- 172.21.15.89:0/1565311852 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a680c6fe0 0x7f2a6814e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:24.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.942+0000 7f2a6d9b3640 1 -- 172.21.15.89:0/1565311852 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a68153bf0 msgr2=0x7f2a68155b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:24.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.942+0000 7f2a6d9b3640 1 --2- 172.21.15.89:0/1565311852 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a68153bf0 0x7f2a68155b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:24.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.942+0000 7f2a6d9b3640 1 -- 172.21.15.89:0/1565311852 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a640094d0 con 0x7f2a680c5f60 2023-10-30T12:51:24.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.942+0000 7f2a6d9b3640 1 --2- 172.21.15.89:0/1565311852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a680c5f60 0x7f2a680c2170 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f2a6401b8c0 tx=0x7f2a6401ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:24.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.942+0000 7f2a5e7fc640 1 -- 172.21.15.89:0/1565311852 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a640b1d90 con 0x7f2a680c5f60 2023-10-30T12:51:24.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.942+0000 7f2a5e7fc640 1 -- 172.21.15.89:0/1565311852 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a640bb470 con 0x7f2a680c5f60 2023-10-30T12:51:24.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.942+0000 7f2a5e7fc640 1 -- 172.21.15.89:0/1565311852 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a640bb800 con 0x7f2a680c5f60 2023-10-30T12:51:24.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.942+0000 7f2a6fc17640 1 -- 172.21.15.89:0/1565311852 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a68155340 con 0x7f2a680c5f60 2023-10-30T12:51:24.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.942+0000 7f2a6fc17640 1 -- 172.21.15.89:0/1565311852 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a68159080 con 0x7f2a680c5f60 2023-10-30T12:51:24.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.942+0000 7f2a6fc17640 1 -- 172.21.15.89:0/1565311852 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a6804e170 con 0x7f2a680c5f60 2023-10-30T12:51:24.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.942+0000 7f2a5e7fc640 1 -- 172.21.15.89:0/1565311852 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2a640b7070 con 0x7f2a680c5f60 2023-10-30T12:51:24.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.942+0000 7f2a5e7fc640 1 --2- 172.21.15.89:0/1565311852 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2a4c0664d0 0x7f2a4c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:24.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.942+0000 7f2a5e7fc640 1 -- 172.21.15.89:0/1565311852 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f2a641222c0 con 0x7f2a680c5f60 2023-10-30T12:51:24.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.946+0000 7f2a6d1b2640 1 --2- 172.21.15.89:0/1565311852 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2a4c0664d0 0x7f2a4c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:24.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.946+0000 7f2a6d1b2640 1 --2- 172.21.15.89:0/1565311852 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2a4c0664d0 0x7f2a4c068990 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f2a6806cd40 tx=0x7f2a58009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:24.947 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:24.946+0000 7f2a5e7fc640 1 -- 172.21.15.89:0/1565311852 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2a640ef3a0 con 0x7f2a680c5f60 2023-10-30T12:51:25.138 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:25.138+0000 7f2a6fc17640 1 -- 172.21.15.89:0/1565311852 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a6804e170 con 0x7f2a680c5f60 2023-10-30T12:51:25.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:25.138+0000 7f2a5e7fc640 1 -- 172.21.15.89:0/1565311852 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+6541 (secure 0 0 0) 0x7f2a640f29c0 con 0x7f2a680c5f60 2023-10-30T12:51:25.139 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:51:25.139 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":118,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5906,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/4085020291","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":4085020291},{"type":"v1","addr":"172.21.15.192:6837","nonce":4085020291}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:51:23.555741+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6271,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6271":{"gid":6271,"name":"b","rank":0,"incarnation":110,"state":"up:active","state_seq":29,"addr":"172.21.15.192:6835/659375964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":659375964},{"type":"v1","addr":"172.21.15.192:6835","nonce":659375964}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:25.140 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 118 2023-10-30T12:51:25.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:25.142+0000 7f2a6fc17640 1 -- 172.21.15.89:0/1565311852 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2a4c0664d0 msgr2=0x7f2a4c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:25.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:25.142+0000 7f2a6fc17640 1 --2- 172.21.15.89:0/1565311852 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2a4c0664d0 0x7f2a4c068990 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f2a6806cd40 tx=0x7f2a58009040 comp rx=0 tx=0).stop 2023-10-30T12:51:25.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:25.142+0000 7f2a6fc17640 1 -- 172.21.15.89:0/1565311852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a680c5f60 msgr2=0x7f2a680c2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:25.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:25.142+0000 7f2a6fc17640 1 --2- 172.21.15.89:0/1565311852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a680c5f60 0x7f2a680c2170 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f2a6401b8c0 tx=0x7f2a6401ba50 comp rx=0 tx=0).stop 2023-10-30T12:51:25.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:25.142+0000 7f2a6fc17640 1 -- 172.21.15.89:0/1565311852 shutdown_connections 2023-10-30T12:51:25.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:25.142+0000 7f2a6fc17640 1 --2- 172.21.15.89:0/1565311852 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2a4c0664d0 0x7f2a4c068990 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:25.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:25.142+0000 7f2a6fc17640 1 --2- 172.21.15.89:0/1565311852 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a68153bf0 0x7f2a68155b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:25.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:25.142+0000 7f2a6fc17640 1 --2- 172.21.15.89:0/1565311852 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2a680c6fe0 0x7f2a6814e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:25.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:25.142+0000 7f2a6fc17640 1 --2- 172.21.15.89:0/1565311852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a680c5f60 0x7f2a680c2170 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:25.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:25.142+0000 7f2a6fc17640 1 -- 172.21.15.89:0/1565311852 >> 172.21.15.89:0/1565311852 conn(0x7f2a6806c6e0 msgr2=0x7f2a680c5160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:25.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:25.142+0000 7f2a6fc17640 1 -- 172.21.15.89:0/1565311852 shutdown_connections 2023-10-30T12:51:25.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:25.142+0000 7f2a6fc17640 1 -- 172.21.15.89:0/1565311852 wait complete. 2023-10-30T12:51:25.159 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-10-30T12:51:25.160 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 117 secs before thrashing 2023-10-30T12:51:26.574 INFO:tasks.workunit.client.0.smithi089.stderr:Updating files: 96% (12470/12861) Updating files: 97% (12476/12861) Updating files: 97% (12502/12861) Updating files: 97% (12599/12861) Updating files: 98% (12604/12861) Updating files: 98% (12680/12861) Updating files: 98% (12689/12861) Updating files: 98% (12717/12861) Updating files: 98% (12727/12861) Updating files: 99% (12733/12861) Updating files: 99% (12825/12861) Updating files: 100% (12861/12861) Updating files: 100% (12861/12861), done. 2023-10-30T12:51:26.575 INFO:tasks.workunit.client.0.smithi089.stderr:fatal: Could not reset index file to revision 'v0.1'. 2023-10-30T12:51:27.305 DEBUG:teuthology.orchestra.run:got remote process result: 128 2023-10-30T12:51:27.305 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0... 2023-10-30T12:51:27.306 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-30T12:51:28.134 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi089 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-10-30T12:51:28.204 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=7a6e87c27f194713999192fee0906059 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi089 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-10-30T12:51:28.207 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2023-10-30T12:51:28.217 INFO:tasks.mds_thrash:joining mds_thrasher 2023-10-30T12:51:28.220 INFO:tasks.mds_thrash.fs.[cephfs]:stat['max_mds'] = 0 2023-10-30T12:51:28.220 INFO:tasks.mds_thrash.fs.[cephfs]:stat['deactivate'] = 0 2023-10-30T12:51:28.220 INFO:tasks.mds_thrash.fs.[cephfs]:stat['kill'] = 11 2023-10-30T12:51:28.221 INFO:tasks.mds_thrash:done joining 2023-10-30T12:51:28.221 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-10-30T12:51:28.230 INFO:tasks.kclient:Unmounting kernel clients... 2023-10-30T12:51:28.230 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-10-30T12:51:28.231 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-30T12:51:28.252 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-10-30T12:51:28.252 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-30T12:51:28.299 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:51:28.479 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.478+0000 7ff903980640 1 Processor -- start 2023-10-30T12:51:28.480 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.482+0000 7ff903980640 1 -- start start 2023-10-30T12:51:28.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.482+0000 7ff903980640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8fc1538a0 0x7ff8fc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:28.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.482+0000 7ff903980640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff8fc0ca0e0 0x7ff8fc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:28.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.482+0000 7ff903980640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8fc0cb800 con 0x7ff8fc0cf160 2023-10-30T12:51:28.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.482+0000 7ff903980640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8fc15dc30 con 0x7ff8fc0ca0e0 2023-10-30T12:51:28.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.482+0000 7ff903980640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8fc0cb330 con 0x7ff8fc1538a0 2023-10-30T12:51:28.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.482+0000 7ff90171c640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8fc1538a0 0x7ff8fc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:28.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.482+0000 7ff900f1b640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff8fc0ca0e0 0x7ff8fc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:28.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.482+0000 7ff90171c640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8fc1538a0 0x7ff8fc15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:60476/0 (socket says 172.21.15.89:60476) 2023-10-30T12:51:28.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.482+0000 7ff90171c640 1 -- 172.21.15.89:0/47443569 learned_addr learned my addr 172.21.15.89:0/47443569 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:28.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.482+0000 7ff90171c640 1 -- 172.21.15.89:0/47443569 >> v2:172.21.15.192:3300/0 conn(0x7ff8fc0ca0e0 msgr2=0x7ff8fc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:28.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.482+0000 7ff90171c640 1 --2- 172.21.15.89:0/47443569 >> v2:172.21.15.192:3300/0 conn(0x7ff8fc0ca0e0 0x7ff8fc15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:28.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.482+0000 7ff90171c640 1 -- 172.21.15.89:0/47443569 >> v1:172.21.15.192:6789/0 conn(0x7ff8fc0cf160 legacy=0x7ff8fc0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:28.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.482+0000 7ff90171c640 1 -- 172.21.15.89:0/47443569 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8fc0cad70 con 0x7ff8fc1538a0 2023-10-30T12:51:28.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.486+0000 7ff90171c640 1 --2- 172.21.15.89:0/47443569 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8fc1538a0 0x7ff8fc15d6f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff8f8009870 tx=0x7ff8f801ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66cd810de072e818 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:28.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.486+0000 7ff8f3fff640 1 -- 172.21.15.89:0/47443569 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f80a8a30 con 0x7ff8fc1538a0 2023-10-30T12:51:28.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.486+0000 7ff8f3fff640 1 -- 172.21.15.89:0/47443569 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8f80a8b90 con 0x7ff8fc1538a0 2023-10-30T12:51:28.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.486+0000 7ff8f3fff640 1 -- 172.21.15.89:0/47443569 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f80b0440 con 0x7ff8fc1538a0 2023-10-30T12:51:28.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.486+0000 7ff903980640 1 -- 172.21.15.89:0/47443569 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8fc1538a0 msgr2=0x7ff8fc15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:28.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.486+0000 7ff903980640 1 --2- 172.21.15.89:0/47443569 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8fc1538a0 0x7ff8fc15d6f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff8f8009870 tx=0x7ff8f801ed30 comp rx=0 tx=0).stop 2023-10-30T12:51:28.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.486+0000 7ff903980640 1 -- 172.21.15.89:0/47443569 shutdown_connections 2023-10-30T12:51:28.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.486+0000 7ff903980640 1 --2- 172.21.15.89:0/47443569 >> v2:172.21.15.192:3300/0 conn(0x7ff8fc0ca0e0 0x7ff8fc15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:28.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.486+0000 7ff903980640 1 --2- 172.21.15.89:0/47443569 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8fc1538a0 0x7ff8fc15d6f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:28.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.486+0000 7ff903980640 1 -- 172.21.15.89:0/47443569 >> 172.21.15.89:0/47443569 conn(0x7ff8fc06bfe0 msgr2=0x7ff8fc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:28.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.486+0000 7ff903980640 1 -- 172.21.15.89:0/47443569 shutdown_connections 2023-10-30T12:51:28.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.486+0000 7ff903980640 1 -- 172.21.15.89:0/47443569 wait complete. 2023-10-30T12:51:28.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.490+0000 7ff903980640 1 Processor -- start 2023-10-30T12:51:28.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.490+0000 7ff903980640 1 -- start start 2023-10-30T12:51:28.492 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.490+0000 7ff903980640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8fc0ca0e0 0x7ff8fc0b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:28.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.490+0000 7ff903980640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8fc0cf160 0x7ff8fc15b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:28.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.490+0000 7ff903980640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8fc1538a0 0x7ff8fc1475a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:28.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.490+0000 7ff903980640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8fc158920 con 0x7ff8fc0ca0e0 2023-10-30T12:51:28.493 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.490+0000 7ff903980640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8fc157d90 con 0x7ff8fc0cf160 2023-10-30T12:51:28.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.490+0000 7ff903980640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8fc1594e0 con 0x7ff8fc1538a0 2023-10-30T12:51:28.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.490+0000 7ff90171c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8fc0ca0e0 0x7ff8fc0b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:28.494 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.490+0000 7ff90171c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8fc0ca0e0 0x7ff8fc0b4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40432/0 (socket says 172.21.15.89:40432) 2023-10-30T12:51:28.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.490+0000 7ff90171c640 1 -- 172.21.15.89:0/2302207582 learned_addr learned my addr 172.21.15.89:0/2302207582 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:28.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.490+0000 7ff900f1b640 1 --2- 172.21.15.89:0/2302207582 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8fc0cf160 0x7ff8fc15b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:28.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.490+0000 7ff901f1d640 1 --2- 172.21.15.89:0/2302207582 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8fc1538a0 0x7ff8fc1475a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:28.495 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.490+0000 7ff90171c640 1 -- 172.21.15.89:0/2302207582 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8fc1538a0 msgr2=0x7ff8fc1475a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:28.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.490+0000 7ff90171c640 1 --2- 172.21.15.89:0/2302207582 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8fc1538a0 0x7ff8fc1475a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:28.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.490+0000 7ff90171c640 1 -- 172.21.15.89:0/2302207582 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8fc0cf160 msgr2=0x7ff8fc15b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:28.496 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.490+0000 7ff90171c640 1 --2- 172.21.15.89:0/2302207582 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8fc0cf160 0x7ff8fc15b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:28.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.490+0000 7ff90171c640 1 -- 172.21.15.89:0/2302207582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f80094d0 con 0x7ff8fc0ca0e0 2023-10-30T12:51:28.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.490+0000 7ff90171c640 1 --2- 172.21.15.89:0/2302207582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8fc0ca0e0 0x7ff8fc0b4450 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7ff8f801d3e0 tx=0x7ff8f801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:28.497 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.490+0000 7ff8f1ffb640 1 -- 172.21.15.89:0/2302207582 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f80bf070 con 0x7ff8fc0ca0e0 2023-10-30T12:51:28.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.490+0000 7ff8f1ffb640 1 -- 172.21.15.89:0/2302207582 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8f80b0b70 con 0x7ff8fc0ca0e0 2023-10-30T12:51:28.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.490+0000 7ff903980640 1 -- 172.21.15.89:0/2302207582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8fc157ff0 con 0x7ff8fc0ca0e0 2023-10-30T12:51:28.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.490+0000 7ff8f1ffb640 1 -- 172.21.15.89:0/2302207582 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f80ba440 con 0x7ff8fc0ca0e0 2023-10-30T12:51:28.498 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.494+0000 7ff903980640 1 -- 172.21.15.89:0/2302207582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8fc1585b0 con 0x7ff8fc0ca0e0 2023-10-30T12:51:28.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.494+0000 7ff903980640 1 -- 172.21.15.89:0/2302207582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8fc0b2a50 con 0x7ff8fc0ca0e0 2023-10-30T12:51:28.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.494+0000 7ff8f1ffb640 1 -- 172.21.15.89:0/2302207582 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff8f80ba5a0 con 0x7ff8fc0ca0e0 2023-10-30T12:51:28.500 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.494+0000 7ff8f1ffb640 1 --2- 172.21.15.89:0/2302207582 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff8e00664d0 0x7ff8e0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:28.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.494+0000 7ff8f1ffb640 1 -- 172.21.15.89:0/2302207582 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7ff8f8121a70 con 0x7ff8fc0ca0e0 2023-10-30T12:51:28.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.494+0000 7ff900f1b640 1 --2- 172.21.15.89:0/2302207582 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff8e00664d0 0x7ff8e0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:28.501 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.494+0000 7ff900f1b640 1 --2- 172.21.15.89:0/2302207582 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff8e00664d0 0x7ff8e0068990 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc0b1da0 tx=0x7ff8ec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:28.502 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.498+0000 7ff8f1ffb640 1 -- 172.21.15.89:0/2302207582 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff8f80eea40 con 0x7ff8fc0ca0e0 2023-10-30T12:51:28.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.654+0000 7ff903980640 1 -- 172.21.15.89:0/2302207582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8fc04e170 con 0x7ff8fc0ca0e0 2023-10-30T12:51:28.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.658+0000 7ff8f1ffb640 1 -- 172.21.15.89:0/2302207582 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+18124 (secure 0 0 0) 0x7ff8f80f2060 con 0x7ff8fc0ca0e0 2023-10-30T12:51:28.656 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:51:28.656 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":66,"fsid":"401641be-db6e-4c16-b3ac-fd84a86c4de4","created":"2023-10-30T12:28:26.198532+0000","modified":"2023-10-30T12:50:41.318657+0000","last_up_change":"2023-10-30T12:28:37.212205+0000","last_in_change":"2023-10-30T12:28:28.517734+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:39.004943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T12:28:48.935909+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T12:28:49.716098+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":58,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72595c54-fd6e-4a5a-bb14-ff27e9377b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6801","nonce":1673955622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6803","nonce":1673955622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6807","nonce":1673955622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6805","nonce":1673955622}]},"public_addr":"172.21.15.89:6801/1673955622","cluster_addr":"172.21.15.89:6803/1673955622","heartbeat_back_addr":"172.21.15.89:6807/1673955622","heartbeat_front_addr":"172.21.15.89:6805/1673955622","state":["exists","up"]},{"osd":1,"uuid":"0305670b-d7c4-4193-bec2-8b7351ff590a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6817","nonce":3340572039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6819","nonce":3340572039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6823","nonce":3340572039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6821","nonce":3340572039}]},"public_addr":"172.21.15.89:6817/3340572039","cluster_addr":"172.21.15.89:6819/3340572039","heartbeat_back_addr":"172.21.15.89:6823/3340572039","heartbeat_front_addr":"172.21.15.89:6821/3340572039","state":["exists","up"]},{"osd":2,"uuid":"4bab5426-a3c3-46b0-91fc-546c254006c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6809","nonce":1929940799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6811","nonce":1929940799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6815","nonce":1929940799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6813","nonce":1929940799}]},"public_addr":"172.21.15.89:6809/1929940799","cluster_addr":"172.21.15.89:6811/1929940799","heartbeat_back_addr":"172.21.15.89:6815/1929940799","heartbeat_front_addr":"172.21.15.89:6813/1929940799","state":["exists","up"]},{"osd":3,"uuid":"7d47aea8-27cd-42be-8591-d63ad68c90e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6825","nonce":345087939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6827","nonce":345087939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6831","nonce":345087939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6829","nonce":345087939}]},"public_addr":"172.21.15.89:6825/345087939","cluster_addr":"172.21.15.89:6827/345087939","heartbeat_back_addr":"172.21.15.89:6831/345087939","heartbeat_front_addr":"172.21.15.89:6829/345087939","state":["exists","up"]},{"osd":4,"uuid":"2e2ae8f8-d097-4bba-9448-95047fbfacad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6801","nonce":2138125545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6803","nonce":2138125545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6809","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6811","nonce":2138125545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6805","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6807","nonce":2138125545}]},"public_addr":"172.21.15.192:6801/2138125545","cluster_addr":"172.21.15.192:6803/2138125545","heartbeat_back_addr":"172.21.15.192:6811/2138125545","heartbeat_front_addr":"172.21.15.192:6807/2138125545","state":["exists","up"]},{"osd":5,"uuid":"49be5ce5-12f4-4c26-8d39-864b6ff67313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6806","nonce":2360218671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6810","nonce":2360218671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6815","nonce":2360218671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6813","nonce":2360218671}]},"public_addr":"172.21.15.192:6806/2360218671","cluster_addr":"172.21.15.192:6810/2360218671","heartbeat_back_addr":"172.21.15.192:6815/2360218671","heartbeat_front_addr":"172.21.15.192:6813/2360218671","state":["exists","up"]},{"osd":6,"uuid":"48623452-d456-4ea2-a0de-5fa4b37e6d10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6825","nonce":3707671287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6827","nonce":3707671287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6831","nonce":3707671287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6829","nonce":3707671287}]},"public_addr":"172.21.15.192:6825/3707671287","cluster_addr":"172.21.15.192:6827/3707671287","heartbeat_back_addr":"172.21.15.192:6831/3707671287","heartbeat_front_addr":"172.21.15.192:6829/3707671287","state":["exists","up"]},{"osd":7,"uuid":"ff0a4384-54f8-43ec-a3ce-9c8ae3bf8a94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6817","nonce":3111945871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6819","nonce":3111945871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6823","nonce":3111945871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6821","nonce":3111945871}]},"public_addr":"172.21.15.192:6817/3111945871","cluster_addr":"172.21.15.192:6819/3111945871","heartbeat_back_addr":"172.21.15.192:6823/3111945871","heartbeat_front_addr":"172.21.15.192:6821/3111945871","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.183065+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.248650+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.194576+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.397483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.043648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.186624+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.159674+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.100851+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2},{"from":5,"to":1}]},{"pgid":"3.13","mappings":[{"from":7,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.192:6837/2130807115":"2023-10-31T12:50:41.318586+0000","172.21.15.192:6834/501291700":"2023-10-31T12:49:01.291877+0000","172.21.15.192:6837/604357386":"2023-10-31T12:48:11.263450+0000","172.21.15.192:6836/604357386":"2023-10-31T12:48:11.263450+0000","172.21.15.89:6836/951276252":"2023-10-31T12:45:51.226535+0000","172.21.15.192:6834/1289220491":"2023-10-31T12:31:20.953485+0000","172.21.15.192:6835/1289220491":"2023-10-31T12:31:20.953485+0000","172.21.15.192:6837/1413360446":"2023-10-31T12:43:36.177020+0000","172.21.15.192:6839/3339589587":"2023-10-31T12:34:56.026152+0000","172.21.15.192:6838/93914587":"2023-10-31T12:42:26.158883+0000","172.21.15.89:6837/2053475104":"2023-10-31T12:33:10.981311+0000","172.21.15.192:6836/2130807115":"2023-10-31T12:50:41.318586+0000","172.21.15.192:6836/1413360446":"2023-10-31T12:43:36.177020+0000","172.21.15.192:6838/3339589587":"2023-10-31T12:34:56.026152+0000","172.21.15.89:6837/951276252":"2023-10-31T12:45:51.226535+0000","172.21.15.192:6836/1002124471":"2023-10-31T12:37:36.081217+0000","172.21.15.192:6837/1002124471":"2023-10-31T12:37:36.081217+0000","172.21.15.192:6835/501291700":"2023-10-31T12:49:01.291877+0000","172.21.15.192:6834/654256299":"2023-10-31T12:40:11.121122+0000","172.21.15.89:6836/2053475104":"2023-10-31T12:33:10.981311+0000","172.21.15.192:6835/654256299":"2023-10-31T12:40:11.121122+0000","172.21.15.192:6839/93914587":"2023-10-31T12:42:26.158883+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:51:28.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.658+0000 7ff903980640 1 -- 172.21.15.89:0/2302207582 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff8e00664d0 msgr2=0x7ff8e0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:28.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.658+0000 7ff903980640 1 --2- 172.21.15.89:0/2302207582 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff8e00664d0 0x7ff8e0068990 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc0b1da0 tx=0x7ff8ec008040 comp rx=0 tx=0).stop 2023-10-30T12:51:28.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.658+0000 7ff903980640 1 -- 172.21.15.89:0/2302207582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8fc0ca0e0 msgr2=0x7ff8fc0b4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:28.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.658+0000 7ff903980640 1 --2- 172.21.15.89:0/2302207582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8fc0ca0e0 0x7ff8fc0b4450 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7ff8f801d3e0 tx=0x7ff8f801bad0 comp rx=0 tx=0).stop 2023-10-30T12:51:28.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.658+0000 7ff903980640 1 -- 172.21.15.89:0/2302207582 shutdown_connections 2023-10-30T12:51:28.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.658+0000 7ff903980640 1 --2- 172.21.15.89:0/2302207582 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7ff8e00664d0 0x7ff8e0068990 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:28.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.658+0000 7ff903980640 1 --2- 172.21.15.89:0/2302207582 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8fc0cf160 0x7ff8fc15b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:28.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.658+0000 7ff903980640 1 --2- 172.21.15.89:0/2302207582 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8fc1538a0 0x7ff8fc1475a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:28.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.658+0000 7ff903980640 1 --2- 172.21.15.89:0/2302207582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8fc0ca0e0 0x7ff8fc0b4450 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:28.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.658+0000 7ff903980640 1 -- 172.21.15.89:0/2302207582 >> 172.21.15.89:0/2302207582 conn(0x7ff8fc06bfe0 msgr2=0x7ff8fc14fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:28.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.658+0000 7ff903980640 1 -- 172.21.15.89:0/2302207582 shutdown_connections 2023-10-30T12:51:28.661 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.658+0000 7ff903980640 1 -- 172.21.15.89:0/2302207582 wait complete. 2023-10-30T12:51:28.677 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T12:28:39.004943+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T12:28:48.935909+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T12:28:49.716098+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 505, 'snap_epoch': 58, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T12:51:28.678 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T12:51:28.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.818+0000 7f7fe093f640 1 Processor -- start 2023-10-30T12:51:28.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.818+0000 7f7fe093f640 1 -- start start 2023-10-30T12:51:28.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.822+0000 7f7fe093f640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7fdc153c20 0x7f7fdc15a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:28.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.822+0000 7f7fe093f640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7fdc0cb4c0 0x7f7fdc0cef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:28.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.822+0000 7f7fe093f640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fdc15a6b0 con 0x7f7fdc153840 2023-10-30T12:51:28.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.822+0000 7f7fe093f640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fdc15dbc0 con 0x7f7fdc0cb4c0 2023-10-30T12:51:28.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.822+0000 7f7fe093f640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fdc15d730 con 0x7f7fdc153c20 2023-10-30T12:51:28.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.822+0000 7f7fdbfff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7fdc0cb4c0 0x7f7fdc0cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:28.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.822+0000 7f7fdbfff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f7fdc0cb4c0 0x7f7fdc0cef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40448/0 (socket says 172.21.15.89:40448) 2023-10-30T12:51:28.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.822+0000 7f7fdbfff640 1 -- 172.21.15.89:0/2442018048 learned_addr learned my addr 172.21.15.89:0/2442018048 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:28.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.822+0000 7f7fdb7fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f7fdc153840 0x7f7fdc0c3eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39326/0 (socket says 172.21.15.89:39326) 2023-10-30T12:51:28.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.822+0000 7f7fd3fff640 1 --2- 172.21.15.89:0/2442018048 >> v2:172.21.15.192:3300/0 conn(0x7f7fdc153c20 0x7f7fdc15a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:28.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.822+0000 7f7fdbfff640 1 -- 172.21.15.89:0/2442018048 >> v2:172.21.15.192:3300/0 conn(0x7f7fdc153c20 msgr2=0x7f7fdc15a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:28.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.822+0000 7f7fdbfff640 1 --2- 172.21.15.89:0/2442018048 >> v2:172.21.15.192:3300/0 conn(0x7f7fdc153c20 0x7f7fdc15a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:28.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.822+0000 7f7fdbfff640 1 -- 172.21.15.89:0/2442018048 >> v1:172.21.15.89:6789/0 conn(0x7f7fdc153840 legacy=0x7f7fdc0c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:28.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.822+0000 7f7fdbfff640 1 -- 172.21.15.89:0/2442018048 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fdc0ca8f0 con 0x7f7fdc0cb4c0 2023-10-30T12:51:28.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.822+0000 7f7fd3fff640 1 --2- 172.21.15.89:0/2442018048 >> v2:172.21.15.192:3300/0 conn(0x7f7fdc153c20 0x7f7fdc15a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:51:28.826 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.826+0000 7f7fdbfff640 1 --2- 172.21.15.89:0/2442018048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fdc0cb4c0 0x7f7fdc0cef40 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc08cfe0 tx=0x7f7fd401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11327f3ecfb65881 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:28.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.826+0000 7f7fdaffd640 1 -- 172.21.15.89:0/2442018048 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd40af400 con 0x7f7fdc0cb4c0 2023-10-30T12:51:28.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.826+0000 7f7fdaffd640 1 -- 172.21.15.89:0/2442018048 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd40ab050 con 0x7f7fdc0cb4c0 2023-10-30T12:51:28.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.826+0000 7f7fdaffd640 1 -- 172.21.15.89:0/2442018048 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd40af780 con 0x7f7fdc0cb4c0 2023-10-30T12:51:28.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.826+0000 7f7fe093f640 1 -- 172.21.15.89:0/2442018048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fdc0cb4c0 msgr2=0x7f7fdc0cef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:28.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.826+0000 7f7fe093f640 1 --2- 172.21.15.89:0/2442018048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fdc0cb4c0 0x7f7fdc0cef40 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc08cfe0 tx=0x7f7fd401f1c0 comp rx=0 tx=0).stop 2023-10-30T12:51:28.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.826+0000 7f7fe093f640 1 -- 172.21.15.89:0/2442018048 shutdown_connections 2023-10-30T12:51:28.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.826+0000 7f7fe093f640 1 --2- 172.21.15.89:0/2442018048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fdc0cb4c0 0x7f7fdc0cef40 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:28.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.826+0000 7f7fe093f640 1 --2- 172.21.15.89:0/2442018048 >> v2:172.21.15.192:3300/0 conn(0x7f7fdc153c20 0x7f7fdc15a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:28.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.826+0000 7f7fe093f640 1 -- 172.21.15.89:0/2442018048 >> 172.21.15.89:0/2442018048 conn(0x7f7fdc06be40 msgr2=0x7f7fdc06c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:28.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.826+0000 7f7fe093f640 1 -- 172.21.15.89:0/2442018048 shutdown_connections 2023-10-30T12:51:28.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.826+0000 7f7fe093f640 1 -- 172.21.15.89:0/2442018048 wait complete. 2023-10-30T12:51:28.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.826+0000 7f7fe093f640 1 Processor -- start 2023-10-30T12:51:28.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.826+0000 7f7fe093f640 1 -- start start 2023-10-30T12:51:28.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.826+0000 7f7fe093f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7fdc0cb4c0 0x7f7fdc1547c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:28.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.826+0000 7f7fe093f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7fdc153840 0x7f7fdc158930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:28.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.830+0000 7f7fe093f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fdc153c20 0x7f7fdc158e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:28.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.830+0000 7f7fe093f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fdc154de0 con 0x7f7fdc153c20 2023-10-30T12:51:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.830+0000 7f7fe093f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fdc1584a0 con 0x7f7fdc0cb4c0 2023-10-30T12:51:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.830+0000 7f7fe093f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fdc156450 con 0x7f7fdc153840 2023-10-30T12:51:28.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.830+0000 7f7fdbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fdc153c20 0x7f7fdc158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:28.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.830+0000 7f7fdbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fdc153c20 0x7f7fdc158e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40454/0 (socket says 172.21.15.89:40454) 2023-10-30T12:51:28.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.830+0000 7f7fdbfff640 1 -- 172.21.15.89:0/1451460278 learned_addr learned my addr 172.21.15.89:0/1451460278 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:28.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.830+0000 7f7fdb7fe640 1 --2- 172.21.15.89:0/1451460278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7fdc0cb4c0 0x7f7fdc1547c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:28.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.830+0000 7f7fd3fff640 1 --2- 172.21.15.89:0/1451460278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7fdc153840 0x7f7fdc158930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:28.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.830+0000 7f7fdbfff640 1 -- 172.21.15.89:0/1451460278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7fdc153840 msgr2=0x7f7fdc158930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:28.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.830+0000 7f7fdbfff640 1 --2- 172.21.15.89:0/1451460278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7fdc153840 0x7f7fdc158930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:28.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.830+0000 7f7fdbfff640 1 -- 172.21.15.89:0/1451460278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7fdc0cb4c0 msgr2=0x7f7fdc1547c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:28.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.830+0000 7f7fdbfff640 1 --2- 172.21.15.89:0/1451460278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7fdc0cb4c0 0x7f7fdc1547c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:28.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.830+0000 7f7fdbfff640 1 -- 172.21.15.89:0/1451460278 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd4009b30 con 0x7f7fdc153c20 2023-10-30T12:51:28.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.830+0000 7f7fdbfff640 1 --2- 172.21.15.89:0/1451460278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fdc153c20 0x7f7fdc158e70 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f7fd40b2040 tx=0x7f7fd401bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:28.837 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.830+0000 7f7fd8ff9640 1 -- 172.21.15.89:0/1451460278 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd40afde0 con 0x7f7fdc153c20 2023-10-30T12:51:28.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.830+0000 7f7fd8ff9640 1 -- 172.21.15.89:0/1451460278 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd4009510 con 0x7f7fdc153c20 2023-10-30T12:51:28.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.830+0000 7f7fe093f640 1 -- 172.21.15.89:0/1451460278 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fdc1566f0 con 0x7f7fdc153c20 2023-10-30T12:51:28.838 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.830+0000 7f7fd8ff9640 1 -- 172.21.15.89:0/1451460278 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd40ba660 con 0x7f7fdc153c20 2023-10-30T12:51:28.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.830+0000 7f7fe093f640 1 -- 172.21.15.89:0/1451460278 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fdc158130 con 0x7f7fdc153c20 2023-10-30T12:51:28.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.830+0000 7f7fe093f640 1 -- 172.21.15.89:0/1451460278 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fdc157f70 con 0x7f7fdc153c20 2023-10-30T12:51:28.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.830+0000 7f7fd8ff9640 1 -- 172.21.15.89:0/1451460278 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7fd40b6040 con 0x7f7fdc153c20 2023-10-30T12:51:28.839 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.834+0000 7f7fd8ff9640 1 --2- 172.21.15.89:0/1451460278 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f7fbc0664d0 0x7f7fbc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:28.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.834+0000 7f7fd8ff9640 1 -- 172.21.15.89:0/1451460278 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f7fd4122640 con 0x7f7fdc153c20 2023-10-30T12:51:28.840 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.834+0000 7f7fdb7fe640 1 --2- 172.21.15.89:0/1451460278 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f7fbc0664d0 0x7f7fbc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:28.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.834+0000 7f7fdb7fe640 1 --2- 172.21.15.89:0/1451460278 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f7fbc0664d0 0x7f7fbc068990 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc000f70 tx=0x7f7fcc007c90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:28.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.838+0000 7f7fd8ff9640 1 -- 172.21.15.89:0/1451460278 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7fd40ef6d0 con 0x7f7fdc153c20 2023-10-30T12:51:28.984 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.986+0000 7f7fe093f640 1 -- 172.21.15.89:0/1451460278 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7fdc158610 con 0x7f7fdc153c20 2023-10-30T12:51:28.985 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.986+0000 7f7fd8ff9640 1 -- 172.21.15.89:0/1451460278 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v66) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7fd40f2cf0 con 0x7f7fdc153c20 2023-10-30T12:51:28.985 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2023-10-30T12:51:28.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.986+0000 7f7fe093f640 1 -- 172.21.15.89:0/1451460278 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f7fbc0664d0 msgr2=0x7f7fbc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:28.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.986+0000 7f7fe093f640 1 --2- 172.21.15.89:0/1451460278 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f7fbc0664d0 0x7f7fbc068990 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc000f70 tx=0x7f7fcc007c90 comp rx=0 tx=0).stop 2023-10-30T12:51:28.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.986+0000 7f7fe093f640 1 -- 172.21.15.89:0/1451460278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fdc153c20 msgr2=0x7f7fdc158e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:28.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.986+0000 7f7fe093f640 1 --2- 172.21.15.89:0/1451460278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fdc153c20 0x7f7fdc158e70 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f7fd40b2040 tx=0x7f7fd401bf60 comp rx=0 tx=0).stop 2023-10-30T12:51:28.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.986+0000 7f7fe093f640 1 -- 172.21.15.89:0/1451460278 shutdown_connections 2023-10-30T12:51:28.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.986+0000 7f7fe093f640 1 --2- 172.21.15.89:0/1451460278 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f7fbc0664d0 0x7f7fbc068990 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:28.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.986+0000 7f7fe093f640 1 --2- 172.21.15.89:0/1451460278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fdc153c20 0x7f7fdc158e70 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:28.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.986+0000 7f7fe093f640 1 --2- 172.21.15.89:0/1451460278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7fdc153840 0x7f7fdc158930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:28.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.986+0000 7f7fe093f640 1 --2- 172.21.15.89:0/1451460278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7fdc0cb4c0 0x7f7fdc1547c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:28.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.986+0000 7f7fe093f640 1 -- 172.21.15.89:0/1451460278 >> 172.21.15.89:0/1451460278 conn(0x7f7fdc06be40 msgr2=0x7f7fdc06c220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:28.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.986+0000 7f7fe093f640 1 -- 172.21.15.89:0/1451460278 shutdown_connections 2023-10-30T12:51:28.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:28.986+0000 7f7fe093f640 1 -- 172.21.15.89:0/1451460278 wait complete. 2023-10-30T12:51:29.006 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T12:51:29.145 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.146+0000 7facbd8d6640 1 Processor -- start 2023-10-30T12:51:29.146 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.146+0000 7facbd8d6640 1 -- start start 2023-10-30T12:51:29.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.146+0000 7facbd8d6640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7facb81508c0 0x7facb80c6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:29.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.146+0000 7facbd8d6640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facb80722a0 0x7facb80c6e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:29.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.146+0000 7facbd8d6640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facb80c0a30 con 0x7facb80bfec0 2023-10-30T12:51:29.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.146+0000 7facbd8d6640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facb80c85b0 con 0x7facb81508c0 2023-10-30T12:51:29.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.146+0000 7facbd8d6640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facb806f650 con 0x7facb80722a0 2023-10-30T12:51:29.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.146+0000 7facb6ffd640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7facb81508c0 0x7facb80c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:29.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.146+0000 7facb6ffd640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7facb81508c0 0x7facb80c6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40470/0 (socket says 172.21.15.89:40470) 2023-10-30T12:51:29.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.146+0000 7facb6ffd640 1 -- 172.21.15.89:0/3571836164 learned_addr learned my addr 172.21.15.89:0/3571836164 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:29.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.146+0000 7facb67fc640 1 --2- 172.21.15.89:0/3571836164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facb80722a0 0x7facb80c6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:29.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.150+0000 7facb6ffd640 1 -- 172.21.15.89:0/3571836164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facb80722a0 msgr2=0x7facb80c6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:29.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.150+0000 7facb6ffd640 1 --2- 172.21.15.89:0/3571836164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facb80722a0 0x7facb80c6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.150+0000 7facb6ffd640 1 -- 172.21.15.89:0/3571836164 >> v1:172.21.15.192:6789/0 conn(0x7facb80bfec0 legacy=0x7facb8156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:29.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.150+0000 7facb6ffd640 1 -- 172.21.15.89:0/3571836164 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb806f930 con 0x7facb81508c0 2023-10-30T12:51:29.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.150+0000 7facb67fc640 1 --2- 172.21.15.89:0/3571836164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facb80722a0 0x7facb80c6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:29.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.150+0000 7facb6ffd640 1 --2- 172.21.15.89:0/3571836164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facb81508c0 0x7facb80c6920 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7faca8009860 tx=0x7faca801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ef53eb131f9b7b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:29.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.150+0000 7facb5ffb640 1 -- 172.21.15.89:0/3571836164 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca80ad680 con 0x7facb81508c0 2023-10-30T12:51:29.157 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.150+0000 7facb5ffb640 1 -- 172.21.15.89:0/3571836164 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faca80ad7e0 con 0x7facb81508c0 2023-10-30T12:51:29.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.150+0000 7facb5ffb640 1 -- 172.21.15.89:0/3571836164 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca80adb70 con 0x7facb81508c0 2023-10-30T12:51:29.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.150+0000 7facbd8d6640 1 -- 172.21.15.89:0/3571836164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facb81508c0 msgr2=0x7facb80c6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:29.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.150+0000 7facbd8d6640 1 --2- 172.21.15.89:0/3571836164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facb81508c0 0x7facb80c6920 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7faca8009860 tx=0x7faca801c490 comp rx=0 tx=0).stop 2023-10-30T12:51:29.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.150+0000 7facbd8d6640 1 -- 172.21.15.89:0/3571836164 shutdown_connections 2023-10-30T12:51:29.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.150+0000 7facbd8d6640 1 --2- 172.21.15.89:0/3571836164 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facb80722a0 0x7facb80c6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.150+0000 7facbd8d6640 1 --2- 172.21.15.89:0/3571836164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facb81508c0 0x7facb80c6920 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.150+0000 7facbd8d6640 1 -- 172.21.15.89:0/3571836164 >> 172.21.15.89:0/3571836164 conn(0x7facb80c7910 msgr2=0x7facb80c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:29.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.154+0000 7facbd8d6640 1 -- 172.21.15.89:0/3571836164 shutdown_connections 2023-10-30T12:51:29.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.154+0000 7facbd8d6640 1 -- 172.21.15.89:0/3571836164 wait complete. 2023-10-30T12:51:29.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.154+0000 7facbd8d6640 1 Processor -- start 2023-10-30T12:51:29.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.154+0000 7facbd8d6640 1 -- start start 2023-10-30T12:51:29.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.154+0000 7facbd8d6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facb80722a0 0x7facb80c9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:29.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.154+0000 7facbd8d6640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facb80bfec0 0x7facb81f5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:29.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.154+0000 7facbd8d6640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7facb81508c0 0x7facb8145220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:29.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.154+0000 7facbd8d6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb814ec00 con 0x7facb80722a0 2023-10-30T12:51:29.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.154+0000 7facbd8d6640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb8155ef0 con 0x7facb81508c0 2023-10-30T12:51:29.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.154+0000 7facbd8d6640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facb80c9890 con 0x7facb80bfec0 2023-10-30T12:51:29.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.154+0000 7facb6ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facb80722a0 0x7facb80c9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:29.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.154+0000 7facb6ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facb80722a0 0x7facb80c9210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40474/0 (socket says 172.21.15.89:40474) 2023-10-30T12:51:29.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.154+0000 7facb6ffd640 1 -- 172.21.15.89:0/414489502 learned_addr learned my addr 172.21.15.89:0/414489502 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:29.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.154+0000 7facb67fc640 1 --2- 172.21.15.89:0/414489502 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facb80bfec0 0x7facb81f5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:29.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.154+0000 7facb77fe640 1 --2- 172.21.15.89:0/414489502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7facb81508c0 0x7facb8145220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:29.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.154+0000 7facb6ffd640 1 -- 172.21.15.89:0/414489502 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facb80bfec0 msgr2=0x7facb81f5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:29.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.154+0000 7facb6ffd640 1 --2- 172.21.15.89:0/414489502 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facb80bfec0 0x7facb81f5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.154+0000 7facb6ffd640 1 -- 172.21.15.89:0/414489502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7facb81508c0 msgr2=0x7facb8145220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:29.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.154+0000 7facb6ffd640 1 --2- 172.21.15.89:0/414489502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7facb81508c0 0x7facb8145220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.154+0000 7facb6ffd640 1 -- 172.21.15.89:0/414489502 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca8009470 con 0x7facb80722a0 2023-10-30T12:51:29.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.154+0000 7facb6ffd640 1 --2- 172.21.15.89:0/414489502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facb80722a0 0x7facb80c9210 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7faca801ab40 tx=0x7faca801acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:29.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.154+0000 7fac9ffff640 1 -- 172.21.15.89:0/414489502 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca80adea0 con 0x7facb80722a0 2023-10-30T12:51:29.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.154+0000 7fac9ffff640 1 -- 172.21.15.89:0/414489502 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faca80b8420 con 0x7facb80722a0 2023-10-30T12:51:29.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.154+0000 7fac9ffff640 1 -- 172.21.15.89:0/414489502 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca80b87d0 con 0x7facb80722a0 2023-10-30T12:51:29.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.154+0000 7facbd8d6640 1 -- 172.21.15.89:0/414489502 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facb80c9b70 con 0x7facb80722a0 2023-10-30T12:51:29.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.158+0000 7facbd8d6640 1 -- 172.21.15.89:0/414489502 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facb81f6760 con 0x7facb80722a0 2023-10-30T12:51:29.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.158+0000 7facbd8d6640 1 -- 172.21.15.89:0/414489502 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facb804e170 con 0x7facb80722a0 2023-10-30T12:51:29.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.158+0000 7fac9ffff640 1 -- 172.21.15.89:0/414489502 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faca80b4040 con 0x7facb80722a0 2023-10-30T12:51:29.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.158+0000 7fac9ffff640 1 --2- 172.21.15.89:0/414489502 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faca40664a0 0x7faca4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:29.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.158+0000 7fac9ffff640 1 -- 172.21.15.89:0/414489502 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7faca811ea50 con 0x7facb80722a0 2023-10-30T12:51:29.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.158+0000 7facb67fc640 1 --2- 172.21.15.89:0/414489502 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faca40664a0 0x7faca4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:29.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.162+0000 7facb67fc640 1 --2- 172.21.15.89:0/414489502 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faca40664a0 0x7faca4068960 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7facb8146580 tx=0x7facac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:29.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.162+0000 7fac9ffff640 1 -- 172.21.15.89:0/414489502 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7faca80ebad0 con 0x7facb80722a0 2023-10-30T12:51:29.371 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.374+0000 7facbd8d6640 1 -- 172.21.15.89:0/414489502 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7facb814ee30 con 0x7facb80722a0 2023-10-30T12:51:29.372 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.374+0000 7fac9ffff640 1 -- 172.21.15.89:0/414489502 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v66) v1 ==== 104+0+11 (secure 0 0 0) 0x7faca80ef0f0 con 0x7facb80722a0 2023-10-30T12:51:29.373 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2023-10-30T12:51:29.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.374+0000 7facbd8d6640 1 -- 172.21.15.89:0/414489502 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faca40664a0 msgr2=0x7faca4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:29.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.374+0000 7facbd8d6640 1 --2- 172.21.15.89:0/414489502 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faca40664a0 0x7faca4068960 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7facb8146580 tx=0x7facac008040 comp rx=0 tx=0).stop 2023-10-30T12:51:29.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.374+0000 7facbd8d6640 1 -- 172.21.15.89:0/414489502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facb80722a0 msgr2=0x7facb80c9210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:29.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.374+0000 7facbd8d6640 1 --2- 172.21.15.89:0/414489502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facb80722a0 0x7facb80c9210 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7faca801ab40 tx=0x7faca801acd0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.374+0000 7facbd8d6640 1 -- 172.21.15.89:0/414489502 shutdown_connections 2023-10-30T12:51:29.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.374+0000 7facbd8d6640 1 --2- 172.21.15.89:0/414489502 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faca40664a0 0x7faca4068960 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.374+0000 7facbd8d6640 1 --2- 172.21.15.89:0/414489502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7facb81508c0 0x7facb8145220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.374+0000 7facbd8d6640 1 --2- 172.21.15.89:0/414489502 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facb80bfec0 0x7facb81f5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.374+0000 7facbd8d6640 1 --2- 172.21.15.89:0/414489502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7facb80722a0 0x7facb80c9210 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.374+0000 7facbd8d6640 1 -- 172.21.15.89:0/414489502 >> 172.21.15.89:0/414489502 conn(0x7facb80c7910 msgr2=0x7facb80c0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:29.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.374+0000 7facbd8d6640 1 -- 172.21.15.89:0/414489502 shutdown_connections 2023-10-30T12:51:29.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.374+0000 7facbd8d6640 1 -- 172.21.15.89:0/414489502 wait complete. 2023-10-30T12:51:29.402 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T12:51:29.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.558+0000 7f8f10d0f640 1 Processor -- start 2023-10-30T12:51:29.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.558+0000 7f8f10d0f640 1 -- start start 2023-10-30T12:51:29.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.558+0000 7f8f10d0f640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8f0c1508c0 0x7f8f0c0c6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:29.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.558+0000 7f8f10d0f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f0c0722a0 0x7f8f0c0c6e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:29.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.558+0000 7f8f10d0f640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f0c0c0a30 con 0x7f8f0c0bfec0 2023-10-30T12:51:29.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.558+0000 7f8f10d0f640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f0c0c85b0 con 0x7f8f0c1508c0 2023-10-30T12:51:29.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.558+0000 7f8f10d0f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c06f650 con 0x7f8f0c0722a0 2023-10-30T12:51:29.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.558+0000 7f8f0ad9d640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f8f0c0bfec0 0x7f8f0c156930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:39342/0 (socket says 172.21.15.89:39342) 2023-10-30T12:51:29.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.558+0000 7f8f0ad9d640 1 -- 172.21.15.89:0/409789952 learned_addr learned my addr 172.21.15.89:0/409789952 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:51:29.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.558+0000 7f8f0a59c640 1 --2- 172.21.15.89:0/409789952 >> v2:172.21.15.192:3300/0 conn(0x7f8f0c1508c0 0x7f8f0c0c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:29.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.558+0000 7f8f09d9b640 1 --2- 172.21.15.89:0/409789952 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f0c0722a0 0x7f8f0c0c6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:29.564 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.558+0000 7f8f0959a640 1 -- 172.21.15.89:0/409789952 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2628639566 0 0) 0x7f8f0c0c0a30 con 0x7f8f0c0bfec0 2023-10-30T12:51:29.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.562+0000 7f8f0959a640 1 -- 172.21.15.89:0/409789952 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ef4003530 con 0x7f8f0c0bfec0 2023-10-30T12:51:29.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.562+0000 7f8f0a59c640 1 -- 172.21.15.89:0/409789952 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f0c0722a0 msgr2=0x7f8f0c0c6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:29.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.562+0000 7f8f0959a640 1 -- 172.21.15.89:0/409789952 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1070137704 0 0) 0x7f8ef4003530 con 0x7f8f0c0bfec0 2023-10-30T12:51:29.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.562+0000 7f8f0a59c640 1 --2- 172.21.15.89:0/409789952 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f0c0722a0 0x7f8f0c0c6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.562+0000 7f8f0a59c640 1 -- 172.21.15.89:0/409789952 >> v1:172.21.15.89:6789/0 conn(0x7f8f0c0bfec0 legacy=0x7f8f0c156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:29.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.562+0000 7f8f0a59c640 1 -- 172.21.15.89:0/409789952 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f0c06f930 con 0x7f8f0c1508c0 2023-10-30T12:51:29.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.562+0000 7f8f0a59c640 1 --2- 172.21.15.89:0/409789952 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f0c1508c0 0x7f8f0c0c6920 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8efc0097d0 tx=0x7f8efc01c2f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bb24eb0dfd3162 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:29.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.562+0000 7f8f0959a640 1 -- 172.21.15.89:0/409789952 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f04001a00 con 0x7f8f0c1508c0 2023-10-30T12:51:29.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.562+0000 7f8f0959a640 1 -- 172.21.15.89:0/409789952 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8efc0a7050 con 0x7f8f0c1508c0 2023-10-30T12:51:29.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.562+0000 7f8f0959a640 1 -- 172.21.15.89:0/409789952 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8efc0ab600 con 0x7f8f0c1508c0 2023-10-30T12:51:29.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.566+0000 7f8f10d0f640 1 -- 172.21.15.89:0/409789952 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f0c1508c0 msgr2=0x7f8f0c0c6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:29.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.566+0000 7f8f10d0f640 1 --2- 172.21.15.89:0/409789952 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f0c1508c0 0x7f8f0c0c6920 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8efc0097d0 tx=0x7f8efc01c2f0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.568 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.566+0000 7f8f10d0f640 1 -- 172.21.15.89:0/409789952 shutdown_connections 2023-10-30T12:51:29.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.566+0000 7f8f10d0f640 1 --2- 172.21.15.89:0/409789952 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f0c0722a0 0x7f8f0c0c6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.566+0000 7f8f10d0f640 1 --2- 172.21.15.89:0/409789952 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f0c1508c0 0x7f8f0c0c6920 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.566+0000 7f8f10d0f640 1 -- 172.21.15.89:0/409789952 >> 172.21.15.89:0/409789952 conn(0x7f8f0c0c7910 msgr2=0x7f8f0c0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:29.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.566+0000 7f8f10d0f640 1 -- 172.21.15.89:0/409789952 shutdown_connections 2023-10-30T12:51:29.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.566+0000 7f8f10d0f640 1 -- 172.21.15.89:0/409789952 wait complete. 2023-10-30T12:51:29.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.566+0000 7f8f10d0f640 1 Processor -- start 2023-10-30T12:51:29.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.566+0000 7f8f10d0f640 1 -- start start 2023-10-30T12:51:29.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.566+0000 7f8f10d0f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f0c0722a0 0x7f8f0c154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:29.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.566+0000 7f8f10d0f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f0c0bfec0 0x7f8f0c154fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:29.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.566+0000 7f8f10d0f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f0c1508c0 0x7f8f0c149ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:29.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.566+0000 7f8f10d0f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c154640 con 0x7f8f0c0722a0 2023-10-30T12:51:29.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.566+0000 7f8f10d0f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c0c7640 con 0x7f8f0c0bfec0 2023-10-30T12:51:29.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.566+0000 7f8f10d0f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c1f37e0 con 0x7f8f0c1508c0 2023-10-30T12:51:29.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.566+0000 7f8f0a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f0c0722a0 0x7f8f0c154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:29.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.566+0000 7f8f0a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f0c0722a0 0x7f8f0c154a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40480/0 (socket says 172.21.15.89:40480) 2023-10-30T12:51:29.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.566+0000 7f8f0a59c640 1 -- 172.21.15.89:0/3975851855 learned_addr learned my addr 172.21.15.89:0/3975851855 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:29.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.566+0000 7f8f0ad9d640 1 --2- 172.21.15.89:0/3975851855 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f0c1508c0 0x7f8f0c149ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:29.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.566+0000 7f8f09d9b640 1 --2- 172.21.15.89:0/3975851855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f0c0bfec0 0x7f8f0c154fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:29.576 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.566+0000 7f8f0a59c640 1 -- 172.21.15.89:0/3975851855 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f0c1508c0 msgr2=0x7f8f0c149ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:29.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.566+0000 7f8f0a59c640 1 --2- 172.21.15.89:0/3975851855 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f0c1508c0 0x7f8f0c149ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.566+0000 7f8f0a59c640 1 -- 172.21.15.89:0/3975851855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f0c0bfec0 msgr2=0x7f8f0c154fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:29.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.566+0000 7f8f0a59c640 1 --2- 172.21.15.89:0/3975851855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f0c0bfec0 0x7f8f0c154fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.566+0000 7f8f0a59c640 1 -- 172.21.15.89:0/3975851855 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8efc0093b0 con 0x7f8f0c0722a0 2023-10-30T12:51:29.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.566+0000 7f8f09d9b640 1 --2- 172.21.15.89:0/3975851855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f0c0bfec0 0x7f8f0c154fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:29.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.570+0000 7f8f0a59c640 1 --2- 172.21.15.89:0/3975851855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f0c0722a0 0x7f8f0c154a60 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f8efc01a9a0 tx=0x7f8efc019090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:29.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.570+0000 7f8efaffd640 1 -- 172.21.15.89:0/3975851855 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8efc0abce0 con 0x7f8f0c0722a0 2023-10-30T12:51:29.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.570+0000 7f8efaffd640 1 -- 172.21.15.89:0/3975851855 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8efc0abe40 con 0x7f8f0c0722a0 2023-10-30T12:51:29.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.570+0000 7f8f10d0f640 1 -- 172.21.15.89:0/3975851855 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f0c1f3aa0 con 0x7f8f0c0722a0 2023-10-30T12:51:29.579 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.570+0000 7f8efaffd640 1 -- 172.21.15.89:0/3975851855 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8efc0b4640 con 0x7f8f0c0722a0 2023-10-30T12:51:29.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.570+0000 7f8f10d0f640 1 -- 172.21.15.89:0/3975851855 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f0c1f4060 con 0x7f8f0c0722a0 2023-10-30T12:51:29.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.570+0000 7f8f10d0f640 1 -- 172.21.15.89:0/3975851855 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f0c04e170 con 0x7f8f0c0722a0 2023-10-30T12:51:29.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.570+0000 7f8efaffd640 1 -- 172.21.15.89:0/3975851855 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8efc0b2040 con 0x7f8f0c0722a0 2023-10-30T12:51:29.580 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.570+0000 7f8efaffd640 1 --2- 172.21.15.89:0/3975851855 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8eec0664d0 0x7f8eec068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:29.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.570+0000 7f8efaffd640 1 -- 172.21.15.89:0/3975851855 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f8efc11dfc0 con 0x7f8f0c0722a0 2023-10-30T12:51:29.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.570+0000 7f8f09d9b640 1 --2- 172.21.15.89:0/3975851855 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8eec0664d0 0x7f8eec068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:29.581 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.574+0000 7f8f09d9b640 1 --2- 172.21.15.89:0/3975851855 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8eec0664d0 0x7f8eec068990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c14c890 tx=0x7f8f00008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:29.582 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.574+0000 7f8efaffd640 1 -- 172.21.15.89:0/3975851855 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8efc14f090 con 0x7f8f0c0722a0 2023-10-30T12:51:29.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.734+0000 7f8f10d0f640 1 -- 172.21.15.89:0/3975851855 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8f0c1548b0 con 0x7f8f0c0722a0 2023-10-30T12:51:29.736 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.738+0000 7f8efaffd640 1 -- 172.21.15.89:0/3975851855 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v66) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8efc0b90a0 con 0x7f8f0c0722a0 2023-10-30T12:51:29.736 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2023-10-30T12:51:29.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.738+0000 7f8f10d0f640 1 -- 172.21.15.89:0/3975851855 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8eec0664d0 msgr2=0x7f8eec068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:29.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.738+0000 7f8f10d0f640 1 --2- 172.21.15.89:0/3975851855 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8eec0664d0 0x7f8eec068990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c14c890 tx=0x7f8f00008040 comp rx=0 tx=0).stop 2023-10-30T12:51:29.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.738+0000 7f8f10d0f640 1 -- 172.21.15.89:0/3975851855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f0c0722a0 msgr2=0x7f8f0c154a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:29.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.738+0000 7f8f10d0f640 1 --2- 172.21.15.89:0/3975851855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f0c0722a0 0x7f8f0c154a60 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f8efc01a9a0 tx=0x7f8efc019090 comp rx=0 tx=0).stop 2023-10-30T12:51:29.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.738+0000 7f8f10d0f640 1 -- 172.21.15.89:0/3975851855 shutdown_connections 2023-10-30T12:51:29.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.738+0000 7f8f10d0f640 1 --2- 172.21.15.89:0/3975851855 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f8eec0664d0 0x7f8eec068990 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.738+0000 7f8f10d0f640 1 --2- 172.21.15.89:0/3975851855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f0c0bfec0 0x7f8f0c154fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.738+0000 7f8f10d0f640 1 --2- 172.21.15.89:0/3975851855 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8f0c1508c0 0x7f8f0c149ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.738+0000 7f8f10d0f640 1 --2- 172.21.15.89:0/3975851855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f0c0722a0 0x7f8f0c154a60 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.738+0000 7f8f10d0f640 1 -- 172.21.15.89:0/3975851855 >> 172.21.15.89:0/3975851855 conn(0x7f8f0c0c7910 msgr2=0x7f8f0c071790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:29.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.738+0000 7f8f10d0f640 1 -- 172.21.15.89:0/3975851855 shutdown_connections 2023-10-30T12:51:29.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.738+0000 7f8f10d0f640 1 -- 172.21.15.89:0/3975851855 wait complete. 2023-10-30T12:51:29.757 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-30T12:51:29.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.898+0000 7f3ea0809640 1 Processor -- start 2023-10-30T12:51:29.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.898+0000 7f3ea0809640 1 -- start start 2023-10-30T12:51:29.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.898+0000 7f3ea0809640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e9c15a000 0x7f3e9c15a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:29.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.898+0000 7f3ea0809640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3e9c0cb570 0x7f3e9c0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:29.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.898+0000 7f3ea0809640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e9c15ce10 con 0x7f3e9c153d10 2023-10-30T12:51:29.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.898+0000 7f3ea0809640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e9c159820 con 0x7f3e9c0cb570 2023-10-30T12:51:29.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.898+0000 7f3ea0809640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c0c8430 con 0x7f3e9c15a000 2023-10-30T12:51:29.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.898+0000 7f3e9ad9d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3e9c0cb570 0x7f3e9c0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:29.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.898+0000 7f3e9ad9d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f3e9c0cb570 0x7f3e9c0cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47260/0 (socket says 172.21.15.89:47260) 2023-10-30T12:51:29.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.898+0000 7f3e9ad9d640 1 -- 172.21.15.89:0/2195682269 learned_addr learned my addr 172.21.15.89:0/2195682269 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:29.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.898+0000 7f3e99d9b640 1 --2- 172.21.15.89:0/2195682269 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e9c15a000 0x7f3e9c15a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:29.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.898+0000 7f3e9ad9d640 1 -- 172.21.15.89:0/2195682269 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e9c15a000 msgr2=0x7f3e9c15a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:29.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.898+0000 7f3e9ad9d640 1 --2- 172.21.15.89:0/2195682269 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e9c15a000 0x7f3e9c15a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.898+0000 7f3e9ad9d640 1 -- 172.21.15.89:0/2195682269 >> v1:172.21.15.192:6789/0 conn(0x7f3e9c153d10 legacy=0x7f3e9c0c5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:29.902 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.898+0000 7f3e9ad9d640 1 -- 172.21.15.89:0/2195682269 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e9c0c8710 con 0x7f3e9c0cb570 2023-10-30T12:51:29.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.902+0000 7f3e9ad9d640 1 --2- 172.21.15.89:0/2195682269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e9c0cb570 0x7f3e9c0cb9e0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f3e94009e50 tx=0x7f3e9401c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df00dd35e77608d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:29.903 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.902+0000 7f3e9959a640 1 -- 172.21.15.89:0/2195682269 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e940a58a0 con 0x7f3e9c0cb570 2023-10-30T12:51:29.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.902+0000 7f3e9959a640 1 -- 172.21.15.89:0/2195682269 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e940a5a00 con 0x7f3e9c0cb570 2023-10-30T12:51:29.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.902+0000 7f3e9959a640 1 -- 172.21.15.89:0/2195682269 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e940a5d90 con 0x7f3e9c0cb570 2023-10-30T12:51:29.904 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.902+0000 7f3ea0809640 1 -- 172.21.15.89:0/2195682269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e9c0cb570 msgr2=0x7f3e9c0cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:29.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.902+0000 7f3ea0809640 1 --2- 172.21.15.89:0/2195682269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e9c0cb570 0x7f3e9c0cb9e0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f3e94009e50 tx=0x7f3e9401c9e0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.902+0000 7f3ea0809640 1 -- 172.21.15.89:0/2195682269 shutdown_connections 2023-10-30T12:51:29.905 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.902+0000 7f3ea0809640 1 --2- 172.21.15.89:0/2195682269 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e9c15a000 0x7f3e9c15a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.902+0000 7f3ea0809640 1 --2- 172.21.15.89:0/2195682269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e9c0cb570 0x7f3e9c0cb9e0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.902+0000 7f3ea0809640 1 -- 172.21.15.89:0/2195682269 >> 172.21.15.89:0/2195682269 conn(0x7f3e9c06c6b0 msgr2=0x7f3e9c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:29.906 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.902+0000 7f3ea0809640 1 -- 172.21.15.89:0/2195682269 shutdown_connections 2023-10-30T12:51:29.907 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.902+0000 7f3ea0809640 1 -- 172.21.15.89:0/2195682269 wait complete. 2023-10-30T12:51:29.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.906+0000 7f3ea0809640 1 Processor -- start 2023-10-30T12:51:29.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.906+0000 7f3ea0809640 1 -- start start 2023-10-30T12:51:29.909 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.906+0000 7f3ea0809640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e9c0cb570 0x7f3e9c14ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:29.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.906+0000 7f3ea0809640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e9c153d10 0x7f3e9c1f6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:29.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.906+0000 7f3ea0809640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e9c15a000 0x7f3e9c154190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:29.910 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.906+0000 7f3ea0809640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c155b00 con 0x7f3e9c153d10 2023-10-30T12:51:29.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.906+0000 7f3ea0809640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c14f120 con 0x7f3e9c15a000 2023-10-30T12:51:29.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.906+0000 7f3ea0809640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c1f6f80 con 0x7f3e9c0cb570 2023-10-30T12:51:29.911 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.906+0000 7f3e99d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e9c153d10 0x7f3e9c1f6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:29.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.906+0000 7f3e99d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e9c153d10 0x7f3e9c1f6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47268/0 (socket says 172.21.15.89:47268) 2023-10-30T12:51:29.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.906+0000 7f3e99d9b640 1 -- 172.21.15.89:0/4282099740 learned_addr learned my addr 172.21.15.89:0/4282099740 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:29.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.906+0000 7f3e9a59c640 1 --2- 172.21.15.89:0/4282099740 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e9c0cb570 0x7f3e9c14ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:29.912 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.906+0000 7f3e9ad9d640 1 --2- 172.21.15.89:0/4282099740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e9c15a000 0x7f3e9c154190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:29.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.906+0000 7f3e99d9b640 1 -- 172.21.15.89:0/4282099740 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e9c0cb570 msgr2=0x7f3e9c14ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:29.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.906+0000 7f3e99d9b640 1 --2- 172.21.15.89:0/4282099740 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e9c0cb570 0x7f3e9c14ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.913 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.906+0000 7f3e99d9b640 1 -- 172.21.15.89:0/4282099740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e9c15a000 msgr2=0x7f3e9c154190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:29.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.906+0000 7f3e99d9b640 1 --2- 172.21.15.89:0/4282099740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e9c15a000 0x7f3e9c154190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.906+0000 7f3e99d9b640 1 -- 172.21.15.89:0/4282099740 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e94009b30 con 0x7f3e9c153d10 2023-10-30T12:51:29.914 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.906+0000 7f3e9ad9d640 1 --2- 172.21.15.89:0/4282099740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e9c15a000 0x7f3e9c154190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:29.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.906+0000 7f3e99d9b640 1 --2- 172.21.15.89:0/4282099740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e9c153d10 0x7f3e9c1f6a40 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f3e9000d5f0 tx=0x7f3e9000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:29.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.906+0000 7f3e8affd640 1 -- 172.21.15.89:0/4282099740 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e900125f0 con 0x7f3e9c153d10 2023-10-30T12:51:29.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.906+0000 7f3e8affd640 1 -- 172.21.15.89:0/4282099740 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e90012750 con 0x7f3e9c153d10 2023-10-30T12:51:29.915 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.906+0000 7f3e8affd640 1 -- 172.21.15.89:0/4282099740 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e90012b00 con 0x7f3e9c153d10 2023-10-30T12:51:29.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.906+0000 7f3ea0809640 1 -- 172.21.15.89:0/4282099740 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e9c158b60 con 0x7f3e9c153d10 2023-10-30T12:51:29.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.906+0000 7f3ea0809640 1 -- 172.21.15.89:0/4282099740 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e9c155d50 con 0x7f3e9c153d10 2023-10-30T12:51:29.916 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.910+0000 7f3ea0809640 1 -- 172.21.15.89:0/4282099740 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e9c04e170 con 0x7f3e9c153d10 2023-10-30T12:51:29.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.910+0000 7f3e8affd640 1 -- 172.21.15.89:0/4282099740 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3e90012c60 con 0x7f3e9c153d10 2023-10-30T12:51:29.917 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.910+0000 7f3e8affd640 1 --2- 172.21.15.89:0/4282099740 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3e7c0664a0 0x7f3e7c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:29.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.910+0000 7f3e8affd640 1 -- 172.21.15.89:0/4282099740 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f3e900847c0 con 0x7f3e9c153d10 2023-10-30T12:51:29.918 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.910+0000 7f3e9a59c640 1 --2- 172.21.15.89:0/4282099740 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3e7c0664a0 0x7f3e7c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:29.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.910+0000 7f3e9a59c640 1 --2- 172.21.15.89:0/4282099740 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3e7c0664a0 0x7f3e7c068960 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c000f70 tx=0x7f3e8c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:29.919 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:29.914+0000 7f3e8affd640 1 -- 172.21.15.89:0/4282099740 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3e90051830 con 0x7f3e9c153d10 2023-10-30T12:51:30.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.078+0000 7f3ea0809640 1 -- 172.21.15.89:0/4282099740 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3e9c15dd40 con 0x7f3e9c153d10 2023-10-30T12:51:30.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.078+0000 7f3e8affd640 1 -- 172.21.15.89:0/4282099740 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 22 entries v66) v1 ==== 81+0+1348 (secure 0 0 0) 0x7f3e90054e50 con 0x7f3e9c153d10 2023-10-30T12:51:30.078 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.192:6837/2130807115 2023-10-31T12:50:41.318586+0000 2023-10-30T12:51:30.078 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.192:6834/501291700 2023-10-31T12:49:01.291877+0000 2023-10-30T12:51:30.078 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.192:6837/604357386 2023-10-31T12:48:11.263450+0000 2023-10-30T12:51:30.079 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.192:6836/604357386 2023-10-31T12:48:11.263450+0000 2023-10-30T12:51:30.079 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6836/951276252 2023-10-31T12:45:51.226535+0000 2023-10-30T12:51:30.079 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.192:6834/1289220491 2023-10-31T12:31:20.953485+0000 2023-10-30T12:51:30.080 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.192:6835/1289220491 2023-10-31T12:31:20.953485+0000 2023-10-30T12:51:30.080 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.192:6837/1413360446 2023-10-31T12:43:36.177020+0000 2023-10-30T12:51:30.080 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.192:6839/3339589587 2023-10-31T12:34:56.026152+0000 2023-10-30T12:51:30.080 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.192:6838/93914587 2023-10-31T12:42:26.158883+0000 2023-10-30T12:51:30.081 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6837/2053475104 2023-10-31T12:33:10.981311+0000 2023-10-30T12:51:30.081 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.192:6836/2130807115 2023-10-31T12:50:41.318586+0000 2023-10-30T12:51:30.081 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.192:6836/1413360446 2023-10-31T12:43:36.177020+0000 2023-10-30T12:51:30.082 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.192:6838/3339589587 2023-10-31T12:34:56.026152+0000 2023-10-30T12:51:30.082 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6837/951276252 2023-10-31T12:45:51.226535+0000 2023-10-30T12:51:30.082 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.192:6836/1002124471 2023-10-31T12:37:36.081217+0000 2023-10-30T12:51:30.082 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.192:6837/1002124471 2023-10-31T12:37:36.081217+0000 2023-10-30T12:51:30.083 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.192:6835/501291700 2023-10-31T12:49:01.291877+0000 2023-10-30T12:51:30.083 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.192:6834/654256299 2023-10-31T12:40:11.121122+0000 2023-10-30T12:51:30.083 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6836/2053475104 2023-10-31T12:33:10.981311+0000 2023-10-30T12:51:30.083 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.192:6835/654256299 2023-10-31T12:40:11.121122+0000 2023-10-30T12:51:30.084 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.192:6839/93914587 2023-10-31T12:42:26.158883+0000 2023-10-30T12:51:30.085 INFO:teuthology.orchestra.run.smithi089.stderr:listed 22 entries 2023-10-30T12:51:30.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.078+0000 7f3ea0809640 1 -- 172.21.15.89:0/4282099740 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3e7c0664a0 msgr2=0x7f3e7c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:30.085 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.078+0000 7f3ea0809640 1 --2- 172.21.15.89:0/4282099740 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3e7c0664a0 0x7f3e7c068960 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c000f70 tx=0x7f3e8c009040 comp rx=0 tx=0).stop 2023-10-30T12:51:30.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.078+0000 7f3ea0809640 1 -- 172.21.15.89:0/4282099740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e9c153d10 msgr2=0x7f3e9c1f6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:30.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.078+0000 7f3ea0809640 1 --2- 172.21.15.89:0/4282099740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e9c153d10 0x7f3e9c1f6a40 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f3e9000d5f0 tx=0x7f3e9000dab0 comp rx=0 tx=0).stop 2023-10-30T12:51:30.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.078+0000 7f3ea0809640 1 -- 172.21.15.89:0/4282099740 shutdown_connections 2023-10-30T12:51:30.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.078+0000 7f3ea0809640 1 --2- 172.21.15.89:0/4282099740 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f3e7c0664a0 0x7f3e7c068960 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:30.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.078+0000 7f3ea0809640 1 --2- 172.21.15.89:0/4282099740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e9c15a000 0x7f3e9c154190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:30.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.078+0000 7f3ea0809640 1 --2- 172.21.15.89:0/4282099740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e9c153d10 0x7f3e9c1f6a40 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:30.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.078+0000 7f3ea0809640 1 --2- 172.21.15.89:0/4282099740 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e9c0cb570 0x7f3e9c14ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:30.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.078+0000 7f3ea0809640 1 -- 172.21.15.89:0/4282099740 >> 172.21.15.89:0/4282099740 conn(0x7f3e9c06c6b0 msgr2=0x7f3e9c0c70e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:30.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.078+0000 7f3ea0809640 1 -- 172.21.15.89:0/4282099740 shutdown_connections 2023-10-30T12:51:30.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.078+0000 7f3ea0809640 1 -- 172.21.15.89:0/4282099740 wait complete. 2023-10-30T12:51:30.098 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-30T12:51:30.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:51:30.099 DEBUG:teuthology.orchestra.run.smithi089:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-30T12:51:30.475 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2023-10-30T12:51:30.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:51:30.476 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-30T12:51:30.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:51:30.487 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2023-10-30T12:51:30.539 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-30T12:51:30.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:51:30.541 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-30T12:51:30.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:51:30.608 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link delete ceph-brx 2023-10-30T12:51:30.826 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-30T12:51:30.836 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-30T12:51:30.836 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:51:30.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.974+0000 7fba41046640 1 Processor -- start 2023-10-30T12:51:30.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.974+0000 7fba41046640 1 -- start start 2023-10-30T12:51:30.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.974+0000 7fba41046640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fba3c0c6b80 0x7fba3c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:30.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.974+0000 7fba41046640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba3c13f4a0 0x7fba3c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:30.976 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.974+0000 7fba41046640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba3c0c0a50 con 0x7fba3c0c6f60 2023-10-30T12:51:30.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.974+0000 7fba41046640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba3c06f6e0 con 0x7fba3c0c6b80 2023-10-30T12:51:30.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.974+0000 7fba41046640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c06f9c0 con 0x7fba3c13f4a0 2023-10-30T12:51:30.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.978+0000 7fba3ad9d640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fba3c0c6b80 0x7fba3c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:30.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.978+0000 7fba3b59e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba3c13f4a0 0x7fba3c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:30.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.978+0000 7fba3b59e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba3c13f4a0 0x7fba3c0c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.89:56876/0 (socket says 172.21.15.89:56876) 2023-10-30T12:51:30.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.978+0000 7fba3b59e640 1 -- 172.21.15.89:0/400389687 learned_addr learned my addr 172.21.15.89:0/400389687 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:30.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.978+0000 7fba3ad9d640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fba3c0c6b80 0x7fba3c0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.89:56474/0 (socket says 172.21.15.89:56474) 2023-10-30T12:51:30.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.978+0000 7fba3b59e640 1 -- 172.21.15.89:0/400389687 >> v2:172.21.15.192:3300/0 conn(0x7fba3c0c6b80 msgr2=0x7fba3c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:30.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.978+0000 7fba3b59e640 1 --2- 172.21.15.89:0/400389687 >> v2:172.21.15.192:3300/0 conn(0x7fba3c0c6b80 0x7fba3c0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:30.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.978+0000 7fba3b59e640 1 -- 172.21.15.89:0/400389687 >> v1:172.21.15.192:6789/0 conn(0x7fba3c0c6f60 legacy=0x7fba3c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:30.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.978+0000 7fba3b59e640 1 -- 172.21.15.89:0/400389687 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba3c06fca0 con 0x7fba3c13f4a0 2023-10-30T12:51:30.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.978+0000 7fba3ad9d640 1 --2- 172.21.15.89:0/400389687 >> v2:172.21.15.192:3300/0 conn(0x7fba3c0c6b80 0x7fba3c0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:51:30.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.982+0000 7fba3b59e640 1 --2- 172.21.15.89:0/400389687 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba3c13f4a0 0x7fba3c0c0400 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fba34009870 tx=0x7fba3401c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f97eaaaf6ff8f749 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:30.982 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.982+0000 7fba3a59c640 1 -- 172.21.15.89:0/400389687 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba340ab650 con 0x7fba3c13f4a0 2023-10-30T12:51:30.982 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.982+0000 7fba3a59c640 1 -- 172.21.15.89:0/400389687 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba340ab7b0 con 0x7fba3c13f4a0 2023-10-30T12:51:30.982 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.982+0000 7fba3a59c640 1 -- 172.21.15.89:0/400389687 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba340abb40 con 0x7fba3c13f4a0 2023-10-30T12:51:30.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.982+0000 7fba41046640 1 -- 172.21.15.89:0/400389687 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba3c13f4a0 msgr2=0x7fba3c0c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:30.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.982+0000 7fba41046640 1 --2- 172.21.15.89:0/400389687 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba3c13f4a0 0x7fba3c0c0400 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fba34009870 tx=0x7fba3401c540 comp rx=0 tx=0).stop 2023-10-30T12:51:30.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.982+0000 7fba41046640 1 -- 172.21.15.89:0/400389687 shutdown_connections 2023-10-30T12:51:30.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.982+0000 7fba41046640 1 --2- 172.21.15.89:0/400389687 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba3c13f4a0 0x7fba3c0c0400 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:30.984 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.982+0000 7fba41046640 1 --2- 172.21.15.89:0/400389687 >> v2:172.21.15.192:3300/0 conn(0x7fba3c0c6b80 0x7fba3c0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:30.984 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.982+0000 7fba41046640 1 -- 172.21.15.89:0/400389687 >> 172.21.15.89:0/400389687 conn(0x7fba3c0c7910 msgr2=0x7fba3c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:30.984 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.982+0000 7fba41046640 1 -- 172.21.15.89:0/400389687 shutdown_connections 2023-10-30T12:51:30.985 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.982+0000 7fba41046640 1 -- 172.21.15.89:0/400389687 wait complete. 2023-10-30T12:51:30.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.982+0000 7fba41046640 1 Processor -- start 2023-10-30T12:51:30.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.986+0000 7fba41046640 1 -- start start 2023-10-30T12:51:30.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.986+0000 7fba41046640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba3c0c6b80 0x7fba3c15e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:30.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.986+0000 7fba41046640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba3c0c6f60 0x7fba3c155750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:30.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.986+0000 7fba41046640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba3c13f4a0 0x7fba3c15f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:30.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.986+0000 7fba41046640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c0c9c50 con 0x7fba3c0c6f60 2023-10-30T12:51:30.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.986+0000 7fba41046640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c0c9620 con 0x7fba3c0c6b80 2023-10-30T12:51:30.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.986+0000 7fba41046640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c0ca520 con 0x7fba3c13f4a0 2023-10-30T12:51:30.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.986+0000 7fba3259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba3c0c6f60 0x7fba3c155750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:30.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.986+0000 7fba3259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba3c0c6f60 0x7fba3c155750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47276/0 (socket says 172.21.15.89:47276) 2023-10-30T12:51:30.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.986+0000 7fba3b59e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba3c13f4a0 0x7fba3c15f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:30.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.986+0000 7fba3259c640 1 -- 172.21.15.89:0/1507409969 learned_addr learned my addr 172.21.15.89:0/1507409969 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:30.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.986+0000 7fba3ad9d640 1 --2- 172.21.15.89:0/1507409969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba3c0c6b80 0x7fba3c15e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:30.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.986+0000 7fba3259c640 1 -- 172.21.15.89:0/1507409969 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba3c13f4a0 msgr2=0x7fba3c15f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:30.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.986+0000 7fba3259c640 1 --2- 172.21.15.89:0/1507409969 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba3c13f4a0 0x7fba3c15f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:30.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.986+0000 7fba3259c640 1 -- 172.21.15.89:0/1507409969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba3c0c6b80 msgr2=0x7fba3c15e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:30.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.986+0000 7fba3259c640 1 --2- 172.21.15.89:0/1507409969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba3c0c6b80 0x7fba3c15e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:30.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.986+0000 7fba3259c640 1 -- 172.21.15.89:0/1507409969 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba2c009a70 con 0x7fba3c0c6f60 2023-10-30T12:51:30.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.986+0000 7fba3259c640 1 --2- 172.21.15.89:0/1507409969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba3c0c6f60 0x7fba3c155750 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fba2400c530 tx=0x7fba2400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:30.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.986+0000 7fba33fff640 1 -- 172.21.15.89:0/1507409969 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba24005220 con 0x7fba3c0c6f60 2023-10-30T12:51:30.994 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.986+0000 7fba33fff640 1 -- 172.21.15.89:0/1507409969 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba24005380 con 0x7fba3c0c6f60 2023-10-30T12:51:30.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.986+0000 7fba33fff640 1 -- 172.21.15.89:0/1507409969 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba24007270 con 0x7fba3c0c6f60 2023-10-30T12:51:30.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.986+0000 7fba41046640 1 -- 172.21.15.89:0/1507409969 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba340094d0 con 0x7fba3c0c6f60 2023-10-30T12:51:30.995 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.986+0000 7fba41046640 1 -- 172.21.15.89:0/1507409969 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba3c153f00 con 0x7fba3c0c6f60 2023-10-30T12:51:30.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.986+0000 7fba41046640 1 -- 172.21.15.89:0/1507409969 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba3c0c9d90 con 0x7fba3c0c6f60 2023-10-30T12:51:30.997 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.990+0000 7fba33fff640 1 -- 172.21.15.89:0/1507409969 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fba240073d0 con 0x7fba3c0c6f60 2023-10-30T12:51:30.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.990+0000 7fba33fff640 1 --2- 172.21.15.89:0/1507409969 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fba1c0664a0 0x7fba1c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:30.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.990+0000 7fba33fff640 1 -- 172.21.15.89:0/1507409969 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7fba240839b0 con 0x7fba3c0c6f60 2023-10-30T12:51:30.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.990+0000 7fba3ad9d640 1 --2- 172.21.15.89:0/1507409969 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fba1c0664a0 0x7fba1c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:30.998 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.990+0000 7fba3ad9d640 1 --2- 172.21.15.89:0/1507409969 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fba1c0664a0 0x7fba1c068960 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fba2c009fd0 tx=0x7fba2c00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:30.999 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:30.994+0000 7fba33fff640 1 -- 172.21.15.89:0/1507409969 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fba3c0c9d90 con 0x7fba3c0c6f60 2023-10-30T12:51:31.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.142+0000 7fba41046640 1 -- 172.21.15.89:0/1507409969 --> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fba3c0c9d90 con 0x7fba1c0664a0 2023-10-30T12:51:31.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.170+0000 7fba33fff640 1 -- 172.21.15.89:0/1507409969 <== mgr.4105 v2:172.21.15.192:6832/4076276638 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+292222 (secure 0 0 0) 0x7fba3c0c9d90 con 0x7fba1c0664a0 2023-10-30T12:51:31.173 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:51:31.175 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-10-30T12:51:31.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.174+0000 7fba41046640 1 -- 172.21.15.89:0/1507409969 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fba1c0664a0 msgr2=0x7fba1c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:31.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.174+0000 7fba41046640 1 --2- 172.21.15.89:0/1507409969 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fba1c0664a0 0x7fba1c068960 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fba2c009fd0 tx=0x7fba2c00b040 comp rx=0 tx=0).stop 2023-10-30T12:51:31.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.174+0000 7fba41046640 1 -- 172.21.15.89:0/1507409969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba3c0c6f60 msgr2=0x7fba3c155750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:31.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.174+0000 7fba41046640 1 --2- 172.21.15.89:0/1507409969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba3c0c6f60 0x7fba3c155750 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fba2400c530 tx=0x7fba2400c9f0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.174+0000 7fba41046640 1 -- 172.21.15.89:0/1507409969 shutdown_connections 2023-10-30T12:51:31.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.174+0000 7fba41046640 1 --2- 172.21.15.89:0/1507409969 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fba1c0664a0 0x7fba1c068960 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.174+0000 7fba41046640 1 --2- 172.21.15.89:0/1507409969 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba3c13f4a0 0x7fba3c15f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.174+0000 7fba41046640 1 --2- 172.21.15.89:0/1507409969 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba3c0c6f60 0x7fba3c155750 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.174+0000 7fba41046640 1 --2- 172.21.15.89:0/1507409969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba3c0c6b80 0x7fba3c15e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.174+0000 7fba41046640 1 -- 172.21.15.89:0/1507409969 >> 172.21.15.89:0/1507409969 conn(0x7fba3c0c7910 msgr2=0x7fba3c13ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:31.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.174+0000 7fba41046640 1 -- 172.21.15.89:0/1507409969 shutdown_connections 2023-10-30T12:51:31.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.174+0000 7fba41046640 1 -- 172.21.15.89:0/1507409969 wait complete. 2023-10-30T12:51:31.193 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":745,"stamp":"2023-10-30T12:51:29.428334+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1063030610,"num_objects":1094,"num_object_clones":0,"num_object_copies":2188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1094,"num_whiteouts":0,"num_read":4359,"num_read_kb":813147,"num_write":41998,"num_write_kb":1354122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":78,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45201,"ondisk_log_size":45201,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2612756,"kb_used_data":2085780,"kb_used_omap":12,"kb_used_meta":526899,"kb_avail":752361964,"statfs":{"total":773094113280,"available":770418651136,"internally_reserved":0,"allocated":2135838720,"data_stored":2129160931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12741,"internal_metadata":539545147},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2682127,"num_objects":-61,"num_object_clones":0,"num_object_copies":-122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-61,"num_whiteouts":0,"num_read":67,"num_read_kb":25,"num_write":252,"num_write_kb":3463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004750"},"pg_stats":[{"pgid":"3.3f","version":"65'525","reported_seq":2677,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.734150+0000","last_change":"2023-10-30T12:38:17.335461+0000","last_active":"2023-10-30T12:51:12.734150+0000","last_peered":"2023-10-30T12:51:12.734150+0000","last_clean":"2023-10-30T12:51:12.734150+0000","last_became_active":"2023-10-30T12:28:50.979351+0000","last_became_peered":"2023-10-30T12:28:50.979351+0000","last_unstale":"2023-10-30T12:51:12.734150+0000","last_undegraded":"2023-10-30T12:51:12.734150+0000","last_fullsized":"2023-10-30T12:51:12.734150+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":525,"log_dups_size":0,"ondisk_log_size":525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:04:55.768566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091708599999999994,"stat_sum":{"num_bytes":4266500,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":40,"num_read_kb":15,"num_write":484,"num_write_kb":5355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"66'518","reported_seq":2663,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.730499+0000","last_change":"2023-10-30T12:38:17.104663+0000","last_active":"2023-10-30T12:51:12.730499+0000","last_peered":"2023-10-30T12:51:12.730499+0000","last_clean":"2023-10-30T12:51:12.730499+0000","last_became_active":"2023-10-30T12:28:50.929133+0000","last_became_peered":"2023-10-30T12:28:50.929133+0000","last_unstale":"2023-10-30T12:51:12.730499+0000","last_undegraded":"2023-10-30T12:51:12.730499+0000","last_fullsized":"2023-10-30T12:51:12.730499+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":518,"log_dups_size":0,"ondisk_log_size":518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:05:29.266806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023149070000000002,"stat_sum":{"num_bytes":8405214,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":33,"num_read_kb":10,"num_write":478,"num_write_kb":9481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"66'525","reported_seq":2686,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.742402+0000","last_change":"2023-10-30T12:38:17.643537+0000","last_active":"2023-10-30T12:51:12.742402+0000","last_peered":"2023-10-30T12:51:12.742402+0000","last_clean":"2023-10-30T12:51:12.742402+0000","last_became_active":"2023-10-30T12:28:50.933896+0000","last_became_peered":"2023-10-30T12:28:50.933896+0000","last_unstale":"2023-10-30T12:51:12.742402+0000","last_undegraded":"2023-10-30T12:51:12.742402+0000","last_fullsized":"2023-10-30T12:51:12.742402+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":525,"log_dups_size":0,"ondisk_log_size":525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:45:03.377748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029033520000000001,"stat_sum":{"num_bytes":12676713,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":49,"num_read_kb":8209,"num_write":482,"num_write_kb":25758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"66'502","reported_seq":2645,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.744598+0000","last_change":"2023-10-30T12:38:17.117873+0000","last_active":"2023-10-30T12:51:12.744598+0000","last_peered":"2023-10-30T12:51:12.744598+0000","last_clean":"2023-10-30T12:51:12.744598+0000","last_became_active":"2023-10-30T12:28:50.967774+0000","last_became_peered":"2023-10-30T12:28:50.967774+0000","last_unstale":"2023-10-30T12:51:12.744598+0000","last_undegraded":"2023-10-30T12:51:12.744598+0000","last_fullsized":"2023-10-30T12:51:12.744598+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":502,"log_dups_size":0,"ondisk_log_size":502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:56:36.456625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075479680000000004,"stat_sum":{"num_bytes":4269565,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":30,"num_read_kb":13,"num_write":476,"num_write_kb":5875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"65'508","reported_seq":2666,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.736735+0000","last_change":"2023-10-30T12:38:17.333966+0000","last_active":"2023-10-30T12:51:12.736735+0000","last_peered":"2023-10-30T12:51:12.736735+0000","last_clean":"2023-10-30T12:51:12.736735+0000","last_became_active":"2023-10-30T12:28:50.973132+0000","last_became_peered":"2023-10-30T12:28:50.973132+0000","last_unstale":"2023-10-30T12:51:12.736735+0000","last_undegraded":"2023-10-30T12:51:12.736735+0000","last_fullsized":"2023-10-30T12:51:12.736735+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":508,"log_dups_size":0,"ondisk_log_size":508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:43:42.826908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0090421609999999999,"stat_sum":{"num_bytes":12646568,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":47,"num_read_kb":16,"num_write":486,"num_write_kb":14458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"66'522","reported_seq":2671,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:22.381126+0000","last_change":"2023-10-30T12:38:17.112051+0000","last_active":"2023-10-30T12:51:22.381126+0000","last_peered":"2023-10-30T12:51:22.381126+0000","last_clean":"2023-10-30T12:51:22.381126+0000","last_became_active":"2023-10-30T12:28:50.967910+0000","last_became_peered":"2023-10-30T12:28:50.967910+0000","last_unstale":"2023-10-30T12:51:22.381126+0000","last_undegraded":"2023-10-30T12:51:22.381126+0000","last_fullsized":"2023-10-30T12:51:22.381126+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":522,"log_dups_size":0,"ondisk_log_size":522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:31:53.805701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040413009999999997,"stat_sum":{"num_bytes":12682645,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":38,"num_read_kb":16,"num_write":498,"num_write_kb":14506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"66'531","reported_seq":2683,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:22.402041+0000","last_change":"2023-10-30T12:38:17.328004+0000","last_active":"2023-10-30T12:51:22.402041+0000","last_peered":"2023-10-30T12:51:22.402041+0000","last_clean":"2023-10-30T12:51:22.402041+0000","last_became_active":"2023-10-30T12:28:50.973288+0000","last_became_peered":"2023-10-30T12:28:50.973288+0000","last_unstale":"2023-10-30T12:51:22.402041+0000","last_undegraded":"2023-10-30T12:51:22.402041+0000","last_fullsized":"2023-10-30T12:51:22.402041+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":531,"log_dups_size":0,"ondisk_log_size":531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:42:52.258224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037299339999999999,"stat_sum":{"num_bytes":4559885,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":41,"num_read_kb":11,"num_write":501,"num_write_kb":7147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"65'470","reported_seq":2620,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.696226+0000","last_change":"2023-10-30T12:38:17.330519+0000","last_active":"2023-10-30T12:51:12.696226+0000","last_peered":"2023-10-30T12:51:12.696226+0000","last_clean":"2023-10-30T12:51:12.696226+0000","last_became_active":"2023-10-30T12:28:50.954988+0000","last_became_peered":"2023-10-30T12:28:50.954988+0000","last_unstale":"2023-10-30T12:51:12.696226+0000","last_undegraded":"2023-10-30T12:51:12.696226+0000","last_fullsized":"2023-10-30T12:51:12.696226+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":470,"log_dups_size":0,"ondisk_log_size":470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:01:29.621026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028962670000000001,"stat_sum":{"num_bytes":8415120,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":15,"num_write":450,"num_write_kb":10234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"66'548","reported_seq":2708,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.686154+0000","last_change":"2023-10-30T12:38:17.336435+0000","last_active":"2023-10-30T12:51:23.686154+0000","last_peered":"2023-10-30T12:51:23.686154+0000","last_clean":"2023-10-30T12:51:23.686154+0000","last_became_active":"2023-10-30T12:28:50.976564+0000","last_became_peered":"2023-10-30T12:28:50.976564+0000","last_unstale":"2023-10-30T12:51:23.686154+0000","last_undegraded":"2023-10-30T12:51:23.686154+0000","last_fullsized":"2023-10-30T12:51:23.686154+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":548,"log_dups_size":0,"ondisk_log_size":548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:02.733738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0094697280000000002,"stat_sum":{"num_bytes":16822419,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":49,"num_read_kb":4116,"num_write":506,"num_write_kb":27572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"66'518","reported_seq":2688,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.731428+0000","last_change":"2023-10-30T12:38:17.327384+0000","last_active":"2023-10-30T12:51:12.731428+0000","last_peered":"2023-10-30T12:51:12.731428+0000","last_clean":"2023-10-30T12:51:12.731428+0000","last_became_active":"2023-10-30T12:28:50.951809+0000","last_became_peered":"2023-10-30T12:28:50.951809+0000","last_unstale":"2023-10-30T12:51:12.731428+0000","last_undegraded":"2023-10-30T12:51:12.731428+0000","last_fullsized":"2023-10-30T12:51:12.731428+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":518,"log_dups_size":0,"ondisk_log_size":518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:26:18.073656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0078740100000000007,"stat_sum":{"num_bytes":16937210,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":59,"num_read_kb":29,"num_write":484,"num_write_kb":22711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"66'547","reported_seq":2711,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:13.217881+0000","last_change":"2023-10-30T12:38:17.323179+0000","last_active":"2023-10-30T12:51:13.217881+0000","last_peered":"2023-10-30T12:51:13.217881+0000","last_clean":"2023-10-30T12:51:13.217881+0000","last_became_active":"2023-10-30T12:29:32.958473+0000","last_became_peered":"2023-10-30T12:29:32.958473+0000","last_unstale":"2023-10-30T12:51:13.217881+0000","last_undegraded":"2023-10-30T12:51:13.217881+0000","last_fullsized":"2023-10-30T12:51:13.217881+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":547,"log_dups_size":0,"ondisk_log_size":547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:07:47.715571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045626240000000004,"stat_sum":{"num_bytes":13032007,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":46,"num_read_kb":17,"num_write":521,"num_write_kb":14861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"65'484","reported_seq":2629,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.654545+0000","last_change":"2023-10-30T12:38:17.113891+0000","last_active":"2023-10-30T12:51:12.654545+0000","last_peered":"2023-10-30T12:51:12.654545+0000","last_clean":"2023-10-30T12:51:12.654545+0000","last_became_active":"2023-10-30T12:28:50.943787+0000","last_became_peered":"2023-10-30T12:28:50.943787+0000","last_unstale":"2023-10-30T12:51:12.654545+0000","last_undegraded":"2023-10-30T12:51:12.654545+0000","last_fullsized":"2023-10-30T12:51:12.654545+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":484,"log_dups_size":0,"ondisk_log_size":484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:51:54.467182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047342360000000002,"stat_sum":{"num_bytes":12598536,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":34,"num_read_kb":4115,"num_write":456,"num_write_kb":18754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"66'573","reported_seq":2734,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.692359+0000","last_change":"2023-10-30T12:38:17.331215+0000","last_active":"2023-10-30T12:51:23.692359+0000","last_peered":"2023-10-30T12:51:23.692359+0000","last_clean":"2023-10-30T12:51:23.692359+0000","last_became_active":"2023-10-30T12:28:50.937906+0000","last_became_peered":"2023-10-30T12:28:50.937906+0000","last_unstale":"2023-10-30T12:51:23.692359+0000","last_undegraded":"2023-10-30T12:51:23.692359+0000","last_fullsized":"2023-10-30T12:51:23.692359+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":573,"log_dups_size":0,"ondisk_log_size":573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:43:06.819593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003369164,"stat_sum":{"num_bytes":4252811,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":50,"num_read_kb":8215,"num_write":540,"num_write_kb":14536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"66'509","reported_seq":2667,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.741036+0000","last_change":"2023-10-30T12:38:17.643256+0000","last_active":"2023-10-30T12:51:12.741036+0000","last_peered":"2023-10-30T12:51:12.741036+0000","last_clean":"2023-10-30T12:51:12.741036+0000","last_became_active":"2023-10-30T12:28:50.932753+0000","last_became_peered":"2023-10-30T12:28:50.932753+0000","last_unstale":"2023-10-30T12:51:12.741036+0000","last_undegraded":"2023-10-30T12:51:12.741036+0000","last_fullsized":"2023-10-30T12:51:12.741036+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":509,"log_dups_size":0,"ondisk_log_size":509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:37:01.519745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027063,"stat_sum":{"num_bytes":16922391,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":47,"num_read_kb":17,"num_write":484,"num_write_kb":19095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"65'539","reported_seq":2700,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.711318+0000","last_change":"2023-10-30T12:38:17.103685+0000","last_active":"2023-10-30T12:51:12.711318+0000","last_peered":"2023-10-30T12:51:12.711318+0000","last_clean":"2023-10-30T12:51:12.711318+0000","last_became_active":"2023-10-30T12:28:50.943011+0000","last_became_peered":"2023-10-30T12:28:50.943011+0000","last_unstale":"2023-10-30T12:51:12.711318+0000","last_undegraded":"2023-10-30T12:51:12.711318+0000","last_fullsized":"2023-10-30T12:51:12.711318+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":539,"log_dups_size":0,"ondisk_log_size":539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:43:12.517498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001813863,"stat_sum":{"num_bytes":21134561,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":51,"num_read_kb":4115,"num_write":503,"num_write_kb":22902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"66'537","reported_seq":2694,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.683828+0000","last_change":"2023-10-30T12:38:17.111689+0000","last_active":"2023-10-30T12:51:23.683828+0000","last_peered":"2023-10-30T12:51:23.683828+0000","last_clean":"2023-10-30T12:51:23.683828+0000","last_became_active":"2023-10-30T12:28:50.952744+0000","last_became_peered":"2023-10-30T12:28:50.952744+0000","last_unstale":"2023-10-30T12:51:23.683828+0000","last_undegraded":"2023-10-30T12:51:23.683828+0000","last_fullsized":"2023-10-30T12:51:23.683828+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":537,"log_dups_size":0,"ondisk_log_size":537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:46:48.822756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034371530000000001,"stat_sum":{"num_bytes":4653479,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":18,"num_write":512,"num_write_kb":9410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"66'554","reported_seq":2705,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:18.626233+0000","last_change":"2023-10-30T12:38:17.120572+0000","last_active":"2023-10-30T12:51:18.626233+0000","last_peered":"2023-10-30T12:51:18.626233+0000","last_clean":"2023-10-30T12:51:18.626233+0000","last_became_active":"2023-10-30T12:28:50.974576+0000","last_became_peered":"2023-10-30T12:28:50.974576+0000","last_unstale":"2023-10-30T12:51:18.626233+0000","last_undegraded":"2023-10-30T12:51:18.626233+0000","last_fullsized":"2023-10-30T12:51:18.626233+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":554,"log_dups_size":0,"ondisk_log_size":554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:19:51.714260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080003049999999992,"stat_sum":{"num_bytes":8578534,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":4113,"num_write":520,"num_write_kb":16515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"66'507","reported_seq":2676,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.747521+0000","last_change":"2023-10-30T12:38:17.285320+0000","last_active":"2023-10-30T12:51:12.747521+0000","last_peered":"2023-10-30T12:51:12.747521+0000","last_clean":"2023-10-30T12:51:12.747521+0000","last_became_active":"2023-10-30T12:28:50.937384+0000","last_became_peered":"2023-10-30T12:28:50.937384+0000","last_unstale":"2023-10-30T12:51:12.747521+0000","last_undegraded":"2023-10-30T12:51:12.747521+0000","last_fullsized":"2023-10-30T12:51:12.747521+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:54:42.597082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001162803,"stat_sum":{"num_bytes":8483984,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":58,"num_read_kb":27,"num_write":482,"num_write_kb":9419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"66'517","reported_seq":2684,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.745991+0000","last_change":"2023-10-30T12:38:17.336198+0000","last_active":"2023-10-30T12:51:12.745991+0000","last_peered":"2023-10-30T12:51:12.745991+0000","last_clean":"2023-10-30T12:51:12.745991+0000","last_became_active":"2023-10-30T12:28:50.973369+0000","last_became_peered":"2023-10-30T12:28:50.973369+0000","last_unstale":"2023-10-30T12:51:12.745991+0000","last_undegraded":"2023-10-30T12:51:12.745991+0000","last_fullsized":"2023-10-30T12:51:12.745991+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:57:45.650842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093923919999999994,"stat_sum":{"num_bytes":12735800,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":55,"num_read_kb":4114,"num_write":492,"num_write_kb":18044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"66'550","reported_seq":2711,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:22.554611+0000","last_change":"2023-10-30T12:38:17.325892+0000","last_active":"2023-10-30T12:51:22.554611+0000","last_peered":"2023-10-30T12:51:22.554611+0000","last_clean":"2023-10-30T12:51:22.554611+0000","last_became_active":"2023-10-30T12:28:50.954831+0000","last_became_peered":"2023-10-30T12:28:50.954831+0000","last_unstale":"2023-10-30T12:51:22.554611+0000","last_undegraded":"2023-10-30T12:51:22.554611+0000","last_fullsized":"2023-10-30T12:51:22.554611+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":550,"log_dups_size":0,"ondisk_log_size":550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:54:29.936622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007119116,"stat_sum":{"num_bytes":21023102,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":49,"num_read_kb":4113,"num_write":518,"num_write_kb":30328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"66'515","reported_seq":2668,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.686674+0000","last_change":"2023-10-30T12:38:17.644608+0000","last_active":"2023-10-30T12:51:23.686674+0000","last_peered":"2023-10-30T12:51:23.686674+0000","last_clean":"2023-10-30T12:51:23.686674+0000","last_became_active":"2023-10-30T12:28:50.962065+0000","last_became_peered":"2023-10-30T12:28:50.962065+0000","last_unstale":"2023-10-30T12:51:23.686674+0000","last_undegraded":"2023-10-30T12:51:23.686674+0000","last_fullsized":"2023-10-30T12:51:23.686674+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:07:44.264938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037739459999999998,"stat_sum":{"num_bytes":25185776,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":41,"num_read_kb":21,"num_write":488,"num_write_kb":26337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"66'577","reported_seq":2735,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:22.381186+0000","last_change":"2023-10-30T12:38:17.118400+0000","last_active":"2023-10-30T12:51:22.381186+0000","last_peered":"2023-10-30T12:51:22.381186+0000","last_clean":"2023-10-30T12:51:22.381186+0000","last_became_active":"2023-10-30T12:28:50.967651+0000","last_became_peered":"2023-10-30T12:28:50.967651+0000","last_unstale":"2023-10-30T12:51:22.381186+0000","last_undegraded":"2023-10-30T12:51:22.381186+0000","last_fullsized":"2023-10-30T12:51:22.381186+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":577,"log_dups_size":0,"ondisk_log_size":577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:19:34.662364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081085610000000002,"stat_sum":{"num_bytes":8464992,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":21,"num_write":548,"num_write_kb":13569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"66'530","reported_seq":2692,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.738894+0000","last_change":"2023-10-30T12:38:17.332602+0000","last_active":"2023-10-30T12:51:12.738894+0000","last_peered":"2023-10-30T12:51:12.738894+0000","last_clean":"2023-10-30T12:51:12.738894+0000","last_became_active":"2023-10-30T12:28:50.962290+0000","last_became_peered":"2023-10-30T12:28:50.962290+0000","last_unstale":"2023-10-30T12:51:12.738894+0000","last_undegraded":"2023-10-30T12:51:12.738894+0000","last_fullsized":"2023-10-30T12:51:12.738894+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:45:30.777077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042026670000000002,"stat_sum":{"num_bytes":57151,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":50,"num_read_kb":4117,"num_write":500,"num_write_kb":6118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"66'550","reported_seq":2702,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:24.465670+0000","last_change":"2023-10-30T12:38:17.114285+0000","last_active":"2023-10-30T12:51:24.465670+0000","last_peered":"2023-10-30T12:51:24.465670+0000","last_clean":"2023-10-30T12:51:24.465670+0000","last_became_active":"2023-10-30T12:28:50.965713+0000","last_became_peered":"2023-10-30T12:28:50.965713+0000","last_unstale":"2023-10-30T12:51:24.465670+0000","last_undegraded":"2023-10-30T12:51:24.465670+0000","last_fullsized":"2023-10-30T12:51:24.465670+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":550,"log_dups_size":0,"ondisk_log_size":550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:08:31.859565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056034209999999999,"stat_sum":{"num_bytes":12649475,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":40,"num_read_kb":16,"num_write":522,"num_write_kb":18815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"66'503","reported_seq":2646,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:18.626419+0000","last_change":"2023-10-30T12:38:17.330864+0000","last_active":"2023-10-30T12:51:18.626419+0000","last_peered":"2023-10-30T12:51:18.626419+0000","last_clean":"2023-10-30T12:51:18.626419+0000","last_became_active":"2023-10-30T12:28:50.951026+0000","last_became_peered":"2023-10-30T12:28:50.951026+0000","last_unstale":"2023-10-30T12:51:18.626419+0000","last_undegraded":"2023-10-30T12:51:18.626419+0000","last_fullsized":"2023-10-30T12:51:18.626419+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":503,"log_dups_size":0,"ondisk_log_size":503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:39:54.872161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031697840000000001,"stat_sum":{"num_bytes":13094288,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":31,"num_read_kb":12,"num_write":474,"num_write_kb":19622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"66'579","reported_seq":2742,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.737087+0000","last_change":"2023-10-30T12:38:17.327887+0000","last_active":"2023-10-30T12:51:12.737087+0000","last_peered":"2023-10-30T12:51:12.737087+0000","last_clean":"2023-10-30T12:51:12.737087+0000","last_became_active":"2023-10-30T12:28:50.962484+0000","last_became_peered":"2023-10-30T12:28:50.962484+0000","last_unstale":"2023-10-30T12:51:12.737087+0000","last_undegraded":"2023-10-30T12:51:12.737087+0000","last_fullsized":"2023-10-30T12:51:12.737087+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:41:18.860625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083773500000000004,"stat_sum":{"num_bytes":13048022,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":51,"num_read_kb":23,"num_write":546,"num_write_kb":15053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"66'481","reported_seq":2635,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.743969+0000","last_change":"2023-10-30T12:38:17.104453+0000","last_active":"2023-10-30T12:51:12.743969+0000","last_peered":"2023-10-30T12:51:12.743969+0000","last_clean":"2023-10-30T12:51:12.743969+0000","last_became_active":"2023-10-30T12:28:50.933093+0000","last_became_peered":"2023-10-30T12:28:50.933093+0000","last_unstale":"2023-10-30T12:51:12.743969+0000","last_undegraded":"2023-10-30T12:51:12.743969+0000","last_fullsized":"2023-10-30T12:51:12.743969+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":481,"log_dups_size":0,"ondisk_log_size":481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:58:51.514957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022251829999999999,"stat_sum":{"num_bytes":25268103,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":42,"num_read_kb":8207,"num_write":439,"num_write_kb":27476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"65'470","reported_seq":2608,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.727936+0000","last_change":"2023-10-30T12:38:17.332060+0000","last_active":"2023-10-30T12:51:12.727936+0000","last_peered":"2023-10-30T12:51:12.727936+0000","last_clean":"2023-10-30T12:51:12.727936+0000","last_became_active":"2023-10-30T12:28:50.952531+0000","last_became_peered":"2023-10-30T12:28:50.952531+0000","last_unstale":"2023-10-30T12:51:12.727936+0000","last_undegraded":"2023-10-30T12:51:12.727936+0000","last_fullsized":"2023-10-30T12:51:12.727936+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":470,"log_dups_size":0,"ondisk_log_size":470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:22:53.945790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039301029999999999,"stat_sum":{"num_bytes":4497430,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":26,"num_read_kb":3284,"num_write":447,"num_write_kb":14120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"65'492","reported_seq":2649,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.733923+0000","last_change":"2023-10-30T12:38:17.644463+0000","last_active":"2023-10-30T12:51:12.733923+0000","last_peered":"2023-10-30T12:51:12.733923+0000","last_clean":"2023-10-30T12:51:12.733923+0000","last_became_active":"2023-10-30T12:28:50.933124+0000","last_became_peered":"2023-10-30T12:28:50.933124+0000","last_unstale":"2023-10-30T12:51:12.733923+0000","last_undegraded":"2023-10-30T12:51:12.733923+0000","last_fullsized":"2023-10-30T12:51:12.733923+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":492,"log_dups_size":0,"ondisk_log_size":492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:25:34.819552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034752709999999998,"stat_sum":{"num_bytes":8485989,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":45,"num_read_kb":14,"num_write":462,"num_write_kb":10450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"66'566","reported_seq":2729,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.737666+0000","last_change":"2023-10-30T12:38:17.118502+0000","last_active":"2023-10-30T12:51:12.737666+0000","last_peered":"2023-10-30T12:51:12.737666+0000","last_clean":"2023-10-30T12:51:12.737666+0000","last_became_active":"2023-10-30T12:28:50.965639+0000","last_became_peered":"2023-10-30T12:28:50.965639+0000","last_unstale":"2023-10-30T12:51:12.737666+0000","last_undegraded":"2023-10-30T12:51:12.737666+0000","last_fullsized":"2023-10-30T12:51:12.737666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":566,"log_dups_size":0,"ondisk_log_size":566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:46:12.692237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076861280000000004,"stat_sum":{"num_bytes":8419509,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":51,"num_read_kb":20,"num_write":541,"num_write_kb":11490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"66'561","reported_seq":2723,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.739391+0000","last_change":"2023-10-30T12:38:17.285366+0000","last_active":"2023-10-30T12:51:12.739391+0000","last_peered":"2023-10-30T12:51:12.739391+0000","last_clean":"2023-10-30T12:51:12.739391+0000","last_became_active":"2023-10-30T12:28:50.955084+0000","last_became_peered":"2023-10-30T12:28:50.955084+0000","last_unstale":"2023-10-30T12:51:12.739391+0000","last_undegraded":"2023-10-30T12:51:12.739391+0000","last_fullsized":"2023-10-30T12:51:12.739391+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":561,"log_dups_size":0,"ondisk_log_size":561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:56:51.880241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012043119999999999,"stat_sum":{"num_bytes":4261339,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":50,"num_read_kb":22,"num_write":525,"num_write_kb":9810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"65'533","reported_seq":2698,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.730032+0000","last_change":"2023-10-30T12:38:17.326196+0000","last_active":"2023-10-30T12:51:12.730032+0000","last_peered":"2023-10-30T12:51:12.730032+0000","last_clean":"2023-10-30T12:51:12.730032+0000","last_became_active":"2023-10-30T12:28:50.944775+0000","last_became_peered":"2023-10-30T12:28:50.944775+0000","last_unstale":"2023-10-30T12:51:12.730032+0000","last_undegraded":"2023-10-30T12:51:12.730032+0000","last_fullsized":"2023-10-30T12:51:12.730032+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":533,"log_dups_size":0,"ondisk_log_size":533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:02:19.297081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070573429999999998,"stat_sum":{"num_bytes":12715391,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":53,"num_read_kb":21,"num_write":506,"num_write_kb":13685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"66'553","reported_seq":2704,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.740036+0000","last_change":"2023-10-30T12:38:17.109227+0000","last_active":"2023-10-30T12:51:12.740036+0000","last_peered":"2023-10-30T12:51:12.740036+0000","last_clean":"2023-10-30T12:51:12.740036+0000","last_became_active":"2023-10-30T12:28:50.940567+0000","last_became_peered":"2023-10-30T12:28:50.940567+0000","last_unstale":"2023-10-30T12:51:12.740036+0000","last_undegraded":"2023-10-30T12:51:12.740036+0000","last_fullsized":"2023-10-30T12:51:12.740036+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":553,"log_dups_size":0,"ondisk_log_size":553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:56:41.791482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028523340000000002,"stat_sum":{"num_bytes":9733426,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":40,"num_read_kb":17,"num_write":523,"num_write_kb":12778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"65'278","reported_seq":357,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:46.346718+0000","last_change":"2023-10-30T12:28:50.395670+0000","last_active":"2023-10-30T12:50:46.346718+0000","last_peered":"2023-10-30T12:50:46.346718+0000","last_clean":"2023-10-30T12:50:46.346718+0000","last_became_active":"2023-10-30T12:28:50.395382+0000","last_became_peered":"2023-10-30T12:28:50.395382+0000","last_unstale":"2023-10-30T12:50:46.346718+0000","last_undegraded":"2023-10-30T12:50:46.346718+0000","last_fullsized":"2023-10-30T12:50:46.346718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":278,"log_dups_size":0,"ondisk_log_size":278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:14.858121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582946,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":45059,"num_write":213,"num_write_kb":12413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"66'472","reported_seq":2628,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:18.626809+0000","last_change":"2023-10-30T12:38:17.644896+0000","last_active":"2023-10-30T12:51:18.626809+0000","last_peered":"2023-10-30T12:51:18.626809+0000","last_clean":"2023-10-30T12:51:18.626809+0000","last_became_active":"2023-10-30T12:28:50.952613+0000","last_became_peered":"2023-10-30T12:28:50.952613+0000","last_unstale":"2023-10-30T12:51:18.626809+0000","last_undegraded":"2023-10-30T12:51:18.626809+0000","last_fullsized":"2023-10-30T12:51:18.626809+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":472,"log_dups_size":0,"ondisk_log_size":472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:16:49.712167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038505420000000002,"stat_sum":{"num_bytes":21060260,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":45,"num_read_kb":18,"num_write":442,"num_write_kb":22001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"65'896","reported_seq":1174,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:47.899291+0000","last_change":"2023-10-30T12:28:50.384598+0000","last_active":"2023-10-30T12:50:47.899291+0000","last_peered":"2023-10-30T12:50:47.899291+0000","last_clean":"2023-10-30T12:50:47.899291+0000","last_became_active":"2023-10-30T12:28:50.383898+0000","last_became_peered":"2023-10-30T12:28:50.383898+0000","last_unstale":"2023-10-30T12:50:47.899291+0000","last_undegraded":"2023-10-30T12:50:47.899291+0000","last_fullsized":"2023-10-30T12:50:47.899291+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":896,"log_dups_size":0,"ondisk_log_size":896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:09:50.290004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21559845,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":229,"num_read_kb":49875,"num_write":837,"num_write_kb":21586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"66'442","reported_seq":733,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:26.502943+0000","last_change":"2023-10-30T12:28:50.386193+0000","last_active":"2023-10-30T12:51:26.502943+0000","last_peered":"2023-10-30T12:51:26.502943+0000","last_clean":"2023-10-30T12:51:26.502943+0000","last_became_active":"2023-10-30T12:28:50.385963+0000","last_became_peered":"2023-10-30T12:28:50.385963+0000","last_unstale":"2023-10-30T12:51:26.502943+0000","last_undegraded":"2023-10-30T12:51:26.502943+0000","last_fullsized":"2023-10-30T12:51:26.502943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":442,"log_dups_size":0,"ondisk_log_size":442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:00:55.603212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7740025,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":238,"num_read_kb":9091,"num_write":376,"num_write_kb":7812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"66'601","reported_seq":2768,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:18.629254+0000","last_change":"2023-10-30T12:38:17.330785+0000","last_active":"2023-10-30T12:51:18.629254+0000","last_peered":"2023-10-30T12:51:18.629254+0000","last_clean":"2023-10-30T12:51:18.629254+0000","last_became_active":"2023-10-30T12:28:50.935764+0000","last_became_peered":"2023-10-30T12:28:50.935764+0000","last_unstale":"2023-10-30T12:51:18.629254+0000","last_undegraded":"2023-10-30T12:51:18.629254+0000","last_fullsized":"2023-10-30T12:51:18.629254+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":601,"log_dups_size":0,"ondisk_log_size":601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:59:59.545417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031329320000000002,"stat_sum":{"num_bytes":4238419,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":56,"num_read_kb":4120,"num_write":560,"num_write_kb":13730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"65'568","reported_seq":643,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:46.705145+0000","last_change":"2023-10-30T12:28:50.392213+0000","last_active":"2023-10-30T12:50:46.705145+0000","last_peered":"2023-10-30T12:50:46.705145+0000","last_clean":"2023-10-30T12:50:46.705145+0000","last_became_active":"2023-10-30T12:28:50.391494+0000","last_became_peered":"2023-10-30T12:28:50.391494+0000","last_unstale":"2023-10-30T12:50:46.705145+0000","last_undegraded":"2023-10-30T12:50:46.705145+0000","last_fullsized":"2023-10-30T12:50:46.705145+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:38:37.301840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":36864,"num_write":519,"num_write_kb":12589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"66'565","reported_seq":2722,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.747887+0000","last_change":"2023-10-30T12:38:17.103988+0000","last_active":"2023-10-30T12:51:12.747887+0000","last_peered":"2023-10-30T12:51:12.747887+0000","last_clean":"2023-10-30T12:51:12.747887+0000","last_became_active":"2023-10-30T12:28:50.933062+0000","last_became_peered":"2023-10-30T12:28:50.933062+0000","last_unstale":"2023-10-30T12:51:12.747887+0000","last_undegraded":"2023-10-30T12:51:12.747887+0000","last_fullsized":"2023-10-30T12:51:12.747887+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":565,"log_dups_size":0,"ondisk_log_size":565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:52:57.336116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021140489999999998,"stat_sum":{"num_bytes":33649708,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":46,"num_read_kb":4115,"num_write":540,"num_write_kb":39124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"66'128","reported_seq":201,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.642253+0000","last_change":"2023-10-30T12:28:50.394368+0000","last_active":"2023-10-30T12:51:12.642253+0000","last_peered":"2023-10-30T12:51:12.642253+0000","last_clean":"2023-10-30T12:51:12.642253+0000","last_became_active":"2023-10-30T12:28:50.393686+0000","last_became_peered":"2023-10-30T12:28:50.393686+0000","last_unstale":"2023-10-30T12:51:12.642253+0000","last_undegraded":"2023-10-30T12:51:12.642253+0000","last_fullsized":"2023-10-30T12:51:12.642253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:43:17.726462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":73,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"65'540","reported_seq":2702,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.734949+0000","last_change":"2023-10-30T12:38:17.119770+0000","last_active":"2023-10-30T12:51:12.734949+0000","last_peered":"2023-10-30T12:51:12.734949+0000","last_clean":"2023-10-30T12:51:12.734949+0000","last_became_active":"2023-10-30T12:28:50.963105+0000","last_became_peered":"2023-10-30T12:28:50.963105+0000","last_unstale":"2023-10-30T12:51:12.734949+0000","last_undegraded":"2023-10-30T12:51:12.734949+0000","last_fullsized":"2023-10-30T12:51:12.734949+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":540,"log_dups_size":0,"ondisk_log_size":540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:12:25.298678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080066809999999999,"stat_sum":{"num_bytes":16891650,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":51,"num_read_kb":18,"num_write":510,"num_write_kb":18448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"66'281","reported_seq":354,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:47.673064+0000","last_change":"2023-10-30T12:28:50.397679+0000","last_active":"2023-10-30T12:50:47.673064+0000","last_peered":"2023-10-30T12:50:47.673064+0000","last_clean":"2023-10-30T12:50:47.673064+0000","last_became_active":"2023-10-30T12:28:50.397041+0000","last_became_peered":"2023-10-30T12:28:50.397041+0000","last_unstale":"2023-10-30T12:50:47.673064+0000","last_undegraded":"2023-10-30T12:50:47.673064+0000","last_fullsized":"2023-10-30T12:50:47.673064+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:52:22.564616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":16396,"num_write":240,"num_write_kb":8312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"66'521","reported_seq":2674,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.744719+0000","last_change":"2023-10-30T12:38:17.644799+0000","last_active":"2023-10-30T12:51:12.744719+0000","last_peered":"2023-10-30T12:51:12.744719+0000","last_clean":"2023-10-30T12:51:12.744719+0000","last_became_active":"2023-10-30T12:28:50.966569+0000","last_became_peered":"2023-10-30T12:28:50.966569+0000","last_unstale":"2023-10-30T12:51:12.744719+0000","last_undegraded":"2023-10-30T12:51:12.744719+0000","last_fullsized":"2023-10-30T12:51:12.744719+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:43:03.050908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037972850000000001,"stat_sum":{"num_bytes":16921261,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":42,"num_read_kb":19,"num_write":497,"num_write_kb":17986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"66'370","reported_seq":449,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:18.810052+0000","last_change":"2023-10-30T12:28:50.391527+0000","last_active":"2023-10-30T12:51:18.810052+0000","last_peered":"2023-10-30T12:51:18.810052+0000","last_clean":"2023-10-30T12:51:18.810052+0000","last_became_active":"2023-10-30T12:28:50.391260+0000","last_became_peered":"2023-10-30T12:28:50.391260+0000","last_unstale":"2023-10-30T12:51:18.810052+0000","last_undegraded":"2023-10-30T12:51:18.810052+0000","last_fullsized":"2023-10-30T12:51:18.810052+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:29:34.772490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":22,"num_read_kb":7467,"num_write":318,"num_write_kb":4400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"66'490","reported_seq":2648,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.693047+0000","last_change":"2023-10-30T12:38:17.114492+0000","last_active":"2023-10-30T12:51:12.693047+0000","last_peered":"2023-10-30T12:51:12.693047+0000","last_clean":"2023-10-30T12:51:12.693047+0000","last_became_active":"2023-10-30T12:28:50.974843+0000","last_became_peered":"2023-10-30T12:28:50.974843+0000","last_unstale":"2023-10-30T12:51:12.693047+0000","last_undegraded":"2023-10-30T12:51:12.693047+0000","last_fullsized":"2023-10-30T12:51:12.693047+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":490,"log_dups_size":0,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:31:41.341035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005161372,"stat_sum":{"num_bytes":4290434,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":47,"num_read_kb":17,"num_write":450,"num_write_kb":9837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"66'349","reported_seq":439,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.670523+0000","last_change":"2023-10-30T12:28:50.374838+0000","last_active":"2023-10-30T12:51:12.670523+0000","last_peered":"2023-10-30T12:51:12.670523+0000","last_clean":"2023-10-30T12:51:12.670523+0000","last_became_active":"2023-10-30T12:28:50.373581+0000","last_became_peered":"2023-10-30T12:28:50.373581+0000","last_unstale":"2023-10-30T12:51:12.670523+0000","last_undegraded":"2023-10-30T12:51:12.670523+0000","last_fullsized":"2023-10-30T12:51:12.670523+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:14:51.506745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":43,"num_read_kb":32803,"num_write":291,"num_write_kb":12542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"65'466","reported_seq":2615,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.728219+0000","last_change":"2023-10-30T12:38:17.328512+0000","last_active":"2023-10-30T12:51:12.728219+0000","last_peered":"2023-10-30T12:51:12.728219+0000","last_clean":"2023-10-30T12:51:12.728219+0000","last_became_active":"2023-10-30T12:28:50.973269+0000","last_became_peered":"2023-10-30T12:28:50.973269+0000","last_unstale":"2023-10-30T12:51:12.728219+0000","last_undegraded":"2023-10-30T12:51:12.728219+0000","last_fullsized":"2023-10-30T12:51:12.728219+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":466,"log_dups_size":0,"ondisk_log_size":466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:26:11.417670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004001498,"stat_sum":{"num_bytes":12676913,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":38,"num_read_kb":11,"num_write":436,"num_write_kb":13453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"66'436","reported_seq":527,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.511431+0000","last_change":"2023-10-30T12:28:50.366590+0000","last_active":"2023-10-30T12:51:12.511431+0000","last_peered":"2023-10-30T12:51:12.511431+0000","last_clean":"2023-10-30T12:51:12.511431+0000","last_became_active":"2023-10-30T12:28:50.366269+0000","last_became_peered":"2023-10-30T12:28:50.366269+0000","last_unstale":"2023-10-30T12:51:12.511431+0000","last_undegraded":"2023-10-30T12:51:12.511431+0000","last_fullsized":"2023-10-30T12:51:12.511431+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":436,"log_dups_size":0,"ondisk_log_size":436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:15:11.963438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":52,"num_read_kb":36923,"num_write":386,"num_write_kb":8499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"65'536","reported_seq":2699,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.738218+0000","last_change":"2023-10-30T12:38:17.644043+0000","last_active":"2023-10-30T12:51:12.738218+0000","last_peered":"2023-10-30T12:51:12.738218+0000","last_clean":"2023-10-30T12:51:12.738218+0000","last_became_active":"2023-10-30T12:28:50.961805+0000","last_became_peered":"2023-10-30T12:28:50.961805+0000","last_unstale":"2023-10-30T12:51:12.738218+0000","last_undegraded":"2023-10-30T12:51:12.738218+0000","last_fullsized":"2023-10-30T12:51:12.738218+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":536,"log_dups_size":0,"ondisk_log_size":536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:50:40.603836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032264350000000002,"stat_sum":{"num_bytes":4573455,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":52,"num_read_kb":4114,"num_write":491,"num_write_kb":14136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"66'258","reported_seq":333,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:07.342026+0000","last_change":"2023-10-30T12:28:50.397244+0000","last_active":"2023-10-30T12:51:07.342026+0000","last_peered":"2023-10-30T12:51:07.342026+0000","last_clean":"2023-10-30T12:51:07.342026+0000","last_became_active":"2023-10-30T12:28:50.396727+0000","last_became_peered":"2023-10-30T12:28:50.396727+0000","last_unstale":"2023-10-30T12:51:07.342026+0000","last_undegraded":"2023-10-30T12:51:07.342026+0000","last_fullsized":"2023-10-30T12:51:07.342026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:33.493343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":16164,"num_write":201,"num_write_kb":8311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"65'535","reported_seq":2701,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.708532+0000","last_change":"2023-10-30T12:38:17.284992+0000","last_active":"2023-10-30T12:51:12.708532+0000","last_peered":"2023-10-30T12:51:12.708532+0000","last_clean":"2023-10-30T12:51:12.708532+0000","last_became_active":"2023-10-30T12:29:32.960608+0000","last_became_peered":"2023-10-30T12:29:32.960608+0000","last_unstale":"2023-10-30T12:51:12.708532+0000","last_undegraded":"2023-10-30T12:51:12.708532+0000","last_fullsized":"2023-10-30T12:51:12.708532+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":535,"log_dups_size":0,"ondisk_log_size":535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:22:33.324798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012934999999999999,"stat_sum":{"num_bytes":4287412,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":59,"num_read_kb":33,"num_write":498,"num_write_kb":6237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"65'677","reported_seq":758,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-30T12:50:01.976866+0000","last_change":"2023-10-30T12:28:50.383880+0000","last_active":"2023-10-30T12:50:01.976866+0000","last_peered":"2023-10-30T12:50:01.976866+0000","last_clean":"2023-10-30T12:50:01.976866+0000","last_became_active":"2023-10-30T12:28:50.383570+0000","last_became_peered":"2023-10-30T12:28:50.383570+0000","last_unstale":"2023-10-30T12:50:01.976866+0000","last_undegraded":"2023-10-30T12:50:01.976866+0000","last_fullsized":"2023-10-30T12:50:01.976866+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":677,"log_dups_size":0,"ondisk_log_size":677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:19:42.047520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":37306,"num_write":619,"num_write_kb":16724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"65'498","reported_seq":2642,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.684505+0000","last_change":"2023-10-30T12:38:17.322231+0000","last_active":"2023-10-30T12:51:12.684505+0000","last_peered":"2023-10-30T12:51:12.684505+0000","last_clean":"2023-10-30T12:51:12.684505+0000","last_became_active":"2023-10-30T12:28:50.949485+0000","last_became_peered":"2023-10-30T12:28:50.949485+0000","last_unstale":"2023-10-30T12:51:12.684505+0000","last_undegraded":"2023-10-30T12:51:12.684505+0000","last_fullsized":"2023-10-30T12:51:12.684505+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:30:36.516655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040897299999999998,"stat_sum":{"num_bytes":12614326,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":469,"num_write_kb":14309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"66'437","reported_seq":543,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.615726+0000","last_change":"2023-10-30T12:28:50.362979+0000","last_active":"2023-10-30T12:51:12.615726+0000","last_peered":"2023-10-30T12:51:12.615726+0000","last_clean":"2023-10-30T12:51:12.615726+0000","last_became_active":"2023-10-30T12:28:50.362624+0000","last_became_peered":"2023-10-30T12:28:50.362624+0000","last_unstale":"2023-10-30T12:51:12.615726+0000","last_undegraded":"2023-10-30T12:51:12.615726+0000","last_fullsized":"2023-10-30T12:51:12.615726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:01:13.111816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583024,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":93,"num_read_kb":24662,"num_write":374,"num_write_kb":12554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"66'553","reported_seq":2714,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:18.625349+0000","last_change":"2023-10-30T12:38:17.334068+0000","last_active":"2023-10-30T12:51:18.625349+0000","last_peered":"2023-10-30T12:51:18.625349+0000","last_clean":"2023-10-30T12:51:18.625349+0000","last_became_active":"2023-10-30T12:28:50.979344+0000","last_became_peered":"2023-10-30T12:28:50.979344+0000","last_unstale":"2023-10-30T12:51:18.625349+0000","last_undegraded":"2023-10-30T12:51:18.625349+0000","last_fullsized":"2023-10-30T12:51:18.625349+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":553,"log_dups_size":0,"ondisk_log_size":553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:32:04.956803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0092295529999999997,"stat_sum":{"num_bytes":12635563,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":50,"num_read_kb":8210,"num_write":525,"num_write_kb":23175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"66'245","reported_seq":322,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:25.413473+0000","last_change":"2023-10-30T12:28:50.382175+0000","last_active":"2023-10-30T12:51:25.413473+0000","last_peered":"2023-10-30T12:51:25.413473+0000","last_clean":"2023-10-30T12:51:25.413473+0000","last_became_active":"2023-10-30T12:28:50.381125+0000","last_became_peered":"2023-10-30T12:28:50.381125+0000","last_unstale":"2023-10-30T12:51:25.413473+0000","last_undegraded":"2023-10-30T12:51:25.413473+0000","last_fullsized":"2023-10-30T12:51:25.413473+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":245,"log_dups_size":0,"ondisk_log_size":245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:36:02.652665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":12378,"num_write":394,"num_write_kb":4652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"65'503","reported_seq":2651,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.747470+0000","last_change":"2023-10-30T12:38:17.334809+0000","last_active":"2023-10-30T12:51:12.747470+0000","last_peered":"2023-10-30T12:51:12.747470+0000","last_clean":"2023-10-30T12:51:12.747470+0000","last_became_active":"2023-10-30T12:28:50.977585+0000","last_became_peered":"2023-10-30T12:28:50.977585+0000","last_unstale":"2023-10-30T12:51:12.747470+0000","last_undegraded":"2023-10-30T12:51:12.747470+0000","last_fullsized":"2023-10-30T12:51:12.747470+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":503,"log_dups_size":0,"ondisk_log_size":503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:33:51.736957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089211889999999995,"stat_sum":{"num_bytes":12866878,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":36,"num_read_kb":8205,"num_write":472,"num_write_kb":18794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"65'355","reported_seq":432,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.691046+0000","last_change":"2023-10-30T12:28:50.367258+0000","last_active":"2023-10-30T12:51:12.691046+0000","last_peered":"2023-10-30T12:51:12.691046+0000","last_clean":"2023-10-30T12:51:12.691046+0000","last_became_active":"2023-10-30T12:28:50.366974+0000","last_became_peered":"2023-10-30T12:28:50.366974+0000","last_unstale":"2023-10-30T12:51:12.691046+0000","last_undegraded":"2023-10-30T12:51:12.691046+0000","last_fullsized":"2023-10-30T12:51:12.691046+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:14:48.823902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":15324,"num_write":292,"num_write_kb":8361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"66'510","reported_seq":2672,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:24.460236+0000","last_change":"2023-10-30T12:38:17.337139+0000","last_active":"2023-10-30T12:51:24.460236+0000","last_peered":"2023-10-30T12:51:24.460236+0000","last_clean":"2023-10-30T12:51:24.460236+0000","last_became_active":"2023-10-30T12:28:50.981193+0000","last_became_peered":"2023-10-30T12:28:50.981193+0000","last_unstale":"2023-10-30T12:51:24.460236+0000","last_undegraded":"2023-10-30T12:51:24.460236+0000","last_fullsized":"2023-10-30T12:51:24.460236+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":510,"log_dups_size":0,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:32:10.843997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096579700000000001,"stat_sum":{"num_bytes":12827883,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":50,"num_read_kb":4114,"num_write":480,"num_write_kb":17991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"66'96","reported_seq":169,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:22.334230+0000","last_change":"2023-10-30T12:28:50.368011+0000","last_active":"2023-10-30T12:51:22.334230+0000","last_peered":"2023-10-30T12:51:22.334230+0000","last_clean":"2023-10-30T12:51:22.334230+0000","last_became_active":"2023-10-30T12:28:50.367730+0000","last_became_peered":"2023-10-30T12:28:50.367730+0000","last_unstale":"2023-10-30T12:51:22.334230+0000","last_undegraded":"2023-10-30T12:51:22.334230+0000","last_fullsized":"2023-10-30T12:51:22.334230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:59:41.946550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":12292,"num_write":45,"num_write_kb":4186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"65'566","reported_seq":2734,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.746266+0000","last_change":"2023-10-30T12:38:17.119150+0000","last_active":"2023-10-30T12:51:12.746266+0000","last_peered":"2023-10-30T12:51:12.746266+0000","last_clean":"2023-10-30T12:51:12.746266+0000","last_became_active":"2023-10-30T12:28:50.966246+0000","last_became_peered":"2023-10-30T12:28:50.966246+0000","last_unstale":"2023-10-30T12:51:12.746266+0000","last_undegraded":"2023-10-30T12:51:12.746266+0000","last_fullsized":"2023-10-30T12:51:12.746266+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":566,"log_dups_size":0,"ondisk_log_size":566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:39:33.054939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007703601,"stat_sum":{"num_bytes":8540203,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":56,"num_read_kb":18,"num_write":531,"num_write_kb":11283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"66'83","reported_seq":159,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:18.809472+0000","last_change":"2023-10-30T12:28:50.394060+0000","last_active":"2023-10-30T12:51:18.809472+0000","last_peered":"2023-10-30T12:51:18.809472+0000","last_clean":"2023-10-30T12:51:18.809472+0000","last_became_active":"2023-10-30T12:28:50.393642+0000","last_became_peered":"2023-10-30T12:28:50.393642+0000","last_unstale":"2023-10-30T12:51:18.809472+0000","last_undegraded":"2023-10-30T12:51:18.809472+0000","last_fullsized":"2023-10-30T12:51:18.809472+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:21.643882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":15,"num_write_kb":101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"65'532","reported_seq":2688,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.708661+0000","last_change":"2023-10-30T12:38:17.335584+0000","last_active":"2023-10-30T12:51:12.708661+0000","last_peered":"2023-10-30T12:51:12.708661+0000","last_clean":"2023-10-30T12:51:12.708661+0000","last_became_active":"2023-10-30T12:28:50.976343+0000","last_became_peered":"2023-10-30T12:28:50.976343+0000","last_unstale":"2023-10-30T12:51:12.708661+0000","last_undegraded":"2023-10-30T12:51:12.708661+0000","last_fullsized":"2023-10-30T12:51:12.708661+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":532,"log_dups_size":0,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:03:37.466766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091875959999999993,"stat_sum":{"num_bytes":16963416,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":44,"num_read_kb":14,"num_write":504,"num_write_kb":22707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":220,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:41.797344+0000","last_change":"2023-10-30T12:28:41.203990+0000","last_active":"2023-10-30T12:50:41.797344+0000","last_peered":"2023-10-30T12:50:41.797344+0000","last_clean":"2023-10-30T12:50:41.797344+0000","last_became_active":"2023-10-30T12:28:41.201796+0000","last_became_peered":"2023-10-30T12:28:41.201796+0000","last_unstale":"2023-10-30T12:50:41.797344+0000","last_undegraded":"2023-10-30T12:50:41.797344+0000","last_fullsized":"2023-10-30T12:50:41.797344+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:39.218257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:39.218257+0000","last_clean_scrub_stamp":"2023-10-30T12:28:39.218257+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:21:00.605462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"66'193","reported_seq":273,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.748291+0000","last_change":"2023-10-30T12:28:50.372651+0000","last_active":"2023-10-30T12:51:23.748291+0000","last_peered":"2023-10-30T12:51:23.748291+0000","last_clean":"2023-10-30T12:51:23.748291+0000","last_became_active":"2023-10-30T12:28:50.367593+0000","last_became_peered":"2023-10-30T12:28:50.367593+0000","last_unstale":"2023-10-30T12:51:23.748291+0000","last_undegraded":"2023-10-30T12:51:23.748291+0000","last_fullsized":"2023-10-30T12:51:23.748291+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:32:10.492061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194338,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":24,"num_read_kb":12326,"num_write":144,"num_write_kb":4223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"65'492","reported_seq":2637,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.709943+0000","last_change":"2023-10-30T12:38:17.285044+0000","last_active":"2023-10-30T12:51:12.709943+0000","last_peered":"2023-10-30T12:51:12.709943+0000","last_clean":"2023-10-30T12:51:12.709943+0000","last_became_active":"2023-10-30T12:29:32.997301+0000","last_became_peered":"2023-10-30T12:29:32.997301+0000","last_unstale":"2023-10-30T12:51:12.709943+0000","last_undegraded":"2023-10-30T12:51:12.709943+0000","last_fullsized":"2023-10-30T12:51:12.709943+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":492,"log_dups_size":0,"ondisk_log_size":492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:12:18.045691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001366412,"stat_sum":{"num_bytes":4230333,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":36,"num_read_kb":11,"num_write":471,"num_write_kb":13453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"66'499","reported_seq":586,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:18.810050+0000","last_change":"2023-10-30T12:28:50.392461+0000","last_active":"2023-10-30T12:51:18.810050+0000","last_peered":"2023-10-30T12:51:18.810050+0000","last_clean":"2023-10-30T12:51:18.810050+0000","last_became_active":"2023-10-30T12:28:50.391824+0000","last_became_peered":"2023-10-30T12:28:50.391824+0000","last_unstale":"2023-10-30T12:51:18.810050+0000","last_undegraded":"2023-10-30T12:51:18.810050+0000","last_fullsized":"2023-10-30T12:51:18.810050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":499,"log_dups_size":0,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:56:10.373046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":32,"num_read_kb":50322,"num_write":431,"num_write_kb":16616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"66'503","reported_seq":2644,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.731399+0000","last_change":"2023-10-30T12:38:17.643111+0000","last_active":"2023-10-30T12:51:12.731399+0000","last_peered":"2023-10-30T12:51:12.731399+0000","last_clean":"2023-10-30T12:51:12.731399+0000","last_became_active":"2023-10-30T12:28:50.952625+0000","last_became_peered":"2023-10-30T12:28:50.952625+0000","last_unstale":"2023-10-30T12:51:12.731399+0000","last_undegraded":"2023-10-30T12:51:12.731399+0000","last_fullsized":"2023-10-30T12:51:12.731399+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":503,"log_dups_size":0,"ondisk_log_size":503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:43:17.223141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026695970000000001,"stat_sum":{"num_bytes":16935229,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":29,"num_read_kb":10,"num_write":464,"num_write_kb":20216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"66'579","reported_seq":882,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:25.409837+0000","last_change":"2023-10-30T12:28:50.367547+0000","last_active":"2023-10-30T12:51:25.409837+0000","last_peered":"2023-10-30T12:51:25.409837+0000","last_clean":"2023-10-30T12:51:25.409837+0000","last_became_active":"2023-10-30T12:28:50.367230+0000","last_became_peered":"2023-10-30T12:28:50.367230+0000","last_unstale":"2023-10-30T12:51:25.409837+0000","last_undegraded":"2023-10-30T12:51:25.409837+0000","last_fullsized":"2023-10-30T12:51:25.409837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:42.570786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11931699,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":249,"num_read_kb":13021,"num_write":520,"num_write_kb":11989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"66'527","reported_seq":2687,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.683264+0000","last_change":"2023-10-30T12:38:17.104045+0000","last_active":"2023-10-30T12:51:23.683264+0000","last_peered":"2023-10-30T12:51:23.683264+0000","last_clean":"2023-10-30T12:51:23.683264+0000","last_became_active":"2023-10-30T12:28:50.928608+0000","last_became_peered":"2023-10-30T12:28:50.928608+0000","last_unstale":"2023-10-30T12:51:23.683264+0000","last_undegraded":"2023-10-30T12:51:23.683264+0000","last_fullsized":"2023-10-30T12:51:23.683264+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":527,"log_dups_size":0,"ondisk_log_size":527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:17:59.249824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002074275,"stat_sum":{"num_bytes":8442985,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":47,"num_read_kb":21,"num_write":493,"num_write_kb":11774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"66'493","reported_seq":572,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:46.897234+0000","last_change":"2023-10-30T12:28:50.391240+0000","last_active":"2023-10-30T12:50:46.897234+0000","last_peered":"2023-10-30T12:50:46.897234+0000","last_clean":"2023-10-30T12:50:46.897234+0000","last_became_active":"2023-10-30T12:28:50.390864+0000","last_became_peered":"2023-10-30T12:28:50.390864+0000","last_unstale":"2023-10-30T12:50:46.897234+0000","last_undegraded":"2023-10-30T12:50:46.897234+0000","last_fullsized":"2023-10-30T12:50:46.897234+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":493,"log_dups_size":0,"ondisk_log_size":493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:59:40.285830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":45183,"num_write":427,"num_write_kb":16635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"66'498","reported_seq":2645,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:24.458791+0000","last_change":"2023-10-30T12:38:17.111185+0000","last_active":"2023-10-30T12:51:24.458791+0000","last_peered":"2023-10-30T12:51:24.458791+0000","last_clean":"2023-10-30T12:51:24.458791+0000","last_became_active":"2023-10-30T12:28:50.949122+0000","last_became_peered":"2023-10-30T12:28:50.949122+0000","last_unstale":"2023-10-30T12:51:24.458791+0000","last_undegraded":"2023-10-30T12:51:24.458791+0000","last_fullsized":"2023-10-30T12:51:24.458791+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:32:08.636317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004393348,"stat_sum":{"num_bytes":12654257,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":35,"num_read_kb":65,"num_write":459,"num_write_kb":14113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"66'266","reported_seq":335,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:49.583265+0000","last_change":"2023-10-30T12:28:50.358700+0000","last_active":"2023-10-30T12:50:49.583265+0000","last_peered":"2023-10-30T12:50:49.583265+0000","last_clean":"2023-10-30T12:50:49.583265+0000","last_became_active":"2023-10-30T12:28:50.358310+0000","last_became_peered":"2023-10-30T12:28:50.358310+0000","last_unstale":"2023-10-30T12:50:49.583265+0000","last_undegraded":"2023-10-30T12:50:49.583265+0000","last_fullsized":"2023-10-30T12:50:49.583265+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":266,"log_dups_size":0,"ondisk_log_size":266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:23:30.910622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":5758,"num_write":222,"num_write_kb":4242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"65'512","reported_seq":2657,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.699965+0000","last_change":"2023-10-30T12:38:17.121187+0000","last_active":"2023-10-30T12:51:12.699965+0000","last_peered":"2023-10-30T12:51:12.699965+0000","last_clean":"2023-10-30T12:51:12.699965+0000","last_became_active":"2023-10-30T12:28:50.965626+0000","last_became_peered":"2023-10-30T12:28:50.965626+0000","last_unstale":"2023-10-30T12:51:12.699965+0000","last_undegraded":"2023-10-30T12:51:12.699965+0000","last_fullsized":"2023-10-30T12:51:12.699965+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":512,"log_dups_size":0,"ondisk_log_size":512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:44:24.255740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0079099229999999993,"stat_sum":{"num_bytes":29393213,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":33,"num_read_kb":4109,"num_write":475,"num_write_kb":37010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"66'399","reported_seq":470,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:47.791499+0000","last_change":"2023-10-30T12:28:50.395645+0000","last_active":"2023-10-30T12:50:47.791499+0000","last_peered":"2023-10-30T12:50:47.791499+0000","last_clean":"2023-10-30T12:50:47.791499+0000","last_became_active":"2023-10-30T12:28:50.395090+0000","last_became_peered":"2023-10-30T12:28:50.395090+0000","last_unstale":"2023-10-30T12:50:47.791499+0000","last_undegraded":"2023-10-30T12:50:47.791499+0000","last_fullsized":"2023-10-30T12:50:47.791499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:45:44.678383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582934,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":8195,"num_write":330,"num_write_kb":12489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"66'511","reported_seq":2673,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.700882+0000","last_change":"2023-10-30T12:38:17.104334+0000","last_active":"2023-10-30T12:51:12.700882+0000","last_peered":"2023-10-30T12:51:12.700882+0000","last_clean":"2023-10-30T12:51:12.700882+0000","last_became_active":"2023-10-30T12:28:50.929337+0000","last_became_peered":"2023-10-30T12:28:50.929337+0000","last_unstale":"2023-10-30T12:51:12.700882+0000","last_undegraded":"2023-10-30T12:51:12.700882+0000","last_fullsized":"2023-10-30T12:51:12.700882+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":511,"log_dups_size":0,"ondisk_log_size":511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:32:28.782456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023039890000000002,"stat_sum":{"num_bytes":14084554,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":50,"num_read_kb":18,"num_write":480,"num_write_kb":15463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"66'350","reported_seq":428,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.563637+0000","last_change":"2023-10-30T12:28:50.380474+0000","last_active":"2023-10-30T12:51:23.563637+0000","last_peered":"2023-10-30T12:51:23.563637+0000","last_clean":"2023-10-30T12:51:23.563637+0000","last_became_active":"2023-10-30T12:28:50.380247+0000","last_became_peered":"2023-10-30T12:28:50.380247+0000","last_unstale":"2023-10-30T12:51:23.563637+0000","last_undegraded":"2023-10-30T12:51:23.563637+0000","last_fullsized":"2023-10-30T12:51:23.563637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":350,"log_dups_size":0,"ondisk_log_size":350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:52:03.952617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":8272,"num_write":524,"num_write_kb":5251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"66'511","reported_seq":2653,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.729280+0000","last_change":"2023-10-30T12:38:17.331272+0000","last_active":"2023-10-30T12:51:12.729280+0000","last_peered":"2023-10-30T12:51:12.729280+0000","last_clean":"2023-10-30T12:51:12.729280+0000","last_became_active":"2023-10-30T12:28:50.973086+0000","last_became_peered":"2023-10-30T12:28:50.973086+0000","last_unstale":"2023-10-30T12:51:12.729280+0000","last_undegraded":"2023-10-30T12:51:12.729280+0000","last_fullsized":"2023-10-30T12:51:12.729280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":511,"log_dups_size":0,"ondisk_log_size":511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:53:48.263747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034133169999999999,"stat_sum":{"num_bytes":16794570,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":30,"num_read_kb":14,"num_write":487,"num_write_kb":22103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"66'198","reported_seq":272,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:25.415320+0000","last_change":"2023-10-30T12:28:50.386277+0000","last_active":"2023-10-30T12:51:25.415320+0000","last_peered":"2023-10-30T12:51:25.415320+0000","last_clean":"2023-10-30T12:51:25.415320+0000","last_became_active":"2023-10-30T12:28:50.385924+0000","last_became_peered":"2023-10-30T12:28:50.385924+0000","last_unstale":"2023-10-30T12:51:25.415320+0000","last_undegraded":"2023-10-30T12:51:25.415320+0000","last_fullsized":"2023-10-30T12:51:25.415320+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:35:03.837523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1958032,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":136,"num_write_kb":2002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'500","reported_seq":2655,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.717097+0000","last_change":"2023-10-30T12:38:17.643575+0000","last_active":"2023-10-30T12:51:12.717097+0000","last_peered":"2023-10-30T12:51:12.717097+0000","last_clean":"2023-10-30T12:51:12.717097+0000","last_became_active":"2023-10-30T12:28:50.929259+0000","last_became_peered":"2023-10-30T12:28:50.929259+0000","last_unstale":"2023-10-30T12:51:12.717097+0000","last_undegraded":"2023-10-30T12:51:12.717097+0000","last_fullsized":"2023-10-30T12:51:12.717097+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":500,"log_dups_size":0,"ondisk_log_size":500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:49:01.397563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029033230000000002,"stat_sum":{"num_bytes":8412687,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":43,"num_read_kb":22,"num_write":470,"num_write_kb":11464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"65'373","reported_seq":455,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:41.796144+0000","last_change":"2023-10-30T12:28:50.386426+0000","last_active":"2023-10-30T12:50:41.796144+0000","last_peered":"2023-10-30T12:50:41.796144+0000","last_clean":"2023-10-30T12:50:41.796144+0000","last_became_active":"2023-10-30T12:28:50.385305+0000","last_became_peered":"2023-10-30T12:28:50.385305+0000","last_unstale":"2023-10-30T12:50:41.796144+0000","last_undegraded":"2023-10-30T12:50:41.796144+0000","last_fullsized":"2023-10-30T12:50:41.796144+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":373,"log_dups_size":0,"ondisk_log_size":373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:13:05.376581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":37322,"num_write":310,"num_write_kb":12446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"66'520","reported_seq":2669,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:18.630606+0000","last_change":"2023-10-30T12:38:17.111500+0000","last_active":"2023-10-30T12:51:18.630606+0000","last_peered":"2023-10-30T12:51:18.630606+0000","last_clean":"2023-10-30T12:51:18.630606+0000","last_became_active":"2023-10-30T12:28:50.940846+0000","last_became_peered":"2023-10-30T12:28:50.940846+0000","last_unstale":"2023-10-30T12:51:18.630606+0000","last_undegraded":"2023-10-30T12:51:18.630606+0000","last_fullsized":"2023-10-30T12:51:18.630606+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":520,"log_dups_size":0,"ondisk_log_size":520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:27:33.103707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046016829999999996,"stat_sum":{"num_bytes":12660803,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":37,"num_read_kb":7,"num_write":492,"num_write_kb":18951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"66'441","reported_seq":518,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.687550+0000","last_change":"2023-10-30T12:28:50.392333+0000","last_active":"2023-10-30T12:51:23.687550+0000","last_peered":"2023-10-30T12:51:23.687550+0000","last_clean":"2023-10-30T12:51:23.687550+0000","last_became_active":"2023-10-30T12:28:50.392068+0000","last_became_peered":"2023-10-30T12:28:50.392068+0000","last_unstale":"2023-10-30T12:51:23.687550+0000","last_undegraded":"2023-10-30T12:51:23.687550+0000","last_fullsized":"2023-10-30T12:51:23.687550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":441,"log_dups_size":0,"ondisk_log_size":441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:47:24.674925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388698,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":19,"num_read_kb":11375,"num_write":386,"num_write_kb":8543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"66'523","reported_seq":2679,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.682847+0000","last_change":"2023-10-30T12:38:17.331599+0000","last_active":"2023-10-30T12:51:23.682847+0000","last_peered":"2023-10-30T12:51:23.682847+0000","last_clean":"2023-10-30T12:51:23.682847+0000","last_became_active":"2023-10-30T12:28:50.931898+0000","last_became_peered":"2023-10-30T12:28:50.931898+0000","last_unstale":"2023-10-30T12:51:23.682847+0000","last_undegraded":"2023-10-30T12:51:23.682847+0000","last_fullsized":"2023-10-30T12:51:23.682847+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":523,"log_dups_size":0,"ondisk_log_size":523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:49:23.804972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003570148,"stat_sum":{"num_bytes":4255498,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":44,"num_read_kb":18,"num_write":498,"num_write_kb":10461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"66'553","reported_seq":626,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:24.395115+0000","last_change":"2023-10-30T12:28:50.365055+0000","last_active":"2023-10-30T12:51:24.395115+0000","last_peered":"2023-10-30T12:51:24.395115+0000","last_clean":"2023-10-30T12:51:24.395115+0000","last_became_active":"2023-10-30T12:28:50.364797+0000","last_became_peered":"2023-10-30T12:28:50.364797+0000","last_unstale":"2023-10-30T12:51:24.395115+0000","last_undegraded":"2023-10-30T12:51:24.395115+0000","last_fullsized":"2023-10-30T12:51:24.395115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":553,"log_dups_size":0,"ondisk_log_size":553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:06:23.067895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":10,"num_read_kb":8489,"num_write":689,"num_write_kb":13252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"66'528","reported_seq":2689,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.685808+0000","last_change":"2023-10-30T12:38:17.331931+0000","last_active":"2023-10-30T12:51:23.685808+0000","last_peered":"2023-10-30T12:51:23.685808+0000","last_clean":"2023-10-30T12:51:23.685808+0000","last_became_active":"2023-10-30T12:28:50.935657+0000","last_became_peered":"2023-10-30T12:28:50.935657+0000","last_unstale":"2023-10-30T12:51:23.685808+0000","last_undegraded":"2023-10-30T12:51:23.685808+0000","last_fullsized":"2023-10-30T12:51:23.685808+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":528,"log_dups_size":0,"ondisk_log_size":528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:42:39.657789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038543219999999999,"stat_sum":{"num_bytes":16855474,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":49,"num_read_kb":4112,"num_write":496,"num_write_kb":17584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"66'244","reported_seq":324,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.643888+0000","last_change":"2023-10-30T12:28:50.393465+0000","last_active":"2023-10-30T12:51:12.643888+0000","last_peered":"2023-10-30T12:51:12.643888+0000","last_clean":"2023-10-30T12:51:12.643888+0000","last_became_active":"2023-10-30T12:28:50.392807+0000","last_became_peered":"2023-10-30T12:28:50.392807+0000","last_unstale":"2023-10-30T12:51:12.643888+0000","last_undegraded":"2023-10-30T12:51:12.643888+0000","last_fullsized":"2023-10-30T12:51:12.643888+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":244,"log_dups_size":0,"ondisk_log_size":244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:35:37.976956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":24596,"num_write":178,"num_write_kb":8294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"65'481","reported_seq":2631,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.730761+0000","last_change":"2023-10-30T12:38:17.325460+0000","last_active":"2023-10-30T12:51:12.730761+0000","last_peered":"2023-10-30T12:51:12.730761+0000","last_clean":"2023-10-30T12:51:12.730761+0000","last_became_active":"2023-10-30T12:28:50.954291+0000","last_became_peered":"2023-10-30T12:28:50.954291+0000","last_unstale":"2023-10-30T12:51:12.730761+0000","last_undegraded":"2023-10-30T12:51:12.730761+0000","last_fullsized":"2023-10-30T12:51:12.730761+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":481,"log_dups_size":0,"ondisk_log_size":481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:58:33.115877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047974480000000002,"stat_sum":{"num_bytes":12640290,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":38,"num_read_kb":15,"num_write":450,"num_write_kb":14582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"65'505","reported_seq":2666,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.715029+0000","last_change":"2023-10-30T12:38:17.120085+0000","last_active":"2023-10-30T12:51:12.715029+0000","last_peered":"2023-10-30T12:51:12.715029+0000","last_clean":"2023-10-30T12:51:12.715029+0000","last_became_active":"2023-10-30T12:28:50.962817+0000","last_became_peered":"2023-10-30T12:28:50.962817+0000","last_unstale":"2023-10-30T12:51:12.715029+0000","last_undegraded":"2023-10-30T12:51:12.715029+0000","last_fullsized":"2023-10-30T12:51:12.715029+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":505,"log_dups_size":0,"ondisk_log_size":505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:14:10.476055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070898599999999999,"stat_sum":{"num_bytes":8463933,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":50,"num_read_kb":25,"num_write":479,"num_write_kb":14576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"65'173","reported_seq":249,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.684872+0000","last_change":"2023-10-30T12:28:50.402179+0000","last_active":"2023-10-30T12:51:12.684872+0000","last_peered":"2023-10-30T12:51:12.684872+0000","last_clean":"2023-10-30T12:51:12.684872+0000","last_became_active":"2023-10-30T12:28:50.401677+0000","last_became_peered":"2023-10-30T12:28:50.401677+0000","last_unstale":"2023-10-30T12:51:12.684872+0000","last_undegraded":"2023-10-30T12:51:12.684872+0000","last_fullsized":"2023-10-30T12:51:12.684872+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:39:44.432214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":20498,"num_write":107,"num_write_kb":8247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"66'474","reported_seq":2629,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:22.381301+0000","last_change":"2023-10-30T12:38:17.119058+0000","last_active":"2023-10-30T12:51:22.381301+0000","last_peered":"2023-10-30T12:51:22.381301+0000","last_clean":"2023-10-30T12:51:22.381301+0000","last_became_active":"2023-10-30T12:28:50.967591+0000","last_became_peered":"2023-10-30T12:28:50.967591+0000","last_unstale":"2023-10-30T12:51:22.381301+0000","last_undegraded":"2023-10-30T12:51:22.381301+0000","last_fullsized":"2023-10-30T12:51:22.381301+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":474,"log_dups_size":0,"ondisk_log_size":474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:15:53.272647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081968709999999997,"stat_sum":{"num_bytes":10507344,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":44,"num_read_kb":1610,"num_write":436,"num_write_kb":22884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"66'363","reported_seq":450,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:22.402277+0000","last_change":"2023-10-30T12:28:50.392617+0000","last_active":"2023-10-30T12:51:22.402277+0000","last_peered":"2023-10-30T12:51:22.402277+0000","last_clean":"2023-10-30T12:51:22.402277+0000","last_became_active":"2023-10-30T12:28:50.392021+0000","last_became_peered":"2023-10-30T12:28:50.392021+0000","last_unstale":"2023-10-30T12:51:22.402277+0000","last_undegraded":"2023-10-30T12:51:22.402277+0000","last_fullsized":"2023-10-30T12:51:22.402277+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:52:30.621125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":57384,"num_write":298,"num_write_kb":16576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"65'482","reported_seq":2638,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.744064+0000","last_change":"2023-10-30T12:38:17.643950+0000","last_active":"2023-10-30T12:51:12.744064+0000","last_peered":"2023-10-30T12:51:12.744064+0000","last_clean":"2023-10-30T12:51:12.744064+0000","last_became_active":"2023-10-30T12:28:50.967186+0000","last_became_peered":"2023-10-30T12:28:50.967186+0000","last_unstale":"2023-10-30T12:51:12.744064+0000","last_undegraded":"2023-10-30T12:51:12.744064+0000","last_fullsized":"2023-10-30T12:51:12.744064+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":482,"log_dups_size":0,"ondisk_log_size":482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:48:20.159300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003136738,"stat_sum":{"num_bytes":18043,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":17,"num_write":445,"num_write_kb":3039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"65'227","reported_seq":303,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:46.350266+0000","last_change":"2023-10-30T12:28:50.363230+0000","last_active":"2023-10-30T12:50:46.350266+0000","last_peered":"2023-10-30T12:50:46.350266+0000","last_clean":"2023-10-30T12:50:46.350266+0000","last_became_active":"2023-10-30T12:28:50.362929+0000","last_became_peered":"2023-10-30T12:28:50.362929+0000","last_unstale":"2023-10-30T12:50:46.350266+0000","last_undegraded":"2023-10-30T12:50:46.350266+0000","last_fullsized":"2023-10-30T12:50:46.350266+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:41:27.760578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":9,"num_read_kb":32768,"num_write":162,"num_write_kb":8341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"66'542","reported_seq":2685,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.687065+0000","last_change":"2023-10-30T12:38:17.331548+0000","last_active":"2023-10-30T12:51:23.687065+0000","last_peered":"2023-10-30T12:51:23.687065+0000","last_clean":"2023-10-30T12:51:23.687065+0000","last_became_active":"2023-10-30T12:28:50.937458+0000","last_became_peered":"2023-10-30T12:28:50.937458+0000","last_unstale":"2023-10-30T12:51:23.687065+0000","last_undegraded":"2023-10-30T12:51:23.687065+0000","last_fullsized":"2023-10-30T12:51:23.687065+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":542,"log_dups_size":0,"ondisk_log_size":542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:25:59.947416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003544045,"stat_sum":{"num_bytes":16835848,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":30,"num_read_kb":11,"num_write":511,"num_write_kb":19769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"66'443","reported_seq":537,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:49.583519+0000","last_change":"2023-10-30T12:28:50.386782+0000","last_active":"2023-10-30T12:50:49.583519+0000","last_peered":"2023-10-30T12:50:49.583519+0000","last_clean":"2023-10-30T12:50:49.583519+0000","last_became_active":"2023-10-30T12:28:50.386522+0000","last_became_peered":"2023-10-30T12:28:50.386522+0000","last_unstale":"2023-10-30T12:50:49.583519+0000","last_undegraded":"2023-10-30T12:50:49.583519+0000","last_fullsized":"2023-10-30T12:50:49.583519+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":443,"log_dups_size":0,"ondisk_log_size":443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:44:03.517641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583494,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":64,"num_read_kb":24634,"num_write":377,"num_write_kb":12490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":759200777,"num_objects":915,"num_object_clones":0,"num_object_copies":1830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":915,"num_whiteouts":0,"num_read":2811,"num_read_kb":100236,"num_write":31405,"num_write_kb":1046937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1521901568,"data_stored":1518401554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33391,"ondisk_log_size":33391,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":303239465,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1432,"num_read_kb":712811,"num_write":10434,"num_write_kb":304769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":78,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":606633984,"data_stored":606478930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11693,"ondisk_log_size":11693,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542422,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":389288,"kb_used_data":322016,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93982552,"statfs":{"total":96636764160,"available":96238133248,"internally_reserved":0,"allocated":329744384,"data_stored":328817986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":68876747},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.78300000000000003,"15min":0.81799999999999995},"min":{"1min":0.52500000000000002,"5min":0.45900000000000002,"15min":0.40000000000000002},"max":{"1min":1.093,"5min":1.2050000000000001,"15min":1.595},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.78400000000000003,"15min":0.84199999999999997},"min":{"1min":0.49099999999999999,"5min":0.441,"15min":0.40999999999999998},"max":{"1min":1.113,"5min":1.302,"15min":1.4830000000000001},"last":0.57299999999999995}]},{"osd":1,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.85099999999999998,"15min":0.88800000000000001},"min":{"1min":0.57199999999999995,"5min":0.52700000000000002,"15min":0.441},"max":{"1min":1.2709999999999999,"5min":3.4329999999999998,"15min":3.4329999999999998},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.85399999999999998,"15min":0.88500000000000001},"min":{"1min":0.56999999999999995,"5min":0.505,"15min":0.505},"max":{"1min":1.1419999999999999,"5min":1.383,"15min":1.3919999999999999},"last":0.90100000000000002}]},{"osd":2,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.89000000000000001,"15min":0.92200000000000004},"min":{"1min":0.58899999999999997,"5min":0.55100000000000005,"15min":0.45900000000000002},"max":{"1min":1.163,"5min":3.4820000000000002,"15min":3.4820000000000002},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.877,"15min":0.83699999999999997},"min":{"1min":0.65000000000000002,"5min":0.52800000000000002,"15min":0.47099999999999997},"max":{"1min":1.2330000000000001,"5min":1.4570000000000001,"15min":1.6639999999999999},"last":0.73799999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":1.0069999999999999,"15min":0.96899999999999997},"min":{"1min":0.61299999999999999,"5min":0.60999999999999999,"15min":0.45500000000000002},"max":{"1min":1.1439999999999999,"5min":3.5190000000000001,"15min":3.5190000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.95599999999999996,"15min":0.99399999999999999},"min":{"1min":0.72899999999999998,"5min":0.54500000000000004,"15min":0.441},"max":{"1min":1.3220000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.94399999999999995}]},{"osd":4,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.97599999999999998,"15min":1.002},"min":{"1min":0.63800000000000001,"5min":0.42499999999999999,"15min":0.34399999999999997},"max":{"1min":1.2070000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":1.028},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.999,"15min":1.042},"min":{"1min":0.61799999999999999,"5min":0.45000000000000001,"15min":0.33800000000000002},"max":{"1min":1.1910000000000001,"5min":1.746,"15min":1.746},"last":1.0489999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.016,"15min":0.96099999999999997},"min":{"1min":0.76700000000000002,"5min":0.53700000000000003,"15min":0.43099999999999999},"max":{"1min":1.2949999999999999,"5min":1.6160000000000001,"15min":1.7549999999999999},"last":1.081},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.117,"15min":1.0680000000000001},"min":{"1min":0.71499999999999997,"5min":0.57899999999999996,"15min":0.35999999999999999},"max":{"1min":1.2370000000000001,"5min":2.1600000000000001,"15min":2.1600000000000001},"last":1.1990000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.141,"15min":1.123},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.39000000000000001},"max":{"1min":1.216,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.129},{"interface":"front","average":{"1min":1,"5min":1.1259999999999999,"15min":1.167},"min":{"1min":0.68200000000000005,"5min":0.56299999999999994,"15min":0.47199999999999998},"max":{"1min":1.3300000000000001,"5min":3.8140000000000001,"15min":3.8140000000000001},"last":0.998}]}]},{"osd":6,"up_from":14,"seq":60129542422,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":296948,"kb_used_data":233900,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94074892,"statfs":{"total":96636764160,"available":96332689408,"internally_reserved":0,"allocated":239513600,"data_stored":238814643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":64551368},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.83299999999999996,"15min":0.85999999999999999},"min":{"1min":0.52900000000000003,"5min":0.46600000000000003,"15min":0.35599999999999998},"max":{"1min":1.155,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.77000000000000002,"15min":0.78800000000000003},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.42999999999999999},"max":{"1min":1.05,"5min":1.109,"15min":1.214},"last":0.92400000000000004}]},{"osd":1,"last update":"Mon Oct 30 12:50:23 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.79900000000000004,"15min":0.81299999999999994},"min":{"1min":0.57299999999999995,"5min":0.42299999999999999,"15min":0.317},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.3140000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.89000000000000001,"15min":0.88800000000000001},"min":{"1min":0.55800000000000005,"5min":0.51100000000000001,"15min":0.40999999999999998},"max":{"1min":1.149,"5min":1.3620000000000001,"15min":1.423},"last":0.78900000000000003}]},{"osd":2,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.93600000000000005,"15min":0.97499999999999998},"min":{"1min":0.495,"5min":0.495,"15min":0.38800000000000001},"max":{"1min":1.373,"5min":1.47,"15min":4.3360000000000003},"last":1.04},{"interface":"front","average":{"1min":1.0409999999999999,"5min":0.97999999999999998,"15min":1.0089999999999999},"min":{"1min":0.65000000000000002,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.502,"5min":1.502,"15min":1.5629999999999999},"last":1.2390000000000001}]},{"osd":3,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.996,"15min":1.042},"min":{"1min":0.64700000000000002,"5min":0.59099999999999997,"15min":0.58999999999999997},"max":{"1min":1.3340000000000001,"5min":1.5509999999999999,"15min":1.6459999999999999},"last":0.83999999999999997},{"interface":"front","average":{"1min":1.02,"5min":1.0680000000000001,"15min":1.101},"min":{"1min":0.72299999999999998,"5min":0.63300000000000001,"15min":0.59699999999999998},"max":{"1min":1.417,"5min":1.552,"15min":1.7969999999999999},"last":1.095}]},{"osd":4,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.149,"15min":1.075},"min":{"1min":0.72599999999999998,"5min":0.69899999999999995,"15min":0.51500000000000001},"max":{"1min":1.585,"5min":1.6499999999999999,"15min":1.7649999999999999},"last":1.5229999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.89400000000000002,"15min":0.96499999999999997},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.42099999999999999},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.881},"last":1.2030000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":1.0720000000000001,"15min":1.087},"min":{"1min":0.54100000000000004,"5min":0.53200000000000003,"15min":0.48399999999999999},"max":{"1min":1.2849999999999999,"5min":1.75,"15min":2.2770000000000001},"last":1.27},{"interface":"front","average":{"1min":1.056,"5min":1.159,"15min":1.0649999999999999},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.46999999999999997},"max":{"1min":1.4610000000000001,"5min":1.6950000000000001,"15min":4.7850000000000001},"last":1.1359999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.036,"15min":1.0029999999999999},"min":{"1min":0.68799999999999994,"5min":0.64100000000000001,"15min":0.46700000000000003},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.825},"last":1.417},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.0629999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.48999999999999999},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.895},"last":1.3169999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542422,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":338824,"kb_used_data":271552,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94033016,"statfs":{"total":96636764160,"available":96289808384,"internally_reserved":0,"allocated":278069248,"data_stored":277156365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.78400000000000003,"15min":0.80300000000000005},"min":{"1min":0.56299999999999994,"5min":0.33900000000000002,"15min":0.29399999999999998},"max":{"1min":1.1970000000000001,"5min":1.238,"15min":2.4729999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.79700000000000004,"15min":0.80400000000000005},"min":{"1min":0.61199999999999999,"5min":0.39200000000000002,"15min":0.33800000000000002},"max":{"1min":1.2390000000000001,"5min":1.3009999999999999,"15min":1.8759999999999999},"last":0.84299999999999997}]},{"osd":1,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.84199999999999997,"15min":0.84399999999999997},"min":{"1min":0.57399999999999995,"5min":0.40999999999999998,"15min":0.38900000000000001},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":3.0190000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.58499999999999996,"5min":0.44800000000000001,"15min":0.373},"max":{"1min":1.264,"5min":1.276,"15min":1.4870000000000001},"last":0.91800000000000004}]},{"osd":2,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.88400000000000001,"15min":0.84699999999999998},"min":{"1min":0.67300000000000004,"5min":0.55400000000000005,"15min":0.38400000000000001},"max":{"1min":1.157,"5min":1.264,"15min":2.2949999999999999},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.81699999999999995,"15min":0.876},"min":{"1min":0.63600000000000001,"5min":0.434,"15min":0.434},"max":{"1min":1.1140000000000001,"5min":1.171,"15min":1.5800000000000001},"last":0.88600000000000001}]},{"osd":3,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.91800000000000004,"15min":0.93700000000000006},"min":{"1min":0.57199999999999995,"5min":0.53800000000000003,"15min":0.47899999999999998},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.756},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.93600000000000005,"15min":0.94599999999999995},"min":{"1min":0.755,"5min":0.498,"15min":0.41399999999999998},"max":{"1min":1.204,"5min":1.4119999999999999,"15min":2.3679999999999999},"last":1.0680000000000001}]},{"osd":4,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.84499999999999997,"15min":0.90900000000000003},"min":{"1min":0.47499999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.304,"5min":1.304,"15min":3.1190000000000002},"last":0.81100000000000005},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.083,"15min":1.091},"min":{"1min":0.59499999999999997,"5min":0.58699999999999997,"15min":0.50900000000000001},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":2.645},"last":1.026}]},{"osd":6,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.038,"15min":1.042},"min":{"1min":0.51100000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.466,"5min":1.466,"15min":3.1859999999999999},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.056,"15min":1.1060000000000001},"min":{"1min":0.67000000000000004,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.3260000000000001,"5min":1.47,"15min":3.371},"last":1}]},{"osd":7,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.0609999999999999,"15min":1.0800000000000001},"min":{"1min":0.65200000000000002,"5min":0.51400000000000001,"15min":0.47299999999999998},"max":{"1min":1.494,"5min":1.508,"15min":3.5310000000000001},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.282,"5min":1.1970000000000001,"15min":1.2310000000000001},"min":{"1min":0.96799999999999997,"5min":0.64700000000000002,"15min":0.61299999999999999},"max":{"1min":1.605,"5min":1.605,"15min":3.6930000000000001},"last":1.3160000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542421,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369848,"kb_used_data":294256,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94001992,"statfs":{"total":96636764160,"available":96258039808,"internally_reserved":0,"allocated":301318144,"data_stored":300442399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:15 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.81799999999999995,"15min":0.92600000000000005},"min":{"1min":0.48099999999999998,"5min":0.34799999999999998,"15min":0.31},"max":{"1min":1.204,"5min":1.2509999999999999,"15min":11.273},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.92400000000000004,"15min":0.98799999999999999},"min":{"1min":0.58099999999999996,"5min":0.44,"15min":0.44},"max":{"1min":1.411,"5min":1.4590000000000001,"15min":7.1289999999999996},"last":0.81899999999999995}]},{"osd":1,"last update":"Mon Oct 30 12:51:15 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.81499999999999995,"15min":0.90500000000000003},"min":{"1min":0.52800000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0049999999999999,"5min":5.1130000000000004,"15min":6.9199999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.871,"5min":0.84199999999999997,"15min":0.90000000000000002},"min":{"1min":0.63900000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.246,"5min":1.246,"15min":4.75},"last":0.76900000000000002}]},{"osd":2,"last update":"Mon Oct 30 12:51:15 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.84199999999999997,"15min":0.84899999999999998},"min":{"1min":0.51700000000000002,"5min":0.41899999999999998,"15min":0.36899999999999999},"max":{"1min":1.175,"5min":1.21,"15min":2.6970000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.996,"5min":0.91200000000000003,"15min":1.0569999999999999},"min":{"1min":0.60999999999999999,"5min":0.54400000000000004,"15min":0.52900000000000003},"max":{"1min":1.361,"5min":1.361,"15min":18.917999999999999},"last":0.999}]},{"osd":3,"last update":"Mon Oct 30 12:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":0.96599999999999997,"15min":0.997},"min":{"1min":0.71399999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.258,"5min":5.1719999999999997,"15min":7.1879999999999997},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.90800000000000003,"15min":1},"min":{"1min":0.57399999999999995,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.125,"5min":1.3600000000000001,"15min":8.3360000000000003},"last":0.71899999999999997}]},{"osd":5,"last update":"Mon Oct 30 12:51:15 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.88300000000000001,"15min":1.002},"min":{"1min":0.56299999999999994,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.159,"5min":1.3300000000000001,"15min":6.0759999999999996},"last":0.92500000000000004},{"interface":"front","average":{"1min":1.119,"5min":1.105,"15min":1.1890000000000001},"min":{"1min":0.89500000000000002,"5min":0.71899999999999997,"15min":0.52600000000000002},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":18.994},"last":1.0329999999999999}]},{"osd":6,"last update":"Mon Oct 30 12:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.0880000000000001,"15min":1.093},"min":{"1min":0.70199999999999996,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.429,"5min":1.429,"15min":6.1580000000000004},"last":1.115},{"interface":"front","average":{"1min":1.016,"5min":1.0169999999999999,"15min":1.1240000000000001},"min":{"1min":0.629,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.28,"5min":5.218,"15min":7.0549999999999997},"last":0.95999999999999996}]},{"osd":7,"last update":"Mon Oct 30 12:51:15 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.0980000000000001,"15min":1.258},"min":{"1min":0.75800000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.4339999999999999,"5min":5.7160000000000002,"15min":18.015999999999998},"last":1.091},{"interface":"front","average":{"1min":1.236,"5min":1.2050000000000001,"15min":1.2769999999999999},"min":{"1min":1.093,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.4530000000000001,"5min":5.6609999999999996,"15min":19.052},"last":1.226}]}]},{"osd":3,"up_from":14,"seq":60129542422,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388864,"kb_used_data":309112,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93982976,"statfs":{"total":96636764160,"available":96238567424,"internally_reserved":0,"allocated":316530688,"data_stored":315623973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":81656264},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.39900000000000002,"15min":0.41399999999999998},"min":{"1min":0.29399999999999998,"5min":0.23699999999999999,"15min":0.23400000000000001},"max":{"1min":0.58999999999999997,"5min":0.77900000000000003,"15min":1.6899999999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51600000000000001,"15min":0.53300000000000003},"min":{"1min":0.40000000000000002,"5min":0.34300000000000003,"15min":0.32700000000000001},"max":{"1min":0.746,"5min":0.871,"15min":1.728},"last":0.55200000000000005}]},{"osd":1,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.42899999999999999,"15min":0.42999999999999999},"min":{"1min":0.309,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":1.2909999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.496,"15min":0.50600000000000001},"min":{"1min":0.34499999999999997,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.64600000000000002,"5min":0.80600000000000005,"15min":1.3260000000000001},"last":0.501}]},{"osd":2,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.496,"15min":0.502},"min":{"1min":0.37,"5min":0.30099999999999999,"15min":0.29299999999999998},"max":{"1min":0.69899999999999995,"5min":0.78600000000000003,"15min":1.099},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55900000000000005,"15min":0.56699999999999995},"min":{"1min":0.41999999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73599999999999999,"5min":0.82599999999999996,"15min":1.2509999999999999},"last":0.53100000000000003}]},{"osd":4,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.85199999999999998,"15min":0.83099999999999996},"min":{"1min":0.63900000000000001,"5min":0.55400000000000005,"15min":0.40400000000000003},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":2.202},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.82999999999999996,"15min":0.86499999999999999},"min":{"1min":0.70599999999999996,"5min":0.48499999999999999,"15min":0.46800000000000003},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":2.4340000000000002},"last":0.89100000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.81499999999999995,"15min":0.86299999999999999},"min":{"1min":0.66200000000000003,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.169,"5min":1.169,"15min":14.103999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.875,"5min":0.876,"15min":0.86499999999999999},"min":{"1min":0.63900000000000001,"5min":0.55000000000000004,"15min":0.499},"max":{"1min":1.2170000000000001,"5min":1.2869999999999999,"15min":4.4859999999999998},"last":0.85699999999999998}]},{"osd":6,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.85299999999999998,"15min":0.83499999999999996},"min":{"1min":0.64700000000000002,"5min":0.56999999999999995,"15min":0.53100000000000003},"max":{"1min":1.087,"5min":1.2450000000000001,"15min":2.2330000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.87,"15min":0.82899999999999996},"min":{"1min":0.77800000000000002,"5min":0.54100000000000004,"15min":0.45100000000000001},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":4.4409999999999998},"last":0.86699999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.875,"15min":0.86499999999999999},"min":{"1min":0.67500000000000004,"5min":0.57299999999999995,"15min":0.57099999999999995},"max":{"1min":1.1499999999999999,"5min":1.2509999999999999,"15min":3.5459999999999998},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.86399999999999999,"15min":0.84699999999999998},"min":{"1min":0.65700000000000003,"5min":0.57399999999999995,"15min":0.45700000000000002},"max":{"1min":1.1259999999999999,"5min":1.2230000000000001,"15min":6.1360000000000001},"last":0.91800000000000004}]}]},{"osd":2,"up_from":14,"seq":60129542422,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":199872,"kb_used_data":142328,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94171968,"statfs":{"total":96636764160,"available":96432095232,"internally_reserved":0,"allocated":145743872,"data_stored":145027826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.45100000000000001,"15min":0.441},"min":{"1min":0.25,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.59499999999999997,"5min":0.69399999999999995,"15min":0.81200000000000006},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.5,"15min":0.51100000000000001},"min":{"1min":0.37,"5min":0.33300000000000002,"15min":0.31900000000000001},"max":{"1min":0.63,"5min":0.71299999999999997,"15min":1.298},"last":0.47099999999999997}]},{"osd":1,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.47899999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.60199999999999998,"5min":0.65500000000000003,"15min":0.83699999999999997},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.52500000000000002,"15min":0.504},"min":{"1min":0.38900000000000001,"5min":0.33700000000000002,"15min":0.29999999999999999},"max":{"1min":0.61699999999999999,"5min":0.74099999999999999,"15min":1.2250000000000001},"last":0.53800000000000003}]},{"osd":3,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.47399999999999998,"15min":0.54800000000000004},"min":{"1min":0.315,"5min":0.311,"15min":0.311},"max":{"1min":0.58599999999999997,"5min":0.75,"15min":1.673},"last":0.498},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57299999999999995,"15min":0.57399999999999995},"min":{"1min":0.41799999999999998,"5min":0.40200000000000002,"15min":0.35699999999999998},"max":{"1min":0.67900000000000005,"5min":0.79100000000000004,"15min":1.597},"last":0.59499999999999997}]},{"osd":4,"last update":"Mon Oct 30 12:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.74299999999999999,"15min":0.82599999999999996},"min":{"1min":0.47499999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.91100000000000003,"5min":1.0569999999999999,"15min":1.377},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.755,"15min":0.79200000000000004},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.998,"5min":1.0640000000000001,"15min":1.4159999999999999},"last":1.0169999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.84099999999999997,"15min":0.84799999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.093,"5min":1.2430000000000001,"15min":1.78},"last":1.002},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91400000000000003},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.1060000000000001,"5min":1.2729999999999999,"15min":1.7430000000000001},"last":1.0309999999999999}]},{"osd":6,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.86199999999999999,"15min":0.85799999999999998},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.60299999999999998},"max":{"1min":1.095,"5min":1.177,"15min":1.3799999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.86599999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.53300000000000003},"max":{"1min":1.097,"5min":1.1799999999999999,"15min":1.8109999999999999},"last":0.93300000000000005}]},{"osd":7,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.82899999999999996,"15min":0.83099999999999996},"min":{"1min":0.58799999999999997,"5min":0.58299999999999996,"15min":0.48799999999999999},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.4159999999999999},"last":1.0429999999999999},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.85399999999999998,"15min":0.83299999999999996},"min":{"1min":0.75900000000000001,"5min":0.51800000000000002,"15min":0.50800000000000001},"max":{"1min":1.206,"5min":1.206,"15min":1.3049999999999999},"last":1.0780000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542422,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":278984,"kb_used_data":233920,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94092856,"statfs":{"total":96636764160,"available":96351084544,"internally_reserved":0,"allocated":239534080,"data_stored":238814180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:50:34 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.40400000000000003,"15min":0.45400000000000001},"min":{"1min":0.28399999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.63600000000000001,"5min":0.65200000000000002,"15min":11.958},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.495,"15min":0.55700000000000005},"min":{"1min":0.34699999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.73599999999999999,"5min":0.73799999999999999,"15min":12.077},"last":0.53800000000000003}]},{"osd":2,"last update":"Mon Oct 30 12:50:34 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45800000000000002,"15min":0.46000000000000002},"min":{"1min":0.29699999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.121,"5min":1.121,"15min":1.623},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52500000000000002,"15min":0.53300000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.31900000000000001},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.5389999999999999},"last":0.60599999999999998}]},{"osd":3,"last update":"Mon Oct 30 12:50:34 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55400000000000005,"15min":0.56699999999999995},"min":{"1min":0.378,"5min":0.31,"15min":0.31},"max":{"1min":1.216,"5min":1.337,"15min":3.694},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.498,"15min":0.54700000000000004},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.66900000000000004,"5min":0.77200000000000002,"15min":13.199999999999999},"last":0.49299999999999999}]},{"osd":4,"last update":"Mon Oct 30 12:50:34 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.82699999999999996,"15min":0.83199999999999996},"min":{"1min":0.54700000000000004,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1519999999999999,"5min":1.3140000000000001,"15min":2.419},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.871,"15min":0.86699999999999999},"min":{"1min":0.66300000000000003,"5min":0.60899999999999999,"15min":0.52100000000000002},"max":{"1min":1.095,"5min":1.103,"15min":2.282},"last":0.91700000000000004}]},{"osd":5,"last update":"Mon Oct 30 12:50:34 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.80000000000000004,"15min":0.81599999999999995},"min":{"1min":0.61699999999999999,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.01,"5min":1.343,"15min":1.343},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.79000000000000004,"15min":0.85599999999999998},"min":{"1min":0.59099999999999997,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.115,"5min":1.29,"15min":1.29},"last":0.77000000000000002}]},{"osd":6,"last update":"Mon Oct 30 12:50:34 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82799999999999996,"15min":0.79800000000000004},"min":{"1min":0.56399999999999995,"5min":0.52800000000000002,"15min":0.51200000000000001},"max":{"1min":1.3660000000000001,"5min":1.371,"15min":1.371},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.89400000000000002,"15min":0.86699999999999999},"min":{"1min":0.67400000000000004,"5min":0.498,"15min":0.498},"max":{"1min":1.417,"5min":1.417,"15min":2.464},"last":0.97099999999999997}]},{"osd":7,"last update":"Mon Oct 30 12:50:34 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.81100000000000005,"15min":0.83199999999999996},"min":{"1min":0.61899999999999999,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0349999999999999,"5min":1.1659999999999999,"15min":2.5169999999999999},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.875,"5min":0.88200000000000001,"15min":0.90700000000000003},"min":{"1min":0.76900000000000002,"5min":0.61899999999999999,"15min":0.58399999999999996},"max":{"1min":1.133,"5min":1.133,"15min":2.5510000000000002},"last":0.78700000000000003}]}]},{"osd":0,"up_from":14,"seq":60129542422,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350128,"kb_used_data":278696,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94021712,"statfs":{"total":96636764160,"available":96278233088,"internally_reserved":0,"allocated":285384704,"data_stored":284463559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 12:50:26 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.44400000000000001,"15min":0.44900000000000001},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.93700000000000006},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50900000000000001,"15min":0.504},"min":{"1min":0.35299999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.67600000000000005,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.56100000000000005}]},{"osd":2,"last update":"Mon Oct 30 12:50:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.45600000000000002,"15min":0.504},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.629,"5min":0.64900000000000002,"15min":0.97199999999999998},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.55200000000000005,"15min":0.52800000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.27400000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":1.0149999999999999},"last":0.57699999999999996}]},{"osd":3,"last update":"Mon Oct 30 12:50:26 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.46899999999999997,"15min":0.51400000000000001},"min":{"1min":0.30499999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.73499999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.48599999999999999,"15min":0.496},"min":{"1min":0.316,"5min":0.31,"15min":0.31},"max":{"1min":0.74199999999999999,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.59299999999999997}]},{"osd":4,"last update":"Mon Oct 30 12:50:26 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86599999999999999,"15min":0.85499999999999998},"min":{"1min":0.70899999999999996,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.127,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.88500000000000001,"15min":0.85699999999999998},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.48299999999999998},"max":{"1min":1.143,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.92400000000000004}]},{"osd":5,"last update":"Mon Oct 30 12:50:26 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.82299999999999995,"15min":0.84199999999999997},"min":{"1min":0.60499999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.095,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.83299999999999996,"15min":0.87},"min":{"1min":0.58699999999999997,"5min":0.53200000000000003,"15min":0.52800000000000002},"max":{"1min":1.889,"5min":1.889,"15min":1.889},"last":0.89000000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:50:26 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.78400000000000003,"15min":0.79500000000000004},"min":{"1min":0.58299999999999996,"5min":0.56299999999999994,"15min":0.50900000000000001},"max":{"1min":1.0409999999999999,"5min":1.1990000000000001,"15min":2.6480000000000001},"last":1.071},{"interface":"front","average":{"1min":0.871,"5min":0.79300000000000004,"15min":0.82099999999999995},"min":{"1min":0.61199999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0409999999999999,"5min":1.1559999999999999,"15min":5.0099999999999998},"last":1.0249999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:50:26 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.87,"15min":0.88200000000000001},"min":{"1min":0.68899999999999995,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.155,"5min":1.401,"15min":3.3359999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.92400000000000004,"15min":0.94399999999999995},"min":{"1min":0.75700000000000001,"5min":0.59699999999999998,"15min":0.56299999999999994},"max":{"1min":1.1890000000000001,"5min":1.3500000000000001,"15min":3.3580000000000001},"last":1.0069999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62930944,"data_stored":62915220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":83910656,"data_stored":83887024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":37122048,"data_stored":37098251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82608128,"data_stored":82586990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":76099584,"data_stored":76085923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":62935040,"data_stored":62915938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":89997312,"data_stored":89978920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":111030272,"data_stored":111010664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":221097984,"data_stored":220570894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":154857472,"data_stored":154539460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":107859968,"data_stored":107542498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":233156608,"data_stored":232649287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":224452608,"data_stored":223968780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":214372352,"data_stored":213853350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":148750336,"data_stored":148448027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":217354240,"data_stored":216829258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:51:31.201 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:51:31.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.374+0000 7fc5bc394640 1 Processor -- start 2023-10-30T12:51:31.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.374+0000 7fc5bc394640 1 -- start start 2023-10-30T12:51:31.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.374+0000 7fc5bc394640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc5b406f7a0 0x7fc5b406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:31.376 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.374+0000 7fc5bc394640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc5b40c6bd0 0x7fc5b4069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:31.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.374+0000 7fc5bc394640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b406a3d0 con 0x7fc5b4071110 2023-10-30T12:51:31.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.374+0000 7fc5bc394640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5b40700c0 con 0x7fc5b40c6bd0 2023-10-30T12:51:31.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.374+0000 7fc5bc394640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5b4066cf0 con 0x7fc5b406f7a0 2023-10-30T12:51:31.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.378+0000 7fc5b992f640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc5b40c6bd0 0x7fc5b4069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:31.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.378+0000 7fc5b992f640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc5b40c6bd0 0x7fc5b4069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47290/0 (socket says 172.21.15.89:47290) 2023-10-30T12:51:31.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.378+0000 7fc5ba931640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc5b4071110 0x7fc5b4158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36830/0 (socket says 172.21.15.89:36830) 2023-10-30T12:51:31.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.378+0000 7fc5b992f640 1 -- 172.21.15.89:0/1645659874 learned_addr learned my addr 172.21.15.89:0/1645659874 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:31.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.378+0000 7fc5ba130640 1 --2- 172.21.15.89:0/1645659874 >> v2:172.21.15.192:3300/0 conn(0x7fc5b406f7a0 0x7fc5b406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:31.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.378+0000 7fc5b992f640 1 -- 172.21.15.89:0/1645659874 >> v2:172.21.15.192:3300/0 conn(0x7fc5b406f7a0 msgr2=0x7fc5b406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:31.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.378+0000 7fc5b992f640 1 --2- 172.21.15.89:0/1645659874 >> v2:172.21.15.192:3300/0 conn(0x7fc5b406f7a0 0x7fc5b406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.378+0000 7fc5b992f640 1 -- 172.21.15.89:0/1645659874 >> v1:172.21.15.89:6789/0 conn(0x7fc5b4071110 legacy=0x7fc5b4158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:31.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.378+0000 7fc5b992f640 1 -- 172.21.15.89:0/1645659874 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b40c70d0 con 0x7fc5b40c6bd0 2023-10-30T12:51:31.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.378+0000 7fc5b912e640 1 -- 172.21.15.89:0/1645659874 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517547322 0 0) 0x7fc5b406a3d0 con 0x7fc5b4071110 2023-10-30T12:51:31.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.378+0000 7fc5b992f640 1 --2- 172.21.15.89:0/1645659874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5b40c6bd0 0x7fc5b4069e90 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fc5b406c830 tx=0x7fc5a401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38cd6adb890ff383 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:31.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.378+0000 7fc5b912e640 1 -- 172.21.15.89:0/1645659874 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a40a89f0 con 0x7fc5b40c6bd0 2023-10-30T12:51:31.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.378+0000 7fc5b912e640 1 -- 172.21.15.89:0/1645659874 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5a40a8b50 con 0x7fc5b40c6bd0 2023-10-30T12:51:31.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.378+0000 7fc5b912e640 1 -- 172.21.15.89:0/1645659874 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a40b0440 con 0x7fc5b40c6bd0 2023-10-30T12:51:31.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.382+0000 7fc5bc394640 1 -- 172.21.15.89:0/1645659874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5b40c6bd0 msgr2=0x7fc5b4069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:31.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.382+0000 7fc5bc394640 1 --2- 172.21.15.89:0/1645659874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5b40c6bd0 0x7fc5b4069e90 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fc5b406c830 tx=0x7fc5a401ecf0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.382+0000 7fc5bc394640 1 -- 172.21.15.89:0/1645659874 shutdown_connections 2023-10-30T12:51:31.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.382+0000 7fc5bc394640 1 --2- 172.21.15.89:0/1645659874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5b40c6bd0 0x7fc5b4069e90 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.382+0000 7fc5bc394640 1 --2- 172.21.15.89:0/1645659874 >> v2:172.21.15.192:3300/0 conn(0x7fc5b406f7a0 0x7fc5b406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.382+0000 7fc5bc394640 1 -- 172.21.15.89:0/1645659874 >> 172.21.15.89:0/1645659874 conn(0x7fc5b40c7910 msgr2=0x7fc5b40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:31.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.382+0000 7fc5bc394640 1 -- 172.21.15.89:0/1645659874 shutdown_connections 2023-10-30T12:51:31.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.382+0000 7fc5bc394640 1 -- 172.21.15.89:0/1645659874 wait complete. 2023-10-30T12:51:31.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.382+0000 7fc5bc394640 1 Processor -- start 2023-10-30T12:51:31.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.382+0000 7fc5bc394640 1 -- start start 2023-10-30T12:51:31.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.382+0000 7fc5bc394640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5b406f7a0 0x7fc5b4154820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:31.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.382+0000 7fc5bc394640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5b4071110 0x7fc5b4152390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:31.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.382+0000 7fc5bc394640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5b40c6bd0 0x7fc5b41528d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:31.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.382+0000 7fc5bc394640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b40afeb0 con 0x7fc5b406f7a0 2023-10-30T12:51:31.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.382+0000 7fc5bc394640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b40c65e0 con 0x7fc5b40c6bd0 2023-10-30T12:51:31.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.382+0000 7fc5bc394640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b40c6300 con 0x7fc5b4071110 2023-10-30T12:51:31.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.382+0000 7fc5ba130640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5b406f7a0 0x7fc5b4154820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:31.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.382+0000 7fc5ba130640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5b406f7a0 0x7fc5b4154820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47294/0 (socket says 172.21.15.89:47294) 2023-10-30T12:51:31.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.382+0000 7fc5ba130640 1 -- 172.21.15.89:0/1206032108 learned_addr learned my addr 172.21.15.89:0/1206032108 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:31.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.382+0000 7fc5b992f640 1 --2- 172.21.15.89:0/1206032108 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5b4071110 0x7fc5b4152390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:31.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.382+0000 7fc5ba931640 1 --2- 172.21.15.89:0/1206032108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5b40c6bd0 0x7fc5b41528d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:31.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.382+0000 7fc5ba130640 1 -- 172.21.15.89:0/1206032108 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5b4071110 msgr2=0x7fc5b4152390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:31.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.382+0000 7fc5ba130640 1 --2- 172.21.15.89:0/1206032108 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5b4071110 0x7fc5b4152390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.382+0000 7fc5ba130640 1 -- 172.21.15.89:0/1206032108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5b40c6bd0 msgr2=0x7fc5b41528d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:31.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.382+0000 7fc5ba130640 1 --2- 172.21.15.89:0/1206032108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5b40c6bd0 0x7fc5b41528d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.382+0000 7fc5ba130640 1 -- 172.21.15.89:0/1206032108 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a40094d0 con 0x7fc5b406f7a0 2023-10-30T12:51:31.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.386+0000 7fc5ba130640 1 --2- 172.21.15.89:0/1206032108 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5b406f7a0 0x7fc5b4154820 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fc5b000d5f0 tx=0x7fc5b000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:31.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.386+0000 7fc5aaffd640 1 -- 172.21.15.89:0/1206032108 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b00125f0 con 0x7fc5b406f7a0 2023-10-30T12:51:31.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.386+0000 7fc5aaffd640 1 -- 172.21.15.89:0/1206032108 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b0012750 con 0x7fc5b406f7a0 2023-10-30T12:51:31.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.386+0000 7fc5aaffd640 1 -- 172.21.15.89:0/1206032108 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0012b00 con 0x7fc5b406f7a0 2023-10-30T12:51:31.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.386+0000 7fc5bc394640 1 -- 172.21.15.89:0/1206032108 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b40afbd0 con 0x7fc5b406f7a0 2023-10-30T12:51:31.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.386+0000 7fc5bc394640 1 -- 172.21.15.89:0/1206032108 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b4157af0 con 0x7fc5b406f7a0 2023-10-30T12:51:31.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.386+0000 7fc5bc394640 1 -- 172.21.15.89:0/1206032108 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b404e170 con 0x7fc5b406f7a0 2023-10-30T12:51:31.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.386+0000 7fc5aaffd640 1 -- 172.21.15.89:0/1206032108 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc5b0012c60 con 0x7fc5b406f7a0 2023-10-30T12:51:31.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.386+0000 7fc5aaffd640 1 --2- 172.21.15.89:0/1206032108 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc5980664a0 0x7fc598068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:31.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.386+0000 7fc5aaffd640 1 -- 172.21.15.89:0/1206032108 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7fc5b0084770 con 0x7fc5b406f7a0 2023-10-30T12:51:31.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.386+0000 7fc5b992f640 1 --2- 172.21.15.89:0/1206032108 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc5980664a0 0x7fc598068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:31.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.390+0000 7fc5b992f640 1 --2- 172.21.15.89:0/1206032108 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc5980664a0 0x7fc598068960 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc5a40b2000 tx=0x7fc5a40aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:31.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.390+0000 7fc5aaffd640 1 -- 172.21.15.89:0/1206032108 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc5b0051870 con 0x7fc5b406f7a0 2023-10-30T12:51:31.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.542+0000 7fc5bc394640 1 -- 172.21.15.89:0/1206032108 --> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc5b4070350 con 0x7fc5980664a0 2023-10-30T12:51:31.565 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.566+0000 7fc5aaffd640 1 -- 172.21.15.89:0/1206032108 <== mgr.4105 v2:172.21.15.192:6832/4076276638 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+292221 (secure 0 0 0) 0x7fc5a40b0440 con 0x7fc5980664a0 2023-10-30T12:51:31.568 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:51:31.569 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-10-30T12:51:31.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.570+0000 7fc5bc394640 1 -- 172.21.15.89:0/1206032108 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc5980664a0 msgr2=0x7fc598068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:31.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.570+0000 7fc5bc394640 1 --2- 172.21.15.89:0/1206032108 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc5980664a0 0x7fc598068960 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc5a40b2000 tx=0x7fc5a40aa070 comp rx=0 tx=0).stop 2023-10-30T12:51:31.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.570+0000 7fc5bc394640 1 -- 172.21.15.89:0/1206032108 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5b406f7a0 msgr2=0x7fc5b4154820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:31.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.570+0000 7fc5bc394640 1 --2- 172.21.15.89:0/1206032108 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5b406f7a0 0x7fc5b4154820 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fc5b000d5f0 tx=0x7fc5b000dab0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.570+0000 7fc5bc394640 1 -- 172.21.15.89:0/1206032108 shutdown_connections 2023-10-30T12:51:31.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.570+0000 7fc5bc394640 1 --2- 172.21.15.89:0/1206032108 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc5980664a0 0x7fc598068960 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.570+0000 7fc5bc394640 1 --2- 172.21.15.89:0/1206032108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5b40c6bd0 0x7fc5b41528d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.570+0000 7fc5bc394640 1 --2- 172.21.15.89:0/1206032108 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc5b4071110 0x7fc5b4152390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.570+0000 7fc5bc394640 1 --2- 172.21.15.89:0/1206032108 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5b406f7a0 0x7fc5b4154820 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.573 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.570+0000 7fc5bc394640 1 -- 172.21.15.89:0/1206032108 >> 172.21.15.89:0/1206032108 conn(0x7fc5b40c7910 msgr2=0x7fc5b40b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:31.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.570+0000 7fc5bc394640 1 -- 172.21.15.89:0/1206032108 shutdown_connections 2023-10-30T12:51:31.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.570+0000 7fc5bc394640 1 -- 172.21.15.89:0/1206032108 wait complete. 2023-10-30T12:51:31.588 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":746,"stamp":"2023-10-30T12:51:31.428790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1063030610,"num_objects":1094,"num_object_clones":0,"num_object_copies":2188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1094,"num_whiteouts":0,"num_read":4359,"num_read_kb":813147,"num_write":41998,"num_write_kb":1354122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":78,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45201,"ondisk_log_size":45201,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2612756,"kb_used_data":2085780,"kb_used_omap":12,"kb_used_meta":526899,"kb_avail":752361964,"statfs":{"total":773094113280,"available":770418651136,"internally_reserved":0,"allocated":2135838720,"data_stored":2129160931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12741,"internal_metadata":539545147},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2682127,"num_objects":-61,"num_object_clones":0,"num_object_copies":-122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-61,"num_whiteouts":0,"num_read":8,"num_read_kb":19,"num_write":251,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004335"},"pg_stats":[{"pgid":"3.3f","version":"65'525","reported_seq":2677,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.734150+0000","last_change":"2023-10-30T12:38:17.335461+0000","last_active":"2023-10-30T12:51:12.734150+0000","last_peered":"2023-10-30T12:51:12.734150+0000","last_clean":"2023-10-30T12:51:12.734150+0000","last_became_active":"2023-10-30T12:28:50.979351+0000","last_became_peered":"2023-10-30T12:28:50.979351+0000","last_unstale":"2023-10-30T12:51:12.734150+0000","last_undegraded":"2023-10-30T12:51:12.734150+0000","last_fullsized":"2023-10-30T12:51:12.734150+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":525,"log_dups_size":0,"ondisk_log_size":525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:04:55.768566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091708599999999994,"stat_sum":{"num_bytes":4266500,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":40,"num_read_kb":15,"num_write":484,"num_write_kb":5355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"66'518","reported_seq":2663,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.730499+0000","last_change":"2023-10-30T12:38:17.104663+0000","last_active":"2023-10-30T12:51:12.730499+0000","last_peered":"2023-10-30T12:51:12.730499+0000","last_clean":"2023-10-30T12:51:12.730499+0000","last_became_active":"2023-10-30T12:28:50.929133+0000","last_became_peered":"2023-10-30T12:28:50.929133+0000","last_unstale":"2023-10-30T12:51:12.730499+0000","last_undegraded":"2023-10-30T12:51:12.730499+0000","last_fullsized":"2023-10-30T12:51:12.730499+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":518,"log_dups_size":0,"ondisk_log_size":518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:05:29.266806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023149070000000002,"stat_sum":{"num_bytes":8405214,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":33,"num_read_kb":10,"num_write":478,"num_write_kb":9481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"66'525","reported_seq":2686,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.742402+0000","last_change":"2023-10-30T12:38:17.643537+0000","last_active":"2023-10-30T12:51:12.742402+0000","last_peered":"2023-10-30T12:51:12.742402+0000","last_clean":"2023-10-30T12:51:12.742402+0000","last_became_active":"2023-10-30T12:28:50.933896+0000","last_became_peered":"2023-10-30T12:28:50.933896+0000","last_unstale":"2023-10-30T12:51:12.742402+0000","last_undegraded":"2023-10-30T12:51:12.742402+0000","last_fullsized":"2023-10-30T12:51:12.742402+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":525,"log_dups_size":0,"ondisk_log_size":525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:45:03.377748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029033520000000001,"stat_sum":{"num_bytes":12676713,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":49,"num_read_kb":8209,"num_write":482,"num_write_kb":25758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"66'502","reported_seq":2645,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.744598+0000","last_change":"2023-10-30T12:38:17.117873+0000","last_active":"2023-10-30T12:51:12.744598+0000","last_peered":"2023-10-30T12:51:12.744598+0000","last_clean":"2023-10-30T12:51:12.744598+0000","last_became_active":"2023-10-30T12:28:50.967774+0000","last_became_peered":"2023-10-30T12:28:50.967774+0000","last_unstale":"2023-10-30T12:51:12.744598+0000","last_undegraded":"2023-10-30T12:51:12.744598+0000","last_fullsized":"2023-10-30T12:51:12.744598+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":502,"log_dups_size":0,"ondisk_log_size":502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:56:36.456625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075479680000000004,"stat_sum":{"num_bytes":4269565,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":30,"num_read_kb":13,"num_write":476,"num_write_kb":5875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"65'508","reported_seq":2666,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.736735+0000","last_change":"2023-10-30T12:38:17.333966+0000","last_active":"2023-10-30T12:51:12.736735+0000","last_peered":"2023-10-30T12:51:12.736735+0000","last_clean":"2023-10-30T12:51:12.736735+0000","last_became_active":"2023-10-30T12:28:50.973132+0000","last_became_peered":"2023-10-30T12:28:50.973132+0000","last_unstale":"2023-10-30T12:51:12.736735+0000","last_undegraded":"2023-10-30T12:51:12.736735+0000","last_fullsized":"2023-10-30T12:51:12.736735+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":508,"log_dups_size":0,"ondisk_log_size":508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:43:42.826908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0090421609999999999,"stat_sum":{"num_bytes":12646568,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":47,"num_read_kb":16,"num_write":486,"num_write_kb":14458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"66'522","reported_seq":2671,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:22.381126+0000","last_change":"2023-10-30T12:38:17.112051+0000","last_active":"2023-10-30T12:51:22.381126+0000","last_peered":"2023-10-30T12:51:22.381126+0000","last_clean":"2023-10-30T12:51:22.381126+0000","last_became_active":"2023-10-30T12:28:50.967910+0000","last_became_peered":"2023-10-30T12:28:50.967910+0000","last_unstale":"2023-10-30T12:51:22.381126+0000","last_undegraded":"2023-10-30T12:51:22.381126+0000","last_fullsized":"2023-10-30T12:51:22.381126+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":522,"log_dups_size":0,"ondisk_log_size":522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:31:53.805701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040413009999999997,"stat_sum":{"num_bytes":12682645,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":38,"num_read_kb":16,"num_write":498,"num_write_kb":14506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"66'531","reported_seq":2683,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:22.402041+0000","last_change":"2023-10-30T12:38:17.328004+0000","last_active":"2023-10-30T12:51:22.402041+0000","last_peered":"2023-10-30T12:51:22.402041+0000","last_clean":"2023-10-30T12:51:22.402041+0000","last_became_active":"2023-10-30T12:28:50.973288+0000","last_became_peered":"2023-10-30T12:28:50.973288+0000","last_unstale":"2023-10-30T12:51:22.402041+0000","last_undegraded":"2023-10-30T12:51:22.402041+0000","last_fullsized":"2023-10-30T12:51:22.402041+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":531,"log_dups_size":0,"ondisk_log_size":531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:42:52.258224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037299339999999999,"stat_sum":{"num_bytes":4559885,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":41,"num_read_kb":11,"num_write":501,"num_write_kb":7147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"65'470","reported_seq":2620,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.696226+0000","last_change":"2023-10-30T12:38:17.330519+0000","last_active":"2023-10-30T12:51:12.696226+0000","last_peered":"2023-10-30T12:51:12.696226+0000","last_clean":"2023-10-30T12:51:12.696226+0000","last_became_active":"2023-10-30T12:28:50.954988+0000","last_became_peered":"2023-10-30T12:28:50.954988+0000","last_unstale":"2023-10-30T12:51:12.696226+0000","last_undegraded":"2023-10-30T12:51:12.696226+0000","last_fullsized":"2023-10-30T12:51:12.696226+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":470,"log_dups_size":0,"ondisk_log_size":470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:01:29.621026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028962670000000001,"stat_sum":{"num_bytes":8415120,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":15,"num_write":450,"num_write_kb":10234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"66'548","reported_seq":2708,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.686154+0000","last_change":"2023-10-30T12:38:17.336435+0000","last_active":"2023-10-30T12:51:23.686154+0000","last_peered":"2023-10-30T12:51:23.686154+0000","last_clean":"2023-10-30T12:51:23.686154+0000","last_became_active":"2023-10-30T12:28:50.976564+0000","last_became_peered":"2023-10-30T12:28:50.976564+0000","last_unstale":"2023-10-30T12:51:23.686154+0000","last_undegraded":"2023-10-30T12:51:23.686154+0000","last_fullsized":"2023-10-30T12:51:23.686154+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":548,"log_dups_size":0,"ondisk_log_size":548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:02.733738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0094697280000000002,"stat_sum":{"num_bytes":16822419,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":49,"num_read_kb":4116,"num_write":506,"num_write_kb":27572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"66'518","reported_seq":2688,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.731428+0000","last_change":"2023-10-30T12:38:17.327384+0000","last_active":"2023-10-30T12:51:12.731428+0000","last_peered":"2023-10-30T12:51:12.731428+0000","last_clean":"2023-10-30T12:51:12.731428+0000","last_became_active":"2023-10-30T12:28:50.951809+0000","last_became_peered":"2023-10-30T12:28:50.951809+0000","last_unstale":"2023-10-30T12:51:12.731428+0000","last_undegraded":"2023-10-30T12:51:12.731428+0000","last_fullsized":"2023-10-30T12:51:12.731428+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":518,"log_dups_size":0,"ondisk_log_size":518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:26:18.073656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0078740100000000007,"stat_sum":{"num_bytes":16937210,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":59,"num_read_kb":29,"num_write":484,"num_write_kb":22711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"66'547","reported_seq":2711,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:13.217881+0000","last_change":"2023-10-30T12:38:17.323179+0000","last_active":"2023-10-30T12:51:13.217881+0000","last_peered":"2023-10-30T12:51:13.217881+0000","last_clean":"2023-10-30T12:51:13.217881+0000","last_became_active":"2023-10-30T12:29:32.958473+0000","last_became_peered":"2023-10-30T12:29:32.958473+0000","last_unstale":"2023-10-30T12:51:13.217881+0000","last_undegraded":"2023-10-30T12:51:13.217881+0000","last_fullsized":"2023-10-30T12:51:13.217881+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":547,"log_dups_size":0,"ondisk_log_size":547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:07:47.715571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045626240000000004,"stat_sum":{"num_bytes":13032007,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":46,"num_read_kb":17,"num_write":521,"num_write_kb":14861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"65'484","reported_seq":2629,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.654545+0000","last_change":"2023-10-30T12:38:17.113891+0000","last_active":"2023-10-30T12:51:12.654545+0000","last_peered":"2023-10-30T12:51:12.654545+0000","last_clean":"2023-10-30T12:51:12.654545+0000","last_became_active":"2023-10-30T12:28:50.943787+0000","last_became_peered":"2023-10-30T12:28:50.943787+0000","last_unstale":"2023-10-30T12:51:12.654545+0000","last_undegraded":"2023-10-30T12:51:12.654545+0000","last_fullsized":"2023-10-30T12:51:12.654545+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":484,"log_dups_size":0,"ondisk_log_size":484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:51:54.467182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047342360000000002,"stat_sum":{"num_bytes":12598536,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":34,"num_read_kb":4115,"num_write":456,"num_write_kb":18754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"66'573","reported_seq":2734,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.692359+0000","last_change":"2023-10-30T12:38:17.331215+0000","last_active":"2023-10-30T12:51:23.692359+0000","last_peered":"2023-10-30T12:51:23.692359+0000","last_clean":"2023-10-30T12:51:23.692359+0000","last_became_active":"2023-10-30T12:28:50.937906+0000","last_became_peered":"2023-10-30T12:28:50.937906+0000","last_unstale":"2023-10-30T12:51:23.692359+0000","last_undegraded":"2023-10-30T12:51:23.692359+0000","last_fullsized":"2023-10-30T12:51:23.692359+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":573,"log_dups_size":0,"ondisk_log_size":573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:43:06.819593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003369164,"stat_sum":{"num_bytes":4252811,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":50,"num_read_kb":8215,"num_write":540,"num_write_kb":14536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"66'509","reported_seq":2667,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.741036+0000","last_change":"2023-10-30T12:38:17.643256+0000","last_active":"2023-10-30T12:51:12.741036+0000","last_peered":"2023-10-30T12:51:12.741036+0000","last_clean":"2023-10-30T12:51:12.741036+0000","last_became_active":"2023-10-30T12:28:50.932753+0000","last_became_peered":"2023-10-30T12:28:50.932753+0000","last_unstale":"2023-10-30T12:51:12.741036+0000","last_undegraded":"2023-10-30T12:51:12.741036+0000","last_fullsized":"2023-10-30T12:51:12.741036+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":509,"log_dups_size":0,"ondisk_log_size":509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:37:01.519745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027063,"stat_sum":{"num_bytes":16922391,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":47,"num_read_kb":17,"num_write":484,"num_write_kb":19095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"65'539","reported_seq":2700,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.711318+0000","last_change":"2023-10-30T12:38:17.103685+0000","last_active":"2023-10-30T12:51:12.711318+0000","last_peered":"2023-10-30T12:51:12.711318+0000","last_clean":"2023-10-30T12:51:12.711318+0000","last_became_active":"2023-10-30T12:28:50.943011+0000","last_became_peered":"2023-10-30T12:28:50.943011+0000","last_unstale":"2023-10-30T12:51:12.711318+0000","last_undegraded":"2023-10-30T12:51:12.711318+0000","last_fullsized":"2023-10-30T12:51:12.711318+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":539,"log_dups_size":0,"ondisk_log_size":539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:43:12.517498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001813863,"stat_sum":{"num_bytes":21134561,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":51,"num_read_kb":4115,"num_write":503,"num_write_kb":22902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"66'537","reported_seq":2694,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.683828+0000","last_change":"2023-10-30T12:38:17.111689+0000","last_active":"2023-10-30T12:51:23.683828+0000","last_peered":"2023-10-30T12:51:23.683828+0000","last_clean":"2023-10-30T12:51:23.683828+0000","last_became_active":"2023-10-30T12:28:50.952744+0000","last_became_peered":"2023-10-30T12:28:50.952744+0000","last_unstale":"2023-10-30T12:51:23.683828+0000","last_undegraded":"2023-10-30T12:51:23.683828+0000","last_fullsized":"2023-10-30T12:51:23.683828+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":537,"log_dups_size":0,"ondisk_log_size":537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:46:48.822756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034371530000000001,"stat_sum":{"num_bytes":4653479,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":18,"num_write":512,"num_write_kb":9410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"66'554","reported_seq":2705,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:18.626233+0000","last_change":"2023-10-30T12:38:17.120572+0000","last_active":"2023-10-30T12:51:18.626233+0000","last_peered":"2023-10-30T12:51:18.626233+0000","last_clean":"2023-10-30T12:51:18.626233+0000","last_became_active":"2023-10-30T12:28:50.974576+0000","last_became_peered":"2023-10-30T12:28:50.974576+0000","last_unstale":"2023-10-30T12:51:18.626233+0000","last_undegraded":"2023-10-30T12:51:18.626233+0000","last_fullsized":"2023-10-30T12:51:18.626233+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":554,"log_dups_size":0,"ondisk_log_size":554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:19:51.714260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080003049999999992,"stat_sum":{"num_bytes":8578534,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":4113,"num_write":520,"num_write_kb":16515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"66'507","reported_seq":2676,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.747521+0000","last_change":"2023-10-30T12:38:17.285320+0000","last_active":"2023-10-30T12:51:12.747521+0000","last_peered":"2023-10-30T12:51:12.747521+0000","last_clean":"2023-10-30T12:51:12.747521+0000","last_became_active":"2023-10-30T12:28:50.937384+0000","last_became_peered":"2023-10-30T12:28:50.937384+0000","last_unstale":"2023-10-30T12:51:12.747521+0000","last_undegraded":"2023-10-30T12:51:12.747521+0000","last_fullsized":"2023-10-30T12:51:12.747521+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:54:42.597082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001162803,"stat_sum":{"num_bytes":8483984,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":58,"num_read_kb":27,"num_write":482,"num_write_kb":9419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"66'517","reported_seq":2684,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.745991+0000","last_change":"2023-10-30T12:38:17.336198+0000","last_active":"2023-10-30T12:51:12.745991+0000","last_peered":"2023-10-30T12:51:12.745991+0000","last_clean":"2023-10-30T12:51:12.745991+0000","last_became_active":"2023-10-30T12:28:50.973369+0000","last_became_peered":"2023-10-30T12:28:50.973369+0000","last_unstale":"2023-10-30T12:51:12.745991+0000","last_undegraded":"2023-10-30T12:51:12.745991+0000","last_fullsized":"2023-10-30T12:51:12.745991+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:57:45.650842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093923919999999994,"stat_sum":{"num_bytes":12735800,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":55,"num_read_kb":4114,"num_write":492,"num_write_kb":18044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"66'550","reported_seq":2711,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:22.554611+0000","last_change":"2023-10-30T12:38:17.325892+0000","last_active":"2023-10-30T12:51:22.554611+0000","last_peered":"2023-10-30T12:51:22.554611+0000","last_clean":"2023-10-30T12:51:22.554611+0000","last_became_active":"2023-10-30T12:28:50.954831+0000","last_became_peered":"2023-10-30T12:28:50.954831+0000","last_unstale":"2023-10-30T12:51:22.554611+0000","last_undegraded":"2023-10-30T12:51:22.554611+0000","last_fullsized":"2023-10-30T12:51:22.554611+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":550,"log_dups_size":0,"ondisk_log_size":550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:54:29.936622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007119116,"stat_sum":{"num_bytes":21023102,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":49,"num_read_kb":4113,"num_write":518,"num_write_kb":30328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"66'515","reported_seq":2668,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.686674+0000","last_change":"2023-10-30T12:38:17.644608+0000","last_active":"2023-10-30T12:51:23.686674+0000","last_peered":"2023-10-30T12:51:23.686674+0000","last_clean":"2023-10-30T12:51:23.686674+0000","last_became_active":"2023-10-30T12:28:50.962065+0000","last_became_peered":"2023-10-30T12:28:50.962065+0000","last_unstale":"2023-10-30T12:51:23.686674+0000","last_undegraded":"2023-10-30T12:51:23.686674+0000","last_fullsized":"2023-10-30T12:51:23.686674+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:07:44.264938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037739459999999998,"stat_sum":{"num_bytes":25185776,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":41,"num_read_kb":21,"num_write":488,"num_write_kb":26337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"66'577","reported_seq":2735,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:22.381186+0000","last_change":"2023-10-30T12:38:17.118400+0000","last_active":"2023-10-30T12:51:22.381186+0000","last_peered":"2023-10-30T12:51:22.381186+0000","last_clean":"2023-10-30T12:51:22.381186+0000","last_became_active":"2023-10-30T12:28:50.967651+0000","last_became_peered":"2023-10-30T12:28:50.967651+0000","last_unstale":"2023-10-30T12:51:22.381186+0000","last_undegraded":"2023-10-30T12:51:22.381186+0000","last_fullsized":"2023-10-30T12:51:22.381186+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":577,"log_dups_size":0,"ondisk_log_size":577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:19:34.662364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081085610000000002,"stat_sum":{"num_bytes":8464992,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":21,"num_write":548,"num_write_kb":13569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"66'530","reported_seq":2692,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.738894+0000","last_change":"2023-10-30T12:38:17.332602+0000","last_active":"2023-10-30T12:51:12.738894+0000","last_peered":"2023-10-30T12:51:12.738894+0000","last_clean":"2023-10-30T12:51:12.738894+0000","last_became_active":"2023-10-30T12:28:50.962290+0000","last_became_peered":"2023-10-30T12:28:50.962290+0000","last_unstale":"2023-10-30T12:51:12.738894+0000","last_undegraded":"2023-10-30T12:51:12.738894+0000","last_fullsized":"2023-10-30T12:51:12.738894+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:45:30.777077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042026670000000002,"stat_sum":{"num_bytes":57151,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":50,"num_read_kb":4117,"num_write":500,"num_write_kb":6118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"66'550","reported_seq":2702,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:24.465670+0000","last_change":"2023-10-30T12:38:17.114285+0000","last_active":"2023-10-30T12:51:24.465670+0000","last_peered":"2023-10-30T12:51:24.465670+0000","last_clean":"2023-10-30T12:51:24.465670+0000","last_became_active":"2023-10-30T12:28:50.965713+0000","last_became_peered":"2023-10-30T12:28:50.965713+0000","last_unstale":"2023-10-30T12:51:24.465670+0000","last_undegraded":"2023-10-30T12:51:24.465670+0000","last_fullsized":"2023-10-30T12:51:24.465670+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":550,"log_dups_size":0,"ondisk_log_size":550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:08:31.859565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056034209999999999,"stat_sum":{"num_bytes":12649475,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":40,"num_read_kb":16,"num_write":522,"num_write_kb":18815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"66'503","reported_seq":2646,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:18.626419+0000","last_change":"2023-10-30T12:38:17.330864+0000","last_active":"2023-10-30T12:51:18.626419+0000","last_peered":"2023-10-30T12:51:18.626419+0000","last_clean":"2023-10-30T12:51:18.626419+0000","last_became_active":"2023-10-30T12:28:50.951026+0000","last_became_peered":"2023-10-30T12:28:50.951026+0000","last_unstale":"2023-10-30T12:51:18.626419+0000","last_undegraded":"2023-10-30T12:51:18.626419+0000","last_fullsized":"2023-10-30T12:51:18.626419+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":503,"log_dups_size":0,"ondisk_log_size":503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:39:54.872161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031697840000000001,"stat_sum":{"num_bytes":13094288,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":31,"num_read_kb":12,"num_write":474,"num_write_kb":19622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"66'579","reported_seq":2742,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.737087+0000","last_change":"2023-10-30T12:38:17.327887+0000","last_active":"2023-10-30T12:51:12.737087+0000","last_peered":"2023-10-30T12:51:12.737087+0000","last_clean":"2023-10-30T12:51:12.737087+0000","last_became_active":"2023-10-30T12:28:50.962484+0000","last_became_peered":"2023-10-30T12:28:50.962484+0000","last_unstale":"2023-10-30T12:51:12.737087+0000","last_undegraded":"2023-10-30T12:51:12.737087+0000","last_fullsized":"2023-10-30T12:51:12.737087+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:41:18.860625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083773500000000004,"stat_sum":{"num_bytes":13048022,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":51,"num_read_kb":23,"num_write":546,"num_write_kb":15053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"66'481","reported_seq":2635,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.743969+0000","last_change":"2023-10-30T12:38:17.104453+0000","last_active":"2023-10-30T12:51:12.743969+0000","last_peered":"2023-10-30T12:51:12.743969+0000","last_clean":"2023-10-30T12:51:12.743969+0000","last_became_active":"2023-10-30T12:28:50.933093+0000","last_became_peered":"2023-10-30T12:28:50.933093+0000","last_unstale":"2023-10-30T12:51:12.743969+0000","last_undegraded":"2023-10-30T12:51:12.743969+0000","last_fullsized":"2023-10-30T12:51:12.743969+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":481,"log_dups_size":0,"ondisk_log_size":481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:58:51.514957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022251829999999999,"stat_sum":{"num_bytes":25268103,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":42,"num_read_kb":8207,"num_write":439,"num_write_kb":27476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"65'470","reported_seq":2608,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.727936+0000","last_change":"2023-10-30T12:38:17.332060+0000","last_active":"2023-10-30T12:51:12.727936+0000","last_peered":"2023-10-30T12:51:12.727936+0000","last_clean":"2023-10-30T12:51:12.727936+0000","last_became_active":"2023-10-30T12:28:50.952531+0000","last_became_peered":"2023-10-30T12:28:50.952531+0000","last_unstale":"2023-10-30T12:51:12.727936+0000","last_undegraded":"2023-10-30T12:51:12.727936+0000","last_fullsized":"2023-10-30T12:51:12.727936+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":470,"log_dups_size":0,"ondisk_log_size":470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:22:53.945790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039301029999999999,"stat_sum":{"num_bytes":4497430,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":26,"num_read_kb":3284,"num_write":447,"num_write_kb":14120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"65'492","reported_seq":2649,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.733923+0000","last_change":"2023-10-30T12:38:17.644463+0000","last_active":"2023-10-30T12:51:12.733923+0000","last_peered":"2023-10-30T12:51:12.733923+0000","last_clean":"2023-10-30T12:51:12.733923+0000","last_became_active":"2023-10-30T12:28:50.933124+0000","last_became_peered":"2023-10-30T12:28:50.933124+0000","last_unstale":"2023-10-30T12:51:12.733923+0000","last_undegraded":"2023-10-30T12:51:12.733923+0000","last_fullsized":"2023-10-30T12:51:12.733923+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":492,"log_dups_size":0,"ondisk_log_size":492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:25:34.819552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034752709999999998,"stat_sum":{"num_bytes":8485989,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":45,"num_read_kb":14,"num_write":462,"num_write_kb":10450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"66'566","reported_seq":2729,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.737666+0000","last_change":"2023-10-30T12:38:17.118502+0000","last_active":"2023-10-30T12:51:12.737666+0000","last_peered":"2023-10-30T12:51:12.737666+0000","last_clean":"2023-10-30T12:51:12.737666+0000","last_became_active":"2023-10-30T12:28:50.965639+0000","last_became_peered":"2023-10-30T12:28:50.965639+0000","last_unstale":"2023-10-30T12:51:12.737666+0000","last_undegraded":"2023-10-30T12:51:12.737666+0000","last_fullsized":"2023-10-30T12:51:12.737666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":566,"log_dups_size":0,"ondisk_log_size":566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:46:12.692237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076861280000000004,"stat_sum":{"num_bytes":8419509,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":51,"num_read_kb":20,"num_write":541,"num_write_kb":11490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"66'561","reported_seq":2723,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.739391+0000","last_change":"2023-10-30T12:38:17.285366+0000","last_active":"2023-10-30T12:51:12.739391+0000","last_peered":"2023-10-30T12:51:12.739391+0000","last_clean":"2023-10-30T12:51:12.739391+0000","last_became_active":"2023-10-30T12:28:50.955084+0000","last_became_peered":"2023-10-30T12:28:50.955084+0000","last_unstale":"2023-10-30T12:51:12.739391+0000","last_undegraded":"2023-10-30T12:51:12.739391+0000","last_fullsized":"2023-10-30T12:51:12.739391+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":561,"log_dups_size":0,"ondisk_log_size":561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:56:51.880241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012043119999999999,"stat_sum":{"num_bytes":4261339,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":50,"num_read_kb":22,"num_write":525,"num_write_kb":9810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"65'533","reported_seq":2698,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.730032+0000","last_change":"2023-10-30T12:38:17.326196+0000","last_active":"2023-10-30T12:51:12.730032+0000","last_peered":"2023-10-30T12:51:12.730032+0000","last_clean":"2023-10-30T12:51:12.730032+0000","last_became_active":"2023-10-30T12:28:50.944775+0000","last_became_peered":"2023-10-30T12:28:50.944775+0000","last_unstale":"2023-10-30T12:51:12.730032+0000","last_undegraded":"2023-10-30T12:51:12.730032+0000","last_fullsized":"2023-10-30T12:51:12.730032+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":533,"log_dups_size":0,"ondisk_log_size":533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:02:19.297081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070573429999999998,"stat_sum":{"num_bytes":12715391,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":53,"num_read_kb":21,"num_write":506,"num_write_kb":13685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"66'553","reported_seq":2704,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.740036+0000","last_change":"2023-10-30T12:38:17.109227+0000","last_active":"2023-10-30T12:51:12.740036+0000","last_peered":"2023-10-30T12:51:12.740036+0000","last_clean":"2023-10-30T12:51:12.740036+0000","last_became_active":"2023-10-30T12:28:50.940567+0000","last_became_peered":"2023-10-30T12:28:50.940567+0000","last_unstale":"2023-10-30T12:51:12.740036+0000","last_undegraded":"2023-10-30T12:51:12.740036+0000","last_fullsized":"2023-10-30T12:51:12.740036+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":553,"log_dups_size":0,"ondisk_log_size":553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:56:41.791482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028523340000000002,"stat_sum":{"num_bytes":9733426,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":40,"num_read_kb":17,"num_write":523,"num_write_kb":12778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"65'278","reported_seq":357,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:46.346718+0000","last_change":"2023-10-30T12:28:50.395670+0000","last_active":"2023-10-30T12:50:46.346718+0000","last_peered":"2023-10-30T12:50:46.346718+0000","last_clean":"2023-10-30T12:50:46.346718+0000","last_became_active":"2023-10-30T12:28:50.395382+0000","last_became_peered":"2023-10-30T12:28:50.395382+0000","last_unstale":"2023-10-30T12:50:46.346718+0000","last_undegraded":"2023-10-30T12:50:46.346718+0000","last_fullsized":"2023-10-30T12:50:46.346718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":278,"log_dups_size":0,"ondisk_log_size":278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:14.858121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582946,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":45059,"num_write":213,"num_write_kb":12413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"66'472","reported_seq":2628,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:18.626809+0000","last_change":"2023-10-30T12:38:17.644896+0000","last_active":"2023-10-30T12:51:18.626809+0000","last_peered":"2023-10-30T12:51:18.626809+0000","last_clean":"2023-10-30T12:51:18.626809+0000","last_became_active":"2023-10-30T12:28:50.952613+0000","last_became_peered":"2023-10-30T12:28:50.952613+0000","last_unstale":"2023-10-30T12:51:18.626809+0000","last_undegraded":"2023-10-30T12:51:18.626809+0000","last_fullsized":"2023-10-30T12:51:18.626809+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":472,"log_dups_size":0,"ondisk_log_size":472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:16:49.712167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038505420000000002,"stat_sum":{"num_bytes":21060260,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":45,"num_read_kb":18,"num_write":442,"num_write_kb":22001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"65'896","reported_seq":1174,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:47.899291+0000","last_change":"2023-10-30T12:28:50.384598+0000","last_active":"2023-10-30T12:50:47.899291+0000","last_peered":"2023-10-30T12:50:47.899291+0000","last_clean":"2023-10-30T12:50:47.899291+0000","last_became_active":"2023-10-30T12:28:50.383898+0000","last_became_peered":"2023-10-30T12:28:50.383898+0000","last_unstale":"2023-10-30T12:50:47.899291+0000","last_undegraded":"2023-10-30T12:50:47.899291+0000","last_fullsized":"2023-10-30T12:50:47.899291+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":896,"log_dups_size":0,"ondisk_log_size":896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:09:50.290004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21559845,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":229,"num_read_kb":49875,"num_write":837,"num_write_kb":21586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"66'442","reported_seq":733,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:26.502943+0000","last_change":"2023-10-30T12:28:50.386193+0000","last_active":"2023-10-30T12:51:26.502943+0000","last_peered":"2023-10-30T12:51:26.502943+0000","last_clean":"2023-10-30T12:51:26.502943+0000","last_became_active":"2023-10-30T12:28:50.385963+0000","last_became_peered":"2023-10-30T12:28:50.385963+0000","last_unstale":"2023-10-30T12:51:26.502943+0000","last_undegraded":"2023-10-30T12:51:26.502943+0000","last_fullsized":"2023-10-30T12:51:26.502943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":442,"log_dups_size":0,"ondisk_log_size":442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:00:55.603212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7740025,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":238,"num_read_kb":9091,"num_write":376,"num_write_kb":7812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"66'601","reported_seq":2768,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:18.629254+0000","last_change":"2023-10-30T12:38:17.330785+0000","last_active":"2023-10-30T12:51:18.629254+0000","last_peered":"2023-10-30T12:51:18.629254+0000","last_clean":"2023-10-30T12:51:18.629254+0000","last_became_active":"2023-10-30T12:28:50.935764+0000","last_became_peered":"2023-10-30T12:28:50.935764+0000","last_unstale":"2023-10-30T12:51:18.629254+0000","last_undegraded":"2023-10-30T12:51:18.629254+0000","last_fullsized":"2023-10-30T12:51:18.629254+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":601,"log_dups_size":0,"ondisk_log_size":601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:59:59.545417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031329320000000002,"stat_sum":{"num_bytes":4238419,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":56,"num_read_kb":4120,"num_write":560,"num_write_kb":13730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"65'568","reported_seq":643,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:46.705145+0000","last_change":"2023-10-30T12:28:50.392213+0000","last_active":"2023-10-30T12:50:46.705145+0000","last_peered":"2023-10-30T12:50:46.705145+0000","last_clean":"2023-10-30T12:50:46.705145+0000","last_became_active":"2023-10-30T12:28:50.391494+0000","last_became_peered":"2023-10-30T12:28:50.391494+0000","last_unstale":"2023-10-30T12:50:46.705145+0000","last_undegraded":"2023-10-30T12:50:46.705145+0000","last_fullsized":"2023-10-30T12:50:46.705145+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:38:37.301840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":36864,"num_write":519,"num_write_kb":12589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"66'565","reported_seq":2722,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.747887+0000","last_change":"2023-10-30T12:38:17.103988+0000","last_active":"2023-10-30T12:51:12.747887+0000","last_peered":"2023-10-30T12:51:12.747887+0000","last_clean":"2023-10-30T12:51:12.747887+0000","last_became_active":"2023-10-30T12:28:50.933062+0000","last_became_peered":"2023-10-30T12:28:50.933062+0000","last_unstale":"2023-10-30T12:51:12.747887+0000","last_undegraded":"2023-10-30T12:51:12.747887+0000","last_fullsized":"2023-10-30T12:51:12.747887+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":565,"log_dups_size":0,"ondisk_log_size":565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:52:57.336116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021140489999999998,"stat_sum":{"num_bytes":33649708,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":46,"num_read_kb":4115,"num_write":540,"num_write_kb":39124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"66'128","reported_seq":201,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.642253+0000","last_change":"2023-10-30T12:28:50.394368+0000","last_active":"2023-10-30T12:51:12.642253+0000","last_peered":"2023-10-30T12:51:12.642253+0000","last_clean":"2023-10-30T12:51:12.642253+0000","last_became_active":"2023-10-30T12:28:50.393686+0000","last_became_peered":"2023-10-30T12:28:50.393686+0000","last_unstale":"2023-10-30T12:51:12.642253+0000","last_undegraded":"2023-10-30T12:51:12.642253+0000","last_fullsized":"2023-10-30T12:51:12.642253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:43:17.726462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":73,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"65'540","reported_seq":2702,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.734949+0000","last_change":"2023-10-30T12:38:17.119770+0000","last_active":"2023-10-30T12:51:12.734949+0000","last_peered":"2023-10-30T12:51:12.734949+0000","last_clean":"2023-10-30T12:51:12.734949+0000","last_became_active":"2023-10-30T12:28:50.963105+0000","last_became_peered":"2023-10-30T12:28:50.963105+0000","last_unstale":"2023-10-30T12:51:12.734949+0000","last_undegraded":"2023-10-30T12:51:12.734949+0000","last_fullsized":"2023-10-30T12:51:12.734949+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":540,"log_dups_size":0,"ondisk_log_size":540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:12:25.298678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080066809999999999,"stat_sum":{"num_bytes":16891650,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":51,"num_read_kb":18,"num_write":510,"num_write_kb":18448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"66'281","reported_seq":354,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:47.673064+0000","last_change":"2023-10-30T12:28:50.397679+0000","last_active":"2023-10-30T12:50:47.673064+0000","last_peered":"2023-10-30T12:50:47.673064+0000","last_clean":"2023-10-30T12:50:47.673064+0000","last_became_active":"2023-10-30T12:28:50.397041+0000","last_became_peered":"2023-10-30T12:28:50.397041+0000","last_unstale":"2023-10-30T12:50:47.673064+0000","last_undegraded":"2023-10-30T12:50:47.673064+0000","last_fullsized":"2023-10-30T12:50:47.673064+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:52:22.564616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":16396,"num_write":240,"num_write_kb":8312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"66'521","reported_seq":2674,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.744719+0000","last_change":"2023-10-30T12:38:17.644799+0000","last_active":"2023-10-30T12:51:12.744719+0000","last_peered":"2023-10-30T12:51:12.744719+0000","last_clean":"2023-10-30T12:51:12.744719+0000","last_became_active":"2023-10-30T12:28:50.966569+0000","last_became_peered":"2023-10-30T12:28:50.966569+0000","last_unstale":"2023-10-30T12:51:12.744719+0000","last_undegraded":"2023-10-30T12:51:12.744719+0000","last_fullsized":"2023-10-30T12:51:12.744719+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:43:03.050908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037972850000000001,"stat_sum":{"num_bytes":16921261,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":42,"num_read_kb":19,"num_write":497,"num_write_kb":17986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"66'370","reported_seq":449,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:18.810052+0000","last_change":"2023-10-30T12:28:50.391527+0000","last_active":"2023-10-30T12:51:18.810052+0000","last_peered":"2023-10-30T12:51:18.810052+0000","last_clean":"2023-10-30T12:51:18.810052+0000","last_became_active":"2023-10-30T12:28:50.391260+0000","last_became_peered":"2023-10-30T12:28:50.391260+0000","last_unstale":"2023-10-30T12:51:18.810052+0000","last_undegraded":"2023-10-30T12:51:18.810052+0000","last_fullsized":"2023-10-30T12:51:18.810052+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:29:34.772490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":22,"num_read_kb":7467,"num_write":318,"num_write_kb":4400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"66'490","reported_seq":2648,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.693047+0000","last_change":"2023-10-30T12:38:17.114492+0000","last_active":"2023-10-30T12:51:12.693047+0000","last_peered":"2023-10-30T12:51:12.693047+0000","last_clean":"2023-10-30T12:51:12.693047+0000","last_became_active":"2023-10-30T12:28:50.974843+0000","last_became_peered":"2023-10-30T12:28:50.974843+0000","last_unstale":"2023-10-30T12:51:12.693047+0000","last_undegraded":"2023-10-30T12:51:12.693047+0000","last_fullsized":"2023-10-30T12:51:12.693047+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":490,"log_dups_size":0,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:31:41.341035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005161372,"stat_sum":{"num_bytes":4290434,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":47,"num_read_kb":17,"num_write":450,"num_write_kb":9837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"66'349","reported_seq":439,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.670523+0000","last_change":"2023-10-30T12:28:50.374838+0000","last_active":"2023-10-30T12:51:12.670523+0000","last_peered":"2023-10-30T12:51:12.670523+0000","last_clean":"2023-10-30T12:51:12.670523+0000","last_became_active":"2023-10-30T12:28:50.373581+0000","last_became_peered":"2023-10-30T12:28:50.373581+0000","last_unstale":"2023-10-30T12:51:12.670523+0000","last_undegraded":"2023-10-30T12:51:12.670523+0000","last_fullsized":"2023-10-30T12:51:12.670523+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:14:51.506745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":43,"num_read_kb":32803,"num_write":291,"num_write_kb":12542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"65'466","reported_seq":2615,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.728219+0000","last_change":"2023-10-30T12:38:17.328512+0000","last_active":"2023-10-30T12:51:12.728219+0000","last_peered":"2023-10-30T12:51:12.728219+0000","last_clean":"2023-10-30T12:51:12.728219+0000","last_became_active":"2023-10-30T12:28:50.973269+0000","last_became_peered":"2023-10-30T12:28:50.973269+0000","last_unstale":"2023-10-30T12:51:12.728219+0000","last_undegraded":"2023-10-30T12:51:12.728219+0000","last_fullsized":"2023-10-30T12:51:12.728219+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":466,"log_dups_size":0,"ondisk_log_size":466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:26:11.417670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004001498,"stat_sum":{"num_bytes":12676913,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":38,"num_read_kb":11,"num_write":436,"num_write_kb":13453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"66'436","reported_seq":527,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.511431+0000","last_change":"2023-10-30T12:28:50.366590+0000","last_active":"2023-10-30T12:51:12.511431+0000","last_peered":"2023-10-30T12:51:12.511431+0000","last_clean":"2023-10-30T12:51:12.511431+0000","last_became_active":"2023-10-30T12:28:50.366269+0000","last_became_peered":"2023-10-30T12:28:50.366269+0000","last_unstale":"2023-10-30T12:51:12.511431+0000","last_undegraded":"2023-10-30T12:51:12.511431+0000","last_fullsized":"2023-10-30T12:51:12.511431+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":436,"log_dups_size":0,"ondisk_log_size":436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:15:11.963438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":52,"num_read_kb":36923,"num_write":386,"num_write_kb":8499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"65'536","reported_seq":2699,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.738218+0000","last_change":"2023-10-30T12:38:17.644043+0000","last_active":"2023-10-30T12:51:12.738218+0000","last_peered":"2023-10-30T12:51:12.738218+0000","last_clean":"2023-10-30T12:51:12.738218+0000","last_became_active":"2023-10-30T12:28:50.961805+0000","last_became_peered":"2023-10-30T12:28:50.961805+0000","last_unstale":"2023-10-30T12:51:12.738218+0000","last_undegraded":"2023-10-30T12:51:12.738218+0000","last_fullsized":"2023-10-30T12:51:12.738218+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":536,"log_dups_size":0,"ondisk_log_size":536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:50:40.603836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032264350000000002,"stat_sum":{"num_bytes":4573455,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":52,"num_read_kb":4114,"num_write":491,"num_write_kb":14136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"66'258","reported_seq":333,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:07.342026+0000","last_change":"2023-10-30T12:28:50.397244+0000","last_active":"2023-10-30T12:51:07.342026+0000","last_peered":"2023-10-30T12:51:07.342026+0000","last_clean":"2023-10-30T12:51:07.342026+0000","last_became_active":"2023-10-30T12:28:50.396727+0000","last_became_peered":"2023-10-30T12:28:50.396727+0000","last_unstale":"2023-10-30T12:51:07.342026+0000","last_undegraded":"2023-10-30T12:51:07.342026+0000","last_fullsized":"2023-10-30T12:51:07.342026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:33.493343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":16164,"num_write":201,"num_write_kb":8311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"65'535","reported_seq":2701,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.708532+0000","last_change":"2023-10-30T12:38:17.284992+0000","last_active":"2023-10-30T12:51:12.708532+0000","last_peered":"2023-10-30T12:51:12.708532+0000","last_clean":"2023-10-30T12:51:12.708532+0000","last_became_active":"2023-10-30T12:29:32.960608+0000","last_became_peered":"2023-10-30T12:29:32.960608+0000","last_unstale":"2023-10-30T12:51:12.708532+0000","last_undegraded":"2023-10-30T12:51:12.708532+0000","last_fullsized":"2023-10-30T12:51:12.708532+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":535,"log_dups_size":0,"ondisk_log_size":535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:22:33.324798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012934999999999999,"stat_sum":{"num_bytes":4287412,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":59,"num_read_kb":33,"num_write":498,"num_write_kb":6237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"65'677","reported_seq":758,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-30T12:50:01.976866+0000","last_change":"2023-10-30T12:28:50.383880+0000","last_active":"2023-10-30T12:50:01.976866+0000","last_peered":"2023-10-30T12:50:01.976866+0000","last_clean":"2023-10-30T12:50:01.976866+0000","last_became_active":"2023-10-30T12:28:50.383570+0000","last_became_peered":"2023-10-30T12:28:50.383570+0000","last_unstale":"2023-10-30T12:50:01.976866+0000","last_undegraded":"2023-10-30T12:50:01.976866+0000","last_fullsized":"2023-10-30T12:50:01.976866+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":677,"log_dups_size":0,"ondisk_log_size":677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:19:42.047520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":37306,"num_write":619,"num_write_kb":16724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"65'498","reported_seq":2642,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.684505+0000","last_change":"2023-10-30T12:38:17.322231+0000","last_active":"2023-10-30T12:51:12.684505+0000","last_peered":"2023-10-30T12:51:12.684505+0000","last_clean":"2023-10-30T12:51:12.684505+0000","last_became_active":"2023-10-30T12:28:50.949485+0000","last_became_peered":"2023-10-30T12:28:50.949485+0000","last_unstale":"2023-10-30T12:51:12.684505+0000","last_undegraded":"2023-10-30T12:51:12.684505+0000","last_fullsized":"2023-10-30T12:51:12.684505+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:30:36.516655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040897299999999998,"stat_sum":{"num_bytes":12614326,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":469,"num_write_kb":14309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"66'437","reported_seq":543,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.615726+0000","last_change":"2023-10-30T12:28:50.362979+0000","last_active":"2023-10-30T12:51:12.615726+0000","last_peered":"2023-10-30T12:51:12.615726+0000","last_clean":"2023-10-30T12:51:12.615726+0000","last_became_active":"2023-10-30T12:28:50.362624+0000","last_became_peered":"2023-10-30T12:28:50.362624+0000","last_unstale":"2023-10-30T12:51:12.615726+0000","last_undegraded":"2023-10-30T12:51:12.615726+0000","last_fullsized":"2023-10-30T12:51:12.615726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:01:13.111816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583024,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":93,"num_read_kb":24662,"num_write":374,"num_write_kb":12554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"66'553","reported_seq":2714,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:18.625349+0000","last_change":"2023-10-30T12:38:17.334068+0000","last_active":"2023-10-30T12:51:18.625349+0000","last_peered":"2023-10-30T12:51:18.625349+0000","last_clean":"2023-10-30T12:51:18.625349+0000","last_became_active":"2023-10-30T12:28:50.979344+0000","last_became_peered":"2023-10-30T12:28:50.979344+0000","last_unstale":"2023-10-30T12:51:18.625349+0000","last_undegraded":"2023-10-30T12:51:18.625349+0000","last_fullsized":"2023-10-30T12:51:18.625349+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":553,"log_dups_size":0,"ondisk_log_size":553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:32:04.956803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0092295529999999997,"stat_sum":{"num_bytes":12635563,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":50,"num_read_kb":8210,"num_write":525,"num_write_kb":23175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"66'245","reported_seq":322,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:25.413473+0000","last_change":"2023-10-30T12:28:50.382175+0000","last_active":"2023-10-30T12:51:25.413473+0000","last_peered":"2023-10-30T12:51:25.413473+0000","last_clean":"2023-10-30T12:51:25.413473+0000","last_became_active":"2023-10-30T12:28:50.381125+0000","last_became_peered":"2023-10-30T12:28:50.381125+0000","last_unstale":"2023-10-30T12:51:25.413473+0000","last_undegraded":"2023-10-30T12:51:25.413473+0000","last_fullsized":"2023-10-30T12:51:25.413473+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":245,"log_dups_size":0,"ondisk_log_size":245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:36:02.652665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":12378,"num_write":394,"num_write_kb":4652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"65'503","reported_seq":2651,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.747470+0000","last_change":"2023-10-30T12:38:17.334809+0000","last_active":"2023-10-30T12:51:12.747470+0000","last_peered":"2023-10-30T12:51:12.747470+0000","last_clean":"2023-10-30T12:51:12.747470+0000","last_became_active":"2023-10-30T12:28:50.977585+0000","last_became_peered":"2023-10-30T12:28:50.977585+0000","last_unstale":"2023-10-30T12:51:12.747470+0000","last_undegraded":"2023-10-30T12:51:12.747470+0000","last_fullsized":"2023-10-30T12:51:12.747470+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":503,"log_dups_size":0,"ondisk_log_size":503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:33:51.736957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089211889999999995,"stat_sum":{"num_bytes":12866878,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":36,"num_read_kb":8205,"num_write":472,"num_write_kb":18794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"65'355","reported_seq":432,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.691046+0000","last_change":"2023-10-30T12:28:50.367258+0000","last_active":"2023-10-30T12:51:12.691046+0000","last_peered":"2023-10-30T12:51:12.691046+0000","last_clean":"2023-10-30T12:51:12.691046+0000","last_became_active":"2023-10-30T12:28:50.366974+0000","last_became_peered":"2023-10-30T12:28:50.366974+0000","last_unstale":"2023-10-30T12:51:12.691046+0000","last_undegraded":"2023-10-30T12:51:12.691046+0000","last_fullsized":"2023-10-30T12:51:12.691046+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:14:48.823902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":15324,"num_write":292,"num_write_kb":8361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"66'510","reported_seq":2672,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:24.460236+0000","last_change":"2023-10-30T12:38:17.337139+0000","last_active":"2023-10-30T12:51:24.460236+0000","last_peered":"2023-10-30T12:51:24.460236+0000","last_clean":"2023-10-30T12:51:24.460236+0000","last_became_active":"2023-10-30T12:28:50.981193+0000","last_became_peered":"2023-10-30T12:28:50.981193+0000","last_unstale":"2023-10-30T12:51:24.460236+0000","last_undegraded":"2023-10-30T12:51:24.460236+0000","last_fullsized":"2023-10-30T12:51:24.460236+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":510,"log_dups_size":0,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:32:10.843997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096579700000000001,"stat_sum":{"num_bytes":12827883,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":50,"num_read_kb":4114,"num_write":480,"num_write_kb":17991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"66'96","reported_seq":169,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:22.334230+0000","last_change":"2023-10-30T12:28:50.368011+0000","last_active":"2023-10-30T12:51:22.334230+0000","last_peered":"2023-10-30T12:51:22.334230+0000","last_clean":"2023-10-30T12:51:22.334230+0000","last_became_active":"2023-10-30T12:28:50.367730+0000","last_became_peered":"2023-10-30T12:28:50.367730+0000","last_unstale":"2023-10-30T12:51:22.334230+0000","last_undegraded":"2023-10-30T12:51:22.334230+0000","last_fullsized":"2023-10-30T12:51:22.334230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:59:41.946550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":12292,"num_write":45,"num_write_kb":4186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"65'566","reported_seq":2734,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.746266+0000","last_change":"2023-10-30T12:38:17.119150+0000","last_active":"2023-10-30T12:51:12.746266+0000","last_peered":"2023-10-30T12:51:12.746266+0000","last_clean":"2023-10-30T12:51:12.746266+0000","last_became_active":"2023-10-30T12:28:50.966246+0000","last_became_peered":"2023-10-30T12:28:50.966246+0000","last_unstale":"2023-10-30T12:51:12.746266+0000","last_undegraded":"2023-10-30T12:51:12.746266+0000","last_fullsized":"2023-10-30T12:51:12.746266+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":566,"log_dups_size":0,"ondisk_log_size":566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:39:33.054939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007703601,"stat_sum":{"num_bytes":8540203,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":56,"num_read_kb":18,"num_write":531,"num_write_kb":11283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"66'83","reported_seq":159,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:18.809472+0000","last_change":"2023-10-30T12:28:50.394060+0000","last_active":"2023-10-30T12:51:18.809472+0000","last_peered":"2023-10-30T12:51:18.809472+0000","last_clean":"2023-10-30T12:51:18.809472+0000","last_became_active":"2023-10-30T12:28:50.393642+0000","last_became_peered":"2023-10-30T12:28:50.393642+0000","last_unstale":"2023-10-30T12:51:18.809472+0000","last_undegraded":"2023-10-30T12:51:18.809472+0000","last_fullsized":"2023-10-30T12:51:18.809472+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:21.643882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":15,"num_write_kb":101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"65'532","reported_seq":2688,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.708661+0000","last_change":"2023-10-30T12:38:17.335584+0000","last_active":"2023-10-30T12:51:12.708661+0000","last_peered":"2023-10-30T12:51:12.708661+0000","last_clean":"2023-10-30T12:51:12.708661+0000","last_became_active":"2023-10-30T12:28:50.976343+0000","last_became_peered":"2023-10-30T12:28:50.976343+0000","last_unstale":"2023-10-30T12:51:12.708661+0000","last_undegraded":"2023-10-30T12:51:12.708661+0000","last_fullsized":"2023-10-30T12:51:12.708661+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":532,"log_dups_size":0,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:03:37.466766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091875959999999993,"stat_sum":{"num_bytes":16963416,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":44,"num_read_kb":14,"num_write":504,"num_write_kb":22707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":220,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:41.797344+0000","last_change":"2023-10-30T12:28:41.203990+0000","last_active":"2023-10-30T12:50:41.797344+0000","last_peered":"2023-10-30T12:50:41.797344+0000","last_clean":"2023-10-30T12:50:41.797344+0000","last_became_active":"2023-10-30T12:28:41.201796+0000","last_became_peered":"2023-10-30T12:28:41.201796+0000","last_unstale":"2023-10-30T12:50:41.797344+0000","last_undegraded":"2023-10-30T12:50:41.797344+0000","last_fullsized":"2023-10-30T12:50:41.797344+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:39.218257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:39.218257+0000","last_clean_scrub_stamp":"2023-10-30T12:28:39.218257+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:21:00.605462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"66'193","reported_seq":273,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.748291+0000","last_change":"2023-10-30T12:28:50.372651+0000","last_active":"2023-10-30T12:51:23.748291+0000","last_peered":"2023-10-30T12:51:23.748291+0000","last_clean":"2023-10-30T12:51:23.748291+0000","last_became_active":"2023-10-30T12:28:50.367593+0000","last_became_peered":"2023-10-30T12:28:50.367593+0000","last_unstale":"2023-10-30T12:51:23.748291+0000","last_undegraded":"2023-10-30T12:51:23.748291+0000","last_fullsized":"2023-10-30T12:51:23.748291+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:32:10.492061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194338,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":24,"num_read_kb":12326,"num_write":144,"num_write_kb":4223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"65'492","reported_seq":2637,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.709943+0000","last_change":"2023-10-30T12:38:17.285044+0000","last_active":"2023-10-30T12:51:12.709943+0000","last_peered":"2023-10-30T12:51:12.709943+0000","last_clean":"2023-10-30T12:51:12.709943+0000","last_became_active":"2023-10-30T12:29:32.997301+0000","last_became_peered":"2023-10-30T12:29:32.997301+0000","last_unstale":"2023-10-30T12:51:12.709943+0000","last_undegraded":"2023-10-30T12:51:12.709943+0000","last_fullsized":"2023-10-30T12:51:12.709943+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":492,"log_dups_size":0,"ondisk_log_size":492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:12:18.045691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001366412,"stat_sum":{"num_bytes":4230333,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":36,"num_read_kb":11,"num_write":471,"num_write_kb":13453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"66'499","reported_seq":586,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:18.810050+0000","last_change":"2023-10-30T12:28:50.392461+0000","last_active":"2023-10-30T12:51:18.810050+0000","last_peered":"2023-10-30T12:51:18.810050+0000","last_clean":"2023-10-30T12:51:18.810050+0000","last_became_active":"2023-10-30T12:28:50.391824+0000","last_became_peered":"2023-10-30T12:28:50.391824+0000","last_unstale":"2023-10-30T12:51:18.810050+0000","last_undegraded":"2023-10-30T12:51:18.810050+0000","last_fullsized":"2023-10-30T12:51:18.810050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":499,"log_dups_size":0,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:56:10.373046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":32,"num_read_kb":50322,"num_write":431,"num_write_kb":16616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"66'503","reported_seq":2644,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.731399+0000","last_change":"2023-10-30T12:38:17.643111+0000","last_active":"2023-10-30T12:51:12.731399+0000","last_peered":"2023-10-30T12:51:12.731399+0000","last_clean":"2023-10-30T12:51:12.731399+0000","last_became_active":"2023-10-30T12:28:50.952625+0000","last_became_peered":"2023-10-30T12:28:50.952625+0000","last_unstale":"2023-10-30T12:51:12.731399+0000","last_undegraded":"2023-10-30T12:51:12.731399+0000","last_fullsized":"2023-10-30T12:51:12.731399+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":503,"log_dups_size":0,"ondisk_log_size":503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:43:17.223141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026695970000000001,"stat_sum":{"num_bytes":16935229,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":29,"num_read_kb":10,"num_write":464,"num_write_kb":20216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"66'579","reported_seq":882,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:25.409837+0000","last_change":"2023-10-30T12:28:50.367547+0000","last_active":"2023-10-30T12:51:25.409837+0000","last_peered":"2023-10-30T12:51:25.409837+0000","last_clean":"2023-10-30T12:51:25.409837+0000","last_became_active":"2023-10-30T12:28:50.367230+0000","last_became_peered":"2023-10-30T12:28:50.367230+0000","last_unstale":"2023-10-30T12:51:25.409837+0000","last_undegraded":"2023-10-30T12:51:25.409837+0000","last_fullsized":"2023-10-30T12:51:25.409837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:42.570786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11931699,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":249,"num_read_kb":13021,"num_write":520,"num_write_kb":11989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"66'527","reported_seq":2687,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.683264+0000","last_change":"2023-10-30T12:38:17.104045+0000","last_active":"2023-10-30T12:51:23.683264+0000","last_peered":"2023-10-30T12:51:23.683264+0000","last_clean":"2023-10-30T12:51:23.683264+0000","last_became_active":"2023-10-30T12:28:50.928608+0000","last_became_peered":"2023-10-30T12:28:50.928608+0000","last_unstale":"2023-10-30T12:51:23.683264+0000","last_undegraded":"2023-10-30T12:51:23.683264+0000","last_fullsized":"2023-10-30T12:51:23.683264+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":527,"log_dups_size":0,"ondisk_log_size":527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:17:59.249824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002074275,"stat_sum":{"num_bytes":8442985,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":47,"num_read_kb":21,"num_write":493,"num_write_kb":11774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"66'493","reported_seq":572,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:46.897234+0000","last_change":"2023-10-30T12:28:50.391240+0000","last_active":"2023-10-30T12:50:46.897234+0000","last_peered":"2023-10-30T12:50:46.897234+0000","last_clean":"2023-10-30T12:50:46.897234+0000","last_became_active":"2023-10-30T12:28:50.390864+0000","last_became_peered":"2023-10-30T12:28:50.390864+0000","last_unstale":"2023-10-30T12:50:46.897234+0000","last_undegraded":"2023-10-30T12:50:46.897234+0000","last_fullsized":"2023-10-30T12:50:46.897234+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":493,"log_dups_size":0,"ondisk_log_size":493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:59:40.285830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":45183,"num_write":427,"num_write_kb":16635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"66'498","reported_seq":2645,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:24.458791+0000","last_change":"2023-10-30T12:38:17.111185+0000","last_active":"2023-10-30T12:51:24.458791+0000","last_peered":"2023-10-30T12:51:24.458791+0000","last_clean":"2023-10-30T12:51:24.458791+0000","last_became_active":"2023-10-30T12:28:50.949122+0000","last_became_peered":"2023-10-30T12:28:50.949122+0000","last_unstale":"2023-10-30T12:51:24.458791+0000","last_undegraded":"2023-10-30T12:51:24.458791+0000","last_fullsized":"2023-10-30T12:51:24.458791+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:32:08.636317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004393348,"stat_sum":{"num_bytes":12654257,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":35,"num_read_kb":65,"num_write":459,"num_write_kb":14113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"66'266","reported_seq":335,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:49.583265+0000","last_change":"2023-10-30T12:28:50.358700+0000","last_active":"2023-10-30T12:50:49.583265+0000","last_peered":"2023-10-30T12:50:49.583265+0000","last_clean":"2023-10-30T12:50:49.583265+0000","last_became_active":"2023-10-30T12:28:50.358310+0000","last_became_peered":"2023-10-30T12:28:50.358310+0000","last_unstale":"2023-10-30T12:50:49.583265+0000","last_undegraded":"2023-10-30T12:50:49.583265+0000","last_fullsized":"2023-10-30T12:50:49.583265+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":266,"log_dups_size":0,"ondisk_log_size":266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:23:30.910622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":5758,"num_write":222,"num_write_kb":4242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"65'512","reported_seq":2657,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.699965+0000","last_change":"2023-10-30T12:38:17.121187+0000","last_active":"2023-10-30T12:51:12.699965+0000","last_peered":"2023-10-30T12:51:12.699965+0000","last_clean":"2023-10-30T12:51:12.699965+0000","last_became_active":"2023-10-30T12:28:50.965626+0000","last_became_peered":"2023-10-30T12:28:50.965626+0000","last_unstale":"2023-10-30T12:51:12.699965+0000","last_undegraded":"2023-10-30T12:51:12.699965+0000","last_fullsized":"2023-10-30T12:51:12.699965+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":512,"log_dups_size":0,"ondisk_log_size":512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:44:24.255740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0079099229999999993,"stat_sum":{"num_bytes":29393213,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":33,"num_read_kb":4109,"num_write":475,"num_write_kb":37010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"66'399","reported_seq":470,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:47.791499+0000","last_change":"2023-10-30T12:28:50.395645+0000","last_active":"2023-10-30T12:50:47.791499+0000","last_peered":"2023-10-30T12:50:47.791499+0000","last_clean":"2023-10-30T12:50:47.791499+0000","last_became_active":"2023-10-30T12:28:50.395090+0000","last_became_peered":"2023-10-30T12:28:50.395090+0000","last_unstale":"2023-10-30T12:50:47.791499+0000","last_undegraded":"2023-10-30T12:50:47.791499+0000","last_fullsized":"2023-10-30T12:50:47.791499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:45:44.678383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582934,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":8195,"num_write":330,"num_write_kb":12489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"66'511","reported_seq":2673,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.700882+0000","last_change":"2023-10-30T12:38:17.104334+0000","last_active":"2023-10-30T12:51:12.700882+0000","last_peered":"2023-10-30T12:51:12.700882+0000","last_clean":"2023-10-30T12:51:12.700882+0000","last_became_active":"2023-10-30T12:28:50.929337+0000","last_became_peered":"2023-10-30T12:28:50.929337+0000","last_unstale":"2023-10-30T12:51:12.700882+0000","last_undegraded":"2023-10-30T12:51:12.700882+0000","last_fullsized":"2023-10-30T12:51:12.700882+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":511,"log_dups_size":0,"ondisk_log_size":511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:32:28.782456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023039890000000002,"stat_sum":{"num_bytes":14084554,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":50,"num_read_kb":18,"num_write":480,"num_write_kb":15463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"66'350","reported_seq":428,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.563637+0000","last_change":"2023-10-30T12:28:50.380474+0000","last_active":"2023-10-30T12:51:23.563637+0000","last_peered":"2023-10-30T12:51:23.563637+0000","last_clean":"2023-10-30T12:51:23.563637+0000","last_became_active":"2023-10-30T12:28:50.380247+0000","last_became_peered":"2023-10-30T12:28:50.380247+0000","last_unstale":"2023-10-30T12:51:23.563637+0000","last_undegraded":"2023-10-30T12:51:23.563637+0000","last_fullsized":"2023-10-30T12:51:23.563637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":350,"log_dups_size":0,"ondisk_log_size":350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:52:03.952617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":8272,"num_write":524,"num_write_kb":5251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"66'511","reported_seq":2653,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.729280+0000","last_change":"2023-10-30T12:38:17.331272+0000","last_active":"2023-10-30T12:51:12.729280+0000","last_peered":"2023-10-30T12:51:12.729280+0000","last_clean":"2023-10-30T12:51:12.729280+0000","last_became_active":"2023-10-30T12:28:50.973086+0000","last_became_peered":"2023-10-30T12:28:50.973086+0000","last_unstale":"2023-10-30T12:51:12.729280+0000","last_undegraded":"2023-10-30T12:51:12.729280+0000","last_fullsized":"2023-10-30T12:51:12.729280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":511,"log_dups_size":0,"ondisk_log_size":511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:53:48.263747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034133169999999999,"stat_sum":{"num_bytes":16794570,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":30,"num_read_kb":14,"num_write":487,"num_write_kb":22103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"66'198","reported_seq":272,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:25.415320+0000","last_change":"2023-10-30T12:28:50.386277+0000","last_active":"2023-10-30T12:51:25.415320+0000","last_peered":"2023-10-30T12:51:25.415320+0000","last_clean":"2023-10-30T12:51:25.415320+0000","last_became_active":"2023-10-30T12:28:50.385924+0000","last_became_peered":"2023-10-30T12:28:50.385924+0000","last_unstale":"2023-10-30T12:51:25.415320+0000","last_undegraded":"2023-10-30T12:51:25.415320+0000","last_fullsized":"2023-10-30T12:51:25.415320+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:35:03.837523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1958032,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":136,"num_write_kb":2002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'500","reported_seq":2655,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.717097+0000","last_change":"2023-10-30T12:38:17.643575+0000","last_active":"2023-10-30T12:51:12.717097+0000","last_peered":"2023-10-30T12:51:12.717097+0000","last_clean":"2023-10-30T12:51:12.717097+0000","last_became_active":"2023-10-30T12:28:50.929259+0000","last_became_peered":"2023-10-30T12:28:50.929259+0000","last_unstale":"2023-10-30T12:51:12.717097+0000","last_undegraded":"2023-10-30T12:51:12.717097+0000","last_fullsized":"2023-10-30T12:51:12.717097+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":500,"log_dups_size":0,"ondisk_log_size":500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:49:01.397563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029033230000000002,"stat_sum":{"num_bytes":8412687,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":43,"num_read_kb":22,"num_write":470,"num_write_kb":11464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"65'373","reported_seq":455,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:41.796144+0000","last_change":"2023-10-30T12:28:50.386426+0000","last_active":"2023-10-30T12:50:41.796144+0000","last_peered":"2023-10-30T12:50:41.796144+0000","last_clean":"2023-10-30T12:50:41.796144+0000","last_became_active":"2023-10-30T12:28:50.385305+0000","last_became_peered":"2023-10-30T12:28:50.385305+0000","last_unstale":"2023-10-30T12:50:41.796144+0000","last_undegraded":"2023-10-30T12:50:41.796144+0000","last_fullsized":"2023-10-30T12:50:41.796144+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":373,"log_dups_size":0,"ondisk_log_size":373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:13:05.376581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":37322,"num_write":310,"num_write_kb":12446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"66'520","reported_seq":2669,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:18.630606+0000","last_change":"2023-10-30T12:38:17.111500+0000","last_active":"2023-10-30T12:51:18.630606+0000","last_peered":"2023-10-30T12:51:18.630606+0000","last_clean":"2023-10-30T12:51:18.630606+0000","last_became_active":"2023-10-30T12:28:50.940846+0000","last_became_peered":"2023-10-30T12:28:50.940846+0000","last_unstale":"2023-10-30T12:51:18.630606+0000","last_undegraded":"2023-10-30T12:51:18.630606+0000","last_fullsized":"2023-10-30T12:51:18.630606+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":520,"log_dups_size":0,"ondisk_log_size":520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:27:33.103707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046016829999999996,"stat_sum":{"num_bytes":12660803,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":37,"num_read_kb":7,"num_write":492,"num_write_kb":18951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"66'441","reported_seq":518,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.687550+0000","last_change":"2023-10-30T12:28:50.392333+0000","last_active":"2023-10-30T12:51:23.687550+0000","last_peered":"2023-10-30T12:51:23.687550+0000","last_clean":"2023-10-30T12:51:23.687550+0000","last_became_active":"2023-10-30T12:28:50.392068+0000","last_became_peered":"2023-10-30T12:28:50.392068+0000","last_unstale":"2023-10-30T12:51:23.687550+0000","last_undegraded":"2023-10-30T12:51:23.687550+0000","last_fullsized":"2023-10-30T12:51:23.687550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":441,"log_dups_size":0,"ondisk_log_size":441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:47:24.674925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388698,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":19,"num_read_kb":11375,"num_write":386,"num_write_kb":8543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"66'523","reported_seq":2679,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.682847+0000","last_change":"2023-10-30T12:38:17.331599+0000","last_active":"2023-10-30T12:51:23.682847+0000","last_peered":"2023-10-30T12:51:23.682847+0000","last_clean":"2023-10-30T12:51:23.682847+0000","last_became_active":"2023-10-30T12:28:50.931898+0000","last_became_peered":"2023-10-30T12:28:50.931898+0000","last_unstale":"2023-10-30T12:51:23.682847+0000","last_undegraded":"2023-10-30T12:51:23.682847+0000","last_fullsized":"2023-10-30T12:51:23.682847+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":523,"log_dups_size":0,"ondisk_log_size":523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:49:23.804972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003570148,"stat_sum":{"num_bytes":4255498,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":44,"num_read_kb":18,"num_write":498,"num_write_kb":10461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"66'553","reported_seq":626,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:24.395115+0000","last_change":"2023-10-30T12:28:50.365055+0000","last_active":"2023-10-30T12:51:24.395115+0000","last_peered":"2023-10-30T12:51:24.395115+0000","last_clean":"2023-10-30T12:51:24.395115+0000","last_became_active":"2023-10-30T12:28:50.364797+0000","last_became_peered":"2023-10-30T12:28:50.364797+0000","last_unstale":"2023-10-30T12:51:24.395115+0000","last_undegraded":"2023-10-30T12:51:24.395115+0000","last_fullsized":"2023-10-30T12:51:24.395115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":553,"log_dups_size":0,"ondisk_log_size":553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:06:23.067895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":10,"num_read_kb":8489,"num_write":689,"num_write_kb":13252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"66'528","reported_seq":2689,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.685808+0000","last_change":"2023-10-30T12:38:17.331931+0000","last_active":"2023-10-30T12:51:23.685808+0000","last_peered":"2023-10-30T12:51:23.685808+0000","last_clean":"2023-10-30T12:51:23.685808+0000","last_became_active":"2023-10-30T12:28:50.935657+0000","last_became_peered":"2023-10-30T12:28:50.935657+0000","last_unstale":"2023-10-30T12:51:23.685808+0000","last_undegraded":"2023-10-30T12:51:23.685808+0000","last_fullsized":"2023-10-30T12:51:23.685808+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":528,"log_dups_size":0,"ondisk_log_size":528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:42:39.657789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038543219999999999,"stat_sum":{"num_bytes":16855474,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":49,"num_read_kb":4112,"num_write":496,"num_write_kb":17584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"66'244","reported_seq":324,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.643888+0000","last_change":"2023-10-30T12:28:50.393465+0000","last_active":"2023-10-30T12:51:12.643888+0000","last_peered":"2023-10-30T12:51:12.643888+0000","last_clean":"2023-10-30T12:51:12.643888+0000","last_became_active":"2023-10-30T12:28:50.392807+0000","last_became_peered":"2023-10-30T12:28:50.392807+0000","last_unstale":"2023-10-30T12:51:12.643888+0000","last_undegraded":"2023-10-30T12:51:12.643888+0000","last_fullsized":"2023-10-30T12:51:12.643888+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":244,"log_dups_size":0,"ondisk_log_size":244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:35:37.976956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":24596,"num_write":178,"num_write_kb":8294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"65'481","reported_seq":2631,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.730761+0000","last_change":"2023-10-30T12:38:17.325460+0000","last_active":"2023-10-30T12:51:12.730761+0000","last_peered":"2023-10-30T12:51:12.730761+0000","last_clean":"2023-10-30T12:51:12.730761+0000","last_became_active":"2023-10-30T12:28:50.954291+0000","last_became_peered":"2023-10-30T12:28:50.954291+0000","last_unstale":"2023-10-30T12:51:12.730761+0000","last_undegraded":"2023-10-30T12:51:12.730761+0000","last_fullsized":"2023-10-30T12:51:12.730761+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":481,"log_dups_size":0,"ondisk_log_size":481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:58:33.115877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047974480000000002,"stat_sum":{"num_bytes":12640290,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":38,"num_read_kb":15,"num_write":450,"num_write_kb":14582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"65'505","reported_seq":2666,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.715029+0000","last_change":"2023-10-30T12:38:17.120085+0000","last_active":"2023-10-30T12:51:12.715029+0000","last_peered":"2023-10-30T12:51:12.715029+0000","last_clean":"2023-10-30T12:51:12.715029+0000","last_became_active":"2023-10-30T12:28:50.962817+0000","last_became_peered":"2023-10-30T12:28:50.962817+0000","last_unstale":"2023-10-30T12:51:12.715029+0000","last_undegraded":"2023-10-30T12:51:12.715029+0000","last_fullsized":"2023-10-30T12:51:12.715029+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":505,"log_dups_size":0,"ondisk_log_size":505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:14:10.476055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070898599999999999,"stat_sum":{"num_bytes":8463933,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":50,"num_read_kb":25,"num_write":479,"num_write_kb":14576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"65'173","reported_seq":249,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.684872+0000","last_change":"2023-10-30T12:28:50.402179+0000","last_active":"2023-10-30T12:51:12.684872+0000","last_peered":"2023-10-30T12:51:12.684872+0000","last_clean":"2023-10-30T12:51:12.684872+0000","last_became_active":"2023-10-30T12:28:50.401677+0000","last_became_peered":"2023-10-30T12:28:50.401677+0000","last_unstale":"2023-10-30T12:51:12.684872+0000","last_undegraded":"2023-10-30T12:51:12.684872+0000","last_fullsized":"2023-10-30T12:51:12.684872+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:39:44.432214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":20498,"num_write":107,"num_write_kb":8247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"66'474","reported_seq":2629,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:22.381301+0000","last_change":"2023-10-30T12:38:17.119058+0000","last_active":"2023-10-30T12:51:22.381301+0000","last_peered":"2023-10-30T12:51:22.381301+0000","last_clean":"2023-10-30T12:51:22.381301+0000","last_became_active":"2023-10-30T12:28:50.967591+0000","last_became_peered":"2023-10-30T12:28:50.967591+0000","last_unstale":"2023-10-30T12:51:22.381301+0000","last_undegraded":"2023-10-30T12:51:22.381301+0000","last_fullsized":"2023-10-30T12:51:22.381301+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":474,"log_dups_size":0,"ondisk_log_size":474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:15:53.272647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081968709999999997,"stat_sum":{"num_bytes":10507344,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":44,"num_read_kb":1610,"num_write":436,"num_write_kb":22884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"66'363","reported_seq":450,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:22.402277+0000","last_change":"2023-10-30T12:28:50.392617+0000","last_active":"2023-10-30T12:51:22.402277+0000","last_peered":"2023-10-30T12:51:22.402277+0000","last_clean":"2023-10-30T12:51:22.402277+0000","last_became_active":"2023-10-30T12:28:50.392021+0000","last_became_peered":"2023-10-30T12:28:50.392021+0000","last_unstale":"2023-10-30T12:51:22.402277+0000","last_undegraded":"2023-10-30T12:51:22.402277+0000","last_fullsized":"2023-10-30T12:51:22.402277+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:52:30.621125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":57384,"num_write":298,"num_write_kb":16576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"65'482","reported_seq":2638,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.744064+0000","last_change":"2023-10-30T12:38:17.643950+0000","last_active":"2023-10-30T12:51:12.744064+0000","last_peered":"2023-10-30T12:51:12.744064+0000","last_clean":"2023-10-30T12:51:12.744064+0000","last_became_active":"2023-10-30T12:28:50.967186+0000","last_became_peered":"2023-10-30T12:28:50.967186+0000","last_unstale":"2023-10-30T12:51:12.744064+0000","last_undegraded":"2023-10-30T12:51:12.744064+0000","last_fullsized":"2023-10-30T12:51:12.744064+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":482,"log_dups_size":0,"ondisk_log_size":482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:48:20.159300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003136738,"stat_sum":{"num_bytes":18043,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":17,"num_write":445,"num_write_kb":3039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"65'227","reported_seq":303,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:46.350266+0000","last_change":"2023-10-30T12:28:50.363230+0000","last_active":"2023-10-30T12:50:46.350266+0000","last_peered":"2023-10-30T12:50:46.350266+0000","last_clean":"2023-10-30T12:50:46.350266+0000","last_became_active":"2023-10-30T12:28:50.362929+0000","last_became_peered":"2023-10-30T12:28:50.362929+0000","last_unstale":"2023-10-30T12:50:46.350266+0000","last_undegraded":"2023-10-30T12:50:46.350266+0000","last_fullsized":"2023-10-30T12:50:46.350266+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:41:27.760578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":9,"num_read_kb":32768,"num_write":162,"num_write_kb":8341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"66'542","reported_seq":2685,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.687065+0000","last_change":"2023-10-30T12:38:17.331548+0000","last_active":"2023-10-30T12:51:23.687065+0000","last_peered":"2023-10-30T12:51:23.687065+0000","last_clean":"2023-10-30T12:51:23.687065+0000","last_became_active":"2023-10-30T12:28:50.937458+0000","last_became_peered":"2023-10-30T12:28:50.937458+0000","last_unstale":"2023-10-30T12:51:23.687065+0000","last_undegraded":"2023-10-30T12:51:23.687065+0000","last_fullsized":"2023-10-30T12:51:23.687065+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":542,"log_dups_size":0,"ondisk_log_size":542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:25:59.947416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003544045,"stat_sum":{"num_bytes":16835848,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":30,"num_read_kb":11,"num_write":511,"num_write_kb":19769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"66'443","reported_seq":537,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:49.583519+0000","last_change":"2023-10-30T12:28:50.386782+0000","last_active":"2023-10-30T12:50:49.583519+0000","last_peered":"2023-10-30T12:50:49.583519+0000","last_clean":"2023-10-30T12:50:49.583519+0000","last_became_active":"2023-10-30T12:28:50.386522+0000","last_became_peered":"2023-10-30T12:28:50.386522+0000","last_unstale":"2023-10-30T12:50:49.583519+0000","last_undegraded":"2023-10-30T12:50:49.583519+0000","last_fullsized":"2023-10-30T12:50:49.583519+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":443,"log_dups_size":0,"ondisk_log_size":443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:44:03.517641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583494,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":64,"num_read_kb":24634,"num_write":377,"num_write_kb":12490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":759200777,"num_objects":915,"num_object_clones":0,"num_object_copies":1830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":915,"num_whiteouts":0,"num_read":2811,"num_read_kb":100236,"num_write":31405,"num_write_kb":1046937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1521901568,"data_stored":1518401554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33391,"ondisk_log_size":33391,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":303239465,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1432,"num_read_kb":712811,"num_write":10434,"num_write_kb":304769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":78,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":606633984,"data_stored":606478930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11693,"ondisk_log_size":11693,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542422,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":389288,"kb_used_data":322016,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93982552,"statfs":{"total":96636764160,"available":96238133248,"internally_reserved":0,"allocated":329744384,"data_stored":328817986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":68876747},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.78300000000000003,"15min":0.81799999999999995},"min":{"1min":0.52500000000000002,"5min":0.45900000000000002,"15min":0.40000000000000002},"max":{"1min":1.093,"5min":1.2050000000000001,"15min":1.595},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.78400000000000003,"15min":0.84199999999999997},"min":{"1min":0.49099999999999999,"5min":0.441,"15min":0.40999999999999998},"max":{"1min":1.113,"5min":1.302,"15min":1.4830000000000001},"last":0.57299999999999995}]},{"osd":1,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.85099999999999998,"15min":0.88800000000000001},"min":{"1min":0.57199999999999995,"5min":0.52700000000000002,"15min":0.441},"max":{"1min":1.2709999999999999,"5min":3.4329999999999998,"15min":3.4329999999999998},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.85399999999999998,"15min":0.88500000000000001},"min":{"1min":0.56999999999999995,"5min":0.505,"15min":0.505},"max":{"1min":1.1419999999999999,"5min":1.383,"15min":1.3919999999999999},"last":0.90100000000000002}]},{"osd":2,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.89000000000000001,"15min":0.92200000000000004},"min":{"1min":0.58899999999999997,"5min":0.55100000000000005,"15min":0.45900000000000002},"max":{"1min":1.163,"5min":3.4820000000000002,"15min":3.4820000000000002},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.877,"15min":0.83699999999999997},"min":{"1min":0.65000000000000002,"5min":0.52800000000000002,"15min":0.47099999999999997},"max":{"1min":1.2330000000000001,"5min":1.4570000000000001,"15min":1.6639999999999999},"last":0.73799999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":1.0069999999999999,"15min":0.96899999999999997},"min":{"1min":0.61299999999999999,"5min":0.60999999999999999,"15min":0.45500000000000002},"max":{"1min":1.1439999999999999,"5min":3.5190000000000001,"15min":3.5190000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.95599999999999996,"15min":0.99399999999999999},"min":{"1min":0.72899999999999998,"5min":0.54500000000000004,"15min":0.441},"max":{"1min":1.3220000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.94399999999999995}]},{"osd":4,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.97599999999999998,"15min":1.002},"min":{"1min":0.63800000000000001,"5min":0.42499999999999999,"15min":0.34399999999999997},"max":{"1min":1.2070000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":1.028},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.999,"15min":1.042},"min":{"1min":0.61799999999999999,"5min":0.45000000000000001,"15min":0.33800000000000002},"max":{"1min":1.1910000000000001,"5min":1.746,"15min":1.746},"last":1.0489999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.016,"15min":0.96099999999999997},"min":{"1min":0.76700000000000002,"5min":0.53700000000000003,"15min":0.43099999999999999},"max":{"1min":1.2949999999999999,"5min":1.6160000000000001,"15min":1.7549999999999999},"last":1.081},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.117,"15min":1.0680000000000001},"min":{"1min":0.71499999999999997,"5min":0.57899999999999996,"15min":0.35999999999999999},"max":{"1min":1.2370000000000001,"5min":2.1600000000000001,"15min":2.1600000000000001},"last":1.1990000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.141,"15min":1.123},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.39000000000000001},"max":{"1min":1.216,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.129},{"interface":"front","average":{"1min":1,"5min":1.1259999999999999,"15min":1.167},"min":{"1min":0.68200000000000005,"5min":0.56299999999999994,"15min":0.47199999999999998},"max":{"1min":1.3300000000000001,"5min":3.8140000000000001,"15min":3.8140000000000001},"last":0.998}]}]},{"osd":6,"up_from":14,"seq":60129542422,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":296948,"kb_used_data":233900,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94074892,"statfs":{"total":96636764160,"available":96332689408,"internally_reserved":0,"allocated":239513600,"data_stored":238814643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":64551368},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.83299999999999996,"15min":0.85999999999999999},"min":{"1min":0.52900000000000003,"5min":0.46600000000000003,"15min":0.35599999999999998},"max":{"1min":1.155,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.77000000000000002,"15min":0.78800000000000003},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.42999999999999999},"max":{"1min":1.05,"5min":1.109,"15min":1.214},"last":0.92400000000000004}]},{"osd":1,"last update":"Mon Oct 30 12:50:23 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.79900000000000004,"15min":0.81299999999999994},"min":{"1min":0.57299999999999995,"5min":0.42299999999999999,"15min":0.317},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.3140000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.89000000000000001,"15min":0.88800000000000001},"min":{"1min":0.55800000000000005,"5min":0.51100000000000001,"15min":0.40999999999999998},"max":{"1min":1.149,"5min":1.3620000000000001,"15min":1.423},"last":0.78900000000000003}]},{"osd":2,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.93600000000000005,"15min":0.97499999999999998},"min":{"1min":0.495,"5min":0.495,"15min":0.38800000000000001},"max":{"1min":1.373,"5min":1.47,"15min":4.3360000000000003},"last":1.04},{"interface":"front","average":{"1min":1.0409999999999999,"5min":0.97999999999999998,"15min":1.0089999999999999},"min":{"1min":0.65000000000000002,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.502,"5min":1.502,"15min":1.5629999999999999},"last":1.2390000000000001}]},{"osd":3,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.996,"15min":1.042},"min":{"1min":0.64700000000000002,"5min":0.59099999999999997,"15min":0.58999999999999997},"max":{"1min":1.3340000000000001,"5min":1.5509999999999999,"15min":1.6459999999999999},"last":0.83999999999999997},{"interface":"front","average":{"1min":1.02,"5min":1.0680000000000001,"15min":1.101},"min":{"1min":0.72299999999999998,"5min":0.63300000000000001,"15min":0.59699999999999998},"max":{"1min":1.417,"5min":1.552,"15min":1.7969999999999999},"last":1.095}]},{"osd":4,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.149,"15min":1.075},"min":{"1min":0.72599999999999998,"5min":0.69899999999999995,"15min":0.51500000000000001},"max":{"1min":1.585,"5min":1.6499999999999999,"15min":1.7649999999999999},"last":1.5229999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.89400000000000002,"15min":0.96499999999999997},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.42099999999999999},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.881},"last":1.2030000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":1.0720000000000001,"15min":1.087},"min":{"1min":0.54100000000000004,"5min":0.53200000000000003,"15min":0.48399999999999999},"max":{"1min":1.2849999999999999,"5min":1.75,"15min":2.2770000000000001},"last":1.27},{"interface":"front","average":{"1min":1.056,"5min":1.159,"15min":1.0649999999999999},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.46999999999999997},"max":{"1min":1.4610000000000001,"5min":1.6950000000000001,"15min":4.7850000000000001},"last":1.1359999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.036,"15min":1.0029999999999999},"min":{"1min":0.68799999999999994,"5min":0.64100000000000001,"15min":0.46700000000000003},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.825},"last":1.417},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.0629999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.48999999999999999},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.895},"last":1.3169999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542422,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":338824,"kb_used_data":271552,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94033016,"statfs":{"total":96636764160,"available":96289808384,"internally_reserved":0,"allocated":278069248,"data_stored":277156365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.78400000000000003,"15min":0.80300000000000005},"min":{"1min":0.56299999999999994,"5min":0.33900000000000002,"15min":0.29399999999999998},"max":{"1min":1.1970000000000001,"5min":1.238,"15min":2.4729999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.79700000000000004,"15min":0.80400000000000005},"min":{"1min":0.61199999999999999,"5min":0.39200000000000002,"15min":0.33800000000000002},"max":{"1min":1.2390000000000001,"5min":1.3009999999999999,"15min":1.8759999999999999},"last":0.84299999999999997}]},{"osd":1,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.84199999999999997,"15min":0.84399999999999997},"min":{"1min":0.57399999999999995,"5min":0.40999999999999998,"15min":0.38900000000000001},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":3.0190000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.58499999999999996,"5min":0.44800000000000001,"15min":0.373},"max":{"1min":1.264,"5min":1.276,"15min":1.4870000000000001},"last":0.91800000000000004}]},{"osd":2,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.88400000000000001,"15min":0.84699999999999998},"min":{"1min":0.67300000000000004,"5min":0.55400000000000005,"15min":0.38400000000000001},"max":{"1min":1.157,"5min":1.264,"15min":2.2949999999999999},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.81699999999999995,"15min":0.876},"min":{"1min":0.63600000000000001,"5min":0.434,"15min":0.434},"max":{"1min":1.1140000000000001,"5min":1.171,"15min":1.5800000000000001},"last":0.88600000000000001}]},{"osd":3,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.91800000000000004,"15min":0.93700000000000006},"min":{"1min":0.57199999999999995,"5min":0.53800000000000003,"15min":0.47899999999999998},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.756},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.93600000000000005,"15min":0.94599999999999995},"min":{"1min":0.755,"5min":0.498,"15min":0.41399999999999998},"max":{"1min":1.204,"5min":1.4119999999999999,"15min":2.3679999999999999},"last":1.0680000000000001}]},{"osd":4,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.84499999999999997,"15min":0.90900000000000003},"min":{"1min":0.47499999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.304,"5min":1.304,"15min":3.1190000000000002},"last":0.81100000000000005},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.083,"15min":1.091},"min":{"1min":0.59499999999999997,"5min":0.58699999999999997,"15min":0.50900000000000001},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":2.645},"last":1.026}]},{"osd":6,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.038,"15min":1.042},"min":{"1min":0.51100000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.466,"5min":1.466,"15min":3.1859999999999999},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.056,"15min":1.1060000000000001},"min":{"1min":0.67000000000000004,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.3260000000000001,"5min":1.47,"15min":3.371},"last":1}]},{"osd":7,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.0609999999999999,"15min":1.0800000000000001},"min":{"1min":0.65200000000000002,"5min":0.51400000000000001,"15min":0.47299999999999998},"max":{"1min":1.494,"5min":1.508,"15min":3.5310000000000001},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.282,"5min":1.1970000000000001,"15min":1.2310000000000001},"min":{"1min":0.96799999999999997,"5min":0.64700000000000002,"15min":0.61299999999999999},"max":{"1min":1.605,"5min":1.605,"15min":3.6930000000000001},"last":1.3160000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542421,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369848,"kb_used_data":294256,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94001992,"statfs":{"total":96636764160,"available":96258039808,"internally_reserved":0,"allocated":301318144,"data_stored":300442399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:15 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.81799999999999995,"15min":0.92600000000000005},"min":{"1min":0.48099999999999998,"5min":0.34799999999999998,"15min":0.31},"max":{"1min":1.204,"5min":1.2509999999999999,"15min":11.273},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.92400000000000004,"15min":0.98799999999999999},"min":{"1min":0.58099999999999996,"5min":0.44,"15min":0.44},"max":{"1min":1.411,"5min":1.4590000000000001,"15min":7.1289999999999996},"last":0.81899999999999995}]},{"osd":1,"last update":"Mon Oct 30 12:51:15 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.81499999999999995,"15min":0.90500000000000003},"min":{"1min":0.52800000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0049999999999999,"5min":5.1130000000000004,"15min":6.9199999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.871,"5min":0.84199999999999997,"15min":0.90000000000000002},"min":{"1min":0.63900000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.246,"5min":1.246,"15min":4.75},"last":0.76900000000000002}]},{"osd":2,"last update":"Mon Oct 30 12:51:15 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.84199999999999997,"15min":0.84899999999999998},"min":{"1min":0.51700000000000002,"5min":0.41899999999999998,"15min":0.36899999999999999},"max":{"1min":1.175,"5min":1.21,"15min":2.6970000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.996,"5min":0.91200000000000003,"15min":1.0569999999999999},"min":{"1min":0.60999999999999999,"5min":0.54400000000000004,"15min":0.52900000000000003},"max":{"1min":1.361,"5min":1.361,"15min":18.917999999999999},"last":0.999}]},{"osd":3,"last update":"Mon Oct 30 12:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":0.96599999999999997,"15min":0.997},"min":{"1min":0.71399999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.258,"5min":5.1719999999999997,"15min":7.1879999999999997},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.90800000000000003,"15min":1},"min":{"1min":0.57399999999999995,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.125,"5min":1.3600000000000001,"15min":8.3360000000000003},"last":0.71899999999999997}]},{"osd":5,"last update":"Mon Oct 30 12:51:15 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.88300000000000001,"15min":1.002},"min":{"1min":0.56299999999999994,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.159,"5min":1.3300000000000001,"15min":6.0759999999999996},"last":0.92500000000000004},{"interface":"front","average":{"1min":1.119,"5min":1.105,"15min":1.1890000000000001},"min":{"1min":0.89500000000000002,"5min":0.71899999999999997,"15min":0.52600000000000002},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":18.994},"last":1.0329999999999999}]},{"osd":6,"last update":"Mon Oct 30 12:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.0880000000000001,"15min":1.093},"min":{"1min":0.70199999999999996,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.429,"5min":1.429,"15min":6.1580000000000004},"last":1.115},{"interface":"front","average":{"1min":1.016,"5min":1.0169999999999999,"15min":1.1240000000000001},"min":{"1min":0.629,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.28,"5min":5.218,"15min":7.0549999999999997},"last":0.95999999999999996}]},{"osd":7,"last update":"Mon Oct 30 12:51:15 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.0980000000000001,"15min":1.258},"min":{"1min":0.75800000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.4339999999999999,"5min":5.7160000000000002,"15min":18.015999999999998},"last":1.091},{"interface":"front","average":{"1min":1.236,"5min":1.2050000000000001,"15min":1.2769999999999999},"min":{"1min":1.093,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.4530000000000001,"5min":5.6609999999999996,"15min":19.052},"last":1.226}]}]},{"osd":3,"up_from":14,"seq":60129542422,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388864,"kb_used_data":309112,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93982976,"statfs":{"total":96636764160,"available":96238567424,"internally_reserved":0,"allocated":316530688,"data_stored":315623973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":81656264},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.39900000000000002,"15min":0.41399999999999998},"min":{"1min":0.29399999999999998,"5min":0.23699999999999999,"15min":0.23400000000000001},"max":{"1min":0.58999999999999997,"5min":0.77900000000000003,"15min":1.6899999999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51600000000000001,"15min":0.53300000000000003},"min":{"1min":0.40000000000000002,"5min":0.34300000000000003,"15min":0.32700000000000001},"max":{"1min":0.746,"5min":0.871,"15min":1.728},"last":0.55200000000000005}]},{"osd":1,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.42899999999999999,"15min":0.42999999999999999},"min":{"1min":0.309,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":1.2909999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.496,"15min":0.50600000000000001},"min":{"1min":0.34499999999999997,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.64600000000000002,"5min":0.80600000000000005,"15min":1.3260000000000001},"last":0.501}]},{"osd":2,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.496,"15min":0.502},"min":{"1min":0.37,"5min":0.30099999999999999,"15min":0.29299999999999998},"max":{"1min":0.69899999999999995,"5min":0.78600000000000003,"15min":1.099},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55900000000000005,"15min":0.56699999999999995},"min":{"1min":0.41999999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73599999999999999,"5min":0.82599999999999996,"15min":1.2509999999999999},"last":0.53100000000000003}]},{"osd":4,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.85199999999999998,"15min":0.83099999999999996},"min":{"1min":0.63900000000000001,"5min":0.55400000000000005,"15min":0.40400000000000003},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":2.202},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.82999999999999996,"15min":0.86499999999999999},"min":{"1min":0.70599999999999996,"5min":0.48499999999999999,"15min":0.46800000000000003},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":2.4340000000000002},"last":0.89100000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.81499999999999995,"15min":0.86299999999999999},"min":{"1min":0.66200000000000003,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.169,"5min":1.169,"15min":14.103999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.875,"5min":0.876,"15min":0.86499999999999999},"min":{"1min":0.63900000000000001,"5min":0.55000000000000004,"15min":0.499},"max":{"1min":1.2170000000000001,"5min":1.2869999999999999,"15min":4.4859999999999998},"last":0.85699999999999998}]},{"osd":6,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.85299999999999998,"15min":0.83499999999999996},"min":{"1min":0.64700000000000002,"5min":0.56999999999999995,"15min":0.53100000000000003},"max":{"1min":1.087,"5min":1.2450000000000001,"15min":2.2330000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.87,"15min":0.82899999999999996},"min":{"1min":0.77800000000000002,"5min":0.54100000000000004,"15min":0.45100000000000001},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":4.4409999999999998},"last":0.86699999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.875,"15min":0.86499999999999999},"min":{"1min":0.67500000000000004,"5min":0.57299999999999995,"15min":0.57099999999999995},"max":{"1min":1.1499999999999999,"5min":1.2509999999999999,"15min":3.5459999999999998},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.86399999999999999,"15min":0.84699999999999998},"min":{"1min":0.65700000000000003,"5min":0.57399999999999995,"15min":0.45700000000000002},"max":{"1min":1.1259999999999999,"5min":1.2230000000000001,"15min":6.1360000000000001},"last":0.91800000000000004}]}]},{"osd":2,"up_from":14,"seq":60129542422,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":199872,"kb_used_data":142328,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94171968,"statfs":{"total":96636764160,"available":96432095232,"internally_reserved":0,"allocated":145743872,"data_stored":145027826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.45100000000000001,"15min":0.441},"min":{"1min":0.25,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.59499999999999997,"5min":0.69399999999999995,"15min":0.81200000000000006},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.5,"15min":0.51100000000000001},"min":{"1min":0.37,"5min":0.33300000000000002,"15min":0.31900000000000001},"max":{"1min":0.63,"5min":0.71299999999999997,"15min":1.298},"last":0.47099999999999997}]},{"osd":1,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.47899999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.60199999999999998,"5min":0.65500000000000003,"15min":0.83699999999999997},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.52500000000000002,"15min":0.504},"min":{"1min":0.38900000000000001,"5min":0.33700000000000002,"15min":0.29999999999999999},"max":{"1min":0.61699999999999999,"5min":0.74099999999999999,"15min":1.2250000000000001},"last":0.53800000000000003}]},{"osd":3,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.47399999999999998,"15min":0.54800000000000004},"min":{"1min":0.315,"5min":0.311,"15min":0.311},"max":{"1min":0.58599999999999997,"5min":0.75,"15min":1.673},"last":0.498},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57299999999999995,"15min":0.57399999999999995},"min":{"1min":0.41799999999999998,"5min":0.40200000000000002,"15min":0.35699999999999998},"max":{"1min":0.67900000000000005,"5min":0.79100000000000004,"15min":1.597},"last":0.59499999999999997}]},{"osd":4,"last update":"Mon Oct 30 12:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.74299999999999999,"15min":0.82599999999999996},"min":{"1min":0.47499999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.91100000000000003,"5min":1.0569999999999999,"15min":1.377},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.755,"15min":0.79200000000000004},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.998,"5min":1.0640000000000001,"15min":1.4159999999999999},"last":1.0169999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.84099999999999997,"15min":0.84799999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.093,"5min":1.2430000000000001,"15min":1.78},"last":1.002},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91400000000000003},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.1060000000000001,"5min":1.2729999999999999,"15min":1.7430000000000001},"last":1.0309999999999999}]},{"osd":6,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.86199999999999999,"15min":0.85799999999999998},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.60299999999999998},"max":{"1min":1.095,"5min":1.177,"15min":1.3799999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.86599999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.53300000000000003},"max":{"1min":1.097,"5min":1.1799999999999999,"15min":1.8109999999999999},"last":0.93300000000000005}]},{"osd":7,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.82899999999999996,"15min":0.83099999999999996},"min":{"1min":0.58799999999999997,"5min":0.58299999999999996,"15min":0.48799999999999999},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.4159999999999999},"last":1.0429999999999999},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.85399999999999998,"15min":0.83299999999999996},"min":{"1min":0.75900000000000001,"5min":0.51800000000000002,"15min":0.50800000000000001},"max":{"1min":1.206,"5min":1.206,"15min":1.3049999999999999},"last":1.0780000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542422,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":278984,"kb_used_data":233920,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94092856,"statfs":{"total":96636764160,"available":96351084544,"internally_reserved":0,"allocated":239534080,"data_stored":238814180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:50:34 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.40400000000000003,"15min":0.45400000000000001},"min":{"1min":0.28399999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.63600000000000001,"5min":0.65200000000000002,"15min":11.958},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.495,"15min":0.55700000000000005},"min":{"1min":0.34699999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.73599999999999999,"5min":0.73799999999999999,"15min":12.077},"last":0.53800000000000003}]},{"osd":2,"last update":"Mon Oct 30 12:50:34 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45800000000000002,"15min":0.46000000000000002},"min":{"1min":0.29699999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.121,"5min":1.121,"15min":1.623},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52500000000000002,"15min":0.53300000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.31900000000000001},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.5389999999999999},"last":0.60599999999999998}]},{"osd":3,"last update":"Mon Oct 30 12:50:34 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55400000000000005,"15min":0.56699999999999995},"min":{"1min":0.378,"5min":0.31,"15min":0.31},"max":{"1min":1.216,"5min":1.337,"15min":3.694},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.498,"15min":0.54700000000000004},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.66900000000000004,"5min":0.77200000000000002,"15min":13.199999999999999},"last":0.49299999999999999}]},{"osd":4,"last update":"Mon Oct 30 12:50:34 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.82699999999999996,"15min":0.83199999999999996},"min":{"1min":0.54700000000000004,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1519999999999999,"5min":1.3140000000000001,"15min":2.419},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.871,"15min":0.86699999999999999},"min":{"1min":0.66300000000000003,"5min":0.60899999999999999,"15min":0.52100000000000002},"max":{"1min":1.095,"5min":1.103,"15min":2.282},"last":0.91700000000000004}]},{"osd":5,"last update":"Mon Oct 30 12:50:34 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.80000000000000004,"15min":0.81599999999999995},"min":{"1min":0.61699999999999999,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.01,"5min":1.343,"15min":1.343},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.79000000000000004,"15min":0.85599999999999998},"min":{"1min":0.59099999999999997,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.115,"5min":1.29,"15min":1.29},"last":0.77000000000000002}]},{"osd":6,"last update":"Mon Oct 30 12:50:34 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82799999999999996,"15min":0.79800000000000004},"min":{"1min":0.56399999999999995,"5min":0.52800000000000002,"15min":0.51200000000000001},"max":{"1min":1.3660000000000001,"5min":1.371,"15min":1.371},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.89400000000000002,"15min":0.86699999999999999},"min":{"1min":0.67400000000000004,"5min":0.498,"15min":0.498},"max":{"1min":1.417,"5min":1.417,"15min":2.464},"last":0.97099999999999997}]},{"osd":7,"last update":"Mon Oct 30 12:50:34 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.81100000000000005,"15min":0.83199999999999996},"min":{"1min":0.61899999999999999,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0349999999999999,"5min":1.1659999999999999,"15min":2.5169999999999999},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.875,"5min":0.88200000000000001,"15min":0.90700000000000003},"min":{"1min":0.76900000000000002,"5min":0.61899999999999999,"15min":0.58399999999999996},"max":{"1min":1.133,"5min":1.133,"15min":2.5510000000000002},"last":0.78700000000000003}]}]},{"osd":0,"up_from":14,"seq":60129542422,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350128,"kb_used_data":278696,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94021712,"statfs":{"total":96636764160,"available":96278233088,"internally_reserved":0,"allocated":285384704,"data_stored":284463559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 12:50:26 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.44400000000000001,"15min":0.44900000000000001},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.93700000000000006},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50900000000000001,"15min":0.504},"min":{"1min":0.35299999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.67600000000000005,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.56100000000000005}]},{"osd":2,"last update":"Mon Oct 30 12:50:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.45600000000000002,"15min":0.504},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.629,"5min":0.64900000000000002,"15min":0.97199999999999998},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.55200000000000005,"15min":0.52800000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.27400000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":1.0149999999999999},"last":0.57699999999999996}]},{"osd":3,"last update":"Mon Oct 30 12:50:26 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.46899999999999997,"15min":0.51400000000000001},"min":{"1min":0.30499999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.73499999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.48599999999999999,"15min":0.496},"min":{"1min":0.316,"5min":0.31,"15min":0.31},"max":{"1min":0.74199999999999999,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.59299999999999997}]},{"osd":4,"last update":"Mon Oct 30 12:50:26 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86599999999999999,"15min":0.85499999999999998},"min":{"1min":0.70899999999999996,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.127,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.88500000000000001,"15min":0.85699999999999998},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.48299999999999998},"max":{"1min":1.143,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.92400000000000004}]},{"osd":5,"last update":"Mon Oct 30 12:50:26 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.82299999999999995,"15min":0.84199999999999997},"min":{"1min":0.60499999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.095,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.83299999999999996,"15min":0.87},"min":{"1min":0.58699999999999997,"5min":0.53200000000000003,"15min":0.52800000000000002},"max":{"1min":1.889,"5min":1.889,"15min":1.889},"last":0.89000000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:50:26 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.78400000000000003,"15min":0.79500000000000004},"min":{"1min":0.58299999999999996,"5min":0.56299999999999994,"15min":0.50900000000000001},"max":{"1min":1.0409999999999999,"5min":1.1990000000000001,"15min":2.6480000000000001},"last":1.071},{"interface":"front","average":{"1min":0.871,"5min":0.79300000000000004,"15min":0.82099999999999995},"min":{"1min":0.61199999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0409999999999999,"5min":1.1559999999999999,"15min":5.0099999999999998},"last":1.0249999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:50:26 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.87,"15min":0.88200000000000001},"min":{"1min":0.68899999999999995,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.155,"5min":1.401,"15min":3.3359999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.92400000000000004,"15min":0.94399999999999995},"min":{"1min":0.75700000000000001,"5min":0.59699999999999998,"15min":0.56299999999999994},"max":{"1min":1.1890000000000001,"5min":1.3500000000000001,"15min":3.3580000000000001},"last":1.0069999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62930944,"data_stored":62915220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":83910656,"data_stored":83887024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":37122048,"data_stored":37098251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82608128,"data_stored":82586990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":76099584,"data_stored":76085923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":62935040,"data_stored":62915938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":89997312,"data_stored":89978920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":111030272,"data_stored":111010664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":221097984,"data_stored":220570894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":154857472,"data_stored":154539460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":107859968,"data_stored":107542498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":233156608,"data_stored":232649287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":224452608,"data_stored":223968780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":214372352,"data_stored":213853350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":148750336,"data_stored":148448027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":217354240,"data_stored":216829258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:51:31.594 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-30T12:51:31.595 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:51:31.768 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.770+0000 7f92d21aa640 1 Processor -- start 2023-10-30T12:51:31.769 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.770+0000 7f92d21aa640 1 -- start start 2023-10-30T12:51:31.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.770+0000 7f92d21aa640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f92cc1538a0 0x7f92cc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:31.770 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.770+0000 7f92d21aa640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f92cc0cf160 0x7f92cc0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:31.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.770+0000 7f92d21aa640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92cc0cb800 con 0x7f92cc0ca0e0 2023-10-30T12:51:31.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.770+0000 7f92d21aa640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92cc15dc30 con 0x7f92cc0cf160 2023-10-30T12:51:31.771 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.770+0000 7f92d21aa640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92cc0cb330 con 0x7f92cc1538a0 2023-10-30T12:51:31.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.770+0000 7f92cbfff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f92cc0cf160 0x7f92cc0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:31.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.770+0000 7f92cbfff640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f92cc0cf160 0x7f92cc0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47308/0 (socket says 172.21.15.89:47308) 2023-10-30T12:51:31.772 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.770+0000 7f92cbfff640 1 -- 172.21.15.89:0/390421578 learned_addr learned my addr 172.21.15.89:0/390421578 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:31.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.770+0000 7f92cb7fe640 1 --2- 172.21.15.89:0/390421578 >> v2:172.21.15.192:3300/0 conn(0x7f92cc1538a0 0x7f92cc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:31.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.770+0000 7f92cbfff640 1 -- 172.21.15.89:0/390421578 >> v2:172.21.15.192:3300/0 conn(0x7f92cc1538a0 msgr2=0x7f92cc15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:31.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.770+0000 7f92cbfff640 1 --2- 172.21.15.89:0/390421578 >> v2:172.21.15.192:3300/0 conn(0x7f92cc1538a0 0x7f92cc15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.773 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.770+0000 7f92cbfff640 1 -- 172.21.15.89:0/390421578 >> v1:172.21.15.192:6789/0 conn(0x7f92cc0ca0e0 legacy=0x7f92cc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:31.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.770+0000 7f92cbfff640 1 -- 172.21.15.89:0/390421578 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92cc0cad70 con 0x7f92cc0cf160 2023-10-30T12:51:31.774 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.770+0000 7f92cb7fe640 1 --2- 172.21.15.89:0/390421578 >> v2:172.21.15.192:3300/0 conn(0x7f92cc1538a0 0x7f92cc15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:31.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.774+0000 7f92cbfff640 1 --2- 172.21.15.89:0/390421578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92cc0cf160 0x7f92cc0cf590 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f92cc0cfd10 tx=0x7f92c401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=953ec818c48d06dc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:31.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.774+0000 7f92ca7fc640 1 -- 172.21.15.89:0/390421578 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c40af400 con 0x7f92cc0cf160 2023-10-30T12:51:31.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.774+0000 7f92ca7fc640 1 -- 172.21.15.89:0/390421578 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92c40ab050 con 0x7f92cc0cf160 2023-10-30T12:51:31.776 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.774+0000 7f92ca7fc640 1 -- 172.21.15.89:0/390421578 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c40af780 con 0x7f92cc0cf160 2023-10-30T12:51:31.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.774+0000 7f92d21aa640 1 -- 172.21.15.89:0/390421578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92cc0cf160 msgr2=0x7f92cc0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:31.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.774+0000 7f92d21aa640 1 --2- 172.21.15.89:0/390421578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92cc0cf160 0x7f92cc0cf590 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f92cc0cfd10 tx=0x7f92c401f1c0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.774+0000 7f92d21aa640 1 -- 172.21.15.89:0/390421578 shutdown_connections 2023-10-30T12:51:31.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.774+0000 7f92d21aa640 1 --2- 172.21.15.89:0/390421578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92cc0cf160 0x7f92cc0cf590 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.774+0000 7f92d21aa640 1 --2- 172.21.15.89:0/390421578 >> v2:172.21.15.192:3300/0 conn(0x7f92cc1538a0 0x7f92cc15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.774+0000 7f92d21aa640 1 -- 172.21.15.89:0/390421578 >> 172.21.15.89:0/390421578 conn(0x7f92cc06bfe0 msgr2=0x7f92cc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:31.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.774+0000 7f92d21aa640 1 -- 172.21.15.89:0/390421578 shutdown_connections 2023-10-30T12:51:31.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.774+0000 7f92d21aa640 1 -- 172.21.15.89:0/390421578 wait complete. 2023-10-30T12:51:31.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.778+0000 7f92d21aa640 1 Processor -- start 2023-10-30T12:51:31.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.778+0000 7f92d21aa640 1 -- start start 2023-10-30T12:51:31.781 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.778+0000 7f92d21aa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92cc0ca0e0 0x7f92cc143fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:31.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.778+0000 7f92d21aa640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92cc0cf160 0x7f92cc1f32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:31.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.778+0000 7f92d21aa640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f92cc1538a0 0x7f92cc0c4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:31.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.778+0000 7f92d21aa640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc144510 con 0x7f92cc0ca0e0 2023-10-30T12:51:31.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.778+0000 7f92d21aa640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc1f3800 con 0x7f92cc0cf160 2023-10-30T12:51:31.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.778+0000 7f92d21aa640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc0b34d0 con 0x7f92cc1538a0 2023-10-30T12:51:31.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.778+0000 7f92cb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92cc0ca0e0 0x7f92cc143fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:31.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.778+0000 7f92cb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92cc0ca0e0 0x7f92cc143fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47312/0 (socket says 172.21.15.89:47312) 2023-10-30T12:51:31.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.778+0000 7f92cb7fe640 1 -- 172.21.15.89:0/3180375983 learned_addr learned my addr 172.21.15.89:0/3180375983 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:31.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.778+0000 7f92cbfff640 1 --2- 172.21.15.89:0/3180375983 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f92cc1538a0 0x7f92cc0c4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:31.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.778+0000 7f92caffd640 1 --2- 172.21.15.89:0/3180375983 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92cc0cf160 0x7f92cc1f32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:31.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.778+0000 7f92cb7fe640 1 -- 172.21.15.89:0/3180375983 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f92cc1538a0 msgr2=0x7f92cc0c4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:31.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.778+0000 7f92cb7fe640 1 --2- 172.21.15.89:0/3180375983 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f92cc1538a0 0x7f92cc0c4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.778+0000 7f92cb7fe640 1 -- 172.21.15.89:0/3180375983 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92cc0cf160 msgr2=0x7f92cc1f32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:31.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.778+0000 7f92cb7fe640 1 --2- 172.21.15.89:0/3180375983 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92cc0cf160 0x7f92cc1f32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.786 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.778+0000 7f92cb7fe640 1 -- 172.21.15.89:0/3180375983 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c4009b30 con 0x7f92cc0ca0e0 2023-10-30T12:51:31.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.778+0000 7f92cb7fe640 1 --2- 172.21.15.89:0/3180375983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92cc0ca0e0 0x7f92cc143fd0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f92bc00e810 tx=0x7f92bc00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:31.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.778+0000 7f92b3fff640 1 -- 172.21.15.89:0/3180375983 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92bc00c870 con 0x7f92cc0ca0e0 2023-10-30T12:51:31.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.778+0000 7f92b3fff640 1 -- 172.21.15.89:0/3180375983 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92bc00c9d0 con 0x7f92cc0ca0e0 2023-10-30T12:51:31.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.778+0000 7f92b3fff640 1 -- 172.21.15.89:0/3180375983 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92bc00cd60 con 0x7f92cc0ca0e0 2023-10-30T12:51:31.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.778+0000 7f92d21aa640 1 -- 172.21.15.89:0/3180375983 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92cc1ef340 con 0x7f92cc0ca0e0 2023-10-30T12:51:31.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.778+0000 7f92d21aa640 1 -- 172.21.15.89:0/3180375983 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92cc1ef8c0 con 0x7f92cc0ca0e0 2023-10-30T12:51:31.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.778+0000 7f92d21aa640 1 -- 172.21.15.89:0/3180375983 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92cc0b2a50 con 0x7f92cc0ca0e0 2023-10-30T12:51:31.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.782+0000 7f92b3fff640 1 -- 172.21.15.89:0/3180375983 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f92bc013410 con 0x7f92cc0ca0e0 2023-10-30T12:51:31.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.782+0000 7f92b3fff640 1 --2- 172.21.15.89:0/3180375983 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f92b80664a0 0x7f92b8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:31.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.782+0000 7f92b3fff640 1 -- 172.21.15.89:0/3180375983 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f92bc084080 con 0x7f92cc0ca0e0 2023-10-30T12:51:31.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.782+0000 7f92caffd640 1 --2- 172.21.15.89:0/3180375983 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f92b80664a0 0x7f92b8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:31.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.782+0000 7f92caffd640 1 --2- 172.21.15.89:0/3180375983 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f92b80664a0 0x7f92b8068960 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f92cc15dc00 tx=0x7f92c0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:31.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.786+0000 7f92b3fff640 1 -- 172.21.15.89:0/3180375983 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f92bc051070 con 0x7f92cc0ca0e0 2023-10-30T12:51:31.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.930+0000 7f92d21aa640 1 -- 172.21.15.89:0/3180375983 --> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f92cc159890 con 0x7f92b80664a0 2023-10-30T12:51:31.952 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.954+0000 7f92b3fff640 1 -- 172.21.15.89:0/3180375983 <== mgr.4105 v2:172.21.15.192:6832/4076276638 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+292221 (secure 0 0 0) 0x7f92c00050d0 con 0x7f92b80664a0 2023-10-30T12:51:31.955 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:51:31.956 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-10-30T12:51:31.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.958+0000 7f92d21aa640 1 -- 172.21.15.89:0/3180375983 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f92b80664a0 msgr2=0x7f92b8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:31.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.958+0000 7f92d21aa640 1 --2- 172.21.15.89:0/3180375983 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f92b80664a0 0x7f92b8068960 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f92cc15dc00 tx=0x7f92c0009040 comp rx=0 tx=0).stop 2023-10-30T12:51:31.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.958+0000 7f92d21aa640 1 -- 172.21.15.89:0/3180375983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92cc0ca0e0 msgr2=0x7f92cc143fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:31.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.958+0000 7f92d21aa640 1 --2- 172.21.15.89:0/3180375983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92cc0ca0e0 0x7f92cc143fd0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f92bc00e810 tx=0x7f92bc00ecd0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.958+0000 7f92d21aa640 1 -- 172.21.15.89:0/3180375983 shutdown_connections 2023-10-30T12:51:31.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.958+0000 7f92d21aa640 1 --2- 172.21.15.89:0/3180375983 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f92b80664a0 0x7f92b8068960 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.960 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.958+0000 7f92d21aa640 1 --2- 172.21.15.89:0/3180375983 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f92cc0cf160 0x7f92cc1f32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.960 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.958+0000 7f92d21aa640 1 --2- 172.21.15.89:0/3180375983 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f92cc1538a0 0x7f92cc0c4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.960 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.958+0000 7f92d21aa640 1 --2- 172.21.15.89:0/3180375983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92cc0ca0e0 0x7f92cc143fd0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.960 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.958+0000 7f92d21aa640 1 -- 172.21.15.89:0/3180375983 >> 172.21.15.89:0/3180375983 conn(0x7f92cc06bfe0 msgr2=0x7f92cc1513b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:31.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.958+0000 7f92d21aa640 1 -- 172.21.15.89:0/3180375983 shutdown_connections 2023-10-30T12:51:31.961 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:31.958+0000 7f92d21aa640 1 -- 172.21.15.89:0/3180375983 wait complete. 2023-10-30T12:51:31.976 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":746,"stamp":"2023-10-30T12:51:31.428790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1063030610,"num_objects":1094,"num_object_clones":0,"num_object_copies":2188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1094,"num_whiteouts":0,"num_read":4359,"num_read_kb":813147,"num_write":41998,"num_write_kb":1354122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":78,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45201,"ondisk_log_size":45201,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2612756,"kb_used_data":2085780,"kb_used_omap":12,"kb_used_meta":526899,"kb_avail":752361964,"statfs":{"total":773094113280,"available":770418651136,"internally_reserved":0,"allocated":2135838720,"data_stored":2129160931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12741,"internal_metadata":539545147},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2682127,"num_objects":-61,"num_object_clones":0,"num_object_copies":-122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-61,"num_whiteouts":0,"num_read":8,"num_read_kb":19,"num_write":251,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004335"},"pg_stats":[{"pgid":"3.3f","version":"65'525","reported_seq":2677,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.734150+0000","last_change":"2023-10-30T12:38:17.335461+0000","last_active":"2023-10-30T12:51:12.734150+0000","last_peered":"2023-10-30T12:51:12.734150+0000","last_clean":"2023-10-30T12:51:12.734150+0000","last_became_active":"2023-10-30T12:28:50.979351+0000","last_became_peered":"2023-10-30T12:28:50.979351+0000","last_unstale":"2023-10-30T12:51:12.734150+0000","last_undegraded":"2023-10-30T12:51:12.734150+0000","last_fullsized":"2023-10-30T12:51:12.734150+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":525,"log_dups_size":0,"ondisk_log_size":525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:04:55.768566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091708599999999994,"stat_sum":{"num_bytes":4266500,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":40,"num_read_kb":15,"num_write":484,"num_write_kb":5355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"66'518","reported_seq":2663,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.730499+0000","last_change":"2023-10-30T12:38:17.104663+0000","last_active":"2023-10-30T12:51:12.730499+0000","last_peered":"2023-10-30T12:51:12.730499+0000","last_clean":"2023-10-30T12:51:12.730499+0000","last_became_active":"2023-10-30T12:28:50.929133+0000","last_became_peered":"2023-10-30T12:28:50.929133+0000","last_unstale":"2023-10-30T12:51:12.730499+0000","last_undegraded":"2023-10-30T12:51:12.730499+0000","last_fullsized":"2023-10-30T12:51:12.730499+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":518,"log_dups_size":0,"ondisk_log_size":518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:05:29.266806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023149070000000002,"stat_sum":{"num_bytes":8405214,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":33,"num_read_kb":10,"num_write":478,"num_write_kb":9481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"66'525","reported_seq":2686,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.742402+0000","last_change":"2023-10-30T12:38:17.643537+0000","last_active":"2023-10-30T12:51:12.742402+0000","last_peered":"2023-10-30T12:51:12.742402+0000","last_clean":"2023-10-30T12:51:12.742402+0000","last_became_active":"2023-10-30T12:28:50.933896+0000","last_became_peered":"2023-10-30T12:28:50.933896+0000","last_unstale":"2023-10-30T12:51:12.742402+0000","last_undegraded":"2023-10-30T12:51:12.742402+0000","last_fullsized":"2023-10-30T12:51:12.742402+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":525,"log_dups_size":0,"ondisk_log_size":525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:45:03.377748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029033520000000001,"stat_sum":{"num_bytes":12676713,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":49,"num_read_kb":8209,"num_write":482,"num_write_kb":25758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"66'502","reported_seq":2645,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.744598+0000","last_change":"2023-10-30T12:38:17.117873+0000","last_active":"2023-10-30T12:51:12.744598+0000","last_peered":"2023-10-30T12:51:12.744598+0000","last_clean":"2023-10-30T12:51:12.744598+0000","last_became_active":"2023-10-30T12:28:50.967774+0000","last_became_peered":"2023-10-30T12:28:50.967774+0000","last_unstale":"2023-10-30T12:51:12.744598+0000","last_undegraded":"2023-10-30T12:51:12.744598+0000","last_fullsized":"2023-10-30T12:51:12.744598+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":502,"log_dups_size":0,"ondisk_log_size":502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:56:36.456625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075479680000000004,"stat_sum":{"num_bytes":4269565,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":30,"num_read_kb":13,"num_write":476,"num_write_kb":5875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"65'508","reported_seq":2666,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.736735+0000","last_change":"2023-10-30T12:38:17.333966+0000","last_active":"2023-10-30T12:51:12.736735+0000","last_peered":"2023-10-30T12:51:12.736735+0000","last_clean":"2023-10-30T12:51:12.736735+0000","last_became_active":"2023-10-30T12:28:50.973132+0000","last_became_peered":"2023-10-30T12:28:50.973132+0000","last_unstale":"2023-10-30T12:51:12.736735+0000","last_undegraded":"2023-10-30T12:51:12.736735+0000","last_fullsized":"2023-10-30T12:51:12.736735+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":508,"log_dups_size":0,"ondisk_log_size":508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:43:42.826908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0090421609999999999,"stat_sum":{"num_bytes":12646568,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":47,"num_read_kb":16,"num_write":486,"num_write_kb":14458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"66'522","reported_seq":2671,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:22.381126+0000","last_change":"2023-10-30T12:38:17.112051+0000","last_active":"2023-10-30T12:51:22.381126+0000","last_peered":"2023-10-30T12:51:22.381126+0000","last_clean":"2023-10-30T12:51:22.381126+0000","last_became_active":"2023-10-30T12:28:50.967910+0000","last_became_peered":"2023-10-30T12:28:50.967910+0000","last_unstale":"2023-10-30T12:51:22.381126+0000","last_undegraded":"2023-10-30T12:51:22.381126+0000","last_fullsized":"2023-10-30T12:51:22.381126+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":522,"log_dups_size":0,"ondisk_log_size":522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:31:53.805701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040413009999999997,"stat_sum":{"num_bytes":12682645,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":38,"num_read_kb":16,"num_write":498,"num_write_kb":14506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"66'531","reported_seq":2683,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:22.402041+0000","last_change":"2023-10-30T12:38:17.328004+0000","last_active":"2023-10-30T12:51:22.402041+0000","last_peered":"2023-10-30T12:51:22.402041+0000","last_clean":"2023-10-30T12:51:22.402041+0000","last_became_active":"2023-10-30T12:28:50.973288+0000","last_became_peered":"2023-10-30T12:28:50.973288+0000","last_unstale":"2023-10-30T12:51:22.402041+0000","last_undegraded":"2023-10-30T12:51:22.402041+0000","last_fullsized":"2023-10-30T12:51:22.402041+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":531,"log_dups_size":0,"ondisk_log_size":531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:42:52.258224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037299339999999999,"stat_sum":{"num_bytes":4559885,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":41,"num_read_kb":11,"num_write":501,"num_write_kb":7147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"65'470","reported_seq":2620,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.696226+0000","last_change":"2023-10-30T12:38:17.330519+0000","last_active":"2023-10-30T12:51:12.696226+0000","last_peered":"2023-10-30T12:51:12.696226+0000","last_clean":"2023-10-30T12:51:12.696226+0000","last_became_active":"2023-10-30T12:28:50.954988+0000","last_became_peered":"2023-10-30T12:28:50.954988+0000","last_unstale":"2023-10-30T12:51:12.696226+0000","last_undegraded":"2023-10-30T12:51:12.696226+0000","last_fullsized":"2023-10-30T12:51:12.696226+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":470,"log_dups_size":0,"ondisk_log_size":470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:01:29.621026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028962670000000001,"stat_sum":{"num_bytes":8415120,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":15,"num_write":450,"num_write_kb":10234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"66'548","reported_seq":2708,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.686154+0000","last_change":"2023-10-30T12:38:17.336435+0000","last_active":"2023-10-30T12:51:23.686154+0000","last_peered":"2023-10-30T12:51:23.686154+0000","last_clean":"2023-10-30T12:51:23.686154+0000","last_became_active":"2023-10-30T12:28:50.976564+0000","last_became_peered":"2023-10-30T12:28:50.976564+0000","last_unstale":"2023-10-30T12:51:23.686154+0000","last_undegraded":"2023-10-30T12:51:23.686154+0000","last_fullsized":"2023-10-30T12:51:23.686154+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":548,"log_dups_size":0,"ondisk_log_size":548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:02.733738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0094697280000000002,"stat_sum":{"num_bytes":16822419,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":49,"num_read_kb":4116,"num_write":506,"num_write_kb":27572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"66'518","reported_seq":2688,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.731428+0000","last_change":"2023-10-30T12:38:17.327384+0000","last_active":"2023-10-30T12:51:12.731428+0000","last_peered":"2023-10-30T12:51:12.731428+0000","last_clean":"2023-10-30T12:51:12.731428+0000","last_became_active":"2023-10-30T12:28:50.951809+0000","last_became_peered":"2023-10-30T12:28:50.951809+0000","last_unstale":"2023-10-30T12:51:12.731428+0000","last_undegraded":"2023-10-30T12:51:12.731428+0000","last_fullsized":"2023-10-30T12:51:12.731428+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":518,"log_dups_size":0,"ondisk_log_size":518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:26:18.073656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0078740100000000007,"stat_sum":{"num_bytes":16937210,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":59,"num_read_kb":29,"num_write":484,"num_write_kb":22711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"66'547","reported_seq":2711,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:13.217881+0000","last_change":"2023-10-30T12:38:17.323179+0000","last_active":"2023-10-30T12:51:13.217881+0000","last_peered":"2023-10-30T12:51:13.217881+0000","last_clean":"2023-10-30T12:51:13.217881+0000","last_became_active":"2023-10-30T12:29:32.958473+0000","last_became_peered":"2023-10-30T12:29:32.958473+0000","last_unstale":"2023-10-30T12:51:13.217881+0000","last_undegraded":"2023-10-30T12:51:13.217881+0000","last_fullsized":"2023-10-30T12:51:13.217881+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":547,"log_dups_size":0,"ondisk_log_size":547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:07:47.715571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045626240000000004,"stat_sum":{"num_bytes":13032007,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":46,"num_read_kb":17,"num_write":521,"num_write_kb":14861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"65'484","reported_seq":2629,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.654545+0000","last_change":"2023-10-30T12:38:17.113891+0000","last_active":"2023-10-30T12:51:12.654545+0000","last_peered":"2023-10-30T12:51:12.654545+0000","last_clean":"2023-10-30T12:51:12.654545+0000","last_became_active":"2023-10-30T12:28:50.943787+0000","last_became_peered":"2023-10-30T12:28:50.943787+0000","last_unstale":"2023-10-30T12:51:12.654545+0000","last_undegraded":"2023-10-30T12:51:12.654545+0000","last_fullsized":"2023-10-30T12:51:12.654545+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":484,"log_dups_size":0,"ondisk_log_size":484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:51:54.467182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047342360000000002,"stat_sum":{"num_bytes":12598536,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":34,"num_read_kb":4115,"num_write":456,"num_write_kb":18754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"66'573","reported_seq":2734,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.692359+0000","last_change":"2023-10-30T12:38:17.331215+0000","last_active":"2023-10-30T12:51:23.692359+0000","last_peered":"2023-10-30T12:51:23.692359+0000","last_clean":"2023-10-30T12:51:23.692359+0000","last_became_active":"2023-10-30T12:28:50.937906+0000","last_became_peered":"2023-10-30T12:28:50.937906+0000","last_unstale":"2023-10-30T12:51:23.692359+0000","last_undegraded":"2023-10-30T12:51:23.692359+0000","last_fullsized":"2023-10-30T12:51:23.692359+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":573,"log_dups_size":0,"ondisk_log_size":573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:43:06.819593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003369164,"stat_sum":{"num_bytes":4252811,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":50,"num_read_kb":8215,"num_write":540,"num_write_kb":14536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"66'509","reported_seq":2667,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.741036+0000","last_change":"2023-10-30T12:38:17.643256+0000","last_active":"2023-10-30T12:51:12.741036+0000","last_peered":"2023-10-30T12:51:12.741036+0000","last_clean":"2023-10-30T12:51:12.741036+0000","last_became_active":"2023-10-30T12:28:50.932753+0000","last_became_peered":"2023-10-30T12:28:50.932753+0000","last_unstale":"2023-10-30T12:51:12.741036+0000","last_undegraded":"2023-10-30T12:51:12.741036+0000","last_fullsized":"2023-10-30T12:51:12.741036+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":509,"log_dups_size":0,"ondisk_log_size":509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:37:01.519745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027063,"stat_sum":{"num_bytes":16922391,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":47,"num_read_kb":17,"num_write":484,"num_write_kb":19095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"65'539","reported_seq":2700,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.711318+0000","last_change":"2023-10-30T12:38:17.103685+0000","last_active":"2023-10-30T12:51:12.711318+0000","last_peered":"2023-10-30T12:51:12.711318+0000","last_clean":"2023-10-30T12:51:12.711318+0000","last_became_active":"2023-10-30T12:28:50.943011+0000","last_became_peered":"2023-10-30T12:28:50.943011+0000","last_unstale":"2023-10-30T12:51:12.711318+0000","last_undegraded":"2023-10-30T12:51:12.711318+0000","last_fullsized":"2023-10-30T12:51:12.711318+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":539,"log_dups_size":0,"ondisk_log_size":539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:43:12.517498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001813863,"stat_sum":{"num_bytes":21134561,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":51,"num_read_kb":4115,"num_write":503,"num_write_kb":22902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"66'537","reported_seq":2694,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.683828+0000","last_change":"2023-10-30T12:38:17.111689+0000","last_active":"2023-10-30T12:51:23.683828+0000","last_peered":"2023-10-30T12:51:23.683828+0000","last_clean":"2023-10-30T12:51:23.683828+0000","last_became_active":"2023-10-30T12:28:50.952744+0000","last_became_peered":"2023-10-30T12:28:50.952744+0000","last_unstale":"2023-10-30T12:51:23.683828+0000","last_undegraded":"2023-10-30T12:51:23.683828+0000","last_fullsized":"2023-10-30T12:51:23.683828+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":537,"log_dups_size":0,"ondisk_log_size":537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:46:48.822756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034371530000000001,"stat_sum":{"num_bytes":4653479,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":18,"num_write":512,"num_write_kb":9410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"66'554","reported_seq":2705,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:18.626233+0000","last_change":"2023-10-30T12:38:17.120572+0000","last_active":"2023-10-30T12:51:18.626233+0000","last_peered":"2023-10-30T12:51:18.626233+0000","last_clean":"2023-10-30T12:51:18.626233+0000","last_became_active":"2023-10-30T12:28:50.974576+0000","last_became_peered":"2023-10-30T12:28:50.974576+0000","last_unstale":"2023-10-30T12:51:18.626233+0000","last_undegraded":"2023-10-30T12:51:18.626233+0000","last_fullsized":"2023-10-30T12:51:18.626233+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":554,"log_dups_size":0,"ondisk_log_size":554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:19:51.714260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080003049999999992,"stat_sum":{"num_bytes":8578534,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":4113,"num_write":520,"num_write_kb":16515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"66'507","reported_seq":2676,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.747521+0000","last_change":"2023-10-30T12:38:17.285320+0000","last_active":"2023-10-30T12:51:12.747521+0000","last_peered":"2023-10-30T12:51:12.747521+0000","last_clean":"2023-10-30T12:51:12.747521+0000","last_became_active":"2023-10-30T12:28:50.937384+0000","last_became_peered":"2023-10-30T12:28:50.937384+0000","last_unstale":"2023-10-30T12:51:12.747521+0000","last_undegraded":"2023-10-30T12:51:12.747521+0000","last_fullsized":"2023-10-30T12:51:12.747521+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:54:42.597082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001162803,"stat_sum":{"num_bytes":8483984,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":58,"num_read_kb":27,"num_write":482,"num_write_kb":9419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"66'517","reported_seq":2684,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.745991+0000","last_change":"2023-10-30T12:38:17.336198+0000","last_active":"2023-10-30T12:51:12.745991+0000","last_peered":"2023-10-30T12:51:12.745991+0000","last_clean":"2023-10-30T12:51:12.745991+0000","last_became_active":"2023-10-30T12:28:50.973369+0000","last_became_peered":"2023-10-30T12:28:50.973369+0000","last_unstale":"2023-10-30T12:51:12.745991+0000","last_undegraded":"2023-10-30T12:51:12.745991+0000","last_fullsized":"2023-10-30T12:51:12.745991+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:57:45.650842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093923919999999994,"stat_sum":{"num_bytes":12735800,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":55,"num_read_kb":4114,"num_write":492,"num_write_kb":18044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"66'550","reported_seq":2711,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:22.554611+0000","last_change":"2023-10-30T12:38:17.325892+0000","last_active":"2023-10-30T12:51:22.554611+0000","last_peered":"2023-10-30T12:51:22.554611+0000","last_clean":"2023-10-30T12:51:22.554611+0000","last_became_active":"2023-10-30T12:28:50.954831+0000","last_became_peered":"2023-10-30T12:28:50.954831+0000","last_unstale":"2023-10-30T12:51:22.554611+0000","last_undegraded":"2023-10-30T12:51:22.554611+0000","last_fullsized":"2023-10-30T12:51:22.554611+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":550,"log_dups_size":0,"ondisk_log_size":550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:54:29.936622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007119116,"stat_sum":{"num_bytes":21023102,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":49,"num_read_kb":4113,"num_write":518,"num_write_kb":30328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"66'515","reported_seq":2668,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.686674+0000","last_change":"2023-10-30T12:38:17.644608+0000","last_active":"2023-10-30T12:51:23.686674+0000","last_peered":"2023-10-30T12:51:23.686674+0000","last_clean":"2023-10-30T12:51:23.686674+0000","last_became_active":"2023-10-30T12:28:50.962065+0000","last_became_peered":"2023-10-30T12:28:50.962065+0000","last_unstale":"2023-10-30T12:51:23.686674+0000","last_undegraded":"2023-10-30T12:51:23.686674+0000","last_fullsized":"2023-10-30T12:51:23.686674+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:07:44.264938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037739459999999998,"stat_sum":{"num_bytes":25185776,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":41,"num_read_kb":21,"num_write":488,"num_write_kb":26337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"66'577","reported_seq":2735,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:22.381186+0000","last_change":"2023-10-30T12:38:17.118400+0000","last_active":"2023-10-30T12:51:22.381186+0000","last_peered":"2023-10-30T12:51:22.381186+0000","last_clean":"2023-10-30T12:51:22.381186+0000","last_became_active":"2023-10-30T12:28:50.967651+0000","last_became_peered":"2023-10-30T12:28:50.967651+0000","last_unstale":"2023-10-30T12:51:22.381186+0000","last_undegraded":"2023-10-30T12:51:22.381186+0000","last_fullsized":"2023-10-30T12:51:22.381186+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":577,"log_dups_size":0,"ondisk_log_size":577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:19:34.662364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081085610000000002,"stat_sum":{"num_bytes":8464992,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":21,"num_write":548,"num_write_kb":13569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"66'530","reported_seq":2692,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.738894+0000","last_change":"2023-10-30T12:38:17.332602+0000","last_active":"2023-10-30T12:51:12.738894+0000","last_peered":"2023-10-30T12:51:12.738894+0000","last_clean":"2023-10-30T12:51:12.738894+0000","last_became_active":"2023-10-30T12:28:50.962290+0000","last_became_peered":"2023-10-30T12:28:50.962290+0000","last_unstale":"2023-10-30T12:51:12.738894+0000","last_undegraded":"2023-10-30T12:51:12.738894+0000","last_fullsized":"2023-10-30T12:51:12.738894+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:45:30.777077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042026670000000002,"stat_sum":{"num_bytes":57151,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":50,"num_read_kb":4117,"num_write":500,"num_write_kb":6118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"66'550","reported_seq":2702,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:24.465670+0000","last_change":"2023-10-30T12:38:17.114285+0000","last_active":"2023-10-30T12:51:24.465670+0000","last_peered":"2023-10-30T12:51:24.465670+0000","last_clean":"2023-10-30T12:51:24.465670+0000","last_became_active":"2023-10-30T12:28:50.965713+0000","last_became_peered":"2023-10-30T12:28:50.965713+0000","last_unstale":"2023-10-30T12:51:24.465670+0000","last_undegraded":"2023-10-30T12:51:24.465670+0000","last_fullsized":"2023-10-30T12:51:24.465670+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":550,"log_dups_size":0,"ondisk_log_size":550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:08:31.859565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056034209999999999,"stat_sum":{"num_bytes":12649475,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":40,"num_read_kb":16,"num_write":522,"num_write_kb":18815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"66'503","reported_seq":2646,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:18.626419+0000","last_change":"2023-10-30T12:38:17.330864+0000","last_active":"2023-10-30T12:51:18.626419+0000","last_peered":"2023-10-30T12:51:18.626419+0000","last_clean":"2023-10-30T12:51:18.626419+0000","last_became_active":"2023-10-30T12:28:50.951026+0000","last_became_peered":"2023-10-30T12:28:50.951026+0000","last_unstale":"2023-10-30T12:51:18.626419+0000","last_undegraded":"2023-10-30T12:51:18.626419+0000","last_fullsized":"2023-10-30T12:51:18.626419+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":503,"log_dups_size":0,"ondisk_log_size":503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:39:54.872161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031697840000000001,"stat_sum":{"num_bytes":13094288,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":31,"num_read_kb":12,"num_write":474,"num_write_kb":19622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"66'579","reported_seq":2742,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.737087+0000","last_change":"2023-10-30T12:38:17.327887+0000","last_active":"2023-10-30T12:51:12.737087+0000","last_peered":"2023-10-30T12:51:12.737087+0000","last_clean":"2023-10-30T12:51:12.737087+0000","last_became_active":"2023-10-30T12:28:50.962484+0000","last_became_peered":"2023-10-30T12:28:50.962484+0000","last_unstale":"2023-10-30T12:51:12.737087+0000","last_undegraded":"2023-10-30T12:51:12.737087+0000","last_fullsized":"2023-10-30T12:51:12.737087+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:41:18.860625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083773500000000004,"stat_sum":{"num_bytes":13048022,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":51,"num_read_kb":23,"num_write":546,"num_write_kb":15053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"66'481","reported_seq":2635,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.743969+0000","last_change":"2023-10-30T12:38:17.104453+0000","last_active":"2023-10-30T12:51:12.743969+0000","last_peered":"2023-10-30T12:51:12.743969+0000","last_clean":"2023-10-30T12:51:12.743969+0000","last_became_active":"2023-10-30T12:28:50.933093+0000","last_became_peered":"2023-10-30T12:28:50.933093+0000","last_unstale":"2023-10-30T12:51:12.743969+0000","last_undegraded":"2023-10-30T12:51:12.743969+0000","last_fullsized":"2023-10-30T12:51:12.743969+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":481,"log_dups_size":0,"ondisk_log_size":481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:58:51.514957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022251829999999999,"stat_sum":{"num_bytes":25268103,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":42,"num_read_kb":8207,"num_write":439,"num_write_kb":27476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"65'470","reported_seq":2608,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.727936+0000","last_change":"2023-10-30T12:38:17.332060+0000","last_active":"2023-10-30T12:51:12.727936+0000","last_peered":"2023-10-30T12:51:12.727936+0000","last_clean":"2023-10-30T12:51:12.727936+0000","last_became_active":"2023-10-30T12:28:50.952531+0000","last_became_peered":"2023-10-30T12:28:50.952531+0000","last_unstale":"2023-10-30T12:51:12.727936+0000","last_undegraded":"2023-10-30T12:51:12.727936+0000","last_fullsized":"2023-10-30T12:51:12.727936+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":470,"log_dups_size":0,"ondisk_log_size":470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:22:53.945790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039301029999999999,"stat_sum":{"num_bytes":4497430,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":26,"num_read_kb":3284,"num_write":447,"num_write_kb":14120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"65'492","reported_seq":2649,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.733923+0000","last_change":"2023-10-30T12:38:17.644463+0000","last_active":"2023-10-30T12:51:12.733923+0000","last_peered":"2023-10-30T12:51:12.733923+0000","last_clean":"2023-10-30T12:51:12.733923+0000","last_became_active":"2023-10-30T12:28:50.933124+0000","last_became_peered":"2023-10-30T12:28:50.933124+0000","last_unstale":"2023-10-30T12:51:12.733923+0000","last_undegraded":"2023-10-30T12:51:12.733923+0000","last_fullsized":"2023-10-30T12:51:12.733923+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":492,"log_dups_size":0,"ondisk_log_size":492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:25:34.819552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034752709999999998,"stat_sum":{"num_bytes":8485989,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":45,"num_read_kb":14,"num_write":462,"num_write_kb":10450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"66'566","reported_seq":2729,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.737666+0000","last_change":"2023-10-30T12:38:17.118502+0000","last_active":"2023-10-30T12:51:12.737666+0000","last_peered":"2023-10-30T12:51:12.737666+0000","last_clean":"2023-10-30T12:51:12.737666+0000","last_became_active":"2023-10-30T12:28:50.965639+0000","last_became_peered":"2023-10-30T12:28:50.965639+0000","last_unstale":"2023-10-30T12:51:12.737666+0000","last_undegraded":"2023-10-30T12:51:12.737666+0000","last_fullsized":"2023-10-30T12:51:12.737666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":566,"log_dups_size":0,"ondisk_log_size":566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:46:12.692237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076861280000000004,"stat_sum":{"num_bytes":8419509,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":51,"num_read_kb":20,"num_write":541,"num_write_kb":11490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"66'561","reported_seq":2723,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.739391+0000","last_change":"2023-10-30T12:38:17.285366+0000","last_active":"2023-10-30T12:51:12.739391+0000","last_peered":"2023-10-30T12:51:12.739391+0000","last_clean":"2023-10-30T12:51:12.739391+0000","last_became_active":"2023-10-30T12:28:50.955084+0000","last_became_peered":"2023-10-30T12:28:50.955084+0000","last_unstale":"2023-10-30T12:51:12.739391+0000","last_undegraded":"2023-10-30T12:51:12.739391+0000","last_fullsized":"2023-10-30T12:51:12.739391+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":561,"log_dups_size":0,"ondisk_log_size":561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:56:51.880241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012043119999999999,"stat_sum":{"num_bytes":4261339,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":50,"num_read_kb":22,"num_write":525,"num_write_kb":9810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"65'533","reported_seq":2698,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.730032+0000","last_change":"2023-10-30T12:38:17.326196+0000","last_active":"2023-10-30T12:51:12.730032+0000","last_peered":"2023-10-30T12:51:12.730032+0000","last_clean":"2023-10-30T12:51:12.730032+0000","last_became_active":"2023-10-30T12:28:50.944775+0000","last_became_peered":"2023-10-30T12:28:50.944775+0000","last_unstale":"2023-10-30T12:51:12.730032+0000","last_undegraded":"2023-10-30T12:51:12.730032+0000","last_fullsized":"2023-10-30T12:51:12.730032+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":533,"log_dups_size":0,"ondisk_log_size":533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:02:19.297081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070573429999999998,"stat_sum":{"num_bytes":12715391,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":53,"num_read_kb":21,"num_write":506,"num_write_kb":13685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"66'553","reported_seq":2704,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.740036+0000","last_change":"2023-10-30T12:38:17.109227+0000","last_active":"2023-10-30T12:51:12.740036+0000","last_peered":"2023-10-30T12:51:12.740036+0000","last_clean":"2023-10-30T12:51:12.740036+0000","last_became_active":"2023-10-30T12:28:50.940567+0000","last_became_peered":"2023-10-30T12:28:50.940567+0000","last_unstale":"2023-10-30T12:51:12.740036+0000","last_undegraded":"2023-10-30T12:51:12.740036+0000","last_fullsized":"2023-10-30T12:51:12.740036+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":553,"log_dups_size":0,"ondisk_log_size":553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:56:41.791482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028523340000000002,"stat_sum":{"num_bytes":9733426,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":40,"num_read_kb":17,"num_write":523,"num_write_kb":12778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"65'278","reported_seq":357,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:46.346718+0000","last_change":"2023-10-30T12:28:50.395670+0000","last_active":"2023-10-30T12:50:46.346718+0000","last_peered":"2023-10-30T12:50:46.346718+0000","last_clean":"2023-10-30T12:50:46.346718+0000","last_became_active":"2023-10-30T12:28:50.395382+0000","last_became_peered":"2023-10-30T12:28:50.395382+0000","last_unstale":"2023-10-30T12:50:46.346718+0000","last_undegraded":"2023-10-30T12:50:46.346718+0000","last_fullsized":"2023-10-30T12:50:46.346718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":278,"log_dups_size":0,"ondisk_log_size":278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:14.858121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582946,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":45059,"num_write":213,"num_write_kb":12413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"66'472","reported_seq":2628,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:18.626809+0000","last_change":"2023-10-30T12:38:17.644896+0000","last_active":"2023-10-30T12:51:18.626809+0000","last_peered":"2023-10-30T12:51:18.626809+0000","last_clean":"2023-10-30T12:51:18.626809+0000","last_became_active":"2023-10-30T12:28:50.952613+0000","last_became_peered":"2023-10-30T12:28:50.952613+0000","last_unstale":"2023-10-30T12:51:18.626809+0000","last_undegraded":"2023-10-30T12:51:18.626809+0000","last_fullsized":"2023-10-30T12:51:18.626809+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":472,"log_dups_size":0,"ondisk_log_size":472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:16:49.712167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038505420000000002,"stat_sum":{"num_bytes":21060260,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":45,"num_read_kb":18,"num_write":442,"num_write_kb":22001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"65'896","reported_seq":1174,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:47.899291+0000","last_change":"2023-10-30T12:28:50.384598+0000","last_active":"2023-10-30T12:50:47.899291+0000","last_peered":"2023-10-30T12:50:47.899291+0000","last_clean":"2023-10-30T12:50:47.899291+0000","last_became_active":"2023-10-30T12:28:50.383898+0000","last_became_peered":"2023-10-30T12:28:50.383898+0000","last_unstale":"2023-10-30T12:50:47.899291+0000","last_undegraded":"2023-10-30T12:50:47.899291+0000","last_fullsized":"2023-10-30T12:50:47.899291+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":896,"log_dups_size":0,"ondisk_log_size":896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:09:50.290004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21559845,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":229,"num_read_kb":49875,"num_write":837,"num_write_kb":21586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"66'442","reported_seq":733,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:26.502943+0000","last_change":"2023-10-30T12:28:50.386193+0000","last_active":"2023-10-30T12:51:26.502943+0000","last_peered":"2023-10-30T12:51:26.502943+0000","last_clean":"2023-10-30T12:51:26.502943+0000","last_became_active":"2023-10-30T12:28:50.385963+0000","last_became_peered":"2023-10-30T12:28:50.385963+0000","last_unstale":"2023-10-30T12:51:26.502943+0000","last_undegraded":"2023-10-30T12:51:26.502943+0000","last_fullsized":"2023-10-30T12:51:26.502943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":442,"log_dups_size":0,"ondisk_log_size":442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:00:55.603212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7740025,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":238,"num_read_kb":9091,"num_write":376,"num_write_kb":7812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"66'601","reported_seq":2768,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:18.629254+0000","last_change":"2023-10-30T12:38:17.330785+0000","last_active":"2023-10-30T12:51:18.629254+0000","last_peered":"2023-10-30T12:51:18.629254+0000","last_clean":"2023-10-30T12:51:18.629254+0000","last_became_active":"2023-10-30T12:28:50.935764+0000","last_became_peered":"2023-10-30T12:28:50.935764+0000","last_unstale":"2023-10-30T12:51:18.629254+0000","last_undegraded":"2023-10-30T12:51:18.629254+0000","last_fullsized":"2023-10-30T12:51:18.629254+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":601,"log_dups_size":0,"ondisk_log_size":601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:59:59.545417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031329320000000002,"stat_sum":{"num_bytes":4238419,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":56,"num_read_kb":4120,"num_write":560,"num_write_kb":13730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"65'568","reported_seq":643,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:46.705145+0000","last_change":"2023-10-30T12:28:50.392213+0000","last_active":"2023-10-30T12:50:46.705145+0000","last_peered":"2023-10-30T12:50:46.705145+0000","last_clean":"2023-10-30T12:50:46.705145+0000","last_became_active":"2023-10-30T12:28:50.391494+0000","last_became_peered":"2023-10-30T12:28:50.391494+0000","last_unstale":"2023-10-30T12:50:46.705145+0000","last_undegraded":"2023-10-30T12:50:46.705145+0000","last_fullsized":"2023-10-30T12:50:46.705145+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:38:37.301840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":36864,"num_write":519,"num_write_kb":12589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"66'565","reported_seq":2722,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.747887+0000","last_change":"2023-10-30T12:38:17.103988+0000","last_active":"2023-10-30T12:51:12.747887+0000","last_peered":"2023-10-30T12:51:12.747887+0000","last_clean":"2023-10-30T12:51:12.747887+0000","last_became_active":"2023-10-30T12:28:50.933062+0000","last_became_peered":"2023-10-30T12:28:50.933062+0000","last_unstale":"2023-10-30T12:51:12.747887+0000","last_undegraded":"2023-10-30T12:51:12.747887+0000","last_fullsized":"2023-10-30T12:51:12.747887+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":565,"log_dups_size":0,"ondisk_log_size":565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:52:57.336116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021140489999999998,"stat_sum":{"num_bytes":33649708,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":46,"num_read_kb":4115,"num_write":540,"num_write_kb":39124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"66'128","reported_seq":201,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.642253+0000","last_change":"2023-10-30T12:28:50.394368+0000","last_active":"2023-10-30T12:51:12.642253+0000","last_peered":"2023-10-30T12:51:12.642253+0000","last_clean":"2023-10-30T12:51:12.642253+0000","last_became_active":"2023-10-30T12:28:50.393686+0000","last_became_peered":"2023-10-30T12:28:50.393686+0000","last_unstale":"2023-10-30T12:51:12.642253+0000","last_undegraded":"2023-10-30T12:51:12.642253+0000","last_fullsized":"2023-10-30T12:51:12.642253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:43:17.726462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":73,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"65'540","reported_seq":2702,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.734949+0000","last_change":"2023-10-30T12:38:17.119770+0000","last_active":"2023-10-30T12:51:12.734949+0000","last_peered":"2023-10-30T12:51:12.734949+0000","last_clean":"2023-10-30T12:51:12.734949+0000","last_became_active":"2023-10-30T12:28:50.963105+0000","last_became_peered":"2023-10-30T12:28:50.963105+0000","last_unstale":"2023-10-30T12:51:12.734949+0000","last_undegraded":"2023-10-30T12:51:12.734949+0000","last_fullsized":"2023-10-30T12:51:12.734949+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":540,"log_dups_size":0,"ondisk_log_size":540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:12:25.298678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080066809999999999,"stat_sum":{"num_bytes":16891650,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":51,"num_read_kb":18,"num_write":510,"num_write_kb":18448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"66'281","reported_seq":354,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:47.673064+0000","last_change":"2023-10-30T12:28:50.397679+0000","last_active":"2023-10-30T12:50:47.673064+0000","last_peered":"2023-10-30T12:50:47.673064+0000","last_clean":"2023-10-30T12:50:47.673064+0000","last_became_active":"2023-10-30T12:28:50.397041+0000","last_became_peered":"2023-10-30T12:28:50.397041+0000","last_unstale":"2023-10-30T12:50:47.673064+0000","last_undegraded":"2023-10-30T12:50:47.673064+0000","last_fullsized":"2023-10-30T12:50:47.673064+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:52:22.564616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":16396,"num_write":240,"num_write_kb":8312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"66'521","reported_seq":2674,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.744719+0000","last_change":"2023-10-30T12:38:17.644799+0000","last_active":"2023-10-30T12:51:12.744719+0000","last_peered":"2023-10-30T12:51:12.744719+0000","last_clean":"2023-10-30T12:51:12.744719+0000","last_became_active":"2023-10-30T12:28:50.966569+0000","last_became_peered":"2023-10-30T12:28:50.966569+0000","last_unstale":"2023-10-30T12:51:12.744719+0000","last_undegraded":"2023-10-30T12:51:12.744719+0000","last_fullsized":"2023-10-30T12:51:12.744719+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:43:03.050908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037972850000000001,"stat_sum":{"num_bytes":16921261,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":42,"num_read_kb":19,"num_write":497,"num_write_kb":17986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"66'370","reported_seq":449,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:18.810052+0000","last_change":"2023-10-30T12:28:50.391527+0000","last_active":"2023-10-30T12:51:18.810052+0000","last_peered":"2023-10-30T12:51:18.810052+0000","last_clean":"2023-10-30T12:51:18.810052+0000","last_became_active":"2023-10-30T12:28:50.391260+0000","last_became_peered":"2023-10-30T12:28:50.391260+0000","last_unstale":"2023-10-30T12:51:18.810052+0000","last_undegraded":"2023-10-30T12:51:18.810052+0000","last_fullsized":"2023-10-30T12:51:18.810052+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:29:34.772490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":22,"num_read_kb":7467,"num_write":318,"num_write_kb":4400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"66'490","reported_seq":2648,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.693047+0000","last_change":"2023-10-30T12:38:17.114492+0000","last_active":"2023-10-30T12:51:12.693047+0000","last_peered":"2023-10-30T12:51:12.693047+0000","last_clean":"2023-10-30T12:51:12.693047+0000","last_became_active":"2023-10-30T12:28:50.974843+0000","last_became_peered":"2023-10-30T12:28:50.974843+0000","last_unstale":"2023-10-30T12:51:12.693047+0000","last_undegraded":"2023-10-30T12:51:12.693047+0000","last_fullsized":"2023-10-30T12:51:12.693047+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":490,"log_dups_size":0,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:31:41.341035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005161372,"stat_sum":{"num_bytes":4290434,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":47,"num_read_kb":17,"num_write":450,"num_write_kb":9837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"66'349","reported_seq":439,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.670523+0000","last_change":"2023-10-30T12:28:50.374838+0000","last_active":"2023-10-30T12:51:12.670523+0000","last_peered":"2023-10-30T12:51:12.670523+0000","last_clean":"2023-10-30T12:51:12.670523+0000","last_became_active":"2023-10-30T12:28:50.373581+0000","last_became_peered":"2023-10-30T12:28:50.373581+0000","last_unstale":"2023-10-30T12:51:12.670523+0000","last_undegraded":"2023-10-30T12:51:12.670523+0000","last_fullsized":"2023-10-30T12:51:12.670523+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:14:51.506745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":43,"num_read_kb":32803,"num_write":291,"num_write_kb":12542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"65'466","reported_seq":2615,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.728219+0000","last_change":"2023-10-30T12:38:17.328512+0000","last_active":"2023-10-30T12:51:12.728219+0000","last_peered":"2023-10-30T12:51:12.728219+0000","last_clean":"2023-10-30T12:51:12.728219+0000","last_became_active":"2023-10-30T12:28:50.973269+0000","last_became_peered":"2023-10-30T12:28:50.973269+0000","last_unstale":"2023-10-30T12:51:12.728219+0000","last_undegraded":"2023-10-30T12:51:12.728219+0000","last_fullsized":"2023-10-30T12:51:12.728219+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":466,"log_dups_size":0,"ondisk_log_size":466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:26:11.417670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004001498,"stat_sum":{"num_bytes":12676913,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":38,"num_read_kb":11,"num_write":436,"num_write_kb":13453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"66'436","reported_seq":527,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.511431+0000","last_change":"2023-10-30T12:28:50.366590+0000","last_active":"2023-10-30T12:51:12.511431+0000","last_peered":"2023-10-30T12:51:12.511431+0000","last_clean":"2023-10-30T12:51:12.511431+0000","last_became_active":"2023-10-30T12:28:50.366269+0000","last_became_peered":"2023-10-30T12:28:50.366269+0000","last_unstale":"2023-10-30T12:51:12.511431+0000","last_undegraded":"2023-10-30T12:51:12.511431+0000","last_fullsized":"2023-10-30T12:51:12.511431+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":436,"log_dups_size":0,"ondisk_log_size":436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:15:11.963438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":52,"num_read_kb":36923,"num_write":386,"num_write_kb":8499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"65'536","reported_seq":2699,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.738218+0000","last_change":"2023-10-30T12:38:17.644043+0000","last_active":"2023-10-30T12:51:12.738218+0000","last_peered":"2023-10-30T12:51:12.738218+0000","last_clean":"2023-10-30T12:51:12.738218+0000","last_became_active":"2023-10-30T12:28:50.961805+0000","last_became_peered":"2023-10-30T12:28:50.961805+0000","last_unstale":"2023-10-30T12:51:12.738218+0000","last_undegraded":"2023-10-30T12:51:12.738218+0000","last_fullsized":"2023-10-30T12:51:12.738218+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":536,"log_dups_size":0,"ondisk_log_size":536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:50:40.603836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032264350000000002,"stat_sum":{"num_bytes":4573455,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":52,"num_read_kb":4114,"num_write":491,"num_write_kb":14136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"66'258","reported_seq":333,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:07.342026+0000","last_change":"2023-10-30T12:28:50.397244+0000","last_active":"2023-10-30T12:51:07.342026+0000","last_peered":"2023-10-30T12:51:07.342026+0000","last_clean":"2023-10-30T12:51:07.342026+0000","last_became_active":"2023-10-30T12:28:50.396727+0000","last_became_peered":"2023-10-30T12:28:50.396727+0000","last_unstale":"2023-10-30T12:51:07.342026+0000","last_undegraded":"2023-10-30T12:51:07.342026+0000","last_fullsized":"2023-10-30T12:51:07.342026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:33.493343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":16164,"num_write":201,"num_write_kb":8311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"65'535","reported_seq":2701,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.708532+0000","last_change":"2023-10-30T12:38:17.284992+0000","last_active":"2023-10-30T12:51:12.708532+0000","last_peered":"2023-10-30T12:51:12.708532+0000","last_clean":"2023-10-30T12:51:12.708532+0000","last_became_active":"2023-10-30T12:29:32.960608+0000","last_became_peered":"2023-10-30T12:29:32.960608+0000","last_unstale":"2023-10-30T12:51:12.708532+0000","last_undegraded":"2023-10-30T12:51:12.708532+0000","last_fullsized":"2023-10-30T12:51:12.708532+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":535,"log_dups_size":0,"ondisk_log_size":535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:22:33.324798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012934999999999999,"stat_sum":{"num_bytes":4287412,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":59,"num_read_kb":33,"num_write":498,"num_write_kb":6237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"65'677","reported_seq":758,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-30T12:50:01.976866+0000","last_change":"2023-10-30T12:28:50.383880+0000","last_active":"2023-10-30T12:50:01.976866+0000","last_peered":"2023-10-30T12:50:01.976866+0000","last_clean":"2023-10-30T12:50:01.976866+0000","last_became_active":"2023-10-30T12:28:50.383570+0000","last_became_peered":"2023-10-30T12:28:50.383570+0000","last_unstale":"2023-10-30T12:50:01.976866+0000","last_undegraded":"2023-10-30T12:50:01.976866+0000","last_fullsized":"2023-10-30T12:50:01.976866+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":677,"log_dups_size":0,"ondisk_log_size":677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:19:42.047520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":37306,"num_write":619,"num_write_kb":16724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"65'498","reported_seq":2642,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.684505+0000","last_change":"2023-10-30T12:38:17.322231+0000","last_active":"2023-10-30T12:51:12.684505+0000","last_peered":"2023-10-30T12:51:12.684505+0000","last_clean":"2023-10-30T12:51:12.684505+0000","last_became_active":"2023-10-30T12:28:50.949485+0000","last_became_peered":"2023-10-30T12:28:50.949485+0000","last_unstale":"2023-10-30T12:51:12.684505+0000","last_undegraded":"2023-10-30T12:51:12.684505+0000","last_fullsized":"2023-10-30T12:51:12.684505+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:30:36.516655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040897299999999998,"stat_sum":{"num_bytes":12614326,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":469,"num_write_kb":14309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"66'437","reported_seq":543,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.615726+0000","last_change":"2023-10-30T12:28:50.362979+0000","last_active":"2023-10-30T12:51:12.615726+0000","last_peered":"2023-10-30T12:51:12.615726+0000","last_clean":"2023-10-30T12:51:12.615726+0000","last_became_active":"2023-10-30T12:28:50.362624+0000","last_became_peered":"2023-10-30T12:28:50.362624+0000","last_unstale":"2023-10-30T12:51:12.615726+0000","last_undegraded":"2023-10-30T12:51:12.615726+0000","last_fullsized":"2023-10-30T12:51:12.615726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:01:13.111816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583024,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":93,"num_read_kb":24662,"num_write":374,"num_write_kb":12554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"66'553","reported_seq":2714,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:18.625349+0000","last_change":"2023-10-30T12:38:17.334068+0000","last_active":"2023-10-30T12:51:18.625349+0000","last_peered":"2023-10-30T12:51:18.625349+0000","last_clean":"2023-10-30T12:51:18.625349+0000","last_became_active":"2023-10-30T12:28:50.979344+0000","last_became_peered":"2023-10-30T12:28:50.979344+0000","last_unstale":"2023-10-30T12:51:18.625349+0000","last_undegraded":"2023-10-30T12:51:18.625349+0000","last_fullsized":"2023-10-30T12:51:18.625349+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":553,"log_dups_size":0,"ondisk_log_size":553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:32:04.956803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0092295529999999997,"stat_sum":{"num_bytes":12635563,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":50,"num_read_kb":8210,"num_write":525,"num_write_kb":23175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"66'245","reported_seq":322,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:25.413473+0000","last_change":"2023-10-30T12:28:50.382175+0000","last_active":"2023-10-30T12:51:25.413473+0000","last_peered":"2023-10-30T12:51:25.413473+0000","last_clean":"2023-10-30T12:51:25.413473+0000","last_became_active":"2023-10-30T12:28:50.381125+0000","last_became_peered":"2023-10-30T12:28:50.381125+0000","last_unstale":"2023-10-30T12:51:25.413473+0000","last_undegraded":"2023-10-30T12:51:25.413473+0000","last_fullsized":"2023-10-30T12:51:25.413473+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":245,"log_dups_size":0,"ondisk_log_size":245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:36:02.652665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":12378,"num_write":394,"num_write_kb":4652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"65'503","reported_seq":2651,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.747470+0000","last_change":"2023-10-30T12:38:17.334809+0000","last_active":"2023-10-30T12:51:12.747470+0000","last_peered":"2023-10-30T12:51:12.747470+0000","last_clean":"2023-10-30T12:51:12.747470+0000","last_became_active":"2023-10-30T12:28:50.977585+0000","last_became_peered":"2023-10-30T12:28:50.977585+0000","last_unstale":"2023-10-30T12:51:12.747470+0000","last_undegraded":"2023-10-30T12:51:12.747470+0000","last_fullsized":"2023-10-30T12:51:12.747470+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":503,"log_dups_size":0,"ondisk_log_size":503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:33:51.736957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089211889999999995,"stat_sum":{"num_bytes":12866878,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":36,"num_read_kb":8205,"num_write":472,"num_write_kb":18794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"65'355","reported_seq":432,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.691046+0000","last_change":"2023-10-30T12:28:50.367258+0000","last_active":"2023-10-30T12:51:12.691046+0000","last_peered":"2023-10-30T12:51:12.691046+0000","last_clean":"2023-10-30T12:51:12.691046+0000","last_became_active":"2023-10-30T12:28:50.366974+0000","last_became_peered":"2023-10-30T12:28:50.366974+0000","last_unstale":"2023-10-30T12:51:12.691046+0000","last_undegraded":"2023-10-30T12:51:12.691046+0000","last_fullsized":"2023-10-30T12:51:12.691046+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:14:48.823902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":15324,"num_write":292,"num_write_kb":8361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"66'510","reported_seq":2672,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:24.460236+0000","last_change":"2023-10-30T12:38:17.337139+0000","last_active":"2023-10-30T12:51:24.460236+0000","last_peered":"2023-10-30T12:51:24.460236+0000","last_clean":"2023-10-30T12:51:24.460236+0000","last_became_active":"2023-10-30T12:28:50.981193+0000","last_became_peered":"2023-10-30T12:28:50.981193+0000","last_unstale":"2023-10-30T12:51:24.460236+0000","last_undegraded":"2023-10-30T12:51:24.460236+0000","last_fullsized":"2023-10-30T12:51:24.460236+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":510,"log_dups_size":0,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:32:10.843997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096579700000000001,"stat_sum":{"num_bytes":12827883,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":50,"num_read_kb":4114,"num_write":480,"num_write_kb":17991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"66'96","reported_seq":169,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:22.334230+0000","last_change":"2023-10-30T12:28:50.368011+0000","last_active":"2023-10-30T12:51:22.334230+0000","last_peered":"2023-10-30T12:51:22.334230+0000","last_clean":"2023-10-30T12:51:22.334230+0000","last_became_active":"2023-10-30T12:28:50.367730+0000","last_became_peered":"2023-10-30T12:28:50.367730+0000","last_unstale":"2023-10-30T12:51:22.334230+0000","last_undegraded":"2023-10-30T12:51:22.334230+0000","last_fullsized":"2023-10-30T12:51:22.334230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:59:41.946550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":12292,"num_write":45,"num_write_kb":4186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"65'566","reported_seq":2734,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.746266+0000","last_change":"2023-10-30T12:38:17.119150+0000","last_active":"2023-10-30T12:51:12.746266+0000","last_peered":"2023-10-30T12:51:12.746266+0000","last_clean":"2023-10-30T12:51:12.746266+0000","last_became_active":"2023-10-30T12:28:50.966246+0000","last_became_peered":"2023-10-30T12:28:50.966246+0000","last_unstale":"2023-10-30T12:51:12.746266+0000","last_undegraded":"2023-10-30T12:51:12.746266+0000","last_fullsized":"2023-10-30T12:51:12.746266+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":566,"log_dups_size":0,"ondisk_log_size":566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:39:33.054939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007703601,"stat_sum":{"num_bytes":8540203,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":56,"num_read_kb":18,"num_write":531,"num_write_kb":11283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"66'83","reported_seq":159,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:18.809472+0000","last_change":"2023-10-30T12:28:50.394060+0000","last_active":"2023-10-30T12:51:18.809472+0000","last_peered":"2023-10-30T12:51:18.809472+0000","last_clean":"2023-10-30T12:51:18.809472+0000","last_became_active":"2023-10-30T12:28:50.393642+0000","last_became_peered":"2023-10-30T12:28:50.393642+0000","last_unstale":"2023-10-30T12:51:18.809472+0000","last_undegraded":"2023-10-30T12:51:18.809472+0000","last_fullsized":"2023-10-30T12:51:18.809472+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:21.643882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":15,"num_write_kb":101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"65'532","reported_seq":2688,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.708661+0000","last_change":"2023-10-30T12:38:17.335584+0000","last_active":"2023-10-30T12:51:12.708661+0000","last_peered":"2023-10-30T12:51:12.708661+0000","last_clean":"2023-10-30T12:51:12.708661+0000","last_became_active":"2023-10-30T12:28:50.976343+0000","last_became_peered":"2023-10-30T12:28:50.976343+0000","last_unstale":"2023-10-30T12:51:12.708661+0000","last_undegraded":"2023-10-30T12:51:12.708661+0000","last_fullsized":"2023-10-30T12:51:12.708661+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":532,"log_dups_size":0,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:03:37.466766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091875959999999993,"stat_sum":{"num_bytes":16963416,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":44,"num_read_kb":14,"num_write":504,"num_write_kb":22707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":220,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:41.797344+0000","last_change":"2023-10-30T12:28:41.203990+0000","last_active":"2023-10-30T12:50:41.797344+0000","last_peered":"2023-10-30T12:50:41.797344+0000","last_clean":"2023-10-30T12:50:41.797344+0000","last_became_active":"2023-10-30T12:28:41.201796+0000","last_became_peered":"2023-10-30T12:28:41.201796+0000","last_unstale":"2023-10-30T12:50:41.797344+0000","last_undegraded":"2023-10-30T12:50:41.797344+0000","last_fullsized":"2023-10-30T12:50:41.797344+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:39.218257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:39.218257+0000","last_clean_scrub_stamp":"2023-10-30T12:28:39.218257+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:21:00.605462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"66'193","reported_seq":273,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.748291+0000","last_change":"2023-10-30T12:28:50.372651+0000","last_active":"2023-10-30T12:51:23.748291+0000","last_peered":"2023-10-30T12:51:23.748291+0000","last_clean":"2023-10-30T12:51:23.748291+0000","last_became_active":"2023-10-30T12:28:50.367593+0000","last_became_peered":"2023-10-30T12:28:50.367593+0000","last_unstale":"2023-10-30T12:51:23.748291+0000","last_undegraded":"2023-10-30T12:51:23.748291+0000","last_fullsized":"2023-10-30T12:51:23.748291+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:32:10.492061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194338,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":24,"num_read_kb":12326,"num_write":144,"num_write_kb":4223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"65'492","reported_seq":2637,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.709943+0000","last_change":"2023-10-30T12:38:17.285044+0000","last_active":"2023-10-30T12:51:12.709943+0000","last_peered":"2023-10-30T12:51:12.709943+0000","last_clean":"2023-10-30T12:51:12.709943+0000","last_became_active":"2023-10-30T12:29:32.997301+0000","last_became_peered":"2023-10-30T12:29:32.997301+0000","last_unstale":"2023-10-30T12:51:12.709943+0000","last_undegraded":"2023-10-30T12:51:12.709943+0000","last_fullsized":"2023-10-30T12:51:12.709943+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":492,"log_dups_size":0,"ondisk_log_size":492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:12:18.045691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001366412,"stat_sum":{"num_bytes":4230333,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":36,"num_read_kb":11,"num_write":471,"num_write_kb":13453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"66'499","reported_seq":586,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:18.810050+0000","last_change":"2023-10-30T12:28:50.392461+0000","last_active":"2023-10-30T12:51:18.810050+0000","last_peered":"2023-10-30T12:51:18.810050+0000","last_clean":"2023-10-30T12:51:18.810050+0000","last_became_active":"2023-10-30T12:28:50.391824+0000","last_became_peered":"2023-10-30T12:28:50.391824+0000","last_unstale":"2023-10-30T12:51:18.810050+0000","last_undegraded":"2023-10-30T12:51:18.810050+0000","last_fullsized":"2023-10-30T12:51:18.810050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":499,"log_dups_size":0,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:56:10.373046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":32,"num_read_kb":50322,"num_write":431,"num_write_kb":16616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"66'503","reported_seq":2644,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.731399+0000","last_change":"2023-10-30T12:38:17.643111+0000","last_active":"2023-10-30T12:51:12.731399+0000","last_peered":"2023-10-30T12:51:12.731399+0000","last_clean":"2023-10-30T12:51:12.731399+0000","last_became_active":"2023-10-30T12:28:50.952625+0000","last_became_peered":"2023-10-30T12:28:50.952625+0000","last_unstale":"2023-10-30T12:51:12.731399+0000","last_undegraded":"2023-10-30T12:51:12.731399+0000","last_fullsized":"2023-10-30T12:51:12.731399+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":503,"log_dups_size":0,"ondisk_log_size":503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:43:17.223141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026695970000000001,"stat_sum":{"num_bytes":16935229,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":29,"num_read_kb":10,"num_write":464,"num_write_kb":20216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"66'579","reported_seq":882,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:25.409837+0000","last_change":"2023-10-30T12:28:50.367547+0000","last_active":"2023-10-30T12:51:25.409837+0000","last_peered":"2023-10-30T12:51:25.409837+0000","last_clean":"2023-10-30T12:51:25.409837+0000","last_became_active":"2023-10-30T12:28:50.367230+0000","last_became_peered":"2023-10-30T12:28:50.367230+0000","last_unstale":"2023-10-30T12:51:25.409837+0000","last_undegraded":"2023-10-30T12:51:25.409837+0000","last_fullsized":"2023-10-30T12:51:25.409837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:31:42.570786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11931699,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":249,"num_read_kb":13021,"num_write":520,"num_write_kb":11989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"66'527","reported_seq":2687,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.683264+0000","last_change":"2023-10-30T12:38:17.104045+0000","last_active":"2023-10-30T12:51:23.683264+0000","last_peered":"2023-10-30T12:51:23.683264+0000","last_clean":"2023-10-30T12:51:23.683264+0000","last_became_active":"2023-10-30T12:28:50.928608+0000","last_became_peered":"2023-10-30T12:28:50.928608+0000","last_unstale":"2023-10-30T12:51:23.683264+0000","last_undegraded":"2023-10-30T12:51:23.683264+0000","last_fullsized":"2023-10-30T12:51:23.683264+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":527,"log_dups_size":0,"ondisk_log_size":527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:17:59.249824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002074275,"stat_sum":{"num_bytes":8442985,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":47,"num_read_kb":21,"num_write":493,"num_write_kb":11774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"66'493","reported_seq":572,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:46.897234+0000","last_change":"2023-10-30T12:28:50.391240+0000","last_active":"2023-10-30T12:50:46.897234+0000","last_peered":"2023-10-30T12:50:46.897234+0000","last_clean":"2023-10-30T12:50:46.897234+0000","last_became_active":"2023-10-30T12:28:50.390864+0000","last_became_peered":"2023-10-30T12:28:50.390864+0000","last_unstale":"2023-10-30T12:50:46.897234+0000","last_undegraded":"2023-10-30T12:50:46.897234+0000","last_fullsized":"2023-10-30T12:50:46.897234+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":493,"log_dups_size":0,"ondisk_log_size":493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:59:40.285830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":45183,"num_write":427,"num_write_kb":16635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"66'498","reported_seq":2645,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:24.458791+0000","last_change":"2023-10-30T12:38:17.111185+0000","last_active":"2023-10-30T12:51:24.458791+0000","last_peered":"2023-10-30T12:51:24.458791+0000","last_clean":"2023-10-30T12:51:24.458791+0000","last_became_active":"2023-10-30T12:28:50.949122+0000","last_became_peered":"2023-10-30T12:28:50.949122+0000","last_unstale":"2023-10-30T12:51:24.458791+0000","last_undegraded":"2023-10-30T12:51:24.458791+0000","last_fullsized":"2023-10-30T12:51:24.458791+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:32:08.636317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004393348,"stat_sum":{"num_bytes":12654257,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":35,"num_read_kb":65,"num_write":459,"num_write_kb":14113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"66'266","reported_seq":335,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:49.583265+0000","last_change":"2023-10-30T12:28:50.358700+0000","last_active":"2023-10-30T12:50:49.583265+0000","last_peered":"2023-10-30T12:50:49.583265+0000","last_clean":"2023-10-30T12:50:49.583265+0000","last_became_active":"2023-10-30T12:28:50.358310+0000","last_became_peered":"2023-10-30T12:28:50.358310+0000","last_unstale":"2023-10-30T12:50:49.583265+0000","last_undegraded":"2023-10-30T12:50:49.583265+0000","last_fullsized":"2023-10-30T12:50:49.583265+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":266,"log_dups_size":0,"ondisk_log_size":266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:23:30.910622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":5758,"num_write":222,"num_write_kb":4242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"65'512","reported_seq":2657,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.699965+0000","last_change":"2023-10-30T12:38:17.121187+0000","last_active":"2023-10-30T12:51:12.699965+0000","last_peered":"2023-10-30T12:51:12.699965+0000","last_clean":"2023-10-30T12:51:12.699965+0000","last_became_active":"2023-10-30T12:28:50.965626+0000","last_became_peered":"2023-10-30T12:28:50.965626+0000","last_unstale":"2023-10-30T12:51:12.699965+0000","last_undegraded":"2023-10-30T12:51:12.699965+0000","last_fullsized":"2023-10-30T12:51:12.699965+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":512,"log_dups_size":0,"ondisk_log_size":512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:44:24.255740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0079099229999999993,"stat_sum":{"num_bytes":29393213,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":33,"num_read_kb":4109,"num_write":475,"num_write_kb":37010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"66'399","reported_seq":470,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:47.791499+0000","last_change":"2023-10-30T12:28:50.395645+0000","last_active":"2023-10-30T12:50:47.791499+0000","last_peered":"2023-10-30T12:50:47.791499+0000","last_clean":"2023-10-30T12:50:47.791499+0000","last_became_active":"2023-10-30T12:28:50.395090+0000","last_became_peered":"2023-10-30T12:28:50.395090+0000","last_unstale":"2023-10-30T12:50:47.791499+0000","last_undegraded":"2023-10-30T12:50:47.791499+0000","last_fullsized":"2023-10-30T12:50:47.791499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:45:44.678383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582934,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":8195,"num_write":330,"num_write_kb":12489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"66'511","reported_seq":2673,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.700882+0000","last_change":"2023-10-30T12:38:17.104334+0000","last_active":"2023-10-30T12:51:12.700882+0000","last_peered":"2023-10-30T12:51:12.700882+0000","last_clean":"2023-10-30T12:51:12.700882+0000","last_became_active":"2023-10-30T12:28:50.929337+0000","last_became_peered":"2023-10-30T12:28:50.929337+0000","last_unstale":"2023-10-30T12:51:12.700882+0000","last_undegraded":"2023-10-30T12:51:12.700882+0000","last_fullsized":"2023-10-30T12:51:12.700882+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":511,"log_dups_size":0,"ondisk_log_size":511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:32:28.782456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023039890000000002,"stat_sum":{"num_bytes":14084554,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":50,"num_read_kb":18,"num_write":480,"num_write_kb":15463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"66'350","reported_seq":428,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.563637+0000","last_change":"2023-10-30T12:28:50.380474+0000","last_active":"2023-10-30T12:51:23.563637+0000","last_peered":"2023-10-30T12:51:23.563637+0000","last_clean":"2023-10-30T12:51:23.563637+0000","last_became_active":"2023-10-30T12:28:50.380247+0000","last_became_peered":"2023-10-30T12:28:50.380247+0000","last_unstale":"2023-10-30T12:51:23.563637+0000","last_undegraded":"2023-10-30T12:51:23.563637+0000","last_fullsized":"2023-10-30T12:51:23.563637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":350,"log_dups_size":0,"ondisk_log_size":350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:52:03.952617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":8272,"num_write":524,"num_write_kb":5251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"66'511","reported_seq":2653,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.729280+0000","last_change":"2023-10-30T12:38:17.331272+0000","last_active":"2023-10-30T12:51:12.729280+0000","last_peered":"2023-10-30T12:51:12.729280+0000","last_clean":"2023-10-30T12:51:12.729280+0000","last_became_active":"2023-10-30T12:28:50.973086+0000","last_became_peered":"2023-10-30T12:28:50.973086+0000","last_unstale":"2023-10-30T12:51:12.729280+0000","last_undegraded":"2023-10-30T12:51:12.729280+0000","last_fullsized":"2023-10-30T12:51:12.729280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":511,"log_dups_size":0,"ondisk_log_size":511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:53:48.263747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034133169999999999,"stat_sum":{"num_bytes":16794570,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":30,"num_read_kb":14,"num_write":487,"num_write_kb":22103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"66'198","reported_seq":272,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:25.415320+0000","last_change":"2023-10-30T12:28:50.386277+0000","last_active":"2023-10-30T12:51:25.415320+0000","last_peered":"2023-10-30T12:51:25.415320+0000","last_clean":"2023-10-30T12:51:25.415320+0000","last_became_active":"2023-10-30T12:28:50.385924+0000","last_became_peered":"2023-10-30T12:28:50.385924+0000","last_unstale":"2023-10-30T12:51:25.415320+0000","last_undegraded":"2023-10-30T12:51:25.415320+0000","last_fullsized":"2023-10-30T12:51:25.415320+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:35:03.837523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1958032,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":136,"num_write_kb":2002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"65'500","reported_seq":2655,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.717097+0000","last_change":"2023-10-30T12:38:17.643575+0000","last_active":"2023-10-30T12:51:12.717097+0000","last_peered":"2023-10-30T12:51:12.717097+0000","last_clean":"2023-10-30T12:51:12.717097+0000","last_became_active":"2023-10-30T12:28:50.929259+0000","last_became_peered":"2023-10-30T12:28:50.929259+0000","last_unstale":"2023-10-30T12:51:12.717097+0000","last_undegraded":"2023-10-30T12:51:12.717097+0000","last_fullsized":"2023-10-30T12:51:12.717097+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":500,"log_dups_size":0,"ondisk_log_size":500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:49:01.397563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029033230000000002,"stat_sum":{"num_bytes":8412687,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":43,"num_read_kb":22,"num_write":470,"num_write_kb":11464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"65'373","reported_seq":455,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:41.796144+0000","last_change":"2023-10-30T12:28:50.386426+0000","last_active":"2023-10-30T12:50:41.796144+0000","last_peered":"2023-10-30T12:50:41.796144+0000","last_clean":"2023-10-30T12:50:41.796144+0000","last_became_active":"2023-10-30T12:28:50.385305+0000","last_became_peered":"2023-10-30T12:28:50.385305+0000","last_unstale":"2023-10-30T12:50:41.796144+0000","last_undegraded":"2023-10-30T12:50:41.796144+0000","last_fullsized":"2023-10-30T12:50:41.796144+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":373,"log_dups_size":0,"ondisk_log_size":373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:13:05.376581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":37322,"num_write":310,"num_write_kb":12446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"66'520","reported_seq":2669,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:18.630606+0000","last_change":"2023-10-30T12:38:17.111500+0000","last_active":"2023-10-30T12:51:18.630606+0000","last_peered":"2023-10-30T12:51:18.630606+0000","last_clean":"2023-10-30T12:51:18.630606+0000","last_became_active":"2023-10-30T12:28:50.940846+0000","last_became_peered":"2023-10-30T12:28:50.940846+0000","last_unstale":"2023-10-30T12:51:18.630606+0000","last_undegraded":"2023-10-30T12:51:18.630606+0000","last_fullsized":"2023-10-30T12:51:18.630606+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":520,"log_dups_size":0,"ondisk_log_size":520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:27:33.103707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046016829999999996,"stat_sum":{"num_bytes":12660803,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":37,"num_read_kb":7,"num_write":492,"num_write_kb":18951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"66'441","reported_seq":518,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.687550+0000","last_change":"2023-10-30T12:28:50.392333+0000","last_active":"2023-10-30T12:51:23.687550+0000","last_peered":"2023-10-30T12:51:23.687550+0000","last_clean":"2023-10-30T12:51:23.687550+0000","last_became_active":"2023-10-30T12:28:50.392068+0000","last_became_peered":"2023-10-30T12:28:50.392068+0000","last_unstale":"2023-10-30T12:51:23.687550+0000","last_undegraded":"2023-10-30T12:51:23.687550+0000","last_fullsized":"2023-10-30T12:51:23.687550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":441,"log_dups_size":0,"ondisk_log_size":441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:47:24.674925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388698,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":19,"num_read_kb":11375,"num_write":386,"num_write_kb":8543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"66'523","reported_seq":2679,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.682847+0000","last_change":"2023-10-30T12:38:17.331599+0000","last_active":"2023-10-30T12:51:23.682847+0000","last_peered":"2023-10-30T12:51:23.682847+0000","last_clean":"2023-10-30T12:51:23.682847+0000","last_became_active":"2023-10-30T12:28:50.931898+0000","last_became_peered":"2023-10-30T12:28:50.931898+0000","last_unstale":"2023-10-30T12:51:23.682847+0000","last_undegraded":"2023-10-30T12:51:23.682847+0000","last_fullsized":"2023-10-30T12:51:23.682847+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":523,"log_dups_size":0,"ondisk_log_size":523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:49:23.804972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003570148,"stat_sum":{"num_bytes":4255498,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":44,"num_read_kb":18,"num_write":498,"num_write_kb":10461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"66'553","reported_seq":626,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:24.395115+0000","last_change":"2023-10-30T12:28:50.365055+0000","last_active":"2023-10-30T12:51:24.395115+0000","last_peered":"2023-10-30T12:51:24.395115+0000","last_clean":"2023-10-30T12:51:24.395115+0000","last_became_active":"2023-10-30T12:28:50.364797+0000","last_became_peered":"2023-10-30T12:28:50.364797+0000","last_unstale":"2023-10-30T12:51:24.395115+0000","last_undegraded":"2023-10-30T12:51:24.395115+0000","last_fullsized":"2023-10-30T12:51:24.395115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":553,"log_dups_size":0,"ondisk_log_size":553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:06:23.067895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":10,"num_read_kb":8489,"num_write":689,"num_write_kb":13252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"66'528","reported_seq":2689,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.685808+0000","last_change":"2023-10-30T12:38:17.331931+0000","last_active":"2023-10-30T12:51:23.685808+0000","last_peered":"2023-10-30T12:51:23.685808+0000","last_clean":"2023-10-30T12:51:23.685808+0000","last_became_active":"2023-10-30T12:28:50.935657+0000","last_became_peered":"2023-10-30T12:28:50.935657+0000","last_unstale":"2023-10-30T12:51:23.685808+0000","last_undegraded":"2023-10-30T12:51:23.685808+0000","last_fullsized":"2023-10-30T12:51:23.685808+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":528,"log_dups_size":0,"ondisk_log_size":528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:42:39.657789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038543219999999999,"stat_sum":{"num_bytes":16855474,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":49,"num_read_kb":4112,"num_write":496,"num_write_kb":17584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"66'244","reported_seq":324,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.643888+0000","last_change":"2023-10-30T12:28:50.393465+0000","last_active":"2023-10-30T12:51:12.643888+0000","last_peered":"2023-10-30T12:51:12.643888+0000","last_clean":"2023-10-30T12:51:12.643888+0000","last_became_active":"2023-10-30T12:28:50.392807+0000","last_became_peered":"2023-10-30T12:28:50.392807+0000","last_unstale":"2023-10-30T12:51:12.643888+0000","last_undegraded":"2023-10-30T12:51:12.643888+0000","last_fullsized":"2023-10-30T12:51:12.643888+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":244,"log_dups_size":0,"ondisk_log_size":244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:35:37.976956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":24596,"num_write":178,"num_write_kb":8294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"65'481","reported_seq":2631,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.730761+0000","last_change":"2023-10-30T12:38:17.325460+0000","last_active":"2023-10-30T12:51:12.730761+0000","last_peered":"2023-10-30T12:51:12.730761+0000","last_clean":"2023-10-30T12:51:12.730761+0000","last_became_active":"2023-10-30T12:28:50.954291+0000","last_became_peered":"2023-10-30T12:28:50.954291+0000","last_unstale":"2023-10-30T12:51:12.730761+0000","last_undegraded":"2023-10-30T12:51:12.730761+0000","last_fullsized":"2023-10-30T12:51:12.730761+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":481,"log_dups_size":0,"ondisk_log_size":481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:58:33.115877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047974480000000002,"stat_sum":{"num_bytes":12640290,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":38,"num_read_kb":15,"num_write":450,"num_write_kb":14582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"65'505","reported_seq":2666,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.715029+0000","last_change":"2023-10-30T12:38:17.120085+0000","last_active":"2023-10-30T12:51:12.715029+0000","last_peered":"2023-10-30T12:51:12.715029+0000","last_clean":"2023-10-30T12:51:12.715029+0000","last_became_active":"2023-10-30T12:28:50.962817+0000","last_became_peered":"2023-10-30T12:28:50.962817+0000","last_unstale":"2023-10-30T12:51:12.715029+0000","last_undegraded":"2023-10-30T12:51:12.715029+0000","last_fullsized":"2023-10-30T12:51:12.715029+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":505,"log_dups_size":0,"ondisk_log_size":505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:14:10.476055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070898599999999999,"stat_sum":{"num_bytes":8463933,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":50,"num_read_kb":25,"num_write":479,"num_write_kb":14576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"65'173","reported_seq":249,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.684872+0000","last_change":"2023-10-30T12:28:50.402179+0000","last_active":"2023-10-30T12:51:12.684872+0000","last_peered":"2023-10-30T12:51:12.684872+0000","last_clean":"2023-10-30T12:51:12.684872+0000","last_became_active":"2023-10-30T12:28:50.401677+0000","last_became_peered":"2023-10-30T12:28:50.401677+0000","last_unstale":"2023-10-30T12:51:12.684872+0000","last_undegraded":"2023-10-30T12:51:12.684872+0000","last_fullsized":"2023-10-30T12:51:12.684872+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:39:44.432214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":20498,"num_write":107,"num_write_kb":8247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"66'474","reported_seq":2629,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:22.381301+0000","last_change":"2023-10-30T12:38:17.119058+0000","last_active":"2023-10-30T12:51:22.381301+0000","last_peered":"2023-10-30T12:51:22.381301+0000","last_clean":"2023-10-30T12:51:22.381301+0000","last_became_active":"2023-10-30T12:28:50.967591+0000","last_became_peered":"2023-10-30T12:28:50.967591+0000","last_unstale":"2023-10-30T12:51:22.381301+0000","last_undegraded":"2023-10-30T12:51:22.381301+0000","last_fullsized":"2023-10-30T12:51:22.381301+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":474,"log_dups_size":0,"ondisk_log_size":474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:15:53.272647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081968709999999997,"stat_sum":{"num_bytes":10507344,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":44,"num_read_kb":1610,"num_write":436,"num_write_kb":22884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"66'363","reported_seq":450,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:22.402277+0000","last_change":"2023-10-30T12:28:50.392617+0000","last_active":"2023-10-30T12:51:22.402277+0000","last_peered":"2023-10-30T12:51:22.402277+0000","last_clean":"2023-10-30T12:51:22.402277+0000","last_became_active":"2023-10-30T12:28:50.392021+0000","last_became_peered":"2023-10-30T12:28:50.392021+0000","last_unstale":"2023-10-30T12:51:22.402277+0000","last_undegraded":"2023-10-30T12:51:22.402277+0000","last_fullsized":"2023-10-30T12:51:22.402277+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:52:30.621125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":57384,"num_write":298,"num_write_kb":16576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"65'482","reported_seq":2638,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.744064+0000","last_change":"2023-10-30T12:38:17.643950+0000","last_active":"2023-10-30T12:51:12.744064+0000","last_peered":"2023-10-30T12:51:12.744064+0000","last_clean":"2023-10-30T12:51:12.744064+0000","last_became_active":"2023-10-30T12:28:50.967186+0000","last_became_peered":"2023-10-30T12:28:50.967186+0000","last_unstale":"2023-10-30T12:51:12.744064+0000","last_undegraded":"2023-10-30T12:51:12.744064+0000","last_fullsized":"2023-10-30T12:51:12.744064+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":482,"log_dups_size":0,"ondisk_log_size":482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:48:20.159300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003136738,"stat_sum":{"num_bytes":18043,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":17,"num_write":445,"num_write_kb":3039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"65'227","reported_seq":303,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:46.350266+0000","last_change":"2023-10-30T12:28:50.363230+0000","last_active":"2023-10-30T12:50:46.350266+0000","last_peered":"2023-10-30T12:50:46.350266+0000","last_clean":"2023-10-30T12:50:46.350266+0000","last_became_active":"2023-10-30T12:28:50.362929+0000","last_became_peered":"2023-10-30T12:28:50.362929+0000","last_unstale":"2023-10-30T12:50:46.350266+0000","last_undegraded":"2023-10-30T12:50:46.350266+0000","last_fullsized":"2023-10-30T12:50:46.350266+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:41:27.760578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":9,"num_read_kb":32768,"num_write":162,"num_write_kb":8341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"66'542","reported_seq":2685,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.687065+0000","last_change":"2023-10-30T12:38:17.331548+0000","last_active":"2023-10-30T12:51:23.687065+0000","last_peered":"2023-10-30T12:51:23.687065+0000","last_clean":"2023-10-30T12:51:23.687065+0000","last_became_active":"2023-10-30T12:28:50.937458+0000","last_became_peered":"2023-10-30T12:28:50.937458+0000","last_unstale":"2023-10-30T12:51:23.687065+0000","last_undegraded":"2023-10-30T12:51:23.687065+0000","last_fullsized":"2023-10-30T12:51:23.687065+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":542,"log_dups_size":0,"ondisk_log_size":542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:25:59.947416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003544045,"stat_sum":{"num_bytes":16835848,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":30,"num_read_kb":11,"num_write":511,"num_write_kb":19769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"66'443","reported_seq":537,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:49.583519+0000","last_change":"2023-10-30T12:28:50.386782+0000","last_active":"2023-10-30T12:50:49.583519+0000","last_peered":"2023-10-30T12:50:49.583519+0000","last_clean":"2023-10-30T12:50:49.583519+0000","last_became_active":"2023-10-30T12:28:50.386522+0000","last_became_peered":"2023-10-30T12:28:50.386522+0000","last_unstale":"2023-10-30T12:50:49.583519+0000","last_undegraded":"2023-10-30T12:50:49.583519+0000","last_fullsized":"2023-10-30T12:50:49.583519+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":443,"log_dups_size":0,"ondisk_log_size":443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:44:03.517641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583494,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":64,"num_read_kb":24634,"num_write":377,"num_write_kb":12490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":759200777,"num_objects":915,"num_object_clones":0,"num_object_copies":1830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":915,"num_whiteouts":0,"num_read":2811,"num_read_kb":100236,"num_write":31405,"num_write_kb":1046937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1521901568,"data_stored":1518401554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33391,"ondisk_log_size":33391,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":303239465,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1432,"num_read_kb":712811,"num_write":10434,"num_write_kb":304769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":78,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":606633984,"data_stored":606478930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11693,"ondisk_log_size":11693,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542422,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":389288,"kb_used_data":322016,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93982552,"statfs":{"total":96636764160,"available":96238133248,"internally_reserved":0,"allocated":329744384,"data_stored":328817986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":68876747},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.78300000000000003,"15min":0.81799999999999995},"min":{"1min":0.52500000000000002,"5min":0.45900000000000002,"15min":0.40000000000000002},"max":{"1min":1.093,"5min":1.2050000000000001,"15min":1.595},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.78400000000000003,"15min":0.84199999999999997},"min":{"1min":0.49099999999999999,"5min":0.441,"15min":0.40999999999999998},"max":{"1min":1.113,"5min":1.302,"15min":1.4830000000000001},"last":0.57299999999999995}]},{"osd":1,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.85099999999999998,"15min":0.88800000000000001},"min":{"1min":0.57199999999999995,"5min":0.52700000000000002,"15min":0.441},"max":{"1min":1.2709999999999999,"5min":3.4329999999999998,"15min":3.4329999999999998},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.85399999999999998,"15min":0.88500000000000001},"min":{"1min":0.56999999999999995,"5min":0.505,"15min":0.505},"max":{"1min":1.1419999999999999,"5min":1.383,"15min":1.3919999999999999},"last":0.90100000000000002}]},{"osd":2,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.89000000000000001,"15min":0.92200000000000004},"min":{"1min":0.58899999999999997,"5min":0.55100000000000005,"15min":0.45900000000000002},"max":{"1min":1.163,"5min":3.4820000000000002,"15min":3.4820000000000002},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.877,"15min":0.83699999999999997},"min":{"1min":0.65000000000000002,"5min":0.52800000000000002,"15min":0.47099999999999997},"max":{"1min":1.2330000000000001,"5min":1.4570000000000001,"15min":1.6639999999999999},"last":0.73799999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":1.0069999999999999,"15min":0.96899999999999997},"min":{"1min":0.61299999999999999,"5min":0.60999999999999999,"15min":0.45500000000000002},"max":{"1min":1.1439999999999999,"5min":3.5190000000000001,"15min":3.5190000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.95599999999999996,"15min":0.99399999999999999},"min":{"1min":0.72899999999999998,"5min":0.54500000000000004,"15min":0.441},"max":{"1min":1.3220000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.94399999999999995}]},{"osd":4,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.97599999999999998,"15min":1.002},"min":{"1min":0.63800000000000001,"5min":0.42499999999999999,"15min":0.34399999999999997},"max":{"1min":1.2070000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":1.028},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.999,"15min":1.042},"min":{"1min":0.61799999999999999,"5min":0.45000000000000001,"15min":0.33800000000000002},"max":{"1min":1.1910000000000001,"5min":1.746,"15min":1.746},"last":1.0489999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.016,"15min":0.96099999999999997},"min":{"1min":0.76700000000000002,"5min":0.53700000000000003,"15min":0.43099999999999999},"max":{"1min":1.2949999999999999,"5min":1.6160000000000001,"15min":1.7549999999999999},"last":1.081},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.117,"15min":1.0680000000000001},"min":{"1min":0.71499999999999997,"5min":0.57899999999999996,"15min":0.35999999999999999},"max":{"1min":1.2370000000000001,"5min":2.1600000000000001,"15min":2.1600000000000001},"last":1.1990000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.141,"15min":1.123},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.39000000000000001},"max":{"1min":1.216,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.129},{"interface":"front","average":{"1min":1,"5min":1.1259999999999999,"15min":1.167},"min":{"1min":0.68200000000000005,"5min":0.56299999999999994,"15min":0.47199999999999998},"max":{"1min":1.3300000000000001,"5min":3.8140000000000001,"15min":3.8140000000000001},"last":0.998}]}]},{"osd":6,"up_from":14,"seq":60129542422,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":296948,"kb_used_data":233900,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94074892,"statfs":{"total":96636764160,"available":96332689408,"internally_reserved":0,"allocated":239513600,"data_stored":238814643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":64551368},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.83299999999999996,"15min":0.85999999999999999},"min":{"1min":0.52900000000000003,"5min":0.46600000000000003,"15min":0.35599999999999998},"max":{"1min":1.155,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.77000000000000002,"15min":0.78800000000000003},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.42999999999999999},"max":{"1min":1.05,"5min":1.109,"15min":1.214},"last":0.92400000000000004}]},{"osd":1,"last update":"Mon Oct 30 12:50:23 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.79900000000000004,"15min":0.81299999999999994},"min":{"1min":0.57299999999999995,"5min":0.42299999999999999,"15min":0.317},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.3140000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.89000000000000001,"15min":0.88800000000000001},"min":{"1min":0.55800000000000005,"5min":0.51100000000000001,"15min":0.40999999999999998},"max":{"1min":1.149,"5min":1.3620000000000001,"15min":1.423},"last":0.78900000000000003}]},{"osd":2,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.93600000000000005,"15min":0.97499999999999998},"min":{"1min":0.495,"5min":0.495,"15min":0.38800000000000001},"max":{"1min":1.373,"5min":1.47,"15min":4.3360000000000003},"last":1.04},{"interface":"front","average":{"1min":1.0409999999999999,"5min":0.97999999999999998,"15min":1.0089999999999999},"min":{"1min":0.65000000000000002,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.502,"5min":1.502,"15min":1.5629999999999999},"last":1.2390000000000001}]},{"osd":3,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.996,"15min":1.042},"min":{"1min":0.64700000000000002,"5min":0.59099999999999997,"15min":0.58999999999999997},"max":{"1min":1.3340000000000001,"5min":1.5509999999999999,"15min":1.6459999999999999},"last":0.83999999999999997},{"interface":"front","average":{"1min":1.02,"5min":1.0680000000000001,"15min":1.101},"min":{"1min":0.72299999999999998,"5min":0.63300000000000001,"15min":0.59699999999999998},"max":{"1min":1.417,"5min":1.552,"15min":1.7969999999999999},"last":1.095}]},{"osd":4,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.149,"15min":1.075},"min":{"1min":0.72599999999999998,"5min":0.69899999999999995,"15min":0.51500000000000001},"max":{"1min":1.585,"5min":1.6499999999999999,"15min":1.7649999999999999},"last":1.5229999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.89400000000000002,"15min":0.96499999999999997},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.42099999999999999},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.881},"last":1.2030000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":1.0720000000000001,"15min":1.087},"min":{"1min":0.54100000000000004,"5min":0.53200000000000003,"15min":0.48399999999999999},"max":{"1min":1.2849999999999999,"5min":1.75,"15min":2.2770000000000001},"last":1.27},{"interface":"front","average":{"1min":1.056,"5min":1.159,"15min":1.0649999999999999},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.46999999999999997},"max":{"1min":1.4610000000000001,"5min":1.6950000000000001,"15min":4.7850000000000001},"last":1.1359999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.036,"15min":1.0029999999999999},"min":{"1min":0.68799999999999994,"5min":0.64100000000000001,"15min":0.46700000000000003},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.825},"last":1.417},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.0629999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.48999999999999999},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.895},"last":1.3169999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542422,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":338824,"kb_used_data":271552,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94033016,"statfs":{"total":96636764160,"available":96289808384,"internally_reserved":0,"allocated":278069248,"data_stored":277156365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.78400000000000003,"15min":0.80300000000000005},"min":{"1min":0.56299999999999994,"5min":0.33900000000000002,"15min":0.29399999999999998},"max":{"1min":1.1970000000000001,"5min":1.238,"15min":2.4729999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.79700000000000004,"15min":0.80400000000000005},"min":{"1min":0.61199999999999999,"5min":0.39200000000000002,"15min":0.33800000000000002},"max":{"1min":1.2390000000000001,"5min":1.3009999999999999,"15min":1.8759999999999999},"last":0.84299999999999997}]},{"osd":1,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.84199999999999997,"15min":0.84399999999999997},"min":{"1min":0.57399999999999995,"5min":0.40999999999999998,"15min":0.38900000000000001},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":3.0190000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.58499999999999996,"5min":0.44800000000000001,"15min":0.373},"max":{"1min":1.264,"5min":1.276,"15min":1.4870000000000001},"last":0.91800000000000004}]},{"osd":2,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.88400000000000001,"15min":0.84699999999999998},"min":{"1min":0.67300000000000004,"5min":0.55400000000000005,"15min":0.38400000000000001},"max":{"1min":1.157,"5min":1.264,"15min":2.2949999999999999},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.81699999999999995,"15min":0.876},"min":{"1min":0.63600000000000001,"5min":0.434,"15min":0.434},"max":{"1min":1.1140000000000001,"5min":1.171,"15min":1.5800000000000001},"last":0.88600000000000001}]},{"osd":3,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.91800000000000004,"15min":0.93700000000000006},"min":{"1min":0.57199999999999995,"5min":0.53800000000000003,"15min":0.47899999999999998},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.756},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.93600000000000005,"15min":0.94599999999999995},"min":{"1min":0.755,"5min":0.498,"15min":0.41399999999999998},"max":{"1min":1.204,"5min":1.4119999999999999,"15min":2.3679999999999999},"last":1.0680000000000001}]},{"osd":4,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.84499999999999997,"15min":0.90900000000000003},"min":{"1min":0.47499999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.304,"5min":1.304,"15min":3.1190000000000002},"last":0.81100000000000005},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.083,"15min":1.091},"min":{"1min":0.59499999999999997,"5min":0.58699999999999997,"15min":0.50900000000000001},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":2.645},"last":1.026}]},{"osd":6,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.038,"15min":1.042},"min":{"1min":0.51100000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.466,"5min":1.466,"15min":3.1859999999999999},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.056,"15min":1.1060000000000001},"min":{"1min":0.67000000000000004,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.3260000000000001,"5min":1.47,"15min":3.371},"last":1}]},{"osd":7,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.0609999999999999,"15min":1.0800000000000001},"min":{"1min":0.65200000000000002,"5min":0.51400000000000001,"15min":0.47299999999999998},"max":{"1min":1.494,"5min":1.508,"15min":3.5310000000000001},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.282,"5min":1.1970000000000001,"15min":1.2310000000000001},"min":{"1min":0.96799999999999997,"5min":0.64700000000000002,"15min":0.61299999999999999},"max":{"1min":1.605,"5min":1.605,"15min":3.6930000000000001},"last":1.3160000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542421,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369848,"kb_used_data":294256,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94001992,"statfs":{"total":96636764160,"available":96258039808,"internally_reserved":0,"allocated":301318144,"data_stored":300442399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:15 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.81799999999999995,"15min":0.92600000000000005},"min":{"1min":0.48099999999999998,"5min":0.34799999999999998,"15min":0.31},"max":{"1min":1.204,"5min":1.2509999999999999,"15min":11.273},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.92400000000000004,"15min":0.98799999999999999},"min":{"1min":0.58099999999999996,"5min":0.44,"15min":0.44},"max":{"1min":1.411,"5min":1.4590000000000001,"15min":7.1289999999999996},"last":0.81899999999999995}]},{"osd":1,"last update":"Mon Oct 30 12:51:15 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.81499999999999995,"15min":0.90500000000000003},"min":{"1min":0.52800000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0049999999999999,"5min":5.1130000000000004,"15min":6.9199999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.871,"5min":0.84199999999999997,"15min":0.90000000000000002},"min":{"1min":0.63900000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.246,"5min":1.246,"15min":4.75},"last":0.76900000000000002}]},{"osd":2,"last update":"Mon Oct 30 12:51:15 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.84199999999999997,"15min":0.84899999999999998},"min":{"1min":0.51700000000000002,"5min":0.41899999999999998,"15min":0.36899999999999999},"max":{"1min":1.175,"5min":1.21,"15min":2.6970000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.996,"5min":0.91200000000000003,"15min":1.0569999999999999},"min":{"1min":0.60999999999999999,"5min":0.54400000000000004,"15min":0.52900000000000003},"max":{"1min":1.361,"5min":1.361,"15min":18.917999999999999},"last":0.999}]},{"osd":3,"last update":"Mon Oct 30 12:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":0.96599999999999997,"15min":0.997},"min":{"1min":0.71399999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.258,"5min":5.1719999999999997,"15min":7.1879999999999997},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.90800000000000003,"15min":1},"min":{"1min":0.57399999999999995,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.125,"5min":1.3600000000000001,"15min":8.3360000000000003},"last":0.71899999999999997}]},{"osd":5,"last update":"Mon Oct 30 12:51:15 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.88300000000000001,"15min":1.002},"min":{"1min":0.56299999999999994,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.159,"5min":1.3300000000000001,"15min":6.0759999999999996},"last":0.92500000000000004},{"interface":"front","average":{"1min":1.119,"5min":1.105,"15min":1.1890000000000001},"min":{"1min":0.89500000000000002,"5min":0.71899999999999997,"15min":0.52600000000000002},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":18.994},"last":1.0329999999999999}]},{"osd":6,"last update":"Mon Oct 30 12:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.0880000000000001,"15min":1.093},"min":{"1min":0.70199999999999996,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.429,"5min":1.429,"15min":6.1580000000000004},"last":1.115},{"interface":"front","average":{"1min":1.016,"5min":1.0169999999999999,"15min":1.1240000000000001},"min":{"1min":0.629,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.28,"5min":5.218,"15min":7.0549999999999997},"last":0.95999999999999996}]},{"osd":7,"last update":"Mon Oct 30 12:51:15 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.0980000000000001,"15min":1.258},"min":{"1min":0.75800000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.4339999999999999,"5min":5.7160000000000002,"15min":18.015999999999998},"last":1.091},{"interface":"front","average":{"1min":1.236,"5min":1.2050000000000001,"15min":1.2769999999999999},"min":{"1min":1.093,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.4530000000000001,"5min":5.6609999999999996,"15min":19.052},"last":1.226}]}]},{"osd":3,"up_from":14,"seq":60129542422,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388864,"kb_used_data":309112,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93982976,"statfs":{"total":96636764160,"available":96238567424,"internally_reserved":0,"allocated":316530688,"data_stored":315623973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":81656264},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.39900000000000002,"15min":0.41399999999999998},"min":{"1min":0.29399999999999998,"5min":0.23699999999999999,"15min":0.23400000000000001},"max":{"1min":0.58999999999999997,"5min":0.77900000000000003,"15min":1.6899999999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51600000000000001,"15min":0.53300000000000003},"min":{"1min":0.40000000000000002,"5min":0.34300000000000003,"15min":0.32700000000000001},"max":{"1min":0.746,"5min":0.871,"15min":1.728},"last":0.55200000000000005}]},{"osd":1,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.42899999999999999,"15min":0.42999999999999999},"min":{"1min":0.309,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":1.2909999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.496,"15min":0.50600000000000001},"min":{"1min":0.34499999999999997,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.64600000000000002,"5min":0.80600000000000005,"15min":1.3260000000000001},"last":0.501}]},{"osd":2,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.496,"15min":0.502},"min":{"1min":0.37,"5min":0.30099999999999999,"15min":0.29299999999999998},"max":{"1min":0.69899999999999995,"5min":0.78600000000000003,"15min":1.099},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55900000000000005,"15min":0.56699999999999995},"min":{"1min":0.41999999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73599999999999999,"5min":0.82599999999999996,"15min":1.2509999999999999},"last":0.53100000000000003}]},{"osd":4,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.85199999999999998,"15min":0.83099999999999996},"min":{"1min":0.63900000000000001,"5min":0.55400000000000005,"15min":0.40400000000000003},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":2.202},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.82999999999999996,"15min":0.86499999999999999},"min":{"1min":0.70599999999999996,"5min":0.48499999999999999,"15min":0.46800000000000003},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":2.4340000000000002},"last":0.89100000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.81499999999999995,"15min":0.86299999999999999},"min":{"1min":0.66200000000000003,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.169,"5min":1.169,"15min":14.103999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.875,"5min":0.876,"15min":0.86499999999999999},"min":{"1min":0.63900000000000001,"5min":0.55000000000000004,"15min":0.499},"max":{"1min":1.2170000000000001,"5min":1.2869999999999999,"15min":4.4859999999999998},"last":0.85699999999999998}]},{"osd":6,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.85299999999999998,"15min":0.83499999999999996},"min":{"1min":0.64700000000000002,"5min":0.56999999999999995,"15min":0.53100000000000003},"max":{"1min":1.087,"5min":1.2450000000000001,"15min":2.2330000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.87,"15min":0.82899999999999996},"min":{"1min":0.77800000000000002,"5min":0.54100000000000004,"15min":0.45100000000000001},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":4.4409999999999998},"last":0.86699999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.875,"15min":0.86499999999999999},"min":{"1min":0.67500000000000004,"5min":0.57299999999999995,"15min":0.57099999999999995},"max":{"1min":1.1499999999999999,"5min":1.2509999999999999,"15min":3.5459999999999998},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.86399999999999999,"15min":0.84699999999999998},"min":{"1min":0.65700000000000003,"5min":0.57399999999999995,"15min":0.45700000000000002},"max":{"1min":1.1259999999999999,"5min":1.2230000000000001,"15min":6.1360000000000001},"last":0.91800000000000004}]}]},{"osd":2,"up_from":14,"seq":60129542422,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":199872,"kb_used_data":142328,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94171968,"statfs":{"total":96636764160,"available":96432095232,"internally_reserved":0,"allocated":145743872,"data_stored":145027826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.45100000000000001,"15min":0.441},"min":{"1min":0.25,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.59499999999999997,"5min":0.69399999999999995,"15min":0.81200000000000006},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.5,"15min":0.51100000000000001},"min":{"1min":0.37,"5min":0.33300000000000002,"15min":0.31900000000000001},"max":{"1min":0.63,"5min":0.71299999999999997,"15min":1.298},"last":0.47099999999999997}]},{"osd":1,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.47899999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.60199999999999998,"5min":0.65500000000000003,"15min":0.83699999999999997},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.52500000000000002,"15min":0.504},"min":{"1min":0.38900000000000001,"5min":0.33700000000000002,"15min":0.29999999999999999},"max":{"1min":0.61699999999999999,"5min":0.74099999999999999,"15min":1.2250000000000001},"last":0.53800000000000003}]},{"osd":3,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.47399999999999998,"15min":0.54800000000000004},"min":{"1min":0.315,"5min":0.311,"15min":0.311},"max":{"1min":0.58599999999999997,"5min":0.75,"15min":1.673},"last":0.498},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57299999999999995,"15min":0.57399999999999995},"min":{"1min":0.41799999999999998,"5min":0.40200000000000002,"15min":0.35699999999999998},"max":{"1min":0.67900000000000005,"5min":0.79100000000000004,"15min":1.597},"last":0.59499999999999997}]},{"osd":4,"last update":"Mon Oct 30 12:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.74299999999999999,"15min":0.82599999999999996},"min":{"1min":0.47499999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.91100000000000003,"5min":1.0569999999999999,"15min":1.377},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.755,"15min":0.79200000000000004},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.998,"5min":1.0640000000000001,"15min":1.4159999999999999},"last":1.0169999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.84099999999999997,"15min":0.84799999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.093,"5min":1.2430000000000001,"15min":1.78},"last":1.002},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91400000000000003},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.1060000000000001,"5min":1.2729999999999999,"15min":1.7430000000000001},"last":1.0309999999999999}]},{"osd":6,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.86199999999999999,"15min":0.85799999999999998},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.60299999999999998},"max":{"1min":1.095,"5min":1.177,"15min":1.3799999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.86599999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.53300000000000003},"max":{"1min":1.097,"5min":1.1799999999999999,"15min":1.8109999999999999},"last":0.93300000000000005}]},{"osd":7,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.82899999999999996,"15min":0.83099999999999996},"min":{"1min":0.58799999999999997,"5min":0.58299999999999996,"15min":0.48799999999999999},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.4159999999999999},"last":1.0429999999999999},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.85399999999999998,"15min":0.83299999999999996},"min":{"1min":0.75900000000000001,"5min":0.51800000000000002,"15min":0.50800000000000001},"max":{"1min":1.206,"5min":1.206,"15min":1.3049999999999999},"last":1.0780000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542422,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":278984,"kb_used_data":233920,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94092856,"statfs":{"total":96636764160,"available":96351084544,"internally_reserved":0,"allocated":239534080,"data_stored":238814180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:50:34 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.40400000000000003,"15min":0.45400000000000001},"min":{"1min":0.28399999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.63600000000000001,"5min":0.65200000000000002,"15min":11.958},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.495,"15min":0.55700000000000005},"min":{"1min":0.34699999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.73599999999999999,"5min":0.73799999999999999,"15min":12.077},"last":0.53800000000000003}]},{"osd":2,"last update":"Mon Oct 30 12:50:34 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45800000000000002,"15min":0.46000000000000002},"min":{"1min":0.29699999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.121,"5min":1.121,"15min":1.623},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52500000000000002,"15min":0.53300000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.31900000000000001},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.5389999999999999},"last":0.60599999999999998}]},{"osd":3,"last update":"Mon Oct 30 12:50:34 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55400000000000005,"15min":0.56699999999999995},"min":{"1min":0.378,"5min":0.31,"15min":0.31},"max":{"1min":1.216,"5min":1.337,"15min":3.694},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.498,"15min":0.54700000000000004},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.66900000000000004,"5min":0.77200000000000002,"15min":13.199999999999999},"last":0.49299999999999999}]},{"osd":4,"last update":"Mon Oct 30 12:50:34 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.82699999999999996,"15min":0.83199999999999996},"min":{"1min":0.54700000000000004,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1519999999999999,"5min":1.3140000000000001,"15min":2.419},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.871,"15min":0.86699999999999999},"min":{"1min":0.66300000000000003,"5min":0.60899999999999999,"15min":0.52100000000000002},"max":{"1min":1.095,"5min":1.103,"15min":2.282},"last":0.91700000000000004}]},{"osd":5,"last update":"Mon Oct 30 12:50:34 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.80000000000000004,"15min":0.81599999999999995},"min":{"1min":0.61699999999999999,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.01,"5min":1.343,"15min":1.343},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.79000000000000004,"15min":0.85599999999999998},"min":{"1min":0.59099999999999997,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.115,"5min":1.29,"15min":1.29},"last":0.77000000000000002}]},{"osd":6,"last update":"Mon Oct 30 12:50:34 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82799999999999996,"15min":0.79800000000000004},"min":{"1min":0.56399999999999995,"5min":0.52800000000000002,"15min":0.51200000000000001},"max":{"1min":1.3660000000000001,"5min":1.371,"15min":1.371},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.89400000000000002,"15min":0.86699999999999999},"min":{"1min":0.67400000000000004,"5min":0.498,"15min":0.498},"max":{"1min":1.417,"5min":1.417,"15min":2.464},"last":0.97099999999999997}]},{"osd":7,"last update":"Mon Oct 30 12:50:34 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.81100000000000005,"15min":0.83199999999999996},"min":{"1min":0.61899999999999999,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0349999999999999,"5min":1.1659999999999999,"15min":2.5169999999999999},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.875,"5min":0.88200000000000001,"15min":0.90700000000000003},"min":{"1min":0.76900000000000002,"5min":0.61899999999999999,"15min":0.58399999999999996},"max":{"1min":1.133,"5min":1.133,"15min":2.5510000000000002},"last":0.78700000000000003}]}]},{"osd":0,"up_from":14,"seq":60129542422,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350128,"kb_used_data":278696,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94021712,"statfs":{"total":96636764160,"available":96278233088,"internally_reserved":0,"allocated":285384704,"data_stored":284463559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 12:50:26 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.44400000000000001,"15min":0.44900000000000001},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.93700000000000006},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50900000000000001,"15min":0.504},"min":{"1min":0.35299999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.67600000000000005,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.56100000000000005}]},{"osd":2,"last update":"Mon Oct 30 12:50:26 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.45600000000000002,"15min":0.504},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.629,"5min":0.64900000000000002,"15min":0.97199999999999998},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.55200000000000005,"15min":0.52800000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.27400000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":1.0149999999999999},"last":0.57699999999999996}]},{"osd":3,"last update":"Mon Oct 30 12:50:26 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.46899999999999997,"15min":0.51400000000000001},"min":{"1min":0.30499999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.73499999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.48599999999999999,"15min":0.496},"min":{"1min":0.316,"5min":0.31,"15min":0.31},"max":{"1min":0.74199999999999999,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.59299999999999997}]},{"osd":4,"last update":"Mon Oct 30 12:50:26 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86599999999999999,"15min":0.85499999999999998},"min":{"1min":0.70899999999999996,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.127,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.88500000000000001,"15min":0.85699999999999998},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.48299999999999998},"max":{"1min":1.143,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.92400000000000004}]},{"osd":5,"last update":"Mon Oct 30 12:50:26 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.82299999999999995,"15min":0.84199999999999997},"min":{"1min":0.60499999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.095,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.83299999999999996,"15min":0.87},"min":{"1min":0.58699999999999997,"5min":0.53200000000000003,"15min":0.52800000000000002},"max":{"1min":1.889,"5min":1.889,"15min":1.889},"last":0.89000000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:50:26 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.78400000000000003,"15min":0.79500000000000004},"min":{"1min":0.58299999999999996,"5min":0.56299999999999994,"15min":0.50900000000000001},"max":{"1min":1.0409999999999999,"5min":1.1990000000000001,"15min":2.6480000000000001},"last":1.071},{"interface":"front","average":{"1min":0.871,"5min":0.79300000000000004,"15min":0.82099999999999995},"min":{"1min":0.61199999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0409999999999999,"5min":1.1559999999999999,"15min":5.0099999999999998},"last":1.0249999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:50:26 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.87,"15min":0.88200000000000001},"min":{"1min":0.68899999999999995,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.155,"5min":1.401,"15min":3.3359999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.92400000000000004,"15min":0.94399999999999995},"min":{"1min":0.75700000000000001,"5min":0.59699999999999998,"15min":0.56299999999999994},"max":{"1min":1.1890000000000001,"5min":1.3500000000000001,"15min":3.3580000000000001},"last":1.0069999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62930944,"data_stored":62915220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":83910656,"data_stored":83887024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":37122048,"data_stored":37098251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82608128,"data_stored":82586990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":76099584,"data_stored":76085923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":62935040,"data_stored":62915938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":89997312,"data_stored":89978920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":111030272,"data_stored":111010664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":221097984,"data_stored":220570894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":154857472,"data_stored":154539460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":107859968,"data_stored":107542498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":233156608,"data_stored":232649287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":224452608,"data_stored":223968780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":214372352,"data_stored":213853350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":148750336,"data_stored":148448027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":217354240,"data_stored":216829258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:51:31.983 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:51:32.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.158+0000 7fe04ba17640 1 Processor -- start 2023-10-30T12:51:32.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.162+0000 7fe04ba17640 1 -- start start 2023-10-30T12:51:32.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.162+0000 7fe04ba17640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe0440c6bd0 0x7fe04406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:32.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.162+0000 7fe04ba17640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe044071110 0x7fe044158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:32.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.162+0000 7fe04ba17640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe044066cf0 con 0x7fe04406f7a0 2023-10-30T12:51:32.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.162+0000 7fe04ba17640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0440700c0 con 0x7fe044071110 2023-10-30T12:51:32.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.162+0000 7fe04ba17640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0440c70d0 con 0x7fe0440c6bd0 2023-10-30T12:51:32.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.162+0000 7fe049fb4640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe044071110 0x7fe044158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:32.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.162+0000 7fe049fb4640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe044071110 0x7fe044158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47326/0 (socket says 172.21.15.89:47326) 2023-10-30T12:51:32.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.162+0000 7fe049fb4640 1 -- 172.21.15.89:0/1034591402 learned_addr learned my addr 172.21.15.89:0/1034591402 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:32.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.162+0000 7fe0497b3640 1 --1- 172.21.15.89:0/1034591402 >> v1:172.21.15.89:6789/0 conn(0x7fe04406f7a0 0x7fe044069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36832/0 (socket says 172.21.15.89:36832) 2023-10-30T12:51:32.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.162+0000 7fe048fb2640 1 --2- 172.21.15.89:0/1034591402 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe0440c6bd0 0x7fe04406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:32.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.162+0000 7fe049fb4640 1 -- 172.21.15.89:0/1034591402 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe0440c6bd0 msgr2=0x7fe04406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:32.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.162+0000 7fe049fb4640 1 --2- 172.21.15.89:0/1034591402 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe0440c6bd0 0x7fe04406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:32.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.162+0000 7fe049fb4640 1 -- 172.21.15.89:0/1034591402 >> v1:172.21.15.89:6789/0 conn(0x7fe04406f7a0 legacy=0x7fe044069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:32.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.162+0000 7fe03bfff640 1 -- 172.21.15.89:0/1034591402 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633530859 0 0) 0x7fe044066cf0 con 0x7fe04406f7a0 2023-10-30T12:51:32.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.162+0000 7fe049fb4640 1 -- 172.21.15.89:0/1034591402 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0440c7390 con 0x7fe044071110 2023-10-30T12:51:32.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.166+0000 7fe049fb4640 1 --2- 172.21.15.89:0/1034591402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe044071110 0x7fe044158110 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fe03c009e50 tx=0x7fe03c01f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fba6c9b7179566c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:32.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.166+0000 7fe03bfff640 1 -- 172.21.15.89:0/1034591402 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03c0af400 con 0x7fe044071110 2023-10-30T12:51:32.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.166+0000 7fe03bfff640 1 -- 172.21.15.89:0/1034591402 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe03c0ab050 con 0x7fe044071110 2023-10-30T12:51:32.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.166+0000 7fe03bfff640 1 -- 172.21.15.89:0/1034591402 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03c0af780 con 0x7fe044071110 2023-10-30T12:51:32.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.166+0000 7fe04ba17640 1 -- 172.21.15.89:0/1034591402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe044071110 msgr2=0x7fe044158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:32.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.166+0000 7fe04ba17640 1 --2- 172.21.15.89:0/1034591402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe044071110 0x7fe044158110 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fe03c009e50 tx=0x7fe03c01f1f0 comp rx=0 tx=0).stop 2023-10-30T12:51:32.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.166+0000 7fe04ba17640 1 -- 172.21.15.89:0/1034591402 shutdown_connections 2023-10-30T12:51:32.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.166+0000 7fe04ba17640 1 --2- 172.21.15.89:0/1034591402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe044071110 0x7fe044158110 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:32.170 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.166+0000 7fe04ba17640 1 --2- 172.21.15.89:0/1034591402 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe0440c6bd0 0x7fe04406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:32.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.166+0000 7fe04ba17640 1 -- 172.21.15.89:0/1034591402 >> 172.21.15.89:0/1034591402 conn(0x7fe0440c7910 msgr2=0x7fe0440c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:32.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.166+0000 7fe04ba17640 1 -- 172.21.15.89:0/1034591402 shutdown_connections 2023-10-30T12:51:32.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.166+0000 7fe04ba17640 1 -- 172.21.15.89:0/1034591402 wait complete. 2023-10-30T12:51:32.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.166+0000 7fe04ba17640 1 Processor -- start 2023-10-30T12:51:32.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.166+0000 7fe04ba17640 1 -- start start 2023-10-30T12:51:32.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.170+0000 7fe04ba17640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe04406f7a0 0x7fe0440c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:32.172 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.170+0000 7fe04ba17640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe044071110 0x7fe04415a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:32.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.170+0000 7fe04ba17640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0440c6bd0 0x7fe04415a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:32.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.170+0000 7fe04ba17640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe044159910 con 0x7fe0440c6bd0 2023-10-30T12:51:32.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.170+0000 7fe04ba17640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe044159ed0 con 0x7fe044071110 2023-10-30T12:51:32.173 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.170+0000 7fe04ba17640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe044159bd0 con 0x7fe04406f7a0 2023-10-30T12:51:32.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.170+0000 7fe049fb4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0440c6bd0 0x7fe04415a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:32.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.170+0000 7fe049fb4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0440c6bd0 0x7fe04415a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47330/0 (socket says 172.21.15.89:47330) 2023-10-30T12:51:32.174 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.170+0000 7fe049fb4640 1 -- 172.21.15.89:0/2870675831 learned_addr learned my addr 172.21.15.89:0/2870675831 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:32.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.170+0000 7fe0497b3640 1 --2- 172.21.15.89:0/2870675831 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe04406f7a0 0x7fe0440c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:32.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.170+0000 7fe048fb2640 1 --2- 172.21.15.89:0/2870675831 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe044071110 0x7fe04415a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:32.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.170+0000 7fe049fb4640 1 -- 172.21.15.89:0/2870675831 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe04406f7a0 msgr2=0x7fe0440c8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:32.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.170+0000 7fe049fb4640 1 --2- 172.21.15.89:0/2870675831 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe04406f7a0 0x7fe0440c8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:32.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.170+0000 7fe049fb4640 1 -- 172.21.15.89:0/2870675831 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe044071110 msgr2=0x7fe04415a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:32.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.170+0000 7fe049fb4640 1 --2- 172.21.15.89:0/2870675831 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe044071110 0x7fe04415a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:32.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.170+0000 7fe049fb4640 1 -- 172.21.15.89:0/2870675831 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe03c009b30 con 0x7fe0440c6bd0 2023-10-30T12:51:32.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.170+0000 7fe049fb4640 1 --2- 172.21.15.89:0/2870675831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0440c6bd0 0x7fe04415a8a0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fe03c01ce20 tx=0x7fe03c01bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:32.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.170+0000 7fe039ffb640 1 -- 172.21.15.89:0/2870675831 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03c0afde0 con 0x7fe0440c6bd0 2023-10-30T12:51:32.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.170+0000 7fe039ffb640 1 -- 172.21.15.89:0/2870675831 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe03c009510 con 0x7fe0440c6bd0 2023-10-30T12:51:32.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.170+0000 7fe039ffb640 1 -- 172.21.15.89:0/2870675831 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03c0ba680 con 0x7fe0440c6bd0 2023-10-30T12:51:32.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.170+0000 7fe04ba17640 1 -- 172.21.15.89:0/2870675831 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0440c0400 con 0x7fe0440c6bd0 2023-10-30T12:51:32.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.170+0000 7fe04ba17640 1 -- 172.21.15.89:0/2870675831 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0440c09a0 con 0x7fe0440c6bd0 2023-10-30T12:51:32.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.170+0000 7fe04ba17640 1 -- 172.21.15.89:0/2870675831 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe04404e170 con 0x7fe0440c6bd0 2023-10-30T12:51:32.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.170+0000 7fe039ffb640 1 -- 172.21.15.89:0/2870675831 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe03c0b6040 con 0x7fe0440c6bd0 2023-10-30T12:51:32.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.174+0000 7fe039ffb640 1 --2- 172.21.15.89:0/2870675831 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe0280664d0 0x7fe028068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:32.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.174+0000 7fe039ffb640 1 -- 172.21.15.89:0/2870675831 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7fe03c1224a0 con 0x7fe0440c6bd0 2023-10-30T12:51:32.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.174+0000 7fe0497b3640 1 --2- 172.21.15.89:0/2870675831 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe0280664d0 0x7fe028068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:32.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.174+0000 7fe0497b3640 1 --2- 172.21.15.89:0/2870675831 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe0280664d0 0x7fe028068990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fe040000f70 tx=0x7fe040007c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:32.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.178+0000 7fe039ffb640 1 -- 172.21.15.89:0/2870675831 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe03c0ef470 con 0x7fe0440c6bd0 2023-10-30T12:51:32.326 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.326+0000 7fe04ba17640 1 -- 172.21.15.89:0/2870675831 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe04415a010 con 0x7fe0440c6bd0 2023-10-30T12:51:32.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.330+0000 7fe039ffb640 1 -- 172.21.15.89:0/2870675831 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+18124 (secure 0 0 0) 0x7fe03c0f2a90 con 0x7fe0440c6bd0 2023-10-30T12:51:32.328 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:51:32.328 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":66,"fsid":"401641be-db6e-4c16-b3ac-fd84a86c4de4","created":"2023-10-30T12:28:26.198532+0000","modified":"2023-10-30T12:50:41.318657+0000","last_up_change":"2023-10-30T12:28:37.212205+0000","last_in_change":"2023-10-30T12:28:28.517734+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:39.004943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T12:28:48.935909+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T12:28:49.716098+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":58,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72595c54-fd6e-4a5a-bb14-ff27e9377b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6801","nonce":1673955622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6803","nonce":1673955622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6807","nonce":1673955622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6805","nonce":1673955622}]},"public_addr":"172.21.15.89:6801/1673955622","cluster_addr":"172.21.15.89:6803/1673955622","heartbeat_back_addr":"172.21.15.89:6807/1673955622","heartbeat_front_addr":"172.21.15.89:6805/1673955622","state":["exists","up"]},{"osd":1,"uuid":"0305670b-d7c4-4193-bec2-8b7351ff590a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6817","nonce":3340572039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6819","nonce":3340572039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6823","nonce":3340572039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6821","nonce":3340572039}]},"public_addr":"172.21.15.89:6817/3340572039","cluster_addr":"172.21.15.89:6819/3340572039","heartbeat_back_addr":"172.21.15.89:6823/3340572039","heartbeat_front_addr":"172.21.15.89:6821/3340572039","state":["exists","up"]},{"osd":2,"uuid":"4bab5426-a3c3-46b0-91fc-546c254006c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6809","nonce":1929940799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6811","nonce":1929940799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6815","nonce":1929940799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6813","nonce":1929940799}]},"public_addr":"172.21.15.89:6809/1929940799","cluster_addr":"172.21.15.89:6811/1929940799","heartbeat_back_addr":"172.21.15.89:6815/1929940799","heartbeat_front_addr":"172.21.15.89:6813/1929940799","state":["exists","up"]},{"osd":3,"uuid":"7d47aea8-27cd-42be-8591-d63ad68c90e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6825","nonce":345087939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6827","nonce":345087939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6831","nonce":345087939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6829","nonce":345087939}]},"public_addr":"172.21.15.89:6825/345087939","cluster_addr":"172.21.15.89:6827/345087939","heartbeat_back_addr":"172.21.15.89:6831/345087939","heartbeat_front_addr":"172.21.15.89:6829/345087939","state":["exists","up"]},{"osd":4,"uuid":"2e2ae8f8-d097-4bba-9448-95047fbfacad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6801","nonce":2138125545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6803","nonce":2138125545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6809","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6811","nonce":2138125545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6805","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6807","nonce":2138125545}]},"public_addr":"172.21.15.192:6801/2138125545","cluster_addr":"172.21.15.192:6803/2138125545","heartbeat_back_addr":"172.21.15.192:6811/2138125545","heartbeat_front_addr":"172.21.15.192:6807/2138125545","state":["exists","up"]},{"osd":5,"uuid":"49be5ce5-12f4-4c26-8d39-864b6ff67313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6806","nonce":2360218671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6810","nonce":2360218671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6815","nonce":2360218671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6813","nonce":2360218671}]},"public_addr":"172.21.15.192:6806/2360218671","cluster_addr":"172.21.15.192:6810/2360218671","heartbeat_back_addr":"172.21.15.192:6815/2360218671","heartbeat_front_addr":"172.21.15.192:6813/2360218671","state":["exists","up"]},{"osd":6,"uuid":"48623452-d456-4ea2-a0de-5fa4b37e6d10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6825","nonce":3707671287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6827","nonce":3707671287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6831","nonce":3707671287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6829","nonce":3707671287}]},"public_addr":"172.21.15.192:6825/3707671287","cluster_addr":"172.21.15.192:6827/3707671287","heartbeat_back_addr":"172.21.15.192:6831/3707671287","heartbeat_front_addr":"172.21.15.192:6829/3707671287","state":["exists","up"]},{"osd":7,"uuid":"ff0a4384-54f8-43ec-a3ce-9c8ae3bf8a94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6817","nonce":3111945871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6819","nonce":3111945871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6823","nonce":3111945871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6821","nonce":3111945871}]},"public_addr":"172.21.15.192:6817/3111945871","cluster_addr":"172.21.15.192:6819/3111945871","heartbeat_back_addr":"172.21.15.192:6823/3111945871","heartbeat_front_addr":"172.21.15.192:6821/3111945871","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.183065+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.248650+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.194576+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.397483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.043648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.186624+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.159674+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.100851+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2},{"from":5,"to":1}]},{"pgid":"3.13","mappings":[{"from":7,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.192:6837/2130807115":"2023-10-31T12:50:41.318586+0000","172.21.15.192:6834/501291700":"2023-10-31T12:49:01.291877+0000","172.21.15.192:6837/604357386":"2023-10-31T12:48:11.263450+0000","172.21.15.192:6836/604357386":"2023-10-31T12:48:11.263450+0000","172.21.15.89:6836/951276252":"2023-10-31T12:45:51.226535+0000","172.21.15.192:6834/1289220491":"2023-10-31T12:31:20.953485+0000","172.21.15.192:6835/1289220491":"2023-10-31T12:31:20.953485+0000","172.21.15.192:6837/1413360446":"2023-10-31T12:43:36.177020+0000","172.21.15.192:6839/3339589587":"2023-10-31T12:34:56.026152+0000","172.21.15.192:6838/93914587":"2023-10-31T12:42:26.158883+0000","172.21.15.89:6837/2053475104":"2023-10-31T12:33:10.981311+0000","172.21.15.192:6836/2130807115":"2023-10-31T12:50:41.318586+0000","172.21.15.192:6836/1413360446":"2023-10-31T12:43:36.177020+0000","172.21.15.192:6838/3339589587":"2023-10-31T12:34:56.026152+0000","172.21.15.89:6837/951276252":"2023-10-31T12:45:51.226535+0000","172.21.15.192:6836/1002124471":"2023-10-31T12:37:36.081217+0000","172.21.15.192:6837/1002124471":"2023-10-31T12:37:36.081217+0000","172.21.15.192:6835/501291700":"2023-10-31T12:49:01.291877+0000","172.21.15.192:6834/654256299":"2023-10-31T12:40:11.121122+0000","172.21.15.89:6836/2053475104":"2023-10-31T12:33:10.981311+0000","172.21.15.192:6835/654256299":"2023-10-31T12:40:11.121122+0000","172.21.15.192:6839/93914587":"2023-10-31T12:42:26.158883+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:51:32.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.330+0000 7fe04ba17640 1 -- 172.21.15.89:0/2870675831 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe0280664d0 msgr2=0x7fe028068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:32.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.330+0000 7fe04ba17640 1 --2- 172.21.15.89:0/2870675831 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe0280664d0 0x7fe028068990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fe040000f70 tx=0x7fe040007c10 comp rx=0 tx=0).stop 2023-10-30T12:51:32.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.330+0000 7fe04ba17640 1 -- 172.21.15.89:0/2870675831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0440c6bd0 msgr2=0x7fe04415a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:32.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.330+0000 7fe04ba17640 1 --2- 172.21.15.89:0/2870675831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0440c6bd0 0x7fe04415a8a0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fe03c01ce20 tx=0x7fe03c01bf90 comp rx=0 tx=0).stop 2023-10-30T12:51:32.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.330+0000 7fe04ba17640 1 -- 172.21.15.89:0/2870675831 shutdown_connections 2023-10-30T12:51:32.331 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.330+0000 7fe04ba17640 1 --2- 172.21.15.89:0/2870675831 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe0280664d0 0x7fe028068990 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:32.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.330+0000 7fe04ba17640 1 --2- 172.21.15.89:0/2870675831 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe044071110 0x7fe04415a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:32.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.330+0000 7fe04ba17640 1 --2- 172.21.15.89:0/2870675831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe0440c6bd0 0x7fe04415a8a0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:32.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.330+0000 7fe04ba17640 1 --2- 172.21.15.89:0/2870675831 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe04406f7a0 0x7fe0440c8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:32.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.330+0000 7fe04ba17640 1 -- 172.21.15.89:0/2870675831 >> 172.21.15.89:0/2870675831 conn(0x7fe0440c7910 msgr2=0x7fe0440b0850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:32.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.330+0000 7fe04ba17640 1 -- 172.21.15.89:0/2870675831 shutdown_connections 2023-10-30T12:51:32.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:32.330+0000 7fe04ba17640 1 -- 172.21.15.89:0/2870675831 wait complete. 2023-10-30T12:51:33.352 INFO:tasks.ceph:Scrubbing osd.0 2023-10-30T12:51:33.352 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-30T12:51:33.503 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.501+0000 7f2230846640 1 Processor -- start 2023-10-30T12:51:33.504 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.501+0000 7f2230846640 1 -- start start 2023-10-30T12:51:33.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.505+0000 7f2230846640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f222c1517f0 0x7f222c157620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:33.505 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.505+0000 7f2230846640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f222c0c6490 0x7f222c0c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:33.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.505+0000 7f2230846640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f222c06c810 con 0x7f222c158b10 2023-10-30T12:51:33.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.505+0000 7f2230846640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f222c15bbf0 con 0x7f222c1517f0 2023-10-30T12:51:33.506 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.505+0000 7f2230846640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f222c071ed0 con 0x7f222c0c6490 2023-10-30T12:51:33.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.505+0000 7f222a59c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f222c1517f0 0x7f222c157620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:33.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.505+0000 7f222a59c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f222c1517f0 0x7f222c157620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47334/0 (socket says 172.21.15.89:47334) 2023-10-30T12:51:33.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.505+0000 7f222a59c640 1 -- 172.21.15.89:0/911688256 learned_addr learned my addr 172.21.15.89:0/911688256 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:33.507 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.505+0000 7f2223fff640 1 --1- 172.21.15.89:0/911688256 >> v1:172.21.15.89:6789/0 conn(0x7f222c158b10 0x7f222c15ac10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36840/0 (socket says 172.21.15.89:36840) 2023-10-30T12:51:33.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.505+0000 7f222ad9d640 1 --2- 172.21.15.89:0/911688256 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f222c0c6490 0x7f222c0c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:33.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.505+0000 7f222a59c640 1 -- 172.21.15.89:0/911688256 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f222c0c6490 msgr2=0x7f222c0c6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:33.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.505+0000 7f222a59c640 1 --2- 172.21.15.89:0/911688256 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f222c0c6490 0x7f222c0c6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:33.508 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.505+0000 7f222a59c640 1 -- 172.21.15.89:0/911688256 >> v1:172.21.15.89:6789/0 conn(0x7f222c158b10 legacy=0x7f222c15ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:33.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.505+0000 7f222a59c640 1 -- 172.21.15.89:0/911688256 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222c15b630 con 0x7f222c1517f0 2023-10-30T12:51:33.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.505+0000 7f2229d9b640 1 -- 172.21.15.89:0/911688256 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4004383939 0 0) 0x7f222c06c810 con 0x7f222c158b10 2023-10-30T12:51:33.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.509+0000 7f222a59c640 1 --2- 172.21.15.89:0/911688256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f222c1517f0 0x7f222c157620 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f221c009870 tx=0x7f221c01c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=733bdd89f1ad6656 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:33.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.509+0000 7f2229d9b640 1 -- 172.21.15.89:0/911688256 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f221c0ab620 con 0x7f222c1517f0 2023-10-30T12:51:33.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.509+0000 7f2229d9b640 1 -- 172.21.15.89:0/911688256 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f221c0ab780 con 0x7f222c1517f0 2023-10-30T12:51:33.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.509+0000 7f2229d9b640 1 -- 172.21.15.89:0/911688256 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f221c0abb10 con 0x7f222c1517f0 2023-10-30T12:51:33.511 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.509+0000 7f2230846640 1 -- 172.21.15.89:0/911688256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f222c1517f0 msgr2=0x7f222c157620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:33.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.509+0000 7f2230846640 1 --2- 172.21.15.89:0/911688256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f222c1517f0 0x7f222c157620 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f221c009870 tx=0x7f221c01c510 comp rx=0 tx=0).stop 2023-10-30T12:51:33.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.509+0000 7f2230846640 1 -- 172.21.15.89:0/911688256 shutdown_connections 2023-10-30T12:51:33.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.509+0000 7f2230846640 1 --2- 172.21.15.89:0/911688256 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f222c0c6490 0x7f222c0c6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:33.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.509+0000 7f2230846640 1 --2- 172.21.15.89:0/911688256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f222c1517f0 0x7f222c157620 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:33.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.509+0000 7f2230846640 1 -- 172.21.15.89:0/911688256 >> 172.21.15.89:0/911688256 conn(0x7f222c0c7910 msgr2=0x7f222c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:33.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.509+0000 7f2230846640 1 -- 172.21.15.89:0/911688256 shutdown_connections 2023-10-30T12:51:33.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.509+0000 7f2230846640 1 -- 172.21.15.89:0/911688256 wait complete. 2023-10-30T12:51:33.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.509+0000 7f2230846640 1 Processor -- start 2023-10-30T12:51:33.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.509+0000 7f2230846640 1 -- start start 2023-10-30T12:51:33.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.513+0000 7f2230846640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f222c0c6490 0x7f222c0c1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:33.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.513+0000 7f2230846640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f222c1517f0 0x7f222c0c0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:33.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.513+0000 7f2230846640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f222c158b10 0x7f222c14b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:33.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.513+0000 7f2230846640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222c14d110 con 0x7f222c1517f0 2023-10-30T12:51:33.517 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.513+0000 7f2230846640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222c14ce70 con 0x7f222c158b10 2023-10-30T12:51:33.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.513+0000 7f2230846640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f222c14b1f0 con 0x7f222c0c6490 2023-10-30T12:51:33.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.513+0000 7f2223fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f222c1517f0 0x7f222c0c0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:33.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.513+0000 7f2223fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f222c1517f0 0x7f222c0c0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47348/0 (socket says 172.21.15.89:47348) 2023-10-30T12:51:33.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.513+0000 7f2223fff640 1 -- 172.21.15.89:0/4043015941 learned_addr learned my addr 172.21.15.89:0/4043015941 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:33.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.513+0000 7f222a59c640 1 --2- 172.21.15.89:0/4043015941 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f222c0c6490 0x7f222c0c1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:33.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.513+0000 7f222ad9d640 1 --2- 172.21.15.89:0/4043015941 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f222c158b10 0x7f222c14b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:33.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.513+0000 7f2223fff640 1 -- 172.21.15.89:0/4043015941 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f222c0c6490 msgr2=0x7f222c0c1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:33.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.513+0000 7f2223fff640 1 --2- 172.21.15.89:0/4043015941 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f222c0c6490 0x7f222c0c1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:33.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.513+0000 7f2223fff640 1 -- 172.21.15.89:0/4043015941 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f222c158b10 msgr2=0x7f222c14b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:33.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.513+0000 7f2223fff640 1 --2- 172.21.15.89:0/4043015941 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f222c158b10 0x7f222c14b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:33.520 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.513+0000 7f2223fff640 1 -- 172.21.15.89:0/4043015941 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f221c0094d0 con 0x7f222c1517f0 2023-10-30T12:51:33.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.513+0000 7f2223fff640 1 --2- 172.21.15.89:0/4043015941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f222c1517f0 0x7f222c0c0b30 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f221400c480 tx=0x7f221400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:33.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.513+0000 7f2222ffd640 1 -- 172.21.15.89:0/4043015941 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22140074e0 con 0x7f222c1517f0 2023-10-30T12:51:33.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.513+0000 7f2222ffd640 1 -- 172.21.15.89:0/4043015941 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2214007660 con 0x7f222c1517f0 2023-10-30T12:51:33.521 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.513+0000 7f2222ffd640 1 -- 172.21.15.89:0/4043015941 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2214004a30 con 0x7f222c1517f0 2023-10-30T12:51:33.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.513+0000 7f2230846640 1 -- 172.21.15.89:0/4043015941 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f222c14b4b0 con 0x7f222c1517f0 2023-10-30T12:51:33.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.513+0000 7f2230846640 1 -- 172.21.15.89:0/4043015941 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f222c0c0240 con 0x7f222c1517f0 2023-10-30T12:51:33.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.513+0000 7f2230846640 1 -- 172.21.15.89:0/4043015941 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f222c0c5f70 con 0x7f222c1517f0 2023-10-30T12:51:33.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.517+0000 7f2222ffd640 1 -- 172.21.15.89:0/4043015941 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2214004b90 con 0x7f222c1517f0 2023-10-30T12:51:33.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.517+0000 7f2222ffd640 1 --2- 172.21.15.89:0/4043015941 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f220c0664a0 0x7f220c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:33.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.517+0000 7f2222ffd640 1 -- 172.21.15.89:0/4043015941 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f22140833a0 con 0x7f222c1517f0 2023-10-30T12:51:33.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.517+0000 7f2222ffd640 1 --2- 172.21.15.89:0/4043015941 >> [v2:172.21.15.89:6800/1673955622,v1:172.21.15.89:6801/1673955622] conn(0x7f220c06efa0 0x7f220c071400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:33.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.517+0000 7f2222ffd640 1 -- 172.21.15.89:0/4043015941 --> [v2:172.21.15.89:6800/1673955622,v1:172.21.15.89:6801/1673955622] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f220c071ad0 con 0x7f220c06efa0 2023-10-30T12:51:33.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.517+0000 7f2222ffd640 1 -- 172.21.15.89:0/4043015941 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2214085980 con 0x7f222c1517f0 2023-10-30T12:51:33.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.517+0000 7f222a59c640 1 --2- 172.21.15.89:0/4043015941 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f220c0664a0 0x7f220c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:33.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.517+0000 7f222ad9d640 1 --2- 172.21.15.89:0/4043015941 >> [v2:172.21.15.89:6800/1673955622,v1:172.21.15.89:6801/1673955622] conn(0x7f220c06efa0 0x7f220c071400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:33.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.517+0000 7f222ad9d640 1 --2- 172.21.15.89:0/4043015941 >> [v2:172.21.15.89:6800/1673955622,v1:172.21.15.89:6801/1673955622] conn(0x7f220c06efa0 0x7f220c071400 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:33.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.517+0000 7f222a59c640 1 --2- 172.21.15.89:0/4043015941 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f220c0664a0 0x7f220c068960 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f221c0af000 tx=0x7f221c0b6040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:33.526 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.517+0000 7f2222ffd640 1 -- 172.21.15.89:0/4043015941 <== osd.0 v2:172.21.15.89:6800/1673955622 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f220c071ad0 con 0x7f220c06efa0 2023-10-30T12:51:33.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.529+0000 7f2230846640 1 -- 172.21.15.89:0/4043015941 --> [v2:172.21.15.89:6800/1673955622,v1:172.21.15.89:6801/1673955622] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f222c0c0760 con 0x7f220c06efa0 2023-10-30T12:51:33.531 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.533+0000 7f2222ffd640 1 -- 172.21.15.89:0/4043015941 <== osd.0 v2:172.21.15.89:6800/1673955622 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f222c0c0760 con 0x7f220c06efa0 2023-10-30T12:51:33.532 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-10-30T12:51:33.532 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T12:51:33.533 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-10-30T12:51:33.533 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.533+0000 7f2230846640 1 -- 172.21.15.89:0/4043015941 >> [v2:172.21.15.89:6800/1673955622,v1:172.21.15.89:6801/1673955622] conn(0x7f220c06efa0 msgr2=0x7f220c071400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:33.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.533+0000 7f2230846640 1 --2- 172.21.15.89:0/4043015941 >> [v2:172.21.15.89:6800/1673955622,v1:172.21.15.89:6801/1673955622] conn(0x7f220c06efa0 0x7f220c071400 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:33.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.533+0000 7f2230846640 1 -- 172.21.15.89:0/4043015941 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f220c0664a0 msgr2=0x7f220c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:33.534 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.533+0000 7f2230846640 1 --2- 172.21.15.89:0/4043015941 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f220c0664a0 0x7f220c068960 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f221c0af000 tx=0x7f221c0b6040 comp rx=0 tx=0).stop 2023-10-30T12:51:33.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.533+0000 7f2230846640 1 -- 172.21.15.89:0/4043015941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f222c1517f0 msgr2=0x7f222c0c0b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:33.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.533+0000 7f2230846640 1 --2- 172.21.15.89:0/4043015941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f222c1517f0 0x7f222c0c0b30 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f221400c480 tx=0x7f221400c940 comp rx=0 tx=0).stop 2023-10-30T12:51:33.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.533+0000 7f222ad9d640 1 -- 172.21.15.89:0/4043015941 reap_dead start 2023-10-30T12:51:33.535 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.533+0000 7f2230846640 1 -- 172.21.15.89:0/4043015941 shutdown_connections 2023-10-30T12:51:33.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.533+0000 7f2230846640 1 -- 172.21.15.89:0/4043015941 >> 172.21.15.89:0/4043015941 conn(0x7f222c0c7910 msgr2=0x7f222c0c8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:33.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.533+0000 7f2230846640 1 -- 172.21.15.89:0/4043015941 shutdown_connections 2023-10-30T12:51:33.536 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.533+0000 7f2230846640 1 -- 172.21.15.89:0/4043015941 wait complete. 2023-10-30T12:51:33.551 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-30T12:51:33.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.689+0000 7fef91b9d640 1 Processor -- start 2023-10-30T12:51:33.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.689+0000 7fef91b9d640 1 -- start start 2023-10-30T12:51:33.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.689+0000 7fef91b9d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef8c13fd10 0x7fef8c1565a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:33.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.689+0000 7fef91b9d640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef8c072030 con 0x7fef8c0c6f60 2023-10-30T12:51:33.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.689+0000 7fef91b9d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef8c070010 con 0x7fef8c0c6b80 2023-10-30T12:51:33.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.689+0000 7fef91b9d640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef8c072260 con 0x7fef8c13fd10 2023-10-30T12:51:33.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.693+0000 7fef8affd640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fef8c0c6f60 0x7fef8c13f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36842/0 (socket says 172.21.15.89:36842) 2023-10-30T12:51:33.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.693+0000 7fef8affd640 1 -- 172.21.15.89:0/366943159 learned_addr learned my addr 172.21.15.89:0/366943159 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:51:33.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.693+0000 7fef8bfff640 1 --2- 172.21.15.89:0/366943159 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef8c13fd10 0x7fef8c1565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:33.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.693+0000 7fef8a7fc640 1 -- 172.21.15.89:0/366943159 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 72694908 0 0) 0x7fef8c072030 con 0x7fef8c0c6f60 2023-10-30T12:51:33.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.693+0000 7fef8a7fc640 1 -- 172.21.15.89:0/366943159 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef74005740 con 0x7fef8c0c6f60 2023-10-30T12:51:33.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.693+0000 7fef8a7fc640 1 -- 172.21.15.89:0/366943159 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3954778105 0 0) 0x7fef8c070010 con 0x7fef8c0c6b80 2023-10-30T12:51:33.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.693+0000 7fef8a7fc640 1 -- 172.21.15.89:0/366943159 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef8c072030 con 0x7fef8c0c6b80 2023-10-30T12:51:33.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.693+0000 7fef8a7fc640 1 -- 172.21.15.89:0/366943159 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2093700908 0 0) 0x7fef74005740 con 0x7fef8c0c6f60 2023-10-30T12:51:33.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.693+0000 7fef8a7fc640 1 -- 172.21.15.89:0/366943159 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef8c13fd10 msgr2=0x7fef8c1565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:33.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.693+0000 7fef8a7fc640 1 --2- 172.21.15.89:0/366943159 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef8c13fd10 0x7fef8c1565a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:33.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.693+0000 7fef8a7fc640 1 -- 172.21.15.89:0/366943159 >> v1:172.21.15.192:6789/0 conn(0x7fef8c0c6b80 legacy=0x7fef8c071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:33.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.693+0000 7fef8a7fc640 1 -- 172.21.15.89:0/366943159 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef8c06f6e0 con 0x7fef8c0c6f60 2023-10-30T12:51:33.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.693+0000 7fef8a7fc640 1 -- 172.21.15.89:0/366943159 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7fef80001680 con 0x7fef8c0c6f60 2023-10-30T12:51:33.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.693+0000 7fef8a7fc640 1 -- 172.21.15.89:0/366943159 >> v1:172.21.15.89:6789/0 conn(0x7fef8c0c6f60 legacy=0x7fef8c13f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:33.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.693+0000 7fef8a7fc640 1 --2- 172.21.15.89:0/366943159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef74005d50 0x7fef74006150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:33.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.693+0000 7fef8a7fc640 1 --2- 172.21.15.89:0/366943159 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef740069e0 0x7fef74006e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:33.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.693+0000 7fef8a7fc640 1 --2- 172.21.15.89:0/366943159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef74007360 0x7fef7400dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:33.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.693+0000 7fef8a7fc640 1 -- 172.21.15.89:0/366943159 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8c072260 con 0x7fef74007360 2023-10-30T12:51:33.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.693+0000 7fef8a7fc640 1 -- 172.21.15.89:0/366943159 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7400e0e0 con 0x7fef74005d50 2023-10-30T12:51:33.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.693+0000 7fef8a7fc640 1 -- 172.21.15.89:0/366943159 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef7400e220 con 0x7fef740069e0 2023-10-30T12:51:33.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.693+0000 7fef8bfff640 1 --2- 172.21.15.89:0/366943159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef74007360 0x7fef7400dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:33.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.693+0000 7fef8affd640 1 --2- 172.21.15.89:0/366943159 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef740069e0 0x7fef74006e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:33.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.693+0000 7fef8b7fe640 1 --2- 172.21.15.89:0/366943159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef74005d50 0x7fef74006150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:33.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.693+0000 7fef8bfff640 1 -- 172.21.15.89:0/366943159 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef740069e0 msgr2=0x7fef74006e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:33.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.693+0000 7fef8bfff640 1 --2- 172.21.15.89:0/366943159 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef740069e0 0x7fef74006e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:33.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.693+0000 7fef8bfff640 1 -- 172.21.15.89:0/366943159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef74005d50 msgr2=0x7fef74006150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:33.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.693+0000 7fef8bfff640 1 --2- 172.21.15.89:0/366943159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef74005d50 0x7fef74006150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:33.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.693+0000 7fef8bfff640 1 -- 172.21.15.89:0/366943159 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef74005740 con 0x7fef74007360 2023-10-30T12:51:33.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.697+0000 7fef8bfff640 1 --2- 172.21.15.89:0/366943159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef74007360 0x7fef7400dba0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fef8c0c8ce0 tx=0x7fef8401cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c46f2a7daae3dac2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:33.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.697+0000 7fef8a7fc640 1 -- 172.21.15.89:0/366943159 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef840a5c10 con 0x7fef74007360 2023-10-30T12:51:33.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.697+0000 7fef8a7fc640 1 -- 172.21.15.89:0/366943159 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef8400f040 con 0x7fef74007360 2023-10-30T12:51:33.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.697+0000 7fef8a7fc640 1 -- 172.21.15.89:0/366943159 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef840ad440 con 0x7fef74007360 2023-10-30T12:51:33.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.697+0000 7fef91b9d640 1 -- 172.21.15.89:0/366943159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef74007360 msgr2=0x7fef7400dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:33.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.697+0000 7fef91b9d640 1 --2- 172.21.15.89:0/366943159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef74007360 0x7fef7400dba0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fef8c0c8ce0 tx=0x7fef8401cc00 comp rx=0 tx=0).stop 2023-10-30T12:51:33.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.697+0000 7fef8bfff640 1 -- 172.21.15.89:0/366943159 reap_dead start 2023-10-30T12:51:33.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.697+0000 7fef91b9d640 1 -- 172.21.15.89:0/366943159 shutdown_connections 2023-10-30T12:51:33.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.697+0000 7fef91b9d640 1 -- 172.21.15.89:0/366943159 >> 172.21.15.89:0/366943159 conn(0x7fef8c0c7910 msgr2=0x7fef8c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:33.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.697+0000 7fef91b9d640 1 -- 172.21.15.89:0/366943159 shutdown_connections 2023-10-30T12:51:33.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.697+0000 7fef91b9d640 1 -- 172.21.15.89:0/366943159 wait complete. 2023-10-30T12:51:33.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.697+0000 7fef91b9d640 1 Processor -- start 2023-10-30T12:51:33.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.697+0000 7fef91b9d640 1 -- start start 2023-10-30T12:51:33.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.701+0000 7fef91b9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef74007360 0x7fef8c15a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:33.709 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.701+0000 7fef91b9d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef8c159400 0x7fef8c1f1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:33.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.701+0000 7fef91b9d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef8c157c30 0x7fef8c1e44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:33.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.701+0000 7fef91b9d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8c13a980 con 0x7fef74007360 2023-10-30T12:51:33.710 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.701+0000 7fef91b9d640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8c157700 con 0x7fef8c159400 2023-10-30T12:51:33.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.701+0000 7fef91b9d640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef8c13a420 con 0x7fef8c157c30 2023-10-30T12:51:33.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.701+0000 7fef8b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef74007360 0x7fef8c15a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:33.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.701+0000 7fef8b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef74007360 0x7fef8c15a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47366/0 (socket says 172.21.15.89:47366) 2023-10-30T12:51:33.711 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.701+0000 7fef8b7fe640 1 -- 172.21.15.89:0/2238085743 learned_addr learned my addr 172.21.15.89:0/2238085743 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:33.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.701+0000 7fef8bfff640 1 --2- 172.21.15.89:0/2238085743 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef8c157c30 0x7fef8c1e44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:33.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.701+0000 7fef8affd640 1 --2- 172.21.15.89:0/2238085743 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef8c159400 0x7fef8c1f1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:33.712 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.701+0000 7fef8b7fe640 1 -- 172.21.15.89:0/2238085743 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef8c157c30 msgr2=0x7fef8c1e44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:33.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.701+0000 7fef8b7fe640 1 --2- 172.21.15.89:0/2238085743 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef8c157c30 0x7fef8c1e44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:33.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.701+0000 7fef8b7fe640 1 -- 172.21.15.89:0/2238085743 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef8c159400 msgr2=0x7fef8c1f1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:33.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.701+0000 7fef8b7fe640 1 --2- 172.21.15.89:0/2238085743 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef8c159400 0x7fef8c1f1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:33.713 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.701+0000 7fef8b7fe640 1 -- 172.21.15.89:0/2238085743 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef8c0bff50 con 0x7fef74007360 2023-10-30T12:51:33.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.701+0000 7fef8b7fe640 1 --2- 172.21.15.89:0/2238085743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef74007360 0x7fef8c15a550 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fef7c00e7e0 tx=0x7fef7c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:33.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.701+0000 7fef73fff640 1 -- 172.21.15.89:0/2238085743 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7c00ee90 con 0x7fef74007360 2023-10-30T12:51:33.714 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.701+0000 7fef73fff640 1 -- 172.21.15.89:0/2238085743 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef7c0052e0 con 0x7fef74007360 2023-10-30T12:51:33.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.701+0000 7fef91b9d640 1 -- 172.21.15.89:0/2238085743 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef8c13a6c0 con 0x7fef74007360 2023-10-30T12:51:33.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.701+0000 7fef73fff640 1 -- 172.21.15.89:0/2238085743 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7c00c8a0 con 0x7fef74007360 2023-10-30T12:51:33.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.701+0000 7fef91b9d640 1 -- 172.21.15.89:0/2238085743 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef8c159d50 con 0x7fef74007360 2023-10-30T12:51:33.715 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.701+0000 7fef91b9d640 1 -- 172.21.15.89:0/2238085743 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef8c159bb0 con 0x7fef74007360 2023-10-30T12:51:33.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.701+0000 7fef73fff640 1 -- 172.21.15.89:0/2238085743 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fef7c00ca00 con 0x7fef74007360 2023-10-30T12:51:33.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.705+0000 7fef73fff640 1 --2- 172.21.15.89:0/2238085743 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fef780664a0 0x7fef78068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:33.716 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.705+0000 7fef73fff640 1 -- 172.21.15.89:0/2238085743 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7fef7c084120 con 0x7fef74007360 2023-10-30T12:51:33.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.705+0000 7fef8affd640 1 --2- 172.21.15.89:0/2238085743 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fef780664a0 0x7fef78068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:33.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.705+0000 7fef8affd640 1 --2- 172.21.15.89:0/2238085743 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fef780664a0 0x7fef78068960 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fef8c157400 tx=0x7fef80009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:33.717 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.709+0000 7fef73fff640 1 -- 172.21.15.89:0/2238085743 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fef8c159bb0 con 0x7fef74007360 2023-10-30T12:51:33.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.857+0000 7fef91b9d640 1 -- 172.21.15.89:0/2238085743 --> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fef8c06fca0 con 0x7fef780664a0 2023-10-30T12:51:33.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.857+0000 7fef73fff640 1 -- 172.21.15.89:0/2238085743 <== mgr.4105 v2:172.21.15.192:6832/4076276638 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fef80013a40 con 0x7fef780664a0 2023-10-30T12:51:33.858 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 0 to deep-scrub 2023-10-30T12:51:33.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.857+0000 7fef91b9d640 1 -- 172.21.15.89:0/2238085743 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fef780664a0 msgr2=0x7fef78068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:33.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.857+0000 7fef91b9d640 1 --2- 172.21.15.89:0/2238085743 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fef780664a0 0x7fef78068960 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fef8c157400 tx=0x7fef80009040 comp rx=0 tx=0).stop 2023-10-30T12:51:33.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.861+0000 7fef91b9d640 1 -- 172.21.15.89:0/2238085743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef74007360 msgr2=0x7fef8c15a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:33.860 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.861+0000 7fef91b9d640 1 --2- 172.21.15.89:0/2238085743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef74007360 0x7fef8c15a550 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fef7c00e7e0 tx=0x7fef7c00eca0 comp rx=0 tx=0).stop 2023-10-30T12:51:33.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.861+0000 7fef91b9d640 1 -- 172.21.15.89:0/2238085743 shutdown_connections 2023-10-30T12:51:33.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.861+0000 7fef91b9d640 1 --2- 172.21.15.89:0/2238085743 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fef780664a0 0x7fef78068960 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:33.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.861+0000 7fef91b9d640 1 --2- 172.21.15.89:0/2238085743 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef8c159400 0x7fef8c1f1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:33.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.861+0000 7fef91b9d640 1 --2- 172.21.15.89:0/2238085743 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef8c157c30 0x7fef8c1e44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:33.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.861+0000 7fef91b9d640 1 --2- 172.21.15.89:0/2238085743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef74007360 0x7fef8c15a550 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:33.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.861+0000 7fef91b9d640 1 -- 172.21.15.89:0/2238085743 >> 172.21.15.89:0/2238085743 conn(0x7fef8c0c7910 msgr2=0x7fef8c0c96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:33.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.861+0000 7fef91b9d640 1 -- 172.21.15.89:0/2238085743 shutdown_connections 2023-10-30T12:51:33.863 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:33.861+0000 7fef91b9d640 1 -- 172.21.15.89:0/2238085743 wait complete. 2023-10-30T12:51:33.879 INFO:tasks.ceph:Scrubbing osd.1 2023-10-30T12:51:33.881 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-30T12:51:34.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.021+0000 7fc409b3e640 1 Processor -- start 2023-10-30T12:51:34.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.021+0000 7fc409b3e640 1 -- start start 2023-10-30T12:51:34.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.021+0000 7fc409b3e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc40414cf40 0x7fc404069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.021+0000 7fc409b3e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc40415a700 0x7fc404156490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.021+0000 7fc409b3e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc40415a270 con 0x7fc4041569d0 2023-10-30T12:51:34.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.021+0000 7fc409b3e640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc40406c7c0 con 0x7fc40414cf40 2023-10-30T12:51:34.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.021+0000 7fc409b3e640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc40406a3d0 con 0x7fc40415a700 2023-10-30T12:51:34.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.021+0000 7fc4037fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc40414cf40 0x7fc404069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.021+0000 7fc4037fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fc40414cf40 0x7fc404069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47370/0 (socket says 172.21.15.89:47370) 2023-10-30T12:51:34.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.021+0000 7fc4037fe640 1 -- 172.21.15.89:0/3832755553 learned_addr learned my addr 172.21.15.89:0/3832755553 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:34.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.025+0000 7fc402ffd640 1 --2- 172.21.15.89:0/3832755553 >> v2:172.21.15.192:3300/0 conn(0x7fc40415a700 0x7fc404156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.025+0000 7fc4037fe640 1 -- 172.21.15.89:0/3832755553 >> v2:172.21.15.192:3300/0 conn(0x7fc40415a700 msgr2=0x7fc404156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:34.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.025+0000 7fc4037fe640 1 --2- 172.21.15.89:0/3832755553 >> v2:172.21.15.192:3300/0 conn(0x7fc40415a700 0x7fc404156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.025+0000 7fc4037fe640 1 -- 172.21.15.89:0/3832755553 >> v1:172.21.15.192:6789/0 conn(0x7fc4041569d0 legacy=0x7fc4040c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.025+0000 7fc4037fe640 1 -- 172.21.15.89:0/3832755553 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc404156ed0 con 0x7fc40414cf40 2023-10-30T12:51:34.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.025+0000 7fc4037fe640 1 --2- 172.21.15.89:0/3832755553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc40414cf40 0x7fc404069e90 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fc4040cbd70 tx=0x7fc3f401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cd417d47d8258cd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:34.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.025+0000 7fc4027fc640 1 -- 172.21.15.89:0/3832755553 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f40ab5f0 con 0x7fc40414cf40 2023-10-30T12:51:34.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.025+0000 7fc4027fc640 1 -- 172.21.15.89:0/3832755553 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f40ab750 con 0x7fc40414cf40 2023-10-30T12:51:34.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.025+0000 7fc4027fc640 1 -- 172.21.15.89:0/3832755553 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f40abae0 con 0x7fc40414cf40 2023-10-30T12:51:34.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.029+0000 7fc409b3e640 1 -- 172.21.15.89:0/3832755553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc40414cf40 msgr2=0x7fc404069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.029+0000 7fc409b3e640 1 --2- 172.21.15.89:0/3832755553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc40414cf40 0x7fc404069e90 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fc4040cbd70 tx=0x7fc3f401c500 comp rx=0 tx=0).stop 2023-10-30T12:51:34.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.029+0000 7fc409b3e640 1 -- 172.21.15.89:0/3832755553 shutdown_connections 2023-10-30T12:51:34.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.029+0000 7fc409b3e640 1 --2- 172.21.15.89:0/3832755553 >> v2:172.21.15.192:3300/0 conn(0x7fc40415a700 0x7fc404156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.029+0000 7fc409b3e640 1 --2- 172.21.15.89:0/3832755553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc40414cf40 0x7fc404069e90 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.029+0000 7fc409b3e640 1 -- 172.21.15.89:0/3832755553 >> 172.21.15.89:0/3832755553 conn(0x7fc4040c7910 msgr2=0x7fc4040c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:34.031 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.029+0000 7fc409b3e640 1 -- 172.21.15.89:0/3832755553 shutdown_connections 2023-10-30T12:51:34.032 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.029+0000 7fc409b3e640 1 -- 172.21.15.89:0/3832755553 wait complete. 2023-10-30T12:51:34.034 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.029+0000 7fc409b3e640 1 Processor -- start 2023-10-30T12:51:34.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.029+0000 7fc409b3e640 1 -- start start 2023-10-30T12:51:34.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.029+0000 7fc409b3e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc40414cf40 0x7fc404147b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.029+0000 7fc409b3e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4041569d0 0x7fc4040c9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.035 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.029+0000 7fc409b3e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc40415a700 0x7fc4040ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.029+0000 7fc409b3e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4040c98c0 con 0x7fc4041569d0 2023-10-30T12:51:34.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.029+0000 7fc409b3e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40415f4e0 con 0x7fc40414cf40 2023-10-30T12:51:34.036 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.029+0000 7fc409b3e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc40415f7a0 con 0x7fc40415a700 2023-10-30T12:51:34.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.029+0000 7fc402ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4041569d0 0x7fc4040c9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.029+0000 7fc402ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4041569d0 0x7fc4040c9d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47380/0 (socket says 172.21.15.89:47380) 2023-10-30T12:51:34.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.029+0000 7fc402ffd640 1 -- 172.21.15.89:0/446052129 learned_addr learned my addr 172.21.15.89:0/446052129 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:34.037 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.029+0000 7fc4037fe640 1 --2- 172.21.15.89:0/446052129 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc40414cf40 0x7fc404147b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.029+0000 7fc403fff640 1 --2- 172.21.15.89:0/446052129 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc40415a700 0x7fc4040ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.029+0000 7fc402ffd640 1 -- 172.21.15.89:0/446052129 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc40415a700 msgr2=0x7fc4040ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.029+0000 7fc402ffd640 1 --2- 172.21.15.89:0/446052129 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc40415a700 0x7fc4040ca250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.029+0000 7fc402ffd640 1 -- 172.21.15.89:0/446052129 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc40414cf40 msgr2=0x7fc404147b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.029+0000 7fc402ffd640 1 --2- 172.21.15.89:0/446052129 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc40414cf40 0x7fc404147b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.029+0000 7fc402ffd640 1 -- 172.21.15.89:0/446052129 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f40094d0 con 0x7fc4041569d0 2023-10-30T12:51:34.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.033+0000 7fc402ffd640 1 --2- 172.21.15.89:0/446052129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4041569d0 0x7fc4040c9d10 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fc3f800d5f0 tx=0x7fc3f800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:34.039 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.033+0000 7fc3ebfff640 1 -- 172.21.15.89:0/446052129 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f80125f0 con 0x7fc4041569d0 2023-10-30T12:51:34.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.033+0000 7fc3ebfff640 1 -- 172.21.15.89:0/446052129 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f8012750 con 0x7fc4041569d0 2023-10-30T12:51:34.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.033+0000 7fc3ebfff640 1 -- 172.21.15.89:0/446052129 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f8012b00 con 0x7fc4041569d0 2023-10-30T12:51:34.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.033+0000 7fc409b3e640 1 -- 172.21.15.89:0/446052129 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc40415fa80 con 0x7fc4041569d0 2023-10-30T12:51:34.040 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.033+0000 7fc409b3e640 1 -- 172.21.15.89:0/446052129 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc404160040 con 0x7fc4041569d0 2023-10-30T12:51:34.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.033+0000 7fc409b3e640 1 -- 172.21.15.89:0/446052129 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc40414d320 con 0x7fc4041569d0 2023-10-30T12:51:34.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.033+0000 7fc3ebfff640 1 -- 172.21.15.89:0/446052129 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc3f8012c60 con 0x7fc4041569d0 2023-10-30T12:51:34.041 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.033+0000 7fc3ebfff640 1 --2- 172.21.15.89:0/446052129 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc3f00664a0 0x7fc3f0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.033+0000 7fc3ebfff640 1 -- 172.21.15.89:0/446052129 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7fc3f80848a0 con 0x7fc4041569d0 2023-10-30T12:51:34.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.033+0000 7fc3ebfff640 1 --2- 172.21.15.89:0/446052129 >> [v2:172.21.15.89:6816/3340572039,v1:172.21.15.89:6817/3340572039] conn(0x7fc3f006efa0 0x7fc3f0071400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.033+0000 7fc3ebfff640 1 -- 172.21.15.89:0/446052129 --> [v2:172.21.15.89:6816/3340572039,v1:172.21.15.89:6817/3340572039] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc3f0071ad0 con 0x7fc3f006efa0 2023-10-30T12:51:34.042 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.033+0000 7fc3ebfff640 1 -- 172.21.15.89:0/446052129 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc3f80892a0 con 0x7fc4041569d0 2023-10-30T12:51:34.043 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.033+0000 7fc4037fe640 1 --2- 172.21.15.89:0/446052129 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc3f00664a0 0x7fc3f0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.033+0000 7fc403fff640 1 --2- 172.21.15.89:0/446052129 >> [v2:172.21.15.89:6816/3340572039,v1:172.21.15.89:6817/3340572039] conn(0x7fc3f006efa0 0x7fc3f0071400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.037+0000 7fc403fff640 1 --2- 172.21.15.89:0/446052129 >> [v2:172.21.15.89:6816/3340572039,v1:172.21.15.89:6817/3340572039] conn(0x7fc3f006efa0 0x7fc3f0071400 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:34.044 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.037+0000 7fc4037fe640 1 --2- 172.21.15.89:0/446052129 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc3f00664a0 0x7fc3f0068960 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc3f40af000 tx=0x7fc3f40a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:34.045 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.037+0000 7fc3ebfff640 1 -- 172.21.15.89:0/446052129 <== osd.1 v2:172.21.15.89:6816/3340572039 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fc3f0071ad0 con 0x7fc3f006efa0 2023-10-30T12:51:34.048 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.049+0000 7fc409b3e640 1 -- 172.21.15.89:0/446052129 --> [v2:172.21.15.89:6816/3340572039,v1:172.21.15.89:6817/3340572039] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fc404159f00 con 0x7fc3f006efa0 2023-10-30T12:51:34.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.049+0000 7fc3ebfff640 1 -- 172.21.15.89:0/446052129 <== osd.1 v2:172.21.15.89:6816/3340572039 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fc3fc001e80 con 0x7fc3f006efa0 2023-10-30T12:51:34.049 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-10-30T12:51:34.050 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T12:51:34.050 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-10-30T12:51:34.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.049+0000 7fc409b3e640 1 -- 172.21.15.89:0/446052129 >> [v2:172.21.15.89:6816/3340572039,v1:172.21.15.89:6817/3340572039] conn(0x7fc3f006efa0 msgr2=0x7fc3f0071400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.049+0000 7fc409b3e640 1 --2- 172.21.15.89:0/446052129 >> [v2:172.21.15.89:6816/3340572039,v1:172.21.15.89:6817/3340572039] conn(0x7fc3f006efa0 0x7fc3f0071400 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.049+0000 7fc409b3e640 1 -- 172.21.15.89:0/446052129 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc3f00664a0 msgr2=0x7fc3f0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.049+0000 7fc409b3e640 1 --2- 172.21.15.89:0/446052129 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc3f00664a0 0x7fc3f0068960 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc3f40af000 tx=0x7fc3f40a7000 comp rx=0 tx=0).stop 2023-10-30T12:51:34.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.049+0000 7fc409b3e640 1 -- 172.21.15.89:0/446052129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4041569d0 msgr2=0x7fc4040c9d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.049+0000 7fc409b3e640 1 --2- 172.21.15.89:0/446052129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4041569d0 0x7fc4040c9d10 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fc3f800d5f0 tx=0x7fc3f800dab0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.049+0000 7fc403fff640 1 -- 172.21.15.89:0/446052129 reap_dead start 2023-10-30T12:51:34.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.049+0000 7fc409b3e640 1 -- 172.21.15.89:0/446052129 shutdown_connections 2023-10-30T12:51:34.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.049+0000 7fc409b3e640 1 -- 172.21.15.89:0/446052129 >> 172.21.15.89:0/446052129 conn(0x7fc4040c7910 msgr2=0x7fc40406f690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:34.053 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.049+0000 7fc409b3e640 1 -- 172.21.15.89:0/446052129 shutdown_connections 2023-10-30T12:51:34.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.049+0000 7fc409b3e640 1 -- 172.21.15.89:0/446052129 wait complete. 2023-10-30T12:51:34.068 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-30T12:51:34.205 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.205+0000 7fd27153e640 1 Processor -- start 2023-10-30T12:51:34.206 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.205+0000 7fd27153e640 1 -- start start 2023-10-30T12:51:34.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.205+0000 7fd27153e640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd26c151910 0x7fd26c158120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.205+0000 7fd27153e640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd26c06a4e0 con 0x7fd26c06ade0 2023-10-30T12:51:34.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.205+0000 7fd27153e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd26c158660 con 0x7fd26c06a970 2023-10-30T12:51:34.208 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.205+0000 7fd27153e640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd26c0c6630 con 0x7fd26c151910 2023-10-30T12:51:34.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.205+0000 7fd26b7fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd26c06ade0 0x7fd26c0c5f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36844/0 (socket says 172.21.15.89:36844) 2023-10-30T12:51:34.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.205+0000 7fd26b7fe640 1 -- 172.21.15.89:0/2012789501 learned_addr learned my addr 172.21.15.89:0/2012789501 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:51:34.209 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.205+0000 7fd26affd640 1 --2- 172.21.15.89:0/2012789501 >> v2:172.21.15.192:3300/0 conn(0x7fd26c151910 0x7fd26c158120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.209+0000 7fd26a7fc640 1 -- 172.21.15.89:0/2012789501 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 768730799 0 0) 0x7fd26c06a4e0 con 0x7fd26c06ade0 2023-10-30T12:51:34.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.209+0000 7fd26a7fc640 1 -- 172.21.15.89:0/2012789501 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd258003530 con 0x7fd26c06ade0 2023-10-30T12:51:34.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.209+0000 7fd26a7fc640 1 -- 172.21.15.89:0/2012789501 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4155228161 0 0) 0x7fd258003530 con 0x7fd26c06ade0 2023-10-30T12:51:34.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.209+0000 7fd26affd640 1 -- 172.21.15.89:0/2012789501 >> v1:172.21.15.192:6789/0 conn(0x7fd26c06a970 legacy=0x7fd26c0c5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.209+0000 7fd26affd640 1 -- 172.21.15.89:0/2012789501 >> v1:172.21.15.89:6789/0 conn(0x7fd26c06ade0 legacy=0x7fd26c0c5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.209+0000 7fd26affd640 1 -- 172.21.15.89:0/2012789501 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd26c0c6c50 con 0x7fd26c151910 2023-10-30T12:51:34.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.209+0000 7fd26affd640 1 --2- 172.21.15.89:0/2012789501 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd26c151910 0x7fd26c158120 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd25c0097c0 tx=0x7fd25c01eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4cbf583e19aeeec server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:34.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.209+0000 7fd26a7fc640 1 -- 172.21.15.89:0/2012789501 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd264001a00 con 0x7fd26c151910 2023-10-30T12:51:34.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.209+0000 7fd26a7fc640 1 -- 172.21.15.89:0/2012789501 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd25c0a8610 con 0x7fd26c151910 2023-10-30T12:51:34.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.209+0000 7fd26a7fc640 1 -- 172.21.15.89:0/2012789501 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25c0a8980 con 0x7fd26c151910 2023-10-30T12:51:34.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.213+0000 7fd27153e640 1 -- 172.21.15.89:0/2012789501 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd26c151910 msgr2=0x7fd26c158120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.213+0000 7fd27153e640 1 --2- 172.21.15.89:0/2012789501 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd26c151910 0x7fd26c158120 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd25c0097c0 tx=0x7fd25c01eb90 comp rx=0 tx=0).stop 2023-10-30T12:51:34.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.213+0000 7fd27153e640 1 -- 172.21.15.89:0/2012789501 shutdown_connections 2023-10-30T12:51:34.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.213+0000 7fd27153e640 1 --2- 172.21.15.89:0/2012789501 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd26c151910 0x7fd26c158120 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.213+0000 7fd27153e640 1 -- 172.21.15.89:0/2012789501 >> 172.21.15.89:0/2012789501 conn(0x7fd26c0c7910 msgr2=0x7fd26c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:34.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.213+0000 7fd27153e640 1 -- 172.21.15.89:0/2012789501 shutdown_connections 2023-10-30T12:51:34.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.213+0000 7fd27153e640 1 -- 172.21.15.89:0/2012789501 wait complete. 2023-10-30T12:51:34.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.213+0000 7fd27153e640 1 Processor -- start 2023-10-30T12:51:34.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.213+0000 7fd27153e640 1 -- start start 2023-10-30T12:51:34.218 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.213+0000 7fd27153e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd26c06a970 0x7fd26c0aecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.213+0000 7fd27153e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd26c151910 0x7fd26c157f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.213+0000 7fd27153e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd26c1550e0 0x7fd26c0c0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.213+0000 7fd27153e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26c0aff90 con 0x7fd26c151910 2023-10-30T12:51:34.219 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.213+0000 7fd27153e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26c158480 con 0x7fd26c1550e0 2023-10-30T12:51:34.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.213+0000 7fd27153e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd26c0c0250 con 0x7fd26c06a970 2023-10-30T12:51:34.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.213+0000 7fd263fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd26c151910 0x7fd26c157f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.213+0000 7fd263fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd26c151910 0x7fd26c157f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47390/0 (socket says 172.21.15.89:47390) 2023-10-30T12:51:34.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.213+0000 7fd263fff640 1 -- 172.21.15.89:0/1938478750 learned_addr learned my addr 172.21.15.89:0/1938478750 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:34.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.213+0000 7fd26affd640 1 --2- 172.21.15.89:0/1938478750 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd26c06a970 0x7fd26c0aecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.213+0000 7fd26b7fe640 1 --2- 172.21.15.89:0/1938478750 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd26c1550e0 0x7fd26c0c0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.213+0000 7fd263fff640 1 -- 172.21.15.89:0/1938478750 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd26c06a970 msgr2=0x7fd26c0aecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.213+0000 7fd263fff640 1 --2- 172.21.15.89:0/1938478750 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd26c06a970 0x7fd26c0aecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.213+0000 7fd263fff640 1 -- 172.21.15.89:0/1938478750 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd26c1550e0 msgr2=0x7fd26c0c0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.222 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.213+0000 7fd263fff640 1 --2- 172.21.15.89:0/1938478750 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd26c1550e0 0x7fd26c0c0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.213+0000 7fd263fff640 1 -- 172.21.15.89:0/1938478750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd25c009420 con 0x7fd26c151910 2023-10-30T12:51:34.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.213+0000 7fd26b7fe640 1 --2- 172.21.15.89:0/1938478750 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd26c1550e0 0x7fd26c0c0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:34.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.217+0000 7fd263fff640 1 --2- 172.21.15.89:0/1938478750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd26c151910 0x7fd26c157f40 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fd25400c530 tx=0x7fd25400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:34.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.217+0000 7fd2637fe640 1 -- 172.21.15.89:0/1938478750 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd254005220 con 0x7fd26c151910 2023-10-30T12:51:34.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.217+0000 7fd2637fe640 1 -- 172.21.15.89:0/1938478750 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd254005380 con 0x7fd26c151910 2023-10-30T12:51:34.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.217+0000 7fd2637fe640 1 -- 172.21.15.89:0/1938478750 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd254007270 con 0x7fd26c151910 2023-10-30T12:51:34.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.217+0000 7fd27153e640 1 -- 172.21.15.89:0/1938478750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd26c0af330 con 0x7fd26c151910 2023-10-30T12:51:34.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.217+0000 7fd27153e640 1 -- 172.21.15.89:0/1938478750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd26c0adf20 con 0x7fd26c151910 2023-10-30T12:51:34.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.217+0000 7fd27153e640 1 -- 172.21.15.89:0/1938478750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd26c04e170 con 0x7fd26c151910 2023-10-30T12:51:34.225 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.217+0000 7fd2637fe640 1 -- 172.21.15.89:0/1938478750 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd2540073d0 con 0x7fd26c151910 2023-10-30T12:51:34.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.217+0000 7fd2637fe640 1 --2- 172.21.15.89:0/1938478750 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd24c0664d0 0x7fd24c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.217+0000 7fd2637fe640 1 -- 172.21.15.89:0/1938478750 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7fd254083a70 con 0x7fd26c151910 2023-10-30T12:51:34.226 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.217+0000 7fd26affd640 1 --2- 172.21.15.89:0/1938478750 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd24c0664d0 0x7fd24c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.221+0000 7fd26affd640 1 --2- 172.21.15.89:0/1938478750 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd24c0664d0 0x7fd24c068990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fd25c0b2000 tx=0x7fd25c0aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:34.227 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.221+0000 7fd2637fe640 1 -- 172.21.15.89:0/1938478750 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd254050ad0 con 0x7fd26c151910 2023-10-30T12:51:34.373 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.373+0000 7fd27153e640 1 -- 172.21.15.89:0/1938478750 --> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd26c0c7210 con 0x7fd24c0664d0 2023-10-30T12:51:34.374 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.373+0000 7fd2637fe640 1 -- 172.21.15.89:0/1938478750 <== mgr.4105 v2:172.21.15.192:6832/4076276638 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd25c004380 con 0x7fd24c0664d0 2023-10-30T12:51:34.375 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 1 to deep-scrub 2023-10-30T12:51:34.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.377+0000 7fd27153e640 1 -- 172.21.15.89:0/1938478750 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd24c0664d0 msgr2=0x7fd24c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.377+0000 7fd27153e640 1 --2- 172.21.15.89:0/1938478750 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd24c0664d0 0x7fd24c068990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fd25c0b2000 tx=0x7fd25c0aa070 comp rx=0 tx=0).stop 2023-10-30T12:51:34.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.377+0000 7fd27153e640 1 -- 172.21.15.89:0/1938478750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd26c151910 msgr2=0x7fd26c157f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.377+0000 7fd27153e640 1 --2- 172.21.15.89:0/1938478750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd26c151910 0x7fd26c157f40 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fd25400c530 tx=0x7fd25400c9f0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.377+0000 7fd27153e640 1 -- 172.21.15.89:0/1938478750 shutdown_connections 2023-10-30T12:51:34.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.377+0000 7fd27153e640 1 --2- 172.21.15.89:0/1938478750 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fd24c0664d0 0x7fd24c068990 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.377+0000 7fd27153e640 1 --2- 172.21.15.89:0/1938478750 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd26c1550e0 0x7fd26c0c0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.377+0000 7fd27153e640 1 --2- 172.21.15.89:0/1938478750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd26c151910 0x7fd26c157f40 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.377+0000 7fd27153e640 1 --2- 172.21.15.89:0/1938478750 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd26c06a970 0x7fd26c0aecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.377+0000 7fd27153e640 1 -- 172.21.15.89:0/1938478750 >> 172.21.15.89:0/1938478750 conn(0x7fd26c0c7910 msgr2=0x7fd26c0c91f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:34.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.377+0000 7fd27153e640 1 -- 172.21.15.89:0/1938478750 shutdown_connections 2023-10-30T12:51:34.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.377+0000 7fd27153e640 1 -- 172.21.15.89:0/1938478750 wait complete. 2023-10-30T12:51:34.396 INFO:tasks.ceph:Scrubbing osd.2 2023-10-30T12:51:34.397 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-30T12:51:34.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.541+0000 7f03d3d80640 1 Processor -- start 2023-10-30T12:51:34.540 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.541+0000 7f03d3d80640 1 -- start start 2023-10-30T12:51:34.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.541+0000 7f03d3d80640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03cc06b8f0 0x7f03cc159840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.541+0000 7f03d3d80640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f03cc0c7b10 0x7f03cc164100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.541+0000 7f03d3d80640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03cc0c17e0 con 0x7f03cc0cbb00 2023-10-30T12:51:34.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.541+0000 7f03d3d80640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03cc06bcd0 con 0x7f03cc0c7b10 2023-10-30T12:51:34.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.541+0000 7f03d3d80640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc0c6c60 con 0x7f03cc06b8f0 2023-10-30T12:51:34.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.541+0000 7f03d231d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f03cc0c7b10 0x7f03cc164100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.541+0000 7f03d231d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f03cc0c7b10 0x7f03cc164100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47394/0 (socket says 172.21.15.89:47394) 2023-10-30T12:51:34.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.541+0000 7f03d131b640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f03cc0cbb00 0x7f03cc06c600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36858/0 (socket says 172.21.15.89:36858) 2023-10-30T12:51:34.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.541+0000 7f03d231d640 1 -- 172.21.15.89:0/492368603 learned_addr learned my addr 172.21.15.89:0/492368603 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:34.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.541+0000 7f03d1b1c640 1 --2- 172.21.15.89:0/492368603 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03cc06b8f0 0x7f03cc159840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.541+0000 7f03d231d640 1 -- 172.21.15.89:0/492368603 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03cc06b8f0 msgr2=0x7f03cc159840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:34.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.541+0000 7f03d231d640 1 --2- 172.21.15.89:0/492368603 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03cc06b8f0 0x7f03cc159840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.541+0000 7f03d231d640 1 -- 172.21.15.89:0/492368603 >> v1:172.21.15.89:6789/0 conn(0x7f03cc0cbb00 legacy=0x7f03cc06c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.541+0000 7f03d231d640 1 -- 172.21.15.89:0/492368603 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03cc0c0f40 con 0x7f03cc0c7b10 2023-10-30T12:51:34.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.541+0000 7f03d0b1a640 1 -- 172.21.15.89:0/492368603 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3293221959 0 0) 0x7f03bc001120 con 0x7f03cc0cbb00 2023-10-30T12:51:34.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.545+0000 7f03d231d640 1 --2- 172.21.15.89:0/492368603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03cc0c7b10 0x7f03cc164100 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f03c4009e50 tx=0x7f03c401c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e256867033d1e6ad server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:34.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.545+0000 7f03d0b1a640 1 -- 172.21.15.89:0/492368603 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c40a58a0 con 0x7f03cc0c7b10 2023-10-30T12:51:34.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.545+0000 7f03d0b1a640 1 -- 172.21.15.89:0/492368603 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c40a5a00 con 0x7f03cc0c7b10 2023-10-30T12:51:34.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.545+0000 7f03d0b1a640 1 -- 172.21.15.89:0/492368603 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c40a5d90 con 0x7f03cc0c7b10 2023-10-30T12:51:34.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.545+0000 7f03d3d80640 1 -- 172.21.15.89:0/492368603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03cc0c7b10 msgr2=0x7f03cc164100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.548 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.545+0000 7f03d3d80640 1 --2- 172.21.15.89:0/492368603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03cc0c7b10 0x7f03cc164100 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f03c4009e50 tx=0x7f03c401c9e0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.545+0000 7f03d3d80640 1 -- 172.21.15.89:0/492368603 shutdown_connections 2023-10-30T12:51:34.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.545+0000 7f03d3d80640 1 --2- 172.21.15.89:0/492368603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03cc0c7b10 0x7f03cc164100 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.549 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.545+0000 7f03d3d80640 1 --2- 172.21.15.89:0/492368603 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03cc06b8f0 0x7f03cc159840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.545+0000 7f03d3d80640 1 -- 172.21.15.89:0/492368603 >> 172.21.15.89:0/492368603 conn(0x7f03cc0c8a20 msgr2=0x7f03cc0c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:34.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.545+0000 7f03d3d80640 1 -- 172.21.15.89:0/492368603 shutdown_connections 2023-10-30T12:51:34.550 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.545+0000 7f03d3d80640 1 -- 172.21.15.89:0/492368603 wait complete. 2023-10-30T12:51:34.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.545+0000 7f03d3d80640 1 Processor -- start 2023-10-30T12:51:34.553 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.549+0000 7f03d3d80640 1 -- start start 2023-10-30T12:51:34.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.549+0000 7f03d3d80640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03cc06b8f0 0x7f03cc151cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.549+0000 7f03d3d80640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03cc0c7b10 0x7f03cc152200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.554 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.549+0000 7f03d3d80640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03cc0cbb00 0x7f03cc0ae680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.549+0000 7f03d3d80640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc14bc80 con 0x7f03cc0c7b10 2023-10-30T12:51:34.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.549+0000 7f03d3d80640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc0affd0 con 0x7f03cc0cbb00 2023-10-30T12:51:34.555 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.549+0000 7f03d3d80640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc1644a0 con 0x7f03cc06b8f0 2023-10-30T12:51:34.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.549+0000 7f03d131b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03cc0c7b10 0x7f03cc152200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.549+0000 7f03d131b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03cc0c7b10 0x7f03cc152200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47400/0 (socket says 172.21.15.89:47400) 2023-10-30T12:51:34.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.549+0000 7f03d131b640 1 -- 172.21.15.89:0/232111862 learned_addr learned my addr 172.21.15.89:0/232111862 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:34.556 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.549+0000 7f03d1b1c640 1 --2- 172.21.15.89:0/232111862 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03cc06b8f0 0x7f03cc151cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.549+0000 7f03d231d640 1 --2- 172.21.15.89:0/232111862 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03cc0cbb00 0x7f03cc0ae680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.549+0000 7f03d131b640 1 -- 172.21.15.89:0/232111862 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03cc06b8f0 msgr2=0x7f03cc151cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.549+0000 7f03d131b640 1 --2- 172.21.15.89:0/232111862 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f03cc06b8f0 0x7f03cc151cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.557 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.549+0000 7f03d131b640 1 -- 172.21.15.89:0/232111862 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03cc0cbb00 msgr2=0x7f03cc0ae680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.549+0000 7f03d131b640 1 --2- 172.21.15.89:0/232111862 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03cc0cbb00 0x7f03cc0ae680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.549+0000 7f03d131b640 1 -- 172.21.15.89:0/232111862 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c4009b30 con 0x7f03cc0c7b10 2023-10-30T12:51:34.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.549+0000 7f03d231d640 1 --2- 172.21.15.89:0/232111862 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03cc0cbb00 0x7f03cc0ae680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:34.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.549+0000 7f03d131b640 1 --2- 172.21.15.89:0/232111862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03cc0c7b10 0x7f03cc152200 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f03bc00c4b0 tx=0x7f03bc00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:34.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.549+0000 7f03c27fc640 1 -- 172.21.15.89:0/232111862 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03bc0074e0 con 0x7f03cc0c7b10 2023-10-30T12:51:34.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.549+0000 7f03c27fc640 1 -- 172.21.15.89:0/232111862 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03bc007660 con 0x7f03cc0c7b10 2023-10-30T12:51:34.559 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.549+0000 7f03c27fc640 1 -- 172.21.15.89:0/232111862 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03bc004a30 con 0x7f03cc0c7b10 2023-10-30T12:51:34.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.549+0000 7f03d3d80640 1 -- 172.21.15.89:0/232111862 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03cc0b0200 con 0x7f03cc0c7b10 2023-10-30T12:51:34.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.549+0000 7f03d3d80640 1 -- 172.21.15.89:0/232111862 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03cc14b580 con 0x7f03cc0c7b10 2023-10-30T12:51:34.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.549+0000 7f03d3d80640 1 -- 172.21.15.89:0/232111862 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f03cc0c6920 con 0x7f03cc0c7b10 2023-10-30T12:51:34.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.553+0000 7f03c27fc640 1 -- 172.21.15.89:0/232111862 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f03bc004b90 con 0x7f03cc0c7b10 2023-10-30T12:51:34.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.553+0000 7f03c27fc640 1 --2- 172.21.15.89:0/232111862 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f03b00664a0 0x7f03b0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.553+0000 7f03c27fc640 1 -- 172.21.15.89:0/232111862 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f03bc0831d0 con 0x7f03cc0c7b10 2023-10-30T12:51:34.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.553+0000 7f03c27fc640 1 --2- 172.21.15.89:0/232111862 >> [v2:172.21.15.89:6808/1929940799,v1:172.21.15.89:6809/1929940799] conn(0x7f03b006efa0 0x7f03b0071400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.553+0000 7f03c27fc640 1 -- 172.21.15.89:0/232111862 --> [v2:172.21.15.89:6808/1929940799,v1:172.21.15.89:6809/1929940799] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f03b0071ad0 con 0x7f03b006efa0 2023-10-30T12:51:34.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.553+0000 7f03c27fc640 1 -- 172.21.15.89:0/232111862 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03bc0857b0 con 0x7f03cc0c7b10 2023-10-30T12:51:34.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.553+0000 7f03d1b1c640 1 --2- 172.21.15.89:0/232111862 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f03b00664a0 0x7f03b0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.553+0000 7f03d231d640 1 --2- 172.21.15.89:0/232111862 >> [v2:172.21.15.89:6808/1929940799,v1:172.21.15.89:6809/1929940799] conn(0x7f03b006efa0 0x7f03b0071400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.562 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.553+0000 7f03d231d640 1 --2- 172.21.15.89:0/232111862 >> [v2:172.21.15.89:6808/1929940799,v1:172.21.15.89:6809/1929940799] conn(0x7f03b006efa0 0x7f03b0071400 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:34.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.553+0000 7f03d1b1c640 1 --2- 172.21.15.89:0/232111862 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f03b00664a0 0x7f03b0068960 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f03c8007a60 tx=0x7f03c8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:34.563 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.553+0000 7f03c27fc640 1 -- 172.21.15.89:0/232111862 <== osd.2 v2:172.21.15.89:6808/1929940799 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f03b0071ad0 con 0x7f03b006efa0 2023-10-30T12:51:34.566 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.565+0000 7f03d3d80640 1 -- 172.21.15.89:0/232111862 --> [v2:172.21.15.89:6808/1929940799,v1:172.21.15.89:6809/1929940799] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f03cc0c81c0 con 0x7f03b006efa0 2023-10-30T12:51:34.567 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.565+0000 7f03c27fc640 1 -- 172.21.15.89:0/232111862 <== osd.2 v2:172.21.15.89:6808/1929940799 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f03c40091d0 con 0x7f03b006efa0 2023-10-30T12:51:34.567 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-10-30T12:51:34.568 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T12:51:34.568 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-10-30T12:51:34.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.565+0000 7f03d3d80640 1 -- 172.21.15.89:0/232111862 >> [v2:172.21.15.89:6808/1929940799,v1:172.21.15.89:6809/1929940799] conn(0x7f03b006efa0 msgr2=0x7f03b0071400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.569 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.565+0000 7f03d3d80640 1 --2- 172.21.15.89:0/232111862 >> [v2:172.21.15.89:6808/1929940799,v1:172.21.15.89:6809/1929940799] conn(0x7f03b006efa0 0x7f03b0071400 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.565+0000 7f03d3d80640 1 -- 172.21.15.89:0/232111862 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f03b00664a0 msgr2=0x7f03b0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.565+0000 7f03d3d80640 1 --2- 172.21.15.89:0/232111862 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f03b00664a0 0x7f03b0068960 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f03c8007a60 tx=0x7f03c8008040 comp rx=0 tx=0).stop 2023-10-30T12:51:34.570 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.565+0000 7f03d3d80640 1 -- 172.21.15.89:0/232111862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03cc0c7b10 msgr2=0x7f03cc152200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.565+0000 7f03d3d80640 1 --2- 172.21.15.89:0/232111862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03cc0c7b10 0x7f03cc152200 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f03bc00c4b0 tx=0x7f03bc00c970 comp rx=0 tx=0).stop 2023-10-30T12:51:34.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.565+0000 7f03d231d640 1 -- 172.21.15.89:0/232111862 reap_dead start 2023-10-30T12:51:34.571 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.565+0000 7f03d3d80640 1 -- 172.21.15.89:0/232111862 shutdown_connections 2023-10-30T12:51:34.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.565+0000 7f03d3d80640 1 -- 172.21.15.89:0/232111862 >> 172.21.15.89:0/232111862 conn(0x7f03cc0c8a20 msgr2=0x7f03cc071bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:34.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.569+0000 7f03d3d80640 1 -- 172.21.15.89:0/232111862 shutdown_connections 2023-10-30T12:51:34.572 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.569+0000 7f03d3d80640 1 -- 172.21.15.89:0/232111862 wait complete. 2023-10-30T12:51:34.585 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-30T12:51:34.724 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.725+0000 7f2cf1972640 1 Processor -- start 2023-10-30T12:51:34.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.725+0000 7f2cf1972640 1 -- start start 2023-10-30T12:51:34.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.725+0000 7f2cf1972640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2cec153c60 0x7f2cec0ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.725+0000 7f2cf1972640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2cec159830 0x7f2cec159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.726 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.725+0000 7f2cf1972640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cec0cb780 con 0x7f2cec0c4430 2023-10-30T12:51:34.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.725+0000 7f2cf1972640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cec154040 con 0x7f2cec153c60 2023-10-30T12:51:34.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.725+0000 7f2cf1972640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cec15a2e0 con 0x7f2cec159830 2023-10-30T12:51:34.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.725+0000 7f2ceaffd640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2cec153c60 0x7f2cec0ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.725+0000 7f2ceaffd640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f2cec153c60 0x7f2cec0ca360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47402/0 (socket says 172.21.15.89:47402) 2023-10-30T12:51:34.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.725+0000 7f2ceaffd640 1 -- 172.21.15.89:0/3077470324 learned_addr learned my addr 172.21.15.89:0/3077470324 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:34.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.725+0000 7f2ceb7fe640 1 --2- 172.21.15.89:0/3077470324 >> v2:172.21.15.192:3300/0 conn(0x7f2cec159830 0x7f2cec159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.725+0000 7f2ceaffd640 1 -- 172.21.15.89:0/3077470324 >> v2:172.21.15.192:3300/0 conn(0x7f2cec159830 msgr2=0x7f2cec159cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:34.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.725+0000 7f2ceaffd640 1 --2- 172.21.15.89:0/3077470324 >> v2:172.21.15.192:3300/0 conn(0x7f2cec159830 0x7f2cec159cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.725+0000 7f2ceaffd640 1 -- 172.21.15.89:0/3077470324 >> v1:172.21.15.192:6789/0 conn(0x7f2cec0c4430 legacy=0x7f2cec0cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.725+0000 7f2ceaffd640 1 -- 172.21.15.89:0/3077470324 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cec15a5c0 con 0x7f2cec153c60 2023-10-30T12:51:34.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.729+0000 7f2ceaffd640 1 --2- 172.21.15.89:0/3077470324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cec153c60 0x7f2cec0ca360 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f2cec073da0 tx=0x7f2cdc01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13961c5479e3c3ab server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:34.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.729+0000 7f2cea7fc640 1 -- 172.21.15.89:0/3077470324 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc0ab5f0 con 0x7f2cec153c60 2023-10-30T12:51:34.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.729+0000 7f2cea7fc640 1 -- 172.21.15.89:0/3077470324 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cdc0ab750 con 0x7f2cec153c60 2023-10-30T12:51:34.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.729+0000 7f2cea7fc640 1 -- 172.21.15.89:0/3077470324 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc0abae0 con 0x7f2cec153c60 2023-10-30T12:51:34.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.729+0000 7f2cf1972640 1 -- 172.21.15.89:0/3077470324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cec153c60 msgr2=0x7f2cec0ca360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.729+0000 7f2cf1972640 1 --2- 172.21.15.89:0/3077470324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cec153c60 0x7f2cec0ca360 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f2cec073da0 tx=0x7f2cdc01c500 comp rx=0 tx=0).stop 2023-10-30T12:51:34.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.729+0000 7f2cf1972640 1 -- 172.21.15.89:0/3077470324 shutdown_connections 2023-10-30T12:51:34.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.729+0000 7f2cf1972640 1 --2- 172.21.15.89:0/3077470324 >> v2:172.21.15.192:3300/0 conn(0x7f2cec159830 0x7f2cec159cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.729+0000 7f2cf1972640 1 --2- 172.21.15.89:0/3077470324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cec153c60 0x7f2cec0ca360 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.729+0000 7f2cf1972640 1 -- 172.21.15.89:0/3077470324 >> 172.21.15.89:0/3077470324 conn(0x7f2cec06bfe0 msgr2=0x7f2cec06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:34.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.729+0000 7f2cf1972640 1 -- 172.21.15.89:0/3077470324 shutdown_connections 2023-10-30T12:51:34.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.729+0000 7f2cf1972640 1 -- 172.21.15.89:0/3077470324 wait complete. 2023-10-30T12:51:34.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.733+0000 7f2cf1972640 1 Processor -- start 2023-10-30T12:51:34.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.733+0000 7f2cf1972640 1 -- start start 2023-10-30T12:51:34.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.733+0000 7f2cf1972640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cec0c4430 0x7f2cec0c7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.733+0000 7f2cf1972640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cec153c60 0x7f2cec1f5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.733+0000 7f2cf1972640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cec159830 0x7f2cec145e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.733+0000 7f2cf1972640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cec142530 con 0x7f2cec0c4430 2023-10-30T12:51:34.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.733+0000 7f2cf1972640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cec144940 con 0x7f2cec153c60 2023-10-30T12:51:34.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.733+0000 7f2cf1972640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cec1439e0 con 0x7f2cec159830 2023-10-30T12:51:34.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.733+0000 7f2ceaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cec0c4430 0x7f2cec0c7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.733+0000 7f2ceaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cec0c4430 0x7f2cec0c7860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47412/0 (socket says 172.21.15.89:47412) 2023-10-30T12:51:34.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.733+0000 7f2ceaffd640 1 -- 172.21.15.89:0/955323848 learned_addr learned my addr 172.21.15.89:0/955323848 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:34.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.733+0000 7f2ce3fff640 1 --2- 172.21.15.89:0/955323848 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cec153c60 0x7f2cec1f5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.733+0000 7f2ceb7fe640 1 --2- 172.21.15.89:0/955323848 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cec159830 0x7f2cec145e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.733+0000 7f2ceaffd640 1 -- 172.21.15.89:0/955323848 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cec159830 msgr2=0x7f2cec145e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.733+0000 7f2ceaffd640 1 --2- 172.21.15.89:0/955323848 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cec159830 0x7f2cec145e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.733+0000 7f2ceaffd640 1 -- 172.21.15.89:0/955323848 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cec153c60 msgr2=0x7f2cec1f5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.733+0000 7f2ceaffd640 1 --2- 172.21.15.89:0/955323848 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cec153c60 0x7f2cec1f5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.733+0000 7f2ceaffd640 1 -- 172.21.15.89:0/955323848 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cdc0094d0 con 0x7f2cec0c4430 2023-10-30T12:51:34.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.733+0000 7f2ceaffd640 1 --2- 172.21.15.89:0/955323848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cec0c4430 0x7f2cec0c7860 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc0ad040 tx=0x7f2cdc01ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:34.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.733+0000 7f2ce8ff9640 1 -- 172.21.15.89:0/955323848 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc0be5f0 con 0x7f2cec0c4430 2023-10-30T12:51:34.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.733+0000 7f2ce8ff9640 1 -- 172.21.15.89:0/955323848 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cdc0be750 con 0x7f2cec0c4430 2023-10-30T12:51:34.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.733+0000 7f2ce8ff9640 1 -- 172.21.15.89:0/955323848 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc0beae0 con 0x7f2cec0c4430 2023-10-30T12:51:34.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.733+0000 7f2cf1972640 1 -- 172.21.15.89:0/955323848 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cec1435f0 con 0x7f2cec0c4430 2023-10-30T12:51:34.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.733+0000 7f2cf1972640 1 -- 172.21.15.89:0/955323848 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cec14fd90 con 0x7f2cec0c4430 2023-10-30T12:51:34.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.733+0000 7f2cf1972640 1 -- 172.21.15.89:0/955323848 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cec04e170 con 0x7f2cec0c4430 2023-10-30T12:51:34.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.737+0000 7f2ce8ff9640 1 -- 172.21.15.89:0/955323848 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2cdc0b43f0 con 0x7f2cec0c4430 2023-10-30T12:51:34.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.737+0000 7f2ce8ff9640 1 --2- 172.21.15.89:0/955323848 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2ccc0664d0 0x7f2ccc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.737+0000 7f2ce8ff9640 1 -- 172.21.15.89:0/955323848 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f2cdc0a8050 con 0x7f2cec0c4430 2023-10-30T12:51:34.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.737+0000 7f2ce3fff640 1 --2- 172.21.15.89:0/955323848 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2ccc0664d0 0x7f2ccc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.737+0000 7f2ce3fff640 1 --2- 172.21.15.89:0/955323848 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2ccc0664d0 0x7f2ccc068990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f2cec0c96e0 tx=0x7f2cd4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:34.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.741+0000 7f2ce8ff9640 1 -- 172.21.15.89:0/955323848 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2cdc0eb770 con 0x7f2cec0c4430 2023-10-30T12:51:34.887 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.885+0000 7f2cf1972640 1 -- 172.21.15.89:0/955323848 --> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f2cec0cbf90 con 0x7f2ccc0664d0 2023-10-30T12:51:34.888 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.889+0000 7f2ce8ff9640 1 -- 172.21.15.89:0/955323848 <== mgr.4105 v2:172.21.15.192:6832/4076276638 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2cd4001370 con 0x7f2ccc0664d0 2023-10-30T12:51:34.888 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 2 to deep-scrub 2023-10-30T12:51:34.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.889+0000 7f2cf1972640 1 -- 172.21.15.89:0/955323848 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2ccc0664d0 msgr2=0x7f2ccc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.889+0000 7f2cf1972640 1 --2- 172.21.15.89:0/955323848 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2ccc0664d0 0x7f2ccc068990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f2cec0c96e0 tx=0x7f2cd4009040 comp rx=0 tx=0).stop 2023-10-30T12:51:34.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.889+0000 7f2cf1972640 1 -- 172.21.15.89:0/955323848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cec0c4430 msgr2=0x7f2cec0c7860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.889+0000 7f2cf1972640 1 --2- 172.21.15.89:0/955323848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cec0c4430 0x7f2cec0c7860 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc0ad040 tx=0x7f2cdc01ad40 comp rx=0 tx=0).stop 2023-10-30T12:51:34.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.889+0000 7f2cf1972640 1 -- 172.21.15.89:0/955323848 shutdown_connections 2023-10-30T12:51:34.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.889+0000 7f2cf1972640 1 --2- 172.21.15.89:0/955323848 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2ccc0664d0 0x7f2ccc068990 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.889+0000 7f2cf1972640 1 --2- 172.21.15.89:0/955323848 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cec153c60 0x7f2cec1f5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.889+0000 7f2cf1972640 1 --2- 172.21.15.89:0/955323848 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cec159830 0x7f2cec145e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.889+0000 7f2cf1972640 1 --2- 172.21.15.89:0/955323848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cec0c4430 0x7f2cec0c7860 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.889+0000 7f2cf1972640 1 -- 172.21.15.89:0/955323848 >> 172.21.15.89:0/955323848 conn(0x7f2cec06bfe0 msgr2=0x7f2cec0c56d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:34.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.889+0000 7f2cf1972640 1 -- 172.21.15.89:0/955323848 shutdown_connections 2023-10-30T12:51:34.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:34.889+0000 7f2cf1972640 1 -- 172.21.15.89:0/955323848 wait complete. 2023-10-30T12:51:34.909 INFO:tasks.ceph:Scrubbing osd.3 2023-10-30T12:51:34.909 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-10-30T12:51:35.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.049+0000 7f2e402ea640 1 Processor -- start 2023-10-30T12:51:35.049 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.049+0000 7f2e402ea640 1 -- start start 2023-10-30T12:51:35.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.049+0000 7f2e402ea640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2e3806b8f0 0x7f2e38159840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:35.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.049+0000 7f2e402ea640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e380c17e0 con 0x7f2e380c7b10 2023-10-30T12:51:35.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.049+0000 7f2e402ea640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e3806bcd0 con 0x7f2e380cbb00 2023-10-30T12:51:35.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.049+0000 7f2e402ea640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e380c6c60 con 0x7f2e3806b8f0 2023-10-30T12:51:35.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.049+0000 7f2e3e887640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2e380c7b10 0x7f2e38164100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36874/0 (socket says 172.21.15.89:36874) 2023-10-30T12:51:35.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.049+0000 7f2e3e887640 1 -- 172.21.15.89:0/1778331145 learned_addr learned my addr 172.21.15.89:0/1778331145 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:51:35.052 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.049+0000 7f2e3e086640 1 --2- 172.21.15.89:0/1778331145 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2e3806b8f0 0x7f2e38159840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:35.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.049+0000 7f2e3d084640 1 -- 172.21.15.89:0/1778331145 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 500435190 0 0) 0x7f2e300014a0 con 0x7f2e380c7b10 2023-10-30T12:51:35.054 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.049+0000 7f2e3d084640 1 -- 172.21.15.89:0/1778331145 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e24003530 con 0x7f2e380c7b10 2023-10-30T12:51:35.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.049+0000 7f2e3d084640 1 -- 172.21.15.89:0/1778331145 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1594392816 0 0) 0x7f2e24003530 con 0x7f2e380c7b10 2023-10-30T12:51:35.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.053+0000 7f2e3d084640 1 -- 172.21.15.89:0/1778331145 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2e3806b8f0 msgr2=0x7f2e38159840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:35.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.053+0000 7f2e3d084640 1 --2- 172.21.15.89:0/1778331145 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2e3806b8f0 0x7f2e38159840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.055 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.053+0000 7f2e3d084640 1 -- 172.21.15.89:0/1778331145 >> v1:172.21.15.192:6789/0 conn(0x7f2e380cbb00 legacy=0x7f2e3806c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:35.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.053+0000 7f2e3d084640 1 -- 172.21.15.89:0/1778331145 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e380c0f40 con 0x7f2e380c7b10 2023-10-30T12:51:35.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.053+0000 7f2e3d084640 1 -- 172.21.15.89:0/1778331145 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f2e30001b90 con 0x7f2e380c7b10 2023-10-30T12:51:35.056 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.053+0000 7f2e3d084640 1 -- 172.21.15.89:0/1778331145 >> v1:172.21.15.89:6789/0 conn(0x7f2e380c7b10 legacy=0x7f2e38164100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:35.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.053+0000 7f2e3d084640 1 --2- 172.21.15.89:0/1778331145 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e240039b0 0x7f2e24003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:35.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.053+0000 7f2e3d084640 1 --2- 172.21.15.89:0/1778331145 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2e3806b8f0 0x7f2e240047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:35.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.053+0000 7f2e3d084640 1 --2- 172.21.15.89:0/1778331145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e24004d10 0x7f2e24005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:35.057 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.053+0000 7f2e3d084640 1 -- 172.21.15.89:0/1778331145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e380c6c60 con 0x7f2e24004d10 2023-10-30T12:51:35.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.053+0000 7f2e3d084640 1 -- 172.21.15.89:0/1778331145 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2400b990 con 0x7f2e240039b0 2023-10-30T12:51:35.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.053+0000 7f2e3d084640 1 -- 172.21.15.89:0/1778331145 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2400bb50 con 0x7f2e3806b8f0 2023-10-30T12:51:35.058 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.053+0000 7f2e3e887640 1 --2- 172.21.15.89:0/1778331145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e24004d10 0x7f2e24005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:35.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.053+0000 7f2e3d885640 1 --2- 172.21.15.89:0/1778331145 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2e3806b8f0 0x7f2e240047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:35.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.053+0000 7f2e3e086640 1 --2- 172.21.15.89:0/1778331145 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e240039b0 0x7f2e24003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:35.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.053+0000 7f2e3e887640 1 -- 172.21.15.89:0/1778331145 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2e3806b8f0 msgr2=0x7f2e240047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:35.059 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.053+0000 7f2e3e887640 1 --2- 172.21.15.89:0/1778331145 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2e3806b8f0 0x7f2e240047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.053+0000 7f2e3e887640 1 -- 172.21.15.89:0/1778331145 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e240039b0 msgr2=0x7f2e24003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:35.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.053+0000 7f2e3e887640 1 --2- 172.21.15.89:0/1778331145 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e240039b0 0x7f2e24003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.053+0000 7f2e3e887640 1 -- 172.21.15.89:0/1778331145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e24003530 con 0x7f2e24004d10 2023-10-30T12:51:35.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.053+0000 7f2e3e887640 1 --2- 172.21.15.89:0/1778331145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e24004d10 0x7f2e24005190 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f2e3000a840 tx=0x7f2e3001fa80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eadf1d34907411bb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:35.064 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.053+0000 7f2e3d084640 1 -- 172.21.15.89:0/1778331145 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e300aa6b0 con 0x7f2e24004d10 2023-10-30T12:51:35.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.057+0000 7f2e3d084640 1 -- 172.21.15.89:0/1778331145 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e300aa830 con 0x7f2e24004d10 2023-10-30T12:51:35.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.057+0000 7f2e3d084640 1 -- 172.21.15.89:0/1778331145 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e300aabe0 con 0x7f2e24004d10 2023-10-30T12:51:35.065 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.057+0000 7f2e402ea640 1 -- 172.21.15.89:0/1778331145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e24004d10 msgr2=0x7f2e24005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:35.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.057+0000 7f2e402ea640 1 --2- 172.21.15.89:0/1778331145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e24004d10 0x7f2e24005190 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f2e3000a840 tx=0x7f2e3001fa80 comp rx=0 tx=0).stop 2023-10-30T12:51:35.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.057+0000 7f2e3e887640 1 -- 172.21.15.89:0/1778331145 reap_dead start 2023-10-30T12:51:35.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.057+0000 7f2e402ea640 1 -- 172.21.15.89:0/1778331145 shutdown_connections 2023-10-30T12:51:35.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.057+0000 7f2e402ea640 1 -- 172.21.15.89:0/1778331145 >> 172.21.15.89:0/1778331145 conn(0x7f2e380c8a20 msgr2=0x7f2e380c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:35.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.057+0000 7f2e402ea640 1 -- 172.21.15.89:0/1778331145 shutdown_connections 2023-10-30T12:51:35.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.057+0000 7f2e402ea640 1 -- 172.21.15.89:0/1778331145 wait complete. 2023-10-30T12:51:35.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.057+0000 7f2e402ea640 1 Processor -- start 2023-10-30T12:51:35.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.057+0000 7f2e402ea640 1 -- start start 2023-10-30T12:51:35.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.057+0000 7f2e402ea640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e24004d10 0x7f2e381f2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:35.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.057+0000 7f2e402ea640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2e380ca4e0 0x7f2e381f2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:35.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.057+0000 7f2e402ea640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e381f0430 0x7f2e381f08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:35.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.057+0000 7f2e402ea640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e381f0010 con 0x7f2e24004d10 2023-10-30T12:51:35.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.057+0000 7f2e402ea640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e381f0df0 con 0x7f2e381f0430 2023-10-30T12:51:35.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.057+0000 7f2e402ea640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e381f2c30 con 0x7f2e380ca4e0 2023-10-30T12:51:35.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.057+0000 7f2e3e086640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e24004d10 0x7f2e381f2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:35.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.057+0000 7f2e3e086640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e24004d10 0x7f2e381f2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47428/0 (socket says 172.21.15.89:47428) 2023-10-30T12:51:35.070 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.057+0000 7f2e3e086640 1 -- 172.21.15.89:0/3120524638 learned_addr learned my addr 172.21.15.89:0/3120524638 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:35.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.061+0000 7f2e3e887640 1 --2- 172.21.15.89:0/3120524638 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e381f0430 0x7f2e381f08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:35.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.061+0000 7f2e3d885640 1 --2- 172.21.15.89:0/3120524638 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2e380ca4e0 0x7f2e381f2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:35.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.061+0000 7f2e3e086640 1 -- 172.21.15.89:0/3120524638 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2e380ca4e0 msgr2=0x7f2e381f2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:35.071 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.061+0000 7f2e3e086640 1 --2- 172.21.15.89:0/3120524638 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2e380ca4e0 0x7f2e381f2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.061+0000 7f2e3e086640 1 -- 172.21.15.89:0/3120524638 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e381f0430 msgr2=0x7f2e381f08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:35.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.061+0000 7f2e3e086640 1 --2- 172.21.15.89:0/3120524638 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e381f0430 0x7f2e381f08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.061+0000 7f2e3e086640 1 -- 172.21.15.89:0/3120524638 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e380af060 con 0x7f2e24004d10 2023-10-30T12:51:35.072 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.061+0000 7f2e3e086640 1 --2- 172.21.15.89:0/3120524638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e24004d10 0x7f2e381f2100 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f2e340119d0 tx=0x7f2e34011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:35.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.061+0000 7f2e2effd640 1 -- 172.21.15.89:0/3120524638 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3400fb50 con 0x7f2e24004d10 2023-10-30T12:51:35.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.061+0000 7f2e2effd640 1 -- 172.21.15.89:0/3120524638 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3400fcb0 con 0x7f2e24004d10 2023-10-30T12:51:35.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.061+0000 7f2e2effd640 1 -- 172.21.15.89:0/3120524638 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e34014440 con 0x7f2e24004d10 2023-10-30T12:51:35.073 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.061+0000 7f2e402ea640 1 -- 172.21.15.89:0/3120524638 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e380af340 con 0x7f2e24004d10 2023-10-30T12:51:35.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.061+0000 7f2e402ea640 1 -- 172.21.15.89:0/3120524638 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e38157180 con 0x7f2e24004d10 2023-10-30T12:51:35.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.061+0000 7f2e402ea640 1 -- 172.21.15.89:0/3120524638 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2e380c1500 con 0x7f2e24004d10 2023-10-30T12:51:35.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.061+0000 7f2e2effd640 1 -- 172.21.15.89:0/3120524638 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2e3401d400 con 0x7f2e24004d10 2023-10-30T12:51:35.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.061+0000 7f2e2effd640 1 --2- 172.21.15.89:0/3120524638 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2e1c0664a0 0x7f2e1c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:35.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.061+0000 7f2e2effd640 1 -- 172.21.15.89:0/3120524638 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f2e340851b0 con 0x7f2e24004d10 2023-10-30T12:51:35.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.061+0000 7f2e2effd640 1 --2- 172.21.15.89:0/3120524638 >> [v2:172.21.15.89:6824/345087939,v1:172.21.15.89:6825/345087939] conn(0x7f2e1c06efa0 0x7f2e1c071400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:35.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.061+0000 7f2e2effd640 1 -- 172.21.15.89:0/3120524638 --> [v2:172.21.15.89:6824/345087939,v1:172.21.15.89:6825/345087939] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2e1c071ad0 con 0x7f2e1c06efa0 2023-10-30T12:51:35.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.061+0000 7f2e2effd640 1 -- 172.21.15.89:0/3120524638 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e34085520 con 0x7f2e24004d10 2023-10-30T12:51:35.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.061+0000 7f2e3d885640 1 --2- 172.21.15.89:0/3120524638 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2e1c0664a0 0x7f2e1c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:35.076 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.065+0000 7f2e3e887640 1 --2- 172.21.15.89:0/3120524638 >> [v2:172.21.15.89:6824/345087939,v1:172.21.15.89:6825/345087939] conn(0x7f2e1c06efa0 0x7f2e1c071400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:35.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.065+0000 7f2e3e887640 1 --2- 172.21.15.89:0/3120524638 >> [v2:172.21.15.89:6824/345087939,v1:172.21.15.89:6825/345087939] conn(0x7f2e1c06efa0 0x7f2e1c071400 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:35.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.065+0000 7f2e3d885640 1 --2- 172.21.15.89:0/3120524638 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2e1c0664a0 0x7f2e1c068960 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f2e38153410 tx=0x7f2e28009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:35.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.065+0000 7f2e2effd640 1 -- 172.21.15.89:0/3120524638 <== osd.3 v2:172.21.15.89:6824/345087939 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f2e1c071ad0 con 0x7f2e1c06efa0 2023-10-30T12:51:35.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.077+0000 7f2e402ea640 1 -- 172.21.15.89:0/3120524638 --> [v2:172.21.15.89:6824/345087939,v1:172.21.15.89:6825/345087939] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f2e380c1220 con 0x7f2e1c06efa0 2023-10-30T12:51:35.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.077+0000 7f2e2effd640 1 -- 172.21.15.89:0/3120524638 <== osd.3 v2:172.21.15.89:6824/345087939 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f2e300014a0 con 0x7f2e1c06efa0 2023-10-30T12:51:35.079 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-10-30T12:51:35.079 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T12:51:35.079 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-10-30T12:51:35.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.077+0000 7f2e402ea640 1 -- 172.21.15.89:0/3120524638 >> [v2:172.21.15.89:6824/345087939,v1:172.21.15.89:6825/345087939] conn(0x7f2e1c06efa0 msgr2=0x7f2e1c071400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:35.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.077+0000 7f2e402ea640 1 --2- 172.21.15.89:0/3120524638 >> [v2:172.21.15.89:6824/345087939,v1:172.21.15.89:6825/345087939] conn(0x7f2e1c06efa0 0x7f2e1c071400 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.077+0000 7f2e402ea640 1 -- 172.21.15.89:0/3120524638 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2e1c0664a0 msgr2=0x7f2e1c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:35.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.077+0000 7f2e402ea640 1 --2- 172.21.15.89:0/3120524638 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f2e1c0664a0 0x7f2e1c068960 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f2e38153410 tx=0x7f2e28009040 comp rx=0 tx=0).stop 2023-10-30T12:51:35.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.077+0000 7f2e402ea640 1 -- 172.21.15.89:0/3120524638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e24004d10 msgr2=0x7f2e381f2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:35.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.077+0000 7f2e402ea640 1 --2- 172.21.15.89:0/3120524638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e24004d10 0x7f2e381f2100 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f2e340119d0 tx=0x7f2e34011e90 comp rx=0 tx=0).stop 2023-10-30T12:51:35.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.077+0000 7f2e3e887640 1 -- 172.21.15.89:0/3120524638 reap_dead start 2023-10-30T12:51:35.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.077+0000 7f2e402ea640 1 -- 172.21.15.89:0/3120524638 shutdown_connections 2023-10-30T12:51:35.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.077+0000 7f2e402ea640 1 -- 172.21.15.89:0/3120524638 >> 172.21.15.89:0/3120524638 conn(0x7f2e380c8a20 msgr2=0x7f2e380c6da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:35.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.077+0000 7f2e402ea640 1 -- 172.21.15.89:0/3120524638 shutdown_connections 2023-10-30T12:51:35.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.077+0000 7f2e402ea640 1 -- 172.21.15.89:0/3120524638 wait complete. 2023-10-30T12:51:35.095 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-10-30T12:51:35.234 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.233+0000 7f99f1a91640 1 Processor -- start 2023-10-30T12:51:35.235 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.233+0000 7f99f1a91640 1 -- start start 2023-10-30T12:51:35.236 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.237+0000 7f99f1a91640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f99ec0c7830 0x7f99ec06ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:35.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.237+0000 7f99f1a91640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99ec06d710 con 0x7f99ec0cce00 2023-10-30T12:51:35.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.237+0000 7f99f1a91640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99ec06b560 con 0x7f99ec06e430 2023-10-30T12:51:35.237 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.237+0000 7f99f1a91640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99ec06dd30 con 0x7f99ec0c7830 2023-10-30T12:51:35.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.237+0000 7f99eb7fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f99ec0cce00 0x7f99ec159850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36882/0 (socket says 172.21.15.89:36882) 2023-10-30T12:51:35.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.237+0000 7f99eb7fe640 1 -- 172.21.15.89:0/3443312176 learned_addr learned my addr 172.21.15.89:0/3443312176 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:51:35.238 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.237+0000 7f99eaffd640 1 --2- 172.21.15.89:0/3443312176 >> v2:172.21.15.89:3300/0 conn(0x7f99ec0c7830 0x7f99ec06ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:35.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.237+0000 7f99ea7fc640 1 -- 172.21.15.89:0/3443312176 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 793934488 0 0) 0x7f99ec06d710 con 0x7f99ec0cce00 2023-10-30T12:51:35.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.237+0000 7f99ea7fc640 1 -- 172.21.15.89:0/3443312176 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99d4003530 con 0x7f99ec0cce00 2023-10-30T12:51:35.239 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.237+0000 7f99eaffd640 1 -- 172.21.15.89:0/3443312176 >> v1:172.21.15.192:6789/0 conn(0x7f99ec06e430 legacy=0x7f99ec159f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:35.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.237+0000 7f99eaffd640 1 -- 172.21.15.89:0/3443312176 >> v1:172.21.15.89:6789/0 conn(0x7f99ec0cce00 legacy=0x7f99ec159850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:35.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.237+0000 7f99eaffd640 1 -- 172.21.15.89:0/3443312176 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99ec0c3c80 con 0x7f99ec0c7830 2023-10-30T12:51:35.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.237+0000 7f99eaffd640 1 --2- 172.21.15.89:0/3443312176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99ec0c7830 0x7f99ec06ca50 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f99e00097c0 tx=0x7f99e001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cba65442fb4b904 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:35.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.237+0000 7f99ea7fc640 1 -- 172.21.15.89:0/3443312176 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e00ab600 con 0x7f99ec0c7830 2023-10-30T12:51:35.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.237+0000 7f99ea7fc640 1 -- 172.21.15.89:0/3443312176 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e00a7050 con 0x7f99ec0c7830 2023-10-30T12:51:35.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.237+0000 7f99ea7fc640 1 -- 172.21.15.89:0/3443312176 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e00ab980 con 0x7f99ec0c7830 2023-10-30T12:51:35.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.241+0000 7f99f1a91640 1 -- 172.21.15.89:0/3443312176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99ec0c7830 msgr2=0x7f99ec06ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:35.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.241+0000 7f99f1a91640 1 --2- 172.21.15.89:0/3443312176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99ec0c7830 0x7f99ec06ca50 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f99e00097c0 tx=0x7f99e001c3f0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.241+0000 7f99f1a91640 1 -- 172.21.15.89:0/3443312176 shutdown_connections 2023-10-30T12:51:35.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.241+0000 7f99f1a91640 1 --2- 172.21.15.89:0/3443312176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99ec0c7830 0x7f99ec06ca50 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.241+0000 7f99f1a91640 1 -- 172.21.15.89:0/3443312176 >> 172.21.15.89:0/3443312176 conn(0x7f99ec071ea0 msgr2=0x7f99ec065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:35.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.241+0000 7f99f1a91640 1 -- 172.21.15.89:0/3443312176 shutdown_connections 2023-10-30T12:51:35.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.241+0000 7f99f1a91640 1 -- 172.21.15.89:0/3443312176 wait complete. 2023-10-30T12:51:35.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.241+0000 7f99f1a91640 1 Processor -- start 2023-10-30T12:51:35.247 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.241+0000 7f99f1a91640 1 -- start start 2023-10-30T12:51:35.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.241+0000 7f99f1a91640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99ec06e430 0x7f99ec0c4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:35.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.241+0000 7f99f1a91640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99ec0c7830 0x7f99ec0c5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:35.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.241+0000 7f99f1a91640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99ec0cce00 0x7f99ec152a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:35.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.241+0000 7f99f1a91640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99ec158ae0 con 0x7f99ec0cce00 2023-10-30T12:51:35.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.241+0000 7f99f1a91640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99ec159ab0 con 0x7f99ec06e430 2023-10-30T12:51:35.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.241+0000 7f99f1a91640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99ec0c4850 con 0x7f99ec0c7830 2023-10-30T12:51:35.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.241+0000 7f99ebfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99ec0cce00 0x7f99ec152a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:35.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.241+0000 7f99ebfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99ec0cce00 0x7f99ec152a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47446/0 (socket says 172.21.15.89:47446) 2023-10-30T12:51:35.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.241+0000 7f99ebfff640 1 -- 172.21.15.89:0/3979077649 learned_addr learned my addr 172.21.15.89:0/3979077649 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:35.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.241+0000 7f99eaffd640 1 --2- 172.21.15.89:0/3979077649 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99ec0c7830 0x7f99ec0c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:35.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.241+0000 7f99eb7fe640 1 --2- 172.21.15.89:0/3979077649 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99ec06e430 0x7f99ec0c4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:35.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.245+0000 7f99ebfff640 1 -- 172.21.15.89:0/3979077649 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99ec0c7830 msgr2=0x7f99ec0c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:35.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.245+0000 7f99ebfff640 1 --2- 172.21.15.89:0/3979077649 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99ec0c7830 0x7f99ec0c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.245+0000 7f99ebfff640 1 -- 172.21.15.89:0/3979077649 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99ec06e430 msgr2=0x7f99ec0c4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:35.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.245+0000 7f99ebfff640 1 --2- 172.21.15.89:0/3979077649 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99ec06e430 0x7f99ec0c4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.245+0000 7f99ebfff640 1 -- 172.21.15.89:0/3979077649 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e0009420 con 0x7f99ec0cce00 2023-10-30T12:51:35.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.245+0000 7f99ebfff640 1 --2- 172.21.15.89:0/3979077649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99ec0cce00 0x7f99ec152a30 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f99e400c6e0 tx=0x7f99e400cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:35.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.245+0000 7f99d3fff640 1 -- 172.21.15.89:0/3979077649 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e4003bd0 con 0x7f99ec0cce00 2023-10-30T12:51:35.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.245+0000 7f99d3fff640 1 -- 172.21.15.89:0/3979077649 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e4003d50 con 0x7f99ec0cce00 2023-10-30T12:51:35.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.245+0000 7f99f1a91640 1 -- 172.21.15.89:0/3979077649 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99ec0c4240 con 0x7f99ec0cce00 2023-10-30T12:51:35.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.245+0000 7f99d3fff640 1 -- 172.21.15.89:0/3979077649 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e4005310 con 0x7f99ec0cce00 2023-10-30T12:51:35.253 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.245+0000 7f99f1a91640 1 -- 172.21.15.89:0/3979077649 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99ec0c44e0 con 0x7f99ec0cce00 2023-10-30T12:51:35.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.245+0000 7f99f1a91640 1 -- 172.21.15.89:0/3979077649 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99ec0c40a0 con 0x7f99ec0cce00 2023-10-30T12:51:35.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.245+0000 7f99d3fff640 1 -- 172.21.15.89:0/3979077649 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f99e4005470 con 0x7f99ec0cce00 2023-10-30T12:51:35.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.245+0000 7f99d3fff640 1 --2- 172.21.15.89:0/3979077649 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f99d80664a0 0x7f99d8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:35.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.249+0000 7f99d3fff640 1 -- 172.21.15.89:0/3979077649 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f99e4083fe0 con 0x7f99ec0cce00 2023-10-30T12:51:35.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.249+0000 7f99eb7fe640 1 --2- 172.21.15.89:0/3979077649 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f99d80664a0 0x7f99d8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:35.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.249+0000 7f99eb7fe640 1 --2- 172.21.15.89:0/3979077649 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f99d80664a0 0x7f99d8068960 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f99dc0011f0 tx=0x7f99dc007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:35.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.249+0000 7f99d3fff640 1 -- 172.21.15.89:0/3979077649 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f99ec0c40a0 con 0x7f99ec0cce00 2023-10-30T12:51:35.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.421+0000 7f99f1a91640 1 -- 172.21.15.89:0/3979077649 --> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f99ec15d9d0 con 0x7f99d80664a0 2023-10-30T12:51:35.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.421+0000 7f99d3fff640 1 -- 172.21.15.89:0/3979077649 <== mgr.4105 v2:172.21.15.192:6832/4076276638 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f99dc005860 con 0x7f99d80664a0 2023-10-30T12:51:35.423 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 3 to deep-scrub 2023-10-30T12:51:35.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.425+0000 7f99f1a91640 1 -- 172.21.15.89:0/3979077649 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f99d80664a0 msgr2=0x7f99d8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:35.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.425+0000 7f99f1a91640 1 --2- 172.21.15.89:0/3979077649 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f99d80664a0 0x7f99d8068960 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f99dc0011f0 tx=0x7f99dc007d40 comp rx=0 tx=0).stop 2023-10-30T12:51:35.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.425+0000 7f99f1a91640 1 -- 172.21.15.89:0/3979077649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99ec0cce00 msgr2=0x7f99ec152a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:35.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.425+0000 7f99f1a91640 1 --2- 172.21.15.89:0/3979077649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99ec0cce00 0x7f99ec152a30 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f99e400c6e0 tx=0x7f99e400cba0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.425+0000 7f99f1a91640 1 -- 172.21.15.89:0/3979077649 shutdown_connections 2023-10-30T12:51:35.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.425+0000 7f99f1a91640 1 --2- 172.21.15.89:0/3979077649 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f99d80664a0 0x7f99d8068960 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.425+0000 7f99f1a91640 1 --2- 172.21.15.89:0/3979077649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99ec0cce00 0x7f99ec152a30 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.425+0000 7f99f1a91640 1 --2- 172.21.15.89:0/3979077649 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f99ec0c7830 0x7f99ec0c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.425+0000 7f99f1a91640 1 --2- 172.21.15.89:0/3979077649 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99ec06e430 0x7f99ec0c4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.425+0000 7f99f1a91640 1 -- 172.21.15.89:0/3979077649 >> 172.21.15.89:0/3979077649 conn(0x7f99ec071ea0 msgr2=0x7f99ec0cb490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:35.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.425+0000 7f99f1a91640 1 -- 172.21.15.89:0/3979077649 shutdown_connections 2023-10-30T12:51:35.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.425+0000 7f99f1a91640 1 -- 172.21.15.89:0/3979077649 wait complete. 2023-10-30T12:51:35.445 INFO:tasks.ceph:Scrubbing osd.4 2023-10-30T12:51:35.445 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-10-30T12:51:35.588 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.589+0000 7f0eea19d640 1 Processor -- start 2023-10-30T12:51:35.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.589+0000 7f0eea19d640 1 -- start start 2023-10-30T12:51:35.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.589+0000 7f0eea19d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0ee414cf40 0x7f0ee4069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:35.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.589+0000 7f0eea19d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ee40c8430 0x7f0ee40c6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:35.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.589+0000 7f0eea19d640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee415a270 con 0x7f0ee415a700 2023-10-30T12:51:35.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.589+0000 7f0eea19d640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ee406c7c0 con 0x7f0ee414cf40 2023-10-30T12:51:35.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.589+0000 7f0eea19d640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee406a3d0 con 0x7f0ee40c8430 2023-10-30T12:51:35.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.589+0000 7f0ee37fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f0ee414cf40 0x7f0ee4069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:35.591 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.589+0000 7f0ee2ffd640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f0ee415a700 0x7f0ee4156490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36890/0 (socket says 172.21.15.89:36890) 2023-10-30T12:51:35.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.589+0000 7f0ee2ffd640 1 -- 172.21.15.89:0/2581020552 learned_addr learned my addr 172.21.15.89:0/2581020552 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:51:35.592 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.589+0000 7f0ee3fff640 1 --2- 172.21.15.89:0/2581020552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ee40c8430 0x7f0ee40c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:35.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.589+0000 7f0ee37fe640 1 -- 172.21.15.89:0/2581020552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ee40c8430 msgr2=0x7f0ee40c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:35.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.589+0000 7f0ee37fe640 1 --2- 172.21.15.89:0/2581020552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ee40c8430 0x7f0ee40c6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.589+0000 7f0ee37fe640 1 -- 172.21.15.89:0/2581020552 >> v1:172.21.15.89:6789/0 conn(0x7f0ee415a700 legacy=0x7f0ee4156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:35.593 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.589+0000 7f0ee37fe640 1 -- 172.21.15.89:0/2581020552 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee4156ed0 con 0x7f0ee414cf40 2023-10-30T12:51:35.594 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.589+0000 7f0ee27fc640 1 -- 172.21.15.89:0/2581020552 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3448385685 0 0) 0x7f0ee415a270 con 0x7f0ee415a700 2023-10-30T12:51:35.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.593+0000 7f0ee37fe640 1 --2- 172.21.15.89:0/2581020552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ee414cf40 0x7f0ee4069e90 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f0ed40097c0 tx=0x7f0ed401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb097158c5bed04e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:35.595 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.593+0000 7f0ee27fc640 1 -- 172.21.15.89:0/2581020552 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed40ab600 con 0x7f0ee414cf40 2023-10-30T12:51:35.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.593+0000 7f0ee27fc640 1 -- 172.21.15.89:0/2581020552 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed40a7050 con 0x7f0ee414cf40 2023-10-30T12:51:35.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.593+0000 7f0ee27fc640 1 -- 172.21.15.89:0/2581020552 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed40ab980 con 0x7f0ee414cf40 2023-10-30T12:51:35.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.593+0000 7f0eea19d640 1 -- 172.21.15.89:0/2581020552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ee414cf40 msgr2=0x7f0ee4069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:35.596 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.593+0000 7f0eea19d640 1 --2- 172.21.15.89:0/2581020552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ee414cf40 0x7f0ee4069e90 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f0ed40097c0 tx=0x7f0ed401c480 comp rx=0 tx=0).stop 2023-10-30T12:51:35.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.593+0000 7f0eea19d640 1 -- 172.21.15.89:0/2581020552 shutdown_connections 2023-10-30T12:51:35.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.593+0000 7f0eea19d640 1 --2- 172.21.15.89:0/2581020552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ee40c8430 0x7f0ee40c6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.593+0000 7f0eea19d640 1 --2- 172.21.15.89:0/2581020552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ee414cf40 0x7f0ee4069e90 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.597 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.593+0000 7f0eea19d640 1 -- 172.21.15.89:0/2581020552 >> 172.21.15.89:0/2581020552 conn(0x7f0ee40c7910 msgr2=0x7f0ee40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:35.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.593+0000 7f0eea19d640 1 -- 172.21.15.89:0/2581020552 shutdown_connections 2023-10-30T12:51:35.598 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.593+0000 7f0eea19d640 1 -- 172.21.15.89:0/2581020552 wait complete. 2023-10-30T12:51:35.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.597+0000 7f0eea19d640 1 Processor -- start 2023-10-30T12:51:35.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.597+0000 7f0eea19d640 1 -- start start 2023-10-30T12:51:35.600 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.597+0000 7f0eea19d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ee40c8430 0x7f0ee41efdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:35.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.597+0000 7f0eea19d640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ee414cf40 0x7f0ee4143630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:35.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.597+0000 7f0eea19d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ee415a700 0x7f0ee4146030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:35.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.597+0000 7f0eea19d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee4145c10 con 0x7f0ee40c8430 2023-10-30T12:51:35.601 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.597+0000 7f0eea19d640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee4154bc0 con 0x7f0ee415a700 2023-10-30T12:51:35.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.597+0000 7f0eea19d640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee41f03e0 con 0x7f0ee414cf40 2023-10-30T12:51:35.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.597+0000 7f0ee37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ee40c8430 0x7f0ee41efdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:35.602 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.597+0000 7f0ee37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ee40c8430 0x7f0ee41efdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47472/0 (socket says 172.21.15.89:47472) 2023-10-30T12:51:35.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.597+0000 7f0ee37fe640 1 -- 172.21.15.89:0/2641210097 learned_addr learned my addr 172.21.15.89:0/2641210097 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:35.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.597+0000 7f0ee2ffd640 1 --2- 172.21.15.89:0/2641210097 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ee414cf40 0x7f0ee4143630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:35.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.597+0000 7f0ee3fff640 1 --2- 172.21.15.89:0/2641210097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ee415a700 0x7f0ee4146030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:35.603 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.597+0000 7f0ee37fe640 1 -- 172.21.15.89:0/2641210097 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ee414cf40 msgr2=0x7f0ee4143630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:35.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.597+0000 7f0ee37fe640 1 --2- 172.21.15.89:0/2641210097 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0ee414cf40 0x7f0ee4143630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.597+0000 7f0ee37fe640 1 -- 172.21.15.89:0/2641210097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ee415a700 msgr2=0x7f0ee4146030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:35.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.597+0000 7f0ee37fe640 1 --2- 172.21.15.89:0/2641210097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ee415a700 0x7f0ee4146030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.604 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.597+0000 7f0ee37fe640 1 -- 172.21.15.89:0/2641210097 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed4009420 con 0x7f0ee40c8430 2023-10-30T12:51:35.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.597+0000 7f0ee37fe640 1 --2- 172.21.15.89:0/2641210097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ee40c8430 0x7f0ee41efdc0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f0ed401a0b0 tx=0x7f0ed401acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:35.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.597+0000 7f0ecbfff640 1 -- 172.21.15.89:0/2641210097 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed40b7650 con 0x7f0ee40c8430 2023-10-30T12:51:35.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.597+0000 7f0ecbfff640 1 -- 172.21.15.89:0/2641210097 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed40abe90 con 0x7f0ee40c8430 2023-10-30T12:51:35.605 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.597+0000 7f0ecbfff640 1 -- 172.21.15.89:0/2641210097 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed40b7a10 con 0x7f0ee40c8430 2023-10-30T12:51:35.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.597+0000 7f0eea19d640 1 -- 172.21.15.89:0/2641210097 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ee41f06c0 con 0x7f0ee40c8430 2023-10-30T12:51:35.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.597+0000 7f0eea19d640 1 -- 172.21.15.89:0/2641210097 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ee41f0c80 con 0x7f0ee40c8430 2023-10-30T12:51:35.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.597+0000 7f0eea19d640 1 -- 172.21.15.89:0/2641210097 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0ee414d320 con 0x7f0ee40c8430 2023-10-30T12:51:35.606 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.601+0000 7f0ecbfff640 1 -- 172.21.15.89:0/2641210097 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0ed40b7b70 con 0x7f0ee40c8430 2023-10-30T12:51:35.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.601+0000 7f0ecbfff640 1 --2- 172.21.15.89:0/2641210097 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0ed00664a0 0x7f0ed0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:35.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.601+0000 7f0ecbfff640 1 -- 172.21.15.89:0/2641210097 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f0ed411ec00 con 0x7f0ee40c8430 2023-10-30T12:51:35.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.601+0000 7f0ecbfff640 1 --2- 172.21.15.89:0/2641210097 >> [v2:172.21.15.192:6800/2138125545,v1:172.21.15.192:6801/2138125545] conn(0x7f0ed006efa0 0x7f0ed0071400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:35.608 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.601+0000 7f0ecbfff640 1 -- 172.21.15.89:0/2641210097 --> [v2:172.21.15.192:6800/2138125545,v1:172.21.15.192:6801/2138125545] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0ed0071ad0 con 0x7f0ed006efa0 2023-10-30T12:51:35.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.601+0000 7f0ecbfff640 1 -- 172.21.15.89:0/2641210097 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ed4123240 con 0x7f0ee40c8430 2023-10-30T12:51:35.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.601+0000 7f0ee2ffd640 1 --2- 172.21.15.89:0/2641210097 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0ed00664a0 0x7f0ed0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:35.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.601+0000 7f0ee3fff640 1 --2- 172.21.15.89:0/2641210097 >> [v2:172.21.15.192:6800/2138125545,v1:172.21.15.192:6801/2138125545] conn(0x7f0ed006efa0 0x7f0ed0071400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:35.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.601+0000 7f0ee2ffd640 1 --2- 172.21.15.89:0/2641210097 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0ed00664a0 0x7f0ed0068960 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0ee414bd70 tx=0x7f0ed8007b60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:35.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.601+0000 7f0ee3fff640 1 --2- 172.21.15.89:0/2641210097 >> [v2:172.21.15.192:6800/2138125545,v1:172.21.15.192:6801/2138125545] conn(0x7f0ed006efa0 0x7f0ed0071400 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:35.610 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.605+0000 7f0ecbfff640 1 -- 172.21.15.89:0/2641210097 <== osd.4 v2:172.21.15.192:6800/2138125545 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f0ed0071ad0 con 0x7f0ed006efa0 2023-10-30T12:51:35.617 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.617+0000 7f0eea19d640 1 -- 172.21.15.89:0/2641210097 --> [v2:172.21.15.192:6800/2138125545,v1:172.21.15.192:6801/2138125545] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f0ee4159f00 con 0x7f0ed006efa0 2023-10-30T12:51:35.618 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.617+0000 7f0ecbfff640 1 -- 172.21.15.89:0/2641210097 <== osd.4 v2:172.21.15.192:6800/2138125545 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f0edc009ad0 con 0x7f0ed006efa0 2023-10-30T12:51:35.618 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-10-30T12:51:35.618 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T12:51:35.619 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-10-30T12:51:35.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.617+0000 7f0eea19d640 1 -- 172.21.15.89:0/2641210097 >> [v2:172.21.15.192:6800/2138125545,v1:172.21.15.192:6801/2138125545] conn(0x7f0ed006efa0 msgr2=0x7f0ed0071400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:35.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.617+0000 7f0eea19d640 1 --2- 172.21.15.89:0/2641210097 >> [v2:172.21.15.192:6800/2138125545,v1:172.21.15.192:6801/2138125545] conn(0x7f0ed006efa0 0x7f0ed0071400 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.617+0000 7f0eea19d640 1 -- 172.21.15.89:0/2641210097 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0ed00664a0 msgr2=0x7f0ed0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:35.620 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.617+0000 7f0eea19d640 1 --2- 172.21.15.89:0/2641210097 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f0ed00664a0 0x7f0ed0068960 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0ee414bd70 tx=0x7f0ed8007b60 comp rx=0 tx=0).stop 2023-10-30T12:51:35.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.617+0000 7f0eea19d640 1 -- 172.21.15.89:0/2641210097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ee40c8430 msgr2=0x7f0ee41efdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:35.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.617+0000 7f0eea19d640 1 --2- 172.21.15.89:0/2641210097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ee40c8430 0x7f0ee41efdc0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f0ed401a0b0 tx=0x7f0ed401acc0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.617+0000 7f0ee3fff640 1 -- 172.21.15.89:0/2641210097 reap_dead start 2023-10-30T12:51:35.621 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.617+0000 7f0eea19d640 1 -- 172.21.15.89:0/2641210097 shutdown_connections 2023-10-30T12:51:35.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.617+0000 7f0eea19d640 1 -- 172.21.15.89:0/2641210097 >> 172.21.15.89:0/2641210097 conn(0x7f0ee40c7910 msgr2=0x7f0ee406a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:35.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.617+0000 7f0eea19d640 1 -- 172.21.15.89:0/2641210097 shutdown_connections 2023-10-30T12:51:35.622 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.617+0000 7f0eea19d640 1 -- 172.21.15.89:0/2641210097 wait complete. 2023-10-30T12:51:35.636 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-10-30T12:51:35.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.773+0000 7fef98cf7640 1 Processor -- start 2023-10-30T12:51:35.775 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.773+0000 7fef98cf7640 1 -- start start 2023-10-30T12:51:35.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.777+0000 7fef98cf7640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fef940c74e0 0x7fef94155c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:35.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.777+0000 7fef98cf7640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fef9406f540 0x7fef94156180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:35.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.777+0000 7fef98cf7640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef940c0060 con 0x7fef9414fc00 2023-10-30T12:51:35.777 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.777+0000 7fef98cf7640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef941566c0 con 0x7fef9406f540 2023-10-30T12:51:35.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.777+0000 7fef98cf7640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef940c0340 con 0x7fef940c74e0 2023-10-30T12:51:35.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.777+0000 7fef92d9d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fef9406f540 0x7fef94156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:35.778 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.777+0000 7fef92d9d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fef9406f540 0x7fef94156180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47488/0 (socket says 172.21.15.89:47488) 2023-10-30T12:51:35.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.777+0000 7fef92d9d640 1 -- 172.21.15.89:0/1715672695 learned_addr learned my addr 172.21.15.89:0/1715672695 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:35.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.777+0000 7fef89d9b640 1 --2- 172.21.15.89:0/1715672695 >> v2:172.21.15.192:3300/0 conn(0x7fef940c74e0 0x7fef94155c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:35.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.777+0000 7fef92d9d640 1 -- 172.21.15.89:0/1715672695 >> v2:172.21.15.192:3300/0 conn(0x7fef940c74e0 msgr2=0x7fef94155c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:35.779 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.777+0000 7fef92d9d640 1 --2- 172.21.15.89:0/1715672695 >> v2:172.21.15.192:3300/0 conn(0x7fef940c74e0 0x7fef94155c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.777+0000 7fef92d9d640 1 -- 172.21.15.89:0/1715672695 >> v1:172.21.15.89:6789/0 conn(0x7fef9414fc00 legacy=0x7fef9406a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:35.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.777+0000 7fef91d9b640 1 -- 172.21.15.89:0/1715672695 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3831965921 0 0) 0x7fef940c0060 con 0x7fef9414fc00 2023-10-30T12:51:35.780 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.777+0000 7fef92d9d640 1 -- 172.21.15.89:0/1715672695 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef94071090 con 0x7fef9406f540 2023-10-30T12:51:35.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.781+0000 7fef92d9d640 1 --2- 172.21.15.89:0/1715672695 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef9406f540 0x7fef94156180 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fef940c8660 tx=0x7fef8c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a5253e5bcbd5c53 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:35.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.781+0000 7fef91d9b640 1 -- 172.21.15.89:0/1715672695 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8c0a89f0 con 0x7fef9406f540 2023-10-30T12:51:35.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.781+0000 7fef91d9b640 1 -- 172.21.15.89:0/1715672695 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef8c0a8b50 con 0x7fef9406f540 2023-10-30T12:51:35.782 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.781+0000 7fef91d9b640 1 -- 172.21.15.89:0/1715672695 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8c0b0440 con 0x7fef9406f540 2023-10-30T12:51:35.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.781+0000 7fef98cf7640 1 -- 172.21.15.89:0/1715672695 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef9406f540 msgr2=0x7fef94156180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:35.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.781+0000 7fef98cf7640 1 --2- 172.21.15.89:0/1715672695 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef9406f540 0x7fef94156180 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fef940c8660 tx=0x7fef8c01ecf0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.783 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.781+0000 7fef98cf7640 1 -- 172.21.15.89:0/1715672695 shutdown_connections 2023-10-30T12:51:35.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.781+0000 7fef98cf7640 1 --2- 172.21.15.89:0/1715672695 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef9406f540 0x7fef94156180 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.781+0000 7fef98cf7640 1 --2- 172.21.15.89:0/1715672695 >> v2:172.21.15.192:3300/0 conn(0x7fef940c74e0 0x7fef94155c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.781+0000 7fef98cf7640 1 -- 172.21.15.89:0/1715672695 >> 172.21.15.89:0/1715672695 conn(0x7fef940c7910 msgr2=0x7fef940c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:35.784 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.781+0000 7fef98cf7640 1 -- 172.21.15.89:0/1715672695 shutdown_connections 2023-10-30T12:51:35.785 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.781+0000 7fef98cf7640 1 -- 172.21.15.89:0/1715672695 wait complete. 2023-10-30T12:51:35.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.781+0000 7fef98cf7640 1 Processor -- start 2023-10-30T12:51:35.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.781+0000 7fef98cf7640 1 -- start start 2023-10-30T12:51:35.787 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.785+0000 7fef98cf7640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef9406f540 0x7fef94158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:35.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.785+0000 7fef98cf7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef940c74e0 0x7fef94159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:35.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.785+0000 7fef98cf7640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef9414fc00 0x7fef941491e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:35.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.785+0000 7fef98cf7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef940c5370 con 0x7fef940c74e0 2023-10-30T12:51:35.788 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.785+0000 7fef98cf7640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef941eff20 con 0x7fef9414fc00 2023-10-30T12:51:35.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.785+0000 7fef98cf7640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef941f0410 con 0x7fef9406f540 2023-10-30T12:51:35.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.785+0000 7fef89d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef940c74e0 0x7fef94159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:35.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.785+0000 7fef89d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef940c74e0 0x7fef94159530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47492/0 (socket says 172.21.15.89:47492) 2023-10-30T12:51:35.789 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.785+0000 7fef89d9b640 1 -- 172.21.15.89:0/621203736 learned_addr learned my addr 172.21.15.89:0/621203736 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:35.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.785+0000 7fef9259c640 1 --2- 172.21.15.89:0/621203736 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef9406f540 0x7fef94158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:35.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.785+0000 7fef92d9d640 1 --2- 172.21.15.89:0/621203736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef9414fc00 0x7fef941491e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:35.790 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.785+0000 7fef89d9b640 1 -- 172.21.15.89:0/621203736 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef9406f540 msgr2=0x7fef94158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:35.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.785+0000 7fef89d9b640 1 --2- 172.21.15.89:0/621203736 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef9406f540 0x7fef94158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.785+0000 7fef89d9b640 1 -- 172.21.15.89:0/621203736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef9414fc00 msgr2=0x7fef941491e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:35.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.785+0000 7fef89d9b640 1 --2- 172.21.15.89:0/621203736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef9414fc00 0x7fef941491e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.791 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.785+0000 7fef89d9b640 1 -- 172.21.15.89:0/621203736 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef8c0094d0 con 0x7fef940c74e0 2023-10-30T12:51:35.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.785+0000 7fef89d9b640 1 --2- 172.21.15.89:0/621203736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef940c74e0 0x7fef94159530 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fef7c00d800 tx=0x7fef7c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:35.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.785+0000 7fef8b7fe640 1 -- 172.21.15.89:0/621203736 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7c00b6c0 con 0x7fef940c74e0 2023-10-30T12:51:35.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.785+0000 7fef8b7fe640 1 -- 172.21.15.89:0/621203736 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef7c00b820 con 0x7fef940c74e0 2023-10-30T12:51:35.792 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.785+0000 7fef98cf7640 1 -- 172.21.15.89:0/621203736 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef9406a560 con 0x7fef940c74e0 2023-10-30T12:51:35.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.785+0000 7fef8b7fe640 1 -- 172.21.15.89:0/621203736 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7c00bb90 con 0x7fef940c74e0 2023-10-30T12:51:35.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.785+0000 7fef98cf7640 1 -- 172.21.15.89:0/621203736 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef941f0130 con 0x7fef940c74e0 2023-10-30T12:51:35.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.785+0000 7fef98cf7640 1 -- 172.21.15.89:0/621203736 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef940708c0 con 0x7fef940c74e0 2023-10-30T12:51:35.793 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.785+0000 7fef8b7fe640 1 -- 172.21.15.89:0/621203736 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fef7c00bcf0 con 0x7fef940c74e0 2023-10-30T12:51:35.794 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.789+0000 7fef8b7fe640 1 --2- 172.21.15.89:0/621203736 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fef740664a0 0x7fef74068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:35.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.789+0000 7fef8b7fe640 1 -- 172.21.15.89:0/621203736 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7fef7c084740 con 0x7fef940c74e0 2023-10-30T12:51:35.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.789+0000 7fef9259c640 1 --2- 172.21.15.89:0/621203736 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fef740664a0 0x7fef74068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:35.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.789+0000 7fef9259c640 1 --2- 172.21.15.89:0/621203736 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fef740664a0 0x7fef74068960 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fef840012f0 tx=0x7fef84009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:35.795 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.793+0000 7fef8b7fe640 1 -- 172.21.15.89:0/621203736 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fef7c0517b0 con 0x7fef940c74e0 2023-10-30T12:51:35.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.941+0000 7fef98cf7640 1 -- 172.21.15.89:0/621203736 --> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fef94071650 con 0x7fef740664a0 2023-10-30T12:51:35.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.941+0000 7fef8b7fe640 1 -- 172.21.15.89:0/621203736 <== mgr.4105 v2:172.21.15.192:6832/4076276638 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fef8400fe00 con 0x7fef740664a0 2023-10-30T12:51:35.941 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 4 to deep-scrub 2023-10-30T12:51:35.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.941+0000 7fef98cf7640 1 -- 172.21.15.89:0/621203736 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fef740664a0 msgr2=0x7fef74068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:35.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.941+0000 7fef98cf7640 1 --2- 172.21.15.89:0/621203736 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fef740664a0 0x7fef74068960 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fef840012f0 tx=0x7fef84009040 comp rx=0 tx=0).stop 2023-10-30T12:51:35.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.941+0000 7fef98cf7640 1 -- 172.21.15.89:0/621203736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef940c74e0 msgr2=0x7fef94159530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:35.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.941+0000 7fef98cf7640 1 --2- 172.21.15.89:0/621203736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef940c74e0 0x7fef94159530 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fef7c00d800 tx=0x7fef7c00dcc0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.941+0000 7fef98cf7640 1 -- 172.21.15.89:0/621203736 shutdown_connections 2023-10-30T12:51:35.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.941+0000 7fef98cf7640 1 --2- 172.21.15.89:0/621203736 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fef740664a0 0x7fef74068960 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.944 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.941+0000 7fef98cf7640 1 --2- 172.21.15.89:0/621203736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef9414fc00 0x7fef941491e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.941+0000 7fef98cf7640 1 --2- 172.21.15.89:0/621203736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef940c74e0 0x7fef94159530 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.941+0000 7fef98cf7640 1 --2- 172.21.15.89:0/621203736 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef9406f540 0x7fef94158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:35.945 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.941+0000 7fef98cf7640 1 -- 172.21.15.89:0/621203736 >> 172.21.15.89:0/621203736 conn(0x7fef940c7910 msgr2=0x7fef9414e010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:35.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.941+0000 7fef98cf7640 1 -- 172.21.15.89:0/621203736 shutdown_connections 2023-10-30T12:51:35.946 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:35.941+0000 7fef98cf7640 1 -- 172.21.15.89:0/621203736 wait complete. 2023-10-30T12:51:35.962 INFO:tasks.ceph:Scrubbing osd.5 2023-10-30T12:51:35.963 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-10-30T12:51:36.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.105+0000 7f1478e46640 1 Processor -- start 2023-10-30T12:51:36.107 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.105+0000 7f1478e46640 1 -- start start 2023-10-30T12:51:36.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.109+0000 7f1478e46640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f147414cf40 0x7f1474069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.109 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.109+0000 7f1478e46640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f147415a700 0x7f1474156490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.109+0000 7f1478e46640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f14741569d0 0x7f14740c6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.109+0000 7f1478e46640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f147415a270 con 0x7f14741569d0 2023-10-30T12:51:36.110 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.109+0000 7f1478e46640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f147406c7c0 con 0x7f147415a700 2023-10-30T12:51:36.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.109+0000 7f1478e46640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f147406a3d0 con 0x7f147414cf40 2023-10-30T12:51:36.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.109+0000 7f1472d9d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f14741569d0 0x7f14740c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.109+0000 7f1472d9d640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f14741569d0 0x7f14740c6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47498/0 (socket says 172.21.15.89:47498) 2023-10-30T12:51:36.111 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.109+0000 7f1472d9d640 1 -- 172.21.15.89:0/1135785136 learned_addr learned my addr 172.21.15.89:0/1135785136 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:36.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.109+0000 7f147259c640 1 --2- 172.21.15.89:0/1135785136 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f147414cf40 0x7f1474069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.109+0000 7f1471d9b640 1 --2- 172.21.15.89:0/1135785136 >> v2:172.21.15.192:3300/0 conn(0x7f147415a700 0x7f1474156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.112 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.109+0000 7f1472d9d640 1 -- 172.21.15.89:0/1135785136 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f147414cf40 msgr2=0x7f1474069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:36.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.109+0000 7f1472d9d640 1 --2- 172.21.15.89:0/1135785136 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f147414cf40 0x7f1474069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.109+0000 7f1472d9d640 1 -- 172.21.15.89:0/1135785136 >> v2:172.21.15.192:3300/0 conn(0x7f147415a700 msgr2=0x7f1474156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:36.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.109+0000 7f1472d9d640 1 --2- 172.21.15.89:0/1135785136 >> v2:172.21.15.192:3300/0 conn(0x7f147415a700 0x7f1474156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.113 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.109+0000 7f1472d9d640 1 -- 172.21.15.89:0/1135785136 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1474156ed0 con 0x7f14741569d0 2023-10-30T12:51:36.114 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.109+0000 7f147259c640 1 --2- 172.21.15.89:0/1135785136 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f147414cf40 0x7f1474069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:36.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.113+0000 7f1472d9d640 1 --2- 172.21.15.89:0/1135785136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14741569d0 0x7f14740c6920 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f14740cbda0 tx=0x7f146c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b878fedb3d77739 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:36.115 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.113+0000 7f147159a640 1 -- 172.21.15.89:0/1135785136 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c0ac410 con 0x7f14741569d0 2023-10-30T12:51:36.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.113+0000 7f147159a640 1 -- 172.21.15.89:0/1135785136 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f146c0ac570 con 0x7f14741569d0 2023-10-30T12:51:36.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.113+0000 7f147159a640 1 -- 172.21.15.89:0/1135785136 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c0ac900 con 0x7f14741569d0 2023-10-30T12:51:36.116 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.113+0000 7f1478e46640 1 -- 172.21.15.89:0/1135785136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14741569d0 msgr2=0x7f14740c6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.113+0000 7f1478e46640 1 --2- 172.21.15.89:0/1135785136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14741569d0 0x7f14740c6920 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f14740cbda0 tx=0x7f146c01c500 comp rx=0 tx=0).stop 2023-10-30T12:51:36.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.113+0000 7f1478e46640 1 -- 172.21.15.89:0/1135785136 shutdown_connections 2023-10-30T12:51:36.117 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.113+0000 7f1478e46640 1 --2- 172.21.15.89:0/1135785136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14741569d0 0x7f14740c6920 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.113+0000 7f1478e46640 1 --2- 172.21.15.89:0/1135785136 >> v2:172.21.15.192:3300/0 conn(0x7f147415a700 0x7f1474156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.113+0000 7f1478e46640 1 --2- 172.21.15.89:0/1135785136 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f147414cf40 0x7f1474069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.113+0000 7f1478e46640 1 -- 172.21.15.89:0/1135785136 >> 172.21.15.89:0/1135785136 conn(0x7f14740c7910 msgr2=0x7f14740c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:36.118 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.113+0000 7f1478e46640 1 -- 172.21.15.89:0/1135785136 shutdown_connections 2023-10-30T12:51:36.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.113+0000 7f1478e46640 1 -- 172.21.15.89:0/1135785136 wait complete. 2023-10-30T12:51:36.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.113+0000 7f1478e46640 1 Processor -- start 2023-10-30T12:51:36.119 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.113+0000 7f1478e46640 1 -- start start 2023-10-30T12:51:36.122 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.117+0000 7f1478e46640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f147414cf40 0x7f14741f0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.117+0000 7f1478e46640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14741569d0 0x7f14741447f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.117+0000 7f1478e46640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f147415a700 0x7f14741426c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.117+0000 7f1478e46640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14740ca230 con 0x7f147415a700 2023-10-30T12:51:36.123 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.117+0000 7f1478e46640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14740ca510 con 0x7f14741569d0 2023-10-30T12:51:36.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.117+0000 7f1478e46640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14740ca7f0 con 0x7f147414cf40 2023-10-30T12:51:36.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.117+0000 7f1472d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f147415a700 0x7f14741426c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.117+0000 7f1472d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f147415a700 0x7f14741426c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47502/0 (socket says 172.21.15.89:47502) 2023-10-30T12:51:36.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.117+0000 7f1472d9d640 1 -- 172.21.15.89:0/258554863 learned_addr learned my addr 172.21.15.89:0/258554863 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:36.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.117+0000 7f147259c640 1 --2- 172.21.15.89:0/258554863 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f147414cf40 0x7f14741f0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.117+0000 7f1471d9b640 1 --2- 172.21.15.89:0/258554863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14741569d0 0x7f14741447f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.125 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.117+0000 7f1472d9d640 1 -- 172.21.15.89:0/258554863 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f147414cf40 msgr2=0x7f14741f0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.126 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.117+0000 7f1472d9d640 1 --2- 172.21.15.89:0/258554863 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f147414cf40 0x7f14741f0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.126 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.117+0000 7f1472d9d640 1 -- 172.21.15.89:0/258554863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14741569d0 msgr2=0x7f14741447f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.126 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.117+0000 7f1472d9d640 1 --2- 172.21.15.89:0/258554863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14741569d0 0x7f14741447f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.126 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.117+0000 7f1472d9d640 1 -- 172.21.15.89:0/258554863 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146c0094d0 con 0x7f147415a700 2023-10-30T12:51:36.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.117+0000 7f1472d9d640 1 --2- 172.21.15.89:0/258554863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f147415a700 0x7f14741426c0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f146c0af040 tx=0x7f146c0192a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:36.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.117+0000 7f1462ffd640 1 -- 172.21.15.89:0/258554863 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c0acea0 con 0x7f147415a700 2023-10-30T12:51:36.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.117+0000 7f1462ffd640 1 -- 172.21.15.89:0/258554863 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f146c0b65f0 con 0x7f147415a700 2023-10-30T12:51:36.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.117+0000 7f1462ffd640 1 -- 172.21.15.89:0/258554863 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c0ac410 con 0x7f147415a700 2023-10-30T12:51:36.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.117+0000 7f1478e46640 1 -- 172.21.15.89:0/258554863 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14740caad0 con 0x7f147415a700 2023-10-30T12:51:36.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.117+0000 7f1478e46640 1 -- 172.21.15.89:0/258554863 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14740c67c0 con 0x7f147415a700 2023-10-30T12:51:36.128 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.117+0000 7f1478e46640 1 -- 172.21.15.89:0/258554863 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f14740ca680 con 0x7f147415a700 2023-10-30T12:51:36.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.121+0000 7f1462ffd640 1 -- 172.21.15.89:0/258554863 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f146c0b69a0 con 0x7f147415a700 2023-10-30T12:51:36.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.121+0000 7f1462ffd640 1 --2- 172.21.15.89:0/258554863 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f14540664a0 0x7f1454068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.121+0000 7f1462ffd640 1 -- 172.21.15.89:0/258554863 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f146c11eb50 con 0x7f147415a700 2023-10-30T12:51:36.129 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.121+0000 7f1462ffd640 1 --2- 172.21.15.89:0/258554863 >> [v2:172.21.15.192:6804/2360218671,v1:172.21.15.192:6806/2360218671] conn(0x7f145406efa0 0x7f1454071400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.121+0000 7f1462ffd640 1 -- 172.21.15.89:0/258554863 --> [v2:172.21.15.192:6804/2360218671,v1:172.21.15.192:6806/2360218671] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1454071ad0 con 0x7f145406efa0 2023-10-30T12:51:36.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.121+0000 7f1462ffd640 1 -- 172.21.15.89:0/258554863 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7f146c1232c0 con 0x7f147415a700 2023-10-30T12:51:36.130 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.121+0000 7f147259c640 1 --2- 172.21.15.89:0/258554863 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f14540664a0 0x7f1454068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.121+0000 7f1471d9b640 1 --2- 172.21.15.89:0/258554863 >> [v2:172.21.15.192:6804/2360218671,v1:172.21.15.192:6806/2360218671] conn(0x7f145406efa0 0x7f1454071400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.121+0000 7f147259c640 1 --2- 172.21.15.89:0/258554863 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f14540664a0 0x7f1454068960 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f1464003ab0 tx=0x7f1464009810 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:36.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.121+0000 7f1471d9b640 1 --2- 172.21.15.89:0/258554863 >> [v2:172.21.15.192:6804/2360218671,v1:172.21.15.192:6806/2360218671] conn(0x7f145406efa0 0x7f1454071400 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:36.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.125+0000 7f1462ffd640 1 -- 172.21.15.89:0/258554863 <== osd.5 v2:172.21.15.192:6804/2360218671 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f1454071ad0 con 0x7f145406efa0 2023-10-30T12:51:36.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.137+0000 7f1478e46640 1 -- 172.21.15.89:0/258554863 --> [v2:172.21.15.192:6804/2360218671,v1:172.21.15.192:6806/2360218671] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f147414d320 con 0x7f145406efa0 2023-10-30T12:51:36.137 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.137+0000 7f1462ffd640 1 -- 172.21.15.89:0/258554863 <== osd.5 v2:172.21.15.192:6804/2360218671 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f1468007610 con 0x7f145406efa0 2023-10-30T12:51:36.137 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-10-30T12:51:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T12:51:36.138 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-10-30T12:51:36.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.137+0000 7f1478e46640 1 -- 172.21.15.89:0/258554863 >> [v2:172.21.15.192:6804/2360218671,v1:172.21.15.192:6806/2360218671] conn(0x7f145406efa0 msgr2=0x7f1454071400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.137+0000 7f1478e46640 1 --2- 172.21.15.89:0/258554863 >> [v2:172.21.15.192:6804/2360218671,v1:172.21.15.192:6806/2360218671] conn(0x7f145406efa0 0x7f1454071400 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.137+0000 7f1478e46640 1 -- 172.21.15.89:0/258554863 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f14540664a0 msgr2=0x7f1454068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.137+0000 7f1478e46640 1 --2- 172.21.15.89:0/258554863 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f14540664a0 0x7f1454068960 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f1464003ab0 tx=0x7f1464009810 comp rx=0 tx=0).stop 2023-10-30T12:51:36.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.137+0000 7f1478e46640 1 -- 172.21.15.89:0/258554863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f147415a700 msgr2=0x7f14741426c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.140 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.137+0000 7f1478e46640 1 --2- 172.21.15.89:0/258554863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f147415a700 0x7f14741426c0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f146c0af040 tx=0x7f146c0192a0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.137+0000 7f1472d9d640 1 -- 172.21.15.89:0/258554863 reap_dead start 2023-10-30T12:51:36.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.137+0000 7f1478e46640 1 -- 172.21.15.89:0/258554863 shutdown_connections 2023-10-30T12:51:36.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.137+0000 7f1478e46640 1 -- 172.21.15.89:0/258554863 >> 172.21.15.89:0/258554863 conn(0x7f14740c7910 msgr2=0x7f1474145990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:36.141 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.137+0000 7f1478e46640 1 -- 172.21.15.89:0/258554863 shutdown_connections 2023-10-30T12:51:36.142 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.137+0000 7f1478e46640 1 -- 172.21.15.89:0/258554863 wait complete. 2023-10-30T12:51:36.156 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-10-30T12:51:36.294 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.293+0000 7f83b6291640 1 Processor -- start 2023-10-30T12:51:36.295 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.293+0000 7f83b6291640 1 -- start start 2023-10-30T12:51:36.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.297+0000 7f83b6291640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83b0151910 0x7f83b0158120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.297+0000 7f83b6291640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f83b006adb0 0x7f83b00c5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.296 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.297+0000 7f83b6291640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b006a4e0 con 0x7f83b006a970 2023-10-30T12:51:36.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.297+0000 7f83b6291640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b0158660 con 0x7f83b006adb0 2023-10-30T12:51:36.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.297+0000 7f83b6291640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b00c6c50 con 0x7f83b0151910 2023-10-30T12:51:36.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.297+0000 7f83af7fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f83b006a970 0x7f83b00c5810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36916/0 (socket says 172.21.15.89:36916) 2023-10-30T12:51:36.297 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.297+0000 7f83af7fe640 1 -- 172.21.15.89:0/3149908873 learned_addr learned my addr 172.21.15.89:0/3149908873 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:51:36.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.297+0000 7f83affff640 1 --2- 172.21.15.89:0/3149908873 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83b0151910 0x7f83b0158120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.298 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.297+0000 7f83b482e640 1 --2- 172.21.15.89:0/3149908873 >> v2:172.21.15.192:3300/0 conn(0x7f83b006adb0 0x7f83b00c5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.297+0000 7f83aeffd640 1 -- 172.21.15.89:0/3149908873 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783316697 0 0) 0x7f83b006a4e0 con 0x7f83b006a970 2023-10-30T12:51:36.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.297+0000 7f83aeffd640 1 -- 172.21.15.89:0/3149908873 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8398003530 con 0x7f83b006a970 2023-10-30T12:51:36.299 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.297+0000 7f83affff640 1 -- 172.21.15.89:0/3149908873 >> v2:172.21.15.192:3300/0 conn(0x7f83b006adb0 msgr2=0x7f83b00c5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:36.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.297+0000 7f83affff640 1 --2- 172.21.15.89:0/3149908873 >> v2:172.21.15.192:3300/0 conn(0x7f83b006adb0 0x7f83b00c5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.297+0000 7f83affff640 1 -- 172.21.15.89:0/3149908873 >> v1:172.21.15.89:6789/0 conn(0x7f83b006a970 legacy=0x7f83b00c5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.300 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.297+0000 7f83affff640 1 -- 172.21.15.89:0/3149908873 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b00c6f30 con 0x7f83b0151910 2023-10-30T12:51:36.301 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.301+0000 7f83affff640 1 --2- 172.21.15.89:0/3149908873 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83b0151910 0x7f83b0158120 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f83a00097c0 tx=0x7f83a001ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5214a99e4cc67e1c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:36.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.301+0000 7f83aeffd640 1 -- 172.21.15.89:0/3149908873 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a00a8970 con 0x7f83b0151910 2023-10-30T12:51:36.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.301+0000 7f83aeffd640 1 -- 172.21.15.89:0/3149908873 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a00a8ad0 con 0x7f83b0151910 2023-10-30T12:51:36.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.301+0000 7f83aeffd640 1 -- 172.21.15.89:0/3149908873 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a00a8e60 con 0x7f83b0151910 2023-10-30T12:51:36.302 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.301+0000 7f83b6291640 1 -- 172.21.15.89:0/3149908873 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83b0151910 msgr2=0x7f83b0158120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.301+0000 7f83b6291640 1 --2- 172.21.15.89:0/3149908873 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83b0151910 0x7f83b0158120 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f83a00097c0 tx=0x7f83a001ec70 comp rx=0 tx=0).stop 2023-10-30T12:51:36.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.301+0000 7f83b6291640 1 -- 172.21.15.89:0/3149908873 shutdown_connections 2023-10-30T12:51:36.303 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.301+0000 7f83b6291640 1 --2- 172.21.15.89:0/3149908873 >> v2:172.21.15.192:3300/0 conn(0x7f83b006adb0 0x7f83b00c5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.301+0000 7f83b6291640 1 --2- 172.21.15.89:0/3149908873 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83b0151910 0x7f83b0158120 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.301+0000 7f83b6291640 1 -- 172.21.15.89:0/3149908873 >> 172.21.15.89:0/3149908873 conn(0x7f83b00c7910 msgr2=0x7f83b00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:36.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.301+0000 7f83b6291640 1 -- 172.21.15.89:0/3149908873 shutdown_connections 2023-10-30T12:51:36.304 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.301+0000 7f83b6291640 1 -- 172.21.15.89:0/3149908873 wait complete. 2023-10-30T12:51:36.306 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.301+0000 7f83b6291640 1 Processor -- start 2023-10-30T12:51:36.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.301+0000 7f83b6291640 1 -- start start 2023-10-30T12:51:36.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.305+0000 7f83b6291640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83b006a970 0x7f83b0157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.305+0000 7f83b6291640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83b006adb0 0x7f83b00c36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.307 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.305+0000 7f83b6291640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83b0151910 0x7f83b014c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.305+0000 7f83b6291640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b01f29f0 con 0x7f83b006a970 2023-10-30T12:51:36.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.305+0000 7f83b6291640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b01ef680 con 0x7f83b0151910 2023-10-30T12:51:36.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.305+0000 7f83b6291640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b01f1de0 con 0x7f83b006adb0 2023-10-30T12:51:36.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.305+0000 7f83affff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83b006a970 0x7f83b0157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.305+0000 7f83affff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83b006a970 0x7f83b0157dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47514/0 (socket says 172.21.15.89:47514) 2023-10-30T12:51:36.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.305+0000 7f83affff640 1 -- 172.21.15.89:0/3438857991 learned_addr learned my addr 172.21.15.89:0/3438857991 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:36.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.305+0000 7f83b482e640 1 --2- 172.21.15.89:0/3438857991 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83b0151910 0x7f83b014c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.305+0000 7f83af7fe640 1 --2- 172.21.15.89:0/3438857991 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83b006adb0 0x7f83b00c36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.305+0000 7f83affff640 1 -- 172.21.15.89:0/3438857991 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83b006adb0 msgr2=0x7f83b00c36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.305+0000 7f83affff640 1 --2- 172.21.15.89:0/3438857991 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83b006adb0 0x7f83b00c36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.305+0000 7f83affff640 1 -- 172.21.15.89:0/3438857991 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83b0151910 msgr2=0x7f83b014c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.305+0000 7f83affff640 1 --2- 172.21.15.89:0/3438857991 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83b0151910 0x7f83b014c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.305+0000 7f83affff640 1 -- 172.21.15.89:0/3438857991 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a0009420 con 0x7f83b006a970 2023-10-30T12:51:36.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.305+0000 7f83affff640 1 --2- 172.21.15.89:0/3438857991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83b006a970 0x7f83b0157dc0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f83a001d320 tx=0x7f83a001ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:36.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.305+0000 7f83acff9640 1 -- 172.21.15.89:0/3438857991 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a00b0d60 con 0x7f83b006a970 2023-10-30T12:51:36.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.305+0000 7f83acff9640 1 -- 172.21.15.89:0/3438857991 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a00bb440 con 0x7f83b006a970 2023-10-30T12:51:36.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.305+0000 7f83acff9640 1 -- 172.21.15.89:0/3438857991 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a00bb7f0 con 0x7f83b006a970 2023-10-30T12:51:36.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.305+0000 7f83b6291640 1 -- 172.21.15.89:0/3438857991 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b01f20c0 con 0x7f83b006a970 2023-10-30T12:51:36.312 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.305+0000 7f83b6291640 1 -- 172.21.15.89:0/3438857991 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b01f2680 con 0x7f83b006a970 2023-10-30T12:51:36.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.305+0000 7f83b6291640 1 -- 172.21.15.89:0/3438857991 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b004e170 con 0x7f83b006a970 2023-10-30T12:51:36.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.305+0000 7f83acff9640 1 -- 172.21.15.89:0/3438857991 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f83a00b7040 con 0x7f83b006a970 2023-10-30T12:51:36.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.309+0000 7f83acff9640 1 --2- 172.21.15.89:0/3438857991 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f839c0664d0 0x7f839c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.314 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.309+0000 7f83acff9640 1 -- 172.21.15.89:0/3438857991 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f83a01221b0 con 0x7f83b006a970 2023-10-30T12:51:36.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.309+0000 7f83af7fe640 1 --2- 172.21.15.89:0/3438857991 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f839c0664d0 0x7f839c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.309+0000 7f83af7fe640 1 --2- 172.21.15.89:0/3438857991 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f839c0664d0 0x7f839c068990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f83b0150900 tx=0x7f83a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:36.315 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.313+0000 7f83acff9640 1 -- 172.21.15.89:0/3438857991 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f83a00ef190 con 0x7f83b006a970 2023-10-30T12:51:36.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.461+0000 7f83b6291640 1 -- 172.21.15.89:0/3438857991 --> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f83b0069e90 con 0x7f839c0664d0 2023-10-30T12:51:36.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.465+0000 7f83acff9640 1 -- 172.21.15.89:0/3438857991 <== mgr.4105 v2:172.21.15.192:6832/4076276638 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f83a40013f0 con 0x7f839c0664d0 2023-10-30T12:51:36.464 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 5 to deep-scrub 2023-10-30T12:51:36.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.465+0000 7f83b6291640 1 -- 172.21.15.89:0/3438857991 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f839c0664d0 msgr2=0x7f839c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.465+0000 7f83b6291640 1 --2- 172.21.15.89:0/3438857991 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f839c0664d0 0x7f839c068990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f83b0150900 tx=0x7f83a4009040 comp rx=0 tx=0).stop 2023-10-30T12:51:36.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.465+0000 7f83b6291640 1 -- 172.21.15.89:0/3438857991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83b006a970 msgr2=0x7f83b0157dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.465+0000 7f83b6291640 1 --2- 172.21.15.89:0/3438857991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83b006a970 0x7f83b0157dc0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f83a001d320 tx=0x7f83a001ba10 comp rx=0 tx=0).stop 2023-10-30T12:51:36.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.465+0000 7f83b6291640 1 -- 172.21.15.89:0/3438857991 shutdown_connections 2023-10-30T12:51:36.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.465+0000 7f83b6291640 1 --2- 172.21.15.89:0/3438857991 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f839c0664d0 0x7f839c068990 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.465+0000 7f83b6291640 1 --2- 172.21.15.89:0/3438857991 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f83b0151910 0x7f83b014c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.465+0000 7f83b6291640 1 --2- 172.21.15.89:0/3438857991 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f83b006adb0 0x7f83b00c36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.465+0000 7f83b6291640 1 --2- 172.21.15.89:0/3438857991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f83b006a970 0x7f83b0157dc0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.465+0000 7f83b6291640 1 -- 172.21.15.89:0/3438857991 >> 172.21.15.89:0/3438857991 conn(0x7f83b00c7910 msgr2=0x7f83b006b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:36.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.465+0000 7f83b6291640 1 -- 172.21.15.89:0/3438857991 shutdown_connections 2023-10-30T12:51:36.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.465+0000 7f83b6291640 1 -- 172.21.15.89:0/3438857991 wait complete. 2023-10-30T12:51:36.485 INFO:tasks.ceph:Scrubbing osd.6 2023-10-30T12:51:36.486 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-10-30T12:51:36.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f4898246640 1 Processor -- start 2023-10-30T12:51:36.628 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f4898246640 1 -- start start 2023-10-30T12:51:36.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f4898246640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4890158070 0x7f48900c08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f4898246640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f48900c17f0 0x7f4890069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f4898246640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4890158450 con 0x7f489006a960 2023-10-30T12:51:36.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f4898246640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48900c13d0 con 0x7f4890158070 2023-10-30T12:51:36.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f4898246640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f489015e460 con 0x7f48900c17f0 2023-10-30T12:51:36.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f48957e1640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f489006a960 0x7f489015dc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36924/0 (socket says 172.21.15.89:36924) 2023-10-30T12:51:36.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f48957e1640 1 -- 172.21.15.89:0/440598545 learned_addr learned my addr 172.21.15.89:0/440598545 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:51:36.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f48967e3640 1 --2- 172.21.15.89:0/440598545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f48900c17f0 0x7f4890069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f4895fe2640 1 --2- 172.21.15.89:0/440598545 >> v2:172.21.15.192:3300/0 conn(0x7f4890158070 0x7f48900c08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f4894fe0640 1 -- 172.21.15.89:0/440598545 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2287580751 0 0) 0x7f4890158450 con 0x7f489006a960 2023-10-30T12:51:36.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f4894fe0640 1 -- 172.21.15.89:0/440598545 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f487c003530 con 0x7f489006a960 2023-10-30T12:51:36.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f4894fe0640 1 -- 172.21.15.89:0/440598545 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3756461118 0 0) 0x7f487c003530 con 0x7f489006a960 2023-10-30T12:51:36.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f4894fe0640 1 -- 172.21.15.89:0/440598545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f48900c17f0 msgr2=0x7f4890069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:36.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f4894fe0640 1 --2- 172.21.15.89:0/440598545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f48900c17f0 0x7f4890069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f4894fe0640 1 -- 172.21.15.89:0/440598545 >> v2:172.21.15.192:3300/0 conn(0x7f4890158070 msgr2=0x7f48900c08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:36.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f4894fe0640 1 --2- 172.21.15.89:0/440598545 >> v2:172.21.15.192:3300/0 conn(0x7f4890158070 0x7f48900c08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f4894fe0640 1 -- 172.21.15.89:0/440598545 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f489015e740 con 0x7f489006a960 2023-10-30T12:51:36.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f4894fe0640 1 -- 172.21.15.89:0/440598545 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f4880001680 con 0x7f489006a960 2023-10-30T12:51:36.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f4895fe2640 1 --2- 172.21.15.89:0/440598545 >> v2:172.21.15.192:3300/0 conn(0x7f4890158070 0x7f48900c08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:51:36.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f4894fe0640 1 -- 172.21.15.89:0/440598545 >> v1:172.21.15.89:6789/0 conn(0x7f489006a960 legacy=0x7f489015dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f4894fe0640 1 --2- 172.21.15.89:0/440598545 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f487c003980 0x7f487c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f4894fe0640 1 --2- 172.21.15.89:0/440598545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f487c0047d0 0x7f487c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f4894fe0640 1 --2- 172.21.15.89:0/440598545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f487c005190 0x7f487c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f4894fe0640 1 -- 172.21.15.89:0/440598545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48900c13d0 con 0x7f487c0047d0 2023-10-30T12:51:36.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f4894fe0640 1 -- 172.21.15.89:0/440598545 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489015e460 con 0x7f487c003980 2023-10-30T12:51:36.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f4894fe0640 1 -- 172.21.15.89:0/440598545 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f487c00bf00 con 0x7f487c005190 2023-10-30T12:51:36.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f48967e3640 1 --2- 172.21.15.89:0/440598545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f48900c17f0 0x7f4890069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:51:36.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f48957e1640 1 --2- 172.21.15.89:0/440598545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f487c0047d0 0x7f487c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f4895fe2640 1 --2- 172.21.15.89:0/440598545 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f487c003980 0x7f487c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f48957e1640 1 -- 172.21.15.89:0/440598545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f487c005190 msgr2=0x7f487c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:36.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.629+0000 7f48967e3640 1 --2- 172.21.15.89:0/440598545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f487c005190 0x7f487c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.633+0000 7f48957e1640 1 --2- 172.21.15.89:0/440598545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f487c005190 0x7f487c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.633+0000 7f48957e1640 1 -- 172.21.15.89:0/440598545 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f487c003980 msgr2=0x7f487c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:36.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.633+0000 7f48957e1640 1 --2- 172.21.15.89:0/440598545 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f487c003980 0x7f487c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.641 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.633+0000 7f48957e1640 1 -- 172.21.15.89:0/440598545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f487c003530 con 0x7f487c0047d0 2023-10-30T12:51:36.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.633+0000 7f4895fe2640 1 --2- 172.21.15.89:0/440598545 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f487c003980 0x7f487c003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:36.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.633+0000 7f48967e3640 1 --2- 172.21.15.89:0/440598545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f487c005190 0x7f487c00b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:51:36.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.633+0000 7f48957e1640 1 --2- 172.21.15.89:0/440598545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f487c0047d0 0x7f487c004c50 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f488000c240 tx=0x7f488001cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be708d3d24d21cd0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:36.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.633+0000 7f4894fe0640 1 -- 172.21.15.89:0/440598545 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48800058e0 con 0x7f487c0047d0 2023-10-30T12:51:36.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.633+0000 7f4894fe0640 1 -- 172.21.15.89:0/440598545 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4880005a40 con 0x7f487c0047d0 2023-10-30T12:51:36.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.633+0000 7f4894fe0640 1 -- 172.21.15.89:0/440598545 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4880005e10 con 0x7f487c0047d0 2023-10-30T12:51:36.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.633+0000 7f4898246640 1 -- 172.21.15.89:0/440598545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f487c0047d0 msgr2=0x7f487c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.633+0000 7f4898246640 1 --2- 172.21.15.89:0/440598545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f487c0047d0 0x7f487c004c50 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f488000c240 tx=0x7f488001cc00 comp rx=0 tx=0).stop 2023-10-30T12:51:36.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.633+0000 7f48967e3640 1 -- 172.21.15.89:0/440598545 reap_dead start 2023-10-30T12:51:36.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.633+0000 7f4898246640 1 -- 172.21.15.89:0/440598545 shutdown_connections 2023-10-30T12:51:36.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.633+0000 7f4898246640 1 -- 172.21.15.89:0/440598545 >> 172.21.15.89:0/440598545 conn(0x7f48900c7910 msgr2=0x7f48900c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:36.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.633+0000 7f4898246640 1 -- 172.21.15.89:0/440598545 shutdown_connections 2023-10-30T12:51:36.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.637+0000 7f4898246640 1 -- 172.21.15.89:0/440598545 wait complete. 2023-10-30T12:51:36.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.637+0000 7f4898246640 1 Processor -- start 2023-10-30T12:51:36.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.637+0000 7f4898246640 1 -- start start 2023-10-30T12:51:36.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.637+0000 7f4898246640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f487c0047d0 0x7f4890155630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.637+0000 7f4898246640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f489015dcd0 0x7f48901f7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.637+0000 7f4898246640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48901f83e0 0x7f48901f9b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.637+0000 7f4898246640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48901663e0 con 0x7f48901f83e0 2023-10-30T12:51:36.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.637+0000 7f4898246640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48901f7a10 con 0x7f489015dcd0 2023-10-30T12:51:36.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.637+0000 7f4898246640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48901f8860 con 0x7f487c0047d0 2023-10-30T12:51:36.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.637+0000 7f48967e3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48901f83e0 0x7f48901f9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.637+0000 7f48967e3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48901f83e0 0x7f48901f9b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47542/0 (socket says 172.21.15.89:47542) 2023-10-30T12:51:36.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.637+0000 7f48967e3640 1 -- 172.21.15.89:0/1206750200 learned_addr learned my addr 172.21.15.89:0/1206750200 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:36.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.637+0000 7f48957e1640 1 --2- 172.21.15.89:0/1206750200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f489015dcd0 0x7f48901f7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.637+0000 7f4895fe2640 1 --2- 172.21.15.89:0/1206750200 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f487c0047d0 0x7f4890155630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.637+0000 7f48967e3640 1 -- 172.21.15.89:0/1206750200 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f487c0047d0 msgr2=0x7f4890155630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.637+0000 7f48967e3640 1 --2- 172.21.15.89:0/1206750200 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f487c0047d0 0x7f4890155630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.637+0000 7f48967e3640 1 -- 172.21.15.89:0/1206750200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f489015dcd0 msgr2=0x7f48901f7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.637+0000 7f48967e3640 1 --2- 172.21.15.89:0/1206750200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f489015dcd0 0x7f48901f7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.637+0000 7f48967e3640 1 -- 172.21.15.89:0/1206750200 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f489015c510 con 0x7f48901f83e0 2023-10-30T12:51:36.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.637+0000 7f48967e3640 1 --2- 172.21.15.89:0/1206750200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48901f83e0 0x7f48901f9b10 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f48900c8cc0 tx=0x7f488800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:36.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.637+0000 7f48867fc640 1 -- 172.21.15.89:0/1206750200 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4888017600 con 0x7f48901f83e0 2023-10-30T12:51:36.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.637+0000 7f48867fc640 1 -- 172.21.15.89:0/1206750200 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4888011040 con 0x7f48901f83e0 2023-10-30T12:51:36.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.637+0000 7f48867fc640 1 -- 172.21.15.89:0/1206750200 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4888017980 con 0x7f48901f83e0 2023-10-30T12:51:36.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.637+0000 7f4898246640 1 -- 172.21.15.89:0/1206750200 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f489015cab0 con 0x7f48901f83e0 2023-10-30T12:51:36.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.637+0000 7f4898246640 1 -- 172.21.15.89:0/1206750200 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f489015d190 con 0x7f48901f83e0 2023-10-30T12:51:36.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.641+0000 7f4898246640 1 -- 172.21.15.89:0/1206750200 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4890070850 con 0x7f48901f83e0 2023-10-30T12:51:36.655 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.641+0000 7f48867fc640 1 -- 172.21.15.89:0/1206750200 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f48880203b0 con 0x7f48901f83e0 2023-10-30T12:51:36.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.641+0000 7f48867fc640 1 --2- 172.21.15.89:0/1206750200 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f48740664d0 0x7f4874068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.641+0000 7f48867fc640 1 -- 172.21.15.89:0/1206750200 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f4888084b60 con 0x7f48901f83e0 2023-10-30T12:51:36.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.641+0000 7f48867fc640 1 --2- 172.21.15.89:0/1206750200 >> [v2:172.21.15.192:6824/3707671287,v1:172.21.15.192:6825/3707671287] conn(0x7f487406efd0 0x7f4874071430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.656 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.641+0000 7f48867fc640 1 -- 172.21.15.89:0/1206750200 --> [v2:172.21.15.192:6824/3707671287,v1:172.21.15.192:6825/3707671287] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4874071b00 con 0x7f487406efd0 2023-10-30T12:51:36.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.641+0000 7f48867fc640 1 -- 172.21.15.89:0/1206750200 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4888084fb0 con 0x7f48901f83e0 2023-10-30T12:51:36.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.641+0000 7f4895fe2640 1 --2- 172.21.15.89:0/1206750200 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f48740664d0 0x7f4874068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.641+0000 7f48957e1640 1 --2- 172.21.15.89:0/1206750200 >> [v2:172.21.15.192:6824/3707671287,v1:172.21.15.192:6825/3707671287] conn(0x7f487406efd0 0x7f4874071430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.641+0000 7f48957e1640 1 --2- 172.21.15.89:0/1206750200 >> [v2:172.21.15.192:6824/3707671287,v1:172.21.15.192:6825/3707671287] conn(0x7f487406efd0 0x7f4874071430 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:36.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.645+0000 7f4895fe2640 1 --2- 172.21.15.89:0/1206750200 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f48740664d0 0x7f4874068990 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f488c009b40 tx=0x7f488c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:36.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.645+0000 7f48867fc640 1 -- 172.21.15.89:0/1206750200 <== osd.6 v2:172.21.15.192:6824/3707671287 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f4874071b00 con 0x7f487406efd0 2023-10-30T12:51:36.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.657+0000 7f4898246640 1 -- 172.21.15.89:0/1206750200 --> [v2:172.21.15.192:6824/3707671287,v1:172.21.15.192:6825/3707671287] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f48901f7bc0 con 0x7f487406efd0 2023-10-30T12:51:36.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.657+0000 7f48867fc640 1 -- 172.21.15.89:0/1206750200 <== osd.6 v2:172.21.15.192:6824/3707671287 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f48901f7bc0 con 0x7f487406efd0 2023-10-30T12:51:36.660 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-10-30T12:51:36.660 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T12:51:36.660 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-10-30T12:51:36.661 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.657+0000 7f4898246640 1 -- 172.21.15.89:0/1206750200 >> [v2:172.21.15.192:6824/3707671287,v1:172.21.15.192:6825/3707671287] conn(0x7f487406efd0 msgr2=0x7f4874071430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.657+0000 7f4898246640 1 --2- 172.21.15.89:0/1206750200 >> [v2:172.21.15.192:6824/3707671287,v1:172.21.15.192:6825/3707671287] conn(0x7f487406efd0 0x7f4874071430 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.657+0000 7f4898246640 1 -- 172.21.15.89:0/1206750200 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f48740664d0 msgr2=0x7f4874068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.657+0000 7f4898246640 1 --2- 172.21.15.89:0/1206750200 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f48740664d0 0x7f4874068990 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f488c009b40 tx=0x7f488c008040 comp rx=0 tx=0).stop 2023-10-30T12:51:36.662 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.657+0000 7f4898246640 1 -- 172.21.15.89:0/1206750200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48901f83e0 msgr2=0x7f48901f9b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.657+0000 7f4898246640 1 --2- 172.21.15.89:0/1206750200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48901f83e0 0x7f48901f9b10 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f48900c8cc0 tx=0x7f488800f790 comp rx=0 tx=0).stop 2023-10-30T12:51:36.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.657+0000 7f48967e3640 1 -- 172.21.15.89:0/1206750200 reap_dead start 2023-10-30T12:51:36.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.657+0000 7f4898246640 1 -- 172.21.15.89:0/1206750200 shutdown_connections 2023-10-30T12:51:36.663 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.657+0000 7f4898246640 1 -- 172.21.15.89:0/1206750200 >> 172.21.15.89:0/1206750200 conn(0x7f48900c7910 msgr2=0x7f489006b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:36.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.657+0000 7f4898246640 1 -- 172.21.15.89:0/1206750200 shutdown_connections 2023-10-30T12:51:36.664 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.657+0000 7f4898246640 1 -- 172.21.15.89:0/1206750200 wait complete. 2023-10-30T12:51:36.677 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-10-30T12:51:36.816 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.817+0000 7f34f99f5640 1 Processor -- start 2023-10-30T12:51:36.817 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.817+0000 7f34f99f5640 1 -- start start 2023-10-30T12:51:36.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.817+0000 7f34f99f5640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f34f40c90d0 0x7f34f40cef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.818 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.817+0000 7f34f99f5640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f415ce30 con 0x7f34f40c8cf0 2023-10-30T12:51:36.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.817+0000 7f34f99f5640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f4159880 con 0x7f34f40cf4d0 2023-10-30T12:51:36.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.817+0000 7f34f99f5640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34f4159ae0 con 0x7f34f40c90d0 2023-10-30T12:51:36.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.817+0000 7f34ea7fc640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f34f40c90d0 0x7f34f40cef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.819 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.817+0000 7f34f2ffd640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f34f40c8cf0 0x7f34f40c9e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36934/0 (socket says 172.21.15.89:36934) 2023-10-30T12:51:36.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.817+0000 7f34f2ffd640 1 -- 172.21.15.89:0/3240366664 learned_addr learned my addr 172.21.15.89:0/3240366664 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:51:36.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.817+0000 7f34ea7fc640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f34f40c90d0 0x7f34f40cef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47544/0 (socket says 172.21.15.89:47544) 2023-10-30T12:51:36.820 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.817+0000 7f34ea7fc640 1 -- 172.21.15.89:0/3240366664 >> v1:172.21.15.192:6789/0 conn(0x7f34f40cf4d0 legacy=0x7f34f415d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:36.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.817+0000 7f34ea7fc640 1 -- 172.21.15.89:0/3240366664 >> v1:172.21.15.89:6789/0 conn(0x7f34f40c8cf0 legacy=0x7f34f40c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.817+0000 7f34ea7fc640 1 -- 172.21.15.89:0/3240366664 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f4159dc0 con 0x7f34f40c90d0 2023-10-30T12:51:36.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.817+0000 7f34f27fc640 1 -- 172.21.15.89:0/3240366664 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 408168001 0 0) 0x7f34f415ce30 con 0x7f34f40c8cf0 2023-10-30T12:51:36.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.821+0000 7f34ea7fc640 1 --2- 172.21.15.89:0/3240366664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34f40c90d0 0x7f34f40cef90 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f34e40097c0 tx=0x7f34e401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c67e3dd87a27062d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:36.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.821+0000 7f34f27fc640 1 -- 172.21.15.89:0/3240366664 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e40ab610 con 0x7f34f40c90d0 2023-10-30T12:51:36.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.821+0000 7f34f27fc640 1 -- 172.21.15.89:0/3240366664 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34e40ab770 con 0x7f34f40c90d0 2023-10-30T12:51:36.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.821+0000 7f34f27fc640 1 -- 172.21.15.89:0/3240366664 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e40abb00 con 0x7f34f40c90d0 2023-10-30T12:51:36.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.821+0000 7f34f99f5640 1 -- 172.21.15.89:0/3240366664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34f40c90d0 msgr2=0x7f34f40cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.821+0000 7f34f99f5640 1 --2- 172.21.15.89:0/3240366664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34f40c90d0 0x7f34f40cef90 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f34e40097c0 tx=0x7f34e401c410 comp rx=0 tx=0).stop 2023-10-30T12:51:36.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.821+0000 7f34f99f5640 1 -- 172.21.15.89:0/3240366664 shutdown_connections 2023-10-30T12:51:36.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.821+0000 7f34f99f5640 1 --2- 172.21.15.89:0/3240366664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34f40c90d0 0x7f34f40cef90 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.821+0000 7f34f99f5640 1 -- 172.21.15.89:0/3240366664 >> 172.21.15.89:0/3240366664 conn(0x7f34f406be40 msgr2=0x7f34f406c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:36.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.821+0000 7f34f99f5640 1 -- 172.21.15.89:0/3240366664 shutdown_connections 2023-10-30T12:51:36.825 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.821+0000 7f34f99f5640 1 -- 172.21.15.89:0/3240366664 wait complete. 2023-10-30T12:51:36.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.825+0000 7f34f99f5640 1 Processor -- start 2023-10-30T12:51:36.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.825+0000 7f34f99f5640 1 -- start start 2023-10-30T12:51:36.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.825+0000 7f34f99f5640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f34f40c8cf0 0x7f34f415c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.828 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.825+0000 7f34f99f5640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f34f40c90d0 0x7f34f4152310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.825+0000 7f34f99f5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34f40cf4d0 0x7f34f4152850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.825+0000 7f34f99f5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f4151300 con 0x7f34f40cf4d0 2023-10-30T12:51:36.829 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.825+0000 7f34f99f5640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f4151e80 con 0x7f34f40c90d0 2023-10-30T12:51:36.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.825+0000 7f34f99f5640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f415c880 con 0x7f34f40c8cf0 2023-10-30T12:51:36.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.825+0000 7f34f37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34f40cf4d0 0x7f34f4152850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.825+0000 7f34f37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34f40cf4d0 0x7f34f4152850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47556/0 (socket says 172.21.15.89:47556) 2023-10-30T12:51:36.830 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.825+0000 7f34f37fe640 1 -- 172.21.15.89:0/294038612 learned_addr learned my addr 172.21.15.89:0/294038612 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:36.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.825+0000 7f34f2ffd640 1 --2- 172.21.15.89:0/294038612 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f34f40c8cf0 0x7f34f415c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.825+0000 7f34ea7fc640 1 --2- 172.21.15.89:0/294038612 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f34f40c90d0 0x7f34f4152310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.825+0000 7f34f37fe640 1 -- 172.21.15.89:0/294038612 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f34f40c8cf0 msgr2=0x7f34f415c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.831 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.825+0000 7f34f37fe640 1 --2- 172.21.15.89:0/294038612 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f34f40c8cf0 0x7f34f415c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.825+0000 7f34f37fe640 1 -- 172.21.15.89:0/294038612 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f34f40c90d0 msgr2=0x7f34f4152310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.825+0000 7f34f37fe640 1 --2- 172.21.15.89:0/294038612 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f34f40c90d0 0x7f34f4152310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.832 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.825+0000 7f34f37fe640 1 -- 172.21.15.89:0/294038612 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e4009420 con 0x7f34f40cf4d0 2023-10-30T12:51:36.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.825+0000 7f34f37fe640 1 --2- 172.21.15.89:0/294038612 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34f40cf4d0 0x7f34f4152850 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f34ec00c780 tx=0x7f34ec00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:36.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.825+0000 7f34f0ff9640 1 -- 172.21.15.89:0/294038612 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ec004b70 con 0x7f34f40cf4d0 2023-10-30T12:51:36.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.825+0000 7f34f0ff9640 1 -- 172.21.15.89:0/294038612 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34ec00ce90 con 0x7f34f40cf4d0 2023-10-30T12:51:36.833 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.825+0000 7f34f99f5640 1 -- 172.21.15.89:0/294038612 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f40c8370 con 0x7f34f40cf4d0 2023-10-30T12:51:36.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.825+0000 7f34f0ff9640 1 -- 172.21.15.89:0/294038612 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ec00a850 con 0x7f34f40cf4d0 2023-10-30T12:51:36.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.825+0000 7f34f99f5640 1 -- 172.21.15.89:0/294038612 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f4151830 con 0x7f34f40cf4d0 2023-10-30T12:51:36.834 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.829+0000 7f34f99f5640 1 -- 172.21.15.89:0/294038612 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f4151ff0 con 0x7f34f40cf4d0 2023-10-30T12:51:36.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.829+0000 7f34f0ff9640 1 -- 172.21.15.89:0/294038612 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f34ec00a9b0 con 0x7f34f40cf4d0 2023-10-30T12:51:36.835 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.829+0000 7f34f0ff9640 1 --2- 172.21.15.89:0/294038612 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f34d40664a0 0x7f34d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.829+0000 7f34f0ff9640 1 -- 172.21.15.89:0/294038612 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f34ec0844a0 con 0x7f34f40cf4d0 2023-10-30T12:51:36.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.829+0000 7f34f2ffd640 1 --2- 172.21.15.89:0/294038612 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f34d40664a0 0x7f34d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.829+0000 7f34f2ffd640 1 --2- 172.21.15.89:0/294038612 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f34d40664a0 0x7f34d4068960 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f34dc000f70 tx=0x7f34dc007c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:36.836 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.833+0000 7f34f0ff9640 1 -- 172.21.15.89:0/294038612 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f34ec051470 con 0x7f34f40cf4d0 2023-10-30T12:51:36.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.981+0000 7f34f99f5640 1 -- 172.21.15.89:0/294038612 --> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f34f40ca510 con 0x7f34d40664a0 2023-10-30T12:51:36.984 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.985+0000 7f34f0ff9640 1 -- 172.21.15.89:0/294038612 <== mgr.4105 v2:172.21.15.192:6832/4076276638 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f34dc007da0 con 0x7f34d40664a0 2023-10-30T12:51:36.984 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 6 to deep-scrub 2023-10-30T12:51:36.986 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.985+0000 7f34f99f5640 1 -- 172.21.15.89:0/294038612 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f34d40664a0 msgr2=0x7f34d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.985+0000 7f34f99f5640 1 --2- 172.21.15.89:0/294038612 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f34d40664a0 0x7f34d4068960 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f34dc000f70 tx=0x7f34dc007c10 comp rx=0 tx=0).stop 2023-10-30T12:51:36.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.985+0000 7f34f99f5640 1 -- 172.21.15.89:0/294038612 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34f40cf4d0 msgr2=0x7f34f4152850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.985+0000 7f34f99f5640 1 --2- 172.21.15.89:0/294038612 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34f40cf4d0 0x7f34f4152850 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f34ec00c780 tx=0x7f34ec00cc40 comp rx=0 tx=0).stop 2023-10-30T12:51:36.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.985+0000 7f34f99f5640 1 -- 172.21.15.89:0/294038612 shutdown_connections 2023-10-30T12:51:36.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.985+0000 7f34f99f5640 1 --2- 172.21.15.89:0/294038612 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f34d40664a0 0x7f34d4068960 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.985+0000 7f34f99f5640 1 --2- 172.21.15.89:0/294038612 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f34f40c90d0 0x7f34f4152310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.985+0000 7f34f99f5640 1 --2- 172.21.15.89:0/294038612 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34f40cf4d0 0x7f34f4152850 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.985+0000 7f34f99f5640 1 --2- 172.21.15.89:0/294038612 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f34f40c8cf0 0x7f34f415c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.985+0000 7f34f99f5640 1 -- 172.21.15.89:0/294038612 >> 172.21.15.89:0/294038612 conn(0x7f34f406be40 msgr2=0x7f34f4158700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:36.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.985+0000 7f34f99f5640 1 -- 172.21.15.89:0/294038612 shutdown_connections 2023-10-30T12:51:36.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:36.985+0000 7f34f99f5640 1 -- 172.21.15.89:0/294038612 wait complete. 2023-10-30T12:51:37.005 INFO:tasks.ceph:Scrubbing osd.7 2023-10-30T12:51:37.006 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-10-30T12:51:37.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.145+0000 7f046033e640 1 Processor -- start 2023-10-30T12:51:37.147 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.145+0000 7f046033e640 1 -- start start 2023-10-30T12:51:37.148 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.149+0000 7f046033e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f04580cbb00 0x7f045806c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:37.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.149+0000 7f046033e640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04580c17e0 con 0x7f04580c7b10 2023-10-30T12:51:37.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.149+0000 7f046033e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f045806bcd0 con 0x7f045806b8f0 2023-10-30T12:51:37.149 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.149+0000 7f046033e640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0458159f50 con 0x7f04580cbb00 2023-10-30T12:51:37.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.149+0000 7f045d8d9640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f04580cbb00 0x7f045806c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:37.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.149+0000 7f045e8db640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f04580c7b10 0x7f0458164100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36942/0 (socket says 172.21.15.89:36942) 2023-10-30T12:51:37.150 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.149+0000 7f045e8db640 1 -- 172.21.15.89:0/941191366 learned_addr learned my addr 172.21.15.89:0/941191366 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:51:37.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.149+0000 7f045d8d9640 1 -- 172.21.15.89:0/941191366 >> v1:172.21.15.192:6789/0 conn(0x7f045806b8f0 legacy=0x7f0458159840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:37.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.149+0000 7f045d8d9640 1 -- 172.21.15.89:0/941191366 >> v1:172.21.15.89:6789/0 conn(0x7f04580c7b10 legacy=0x7f0458164100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:37.151 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.149+0000 7f045d8d9640 1 -- 172.21.15.89:0/941191366 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04580c6c60 con 0x7f04580cbb00 2023-10-30T12:51:37.152 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.149+0000 7f045d0d8640 1 -- 172.21.15.89:0/941191366 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3726282886 0 0) 0x7f04500014a0 con 0x7f04580c7b10 2023-10-30T12:51:37.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.153+0000 7f045d8d9640 1 --2- 172.21.15.89:0/941191366 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04580cbb00 0x7f045806c600 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f04480097c0 tx=0x7f044801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbe43865b8d79f78 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:37.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.153+0000 7f045d0d8640 1 -- 172.21.15.89:0/941191366 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04480ab650 con 0x7f04580cbb00 2023-10-30T12:51:37.153 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.153+0000 7f045d0d8640 1 -- 172.21.15.89:0/941191366 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04480ab7b0 con 0x7f04580cbb00 2023-10-30T12:51:37.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.153+0000 7f045d0d8640 1 -- 172.21.15.89:0/941191366 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04480abb40 con 0x7f04580cbb00 2023-10-30T12:51:37.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.153+0000 7f046033e640 1 -- 172.21.15.89:0/941191366 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04580cbb00 msgr2=0x7f045806c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:37.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.153+0000 7f046033e640 1 --2- 172.21.15.89:0/941191366 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04580cbb00 0x7f045806c600 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f04480097c0 tx=0x7f044801c450 comp rx=0 tx=0).stop 2023-10-30T12:51:37.154 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.153+0000 7f046033e640 1 -- 172.21.15.89:0/941191366 shutdown_connections 2023-10-30T12:51:37.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.153+0000 7f046033e640 1 --2- 172.21.15.89:0/941191366 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04580cbb00 0x7f045806c600 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:37.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.153+0000 7f046033e640 1 -- 172.21.15.89:0/941191366 >> 172.21.15.89:0/941191366 conn(0x7f04580c8a20 msgr2=0x7f04580c8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:37.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.153+0000 7f046033e640 1 -- 172.21.15.89:0/941191366 shutdown_connections 2023-10-30T12:51:37.155 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.153+0000 7f046033e640 1 -- 172.21.15.89:0/941191366 wait complete. 2023-10-30T12:51:37.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.153+0000 7f046033e640 1 Processor -- start 2023-10-30T12:51:37.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.153+0000 7f046033e640 1 -- start start 2023-10-30T12:51:37.158 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.157+0000 7f046033e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f045806b8f0 0x7f0458149f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:37.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.157+0000 7f046033e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04580c7b10 0x7f045814fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:37.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.157+0000 7f046033e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04580cbb00 0x7f045814a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:37.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.157+0000 7f046033e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0458151a90 con 0x7f045806b8f0 2023-10-30T12:51:37.159 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.157+0000 7f046033e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0458152610 con 0x7f04580c7b10 2023-10-30T12:51:37.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.157+0000 7f046033e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04580adc40 con 0x7f04580cbb00 2023-10-30T12:51:37.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.157+0000 7f045e0da640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f045806b8f0 0x7f0458149f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:37.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.157+0000 7f045e0da640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f045806b8f0 0x7f0458149f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47568/0 (socket says 172.21.15.89:47568) 2023-10-30T12:51:37.160 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.157+0000 7f045e0da640 1 -- 172.21.15.89:0/1033507718 learned_addr learned my addr 172.21.15.89:0/1033507718 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:37.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.157+0000 7f045d8d9640 1 --2- 172.21.15.89:0/1033507718 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04580c7b10 0x7f045814fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:37.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.157+0000 7f045e8db640 1 --2- 172.21.15.89:0/1033507718 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04580cbb00 0x7f045814a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:37.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.157+0000 7f045e0da640 1 -- 172.21.15.89:0/1033507718 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04580cbb00 msgr2=0x7f045814a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:37.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.157+0000 7f045e0da640 1 --2- 172.21.15.89:0/1033507718 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f04580cbb00 0x7f045814a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:37.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.157+0000 7f045e0da640 1 -- 172.21.15.89:0/1033507718 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04580c7b10 msgr2=0x7f045814fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:37.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.157+0000 7f045e0da640 1 --2- 172.21.15.89:0/1033507718 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04580c7b10 0x7f045814fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:37.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.157+0000 7f045e0da640 1 -- 172.21.15.89:0/1033507718 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0448009420 con 0x7f045806b8f0 2023-10-30T12:51:37.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.157+0000 7f045e0da640 1 --2- 172.21.15.89:0/1033507718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f045806b8f0 0x7f0458149f20 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f045400c3c0 tx=0x7f045400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:37.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.157+0000 7f044effd640 1 -- 172.21.15.89:0/1033507718 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04540052d0 con 0x7f045806b8f0 2023-10-30T12:51:37.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.157+0000 7f044effd640 1 -- 172.21.15.89:0/1033507718 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0454005430 con 0x7f045806b8f0 2023-10-30T12:51:37.163 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.157+0000 7f046033e640 1 -- 172.21.15.89:0/1033507718 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0458152b00 con 0x7f045806b8f0 2023-10-30T12:51:37.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.157+0000 7f044effd640 1 -- 172.21.15.89:0/1033507718 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0454003840 con 0x7f045806b8f0 2023-10-30T12:51:37.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.157+0000 7f046033e640 1 -- 172.21.15.89:0/1033507718 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0458151440 con 0x7f045806b8f0 2023-10-30T12:51:37.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.157+0000 7f046033e640 1 -- 172.21.15.89:0/1033507718 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f04580c81c0 con 0x7f045806b8f0 2023-10-30T12:51:37.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.157+0000 7f044effd640 1 -- 172.21.15.89:0/1033507718 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f04540039a0 con 0x7f045806b8f0 2023-10-30T12:51:37.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.161+0000 7f044effd640 1 --2- 172.21.15.89:0/1033507718 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f043c0664a0 0x7f043c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:37.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.161+0000 7f044effd640 1 -- 172.21.15.89:0/1033507718 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f04540837b0 con 0x7f045806b8f0 2023-10-30T12:51:37.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.161+0000 7f044effd640 1 --2- 172.21.15.89:0/1033507718 >> [v2:172.21.15.192:6816/3111945871,v1:172.21.15.192:6817/3111945871] conn(0x7f043c06ef90 0x7f043c0713f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:37.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.161+0000 7f044effd640 1 -- 172.21.15.89:0/1033507718 --> [v2:172.21.15.192:6816/3111945871,v1:172.21.15.192:6817/3111945871] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f043c071ac0 con 0x7f043c06ef90 2023-10-30T12:51:37.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.161+0000 7f044effd640 1 -- 172.21.15.89:0/1033507718 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0454085e70 con 0x7f045806b8f0 2023-10-30T12:51:37.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.161+0000 7f045d8d9640 1 --2- 172.21.15.89:0/1033507718 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f043c0664a0 0x7f043c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:37.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.161+0000 7f045e8db640 1 --2- 172.21.15.89:0/1033507718 >> [v2:172.21.15.192:6816/3111945871,v1:172.21.15.192:6817/3111945871] conn(0x7f043c06ef90 0x7f043c0713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:37.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.161+0000 7f045d8d9640 1 --2- 172.21.15.89:0/1033507718 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f043c0664a0 0x7f043c068960 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f04480af000 tx=0x7f04480a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:37.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.161+0000 7f045e8db640 1 --2- 172.21.15.89:0/1033507718 >> [v2:172.21.15.192:6816/3111945871,v1:172.21.15.192:6817/3111945871] conn(0x7f043c06ef90 0x7f043c0713f0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:37.168 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.161+0000 7f044effd640 1 -- 172.21.15.89:0/1033507718 <== osd.7 v2:172.21.15.192:6816/3111945871 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f043c071ac0 con 0x7f043c06ef90 2023-10-30T12:51:37.175 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.173+0000 7f046033e640 1 -- 172.21.15.89:0/1033507718 --> [v2:172.21.15.192:6816/3111945871,v1:172.21.15.192:6817/3111945871] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f04580c1500 con 0x7f043c06ef90 2023-10-30T12:51:37.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.177+0000 7f044effd640 1 -- 172.21.15.89:0/1033507718 <== osd.7 v2:172.21.15.192:6816/3111945871 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f0450001b10 con 0x7f043c06ef90 2023-10-30T12:51:37.176 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-10-30T12:51:37.177 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T12:51:37.177 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-10-30T12:51:37.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.177+0000 7f046033e640 1 -- 172.21.15.89:0/1033507718 >> [v2:172.21.15.192:6816/3111945871,v1:172.21.15.192:6817/3111945871] conn(0x7f043c06ef90 msgr2=0x7f043c0713f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:37.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.177+0000 7f046033e640 1 --2- 172.21.15.89:0/1033507718 >> [v2:172.21.15.192:6816/3111945871,v1:172.21.15.192:6817/3111945871] conn(0x7f043c06ef90 0x7f043c0713f0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:37.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.177+0000 7f046033e640 1 -- 172.21.15.89:0/1033507718 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f043c0664a0 msgr2=0x7f043c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:37.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.177+0000 7f046033e640 1 --2- 172.21.15.89:0/1033507718 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f043c0664a0 0x7f043c068960 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f04480af000 tx=0x7f04480a7000 comp rx=0 tx=0).stop 2023-10-30T12:51:37.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.177+0000 7f046033e640 1 -- 172.21.15.89:0/1033507718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f045806b8f0 msgr2=0x7f0458149f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:37.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.177+0000 7f046033e640 1 --2- 172.21.15.89:0/1033507718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f045806b8f0 0x7f0458149f20 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f045400c3c0 tx=0x7f045400c880 comp rx=0 tx=0).stop 2023-10-30T12:51:37.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.177+0000 7f045e8db640 1 -- 172.21.15.89:0/1033507718 reap_dead start 2023-10-30T12:51:37.180 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.177+0000 7f046033e640 1 -- 172.21.15.89:0/1033507718 shutdown_connections 2023-10-30T12:51:37.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.177+0000 7f046033e640 1 -- 172.21.15.89:0/1033507718 >> 172.21.15.89:0/1033507718 conn(0x7f04580c8a20 msgr2=0x7f04580c8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:37.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.177+0000 7f046033e640 1 -- 172.21.15.89:0/1033507718 shutdown_connections 2023-10-30T12:51:37.181 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.177+0000 7f046033e640 1 -- 172.21.15.89:0/1033507718 wait complete. 2023-10-30T12:51:37.195 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-10-30T12:51:37.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.333+0000 7faf3b5c4640 1 Processor -- start 2023-10-30T12:51:37.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.333+0000 7faf3b5c4640 1 -- start start 2023-10-30T12:51:37.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.337+0000 7faf3b5c4640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faf340c6b40 0x7faf34161be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:37.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.337+0000 7faf3b5c4640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf340c72b0 con 0x7faf340c0ed0 2023-10-30T12:51:37.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.337+0000 7faf3b5c4640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf3406acb0 con 0x7faf34158090 2023-10-30T12:51:37.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.337+0000 7faf3b5c4640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf34162120 con 0x7faf340c6b40 2023-10-30T12:51:37.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.337+0000 7faf39b61640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7faf340c0ed0 0x7faf3406a5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36958/0 (socket says 172.21.15.89:36958) 2023-10-30T12:51:37.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.337+0000 7faf39b61640 1 -- 172.21.15.89:0/3408535031 learned_addr learned my addr 172.21.15.89:0/3408535031 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:51:37.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.337+0000 7faf38b5f640 1 --2- 172.21.15.89:0/3408535031 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faf340c6b40 0x7faf34161be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:37.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.337+0000 7faf2bfff640 1 -- 172.21.15.89:0/3408535031 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079418966 0 0) 0x7faf340c72b0 con 0x7faf340c0ed0 2023-10-30T12:51:37.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.337+0000 7faf2bfff640 1 -- 172.21.15.89:0/3408535031 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf20003530 con 0x7faf340c0ed0 2023-10-30T12:51:37.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.337+0000 7faf2bfff640 1 -- 172.21.15.89:0/3408535031 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 682370964 0 0) 0x7faf20003530 con 0x7faf340c0ed0 2023-10-30T12:51:37.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.337+0000 7faf2bfff640 1 -- 172.21.15.89:0/3408535031 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faf340c6b40 msgr2=0x7faf34161be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:37.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.337+0000 7faf2bfff640 1 --2- 172.21.15.89:0/3408535031 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faf340c6b40 0x7faf34161be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:37.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.337+0000 7faf2bfff640 1 -- 172.21.15.89:0/3408535031 >> v1:172.21.15.192:6789/0 conn(0x7faf34158090 legacy=0x7faf34069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:37.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.337+0000 7faf2bfff640 1 -- 172.21.15.89:0/3408535031 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf340c6fd0 con 0x7faf340c0ed0 2023-10-30T12:51:37.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.337+0000 7faf2bfff640 1 -- 172.21.15.89:0/3408535031 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7faf2c001a00 con 0x7faf340c0ed0 2023-10-30T12:51:37.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.337+0000 7faf2bfff640 1 -- 172.21.15.89:0/3408535031 >> v1:172.21.15.89:6789/0 conn(0x7faf340c0ed0 legacy=0x7faf3406a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:37.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.337+0000 7faf2bfff640 1 --2- 172.21.15.89:0/3408535031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf200039b0 0x7faf20003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:37.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.337+0000 7faf2bfff640 1 --2- 172.21.15.89:0/3408535031 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faf200047d0 0x7faf20004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:37.342 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.337+0000 7faf2bfff640 1 --2- 172.21.15.89:0/3408535031 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faf20005190 0x7faf2000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:37.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.337+0000 7faf2bfff640 1 -- 172.21.15.89:0/3408535031 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf34162120 con 0x7faf200039b0 2023-10-30T12:51:37.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.337+0000 7faf2bfff640 1 -- 172.21.15.89:0/3408535031 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2000bed0 con 0x7faf200047d0 2023-10-30T12:51:37.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.337+0000 7faf2bfff640 1 -- 172.21.15.89:0/3408535031 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf2000c040 con 0x7faf20005190 2023-10-30T12:51:37.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.337+0000 7faf39360640 1 --2- 172.21.15.89:0/3408535031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf200039b0 0x7faf20003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:37.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.337+0000 7faf38b5f640 1 --2- 172.21.15.89:0/3408535031 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faf200047d0 0x7faf20004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:37.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.337+0000 7faf39b61640 1 --2- 172.21.15.89:0/3408535031 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faf20005190 0x7faf2000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:37.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.337+0000 7faf39360640 1 -- 172.21.15.89:0/3408535031 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faf20005190 msgr2=0x7faf2000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:37.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.337+0000 7faf39360640 1 --2- 172.21.15.89:0/3408535031 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faf20005190 0x7faf2000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:37.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.337+0000 7faf39360640 1 -- 172.21.15.89:0/3408535031 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faf200047d0 msgr2=0x7faf20004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:37.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.337+0000 7faf39360640 1 --2- 172.21.15.89:0/3408535031 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faf200047d0 0x7faf20004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:37.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.337+0000 7faf39360640 1 -- 172.21.15.89:0/3408535031 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf20003530 con 0x7faf200039b0 2023-10-30T12:51:37.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.341+0000 7faf39360640 1 --2- 172.21.15.89:0/3408535031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf200039b0 0x7faf20003dd0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7faf30001780 tx=0x7faf3001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cf47fc89c113ade server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:37.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.341+0000 7faf2bfff640 1 -- 172.21.15.89:0/3408535031 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf300052d0 con 0x7faf200039b0 2023-10-30T12:51:37.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.341+0000 7faf2bfff640 1 -- 172.21.15.89:0/3408535031 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf300a7050 con 0x7faf200039b0 2023-10-30T12:51:37.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.341+0000 7faf2bfff640 1 -- 172.21.15.89:0/3408535031 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf30003600 con 0x7faf200039b0 2023-10-30T12:51:37.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.341+0000 7faf3b5c4640 1 -- 172.21.15.89:0/3408535031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf200039b0 msgr2=0x7faf20003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:37.350 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.341+0000 7faf3b5c4640 1 --2- 172.21.15.89:0/3408535031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf200039b0 0x7faf20003dd0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7faf30001780 tx=0x7faf3001c400 comp rx=0 tx=0).stop 2023-10-30T12:51:37.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.341+0000 7faf39b61640 1 -- 172.21.15.89:0/3408535031 reap_dead start 2023-10-30T12:51:37.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.341+0000 7faf3b5c4640 1 -- 172.21.15.89:0/3408535031 shutdown_connections 2023-10-30T12:51:37.351 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.341+0000 7faf3b5c4640 1 -- 172.21.15.89:0/3408535031 >> 172.21.15.89:0/3408535031 conn(0x7faf340c7910 msgr2=0x7faf340c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:37.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.341+0000 7faf3b5c4640 1 -- 172.21.15.89:0/3408535031 shutdown_connections 2023-10-30T12:51:37.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.341+0000 7faf3b5c4640 1 -- 172.21.15.89:0/3408535031 wait complete. 2023-10-30T12:51:37.352 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.341+0000 7faf3b5c4640 1 Processor -- start 2023-10-30T12:51:37.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.341+0000 7faf3b5c4640 1 -- start start 2023-10-30T12:51:37.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.345+0000 7faf3b5c4640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faf200039b0 0x7faf3415c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:37.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.345+0000 7faf3b5c4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf34161990 0x7faf341f4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:37.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.345+0000 7faf3b5c4640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faf3414fd20 0x7faf341501a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:37.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.345+0000 7faf3b5c4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf3415b320 con 0x7faf34161990 2023-10-30T12:51:37.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.345+0000 7faf3b5c4640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf3415b5c0 con 0x7faf3414fd20 2023-10-30T12:51:37.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.345+0000 7faf3b5c4640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf341f2d50 con 0x7faf200039b0 2023-10-30T12:51:37.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.345+0000 7faf38b5f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf34161990 0x7faf341f4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:37.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.345+0000 7faf38b5f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf34161990 0x7faf341f4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47592/0 (socket says 172.21.15.89:47592) 2023-10-30T12:51:37.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.345+0000 7faf38b5f640 1 -- 172.21.15.89:0/96629470 learned_addr learned my addr 172.21.15.89:0/96629470 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:37.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.345+0000 7faf39360640 1 --2- 172.21.15.89:0/96629470 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faf200039b0 0x7faf3415c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:37.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.345+0000 7faf39b61640 1 --2- 172.21.15.89:0/96629470 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faf3414fd20 0x7faf341501a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:37.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.345+0000 7faf38b5f640 1 -- 172.21.15.89:0/96629470 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faf200039b0 msgr2=0x7faf3415c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:37.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.345+0000 7faf38b5f640 1 --2- 172.21.15.89:0/96629470 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faf200039b0 0x7faf3415c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:37.357 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.345+0000 7faf38b5f640 1 -- 172.21.15.89:0/96629470 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faf3414fd20 msgr2=0x7faf341501a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:37.357 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.345+0000 7faf38b5f640 1 --2- 172.21.15.89:0/96629470 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faf3414fd20 0x7faf341501a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:37.357 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.345+0000 7faf38b5f640 1 -- 172.21.15.89:0/96629470 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf341f3030 con 0x7faf34161990 2023-10-30T12:51:37.357 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.345+0000 7faf38b5f640 1 --2- 172.21.15.89:0/96629470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf34161990 0x7faf341f4130 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7faf240115e0 tx=0x7faf24011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:37.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.345+0000 7faf29ffb640 1 -- 172.21.15.89:0/96629470 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2400f8f0 con 0x7faf34161990 2023-10-30T12:51:37.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.345+0000 7faf29ffb640 1 -- 172.21.15.89:0/96629470 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf2400fa50 con 0x7faf34161990 2023-10-30T12:51:37.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.345+0000 7faf29ffb640 1 -- 172.21.15.89:0/96629470 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2400fdc0 con 0x7faf34161990 2023-10-30T12:51:37.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.345+0000 7faf3b5c4640 1 -- 172.21.15.89:0/96629470 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf341f3310 con 0x7faf34161990 2023-10-30T12:51:37.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.345+0000 7faf3b5c4640 1 -- 172.21.15.89:0/96629470 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf341603d0 con 0x7faf34161990 2023-10-30T12:51:37.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.345+0000 7faf3b5c4640 1 -- 172.21.15.89:0/96629470 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf3404e170 con 0x7faf34161990 2023-10-30T12:51:37.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.345+0000 7faf29ffb640 1 -- 172.21.15.89:0/96629470 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faf24015460 con 0x7faf34161990 2023-10-30T12:51:37.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.349+0000 7faf29ffb640 1 --2- 172.21.15.89:0/96629470 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faf180664a0 0x7faf18068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:37.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.349+0000 7faf29ffb640 1 -- 172.21.15.89:0/96629470 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7faf24084650 con 0x7faf34161990 2023-10-30T12:51:37.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.349+0000 7faf39360640 1 --2- 172.21.15.89:0/96629470 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faf180664a0 0x7faf18068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:37.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.349+0000 7faf39360640 1 --2- 172.21.15.89:0/96629470 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faf180664a0 0x7faf18068960 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7faf30001750 tx=0x7faf300011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:37.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.353+0000 7faf29ffb640 1 -- 172.21.15.89:0/96629470 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7faf240516e0 con 0x7faf34161990 2023-10-30T12:51:37.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.509+0000 7faf3b5c4640 1 -- 172.21.15.89:0/96629470 --> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7faf341f2ec0 con 0x7faf180664a0 2023-10-30T12:51:37.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.509+0000 7faf29ffb640 1 -- 172.21.15.89:0/96629470 <== mgr.4105 v2:172.21.15.192:6832/4076276638 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7faf341f2ec0 con 0x7faf180664a0 2023-10-30T12:51:37.511 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 7 to deep-scrub 2023-10-30T12:51:37.512 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.513+0000 7faf3b5c4640 1 -- 172.21.15.89:0/96629470 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faf180664a0 msgr2=0x7faf18068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:37.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.513+0000 7faf3b5c4640 1 --2- 172.21.15.89:0/96629470 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faf180664a0 0x7faf18068960 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7faf30001750 tx=0x7faf300011a0 comp rx=0 tx=0).stop 2023-10-30T12:51:37.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.513+0000 7faf3b5c4640 1 -- 172.21.15.89:0/96629470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf34161990 msgr2=0x7faf341f4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:37.513 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.513+0000 7faf3b5c4640 1 --2- 172.21.15.89:0/96629470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf34161990 0x7faf341f4130 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7faf240115e0 tx=0x7faf24011aa0 comp rx=0 tx=0).stop 2023-10-30T12:51:37.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.513+0000 7faf3b5c4640 1 -- 172.21.15.89:0/96629470 shutdown_connections 2023-10-30T12:51:37.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.513+0000 7faf3b5c4640 1 --2- 172.21.15.89:0/96629470 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faf180664a0 0x7faf18068960 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:37.514 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.513+0000 7faf3b5c4640 1 --2- 172.21.15.89:0/96629470 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faf3414fd20 0x7faf341501a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:37.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.513+0000 7faf3b5c4640 1 --2- 172.21.15.89:0/96629470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf34161990 0x7faf341f4130 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:37.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.513+0000 7faf3b5c4640 1 --2- 172.21.15.89:0/96629470 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faf200039b0 0x7faf3415c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:37.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.513+0000 7faf3b5c4640 1 -- 172.21.15.89:0/96629470 >> 172.21.15.89:0/96629470 conn(0x7faf340c7910 msgr2=0x7faf34071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:37.515 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.513+0000 7faf3b5c4640 1 -- 172.21.15.89:0/96629470 shutdown_connections 2023-10-30T12:51:37.516 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.513+0000 7faf3b5c4640 1 -- 172.21.15.89:0/96629470 wait complete. 2023-10-30T12:51:37.532 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:51:37.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.669+0000 7fa229e79640 1 Processor -- start 2023-10-30T12:51:37.670 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.669+0000 7fa229e79640 1 -- start start 2023-10-30T12:51:37.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.669+0000 7fa229e79640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2240c6fe0 0x7fa224071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:37.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.669+0000 7fa229e79640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa2240cb550 0x7fa2240cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:37.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.669+0000 7fa229e79640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2240c7b90 con 0x7fa224153bf0 2023-10-30T12:51:37.673 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.669+0000 7fa229e79640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa22415d870 con 0x7fa2240cb550 2023-10-30T12:51:37.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.669+0000 7fa229e79640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2240c60e0 con 0x7fa2240c6fe0 2023-10-30T12:51:37.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.669+0000 7fa2237fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fa224153bf0 0x7fa2240c7480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:36966/0 (socket says 172.21.15.89:36966) 2023-10-30T12:51:37.674 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.669+0000 7fa2237fe640 1 -- 172.21.15.89:0/3352053486 learned_addr learned my addr 172.21.15.89:0/3352053486 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:51:37.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.673+0000 7fa222ffd640 1 --2- 172.21.15.89:0/3352053486 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2240c6fe0 0x7fa224071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:37.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.673+0000 7fa223fff640 1 --2- 172.21.15.89:0/3352053486 >> v2:172.21.15.192:3300/0 conn(0x7fa2240cb550 0x7fa2240cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:37.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.673+0000 7fa2227fc640 1 -- 172.21.15.89:0/3352053486 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1373649953 0 0) 0x7fa214001120 con 0x7fa224153bf0 2023-10-30T12:51:37.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.673+0000 7fa2227fc640 1 -- 172.21.15.89:0/3352053486 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa20c003530 con 0x7fa224153bf0 2023-10-30T12:51:37.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.673+0000 7fa2227fc640 1 -- 172.21.15.89:0/3352053486 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2396694087 0 0) 0x7fa20c003530 con 0x7fa224153bf0 2023-10-30T12:51:37.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.673+0000 7fa2227fc640 1 -- 172.21.15.89:0/3352053486 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2240c6fe0 msgr2=0x7fa224071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:37.676 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.673+0000 7fa2227fc640 1 --2- 172.21.15.89:0/3352053486 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2240c6fe0 0x7fa224071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:37.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.673+0000 7fa2227fc640 1 -- 172.21.15.89:0/3352053486 >> v2:172.21.15.192:3300/0 conn(0x7fa2240cb550 msgr2=0x7fa2240cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:37.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.673+0000 7fa2227fc640 1 --2- 172.21.15.89:0/3352053486 >> v2:172.21.15.192:3300/0 conn(0x7fa2240cb550 0x7fa2240cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:37.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.673+0000 7fa2227fc640 1 -- 172.21.15.89:0/3352053486 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2240c63a0 con 0x7fa224153bf0 2023-10-30T12:51:37.677 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.673+0000 7fa2227fc640 1 -- 172.21.15.89:0/3352053486 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7fa214001810 con 0x7fa224153bf0 2023-10-30T12:51:37.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.673+0000 7fa2227fc640 1 -- 172.21.15.89:0/3352053486 >> v1:172.21.15.89:6789/0 conn(0x7fa224153bf0 legacy=0x7fa2240c7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:37.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.673+0000 7fa2227fc640 1 --2- 172.21.15.89:0/3352053486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa20c003980 0x7fa20c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:37.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.673+0000 7fa2227fc640 1 --2- 172.21.15.89:0/3352053486 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa20c0047d0 0x7fa20c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:37.678 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.673+0000 7fa2227fc640 1 --2- 172.21.15.89:0/3352053486 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2240c6fe0 0x7fa20c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:37.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.673+0000 7fa2227fc640 1 -- 172.21.15.89:0/3352053486 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22415d870 con 0x7fa20c003980 2023-10-30T12:51:37.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.673+0000 7fa2227fc640 1 -- 172.21.15.89:0/3352053486 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2240c60e0 con 0x7fa20c0047d0 2023-10-30T12:51:37.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.673+0000 7fa2227fc640 1 -- 172.21.15.89:0/3352053486 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa20c00b990 con 0x7fa2240c6fe0 2023-10-30T12:51:37.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.673+0000 7fa2237fe640 1 --2- 172.21.15.89:0/3352053486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa20c003980 0x7fa20c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:37.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.673+0000 7fa222ffd640 1 --2- 172.21.15.89:0/3352053486 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa20c0047d0 0x7fa20c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:37.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.673+0000 7fa223fff640 1 --2- 172.21.15.89:0/3352053486 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2240c6fe0 0x7fa20c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:37.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.673+0000 7fa2237fe640 1 -- 172.21.15.89:0/3352053486 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2240c6fe0 msgr2=0x7fa20c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:37.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.673+0000 7fa2237fe640 1 --2- 172.21.15.89:0/3352053486 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa2240c6fe0 0x7fa20c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:37.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.673+0000 7fa2237fe640 1 -- 172.21.15.89:0/3352053486 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa20c0047d0 msgr2=0x7fa20c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:37.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.673+0000 7fa2237fe640 1 --2- 172.21.15.89:0/3352053486 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa20c0047d0 0x7fa20c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:37.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.673+0000 7fa2237fe640 1 -- 172.21.15.89:0/3352053486 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa20c003530 con 0x7fa20c003980 2023-10-30T12:51:37.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.677+0000 7fa2237fe640 1 --2- 172.21.15.89:0/3352053486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa20c003980 0x7fa20c003da0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fa21400c1c0 tx=0x7fa21401cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f4422a8a7704aa4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:37.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.677+0000 7fa2227fc640 1 -- 172.21.15.89:0/3352053486 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2140058e0 con 0x7fa20c003980 2023-10-30T12:51:37.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.677+0000 7fa2227fc640 1 -- 172.21.15.89:0/3352053486 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa214005a40 con 0x7fa20c003980 2023-10-30T12:51:37.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.677+0000 7fa2227fc640 1 -- 172.21.15.89:0/3352053486 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa214005dd0 con 0x7fa20c003980 2023-10-30T12:51:37.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.677+0000 7fa229e79640 1 -- 172.21.15.89:0/3352053486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa20c003980 msgr2=0x7fa20c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:37.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.677+0000 7fa229e79640 1 --2- 172.21.15.89:0/3352053486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa20c003980 0x7fa20c003da0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fa21400c1c0 tx=0x7fa21401cc10 comp rx=0 tx=0).stop 2023-10-30T12:51:37.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.677+0000 7fa223fff640 1 -- 172.21.15.89:0/3352053486 reap_dead start 2023-10-30T12:51:37.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.677+0000 7fa229e79640 1 -- 172.21.15.89:0/3352053486 shutdown_connections 2023-10-30T12:51:37.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.677+0000 7fa229e79640 1 -- 172.21.15.89:0/3352053486 >> 172.21.15.89:0/3352053486 conn(0x7fa22406c6e0 msgr2=0x7fa224069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:37.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.677+0000 7fa229e79640 1 -- 172.21.15.89:0/3352053486 shutdown_connections 2023-10-30T12:51:37.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.677+0000 7fa229e79640 1 -- 172.21.15.89:0/3352053486 wait complete. 2023-10-30T12:51:37.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.677+0000 7fa229e79640 1 Processor -- start 2023-10-30T12:51:37.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.677+0000 7fa229e79640 1 -- start start 2023-10-30T12:51:37.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.677+0000 7fa229e79640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa20c003980 0x7fa224146180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:37.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.677+0000 7fa229e79640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa22415c960 0x7fa2241f2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:37.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.677+0000 7fa229e79640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa22414ac90 0x7fa22414b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:37.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.677+0000 7fa229e79640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22415abc0 con 0x7fa22414ac90 2023-10-30T12:51:37.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.677+0000 7fa229e79640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2241466c0 con 0x7fa20c003980 2023-10-30T12:51:37.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.677+0000 7fa229e79640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2241573d0 con 0x7fa22415c960 2023-10-30T12:51:37.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.681+0000 7fa223fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa22414ac90 0x7fa22414b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:37.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.681+0000 7fa223fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa22414ac90 0x7fa22414b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47608/0 (socket says 172.21.15.89:47608) 2023-10-30T12:51:37.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.681+0000 7fa223fff640 1 -- 172.21.15.89:0/3753717835 learned_addr learned my addr 172.21.15.89:0/3753717835 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:37.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.681+0000 7fa222ffd640 1 --2- 172.21.15.89:0/3753717835 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa22415c960 0x7fa2241f2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:37.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.681+0000 7fa2237fe640 1 --2- 172.21.15.89:0/3753717835 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa20c003980 0x7fa224146180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:37.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.681+0000 7fa223fff640 1 -- 172.21.15.89:0/3753717835 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa22415c960 msgr2=0x7fa2241f2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:37.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.681+0000 7fa223fff640 1 --2- 172.21.15.89:0/3753717835 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa22415c960 0x7fa2241f2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:37.692 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.681+0000 7fa223fff640 1 -- 172.21.15.89:0/3753717835 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa20c003980 msgr2=0x7fa224146180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:37.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.681+0000 7fa223fff640 1 --2- 172.21.15.89:0/3753717835 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa20c003980 0x7fa224146180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:37.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.681+0000 7fa223fff640 1 -- 172.21.15.89:0/3753717835 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2241f2ec0 con 0x7fa22414ac90 2023-10-30T12:51:37.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.681+0000 7fa223fff640 1 --2- 172.21.15.89:0/3753717835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa22414ac90 0x7fa22414b0f0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fa2240723d0 tx=0x7fa21c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:37.693 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.681+0000 7fa20bfff640 1 -- 172.21.15.89:0/3753717835 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21c00e070 con 0x7fa22414ac90 2023-10-30T12:51:37.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.681+0000 7fa20bfff640 1 -- 172.21.15.89:0/3753717835 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa21c011e40 con 0x7fa22414ac90 2023-10-30T12:51:37.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.681+0000 7fa229e79640 1 -- 172.21.15.89:0/3753717835 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa22415ddd0 con 0x7fa22414ac90 2023-10-30T12:51:37.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.681+0000 7fa20bfff640 1 -- 172.21.15.89:0/3753717835 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21c0039a0 con 0x7fa22414ac90 2023-10-30T12:51:37.694 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.681+0000 7fa229e79640 1 -- 172.21.15.89:0/3753717835 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa22415e390 con 0x7fa22414ac90 2023-10-30T12:51:37.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.681+0000 7fa229e79640 1 -- 172.21.15.89:0/3753717835 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa22415ad00 con 0x7fa22414ac90 2023-10-30T12:51:37.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.681+0000 7fa20bfff640 1 -- 172.21.15.89:0/3753717835 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa21c003b00 con 0x7fa22414ac90 2023-10-30T12:51:37.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.685+0000 7fa20bfff640 1 --2- 172.21.15.89:0/3753717835 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa2100664d0 0x7fa210068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:37.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.685+0000 7fa20bfff640 1 -- 172.21.15.89:0/3753717835 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7fa21c0848b0 con 0x7fa22414ac90 2023-10-30T12:51:37.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.685+0000 7fa2237fe640 1 --2- 172.21.15.89:0/3753717835 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa2100664d0 0x7fa210068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:37.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.685+0000 7fa2237fe640 1 --2- 172.21.15.89:0/3753717835 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa2100664d0 0x7fa210068990 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa2140aa000 tx=0x7fa214001ba0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:37.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.685+0000 7fa20bfff640 1 -- 172.21.15.89:0/3753717835 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa22415ad00 con 0x7fa22414ac90 2023-10-30T12:51:37.841 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.841+0000 7fa229e79640 1 -- 172.21.15.89:0/3753717835 --> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa22415d490 con 0x7fa2100664d0 2023-10-30T12:51:37.862 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.865+0000 7fa20bfff640 1 -- 172.21.15.89:0/3753717835 <== mgr.4105 v2:172.21.15.192:6832/4076276638 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289877 (secure 0 0 0) 0x7fa214001650 con 0x7fa2100664d0 2023-10-30T12:51:37.865 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:51:37.867 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-10-30T12:51:37.868 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.865+0000 7fa229e79640 1 -- 172.21.15.89:0/3753717835 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa2100664d0 msgr2=0x7fa210068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:37.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.865+0000 7fa229e79640 1 --2- 172.21.15.89:0/3753717835 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa2100664d0 0x7fa210068990 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa2140aa000 tx=0x7fa214001ba0 comp rx=0 tx=0).stop 2023-10-30T12:51:37.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.865+0000 7fa229e79640 1 -- 172.21.15.89:0/3753717835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa22414ac90 msgr2=0x7fa22414b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:37.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.865+0000 7fa229e79640 1 --2- 172.21.15.89:0/3753717835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa22414ac90 0x7fa22414b0f0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fa2240723d0 tx=0x7fa21c0115a0 comp rx=0 tx=0).stop 2023-10-30T12:51:37.869 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.865+0000 7fa229e79640 1 -- 172.21.15.89:0/3753717835 shutdown_connections 2023-10-30T12:51:37.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.865+0000 7fa229e79640 1 --2- 172.21.15.89:0/3753717835 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fa2100664d0 0x7fa210068990 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:37.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.865+0000 7fa229e79640 1 --2- 172.21.15.89:0/3753717835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa22414ac90 0x7fa22414b0f0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:37.870 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.865+0000 7fa229e79640 1 --2- 172.21.15.89:0/3753717835 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa22415c960 0x7fa2241f2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:37.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.865+0000 7fa229e79640 1 --2- 172.21.15.89:0/3753717835 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa20c003980 0x7fa224146180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:37.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.865+0000 7fa229e79640 1 -- 172.21.15.89:0/3753717835 >> 172.21.15.89:0/3753717835 conn(0x7fa22406c6e0 msgr2=0x7fa224149140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:37.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.865+0000 7fa229e79640 1 -- 172.21.15.89:0/3753717835 shutdown_connections 2023-10-30T12:51:37.871 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:37.865+0000 7fa229e79640 1 -- 172.21.15.89:0/3753717835 wait complete. 2023-10-30T12:51:37.885 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":749,"stamp":"2023-10-30T12:51:37.430965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1063108251,"num_objects":1371,"num_object_clones":0,"num_object_copies":2742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1371,"num_whiteouts":0,"num_read":4367,"num_read_kb":813163,"num_write":42589,"num_write_kb":1358931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":78,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":16940,"num_omap_keys":32,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47122,"ondisk_log_size":47122,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2597756,"kb_used_data":2087420,"kb_used_omap":12,"kb_used_meta":510259,"kb_avail":752376964,"statfs":{"total":773094113280,"available":770434011136,"internally_reserved":0,"allocated":2137518080,"data_stored":2129370939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12741,"internal_metadata":522505787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1497144,"num_objects":228,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":14,"num_read_kb":31,"num_write":680,"num_write_kb":6515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":16940,"num_omap_keys":32,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004007"},"pg_stats":[{"pgid":"3.3f","version":"66'551","reported_seq":2703,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.506666+0000","last_change":"2023-10-30T12:38:17.335461+0000","last_active":"2023-10-30T12:51:30.506666+0000","last_peered":"2023-10-30T12:51:30.506666+0000","last_clean":"2023-10-30T12:51:30.506666+0000","last_became_active":"2023-10-30T12:28:50.979351+0000","last_became_peered":"2023-10-30T12:28:50.979351+0000","last_unstale":"2023-10-30T12:51:30.506666+0000","last_undegraded":"2023-10-30T12:51:30.506666+0000","last_fullsized":"2023-10-30T12:51:30.506666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":551,"log_dups_size":0,"ondisk_log_size":551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:04:55.768566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091708599999999994,"stat_sum":{"num_bytes":4321961,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":40,"num_read_kb":15,"num_write":493,"num_write_kb":5417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"66'551","reported_seq":2697,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.403605+0000","last_change":"2023-10-30T12:38:17.104663+0000","last_active":"2023-10-30T12:51:30.403605+0000","last_peered":"2023-10-30T12:51:30.403605+0000","last_clean":"2023-10-30T12:51:30.403605+0000","last_became_active":"2023-10-30T12:28:50.929133+0000","last_became_peered":"2023-10-30T12:28:50.929133+0000","last_unstale":"2023-10-30T12:51:30.403605+0000","last_undegraded":"2023-10-30T12:51:30.403605+0000","last_fullsized":"2023-10-30T12:51:30.403605+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":551,"log_dups_size":0,"ondisk_log_size":551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023149070000000002,"stat_sum":{"num_bytes":8426607,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":33,"num_read_kb":10,"num_write":489,"num_write_kb":9508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"66'550","reported_seq":2712,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.399399+0000","last_change":"2023-10-30T12:38:17.643537+0000","last_active":"2023-10-30T12:51:30.399399+0000","last_peered":"2023-10-30T12:51:30.399399+0000","last_clean":"2023-10-30T12:51:30.399399+0000","last_became_active":"2023-10-30T12:28:50.933896+0000","last_became_peered":"2023-10-30T12:28:50.933896+0000","last_unstale":"2023-10-30T12:51:30.399399+0000","last_undegraded":"2023-10-30T12:51:30.399399+0000","last_fullsized":"2023-10-30T12:51:30.399399+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":550,"log_dups_size":0,"ondisk_log_size":550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029033520000000001,"stat_sum":{"num_bytes":12697764,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":49,"num_read_kb":8209,"num_write":487,"num_write_kb":25783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"66'517","reported_seq":2661,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.473760+0000","last_change":"2023-10-30T12:38:17.117873+0000","last_active":"2023-10-30T12:51:30.473760+0000","last_peered":"2023-10-30T12:51:30.473760+0000","last_clean":"2023-10-30T12:51:30.473760+0000","last_became_active":"2023-10-30T12:28:50.967774+0000","last_became_peered":"2023-10-30T12:28:50.967774+0000","last_unstale":"2023-10-30T12:51:30.473760+0000","last_undegraded":"2023-10-30T12:51:30.473760+0000","last_fullsized":"2023-10-30T12:51:30.473760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075479680000000004,"stat_sum":{"num_bytes":4272818,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":30,"num_read_kb":13,"num_write":479,"num_write_kb":5884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"66'542","reported_seq":2700,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.506017+0000","last_change":"2023-10-30T12:38:17.333966+0000","last_active":"2023-10-30T12:51:30.506017+0000","last_peered":"2023-10-30T12:51:30.506017+0000","last_clean":"2023-10-30T12:51:30.506017+0000","last_became_active":"2023-10-30T12:28:50.973132+0000","last_became_peered":"2023-10-30T12:28:50.973132+0000","last_unstale":"2023-10-30T12:51:30.506017+0000","last_undegraded":"2023-10-30T12:51:30.506017+0000","last_fullsized":"2023-10-30T12:51:30.506017+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":542,"log_dups_size":0,"ondisk_log_size":542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:43:42.826908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0090421609999999999,"stat_sum":{"num_bytes":12692456,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":47,"num_read_kb":16,"num_write":494,"num_write_kb":14525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"66'551","reported_seq":2701,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.474891+0000","last_change":"2023-10-30T12:38:17.112051+0000","last_active":"2023-10-30T12:51:30.474891+0000","last_peered":"2023-10-30T12:51:30.474891+0000","last_clean":"2023-10-30T12:51:30.474891+0000","last_became_active":"2023-10-30T12:28:50.967910+0000","last_became_peered":"2023-10-30T12:28:50.967910+0000","last_unstale":"2023-10-30T12:51:30.474891+0000","last_undegraded":"2023-10-30T12:51:30.474891+0000","last_fullsized":"2023-10-30T12:51:30.474891+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":551,"log_dups_size":0,"ondisk_log_size":551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040413009999999997,"stat_sum":{"num_bytes":12623095,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":38,"num_read_kb":16,"num_write":503,"num_write_kb":14521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"66'555","reported_seq":2707,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.506495+0000","last_change":"2023-10-30T12:38:17.328004+0000","last_active":"2023-10-30T12:51:30.506495+0000","last_peered":"2023-10-30T12:51:30.506495+0000","last_clean":"2023-10-30T12:51:30.506495+0000","last_became_active":"2023-10-30T12:28:50.973288+0000","last_became_peered":"2023-10-30T12:28:50.973288+0000","last_unstale":"2023-10-30T12:51:30.506495+0000","last_undegraded":"2023-10-30T12:51:30.506495+0000","last_fullsized":"2023-10-30T12:51:30.506495+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":555,"log_dups_size":0,"ondisk_log_size":555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:42:52.258224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037299339999999999,"stat_sum":{"num_bytes":4302112,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":41,"num_read_kb":11,"num_write":510,"num_write_kb":7154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"66'500","reported_seq":2651,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:31.291065+0000","last_change":"2023-10-30T12:38:17.330519+0000","last_active":"2023-10-30T12:51:31.291065+0000","last_peered":"2023-10-30T12:51:31.291065+0000","last_clean":"2023-10-30T12:51:31.291065+0000","last_became_active":"2023-10-30T12:28:50.954988+0000","last_became_peered":"2023-10-30T12:28:50.954988+0000","last_unstale":"2023-10-30T12:51:31.291065+0000","last_undegraded":"2023-10-30T12:51:31.291065+0000","last_fullsized":"2023-10-30T12:51:31.291065+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":500,"log_dups_size":0,"ondisk_log_size":500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028962670000000001,"stat_sum":{"num_bytes":8424944,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":39,"num_read_kb":15,"num_write":456,"num_write_kb":10248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"66'573","reported_seq":2733,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.506849+0000","last_change":"2023-10-30T12:38:17.336435+0000","last_active":"2023-10-30T12:51:30.506849+0000","last_peered":"2023-10-30T12:51:30.506849+0000","last_clean":"2023-10-30T12:51:30.506849+0000","last_became_active":"2023-10-30T12:28:50.976564+0000","last_became_peered":"2023-10-30T12:28:50.976564+0000","last_unstale":"2023-10-30T12:51:30.506849+0000","last_undegraded":"2023-10-30T12:51:30.506849+0000","last_fullsized":"2023-10-30T12:51:30.506849+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":573,"log_dups_size":0,"ondisk_log_size":573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:02.733738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0094697280000000002,"stat_sum":{"num_bytes":16840394,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":49,"num_read_kb":4116,"num_write":513,"num_write_kb":27605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"66'546","reported_seq":2716,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.460821+0000","last_change":"2023-10-30T12:38:17.327384+0000","last_active":"2023-10-30T12:51:30.460821+0000","last_peered":"2023-10-30T12:51:30.460821+0000","last_clean":"2023-10-30T12:51:30.460821+0000","last_became_active":"2023-10-30T12:28:50.951809+0000","last_became_peered":"2023-10-30T12:28:50.951809+0000","last_unstale":"2023-10-30T12:51:30.460821+0000","last_undegraded":"2023-10-30T12:51:30.460821+0000","last_fullsized":"2023-10-30T12:51:30.460821+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":546,"log_dups_size":0,"ondisk_log_size":546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:26:18.073656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0078740100000000007,"stat_sum":{"num_bytes":16924216,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":59,"num_read_kb":29,"num_write":493,"num_write_kb":22764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"66'577","reported_seq":2741,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.404379+0000","last_change":"2023-10-30T12:38:17.323179+0000","last_active":"2023-10-30T12:51:30.404379+0000","last_peered":"2023-10-30T12:51:30.404379+0000","last_clean":"2023-10-30T12:51:30.404379+0000","last_became_active":"2023-10-30T12:29:32.958473+0000","last_became_peered":"2023-10-30T12:29:32.958473+0000","last_unstale":"2023-10-30T12:51:30.404379+0000","last_undegraded":"2023-10-30T12:51:30.404379+0000","last_fullsized":"2023-10-30T12:51:30.404379+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":577,"log_dups_size":0,"ondisk_log_size":577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:07:47.715571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045626240000000004,"stat_sum":{"num_bytes":12721502,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":46,"num_read_kb":17,"num_write":533,"num_write_kb":14913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"66'517","reported_seq":2663,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.496045+0000","last_change":"2023-10-30T12:38:17.113891+0000","last_active":"2023-10-30T12:51:30.496045+0000","last_peered":"2023-10-30T12:51:30.496045+0000","last_clean":"2023-10-30T12:51:30.496045+0000","last_became_active":"2023-10-30T12:28:50.943787+0000","last_became_peered":"2023-10-30T12:28:50.943787+0000","last_unstale":"2023-10-30T12:51:30.496045+0000","last_undegraded":"2023-10-30T12:51:30.496045+0000","last_fullsized":"2023-10-30T12:51:30.496045+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047342360000000002,"stat_sum":{"num_bytes":12634629,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":34,"num_read_kb":4115,"num_write":467,"num_write_kb":18795,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"66'600","reported_seq":2762,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.405860+0000","last_change":"2023-10-30T12:38:17.331215+0000","last_active":"2023-10-30T12:51:30.405860+0000","last_peered":"2023-10-30T12:51:30.405860+0000","last_clean":"2023-10-30T12:51:30.405860+0000","last_became_active":"2023-10-30T12:28:50.937906+0000","last_became_peered":"2023-10-30T12:28:50.937906+0000","last_unstale":"2023-10-30T12:51:30.405860+0000","last_undegraded":"2023-10-30T12:51:30.405860+0000","last_fullsized":"2023-10-30T12:51:30.405860+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":600,"log_dups_size":0,"ondisk_log_size":600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003369164,"stat_sum":{"num_bytes":4268758,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":50,"num_read_kb":8215,"num_write":547,"num_write_kb":14586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"66'535","reported_seq":2694,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:31.168634+0000","last_change":"2023-10-30T12:38:17.643256+0000","last_active":"2023-10-30T12:51:31.168634+0000","last_peered":"2023-10-30T12:51:31.168634+0000","last_clean":"2023-10-30T12:51:31.168634+0000","last_became_active":"2023-10-30T12:28:50.932753+0000","last_became_peered":"2023-10-30T12:28:50.932753+0000","last_unstale":"2023-10-30T12:51:31.168634+0000","last_undegraded":"2023-10-30T12:51:31.168634+0000","last_fullsized":"2023-10-30T12:51:31.168634+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":535,"log_dups_size":0,"ondisk_log_size":535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027063,"stat_sum":{"num_bytes":16858869,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":47,"num_read_kb":17,"num_write":493,"num_write_kb":19132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"66'563","reported_seq":2725,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.484481+0000","last_change":"2023-10-30T12:38:17.103685+0000","last_active":"2023-10-30T12:51:30.484481+0000","last_peered":"2023-10-30T12:51:30.484481+0000","last_clean":"2023-10-30T12:51:30.484481+0000","last_became_active":"2023-10-30T12:28:50.943011+0000","last_became_peered":"2023-10-30T12:28:50.943011+0000","last_unstale":"2023-10-30T12:51:30.484481+0000","last_undegraded":"2023-10-30T12:51:30.484481+0000","last_fullsized":"2023-10-30T12:51:30.484481+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":563,"log_dups_size":0,"ondisk_log_size":563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001813863,"stat_sum":{"num_bytes":21155803,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":51,"num_read_kb":4115,"num_write":508,"num_write_kb":22928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"66'566","reported_seq":2724,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.471687+0000","last_change":"2023-10-30T12:38:17.111689+0000","last_active":"2023-10-30T12:51:30.471687+0000","last_peered":"2023-10-30T12:51:30.471687+0000","last_clean":"2023-10-30T12:51:30.471687+0000","last_became_active":"2023-10-30T12:28:50.952744+0000","last_became_peered":"2023-10-30T12:28:50.952744+0000","last_unstale":"2023-10-30T12:51:30.471687+0000","last_undegraded":"2023-10-30T12:51:30.471687+0000","last_fullsized":"2023-10-30T12:51:30.471687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":566,"log_dups_size":0,"ondisk_log_size":566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034371530000000001,"stat_sum":{"num_bytes":4262714,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":46,"num_read_kb":18,"num_write":521,"num_write_kb":9441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"66'590","reported_seq":2742,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:31.170634+0000","last_change":"2023-10-30T12:38:17.120572+0000","last_active":"2023-10-30T12:51:31.170634+0000","last_peered":"2023-10-30T12:51:31.170634+0000","last_clean":"2023-10-30T12:51:31.170634+0000","last_became_active":"2023-10-30T12:28:50.974576+0000","last_became_peered":"2023-10-30T12:28:50.974576+0000","last_unstale":"2023-10-30T12:51:31.170634+0000","last_undegraded":"2023-10-30T12:51:31.170634+0000","last_fullsized":"2023-10-30T12:51:31.170634+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":590,"log_dups_size":0,"ondisk_log_size":590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080003049999999992,"stat_sum":{"num_bytes":8462359,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":40,"num_read_kb":4113,"num_write":531,"num_write_kb":16554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"66'531","reported_seq":2701,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.409262+0000","last_change":"2023-10-30T12:38:17.285320+0000","last_active":"2023-10-30T12:51:30.409262+0000","last_peered":"2023-10-30T12:51:30.409262+0000","last_clean":"2023-10-30T12:51:30.409262+0000","last_became_active":"2023-10-30T12:28:50.937384+0000","last_became_peered":"2023-10-30T12:28:50.937384+0000","last_unstale":"2023-10-30T12:51:30.409262+0000","last_undegraded":"2023-10-30T12:51:30.409262+0000","last_fullsized":"2023-10-30T12:51:30.409262+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":531,"log_dups_size":0,"ondisk_log_size":531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001162803,"stat_sum":{"num_bytes":8456287,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":58,"num_read_kb":27,"num_write":488,"num_write_kb":9441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"66'554","reported_seq":2721,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.506921+0000","last_change":"2023-10-30T12:38:17.336198+0000","last_active":"2023-10-30T12:51:30.506921+0000","last_peered":"2023-10-30T12:51:30.506921+0000","last_clean":"2023-10-30T12:51:30.506921+0000","last_became_active":"2023-10-30T12:28:50.973369+0000","last_became_peered":"2023-10-30T12:28:50.973369+0000","last_unstale":"2023-10-30T12:51:30.506921+0000","last_undegraded":"2023-10-30T12:51:30.506921+0000","last_fullsized":"2023-10-30T12:51:30.506921+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":554,"log_dups_size":0,"ondisk_log_size":554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:57:45.650842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093923919999999994,"stat_sum":{"num_bytes":12689191,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":55,"num_read_kb":4114,"num_write":505,"num_write_kb":18071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"66'577","reported_seq":2738,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.482016+0000","last_change":"2023-10-30T12:38:17.325892+0000","last_active":"2023-10-30T12:51:30.482016+0000","last_peered":"2023-10-30T12:51:30.482016+0000","last_clean":"2023-10-30T12:51:30.482016+0000","last_became_active":"2023-10-30T12:28:50.954831+0000","last_became_peered":"2023-10-30T12:28:50.954831+0000","last_unstale":"2023-10-30T12:51:30.482016+0000","last_undegraded":"2023-10-30T12:51:30.482016+0000","last_fullsized":"2023-10-30T12:51:30.482016+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":577,"log_dups_size":0,"ondisk_log_size":577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:54:29.936622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007119116,"stat_sum":{"num_bytes":21055454,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":49,"num_read_kb":4113,"num_write":524,"num_write_kb":30367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"66'532","reported_seq":2686,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.465812+0000","last_change":"2023-10-30T12:38:17.644608+0000","last_active":"2023-10-30T12:51:30.465812+0000","last_peered":"2023-10-30T12:51:30.465812+0000","last_clean":"2023-10-30T12:51:30.465812+0000","last_became_active":"2023-10-30T12:28:50.962065+0000","last_became_peered":"2023-10-30T12:28:50.962065+0000","last_unstale":"2023-10-30T12:51:30.465812+0000","last_undegraded":"2023-10-30T12:51:30.465812+0000","last_fullsized":"2023-10-30T12:51:30.465812+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":532,"log_dups_size":0,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037739459999999998,"stat_sum":{"num_bytes":25200014,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":41,"num_read_kb":21,"num_write":490,"num_write_kb":26352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"66'608","reported_seq":2767,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.506121+0000","last_change":"2023-10-30T12:38:17.118400+0000","last_active":"2023-10-30T12:51:30.506121+0000","last_peered":"2023-10-30T12:51:30.506121+0000","last_clean":"2023-10-30T12:51:30.506121+0000","last_became_active":"2023-10-30T12:28:50.967651+0000","last_became_peered":"2023-10-30T12:28:50.967651+0000","last_unstale":"2023-10-30T12:51:30.506121+0000","last_undegraded":"2023-10-30T12:51:30.506121+0000","last_fullsized":"2023-10-30T12:51:30.506121+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":608,"log_dups_size":0,"ondisk_log_size":608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081085610000000002,"stat_sum":{"num_bytes":8425872,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":46,"num_read_kb":21,"num_write":555,"num_write_kb":13596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"66'560","reported_seq":2723,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.484231+0000","last_change":"2023-10-30T12:38:17.332602+0000","last_active":"2023-10-30T12:51:30.484231+0000","last_peered":"2023-10-30T12:51:30.484231+0000","last_clean":"2023-10-30T12:51:30.484231+0000","last_became_active":"2023-10-30T12:28:50.962290+0000","last_became_peered":"2023-10-30T12:28:50.962290+0000","last_unstale":"2023-10-30T12:51:30.484231+0000","last_undegraded":"2023-10-30T12:51:30.484231+0000","last_fullsized":"2023-10-30T12:51:30.484231+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":560,"log_dups_size":0,"ondisk_log_size":560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042026670000000002,"stat_sum":{"num_bytes":195030,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":50,"num_read_kb":4117,"num_write":509,"num_write_kb":6258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"66'575","reported_seq":2728,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:31.170566+0000","last_change":"2023-10-30T12:38:17.114285+0000","last_active":"2023-10-30T12:51:31.170566+0000","last_peered":"2023-10-30T12:51:31.170566+0000","last_clean":"2023-10-30T12:51:31.170566+0000","last_became_active":"2023-10-30T12:28:50.965713+0000","last_became_peered":"2023-10-30T12:28:50.965713+0000","last_unstale":"2023-10-30T12:51:31.170566+0000","last_undegraded":"2023-10-30T12:51:31.170566+0000","last_fullsized":"2023-10-30T12:51:31.170566+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":575,"log_dups_size":0,"ondisk_log_size":575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056034209999999999,"stat_sum":{"num_bytes":12659519,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":40,"num_read_kb":16,"num_write":528,"num_write_kb":18837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"66'533","reported_seq":2677,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.455876+0000","last_change":"2023-10-30T12:38:17.330864+0000","last_active":"2023-10-30T12:51:30.455876+0000","last_peered":"2023-10-30T12:51:30.455876+0000","last_clean":"2023-10-30T12:51:30.455876+0000","last_became_active":"2023-10-30T12:28:50.951026+0000","last_became_peered":"2023-10-30T12:28:50.951026+0000","last_unstale":"2023-10-30T12:51:30.455876+0000","last_undegraded":"2023-10-30T12:51:30.455876+0000","last_fullsized":"2023-10-30T12:51:30.455876+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":533,"log_dups_size":0,"ondisk_log_size":533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031697840000000001,"stat_sum":{"num_bytes":12597918,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":31,"num_read_kb":12,"num_write":481,"num_write_kb":19629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"66'607","reported_seq":2770,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.466642+0000","last_change":"2023-10-30T12:38:17.327887+0000","last_active":"2023-10-30T12:51:30.466642+0000","last_peered":"2023-10-30T12:51:30.466642+0000","last_clean":"2023-10-30T12:51:30.466642+0000","last_became_active":"2023-10-30T12:28:50.962484+0000","last_became_peered":"2023-10-30T12:28:50.962484+0000","last_unstale":"2023-10-30T12:51:30.466642+0000","last_undegraded":"2023-10-30T12:51:30.466642+0000","last_fullsized":"2023-10-30T12:51:30.466642+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":607,"log_dups_size":0,"ondisk_log_size":607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:41:18.860625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083773500000000004,"stat_sum":{"num_bytes":12675985,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":51,"num_read_kb":23,"num_write":557,"num_write_kb":15080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"66'518","reported_seq":2673,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.385724+0000","last_change":"2023-10-30T12:38:17.104453+0000","last_active":"2023-10-30T12:51:30.385724+0000","last_peered":"2023-10-30T12:51:30.385724+0000","last_clean":"2023-10-30T12:51:30.385724+0000","last_became_active":"2023-10-30T12:28:50.933093+0000","last_became_peered":"2023-10-30T12:28:50.933093+0000","last_unstale":"2023-10-30T12:51:30.385724+0000","last_undegraded":"2023-10-30T12:51:30.385724+0000","last_fullsized":"2023-10-30T12:51:30.385724+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":518,"log_dups_size":0,"ondisk_log_size":518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022251829999999999,"stat_sum":{"num_bytes":25274091,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":42,"num_read_kb":8207,"num_write":449,"num_write_kb":27548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"66'499","reported_seq":2638,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.474390+0000","last_change":"2023-10-30T12:38:17.332060+0000","last_active":"2023-10-30T12:51:30.474390+0000","last_peered":"2023-10-30T12:51:30.474390+0000","last_clean":"2023-10-30T12:51:30.474390+0000","last_became_active":"2023-10-30T12:28:50.952531+0000","last_became_peered":"2023-10-30T12:28:50.952531+0000","last_unstale":"2023-10-30T12:51:30.474390+0000","last_undegraded":"2023-10-30T12:51:30.474390+0000","last_fullsized":"2023-10-30T12:51:30.474390+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":499,"log_dups_size":0,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039301029999999999,"stat_sum":{"num_bytes":4373272,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":26,"num_read_kb":3284,"num_write":458,"num_write_kb":14265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"66'523","reported_seq":2681,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.403579+0000","last_change":"2023-10-30T12:38:17.644463+0000","last_active":"2023-10-30T12:51:30.403579+0000","last_peered":"2023-10-30T12:51:30.403579+0000","last_clean":"2023-10-30T12:51:30.403579+0000","last_became_active":"2023-10-30T12:28:50.933124+0000","last_became_peered":"2023-10-30T12:28:50.933124+0000","last_unstale":"2023-10-30T12:51:30.403579+0000","last_undegraded":"2023-10-30T12:51:30.403579+0000","last_fullsized":"2023-10-30T12:51:30.403579+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":523,"log_dups_size":0,"ondisk_log_size":523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034752709999999998,"stat_sum":{"num_bytes":8490168,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":45,"num_read_kb":14,"num_write":469,"num_write_kb":10488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"66'594","reported_seq":2758,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.475647+0000","last_change":"2023-10-30T12:38:17.118502+0000","last_active":"2023-10-30T12:51:30.475647+0000","last_peered":"2023-10-30T12:51:30.475647+0000","last_clean":"2023-10-30T12:51:30.475647+0000","last_became_active":"2023-10-30T12:28:50.965639+0000","last_became_peered":"2023-10-30T12:28:50.965639+0000","last_unstale":"2023-10-30T12:51:30.475647+0000","last_undegraded":"2023-10-30T12:51:30.475647+0000","last_fullsized":"2023-10-30T12:51:30.475647+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":594,"log_dups_size":0,"ondisk_log_size":594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076861280000000004,"stat_sum":{"num_bytes":8432593,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":51,"num_read_kb":20,"num_write":547,"num_write_kb":11509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"66'587","reported_seq":2750,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:31.171384+0000","last_change":"2023-10-30T12:38:17.285366+0000","last_active":"2023-10-30T12:51:31.171384+0000","last_peered":"2023-10-30T12:51:31.171384+0000","last_clean":"2023-10-30T12:51:31.171384+0000","last_became_active":"2023-10-30T12:28:50.955084+0000","last_became_peered":"2023-10-30T12:28:50.955084+0000","last_unstale":"2023-10-30T12:51:31.171384+0000","last_undegraded":"2023-10-30T12:51:31.171384+0000","last_fullsized":"2023-10-30T12:51:31.171384+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":587,"log_dups_size":0,"ondisk_log_size":587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012043119999999999,"stat_sum":{"num_bytes":4257179,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":50,"num_read_kb":22,"num_write":532,"num_write_kb":9837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"66'557","reported_seq":2722,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.406641+0000","last_change":"2023-10-30T12:38:17.326196+0000","last_active":"2023-10-30T12:51:30.406641+0000","last_peered":"2023-10-30T12:51:30.406641+0000","last_clean":"2023-10-30T12:51:30.406641+0000","last_became_active":"2023-10-30T12:28:50.944775+0000","last_became_peered":"2023-10-30T12:28:50.944775+0000","last_unstale":"2023-10-30T12:51:30.406641+0000","last_undegraded":"2023-10-30T12:51:30.406641+0000","last_fullsized":"2023-10-30T12:51:30.406641+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:02:19.297081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070573429999999998,"stat_sum":{"num_bytes":12763889,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":53,"num_read_kb":21,"num_write":514,"num_write_kb":13736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"66'582","reported_seq":2734,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.479974+0000","last_change":"2023-10-30T12:38:17.109227+0000","last_active":"2023-10-30T12:51:30.479974+0000","last_peered":"2023-10-30T12:51:30.479974+0000","last_clean":"2023-10-30T12:51:30.479974+0000","last_became_active":"2023-10-30T12:28:50.940567+0000","last_became_peered":"2023-10-30T12:28:50.940567+0000","last_unstale":"2023-10-30T12:51:30.479974+0000","last_undegraded":"2023-10-30T12:51:30.479974+0000","last_fullsized":"2023-10-30T12:51:30.479974+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":582,"log_dups_size":0,"ondisk_log_size":582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028523340000000002,"stat_sum":{"num_bytes":9696728,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":17,"num_write":529,"num_write_kb":12781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"65'278","reported_seq":358,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:22.401263+0000","last_change":"2023-10-30T12:28:50.395670+0000","last_active":"2023-10-30T12:51:22.401263+0000","last_peered":"2023-10-30T12:51:22.401263+0000","last_clean":"2023-10-30T12:51:22.401263+0000","last_became_active":"2023-10-30T12:28:50.395382+0000","last_became_peered":"2023-10-30T12:28:50.395382+0000","last_unstale":"2023-10-30T12:51:22.401263+0000","last_undegraded":"2023-10-30T12:51:22.401263+0000","last_fullsized":"2023-10-30T12:51:22.401263+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":278,"log_dups_size":0,"ondisk_log_size":278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582946,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":45059,"num_write":213,"num_write_kb":12413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"66'498","reported_seq":2655,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.483116+0000","last_change":"2023-10-30T12:38:17.644896+0000","last_active":"2023-10-30T12:51:30.483116+0000","last_peered":"2023-10-30T12:51:30.483116+0000","last_clean":"2023-10-30T12:51:30.483116+0000","last_became_active":"2023-10-30T12:28:50.952613+0000","last_became_peered":"2023-10-30T12:28:50.952613+0000","last_unstale":"2023-10-30T12:51:30.483116+0000","last_undegraded":"2023-10-30T12:51:30.483116+0000","last_fullsized":"2023-10-30T12:51:30.483116+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038505420000000002,"stat_sum":{"num_bytes":21061321,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":45,"num_read_kb":18,"num_write":449,"num_write_kb":22047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"66'900","reported_seq":1182,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:31.165207+0000","last_change":"2023-10-30T12:28:50.384598+0000","last_active":"2023-10-30T12:51:31.165207+0000","last_peered":"2023-10-30T12:51:31.165207+0000","last_clean":"2023-10-30T12:51:31.165207+0000","last_became_active":"2023-10-30T12:28:50.383898+0000","last_became_peered":"2023-10-30T12:28:50.383898+0000","last_unstale":"2023-10-30T12:51:31.165207+0000","last_undegraded":"2023-10-30T12:51:31.165207+0000","last_fullsized":"2023-10-30T12:51:31.165207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:09:50.290004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21567218,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":233,"num_read_kb":49885,"num_write":841,"num_write_kb":21596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"66'452","reported_seq":746,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:35.471144+0000","last_change":"2023-10-30T12:28:50.386193+0000","last_active":"2023-10-30T12:51:35.471144+0000","last_peered":"2023-10-30T12:51:35.471144+0000","last_clean":"2023-10-30T12:51:35.471144+0000","last_became_active":"2023-10-30T12:28:50.385963+0000","last_became_peered":"2023-10-30T12:28:50.385963+0000","last_unstale":"2023-10-30T12:51:35.471144+0000","last_undegraded":"2023-10-30T12:51:35.471144+0000","last_fullsized":"2023-10-30T12:51:35.471144+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":452,"log_dups_size":0,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7993138,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":240,"num_read_kb":9095,"num_write":386,"num_write_kb":8066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"66'626","reported_seq":2794,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.404259+0000","last_change":"2023-10-30T12:38:17.330785+0000","last_active":"2023-10-30T12:51:30.404259+0000","last_peered":"2023-10-30T12:51:30.404259+0000","last_clean":"2023-10-30T12:51:30.404259+0000","last_became_active":"2023-10-30T12:28:50.935764+0000","last_became_peered":"2023-10-30T12:28:50.935764+0000","last_unstale":"2023-10-30T12:51:30.404259+0000","last_undegraded":"2023-10-30T12:51:30.404259+0000","last_fullsized":"2023-10-30T12:51:30.404259+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":626,"log_dups_size":0,"ondisk_log_size":626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031329320000000002,"stat_sum":{"num_bytes":4280685,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":56,"num_read_kb":4120,"num_write":569,"num_write_kb":13778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"65'568","reported_seq":644,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:46.985469+0000","last_change":"2023-10-30T12:28:50.392213+0000","last_active":"2023-10-30T12:50:46.985469+0000","last_peered":"2023-10-30T12:50:46.985469+0000","last_clean":"2023-10-30T12:50:46.985469+0000","last_became_active":"2023-10-30T12:28:50.391494+0000","last_became_peered":"2023-10-30T12:28:50.391494+0000","last_unstale":"2023-10-30T12:50:46.985469+0000","last_undegraded":"2023-10-30T12:50:46.985469+0000","last_fullsized":"2023-10-30T12:50:46.985469+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":36864,"num_write":519,"num_write_kb":12589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"66'592","reported_seq":2750,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.380764+0000","last_change":"2023-10-30T12:38:17.103988+0000","last_active":"2023-10-30T12:51:30.380764+0000","last_peered":"2023-10-30T12:51:30.380764+0000","last_clean":"2023-10-30T12:51:30.380764+0000","last_became_active":"2023-10-30T12:28:50.933062+0000","last_became_peered":"2023-10-30T12:28:50.933062+0000","last_unstale":"2023-10-30T12:51:30.380764+0000","last_undegraded":"2023-10-30T12:51:30.380764+0000","last_fullsized":"2023-10-30T12:51:30.380764+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":592,"log_dups_size":0,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021140489999999998,"stat_sum":{"num_bytes":33653289,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":46,"num_read_kb":4115,"num_write":549,"num_write_kb":39173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"66'128","reported_seq":201,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.642253+0000","last_change":"2023-10-30T12:28:50.394368+0000","last_active":"2023-10-30T12:51:12.642253+0000","last_peered":"2023-10-30T12:51:12.642253+0000","last_clean":"2023-10-30T12:51:12.642253+0000","last_became_active":"2023-10-30T12:28:50.393686+0000","last_became_peered":"2023-10-30T12:28:50.393686+0000","last_unstale":"2023-10-30T12:51:12.642253+0000","last_undegraded":"2023-10-30T12:51:12.642253+0000","last_fullsized":"2023-10-30T12:51:12.642253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:43:17.726462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":73,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"66'567","reported_seq":2730,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.467047+0000","last_change":"2023-10-30T12:38:17.119770+0000","last_active":"2023-10-30T12:51:30.467047+0000","last_peered":"2023-10-30T12:51:30.467047+0000","last_clean":"2023-10-30T12:51:30.467047+0000","last_became_active":"2023-10-30T12:28:50.963105+0000","last_became_peered":"2023-10-30T12:28:50.963105+0000","last_unstale":"2023-10-30T12:51:30.467047+0000","last_undegraded":"2023-10-30T12:51:30.467047+0000","last_fullsized":"2023-10-30T12:51:30.467047+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":567,"log_dups_size":0,"ondisk_log_size":567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080066809999999999,"stat_sum":{"num_bytes":16920090,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":51,"num_read_kb":18,"num_write":515,"num_write_kb":18479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"66'281","reported_seq":355,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:24.397165+0000","last_change":"2023-10-30T12:28:50.397679+0000","last_active":"2023-10-30T12:51:24.397165+0000","last_peered":"2023-10-30T12:51:24.397165+0000","last_clean":"2023-10-30T12:51:24.397165+0000","last_became_active":"2023-10-30T12:28:50.397041+0000","last_became_peered":"2023-10-30T12:28:50.397041+0000","last_unstale":"2023-10-30T12:51:24.397165+0000","last_undegraded":"2023-10-30T12:51:24.397165+0000","last_fullsized":"2023-10-30T12:51:24.397165+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":16396,"num_write":240,"num_write_kb":8312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"66'548","reported_seq":2702,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:31.170135+0000","last_change":"2023-10-30T12:38:17.644799+0000","last_active":"2023-10-30T12:51:31.170135+0000","last_peered":"2023-10-30T12:51:31.170135+0000","last_clean":"2023-10-30T12:51:31.170135+0000","last_became_active":"2023-10-30T12:28:50.966569+0000","last_became_peered":"2023-10-30T12:28:50.966569+0000","last_unstale":"2023-10-30T12:51:31.170135+0000","last_undegraded":"2023-10-30T12:51:31.170135+0000","last_fullsized":"2023-10-30T12:51:31.170135+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":548,"log_dups_size":0,"ondisk_log_size":548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037972850000000001,"stat_sum":{"num_bytes":16988939,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":42,"num_read_kb":19,"num_write":508,"num_write_kb":18140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"66'371","reported_seq":451,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:29.590427+0000","last_change":"2023-10-30T12:28:50.391527+0000","last_active":"2023-10-30T12:51:29.590427+0000","last_peered":"2023-10-30T12:51:29.590427+0000","last_clean":"2023-10-30T12:51:29.590427+0000","last_became_active":"2023-10-30T12:28:50.391260+0000","last_became_peered":"2023-10-30T12:28:50.391260+0000","last_unstale":"2023-10-30T12:51:29.590427+0000","last_undegraded":"2023-10-30T12:51:29.590427+0000","last_fullsized":"2023-10-30T12:51:29.590427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":22,"num_read_kb":7467,"num_write":319,"num_write_kb":4401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"66'516","reported_seq":2675,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.503653+0000","last_change":"2023-10-30T12:38:17.114492+0000","last_active":"2023-10-30T12:51:30.503653+0000","last_peered":"2023-10-30T12:51:30.503653+0000","last_clean":"2023-10-30T12:51:30.503653+0000","last_became_active":"2023-10-30T12:28:50.974843+0000","last_became_peered":"2023-10-30T12:28:50.974843+0000","last_unstale":"2023-10-30T12:51:30.503653+0000","last_undegraded":"2023-10-30T12:51:30.503653+0000","last_fullsized":"2023-10-30T12:51:30.503653+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":516,"log_dups_size":0,"ondisk_log_size":516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005161372,"stat_sum":{"num_bytes":4244440,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":47,"num_read_kb":17,"num_write":458,"num_write_kb":9849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"66'350","reported_seq":440,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:18.809164+0000","last_change":"2023-10-30T12:28:50.374838+0000","last_active":"2023-10-30T12:51:18.809164+0000","last_peered":"2023-10-30T12:51:18.809164+0000","last_clean":"2023-10-30T12:51:18.809164+0000","last_became_active":"2023-10-30T12:28:50.373581+0000","last_became_peered":"2023-10-30T12:28:50.373581+0000","last_unstale":"2023-10-30T12:51:18.809164+0000","last_undegraded":"2023-10-30T12:51:18.809164+0000","last_fullsized":"2023-10-30T12:51:18.809164+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":350,"log_dups_size":0,"ondisk_log_size":350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:14:51.506745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":43,"num_read_kb":32803,"num_write":292,"num_write_kb":12543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"66'499","reported_seq":2648,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.507173+0000","last_change":"2023-10-30T12:38:17.328512+0000","last_active":"2023-10-30T12:51:30.507173+0000","last_peered":"2023-10-30T12:51:30.507173+0000","last_clean":"2023-10-30T12:51:30.507173+0000","last_became_active":"2023-10-30T12:28:50.973269+0000","last_became_peered":"2023-10-30T12:28:50.973269+0000","last_unstale":"2023-10-30T12:51:30.507173+0000","last_undegraded":"2023-10-30T12:51:30.507173+0000","last_fullsized":"2023-10-30T12:51:30.507173+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":499,"log_dups_size":0,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:26:11.417670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004001498,"stat_sum":{"num_bytes":12707497,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":38,"num_read_kb":11,"num_write":446,"num_write_kb":13522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"66'436","reported_seq":528,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:18.810393+0000","last_change":"2023-10-30T12:28:50.366590+0000","last_active":"2023-10-30T12:51:18.810393+0000","last_peered":"2023-10-30T12:51:18.810393+0000","last_clean":"2023-10-30T12:51:18.810393+0000","last_became_active":"2023-10-30T12:28:50.366269+0000","last_became_peered":"2023-10-30T12:28:50.366269+0000","last_unstale":"2023-10-30T12:51:18.810393+0000","last_undegraded":"2023-10-30T12:51:18.810393+0000","last_fullsized":"2023-10-30T12:51:18.810393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":436,"log_dups_size":0,"ondisk_log_size":436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":52,"num_read_kb":36923,"num_write":386,"num_write_kb":8499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"66'574","reported_seq":2738,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.466545+0000","last_change":"2023-10-30T12:38:17.644043+0000","last_active":"2023-10-30T12:51:30.466545+0000","last_peered":"2023-10-30T12:51:30.466545+0000","last_clean":"2023-10-30T12:51:30.466545+0000","last_became_active":"2023-10-30T12:28:50.961805+0000","last_became_peered":"2023-10-30T12:28:50.961805+0000","last_unstale":"2023-10-30T12:51:30.466545+0000","last_undegraded":"2023-10-30T12:51:30.466545+0000","last_fullsized":"2023-10-30T12:51:30.466545+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":574,"log_dups_size":0,"ondisk_log_size":574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032264350000000002,"stat_sum":{"num_bytes":4388445,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":52,"num_read_kb":4114,"num_write":506,"num_write_kb":14199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"66'258","reported_seq":334,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:24.396418+0000","last_change":"2023-10-30T12:28:50.397244+0000","last_active":"2023-10-30T12:51:24.396418+0000","last_peered":"2023-10-30T12:51:24.396418+0000","last_clean":"2023-10-30T12:51:24.396418+0000","last_became_active":"2023-10-30T12:28:50.396727+0000","last_became_peered":"2023-10-30T12:28:50.396727+0000","last_unstale":"2023-10-30T12:51:24.396418+0000","last_undegraded":"2023-10-30T12:51:24.396418+0000","last_fullsized":"2023-10-30T12:51:24.396418+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":16164,"num_write":201,"num_write_kb":8311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"66'560","reported_seq":2727,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.490093+0000","last_change":"2023-10-30T12:38:17.284992+0000","last_active":"2023-10-30T12:51:30.490093+0000","last_peered":"2023-10-30T12:51:30.490093+0000","last_clean":"2023-10-30T12:51:30.490093+0000","last_became_active":"2023-10-30T12:29:32.960608+0000","last_became_peered":"2023-10-30T12:29:32.960608+0000","last_unstale":"2023-10-30T12:51:30.490093+0000","last_undegraded":"2023-10-30T12:51:30.490093+0000","last_fullsized":"2023-10-30T12:51:30.490093+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":560,"log_dups_size":0,"ondisk_log_size":560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012934999999999999,"stat_sum":{"num_bytes":4430082,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":59,"num_read_kb":33,"num_write":506,"num_write_kb":6384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"65'677","reported_seq":758,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-10-30T12:50:01.976866+0000","last_change":"2023-10-30T12:28:50.383880+0000","last_active":"2023-10-30T12:50:01.976866+0000","last_peered":"2023-10-30T12:50:01.976866+0000","last_clean":"2023-10-30T12:50:01.976866+0000","last_became_active":"2023-10-30T12:28:50.383570+0000","last_became_peered":"2023-10-30T12:28:50.383570+0000","last_unstale":"2023-10-30T12:50:01.976866+0000","last_undegraded":"2023-10-30T12:50:01.976866+0000","last_fullsized":"2023-10-30T12:50:01.976866+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":677,"log_dups_size":0,"ondisk_log_size":677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:19:42.047520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":37306,"num_write":619,"num_write_kb":16724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"66'541","reported_seq":2685,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.407885+0000","last_change":"2023-10-30T12:38:17.322231+0000","last_active":"2023-10-30T12:51:30.407885+0000","last_peered":"2023-10-30T12:51:30.407885+0000","last_clean":"2023-10-30T12:51:30.407885+0000","last_became_active":"2023-10-30T12:28:50.949485+0000","last_became_peered":"2023-10-30T12:28:50.949485+0000","last_unstale":"2023-10-30T12:51:30.407885+0000","last_undegraded":"2023-10-30T12:51:30.407885+0000","last_fullsized":"2023-10-30T12:51:30.407885+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":541,"log_dups_size":0,"ondisk_log_size":541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:30:36.516655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040897299999999998,"stat_sum":{"num_bytes":12622957,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":477,"num_write_kb":14339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"66'440","reported_seq":547,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:31.173133+0000","last_change":"2023-10-30T12:28:50.362979+0000","last_active":"2023-10-30T12:51:31.173133+0000","last_peered":"2023-10-30T12:51:31.173133+0000","last_clean":"2023-10-30T12:51:31.173133+0000","last_became_active":"2023-10-30T12:28:50.362624+0000","last_became_peered":"2023-10-30T12:28:50.362624+0000","last_unstale":"2023-10-30T12:51:31.173133+0000","last_undegraded":"2023-10-30T12:51:31.173133+0000","last_fullsized":"2023-10-30T12:51:31.173133+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":440,"log_dups_size":0,"ondisk_log_size":440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583024,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":93,"num_read_kb":24662,"num_write":377,"num_write_kb":12557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"66'582","reported_seq":2743,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.506557+0000","last_change":"2023-10-30T12:38:17.334068+0000","last_active":"2023-10-30T12:51:30.506557+0000","last_peered":"2023-10-30T12:51:30.506557+0000","last_clean":"2023-10-30T12:51:30.506557+0000","last_became_active":"2023-10-30T12:28:50.979344+0000","last_became_peered":"2023-10-30T12:28:50.979344+0000","last_unstale":"2023-10-30T12:51:30.506557+0000","last_undegraded":"2023-10-30T12:51:30.506557+0000","last_fullsized":"2023-10-30T12:51:30.506557+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":582,"log_dups_size":0,"ondisk_log_size":582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:32:04.956803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0092295529999999997,"stat_sum":{"num_bytes":12647197,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":50,"num_read_kb":8210,"num_write":532,"num_write_kb":23192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"66'248","reported_seq":334,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:37.268010+0000","last_change":"2023-10-30T12:51:37.268108+0000","last_active":"2023-10-30T12:51:37.268010+0000","last_peered":"2023-10-30T12:51:37.268010+0000","last_clean":"2023-10-30T12:51:37.268010+0000","last_became_active":"2023-10-30T12:28:50.381125+0000","last_became_peered":"2023-10-30T12:28:50.381125+0000","last_unstale":"2023-10-30T12:51:37.268010+0000","last_undegraded":"2023-10-30T12:51:37.268010+0000","last_fullsized":"2023-10-30T12:51:37.268010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'248","last_scrub_stamp":"2023-10-30T12:51:37.267826+0000","last_deep_scrub":"66'248","last_deep_scrub_stamp":"2023-10-30T12:51:37.267826+0000","last_clean_scrub_stamp":"2023-10-30T12:51:37.267826+0000","objects_scrubbed":3,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:30:50.232542+0000","scrub_duration":0.054105548000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":12378,"num_write":396,"num_write_kb":4652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"66'531","reported_seq":2679,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.505964+0000","last_change":"2023-10-30T12:38:17.334809+0000","last_active":"2023-10-30T12:51:30.505964+0000","last_peered":"2023-10-30T12:51:30.505964+0000","last_clean":"2023-10-30T12:51:30.505964+0000","last_became_active":"2023-10-30T12:28:50.977585+0000","last_became_peered":"2023-10-30T12:28:50.977585+0000","last_unstale":"2023-10-30T12:51:30.505964+0000","last_undegraded":"2023-10-30T12:51:30.505964+0000","last_fullsized":"2023-10-30T12:51:30.505964+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":531,"log_dups_size":0,"ondisk_log_size":531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:33:51.736957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089211889999999995,"stat_sum":{"num_bytes":12933034,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":36,"num_read_kb":8205,"num_write":483,"num_write_kb":18997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"65'355","reported_seq":433,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:22.420472+0000","last_change":"2023-10-30T12:28:50.367258+0000","last_active":"2023-10-30T12:51:22.420472+0000","last_peered":"2023-10-30T12:51:22.420472+0000","last_clean":"2023-10-30T12:51:22.420472+0000","last_became_active":"2023-10-30T12:28:50.366974+0000","last_became_peered":"2023-10-30T12:28:50.366974+0000","last_unstale":"2023-10-30T12:51:22.420472+0000","last_undegraded":"2023-10-30T12:51:22.420472+0000","last_fullsized":"2023-10-30T12:51:22.420472+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":15324,"num_write":292,"num_write_kb":8361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"66'534","reported_seq":2696,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.505952+0000","last_change":"2023-10-30T12:38:17.337139+0000","last_active":"2023-10-30T12:51:30.505952+0000","last_peered":"2023-10-30T12:51:30.505952+0000","last_clean":"2023-10-30T12:51:30.505952+0000","last_became_active":"2023-10-30T12:28:50.981193+0000","last_became_peered":"2023-10-30T12:28:50.981193+0000","last_unstale":"2023-10-30T12:51:30.505952+0000","last_undegraded":"2023-10-30T12:51:30.505952+0000","last_fullsized":"2023-10-30T12:51:30.505952+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":534,"log_dups_size":0,"ondisk_log_size":534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:32:10.843997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096579700000000001,"stat_sum":{"num_bytes":12674006,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":50,"num_read_kb":4114,"num_write":492,"num_write_kb":18051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"66'96","reported_seq":170,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:23.570444+0000","last_change":"2023-10-30T12:28:50.368011+0000","last_active":"2023-10-30T12:51:23.570444+0000","last_peered":"2023-10-30T12:51:23.570444+0000","last_clean":"2023-10-30T12:51:23.570444+0000","last_became_active":"2023-10-30T12:28:50.367730+0000","last_became_peered":"2023-10-30T12:28:50.367730+0000","last_unstale":"2023-10-30T12:51:23.570444+0000","last_undegraded":"2023-10-30T12:51:23.570444+0000","last_fullsized":"2023-10-30T12:51:23.570444+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":12292,"num_write":45,"num_write_kb":4186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"66'594","reported_seq":2763,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.473832+0000","last_change":"2023-10-30T12:38:17.119150+0000","last_active":"2023-10-30T12:51:30.473832+0000","last_peered":"2023-10-30T12:51:30.473832+0000","last_clean":"2023-10-30T12:51:30.473832+0000","last_became_active":"2023-10-30T12:28:50.966246+0000","last_became_peered":"2023-10-30T12:28:50.966246+0000","last_unstale":"2023-10-30T12:51:30.473832+0000","last_undegraded":"2023-10-30T12:51:30.473832+0000","last_fullsized":"2023-10-30T12:51:30.473832+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":594,"log_dups_size":0,"ondisk_log_size":594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007703601,"stat_sum":{"num_bytes":8520713,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":56,"num_read_kb":18,"num_write":541,"num_write_kb":11343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"66'83","reported_seq":160,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:22.420454+0000","last_change":"2023-10-30T12:28:50.394060+0000","last_active":"2023-10-30T12:51:22.420454+0000","last_peered":"2023-10-30T12:51:22.420454+0000","last_clean":"2023-10-30T12:51:22.420454+0000","last_became_active":"2023-10-30T12:28:50.393642+0000","last_became_peered":"2023-10-30T12:28:50.393642+0000","last_unstale":"2023-10-30T12:51:22.420454+0000","last_undegraded":"2023-10-30T12:51:22.420454+0000","last_fullsized":"2023-10-30T12:51:22.420454+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":15,"num_write_kb":101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"66'568","reported_seq":2724,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.507096+0000","last_change":"2023-10-30T12:38:17.335584+0000","last_active":"2023-10-30T12:51:30.507096+0000","last_peered":"2023-10-30T12:51:30.507096+0000","last_clean":"2023-10-30T12:51:30.507096+0000","last_became_active":"2023-10-30T12:28:50.976343+0000","last_became_peered":"2023-10-30T12:28:50.976343+0000","last_unstale":"2023-10-30T12:51:30.507096+0000","last_undegraded":"2023-10-30T12:51:30.507096+0000","last_fullsized":"2023-10-30T12:51:30.507096+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:03:37.466766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091875959999999993,"stat_sum":{"num_bytes":17078090,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":44,"num_read_kb":14,"num_write":517,"num_write_kb":22829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":220,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:41.797344+0000","last_change":"2023-10-30T12:28:41.203990+0000","last_active":"2023-10-30T12:50:41.797344+0000","last_peered":"2023-10-30T12:50:41.797344+0000","last_clean":"2023-10-30T12:50:41.797344+0000","last_became_active":"2023-10-30T12:28:41.201796+0000","last_became_peered":"2023-10-30T12:28:41.201796+0000","last_unstale":"2023-10-30T12:50:41.797344+0000","last_undegraded":"2023-10-30T12:50:41.797344+0000","last_fullsized":"2023-10-30T12:50:41.797344+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:39.218257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:39.218257+0000","last_clean_scrub_stamp":"2023-10-30T12:28:39.218257+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:21:00.605462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"66'193","reported_seq":274,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:24.394778+0000","last_change":"2023-10-30T12:28:50.372651+0000","last_active":"2023-10-30T12:51:24.394778+0000","last_peered":"2023-10-30T12:51:24.394778+0000","last_clean":"2023-10-30T12:51:24.394778+0000","last_became_active":"2023-10-30T12:28:50.367593+0000","last_became_peered":"2023-10-30T12:28:50.367593+0000","last_unstale":"2023-10-30T12:51:24.394778+0000","last_undegraded":"2023-10-30T12:51:24.394778+0000","last_fullsized":"2023-10-30T12:51:24.394778+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194338,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":24,"num_read_kb":12326,"num_write":144,"num_write_kb":4223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"66'525","reported_seq":2671,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.378780+0000","last_change":"2023-10-30T12:38:17.285044+0000","last_active":"2023-10-30T12:51:30.378780+0000","last_peered":"2023-10-30T12:51:30.378780+0000","last_clean":"2023-10-30T12:51:30.378780+0000","last_became_active":"2023-10-30T12:29:32.997301+0000","last_became_peered":"2023-10-30T12:29:32.997301+0000","last_unstale":"2023-10-30T12:51:30.378780+0000","last_undegraded":"2023-10-30T12:51:30.378780+0000","last_fullsized":"2023-10-30T12:51:30.378780+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":525,"log_dups_size":0,"ondisk_log_size":525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001366412,"stat_sum":{"num_bytes":4237965,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":36,"num_read_kb":11,"num_write":481,"num_write_kb":13486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"66'499","reported_seq":586,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:18.810050+0000","last_change":"2023-10-30T12:28:50.392461+0000","last_active":"2023-10-30T12:51:18.810050+0000","last_peered":"2023-10-30T12:51:18.810050+0000","last_clean":"2023-10-30T12:51:18.810050+0000","last_became_active":"2023-10-30T12:28:50.391824+0000","last_became_peered":"2023-10-30T12:28:50.391824+0000","last_unstale":"2023-10-30T12:51:18.810050+0000","last_undegraded":"2023-10-30T12:51:18.810050+0000","last_fullsized":"2023-10-30T12:51:18.810050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":499,"log_dups_size":0,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:56:10.373046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":32,"num_read_kb":50322,"num_write":431,"num_write_kb":16616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"66'532","reported_seq":2674,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:31.207540+0000","last_change":"2023-10-30T12:38:17.643111+0000","last_active":"2023-10-30T12:51:31.207540+0000","last_peered":"2023-10-30T12:51:31.207540+0000","last_clean":"2023-10-30T12:51:31.207540+0000","last_became_active":"2023-10-30T12:28:50.952625+0000","last_became_peered":"2023-10-30T12:28:50.952625+0000","last_unstale":"2023-10-30T12:51:31.207540+0000","last_undegraded":"2023-10-30T12:51:31.207540+0000","last_fullsized":"2023-10-30T12:51:31.207540+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":532,"log_dups_size":0,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026695970000000001,"stat_sum":{"num_bytes":16953646,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":29,"num_read_kb":10,"num_write":468,"num_write_kb":20237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"66'600","reported_seq":914,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:35.467038+0000","last_change":"2023-10-30T12:51:35.405246+0000","last_active":"2023-10-30T12:51:35.467038+0000","last_peered":"2023-10-30T12:51:35.467038+0000","last_clean":"2023-10-30T12:51:35.467038+0000","last_became_active":"2023-10-30T12:28:50.367230+0000","last_became_peered":"2023-10-30T12:28:50.367230+0000","last_unstale":"2023-10-30T12:51:35.467038+0000","last_undegraded":"2023-10-30T12:51:35.467038+0000","last_fullsized":"2023-10-30T12:51:35.467038+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'598","last_scrub_stamp":"2023-10-30T12:51:35.405133+0000","last_deep_scrub":"66'598","last_deep_scrub_stamp":"2023-10-30T12:51:35.405133+0000","last_clean_scrub_stamp":"2023-10-30T12:51:35.405133+0000","objects_scrubbed":7,"log_size":600,"log_dups_size":0,"ondisk_log_size":600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:34:01.194417+0000","scrub_duration":0.035040017999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12740221,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":251,"num_read_kb":13023,"num_write":537,"num_write_kb":12789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9828,"num_omap_keys":18,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"66'551","reported_seq":2712,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.382287+0000","last_change":"2023-10-30T12:38:17.104045+0000","last_active":"2023-10-30T12:51:30.382287+0000","last_peered":"2023-10-30T12:51:30.382287+0000","last_clean":"2023-10-30T12:51:30.382287+0000","last_became_active":"2023-10-30T12:28:50.928608+0000","last_became_peered":"2023-10-30T12:28:50.928608+0000","last_unstale":"2023-10-30T12:51:30.382287+0000","last_undegraded":"2023-10-30T12:51:30.382287+0000","last_fullsized":"2023-10-30T12:51:30.382287+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":551,"log_dups_size":0,"ondisk_log_size":551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002074275,"stat_sum":{"num_bytes":8441122,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":47,"num_read_kb":21,"num_write":501,"num_write_kb":11799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"66'493","reported_seq":573,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.653410+0000","last_change":"2023-10-30T12:28:50.391240+0000","last_active":"2023-10-30T12:51:12.653410+0000","last_peered":"2023-10-30T12:51:12.653410+0000","last_clean":"2023-10-30T12:51:12.653410+0000","last_became_active":"2023-10-30T12:28:50.390864+0000","last_became_peered":"2023-10-30T12:28:50.390864+0000","last_unstale":"2023-10-30T12:51:12.653410+0000","last_undegraded":"2023-10-30T12:51:12.653410+0000","last_fullsized":"2023-10-30T12:51:12.653410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":493,"log_dups_size":0,"ondisk_log_size":493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":45183,"num_write":427,"num_write_kb":16635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"66'524","reported_seq":2672,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.489071+0000","last_change":"2023-10-30T12:38:17.111185+0000","last_active":"2023-10-30T12:51:30.489071+0000","last_peered":"2023-10-30T12:51:30.489071+0000","last_clean":"2023-10-30T12:51:30.489071+0000","last_became_active":"2023-10-30T12:28:50.949122+0000","last_became_peered":"2023-10-30T12:28:50.949122+0000","last_unstale":"2023-10-30T12:51:30.489071+0000","last_undegraded":"2023-10-30T12:51:30.489071+0000","last_fullsized":"2023-10-30T12:51:30.489071+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":524,"log_dups_size":0,"ondisk_log_size":524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004393348,"stat_sum":{"num_bytes":12684737,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":35,"num_read_kb":65,"num_write":464,"num_write_kb":14146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"66'267","reported_seq":345,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:36.014809+0000","last_change":"2023-10-30T12:51:36.014856+0000","last_active":"2023-10-30T12:51:36.014809+0000","last_peered":"2023-10-30T12:51:36.014809+0000","last_clean":"2023-10-30T12:51:36.014809+0000","last_became_active":"2023-10-30T12:28:50.358310+0000","last_became_peered":"2023-10-30T12:28:50.358310+0000","last_unstale":"2023-10-30T12:51:36.014809+0000","last_undegraded":"2023-10-30T12:51:36.014809+0000","last_fullsized":"2023-10-30T12:51:36.014809+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'267","last_scrub_stamp":"2023-10-30T12:51:36.014733+0000","last_deep_scrub":"66'267","last_deep_scrub_stamp":"2023-10-30T12:51:36.014733+0000","last_clean_scrub_stamp":"2023-10-30T12:51:36.014733+0000","objects_scrubbed":2,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:22:01.486529+0000","scrub_duration":0.015176472999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":5758,"num_write":222,"num_write_kb":4242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7112,"num_omap_keys":14,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"66'540","reported_seq":2686,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.471401+0000","last_change":"2023-10-30T12:38:17.121187+0000","last_active":"2023-10-30T12:51:30.471401+0000","last_peered":"2023-10-30T12:51:30.471401+0000","last_clean":"2023-10-30T12:51:30.471401+0000","last_became_active":"2023-10-30T12:28:50.965626+0000","last_became_peered":"2023-10-30T12:28:50.965626+0000","last_unstale":"2023-10-30T12:51:30.471401+0000","last_undegraded":"2023-10-30T12:51:30.471401+0000","last_fullsized":"2023-10-30T12:51:30.471401+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":540,"log_dups_size":0,"ondisk_log_size":540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0079099229999999993,"stat_sum":{"num_bytes":29382851,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":33,"num_read_kb":4109,"num_write":477,"num_write_kb":37019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"66'399","reported_seq":470,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:47.791499+0000","last_change":"2023-10-30T12:28:50.395645+0000","last_active":"2023-10-30T12:50:47.791499+0000","last_peered":"2023-10-30T12:50:47.791499+0000","last_clean":"2023-10-30T12:50:47.791499+0000","last_became_active":"2023-10-30T12:28:50.395090+0000","last_became_peered":"2023-10-30T12:28:50.395090+0000","last_unstale":"2023-10-30T12:50:47.791499+0000","last_undegraded":"2023-10-30T12:50:47.791499+0000","last_fullsized":"2023-10-30T12:50:47.791499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:45:44.678383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582934,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":8195,"num_write":330,"num_write_kb":12489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"66'551","reported_seq":2714,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.380699+0000","last_change":"2023-10-30T12:38:17.104334+0000","last_active":"2023-10-30T12:51:30.380699+0000","last_peered":"2023-10-30T12:51:30.380699+0000","last_clean":"2023-10-30T12:51:30.380699+0000","last_became_active":"2023-10-30T12:28:50.929337+0000","last_became_peered":"2023-10-30T12:28:50.929337+0000","last_unstale":"2023-10-30T12:51:30.380699+0000","last_undegraded":"2023-10-30T12:51:30.380699+0000","last_fullsized":"2023-10-30T12:51:30.380699+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":551,"log_dups_size":0,"ondisk_log_size":551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023039890000000002,"stat_sum":{"num_bytes":14041074,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":50,"num_read_kb":18,"num_write":487,"num_write_kb":15488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"66'352","reported_seq":431,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:33.562730+0000","last_change":"2023-10-30T12:28:50.380474+0000","last_active":"2023-10-30T12:51:33.562730+0000","last_peered":"2023-10-30T12:51:33.562730+0000","last_clean":"2023-10-30T12:51:33.562730+0000","last_became_active":"2023-10-30T12:28:50.380247+0000","last_became_peered":"2023-10-30T12:28:50.380247+0000","last_unstale":"2023-10-30T12:51:33.562730+0000","last_undegraded":"2023-10-30T12:51:33.562730+0000","last_fullsized":"2023-10-30T12:51:33.562730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":8272,"num_write":528,"num_write_kb":5259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"66'532","reported_seq":2675,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.471259+0000","last_change":"2023-10-30T12:38:17.331272+0000","last_active":"2023-10-30T12:51:30.471259+0000","last_peered":"2023-10-30T12:51:30.471259+0000","last_clean":"2023-10-30T12:51:30.471259+0000","last_became_active":"2023-10-30T12:28:50.973086+0000","last_became_peered":"2023-10-30T12:28:50.973086+0000","last_unstale":"2023-10-30T12:51:30.471259+0000","last_undegraded":"2023-10-30T12:51:30.471259+0000","last_fullsized":"2023-10-30T12:51:30.471259+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":532,"log_dups_size":0,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034133169999999999,"stat_sum":{"num_bytes":16807043,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":30,"num_read_kb":14,"num_write":492,"num_write_kb":22119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"66'249","reported_seq":323,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:31.507410+0000","last_change":"2023-10-30T12:28:50.386277+0000","last_active":"2023-10-30T12:51:31.507410+0000","last_peered":"2023-10-30T12:51:31.507410+0000","last_clean":"2023-10-30T12:51:31.507410+0000","last_became_active":"2023-10-30T12:28:50.385924+0000","last_became_peered":"2023-10-30T12:28:50.385924+0000","last_unstale":"2023-10-30T12:51:31.507410+0000","last_undegraded":"2023-10-30T12:51:31.507410+0000","last_fullsized":"2023-10-30T12:51:31.507410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:35:03.837523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2869345,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":187,"num_write_kb":2926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"66'530","reported_seq":2686,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.403851+0000","last_change":"2023-10-30T12:38:17.643575+0000","last_active":"2023-10-30T12:51:30.403851+0000","last_peered":"2023-10-30T12:51:30.403851+0000","last_clean":"2023-10-30T12:51:30.403851+0000","last_became_active":"2023-10-30T12:28:50.929259+0000","last_became_peered":"2023-10-30T12:28:50.929259+0000","last_unstale":"2023-10-30T12:51:30.403851+0000","last_undegraded":"2023-10-30T12:51:30.403851+0000","last_fullsized":"2023-10-30T12:51:30.403851+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029033230000000002,"stat_sum":{"num_bytes":8423823,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":43,"num_read_kb":22,"num_write":477,"num_write_kb":11480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"65'373","reported_seq":455,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:41.796144+0000","last_change":"2023-10-30T12:28:50.386426+0000","last_active":"2023-10-30T12:50:41.796144+0000","last_peered":"2023-10-30T12:50:41.796144+0000","last_clean":"2023-10-30T12:50:41.796144+0000","last_became_active":"2023-10-30T12:28:50.385305+0000","last_became_peered":"2023-10-30T12:28:50.385305+0000","last_unstale":"2023-10-30T12:50:41.796144+0000","last_undegraded":"2023-10-30T12:50:41.796144+0000","last_fullsized":"2023-10-30T12:50:41.796144+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":373,"log_dups_size":0,"ondisk_log_size":373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:13:05.376581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":37322,"num_write":310,"num_write_kb":12446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"66'547","reported_seq":2697,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.501676+0000","last_change":"2023-10-30T12:38:17.111500+0000","last_active":"2023-10-30T12:51:30.501676+0000","last_peered":"2023-10-30T12:51:30.501676+0000","last_clean":"2023-10-30T12:51:30.501676+0000","last_became_active":"2023-10-30T12:28:50.940846+0000","last_became_peered":"2023-10-30T12:28:50.940846+0000","last_unstale":"2023-10-30T12:51:30.501676+0000","last_undegraded":"2023-10-30T12:51:30.501676+0000","last_fullsized":"2023-10-30T12:51:30.501676+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":547,"log_dups_size":0,"ondisk_log_size":547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046016829999999996,"stat_sum":{"num_bytes":12679067,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":37,"num_read_kb":7,"num_write":498,"num_write_kb":18971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"66'442","reported_seq":520,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:29.706310+0000","last_change":"2023-10-30T12:28:50.392333+0000","last_active":"2023-10-30T12:51:29.706310+0000","last_peered":"2023-10-30T12:51:29.706310+0000","last_clean":"2023-10-30T12:51:29.706310+0000","last_became_active":"2023-10-30T12:28:50.392068+0000","last_became_peered":"2023-10-30T12:28:50.392068+0000","last_unstale":"2023-10-30T12:51:29.706310+0000","last_undegraded":"2023-10-30T12:51:29.706310+0000","last_fullsized":"2023-10-30T12:51:29.706310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":442,"log_dups_size":0,"ondisk_log_size":442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388698,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":19,"num_read_kb":11375,"num_write":387,"num_write_kb":8544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"66'559","reported_seq":2716,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.382600+0000","last_change":"2023-10-30T12:38:17.331599+0000","last_active":"2023-10-30T12:51:30.382600+0000","last_peered":"2023-10-30T12:51:30.382600+0000","last_clean":"2023-10-30T12:51:30.382600+0000","last_became_active":"2023-10-30T12:28:50.931898+0000","last_became_peered":"2023-10-30T12:28:50.931898+0000","last_unstale":"2023-10-30T12:51:30.382600+0000","last_undegraded":"2023-10-30T12:51:30.382600+0000","last_fullsized":"2023-10-30T12:51:30.382600+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":559,"log_dups_size":0,"ondisk_log_size":559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003570148,"stat_sum":{"num_bytes":4263636,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":44,"num_read_kb":18,"num_write":507,"num_write_kb":10484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"66'554","reported_seq":628,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:31.501768+0000","last_change":"2023-10-30T12:28:50.365055+0000","last_active":"2023-10-30T12:51:31.501768+0000","last_peered":"2023-10-30T12:51:31.501768+0000","last_clean":"2023-10-30T12:51:31.501768+0000","last_became_active":"2023-10-30T12:28:50.364797+0000","last_became_peered":"2023-10-30T12:28:50.364797+0000","last_unstale":"2023-10-30T12:51:31.501768+0000","last_undegraded":"2023-10-30T12:51:31.501768+0000","last_fullsized":"2023-10-30T12:51:31.501768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":554,"log_dups_size":0,"ondisk_log_size":554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":10,"num_read_kb":8489,"num_write":691,"num_write_kb":13252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"66'555","reported_seq":2717,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.401646+0000","last_change":"2023-10-30T12:38:17.331931+0000","last_active":"2023-10-30T12:51:30.401646+0000","last_peered":"2023-10-30T12:51:30.401646+0000","last_clean":"2023-10-30T12:51:30.401646+0000","last_became_active":"2023-10-30T12:28:50.935657+0000","last_became_peered":"2023-10-30T12:28:50.935657+0000","last_unstale":"2023-10-30T12:51:30.401646+0000","last_undegraded":"2023-10-30T12:51:30.401646+0000","last_fullsized":"2023-10-30T12:51:30.401646+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":555,"log_dups_size":0,"ondisk_log_size":555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038543219999999999,"stat_sum":{"num_bytes":16829664,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":49,"num_read_kb":4112,"num_write":499,"num_write_kb":17598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"66'244","reported_seq":324,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.643888+0000","last_change":"2023-10-30T12:28:50.393465+0000","last_active":"2023-10-30T12:51:12.643888+0000","last_peered":"2023-10-30T12:51:12.643888+0000","last_clean":"2023-10-30T12:51:12.643888+0000","last_became_active":"2023-10-30T12:28:50.392807+0000","last_became_peered":"2023-10-30T12:28:50.392807+0000","last_unstale":"2023-10-30T12:51:12.643888+0000","last_undegraded":"2023-10-30T12:51:12.643888+0000","last_fullsized":"2023-10-30T12:51:12.643888+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":244,"log_dups_size":0,"ondisk_log_size":244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:35:37.976956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":24596,"num_write":178,"num_write_kb":8294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"66'505","reported_seq":2655,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.457212+0000","last_change":"2023-10-30T12:38:17.325460+0000","last_active":"2023-10-30T12:51:30.457212+0000","last_peered":"2023-10-30T12:51:30.457212+0000","last_clean":"2023-10-30T12:51:30.457212+0000","last_became_active":"2023-10-30T12:28:50.954291+0000","last_became_peered":"2023-10-30T12:28:50.954291+0000","last_unstale":"2023-10-30T12:51:30.457212+0000","last_undegraded":"2023-10-30T12:51:30.457212+0000","last_fullsized":"2023-10-30T12:51:30.457212+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":505,"log_dups_size":0,"ondisk_log_size":505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:58:33.115877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047974480000000002,"stat_sum":{"num_bytes":12709282,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":38,"num_read_kb":15,"num_write":455,"num_write_kb":14651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"66'526","reported_seq":2688,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.476097+0000","last_change":"2023-10-30T12:38:17.120085+0000","last_active":"2023-10-30T12:51:30.476097+0000","last_peered":"2023-10-30T12:51:30.476097+0000","last_clean":"2023-10-30T12:51:30.476097+0000","last_became_active":"2023-10-30T12:28:50.962817+0000","last_became_peered":"2023-10-30T12:28:50.962817+0000","last_unstale":"2023-10-30T12:51:30.476097+0000","last_undegraded":"2023-10-30T12:51:30.476097+0000","last_fullsized":"2023-10-30T12:51:30.476097+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":526,"log_dups_size":0,"ondisk_log_size":526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070898599999999999,"stat_sum":{"num_bytes":8448766,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":50,"num_read_kb":25,"num_write":483,"num_write_kb":14592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"65'173","reported_seq":249,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:12.684872+0000","last_change":"2023-10-30T12:28:50.402179+0000","last_active":"2023-10-30T12:51:12.684872+0000","last_peered":"2023-10-30T12:51:12.684872+0000","last_clean":"2023-10-30T12:51:12.684872+0000","last_became_active":"2023-10-30T12:28:50.401677+0000","last_became_peered":"2023-10-30T12:28:50.401677+0000","last_unstale":"2023-10-30T12:51:12.684872+0000","last_undegraded":"2023-10-30T12:51:12.684872+0000","last_fullsized":"2023-10-30T12:51:12.684872+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:39:44.432214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":20498,"num_write":107,"num_write_kb":8247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"66'500","reported_seq":2656,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.497349+0000","last_change":"2023-10-30T12:38:17.119058+0000","last_active":"2023-10-30T12:51:30.497349+0000","last_peered":"2023-10-30T12:51:30.497349+0000","last_clean":"2023-10-30T12:51:30.497349+0000","last_became_active":"2023-10-30T12:28:50.967591+0000","last_became_peered":"2023-10-30T12:28:50.967591+0000","last_unstale":"2023-10-30T12:51:30.497349+0000","last_undegraded":"2023-10-30T12:51:30.497349+0000","last_fullsized":"2023-10-30T12:51:30.497349+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":500,"log_dups_size":0,"ondisk_log_size":500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081968709999999997,"stat_sum":{"num_bytes":10185055,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":44,"num_read_kb":1610,"num_write":440,"num_write_kb":22911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"66'363","reported_seq":450,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:22.402277+0000","last_change":"2023-10-30T12:28:50.392617+0000","last_active":"2023-10-30T12:51:22.402277+0000","last_peered":"2023-10-30T12:51:22.402277+0000","last_clean":"2023-10-30T12:51:22.402277+0000","last_became_active":"2023-10-30T12:28:50.392021+0000","last_became_peered":"2023-10-30T12:28:50.392021+0000","last_unstale":"2023-10-30T12:51:22.402277+0000","last_undegraded":"2023-10-30T12:51:22.402277+0000","last_fullsized":"2023-10-30T12:51:22.402277+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:52:30.621125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":57384,"num_write":298,"num_write_kb":16576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"66'506","reported_seq":2663,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.462943+0000","last_change":"2023-10-30T12:38:17.643950+0000","last_active":"2023-10-30T12:51:30.462943+0000","last_peered":"2023-10-30T12:51:30.462943+0000","last_clean":"2023-10-30T12:51:30.462943+0000","last_became_active":"2023-10-30T12:28:50.967186+0000","last_became_peered":"2023-10-30T12:28:50.967186+0000","last_unstale":"2023-10-30T12:51:30.462943+0000","last_undegraded":"2023-10-30T12:51:30.462943+0000","last_fullsized":"2023-10-30T12:51:30.462943+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003136738,"stat_sum":{"num_bytes":73381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":44,"num_read_kb":17,"num_write":453,"num_write_kb":3099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"65'227","reported_seq":304,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:50:57.398216+0000","last_change":"2023-10-30T12:28:50.363230+0000","last_active":"2023-10-30T12:50:57.398216+0000","last_peered":"2023-10-30T12:50:57.398216+0000","last_clean":"2023-10-30T12:50:57.398216+0000","last_became_active":"2023-10-30T12:28:50.362929+0000","last_became_peered":"2023-10-30T12:28:50.362929+0000","last_unstale":"2023-10-30T12:50:57.398216+0000","last_undegraded":"2023-10-30T12:50:57.398216+0000","last_fullsized":"2023-10-30T12:50:57.398216+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":9,"num_read_kb":32768,"num_write":162,"num_write_kb":8341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"66'572","reported_seq":2716,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.401679+0000","last_change":"2023-10-30T12:38:17.331548+0000","last_active":"2023-10-30T12:51:30.401679+0000","last_peered":"2023-10-30T12:51:30.401679+0000","last_clean":"2023-10-30T12:51:30.401679+0000","last_became_active":"2023-10-30T12:28:50.937458+0000","last_became_peered":"2023-10-30T12:28:50.937458+0000","last_unstale":"2023-10-30T12:51:30.401679+0000","last_undegraded":"2023-10-30T12:51:30.401679+0000","last_fullsized":"2023-10-30T12:51:30.401679+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":572,"log_dups_size":0,"ondisk_log_size":572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003544045,"stat_sum":{"num_bytes":16852009,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":30,"num_read_kb":11,"num_write":518,"num_write_kb":19789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"66'450","reported_seq":553,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:34.515116+0000","last_change":"2023-10-30T12:51:34.515168+0000","last_active":"2023-10-30T12:51:34.515116+0000","last_peered":"2023-10-30T12:51:34.515116+0000","last_clean":"2023-10-30T12:51:34.515116+0000","last_became_active":"2023-10-30T12:28:50.386522+0000","last_became_peered":"2023-10-30T12:28:50.386522+0000","last_unstale":"2023-10-30T12:51:34.515116+0000","last_undegraded":"2023-10-30T12:51:34.515116+0000","last_fullsized":"2023-10-30T12:51:34.515116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'450","last_scrub_stamp":"2023-10-30T12:51:34.515038+0000","last_deep_scrub":"66'450","last_deep_scrub_stamp":"2023-10-30T12:51:34.515038+0000","last_clean_scrub_stamp":"2023-10-30T12:51:34.515038+0000","objects_scrubbed":7,"log_size":450,"log_dups_size":0,"ondisk_log_size":450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:32:31.022257+0000","scrub_duration":0.052865532999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583494,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":64,"num_read_kb":24634,"num_write":377,"num_write_kb":12490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":757298097,"num_objects":1192,"num_object_clones":0,"num_object_copies":2384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1192,"num_whiteouts":0,"num_read":2811,"num_read_kb":100236,"num_write":31900,"num_write_kb":1049744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1519566848,"data_stored":1514596194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35206,"ondisk_log_size":35206,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":305219786,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1440,"num_read_kb":712827,"num_write":10530,"num_write_kb":306771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":78,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":16940,"num_omap_keys":32,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":610648064,"data_stored":610494298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11799,"ondisk_log_size":11799,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542424,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":389672,"kb_used_data":322400,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93982168,"statfs":{"total":96636764160,"available":96237740032,"internally_reserved":0,"allocated":330137600,"data_stored":328969631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":68876747},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.78300000000000003,"15min":0.81799999999999995},"min":{"1min":0.52500000000000002,"5min":0.45900000000000002,"15min":0.40000000000000002},"max":{"1min":1.093,"5min":1.2050000000000001,"15min":1.595},"last":1.1120000000000001},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.78400000000000003,"15min":0.84199999999999997},"min":{"1min":0.49099999999999999,"5min":0.441,"15min":0.40999999999999998},"max":{"1min":1.113,"5min":1.302,"15min":1.4830000000000001},"last":0.754}]},{"osd":1,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.85099999999999998,"15min":0.88800000000000001},"min":{"1min":0.57199999999999995,"5min":0.52700000000000002,"15min":0.441},"max":{"1min":1.2709999999999999,"5min":3.4329999999999998,"15min":3.4329999999999998},"last":1.056},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.85399999999999998,"15min":0.88500000000000001},"min":{"1min":0.56999999999999995,"5min":0.505,"15min":0.505},"max":{"1min":1.1419999999999999,"5min":1.383,"15min":1.3919999999999999},"last":0.70499999999999996}]},{"osd":2,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.89000000000000001,"15min":0.92200000000000004},"min":{"1min":0.58899999999999997,"5min":0.55100000000000005,"15min":0.45900000000000002},"max":{"1min":1.163,"5min":3.4820000000000002,"15min":3.4820000000000002},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.877,"15min":0.83699999999999997},"min":{"1min":0.65000000000000002,"5min":0.52800000000000002,"15min":0.47099999999999997},"max":{"1min":1.2330000000000001,"5min":1.4570000000000001,"15min":1.6639999999999999},"last":0.84999999999999998}]},{"osd":3,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":1.0069999999999999,"15min":0.96899999999999997},"min":{"1min":0.61299999999999999,"5min":0.60999999999999999,"15min":0.45500000000000002},"max":{"1min":1.1439999999999999,"5min":3.5190000000000001,"15min":3.5190000000000001},"last":1.161},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.95599999999999996,"15min":0.99399999999999999},"min":{"1min":0.72899999999999998,"5min":0.54500000000000004,"15min":0.441},"max":{"1min":1.3220000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":1.0049999999999999}]},{"osd":4,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.97599999999999998,"15min":1.002},"min":{"1min":0.63800000000000001,"5min":0.42499999999999999,"15min":0.34399999999999997},"max":{"1min":1.2070000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.999,"15min":1.042},"min":{"1min":0.61799999999999999,"5min":0.45000000000000001,"15min":0.33800000000000002},"max":{"1min":1.1910000000000001,"5min":1.746,"15min":1.746},"last":1.208}]},{"osd":5,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.016,"15min":0.96099999999999997},"min":{"1min":0.76700000000000002,"5min":0.53700000000000003,"15min":0.43099999999999999},"max":{"1min":1.2949999999999999,"5min":1.6160000000000001,"15min":1.7549999999999999},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.117,"15min":1.0680000000000001},"min":{"1min":0.71499999999999997,"5min":0.57899999999999996,"15min":0.35999999999999999},"max":{"1min":1.2370000000000001,"5min":2.1600000000000001,"15min":2.1600000000000001},"last":1.296}]},{"osd":6,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.141,"15min":1.123},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.39000000000000001},"max":{"1min":1.216,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.3740000000000001},{"interface":"front","average":{"1min":1,"5min":1.1259999999999999,"15min":1.167},"min":{"1min":0.68200000000000005,"5min":0.56299999999999994,"15min":0.47199999999999998},"max":{"1min":1.3300000000000001,"5min":3.8140000000000001,"15min":3.8140000000000001},"last":1.262}]}]},{"osd":6,"up_from":14,"seq":60129542423,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297812,"kb_used_data":234764,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94074028,"statfs":{"total":96636764160,"available":96331804672,"internally_reserved":0,"allocated":240398336,"data_stored":239540666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":64551368},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.83299999999999996,"15min":0.85999999999999999},"min":{"1min":0.52900000000000003,"5min":0.46600000000000003,"15min":0.35599999999999998},"max":{"1min":1.155,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.749,"5min":0.77000000000000002,"15min":0.78800000000000003},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.42999999999999999},"max":{"1min":1.05,"5min":1.109,"15min":1.214},"last":1.0309999999999999}]},{"osd":1,"last update":"Mon Oct 30 12:51:28 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.82099999999999995,"15min":0.81599999999999995},"min":{"1min":0.45000000000000001,"5min":0.42299999999999999,"15min":0.317},"max":{"1min":1.23,"5min":1.2809999999999999,"15min":1.3140000000000001},"last":1.0669999999999999},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.85199999999999998,"15min":0.876},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.40999999999999998},"max":{"1min":1.1399999999999999,"5min":1.2869999999999999,"15min":1.423},"last":0.96999999999999997}]},{"osd":2,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.93600000000000005,"15min":0.97499999999999998},"min":{"1min":0.495,"5min":0.495,"15min":0.38800000000000001},"max":{"1min":1.373,"5min":1.47,"15min":4.3360000000000003},"last":1.002},{"interface":"front","average":{"1min":1.0409999999999999,"5min":0.97999999999999998,"15min":1.0089999999999999},"min":{"1min":0.65000000000000002,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.502,"5min":1.502,"15min":1.5629999999999999},"last":1.153}]},{"osd":3,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.996,"15min":1.042},"min":{"1min":0.64700000000000002,"5min":0.59099999999999997,"15min":0.58999999999999997},"max":{"1min":1.3340000000000001,"5min":1.5509999999999999,"15min":1.6459999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.02,"5min":1.0680000000000001,"15min":1.101},"min":{"1min":0.72299999999999998,"5min":0.63300000000000001,"15min":0.59699999999999998},"max":{"1min":1.417,"5min":1.552,"15min":1.7969999999999999},"last":1.304}]},{"osd":4,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.149,"15min":1.075},"min":{"1min":0.72599999999999998,"5min":0.69899999999999995,"15min":0.51500000000000001},"max":{"1min":1.585,"5min":1.6499999999999999,"15min":1.7649999999999999},"last":1.2430000000000001},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.89400000000000002,"15min":0.96499999999999997},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.42099999999999999},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.881},"last":0.752}]},{"osd":5,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":1.0720000000000001,"15min":1.087},"min":{"1min":0.54100000000000004,"5min":0.53200000000000003,"15min":0.48399999999999999},"max":{"1min":1.2849999999999999,"5min":1.75,"15min":2.2770000000000001},"last":0.85099999999999998},{"interface":"front","average":{"1min":1.056,"5min":1.159,"15min":1.0649999999999999},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.46999999999999997},"max":{"1min":1.4610000000000001,"5min":1.6950000000000001,"15min":4.7850000000000001},"last":1.107}]},{"osd":7,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.036,"15min":1.0029999999999999},"min":{"1min":0.68799999999999994,"5min":0.64100000000000001,"15min":0.46700000000000003},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.825},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.0629999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.48999999999999999},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.895},"last":1.206}]}]},{"osd":5,"up_from":14,"seq":60129542424,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":338384,"kb_used_data":271112,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94033456,"statfs":{"total":96636764160,"available":96290258944,"internally_reserved":0,"allocated":277618688,"data_stored":276486603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.78400000000000003,"15min":0.80300000000000005},"min":{"1min":0.56299999999999994,"5min":0.33900000000000002,"15min":0.29399999999999998},"max":{"1min":1.1970000000000001,"5min":1.238,"15min":2.4729999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.79700000000000004,"15min":0.80400000000000005},"min":{"1min":0.61199999999999999,"5min":0.39200000000000002,"15min":0.33800000000000002},"max":{"1min":1.2390000000000001,"5min":1.3009999999999999,"15min":1.8759999999999999},"last":0.69499999999999995}]},{"osd":1,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.84199999999999997,"15min":0.84399999999999997},"min":{"1min":0.57399999999999995,"5min":0.40999999999999998,"15min":0.38900000000000001},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":3.0190000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.58499999999999996,"5min":0.44800000000000001,"15min":0.373},"max":{"1min":1.264,"5min":1.276,"15min":1.4870000000000001},"last":0.86199999999999999}]},{"osd":2,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.88400000000000001,"15min":0.84699999999999998},"min":{"1min":0.67300000000000004,"5min":0.55400000000000005,"15min":0.38400000000000001},"max":{"1min":1.157,"5min":1.264,"15min":2.2949999999999999},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.81699999999999995,"15min":0.876},"min":{"1min":0.63600000000000001,"5min":0.434,"15min":0.434},"max":{"1min":1.1140000000000001,"5min":1.171,"15min":1.5800000000000001},"last":0.64000000000000001}]},{"osd":3,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.91800000000000004,"15min":0.93700000000000006},"min":{"1min":0.57199999999999995,"5min":0.53800000000000003,"15min":0.47899999999999998},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.756},"last":1.03},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.93600000000000005,"15min":0.94599999999999995},"min":{"1min":0.755,"5min":0.498,"15min":0.41399999999999998},"max":{"1min":1.204,"5min":1.4119999999999999,"15min":2.3679999999999999},"last":1.113}]},{"osd":4,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.84499999999999997,"15min":0.90900000000000003},"min":{"1min":0.47499999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.304,"5min":1.304,"15min":3.1190000000000002},"last":0.80800000000000005},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.083,"15min":1.091},"min":{"1min":0.59499999999999997,"5min":0.58699999999999997,"15min":0.50900000000000001},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":2.645},"last":1.0700000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.038,"15min":1.042},"min":{"1min":0.51100000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.466,"5min":1.466,"15min":3.1859999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.056,"15min":1.1060000000000001},"min":{"1min":0.67000000000000004,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.3260000000000001,"5min":1.47,"15min":3.371},"last":1.1659999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.0609999999999999,"15min":1.0800000000000001},"min":{"1min":0.65200000000000002,"5min":0.51400000000000001,"15min":0.47299999999999998},"max":{"1min":1.494,"5min":1.508,"15min":3.5310000000000001},"last":1.218},{"interface":"front","average":{"1min":1.282,"5min":1.1970000000000001,"15min":1.2310000000000001},"min":{"1min":0.96799999999999997,"5min":0.64700000000000002,"15min":0.61299999999999999},"max":{"1min":1.605,"5min":1.605,"15min":3.6930000000000001},"last":1.266}]}]},{"osd":4,"up_from":14,"seq":60129542423,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369460,"kb_used_data":293868,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94002380,"statfs":{"total":96636764160,"available":96258437120,"internally_reserved":0,"allocated":300920832,"data_stored":299840459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:15 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.81799999999999995,"15min":0.92600000000000005},"min":{"1min":0.48099999999999998,"5min":0.34799999999999998,"15min":0.31},"max":{"1min":1.204,"5min":1.2509999999999999,"15min":11.273},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.92400000000000004,"15min":0.98799999999999999},"min":{"1min":0.58099999999999996,"5min":0.44,"15min":0.44},"max":{"1min":1.411,"5min":1.4590000000000001,"15min":7.1289999999999996},"last":0.90200000000000002}]},{"osd":1,"last update":"Mon Oct 30 12:51:15 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.81499999999999995,"15min":0.90500000000000003},"min":{"1min":0.52800000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0049999999999999,"5min":5.1130000000000004,"15min":6.9199999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.871,"5min":0.84199999999999997,"15min":0.90000000000000002},"min":{"1min":0.63900000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.246,"5min":1.246,"15min":4.75},"last":0.72899999999999998}]},{"osd":2,"last update":"Mon Oct 30 12:51:15 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.84199999999999997,"15min":0.84899999999999998},"min":{"1min":0.51700000000000002,"5min":0.41899999999999998,"15min":0.36899999999999999},"max":{"1min":1.175,"5min":1.21,"15min":2.6970000000000001},"last":1.0780000000000001},{"interface":"front","average":{"1min":0.996,"5min":0.91200000000000003,"15min":1.0569999999999999},"min":{"1min":0.60999999999999999,"5min":0.54400000000000004,"15min":0.52900000000000003},"max":{"1min":1.361,"5min":1.361,"15min":18.917999999999999},"last":0.85999999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":0.96599999999999997,"15min":0.997},"min":{"1min":0.71399999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.258,"5min":5.1719999999999997,"15min":7.1879999999999997},"last":1.1739999999999999},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.90800000000000003,"15min":1},"min":{"1min":0.57399999999999995,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.125,"5min":1.3600000000000001,"15min":8.3360000000000003},"last":0.81200000000000006}]},{"osd":5,"last update":"Mon Oct 30 12:51:15 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.88300000000000001,"15min":1.002},"min":{"1min":0.56299999999999994,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.159,"5min":1.3300000000000001,"15min":6.0759999999999996},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.119,"5min":1.105,"15min":1.1890000000000001},"min":{"1min":0.89500000000000002,"5min":0.71899999999999997,"15min":0.52600000000000002},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":18.994},"last":1.127}]},{"osd":6,"last update":"Mon Oct 30 12:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.0880000000000001,"15min":1.093},"min":{"1min":0.70199999999999996,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.429,"5min":1.429,"15min":6.1580000000000004},"last":1.258},{"interface":"front","average":{"1min":1.016,"5min":1.0169999999999999,"15min":1.1240000000000001},"min":{"1min":0.629,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.28,"5min":5.218,"15min":7.0549999999999997},"last":1.0429999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:51:15 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.0980000000000001,"15min":1.258},"min":{"1min":0.75800000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.4339999999999999,"5min":5.7160000000000002,"15min":18.015999999999998},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.236,"5min":1.2050000000000001,"15min":1.2769999999999999},"min":{"1min":1.093,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.4530000000000001,"5min":5.6609999999999996,"15min":19.052},"last":1.345}]}]},{"osd":3,"up_from":14,"seq":60129542424,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":373108,"kb_used_data":309996,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93998732,"statfs":{"total":96636764160,"available":96254701568,"internally_reserved":0,"allocated":317435904,"data_stored":316322819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":64616904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.39900000000000002,"15min":0.41399999999999998},"min":{"1min":0.29399999999999998,"5min":0.23699999999999999,"15min":0.23400000000000001},"max":{"1min":0.58999999999999997,"5min":0.77900000000000003,"15min":1.6899999999999999},"last":0.435},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51600000000000001,"15min":0.53300000000000003},"min":{"1min":0.40000000000000002,"5min":0.34300000000000003,"15min":0.32700000000000001},"max":{"1min":0.746,"5min":0.871,"15min":1.728},"last":0.52400000000000002}]},{"osd":1,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.42899999999999999,"15min":0.42999999999999999},"min":{"1min":0.309,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":1.2909999999999999},"last":0.308},{"interface":"front","average":{"1min":0.497,"5min":0.496,"15min":0.50600000000000001},"min":{"1min":0.34499999999999997,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.64600000000000002,"5min":0.80600000000000005,"15min":1.3260000000000001},"last":0.47899999999999998}]},{"osd":2,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.496,"15min":0.502},"min":{"1min":0.37,"5min":0.30099999999999999,"15min":0.29299999999999998},"max":{"1min":0.69899999999999995,"5min":0.78600000000000003,"15min":1.099},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55900000000000005,"15min":0.56699999999999995},"min":{"1min":0.41999999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73599999999999999,"5min":0.82599999999999996,"15min":1.2509999999999999},"last":0.45000000000000001}]},{"osd":4,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.85199999999999998,"15min":0.83099999999999996},"min":{"1min":0.63900000000000001,"5min":0.55400000000000005,"15min":0.40400000000000003},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":2.202},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.82999999999999996,"15min":0.86499999999999999},"min":{"1min":0.70599999999999996,"5min":0.48499999999999999,"15min":0.46800000000000003},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":2.4340000000000002},"last":0.78100000000000003}]},{"osd":5,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.81499999999999995,"15min":0.86299999999999999},"min":{"1min":0.66200000000000003,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.169,"5min":1.169,"15min":14.103999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.875,"5min":0.876,"15min":0.86499999999999999},"min":{"1min":0.63900000000000001,"5min":0.55000000000000004,"15min":0.499},"max":{"1min":1.2170000000000001,"5min":1.2869999999999999,"15min":4.4859999999999998},"last":0.82399999999999995}]},{"osd":6,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.85299999999999998,"15min":0.83499999999999996},"min":{"1min":0.64700000000000002,"5min":0.56999999999999995,"15min":0.53100000000000003},"max":{"1min":1.087,"5min":1.2450000000000001,"15min":2.2330000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.87,"15min":0.82899999999999996},"min":{"1min":0.77800000000000002,"5min":0.54100000000000004,"15min":0.45100000000000001},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":4.4409999999999998},"last":0.79800000000000004}]},{"osd":7,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.875,"15min":0.86499999999999999},"min":{"1min":0.67500000000000004,"5min":0.57299999999999995,"15min":0.57099999999999995},"max":{"1min":1.1499999999999999,"5min":1.2509999999999999,"15min":3.5459999999999998},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.86399999999999999,"15min":0.84699999999999998},"min":{"1min":0.65700000000000003,"5min":0.57399999999999995,"15min":0.45700000000000002},"max":{"1min":1.1259999999999999,"5min":1.2230000000000001,"15min":6.1360000000000001},"last":0.73899999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542424,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":200564,"kb_used_data":143020,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94171276,"statfs":{"total":96636764160,"available":96431386624,"internally_reserved":0,"allocated":146452480,"data_stored":145637539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.45100000000000001,"15min":0.441},"min":{"1min":0.25,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.59499999999999997,"5min":0.69399999999999995,"15min":0.81200000000000006},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.5,"15min":0.51100000000000001},"min":{"1min":0.37,"5min":0.33300000000000002,"15min":0.31900000000000001},"max":{"1min":0.63,"5min":0.71299999999999997,"15min":1.298},"last":0.442}]},{"osd":1,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.47899999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.60199999999999998,"5min":0.65500000000000003,"15min":0.83699999999999997},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.52500000000000002,"15min":0.504},"min":{"1min":0.38900000000000001,"5min":0.33700000000000002,"15min":0.29999999999999999},"max":{"1min":0.61699999999999999,"5min":0.74099999999999999,"15min":1.2250000000000001},"last":0.39700000000000002}]},{"osd":3,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.47399999999999998,"15min":0.54800000000000004},"min":{"1min":0.315,"5min":0.311,"15min":0.311},"max":{"1min":0.58599999999999997,"5min":0.75,"15min":1.673},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57299999999999995,"15min":0.57399999999999995},"min":{"1min":0.41799999999999998,"5min":0.40200000000000002,"15min":0.35699999999999998},"max":{"1min":0.67900000000000005,"5min":0.79100000000000004,"15min":1.597},"last":0.54700000000000004}]},{"osd":4,"last update":"Mon Oct 30 12:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.74299999999999999,"15min":0.82599999999999996},"min":{"1min":0.47499999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.91100000000000003,"5min":1.0569999999999999,"15min":1.377},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.755,"15min":0.79200000000000004},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.998,"5min":1.0640000000000001,"15min":1.4159999999999999},"last":0.76700000000000002}]},{"osd":5,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.84099999999999997,"15min":0.84799999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.093,"5min":1.2430000000000001,"15min":1.78},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91400000000000003},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.1060000000000001,"5min":1.2729999999999999,"15min":1.7430000000000001},"last":0.751}]},{"osd":6,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.86199999999999999,"15min":0.85799999999999998},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.60299999999999998},"max":{"1min":1.095,"5min":1.177,"15min":1.3799999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.86599999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.53300000000000003},"max":{"1min":1.097,"5min":1.1799999999999999,"15min":1.8109999999999999},"last":0.84399999999999997}]},{"osd":7,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.82899999999999996,"15min":0.83099999999999996},"min":{"1min":0.58799999999999997,"5min":0.58299999999999996,"15min":0.48799999999999999},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.4159999999999999},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.85399999999999998,"15min":0.83299999999999996},"min":{"1min":0.75900000000000001,"5min":0.51800000000000002,"15min":0.50800000000000001},"max":{"1min":1.206,"5min":1.206,"15min":1.3049999999999999},"last":0.89100000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542424,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":279036,"kb_used_data":233972,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94092804,"statfs":{"total":96636764160,"available":96351031296,"internally_reserved":0,"allocated":239587328,"data_stored":238747727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:50:34 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.40400000000000003,"15min":0.45400000000000001},"min":{"1min":0.28399999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.63600000000000001,"5min":0.65200000000000002,"15min":11.958},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.495,"15min":0.55700000000000005},"min":{"1min":0.34699999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.73599999999999999,"5min":0.73799999999999999,"15min":12.077},"last":0.35199999999999998}]},{"osd":2,"last update":"Mon Oct 30 12:50:34 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45800000000000002,"15min":0.46000000000000002},"min":{"1min":0.29699999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.121,"5min":1.121,"15min":1.623},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52500000000000002,"15min":0.53300000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.31900000000000001},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.5389999999999999},"last":0.51900000000000002}]},{"osd":3,"last update":"Mon Oct 30 12:50:34 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55400000000000005,"15min":0.56699999999999995},"min":{"1min":0.378,"5min":0.31,"15min":0.31},"max":{"1min":1.216,"5min":1.337,"15min":3.694},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.498,"15min":0.54700000000000004},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.66900000000000004,"5min":0.77200000000000002,"15min":13.199999999999999},"last":0.499}]},{"osd":4,"last update":"Mon Oct 30 12:50:34 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.82699999999999996,"15min":0.83199999999999996},"min":{"1min":0.54700000000000004,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1519999999999999,"5min":1.3140000000000001,"15min":2.419},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.871,"15min":0.86699999999999999},"min":{"1min":0.66300000000000003,"5min":0.60899999999999999,"15min":0.52100000000000002},"max":{"1min":1.095,"5min":1.103,"15min":2.282},"last":0.80900000000000005}]},{"osd":5,"last update":"Mon Oct 30 12:50:34 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.80000000000000004,"15min":0.81599999999999995},"min":{"1min":0.61699999999999999,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.01,"5min":1.343,"15min":1.343},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.79000000000000004,"15min":0.85599999999999998},"min":{"1min":0.59099999999999997,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.115,"5min":1.29,"15min":1.29},"last":0.58499999999999996}]},{"osd":6,"last update":"Mon Oct 30 12:50:34 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82799999999999996,"15min":0.79800000000000004},"min":{"1min":0.56399999999999995,"5min":0.52800000000000002,"15min":0.51200000000000001},"max":{"1min":1.3660000000000001,"5min":1.371,"15min":1.371},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.89400000000000002,"15min":0.86699999999999999},"min":{"1min":0.67400000000000004,"5min":0.498,"15min":0.498},"max":{"1min":1.417,"5min":1.417,"15min":2.464},"last":0.79300000000000004}]},{"osd":7,"last update":"Mon Oct 30 12:50:34 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.81100000000000005,"15min":0.83199999999999996},"min":{"1min":0.61899999999999999,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0349999999999999,"5min":1.1659999999999999,"15min":2.5169999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.875,"5min":0.88200000000000001,"15min":0.90700000000000003},"min":{"1min":0.76900000000000002,"5min":0.61899999999999999,"15min":0.58399999999999996},"max":{"1min":1.133,"5min":1.133,"15min":2.5510000000000002},"last":0.88800000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542424,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":349720,"kb_used_data":278288,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94022120,"statfs":{"total":96636764160,"available":96278650880,"internally_reserved":0,"allocated":284966912,"data_stored":283825495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 12:51:29 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.45700000000000002,"15min":0.45200000000000001},"min":{"1min":0.28699999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.64600000000000002,"5min":0.71499999999999997,"15min":0.93700000000000006},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51500000000000001,"15min":0.50700000000000001},"min":{"1min":0.38700000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.73799999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.443}]},{"osd":2,"last update":"Mon Oct 30 12:51:29 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45600000000000002,"15min":0.502},"min":{"1min":0.315,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.97199999999999998},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56000000000000005,"15min":0.53300000000000003},"min":{"1min":0.437,"5min":0.36799999999999999,"15min":0.27400000000000002},"max":{"1min":0.79000000000000004,"5min":0.83299999999999996,"15min":1.0149999999999999},"last":0.45800000000000002}]},{"osd":3,"last update":"Mon Oct 30 12:51:29 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47899999999999998,"15min":0.51300000000000001},"min":{"1min":0.36099999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.67100000000000004,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.5,"15min":0.497},"min":{"1min":0.41099999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.71199999999999997,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.48499999999999999}]},{"osd":4,"last update":"Mon Oct 30 12:51:29 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.87,"15min":0.85299999999999998},"min":{"1min":0.57699999999999996,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1899999999999999,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.877,"15min":0.85899999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.48299999999999998},"max":{"1min":1.0189999999999999,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.86899999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:51:29 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.81399999999999995,"15min":0.83299999999999996},"min":{"1min":0.55200000000000005,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.163,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83699999999999997,"15min":0.86799999999999999},"min":{"1min":0.68999999999999995,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0289999999999999,"5min":1.889,"15min":1.889},"last":0.67800000000000005}]},{"osd":6,"last update":"Mon Oct 30 12:51:29 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80300000000000005,"15min":0.79500000000000004},"min":{"1min":0.63100000000000001,"5min":0.56299999999999994,"15min":0.52200000000000002},"max":{"1min":1.071,"5min":1.1990000000000001,"15min":2.6480000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.80100000000000005,"15min":0.81999999999999995},"min":{"1min":0.60999999999999999,"5min":0.48799999999999999,"15min":0.46999999999999997},"max":{"1min":1.071,"5min":1.1559999999999999,"15min":5.0099999999999998},"last":0.621}]},{"osd":7,"last update":"Mon Oct 30 12:51:29 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.872,"15min":0.88300000000000001},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.55000000000000004},"max":{"1min":1.25,"5min":1.401,"15min":3.3359999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.93899999999999995,"15min":0.94799999999999995},"min":{"1min":0.73899999999999999,"5min":0.65100000000000002,"15min":0.56299999999999994},"max":{"1min":1.2729999999999999,"5min":1.3500000000000001,"15min":3.3580000000000001},"last":0.88200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62930944,"data_stored":62915220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":83910656,"data_stored":83887024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":37928960,"data_stored":37906773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":83668992,"data_stored":83648625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":76107776,"data_stored":76093296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":62935040,"data_stored":62915938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":91164672,"data_stored":91146041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":112001024,"data_stored":111981381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":220680192,"data_stored":219932830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":154910720,"data_stored":154473007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":107761664,"data_stored":107343689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":233000960,"data_stored":232286498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":224047104,"data_stored":223359467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":213921792,"data_stored":213183588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":148467712,"data_stored":148006929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":216776704,"data_stored":216010186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:51:37.893 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.894 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.894 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.895 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.895 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.895 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.896 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.896 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.897 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.897 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.897 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.898 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.898 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.898 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.899 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.899 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.900 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.900 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.900 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.901 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.901 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.901 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.902 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.902 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.902 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.903 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.903 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.904 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.904 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.904 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.905 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.905 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.905 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.906 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.906 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.907 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.907 INFO:tasks.ceph:pgid 2.19 last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.907 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.908 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.908 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.908 INFO:tasks.ceph:pgid 2.17 last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.909 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.909 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.910 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.910 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.911 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.911 INFO:tasks.ceph:pgid 2.14 last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.911 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.912 INFO:tasks.ceph:pgid 2.13 last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.912 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.913 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.913 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.913 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.914 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.914 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.914 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.915 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.915 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.915 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.916 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.916 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.916 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.917 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.917 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-30T12:28:39.218257+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=39, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.918 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.918 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.918 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.919 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.919 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.919 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.920 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.920 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.920 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.921 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.921 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.921 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.922 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.922 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.923 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.923 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.923 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.924 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.924 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.925 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.925 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.925 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.926 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.926 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.927 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.927 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.927 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.928 INFO:tasks.ceph:pgid 2.1e last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.928 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:37.928 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T12:51:48.818 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:51:48.822 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:51:57.930 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:51:58.078 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.077+0000 7f4131b0f640 1 Processor -- start 2023-10-30T12:51:58.079 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.077+0000 7f4131b0f640 1 -- start start 2023-10-30T12:51:58.080 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.081+0000 7f4131b0f640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f412c0c6b80 0x7f412c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:58.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.081+0000 7f4131b0f640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f412c0c6f60 0x7f412c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:58.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.081+0000 7f4131b0f640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f412c0c0a50 con 0x7f412c0c8430 2023-10-30T12:51:58.081 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.081+0000 7f4131b0f640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f412c06f6e0 con 0x7f412c0c6f60 2023-10-30T12:51:58.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.081+0000 7f4131b0f640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f412c06f9c0 con 0x7f412c0c6b80 2023-10-30T12:51:58.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.081+0000 7f412affd640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f412c0c6f60 0x7f412c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:58.082 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.081+0000 7f412affd640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f412c0c6f60 0x7f412c0c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37688/0 (socket says 172.21.15.89:37688) 2023-10-30T12:51:58.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.081+0000 7f412affd640 1 -- 172.21.15.89:0/372210151 learned_addr learned my addr 172.21.15.89:0/372210151 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:58.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.081+0000 7f412b7fe640 1 --2- 172.21.15.89:0/372210151 >> v2:172.21.15.192:3300/0 conn(0x7f412c0c6b80 0x7f412c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:58.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.081+0000 7f412affd640 1 -- 172.21.15.89:0/372210151 >> v2:172.21.15.192:3300/0 conn(0x7f412c0c6b80 msgr2=0x7f412c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:58.083 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.081+0000 7f412affd640 1 --2- 172.21.15.89:0/372210151 >> v2:172.21.15.192:3300/0 conn(0x7f412c0c6b80 0x7f412c0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:58.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.081+0000 7f412affd640 1 -- 172.21.15.89:0/372210151 >> v1:172.21.15.89:6789/0 conn(0x7f412c0c8430 legacy=0x7f412c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:58.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.081+0000 7f412affd640 1 -- 172.21.15.89:0/372210151 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f412c06fca0 con 0x7f412c0c6f60 2023-10-30T12:51:58.084 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.081+0000 7f412a7fc640 1 -- 172.21.15.89:0/372210151 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 644853934 0 0) 0x7f412c0c0a50 con 0x7f412c0c8430 2023-10-30T12:51:58.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.081+0000 7f412affd640 1 --2- 172.21.15.89:0/372210151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c0c6f60 0x7f412c0c0400 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f412c06f1b0 tx=0x7f412001ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa07f061b6501c48 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:58.086 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.085+0000 7f412a7fc640 1 -- 172.21.15.89:0/372210151 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41200a7bb0 con 0x7f412c0c6f60 2023-10-30T12:51:58.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.085+0000 7f412a7fc640 1 -- 172.21.15.89:0/372210151 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41200a7d10 con 0x7f412c0c6f60 2023-10-30T12:51:58.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.085+0000 7f412a7fc640 1 -- 172.21.15.89:0/372210151 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41200af660 con 0x7f412c0c6f60 2023-10-30T12:51:58.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.085+0000 7f4131b0f640 1 -- 172.21.15.89:0/372210151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c0c6f60 msgr2=0x7f412c0c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:58.087 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.085+0000 7f4131b0f640 1 --2- 172.21.15.89:0/372210151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c0c6f60 0x7f412c0c0400 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f412c06f1b0 tx=0x7f412001ebd0 comp rx=0 tx=0).stop 2023-10-30T12:51:58.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.085+0000 7f4131b0f640 1 -- 172.21.15.89:0/372210151 shutdown_connections 2023-10-30T12:51:58.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.085+0000 7f4131b0f640 1 --2- 172.21.15.89:0/372210151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c0c6f60 0x7f412c0c0400 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:58.088 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.085+0000 7f4131b0f640 1 --2- 172.21.15.89:0/372210151 >> v2:172.21.15.192:3300/0 conn(0x7f412c0c6b80 0x7f412c0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:58.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.085+0000 7f4131b0f640 1 -- 172.21.15.89:0/372210151 >> 172.21.15.89:0/372210151 conn(0x7f412c0c7910 msgr2=0x7f412c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:58.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.085+0000 7f4131b0f640 1 -- 172.21.15.89:0/372210151 shutdown_connections 2023-10-30T12:51:58.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.085+0000 7f4131b0f640 1 -- 172.21.15.89:0/372210151 wait complete. 2023-10-30T12:51:58.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.085+0000 7f4131b0f640 1 Processor -- start 2023-10-30T12:51:58.090 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.085+0000 7f4131b0f640 1 -- start start 2023-10-30T12:51:58.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.085+0000 7f4131b0f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f412c0c6b80 0x7f412c0c9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:58.092 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.085+0000 7f4131b0f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c0c6f60 0x7f412c132930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:58.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.085+0000 7f4131b0f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f412c0c8430 0x7f412c132e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:58.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.085+0000 7f4131b0f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412c156260 con 0x7f412c0c6f60 2023-10-30T12:51:58.093 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.085+0000 7f4131b0f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412c1526c0 con 0x7f412c0c6b80 2023-10-30T12:51:58.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.085+0000 7f4131b0f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f412c1522c0 con 0x7f412c0c8430 2023-10-30T12:51:58.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.085+0000 7f412affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c0c6f60 0x7f412c132930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:58.094 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.085+0000 7f412affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c0c6f60 0x7f412c132930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37692/0 (socket says 172.21.15.89:37692) 2023-10-30T12:51:58.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.085+0000 7f412affd640 1 -- 172.21.15.89:0/1056867895 learned_addr learned my addr 172.21.15.89:0/1056867895 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:51:58.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.089+0000 7f412b7fe640 1 --2- 172.21.15.89:0/1056867895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f412c0c6b80 0x7f412c0c9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:58.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.089+0000 7f412bfff640 1 --2- 172.21.15.89:0/1056867895 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f412c0c8430 0x7f412c132e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:58.095 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.089+0000 7f412affd640 1 -- 172.21.15.89:0/1056867895 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f412c0c8430 msgr2=0x7f412c132e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:58.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.089+0000 7f412affd640 1 --2- 172.21.15.89:0/1056867895 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f412c0c8430 0x7f412c132e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:58.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.089+0000 7f412affd640 1 -- 172.21.15.89:0/1056867895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f412c0c6b80 msgr2=0x7f412c0c9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:58.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.089+0000 7f412affd640 1 --2- 172.21.15.89:0/1056867895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f412c0c6b80 0x7f412c0c9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:58.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.089+0000 7f412affd640 1 -- 172.21.15.89:0/1056867895 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f412000b770 con 0x7f412c0c6f60 2023-10-30T12:51:58.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.089+0000 7f412b7fe640 1 --2- 172.21.15.89:0/1056867895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f412c0c6b80 0x7f412c0c9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:58.097 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.089+0000 7f412affd640 1 --2- 172.21.15.89:0/1056867895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c0c6f60 0x7f412c132930 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f412001b7e0 tx=0x7f412001b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:58.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.089+0000 7f4113fff640 1 -- 172.21.15.89:0/1056867895 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4120006ee0 con 0x7f412c0c6f60 2023-10-30T12:51:58.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.089+0000 7f4113fff640 1 -- 172.21.15.89:0/1056867895 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41200ba640 con 0x7f412c0c6f60 2023-10-30T12:51:58.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.089+0000 7f4113fff640 1 -- 172.21.15.89:0/1056867895 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41200ba9f0 con 0x7f412c0c6f60 2023-10-30T12:51:58.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.089+0000 7f4131b0f640 1 -- 172.21.15.89:0/1056867895 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f412c134cf0 con 0x7f412c0c6f60 2023-10-30T12:51:58.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.089+0000 7f4131b0f640 1 -- 172.21.15.89:0/1056867895 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f412c153450 con 0x7f412c0c6f60 2023-10-30T12:51:58.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.089+0000 7f4131b0f640 1 -- 172.21.15.89:0/1056867895 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f412c132650 con 0x7f412c0c6f60 2023-10-30T12:51:58.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.089+0000 7f4113fff640 1 -- 172.21.15.89:0/1056867895 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f41200b6040 con 0x7f412c0c6f60 2023-10-30T12:51:58.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.089+0000 7f4113fff640 1 --2- 172.21.15.89:0/1056867895 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f41180664a0 0x7f4118068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:58.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.089+0000 7f4113fff640 1 -- 172.21.15.89:0/1056867895 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f41201217a0 con 0x7f412c0c6f60 2023-10-30T12:51:58.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.093+0000 7f412b7fe640 1 --2- 172.21.15.89:0/1056867895 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f41180664a0 0x7f4118068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:58.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.093+0000 7f412b7fe640 1 --2- 172.21.15.89:0/1056867895 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f41180664a0 0x7f4118068960 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f411c007a70 tx=0x7f411c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:58.101 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.093+0000 7f4113fff640 1 -- 172.21.15.89:0/1056867895 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f412c132650 con 0x7f412c0c6f60 2023-10-30T12:51:58.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.241+0000 7f4131b0f640 1 -- 172.21.15.89:0/1056867895 --> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f412c132650 con 0x7f41180664a0 2023-10-30T12:51:58.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.261+0000 7f4113fff640 1 -- 172.21.15.89:0/1056867895 <== mgr.4105 v2:172.21.15.192:6832/4076276638 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+291926 (secure 0 0 0) 0x7f412c132650 con 0x7f41180664a0 2023-10-30T12:51:58.265 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:51:58.266 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-10-30T12:51:58.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.265+0000 7f4131b0f640 1 -- 172.21.15.89:0/1056867895 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f41180664a0 msgr2=0x7f4118068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:58.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.265+0000 7f4131b0f640 1 --2- 172.21.15.89:0/1056867895 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f41180664a0 0x7f4118068960 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f411c007a70 tx=0x7f411c008040 comp rx=0 tx=0).stop 2023-10-30T12:51:58.268 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.265+0000 7f4131b0f640 1 -- 172.21.15.89:0/1056867895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c0c6f60 msgr2=0x7f412c132930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:58.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.265+0000 7f4131b0f640 1 --2- 172.21.15.89:0/1056867895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c0c6f60 0x7f412c132930 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f412001b7e0 tx=0x7f412001b970 comp rx=0 tx=0).stop 2023-10-30T12:51:58.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.265+0000 7f4131b0f640 1 -- 172.21.15.89:0/1056867895 shutdown_connections 2023-10-30T12:51:58.269 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.265+0000 7f4131b0f640 1 --2- 172.21.15.89:0/1056867895 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f41180664a0 0x7f4118068960 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:58.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.265+0000 7f4131b0f640 1 --2- 172.21.15.89:0/1056867895 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f412c0c8430 0x7f412c132e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:58.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.265+0000 7f4131b0f640 1 --2- 172.21.15.89:0/1056867895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f412c0c6f60 0x7f412c132930 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:58.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.265+0000 7f4131b0f640 1 --2- 172.21.15.89:0/1056867895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f412c0c6b80 0x7f412c0c9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:58.270 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.265+0000 7f4131b0f640 1 -- 172.21.15.89:0/1056867895 >> 172.21.15.89:0/1056867895 conn(0x7f412c0c7910 msgr2=0x7f412c06b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:58.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.265+0000 7f4131b0f640 1 -- 172.21.15.89:0/1056867895 shutdown_connections 2023-10-30T12:51:58.271 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:51:58.265+0000 7f4131b0f640 1 -- 172.21.15.89:0/1056867895 wait complete. 2023-10-30T12:51:58.285 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":759,"stamp":"2023-10-30T12:51:57.438622+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1063412716,"num_objects":1371,"num_object_clones":0,"num_object_copies":2742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1371,"num_whiteouts":0,"num_read":4368,"num_read_kb":813164,"num_write":42615,"num_write_kb":1359244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":78,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":305728,"num_omap_keys":583,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47556,"ondisk_log_size":47556,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2620516,"kb_used_data":2087972,"kb_used_omap":12,"kb_used_meta":532467,"kb_avail":752354204,"statfs":{"total":773094113280,"available":770410704896,"internally_reserved":0,"allocated":2138083328,"data_stored":2129925143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12741,"internal_metadata":545246779},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":115503,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":227368,"num_omap_keys":434,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004349"},"pg_stats":[{"pgid":"3.3f","version":"66'551","reported_seq":2705,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:37.514715+0000","last_change":"2023-10-30T12:38:17.335461+0000","last_active":"2023-10-30T12:51:37.514715+0000","last_peered":"2023-10-30T12:51:37.514715+0000","last_clean":"2023-10-30T12:51:37.514715+0000","last_became_active":"2023-10-30T12:28:50.979351+0000","last_became_peered":"2023-10-30T12:28:50.979351+0000","last_unstale":"2023-10-30T12:51:37.514715+0000","last_undegraded":"2023-10-30T12:51:37.514715+0000","last_fullsized":"2023-10-30T12:51:37.514715+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":551,"log_dups_size":0,"ondisk_log_size":551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091708599999999994,"stat_sum":{"num_bytes":4321961,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":40,"num_read_kb":15,"num_write":493,"num_write_kb":5417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"66'551","reported_seq":2698,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:34.379188+0000","last_change":"2023-10-30T12:38:17.104663+0000","last_active":"2023-10-30T12:51:34.379188+0000","last_peered":"2023-10-30T12:51:34.379188+0000","last_clean":"2023-10-30T12:51:34.379188+0000","last_became_active":"2023-10-30T12:28:50.929133+0000","last_became_peered":"2023-10-30T12:28:50.929133+0000","last_unstale":"2023-10-30T12:51:34.379188+0000","last_undegraded":"2023-10-30T12:51:34.379188+0000","last_fullsized":"2023-10-30T12:51:34.379188+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":551,"log_dups_size":0,"ondisk_log_size":551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023149070000000002,"stat_sum":{"num_bytes":8426607,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":33,"num_read_kb":10,"num_write":489,"num_write_kb":9508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"66'550","reported_seq":2713,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:33.862537+0000","last_change":"2023-10-30T12:38:17.643537+0000","last_active":"2023-10-30T12:51:33.862537+0000","last_peered":"2023-10-30T12:51:33.862537+0000","last_clean":"2023-10-30T12:51:33.862537+0000","last_became_active":"2023-10-30T12:28:50.933896+0000","last_became_peered":"2023-10-30T12:28:50.933896+0000","last_unstale":"2023-10-30T12:51:33.862537+0000","last_undegraded":"2023-10-30T12:51:33.862537+0000","last_fullsized":"2023-10-30T12:51:33.862537+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":550,"log_dups_size":0,"ondisk_log_size":550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029033520000000001,"stat_sum":{"num_bytes":12697764,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":49,"num_read_kb":8209,"num_write":487,"num_write_kb":25783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"66'517","reported_seq":2661,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.473760+0000","last_change":"2023-10-30T12:38:17.117873+0000","last_active":"2023-10-30T12:51:30.473760+0000","last_peered":"2023-10-30T12:51:30.473760+0000","last_clean":"2023-10-30T12:51:30.473760+0000","last_became_active":"2023-10-30T12:28:50.967774+0000","last_became_peered":"2023-10-30T12:28:50.967774+0000","last_unstale":"2023-10-30T12:51:30.473760+0000","last_undegraded":"2023-10-30T12:51:30.473760+0000","last_fullsized":"2023-10-30T12:51:30.473760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075479680000000004,"stat_sum":{"num_bytes":4272818,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":30,"num_read_kb":13,"num_write":479,"num_write_kb":5884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"66'553","reported_seq":2720,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:39.811881+0000","last_change":"2023-10-30T12:51:39.812135+0000","last_active":"2023-10-30T12:51:39.811881+0000","last_peered":"2023-10-30T12:51:39.811881+0000","last_clean":"2023-10-30T12:51:39.811881+0000","last_became_active":"2023-10-30T12:28:50.973132+0000","last_became_peered":"2023-10-30T12:28:50.973132+0000","last_unstale":"2023-10-30T12:51:39.811881+0000","last_undegraded":"2023-10-30T12:51:39.811881+0000","last_fullsized":"2023-10-30T12:51:39.811881+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'553","last_scrub_stamp":"2023-10-30T12:51:39.811639+0000","last_deep_scrub":"66'553","last_deep_scrub_stamp":"2023-10-30T12:51:39.811639+0000","last_clean_scrub_stamp":"2023-10-30T12:51:39.811639+0000","objects_scrubbed":18,"log_size":553,"log_dups_size":0,"ondisk_log_size":553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:25:28.276273+0000","scrub_duration":0.055821471999999997,"objects_trimmed":0,"snaptrim_duration":0.0090421609999999999,"stat_sum":{"num_bytes":12692456,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":47,"num_read_kb":16,"num_write":494,"num_write_kb":14525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"66'551","reported_seq":2701,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.474891+0000","last_change":"2023-10-30T12:38:17.112051+0000","last_active":"2023-10-30T12:51:30.474891+0000","last_peered":"2023-10-30T12:51:30.474891+0000","last_clean":"2023-10-30T12:51:30.474891+0000","last_became_active":"2023-10-30T12:28:50.967910+0000","last_became_peered":"2023-10-30T12:28:50.967910+0000","last_unstale":"2023-10-30T12:51:30.474891+0000","last_undegraded":"2023-10-30T12:51:30.474891+0000","last_fullsized":"2023-10-30T12:51:30.474891+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":551,"log_dups_size":0,"ondisk_log_size":551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040413009999999997,"stat_sum":{"num_bytes":12623095,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":38,"num_read_kb":16,"num_write":503,"num_write_kb":14521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"66'569","reported_seq":2730,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:55.514716+0000","last_change":"2023-10-30T12:51:55.514849+0000","last_active":"2023-10-30T12:51:55.514716+0000","last_peered":"2023-10-30T12:51:55.514716+0000","last_clean":"2023-10-30T12:51:55.514716+0000","last_became_active":"2023-10-30T12:28:50.973288+0000","last_became_peered":"2023-10-30T12:28:50.973288+0000","last_unstale":"2023-10-30T12:51:55.514716+0000","last_undegraded":"2023-10-30T12:51:55.514716+0000","last_fullsized":"2023-10-30T12:51:55.514716+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'569","last_scrub_stamp":"2023-10-30T12:51:55.514522+0000","last_deep_scrub":"66'569","last_deep_scrub_stamp":"2023-10-30T12:51:55.514522+0000","last_clean_scrub_stamp":"2023-10-30T12:51:55.514522+0000","objects_scrubbed":18,"log_size":569,"log_dups_size":0,"ondisk_log_size":569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:36.003579+0000","scrub_duration":0.043916595000000003,"objects_trimmed":0,"snaptrim_duration":0.0037299339999999999,"stat_sum":{"num_bytes":4302112,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":41,"num_read_kb":11,"num_write":510,"num_write_kb":7154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"66'500","reported_seq":2651,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:31.291065+0000","last_change":"2023-10-30T12:38:17.330519+0000","last_active":"2023-10-30T12:51:31.291065+0000","last_peered":"2023-10-30T12:51:31.291065+0000","last_clean":"2023-10-30T12:51:31.291065+0000","last_became_active":"2023-10-30T12:28:50.954988+0000","last_became_peered":"2023-10-30T12:28:50.954988+0000","last_unstale":"2023-10-30T12:51:31.291065+0000","last_undegraded":"2023-10-30T12:51:31.291065+0000","last_fullsized":"2023-10-30T12:51:31.291065+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":500,"log_dups_size":0,"ondisk_log_size":500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028962670000000001,"stat_sum":{"num_bytes":8424944,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":39,"num_read_kb":15,"num_write":456,"num_write_kb":10248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"66'573","reported_seq":2734,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.573146+0000","last_change":"2023-10-30T12:38:17.336435+0000","last_active":"2023-10-30T12:51:30.573146+0000","last_peered":"2023-10-30T12:51:30.573146+0000","last_clean":"2023-10-30T12:51:30.573146+0000","last_became_active":"2023-10-30T12:28:50.976564+0000","last_became_peered":"2023-10-30T12:28:50.976564+0000","last_unstale":"2023-10-30T12:51:30.573146+0000","last_undegraded":"2023-10-30T12:51:30.573146+0000","last_fullsized":"2023-10-30T12:51:30.573146+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":573,"log_dups_size":0,"ondisk_log_size":573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0094697280000000002,"stat_sum":{"num_bytes":16840394,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":49,"num_read_kb":4116,"num_write":513,"num_write_kb":27605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"66'546","reported_seq":2717,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.460931+0000","last_change":"2023-10-30T12:38:17.327384+0000","last_active":"2023-10-30T12:51:30.460931+0000","last_peered":"2023-10-30T12:51:30.460931+0000","last_clean":"2023-10-30T12:51:30.460931+0000","last_became_active":"2023-10-30T12:28:50.951809+0000","last_became_peered":"2023-10-30T12:28:50.951809+0000","last_unstale":"2023-10-30T12:51:30.460931+0000","last_undegraded":"2023-10-30T12:51:30.460931+0000","last_fullsized":"2023-10-30T12:51:30.460931+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":546,"log_dups_size":0,"ondisk_log_size":546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0078740100000000007,"stat_sum":{"num_bytes":16924216,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":59,"num_read_kb":29,"num_write":493,"num_write_kb":22764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"66'577","reported_seq":2742,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.404446+0000","last_change":"2023-10-30T12:38:17.323179+0000","last_active":"2023-10-30T12:51:30.404446+0000","last_peered":"2023-10-30T12:51:30.404446+0000","last_clean":"2023-10-30T12:51:30.404446+0000","last_became_active":"2023-10-30T12:29:32.958473+0000","last_became_peered":"2023-10-30T12:29:32.958473+0000","last_unstale":"2023-10-30T12:51:30.404446+0000","last_undegraded":"2023-10-30T12:51:30.404446+0000","last_fullsized":"2023-10-30T12:51:30.404446+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":577,"log_dups_size":0,"ondisk_log_size":577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045626240000000004,"stat_sum":{"num_bytes":12721502,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":46,"num_read_kb":17,"num_write":533,"num_write_kb":14913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"66'526","reported_seq":2680,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:43.156958+0000","last_change":"2023-10-30T12:51:43.157088+0000","last_active":"2023-10-30T12:51:43.156958+0000","last_peered":"2023-10-30T12:51:43.156958+0000","last_clean":"2023-10-30T12:51:43.156958+0000","last_became_active":"2023-10-30T12:28:50.943787+0000","last_became_peered":"2023-10-30T12:28:50.943787+0000","last_unstale":"2023-10-30T12:51:43.156958+0000","last_undegraded":"2023-10-30T12:51:43.156958+0000","last_fullsized":"2023-10-30T12:51:43.156958+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'526","last_scrub_stamp":"2023-10-30T12:51:43.156744+0000","last_deep_scrub":"66'526","last_deep_scrub_stamp":"2023-10-30T12:51:43.156744+0000","last_clean_scrub_stamp":"2023-10-30T12:51:43.156744+0000","objects_scrubbed":20,"log_size":526,"log_dups_size":0,"ondisk_log_size":526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:59:42.865066+0000","scrub_duration":0.057927502999999998,"objects_trimmed":0,"snaptrim_duration":0.0047342360000000002,"stat_sum":{"num_bytes":12634629,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":34,"num_read_kb":4115,"num_write":467,"num_write_kb":18795,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"66'600","reported_seq":2762,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.405860+0000","last_change":"2023-10-30T12:38:17.331215+0000","last_active":"2023-10-30T12:51:30.405860+0000","last_peered":"2023-10-30T12:51:30.405860+0000","last_clean":"2023-10-30T12:51:30.405860+0000","last_became_active":"2023-10-30T12:28:50.937906+0000","last_became_peered":"2023-10-30T12:28:50.937906+0000","last_unstale":"2023-10-30T12:51:30.405860+0000","last_undegraded":"2023-10-30T12:51:30.405860+0000","last_fullsized":"2023-10-30T12:51:30.405860+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":600,"log_dups_size":0,"ondisk_log_size":600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003369164,"stat_sum":{"num_bytes":4268758,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":50,"num_read_kb":8215,"num_write":547,"num_write_kb":14586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"66'554","reported_seq":2722,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:46.367606+0000","last_change":"2023-10-30T12:51:46.367652+0000","last_active":"2023-10-30T12:51:46.367606+0000","last_peered":"2023-10-30T12:51:46.367606+0000","last_clean":"2023-10-30T12:51:46.367606+0000","last_became_active":"2023-10-30T12:28:50.932753+0000","last_became_peered":"2023-10-30T12:28:50.932753+0000","last_unstale":"2023-10-30T12:51:46.367606+0000","last_undegraded":"2023-10-30T12:51:46.367606+0000","last_fullsized":"2023-10-30T12:51:46.367606+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'554","last_scrub_stamp":"2023-10-30T12:51:46.367517+0000","last_deep_scrub":"66'554","last_deep_scrub_stamp":"2023-10-30T12:51:46.367517+0000","last_clean_scrub_stamp":"2023-10-30T12:51:46.367517+0000","objects_scrubbed":25,"log_size":554,"log_dups_size":0,"ondisk_log_size":554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:15:04.516611+0000","scrub_duration":0.047742825000000003,"objects_trimmed":0,"snaptrim_duration":0.0027063,"stat_sum":{"num_bytes":16858869,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":47,"num_read_kb":17,"num_write":493,"num_write_kb":19132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"66'580","reported_seq":2750,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:49.898201+0000","last_change":"2023-10-30T12:51:49.898263+0000","last_active":"2023-10-30T12:51:49.898201+0000","last_peered":"2023-10-30T12:51:49.898201+0000","last_clean":"2023-10-30T12:51:49.898201+0000","last_became_active":"2023-10-30T12:28:50.943011+0000","last_became_peered":"2023-10-30T12:28:50.943011+0000","last_unstale":"2023-10-30T12:51:49.898201+0000","last_undegraded":"2023-10-30T12:51:49.898201+0000","last_fullsized":"2023-10-30T12:51:49.898201+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'580","last_scrub_stamp":"2023-10-30T12:51:49.898122+0000","last_deep_scrub":"66'580","last_deep_scrub_stamp":"2023-10-30T12:51:49.898122+0000","last_clean_scrub_stamp":"2023-10-30T12:51:49.898122+0000","objects_scrubbed":21,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:56:15.625962+0000","scrub_duration":0.069989635999999994,"objects_trimmed":0,"snaptrim_duration":0.001813863,"stat_sum":{"num_bytes":21155803,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":51,"num_read_kb":4115,"num_write":508,"num_write_kb":22928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"66'566","reported_seq":2724,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.471687+0000","last_change":"2023-10-30T12:38:17.111689+0000","last_active":"2023-10-30T12:51:30.471687+0000","last_peered":"2023-10-30T12:51:30.471687+0000","last_clean":"2023-10-30T12:51:30.471687+0000","last_became_active":"2023-10-30T12:28:50.952744+0000","last_became_peered":"2023-10-30T12:28:50.952744+0000","last_unstale":"2023-10-30T12:51:30.471687+0000","last_undegraded":"2023-10-30T12:51:30.471687+0000","last_fullsized":"2023-10-30T12:51:30.471687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":566,"log_dups_size":0,"ondisk_log_size":566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034371530000000001,"stat_sum":{"num_bytes":4262714,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":46,"num_read_kb":18,"num_write":521,"num_write_kb":9441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"66'590","reported_seq":2742,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:31.170634+0000","last_change":"2023-10-30T12:38:17.120572+0000","last_active":"2023-10-30T12:51:31.170634+0000","last_peered":"2023-10-30T12:51:31.170634+0000","last_clean":"2023-10-30T12:51:31.170634+0000","last_became_active":"2023-10-30T12:28:50.974576+0000","last_became_peered":"2023-10-30T12:28:50.974576+0000","last_unstale":"2023-10-30T12:51:31.170634+0000","last_undegraded":"2023-10-30T12:51:31.170634+0000","last_fullsized":"2023-10-30T12:51:31.170634+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":590,"log_dups_size":0,"ondisk_log_size":590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080003049999999992,"stat_sum":{"num_bytes":8462359,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":40,"num_read_kb":4113,"num_write":531,"num_write_kb":16554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"66'543","reported_seq":2721,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:46.554437+0000","last_change":"2023-10-30T12:51:46.554500+0000","last_active":"2023-10-30T12:51:46.554437+0000","last_peered":"2023-10-30T12:51:46.554437+0000","last_clean":"2023-10-30T12:51:46.554437+0000","last_became_active":"2023-10-30T12:28:50.937384+0000","last_became_peered":"2023-10-30T12:28:50.937384+0000","last_unstale":"2023-10-30T12:51:46.554437+0000","last_undegraded":"2023-10-30T12:51:46.554437+0000","last_fullsized":"2023-10-30T12:51:46.554437+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'543","last_scrub_stamp":"2023-10-30T12:51:46.554353+0000","last_deep_scrub":"66'543","last_deep_scrub_stamp":"2023-10-30T12:51:46.554353+0000","last_clean_scrub_stamp":"2023-10-30T12:51:46.554353+0000","objects_scrubbed":16,"log_size":543,"log_dups_size":0,"ondisk_log_size":543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:23:08.192334+0000","scrub_duration":0.042828463999999997,"objects_trimmed":0,"snaptrim_duration":0.001162803,"stat_sum":{"num_bytes":8456287,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":58,"num_read_kb":27,"num_write":488,"num_write_kb":9441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"66'554","reported_seq":2722,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.507253+0000","last_change":"2023-10-30T12:38:17.336198+0000","last_active":"2023-10-30T12:51:30.507253+0000","last_peered":"2023-10-30T12:51:30.507253+0000","last_clean":"2023-10-30T12:51:30.507253+0000","last_became_active":"2023-10-30T12:28:50.973369+0000","last_became_peered":"2023-10-30T12:28:50.973369+0000","last_unstale":"2023-10-30T12:51:30.507253+0000","last_undegraded":"2023-10-30T12:51:30.507253+0000","last_fullsized":"2023-10-30T12:51:30.507253+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":554,"log_dups_size":0,"ondisk_log_size":554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093923919999999994,"stat_sum":{"num_bytes":12689191,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":55,"num_read_kb":4114,"num_write":505,"num_write_kb":18071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"66'592","reported_seq":2762,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:47.730012+0000","last_change":"2023-10-30T12:51:47.730137+0000","last_active":"2023-10-30T12:51:47.730012+0000","last_peered":"2023-10-30T12:51:47.730012+0000","last_clean":"2023-10-30T12:51:47.730012+0000","last_became_active":"2023-10-30T12:28:50.954831+0000","last_became_peered":"2023-10-30T12:28:50.954831+0000","last_unstale":"2023-10-30T12:51:47.730012+0000","last_undegraded":"2023-10-30T12:51:47.730012+0000","last_fullsized":"2023-10-30T12:51:47.730012+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'592","last_scrub_stamp":"2023-10-30T12:51:47.729828+0000","last_deep_scrub":"66'592","last_deep_scrub_stamp":"2023-10-30T12:51:47.729828+0000","last_clean_scrub_stamp":"2023-10-30T12:51:47.729828+0000","objects_scrubbed":20,"log_size":592,"log_dups_size":0,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:14:47.264586+0000","scrub_duration":0.089463355999999994,"objects_trimmed":0,"snaptrim_duration":0.007119116,"stat_sum":{"num_bytes":21055454,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":49,"num_read_kb":4113,"num_write":524,"num_write_kb":30367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"66'532","reported_seq":2686,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.465812+0000","last_change":"2023-10-30T12:38:17.644608+0000","last_active":"2023-10-30T12:51:30.465812+0000","last_peered":"2023-10-30T12:51:30.465812+0000","last_clean":"2023-10-30T12:51:30.465812+0000","last_became_active":"2023-10-30T12:28:50.962065+0000","last_became_peered":"2023-10-30T12:28:50.962065+0000","last_unstale":"2023-10-30T12:51:30.465812+0000","last_undegraded":"2023-10-30T12:51:30.465812+0000","last_fullsized":"2023-10-30T12:51:30.465812+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":532,"log_dups_size":0,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037739459999999998,"stat_sum":{"num_bytes":25200014,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":41,"num_read_kb":21,"num_write":490,"num_write_kb":26352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"66'608","reported_seq":2767,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.506121+0000","last_change":"2023-10-30T12:38:17.118400+0000","last_active":"2023-10-30T12:51:30.506121+0000","last_peered":"2023-10-30T12:51:30.506121+0000","last_clean":"2023-10-30T12:51:30.506121+0000","last_became_active":"2023-10-30T12:28:50.967651+0000","last_became_peered":"2023-10-30T12:28:50.967651+0000","last_unstale":"2023-10-30T12:51:30.506121+0000","last_undegraded":"2023-10-30T12:51:30.506121+0000","last_fullsized":"2023-10-30T12:51:30.506121+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":608,"log_dups_size":0,"ondisk_log_size":608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081085610000000002,"stat_sum":{"num_bytes":8425872,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":46,"num_read_kb":21,"num_write":555,"num_write_kb":13596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"66'560","reported_seq":2723,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.484231+0000","last_change":"2023-10-30T12:38:17.332602+0000","last_active":"2023-10-30T12:51:30.484231+0000","last_peered":"2023-10-30T12:51:30.484231+0000","last_clean":"2023-10-30T12:51:30.484231+0000","last_became_active":"2023-10-30T12:28:50.962290+0000","last_became_peered":"2023-10-30T12:28:50.962290+0000","last_unstale":"2023-10-30T12:51:30.484231+0000","last_undegraded":"2023-10-30T12:51:30.484231+0000","last_fullsized":"2023-10-30T12:51:30.484231+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":560,"log_dups_size":0,"ondisk_log_size":560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042026670000000002,"stat_sum":{"num_bytes":195030,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":50,"num_read_kb":4117,"num_write":509,"num_write_kb":6258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"66'575","reported_seq":2728,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:31.170566+0000","last_change":"2023-10-30T12:38:17.114285+0000","last_active":"2023-10-30T12:51:31.170566+0000","last_peered":"2023-10-30T12:51:31.170566+0000","last_clean":"2023-10-30T12:51:31.170566+0000","last_became_active":"2023-10-30T12:28:50.965713+0000","last_became_peered":"2023-10-30T12:28:50.965713+0000","last_unstale":"2023-10-30T12:51:31.170566+0000","last_undegraded":"2023-10-30T12:51:31.170566+0000","last_fullsized":"2023-10-30T12:51:31.170566+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":575,"log_dups_size":0,"ondisk_log_size":575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056034209999999999,"stat_sum":{"num_bytes":12659519,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":40,"num_read_kb":16,"num_write":528,"num_write_kb":18837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"66'533","reported_seq":2677,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.455876+0000","last_change":"2023-10-30T12:38:17.330864+0000","last_active":"2023-10-30T12:51:30.455876+0000","last_peered":"2023-10-30T12:51:30.455876+0000","last_clean":"2023-10-30T12:51:30.455876+0000","last_became_active":"2023-10-30T12:28:50.951026+0000","last_became_peered":"2023-10-30T12:28:50.951026+0000","last_unstale":"2023-10-30T12:51:30.455876+0000","last_undegraded":"2023-10-30T12:51:30.455876+0000","last_fullsized":"2023-10-30T12:51:30.455876+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":533,"log_dups_size":0,"ondisk_log_size":533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031697840000000001,"stat_sum":{"num_bytes":12597918,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":31,"num_read_kb":12,"num_write":481,"num_write_kb":19629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"66'607","reported_seq":2775,"reported_epoch":66,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-30T12:51:52.524859+0000","last_change":"2023-10-30T12:51:52.523378+0000","last_active":"2023-10-30T12:51:52.524859+0000","last_peered":"2023-10-30T12:51:52.524859+0000","last_clean":"2023-10-30T12:51:52.524859+0000","last_became_active":"2023-10-30T12:28:50.962484+0000","last_became_peered":"2023-10-30T12:28:50.962484+0000","last_unstale":"2023-10-30T12:51:52.524859+0000","last_undegraded":"2023-10-30T12:51:52.524859+0000","last_fullsized":"2023-10-30T12:51:52.524859+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":607,"log_dups_size":0,"ondisk_log_size":607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083773500000000004,"stat_sum":{"num_bytes":12675985,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":51,"num_read_kb":23,"num_write":557,"num_write_kb":15080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"66'532","reported_seq":2695,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:51.872361+0000","last_change":"2023-10-30T12:51:51.872430+0000","last_active":"2023-10-30T12:51:51.872361+0000","last_peered":"2023-10-30T12:51:51.872361+0000","last_clean":"2023-10-30T12:51:51.872361+0000","last_became_active":"2023-10-30T12:28:50.933093+0000","last_became_peered":"2023-10-30T12:28:50.933093+0000","last_unstale":"2023-10-30T12:51:51.872361+0000","last_undegraded":"2023-10-30T12:51:51.872361+0000","last_fullsized":"2023-10-30T12:51:51.872361+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'532","last_scrub_stamp":"2023-10-30T12:51:51.872277+0000","last_deep_scrub":"66'532","last_deep_scrub_stamp":"2023-10-30T12:51:51.872277+0000","last_clean_scrub_stamp":"2023-10-30T12:51:51.872277+0000","objects_scrubbed":22,"log_size":532,"log_dups_size":0,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:06:03.858620+0000","scrub_duration":0.057177436999999998,"objects_trimmed":0,"snaptrim_duration":0.0022251829999999999,"stat_sum":{"num_bytes":25274091,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":42,"num_read_kb":8207,"num_write":449,"num_write_kb":27548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"66'499","reported_seq":2638,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.474390+0000","last_change":"2023-10-30T12:38:17.332060+0000","last_active":"2023-10-30T12:51:30.474390+0000","last_peered":"2023-10-30T12:51:30.474390+0000","last_clean":"2023-10-30T12:51:30.474390+0000","last_became_active":"2023-10-30T12:28:50.952531+0000","last_became_peered":"2023-10-30T12:28:50.952531+0000","last_unstale":"2023-10-30T12:51:30.474390+0000","last_undegraded":"2023-10-30T12:51:30.474390+0000","last_fullsized":"2023-10-30T12:51:30.474390+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":499,"log_dups_size":0,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039301029999999999,"stat_sum":{"num_bytes":4373272,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":26,"num_read_kb":3284,"num_write":458,"num_write_kb":14265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"66'523","reported_seq":2682,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:33.862521+0000","last_change":"2023-10-30T12:38:17.644463+0000","last_active":"2023-10-30T12:51:33.862521+0000","last_peered":"2023-10-30T12:51:33.862521+0000","last_clean":"2023-10-30T12:51:33.862521+0000","last_became_active":"2023-10-30T12:28:50.933124+0000","last_became_peered":"2023-10-30T12:28:50.933124+0000","last_unstale":"2023-10-30T12:51:33.862521+0000","last_undegraded":"2023-10-30T12:51:33.862521+0000","last_fullsized":"2023-10-30T12:51:33.862521+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":523,"log_dups_size":0,"ondisk_log_size":523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034752709999999998,"stat_sum":{"num_bytes":8490168,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":45,"num_read_kb":14,"num_write":469,"num_write_kb":10488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"66'594","reported_seq":2759,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:36.467890+0000","last_change":"2023-10-30T12:38:17.118502+0000","last_active":"2023-10-30T12:51:36.467890+0000","last_peered":"2023-10-30T12:51:36.467890+0000","last_clean":"2023-10-30T12:51:36.467890+0000","last_became_active":"2023-10-30T12:28:50.965639+0000","last_became_peered":"2023-10-30T12:28:50.965639+0000","last_unstale":"2023-10-30T12:51:36.467890+0000","last_undegraded":"2023-10-30T12:51:36.467890+0000","last_fullsized":"2023-10-30T12:51:36.467890+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":594,"log_dups_size":0,"ondisk_log_size":594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076861280000000004,"stat_sum":{"num_bytes":8432593,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":51,"num_read_kb":20,"num_write":547,"num_write_kb":11509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"66'595","reported_seq":2767,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:45.496340+0000","last_change":"2023-10-30T12:51:45.496340+0000","last_active":"2023-10-30T12:51:45.496340+0000","last_peered":"2023-10-30T12:51:45.496340+0000","last_clean":"2023-10-30T12:51:45.496340+0000","last_became_active":"2023-10-30T12:28:50.955084+0000","last_became_peered":"2023-10-30T12:28:50.955084+0000","last_unstale":"2023-10-30T12:51:45.496340+0000","last_undegraded":"2023-10-30T12:51:45.496340+0000","last_fullsized":"2023-10-30T12:51:45.496340+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'595","last_scrub_stamp":"2023-10-30T12:51:45.496211+0000","last_deep_scrub":"66'595","last_deep_scrub_stamp":"2023-10-30T12:51:45.496211+0000","last_clean_scrub_stamp":"2023-10-30T12:51:45.496211+0000","objects_scrubbed":12,"log_size":595,"log_dups_size":0,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:13:37.745803+0000","scrub_duration":0.027110084,"objects_trimmed":0,"snaptrim_duration":0.0012043119999999999,"stat_sum":{"num_bytes":4257179,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":50,"num_read_kb":22,"num_write":532,"num_write_kb":9837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"66'557","reported_seq":2724,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:36.988290+0000","last_change":"2023-10-30T12:38:17.326196+0000","last_active":"2023-10-30T12:51:36.988290+0000","last_peered":"2023-10-30T12:51:36.988290+0000","last_clean":"2023-10-30T12:51:36.988290+0000","last_became_active":"2023-10-30T12:28:50.944775+0000","last_became_peered":"2023-10-30T12:28:50.944775+0000","last_unstale":"2023-10-30T12:51:36.988290+0000","last_undegraded":"2023-10-30T12:51:36.988290+0000","last_fullsized":"2023-10-30T12:51:36.988290+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070573429999999998,"stat_sum":{"num_bytes":12763889,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":53,"num_read_kb":21,"num_write":514,"num_write_kb":13736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"66'590","reported_seq":2750,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:44.129343+0000","last_change":"2023-10-30T12:51:44.129472+0000","last_active":"2023-10-30T12:51:44.129343+0000","last_peered":"2023-10-30T12:51:44.129343+0000","last_clean":"2023-10-30T12:51:44.129343+0000","last_became_active":"2023-10-30T12:28:50.940567+0000","last_became_peered":"2023-10-30T12:28:50.940567+0000","last_unstale":"2023-10-30T12:51:44.129343+0000","last_undegraded":"2023-10-30T12:51:44.129343+0000","last_fullsized":"2023-10-30T12:51:44.129343+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'590","last_scrub_stamp":"2023-10-30T12:51:44.129123+0000","last_deep_scrub":"66'590","last_deep_scrub_stamp":"2023-10-30T12:51:44.129123+0000","last_clean_scrub_stamp":"2023-10-30T12:51:44.129123+0000","objects_scrubbed":10,"log_size":590,"log_dups_size":0,"ondisk_log_size":590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:25:14.156000+0000","scrub_duration":0.048555811999999997,"objects_trimmed":0,"snaptrim_duration":0.0028523340000000002,"stat_sum":{"num_bytes":9696728,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":17,"num_write":529,"num_write_kb":12781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"66'283","reported_seq":371,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:45.780003+0000","last_change":"2023-10-30T12:51:45.780059+0000","last_active":"2023-10-30T12:51:45.780003+0000","last_peered":"2023-10-30T12:51:45.780003+0000","last_clean":"2023-10-30T12:51:45.780003+0000","last_became_active":"2023-10-30T12:28:50.395382+0000","last_became_peered":"2023-10-30T12:28:50.395382+0000","last_unstale":"2023-10-30T12:51:45.780003+0000","last_undegraded":"2023-10-30T12:51:45.780003+0000","last_fullsized":"2023-10-30T12:51:45.780003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'283","last_scrub_stamp":"2023-10-30T12:51:45.779902+0000","last_deep_scrub":"66'283","last_deep_scrub_stamp":"2023-10-30T12:51:45.779902+0000","last_clean_scrub_stamp":"2023-10-30T12:51:45.779902+0000","objects_scrubbed":5,"log_size":283,"log_dups_size":0,"ondisk_log_size":283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:36:15.846905+0000","scrub_duration":0.042332846,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582946,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":45059,"num_write":213,"num_write_kb":12413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8500,"num_omap_keys":17,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"66'511","reported_seq":2676,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:45.384937+0000","last_change":"2023-10-30T12:51:45.385000+0000","last_active":"2023-10-30T12:51:45.384937+0000","last_peered":"2023-10-30T12:51:45.384937+0000","last_clean":"2023-10-30T12:51:45.384937+0000","last_became_active":"2023-10-30T12:28:50.952613+0000","last_became_peered":"2023-10-30T12:28:50.952613+0000","last_unstale":"2023-10-30T12:51:45.384937+0000","last_undegraded":"2023-10-30T12:51:45.384937+0000","last_fullsized":"2023-10-30T12:51:45.384937+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'511","last_scrub_stamp":"2023-10-30T12:51:45.384858+0000","last_deep_scrub":"66'511","last_deep_scrub_stamp":"2023-10-30T12:51:45.384858+0000","last_clean_scrub_stamp":"2023-10-30T12:51:45.384858+0000","objects_scrubbed":19,"log_size":511,"log_dups_size":0,"ondisk_log_size":511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:37:21.237208+0000","scrub_duration":0.069355456999999995,"objects_trimmed":0,"snaptrim_duration":0.0038505420000000002,"stat_sum":{"num_bytes":21061321,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":45,"num_read_kb":18,"num_write":449,"num_write_kb":22047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"66'909","reported_seq":1201,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:49.177081+0000","last_change":"2023-10-30T12:51:42.662808+0000","last_active":"2023-10-30T12:51:49.177081+0000","last_peered":"2023-10-30T12:51:49.177081+0000","last_clean":"2023-10-30T12:51:49.177081+0000","last_became_active":"2023-10-30T12:28:50.383898+0000","last_became_peered":"2023-10-30T12:28:50.383898+0000","last_unstale":"2023-10-30T12:51:49.177081+0000","last_undegraded":"2023-10-30T12:51:49.177081+0000","last_fullsized":"2023-10-30T12:51:49.177081+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'908","last_scrub_stamp":"2023-10-30T12:51:42.662429+0000","last_deep_scrub":"66'908","last_deep_scrub_stamp":"2023-10-30T12:51:42.662429+0000","last_clean_scrub_stamp":"2023-10-30T12:51:42.662429+0000","objects_scrubbed":9,"log_size":909,"log_dups_size":0,"ondisk_log_size":909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:01:46.593761+0000","scrub_duration":0.098226147999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21567319,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":234,"num_read_kb":49886,"num_write":842,"num_write_kb":21597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":23432,"num_omap_keys":43,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"66'457","reported_seq":759,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:48.571287+0000","last_change":"2023-10-30T12:51:41.780215+0000","last_active":"2023-10-30T12:51:48.571287+0000","last_peered":"2023-10-30T12:51:48.571287+0000","last_clean":"2023-10-30T12:51:48.571287+0000","last_became_active":"2023-10-30T12:28:50.385963+0000","last_became_peered":"2023-10-30T12:28:50.385963+0000","last_unstale":"2023-10-30T12:51:48.571287+0000","last_undegraded":"2023-10-30T12:51:48.571287+0000","last_fullsized":"2023-10-30T12:51:48.571287+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'454","last_scrub_stamp":"2023-10-30T12:51:41.780079+0000","last_deep_scrub":"66'454","last_deep_scrub_stamp":"2023-10-30T12:51:41.780079+0000","last_clean_scrub_stamp":"2023-10-30T12:51:41.780079+0000","objects_scrubbed":4,"log_size":457,"log_dups_size":0,"ondisk_log_size":457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:29:54.800743+0000","scrub_duration":0.030692146999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8006399,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":240,"num_read_kb":9095,"num_write":389,"num_write_kb":8081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"66'639","reported_seq":2815,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:51.809783+0000","last_change":"2023-10-30T12:51:51.809888+0000","last_active":"2023-10-30T12:51:51.809783+0000","last_peered":"2023-10-30T12:51:51.809783+0000","last_clean":"2023-10-30T12:51:51.809783+0000","last_became_active":"2023-10-30T12:28:50.935764+0000","last_became_peered":"2023-10-30T12:28:50.935764+0000","last_unstale":"2023-10-30T12:51:51.809783+0000","last_undegraded":"2023-10-30T12:51:51.809783+0000","last_fullsized":"2023-10-30T12:51:51.809783+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'639","last_scrub_stamp":"2023-10-30T12:51:51.809674+0000","last_deep_scrub":"66'639","last_deep_scrub_stamp":"2023-10-30T12:51:51.809674+0000","last_clean_scrub_stamp":"2023-10-30T12:51:51.809674+0000","objects_scrubbed":21,"log_size":639,"log_dups_size":0,"ondisk_log_size":639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:53.793939+0000","scrub_duration":0.025406960999999999,"objects_trimmed":0,"snaptrim_duration":0.0031329320000000002,"stat_sum":{"num_bytes":4280685,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":56,"num_read_kb":4120,"num_write":569,"num_write_kb":13778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"66'571","reported_seq":655,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:43.489239+0000","last_change":"2023-10-30T12:51:43.489353+0000","last_active":"2023-10-30T12:51:43.489239+0000","last_peered":"2023-10-30T12:51:43.489239+0000","last_clean":"2023-10-30T12:51:43.489239+0000","last_became_active":"2023-10-30T12:28:50.391494+0000","last_became_peered":"2023-10-30T12:28:50.391494+0000","last_unstale":"2023-10-30T12:51:43.489239+0000","last_undegraded":"2023-10-30T12:51:43.489239+0000","last_fullsized":"2023-10-30T12:51:43.489239+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'571","last_scrub_stamp":"2023-10-30T12:51:43.489033+0000","last_deep_scrub":"66'571","last_deep_scrub_stamp":"2023-10-30T12:51:43.489033+0000","last_clean_scrub_stamp":"2023-10-30T12:51:43.489033+0000","objects_scrubbed":3,"log_size":571,"log_dups_size":0,"ondisk_log_size":571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:24:33.871160+0000","scrub_duration":0.058779308000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":36864,"num_write":519,"num_write_kb":12589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"66'592","reported_seq":2750,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.380764+0000","last_change":"2023-10-30T12:38:17.103988+0000","last_active":"2023-10-30T12:51:30.380764+0000","last_peered":"2023-10-30T12:51:30.380764+0000","last_clean":"2023-10-30T12:51:30.380764+0000","last_became_active":"2023-10-30T12:28:50.933062+0000","last_became_peered":"2023-10-30T12:28:50.933062+0000","last_unstale":"2023-10-30T12:51:30.380764+0000","last_undegraded":"2023-10-30T12:51:30.380764+0000","last_fullsized":"2023-10-30T12:51:30.380764+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":592,"log_dups_size":0,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021140489999999998,"stat_sum":{"num_bytes":33653289,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":46,"num_read_kb":4115,"num_write":549,"num_write_kb":39173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"66'132","reported_seq":214,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:46.616340+0000","last_change":"2023-10-30T12:51:46.616340+0000","last_active":"2023-10-30T12:51:46.616340+0000","last_peered":"2023-10-30T12:51:46.616340+0000","last_clean":"2023-10-30T12:51:46.616340+0000","last_became_active":"2023-10-30T12:28:50.393686+0000","last_became_peered":"2023-10-30T12:28:50.393686+0000","last_unstale":"2023-10-30T12:51:46.616340+0000","last_undegraded":"2023-10-30T12:51:46.616340+0000","last_fullsized":"2023-10-30T12:51:46.616340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'131","last_scrub_stamp":"2023-10-30T12:51:46.615853+0000","last_deep_scrub":"66'131","last_deep_scrub_stamp":"2023-10-30T12:51:46.615853+0000","last_clean_scrub_stamp":"2023-10-30T12:51:46.615853+0000","objects_scrubbed":3,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:42:31.808839+0000","scrub_duration":0.0095666550000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":75,"num_write_kb":93,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9282,"num_omap_keys":17,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"66'567","reported_seq":2730,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.467047+0000","last_change":"2023-10-30T12:38:17.119770+0000","last_active":"2023-10-30T12:51:30.467047+0000","last_peered":"2023-10-30T12:51:30.467047+0000","last_clean":"2023-10-30T12:51:30.467047+0000","last_became_active":"2023-10-30T12:28:50.963105+0000","last_became_peered":"2023-10-30T12:28:50.963105+0000","last_unstale":"2023-10-30T12:51:30.467047+0000","last_undegraded":"2023-10-30T12:51:30.467047+0000","last_fullsized":"2023-10-30T12:51:30.467047+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":567,"log_dups_size":0,"ondisk_log_size":567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080066809999999999,"stat_sum":{"num_bytes":16920090,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":51,"num_read_kb":18,"num_write":515,"num_write_kb":18479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"66'281","reported_seq":355,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:24.397165+0000","last_change":"2023-10-30T12:28:50.397679+0000","last_active":"2023-10-30T12:51:24.397165+0000","last_peered":"2023-10-30T12:51:24.397165+0000","last_clean":"2023-10-30T12:51:24.397165+0000","last_became_active":"2023-10-30T12:28:50.397041+0000","last_became_peered":"2023-10-30T12:28:50.397041+0000","last_unstale":"2023-10-30T12:51:24.397165+0000","last_undegraded":"2023-10-30T12:51:24.397165+0000","last_fullsized":"2023-10-30T12:51:24.397165+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":16396,"num_write":240,"num_write_kb":8312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"66'548","reported_seq":2702,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:31.170135+0000","last_change":"2023-10-30T12:38:17.644799+0000","last_active":"2023-10-30T12:51:31.170135+0000","last_peered":"2023-10-30T12:51:31.170135+0000","last_clean":"2023-10-30T12:51:31.170135+0000","last_became_active":"2023-10-30T12:28:50.966569+0000","last_became_peered":"2023-10-30T12:28:50.966569+0000","last_unstale":"2023-10-30T12:51:31.170135+0000","last_undegraded":"2023-10-30T12:51:31.170135+0000","last_fullsized":"2023-10-30T12:51:31.170135+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":548,"log_dups_size":0,"ondisk_log_size":548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037972850000000001,"stat_sum":{"num_bytes":16988939,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":42,"num_read_kb":19,"num_write":508,"num_write_kb":18140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"66'375","reported_seq":463,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:45.949491+0000","last_change":"2023-10-30T12:51:45.949541+0000","last_active":"2023-10-30T12:51:45.949491+0000","last_peered":"2023-10-30T12:51:45.949491+0000","last_clean":"2023-10-30T12:51:45.949491+0000","last_became_active":"2023-10-30T12:28:50.391260+0000","last_became_peered":"2023-10-30T12:28:50.391260+0000","last_unstale":"2023-10-30T12:51:45.949491+0000","last_undegraded":"2023-10-30T12:51:45.949491+0000","last_fullsized":"2023-10-30T12:51:45.949491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'375","last_scrub_stamp":"2023-10-30T12:51:45.949405+0000","last_deep_scrub":"66'375","last_deep_scrub_stamp":"2023-10-30T12:51:45.949405+0000","last_clean_scrub_stamp":"2023-10-30T12:51:45.949405+0000","objects_scrubbed":6,"log_size":375,"log_dups_size":0,"ondisk_log_size":375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:31:43.191871+0000","scrub_duration":0.024520874000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":22,"num_read_kb":7467,"num_write":319,"num_write_kb":4401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7112,"num_omap_keys":14,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"66'524","reported_seq":2691,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:42.140431+0000","last_change":"2023-10-30T12:51:42.140557+0000","last_active":"2023-10-30T12:51:42.140431+0000","last_peered":"2023-10-30T12:51:42.140431+0000","last_clean":"2023-10-30T12:51:42.140431+0000","last_became_active":"2023-10-30T12:28:50.974843+0000","last_became_peered":"2023-10-30T12:28:50.974843+0000","last_unstale":"2023-10-30T12:51:42.140431+0000","last_undegraded":"2023-10-30T12:51:42.140431+0000","last_fullsized":"2023-10-30T12:51:42.140431+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'524","last_scrub_stamp":"2023-10-30T12:51:42.140220+0000","last_deep_scrub":"66'524","last_deep_scrub_stamp":"2023-10-30T12:51:42.140220+0000","last_clean_scrub_stamp":"2023-10-30T12:51:42.140220+0000","objects_scrubbed":12,"log_size":524,"log_dups_size":0,"ondisk_log_size":524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:15:48.848274+0000","scrub_duration":0.033227912999999998,"objects_trimmed":0,"snaptrim_duration":0.005161372,"stat_sum":{"num_bytes":4244440,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":47,"num_read_kb":17,"num_write":458,"num_write_kb":9849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"66'358","reported_seq":457,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:40.635013+0000","last_change":"2023-10-30T12:51:40.635013+0000","last_active":"2023-10-30T12:51:40.635013+0000","last_peered":"2023-10-30T12:51:40.635013+0000","last_clean":"2023-10-30T12:51:40.635013+0000","last_became_active":"2023-10-30T12:28:50.373581+0000","last_became_peered":"2023-10-30T12:28:50.373581+0000","last_unstale":"2023-10-30T12:51:40.635013+0000","last_undegraded":"2023-10-30T12:51:40.635013+0000","last_fullsized":"2023-10-30T12:51:40.635013+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'357","last_scrub_stamp":"2023-10-30T12:51:40.634678+0000","last_deep_scrub":"66'357","last_deep_scrub_stamp":"2023-10-30T12:51:40.634678+0000","last_clean_scrub_stamp":"2023-10-30T12:51:40.634678+0000","objects_scrubbed":9,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:38:14.782628+0000","scrub_duration":0.051559438999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":43,"num_read_kb":32803,"num_write":293,"num_write_kb":12544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7620,"num_omap_keys":15,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"66'499","reported_seq":2649,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.507327+0000","last_change":"2023-10-30T12:38:17.328512+0000","last_active":"2023-10-30T12:51:30.507327+0000","last_peered":"2023-10-30T12:51:30.507327+0000","last_clean":"2023-10-30T12:51:30.507327+0000","last_became_active":"2023-10-30T12:28:50.973269+0000","last_became_peered":"2023-10-30T12:28:50.973269+0000","last_unstale":"2023-10-30T12:51:30.507327+0000","last_undegraded":"2023-10-30T12:51:30.507327+0000","last_fullsized":"2023-10-30T12:51:30.507327+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":499,"log_dups_size":0,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004001498,"stat_sum":{"num_bytes":12707497,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":38,"num_read_kb":11,"num_write":446,"num_write_kb":13522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"66'444","reported_seq":544,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:37.461835+0000","last_change":"2023-10-30T12:51:37.461917+0000","last_active":"2023-10-30T12:51:37.461835+0000","last_peered":"2023-10-30T12:51:37.461835+0000","last_clean":"2023-10-30T12:51:37.461835+0000","last_became_active":"2023-10-30T12:28:50.366269+0000","last_became_peered":"2023-10-30T12:28:50.366269+0000","last_unstale":"2023-10-30T12:51:37.461835+0000","last_undegraded":"2023-10-30T12:51:37.461835+0000","last_fullsized":"2023-10-30T12:51:37.461835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'444","last_scrub_stamp":"2023-10-30T12:51:37.461743+0000","last_deep_scrub":"66'444","last_deep_scrub_stamp":"2023-10-30T12:51:37.461743+0000","last_clean_scrub_stamp":"2023-10-30T12:51:37.461743+0000","objects_scrubbed":9,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:35:49.998956+0000","scrub_duration":0.040110959000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":52,"num_read_kb":36923,"num_write":386,"num_write_kb":8499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":16168,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"66'586","reported_seq":2758,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:43.354673+0000","last_change":"2023-10-30T12:51:43.354752+0000","last_active":"2023-10-30T12:51:43.354673+0000","last_peered":"2023-10-30T12:51:43.354673+0000","last_clean":"2023-10-30T12:51:43.354673+0000","last_became_active":"2023-10-30T12:28:50.961805+0000","last_became_peered":"2023-10-30T12:28:50.961805+0000","last_unstale":"2023-10-30T12:51:43.354673+0000","last_undegraded":"2023-10-30T12:51:43.354673+0000","last_fullsized":"2023-10-30T12:51:43.354673+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'586","last_scrub_stamp":"2023-10-30T12:51:43.354576+0000","last_deep_scrub":"66'586","last_deep_scrub_stamp":"2023-10-30T12:51:43.354576+0000","last_clean_scrub_stamp":"2023-10-30T12:51:43.354576+0000","objects_scrubbed":22,"log_size":586,"log_dups_size":0,"ondisk_log_size":586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:44:38.376225+0000","scrub_duration":0.037946186999999999,"objects_trimmed":0,"snaptrim_duration":0.0032264350000000002,"stat_sum":{"num_bytes":4388445,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":52,"num_read_kb":4114,"num_write":506,"num_write_kb":14199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"66'258","reported_seq":334,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:24.396418+0000","last_change":"2023-10-30T12:28:50.397244+0000","last_active":"2023-10-30T12:51:24.396418+0000","last_peered":"2023-10-30T12:51:24.396418+0000","last_clean":"2023-10-30T12:51:24.396418+0000","last_became_active":"2023-10-30T12:28:50.396727+0000","last_became_peered":"2023-10-30T12:28:50.396727+0000","last_unstale":"2023-10-30T12:51:24.396418+0000","last_undegraded":"2023-10-30T12:51:24.396418+0000","last_fullsized":"2023-10-30T12:51:24.396418+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":16164,"num_write":201,"num_write_kb":8311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"66'568","reported_seq":2743,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:49.633783+0000","last_change":"2023-10-30T12:51:49.633887+0000","last_active":"2023-10-30T12:51:49.633783+0000","last_peered":"2023-10-30T12:51:49.633783+0000","last_clean":"2023-10-30T12:51:49.633783+0000","last_became_active":"2023-10-30T12:29:32.960608+0000","last_became_peered":"2023-10-30T12:29:32.960608+0000","last_unstale":"2023-10-30T12:51:49.633783+0000","last_undegraded":"2023-10-30T12:51:49.633783+0000","last_fullsized":"2023-10-30T12:51:49.633783+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'568","last_scrub_stamp":"2023-10-30T12:51:49.633664+0000","last_deep_scrub":"66'568","last_deep_scrub_stamp":"2023-10-30T12:51:49.633664+0000","last_clean_scrub_stamp":"2023-10-30T12:51:49.633664+0000","objects_scrubbed":14,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:00:42.755675+0000","scrub_duration":0.028072751999999999,"objects_trimmed":0,"snaptrim_duration":0.0012934999999999999,"stat_sum":{"num_bytes":4430082,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":59,"num_read_kb":33,"num_write":506,"num_write_kb":6384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"66'684","reported_seq":774,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:44.639409+0000","last_change":"2023-10-30T12:51:44.639628+0000","last_active":"2023-10-30T12:51:44.639409+0000","last_peered":"2023-10-30T12:51:44.639409+0000","last_clean":"2023-10-30T12:51:44.639409+0000","last_became_active":"2023-10-30T12:28:50.383570+0000","last_became_peered":"2023-10-30T12:28:50.383570+0000","last_unstale":"2023-10-30T12:51:44.639409+0000","last_undegraded":"2023-10-30T12:51:44.639409+0000","last_fullsized":"2023-10-30T12:51:44.639409+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'684","last_scrub_stamp":"2023-10-30T12:51:44.639059+0000","last_deep_scrub":"66'684","last_deep_scrub_stamp":"2023-10-30T12:51:44.639059+0000","last_clean_scrub_stamp":"2023-10-30T12:51:44.639059+0000","objects_scrubbed":7,"log_size":684,"log_dups_size":0,"ondisk_log_size":684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:23:55.453101+0000","scrub_duration":0.079897833000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":37306,"num_write":619,"num_write_kb":16724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15312,"num_omap_keys":30,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"66'541","reported_seq":2686,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.407985+0000","last_change":"2023-10-30T12:38:17.322231+0000","last_active":"2023-10-30T12:51:30.407985+0000","last_peered":"2023-10-30T12:51:30.407985+0000","last_clean":"2023-10-30T12:51:30.407985+0000","last_became_active":"2023-10-30T12:28:50.949485+0000","last_became_peered":"2023-10-30T12:28:50.949485+0000","last_unstale":"2023-10-30T12:51:30.407985+0000","last_undegraded":"2023-10-30T12:51:30.407985+0000","last_fullsized":"2023-10-30T12:51:30.407985+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":541,"log_dups_size":0,"ondisk_log_size":541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040897299999999998,"stat_sum":{"num_bytes":12622957,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":477,"num_write_kb":14339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"66'453","reported_seq":568,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:41.463979+0000","last_change":"2023-10-30T12:51:41.463979+0000","last_active":"2023-10-30T12:51:41.463979+0000","last_peered":"2023-10-30T12:51:41.463979+0000","last_clean":"2023-10-30T12:51:41.463979+0000","last_became_active":"2023-10-30T12:28:50.362624+0000","last_became_peered":"2023-10-30T12:28:50.362624+0000","last_unstale":"2023-10-30T12:51:41.463979+0000","last_undegraded":"2023-10-30T12:51:41.463979+0000","last_fullsized":"2023-10-30T12:51:41.463979+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'452","last_scrub_stamp":"2023-10-30T12:51:41.463765+0000","last_deep_scrub":"66'452","last_deep_scrub_stamp":"2023-10-30T12:51:41.463765+0000","last_clean_scrub_stamp":"2023-10-30T12:51:41.463765+0000","objects_scrubbed":13,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:18.726132+0000","scrub_duration":0.043492996999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583024,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":93,"num_read_kb":24662,"num_write":378,"num_write_kb":12558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6552,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"66'582","reported_seq":2744,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.506617+0000","last_change":"2023-10-30T12:38:17.334068+0000","last_active":"2023-10-30T12:51:30.506617+0000","last_peered":"2023-10-30T12:51:30.506617+0000","last_clean":"2023-10-30T12:51:30.506617+0000","last_became_active":"2023-10-30T12:28:50.979344+0000","last_became_peered":"2023-10-30T12:28:50.979344+0000","last_unstale":"2023-10-30T12:51:30.506617+0000","last_undegraded":"2023-10-30T12:51:30.506617+0000","last_fullsized":"2023-10-30T12:51:30.506617+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":582,"log_dups_size":0,"ondisk_log_size":582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0092295529999999997,"stat_sum":{"num_bytes":12647197,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":50,"num_read_kb":8210,"num_write":532,"num_write_kb":23192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"66'248","reported_seq":334,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:37.268010+0000","last_change":"2023-10-30T12:51:37.268108+0000","last_active":"2023-10-30T12:51:37.268010+0000","last_peered":"2023-10-30T12:51:37.268010+0000","last_clean":"2023-10-30T12:51:37.268010+0000","last_became_active":"2023-10-30T12:28:50.381125+0000","last_became_peered":"2023-10-30T12:28:50.381125+0000","last_unstale":"2023-10-30T12:51:37.268010+0000","last_undegraded":"2023-10-30T12:51:37.268010+0000","last_fullsized":"2023-10-30T12:51:37.268010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'248","last_scrub_stamp":"2023-10-30T12:51:37.267826+0000","last_deep_scrub":"66'248","last_deep_scrub_stamp":"2023-10-30T12:51:37.267826+0000","last_clean_scrub_stamp":"2023-10-30T12:51:37.267826+0000","objects_scrubbed":3,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:30:50.232542+0000","scrub_duration":0.054105548000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":12378,"num_write":396,"num_write_kb":4652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"66'531","reported_seq":2680,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.506015+0000","last_change":"2023-10-30T12:38:17.334809+0000","last_active":"2023-10-30T12:51:30.506015+0000","last_peered":"2023-10-30T12:51:30.506015+0000","last_clean":"2023-10-30T12:51:30.506015+0000","last_became_active":"2023-10-30T12:28:50.977585+0000","last_became_peered":"2023-10-30T12:28:50.977585+0000","last_unstale":"2023-10-30T12:51:30.506015+0000","last_undegraded":"2023-10-30T12:51:30.506015+0000","last_fullsized":"2023-10-30T12:51:30.506015+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":531,"log_dups_size":0,"ondisk_log_size":531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089211889999999995,"stat_sum":{"num_bytes":12933034,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":36,"num_read_kb":8205,"num_write":483,"num_write_kb":18997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"66'359","reported_seq":445,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:38.474826+0000","last_change":"2023-10-30T12:51:38.474877+0000","last_active":"2023-10-30T12:51:38.474826+0000","last_peered":"2023-10-30T12:51:38.474826+0000","last_clean":"2023-10-30T12:51:38.474826+0000","last_became_active":"2023-10-30T12:28:50.366974+0000","last_became_peered":"2023-10-30T12:28:50.366974+0000","last_unstale":"2023-10-30T12:51:38.474826+0000","last_undegraded":"2023-10-30T12:51:38.474826+0000","last_fullsized":"2023-10-30T12:51:38.474826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'359","last_scrub_stamp":"2023-10-30T12:51:38.474746+0000","last_deep_scrub":"66'359","last_deep_scrub_stamp":"2023-10-30T12:51:38.474746+0000","last_clean_scrub_stamp":"2023-10-30T12:51:38.474746+0000","objects_scrubbed":4,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:46:34.826564+0000","scrub_duration":0.021991255000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":15324,"num_write":292,"num_write_kb":8361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10374,"num_omap_keys":19,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"66'534","reported_seq":2697,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.506599+0000","last_change":"2023-10-30T12:38:17.337139+0000","last_active":"2023-10-30T12:51:30.506599+0000","last_peered":"2023-10-30T12:51:30.506599+0000","last_clean":"2023-10-30T12:51:30.506599+0000","last_became_active":"2023-10-30T12:28:50.981193+0000","last_became_peered":"2023-10-30T12:28:50.981193+0000","last_unstale":"2023-10-30T12:51:30.506599+0000","last_undegraded":"2023-10-30T12:51:30.506599+0000","last_fullsized":"2023-10-30T12:51:30.506599+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":534,"log_dups_size":0,"ondisk_log_size":534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096579700000000001,"stat_sum":{"num_bytes":12674006,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":50,"num_read_kb":4114,"num_write":492,"num_write_kb":18051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"66'98","reported_seq":180,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:44.901113+0000","last_change":"2023-10-30T12:51:44.901168+0000","last_active":"2023-10-30T12:51:44.901113+0000","last_peered":"2023-10-30T12:51:44.901113+0000","last_clean":"2023-10-30T12:51:44.901113+0000","last_became_active":"2023-10-30T12:28:50.367730+0000","last_became_peered":"2023-10-30T12:28:50.367730+0000","last_unstale":"2023-10-30T12:51:44.901113+0000","last_undegraded":"2023-10-30T12:51:44.901113+0000","last_fullsized":"2023-10-30T12:51:44.901113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'98","last_scrub_stamp":"2023-10-30T12:51:44.900977+0000","last_deep_scrub":"66'98","last_deep_scrub_stamp":"2023-10-30T12:51:44.900977+0000","last_clean_scrub_stamp":"2023-10-30T12:51:44.900977+0000","objects_scrubbed":4,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:46:56.082172+0000","scrub_duration":0.018334822000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":12292,"num_write":45,"num_write_kb":4186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8128,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"66'594","reported_seq":2763,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.473832+0000","last_change":"2023-10-30T12:38:17.119150+0000","last_active":"2023-10-30T12:51:30.473832+0000","last_peered":"2023-10-30T12:51:30.473832+0000","last_clean":"2023-10-30T12:51:30.473832+0000","last_became_active":"2023-10-30T12:28:50.966246+0000","last_became_peered":"2023-10-30T12:28:50.966246+0000","last_unstale":"2023-10-30T12:51:30.473832+0000","last_undegraded":"2023-10-30T12:51:30.473832+0000","last_fullsized":"2023-10-30T12:51:30.473832+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":594,"log_dups_size":0,"ondisk_log_size":594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007703601,"stat_sum":{"num_bytes":8520713,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":56,"num_read_kb":18,"num_write":541,"num_write_kb":11343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"66'87","reported_seq":172,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:42.434625+0000","last_change":"2023-10-30T12:51:42.434673+0000","last_active":"2023-10-30T12:51:42.434625+0000","last_peered":"2023-10-30T12:51:42.434625+0000","last_clean":"2023-10-30T12:51:42.434625+0000","last_became_active":"2023-10-30T12:28:50.393642+0000","last_became_peered":"2023-10-30T12:28:50.393642+0000","last_unstale":"2023-10-30T12:51:42.434625+0000","last_undegraded":"2023-10-30T12:51:42.434625+0000","last_fullsized":"2023-10-30T12:51:42.434625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'87","last_scrub_stamp":"2023-10-30T12:51:42.434545+0000","last_deep_scrub":"66'87","last_deep_scrub_stamp":"2023-10-30T12:51:42.434545+0000","last_clean_scrub_stamp":"2023-10-30T12:51:42.434545+0000","objects_scrubbed":5,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:27:29.743857+0000","scrub_duration":0.0090945590000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":15,"num_write_kb":101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17524,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"66'579","reported_seq":2744,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:56.560776+0000","last_change":"2023-10-30T12:51:56.560943+0000","last_active":"2023-10-30T12:51:56.560776+0000","last_peered":"2023-10-30T12:51:56.560776+0000","last_clean":"2023-10-30T12:51:56.560776+0000","last_became_active":"2023-10-30T12:28:50.976343+0000","last_became_peered":"2023-10-30T12:28:50.976343+0000","last_unstale":"2023-10-30T12:51:56.560776+0000","last_undegraded":"2023-10-30T12:51:56.560776+0000","last_fullsized":"2023-10-30T12:51:56.560776+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'579","last_scrub_stamp":"2023-10-30T12:51:56.560556+0000","last_deep_scrub":"66'579","last_deep_scrub_stamp":"2023-10-30T12:51:56.560556+0000","last_clean_scrub_stamp":"2023-10-30T12:51:56.560556+0000","objects_scrubbed":23,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:00:58.734907+0000","scrub_duration":0.074070374999999994,"objects_trimmed":0,"snaptrim_duration":0.0091875959999999993,"stat_sum":{"num_bytes":17078090,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":44,"num_read_kb":14,"num_write":517,"num_write_kb":22829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"66'119","reported_seq":231,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:44.641495+0000","last_change":"2023-10-30T12:51:44.641713+0000","last_active":"2023-10-30T12:51:44.641495+0000","last_peered":"2023-10-30T12:51:44.641495+0000","last_clean":"2023-10-30T12:51:44.641495+0000","last_became_active":"2023-10-30T12:28:41.201796+0000","last_became_peered":"2023-10-30T12:28:41.201796+0000","last_unstale":"2023-10-30T12:51:44.641495+0000","last_undegraded":"2023-10-30T12:51:44.641495+0000","last_fullsized":"2023-10-30T12:51:44.641495+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'119","last_scrub_stamp":"2023-10-30T12:51:44.641256+0000","last_deep_scrub":"66'119","last_deep_scrub_stamp":"2023-10-30T12:51:44.641256+0000","last_clean_scrub_stamp":"2023-10-30T12:51:44.641256+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:26:13.839326+0000","scrub_duration":0.013391731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"66'201","reported_seq":291,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:50.447569+0000","last_change":"2023-10-30T12:51:50.447661+0000","last_active":"2023-10-30T12:51:50.447569+0000","last_peered":"2023-10-30T12:51:50.447569+0000","last_clean":"2023-10-30T12:51:50.447569+0000","last_became_active":"2023-10-30T12:28:50.367593+0000","last_became_peered":"2023-10-30T12:28:50.367593+0000","last_unstale":"2023-10-30T12:51:50.447569+0000","last_undegraded":"2023-10-30T12:51:50.447569+0000","last_fullsized":"2023-10-30T12:51:50.447569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'201","last_scrub_stamp":"2023-10-30T12:51:50.447458+0000","last_deep_scrub":"66'201","last_deep_scrub_stamp":"2023-10-30T12:51:50.447458+0000","last_clean_scrub_stamp":"2023-10-30T12:51:50.447458+0000","objects_scrubbed":9,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:35:16.204652+0000","scrub_duration":0.025712215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194338,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":24,"num_read_kb":12326,"num_write":144,"num_write_kb":4223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":49160,"num_omap_keys":96,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"66'535","reported_seq":2689,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:48.599000+0000","last_change":"2023-10-30T12:51:48.599069+0000","last_active":"2023-10-30T12:51:48.599000+0000","last_peered":"2023-10-30T12:51:48.599000+0000","last_clean":"2023-10-30T12:51:48.599000+0000","last_became_active":"2023-10-30T12:29:32.997301+0000","last_became_peered":"2023-10-30T12:29:32.997301+0000","last_unstale":"2023-10-30T12:51:48.599000+0000","last_undegraded":"2023-10-30T12:51:48.599000+0000","last_fullsized":"2023-10-30T12:51:48.599000+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'535","last_scrub_stamp":"2023-10-30T12:51:48.598899+0000","last_deep_scrub":"66'535","last_deep_scrub_stamp":"2023-10-30T12:51:48.598899+0000","last_clean_scrub_stamp":"2023-10-30T12:51:48.598899+0000","objects_scrubbed":17,"log_size":535,"log_dups_size":0,"ondisk_log_size":535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:04:11.091469+0000","scrub_duration":0.019854914000000001,"objects_trimmed":0,"snaptrim_duration":0.001366412,"stat_sum":{"num_bytes":4237965,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":36,"num_read_kb":11,"num_write":481,"num_write_kb":13486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"66'504","reported_seq":600,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:47.588384+0000","last_change":"2023-10-30T12:51:47.588509+0000","last_active":"2023-10-30T12:51:47.588384+0000","last_peered":"2023-10-30T12:51:47.588384+0000","last_clean":"2023-10-30T12:51:47.588384+0000","last_became_active":"2023-10-30T12:28:50.391824+0000","last_became_peered":"2023-10-30T12:28:50.391824+0000","last_unstale":"2023-10-30T12:51:47.588384+0000","last_undegraded":"2023-10-30T12:51:47.588384+0000","last_fullsized":"2023-10-30T12:51:47.588384+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'504","last_scrub_stamp":"2023-10-30T12:51:47.588159+0000","last_deep_scrub":"66'504","last_deep_scrub_stamp":"2023-10-30T12:51:47.588159+0000","last_clean_scrub_stamp":"2023-10-30T12:51:47.588159+0000","objects_scrubbed":7,"log_size":504,"log_dups_size":0,"ondisk_log_size":504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:53:44.520327+0000","scrub_duration":0.072404271000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":32,"num_read_kb":50322,"num_write":431,"num_write_kb":16616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13208,"num_omap_keys":26,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"66'532","reported_seq":2675,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:33.862317+0000","last_change":"2023-10-30T12:38:17.643111+0000","last_active":"2023-10-30T12:51:33.862317+0000","last_peered":"2023-10-30T12:51:33.862317+0000","last_clean":"2023-10-30T12:51:33.862317+0000","last_became_active":"2023-10-30T12:28:50.952625+0000","last_became_peered":"2023-10-30T12:28:50.952625+0000","last_unstale":"2023-10-30T12:51:33.862317+0000","last_undegraded":"2023-10-30T12:51:33.862317+0000","last_fullsized":"2023-10-30T12:51:33.862317+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":532,"log_dups_size":0,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026695970000000001,"stat_sum":{"num_bytes":16953646,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":29,"num_read_kb":10,"num_write":468,"num_write_kb":20237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"66'605","reported_seq":920,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:48.574934+0000","last_change":"2023-10-30T12:51:35.405246+0000","last_active":"2023-10-30T12:51:48.574934+0000","last_peered":"2023-10-30T12:51:48.574934+0000","last_clean":"2023-10-30T12:51:48.574934+0000","last_became_active":"2023-10-30T12:28:50.367230+0000","last_became_peered":"2023-10-30T12:28:50.367230+0000","last_unstale":"2023-10-30T12:51:48.574934+0000","last_undegraded":"2023-10-30T12:51:48.574934+0000","last_fullsized":"2023-10-30T12:51:48.574934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'598","last_scrub_stamp":"2023-10-30T12:51:35.405133+0000","last_deep_scrub":"66'598","last_deep_scrub_stamp":"2023-10-30T12:51:35.405133+0000","last_clean_scrub_stamp":"2023-10-30T12:51:35.405133+0000","objects_scrubbed":7,"log_size":605,"log_dups_size":0,"ondisk_log_size":605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:34:01.194417+0000","scrub_duration":0.035040017999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12778646,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":251,"num_read_kb":13023,"num_write":542,"num_write_kb":12830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9828,"num_omap_keys":18,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"66'551","reported_seq":2712,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.382287+0000","last_change":"2023-10-30T12:38:17.104045+0000","last_active":"2023-10-30T12:51:30.382287+0000","last_peered":"2023-10-30T12:51:30.382287+0000","last_clean":"2023-10-30T12:51:30.382287+0000","last_became_active":"2023-10-30T12:28:50.928608+0000","last_became_peered":"2023-10-30T12:28:50.928608+0000","last_unstale":"2023-10-30T12:51:30.382287+0000","last_undegraded":"2023-10-30T12:51:30.382287+0000","last_fullsized":"2023-10-30T12:51:30.382287+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":551,"log_dups_size":0,"ondisk_log_size":551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002074275,"stat_sum":{"num_bytes":8441122,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":47,"num_read_kb":21,"num_write":501,"num_write_kb":11799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"66'493","reported_seq":574,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:36.467505+0000","last_change":"2023-10-30T12:28:50.391240+0000","last_active":"2023-10-30T12:51:36.467505+0000","last_peered":"2023-10-30T12:51:36.467505+0000","last_clean":"2023-10-30T12:51:36.467505+0000","last_became_active":"2023-10-30T12:28:50.390864+0000","last_became_peered":"2023-10-30T12:28:50.390864+0000","last_unstale":"2023-10-30T12:51:36.467505+0000","last_undegraded":"2023-10-30T12:51:36.467505+0000","last_fullsized":"2023-10-30T12:51:36.467505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:49.324113+0000","last_clean_scrub_stamp":"2023-10-30T12:28:49.324113+0000","objects_scrubbed":0,"log_size":493,"log_dups_size":0,"ondisk_log_size":493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":45183,"num_write":427,"num_write_kb":16635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"66'536","reported_seq":2693,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:45.113576+0000","last_change":"2023-10-30T12:51:45.113576+0000","last_active":"2023-10-30T12:51:45.113576+0000","last_peered":"2023-10-30T12:51:45.113576+0000","last_clean":"2023-10-30T12:51:45.113576+0000","last_became_active":"2023-10-30T12:28:50.949122+0000","last_became_peered":"2023-10-30T12:28:50.949122+0000","last_unstale":"2023-10-30T12:51:45.113576+0000","last_undegraded":"2023-10-30T12:51:45.113576+0000","last_fullsized":"2023-10-30T12:51:45.113576+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'536","last_scrub_stamp":"2023-10-30T12:51:45.113133+0000","last_deep_scrub":"66'536","last_deep_scrub_stamp":"2023-10-30T12:51:45.113133+0000","last_clean_scrub_stamp":"2023-10-30T12:51:45.113133+0000","objects_scrubbed":16,"log_size":536,"log_dups_size":0,"ondisk_log_size":536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:45:58.801601+0000","scrub_duration":0.060099805999999999,"objects_trimmed":0,"snaptrim_duration":0.004393348,"stat_sum":{"num_bytes":12684737,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":35,"num_read_kb":65,"num_write":464,"num_write_kb":14146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"66'267","reported_seq":346,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:36.014856+0000","last_change":"2023-10-30T12:51:36.014856+0000","last_active":"2023-10-30T12:51:36.014856+0000","last_peered":"2023-10-30T12:51:36.014856+0000","last_clean":"2023-10-30T12:51:36.014856+0000","last_became_active":"2023-10-30T12:28:50.358310+0000","last_became_peered":"2023-10-30T12:28:50.358310+0000","last_unstale":"2023-10-30T12:51:36.014856+0000","last_undegraded":"2023-10-30T12:51:36.014856+0000","last_fullsized":"2023-10-30T12:51:36.014856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'267","last_scrub_stamp":"2023-10-30T12:51:36.014733+0000","last_deep_scrub":"66'267","last_deep_scrub_stamp":"2023-10-30T12:51:36.014733+0000","last_clean_scrub_stamp":"2023-10-30T12:51:36.014733+0000","objects_scrubbed":2,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:22:01.486529+0000","scrub_duration":0.015176472999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":5758,"num_write":222,"num_write_kb":4242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7112,"num_omap_keys":14,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"66'554","reported_seq":2709,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:38.621754+0000","last_change":"2023-10-30T12:51:38.621754+0000","last_active":"2023-10-30T12:51:38.621754+0000","last_peered":"2023-10-30T12:51:38.621754+0000","last_clean":"2023-10-30T12:51:38.621754+0000","last_became_active":"2023-10-30T12:28:50.965626+0000","last_became_peered":"2023-10-30T12:28:50.965626+0000","last_unstale":"2023-10-30T12:51:38.621754+0000","last_undegraded":"2023-10-30T12:51:38.621754+0000","last_fullsized":"2023-10-30T12:51:38.621754+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'554","last_scrub_stamp":"2023-10-30T12:51:38.621402+0000","last_deep_scrub":"66'554","last_deep_scrub_stamp":"2023-10-30T12:51:38.621402+0000","last_clean_scrub_stamp":"2023-10-30T12:51:38.621402+0000","objects_scrubbed":15,"log_size":554,"log_dups_size":0,"ondisk_log_size":554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:32:24.000396+0000","scrub_duration":0.10283745599999999,"objects_trimmed":0,"snaptrim_duration":0.0079099229999999993,"stat_sum":{"num_bytes":29382851,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":33,"num_read_kb":4109,"num_write":477,"num_write_kb":37019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"66'404","reported_seq":485,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:53.568757+0000","last_change":"2023-10-30T12:51:53.568971+0000","last_active":"2023-10-30T12:51:53.568757+0000","last_peered":"2023-10-30T12:51:53.568757+0000","last_clean":"2023-10-30T12:51:53.568757+0000","last_became_active":"2023-10-30T12:28:50.395090+0000","last_became_peered":"2023-10-30T12:28:50.395090+0000","last_unstale":"2023-10-30T12:51:53.568757+0000","last_undegraded":"2023-10-30T12:51:53.568757+0000","last_fullsized":"2023-10-30T12:51:53.568757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'404","last_scrub_stamp":"2023-10-30T12:51:53.568460+0000","last_deep_scrub":"66'404","last_deep_scrub_stamp":"2023-10-30T12:51:53.568460+0000","last_clean_scrub_stamp":"2023-10-30T12:51:53.568460+0000","objects_scrubbed":5,"log_size":404,"log_dups_size":0,"ondisk_log_size":404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:27:52.316450+0000","scrub_duration":0.055684696999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582934,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":8195,"num_write":330,"num_write_kb":12489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12012,"num_omap_keys":22,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"66'551","reported_seq":2715,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:34.379165+0000","last_change":"2023-10-30T12:38:17.104334+0000","last_active":"2023-10-30T12:51:34.379165+0000","last_peered":"2023-10-30T12:51:34.379165+0000","last_clean":"2023-10-30T12:51:34.379165+0000","last_became_active":"2023-10-30T12:28:50.929337+0000","last_became_peered":"2023-10-30T12:28:50.929337+0000","last_unstale":"2023-10-30T12:51:34.379165+0000","last_undegraded":"2023-10-30T12:51:34.379165+0000","last_fullsized":"2023-10-30T12:51:34.379165+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":551,"log_dups_size":0,"ondisk_log_size":551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023039890000000002,"stat_sum":{"num_bytes":14041074,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":50,"num_read_kb":18,"num_write":487,"num_write_kb":15488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"66'356","reported_seq":443,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:41.110896+0000","last_change":"2023-10-30T12:51:41.110963+0000","last_active":"2023-10-30T12:51:41.110896+0000","last_peered":"2023-10-30T12:51:41.110896+0000","last_clean":"2023-10-30T12:51:41.110896+0000","last_became_active":"2023-10-30T12:28:50.380247+0000","last_became_peered":"2023-10-30T12:28:50.380247+0000","last_unstale":"2023-10-30T12:51:41.110896+0000","last_undegraded":"2023-10-30T12:51:41.110896+0000","last_fullsized":"2023-10-30T12:51:41.110896+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'356","last_scrub_stamp":"2023-10-30T12:51:41.110806+0000","last_deep_scrub":"66'356","last_deep_scrub_stamp":"2023-10-30T12:51:41.110806+0000","last_clean_scrub_stamp":"2023-10-30T12:51:41.110806+0000","objects_scrubbed":5,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:36:44.682836+0000","scrub_duration":0.090051573999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":8272,"num_write":528,"num_write_kb":5259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18674,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"66'532","reported_seq":2675,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.471259+0000","last_change":"2023-10-30T12:38:17.331272+0000","last_active":"2023-10-30T12:51:30.471259+0000","last_peered":"2023-10-30T12:51:30.471259+0000","last_clean":"2023-10-30T12:51:30.471259+0000","last_became_active":"2023-10-30T12:28:50.973086+0000","last_became_peered":"2023-10-30T12:28:50.973086+0000","last_unstale":"2023-10-30T12:51:30.471259+0000","last_undegraded":"2023-10-30T12:51:30.471259+0000","last_fullsized":"2023-10-30T12:51:30.471259+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":532,"log_dups_size":0,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034133169999999999,"stat_sum":{"num_bytes":16807043,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":30,"num_read_kb":14,"num_write":492,"num_write_kb":22119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"66'266","reported_seq":349,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:48.569486+0000","last_change":"2023-10-30T12:51:43.564434+0000","last_active":"2023-10-30T12:51:48.569486+0000","last_peered":"2023-10-30T12:51:48.569486+0000","last_clean":"2023-10-30T12:51:48.569486+0000","last_became_active":"2023-10-30T12:28:50.385924+0000","last_became_peered":"2023-10-30T12:28:50.385924+0000","last_unstale":"2023-10-30T12:51:48.569486+0000","last_undegraded":"2023-10-30T12:51:48.569486+0000","last_fullsized":"2023-10-30T12:51:48.569486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'261","last_scrub_stamp":"2023-10-30T12:51:43.564089+0000","last_deep_scrub":"66'261","last_deep_scrub_stamp":"2023-10-30T12:51:43.564089+0000","last_clean_scrub_stamp":"2023-10-30T12:51:43.564089+0000","objects_scrubbed":5,"log_size":266,"log_dups_size":0,"ondisk_log_size":266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:34:21.648949+0000","scrub_duration":0.022937485000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3122023,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":200,"num_write_kb":3178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13032,"num_omap_keys":26,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"66'541","reported_seq":2705,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:49.313126+0000","last_change":"2023-10-30T12:51:49.313203+0000","last_active":"2023-10-30T12:51:49.313126+0000","last_peered":"2023-10-30T12:51:49.313126+0000","last_clean":"2023-10-30T12:51:49.313126+0000","last_became_active":"2023-10-30T12:28:50.929259+0000","last_became_peered":"2023-10-30T12:28:50.929259+0000","last_unstale":"2023-10-30T12:51:49.313126+0000","last_undegraded":"2023-10-30T12:51:49.313126+0000","last_fullsized":"2023-10-30T12:51:49.313126+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'541","last_scrub_stamp":"2023-10-30T12:51:49.313036+0000","last_deep_scrub":"66'541","last_deep_scrub_stamp":"2023-10-30T12:51:49.313036+0000","last_clean_scrub_stamp":"2023-10-30T12:51:49.313036+0000","objects_scrubbed":17,"log_size":541,"log_dups_size":0,"ondisk_log_size":541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:00:33.907215+0000","scrub_duration":0.028828065,"objects_trimmed":0,"snaptrim_duration":0.0029033230000000002,"stat_sum":{"num_bytes":8423823,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":43,"num_read_kb":22,"num_write":477,"num_write_kb":11480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"66'379","reported_seq":470,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:46.586619+0000","last_change":"2023-10-30T12:51:46.586619+0000","last_active":"2023-10-30T12:51:46.586619+0000","last_peered":"2023-10-30T12:51:46.586619+0000","last_clean":"2023-10-30T12:51:46.586619+0000","last_became_active":"2023-10-30T12:28:50.385305+0000","last_became_peered":"2023-10-30T12:28:50.385305+0000","last_unstale":"2023-10-30T12:51:46.586619+0000","last_undegraded":"2023-10-30T12:51:46.586619+0000","last_fullsized":"2023-10-30T12:51:46.586619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'378","last_scrub_stamp":"2023-10-30T12:51:46.586293+0000","last_deep_scrub":"66'378","last_deep_scrub_stamp":"2023-10-30T12:51:46.586293+0000","last_clean_scrub_stamp":"2023-10-30T12:51:46.586293+0000","objects_scrubbed":5,"log_size":379,"log_dups_size":0,"ondisk_log_size":379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:03:32.538098+0000","scrub_duration":0.051218165000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":37322,"num_write":310,"num_write_kb":12446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":508,"num_omap_keys":1,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"66'562","reported_seq":2720,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:38.230220+0000","last_change":"2023-10-30T12:51:38.230463+0000","last_active":"2023-10-30T12:51:38.230220+0000","last_peered":"2023-10-30T12:51:38.230220+0000","last_clean":"2023-10-30T12:51:38.230220+0000","last_became_active":"2023-10-30T12:28:50.940846+0000","last_became_peered":"2023-10-30T12:28:50.940846+0000","last_unstale":"2023-10-30T12:51:38.230220+0000","last_undegraded":"2023-10-30T12:51:38.230220+0000","last_fullsized":"2023-10-30T12:51:38.230220+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'562","last_scrub_stamp":"2023-10-30T12:51:38.229988+0000","last_deep_scrub":"66'562","last_deep_scrub_stamp":"2023-10-30T12:51:38.229988+0000","last_clean_scrub_stamp":"2023-10-30T12:51:38.229988+0000","objects_scrubbed":21,"log_size":562,"log_dups_size":0,"ondisk_log_size":562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:31:03.842668+0000","scrub_duration":0.057637995999999997,"objects_trimmed":0,"snaptrim_duration":0.0046016829999999996,"stat_sum":{"num_bytes":12679067,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":37,"num_read_kb":7,"num_write":498,"num_write_kb":18971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"66'448","reported_seq":534,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:42.982397+0000","last_change":"2023-10-30T12:51:42.982473+0000","last_active":"2023-10-30T12:51:42.982397+0000","last_peered":"2023-10-30T12:51:42.982397+0000","last_clean":"2023-10-30T12:51:42.982397+0000","last_became_active":"2023-10-30T12:28:50.392068+0000","last_became_peered":"2023-10-30T12:28:50.392068+0000","last_unstale":"2023-10-30T12:51:42.982397+0000","last_undegraded":"2023-10-30T12:51:42.982397+0000","last_fullsized":"2023-10-30T12:51:42.982397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'448","last_scrub_stamp":"2023-10-30T12:51:42.982269+0000","last_deep_scrub":"66'448","last_deep_scrub_stamp":"2023-10-30T12:51:42.982269+0000","last_clean_scrub_stamp":"2023-10-30T12:51:42.982269+0000","objects_scrubbed":7,"log_size":448,"log_dups_size":0,"ondisk_log_size":448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:55:25.126896+0000","scrub_duration":0.032966065000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388698,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":19,"num_read_kb":11375,"num_write":387,"num_write_kb":8544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":27300,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"66'570","reported_seq":2735,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:49.808293+0000","last_change":"2023-10-30T12:51:49.808358+0000","last_active":"2023-10-30T12:51:49.808293+0000","last_peered":"2023-10-30T12:51:49.808293+0000","last_clean":"2023-10-30T12:51:49.808293+0000","last_became_active":"2023-10-30T12:28:50.931898+0000","last_became_peered":"2023-10-30T12:28:50.931898+0000","last_unstale":"2023-10-30T12:51:49.808293+0000","last_undegraded":"2023-10-30T12:51:49.808293+0000","last_fullsized":"2023-10-30T12:51:49.808293+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'570","last_scrub_stamp":"2023-10-30T12:51:49.808201+0000","last_deep_scrub":"66'570","last_deep_scrub_stamp":"2023-10-30T12:51:49.808201+0000","last_clean_scrub_stamp":"2023-10-30T12:51:49.808201+0000","objects_scrubbed":18,"log_size":570,"log_dups_size":0,"ondisk_log_size":570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:46:36.932799+0000","scrub_duration":0.022998979999999999,"objects_trimmed":0,"snaptrim_duration":0.003570148,"stat_sum":{"num_bytes":4263636,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":44,"num_read_kb":18,"num_write":507,"num_write_kb":10484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"66'557","reported_seq":639,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:42.047483+0000","last_change":"2023-10-30T12:51:42.047533+0000","last_active":"2023-10-30T12:51:42.047483+0000","last_peered":"2023-10-30T12:51:42.047483+0000","last_clean":"2023-10-30T12:51:42.047483+0000","last_became_active":"2023-10-30T12:28:50.364797+0000","last_became_peered":"2023-10-30T12:28:50.364797+0000","last_unstale":"2023-10-30T12:51:42.047483+0000","last_undegraded":"2023-10-30T12:51:42.047483+0000","last_fullsized":"2023-10-30T12:51:42.047483+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'557","last_scrub_stamp":"2023-10-30T12:51:42.047398+0000","last_deep_scrub":"66'557","last_deep_scrub_stamp":"2023-10-30T12:51:42.047398+0000","last_clean_scrub_stamp":"2023-10-30T12:51:42.047398+0000","objects_scrubbed":4,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:36:26.216418+0000","scrub_duration":0.054259416999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":10,"num_read_kb":8489,"num_write":691,"num_write_kb":13252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"66'568","reported_seq":2738,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:50.791335+0000","last_change":"2023-10-30T12:51:50.791392+0000","last_active":"2023-10-30T12:51:50.791335+0000","last_peered":"2023-10-30T12:51:50.791335+0000","last_clean":"2023-10-30T12:51:50.791335+0000","last_became_active":"2023-10-30T12:28:50.935657+0000","last_became_peered":"2023-10-30T12:28:50.935657+0000","last_unstale":"2023-10-30T12:51:50.791335+0000","last_undegraded":"2023-10-30T12:51:50.791335+0000","last_fullsized":"2023-10-30T12:51:50.791335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'568","last_scrub_stamp":"2023-10-30T12:51:50.791255+0000","last_deep_scrub":"66'568","last_deep_scrub_stamp":"2023-10-30T12:51:50.791255+0000","last_clean_scrub_stamp":"2023-10-30T12:51:50.791255+0000","objects_scrubbed":15,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:03:42.487468+0000","scrub_duration":0.039011377999999999,"objects_trimmed":0,"snaptrim_duration":0.0038543219999999999,"stat_sum":{"num_bytes":16829664,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":49,"num_read_kb":4112,"num_write":499,"num_write_kb":17598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"66'249","reported_seq":338,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:54.509810+0000","last_change":"2023-10-30T12:51:54.509947+0000","last_active":"2023-10-30T12:51:54.509810+0000","last_peered":"2023-10-30T12:51:54.509810+0000","last_clean":"2023-10-30T12:51:54.509810+0000","last_became_active":"2023-10-30T12:28:50.392807+0000","last_became_peered":"2023-10-30T12:28:50.392807+0000","last_unstale":"2023-10-30T12:51:54.509810+0000","last_undegraded":"2023-10-30T12:51:54.509810+0000","last_fullsized":"2023-10-30T12:51:54.509810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'249","last_scrub_stamp":"2023-10-30T12:51:54.509581+0000","last_deep_scrub":"66'249","last_deep_scrub_stamp":"2023-10-30T12:51:54.509581+0000","last_clean_scrub_stamp":"2023-10-30T12:51:54.509581+0000","objects_scrubbed":5,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:12:36.299114+0000","scrub_duration":0.039282915000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":24596,"num_write":178,"num_write_kb":8294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"66'505","reported_seq":2656,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.457322+0000","last_change":"2023-10-30T12:38:17.325460+0000","last_active":"2023-10-30T12:51:30.457322+0000","last_peered":"2023-10-30T12:51:30.457322+0000","last_clean":"2023-10-30T12:51:30.457322+0000","last_became_active":"2023-10-30T12:28:50.954291+0000","last_became_peered":"2023-10-30T12:28:50.954291+0000","last_unstale":"2023-10-30T12:51:30.457322+0000","last_undegraded":"2023-10-30T12:51:30.457322+0000","last_fullsized":"2023-10-30T12:51:30.457322+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":505,"log_dups_size":0,"ondisk_log_size":505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047974480000000002,"stat_sum":{"num_bytes":12709282,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":38,"num_read_kb":15,"num_write":455,"num_write_kb":14651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"66'526","reported_seq":2688,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.476097+0000","last_change":"2023-10-30T12:38:17.120085+0000","last_active":"2023-10-30T12:51:30.476097+0000","last_peered":"2023-10-30T12:51:30.476097+0000","last_clean":"2023-10-30T12:51:30.476097+0000","last_became_active":"2023-10-30T12:28:50.962817+0000","last_became_peered":"2023-10-30T12:28:50.962817+0000","last_unstale":"2023-10-30T12:51:30.476097+0000","last_undegraded":"2023-10-30T12:51:30.476097+0000","last_fullsized":"2023-10-30T12:51:30.476097+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":526,"log_dups_size":0,"ondisk_log_size":526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070898599999999999,"stat_sum":{"num_bytes":8448766,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":50,"num_read_kb":25,"num_write":483,"num_write_kb":14592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"66'176","reported_seq":261,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:51.555385+0000","last_change":"2023-10-30T12:51:51.555592+0000","last_active":"2023-10-30T12:51:51.555385+0000","last_peered":"2023-10-30T12:51:51.555385+0000","last_clean":"2023-10-30T12:51:51.555385+0000","last_became_active":"2023-10-30T12:28:50.401677+0000","last_became_peered":"2023-10-30T12:28:50.401677+0000","last_unstale":"2023-10-30T12:51:51.555385+0000","last_undegraded":"2023-10-30T12:51:51.555385+0000","last_fullsized":"2023-10-30T12:51:51.555385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'176","last_scrub_stamp":"2023-10-30T12:51:51.555145+0000","last_deep_scrub":"66'176","last_deep_scrub_stamp":"2023-10-30T12:51:51.555145+0000","last_clean_scrub_stamp":"2023-10-30T12:51:51.555145+0000","objects_scrubbed":3,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:07:33.229250+0000","scrub_duration":0.033175365999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":20498,"num_write":107,"num_write_kb":8247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"66'500","reported_seq":2656,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.497349+0000","last_change":"2023-10-30T12:38:17.119058+0000","last_active":"2023-10-30T12:51:30.497349+0000","last_peered":"2023-10-30T12:51:30.497349+0000","last_clean":"2023-10-30T12:51:30.497349+0000","last_became_active":"2023-10-30T12:28:50.967591+0000","last_became_peered":"2023-10-30T12:28:50.967591+0000","last_unstale":"2023-10-30T12:51:30.497349+0000","last_undegraded":"2023-10-30T12:51:30.497349+0000","last_fullsized":"2023-10-30T12:51:30.497349+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":500,"log_dups_size":0,"ondisk_log_size":500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081968709999999997,"stat_sum":{"num_bytes":10185055,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":44,"num_read_kb":1610,"num_write":440,"num_write_kb":22911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"66'370","reported_seq":466,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:50.641824+0000","last_change":"2023-10-30T12:51:50.641945+0000","last_active":"2023-10-30T12:51:50.641824+0000","last_peered":"2023-10-30T12:51:50.641824+0000","last_clean":"2023-10-30T12:51:50.641824+0000","last_became_active":"2023-10-30T12:28:50.392021+0000","last_became_peered":"2023-10-30T12:28:50.392021+0000","last_unstale":"2023-10-30T12:51:50.641824+0000","last_undegraded":"2023-10-30T12:51:50.641824+0000","last_fullsized":"2023-10-30T12:51:50.641824+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'370","last_scrub_stamp":"2023-10-30T12:51:50.641619+0000","last_deep_scrub":"66'370","last_deep_scrub_stamp":"2023-10-30T12:51:50.641619+0000","last_clean_scrub_stamp":"2023-10-30T12:51:50.641619+0000","objects_scrubbed":8,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:05:24.267901+0000","scrub_duration":0.078330257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":57384,"num_write":298,"num_write_kb":16576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22858,"num_omap_keys":43,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"66'506","reported_seq":2664,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:33.862492+0000","last_change":"2023-10-30T12:38:17.643950+0000","last_active":"2023-10-30T12:51:33.862492+0000","last_peered":"2023-10-30T12:51:33.862492+0000","last_clean":"2023-10-30T12:51:33.862492+0000","last_became_active":"2023-10-30T12:28:50.967186+0000","last_became_peered":"2023-10-30T12:28:50.967186+0000","last_unstale":"2023-10-30T12:51:33.862492+0000","last_undegraded":"2023-10-30T12:51:33.862492+0000","last_fullsized":"2023-10-30T12:51:33.862492+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003136738,"stat_sum":{"num_bytes":73381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":44,"num_read_kb":17,"num_write":453,"num_write_kb":3099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"66'231","reported_seq":316,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:37.827427+0000","last_change":"2023-10-30T12:51:37.827474+0000","last_active":"2023-10-30T12:51:37.827427+0000","last_peered":"2023-10-30T12:51:37.827427+0000","last_clean":"2023-10-30T12:51:37.827427+0000","last_became_active":"2023-10-30T12:28:50.362929+0000","last_became_peered":"2023-10-30T12:28:50.362929+0000","last_unstale":"2023-10-30T12:51:37.827427+0000","last_undegraded":"2023-10-30T12:51:37.827427+0000","last_fullsized":"2023-10-30T12:51:37.827427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'231","last_scrub_stamp":"2023-10-30T12:51:37.827339+0000","last_deep_scrub":"66'231","last_deep_scrub_stamp":"2023-10-30T12:51:37.827339+0000","last_clean_scrub_stamp":"2023-10-30T12:51:37.827339+0000","objects_scrubbed":4,"log_size":231,"log_dups_size":0,"ondisk_log_size":231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:06:46.012417+0000","scrub_duration":0.026390808000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":9,"num_read_kb":32768,"num_write":162,"num_write_kb":8341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":4,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"66'572","reported_seq":2716,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.401679+0000","last_change":"2023-10-30T12:38:17.331548+0000","last_active":"2023-10-30T12:51:30.401679+0000","last_peered":"2023-10-30T12:51:30.401679+0000","last_clean":"2023-10-30T12:51:30.401679+0000","last_became_active":"2023-10-30T12:28:50.937458+0000","last_became_peered":"2023-10-30T12:28:50.937458+0000","last_unstale":"2023-10-30T12:51:30.401679+0000","last_undegraded":"2023-10-30T12:51:30.401679+0000","last_fullsized":"2023-10-30T12:51:30.401679+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":572,"log_dups_size":0,"ondisk_log_size":572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003544045,"stat_sum":{"num_bytes":16852009,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":30,"num_read_kb":11,"num_write":518,"num_write_kb":19789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"66'450","reported_seq":554,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:34.515168+0000","last_change":"2023-10-30T12:51:34.515168+0000","last_active":"2023-10-30T12:51:34.515168+0000","last_peered":"2023-10-30T12:51:34.515168+0000","last_clean":"2023-10-30T12:51:34.515168+0000","last_became_active":"2023-10-30T12:28:50.386522+0000","last_became_peered":"2023-10-30T12:28:50.386522+0000","last_unstale":"2023-10-30T12:51:34.515168+0000","last_undegraded":"2023-10-30T12:51:34.515168+0000","last_fullsized":"2023-10-30T12:51:34.515168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'450","last_scrub_stamp":"2023-10-30T12:51:34.515038+0000","last_deep_scrub":"66'450","last_deep_scrub_stamp":"2023-10-30T12:51:34.515038+0000","last_clean_scrub_stamp":"2023-10-30T12:51:34.515038+0000","objects_scrubbed":7,"log_size":450,"log_dups_size":0,"ondisk_log_size":450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:32:31.022257+0000","scrub_duration":0.052865532999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583494,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":64,"num_read_kb":24634,"num_write":377,"num_write_kb":12490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":757298097,"num_objects":1192,"num_object_clones":0,"num_object_copies":2384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1192,"num_whiteouts":0,"num_read":2811,"num_read_kb":100236,"num_write":31900,"num_write_kb":1049744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1519566848,"data_stored":1514596194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35484,"ondisk_log_size":35484,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":305524251,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1441,"num_read_kb":712828,"num_write":10556,"num_write_kb":307084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":78,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":305728,"num_omap_keys":583,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":611213312,"data_stored":611048502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11953,"ondisk_log_size":11953,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542428,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":394020,"kb_used_data":322588,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93977820,"statfs":{"total":96636764160,"available":96233287680,"internally_reserved":0,"allocated":330330112,"data_stored":329162905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":73136587},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.78300000000000003,"15min":0.81799999999999995},"min":{"1min":0.52500000000000002,"5min":0.45900000000000002,"15min":0.40000000000000002},"max":{"1min":1.093,"5min":1.2050000000000001,"15min":1.595},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.78400000000000003,"15min":0.84199999999999997},"min":{"1min":0.49099999999999999,"5min":0.441,"15min":0.40999999999999998},"max":{"1min":1.113,"5min":1.302,"15min":1.4830000000000001},"last":0.78000000000000003}]},{"osd":1,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.85099999999999998,"15min":0.88800000000000001},"min":{"1min":0.57199999999999995,"5min":0.52700000000000002,"15min":0.441},"max":{"1min":1.2709999999999999,"5min":3.4329999999999998,"15min":3.4329999999999998},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.85399999999999998,"15min":0.88500000000000001},"min":{"1min":0.56999999999999995,"5min":0.505,"15min":0.505},"max":{"1min":1.1419999999999999,"5min":1.383,"15min":1.3919999999999999},"last":1.0329999999999999}]},{"osd":2,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.89000000000000001,"15min":0.92200000000000004},"min":{"1min":0.58899999999999997,"5min":0.55100000000000005,"15min":0.45900000000000002},"max":{"1min":1.163,"5min":3.4820000000000002,"15min":3.4820000000000002},"last":1.083},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.877,"15min":0.83699999999999997},"min":{"1min":0.65000000000000002,"5min":0.52800000000000002,"15min":0.47099999999999997},"max":{"1min":1.2330000000000001,"5min":1.4570000000000001,"15min":1.6639999999999999},"last":0.72999999999999998}]},{"osd":3,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":1.0069999999999999,"15min":0.96899999999999997},"min":{"1min":0.61299999999999999,"5min":0.60999999999999999,"15min":0.45500000000000002},"max":{"1min":1.1439999999999999,"5min":3.5190000000000001,"15min":3.5190000000000001},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.95599999999999996,"15min":0.99399999999999999},"min":{"1min":0.72899999999999998,"5min":0.54500000000000004,"15min":0.441},"max":{"1min":1.3220000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.92900000000000005}]},{"osd":4,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.97599999999999998,"15min":1.002},"min":{"1min":0.63800000000000001,"5min":0.42499999999999999,"15min":0.34399999999999997},"max":{"1min":1.2070000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.999,"15min":1.042},"min":{"1min":0.61799999999999999,"5min":0.45000000000000001,"15min":0.33800000000000002},"max":{"1min":1.1910000000000001,"5min":1.746,"15min":1.746},"last":1.0069999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.016,"15min":0.96099999999999997},"min":{"1min":0.76700000000000002,"5min":0.53700000000000003,"15min":0.43099999999999999},"max":{"1min":1.2949999999999999,"5min":1.6160000000000001,"15min":1.7549999999999999},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.117,"15min":1.0680000000000001},"min":{"1min":0.71499999999999997,"5min":0.57899999999999996,"15min":0.35999999999999999},"max":{"1min":1.2370000000000001,"5min":2.1600000000000001,"15min":2.1600000000000001},"last":1.105}]},{"osd":6,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.141,"15min":1.123},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.39000000000000001},"max":{"1min":1.216,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.0620000000000001},{"interface":"front","average":{"1min":1,"5min":1.1259999999999999,"15min":1.167},"min":{"1min":0.68200000000000005,"5min":0.56299999999999994,"15min":0.47199999999999998},"max":{"1min":1.3300000000000001,"5min":3.8140000000000001,"15min":3.8140000000000001},"last":1.149}]}]},{"osd":6,"up_from":14,"seq":60129542427,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298080,"kb_used_data":235032,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94073760,"statfs":{"total":96636764160,"available":96331530240,"internally_reserved":0,"allocated":240672768,"data_stored":239811384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":64551368},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.83299999999999996,"15min":0.85999999999999999},"min":{"1min":0.52900000000000003,"5min":0.46600000000000003,"15min":0.35599999999999998},"max":{"1min":1.155,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":1.085},{"interface":"front","average":{"1min":0.749,"5min":0.77000000000000002,"15min":0.78800000000000003},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.42999999999999999},"max":{"1min":1.05,"5min":1.109,"15min":1.214},"last":0.68400000000000005}]},{"osd":1,"last update":"Mon Oct 30 12:51:28 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.82099999999999995,"15min":0.81599999999999995},"min":{"1min":0.45000000000000001,"5min":0.42299999999999999,"15min":0.317},"max":{"1min":1.23,"5min":1.2809999999999999,"15min":1.3140000000000001},"last":0.63},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.85199999999999998,"15min":0.876},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.40999999999999998},"max":{"1min":1.1399999999999999,"5min":1.2869999999999999,"15min":1.423},"last":0.78000000000000003}]},{"osd":2,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.93600000000000005,"15min":0.97499999999999998},"min":{"1min":0.495,"5min":0.495,"15min":0.38800000000000001},"max":{"1min":1.373,"5min":1.47,"15min":4.3360000000000003},"last":0.90200000000000002},{"interface":"front","average":{"1min":1.0409999999999999,"5min":0.97999999999999998,"15min":1.0089999999999999},"min":{"1min":0.65000000000000002,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.502,"5min":1.502,"15min":1.5629999999999999},"last":0.94799999999999995}]},{"osd":3,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.996,"15min":1.042},"min":{"1min":0.64700000000000002,"5min":0.59099999999999997,"15min":0.58999999999999997},"max":{"1min":1.3340000000000001,"5min":1.5509999999999999,"15min":1.6459999999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.02,"5min":1.0680000000000001,"15min":1.101},"min":{"1min":0.72299999999999998,"5min":0.63300000000000001,"15min":0.59699999999999998},"max":{"1min":1.417,"5min":1.552,"15min":1.7969999999999999},"last":0.73499999999999999}]},{"osd":4,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.149,"15min":1.075},"min":{"1min":0.72599999999999998,"5min":0.69899999999999995,"15min":0.51500000000000001},"max":{"1min":1.585,"5min":1.6499999999999999,"15min":1.7649999999999999},"last":1.0229999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.89400000000000002,"15min":0.96499999999999997},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.42099999999999999},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.881},"last":1.137}]},{"osd":5,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":1.0720000000000001,"15min":1.087},"min":{"1min":0.54100000000000004,"5min":0.53200000000000003,"15min":0.48399999999999999},"max":{"1min":1.2849999999999999,"5min":1.75,"15min":2.2770000000000001},"last":0.86699999999999999},{"interface":"front","average":{"1min":1.056,"5min":1.159,"15min":1.0649999999999999},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.46999999999999997},"max":{"1min":1.4610000000000001,"5min":1.6950000000000001,"15min":4.7850000000000001},"last":1.1120000000000001}]},{"osd":7,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.036,"15min":1.0029999999999999},"min":{"1min":0.68799999999999994,"5min":0.64100000000000001,"15min":0.46700000000000003},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.825},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.0629999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.48999999999999999},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.895},"last":1.052}]}]},{"osd":5,"up_from":14,"seq":60129542428,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":338384,"kb_used_data":271112,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94033456,"statfs":{"total":96636764160,"available":96290258944,"internally_reserved":0,"allocated":277618688,"data_stored":276486603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.78400000000000003,"15min":0.80300000000000005},"min":{"1min":0.56299999999999994,"5min":0.33900000000000002,"15min":0.29399999999999998},"max":{"1min":1.1970000000000001,"5min":1.238,"15min":2.4729999999999999},"last":0.749},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.79700000000000004,"15min":0.80400000000000005},"min":{"1min":0.61199999999999999,"5min":0.39200000000000002,"15min":0.33800000000000002},"max":{"1min":1.2390000000000001,"5min":1.3009999999999999,"15min":1.8759999999999999},"last":0.89600000000000002}]},{"osd":1,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.84199999999999997,"15min":0.84399999999999997},"min":{"1min":0.57399999999999995,"5min":0.40999999999999998,"15min":0.38900000000000001},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":3.0190000000000001},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.58499999999999996,"5min":0.44800000000000001,"15min":0.373},"max":{"1min":1.264,"5min":1.276,"15min":1.4870000000000001},"last":1.1220000000000001}]},{"osd":2,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.88400000000000001,"15min":0.84699999999999998},"min":{"1min":0.67300000000000004,"5min":0.55400000000000005,"15min":0.38400000000000001},"max":{"1min":1.157,"5min":1.264,"15min":2.2949999999999999},"last":1.042},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.81699999999999995,"15min":0.876},"min":{"1min":0.63600000000000001,"5min":0.434,"15min":0.434},"max":{"1min":1.1140000000000001,"5min":1.171,"15min":1.5800000000000001},"last":0.97799999999999998}]},{"osd":3,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.91800000000000004,"15min":0.93700000000000006},"min":{"1min":0.57199999999999995,"5min":0.53800000000000003,"15min":0.47899999999999998},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.756},"last":1.2410000000000001},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.93600000000000005,"15min":0.94599999999999995},"min":{"1min":0.755,"5min":0.498,"15min":0.41399999999999998},"max":{"1min":1.204,"5min":1.4119999999999999,"15min":2.3679999999999999},"last":1.5369999999999999}]},{"osd":4,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.84499999999999997,"15min":0.90900000000000003},"min":{"1min":0.47499999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.304,"5min":1.304,"15min":3.1190000000000002},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.083,"15min":1.091},"min":{"1min":0.59499999999999997,"5min":0.58699999999999997,"15min":0.50900000000000001},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":2.645},"last":1.153}]},{"osd":6,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.038,"15min":1.042},"min":{"1min":0.51100000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.466,"5min":1.466,"15min":3.1859999999999999},"last":1.48},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.056,"15min":1.1060000000000001},"min":{"1min":0.67000000000000004,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.3260000000000001,"5min":1.47,"15min":3.371},"last":1.3260000000000001}]},{"osd":7,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.0609999999999999,"15min":1.0800000000000001},"min":{"1min":0.65200000000000002,"5min":0.51400000000000001,"15min":0.47299999999999998},"max":{"1min":1.494,"5min":1.508,"15min":3.5310000000000001},"last":1.7569999999999999},{"interface":"front","average":{"1min":1.282,"5min":1.1970000000000001,"15min":1.2310000000000001},"min":{"1min":0.96799999999999997,"5min":0.64700000000000002,"15min":0.61299999999999999},"max":{"1min":1.605,"5min":1.605,"15min":3.6930000000000001},"last":1.903}]}]},{"osd":4,"up_from":14,"seq":60129542427,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369460,"kb_used_data":293868,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94002380,"statfs":{"total":96636764160,"available":96258437120,"internally_reserved":0,"allocated":300920832,"data_stored":299840560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:15 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.81799999999999995,"15min":0.92600000000000005},"min":{"1min":0.48099999999999998,"5min":0.34799999999999998,"15min":0.31},"max":{"1min":1.204,"5min":1.2509999999999999,"15min":11.273},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.92400000000000004,"15min":0.98799999999999999},"min":{"1min":0.58099999999999996,"5min":0.44,"15min":0.44},"max":{"1min":1.411,"5min":1.4590000000000001,"15min":7.1289999999999996},"last":0.82199999999999995}]},{"osd":1,"last update":"Mon Oct 30 12:51:15 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.81499999999999995,"15min":0.90500000000000003},"min":{"1min":0.52800000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0049999999999999,"5min":5.1130000000000004,"15min":6.9199999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.871,"5min":0.84199999999999997,"15min":0.90000000000000002},"min":{"1min":0.63900000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.246,"5min":1.246,"15min":4.75},"last":0.98099999999999998}]},{"osd":2,"last update":"Mon Oct 30 12:51:15 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.84199999999999997,"15min":0.84899999999999998},"min":{"1min":0.51700000000000002,"5min":0.41899999999999998,"15min":0.36899999999999999},"max":{"1min":1.175,"5min":1.21,"15min":2.6970000000000001},"last":1.073},{"interface":"front","average":{"1min":0.996,"5min":0.91200000000000003,"15min":1.0569999999999999},"min":{"1min":0.60999999999999999,"5min":0.54400000000000004,"15min":0.52900000000000003},"max":{"1min":1.361,"5min":1.361,"15min":18.917999999999999},"last":1.1299999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":0.96599999999999997,"15min":0.997},"min":{"1min":0.71399999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.258,"5min":5.1719999999999997,"15min":7.1879999999999997},"last":1.179},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.90800000000000003,"15min":1},"min":{"1min":0.57399999999999995,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.125,"5min":1.3600000000000001,"15min":8.3360000000000003},"last":0.69199999999999995}]},{"osd":5,"last update":"Mon Oct 30 12:51:15 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.88300000000000001,"15min":1.002},"min":{"1min":0.56299999999999994,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.159,"5min":1.3300000000000001,"15min":6.0759999999999996},"last":0.63900000000000001},{"interface":"front","average":{"1min":1.119,"5min":1.105,"15min":1.1890000000000001},"min":{"1min":0.89500000000000002,"5min":0.71899999999999997,"15min":0.52600000000000002},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":18.994},"last":1.0249999999999999}]},{"osd":6,"last update":"Mon Oct 30 12:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.0880000000000001,"15min":1.093},"min":{"1min":0.70199999999999996,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.429,"5min":1.429,"15min":6.1580000000000004},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.016,"5min":1.0169999999999999,"15min":1.1240000000000001},"min":{"1min":0.629,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.28,"5min":5.218,"15min":7.0549999999999997},"last":0.872}]},{"osd":7,"last update":"Mon Oct 30 12:51:15 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.0980000000000001,"15min":1.258},"min":{"1min":0.75800000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.4339999999999999,"5min":5.7160000000000002,"15min":18.015999999999998},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.236,"5min":1.2050000000000001,"15min":1.2769999999999999},"min":{"1min":1.093,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.4530000000000001,"5min":5.6609999999999996,"15min":19.052},"last":1.222}]}]},{"osd":3,"up_from":14,"seq":60129542428,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391212,"kb_used_data":310052,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93980628,"statfs":{"total":96636764160,"available":96236163072,"internally_reserved":0,"allocated":317493248,"data_stored":316374505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":83098056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.39900000000000002,"15min":0.41399999999999998},"min":{"1min":0.29399999999999998,"5min":0.23699999999999999,"15min":0.23400000000000001},"max":{"1min":0.58999999999999997,"5min":0.77900000000000003,"15min":1.6899999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51600000000000001,"15min":0.53300000000000003},"min":{"1min":0.40000000000000002,"5min":0.34300000000000003,"15min":0.32700000000000001},"max":{"1min":0.746,"5min":0.871,"15min":1.728},"last":0.55700000000000005}]},{"osd":1,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.42899999999999999,"15min":0.42999999999999999},"min":{"1min":0.309,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":1.2909999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.496,"15min":0.50600000000000001},"min":{"1min":0.34499999999999997,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.64600000000000002,"5min":0.80600000000000005,"15min":1.3260000000000001},"last":0.51900000000000002}]},{"osd":2,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.496,"15min":0.502},"min":{"1min":0.37,"5min":0.30099999999999999,"15min":0.29299999999999998},"max":{"1min":0.69899999999999995,"5min":0.78600000000000003,"15min":1.099},"last":0.501},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55900000000000005,"15min":0.56699999999999995},"min":{"1min":0.41999999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73599999999999999,"5min":0.82599999999999996,"15min":1.2509999999999999},"last":0.53900000000000003}]},{"osd":4,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.85199999999999998,"15min":0.83099999999999996},"min":{"1min":0.63900000000000001,"5min":0.55400000000000005,"15min":0.40400000000000003},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":2.202},"last":1.01},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.82999999999999996,"15min":0.86499999999999999},"min":{"1min":0.70599999999999996,"5min":0.48499999999999999,"15min":0.46800000000000003},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":2.4340000000000002},"last":0.95499999999999996}]},{"osd":5,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.81499999999999995,"15min":0.86299999999999999},"min":{"1min":0.66200000000000003,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.169,"5min":1.169,"15min":14.103999999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.875,"5min":0.876,"15min":0.86499999999999999},"min":{"1min":0.63900000000000001,"5min":0.55000000000000004,"15min":0.499},"max":{"1min":1.2170000000000001,"5min":1.2869999999999999,"15min":4.4859999999999998},"last":0.872}]},{"osd":6,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.85299999999999998,"15min":0.83499999999999996},"min":{"1min":0.64700000000000002,"5min":0.56999999999999995,"15min":0.53100000000000003},"max":{"1min":1.087,"5min":1.2450000000000001,"15min":2.2330000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.87,"15min":0.82899999999999996},"min":{"1min":0.77800000000000002,"5min":0.54100000000000004,"15min":0.45100000000000001},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":4.4409999999999998},"last":0.97499999999999998}]},{"osd":7,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.875,"15min":0.86499999999999999},"min":{"1min":0.67500000000000004,"5min":0.57299999999999995,"15min":0.57099999999999995},"max":{"1min":1.1499999999999999,"5min":1.2509999999999999,"15min":3.5459999999999998},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.86399999999999999,"15min":0.84699999999999998},"min":{"1min":0.65700000000000003,"5min":0.57399999999999995,"15min":0.45700000000000002},"max":{"1min":1.1259999999999999,"5min":1.2230000000000001,"15min":6.1360000000000001},"last":0.89400000000000002}]}]},{"osd":2,"up_from":14,"seq":60129542428,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":200604,"kb_used_data":143060,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94171236,"statfs":{"total":96636764160,"available":96431345664,"internally_reserved":0,"allocated":146493440,"data_stored":145675964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.45100000000000001,"15min":0.441},"min":{"1min":0.25,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.59499999999999997,"5min":0.69399999999999995,"15min":0.81200000000000006},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.5,"15min":0.51100000000000001},"min":{"1min":0.37,"5min":0.33300000000000002,"15min":0.31900000000000001},"max":{"1min":0.63,"5min":0.71299999999999997,"15min":1.298},"last":0.53300000000000003}]},{"osd":1,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.47899999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.60199999999999998,"5min":0.65500000000000003,"15min":0.83699999999999997},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.52500000000000002,"15min":0.504},"min":{"1min":0.38900000000000001,"5min":0.33700000000000002,"15min":0.29999999999999999},"max":{"1min":0.61699999999999999,"5min":0.74099999999999999,"15min":1.2250000000000001},"last":0.60199999999999998}]},{"osd":3,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.47399999999999998,"15min":0.54800000000000004},"min":{"1min":0.315,"5min":0.311,"15min":0.311},"max":{"1min":0.58599999999999997,"5min":0.75,"15min":1.673},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57299999999999995,"15min":0.57399999999999995},"min":{"1min":0.41799999999999998,"5min":0.40200000000000002,"15min":0.35699999999999998},"max":{"1min":0.67900000000000005,"5min":0.79100000000000004,"15min":1.597},"last":0.72299999999999998}]},{"osd":4,"last update":"Mon Oct 30 12:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.74299999999999999,"15min":0.82599999999999996},"min":{"1min":0.47499999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.91100000000000003,"5min":1.0569999999999999,"15min":1.377},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.75,"5min":0.755,"15min":0.79200000000000004},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.998,"5min":1.0640000000000001,"15min":1.4159999999999999},"last":0.98999999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.84099999999999997,"15min":0.84799999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.093,"5min":1.2430000000000001,"15min":1.78},"last":1.0049999999999999},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91400000000000003},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.1060000000000001,"5min":1.2729999999999999,"15min":1.7430000000000001},"last":0.95199999999999996}]},{"osd":6,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.86199999999999999,"15min":0.85799999999999998},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.60299999999999998},"max":{"1min":1.095,"5min":1.177,"15min":1.3799999999999999},"last":1.048},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.86599999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.53300000000000003},"max":{"1min":1.097,"5min":1.1799999999999999,"15min":1.8109999999999999},"last":0.97799999999999998}]},{"osd":7,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.82899999999999996,"15min":0.83099999999999996},"min":{"1min":0.58799999999999997,"5min":0.58299999999999996,"15min":0.48799999999999999},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.4159999999999999},"last":1.022},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.85399999999999998,"15min":0.83299999999999996},"min":{"1min":0.75900000000000001,"5min":0.51800000000000002,"15min":0.50800000000000001},"max":{"1min":1.206,"5min":1.206,"15min":1.3049999999999999},"last":0.96399999999999997}]}]},{"osd":1,"up_from":14,"seq":60129542428,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":279036,"kb_used_data":233972,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94092804,"statfs":{"total":96636764160,"available":96351031296,"internally_reserved":0,"allocated":239587328,"data_stored":238747727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:38 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.41399999999999998,"15min":0.45700000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.24299999999999999},"max":{"1min":0.65100000000000002,"5min":0.65200000000000002,"15min":11.958},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.497,"15min":0.55500000000000005},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.751,"5min":0.751,"15min":12.077},"last":0.42299999999999999}]},{"osd":2,"last update":"Mon Oct 30 12:51:38 2023","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.46200000000000002,"15min":0.45800000000000002},"min":{"1min":0.32700000000000001,"5min":0.29699999999999999,"15min":0.26400000000000001},"max":{"1min":0.54100000000000004,"5min":1.121,"15min":1.623},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.53200000000000003,"15min":0.53100000000000003},"min":{"1min":0.37,"5min":0.34200000000000003,"15min":0.31900000000000001},"max":{"1min":0.84499999999999997,"5min":1.1699999999999999,"15min":1.5389999999999999},"last":0.53400000000000003}]},{"osd":3,"last update":"Mon Oct 30 12:51:38 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56499999999999995,"15min":0.56599999999999995},"min":{"1min":0.40999999999999998,"5min":0.378,"15min":0.31},"max":{"1min":0.72099999999999997,"5min":1.337,"15min":3.694},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.504,"15min":0.53700000000000003},"min":{"1min":0.308,"5min":0.307,"15min":0.307},"max":{"1min":0.69699999999999995,"5min":0.77200000000000002,"15min":13.199999999999999},"last":0.50900000000000001}]},{"osd":4,"last update":"Mon Oct 30 12:51:38 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.82299999999999995,"15min":0.82699999999999996},"min":{"1min":0.54700000000000004,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.94599999999999995,"5min":1.3140000000000001,"15min":2.419},"last":0.97399999999999998},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.873,"15min":0.86499999999999999},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.52100000000000002},"max":{"1min":1.012,"5min":1.103,"15min":2.282},"last":1.002}]},{"osd":5,"last update":"Mon Oct 30 12:51:38 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.79800000000000004,"15min":0.81399999999999995},"min":{"1min":0.54100000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.95799999999999996,"5min":1.343,"15min":1.343},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.79000000000000004,"15min":0.84799999999999998},"min":{"1min":0.51700000000000002,"5min":0.50800000000000001,"15min":0.44900000000000001},"max":{"1min":0.98999999999999999,"5min":1.2809999999999999,"15min":1.29},"last":0.61699999999999999}]},{"osd":6,"last update":"Mon Oct 30 12:51:38 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.83399999999999996,"15min":0.80000000000000004},"min":{"1min":0.56200000000000006,"5min":0.52800000000000002,"15min":0.51200000000000001},"max":{"1min":1.1279999999999999,"5min":1.371,"15min":1.371},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.91200000000000003,"15min":0.86699999999999999},"min":{"1min":0.628,"5min":0.62,"15min":0.498},"max":{"1min":1.173,"5min":1.417,"15min":2.464},"last":0.83999999999999997}]},{"osd":7,"last update":"Mon Oct 30 12:51:38 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.80300000000000005,"15min":0.82399999999999995},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.075,"5min":1.1659999999999999,"15min":2.5169999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.88500000000000001,"15min":0.90100000000000002},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.1140000000000001,"5min":1.133,"15min":2.5510000000000002},"last":0.80200000000000005}]}]},{"osd":0,"up_from":14,"seq":60129542428,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":349720,"kb_used_data":278288,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94022120,"statfs":{"total":96636764160,"available":96278650880,"internally_reserved":0,"allocated":284966912,"data_stored":283825495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 12:51:29 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.45700000000000002,"15min":0.45200000000000001},"min":{"1min":0.28699999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.64600000000000002,"5min":0.71499999999999997,"15min":0.93700000000000006},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51500000000000001,"15min":0.50700000000000001},"min":{"1min":0.38700000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.73799999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.45100000000000001}]},{"osd":2,"last update":"Mon Oct 30 12:51:29 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45600000000000002,"15min":0.502},"min":{"1min":0.315,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.97199999999999998},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56000000000000005,"15min":0.53300000000000003},"min":{"1min":0.437,"5min":0.36799999999999999,"15min":0.27400000000000002},"max":{"1min":0.79000000000000004,"5min":0.83299999999999996,"15min":1.0149999999999999},"last":0.60699999999999998}]},{"osd":3,"last update":"Mon Oct 30 12:51:29 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47899999999999998,"15min":0.51300000000000001},"min":{"1min":0.36099999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.67100000000000004,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.5,"15min":0.497},"min":{"1min":0.41099999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.71199999999999997,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.51300000000000001}]},{"osd":4,"last update":"Mon Oct 30 12:51:29 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.87,"15min":0.85299999999999998},"min":{"1min":0.57699999999999996,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1899999999999999,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.877,"15min":0.85899999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.48299999999999998},"max":{"1min":1.0189999999999999,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.78700000000000003}]},{"osd":5,"last update":"Mon Oct 30 12:51:29 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.81399999999999995,"15min":0.83299999999999996},"min":{"1min":0.55200000000000005,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.163,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83699999999999997,"15min":0.86799999999999999},"min":{"1min":0.68999999999999995,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0289999999999999,"5min":1.889,"15min":1.889},"last":0.84399999999999997}]},{"osd":6,"last update":"Mon Oct 30 12:51:29 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80300000000000005,"15min":0.79500000000000004},"min":{"1min":0.63100000000000001,"5min":0.56299999999999994,"15min":0.52200000000000002},"max":{"1min":1.071,"5min":1.1990000000000001,"15min":2.6480000000000001},"last":0.747},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.80100000000000005,"15min":0.81999999999999995},"min":{"1min":0.60999999999999999,"5min":0.48799999999999999,"15min":0.46999999999999997},"max":{"1min":1.071,"5min":1.1559999999999999,"15min":5.0099999999999998},"last":0.91600000000000004}]},{"osd":7,"last update":"Mon Oct 30 12:51:29 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.872,"15min":0.88300000000000001},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.55000000000000004},"max":{"1min":1.25,"5min":1.401,"15min":3.3359999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.93899999999999995,"15min":0.94799999999999995},"min":{"1min":0.73899999999999999,"5min":0.65100000000000002,"15min":0.56299999999999994},"max":{"1min":1.2729999999999999,"5min":1.3500000000000001,"15min":3.3580000000000001},"last":0.89800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62930944,"data_stored":62915220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":83910656,"data_stored":83887024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":37969920,"data_stored":37945198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":83726336,"data_stored":83700311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":76107776,"data_stored":76093397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":62935040,"data_stored":62915938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":91439104,"data_stored":91416759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":112193536,"data_stored":112174655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":220680192,"data_stored":219932830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":154910720,"data_stored":154473007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":107761664,"data_stored":107343689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":233000960,"data_stored":232286498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":224047104,"data_stored":223359467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":213921792,"data_stored":213183588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":148467712,"data_stored":148006929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":216776704,"data_stored":216010186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:51:58.292 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.292 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.293 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.293 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.294 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.294 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.294 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.295 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.295 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.296 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.296 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.296 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.297 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.297 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.297 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.298 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.298 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.299 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.299 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.299 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.300 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.300 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.301 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.301 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.301 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.302 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.302 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.303 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.303 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.304 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.304 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.304 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.305 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.305 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.306 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.306 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.306 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-30T12:28:49.324113+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=49, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.307 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.307 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.308 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.308 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.309 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.309 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.309 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:51:58.310 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T12:52:18.311 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:52:18.463 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.460+0000 7fc6e349f640 1 Processor -- start 2023-10-30T12:52:18.464 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e349f640 1 -- start start 2023-10-30T12:52:18.465 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e349f640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc6dc15cd50 0x7fc6dc0c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:18.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e349f640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6dc15a6c0 con 0x7fc6dc1538f0 2023-10-30T12:52:18.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e349f640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6dc15d300 con 0x7fc6dc0cefa0 2023-10-30T12:52:18.466 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e349f640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6dc15d5e0 con 0x7fc6dc15cd50 2023-10-30T12:52:18.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e249d640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc6dc1538f0 0x7fc6dc159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:40444/0 (socket says 172.21.15.89:40444) 2023-10-30T12:52:18.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e249d640 1 -- 172.21.15.89:0/3482833999 learned_addr learned my addr 172.21.15.89:0/3482833999 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:52:18.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e1c9c640 1 --2- 172.21.15.89:0/3482833999 >> v2:172.21.15.192:3300/0 conn(0x7fc6dc15cd50 0x7fc6dc0c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:18.467 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e149b640 1 -- 172.21.15.89:0/3482833999 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 181771345 0 0) 0x7fc6dc15a6c0 con 0x7fc6dc1538f0 2023-10-30T12:52:18.468 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e149b640 1 -- 172.21.15.89:0/3482833999 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6c8003530 con 0x7fc6dc1538f0 2023-10-30T12:52:18.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e149b640 1 -- 172.21.15.89:0/3482833999 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3439904622 0 0) 0x7fc6c8003530 con 0x7fc6dc1538f0 2023-10-30T12:52:18.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e149b640 1 -- 172.21.15.89:0/3482833999 >> v2:172.21.15.192:3300/0 conn(0x7fc6dc15cd50 msgr2=0x7fc6dc0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:18.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e149b640 1 --2- 172.21.15.89:0/3482833999 >> v2:172.21.15.192:3300/0 conn(0x7fc6dc15cd50 0x7fc6dc0c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:18.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e149b640 1 -- 172.21.15.89:0/3482833999 >> v1:172.21.15.192:6789/0 conn(0x7fc6dc0cefa0 legacy=0x7fc6dc1640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:18.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e149b640 1 -- 172.21.15.89:0/3482833999 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6dc15d8c0 con 0x7fc6dc1538f0 2023-10-30T12:52:18.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e149b640 1 -- 172.21.15.89:0/3482833999 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7fc6d8001680 con 0x7fc6dc1538f0 2023-10-30T12:52:18.471 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e149b640 1 -- 172.21.15.89:0/3482833999 >> v1:172.21.15.89:6789/0 conn(0x7fc6dc1538f0 legacy=0x7fc6dc159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:18.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e149b640 1 --2- 172.21.15.89:0/3482833999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6c8003980 0x7fc6c8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:18.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e149b640 1 --2- 172.21.15.89:0/3482833999 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6c80047d0 0x7fc6c8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:18.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e149b640 1 --2- 172.21.15.89:0/3482833999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6c8005190 0x7fc6c800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:18.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e149b640 1 -- 172.21.15.89:0/3482833999 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc15d5e0 con 0x7fc6c8003980 2023-10-30T12:52:18.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e149b640 1 -- 172.21.15.89:0/3482833999 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c800c040 con 0x7fc6c8005190 2023-10-30T12:52:18.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e149b640 1 -- 172.21.15.89:0/3482833999 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c800c200 con 0x7fc6c80047d0 2023-10-30T12:52:18.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e249d640 1 --2- 172.21.15.89:0/3482833999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6c8003980 0x7fc6c8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:18.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e2c9e640 1 --2- 172.21.15.89:0/3482833999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6c8005190 0x7fc6c800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:18.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e1c9c640 1 --2- 172.21.15.89:0/3482833999 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6c80047d0 0x7fc6c8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:18.474 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e249d640 1 -- 172.21.15.89:0/3482833999 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6c80047d0 msgr2=0x7fc6c8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:18.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e249d640 1 --2- 172.21.15.89:0/3482833999 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6c80047d0 0x7fc6c8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:18.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e249d640 1 -- 172.21.15.89:0/3482833999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6c8005190 msgr2=0x7fc6c800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:18.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e249d640 1 --2- 172.21.15.89:0/3482833999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6c8005190 0x7fc6c800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:18.475 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e249d640 1 -- 172.21.15.89:0/3482833999 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c8003530 con 0x7fc6c8003980 2023-10-30T12:52:18.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e2c9e640 1 --2- 172.21.15.89:0/3482833999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6c8005190 0x7fc6c800b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:52:18.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.464+0000 7fc6e2c9e640 1 -- 172.21.15.89:0/3482833999 reap_dead start 2023-10-30T12:52:18.476 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.468+0000 7fc6e249d640 1 --2- 172.21.15.89:0/3482833999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6c8003980 0x7fc6c8003da0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fc6d800c240 tx=0x7fc6d801f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d980daf1cf585b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:18.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.468+0000 7fc6e149b640 1 -- 172.21.15.89:0/3482833999 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8005ae0 con 0x7fc6c8003980 2023-10-30T12:52:18.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.468+0000 7fc6e149b640 1 -- 172.21.15.89:0/3482833999 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d80ab070 con 0x7fc6c8003980 2023-10-30T12:52:18.477 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.468+0000 7fc6e149b640 1 -- 172.21.15.89:0/3482833999 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8007070 con 0x7fc6c8003980 2023-10-30T12:52:18.480 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.468+0000 7fc6e349f640 1 -- 172.21.15.89:0/3482833999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6c8003980 msgr2=0x7fc6c8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:18.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.468+0000 7fc6e349f640 1 --2- 172.21.15.89:0/3482833999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6c8003980 0x7fc6c8003da0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fc6d800c240 tx=0x7fc6d801f3b0 comp rx=0 tx=0).stop 2023-10-30T12:52:18.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.468+0000 7fc6e349f640 1 -- 172.21.15.89:0/3482833999 shutdown_connections 2023-10-30T12:52:18.481 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.468+0000 7fc6e349f640 1 --2- 172.21.15.89:0/3482833999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6c8003980 0x7fc6c8003da0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:18.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.468+0000 7fc6e349f640 1 -- 172.21.15.89:0/3482833999 >> 172.21.15.89:0/3482833999 conn(0x7fc6dc06a640 msgr2=0x7fc6dc065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:18.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.468+0000 7fc6e349f640 1 -- 172.21.15.89:0/3482833999 shutdown_connections 2023-10-30T12:52:18.482 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.468+0000 7fc6e349f640 1 -- 172.21.15.89:0/3482833999 wait complete. 2023-10-30T12:52:18.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.472+0000 7fc6e349f640 1 Processor -- start 2023-10-30T12:52:18.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.472+0000 7fc6e349f640 1 -- start start 2023-10-30T12:52:18.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.472+0000 7fc6e349f640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6c8003980 0x7fc6dc14ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:18.483 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.472+0000 7fc6e349f640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6dc1f3210 0x7fc6dc15aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:18.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.472+0000 7fc6e349f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6dc1f0700 0x7fc6dc06c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:18.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.472+0000 7fc6e349f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc14f240 con 0x7fc6dc1f0700 2023-10-30T12:52:18.484 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.472+0000 7fc6e349f640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc14fa70 con 0x7fc6c8003980 2023-10-30T12:52:18.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.472+0000 7fc6e349f640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc14f4b0 con 0x7fc6dc1f3210 2023-10-30T12:52:18.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.472+0000 7fc6e2c9e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6dc1f0700 0x7fc6dc06c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:18.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.472+0000 7fc6e2c9e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6dc1f0700 0x7fc6dc06c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55540/0 (socket says 172.21.15.89:55540) 2023-10-30T12:52:18.485 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.472+0000 7fc6e2c9e640 1 -- 172.21.15.89:0/3124589556 learned_addr learned my addr 172.21.15.89:0/3124589556 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:52:18.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.472+0000 7fc6e249d640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6c8003980 0x7fc6dc14ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:18.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.472+0000 7fc6e1c9c640 1 --2- 172.21.15.89:0/3124589556 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6dc1f3210 0x7fc6dc15aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:18.486 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.472+0000 7fc6e2c9e640 1 -- 172.21.15.89:0/3124589556 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6dc1f3210 msgr2=0x7fc6dc15aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:18.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.472+0000 7fc6e2c9e640 1 --2- 172.21.15.89:0/3124589556 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6dc1f3210 0x7fc6dc15aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:18.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.472+0000 7fc6e2c9e640 1 -- 172.21.15.89:0/3124589556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6c8003980 msgr2=0x7fc6dc14ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:18.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.472+0000 7fc6e2c9e640 1 --2- 172.21.15.89:0/3124589556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6c8003980 0x7fc6dc14ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:18.487 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.472+0000 7fc6e2c9e640 1 -- 172.21.15.89:0/3124589556 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6dc1f3700 con 0x7fc6dc1f0700 2023-10-30T12:52:18.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.472+0000 7fc6e2c9e640 1 --2- 172.21.15.89:0/3124589556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6dc1f0700 0x7fc6dc06c380 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc066500 tx=0x7fc6d400eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:18.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.472+0000 7fc6d2ffd640 1 -- 172.21.15.89:0/3124589556 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d400eed0 con 0x7fc6dc1f0700 2023-10-30T12:52:18.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.472+0000 7fc6d2ffd640 1 -- 172.21.15.89:0/3124589556 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d4003980 con 0x7fc6dc1f0700 2023-10-30T12:52:18.488 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.472+0000 7fc6d2ffd640 1 -- 172.21.15.89:0/3124589556 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d4003d10 con 0x7fc6dc1f0700 2023-10-30T12:52:18.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.472+0000 7fc6e349f640 1 -- 172.21.15.89:0/3124589556 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6dc15c750 con 0x7fc6dc1f0700 2023-10-30T12:52:18.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.472+0000 7fc6e349f640 1 -- 172.21.15.89:0/3124589556 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6dc14f700 con 0x7fc6dc1f0700 2023-10-30T12:52:18.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.476+0000 7fc6e349f640 1 -- 172.21.15.89:0/3124589556 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6dc04e170 con 0x7fc6dc1f0700 2023-10-30T12:52:18.489 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.476+0000 7fc6d2ffd640 1 -- 172.21.15.89:0/3124589556 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc6d40050d0 con 0x7fc6dc1f0700 2023-10-30T12:52:18.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.476+0000 7fc6d2ffd640 1 --2- 172.21.15.89:0/3124589556 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc6c00664a0 0x7fc6c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:18.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.476+0000 7fc6d2ffd640 1 -- 172.21.15.89:0/3124589556 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7fc6d4083d90 con 0x7fc6dc1f0700 2023-10-30T12:52:18.490 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.476+0000 7fc6e249d640 1 --2- 172.21.15.89:0/3124589556 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc6c00664a0 0x7fc6c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:18.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.476+0000 7fc6e249d640 1 --2- 172.21.15.89:0/3124589556 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc6c00664a0 0x7fc6c0068960 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc6d80af000 tx=0x7fc6d80aa000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:18.491 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.480+0000 7fc6d2ffd640 1 -- 172.21.15.89:0/3124589556 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc6d4050f70 con 0x7fc6dc1f0700 2023-10-30T12:52:18.630 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.628+0000 7fc6e349f640 1 -- 172.21.15.89:0/3124589556 --> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc6dc159a90 con 0x7fc6c00664a0 2023-10-30T12:52:18.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.648+0000 7fc6d2ffd640 1 -- 172.21.15.89:0/3124589556 <== mgr.4105 v2:172.21.15.192:6832/4076276638 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+294139 (secure 0 0 0) 0x7fc6d80047f0 con 0x7fc6c00664a0 2023-10-30T12:52:18.654 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:52:18.656 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-10-30T12:52:18.657 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.652+0000 7fc6e349f640 1 -- 172.21.15.89:0/3124589556 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc6c00664a0 msgr2=0x7fc6c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:18.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.652+0000 7fc6e349f640 1 --2- 172.21.15.89:0/3124589556 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc6c00664a0 0x7fc6c0068960 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc6d80af000 tx=0x7fc6d80aa000 comp rx=0 tx=0).stop 2023-10-30T12:52:18.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.652+0000 7fc6e349f640 1 -- 172.21.15.89:0/3124589556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6dc1f0700 msgr2=0x7fc6dc06c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:18.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.652+0000 7fc6e349f640 1 --2- 172.21.15.89:0/3124589556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6dc1f0700 0x7fc6dc06c380 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc066500 tx=0x7fc6d400eb90 comp rx=0 tx=0).stop 2023-10-30T12:52:18.658 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.652+0000 7fc6e349f640 1 -- 172.21.15.89:0/3124589556 shutdown_connections 2023-10-30T12:52:18.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.652+0000 7fc6e349f640 1 --2- 172.21.15.89:0/3124589556 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fc6c00664a0 0x7fc6c0068960 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:18.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.652+0000 7fc6e349f640 1 --2- 172.21.15.89:0/3124589556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc6dc1f0700 0x7fc6dc06c380 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:18.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.652+0000 7fc6e349f640 1 --2- 172.21.15.89:0/3124589556 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc6dc1f3210 0x7fc6dc15aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:18.659 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.652+0000 7fc6e349f640 1 --2- 172.21.15.89:0/3124589556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc6c8003980 0x7fc6dc14ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:18.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.652+0000 7fc6e349f640 1 -- 172.21.15.89:0/3124589556 >> 172.21.15.89:0/3124589556 conn(0x7fc6dc06a640 msgr2=0x7fc6dc158bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:18.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.652+0000 7fc6e349f640 1 -- 172.21.15.89:0/3124589556 shutdown_connections 2023-10-30T12:52:18.660 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:18.652+0000 7fc6e349f640 1 -- 172.21.15.89:0/3124589556 wait complete. 2023-10-30T12:52:18.675 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":769,"stamp":"2023-10-30T12:52:17.446121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1063570331,"num_objects":1371,"num_object_clones":0,"num_object_copies":2742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1371,"num_whiteouts":0,"num_read":4368,"num_read_kb":813164,"num_write":42623,"num_write_kb":1359402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":78,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":305728,"num_omap_keys":583,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48148,"ondisk_log_size":48148,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2624980,"kb_used_data":2088276,"kb_used_omap":12,"kb_used_meta":536627,"kb_avail":752349740,"statfs":{"total":773094113280,"available":770406133760,"internally_reserved":0,"allocated":2138394624,"data_stored":2130240373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12741,"internal_metadata":549506619},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":93646,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004209"},"pg_stats":[{"pgid":"3.3f","version":"66'572","reported_seq":2734,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:15.560856+0000","last_change":"2023-10-30T12:52:15.561063+0000","last_active":"2023-10-30T12:52:15.560856+0000","last_peered":"2023-10-30T12:52:15.560856+0000","last_clean":"2023-10-30T12:52:15.560856+0000","last_became_active":"2023-10-30T12:28:50.979351+0000","last_became_peered":"2023-10-30T12:28:50.979351+0000","last_unstale":"2023-10-30T12:52:15.560856+0000","last_undegraded":"2023-10-30T12:52:15.560856+0000","last_fullsized":"2023-10-30T12:52:15.560856+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'572","last_scrub_stamp":"2023-10-30T12:52:15.560583+0000","last_deep_scrub":"66'572","last_deep_scrub_stamp":"2023-10-30T12:52:15.560583+0000","last_clean_scrub_stamp":"2023-10-30T12:52:15.560583+0000","objects_scrubbed":21,"log_size":572,"log_dups_size":0,"ondisk_log_size":572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:47:12.132657+0000","scrub_duration":0.060883743999999997,"objects_trimmed":0,"snaptrim_duration":0.0091708599999999994,"stat_sum":{"num_bytes":4321961,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":40,"num_read_kb":15,"num_write":493,"num_write_kb":5417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"66'570","reported_seq":2725,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:02.815587+0000","last_change":"2023-10-30T12:52:02.815654+0000","last_active":"2023-10-30T12:52:02.815587+0000","last_peered":"2023-10-30T12:52:02.815587+0000","last_clean":"2023-10-30T12:52:02.815587+0000","last_became_active":"2023-10-30T12:28:50.929133+0000","last_became_peered":"2023-10-30T12:28:50.929133+0000","last_unstale":"2023-10-30T12:52:02.815587+0000","last_undegraded":"2023-10-30T12:52:02.815587+0000","last_fullsized":"2023-10-30T12:52:02.815587+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'570","last_scrub_stamp":"2023-10-30T12:52:02.815504+0000","last_deep_scrub":"66'570","last_deep_scrub_stamp":"2023-10-30T12:52:02.815504+0000","last_clean_scrub_stamp":"2023-10-30T12:52:02.815504+0000","objects_scrubbed":19,"log_size":570,"log_dups_size":0,"ondisk_log_size":570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:57:55.912117+0000","scrub_duration":0.032094442000000001,"objects_trimmed":0,"snaptrim_duration":0.0023149070000000002,"stat_sum":{"num_bytes":8426607,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":33,"num_read_kb":10,"num_write":489,"num_write_kb":9508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"66'568","reported_seq":2739,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:12.337570+0000","last_change":"2023-10-30T12:52:12.337656+0000","last_active":"2023-10-30T12:52:12.337570+0000","last_peered":"2023-10-30T12:52:12.337570+0000","last_clean":"2023-10-30T12:52:12.337570+0000","last_became_active":"2023-10-30T12:28:50.933896+0000","last_became_peered":"2023-10-30T12:28:50.933896+0000","last_unstale":"2023-10-30T12:52:12.337570+0000","last_undegraded":"2023-10-30T12:52:12.337570+0000","last_fullsized":"2023-10-30T12:52:12.337570+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'568","last_scrub_stamp":"2023-10-30T12:52:12.337465+0000","last_deep_scrub":"66'568","last_deep_scrub_stamp":"2023-10-30T12:52:12.337465+0000","last_clean_scrub_stamp":"2023-10-30T12:52:12.337465+0000","objects_scrubbed":18,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:24:10.063858+0000","scrub_duration":0.040558903,"objects_trimmed":0,"snaptrim_duration":0.0029033520000000001,"stat_sum":{"num_bytes":12697764,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":49,"num_read_kb":8209,"num_write":487,"num_write_kb":25783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"66'517","reported_seq":2661,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.473760+0000","last_change":"2023-10-30T12:38:17.117873+0000","last_active":"2023-10-30T12:51:30.473760+0000","last_peered":"2023-10-30T12:51:30.473760+0000","last_clean":"2023-10-30T12:51:30.473760+0000","last_became_active":"2023-10-30T12:28:50.967774+0000","last_became_peered":"2023-10-30T12:28:50.967774+0000","last_unstale":"2023-10-30T12:51:30.473760+0000","last_undegraded":"2023-10-30T12:51:30.473760+0000","last_fullsized":"2023-10-30T12:51:30.473760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075479680000000004,"stat_sum":{"num_bytes":4272818,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":30,"num_read_kb":13,"num_write":479,"num_write_kb":5884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"66'553","reported_seq":2720,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:39.811881+0000","last_change":"2023-10-30T12:51:39.812135+0000","last_active":"2023-10-30T12:51:39.811881+0000","last_peered":"2023-10-30T12:51:39.811881+0000","last_clean":"2023-10-30T12:51:39.811881+0000","last_became_active":"2023-10-30T12:28:50.973132+0000","last_became_peered":"2023-10-30T12:28:50.973132+0000","last_unstale":"2023-10-30T12:51:39.811881+0000","last_undegraded":"2023-10-30T12:51:39.811881+0000","last_fullsized":"2023-10-30T12:51:39.811881+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'553","last_scrub_stamp":"2023-10-30T12:51:39.811639+0000","last_deep_scrub":"66'553","last_deep_scrub_stamp":"2023-10-30T12:51:39.811639+0000","last_clean_scrub_stamp":"2023-10-30T12:51:39.811639+0000","objects_scrubbed":18,"log_size":553,"log_dups_size":0,"ondisk_log_size":553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:25:28.276273+0000","scrub_duration":0.055821471999999997,"objects_trimmed":0,"snaptrim_duration":0.0090421609999999999,"stat_sum":{"num_bytes":12692456,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":47,"num_read_kb":16,"num_write":494,"num_write_kb":14525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"66'551","reported_seq":2701,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.474891+0000","last_change":"2023-10-30T12:38:17.112051+0000","last_active":"2023-10-30T12:51:30.474891+0000","last_peered":"2023-10-30T12:51:30.474891+0000","last_clean":"2023-10-30T12:51:30.474891+0000","last_became_active":"2023-10-30T12:28:50.967910+0000","last_became_peered":"2023-10-30T12:28:50.967910+0000","last_unstale":"2023-10-30T12:51:30.474891+0000","last_undegraded":"2023-10-30T12:51:30.474891+0000","last_fullsized":"2023-10-30T12:51:30.474891+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":551,"log_dups_size":0,"ondisk_log_size":551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040413009999999997,"stat_sum":{"num_bytes":12623095,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":38,"num_read_kb":16,"num_write":503,"num_write_kb":14521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"66'569","reported_seq":2730,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:55.514716+0000","last_change":"2023-10-30T12:51:55.514849+0000","last_active":"2023-10-30T12:51:55.514716+0000","last_peered":"2023-10-30T12:51:55.514716+0000","last_clean":"2023-10-30T12:51:55.514716+0000","last_became_active":"2023-10-30T12:28:50.973288+0000","last_became_peered":"2023-10-30T12:28:50.973288+0000","last_unstale":"2023-10-30T12:51:55.514716+0000","last_undegraded":"2023-10-30T12:51:55.514716+0000","last_fullsized":"2023-10-30T12:51:55.514716+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'569","last_scrub_stamp":"2023-10-30T12:51:55.514522+0000","last_deep_scrub":"66'569","last_deep_scrub_stamp":"2023-10-30T12:51:55.514522+0000","last_clean_scrub_stamp":"2023-10-30T12:51:55.514522+0000","objects_scrubbed":18,"log_size":569,"log_dups_size":0,"ondisk_log_size":569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:36.003579+0000","scrub_duration":0.043916595000000003,"objects_trimmed":0,"snaptrim_duration":0.0037299339999999999,"stat_sum":{"num_bytes":4302112,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":41,"num_read_kb":11,"num_write":510,"num_write_kb":7154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"66'500","reported_seq":2651,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:31.291065+0000","last_change":"2023-10-30T12:38:17.330519+0000","last_active":"2023-10-30T12:51:31.291065+0000","last_peered":"2023-10-30T12:51:31.291065+0000","last_clean":"2023-10-30T12:51:31.291065+0000","last_became_active":"2023-10-30T12:28:50.954988+0000","last_became_peered":"2023-10-30T12:28:50.954988+0000","last_unstale":"2023-10-30T12:51:31.291065+0000","last_undegraded":"2023-10-30T12:51:31.291065+0000","last_fullsized":"2023-10-30T12:51:31.291065+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":500,"log_dups_size":0,"ondisk_log_size":500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028962670000000001,"stat_sum":{"num_bytes":8424944,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":39,"num_read_kb":15,"num_write":456,"num_write_kb":10248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"66'595","reported_seq":2764,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:04.680217+0000","last_change":"2023-10-30T12:52:04.680454+0000","last_active":"2023-10-30T12:52:04.680217+0000","last_peered":"2023-10-30T12:52:04.680217+0000","last_clean":"2023-10-30T12:52:04.680217+0000","last_became_active":"2023-10-30T12:28:50.976564+0000","last_became_peered":"2023-10-30T12:28:50.976564+0000","last_unstale":"2023-10-30T12:52:04.680217+0000","last_undegraded":"2023-10-30T12:52:04.680217+0000","last_fullsized":"2023-10-30T12:52:04.680217+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'595","last_scrub_stamp":"2023-10-30T12:52:04.679935+0000","last_deep_scrub":"66'595","last_deep_scrub_stamp":"2023-10-30T12:52:04.679935+0000","last_clean_scrub_stamp":"2023-10-30T12:52:04.679935+0000","objects_scrubbed":22,"log_size":595,"log_dups_size":0,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:37:06.466494+0000","scrub_duration":0.091817582999999994,"objects_trimmed":0,"snaptrim_duration":0.0094697280000000002,"stat_sum":{"num_bytes":16840394,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":49,"num_read_kb":4116,"num_write":513,"num_write_kb":27605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"66'566","reported_seq":2747,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:57.616031+0000","last_change":"2023-10-30T12:51:57.616119+0000","last_active":"2023-10-30T12:51:57.616031+0000","last_peered":"2023-10-30T12:51:57.616031+0000","last_clean":"2023-10-30T12:51:57.616031+0000","last_became_active":"2023-10-30T12:28:50.951809+0000","last_became_peered":"2023-10-30T12:28:50.951809+0000","last_unstale":"2023-10-30T12:51:57.616031+0000","last_undegraded":"2023-10-30T12:51:57.616031+0000","last_fullsized":"2023-10-30T12:51:57.616031+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'566","last_scrub_stamp":"2023-10-30T12:51:57.615865+0000","last_deep_scrub":"66'566","last_deep_scrub_stamp":"2023-10-30T12:51:57.615865+0000","last_clean_scrub_stamp":"2023-10-30T12:51:57.615865+0000","objects_scrubbed":27,"log_size":566,"log_dups_size":0,"ondisk_log_size":566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:45:34.480445+0000","scrub_duration":0.097301143000000007,"objects_trimmed":0,"snaptrim_duration":0.0078740100000000007,"stat_sum":{"num_bytes":16924216,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":59,"num_read_kb":29,"num_write":493,"num_write_kb":22764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"66'600","reported_seq":2773,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:10.791677+0000","last_change":"2023-10-30T12:52:10.791921+0000","last_active":"2023-10-30T12:52:10.791677+0000","last_peered":"2023-10-30T12:52:10.791677+0000","last_clean":"2023-10-30T12:52:10.791677+0000","last_became_active":"2023-10-30T12:29:32.958473+0000","last_became_peered":"2023-10-30T12:29:32.958473+0000","last_unstale":"2023-10-30T12:52:10.791677+0000","last_undegraded":"2023-10-30T12:52:10.791677+0000","last_fullsized":"2023-10-30T12:52:10.791677+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'600","last_scrub_stamp":"2023-10-30T12:52:10.791444+0000","last_deep_scrub":"66'600","last_deep_scrub_stamp":"2023-10-30T12:52:10.791444+0000","last_clean_scrub_stamp":"2023-10-30T12:52:10.791444+0000","objects_scrubbed":23,"log_size":600,"log_dups_size":0,"ondisk_log_size":600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:50:18.491558+0000","scrub_duration":0.058907956999999997,"objects_trimmed":0,"snaptrim_duration":0.0045626240000000004,"stat_sum":{"num_bytes":12721502,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":46,"num_read_kb":17,"num_write":533,"num_write_kb":14913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"66'526","reported_seq":2680,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:43.156958+0000","last_change":"2023-10-30T12:51:43.157088+0000","last_active":"2023-10-30T12:51:43.156958+0000","last_peered":"2023-10-30T12:51:43.156958+0000","last_clean":"2023-10-30T12:51:43.156958+0000","last_became_active":"2023-10-30T12:28:50.943787+0000","last_became_peered":"2023-10-30T12:28:50.943787+0000","last_unstale":"2023-10-30T12:51:43.156958+0000","last_undegraded":"2023-10-30T12:51:43.156958+0000","last_fullsized":"2023-10-30T12:51:43.156958+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'526","last_scrub_stamp":"2023-10-30T12:51:43.156744+0000","last_deep_scrub":"66'526","last_deep_scrub_stamp":"2023-10-30T12:51:43.156744+0000","last_clean_scrub_stamp":"2023-10-30T12:51:43.156744+0000","objects_scrubbed":20,"log_size":526,"log_dups_size":0,"ondisk_log_size":526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:59:42.865066+0000","scrub_duration":0.057927502999999998,"objects_trimmed":0,"snaptrim_duration":0.0047342360000000002,"stat_sum":{"num_bytes":12634629,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":34,"num_read_kb":4115,"num_write":467,"num_write_kb":18795,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"66'611","reported_seq":2781,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:07.692371+0000","last_change":"2023-10-30T12:52:07.692447+0000","last_active":"2023-10-30T12:52:07.692371+0000","last_peered":"2023-10-30T12:52:07.692371+0000","last_clean":"2023-10-30T12:52:07.692371+0000","last_became_active":"2023-10-30T12:28:50.937906+0000","last_became_peered":"2023-10-30T12:28:50.937906+0000","last_unstale":"2023-10-30T12:52:07.692371+0000","last_undegraded":"2023-10-30T12:52:07.692371+0000","last_fullsized":"2023-10-30T12:52:07.692371+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'611","last_scrub_stamp":"2023-10-30T12:52:07.692259+0000","last_deep_scrub":"66'611","last_deep_scrub_stamp":"2023-10-30T12:52:07.692259+0000","last_clean_scrub_stamp":"2023-10-30T12:52:07.692259+0000","objects_scrubbed":11,"log_size":611,"log_dups_size":0,"ondisk_log_size":611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:46:50.215640+0000","scrub_duration":0.029019208000000001,"objects_trimmed":0,"snaptrim_duration":0.003369164,"stat_sum":{"num_bytes":4268758,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":50,"num_read_kb":8215,"num_write":547,"num_write_kb":14586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"66'554","reported_seq":2723,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:46.367652+0000","last_change":"2023-10-30T12:51:46.367652+0000","last_active":"2023-10-30T12:51:46.367652+0000","last_peered":"2023-10-30T12:51:46.367652+0000","last_clean":"2023-10-30T12:51:46.367652+0000","last_became_active":"2023-10-30T12:28:50.932753+0000","last_became_peered":"2023-10-30T12:28:50.932753+0000","last_unstale":"2023-10-30T12:51:46.367652+0000","last_undegraded":"2023-10-30T12:51:46.367652+0000","last_fullsized":"2023-10-30T12:51:46.367652+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'554","last_scrub_stamp":"2023-10-30T12:51:46.367517+0000","last_deep_scrub":"66'554","last_deep_scrub_stamp":"2023-10-30T12:51:46.367517+0000","last_clean_scrub_stamp":"2023-10-30T12:51:46.367517+0000","objects_scrubbed":25,"log_size":554,"log_dups_size":0,"ondisk_log_size":554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:15:04.516611+0000","scrub_duration":0.047742825000000003,"objects_trimmed":0,"snaptrim_duration":0.0027063,"stat_sum":{"num_bytes":16858869,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":47,"num_read_kb":17,"num_write":493,"num_write_kb":19132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"66'580","reported_seq":2751,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:49.898263+0000","last_change":"2023-10-30T12:51:49.898263+0000","last_active":"2023-10-30T12:51:49.898263+0000","last_peered":"2023-10-30T12:51:49.898263+0000","last_clean":"2023-10-30T12:51:49.898263+0000","last_became_active":"2023-10-30T12:28:50.943011+0000","last_became_peered":"2023-10-30T12:28:50.943011+0000","last_unstale":"2023-10-30T12:51:49.898263+0000","last_undegraded":"2023-10-30T12:51:49.898263+0000","last_fullsized":"2023-10-30T12:51:49.898263+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'580","last_scrub_stamp":"2023-10-30T12:51:49.898122+0000","last_deep_scrub":"66'580","last_deep_scrub_stamp":"2023-10-30T12:51:49.898122+0000","last_clean_scrub_stamp":"2023-10-30T12:51:49.898122+0000","objects_scrubbed":21,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:56:15.625962+0000","scrub_duration":0.069989635999999994,"objects_trimmed":0,"snaptrim_duration":0.001813863,"stat_sum":{"num_bytes":21155803,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":51,"num_read_kb":4115,"num_write":508,"num_write_kb":22928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"66'583","reported_seq":2750,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:16.300403+0000","last_change":"2023-10-30T12:52:16.300527+0000","last_active":"2023-10-30T12:52:16.300403+0000","last_peered":"2023-10-30T12:52:16.300403+0000","last_clean":"2023-10-30T12:52:16.300403+0000","last_became_active":"2023-10-30T12:28:50.952744+0000","last_became_peered":"2023-10-30T12:28:50.952744+0000","last_unstale":"2023-10-30T12:52:16.300403+0000","last_undegraded":"2023-10-30T12:52:16.300403+0000","last_fullsized":"2023-10-30T12:52:16.300403+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'583","last_scrub_stamp":"2023-10-30T12:52:16.300199+0000","last_deep_scrub":"66'583","last_deep_scrub_stamp":"2023-10-30T12:52:16.300199+0000","last_clean_scrub_stamp":"2023-10-30T12:52:16.300199+0000","objects_scrubbed":17,"log_size":583,"log_dups_size":0,"ondisk_log_size":583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:11:54.600332+0000","scrub_duration":0.041195979000000001,"objects_trimmed":0,"snaptrim_duration":0.0034371530000000001,"stat_sum":{"num_bytes":4262714,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":46,"num_read_kb":18,"num_write":521,"num_write_kb":9441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"66'590","reported_seq":2743,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:36.468011+0000","last_change":"2023-10-30T12:38:17.120572+0000","last_active":"2023-10-30T12:51:36.468011+0000","last_peered":"2023-10-30T12:51:36.468011+0000","last_clean":"2023-10-30T12:51:36.468011+0000","last_became_active":"2023-10-30T12:28:50.974576+0000","last_became_peered":"2023-10-30T12:28:50.974576+0000","last_unstale":"2023-10-30T12:51:36.468011+0000","last_undegraded":"2023-10-30T12:51:36.468011+0000","last_fullsized":"2023-10-30T12:51:36.468011+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":590,"log_dups_size":0,"ondisk_log_size":590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080003049999999992,"stat_sum":{"num_bytes":8462359,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":40,"num_read_kb":4113,"num_write":531,"num_write_kb":16554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"66'543","reported_seq":2722,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:46.554500+0000","last_change":"2023-10-30T12:51:46.554500+0000","last_active":"2023-10-30T12:51:46.554500+0000","last_peered":"2023-10-30T12:51:46.554500+0000","last_clean":"2023-10-30T12:51:46.554500+0000","last_became_active":"2023-10-30T12:28:50.937384+0000","last_became_peered":"2023-10-30T12:28:50.937384+0000","last_unstale":"2023-10-30T12:51:46.554500+0000","last_undegraded":"2023-10-30T12:51:46.554500+0000","last_fullsized":"2023-10-30T12:51:46.554500+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'543","last_scrub_stamp":"2023-10-30T12:51:46.554353+0000","last_deep_scrub":"66'543","last_deep_scrub_stamp":"2023-10-30T12:51:46.554353+0000","last_clean_scrub_stamp":"2023-10-30T12:51:46.554353+0000","objects_scrubbed":16,"log_size":543,"log_dups_size":0,"ondisk_log_size":543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:23:08.192334+0000","scrub_duration":0.042828463999999997,"objects_trimmed":0,"snaptrim_duration":0.001162803,"stat_sum":{"num_bytes":8456287,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":58,"num_read_kb":27,"num_write":488,"num_write_kb":9441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"66'581","reported_seq":2758,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:14.634497+0000","last_change":"2023-10-30T12:52:14.634614+0000","last_active":"2023-10-30T12:52:14.634497+0000","last_peered":"2023-10-30T12:52:14.634497+0000","last_clean":"2023-10-30T12:52:14.634497+0000","last_became_active":"2023-10-30T12:28:50.973369+0000","last_became_peered":"2023-10-30T12:28:50.973369+0000","last_unstale":"2023-10-30T12:52:14.634497+0000","last_undegraded":"2023-10-30T12:52:14.634497+0000","last_fullsized":"2023-10-30T12:52:14.634497+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'581","last_scrub_stamp":"2023-10-30T12:52:14.634290+0000","last_deep_scrub":"66'581","last_deep_scrub_stamp":"2023-10-30T12:52:14.634290+0000","last_clean_scrub_stamp":"2023-10-30T12:52:14.634290+0000","objects_scrubbed":27,"log_size":581,"log_dups_size":0,"ondisk_log_size":581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:50:37.072999+0000","scrub_duration":0.087054615000000002,"objects_trimmed":0,"snaptrim_duration":0.0093923919999999994,"stat_sum":{"num_bytes":12689191,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":55,"num_read_kb":4114,"num_write":505,"num_write_kb":18071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"66'592","reported_seq":2762,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:47.730012+0000","last_change":"2023-10-30T12:51:47.730137+0000","last_active":"2023-10-30T12:51:47.730012+0000","last_peered":"2023-10-30T12:51:47.730012+0000","last_clean":"2023-10-30T12:51:47.730012+0000","last_became_active":"2023-10-30T12:28:50.954831+0000","last_became_peered":"2023-10-30T12:28:50.954831+0000","last_unstale":"2023-10-30T12:51:47.730012+0000","last_undegraded":"2023-10-30T12:51:47.730012+0000","last_fullsized":"2023-10-30T12:51:47.730012+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'592","last_scrub_stamp":"2023-10-30T12:51:47.729828+0000","last_deep_scrub":"66'592","last_deep_scrub_stamp":"2023-10-30T12:51:47.729828+0000","last_clean_scrub_stamp":"2023-10-30T12:51:47.729828+0000","objects_scrubbed":20,"log_size":592,"log_dups_size":0,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:14:47.264586+0000","scrub_duration":0.089463355999999994,"objects_trimmed":0,"snaptrim_duration":0.007119116,"stat_sum":{"num_bytes":21055454,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":49,"num_read_kb":4113,"num_write":524,"num_write_kb":30367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"66'549","reported_seq":2711,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:10.434180+0000","last_change":"2023-10-30T12:52:10.434275+0000","last_active":"2023-10-30T12:52:10.434180+0000","last_peered":"2023-10-30T12:52:10.434180+0000","last_clean":"2023-10-30T12:52:10.434180+0000","last_became_active":"2023-10-30T12:28:50.962065+0000","last_became_peered":"2023-10-30T12:28:50.962065+0000","last_unstale":"2023-10-30T12:52:10.434180+0000","last_undegraded":"2023-10-30T12:52:10.434180+0000","last_fullsized":"2023-10-30T12:52:10.434180+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'549","last_scrub_stamp":"2023-10-30T12:52:10.434060+0000","last_deep_scrub":"66'549","last_deep_scrub_stamp":"2023-10-30T12:52:10.434060+0000","last_clean_scrub_stamp":"2023-10-30T12:52:10.434060+0000","objects_scrubbed":17,"log_size":549,"log_dups_size":0,"ondisk_log_size":549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:18:27.491696+0000","scrub_duration":0.079332753000000006,"objects_trimmed":0,"snaptrim_duration":0.0037739459999999998,"stat_sum":{"num_bytes":25200014,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":41,"num_read_kb":21,"num_write":490,"num_write_kb":26352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"66'608","reported_seq":2767,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.506121+0000","last_change":"2023-10-30T12:38:17.118400+0000","last_active":"2023-10-30T12:51:30.506121+0000","last_peered":"2023-10-30T12:51:30.506121+0000","last_clean":"2023-10-30T12:51:30.506121+0000","last_became_active":"2023-10-30T12:28:50.967651+0000","last_became_peered":"2023-10-30T12:28:50.967651+0000","last_unstale":"2023-10-30T12:51:30.506121+0000","last_undegraded":"2023-10-30T12:51:30.506121+0000","last_fullsized":"2023-10-30T12:51:30.506121+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":608,"log_dups_size":0,"ondisk_log_size":608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081085610000000002,"stat_sum":{"num_bytes":8425872,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":46,"num_read_kb":21,"num_write":555,"num_write_kb":13596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"66'584","reported_seq":2755,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:10.755998+0000","last_change":"2023-10-30T12:52:10.756068+0000","last_active":"2023-10-30T12:52:10.755998+0000","last_peered":"2023-10-30T12:52:10.755998+0000","last_clean":"2023-10-30T12:52:10.755998+0000","last_became_active":"2023-10-30T12:28:50.962290+0000","last_became_peered":"2023-10-30T12:28:50.962290+0000","last_unstale":"2023-10-30T12:52:10.755998+0000","last_undegraded":"2023-10-30T12:52:10.755998+0000","last_fullsized":"2023-10-30T12:52:10.755998+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'584","last_scrub_stamp":"2023-10-30T12:52:10.755902+0000","last_deep_scrub":"66'584","last_deep_scrub_stamp":"2023-10-30T12:52:10.755902+0000","last_clean_scrub_stamp":"2023-10-30T12:52:10.755902+0000","objects_scrubbed":24,"log_size":584,"log_dups_size":0,"ondisk_log_size":584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:10:48.108121+0000","scrub_duration":0.037006553999999997,"objects_trimmed":0,"snaptrim_duration":0.0042026670000000002,"stat_sum":{"num_bytes":195030,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":50,"num_read_kb":4117,"num_write":509,"num_write_kb":6258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"66'591","reported_seq":2752,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:13.307896+0000","last_change":"2023-10-30T12:52:13.308161+0000","last_active":"2023-10-30T12:52:13.307896+0000","last_peered":"2023-10-30T12:52:13.307896+0000","last_clean":"2023-10-30T12:52:13.307896+0000","last_became_active":"2023-10-30T12:28:50.965713+0000","last_became_peered":"2023-10-30T12:28:50.965713+0000","last_unstale":"2023-10-30T12:52:13.307896+0000","last_undegraded":"2023-10-30T12:52:13.307896+0000","last_fullsized":"2023-10-30T12:52:13.307896+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'591","last_scrub_stamp":"2023-10-30T12:52:13.307638+0000","last_deep_scrub":"66'591","last_deep_scrub_stamp":"2023-10-30T12:52:13.307638+0000","last_clean_scrub_stamp":"2023-10-30T12:52:13.307638+0000","objects_scrubbed":16,"log_size":591,"log_dups_size":0,"ondisk_log_size":591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:12:08.906462+0000","scrub_duration":0.066437737999999996,"objects_trimmed":0,"snaptrim_duration":0.0056034209999999999,"stat_sum":{"num_bytes":12659519,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":40,"num_read_kb":16,"num_write":528,"num_write_kb":18837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"66'533","reported_seq":2677,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.455876+0000","last_change":"2023-10-30T12:38:17.330864+0000","last_active":"2023-10-30T12:51:30.455876+0000","last_peered":"2023-10-30T12:51:30.455876+0000","last_clean":"2023-10-30T12:51:30.455876+0000","last_became_active":"2023-10-30T12:28:50.951026+0000","last_became_peered":"2023-10-30T12:28:50.951026+0000","last_unstale":"2023-10-30T12:51:30.455876+0000","last_undegraded":"2023-10-30T12:51:30.455876+0000","last_fullsized":"2023-10-30T12:51:30.455876+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":533,"log_dups_size":0,"ondisk_log_size":533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031697840000000001,"stat_sum":{"num_bytes":12597918,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":31,"num_read_kb":12,"num_write":481,"num_write_kb":19629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"66'624","reported_seq":2798,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:52.603819+0000","last_change":"2023-10-30T12:51:52.604014+0000","last_active":"2023-10-30T12:51:52.603819+0000","last_peered":"2023-10-30T12:51:52.603819+0000","last_clean":"2023-10-30T12:51:52.603819+0000","last_became_active":"2023-10-30T12:28:50.962484+0000","last_became_peered":"2023-10-30T12:28:50.962484+0000","last_unstale":"2023-10-30T12:51:52.603819+0000","last_undegraded":"2023-10-30T12:51:52.603819+0000","last_fullsized":"2023-10-30T12:51:52.603819+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'624","last_scrub_stamp":"2023-10-30T12:51:52.603583+0000","last_deep_scrub":"66'624","last_deep_scrub_stamp":"2023-10-30T12:51:52.603583+0000","last_clean_scrub_stamp":"2023-10-30T12:51:52.603583+0000","objects_scrubbed":26,"log_size":624,"log_dups_size":0,"ondisk_log_size":624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:26:46.288012+0000","scrub_duration":0.079879599999999995,"objects_trimmed":0,"snaptrim_duration":0.0083773500000000004,"stat_sum":{"num_bytes":12675985,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":51,"num_read_kb":23,"num_write":557,"num_write_kb":15080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"66'532","reported_seq":2695,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:51.872361+0000","last_change":"2023-10-30T12:51:51.872430+0000","last_active":"2023-10-30T12:51:51.872361+0000","last_peered":"2023-10-30T12:51:51.872361+0000","last_clean":"2023-10-30T12:51:51.872361+0000","last_became_active":"2023-10-30T12:28:50.933093+0000","last_became_peered":"2023-10-30T12:28:50.933093+0000","last_unstale":"2023-10-30T12:51:51.872361+0000","last_undegraded":"2023-10-30T12:51:51.872361+0000","last_fullsized":"2023-10-30T12:51:51.872361+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'532","last_scrub_stamp":"2023-10-30T12:51:51.872277+0000","last_deep_scrub":"66'532","last_deep_scrub_stamp":"2023-10-30T12:51:51.872277+0000","last_clean_scrub_stamp":"2023-10-30T12:51:51.872277+0000","objects_scrubbed":22,"log_size":532,"log_dups_size":0,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:06:03.858620+0000","scrub_duration":0.057177436999999998,"objects_trimmed":0,"snaptrim_duration":0.0022251829999999999,"stat_sum":{"num_bytes":25274091,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":42,"num_read_kb":8207,"num_write":449,"num_write_kb":27548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"66'517","reported_seq":2664,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:04.763148+0000","last_change":"2023-10-30T12:52:04.763210+0000","last_active":"2023-10-30T12:52:04.763148+0000","last_peered":"2023-10-30T12:52:04.763148+0000","last_clean":"2023-10-30T12:52:04.763148+0000","last_became_active":"2023-10-30T12:28:50.952531+0000","last_became_peered":"2023-10-30T12:28:50.952531+0000","last_unstale":"2023-10-30T12:52:04.763148+0000","last_undegraded":"2023-10-30T12:52:04.763148+0000","last_fullsized":"2023-10-30T12:52:04.763148+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'517","last_scrub_stamp":"2023-10-30T12:52:04.763062+0000","last_deep_scrub":"66'517","last_deep_scrub_stamp":"2023-10-30T12:52:04.763062+0000","last_clean_scrub_stamp":"2023-10-30T12:52:04.763062+0000","objects_scrubbed":18,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:37:42.920027+0000","scrub_duration":0.034901690999999999,"objects_trimmed":0,"snaptrim_duration":0.0039301029999999999,"stat_sum":{"num_bytes":4373272,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":26,"num_read_kb":3284,"num_write":458,"num_write_kb":14265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"66'542","reported_seq":2709,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:59.310588+0000","last_change":"2023-10-30T12:51:59.310659+0000","last_active":"2023-10-30T12:51:59.310588+0000","last_peered":"2023-10-30T12:51:59.310588+0000","last_clean":"2023-10-30T12:51:59.310588+0000","last_became_active":"2023-10-30T12:28:50.933124+0000","last_became_peered":"2023-10-30T12:28:50.933124+0000","last_unstale":"2023-10-30T12:51:59.310588+0000","last_undegraded":"2023-10-30T12:51:59.310588+0000","last_fullsized":"2023-10-30T12:51:59.310588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'542","last_scrub_stamp":"2023-10-30T12:51:59.310503+0000","last_deep_scrub":"66'542","last_deep_scrub_stamp":"2023-10-30T12:51:59.310503+0000","last_clean_scrub_stamp":"2023-10-30T12:51:59.310503+0000","objects_scrubbed":24,"log_size":542,"log_dups_size":0,"ondisk_log_size":542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:48:29.721009+0000","scrub_duration":0.033010021000000001,"objects_trimmed":0,"snaptrim_duration":0.0034752709999999998,"stat_sum":{"num_bytes":8490168,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":45,"num_read_kb":14,"num_write":469,"num_write_kb":10488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"66'594","reported_seq":2759,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:36.467890+0000","last_change":"2023-10-30T12:38:17.118502+0000","last_active":"2023-10-30T12:51:36.467890+0000","last_peered":"2023-10-30T12:51:36.467890+0000","last_clean":"2023-10-30T12:51:36.467890+0000","last_became_active":"2023-10-30T12:28:50.965639+0000","last_became_peered":"2023-10-30T12:28:50.965639+0000","last_unstale":"2023-10-30T12:51:36.467890+0000","last_undegraded":"2023-10-30T12:51:36.467890+0000","last_fullsized":"2023-10-30T12:51:36.467890+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":594,"log_dups_size":0,"ondisk_log_size":594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076861280000000004,"stat_sum":{"num_bytes":8432593,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":51,"num_read_kb":20,"num_write":547,"num_write_kb":11509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"66'595","reported_seq":2767,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:45.496340+0000","last_change":"2023-10-30T12:51:45.496340+0000","last_active":"2023-10-30T12:51:45.496340+0000","last_peered":"2023-10-30T12:51:45.496340+0000","last_clean":"2023-10-30T12:51:45.496340+0000","last_became_active":"2023-10-30T12:28:50.955084+0000","last_became_peered":"2023-10-30T12:28:50.955084+0000","last_unstale":"2023-10-30T12:51:45.496340+0000","last_undegraded":"2023-10-30T12:51:45.496340+0000","last_fullsized":"2023-10-30T12:51:45.496340+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'595","last_scrub_stamp":"2023-10-30T12:51:45.496211+0000","last_deep_scrub":"66'595","last_deep_scrub_stamp":"2023-10-30T12:51:45.496211+0000","last_clean_scrub_stamp":"2023-10-30T12:51:45.496211+0000","objects_scrubbed":12,"log_size":595,"log_dups_size":0,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:13:37.745803+0000","scrub_duration":0.027110084,"objects_trimmed":0,"snaptrim_duration":0.0012043119999999999,"stat_sum":{"num_bytes":4257179,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":50,"num_read_kb":22,"num_write":532,"num_write_kb":9837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"66'569","reported_seq":2744,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:54.604746+0000","last_change":"2023-10-30T12:51:54.604869+0000","last_active":"2023-10-30T12:51:54.604746+0000","last_peered":"2023-10-30T12:51:54.604746+0000","last_clean":"2023-10-30T12:51:54.604746+0000","last_became_active":"2023-10-30T12:28:50.944775+0000","last_became_peered":"2023-10-30T12:28:50.944775+0000","last_unstale":"2023-10-30T12:51:54.604746+0000","last_undegraded":"2023-10-30T12:51:54.604746+0000","last_fullsized":"2023-10-30T12:51:54.604746+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'569","last_scrub_stamp":"2023-10-30T12:51:54.604552+0000","last_deep_scrub":"66'569","last_deep_scrub_stamp":"2023-10-30T12:51:54.604552+0000","last_clean_scrub_stamp":"2023-10-30T12:51:54.604552+0000","objects_scrubbed":20,"log_size":569,"log_dups_size":0,"ondisk_log_size":569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:33:53.639827+0000","scrub_duration":0.054468190999999999,"objects_trimmed":0,"snaptrim_duration":0.0070573429999999998,"stat_sum":{"num_bytes":12763889,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":53,"num_read_kb":21,"num_write":514,"num_write_kb":13736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"66'590","reported_seq":2750,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:44.129343+0000","last_change":"2023-10-30T12:51:44.129472+0000","last_active":"2023-10-30T12:51:44.129343+0000","last_peered":"2023-10-30T12:51:44.129343+0000","last_clean":"2023-10-30T12:51:44.129343+0000","last_became_active":"2023-10-30T12:28:50.940567+0000","last_became_peered":"2023-10-30T12:28:50.940567+0000","last_unstale":"2023-10-30T12:51:44.129343+0000","last_undegraded":"2023-10-30T12:51:44.129343+0000","last_fullsized":"2023-10-30T12:51:44.129343+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'590","last_scrub_stamp":"2023-10-30T12:51:44.129123+0000","last_deep_scrub":"66'590","last_deep_scrub_stamp":"2023-10-30T12:51:44.129123+0000","last_clean_scrub_stamp":"2023-10-30T12:51:44.129123+0000","objects_scrubbed":10,"log_size":590,"log_dups_size":0,"ondisk_log_size":590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:25:14.156000+0000","scrub_duration":0.048555811999999997,"objects_trimmed":0,"snaptrim_duration":0.0028523340000000002,"stat_sum":{"num_bytes":9696728,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":17,"num_write":529,"num_write_kb":12781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"66'283","reported_seq":371,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:45.780003+0000","last_change":"2023-10-30T12:51:45.780059+0000","last_active":"2023-10-30T12:51:45.780003+0000","last_peered":"2023-10-30T12:51:45.780003+0000","last_clean":"2023-10-30T12:51:45.780003+0000","last_became_active":"2023-10-30T12:28:50.395382+0000","last_became_peered":"2023-10-30T12:28:50.395382+0000","last_unstale":"2023-10-30T12:51:45.780003+0000","last_undegraded":"2023-10-30T12:51:45.780003+0000","last_fullsized":"2023-10-30T12:51:45.780003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'283","last_scrub_stamp":"2023-10-30T12:51:45.779902+0000","last_deep_scrub":"66'283","last_deep_scrub_stamp":"2023-10-30T12:51:45.779902+0000","last_clean_scrub_stamp":"2023-10-30T12:51:45.779902+0000","objects_scrubbed":5,"log_size":283,"log_dups_size":0,"ondisk_log_size":283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:36:15.846905+0000","scrub_duration":0.042332846,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582946,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":45059,"num_write":213,"num_write_kb":12413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8500,"num_omap_keys":17,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"66'511","reported_seq":2676,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:45.384937+0000","last_change":"2023-10-30T12:51:45.385000+0000","last_active":"2023-10-30T12:51:45.384937+0000","last_peered":"2023-10-30T12:51:45.384937+0000","last_clean":"2023-10-30T12:51:45.384937+0000","last_became_active":"2023-10-30T12:28:50.952613+0000","last_became_peered":"2023-10-30T12:28:50.952613+0000","last_unstale":"2023-10-30T12:51:45.384937+0000","last_undegraded":"2023-10-30T12:51:45.384937+0000","last_fullsized":"2023-10-30T12:51:45.384937+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'511","last_scrub_stamp":"2023-10-30T12:51:45.384858+0000","last_deep_scrub":"66'511","last_deep_scrub_stamp":"2023-10-30T12:51:45.384858+0000","last_clean_scrub_stamp":"2023-10-30T12:51:45.384858+0000","objects_scrubbed":19,"log_size":511,"log_dups_size":0,"ondisk_log_size":511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:37:21.237208+0000","scrub_duration":0.069355456999999995,"objects_trimmed":0,"snaptrim_duration":0.0038505420000000002,"stat_sum":{"num_bytes":21061321,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":45,"num_read_kb":18,"num_write":449,"num_write_kb":22047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"66'909","reported_seq":1201,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:49.177081+0000","last_change":"2023-10-30T12:51:42.662808+0000","last_active":"2023-10-30T12:51:49.177081+0000","last_peered":"2023-10-30T12:51:49.177081+0000","last_clean":"2023-10-30T12:51:49.177081+0000","last_became_active":"2023-10-30T12:28:50.383898+0000","last_became_peered":"2023-10-30T12:28:50.383898+0000","last_unstale":"2023-10-30T12:51:49.177081+0000","last_undegraded":"2023-10-30T12:51:49.177081+0000","last_fullsized":"2023-10-30T12:51:49.177081+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'908","last_scrub_stamp":"2023-10-30T12:51:42.662429+0000","last_deep_scrub":"66'908","last_deep_scrub_stamp":"2023-10-30T12:51:42.662429+0000","last_clean_scrub_stamp":"2023-10-30T12:51:42.662429+0000","objects_scrubbed":9,"log_size":909,"log_dups_size":0,"ondisk_log_size":909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:01:46.593761+0000","scrub_duration":0.098226147999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21567319,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":234,"num_read_kb":49886,"num_write":842,"num_write_kb":21597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":23432,"num_omap_keys":43,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"66'457","reported_seq":759,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:48.571287+0000","last_change":"2023-10-30T12:51:41.780215+0000","last_active":"2023-10-30T12:51:48.571287+0000","last_peered":"2023-10-30T12:51:48.571287+0000","last_clean":"2023-10-30T12:51:48.571287+0000","last_became_active":"2023-10-30T12:28:50.385963+0000","last_became_peered":"2023-10-30T12:28:50.385963+0000","last_unstale":"2023-10-30T12:51:48.571287+0000","last_undegraded":"2023-10-30T12:51:48.571287+0000","last_fullsized":"2023-10-30T12:51:48.571287+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'454","last_scrub_stamp":"2023-10-30T12:51:41.780079+0000","last_deep_scrub":"66'454","last_deep_scrub_stamp":"2023-10-30T12:51:41.780079+0000","last_clean_scrub_stamp":"2023-10-30T12:51:41.780079+0000","objects_scrubbed":4,"log_size":457,"log_dups_size":0,"ondisk_log_size":457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:29:54.800743+0000","scrub_duration":0.030692146999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8006399,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":240,"num_read_kb":9095,"num_write":389,"num_write_kb":8081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"66'639","reported_seq":2815,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:51.809783+0000","last_change":"2023-10-30T12:51:51.809888+0000","last_active":"2023-10-30T12:51:51.809783+0000","last_peered":"2023-10-30T12:51:51.809783+0000","last_clean":"2023-10-30T12:51:51.809783+0000","last_became_active":"2023-10-30T12:28:50.935764+0000","last_became_peered":"2023-10-30T12:28:50.935764+0000","last_unstale":"2023-10-30T12:51:51.809783+0000","last_undegraded":"2023-10-30T12:51:51.809783+0000","last_fullsized":"2023-10-30T12:51:51.809783+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'639","last_scrub_stamp":"2023-10-30T12:51:51.809674+0000","last_deep_scrub":"66'639","last_deep_scrub_stamp":"2023-10-30T12:51:51.809674+0000","last_clean_scrub_stamp":"2023-10-30T12:51:51.809674+0000","objects_scrubbed":21,"log_size":639,"log_dups_size":0,"ondisk_log_size":639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:53.793939+0000","scrub_duration":0.025406960999999999,"objects_trimmed":0,"snaptrim_duration":0.0031329320000000002,"stat_sum":{"num_bytes":4280685,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":56,"num_read_kb":4120,"num_write":569,"num_write_kb":13778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"66'571","reported_seq":655,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:43.489239+0000","last_change":"2023-10-30T12:51:43.489353+0000","last_active":"2023-10-30T12:51:43.489239+0000","last_peered":"2023-10-30T12:51:43.489239+0000","last_clean":"2023-10-30T12:51:43.489239+0000","last_became_active":"2023-10-30T12:28:50.391494+0000","last_became_peered":"2023-10-30T12:28:50.391494+0000","last_unstale":"2023-10-30T12:51:43.489239+0000","last_undegraded":"2023-10-30T12:51:43.489239+0000","last_fullsized":"2023-10-30T12:51:43.489239+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'571","last_scrub_stamp":"2023-10-30T12:51:43.489033+0000","last_deep_scrub":"66'571","last_deep_scrub_stamp":"2023-10-30T12:51:43.489033+0000","last_clean_scrub_stamp":"2023-10-30T12:51:43.489033+0000","objects_scrubbed":3,"log_size":571,"log_dups_size":0,"ondisk_log_size":571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:24:33.871160+0000","scrub_duration":0.058779308000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":36864,"num_write":519,"num_write_kb":12589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"66'619","reported_seq":2786,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:07.826653+0000","last_change":"2023-10-30T12:52:07.826695+0000","last_active":"2023-10-30T12:52:07.826653+0000","last_peered":"2023-10-30T12:52:07.826653+0000","last_clean":"2023-10-30T12:52:07.826653+0000","last_became_active":"2023-10-30T12:28:50.933062+0000","last_became_peered":"2023-10-30T12:28:50.933062+0000","last_unstale":"2023-10-30T12:52:07.826653+0000","last_undegraded":"2023-10-30T12:52:07.826653+0000","last_fullsized":"2023-10-30T12:52:07.826653+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'619","last_scrub_stamp":"2023-10-30T12:52:07.826574+0000","last_deep_scrub":"66'619","last_deep_scrub_stamp":"2023-10-30T12:52:07.826574+0000","last_clean_scrub_stamp":"2023-10-30T12:52:07.826574+0000","objects_scrubbed":27,"log_size":619,"log_dups_size":0,"ondisk_log_size":619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:21:58.906873+0000","scrub_duration":0.081541012999999996,"objects_trimmed":0,"snaptrim_duration":0.0021140489999999998,"stat_sum":{"num_bytes":33653289,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":46,"num_read_kb":4115,"num_write":549,"num_write_kb":39173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"66'132","reported_seq":214,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:46.616340+0000","last_change":"2023-10-30T12:51:46.616340+0000","last_active":"2023-10-30T12:51:46.616340+0000","last_peered":"2023-10-30T12:51:46.616340+0000","last_clean":"2023-10-30T12:51:46.616340+0000","last_became_active":"2023-10-30T12:28:50.393686+0000","last_became_peered":"2023-10-30T12:28:50.393686+0000","last_unstale":"2023-10-30T12:51:46.616340+0000","last_undegraded":"2023-10-30T12:51:46.616340+0000","last_fullsized":"2023-10-30T12:51:46.616340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'131","last_scrub_stamp":"2023-10-30T12:51:46.615853+0000","last_deep_scrub":"66'131","last_deep_scrub_stamp":"2023-10-30T12:51:46.615853+0000","last_clean_scrub_stamp":"2023-10-30T12:51:46.615853+0000","objects_scrubbed":3,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:42:31.808839+0000","scrub_duration":0.0095666550000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":75,"num_write_kb":93,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9282,"num_omap_keys":17,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"66'589","reported_seq":2760,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:15.339095+0000","last_change":"2023-10-30T12:52:15.339230+0000","last_active":"2023-10-30T12:52:15.339095+0000","last_peered":"2023-10-30T12:52:15.339095+0000","last_clean":"2023-10-30T12:52:15.339095+0000","last_became_active":"2023-10-30T12:28:50.963105+0000","last_became_peered":"2023-10-30T12:28:50.963105+0000","last_unstale":"2023-10-30T12:52:15.339095+0000","last_undegraded":"2023-10-30T12:52:15.339095+0000","last_fullsized":"2023-10-30T12:52:15.339095+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'589","last_scrub_stamp":"2023-10-30T12:52:15.338927+0000","last_deep_scrub":"66'589","last_deep_scrub_stamp":"2023-10-30T12:52:15.338927+0000","last_clean_scrub_stamp":"2023-10-30T12:52:15.338927+0000","objects_scrubbed":22,"log_size":589,"log_dups_size":0,"ondisk_log_size":589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:22:02.826284+0000","scrub_duration":0.088894443000000004,"objects_trimmed":0,"snaptrim_duration":0.0080066809999999999,"stat_sum":{"num_bytes":16920090,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":51,"num_read_kb":18,"num_write":515,"num_write_kb":18479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"66'284","reported_seq":366,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:59.398926+0000","last_change":"2023-10-30T12:51:59.399057+0000","last_active":"2023-10-30T12:51:59.398926+0000","last_peered":"2023-10-30T12:51:59.398926+0000","last_clean":"2023-10-30T12:51:59.398926+0000","last_became_active":"2023-10-30T12:28:50.397041+0000","last_became_peered":"2023-10-30T12:28:50.397041+0000","last_unstale":"2023-10-30T12:51:59.398926+0000","last_undegraded":"2023-10-30T12:51:59.398926+0000","last_fullsized":"2023-10-30T12:51:59.398926+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'284","last_scrub_stamp":"2023-10-30T12:51:59.398723+0000","last_deep_scrub":"66'284","last_deep_scrub_stamp":"2023-10-30T12:51:59.398723+0000","last_clean_scrub_stamp":"2023-10-30T12:51:59.398723+0000","objects_scrubbed":3,"log_size":284,"log_dups_size":0,"ondisk_log_size":284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:44:04.485785+0000","scrub_duration":0.038907621000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":16396,"num_write":240,"num_write_kb":8312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"66'570","reported_seq":2732,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:11.399647+0000","last_change":"2023-10-30T12:52:11.399712+0000","last_active":"2023-10-30T12:52:11.399647+0000","last_peered":"2023-10-30T12:52:11.399647+0000","last_clean":"2023-10-30T12:52:11.399647+0000","last_became_active":"2023-10-30T12:28:50.966569+0000","last_became_peered":"2023-10-30T12:28:50.966569+0000","last_unstale":"2023-10-30T12:52:11.399647+0000","last_undegraded":"2023-10-30T12:52:11.399647+0000","last_fullsized":"2023-10-30T12:52:11.399647+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'570","last_scrub_stamp":"2023-10-30T12:52:11.399561+0000","last_deep_scrub":"66'570","last_deep_scrub_stamp":"2023-10-30T12:52:11.399561+0000","last_clean_scrub_stamp":"2023-10-30T12:52:11.399561+0000","objects_scrubbed":22,"log_size":570,"log_dups_size":0,"ondisk_log_size":570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:55.041154+0000","scrub_duration":0.068592573000000004,"objects_trimmed":0,"snaptrim_duration":0.0037972850000000001,"stat_sum":{"num_bytes":16988939,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":42,"num_read_kb":19,"num_write":508,"num_write_kb":18140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"66'375","reported_seq":463,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:45.949491+0000","last_change":"2023-10-30T12:51:45.949541+0000","last_active":"2023-10-30T12:51:45.949491+0000","last_peered":"2023-10-30T12:51:45.949491+0000","last_clean":"2023-10-30T12:51:45.949491+0000","last_became_active":"2023-10-30T12:28:50.391260+0000","last_became_peered":"2023-10-30T12:28:50.391260+0000","last_unstale":"2023-10-30T12:51:45.949491+0000","last_undegraded":"2023-10-30T12:51:45.949491+0000","last_fullsized":"2023-10-30T12:51:45.949491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'375","last_scrub_stamp":"2023-10-30T12:51:45.949405+0000","last_deep_scrub":"66'375","last_deep_scrub_stamp":"2023-10-30T12:51:45.949405+0000","last_clean_scrub_stamp":"2023-10-30T12:51:45.949405+0000","objects_scrubbed":6,"log_size":375,"log_dups_size":0,"ondisk_log_size":375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:31:43.191871+0000","scrub_duration":0.024520874000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":22,"num_read_kb":7467,"num_write":319,"num_write_kb":4401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7112,"num_omap_keys":14,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"66'524","reported_seq":2692,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:42.140557+0000","last_change":"2023-10-30T12:51:42.140557+0000","last_active":"2023-10-30T12:51:42.140557+0000","last_peered":"2023-10-30T12:51:42.140557+0000","last_clean":"2023-10-30T12:51:42.140557+0000","last_became_active":"2023-10-30T12:28:50.974843+0000","last_became_peered":"2023-10-30T12:28:50.974843+0000","last_unstale":"2023-10-30T12:51:42.140557+0000","last_undegraded":"2023-10-30T12:51:42.140557+0000","last_fullsized":"2023-10-30T12:51:42.140557+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'524","last_scrub_stamp":"2023-10-30T12:51:42.140220+0000","last_deep_scrub":"66'524","last_deep_scrub_stamp":"2023-10-30T12:51:42.140220+0000","last_clean_scrub_stamp":"2023-10-30T12:51:42.140220+0000","objects_scrubbed":12,"log_size":524,"log_dups_size":0,"ondisk_log_size":524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:15:48.848274+0000","scrub_duration":0.033227912999999998,"objects_trimmed":0,"snaptrim_duration":0.005161372,"stat_sum":{"num_bytes":4244440,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":47,"num_read_kb":17,"num_write":458,"num_write_kb":9849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"66'359","reported_seq":459,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:08.560979+0000","last_change":"2023-10-30T12:51:40.635013+0000","last_active":"2023-10-30T12:52:08.560979+0000","last_peered":"2023-10-30T12:52:08.560979+0000","last_clean":"2023-10-30T12:52:08.560979+0000","last_became_active":"2023-10-30T12:28:50.373581+0000","last_became_peered":"2023-10-30T12:28:50.373581+0000","last_unstale":"2023-10-30T12:52:08.560979+0000","last_undegraded":"2023-10-30T12:52:08.560979+0000","last_fullsized":"2023-10-30T12:52:08.560979+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'357","last_scrub_stamp":"2023-10-30T12:51:40.634678+0000","last_deep_scrub":"66'357","last_deep_scrub_stamp":"2023-10-30T12:51:40.634678+0000","last_clean_scrub_stamp":"2023-10-30T12:51:40.634678+0000","objects_scrubbed":9,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:38:14.782628+0000","scrub_duration":0.051559438999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":43,"num_read_kb":32803,"num_write":294,"num_write_kb":12545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7620,"num_omap_keys":15,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"66'519","reported_seq":2677,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:08.630116+0000","last_change":"2023-10-30T12:52:08.630261+0000","last_active":"2023-10-30T12:52:08.630116+0000","last_peered":"2023-10-30T12:52:08.630116+0000","last_clean":"2023-10-30T12:52:08.630116+0000","last_became_active":"2023-10-30T12:28:50.973269+0000","last_became_peered":"2023-10-30T12:28:50.973269+0000","last_unstale":"2023-10-30T12:52:08.630116+0000","last_undegraded":"2023-10-30T12:52:08.630116+0000","last_fullsized":"2023-10-30T12:52:08.630116+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'519","last_scrub_stamp":"2023-10-30T12:52:08.629908+0000","last_deep_scrub":"66'519","last_deep_scrub_stamp":"2023-10-30T12:52:08.629908+0000","last_clean_scrub_stamp":"2023-10-30T12:52:08.629908+0000","objects_scrubbed":20,"log_size":519,"log_dups_size":0,"ondisk_log_size":519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:08.922310+0000","scrub_duration":0.063779350999999998,"objects_trimmed":0,"snaptrim_duration":0.004001498,"stat_sum":{"num_bytes":12707497,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":38,"num_read_kb":11,"num_write":446,"num_write_kb":13522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"66'444","reported_seq":545,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:37.461917+0000","last_change":"2023-10-30T12:51:37.461917+0000","last_active":"2023-10-30T12:51:37.461917+0000","last_peered":"2023-10-30T12:51:37.461917+0000","last_clean":"2023-10-30T12:51:37.461917+0000","last_became_active":"2023-10-30T12:28:50.366269+0000","last_became_peered":"2023-10-30T12:28:50.366269+0000","last_unstale":"2023-10-30T12:51:37.461917+0000","last_undegraded":"2023-10-30T12:51:37.461917+0000","last_fullsized":"2023-10-30T12:51:37.461917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'444","last_scrub_stamp":"2023-10-30T12:51:37.461743+0000","last_deep_scrub":"66'444","last_deep_scrub_stamp":"2023-10-30T12:51:37.461743+0000","last_clean_scrub_stamp":"2023-10-30T12:51:37.461743+0000","objects_scrubbed":9,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:35:49.998956+0000","scrub_duration":0.040110959000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":52,"num_read_kb":36923,"num_write":386,"num_write_kb":8499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":16168,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"66'586","reported_seq":2759,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:43.354752+0000","last_change":"2023-10-30T12:51:43.354752+0000","last_active":"2023-10-30T12:51:43.354752+0000","last_peered":"2023-10-30T12:51:43.354752+0000","last_clean":"2023-10-30T12:51:43.354752+0000","last_became_active":"2023-10-30T12:28:50.961805+0000","last_became_peered":"2023-10-30T12:28:50.961805+0000","last_unstale":"2023-10-30T12:51:43.354752+0000","last_undegraded":"2023-10-30T12:51:43.354752+0000","last_fullsized":"2023-10-30T12:51:43.354752+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'586","last_scrub_stamp":"2023-10-30T12:51:43.354576+0000","last_deep_scrub":"66'586","last_deep_scrub_stamp":"2023-10-30T12:51:43.354576+0000","last_clean_scrub_stamp":"2023-10-30T12:51:43.354576+0000","objects_scrubbed":22,"log_size":586,"log_dups_size":0,"ondisk_log_size":586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:44:38.376225+0000","scrub_duration":0.037946186999999999,"objects_trimmed":0,"snaptrim_duration":0.0032264350000000002,"stat_sum":{"num_bytes":4388445,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":52,"num_read_kb":4114,"num_write":506,"num_write_kb":14199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"66'261","reported_seq":346,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:08.296017+0000","last_change":"2023-10-30T12:52:08.296017+0000","last_active":"2023-10-30T12:52:08.296017+0000","last_peered":"2023-10-30T12:52:08.296017+0000","last_clean":"2023-10-30T12:52:08.296017+0000","last_became_active":"2023-10-30T12:28:50.396727+0000","last_became_peered":"2023-10-30T12:28:50.396727+0000","last_unstale":"2023-10-30T12:52:08.296017+0000","last_undegraded":"2023-10-30T12:52:08.296017+0000","last_fullsized":"2023-10-30T12:52:08.296017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'261","last_scrub_stamp":"2023-10-30T12:52:08.295709+0000","last_deep_scrub":"66'261","last_deep_scrub_stamp":"2023-10-30T12:52:08.295709+0000","last_clean_scrub_stamp":"2023-10-30T12:52:08.295709+0000","objects_scrubbed":3,"log_size":261,"log_dups_size":0,"ondisk_log_size":261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:43:47.836137+0000","scrub_duration":0.032577411000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":16164,"num_write":201,"num_write_kb":8311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"66'568","reported_seq":2743,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:49.633783+0000","last_change":"2023-10-30T12:51:49.633887+0000","last_active":"2023-10-30T12:51:49.633783+0000","last_peered":"2023-10-30T12:51:49.633783+0000","last_clean":"2023-10-30T12:51:49.633783+0000","last_became_active":"2023-10-30T12:29:32.960608+0000","last_became_peered":"2023-10-30T12:29:32.960608+0000","last_unstale":"2023-10-30T12:51:49.633783+0000","last_undegraded":"2023-10-30T12:51:49.633783+0000","last_fullsized":"2023-10-30T12:51:49.633783+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'568","last_scrub_stamp":"2023-10-30T12:51:49.633664+0000","last_deep_scrub":"66'568","last_deep_scrub_stamp":"2023-10-30T12:51:49.633664+0000","last_clean_scrub_stamp":"2023-10-30T12:51:49.633664+0000","objects_scrubbed":14,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:00:42.755675+0000","scrub_duration":0.028072751999999999,"objects_trimmed":0,"snaptrim_duration":0.0012934999999999999,"stat_sum":{"num_bytes":4430082,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":59,"num_read_kb":33,"num_write":506,"num_write_kb":6384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"66'684","reported_seq":775,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:44.639628+0000","last_change":"2023-10-30T12:51:44.639628+0000","last_active":"2023-10-30T12:51:44.639628+0000","last_peered":"2023-10-30T12:51:44.639628+0000","last_clean":"2023-10-30T12:51:44.639628+0000","last_became_active":"2023-10-30T12:28:50.383570+0000","last_became_peered":"2023-10-30T12:28:50.383570+0000","last_unstale":"2023-10-30T12:51:44.639628+0000","last_undegraded":"2023-10-30T12:51:44.639628+0000","last_fullsized":"2023-10-30T12:51:44.639628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'684","last_scrub_stamp":"2023-10-30T12:51:44.639059+0000","last_deep_scrub":"66'684","last_deep_scrub_stamp":"2023-10-30T12:51:44.639059+0000","last_clean_scrub_stamp":"2023-10-30T12:51:44.639059+0000","objects_scrubbed":7,"log_size":684,"log_dups_size":0,"ondisk_log_size":684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:23:55.453101+0000","scrub_duration":0.079897833000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":37306,"num_write":619,"num_write_kb":16724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15312,"num_omap_keys":30,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"66'556","reported_seq":2710,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:01.705427+0000","last_change":"2023-10-30T12:52:01.705427+0000","last_active":"2023-10-30T12:52:01.705427+0000","last_peered":"2023-10-30T12:52:01.705427+0000","last_clean":"2023-10-30T12:52:01.705427+0000","last_became_active":"2023-10-30T12:28:50.949485+0000","last_became_peered":"2023-10-30T12:28:50.949485+0000","last_unstale":"2023-10-30T12:52:01.705427+0000","last_undegraded":"2023-10-30T12:52:01.705427+0000","last_fullsized":"2023-10-30T12:52:01.705427+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'556","last_scrub_stamp":"2023-10-30T12:52:01.705046+0000","last_deep_scrub":"66'556","last_deep_scrub_stamp":"2023-10-30T12:52:01.705046+0000","last_clean_scrub_stamp":"2023-10-30T12:52:01.705046+0000","objects_scrubbed":15,"log_size":556,"log_dups_size":0,"ondisk_log_size":556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:05:15.771642+0000","scrub_duration":0.064202585000000006,"objects_trimmed":0,"snaptrim_duration":0.0040897299999999998,"stat_sum":{"num_bytes":12622957,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":477,"num_write_kb":14339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"66'453","reported_seq":569,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:49.184348+0000","last_change":"2023-10-30T12:51:41.463979+0000","last_active":"2023-10-30T12:51:49.184348+0000","last_peered":"2023-10-30T12:51:49.184348+0000","last_clean":"2023-10-30T12:51:49.184348+0000","last_became_active":"2023-10-30T12:28:50.362624+0000","last_became_peered":"2023-10-30T12:28:50.362624+0000","last_unstale":"2023-10-30T12:51:49.184348+0000","last_undegraded":"2023-10-30T12:51:49.184348+0000","last_fullsized":"2023-10-30T12:51:49.184348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'452","last_scrub_stamp":"2023-10-30T12:51:41.463765+0000","last_deep_scrub":"66'452","last_deep_scrub_stamp":"2023-10-30T12:51:41.463765+0000","last_clean_scrub_stamp":"2023-10-30T12:51:41.463765+0000","objects_scrubbed":13,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:18.726132+0000","scrub_duration":0.043492996999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583024,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":93,"num_read_kb":24662,"num_write":378,"num_write_kb":12558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6552,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"66'600","reported_seq":2771,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:57.546471+0000","last_change":"2023-10-30T12:51:57.546471+0000","last_active":"2023-10-30T12:51:57.546471+0000","last_peered":"2023-10-30T12:51:57.546471+0000","last_clean":"2023-10-30T12:51:57.546471+0000","last_became_active":"2023-10-30T12:28:50.979344+0000","last_became_peered":"2023-10-30T12:28:50.979344+0000","last_unstale":"2023-10-30T12:51:57.546471+0000","last_undegraded":"2023-10-30T12:51:57.546471+0000","last_fullsized":"2023-10-30T12:51:57.546471+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'600","last_scrub_stamp":"2023-10-30T12:51:57.546007+0000","last_deep_scrub":"66'600","last_deep_scrub_stamp":"2023-10-30T12:51:57.546007+0000","last_clean_scrub_stamp":"2023-10-30T12:51:57.546007+0000","objects_scrubbed":24,"log_size":600,"log_dups_size":0,"ondisk_log_size":600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:44:55.842195+0000","scrub_duration":0.075819872999999996,"objects_trimmed":0,"snaptrim_duration":0.0092295529999999997,"stat_sum":{"num_bytes":12647197,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":50,"num_read_kb":8210,"num_write":532,"num_write_kb":23192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"66'248","reported_seq":334,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:37.268010+0000","last_change":"2023-10-30T12:51:37.268108+0000","last_active":"2023-10-30T12:51:37.268010+0000","last_peered":"2023-10-30T12:51:37.268010+0000","last_clean":"2023-10-30T12:51:37.268010+0000","last_became_active":"2023-10-30T12:28:50.381125+0000","last_became_peered":"2023-10-30T12:28:50.381125+0000","last_unstale":"2023-10-30T12:51:37.268010+0000","last_undegraded":"2023-10-30T12:51:37.268010+0000","last_fullsized":"2023-10-30T12:51:37.268010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'248","last_scrub_stamp":"2023-10-30T12:51:37.267826+0000","last_deep_scrub":"66'248","last_deep_scrub_stamp":"2023-10-30T12:51:37.267826+0000","last_clean_scrub_stamp":"2023-10-30T12:51:37.267826+0000","objects_scrubbed":3,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:30:50.232542+0000","scrub_duration":0.054105548000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":12378,"num_write":396,"num_write_kb":4652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"66'548","reported_seq":2705,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:09.640556+0000","last_change":"2023-10-30T12:52:09.640712+0000","last_active":"2023-10-30T12:52:09.640556+0000","last_peered":"2023-10-30T12:52:09.640556+0000","last_clean":"2023-10-30T12:52:09.640556+0000","last_became_active":"2023-10-30T12:28:50.977585+0000","last_became_peered":"2023-10-30T12:28:50.977585+0000","last_unstale":"2023-10-30T12:52:09.640556+0000","last_undegraded":"2023-10-30T12:52:09.640556+0000","last_fullsized":"2023-10-30T12:52:09.640556+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'548","last_scrub_stamp":"2023-10-30T12:52:09.640316+0000","last_deep_scrub":"66'548","last_deep_scrub_stamp":"2023-10-30T12:52:09.640316+0000","last_clean_scrub_stamp":"2023-10-30T12:52:09.640316+0000","objects_scrubbed":17,"log_size":548,"log_dups_size":0,"ondisk_log_size":548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:06:42.121080+0000","scrub_duration":0.072014062000000004,"objects_trimmed":0,"snaptrim_duration":0.0089211889999999995,"stat_sum":{"num_bytes":12933034,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":36,"num_read_kb":8205,"num_write":483,"num_write_kb":18997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"66'359","reported_seq":445,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:38.474826+0000","last_change":"2023-10-30T12:51:38.474877+0000","last_active":"2023-10-30T12:51:38.474826+0000","last_peered":"2023-10-30T12:51:38.474826+0000","last_clean":"2023-10-30T12:51:38.474826+0000","last_became_active":"2023-10-30T12:28:50.366974+0000","last_became_peered":"2023-10-30T12:28:50.366974+0000","last_unstale":"2023-10-30T12:51:38.474826+0000","last_undegraded":"2023-10-30T12:51:38.474826+0000","last_fullsized":"2023-10-30T12:51:38.474826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'359","last_scrub_stamp":"2023-10-30T12:51:38.474746+0000","last_deep_scrub":"66'359","last_deep_scrub_stamp":"2023-10-30T12:51:38.474746+0000","last_clean_scrub_stamp":"2023-10-30T12:51:38.474746+0000","objects_scrubbed":4,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:46:34.826564+0000","scrub_duration":0.021991255000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":15324,"num_write":292,"num_write_kb":8361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10374,"num_omap_keys":19,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"66'558","reported_seq":2729,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:11.609600+0000","last_change":"2023-10-30T12:52:11.609787+0000","last_active":"2023-10-30T12:52:11.609600+0000","last_peered":"2023-10-30T12:52:11.609600+0000","last_clean":"2023-10-30T12:52:11.609600+0000","last_became_active":"2023-10-30T12:28:50.981193+0000","last_became_peered":"2023-10-30T12:28:50.981193+0000","last_unstale":"2023-10-30T12:52:11.609600+0000","last_undegraded":"2023-10-30T12:52:11.609600+0000","last_fullsized":"2023-10-30T12:52:11.609600+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'558","last_scrub_stamp":"2023-10-30T12:52:11.609388+0000","last_deep_scrub":"66'558","last_deep_scrub_stamp":"2023-10-30T12:52:11.609388+0000","last_clean_scrub_stamp":"2023-10-30T12:52:11.609388+0000","objects_scrubbed":24,"log_size":558,"log_dups_size":0,"ondisk_log_size":558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:35:18.683385+0000","scrub_duration":0.079771501999999994,"objects_trimmed":0,"snaptrim_duration":0.0096579700000000001,"stat_sum":{"num_bytes":12674006,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":50,"num_read_kb":4114,"num_write":492,"num_write_kb":18051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"66'98","reported_seq":180,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:44.901113+0000","last_change":"2023-10-30T12:51:44.901168+0000","last_active":"2023-10-30T12:51:44.901113+0000","last_peered":"2023-10-30T12:51:44.901113+0000","last_clean":"2023-10-30T12:51:44.901113+0000","last_became_active":"2023-10-30T12:28:50.367730+0000","last_became_peered":"2023-10-30T12:28:50.367730+0000","last_unstale":"2023-10-30T12:51:44.901113+0000","last_undegraded":"2023-10-30T12:51:44.901113+0000","last_fullsized":"2023-10-30T12:51:44.901113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'98","last_scrub_stamp":"2023-10-30T12:51:44.900977+0000","last_deep_scrub":"66'98","last_deep_scrub_stamp":"2023-10-30T12:51:44.900977+0000","last_clean_scrub_stamp":"2023-10-30T12:51:44.900977+0000","objects_scrubbed":4,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:46:56.082172+0000","scrub_duration":0.018334822000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":12292,"num_write":45,"num_write_kb":4186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8128,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"66'594","reported_seq":2763,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.473832+0000","last_change":"2023-10-30T12:38:17.119150+0000","last_active":"2023-10-30T12:51:30.473832+0000","last_peered":"2023-10-30T12:51:30.473832+0000","last_clean":"2023-10-30T12:51:30.473832+0000","last_became_active":"2023-10-30T12:28:50.966246+0000","last_became_peered":"2023-10-30T12:28:50.966246+0000","last_unstale":"2023-10-30T12:51:30.473832+0000","last_undegraded":"2023-10-30T12:51:30.473832+0000","last_fullsized":"2023-10-30T12:51:30.473832+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":594,"log_dups_size":0,"ondisk_log_size":594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007703601,"stat_sum":{"num_bytes":8520713,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":56,"num_read_kb":18,"num_write":541,"num_write_kb":11343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"66'87","reported_seq":172,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:42.434625+0000","last_change":"2023-10-30T12:51:42.434673+0000","last_active":"2023-10-30T12:51:42.434625+0000","last_peered":"2023-10-30T12:51:42.434625+0000","last_clean":"2023-10-30T12:51:42.434625+0000","last_became_active":"2023-10-30T12:28:50.393642+0000","last_became_peered":"2023-10-30T12:28:50.393642+0000","last_unstale":"2023-10-30T12:51:42.434625+0000","last_undegraded":"2023-10-30T12:51:42.434625+0000","last_fullsized":"2023-10-30T12:51:42.434625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'87","last_scrub_stamp":"2023-10-30T12:51:42.434545+0000","last_deep_scrub":"66'87","last_deep_scrub_stamp":"2023-10-30T12:51:42.434545+0000","last_clean_scrub_stamp":"2023-10-30T12:51:42.434545+0000","objects_scrubbed":5,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:27:29.743857+0000","scrub_duration":0.0090945590000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":15,"num_write_kb":101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17524,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"66'579","reported_seq":2744,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:56.560776+0000","last_change":"2023-10-30T12:51:56.560943+0000","last_active":"2023-10-30T12:51:56.560776+0000","last_peered":"2023-10-30T12:51:56.560776+0000","last_clean":"2023-10-30T12:51:56.560776+0000","last_became_active":"2023-10-30T12:28:50.976343+0000","last_became_peered":"2023-10-30T12:28:50.976343+0000","last_unstale":"2023-10-30T12:51:56.560776+0000","last_undegraded":"2023-10-30T12:51:56.560776+0000","last_fullsized":"2023-10-30T12:51:56.560776+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'579","last_scrub_stamp":"2023-10-30T12:51:56.560556+0000","last_deep_scrub":"66'579","last_deep_scrub_stamp":"2023-10-30T12:51:56.560556+0000","last_clean_scrub_stamp":"2023-10-30T12:51:56.560556+0000","objects_scrubbed":23,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:00:58.734907+0000","scrub_duration":0.074070374999999994,"objects_trimmed":0,"snaptrim_duration":0.0091875959999999993,"stat_sum":{"num_bytes":17078090,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":44,"num_read_kb":14,"num_write":517,"num_write_kb":22829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"66'119","reported_seq":231,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:44.641495+0000","last_change":"2023-10-30T12:51:44.641713+0000","last_active":"2023-10-30T12:51:44.641495+0000","last_peered":"2023-10-30T12:51:44.641495+0000","last_clean":"2023-10-30T12:51:44.641495+0000","last_became_active":"2023-10-30T12:28:41.201796+0000","last_became_peered":"2023-10-30T12:28:41.201796+0000","last_unstale":"2023-10-30T12:51:44.641495+0000","last_undegraded":"2023-10-30T12:51:44.641495+0000","last_fullsized":"2023-10-30T12:51:44.641495+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'119","last_scrub_stamp":"2023-10-30T12:51:44.641256+0000","last_deep_scrub":"66'119","last_deep_scrub_stamp":"2023-10-30T12:51:44.641256+0000","last_clean_scrub_stamp":"2023-10-30T12:51:44.641256+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:26:13.839326+0000","scrub_duration":0.013391731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"66'201","reported_seq":291,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:50.447569+0000","last_change":"2023-10-30T12:51:50.447661+0000","last_active":"2023-10-30T12:51:50.447569+0000","last_peered":"2023-10-30T12:51:50.447569+0000","last_clean":"2023-10-30T12:51:50.447569+0000","last_became_active":"2023-10-30T12:28:50.367593+0000","last_became_peered":"2023-10-30T12:28:50.367593+0000","last_unstale":"2023-10-30T12:51:50.447569+0000","last_undegraded":"2023-10-30T12:51:50.447569+0000","last_fullsized":"2023-10-30T12:51:50.447569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'201","last_scrub_stamp":"2023-10-30T12:51:50.447458+0000","last_deep_scrub":"66'201","last_deep_scrub_stamp":"2023-10-30T12:51:50.447458+0000","last_clean_scrub_stamp":"2023-10-30T12:51:50.447458+0000","objects_scrubbed":9,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:35:16.204652+0000","scrub_duration":0.025712215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194338,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":24,"num_read_kb":12326,"num_write":144,"num_write_kb":4223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":49160,"num_omap_keys":96,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"66'535","reported_seq":2689,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:48.599000+0000","last_change":"2023-10-30T12:51:48.599069+0000","last_active":"2023-10-30T12:51:48.599000+0000","last_peered":"2023-10-30T12:51:48.599000+0000","last_clean":"2023-10-30T12:51:48.599000+0000","last_became_active":"2023-10-30T12:29:32.997301+0000","last_became_peered":"2023-10-30T12:29:32.997301+0000","last_unstale":"2023-10-30T12:51:48.599000+0000","last_undegraded":"2023-10-30T12:51:48.599000+0000","last_fullsized":"2023-10-30T12:51:48.599000+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'535","last_scrub_stamp":"2023-10-30T12:51:48.598899+0000","last_deep_scrub":"66'535","last_deep_scrub_stamp":"2023-10-30T12:51:48.598899+0000","last_clean_scrub_stamp":"2023-10-30T12:51:48.598899+0000","objects_scrubbed":17,"log_size":535,"log_dups_size":0,"ondisk_log_size":535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:04:11.091469+0000","scrub_duration":0.019854914000000001,"objects_trimmed":0,"snaptrim_duration":0.001366412,"stat_sum":{"num_bytes":4237965,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":36,"num_read_kb":11,"num_write":481,"num_write_kb":13486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"66'504","reported_seq":600,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:47.588384+0000","last_change":"2023-10-30T12:51:47.588509+0000","last_active":"2023-10-30T12:51:47.588384+0000","last_peered":"2023-10-30T12:51:47.588384+0000","last_clean":"2023-10-30T12:51:47.588384+0000","last_became_active":"2023-10-30T12:28:50.391824+0000","last_became_peered":"2023-10-30T12:28:50.391824+0000","last_unstale":"2023-10-30T12:51:47.588384+0000","last_undegraded":"2023-10-30T12:51:47.588384+0000","last_fullsized":"2023-10-30T12:51:47.588384+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'504","last_scrub_stamp":"2023-10-30T12:51:47.588159+0000","last_deep_scrub":"66'504","last_deep_scrub_stamp":"2023-10-30T12:51:47.588159+0000","last_clean_scrub_stamp":"2023-10-30T12:51:47.588159+0000","objects_scrubbed":7,"log_size":504,"log_dups_size":0,"ondisk_log_size":504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:53:44.520327+0000","scrub_duration":0.072404271000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":32,"num_read_kb":50322,"num_write":431,"num_write_kb":16616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13208,"num_omap_keys":26,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"66'547","reported_seq":2698,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:16.385607+0000","last_change":"2023-10-30T12:52:16.385672+0000","last_active":"2023-10-30T12:52:16.385607+0000","last_peered":"2023-10-30T12:52:16.385607+0000","last_clean":"2023-10-30T12:52:16.385607+0000","last_became_active":"2023-10-30T12:28:50.952625+0000","last_became_peered":"2023-10-30T12:28:50.952625+0000","last_unstale":"2023-10-30T12:52:16.385607+0000","last_undegraded":"2023-10-30T12:52:16.385607+0000","last_fullsized":"2023-10-30T12:52:16.385607+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'547","last_scrub_stamp":"2023-10-30T12:52:16.385522+0000","last_deep_scrub":"66'547","last_deep_scrub_stamp":"2023-10-30T12:52:16.385522+0000","last_clean_scrub_stamp":"2023-10-30T12:52:16.385522+0000","objects_scrubbed":15,"log_size":547,"log_dups_size":0,"ondisk_log_size":547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:48:56.290695+0000","scrub_duration":0.061983638000000001,"objects_trimmed":0,"snaptrim_duration":0.0026695970000000001,"stat_sum":{"num_bytes":16953646,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":29,"num_read_kb":10,"num_write":468,"num_write_kb":20237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"66'605","reported_seq":920,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:48.574934+0000","last_change":"2023-10-30T12:51:35.405246+0000","last_active":"2023-10-30T12:51:48.574934+0000","last_peered":"2023-10-30T12:51:48.574934+0000","last_clean":"2023-10-30T12:51:48.574934+0000","last_became_active":"2023-10-30T12:28:50.367230+0000","last_became_peered":"2023-10-30T12:28:50.367230+0000","last_unstale":"2023-10-30T12:51:48.574934+0000","last_undegraded":"2023-10-30T12:51:48.574934+0000","last_fullsized":"2023-10-30T12:51:48.574934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'598","last_scrub_stamp":"2023-10-30T12:51:35.405133+0000","last_deep_scrub":"66'598","last_deep_scrub_stamp":"2023-10-30T12:51:35.405133+0000","last_clean_scrub_stamp":"2023-10-30T12:51:35.405133+0000","objects_scrubbed":7,"log_size":605,"log_dups_size":0,"ondisk_log_size":605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:34:01.194417+0000","scrub_duration":0.035040017999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12778646,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":251,"num_read_kb":13023,"num_write":542,"num_write_kb":12830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9828,"num_omap_keys":18,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"66'565","reported_seq":2734,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:57.726014+0000","last_change":"2023-10-30T12:51:57.726097+0000","last_active":"2023-10-30T12:51:57.726014+0000","last_peered":"2023-10-30T12:51:57.726014+0000","last_clean":"2023-10-30T12:51:57.726014+0000","last_became_active":"2023-10-30T12:28:50.928608+0000","last_became_peered":"2023-10-30T12:28:50.928608+0000","last_unstale":"2023-10-30T12:51:57.726014+0000","last_undegraded":"2023-10-30T12:51:57.726014+0000","last_fullsized":"2023-10-30T12:51:57.726014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'565","last_scrub_stamp":"2023-10-30T12:51:57.725915+0000","last_deep_scrub":"66'565","last_deep_scrub_stamp":"2023-10-30T12:51:57.725915+0000","last_clean_scrub_stamp":"2023-10-30T12:51:57.725915+0000","objects_scrubbed":21,"log_size":565,"log_dups_size":0,"ondisk_log_size":565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:24:52.304705+0000","scrub_duration":0.032239474999999997,"objects_trimmed":0,"snaptrim_duration":0.002074275,"stat_sum":{"num_bytes":8441122,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":47,"num_read_kb":21,"num_write":501,"num_write_kb":11799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"66'497","reported_seq":586,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:02.360999+0000","last_change":"2023-10-30T12:52:02.361114+0000","last_active":"2023-10-30T12:52:02.360999+0000","last_peered":"2023-10-30T12:52:02.360999+0000","last_clean":"2023-10-30T12:52:02.360999+0000","last_became_active":"2023-10-30T12:28:50.390864+0000","last_became_peered":"2023-10-30T12:28:50.390864+0000","last_unstale":"2023-10-30T12:52:02.360999+0000","last_undegraded":"2023-10-30T12:52:02.360999+0000","last_fullsized":"2023-10-30T12:52:02.360999+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'497","last_scrub_stamp":"2023-10-30T12:52:02.360776+0000","last_deep_scrub":"66'497","last_deep_scrub_stamp":"2023-10-30T12:52:02.360776+0000","last_clean_scrub_stamp":"2023-10-30T12:52:02.360776+0000","objects_scrubbed":4,"log_size":497,"log_dups_size":0,"ondisk_log_size":497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:40:59.407446+0000","scrub_duration":0.065243972999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":45183,"num_write":427,"num_write_kb":16635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"66'536","reported_seq":2693,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:45.113576+0000","last_change":"2023-10-30T12:51:45.113576+0000","last_active":"2023-10-30T12:51:45.113576+0000","last_peered":"2023-10-30T12:51:45.113576+0000","last_clean":"2023-10-30T12:51:45.113576+0000","last_became_active":"2023-10-30T12:28:50.949122+0000","last_became_peered":"2023-10-30T12:28:50.949122+0000","last_unstale":"2023-10-30T12:51:45.113576+0000","last_undegraded":"2023-10-30T12:51:45.113576+0000","last_fullsized":"2023-10-30T12:51:45.113576+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'536","last_scrub_stamp":"2023-10-30T12:51:45.113133+0000","last_deep_scrub":"66'536","last_deep_scrub_stamp":"2023-10-30T12:51:45.113133+0000","last_clean_scrub_stamp":"2023-10-30T12:51:45.113133+0000","objects_scrubbed":16,"log_size":536,"log_dups_size":0,"ondisk_log_size":536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:45:58.801601+0000","scrub_duration":0.060099805999999999,"objects_trimmed":0,"snaptrim_duration":0.004393348,"stat_sum":{"num_bytes":12684737,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":35,"num_read_kb":65,"num_write":464,"num_write_kb":14146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"66'267","reported_seq":346,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:36.014856+0000","last_change":"2023-10-30T12:51:36.014856+0000","last_active":"2023-10-30T12:51:36.014856+0000","last_peered":"2023-10-30T12:51:36.014856+0000","last_clean":"2023-10-30T12:51:36.014856+0000","last_became_active":"2023-10-30T12:28:50.358310+0000","last_became_peered":"2023-10-30T12:28:50.358310+0000","last_unstale":"2023-10-30T12:51:36.014856+0000","last_undegraded":"2023-10-30T12:51:36.014856+0000","last_fullsized":"2023-10-30T12:51:36.014856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'267","last_scrub_stamp":"2023-10-30T12:51:36.014733+0000","last_deep_scrub":"66'267","last_deep_scrub_stamp":"2023-10-30T12:51:36.014733+0000","last_clean_scrub_stamp":"2023-10-30T12:51:36.014733+0000","objects_scrubbed":2,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:22:01.486529+0000","scrub_duration":0.015176472999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":5758,"num_write":222,"num_write_kb":4242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7112,"num_omap_keys":14,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"66'554","reported_seq":2709,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:38.621754+0000","last_change":"2023-10-30T12:51:38.621754+0000","last_active":"2023-10-30T12:51:38.621754+0000","last_peered":"2023-10-30T12:51:38.621754+0000","last_clean":"2023-10-30T12:51:38.621754+0000","last_became_active":"2023-10-30T12:28:50.965626+0000","last_became_peered":"2023-10-30T12:28:50.965626+0000","last_unstale":"2023-10-30T12:51:38.621754+0000","last_undegraded":"2023-10-30T12:51:38.621754+0000","last_fullsized":"2023-10-30T12:51:38.621754+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'554","last_scrub_stamp":"2023-10-30T12:51:38.621402+0000","last_deep_scrub":"66'554","last_deep_scrub_stamp":"2023-10-30T12:51:38.621402+0000","last_clean_scrub_stamp":"2023-10-30T12:51:38.621402+0000","objects_scrubbed":15,"log_size":554,"log_dups_size":0,"ondisk_log_size":554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:32:24.000396+0000","scrub_duration":0.10283745599999999,"objects_trimmed":0,"snaptrim_duration":0.0079099229999999993,"stat_sum":{"num_bytes":29382851,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":33,"num_read_kb":4109,"num_write":477,"num_write_kb":37019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"66'404","reported_seq":485,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:53.568757+0000","last_change":"2023-10-30T12:51:53.568971+0000","last_active":"2023-10-30T12:51:53.568757+0000","last_peered":"2023-10-30T12:51:53.568757+0000","last_clean":"2023-10-30T12:51:53.568757+0000","last_became_active":"2023-10-30T12:28:50.395090+0000","last_became_peered":"2023-10-30T12:28:50.395090+0000","last_unstale":"2023-10-30T12:51:53.568757+0000","last_undegraded":"2023-10-30T12:51:53.568757+0000","last_fullsized":"2023-10-30T12:51:53.568757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'404","last_scrub_stamp":"2023-10-30T12:51:53.568460+0000","last_deep_scrub":"66'404","last_deep_scrub_stamp":"2023-10-30T12:51:53.568460+0000","last_clean_scrub_stamp":"2023-10-30T12:51:53.568460+0000","objects_scrubbed":5,"log_size":404,"log_dups_size":0,"ondisk_log_size":404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:27:52.316450+0000","scrub_duration":0.055684696999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582934,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":8195,"num_write":330,"num_write_kb":12489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12012,"num_omap_keys":22,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"66'568","reported_seq":2740,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:04.826459+0000","last_change":"2023-10-30T12:52:04.826539+0000","last_active":"2023-10-30T12:52:04.826459+0000","last_peered":"2023-10-30T12:52:04.826459+0000","last_clean":"2023-10-30T12:52:04.826459+0000","last_became_active":"2023-10-30T12:28:50.929337+0000","last_became_peered":"2023-10-30T12:28:50.929337+0000","last_unstale":"2023-10-30T12:52:04.826459+0000","last_undegraded":"2023-10-30T12:52:04.826459+0000","last_fullsized":"2023-10-30T12:52:04.826459+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'568","last_scrub_stamp":"2023-10-30T12:52:04.826355+0000","last_deep_scrub":"66'568","last_deep_scrub_stamp":"2023-10-30T12:52:04.826355+0000","last_clean_scrub_stamp":"2023-10-30T12:52:04.826355+0000","objects_scrubbed":17,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:47:52.867591+0000","scrub_duration":0.039134962000000002,"objects_trimmed":0,"snaptrim_duration":0.0023039890000000002,"stat_sum":{"num_bytes":14041074,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":50,"num_read_kb":18,"num_write":487,"num_write_kb":15488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"66'356","reported_seq":443,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:41.110896+0000","last_change":"2023-10-30T12:51:41.110963+0000","last_active":"2023-10-30T12:51:41.110896+0000","last_peered":"2023-10-30T12:51:41.110896+0000","last_clean":"2023-10-30T12:51:41.110896+0000","last_became_active":"2023-10-30T12:28:50.380247+0000","last_became_peered":"2023-10-30T12:28:50.380247+0000","last_unstale":"2023-10-30T12:51:41.110896+0000","last_undegraded":"2023-10-30T12:51:41.110896+0000","last_fullsized":"2023-10-30T12:51:41.110896+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'356","last_scrub_stamp":"2023-10-30T12:51:41.110806+0000","last_deep_scrub":"66'356","last_deep_scrub_stamp":"2023-10-30T12:51:41.110806+0000","last_clean_scrub_stamp":"2023-10-30T12:51:41.110806+0000","objects_scrubbed":5,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:36:44.682836+0000","scrub_duration":0.090051573999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":8272,"num_write":528,"num_write_kb":5259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18674,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"66'544","reported_seq":2695,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:16.880659+0000","last_change":"2023-10-30T12:52:16.880748+0000","last_active":"2023-10-30T12:52:16.880659+0000","last_peered":"2023-10-30T12:52:16.880659+0000","last_clean":"2023-10-30T12:52:16.880659+0000","last_became_active":"2023-10-30T12:28:50.973086+0000","last_became_peered":"2023-10-30T12:28:50.973086+0000","last_unstale":"2023-10-30T12:52:16.880659+0000","last_undegraded":"2023-10-30T12:52:16.880659+0000","last_fullsized":"2023-10-30T12:52:16.880659+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'544","last_scrub_stamp":"2023-10-30T12:52:16.880542+0000","last_deep_scrub":"66'544","last_deep_scrub_stamp":"2023-10-30T12:52:16.880542+0000","last_clean_scrub_stamp":"2023-10-30T12:52:16.880542+0000","objects_scrubbed":12,"log_size":544,"log_dups_size":0,"ondisk_log_size":544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:23:12.413779+0000","scrub_duration":0.056029834000000001,"objects_trimmed":0,"snaptrim_duration":0.0034133169999999999,"stat_sum":{"num_bytes":16807043,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":30,"num_read_kb":14,"num_write":492,"num_write_kb":22119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"66'273","reported_seq":356,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:08.564387+0000","last_change":"2023-10-30T12:51:43.564434+0000","last_active":"2023-10-30T12:52:08.564387+0000","last_peered":"2023-10-30T12:52:08.564387+0000","last_clean":"2023-10-30T12:52:08.564387+0000","last_became_active":"2023-10-30T12:28:50.385924+0000","last_became_peered":"2023-10-30T12:28:50.385924+0000","last_unstale":"2023-10-30T12:52:08.564387+0000","last_undegraded":"2023-10-30T12:52:08.564387+0000","last_fullsized":"2023-10-30T12:52:08.564387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'261","last_scrub_stamp":"2023-10-30T12:51:43.564089+0000","last_deep_scrub":"66'261","last_deep_scrub_stamp":"2023-10-30T12:51:43.564089+0000","last_clean_scrub_stamp":"2023-10-30T12:51:43.564089+0000","objects_scrubbed":5,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:34:21.648949+0000","scrub_duration":0.022937485000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3279638,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":207,"num_write_kb":3335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13032,"num_omap_keys":26,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"66'541","reported_seq":2705,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:49.313126+0000","last_change":"2023-10-30T12:51:49.313203+0000","last_active":"2023-10-30T12:51:49.313126+0000","last_peered":"2023-10-30T12:51:49.313126+0000","last_clean":"2023-10-30T12:51:49.313126+0000","last_became_active":"2023-10-30T12:28:50.929259+0000","last_became_peered":"2023-10-30T12:28:50.929259+0000","last_unstale":"2023-10-30T12:51:49.313126+0000","last_undegraded":"2023-10-30T12:51:49.313126+0000","last_fullsized":"2023-10-30T12:51:49.313126+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'541","last_scrub_stamp":"2023-10-30T12:51:49.313036+0000","last_deep_scrub":"66'541","last_deep_scrub_stamp":"2023-10-30T12:51:49.313036+0000","last_clean_scrub_stamp":"2023-10-30T12:51:49.313036+0000","objects_scrubbed":17,"log_size":541,"log_dups_size":0,"ondisk_log_size":541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:00:33.907215+0000","scrub_duration":0.028828065,"objects_trimmed":0,"snaptrim_duration":0.0029033230000000002,"stat_sum":{"num_bytes":8423823,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":43,"num_read_kb":22,"num_write":477,"num_write_kb":11480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"66'379","reported_seq":470,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:46.586619+0000","last_change":"2023-10-30T12:51:46.586619+0000","last_active":"2023-10-30T12:51:46.586619+0000","last_peered":"2023-10-30T12:51:46.586619+0000","last_clean":"2023-10-30T12:51:46.586619+0000","last_became_active":"2023-10-30T12:28:50.385305+0000","last_became_peered":"2023-10-30T12:28:50.385305+0000","last_unstale":"2023-10-30T12:51:46.586619+0000","last_undegraded":"2023-10-30T12:51:46.586619+0000","last_fullsized":"2023-10-30T12:51:46.586619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'378","last_scrub_stamp":"2023-10-30T12:51:46.586293+0000","last_deep_scrub":"66'378","last_deep_scrub_stamp":"2023-10-30T12:51:46.586293+0000","last_clean_scrub_stamp":"2023-10-30T12:51:46.586293+0000","objects_scrubbed":5,"log_size":379,"log_dups_size":0,"ondisk_log_size":379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:03:32.538098+0000","scrub_duration":0.051218165000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":37322,"num_write":310,"num_write_kb":12446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":508,"num_omap_keys":1,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"66'562","reported_seq":2720,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:38.230220+0000","last_change":"2023-10-30T12:51:38.230463+0000","last_active":"2023-10-30T12:51:38.230220+0000","last_peered":"2023-10-30T12:51:38.230220+0000","last_clean":"2023-10-30T12:51:38.230220+0000","last_became_active":"2023-10-30T12:28:50.940846+0000","last_became_peered":"2023-10-30T12:28:50.940846+0000","last_unstale":"2023-10-30T12:51:38.230220+0000","last_undegraded":"2023-10-30T12:51:38.230220+0000","last_fullsized":"2023-10-30T12:51:38.230220+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'562","last_scrub_stamp":"2023-10-30T12:51:38.229988+0000","last_deep_scrub":"66'562","last_deep_scrub_stamp":"2023-10-30T12:51:38.229988+0000","last_clean_scrub_stamp":"2023-10-30T12:51:38.229988+0000","objects_scrubbed":21,"log_size":562,"log_dups_size":0,"ondisk_log_size":562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:31:03.842668+0000","scrub_duration":0.057637995999999997,"objects_trimmed":0,"snaptrim_duration":0.0046016829999999996,"stat_sum":{"num_bytes":12679067,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":37,"num_read_kb":7,"num_write":498,"num_write_kb":18971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"66'448","reported_seq":534,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:42.982397+0000","last_change":"2023-10-30T12:51:42.982473+0000","last_active":"2023-10-30T12:51:42.982397+0000","last_peered":"2023-10-30T12:51:42.982397+0000","last_clean":"2023-10-30T12:51:42.982397+0000","last_became_active":"2023-10-30T12:28:50.392068+0000","last_became_peered":"2023-10-30T12:28:50.392068+0000","last_unstale":"2023-10-30T12:51:42.982397+0000","last_undegraded":"2023-10-30T12:51:42.982397+0000","last_fullsized":"2023-10-30T12:51:42.982397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'448","last_scrub_stamp":"2023-10-30T12:51:42.982269+0000","last_deep_scrub":"66'448","last_deep_scrub_stamp":"2023-10-30T12:51:42.982269+0000","last_clean_scrub_stamp":"2023-10-30T12:51:42.982269+0000","objects_scrubbed":7,"log_size":448,"log_dups_size":0,"ondisk_log_size":448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:55:25.126896+0000","scrub_duration":0.032966065000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388698,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":19,"num_read_kb":11375,"num_write":387,"num_write_kb":8544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":27300,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"66'570","reported_seq":2735,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:49.808293+0000","last_change":"2023-10-30T12:51:49.808358+0000","last_active":"2023-10-30T12:51:49.808293+0000","last_peered":"2023-10-30T12:51:49.808293+0000","last_clean":"2023-10-30T12:51:49.808293+0000","last_became_active":"2023-10-30T12:28:50.931898+0000","last_became_peered":"2023-10-30T12:28:50.931898+0000","last_unstale":"2023-10-30T12:51:49.808293+0000","last_undegraded":"2023-10-30T12:51:49.808293+0000","last_fullsized":"2023-10-30T12:51:49.808293+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'570","last_scrub_stamp":"2023-10-30T12:51:49.808201+0000","last_deep_scrub":"66'570","last_deep_scrub_stamp":"2023-10-30T12:51:49.808201+0000","last_clean_scrub_stamp":"2023-10-30T12:51:49.808201+0000","objects_scrubbed":18,"log_size":570,"log_dups_size":0,"ondisk_log_size":570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:46:36.932799+0000","scrub_duration":0.022998979999999999,"objects_trimmed":0,"snaptrim_duration":0.003570148,"stat_sum":{"num_bytes":4263636,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":44,"num_read_kb":18,"num_write":507,"num_write_kb":10484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"66'557","reported_seq":639,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:42.047483+0000","last_change":"2023-10-30T12:51:42.047533+0000","last_active":"2023-10-30T12:51:42.047483+0000","last_peered":"2023-10-30T12:51:42.047483+0000","last_clean":"2023-10-30T12:51:42.047483+0000","last_became_active":"2023-10-30T12:28:50.364797+0000","last_became_peered":"2023-10-30T12:28:50.364797+0000","last_unstale":"2023-10-30T12:51:42.047483+0000","last_undegraded":"2023-10-30T12:51:42.047483+0000","last_fullsized":"2023-10-30T12:51:42.047483+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'557","last_scrub_stamp":"2023-10-30T12:51:42.047398+0000","last_deep_scrub":"66'557","last_deep_scrub_stamp":"2023-10-30T12:51:42.047398+0000","last_clean_scrub_stamp":"2023-10-30T12:51:42.047398+0000","objects_scrubbed":4,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:36:26.216418+0000","scrub_duration":0.054259416999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":10,"num_read_kb":8489,"num_write":691,"num_write_kb":13252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"66'568","reported_seq":2738,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:50.791335+0000","last_change":"2023-10-30T12:51:50.791392+0000","last_active":"2023-10-30T12:51:50.791335+0000","last_peered":"2023-10-30T12:51:50.791335+0000","last_clean":"2023-10-30T12:51:50.791335+0000","last_became_active":"2023-10-30T12:28:50.935657+0000","last_became_peered":"2023-10-30T12:28:50.935657+0000","last_unstale":"2023-10-30T12:51:50.791335+0000","last_undegraded":"2023-10-30T12:51:50.791335+0000","last_fullsized":"2023-10-30T12:51:50.791335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'568","last_scrub_stamp":"2023-10-30T12:51:50.791255+0000","last_deep_scrub":"66'568","last_deep_scrub_stamp":"2023-10-30T12:51:50.791255+0000","last_clean_scrub_stamp":"2023-10-30T12:51:50.791255+0000","objects_scrubbed":15,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:03:42.487468+0000","scrub_duration":0.039011377999999999,"objects_trimmed":0,"snaptrim_duration":0.0038543219999999999,"stat_sum":{"num_bytes":16829664,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":49,"num_read_kb":4112,"num_write":499,"num_write_kb":17598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"66'249","reported_seq":338,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:54.509810+0000","last_change":"2023-10-30T12:51:54.509947+0000","last_active":"2023-10-30T12:51:54.509810+0000","last_peered":"2023-10-30T12:51:54.509810+0000","last_clean":"2023-10-30T12:51:54.509810+0000","last_became_active":"2023-10-30T12:28:50.392807+0000","last_became_peered":"2023-10-30T12:28:50.392807+0000","last_unstale":"2023-10-30T12:51:54.509810+0000","last_undegraded":"2023-10-30T12:51:54.509810+0000","last_fullsized":"2023-10-30T12:51:54.509810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'249","last_scrub_stamp":"2023-10-30T12:51:54.509581+0000","last_deep_scrub":"66'249","last_deep_scrub_stamp":"2023-10-30T12:51:54.509581+0000","last_clean_scrub_stamp":"2023-10-30T12:51:54.509581+0000","objects_scrubbed":5,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:12:36.299114+0000","scrub_duration":0.039282915000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":24596,"num_write":178,"num_write_kb":8294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"66'524","reported_seq":2683,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:02.742094+0000","last_change":"2023-10-30T12:52:02.742242+0000","last_active":"2023-10-30T12:52:02.742094+0000","last_peered":"2023-10-30T12:52:02.742094+0000","last_clean":"2023-10-30T12:52:02.742094+0000","last_became_active":"2023-10-30T12:28:50.954291+0000","last_became_peered":"2023-10-30T12:28:50.954291+0000","last_unstale":"2023-10-30T12:52:02.742094+0000","last_undegraded":"2023-10-30T12:52:02.742094+0000","last_fullsized":"2023-10-30T12:52:02.742094+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'524","last_scrub_stamp":"2023-10-30T12:52:02.741848+0000","last_deep_scrub":"66'524","last_deep_scrub_stamp":"2023-10-30T12:52:02.741848+0000","last_clean_scrub_stamp":"2023-10-30T12:52:02.741848+0000","objects_scrubbed":19,"log_size":524,"log_dups_size":0,"ondisk_log_size":524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:24:29.376137+0000","scrub_duration":0.074825178000000006,"objects_trimmed":0,"snaptrim_duration":0.0047974480000000002,"stat_sum":{"num_bytes":12709282,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":38,"num_read_kb":15,"num_write":455,"num_write_kb":14651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"66'526","reported_seq":2688,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.476097+0000","last_change":"2023-10-30T12:38:17.120085+0000","last_active":"2023-10-30T12:51:30.476097+0000","last_peered":"2023-10-30T12:51:30.476097+0000","last_clean":"2023-10-30T12:51:30.476097+0000","last_became_active":"2023-10-30T12:28:50.962817+0000","last_became_peered":"2023-10-30T12:28:50.962817+0000","last_unstale":"2023-10-30T12:51:30.476097+0000","last_undegraded":"2023-10-30T12:51:30.476097+0000","last_fullsized":"2023-10-30T12:51:30.476097+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":526,"log_dups_size":0,"ondisk_log_size":526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070898599999999999,"stat_sum":{"num_bytes":8448766,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":50,"num_read_kb":25,"num_write":483,"num_write_kb":14592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"66'176","reported_seq":261,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:51.555385+0000","last_change":"2023-10-30T12:51:51.555592+0000","last_active":"2023-10-30T12:51:51.555385+0000","last_peered":"2023-10-30T12:51:51.555385+0000","last_clean":"2023-10-30T12:51:51.555385+0000","last_became_active":"2023-10-30T12:28:50.401677+0000","last_became_peered":"2023-10-30T12:28:50.401677+0000","last_unstale":"2023-10-30T12:51:51.555385+0000","last_undegraded":"2023-10-30T12:51:51.555385+0000","last_fullsized":"2023-10-30T12:51:51.555385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'176","last_scrub_stamp":"2023-10-30T12:51:51.555145+0000","last_deep_scrub":"66'176","last_deep_scrub_stamp":"2023-10-30T12:51:51.555145+0000","last_clean_scrub_stamp":"2023-10-30T12:51:51.555145+0000","objects_scrubbed":3,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:07:33.229250+0000","scrub_duration":0.033175365999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":20498,"num_write":107,"num_write_kb":8247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"66'500","reported_seq":2656,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:30.497349+0000","last_change":"2023-10-30T12:38:17.119058+0000","last_active":"2023-10-30T12:51:30.497349+0000","last_peered":"2023-10-30T12:51:30.497349+0000","last_clean":"2023-10-30T12:51:30.497349+0000","last_became_active":"2023-10-30T12:28:50.967591+0000","last_became_peered":"2023-10-30T12:28:50.967591+0000","last_unstale":"2023-10-30T12:51:30.497349+0000","last_undegraded":"2023-10-30T12:51:30.497349+0000","last_fullsized":"2023-10-30T12:51:30.497349+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":500,"log_dups_size":0,"ondisk_log_size":500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081968709999999997,"stat_sum":{"num_bytes":10185055,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":44,"num_read_kb":1610,"num_write":440,"num_write_kb":22911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"66'370","reported_seq":466,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:50.641824+0000","last_change":"2023-10-30T12:51:50.641945+0000","last_active":"2023-10-30T12:51:50.641824+0000","last_peered":"2023-10-30T12:51:50.641824+0000","last_clean":"2023-10-30T12:51:50.641824+0000","last_became_active":"2023-10-30T12:28:50.392021+0000","last_became_peered":"2023-10-30T12:28:50.392021+0000","last_unstale":"2023-10-30T12:51:50.641824+0000","last_undegraded":"2023-10-30T12:51:50.641824+0000","last_fullsized":"2023-10-30T12:51:50.641824+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'370","last_scrub_stamp":"2023-10-30T12:51:50.641619+0000","last_deep_scrub":"66'370","last_deep_scrub_stamp":"2023-10-30T12:51:50.641619+0000","last_clean_scrub_stamp":"2023-10-30T12:51:50.641619+0000","objects_scrubbed":8,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:05:24.267901+0000","scrub_duration":0.078330257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":57384,"num_write":298,"num_write_kb":16576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22858,"num_omap_keys":43,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"66'521","reported_seq":2687,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:00.262491+0000","last_change":"2023-10-30T12:52:00.262554+0000","last_active":"2023-10-30T12:52:00.262491+0000","last_peered":"2023-10-30T12:52:00.262491+0000","last_clean":"2023-10-30T12:52:00.262491+0000","last_became_active":"2023-10-30T12:28:50.967186+0000","last_became_peered":"2023-10-30T12:28:50.967186+0000","last_unstale":"2023-10-30T12:52:00.262491+0000","last_undegraded":"2023-10-30T12:52:00.262491+0000","last_fullsized":"2023-10-30T12:52:00.262491+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'521","last_scrub_stamp":"2023-10-30T12:52:00.262413+0000","last_deep_scrub":"66'521","last_deep_scrub_stamp":"2023-10-30T12:52:00.262413+0000","last_clean_scrub_stamp":"2023-10-30T12:52:00.262413+0000","objects_scrubbed":15,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:28:18.321174+0000","scrub_duration":0.025087834999999999,"objects_trimmed":0,"snaptrim_duration":0.003136738,"stat_sum":{"num_bytes":73381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":44,"num_read_kb":17,"num_write":453,"num_write_kb":3099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"66'231","reported_seq":316,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:37.827427+0000","last_change":"2023-10-30T12:51:37.827474+0000","last_active":"2023-10-30T12:51:37.827427+0000","last_peered":"2023-10-30T12:51:37.827427+0000","last_clean":"2023-10-30T12:51:37.827427+0000","last_became_active":"2023-10-30T12:28:50.362929+0000","last_became_peered":"2023-10-30T12:28:50.362929+0000","last_unstale":"2023-10-30T12:51:37.827427+0000","last_undegraded":"2023-10-30T12:51:37.827427+0000","last_fullsized":"2023-10-30T12:51:37.827427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'231","last_scrub_stamp":"2023-10-30T12:51:37.827339+0000","last_deep_scrub":"66'231","last_deep_scrub_stamp":"2023-10-30T12:51:37.827339+0000","last_clean_scrub_stamp":"2023-10-30T12:51:37.827339+0000","objects_scrubbed":4,"log_size":231,"log_dups_size":0,"ondisk_log_size":231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:06:46.012417+0000","scrub_duration":0.026390808000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":9,"num_read_kb":32768,"num_write":162,"num_write_kb":8341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":4,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"66'588","reported_seq":2740,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:11.804649+0000","last_change":"2023-10-30T12:52:11.804712+0000","last_active":"2023-10-30T12:52:11.804649+0000","last_peered":"2023-10-30T12:52:11.804649+0000","last_clean":"2023-10-30T12:52:11.804649+0000","last_became_active":"2023-10-30T12:28:50.937458+0000","last_became_peered":"2023-10-30T12:28:50.937458+0000","last_unstale":"2023-10-30T12:52:11.804649+0000","last_undegraded":"2023-10-30T12:52:11.804649+0000","last_fullsized":"2023-10-30T12:52:11.804649+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'588","last_scrub_stamp":"2023-10-30T12:52:11.804563+0000","last_deep_scrub":"66'588","last_deep_scrub_stamp":"2023-10-30T12:52:11.804563+0000","last_clean_scrub_stamp":"2023-10-30T12:52:11.804563+0000","objects_scrubbed":16,"log_size":588,"log_dups_size":0,"ondisk_log_size":588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:39:38.027390+0000","scrub_duration":0.058327041000000003,"objects_trimmed":0,"snaptrim_duration":0.003544045,"stat_sum":{"num_bytes":16852009,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":30,"num_read_kb":11,"num_write":518,"num_write_kb":19789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"66'450","reported_seq":554,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:34.515168+0000","last_change":"2023-10-30T12:51:34.515168+0000","last_active":"2023-10-30T12:51:34.515168+0000","last_peered":"2023-10-30T12:51:34.515168+0000","last_clean":"2023-10-30T12:51:34.515168+0000","last_became_active":"2023-10-30T12:28:50.386522+0000","last_became_peered":"2023-10-30T12:28:50.386522+0000","last_unstale":"2023-10-30T12:51:34.515168+0000","last_undegraded":"2023-10-30T12:51:34.515168+0000","last_fullsized":"2023-10-30T12:51:34.515168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'450","last_scrub_stamp":"2023-10-30T12:51:34.515038+0000","last_deep_scrub":"66'450","last_deep_scrub_stamp":"2023-10-30T12:51:34.515038+0000","last_clean_scrub_stamp":"2023-10-30T12:51:34.515038+0000","objects_scrubbed":7,"log_size":450,"log_dups_size":0,"ondisk_log_size":450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:32:31.022257+0000","scrub_duration":0.052865532999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583494,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":64,"num_read_kb":24634,"num_write":377,"num_write_kb":12490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":757298097,"num_objects":1192,"num_object_clones":0,"num_object_copies":2384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1192,"num_whiteouts":0,"num_read":2811,"num_read_kb":100236,"num_write":31900,"num_write_kb":1049744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1519566848,"data_stored":1514596194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36058,"ondisk_log_size":36058,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":305681866,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1441,"num_read_kb":712828,"num_write":10564,"num_write_kb":307242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":78,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":305728,"num_omap_keys":583,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":611524608,"data_stored":611363732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11971,"ondisk_log_size":11971,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542432,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":394172,"kb_used_data":322740,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93977668,"statfs":{"total":96636764160,"available":96233132032,"internally_reserved":0,"allocated":330485760,"data_stored":329320520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":73136587},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.78300000000000003,"15min":0.81799999999999995},"min":{"1min":0.52500000000000002,"5min":0.45900000000000002,"15min":0.40000000000000002},"max":{"1min":1.093,"5min":1.2050000000000001,"15min":1.595},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.78400000000000003,"15min":0.84199999999999997},"min":{"1min":0.49099999999999999,"5min":0.441,"15min":0.40999999999999998},"max":{"1min":1.113,"5min":1.302,"15min":1.4830000000000001},"last":1.0089999999999999}]},{"osd":1,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.85099999999999998,"15min":0.88800000000000001},"min":{"1min":0.57199999999999995,"5min":0.52700000000000002,"15min":0.441},"max":{"1min":1.2709999999999999,"5min":3.4329999999999998,"15min":3.4329999999999998},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.85399999999999998,"15min":0.88500000000000001},"min":{"1min":0.56999999999999995,"5min":0.505,"15min":0.505},"max":{"1min":1.1419999999999999,"5min":1.383,"15min":1.3919999999999999},"last":0.79800000000000004}]},{"osd":2,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.89000000000000001,"15min":0.92200000000000004},"min":{"1min":0.58899999999999997,"5min":0.55100000000000005,"15min":0.45900000000000002},"max":{"1min":1.163,"5min":3.4820000000000002,"15min":3.4820000000000002},"last":0.97599999999999998},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.877,"15min":0.83699999999999997},"min":{"1min":0.65000000000000002,"5min":0.52800000000000002,"15min":0.47099999999999997},"max":{"1min":1.2330000000000001,"5min":1.4570000000000001,"15min":1.6639999999999999},"last":1.1639999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":1.0069999999999999,"15min":0.96899999999999997},"min":{"1min":0.61299999999999999,"5min":0.60999999999999999,"15min":0.45500000000000002},"max":{"1min":1.1439999999999999,"5min":3.5190000000000001,"15min":3.5190000000000001},"last":1.127},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.95599999999999996,"15min":0.99399999999999999},"min":{"1min":0.72899999999999998,"5min":0.54500000000000004,"15min":0.441},"max":{"1min":1.3220000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":1.093}]},{"osd":4,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.97599999999999998,"15min":1.002},"min":{"1min":0.63800000000000001,"5min":0.42499999999999999,"15min":0.34399999999999997},"max":{"1min":1.2070000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":1.2410000000000001},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.999,"15min":1.042},"min":{"1min":0.61799999999999999,"5min":0.45000000000000001,"15min":0.33800000000000002},"max":{"1min":1.1910000000000001,"5min":1.746,"15min":1.746},"last":1.2949999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.016,"15min":0.96099999999999997},"min":{"1min":0.76700000000000002,"5min":0.53700000000000003,"15min":0.43099999999999999},"max":{"1min":1.2949999999999999,"5min":1.6160000000000001,"15min":1.7549999999999999},"last":1.0509999999999999},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.117,"15min":1.0680000000000001},"min":{"1min":0.71499999999999997,"5min":0.57899999999999996,"15min":0.35999999999999999},"max":{"1min":1.2370000000000001,"5min":2.1600000000000001,"15min":2.1600000000000001},"last":1.379}]},{"osd":6,"last update":"Mon Oct 30 12:51:24 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.141,"15min":1.123},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.39000000000000001},"max":{"1min":1.216,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.2070000000000001},{"interface":"front","average":{"1min":1,"5min":1.1259999999999999,"15min":1.167},"min":{"1min":0.68200000000000005,"5min":0.56299999999999994,"15min":0.47199999999999998},"max":{"1min":1.3300000000000001,"5min":3.8140000000000001,"15min":3.8140000000000001},"last":1.27}]}]},{"osd":6,"up_from":14,"seq":60129542431,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298232,"kb_used_data":235184,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94073608,"statfs":{"total":96636764160,"available":96331374592,"internally_reserved":0,"allocated":240828416,"data_stored":239968999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":64551368},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.83299999999999996,"15min":0.85999999999999999},"min":{"1min":0.52900000000000003,"5min":0.46600000000000003,"15min":0.35599999999999998},"max":{"1min":1.155,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":1.256},{"interface":"front","average":{"1min":0.749,"5min":0.77000000000000002,"15min":0.78800000000000003},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.42999999999999999},"max":{"1min":1.05,"5min":1.109,"15min":1.214},"last":0.97399999999999998}]},{"osd":1,"last update":"Mon Oct 30 12:51:28 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.82099999999999995,"15min":0.81599999999999995},"min":{"1min":0.45000000000000001,"5min":0.42299999999999999,"15min":0.317},"max":{"1min":1.23,"5min":1.2809999999999999,"15min":1.3140000000000001},"last":1.1060000000000001},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.85199999999999998,"15min":0.876},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.40999999999999998},"max":{"1min":1.1399999999999999,"5min":1.2869999999999999,"15min":1.423},"last":1.026}]},{"osd":2,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.93600000000000005,"15min":0.97499999999999998},"min":{"1min":0.495,"5min":0.495,"15min":0.38800000000000001},"max":{"1min":1.373,"5min":1.47,"15min":4.3360000000000003},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.0409999999999999,"5min":0.97999999999999998,"15min":1.0089999999999999},"min":{"1min":0.65000000000000002,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.502,"5min":1.502,"15min":1.5629999999999999},"last":1.1619999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.996,"15min":1.042},"min":{"1min":0.64700000000000002,"5min":0.59099999999999997,"15min":0.58999999999999997},"max":{"1min":1.3340000000000001,"5min":1.5509999999999999,"15min":1.6459999999999999},"last":1.048},{"interface":"front","average":{"1min":1.02,"5min":1.0680000000000001,"15min":1.101},"min":{"1min":0.72299999999999998,"5min":0.63300000000000001,"15min":0.59699999999999998},"max":{"1min":1.417,"5min":1.552,"15min":1.7969999999999999},"last":1.2090000000000001}]},{"osd":4,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.149,"15min":1.075},"min":{"1min":0.72599999999999998,"5min":0.69899999999999995,"15min":0.51500000000000001},"max":{"1min":1.585,"5min":1.6499999999999999,"15min":1.7649999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.89400000000000002,"15min":0.96499999999999997},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.42099999999999999},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.881},"last":1.3009999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":1.0720000000000001,"15min":1.087},"min":{"1min":0.54100000000000004,"5min":0.53200000000000003,"15min":0.48399999999999999},"max":{"1min":1.2849999999999999,"5min":1.75,"15min":2.2770000000000001},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.056,"5min":1.159,"15min":1.0649999999999999},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.46999999999999997},"max":{"1min":1.4610000000000001,"5min":1.6950000000000001,"15min":4.7850000000000001},"last":1.0780000000000001}]},{"osd":7,"last update":"Mon Oct 30 12:51:27 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.036,"15min":1.0029999999999999},"min":{"1min":0.68799999999999994,"5min":0.64100000000000001,"15min":0.46700000000000003},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.825},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.0629999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.48999999999999999},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.895},"last":1.0029999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542432,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":342544,"kb_used_data":271112,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94029296,"statfs":{"total":96636764160,"available":96285999104,"internally_reserved":0,"allocated":277618688,"data_stored":276486603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.78400000000000003,"15min":0.80300000000000005},"min":{"1min":0.56299999999999994,"5min":0.33900000000000002,"15min":0.29399999999999998},"max":{"1min":1.1970000000000001,"5min":1.238,"15min":2.4729999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.79700000000000004,"15min":0.80400000000000005},"min":{"1min":0.61199999999999999,"5min":0.39200000000000002,"15min":0.33800000000000002},"max":{"1min":1.2390000000000001,"5min":1.3009999999999999,"15min":1.8759999999999999},"last":0.72099999999999997}]},{"osd":1,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.84199999999999997,"15min":0.84399999999999997},"min":{"1min":0.57399999999999995,"5min":0.40999999999999998,"15min":0.38900000000000001},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":3.0190000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.58499999999999996,"5min":0.44800000000000001,"15min":0.373},"max":{"1min":1.264,"5min":1.276,"15min":1.4870000000000001},"last":0.82399999999999995}]},{"osd":2,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.88400000000000001,"15min":0.84699999999999998},"min":{"1min":0.67300000000000004,"5min":0.55400000000000005,"15min":0.38400000000000001},"max":{"1min":1.157,"5min":1.264,"15min":2.2949999999999999},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.81699999999999995,"15min":0.876},"min":{"1min":0.63600000000000001,"5min":0.434,"15min":0.434},"max":{"1min":1.1140000000000001,"5min":1.171,"15min":1.5800000000000001},"last":0.877}]},{"osd":3,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.91800000000000004,"15min":0.93700000000000006},"min":{"1min":0.57199999999999995,"5min":0.53800000000000003,"15min":0.47899999999999998},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.756},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.93600000000000005,"15min":0.94599999999999995},"min":{"1min":0.755,"5min":0.498,"15min":0.41399999999999998},"max":{"1min":1.204,"5min":1.4119999999999999,"15min":2.3679999999999999},"last":1.0229999999999999}]},{"osd":4,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.84499999999999997,"15min":0.90900000000000003},"min":{"1min":0.47499999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.304,"5min":1.304,"15min":3.1190000000000002},"last":1.351},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.083,"15min":1.091},"min":{"1min":0.59499999999999997,"5min":0.58699999999999997,"15min":0.50900000000000001},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":2.645},"last":1.5149999999999999}]},{"osd":6,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.038,"15min":1.042},"min":{"1min":0.51100000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.466,"5min":1.466,"15min":3.1859999999999999},"last":1.4570000000000001},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.056,"15min":1.1060000000000001},"min":{"1min":0.67000000000000004,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.3260000000000001,"5min":1.47,"15min":3.371},"last":1.3069999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.0609999999999999,"15min":1.0800000000000001},"min":{"1min":0.65200000000000002,"5min":0.51400000000000001,"15min":0.47299999999999998},"max":{"1min":1.494,"5min":1.508,"15min":3.5310000000000001},"last":1.129},{"interface":"front","average":{"1min":1.282,"5min":1.1970000000000001,"15min":1.2310000000000001},"min":{"1min":0.96799999999999997,"5min":0.64700000000000002,"15min":0.61299999999999999},"max":{"1min":1.605,"5min":1.605,"15min":3.6930000000000001},"last":1.208}]}]},{"osd":4,"up_from":14,"seq":60129542431,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369460,"kb_used_data":293868,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94002380,"statfs":{"total":96636764160,"available":96258437120,"internally_reserved":0,"allocated":300920832,"data_stored":299840560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.80100000000000005,"15min":0.92600000000000005},"min":{"1min":0.46800000000000003,"5min":0.433,"15min":0.31},"max":{"1min":1.159,"5min":1.2509999999999999,"15min":11.273},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.91800000000000004,"15min":0.98299999999999998},"min":{"1min":0.63600000000000001,"5min":0.53900000000000003,"15min":0.44},"max":{"1min":1.3120000000000001,"5min":1.4590000000000001,"15min":7.1289999999999996},"last":1.036}]},{"osd":1,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.78100000000000003,"15min":0.89300000000000002},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0229999999999999,"5min":1.2230000000000001,"15min":6.9199999999999999},"last":0.88},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.83899999999999997,"15min":0.89200000000000002},"min":{"1min":0.52200000000000002,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.1160000000000001,"5min":1.246,"15min":4.75},"last":0.90400000000000003}]},{"osd":2,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.86199999999999999,"15min":0.85099999999999998},"min":{"1min":0.56200000000000006,"5min":0.51700000000000002,"15min":0.36899999999999999},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":2.6970000000000001},"last":1.018},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.91100000000000003,"15min":1.0620000000000001},"min":{"1min":0.66900000000000004,"5min":0.54400000000000004,"15min":0.52900000000000003},"max":{"1min":1.1299999999999999,"5min":1.361,"15min":18.917999999999999},"last":1.077}]},{"osd":3,"last update":"Mon Oct 30 12:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":0.96599999999999997,"15min":0.997},"min":{"1min":0.71399999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.258,"5min":5.1719999999999997,"15min":7.1879999999999997},"last":1.236},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.90800000000000003,"15min":1},"min":{"1min":0.57399999999999995,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.125,"5min":1.3600000000000001,"15min":8.3360000000000003},"last":0.79300000000000004}]},{"osd":5,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.86899999999999999,"15min":0.996},"min":{"1min":0.57599999999999996,"5min":0.56299999999999994,"15min":0.54700000000000004},"max":{"1min":1.026,"5min":1.2509999999999999,"15min":6.0759999999999996},"last":0.68500000000000005},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.113,"15min":1.1859999999999999},"min":{"1min":0.95399999999999996,"5min":0.77400000000000002,"15min":0.52600000000000002},"max":{"1min":1.3660000000000001,"5min":1.4810000000000001,"15min":18.994},"last":1.1220000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:51:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.0880000000000001,"15min":1.093},"min":{"1min":0.70199999999999996,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.429,"5min":1.429,"15min":6.1580000000000004},"last":1.181},{"interface":"front","average":{"1min":1.016,"5min":1.0169999999999999,"15min":1.1240000000000001},"min":{"1min":0.629,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.28,"5min":5.218,"15min":7.0549999999999997},"last":0.99299999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.056,"15min":1.2529999999999999},"min":{"1min":0.83799999999999997,"5min":0.56399999999999995,"15min":0.50700000000000001},"max":{"1min":1.2749999999999999,"5min":1.4850000000000001,"15min":18.015999999999998},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.1579999999999999,"15min":1.2769999999999999},"min":{"1min":0.92100000000000004,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.4339999999999999,"5min":1.4570000000000001,"15min":19.052},"last":1.2809999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542432,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391212,"kb_used_data":310052,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93980628,"statfs":{"total":96636764160,"available":96236163072,"internally_reserved":0,"allocated":317493248,"data_stored":316374505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":83098056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.39900000000000002,"15min":0.41399999999999998},"min":{"1min":0.29399999999999998,"5min":0.23699999999999999,"15min":0.23400000000000001},"max":{"1min":0.58999999999999997,"5min":0.77900000000000003,"15min":1.6899999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51600000000000001,"15min":0.53300000000000003},"min":{"1min":0.40000000000000002,"5min":0.34300000000000003,"15min":0.32700000000000001},"max":{"1min":0.746,"5min":0.871,"15min":1.728},"last":0.49099999999999999}]},{"osd":1,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.42899999999999999,"15min":0.42999999999999999},"min":{"1min":0.309,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":1.2909999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.497,"5min":0.496,"15min":0.50600000000000001},"min":{"1min":0.34499999999999997,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.64600000000000002,"5min":0.80600000000000005,"15min":1.3260000000000001},"last":0.54000000000000004}]},{"osd":2,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.496,"15min":0.502},"min":{"1min":0.37,"5min":0.30099999999999999,"15min":0.29299999999999998},"max":{"1min":0.69899999999999995,"5min":0.78600000000000003,"15min":1.099},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55900000000000005,"15min":0.56699999999999995},"min":{"1min":0.41999999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73599999999999999,"5min":0.82599999999999996,"15min":1.2509999999999999},"last":0.55500000000000005}]},{"osd":4,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.85199999999999998,"15min":0.83099999999999996},"min":{"1min":0.63900000000000001,"5min":0.55400000000000005,"15min":0.40400000000000003},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":2.202},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.82999999999999996,"15min":0.86499999999999999},"min":{"1min":0.70599999999999996,"5min":0.48499999999999999,"15min":0.46800000000000003},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":2.4340000000000002},"last":0.92300000000000004}]},{"osd":5,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.81499999999999995,"15min":0.86299999999999999},"min":{"1min":0.66200000000000003,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.169,"5min":1.169,"15min":14.103999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.875,"5min":0.876,"15min":0.86499999999999999},"min":{"1min":0.63900000000000001,"5min":0.55000000000000004,"15min":0.499},"max":{"1min":1.2170000000000001,"5min":1.2869999999999999,"15min":4.4859999999999998},"last":0.88500000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.85299999999999998,"15min":0.83499999999999996},"min":{"1min":0.64700000000000002,"5min":0.56999999999999995,"15min":0.53100000000000003},"max":{"1min":1.087,"5min":1.2450000000000001,"15min":2.2330000000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.87,"15min":0.82899999999999996},"min":{"1min":0.77800000000000002,"5min":0.54100000000000004,"15min":0.45100000000000001},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":4.4409999999999998},"last":0.94499999999999995}]},{"osd":7,"last update":"Mon Oct 30 12:51:25 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.875,"15min":0.86499999999999999},"min":{"1min":0.67500000000000004,"5min":0.57299999999999995,"15min":0.57099999999999995},"max":{"1min":1.1499999999999999,"5min":1.2509999999999999,"15min":3.5459999999999998},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.86399999999999999,"15min":0.84699999999999998},"min":{"1min":0.65700000000000003,"5min":0.57399999999999995,"15min":0.45700000000000002},"max":{"1min":1.1259999999999999,"5min":1.2230000000000001,"15min":6.1360000000000001},"last":0.97399999999999998}]}]},{"osd":2,"up_from":14,"seq":60129542432,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":200604,"kb_used_data":143060,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94171236,"statfs":{"total":96636764160,"available":96431345664,"internally_reserved":0,"allocated":146493440,"data_stored":145675964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.45100000000000001,"15min":0.441},"min":{"1min":0.25,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.59499999999999997,"5min":0.69399999999999995,"15min":0.81200000000000006},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.5,"15min":0.51100000000000001},"min":{"1min":0.37,"5min":0.33300000000000002,"15min":0.31900000000000001},"max":{"1min":0.63,"5min":0.71299999999999997,"15min":1.298},"last":0.435}]},{"osd":1,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.47899999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.60199999999999998,"5min":0.65500000000000003,"15min":0.83699999999999997},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.52500000000000002,"15min":0.504},"min":{"1min":0.38900000000000001,"5min":0.33700000000000002,"15min":0.29999999999999999},"max":{"1min":0.61699999999999999,"5min":0.74099999999999999,"15min":1.2250000000000001},"last":0.497}]},{"osd":3,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.47399999999999998,"15min":0.54800000000000004},"min":{"1min":0.315,"5min":0.311,"15min":0.311},"max":{"1min":0.58599999999999997,"5min":0.75,"15min":1.673},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57299999999999995,"15min":0.57399999999999995},"min":{"1min":0.41799999999999998,"5min":0.40200000000000002,"15min":0.35699999999999998},"max":{"1min":0.67900000000000005,"5min":0.79100000000000004,"15min":1.597},"last":0.56399999999999995}]},{"osd":4,"last update":"Mon Oct 30 12:51:21 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.74299999999999999,"15min":0.82599999999999996},"min":{"1min":0.47499999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.91100000000000003,"5min":1.0569999999999999,"15min":1.377},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.75,"5min":0.755,"15min":0.79200000000000004},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.998,"5min":1.0640000000000001,"15min":1.4159999999999999},"last":0.91000000000000003}]},{"osd":5,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.84099999999999997,"15min":0.84799999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.093,"5min":1.2430000000000001,"15min":1.78},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91400000000000003},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.1060000000000001,"5min":1.2729999999999999,"15min":1.7430000000000001},"last":0.75}]},{"osd":6,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.86199999999999999,"15min":0.85799999999999998},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.60299999999999998},"max":{"1min":1.095,"5min":1.177,"15min":1.3799999999999999},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.86599999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.53300000000000003},"max":{"1min":1.097,"5min":1.1799999999999999,"15min":1.8109999999999999},"last":0.876}]},{"osd":7,"last update":"Mon Oct 30 12:51:16 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.82899999999999996,"15min":0.83099999999999996},"min":{"1min":0.58799999999999997,"5min":0.58299999999999996,"15min":0.48799999999999999},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.4159999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.85399999999999998,"15min":0.83299999999999996},"min":{"1min":0.75900000000000001,"5min":0.51800000000000002,"15min":0.50800000000000001},"max":{"1min":1.206,"5min":1.206,"15min":1.3049999999999999},"last":0.76200000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542432,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":279036,"kb_used_data":233972,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94092804,"statfs":{"total":96636764160,"available":96351031296,"internally_reserved":0,"allocated":239587328,"data_stored":238747727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:38 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.41399999999999998,"15min":0.45700000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.24299999999999999},"max":{"1min":0.65100000000000002,"5min":0.65200000000000002,"15min":11.958},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.497,"15min":0.55500000000000005},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.751,"5min":0.751,"15min":12.077},"last":0.56299999999999994}]},{"osd":2,"last update":"Mon Oct 30 12:51:38 2023","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.46200000000000002,"15min":0.45800000000000002},"min":{"1min":0.32700000000000001,"5min":0.29699999999999999,"15min":0.26400000000000001},"max":{"1min":0.54100000000000004,"5min":1.121,"15min":1.623},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.53200000000000003,"15min":0.53100000000000003},"min":{"1min":0.37,"5min":0.34200000000000003,"15min":0.31900000000000001},"max":{"1min":0.84499999999999997,"5min":1.1699999999999999,"15min":1.5389999999999999},"last":0.48899999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:51:38 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56499999999999995,"15min":0.56599999999999995},"min":{"1min":0.40999999999999998,"5min":0.378,"15min":0.31},"max":{"1min":0.72099999999999997,"5min":1.337,"15min":3.694},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.504,"15min":0.53700000000000003},"min":{"1min":0.308,"5min":0.307,"15min":0.307},"max":{"1min":0.69699999999999995,"5min":0.77200000000000002,"15min":13.199999999999999},"last":0.51300000000000001}]},{"osd":4,"last update":"Mon Oct 30 12:51:38 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.82299999999999995,"15min":0.82699999999999996},"min":{"1min":0.54700000000000004,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.94599999999999995,"5min":1.3140000000000001,"15min":2.419},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.873,"15min":0.86499999999999999},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.52100000000000002},"max":{"1min":1.012,"5min":1.103,"15min":2.282},"last":0.95499999999999996}]},{"osd":5,"last update":"Mon Oct 30 12:51:38 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.79800000000000004,"15min":0.81399999999999995},"min":{"1min":0.54100000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.95799999999999996,"5min":1.343,"15min":1.343},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.751,"5min":0.79000000000000004,"15min":0.84799999999999998},"min":{"1min":0.51700000000000002,"5min":0.50800000000000001,"15min":0.44900000000000001},"max":{"1min":0.98999999999999999,"5min":1.2809999999999999,"15min":1.29},"last":0.88200000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:51:38 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.83399999999999996,"15min":0.80000000000000004},"min":{"1min":0.56200000000000006,"5min":0.52800000000000002,"15min":0.51200000000000001},"max":{"1min":1.1279999999999999,"5min":1.371,"15min":1.371},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.91200000000000003,"15min":0.86699999999999999},"min":{"1min":0.628,"5min":0.62,"15min":0.498},"max":{"1min":1.173,"5min":1.417,"15min":2.464},"last":0.91800000000000004}]},{"osd":7,"last update":"Mon Oct 30 12:51:38 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.80300000000000005,"15min":0.82399999999999995},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.075,"5min":1.1659999999999999,"15min":2.5169999999999999},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.88500000000000001,"15min":0.90100000000000002},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.1140000000000001,"5min":1.133,"15min":2.5510000000000002},"last":0.98399999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542432,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":349720,"kb_used_data":278288,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94022120,"statfs":{"total":96636764160,"available":96278650880,"internally_reserved":0,"allocated":284966912,"data_stored":283825495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 12:51:29 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.45700000000000002,"15min":0.45200000000000001},"min":{"1min":0.28699999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.64600000000000002,"5min":0.71499999999999997,"15min":0.93700000000000006},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51500000000000001,"15min":0.50700000000000001},"min":{"1min":0.38700000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.73799999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.437}]},{"osd":2,"last update":"Mon Oct 30 12:51:29 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45600000000000002,"15min":0.502},"min":{"1min":0.315,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.97199999999999998},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56000000000000005,"15min":0.53300000000000003},"min":{"1min":0.437,"5min":0.36799999999999999,"15min":0.27400000000000002},"max":{"1min":0.79000000000000004,"5min":0.83299999999999996,"15min":1.0149999999999999},"last":0.50800000000000001}]},{"osd":3,"last update":"Mon Oct 30 12:51:29 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47899999999999998,"15min":0.51300000000000001},"min":{"1min":0.36099999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.67100000000000004,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.5,"15min":0.497},"min":{"1min":0.41099999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.71199999999999997,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.47599999999999998}]},{"osd":4,"last update":"Mon Oct 30 12:51:29 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.87,"15min":0.85299999999999998},"min":{"1min":0.57699999999999996,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1899999999999999,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.877,"15min":0.85899999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.48299999999999998},"max":{"1min":1.0189999999999999,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.872}]},{"osd":5,"last update":"Mon Oct 30 12:51:29 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.81399999999999995,"15min":0.83299999999999996},"min":{"1min":0.55200000000000005,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.163,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83699999999999997,"15min":0.86799999999999999},"min":{"1min":0.68999999999999995,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0289999999999999,"5min":1.889,"15min":1.889},"last":0.84299999999999997}]},{"osd":6,"last update":"Mon Oct 30 12:51:29 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80300000000000005,"15min":0.79500000000000004},"min":{"1min":0.63100000000000001,"5min":0.56299999999999994,"15min":0.52200000000000002},"max":{"1min":1.071,"5min":1.1990000000000001,"15min":2.6480000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.80100000000000005,"15min":0.81999999999999995},"min":{"1min":0.60999999999999999,"5min":0.48799999999999999,"15min":0.46999999999999997},"max":{"1min":1.071,"5min":1.1559999999999999,"15min":5.0099999999999998},"last":0.82199999999999995}]},{"osd":7,"last update":"Mon Oct 30 12:51:29 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.872,"15min":0.88300000000000001},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.55000000000000004},"max":{"1min":1.25,"5min":1.401,"15min":3.3359999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.93899999999999995,"15min":0.94799999999999995},"min":{"1min":0.73899999999999999,"5min":0.65100000000000002,"15min":0.56299999999999994},"max":{"1min":1.2729999999999999,"5min":1.3500000000000001,"15min":3.3580000000000001},"last":0.93100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62930944,"data_stored":62915220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":83910656,"data_stored":83887024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":37969920,"data_stored":37945198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":83726336,"data_stored":83700311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":76107776,"data_stored":76093397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":62935040,"data_stored":62915938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":91594752,"data_stored":91574374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":112349184,"data_stored":112332270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":220680192,"data_stored":219932830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":154910720,"data_stored":154473007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":107761664,"data_stored":107343689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":233000960,"data_stored":232286498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":224047104,"data_stored":223359467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":213921792,"data_stored":213183588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":148467712,"data_stored":148006929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":216776704,"data_stored":216010186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:52:18.717 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:52:18.717 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:52:18.718 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:52:18.718 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:52:18.719 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:52:18.719 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:52:18.719 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:52:18.720 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:52:18.721 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:52:18.721 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:52:18.722 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T12:52:18.891 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:52:18.895 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:52:38.723 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:52:38.873 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.872+0000 7faa7f009640 1 Processor -- start 2023-10-30T12:52:38.874 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.872+0000 7faa7f009640 1 -- start start 2023-10-30T12:52:38.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.872+0000 7faa7f009640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7faa781539a0 0x7faa7815a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:38.875 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.872+0000 7faa7f009640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa7815a8c0 con 0x7faa78069e90 2023-10-30T12:52:38.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.872+0000 7faa7f009640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa78153d80 con 0x7faa78070a00 2023-10-30T12:52:38.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.872+0000 7faa7f009640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa7806fbb0 con 0x7faa781539a0 2023-10-30T12:52:38.876 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.872+0000 7faa7d5a6640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7faa78069e90 0x7faa7806a300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:45148/0 (socket says 172.21.15.89:45148) 2023-10-30T12:52:38.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.872+0000 7faa7d5a6640 1 -- 172.21.15.89:0/2558274293 learned_addr learned my addr 172.21.15.89:0/2558274293 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:52:38.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.872+0000 7faa7cda5640 1 --2- 172.21.15.89:0/2558274293 >> v2:172.21.15.192:3300/0 conn(0x7faa781539a0 0x7faa7815a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:38.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.872+0000 7faa6f7fe640 1 -- 172.21.15.89:0/2558274293 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 298386557 0 0) 0x7faa7815a8c0 con 0x7faa78069e90 2023-10-30T12:52:38.877 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.872+0000 7faa6f7fe640 1 -- 172.21.15.89:0/2558274293 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa60003530 con 0x7faa78069e90 2023-10-30T12:52:38.879 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.876+0000 7faa6f7fe640 1 -- 172.21.15.89:0/2558274293 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2257467946 0 0) 0x7faa60003530 con 0x7faa78069e90 2023-10-30T12:52:38.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.876+0000 7faa6f7fe640 1 -- 172.21.15.89:0/2558274293 >> v2:172.21.15.192:3300/0 conn(0x7faa781539a0 msgr2=0x7faa7815a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:38.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.876+0000 7faa6f7fe640 1 --2- 172.21.15.89:0/2558274293 >> v2:172.21.15.192:3300/0 conn(0x7faa781539a0 0x7faa7815a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:38.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.876+0000 7faa6f7fe640 1 -- 172.21.15.89:0/2558274293 >> v1:172.21.15.192:6789/0 conn(0x7faa78070a00 legacy=0x7faa78070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:38.880 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.876+0000 7faa6f7fe640 1 -- 172.21.15.89:0/2558274293 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa780c5c50 con 0x7faa78069e90 2023-10-30T12:52:38.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.876+0000 7faa6f7fe640 1 -- 172.21.15.89:0/2558274293 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7faa70001a00 con 0x7faa78069e90 2023-10-30T12:52:38.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.876+0000 7faa6f7fe640 1 -- 172.21.15.89:0/2558274293 >> v1:172.21.15.89:6789/0 conn(0x7faa78069e90 legacy=0x7faa7806a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:38.881 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.876+0000 7faa6f7fe640 1 --2- 172.21.15.89:0/2558274293 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa60003980 0x7faa60003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:38.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.876+0000 7faa6f7fe640 1 --2- 172.21.15.89:0/2558274293 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa600047d0 0x7faa60004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:38.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.876+0000 7faa6f7fe640 1 --2- 172.21.15.89:0/2558274293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa60005190 0x7faa6000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:38.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.876+0000 7faa6f7fe640 1 -- 172.21.15.89:0/2558274293 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7806fbb0 con 0x7faa600047d0 2023-10-30T12:52:38.882 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.876+0000 7faa6f7fe640 1 -- 172.21.15.89:0/2558274293 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6000c130 con 0x7faa60005190 2023-10-30T12:52:38.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.876+0000 7faa6f7fe640 1 -- 172.21.15.89:0/2558274293 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa6000c2f0 con 0x7faa60003980 2023-10-30T12:52:38.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.876+0000 7faa6ffff640 1 --2- 172.21.15.89:0/2558274293 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa600047d0 0x7faa60004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:38.883 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.876+0000 7faa7cda5640 1 --2- 172.21.15.89:0/2558274293 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa60003980 0x7faa60003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:38.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.876+0000 7faa7d5a6640 1 --2- 172.21.15.89:0/2558274293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa60005190 0x7faa6000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:38.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.876+0000 7faa6ffff640 1 -- 172.21.15.89:0/2558274293 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa60003980 msgr2=0x7faa60003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:38.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.876+0000 7faa6ffff640 1 --2- 172.21.15.89:0/2558274293 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa60003980 0x7faa60003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:38.884 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.876+0000 7faa6ffff640 1 -- 172.21.15.89:0/2558274293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa60005190 msgr2=0x7faa6000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:38.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.876+0000 7faa6ffff640 1 --2- 172.21.15.89:0/2558274293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa60005190 0x7faa6000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:38.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.876+0000 7faa6ffff640 1 -- 172.21.15.89:0/2558274293 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa60003530 con 0x7faa600047d0 2023-10-30T12:52:38.885 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.876+0000 7faa7d5a6640 1 --2- 172.21.15.89:0/2558274293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa60005190 0x7faa6000b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:52:38.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.876+0000 7faa7cda5640 1 --2- 172.21.15.89:0/2558274293 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa60003980 0x7faa60003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:52:38.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.876+0000 7faa7d5a6640 1 -- 172.21.15.89:0/2558274293 reap_dead start 2023-10-30T12:52:38.886 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.880+0000 7faa6ffff640 1 --2- 172.21.15.89:0/2558274293 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa600047d0 0x7faa60004c50 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7faa68001780 tx=0x7faa6801c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e98a7c56dfac37d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:38.889 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.880+0000 7faa6f7fe640 1 -- 172.21.15.89:0/2558274293 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa680052d0 con 0x7faa600047d0 2023-10-30T12:52:38.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.880+0000 7faa6f7fe640 1 -- 172.21.15.89:0/2558274293 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa680a7050 con 0x7faa600047d0 2023-10-30T12:52:38.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.880+0000 7faa6f7fe640 1 -- 172.21.15.89:0/2558274293 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa68003600 con 0x7faa600047d0 2023-10-30T12:52:38.890 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.880+0000 7faa7f009640 1 -- 172.21.15.89:0/2558274293 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa600047d0 msgr2=0x7faa60004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:38.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.880+0000 7faa7f009640 1 --2- 172.21.15.89:0/2558274293 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa600047d0 0x7faa60004c50 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7faa68001780 tx=0x7faa6801c3b0 comp rx=0 tx=0).stop 2023-10-30T12:52:38.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.880+0000 7faa7f009640 1 -- 172.21.15.89:0/2558274293 shutdown_connections 2023-10-30T12:52:38.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.880+0000 7faa7f009640 1 --2- 172.21.15.89:0/2558274293 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa600047d0 0x7faa60004c50 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:38.891 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.880+0000 7faa7f009640 1 -- 172.21.15.89:0/2558274293 >> 172.21.15.89:0/2558274293 conn(0x7faa780c7910 msgr2=0x7faa780c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:38.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.880+0000 7faa7f009640 1 -- 172.21.15.89:0/2558274293 shutdown_connections 2023-10-30T12:52:38.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.880+0000 7faa7f009640 1 -- 172.21.15.89:0/2558274293 wait complete. 2023-10-30T12:52:38.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.880+0000 7faa7f009640 1 Processor -- start 2023-10-30T12:52:38.892 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.880+0000 7faa7f009640 1 -- start start 2023-10-30T12:52:38.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.880+0000 7faa7f009640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa600047d0 0x7faa780c4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:38.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.880+0000 7faa7f009640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa7814b710 0x7faa7814bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:38.893 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.880+0000 7faa7f009640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa7814a6d0 0x7faa7814ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:38.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.884+0000 7faa7f009640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa78147370 con 0x7faa600047d0 2023-10-30T12:52:38.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.884+0000 7faa7f009640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa780c52d0 con 0x7faa7814a6d0 2023-10-30T12:52:38.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.884+0000 7faa7f009640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa78157530 con 0x7faa7814b710 2023-10-30T12:52:38.894 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.884+0000 7faa7cda5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa600047d0 0x7faa780c4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:38.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.884+0000 7faa7cda5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa600047d0 0x7faa780c4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60174/0 (socket says 172.21.15.89:60174) 2023-10-30T12:52:38.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.884+0000 7faa7cda5640 1 -- 172.21.15.89:0/340848337 learned_addr learned my addr 172.21.15.89:0/340848337 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:52:38.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.884+0000 7faa6ffff640 1 --2- 172.21.15.89:0/340848337 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa7814b710 0x7faa7814bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:38.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.884+0000 7faa7d5a6640 1 --2- 172.21.15.89:0/340848337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa7814a6d0 0x7faa7814ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:38.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.884+0000 7faa7cda5640 1 -- 172.21.15.89:0/340848337 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa7814b710 msgr2=0x7faa7814bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:38.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.884+0000 7faa7cda5640 1 --2- 172.21.15.89:0/340848337 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa7814b710 0x7faa7814bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:38.896 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.884+0000 7faa7cda5640 1 -- 172.21.15.89:0/340848337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa7814a6d0 msgr2=0x7faa7814ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:38.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.884+0000 7faa7cda5640 1 --2- 172.21.15.89:0/340848337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa7814a6d0 0x7faa7814ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:38.897 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.884+0000 7faa7cda5640 1 -- 172.21.15.89:0/340848337 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7814c090 con 0x7faa600047d0 2023-10-30T12:52:38.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.884+0000 7faa7cda5640 1 --2- 172.21.15.89:0/340848337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa600047d0 0x7faa780c4d90 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7faa74011510 tx=0x7faa740119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:38.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.884+0000 7faa6d7fa640 1 -- 172.21.15.89:0/340848337 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa74004220 con 0x7faa600047d0 2023-10-30T12:52:38.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.884+0000 7faa6d7fa640 1 -- 172.21.15.89:0/340848337 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa74004380 con 0x7faa600047d0 2023-10-30T12:52:38.898 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.884+0000 7faa6d7fa640 1 -- 172.21.15.89:0/340848337 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa74015890 con 0x7faa600047d0 2023-10-30T12:52:38.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.884+0000 7faa7f009640 1 -- 172.21.15.89:0/340848337 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa78159b50 con 0x7faa600047d0 2023-10-30T12:52:38.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.884+0000 7faa7f009640 1 -- 172.21.15.89:0/340848337 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa781f2d70 con 0x7faa600047d0 2023-10-30T12:52:38.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.884+0000 7faa7f009640 1 -- 172.21.15.89:0/340848337 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa781f2f30 con 0x7faa600047d0 2023-10-30T12:52:38.899 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.884+0000 7faa6d7fa640 1 -- 172.21.15.89:0/340848337 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faa740159f0 con 0x7faa600047d0 2023-10-30T12:52:38.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.888+0000 7faa6d7fa640 1 --2- 172.21.15.89:0/340848337 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faa640664a0 0x7faa64068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:38.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.888+0000 7faa6d7fa640 1 -- 172.21.15.89:0/340848337 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7faa740848d0 con 0x7faa600047d0 2023-10-30T12:52:38.900 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.888+0000 7faa6ffff640 1 --2- 172.21.15.89:0/340848337 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faa640664a0 0x7faa64068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:38.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.888+0000 7faa6ffff640 1 --2- 172.21.15.89:0/340848337 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faa640664a0 0x7faa64068960 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7faa68001750 tx=0x7faa680011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:38.901 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:38.892+0000 7faa6d7fa640 1 -- 172.21.15.89:0/340848337 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7faa74051810 con 0x7faa600047d0 2023-10-30T12:52:39.038 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:39.036+0000 7faa7f009640 1 -- 172.21.15.89:0/340848337 --> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faa780c67d0 con 0x7faa640664a0 2023-10-30T12:52:39.060 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:39.060+0000 7faa6d7fa640 1 -- 172.21.15.89:0/340848337 <== mgr.4105 v2:172.21.15.192:6832/4076276638 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+294653 (secure 0 0 0) 0x7faa68005390 con 0x7faa640664a0 2023-10-30T12:52:39.063 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:52:39.064 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-10-30T12:52:39.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:39.064+0000 7faa7f009640 1 -- 172.21.15.89:0/340848337 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faa640664a0 msgr2=0x7faa64068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:39.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:39.064+0000 7faa7f009640 1 --2- 172.21.15.89:0/340848337 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faa640664a0 0x7faa64068960 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7faa68001750 tx=0x7faa680011a0 comp rx=0 tx=0).stop 2023-10-30T12:52:39.066 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:39.064+0000 7faa7f009640 1 -- 172.21.15.89:0/340848337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa600047d0 msgr2=0x7faa780c4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:39.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:39.064+0000 7faa7f009640 1 --2- 172.21.15.89:0/340848337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa600047d0 0x7faa780c4d90 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7faa74011510 tx=0x7faa740119d0 comp rx=0 tx=0).stop 2023-10-30T12:52:39.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:39.064+0000 7faa7f009640 1 -- 172.21.15.89:0/340848337 shutdown_connections 2023-10-30T12:52:39.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:39.064+0000 7faa7f009640 1 --2- 172.21.15.89:0/340848337 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7faa640664a0 0x7faa64068960 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:39.067 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:39.064+0000 7faa7f009640 1 --2- 172.21.15.89:0/340848337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa7814a6d0 0x7faa7814ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:39.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:39.064+0000 7faa7f009640 1 --2- 172.21.15.89:0/340848337 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa7814b710 0x7faa7814bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:39.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:39.064+0000 7faa7f009640 1 --2- 172.21.15.89:0/340848337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa600047d0 0x7faa780c4d90 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:39.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:39.064+0000 7faa7f009640 1 -- 172.21.15.89:0/340848337 >> 172.21.15.89:0/340848337 conn(0x7faa780c7910 msgr2=0x7faa7814e340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:39.068 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:39.064+0000 7faa7f009640 1 -- 172.21.15.89:0/340848337 shutdown_connections 2023-10-30T12:52:39.069 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:39.064+0000 7faa7f009640 1 -- 172.21.15.89:0/340848337 wait complete. 2023-10-30T12:52:39.083 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":779,"stamp":"2023-10-30T12:52:37.453738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1063659013,"num_objects":1371,"num_object_clones":0,"num_object_copies":2742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1371,"num_whiteouts":0,"num_read":4368,"num_read_kb":813164,"num_write":42625,"num_write_kb":1359490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":78,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":305728,"num_omap_keys":583,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48295,"ondisk_log_size":48295,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2629316,"kb_used_data":2088452,"kb_used_omap":12,"kb_used_meta":540787,"kb_avail":752345404,"statfs":{"total":773094113280,"available":770401693696,"internally_reserved":0,"allocated":2138574848,"data_stored":2130417737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12741,"internal_metadata":553766459},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004456"},"pg_stats":[{"pgid":"3.3f","version":"66'572","reported_seq":2734,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:15.560856+0000","last_change":"2023-10-30T12:52:15.561063+0000","last_active":"2023-10-30T12:52:15.560856+0000","last_peered":"2023-10-30T12:52:15.560856+0000","last_clean":"2023-10-30T12:52:15.560856+0000","last_became_active":"2023-10-30T12:28:50.979351+0000","last_became_peered":"2023-10-30T12:28:50.979351+0000","last_unstale":"2023-10-30T12:52:15.560856+0000","last_undegraded":"2023-10-30T12:52:15.560856+0000","last_fullsized":"2023-10-30T12:52:15.560856+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'572","last_scrub_stamp":"2023-10-30T12:52:15.560583+0000","last_deep_scrub":"66'572","last_deep_scrub_stamp":"2023-10-30T12:52:15.560583+0000","last_clean_scrub_stamp":"2023-10-30T12:52:15.560583+0000","objects_scrubbed":21,"log_size":572,"log_dups_size":0,"ondisk_log_size":572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:47:12.132657+0000","scrub_duration":0.060883743999999997,"objects_trimmed":0,"snaptrim_duration":0.0091708599999999994,"stat_sum":{"num_bytes":4321961,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":40,"num_read_kb":15,"num_write":493,"num_write_kb":5417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"66'570","reported_seq":2725,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:02.815587+0000","last_change":"2023-10-30T12:52:02.815654+0000","last_active":"2023-10-30T12:52:02.815587+0000","last_peered":"2023-10-30T12:52:02.815587+0000","last_clean":"2023-10-30T12:52:02.815587+0000","last_became_active":"2023-10-30T12:28:50.929133+0000","last_became_peered":"2023-10-30T12:28:50.929133+0000","last_unstale":"2023-10-30T12:52:02.815587+0000","last_undegraded":"2023-10-30T12:52:02.815587+0000","last_fullsized":"2023-10-30T12:52:02.815587+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'570","last_scrub_stamp":"2023-10-30T12:52:02.815504+0000","last_deep_scrub":"66'570","last_deep_scrub_stamp":"2023-10-30T12:52:02.815504+0000","last_clean_scrub_stamp":"2023-10-30T12:52:02.815504+0000","objects_scrubbed":19,"log_size":570,"log_dups_size":0,"ondisk_log_size":570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:57:55.912117+0000","scrub_duration":0.032094442000000001,"objects_trimmed":0,"snaptrim_duration":0.0023149070000000002,"stat_sum":{"num_bytes":8426607,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":33,"num_read_kb":10,"num_write":489,"num_write_kb":9508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"66'568","reported_seq":2739,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:12.337570+0000","last_change":"2023-10-30T12:52:12.337656+0000","last_active":"2023-10-30T12:52:12.337570+0000","last_peered":"2023-10-30T12:52:12.337570+0000","last_clean":"2023-10-30T12:52:12.337570+0000","last_became_active":"2023-10-30T12:28:50.933896+0000","last_became_peered":"2023-10-30T12:28:50.933896+0000","last_unstale":"2023-10-30T12:52:12.337570+0000","last_undegraded":"2023-10-30T12:52:12.337570+0000","last_fullsized":"2023-10-30T12:52:12.337570+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'568","last_scrub_stamp":"2023-10-30T12:52:12.337465+0000","last_deep_scrub":"66'568","last_deep_scrub_stamp":"2023-10-30T12:52:12.337465+0000","last_clean_scrub_stamp":"2023-10-30T12:52:12.337465+0000","objects_scrubbed":18,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:24:10.063858+0000","scrub_duration":0.040558903,"objects_trimmed":0,"snaptrim_duration":0.0029033520000000001,"stat_sum":{"num_bytes":12697764,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":49,"num_read_kb":8209,"num_write":487,"num_write_kb":25783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"66'530","reported_seq":2682,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:32.236357+0000","last_change":"2023-10-30T12:52:32.236477+0000","last_active":"2023-10-30T12:52:32.236357+0000","last_peered":"2023-10-30T12:52:32.236357+0000","last_clean":"2023-10-30T12:52:32.236357+0000","last_became_active":"2023-10-30T12:28:50.967774+0000","last_became_peered":"2023-10-30T12:28:50.967774+0000","last_unstale":"2023-10-30T12:52:32.236357+0000","last_undegraded":"2023-10-30T12:52:32.236357+0000","last_fullsized":"2023-10-30T12:52:32.236357+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'530","last_scrub_stamp":"2023-10-30T12:52:32.236159+0000","last_deep_scrub":"66'530","last_deep_scrub_stamp":"2023-10-30T12:52:32.236159+0000","last_clean_scrub_stamp":"2023-10-30T12:52:32.236159+0000","objects_scrubbed":13,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:31:41.392978+0000","scrub_duration":0.034302944000000002,"objects_trimmed":0,"snaptrim_duration":0.0075479680000000004,"stat_sum":{"num_bytes":4272818,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":30,"num_read_kb":13,"num_write":479,"num_write_kb":5884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"66'553","reported_seq":2720,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:39.811881+0000","last_change":"2023-10-30T12:51:39.812135+0000","last_active":"2023-10-30T12:51:39.811881+0000","last_peered":"2023-10-30T12:51:39.811881+0000","last_clean":"2023-10-30T12:51:39.811881+0000","last_became_active":"2023-10-30T12:28:50.973132+0000","last_became_peered":"2023-10-30T12:28:50.973132+0000","last_unstale":"2023-10-30T12:51:39.811881+0000","last_undegraded":"2023-10-30T12:51:39.811881+0000","last_fullsized":"2023-10-30T12:51:39.811881+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'553","last_scrub_stamp":"2023-10-30T12:51:39.811639+0000","last_deep_scrub":"66'553","last_deep_scrub_stamp":"2023-10-30T12:51:39.811639+0000","last_clean_scrub_stamp":"2023-10-30T12:51:39.811639+0000","objects_scrubbed":18,"log_size":553,"log_dups_size":0,"ondisk_log_size":553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:25:28.276273+0000","scrub_duration":0.055821471999999997,"objects_trimmed":0,"snaptrim_duration":0.0090421609999999999,"stat_sum":{"num_bytes":12692456,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":47,"num_read_kb":16,"num_write":494,"num_write_kb":14525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"66'566","reported_seq":2724,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:35.199332+0000","last_change":"2023-10-30T12:52:35.199562+0000","last_active":"2023-10-30T12:52:35.199332+0000","last_peered":"2023-10-30T12:52:35.199332+0000","last_clean":"2023-10-30T12:52:35.199332+0000","last_became_active":"2023-10-30T12:28:50.967910+0000","last_became_peered":"2023-10-30T12:28:50.967910+0000","last_unstale":"2023-10-30T12:52:35.199332+0000","last_undegraded":"2023-10-30T12:52:35.199332+0000","last_fullsized":"2023-10-30T12:52:35.199332+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'566","last_scrub_stamp":"2023-10-30T12:52:35.199061+0000","last_deep_scrub":"66'566","last_deep_scrub_stamp":"2023-10-30T12:52:35.199061+0000","last_clean_scrub_stamp":"2023-10-30T12:52:35.199061+0000","objects_scrubbed":15,"log_size":566,"log_dups_size":0,"ondisk_log_size":566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:50:48.082035+0000","scrub_duration":0.053241647000000003,"objects_trimmed":0,"snaptrim_duration":0.0040413009999999997,"stat_sum":{"num_bytes":12623095,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":38,"num_read_kb":16,"num_write":503,"num_write_kb":14521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"66'569","reported_seq":2730,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:55.514716+0000","last_change":"2023-10-30T12:51:55.514849+0000","last_active":"2023-10-30T12:51:55.514716+0000","last_peered":"2023-10-30T12:51:55.514716+0000","last_clean":"2023-10-30T12:51:55.514716+0000","last_became_active":"2023-10-30T12:28:50.973288+0000","last_became_peered":"2023-10-30T12:28:50.973288+0000","last_unstale":"2023-10-30T12:51:55.514716+0000","last_undegraded":"2023-10-30T12:51:55.514716+0000","last_fullsized":"2023-10-30T12:51:55.514716+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'569","last_scrub_stamp":"2023-10-30T12:51:55.514522+0000","last_deep_scrub":"66'569","last_deep_scrub_stamp":"2023-10-30T12:51:55.514522+0000","last_clean_scrub_stamp":"2023-10-30T12:51:55.514522+0000","objects_scrubbed":18,"log_size":569,"log_dups_size":0,"ondisk_log_size":569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:36.003579+0000","scrub_duration":0.043916595000000003,"objects_trimmed":0,"snaptrim_duration":0.0037299339999999999,"stat_sum":{"num_bytes":4302112,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":41,"num_read_kb":11,"num_write":510,"num_write_kb":7154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"66'518","reported_seq":2677,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:19.850824+0000","last_change":"2023-10-30T12:52:19.850891+0000","last_active":"2023-10-30T12:52:19.850824+0000","last_peered":"2023-10-30T12:52:19.850824+0000","last_clean":"2023-10-30T12:52:19.850824+0000","last_became_active":"2023-10-30T12:28:50.954988+0000","last_became_peered":"2023-10-30T12:28:50.954988+0000","last_unstale":"2023-10-30T12:52:19.850824+0000","last_undegraded":"2023-10-30T12:52:19.850824+0000","last_fullsized":"2023-10-30T12:52:19.850824+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'518","last_scrub_stamp":"2023-10-30T12:52:19.850743+0000","last_deep_scrub":"66'518","last_deep_scrub_stamp":"2023-10-30T12:52:19.850743+0000","last_clean_scrub_stamp":"2023-10-30T12:52:19.850743+0000","objects_scrubbed":18,"log_size":518,"log_dups_size":0,"ondisk_log_size":518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:55:39.941900+0000","scrub_duration":0.048490766999999997,"objects_trimmed":0,"snaptrim_duration":0.0028962670000000001,"stat_sum":{"num_bytes":8424944,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":39,"num_read_kb":15,"num_write":456,"num_write_kb":10248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"66'595","reported_seq":2764,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:04.680217+0000","last_change":"2023-10-30T12:52:04.680454+0000","last_active":"2023-10-30T12:52:04.680217+0000","last_peered":"2023-10-30T12:52:04.680217+0000","last_clean":"2023-10-30T12:52:04.680217+0000","last_became_active":"2023-10-30T12:28:50.976564+0000","last_became_peered":"2023-10-30T12:28:50.976564+0000","last_unstale":"2023-10-30T12:52:04.680217+0000","last_undegraded":"2023-10-30T12:52:04.680217+0000","last_fullsized":"2023-10-30T12:52:04.680217+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'595","last_scrub_stamp":"2023-10-30T12:52:04.679935+0000","last_deep_scrub":"66'595","last_deep_scrub_stamp":"2023-10-30T12:52:04.679935+0000","last_clean_scrub_stamp":"2023-10-30T12:52:04.679935+0000","objects_scrubbed":22,"log_size":595,"log_dups_size":0,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:37:06.466494+0000","scrub_duration":0.091817582999999994,"objects_trimmed":0,"snaptrim_duration":0.0094697280000000002,"stat_sum":{"num_bytes":16840394,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":49,"num_read_kb":4116,"num_write":513,"num_write_kb":27605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"66'566","reported_seq":2747,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:57.616031+0000","last_change":"2023-10-30T12:51:57.616119+0000","last_active":"2023-10-30T12:51:57.616031+0000","last_peered":"2023-10-30T12:51:57.616031+0000","last_clean":"2023-10-30T12:51:57.616031+0000","last_became_active":"2023-10-30T12:28:50.951809+0000","last_became_peered":"2023-10-30T12:28:50.951809+0000","last_unstale":"2023-10-30T12:51:57.616031+0000","last_undegraded":"2023-10-30T12:51:57.616031+0000","last_fullsized":"2023-10-30T12:51:57.616031+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'566","last_scrub_stamp":"2023-10-30T12:51:57.615865+0000","last_deep_scrub":"66'566","last_deep_scrub_stamp":"2023-10-30T12:51:57.615865+0000","last_clean_scrub_stamp":"2023-10-30T12:51:57.615865+0000","objects_scrubbed":27,"log_size":566,"log_dups_size":0,"ondisk_log_size":566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:45:34.480445+0000","scrub_duration":0.097301143000000007,"objects_trimmed":0,"snaptrim_duration":0.0078740100000000007,"stat_sum":{"num_bytes":16924216,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":59,"num_read_kb":29,"num_write":493,"num_write_kb":22764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"66'600","reported_seq":2773,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:10.791677+0000","last_change":"2023-10-30T12:52:10.791921+0000","last_active":"2023-10-30T12:52:10.791677+0000","last_peered":"2023-10-30T12:52:10.791677+0000","last_clean":"2023-10-30T12:52:10.791677+0000","last_became_active":"2023-10-30T12:29:32.958473+0000","last_became_peered":"2023-10-30T12:29:32.958473+0000","last_unstale":"2023-10-30T12:52:10.791677+0000","last_undegraded":"2023-10-30T12:52:10.791677+0000","last_fullsized":"2023-10-30T12:52:10.791677+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'600","last_scrub_stamp":"2023-10-30T12:52:10.791444+0000","last_deep_scrub":"66'600","last_deep_scrub_stamp":"2023-10-30T12:52:10.791444+0000","last_clean_scrub_stamp":"2023-10-30T12:52:10.791444+0000","objects_scrubbed":23,"log_size":600,"log_dups_size":0,"ondisk_log_size":600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:50:18.491558+0000","scrub_duration":0.058907956999999997,"objects_trimmed":0,"snaptrim_duration":0.0045626240000000004,"stat_sum":{"num_bytes":12721502,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":46,"num_read_kb":17,"num_write":533,"num_write_kb":14913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"66'526","reported_seq":2680,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:43.156958+0000","last_change":"2023-10-30T12:51:43.157088+0000","last_active":"2023-10-30T12:51:43.156958+0000","last_peered":"2023-10-30T12:51:43.156958+0000","last_clean":"2023-10-30T12:51:43.156958+0000","last_became_active":"2023-10-30T12:28:50.943787+0000","last_became_peered":"2023-10-30T12:28:50.943787+0000","last_unstale":"2023-10-30T12:51:43.156958+0000","last_undegraded":"2023-10-30T12:51:43.156958+0000","last_fullsized":"2023-10-30T12:51:43.156958+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'526","last_scrub_stamp":"2023-10-30T12:51:43.156744+0000","last_deep_scrub":"66'526","last_deep_scrub_stamp":"2023-10-30T12:51:43.156744+0000","last_clean_scrub_stamp":"2023-10-30T12:51:43.156744+0000","objects_scrubbed":20,"log_size":526,"log_dups_size":0,"ondisk_log_size":526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:59:42.865066+0000","scrub_duration":0.057927502999999998,"objects_trimmed":0,"snaptrim_duration":0.0047342360000000002,"stat_sum":{"num_bytes":12634629,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":34,"num_read_kb":4115,"num_write":467,"num_write_kb":18795,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"66'611","reported_seq":2781,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:07.692371+0000","last_change":"2023-10-30T12:52:07.692447+0000","last_active":"2023-10-30T12:52:07.692371+0000","last_peered":"2023-10-30T12:52:07.692371+0000","last_clean":"2023-10-30T12:52:07.692371+0000","last_became_active":"2023-10-30T12:28:50.937906+0000","last_became_peered":"2023-10-30T12:28:50.937906+0000","last_unstale":"2023-10-30T12:52:07.692371+0000","last_undegraded":"2023-10-30T12:52:07.692371+0000","last_fullsized":"2023-10-30T12:52:07.692371+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'611","last_scrub_stamp":"2023-10-30T12:52:07.692259+0000","last_deep_scrub":"66'611","last_deep_scrub_stamp":"2023-10-30T12:52:07.692259+0000","last_clean_scrub_stamp":"2023-10-30T12:52:07.692259+0000","objects_scrubbed":11,"log_size":611,"log_dups_size":0,"ondisk_log_size":611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:46:50.215640+0000","scrub_duration":0.029019208000000001,"objects_trimmed":0,"snaptrim_duration":0.003369164,"stat_sum":{"num_bytes":4268758,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":50,"num_read_kb":8215,"num_write":547,"num_write_kb":14586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"66'554","reported_seq":2723,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:46.367652+0000","last_change":"2023-10-30T12:51:46.367652+0000","last_active":"2023-10-30T12:51:46.367652+0000","last_peered":"2023-10-30T12:51:46.367652+0000","last_clean":"2023-10-30T12:51:46.367652+0000","last_became_active":"2023-10-30T12:28:50.932753+0000","last_became_peered":"2023-10-30T12:28:50.932753+0000","last_unstale":"2023-10-30T12:51:46.367652+0000","last_undegraded":"2023-10-30T12:51:46.367652+0000","last_fullsized":"2023-10-30T12:51:46.367652+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'554","last_scrub_stamp":"2023-10-30T12:51:46.367517+0000","last_deep_scrub":"66'554","last_deep_scrub_stamp":"2023-10-30T12:51:46.367517+0000","last_clean_scrub_stamp":"2023-10-30T12:51:46.367517+0000","objects_scrubbed":25,"log_size":554,"log_dups_size":0,"ondisk_log_size":554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:15:04.516611+0000","scrub_duration":0.047742825000000003,"objects_trimmed":0,"snaptrim_duration":0.0027063,"stat_sum":{"num_bytes":16858869,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":47,"num_read_kb":17,"num_write":493,"num_write_kb":19132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"66'580","reported_seq":2751,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:49.898263+0000","last_change":"2023-10-30T12:51:49.898263+0000","last_active":"2023-10-30T12:51:49.898263+0000","last_peered":"2023-10-30T12:51:49.898263+0000","last_clean":"2023-10-30T12:51:49.898263+0000","last_became_active":"2023-10-30T12:28:50.943011+0000","last_became_peered":"2023-10-30T12:28:50.943011+0000","last_unstale":"2023-10-30T12:51:49.898263+0000","last_undegraded":"2023-10-30T12:51:49.898263+0000","last_fullsized":"2023-10-30T12:51:49.898263+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'580","last_scrub_stamp":"2023-10-30T12:51:49.898122+0000","last_deep_scrub":"66'580","last_deep_scrub_stamp":"2023-10-30T12:51:49.898122+0000","last_clean_scrub_stamp":"2023-10-30T12:51:49.898122+0000","objects_scrubbed":21,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:56:15.625962+0000","scrub_duration":0.069989635999999994,"objects_trimmed":0,"snaptrim_duration":0.001813863,"stat_sum":{"num_bytes":21155803,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":51,"num_read_kb":4115,"num_write":508,"num_write_kb":22928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"66'583","reported_seq":2750,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:16.300403+0000","last_change":"2023-10-30T12:52:16.300527+0000","last_active":"2023-10-30T12:52:16.300403+0000","last_peered":"2023-10-30T12:52:16.300403+0000","last_clean":"2023-10-30T12:52:16.300403+0000","last_became_active":"2023-10-30T12:28:50.952744+0000","last_became_peered":"2023-10-30T12:28:50.952744+0000","last_unstale":"2023-10-30T12:52:16.300403+0000","last_undegraded":"2023-10-30T12:52:16.300403+0000","last_fullsized":"2023-10-30T12:52:16.300403+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'583","last_scrub_stamp":"2023-10-30T12:52:16.300199+0000","last_deep_scrub":"66'583","last_deep_scrub_stamp":"2023-10-30T12:52:16.300199+0000","last_clean_scrub_stamp":"2023-10-30T12:52:16.300199+0000","objects_scrubbed":17,"log_size":583,"log_dups_size":0,"ondisk_log_size":583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:11:54.600332+0000","scrub_duration":0.041195979000000001,"objects_trimmed":0,"snaptrim_duration":0.0034371530000000001,"stat_sum":{"num_bytes":4262714,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":46,"num_read_kb":18,"num_write":521,"num_write_kb":9441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"66'590","reported_seq":2743,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:36.468011+0000","last_change":"2023-10-30T12:38:17.120572+0000","last_active":"2023-10-30T12:51:36.468011+0000","last_peered":"2023-10-30T12:51:36.468011+0000","last_clean":"2023-10-30T12:51:36.468011+0000","last_became_active":"2023-10-30T12:28:50.974576+0000","last_became_peered":"2023-10-30T12:28:50.974576+0000","last_unstale":"2023-10-30T12:51:36.468011+0000","last_undegraded":"2023-10-30T12:51:36.468011+0000","last_fullsized":"2023-10-30T12:51:36.468011+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:50.333136+0000","last_clean_scrub_stamp":"2023-10-30T12:28:50.333136+0000","objects_scrubbed":0,"log_size":590,"log_dups_size":0,"ondisk_log_size":590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080003049999999992,"stat_sum":{"num_bytes":8462359,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":40,"num_read_kb":4113,"num_write":531,"num_write_kb":16554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"66'543","reported_seq":2722,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:46.554500+0000","last_change":"2023-10-30T12:51:46.554500+0000","last_active":"2023-10-30T12:51:46.554500+0000","last_peered":"2023-10-30T12:51:46.554500+0000","last_clean":"2023-10-30T12:51:46.554500+0000","last_became_active":"2023-10-30T12:28:50.937384+0000","last_became_peered":"2023-10-30T12:28:50.937384+0000","last_unstale":"2023-10-30T12:51:46.554500+0000","last_undegraded":"2023-10-30T12:51:46.554500+0000","last_fullsized":"2023-10-30T12:51:46.554500+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'543","last_scrub_stamp":"2023-10-30T12:51:46.554353+0000","last_deep_scrub":"66'543","last_deep_scrub_stamp":"2023-10-30T12:51:46.554353+0000","last_clean_scrub_stamp":"2023-10-30T12:51:46.554353+0000","objects_scrubbed":16,"log_size":543,"log_dups_size":0,"ondisk_log_size":543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:23:08.192334+0000","scrub_duration":0.042828463999999997,"objects_trimmed":0,"snaptrim_duration":0.001162803,"stat_sum":{"num_bytes":8456287,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":58,"num_read_kb":27,"num_write":488,"num_write_kb":9441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"66'581","reported_seq":2758,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:14.634497+0000","last_change":"2023-10-30T12:52:14.634614+0000","last_active":"2023-10-30T12:52:14.634497+0000","last_peered":"2023-10-30T12:52:14.634497+0000","last_clean":"2023-10-30T12:52:14.634497+0000","last_became_active":"2023-10-30T12:28:50.973369+0000","last_became_peered":"2023-10-30T12:28:50.973369+0000","last_unstale":"2023-10-30T12:52:14.634497+0000","last_undegraded":"2023-10-30T12:52:14.634497+0000","last_fullsized":"2023-10-30T12:52:14.634497+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'581","last_scrub_stamp":"2023-10-30T12:52:14.634290+0000","last_deep_scrub":"66'581","last_deep_scrub_stamp":"2023-10-30T12:52:14.634290+0000","last_clean_scrub_stamp":"2023-10-30T12:52:14.634290+0000","objects_scrubbed":27,"log_size":581,"log_dups_size":0,"ondisk_log_size":581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:50:37.072999+0000","scrub_duration":0.087054615000000002,"objects_trimmed":0,"snaptrim_duration":0.0093923919999999994,"stat_sum":{"num_bytes":12689191,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":55,"num_read_kb":4114,"num_write":505,"num_write_kb":18071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"66'592","reported_seq":2762,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:47.730012+0000","last_change":"2023-10-30T12:51:47.730137+0000","last_active":"2023-10-30T12:51:47.730012+0000","last_peered":"2023-10-30T12:51:47.730012+0000","last_clean":"2023-10-30T12:51:47.730012+0000","last_became_active":"2023-10-30T12:28:50.954831+0000","last_became_peered":"2023-10-30T12:28:50.954831+0000","last_unstale":"2023-10-30T12:51:47.730012+0000","last_undegraded":"2023-10-30T12:51:47.730012+0000","last_fullsized":"2023-10-30T12:51:47.730012+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'592","last_scrub_stamp":"2023-10-30T12:51:47.729828+0000","last_deep_scrub":"66'592","last_deep_scrub_stamp":"2023-10-30T12:51:47.729828+0000","last_clean_scrub_stamp":"2023-10-30T12:51:47.729828+0000","objects_scrubbed":20,"log_size":592,"log_dups_size":0,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:14:47.264586+0000","scrub_duration":0.089463355999999994,"objects_trimmed":0,"snaptrim_duration":0.007119116,"stat_sum":{"num_bytes":21055454,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":49,"num_read_kb":4113,"num_write":524,"num_write_kb":30367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"66'549","reported_seq":2711,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:10.434180+0000","last_change":"2023-10-30T12:52:10.434275+0000","last_active":"2023-10-30T12:52:10.434180+0000","last_peered":"2023-10-30T12:52:10.434180+0000","last_clean":"2023-10-30T12:52:10.434180+0000","last_became_active":"2023-10-30T12:28:50.962065+0000","last_became_peered":"2023-10-30T12:28:50.962065+0000","last_unstale":"2023-10-30T12:52:10.434180+0000","last_undegraded":"2023-10-30T12:52:10.434180+0000","last_fullsized":"2023-10-30T12:52:10.434180+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'549","last_scrub_stamp":"2023-10-30T12:52:10.434060+0000","last_deep_scrub":"66'549","last_deep_scrub_stamp":"2023-10-30T12:52:10.434060+0000","last_clean_scrub_stamp":"2023-10-30T12:52:10.434060+0000","objects_scrubbed":17,"log_size":549,"log_dups_size":0,"ondisk_log_size":549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:18:27.491696+0000","scrub_duration":0.079332753000000006,"objects_trimmed":0,"snaptrim_duration":0.0037739459999999998,"stat_sum":{"num_bytes":25200014,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":41,"num_read_kb":21,"num_write":490,"num_write_kb":26352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"66'625","reported_seq":2792,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:33.218192+0000","last_change":"2023-10-30T12:52:33.218401+0000","last_active":"2023-10-30T12:52:33.218192+0000","last_peered":"2023-10-30T12:52:33.218192+0000","last_clean":"2023-10-30T12:52:33.218192+0000","last_became_active":"2023-10-30T12:28:50.967651+0000","last_became_peered":"2023-10-30T12:28:50.967651+0000","last_unstale":"2023-10-30T12:52:33.218192+0000","last_undegraded":"2023-10-30T12:52:33.218192+0000","last_fullsized":"2023-10-30T12:52:33.218192+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'625","last_scrub_stamp":"2023-10-30T12:52:33.217988+0000","last_deep_scrub":"66'625","last_deep_scrub_stamp":"2023-10-30T12:52:33.217988+0000","last_clean_scrub_stamp":"2023-10-30T12:52:33.217988+0000","objects_scrubbed":17,"log_size":625,"log_dups_size":0,"ondisk_log_size":625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:41:02.531460+0000","scrub_duration":0.058304638999999998,"objects_trimmed":0,"snaptrim_duration":0.0081085610000000002,"stat_sum":{"num_bytes":8425872,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":46,"num_read_kb":21,"num_write":555,"num_write_kb":13596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"66'584","reported_seq":2755,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:10.755998+0000","last_change":"2023-10-30T12:52:10.756068+0000","last_active":"2023-10-30T12:52:10.755998+0000","last_peered":"2023-10-30T12:52:10.755998+0000","last_clean":"2023-10-30T12:52:10.755998+0000","last_became_active":"2023-10-30T12:28:50.962290+0000","last_became_peered":"2023-10-30T12:28:50.962290+0000","last_unstale":"2023-10-30T12:52:10.755998+0000","last_undegraded":"2023-10-30T12:52:10.755998+0000","last_fullsized":"2023-10-30T12:52:10.755998+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'584","last_scrub_stamp":"2023-10-30T12:52:10.755902+0000","last_deep_scrub":"66'584","last_deep_scrub_stamp":"2023-10-30T12:52:10.755902+0000","last_clean_scrub_stamp":"2023-10-30T12:52:10.755902+0000","objects_scrubbed":24,"log_size":584,"log_dups_size":0,"ondisk_log_size":584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:10:48.108121+0000","scrub_duration":0.037006553999999997,"objects_trimmed":0,"snaptrim_duration":0.0042026670000000002,"stat_sum":{"num_bytes":195030,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":50,"num_read_kb":4117,"num_write":509,"num_write_kb":6258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"66'591","reported_seq":2752,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:13.307896+0000","last_change":"2023-10-30T12:52:13.308161+0000","last_active":"2023-10-30T12:52:13.307896+0000","last_peered":"2023-10-30T12:52:13.307896+0000","last_clean":"2023-10-30T12:52:13.307896+0000","last_became_active":"2023-10-30T12:28:50.965713+0000","last_became_peered":"2023-10-30T12:28:50.965713+0000","last_unstale":"2023-10-30T12:52:13.307896+0000","last_undegraded":"2023-10-30T12:52:13.307896+0000","last_fullsized":"2023-10-30T12:52:13.307896+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'591","last_scrub_stamp":"2023-10-30T12:52:13.307638+0000","last_deep_scrub":"66'591","last_deep_scrub_stamp":"2023-10-30T12:52:13.307638+0000","last_clean_scrub_stamp":"2023-10-30T12:52:13.307638+0000","objects_scrubbed":16,"log_size":591,"log_dups_size":0,"ondisk_log_size":591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:12:08.906462+0000","scrub_duration":0.066437737999999996,"objects_trimmed":0,"snaptrim_duration":0.0056034209999999999,"stat_sum":{"num_bytes":12659519,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":40,"num_read_kb":16,"num_write":528,"num_write_kb":18837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"66'544","reported_seq":2696,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:22.817181+0000","last_change":"2023-10-30T12:52:22.817243+0000","last_active":"2023-10-30T12:52:22.817181+0000","last_peered":"2023-10-30T12:52:22.817181+0000","last_clean":"2023-10-30T12:52:22.817181+0000","last_became_active":"2023-10-30T12:28:50.951026+0000","last_became_peered":"2023-10-30T12:28:50.951026+0000","last_unstale":"2023-10-30T12:52:22.817181+0000","last_undegraded":"2023-10-30T12:52:22.817181+0000","last_fullsized":"2023-10-30T12:52:22.817181+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'544","last_scrub_stamp":"2023-10-30T12:52:22.817093+0000","last_deep_scrub":"66'544","last_deep_scrub_stamp":"2023-10-30T12:52:22.817093+0000","last_clean_scrub_stamp":"2023-10-30T12:52:22.817093+0000","objects_scrubbed":11,"log_size":544,"log_dups_size":0,"ondisk_log_size":544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:07:21.492835+0000","scrub_duration":0.052369851000000002,"objects_trimmed":0,"snaptrim_duration":0.0031697840000000001,"stat_sum":{"num_bytes":12597918,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":31,"num_read_kb":12,"num_write":481,"num_write_kb":19629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"66'624","reported_seq":2798,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:52.603819+0000","last_change":"2023-10-30T12:51:52.604014+0000","last_active":"2023-10-30T12:51:52.603819+0000","last_peered":"2023-10-30T12:51:52.603819+0000","last_clean":"2023-10-30T12:51:52.603819+0000","last_became_active":"2023-10-30T12:28:50.962484+0000","last_became_peered":"2023-10-30T12:28:50.962484+0000","last_unstale":"2023-10-30T12:51:52.603819+0000","last_undegraded":"2023-10-30T12:51:52.603819+0000","last_fullsized":"2023-10-30T12:51:52.603819+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'624","last_scrub_stamp":"2023-10-30T12:51:52.603583+0000","last_deep_scrub":"66'624","last_deep_scrub_stamp":"2023-10-30T12:51:52.603583+0000","last_clean_scrub_stamp":"2023-10-30T12:51:52.603583+0000","objects_scrubbed":26,"log_size":624,"log_dups_size":0,"ondisk_log_size":624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:26:46.288012+0000","scrub_duration":0.079879599999999995,"objects_trimmed":0,"snaptrim_duration":0.0083773500000000004,"stat_sum":{"num_bytes":12675985,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":51,"num_read_kb":23,"num_write":557,"num_write_kb":15080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"66'532","reported_seq":2695,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:51.872361+0000","last_change":"2023-10-30T12:51:51.872430+0000","last_active":"2023-10-30T12:51:51.872361+0000","last_peered":"2023-10-30T12:51:51.872361+0000","last_clean":"2023-10-30T12:51:51.872361+0000","last_became_active":"2023-10-30T12:28:50.933093+0000","last_became_peered":"2023-10-30T12:28:50.933093+0000","last_unstale":"2023-10-30T12:51:51.872361+0000","last_undegraded":"2023-10-30T12:51:51.872361+0000","last_fullsized":"2023-10-30T12:51:51.872361+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'532","last_scrub_stamp":"2023-10-30T12:51:51.872277+0000","last_deep_scrub":"66'532","last_deep_scrub_stamp":"2023-10-30T12:51:51.872277+0000","last_clean_scrub_stamp":"2023-10-30T12:51:51.872277+0000","objects_scrubbed":22,"log_size":532,"log_dups_size":0,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:06:03.858620+0000","scrub_duration":0.057177436999999998,"objects_trimmed":0,"snaptrim_duration":0.0022251829999999999,"stat_sum":{"num_bytes":25274091,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":42,"num_read_kb":8207,"num_write":449,"num_write_kb":27548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"66'517","reported_seq":2664,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:04.763148+0000","last_change":"2023-10-30T12:52:04.763210+0000","last_active":"2023-10-30T12:52:04.763148+0000","last_peered":"2023-10-30T12:52:04.763148+0000","last_clean":"2023-10-30T12:52:04.763148+0000","last_became_active":"2023-10-30T12:28:50.952531+0000","last_became_peered":"2023-10-30T12:28:50.952531+0000","last_unstale":"2023-10-30T12:52:04.763148+0000","last_undegraded":"2023-10-30T12:52:04.763148+0000","last_fullsized":"2023-10-30T12:52:04.763148+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'517","last_scrub_stamp":"2023-10-30T12:52:04.763062+0000","last_deep_scrub":"66'517","last_deep_scrub_stamp":"2023-10-30T12:52:04.763062+0000","last_clean_scrub_stamp":"2023-10-30T12:52:04.763062+0000","objects_scrubbed":18,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:37:42.920027+0000","scrub_duration":0.034901690999999999,"objects_trimmed":0,"snaptrim_duration":0.0039301029999999999,"stat_sum":{"num_bytes":4373272,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":26,"num_read_kb":3284,"num_write":458,"num_write_kb":14265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"66'542","reported_seq":2709,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:59.310588+0000","last_change":"2023-10-30T12:51:59.310659+0000","last_active":"2023-10-30T12:51:59.310588+0000","last_peered":"2023-10-30T12:51:59.310588+0000","last_clean":"2023-10-30T12:51:59.310588+0000","last_became_active":"2023-10-30T12:28:50.933124+0000","last_became_peered":"2023-10-30T12:28:50.933124+0000","last_unstale":"2023-10-30T12:51:59.310588+0000","last_undegraded":"2023-10-30T12:51:59.310588+0000","last_fullsized":"2023-10-30T12:51:59.310588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'542","last_scrub_stamp":"2023-10-30T12:51:59.310503+0000","last_deep_scrub":"66'542","last_deep_scrub_stamp":"2023-10-30T12:51:59.310503+0000","last_clean_scrub_stamp":"2023-10-30T12:51:59.310503+0000","objects_scrubbed":24,"log_size":542,"log_dups_size":0,"ondisk_log_size":542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:48:29.721009+0000","scrub_duration":0.033010021000000001,"objects_trimmed":0,"snaptrim_duration":0.0034752709999999998,"stat_sum":{"num_bytes":8490168,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":45,"num_read_kb":14,"num_write":469,"num_write_kb":10488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"66'611","reported_seq":2784,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:21.315467+0000","last_change":"2023-10-30T12:52:21.315616+0000","last_active":"2023-10-30T12:52:21.315467+0000","last_peered":"2023-10-30T12:52:21.315467+0000","last_clean":"2023-10-30T12:52:21.315467+0000","last_became_active":"2023-10-30T12:28:50.965639+0000","last_became_peered":"2023-10-30T12:28:50.965639+0000","last_unstale":"2023-10-30T12:52:21.315467+0000","last_undegraded":"2023-10-30T12:52:21.315467+0000","last_fullsized":"2023-10-30T12:52:21.315467+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'611","last_scrub_stamp":"2023-10-30T12:52:21.315261+0000","last_deep_scrub":"66'611","last_deep_scrub_stamp":"2023-10-30T12:52:21.315261+0000","last_clean_scrub_stamp":"2023-10-30T12:52:21.315261+0000","objects_scrubbed":17,"log_size":611,"log_dups_size":0,"ondisk_log_size":611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:59:20.933491+0000","scrub_duration":0.051029466000000002,"objects_trimmed":0,"snaptrim_duration":0.0076861280000000004,"stat_sum":{"num_bytes":8432593,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":51,"num_read_kb":20,"num_write":547,"num_write_kb":11509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"66'595","reported_seq":2767,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:45.496340+0000","last_change":"2023-10-30T12:51:45.496340+0000","last_active":"2023-10-30T12:51:45.496340+0000","last_peered":"2023-10-30T12:51:45.496340+0000","last_clean":"2023-10-30T12:51:45.496340+0000","last_became_active":"2023-10-30T12:28:50.955084+0000","last_became_peered":"2023-10-30T12:28:50.955084+0000","last_unstale":"2023-10-30T12:51:45.496340+0000","last_undegraded":"2023-10-30T12:51:45.496340+0000","last_fullsized":"2023-10-30T12:51:45.496340+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'595","last_scrub_stamp":"2023-10-30T12:51:45.496211+0000","last_deep_scrub":"66'595","last_deep_scrub_stamp":"2023-10-30T12:51:45.496211+0000","last_clean_scrub_stamp":"2023-10-30T12:51:45.496211+0000","objects_scrubbed":12,"log_size":595,"log_dups_size":0,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:13:37.745803+0000","scrub_duration":0.027110084,"objects_trimmed":0,"snaptrim_duration":0.0012043119999999999,"stat_sum":{"num_bytes":4257179,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":50,"num_read_kb":22,"num_write":532,"num_write_kb":9837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"66'569","reported_seq":2744,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:54.604746+0000","last_change":"2023-10-30T12:51:54.604869+0000","last_active":"2023-10-30T12:51:54.604746+0000","last_peered":"2023-10-30T12:51:54.604746+0000","last_clean":"2023-10-30T12:51:54.604746+0000","last_became_active":"2023-10-30T12:28:50.944775+0000","last_became_peered":"2023-10-30T12:28:50.944775+0000","last_unstale":"2023-10-30T12:51:54.604746+0000","last_undegraded":"2023-10-30T12:51:54.604746+0000","last_fullsized":"2023-10-30T12:51:54.604746+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'569","last_scrub_stamp":"2023-10-30T12:51:54.604552+0000","last_deep_scrub":"66'569","last_deep_scrub_stamp":"2023-10-30T12:51:54.604552+0000","last_clean_scrub_stamp":"2023-10-30T12:51:54.604552+0000","objects_scrubbed":20,"log_size":569,"log_dups_size":0,"ondisk_log_size":569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:33:53.639827+0000","scrub_duration":0.054468190999999999,"objects_trimmed":0,"snaptrim_duration":0.0070573429999999998,"stat_sum":{"num_bytes":12763889,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":53,"num_read_kb":21,"num_write":514,"num_write_kb":13736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"66'590","reported_seq":2750,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:44.129343+0000","last_change":"2023-10-30T12:51:44.129472+0000","last_active":"2023-10-30T12:51:44.129343+0000","last_peered":"2023-10-30T12:51:44.129343+0000","last_clean":"2023-10-30T12:51:44.129343+0000","last_became_active":"2023-10-30T12:28:50.940567+0000","last_became_peered":"2023-10-30T12:28:50.940567+0000","last_unstale":"2023-10-30T12:51:44.129343+0000","last_undegraded":"2023-10-30T12:51:44.129343+0000","last_fullsized":"2023-10-30T12:51:44.129343+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'590","last_scrub_stamp":"2023-10-30T12:51:44.129123+0000","last_deep_scrub":"66'590","last_deep_scrub_stamp":"2023-10-30T12:51:44.129123+0000","last_clean_scrub_stamp":"2023-10-30T12:51:44.129123+0000","objects_scrubbed":10,"log_size":590,"log_dups_size":0,"ondisk_log_size":590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:25:14.156000+0000","scrub_duration":0.048555811999999997,"objects_trimmed":0,"snaptrim_duration":0.0028523340000000002,"stat_sum":{"num_bytes":9696728,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":17,"num_write":529,"num_write_kb":12781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"66'283","reported_seq":371,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:45.780003+0000","last_change":"2023-10-30T12:51:45.780059+0000","last_active":"2023-10-30T12:51:45.780003+0000","last_peered":"2023-10-30T12:51:45.780003+0000","last_clean":"2023-10-30T12:51:45.780003+0000","last_became_active":"2023-10-30T12:28:50.395382+0000","last_became_peered":"2023-10-30T12:28:50.395382+0000","last_unstale":"2023-10-30T12:51:45.780003+0000","last_undegraded":"2023-10-30T12:51:45.780003+0000","last_fullsized":"2023-10-30T12:51:45.780003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'283","last_scrub_stamp":"2023-10-30T12:51:45.779902+0000","last_deep_scrub":"66'283","last_deep_scrub_stamp":"2023-10-30T12:51:45.779902+0000","last_clean_scrub_stamp":"2023-10-30T12:51:45.779902+0000","objects_scrubbed":5,"log_size":283,"log_dups_size":0,"ondisk_log_size":283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:36:15.846905+0000","scrub_duration":0.042332846,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582946,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":45059,"num_write":213,"num_write_kb":12413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8500,"num_omap_keys":17,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"66'511","reported_seq":2676,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:45.384937+0000","last_change":"2023-10-30T12:51:45.385000+0000","last_active":"2023-10-30T12:51:45.384937+0000","last_peered":"2023-10-30T12:51:45.384937+0000","last_clean":"2023-10-30T12:51:45.384937+0000","last_became_active":"2023-10-30T12:28:50.952613+0000","last_became_peered":"2023-10-30T12:28:50.952613+0000","last_unstale":"2023-10-30T12:51:45.384937+0000","last_undegraded":"2023-10-30T12:51:45.384937+0000","last_fullsized":"2023-10-30T12:51:45.384937+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'511","last_scrub_stamp":"2023-10-30T12:51:45.384858+0000","last_deep_scrub":"66'511","last_deep_scrub_stamp":"2023-10-30T12:51:45.384858+0000","last_clean_scrub_stamp":"2023-10-30T12:51:45.384858+0000","objects_scrubbed":19,"log_size":511,"log_dups_size":0,"ondisk_log_size":511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:37:21.237208+0000","scrub_duration":0.069355456999999995,"objects_trimmed":0,"snaptrim_duration":0.0038505420000000002,"stat_sum":{"num_bytes":21061321,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":45,"num_read_kb":18,"num_write":449,"num_write_kb":22047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"66'909","reported_seq":1201,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:49.177081+0000","last_change":"2023-10-30T12:51:42.662808+0000","last_active":"2023-10-30T12:51:49.177081+0000","last_peered":"2023-10-30T12:51:49.177081+0000","last_clean":"2023-10-30T12:51:49.177081+0000","last_became_active":"2023-10-30T12:28:50.383898+0000","last_became_peered":"2023-10-30T12:28:50.383898+0000","last_unstale":"2023-10-30T12:51:49.177081+0000","last_undegraded":"2023-10-30T12:51:49.177081+0000","last_fullsized":"2023-10-30T12:51:49.177081+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'908","last_scrub_stamp":"2023-10-30T12:51:42.662429+0000","last_deep_scrub":"66'908","last_deep_scrub_stamp":"2023-10-30T12:51:42.662429+0000","last_clean_scrub_stamp":"2023-10-30T12:51:42.662429+0000","objects_scrubbed":9,"log_size":909,"log_dups_size":0,"ondisk_log_size":909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:01:46.593761+0000","scrub_duration":0.098226147999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21567319,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":234,"num_read_kb":49886,"num_write":842,"num_write_kb":21597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":23432,"num_omap_keys":43,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"66'457","reported_seq":759,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:48.571287+0000","last_change":"2023-10-30T12:51:41.780215+0000","last_active":"2023-10-30T12:51:48.571287+0000","last_peered":"2023-10-30T12:51:48.571287+0000","last_clean":"2023-10-30T12:51:48.571287+0000","last_became_active":"2023-10-30T12:28:50.385963+0000","last_became_peered":"2023-10-30T12:28:50.385963+0000","last_unstale":"2023-10-30T12:51:48.571287+0000","last_undegraded":"2023-10-30T12:51:48.571287+0000","last_fullsized":"2023-10-30T12:51:48.571287+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'454","last_scrub_stamp":"2023-10-30T12:51:41.780079+0000","last_deep_scrub":"66'454","last_deep_scrub_stamp":"2023-10-30T12:51:41.780079+0000","last_clean_scrub_stamp":"2023-10-30T12:51:41.780079+0000","objects_scrubbed":4,"log_size":457,"log_dups_size":0,"ondisk_log_size":457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:29:54.800743+0000","scrub_duration":0.030692146999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8006399,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":240,"num_read_kb":9095,"num_write":389,"num_write_kb":8081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"66'639","reported_seq":2815,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:51.809783+0000","last_change":"2023-10-30T12:51:51.809888+0000","last_active":"2023-10-30T12:51:51.809783+0000","last_peered":"2023-10-30T12:51:51.809783+0000","last_clean":"2023-10-30T12:51:51.809783+0000","last_became_active":"2023-10-30T12:28:50.935764+0000","last_became_peered":"2023-10-30T12:28:50.935764+0000","last_unstale":"2023-10-30T12:51:51.809783+0000","last_undegraded":"2023-10-30T12:51:51.809783+0000","last_fullsized":"2023-10-30T12:51:51.809783+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'639","last_scrub_stamp":"2023-10-30T12:51:51.809674+0000","last_deep_scrub":"66'639","last_deep_scrub_stamp":"2023-10-30T12:51:51.809674+0000","last_clean_scrub_stamp":"2023-10-30T12:51:51.809674+0000","objects_scrubbed":21,"log_size":639,"log_dups_size":0,"ondisk_log_size":639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:53.793939+0000","scrub_duration":0.025406960999999999,"objects_trimmed":0,"snaptrim_duration":0.0031329320000000002,"stat_sum":{"num_bytes":4280685,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":56,"num_read_kb":4120,"num_write":569,"num_write_kb":13778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"66'571","reported_seq":655,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:43.489239+0000","last_change":"2023-10-30T12:51:43.489353+0000","last_active":"2023-10-30T12:51:43.489239+0000","last_peered":"2023-10-30T12:51:43.489239+0000","last_clean":"2023-10-30T12:51:43.489239+0000","last_became_active":"2023-10-30T12:28:50.391494+0000","last_became_peered":"2023-10-30T12:28:50.391494+0000","last_unstale":"2023-10-30T12:51:43.489239+0000","last_undegraded":"2023-10-30T12:51:43.489239+0000","last_fullsized":"2023-10-30T12:51:43.489239+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'571","last_scrub_stamp":"2023-10-30T12:51:43.489033+0000","last_deep_scrub":"66'571","last_deep_scrub_stamp":"2023-10-30T12:51:43.489033+0000","last_clean_scrub_stamp":"2023-10-30T12:51:43.489033+0000","objects_scrubbed":3,"log_size":571,"log_dups_size":0,"ondisk_log_size":571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:24:33.871160+0000","scrub_duration":0.058779308000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":36864,"num_write":519,"num_write_kb":12589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"66'619","reported_seq":2786,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:07.826653+0000","last_change":"2023-10-30T12:52:07.826695+0000","last_active":"2023-10-30T12:52:07.826653+0000","last_peered":"2023-10-30T12:52:07.826653+0000","last_clean":"2023-10-30T12:52:07.826653+0000","last_became_active":"2023-10-30T12:28:50.933062+0000","last_became_peered":"2023-10-30T12:28:50.933062+0000","last_unstale":"2023-10-30T12:52:07.826653+0000","last_undegraded":"2023-10-30T12:52:07.826653+0000","last_fullsized":"2023-10-30T12:52:07.826653+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'619","last_scrub_stamp":"2023-10-30T12:52:07.826574+0000","last_deep_scrub":"66'619","last_deep_scrub_stamp":"2023-10-30T12:52:07.826574+0000","last_clean_scrub_stamp":"2023-10-30T12:52:07.826574+0000","objects_scrubbed":27,"log_size":619,"log_dups_size":0,"ondisk_log_size":619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:21:58.906873+0000","scrub_duration":0.081541012999999996,"objects_trimmed":0,"snaptrim_duration":0.0021140489999999998,"stat_sum":{"num_bytes":33653289,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":46,"num_read_kb":4115,"num_write":549,"num_write_kb":39173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"66'132","reported_seq":214,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:46.616340+0000","last_change":"2023-10-30T12:51:46.616340+0000","last_active":"2023-10-30T12:51:46.616340+0000","last_peered":"2023-10-30T12:51:46.616340+0000","last_clean":"2023-10-30T12:51:46.616340+0000","last_became_active":"2023-10-30T12:28:50.393686+0000","last_became_peered":"2023-10-30T12:28:50.393686+0000","last_unstale":"2023-10-30T12:51:46.616340+0000","last_undegraded":"2023-10-30T12:51:46.616340+0000","last_fullsized":"2023-10-30T12:51:46.616340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'131","last_scrub_stamp":"2023-10-30T12:51:46.615853+0000","last_deep_scrub":"66'131","last_deep_scrub_stamp":"2023-10-30T12:51:46.615853+0000","last_clean_scrub_stamp":"2023-10-30T12:51:46.615853+0000","objects_scrubbed":3,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:42:31.808839+0000","scrub_duration":0.0095666550000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":75,"num_write_kb":93,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9282,"num_omap_keys":17,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"66'589","reported_seq":2760,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:15.339095+0000","last_change":"2023-10-30T12:52:15.339230+0000","last_active":"2023-10-30T12:52:15.339095+0000","last_peered":"2023-10-30T12:52:15.339095+0000","last_clean":"2023-10-30T12:52:15.339095+0000","last_became_active":"2023-10-30T12:28:50.963105+0000","last_became_peered":"2023-10-30T12:28:50.963105+0000","last_unstale":"2023-10-30T12:52:15.339095+0000","last_undegraded":"2023-10-30T12:52:15.339095+0000","last_fullsized":"2023-10-30T12:52:15.339095+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'589","last_scrub_stamp":"2023-10-30T12:52:15.338927+0000","last_deep_scrub":"66'589","last_deep_scrub_stamp":"2023-10-30T12:52:15.338927+0000","last_clean_scrub_stamp":"2023-10-30T12:52:15.338927+0000","objects_scrubbed":22,"log_size":589,"log_dups_size":0,"ondisk_log_size":589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:22:02.826284+0000","scrub_duration":0.088894443000000004,"objects_trimmed":0,"snaptrim_duration":0.0080066809999999999,"stat_sum":{"num_bytes":16920090,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":51,"num_read_kb":18,"num_write":515,"num_write_kb":18479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"66'284","reported_seq":366,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:59.398926+0000","last_change":"2023-10-30T12:51:59.399057+0000","last_active":"2023-10-30T12:51:59.398926+0000","last_peered":"2023-10-30T12:51:59.398926+0000","last_clean":"2023-10-30T12:51:59.398926+0000","last_became_active":"2023-10-30T12:28:50.397041+0000","last_became_peered":"2023-10-30T12:28:50.397041+0000","last_unstale":"2023-10-30T12:51:59.398926+0000","last_undegraded":"2023-10-30T12:51:59.398926+0000","last_fullsized":"2023-10-30T12:51:59.398926+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'284","last_scrub_stamp":"2023-10-30T12:51:59.398723+0000","last_deep_scrub":"66'284","last_deep_scrub_stamp":"2023-10-30T12:51:59.398723+0000","last_clean_scrub_stamp":"2023-10-30T12:51:59.398723+0000","objects_scrubbed":3,"log_size":284,"log_dups_size":0,"ondisk_log_size":284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:44:04.485785+0000","scrub_duration":0.038907621000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":16396,"num_write":240,"num_write_kb":8312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"66'570","reported_seq":2732,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:11.399647+0000","last_change":"2023-10-30T12:52:11.399712+0000","last_active":"2023-10-30T12:52:11.399647+0000","last_peered":"2023-10-30T12:52:11.399647+0000","last_clean":"2023-10-30T12:52:11.399647+0000","last_became_active":"2023-10-30T12:28:50.966569+0000","last_became_peered":"2023-10-30T12:28:50.966569+0000","last_unstale":"2023-10-30T12:52:11.399647+0000","last_undegraded":"2023-10-30T12:52:11.399647+0000","last_fullsized":"2023-10-30T12:52:11.399647+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'570","last_scrub_stamp":"2023-10-30T12:52:11.399561+0000","last_deep_scrub":"66'570","last_deep_scrub_stamp":"2023-10-30T12:52:11.399561+0000","last_clean_scrub_stamp":"2023-10-30T12:52:11.399561+0000","objects_scrubbed":22,"log_size":570,"log_dups_size":0,"ondisk_log_size":570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:55.041154+0000","scrub_duration":0.068592573000000004,"objects_trimmed":0,"snaptrim_duration":0.0037972850000000001,"stat_sum":{"num_bytes":16988939,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":42,"num_read_kb":19,"num_write":508,"num_write_kb":18140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"66'375","reported_seq":463,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:45.949491+0000","last_change":"2023-10-30T12:51:45.949541+0000","last_active":"2023-10-30T12:51:45.949491+0000","last_peered":"2023-10-30T12:51:45.949491+0000","last_clean":"2023-10-30T12:51:45.949491+0000","last_became_active":"2023-10-30T12:28:50.391260+0000","last_became_peered":"2023-10-30T12:28:50.391260+0000","last_unstale":"2023-10-30T12:51:45.949491+0000","last_undegraded":"2023-10-30T12:51:45.949491+0000","last_fullsized":"2023-10-30T12:51:45.949491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'375","last_scrub_stamp":"2023-10-30T12:51:45.949405+0000","last_deep_scrub":"66'375","last_deep_scrub_stamp":"2023-10-30T12:51:45.949405+0000","last_clean_scrub_stamp":"2023-10-30T12:51:45.949405+0000","objects_scrubbed":6,"log_size":375,"log_dups_size":0,"ondisk_log_size":375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:31:43.191871+0000","scrub_duration":0.024520874000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":22,"num_read_kb":7467,"num_write":319,"num_write_kb":4401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7112,"num_omap_keys":14,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"66'524","reported_seq":2692,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:42.140557+0000","last_change":"2023-10-30T12:51:42.140557+0000","last_active":"2023-10-30T12:51:42.140557+0000","last_peered":"2023-10-30T12:51:42.140557+0000","last_clean":"2023-10-30T12:51:42.140557+0000","last_became_active":"2023-10-30T12:28:50.974843+0000","last_became_peered":"2023-10-30T12:28:50.974843+0000","last_unstale":"2023-10-30T12:51:42.140557+0000","last_undegraded":"2023-10-30T12:51:42.140557+0000","last_fullsized":"2023-10-30T12:51:42.140557+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'524","last_scrub_stamp":"2023-10-30T12:51:42.140220+0000","last_deep_scrub":"66'524","last_deep_scrub_stamp":"2023-10-30T12:51:42.140220+0000","last_clean_scrub_stamp":"2023-10-30T12:51:42.140220+0000","objects_scrubbed":12,"log_size":524,"log_dups_size":0,"ondisk_log_size":524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:15:48.848274+0000","scrub_duration":0.033227912999999998,"objects_trimmed":0,"snaptrim_duration":0.005161372,"stat_sum":{"num_bytes":4244440,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":47,"num_read_kb":17,"num_write":458,"num_write_kb":9849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"66'359","reported_seq":459,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:08.560979+0000","last_change":"2023-10-30T12:51:40.635013+0000","last_active":"2023-10-30T12:52:08.560979+0000","last_peered":"2023-10-30T12:52:08.560979+0000","last_clean":"2023-10-30T12:52:08.560979+0000","last_became_active":"2023-10-30T12:28:50.373581+0000","last_became_peered":"2023-10-30T12:28:50.373581+0000","last_unstale":"2023-10-30T12:52:08.560979+0000","last_undegraded":"2023-10-30T12:52:08.560979+0000","last_fullsized":"2023-10-30T12:52:08.560979+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'357","last_scrub_stamp":"2023-10-30T12:51:40.634678+0000","last_deep_scrub":"66'357","last_deep_scrub_stamp":"2023-10-30T12:51:40.634678+0000","last_clean_scrub_stamp":"2023-10-30T12:51:40.634678+0000","objects_scrubbed":9,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:38:14.782628+0000","scrub_duration":0.051559438999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":43,"num_read_kb":32803,"num_write":294,"num_write_kb":12545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7620,"num_omap_keys":15,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"66'519","reported_seq":2677,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:08.630116+0000","last_change":"2023-10-30T12:52:08.630261+0000","last_active":"2023-10-30T12:52:08.630116+0000","last_peered":"2023-10-30T12:52:08.630116+0000","last_clean":"2023-10-30T12:52:08.630116+0000","last_became_active":"2023-10-30T12:28:50.973269+0000","last_became_peered":"2023-10-30T12:28:50.973269+0000","last_unstale":"2023-10-30T12:52:08.630116+0000","last_undegraded":"2023-10-30T12:52:08.630116+0000","last_fullsized":"2023-10-30T12:52:08.630116+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'519","last_scrub_stamp":"2023-10-30T12:52:08.629908+0000","last_deep_scrub":"66'519","last_deep_scrub_stamp":"2023-10-30T12:52:08.629908+0000","last_clean_scrub_stamp":"2023-10-30T12:52:08.629908+0000","objects_scrubbed":20,"log_size":519,"log_dups_size":0,"ondisk_log_size":519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:08.922310+0000","scrub_duration":0.063779350999999998,"objects_trimmed":0,"snaptrim_duration":0.004001498,"stat_sum":{"num_bytes":12707497,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":38,"num_read_kb":11,"num_write":446,"num_write_kb":13522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"66'444","reported_seq":545,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:37.461917+0000","last_change":"2023-10-30T12:51:37.461917+0000","last_active":"2023-10-30T12:51:37.461917+0000","last_peered":"2023-10-30T12:51:37.461917+0000","last_clean":"2023-10-30T12:51:37.461917+0000","last_became_active":"2023-10-30T12:28:50.366269+0000","last_became_peered":"2023-10-30T12:28:50.366269+0000","last_unstale":"2023-10-30T12:51:37.461917+0000","last_undegraded":"2023-10-30T12:51:37.461917+0000","last_fullsized":"2023-10-30T12:51:37.461917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'444","last_scrub_stamp":"2023-10-30T12:51:37.461743+0000","last_deep_scrub":"66'444","last_deep_scrub_stamp":"2023-10-30T12:51:37.461743+0000","last_clean_scrub_stamp":"2023-10-30T12:51:37.461743+0000","objects_scrubbed":9,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:35:49.998956+0000","scrub_duration":0.040110959000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":52,"num_read_kb":36923,"num_write":386,"num_write_kb":8499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":16168,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"66'586","reported_seq":2759,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:43.354752+0000","last_change":"2023-10-30T12:51:43.354752+0000","last_active":"2023-10-30T12:51:43.354752+0000","last_peered":"2023-10-30T12:51:43.354752+0000","last_clean":"2023-10-30T12:51:43.354752+0000","last_became_active":"2023-10-30T12:28:50.961805+0000","last_became_peered":"2023-10-30T12:28:50.961805+0000","last_unstale":"2023-10-30T12:51:43.354752+0000","last_undegraded":"2023-10-30T12:51:43.354752+0000","last_fullsized":"2023-10-30T12:51:43.354752+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'586","last_scrub_stamp":"2023-10-30T12:51:43.354576+0000","last_deep_scrub":"66'586","last_deep_scrub_stamp":"2023-10-30T12:51:43.354576+0000","last_clean_scrub_stamp":"2023-10-30T12:51:43.354576+0000","objects_scrubbed":22,"log_size":586,"log_dups_size":0,"ondisk_log_size":586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:44:38.376225+0000","scrub_duration":0.037946186999999999,"objects_trimmed":0,"snaptrim_duration":0.0032264350000000002,"stat_sum":{"num_bytes":4388445,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":52,"num_read_kb":4114,"num_write":506,"num_write_kb":14199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"66'261","reported_seq":346,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:08.296017+0000","last_change":"2023-10-30T12:52:08.296017+0000","last_active":"2023-10-30T12:52:08.296017+0000","last_peered":"2023-10-30T12:52:08.296017+0000","last_clean":"2023-10-30T12:52:08.296017+0000","last_became_active":"2023-10-30T12:28:50.396727+0000","last_became_peered":"2023-10-30T12:28:50.396727+0000","last_unstale":"2023-10-30T12:52:08.296017+0000","last_undegraded":"2023-10-30T12:52:08.296017+0000","last_fullsized":"2023-10-30T12:52:08.296017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'261","last_scrub_stamp":"2023-10-30T12:52:08.295709+0000","last_deep_scrub":"66'261","last_deep_scrub_stamp":"2023-10-30T12:52:08.295709+0000","last_clean_scrub_stamp":"2023-10-30T12:52:08.295709+0000","objects_scrubbed":3,"log_size":261,"log_dups_size":0,"ondisk_log_size":261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:43:47.836137+0000","scrub_duration":0.032577411000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":16164,"num_write":201,"num_write_kb":8311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"66'568","reported_seq":2743,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:49.633783+0000","last_change":"2023-10-30T12:51:49.633887+0000","last_active":"2023-10-30T12:51:49.633783+0000","last_peered":"2023-10-30T12:51:49.633783+0000","last_clean":"2023-10-30T12:51:49.633783+0000","last_became_active":"2023-10-30T12:29:32.960608+0000","last_became_peered":"2023-10-30T12:29:32.960608+0000","last_unstale":"2023-10-30T12:51:49.633783+0000","last_undegraded":"2023-10-30T12:51:49.633783+0000","last_fullsized":"2023-10-30T12:51:49.633783+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'568","last_scrub_stamp":"2023-10-30T12:51:49.633664+0000","last_deep_scrub":"66'568","last_deep_scrub_stamp":"2023-10-30T12:51:49.633664+0000","last_clean_scrub_stamp":"2023-10-30T12:51:49.633664+0000","objects_scrubbed":14,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:00:42.755675+0000","scrub_duration":0.028072751999999999,"objects_trimmed":0,"snaptrim_duration":0.0012934999999999999,"stat_sum":{"num_bytes":4430082,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":59,"num_read_kb":33,"num_write":506,"num_write_kb":6384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"66'684","reported_seq":775,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:44.639628+0000","last_change":"2023-10-30T12:51:44.639628+0000","last_active":"2023-10-30T12:51:44.639628+0000","last_peered":"2023-10-30T12:51:44.639628+0000","last_clean":"2023-10-30T12:51:44.639628+0000","last_became_active":"2023-10-30T12:28:50.383570+0000","last_became_peered":"2023-10-30T12:28:50.383570+0000","last_unstale":"2023-10-30T12:51:44.639628+0000","last_undegraded":"2023-10-30T12:51:44.639628+0000","last_fullsized":"2023-10-30T12:51:44.639628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'684","last_scrub_stamp":"2023-10-30T12:51:44.639059+0000","last_deep_scrub":"66'684","last_deep_scrub_stamp":"2023-10-30T12:51:44.639059+0000","last_clean_scrub_stamp":"2023-10-30T12:51:44.639059+0000","objects_scrubbed":7,"log_size":684,"log_dups_size":0,"ondisk_log_size":684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:23:55.453101+0000","scrub_duration":0.079897833000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":37306,"num_write":619,"num_write_kb":16724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15312,"num_omap_keys":30,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"66'556","reported_seq":2710,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:01.705427+0000","last_change":"2023-10-30T12:52:01.705427+0000","last_active":"2023-10-30T12:52:01.705427+0000","last_peered":"2023-10-30T12:52:01.705427+0000","last_clean":"2023-10-30T12:52:01.705427+0000","last_became_active":"2023-10-30T12:28:50.949485+0000","last_became_peered":"2023-10-30T12:28:50.949485+0000","last_unstale":"2023-10-30T12:52:01.705427+0000","last_undegraded":"2023-10-30T12:52:01.705427+0000","last_fullsized":"2023-10-30T12:52:01.705427+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'556","last_scrub_stamp":"2023-10-30T12:52:01.705046+0000","last_deep_scrub":"66'556","last_deep_scrub_stamp":"2023-10-30T12:52:01.705046+0000","last_clean_scrub_stamp":"2023-10-30T12:52:01.705046+0000","objects_scrubbed":15,"log_size":556,"log_dups_size":0,"ondisk_log_size":556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:05:15.771642+0000","scrub_duration":0.064202585000000006,"objects_trimmed":0,"snaptrim_duration":0.0040897299999999998,"stat_sum":{"num_bytes":12622957,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":477,"num_write_kb":14339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"66'453","reported_seq":569,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:49.184348+0000","last_change":"2023-10-30T12:51:41.463979+0000","last_active":"2023-10-30T12:51:49.184348+0000","last_peered":"2023-10-30T12:51:49.184348+0000","last_clean":"2023-10-30T12:51:49.184348+0000","last_became_active":"2023-10-30T12:28:50.362624+0000","last_became_peered":"2023-10-30T12:28:50.362624+0000","last_unstale":"2023-10-30T12:51:49.184348+0000","last_undegraded":"2023-10-30T12:51:49.184348+0000","last_fullsized":"2023-10-30T12:51:49.184348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'452","last_scrub_stamp":"2023-10-30T12:51:41.463765+0000","last_deep_scrub":"66'452","last_deep_scrub_stamp":"2023-10-30T12:51:41.463765+0000","last_clean_scrub_stamp":"2023-10-30T12:51:41.463765+0000","objects_scrubbed":13,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:18.726132+0000","scrub_duration":0.043492996999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583024,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":93,"num_read_kb":24662,"num_write":378,"num_write_kb":12558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6552,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"66'600","reported_seq":2771,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:57.546471+0000","last_change":"2023-10-30T12:51:57.546471+0000","last_active":"2023-10-30T12:51:57.546471+0000","last_peered":"2023-10-30T12:51:57.546471+0000","last_clean":"2023-10-30T12:51:57.546471+0000","last_became_active":"2023-10-30T12:28:50.979344+0000","last_became_peered":"2023-10-30T12:28:50.979344+0000","last_unstale":"2023-10-30T12:51:57.546471+0000","last_undegraded":"2023-10-30T12:51:57.546471+0000","last_fullsized":"2023-10-30T12:51:57.546471+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'600","last_scrub_stamp":"2023-10-30T12:51:57.546007+0000","last_deep_scrub":"66'600","last_deep_scrub_stamp":"2023-10-30T12:51:57.546007+0000","last_clean_scrub_stamp":"2023-10-30T12:51:57.546007+0000","objects_scrubbed":24,"log_size":600,"log_dups_size":0,"ondisk_log_size":600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:44:55.842195+0000","scrub_duration":0.075819872999999996,"objects_trimmed":0,"snaptrim_duration":0.0092295529999999997,"stat_sum":{"num_bytes":12647197,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":50,"num_read_kb":8210,"num_write":532,"num_write_kb":23192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"66'248","reported_seq":334,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:37.268010+0000","last_change":"2023-10-30T12:51:37.268108+0000","last_active":"2023-10-30T12:51:37.268010+0000","last_peered":"2023-10-30T12:51:37.268010+0000","last_clean":"2023-10-30T12:51:37.268010+0000","last_became_active":"2023-10-30T12:28:50.381125+0000","last_became_peered":"2023-10-30T12:28:50.381125+0000","last_unstale":"2023-10-30T12:51:37.268010+0000","last_undegraded":"2023-10-30T12:51:37.268010+0000","last_fullsized":"2023-10-30T12:51:37.268010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'248","last_scrub_stamp":"2023-10-30T12:51:37.267826+0000","last_deep_scrub":"66'248","last_deep_scrub_stamp":"2023-10-30T12:51:37.267826+0000","last_clean_scrub_stamp":"2023-10-30T12:51:37.267826+0000","objects_scrubbed":3,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:30:50.232542+0000","scrub_duration":0.054105548000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":12378,"num_write":396,"num_write_kb":4652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"66'548","reported_seq":2705,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:09.640556+0000","last_change":"2023-10-30T12:52:09.640712+0000","last_active":"2023-10-30T12:52:09.640556+0000","last_peered":"2023-10-30T12:52:09.640556+0000","last_clean":"2023-10-30T12:52:09.640556+0000","last_became_active":"2023-10-30T12:28:50.977585+0000","last_became_peered":"2023-10-30T12:28:50.977585+0000","last_unstale":"2023-10-30T12:52:09.640556+0000","last_undegraded":"2023-10-30T12:52:09.640556+0000","last_fullsized":"2023-10-30T12:52:09.640556+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'548","last_scrub_stamp":"2023-10-30T12:52:09.640316+0000","last_deep_scrub":"66'548","last_deep_scrub_stamp":"2023-10-30T12:52:09.640316+0000","last_clean_scrub_stamp":"2023-10-30T12:52:09.640316+0000","objects_scrubbed":17,"log_size":548,"log_dups_size":0,"ondisk_log_size":548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:06:42.121080+0000","scrub_duration":0.072014062000000004,"objects_trimmed":0,"snaptrim_duration":0.0089211889999999995,"stat_sum":{"num_bytes":12933034,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":36,"num_read_kb":8205,"num_write":483,"num_write_kb":18997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"66'359","reported_seq":445,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:38.474826+0000","last_change":"2023-10-30T12:51:38.474877+0000","last_active":"2023-10-30T12:51:38.474826+0000","last_peered":"2023-10-30T12:51:38.474826+0000","last_clean":"2023-10-30T12:51:38.474826+0000","last_became_active":"2023-10-30T12:28:50.366974+0000","last_became_peered":"2023-10-30T12:28:50.366974+0000","last_unstale":"2023-10-30T12:51:38.474826+0000","last_undegraded":"2023-10-30T12:51:38.474826+0000","last_fullsized":"2023-10-30T12:51:38.474826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'359","last_scrub_stamp":"2023-10-30T12:51:38.474746+0000","last_deep_scrub":"66'359","last_deep_scrub_stamp":"2023-10-30T12:51:38.474746+0000","last_clean_scrub_stamp":"2023-10-30T12:51:38.474746+0000","objects_scrubbed":4,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:46:34.826564+0000","scrub_duration":0.021991255000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":15324,"num_write":292,"num_write_kb":8361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10374,"num_omap_keys":19,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"66'558","reported_seq":2729,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:11.609600+0000","last_change":"2023-10-30T12:52:11.609787+0000","last_active":"2023-10-30T12:52:11.609600+0000","last_peered":"2023-10-30T12:52:11.609600+0000","last_clean":"2023-10-30T12:52:11.609600+0000","last_became_active":"2023-10-30T12:28:50.981193+0000","last_became_peered":"2023-10-30T12:28:50.981193+0000","last_unstale":"2023-10-30T12:52:11.609600+0000","last_undegraded":"2023-10-30T12:52:11.609600+0000","last_fullsized":"2023-10-30T12:52:11.609600+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'558","last_scrub_stamp":"2023-10-30T12:52:11.609388+0000","last_deep_scrub":"66'558","last_deep_scrub_stamp":"2023-10-30T12:52:11.609388+0000","last_clean_scrub_stamp":"2023-10-30T12:52:11.609388+0000","objects_scrubbed":24,"log_size":558,"log_dups_size":0,"ondisk_log_size":558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:35:18.683385+0000","scrub_duration":0.079771501999999994,"objects_trimmed":0,"snaptrim_duration":0.0096579700000000001,"stat_sum":{"num_bytes":12674006,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":50,"num_read_kb":4114,"num_write":492,"num_write_kb":18051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"66'98","reported_seq":180,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:44.901113+0000","last_change":"2023-10-30T12:51:44.901168+0000","last_active":"2023-10-30T12:51:44.901113+0000","last_peered":"2023-10-30T12:51:44.901113+0000","last_clean":"2023-10-30T12:51:44.901113+0000","last_became_active":"2023-10-30T12:28:50.367730+0000","last_became_peered":"2023-10-30T12:28:50.367730+0000","last_unstale":"2023-10-30T12:51:44.901113+0000","last_undegraded":"2023-10-30T12:51:44.901113+0000","last_fullsized":"2023-10-30T12:51:44.901113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'98","last_scrub_stamp":"2023-10-30T12:51:44.900977+0000","last_deep_scrub":"66'98","last_deep_scrub_stamp":"2023-10-30T12:51:44.900977+0000","last_clean_scrub_stamp":"2023-10-30T12:51:44.900977+0000","objects_scrubbed":4,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:46:56.082172+0000","scrub_duration":0.018334822000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":12292,"num_write":45,"num_write_kb":4186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8128,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"66'621","reported_seq":2799,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:23.291318+0000","last_change":"2023-10-30T12:52:23.291517+0000","last_active":"2023-10-30T12:52:23.291318+0000","last_peered":"2023-10-30T12:52:23.291318+0000","last_clean":"2023-10-30T12:52:23.291318+0000","last_became_active":"2023-10-30T12:28:50.966246+0000","last_became_peered":"2023-10-30T12:28:50.966246+0000","last_unstale":"2023-10-30T12:52:23.291318+0000","last_undegraded":"2023-10-30T12:52:23.291318+0000","last_fullsized":"2023-10-30T12:52:23.291318+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'621","last_scrub_stamp":"2023-10-30T12:52:23.291084+0000","last_deep_scrub":"66'621","last_deep_scrub_stamp":"2023-10-30T12:52:23.291084+0000","last_clean_scrub_stamp":"2023-10-30T12:52:23.291084+0000","objects_scrubbed":27,"log_size":621,"log_dups_size":0,"ondisk_log_size":621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:33:14.777415+0000","scrub_duration":0.089234725000000001,"objects_trimmed":0,"snaptrim_duration":0.007703601,"stat_sum":{"num_bytes":8520713,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":56,"num_read_kb":18,"num_write":541,"num_write_kb":11343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"66'87","reported_seq":172,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:42.434625+0000","last_change":"2023-10-30T12:51:42.434673+0000","last_active":"2023-10-30T12:51:42.434625+0000","last_peered":"2023-10-30T12:51:42.434625+0000","last_clean":"2023-10-30T12:51:42.434625+0000","last_became_active":"2023-10-30T12:28:50.393642+0000","last_became_peered":"2023-10-30T12:28:50.393642+0000","last_unstale":"2023-10-30T12:51:42.434625+0000","last_undegraded":"2023-10-30T12:51:42.434625+0000","last_fullsized":"2023-10-30T12:51:42.434625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'87","last_scrub_stamp":"2023-10-30T12:51:42.434545+0000","last_deep_scrub":"66'87","last_deep_scrub_stamp":"2023-10-30T12:51:42.434545+0000","last_clean_scrub_stamp":"2023-10-30T12:51:42.434545+0000","objects_scrubbed":5,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:27:29.743857+0000","scrub_duration":0.0090945590000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":15,"num_write_kb":101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17524,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"66'579","reported_seq":2744,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:56.560776+0000","last_change":"2023-10-30T12:51:56.560943+0000","last_active":"2023-10-30T12:51:56.560776+0000","last_peered":"2023-10-30T12:51:56.560776+0000","last_clean":"2023-10-30T12:51:56.560776+0000","last_became_active":"2023-10-30T12:28:50.976343+0000","last_became_peered":"2023-10-30T12:28:50.976343+0000","last_unstale":"2023-10-30T12:51:56.560776+0000","last_undegraded":"2023-10-30T12:51:56.560776+0000","last_fullsized":"2023-10-30T12:51:56.560776+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'579","last_scrub_stamp":"2023-10-30T12:51:56.560556+0000","last_deep_scrub":"66'579","last_deep_scrub_stamp":"2023-10-30T12:51:56.560556+0000","last_clean_scrub_stamp":"2023-10-30T12:51:56.560556+0000","objects_scrubbed":23,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:00:58.734907+0000","scrub_duration":0.074070374999999994,"objects_trimmed":0,"snaptrim_duration":0.0091875959999999993,"stat_sum":{"num_bytes":17078090,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":44,"num_read_kb":14,"num_write":517,"num_write_kb":22829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"66'119","reported_seq":231,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:44.641495+0000","last_change":"2023-10-30T12:51:44.641713+0000","last_active":"2023-10-30T12:51:44.641495+0000","last_peered":"2023-10-30T12:51:44.641495+0000","last_clean":"2023-10-30T12:51:44.641495+0000","last_became_active":"2023-10-30T12:28:41.201796+0000","last_became_peered":"2023-10-30T12:28:41.201796+0000","last_unstale":"2023-10-30T12:51:44.641495+0000","last_undegraded":"2023-10-30T12:51:44.641495+0000","last_fullsized":"2023-10-30T12:51:44.641495+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'119","last_scrub_stamp":"2023-10-30T12:51:44.641256+0000","last_deep_scrub":"66'119","last_deep_scrub_stamp":"2023-10-30T12:51:44.641256+0000","last_clean_scrub_stamp":"2023-10-30T12:51:44.641256+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:26:13.839326+0000","scrub_duration":0.013391731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"66'201","reported_seq":291,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:50.447569+0000","last_change":"2023-10-30T12:51:50.447661+0000","last_active":"2023-10-30T12:51:50.447569+0000","last_peered":"2023-10-30T12:51:50.447569+0000","last_clean":"2023-10-30T12:51:50.447569+0000","last_became_active":"2023-10-30T12:28:50.367593+0000","last_became_peered":"2023-10-30T12:28:50.367593+0000","last_unstale":"2023-10-30T12:51:50.447569+0000","last_undegraded":"2023-10-30T12:51:50.447569+0000","last_fullsized":"2023-10-30T12:51:50.447569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'201","last_scrub_stamp":"2023-10-30T12:51:50.447458+0000","last_deep_scrub":"66'201","last_deep_scrub_stamp":"2023-10-30T12:51:50.447458+0000","last_clean_scrub_stamp":"2023-10-30T12:51:50.447458+0000","objects_scrubbed":9,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:35:16.204652+0000","scrub_duration":0.025712215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194338,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":24,"num_read_kb":12326,"num_write":144,"num_write_kb":4223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":49160,"num_omap_keys":96,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"66'535","reported_seq":2690,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:48.599069+0000","last_change":"2023-10-30T12:51:48.599069+0000","last_active":"2023-10-30T12:51:48.599069+0000","last_peered":"2023-10-30T12:51:48.599069+0000","last_clean":"2023-10-30T12:51:48.599069+0000","last_became_active":"2023-10-30T12:29:32.997301+0000","last_became_peered":"2023-10-30T12:29:32.997301+0000","last_unstale":"2023-10-30T12:51:48.599069+0000","last_undegraded":"2023-10-30T12:51:48.599069+0000","last_fullsized":"2023-10-30T12:51:48.599069+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'535","last_scrub_stamp":"2023-10-30T12:51:48.598899+0000","last_deep_scrub":"66'535","last_deep_scrub_stamp":"2023-10-30T12:51:48.598899+0000","last_clean_scrub_stamp":"2023-10-30T12:51:48.598899+0000","objects_scrubbed":17,"log_size":535,"log_dups_size":0,"ondisk_log_size":535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:04:11.091469+0000","scrub_duration":0.019854914000000001,"objects_trimmed":0,"snaptrim_duration":0.001366412,"stat_sum":{"num_bytes":4237965,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":36,"num_read_kb":11,"num_write":481,"num_write_kb":13486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"66'504","reported_seq":600,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:47.588384+0000","last_change":"2023-10-30T12:51:47.588509+0000","last_active":"2023-10-30T12:51:47.588384+0000","last_peered":"2023-10-30T12:51:47.588384+0000","last_clean":"2023-10-30T12:51:47.588384+0000","last_became_active":"2023-10-30T12:28:50.391824+0000","last_became_peered":"2023-10-30T12:28:50.391824+0000","last_unstale":"2023-10-30T12:51:47.588384+0000","last_undegraded":"2023-10-30T12:51:47.588384+0000","last_fullsized":"2023-10-30T12:51:47.588384+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'504","last_scrub_stamp":"2023-10-30T12:51:47.588159+0000","last_deep_scrub":"66'504","last_deep_scrub_stamp":"2023-10-30T12:51:47.588159+0000","last_clean_scrub_stamp":"2023-10-30T12:51:47.588159+0000","objects_scrubbed":7,"log_size":504,"log_dups_size":0,"ondisk_log_size":504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:53:44.520327+0000","scrub_duration":0.072404271000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":32,"num_read_kb":50322,"num_write":431,"num_write_kb":16616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13208,"num_omap_keys":26,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"66'547","reported_seq":2698,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:16.385607+0000","last_change":"2023-10-30T12:52:16.385672+0000","last_active":"2023-10-30T12:52:16.385607+0000","last_peered":"2023-10-30T12:52:16.385607+0000","last_clean":"2023-10-30T12:52:16.385607+0000","last_became_active":"2023-10-30T12:28:50.952625+0000","last_became_peered":"2023-10-30T12:28:50.952625+0000","last_unstale":"2023-10-30T12:52:16.385607+0000","last_undegraded":"2023-10-30T12:52:16.385607+0000","last_fullsized":"2023-10-30T12:52:16.385607+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'547","last_scrub_stamp":"2023-10-30T12:52:16.385522+0000","last_deep_scrub":"66'547","last_deep_scrub_stamp":"2023-10-30T12:52:16.385522+0000","last_clean_scrub_stamp":"2023-10-30T12:52:16.385522+0000","objects_scrubbed":15,"log_size":547,"log_dups_size":0,"ondisk_log_size":547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:48:56.290695+0000","scrub_duration":0.061983638000000001,"objects_trimmed":0,"snaptrim_duration":0.0026695970000000001,"stat_sum":{"num_bytes":16953646,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":29,"num_read_kb":10,"num_write":468,"num_write_kb":20237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"66'605","reported_seq":920,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:48.574934+0000","last_change":"2023-10-30T12:51:35.405246+0000","last_active":"2023-10-30T12:51:48.574934+0000","last_peered":"2023-10-30T12:51:48.574934+0000","last_clean":"2023-10-30T12:51:48.574934+0000","last_became_active":"2023-10-30T12:28:50.367230+0000","last_became_peered":"2023-10-30T12:28:50.367230+0000","last_unstale":"2023-10-30T12:51:48.574934+0000","last_undegraded":"2023-10-30T12:51:48.574934+0000","last_fullsized":"2023-10-30T12:51:48.574934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'598","last_scrub_stamp":"2023-10-30T12:51:35.405133+0000","last_deep_scrub":"66'598","last_deep_scrub_stamp":"2023-10-30T12:51:35.405133+0000","last_clean_scrub_stamp":"2023-10-30T12:51:35.405133+0000","objects_scrubbed":7,"log_size":605,"log_dups_size":0,"ondisk_log_size":605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:34:01.194417+0000","scrub_duration":0.035040017999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12778646,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":251,"num_read_kb":13023,"num_write":542,"num_write_kb":12830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9828,"num_omap_keys":18,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"66'565","reported_seq":2734,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:57.726014+0000","last_change":"2023-10-30T12:51:57.726097+0000","last_active":"2023-10-30T12:51:57.726014+0000","last_peered":"2023-10-30T12:51:57.726014+0000","last_clean":"2023-10-30T12:51:57.726014+0000","last_became_active":"2023-10-30T12:28:50.928608+0000","last_became_peered":"2023-10-30T12:28:50.928608+0000","last_unstale":"2023-10-30T12:51:57.726014+0000","last_undegraded":"2023-10-30T12:51:57.726014+0000","last_fullsized":"2023-10-30T12:51:57.726014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'565","last_scrub_stamp":"2023-10-30T12:51:57.725915+0000","last_deep_scrub":"66'565","last_deep_scrub_stamp":"2023-10-30T12:51:57.725915+0000","last_clean_scrub_stamp":"2023-10-30T12:51:57.725915+0000","objects_scrubbed":21,"log_size":565,"log_dups_size":0,"ondisk_log_size":565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:24:52.304705+0000","scrub_duration":0.032239474999999997,"objects_trimmed":0,"snaptrim_duration":0.002074275,"stat_sum":{"num_bytes":8441122,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":47,"num_read_kb":21,"num_write":501,"num_write_kb":11799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"66'497","reported_seq":586,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:02.360999+0000","last_change":"2023-10-30T12:52:02.361114+0000","last_active":"2023-10-30T12:52:02.360999+0000","last_peered":"2023-10-30T12:52:02.360999+0000","last_clean":"2023-10-30T12:52:02.360999+0000","last_became_active":"2023-10-30T12:28:50.390864+0000","last_became_peered":"2023-10-30T12:28:50.390864+0000","last_unstale":"2023-10-30T12:52:02.360999+0000","last_undegraded":"2023-10-30T12:52:02.360999+0000","last_fullsized":"2023-10-30T12:52:02.360999+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'497","last_scrub_stamp":"2023-10-30T12:52:02.360776+0000","last_deep_scrub":"66'497","last_deep_scrub_stamp":"2023-10-30T12:52:02.360776+0000","last_clean_scrub_stamp":"2023-10-30T12:52:02.360776+0000","objects_scrubbed":4,"log_size":497,"log_dups_size":0,"ondisk_log_size":497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:40:59.407446+0000","scrub_duration":0.065243972999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":45183,"num_write":427,"num_write_kb":16635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"66'536","reported_seq":2693,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:45.113576+0000","last_change":"2023-10-30T12:51:45.113576+0000","last_active":"2023-10-30T12:51:45.113576+0000","last_peered":"2023-10-30T12:51:45.113576+0000","last_clean":"2023-10-30T12:51:45.113576+0000","last_became_active":"2023-10-30T12:28:50.949122+0000","last_became_peered":"2023-10-30T12:28:50.949122+0000","last_unstale":"2023-10-30T12:51:45.113576+0000","last_undegraded":"2023-10-30T12:51:45.113576+0000","last_fullsized":"2023-10-30T12:51:45.113576+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'536","last_scrub_stamp":"2023-10-30T12:51:45.113133+0000","last_deep_scrub":"66'536","last_deep_scrub_stamp":"2023-10-30T12:51:45.113133+0000","last_clean_scrub_stamp":"2023-10-30T12:51:45.113133+0000","objects_scrubbed":16,"log_size":536,"log_dups_size":0,"ondisk_log_size":536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:45:58.801601+0000","scrub_duration":0.060099805999999999,"objects_trimmed":0,"snaptrim_duration":0.004393348,"stat_sum":{"num_bytes":12684737,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":35,"num_read_kb":65,"num_write":464,"num_write_kb":14146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"66'267","reported_seq":346,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:36.014856+0000","last_change":"2023-10-30T12:51:36.014856+0000","last_active":"2023-10-30T12:51:36.014856+0000","last_peered":"2023-10-30T12:51:36.014856+0000","last_clean":"2023-10-30T12:51:36.014856+0000","last_became_active":"2023-10-30T12:28:50.358310+0000","last_became_peered":"2023-10-30T12:28:50.358310+0000","last_unstale":"2023-10-30T12:51:36.014856+0000","last_undegraded":"2023-10-30T12:51:36.014856+0000","last_fullsized":"2023-10-30T12:51:36.014856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'267","last_scrub_stamp":"2023-10-30T12:51:36.014733+0000","last_deep_scrub":"66'267","last_deep_scrub_stamp":"2023-10-30T12:51:36.014733+0000","last_clean_scrub_stamp":"2023-10-30T12:51:36.014733+0000","objects_scrubbed":2,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:22:01.486529+0000","scrub_duration":0.015176472999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":5758,"num_write":222,"num_write_kb":4242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7112,"num_omap_keys":14,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"66'554","reported_seq":2709,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:38.621754+0000","last_change":"2023-10-30T12:51:38.621754+0000","last_active":"2023-10-30T12:51:38.621754+0000","last_peered":"2023-10-30T12:51:38.621754+0000","last_clean":"2023-10-30T12:51:38.621754+0000","last_became_active":"2023-10-30T12:28:50.965626+0000","last_became_peered":"2023-10-30T12:28:50.965626+0000","last_unstale":"2023-10-30T12:51:38.621754+0000","last_undegraded":"2023-10-30T12:51:38.621754+0000","last_fullsized":"2023-10-30T12:51:38.621754+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'554","last_scrub_stamp":"2023-10-30T12:51:38.621402+0000","last_deep_scrub":"66'554","last_deep_scrub_stamp":"2023-10-30T12:51:38.621402+0000","last_clean_scrub_stamp":"2023-10-30T12:51:38.621402+0000","objects_scrubbed":15,"log_size":554,"log_dups_size":0,"ondisk_log_size":554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:32:24.000396+0000","scrub_duration":0.10283745599999999,"objects_trimmed":0,"snaptrim_duration":0.0079099229999999993,"stat_sum":{"num_bytes":29382851,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":33,"num_read_kb":4109,"num_write":477,"num_write_kb":37019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"66'404","reported_seq":485,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:53.568757+0000","last_change":"2023-10-30T12:51:53.568971+0000","last_active":"2023-10-30T12:51:53.568757+0000","last_peered":"2023-10-30T12:51:53.568757+0000","last_clean":"2023-10-30T12:51:53.568757+0000","last_became_active":"2023-10-30T12:28:50.395090+0000","last_became_peered":"2023-10-30T12:28:50.395090+0000","last_unstale":"2023-10-30T12:51:53.568757+0000","last_undegraded":"2023-10-30T12:51:53.568757+0000","last_fullsized":"2023-10-30T12:51:53.568757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'404","last_scrub_stamp":"2023-10-30T12:51:53.568460+0000","last_deep_scrub":"66'404","last_deep_scrub_stamp":"2023-10-30T12:51:53.568460+0000","last_clean_scrub_stamp":"2023-10-30T12:51:53.568460+0000","objects_scrubbed":5,"log_size":404,"log_dups_size":0,"ondisk_log_size":404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:27:52.316450+0000","scrub_duration":0.055684696999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582934,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":8195,"num_write":330,"num_write_kb":12489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12012,"num_omap_keys":22,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"66'568","reported_seq":2740,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:04.826459+0000","last_change":"2023-10-30T12:52:04.826539+0000","last_active":"2023-10-30T12:52:04.826459+0000","last_peered":"2023-10-30T12:52:04.826459+0000","last_clean":"2023-10-30T12:52:04.826459+0000","last_became_active":"2023-10-30T12:28:50.929337+0000","last_became_peered":"2023-10-30T12:28:50.929337+0000","last_unstale":"2023-10-30T12:52:04.826459+0000","last_undegraded":"2023-10-30T12:52:04.826459+0000","last_fullsized":"2023-10-30T12:52:04.826459+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'568","last_scrub_stamp":"2023-10-30T12:52:04.826355+0000","last_deep_scrub":"66'568","last_deep_scrub_stamp":"2023-10-30T12:52:04.826355+0000","last_clean_scrub_stamp":"2023-10-30T12:52:04.826355+0000","objects_scrubbed":17,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:47:52.867591+0000","scrub_duration":0.039134962000000002,"objects_trimmed":0,"snaptrim_duration":0.0023039890000000002,"stat_sum":{"num_bytes":14041074,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":50,"num_read_kb":18,"num_write":487,"num_write_kb":15488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"66'356","reported_seq":443,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:41.110896+0000","last_change":"2023-10-30T12:51:41.110963+0000","last_active":"2023-10-30T12:51:41.110896+0000","last_peered":"2023-10-30T12:51:41.110896+0000","last_clean":"2023-10-30T12:51:41.110896+0000","last_became_active":"2023-10-30T12:28:50.380247+0000","last_became_peered":"2023-10-30T12:28:50.380247+0000","last_unstale":"2023-10-30T12:51:41.110896+0000","last_undegraded":"2023-10-30T12:51:41.110896+0000","last_fullsized":"2023-10-30T12:51:41.110896+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'356","last_scrub_stamp":"2023-10-30T12:51:41.110806+0000","last_deep_scrub":"66'356","last_deep_scrub_stamp":"2023-10-30T12:51:41.110806+0000","last_clean_scrub_stamp":"2023-10-30T12:51:41.110806+0000","objects_scrubbed":5,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:36:44.682836+0000","scrub_duration":0.090051573999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":8272,"num_write":528,"num_write_kb":5259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18674,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"66'544","reported_seq":2695,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:16.880659+0000","last_change":"2023-10-30T12:52:16.880748+0000","last_active":"2023-10-30T12:52:16.880659+0000","last_peered":"2023-10-30T12:52:16.880659+0000","last_clean":"2023-10-30T12:52:16.880659+0000","last_became_active":"2023-10-30T12:28:50.973086+0000","last_became_peered":"2023-10-30T12:28:50.973086+0000","last_unstale":"2023-10-30T12:52:16.880659+0000","last_undegraded":"2023-10-30T12:52:16.880659+0000","last_fullsized":"2023-10-30T12:52:16.880659+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'544","last_scrub_stamp":"2023-10-30T12:52:16.880542+0000","last_deep_scrub":"66'544","last_deep_scrub_stamp":"2023-10-30T12:52:16.880542+0000","last_clean_scrub_stamp":"2023-10-30T12:52:16.880542+0000","objects_scrubbed":12,"log_size":544,"log_dups_size":0,"ondisk_log_size":544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:23:12.413779+0000","scrub_duration":0.056029834000000001,"objects_trimmed":0,"snaptrim_duration":0.0034133169999999999,"stat_sum":{"num_bytes":16807043,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":30,"num_read_kb":14,"num_write":492,"num_write_kb":22119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"66'275","reported_seq":358,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:18.562313+0000","last_change":"2023-10-30T12:51:43.564434+0000","last_active":"2023-10-30T12:52:18.562313+0000","last_peered":"2023-10-30T12:52:18.562313+0000","last_clean":"2023-10-30T12:52:18.562313+0000","last_became_active":"2023-10-30T12:28:50.385924+0000","last_became_peered":"2023-10-30T12:28:50.385924+0000","last_unstale":"2023-10-30T12:52:18.562313+0000","last_undegraded":"2023-10-30T12:52:18.562313+0000","last_fullsized":"2023-10-30T12:52:18.562313+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'261","last_scrub_stamp":"2023-10-30T12:51:43.564089+0000","last_deep_scrub":"66'261","last_deep_scrub_stamp":"2023-10-30T12:51:43.564089+0000","last_clean_scrub_stamp":"2023-10-30T12:51:43.564089+0000","objects_scrubbed":5,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:34:21.648949+0000","scrub_duration":0.022937485000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3368320,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":209,"num_write_kb":3423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13032,"num_omap_keys":26,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"66'541","reported_seq":2705,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:49.313126+0000","last_change":"2023-10-30T12:51:49.313203+0000","last_active":"2023-10-30T12:51:49.313126+0000","last_peered":"2023-10-30T12:51:49.313126+0000","last_clean":"2023-10-30T12:51:49.313126+0000","last_became_active":"2023-10-30T12:28:50.929259+0000","last_became_peered":"2023-10-30T12:28:50.929259+0000","last_unstale":"2023-10-30T12:51:49.313126+0000","last_undegraded":"2023-10-30T12:51:49.313126+0000","last_fullsized":"2023-10-30T12:51:49.313126+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'541","last_scrub_stamp":"2023-10-30T12:51:49.313036+0000","last_deep_scrub":"66'541","last_deep_scrub_stamp":"2023-10-30T12:51:49.313036+0000","last_clean_scrub_stamp":"2023-10-30T12:51:49.313036+0000","objects_scrubbed":17,"log_size":541,"log_dups_size":0,"ondisk_log_size":541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:00:33.907215+0000","scrub_duration":0.028828065,"objects_trimmed":0,"snaptrim_duration":0.0029033230000000002,"stat_sum":{"num_bytes":8423823,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":43,"num_read_kb":22,"num_write":477,"num_write_kb":11480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"66'379","reported_seq":470,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:46.586619+0000","last_change":"2023-10-30T12:51:46.586619+0000","last_active":"2023-10-30T12:51:46.586619+0000","last_peered":"2023-10-30T12:51:46.586619+0000","last_clean":"2023-10-30T12:51:46.586619+0000","last_became_active":"2023-10-30T12:28:50.385305+0000","last_became_peered":"2023-10-30T12:28:50.385305+0000","last_unstale":"2023-10-30T12:51:46.586619+0000","last_undegraded":"2023-10-30T12:51:46.586619+0000","last_fullsized":"2023-10-30T12:51:46.586619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'378","last_scrub_stamp":"2023-10-30T12:51:46.586293+0000","last_deep_scrub":"66'378","last_deep_scrub_stamp":"2023-10-30T12:51:46.586293+0000","last_clean_scrub_stamp":"2023-10-30T12:51:46.586293+0000","objects_scrubbed":5,"log_size":379,"log_dups_size":0,"ondisk_log_size":379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:03:32.538098+0000","scrub_duration":0.051218165000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":37322,"num_write":310,"num_write_kb":12446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":508,"num_omap_keys":1,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"66'562","reported_seq":2720,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:38.230220+0000","last_change":"2023-10-30T12:51:38.230463+0000","last_active":"2023-10-30T12:51:38.230220+0000","last_peered":"2023-10-30T12:51:38.230220+0000","last_clean":"2023-10-30T12:51:38.230220+0000","last_became_active":"2023-10-30T12:28:50.940846+0000","last_became_peered":"2023-10-30T12:28:50.940846+0000","last_unstale":"2023-10-30T12:51:38.230220+0000","last_undegraded":"2023-10-30T12:51:38.230220+0000","last_fullsized":"2023-10-30T12:51:38.230220+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'562","last_scrub_stamp":"2023-10-30T12:51:38.229988+0000","last_deep_scrub":"66'562","last_deep_scrub_stamp":"2023-10-30T12:51:38.229988+0000","last_clean_scrub_stamp":"2023-10-30T12:51:38.229988+0000","objects_scrubbed":21,"log_size":562,"log_dups_size":0,"ondisk_log_size":562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:31:03.842668+0000","scrub_duration":0.057637995999999997,"objects_trimmed":0,"snaptrim_duration":0.0046016829999999996,"stat_sum":{"num_bytes":12679067,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":37,"num_read_kb":7,"num_write":498,"num_write_kb":18971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"66'448","reported_seq":534,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:42.982397+0000","last_change":"2023-10-30T12:51:42.982473+0000","last_active":"2023-10-30T12:51:42.982397+0000","last_peered":"2023-10-30T12:51:42.982397+0000","last_clean":"2023-10-30T12:51:42.982397+0000","last_became_active":"2023-10-30T12:28:50.392068+0000","last_became_peered":"2023-10-30T12:28:50.392068+0000","last_unstale":"2023-10-30T12:51:42.982397+0000","last_undegraded":"2023-10-30T12:51:42.982397+0000","last_fullsized":"2023-10-30T12:51:42.982397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'448","last_scrub_stamp":"2023-10-30T12:51:42.982269+0000","last_deep_scrub":"66'448","last_deep_scrub_stamp":"2023-10-30T12:51:42.982269+0000","last_clean_scrub_stamp":"2023-10-30T12:51:42.982269+0000","objects_scrubbed":7,"log_size":448,"log_dups_size":0,"ondisk_log_size":448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:55:25.126896+0000","scrub_duration":0.032966065000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388698,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":19,"num_read_kb":11375,"num_write":387,"num_write_kb":8544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":27300,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"66'570","reported_seq":2735,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:49.808293+0000","last_change":"2023-10-30T12:51:49.808358+0000","last_active":"2023-10-30T12:51:49.808293+0000","last_peered":"2023-10-30T12:51:49.808293+0000","last_clean":"2023-10-30T12:51:49.808293+0000","last_became_active":"2023-10-30T12:28:50.931898+0000","last_became_peered":"2023-10-30T12:28:50.931898+0000","last_unstale":"2023-10-30T12:51:49.808293+0000","last_undegraded":"2023-10-30T12:51:49.808293+0000","last_fullsized":"2023-10-30T12:51:49.808293+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'570","last_scrub_stamp":"2023-10-30T12:51:49.808201+0000","last_deep_scrub":"66'570","last_deep_scrub_stamp":"2023-10-30T12:51:49.808201+0000","last_clean_scrub_stamp":"2023-10-30T12:51:49.808201+0000","objects_scrubbed":18,"log_size":570,"log_dups_size":0,"ondisk_log_size":570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:46:36.932799+0000","scrub_duration":0.022998979999999999,"objects_trimmed":0,"snaptrim_duration":0.003570148,"stat_sum":{"num_bytes":4263636,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":44,"num_read_kb":18,"num_write":507,"num_write_kb":10484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"66'557","reported_seq":639,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:42.047483+0000","last_change":"2023-10-30T12:51:42.047533+0000","last_active":"2023-10-30T12:51:42.047483+0000","last_peered":"2023-10-30T12:51:42.047483+0000","last_clean":"2023-10-30T12:51:42.047483+0000","last_became_active":"2023-10-30T12:28:50.364797+0000","last_became_peered":"2023-10-30T12:28:50.364797+0000","last_unstale":"2023-10-30T12:51:42.047483+0000","last_undegraded":"2023-10-30T12:51:42.047483+0000","last_fullsized":"2023-10-30T12:51:42.047483+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'557","last_scrub_stamp":"2023-10-30T12:51:42.047398+0000","last_deep_scrub":"66'557","last_deep_scrub_stamp":"2023-10-30T12:51:42.047398+0000","last_clean_scrub_stamp":"2023-10-30T12:51:42.047398+0000","objects_scrubbed":4,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:36:26.216418+0000","scrub_duration":0.054259416999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":10,"num_read_kb":8489,"num_write":691,"num_write_kb":13252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"66'568","reported_seq":2738,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:50.791335+0000","last_change":"2023-10-30T12:51:50.791392+0000","last_active":"2023-10-30T12:51:50.791335+0000","last_peered":"2023-10-30T12:51:50.791335+0000","last_clean":"2023-10-30T12:51:50.791335+0000","last_became_active":"2023-10-30T12:28:50.935657+0000","last_became_peered":"2023-10-30T12:28:50.935657+0000","last_unstale":"2023-10-30T12:51:50.791335+0000","last_undegraded":"2023-10-30T12:51:50.791335+0000","last_fullsized":"2023-10-30T12:51:50.791335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'568","last_scrub_stamp":"2023-10-30T12:51:50.791255+0000","last_deep_scrub":"66'568","last_deep_scrub_stamp":"2023-10-30T12:51:50.791255+0000","last_clean_scrub_stamp":"2023-10-30T12:51:50.791255+0000","objects_scrubbed":15,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:03:42.487468+0000","scrub_duration":0.039011377999999999,"objects_trimmed":0,"snaptrim_duration":0.0038543219999999999,"stat_sum":{"num_bytes":16829664,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":49,"num_read_kb":4112,"num_write":499,"num_write_kb":17598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"66'249","reported_seq":338,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:54.509810+0000","last_change":"2023-10-30T12:51:54.509947+0000","last_active":"2023-10-30T12:51:54.509810+0000","last_peered":"2023-10-30T12:51:54.509810+0000","last_clean":"2023-10-30T12:51:54.509810+0000","last_became_active":"2023-10-30T12:28:50.392807+0000","last_became_peered":"2023-10-30T12:28:50.392807+0000","last_unstale":"2023-10-30T12:51:54.509810+0000","last_undegraded":"2023-10-30T12:51:54.509810+0000","last_fullsized":"2023-10-30T12:51:54.509810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'249","last_scrub_stamp":"2023-10-30T12:51:54.509581+0000","last_deep_scrub":"66'249","last_deep_scrub_stamp":"2023-10-30T12:51:54.509581+0000","last_clean_scrub_stamp":"2023-10-30T12:51:54.509581+0000","objects_scrubbed":5,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:12:36.299114+0000","scrub_duration":0.039282915000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":24596,"num_write":178,"num_write_kb":8294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"66'524","reported_seq":2683,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:02.742094+0000","last_change":"2023-10-30T12:52:02.742242+0000","last_active":"2023-10-30T12:52:02.742094+0000","last_peered":"2023-10-30T12:52:02.742094+0000","last_clean":"2023-10-30T12:52:02.742094+0000","last_became_active":"2023-10-30T12:28:50.954291+0000","last_became_peered":"2023-10-30T12:28:50.954291+0000","last_unstale":"2023-10-30T12:52:02.742094+0000","last_undegraded":"2023-10-30T12:52:02.742094+0000","last_fullsized":"2023-10-30T12:52:02.742094+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'524","last_scrub_stamp":"2023-10-30T12:52:02.741848+0000","last_deep_scrub":"66'524","last_deep_scrub_stamp":"2023-10-30T12:52:02.741848+0000","last_clean_scrub_stamp":"2023-10-30T12:52:02.741848+0000","objects_scrubbed":19,"log_size":524,"log_dups_size":0,"ondisk_log_size":524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:24:29.376137+0000","scrub_duration":0.074825178000000006,"objects_trimmed":0,"snaptrim_duration":0.0047974480000000002,"stat_sum":{"num_bytes":12709282,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":38,"num_read_kb":15,"num_write":455,"num_write_kb":14651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"66'539","reported_seq":2709,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:18.285265+0000","last_change":"2023-10-30T12:52:18.285390+0000","last_active":"2023-10-30T12:52:18.285265+0000","last_peered":"2023-10-30T12:52:18.285265+0000","last_clean":"2023-10-30T12:52:18.285265+0000","last_became_active":"2023-10-30T12:28:50.962817+0000","last_became_peered":"2023-10-30T12:28:50.962817+0000","last_unstale":"2023-10-30T12:52:18.285265+0000","last_undegraded":"2023-10-30T12:52:18.285265+0000","last_fullsized":"2023-10-30T12:52:18.285265+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'539","last_scrub_stamp":"2023-10-30T12:52:18.285036+0000","last_deep_scrub":"66'539","last_deep_scrub_stamp":"2023-10-30T12:52:18.285036+0000","last_clean_scrub_stamp":"2023-10-30T12:52:18.285036+0000","objects_scrubbed":13,"log_size":539,"log_dups_size":0,"ondisk_log_size":539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:28:02.616414+0000","scrub_duration":0.052547446999999997,"objects_trimmed":0,"snaptrim_duration":0.0070898599999999999,"stat_sum":{"num_bytes":8448766,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":50,"num_read_kb":25,"num_write":483,"num_write_kb":14592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"66'176","reported_seq":261,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:51.555385+0000","last_change":"2023-10-30T12:51:51.555592+0000","last_active":"2023-10-30T12:51:51.555385+0000","last_peered":"2023-10-30T12:51:51.555385+0000","last_clean":"2023-10-30T12:51:51.555385+0000","last_became_active":"2023-10-30T12:28:50.401677+0000","last_became_peered":"2023-10-30T12:28:50.401677+0000","last_unstale":"2023-10-30T12:51:51.555385+0000","last_undegraded":"2023-10-30T12:51:51.555385+0000","last_fullsized":"2023-10-30T12:51:51.555385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'176","last_scrub_stamp":"2023-10-30T12:51:51.555145+0000","last_deep_scrub":"66'176","last_deep_scrub_stamp":"2023-10-30T12:51:51.555145+0000","last_clean_scrub_stamp":"2023-10-30T12:51:51.555145+0000","objects_scrubbed":3,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:07:33.229250+0000","scrub_duration":0.033175365999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":20498,"num_write":107,"num_write_kb":8247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"66'514","reported_seq":2678,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:31.213489+0000","last_change":"2023-10-30T12:52:31.213673+0000","last_active":"2023-10-30T12:52:31.213489+0000","last_peered":"2023-10-30T12:52:31.213489+0000","last_clean":"2023-10-30T12:52:31.213489+0000","last_became_active":"2023-10-30T12:28:50.967591+0000","last_became_peered":"2023-10-30T12:28:50.967591+0000","last_unstale":"2023-10-30T12:52:31.213489+0000","last_undegraded":"2023-10-30T12:52:31.213489+0000","last_fullsized":"2023-10-30T12:52:31.213489+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'514","last_scrub_stamp":"2023-10-30T12:52:31.213261+0000","last_deep_scrub":"66'514","last_deep_scrub_stamp":"2023-10-30T12:52:31.213261+0000","last_clean_scrub_stamp":"2023-10-30T12:52:31.213261+0000","objects_scrubbed":14,"log_size":514,"log_dups_size":0,"ondisk_log_size":514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:25:59.281273+0000","scrub_duration":0.057278369000000003,"objects_trimmed":0,"snaptrim_duration":0.0081968709999999997,"stat_sum":{"num_bytes":10185055,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":44,"num_read_kb":1610,"num_write":440,"num_write_kb":22911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"66'370","reported_seq":466,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:50.641824+0000","last_change":"2023-10-30T12:51:50.641945+0000","last_active":"2023-10-30T12:51:50.641824+0000","last_peered":"2023-10-30T12:51:50.641824+0000","last_clean":"2023-10-30T12:51:50.641824+0000","last_became_active":"2023-10-30T12:28:50.392021+0000","last_became_peered":"2023-10-30T12:28:50.392021+0000","last_unstale":"2023-10-30T12:51:50.641824+0000","last_undegraded":"2023-10-30T12:51:50.641824+0000","last_fullsized":"2023-10-30T12:51:50.641824+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'370","last_scrub_stamp":"2023-10-30T12:51:50.641619+0000","last_deep_scrub":"66'370","last_deep_scrub_stamp":"2023-10-30T12:51:50.641619+0000","last_clean_scrub_stamp":"2023-10-30T12:51:50.641619+0000","objects_scrubbed":8,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:05:24.267901+0000","scrub_duration":0.078330257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":57384,"num_write":298,"num_write_kb":16576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22858,"num_omap_keys":43,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"66'521","reported_seq":2687,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:00.262491+0000","last_change":"2023-10-30T12:52:00.262554+0000","last_active":"2023-10-30T12:52:00.262491+0000","last_peered":"2023-10-30T12:52:00.262491+0000","last_clean":"2023-10-30T12:52:00.262491+0000","last_became_active":"2023-10-30T12:28:50.967186+0000","last_became_peered":"2023-10-30T12:28:50.967186+0000","last_unstale":"2023-10-30T12:52:00.262491+0000","last_undegraded":"2023-10-30T12:52:00.262491+0000","last_fullsized":"2023-10-30T12:52:00.262491+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'521","last_scrub_stamp":"2023-10-30T12:52:00.262413+0000","last_deep_scrub":"66'521","last_deep_scrub_stamp":"2023-10-30T12:52:00.262413+0000","last_clean_scrub_stamp":"2023-10-30T12:52:00.262413+0000","objects_scrubbed":15,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:28:18.321174+0000","scrub_duration":0.025087834999999999,"objects_trimmed":0,"snaptrim_duration":0.003136738,"stat_sum":{"num_bytes":73381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":44,"num_read_kb":17,"num_write":453,"num_write_kb":3099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"66'231","reported_seq":316,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:37.827427+0000","last_change":"2023-10-30T12:51:37.827474+0000","last_active":"2023-10-30T12:51:37.827427+0000","last_peered":"2023-10-30T12:51:37.827427+0000","last_clean":"2023-10-30T12:51:37.827427+0000","last_became_active":"2023-10-30T12:28:50.362929+0000","last_became_peered":"2023-10-30T12:28:50.362929+0000","last_unstale":"2023-10-30T12:51:37.827427+0000","last_undegraded":"2023-10-30T12:51:37.827427+0000","last_fullsized":"2023-10-30T12:51:37.827427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'231","last_scrub_stamp":"2023-10-30T12:51:37.827339+0000","last_deep_scrub":"66'231","last_deep_scrub_stamp":"2023-10-30T12:51:37.827339+0000","last_clean_scrub_stamp":"2023-10-30T12:51:37.827339+0000","objects_scrubbed":4,"log_size":231,"log_dups_size":0,"ondisk_log_size":231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:06:46.012417+0000","scrub_duration":0.026390808000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":9,"num_read_kb":32768,"num_write":162,"num_write_kb":8341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":4,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"66'588","reported_seq":2740,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:11.804649+0000","last_change":"2023-10-30T12:52:11.804712+0000","last_active":"2023-10-30T12:52:11.804649+0000","last_peered":"2023-10-30T12:52:11.804649+0000","last_clean":"2023-10-30T12:52:11.804649+0000","last_became_active":"2023-10-30T12:28:50.937458+0000","last_became_peered":"2023-10-30T12:28:50.937458+0000","last_unstale":"2023-10-30T12:52:11.804649+0000","last_undegraded":"2023-10-30T12:52:11.804649+0000","last_fullsized":"2023-10-30T12:52:11.804649+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'588","last_scrub_stamp":"2023-10-30T12:52:11.804563+0000","last_deep_scrub":"66'588","last_deep_scrub_stamp":"2023-10-30T12:52:11.804563+0000","last_clean_scrub_stamp":"2023-10-30T12:52:11.804563+0000","objects_scrubbed":16,"log_size":588,"log_dups_size":0,"ondisk_log_size":588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:39:38.027390+0000","scrub_duration":0.058327041000000003,"objects_trimmed":0,"snaptrim_duration":0.003544045,"stat_sum":{"num_bytes":16852009,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":30,"num_read_kb":11,"num_write":518,"num_write_kb":19789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"66'450","reported_seq":554,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:34.515168+0000","last_change":"2023-10-30T12:51:34.515168+0000","last_active":"2023-10-30T12:51:34.515168+0000","last_peered":"2023-10-30T12:51:34.515168+0000","last_clean":"2023-10-30T12:51:34.515168+0000","last_became_active":"2023-10-30T12:28:50.386522+0000","last_became_peered":"2023-10-30T12:28:50.386522+0000","last_unstale":"2023-10-30T12:51:34.515168+0000","last_undegraded":"2023-10-30T12:51:34.515168+0000","last_fullsized":"2023-10-30T12:51:34.515168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'450","last_scrub_stamp":"2023-10-30T12:51:34.515038+0000","last_deep_scrub":"66'450","last_deep_scrub_stamp":"2023-10-30T12:51:34.515038+0000","last_clean_scrub_stamp":"2023-10-30T12:51:34.515038+0000","objects_scrubbed":7,"log_size":450,"log_dups_size":0,"ondisk_log_size":450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:32:31.022257+0000","scrub_duration":0.052865532999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583494,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":64,"num_read_kb":24634,"num_write":377,"num_write_kb":12490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":757298097,"num_objects":1192,"num_object_clones":0,"num_object_copies":2384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1192,"num_whiteouts":0,"num_read":2811,"num_read_kb":100236,"num_write":31900,"num_write_kb":1049744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1519566848,"data_stored":1514596194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36203,"ondisk_log_size":36203,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":305770548,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1441,"num_read_kb":712828,"num_write":10566,"num_write_kb":307330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":78,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":305728,"num_omap_keys":583,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":611704832,"data_stored":611541096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11973,"ondisk_log_size":11973,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542436,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":394260,"kb_used_data":322828,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93977580,"statfs":{"total":96636764160,"available":96233041920,"internally_reserved":0,"allocated":330575872,"data_stored":329409202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":73136587},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:52:25 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.77600000000000002,"15min":0.81399999999999995},"min":{"1min":0.504,"5min":0.45900000000000002,"15min":0.40000000000000002},"max":{"1min":1.266,"5min":1.266,"15min":1.595},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.78500000000000003,"15min":0.84099999999999997},"min":{"1min":0.502,"5min":0.49099999999999999,"15min":0.40999999999999998},"max":{"1min":1.1279999999999999,"5min":1.302,"15min":1.4830000000000001},"last":0.85599999999999998}]},{"osd":1,"last update":"Mon Oct 30 12:52:25 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.83799999999999997,"15min":0.88400000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.441},"max":{"1min":1.089,"5min":3.4329999999999998,"15min":3.4329999999999998},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.81599999999999995,"15min":0.878},"min":{"1min":0.55300000000000005,"5min":0.505,"15min":0.505},"max":{"1min":1.117,"5min":1.268,"15min":1.3919999999999999},"last":0.90000000000000002}]},{"osd":2,"last update":"Mon Oct 30 12:52:25 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.86799999999999999,"15min":0.91900000000000004},"min":{"1min":0.59099999999999997,"5min":0.55100000000000005,"15min":0.45900000000000002},"max":{"1min":1.153,"5min":3.4820000000000002,"15min":3.4820000000000002},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.876,"15min":0.83899999999999997},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.503},"max":{"1min":1.1930000000000001,"5min":1.4570000000000001,"15min":1.6639999999999999},"last":0.80000000000000004}]},{"osd":3,"last update":"Mon Oct 30 12:52:30 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":0.99299999999999999,"15min":0.97599999999999998},"min":{"1min":0.79000000000000004,"5min":0.60999999999999999,"15min":0.45500000000000002},"max":{"1min":1.25,"5min":3.5190000000000001,"15min":3.5190000000000001},"last":1.002},{"interface":"front","average":{"1min":1.0009999999999999,"5min":0.96499999999999997,"15min":0.99099999999999999},"min":{"1min":0.67500000000000004,"5min":0.60999999999999999,"15min":0.53300000000000003},"max":{"1min":1.298,"5min":1.4259999999999999,"15min":1.5880000000000001},"last":0.97699999999999998}]},{"osd":4,"last update":"Mon Oct 30 12:52:25 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.95899999999999996,"15min":0.998},"min":{"1min":0.70499999999999996,"5min":0.42499999999999999,"15min":0.34399999999999997},"max":{"1min":1.4339999999999999,"5min":1.542,"15min":1.7350000000000001},"last":1.103},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1,"15min":1.0449999999999999},"min":{"1min":0.76400000000000001,"5min":0.45000000000000001,"15min":0.33800000000000002},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.746},"last":1.0609999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:52:25 2023","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0329999999999999,"15min":0.96399999999999997},"min":{"1min":0.83499999999999996,"5min":0.53700000000000003,"15min":0.43099999999999999},"max":{"1min":1.5640000000000001,"5min":1.6160000000000001,"15min":1.7549999999999999},"last":1.028},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.1180000000000001,"15min":1.083},"min":{"1min":0.89000000000000001,"5min":0.57899999999999996,"15min":0.35999999999999999},"max":{"1min":1.379,"5min":1.643,"15min":2.1600000000000001},"last":1.139}]},{"osd":6,"last update":"Mon Oct 30 12:52:25 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.129,"15min":1.131},"min":{"1min":0.73999999999999999,"5min":0.71299999999999997,"15min":0.39000000000000001},"max":{"1min":1.5149999999999999,"5min":1.5609999999999999,"15min":2.3809999999999998},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.119,"15min":1.165},"min":{"1min":0.93000000000000005,"5min":0.56299999999999994,"15min":0.47199999999999998},"max":{"1min":1.637,"5min":3.8140000000000001,"15min":3.8140000000000001},"last":0.76200000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542435,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302480,"kb_used_data":235272,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94069360,"statfs":{"total":96636764160,"available":96327024640,"internally_reserved":0,"allocated":240918528,"data_stored":240057681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":68811208},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:52:28 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.83399999999999996,"15min":0.85499999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.436},"max":{"1min":1.256,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.75900000000000001,"15min":0.78000000000000003},"min":{"1min":0.434,"5min":0.43099999999999999,"15min":0.42999999999999999},"max":{"1min":1.0309999999999999,"5min":1.0780000000000001,"15min":1.214},"last":0.622}]},{"osd":1,"last update":"Mon Oct 30 12:51:28 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.82099999999999995,"15min":0.81599999999999995},"min":{"1min":0.45000000000000001,"5min":0.42299999999999999,"15min":0.317},"max":{"1min":1.23,"5min":1.2809999999999999,"15min":1.3140000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.85199999999999998,"15min":0.876},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.40999999999999998},"max":{"1min":1.1399999999999999,"5min":1.2869999999999999,"15min":1.423},"last":0.85799999999999998}]},{"osd":2,"last update":"Mon Oct 30 12:52:28 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.89300000000000002,"15min":0.95999999999999996},"min":{"1min":0.58399999999999996,"5min":0.495,"15min":0.495},"max":{"1min":1.085,"5min":1.373,"15min":4.3360000000000003},"last":0.752},{"interface":"front","average":{"1min":1.044,"5min":0.98799999999999999,"15min":1.016},"min":{"1min":0.65100000000000002,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.409,"5min":1.502,"15min":1.5629999999999999},"last":0.98499999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:52:28 2023","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":0.98699999999999999,"15min":1.042},"min":{"1min":0.78100000000000003,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.5009999999999999,"5min":1.5049999999999999,"15min":1.6459999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":1.012,"5min":1.0409999999999999,"15min":1.095},"min":{"1min":0.73499999999999999,"5min":0.63300000000000001,"15min":0.59699999999999998},"max":{"1min":1.573,"5min":1.573,"15min":1.5800000000000001},"last":0.90100000000000002}]},{"osd":4,"last update":"Mon Oct 30 12:52:28 2023","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.127,"15min":1.087},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.55700000000000005},"max":{"1min":1.6599999999999999,"5min":1.6599999999999999,"15min":1.7649999999999999},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.1910000000000001,"5min":0.95099999999999996,"15min":0.96999999999999997},"min":{"1min":0.752,"5min":0.47199999999999998,"15min":0.42099999999999999},"max":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":1.881},"last":1.127}]},{"osd":5,"last update":"Mon Oct 30 12:52:28 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.006,"15min":1.0860000000000001},"min":{"1min":0.65200000000000002,"5min":0.53200000000000003,"15min":0.48399999999999999},"max":{"1min":1.6990000000000001,"5min":1.6990000000000001,"15min":2.2770000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":1.016,"5min":1.1419999999999999,"15min":1.069},"min":{"1min":0.69099999999999995,"5min":0.66800000000000004,"15min":0.46999999999999997},"max":{"1min":1.208,"5min":1.6950000000000001,"15min":4.7850000000000001},"last":1.038}]},{"osd":7,"last update":"Mon Oct 30 12:52:28 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":1.0209999999999999,"15min":0.998},"min":{"1min":0.64200000000000002,"5min":0.64100000000000001,"15min":0.52300000000000002},"max":{"1min":1.2090000000000001,"5min":1.5609999999999999,"15min":1.825},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.071,"5min":1.1160000000000001,"15min":1.0700000000000001},"min":{"1min":0.76000000000000001,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.337,"5min":1.6899999999999999,"15min":1.895},"last":1.099}]}]},{"osd":5,"up_from":14,"seq":60129542436,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":342544,"kb_used_data":271112,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94029296,"statfs":{"total":96636764160,"available":96285999104,"internally_reserved":0,"allocated":277618688,"data_stored":276486603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":1.7969999999999999,"5min":0.99099999999999999,"15min":0.86399999999999999},"min":{"1min":0.53500000000000003,"5min":0.33900000000000002,"15min":0.29399999999999998},"max":{"1min":19.593,"5min":19.593,"15min":19.593},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.80400000000000005,"15min":0.80300000000000005},"min":{"1min":0.59299999999999997,"5min":0.39200000000000002,"15min":0.33800000000000002},"max":{"1min":1.121,"5min":1.3009999999999999,"15min":1.8759999999999999},"last":0.97899999999999998}]},{"osd":1,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.83599999999999997,"15min":0.83999999999999997},"min":{"1min":0.59299999999999997,"5min":0.40999999999999998,"15min":0.38900000000000001},"max":{"1min":1.1859999999999999,"5min":1.2909999999999999,"15min":3.0190000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.89100000000000001,"15min":0.85899999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.373},"max":{"1min":1.1870000000000001,"5min":1.276,"15min":1.4870000000000001},"last":0.63100000000000001}]},{"osd":2,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.90100000000000002,"15min":0.84999999999999998},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.38400000000000001},"max":{"1min":1.306,"5min":1.306,"15min":2.2949999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":1.954,"5min":1.046,"15min":0.94599999999999995},"min":{"1min":0.64000000000000001,"5min":0.434,"15min":0.434},"max":{"1min":19.640000000000001,"5min":19.640000000000001,"15min":19.640000000000001},"last":0.78500000000000003}]},{"osd":3,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.92200000000000004,"15min":0.93200000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.2410000000000001,"5min":1.3999999999999999,"15min":1.756},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.0660000000000001,"5min":0.96699999999999997,"15min":0.95399999999999996},"min":{"1min":0.64900000000000002,"5min":0.498,"15min":0.41399999999999998},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":2.3679999999999999},"last":0.83399999999999996}]},{"osd":4,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":2.0680000000000001,"5min":1.099,"15min":0.98699999999999999},"min":{"1min":0.53700000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":19.675999999999998,"5min":19.675999999999998,"15min":19.675999999999998},"last":1.0349999999999999},{"interface":"front","average":{"1min":2.2909999999999999,"5min":1.341,"15min":1.173},"min":{"1min":0.72499999999999998,"5min":0.59499999999999997,"15min":0.50900000000000001},"max":{"1min":20.036999999999999,"5min":20.036999999999999,"15min":20.036999999999999},"last":1.0920000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":2.3090000000000002,"5min":1.304,"15min":1.129},"min":{"1min":0.88200000000000001,"5min":0.45800000000000002,"15min":0.33600000000000002},"max":{"1min":21.526,"5min":21.526,"15min":21.526},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.103,"5min":1.079,"15min":1.1020000000000001},"min":{"1min":0.46400000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.387,"5min":1.47,"15min":3.371},"last":1.0620000000000001}]},{"osd":7,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.1060000000000001,"15min":1.0840000000000001},"min":{"1min":0.57599999999999996,"5min":0.51400000000000001,"15min":0.47299999999999998},"max":{"1min":1.7569999999999999,"5min":1.7569999999999999,"15min":3.5310000000000001},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.236,"15min":1.232},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.903,"5min":1.903,"15min":3.6930000000000001},"last":1.01}]}]},{"osd":4,"up_from":14,"seq":60129542435,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369460,"kb_used_data":293868,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94002380,"statfs":{"total":96636764160,"available":96258437120,"internally_reserved":0,"allocated":300920832,"data_stored":299840560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.80100000000000005,"15min":0.92600000000000005},"min":{"1min":0.46800000000000003,"5min":0.433,"15min":0.31},"max":{"1min":1.159,"5min":1.2509999999999999,"15min":11.273},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.91800000000000004,"15min":0.98299999999999998},"min":{"1min":0.63600000000000001,"5min":0.53900000000000003,"15min":0.44},"max":{"1min":1.3120000000000001,"5min":1.4590000000000001,"15min":7.1289999999999996},"last":0.85799999999999998}]},{"osd":1,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.78100000000000003,"15min":0.89300000000000002},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0229999999999999,"5min":1.2230000000000001,"15min":6.9199999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.83899999999999997,"15min":0.89200000000000002},"min":{"1min":0.52200000000000002,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.1160000000000001,"5min":1.246,"15min":4.75},"last":0.68100000000000005}]},{"osd":2,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.86199999999999999,"15min":0.85099999999999998},"min":{"1min":0.56200000000000006,"5min":0.51700000000000002,"15min":0.36899999999999999},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":2.6970000000000001},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.91100000000000003,"15min":1.0620000000000001},"min":{"1min":0.66900000000000004,"5min":0.54400000000000004,"15min":0.52900000000000003},"max":{"1min":1.1299999999999999,"5min":1.361,"15min":18.917999999999999},"last":0.79000000000000004}]},{"osd":3,"last update":"Mon Oct 30 12:52:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":0.96099999999999997,"15min":1.006},"min":{"1min":0.67700000000000005,"5min":0.60799999999999998,"15min":0.51400000000000001},"max":{"1min":1.236,"5min":1.284,"15min":7.1879999999999997},"last":1.1200000000000001},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.86199999999999999,"15min":0.98999999999999999},"min":{"1min":0.58699999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.1040000000000001,"5min":1.343,"15min":8.3360000000000003},"last":0.72599999999999998}]},{"osd":5,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.86899999999999999,"15min":0.996},"min":{"1min":0.57599999999999996,"5min":0.56299999999999994,"15min":0.54700000000000004},"max":{"1min":1.026,"5min":1.2509999999999999,"15min":6.0759999999999996},"last":1.071},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.113,"15min":1.1859999999999999},"min":{"1min":0.95399999999999996,"5min":0.77400000000000002,"15min":0.52600000000000002},"max":{"1min":1.3660000000000001,"5min":1.4810000000000001,"15min":18.994},"last":1.202}]},{"osd":6,"last update":"Mon Oct 30 12:52:26 2023","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.099,"15min":1.099},"min":{"1min":0.77400000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.381,"5min":1.429,"15min":6.1580000000000004},"last":1.2430000000000001},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.96999999999999997,"15min":1.1220000000000001},"min":{"1min":0.66900000000000004,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.179,"5min":1.524,"15min":7.0549999999999997},"last":1.159}]},{"osd":7,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.056,"15min":1.2529999999999999},"min":{"1min":0.83799999999999997,"5min":0.56399999999999995,"15min":0.50700000000000001},"max":{"1min":1.2749999999999999,"5min":1.4850000000000001,"15min":18.015999999999998},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.1579999999999999,"15min":1.2769999999999999},"min":{"1min":0.92100000000000004,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.4339999999999999,"5min":1.4570000000000001,"15min":19.052},"last":1.3260000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542436,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391212,"kb_used_data":310052,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93980628,"statfs":{"total":96636764160,"available":96236163072,"internally_reserved":0,"allocated":317493248,"data_stored":316374505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":83098056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:52:26 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.40500000000000003,"15min":0.41699999999999998},"min":{"1min":0.31900000000000001,"5min":0.23699999999999999,"15min":0.23400000000000001},"max":{"1min":0.61799999999999999,"5min":0.77900000000000003,"15min":1.6899999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52300000000000002,"15min":0.53400000000000003},"min":{"1min":0.40500000000000003,"5min":0.34300000000000003,"15min":0.32700000000000001},"max":{"1min":0.71699999999999997,"5min":0.871,"15min":1.728},"last":0.59499999999999997}]},{"osd":1,"last update":"Mon Oct 30 12:52:26 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.438,"15min":0.432},"min":{"1min":0.308,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":1.2909999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.501,"15min":0.50900000000000001},"min":{"1min":0.36299999999999999,"5min":0.31900000000000001,"15min":0.29499999999999998},"max":{"1min":0.73199999999999998,"5min":0.80600000000000005,"15min":1.3260000000000001},"last":0.51400000000000001}]},{"osd":2,"last update":"Mon Oct 30 12:52:26 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.504,"15min":0.501},"min":{"1min":0.35199999999999998,"5min":0.30099999999999999,"15min":0.29299999999999998},"max":{"1min":0.69999999999999996,"5min":0.78600000000000003,"15min":1.099},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56200000000000006,"15min":0.56399999999999995},"min":{"1min":0.41099999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.755,"5min":0.82599999999999996,"15min":1.2509999999999999},"last":0.57499999999999996}]},{"osd":4,"last update":"Mon Oct 30 12:52:26 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.86099999999999999,"15min":0.83699999999999997},"min":{"1min":0.69299999999999995,"5min":0.55400000000000005,"15min":0.40400000000000003},"max":{"1min":1.1679999999999999,"5min":1.2310000000000001,"15min":2.202},"last":0.755},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.84299999999999997,"15min":0.86899999999999999},"min":{"1min":0.78100000000000003,"5min":0.48499999999999999,"15min":0.46800000000000003},"max":{"1min":1.0669999999999999,"5min":1.1950000000000001,"15min":2.4340000000000002},"last":0.80300000000000005}]},{"osd":5,"last update":"Mon Oct 30 12:52:26 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.81599999999999995,"15min":0.86199999999999999},"min":{"1min":0.58399999999999996,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.0649999999999999,"5min":1.169,"15min":14.103999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.873,"15min":0.872},"min":{"1min":0.75800000000000001,"5min":0.55000000000000004,"15min":0.499},"max":{"1min":1.0469999999999999,"5min":1.2170000000000001,"15min":4.4859999999999998},"last":0.85499999999999998}]},{"osd":6,"last update":"Mon Oct 30 12:52:26 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.84299999999999997,"15min":0.83399999999999996},"min":{"1min":0.57399999999999995,"5min":0.56999999999999995,"15min":0.53100000000000003},"max":{"1min":1.123,"5min":1.2,"15min":2.2330000000000001},"last":0.878},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.90500000000000003,"15min":0.83799999999999997},"min":{"1min":0.79800000000000004,"5min":0.57499999999999996,"15min":0.45100000000000001},"max":{"1min":1.0800000000000001,"5min":1.2609999999999999,"15min":4.4409999999999998},"last":0.91900000000000004}]},{"osd":7,"last update":"Mon Oct 30 12:52:26 2023","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.88200000000000001,"15min":0.86899999999999999},"min":{"1min":0.68100000000000005,"5min":0.59599999999999997,"15min":0.57099999999999995},"max":{"1min":1.0980000000000001,"5min":1.1499999999999999,"15min":3.5459999999999998},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.89400000000000002,"15min":0.85599999999999998},"min":{"1min":0.64100000000000001,"5min":0.57399999999999995,"15min":0.45700000000000002},"max":{"1min":1.2090000000000001,"5min":1.2230000000000001,"15min":6.1360000000000001},"last":0.95399999999999996}]}]},{"osd":2,"up_from":14,"seq":60129542436,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":200604,"kb_used_data":143060,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94171236,"statfs":{"total":96636764160,"available":96431345664,"internally_reserved":0,"allocated":146493440,"data_stored":145675964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.442,"15min":0.441},"min":{"1min":0.28000000000000003,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.55300000000000005,"5min":0.69399999999999995,"15min":0.81200000000000006},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.504,"5min":0.502,"15min":0.51100000000000001},"min":{"1min":0.39900000000000002,"5min":0.36099999999999999,"15min":0.31900000000000001},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":1.298},"last":0.64600000000000002}]},{"osd":1,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.47499999999999998},"min":{"1min":0.315,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.57699999999999996,"5min":0.64800000000000002,"15min":0.83699999999999997},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.51800000000000002,"15min":0.501},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.29999999999999999},"max":{"1min":0.61799999999999999,"5min":0.74099999999999999,"15min":1.2250000000000001},"last":0.54800000000000004}]},{"osd":3,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.48299999999999998,"15min":0.54200000000000004},"min":{"1min":0.33500000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.752,"5min":0.752,"15min":1.673},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57299999999999995,"15min":0.57099999999999995},"min":{"1min":0.47299999999999998,"5min":0.40200000000000002,"15min":0.35699999999999998},"max":{"1min":0.72299999999999998,"5min":0.79100000000000004,"15min":1.597},"last":0.625}]},{"osd":4,"last update":"Mon Oct 30 12:52:21 2023","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.75700000000000001,"15min":0.82199999999999995},"min":{"1min":0.69599999999999995,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.038,"5min":1.038,"15min":1.377},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.876,"5min":0.77800000000000002,"15min":0.79500000000000004},"min":{"1min":0.74099999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.4159999999999999},"last":1.022}]},{"osd":5,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.84899999999999998,"15min":0.84399999999999997},"min":{"1min":0.60199999999999998,"5min":0.502,"15min":0.502},"max":{"1min":1.0900000000000001,"5min":1.113,"15min":1.78},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.91800000000000004,"15min":0.91400000000000003},"min":{"1min":0.75,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.135,"5min":1.1379999999999999,"15min":1.7430000000000001},"last":1.044}]},{"osd":6,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.85899999999999999,"15min":0.86099999999999999},"min":{"1min":0.66000000000000003,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.1659999999999999,"5min":1.1759999999999999,"15min":1.3799999999999999},"last":1.077},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.91400000000000003,"15min":0.86899999999999999},"min":{"1min":0.65600000000000003,"5min":0.58299999999999996,"15min":0.53300000000000003},"max":{"1min":1.081,"5min":1.1799999999999999,"15min":1.8109999999999999},"last":1.0029999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.84399999999999997,"15min":0.83099999999999996},"min":{"1min":0.69899999999999995,"5min":0.58299999999999996,"15min":0.48799999999999999},"max":{"1min":1.1379999999999999,"5min":1.1599999999999999,"15min":1.4159999999999999},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.872,"15min":0.83899999999999997},"min":{"1min":0.69399999999999995,"5min":0.51800000000000002,"15min":0.50800000000000001},"max":{"1min":1.238,"5min":1.238,"15min":1.3049999999999999},"last":0.98799999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542436,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":279036,"kb_used_data":233972,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94092804,"statfs":{"total":96636764160,"available":96351031296,"internally_reserved":0,"allocated":239587328,"data_stored":238747727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:51:38 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.41399999999999998,"15min":0.45700000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.24299999999999999},"max":{"1min":0.65100000000000002,"5min":0.65200000000000002,"15min":11.958},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.497,"15min":0.55500000000000005},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.751,"5min":0.751,"15min":12.077},"last":0.59199999999999997}]},{"osd":2,"last update":"Mon Oct 30 12:51:38 2023","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.46200000000000002,"15min":0.45800000000000002},"min":{"1min":0.32700000000000001,"5min":0.29699999999999999,"15min":0.26400000000000001},"max":{"1min":0.54100000000000004,"5min":1.121,"15min":1.623},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.53200000000000003,"15min":0.53100000000000003},"min":{"1min":0.37,"5min":0.34200000000000003,"15min":0.31900000000000001},"max":{"1min":0.84499999999999997,"5min":1.1699999999999999,"15min":1.5389999999999999},"last":0.64000000000000001}]},{"osd":3,"last update":"Mon Oct 30 12:51:38 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56499999999999995,"15min":0.56599999999999995},"min":{"1min":0.40999999999999998,"5min":0.378,"15min":0.31},"max":{"1min":0.72099999999999997,"5min":1.337,"15min":3.694},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.504,"15min":0.53700000000000003},"min":{"1min":0.308,"5min":0.307,"15min":0.307},"max":{"1min":0.69699999999999995,"5min":0.77200000000000002,"15min":13.199999999999999},"last":0.70599999999999996}]},{"osd":4,"last update":"Mon Oct 30 12:51:38 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.82299999999999995,"15min":0.82699999999999996},"min":{"1min":0.54700000000000004,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.94599999999999995,"5min":1.3140000000000001,"15min":2.419},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.873,"15min":0.86499999999999999},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.52100000000000002},"max":{"1min":1.012,"5min":1.103,"15min":2.282},"last":0.95299999999999996}]},{"osd":5,"last update":"Mon Oct 30 12:51:38 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.79800000000000004,"15min":0.81399999999999995},"min":{"1min":0.54100000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.95799999999999996,"5min":1.343,"15min":1.343},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.751,"5min":0.79000000000000004,"15min":0.84799999999999998},"min":{"1min":0.51700000000000002,"5min":0.50800000000000001,"15min":0.44900000000000001},"max":{"1min":0.98999999999999999,"5min":1.2809999999999999,"15min":1.29},"last":0.93100000000000005}]},{"osd":6,"last update":"Mon Oct 30 12:51:38 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.83399999999999996,"15min":0.80000000000000004},"min":{"1min":0.56200000000000006,"5min":0.52800000000000002,"15min":0.51200000000000001},"max":{"1min":1.1279999999999999,"5min":1.371,"15min":1.371},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.91200000000000003,"15min":0.86699999999999999},"min":{"1min":0.628,"5min":0.62,"15min":0.498},"max":{"1min":1.173,"5min":1.417,"15min":2.464},"last":0.98499999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:51:38 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.80300000000000005,"15min":0.82399999999999995},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.075,"5min":1.1659999999999999,"15min":2.5169999999999999},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.88500000000000001,"15min":0.90100000000000002},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.1140000000000001,"5min":1.133,"15min":2.5510000000000002},"last":1.006}]}]},{"osd":0,"up_from":14,"seq":60129542436,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":349720,"kb_used_data":278288,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94022120,"statfs":{"total":96636764160,"available":96278650880,"internally_reserved":0,"allocated":284966912,"data_stored":283825495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 12:52:30 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.46100000000000002,"15min":0.45200000000000001},"min":{"1min":0.27500000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.93700000000000006},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52100000000000002,"15min":0.51100000000000001},"min":{"1min":0.40200000000000002,"5min":0.35299999999999998,"15min":0.26500000000000001},"max":{"1min":0.72099999999999997,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.52000000000000002}]},{"osd":2,"last update":"Mon Oct 30 12:52:30 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46100000000000002,"15min":0.499},"min":{"1min":0.30099999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.97199999999999998},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.53900000000000003},"min":{"1min":0.42399999999999999,"5min":0.36799999999999999,"15min":0.27400000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":1.0149999999999999},"last":0.497}]},{"osd":3,"last update":"Mon Oct 30 12:52:30 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.48499999999999999,"15min":0.50800000000000001},"min":{"1min":0.35199999999999998,"5min":0.30499999999999999,"15min":0.29499999999999998},"max":{"1min":0.82799999999999996,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.37},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51100000000000001,"15min":0.499},"min":{"1min":0.313,"5min":0.313,"15min":0.31},"max":{"1min":0.76000000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.44900000000000001}]},{"osd":4,"last update":"Mon Oct 30 12:52:30 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.86399999999999999,"15min":0.85299999999999998},"min":{"1min":0.58199999999999996,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.2250000000000001,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.874,"15min":0.86099999999999999},"min":{"1min":0.71699999999999997,"5min":0.52600000000000002,"15min":0.48299999999999998},"max":{"1min":1.248,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.84099999999999997}]},{"osd":5,"last update":"Mon Oct 30 12:52:30 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.80300000000000005,"15min":0.82799999999999996},"min":{"1min":0.55700000000000005,"5min":0.55200000000000005,"15min":0.46200000000000002},"max":{"1min":1.079,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.85299999999999998,"15min":0.86499999999999999},"min":{"1min":0.67800000000000005,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.1779999999999999,"5min":1.889,"15min":1.889},"last":0.88800000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:52:30 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.81299999999999994,"15min":0.79800000000000004},"min":{"1min":0.69399999999999995,"5min":0.56299999999999994,"15min":0.52200000000000002},"max":{"1min":1.294,"5min":1.294,"15min":2.6480000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.81000000000000005,"15min":0.81799999999999995},"min":{"1min":0.621,"5min":0.57699999999999996,"15min":0.46999999999999997},"max":{"1min":1.0389999999999999,"5min":1.1559999999999999,"15min":5.0099999999999998},"last":0.69999999999999996}]},{"osd":7,"last update":"Mon Oct 30 12:52:30 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.879,"15min":0.88600000000000001},"min":{"1min":0.63400000000000001,"5min":0.59699999999999998,"15min":0.55000000000000004},"max":{"1min":1.1539999999999999,"5min":1.401,"15min":3.3359999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.94199999999999995,"15min":0.94899999999999995},"min":{"1min":0.70099999999999996,"5min":0.65100000000000002,"15min":0.59699999999999998},"max":{"1min":1.2150000000000001,"5min":1.3500000000000001,"15min":3.3580000000000001},"last":0.85399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62930944,"data_stored":62915220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":83910656,"data_stored":83887024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":37969920,"data_stored":37945198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":83726336,"data_stored":83700311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":76107776,"data_stored":76093397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":62935040,"data_stored":62915938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":91684864,"data_stored":91663056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":112439296,"data_stored":112420952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":220680192,"data_stored":219932830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":154910720,"data_stored":154473007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":107761664,"data_stored":107343689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":233000960,"data_stored":232286498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":224047104,"data_stored":223359467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":213921792,"data_stored":213183588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":148467712,"data_stored":148006929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":216776704,"data_stored":216010186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:52:39.090 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2023-10-30T12:28:50.333136+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=50, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=51, tm_sec=32, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T12:52:39.091 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T12:52:48.945 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:52:48.949 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:52:59.092 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:52:59.240 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.243+0000 7f237be46640 1 Processor -- start 2023-10-30T12:52:59.241 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.243+0000 7f237be46640 1 -- start start 2023-10-30T12:52:59.242 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.243+0000 7f237be46640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f237406f7a0 0x7f237406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:59.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.243+0000 7f237be46640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f23740c6bd0 0x7f2374069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:59.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.243+0000 7f237be46640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237406a3d0 con 0x7f2374071110 2023-10-30T12:52:59.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.243+0000 7f237be46640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23740700c0 con 0x7f23740c6bd0 2023-10-30T12:52:59.243 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.243+0000 7f237be46640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2374066cf0 con 0x7f237406f7a0 2023-10-30T12:52:59.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.243+0000 7f23793e1640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f23740c6bd0 0x7f2374069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:59.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.243+0000 7f237a3e3640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2374071110 0x7f2374158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:50972/0 (socket says 172.21.15.89:50972) 2023-10-30T12:52:59.244 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.243+0000 7f237a3e3640 1 -- 172.21.15.89:0/2630956 learned_addr learned my addr 172.21.15.89:0/2630956 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:52:59.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.243+0000 7f2379be2640 1 --2- 172.21.15.89:0/2630956 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f237406f7a0 0x7f237406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:59.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.243+0000 7f23793e1640 1 -- 172.21.15.89:0/2630956 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f237406f7a0 msgr2=0x7f237406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:59.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.243+0000 7f23793e1640 1 --2- 172.21.15.89:0/2630956 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f237406f7a0 0x7f237406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:59.245 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.243+0000 7f23793e1640 1 -- 172.21.15.89:0/2630956 >> v1:172.21.15.89:6789/0 conn(0x7f2374071110 legacy=0x7f2374158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:59.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.243+0000 7f23793e1640 1 -- 172.21.15.89:0/2630956 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23740c70d0 con 0x7f23740c6bd0 2023-10-30T12:52:59.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.243+0000 7f2378be0640 1 -- 172.21.15.89:0/2630956 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1270532427 0 0) 0x7f237406a3d0 con 0x7f2374071110 2023-10-30T12:52:59.246 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.243+0000 7f2379be2640 1 --2- 172.21.15.89:0/2630956 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f237406f7a0 0x7f237406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:52:59.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.247+0000 7f23793e1640 1 --2- 172.21.15.89:0/2630956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23740c6bd0 0x7f2374069e90 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f23640097c0 tx=0x7f236401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef621cb3e7102fb7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:59.248 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.247+0000 7f2378be0640 1 -- 172.21.15.89:0/2630956 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23640a8970 con 0x7f23740c6bd0 2023-10-30T12:52:59.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.247+0000 7f2378be0640 1 -- 172.21.15.89:0/2630956 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23640a8ad0 con 0x7f23740c6bd0 2023-10-30T12:52:59.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.247+0000 7f2378be0640 1 -- 172.21.15.89:0/2630956 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23640a8e60 con 0x7f23740c6bd0 2023-10-30T12:52:59.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.247+0000 7f237be46640 1 -- 172.21.15.89:0/2630956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23740c6bd0 msgr2=0x7f2374069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:59.249 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.247+0000 7f237be46640 1 --2- 172.21.15.89:0/2630956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23740c6bd0 0x7f2374069e90 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f23640097c0 tx=0x7f236401ec70 comp rx=0 tx=0).stop 2023-10-30T12:52:59.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.247+0000 7f237be46640 1 -- 172.21.15.89:0/2630956 shutdown_connections 2023-10-30T12:52:59.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.247+0000 7f237be46640 1 --2- 172.21.15.89:0/2630956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23740c6bd0 0x7f2374069e90 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:59.250 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.247+0000 7f237be46640 1 --2- 172.21.15.89:0/2630956 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f237406f7a0 0x7f237406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:59.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.247+0000 7f237be46640 1 -- 172.21.15.89:0/2630956 >> 172.21.15.89:0/2630956 conn(0x7f23740c7910 msgr2=0x7f23740c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:59.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.247+0000 7f237be46640 1 -- 172.21.15.89:0/2630956 shutdown_connections 2023-10-30T12:52:59.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.247+0000 7f237be46640 1 -- 172.21.15.89:0/2630956 wait complete. 2023-10-30T12:52:59.251 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.251+0000 7f237be46640 1 Processor -- start 2023-10-30T12:52:59.252 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.251+0000 7f237be46640 1 -- start start 2023-10-30T12:52:59.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.251+0000 7f237be46640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f237406f7a0 0x7f23741ef130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:59.254 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.251+0000 7f237be46640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2374071110 0x7f23740aec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:59.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.251+0000 7f237be46640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23740c6bd0 0x7f23740c1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:59.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.251+0000 7f237be46640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237415bc10 con 0x7f237406f7a0 2023-10-30T12:52:59.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.251+0000 7f237be46640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23740ae7f0 con 0x7f2374071110 2023-10-30T12:52:59.255 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.251+0000 7f237be46640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23740ad970 con 0x7f23740c6bd0 2023-10-30T12:52:59.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.251+0000 7f2379be2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f237406f7a0 0x7f23741ef130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:59.256 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.251+0000 7f2379be2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f237406f7a0 0x7f23741ef130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53954/0 (socket says 172.21.15.89:53954) 2023-10-30T12:52:59.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.251+0000 7f2379be2640 1 -- 172.21.15.89:0/1289468668 learned_addr learned my addr 172.21.15.89:0/1289468668 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:52:59.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.251+0000 7f237a3e3640 1 --2- 172.21.15.89:0/1289468668 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23740c6bd0 0x7f23740c1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:59.257 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.251+0000 7f23793e1640 1 --2- 172.21.15.89:0/1289468668 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2374071110 0x7f23740aec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:59.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.251+0000 7f2379be2640 1 -- 172.21.15.89:0/1289468668 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23740c6bd0 msgr2=0x7f23740c1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:59.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.251+0000 7f2379be2640 1 --2- 172.21.15.89:0/1289468668 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23740c6bd0 0x7f23740c1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:59.258 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.251+0000 7f2379be2640 1 -- 172.21.15.89:0/1289468668 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2374071110 msgr2=0x7f23740aec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:59.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.251+0000 7f2379be2640 1 --2- 172.21.15.89:0/1289468668 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2374071110 0x7f23740aec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:59.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.251+0000 7f2379be2640 1 -- 172.21.15.89:0/1289468668 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2364009420 con 0x7f237406f7a0 2023-10-30T12:52:59.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.251+0000 7f23793e1640 1 --2- 172.21.15.89:0/1289468668 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2374071110 0x7f23740aec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:52:59.259 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.251+0000 7f2379be2640 1 --2- 172.21.15.89:0/1289468668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f237406f7a0 0x7f23741ef130 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f237000d830 tx=0x7f237000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:59.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.251+0000 7f236a7fc640 1 -- 172.21.15.89:0/1289468668 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23700096c0 con 0x7f237406f7a0 2023-10-30T12:52:59.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.251+0000 7f236a7fc640 1 -- 172.21.15.89:0/1289468668 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2370009820 con 0x7f237406f7a0 2023-10-30T12:52:59.260 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.251+0000 7f237be46640 1 -- 172.21.15.89:0/1289468668 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23740af1c0 con 0x7f237406f7a0 2023-10-30T12:52:59.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.251+0000 7f236a7fc640 1 -- 172.21.15.89:0/1289468668 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2370009bd0 con 0x7f237406f7a0 2023-10-30T12:52:59.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.251+0000 7f237be46640 1 -- 172.21.15.89:0/1289468668 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23740adec0 con 0x7f237406f7a0 2023-10-30T12:52:59.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.251+0000 7f237be46640 1 -- 172.21.15.89:0/1289468668 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f237404e170 con 0x7f237406f7a0 2023-10-30T12:52:59.261 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.255+0000 7f236a7fc640 1 -- 172.21.15.89:0/1289468668 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2370014410 con 0x7f237406f7a0 2023-10-30T12:52:59.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.255+0000 7f236a7fc640 1 --2- 172.21.15.89:0/1289468668 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23580664d0 0x7f2358068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:59.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.255+0000 7f236a7fc640 1 -- 172.21.15.89:0/1289468668 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f2370084c10 con 0x7f237406f7a0 2023-10-30T12:52:59.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.255+0000 7f23793e1640 1 --2- 172.21.15.89:0/1289468668 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23580664d0 0x7f2358068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:59.262 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.255+0000 7f23793e1640 1 --2- 172.21.15.89:0/1289468668 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23580664d0 0x7f2358068990 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f23640b2000 tx=0x7f2364008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:59.263 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.259+0000 7f236a7fc640 1 -- 172.21.15.89:0/1289468668 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2370051bf0 con 0x7f237406f7a0 2023-10-30T12:52:59.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.403+0000 7f237be46640 1 -- 172.21.15.89:0/1289468668 --> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2374070350 con 0x7f23580664d0 2023-10-30T12:52:59.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.427+0000 7f236a7fc640 1 -- 172.21.15.89:0/1289468668 <== mgr.4105 v2:172.21.15.192:6832/4076276638 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+294845 (secure 0 0 0) 0x7f23640b04c0 con 0x7f23580664d0 2023-10-30T12:52:59.428 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:52:59.429 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2023-10-30T12:52:59.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.431+0000 7f237be46640 1 -- 172.21.15.89:0/1289468668 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23580664d0 msgr2=0x7f2358068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:59.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.431+0000 7f237be46640 1 --2- 172.21.15.89:0/1289468668 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23580664d0 0x7f2358068990 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f23640b2000 tx=0x7f2364008040 comp rx=0 tx=0).stop 2023-10-30T12:52:59.431 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.431+0000 7f237be46640 1 -- 172.21.15.89:0/1289468668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f237406f7a0 msgr2=0x7f23741ef130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:59.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.431+0000 7f237be46640 1 --2- 172.21.15.89:0/1289468668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f237406f7a0 0x7f23741ef130 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f237000d830 tx=0x7f237000dcf0 comp rx=0 tx=0).stop 2023-10-30T12:52:59.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.431+0000 7f237be46640 1 -- 172.21.15.89:0/1289468668 shutdown_connections 2023-10-30T12:52:59.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.431+0000 7f237be46640 1 --2- 172.21.15.89:0/1289468668 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f23580664d0 0x7f2358068990 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:59.432 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.431+0000 7f237be46640 1 --2- 172.21.15.89:0/1289468668 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2374071110 0x7f23740aec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:59.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.431+0000 7f237be46640 1 --2- 172.21.15.89:0/1289468668 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f23740c6bd0 0x7f23740c1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:59.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.431+0000 7f237be46640 1 --2- 172.21.15.89:0/1289468668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f237406f7a0 0x7f23741ef130 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:59.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.431+0000 7f237be46640 1 -- 172.21.15.89:0/1289468668 >> 172.21.15.89:0/1289468668 conn(0x7f23740c7910 msgr2=0x7f23740c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:59.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.431+0000 7f237be46640 1 -- 172.21.15.89:0/1289468668 shutdown_connections 2023-10-30T12:52:59.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.431+0000 7f237be46640 1 -- 172.21.15.89:0/1289468668 wait complete. 2023-10-30T12:52:59.447 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":789,"stamp":"2023-10-30T12:52:57.460716+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1063659013,"num_objects":1371,"num_object_clones":0,"num_object_copies":2742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1371,"num_whiteouts":0,"num_read":4368,"num_read_kb":813164,"num_write":42625,"num_write_kb":1359490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":78,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":305728,"num_omap_keys":583,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48314,"ondisk_log_size":48314,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2629316,"kb_used_data":2088452,"kb_used_omap":12,"kb_used_meta":540787,"kb_avail":752345404,"statfs":{"total":773094113280,"available":770401693696,"internally_reserved":0,"allocated":2138574848,"data_stored":2130417737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12741,"internal_metadata":553766459},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003958"},"pg_stats":[{"pgid":"3.3f","version":"66'572","reported_seq":2734,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:15.560856+0000","last_change":"2023-10-30T12:52:15.561063+0000","last_active":"2023-10-30T12:52:15.560856+0000","last_peered":"2023-10-30T12:52:15.560856+0000","last_clean":"2023-10-30T12:52:15.560856+0000","last_became_active":"2023-10-30T12:28:50.979351+0000","last_became_peered":"2023-10-30T12:28:50.979351+0000","last_unstale":"2023-10-30T12:52:15.560856+0000","last_undegraded":"2023-10-30T12:52:15.560856+0000","last_fullsized":"2023-10-30T12:52:15.560856+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'572","last_scrub_stamp":"2023-10-30T12:52:15.560583+0000","last_deep_scrub":"66'572","last_deep_scrub_stamp":"2023-10-30T12:52:15.560583+0000","last_clean_scrub_stamp":"2023-10-30T12:52:15.560583+0000","objects_scrubbed":21,"log_size":572,"log_dups_size":0,"ondisk_log_size":572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:47:12.132657+0000","scrub_duration":0.060883743999999997,"objects_trimmed":0,"snaptrim_duration":0.0091708599999999994,"stat_sum":{"num_bytes":4321961,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":40,"num_read_kb":15,"num_write":493,"num_write_kb":5417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"66'570","reported_seq":2725,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:02.815587+0000","last_change":"2023-10-30T12:52:02.815654+0000","last_active":"2023-10-30T12:52:02.815587+0000","last_peered":"2023-10-30T12:52:02.815587+0000","last_clean":"2023-10-30T12:52:02.815587+0000","last_became_active":"2023-10-30T12:28:50.929133+0000","last_became_peered":"2023-10-30T12:28:50.929133+0000","last_unstale":"2023-10-30T12:52:02.815587+0000","last_undegraded":"2023-10-30T12:52:02.815587+0000","last_fullsized":"2023-10-30T12:52:02.815587+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'570","last_scrub_stamp":"2023-10-30T12:52:02.815504+0000","last_deep_scrub":"66'570","last_deep_scrub_stamp":"2023-10-30T12:52:02.815504+0000","last_clean_scrub_stamp":"2023-10-30T12:52:02.815504+0000","objects_scrubbed":19,"log_size":570,"log_dups_size":0,"ondisk_log_size":570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:57:55.912117+0000","scrub_duration":0.032094442000000001,"objects_trimmed":0,"snaptrim_duration":0.0023149070000000002,"stat_sum":{"num_bytes":8426607,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":33,"num_read_kb":10,"num_write":489,"num_write_kb":9508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"66'568","reported_seq":2739,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:12.337570+0000","last_change":"2023-10-30T12:52:12.337656+0000","last_active":"2023-10-30T12:52:12.337570+0000","last_peered":"2023-10-30T12:52:12.337570+0000","last_clean":"2023-10-30T12:52:12.337570+0000","last_became_active":"2023-10-30T12:28:50.933896+0000","last_became_peered":"2023-10-30T12:28:50.933896+0000","last_unstale":"2023-10-30T12:52:12.337570+0000","last_undegraded":"2023-10-30T12:52:12.337570+0000","last_fullsized":"2023-10-30T12:52:12.337570+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'568","last_scrub_stamp":"2023-10-30T12:52:12.337465+0000","last_deep_scrub":"66'568","last_deep_scrub_stamp":"2023-10-30T12:52:12.337465+0000","last_clean_scrub_stamp":"2023-10-30T12:52:12.337465+0000","objects_scrubbed":18,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:24:10.063858+0000","scrub_duration":0.040558903,"objects_trimmed":0,"snaptrim_duration":0.0029033520000000001,"stat_sum":{"num_bytes":12697764,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":49,"num_read_kb":8209,"num_write":487,"num_write_kb":25783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"66'530","reported_seq":2683,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:32.236477+0000","last_change":"2023-10-30T12:52:32.236477+0000","last_active":"2023-10-30T12:52:32.236477+0000","last_peered":"2023-10-30T12:52:32.236477+0000","last_clean":"2023-10-30T12:52:32.236477+0000","last_became_active":"2023-10-30T12:28:50.967774+0000","last_became_peered":"2023-10-30T12:28:50.967774+0000","last_unstale":"2023-10-30T12:52:32.236477+0000","last_undegraded":"2023-10-30T12:52:32.236477+0000","last_fullsized":"2023-10-30T12:52:32.236477+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'530","last_scrub_stamp":"2023-10-30T12:52:32.236159+0000","last_deep_scrub":"66'530","last_deep_scrub_stamp":"2023-10-30T12:52:32.236159+0000","last_clean_scrub_stamp":"2023-10-30T12:52:32.236159+0000","objects_scrubbed":13,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:31:41.392978+0000","scrub_duration":0.034302944000000002,"objects_trimmed":0,"snaptrim_duration":0.0075479680000000004,"stat_sum":{"num_bytes":4272818,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":30,"num_read_kb":13,"num_write":479,"num_write_kb":5884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"66'553","reported_seq":2720,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:39.811881+0000","last_change":"2023-10-30T12:51:39.812135+0000","last_active":"2023-10-30T12:51:39.811881+0000","last_peered":"2023-10-30T12:51:39.811881+0000","last_clean":"2023-10-30T12:51:39.811881+0000","last_became_active":"2023-10-30T12:28:50.973132+0000","last_became_peered":"2023-10-30T12:28:50.973132+0000","last_unstale":"2023-10-30T12:51:39.811881+0000","last_undegraded":"2023-10-30T12:51:39.811881+0000","last_fullsized":"2023-10-30T12:51:39.811881+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'553","last_scrub_stamp":"2023-10-30T12:51:39.811639+0000","last_deep_scrub":"66'553","last_deep_scrub_stamp":"2023-10-30T12:51:39.811639+0000","last_clean_scrub_stamp":"2023-10-30T12:51:39.811639+0000","objects_scrubbed":18,"log_size":553,"log_dups_size":0,"ondisk_log_size":553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:25:28.276273+0000","scrub_duration":0.055821471999999997,"objects_trimmed":0,"snaptrim_duration":0.0090421609999999999,"stat_sum":{"num_bytes":12692456,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":47,"num_read_kb":16,"num_write":494,"num_write_kb":14525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"66'566","reported_seq":2724,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:35.199332+0000","last_change":"2023-10-30T12:52:35.199562+0000","last_active":"2023-10-30T12:52:35.199332+0000","last_peered":"2023-10-30T12:52:35.199332+0000","last_clean":"2023-10-30T12:52:35.199332+0000","last_became_active":"2023-10-30T12:28:50.967910+0000","last_became_peered":"2023-10-30T12:28:50.967910+0000","last_unstale":"2023-10-30T12:52:35.199332+0000","last_undegraded":"2023-10-30T12:52:35.199332+0000","last_fullsized":"2023-10-30T12:52:35.199332+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'566","last_scrub_stamp":"2023-10-30T12:52:35.199061+0000","last_deep_scrub":"66'566","last_deep_scrub_stamp":"2023-10-30T12:52:35.199061+0000","last_clean_scrub_stamp":"2023-10-30T12:52:35.199061+0000","objects_scrubbed":15,"log_size":566,"log_dups_size":0,"ondisk_log_size":566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:50:48.082035+0000","scrub_duration":0.053241647000000003,"objects_trimmed":0,"snaptrim_duration":0.0040413009999999997,"stat_sum":{"num_bytes":12623095,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":38,"num_read_kb":16,"num_write":503,"num_write_kb":14521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"66'569","reported_seq":2730,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:55.514716+0000","last_change":"2023-10-30T12:51:55.514849+0000","last_active":"2023-10-30T12:51:55.514716+0000","last_peered":"2023-10-30T12:51:55.514716+0000","last_clean":"2023-10-30T12:51:55.514716+0000","last_became_active":"2023-10-30T12:28:50.973288+0000","last_became_peered":"2023-10-30T12:28:50.973288+0000","last_unstale":"2023-10-30T12:51:55.514716+0000","last_undegraded":"2023-10-30T12:51:55.514716+0000","last_fullsized":"2023-10-30T12:51:55.514716+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'569","last_scrub_stamp":"2023-10-30T12:51:55.514522+0000","last_deep_scrub":"66'569","last_deep_scrub_stamp":"2023-10-30T12:51:55.514522+0000","last_clean_scrub_stamp":"2023-10-30T12:51:55.514522+0000","objects_scrubbed":18,"log_size":569,"log_dups_size":0,"ondisk_log_size":569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:24:36.003579+0000","scrub_duration":0.043916595000000003,"objects_trimmed":0,"snaptrim_duration":0.0037299339999999999,"stat_sum":{"num_bytes":4302112,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":41,"num_read_kb":11,"num_write":510,"num_write_kb":7154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"66'518","reported_seq":2677,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:19.850824+0000","last_change":"2023-10-30T12:52:19.850891+0000","last_active":"2023-10-30T12:52:19.850824+0000","last_peered":"2023-10-30T12:52:19.850824+0000","last_clean":"2023-10-30T12:52:19.850824+0000","last_became_active":"2023-10-30T12:28:50.954988+0000","last_became_peered":"2023-10-30T12:28:50.954988+0000","last_unstale":"2023-10-30T12:52:19.850824+0000","last_undegraded":"2023-10-30T12:52:19.850824+0000","last_fullsized":"2023-10-30T12:52:19.850824+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'518","last_scrub_stamp":"2023-10-30T12:52:19.850743+0000","last_deep_scrub":"66'518","last_deep_scrub_stamp":"2023-10-30T12:52:19.850743+0000","last_clean_scrub_stamp":"2023-10-30T12:52:19.850743+0000","objects_scrubbed":18,"log_size":518,"log_dups_size":0,"ondisk_log_size":518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:55:39.941900+0000","scrub_duration":0.048490766999999997,"objects_trimmed":0,"snaptrim_duration":0.0028962670000000001,"stat_sum":{"num_bytes":8424944,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":39,"num_read_kb":15,"num_write":456,"num_write_kb":10248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"66'595","reported_seq":2764,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:04.680217+0000","last_change":"2023-10-30T12:52:04.680454+0000","last_active":"2023-10-30T12:52:04.680217+0000","last_peered":"2023-10-30T12:52:04.680217+0000","last_clean":"2023-10-30T12:52:04.680217+0000","last_became_active":"2023-10-30T12:28:50.976564+0000","last_became_peered":"2023-10-30T12:28:50.976564+0000","last_unstale":"2023-10-30T12:52:04.680217+0000","last_undegraded":"2023-10-30T12:52:04.680217+0000","last_fullsized":"2023-10-30T12:52:04.680217+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'595","last_scrub_stamp":"2023-10-30T12:52:04.679935+0000","last_deep_scrub":"66'595","last_deep_scrub_stamp":"2023-10-30T12:52:04.679935+0000","last_clean_scrub_stamp":"2023-10-30T12:52:04.679935+0000","objects_scrubbed":22,"log_size":595,"log_dups_size":0,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:37:06.466494+0000","scrub_duration":0.091817582999999994,"objects_trimmed":0,"snaptrim_duration":0.0094697280000000002,"stat_sum":{"num_bytes":16840394,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":49,"num_read_kb":4116,"num_write":513,"num_write_kb":27605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"66'566","reported_seq":2747,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:57.616031+0000","last_change":"2023-10-30T12:51:57.616119+0000","last_active":"2023-10-30T12:51:57.616031+0000","last_peered":"2023-10-30T12:51:57.616031+0000","last_clean":"2023-10-30T12:51:57.616031+0000","last_became_active":"2023-10-30T12:28:50.951809+0000","last_became_peered":"2023-10-30T12:28:50.951809+0000","last_unstale":"2023-10-30T12:51:57.616031+0000","last_undegraded":"2023-10-30T12:51:57.616031+0000","last_fullsized":"2023-10-30T12:51:57.616031+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'566","last_scrub_stamp":"2023-10-30T12:51:57.615865+0000","last_deep_scrub":"66'566","last_deep_scrub_stamp":"2023-10-30T12:51:57.615865+0000","last_clean_scrub_stamp":"2023-10-30T12:51:57.615865+0000","objects_scrubbed":27,"log_size":566,"log_dups_size":0,"ondisk_log_size":566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:45:34.480445+0000","scrub_duration":0.097301143000000007,"objects_trimmed":0,"snaptrim_duration":0.0078740100000000007,"stat_sum":{"num_bytes":16924216,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":59,"num_read_kb":29,"num_write":493,"num_write_kb":22764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"66'600","reported_seq":2773,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:10.791677+0000","last_change":"2023-10-30T12:52:10.791921+0000","last_active":"2023-10-30T12:52:10.791677+0000","last_peered":"2023-10-30T12:52:10.791677+0000","last_clean":"2023-10-30T12:52:10.791677+0000","last_became_active":"2023-10-30T12:29:32.958473+0000","last_became_peered":"2023-10-30T12:29:32.958473+0000","last_unstale":"2023-10-30T12:52:10.791677+0000","last_undegraded":"2023-10-30T12:52:10.791677+0000","last_fullsized":"2023-10-30T12:52:10.791677+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'600","last_scrub_stamp":"2023-10-30T12:52:10.791444+0000","last_deep_scrub":"66'600","last_deep_scrub_stamp":"2023-10-30T12:52:10.791444+0000","last_clean_scrub_stamp":"2023-10-30T12:52:10.791444+0000","objects_scrubbed":23,"log_size":600,"log_dups_size":0,"ondisk_log_size":600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:50:18.491558+0000","scrub_duration":0.058907956999999997,"objects_trimmed":0,"snaptrim_duration":0.0045626240000000004,"stat_sum":{"num_bytes":12721502,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":46,"num_read_kb":17,"num_write":533,"num_write_kb":14913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"66'526","reported_seq":2680,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:43.156958+0000","last_change":"2023-10-30T12:51:43.157088+0000","last_active":"2023-10-30T12:51:43.156958+0000","last_peered":"2023-10-30T12:51:43.156958+0000","last_clean":"2023-10-30T12:51:43.156958+0000","last_became_active":"2023-10-30T12:28:50.943787+0000","last_became_peered":"2023-10-30T12:28:50.943787+0000","last_unstale":"2023-10-30T12:51:43.156958+0000","last_undegraded":"2023-10-30T12:51:43.156958+0000","last_fullsized":"2023-10-30T12:51:43.156958+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'526","last_scrub_stamp":"2023-10-30T12:51:43.156744+0000","last_deep_scrub":"66'526","last_deep_scrub_stamp":"2023-10-30T12:51:43.156744+0000","last_clean_scrub_stamp":"2023-10-30T12:51:43.156744+0000","objects_scrubbed":20,"log_size":526,"log_dups_size":0,"ondisk_log_size":526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:59:42.865066+0000","scrub_duration":0.057927502999999998,"objects_trimmed":0,"snaptrim_duration":0.0047342360000000002,"stat_sum":{"num_bytes":12634629,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":34,"num_read_kb":4115,"num_write":467,"num_write_kb":18795,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"66'611","reported_seq":2781,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:07.692371+0000","last_change":"2023-10-30T12:52:07.692447+0000","last_active":"2023-10-30T12:52:07.692371+0000","last_peered":"2023-10-30T12:52:07.692371+0000","last_clean":"2023-10-30T12:52:07.692371+0000","last_became_active":"2023-10-30T12:28:50.937906+0000","last_became_peered":"2023-10-30T12:28:50.937906+0000","last_unstale":"2023-10-30T12:52:07.692371+0000","last_undegraded":"2023-10-30T12:52:07.692371+0000","last_fullsized":"2023-10-30T12:52:07.692371+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'611","last_scrub_stamp":"2023-10-30T12:52:07.692259+0000","last_deep_scrub":"66'611","last_deep_scrub_stamp":"2023-10-30T12:52:07.692259+0000","last_clean_scrub_stamp":"2023-10-30T12:52:07.692259+0000","objects_scrubbed":11,"log_size":611,"log_dups_size":0,"ondisk_log_size":611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:46:50.215640+0000","scrub_duration":0.029019208000000001,"objects_trimmed":0,"snaptrim_duration":0.003369164,"stat_sum":{"num_bytes":4268758,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":50,"num_read_kb":8215,"num_write":547,"num_write_kb":14586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"66'554","reported_seq":2723,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:46.367652+0000","last_change":"2023-10-30T12:51:46.367652+0000","last_active":"2023-10-30T12:51:46.367652+0000","last_peered":"2023-10-30T12:51:46.367652+0000","last_clean":"2023-10-30T12:51:46.367652+0000","last_became_active":"2023-10-30T12:28:50.932753+0000","last_became_peered":"2023-10-30T12:28:50.932753+0000","last_unstale":"2023-10-30T12:51:46.367652+0000","last_undegraded":"2023-10-30T12:51:46.367652+0000","last_fullsized":"2023-10-30T12:51:46.367652+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'554","last_scrub_stamp":"2023-10-30T12:51:46.367517+0000","last_deep_scrub":"66'554","last_deep_scrub_stamp":"2023-10-30T12:51:46.367517+0000","last_clean_scrub_stamp":"2023-10-30T12:51:46.367517+0000","objects_scrubbed":25,"log_size":554,"log_dups_size":0,"ondisk_log_size":554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:15:04.516611+0000","scrub_duration":0.047742825000000003,"objects_trimmed":0,"snaptrim_duration":0.0027063,"stat_sum":{"num_bytes":16858869,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":47,"num_read_kb":17,"num_write":493,"num_write_kb":19132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"66'580","reported_seq":2751,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:49.898263+0000","last_change":"2023-10-30T12:51:49.898263+0000","last_active":"2023-10-30T12:51:49.898263+0000","last_peered":"2023-10-30T12:51:49.898263+0000","last_clean":"2023-10-30T12:51:49.898263+0000","last_became_active":"2023-10-30T12:28:50.943011+0000","last_became_peered":"2023-10-30T12:28:50.943011+0000","last_unstale":"2023-10-30T12:51:49.898263+0000","last_undegraded":"2023-10-30T12:51:49.898263+0000","last_fullsized":"2023-10-30T12:51:49.898263+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'580","last_scrub_stamp":"2023-10-30T12:51:49.898122+0000","last_deep_scrub":"66'580","last_deep_scrub_stamp":"2023-10-30T12:51:49.898122+0000","last_clean_scrub_stamp":"2023-10-30T12:51:49.898122+0000","objects_scrubbed":21,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:56:15.625962+0000","scrub_duration":0.069989635999999994,"objects_trimmed":0,"snaptrim_duration":0.001813863,"stat_sum":{"num_bytes":21155803,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":51,"num_read_kb":4115,"num_write":508,"num_write_kb":22928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"66'583","reported_seq":2750,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:16.300403+0000","last_change":"2023-10-30T12:52:16.300527+0000","last_active":"2023-10-30T12:52:16.300403+0000","last_peered":"2023-10-30T12:52:16.300403+0000","last_clean":"2023-10-30T12:52:16.300403+0000","last_became_active":"2023-10-30T12:28:50.952744+0000","last_became_peered":"2023-10-30T12:28:50.952744+0000","last_unstale":"2023-10-30T12:52:16.300403+0000","last_undegraded":"2023-10-30T12:52:16.300403+0000","last_fullsized":"2023-10-30T12:52:16.300403+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'583","last_scrub_stamp":"2023-10-30T12:52:16.300199+0000","last_deep_scrub":"66'583","last_deep_scrub_stamp":"2023-10-30T12:52:16.300199+0000","last_clean_scrub_stamp":"2023-10-30T12:52:16.300199+0000","objects_scrubbed":17,"log_size":583,"log_dups_size":0,"ondisk_log_size":583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:11:54.600332+0000","scrub_duration":0.041195979000000001,"objects_trimmed":0,"snaptrim_duration":0.0034371530000000001,"stat_sum":{"num_bytes":4262714,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":46,"num_read_kb":18,"num_write":521,"num_write_kb":9441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"66'609","reported_seq":2770,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:38.248370+0000","last_change":"2023-10-30T12:52:38.248503+0000","last_active":"2023-10-30T12:52:38.248370+0000","last_peered":"2023-10-30T12:52:38.248370+0000","last_clean":"2023-10-30T12:52:38.248370+0000","last_became_active":"2023-10-30T12:28:50.974576+0000","last_became_peered":"2023-10-30T12:28:50.974576+0000","last_unstale":"2023-10-30T12:52:38.248370+0000","last_undegraded":"2023-10-30T12:52:38.248370+0000","last_fullsized":"2023-10-30T12:52:38.248370+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'609","last_scrub_stamp":"2023-10-30T12:52:38.248115+0000","last_deep_scrub":"66'609","last_deep_scrub_stamp":"2023-10-30T12:52:38.248115+0000","last_clean_scrub_stamp":"2023-10-30T12:52:38.248115+0000","objects_scrubbed":19,"log_size":609,"log_dups_size":0,"ondisk_log_size":609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:18:26.030262+0000","scrub_duration":0.066337507000000004,"objects_trimmed":0,"snaptrim_duration":0.0080003049999999992,"stat_sum":{"num_bytes":8462359,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":40,"num_read_kb":4113,"num_write":531,"num_write_kb":16554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"66'543","reported_seq":2722,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:46.554500+0000","last_change":"2023-10-30T12:51:46.554500+0000","last_active":"2023-10-30T12:51:46.554500+0000","last_peered":"2023-10-30T12:51:46.554500+0000","last_clean":"2023-10-30T12:51:46.554500+0000","last_became_active":"2023-10-30T12:28:50.937384+0000","last_became_peered":"2023-10-30T12:28:50.937384+0000","last_unstale":"2023-10-30T12:51:46.554500+0000","last_undegraded":"2023-10-30T12:51:46.554500+0000","last_fullsized":"2023-10-30T12:51:46.554500+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'543","last_scrub_stamp":"2023-10-30T12:51:46.554353+0000","last_deep_scrub":"66'543","last_deep_scrub_stamp":"2023-10-30T12:51:46.554353+0000","last_clean_scrub_stamp":"2023-10-30T12:51:46.554353+0000","objects_scrubbed":16,"log_size":543,"log_dups_size":0,"ondisk_log_size":543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:23:08.192334+0000","scrub_duration":0.042828463999999997,"objects_trimmed":0,"snaptrim_duration":0.001162803,"stat_sum":{"num_bytes":8456287,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":58,"num_read_kb":27,"num_write":488,"num_write_kb":9441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"66'581","reported_seq":2758,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:14.634497+0000","last_change":"2023-10-30T12:52:14.634614+0000","last_active":"2023-10-30T12:52:14.634497+0000","last_peered":"2023-10-30T12:52:14.634497+0000","last_clean":"2023-10-30T12:52:14.634497+0000","last_became_active":"2023-10-30T12:28:50.973369+0000","last_became_peered":"2023-10-30T12:28:50.973369+0000","last_unstale":"2023-10-30T12:52:14.634497+0000","last_undegraded":"2023-10-30T12:52:14.634497+0000","last_fullsized":"2023-10-30T12:52:14.634497+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'581","last_scrub_stamp":"2023-10-30T12:52:14.634290+0000","last_deep_scrub":"66'581","last_deep_scrub_stamp":"2023-10-30T12:52:14.634290+0000","last_clean_scrub_stamp":"2023-10-30T12:52:14.634290+0000","objects_scrubbed":27,"log_size":581,"log_dups_size":0,"ondisk_log_size":581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:50:37.072999+0000","scrub_duration":0.087054615000000002,"objects_trimmed":0,"snaptrim_duration":0.0093923919999999994,"stat_sum":{"num_bytes":12689191,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":55,"num_read_kb":4114,"num_write":505,"num_write_kb":18071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"66'592","reported_seq":2762,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:47.730012+0000","last_change":"2023-10-30T12:51:47.730137+0000","last_active":"2023-10-30T12:51:47.730012+0000","last_peered":"2023-10-30T12:51:47.730012+0000","last_clean":"2023-10-30T12:51:47.730012+0000","last_became_active":"2023-10-30T12:28:50.954831+0000","last_became_peered":"2023-10-30T12:28:50.954831+0000","last_unstale":"2023-10-30T12:51:47.730012+0000","last_undegraded":"2023-10-30T12:51:47.730012+0000","last_fullsized":"2023-10-30T12:51:47.730012+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'592","last_scrub_stamp":"2023-10-30T12:51:47.729828+0000","last_deep_scrub":"66'592","last_deep_scrub_stamp":"2023-10-30T12:51:47.729828+0000","last_clean_scrub_stamp":"2023-10-30T12:51:47.729828+0000","objects_scrubbed":20,"log_size":592,"log_dups_size":0,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:14:47.264586+0000","scrub_duration":0.089463355999999994,"objects_trimmed":0,"snaptrim_duration":0.007119116,"stat_sum":{"num_bytes":21055454,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":49,"num_read_kb":4113,"num_write":524,"num_write_kb":30367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"66'549","reported_seq":2711,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:10.434180+0000","last_change":"2023-10-30T12:52:10.434275+0000","last_active":"2023-10-30T12:52:10.434180+0000","last_peered":"2023-10-30T12:52:10.434180+0000","last_clean":"2023-10-30T12:52:10.434180+0000","last_became_active":"2023-10-30T12:28:50.962065+0000","last_became_peered":"2023-10-30T12:28:50.962065+0000","last_unstale":"2023-10-30T12:52:10.434180+0000","last_undegraded":"2023-10-30T12:52:10.434180+0000","last_fullsized":"2023-10-30T12:52:10.434180+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'549","last_scrub_stamp":"2023-10-30T12:52:10.434060+0000","last_deep_scrub":"66'549","last_deep_scrub_stamp":"2023-10-30T12:52:10.434060+0000","last_clean_scrub_stamp":"2023-10-30T12:52:10.434060+0000","objects_scrubbed":17,"log_size":549,"log_dups_size":0,"ondisk_log_size":549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:18:27.491696+0000","scrub_duration":0.079332753000000006,"objects_trimmed":0,"snaptrim_duration":0.0037739459999999998,"stat_sum":{"num_bytes":25200014,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":41,"num_read_kb":21,"num_write":490,"num_write_kb":26352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"66'625","reported_seq":2792,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:33.218192+0000","last_change":"2023-10-30T12:52:33.218401+0000","last_active":"2023-10-30T12:52:33.218192+0000","last_peered":"2023-10-30T12:52:33.218192+0000","last_clean":"2023-10-30T12:52:33.218192+0000","last_became_active":"2023-10-30T12:28:50.967651+0000","last_became_peered":"2023-10-30T12:28:50.967651+0000","last_unstale":"2023-10-30T12:52:33.218192+0000","last_undegraded":"2023-10-30T12:52:33.218192+0000","last_fullsized":"2023-10-30T12:52:33.218192+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'625","last_scrub_stamp":"2023-10-30T12:52:33.217988+0000","last_deep_scrub":"66'625","last_deep_scrub_stamp":"2023-10-30T12:52:33.217988+0000","last_clean_scrub_stamp":"2023-10-30T12:52:33.217988+0000","objects_scrubbed":17,"log_size":625,"log_dups_size":0,"ondisk_log_size":625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:41:02.531460+0000","scrub_duration":0.058304638999999998,"objects_trimmed":0,"snaptrim_duration":0.0081085610000000002,"stat_sum":{"num_bytes":8425872,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":46,"num_read_kb":21,"num_write":555,"num_write_kb":13596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"66'584","reported_seq":2755,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:10.755998+0000","last_change":"2023-10-30T12:52:10.756068+0000","last_active":"2023-10-30T12:52:10.755998+0000","last_peered":"2023-10-30T12:52:10.755998+0000","last_clean":"2023-10-30T12:52:10.755998+0000","last_became_active":"2023-10-30T12:28:50.962290+0000","last_became_peered":"2023-10-30T12:28:50.962290+0000","last_unstale":"2023-10-30T12:52:10.755998+0000","last_undegraded":"2023-10-30T12:52:10.755998+0000","last_fullsized":"2023-10-30T12:52:10.755998+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'584","last_scrub_stamp":"2023-10-30T12:52:10.755902+0000","last_deep_scrub":"66'584","last_deep_scrub_stamp":"2023-10-30T12:52:10.755902+0000","last_clean_scrub_stamp":"2023-10-30T12:52:10.755902+0000","objects_scrubbed":24,"log_size":584,"log_dups_size":0,"ondisk_log_size":584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:10:48.108121+0000","scrub_duration":0.037006553999999997,"objects_trimmed":0,"snaptrim_duration":0.0042026670000000002,"stat_sum":{"num_bytes":195030,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":50,"num_read_kb":4117,"num_write":509,"num_write_kb":6258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"66'591","reported_seq":2753,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:13.308161+0000","last_change":"2023-10-30T12:52:13.308161+0000","last_active":"2023-10-30T12:52:13.308161+0000","last_peered":"2023-10-30T12:52:13.308161+0000","last_clean":"2023-10-30T12:52:13.308161+0000","last_became_active":"2023-10-30T12:28:50.965713+0000","last_became_peered":"2023-10-30T12:28:50.965713+0000","last_unstale":"2023-10-30T12:52:13.308161+0000","last_undegraded":"2023-10-30T12:52:13.308161+0000","last_fullsized":"2023-10-30T12:52:13.308161+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'591","last_scrub_stamp":"2023-10-30T12:52:13.307638+0000","last_deep_scrub":"66'591","last_deep_scrub_stamp":"2023-10-30T12:52:13.307638+0000","last_clean_scrub_stamp":"2023-10-30T12:52:13.307638+0000","objects_scrubbed":16,"log_size":591,"log_dups_size":0,"ondisk_log_size":591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:12:08.906462+0000","scrub_duration":0.066437737999999996,"objects_trimmed":0,"snaptrim_duration":0.0056034209999999999,"stat_sum":{"num_bytes":12659519,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":40,"num_read_kb":16,"num_write":528,"num_write_kb":18837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"66'544","reported_seq":2697,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:22.817243+0000","last_change":"2023-10-30T12:52:22.817243+0000","last_active":"2023-10-30T12:52:22.817243+0000","last_peered":"2023-10-30T12:52:22.817243+0000","last_clean":"2023-10-30T12:52:22.817243+0000","last_became_active":"2023-10-30T12:28:50.951026+0000","last_became_peered":"2023-10-30T12:28:50.951026+0000","last_unstale":"2023-10-30T12:52:22.817243+0000","last_undegraded":"2023-10-30T12:52:22.817243+0000","last_fullsized":"2023-10-30T12:52:22.817243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'544","last_scrub_stamp":"2023-10-30T12:52:22.817093+0000","last_deep_scrub":"66'544","last_deep_scrub_stamp":"2023-10-30T12:52:22.817093+0000","last_clean_scrub_stamp":"2023-10-30T12:52:22.817093+0000","objects_scrubbed":11,"log_size":544,"log_dups_size":0,"ondisk_log_size":544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:07:21.492835+0000","scrub_duration":0.052369851000000002,"objects_trimmed":0,"snaptrim_duration":0.0031697840000000001,"stat_sum":{"num_bytes":12597918,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":31,"num_read_kb":12,"num_write":481,"num_write_kb":19629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"66'624","reported_seq":2798,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:52.603819+0000","last_change":"2023-10-30T12:51:52.604014+0000","last_active":"2023-10-30T12:51:52.603819+0000","last_peered":"2023-10-30T12:51:52.603819+0000","last_clean":"2023-10-30T12:51:52.603819+0000","last_became_active":"2023-10-30T12:28:50.962484+0000","last_became_peered":"2023-10-30T12:28:50.962484+0000","last_unstale":"2023-10-30T12:51:52.603819+0000","last_undegraded":"2023-10-30T12:51:52.603819+0000","last_fullsized":"2023-10-30T12:51:52.603819+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'624","last_scrub_stamp":"2023-10-30T12:51:52.603583+0000","last_deep_scrub":"66'624","last_deep_scrub_stamp":"2023-10-30T12:51:52.603583+0000","last_clean_scrub_stamp":"2023-10-30T12:51:52.603583+0000","objects_scrubbed":26,"log_size":624,"log_dups_size":0,"ondisk_log_size":624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:26:46.288012+0000","scrub_duration":0.079879599999999995,"objects_trimmed":0,"snaptrim_duration":0.0083773500000000004,"stat_sum":{"num_bytes":12675985,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":51,"num_read_kb":23,"num_write":557,"num_write_kb":15080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"66'532","reported_seq":2696,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:51.872430+0000","last_change":"2023-10-30T12:51:51.872430+0000","last_active":"2023-10-30T12:51:51.872430+0000","last_peered":"2023-10-30T12:51:51.872430+0000","last_clean":"2023-10-30T12:51:51.872430+0000","last_became_active":"2023-10-30T12:28:50.933093+0000","last_became_peered":"2023-10-30T12:28:50.933093+0000","last_unstale":"2023-10-30T12:51:51.872430+0000","last_undegraded":"2023-10-30T12:51:51.872430+0000","last_fullsized":"2023-10-30T12:51:51.872430+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'532","last_scrub_stamp":"2023-10-30T12:51:51.872277+0000","last_deep_scrub":"66'532","last_deep_scrub_stamp":"2023-10-30T12:51:51.872277+0000","last_clean_scrub_stamp":"2023-10-30T12:51:51.872277+0000","objects_scrubbed":22,"log_size":532,"log_dups_size":0,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:06:03.858620+0000","scrub_duration":0.057177436999999998,"objects_trimmed":0,"snaptrim_duration":0.0022251829999999999,"stat_sum":{"num_bytes":25274091,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":42,"num_read_kb":8207,"num_write":449,"num_write_kb":27548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"66'517","reported_seq":2665,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:04.763210+0000","last_change":"2023-10-30T12:52:04.763210+0000","last_active":"2023-10-30T12:52:04.763210+0000","last_peered":"2023-10-30T12:52:04.763210+0000","last_clean":"2023-10-30T12:52:04.763210+0000","last_became_active":"2023-10-30T12:28:50.952531+0000","last_became_peered":"2023-10-30T12:28:50.952531+0000","last_unstale":"2023-10-30T12:52:04.763210+0000","last_undegraded":"2023-10-30T12:52:04.763210+0000","last_fullsized":"2023-10-30T12:52:04.763210+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'517","last_scrub_stamp":"2023-10-30T12:52:04.763062+0000","last_deep_scrub":"66'517","last_deep_scrub_stamp":"2023-10-30T12:52:04.763062+0000","last_clean_scrub_stamp":"2023-10-30T12:52:04.763062+0000","objects_scrubbed":18,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:37:42.920027+0000","scrub_duration":0.034901690999999999,"objects_trimmed":0,"snaptrim_duration":0.0039301029999999999,"stat_sum":{"num_bytes":4373272,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":26,"num_read_kb":3284,"num_write":458,"num_write_kb":14265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"66'542","reported_seq":2709,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:59.310588+0000","last_change":"2023-10-30T12:51:59.310659+0000","last_active":"2023-10-30T12:51:59.310588+0000","last_peered":"2023-10-30T12:51:59.310588+0000","last_clean":"2023-10-30T12:51:59.310588+0000","last_became_active":"2023-10-30T12:28:50.933124+0000","last_became_peered":"2023-10-30T12:28:50.933124+0000","last_unstale":"2023-10-30T12:51:59.310588+0000","last_undegraded":"2023-10-30T12:51:59.310588+0000","last_fullsized":"2023-10-30T12:51:59.310588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'542","last_scrub_stamp":"2023-10-30T12:51:59.310503+0000","last_deep_scrub":"66'542","last_deep_scrub_stamp":"2023-10-30T12:51:59.310503+0000","last_clean_scrub_stamp":"2023-10-30T12:51:59.310503+0000","objects_scrubbed":24,"log_size":542,"log_dups_size":0,"ondisk_log_size":542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:48:29.721009+0000","scrub_duration":0.033010021000000001,"objects_trimmed":0,"snaptrim_duration":0.0034752709999999998,"stat_sum":{"num_bytes":8490168,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":45,"num_read_kb":14,"num_write":469,"num_write_kb":10488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"66'611","reported_seq":2784,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:21.315467+0000","last_change":"2023-10-30T12:52:21.315616+0000","last_active":"2023-10-30T12:52:21.315467+0000","last_peered":"2023-10-30T12:52:21.315467+0000","last_clean":"2023-10-30T12:52:21.315467+0000","last_became_active":"2023-10-30T12:28:50.965639+0000","last_became_peered":"2023-10-30T12:28:50.965639+0000","last_unstale":"2023-10-30T12:52:21.315467+0000","last_undegraded":"2023-10-30T12:52:21.315467+0000","last_fullsized":"2023-10-30T12:52:21.315467+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'611","last_scrub_stamp":"2023-10-30T12:52:21.315261+0000","last_deep_scrub":"66'611","last_deep_scrub_stamp":"2023-10-30T12:52:21.315261+0000","last_clean_scrub_stamp":"2023-10-30T12:52:21.315261+0000","objects_scrubbed":17,"log_size":611,"log_dups_size":0,"ondisk_log_size":611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:59:20.933491+0000","scrub_duration":0.051029466000000002,"objects_trimmed":0,"snaptrim_duration":0.0076861280000000004,"stat_sum":{"num_bytes":8432593,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":51,"num_read_kb":20,"num_write":547,"num_write_kb":11509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"66'595","reported_seq":2767,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:45.496340+0000","last_change":"2023-10-30T12:51:45.496340+0000","last_active":"2023-10-30T12:51:45.496340+0000","last_peered":"2023-10-30T12:51:45.496340+0000","last_clean":"2023-10-30T12:51:45.496340+0000","last_became_active":"2023-10-30T12:28:50.955084+0000","last_became_peered":"2023-10-30T12:28:50.955084+0000","last_unstale":"2023-10-30T12:51:45.496340+0000","last_undegraded":"2023-10-30T12:51:45.496340+0000","last_fullsized":"2023-10-30T12:51:45.496340+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'595","last_scrub_stamp":"2023-10-30T12:51:45.496211+0000","last_deep_scrub":"66'595","last_deep_scrub_stamp":"2023-10-30T12:51:45.496211+0000","last_clean_scrub_stamp":"2023-10-30T12:51:45.496211+0000","objects_scrubbed":12,"log_size":595,"log_dups_size":0,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:13:37.745803+0000","scrub_duration":0.027110084,"objects_trimmed":0,"snaptrim_duration":0.0012043119999999999,"stat_sum":{"num_bytes":4257179,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":50,"num_read_kb":22,"num_write":532,"num_write_kb":9837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"66'569","reported_seq":2744,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:54.604746+0000","last_change":"2023-10-30T12:51:54.604869+0000","last_active":"2023-10-30T12:51:54.604746+0000","last_peered":"2023-10-30T12:51:54.604746+0000","last_clean":"2023-10-30T12:51:54.604746+0000","last_became_active":"2023-10-30T12:28:50.944775+0000","last_became_peered":"2023-10-30T12:28:50.944775+0000","last_unstale":"2023-10-30T12:51:54.604746+0000","last_undegraded":"2023-10-30T12:51:54.604746+0000","last_fullsized":"2023-10-30T12:51:54.604746+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'569","last_scrub_stamp":"2023-10-30T12:51:54.604552+0000","last_deep_scrub":"66'569","last_deep_scrub_stamp":"2023-10-30T12:51:54.604552+0000","last_clean_scrub_stamp":"2023-10-30T12:51:54.604552+0000","objects_scrubbed":20,"log_size":569,"log_dups_size":0,"ondisk_log_size":569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:33:53.639827+0000","scrub_duration":0.054468190999999999,"objects_trimmed":0,"snaptrim_duration":0.0070573429999999998,"stat_sum":{"num_bytes":12763889,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":53,"num_read_kb":21,"num_write":514,"num_write_kb":13736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"66'590","reported_seq":2750,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:44.129343+0000","last_change":"2023-10-30T12:51:44.129472+0000","last_active":"2023-10-30T12:51:44.129343+0000","last_peered":"2023-10-30T12:51:44.129343+0000","last_clean":"2023-10-30T12:51:44.129343+0000","last_became_active":"2023-10-30T12:28:50.940567+0000","last_became_peered":"2023-10-30T12:28:50.940567+0000","last_unstale":"2023-10-30T12:51:44.129343+0000","last_undegraded":"2023-10-30T12:51:44.129343+0000","last_fullsized":"2023-10-30T12:51:44.129343+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'590","last_scrub_stamp":"2023-10-30T12:51:44.129123+0000","last_deep_scrub":"66'590","last_deep_scrub_stamp":"2023-10-30T12:51:44.129123+0000","last_clean_scrub_stamp":"2023-10-30T12:51:44.129123+0000","objects_scrubbed":10,"log_size":590,"log_dups_size":0,"ondisk_log_size":590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:25:14.156000+0000","scrub_duration":0.048555811999999997,"objects_trimmed":0,"snaptrim_duration":0.0028523340000000002,"stat_sum":{"num_bytes":9696728,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":17,"num_write":529,"num_write_kb":12781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"66'283","reported_seq":371,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:45.780003+0000","last_change":"2023-10-30T12:51:45.780059+0000","last_active":"2023-10-30T12:51:45.780003+0000","last_peered":"2023-10-30T12:51:45.780003+0000","last_clean":"2023-10-30T12:51:45.780003+0000","last_became_active":"2023-10-30T12:28:50.395382+0000","last_became_peered":"2023-10-30T12:28:50.395382+0000","last_unstale":"2023-10-30T12:51:45.780003+0000","last_undegraded":"2023-10-30T12:51:45.780003+0000","last_fullsized":"2023-10-30T12:51:45.780003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'283","last_scrub_stamp":"2023-10-30T12:51:45.779902+0000","last_deep_scrub":"66'283","last_deep_scrub_stamp":"2023-10-30T12:51:45.779902+0000","last_clean_scrub_stamp":"2023-10-30T12:51:45.779902+0000","objects_scrubbed":5,"log_size":283,"log_dups_size":0,"ondisk_log_size":283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:36:15.846905+0000","scrub_duration":0.042332846,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582946,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":45059,"num_write":213,"num_write_kb":12413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8500,"num_omap_keys":17,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"66'511","reported_seq":2676,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:45.384937+0000","last_change":"2023-10-30T12:51:45.385000+0000","last_active":"2023-10-30T12:51:45.384937+0000","last_peered":"2023-10-30T12:51:45.384937+0000","last_clean":"2023-10-30T12:51:45.384937+0000","last_became_active":"2023-10-30T12:28:50.952613+0000","last_became_peered":"2023-10-30T12:28:50.952613+0000","last_unstale":"2023-10-30T12:51:45.384937+0000","last_undegraded":"2023-10-30T12:51:45.384937+0000","last_fullsized":"2023-10-30T12:51:45.384937+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'511","last_scrub_stamp":"2023-10-30T12:51:45.384858+0000","last_deep_scrub":"66'511","last_deep_scrub_stamp":"2023-10-30T12:51:45.384858+0000","last_clean_scrub_stamp":"2023-10-30T12:51:45.384858+0000","objects_scrubbed":19,"log_size":511,"log_dups_size":0,"ondisk_log_size":511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:37:21.237208+0000","scrub_duration":0.069355456999999995,"objects_trimmed":0,"snaptrim_duration":0.0038505420000000002,"stat_sum":{"num_bytes":21061321,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":45,"num_read_kb":18,"num_write":449,"num_write_kb":22047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"66'909","reported_seq":1201,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:49.177081+0000","last_change":"2023-10-30T12:51:42.662808+0000","last_active":"2023-10-30T12:51:49.177081+0000","last_peered":"2023-10-30T12:51:49.177081+0000","last_clean":"2023-10-30T12:51:49.177081+0000","last_became_active":"2023-10-30T12:28:50.383898+0000","last_became_peered":"2023-10-30T12:28:50.383898+0000","last_unstale":"2023-10-30T12:51:49.177081+0000","last_undegraded":"2023-10-30T12:51:49.177081+0000","last_fullsized":"2023-10-30T12:51:49.177081+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'908","last_scrub_stamp":"2023-10-30T12:51:42.662429+0000","last_deep_scrub":"66'908","last_deep_scrub_stamp":"2023-10-30T12:51:42.662429+0000","last_clean_scrub_stamp":"2023-10-30T12:51:42.662429+0000","objects_scrubbed":9,"log_size":909,"log_dups_size":0,"ondisk_log_size":909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:01:46.593761+0000","scrub_duration":0.098226147999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21567319,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":234,"num_read_kb":49886,"num_write":842,"num_write_kb":21597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":23432,"num_omap_keys":43,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"66'457","reported_seq":759,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:48.571287+0000","last_change":"2023-10-30T12:51:41.780215+0000","last_active":"2023-10-30T12:51:48.571287+0000","last_peered":"2023-10-30T12:51:48.571287+0000","last_clean":"2023-10-30T12:51:48.571287+0000","last_became_active":"2023-10-30T12:28:50.385963+0000","last_became_peered":"2023-10-30T12:28:50.385963+0000","last_unstale":"2023-10-30T12:51:48.571287+0000","last_undegraded":"2023-10-30T12:51:48.571287+0000","last_fullsized":"2023-10-30T12:51:48.571287+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'454","last_scrub_stamp":"2023-10-30T12:51:41.780079+0000","last_deep_scrub":"66'454","last_deep_scrub_stamp":"2023-10-30T12:51:41.780079+0000","last_clean_scrub_stamp":"2023-10-30T12:51:41.780079+0000","objects_scrubbed":4,"log_size":457,"log_dups_size":0,"ondisk_log_size":457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:29:54.800743+0000","scrub_duration":0.030692146999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8006399,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":240,"num_read_kb":9095,"num_write":389,"num_write_kb":8081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"66'639","reported_seq":2815,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:51.809783+0000","last_change":"2023-10-30T12:51:51.809888+0000","last_active":"2023-10-30T12:51:51.809783+0000","last_peered":"2023-10-30T12:51:51.809783+0000","last_clean":"2023-10-30T12:51:51.809783+0000","last_became_active":"2023-10-30T12:28:50.935764+0000","last_became_peered":"2023-10-30T12:28:50.935764+0000","last_unstale":"2023-10-30T12:51:51.809783+0000","last_undegraded":"2023-10-30T12:51:51.809783+0000","last_fullsized":"2023-10-30T12:51:51.809783+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'639","last_scrub_stamp":"2023-10-30T12:51:51.809674+0000","last_deep_scrub":"66'639","last_deep_scrub_stamp":"2023-10-30T12:51:51.809674+0000","last_clean_scrub_stamp":"2023-10-30T12:51:51.809674+0000","objects_scrubbed":21,"log_size":639,"log_dups_size":0,"ondisk_log_size":639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:53.793939+0000","scrub_duration":0.025406960999999999,"objects_trimmed":0,"snaptrim_duration":0.0031329320000000002,"stat_sum":{"num_bytes":4280685,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":56,"num_read_kb":4120,"num_write":569,"num_write_kb":13778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"66'571","reported_seq":655,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:43.489239+0000","last_change":"2023-10-30T12:51:43.489353+0000","last_active":"2023-10-30T12:51:43.489239+0000","last_peered":"2023-10-30T12:51:43.489239+0000","last_clean":"2023-10-30T12:51:43.489239+0000","last_became_active":"2023-10-30T12:28:50.391494+0000","last_became_peered":"2023-10-30T12:28:50.391494+0000","last_unstale":"2023-10-30T12:51:43.489239+0000","last_undegraded":"2023-10-30T12:51:43.489239+0000","last_fullsized":"2023-10-30T12:51:43.489239+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'571","last_scrub_stamp":"2023-10-30T12:51:43.489033+0000","last_deep_scrub":"66'571","last_deep_scrub_stamp":"2023-10-30T12:51:43.489033+0000","last_clean_scrub_stamp":"2023-10-30T12:51:43.489033+0000","objects_scrubbed":3,"log_size":571,"log_dups_size":0,"ondisk_log_size":571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:24:33.871160+0000","scrub_duration":0.058779308000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":36864,"num_write":519,"num_write_kb":12589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"66'619","reported_seq":2786,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:07.826653+0000","last_change":"2023-10-30T12:52:07.826695+0000","last_active":"2023-10-30T12:52:07.826653+0000","last_peered":"2023-10-30T12:52:07.826653+0000","last_clean":"2023-10-30T12:52:07.826653+0000","last_became_active":"2023-10-30T12:28:50.933062+0000","last_became_peered":"2023-10-30T12:28:50.933062+0000","last_unstale":"2023-10-30T12:52:07.826653+0000","last_undegraded":"2023-10-30T12:52:07.826653+0000","last_fullsized":"2023-10-30T12:52:07.826653+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'619","last_scrub_stamp":"2023-10-30T12:52:07.826574+0000","last_deep_scrub":"66'619","last_deep_scrub_stamp":"2023-10-30T12:52:07.826574+0000","last_clean_scrub_stamp":"2023-10-30T12:52:07.826574+0000","objects_scrubbed":27,"log_size":619,"log_dups_size":0,"ondisk_log_size":619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:21:58.906873+0000","scrub_duration":0.081541012999999996,"objects_trimmed":0,"snaptrim_duration":0.0021140489999999998,"stat_sum":{"num_bytes":33653289,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":46,"num_read_kb":4115,"num_write":549,"num_write_kb":39173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"66'132","reported_seq":214,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:46.616340+0000","last_change":"2023-10-30T12:51:46.616340+0000","last_active":"2023-10-30T12:51:46.616340+0000","last_peered":"2023-10-30T12:51:46.616340+0000","last_clean":"2023-10-30T12:51:46.616340+0000","last_became_active":"2023-10-30T12:28:50.393686+0000","last_became_peered":"2023-10-30T12:28:50.393686+0000","last_unstale":"2023-10-30T12:51:46.616340+0000","last_undegraded":"2023-10-30T12:51:46.616340+0000","last_fullsized":"2023-10-30T12:51:46.616340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'131","last_scrub_stamp":"2023-10-30T12:51:46.615853+0000","last_deep_scrub":"66'131","last_deep_scrub_stamp":"2023-10-30T12:51:46.615853+0000","last_clean_scrub_stamp":"2023-10-30T12:51:46.615853+0000","objects_scrubbed":3,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:42:31.808839+0000","scrub_duration":0.0095666550000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":75,"num_write_kb":93,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9282,"num_omap_keys":17,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"66'589","reported_seq":2760,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:15.339095+0000","last_change":"2023-10-30T12:52:15.339230+0000","last_active":"2023-10-30T12:52:15.339095+0000","last_peered":"2023-10-30T12:52:15.339095+0000","last_clean":"2023-10-30T12:52:15.339095+0000","last_became_active":"2023-10-30T12:28:50.963105+0000","last_became_peered":"2023-10-30T12:28:50.963105+0000","last_unstale":"2023-10-30T12:52:15.339095+0000","last_undegraded":"2023-10-30T12:52:15.339095+0000","last_fullsized":"2023-10-30T12:52:15.339095+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'589","last_scrub_stamp":"2023-10-30T12:52:15.338927+0000","last_deep_scrub":"66'589","last_deep_scrub_stamp":"2023-10-30T12:52:15.338927+0000","last_clean_scrub_stamp":"2023-10-30T12:52:15.338927+0000","objects_scrubbed":22,"log_size":589,"log_dups_size":0,"ondisk_log_size":589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:22:02.826284+0000","scrub_duration":0.088894443000000004,"objects_trimmed":0,"snaptrim_duration":0.0080066809999999999,"stat_sum":{"num_bytes":16920090,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":51,"num_read_kb":18,"num_write":515,"num_write_kb":18479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"66'284","reported_seq":366,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:59.398926+0000","last_change":"2023-10-30T12:51:59.399057+0000","last_active":"2023-10-30T12:51:59.398926+0000","last_peered":"2023-10-30T12:51:59.398926+0000","last_clean":"2023-10-30T12:51:59.398926+0000","last_became_active":"2023-10-30T12:28:50.397041+0000","last_became_peered":"2023-10-30T12:28:50.397041+0000","last_unstale":"2023-10-30T12:51:59.398926+0000","last_undegraded":"2023-10-30T12:51:59.398926+0000","last_fullsized":"2023-10-30T12:51:59.398926+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'284","last_scrub_stamp":"2023-10-30T12:51:59.398723+0000","last_deep_scrub":"66'284","last_deep_scrub_stamp":"2023-10-30T12:51:59.398723+0000","last_clean_scrub_stamp":"2023-10-30T12:51:59.398723+0000","objects_scrubbed":3,"log_size":284,"log_dups_size":0,"ondisk_log_size":284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:44:04.485785+0000","scrub_duration":0.038907621000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":16396,"num_write":240,"num_write_kb":8312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"66'570","reported_seq":2732,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:11.399647+0000","last_change":"2023-10-30T12:52:11.399712+0000","last_active":"2023-10-30T12:52:11.399647+0000","last_peered":"2023-10-30T12:52:11.399647+0000","last_clean":"2023-10-30T12:52:11.399647+0000","last_became_active":"2023-10-30T12:28:50.966569+0000","last_became_peered":"2023-10-30T12:28:50.966569+0000","last_unstale":"2023-10-30T12:52:11.399647+0000","last_undegraded":"2023-10-30T12:52:11.399647+0000","last_fullsized":"2023-10-30T12:52:11.399647+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'570","last_scrub_stamp":"2023-10-30T12:52:11.399561+0000","last_deep_scrub":"66'570","last_deep_scrub_stamp":"2023-10-30T12:52:11.399561+0000","last_clean_scrub_stamp":"2023-10-30T12:52:11.399561+0000","objects_scrubbed":22,"log_size":570,"log_dups_size":0,"ondisk_log_size":570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:55:55.041154+0000","scrub_duration":0.068592573000000004,"objects_trimmed":0,"snaptrim_duration":0.0037972850000000001,"stat_sum":{"num_bytes":16988939,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":42,"num_read_kb":19,"num_write":508,"num_write_kb":18140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"66'375","reported_seq":463,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:45.949491+0000","last_change":"2023-10-30T12:51:45.949541+0000","last_active":"2023-10-30T12:51:45.949491+0000","last_peered":"2023-10-30T12:51:45.949491+0000","last_clean":"2023-10-30T12:51:45.949491+0000","last_became_active":"2023-10-30T12:28:50.391260+0000","last_became_peered":"2023-10-30T12:28:50.391260+0000","last_unstale":"2023-10-30T12:51:45.949491+0000","last_undegraded":"2023-10-30T12:51:45.949491+0000","last_fullsized":"2023-10-30T12:51:45.949491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'375","last_scrub_stamp":"2023-10-30T12:51:45.949405+0000","last_deep_scrub":"66'375","last_deep_scrub_stamp":"2023-10-30T12:51:45.949405+0000","last_clean_scrub_stamp":"2023-10-30T12:51:45.949405+0000","objects_scrubbed":6,"log_size":375,"log_dups_size":0,"ondisk_log_size":375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:31:43.191871+0000","scrub_duration":0.024520874000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":22,"num_read_kb":7467,"num_write":319,"num_write_kb":4401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7112,"num_omap_keys":14,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"66'524","reported_seq":2692,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:42.140557+0000","last_change":"2023-10-30T12:51:42.140557+0000","last_active":"2023-10-30T12:51:42.140557+0000","last_peered":"2023-10-30T12:51:42.140557+0000","last_clean":"2023-10-30T12:51:42.140557+0000","last_became_active":"2023-10-30T12:28:50.974843+0000","last_became_peered":"2023-10-30T12:28:50.974843+0000","last_unstale":"2023-10-30T12:51:42.140557+0000","last_undegraded":"2023-10-30T12:51:42.140557+0000","last_fullsized":"2023-10-30T12:51:42.140557+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'524","last_scrub_stamp":"2023-10-30T12:51:42.140220+0000","last_deep_scrub":"66'524","last_deep_scrub_stamp":"2023-10-30T12:51:42.140220+0000","last_clean_scrub_stamp":"2023-10-30T12:51:42.140220+0000","objects_scrubbed":12,"log_size":524,"log_dups_size":0,"ondisk_log_size":524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:15:48.848274+0000","scrub_duration":0.033227912999999998,"objects_trimmed":0,"snaptrim_duration":0.005161372,"stat_sum":{"num_bytes":4244440,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":47,"num_read_kb":17,"num_write":458,"num_write_kb":9849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"66'359","reported_seq":459,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:08.560979+0000","last_change":"2023-10-30T12:51:40.635013+0000","last_active":"2023-10-30T12:52:08.560979+0000","last_peered":"2023-10-30T12:52:08.560979+0000","last_clean":"2023-10-30T12:52:08.560979+0000","last_became_active":"2023-10-30T12:28:50.373581+0000","last_became_peered":"2023-10-30T12:28:50.373581+0000","last_unstale":"2023-10-30T12:52:08.560979+0000","last_undegraded":"2023-10-30T12:52:08.560979+0000","last_fullsized":"2023-10-30T12:52:08.560979+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'357","last_scrub_stamp":"2023-10-30T12:51:40.634678+0000","last_deep_scrub":"66'357","last_deep_scrub_stamp":"2023-10-30T12:51:40.634678+0000","last_clean_scrub_stamp":"2023-10-30T12:51:40.634678+0000","objects_scrubbed":9,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:38:14.782628+0000","scrub_duration":0.051559438999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583584,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":43,"num_read_kb":32803,"num_write":294,"num_write_kb":12545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7620,"num_omap_keys":15,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"66'519","reported_seq":2677,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:08.630116+0000","last_change":"2023-10-30T12:52:08.630261+0000","last_active":"2023-10-30T12:52:08.630116+0000","last_peered":"2023-10-30T12:52:08.630116+0000","last_clean":"2023-10-30T12:52:08.630116+0000","last_became_active":"2023-10-30T12:28:50.973269+0000","last_became_peered":"2023-10-30T12:28:50.973269+0000","last_unstale":"2023-10-30T12:52:08.630116+0000","last_undegraded":"2023-10-30T12:52:08.630116+0000","last_fullsized":"2023-10-30T12:52:08.630116+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'519","last_scrub_stamp":"2023-10-30T12:52:08.629908+0000","last_deep_scrub":"66'519","last_deep_scrub_stamp":"2023-10-30T12:52:08.629908+0000","last_clean_scrub_stamp":"2023-10-30T12:52:08.629908+0000","objects_scrubbed":20,"log_size":519,"log_dups_size":0,"ondisk_log_size":519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:08.922310+0000","scrub_duration":0.063779350999999998,"objects_trimmed":0,"snaptrim_duration":0.004001498,"stat_sum":{"num_bytes":12707497,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":38,"num_read_kb":11,"num_write":446,"num_write_kb":13522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"66'444","reported_seq":545,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:37.461917+0000","last_change":"2023-10-30T12:51:37.461917+0000","last_active":"2023-10-30T12:51:37.461917+0000","last_peered":"2023-10-30T12:51:37.461917+0000","last_clean":"2023-10-30T12:51:37.461917+0000","last_became_active":"2023-10-30T12:28:50.366269+0000","last_became_peered":"2023-10-30T12:28:50.366269+0000","last_unstale":"2023-10-30T12:51:37.461917+0000","last_undegraded":"2023-10-30T12:51:37.461917+0000","last_fullsized":"2023-10-30T12:51:37.461917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'444","last_scrub_stamp":"2023-10-30T12:51:37.461743+0000","last_deep_scrub":"66'444","last_deep_scrub_stamp":"2023-10-30T12:51:37.461743+0000","last_clean_scrub_stamp":"2023-10-30T12:51:37.461743+0000","objects_scrubbed":9,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:35:49.998956+0000","scrub_duration":0.040110959000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388642,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":52,"num_read_kb":36923,"num_write":386,"num_write_kb":8499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":16168,"num_omap_keys":32,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"66'586","reported_seq":2759,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:43.354752+0000","last_change":"2023-10-30T12:51:43.354752+0000","last_active":"2023-10-30T12:51:43.354752+0000","last_peered":"2023-10-30T12:51:43.354752+0000","last_clean":"2023-10-30T12:51:43.354752+0000","last_became_active":"2023-10-30T12:28:50.961805+0000","last_became_peered":"2023-10-30T12:28:50.961805+0000","last_unstale":"2023-10-30T12:51:43.354752+0000","last_undegraded":"2023-10-30T12:51:43.354752+0000","last_fullsized":"2023-10-30T12:51:43.354752+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'586","last_scrub_stamp":"2023-10-30T12:51:43.354576+0000","last_deep_scrub":"66'586","last_deep_scrub_stamp":"2023-10-30T12:51:43.354576+0000","last_clean_scrub_stamp":"2023-10-30T12:51:43.354576+0000","objects_scrubbed":22,"log_size":586,"log_dups_size":0,"ondisk_log_size":586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:44:38.376225+0000","scrub_duration":0.037946186999999999,"objects_trimmed":0,"snaptrim_duration":0.0032264350000000002,"stat_sum":{"num_bytes":4388445,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":52,"num_read_kb":4114,"num_write":506,"num_write_kb":14199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"66'261","reported_seq":346,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:08.296017+0000","last_change":"2023-10-30T12:52:08.296017+0000","last_active":"2023-10-30T12:52:08.296017+0000","last_peered":"2023-10-30T12:52:08.296017+0000","last_clean":"2023-10-30T12:52:08.296017+0000","last_became_active":"2023-10-30T12:28:50.396727+0000","last_became_peered":"2023-10-30T12:28:50.396727+0000","last_unstale":"2023-10-30T12:52:08.296017+0000","last_undegraded":"2023-10-30T12:52:08.296017+0000","last_fullsized":"2023-10-30T12:52:08.296017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'261","last_scrub_stamp":"2023-10-30T12:52:08.295709+0000","last_deep_scrub":"66'261","last_deep_scrub_stamp":"2023-10-30T12:52:08.295709+0000","last_clean_scrub_stamp":"2023-10-30T12:52:08.295709+0000","objects_scrubbed":3,"log_size":261,"log_dups_size":0,"ondisk_log_size":261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:43:47.836137+0000","scrub_duration":0.032577411000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":16164,"num_write":201,"num_write_kb":8311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"66'568","reported_seq":2744,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:49.633887+0000","last_change":"2023-10-30T12:51:49.633887+0000","last_active":"2023-10-30T12:51:49.633887+0000","last_peered":"2023-10-30T12:51:49.633887+0000","last_clean":"2023-10-30T12:51:49.633887+0000","last_became_active":"2023-10-30T12:29:32.960608+0000","last_became_peered":"2023-10-30T12:29:32.960608+0000","last_unstale":"2023-10-30T12:51:49.633887+0000","last_undegraded":"2023-10-30T12:51:49.633887+0000","last_fullsized":"2023-10-30T12:51:49.633887+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'568","last_scrub_stamp":"2023-10-30T12:51:49.633664+0000","last_deep_scrub":"66'568","last_deep_scrub_stamp":"2023-10-30T12:51:49.633664+0000","last_clean_scrub_stamp":"2023-10-30T12:51:49.633664+0000","objects_scrubbed":14,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:00:42.755675+0000","scrub_duration":0.028072751999999999,"objects_trimmed":0,"snaptrim_duration":0.0012934999999999999,"stat_sum":{"num_bytes":4430082,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":59,"num_read_kb":33,"num_write":506,"num_write_kb":6384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"66'684","reported_seq":775,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:44.639628+0000","last_change":"2023-10-30T12:51:44.639628+0000","last_active":"2023-10-30T12:51:44.639628+0000","last_peered":"2023-10-30T12:51:44.639628+0000","last_clean":"2023-10-30T12:51:44.639628+0000","last_became_active":"2023-10-30T12:28:50.383570+0000","last_became_peered":"2023-10-30T12:28:50.383570+0000","last_unstale":"2023-10-30T12:51:44.639628+0000","last_undegraded":"2023-10-30T12:51:44.639628+0000","last_fullsized":"2023-10-30T12:51:44.639628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'684","last_scrub_stamp":"2023-10-30T12:51:44.639059+0000","last_deep_scrub":"66'684","last_deep_scrub_stamp":"2023-10-30T12:51:44.639059+0000","last_clean_scrub_stamp":"2023-10-30T12:51:44.639059+0000","objects_scrubbed":7,"log_size":684,"log_dups_size":0,"ondisk_log_size":684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:23:55.453101+0000","scrub_duration":0.079897833000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":37306,"num_write":619,"num_write_kb":16724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15312,"num_omap_keys":30,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"66'556","reported_seq":2710,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:01.705427+0000","last_change":"2023-10-30T12:52:01.705427+0000","last_active":"2023-10-30T12:52:01.705427+0000","last_peered":"2023-10-30T12:52:01.705427+0000","last_clean":"2023-10-30T12:52:01.705427+0000","last_became_active":"2023-10-30T12:28:50.949485+0000","last_became_peered":"2023-10-30T12:28:50.949485+0000","last_unstale":"2023-10-30T12:52:01.705427+0000","last_undegraded":"2023-10-30T12:52:01.705427+0000","last_fullsized":"2023-10-30T12:52:01.705427+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'556","last_scrub_stamp":"2023-10-30T12:52:01.705046+0000","last_deep_scrub":"66'556","last_deep_scrub_stamp":"2023-10-30T12:52:01.705046+0000","last_clean_scrub_stamp":"2023-10-30T12:52:01.705046+0000","objects_scrubbed":15,"log_size":556,"log_dups_size":0,"ondisk_log_size":556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:05:15.771642+0000","scrub_duration":0.064202585000000006,"objects_trimmed":0,"snaptrim_duration":0.0040897299999999998,"stat_sum":{"num_bytes":12622957,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":477,"num_write_kb":14339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"66'453","reported_seq":569,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:49.184348+0000","last_change":"2023-10-30T12:51:41.463979+0000","last_active":"2023-10-30T12:51:49.184348+0000","last_peered":"2023-10-30T12:51:49.184348+0000","last_clean":"2023-10-30T12:51:49.184348+0000","last_became_active":"2023-10-30T12:28:50.362624+0000","last_became_peered":"2023-10-30T12:28:50.362624+0000","last_unstale":"2023-10-30T12:51:49.184348+0000","last_undegraded":"2023-10-30T12:51:49.184348+0000","last_fullsized":"2023-10-30T12:51:49.184348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'452","last_scrub_stamp":"2023-10-30T12:51:41.463765+0000","last_deep_scrub":"66'452","last_deep_scrub_stamp":"2023-10-30T12:51:41.463765+0000","last_clean_scrub_stamp":"2023-10-30T12:51:41.463765+0000","objects_scrubbed":13,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:18.726132+0000","scrub_duration":0.043492996999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583024,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":93,"num_read_kb":24662,"num_write":378,"num_write_kb":12558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6552,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"66'600","reported_seq":2771,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:57.546471+0000","last_change":"2023-10-30T12:51:57.546471+0000","last_active":"2023-10-30T12:51:57.546471+0000","last_peered":"2023-10-30T12:51:57.546471+0000","last_clean":"2023-10-30T12:51:57.546471+0000","last_became_active":"2023-10-30T12:28:50.979344+0000","last_became_peered":"2023-10-30T12:28:50.979344+0000","last_unstale":"2023-10-30T12:51:57.546471+0000","last_undegraded":"2023-10-30T12:51:57.546471+0000","last_fullsized":"2023-10-30T12:51:57.546471+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'600","last_scrub_stamp":"2023-10-30T12:51:57.546007+0000","last_deep_scrub":"66'600","last_deep_scrub_stamp":"2023-10-30T12:51:57.546007+0000","last_clean_scrub_stamp":"2023-10-30T12:51:57.546007+0000","objects_scrubbed":24,"log_size":600,"log_dups_size":0,"ondisk_log_size":600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:44:55.842195+0000","scrub_duration":0.075819872999999996,"objects_trimmed":0,"snaptrim_duration":0.0092295529999999997,"stat_sum":{"num_bytes":12647197,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":50,"num_read_kb":8210,"num_write":532,"num_write_kb":23192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"66'248","reported_seq":334,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:37.268010+0000","last_change":"2023-10-30T12:51:37.268108+0000","last_active":"2023-10-30T12:51:37.268010+0000","last_peered":"2023-10-30T12:51:37.268010+0000","last_clean":"2023-10-30T12:51:37.268010+0000","last_became_active":"2023-10-30T12:28:50.381125+0000","last_became_peered":"2023-10-30T12:28:50.381125+0000","last_unstale":"2023-10-30T12:51:37.268010+0000","last_undegraded":"2023-10-30T12:51:37.268010+0000","last_fullsized":"2023-10-30T12:51:37.268010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'248","last_scrub_stamp":"2023-10-30T12:51:37.267826+0000","last_deep_scrub":"66'248","last_deep_scrub_stamp":"2023-10-30T12:51:37.267826+0000","last_clean_scrub_stamp":"2023-10-30T12:51:37.267826+0000","objects_scrubbed":3,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:30:50.232542+0000","scrub_duration":0.054105548000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":12378,"num_write":396,"num_write_kb":4652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"66'548","reported_seq":2705,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:09.640556+0000","last_change":"2023-10-30T12:52:09.640712+0000","last_active":"2023-10-30T12:52:09.640556+0000","last_peered":"2023-10-30T12:52:09.640556+0000","last_clean":"2023-10-30T12:52:09.640556+0000","last_became_active":"2023-10-30T12:28:50.977585+0000","last_became_peered":"2023-10-30T12:28:50.977585+0000","last_unstale":"2023-10-30T12:52:09.640556+0000","last_undegraded":"2023-10-30T12:52:09.640556+0000","last_fullsized":"2023-10-30T12:52:09.640556+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'548","last_scrub_stamp":"2023-10-30T12:52:09.640316+0000","last_deep_scrub":"66'548","last_deep_scrub_stamp":"2023-10-30T12:52:09.640316+0000","last_clean_scrub_stamp":"2023-10-30T12:52:09.640316+0000","objects_scrubbed":17,"log_size":548,"log_dups_size":0,"ondisk_log_size":548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:06:42.121080+0000","scrub_duration":0.072014062000000004,"objects_trimmed":0,"snaptrim_duration":0.0089211889999999995,"stat_sum":{"num_bytes":12933034,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":36,"num_read_kb":8205,"num_write":483,"num_write_kb":18997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"66'359","reported_seq":445,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:38.474826+0000","last_change":"2023-10-30T12:51:38.474877+0000","last_active":"2023-10-30T12:51:38.474826+0000","last_peered":"2023-10-30T12:51:38.474826+0000","last_clean":"2023-10-30T12:51:38.474826+0000","last_became_active":"2023-10-30T12:28:50.366974+0000","last_became_peered":"2023-10-30T12:28:50.366974+0000","last_unstale":"2023-10-30T12:51:38.474826+0000","last_undegraded":"2023-10-30T12:51:38.474826+0000","last_fullsized":"2023-10-30T12:51:38.474826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'359","last_scrub_stamp":"2023-10-30T12:51:38.474746+0000","last_deep_scrub":"66'359","last_deep_scrub_stamp":"2023-10-30T12:51:38.474746+0000","last_clean_scrub_stamp":"2023-10-30T12:51:38.474746+0000","objects_scrubbed":4,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:46:34.826564+0000","scrub_duration":0.021991255000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":15324,"num_write":292,"num_write_kb":8361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10374,"num_omap_keys":19,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"66'558","reported_seq":2729,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:11.609600+0000","last_change":"2023-10-30T12:52:11.609787+0000","last_active":"2023-10-30T12:52:11.609600+0000","last_peered":"2023-10-30T12:52:11.609600+0000","last_clean":"2023-10-30T12:52:11.609600+0000","last_became_active":"2023-10-30T12:28:50.981193+0000","last_became_peered":"2023-10-30T12:28:50.981193+0000","last_unstale":"2023-10-30T12:52:11.609600+0000","last_undegraded":"2023-10-30T12:52:11.609600+0000","last_fullsized":"2023-10-30T12:52:11.609600+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'558","last_scrub_stamp":"2023-10-30T12:52:11.609388+0000","last_deep_scrub":"66'558","last_deep_scrub_stamp":"2023-10-30T12:52:11.609388+0000","last_clean_scrub_stamp":"2023-10-30T12:52:11.609388+0000","objects_scrubbed":24,"log_size":558,"log_dups_size":0,"ondisk_log_size":558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:35:18.683385+0000","scrub_duration":0.079771501999999994,"objects_trimmed":0,"snaptrim_duration":0.0096579700000000001,"stat_sum":{"num_bytes":12674006,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":50,"num_read_kb":4114,"num_write":492,"num_write_kb":18051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"66'98","reported_seq":180,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:44.901113+0000","last_change":"2023-10-30T12:51:44.901168+0000","last_active":"2023-10-30T12:51:44.901113+0000","last_peered":"2023-10-30T12:51:44.901113+0000","last_clean":"2023-10-30T12:51:44.901113+0000","last_became_active":"2023-10-30T12:28:50.367730+0000","last_became_peered":"2023-10-30T12:28:50.367730+0000","last_unstale":"2023-10-30T12:51:44.901113+0000","last_undegraded":"2023-10-30T12:51:44.901113+0000","last_fullsized":"2023-10-30T12:51:44.901113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'98","last_scrub_stamp":"2023-10-30T12:51:44.900977+0000","last_deep_scrub":"66'98","last_deep_scrub_stamp":"2023-10-30T12:51:44.900977+0000","last_clean_scrub_stamp":"2023-10-30T12:51:44.900977+0000","objects_scrubbed":4,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:46:56.082172+0000","scrub_duration":0.018334822000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194374,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":12292,"num_write":45,"num_write_kb":4186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8128,"num_omap_keys":16,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"66'621","reported_seq":2799,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:23.291318+0000","last_change":"2023-10-30T12:52:23.291517+0000","last_active":"2023-10-30T12:52:23.291318+0000","last_peered":"2023-10-30T12:52:23.291318+0000","last_clean":"2023-10-30T12:52:23.291318+0000","last_became_active":"2023-10-30T12:28:50.966246+0000","last_became_peered":"2023-10-30T12:28:50.966246+0000","last_unstale":"2023-10-30T12:52:23.291318+0000","last_undegraded":"2023-10-30T12:52:23.291318+0000","last_fullsized":"2023-10-30T12:52:23.291318+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'621","last_scrub_stamp":"2023-10-30T12:52:23.291084+0000","last_deep_scrub":"66'621","last_deep_scrub_stamp":"2023-10-30T12:52:23.291084+0000","last_clean_scrub_stamp":"2023-10-30T12:52:23.291084+0000","objects_scrubbed":27,"log_size":621,"log_dups_size":0,"ondisk_log_size":621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:33:14.777415+0000","scrub_duration":0.089234725000000001,"objects_trimmed":0,"snaptrim_duration":0.007703601,"stat_sum":{"num_bytes":8520713,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":56,"num_read_kb":18,"num_write":541,"num_write_kb":11343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"66'87","reported_seq":172,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:42.434625+0000","last_change":"2023-10-30T12:51:42.434673+0000","last_active":"2023-10-30T12:51:42.434625+0000","last_peered":"2023-10-30T12:51:42.434625+0000","last_clean":"2023-10-30T12:51:42.434625+0000","last_became_active":"2023-10-30T12:28:50.393642+0000","last_became_peered":"2023-10-30T12:28:50.393642+0000","last_unstale":"2023-10-30T12:51:42.434625+0000","last_undegraded":"2023-10-30T12:51:42.434625+0000","last_fullsized":"2023-10-30T12:51:42.434625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'87","last_scrub_stamp":"2023-10-30T12:51:42.434545+0000","last_deep_scrub":"66'87","last_deep_scrub_stamp":"2023-10-30T12:51:42.434545+0000","last_clean_scrub_stamp":"2023-10-30T12:51:42.434545+0000","objects_scrubbed":5,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:27:29.743857+0000","scrub_duration":0.0090945590000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":15,"num_write_kb":101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17524,"num_omap_keys":33,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"66'579","reported_seq":2744,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:56.560776+0000","last_change":"2023-10-30T12:51:56.560943+0000","last_active":"2023-10-30T12:51:56.560776+0000","last_peered":"2023-10-30T12:51:56.560776+0000","last_clean":"2023-10-30T12:51:56.560776+0000","last_became_active":"2023-10-30T12:28:50.976343+0000","last_became_peered":"2023-10-30T12:28:50.976343+0000","last_unstale":"2023-10-30T12:51:56.560776+0000","last_undegraded":"2023-10-30T12:51:56.560776+0000","last_fullsized":"2023-10-30T12:51:56.560776+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'579","last_scrub_stamp":"2023-10-30T12:51:56.560556+0000","last_deep_scrub":"66'579","last_deep_scrub_stamp":"2023-10-30T12:51:56.560556+0000","last_clean_scrub_stamp":"2023-10-30T12:51:56.560556+0000","objects_scrubbed":23,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:00:58.734907+0000","scrub_duration":0.074070374999999994,"objects_trimmed":0,"snaptrim_duration":0.0091875959999999993,"stat_sum":{"num_bytes":17078090,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":44,"num_read_kb":14,"num_write":517,"num_write_kb":22829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"66'119","reported_seq":232,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:44.641713+0000","last_change":"2023-10-30T12:51:44.641713+0000","last_active":"2023-10-30T12:51:44.641713+0000","last_peered":"2023-10-30T12:51:44.641713+0000","last_clean":"2023-10-30T12:51:44.641713+0000","last_became_active":"2023-10-30T12:28:41.201796+0000","last_became_peered":"2023-10-30T12:28:41.201796+0000","last_unstale":"2023-10-30T12:51:44.641713+0000","last_undegraded":"2023-10-30T12:51:44.641713+0000","last_fullsized":"2023-10-30T12:51:44.641713+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'119","last_scrub_stamp":"2023-10-30T12:51:44.641256+0000","last_deep_scrub":"66'119","last_deep_scrub_stamp":"2023-10-30T12:51:44.641256+0000","last_clean_scrub_stamp":"2023-10-30T12:51:44.641256+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:26:13.839326+0000","scrub_duration":0.013391731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"66'201","reported_seq":291,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:50.447569+0000","last_change":"2023-10-30T12:51:50.447661+0000","last_active":"2023-10-30T12:51:50.447569+0000","last_peered":"2023-10-30T12:51:50.447569+0000","last_clean":"2023-10-30T12:51:50.447569+0000","last_became_active":"2023-10-30T12:28:50.367593+0000","last_became_peered":"2023-10-30T12:28:50.367593+0000","last_unstale":"2023-10-30T12:51:50.447569+0000","last_undegraded":"2023-10-30T12:51:50.447569+0000","last_fullsized":"2023-10-30T12:51:50.447569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'201","last_scrub_stamp":"2023-10-30T12:51:50.447458+0000","last_deep_scrub":"66'201","last_deep_scrub_stamp":"2023-10-30T12:51:50.447458+0000","last_clean_scrub_stamp":"2023-10-30T12:51:50.447458+0000","objects_scrubbed":9,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:35:16.204652+0000","scrub_duration":0.025712215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194338,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":24,"num_read_kb":12326,"num_write":144,"num_write_kb":4223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":49160,"num_omap_keys":96,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"66'535","reported_seq":2690,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:48.599069+0000","last_change":"2023-10-30T12:51:48.599069+0000","last_active":"2023-10-30T12:51:48.599069+0000","last_peered":"2023-10-30T12:51:48.599069+0000","last_clean":"2023-10-30T12:51:48.599069+0000","last_became_active":"2023-10-30T12:29:32.997301+0000","last_became_peered":"2023-10-30T12:29:32.997301+0000","last_unstale":"2023-10-30T12:51:48.599069+0000","last_undegraded":"2023-10-30T12:51:48.599069+0000","last_fullsized":"2023-10-30T12:51:48.599069+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'535","last_scrub_stamp":"2023-10-30T12:51:48.598899+0000","last_deep_scrub":"66'535","last_deep_scrub_stamp":"2023-10-30T12:51:48.598899+0000","last_clean_scrub_stamp":"2023-10-30T12:51:48.598899+0000","objects_scrubbed":17,"log_size":535,"log_dups_size":0,"ondisk_log_size":535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:04:11.091469+0000","scrub_duration":0.019854914000000001,"objects_trimmed":0,"snaptrim_duration":0.001366412,"stat_sum":{"num_bytes":4237965,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":36,"num_read_kb":11,"num_write":481,"num_write_kb":13486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"66'504","reported_seq":601,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:47.588509+0000","last_change":"2023-10-30T12:51:47.588509+0000","last_active":"2023-10-30T12:51:47.588509+0000","last_peered":"2023-10-30T12:51:47.588509+0000","last_clean":"2023-10-30T12:51:47.588509+0000","last_became_active":"2023-10-30T12:28:50.391824+0000","last_became_peered":"2023-10-30T12:28:50.391824+0000","last_unstale":"2023-10-30T12:51:47.588509+0000","last_undegraded":"2023-10-30T12:51:47.588509+0000","last_fullsized":"2023-10-30T12:51:47.588509+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'504","last_scrub_stamp":"2023-10-30T12:51:47.588159+0000","last_deep_scrub":"66'504","last_deep_scrub_stamp":"2023-10-30T12:51:47.588159+0000","last_clean_scrub_stamp":"2023-10-30T12:51:47.588159+0000","objects_scrubbed":7,"log_size":504,"log_dups_size":0,"ondisk_log_size":504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:53:44.520327+0000","scrub_duration":0.072404271000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":32,"num_read_kb":50322,"num_write":431,"num_write_kb":16616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13208,"num_omap_keys":26,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"66'547","reported_seq":2698,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:16.385607+0000","last_change":"2023-10-30T12:52:16.385672+0000","last_active":"2023-10-30T12:52:16.385607+0000","last_peered":"2023-10-30T12:52:16.385607+0000","last_clean":"2023-10-30T12:52:16.385607+0000","last_became_active":"2023-10-30T12:28:50.952625+0000","last_became_peered":"2023-10-30T12:28:50.952625+0000","last_unstale":"2023-10-30T12:52:16.385607+0000","last_undegraded":"2023-10-30T12:52:16.385607+0000","last_fullsized":"2023-10-30T12:52:16.385607+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'547","last_scrub_stamp":"2023-10-30T12:52:16.385522+0000","last_deep_scrub":"66'547","last_deep_scrub_stamp":"2023-10-30T12:52:16.385522+0000","last_clean_scrub_stamp":"2023-10-30T12:52:16.385522+0000","objects_scrubbed":15,"log_size":547,"log_dups_size":0,"ondisk_log_size":547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:48:56.290695+0000","scrub_duration":0.061983638000000001,"objects_trimmed":0,"snaptrim_duration":0.0026695970000000001,"stat_sum":{"num_bytes":16953646,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":29,"num_read_kb":10,"num_write":468,"num_write_kb":20237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"66'605","reported_seq":920,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:48.574934+0000","last_change":"2023-10-30T12:51:35.405246+0000","last_active":"2023-10-30T12:51:48.574934+0000","last_peered":"2023-10-30T12:51:48.574934+0000","last_clean":"2023-10-30T12:51:48.574934+0000","last_became_active":"2023-10-30T12:28:50.367230+0000","last_became_peered":"2023-10-30T12:28:50.367230+0000","last_unstale":"2023-10-30T12:51:48.574934+0000","last_undegraded":"2023-10-30T12:51:48.574934+0000","last_fullsized":"2023-10-30T12:51:48.574934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'598","last_scrub_stamp":"2023-10-30T12:51:35.405133+0000","last_deep_scrub":"66'598","last_deep_scrub_stamp":"2023-10-30T12:51:35.405133+0000","last_clean_scrub_stamp":"2023-10-30T12:51:35.405133+0000","objects_scrubbed":7,"log_size":605,"log_dups_size":0,"ondisk_log_size":605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:34:01.194417+0000","scrub_duration":0.035040017999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12778646,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":251,"num_read_kb":13023,"num_write":542,"num_write_kb":12830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9828,"num_omap_keys":18,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"66'565","reported_seq":2735,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:57.726097+0000","last_change":"2023-10-30T12:51:57.726097+0000","last_active":"2023-10-30T12:51:57.726097+0000","last_peered":"2023-10-30T12:51:57.726097+0000","last_clean":"2023-10-30T12:51:57.726097+0000","last_became_active":"2023-10-30T12:28:50.928608+0000","last_became_peered":"2023-10-30T12:28:50.928608+0000","last_unstale":"2023-10-30T12:51:57.726097+0000","last_undegraded":"2023-10-30T12:51:57.726097+0000","last_fullsized":"2023-10-30T12:51:57.726097+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'565","last_scrub_stamp":"2023-10-30T12:51:57.725915+0000","last_deep_scrub":"66'565","last_deep_scrub_stamp":"2023-10-30T12:51:57.725915+0000","last_clean_scrub_stamp":"2023-10-30T12:51:57.725915+0000","objects_scrubbed":21,"log_size":565,"log_dups_size":0,"ondisk_log_size":565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:24:52.304705+0000","scrub_duration":0.032239474999999997,"objects_trimmed":0,"snaptrim_duration":0.002074275,"stat_sum":{"num_bytes":8441122,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":47,"num_read_kb":21,"num_write":501,"num_write_kb":11799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"66'497","reported_seq":586,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:02.360999+0000","last_change":"2023-10-30T12:52:02.361114+0000","last_active":"2023-10-30T12:52:02.360999+0000","last_peered":"2023-10-30T12:52:02.360999+0000","last_clean":"2023-10-30T12:52:02.360999+0000","last_became_active":"2023-10-30T12:28:50.390864+0000","last_became_peered":"2023-10-30T12:28:50.390864+0000","last_unstale":"2023-10-30T12:52:02.360999+0000","last_undegraded":"2023-10-30T12:52:02.360999+0000","last_fullsized":"2023-10-30T12:52:02.360999+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'497","last_scrub_stamp":"2023-10-30T12:52:02.360776+0000","last_deep_scrub":"66'497","last_deep_scrub_stamp":"2023-10-30T12:52:02.360776+0000","last_clean_scrub_stamp":"2023-10-30T12:52:02.360776+0000","objects_scrubbed":4,"log_size":497,"log_dups_size":0,"ondisk_log_size":497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:40:59.407446+0000","scrub_duration":0.065243972999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":45183,"num_write":427,"num_write_kb":16635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"66'536","reported_seq":2693,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:45.113576+0000","last_change":"2023-10-30T12:51:45.113576+0000","last_active":"2023-10-30T12:51:45.113576+0000","last_peered":"2023-10-30T12:51:45.113576+0000","last_clean":"2023-10-30T12:51:45.113576+0000","last_became_active":"2023-10-30T12:28:50.949122+0000","last_became_peered":"2023-10-30T12:28:50.949122+0000","last_unstale":"2023-10-30T12:51:45.113576+0000","last_undegraded":"2023-10-30T12:51:45.113576+0000","last_fullsized":"2023-10-30T12:51:45.113576+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'536","last_scrub_stamp":"2023-10-30T12:51:45.113133+0000","last_deep_scrub":"66'536","last_deep_scrub_stamp":"2023-10-30T12:51:45.113133+0000","last_clean_scrub_stamp":"2023-10-30T12:51:45.113133+0000","objects_scrubbed":16,"log_size":536,"log_dups_size":0,"ondisk_log_size":536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:45:58.801601+0000","scrub_duration":0.060099805999999999,"objects_trimmed":0,"snaptrim_duration":0.004393348,"stat_sum":{"num_bytes":12684737,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":35,"num_read_kb":65,"num_write":464,"num_write_kb":14146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"66'267","reported_seq":346,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:36.014856+0000","last_change":"2023-10-30T12:51:36.014856+0000","last_active":"2023-10-30T12:51:36.014856+0000","last_peered":"2023-10-30T12:51:36.014856+0000","last_clean":"2023-10-30T12:51:36.014856+0000","last_became_active":"2023-10-30T12:28:50.358310+0000","last_became_peered":"2023-10-30T12:28:50.358310+0000","last_unstale":"2023-10-30T12:51:36.014856+0000","last_undegraded":"2023-10-30T12:51:36.014856+0000","last_fullsized":"2023-10-30T12:51:36.014856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'267","last_scrub_stamp":"2023-10-30T12:51:36.014733+0000","last_deep_scrub":"66'267","last_deep_scrub_stamp":"2023-10-30T12:51:36.014733+0000","last_clean_scrub_stamp":"2023-10-30T12:51:36.014733+0000","objects_scrubbed":2,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:22:01.486529+0000","scrub_duration":0.015176472999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":5758,"num_write":222,"num_write_kb":4242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7112,"num_omap_keys":14,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"66'554","reported_seq":2709,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:38.621754+0000","last_change":"2023-10-30T12:51:38.621754+0000","last_active":"2023-10-30T12:51:38.621754+0000","last_peered":"2023-10-30T12:51:38.621754+0000","last_clean":"2023-10-30T12:51:38.621754+0000","last_became_active":"2023-10-30T12:28:50.965626+0000","last_became_peered":"2023-10-30T12:28:50.965626+0000","last_unstale":"2023-10-30T12:51:38.621754+0000","last_undegraded":"2023-10-30T12:51:38.621754+0000","last_fullsized":"2023-10-30T12:51:38.621754+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'554","last_scrub_stamp":"2023-10-30T12:51:38.621402+0000","last_deep_scrub":"66'554","last_deep_scrub_stamp":"2023-10-30T12:51:38.621402+0000","last_clean_scrub_stamp":"2023-10-30T12:51:38.621402+0000","objects_scrubbed":15,"log_size":554,"log_dups_size":0,"ondisk_log_size":554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:32:24.000396+0000","scrub_duration":0.10283745599999999,"objects_trimmed":0,"snaptrim_duration":0.0079099229999999993,"stat_sum":{"num_bytes":29382851,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":33,"num_read_kb":4109,"num_write":477,"num_write_kb":37019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"66'404","reported_seq":485,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:53.568757+0000","last_change":"2023-10-30T12:51:53.568971+0000","last_active":"2023-10-30T12:51:53.568757+0000","last_peered":"2023-10-30T12:51:53.568757+0000","last_clean":"2023-10-30T12:51:53.568757+0000","last_became_active":"2023-10-30T12:28:50.395090+0000","last_became_peered":"2023-10-30T12:28:50.395090+0000","last_unstale":"2023-10-30T12:51:53.568757+0000","last_undegraded":"2023-10-30T12:51:53.568757+0000","last_fullsized":"2023-10-30T12:51:53.568757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'404","last_scrub_stamp":"2023-10-30T12:51:53.568460+0000","last_deep_scrub":"66'404","last_deep_scrub_stamp":"2023-10-30T12:51:53.568460+0000","last_clean_scrub_stamp":"2023-10-30T12:51:53.568460+0000","objects_scrubbed":5,"log_size":404,"log_dups_size":0,"ondisk_log_size":404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:27:52.316450+0000","scrub_duration":0.055684696999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582934,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":8195,"num_write":330,"num_write_kb":12489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12012,"num_omap_keys":22,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"66'568","reported_seq":2740,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:04.826459+0000","last_change":"2023-10-30T12:52:04.826539+0000","last_active":"2023-10-30T12:52:04.826459+0000","last_peered":"2023-10-30T12:52:04.826459+0000","last_clean":"2023-10-30T12:52:04.826459+0000","last_became_active":"2023-10-30T12:28:50.929337+0000","last_became_peered":"2023-10-30T12:28:50.929337+0000","last_unstale":"2023-10-30T12:52:04.826459+0000","last_undegraded":"2023-10-30T12:52:04.826459+0000","last_fullsized":"2023-10-30T12:52:04.826459+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'568","last_scrub_stamp":"2023-10-30T12:52:04.826355+0000","last_deep_scrub":"66'568","last_deep_scrub_stamp":"2023-10-30T12:52:04.826355+0000","last_clean_scrub_stamp":"2023-10-30T12:52:04.826355+0000","objects_scrubbed":17,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:47:52.867591+0000","scrub_duration":0.039134962000000002,"objects_trimmed":0,"snaptrim_duration":0.0023039890000000002,"stat_sum":{"num_bytes":14041074,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":50,"num_read_kb":18,"num_write":487,"num_write_kb":15488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"66'356","reported_seq":444,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:41.110963+0000","last_change":"2023-10-30T12:51:41.110963+0000","last_active":"2023-10-30T12:51:41.110963+0000","last_peered":"2023-10-30T12:51:41.110963+0000","last_clean":"2023-10-30T12:51:41.110963+0000","last_became_active":"2023-10-30T12:28:50.380247+0000","last_became_peered":"2023-10-30T12:28:50.380247+0000","last_unstale":"2023-10-30T12:51:41.110963+0000","last_undegraded":"2023-10-30T12:51:41.110963+0000","last_fullsized":"2023-10-30T12:51:41.110963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'356","last_scrub_stamp":"2023-10-30T12:51:41.110806+0000","last_deep_scrub":"66'356","last_deep_scrub_stamp":"2023-10-30T12:51:41.110806+0000","last_clean_scrub_stamp":"2023-10-30T12:51:41.110806+0000","objects_scrubbed":5,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:36:44.682836+0000","scrub_duration":0.090051573999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":8272,"num_write":528,"num_write_kb":5259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18674,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"66'544","reported_seq":2696,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:16.880748+0000","last_change":"2023-10-30T12:52:16.880748+0000","last_active":"2023-10-30T12:52:16.880748+0000","last_peered":"2023-10-30T12:52:16.880748+0000","last_clean":"2023-10-30T12:52:16.880748+0000","last_became_active":"2023-10-30T12:28:50.973086+0000","last_became_peered":"2023-10-30T12:28:50.973086+0000","last_unstale":"2023-10-30T12:52:16.880748+0000","last_undegraded":"2023-10-30T12:52:16.880748+0000","last_fullsized":"2023-10-30T12:52:16.880748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'544","last_scrub_stamp":"2023-10-30T12:52:16.880542+0000","last_deep_scrub":"66'544","last_deep_scrub_stamp":"2023-10-30T12:52:16.880542+0000","last_clean_scrub_stamp":"2023-10-30T12:52:16.880542+0000","objects_scrubbed":12,"log_size":544,"log_dups_size":0,"ondisk_log_size":544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:23:12.413779+0000","scrub_duration":0.056029834000000001,"objects_trimmed":0,"snaptrim_duration":0.0034133169999999999,"stat_sum":{"num_bytes":16807043,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":30,"num_read_kb":14,"num_write":492,"num_write_kb":22119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"66'275","reported_seq":359,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:18.562955+0000","last_change":"2023-10-30T12:51:43.564434+0000","last_active":"2023-10-30T12:52:18.562955+0000","last_peered":"2023-10-30T12:52:18.562955+0000","last_clean":"2023-10-30T12:52:18.562955+0000","last_became_active":"2023-10-30T12:28:50.385924+0000","last_became_peered":"2023-10-30T12:28:50.385924+0000","last_unstale":"2023-10-30T12:52:18.562955+0000","last_undegraded":"2023-10-30T12:52:18.562955+0000","last_fullsized":"2023-10-30T12:52:18.562955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'261","last_scrub_stamp":"2023-10-30T12:51:43.564089+0000","last_deep_scrub":"66'261","last_deep_scrub_stamp":"2023-10-30T12:51:43.564089+0000","last_clean_scrub_stamp":"2023-10-30T12:51:43.564089+0000","objects_scrubbed":5,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:34:21.648949+0000","scrub_duration":0.022937485000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3368320,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":209,"num_write_kb":3423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13032,"num_omap_keys":26,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"66'541","reported_seq":2706,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:49.313203+0000","last_change":"2023-10-30T12:51:49.313203+0000","last_active":"2023-10-30T12:51:49.313203+0000","last_peered":"2023-10-30T12:51:49.313203+0000","last_clean":"2023-10-30T12:51:49.313203+0000","last_became_active":"2023-10-30T12:28:50.929259+0000","last_became_peered":"2023-10-30T12:28:50.929259+0000","last_unstale":"2023-10-30T12:51:49.313203+0000","last_undegraded":"2023-10-30T12:51:49.313203+0000","last_fullsized":"2023-10-30T12:51:49.313203+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'541","last_scrub_stamp":"2023-10-30T12:51:49.313036+0000","last_deep_scrub":"66'541","last_deep_scrub_stamp":"2023-10-30T12:51:49.313036+0000","last_clean_scrub_stamp":"2023-10-30T12:51:49.313036+0000","objects_scrubbed":17,"log_size":541,"log_dups_size":0,"ondisk_log_size":541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:00:33.907215+0000","scrub_duration":0.028828065,"objects_trimmed":0,"snaptrim_duration":0.0029033230000000002,"stat_sum":{"num_bytes":8423823,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":43,"num_read_kb":22,"num_write":477,"num_write_kb":11480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"66'379","reported_seq":471,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:49.180993+0000","last_change":"2023-10-30T12:51:46.586619+0000","last_active":"2023-10-30T12:51:49.180993+0000","last_peered":"2023-10-30T12:51:49.180993+0000","last_clean":"2023-10-30T12:51:49.180993+0000","last_became_active":"2023-10-30T12:28:50.385305+0000","last_became_peered":"2023-10-30T12:28:50.385305+0000","last_unstale":"2023-10-30T12:51:49.180993+0000","last_undegraded":"2023-10-30T12:51:49.180993+0000","last_fullsized":"2023-10-30T12:51:49.180993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'378","last_scrub_stamp":"2023-10-30T12:51:46.586293+0000","last_deep_scrub":"66'378","last_deep_scrub_stamp":"2023-10-30T12:51:46.586293+0000","last_clean_scrub_stamp":"2023-10-30T12:51:46.586293+0000","objects_scrubbed":5,"log_size":379,"log_dups_size":0,"ondisk_log_size":379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:03:32.538098+0000","scrub_duration":0.051218165000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":37322,"num_write":310,"num_write_kb":12446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":508,"num_omap_keys":1,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"66'562","reported_seq":2721,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:38.230463+0000","last_change":"2023-10-30T12:51:38.230463+0000","last_active":"2023-10-30T12:51:38.230463+0000","last_peered":"2023-10-30T12:51:38.230463+0000","last_clean":"2023-10-30T12:51:38.230463+0000","last_became_active":"2023-10-30T12:28:50.940846+0000","last_became_peered":"2023-10-30T12:28:50.940846+0000","last_unstale":"2023-10-30T12:51:38.230463+0000","last_undegraded":"2023-10-30T12:51:38.230463+0000","last_fullsized":"2023-10-30T12:51:38.230463+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'562","last_scrub_stamp":"2023-10-30T12:51:38.229988+0000","last_deep_scrub":"66'562","last_deep_scrub_stamp":"2023-10-30T12:51:38.229988+0000","last_clean_scrub_stamp":"2023-10-30T12:51:38.229988+0000","objects_scrubbed":21,"log_size":562,"log_dups_size":0,"ondisk_log_size":562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:31:03.842668+0000","scrub_duration":0.057637995999999997,"objects_trimmed":0,"snaptrim_duration":0.0046016829999999996,"stat_sum":{"num_bytes":12679067,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":37,"num_read_kb":7,"num_write":498,"num_write_kb":18971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"66'448","reported_seq":535,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:42.982473+0000","last_change":"2023-10-30T12:51:42.982473+0000","last_active":"2023-10-30T12:51:42.982473+0000","last_peered":"2023-10-30T12:51:42.982473+0000","last_clean":"2023-10-30T12:51:42.982473+0000","last_became_active":"2023-10-30T12:28:50.392068+0000","last_became_peered":"2023-10-30T12:28:50.392068+0000","last_unstale":"2023-10-30T12:51:42.982473+0000","last_undegraded":"2023-10-30T12:51:42.982473+0000","last_fullsized":"2023-10-30T12:51:42.982473+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'448","last_scrub_stamp":"2023-10-30T12:51:42.982269+0000","last_deep_scrub":"66'448","last_deep_scrub_stamp":"2023-10-30T12:51:42.982269+0000","last_clean_scrub_stamp":"2023-10-30T12:51:42.982269+0000","objects_scrubbed":7,"log_size":448,"log_dups_size":0,"ondisk_log_size":448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:55:25.126896+0000","scrub_duration":0.032966065000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388698,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":19,"num_read_kb":11375,"num_write":387,"num_write_kb":8544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":27300,"num_omap_keys":50,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"66'570","reported_seq":2736,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:49.808358+0000","last_change":"2023-10-30T12:51:49.808358+0000","last_active":"2023-10-30T12:51:49.808358+0000","last_peered":"2023-10-30T12:51:49.808358+0000","last_clean":"2023-10-30T12:51:49.808358+0000","last_became_active":"2023-10-30T12:28:50.931898+0000","last_became_peered":"2023-10-30T12:28:50.931898+0000","last_unstale":"2023-10-30T12:51:49.808358+0000","last_undegraded":"2023-10-30T12:51:49.808358+0000","last_fullsized":"2023-10-30T12:51:49.808358+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'570","last_scrub_stamp":"2023-10-30T12:51:49.808201+0000","last_deep_scrub":"66'570","last_deep_scrub_stamp":"2023-10-30T12:51:49.808201+0000","last_clean_scrub_stamp":"2023-10-30T12:51:49.808201+0000","objects_scrubbed":18,"log_size":570,"log_dups_size":0,"ondisk_log_size":570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:46:36.932799+0000","scrub_duration":0.022998979999999999,"objects_trimmed":0,"snaptrim_duration":0.003570148,"stat_sum":{"num_bytes":4263636,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":44,"num_read_kb":18,"num_write":507,"num_write_kb":10484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"66'557","reported_seq":640,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:42.047533+0000","last_change":"2023-10-30T12:51:42.047533+0000","last_active":"2023-10-30T12:51:42.047533+0000","last_peered":"2023-10-30T12:51:42.047533+0000","last_clean":"2023-10-30T12:51:42.047533+0000","last_became_active":"2023-10-30T12:28:50.364797+0000","last_became_peered":"2023-10-30T12:28:50.364797+0000","last_unstale":"2023-10-30T12:51:42.047533+0000","last_undegraded":"2023-10-30T12:51:42.047533+0000","last_fullsized":"2023-10-30T12:51:42.047533+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'557","last_scrub_stamp":"2023-10-30T12:51:42.047398+0000","last_deep_scrub":"66'557","last_deep_scrub_stamp":"2023-10-30T12:51:42.047398+0000","last_clean_scrub_stamp":"2023-10-30T12:51:42.047398+0000","objects_scrubbed":4,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:36:26.216418+0000","scrub_duration":0.054259416999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":10,"num_read_kb":8489,"num_write":691,"num_write_kb":13252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"66'568","reported_seq":2738,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:50.791335+0000","last_change":"2023-10-30T12:51:50.791392+0000","last_active":"2023-10-30T12:51:50.791335+0000","last_peered":"2023-10-30T12:51:50.791335+0000","last_clean":"2023-10-30T12:51:50.791335+0000","last_became_active":"2023-10-30T12:28:50.935657+0000","last_became_peered":"2023-10-30T12:28:50.935657+0000","last_unstale":"2023-10-30T12:51:50.791335+0000","last_undegraded":"2023-10-30T12:51:50.791335+0000","last_fullsized":"2023-10-30T12:51:50.791335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'568","last_scrub_stamp":"2023-10-30T12:51:50.791255+0000","last_deep_scrub":"66'568","last_deep_scrub_stamp":"2023-10-30T12:51:50.791255+0000","last_clean_scrub_stamp":"2023-10-30T12:51:50.791255+0000","objects_scrubbed":15,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:03:42.487468+0000","scrub_duration":0.039011377999999999,"objects_trimmed":0,"snaptrim_duration":0.0038543219999999999,"stat_sum":{"num_bytes":16829664,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":49,"num_read_kb":4112,"num_write":499,"num_write_kb":17598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"66'249","reported_seq":338,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:54.509810+0000","last_change":"2023-10-30T12:51:54.509947+0000","last_active":"2023-10-30T12:51:54.509810+0000","last_peered":"2023-10-30T12:51:54.509810+0000","last_clean":"2023-10-30T12:51:54.509810+0000","last_became_active":"2023-10-30T12:28:50.392807+0000","last_became_peered":"2023-10-30T12:28:50.392807+0000","last_unstale":"2023-10-30T12:51:54.509810+0000","last_undegraded":"2023-10-30T12:51:54.509810+0000","last_fullsized":"2023-10-30T12:51:54.509810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'249","last_scrub_stamp":"2023-10-30T12:51:54.509581+0000","last_deep_scrub":"66'249","last_deep_scrub_stamp":"2023-10-30T12:51:54.509581+0000","last_clean_scrub_stamp":"2023-10-30T12:51:54.509581+0000","objects_scrubbed":5,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:12:36.299114+0000","scrub_duration":0.039282915000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":24596,"num_write":178,"num_write_kb":8294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"66'524","reported_seq":2684,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:02.742242+0000","last_change":"2023-10-30T12:52:02.742242+0000","last_active":"2023-10-30T12:52:02.742242+0000","last_peered":"2023-10-30T12:52:02.742242+0000","last_clean":"2023-10-30T12:52:02.742242+0000","last_became_active":"2023-10-30T12:28:50.954291+0000","last_became_peered":"2023-10-30T12:28:50.954291+0000","last_unstale":"2023-10-30T12:52:02.742242+0000","last_undegraded":"2023-10-30T12:52:02.742242+0000","last_fullsized":"2023-10-30T12:52:02.742242+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'524","last_scrub_stamp":"2023-10-30T12:52:02.741848+0000","last_deep_scrub":"66'524","last_deep_scrub_stamp":"2023-10-30T12:52:02.741848+0000","last_clean_scrub_stamp":"2023-10-30T12:52:02.741848+0000","objects_scrubbed":19,"log_size":524,"log_dups_size":0,"ondisk_log_size":524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:24:29.376137+0000","scrub_duration":0.074825178000000006,"objects_trimmed":0,"snaptrim_duration":0.0047974480000000002,"stat_sum":{"num_bytes":12709282,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":38,"num_read_kb":15,"num_write":455,"num_write_kb":14651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"66'539","reported_seq":2709,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:18.285265+0000","last_change":"2023-10-30T12:52:18.285390+0000","last_active":"2023-10-30T12:52:18.285265+0000","last_peered":"2023-10-30T12:52:18.285265+0000","last_clean":"2023-10-30T12:52:18.285265+0000","last_became_active":"2023-10-30T12:28:50.962817+0000","last_became_peered":"2023-10-30T12:28:50.962817+0000","last_unstale":"2023-10-30T12:52:18.285265+0000","last_undegraded":"2023-10-30T12:52:18.285265+0000","last_fullsized":"2023-10-30T12:52:18.285265+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'539","last_scrub_stamp":"2023-10-30T12:52:18.285036+0000","last_deep_scrub":"66'539","last_deep_scrub_stamp":"2023-10-30T12:52:18.285036+0000","last_clean_scrub_stamp":"2023-10-30T12:52:18.285036+0000","objects_scrubbed":13,"log_size":539,"log_dups_size":0,"ondisk_log_size":539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:28:02.616414+0000","scrub_duration":0.052547446999999997,"objects_trimmed":0,"snaptrim_duration":0.0070898599999999999,"stat_sum":{"num_bytes":8448766,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":50,"num_read_kb":25,"num_write":483,"num_write_kb":14592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"66'176","reported_seq":261,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:51.555385+0000","last_change":"2023-10-30T12:51:51.555592+0000","last_active":"2023-10-30T12:51:51.555385+0000","last_peered":"2023-10-30T12:51:51.555385+0000","last_clean":"2023-10-30T12:51:51.555385+0000","last_became_active":"2023-10-30T12:28:50.401677+0000","last_became_peered":"2023-10-30T12:28:50.401677+0000","last_unstale":"2023-10-30T12:51:51.555385+0000","last_undegraded":"2023-10-30T12:51:51.555385+0000","last_fullsized":"2023-10-30T12:51:51.555385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'176","last_scrub_stamp":"2023-10-30T12:51:51.555145+0000","last_deep_scrub":"66'176","last_deep_scrub_stamp":"2023-10-30T12:51:51.555145+0000","last_clean_scrub_stamp":"2023-10-30T12:51:51.555145+0000","objects_scrubbed":3,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:07:33.229250+0000","scrub_duration":0.033175365999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":20498,"num_write":107,"num_write_kb":8247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"66'514","reported_seq":2678,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:31.213489+0000","last_change":"2023-10-30T12:52:31.213673+0000","last_active":"2023-10-30T12:52:31.213489+0000","last_peered":"2023-10-30T12:52:31.213489+0000","last_clean":"2023-10-30T12:52:31.213489+0000","last_became_active":"2023-10-30T12:28:50.967591+0000","last_became_peered":"2023-10-30T12:28:50.967591+0000","last_unstale":"2023-10-30T12:52:31.213489+0000","last_undegraded":"2023-10-30T12:52:31.213489+0000","last_fullsized":"2023-10-30T12:52:31.213489+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'514","last_scrub_stamp":"2023-10-30T12:52:31.213261+0000","last_deep_scrub":"66'514","last_deep_scrub_stamp":"2023-10-30T12:52:31.213261+0000","last_clean_scrub_stamp":"2023-10-30T12:52:31.213261+0000","objects_scrubbed":14,"log_size":514,"log_dups_size":0,"ondisk_log_size":514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:25:59.281273+0000","scrub_duration":0.057278369000000003,"objects_trimmed":0,"snaptrim_duration":0.0081968709999999997,"stat_sum":{"num_bytes":10185055,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":44,"num_read_kb":1610,"num_write":440,"num_write_kb":22911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"66'370","reported_seq":466,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:50.641824+0000","last_change":"2023-10-30T12:51:50.641945+0000","last_active":"2023-10-30T12:51:50.641824+0000","last_peered":"2023-10-30T12:51:50.641824+0000","last_clean":"2023-10-30T12:51:50.641824+0000","last_became_active":"2023-10-30T12:28:50.392021+0000","last_became_peered":"2023-10-30T12:28:50.392021+0000","last_unstale":"2023-10-30T12:51:50.641824+0000","last_undegraded":"2023-10-30T12:51:50.641824+0000","last_fullsized":"2023-10-30T12:51:50.641824+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'370","last_scrub_stamp":"2023-10-30T12:51:50.641619+0000","last_deep_scrub":"66'370","last_deep_scrub_stamp":"2023-10-30T12:51:50.641619+0000","last_clean_scrub_stamp":"2023-10-30T12:51:50.641619+0000","objects_scrubbed":8,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:05:24.267901+0000","scrub_duration":0.078330257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":57384,"num_write":298,"num_write_kb":16576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22858,"num_omap_keys":43,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"66'521","reported_seq":2687,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:00.262491+0000","last_change":"2023-10-30T12:52:00.262554+0000","last_active":"2023-10-30T12:52:00.262491+0000","last_peered":"2023-10-30T12:52:00.262491+0000","last_clean":"2023-10-30T12:52:00.262491+0000","last_became_active":"2023-10-30T12:28:50.967186+0000","last_became_peered":"2023-10-30T12:28:50.967186+0000","last_unstale":"2023-10-30T12:52:00.262491+0000","last_undegraded":"2023-10-30T12:52:00.262491+0000","last_fullsized":"2023-10-30T12:52:00.262491+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'521","last_scrub_stamp":"2023-10-30T12:52:00.262413+0000","last_deep_scrub":"66'521","last_deep_scrub_stamp":"2023-10-30T12:52:00.262413+0000","last_clean_scrub_stamp":"2023-10-30T12:52:00.262413+0000","objects_scrubbed":15,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:28:18.321174+0000","scrub_duration":0.025087834999999999,"objects_trimmed":0,"snaptrim_duration":0.003136738,"stat_sum":{"num_bytes":73381,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":44,"num_read_kb":17,"num_write":453,"num_write_kb":3099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"66'231","reported_seq":317,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:37.827474+0000","last_change":"2023-10-30T12:51:37.827474+0000","last_active":"2023-10-30T12:51:37.827474+0000","last_peered":"2023-10-30T12:51:37.827474+0000","last_clean":"2023-10-30T12:51:37.827474+0000","last_became_active":"2023-10-30T12:28:50.362929+0000","last_became_peered":"2023-10-30T12:28:50.362929+0000","last_unstale":"2023-10-30T12:51:37.827474+0000","last_undegraded":"2023-10-30T12:51:37.827474+0000","last_fullsized":"2023-10-30T12:51:37.827474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'231","last_scrub_stamp":"2023-10-30T12:51:37.827339+0000","last_deep_scrub":"66'231","last_deep_scrub_stamp":"2023-10-30T12:51:37.827339+0000","last_clean_scrub_stamp":"2023-10-30T12:51:37.827339+0000","objects_scrubbed":4,"log_size":231,"log_dups_size":0,"ondisk_log_size":231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:06:46.012417+0000","scrub_duration":0.026390808000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":9,"num_read_kb":32768,"num_write":162,"num_write_kb":8341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":4,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"66'588","reported_seq":2741,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:52:11.804712+0000","last_change":"2023-10-30T12:52:11.804712+0000","last_active":"2023-10-30T12:52:11.804712+0000","last_peered":"2023-10-30T12:52:11.804712+0000","last_clean":"2023-10-30T12:52:11.804712+0000","last_became_active":"2023-10-30T12:28:50.937458+0000","last_became_peered":"2023-10-30T12:28:50.937458+0000","last_unstale":"2023-10-30T12:52:11.804712+0000","last_undegraded":"2023-10-30T12:52:11.804712+0000","last_fullsized":"2023-10-30T12:52:11.804712+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'588","last_scrub_stamp":"2023-10-30T12:52:11.804563+0000","last_deep_scrub":"66'588","last_deep_scrub_stamp":"2023-10-30T12:52:11.804563+0000","last_clean_scrub_stamp":"2023-10-30T12:52:11.804563+0000","objects_scrubbed":16,"log_size":588,"log_dups_size":0,"ondisk_log_size":588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:39:38.027390+0000","scrub_duration":0.058327041000000003,"objects_trimmed":0,"snaptrim_duration":0.003544045,"stat_sum":{"num_bytes":16852009,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":30,"num_read_kb":11,"num_write":518,"num_write_kb":19789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"66'450","reported_seq":554,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-30T12:51:34.515168+0000","last_change":"2023-10-30T12:51:34.515168+0000","last_active":"2023-10-30T12:51:34.515168+0000","last_peered":"2023-10-30T12:51:34.515168+0000","last_clean":"2023-10-30T12:51:34.515168+0000","last_became_active":"2023-10-30T12:28:50.386522+0000","last_became_peered":"2023-10-30T12:28:50.386522+0000","last_unstale":"2023-10-30T12:51:34.515168+0000","last_undegraded":"2023-10-30T12:51:34.515168+0000","last_fullsized":"2023-10-30T12:51:34.515168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'450","last_scrub_stamp":"2023-10-30T12:51:34.515038+0000","last_deep_scrub":"66'450","last_deep_scrub_stamp":"2023-10-30T12:51:34.515038+0000","last_clean_scrub_stamp":"2023-10-30T12:51:34.515038+0000","objects_scrubbed":7,"log_size":450,"log_dups_size":0,"ondisk_log_size":450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:32:31.022257+0000","scrub_duration":0.052865532999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583494,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":64,"num_read_kb":24634,"num_write":377,"num_write_kb":12490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":757298097,"num_objects":1192,"num_object_clones":0,"num_object_copies":2384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1192,"num_whiteouts":0,"num_read":2811,"num_read_kb":100236,"num_write":31900,"num_write_kb":1049744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1519566848,"data_stored":1514596194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36222,"ondisk_log_size":36222,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":305770548,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1441,"num_read_kb":712828,"num_write":10566,"num_write_kb":307330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":78,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":305728,"num_omap_keys":583,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":611704832,"data_stored":611541096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11973,"ondisk_log_size":11973,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542440,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":394260,"kb_used_data":322828,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93977580,"statfs":{"total":96636764160,"available":96233041920,"internally_reserved":0,"allocated":330575872,"data_stored":329409202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":73136587},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:52:25 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.77600000000000002,"15min":0.81399999999999995},"min":{"1min":0.504,"5min":0.45900000000000002,"15min":0.40000000000000002},"max":{"1min":1.266,"5min":1.266,"15min":1.595},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.78500000000000003,"15min":0.84099999999999997},"min":{"1min":0.502,"5min":0.49099999999999999,"15min":0.40999999999999998},"max":{"1min":1.1279999999999999,"5min":1.302,"15min":1.4830000000000001},"last":0.59899999999999998}]},{"osd":1,"last update":"Mon Oct 30 12:52:25 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.83799999999999997,"15min":0.88400000000000001},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.441},"max":{"1min":1.089,"5min":3.4329999999999998,"15min":3.4329999999999998},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.81599999999999995,"15min":0.878},"min":{"1min":0.55300000000000005,"5min":0.505,"15min":0.505},"max":{"1min":1.117,"5min":1.268,"15min":1.3919999999999999},"last":0.752}]},{"osd":2,"last update":"Mon Oct 30 12:52:25 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.86799999999999999,"15min":0.91900000000000004},"min":{"1min":0.59099999999999997,"5min":0.55100000000000005,"15min":0.45900000000000002},"max":{"1min":1.153,"5min":3.4820000000000002,"15min":3.4820000000000002},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.876,"15min":0.83899999999999997},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.503},"max":{"1min":1.1930000000000001,"5min":1.4570000000000001,"15min":1.6639999999999999},"last":0.83399999999999996}]},{"osd":3,"last update":"Mon Oct 30 12:52:30 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":0.99299999999999999,"15min":0.97599999999999998},"min":{"1min":0.79000000000000004,"5min":0.60999999999999999,"15min":0.45500000000000002},"max":{"1min":1.25,"5min":3.5190000000000001,"15min":3.5190000000000001},"last":0.79900000000000004},{"interface":"front","average":{"1min":1.0009999999999999,"5min":0.96499999999999997,"15min":0.99099999999999999},"min":{"1min":0.67500000000000004,"5min":0.60999999999999999,"15min":0.53300000000000003},"max":{"1min":1.298,"5min":1.4259999999999999,"15min":1.5880000000000001},"last":0.86099999999999999}]},{"osd":4,"last update":"Mon Oct 30 12:52:25 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.95899999999999996,"15min":0.998},"min":{"1min":0.70499999999999996,"5min":0.42499999999999999,"15min":0.34399999999999997},"max":{"1min":1.4339999999999999,"5min":1.542,"15min":1.7350000000000001},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1,"15min":1.0449999999999999},"min":{"1min":0.76400000000000001,"5min":0.45000000000000001,"15min":0.33800000000000002},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.746},"last":1.077}]},{"osd":5,"last update":"Mon Oct 30 12:52:25 2023","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0329999999999999,"15min":0.96399999999999997},"min":{"1min":0.83499999999999996,"5min":0.53700000000000003,"15min":0.43099999999999999},"max":{"1min":1.5640000000000001,"5min":1.6160000000000001,"15min":1.7549999999999999},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.1180000000000001,"15min":1.083},"min":{"1min":0.89000000000000001,"5min":0.57899999999999996,"15min":0.35999999999999999},"max":{"1min":1.379,"5min":1.643,"15min":2.1600000000000001},"last":0.68400000000000005}]},{"osd":6,"last update":"Mon Oct 30 12:52:25 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.129,"15min":1.131},"min":{"1min":0.73999999999999999,"5min":0.71299999999999997,"15min":0.39000000000000001},"max":{"1min":1.5149999999999999,"5min":1.5609999999999999,"15min":2.3809999999999998},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.119,"15min":1.165},"min":{"1min":0.93000000000000005,"5min":0.56299999999999994,"15min":0.47199999999999998},"max":{"1min":1.637,"5min":3.8140000000000001,"15min":3.8140000000000001},"last":1.0349999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542439,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302480,"kb_used_data":235272,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94069360,"statfs":{"total":96636764160,"available":96327024640,"internally_reserved":0,"allocated":240918528,"data_stored":240057681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":68811208},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:52:28 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.83399999999999996,"15min":0.85499999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.436},"max":{"1min":1.256,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.75900000000000001,"15min":0.78000000000000003},"min":{"1min":0.434,"5min":0.43099999999999999,"15min":0.42999999999999999},"max":{"1min":1.0309999999999999,"5min":1.0780000000000001,"15min":1.214},"last":1.0860000000000001}]},{"osd":1,"last update":"Mon Oct 30 12:52:33 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.84599999999999997,"15min":0.81499999999999995},"min":{"1min":0.56000000000000005,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.29,"5min":1.29,"15min":1.3140000000000001},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.83999999999999997,"15min":0.876},"min":{"1min":0.63300000000000001,"5min":0.45500000000000002,"15min":0.44400000000000001},"max":{"1min":1.107,"5min":1.2869999999999999,"15min":1.423},"last":0.86799999999999999}]},{"osd":2,"last update":"Mon Oct 30 12:52:28 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.89300000000000002,"15min":0.95999999999999996},"min":{"1min":0.58399999999999996,"5min":0.495,"15min":0.495},"max":{"1min":1.085,"5min":1.373,"15min":4.3360000000000003},"last":0.72699999999999998},{"interface":"front","average":{"1min":1.044,"5min":0.98799999999999999,"15min":1.016},"min":{"1min":0.65100000000000002,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.409,"5min":1.502,"15min":1.5629999999999999},"last":1.157}]},{"osd":3,"last update":"Mon Oct 30 12:52:28 2023","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":0.98699999999999999,"15min":1.042},"min":{"1min":0.78100000000000003,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.5009999999999999,"5min":1.5049999999999999,"15min":1.6459999999999999},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.012,"5min":1.0409999999999999,"15min":1.095},"min":{"1min":0.73499999999999999,"5min":0.63300000000000001,"15min":0.59699999999999998},"max":{"1min":1.573,"5min":1.573,"15min":1.5800000000000001},"last":0.92600000000000005}]},{"osd":4,"last update":"Mon Oct 30 12:52:28 2023","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.127,"15min":1.087},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.55700000000000005},"max":{"1min":1.6599999999999999,"5min":1.6599999999999999,"15min":1.7649999999999999},"last":1.296},{"interface":"front","average":{"1min":1.1910000000000001,"5min":0.95099999999999996,"15min":0.96999999999999997},"min":{"1min":0.752,"5min":0.47199999999999998,"15min":0.42099999999999999},"max":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":1.881},"last":1.3440000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:52:28 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.006,"15min":1.0860000000000001},"min":{"1min":0.65200000000000002,"5min":0.53200000000000003,"15min":0.48399999999999999},"max":{"1min":1.6990000000000001,"5min":1.6990000000000001,"15min":2.2770000000000001},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.016,"5min":1.1419999999999999,"15min":1.069},"min":{"1min":0.69099999999999995,"5min":0.66800000000000004,"15min":0.46999999999999997},"max":{"1min":1.208,"5min":1.6950000000000001,"15min":4.7850000000000001},"last":0.95899999999999996}]},{"osd":7,"last update":"Mon Oct 30 12:52:28 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":1.0209999999999999,"15min":0.998},"min":{"1min":0.64200000000000002,"5min":0.64100000000000001,"15min":0.52300000000000002},"max":{"1min":1.2090000000000001,"5min":1.5609999999999999,"15min":1.825},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.071,"5min":1.1160000000000001,"15min":1.0700000000000001},"min":{"1min":0.76000000000000001,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.337,"5min":1.6899999999999999,"15min":1.895},"last":1.212}]}]},{"osd":5,"up_from":14,"seq":60129542440,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":342544,"kb_used_data":271112,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94029296,"statfs":{"total":96636764160,"available":96285999104,"internally_reserved":0,"allocated":277618688,"data_stored":276486603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":1.7969999999999999,"5min":0.99099999999999999,"15min":0.86399999999999999},"min":{"1min":0.53500000000000003,"5min":0.33900000000000002,"15min":0.29399999999999998},"max":{"1min":19.593,"5min":19.593,"15min":19.593},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.80400000000000005,"15min":0.80300000000000005},"min":{"1min":0.59299999999999997,"5min":0.39200000000000002,"15min":0.33800000000000002},"max":{"1min":1.121,"5min":1.3009999999999999,"15min":1.8759999999999999},"last":0.75900000000000001}]},{"osd":1,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.83599999999999997,"15min":0.83999999999999997},"min":{"1min":0.59299999999999997,"5min":0.40999999999999998,"15min":0.38900000000000001},"max":{"1min":1.1859999999999999,"5min":1.2909999999999999,"15min":3.0190000000000001},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.89100000000000001,"15min":0.85899999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.373},"max":{"1min":1.1870000000000001,"5min":1.276,"15min":1.4870000000000001},"last":0.96299999999999997}]},{"osd":2,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.90100000000000002,"15min":0.84999999999999998},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.38400000000000001},"max":{"1min":1.306,"5min":1.306,"15min":2.2949999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.954,"5min":1.046,"15min":0.94599999999999995},"min":{"1min":0.64000000000000001,"5min":0.434,"15min":0.434},"max":{"1min":19.640000000000001,"5min":19.640000000000001,"15min":19.640000000000001},"last":1.0129999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.92200000000000004,"15min":0.93200000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.2410000000000001,"5min":1.3999999999999999,"15min":1.756},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.0660000000000001,"5min":0.96699999999999997,"15min":0.95399999999999996},"min":{"1min":0.64900000000000002,"5min":0.498,"15min":0.41399999999999998},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":2.3679999999999999},"last":1.1579999999999999}]},{"osd":4,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":2.0680000000000001,"5min":1.099,"15min":0.98699999999999999},"min":{"1min":0.53700000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":19.675999999999998,"5min":19.675999999999998,"15min":19.675999999999998},"last":1.113},{"interface":"front","average":{"1min":2.2909999999999999,"5min":1.341,"15min":1.173},"min":{"1min":0.72499999999999998,"5min":0.59499999999999997,"15min":0.50900000000000001},"max":{"1min":20.036999999999999,"5min":20.036999999999999,"15min":20.036999999999999},"last":1.3320000000000001}]},{"osd":6,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":2.3090000000000002,"5min":1.304,"15min":1.129},"min":{"1min":0.88200000000000001,"5min":0.45800000000000002,"15min":0.33600000000000002},"max":{"1min":21.526,"5min":21.526,"15min":21.526},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.103,"5min":1.079,"15min":1.1020000000000001},"min":{"1min":0.46400000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.387,"5min":1.47,"15min":3.371},"last":1.2949999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.1060000000000001,"15min":1.0840000000000001},"min":{"1min":0.57599999999999996,"5min":0.51400000000000001,"15min":0.47299999999999998},"max":{"1min":1.7569999999999999,"5min":1.7569999999999999,"15min":3.5310000000000001},"last":1.536},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.236,"15min":1.232},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.903,"5min":1.903,"15min":3.6930000000000001},"last":1.613}]}]},{"osd":4,"up_from":14,"seq":60129542439,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369460,"kb_used_data":293868,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94002380,"statfs":{"total":96636764160,"available":96258437120,"internally_reserved":0,"allocated":300920832,"data_stored":299840560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.80100000000000005,"15min":0.92600000000000005},"min":{"1min":0.46800000000000003,"5min":0.433,"15min":0.31},"max":{"1min":1.159,"5min":1.2509999999999999,"15min":11.273},"last":1.0940000000000001},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.91800000000000004,"15min":0.98299999999999998},"min":{"1min":0.63600000000000001,"5min":0.53900000000000003,"15min":0.44},"max":{"1min":1.3120000000000001,"5min":1.4590000000000001,"15min":7.1289999999999996},"last":0.96999999999999997}]},{"osd":1,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.78100000000000003,"15min":0.89300000000000002},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0229999999999999,"5min":1.2230000000000001,"15min":6.9199999999999999},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.83899999999999997,"15min":0.89200000000000002},"min":{"1min":0.52200000000000002,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.1160000000000001,"5min":1.246,"15min":4.75},"last":1.1719999999999999}]},{"osd":2,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.86199999999999999,"15min":0.85099999999999998},"min":{"1min":0.56200000000000006,"5min":0.51700000000000002,"15min":0.36899999999999999},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":2.6970000000000001},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.91100000000000003,"15min":1.0620000000000001},"min":{"1min":0.66900000000000004,"5min":0.54400000000000004,"15min":0.52900000000000003},"max":{"1min":1.1299999999999999,"5min":1.361,"15min":18.917999999999999},"last":1.0529999999999999}]},{"osd":3,"last update":"Mon Oct 30 12:52:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":0.96099999999999997,"15min":1.006},"min":{"1min":0.67700000000000005,"5min":0.60799999999999998,"15min":0.51400000000000001},"max":{"1min":1.236,"5min":1.284,"15min":7.1879999999999997},"last":1.1240000000000001},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.86199999999999999,"15min":0.98999999999999999},"min":{"1min":0.58699999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.1040000000000001,"5min":1.343,"15min":8.3360000000000003},"last":0.998}]},{"osd":5,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.86899999999999999,"15min":0.996},"min":{"1min":0.57599999999999996,"5min":0.56299999999999994,"15min":0.54700000000000004},"max":{"1min":1.026,"5min":1.2509999999999999,"15min":6.0759999999999996},"last":0.70699999999999996},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.113,"15min":1.1859999999999999},"min":{"1min":0.95399999999999996,"5min":0.77400000000000002,"15min":0.52600000000000002},"max":{"1min":1.3660000000000001,"5min":1.4810000000000001,"15min":18.994},"last":1.0209999999999999}]},{"osd":6,"last update":"Mon Oct 30 12:52:26 2023","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.099,"15min":1.099},"min":{"1min":0.77400000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.381,"5min":1.429,"15min":6.1580000000000004},"last":1.22},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.96999999999999997,"15min":1.1220000000000001},"min":{"1min":0.66900000000000004,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.179,"5min":1.524,"15min":7.0549999999999997},"last":0.92300000000000004}]},{"osd":7,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.056,"15min":1.2529999999999999},"min":{"1min":0.83799999999999997,"5min":0.56399999999999995,"15min":0.50700000000000001},"max":{"1min":1.2749999999999999,"5min":1.4850000000000001,"15min":18.015999999999998},"last":0.88400000000000001},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.1579999999999999,"15min":1.2769999999999999},"min":{"1min":0.92100000000000004,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.4339999999999999,"5min":1.4570000000000001,"15min":19.052},"last":1.272}]}]},{"osd":3,"up_from":14,"seq":60129542440,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391212,"kb_used_data":310052,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93980628,"statfs":{"total":96636764160,"available":96236163072,"internally_reserved":0,"allocated":317493248,"data_stored":316374505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":83098056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:52:26 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.40500000000000003,"15min":0.41699999999999998},"min":{"1min":0.31900000000000001,"5min":0.23699999999999999,"15min":0.23400000000000001},"max":{"1min":0.61799999999999999,"5min":0.77900000000000003,"15min":1.6899999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52300000000000002,"15min":0.53400000000000003},"min":{"1min":0.40500000000000003,"5min":0.34300000000000003,"15min":0.32700000000000001},"max":{"1min":0.71699999999999997,"5min":0.871,"15min":1.728},"last":0.53600000000000003}]},{"osd":1,"last update":"Mon Oct 30 12:52:26 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.438,"15min":0.432},"min":{"1min":0.308,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":1.2909999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.501,"15min":0.50900000000000001},"min":{"1min":0.36299999999999999,"5min":0.31900000000000001,"15min":0.29499999999999998},"max":{"1min":0.73199999999999998,"5min":0.80600000000000005,"15min":1.3260000000000001},"last":0.63200000000000001}]},{"osd":2,"last update":"Mon Oct 30 12:52:26 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.504,"15min":0.501},"min":{"1min":0.35199999999999998,"5min":0.30099999999999999,"15min":0.29299999999999998},"max":{"1min":0.69999999999999996,"5min":0.78600000000000003,"15min":1.099},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56200000000000006,"15min":0.56399999999999995},"min":{"1min":0.41099999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.755,"5min":0.82599999999999996,"15min":1.2509999999999999},"last":0.60699999999999998}]},{"osd":4,"last update":"Mon Oct 30 12:52:26 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.86099999999999999,"15min":0.83699999999999997},"min":{"1min":0.69299999999999995,"5min":0.55400000000000005,"15min":0.40400000000000003},"max":{"1min":1.1679999999999999,"5min":1.2310000000000001,"15min":2.202},"last":1.032},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.84299999999999997,"15min":0.86899999999999999},"min":{"1min":0.78100000000000003,"5min":0.48499999999999999,"15min":0.46800000000000003},"max":{"1min":1.0669999999999999,"5min":1.1950000000000001,"15min":2.4340000000000002},"last":0.93799999999999994}]},{"osd":5,"last update":"Mon Oct 30 12:52:26 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.81599999999999995,"15min":0.86199999999999999},"min":{"1min":0.58399999999999996,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.0649999999999999,"5min":1.169,"15min":14.103999999999999},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.873,"15min":0.872},"min":{"1min":0.75800000000000001,"5min":0.55000000000000004,"15min":0.499},"max":{"1min":1.0469999999999999,"5min":1.2170000000000001,"15min":4.4859999999999998},"last":0.85099999999999998}]},{"osd":6,"last update":"Mon Oct 30 12:52:26 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.84299999999999997,"15min":0.83399999999999996},"min":{"1min":0.57399999999999995,"5min":0.56999999999999995,"15min":0.53100000000000003},"max":{"1min":1.123,"5min":1.2,"15min":2.2330000000000001},"last":1.006},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.90500000000000003,"15min":0.83799999999999997},"min":{"1min":0.79800000000000004,"5min":0.57499999999999996,"15min":0.45100000000000001},"max":{"1min":1.0800000000000001,"5min":1.2609999999999999,"15min":4.4409999999999998},"last":1.0569999999999999}]},{"osd":7,"last update":"Mon Oct 30 12:52:26 2023","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.88200000000000001,"15min":0.86899999999999999},"min":{"1min":0.68100000000000005,"5min":0.59599999999999997,"15min":0.57099999999999995},"max":{"1min":1.0980000000000001,"5min":1.1499999999999999,"15min":3.5459999999999998},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.89400000000000002,"15min":0.85599999999999998},"min":{"1min":0.64100000000000001,"5min":0.57399999999999995,"15min":0.45700000000000002},"max":{"1min":1.2090000000000001,"5min":1.2230000000000001,"15min":6.1360000000000001},"last":1.0840000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542440,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":200604,"kb_used_data":143060,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94171236,"statfs":{"total":96636764160,"available":96431345664,"internally_reserved":0,"allocated":146493440,"data_stored":145675964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.442,"15min":0.441},"min":{"1min":0.28000000000000003,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.55300000000000005,"5min":0.69399999999999995,"15min":0.81200000000000006},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.502,"15min":0.51100000000000001},"min":{"1min":0.39900000000000002,"5min":0.36099999999999999,"15min":0.31900000000000001},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":1.298},"last":0.61299999999999999}]},{"osd":1,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.47499999999999998},"min":{"1min":0.315,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.57699999999999996,"5min":0.64800000000000002,"15min":0.83699999999999997},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.51800000000000002,"15min":0.501},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.29999999999999999},"max":{"1min":0.61799999999999999,"5min":0.74099999999999999,"15min":1.2250000000000001},"last":0.66800000000000004}]},{"osd":3,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.48299999999999998,"15min":0.54200000000000004},"min":{"1min":0.33500000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.752,"5min":0.752,"15min":1.673},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57299999999999995,"15min":0.57099999999999995},"min":{"1min":0.47299999999999998,"5min":0.40200000000000002,"15min":0.35699999999999998},"max":{"1min":0.72299999999999998,"5min":0.79100000000000004,"15min":1.597},"last":0.745}]},{"osd":4,"last update":"Mon Oct 30 12:52:21 2023","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.75700000000000001,"15min":0.82199999999999995},"min":{"1min":0.69599999999999995,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.038,"5min":1.038,"15min":1.377},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.876,"5min":0.77800000000000002,"15min":0.79500000000000004},"min":{"1min":0.74099999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.4159999999999999},"last":0.92500000000000004}]},{"osd":5,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.84899999999999998,"15min":0.84399999999999997},"min":{"1min":0.60199999999999998,"5min":0.502,"15min":0.502},"max":{"1min":1.0900000000000001,"5min":1.113,"15min":1.78},"last":1.0309999999999999},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.91800000000000004,"15min":0.91400000000000003},"min":{"1min":0.75,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.135,"5min":1.1379999999999999,"15min":1.7430000000000001},"last":0.85499999999999998}]},{"osd":6,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.85899999999999999,"15min":0.86099999999999999},"min":{"1min":0.66000000000000003,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.1659999999999999,"5min":1.1759999999999999,"15min":1.3799999999999999},"last":1.069},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.91400000000000003,"15min":0.86899999999999999},"min":{"1min":0.65600000000000003,"5min":0.58299999999999996,"15min":0.53300000000000003},"max":{"1min":1.081,"5min":1.1799999999999999,"15min":1.8109999999999999},"last":1.097}]},{"osd":7,"last update":"Mon Oct 30 12:52:17 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.84399999999999997,"15min":0.83099999999999996},"min":{"1min":0.69899999999999995,"5min":0.58299999999999996,"15min":0.48799999999999999},"max":{"1min":1.1379999999999999,"5min":1.1599999999999999,"15min":1.4159999999999999},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.872,"15min":0.83899999999999997},"min":{"1min":0.69399999999999995,"5min":0.51800000000000002,"15min":0.50800000000000001},"max":{"1min":1.238,"5min":1.238,"15min":1.3049999999999999},"last":0.95999999999999996}]}]},{"osd":1,"up_from":14,"seq":60129542440,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":279036,"kb_used_data":233972,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94092804,"statfs":{"total":96636764160,"available":96351031296,"internally_reserved":0,"allocated":239587328,"data_stored":238747727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 12:52:42 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.41099999999999998,"15min":0.45700000000000002},"min":{"1min":0.28899999999999998,"5min":0.25600000000000001,"15min":0.24299999999999999},"max":{"1min":0.60499999999999998,"5min":0.65200000000000002,"15min":11.958},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.55300000000000005},"min":{"1min":0.38400000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68000000000000005,"5min":0.751,"15min":12.077},"last":0.41399999999999998}]},{"osd":2,"last update":"Mon Oct 30 12:52:42 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.45700000000000002,"15min":0.46000000000000002},"min":{"1min":0.33300000000000002,"5min":0.29699999999999999,"15min":0.26400000000000001},"max":{"1min":0.60399999999999998,"5min":1.121,"15min":1.623},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53000000000000003,"15min":0.53200000000000003},"min":{"1min":0.38,"5min":0.34200000000000003,"15min":0.31900000000000001},"max":{"1min":0.71799999999999997,"5min":1.1699999999999999,"15min":1.5389999999999999},"last":0.51900000000000002}]},{"osd":3,"last update":"Mon Oct 30 12:52:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56299999999999994,"15min":0.56599999999999995},"min":{"1min":0.45100000000000001,"5min":0.378,"15min":0.31},"max":{"1min":0.74199999999999999,"5min":1.337,"15min":3.694},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.498,"15min":0.53900000000000003},"min":{"1min":0.373,"5min":0.307,"15min":0.307},"max":{"1min":0.70599999999999996,"5min":0.73199999999999998,"15min":13.199999999999999},"last":0.55400000000000005}]},{"osd":4,"last update":"Mon Oct 30 12:52:42 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.80900000000000005,"15min":0.82499999999999996},"min":{"1min":0.60299999999999998,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.0249999999999999,"5min":1.3140000000000001,"15min":2.419},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.871,"15min":0.86699999999999999},"min":{"1min":0.72099999999999997,"5min":0.60699999999999998,"15min":0.52100000000000002},"max":{"1min":1.0309999999999999,"5min":1.0960000000000001,"15min":2.282},"last":0.88600000000000001}]},{"osd":5,"last update":"Mon Oct 30 12:52:42 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.78600000000000003,"15min":0.81399999999999995},"min":{"1min":0.60399999999999998,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.01,"5min":1.343,"15min":1.343},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.79200000000000004,"15min":0.84499999999999997},"min":{"1min":0.57199999999999995,"5min":0.50800000000000001,"15min":0.44900000000000001},"max":{"1min":1.0920000000000001,"5min":1.2809999999999999,"15min":1.29},"last":0.80500000000000005}]},{"osd":6,"last update":"Mon Oct 30 12:52:42 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.84799999999999998,"15min":0.80600000000000005},"min":{"1min":0.73299999999999998,"5min":0.52800000000000002,"15min":0.51200000000000001},"max":{"1min":1.127,"5min":1.371,"15min":1.371},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.92100000000000004,"15min":0.871},"min":{"1min":0.79100000000000004,"5min":0.62,"15min":0.498},"max":{"1min":1.0960000000000001,"5min":1.417,"15min":2.464},"last":0.625}]},{"osd":7,"last update":"Mon Oct 30 12:52:42 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.81599999999999995,"15min":0.82499999999999996},"min":{"1min":0.66600000000000004,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":2.5169999999999999},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.89800000000000002,"15min":0.90300000000000002},"min":{"1min":0.80200000000000005,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":2.5510000000000002},"last":0.81999999999999995}]}]},{"osd":0,"up_from":14,"seq":60129542440,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":349720,"kb_used_data":278288,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94022120,"statfs":{"total":96636764160,"available":96278650880,"internally_reserved":0,"allocated":284966912,"data_stored":283825495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 12:52:30 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.46100000000000002,"15min":0.45200000000000001},"min":{"1min":0.27500000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.93700000000000006},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52100000000000002,"15min":0.51100000000000001},"min":{"1min":0.40200000000000002,"5min":0.35299999999999998,"15min":0.26500000000000001},"max":{"1min":0.72099999999999997,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.56399999999999995}]},{"osd":2,"last update":"Mon Oct 30 12:52:30 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46100000000000002,"15min":0.499},"min":{"1min":0.30099999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.97199999999999998},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.53900000000000003},"min":{"1min":0.42399999999999999,"5min":0.36799999999999999,"15min":0.27400000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":1.0149999999999999},"last":0.54300000000000004}]},{"osd":3,"last update":"Mon Oct 30 12:52:30 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.48499999999999999,"15min":0.50800000000000001},"min":{"1min":0.35199999999999998,"5min":0.30499999999999999,"15min":0.29499999999999998},"max":{"1min":0.82799999999999996,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51100000000000001,"15min":0.499},"min":{"1min":0.313,"5min":0.313,"15min":0.31},"max":{"1min":0.76000000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.52000000000000002}]},{"osd":4,"last update":"Mon Oct 30 12:52:30 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.86399999999999999,"15min":0.85299999999999998},"min":{"1min":0.58199999999999996,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.2250000000000001,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.874,"15min":0.86099999999999999},"min":{"1min":0.71699999999999997,"5min":0.52600000000000002,"15min":0.48299999999999998},"max":{"1min":1.248,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.86699999999999999}]},{"osd":5,"last update":"Mon Oct 30 12:52:30 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.80300000000000005,"15min":0.82799999999999996},"min":{"1min":0.55700000000000005,"5min":0.55200000000000005,"15min":0.46200000000000002},"max":{"1min":1.079,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.85299999999999998,"15min":0.86499999999999999},"min":{"1min":0.67800000000000005,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.1779999999999999,"5min":1.889,"15min":1.889},"last":0.90200000000000002}]},{"osd":6,"last update":"Mon Oct 30 12:52:30 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.81299999999999994,"15min":0.79800000000000004},"min":{"1min":0.69399999999999995,"5min":0.56299999999999994,"15min":0.52200000000000002},"max":{"1min":1.294,"5min":1.294,"15min":2.6480000000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.81000000000000005,"15min":0.81799999999999995},"min":{"1min":0.621,"5min":0.57699999999999996,"15min":0.46999999999999997},"max":{"1min":1.0389999999999999,"5min":1.1559999999999999,"15min":5.0099999999999998},"last":0.91600000000000004}]},{"osd":7,"last update":"Mon Oct 30 12:52:30 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.879,"15min":0.88600000000000001},"min":{"1min":0.63400000000000001,"5min":0.59699999999999998,"15min":0.55000000000000004},"max":{"1min":1.1539999999999999,"5min":1.401,"15min":3.3359999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.94199999999999995,"15min":0.94899999999999995},"min":{"1min":0.70099999999999996,"5min":0.65100000000000002,"15min":0.59699999999999998},"max":{"1min":1.2150000000000001,"5min":1.3500000000000001,"15min":3.3580000000000001},"last":0.94499999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62930944,"data_stored":62915220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":83910656,"data_stored":83887024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":37969920,"data_stored":37945198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":83726336,"data_stored":83700311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":76107776,"data_stored":76093397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":62935040,"data_stored":62915938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":91684864,"data_stored":91663056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":112439296,"data_stored":112420952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":220680192,"data_stored":219932830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":154910720,"data_stored":154473007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":107761664,"data_stored":107343689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":233000960,"data_stored":232286498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":224047104,"data_stored":223359467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":213921792,"data_stored":213183588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":148467712,"data_stored":148006929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":216776704,"data_stored":216010186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:52:59.455 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-30T12:52:59.631 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.631+0000 7f4b7bf3e640 1 Processor -- start 2023-10-30T12:52:59.632 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.635+0000 7f4b7bf3e640 1 -- start start 2023-10-30T12:52:59.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.635+0000 7f4b7bf3e640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4b741508c0 0x7f4b740c6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:59.633 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.635+0000 7f4b7bf3e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b740c6e60 0x7f4b74156930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:59.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.635+0000 7f4b7bf3e640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b740c0a30 con 0x7f4b740722a0 2023-10-30T12:52:59.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.635+0000 7f4b7bf3e640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b740c85b0 con 0x7f4b741508c0 2023-10-30T12:52:59.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.635+0000 7f4b7bf3e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7406f650 con 0x7f4b740c6e60 2023-10-30T12:52:59.634 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.635+0000 7f4b79cda640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4b741508c0 0x7f4b740c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:59.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.635+0000 7f4b79cda640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f4b741508c0 0x7f4b740c6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53962/0 (socket says 172.21.15.89:53962) 2023-10-30T12:52:59.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.635+0000 7f4b79cda640 1 -- 172.21.15.89:0/3414410107 learned_addr learned my addr 172.21.15.89:0/3414410107 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:52:59.635 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.635+0000 7f4b7a4db640 1 --2- 172.21.15.89:0/3414410107 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b740c6e60 0x7f4b74156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:59.636 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.635+0000 7f4b79cda640 1 -- 172.21.15.89:0/3414410107 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b740c6e60 msgr2=0x7f4b74156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:59.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.635+0000 7f4b79cda640 1 --2- 172.21.15.89:0/3414410107 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b740c6e60 0x7f4b74156930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:59.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.635+0000 7f4b79cda640 1 -- 172.21.15.89:0/3414410107 >> v1:172.21.15.192:6789/0 conn(0x7f4b740722a0 legacy=0x7f4b740bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:59.637 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.635+0000 7f4b79cda640 1 -- 172.21.15.89:0/3414410107 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b7406f930 con 0x7f4b741508c0 2023-10-30T12:52:59.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.639+0000 7f4b79cda640 1 --2- 172.21.15.89:0/3414410107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b741508c0 0x7f4b740c6920 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f4b70006fd0 tx=0x7f4b7001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31812f72eaa04652 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:59.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.639+0000 7f4b78cd8640 1 -- 172.21.15.89:0/3414410107 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b700ad680 con 0x7f4b741508c0 2023-10-30T12:52:59.638 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.639+0000 7f4b78cd8640 1 -- 172.21.15.89:0/3414410107 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b700ad7e0 con 0x7f4b741508c0 2023-10-30T12:52:59.639 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.639+0000 7f4b78cd8640 1 -- 172.21.15.89:0/3414410107 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b700adb70 con 0x7f4b741508c0 2023-10-30T12:52:59.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.639+0000 7f4b7bf3e640 1 -- 172.21.15.89:0/3414410107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b741508c0 msgr2=0x7f4b740c6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:59.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.639+0000 7f4b7bf3e640 1 --2- 172.21.15.89:0/3414410107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b741508c0 0x7f4b740c6920 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f4b70006fd0 tx=0x7f4b7001c490 comp rx=0 tx=0).stop 2023-10-30T12:52:59.642 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.639+0000 7f4b7bf3e640 1 -- 172.21.15.89:0/3414410107 shutdown_connections 2023-10-30T12:52:59.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.639+0000 7f4b7bf3e640 1 --2- 172.21.15.89:0/3414410107 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b740c6e60 0x7f4b74156930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:59.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.639+0000 7f4b7bf3e640 1 --2- 172.21.15.89:0/3414410107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b741508c0 0x7f4b740c6920 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:59.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.639+0000 7f4b7bf3e640 1 -- 172.21.15.89:0/3414410107 >> 172.21.15.89:0/3414410107 conn(0x7f4b740c7910 msgr2=0x7f4b740c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:59.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.639+0000 7f4b7bf3e640 1 -- 172.21.15.89:0/3414410107 shutdown_connections 2023-10-30T12:52:59.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.639+0000 7f4b7bf3e640 1 -- 172.21.15.89:0/3414410107 wait complete. 2023-10-30T12:52:59.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.639+0000 7f4b7bf3e640 1 Processor -- start 2023-10-30T12:52:59.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.639+0000 7f4b7bf3e640 1 -- start start 2023-10-30T12:52:59.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.643+0000 7f4b7bf3e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b740722a0 0x7f4b740ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:59.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.643+0000 7f4b7bf3e640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b740c6e60 0x7f4b74154410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:59.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.643+0000 7f4b7bf3e640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b741508c0 0x7f4b741435b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:59.645 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.643+0000 7f4b7bf3e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b740b0440 con 0x7f4b740722a0 2023-10-30T12:52:59.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.643+0000 7f4b7bf3e640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b740af220 con 0x7f4b741508c0 2023-10-30T12:52:59.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.643+0000 7f4b7bf3e640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b74153950 con 0x7f4b740c6e60 2023-10-30T12:52:59.646 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.643+0000 7f4b79cda640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b740722a0 0x7f4b740ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:59.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.643+0000 7f4b79cda640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b740722a0 0x7f4b740ad8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53964/0 (socket says 172.21.15.89:53964) 2023-10-30T12:52:59.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.643+0000 7f4b79cda640 1 -- 172.21.15.89:0/4259086263 learned_addr learned my addr 172.21.15.89:0/4259086263 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:52:59.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.643+0000 7f4b7a4db640 1 --2- 172.21.15.89:0/4259086263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b741508c0 0x7f4b741435b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:59.647 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.643+0000 7f4b794d9640 1 --2- 172.21.15.89:0/4259086263 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b740c6e60 0x7f4b74154410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:59.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.643+0000 7f4b79cda640 1 -- 172.21.15.89:0/4259086263 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b740c6e60 msgr2=0x7f4b74154410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:59.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.643+0000 7f4b79cda640 1 --2- 172.21.15.89:0/4259086263 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b740c6e60 0x7f4b74154410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:59.648 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.643+0000 7f4b79cda640 1 -- 172.21.15.89:0/4259086263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b741508c0 msgr2=0x7f4b741435b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:59.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.643+0000 7f4b79cda640 1 --2- 172.21.15.89:0/4259086263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b741508c0 0x7f4b741435b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:59.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.643+0000 7f4b79cda640 1 -- 172.21.15.89:0/4259086263 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b70009480 con 0x7f4b740722a0 2023-10-30T12:52:59.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.643+0000 7f4b794d9640 1 --2- 172.21.15.89:0/4259086263 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b740c6e60 0x7f4b74154410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:52:59.649 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.643+0000 7f4b79cda640 1 --2- 172.21.15.89:0/4259086263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b740722a0 0x7f4b740ad8d0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f4b7001a0c0 tx=0x7f4b70019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:59.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.643+0000 7f4b6a7fc640 1 -- 172.21.15.89:0/4259086263 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b70004450 con 0x7f4b740722a0 2023-10-30T12:52:59.650 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.643+0000 7f4b6a7fc640 1 -- 172.21.15.89:0/4259086263 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b700ad950 con 0x7f4b740722a0 2023-10-30T12:52:59.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.643+0000 7f4b6a7fc640 1 -- 172.21.15.89:0/4259086263 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b700add00 con 0x7f4b740722a0 2023-10-30T12:52:59.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.643+0000 7f4b7bf3e640 1 -- 172.21.15.89:0/4259086263 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b74153c30 con 0x7f4b740722a0 2023-10-30T12:52:59.651 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.643+0000 7f4b7bf3e640 1 -- 172.21.15.89:0/4259086263 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b740b0060 con 0x7f4b740722a0 2023-10-30T12:52:59.652 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.643+0000 7f4b7bf3e640 1 -- 172.21.15.89:0/4259086263 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b7404e170 con 0x7f4b740722a0 2023-10-30T12:52:59.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.643+0000 7f4b6a7fc640 1 -- 172.21.15.89:0/4259086263 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4b700b4040 con 0x7f4b740722a0 2023-10-30T12:52:59.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.647+0000 7f4b6a7fc640 1 --2- 172.21.15.89:0/4259086263 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4b580664a0 0x7f4b58068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:59.653 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.647+0000 7f4b6a7fc640 1 -- 172.21.15.89:0/4259086263 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f4b7011e420 con 0x7f4b740722a0 2023-10-30T12:52:59.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.647+0000 7f4b794d9640 1 --2- 172.21.15.89:0/4259086263 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4b580664a0 0x7f4b58068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:59.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.647+0000 7f4b794d9640 1 --2- 172.21.15.89:0/4259086263 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4b580664a0 0x7f4b58068960 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f4b741f37d0 tx=0x7f4b64009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:59.654 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.651+0000 7f4b6a7fc640 1 -- 172.21.15.89:0/4259086263 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4b700eb5d0 con 0x7f4b740722a0 2023-10-30T12:52:59.798 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.799+0000 7f4b7bf3e640 1 -- 172.21.15.89:0/4259086263 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f4b740c5810 con 0x7f4b740722a0 2023-10-30T12:52:59.803 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.803+0000 7f4b6a7fc640 1 -- 172.21.15.89:0/4259086263 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v10)=0 v10) v1 ==== 123+0+0 (secure 0 0 0) 0x7f4b700ade60 con 0x7f4b740722a0 2023-10-30T12:52:59.805 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.807+0000 7f4b6a7fc640 1 -- 172.21.15.89:0/4259086263 <== mon.0 v2:172.21.15.89:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f4b700eb5d0 con 0x7f4b740722a0 2023-10-30T12:52:59.806 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.807+0000 7f4b7bf3e640 1 -- 172.21.15.89:0/4259086263 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4b580664a0 msgr2=0x7f4b58068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:59.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.807+0000 7f4b7bf3e640 1 --2- 172.21.15.89:0/4259086263 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4b580664a0 0x7f4b58068960 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f4b741f37d0 tx=0x7f4b64009040 comp rx=0 tx=0).stop 2023-10-30T12:52:59.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.807+0000 7f4b7bf3e640 1 -- 172.21.15.89:0/4259086263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b740722a0 msgr2=0x7f4b740ad8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:59.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.807+0000 7f4b7bf3e640 1 --2- 172.21.15.89:0/4259086263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b740722a0 0x7f4b740ad8d0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f4b7001a0c0 tx=0x7f4b70019230 comp rx=0 tx=0).stop 2023-10-30T12:52:59.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.807+0000 7f4b7bf3e640 1 -- 172.21.15.89:0/4259086263 shutdown_connections 2023-10-30T12:52:59.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.807+0000 7f4b7bf3e640 1 --2- 172.21.15.89:0/4259086263 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f4b580664a0 0x7f4b58068960 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:59.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.807+0000 7f4b7bf3e640 1 --2- 172.21.15.89:0/4259086263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b741508c0 0x7f4b741435b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:59.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.807+0000 7f4b7bf3e640 1 --2- 172.21.15.89:0/4259086263 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b740c6e60 0x7f4b74154410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:59.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.807+0000 7f4b7bf3e640 1 --2- 172.21.15.89:0/4259086263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b740722a0 0x7f4b740ad8d0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:59.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.807+0000 7f4b7bf3e640 1 -- 172.21.15.89:0/4259086263 >> 172.21.15.89:0/4259086263 conn(0x7f4b740c7910 msgr2=0x7f4b740c6200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:59.809 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.807+0000 7f4b7bf3e640 1 -- 172.21.15.89:0/4259086263 shutdown_connections 2023-10-30T12:52:59.810 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.807+0000 7f4b7bf3e640 1 -- 172.21.15.89:0/4259086263 wait complete. 2023-10-30T12:52:59.825 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-30T12:52:59.826 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:52:59.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.963+0000 7f970f765640 1 Processor -- start 2023-10-30T12:52:59.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.963+0000 7f970f765640 1 -- start start 2023-10-30T12:52:59.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.967+0000 7f970f765640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97081539a0 0x7f970815a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:59.965 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.967+0000 7f970f765640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9708070a00 0x7f9708070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:59.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.967+0000 7f970f765640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9708069e90 0x7f970806a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:59.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.967+0000 7f970f765640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f970806fbb0 con 0x7f9708070a00 2023-10-30T12:52:59.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.967+0000 7f970f765640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f970815a8c0 con 0x7f9708069e90 2023-10-30T12:52:59.966 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.967+0000 7f970f765640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9708071380 con 0x7f97081539a0 2023-10-30T12:52:59.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.967+0000 7f970cd00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9708070a00 0x7f9708070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:59.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.967+0000 7f970cd00640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f9708070a00 0x7f9708070e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48712/0 (socket says 172.21.15.89:48712) 2023-10-30T12:52:59.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.967+0000 7f970cd00640 1 -- 172.21.15.89:0/3242464010 learned_addr learned my addr 172.21.15.89:0/3242464010 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:52:59.967 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.967+0000 7f970d501640 1 --2- 172.21.15.89:0/3242464010 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97081539a0 0x7f970815a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:59.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.967+0000 7f970dd02640 1 --2- 172.21.15.89:0/3242464010 >> v2:172.21.15.192:3300/0 conn(0x7f9708069e90 0x7f970806a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:59.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.967+0000 7f970cd00640 1 -- 172.21.15.89:0/3242464010 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97081539a0 msgr2=0x7f970815a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:59.968 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.967+0000 7f970cd00640 1 --2- 172.21.15.89:0/3242464010 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97081539a0 0x7f970815a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:59.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.967+0000 7f970cd00640 1 -- 172.21.15.89:0/3242464010 >> v2:172.21.15.192:3300/0 conn(0x7f9708069e90 msgr2=0x7f970806a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:59.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.967+0000 7f970cd00640 1 --2- 172.21.15.89:0/3242464010 >> v2:172.21.15.192:3300/0 conn(0x7f9708069e90 0x7f970806a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:59.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.967+0000 7f970cd00640 1 -- 172.21.15.89:0/3242464010 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9708153d80 con 0x7f9708070a00 2023-10-30T12:52:59.969 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.967+0000 7f970d501640 1 --2- 172.21.15.89:0/3242464010 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97081539a0 0x7f970815a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:52:59.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.971+0000 7f970cd00640 1 --2- 172.21.15.89:0/3242464010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9708070a00 0x7f9708070e40 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f970806cdd0 tx=0x7f96f801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7888b5a64c7a7983 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:59.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.971+0000 7f96fffff640 1 -- 172.21.15.89:0/3242464010 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f80ac640 con 0x7f9708070a00 2023-10-30T12:52:59.971 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.971+0000 7f96fffff640 1 -- 172.21.15.89:0/3242464010 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f96f80ac7a0 con 0x7f9708070a00 2023-10-30T12:52:59.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.971+0000 7f96fffff640 1 -- 172.21.15.89:0/3242464010 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f80acb50 con 0x7f9708070a00 2023-10-30T12:52:59.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.971+0000 7f970f765640 1 -- 172.21.15.89:0/3242464010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9708070a00 msgr2=0x7f9708070e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:59.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.971+0000 7f970f765640 1 --2- 172.21.15.89:0/3242464010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9708070a00 0x7f9708070e40 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f970806cdd0 tx=0x7f96f801c440 comp rx=0 tx=0).stop 2023-10-30T12:52:59.972 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.971+0000 7f970f765640 1 -- 172.21.15.89:0/3242464010 shutdown_connections 2023-10-30T12:52:59.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.971+0000 7f970f765640 1 --2- 172.21.15.89:0/3242464010 >> v2:172.21.15.192:3300/0 conn(0x7f9708069e90 0x7f970806a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:59.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.971+0000 7f970f765640 1 --2- 172.21.15.89:0/3242464010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9708070a00 0x7f9708070e40 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:59.973 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.971+0000 7f970f765640 1 --2- 172.21.15.89:0/3242464010 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97081539a0 0x7f970815a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:59.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.971+0000 7f970f765640 1 -- 172.21.15.89:0/3242464010 >> 172.21.15.89:0/3242464010 conn(0x7f97080c7910 msgr2=0x7f97080c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:59.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.971+0000 7f970f765640 1 -- 172.21.15.89:0/3242464010 shutdown_connections 2023-10-30T12:52:59.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.971+0000 7f970f765640 1 -- 172.21.15.89:0/3242464010 wait complete. 2023-10-30T12:52:59.974 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.971+0000 7f970f765640 1 Processor -- start 2023-10-30T12:52:59.975 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.971+0000 7f970f765640 1 -- start start 2023-10-30T12:52:59.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.975+0000 7f970f765640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9708069e90 0x7f97081589d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:59.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.975+0000 7f970f765640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9708070a00 0x7f97080c0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:59.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.975+0000 7f970f765640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97081539a0 0x7f9708145790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:59.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.975+0000 7f970f765640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9708157770 con 0x7f9708070a00 2023-10-30T12:52:59.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.975+0000 7f970f765640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970815c000 con 0x7f9708069e90 2023-10-30T12:52:59.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.975+0000 7f970f765640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f970815cb80 con 0x7f97081539a0 2023-10-30T12:52:59.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.975+0000 7f970cd00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9708070a00 0x7f97080c0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:59.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.975+0000 7f970cd00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9708070a00 0x7f97080c0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48728/0 (socket says 172.21.15.89:48728) 2023-10-30T12:52:59.979 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.975+0000 7f970cd00640 1 -- 172.21.15.89:0/1900555865 learned_addr learned my addr 172.21.15.89:0/1900555865 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:52:59.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.975+0000 7f970d501640 1 --2- 172.21.15.89:0/1900555865 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9708069e90 0x7f97081589d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:59.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.975+0000 7f970dd02640 1 --2- 172.21.15.89:0/1900555865 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97081539a0 0x7f9708145790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:59.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.975+0000 7f970cd00640 1 -- 172.21.15.89:0/1900555865 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97081539a0 msgr2=0x7f9708145790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:59.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.975+0000 7f970cd00640 1 --2- 172.21.15.89:0/1900555865 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97081539a0 0x7f9708145790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:59.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.975+0000 7f970cd00640 1 -- 172.21.15.89:0/1900555865 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9708069e90 msgr2=0x7f97081589d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:59.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.975+0000 7f970cd00640 1 --2- 172.21.15.89:0/1900555865 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9708069e90 0x7f97081589d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:59.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.975+0000 7f970cd00640 1 -- 172.21.15.89:0/1900555865 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f80094d0 con 0x7f9708070a00 2023-10-30T12:52:59.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.975+0000 7f970cd00640 1 --2- 172.21.15.89:0/1900555865 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9708070a00 0x7f97080c0120 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f96f801aaf0 tx=0x7f96f80191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:59.982 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.975+0000 7f96fdffb640 1 -- 172.21.15.89:0/1900555865 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f80b5850 con 0x7f9708070a00 2023-10-30T12:52:59.982 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.975+0000 7f96fdffb640 1 -- 172.21.15.89:0/1900555865 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f96f80b59b0 con 0x7f9708070a00 2023-10-30T12:52:59.982 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.975+0000 7f970f765640 1 -- 172.21.15.89:0/1900555865 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97081579a0 con 0x7f9708070a00 2023-10-30T12:52:59.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.975+0000 7f96fdffb640 1 -- 172.21.15.89:0/1900555865 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f80b5d60 con 0x7f9708070a00 2023-10-30T12:52:59.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.975+0000 7f970f765640 1 -- 172.21.15.89:0/1900555865 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f970815c530 con 0x7f9708070a00 2023-10-30T12:52:59.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.975+0000 7f970f765640 1 -- 172.21.15.89:0/1900555865 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f970804e170 con 0x7f9708070a00 2023-10-30T12:52:59.983 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.975+0000 7f96fdffb640 1 -- 172.21.15.89:0/1900555865 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f96f80b3070 con 0x7f9708070a00 2023-10-30T12:52:59.984 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.979+0000 7f96fdffb640 1 --2- 172.21.15.89:0/1900555865 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f96ec0664a0 0x7f96ec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:59.984 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.979+0000 7f96fdffb640 1 -- 172.21.15.89:0/1900555865 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f96f811e550 con 0x7f9708070a00 2023-10-30T12:52:59.984 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.979+0000 7f970d501640 1 --2- 172.21.15.89:0/1900555865 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f96ec0664a0 0x7f96ec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:59.984 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.979+0000 7f970d501640 1 --2- 172.21.15.89:0/1900555865 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f96ec0664a0 0x7f96ec068960 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f97040045f0 tx=0x7f9704008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:59.985 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:52:59.983+0000 7f96fdffb640 1 -- 172.21.15.89:0/1900555865 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f96f80eb530 con 0x7f9708070a00 2023-10-30T12:53:00.161 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.163+0000 7f970f765640 1 -- 172.21.15.89:0/1900555865 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f970815afa0 con 0x7f9708070a00 2023-10-30T12:53:00.162 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.163+0000 7f96fdffb640 1 -- 172.21.15.89:0/1900555865 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+6540 (secure 0 0 0) 0x7f96f80ac600 con 0x7f9708070a00 2023-10-30T12:53:00.162 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:53:00.163 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":121,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5906,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/4085020291","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":4085020291},{"type":"v1","addr":"172.21.15.192:6837","nonce":4085020291}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:52:51.500390+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6271,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6271":{"gid":6271,"name":"b","rank":0,"incarnation":110,"state":"up:active","state_seq":29,"addr":"172.21.15.192:6835/659375964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":659375964},{"type":"v1","addr":"172.21.15.192:6835","nonce":659375964}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:53:00.164 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 121 2023-10-30T12:53:00.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.167+0000 7f970f765640 1 -- 172.21.15.89:0/1900555865 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f96ec0664a0 msgr2=0x7f96ec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:00.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.167+0000 7f970f765640 1 --2- 172.21.15.89:0/1900555865 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f96ec0664a0 0x7f96ec068960 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f97040045f0 tx=0x7f9704008040 comp rx=0 tx=0).stop 2023-10-30T12:53:00.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.167+0000 7f970f765640 1 -- 172.21.15.89:0/1900555865 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9708070a00 msgr2=0x7f97080c0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:00.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.167+0000 7f970f765640 1 --2- 172.21.15.89:0/1900555865 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9708070a00 0x7f97080c0120 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f96f801aaf0 tx=0x7f96f80191e0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.167+0000 7f970f765640 1 -- 172.21.15.89:0/1900555865 shutdown_connections 2023-10-30T12:53:00.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.167+0000 7f970f765640 1 --2- 172.21.15.89:0/1900555865 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f96ec0664a0 0x7f96ec068960 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.167+0000 7f970f765640 1 --2- 172.21.15.89:0/1900555865 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97081539a0 0x7f9708145790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.167+0000 7f970f765640 1 --2- 172.21.15.89:0/1900555865 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9708070a00 0x7f97080c0120 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.167+0000 7f970f765640 1 --2- 172.21.15.89:0/1900555865 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9708069e90 0x7f97081589d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.167+0000 7f970f765640 1 -- 172.21.15.89:0/1900555865 >> 172.21.15.89:0/1900555865 conn(0x7f97080c7910 msgr2=0x7f97081499d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:53:00.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.167+0000 7f970f765640 1 -- 172.21.15.89:0/1900555865 shutdown_connections 2023-10-30T12:53:00.169 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.167+0000 7f970f765640 1 -- 172.21.15.89:0/1900555865 wait complete. 2023-10-30T12:53:00.183 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:53:00.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.323+0000 7fe354380640 1 Processor -- start 2023-10-30T12:53:00.320 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.323+0000 7fe354380640 1 -- start start 2023-10-30T12:53:00.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.323+0000 7fe354380640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe34c0c6b80 0x7fe34c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:00.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.323+0000 7fe354380640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe34c0c6f60 0x7fe34c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:00.322 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.323+0000 7fe354380640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe34c0c0a50 con 0x7fe34c0c8430 2023-10-30T12:53:00.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.323+0000 7fe354380640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe34c06f6e0 con 0x7fe34c0c6b80 2023-10-30T12:53:00.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.323+0000 7fe354380640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe34c06f9c0 con 0x7fe34c0c6f60 2023-10-30T12:53:00.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.323+0000 7fe35211c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe34c0c6b80 0x7fe34c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:00.323 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.323+0000 7fe35211c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fe34c0c6b80 0x7fe34c0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48736/0 (socket says 172.21.15.89:48736) 2023-10-30T12:53:00.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.323+0000 7fe35211c640 1 -- 172.21.15.89:0/3823943419 learned_addr learned my addr 172.21.15.89:0/3823943419 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:53:00.324 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.323+0000 7fe35191b640 1 --2- 172.21.15.89:0/3823943419 >> v2:172.21.15.192:3300/0 conn(0x7fe34c0c6f60 0x7fe34c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:00.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.323+0000 7fe35211c640 1 -- 172.21.15.89:0/3823943419 >> v2:172.21.15.192:3300/0 conn(0x7fe34c0c6f60 msgr2=0x7fe34c0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:53:00.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.323+0000 7fe35211c640 1 --2- 172.21.15.89:0/3823943419 >> v2:172.21.15.192:3300/0 conn(0x7fe34c0c6f60 0x7fe34c0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.325 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.323+0000 7fe35211c640 1 -- 172.21.15.89:0/3823943419 >> v1:172.21.15.192:6789/0 conn(0x7fe34c0c8430 legacy=0x7fe34c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:53:00.326 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.323+0000 7fe35211c640 1 -- 172.21.15.89:0/3823943419 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe34c06fca0 con 0x7fe34c0c6b80 2023-10-30T12:53:00.326 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.327+0000 7fe35211c640 1 --2- 172.21.15.89:0/3823943419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe34c0c6b80 0x7fe34c0bfec0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fe34c159b90 tx=0x7fe34801c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7af1a04eaee28a31 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:53:00.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.327+0000 7fe35111a640 1 -- 172.21.15.89:0/3823943419 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3480ac630 con 0x7fe34c0c6b80 2023-10-30T12:53:00.327 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.327+0000 7fe35111a640 1 -- 172.21.15.89:0/3823943419 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fe3480ac790 con 0x7fe34c0c6b80 2023-10-30T12:53:00.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.327+0000 7fe35111a640 1 -- 172.21.15.89:0/3823943419 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3480acb40 con 0x7fe34c0c6b80 2023-10-30T12:53:00.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.327+0000 7fe354380640 1 -- 172.21.15.89:0/3823943419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe34c0c6b80 msgr2=0x7fe34c0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:00.328 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.327+0000 7fe354380640 1 --2- 172.21.15.89:0/3823943419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe34c0c6b80 0x7fe34c0bfec0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fe34c159b90 tx=0x7fe34801c4c0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.327+0000 7fe354380640 1 -- 172.21.15.89:0/3823943419 shutdown_connections 2023-10-30T12:53:00.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.327+0000 7fe354380640 1 --2- 172.21.15.89:0/3823943419 >> v2:172.21.15.192:3300/0 conn(0x7fe34c0c6f60 0x7fe34c0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.327+0000 7fe354380640 1 --2- 172.21.15.89:0/3823943419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe34c0c6b80 0x7fe34c0bfec0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.329 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.327+0000 7fe354380640 1 -- 172.21.15.89:0/3823943419 >> 172.21.15.89:0/3823943419 conn(0x7fe34c0c7910 msgr2=0x7fe34c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:53:00.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.327+0000 7fe354380640 1 -- 172.21.15.89:0/3823943419 shutdown_connections 2023-10-30T12:53:00.330 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.327+0000 7fe354380640 1 -- 172.21.15.89:0/3823943419 wait complete. 2023-10-30T12:53:00.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.331+0000 7fe354380640 1 Processor -- start 2023-10-30T12:53:00.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.331+0000 7fe354380640 1 -- start start 2023-10-30T12:53:00.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.331+0000 7fe354380640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe34c0c6b80 0x7fe34c135690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:00.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.331+0000 7fe354380640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe34c0c6f60 0x7fe34c1510a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:00.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.331+0000 7fe354380640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe34c0c8430 0x7fe34c14fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:00.333 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.331+0000 7fe354380640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c13b870 con 0x7fe34c0c8430 2023-10-30T12:53:00.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.331+0000 7fe354380640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c13d370 con 0x7fe34c0c6f60 2023-10-30T12:53:00.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.331+0000 7fe354380640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c155f90 con 0x7fe34c0c6b80 2023-10-30T12:53:00.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.331+0000 7fe35291d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe34c0c8430 0x7fe34c14fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:00.334 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.331+0000 7fe35291d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe34c0c8430 0x7fe34c14fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48748/0 (socket says 172.21.15.89:48748) 2023-10-30T12:53:00.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.331+0000 7fe35291d640 1 -- 172.21.15.89:0/2366114839 learned_addr learned my addr 172.21.15.89:0/2366114839 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:53:00.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.331+0000 7fe35191b640 1 --2- 172.21.15.89:0/2366114839 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe34c0c6f60 0x7fe34c1510a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:00.335 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.331+0000 7fe35211c640 1 --2- 172.21.15.89:0/2366114839 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe34c0c6b80 0x7fe34c135690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:00.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.331+0000 7fe35291d640 1 -- 172.21.15.89:0/2366114839 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe34c0c6b80 msgr2=0x7fe34c135690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:00.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.331+0000 7fe35291d640 1 --2- 172.21.15.89:0/2366114839 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe34c0c6b80 0x7fe34c135690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.331+0000 7fe35291d640 1 -- 172.21.15.89:0/2366114839 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe34c0c6f60 msgr2=0x7fe34c1510a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:00.336 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.331+0000 7fe35291d640 1 --2- 172.21.15.89:0/2366114839 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe34c0c6f60 0x7fe34c1510a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.331+0000 7fe35291d640 1 -- 172.21.15.89:0/2366114839 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3480094d0 con 0x7fe34c0c8430 2023-10-30T12:53:00.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.331+0000 7fe35291d640 1 --2- 172.21.15.89:0/2366114839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe34c0c8430 0x7fe34c14fdc0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fe34400b970 tx=0x7fe34400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:53:00.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.331+0000 7fe342ffd640 1 -- 172.21.15.89:0/2366114839 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34400cb40 con 0x7fe34c0c8430 2023-10-30T12:53:00.337 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.331+0000 7fe342ffd640 1 -- 172.21.15.89:0/2366114839 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fe34400cca0 con 0x7fe34c0c8430 2023-10-30T12:53:00.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.331+0000 7fe354380640 1 -- 172.21.15.89:0/2366114839 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe34c156270 con 0x7fe34c0c8430 2023-10-30T12:53:00.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.331+0000 7fe342ffd640 1 -- 172.21.15.89:0/2366114839 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3440146a0 con 0x7fe34c0c8430 2023-10-30T12:53:00.338 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.331+0000 7fe354380640 1 -- 172.21.15.89:0/2366114839 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe34c13cb70 con 0x7fe34c0c8430 2023-10-30T12:53:00.339 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.335+0000 7fe354380640 1 -- 172.21.15.89:0/2366114839 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe34c13c6f0 con 0x7fe34c0c8430 2023-10-30T12:53:00.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.335+0000 7fe342ffd640 1 -- 172.21.15.89:0/2366114839 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe34401d3b0 con 0x7fe34c0c8430 2023-10-30T12:53:00.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.335+0000 7fe342ffd640 1 --2- 172.21.15.89:0/2366114839 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe3300664d0 0x7fe330068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:00.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.335+0000 7fe342ffd640 1 -- 172.21.15.89:0/2366114839 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7fe344083f70 con 0x7fe34c0c8430 2023-10-30T12:53:00.340 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.335+0000 7fe35211c640 1 --2- 172.21.15.89:0/2366114839 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe3300664d0 0x7fe330068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:00.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.335+0000 7fe35211c640 1 --2- 172.21.15.89:0/2366114839 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe3300664d0 0x7fe330068990 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fe3480ae000 tx=0x7fe348008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:53:00.341 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.339+0000 7fe342ffd640 1 -- 172.21.15.89:0/2366114839 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe3440b5090 con 0x7fe34c0c8430 2023-10-30T12:53:00.518 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.519+0000 7fe354380640 1 -- 172.21.15.89:0/2366114839 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe34c04e170 con 0x7fe34c0c8430 2023-10-30T12:53:00.519 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.519+0000 7fe342ffd640 1 -- 172.21.15.89:0/2366114839 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+6540 (secure 0 0 0) 0x7fe34c13c6f0 con 0x7fe34c0c8430 2023-10-30T12:53:00.520 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:53:00.520 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":121,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4236,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6834/2909790173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2909790173},{"type":"v1","addr":"172.21.15.89:6834","nonce":2909790173}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/2387357615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6833","nonce":2387357615},{"type":"v1","addr":"172.21.15.89:6835","nonce":2387357615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5906,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/4085020291","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":4085020291},{"type":"v1","addr":"172.21.15.192:6837","nonce":4085020291}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:50.729245+0000","modified":"2023-10-30T12:52:51.500390+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6271,"mds_1":6075,"mds_2":5728},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5728":{"gid":5728,"name":"d","rank":2,"incarnation":73,"state":"up:active","state_seq":19,"addr":"172.21.15.192:6839/3991891597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":3991891597},{"type":"v1","addr":"172.21.15.192:6839","nonce":3991891597}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6075":{"gid":6075,"name":"c","rank":1,"incarnation":92,"state":"up:active","state_seq":38,"addr":"172.21.15.89:6837/2412618858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":2412618858},{"type":"v1","addr":"172.21.15.89:6837","nonce":2412618858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6271":{"gid":6271,"name":"b","rank":0,"incarnation":110,"state":"up:active","state_seq":29,"addr":"172.21.15.192:6835/659375964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":659375964},{"type":"v1","addr":"172.21.15.192:6835","nonce":659375964}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:53:00.521 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 121 2023-10-30T12:53:00.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.523+0000 7fe354380640 1 -- 172.21.15.89:0/2366114839 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe3300664d0 msgr2=0x7fe330068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:00.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.523+0000 7fe354380640 1 --2- 172.21.15.89:0/2366114839 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe3300664d0 0x7fe330068990 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fe3480ae000 tx=0x7fe348008040 comp rx=0 tx=0).stop 2023-10-30T12:53:00.522 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.523+0000 7fe354380640 1 -- 172.21.15.89:0/2366114839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe34c0c8430 msgr2=0x7fe34c14fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:00.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.523+0000 7fe354380640 1 --2- 172.21.15.89:0/2366114839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe34c0c8430 0x7fe34c14fdc0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fe34400b970 tx=0x7fe34400be30 comp rx=0 tx=0).stop 2023-10-30T12:53:00.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.523+0000 7fe354380640 1 -- 172.21.15.89:0/2366114839 shutdown_connections 2023-10-30T12:53:00.523 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.523+0000 7fe354380640 1 --2- 172.21.15.89:0/2366114839 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7fe3300664d0 0x7fe330068990 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.523+0000 7fe354380640 1 --2- 172.21.15.89:0/2366114839 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe34c0c6f60 0x7fe34c1510a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.523+0000 7fe354380640 1 --2- 172.21.15.89:0/2366114839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe34c0c8430 0x7fe34c14fdc0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.523+0000 7fe354380640 1 --2- 172.21.15.89:0/2366114839 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe34c0c6b80 0x7fe34c135690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.524 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.523+0000 7fe354380640 1 -- 172.21.15.89:0/2366114839 >> 172.21.15.89:0/2366114839 conn(0x7fe34c0c7910 msgr2=0x7fe34c1532b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:53:00.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.523+0000 7fe354380640 1 -- 172.21.15.89:0/2366114839 shutdown_connections 2023-10-30T12:53:00.525 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.523+0000 7fe354380640 1 -- 172.21.15.89:0/2366114839 wait complete. 2023-10-30T12:53:00.541 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:53:00.679 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.679+0000 7f9b69605640 1 Processor -- start 2023-10-30T12:53:00.680 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.679+0000 7f9b69605640 1 -- start start 2023-10-30T12:53:00.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b69605640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b641538a0 0x7f9b6415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:00.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b69605640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b640cb800 con 0x7f9b640cf160 2023-10-30T12:53:00.681 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b69605640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b6415dc30 con 0x7f9b640ca0e0 2023-10-30T12:53:00.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b69605640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b640cb330 con 0x7f9b641538a0 2023-10-30T12:53:00.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b637fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9b640cf160 0x7f9b64164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38118/0 (socket says 172.21.15.89:38118) 2023-10-30T12:53:00.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b637fe640 1 -- 172.21.15.89:0/2199308818 learned_addr learned my addr 172.21.15.89:0/2199308818 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:53:00.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b62ffd640 1 --2- 172.21.15.89:0/2199308818 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b641538a0 0x7f9b6415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:00.684 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b61ffb640 1 -- 172.21.15.89:0/2199308818 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 91968506 0 0) 0x7f9b640cb800 con 0x7f9b640cf160 2023-10-30T12:53:00.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b61ffb640 1 -- 172.21.15.89:0/2199308818 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b4c003530 con 0x7f9b640cf160 2023-10-30T12:53:00.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b61ffb640 1 -- 172.21.15.89:0/2199308818 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3786627433 0 0) 0x7f9b58001120 con 0x7f9b640ca0e0 2023-10-30T12:53:00.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b61ffb640 1 -- 172.21.15.89:0/2199308818 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b640cb800 con 0x7f9b640ca0e0 2023-10-30T12:53:00.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b61ffb640 1 -- 172.21.15.89:0/2199308818 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 245872189 0 0) 0x7f9b4c003530 con 0x7f9b640cf160 2023-10-30T12:53:00.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b61ffb640 1 -- 172.21.15.89:0/2199308818 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b641538a0 msgr2=0x7f9b6415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:53:00.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b61ffb640 1 --2- 172.21.15.89:0/2199308818 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b641538a0 0x7f9b6415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b61ffb640 1 -- 172.21.15.89:0/2199308818 >> v1:172.21.15.192:6789/0 conn(0x7f9b640ca0e0 legacy=0x7f9b6415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:00.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b61ffb640 1 -- 172.21.15.89:0/2199308818 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b640cad70 con 0x7f9b640cf160 2023-10-30T12:53:00.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b61ffb640 1 -- 172.21.15.89:0/2199308818 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f9b5c001a00 con 0x7f9b640cf160 2023-10-30T12:53:00.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b62ffd640 1 --2- 172.21.15.89:0/2199308818 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b641538a0 0x7f9b6415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:53:00.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b61ffb640 1 -- 172.21.15.89:0/2199308818 >> v1:172.21.15.89:6789/0 conn(0x7f9b640cf160 legacy=0x7f9b64164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:00.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b61ffb640 1 --2- 172.21.15.89:0/2199308818 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b4c003b40 0x7f9b4c003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:00.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b61ffb640 1 --2- 172.21.15.89:0/2199308818 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b641538a0 0x7f9b4c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:00.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b61ffb640 1 --2- 172.21.15.89:0/2199308818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4c004d10 0x7f9b4c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:00.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b61ffb640 1 -- 172.21.15.89:0/2199308818 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b640cb330 con 0x7f9b4c004d10 2023-10-30T12:53:00.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b61ffb640 1 -- 172.21.15.89:0/2199308818 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b4c00ba40 con 0x7f9b4c003b40 2023-10-30T12:53:00.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b61ffb640 1 -- 172.21.15.89:0/2199308818 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b4c00bb80 con 0x7f9b641538a0 2023-10-30T12:53:00.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b637fe640 1 --2- 172.21.15.89:0/2199308818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4c004d10 0x7f9b4c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:00.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b62ffd640 1 --2- 172.21.15.89:0/2199308818 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b4c003b40 0x7f9b4c003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:00.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b627fc640 1 --2- 172.21.15.89:0/2199308818 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b641538a0 0x7f9b4c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:00.690 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b637fe640 1 -- 172.21.15.89:0/2199308818 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b641538a0 msgr2=0x7f9b4c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:53:00.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b637fe640 1 --2- 172.21.15.89:0/2199308818 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b641538a0 0x7f9b4c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b637fe640 1 -- 172.21.15.89:0/2199308818 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b4c003b40 msgr2=0x7f9b4c003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:53:00.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b637fe640 1 --2- 172.21.15.89:0/2199308818 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b4c003b40 0x7f9b4c003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.691 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.683+0000 7f9b637fe640 1 -- 172.21.15.89:0/2199308818 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b54009420 con 0x7f9b4c004d10 2023-10-30T12:53:00.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.687+0000 7f9b637fe640 1 --2- 172.21.15.89:0/2199308818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4c004d10 0x7f9b4c005190 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c00a800 tx=0x7f9b5c01d280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d6c038267057d1d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:53:00.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.687+0000 7f9b61ffb640 1 -- 172.21.15.89:0/2199308818 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c0ad630 con 0x7f9b4c004d10 2023-10-30T12:53:00.695 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.687+0000 7f9b61ffb640 1 -- 172.21.15.89:0/2199308818 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f9b5c0a8070 con 0x7f9b4c004d10 2023-10-30T12:53:00.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.687+0000 7f9b61ffb640 1 -- 172.21.15.89:0/2199308818 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c0ada10 con 0x7f9b4c004d10 2023-10-30T12:53:00.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.687+0000 7f9b69605640 1 -- 172.21.15.89:0/2199308818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4c004d10 msgr2=0x7f9b4c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:00.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.687+0000 7f9b69605640 1 --2- 172.21.15.89:0/2199308818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4c004d10 0x7f9b4c005190 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c00a800 tx=0x7f9b5c01d280 comp rx=0 tx=0).stop 2023-10-30T12:53:00.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.687+0000 7f9b637fe640 1 -- 172.21.15.89:0/2199308818 reap_dead start 2023-10-30T12:53:00.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.687+0000 7f9b69605640 1 -- 172.21.15.89:0/2199308818 shutdown_connections 2023-10-30T12:53:00.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.687+0000 7f9b69605640 1 -- 172.21.15.89:0/2199308818 >> 172.21.15.89:0/2199308818 conn(0x7f9b6406bfe0 msgr2=0x7f9b6406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:53:00.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.687+0000 7f9b69605640 1 -- 172.21.15.89:0/2199308818 shutdown_connections 2023-10-30T12:53:00.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.687+0000 7f9b69605640 1 -- 172.21.15.89:0/2199308818 wait complete. 2023-10-30T12:53:00.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.691+0000 7f9b69605640 1 Processor -- start 2023-10-30T12:53:00.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.691+0000 7f9b69605640 1 -- start start 2023-10-30T12:53:00.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.691+0000 7f9b69605640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4c004d10 0x7f9b6414e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:00.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.691+0000 7f9b69605640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b64158960 0x7f9b641e2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:00.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.691+0000 7f9b69605640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b6415a8b0 0x7f9b6414bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:00.699 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.691+0000 7f9b69605640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b64159730 con 0x7f9b4c004d10 2023-10-30T12:53:00.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.691+0000 7f9b69605640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b641e53c0 con 0x7f9b6415a8b0 2023-10-30T12:53:00.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.691+0000 7f9b69605640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b641f3390 con 0x7f9b64158960 2023-10-30T12:53:00.700 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.691+0000 7f9b62ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4c004d10 0x7f9b6414e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:00.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.691+0000 7f9b62ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4c004d10 0x7f9b6414e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48762/0 (socket says 172.21.15.89:48762) 2023-10-30T12:53:00.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.691+0000 7f9b62ffd640 1 -- 172.21.15.89:0/399122476 learned_addr learned my addr 172.21.15.89:0/399122476 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:53:00.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.691+0000 7f9b627fc640 1 --2- 172.21.15.89:0/399122476 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b64158960 0x7f9b641e2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:00.701 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.691+0000 7f9b637fe640 1 --2- 172.21.15.89:0/399122476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b6415a8b0 0x7f9b6414bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:00.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.691+0000 7f9b62ffd640 1 -- 172.21.15.89:0/399122476 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b64158960 msgr2=0x7f9b641e2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:00.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.691+0000 7f9b62ffd640 1 --2- 172.21.15.89:0/399122476 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b64158960 0x7f9b641e2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.702 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.691+0000 7f9b62ffd640 1 -- 172.21.15.89:0/399122476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b6415a8b0 msgr2=0x7f9b6414bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:00.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.691+0000 7f9b62ffd640 1 --2- 172.21.15.89:0/399122476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b6415a8b0 0x7f9b6414bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.691+0000 7f9b62ffd640 1 -- 172.21.15.89:0/399122476 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b641f2f00 con 0x7f9b4c004d10 2023-10-30T12:53:00.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.691+0000 7f9b62ffd640 1 --2- 172.21.15.89:0/399122476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4c004d10 0x7f9b6414e350 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f9b54011930 tx=0x7f9b54011df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:53:00.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.691+0000 7f9b4b7fe640 1 -- 172.21.15.89:0/399122476 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b5400fbd0 con 0x7f9b4c004d10 2023-10-30T12:53:00.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.691+0000 7f9b4b7fe640 1 -- 172.21.15.89:0/399122476 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f9b5400fd30 con 0x7f9b4c004d10 2023-10-30T12:53:00.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.691+0000 7f9b4b7fe640 1 -- 172.21.15.89:0/399122476 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b54015680 con 0x7f9b4c004d10 2023-10-30T12:53:00.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.691+0000 7f9b69605640 1 -- 172.21.15.89:0/399122476 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b64159490 con 0x7f9b4c004d10 2023-10-30T12:53:00.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.691+0000 7f9b69605640 1 -- 172.21.15.89:0/399122476 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b641f28b0 con 0x7f9b4c004d10 2023-10-30T12:53:00.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.691+0000 7f9b69605640 1 -- 172.21.15.89:0/399122476 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b64159870 con 0x7f9b4c004d10 2023-10-30T12:53:00.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.695+0000 7f9b4b7fe640 1 -- 172.21.15.89:0/399122476 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9b540157e0 con 0x7f9b4c004d10 2023-10-30T12:53:00.705 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.695+0000 7f9b4b7fe640 1 --2- 172.21.15.89:0/399122476 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9b500664d0 0x7f9b50068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:00.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.695+0000 7f9b4b7fe640 1 -- 172.21.15.89:0/399122476 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7f9b54084d80 con 0x7f9b4c004d10 2023-10-30T12:53:00.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.695+0000 7f9b627fc640 1 --2- 172.21.15.89:0/399122476 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9b500664d0 0x7f9b50068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:00.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.695+0000 7f9b627fc640 1 --2- 172.21.15.89:0/399122476 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9b500664d0 0x7f9b50068990 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f9b58001280 tx=0x7f9b58009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:53:00.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.699+0000 7f9b4b7fe640 1 -- 172.21.15.89:0/399122476 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9b64159870 con 0x7f9b4c004d10 2023-10-30T12:53:00.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.847+0000 7f9b69605640 1 -- 172.21.15.89:0/399122476 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b64159870 con 0x7f9b4c004d10 2023-10-30T12:53:00.848 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.851+0000 7f9b4b7fe640 1 -- 172.21.15.89:0/399122476 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+18124 (secure 0 0 0) 0x7f9b64159870 con 0x7f9b4c004d10 2023-10-30T12:53:00.849 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-10-30T12:53:00.849 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":66,"fsid":"401641be-db6e-4c16-b3ac-fd84a86c4de4","created":"2023-10-30T12:28:26.198532+0000","modified":"2023-10-30T12:50:41.318657+0000","last_up_change":"2023-10-30T12:28:37.212205+0000","last_in_change":"2023-10-30T12:28:28.517734+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:39.004943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T12:28:48.935909+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T12:28:49.716098+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":58,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"72595c54-fd6e-4a5a-bb14-ff27e9377b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6801","nonce":1673955622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6803","nonce":1673955622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6807","nonce":1673955622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":1673955622},{"type":"v1","addr":"172.21.15.89:6805","nonce":1673955622}]},"public_addr":"172.21.15.89:6801/1673955622","cluster_addr":"172.21.15.89:6803/1673955622","heartbeat_back_addr":"172.21.15.89:6807/1673955622","heartbeat_front_addr":"172.21.15.89:6805/1673955622","state":["exists","up"]},{"osd":1,"uuid":"0305670b-d7c4-4193-bec2-8b7351ff590a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6817","nonce":3340572039}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6819","nonce":3340572039}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6823","nonce":3340572039}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3340572039},{"type":"v1","addr":"172.21.15.89:6821","nonce":3340572039}]},"public_addr":"172.21.15.89:6817/3340572039","cluster_addr":"172.21.15.89:6819/3340572039","heartbeat_back_addr":"172.21.15.89:6823/3340572039","heartbeat_front_addr":"172.21.15.89:6821/3340572039","state":["exists","up"]},{"osd":2,"uuid":"4bab5426-a3c3-46b0-91fc-546c254006c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6809","nonce":1929940799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6811","nonce":1929940799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6815","nonce":1929940799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":1929940799},{"type":"v1","addr":"172.21.15.89:6813","nonce":1929940799}]},"public_addr":"172.21.15.89:6809/1929940799","cluster_addr":"172.21.15.89:6811/1929940799","heartbeat_back_addr":"172.21.15.89:6815/1929940799","heartbeat_front_addr":"172.21.15.89:6813/1929940799","state":["exists","up"]},{"osd":3,"uuid":"7d47aea8-27cd-42be-8591-d63ad68c90e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6825","nonce":345087939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6827","nonce":345087939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6831","nonce":345087939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":345087939},{"type":"v1","addr":"172.21.15.89:6829","nonce":345087939}]},"public_addr":"172.21.15.89:6825/345087939","cluster_addr":"172.21.15.89:6827/345087939","heartbeat_back_addr":"172.21.15.89:6831/345087939","heartbeat_front_addr":"172.21.15.89:6829/345087939","state":["exists","up"]},{"osd":4,"uuid":"2e2ae8f8-d097-4bba-9448-95047fbfacad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6801","nonce":2138125545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6803","nonce":2138125545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6809","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6811","nonce":2138125545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6805","nonce":2138125545},{"type":"v1","addr":"172.21.15.192:6807","nonce":2138125545}]},"public_addr":"172.21.15.192:6801/2138125545","cluster_addr":"172.21.15.192:6803/2138125545","heartbeat_back_addr":"172.21.15.192:6811/2138125545","heartbeat_front_addr":"172.21.15.192:6807/2138125545","state":["exists","up"]},{"osd":5,"uuid":"49be5ce5-12f4-4c26-8d39-864b6ff67313","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6806","nonce":2360218671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6810","nonce":2360218671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6815","nonce":2360218671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2360218671},{"type":"v1","addr":"172.21.15.192:6813","nonce":2360218671}]},"public_addr":"172.21.15.192:6806/2360218671","cluster_addr":"172.21.15.192:6810/2360218671","heartbeat_back_addr":"172.21.15.192:6815/2360218671","heartbeat_front_addr":"172.21.15.192:6813/2360218671","state":["exists","up"]},{"osd":6,"uuid":"48623452-d456-4ea2-a0de-5fa4b37e6d10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6825","nonce":3707671287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6827","nonce":3707671287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6831","nonce":3707671287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3707671287},{"type":"v1","addr":"172.21.15.192:6829","nonce":3707671287}]},"public_addr":"172.21.15.192:6825/3707671287","cluster_addr":"172.21.15.192:6827/3707671287","heartbeat_back_addr":"172.21.15.192:6831/3707671287","heartbeat_front_addr":"172.21.15.192:6829/3707671287","state":["exists","up"]},{"osd":7,"uuid":"ff0a4384-54f8-43ec-a3ce-9c8ae3bf8a94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6817","nonce":3111945871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6819","nonce":3111945871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6823","nonce":3111945871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3111945871},{"type":"v1","addr":"172.21.15.192:6821","nonce":3111945871}]},"public_addr":"172.21.15.192:6817/3111945871","cluster_addr":"172.21.15.192:6819/3111945871","heartbeat_back_addr":"172.21.15.192:6823/3111945871","heartbeat_front_addr":"172.21.15.192:6821/3111945871","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.183065+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.248650+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.194576+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.397483+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.043648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.186624+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.159674+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:34.100851+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2},{"from":5,"to":1}]},{"pgid":"3.13","mappings":[{"from":7,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.192:6837/2130807115":"2023-10-31T12:50:41.318586+0000","172.21.15.192:6834/501291700":"2023-10-31T12:49:01.291877+0000","172.21.15.192:6837/604357386":"2023-10-31T12:48:11.263450+0000","172.21.15.192:6836/604357386":"2023-10-31T12:48:11.263450+0000","172.21.15.89:6836/951276252":"2023-10-31T12:45:51.226535+0000","172.21.15.192:6834/1289220491":"2023-10-31T12:31:20.953485+0000","172.21.15.192:6835/1289220491":"2023-10-31T12:31:20.953485+0000","172.21.15.192:6837/1413360446":"2023-10-31T12:43:36.177020+0000","172.21.15.192:6839/3339589587":"2023-10-31T12:34:56.026152+0000","172.21.15.192:6838/93914587":"2023-10-31T12:42:26.158883+0000","172.21.15.89:6837/2053475104":"2023-10-31T12:33:10.981311+0000","172.21.15.192:6836/2130807115":"2023-10-31T12:50:41.318586+0000","172.21.15.192:6836/1413360446":"2023-10-31T12:43:36.177020+0000","172.21.15.192:6838/3339589587":"2023-10-31T12:34:56.026152+0000","172.21.15.89:6837/951276252":"2023-10-31T12:45:51.226535+0000","172.21.15.192:6836/1002124471":"2023-10-31T12:37:36.081217+0000","172.21.15.192:6837/1002124471":"2023-10-31T12:37:36.081217+0000","172.21.15.192:6835/501291700":"2023-10-31T12:49:01.291877+0000","172.21.15.192:6834/654256299":"2023-10-31T12:40:11.121122+0000","172.21.15.89:6836/2053475104":"2023-10-31T12:33:10.981311+0000","172.21.15.192:6835/654256299":"2023-10-31T12:40:11.121122+0000","172.21.15.192:6839/93914587":"2023-10-31T12:42:26.158883+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:53:00.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.851+0000 7f9b69605640 1 -- 172.21.15.89:0/399122476 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9b500664d0 msgr2=0x7f9b50068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:00.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.851+0000 7f9b69605640 1 --2- 172.21.15.89:0/399122476 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9b500664d0 0x7f9b50068990 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f9b58001280 tx=0x7f9b58009040 comp rx=0 tx=0).stop 2023-10-30T12:53:00.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.851+0000 7f9b69605640 1 -- 172.21.15.89:0/399122476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4c004d10 msgr2=0x7f9b6414e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:00.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.851+0000 7f9b69605640 1 --2- 172.21.15.89:0/399122476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4c004d10 0x7f9b6414e350 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f9b54011930 tx=0x7f9b54011df0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.851 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.851+0000 7f9b69605640 1 -- 172.21.15.89:0/399122476 shutdown_connections 2023-10-30T12:53:00.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.851+0000 7f9b69605640 1 --2- 172.21.15.89:0/399122476 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f9b500664d0 0x7f9b50068990 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.851+0000 7f9b69605640 1 --2- 172.21.15.89:0/399122476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b6415a8b0 0x7f9b6414bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.852 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.851+0000 7f9b69605640 1 --2- 172.21.15.89:0/399122476 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9b64158960 0x7f9b641e2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.851+0000 7f9b69605640 1 --2- 172.21.15.89:0/399122476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9b4c004d10 0x7f9b6414e350 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.851+0000 7f9b69605640 1 -- 172.21.15.89:0/399122476 >> 172.21.15.89:0/399122476 conn(0x7f9b6406bfe0 msgr2=0x7f9b6414fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:53:00.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.851+0000 7f9b69605640 1 -- 172.21.15.89:0/399122476 shutdown_connections 2023-10-30T12:53:00.853 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:00.851+0000 7f9b69605640 1 -- 172.21.15.89:0/399122476 wait complete. 2023-10-30T12:53:00.870 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-30T12:53:01.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.011+0000 7fad157f8640 1 Processor -- start 2023-10-30T12:53:01.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.011+0000 7fad157f8640 1 -- start start 2023-10-30T12:53:01.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.011+0000 7fad157f8640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad100c6b80 0x7fad10071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:01.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.011+0000 7fad157f8640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fad1013fd10 0x7fad101565a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:01.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.011+0000 7fad157f8640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad10070010 con 0x7fad100c6f60 2023-10-30T12:53:01.011 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.011+0000 7fad157f8640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad10072260 con 0x7fad1013fd10 2023-10-30T12:53:01.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.011+0000 7fad157f8640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad1006f6e0 con 0x7fad100c6b80 2023-10-30T12:53:01.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.011+0000 7fad0f7fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fad1013fd10 0x7fad101565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:01.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.011+0000 7fad0f7fe640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7fad1013fd10 0x7fad101565a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48770/0 (socket says 172.21.15.89:48770) 2023-10-30T12:53:01.012 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.011+0000 7fad0f7fe640 1 -- 172.21.15.89:0/3393774623 learned_addr learned my addr 172.21.15.89:0/3393774623 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:53:01.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.011+0000 7fad0effd640 1 --2- 172.21.15.89:0/3393774623 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad100c6b80 0x7fad10071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:01.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.011+0000 7fad0f7fe640 1 -- 172.21.15.89:0/3393774623 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad100c6b80 msgr2=0x7fad10071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:53:01.013 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.011+0000 7fad0f7fe640 1 --2- 172.21.15.89:0/3393774623 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad100c6b80 0x7fad10071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:01.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.011+0000 7fad0f7fe640 1 -- 172.21.15.89:0/3393774623 >> v1:172.21.15.192:6789/0 conn(0x7fad100c6f60 legacy=0x7fad1013f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:01.014 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.011+0000 7fad0f7fe640 1 -- 172.21.15.89:0/3393774623 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad1006f9c0 con 0x7fad1013fd10 2023-10-30T12:53:01.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.015+0000 7fad0f7fe640 1 --2- 172.21.15.89:0/3393774623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad1013fd10 0x7fad101565a0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fad08009e50 tx=0x7fad0801c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38d46de521a5b77 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:53:01.015 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.015+0000 7fad0dffb640 1 -- 172.21.15.89:0/3393774623 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad080a58d0 con 0x7fad1013fd10 2023-10-30T12:53:01.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.015+0000 7fad0dffb640 1 -- 172.21.15.89:0/3393774623 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fad080a5a30 con 0x7fad1013fd10 2023-10-30T12:53:01.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.015+0000 7fad0dffb640 1 -- 172.21.15.89:0/3393774623 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad080a5de0 con 0x7fad1013fd10 2023-10-30T12:53:01.016 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.015+0000 7fad157f8640 1 -- 172.21.15.89:0/3393774623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad1013fd10 msgr2=0x7fad101565a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:01.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.015+0000 7fad157f8640 1 --2- 172.21.15.89:0/3393774623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad1013fd10 0x7fad101565a0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fad08009e50 tx=0x7fad0801c9e0 comp rx=0 tx=0).stop 2023-10-30T12:53:01.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.015+0000 7fad157f8640 1 -- 172.21.15.89:0/3393774623 shutdown_connections 2023-10-30T12:53:01.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.015+0000 7fad157f8640 1 --2- 172.21.15.89:0/3393774623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad1013fd10 0x7fad101565a0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:01.017 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.015+0000 7fad157f8640 1 --2- 172.21.15.89:0/3393774623 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad100c6b80 0x7fad10071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:01.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.015+0000 7fad157f8640 1 -- 172.21.15.89:0/3393774623 >> 172.21.15.89:0/3393774623 conn(0x7fad100c7910 msgr2=0x7fad100c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:53:01.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.015+0000 7fad157f8640 1 -- 172.21.15.89:0/3393774623 shutdown_connections 2023-10-30T12:53:01.018 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.015+0000 7fad157f8640 1 -- 172.21.15.89:0/3393774623 wait complete. 2023-10-30T12:53:01.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.019+0000 7fad157f8640 1 Processor -- start 2023-10-30T12:53:01.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.019+0000 7fad157f8640 1 -- start start 2023-10-30T12:53:01.021 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.019+0000 7fad157f8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad100c6b80 0x7fad101f40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:01.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.019+0000 7fad157f8640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad100c6f60 0x7fad10131420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:01.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.019+0000 7fad157f8640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad10157700 0x7fad1013e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:01.022 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.019+0000 7fad157f8640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad10139ac0 con 0x7fad100c6b80 2023-10-30T12:53:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.019+0000 7fad157f8640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad10133590 con 0x7fad10157700 2023-10-30T12:53:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.019+0000 7fad157f8640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad10132fd0 con 0x7fad100c6f60 2023-10-30T12:53:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.019+0000 7fad0effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad100c6b80 0x7fad101f40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:01.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.019+0000 7fad0effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad100c6b80 0x7fad101f40e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48772/0 (socket says 172.21.15.89:48772) 2023-10-30T12:53:01.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.019+0000 7fad0effd640 1 -- 172.21.15.89:0/2534041816 learned_addr learned my addr 172.21.15.89:0/2534041816 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:53:01.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.019+0000 7fad0f7fe640 1 --2- 172.21.15.89:0/2534041816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad10157700 0x7fad1013e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:01.024 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.019+0000 7fad0e7fc640 1 --2- 172.21.15.89:0/2534041816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad100c6f60 0x7fad10131420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.019+0000 7fad0effd640 1 -- 172.21.15.89:0/2534041816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad100c6f60 msgr2=0x7fad10131420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.019+0000 7fad0effd640 1 --2- 172.21.15.89:0/2534041816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad100c6f60 0x7fad10131420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.019+0000 7fad0effd640 1 -- 172.21.15.89:0/2534041816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad10157700 msgr2=0x7fad1013e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.019+0000 7fad0effd640 1 --2- 172.21.15.89:0/2534041816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad10157700 0x7fad1013e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.019+0000 7fad0effd640 1 -- 172.21.15.89:0/2534041816 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad08009b30 con 0x7fad100c6b80 2023-10-30T12:53:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.019+0000 7fad0effd640 1 --2- 172.21.15.89:0/2534041816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad100c6b80 0x7fad101f40e0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fad0000d800 tx=0x7fad0000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:53:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.019+0000 7facf77fe640 1 -- 172.21.15.89:0/2534041816 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0000b6c0 con 0x7fad100c6b80 2023-10-30T12:53:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.019+0000 7facf77fe640 1 -- 172.21.15.89:0/2534041816 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fad0000b820 con 0x7fad100c6b80 2023-10-30T12:53:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.019+0000 7fad157f8640 1 -- 172.21.15.89:0/2534041816 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad101332b0 con 0x7fad100c6b80 2023-10-30T12:53:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.019+0000 7facf77fe640 1 -- 172.21.15.89:0/2534041816 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0000bbb0 con 0x7fad100c6b80 2023-10-30T12:53:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.019+0000 7fad157f8640 1 -- 172.21.15.89:0/2534041816 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad101f38e0 con 0x7fad100c6b80 2023-10-30T12:53:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.019+0000 7fad157f8640 1 -- 172.21.15.89:0/2534041816 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad1004e170 con 0x7fad100c6b80 2023-10-30T12:53:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.023+0000 7facf77fe640 1 -- 172.21.15.89:0/2534041816 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fad0000bd10 con 0x7fad100c6b80 2023-10-30T12:53:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.023+0000 7facf77fe640 1 --2- 172.21.15.89:0/2534041816 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7facfc0664a0 0x7facfc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:01.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.023+0000 7facf77fe640 1 -- 172.21.15.89:0/2534041816 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6750+0+0 (secure 0 0 0) 0x7fad00084850 con 0x7fad100c6b80 2023-10-30T12:53:01.029 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.023+0000 7fad0e7fc640 1 --2- 172.21.15.89:0/2534041816 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7facfc0664a0 0x7facfc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.023+0000 7fad0e7fc640 1 --2- 172.21.15.89:0/2534041816 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7facfc0664a0 0x7facfc068960 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fad1013ec60 tx=0x7fad04009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:53:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.027+0000 7facf77fe640 1 -- 172.21.15.89:0/2534041816 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fad000518c0 con 0x7fad100c6b80 2023-10-30T12:53:01.204 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.207+0000 7fad157f8640 1 -- 172.21.15.89:0/2534041816 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fad1013d3a0 con 0x7fad100c6b80 2023-10-30T12:53:01.207 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.207+0000 7facf77fe640 1 -- 172.21.15.89:0/2534041816 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v122) v1 ==== 161+0+0 (secure 0 0 0) 0x7fad00054ee0 con 0x7fad100c6b80 2023-10-30T12:53:01.208 INFO:teuthology.orchestra.run.smithi089.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-30T12:53:01.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.211+0000 7fad157f8640 1 -- 172.21.15.89:0/2534041816 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7facfc0664a0 msgr2=0x7facfc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:01.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.211+0000 7fad157f8640 1 --2- 172.21.15.89:0/2534041816 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7facfc0664a0 0x7facfc068960 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fad1013ec60 tx=0x7fad04009040 comp rx=0 tx=0).stop 2023-10-30T12:53:01.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.211+0000 7fad157f8640 1 -- 172.21.15.89:0/2534041816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad100c6b80 msgr2=0x7fad101f40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:01.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.211+0000 7fad157f8640 1 --2- 172.21.15.89:0/2534041816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad100c6b80 0x7fad101f40e0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fad0000d800 tx=0x7fad0000dcc0 comp rx=0 tx=0).stop 2023-10-30T12:53:01.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.211+0000 7fad157f8640 1 -- 172.21.15.89:0/2534041816 shutdown_connections 2023-10-30T12:53:01.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.211+0000 7fad157f8640 1 --2- 172.21.15.89:0/2534041816 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7facfc0664a0 0x7facfc068960 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:01.211 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.211+0000 7fad157f8640 1 --2- 172.21.15.89:0/2534041816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fad10157700 0x7fad1013e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:01.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.211+0000 7fad157f8640 1 --2- 172.21.15.89:0/2534041816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fad100c6f60 0x7fad10131420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:01.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.211+0000 7fad157f8640 1 --2- 172.21.15.89:0/2534041816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad100c6b80 0x7fad101f40e0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:01.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.211+0000 7fad157f8640 1 -- 172.21.15.89:0/2534041816 >> 172.21.15.89:0/2534041816 conn(0x7fad100c7910 msgr2=0x7fad1006a960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:53:01.212 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.211+0000 7fad157f8640 1 -- 172.21.15.89:0/2534041816 shutdown_connections 2023-10-30T12:53:01.213 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.211+0000 7fad157f8640 1 -- 172.21.15.89:0/2534041816 wait complete. 2023-10-30T12:53:01.232 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-30T12:53:01.278 INFO:tasks.ceph.mds.c.smithi089.stdout:starting mds.c at 2023-10-30T12:53:01.303 INFO:tasks.ceph.mds.b.smithi192.stdout:starting mds.b at 2023-10-30T12:53:01.305 INFO:tasks.ceph.mds.d.smithi192.stdout:starting mds.d at 2023-10-30T12:53:01.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.375+0000 7f52741f5640 1 Processor -- start 2023-10-30T12:53:01.375 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.375+0000 7f52741f5640 1 -- start start 2023-10-30T12:53:01.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.379+0000 7f52741f5640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f526c1508c0 0x7f526c0c6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:01.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.379+0000 7f52741f5640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f526c0722a0 0x7f526c0c6e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:01.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.379+0000 7f52741f5640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f526c0c0a30 con 0x7f526c0bfec0 2023-10-30T12:53:01.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.379+0000 7f52741f5640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f526c0c85b0 con 0x7f526c0722a0 2023-10-30T12:53:01.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.379+0000 7f52741f5640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f526c06f650 con 0x7f526c1508c0 2023-10-30T12:53:01.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.379+0000 7f5271790640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f526c0722a0 0x7f526c0c6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:01.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.379+0000 7f5271790640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f526c0722a0 0x7f526c0c6e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48794/0 (socket says 172.21.15.89:48794) 2023-10-30T12:53:01.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.379+0000 7f5271790640 1 -- 172.21.15.89:0/1359985165 learned_addr learned my addr 172.21.15.89:0/1359985165 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:53:01.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.379+0000 7f5272792640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f526c0bfec0 0x7f526c156930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38120/0 (socket says 172.21.15.89:38120) 2023-10-30T12:53:01.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.379+0000 7f5271f91640 1 --2- 172.21.15.89:0/1359985165 >> v2:172.21.15.192:3300/0 conn(0x7f526c1508c0 0x7f526c0c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:01.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.379+0000 7f5271790640 1 -- 172.21.15.89:0/1359985165 >> v2:172.21.15.192:3300/0 conn(0x7f526c1508c0 msgr2=0x7f526c0c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:53:01.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.379+0000 7f5271790640 1 --2- 172.21.15.89:0/1359985165 >> v2:172.21.15.192:3300/0 conn(0x7f526c1508c0 0x7f526c0c6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:01.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.379+0000 7f5271790640 1 -- 172.21.15.89:0/1359985165 >> v1:172.21.15.89:6789/0 conn(0x7f526c0bfec0 legacy=0x7f526c156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:01.381 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.379+0000 7f5271790640 1 -- 172.21.15.89:0/1359985165 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f526c06f930 con 0x7f526c0722a0 2023-10-30T12:53:01.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.383+0000 7f5271790640 1 --2- 172.21.15.89:0/1359985165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f526c0722a0 0x7f526c0c6e60 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f526c0c7540 tx=0x7f525c01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c3f63dc3a6d2247 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:53:01.382 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.383+0000 7f5270f8f640 1 -- 172.21.15.89:0/1359985165 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525c0ad660 con 0x7f526c0722a0 2023-10-30T12:53:01.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.383+0000 7f5270f8f640 1 -- 172.21.15.89:0/1359985165 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f525c0ad7c0 con 0x7f526c0722a0 2023-10-30T12:53:01.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.383+0000 7f5270f8f640 1 -- 172.21.15.89:0/1359985165 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525c0adb70 con 0x7f526c0722a0 2023-10-30T12:53:01.383 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.383+0000 7f52741f5640 1 -- 172.21.15.89:0/1359985165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f526c0722a0 msgr2=0x7f526c0c6e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:01.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.383+0000 7f52741f5640 1 --2- 172.21.15.89:0/1359985165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f526c0722a0 0x7f526c0c6e60 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f526c0c7540 tx=0x7f525c01c470 comp rx=0 tx=0).stop 2023-10-30T12:53:01.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.383+0000 7f52741f5640 1 -- 172.21.15.89:0/1359985165 shutdown_connections 2023-10-30T12:53:01.384 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.383+0000 7f52741f5640 1 --2- 172.21.15.89:0/1359985165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f526c0722a0 0x7f526c0c6e60 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:01.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.383+0000 7f52741f5640 1 --2- 172.21.15.89:0/1359985165 >> v2:172.21.15.192:3300/0 conn(0x7f526c1508c0 0x7f526c0c6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:01.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.383+0000 7f52741f5640 1 -- 172.21.15.89:0/1359985165 >> 172.21.15.89:0/1359985165 conn(0x7f526c0c7910 msgr2=0x7f526c0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:53:01.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.383+0000 7f52741f5640 1 -- 172.21.15.89:0/1359985165 shutdown_connections 2023-10-30T12:53:01.386 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.383+0000 7f52741f5640 1 -- 172.21.15.89:0/1359985165 wait complete. 2023-10-30T12:53:01.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.383+0000 7f52741f5640 1 Processor -- start 2023-10-30T12:53:01.388 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.383+0000 7f52741f5640 1 -- start start 2023-10-30T12:53:01.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.387+0000 7f52741f5640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f526c0722a0 0x7f526c14c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:01.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.387+0000 7f52741f5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f526c0bfec0 0x7f526c14e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:01.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.387+0000 7f52741f5640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f526c1508c0 0x7f526c14eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:01.389 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.387+0000 7f52741f5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f526c0caba0 con 0x7f526c0bfec0 2023-10-30T12:53:01.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.387+0000 7f52741f5640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f526c0c9f90 con 0x7f526c1508c0 2023-10-30T12:53:01.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.387+0000 7f52741f5640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f526c0ca270 con 0x7f526c0722a0 2023-10-30T12:53:01.390 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.387+0000 7f5271790640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f526c0bfec0 0x7f526c14e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:01.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.387+0000 7f5271790640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f526c0bfec0 0x7f526c14e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48808/0 (socket says 172.21.15.89:48808) 2023-10-30T12:53:01.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.387+0000 7f5271790640 1 -- 172.21.15.89:0/433396387 learned_addr learned my addr 172.21.15.89:0/433396387 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:53:01.391 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.387+0000 7f5271f91640 1 --2- 172.21.15.89:0/433396387 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f526c0722a0 0x7f526c14c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:01.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.387+0000 7f5272792640 1 --2- 172.21.15.89:0/433396387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f526c1508c0 0x7f526c14eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:01.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.387+0000 7f5271790640 1 -- 172.21.15.89:0/433396387 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f526c0722a0 msgr2=0x7f526c14c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:01.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.387+0000 7f5271790640 1 --2- 172.21.15.89:0/433396387 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f526c0722a0 0x7f526c14c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:01.392 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.387+0000 7f5271790640 1 -- 172.21.15.89:0/433396387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f526c1508c0 msgr2=0x7f526c14eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:01.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.387+0000 7f5271790640 1 --2- 172.21.15.89:0/433396387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f526c1508c0 0x7f526c14eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:01.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.387+0000 7f5271790640 1 -- 172.21.15.89:0/433396387 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f525c0094d0 con 0x7f526c0bfec0 2023-10-30T12:53:01.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.387+0000 7f5272792640 1 --2- 172.21.15.89:0/433396387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f526c1508c0 0x7f526c14eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:53:01.393 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.387+0000 7f5271790640 1 --2- 172.21.15.89:0/433396387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f526c0bfec0 0x7f526c14e5d0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f525c019080 tx=0x7f525c019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:53:01.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.387+0000 7f52627fc640 1 -- 172.21.15.89:0/433396387 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525c0b6640 con 0x7f526c0bfec0 2023-10-30T12:53:01.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.387+0000 7f52627fc640 1 -- 172.21.15.89:0/433396387 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f525c0b67a0 con 0x7f526c0bfec0 2023-10-30T12:53:01.394 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.387+0000 7f52741f5640 1 -- 172.21.15.89:0/433396387 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f526c0ca550 con 0x7f526c0bfec0 2023-10-30T12:53:01.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.387+0000 7f52627fc640 1 -- 172.21.15.89:0/433396387 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525c0b6b50 con 0x7f526c0bfec0 2023-10-30T12:53:01.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.387+0000 7f52741f5640 1 -- 172.21.15.89:0/433396387 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f526c156bb0 con 0x7f526c0bfec0 2023-10-30T12:53:01.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.387+0000 7f52741f5640 1 -- 172.21.15.89:0/433396387 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f526c04e170 con 0x7f526c0bfec0 2023-10-30T12:53:01.395 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.387+0000 7f52627fc640 1 -- 172.21.15.89:0/433396387 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f525c0b4070 con 0x7f526c0bfec0 2023-10-30T12:53:01.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.391+0000 7f52627fc640 1 --2- 172.21.15.89:0/433396387 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f52500664a0 0x7f5250068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:01.396 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.391+0000 7f52627fc640 1 -- 172.21.15.89:0/433396387 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7008+0+0 (secure 0 0 0) 0x7f525c11e7f0 con 0x7f526c0bfec0 2023-10-30T12:53:01.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.391+0000 7f5271f91640 1 --2- 172.21.15.89:0/433396387 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f52500664a0 0x7f5250068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:01.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.391+0000 7f5271f91640 1 --2- 172.21.15.89:0/433396387 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f52500664a0 0x7f5250068960 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f5268006fd0 tx=0x7f5268008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:53:01.397 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.395+0000 7f52627fc640 1 -- 172.21.15.89:0/433396387 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f525c0eb660 con 0x7f526c0bfec0 2023-10-30T12:53:01.577 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:01.579+0000 7f52741f5640 1 -- 172.21.15.89:0/433396387 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f526c0c54f0 con 0x7f526c0bfec0 2023-10-30T12:53:02.210 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.211+0000 7f52627fc640 1 -- 172.21.15.89:0/433396387 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v123) v1 ==== 104+0+0 (secure 0 0 0) 0x7f525c0b6cb0 con 0x7f526c0bfec0 2023-10-30T12:53:02.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.215+0000 7f52741f5640 1 -- 172.21.15.89:0/433396387 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f52500664a0 msgr2=0x7f5250068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:02.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.215+0000 7f52741f5640 1 --2- 172.21.15.89:0/433396387 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f52500664a0 0x7f5250068960 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f5268006fd0 tx=0x7f5268008040 comp rx=0 tx=0).stop 2023-10-30T12:53:02.214 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.215+0000 7f52741f5640 1 -- 172.21.15.89:0/433396387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f526c0bfec0 msgr2=0x7f526c14e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:02.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.215+0000 7f52741f5640 1 --2- 172.21.15.89:0/433396387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f526c0bfec0 0x7f526c14e5d0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f525c019080 tx=0x7f525c019210 comp rx=0 tx=0).stop 2023-10-30T12:53:02.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.215+0000 7f52741f5640 1 -- 172.21.15.89:0/433396387 shutdown_connections 2023-10-30T12:53:02.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.215+0000 7f52741f5640 1 --2- 172.21.15.89:0/433396387 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f52500664a0 0x7f5250068960 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:02.215 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.215+0000 7f52741f5640 1 --2- 172.21.15.89:0/433396387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f526c1508c0 0x7f526c14eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:02.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.215+0000 7f52741f5640 1 --2- 172.21.15.89:0/433396387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f526c0bfec0 0x7f526c14e5d0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:02.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.215+0000 7f52741f5640 1 --2- 172.21.15.89:0/433396387 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f526c0722a0 0x7f526c14c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:02.216 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.215+0000 7f52741f5640 1 -- 172.21.15.89:0/433396387 >> 172.21.15.89:0/433396387 conn(0x7f526c0c7910 msgr2=0x7f526c0c1180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:53:02.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.215+0000 7f52741f5640 1 -- 172.21.15.89:0/433396387 shutdown_connections 2023-10-30T12:53:02.217 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.215+0000 7f52741f5640 1 -- 172.21.15.89:0/433396387 wait complete. 2023-10-30T12:53:02.234 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-30T12:53:02.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.419+0000 7f1311fa7640 1 Processor -- start 2023-10-30T12:53:02.418 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.419+0000 7f1311fa7640 1 -- start start 2023-10-30T12:53:02.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.419+0000 7f1311fa7640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f130c1538f0 0x7f130c159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:02.421 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.419+0000 7f1311fa7640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f130c0ca6a0 con 0x7f130c15cd50 2023-10-30T12:53:02.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.419+0000 7f1311fa7640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f130c15d300 con 0x7f130c0cefa0 2023-10-30T12:53:02.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.419+0000 7f1311fa7640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f130c15d5e0 con 0x7f130c1538f0 2023-10-30T12:53:02.422 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.419+0000 7f130bfff640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f130c15cd50 0x7f130c0c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:38128/0 (socket says 172.21.15.89:38128) 2023-10-30T12:53:02.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.419+0000 7f130bfff640 1 -- 172.21.15.89:0/542056319 learned_addr learned my addr 172.21.15.89:0/542056319 (peer_addr_for_me v1:172.21.15.89:0/0) 2023-10-30T12:53:02.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.423+0000 7f1310fa5640 1 --2- 172.21.15.89:0/542056319 >> v2:172.21.15.192:3300/0 conn(0x7f130c1538f0 0x7f130c159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:02.423 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.423+0000 7f130b7fe640 1 -- 172.21.15.89:0/542056319 <== mon.0 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 437572048 0 0) 0x7f130c0ca6a0 con 0x7f130c15cd50 2023-10-30T12:53:02.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.423+0000 7f130b7fe640 1 -- 172.21.15.89:0/542056319 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12fc003530 con 0x7f130c15cd50 2023-10-30T12:53:02.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.423+0000 7f130b7fe640 1 -- 172.21.15.89:0/542056319 <== mon.0 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3598432584 0 0) 0x7f12fc003530 con 0x7f130c15cd50 2023-10-30T12:53:02.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.423+0000 7f130b7fe640 1 -- 172.21.15.89:0/542056319 >> v2:172.21.15.192:3300/0 conn(0x7f130c1538f0 msgr2=0x7f130c159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:53:02.424 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.423+0000 7f130b7fe640 1 --2- 172.21.15.89:0/542056319 >> v2:172.21.15.192:3300/0 conn(0x7f130c1538f0 0x7f130c159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:02.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.423+0000 7f130b7fe640 1 -- 172.21.15.89:0/542056319 >> v1:172.21.15.192:6789/0 conn(0x7f130c0cefa0 legacy=0x7f130c1640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:02.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.423+0000 7f130b7fe640 1 -- 172.21.15.89:0/542056319 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130c15d8c0 con 0x7f130c15cd50 2023-10-30T12:53:02.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.423+0000 7f130b7fe640 1 -- 172.21.15.89:0/542056319 <== mon.0 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4083464149 0 0) 0x7f12f8001680 con 0x7f130c15cd50 2023-10-30T12:53:02.425 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.423+0000 7f130b7fe640 1 -- 172.21.15.89:0/542056319 >> v1:172.21.15.89:6789/0 conn(0x7f130c15cd50 legacy=0x7f130c0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:02.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.423+0000 7f130b7fe640 1 --2- 172.21.15.89:0/542056319 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12fc003980 0x7f12fc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:02.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.423+0000 7f130b7fe640 1 --2- 172.21.15.89:0/542056319 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12fc0047d0 0x7f12fc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:02.426 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.423+0000 7f130b7fe640 1 --2- 172.21.15.89:0/542056319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12fc005190 0x7f12fc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:02.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.423+0000 7f130b7fe640 1 -- 172.21.15.89:0/542056319 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130c15d5e0 con 0x7f12fc005190 2023-10-30T12:53:02.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.423+0000 7f130b7fe640 1 -- 172.21.15.89:0/542056319 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12fc00c040 con 0x7f12fc0047d0 2023-10-30T12:53:02.427 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.423+0000 7f130b7fe640 1 -- 172.21.15.89:0/542056319 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12fc00c200 con 0x7f12fc003980 2023-10-30T12:53:02.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.423+0000 7f13117a6640 1 --2- 172.21.15.89:0/542056319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12fc005190 0x7f12fc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:02.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.423+0000 7f1310fa5640 1 --2- 172.21.15.89:0/542056319 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12fc003980 0x7f12fc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:02.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.423+0000 7f130bfff640 1 --2- 172.21.15.89:0/542056319 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12fc0047d0 0x7f12fc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:02.428 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.423+0000 7f13117a6640 1 -- 172.21.15.89:0/542056319 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12fc003980 msgr2=0x7f12fc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:53:02.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.423+0000 7f13117a6640 1 --2- 172.21.15.89:0/542056319 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12fc003980 0x7f12fc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:02.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.423+0000 7f13117a6640 1 -- 172.21.15.89:0/542056319 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12fc0047d0 msgr2=0x7f12fc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:53:02.429 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.423+0000 7f13117a6640 1 --2- 172.21.15.89:0/542056319 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12fc0047d0 0x7f12fc004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:02.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.423+0000 7f13117a6640 1 -- 172.21.15.89:0/542056319 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12fc003530 con 0x7f12fc005190 2023-10-30T12:53:02.430 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.423+0000 7f1310fa5640 1 --2- 172.21.15.89:0/542056319 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12fc003980 0x7f12fc003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:53:02.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.427+0000 7f13117a6640 1 -- 172.21.15.89:0/542056319 reap_dead start 2023-10-30T12:53:02.433 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.427+0000 7f13117a6640 1 --2- 172.21.15.89:0/542056319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12fc005190 0x7f12fc00b990 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f1304001e70 tx=0x7f130401c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ee6810b02aaa196 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:53:02.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.427+0000 7f130b7fe640 1 -- 172.21.15.89:0/542056319 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1304003b80 con 0x7f12fc005190 2023-10-30T12:53:02.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.427+0000 7f130b7fe640 1 -- 172.21.15.89:0/542056319 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f1304003d20 con 0x7f12fc005190 2023-10-30T12:53:02.434 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.427+0000 7f130b7fe640 1 -- 172.21.15.89:0/542056319 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1304005280 con 0x7f12fc005190 2023-10-30T12:53:02.435 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.427+0000 7f1311fa7640 1 -- 172.21.15.89:0/542056319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12fc005190 msgr2=0x7f12fc00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:02.435 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.427+0000 7f1311fa7640 1 --2- 172.21.15.89:0/542056319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12fc005190 0x7f12fc00b990 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f1304001e70 tx=0x7f130401c760 comp rx=0 tx=0).stop 2023-10-30T12:53:02.435 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.427+0000 7f1311fa7640 1 -- 172.21.15.89:0/542056319 shutdown_connections 2023-10-30T12:53:02.436 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.427+0000 7f1311fa7640 1 --2- 172.21.15.89:0/542056319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12fc005190 0x7f12fc00b990 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:02.436 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.427+0000 7f1311fa7640 1 -- 172.21.15.89:0/542056319 >> 172.21.15.89:0/542056319 conn(0x7f130c06a640 msgr2=0x7f130c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:53:02.436 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.427+0000 7f1311fa7640 1 -- 172.21.15.89:0/542056319 shutdown_connections 2023-10-30T12:53:02.436 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.427+0000 7f1311fa7640 1 -- 172.21.15.89:0/542056319 wait complete. 2023-10-30T12:53:02.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.427+0000 7f1311fa7640 1 Processor -- start 2023-10-30T12:53:02.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.427+0000 7f1311fa7640 1 -- start start 2023-10-30T12:53:02.437 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.431+0000 7f1311fa7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12fc005190 0x7f130c0b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:02.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.431+0000 7f1311fa7640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f130c1f31e0 0x7f130c1ef060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:02.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.431+0000 7f1311fa7640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f130c13c650 0x7f130c141be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:02.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.431+0000 7f1311fa7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130c1f2d50 con 0x7f12fc005190 2023-10-30T12:53:02.438 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.431+0000 7f1311fa7640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130c15ca00 con 0x7f130c1f31e0 2023-10-30T12:53:02.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.431+0000 7f1311fa7640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f130c15c4b0 con 0x7f130c13c650 2023-10-30T12:53:02.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.431+0000 7f1310fa5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12fc005190 0x7f130c0b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:02.439 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.431+0000 7f1310fa5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12fc005190 0x7f130c0b39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48830/0 (socket says 172.21.15.89:48830) 2023-10-30T12:53:02.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.431+0000 7f1310fa5640 1 -- 172.21.15.89:0/1643685741 learned_addr learned my addr 172.21.15.89:0/1643685741 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:53:02.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.431+0000 7f130bfff640 1 --2- 172.21.15.89:0/1643685741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f130c1f31e0 0x7f130c1ef060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:02.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.431+0000 7f13117a6640 1 --2- 172.21.15.89:0/1643685741 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f130c13c650 0x7f130c141be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:02.440 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.431+0000 7f1310fa5640 1 -- 172.21.15.89:0/1643685741 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f130c13c650 msgr2=0x7f130c141be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:02.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.431+0000 7f1310fa5640 1 --2- 172.21.15.89:0/1643685741 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f130c13c650 0x7f130c141be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:02.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.431+0000 7f1310fa5640 1 -- 172.21.15.89:0/1643685741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f130c1f31e0 msgr2=0x7f130c1ef060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:02.441 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.431+0000 7f1310fa5640 1 --2- 172.21.15.89:0/1643685741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f130c1f31e0 0x7f130c1ef060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:02.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.431+0000 7f1310fa5640 1 -- 172.21.15.89:0/1643685741 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130c15c720 con 0x7f12fc005190 2023-10-30T12:53:02.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.431+0000 7f1310fa5640 1 --2- 172.21.15.89:0/1643685741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12fc005190 0x7f130c0b39d0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f1300009160 tx=0x7f13000119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:53:02.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.431+0000 7f13097fa640 1 -- 172.21.15.89:0/1643685741 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1300004220 con 0x7f12fc005190 2023-10-30T12:53:02.442 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.431+0000 7f13097fa640 1 -- 172.21.15.89:0/1643685741 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f1300004380 con 0x7f12fc005190 2023-10-30T12:53:02.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.431+0000 7f1311fa7640 1 -- 172.21.15.89:0/1643685741 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f130c1f2700 con 0x7f12fc005190 2023-10-30T12:53:02.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.431+0000 7f13097fa640 1 -- 172.21.15.89:0/1643685741 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13000158b0 con 0x7f12fc005190 2023-10-30T12:53:02.443 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.431+0000 7f1311fa7640 1 -- 172.21.15.89:0/1643685741 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f130c1ef5a0 con 0x7f12fc005190 2023-10-30T12:53:02.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.431+0000 7f1311fa7640 1 -- 172.21.15.89:0/1643685741 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f130c04e170 con 0x7f12fc005190 2023-10-30T12:53:02.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.431+0000 7f13097fa640 1 -- 172.21.15.89:0/1643685741 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1300015a10 con 0x7f12fc005190 2023-10-30T12:53:02.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.435+0000 7f13097fa640 1 --2- 172.21.15.89:0/1643685741 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f12f00664d0 0x7f12f0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:02.444 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.435+0000 7f13097fa640 1 -- 172.21.15.89:0/1643685741 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7008+0+0 (secure 0 0 0) 0x7f1300084740 con 0x7f12fc005190 2023-10-30T12:53:02.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.435+0000 7f130bfff640 1 --2- 172.21.15.89:0/1643685741 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f12f00664d0 0x7f12f0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:02.445 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.435+0000 7f130bfff640 1 --2- 172.21.15.89:0/1643685741 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f12f00664d0 0x7f12f0068990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f130c146140 tx=0x7f12f8000ee0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:53:02.446 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.439+0000 7f13097fa640 1 -- 172.21.15.89:0/1643685741 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f13000516e0 con 0x7f12fc005190 2023-10-30T12:53:02.589 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:02.591+0000 7f1311fa7640 1 -- 172.21.15.89:0/1643685741 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f130c146190 con 0x7f12fc005190 2023-10-30T12:53:03.537 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.539+0000 7f13097fa640 1 -- 172.21.15.89:0/1643685741 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v68) v1 ==== 181+0+0 (secure 0 0 0) 0x7f1300054d00 con 0x7f12fc005190 2023-10-30T12:53:03.537 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' removed 2023-10-30T12:53:03.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.543+0000 7f1311fa7640 1 -- 172.21.15.89:0/1643685741 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f12f00664d0 msgr2=0x7f12f0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:03.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.543+0000 7f1311fa7640 1 --2- 172.21.15.89:0/1643685741 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f12f00664d0 0x7f12f0068990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f130c146140 tx=0x7f12f8000ee0 comp rx=0 tx=0).stop 2023-10-30T12:53:03.541 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.543+0000 7f1311fa7640 1 -- 172.21.15.89:0/1643685741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12fc005190 msgr2=0x7f130c0b39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:03.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.543+0000 7f1311fa7640 1 --2- 172.21.15.89:0/1643685741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12fc005190 0x7f130c0b39d0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f1300009160 tx=0x7f13000119a0 comp rx=0 tx=0).stop 2023-10-30T12:53:03.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.543+0000 7f1311fa7640 1 -- 172.21.15.89:0/1643685741 shutdown_connections 2023-10-30T12:53:03.542 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.543+0000 7f1311fa7640 1 --2- 172.21.15.89:0/1643685741 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f12f00664d0 0x7f12f0068990 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:03.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.543+0000 7f1311fa7640 1 --2- 172.21.15.89:0/1643685741 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f130c1f31e0 0x7f130c1ef060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:03.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.543+0000 7f1311fa7640 1 --2- 172.21.15.89:0/1643685741 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f130c13c650 0x7f130c141be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:03.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.543+0000 7f1311fa7640 1 --2- 172.21.15.89:0/1643685741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12fc005190 0x7f130c0b39d0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:03.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.543+0000 7f1311fa7640 1 -- 172.21.15.89:0/1643685741 >> 172.21.15.89:0/1643685741 conn(0x7f130c06a640 msgr2=0x7f130c0c9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:53:03.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.543+0000 7f1311fa7640 1 -- 172.21.15.89:0/1643685741 shutdown_connections 2023-10-30T12:53:03.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.543+0000 7f1311fa7640 1 -- 172.21.15.89:0/1643685741 wait complete. 2023-10-30T12:53:03.577 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-30T12:53:03.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.727+0000 7f78a4691640 1 Processor -- start 2023-10-30T12:53:03.725 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.727+0000 7f78a4691640 1 -- start start 2023-10-30T12:53:03.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.727+0000 7f78a4691640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f789c15d9d0 0x7f789c0cb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:03.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.727+0000 7f78a4691640 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f789c0c4000 0x7f789c0cf090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:03.727 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.727+0000 7f78a4691640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f789c15a210 con 0x7f789c1536d0 2023-10-30T12:53:03.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.727+0000 7f78a4691640 1 -- --> v2:172.21.15.89:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f789c15a4f0 con 0x7f789c15d9d0 2023-10-30T12:53:03.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.727+0000 7f78a4691640 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f789c15cef0 con 0x7f789c0c4000 2023-10-30T12:53:03.728 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.727+0000 7f78a1c2c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f789c15d9d0 0x7f789c0cb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:03.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.727+0000 7f78a1c2c640 1 --2- >> v2:172.21.15.89:3300/0 conn(0x7f789c15d9d0 0x7f789c0cb3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48834/0 (socket says 172.21.15.89:48834) 2023-10-30T12:53:03.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.727+0000 7f78a1c2c640 1 -- 172.21.15.89:0/2046486135 learned_addr learned my addr 172.21.15.89:0/2046486135 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:53:03.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.727+0000 7f78a2c2e640 1 --2- 172.21.15.89:0/2046486135 >> v2:172.21.15.192:3300/0 conn(0x7f789c0c4000 0x7f789c0cf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:03.729 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.727+0000 7f78a1c2c640 1 -- 172.21.15.89:0/2046486135 >> v2:172.21.15.192:3300/0 conn(0x7f789c0c4000 msgr2=0x7f789c0cf090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:53:03.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.727+0000 7f78a1c2c640 1 --2- 172.21.15.89:0/2046486135 >> v2:172.21.15.192:3300/0 conn(0x7f789c0c4000 0x7f789c0cf090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:03.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.727+0000 7f78a1c2c640 1 -- 172.21.15.89:0/2046486135 >> v1:172.21.15.192:6789/0 conn(0x7f789c1536d0 legacy=0x7f789c159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:53:03.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.727+0000 7f78a1c2c640 1 -- 172.21.15.89:0/2046486135 --> v2:172.21.15.89:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f789c15d1d0 con 0x7f789c15d9d0 2023-10-30T12:53:03.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.731+0000 7f78a1c2c640 1 --2- 172.21.15.89:0/2046486135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f789c15d9d0 0x7f789c0cb3c0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f789c0d01b0 tx=0x7f788c01c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5477ac2627b09239 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:53:03.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.731+0000 7f78a142b640 1 -- 172.21.15.89:0/2046486135 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c0ab600 con 0x7f789c15d9d0 2023-10-30T12:53:03.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.731+0000 7f78a142b640 1 -- 172.21.15.89:0/2046486135 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f788c0a7050 con 0x7f789c15d9d0 2023-10-30T12:53:03.732 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.731+0000 7f78a142b640 1 -- 172.21.15.89:0/2046486135 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c0ab9a0 con 0x7f789c15d9d0 2023-10-30T12:53:03.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.731+0000 7f78a4691640 1 -- 172.21.15.89:0/2046486135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f789c15d9d0 msgr2=0x7f789c0cb3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:03.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.731+0000 7f78a4691640 1 --2- 172.21.15.89:0/2046486135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f789c15d9d0 0x7f789c0cb3c0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f789c0d01b0 tx=0x7f788c01c4c0 comp rx=0 tx=0).stop 2023-10-30T12:53:03.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.731+0000 7f78a4691640 1 -- 172.21.15.89:0/2046486135 shutdown_connections 2023-10-30T12:53:03.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.731+0000 7f78a4691640 1 --2- 172.21.15.89:0/2046486135 >> v2:172.21.15.192:3300/0 conn(0x7f789c0c4000 0x7f789c0cf090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:03.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.731+0000 7f78a4691640 1 --2- 172.21.15.89:0/2046486135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f789c15d9d0 0x7f789c0cb3c0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:03.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.731+0000 7f78a4691640 1 -- 172.21.15.89:0/2046486135 >> 172.21.15.89:0/2046486135 conn(0x7f789c06bf70 msgr2=0x7f789c06c380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:53:03.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.731+0000 7f78a4691640 1 -- 172.21.15.89:0/2046486135 shutdown_connections 2023-10-30T12:53:03.735 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.731+0000 7f78a4691640 1 -- 172.21.15.89:0/2046486135 wait complete. 2023-10-30T12:53:03.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.735+0000 7f78a4691640 1 Processor -- start 2023-10-30T12:53:03.737 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.735+0000 7f78a4691640 1 -- start start 2023-10-30T12:53:03.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.735+0000 7f78a4691640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f789c0c4000 0x7f789c0b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:03.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.735+0000 7f78a4691640 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f789c1536d0 0x7f789c14b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:03.738 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.735+0000 7f78a4691640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f789c15d9d0 0x7f789c14bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:03.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.735+0000 7f78a4691640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789c159c60 con 0x7f789c15d9d0 2023-10-30T12:53:03.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.735+0000 7f78a4691640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789c14c170 con 0x7f789c0c4000 2023-10-30T12:53:03.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.735+0000 7f78a4691640 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f789c158340 con 0x7f789c1536d0 2023-10-30T12:53:03.739 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.735+0000 7f78a2c2e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f789c15d9d0 0x7f789c14bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:03.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.735+0000 7f78a2c2e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f789c15d9d0 0x7f789c14bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48850/0 (socket says 172.21.15.89:48850) 2023-10-30T12:53:03.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.735+0000 7f78a2c2e640 1 -- 172.21.15.89:0/1869838349 learned_addr learned my addr 172.21.15.89:0/1869838349 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-10-30T12:53:03.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.735+0000 7f78a242d640 1 --2- 172.21.15.89:0/1869838349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f789c0c4000 0x7f789c0b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:03.740 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.735+0000 7f78a1c2c640 1 --2- 172.21.15.89:0/1869838349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f789c1536d0 0x7f789c14b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:03.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.735+0000 7f78a2c2e640 1 -- 172.21.15.89:0/1869838349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f789c1536d0 msgr2=0x7f789c14b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:03.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.735+0000 7f78a2c2e640 1 --2- 172.21.15.89:0/1869838349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f789c1536d0 0x7f789c14b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:03.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.735+0000 7f78a2c2e640 1 -- 172.21.15.89:0/1869838349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f789c0c4000 msgr2=0x7f789c0b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:03.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.735+0000 7f78a2c2e640 1 --2- 172.21.15.89:0/1869838349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f789c0c4000 0x7f789c0b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:03.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.735+0000 7f78a2c2e640 1 -- 172.21.15.89:0/1869838349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788c0094d0 con 0x7f789c15d9d0 2023-10-30T12:53:03.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.735+0000 7f78a2c2e640 1 --2- 172.21.15.89:0/1869838349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f789c15d9d0 0x7f789c14bc30 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f789400cda0 tx=0x7f789400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:53:03.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.735+0000 7f7892ffd640 1 -- 172.21.15.89:0/1869838349 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7894003ba0 con 0x7f789c15d9d0 2023-10-30T12:53:03.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.735+0000 7f7892ffd640 1 -- 172.21.15.89:0/1869838349 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f7894003d00 con 0x7f789c15d9d0 2023-10-30T12:53:03.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.735+0000 7f7892ffd640 1 -- 172.21.15.89:0/1869838349 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7894004260 con 0x7f789c15d9d0 2023-10-30T12:53:03.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.735+0000 7f78a4691640 1 -- 172.21.15.89:0/1869838349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f789c14b260 con 0x7f789c15d9d0 2023-10-30T12:53:03.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.735+0000 7f78a4691640 1 -- 172.21.15.89:0/1869838349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f789c0c58f0 con 0x7f789c15d9d0 2023-10-30T12:53:03.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.739+0000 7f78a4691640 1 -- 172.21.15.89:0/1869838349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f789c14b3f0 con 0x7f789c15d9d0 2023-10-30T12:53:03.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.739+0000 7f7892ffd640 1 -- 172.21.15.89:0/1869838349 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f78940043c0 con 0x7f789c15d9d0 2023-10-30T12:53:03.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.739+0000 7f7892ffd640 1 --2- 172.21.15.89:0/1869838349 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f78800664a0 0x7f7880068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:03.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.739+0000 7f7892ffd640 1 -- 172.21.15.89:0/1869838349 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6570+0+0 (secure 0 0 0) 0x7f7894012070 con 0x7f789c15d9d0 2023-10-30T12:53:03.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.739+0000 7f78a242d640 1 --2- 172.21.15.89:0/1869838349 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f78800664a0 0x7f7880068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:03.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.739+0000 7f78a242d640 1 --2- 172.21.15.89:0/1869838349 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f78800664a0 0x7f7880068960 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f7898004e30 tx=0x7f7898009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:53:03.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.743+0000 7f7892ffd640 1 -- 172.21.15.89:0/1869838349 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7894051cf0 con 0x7f789c15d9d0 2023-10-30T12:53:03.895 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:03.895+0000 7f78a4691640 1 -- 172.21.15.89:0/1869838349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f789c04e170 con 0x7f789c15d9d0 2023-10-30T12:53:04.539 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:04.539+0000 7f7892ffd640 1 -- 172.21.15.89:0/1869838349 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v69) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7894053c20 con 0x7f789c15d9d0 2023-10-30T12:53:04.540 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' removed 2023-10-30T12:53:04.543 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:04.543+0000 7f78a4691640 1 -- 172.21.15.89:0/1869838349 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f78800664a0 msgr2=0x7f7880068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:04.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:04.543+0000 7f78a4691640 1 --2- 172.21.15.89:0/1869838349 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f78800664a0 0x7f7880068960 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f7898004e30 tx=0x7f7898009040 comp rx=0 tx=0).stop 2023-10-30T12:53:04.544 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:04.543+0000 7f78a4691640 1 -- 172.21.15.89:0/1869838349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f789c15d9d0 msgr2=0x7f789c14bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:04.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:04.543+0000 7f78a4691640 1 --2- 172.21.15.89:0/1869838349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f789c15d9d0 0x7f789c14bc30 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f789400cda0 tx=0x7f789400a660 comp rx=0 tx=0).stop 2023-10-30T12:53:04.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:04.543+0000 7f78a4691640 1 -- 172.21.15.89:0/1869838349 shutdown_connections 2023-10-30T12:53:04.545 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:04.543+0000 7f78a4691640 1 --2- 172.21.15.89:0/1869838349 >> [v2:172.21.15.192:6832/4076276638,v1:172.21.15.192:6833/4076276638] conn(0x7f78800664a0 0x7f7880068960 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:04.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:04.543+0000 7f78a4691640 1 --2- 172.21.15.89:0/1869838349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f789c15d9d0 0x7f789c14bc30 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:04.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:04.543+0000 7f78a4691640 1 --2- 172.21.15.89:0/1869838349 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f789c1536d0 0x7f789c14b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:04.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:04.543+0000 7f78a4691640 1 --2- 172.21.15.89:0/1869838349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f789c0c4000 0x7f789c0b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:04.546 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:04.543+0000 7f78a4691640 1 -- 172.21.15.89:0/1869838349 >> 172.21.15.89:0/1869838349 conn(0x7f789c06bf70 msgr2=0x7f789c0cc230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:53:04.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:04.543+0000 7f78a4691640 1 -- 172.21.15.89:0/1869838349 shutdown_connections 2023-10-30T12:53:04.547 INFO:teuthology.orchestra.run.smithi089.stderr:2023-10-30T12:53:04.543+0000 7f78a4691640 1 -- 172.21.15.89:0/1869838349 wait complete. 2023-10-30T12:53:04.573 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-30T12:53:04.574 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T12:53:04.574 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:53:04.587 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T12:53:04.588 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T12:53:04.588 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:53:04.644 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T12:53:04.645 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T12:53:04.646 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:53:04.703 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T12:53:04.704 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-30T12:53:04.704 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:53:04.714 INFO:tasks.ceph.mds.b:Stopped 2023-10-30T12:53:04.714 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-30T12:53:04.715 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:53:04.767 INFO:tasks.ceph.mds.d:Stopped 2023-10-30T12:53:04.768 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-30T12:53:04.768 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:53:04.824 INFO:tasks.ceph.mds.f:Stopped 2023-10-30T12:53:04.825 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-30T12:53:04.825 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-30T12:53:04.825 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:53:04.910 INFO:tasks.ceph.osd.0:Stopped 2023-10-30T12:53:04.911 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-30T12:53:04.911 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:53:09.884 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-10-30T12:53:11.013 INFO:tasks.ceph.osd.1:Stopped 2023-10-30T12:53:11.014 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-30T12:53:11.015 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:53:11.104 INFO:tasks.ceph.osd.2:Stopped 2023-10-30T12:53:11.105 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-10-30T12:53:11.105 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:53:11.206 INFO:tasks.ceph.osd.3:Stopped 2023-10-30T12:53:11.207 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-10-30T12:53:11.207 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:53:11.306 INFO:tasks.ceph.osd.4:Stopped 2023-10-30T12:53:11.306 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-10-30T12:53:11.307 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:53:11.398 INFO:tasks.ceph.osd.5:Stopped 2023-10-30T12:53:11.399 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-10-30T12:53:11.399 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:53:11.489 INFO:tasks.ceph.osd.6:Stopped 2023-10-30T12:53:11.490 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-10-30T12:53:11.490 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:53:15.388 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2023-10-30T12:53:15.389 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-10-30T12:53:17.592 INFO:tasks.ceph.osd.7:Stopped 2023-10-30T12:53:17.593 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-30T12:53:17.594 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-10-30T12:53:17.594 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:53:17.646 INFO:tasks.ceph.mgr.y:Stopped 2023-10-30T12:53:17.646 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-30T12:53:17.647 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:53:17.676 INFO:tasks.ceph.mgr.x:Stopped 2023-10-30T12:53:17.677 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-30T12:53:17.677 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-30T12:53:17.677 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:53:17.699 INFO:tasks.ceph.mon.a:Stopped 2023-10-30T12:53:17.699 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-30T12:53:17.700 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:53:17.734 INFO:tasks.ceph.mon.b:Stopped 2023-10-30T12:53:17.735 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-30T12:53:17.735 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:53:17.791 INFO:tasks.ceph.mon.c:Stopped 2023-10-30T12:53:17.792 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-30T12:53:17.793 DEBUG:teuthology.orchestra.run.smithi089:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v 'Replacing daemon mds' | head -n 1 2023-10-30T12:53:17.832 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi089.front.sepia.ceph.com 2023-10-30T12:53:17.833 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-30T12:53:18.205 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi089.front.sepia.ceph.com 2023-10-30T12:53:18.206 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-30T12:53:18.418 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi089.front.sepia.ceph.com 2023-10-30T12:53:18.419 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-30T12:53:18.639 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi089.front.sepia.ceph.com 2023-10-30T12:53:18.640 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-10-30T12:53:18.851 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi192.front.sepia.ceph.com 2023-10-30T12:53:18.852 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-10-30T12:53:19.101 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi192.front.sepia.ceph.com 2023-10-30T12:53:19.103 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-10-30T12:53:19.150 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:53:19.155 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:53:19.352 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi192.front.sepia.ceph.com 2023-10-30T12:53:19.352 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-10-30T12:53:19.634 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi192.front.sepia.ceph.com 2023-10-30T12:53:19.635 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-10-30T12:53:19.866 INFO:tasks.ceph:Archiving mon data... 2023-10-30T12:53:19.867 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441018/data/mon.a.tgz 2023-10-30T12:53:19.868 DEBUG:teuthology.orchestra.run.smithi089:> mktemp 2023-10-30T12:53:19.879 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/tmp.GdSbax0AlS 2023-10-30T12:53:19.880 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.GdSbax0AlS 2023-10-30T12:53:20.020 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0666 /tmp/tmp.GdSbax0AlS 2023-10-30T12:53:20.096 DEBUG:teuthology.orchestra.remote:smithi089:/tmp/tmp.GdSbax0AlS is 390KB 2023-10-30T12:53:20.162 DEBUG:teuthology.orchestra.run.smithi089:> rm -fr /tmp/tmp.GdSbax0AlS 2023-10-30T12:53:20.171 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441018/data/mon.b.tgz 2023-10-30T12:53:20.171 DEBUG:teuthology.orchestra.run.smithi192:> mktemp 2023-10-30T12:53:20.177 INFO:teuthology.orchestra.run.smithi192.stdout:/tmp/tmp.Ry2OpJM4Lx 2023-10-30T12:53:20.178 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.Ry2OpJM4Lx 2023-10-30T12:53:20.325 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0666 /tmp/tmp.Ry2OpJM4Lx 2023-10-30T12:53:20.389 DEBUG:teuthology.orchestra.remote:smithi192:/tmp/tmp.Ry2OpJM4Lx is 400KB 2023-10-30T12:53:20.390 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2023-10-30T12:53:20.452 DEBUG:teuthology.orchestra.run.smithi192:> rm -fr /tmp/tmp.Ry2OpJM4Lx 2023-10-30T12:53:20.458 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441018/data/mon.c.tgz 2023-10-30T12:53:20.458 DEBUG:teuthology.orchestra.run.smithi192:> mktemp 2023-10-30T12:53:20.507 INFO:teuthology.orchestra.run.smithi192.stdout:/tmp/tmp.dwV9CPTl3I 2023-10-30T12:53:20.508 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.dwV9CPTl3I 2023-10-30T12:53:20.644 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0666 /tmp/tmp.dwV9CPTl3I 2023-10-30T12:53:20.706 DEBUG:teuthology.orchestra.remote:smithi192:/tmp/tmp.dwV9CPTl3I is 400KB 2023-10-30T12:53:20.724 DEBUG:teuthology.orchestra.run.smithi192:> rm -fr /tmp/tmp.dwV9CPTl3I 2023-10-30T12:53:20.730 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-30T12:53:20.730 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-30T12:53:20.734 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-30T12:53:20.786 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-30T12:53:20.786 DEBUG:teuthology.orchestra.run.smithi089:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-30T12:53:20.798 DEBUG:teuthology.orchestra.run.smithi192:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-30T12:53:20.834 INFO:teuthology.orchestra.run.smithi089.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-30T12:53:20.848 INFO:teuthology.orchestra.run.smithi192.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-30T12:53:20.852 INFO:tasks.ceph:Archiving crash dumps... 2023-10-30T12:53:20.854 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441018/remote/smithi089/crash 2023-10-30T12:53:20.856 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-30T12:53:20.908 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441018/remote/smithi192/crash 2023-10-30T12:53:20.909 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-30T12:53:20.920 INFO:tasks.ceph:Shutting down logrotate 2023-10-30T12:53:20.921 DEBUG:teuthology.orchestra.run.smithi089:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:53:20.958 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:53:21.010 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-30T12:53:21.030 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-30T12:53:21.065 INFO:tasks.ceph:Compressing logs... 2023-10-30T12:53:21.066 DEBUG:teuthology.orchestra.run.smithi089:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-30T12:53:21.074 DEBUG:teuthology.orchestra.run.smithi192:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-30T12:53:21.091 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52514.log 2023-10-30T12:53:21.092 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53735.log 2023-10-30T12:53:21.093 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45602.log 2023-10-30T12:53:21.093 INFO:teuthology.orchestra.run.smithi089.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52514.log.gz 2023-10-30T12:53:21.094 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51636.log 2023-10-30T12:53:21.094 INFO:teuthology.orchestra.run.smithi089.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53735.log.gz 2023-10-30T12:53:21.095 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.45602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51295.log 2023-10-30T12:53:21.095 INFO:teuthology.orchestra.run.smithi089.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.45602.log.gz 2023-10-30T12:53:21.096 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49058.log 2023-10-30T12:53:21.097 INFO:teuthology.orchestra.run.smithi089.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51636.log.gz 2023-10-30T12:53:21.097 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52592.log 2023-10-30T12:53:21.098 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51295.log.gz 2023-10-30T12:53:21.099 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.49058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44652.log 2023-10-30T12:53:21.099 INFO:teuthology.orchestra.run.smithi089.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49058.log.gz 2023-10-30T12:53:21.100 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38324.log 2023-10-30T12:53:21.100 INFO:teuthology.orchestra.run.smithi089.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52592.log.gz 2023-10-30T12:53:21.101 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45394.log 2023-10-30T12:53:21.101 INFO:teuthology.orchestra.run.smithi089.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44652.log.gz 2023-10-30T12:53:21.102 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.38324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52356.log 2023-10-30T12:53:21.102 INFO:teuthology.orchestra.run.smithi089.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38324.log.gz 2023-10-30T12:53:21.103 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.45394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45014.log 2023-10-30T12:53:21.103 INFO:teuthology.orchestra.run.smithi089.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45394.log.gz 2023-10-30T12:53:21.103 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50064.log 2023-10-30T12:53:21.104 INFO:teuthology.orchestra.run.smithi089.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52356.log.gz 2023-10-30T12:53:21.105 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.45014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52239.log 2023-10-30T12:53:21.105 INFO:teuthology.orchestra.run.smithi089.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.45014.log.gz 2023-10-30T12:53:21.105 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.50064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54343.log 2023-10-30T12:53:21.106 INFO:teuthology.orchestra.run.smithi089.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50064.log.gz 2023-10-30T12:53:21.107 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52239.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-10-30T12:53:21.107 INFO:teuthology.orchestra.run.smithi089.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52239.log.gz 2023-10-30T12:53:21.107 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38200.log 2023-10-30T12:53:21.108 INFO:teuthology.orchestra.run.smithi089.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54343.log.gz 2023-10-30T12:53:21.108 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46630.log 2023-10-30T12:53:21.109 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.38200.log: gzip -5 87.5% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.38200.log.gz /var/log/ceph/ceph-client.admin.55125.log 2023-10-30T12:53:21.109 INFO:teuthology.orchestra.run.smithi089.stderr: 2023-10-30T12:53:21.111 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.46630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47025.log 2023-10-30T12:53:21.111 INFO:teuthology.orchestra.run.smithi089.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46630.log.gz 2023-10-30T12:53:21.111 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.55125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50090.log 2023-10-30T12:53:21.112 INFO:teuthology.orchestra.run.smithi089.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55125.log.gz 2023-10-30T12:53:21.113 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.47025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55151.log 2023-10-30T12:53:21.114 INFO:teuthology.orchestra.run.smithi089.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47025.log.gz 2023-10-30T12:53:21.114 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.50090.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50090.log.gz 2023-10-30T12:53:21.114 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52303.log 2023-10-30T12:53:21.115 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.55151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43508.log 2023-10-30T12:53:21.116 INFO:teuthology.orchestra.run.smithi089.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55151.log.gz 2023-10-30T12:53:21.116 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52303.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52303.log.gz 2023-10-30T12:53:21.116 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50116.log 2023-10-30T12:53:21.117 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43508.log: gzip -5 --verbose -- 88.2% /var/log/ceph/ceph-client.admin.51478.log -- replaced with /var/log/ceph/ceph-client.admin.43508.log.gz 2023-10-30T12:53:21.118 INFO:teuthology.orchestra.run.smithi089.stderr: 2023-10-30T12:53:21.118 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.50116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52488.log 2023-10-30T12:53:21.118 INFO:teuthology.orchestra.run.smithi089.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50116.log.gz 2023-10-30T12:53:21.119 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44383.log 2023-10-30T12:53:21.120 INFO:teuthology.orchestra.run.smithi089.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51478.log.gz 2023-10-30T12:53:21.120 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.45244.log 2023-10-30T12:53:21.120 INFO:teuthology.orchestra.run.smithi089.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52488.log.gz 2023-10-30T12:53:21.121 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44383.log: 95.0% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-10-30T12:53:21.122 INFO:teuthology.orchestra.run.smithi089.stderr:gzip 88.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.44383.log.gz 2023-10-30T12:53:21.122 INFO:teuthology.orchestra.run.smithi089.stderr: --verbose -- /var/log/ceph/ceph-client.admin.54049.log 2023-10-30T12:53:21.122 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.0.45244.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.45244.log.gz 2023-10-30T12:53:21.123 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43978.log 2023-10-30T12:53:21.124 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53069.log 2023-10-30T12:53:21.124 INFO:teuthology.orchestra.run.smithi089.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54049.log.gz 2023-10-30T12:53:21.125 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44730.log 2023-10-30T12:53:21.125 INFO:teuthology.orchestra.run.smithi089.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43978.log.gz 2023-10-30T12:53:21.126 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53069.log: gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53069.log.gz -5 2023-10-30T12:53:21.126 INFO:teuthology.orchestra.run.smithi089.stderr: --verbose -- /var/log/ceph/ceph-client.admin.51584.log 2023-10-30T12:53:21.127 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50866.log 2023-10-30T12:53:21.127 INFO:teuthology.orchestra.run.smithi089.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44730.log.gz 2023-10-30T12:53:21.128 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51425.log 2023-10-30T12:53:21.128 INFO:teuthology.orchestra.run.smithi089.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51584.log.gz 2023-10-30T12:53:21.129 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.50866.log: gzip -5 88.6% --verbose -- replaced with /var/log/ceph/ceph-client.admin.50866.log.gz -- 2023-10-30T12:53:21.129 INFO:teuthology.orchestra.run.smithi089.stderr: /var/log/ceph/ceph-client.admin.51085.log 2023-10-30T12:53:21.133 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51425.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-30T12:53:21.133 INFO:teuthology.orchestra.run.smithi089.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51425.log.gz 2023-10-30T12:53:21.133 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51085.log: gzip -5 --verbose 88.1% -- -- replaced with /var/log/ceph/ceph-client.admin.51085.log.gz /var/log/ceph/ceph-client.admin.49958.log 2023-10-30T12:53:21.134 INFO:teuthology.orchestra.run.smithi089.stderr: 2023-10-30T12:53:21.134 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.33239.log 2023-10-30T12:53:21.134 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.49958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51925.log 2023-10-30T12:53:21.135 INFO:teuthology.orchestra.run.smithi089.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49958.log.gz 2023-10-30T12:53:21.135 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph.tmp-client.admin.33239.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.33239.log.gz 2023-10-30T12:53:21.135 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53128.log 2023-10-30T12:53:21.137 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38745.log 2023-10-30T12:53:21.138 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-10-30T12:53:21.138 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-10-30T12:53:21.138 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-client.admin.38745.log: /var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-10-30T12:53:21.139 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38745.log.gz 2023-10-30T12:53:21.139 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38721.log 2023-10-30T12:53:21.139 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38697.log 2023-10-30T12:53:21.139 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-client.admin.38721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38817.log 2023-10-30T12:53:21.140 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-client.admin.38697.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-30T12:53:21.140 INFO:teuthology.orchestra.run.smithi192.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38721.log.gz 2023-10-30T12:53:21.140 INFO:teuthology.orchestra.run.smithi192.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38697.log.gz 2023-10-30T12:53:21.140 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2023-10-30T12:53:21.141 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-client.admin.38817.log: /var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-30T12:53:21.141 INFO:teuthology.orchestra.run.smithi192.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38817.log.gz 2023-10-30T12:53:21.141 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-10-30T12:53:21.142 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-30T12:53:21.142 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2023-10-30T12:53:21.142 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph.audit.log: 94.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-30T12:53:21.147 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38769.log 2023-10-30T12:53:21.147 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-mds.d.log: 92.9% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-30T12:53:21.148 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38649.log 2023-10-30T12:53:21.148 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-client.admin.38769.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38769.log.gz 2023-10-30T12:53:21.148 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2023-10-30T12:53:21.148 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-client.admin.38649.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38649.log.gz 2023-10-30T12:53:21.149 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-10-30T12:53:21.152 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51925.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51925.log.gz 2023-10-30T12:53:21.152 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46385.log 2023-10-30T12:53:21.152 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50629.log 2023-10-30T12:53:21.152 INFO:teuthology.orchestra.run.smithi089.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53128.log.gz 2023-10-30T12:53:21.153 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.46385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52029.log 2023-10-30T12:53:21.153 INFO:teuthology.orchestra.run.smithi089.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46385.log.gz 2023-10-30T12:53:21.153 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.50629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46922.log 2023-10-30T12:53:21.154 INFO:teuthology.orchestra.run.smithi089.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50629.log.gz 2023-10-30T12:53:21.154 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44678.log 2023-10-30T12:53:21.154 INFO:teuthology.orchestra.run.smithi089.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52029.log.gz 2023-10-30T12:53:21.154 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.46922.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46922.log.gz 2023-10-30T12:53:21.155 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54501.log 2023-10-30T12:53:21.155 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53499.log 2023-10-30T12:53:21.155 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44678.log.gz 2023-10-30T12:53:21.155 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54501.log: 88.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.54501.log.gz 2023-10-30T12:53:21.156 INFO:teuthology.orchestra.run.smithi089.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47638.log 2023-10-30T12:53:21.156 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54657.log 2023-10-30T12:53:21.156 INFO:teuthology.orchestra.run.smithi089.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53499.log.gz 2023-10-30T12:53:21.157 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.47638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54553.log 2023-10-30T12:53:21.157 INFO:teuthology.orchestra.run.smithi089.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47638.log.gz 2023-10-30T12:53:21.157 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45368.log 2023-10-30T12:53:21.157 INFO:teuthology.orchestra.run.smithi089.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54657.log.gz 2023-10-30T12:53:21.158 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44574.log 2023-10-30T12:53:21.158 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54553.log.gz 2023-10-30T12:53:21.158 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.45368.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45368.log.gz 2023-10-30T12:53:21.159 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52003.log 2023-10-30T12:53:21.159 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54134.log 2023-10-30T12:53:21.159 INFO:teuthology.orchestra.run.smithi089.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44574.log.gz 2023-10-30T12:53:21.159 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52267.log 2023-10-30T12:53:21.160 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52003.log.gz 2023-10-30T12:53:21.160 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50681.log 2023-10-30T12:53:21.160 INFO:teuthology.orchestra.run.smithi089.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54134.log.gz 2023-10-30T12:53:21.160 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52267.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52267.log.gz 2023-10-30T12:53:21.161 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55464.log 2023-10-30T12:53:21.161 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.50681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45446.log 2023-10-30T12:53:21.161 INFO:teuthology.orchestra.run.smithi089.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50681.log.gz 2023-10-30T12:53:21.162 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.55464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43244.log 2023-10-30T12:53:21.162 INFO:teuthology.orchestra.run.smithi089.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55464.log.gz 2023-10-30T12:53:21.163 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38673.log 2023-10-30T12:53:21.168 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.45446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55800.log 2023-10-30T12:53:21.169 INFO:teuthology.orchestra.run.smithi089.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45446.log.gz 2023-10-30T12:53:21.169 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43244.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43244.log.gz 2023-10-30T12:53:21.169 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49295.log 2023-10-30T12:53:21.170 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.55800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46868.log 2023-10-30T12:53:21.170 INFO:teuthology.orchestra.run.smithi089.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55800.log.gz 2023-10-30T12:53:21.170 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.49295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54238.log 2023-10-30T12:53:21.170 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49295.log.gz 2023-10-30T12:53:21.171 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.46868.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46868.log.gz 2023-10-30T12:53:21.171 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55229.log 2023-10-30T12:53:21.171 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47586.log 2023-10-30T12:53:21.172 INFO:teuthology.orchestra.run.smithi089.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54238.log.gz 2023-10-30T12:53:21.172 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.55229.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55229.log.gz 2023-10-30T12:53:21.172 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45472.log 2023-10-30T12:53:21.172 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.47586.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47586.log.gz 2023-10-30T12:53:21.173 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52434.log 2023-10-30T12:53:21.173 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.45472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43963.log 2023-10-30T12:53:21.173 INFO:teuthology.orchestra.run.smithi089.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45472.log.gz 2023-10-30T12:53:21.174 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52434.log: gzip -5 --verbose -- 88.8% /var/log/ceph/ceph-client.admin.54795.log -- replaced with /var/log/ceph/ceph-client.admin.52434.log.gz 2023-10-30T12:53:21.174 INFO:teuthology.orchestra.run.smithi089.stderr: 2023-10-30T12:53:21.174 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43764.log 2023-10-30T12:53:21.174 INFO:teuthology.orchestra.run.smithi089.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43963.log.gz 2023-10-30T12:53:21.175 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54795.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54795.log.gz 2023-10-30T12:53:21.175 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55412.log 2023-10-30T12:53:21.175 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52213.log 2023-10-30T12:53:21.176 INFO:teuthology.orchestra.run.smithi089.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43764.log.gz 2023-10-30T12:53:21.176 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.55412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54683.log 2023-10-30T12:53:21.176 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55412.log.gz 2023-10-30T12:53:21.176 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53341.log 2023-10-30T12:53:21.177 INFO:teuthology.orchestra.run.smithi089.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52213.log.gz 2023-10-30T12:53:21.177 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50551.log 2023-10-30T12:53:21.177 INFO:teuthology.orchestra.run.smithi089.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54683.log.gz 2023-10-30T12:53:21.178 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45316.log 2023-10-30T12:53:21.178 INFO:teuthology.orchestra.run.smithi089.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53341.log.gz 2023-10-30T12:53:21.182 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.50551.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50551.log.gz 2023-10-30T12:53:21.183 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52722.log 2023-10-30T12:53:21.183 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.45316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50012.log 2023-10-30T12:53:21.183 INFO:teuthology.orchestra.run.smithi089.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45316.log.gz 2023-10-30T12:53:21.184 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51452.log 2023-10-30T12:53:21.184 INFO:teuthology.orchestra.run.smithi089.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52722.log.gz 2023-10-30T12:53:21.184 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.50012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54899.log 2023-10-30T12:53:21.185 INFO:teuthology.orchestra.run.smithi089.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50012.log.gz 2023-10-30T12:53:21.185 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51452.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51452.log.gz 2023-10-30T12:53:21.185 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44366.log 2023-10-30T12:53:21.185 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44414.log 2023-10-30T12:53:21.186 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54899.log.gz 2023-10-30T12:53:21.186 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53945.log 2023-10-30T12:53:21.186 INFO:teuthology.orchestra.run.smithi089.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44366.log.gz 2023-10-30T12:53:21.187 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50996.log 2023-10-30T12:53:21.187 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44414.log.gz 2023-10-30T12:53:21.187 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53945.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-30T12:53:21.187 INFO:teuthology.orchestra.run.smithi089.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53945.log.gz 2023-10-30T12:53:21.188 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.50996.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50996.log.gz 2023-10-30T12:53:21.188 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52877.log 2023-10-30T12:53:21.188 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55703.log 2023-10-30T12:53:21.188 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52877.log: gzip 88.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.52877.log.gz 2023-10-30T12:53:21.189 INFO:teuthology.orchestra.run.smithi089.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48921.log 2023-10-30T12:53:21.189 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.55703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54317.log 2023-10-30T12:53:21.189 INFO:teuthology.orchestra.run.smithi089.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55703.log.gz 2023-10-30T12:53:21.191 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-10-30T12:53:21.191 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-client.admin.38673.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38673.log.gz 2023-10-30T12:53:21.195 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.48921.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48921.log.gz 2023-10-30T12:53:21.195 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52109.log 2023-10-30T12:53:21.195 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43976.log 2023-10-30T12:53:21.196 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54317.log.gz 2023-10-30T12:53:21.196 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52109.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52109.log.gz 2023-10-30T12:53:21.196 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53893.log 2023-10-30T12:53:21.196 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45290.log 2023-10-30T12:53:21.197 INFO:teuthology.orchestra.run.smithi089.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43976.log.gz 2023-10-30T12:53:21.197 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53893.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53893.log.gz 2023-10-30T12:53:21.197 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50892.log 2023-10-30T12:53:21.197 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.45290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54995.log 2023-10-30T12:53:21.198 INFO:teuthology.orchestra.run.smithi089.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45290.log.gz 2023-10-30T12:53:21.198 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.50892.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50892.log.gz 2023-10-30T12:53:21.198 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52460.log 2023-10-30T12:53:21.199 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54995.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54995.log.gz 2023-10-30T12:53:21.199 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53971.log 2023-10-30T12:53:21.199 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52460.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52460.log.gz 2023-10-30T12:53:21.199 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53043.log 2023-10-30T12:53:21.200 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53971.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53971.log.gz 2023-10-30T12:53:21.200 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49032.log 2023-10-30T12:53:21.200 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49775.log 2023-10-30T12:53:21.200 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53043.log.gz 2023-10-30T12:53:21.201 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38793.log 2023-10-30T12:53:21.204 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.49032.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49032.log.gz 2023-10-30T12:53:21.204 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-30T12:53:21.204 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.49775.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49775.log.gz 2023-10-30T12:53:21.205 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51269.log 2023-10-30T12:53:21.205 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44756.log 2023-10-30T12:53:21.205 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51269.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51269.log.gz 2023-10-30T12:53:21.205 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46763.log 2023-10-30T12:53:21.206 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48974.log 2023-10-30T12:53:21.206 INFO:teuthology.orchestra.run.smithi089.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44756.log.gz 2023-10-30T12:53:21.206 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.46763.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46763.log.gz 2023-10-30T12:53:21.207 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53839.log 2023-10-30T12:53:21.207 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.48974.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48974.log.gz 2023-10-30T12:53:21.207 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52965.log 2023-10-30T12:53:21.209 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53839.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53839.log.gz 2023-10-30T12:53:21.209 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44182.log 2023-10-30T12:53:21.210 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52965.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52965.log.gz 2023-10-30T12:53:21.210 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52187.log 2023-10-30T12:53:21.210 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49165.log 2023-10-30T12:53:21.210 INFO:teuthology.orchestra.run.smithi089.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44182.log.gz 2023-10-30T12:53:21.211 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52187.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52187.log.gz 2023-10-30T12:53:21.211 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46463.log 2023-10-30T12:53:21.212 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.49165.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49165.log.gz 2023-10-30T12:53:21.213 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43754.log 2023-10-30T12:53:21.213 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.46463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48756.log 2023-10-30T12:53:21.213 INFO:teuthology.orchestra.run.smithi089.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46463.log.gz 2023-10-30T12:53:21.214 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43754.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43754.log.gz 2023-10-30T12:53:21.214 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53315.log 2023-10-30T12:53:21.215 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-client.admin.38793.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38793.log.gz 2023-10-30T12:53:21.216 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.48756.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48756.log.gz 2023-10-30T12:53:21.216 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55359.log 2023-10-30T12:53:21.217 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53315.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53315.log.gz 2023-10-30T12:53:21.217 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53180.log 2023-10-30T12:53:21.218 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.55359.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55359.log.gz 2023-10-30T12:53:21.218 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50525.log 2023-10-30T12:53:21.219 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44380.log 2023-10-30T12:53:21.219 INFO:teuthology.orchestra.run.smithi089.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53180.log.gz 2023-10-30T12:53:21.220 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.50525.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50525.log.gz 2023-10-30T12:53:21.220 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48862.log 2023-10-30T12:53:21.221 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44380.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44380.log.gz 2023-10-30T12:53:21.222 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43974.log 2023-10-30T12:53:21.222 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.48862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43322.log 2023-10-30T12:53:21.223 INFO:teuthology.orchestra.run.smithi089.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48862.log.gz 2023-10-30T12:53:21.224 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43974.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43974.log.gz 2023-10-30T12:53:21.225 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43957.log 2023-10-30T12:53:21.225 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43322.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43322.log.gz 2023-10-30T12:53:21.226 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49801.log 2023-10-30T12:53:21.227 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43957.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43957.log.gz 2023-10-30T12:53:21.227 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46896.log 2023-10-30T12:53:21.228 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.49801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50655.log 2023-10-30T12:53:21.229 INFO:teuthology.orchestra.run.smithi089.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49801.log.gz 2023-10-30T12:53:21.230 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.46896.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46896.log.gz 2023-10-30T12:53:21.230 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53709.log 2023-10-30T12:53:21.231 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.50655.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50655.log.gz 2023-10-30T12:53:21.231 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51898.log 2023-10-30T12:53:21.232 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49827.log 2023-10-30T12:53:21.233 INFO:teuthology.orchestra.run.smithi089.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53709.log.gz/var/log/ceph/ceph-client.admin.51898.log: 2023-10-30T12:53:21.233 INFO:teuthology.orchestra.run.smithi089.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51898.log.gz 2023-10-30T12:53:21.234 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55547.log 2023-10-30T12:53:21.235 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.49827.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49827.log.gz 2023-10-30T12:53:21.235 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43716.log 2023-10-30T12:53:21.236 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.55547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50196.log 2023-10-30T12:53:21.237 INFO:teuthology.orchestra.run.smithi089.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55547.log.gz 2023-10-30T12:53:21.238 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43716.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43716.log.gz 2023-10-30T12:53:21.238 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43164.log 2023-10-30T12:53:21.239 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.50196.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50196.log.gz 2023-10-30T12:53:21.240 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45064.log 2023-10-30T12:53:21.241 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43164.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43164.log.gz 2023-10-30T12:53:21.241 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52670.log 2023-10-30T12:53:21.242 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.45064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55255.log 2023-10-30T12:53:21.243 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52670.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.45064.log.gz 2023-10-30T12:53:21.243 INFO:teuthology.orchestra.run.smithi089.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52670.log.gz 2023-10-30T12:53:21.244 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51610.log 2023-10-30T12:53:21.245 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.55255.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55255.log.gz 2023-10-30T12:53:21.245 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53473.log 2023-10-30T12:53:21.246 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51610.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51610.log.gz 2023-10-30T12:53:21.247 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48888.log 2023-10-30T12:53:21.248 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52083.log 2023-10-30T12:53:21.248 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53473.log.gz 2023-10-30T12:53:21.249 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.48888.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48888.log.gz 2023-10-30T12:53:21.249 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54847.log 2023-10-30T12:53:21.250 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52083.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52083.log.gz 2023-10-30T12:53:21.255 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43770.log 2023-10-30T12:53:21.255 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54847.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54847.log.gz 2023-10-30T12:53:21.256 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51347.log 2023-10-30T12:53:21.256 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43770.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43770.log.gz 2023-10-30T12:53:21.256 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55599.log 2023-10-30T12:53:21.256 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51347.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51347.log.gz 2023-10-30T12:53:21.257 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46604.log 2023-10-30T12:53:21.258 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.55599.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55599.log.gz 2023-10-30T12:53:21.258 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55281.log 2023-10-30T12:53:21.259 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.46604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44834.log 2023-10-30T12:53:21.259 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46604.log.gz 2023-10-30T12:53:21.260 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.55281.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55281.log.gz 2023-10-30T12:53:21.261 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.45235.log 2023-10-30T12:53:21.261 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53657.log 2023-10-30T12:53:21.261 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44834.log.gz 2023-10-30T12:53:21.262 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.0.45235.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.45235.log.gz 2023-10-30T12:53:21.262 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52618.log 2023-10-30T12:53:21.263 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53657.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53657.log.gz 2023-10-30T12:53:21.264 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52408.log 2023-10-30T12:53:21.264 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52618.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52618.log.gz 2023-10-30T12:53:21.265 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54765.log 2023-10-30T12:53:21.266 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52408.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52408.log.gz 2023-10-30T12:53:21.266 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46737.log 2023-10-30T12:53:21.267 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54765.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54765.log.gz 2023-10-30T12:53:21.268 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51977.log 2023-10-30T12:53:21.268 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.46737.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46737.log.gz 2023-10-30T12:53:21.269 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47874.log 2023-10-30T12:53:21.270 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51977.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51977.log.gz 2023-10-30T12:53:21.271 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51399.log 2023-10-30T12:53:21.272 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.47874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50577.log 2023-10-30T12:53:21.272 INFO:teuthology.orchestra.run.smithi089.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47874.log.gz 2023-10-30T12:53:21.273 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51399.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51399.log.gz 2023-10-30T12:53:21.274 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54873.log 2023-10-30T12:53:21.275 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.50577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50499.log 2023-10-30T12:53:21.275 INFO:teuthology.orchestra.run.smithi089.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50577.log.gz 2023-10-30T12:53:21.276 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54873.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54873.log.gz 2023-10-30T12:53:21.276 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44938.log 2023-10-30T12:53:21.277 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.50499.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50499.log.gz 2023-10-30T12:53:21.278 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43752.log 2023-10-30T12:53:21.279 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44938.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44938.log.gz 2023-10-30T12:53:21.279 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49006.log 2023-10-30T12:53:21.280 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50787.log 2023-10-30T12:53:21.281 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43752.log.gz 2023-10-30T12:53:21.282 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.49006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47768.log 2023-10-30T12:53:21.282 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49006.log.gz 2023-10-30T12:53:21.283 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.50787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52774.log 2023-10-30T12:53:21.284 INFO:teuthology.orchestra.run.smithi089.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50787.log.gz 2023-10-30T12:53:21.285 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.47768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46656.log 2023-10-30T12:53:21.285 INFO:teuthology.orchestra.run.smithi089.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47768.log.gz 2023-10-30T12:53:21.286 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49932.log 2023-10-30T12:53:21.287 INFO:teuthology.orchestra.run.smithi089.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52774.log.gz 2023-10-30T12:53:21.288 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.46656.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-10-30T12:53:21.289 INFO:teuthology.orchestra.run.smithi089.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46656.log.gz 2023-10-30T12:53:21.290 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.49932.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49932.log.gz 2023-10-30T12:53:21.290 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50813.log 2023-10-30T12:53:21.291 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51243.log 2023-10-30T12:53:21.292 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.50813.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50813.log.gz 2023-10-30T12:53:21.292 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48947.log 2023-10-30T12:53:21.293 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51243.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51243.log.gz 2023-10-30T12:53:21.294 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43560.log 2023-10-30T12:53:21.295 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.48947.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48947.log.gz 2023-10-30T12:53:21.295 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54579.log 2023-10-30T12:53:21.296 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43560.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43560.log.gz 2023-10-30T12:53:21.297 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53553.log 2023-10-30T12:53:21.298 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54579.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54579.log.gz 2023-10-30T12:53:21.298 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47926.log 2023-10-30T12:53:21.300 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53553.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53553.log.gz 2023-10-30T12:53:21.300 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53761.log 2023-10-30T12:53:21.301 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.47926.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47926.log.gz 2023-10-30T12:53:21.302 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43456.log 2023-10-30T12:53:21.303 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53761.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53761.log.gz 2023-10-30T12:53:21.304 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43534.log 2023-10-30T12:53:21.305 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43456.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43456.log.gz 2023-10-30T12:53:21.305 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50918.log 2023-10-30T12:53:21.306 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43534.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43534.log.gz 2023-10-30T12:53:21.307 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51321.log 2023-10-30T12:53:21.308 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.50918.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50918.log.gz 2023-10-30T12:53:21.308 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45119.log 2023-10-30T12:53:21.310 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38089.log 2023-10-30T12:53:21.310 INFO:teuthology.orchestra.run.smithi089.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51321.log.gz 2023-10-30T12:53:21.311 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.45119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44170.log 2023-10-30T12:53:21.311 INFO:teuthology.orchestra.run.smithi089.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45119.log.gz 2023-10-30T12:53:21.313 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.38089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53605.log 2023-10-30T12:53:21.313 INFO:teuthology.orchestra.run.smithi089.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.38089.log.gz 2023-10-30T12:53:21.314 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46489.log 2023-10-30T12:53:21.315 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44170.log.gz 2023-10-30T12:53:21.315 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53605.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53605.log.gz 2023-10-30T12:53:21.316 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49749.log 2023-10-30T12:53:21.317 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.46489.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46489.log.gz 2023-10-30T12:53:21.317 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45342.log 2023-10-30T12:53:21.318 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.49749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51137.log 2023-10-30T12:53:21.318 INFO:teuthology.orchestra.run.smithi089.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49749.log.gz 2023-10-30T12:53:21.319 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.45342.log: 89.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.45342.log.gz 2023-10-30T12:53:21.320 INFO:teuthology.orchestra.run.smithi089.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49880.log 2023-10-30T12:53:21.321 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51872.log 2023-10-30T12:53:21.321 INFO:teuthology.orchestra.run.smithi089.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51137.log.gz 2023-10-30T12:53:21.323 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.49880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54943.log 2023-10-30T12:53:21.323 INFO:teuthology.orchestra.run.smithi089.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49880.log.gz 2023-10-30T12:53:21.324 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49322.log 2023-10-30T12:53:21.324 INFO:teuthology.orchestra.run.smithi089.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51872.log.gz 2023-10-30T12:53:21.326 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43404.log 2023-10-30T12:53:21.326 INFO:teuthology.orchestra.run.smithi089.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54943.log.gz 2023-10-30T12:53:21.327 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.49322.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-30T12:53:21.327 INFO:teuthology.orchestra.run.smithi089.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49322.log.gz 2023-10-30T12:53:21.329 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43404.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43404.log.gz 2023-10-30T12:53:21.329 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52696.log 2023-10-30T12:53:21.330 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54449.log 2023-10-30T12:53:21.332 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53865.log 2023-10-30T12:53:21.332 INFO:teuthology.orchestra.run.smithi089.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52696.log.gz 2023-10-30T12:53:21.333 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54449.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54449.log.gz 2023-10-30T12:53:21.334 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51716.log 2023-10-30T12:53:21.334 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53865.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53865.log.gz 2023-10-30T12:53:21.335 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53997.log 2023-10-30T12:53:21.336 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49615.log 2023-10-30T12:53:21.337 INFO:teuthology.orchestra.run.smithi089.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51716.log.gz 2023-10-30T12:53:21.338 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45524.log 2023-10-30T12:53:21.338 INFO:teuthology.orchestra.run.smithi089.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53997.log.gz 2023-10-30T12:53:21.339 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.49615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52829.log 2023-10-30T12:53:21.340 INFO:teuthology.orchestra.run.smithi089.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49615.log.gz 2023-10-30T12:53:21.340 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.45524.log: gzip 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45524.log.gz 2023-10-30T12:53:21.341 INFO:teuthology.orchestra.run.smithi089.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53446.log 2023-10-30T12:53:21.342 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52829.log: gzip 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52829.log.gz -5 2023-10-30T12:53:21.343 INFO:teuthology.orchestra.run.smithi089.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44179.log 2023-10-30T12:53:21.344 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53393.log 2023-10-30T12:53:21.344 INFO:teuthology.orchestra.run.smithi089.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53446.log.gz 2023-10-30T12:53:21.345 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44179.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44179.log.gz 2023-10-30T12:53:21.346 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51373.log 2023-10-30T12:53:21.347 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54475.log 2023-10-30T12:53:21.347 INFO:teuthology.orchestra.run.smithi089.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53393.log.gz 2023-10-30T12:53:21.348 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43964.log 2023-10-30T12:53:21.349 INFO:teuthology.orchestra.run.smithi089.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51373.log.gz 2023-10-30T12:53:21.350 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54475.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54475.log.gz 2023-10-30T12:53:21.351 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43964.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43964.log.gz 2023-10-30T12:53:21.351 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49217.log 2023-10-30T12:53:21.353 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50970.log 2023-10-30T12:53:21.354 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.49217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55438.log 2023-10-30T12:53:21.355 INFO:teuthology.orchestra.run.smithi089.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49217.log.gz 2023-10-30T12:53:21.356 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.50970.log: gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50970.log.gz -5 2023-10-30T12:53:21.357 INFO:teuthology.orchestra.run.smithi089.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43690.log 2023-10-30T12:53:21.357 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.55438.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55438.log.gz 2023-10-30T12:53:21.358 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44886.log 2023-10-30T12:53:21.359 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52382.log 2023-10-30T12:53:21.359 INFO:teuthology.orchestra.run.smithi089.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43690.log.gz 2023-10-30T12:53:21.361 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53683.log 2023-10-30T12:53:21.361 INFO:teuthology.orchestra.run.smithi089.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44886.log.gz 2023-10-30T12:53:21.362 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52382.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52382.log.gz 2023-10-30T12:53:21.363 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43296.log 2023-10-30T12:53:21.363 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53683.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53683.log.gz 2023-10-30T12:53:21.364 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48207.log 2023-10-30T12:53:21.365 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53209.log 2023-10-30T12:53:21.366 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43296.log.gz 2023-10-30T12:53:21.367 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.48207.log: gzip -5 --verbose 88.5% -- -- replaced with /var/log/ceph/ceph-client.admin.48207.log.gz /var/log/ceph/ceph-client.admin.54108.log 2023-10-30T12:53:21.367 INFO:teuthology.orchestra.run.smithi089.stderr: 2023-10-30T12:53:21.368 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54821.log 88.1% 2023-10-30T12:53:21.369 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53209.log.gz 2023-10-30T12:53:21.369 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54108.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.54108.log.gz 2023-10-30T12:53:21.370 INFO:teuthology.orchestra.run.smithi089.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54605.log 2023-10-30T12:53:21.371 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44389.log 2023-10-30T12:53:21.371 INFO:teuthology.orchestra.run.smithi089.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54821.log.gz 2023-10-30T12:53:21.373 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47690.log 2023-10-30T12:53:21.373 INFO:teuthology.orchestra.run.smithi089.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54605.log.gz 2023-10-30T12:53:21.374 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44389.log: gzip -5 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44389.log.gz 2023-10-30T12:53:21.375 INFO:teuthology.orchestra.run.smithi089.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49723.log 2023-10-30T12:53:21.376 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.47690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46411.log 2023-10-30T12:53:21.376 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47690.log.gz 2023-10-30T12:53:21.377 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.49723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52329.log 2023-10-30T12:53:21.377 INFO:teuthology.orchestra.run.smithi089.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49723.log.gz 2023-10-30T12:53:21.379 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.46411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53919.log 2023-10-30T12:53:21.379 INFO:teuthology.orchestra.run.smithi089.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46411.log.gz 2023-10-30T12:53:21.380 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53154.log 2023-10-30T12:53:21.380 INFO:teuthology.orchestra.run.smithi089.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52329.log.gz 2023-10-30T12:53:21.381 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45706.log 2023-10-30T12:53:21.382 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53919.log.gz 2023-10-30T12:53:21.383 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51504.log 2023-10-30T12:53:21.383 INFO:teuthology.orchestra.run.smithi089.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53154.log.gz 2023-10-30T12:53:21.384 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.45706.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45706.log.gz 2023-10-30T12:53:21.385 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46842.log 2023-10-30T12:53:21.386 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55073.log 2023-10-30T12:53:21.386 INFO:teuthology.orchestra.run.smithi089.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51504.log.gz 2023-10-30T12:53:21.387 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-30T12:53:21.388 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.46842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44782.log 2023-10-30T12:53:21.389 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46842.log.gz 2023-10-30T12:53:21.389 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.55073.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55073.log.gz 2023-10-30T12:53:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53017.log 2023-10-30T12:53:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55385.log 2023-10-30T12:53:21.391 INFO:teuthology.orchestra.run.smithi089.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44782.log.gz 2023-10-30T12:53:21.392 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45550.log 2023-10-30T12:53:21.392 INFO:teuthology.orchestra.run.smithi089.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53017.log.gz 2023-10-30T12:53:21.393 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.55385.log: gzip 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55385.log.gz 2023-10-30T12:53:21.394 INFO:teuthology.orchestra.run.smithi089.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54527.log 2023-10-30T12:53:21.395 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.45550.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45550.log.gz 2023-10-30T12:53:21.395 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52161.log 2023-10-30T12:53:21.396 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54527.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-30T12:53:21.397 INFO:teuthology.orchestra.run.smithi089.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54527.log.gz 2023-10-30T12:53:21.398 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49111.log 2023-10-30T12:53:21.398 INFO:teuthology.orchestra.run.smithi089.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52161.log.gz 2023-10-30T12:53:21.399 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44626.log 2023-10-30T12:53:21.401 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.49111.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49111.log.gz 2023-10-30T12:53:21.402 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44626.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44626.log.gz 2023-10-30T12:53:21.403 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43482.log 2023-10-30T12:53:21.404 INFO:teuthology.orchestra.run.smithi089.stderr: 87.4% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-30T12:53:21.404 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46545.log 2023-10-30T12:53:21.405 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51768.log 2023-10-30T12:53:21.406 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43482.log.gz 2023-10-30T12:53:21.407 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.46545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44186.log 2023-10-30T12:53:21.407 INFO:teuthology.orchestra.run.smithi089.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46545.log.gz 2023-10-30T12:53:21.408 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55307.log 2023-10-30T12:53:21.409 INFO:teuthology.orchestra.run.smithi089.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51768.log.gz 2023-10-30T12:53:21.410 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50759.log 2023-10-30T12:53:21.410 INFO:teuthology.orchestra.run.smithi089.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44186.log.gz 2023-10-30T12:53:21.411 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.55307.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55307.log.gz 2023-10-30T12:53:21.412 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51556.log 2023-10-30T12:53:21.413 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.50759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49085.log 2023-10-30T12:53:21.413 INFO:teuthology.orchestra.run.smithi089.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50759.log.gz 2023-10-30T12:53:21.414 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53579.log 2023-10-30T12:53:21.415 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51556.log.gz 2023-10-30T12:53:21.416 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.49085.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49085.log.gz 2023-10-30T12:53:21.416 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46517.log 2023-10-30T12:53:21.417 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53579.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53579.log.gz 2023-10-30T12:53:21.418 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53289.log 2023-10-30T12:53:21.419 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.46517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53095.log 2023-10-30T12:53:21.420 INFO:teuthology.orchestra.run.smithi089.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46517.log.gz 2023-10-30T12:53:21.421 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53289.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53289.log.gz 2023-10-30T12:53:21.421 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51023.log 2023-10-30T12:53:21.422 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53095.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53095.log.gz 2023-10-30T12:53:21.423 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51846.log 2023-10-30T12:53:21.424 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51023.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51023.log.gz 2023-10-30T12:53:21.424 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46816.log 2023-10-30T12:53:21.425 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51846.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51846.log.gz 2023-10-30T12:53:21.426 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54081.log 2023-10-30T12:53:21.427 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.46816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44912.log 2023-10-30T12:53:21.428 INFO:teuthology.orchestra.run.smithi089.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46816.log.gz 2023-10-30T12:53:21.429 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54081.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54081.log.gz 2023-10-30T12:53:21.429 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45264.log 2023-10-30T12:53:21.430 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44912.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44912.log.gz 2023-10-30T12:53:21.431 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46359.log 2023-10-30T12:53:21.432 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.45264.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45264.log.gz 2023-10-30T12:53:21.432 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46790.log 2023-10-30T12:53:21.434 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.46359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53631.log 2023-10-30T12:53:21.434 INFO:teuthology.orchestra.run.smithi089.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46359.log.gz 2023-10-30T12:53:21.435 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.46790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54186.log 2023-10-30T12:53:21.435 INFO:teuthology.orchestra.run.smithi089.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46790.log.gz 2023-10-30T12:53:21.437 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50603.log 2023-10-30T12:53:21.437 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53631.log.gz 2023-10-30T12:53:21.438 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47794.log 2023-10-30T12:53:21.438 INFO:teuthology.orchestra.run.smithi089.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54186.log.gz 2023-10-30T12:53:21.439 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.50603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52913.log 2023-10-30T12:53:21.440 INFO:teuthology.orchestra.run.smithi089.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50603.log.gz 2023-10-30T12:53:21.441 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.47794.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47794.log.gz 2023-10-30T12:53:21.441 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50733.log 2023-10-30T12:53:21.443 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47900.log 2023-10-30T12:53:21.443 INFO:teuthology.orchestra.run.smithi089.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52913.log.gz 2023-10-30T12:53:21.444 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.50733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55203.log 2023-10-30T12:53:21.444 INFO:teuthology.orchestra.run.smithi089.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50733.log.gz 2023-10-30T12:53:21.445 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.47900.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47900.log.gz 2023-10-30T12:53:21.447 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.55203.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55203.log.gz 2023-10-30T12:53:21.447 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46685.log 2023-10-30T12:53:21.449 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52991.log 2023-10-30T12:53:21.450 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.46685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53237.log 2023-10-30T12:53:21.451 INFO:teuthology.orchestra.run.smithi089.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46685.log.gz 2023-10-30T12:53:21.451 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52991.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52991.log.gz 2023-10-30T12:53:21.453 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53237.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53237.log.gz 2023-10-30T12:53:21.454 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54395.log 2023-10-30T12:53:21.455 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43270.log 2023-10-30T12:53:21.457 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44158.log 2023-10-30T12:53:21.457 INFO:teuthology.orchestra.run.smithi089.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54395.log.gz 2023-10-30T12:53:21.458 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43664.log 2023-10-30T12:53:21.458 INFO:teuthology.orchestra.run.smithi089.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43270.log.gz 2023-10-30T12:53:21.460 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44158.log: gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44158.log.gz -5 2023-10-30T12:53:21.460 INFO:teuthology.orchestra.run.smithi089.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48030.log 2023-10-30T12:53:21.461 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50142.log 2023-10-30T12:53:21.461 INFO:teuthology.orchestra.run.smithi089.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43664.log.gz 2023-10-30T12:53:21.462 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.48030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50707.log 2023-10-30T12:53:21.463 INFO:teuthology.orchestra.run.smithi089.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48030.log.gz 2023-10-30T12:53:21.463 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.50142.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50142.log.gz 2023-10-30T12:53:21.464 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55774.log 2023-10-30T12:53:21.466 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.50707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42986.log 2023-10-30T12:53:21.466 INFO:teuthology.orchestra.run.smithi089.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50707.log.gz 2023-10-30T12:53:21.467 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.55774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49243.log 2023-10-30T12:53:21.467 INFO:teuthology.orchestra.run.smithi089.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55774.log.gz 2023-10-30T12:53:21.468 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.42986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52800.log 2023-10-30T12:53:21.469 INFO:teuthology.orchestra.run.smithi089.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42986.log.gz 2023-10-30T12:53:21.470 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.49243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47952.log 2023-10-30T12:53:21.470 INFO:teuthology.orchestra.run.smithi089.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49243.log.gz 2023-10-30T12:53:21.471 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48810.log 2023-10-30T12:53:21.472 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52800.log.gz 2023-10-30T12:53:21.473 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.47952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47489.log 2023-10-30T12:53:21.473 INFO:teuthology.orchestra.run.smithi089.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47952.log.gz 2023-10-30T12:53:21.474 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.48810.log: gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48810.log.gz 2023-10-30T12:53:21.475 INFO:teuthology.orchestra.run.smithi089.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45628.log 2023-10-30T12:53:21.475 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.47489.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47489.log.gz 2023-10-30T12:53:21.476 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44177.log 2023-10-30T12:53:21.477 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.45628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47664.log 2023-10-30T12:53:21.478 INFO:teuthology.orchestra.run.smithi089.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45628.log.gz 2023-10-30T12:53:21.479 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43190.log 2023-10-30T12:53:21.479 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44177.log.gz 2023-10-30T12:53:21.480 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.47664.log: 89.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.47664.log.gz -5 2023-10-30T12:53:21.481 INFO:teuthology.orchestra.run.smithi089.stderr: --verbose -- /var/log/ceph/ceph-client.admin.54737.log 2023-10-30T12:53:21.482 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44600.log 2023-10-30T12:53:21.482 INFO:teuthology.orchestra.run.smithi089.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43190.log.gz 2023-10-30T12:53:21.483 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54291.log 2023-10-30T12:53:21.484 INFO:teuthology.orchestra.run.smithi089.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54737.log.gz 2023-10-30T12:53:21.484 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44600.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44600.log.gz 2023-10-30T12:53:21.485 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45680.log 2023-10-30T12:53:21.486 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51951.log 2023-10-30T12:53:21.486 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54291.log.gz 2023-10-30T12:53:21.488 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.45680.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45680.log.gz 2023-10-30T12:53:21.488 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43080.log 2023-10-30T12:53:21.489 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51951.log: 87.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.51951.log.gz -5 2023-10-30T12:53:21.490 INFO:teuthology.orchestra.run.smithi089.stderr: --verbose -- /var/log/ceph/ceph-client.admin.54423.log 2023-10-30T12:53:21.491 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43080.log: gzip 88.7% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.43080.log.gz -- 2023-10-30T12:53:21.491 INFO:teuthology.orchestra.run.smithi089.stderr: /var/log/ceph/ceph-client.admin.52055.log 2023-10-30T12:53:21.492 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54423.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54423.log.gz 2023-10-30T12:53:21.493 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50168.log 2023-10-30T12:53:21.494 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52055.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52055.log.gz 2023-10-30T12:53:21.494 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49854.log 2023-10-30T12:53:21.495 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.50168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46711.log 2023-10-30T12:53:21.496 INFO:teuthology.orchestra.run.smithi089.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50168.log.gz 2023-10-30T12:53:21.496 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.49854.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49854.log.gz 2023-10-30T12:53:21.497 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54023.log 2023-10-30T12:53:21.499 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.46711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51794.log 2023-10-30T12:53:21.499 INFO:teuthology.orchestra.run.smithi089.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46711.log.gz 2023-10-30T12:53:21.500 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54023.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54023.log.gz 2023-10-30T12:53:21.500 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43612.log 2023-10-30T12:53:21.502 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53813.log 2023-10-30T12:53:21.502 INFO:teuthology.orchestra.run.smithi089.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51794.log.gz 2023-10-30T12:53:21.503 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43612.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43612.log.gz 2023-10-30T12:53:21.504 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-30T12:53:21.504 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53813.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53813.log.gz 2023-10-30T12:53:21.505 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55047.log 2023-10-30T12:53:21.506 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2023-10-30T12:53:21.508 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.55047.log: 88.2% 89.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-30T12:53:21.508 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55047.log.gz 2023-10-30T12:53:21.512 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44387.log 2023-10-30T12:53:21.513 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51690.log 2023-10-30T12:53:21.515 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44387.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44387.log.gz 2023-10-30T12:53:21.515 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52644.log 2023-10-30T12:53:21.516 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51690.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51690.log.gz 2023-10-30T12:53:21.518 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52644.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52644.log.gz 2023-10-30T12:53:21.518 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43638.log 2023-10-30T12:53:21.520 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54264.log 2023-10-30T12:53:21.521 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43638.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43638.log.gz 2023-10-30T12:53:21.522 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54264.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54264.log.gz 2023-10-30T12:53:21.523 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2023-10-30T12:53:21.525 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43767.log 2023-10-30T12:53:21.527 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.43767.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43767.log.gz 2023-10-30T12:53:21.528 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46437.log 2023-10-30T12:53:21.530 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52566.log 2023-10-30T12:53:21.531 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.46437.log: 94.2% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-10-30T12:53:21.532 INFO:teuthology.orchestra.run.smithi089.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46437.log.gz 2023-10-30T12:53:21.533 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49986.log 2023-10-30T12:53:21.534 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52566.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.52566.log.gz 2023-10-30T12:53:21.536 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.49986.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49986.log.gz 2023-10-30T12:53:21.536 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54160.log 2023-10-30T12:53:21.538 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55575.log 2023-10-30T12:53:21.539 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54160.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54160.log.gz 2023-10-30T12:53:21.540 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.55575.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55575.log.gz 2023-10-30T12:53:21.541 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44808.log 2023-10-30T12:53:21.542 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52540.log 2023-10-30T12:53:21.544 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44808.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44808.log.gz 2023-10-30T12:53:21.545 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52540.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52540.log.gz 2023-10-30T12:53:21.546 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54212.log 2023-10-30T12:53:21.547 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50944.log 2023-10-30T12:53:21.549 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54212.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54212.log.gz 2023-10-30T12:53:21.550 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.50944.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50944.log.gz 2023-10-30T12:53:21.550 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54369.log 2023-10-30T12:53:21.552 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53419.log 2023-10-30T12:53:21.553 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54369.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54369.log.gz 2023-10-30T12:53:21.554 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53419.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53419.log.gz 2023-10-30T12:53:21.555 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44704.log 2023-10-30T12:53:21.556 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51530.log 2023-10-30T12:53:21.558 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44704.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44704.log.gz 2023-10-30T12:53:21.559 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51530.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51530.log.gz 2023-10-30T12:53:21.559 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48836.log 2023-10-30T12:53:21.561 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51820.log 2023-10-30T12:53:21.562 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.48836.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48836.log.gz 2023-10-30T12:53:21.563 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51820.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.51820.log.gz 2023-10-30T12:53:21.564 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55177.log 2023-10-30T12:53:21.565 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46571.log 2023-10-30T12:53:21.567 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.55177.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55177.log.gz 2023-10-30T12:53:21.568 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.46571.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46571.log.gz 2023-10-30T12:53:21.569 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51111.log 2023-10-30T12:53:21.570 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51163.log 2023-10-30T12:53:21.571 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51111.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51111.log.gz 2023-10-30T12:53:21.573 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51163.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51163.log.gz 2023-10-30T12:53:21.573 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47742.log 2023-10-30T12:53:21.575 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43378.log 2023-10-30T12:53:21.576 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.47742.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47742.log.gz 2023-10-30T12:53:21.577 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43378.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43378.log.gz 2023-10-30T12:53:21.578 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48730.log 2023-10-30T12:53:21.579 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53367.log 2023-10-30T12:53:21.581 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.48730.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48730.log.gz 2023-10-30T12:53:21.582 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53367.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53367.log.gz 2023-10-30T12:53:21.583 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47532.log 2023-10-30T12:53:21.584 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54969.log 2023-10-30T12:53:21.585 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.47532.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47532.log.gz 2023-10-30T12:53:21.586 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54969.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54969.log.gz 2023-10-30T12:53:21.587 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45420.log 2023-10-30T12:53:21.588 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43218.log 2023-10-30T12:53:21.590 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.45420.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45420.log.gz 2023-10-30T12:53:21.591 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43218.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43218.log.gz 2023-10-30T12:53:21.592 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50038.log 2023-10-30T12:53:21.593 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52939.log 2023-10-30T12:53:21.594 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.50038.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50038.log.gz 2023-10-30T12:53:21.595 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52939.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52939.log.gz 2023-10-30T12:53:21.596 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43769.log 2023-10-30T12:53:21.598 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55651.log 2023-10-30T12:53:21.599 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43769.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43769.log.gz 2023-10-30T12:53:21.600 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.55651.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55651.log.gz 2023-10-30T12:53:21.601 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51662.log 2023-10-30T12:53:21.602 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48704.log 2023-10-30T12:53:21.604 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51662.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51662.log.gz 2023-10-30T12:53:21.605 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.48704.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48704.log.gz 2023-10-30T12:53:21.605 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43586.log 2023-10-30T12:53:21.607 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52748.log 2023-10-30T12:53:21.609 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43586.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43586.log.gz 2023-10-30T12:53:21.610 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52748.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52748.log.gz 2023-10-30T12:53:21.610 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47848.log 2023-10-30T12:53:21.612 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43977.log 2023-10-30T12:53:21.613 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.47848.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47848.log.gz 2023-10-30T12:53:21.614 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43977.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43977.log.gz 2023-10-30T12:53:21.615 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44964.log 2023-10-30T12:53:21.616 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55729.log 2023-10-30T12:53:21.618 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44964.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.44964.log.gz 2023-10-30T12:53:21.619 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.55729.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55729.log.gz 2023-10-30T12:53:21.619 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55677.log 2023-10-30T12:53:21.621 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43042.log 2023-10-30T12:53:21.622 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.55677.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55677.log.gz 2023-10-30T12:53:21.624 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43042.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43042.log.gz 2023-10-30T12:53:21.624 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47822.log 2023-10-30T12:53:21.625 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43430.log 2023-10-30T12:53:21.627 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.47822.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47822.log.gz 2023-10-30T12:53:21.628 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43430.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.43430.log.gz 2023-10-30T12:53:21.629 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45654.log 2023-10-30T12:53:21.630 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47560.log 2023-10-30T12:53:21.632 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.45654.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45654.log.gz 2023-10-30T12:53:21.632 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.47560.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47560.log.gz 2023-10-30T12:53:21.633 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55021.log 2023-10-30T12:53:21.635 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49697.log 2023-10-30T12:53:21.636 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.55021.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55021.log.gz 2023-10-30T12:53:21.637 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.49697.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49697.log.gz 2023-10-30T12:53:21.638 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51049.log 2023-10-30T12:53:21.642 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44860.log 2023-10-30T12:53:21.643 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51049.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.51049.log.gz 2023-10-30T12:53:21.643 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44860.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44860.log.gz 2023-10-30T12:53:21.643 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44391.log 2023-10-30T12:53:21.644 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54631.log 2023-10-30T12:53:21.646 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44391.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44391.log.gz 2023-10-30T12:53:21.647 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54631.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54631.log.gz 2023-10-30T12:53:21.647 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47612.log 2023-10-30T12:53:21.649 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47978.log 2023-10-30T12:53:21.650 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.47612.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47612.log.gz 2023-10-30T12:53:21.651 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.47978.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47978.log.gz 2023-10-30T12:53:21.652 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43970.log 2023-10-30T12:53:21.654 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53263.log 2023-10-30T12:53:21.655 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43970.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43970.log.gz 2023-10-30T12:53:21.656 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53263.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53263.log.gz 2023-10-30T12:53:21.657 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53787.log 2023-10-30T12:53:21.658 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54709.log 2023-10-30T12:53:21.660 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53787.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53787.log.gz 2023-10-30T12:53:21.661 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.54709.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54709.log.gz 2023-10-30T12:53:21.662 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48004.log 2023-10-30T12:53:21.663 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50839.log 2023-10-30T12:53:21.665 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.48004.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48004.log.gz 2023-10-30T12:53:21.666 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.50839.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50839.log.gz 2023-10-30T12:53:21.666 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50473.log 2023-10-30T12:53:21.668 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44370.log 2023-10-30T12:53:21.670 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.50473.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50473.log.gz 2023-10-30T12:53:21.671 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44370.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44370.log.gz 2023-10-30T12:53:21.671 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49137.log 2023-10-30T12:53:21.673 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48782.log 2023-10-30T12:53:21.674 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.49137.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49137.log.gz 2023-10-30T12:53:21.675 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.48782.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48782.log.gz 2023-10-30T12:53:21.676 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43763.log 2023-10-30T12:53:21.677 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51217.log 2023-10-30T12:53:21.679 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43763.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43763.log.gz 2023-10-30T12:53:21.680 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51217.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51217.log.gz 2023-10-30T12:53:21.681 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49906.log 2023-10-30T12:53:21.682 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45576.log 2023-10-30T12:53:21.684 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.49906.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49906.log.gz 2023-10-30T12:53:21.685 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.45576.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45576.log.gz 2023-10-30T12:53:21.685 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55492.log 2023-10-30T12:53:21.687 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53527.log 2023-10-30T12:53:21.688 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.55492.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55492.log.gz 2023-10-30T12:53:21.690 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.53527.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53527.log.gz 2023-10-30T12:53:21.690 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44180.log 2023-10-30T12:53:21.692 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49191.log 2023-10-30T12:53:21.695 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44180.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44180.log.gz 2023-10-30T12:53:21.695 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.49191.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49191.log.gz 2023-10-30T12:53:21.695 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52135.log 2023-10-30T12:53:21.697 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55099.log 2023-10-30T12:53:21.698 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.52135.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52135.log.gz 2023-10-30T12:53:21.699 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.55099.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55099.log.gz 2023-10-30T12:53:21.700 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43350.log 2023-10-30T12:53:21.701 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51189.log 2023-10-30T12:53:21.703 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43350.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43350.log.gz 2023-10-30T12:53:21.704 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49269.log 2023-10-30T12:53:21.706 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51189.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51189.log.gz 2023-10-30T12:53:21.706 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49671.log 2023-10-30T12:53:21.707 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.49269.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49269.log.gz 2023-10-30T12:53:21.708 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.49671.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49671.log.gz 2023-10-30T12:53:21.709 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55333.log 2023-10-30T12:53:21.710 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44172.log 2023-10-30T12:53:21.712 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.55333.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55333.log.gz 2023-10-30T12:53:21.713 INFO:teuthology.orchestra.run.smithi192.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-10-30T12:53:21.713 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.44172.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44172.log.gz 2023-10-30T12:53:21.714 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49645.log 2023-10-30T12:53:21.715 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2023-10-30T12:53:21.716 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.49645.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49645.log.gz 2023-10-30T12:53:21.720 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55625.log 2023-10-30T12:53:21.721 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45498.log 2023-10-30T12:53:21.722 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.55625.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55625.log.gz 2023-10-30T12:53:21.724 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51742.log 2023-10-30T12:53:21.725 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.45498.log: 94.3% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2023-10-30T12:53:21.726 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55518.log 2023-10-30T12:53:21.727 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.51742.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45498.log.gz 2023-10-30T12:53:21.727 INFO:teuthology.orchestra.run.smithi089.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51742.log.gz 2023-10-30T12:53:21.729 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.55518.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55518.log.gz 2023-10-30T12:53:21.730 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47716.log 2023-10-30T12:53:21.731 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43757.log 2023-10-30T12:53:21.733 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.47716.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47716.log.gz 2023-10-30T12:53:21.734 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.43757.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43757.log.gz 2023-10-30T12:53:21.876 INFO:teuthology.orchestra.run.smithi192.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-10-30T12:53:22.557 INFO:teuthology.orchestra.run.smithi089.stderr: 90.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-30T12:53:23.285 INFO:teuthology.orchestra.run.smithi089.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-30T12:53:23.948 INFO:teuthology.orchestra.run.smithi089.stderr: 94.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-30T12:53:24.643 INFO:teuthology.orchestra.run.smithi192.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-10-30T12:53:24.823 INFO:teuthology.orchestra.run.smithi089.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-10-30T12:53:24.952 INFO:teuthology.orchestra.run.smithi089.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-30T12:53:25.119 INFO:teuthology.orchestra.run.smithi192.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-10-30T12:53:25.905 INFO:teuthology.orchestra.run.smithi192.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-10-30T12:53:26.261 INFO:teuthology.orchestra.run.smithi192.stderr: 94.5% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-10-30T12:53:37.661 INFO:teuthology.orchestra.run.smithi192.stderr: 93.6% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-10-30T12:53:39.786 INFO:teuthology.orchestra.run.smithi192.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-10-30T12:53:40.692 INFO:teuthology.orchestra.run.smithi192.stderr: 94.0% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2023-10-30T12:53:40.693 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-10-30T12:53:40.693 INFO:teuthology.orchestra.run.smithi192.stderr:real 0m19.579s 2023-10-30T12:53:40.694 INFO:teuthology.orchestra.run.smithi192.stderr:user 1m10.532s 2023-10-30T12:53:40.694 INFO:teuthology.orchestra.run.smithi192.stderr:sys 0m2.381s 2023-10-30T12:53:41.035 INFO:teuthology.orchestra.run.smithi089.stderr: 93.6% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-10-30T12:53:41.037 INFO:teuthology.orchestra.run.smithi089.stderr: 2023-10-30T12:53:41.037 INFO:teuthology.orchestra.run.smithi089.stderr:real 0m19.959s 2023-10-30T12:53:41.037 INFO:teuthology.orchestra.run.smithi089.stderr:user 0m31.211s 2023-10-30T12:53:41.037 INFO:teuthology.orchestra.run.smithi089.stderr:sys 0m3.004s 2023-10-30T12:53:41.039 INFO:tasks.ceph:Archiving logs... 2023-10-30T12:53:41.039 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/log/ceph to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441018/remote/smithi089/log 2023-10-30T12:53:41.041 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-30T12:53:44.918 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/log/ceph to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441018/remote/smithi192/log 2023-10-30T12:53:44.919 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-30T12:53:52.218 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-30T12:53:52.228 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-30T12:53:52.228 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-30T12:53:52.233 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-30T12:53:52.312 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-30T12:53:52.322 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-30T12:53:52.323 DEBUG:teuthology.orchestra.run.smithi089:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T12:53:52.327 DEBUG:teuthology.orchestra.run.smithi192:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T12:53:52.339 INFO:teuthology.orchestra.run.smithi089.stdout: remote refid st t when poll reach delay offset jitter 2023-10-30T12:53:52.340 INFO:teuthology.orchestra.run.smithi089.stdout:============================================================================== 2023-10-30T12:53:52.340 INFO:teuthology.orchestra.run.smithi089.stdout:*hv01.front.sepi 44.4.53.2 2 u 34 64 377 0.098 -8.029 0.843 2023-10-30T12:53:52.340 INFO:teuthology.orchestra.run.smithi089.stdout:+hv02.front.sepi 63.231.80.2 3 u 33 64 377 0.098 -4.221 0.814 2023-10-30T12:53:52.340 INFO:teuthology.orchestra.run.smithi089.stdout:+hv03.front.sepi 74.6.168.72 3 u 30 64 377 0.092 -5.091 1.095 2023-10-30T12:53:52.341 INFO:teuthology.orchestra.run.smithi089.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-10-30T12:53:52.342 INFO:teuthology.orchestra.run.smithi192.stdout: remote refid st t when poll reach delay offset jitter 2023-10-30T12:53:52.343 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================== 2023-10-30T12:53:52.343 INFO:teuthology.orchestra.run.smithi192.stdout:*hv01.front.sepi 44.4.53.2 2 u 33 64 377 0.080 -6.366 0.537 2023-10-30T12:53:52.343 INFO:teuthology.orchestra.run.smithi192.stdout:+hv02.front.sepi 63.231.80.2 3 u 35 64 377 0.115 -2.222 0.390 2023-10-30T12:53:52.343 INFO:teuthology.orchestra.run.smithi192.stdout:+hv03.front.sepi 74.6.168.72 3 u 21 64 377 0.123 -4.827 0.635 2023-10-30T12:53:52.344 INFO:teuthology.orchestra.run.smithi192.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-10-30T12:53:52.344 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-30T12:53:52.353 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-30T12:53:52.354 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-30T12:53:52.362 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-30T12:53:52.371 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-30T12:53:52.381 INFO:teuthology.task.internal:Duration was 2255.043195 seconds 2023-10-30T12:53:52.381 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-30T12:53:52.390 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-30T12:53:52.391 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-30T12:53:52.395 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-30T12:53:52.493 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-30T12:53:52.494 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi089.front.sepia.ceph.com 2023-10-30T12:53:52.494 DEBUG:teuthology.orchestra.run.smithi089:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-30T12:53:52.517 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi192.front.sepia.ceph.com 2023-10-30T12:53:52.517 DEBUG:teuthology.orchestra.run.smithi192:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-30T12:53:52.526 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-30T12:53:52.527 DEBUG:teuthology.orchestra.run.smithi089:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-30T12:53:52.566 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-30T12:53:52.599 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-30T12:53:52.600 DEBUG:teuthology.orchestra.run.smithi089:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-30T12:53:52.650 DEBUG:teuthology.orchestra.run.smithi192:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-30T12:53:52.724 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-30T12:53:52.734 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-30T12:53:52.734 DEBUG:teuthology.orchestra.run.smithi089:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-30T12:53:52.755 DEBUG:teuthology.orchestra.run.smithi192:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-30T12:53:52.767 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-30T12:53:52.777 DEBUG:teuthology.orchestra.run.smithi089:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-30T12:53:52.802 DEBUG:teuthology.orchestra.run.smithi192:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-30T12:53:52.817 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = core 2023-10-30T12:53:52.821 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = core 2023-10-30T12:53:52.847 DEBUG:teuthology.orchestra.run.smithi089:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-30T12:53:52.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:53:52.882 DEBUG:teuthology.orchestra.run.smithi192:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-30T12:53:52.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:53:52.894 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-30T12:53:52.904 INFO:teuthology.task.internal:Transferring archived files... 2023-10-30T12:53:52.905 DEBUG:teuthology.misc:Transferring archived files from smithi089:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441018/remote/smithi089 2023-10-30T12:53:52.906 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-30T12:53:52.951 DEBUG:teuthology.misc:Transferring archived files from smithi192:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441018/remote/smithi192 2023-10-30T12:53:52.952 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-30T12:53:52.969 INFO:teuthology.task.internal:Removing archive directory... 2023-10-30T12:53:52.970 DEBUG:teuthology.orchestra.run.smithi089:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-30T12:53:52.994 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-30T12:53:53.015 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-30T12:53:53.024 INFO:teuthology.task.internal:Not uploading archives. 2023-10-30T12:53:53.025 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-30T12:53:53.035 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-30T12:53:53.035 DEBUG:teuthology.orchestra.run.smithi089:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-30T12:53:53.046 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-30T12:53:53.051 INFO:teuthology.orchestra.run.smithi089.stdout: 657973 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 30 12:53 /home/ubuntu/cephtest 2023-10-30T12:53:53.063 INFO:teuthology.orchestra.run.smithi192.stdout: 524293 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 30 12:53 /home/ubuntu/cephtest 2023-10-30T12:53:53.064 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-30T12:53:53.074 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-30T12:53:53.097 INFO:teuthology.nuke:Checking targets against current locks 2023-10-30T12:53:53.124 DEBUG:teuthology.nuke:shortname: smithi089 2023-10-30T12:53:53.124 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T12:53:53.148 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441018', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 12:05:16.663906', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOag3my8IamJNicmi37LZgXQEl7nS8WHId2QtJcbtUnA2zSfW/3voFcVlZ9NqpZOgjD5TywEhthoXXM1YcaRGLg='} 2023-10-30T12:53:53.151 DEBUG:teuthology.nuke:shortname: smithi192 2023-10-30T12:53:53.152 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T12:53:53.175 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441018', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 12:05:16.662130', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJcRRf5xTajm0BZJQJLWhF2zIJij3ZMLY/mNaetV4sVgWPO7jUH10GB2CXSpPhG7M4/lSP5SCEra8BopJzAdnzY='} 2023-10-30T12:53:53.198 INFO:teuthology.orchestra.console.smithi089:Power off 2023-10-30T12:53:53.199 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-30T12:53:53.221 DEBUG:teuthology.orchestra.console.smithi089:power off output: Chassis Power Control: Down/Off 2023-10-30T12:53:53.222 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T12:53:53.238 INFO:teuthology.orchestra.console.smithi192:Power off 2023-10-30T12:53:53.239 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-30T12:53:53.248 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is on 2023-10-30T12:53:53.260 DEBUG:teuthology.orchestra.console.smithi192:power off output: Chassis Power Control: Down/Off 2023-10-30T12:53:53.260 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T12:53:53.283 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2023-10-30T12:53:57.250 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T12:53:57.284 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T12:53:57.360 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is on 2023-10-30T12:53:57.394 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2023-10-30T12:54:01.362 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T12:54:01.395 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T12:54:01.472 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is on 2023-10-30T12:54:01.505 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is off 2023-10-30T12:54:01.606 INFO:teuthology.orchestra.console.smithi192:Power off completed 2023-10-30T12:54:05.473 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T12:54:05.583 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is off 2023-10-30T12:54:05.684 INFO:teuthology.orchestra.console.smithi089:Power off completed 2023-10-30T12:54:05.790 INFO:teuthology.run:Summary data: client.0-kernel-sha1: eacf00cc51e204f401b4d5d4f84825a68f373c74 description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} duration: 2255.0431954860687 failure_reason: 'Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi089 with status 128: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh''' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=7a6e87c27f194713999192fee0906059 status: fail success: false 2023-10-30T12:54:05.791 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-30T12:54:05.871 INFO:teuthology.run:FAIL